farblog

In a footnote to a previous post, I mentioned that I’d recently done some TCP tuning to increase initcwnd on this website, which runs on Google Compute Engine (GCE from here on out). Configuring this turned out to be surprisingly tricky, so I thought it would be useful to write up how it all fits together and how to do this on GCE.

Note that I’m specifically going to be talking about the Debian 13 (Trixie) cloud images, which configure networking via systemd-networkd managed by Netplan. Netplan isn’t critical here, but if you’re not using systemd-networkd, this post probably won’t be that useful.

First, what was I trying to achieve?

initcwnd is the TCP initial congestion window, and controls how much data a server can transmit at the start of a connection before it needs to wait for an acknowledgement from the client.

Linux 2.6.39+ defaults to an initcwnd of 10 segments¹ (up to 10 × MSS bytes). On GCE, where the default virtual network MTU is 1460 bytes, this allows transmitting almost 14 KiB in a single go. For a webserver, this means that if an HTTP response (including headers) is larger than that, the server must pause partway to wait for the client to send a TCP ACK, and only then continue with the remainder of the response.

Increasing initcwnd to 20 segments is therefore a quick way to reduce latency for small (approximately, 14–28 KiB) responses, since it effectively cuts their transfer time in half. For most webservers, there’s quite a lot that fits in that size range: for example, while this post by itself is comfortably smaller than 14 KiB (assuming compression), my front page (compressed) will typically be just a little larger than that threshold.

The hard limit for the initial TCP window is 65535 bytes, which in practice would be about 44 segments². However, setting initcwnd higher than about 20 is usually not recommended: sending too many packets can overflow intermediate routers’ receive buffers (leading to packet loss — and packet loss during the initial TCP connection can stall the connection for seconds), and can also contribute to bufferbloat, potentially causing latency spikes for other traffic. 20 is a ‘reasonable’ figure that noticeably improves latency without triggering any of those side effects.

Note that this does assume that the ultimate receiver can actually handle this much data: some very old mobile clients and modern IoT clients (and e.g. Windows XP) will advertise a smaller receive window (their initrwnd), which will cap the size of the initial window.

Anyway, that’s why I wanted to configure initcwnd on this webserver (also, because it’s fun). The rest is how to do so.

As is common, on the GCE version of Debian, the internal IPv4 address is set by DHCP. As is slightly less common, DHCP also provides the server with default and gateway routes (per RFC 3442).

In modern versions of systemd, systemd-networkd can be told how to set initcwnd via an InitialCongestionWindow setting in a .network file.

We can see which .network file systemd-networkd is using with networkctl status (which accepts interface names or shell-style wildcards):

$ networkctl status 'en*'
● 2: ens4
                   Link File: /usr/lib/systemd/network/99-default.link
                Network File: /run/systemd/network/10-netplan-all-en.network
                       State: routable (configured)
...

(or networkctl with no arguments will list all the network links that systemd-networkd is aware of.)

Network File: is the important part here: it shows the name of the .network file that we’d need to change.

However, here we have a dynamic file (in /run/) generated by Netplan, so we can’t modify it directly. Fortunately, systemd-networkd also supports the idea of “drop-in files” that can be used to add additional configuration to an existing network; see man systemd.network.

For example, for the network link above, we could put extra configuration into /etc/systemd/network/10-netplan-all-en.network.d/*.conf, where the name of the .d directory has to match the name of the network file.

So here we’d just need to add a [DHCPv4] section with InitialCongestionWindow=20 to achieve what we want to do.

This approach does rely on the Netplan-generated filename staying static, but that should be the case: it seems to use a hardcoded 10-netplan- prefix, plus a string from the /etc/netplan/ configuration, which should be unlikely to change. Of course, if you’re not using Netplan, it’s even easier, since you just edit the .network file directly.

So IPv4 is pretty simple, once we know what file to create (I’ve put a complete example below), but IPv6 is a little more mysterious.

For IPv6, you might expect to be able to add a similar setting under one of the IPv6-related sections, but the specific setting we’re after can only be specified under [Route] and [DHCPv4], at least as of systemd 261³.

We do have a solution, thankfully, but it requires understanding how routing is configured for dynamic IPv6:

The IPv6 address is assigned by stateful DHCPv6, where systemd-networkd runs a client to lease the public /128 address from its allocated subnet. GCE does not use SLAAC for address assignment. However, DHCPv6 only provisions addresses, not routes.

The IPv6 route is assigned by watching for periodic ICMPv6 Router Advertisement broadcasts⁴.

When an RA packet arrives, systemd-networkd processes it and normally installs the default route. RA packets only supply IPv6 routing parameters (and we can’t change what GCE sends anyway), and as noted above, systemd-networkd doesn’t have any obvious place to attach route metrics like initcwnd to the result.

Fortunately, we can switch off the automatic creation of that default route and instead configure a route of our own that will become the default route.

To cut to the chase, we can set initcwnd for both IPv4 and IPv6 by creating a single drop-in file with a name matching the network file shown by networkctl status:

# /etc/systemd/network/10-netplan-all-en.network.d/initcwnd.conf
[DHCPv4]
InitialCongestionWindow=20

[IPv6AcceptRA]
UseGateway=no

[Route]
Destination=::/0
Gateway=_ipv6ra
InitialCongestionWindow=20

Setting UseGateway=no stops systemd-networkd from creating its own default route when receiving the RA; instead, it finds the [Route] that defines Gateway=_ipv6ra, and installs that route using the advertised gateway. Since our route also defines Destination=::/0, it becomes the default route.

While my focus was on initcwnd, you can obviously also use this to set any other per-route settings, such as InitialAdvertisedReceiveWindow (initrwnd). Some other per-route settings can already be set directly in the [IPv6AcceptRA] section, which is probably a better place for them. (If a future version of systemd adds support for the missing settings to [IPv6AcceptRA], the above would become a little simpler.)

sudo networkctl reload reloads the configuration and updates the IPv4 route immediately from the cached DHCP lease. For IPv6, it’ll drop the default route (the one created by UseGateway=yes) and request a new RA by sending out a Router Solicitation packet. That should trigger a new RA, installing the new IPv6 default route within a second. (Alternatively, you can just reboot the VM.)

Once everything’s been updated, we can see the results with ip route:

$ ip -4 route show default
default via 10.154.0.1 dev ens4 proto dhcp src 10.154.15.203 metric 100 initcwnd 20

$ ip -6 route show default
default nhid 886644799 via fe80::4001:aff:fe9b:1 dev ens4 proto ra metric 100 expires 85sec initcwnd 20 pref medium

The two via addresses are the local gateways, and proto dhcp and proto ra show what protocols provided the routes: DHCP for the IPv4 route, and a Router Advertisement broadcast for the IPv6 route. Both now have initcwnd 20, which was what I wanted to achieve.

Well, it took a bit of digging through man pages to figure out what was going on here, but the end result is that I now have a much better understanding of modern networking clients, and of IPv6 — which was really the main reason I wanted to look into this in the first place. Writing it up here should also give me a place to look the next time I want to remember what’s going on.

Oh, and this server is very slightly faster now too.