farblog

Here’s the tl;dr (also, spoilers): if you’re using Prometheus’ blackbox exporter to probe a target, you should probably explicitly specify the IP protocol (IPv4 or IPv6) to use for each probe, especially if you have an IPv4-only target.

I have a very small home monitoring setup using Prometheus and various exporters to monitor this website, plus a few other sites and machines that I care about. For HTTP monitoring, I’m using Prometheus’ blackbox exporter to probe a target URL (the front page of this site) every so often.

The probe configuration is a little bit confusing, as the targets (what to monitor, i.e. the target URL) are set in the main Prometheus config, while the probe definition and expectations (how to monitor) are set in the blackbox exporter config, but the latter looks something like this:

modules:
  farblog_https:
    prober: http
    http:
      headers:
        accept-encoding: br
      compression: br
      fail_if_not_ssl: true
      follow_redirects: false
      fail_if_body_not_matches_regexp:
        - "farblog"

This probe succeeds if the site is responding reasonably, and each probe attempt also records various metrics that I can use elsewhere (probe_ssl_earliest_cert_expiry, probe_success, etc).

The automated checks I’m running are pretty basic: verifying that the site is up, that the TLS certificate isn’t about to expire, that kind of thing, but I also have some basic monitoring for probe latency, mostly to detect accidental regressions (or just let me know if something weird is going on).

In the past I’ve used this historical record to test the impact of server changes (for example, I now pre-generate compressed gzip and Brotli representations of most content, which saves a few milliseconds), but as with most things here, it’s just a fun little project to play with.

For some time though, I’d been aware that the response times I was seeing were a little spiky, which bothered me: my home internet connection should be fairly stable, and the site itself (which is running nginx on a small GCE node) should be very predictable, and yet my graphs still looked like this:

I’d assumed that this was down to something in my home networking setup that I hadn’t taken account of, but then more recently I noticed that most of the probes to other sites were much more consistent; so what was the difference with this one?

Zooming in to show an hour rather than the whole day (the highlighted part of the graph above) initially didn’t help much, but breaking down the probe duration by the different probe stages (the probe_http_duration_seconds metric) does show something interesting:

This graph is fairly unsurprising: as a static site, most of the time is determined by the number of roundtrips we need to make from the probe origin to the target¹, but the resolve stage is clearly the issue here.

Isolating just resolve makes this even clearer:

This is why my probe latencies were spiky: address resolution usually takes a little over a millisecond, but every so often, it takes about 7ms instead. What’s going on here?

1–2ms is about² the time it takes to resolve an address that’s cached by my router, while 7ms is about³ the time it takes to resolve the target address via my upstream nameservers, so something was clearly making extra queries to my upstream.

My router should definitely be caching the DNS results, and while we absolutely would expect to see a spike when that cached result expires, that should manifest as an extra delay once every 12 hours (the TTL for www.farside.org.uk) — not what we’re seeing here.

One possibility I considered was that my router was prematurely evicting results from its cache, but it’s pretty clear from the graph that “every so often” is actually “every five minutes”, and that’s a big clue: the MINIMUM time for my domain (in the SOA record) is 300 seconds, or five minutes, so this is likely a negative cache result expiring and triggering a query to an upstream nameserver (per RFC 2308). But what non-existent name could it be trying to resolve?

The answer turns out to be IPv6.

At the time I was looking into this, this site only served over IPv4 — GCE didn’t support IPv6 until 2022 or so, and I’d never gone back to turn it on — and so it only had an A record for the IPv4 address, not the AAAA record needed for IPv6.

Somewhat unexpectedly, blackbox exporter’s default behaviour is to resolve both A and AAAA records in parallel, and then only continue once both queries have an answer.⁴

Since in my case there was no AAAA record to resolve, my upstream nameserver would return a NODATA response, and my router would dutifully cache that for the amount of time specified in the SOA record.

The solution is to tell blackbox exporter to resolve only one address family by disabling ip_protocol_fallback, and then to pick a protocol (or use the default of IPv6) using preferred_ip_protocol:

modules:
  farblog_https:
    prober: http
    http:
      # (as before…)

      # Prefer IPv4, and disable protocol fallback so that we only
      # attempt to resolve an address for one address family.
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: false

This is also what you’ll want to do if you have an IPv4-only client probing a target that does have an IPv6 address, though in that case it’ll be a lot more obvious, as blackbox exporter will resolve both and then fail to connect via IPv6.

With this change, blackbox exporter only issues a single A record query, which will usually be cached locally. The effect was immediately visible:

This was also the explanation as to why only some of my probes were noticeably affected by this problem: the others were probing dual-stack targets, where the IPv4 and IPv6 addresses would both exist, and would typically have comparable TTLs. This still wasn’t ideal, since the probes would be waiting for answers to DNS queries they wouldn’t use, but in practice it didn’t have anywhere near the same effect on the probe duration.

I do think that blackbox exporter’s default behaviour is a little unfortunate here, since it abstracts away too many of the details (and until fairly recently, was mostly undocumented). However, the documentation has now been updated to describe the address selection process, and to recommend that dual-stack targets, at least, should have separate IPv4 and IPv6 probes.

Does any of this matter to my little website? Of course not, but I had fun digging into the problem, and learned a fair amount along the way. And maybe this write-up will be useful to someone else too.

Coda

In related news, this finally prompted me to put in the work to enable IPv6 on this site, which turned out be to pretty trivial to do. So now I have two separate sets of latency metrics to look at!

(The title of this post is derived from RFC 8305, the “Happy Eyeballs” algorithm that helps applications to connect to dual-stack hosts. While a low-level prober isn’t the kind of application that would actually want to use this algorithm, I thought the reference was too good to pass up!)