milestone 31: concurrent upstream exchanges, dot session reuse, queue metrics

docs/how-to/troubleshoot.md

@@ -257,6 +257,22 @@ cat /etc/resolv.conf
 
 **Fix.** If it points at the nxdns container, repoint it at a real resolver. The container resolves its upstream DoH and DoT hostnames through the host's DNS configuration, so pointing that at nxdns makes nxdns depend on itself to start. LAN clients point at nxdns; the container's own host does not.
 
+## SERVFAIL in bursts, only under concurrent load
+
+**Symptom.** Single queries answer normally, but a burst of unique names — a page load, a device waking up, a `dig` loop — returns SERVFAIL for some of them, and the upstream itself is healthy on `/api/health`.
+
+**Diagnosis.** Scrape the queue counters:
+
+```sh
+curl -s http://127.0.0.1:8080/metrics | grep nxdns_upstream_
+```
+
+`nxdns_upstream_in_flight` against `nxdns_upstream_slots` is how much of an upstream's concurrency is in use right now, and `nxdns_upstream_queued_total` counts exchanges that had to wait for a slot (with `nxdns_upstream_queued_seconds_total` for how long they waited in total). Both queue counters are approximate — they are sampled when a query is admitted, not measured as a queue length. A `queued_total` climbing with each burst means queries are waiting on the upstream rather than failing at it, and a query that waits past `upstream.total_timeout_ms` is canceled in the queue and answered SERVFAIL.
+
+`nxdns_upstream_reuse_recoveries_total` is the other half of the picture: it counts DoT connections that went stale between exchanges and were redialed. A few are normal — a resolver is free to close an idle connection. One per query means the connection is never being reused, and every query is paying a full TLS handshake.
+
+**Fix.** Raise `upstream.total_timeout_ms` if the queue drains but drains too slowly for the budget. Otherwise the queue is telling you the upstream is slow: an upstream answering in a few milliseconds does not fill eight concurrent slots at household query rates, so sustained queueing points at the resolver you configured, and a faster one is the fix. Adding a second upstream is not: failover is strict priority, not load spreading — a query waits for a slot on the first available upstream and only reaches the next one when that upstream fails or is in backoff, so a second entry adds no concurrent capacity to the first. The per-upstream slot count is compiled, not configured, so there is no knob to widen one upstream either.
+
 ## The disk is filling up
 
 **Symptom.** Writes stop but DNS keeps answering. The journal shows the transition: