upstream: one absolute per-query budget across queueing and failover
Gates / frontend (push) Successful in 1m46s
Gates / test (push) Successful in 2m32s
Gates / package (push) Successful in 4m20s
Gates / test-aarch64 (push) Successful in 8m15s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 30m33s

waiting for a slot now spends the query budget; truncated attempts that
expire fault the budget, not the upstream, and are never attributed.
admission sweeps in priority order before blocking. forward zones spend
read_timeout_ms once across udp, truncation and tcp. adds
nxdns_upstream_budget_exhausted_total and a 64-upstream validation limit.
This commit is contained in:
2026-08-27 21:10:43 +02:00
parent d2e12ae0e2
commit a3aa7febb4
16 changed files with 1598 additions and 207 deletions
+5 -3
View File
@@ -37,12 +37,14 @@ Timeouts for talking to upstream resolvers.
| Key | Type | Default | Unit | Validation | Consumed by |
|---|---|---|---|---|---|
| `upstream.attempt_timeout_ms` | u32 | 2500 | ms | 100120000, and not above `total_timeout_ms` | deadline on one attempt against one upstream inside the pool's failover loop (`src/upstream/pool.zig`), the whole attempt including the connect |
| `upstream.read_timeout_ms` | u32 | 3000 | ms | 100120000 | read deadline on conditional-forward-zone exchanges (`src/local/forward_client.zig`) |
| `upstream.read_timeout_ms` | u32 | 3000 | ms | 100120000 | budget for a whole conditional-forward-zone exchange (`src/local/forward_client.zig`): the UDP attempt, a TC=1 fallback and the TCP retry together |
| `upstream.total_timeout_ms` | u32 | 5000 | ms | 100120000 | per-query budget of the upstream pool (`src/upstream/pool.zig`): every failover attempt together, not one of them; also the `nxdns check` probe deadline |
The two pool budgets nest. `attempt_timeout_ms` bounds one try against one upstream; when it expires the pool records the failure and moves to the next candidate. `total_timeout_ms` bounds the whole loop, so a query against five unreachable upstreams costs the total budget once, not five attempt budgets in a row. When the total expires the in-flight attempt is canceled and the query fails with a timeout.
`read_timeout_ms` is unrelated to both. It bounds a different subsystem — the conditional-forward-zone client — so no cross-check relates it to the pool's budgets, and it is free to sit above either of them.
Budget semantics. The deadline is an instant, computed once when the query enters the pool, and every blocking step spends against it — waiting for a free slot on an upstream as much as the exchange itself. An attempt therefore runs against `min(now + attempt_timeout_ms, deadline)`, which near the end of the budget is *truncated*: shorter than the configured attempt. Attribution follows from that. A truncated attempt that expires is evidence about the budget, not about the upstream, so it leaves that upstream's health and success rate untouched and the query log's `upstream` field unchanged — the row may still name the endpoint of the preceding attributable attempt, and is null only when there was none — and it increments the pool-wide `nxdns_upstream_budget_exhausted_total` metric. An attempt that expires on its full budget, or that fails outright (a refused connection, a bad answer, the peer's own timeout), is evidence about the upstream: it is recorded against that upstream's health and names it in the query log. Either way the client is answered SERVFAIL. Forward-zone queries are not affected — they have one configured resolver, so a timeout there always names it.
`read_timeout_ms` is unrelated to both pool budgets. It bounds a different subsystem — the conditional-forward-zone client — so no cross-check relates it to the pool's budgets, and it is free to sit above either of them. Within that subsystem it is one budget for the whole exchange: a query that goes out over UDP, comes back truncated and is retried over TCP has the two legs and the fallback share `read_timeout_ms`, never one each.
### dns
@@ -318,7 +320,7 @@ Zones resolved by a specific resolver instead of the configured upstreams, for L
| `zone` | string | required | a valid domain name; unique |
| `resolver` | string | required | `udp://IP:port` or `tcp://IP:port`; the host must be an IP literal and the port is mandatory |
The resolver host must be an IP literal because resolving the resolver's own name would be a bootstrap problem. Matching is longest suffix (`src/local/forward_zones.zig`); the exchange is UDP then TCP (`src/local/forward_client.zig`) with `upstream.read_timeout_ms` as the read deadline.
The resolver host must be an IP literal because resolving the resolver's own name would be a bootstrap problem. Matching is longest suffix (`src/local/forward_zones.zig`); the exchange is UDP then TCP (`src/local/forward_client.zig`), with `upstream.read_timeout_ms` bounding the whole exchange rather than each leg.
Reverse zones are declared the same way, and one is the prerequisite for [learned client names](#learned-names):