server: a query that finds every permit taken waits for one instead of failing, the budget is 64, permit and upstream turns wait on a futex with one thread each, the thread pool has no task limit, and transparent huge pages are off for the process

specs/memory-budget.md

@@ -22,7 +22,7 @@ Why freed small blocks stay resident: with libc linked, `malloc` is zig's `lib/c
 
 - Fix each owner at its root. Do not bypass the safe-build 0xaa fill: it only makes the oversizing visible.
 - Do not change the allocator in this work. Replacing `init.gpa` does not move SQLite, which calls C `malloc`. Revisit only if the measurement gate still names small-class slabs as a top owner.
-- No compiled-format change, no config-key migration, no new deadline mechanism. Each of those was considered and rejected in review: the format change makes the upgrade depend on the network, packed query-log rings cost more complexity than sizing the queue, and a socket-shutdown deadline cannot bound `connect` (`std.Io.Threaded` has no connect timeout) or the pool admission wait.
+- No compiled-format change, no config-key migration, no socket-shutdown deadline. Each of those was considered and rejected in review: the format change makes the upgrade depend on the network, packed query-log rings cost more complexity than sizing the queue, and a socket-shutdown deadline cannot bound `connect` (`std.Io.Threaded` has no connect timeout) or a wait for a permit. Permit waits take their deadline on a futex instead (item 3).
 - Not `SQLITE_CONFIG_PAGECACHE`, not a database page-size migration.
 
 ## Contract
@@ -56,13 +56,19 @@ Within one source, as compiled entries, an exact `x` in `.list` together with `x
 
 ### 3. Bound concurrent DNS work
 
-- `main` takes `std.process.Init.Minimal` and owns its `std.Io.Threaded`, with a `concurrent_limit` derived from the chosen query and connection caps, the tasks each can nest, and the fixed services. The measured task high-water already counts every caller, so it is the cross-check, not a term in the sum.
-- `ConcurrencyUnavailable` is counted at every boundary where it can surface. The spec work enumerates each `io.concurrent` and `Group.concurrent` call; today the logger turns the error into an early flush without counting it (`src/storage/logger.zig:748`) and the listener races fold it into a generic failure (`src/server/listener.zig:102`). Each gets its own counter or a reason on an existing one.
-- Two kinds of bound: one cap on active DNS queries, shared by every listener (UDP and TCP, both families, DoT/DoH when enabled), and a slot cap per stream listener on accepted connections, each from the item-0 high-waters. A query takes a permit only while it does work; a keep-alive connection releases its permit between requests.
-- UDP keeps today's counted drop when no permit is free (`dropped_no_slot`). The kernel buffer does not absorb overload invisibly.
-- The upstream exchange keeps its whole-operation `Select` deadline and its distinction between peer result, budget expiry and cancellation. The thread high-water falls because the query budget is smaller, not because the race changes.
-- UDP slots and TCP connection buffers are sized from these bounds, not 64 per listener.
-- As built: 32 query permits for the whole process (`query_budget.permits`; the Pi's `nxdns_dns_in_flight_peak` was 27 over 37 h), each holding the datagram, the reply buffer and the handler scratch, about 2.4 MiB in all. 16 connections per DNS stream listener (`listener.dns_max_connections`), a policy rather than a measurement: the Pi's TCP peak was 0 and it runs neither DoT nor DoH. A stream query that finds no permit closes its connection and is counted as `dropped_no_permit`, on DoH too, where a 503 would need a reply buffer the query does not have. `async_limit` is `.nothing`, so every pool task is a counted `concurrent` one, and `app.concurrent_limit` enumerates them. `ConcurrencyUnavailable` is `nxdns_concurrency_unavailable_total{site}`, one label per boundary (`platform/concurrency.zig`).
+- Two kinds of bound: one budget of query permits, shared by every listener (UDP and TCP, both families, DoT/DoH when enabled), and a slot cap per stream listener on accepted connections. A query takes a permit only while it does work; a keep-alive connection releases its permit between requests.
+- The work is bounded where it enters, not in the thread pool. `main` keeps `std.process.Init` and its `std.Io.Threaded` sets no `concurrent_limit`: a limit below the tasks the admitted work can nest turns a burst into `ConcurrencyUnavailable` failures inside admitted queries, and a limit sized to cover them bounds nothing the budget does not already bound. `ConcurrencyUnavailable` therefore means the system refused a thread or a task could not be allocated, and it is counted at every boundary where it can surface as `nxdns_concurrency_unavailable_total{site}` (`platform/concurrency.zig`).
+- A query that finds every permit taken waits for one; it is neither dropped nor answered SERVFAIL. glibc's stub retries a SERVFAIL once at once and then fails the lookup, and a dropped UDP query costs the client its whole retry timeout, while a wait behind a permit that frees within the upstream deadline is invisible to it. Every held permit is bounded (the upstream total deadline, the stream write budget), so a wait always ends.
+  - UDP: the receive loop receives into its own staging buffer, then waits for a permit holding that datagram, and receives nothing meanwhile. The backlog stays in the socket's kernel buffer, bounded by `SO_RCVBUF`; what overflows it is dropped by the kernel and exported from `/proc/net/udp{,6}` by socket inode as `nxdns_udp_server_kernel_drops_total`, with the queued bytes as `nxdns_udp_server_receive_queue_bytes` and a failed read as `nxdns_udp_server_kernel_read_failures_total`.
+  - Streams: a whole query read off the connection waits at most the connection's idle budget. A wait that outlasts it ends the connection unanswered, because DNS-over-TCP has no framing for "no answer"; DoH closes too.
+  - Every wait is counted: `nxdns_dns_query_permit_waits_total`, `nxdns_dns_query_permit_wait_seconds_total`, `nxdns_dns_query_permit_wait_timeouts_total` and the gauge `nxdns_dns_query_permits_waiting`.
+  - A listener that is closing wakes the budget's waiters, so its `deinit` does not wait for a permit to free.
+- A query waits on its own thread. The budget and the upstream pool's per-entry admission sleep on a futex word with the caller's deadline (`platform/wake_word.zig`) instead of racing a condition wait against a timer task, which cost two extra threads per waiter. A waiter that leaves without a permit hands its wake on if a permit is free.
+- The upstream exchange keeps its whole-operation deadline race and its distinction between peer result, budget expiry and cancellation.
+- Stream listeners keep `listener.race` for their idle reads, writes and handshakes: three threads per open connection, at most 16 per listener. Removing those two threads needs a non-blocking, socket-aware TLS and HTTP transport with its own deadline, reset and partial-I/O handling; std's timed file reads do not provide one (a blocking syscall follows the poll, and `ECONNRESET` on a timed write is an `errnoBug`). That transport costs more than the threads it saves at household scale.
+- The process opts out of transparent huge pages at startup (`PR_SET_THP_DISABLE`, `platform/huge_pages.zig`), with one warning if the kernel refuses. With THP set to `always`, a worker stack that touches a few KiB holds a 2 MiB page and keeps it, because workers are never retired: a 64-query burst on such a host left about 40 of 200 workers at 2 MiB each.
+- What this costs, stated plainly: a burst beyond the budget is answered later, not faster; a UDP burst that also outruns the socket buffer loses datagrams to the kernel, counted; a stream query can wait out its idle budget and get no answer; an upstream in backoff is still probed, so a dead upstream costs each query its deadline and ends in SERVFAIL from the handler, as before.
+- As built: 64 query permits for the whole process (`query_budget.permits`), each holding the datagram, the reply buffer and the handler scratch, about 4.6 MiB in all. The Pi's `nxdns_dns_in_flight_peak` was 27 over a day of household traffic; 64 simultaneous uncached queries on two DoT upstreams (x86_64 musl ReleaseSafe) were all answered, the slowest in 0.2 s, and grew the pool from 14 to 205 threads, which is the all-active envelope of 64 × 3 plus the fixed services, about 40 MiB of stack at the measured 200 KiB high-water per worker. 16 connections per DNS stream listener (`listener.dns_max_connections`), a policy rather than a measurement: the Pi's TCP peak was 0 and it runs neither DoT nor DoH.
 
 ### 4. Size the query-log buffers by use
 
@@ -90,7 +96,7 @@ Within one source, as compiled entries, an exact `x` in `.list` together with `x
 - [ ] Builder: a reload holds at most one source's bodies (tested through an allocator that records peak bytes).
 - [ ] Boot: one snapshot build when no source refreshes; listeners never start without the initial load having run.
 - [ ] Measurements: each of the five item-0 gauges rises under a scripted load and is visible in `/metrics`; the thread count and the worker stack residency are recorded from `/proc` beside them.
-- [ ] Bounds: with the query budget full, a further UDP query is dropped and counted; a stream listener at its slot cap refuses and counts; a keep-alive connection between requests holds no query permit; `ConcurrencyUnavailable` is counted at each enumerated boundary, including the logger wait and the listener races.
+- [ ] Bounds: with the query budget full, a further UDP query waits and is answered once a permit frees, and the wait is counted; a stream query waits at most its idle budget, and a timeout ends the connection and is counted; closing a listener ends its waits without a permit freeing; a waiter canceled or timed out hands its wake on; a stream listener at its slot cap refuses and counts; a keep-alive connection between requests holds no query permit; the UDP sockets' kernel queue and drops are exported; `ConcurrencyUnavailable` is counted at each enumerated boundary, including the logger wait and the listener races.
 - [ ] SSE: no ring is allocated without a subscriber; an allocation failure is counted, reported as its own error and leaves the slot reusable; overflow still ends the stream.
 - [ ] SQLite: each runtime handle reports its configured `cache_size`; migrations open without it.
 - [ ] DNS cache: puts beyond the byte budget evict; live bytes never exceed it; an oversized response is skipped and counted; a failed replacement keeps the old entry; a failed copy allocation is counted and the answer is still sent.