metrics: peak gauges for in-flight queries, stream connections and the query log queue

specs/memory-budget.md

@@ -31,13 +31,15 @@ Why freed small blocks stay resident: with libc linked, `malloc` is zig's `lib/c
 
 The admission budget, the listener caps, the stack size and the queue size are chosen from the Pi's real peaks, so those peaks become measurements before anything is resized:
 
-- high-water of in-flight DNS queries;
-- high-water of active stream connections, per listener (today `connections` counts accepts over time, not concurrency);
-- high-water of concurrent `std.Io.Threaded` tasks;
-- high-water of the query-log writer queue's occupancy (not arrivals per interval: a full batch flushes early, and a disk gate or a slow write lets backlog build past one interval);
-- worker call-stack use, measured on representative workers. The signal stack (`std_options.signal_stack_size`) is a separate allocation for signal handling and stack traces and keeps its stdlib default unless its own use is measured.
+- high-water of in-flight DNS queries: `nxdns_dns_in_flight_peak`, counted around `Handler.handle`, every listener together;
+- high-water of active stream connections, per listener (today `connections` counts accepts over time, not concurrency): `nxdns_tcp_server_connections_peak` (the larger of the two address families), `nxdns_dot_server_connections_peak`, `nxdns_doh_server_connections_peak`;
+- high-water of the `std.Io.Threaded` worker pool: the process thread count. `Threaded` spawns a worker only when every worker is busy and never retires one (`Threaded.zig:2110`), so the count is that high-water already and needs no code. It counts workers, not tasks: `async` work can run inline or wait in the run queue;
+- high-water of the query-log writer queue's occupancy (not arrivals per interval: a full batch flushes early, and a disk gate or a slow write lets backlog build past one interval): `nxdns_querylog_queue_peak`. `std.Io.Queue` exposes no length, so a count beside it is raised before each insert and lowered after each removal; it can overstate the peak by the entries in transit, never understate it;
+- worker call-stack use, read from `/proc/<pid>/smaps`: each worker stack is an anonymous mapping directly above its own guard page, all of one size, and its `Rss` is what the thread has touched and still holds resident. That finds the expensive paths; it is not a depth bound. Before item 3 lowers `stack_size`, each expensive path gets a measured stack high-water, and the size keeps headroom above the deepest one. The signal stack (`std_options.signal_stack_size`) is a separate allocation for signal handling and stack traces and keeps its stdlib default unless its own use is measured.
 
-The counters appear in diagnostics beside the existing per-reason drop counters. One release carrying only these measurements runs on the Pi before the budget values are fixed.
+The five gauges are in `/metrics` beside the existing per-reason drop counters. One release carrying only these measurements runs on the Pi before the budget values are fixed.
+
+First reading, static x86_64 musl build, one burst of 300 UDP queries: the in-flight peak reached 64 (the UDP slot pool), threads went from 14 to 205, and of 201 worker stacks 175 touched 32 kB or less while 26 touched 2 MiB each — the same 2 MiB with only forward-zone traffic and no DoT exchange, so the TLS client is not the owner. Item 3 identifies that owner before it lowers `stack_size`: a stack smaller than that path needs is a crash, not a saving.
 
 ### 1. Store each blocklist name once in memory
 
@@ -86,7 +88,7 @@ Within one source, as compiled entries, an exact `x` in `.list` together with `x
 - [ ] Matcher equivalence: a deterministic corpus (hosts, domains, ABP with `||x^`, `*.x` alone, `x` alone, `x` with `*.x`, `@@`, an exact entry for a subdomain of a wildcard, nested wildcards `*.x` and `*.y.x`, overlapping sources) gives the same `Decision` for every probe name before and after item 1.
 - [ ] Builder: a reload holds at most one source's bodies (tested through an allocator that records peak bytes).
 - [ ] Boot: one snapshot build when no source refreshes; listeners never start without the initial load having run.
-- [ ] Measurements: each item-0 high-water rises under a scripted load and is visible in diagnostics.
+- [ ] Measurements: each of the five item-0 gauges rises under a scripted load and is visible in `/metrics`; the thread count and the worker stack residency are recorded from `/proc` beside them.
 - [ ] Bounds: with the query budget full, a further UDP query is dropped and counted; a stream listener at its slot cap refuses and counts; a keep-alive connection between requests holds no query permit; `ConcurrencyUnavailable` is counted at each enumerated boundary, including the logger wait and the listener races.
 - [ ] SSE: no ring is allocated without a subscriber; an allocation failure is counted, reported as its own error and leaves the slot reusable; overflow still ends the stream.
 - [ ] SQLite: each runtime handle reports its configured `cache_size`; migrations open without it.