nxdns/specs/milestone-16.md

Milestone 16: contained behavioral fixes

Goal: fix the verified bugs and silent failures from TECH_DEBT.md themes 3, 5, 6 and 7 — each one localized, each with a test that fails before and passes after. No refactors: the duplication work is milestone 18, and a refactor that must simultaneously fix behavior is how mirrored copies diverge further.

PROVISIONAL. Written before milestone 15 was built. m15 touches build.zig, workflows, fuzz files, filter_integration_test.zig, cli.zig, docs_drift_test.zig and logging.zig — re-verify line references in those files before starting.

Rulings (binding)

1. loadSource propagates cancellation

src/filter/manager.zig:541-544 and :547-550: two identical catch sites fold error.Canceled into loadFailure(...), recording a bogus "Canceled" load-failed status, consuming the one-shot cancellation, and publishing a snapshot with the source excluded. Ten other catch sites in the file propagate it correctly. Add if (err == error.Canceled) return error.Canceled; beside the existing OutOfMemory arm at both sites — Manager.Error already has the member. Unit test: a reader that fails with Canceled during loadSource makes the whole call return Canceled and writes no status.

2. commitStatus never drops an outcome silently

src/filter/manager.zig:1237-1250: the id-match loop falls through with no log when a source has no status entry (reachable via a startupPass / web-insert race; milestone-5 policy says every non-ok state is recorded and surfaced). Add a log.warn after the loop naming the source id. Unit test: committing a status for an unknown id emits the warning and does not crash.

3. Downloads and compiles leave the writer lock

src/filter/manager.zig:609-624: refreshAll holds writer_lock across every download (300 s budget each, app.zig:85) and every compile; every web mutation ends in Manager.reload on the same lock, so an unrelated rule save parks uncancelably — on a request path with no timeout, occupying one of 64 web slots — until the pass ends. The download-to-tmp flow already exists (download writes <id>.raw.tmp via compiledName, with deleteQuietly defers at :646-648); the lock is just taken too early.

Restructure:

• New refresh_lock: std.Io.Mutex on Manager, serializing refresh passes against each other only.
• Split refreshOne (:626-729): the fetch/detect/compile stages (:650, :659, :668) run under refresh_lock alone; the publish stage (:711), the status commit and the final reloadLocked run under writer_lock.
• refreshAll takes refresh_lock for the pass, and writer_lock only per-source around publish plus once for the final reload. refreshSource (:582-586) follows the same split.
• The lock-ordering rule, documented on both mutex fields: refresh_lock is never acquired while holding writer_lock.
• refresh_lock serializes blocklist-directory maintenance too, not just refresh passes: pruneOrphans (:1040, invariant comment at :1130-1136) today assumes every temp-file writer holds writer_lock; once downloads write .raw.tmp/.list.tmp/.wild.tmp under refresh_lock alone, a concurrent source deletion (blocklists.zig delete → reload → pruneFiles) could sweep an in-flight refresh's temp files. pruneOrphans therefore takes refresh_lock first (then writer_lock if it needs it — same order as everywhere), and its invariant comment is rewritten. Regression test: a source deletion during a stalled refresh must not delete the stalled refresh's temp files.

Integration test (filter_integration_test.zig): with a refresh pass parked on a deliberately stalled fixture route, a concurrent rule mutation completes without waiting for the pass. DNS serving was never affected and stays covered by the existing tests.

4. Truncated responses are never cached

src/cache/dns_cache.zig:106-131: classify reads rcode and ancount and never flags.tc, so a TC=1 answer from a misbehaving upstream is cached for up to max_ttl_seconds (86 400 s) and re-served — TC bit intact, even over TCP, which can loop retrying clients (RFC 2181 §9). Add if (p.header.flags.tc) return null; (flags.tc is src/dns/header.zig:20).

transport.validateResponse is deliberately untouched: rejecting TC there would change failover semantics for every exchange, and the forward client legitimately reads TC for its UDP-to-TCP retry (forward_client.zig:178). Record: considered, rejected. Unit test beside the existing classify tests: a TC=1 NOERROR response classifies as null.

5. The format sniffer stops eating hosts files with ## banners

src/filter/parsers.zig:54-73: hasAbpMarker runs before isComment, and isElementHiding (:86-93) matches ## (and #@#, #?#, #$#, #%#) unanchored with no comment guard — unlike the $ branch two lines later, which is guarded. A hosts file with a ## banner parses whole as ABP: 0.0.0.0 enters the domain set and hosts lines with inline URL comments are silently dropped (real blocking loss, reproduced with the URLhaus banner style). The milestone-5 as-built note (specs/milestone-5.md:1677-1678) already claims the separators are matched anchored — the code never was. Guarding the branch with !isComment is a circular no-op (isComment consults isElementHiding); do not attempt it.

Fix — a positional predicate in isElementHiding: a separator at offset p counts as element hiding only when

• p == 0 and the character after the separator is not whitespace, #, or end of line (a generic rule ##.ad counts; a banner ## Title, ####, or bare ## does not), or
• p > 0 and line[p - 1] is not whitespace and not # (a rule example.com##.ad counts; prose see ## below does not).

Required test cases, in parsers.zig: the URLhaus banner style sniffs as hosts; ##.ad-banner sniffs as ABP; example.com##.ad sniffs as ABP; #@#exception after a domain sniffs as ABP; a #-initial line containing ## sniffs as a comment. The fuzz target (blocklist_fuzz.zig) already covers detectFormat for crashes.

6. VACUUM respects the disk monitor

src/storage/retention.zig: runOnce (:78) does prune → checkpoint → (every 7th pass, :94-95) VACUUM, and takes no monitor — the word does not appear in the file — while its two sibling tasks gate on disk_monitor.writesAllowed() (logger.zig:386 as a parameter, manager.zig:1058 as a field). VACUUM is the most expensive write the program makes and fires unconditionally on the exact filesystem the monitor watches, then fails SQLITE_FULL with no retry for seven daily passes.

runOnce and run gain monitor: ?*disk_monitor.Monitor (the logger's parameter shape). Only the VACUUM step gates: prune and checkpoint keep running — they free space. A skipped vacuum increments a new Stats.vacuums_gated and retries on the next pass (drop the modulo-only trigger for a passes_since_vacuum >= vacuum_every_passes counter that resets on success). app.zig:561 passes the monitor like :560 does for the logger. Tests: extend the cadence tests (:223) with a gated pass — the vacuum is skipped, counted, and runs on the next allowed pass.

7. An oversized Cookie header degrades loudly, and the session survives

src/web/server.zig:612-620 (copyHeader): a header value over the buffer (cookie buffer = http_util.max_cookie_len = 1024, http_util.zig:35) returns "" with no log — under the documented reverse-proxy-on-shared- domain deployment (foreign cookies riding along), every request silently 401s and the operator sees an unexplained login loop.

copyHeader stays generic. The cookie call site (:510) changes: when the raw header value exceeds the buffer, extract only the session pair by running http_util.cookieValue (:200 — it already parses pairs and returns a slice of the original header) with the existing session-cookie name constant against the full value, copy that pair into the buffer as <name>=<value>, and emit one log.debug with the dropped size (the .web_server scope at :55; no secret is logged). If even the pair does not fit, keep the empty result but still log. Integration test (web_integration_test.zig): a request with a 2 KiB cookie header whose session pair is valid is authenticated; the same header without the pair 401s.

8. Live view detects fatal SSE rejections

web/src/features/live/useLiveQueries.ts: per the WHATWG spec a non-200 response fails an EventSource permanently after one error event, so the 3-consecutive-errors threshold (:29, :125-133) is unreachable on exactly the 429/401 paths it was built for — the UI shows "Reconnecting…" forever and the capped state and session probe are dead. FakeEventSource has no readyState, so tests pass — the mocked-network class again.

• EventSourceLike (:9-13) gains readyState: number; export const EVENT_SOURCE_CLOSED = 2. The browser EventSource satisfies it structurally.
• In the error handler: readyState === EVENT_SOURCE_CLOSED is a permanent failure — close, set capped, run the session probe immediately, bypassing the counter. Transient errors (browser auto-retry pending) keep the existing counter path.
• fakeEventSource.ts gains readyState with the real lifecycle (CONNECTING → OPEN on emit("open"), CLOSED on close() and on failFatal(), a new test helper).
• New tests: a fatal rejection (single error event, readyState CLOSED) reaches capped and calls probeSession; a transient error still takes three to trip.

9. A timed-out DoH handshake is a handshake failure

src/server/doh_server.zig:311-315 counts a handshake-race timeout as idle_timeouts; DoT (:313-317) counts it as tls_handshake_failures, which is the recorded spec ruling. Since DoH requests have no other timer, its idle_timeouts metric can only ever mean handshake stalls — the same exported name with disjoint semantics per listener. Align DoH's .timed_out arm to tls_handshake_failures.

10. Idle DoH keep-alive connections are reclaimed

src/server/doh_server.zig:70: idle_timeout bounds only the handshake; its own doc comment admits requests have none. DoH stubs hold keep-alives by design, and with no TCP keepalive a vanished peer pins one of 64 slots until restart — while DoT on the same LAN reclaims after 10 s.

Extend the existing race to receiveHead (:333), the wait for the next request on a keep-alive connection, using the DoT out-param precedent (readPrefix's out_len): a small wrapper writes the received Request (or its error) through a pointer so the raced function stays anyerror!void. On .timed_out: bump idle_timeouts (now truthfully named again after ruling 9) and close the connection. Preserve the existing error triage at :334-346 (HttpConnectionClosing and ReadFailed return uncounted; the three structural errors count bad_requests/connection_errors as today). The body read and handleRequest stay untimed, like the web listener.

New integration-gated test mirroring the DoT idle test (dot_server.zig:1098): a client that completes the handshake and one request then goes quiet is closed after a short idle budget; idle_timeouts == 1, tls_handshake_failures == 0. DoH currently has no idle test at all.

11. SSE shutdown does not wait for a heartbeat

src/web/sse.zig: the Hub has no shutdown signal, so graceful drain parks up to 15 s (heartbeat_interval, web/handlers/live.zig:32) per idle subscriber — the web beginShutdown (web/server.zig:593-605) shuts down sockets but nothing wakes a task inside Hub.wait. Measured: the SSE integration test budgets 40 s for exactly this (web_integration_test.zig:74-75).

• Wake (:33) gains .closed. Hub gains closing: bool and pub fn close(self: *Hub, io: std.Io) void: under the mutex, set the flag and event.set every active slot. wait returns .closed immediately when the flag is set.
• live.zig:114: .closed => return.
• web/server.zig deinit (:349): call the hub's close (via the state's hub pointer) immediately before beginShutdown (:360).
• Test: a subscriber parked in wait returns .closed promptly after close; the W10 integration teardown no longer stalls (tighten sse_budget only if the heartbeat assertion itself allows it).

12. The API limiter's sweep runs

src/web/api_limiter.zig:210: sweep exists, preallocates stale_keys so it never allocates, is tested — and has no production caller; only the DNS limiter got scheduled. Once 4096 distinct addresses have been seen, the table stays full forever and every unknown-address request pays an O(4096) eviction scan under the limiter mutex.

runMaintenance (app.zig:709) gains an api_limiter: ?*ApiLimiter parameter, wired at :565 from the instance built at :380-388. In the loop, beside the two existing tasks: _ = api.sweep(io, Clock.awake.now(io)); — note it locks itself, unlike the DNS limiter. Test: the maintenance-loop integration coverage asserts a stale bucket disappears.

13. UDP/53 and TCP/53 stats reach /metrics

src/server/udp_server.zig:38-49 and tcp_server.zig:54-60: both stats structs are written and read by nothing outside their own integration tests — dropped_no_slot/dropped_oversize have no metric, no API field, and errors log below the default level, while the DoT/DoH siblings export equivalent counters. The module doc sells "dropped and counted"; the count is unobservable.

• Both gain pub const Snapshot + pub fn snapshotStats in the exact DoT shape (dot_server.zig:61, :232). Field names stay as-is (the accepted/connections unification is milestone-18 work).
• WebState (src/web/server.zig) gains udp_listeners: []const *udp_server.UdpServer = &.{} and tcp_listeners: []const *tcp_server.TcpServer = &.{} — S4 adds the fields with exactly these names and types; S2 consumes them (the app builds four listeners: udp6/udp4/tcp6/tcp4, app.zig:506-528).
• metrics.zig: sum each family across its listeners into one nxdns_udp_server_*_total / nxdns_tcp_server_*_total group via the existing counterGroup derivation; extend the name-assertion test (:722-743).
• app.zig wires the slices.

14. Forward-client counters survive the query

src/local/forward_client.zig:44-57: Stats (queries, udp_truncated, foreign_datagrams, failures) is instrumented, documented as preventing "an unrecorded failure mode" — and the only production caller builds a stack-local client per query (handler.zig:394) and drops it, so the spoofing signal does not exist.

Handler.Stats (handler.zig:130-154) gains forward_udp_truncated, forward_foreign_datagrams, forward_failures (atomic, like the other 17; queries are already counted by forward_zone_answers). After the exchange in viaForwardZone, fetchAdd the client's counters into them. The metrics reflection (dns_stat_fields, metrics.zig:48) picks the new fields up without an edit — assert the three new nxdns_dns_*_total names in the metrics test. ForwardClient.Stats itself stays plain and per-instance.

15. The TLS server keeps the concrete transport cause

src/platform/tls_server.zig:418-438: both BIO callbacks discard the stashed cause — net_reader.err / net_writer.err (which hold Reset/Timeout/Canceled, std Io/net.zig:1260/:1324) are never read anywhere in the file — so a routine idle-budget cancel surfaces as a warn "mbedtls_ssl_read failed" and peer resets are indistinguishable from timeouts. The client side solved exactly this (dot_client.zig concreteRead/concreteWrite, :303-318).

• ServerStream gains recv_cause: ?anyerror = null, send_cause: ?anyerror = null; bioRecv/bioSend stash self.net_reader.err/self.net_writer.err on failure before returning the mbedtls code.
• ReadError (:216-223) widens with Canceled; the read path (readIntoBuffer, :334) maps a stashed error.Canceled to it instead of TlsFailed. Callers' race harnesses already treat cancellation separately.
• Unit tests in the file's existing stub style: a reader whose err is Canceled yields error.Canceled; a Reset yields TlsFailed with the cause stashed for logging (ruling 16).

16. Peer misbehavior logs at debug, like the read path already rules

src/platform/tls_server.zig: the read path deliberately logs a peer TCP drop at debug (:362-364, "a louder level would be a log-spam vector"), then close warns on close_notify against the same dead socket (:311) and handshake warns per probe (:287). .tls_server is not in the dedup scope set (logging.zig:103-108), so peer-driven warns can evict genuine warnings from the rotating log.

• report (:474-479) gains a level parameter. The close_notify site passes debug when peer_closed is set or the stashed cause (ruling 15) is a peer-class error; the handshake site likewise. Local/config failures keep warn.
• Add .tls_server to isDedupScope (logging.zig:103-108) and to its membership test (:637-644).

17. The query log heals its own gaps

web/src/features/queries/QueryLogPage.tsx: the hand-rolled accumulation (extra, cursorOverride, generation, :79-98) develops a silent mid-table row gap when the base page refetches after 30 s staleness — the newest-100 boundary moves up while extra starts below the old cursor, and the stale override means load-more never heals it. On a live DNS server the trigger is routine.

Migrate to useInfiniteQuery: the endpoint already maps onto it (getNextPageParam: (last) => last.next_before ?? undefined; keyset pagination confirmed server-side, handlers/queries.zig:103-114). A background refetch then refetches all pages in order — consistent, no gap. The three behaviors the old code carried by hand: staleness discard is handled by the query itself; keep the isPlaceholderData disable on the button; the 401 branch is already covered by the global cache-level handleUnauthorized (queryClient.ts:29-30). Delete extra, cursorOverride, generation, loadingMore, moreError. Rewrite the five load-more tests; add one for the gap scenario: base refetch with new rows between page renders leaves no discontinuity.

18. Password hashing leaves the config lock

src/web/handlers/settings.zig: applyPut holds config_lock from :286 to :341, and the argon2id call (t=2, m=19 MiB, :373-391) sits inside it at :312 — every settings GET (:407-409 takes the same lock) and every mutation stalls for the hash duration on the Pi 5. The login path already does it right: copy under lock, hash unlocked, re-check generation under lock (auth.zig:47-71), and the LiveHash generation check (auth.zig:73-79) already closes the concurrent-install race.

Move the password-length check and the hashPassword call before the lock acquisition; everything from loadConfig on stays inside. The hash input is the parsed patch only, so nothing under the lock is needed. Tests: the existing applyPut suite passes unchanged; add one that actually distinguishes the new behavior — "both complete" already held before the fix, the GET merely waited out the hash. A stall seam under builtin.is_test (the milestone-15 rotation-seam shape) parks hashPassword on an event; the test starts a password PUT, waits until the hash is parked, completes a settings GET while the hash is still parked, then releases the seam and joins the PUT.

Sessions

S1-S5 run in parallel. One cross-session interface, fixed here: S4 adds the WebState.udp_listeners/tcp_listeners fields exactly as ruling 13 names them; S2 wires and consumes them without touching web/server.zig.

Session S1: filter

Owns src/filter/manager.zig, src/filter/parsers.zig, src/filter/filter_integration_test.zig. Rulings 1, 2, 3, 5.

Session S2: cache, retention, counters, wiring

Owns src/cache/dns_cache.zig, src/storage/retention.zig, src/app.zig, src/server/udp_server.zig, src/server/tcp_server.zig, src/local/forward_client.zig, src/server/handler.zig, src/web/metrics.zig, and the storage/server integration tests those touch. Rulings 4, 6, 12, 13 (except the WebState fields), 14.

Session S3: TLS listeners

Owns src/platform/tls_server.zig, src/platform/logging.zig, src/server/doh_server.zig. Rulings 9, 10, 15, 16.

Session S4: web server

Owns src/web/server.zig, src/web/sse.zig, src/web/handlers/live.zig, src/web/handlers/settings.zig, src/web/web_integration_test.zig, plus the ruling-13 field additions. Rulings 7, 11, 18.

Session S5: frontend

Owns web/src/features/live/*, web/src/features/queries/*. Rulings 8, 17.

Orchestrator

Strikes the closed findings in TECH_DEBT.md; updates the stale milestone-5 as-built line if ruling 5 changed its truth value.

Acceptance (milestone complete)

• Both loadSource catch sites propagate Canceled; the new unit test passes; no status row ever reads "Canceled".
• commitStatus warns on an unknown id (test).
• With a stalled download in flight, a concurrent rule mutation completes (integration test); the lock-ordering comment exists on both mutexes; a source deletion during a stalled refresh leaves the refresh's temp files alone (regression test).
• A TC=1 response classifies as null (test); validateResponse is untouched.
• The five sniffer cases pass; the URLhaus banner file parses as hosts.
• A gated pass skips only the vacuum, counts vacuums_gated, and vacuums on the next allowed pass (test).
• The 2 KiB-cookie tests pass: session extracted, non-session 401s, debug line emitted.
• A fatal SSE rejection reaches capped and probes the session (frontend test); a transient error still takes three.
• DoH .timed_out handshake counts tls_handshake_failures; the metrics name test still passes.
• The new DoH idle test passes: quiet keep-alive closed, idle_timeouts == 1.
• Hub.close wakes a parked subscriber promptly (test); web deinit calls it before beginShutdown.
• The API limiter sweep runs in maintenance (test).
• /metrics carries nxdns_udp_server_* and nxdns_tcp_server_* families summed over four listeners (name test extended).
• /metrics carries the three new nxdns_dns_forward_* counters.
• The TLS-server cause tests pass: Canceled surfaces as Canceled, never as a warn line; close_notify after a peer drop logs debug; .tls_server is in the dedup set.
• The query log uses useInfiniteQuery; the gap-scenario test passes; the accumulation state is gone.
• Hashing runs before config_lock; the settings suite passes.
• Full suite green: zig build test -Dintegration, npm run test, npm run typecheck.

Anti-requirements

• No listener-core extraction, no shared repo helpers, no shared transport helpers — milestone 18.
• No stats field renames (accepted vs connections) — milestone 18.
• No config key renames and no trusted-proxy setting — milestone 17.
• No useInfiniteQuery migration anywhere but the query log.
• No new metrics beyond the families this spec names.
• No timeout on DoH request bodies or handlers — only receiveHead races.