nxdns/specs/milestone-10.md

Milestone 10: DoH/DoT server + cert watcher + certs/reload (PLAN Phase 9)

Goal: LAN clients resolve via DoH (RFC 8484 over HTTP/1.1 + TLS) and DoT (RFC 7858) against local certs; certs hot-reload via a polling watcher and POST /api/certs/reload (endpoint + openapi entry + contract test, deferred whole from milestone-8 ruling 2).

Ground truth: PLAN.md:23,155,194,232,500-503,525,547,615-616,647,664 (scope, layout, config keys, endpoint, exit); PLAN.md:684 — HTTP/2 and DoQ are OUT; PLAN.md:297 — "a failed reload publishes neither". Explore facts (verified this session):

• platform/tls_server.zig: ServerContext.init(gpa, cert_pem, key_pem) shared across connections; binding is per-accept (mbedtls_ssl_setup at accept), so reload = publish a new context pointer; the OLD context must outlive every stream set up against it. ServerStream.reader()/writer() are real *Io.Reader/*Io.Writer; close(gpa) sends close_notify but does NOT close the TCP stream. MBEDTLS_THREADING_C + PTHREAD are ON (build.zig:255) — one context is safe under concurrent handshakes.
• std.http.Server.init(in: *Reader, out: *Writer) takes ARBITRARY reader/writer (http/Server.zig:25) — web/server.zig's serveConn transfers with three lines changed.
• No usable file watching in 0.16 (inotify/fanotify are raw syscalls outside Io) — the watcher POLLS Io.File.stat mtime+size.
• Handler.handle(self, io, which: Transport, from, query, response_buf, scratch) (handler.zig:164); Transport {udp,tcp} drives truncation only; response_buf 512..65535; Scratch ~7 KiB per connection. tcp_server serveConn (tcp_server.zig:265-338) is the DoT loop verbatim modulo the stream; dns/transport.zig has prefix_len/parsePrefix/ framePrefix/max_message_len.
• doh_client.zig: media_type "application/dns-message" (:16), contentTypeOk exported (:171). base64 url_safe_no_pad = the RFC 8484 GET codec. http_util.queryPairs for ?dns=.
• Fixtures: tests/fixtures self_signed cert/key via the test_fixtures anonymous import.
• Config: model.TlsEndpoint already exists (doh_server port 443, dot_server 853), validated, settings-persisted, checked by nxdns check. app.zig ignores both today.

Rulings (binding)

1. Two listeners: src/server/doh_server.zig, src/server/dot_server.zig (PLAN:194), constructed only when their endpoint is enabled. Bind failure = warn + continue (web precedent). Both join app.zig's group; cancel semantics mirror tcp_server's Stop enum (.canceled cancels the connection group; listener close drains).
2. DoH protocol: HTTP/1.1 only (PLAN:684 excludes h2). Path /dns-query exactly. POST with content-type: application/dns-message (checked via doh_client.contentTypeOk) and GET ?dns=<base64url unpadded>. Responses: 200 application/dns-message with the reply bytes; handler .drop → 502 with empty body reason? NO — .drop means "no answer on purpose" (malformed/refused-silent): close the connection for POST/GET alike via 400. Errors: 404 non-/dns-query path, 405 other methods (Allow: GET, POST), 415 wrong content-type, 400 missing/invalid dns param or empty body, 413 body > dns/transport.max_message_len. No cache-control headers (LAN, no intermediaries). Keep-alive per std.http.Server loop; the bodyless-POST normalization from web/server.zig:443 is copied (same stdlib assert).
3. DoT protocol: RFC 7858 = the tcp_server loop over ServerStream: 2-byte prefix, parsePrefix, reject 0, read body, handle, frame + write + flush. idle_timeout race identical (same Options shape, max_connections 64, per-conn Scratch). The TLS handshake itself runs under the same race budget (a stalled handshake must not pin a connection slot).
4. Transport enum stays {udp, tcp}: DoH/DoT call handle with .tcp (it only drives truncation, which never applies to stream transports). No unused surface.
5. ALPN, additive to platform: ServerContext.init gains alpn: ?[*:null]const ?[*:0]const u8 (mbedtls_ssl_conf_alpn_protocols requires a NULL-terminated array that OUTLIVES the config — use comptime-constant arrays). DoH advertises ["http/1.1"], DoT ["dot"]. A client with no ALPN still connects (mbedTLS only enforces when the client sends the extension). Existing callers pass null; the milestone-1 spec note about "no ALPN" is superseded.
6. Cert holder + reload: new src/server/cert_store.zig. CertStore owns {mutex, current: *Entry, cert_path, key_path, gpa} where Entry = {ctx: ServerContext, refs: u32, retired: bool}. acquire(io) *Entry (+1 under mutex), release(io, entry) (−1; free when retired and 0). reload(io) ReloadError!void: read both PEM files (NUL-terminated dupes, 64 KiB cap each), ServerContext.init, on success swap + retire the old (freed when its refs drain), on ANY failure publish nothing (PLAN:297). Listeners acquire per connection (before accept) and release when the connection ends — so in-flight streams keep their generation alive. File read errors and init errors map to a typed error + a human message for the API.
7. Cert watcher: one task per enabled endpoint inside cert_store (watch(store, io) house loop: Cancelable!void, .boot clock, warn-and-continue). Poll every 30 s: stat cert AND key (Io.File.stat), compare mtime+size against the last LOADED pair; any change → reload(); reload failure → warn (the old cert keeps serving) + a reload_failures counter; success → info-level "certificate reloaded". mtime+size both compared (same-second atomic renames).
8. POST /api/certs/reload (PLAN:547): handler src/web/handlers/certs.zig (PLAN:232). auth = .session, rate_limit = .counted. Reloads every enabled endpoint's store; response 200 {"doh": {"enabled": bool, "reloaded": bool, "error": <msg>|null}, "dot": {...}} — per-endpoint outcome, 200 even when a reload fails (the outcome IS the payload; nothing about the server's own state is exceptional). Disabled endpoint: {enabled:false, reloaded:false, error:null}. WebState gains doh_certs: ?*CertStore and dot_certs: ?*CertStore (null-defaulted, W3 pattern).
9. openapi.yaml + contract test land together (m8 ruling 2): the path entry with the exact schema, and a contract-table entry + response-shape test in web_integration_test.zig. Drift guard (b) count moves 55 → 56 — the test derives the count from router.routes so only the yaml operation count assertion data changes.
10. Observability: each listener keeps tcp_server-shaped stats (connections, tls_handshake_failures, idle_timeouts, connection_errors) + cert_store keeps {reloads, reload_failures, last_reload_unix}. metrics.zig additively gains nxdns_doh_server_*, nxdns_dot_server_*, nxdns_cert_reloads_total, nxdns_cert_reload_failures_total families (counterGroup walks the structs — follow its existing pattern). /api/health is unchanged (no new rollup this milestone; the reload endpoint reports cert state on demand).
11. app.zig wiring: when cfg.doh_server.enabled, build its CertStore (initial load happens in serve BEFORE the group — a bad cert at boot is exit 2 with the validate-style message, matching nxdns check's contract that enabled endpoints have readable certs; the WATCHER handles later breakage gracefully), then the listener; same for dot. Watcher tasks + listener tasks join the group. WebState seams wired when web is also enabled. SIGTERM path unchanged.
12. Integration tests (both gated -Dintegration, loopback, fixtures): DoT: std.crypto.tls.Client (no_verification) → framed A query → framed reply (reuse resolver_integration_test's fake-upstream pattern or a local-records-only handler). DoH POST + GET: raw HTTP/1.1 text over std.crypto.tls.Client (no std.http.Client CA ceremony), assert status/content-type/bytes; 415/405/404/400 matrix; keep-alive two-requests test. Reload: swap fixture cert (write a second self-signed pair fixture), reload, NEW connection sees the new cert (std.crypto.tls.Client exposes the peer cert? if not: assert old connections still serve and new handshakes succeed — the observable contract), old connection still answers. Watcher: unit-test the decision function (stat-pair compare) pure; do not test 30 s timing.
13. New fixture: tests/fixtures/self_signed_cert2.pem + key2 (openssl one-liner, same profile as the existing pair; README updated) for reload tests.
14. Docs: NOT this milestone beyond openapi.yaml (docs/api rendering is Phase 10 — m8 ruling 3 unchanged).
15. Zig freeze elsewhere: dns/, filter/, cache/, storage/, existing server files (udp/tcp/handler) untouched except: app.zig (T6), metrics.zig + routes.zig + openapi.yaml + web_integration_test.zig + web/server.zig(WebState fields only) (T5). handler.zig is NOT touched (ruling 4).

Sessions

T1+T2 parallel first; then T3+T4+T5 parallel; then T6.

---

Session T1: platform ALPN

Owns src/platform/tls_server.zig (+ its tests). Add the alpn parameter per ruling 5 (extern mbedtls_ssl_conf_alpn_protocols, comptime NULL-terminated arrays, doc the lifetime rule), update existing callers/tests with null, add one test that a client negotiating "http/1.1" succeeds against a context advertising it (the loopback integration test grows the assertion — std.crypto.tls.Client: check whether it can send ALPN in 0.16; if it cannot, the unit test just proves config acceptance and the lifetime doc stands).

Acceptance: fmt/ast clean; plain + integration suites 0 failed.

T1 As built

ServerContext.init(gpa, cert_pem, key_pem, alpn: ?[*:null]const ?[*:0]const u8). Extern mbedtls_ssl_conf_alpn_protocols verified against vendored 3.6.7 ssl.h:4314 (pointer recorded, not copied — comptime-constant NULL-terminated arrays required, documented on init). MBEDTLS_SSL_ALPN is on in the stock config. std.crypto.tls.Client CANNOT send ALPN in 0.16 (no Options field) — negotiation untestable client-side; the unit test proves config acceptance, the integration echo proves no-ALPN clients still handshake against an advertising context. Servers can assert server-side via mbedtls_ssl_get_alpn_protocol if ever needed (extern not declared — unused surface). Mechanical ripple: tls_client_integration_test.zig:193 caller gained , null.

Session T2: cert_store

Owns src/server/cert_store.zig (new). Ruling 6 (holder + refcount + reload) and ruling 7 (watch loop + pure stat-compare decision fn) + ruling 10's counters + snapshotStats(). Unit tests: acquire/release refcounting across a swap (old entry freed only after last release), reload failure publishes nothing (bad path, bad PEM), NUL-termination and size cap, stat-compare decision table. Uses fixtures via test_fixtures import.

Acceptance: fmt/ast clean; temp-root green (needs -lc + vendored mbedtls objects).

T2 As built

16 in-file tests; temp-root 22 passed / 0 failed. Surface: poll_interval_s=30, max_pem_bytes=64KiB, ReloadError (10 cases) + humanMessage(err) []const u8; FileSig{mtime_ns: i96, size}, Signature{cert,key}, pure changed(loaded, observed) bool; Entry{ctx, refs, retired}; CertStore.init(gpa, io, cert_path, key_path, alpn) (boot load; typed error for T6's exit-2), deinit(io) (asserts current refs==0 — join listeners first), acquire(io) *Entry / release(io, entry) (release safe from any task; last releaser frees a retired entry), reload(io) ReloadError!void, watch(io) Cancelable!void (sleeps BEFORE first poll, .boot clock), pollOnce(io) (testable watcher pass), snapshotStats() (no io; {reloads, reload_failures, last_reload_unix}). Notes: use &entry.ctx for ServerStream.accept; paths are BORROWED (config outlives the store); reloads counts successes only; stat failures warn but are not reload_failures; last_reload_unix set at init too.

Session T3: dot_server (after T1+T2)

Owns src/server/dot_server.zig (new) + its integration test file if separate (prefer tests inside the file, house pattern). Ruling 3: tcp_server loop shape over ServerStream; acquire cert entry per connection, release on exit; handshake under the race budget; stats per ruling 10; Stop enum; max_connections 64; ALPN ["dot"]. Integration tests per ruling 12 (DoT half).

Acceptance: fmt/ast clean; both suites 0 failed.

T3 As built

DotServer.listen(gpa, io, listen_address, h: *handler.Handler, certs: *cert_store.CertStore, options) / serve(io) / boundAddress() / deinit(io). Options {max_connections: u16 = 64, idle_timeout = 10s}; the handshake runs under the same idle budget. Deviation from tcp_server: deinit takes no gpa — the allocator is stored at listen because ServerStream.accept heap-allocates one mbedTLS ssl context per connection (documented in the file header). Stats atomics: the four ruling-10 fields plus tcp-pattern internals rejected_at_capacity, rejected_at_shutdown, accept_errors; snapshotStats() returns exactly the ruling-10 four. Loop: accept → claim slot → certs.acquire (released on every exit) → handshake raced against the idle budget → 2-byte-prefix loop over ServerStream reader/writer, flush per reply, handler called with .tcp, tls.close(gpa) (close_notify) on every post-handshake exit. A handshook flag ensures the ssl context closes exactly once when Select reports expiry/cancel after a successful handshake; a timed-out handshake counts as tls_handshake_failures. ALPN is set only via the store (T6). 7 unit + 4 integration tests in-file. Teardown order: server.deinit(io) before CertStore.deinit.

Session T4: doh_server (after T1+T2)

Owns src/server/doh_server.zig (new). Ruling 2 in full: std.http.Server over ServerStream reader/writer (recv buffer 8 KiB = head cap, send 4 KiB), connection group + 64 cap + 503-free refusal is NOT applicable here (no HTTP yet at accept overflow: over cap → close raw, cheapest honest behavior — document); per-connection Scratch + response_buf (65535); GET base64url decode (url_safe_no_pad, reject padding/ whitespace), POST body cap via Io.Limit; keep-alive; bodyless-POST normalization; ALPN ["http/1.1"]; cert acquire/release per connection; stats. Integration tests per ruling 12 (DoH half: POST, GET, 415/405/404/400, keep-alive).

Acceptance: fmt/ast clean; both suites 0 failed.

T4 As built

DohServer.listen(gpa, io, address, *Handler, *CertStore, Options) / serve(io) / deinit(gpa, io) / boundAddress() / snapshotStats(). Options {max_connections = 64, idle_timeout = 10s} — the budget races the TLS handshake only; requests have no timeout (web-listener precedent). Module-level serve(gpa, io, model.TlsEndpoint, *Handler, *CertStore) is the bind-warn-and-continue entry for T6. Exposed comptime alpn_protocols (["http/1.1"]) and dns_query_path. Stats: the ruling-10 four + bad_requests + tcp-shaped accept counters (rejected_at_capacity, rejected_at_shutdown, accept_errors) — over-capacity refusals must stay visible. Deviations: POST cap via content-length precheck + fixed buffer + one-byte probe for chunked bodies instead of Io.Limit (no allocation on the hot path); respond can raise HttpExpectationFailed (100-continue), mapped to connection_errors; no 413 integration test (ruling 12's matrix is 415/405/404/400). GET ?dns= parsed by a local validating decoder (duplicate param → 400, % and padding/whitespace rejected). Oversize POST answers 413 with keep_alive=false (body never drained). Over the 64 cap the raw TCP stream closes (no HTTP before a handshake). 8 unit + 4 integration tests. T6 recipe: CertStore.init(gpa, io, cert_path, key_path, doh_server.alpn_protocols); run doh_server.serve and store.watch as group tasks; DohServer.deinit before CertStore.deinit.

Session T5: web endpoint + metrics (after T2, parallel with T3/T4)

Owns src/web/handlers/certs.zig (new), src/web/routes.zig (one entry), src/web/openapi.yaml (one path), src/web/server.zig (two null-defaulted WebState fields ONLY), src/web/metrics.zig (ruling 10 families), and the additive web_integration_test.zig edits (contract entry + shape test + guard-b count). Handler per ruling 8 (apply/respond split, house pattern). Unit tests for the apply fn (both stores null; one store failing reload → error message in payload, still 200).

Acceptance: fmt/ast clean; plain suite 0 failed; drift guards green.

T5 As built

handlers/certs.zig: apply/respond split — applyReload(state, io) View with View{doh, dot} and Outcome{enabled, reloaded, @"error": ?[]const u8} (serialized as error); post always answers 200. Route: POST /api/certs/reload, .session, .counted; table 55→56; contract entry + strict shape test + drift guards green at 56. metrics.zig: the two ruling-10 cert counters share one family pair (nxdns_cert_reloads_total / nxdns_cert_reload_failures_total) with an endpoint="doh"|"dot" label (upstream-label pattern); unwired stores omit the family; last_reload_unix not exported. WebState gains doh_certs/dot_certs: ?*cert_store.CertStore = null. Deviation (accepted): deleted the m8 tripwire test in src/web/openapi.zig ("the document does not promise what phase 9 owns") — it exists to fire when this endpoint lands. Deferred to T6: nxdns_doh_server_* / nxdns_dot_server_* listener families (T3/T4 types did not exist at T5 build time).

Session T6: app wiring + fixtures + smoke (after all)

Owns src/app.zig, tests/fixtures/ (cert2/key2 + README + fixtures.zig), and src/tests.zig is the ORCHESTRATOR's (report lines). Ruling 11 wiring; ruling 13 fixture. Smoke (report transcript): boot with doh+dot enabled on unprivileged ports with fixture certs; DoT query via a scripted TLS client (a tiny zig run or the integration test binary is fine — no new tooling); DoH POST via curl --insecure --http1.1 with a binary body; certs/reload via the API twice (success, then chmod the key unreadable → error in payload, old cert still serving a new connection); SIGTERM 0. Both suites + cross ReleaseSafe with the SPA dist.

Acceptance: full gates green; smoke transcript.

T6 As built

app.zig realizes ruling 11 in-frame: openCertStore maps non-OOM ReloadError to error.BadCertificate (exit 2, prints both paths + cert_store.humanMessage); bindDoh/bindDot warn-and-continue per ruling 1; listeners + two store.watch loops join the group; stores are declared before listeners so teardown runs listener-deinit → store-deinit and the refs==0 assert holds. dot_alpn (["dot"]) lives in app.zig. Deviation: doh_server.serve module entry is NOT used — it constructs the DohServer in its own task frame, so WebState could never get the pointer the nxdns_doh_server_* family needs; both listeners bind in app.zig's frame instead (tcp_server style). doh_server.serve remains as unused pub API. metrics.zig: DohListenerSample (ruling-10 four + bad_requests); DoT renders dot_server.StatsSnapshot directly; accept-side counters stay off the exposition; unwired listeners omit the families. WebState gains doh_listener/dot_listener optional pointers. build.zig (out of ownership, necessary): the exe now compiles mbedtls_shim.c with the threading macros — first exe-side tls_server reference; mirrors the tests wiring. Fixtures: cert2/key2 exported as cert2_pem/key2_pem (distinct fingerprint), currently unreferenced by tests. Config: model.TlsEndpoint already existed; no model/validate/schema edits. Smoke: DoT framed query answered from local records; DoH POST 200 application/dns-message; reload success then key chmod 000 → reloaded:false with "private key file is not readable" while the old cert keeps serving; SIGTERM 0; boot with unreadable key exits 2.

---

Review (Codex, as built)

Four rounds on one thread; round 4 returned "No findings."

Round 1 (3 important, 2 minor): doh_server leaked the TLS context when the handshake won a race the Select reported as timeout/cancel → handshook flag (dot_server pattern). cert_store reloads were not serialized across read→build→publish, so an older read could publish last → reload_mutex held across the whole sequence (lock order: reload_mutex before the entry mutex; accept path untouched) + a hook-pinned regression test (after_load_hook seam). doh bodied-GET smuggling: std leaves unframed body bytes for bodyless methods to be parsed as the next request → framesUnreadableBody guard, 400 + close, no byte read. .drop now closes per ruling 2. cert2/key2 were unreferenced → DoT reload-to-distinct-identity integration test (old connection keeps serving, new handshake succeeds, distinct entry pointers; the 0.16 tls Client does not retain the peer chain, so identity is proven via entry pointers).

Round 2 (2 minor): the smuggling guard ran before routing and stole 404/405 statuses → routing decides the status, framed requests only force close. Successful POSTs ignored the client's Connection: close → answer/refuse honor head.keep_alive in the Next enum (the wire already did via Server.zig:628).

Round 3 (1 important, introduced by the round-2 reshape): bodied refusals (404/405/415) kept keep-alive, so respond drained the whole body first — an unterminated chunked body could pin all 64 slots with no response → framesBody helper; any pre-body-read refusal closes when a body is framed; drain-free integration tests (missing terminal chunk → 404+EOF at once; undelivered content-length 4096 → 415+EOF); errorMatrix's 415 became a zero-length wrong-type POST so its keep-alive reuse stays legitimate.

Final counts: doh_server.zig 19 tests, dot_server.zig 12, cert_store.zig 17. Final gates: plain 1166/1279 passed, 113 skipped (integration-gated), 0 failed; integration 1275/1279, 4 skipped (live-network by design), 0 failed.

Module layout (new)

src/server/{doh_server,dot_server,cert_store}.zig, src/web/handlers/certs.zig, tests/fixtures/{self_signed_cert2.pem,self_signed_key2.pem}.

File ownership

T1 platform/tls_server.zig; T2 server/cert_store.zig; T3 server/dot_server.zig; T4 server/doh_server.zig; T5 web/{handlers/certs.zig,routes.zig,openapi.yaml, server.zig(fields),metrics.zig,web_integration_test.zig}; T6 app.zig + fixtures. Orchestrator: src/tests.zig, spec. Parallel sessions never share a file.

Acceptance (milestone complete)

• Both suites 0 failed (plain + -Dintegration), zero err lines; cross ReleaseSafe with SPA dist green.
• DoT and DoH loopback integration tests green (query → reply, error matrix, keep-alive).
• certs/reload endpoint + openapi entry + contract test land together; drift guards green at 56.
• Cert watcher reloads on file change; failed reload keeps serving the old cert.
• T6 smoke transcript incl. reload-failure path and SIGTERM 0.
• Spec As-built synced per session.

Anti-requirements

No HTTP/2, no DoQ, no SNI/multi-cert, no client-cert auth, no session tickets, no proxy-protocol/XFF handling, no cert generation (ACME etc.), no cache-control on DoH, no handler.zig changes, no docs/api rendering (Phase 10), no config schema changes (TlsEndpoint exists), no inotify.