# Milestone 10: DoH/DoT server + cert watcher + certs/reload (PLAN Phase 9)

Goal: LAN clients resolve via DoH (RFC 8484 over HTTP/1.1 + TLS) and DoT (RFC 7858) against local certs; certs hot-reload via a polling watcher and `POST /api/certs/reload` (endpoint + openapi entry + contract test, deferred whole from milestone-8 ruling 2).

Ground truth: PLAN.md:23,155,194,232,500-503,525,547,615-616,647,664 (scope, layout, config keys, endpoint, exit); PLAN.md:684 — HTTP/2 and DoQ are OUT; PLAN.md:297 — "a failed reload publishes neither". Explore facts (verified this session):
- platform/tls_server.zig: ServerContext.init(gpa, cert_pem, key_pem) shared across connections; binding is per-accept (mbedtls_ssl_setup at accept), so reload = publish a new context pointer; the OLD context must outlive every stream set up against it. ServerStream.reader()/writer() are real *Io.Reader/*Io.Writer; close(gpa) sends close_notify but does NOT close the TCP stream. MBEDTLS_THREADING_C + PTHREAD are ON (build.zig:255) — one context is safe under concurrent handshakes.
- std.http.Server.init(in: *Reader, out: *Writer) takes ARBITRARY reader/writer (http/Server.zig:25) — web/server.zig's serveConn transfers with three lines changed.
- No usable file watching in 0.16 (inotify/fanotify are raw syscalls outside Io) — the watcher POLLS Io.File.stat mtime+size.
- Handler.handle(self, io, which: Transport, from, query, response_buf, scratch) (handler.zig:164); Transport {udp,tcp} drives truncation only; response_buf 512..65535; Scratch ~7 KiB per connection. tcp_server serveConn (tcp_server.zig:265-338) is the DoT loop verbatim modulo the stream; dns/transport.zig has prefix_len/parsePrefix/ framePrefix/max_message_len.
- doh_client.zig: media_type "application/dns-message" (:16), contentTypeOk exported (:171). base64 url_safe_no_pad = the RFC 8484 GET codec. http_util.queryPairs for ?dns=.
- Fixtures: tests/fixtures self_signed cert/key via the test_fixtures anonymous import.
- Config: model.TlsEndpoint already exists (doh_server port 443, dot_server 853), validated, settings-persisted, checked by `nxdns check`. app.zig ignores both today.

## Rulings (binding)

1. **Two listeners**: `src/server/doh_server.zig`, `src/server/dot_server.zig` (PLAN:194), constructed only when their endpoint is enabled. Bind failure = warn + continue (web precedent). Both join app.zig's group; cancel semantics mirror tcp_server's Stop enum (.canceled cancels the connection group; listener close drains).
2. **DoH protocol**: HTTP/1.1 only (PLAN:684 excludes h2). Path `/dns-query` exactly. `POST` with `content-type: application/dns-message` (checked via doh_client.contentTypeOk) and `GET ?dns=<base64url unpadded>`. Responses: 200 `application/dns-message` with the reply bytes; handler `.drop` → 502 with empty body reason? NO — `.drop` means "no answer on purpose" (malformed/refused-silent): close the connection for POST/GET alike via 400. Errors: 404 non-/dns-query path, 405 other methods (Allow: GET, POST), 415 wrong content-type, 400 missing/invalid `dns` param or empty body, 413 body > dns/transport.max_message_len. No cache-control headers (LAN, no intermediaries). Keep-alive per std.http.Server loop; the bodyless-POST normalization from web/server.zig:443 is copied (same stdlib assert).
3. **DoT protocol**: RFC 7858 = the tcp_server loop over ServerStream: 2-byte prefix, parsePrefix, reject 0, read body, handle, frame + write + flush. idle_timeout race identical (same Options shape, max_connections 64, per-conn Scratch). The TLS handshake itself runs under the same race budget (a stalled handshake must not pin a connection slot).
4. **Transport enum stays {udp, tcp}**: DoH/DoT call handle with `.tcp` (it only drives truncation, which never applies to stream transports). No unused surface.
5. **ALPN, additive to platform**: ServerContext.init gains `alpn: ?[*:null]const ?[*:0]const u8` (mbedtls_ssl_conf_alpn_protocols requires a NULL-terminated array that OUTLIVES the config — use comptime-constant arrays). DoH advertises ["http/1.1"], DoT ["dot"]. A client with no ALPN still connects (mbedTLS only enforces when the client sends the extension). Existing callers pass null; the milestone-1 spec note about "no ALPN" is superseded.
6. **Cert holder + reload**: new `src/server/cert_store.zig`. `CertStore` owns {mutex, current: *Entry, cert_path, key_path, gpa} where Entry = {ctx: ServerContext, refs: u32, retired: bool}. `acquire(io) *Entry` (+1 under mutex), `release(io, entry)` (−1; free when retired and 0). `reload(io) ReloadError!void`: read both PEM files (NUL-terminated dupes, 64 KiB cap each), ServerContext.init, on success swap + retire the old (freed when its refs drain), on ANY failure publish nothing (PLAN:297). Listeners acquire per connection (before accept) and release when the connection ends — so in-flight streams keep their generation alive. File read errors and init errors map to a typed error + a human message for the API.
7. **Cert watcher**: one task per enabled endpoint inside cert_store (`watch(store, io)` house loop: Cancelable!void, .boot clock, warn-and-continue). Poll every 30 s: stat cert AND key (Io.File.stat), compare mtime+size against the last LOADED pair; any change → reload(); reload failure → warn (the old cert keeps serving) + a `reload_failures` counter; success → info-level "certificate reloaded". mtime+size both compared (same-second atomic renames).
8. **`POST /api/certs/reload`** (PLAN:547): handler `src/web/handlers/certs.zig` (PLAN:232). auth = .session, rate_limit = .counted. Reloads every enabled endpoint's store; response 200 `{"doh": {"enabled": bool, "reloaded": bool, "error": <msg>|null}, "dot": {...}}` — per-endpoint outcome, 200 even when a reload fails (the outcome IS the payload; nothing about the server's own state is exceptional). Disabled endpoint: {enabled:false, reloaded:false, error:null}. WebState gains `doh_certs: ?*CertStore` and `dot_certs: ?*CertStore` (null-defaulted, W3 pattern).
9. **openapi.yaml + contract test land together** (m8 ruling 2): the path entry with the exact schema, and a contract-table entry + response-shape test in web_integration_test.zig. Drift guard (b) count moves 55 → 56 — the test derives the count from router.routes so only the yaml operation count assertion data changes.
10. **Observability**: each listener keeps tcp_server-shaped stats (connections, tls_handshake_failures, idle_timeouts, connection_errors) + cert_store keeps {reloads, reload_failures, last_reload_unix}. metrics.zig additively gains `nxdns_doh_server_*`, `nxdns_dot_server_*`, `nxdns_cert_reloads_total`, `nxdns_cert_reload_failures_total` families (counterGroup walks the structs — follow its existing pattern). /api/health is unchanged (no new rollup this milestone; the reload endpoint reports cert state on demand).
11. **app.zig wiring**: when `cfg.doh_server.enabled`, build its CertStore (initial load happens in serve BEFORE the group — a bad cert at boot is exit 2 with the validate-style message, matching `nxdns check`'s contract that enabled endpoints have readable certs; the WATCHER handles later breakage gracefully), then the listener; same for dot. Watcher tasks + listener tasks join the group. WebState seams wired when web is also enabled. SIGTERM path unchanged.
12. **Integration tests** (both gated `-Dintegration`, loopback, fixtures): DoT: std.crypto.tls.Client (no_verification) → framed A query → framed reply (reuse resolver_integration_test's fake-upstream pattern or a local-records-only handler). DoH POST + GET: raw HTTP/1.1 text over std.crypto.tls.Client (no std.http.Client CA ceremony), assert status/content-type/bytes; 415/405/404/400 matrix; keep-alive two-requests test. Reload: swap fixture cert (write a second self-signed pair fixture), reload, NEW connection sees the new cert (std.crypto.tls.Client exposes the peer cert? if not: assert old connections still serve and new handshakes succeed — the observable contract), old connection still answers. Watcher: unit-test the decision function (stat-pair compare) pure; do not test 30 s timing.
13. **New fixture**: `tests/fixtures/self_signed_cert2.pem` + key2 (openssl one-liner, same profile as the existing pair; README updated) for reload tests.
14. **Docs**: NOT this milestone beyond openapi.yaml (docs/api rendering is Phase 10 — m8 ruling 3 unchanged).
15. **Zig freeze elsewhere**: dns/, filter/, cache/, storage/, existing server files (udp/tcp/handler) untouched except: app.zig (T6), metrics.zig + routes.zig + openapi.yaml + web_integration_test.zig + web/server.zig(WebState fields only) (T5). handler.zig is NOT touched (ruling 4).

## Sessions

T1+T2 parallel first; then T3+T4+T5 parallel; then T6.

---

## Session T1: platform ALPN

Owns `src/platform/tls_server.zig` (+ its tests). Add the alpn parameter per ruling 5 (extern mbedtls_ssl_conf_alpn_protocols, comptime NULL-terminated arrays, doc the lifetime rule), update existing callers/tests with null, add one test that a client negotiating "http/1.1" succeeds against a context advertising it (the loopback integration test grows the assertion — std.crypto.tls.Client: check whether it can send ALPN in 0.16; if it cannot, the unit test just proves config acceptance and the lifetime doc stands).

Acceptance: fmt/ast clean; plain + integration suites 0 failed.

### T1 As built

`ServerContext.init(gpa, cert_pem, key_pem, alpn: ?[*:null]const ?[*:0]const u8)`. Extern `mbedtls_ssl_conf_alpn_protocols` verified against vendored 3.6.7 ssl.h:4314 (pointer recorded, not copied — comptime-constant NULL-terminated arrays required, documented on init). MBEDTLS_SSL_ALPN is on in the stock config. std.crypto.tls.Client CANNOT send ALPN in 0.16 (no Options field) — negotiation untestable client-side; the unit test proves config acceptance, the integration echo proves no-ALPN clients still handshake against an advertising context. Servers can assert server-side via mbedtls_ssl_get_alpn_protocol if ever needed (extern not declared — unused surface). Mechanical ripple: tls_client_integration_test.zig:193 caller gained `, null`.

## Session T2: cert_store

Owns `src/server/cert_store.zig` (new). Ruling 6 (holder + refcount + reload) and ruling 7 (watch loop + pure stat-compare decision fn) + ruling 10's counters + `snapshotStats()`. Unit tests: acquire/release refcounting across a swap (old entry freed only after last release), reload failure publishes nothing (bad path, bad PEM), NUL-termination and size cap, stat-compare decision table. Uses fixtures via test_fixtures import.

Acceptance: fmt/ast clean; temp-root green (needs -lc + vendored mbedtls objects).

### T2 As built

16 in-file tests; temp-root 22 passed / 0 failed. Surface: `poll_interval_s=30`, `max_pem_bytes=64KiB`, `ReloadError` (10 cases) + `humanMessage(err) []const u8`; `FileSig{mtime_ns: i96, size}`, `Signature{cert,key}`, pure `changed(loaded, observed) bool`; `Entry{ctx, refs, retired}`; `CertStore.init(gpa, io, cert_path, key_path, alpn)` (boot load; typed error for T6's exit-2), `deinit(io)` (asserts current refs==0 — join listeners first), `acquire(io) *Entry` / `release(io, entry)` (release safe from any task; last releaser frees a retired entry), `reload(io) ReloadError!void`, `watch(io) Cancelable!void` (sleeps BEFORE first poll, .boot clock), `pollOnce(io)` (testable watcher pass), `snapshotStats()` (no io; {reloads, reload_failures, last_reload_unix}). Notes: use `&entry.ctx` for ServerStream.accept; paths are BORROWED (config outlives the store); `reloads` counts successes only; stat failures warn but are not reload_failures; `last_reload_unix` set at init too.

## Session T3: dot_server (after T1+T2)

Owns `src/server/dot_server.zig` (new) + its integration test file if separate (prefer tests inside the file, house pattern). Ruling 3: tcp_server loop shape over ServerStream; acquire cert entry per connection, release on exit; handshake under the race budget; stats per ruling 10; Stop enum; max_connections 64; ALPN ["dot"]. Integration tests per ruling 12 (DoT half).

Acceptance: fmt/ast clean; both suites 0 failed.

### T3 As built

`DotServer.listen(gpa, io, listen_address, h: *handler.Handler, certs: *cert_store.CertStore, options)` / `serve(io)` / `boundAddress()` / `deinit(io)`. Options `{max_connections: u16 = 64, idle_timeout = 10s}`; the handshake runs under the same idle budget. Deviation from tcp_server: `deinit` takes no gpa — the allocator is stored at `listen` because `ServerStream.accept` heap-allocates one mbedTLS ssl context per connection (documented in the file header). Stats atomics: the four ruling-10 fields plus tcp-pattern internals `rejected_at_capacity, rejected_at_shutdown, accept_errors`; `snapshotStats()` returns exactly the ruling-10 four. Loop: accept → claim slot → `certs.acquire` (released on every exit) → handshake raced against the idle budget → 2-byte-prefix loop over ServerStream reader/writer, flush per reply, handler called with `.tcp`, `tls.close(gpa)` (close_notify) on every post-handshake exit. A `handshook` flag ensures the ssl context closes exactly once when Select reports expiry/cancel after a successful handshake; a timed-out handshake counts as `tls_handshake_failures`. ALPN is set only via the store (T6). 7 unit + 4 integration tests in-file. Teardown order: `server.deinit(io)` before `CertStore.deinit`.

## Session T4: doh_server (after T1+T2)

Owns `src/server/doh_server.zig` (new). Ruling 2 in full: std.http.Server over ServerStream reader/writer (recv buffer 8 KiB = head cap, send 4 KiB), connection group + 64 cap + 503-free refusal is NOT applicable here (no HTTP yet at accept overflow: over cap → close raw, cheapest honest behavior — document); per-connection Scratch + response_buf (65535); GET base64url decode (url_safe_no_pad, reject padding/ whitespace), POST body cap via Io.Limit; keep-alive; bodyless-POST normalization; ALPN ["http/1.1"]; cert acquire/release per connection; stats. Integration tests per ruling 12 (DoH half: POST, GET, 415/405/404/400, keep-alive).

Acceptance: fmt/ast clean; both suites 0 failed.

### T4 As built

`DohServer.listen(gpa, io, address, *Handler, *CertStore, Options)` / `serve(io)` / `deinit(gpa, io)` / `boundAddress()` / `snapshotStats()`. Options `{max_connections = 64, idle_timeout = 10s}` — the budget races the TLS handshake only; requests have no timeout (web-listener precedent). Module-level `serve(gpa, io, model.TlsEndpoint, *Handler, *CertStore)` is the bind-warn-and-continue entry for T6. Exposed comptime `alpn_protocols` (["http/1.1"]) and `dns_query_path`. Stats: the ruling-10 four + `bad_requests` + tcp-shaped accept counters (`rejected_at_capacity, rejected_at_shutdown, accept_errors`) — over-capacity refusals must stay visible. Deviations: POST cap via content-length precheck + fixed buffer + one-byte probe for chunked bodies instead of Io.Limit (no allocation on the hot path); `respond` can raise `HttpExpectationFailed` (100-continue), mapped to `connection_errors`; no 413 integration test (ruling 12's matrix is 415/405/404/400). GET `?dns=` parsed by a local validating decoder (duplicate param → 400, `%` and padding/whitespace rejected). Oversize POST answers 413 with `keep_alive=false` (body never drained). Over the 64 cap the raw TCP stream closes (no HTTP before a handshake). 8 unit + 4 integration tests. T6 recipe: `CertStore.init(gpa, io, cert_path, key_path, doh_server.alpn_protocols)`; run `doh_server.serve` and `store.watch` as group tasks; `DohServer.deinit` before `CertStore.deinit`.

## Session T5: web endpoint + metrics (after T2, parallel with T3/T4)

Owns `src/web/handlers/certs.zig` (new), `src/web/routes.zig` (one entry), `src/web/openapi.yaml` (one path), `src/web/server.zig` (two null-defaulted WebState fields ONLY), `src/web/metrics.zig` (ruling 10 families), and the additive web_integration_test.zig edits (contract entry + shape test + guard-b count). Handler per ruling 8 (apply/respond split, house pattern). Unit tests for the apply fn (both stores null; one store failing reload → error message in payload, still 200).

Acceptance: fmt/ast clean; plain suite 0 failed; drift guards green.

### T5 As built

handlers/certs.zig: apply/respond split — `applyReload(state, io) View` with `View{doh, dot}` and `Outcome{enabled, reloaded, @"error": ?[]const u8}` (serialized as `error`); `post` always answers 200. Route: `POST /api/certs/reload`, `.session`, `.counted`; table 55→56; contract entry + strict shape test + drift guards green at 56. metrics.zig: the two ruling-10 cert counters share one family pair (`nxdns_cert_reloads_total` / `nxdns_cert_reload_failures_total`) with an `endpoint="doh"|"dot"` label (upstream-label pattern); unwired stores omit the family; `last_reload_unix` not exported. WebState gains `doh_certs`/`dot_certs: ?*cert_store.CertStore = null`. Deviation (accepted): deleted the m8 tripwire test in `src/web/openapi.zig` ("the document does not promise what phase 9 owns") — it exists to fire when this endpoint lands. Deferred to T6: `nxdns_doh_server_*` / `nxdns_dot_server_*` listener families (T3/T4 types did not exist at T5 build time).

## Session T6: app wiring + fixtures + smoke (after all)

Owns `src/app.zig`, `tests/fixtures/` (cert2/key2 + README + fixtures.zig), and `src/tests.zig` is the ORCHESTRATOR's (report lines). Ruling 11 wiring; ruling 13 fixture. Smoke (report transcript): boot with doh+dot enabled on unprivileged ports with fixture certs; DoT query via a scripted TLS client (a tiny zig run or the integration test binary is fine — no new tooling); DoH POST via curl --insecure --http1.1 with a binary body; certs/reload via the API twice (success, then chmod the key unreadable → error in payload, old cert still serving a new connection); SIGTERM 0. Both suites + cross ReleaseSafe with the SPA dist.

Acceptance: full gates green; smoke transcript.

### T6 As built

app.zig realizes ruling 11 in-frame: `openCertStore` maps non-OOM `ReloadError` to `error.BadCertificate` (exit 2, prints both paths + `cert_store.humanMessage`); `bindDoh`/`bindDot` warn-and-continue per ruling 1; listeners + two `store.watch` loops join the group; stores are declared before listeners so teardown runs listener-deinit → store-deinit and the refs==0 assert holds. `dot_alpn` (["dot"]) lives in app.zig. Deviation: `doh_server.serve` module entry is NOT used — it constructs the DohServer in its own task frame, so WebState could never get the pointer the `nxdns_doh_server_*` family needs; both listeners bind in app.zig's frame instead (tcp_server style). `doh_server.serve` remains as unused pub API. **Superseded by milestone-18 ruling 1: the unused `doh_server.serve` is deleted. Nothing ever called it, and a dead second composition path over the listener core is exactly the divergence that milestone collapsed. app.zig's in-frame bind is now the only way DoH comes up.** metrics.zig: `DohListenerSample` (ruling-10 four + `bad_requests`); DoT renders `dot_server.StatsSnapshot` directly; accept-side counters stay off the exposition; unwired listeners omit the families. WebState gains `doh_listener`/`dot_listener` optional pointers. build.zig (out of ownership, necessary): the exe now compiles `mbedtls_shim.c` with the threading macros — first exe-side tls_server reference; mirrors the tests wiring. Fixtures: cert2/key2 exported as `cert2_pem`/`key2_pem` (distinct fingerprint), currently unreferenced by tests. Config: `model.TlsEndpoint` already existed; no model/validate/schema edits. Smoke: DoT framed query answered from local records; DoH POST 200 application/dns-message; reload success then key chmod 000 → `reloaded:false` with "private key file is not readable" while the old cert keeps serving; SIGTERM 0; boot with unreadable key exits 2.

---

## Review (Codex, as built)

Four rounds on one thread; round 4 returned "No findings."

Round 1 (3 important, 2 minor): doh_server leaked the TLS context when the handshake won a race the Select reported as timeout/cancel → handshook flag (dot_server pattern). cert_store reloads were not serialized across read→build→publish, so an older read could publish last → `reload_mutex` held across the whole sequence (lock order: reload_mutex before the entry mutex; accept path untouched) + a hook-pinned regression test (`after_load_hook` seam). doh bodied-GET smuggling: std leaves unframed body bytes for bodyless methods to be parsed as the next request → `framesUnreadableBody` guard, 400 + close, no byte read. `.drop` now closes per ruling 2. cert2/key2 were unreferenced → DoT reload-to-distinct-identity integration test (old connection keeps serving, new handshake succeeds, distinct entry pointers; the 0.16 tls Client does not retain the peer chain, so identity is proven via entry pointers).

Round 2 (2 minor): the smuggling guard ran before routing and stole 404/405 statuses → routing decides the status, framed requests only force close. Successful POSTs ignored the client's `Connection: close` → answer/refuse honor `head.keep_alive` in the Next enum (the wire already did via Server.zig:628).

Round 3 (1 important, introduced by the round-2 reshape): bodied refusals (404/405/415) kept keep-alive, so respond drained the whole body first — an unterminated chunked body could pin all 64 slots with no response → `framesBody` helper; any pre-body-read refusal closes when a body is framed; drain-free integration tests (missing terminal chunk → 404+EOF at once; undelivered content-length 4096 → 415+EOF); errorMatrix's 415 became a zero-length wrong-type POST so its keep-alive reuse stays legitimate.

Final counts: doh_server.zig 19 tests, dot_server.zig 12, cert_store.zig 17. Final gates: plain 1166/1279 passed, 113 skipped (integration-gated), 0 failed; integration 1275/1279, 4 skipped (live-network by design), 0 failed.

## Module layout (new)

src/server/{doh_server,dot_server,cert_store}.zig, src/web/handlers/certs.zig, tests/fixtures/{self_signed_cert2.pem,self_signed_key2.pem}.

## File ownership

T1 platform/tls_server.zig; T2 server/cert_store.zig; T3 server/dot_server.zig; T4 server/doh_server.zig; T5 web/{handlers/certs.zig,routes.zig,openapi.yaml, server.zig(fields),metrics.zig,web_integration_test.zig}; T6 app.zig + fixtures. Orchestrator: src/tests.zig, spec. Parallel sessions never share a file.

## Acceptance (milestone complete)

- [ ] Both suites 0 failed (plain + -Dintegration), zero err lines; cross ReleaseSafe
      with SPA dist green.
- [ ] DoT and DoH loopback integration tests green (query → reply, error matrix,
      keep-alive).
- [ ] certs/reload endpoint + openapi entry + contract test land together; drift
      guards green at 56.
- [ ] Cert watcher reloads on file change; failed reload keeps serving the old cert.
- [ ] T6 smoke transcript incl. reload-failure path and SIGTERM 0.
- [ ] Spec As-built synced per session.

## Anti-requirements

No HTTP/2, no DoQ, no SNI/multi-cert, no client-cert auth, no session tickets, no proxy-protocol/XFF handling, no cert generation (ACME etc.), no cache-control on DoH, no handler.zig changes, no docs/api rendering (Phase 10), no config schema changes (TlsEndpoint exists), no inotify.
