nxdns/specs/milestone-10.md

# Milestone 10: DoH/DoT server + cert watcher + certs/reload (PLAN Phase 9)

Goal: LAN clients resolve via DoH (RFC 8484 over HTTP/1.1 + TLS) and DoT (RFC 7858)
against local certs; certs hot-reload via a polling watcher and `POST /api/certs/reload`
(endpoint + openapi entry + contract test, deferred whole from milestone-8 ruling 2).

Ground truth: PLAN.md:23,155,194,232,500-503,525,547,615-616,647,664 (scope, layout,
config keys, endpoint, exit); PLAN.md:684 — HTTP/2 and DoQ are OUT; PLAN.md:297 —
"a failed reload publishes neither". Explore facts (verified this session):
- platform/tls_server.zig: ServerContext.init(gpa, cert_pem, key_pem) shared across
  connections; binding is per-accept (mbedtls_ssl_setup at accept), so reload = publish
  a new context pointer; the OLD context must outlive every stream set up against it.
  ServerStream.reader()/writer() are real *Io.Reader/*Io.Writer; close(gpa) sends
  close_notify but does NOT close the TCP stream. MBEDTLS_THREADING_C + PTHREAD are ON
  (build.zig:255) — one context is safe under concurrent handshakes.
- std.http.Server.init(in: *Reader, out: *Writer) takes ARBITRARY reader/writer
  (http/Server.zig:25) — web/server.zig's serveConn transfers with three lines changed.
- No usable file watching in 0.16 (inotify/fanotify are raw syscalls outside Io) —
  the watcher POLLS Io.File.stat mtime+size.
- Handler.handle(self, io, which: Transport, from, query, response_buf, scratch)
  (handler.zig:164); Transport {udp,tcp} drives truncation only; response_buf 512..65535;
  Scratch ~7 KiB per connection. tcp_server serveConn (tcp_server.zig:265-338) is the
  DoT loop verbatim modulo the stream; dns/transport.zig has prefix_len/parsePrefix/
  framePrefix/max_message_len.
- doh_client.zig: media_type "application/dns-message" (:16), contentTypeOk exported
  (:171). base64 url_safe_no_pad = the RFC 8484 GET codec. http_util.queryPairs for ?dns=.
- Fixtures: tests/fixtures self_signed cert/key via the test_fixtures anonymous import.
- Config: model.TlsEndpoint already exists (doh_server port 443, dot_server 853),
  validated, settings-persisted, checked by `nxdns check`. app.zig ignores both today.

## Rulings (binding)

1. **Two listeners**: `src/server/doh_server.zig`, `src/server/dot_server.zig`
   (PLAN:194), constructed only when their endpoint is enabled. Bind failure = warn +
   continue (web precedent). Both join app.zig's group; cancel semantics mirror
   tcp_server's Stop enum (.canceled cancels the connection group; listener close
   drains).
2. **DoH protocol**: HTTP/1.1 only (PLAN:684 excludes h2). Path `/dns-query` exactly.
   `POST` with `content-type: application/dns-message` (checked via
   doh_client.contentTypeOk) and `GET ?dns=<base64url unpadded>`. Responses:
   200 `application/dns-message` with the reply bytes; handler `.drop` → 502 with empty
   body reason? NO — `.drop` means "no answer on purpose" (malformed/refused-silent):
   close the connection for POST/GET alike via 400. Errors: 404 non-/dns-query path,
   405 other methods (Allow: GET, POST), 415 wrong content-type, 400 missing/invalid
   `dns` param or empty body, 413 body > dns/transport.max_message_len. No
   cache-control headers (LAN, no intermediaries). Keep-alive per std.http.Server loop;
   the bodyless-POST normalization from web/server.zig:443 is copied (same stdlib
   assert).
3. **DoT protocol**: RFC 7858 = the tcp_server loop over ServerStream: 2-byte prefix,
   parsePrefix, reject 0, read body, handle, frame + write + flush. idle_timeout race
   identical (same Options shape, max_connections 64, per-conn Scratch). The TLS
   handshake itself runs under the same race budget (a stalled handshake must not pin a
   connection slot).
4. **Transport enum stays {udp, tcp}**: DoH/DoT call handle with `.tcp` (it only
   drives truncation, which never applies to stream transports). No unused surface.
5. **ALPN, additive to platform**: ServerContext.init gains
   `alpn: ?[*:null]const ?[*:0]const u8` (mbedtls_ssl_conf_alpn_protocols requires a
   NULL-terminated array that OUTLIVES the config — use comptime-constant arrays).
   DoH advertises ["http/1.1"], DoT ["dot"]. A client with no ALPN still connects
   (mbedTLS only enforces when the client sends the extension). Existing callers pass
   null; the milestone-1 spec note about "no ALPN" is superseded.
6. **Cert holder + reload**: new `src/server/cert_store.zig`. `CertStore` owns
   {mutex, current: *Entry, cert_path, key_path, gpa} where Entry = {ctx: ServerContext,
   refs: u32, retired: bool}. `acquire(io) *Entry` (+1 under mutex),
   `release(io, entry)` (−1; free when retired and 0). `reload(io) ReloadError!void`:
   read both PEM files (NUL-terminated dupes, 64 KiB cap each), ServerContext.init, on
   success swap + retire the old (freed when its refs drain), on ANY failure publish
   nothing (PLAN:297). Listeners acquire per connection (before accept) and release
   when the connection ends — so in-flight streams keep their generation alive.
   File read errors and init errors map to a typed error + a human message for the API.
7. **Cert watcher**: one task per enabled endpoint inside cert_store
   (`watch(store, io)` house loop: Cancelable!void, .boot clock, warn-and-continue).
   Poll every 30 s: stat cert AND key (Io.File.stat), compare mtime+size against the
   last LOADED pair; any change → reload(); reload failure → warn (the old cert keeps
   serving) + a `reload_failures` counter; success → info-level "certificate reloaded".
   mtime+size both compared (same-second atomic renames).
8. **`POST /api/certs/reload`** (PLAN:547): handler `src/web/handlers/certs.zig`
   (PLAN:232). auth = .session, rate_limit = .counted. Reloads every enabled endpoint's
   store; response 200 `{"doh": {"enabled": bool, "reloaded": bool, "error": <msg>|null},
   "dot": {...}}` — per-endpoint outcome, 200 even when a reload fails (the outcome IS
   the payload; nothing about the server's own state is exceptional). Disabled endpoint:
   {enabled:false, reloaded:false, error:null}. WebState gains `doh_certs: ?*CertStore`
   and `dot_certs: ?*CertStore` (null-defaulted, W3 pattern).
9. **openapi.yaml + contract test land together** (m8 ruling 2): the path entry with
   the exact schema, and a contract-table entry + response-shape test in
   web_integration_test.zig. Drift guard (b) count moves 55 → 56 — the test derives the
   count from router.routes so only the yaml operation count assertion data changes.
10. **Observability**: each listener keeps tcp_server-shaped stats (connections,
    tls_handshake_failures, idle_timeouts, connection_errors) + cert_store keeps
    {reloads, reload_failures, last_reload_unix}. metrics.zig additively gains
    `nxdns_doh_server_*`, `nxdns_dot_server_*`, `nxdns_cert_reloads_total`,
    `nxdns_cert_reload_failures_total` families (counterGroup walks the structs —
    follow its existing pattern). /api/health is unchanged (no new rollup this
    milestone; the reload endpoint reports cert state on demand).
11. **app.zig wiring**: when `cfg.doh_server.enabled`, build its CertStore (initial
    load happens in serve BEFORE the group — a bad cert at boot is exit 2 with the
    validate-style message, matching `nxdns check`'s contract that enabled endpoints
    have readable certs; the WATCHER handles later breakage gracefully), then the
    listener; same for dot. Watcher tasks + listener tasks join the group. WebState
    seams wired when web is also enabled. SIGTERM path unchanged.
12. **Integration tests** (both gated `-Dintegration`, loopback, fixtures):
    DoT: std.crypto.tls.Client (no_verification) → framed A query → framed reply
    (reuse resolver_integration_test's fake-upstream pattern or a local-records-only
    handler). DoH POST + GET: raw HTTP/1.1 text over std.crypto.tls.Client (no
    std.http.Client CA ceremony), assert status/content-type/bytes; 415/405/404/400
    matrix; keep-alive two-requests test. Reload: swap fixture cert (write a second
    self-signed pair fixture), reload, NEW connection sees the new cert
    (std.crypto.tls.Client exposes the peer cert? if not: assert old connections
    still serve and new handshakes succeed — the observable contract), old connection
    still answers. Watcher: unit-test the decision function (stat-pair compare) pure;
    do not test 30 s timing.
13. **New fixture**: `tests/fixtures/self_signed_cert2.pem` + key2 (openssl one-liner,
    same profile as the existing pair; README updated) for reload tests.
14. **Docs**: NOT this milestone beyond openapi.yaml (docs/api rendering is Phase 10 —
    m8 ruling 3 unchanged).
15. **Zig freeze elsewhere**: dns/, filter/, cache/, storage/, existing server files
    (udp/tcp/handler) untouched except: app.zig (T6), metrics.zig + routes.zig +
    openapi.yaml + web_integration_test.zig + web/server.zig(WebState fields only)
    (T5). handler.zig is NOT touched (ruling 4).

## Sessions

T1+T2 parallel first; then T3+T4+T5 parallel; then T6.

---

## Session T1: platform ALPN

Owns `src/platform/tls_server.zig` (+ its tests). Add the alpn parameter per ruling 5
(extern mbedtls_ssl_conf_alpn_protocols, comptime NULL-terminated arrays, doc the
lifetime rule), update existing callers/tests with null, add one test that a client
negotiating "http/1.1" succeeds against a context advertising it (the loopback
integration test grows the assertion — std.crypto.tls.Client: check whether it can
send ALPN in 0.16; if it cannot, the unit test just proves config acceptance and the
lifetime doc stands).

Acceptance: fmt/ast clean; plain + integration suites 0 failed.

### T1 As built

`ServerContext.init(gpa, cert_pem, key_pem, alpn: ?[*:null]const ?[*:0]const u8)`.
Extern `mbedtls_ssl_conf_alpn_protocols` verified against vendored 3.6.7 ssl.h:4314
(pointer recorded, not copied — comptime-constant NULL-terminated arrays required,
documented on init). MBEDTLS_SSL_ALPN is on in the stock config. std.crypto.tls.Client
CANNOT send ALPN in 0.16 (no Options field) — negotiation untestable client-side; the
unit test proves config acceptance, the integration echo proves no-ALPN clients still
handshake against an advertising context. Servers can assert server-side via
mbedtls_ssl_get_alpn_protocol if ever needed (extern not declared — unused surface).
Mechanical ripple: tls_client_integration_test.zig:193 caller gained `, null`.

## Session T2: cert_store

Owns `src/server/cert_store.zig` (new). Ruling 6 (holder + refcount + reload) and
ruling 7 (watch loop + pure stat-compare decision fn) + ruling 10's counters +
`snapshotStats()`. Unit tests: acquire/release refcounting across a swap (old entry
freed only after last release), reload failure publishes nothing (bad path, bad PEM),
NUL-termination and size cap, stat-compare decision table. Uses fixtures via
test_fixtures import.

Acceptance: fmt/ast clean; temp-root green (needs -lc + vendored mbedtls objects).

### T2 As built

16 in-file tests; temp-root 22 passed / 0 failed. Surface:
`poll_interval_s=30`, `max_pem_bytes=64KiB`, `ReloadError` (10 cases) +
`humanMessage(err) []const u8`; `FileSig{mtime_ns: i96, size}`, `Signature{cert,key}`,
pure `changed(loaded, observed) bool`; `Entry{ctx, refs, retired}`;
`CertStore.init(gpa, io, cert_path, key_path, alpn)` (boot load; typed error for T6's
exit-2), `deinit(io)` (asserts current refs==0 — join listeners first),
`acquire(io) *Entry` / `release(io, entry)` (release safe from any task; last releaser
frees a retired entry), `reload(io) ReloadError!void`, `watch(io) Cancelable!void`
(sleeps BEFORE first poll, .boot clock), `pollOnce(io)` (testable watcher pass),
`snapshotStats()` (no io; {reloads, reload_failures, last_reload_unix}).
Notes: use `&entry.ctx` for ServerStream.accept; paths are BORROWED (config outlives
the store); `reloads` counts successes only; stat failures warn but are not
reload_failures; `last_reload_unix` set at init too.

## Session T3: dot_server (after T1+T2)

Owns `src/server/dot_server.zig` (new) + its integration test file if separate
(prefer tests inside the file, house pattern). Ruling 3: tcp_server loop shape over
ServerStream; acquire cert entry per connection, release on exit; handshake under the
race budget; stats per ruling 10; Stop enum; max_connections 64; ALPN ["dot"].
Integration tests per ruling 12 (DoT half).

Acceptance: fmt/ast clean; both suites 0 failed.

### T3 As built

`DotServer.listen(gpa, io, listen_address, h: *handler.Handler, certs: *cert_store.CertStore, options)` /
`serve(io)` / `boundAddress()` / `deinit(io)`. Options `{max_connections: u16 = 64, idle_timeout = 10s}`;
the handshake runs under the same idle budget. Deviation from tcp_server: `deinit` takes no
gpa — the allocator is stored at `listen` because `ServerStream.accept` heap-allocates one
mbedTLS ssl context per connection (documented in the file header). Stats atomics:
the four ruling-10 fields plus tcp-pattern internals `rejected_at_capacity,
rejected_at_shutdown, accept_errors`; `snapshotStats()` returns exactly the ruling-10 four.
Loop: accept → claim slot → `certs.acquire` (released on every exit) → handshake raced
against the idle budget → 2-byte-prefix loop over ServerStream reader/writer, flush per
reply, handler called with `.tcp`, `tls.close(gpa)` (close_notify) on every post-handshake
exit. A `handshook` flag ensures the ssl context closes exactly once when Select reports
expiry/cancel after a successful handshake; a timed-out handshake counts as
`tls_handshake_failures`. ALPN is set only via the store (T6). 7 unit + 4 integration
tests in-file. Teardown order: `server.deinit(io)` before `CertStore.deinit`.

## Session T4: doh_server (after T1+T2)

Owns `src/server/doh_server.zig` (new). Ruling 2 in full: std.http.Server over
ServerStream reader/writer (recv buffer 8 KiB = head cap, send 4 KiB), connection
group + 64 cap + 503-free refusal is NOT applicable here (no HTTP yet at accept
overflow: over cap → close raw, cheapest honest behavior — document); per-connection
Scratch + response_buf (65535); GET base64url decode (url_safe_no_pad, reject padding/
whitespace), POST body cap via Io.Limit; keep-alive; bodyless-POST normalization;
ALPN ["http/1.1"]; cert acquire/release per connection; stats. Integration tests per
ruling 12 (DoH half: POST, GET, 415/405/404/400, keep-alive).

Acceptance: fmt/ast clean; both suites 0 failed.

### T4 As built

`DohServer.listen(gpa, io, address, *Handler, *CertStore, Options)` / `serve(io)` /
`deinit(gpa, io)` / `boundAddress()` / `snapshotStats()`. Options
`{max_connections = 64, idle_timeout = 10s}` — the budget races the TLS handshake only;
requests have no timeout (web-listener precedent). Module-level
`serve(gpa, io, model.TlsEndpoint, *Handler, *CertStore)` is the bind-warn-and-continue
entry for T6. Exposed comptime `alpn_protocols` (["http/1.1"]) and `dns_query_path`.
Stats: the ruling-10 four + `bad_requests` + tcp-shaped accept counters
(`rejected_at_capacity, rejected_at_shutdown, accept_errors`) — over-capacity refusals
must stay visible. Deviations: POST cap via content-length precheck + fixed buffer +
one-byte probe for chunked bodies instead of Io.Limit (no allocation on the hot path);
`respond` can raise `HttpExpectationFailed` (100-continue), mapped to `connection_errors`;
no 413 integration test (ruling 12's matrix is 415/405/404/400). GET `?dns=` parsed by a
local validating decoder (duplicate param → 400, `%` and padding/whitespace rejected).
Oversize POST answers 413 with `keep_alive=false` (body never drained). Over the 64 cap
the raw TCP stream closes (no HTTP before a handshake). 8 unit + 4 integration tests.
T6 recipe: `CertStore.init(gpa, io, cert_path, key_path, doh_server.alpn_protocols)`;
run `doh_server.serve` and `store.watch` as group tasks; `DohServer.deinit` before
`CertStore.deinit`.

## Session T5: web endpoint + metrics (after T2, parallel with T3/T4)

Owns `src/web/handlers/certs.zig` (new), `src/web/routes.zig` (one entry),
`src/web/openapi.yaml` (one path), `src/web/server.zig` (two null-defaulted WebState
fields ONLY), `src/web/metrics.zig` (ruling 10 families), and the additive
web_integration_test.zig edits (contract entry + shape test + guard-b count).
Handler per ruling 8 (apply/respond split, house pattern). Unit tests for the
apply fn (both stores null; one store failing reload → error message in payload,
still 200).

Acceptance: fmt/ast clean; plain suite 0 failed; drift guards green.

### T5 As built

handlers/certs.zig: apply/respond split — `applyReload(state, io) View` with
`View{doh, dot}` and `Outcome{enabled, reloaded, @"error": ?[]const u8}` (serialized as
`error`); `post` always answers 200. Route: `POST /api/certs/reload`, `.session`,
`.counted`; table 55→56; contract entry + strict shape test + drift guards green at 56.
metrics.zig: the two ruling-10 cert counters share one family pair
(`nxdns_cert_reloads_total` / `nxdns_cert_reload_failures_total`) with an
`endpoint="doh"|"dot"` label (upstream-label pattern); unwired stores omit the family;
`last_reload_unix` not exported. WebState gains `doh_certs`/`dot_certs:
?*cert_store.CertStore = null`. Deviation (accepted): deleted the m8 tripwire test in
`src/web/openapi.zig` ("the document does not promise what phase 9 owns") — it exists to
fire when this endpoint lands. Deferred to T6: `nxdns_doh_server_*` / `nxdns_dot_server_*`
listener families (T3/T4 types did not exist at T5 build time).

## Session T6: app wiring + fixtures + smoke (after all)

Owns `src/app.zig`, `tests/fixtures/` (cert2/key2 + README + fixtures.zig), and
`src/tests.zig` is the ORCHESTRATOR's (report lines). Ruling 11 wiring; ruling 13
fixture. Smoke (report transcript): boot with doh+dot enabled on unprivileged ports
with fixture certs; DoT query via a scripted TLS client (a tiny zig run or the
integration test binary is fine — no new tooling); DoH POST via curl --insecure
--http1.1 with a binary body; certs/reload via the API twice (success, then chmod the
key unreadable → error in payload, old cert still serving a new connection); SIGTERM 0.
Both suites + cross ReleaseSafe with the SPA dist.

Acceptance: full gates green; smoke transcript.

### T6 As built

app.zig realizes ruling 11 in-frame: `openCertStore` maps non-OOM `ReloadError` to
`error.BadCertificate` (exit 2, prints both paths + `cert_store.humanMessage`);
`bindDoh`/`bindDot` warn-and-continue per ruling 1; listeners + two `store.watch` loops
join the group; stores are declared before listeners so teardown runs listener-deinit →
store-deinit and the refs==0 assert holds. `dot_alpn` (["dot"]) lives in app.zig.
Deviation: `doh_server.serve` module entry is NOT used — it constructs the DohServer in
its own task frame, so WebState could never get the pointer the `nxdns_doh_server_*`
family needs; both listeners bind in app.zig's frame instead (tcp_server style).
`doh_server.serve` remains as unused pub API. **Superseded by milestone-18 ruling 1: the
unused `doh_server.serve` is deleted. Nothing ever called it, and a dead second
composition path over the listener core is exactly the divergence that milestone
collapsed. app.zig's in-frame bind is now the only way DoH comes up.** metrics.zig: `DohListenerSample`
(ruling-10 four + `bad_requests`); DoT renders `dot_server.StatsSnapshot` directly;
accept-side counters stay off the exposition; unwired listeners omit the families.
WebState gains `doh_listener`/`dot_listener` optional pointers. build.zig (out of
ownership, necessary): the exe now compiles `mbedtls_shim.c` with the threading macros —
first exe-side tls_server reference; mirrors the tests wiring. Fixtures: cert2/key2
exported as `cert2_pem`/`key2_pem` (distinct fingerprint), currently unreferenced by
tests. Config: `model.TlsEndpoint` already existed; no model/validate/schema edits.
Smoke: DoT framed query answered from local records; DoH POST 200
application/dns-message; reload success then key chmod 000 → `reloaded:false` with
"private key file is not readable" while the old cert keeps serving; SIGTERM 0; boot
with unreadable key exits 2.

---

## Review (Codex, as built)

Four rounds on one thread; round 4 returned "No findings."

Round 1 (3 important, 2 minor): doh_server leaked the TLS context when the handshake
won a race the Select reported as timeout/cancel → handshook flag (dot_server pattern).
cert_store reloads were not serialized across read→build→publish, so an older read could
publish last → `reload_mutex` held across the whole sequence (lock order: reload_mutex
before the entry mutex; accept path untouched) + a hook-pinned regression test
(`after_load_hook` seam). doh bodied-GET smuggling: std leaves unframed body bytes for
bodyless methods to be parsed as the next request → `framesUnreadableBody` guard, 400 +
close, no byte read. `.drop` now closes per ruling 2. cert2/key2 were unreferenced →
DoT reload-to-distinct-identity integration test (old connection keeps serving, new
handshake succeeds, distinct entry pointers; the 0.16 tls Client does not retain the
peer chain, so identity is proven via entry pointers).

Round 2 (2 minor): the smuggling guard ran before routing and stole 404/405 statuses →
routing decides the status, framed requests only force close. Successful POSTs ignored
the client's `Connection: close` → answer/refuse honor `head.keep_alive` in the Next
enum (the wire already did via Server.zig:628).

Round 3 (1 important, introduced by the round-2 reshape): bodied refusals (404/405/415)
kept keep-alive, so respond drained the whole body first — an unterminated chunked body
could pin all 64 slots with no response → `framesBody` helper; any pre-body-read refusal
closes when a body is framed; drain-free integration tests (missing terminal chunk →
404+EOF at once; undelivered content-length 4096 → 415+EOF); errorMatrix's 415 became a
zero-length wrong-type POST so its keep-alive reuse stays legitimate.

Final counts: doh_server.zig 19 tests, dot_server.zig 12, cert_store.zig 17. Final
gates: plain 1166/1279 passed, 113 skipped (integration-gated), 0 failed; integration
1275/1279, 4 skipped (live-network by design), 0 failed.

## Module layout (new)

src/server/{doh_server,dot_server,cert_store}.zig, src/web/handlers/certs.zig,
tests/fixtures/{self_signed_cert2.pem,self_signed_key2.pem}.

## File ownership

T1 platform/tls_server.zig; T2 server/cert_store.zig; T3 server/dot_server.zig;
T4 server/doh_server.zig; T5 web/{handlers/certs.zig,routes.zig,openapi.yaml,
server.zig(fields),metrics.zig,web_integration_test.zig}; T6 app.zig + fixtures.
Orchestrator: src/tests.zig, spec. Parallel sessions never share a file.

## Acceptance (milestone complete)

- [ ] Both suites 0 failed (plain + -Dintegration), zero err lines; cross ReleaseSafe
      with SPA dist green.
- [ ] DoT and DoH loopback integration tests green (query → reply, error matrix,
      keep-alive).
- [ ] certs/reload endpoint + openapi entry + contract test land together; drift
      guards green at 56.
- [ ] Cert watcher reloads on file change; failed reload keeps serving the old cert.
- [ ] T6 smoke transcript incl. reload-failure path and SIGTERM 0.
- [ ] Spec As-built synced per session.

## Anti-requirements

No HTTP/2, no DoQ, no SNI/multi-cert, no client-cert auth, no session tickets, no
proxy-protocol/XFF handling, no cert generation (ACME etc.), no cache-control on DoH,
no handler.zig changes, no docs/api rendering (Phase 10), no config schema changes
(TlsEndpoint exists), no inotify.