milestone 11: systemd and docker packaging, operator and architecture docs, config and api reference, docs drift guards

specs/milestone-11.md

@@ -0,0 +1,273 @@
+# Milestone 11: packaging, ops and docs (PLAN Phase 10)
+
+Goal: systemd unit, Dockerfile + compose, and the four docs (operator, architecture,
+config-reference, API) — the documented deployment must work end-to-end; docs are
+drift-guarded where a guard is cheap and honest.
+
+## Rulings (binding)
+
+1. **Layout.** `deploy/systemd/nxdns.service` + `deploy/systemd/sysusers.conf`;
+   `deploy/docker/{Dockerfile,compose.yaml,.dockerignore}`; `docs/{operator.md,
+   architecture.md,config-reference.md,api.md}`; `README.md` at the root (the repo has
+   none; a portfolio repo needs a front door — short: what, why, quickstart, doc links).
+   PLAN.md:236 sketches docs/ subdirectories; single files need no subdirectories.
+
+2. **API docs = hand-written `docs/api.md` + drift test.** No renderer is vendored
+   (redoc/scalar are exactly the dependency liability AGENTS.md refuses), and a
+   build-time YAML parser for rendering is scope the yaml does not justify — the yaml
+   itself is already served unauthenticated at `GET /api/openapi.yaml` (routes.zig:49)
+   and is the exhaustive contract. `docs/api.md` gives human-readable orientation:
+   auth model (cookie session, login flow), rate limiting, error envelope, SSE
+   semantics, then one line per operation (method, path, auth, one-sentence purpose)
+   and a pointer to the yaml for schemas. A drift test asserts every served route
+   appears textually in api.md (mirror of openapi.zig:34's guard). This satisfies
+   m8 ruling 3's deferred "docs/api rendering" as the engineering call: rendered =
+   readable, guarded, in-repo; not = a vendored JS bundle.
+
+3. **Docs drift guards.** New `src/docs_drift_test.zig` (ORCHESTRATOR-owned, written
+   after the doc sessions land): (a) every route in `router.routes` appears in
+   docs/api.md; (b) every settings key from `model.toSettings` (the 44 keys) appears
+   in docs/config-reference.md; (c) every CLI subcommand name appears in
+   docs/operator.md. Docs embedded via a `docs_files` anonymous import added in
+   build.zig (test_fixtures pattern, build.zig:61). Guards are textual-containment
+   only — cheap, zero false authority.
+
+4. **systemd unit.** `Type=simple` (no forking, shutdown.zig:35 handles SIGTERM),
+   `ExecStart=/usr/local/bin/nxdns run`, stderr → journald (logging.zig:301 already
+   states this; `logging.output=stderr` stays the default). Static system user `nxdns`
+   via `deploy/systemd/sysusers.conf` (`u nxdns - "nxdns DNS sinkhole"`), NOT
+   DynamicUser — the TLS key must be chown-able to a stable uid ("TLS keys readable by
+   service user only", PLAN §19). `StateDirectory=nxdns` (0700 matches cli.zig:226),
+   `LogsDirectory=nxdns` (covers logging.output=file; the binary does not create the
+   directory, logging.zig:502), `ConfigurationDirectory=nxdns`.
+   `AmbientCapabilities=CAP_NET_BIND_SERVICE` + `CapabilityBoundingSet=` the same
+   (port 53; 443/853 covered by the same cap). Hardening: `NoNewPrivileges=yes`,
+   `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `PrivateDevices=yes`,
+   `ProtectKernelTunables/Modules/Logs=yes`, `ProtectControlGroups=yes`,
+   `ProtectClock=yes`, `ProtectHostname=yes`, `ProtectProc=invisible`,
+   `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `RestrictNamespaces=yes`,
+   `RestrictRealtime=yes`, `RestrictSUIDSGID=yes`, `LockPersonality=yes`,
+   `MemoryDenyWriteExecute=yes` (static Zig binary, no JIT), `UMask=0077`,
+   `SystemCallFilter=@system-service`, `SystemCallArchitectures=native`,
+   `Restart=on-failure`, `RestartSec=2`. No `ReadWritePaths` beyond what
+   StateDirectory/LogsDirectory grant. Validate with `systemd-analyze verify` if the
+   build host has it; report honestly if not.
+
+5. **Docker.** Multi-stage: builder stage only stages `ca-certificates` (upstream
+   TLS verification rescans the system CA bundle, tls_client.zig:216 — a scratch
+   image without a bundle breaks every DoH/DoT upstream); final `FROM scratch` with
+   the static musl binary, `/etc/ssl/certs/ca-certificates.crt`, a nonroot numeric
+   `USER 65532:65532`, `VOLUME /var/lib/nxdns`, `EXPOSE 53/udp 53/tcp 8080 443 853`,
+   `ENTRYPOINT ["/nxdns"]`, `CMD ["run"]`. The binary is NOT built inside the
+   Dockerfile (the repo builds it with zig; the Dockerfile COPYes
+   `zig-out/cross/$TARGETARCH-…/nxdns` via a build arg or buildx TARGETARCH mapping —
+   keep it working for both arches). compose.yaml: ports 53:53/udp+tcp and 8080:8080
+   (443/853 commented), bind-mount `./etc-nxdns:/etc/nxdns:ro`, named volume for
+   `/var/lib/nxdns`, `sysctls: net.ipv4.ip_unprivileged_port_start=0` so the nonroot
+   user binds 53 (per-netns sysctl; documented), `restart: unless-stopped`. First
+   boot needs a seeded `/etc/nxdns/config.zon` with a `default` group + one enabled
+   upstream or the container exits 2 (bootstrap.zig:38, app.zig:93) — operator.md and
+   a compose comment both say so. Do NOT point the host's resolv.conf at nxdns
+   itself for the container's own lookups.
+
+6. **CI.** One added job `docker` in ci.yml: after building the x86_64 exe
+   (ReleaseSafe, with the SPA dist like the cross job), `docker build` the image and
+   run a container smoke (seed a minimal config.zon; `nxdns version` + boot + one
+   `dig`-equivalent via the test client or `curl` on 8080/api/health; SIGTERM 0).
+   No registry push — no publish step exists anywhere in this repo or the infra
+   repo's CI, and registry credentials are an infra decision outside this repo.
+   Manual publishing to git.mial.net stays possible and is documented in operator.md
+   in one paragraph.
+
+7. **Docs content contracts.**
+   - operator.md: install (systemd path and docker path, both complete), first boot +
+     config.zon seeding semantics (file seeds DB once, DB is truth thereafter,
+     bootstrap.zig:38), auth setup (`web.password` hashed on import, never stored,
+     export writes ""), TLS cert/key provisioning + 0600 expectations + the reload
+     API/watcher, backup/restore = `nxdns export`/`import --force` (+ the 0600 export
+     mode and why), upgrades (schema migration = install + restart, PLAN §20.11),
+     data-dir layout table, exit codes (0/1/2/64, cli.zig:35), CLI reference (all six
+     subcommands + flags), troubleshooting (exit 2 causes, `nxdns check` semantics
+     incl. source-selection order cli.zig:511, disk-full degradation, port 53
+     conflicts with systemd-resolved — include the disable recipe).
+   - architecture.md: module map (the src/ inventory), the purity rule (dns/, filter/,
+     local/, cache/ take bytes, no Io — AGENTS.md), std.Io injection + Threaded
+     backend, data flow for one query (listener → handler → filter/cache/local →
+     upstream → sink/logger), storage split (config.db truth / querylog.db expendable),
+     web stack (std.http over TLS optional, SPA embedded via web_assets, SSE), cert
+     hot-reload design (refcounted CertStore), failure-visibility doctrine (counters +
+     /metrics over log spam). Concise — a map, not a novel.
+   - config-reference.md: complete — every section/field with type, default, unit,
+     validation range, and which subsystem consumes it; collections with required
+     fields; DB-vs-file truth explanation; the `web.password`/`password_hash`
+     exclusivity; `logging.level=.err` serializes as "error" (model.zig:154). The
+     explorer inventory in this milestone's research is the skeleton; verify against
+     model.zig/validate.zig while writing, do not trust the summary blindly.
+   - api.md: per ruling 2.
+   - README.md: ≤120 lines; what nxdns is, feature list (honest, shipping features
+     only), quickstart (docker compose path), build-from-source (zig build, node for
+     the SPA), doc links, license note if a LICENSE exists (do not invent one).
+
+8. **No new runtime code.** This milestone adds zero behavior to the binary. The only
+   src/ change is the orchestrator's docs_drift_test.zig + its build wiring. If a doc
+   session finds a bug while documenting, it REPORTS it (no fix); the orchestrator
+   triages.
+
+9. **Pi 5 end-to-end**: the exit criterion runs on hardware this environment does not
+   have. The deliverable here is: both suites green, docker smoke green on x86_64,
+   `systemd-analyze verify` clean (or honestly reported unavailable), aarch64 binary
+   built and statically verified (existing cross job). The operator doc's Pi 5 recipe
+   is written to be executed by the user; the spec records this boundary explicitly.
+
+## Sessions
+
+U1, U3, U4, U5 parallel; U2 after U1 (documents the artifacts U1 produces);
+orchestrator wiring (ruling 3) after U3+U5.
+
+## Session U1: deploy artifacts + CI
+
+Owns `deploy/systemd/nxdns.service`, `deploy/systemd/sysusers.conf`,
+`deploy/docker/{Dockerfile,compose.yaml,.dockerignore}`, `.gitea/workflows/ci.yml`
+(one added job). Rulings 4, 5, 6. Verify: `systemd-analyze verify` (or report
+unavailable), local `docker build` + container smoke if the docker daemon is
+reachable (report honestly either way), `zig build test` untouched-green.
+
+## Session U2: operator.md + README.md (after U1)
+
+Owns `docs/operator.md`, `README.md`. Rulings 7 (operator + README). Reads U1's
+artifacts and the runtime/CLI facts from the code (verify against src/cli.zig,
+src/app.zig, src/config/bootstrap.zig — not from memory).
+
+## Session U3: config-reference.md
+
+Owns `docs/config-reference.md`. Ruling 7. Source of truth: src/config/model.zig +
+validate.zig + import/export/bootstrap. Every field, no sampling.
+
+## Session U4: architecture.md
+
+Owns `docs/architecture.md`. Ruling 7. Reads module headers; no deep dives needed
+beyond what the doc claims.
+
+## Session U5: api.md
+
+Owns `docs/api.md`. Rulings 2, 7. Source of truth: src/web/routes.zig (the served
+table: method, path, auth, limiter) + openapi.yaml summaries + auth.zig/sse.zig for
+the auth and SSE prose. Every route, no sampling.
+
+## As built
+
+**U1** delivered per rulings 4-6 with accepted deviations: `StateDirectoryMode=0700`
+(systemd defaults 0755; the binary cannot tighten a pre-existing directory) and
+`User=nxdns`/`Group=nxdns` added to the unit; the Dockerfile's builder stage also maps
+buildx TARGETARCH → cross-target dir and pre-chowns `/var/lib/nxdns` to 65532 (a named
+volume seeded from a root-owned image dir would be unwritable on first boot); compose
+gained a `build:` block; `deploy/docker/.dockerignore` is documentation-grade under
+BuildKit (only a root `.dockerignore` or `Dockerfile.dockerignore` is honored — the
+file's header says so). Verified: `systemd-analyze verify` clean modulo the off-host
+ExecStart path (an ExecStart=/bin/true copy verifies exit 0); full local docker build +
+smoke passed (binds 53 as uid 65532, /api/health ok, SIGTERM exit 0). The CI docker
+job probes both 127.0.0.1 and the container IP to survive either runner topology.
+Compose expects the operator-created seed at `deploy/docker/etc-nxdns/config.zon`
+(minimal: a `default` group + one enabled upstream), else exit 2.
+
+**U2** delivered docs/operator.md (397 lines; systemd + docker + Pi 5 recipes, seeding
+semantics, auth, TLS, backup/restore, data-dir table, full CLI reference, exit codes,
+troubleshooting incl. the systemd-resolved DNSStubListener recipe) and README.md
+(72 lines, no license section — no LICENSE exists). All facts source-verified.
+
+**U3** delivered docs/config-reference.md (12 scalar sections, 9 collections, DB-vs-file
+truth model, auth section, minimal + annotated examples) and surfaced five code
+discrepancies during writing (see fix wave below).
+
+**U4** delivered docs/architecture.md (module map from the //! headers, purity rule
+with the honest exceptions, life-of-one-query pipeline verified against handler.zig,
+storage split, web stack, CertStore design, failure-visibility doctrine). Reported one
+stale comment (logging.zig:17 cited a moved cli.zig line) — orchestrator fixed the
+comment to cite start.zig:724 via std.process.Init.
+
+**U5** delivered docs/api.md: all 56 operations (matches router.routes.len), auth /
+rate-limit / SSE prose, error envelope, openapi.yaml pointer. No code-vs-yaml
+discrepancies found.
+
+**Orchestrator wiring (ruling 3)**: docs/docs.zig (embeds api.md, config-reference.md,
+operator.md), `docs_files` anonymous import on the test module in build.zig,
+src/docs_drift_test.zig with three containment guards (routes → api.md; toSettings
+keys → config-reference.md; the six subcommand names → operator.md), tests.zig import.
+
+**Fix wave (orchestrator-triaged; ruling 8's no-runtime-code rule lifted for exactly
+these)** — U3's five discrepancies, triaged with stdlib evidence:
+1. `runtime.io_backend` DELETED end to end (model, settings handler + view, openapi,
+   web types/SettingsPage/settingsDiff + tests, docs). Nothing consumed it — main uses
+   init.io (stdlib Threaded, start.zig:724), and 0.16's std.Io.Evented has stubbed
+   networking (Uring.zig netConnectIp → error.NetworkDown), so PLAN decision E's
+   "io_uring via flag" is not deliverable at this tag. Re-add when std ships working
+   evented net. Old DB rows warn-and-ignore via fromSettings' unknown-key path.
+2. `upstream.connect_timeout_ms` DELETED. No call site; the pool races the whole
+   attempt against total_timeout (app.zig sets it from totalTimeout); Threaded panics
+   on IpAddress.ConnectOptions.timeout != .none; std.http.Client has no knob. The
+   validate cross-check is now total >= read only.
+3. `cache.size` KEPT: 0 is clean documented disabled behavior (put short-circuits;
+   in-file test "a cache of zero entries stores nothing"). Doc row corrected.
+4. `dns.bind_ipv6` TIGHTENED: checkBind generalized to a BindFamily enum; dns.bind_ipv6
+   requires an IPv6 literal (an IPv4 wildcard there made the v4 bind AddressInUse get
+   swallowed with a false "dual-stack" log — silent IPv6 loss). web/TLS binds stay .any.
+5. doh/dot `readTimeout(.{})` sites KEPT: they are test fixtures; the real idle budget
+   is the ruled 10s Options default. The doc claim was wrong and was removed.
+Settings key counts after deletion: toSettings emits 43 (incl. web.password_hash);
+the API-visible restart-required set is 42 (was 44).
+
+## Review (Codex, as built)
+
+Three rounds on one thread; round 3 returned "No findings."
+
+Round 1 (5 important): stale-DB bind_ipv6 rows bypassed the new validate check at boot
+→ app.zig parseBind now enforces the IP family on both dns binds (BadBindAddress,
+exit 2, remedy in the message; the v4 side had the symmetric hole); app.zig's tests
+were not collected by tests.zig at all — the import line was added and the new test
+runs. operator.md's TLS recipe assumed the nxdns host user → Docker path now chowns
+65532:65532 numerically host-side (read-only bind mount). PLAN.md still promised the
+io_uring flag and connect_timeout_ms → synced (Io bullet records the drop with stdlib
+evidence; decision E row; example config). Drift guards were maskable → api.md guard
+anchors the full "| METHOD | `pattern` |" row per operation; operator.md guard anchors
+the "### `name" reference headings.
+
+Round 2 (1 important, 1 minor): seed-permission guidance covered only web.password →
+now web.password or web.password_hash (a restored export); PLAN.md's "stubs all
+networking" overstated 0.16's Uring — now names the stubbed operations precisely.
+
+Final gates: plain 1171/1284 passed, 113 skipped (integration-gated), 0 failed;
+integration 1280/1284, 4 skipped (live-network by design), 0 failed; cross ReleaseSafe
+with the SPA dist 18/18; web suite 121/121 with format/lint/typecheck clean.
+
+## Module layout (new)
+
+deploy/systemd/{nxdns.service,sysusers.conf}, deploy/docker/{Dockerfile,compose.yaml,
+.dockerignore}, docs/{operator,architecture,config-reference,api}.md, README.md,
+src/docs_drift_test.zig (orchestrator).
+
+## File ownership
+
+U1 deploy/* + ci.yml; U2 docs/operator.md + README.md; U3 docs/config-reference.md;
+U4 docs/architecture.md; U5 docs/api.md; orchestrator src/docs_drift_test.zig,
+build.zig (docs_files module), src/tests.zig.
+
+## Acceptance (milestone complete)
+
+- [ ] All files in the module layout exist with the ruling-7 content contracts met.
+- [ ] Docs drift tests pass: route coverage in api.md, settings-key coverage in
+      config-reference.md, subcommand coverage in operator.md.
+- [ ] CI docker job builds the image and the container smoke passes (in CI; locally
+      if the daemon is available).
+- [ ] `systemd-analyze verify deploy/systemd/nxdns.service` clean, or its
+      unavailability recorded in the U1 report.
+- [ ] Both suites 0 failed; cross ReleaseSafe with the SPA dist green.
+- [ ] No runtime-behavior changes (git diff over src/ shows only docs_drift_test.zig
+      + wiring).
+
+## Anti-requirements
+
+- No vendored API-doc renderer (redoc/scalar/swagger-ui), no YAML parser.
+- No registry publish step; no k3s manifests (the infra repo owns deployment there).
+- No SIGHUP/reload feature, no env-var config, no new CLI flags — document what
+  exists; report gaps instead of filling them.
+- No LICENSE invention; no badges or marketing prose in README.