From 5b3d1cd65c16919dbf84c6ee2b856b1dbdb034e4 Mon Sep 17 00:00:00 2001 From: m5r Date: Sat, 15 Aug 2026 16:27:36 +0200 Subject: [PATCH] docs: unwrap hand-wrapped prose repo-wide --- AGENTS.md | 73 +- INSTALL.md | 46 +- PLAN.md | 84 +- README.md | 110 +-- docs/README.md | 68 +- docs/explanation/architecture.md | 182 +---- docs/explanation/configuration-model.md | 204 +---- docs/explanation/performance-and-testing.md | 190 +---- docs/how-to/back-up-and-restore.md | 116 +-- docs/how-to/enable-doh-and-dot.md | 96 +-- docs/how-to/install-with-docker.md | 130 +-- docs/how-to/install-with-systemd.md | 218 ++--- docs/how-to/measure-performance.md | 87 +- docs/how-to/set-up-admin-authentication.md | 138 +--- docs/how-to/troubleshoot.md | 208 ++--- docs/how-to/upgrade.md | 196 ++--- docs/how-to/verify-a-release.md | 193 ++--- docs/reference/api.md | 210 ++--- docs/reference/cli.md | 232 ++---- docs/reference/configuration.md | 269 ++----- docs/reference/files-and-directories.md | 155 +--- docs/reference/performance.md | 47 +- docs/tutorial/first-run.md | 175 +---- specs/milestone-10.md | 318 +------- specs/milestone-11.md | 236 +----- specs/milestone-12.md | 142 +--- specs/milestone-13.md | 549 +++---------- specs/milestone-14.md | 611 +++------------ specs/milestone-15.md | 289 ++----- specs/milestone-16.md | 363 ++------- specs/milestone-17.md | 407 ++-------- specs/milestone-18.md | 441 ++--------- specs/milestone-19.md | 400 ++-------- specs/milestone-20.md | 801 +++---------------- specs/milestone-21.md | 551 ++----------- specs/milestone-22.md | 74 +- specs/milestone-23.md | 412 ++-------- specs/milestone-24.md | 440 ++--------- specs/milestone-25.md | 540 +++---------- specs/milestone-3.md | 419 +++------- specs/milestone-4.md | 829 +++++--------------- specs/milestone-5.md | 751 ++++-------------- specs/milestone-6.md | 453 ++--------- specs/milestone-7.md | 509 ++---------- specs/milestone-8.md | 801 +++---------------- specs/milestone-9.md | 392 ++------- specs/research/zig-0.16-api-notes.md | 217 ++--- tests/fixtures/README.md | 18 +- 48 files changed, 2691 insertions(+), 11699 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index bdebdcf..ea25562 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -2,79 +2,42 @@ ## Aim -nxdns: a self-hosted DNS sinkhole for a household LAN, written in Zig 0.16.0. -Portfolio-grade public repo. PLAN.md is the source of truth for scope and design; -specs/ holds per-milestone contracts; specs/research/ holds verified stdlib facts. +nxdns: a self-hosted DNS sinkhole for a household LAN, written in Zig 0.16.0. Portfolio-grade public repo. PLAN.md is the source of truth for scope and design; specs/ holds per-milestone contracts; specs/research/ holds verified stdlib facts. ## Values -We intentionally architect this code to be robust, maintainable, pragmatic — -good craftsmanship and good engineering. We explicitly avoid tech debt, code -smells, bad architecture decisions, and brittle implementations. +We intentionally architect this code to be robust, maintainable, pragmatic — good craftsmanship and good engineering. We explicitly avoid tech debt, code smells, bad architecture decisions, and brittle implementations. What that means in practice: -- This is a greenfield project. Breaking changes are allowed. Never keep a bad - interface for compatibility; fix it at the root. -- No versioning of scope. A feature is in scope (build it completely) or out of - scope (do not build it). No "v2 later", no stubs left behind. +- This is a greenfield project. Breaking changes are allowed. Never keep a bad interface for compatibility; fix it at the root. +- No versioning of scope. A feature is in scope (build it completely) or out of scope (do not build it). No "v2 later", no stubs left behind. - Fix root causes, not symptoms. Do not iterate on workarounds. -- Scope is small on purpose: household scale, two targets, few dependencies. - Do not add generality nobody asked for. -- Dependencies are liabilities: stdlib first; vendored + pinned C deps - (sqlite3, mbedTLS) only where the stdlib has nothing. -- Verify stdlib claims against ../zig at tag 0.16.0 — pre-0.16 knowledge is - stale (std.Io migration). See specs/research/zig-0.16-api-notes.md. -- Pure core: dns/, filter/, local/, cache/ take bytes and return bytes — no Io, - no sockets, no clocks hidden inside. -- Every failure mode must be visible: no silent drops, no unbounded logs, no - swallowed errors. Counters + health surfaces over log spam. -- Tests are runnable acceptance criteria, not decoration. Required CI stays - deterministic — no network-dependent tests in blocking jobs. -- Comments state constraints the code cannot show. No narration, no - commented-out code. -- Git: GPG-signed commits (`git commit -S`), simple lowercase messages, no - generated-by footers. +- Scope is small on purpose: household scale, two targets, few dependencies. Do not add generality nobody asked for. +- Dependencies are liabilities: stdlib first; vendored + pinned C deps (sqlite3, mbedTLS) only where the stdlib has nothing. +- Verify stdlib claims against ../zig at tag 0.16.0 — pre-0.16 knowledge is stale (std.Io migration). See specs/research/zig-0.16-api-notes.md. +- Pure core: dns/, filter/, local/, cache/ take bytes and return bytes — no Io, no sockets, no clocks hidden inside. +- Every failure mode must be visible: no silent drops, no unbounded logs, no swallowed errors. Counters + health surfaces over log spam. +- Tests are runnable acceptance criteria, not decoration. Required CI stays deterministic — no network-dependent tests in blocking jobs. +- Comments state constraints the code cannot show. No narration, no commented-out code. +- Git: GPG-signed commits (`git commit -S`), simple lowercase messages, no generated-by footers. ## Reading `zig build test` output -A fully passing `zig build test` still prints a line like `failed command: -.../test --cache-dir=... --seed=... --listen=-`, and still exits 0. That line -is a known upstream zig 0.16.0 labelling defect. It does not mean a test -failed, and no test binary crashed. +A fully passing `zig build test` still prints a line like `failed command: .../test --cache-dir=... --seed=... --listen=-`, and still exits 0. That line is a known upstream zig 0.16.0 labelling defect. It does not mean a test failed, and no test binary crashed. -The build runner sets a step's `result_failed_command` on every spawn -(`std/Build/Step/Run.zig:1540`) and never clears it on success. It then prints -a step's diagnostics whenever the step wrote anything to stderr, explicitly "no -matter the result" (`compiler/build_runner.zig:1381`), and that printer emits -the `failed command: ` label unconditionally when the field is set -(`compiler/build_runner.zig:1515`). Our suite writes to stderr on every run, -because the tests that cover the warning paths log through the real sink. A -minimal reproducer with no mbedTLS and no C — one passing test whose body is a -`std.debug.print` — prints the same label and reports "3/3 steps succeeded; -1/1 tests passed"; deleting the print removes the label. No upstream issue -matched a search, so the reference is the 0.16.0 source lines above. +The build runner sets a step's `result_failed_command` on every spawn (`std/Build/Step/Run.zig:1540`) and never clears it on success. It then prints a step's diagnostics whenever the step wrote anything to stderr, explicitly "no matter the result" (`compiler/build_runner.zig:1381`), and that printer emits the `failed command: ` label unconditionally when the field is set (`compiler/build_runner.zig:1515`). Our suite writes to stderr on every run, because the tests that cover the warning paths log through the real sink. A minimal reproducer with no mbedTLS and no C — one passing test whose body is a `std.debug.print` — prints the same label and reports "3/3 steps succeeded; 1/1 tests passed"; deleting the print removes the label. No upstream issue matched a search, so the reference is the 0.16.0 source lines above. -Any *other* failure text is real. Trust the summary line: `zig build test` -exiting non-zero, a `N failed` count, or a panic backtrace all mean a genuine -failure. Do not filter, wrap, or suppress the runner's output to hide the -label — that would hide real failures with it. +Any *other* failure text is real. Trust the summary line: `zig build test` exiting non-zero, a `N failed` count, or a panic backtrace all mean a genuine failure. Do not filter, wrap, or suppress the runner's output to hide the label — that would hide real failures with it. -One trap: running a cached test binary by hand with `--listen=-` aborts with -`internal test runner failure: EndOfStream`. That is not a teardown bug; the -IPC runner is talking to a closed stdin because no build runner is on the other -end. Run the binary with no arguments to get the plain stdio report. +One trap: running a cached test binary by hand with `--listen=-` aborts with `internal test runner failure: EndOfStream`. That is not a teardown bug; the IPC runner is talking to a closed stdin because no build runner is on the other end. Run the binary with no arguments to get the plain stdio report. ## Regenerating the contract samples -`web/src/lib/contractSamples.gen.ts` is a committed golden of canonicalized -API responses, byte-compared against the live server by a `-Dintegration` -test and type-checked by `tsc`. After a deliberate API contract change, -regenerate it with: +`web/src/lib/contractSamples.gen.ts` is a committed golden of canonicalized API responses, byte-compared against the live server by a `-Dintegration` test and type-checked by `tsc`. After a deliberate API contract change, regenerate it with: ``` zig build test -Dintegration -Dcontract-samples-out="$PWD/web/src/lib/contractSamples.gen.ts" ``` -then update `web/src/lib/types.ts` to match and commit both. Never edit the -generated file by hand. +then update `web/src/lib/types.ts` to match and commit both. Never edit the generated file by hand. diff --git a/INSTALL.md b/INSTALL.md index 1404106..b0cbd87 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -11,12 +11,9 @@ This directory is an nxdns release for one architecture. It holds: | `THIRD-PARTY-NOTICES` | Licences of everything compiled or bundled in | | `INSTALL.md` | This file | -The binary is statically linked against musl and needs nothing installed on the -target host. +The binary is statically linked against musl and needs nothing installed on the target host. -Verify the download before you trust it. `docs/how-to/verify-a-release.md` in -the repository covers where the public key comes from, what fingerprint to -expect, and what the signature does and does not prove. +Verify the download before you trust it. `docs/how-to/verify-a-release.md` in the repository covers where the public key comes from, what fingerprint to expect, and what the signature does and does not prove. ## 1. Install the binary, the user and the unit @@ -34,12 +31,9 @@ systemctl daemon-reload mkdir -p -m 0755 /etc/nxdns ``` -`nxdns.conf` ships under the name it is installed as, so there is no rename to -get wrong. +`nxdns.conf` ships under the name it is installed as, so there is no rename to get wrong. -Do not create `/var/lib/nxdns` or `/var/log/nxdns` by hand. The unit's -`StateDirectory` and `LogsDirectory` settings make systemd create them on first -start, `/var/lib/nxdns` at mode 0700 owned by `nxdns`. +Do not create `/var/lib/nxdns` or `/var/log/nxdns` by hand. The unit's `StateDirectory` and `LogsDirectory` settings make systemd create them on first start, `/var/lib/nxdns` at mode 0700 owned by `nxdns`. ## 2. Write the configuration @@ -53,20 +47,14 @@ nxdns will not start with nothing to forward to. Write `/etc/nxdns/config.zon`: } ``` -That file holds a password in plain text. Root's umask is 022 on most -distributions, so restrict it as soon as you have written it: +That file holds a password in plain text. Root's umask is 022 on most distributions, so restrict it as soon as you have written it: ```sh chown root:nxdns /etc/nxdns/config.zon chmod 0640 /etc/nxdns/config.zon ``` -0640 with group `nxdns` rather than 0600: the service runs as `nxdns`, and -systemd leaves `/etc/nxdns` owned by root. Keep that group read bit for good. -Under `run --config` the service reads this file on **every** start, not once, -so tightening the mode after the first boot breaks the next restart. Under -database authority it is `nxdns import` that reads the file, as whoever runs -that command, and a bare `nxdns run` never reads it at all. +0640 with group `nxdns` rather than 0600: the service runs as `nxdns`, and systemd leaves `/etc/nxdns` owned by root. Keep that group read bit for good. Under `run --config` the service reads this file on **every** start, not once, so tightening the mode after the first boot breaks the next restart. Under database authority it is `nxdns import` that reads the file, as whoever runs that command, and a bare `nxdns run` never reads it at all. Check it before starting the service: @@ -74,9 +62,7 @@ Check it before starting the service: nxdns check --config /etc/nxdns/config.zon ``` -A good file ends with `OK: no problems found`. Exit 2 means `check` found -something to fix and printed every problem it found. The upstream probe sends a -real query, so this needs working DNS on the host. +A good file ends with `OK: no problems found`. Exit 2 means `check` found something to fix and printed every problem it found. The upstream probe sends a real query, so this needs working DNS on the host. Load it into the database: @@ -84,10 +70,7 @@ Load it into the database: nxdns import /etc/nxdns/config.zon ``` -The packaged unit runs `nxdns run` with no `--config`, so from here the database -is the configuration and nothing reads the file again. `web.password` is hashed -and the plaintext is never stored, so once you have logged in you can delete the -file: +The packaged unit runs `nxdns run` with no `--config`, so from here the database is the configuration and nothing reads the file again. `web.password` is hashed and the plaintext is never stored, so once you have logged in you can delete the file: ```sh rm /etc/nxdns/config.zon @@ -95,10 +78,7 @@ rm /etc/nxdns/config.zon A kept file is not a backup. `nxdns export` is. -To keep the file as the configuration instead — converged at every start, with -the UI refusing configuration edits — do not delete it, and add a drop-in that -appends `--config=/etc/nxdns/config.zon` to `ExecStart`. See -`docs/how-to/install-with-systemd.md`. +To keep the file as the configuration instead — converged at every start, with the UI refusing configuration edits — do not delete it, and add a drop-in that appends `--config=/etc/nxdns/config.zon` to `ExecStart`. See `docs/how-to/install-with-systemd.md`. ## 3. Start it @@ -107,9 +87,7 @@ systemctl enable --now nxdns journalctl -u nxdns -f ``` -A healthy start logs a line naming every socket it bound. Port 53 is -privileged, and the unit grants `CAP_NET_BIND_SERVICE` through -`AmbientCapabilities`. +A healthy start logs a line naming every socket it bound. Port 53 is privileged, and the unit grants `CAP_NET_BIND_SERVICE` through `AmbientCapabilities`. ## 4. Confirm it answers @@ -119,9 +97,7 @@ From another machine on the LAN: dig @ example.com A +short ``` -The admin interface is on port 8080 by default; log in with the password from -the configuration file. `http://:8080/api/health` reports upstream -availability and disk state without a login. +The admin interface is on port 8080 by default; log in with the password from the configuration file. `http://:8080/api/health` reports upstream availability and disk state without a login. ## More diff --git a/PLAN.md b/PLAN.md index 34520ea..1a67b15 100644 --- a/PLAN.md +++ b/PLAN.md @@ -1,7 +1,6 @@ # nxdns — Implementation Plan v3.0 (Zig 0.16.0 Stable) -Source of truth for **nxdns**, a self-hosted DNS sinkhole written in Zig 0.16.0 stable. -All stdlib claims in this document are verified against the `0.16.0` tag of the Zig repo (`../zig`). +Source of truth for **nxdns**, a self-hosted DNS sinkhole written in Zig 0.16.0 stable. All stdlib claims in this document are verified against the `0.16.0` tag of the Zig repo (`../zig`). There is no v1/v2 versioning. Scope is binary: a feature is in scope (and gets built) or out of scope (and does not). "Done" = everything in scope implemented, tested, documented. @@ -121,16 +120,9 @@ Rule kinds: `exact`, parent-walk (implicit via candidate chain), `wildcard` (`*` Tie-break at same specificity: **allow wins**. -Each level is checked against the whole candidate chain before the next level is -checked against any, which is what makes an allow rule on a parent beat a block -rule on the child. +Each level is checked against the whole candidate chain before the next level is checked against any, which is what makes an allow rule on a parent beat a block rule on the child. -Two positions carry an argument rather than a preference. The regex levels come -last among the operator rules because they are the only ones that are not a set -lookup or a label walk: a regex runs only once every cheaper level has missed. -Blocklist exceptions come below **every** operator level because a downloaded -list may cancel what another list blocked and must never cancel what the -operator decided — no list can open an allow hole the operator did not open. +Two positions carry an argument rather than a preference. The regex levels come last among the operator rules because they are the only ones that are not a set lookup or a label walk: a regex runs only once every cheaper level has missed. Blocklist exceptions come below **every** operator level because a downloaded list may cancel what another list blocked and must never cancel what the operator decided — no list can open an allow hole the operator did not open. ### 3.11 Network Posture @@ -290,8 +282,7 @@ Walk chain to depth 8; any target hitting block logic → synthesize blocked res ### 7.1 Evaluation -For `{domain, group_id}` (the qtype travels with the query for logging and response synthesis, not -for matching): +For `{domain, group_id}` (the qtype travels with the query for logging and response synthesis, not for matching): 1. Normalize: lowercase, trim trailing dot. 2. Build candidate chain (full, parent1, parent2, …). 3. Explicit rules per §3.10 precedence, allow before block at each level: exact rules against every candidate in the chain, then wildcard patterns and then regex patterns against the whole name (both kinds express their own reach, so neither walks the chain). @@ -300,10 +291,7 @@ for matching): 6. Group's blocklist wildcards, matched against every proper parent of the query name. 7. No match → allow. -Blocklist *domain* entries do not parent-walk: they are matched against the query name alone. Wildcard -entries match every proper parent, and exception entries walk the candidate chain the way rules do -(§3.9), so `@@||good.ads.example^` also lifts `y.good.ads.example`. ABP `||x.y^` emits both a domain -entry `x.y` and a wildcard entry `x.y`, which together give domain-and-subdomains semantics. +Blocklist *domain* entries do not parent-walk: they are matched against the query name alone. Wildcard entries match every proper parent, and exception entries walk the candidate chain the way rules do (§3.9), so `@@||good.ads.example^` also lifts `y.good.ads.example`. ABP `||x.y^` emits both a domain entry `x.y` and a wildcard entry `x.y`, which together give domain-and-subdomains semantics. ### 7.2 Group Assignment @@ -314,12 +302,7 @@ entry `x.y` and a wildcard entry `x.y`, which together give domain-and-subdomain ### 7.3 Reload -New immutable matcher built from DB + compiled list files → swap under an `std.Io.RwLock` with a -generation counter. Readers take the shared lock for the microseconds of one evaluate; the writer -takes the exclusive lock only for the swap, and the source status table is installed in the same -critical section, so a failed reload publishes neither. (Deliberate deviation from "readers -lock-free": freeing the old snapshot without a lock needs epoch-based reclamation, unjustifiable at -household scale — see specs/milestone-5.md S8.3.) +New immutable matcher built from DB + compiled list files → swap under an `std.Io.RwLock` with a generation counter. Readers take the shared lock for the microseconds of one evaluate; the writer takes the exclusive lock only for the swap, and the source status table is installed in the same critical section, so a failed reload publishes neither. (Deliberate deviation from "readers lock-free": freeing the old snapshot without a lock needs epoch-based reclamation, unjustifiable at household scale — see specs/milestone-5.md S8.3.) ### 7.4 Safe-Search @@ -365,14 +348,9 @@ Per-group boolean. Rewrites known engine domains to their safe-search CNAME targ ### 11.2 config.db Schema (v1 baseline) -The DDL below is the live schema, kept byte-identical to -`src/storage/config_schema.zig`. `src/storage/migrations.zig` carries it as its -one and only step, so a database is at **version 1** or it does not exist. +The DDL below is the live schema, kept byte-identical to `src/storage/config_schema.zig`. `src/storage/migrations.zig` carries it as its one and only step, so a database is at **version 1** or it does not exist. -Until nxdns reaches v0.1 this baseline is **editable**: a schema change edits -this section and `config_schema.zig` together and adds no migration step. nxdns -has no installs, so there is no database for a step to reconcile. At v0.1 the -baseline freezes and every later change becomes an append-only step. +Until nxdns reaches v0.1 this baseline is **editable**: a schema change edits this section and `config_schema.zig` together and adds no migration step. nxdns has no installs, so there is no database for a step to reconcile. At v0.1 the baseline freezes and every later change becomes an append-only step. ```sql CREATE TABLE schema_version (version INTEGER NOT NULL); @@ -509,14 +487,7 @@ Periodic delete of rows older than `retention_days`; scheduled checkpoint/VACUUM ### 12.1 Config ZON Shape -The canonical shape is not duplicated here. It lives in -[docs/reference/configuration.md](docs/reference/configuration.md), which is -handwritten against `config/model.zig` and only partly guarded (the drift test -covers settings-key rows, not the whole shape, so a new collection can go -undocumented while the guard stays green), and `nxdns export` emits it. A copy in -this document is how §12.1 came to describe an `.upstream.servers` field that -never existed and to omit the required `.groups` and `.upstreams` — a sample -nobody could load. The skeleton, for orientation only: +The canonical shape is not duplicated here. It lives in [docs/reference/configuration.md](docs/reference/configuration.md), which is handwritten against `config/model.zig` and only partly guarded (the drift test covers settings-key rows, not the whole shape, so a new collection can go undocumented while the guard stays green), and `nxdns export` emits it. A copy in this document is how §12.1 came to describe an `.upstream.servers` field that never existed and to omit the required `.groups` and `.upstreams` — a sample nobody could load. The skeleton, for orientation only: ```zon .{ @@ -584,48 +555,37 @@ Requirements: responsive desktop/mobile; route loaders for initial fetch; TanSta ## 16. Implementation Order ### Phase 0 — Build Baseline -Scaffold tree; `build.zig` with 0.16 assertion, musl targets, vendored sqlite3 + mbedtls compiling; version plumbing; Gitea Actions workflows (test, integration, fuzz smoke, OpenAPI lint, frontend build). -Exit: cross-compiled hello-world linking both C deps on both targets; CI green. +Scaffold tree; `build.zig` with 0.16 assertion, musl targets, vendored sqlite3 + mbedtls compiling; version plumbing; Gitea Actions workflows (test, integration, fuzz smoke, OpenAPI lint, frontend build). Exit: cross-compiled hello-world linking both C deps on both targets; CI green. ### Phase 1 — Platform Layer -`platform/address.zig` (v4/v6 parity + canonical keys); `platform/tls_client.zig` (deadlines, error classes); `platform/tls_server.zig` (mbedTLS handshake → `std.Io.Reader`/`Writer`). -Exit: UDP echo over `std.Io`; TLS client handshake against a real host; mbedTLS server terminating a loopback TLS connection. +`platform/address.zig` (v4/v6 parity + canonical keys); `platform/tls_client.zig` (deadlines, error classes); `platform/tls_server.zig` (mbedTLS handshake → `std.Io.Reader`/`Writer`). Exit: UDP echo over `std.Io`; TLS client handshake against a real host; mbedTLS server terminating a loopback TLS connection. ### Phase 2 — DNS Core -Types/header/name/question/record/packet; parser + encoder tests + fuzz target; EDNS + DO passthrough. -Exit: unit + fuzz smoke pass. +Types/header/name/question/record/packet; parser + encoder tests + fuzz target; EDNS + DO passthrough. Exit: unit + fuzz smoke pass. ### Phase 3 — Resolver Transport -UDP server, TCP server, DoH + DoT upstream clients, pool + failover/backoff + health. -Exit: A/AAAA forwarding over UDP + TCP; health populated. +UDP server, TCP server, DoH + DoT upstream clients, pool + failover/backoff + health. Exit: A/AAAA forwarding over UDP + TCP; health populated. ### Phase 4 — Storage + Config -SQLite wrapper; config.db schema + migration runner; querylog.db schema + recreate-on-mismatch; repositories; ZON loading + import/export; `nxdns check`. -Exit: export → import round-trips byte-stable. (The ZON bootstrap this phase shipped was replaced in m20 by the two authority modes above.) +SQLite wrapper; config.db schema + migration runner; querylog.db schema + recreate-on-mismatch; repositories; ZON loading + import/export; `nxdns check`. Exit: export → import round-trips byte-stable. (The ZON bootstrap this phase shipped was replaced in m20 by the two authority modes above.) ### Phase 5 — Filtering + Local DNS -Rule matcher (exact/parent/wildcard; `regex` added in m21); blocklist parsers → compiled file format; fetcher + scheduled update; RCU swap; per-group safe-search; local records; forward zones. -Exit: precedence table validated by tests; local zone answers + conditional forwards work. +Rule matcher (exact/parent/wildcard; `regex` added in m21); blocklist parsers → compiled file format; fetcher + scheduled update; RCU swap; per-group safe-search; local records; forward zones. Exit: precedence table validated by tests; local zone answers + conditional forwards work. ### Phase 6 — Cache + Rate Limit + Logging + Disk Monitor -TTL cache + reconstruction; v4/v6 rate limiter; async query logger + backpressure + retention; DiskMonitor + rotation + error-log dedup. -Exit: disk thresholds trigger degradation + drop counters in integration test. +TTL cache + reconstruction; v4/v6 rate limiter; async query logger + backpressure + retention; DiskMonitor + rotation + error-log dedup. Exit: disk thresholds trigger degradation + drop counters in integration test. ### Phase 7 — Handler Integration -Full pipeline composition; CNAME uncloaking; pause/resume. -Exit: end-to-end DNS flow with blocking, local records, cache, failover. +Full pipeline composition; CNAME uncloaking; pause/resume. Exit: end-to-end DNS flow with blocking, local records, cache, failover. ### Phase 8 — Web / API / SSE / Auth / Metrics -HTTP server + router; handlers; SSE; optional auth; API rate limiting; `/metrics`; OpenAPI served + contract tests; embedded frontend + dev-mode disk serving. -Exit: frontend fully drives config and operations; contract tests green. +HTTP server + router; handlers; SSE; optional auth; API rate limiting; `/metrics`; OpenAPI served + contract tests; embedded frontend + dev-mode disk serving. Exit: frontend fully drives config and operations; contract tests green. ### Phase 9 — Local DoH/DoT Endpoints -DoH server + DoT server on `platform/tls_server.zig`; cert watcher + reload. -Exit: LAN client resolves via DoH and DoT against local certs. +DoH server + DoT server on `platform/tls_server.zig`; cert watcher + reload. Exit: LAN client resolves via DoH and DoT against local certs. ### Phase 10 — Packaging + Ops + Docs -systemd unit (`AmbientCapabilities=CAP_NET_BIND_SERVICE`, hardened, writable `/var/lib/nxdns` + optional `/var/log/nxdns`); Dockerfile + compose (53/udp+tcp, 8080; mounts `/etc/nxdns`, `/var/lib/nxdns`); operator/architecture/config-reference/API docs. -Exit: documented deployment works end-to-end on the Pi 5. +systemd unit (`AmbientCapabilities=CAP_NET_BIND_SERVICE`, hardened, writable `/var/lib/nxdns` + optional `/var/log/nxdns`); Dockerfile + compose (53/udp+tcp, 8080; mounts `/etc/nxdns`, `/var/lib/nxdns`); operator/architecture/config-reference/API docs. Exit: documented deployment works end-to-end on the Pi 5. --- @@ -660,9 +620,7 @@ Exit: documented deployment works end-to-end on the Pi 5. ## 20. Publication -The project publishes released binaries and container images from its own Gitea -instance. Building from source stays fully supported and documented; it is no -longer the only path. +The project publishes released binaries and container images from its own Gitea instance. Building from source stays fully supported and documented; it is no longer the only path. - **Trigger.** Pushing an annotated, GPG-signed tag `vX.Y.Z` to `git.mial.net/mokhtar/nxdns`. Nothing else publishes. Pre-release tags are rejected. - **Version.** The tag is authoritative. `build.zig.zon`'s `.version` must equal the tag, and the packaging gate asserts it. Nowhere else stores a version. diff --git a/README.md b/README.md index 6594ce1..e59c8f7 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,6 @@ # nxdns -nxdns is a DNS sinkhole for your LAN. It blocks the names you do not want, and -forwards the rest over an encrypted connection. +nxdns is a DNS sinkhole for your LAN. It blocks the names you do not want, and forwards the rest over an encrypted connection. ```mermaid flowchart LR @@ -12,49 +11,30 @@ flowchart LR nxdns -- "blocked" --> sink["0.0.0.0 / NXDOMAIN"] ``` -One static Zig binary. SQLite holds the state and the query log, which the web -UI, the REST API and `/metrics` read. +One static Zig binary. SQLite holds the state and the query log, which the web UI, the REST API and `/metrics` read. ## Features -- Blocklist filtering: subscribe to hosts, domain and Adblock Plus lists — - whose `@@` exception lines are honoured — plus your own allow and block rules, - exact, wildcard (`*.example.com`) or regular expression -- Two configuration modes: a database the web UI edits, or a ZON file you keep - in git and converge onto at every start -- Per-client policy groups: different filtering for the kids' tablet and - your workstation +- Blocklist filtering: subscribe to hosts, domain and Adblock Plus lists — whose `@@` exception lines are honoured — plus your own allow and block rules, exact, wildcard (`*.example.com`) or regular expression +- Two configuration modes: a database the web UI edits, or a ZON file you keep in git and converge onto at every start +- Per-client policy groups: different filtering for the kids' tablet and your workstation - Local DNS records and conditional forwarding for internal zones - Encrypted upstreams: DNS-over-HTTPS and DNS-over-TLS with failover - Built-in DoH and DoT server endpoints, with certificate hot-reload - Bounded in-memory DNS cache with TTL-respecting expiry - Query log with retention limits, live-streamed over SSE - Web UI (embedded in the binary) and a REST API with a served OpenAPI spec -- Prometheus-style `/metrics`, per-client rate limiting, disk-full - self-protection +- Prometheus-style `/metrics`, per-client rate limiting, disk-full self-protection ## Install -**No release exists yet.** This repository has no tags, nothing has been -published to , and no container -image has been pushed. Every release URL on this page and in the how-to guides -is a 404 today, and `docker pull` finds nothing. Until the first tag ships, -building from source is the only way to get nxdns. +**No release exists yet.** This repository has no tags, nothing has been published to , and no container image has been pushed. Every release URL on this page and in the how-to guides is a 404 today, and `docker pull` finds nothing. Until the first tag ships, building from source is the only way to get nxdns. -What a tag will publish, once one exists: five assets — two static musl -tarballs (`nxdns--x86_64-linux-musl.tar.gz`, -`nxdns--aarch64-linux-musl.tar.gz`), `IMAGE-DIGEST.txt` naming the -multi-architecture container image by digest, `SHA256SUMS.txt` covering those -three files, and `SHA256SUMS.txt.asc`, a detached OpenPGP signature over the -checksum file. Verify what you downloaded before you run it: -[docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md), which also -says what that signature does and does not prove. +What a tag will publish, once one exists: five assets — two static musl tarballs (`nxdns--x86_64-linux-musl.tar.gz`, `nxdns--aarch64-linux-musl.tar.gz`), `IMAGE-DIGEST.txt` naming the multi-architecture container image by digest, `SHA256SUMS.txt` covering those three files, and `SHA256SUMS.txt.asc`, a detached OpenPGP signature over the checksum file. Verify what you downloaded before you run it: [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md), which also says what that signature does and does not prove. ## Quickstart (docker compose) -Write a minimal configuration and start the published image. This is what the -first release will make possible; it does not work today, because there is no -image in the registry to pull: +Write a minimal configuration and start the published image. This is what the first release will make possible; it does not work today, because there is no image in the registry to pull: ```sh cd deploy/docker @@ -69,32 +49,15 @@ EOF NXDNS_VERSION= docker compose up -d ``` -The compose file defaults to `:latest`; pin a version for anything you intend -to keep running. To run it before a release exists, build the image yourself and -name it — `NXDNS_IMAGE=nxdns docker compose up -d` — as -[docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md) -describes. DNS is on port 53, the web UI on . +The compose file defaults to `:latest`; pin a version for anything you intend to keep running. To run it before a release exists, build the image yourself and name it — `NXDNS_IMAGE=nxdns docker compose up -d` — as [docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md) describes. DNS is on port 53, the web UI on . -The compose file runs `nxdns run --config=/etc/nxdns/config.zon`, which makes -that file the configuration: every start reconciles the database onto it, and -the UI refuses configuration edits. Edit the file and restart to change -anything. Drop the `command:` line to run bare `nxdns run` instead, where the -database is the configuration and changes go through the UI, the API, or -`nxdns export` / `nxdns import` — the packaged systemd unit does that. Which -mode is live is printed at every start (`authority: database` / -`authority: file ()`); see -[docs/explanation/configuration-model.md](docs/explanation/configuration-model.md). +The compose file runs `nxdns run --config=/etc/nxdns/config.zon`, which makes that file the configuration: every start reconciles the database onto it, and the UI refuses configuration edits. Edit the file and restart to change anything. Drop the `command:` line to run bare `nxdns run` instead, where the database is the configuration and changes go through the UI, the API, or `nxdns export` / `nxdns import` — the packaged systemd unit does that. Which mode is live is printed at every start (`authority: database` / `authority: file ()`); see [docs/explanation/configuration-model.md](docs/explanation/configuration-model.md). -Full install instructions, including the systemd path and the Pi 5 recipe, are -in -[docs/how-to/install-with-systemd.md](docs/how-to/install-with-systemd.md) and -[docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md). +Full install instructions, including the systemd path and the Pi 5 recipe, are in [docs/how-to/install-with-systemd.md](docs/how-to/install-with-systemd.md) and [docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md). ## Building from source -Requires [Zig 0.16.0](https://ziglang.org/download/) and Node.js 24 (for -the web UI). C dependencies (SQLite, mbedTLS) are vendored and built by -`zig build`. +Requires [Zig 0.16.0](https://ziglang.org/download/) and Node.js 24 (for the web UI). C dependencies (SQLite, mbedTLS) are vendored and built by `zig build`. ```sh (cd web && npm ci && npm run build) # web UI -> web/dist @@ -102,8 +65,7 @@ zig build -Dweb-dist=web/dist # native binary -> zig-out/bin/nxdn zig build test --summary all # unit tests ``` -The release artifacts come out of the same build graph, so the whole release -build runs on a laptop exactly as it runs on the CI runner: +The release artifacts come out of the same build graph, so the whole release build runs on a laptop exactly as it runs on the CI runner: ```sh (cd web && npm ci && npm run build) # required: dist refuses the placeholder @@ -114,45 +76,25 @@ zig build verify-dist -Dversion-string="$VERSION" -Dgit-commit=$(git rev-parse H -Dweb-dist=web/dist -Doptimize=ReleaseSafe # the release checks ``` -The version comes from `build.zig.zon` because `verify-dist` asserts the two -agree; a tag sets both. +The version comes from `build.zig.zon` because `verify-dist` asserts the two agree; a tag sets both. -That is not a claim that your tarball will hash the same as a published one. -Nothing in this project measures whether two builds of the same commit on two -different machines land on the same bytes, so no document here describes the -build as reproducible. The gate that would settle it is a recorded deferral — -`specs/milestone-14.md` ruling 12 — and -[docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md) explains what -a matching or differing hash is worth in the meantime. +That is not a claim that your tarball will hash the same as a published one. Nothing in this project measures whether two builds of the same commit on two different machines land on the same bytes, so no document here describes the build as reproducible. The gate that would settle it is a recorded deferral — `specs/milestone-14.md` ruling 12 — and [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md) explains what a matching or differing hash is worth in the meantime. ## Documentation -Start at [docs/README.md](docs/README.md), which splits the documentation -into a tutorial, how-to guides, reference and explanation. +Start at [docs/README.md](docs/README.md), which splits the documentation into a tutorial, how-to guides, reference and explanation. -- [docs/tutorial/first-run.md](docs/tutorial/first-run.md) — build it, resolve - a name, block a domain, on a scratch directory -- [docs/how-to/install-with-systemd.md](docs/how-to/install-with-systemd.md) — - a real install, including the Raspberry Pi 5 -- [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md) — checking - the hashes and the signature before you install -- [docs/reference/configuration.md](docs/reference/configuration.md) — every - configuration field +- [docs/tutorial/first-run.md](docs/tutorial/first-run.md) — build it, resolve a name, block a domain, on a scratch directory +- [docs/how-to/install-with-systemd.md](docs/how-to/install-with-systemd.md) — a real install, including the Raspberry Pi 5 +- [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md) — checking the hashes and the signature before you install +- [docs/reference/configuration.md](docs/reference/configuration.md) — every configuration field - [docs/reference/api.md](docs/reference/api.md) — REST API, auth and SSE -- [docs/reference/cli.md](docs/reference/cli.md) — subcommands, flags and exit - codes -- [docs/explanation/architecture.md](docs/explanation/architecture.md) — module - map and design -- [PLAN.md](PLAN.md) and [specs/](specs/) — scope, design decisions and - per-milestone contracts +- [docs/reference/cli.md](docs/reference/cli.md) — subcommands, flags and exit codes +- [docs/explanation/architecture.md](docs/explanation/architecture.md) — module map and design +- [PLAN.md](PLAN.md) and [specs/](specs/) — scope, design decisions and per-milestone contracts ## Licence -Copyright (c) 2026 Mokhtar Mial. nxdns is licensed under the European Union -Public Licence v. 1.2 (`EUPL-1.2`); the full text is in [LICENSE](LICENSE). +Copyright (c) 2026 Mokhtar Mial. nxdns is licensed under the European Union Public Licence v. 1.2 (`EUPL-1.2`); the full text is in [LICENSE](LICENSE). -Every released tarball and image carries a `THIRD-PARTY-NOTICES` file assembled -from the reviewed inventory in [licenses/](licenses/), which covers what the -artifacts actually contain: musl, the Zig runtime, SQLite, Mbed TLS and its -vendored Everest and p256-m code, and the JavaScript and CSS bundled into the -admin UI. +Every released tarball and image carries a `THIRD-PARTY-NOTICES` file assembled from the reviewed inventory in [licenses/](licenses/), which covers what the artifacts actually contain: musl, the Zig runtime, SQLite, Mbed TLS and its vendored Everest and p256-m code, and the JavaScript and CSS bundled into the admin UI. diff --git a/docs/README.md b/docs/README.md index 1fcd813..6a85a90 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,8 +1,6 @@ # nxdns documentation -The pages are split by what you are trying to do, following -[Diátaxis](https://diataxis.fr/). Each page serves one of four purposes, and -knowing which one you want is the fastest way to the right page. +The pages are split by what you are trying to do, following [Diátaxis](https://diataxis.fr/). Each page serves one of four purposes, and knowing which one you want is the fastest way to the right page. | Mode | For | Read it when | | --- | --- | --- | @@ -13,62 +11,40 @@ knowing which one you want is the fastest way to the right page. ## Tutorial -A lesson, not a procedure: one path with one outcome, on a scratch directory you -can delete afterwards. +A lesson, not a procedure: one path with one outcome, on a scratch directory you can delete afterwards. -- [tutorial/first-run.md](tutorial/first-run.md) — build nxdns, resolve a name, - block a domain from a real blocklist, open the web interface, stop cleanly. +- [tutorial/first-run.md](tutorial/first-run.md) — build nxdns, resolve a name, block a domain from a real blocklist, open the web interface, stop cleanly. ## How-to guides Steps for a goal you already have. They assume you know what nxdns is. -- [how-to/verify-a-release.md](how-to/verify-a-release.md) — check the - signature and the checksums before you run anything, and what they prove. -- [how-to/install-with-systemd.md](how-to/install-with-systemd.md) — a real - install as a system service, including the Raspberry Pi 5 aarch64 binary. -- [how-to/install-with-docker.md](how-to/install-with-docker.md) — the published - container image and the compose file. -- [how-to/upgrade.md](how-to/upgrade.md) — move to a new release without losing - state. -- [how-to/troubleshoot.md](how-to/troubleshoot.md) — what to do when it does not - answer, does not block, or will not start. -- [how-to/enable-doh-and-dot.md](how-to/enable-doh-and-dot.md) — serve encrypted - DNS with certificates. -- [how-to/set-up-admin-authentication.md](how-to/set-up-admin-authentication.md) - — put a password on the web interface and the API. -- [how-to/back-up-and-restore.md](how-to/back-up-and-restore.md) — export and - import the configuration, and what to copy. -- [how-to/measure-performance.md](how-to/measure-performance.md) — run the - benchmark harness on your own hardware. +- [how-to/verify-a-release.md](how-to/verify-a-release.md) — check the signature and the checksums before you run anything, and what they prove. +- [how-to/install-with-systemd.md](how-to/install-with-systemd.md) — a real install as a system service, including the Raspberry Pi 5 aarch64 binary. +- [how-to/install-with-docker.md](how-to/install-with-docker.md) — the published container image and the compose file. +- [how-to/upgrade.md](how-to/upgrade.md) — move to a new release without losing state. +- [how-to/troubleshoot.md](how-to/troubleshoot.md) — what to do when it does not answer, does not block, or will not start. +- [how-to/enable-doh-and-dot.md](how-to/enable-doh-and-dot.md) — serve encrypted DNS with certificates. +- [how-to/set-up-admin-authentication.md](how-to/set-up-admin-authentication.md) — put a password on the web interface and the API. +- [how-to/back-up-and-restore.md](how-to/back-up-and-restore.md) — export and import the configuration, and what to copy. +- [how-to/measure-performance.md](how-to/measure-performance.md) — run the benchmark harness on your own hardware. ## Reference Descriptions of what is there. No procedures, no advice. -- [reference/configuration.md](reference/configuration.md) — every - configuration section, field, default and range. -- [reference/api.md](reference/api.md) — every REST route, authentication and - the event stream. -- [reference/cli.md](reference/cli.md) — the six subcommands, every flag, every - exit code. -- [reference/files-and-directories.md](reference/files-and-directories.md) — the - data directory layout and file modes. -- [reference/performance.md](reference/performance.md) — the targets and the - measured numbers. +- [reference/configuration.md](reference/configuration.md) — every configuration section, field, default and range. +- [reference/api.md](reference/api.md) — every REST route, authentication and the event stream. +- [reference/cli.md](reference/cli.md) — the six subcommands, every flag, every exit code. +- [reference/files-and-directories.md](reference/files-and-directories.md) — the data directory layout and file modes. +- [reference/performance.md](reference/performance.md) — the targets and the measured numbers. ## Explanation -Background. Nothing here is needed to operate nxdns; it is here so the decisions -are inspectable. +Background. Nothing here is needed to operate nxdns; it is here so the decisions are inspectable. -- [explanation/architecture.md](explanation/architecture.md) — the module map - and the design it comes from. -- [explanation/configuration-model.md](explanation/configuration-model.md) — why - there are two authority modes, how each one is selected, and what each is for. -- [explanation/performance-and-testing.md](explanation/performance-and-testing.md) - — why the targets exist, why CI does not gate on them, and what the hermetic - tests do and do not prove. +- [explanation/architecture.md](explanation/architecture.md) — the module map and the design it comes from. +- [explanation/configuration-model.md](explanation/configuration-model.md) — why there are two authority modes, how each one is selected, and what each is for. +- [explanation/performance-and-testing.md](explanation/performance-and-testing.md) — why the targets exist, why CI does not gate on them, and what the hermetic tests do and do not prove. -Scope and per-milestone contracts live outside this directory, in -[../PLAN.md](../PLAN.md) and [../specs/](../specs/). +Scope and per-milestone contracts live outside this directory, in [../PLAN.md](../PLAN.md) and [../specs/](../specs/). diff --git a/docs/explanation/architecture.md b/docs/explanation/architecture.md index 55c4567..e4b3f64 100644 --- a/docs/explanation/architecture.md +++ b/docs/explanation/architecture.md @@ -1,15 +1,8 @@ # Architecture -nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary -that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against -blocklists, and serves an embedded admin SPA over HTTP. This page maps the -source tree and explains the few design rules that hold everywhere, and why -they are the rules. +nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against blocklists, and serves an embedded admin SPA over HTTP. This page maps the source tree and explains the few design rules that hold everywhere, and why they are the rules. -For what the configuration fields, API routes and CLI flags actually are, see -[reference/configuration.md](../reference/configuration.md), -[reference/api.md](../reference/api.md) and -[reference/cli.md](../reference/cli.md). This page does not repeat them. +For what the configuration fields, API routes and CLI flags actually are, see [reference/configuration.md](../reference/configuration.md), [reference/api.md](../reference/api.md) and [reference/cli.md](../reference/cli.md). This page does not repeat them. ## Module map @@ -39,8 +32,7 @@ Directories: | `src/web/` | The admin HTTP layer: `server.zig` (listener), `router.zig`/`routes.zig`, one file per resource under `handlers/`, `auth.zig` (sessions), `sse.zig` (live query fanout), `static.zig` (embedded SPA), `metrics.zig` (Prometheus), `openapi.zig` (served contract), `api_limiter.zig`, `http_util.zig`. | | `src/platform/` | OS and TLS edges: IP address values, the `std.log` sink (`logging.zig`), `statfs.zig` (free-space query via libc), client TLS over `std.crypto.tls` (`tls_client.zig`), server TLS over vendored Mbed TLS (`tls_server.zig`). | -The SPA source lives in `web/` at the repo root; the build embeds its `dist/` -output as the `web_assets` module (`-Dweb-dist`). +The SPA source lives in `web/` at the repo root; the build embeds its `dist/` output as the `web_assets` module (`-Dweb-dist`). ``` main.zig ── cli.zig ── app.zig (composition root) @@ -58,48 +50,23 @@ main.zig ── cli.zig ── app.zig (composition root) ## The purity rule -`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no -`std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs -a timestamp takes it as a parameter — the cache, the rate limiter and the -pause flag all work this way, so every decision is testable without a backend. +`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no `std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs a timestamp takes it as a parameter — the cache, the rate limiter and the pause flag all work this way, so every decision is testable without a backend. -The point is not purity for its own sake. A decision that depends on a hidden -clock or a hidden socket can only be tested by arranging the world around it; -one that takes the clock as an argument is tested by passing a number. The -whole filtering and caching pipeline can therefore be exercised in the plain, -network-free test suite, which is what makes that suite worth gating CI on. +The point is not purity for its own sake. A decision that depends on a hidden clock or a hidden socket can only be tested by arranging the world around it; one that takes the clock as an argument is tested by passing a number. The whole filtering and caching pipeline can therefore be exercised in the plain, network-free test suite, which is what makes that suite worth gating CI on. -The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads -lists, `filter/manager.zig` owns the compiled files, the DB columns and the -snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN -resolver. Those three files are the only ones under those four directories -that take a `std.Io`. The decision path a query takes through them allocates -nothing and opens nothing. +The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads lists, `filter/manager.zig` owns the compiled files, the DB columns and the snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN resolver. Those three files are the only ones under those four directories that take a `std.Io`. The decision path a query takes through them allocates nothing and opens nothing. -The honest cost of the exceptions shows up in -[performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is -where the one production crash came from, precisely because it is the file the -pure suite cannot reach. +The honest cost of the exceptions shows up in [performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is where the one production crash came from, precisely because it is the file the pure suite cannot reach. ## std.Io injection -There is one `std.Io` in the process. `main` receives it through -`std.process.Init` — on the standard start path this is the Threaded backend -(`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to -`cli.Runner`, from which `app.zig` threads it into every collaborator as a -parameter. No module constructs its own event loop or reads an ambient clock; -tests build their own `std.Io.Threaded` instance and pass it the same way. +There is one `std.Io` in the process. `main` receives it through `std.process.Init` — on the standard start path this is the Threaded backend (`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to `cli.Runner`, from which `app.zig` threads it into every collaborator as a parameter. No module constructs its own event loop or reads an ambient clock; tests build their own `std.Io.Threaded` instance and pass it the same way. -The one deliberate exception is `storage/db.zig`: SQLite performs its own file -I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's -VFS to route through `std.Io` would be a large amount of C-boundary code to -make one dependency match a convention it does not need. +The one deliberate exception is `storage/db.zig`: SQLite performs its own file I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's VFS to route through `std.Io` would be a large amount of C-boundary code to make one dependency match a convention it does not need. ## Life of one query -The pipeline lives in `src/server/handler.zig` — `Handler.handle` does -validation and setup, then `Context.run` decides the answer. Its order is -PLAN §4; the stages below are the code's actual call chain: +The pipeline lives in `src/server/handler.zig` — `Handler.handle` does validation and setup, then `Context.run` decides the answer. Its order is PLAN §4; the stages below are the code's actual call chain: ``` UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig) @@ -132,139 +99,46 @@ UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig) └─► async logger ─► querylog.db ``` -Local records win over forward zones, and both win over filtering: a name -nxdns answers itself never reaches a blocklist. Pause suspends filtering only; -local records, forward zones, cache, upstream and the query log keep running, -which is what makes pause safe to hand to a household member. A question whose -class is not IN bypasses local answers, filtering and the cache entirely and -goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and -declines to cache answers it does not model. +Local records win over forward zones, and both win over filtering: a name nxdns answers itself never reaches a blocklist. Pause suspends filtering only; local records, forward zones, cache, upstream and the query log keep running, which is what makes pause safe to hand to a household member. A question whose class is not IN bypasses local answers, filtering and the cache entirely and goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and declines to cache answers it does not model. -`handle` returns no error union. Every failure is either a DNS response the -client can act on or a counted drop, because there is no caller above it that -could do anything useful with a Zig error. Two consequences are worth knowing: -answers synthesized from a safe-search rewrite are never cached (the rewrite is -per group, and the cache is not), and a SERVFAIL reply short-circuits before the -query sink, so it appears in the counters but not in the query log. +`handle` returns no error union. Every failure is either a DNS response the client can act on or a counted drop, because there is no caller above it that could do anything useful with a Zig error. Two consequences are worth knowing: answers synthesized from a safe-search rewrite are never cached (the rewrite is per group, and the cache is not), and a SERVFAIL reply short-circuits before the query sink, so it appears in the counters but not in the query log. -The query path never waits on the database. `QuerySink` copies the entry, the -SSE hub gets it first, and one writer task owns the `querylog.db` handle behind -an `std.Io.Queue`. A slow disk delays logging, never resolution. +The query path never waits on the database. `QuerySink` copies the entry, the SSE hub gets it first, and one writer task owns the `querylog.db` handle behind an `std.Io.Queue`. A slow disk delays logging, never resolution. -Two smaller decisions in the same spirit: the upstream pool makes a second -pass that ignores backoff, so "every endpoint is in backoff" degrades to -trying anyway rather than to a blanket SERVFAIL; and a handler that has no -filter snapshot yet answers unfiltered rather than refusing. Both prefer a -working resolver over a correct-looking failure. +Two smaller decisions in the same spirit: the upstream pool makes a second pass that ignores backoff, so "every endpoint is in backoff" degrades to trying anyway rather than to a blanket SERVFAIL; and a handler that has no filter snapshot yet answers unfiltered rather than refusing. Both prefer a working resolver over a correct-looking failure. ## Storage -Two databases with opposite contracts, in one data directory (see -[reference/files-and-directories.md](../reference/files-and-directories.md)). +Two databases with opposite contracts, in one data directory (see [reference/files-and-directories.md](../reference/files-and-directories.md)). -**`config.db` is what the server reads.** Its schema is versioned: `migrations.zig` holds -an ordered list of steps, step 1 being the verbatim DDL from -`config_schema.zig`, each applied inside one transaction. `nxdns import` -replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed -import changes nothing; `nxdns export` renders it back as canonical ZON, -byte-identical across round trips. +**`config.db` is what the server reads.** Its schema is versioned: `migrations.zig` holds an ordered list of steps, step 1 being the verbatim DDL from `config_schema.zig`, each applied inside one transaction. `nxdns import` replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed import changes nothing; `nxdns export` renders it back as canonical ZON, byte-identical across round trips. -Which of the file and the database is *authoritative* is chosen by the -invocation, not by state: bare `nxdns run` serves the database, and -`nxdns run --config FILE` makes the file authoritative and reconciles the -database onto it at every start. `reconcile.zig` is that convergence, matching -rows by identity and writing only differences, so runtime state — blocklist -checksums, compiled snapshots, client history — survives. Why it works that way -is [configuration-model.md](configuration-model.md). +Which of the file and the database is *authoritative* is chosen by the invocation, not by state: bare `nxdns run` serves the database, and `nxdns run --config FILE` makes the file authoritative and reconciles the database onto it at every start. `reconcile.zig` is that convergence, matching rows by identity and writing only differences, so runtime state — blocklist checksums, compiled snapshots, client history — survives. Why it works that way is [configuration-model.md](configuration-model.md). -**`querylog.db` is expendable.** It is never migrated. Its schema carries a -fingerprint derived from the DDL text, and at open, a missing, corrupt, -non-database, `quick_check`-failing or fingerprint-mismatched file is moved -aside and recreated empty — the old file is kept under a new name rather than -deleted, so an operator can still look at it. Retention deletes old rows daily -and periodically rewrites the file to reclaim space. +**`querylog.db` is expendable.** It is never migrated. Its schema carries a fingerprint derived from the DDL text, and at open, a missing, corrupt, non-database, `quick_check`-failing or fingerprint-mismatched file is moved aside and recreated empty — the old file is kept under a new name rather than deleted, so an operator can still look at it. Retention deletes old rows daily and periodically rewrites the file to reclaim space. -The split exists so that the churn of the second database can never endanger -the first. Query logs are high-volume, disposable, and the thing most likely -to be corrupted by a power cut on an SD card; configuration is small, -irreplaceable, and the thing an operator would have to reconstruct by hand. -Giving them one file would force the careful contract onto the noisy data or -the loose contract onto the valuable data. +The split exists so that the churn of the second database can never endanger the first. Query logs are high-volume, disposable, and the thing most likely to be corrupted by a power cut on an SD card; configuration is small, irreplaceable, and the thing an operator would have to reconstruct by hand. Giving them one file would force the careful contract onto the noisy data or the loose contract onto the valuable data. ## Web stack -`web/server.zig` runs one `std.http.Server` per connection over its own accept -loop, with a fixed set of pre-allocated connection slots, optionally behind -TLS. Over capacity it answers 503 rather than queueing without bound — the -admin UI is not the product, and it must not be able to starve DNS. +`web/server.zig` runs one `std.http.Server` per connection over its own accept loop, with a fixed set of pre-allocated connection slots, optionally behind TLS. Over capacity it answers 503 rather than queueing without bound — the admin UI is not the product, and it must not be able to starve DNS. -The SPA is embedded at build time: `static.zig` serves the `web_assets` module -— bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where -it paid off — via a linear scan with no filesystem access at runtime. (The one -exception is `nxdns run --web-dev DIR`, which serves from disk with no cache -headers, for developing the SPA against a running server.) `GET -/api/queries/live` is server-sent events over chunked transfer, fed by the -same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`) -matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML -is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest -by tests that assert every served route appears in it. +The SPA is embedded at build time: `static.zig` serves the `web_assets` module — bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where it paid off — via a linear scan with no filesystem access at runtime. (The one exception is `nxdns run --web-dev DIR`, which serves from disk with no cache headers, for developing the SPA against a running server.) `GET /api/queries/live` is server-sent events over chunked transfer, fed by the same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`) matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest by tests that assert every served route appears in it. -Authentication (`web/auth.zig`): the operator's password is verified against -an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import -and never stored). A successful login mints a 256-bit token carried in a -cookie; the in-memory session table holds only SHA-256 digests of tokens, -compared in constant time, capped at 32 sessions with LRU eviction. Nothing is -persisted, so a restart logs everyone out — for a household LAN that is a -feature, not a gap. Unauthenticated by design: the monitoring endpoints -(health, version, metrics), the served OpenAPI contract, login itself, and the -static SPA assets, which the router hands to the SPA fallback before any auth -check. Everything else requires the cookie, and the API has its own -token-bucket rate limiter. See -[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md). +Authentication (`web/auth.zig`): the operator's password is verified against an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import and never stored). A successful login mints a 256-bit token carried in a cookie; the in-memory session table holds only SHA-256 digests of tokens, compared in constant time, capped at 32 sessions with LRU eviction. Nothing is persisted, so a restart logs everyone out — for a household LAN that is a feature, not a gap. Unauthenticated by design: the monitoring endpoints (health, version, metrics), the served OpenAPI contract, login itself, and the static SPA assets, which the router hands to the SPA fallback before any auth check. Everything else requires the cookie, and the API has its own token-bucket rate limiter. See [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md). ## DoH, DoT and certificate hot-reload -`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and -`server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server -TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as -`std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether -they are encrypted. +`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and `server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as `std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether they are encrypted. -Certificates hot-reload through `server/cert_store.zig`. One refcounted -`CertStore` per endpoint owns the published TLS context generation; listeners -`acquire` it per connection and `release` it when the connection ends, so a -reload never frees a context mid-handshake. Reload publishes nothing on -failure: both PEM files are read and a whole new context is built before -anything swaps, and any failure leaves the old generation serving. A watcher -polls mtime and size of both files every 30 seconds -(`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same -path on demand and reports the per-endpoint outcome as its payload. +Certificates hot-reload through `server/cert_store.zig`. One refcounted `CertStore` per endpoint owns the published TLS context generation; listeners `acquire` it per connection and `release` it when the connection ends, so a reload never frees a context mid-handshake. Reload publishes nothing on failure: both PEM files are read and a whole new context is built before anything swaps, and any failure leaves the old generation serving. A watcher polls mtime and size of both files every 30 seconds (`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same path on demand and reports the per-endpoint outcome as its payload. -The requirement driving all of this is that a certbot renewal must not need a -restart and must not be able to break DNS. A half-swapped context or a -free-while-in-use would do exactly that, so the store is built so neither is -representable. See -[how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md). +The requirement driving all of this is that a certbot renewal must not need a restart and must not be able to break DNS. A half-swapped context or a free-while-in-use would do exactly that, so the store is built so neither is representable. See [how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md). ## Failure visibility -Every failure mode must be visible, and the surface is counters, not log lines -(AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they -are only read after someone already suspects a problem, and on an SD card they -are a way to fill a disk. +Every failure mode must be visible, and the surface is counters, not log lines (AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they are only read after someone already suspects a problem, and on an SD card they are a way to fill a disk. -So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP, -REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and -forward-zone answers, safe-search rewrites, paused and unfiltered queries, and -the tracker-full condition. Listeners count dropped datagrams instead of -queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text -0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers -200: "degraded" is a fact about the box, not a failed request, and a monitor -that cannot distinguish the two is worse than no monitor. +So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP, REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and forward-zone answers, safe-search rewrites, paused and unfiltered queries, and the tracker-full condition. Listeners count dropped datagrams instead of queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text 0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers 200: "degraded" is a fact about the box, not a failed request, and a monitor that cannot distinguish the two is worse than no monitor. -The disk monitor classifies free space against thresholds and gates -non-essential writes; the query logger holds its batches while writes are -disallowed rather than dropping them silently or writing until the filesystem -fills. `std.log` is reserved for failures nobody else records, with -upstream-error deduplication so a flapping resolver cannot fill a disk with -identical lines. +The disk monitor classifies free space against thresholds and gates non-essential writes; the query logger holds its batches while writes are disallowed rather than dropping them silently or writing until the filesystem fills. `std.log` is reserved for failures nobody else records, with upstream-error deduplication so a flapping resolver cannot fill a disk with identical lines. diff --git a/docs/explanation/configuration-model.md b/docs/explanation/configuration-model.md index cf75e09..f8e8b16 100644 --- a/docs/explanation/configuration-model.md +++ b/docs/explanation/configuration-model.md @@ -1,13 +1,8 @@ # The configuration model -nxdns is configured two ways — a ZON file and a web UI — and only one of them -can be the truth at a time. This page explains how that choice is made, what -each mode is for, and why the design leaves the fewest ways to lose an -operator's work. +nxdns is configured two ways — a ZON file and a web UI — and only one of them can be the truth at a time. This page explains how that choice is made, what each mode is for, and why the design leaves the fewest ways to lose an operator's work. -For the fields themselves see -[reference/configuration.md](../reference/configuration.md); for the commands -and their exit codes see [reference/cli.md](../reference/cli.md). +For the fields themselves see [reference/configuration.md](../reference/configuration.md); for the commands and their exit codes see [reference/cli.md](../reference/cli.md). ## The rule @@ -18,37 +13,19 @@ nxdns run the database is the truth nxdns run --config /etc/nxdns/config.zon the file is the truth ``` -That is the entire selection mechanism. There is no mode setting, no default -file path, and nothing recorded in the database about which mode last wrote it. -A `config.zon` that exists but that no invocation names changes nothing at all. +That is the entire selection mechanism. There is no mode setting, no default file path, and nothing recorded in the database about which mode last wrote it. A `config.zon` that exists but that no invocation names changes nothing at all. Two properties fall out of that, and both were chosen on purpose. -**An operator can read `ExecStart` and know which authority is live.** The -alternative — probe a well-known path, and behave differently depending on -whether a file happens to be there — is ambient magic. It is also the exact -class of rule that produced years of documentation lies in this project: the -old design read the file only while the database was empty, which meant the same -command did two different things depending on state nobody could see from the -command line, and every page that described it eventually described it wrongly. +**An operator can read `ExecStart` and know which authority is live.** The alternative — probe a well-known path, and behave differently depending on whether a file happens to be there — is ambient magic. It is also the exact class of rule that produced years of documentation lies in this project: the old design read the file only while the database was empty, which meant the same command did two different things depending on state nobody could see from the command line, and every page that described it eventually described it wrongly. -**A path already expresses a two-state choice, so a mode flag beside it would -be redundant and worse.** An earlier draft had `--config-source=db|file`. A mode -flag next to a path flag manufactures combinations that cannot mean anything — -a path with no mode, a mode with no path — and each one then needs a pairing -rule and a usage error to defend it. Presence-of-path has no invalid -combinations, so there is nothing to defend. +**A path already expresses a two-state choice, so a mode flag beside it would be redundant and worse.** An earlier draft had `--config-source=db|file`. A mode flag next to a path flag manufactures combinations that cannot mean anything — a path with no mode, a mode with no path — and each one then needs a pairing rule and a usage error to defend it. Presence-of-path has no invalid combinations, so there is nothing to defend. ## Database mode -`nxdns run`. `config.db` holds the configuration; the UI, the API and `nxdns -import` write to it; nothing reads a file. This is the appliance: someone sets -the box up once, and afterwards the household member who wants to unblock one -domain clicks a button. +`nxdns run`. `config.db` holds the configuration; the UI, the API and `nxdns import` write to it; nothing reads a file. This is the appliance: someone sets the box up once, and afterwards the household member who wants to unblock one domain clicks a button. -A fresh install in this mode starts from an empty database, which fails -validation on its own terms — there is nowhere to forward a query to — and says -what to do about it: +A fresh install in this mode starts from an empty database, which fails validation on its own terms — there is nowhere to forward a query to — and says what to do about it: ``` nxdns run failed: NoUsableUpstreams @@ -58,188 +35,85 @@ load one with `nxdns import `, or make a file the source of truth with `nx ## File mode -`nxdns run --config FILE`. The file is the sole declarative source, and the -database becomes the runtime substrate: every start reads the file, validates -it, converges the database onto it, and serves from there. Configuration writes -through the API are refused with a 403. +`nxdns run --config FILE`. The file is the sole declarative source, and the database becomes the runtime substrate: every start reads the file, validates it, converges the database onto it, and serves from there. Configuration writes through the API are refused with a 403. -This is the mode for a file kept in git and pushed by Ansible. What it buys is -that the deployed file is what is running — not "was imported once", not -"was imported unless someone clicked something since". +This is the mode for a file kept in git and pushed by Ansible. What it buys is that the deployed file is what is running — not "was imported once", not "was imported unless someone clicked something since". Three properties make it usable rather than merely correct. -**It fails closed.** A file that is missing, unreadable, unparseable, oversized -or invalid stops the start. nxdns never falls back to the database, because a -fallback turns a deploy typo into a configuration that is silently months old -and looks fine. That failure is exit 2, so `nxdns check --config FILE` is a real -pre-restart gate: validate the pushed file in the handler, and a typo is a -failed deploy at noon rather than a dead resolver at the next power cut. +**It fails closed.** A file that is missing, unreadable, unparseable, oversized or invalid stops the start. nxdns never falls back to the database, because a fallback turns a deploy typo into a configuration that is silently months old and looks fine. That failure is exit 2, so `nxdns check --config FILE` is a real pre-restart gate: validate the pushed file in the handler, and a typo is a failed deploy at noon rather than a dead resolver at the next power cut. -**It converges rather than replaces.** Reconciling matches rows by identity and -writes only what differs. A source whose URL has not changed keeps its row id, -its checksum, its counters and its compiled blocklist files — so a restart in -file mode downloads nothing, which is the difference between a design that is -tolerable to restart and one that costs three minutes and 100 MB every time. +**It converges rather than replaces.** Reconciling matches rows by identity and writes only what differs. A source whose URL has not changed keeps its row id, its checksum, its counters and its compiled blocklist files — so a restart in file mode downloads nothing, which is the difference between a design that is tolerable to restart and one that costs three minutes and 100 MB every time. -**An unchanged file writes nothing at all.** Not "writes the same bytes" — -performs zero write statements, and reports it: +**An unchanged file writes nothing at all.** Not "writes the same bytes" — performs zero write statements, and reports it: ``` reconciled '/etc/nxdns/config.zon': no changes ``` -That matters beyond elegance. A box whose SD card is full of query log can still -restart in file mode, because a no-op reconcile needs no write-ahead-log -headroom. +That matters beyond elegance. A box whose SD card is full of query log can still restart in file mode, because a no-op reconcile needs no write-ahead-log headroom. -**Converged at every boot is not a lock between boots.** Nothing stops `nxdns -import` or a `runtime action` route from moving the database while the server -runs. The contract is that the next start puts it back, and says what it -corrected. +**Converged at every boot is not a lock between boots.** Nothing stops `nxdns import` or a `runtime action` route from moving the database while the server runs. The contract is that the next start puts it back, and says what it corrected. ## What the file cannot take away -The file is authoritative over configuration. It is not authoritative over -things it has no vocabulary for, and reconciling has to preserve those or the -mode is unusable. +The file is authoritative over configuration. It is not authoritative over things it has no vocabulary for, and reconciling has to preserve those or the mode is unusable. -- **Blocklist download state.** Checksums, fetch timestamps and domain counts - belong to the network, not the operator. They survive on every matched row. -- **Client history.** Devices nxdns saw on the wire are kept whole. Naming one - in the file promotes that row in place — it keeps its first-seen and - last-seen and its row id, and counts as an update rather than a delete and an - insert. -- **Devices whose group is un-declared.** Remove a group from the file and the - observed clients assigned to it move to `default`. The operator un-declared - the group, not the devices. -- **The password, when the file does not mention it.** See - [the password](#the-password). +- **Blocklist download state.** Checksums, fetch timestamps and domain counts belong to the network, not the operator. They survive on every matched row. +- **Client history.** Devices nxdns saw on the wire are kept whole. Naming one in the file promotes that row in place — it keeps its first-seen and last-seen and its row id, and counts as an update rather than a delete and an insert. +- **Devices whose group is un-declared.** Remove a group from the file and the observed clients assigned to it move to `default`. The operator un-declared the group, not the devices. +- **The password, when the file does not mention it.** See [the password](#the-password). -The one thing identity cannot survive is a change to identity itself. Edit a -source's URL and the engine sees one row gone and one row arrived: new id, fresh -download, and the old compiled files swept. That is consistent — artifacts are -keyed by row id — and it is why the CLI asks for `--allow-delete` when a diff -deletes anything. +The one thing identity cannot survive is a change to identity itself. Edit a source's URL and the engine sees one row gone and one row arrived: new id, fresh download, and the old compiled files swept. That is consistent — artifacts are keyed by row id — and it is why the CLI asks for `--allow-delete` when a diff deletes anything. ## Why the database is the substrate in both modes -Even in file mode the database is where the server reads its effective -configuration from. That is not a leftover; it is what lets one read path serve -both modes, and it is what makes the runtime state above have somewhere to live. +Even in file mode the database is where the server reads its effective configuration from. That is not a leftover; it is what lets one read path serve both modes, and it is what makes the runtime state above have somewhere to live. -If the file were read directly on every query path there would be no place to -keep a checksum, and no way for the UI to show anything. If both file and -database were authoritative there would be a two-way merge, and a merge cannot -work here: when the UI writes a rule and the file still says otherwise, nothing -in the system knows which of two statements is the newer intention. Every -resolution silently destroys work someone meant to keep. +If the file were read directly on every query path there would be no place to keep a checksum, and no way for the UI to show anything. If both file and database were authoritative there would be a two-way merge, and a merge cannot work here: when the UI writes a rule and the file still says otherwise, nothing in the system knows which of two statements is the newer intention. Every resolution silently destroys work someone meant to keep. -So the modes are exclusive, and the failure mode of each is loud. In database -mode, editing the file does nothing, which you find out the first time you look -at the UI. In file mode, the UI refuses the edit to your face with a message -naming the file to edit instead. +So the modes are exclusive, and the failure mode of each is loud. In database mode, editing the file does nothing, which you find out the first time you look at the UI. In file mode, the UI refuses the edit to your face with a message naming the file to edit instead. ## The round trip -The file stays useful as an editing surface in both modes — text is diffable, -reviewable and easy to back up — because `export` renders the database into the -same shape `import` and file mode read: +The file stays useful as an editing surface in both modes — text is diffable, reviewable and easy to back up — because `export` renders the database into the same shape `import` and file mode read: ``` nxdns export → canonical ZON → edit → nxdns import → config.db ``` -`nxdns export` writes canonical ZON: a fixed two-line header, every default -emitted, deterministic ordering from the model's field order and the -repositories' `ORDER BY` clauses, and no timestamps or hostnames anywhere. -Runtime columns are absent from the configuration model on purpose, so two -exports taken from a live, busy server are identical. `export → import → -export` is byte-identical, and a test asserts it. +`nxdns export` writes canonical ZON: a fixed two-line header, every default emitted, deterministic ordering from the model's field order and the repositories' `ORDER BY` clauses, and no timestamps or hostnames anywhere. Runtime columns are absent from the configuration model on purpose, so two exports taken from a live, busy server are identical. `export → import → export` is byte-identical, and a test asserts it. -Byte-stability is not cosmetic. It is what makes an exported file usable in -version control and what makes a diff of two exports mean something: any -difference is a configuration change, never noise from when the export ran. +Byte-stability is not cosmetic. It is what makes an exported file usable in version control and what makes a diff of two exports mean something: any difference is a configuration change, never noise from when the export ran. -The output is also identical in both modes — nothing marks a file as coming -from a file-mode box. That is deliberate, because it is what makes export the -adoption tool: the file you check is the file you deploy, byte for byte. The -label an operator wants is in the unit file, where they put it. +The output is also identical in both modes — nothing marks a file as coming from a file-mode box. That is deliberate, because it is what makes export the adoption tool: the file you check is the file you deploy, byte for byte. The label an operator wants is in the unit file, where they put it. -Reconciling the same file twice produces a byte-identical database — ids, -checksums, `created_at`, the password hash, the whole settings table — including -when the file is written in a non-canonical but equivalent form, such as -`FD00:0:0:0:0:0:0:1` for an address stored as `fd00::1`. Anything that churns -under an unchanged file is a bug in the engine by definition. That single -invariant is what forces most of the design above: matching on canonical forms, -writing only on difference, treating duplicate rule tuples as a multiset, and -verifying a password rather than re-hashing it. +Reconciling the same file twice produces a byte-identical database — ids, checksums, `created_at`, the password hash, the whole settings table — including when the file is written in a non-canonical but equivalent form, such as `FD00:0:0:0:0:0:0:1` for an address stored as `fd00::1`. Anything that churns under an unchanged file is a bug in the engine by definition. That single invariant is what forces most of the design above: matching on canonical forms, writing only on difference, treating duplicate rule tuples as a multiset, and verifying a password rather than re-hashing it. ## Unknown keys, and why the asymmetry is deliberate -An unknown key in the **database** is warned about and ignored. An unknown key -in the **file** is a hard error. +An unknown key in the **database** is warned about and ignored. An unknown key in the **file** is a hard error. -They are different situations. A settings row the running binary does not -recognise is almost always an older binary reading a database written by a newer -one — a downgrade, or a rollback after a bad upgrade. Refusing to start there -would mean a downgrade bricks the config database, and the operator would have -to hand-edit SQLite to recover. Warning and ignoring means the downgrade works, -the unknown setting sits inert, and the upgrade back picks it up again. +They are different situations. A settings row the running binary does not recognise is almost always an older binary reading a database written by a newer one — a downgrade, or a rollback after a bad upgrade. Refusing to start there would mean a downgrade bricks the config database, and the operator would have to hand-edit SQLite to recover. Warning and ignoring means the downgrade works, the unknown setting sits inert, and the upgrade back picks it up again. -A key in a file, by contrast, is something a human just typed. The likeliest -cause is a typo, and the second likeliest is a field that no longer exists. -Silently ignoring it would mean the setting the operator believes they applied -was never applied — exactly the silent-divergence failure the whole model is -built to avoid. So the ZON parser rejects unknown fields with a line and column. +A key in a file, by contrast, is something a human just typed. The likeliest cause is a typo, and the second likeliest is a field that no longer exists. Silently ignoring it would mean the setting the operator believes they applied was never applied — exactly the silent-divergence failure the whole model is built to avoid. So the ZON parser rejects unknown fields with a line and column. -The tolerance has a boundary worth stating plainly: it covers unknown *keys*, -not unparseable *values*. A known key whose stored text does not decode into its -type is an error, not a warning. +The tolerance has a boundary worth stating plainly: it covers unknown *keys*, not unparseable *values*. A known key whose stored text does not decode into its type is an error, not a warning. ## The password -`web.password` is a write-only input. It is never a stored value. A non-empty -one is hashed with argon2id (PHC encoding, OWASP argon2id parameters) into -`web.password_hash`, and the plaintext is cleared before anything is written. -There is no settings row that can hold it: the model skips `web.password` in -both directions of the settings bridge, so the plaintext has nowhere to go even -by accident. +`web.password` is a write-only input. It is never a stored value. A non-empty one is hashed with argon2id (PHC encoding, OWASP argon2id parameters) into `web.password_hash`, and the plaintext is cleared before anything is written. There is no settings row that can hold it: the model skips `web.password` in both directions of the settings bridge, so the plaintext has nowhere to go even by accident. -Both fields are optional, and this is the one deliberate carve-out from -file-as-sole-truth: **a file that mentions neither leaves the stored hash -alone.** +Both fields are optional, and this is the one deliberate carve-out from file-as-sole-truth: **a file that mentions neither leaves the stored hash alone.** -The reason is a trap the design walked into once. An export carries the full PHC -string, which is long and ugly, and an operator committing that file to git will -sooner or later delete the line — meaning "keep the current password". If -absence meant "no password", that edit would reconcile an empty hash over the -stored one and open the admin UI to the entire LAN, silently, because -authentication is on exactly when the hash is non-empty. Silence has to mean -keep. Disabling authentication takes the explicit `password_hash = ""`. +The reason is a trap the design walked into once. An export carries the full PHC string, which is long and ugly, and an operator committing that file to git will sooner or later delete the line — meaning "keep the current password". If absence meant "no password", that edit would reconcile an empty hash over the stored one and open the admin UI to the entire LAN, silently, because authentication is on exactly when the hash is non-empty. Silence has to mean keep. Disabling authentication takes the explicit `password_hash = ""`. -The other end of the same problem is `password = ""`. Hashing the empty string -produces a perfectly valid hash, so authentication would be *on* — while the -login handler refuses every empty password, so it could never be satisfied. -Auth on and unreachable is worse than either alternative, so that file is -refused at validation with a diagnostic naming the remedy. +The other end of the same problem is `password = ""`. Hashing the empty string produces a perfectly valid hash, so authentication would be *on* — while the login handler refuses every empty password, so it could never be satisfied. Auth on and unreachable is worse than either alternative, so that file is refused at validation with a diagnostic naming the remedy. -A plaintext password that has not changed is verified against the stored hash -and kept rather than re-hashed. That is byte-stability, not a saving: -verification recomputes the same argon2id function with the stored salt and -costs exactly what hashing costs. Hashing unconditionally would generate a fresh -salt on every start and break the invariant above. +A plaintext password that has not changed is verified against the stored hash and kept rather than re-hashed. That is byte-stability, not a saving: verification recomputes the same argon2id function with the stored salt and costs exactly what hashing costs. Hashing unconditionally would generate a fresh salt on every start and break the invariant above. -Export's canonical form is therefore `password = null` beside the stored -`password_hash`. Writing an empty *string* there instead would make every export -carry a present-but-empty password next to a hash — tripping the both-set rule -on re-import, so export's own output would fail export's own contract. +Export's canonical form is therefore `password = null` beside the stored `password_hash`. Writing an empty *string* there instead would make every export carry a present-but-empty password next to a hash — tripping the both-set rule on re-import, so export's own output would fail export's own contract. ## What is not configuration -Storage paths are process arguments, not configuration fields: `--data-dir`, -`--config`, `--web-dev`. They cannot live in the file, because the file is found -by way of them — a path that told you where to find the thing that told you the -path would be circular. They are also the settings a supervisor (systemd, -Docker) owns rather than the operator's policy about DNS. See -[reference/files-and-directories.md](../reference/files-and-directories.md). +Storage paths are process arguments, not configuration fields: `--data-dir`, `--config`, `--web-dev`. They cannot live in the file, because the file is found by way of them — a path that told you where to find the thing that told you the path would be circular. They are also the settings a supervisor (systemd, Docker) owns rather than the operator's policy about DNS. See [reference/files-and-directories.md](../reference/files-and-directories.md). diff --git a/docs/explanation/performance-and-testing.md b/docs/explanation/performance-and-testing.md index cc296fe..119d752 100644 --- a/docs/explanation/performance-and-testing.md +++ b/docs/explanation/performance-and-testing.md @@ -1,12 +1,8 @@ # Performance targets and what the tests prove -Two related questions: why the performance numbers are the numbers, and why CI -does not enforce them — and then, less comfortably, what a green test suite -here does and does not tell you. +Two related questions: why the performance numbers are the numbers, and why CI does not enforce them — and then, less comfortably, what a green test suite here does and does not tell you. -For the targets and the measured results as data, see -[reference/performance.md](../reference/performance.md); to run the bench -yourself, [how-to/measure-performance.md](../how-to/measure-performance.md). +For the targets and the measured results as data, see [reference/performance.md](../reference/performance.md); to run the bench yourself, [how-to/measure-performance.md](../how-to/measure-performance.md). ## Where the targets come from @@ -16,197 +12,75 @@ PLAN §18 sets five: - blocklist lookup p95 < 1 ms; - cached response p95 < 5 ms; - memory with ~1M blocked domains < 100 MiB; -- stripped static binary ≤ 10,485,760 bytes per arch, ≤ 15,728,640 bytes with - the embedded frontend. +- stripped static binary ≤ 10,485,760 bytes per arch, ≤ 15,728,640 bytes with the embedded frontend. -They are household-scale numbers, and they are deliberately unambitious. 100 -qps is far more than a house generates; the point of the target is not speed -but that a Pi 5 with an SD card never becomes the reason the internet feels -broken. The latency targets exist for the same reason: DNS sits in front of -every connection anyone makes, so the failure people notice is not throughput -but a stall. The memory target is what keeps a 1M-entry blocklist from -competing with everything else on a 4 GB board. The binary-size target is -about what a static single-binary deployment is for — if it does not fit on a -constrained box and copy over a slow link in one step, the packaging decision -has not paid for itself. +They are household-scale numbers, and they are deliberately unambitious. 100 qps is far more than a house generates; the point of the target is not speed but that a Pi 5 with an SD card never becomes the reason the internet feels broken. The latency targets exist for the same reason: DNS sits in front of every connection anyone makes, so the failure people notice is not throughput but a stall. The memory target is what keeps a 1M-entry blocklist from competing with everything else on a 4 GB board. The binary-size target is about what a static single-binary deployment is for — if it does not fit on a constrained box and copy over a slow link in one step, the packaging decision has not paid for itself. -`tools/bench.zig` (`zig build bench`) measures the three that are measurable -in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry -snapshot), `cache` (key build plus cache get plus id patch), and `compile` -(the blocklist compiler over a 1M-line body, informational — there is no §18 -target for it because no prior datapoint exists). Memory comes from -`/proc/self/status` VmRSS. The qps target is not in the harness at all: it is -end-to-end against the real binary with a DNS load generator, because a -harness number for "queries per second" would measure the harness. +`tools/bench.zig` (`zig build bench`) measures the three that are measurable in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry snapshot), `cache` (key build plus cache get plus id patch), and `compile` (the blocklist compiler over a 1M-line body, informational — there is no §18 target for it because no prior datapoint exists). Memory comes from `/proc/self/status` VmRSS. The qps target is not in the harness at all: it is end-to-end against the real binary with a DNS load generator, because a harness number for "queries per second" would measure the harness. -The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product, -and `src/tests.zig` aggregates everything shippable. +The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product, and `src/tests.zig` aggregates everything shippable. ## Why CI does not gate on performance -Required CI stays deterministic (AGENTS.md). Latency assertions on shared -runners measure the runner's noisy neighbours; the same commit passes and -fails depending on what else the host is doing. A gate that flakes does not -protect anything — it trains people to re-run the job, and once re-running is -routine, a real regression gets re-run too. The flaky gate is worse than no -gate, because it also consumes the attention a real gate would need. +Required CI stays deterministic (AGENTS.md). Latency assertions on shared runners measure the runner's noisy neighbours; the same commit passes and fails depending on what else the host is doing. A gate that flakes does not protect anything — it trains people to re-run the job, and once re-running is routine, a real regression gets re-run too. The flaky gate is worse than no gate, because it also consumes the attention a real gate would need. -So the bench defaults to informational, and `--assert` — which exits non-zero -on a missed target — exists for hardware you control. Run it on the Pi, where -the numbers describe the machine the software actually has to run on. The -x86_64 development-host numbers in -[reference/performance.md](../reference/performance.md) are a regression -baseline for the machine development happens on, not a claim about the target -platform; a Cortex-A76 is far slower and those numbers do not transfer. +So the bench defaults to informational, and `--assert` — which exits non-zero on a missed target — exists for hardware you control. Run it on the Pi, where the numbers describe the machine the software actually has to run on. The x86_64 development-host numbers in [reference/performance.md](../reference/performance.md) are a regression baseline for the machine development happens on, not a claim about the target platform; a Cortex-A76 is far slower and those numbers do not transfer. -CI does gate on the one performance property that *is* deterministic: binary -size. The `package` job builds the release artifacts and `zig build verify-dist` -asserts both §18 budgets against them. Size is a function of the input, not of -the runner's mood, so it is exactly the kind of thing a shared runner can -measure honestly. +CI does gate on the one performance property that *is* deterministic: binary size. The `package` job builds the release artifacts and `zig build verify-dist` asserts both §18 budgets against them. Size is a function of the input, not of the runner's mood, so it is exactly the kind of thing a shared runner can measure honestly. -The budgets are asserted as exact byte counts, and the asset-free budget gets -its own build against a generated empty assets directory rather than against -`web/dist-placeholder`. The placeholder is not buildable by `dist` at all — -that is the guard against a release shipping a stub admin page — and letting it -back in through a size check would have defeated the guard for the sake of one -number. +The budgets are asserted as exact byte counts, and the asset-free budget gets its own build against a generated empty assets directory rather than against `web/dist-placeholder`. The placeholder is not buildable by `dist` at all — that is the guard against a release shipping a stub admin page — and letting it back in through a size check would have defeated the guard for the sake of one number. ## What the test suite is -Every blocking check lives in `.gitea/workflows/gates.yml`, which is a -`workflow_call` workflow with nothing in it but jobs. `ci.yml` calls it on push -and pull request for `master`, and `release.yml` calls it before it builds -anything publishable. That shape exists for one reason: a check that lived in -`ci.yml` alone would be a check a release could skip. +Every blocking check lives in `.gitea/workflows/gates.yml`, which is a `workflow_call` workflow with nothing in it but jobs. `ci.yml` calls it on push and pull request for `master`, and `release.yml` calls it before it builds anything publishable. That shape exists for one reason: a check that lived in `ci.yml` alone would be a check a release could skip. Five jobs, all required: - `test` — the Zig suite with `-Dintegration`. -- `test-aarch64` — the same suite cross-built for aarch64 and executed under - qemu-user, plain tier only. +- `test-aarch64` — the same suite cross-built for aarch64 and executed under qemu-user, plain tier only. - `frontend` — format, lint, typecheck, the vitest cases, build. -- `package` — `zig build dist` and `zig build verify-dist`, which is where the - size budgets, the ELF static-linkage assert and the archive layout checks - are. -- `container` — builds the image, asserts the binary inside it is byte-identical - to the one in the matching tarball, and smoke-tests it by booting the - container and polling `/api/health`. +- `package` — `zig build dist` and `zig build verify-dist`, which is where the size budgets, the ELF static-linkage assert and the archive layout checks are. +- `container` — builds the image, asserts the binary inside it is byte-identical to the one in the matching tarball, and smoke-tests it by booting the container and polling `/api/health`. The Zig suite has three tiers, gated by build flags: -- **plain `zig build test`** — pure logic. No sockets, no threads, no clock - budgets. This is the tier the purity rule - ([architecture.md](architecture.md)) exists to make possible. -- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:` - databases, temp directories. Nothing leaves the host. -- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT - handshakes against real resolvers). Four tests, and they run in a - manual-dispatch workflow, never on push or pull request. +- **plain `zig build test`** — pure logic. No sockets, no threads, no clock budgets. This is the tier the purity rule ([architecture.md](architecture.md)) exists to make possible. +- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:` databases, temp directories. Nothing leaves the host. +- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT handshakes against real resolvers). Four tests, and they run in a manual-dispatch workflow, never on push or pull request. -The aarch64 job runs the plain tier only. The integration tests are -multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown -turns those budgets into a flake source — the same reasoning that keeps the -bench out of CI. What aarch64 needs to prove is portable correctness of the -DNS, filter and cache logic, and the plain tier is exactly that. +The aarch64 job runs the plain tier only. The integration tests are multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown turns those budgets into a flake source — the same reasoning that keeps the bench out of CI. What aarch64 needs to prove is portable correctness of the DNS, filter and cache logic, and the plain tier is exactly that. -At the time of writing, plain `zig build test` is 1175 of 1288 passing with -113 skipped and 0 failed, the skips being the integration-gated tests. -Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with -`-Dintegration` (the 4 skips are the live-network tests) and 1159 passing -under qemu, 0 failed in each. +At the time of writing, plain `zig build test` is 1175 of 1288 passing with 113 skipped and 0 failed, the skips being the integration-gated tests. Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with `-Dintegration` (the 4 skips are the live-network tests) and 1159 passing under qemu, 0 failed in each. ## What it does not prove -The suite is hermetic by design. That is the right default: it is fast, it is -deterministic, it can gate merges. But hermetic and correct are different -properties, and the gap has already cost this project twice. +The suite is hermetic by design. That is the right default: it is fast, it is deterministic, it can gate merges. But hermetic and correct are different properties, and the gap has already cost this project twice. -**The blocklist download aborted the process on first real use.** The fetcher -constructed the HTTP response reader over `transfer_buf` and then read *into* -that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's -already-buffered bytes into the caller's destination — so source and -destination were the same allocation, and Zig's `@memcpy` requires them not to -overlap. It aborts. +**The blocklist download aborted the process on first real use.** The fetcher constructed the HTTP response reader over `transfer_buf` and then read *into* that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's already-buffered bytes into the caller's destination — so source and destination were the same allocation, and Zig's `@memcpy` requires them not to overlap. It aborts. -The reason no test caught it is precise and instructive. The copy length is -zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is -fine. Bytes only accumulate in the reader's own buffer when a read comes back -short of filling the destination and the loop goes round again — that is, when -the body arrives in more than one stream call. The loopback fixture answers -every request with one small in-memory body that lands in a single read, and -the one over-size test never streams a byte, because the fetcher refuses an -oversized `content-length` on the response head. Every test in the suite was -on the zero-length-memcpy side of the branch. The first real download — a -multi-megabyte list over TLS across the WAN, arriving in many TCP segments — -was on the other side, and took the process down. The fix (commit 35f2324) -streams the body straight into the caller's writer, so the reader's buffer is -never a destination slice, and it came with four regression tests that put a -fully-buffered reader into exactly the state the old code could not survive. +The reason no test caught it is precise and instructive. The copy length is zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is fine. Bytes only accumulate in the reader's own buffer when a read comes back short of filling the destination and the loop goes round again — that is, when the body arrives in more than one stream call. The loopback fixture answers every request with one small in-memory body that lands in a single read, and the one over-size test never streams a byte, because the fetcher refuses an oversized `content-length` on the response head. Every test in the suite was on the zero-length-memcpy side of the branch. The first real download — a multi-megabyte list over TLS across the WAN, arriving in many TCP segments — was on the other side, and took the process down. The fix (commit 35f2324) streams the body straight into the caller's writer, so the reader's buffer is never a destination slice, and it came with four regression tests that put a fully-buffered reader into exactly the state the old code could not survive. -**A stale embedded SPA bundle shipped a settings page that crashed on load, -while 121 web tests passed.** `web/dist/` is gitignored and -`-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The -frontend tests ran against the sources, in jsdom, and were green; the binary -carried an older build. The tests were testing something the artifact did not -contain. +**A stale embedded SPA bundle shipped a settings page that crashed on load, while 121 web tests passed.** `web/dist/` is gitignored and `-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The frontend tests ran against the sources, in jsdom, and were green; the binary carried an older build. The tests were testing something the artifact did not contain. -Note what these two have in common. Neither was a logic bug that a better unit -test would have caught. One lived in the seam between the pure core and its -one I/O edge; the other lived in the seam between two build systems. Hermetic -tests are constructed to exclude exactly those seams — that is what makes them -hermetic. +Note what these two have in common. Neither was a logic bug that a better unit test would have caught. One lived in the seam between the pure core and its one I/O edge; the other lived in the seam between two build systems. Hermetic tests are constructed to exclude exactly those seams — that is what makes them hermetic. ## The lesson, and where it now lives -Green hermetic tests are a floor, not a ceiling. They prove the logic is -consistent with itself. They cannot prove the program works, because the -things they deliberately exclude — real network reads, real TLS, real file -sizes, real build artifacts — are where a program meets reality. +Green hermetic tests are a floor, not a ceiling. They prove the logic is consistent with itself. They cannot prove the program works, because the things they deliberately exclude — real network reads, real TLS, real file sizes, real build artifacts — are where a program meets reality. -The response is not to make CI non-deterministic. It is to require that the -real paths get exercised by a human before work is called done. That is now -ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and -the how-to pages is executed verbatim, on the host, by the session that writes -it, and a command that cannot run there is marked in the page as unverified -with the reason. Documentation written from source-reading alone is how both -of these shipped; documentation that has been run is a second, independent -test suite that exercises precisely the paths the hermetic one skips. +The response is not to make CI non-deterministic. It is to require that the real paths get exercised by a human before work is called done. That is now ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and the how-to pages is executed verbatim, on the host, by the session that writes it, and a command that cannot run there is marked in the page as unverified with the reason. Documentation written from source-reading alone is how both of these shipped; documentation that has been run is a second, independent test suite that exercises precisely the paths the hermetic one skips. Two honest gaps remain, stated so nobody has to rediscover them: -- Nothing in the suite drives a multi-read HTTP body through the fetcher end - to end. The regression tests cover `pumpBody` directly over a pre-buffered - reader; the loopback fixture still sends one small body per connection. -- There is no freshness check on `web/dist`. CI cannot embed a stale bundle, - because the jobs that pass `-Dweb-dist` rebuild the frontend immediately - beforehand. A local build can, and will do it without a warning. +- Nothing in the suite drives a multi-read HTTP body through the fetcher end to end. The regression tests cover `pumpBody` directly over a pre-buffered reader; the loopback fixture still sends one small body per connection. +- There is no freshness check on `web/dist`. CI cannot embed a stale bundle, because the jobs that pass `-Dweb-dist` rebuild the frontend immediately beforehand. A local build can, and will do it without a warning. ## What a signed release does not prove either -The same distinction applies one level out, to the artifacts. A release is -signed, and the signature is worth having: it says the artifact came from this -project's pipeline and reached you unaltered. It does not say the binary was -built from the source in this repository, because the machine that ran the -build also held the signing key. An attacker with that machine produces -something that verifies cleanly and contains whatever they put in it. +The same distinction applies one level out, to the artifacts. A release is signed, and the signature is worth having: it says the artifact came from this project's pipeline and reached you unaltered. It does not say the binary was built from the source in this repository, because the machine that ran the build also held the signing key. An attacker with that machine produces something that verifies cleanly and contains whatever they put in it. -The control that closes that gap is a reproducibility gate — an independent -build, in a different directory on a different machine, landing on the same -bytes. It does not exist. It is a recorded deferral (`specs/milestone-14.md` -ruling 12), not something nobody thought of, and until it exists no document -here describes the build as reproducible: nobody has measured whether it is. -The cheap inputs to reproducibility are already in place — `gzip -n`, -`--mtime=@0`, `LC_ALL=C`, `TZ=UTC`, exact Zig and Node pins — which makes the -gate cheap to add later and proves nothing on its own. +The control that closes that gap is a reproducibility gate — an independent build, in a different directory on a different machine, landing on the same bytes. It does not exist. It is a recorded deferral (`specs/milestone-14.md` ruling 12), not something nobody thought of, and until it exists no document here describes the build as reproducible: nobody has measured whether it is. The cheap inputs to reproducibility are already in place — `gzip -n`, `--mtime=@0`, `LC_ALL=C`, `TZ=UTC`, exact Zig and Node pins — which makes the gate cheap to add later and proves nothing on its own. -What the release pipeline is required to hold to is narrower: two runs of -`zig build dist` on the same commit **in the same directory** produce -byte-identical tarballs. Same-directory determinism is a much weaker property -than reproducibility, and conflating the two is exactly the kind of claim this -page exists to refuse. +What the release pipeline is required to hold to is narrower: two runs of `zig build dist` on the same commit **in the same directory** produce byte-identical tarballs. Same-directory determinism is a much weaker property than reproducibility, and conflating the two is exactly the kind of claim this page exists to refuse. -[Verify a release](../how-to/verify-a-release.md) states the same limits where -an operator will actually meet them, and gives the rebuild-and-compare recipe -with the caveat that a differing hash is not evidence of tampering while this -gap is open. +[Verify a release](../how-to/verify-a-release.md) states the same limits where an operator will actually meet them, and gives the rebuild-and-compare recipe with the caveat that a differing hash is not evidence of tampering while this gap is open. diff --git a/docs/how-to/back-up-and-restore.md b/docs/how-to/back-up-and-restore.md index 0ad9323..2f43800 100644 --- a/docs/how-to/back-up-and-restore.md +++ b/docs/how-to/back-up-and-restore.md @@ -1,27 +1,15 @@ # Back up and restore -The configuration is the only state worth keeping. `nxdns export` writes it out -as a ZON file and `nxdns import` writes one back. The query log is deliberately -not part of a backup: it is expendable history, and if it is missing it gets -recreated empty. +The configuration is the only state worth keeping. `nxdns export` writes it out as a ZON file and `nxdns import` writes one back. The query log is deliberately not part of a backup: it is expendable history, and if it is missing it gets recreated empty. -**Which authority the service runs under decides what the backup *is*.** Check -the start log: +**Which authority the service runs under decides what the backup *is*.** Check the start log: -- `authority: database` — `config.db` holds the configuration. Back it up with - `nxdns export`, and restore with `nxdns import` or by replacing the database - file. -- `authority: file ()` — that file holds the configuration, and it is - already a text file you can keep in git. **The file is the backup.** Restoring - means putting the file back and restarting; the database rebuilds itself from - it. `config.db` is a cache of the file in this mode, not the thing to preserve. +- `authority: database` — `config.db` holds the configuration. Back it up with `nxdns export`, and restore with `nxdns import` or by replacing the database file. +- `authority: file ()` — that file holds the configuration, and it is already a text file you can keep in git. **The file is the backup.** Restoring means putting the file back and restarting; the database rebuilds itself from it. `config.db` is a cache of the file in this mode, not the thing to preserve. The rest of this page covers database mode unless it says otherwise. -The commands below use the scratch lab from -[enable DoH and DoT](enable-doh-and-dot.md), data directory -`/tmp/nxdns-lab/data`. On a real install drop `--data-dir` and the default -`/var/lib/nxdns` applies. +The commands below use the scratch lab from [enable DoH and DoT](enable-doh-and-dot.md), data directory `/tmp/nxdns-lab/data`. On a real install drop `--data-dir` and the default `/var/lib/nxdns` applies. ## Back up @@ -35,8 +23,7 @@ nxdns export --data-dir /tmp/nxdns-lab/data --out /tmp/nxdns-lab/backup.zon wrote /tmp/nxdns-lab/backup.zon ``` -The write is a temp file plus a rename, so an interrupted export leaves no -half-written backup, and the result is mode 0600: +The write is a temp file plus a rename, so an interrupted export leaves no half-written backup, and the result is mode 0600: ```sh stat -c '%a %n' /tmp/nxdns-lab/backup.zon @@ -57,15 +44,9 @@ grep password /tmp/nxdns-lab/backup.zon .password_hash = "$argon2id$v=19$m=19456,t=2,p=1$hOjjnTrTZ6kU8XrwQuJ7ZFC3B5LumaB4hRe7kBbzJ6Q$YsC2rCTDKv96eEwPhs+D6vbDliLogEZph8EkagKSuy8", ``` -Treat backups as secrets. `.password` is always exported as `null` — the -plaintext is never stored anywhere — so the file re-imports without anyone -knowing the password. `null` and `""` are different statements here: `null` -means the file says nothing about the password, while an empty `password_hash` -would disable authentication. See -[Password and hash](../reference/configuration.md#password-and-hash). +Treat backups as secrets. `.password` is always exported as `null` — the plaintext is never stored anywhere — so the file re-imports without anyone knowing the password. `null` and `""` are different statements here: `null` means the file says nothing about the password, while an empty `password_hash` would disable authentication. See [Password and hash](../reference/configuration.md#password-and-hash). -Without `--out` the export goes to stdout, where the file mode is your -redirect's problem: +Without `--out` the export goes to stdout, where the file mode is your redirect's problem: ```sh nxdns export --data-dir /tmp/nxdns-lab/data | head -10 @@ -84,15 +65,11 @@ nxdns export --data-dir /tmp/nxdns-lab/data | head -10 .bind_ipv4 = "127.0.0.1", ``` -Runtime facts are left out on purpose: client first-seen and last-seen times, -rule creation times, per-source domain counts and checksums. They are things a -running server produces, not configuration, and including them would make two -exports taken minutes apart differ. +Runtime facts are left out on purpose: client first-seen and last-seen times, rule creation times, per-source domain counts and checksums. They are things a running server produces, not configuration, and including them would make two exports taken minutes apart differ. ## Restore onto a fresh data directory -This is the normal restore: new machine, new disk, empty data directory. -Nothing to overwrite, so nothing to authorise. +This is the normal restore: new machine, new disk, empty data directory. Nothing to overwrite, so nothing to authorise. ```sh nxdns import /tmp/nxdns-lab/backup.zon --data-dir /tmp/nxdns-lab/data-restored @@ -103,15 +80,11 @@ info(migrations): config.db migrated from schema version 0 to 1 imported /tmp/nxdns-lab/backup.zon ``` -The migration line is expected: `import` creates and migrates the database -before writing to it. +The migration line is expected: `import` creates and migrates the database before writing to it. ## Restore over an existing database -**Stop the server first.** `import` rewrites configuration underneath a process -that read it at startup, and a running server picks up only part of it — -filtering follows the new rows at the next reload, while upstreams, listeners -and settings stay at their boot values until a restart. +**Stop the server first.** `import` rewrites configuration underneath a process that read it at startup, and a running server picks up only part of it — filtering follows the new rows at the next reload, while upstreams, listeners and settings stay at their boot values until a restart. ```sh systemctl stop nxdns @@ -119,10 +92,7 @@ nxdns import /var/backups/nxdns-config.zon systemctl start nxdns ``` -`import` converges the database onto the file. Rows the file still names are -matched and updated in place; rows it no longer names are deleted. That last -part is what a restore of an old backup does to everything added since, so it -takes a flag: +`import` converges the database onto the file. Rows the file still names are matched and updated in place; rows it no longer names are deleted. That last part is what a restore of an old backup does to everything added since, so it takes a flag: ```sh nxdns import /tmp/nxdns-lab/old-backup.zon --data-dir /tmp/nxdns-lab/data @@ -133,10 +103,7 @@ FAIL import: this file would delete rows the database holds (upstreams 1); re-ru import failed: DestructiveImport ``` -That exits 2 and rolls the transaction back, so nothing is half-applied. The -message names every table that would lose rows, which is usually enough to tell -an intended restore from the wrong file. Say `--allow-delete` when deleting is -what you mean: +That exits 2 and rolls the transaction back, so nothing is half-applied. The message names every table that would lose rows, which is usually enough to tell an intended restore from the wrong file. Say `--allow-delete` when deleting is what you mean: ```sh nxdns import /tmp/nxdns-lab/old-backup.zon --allow-delete --data-dir /tmp/nxdns-lab/data @@ -146,20 +113,9 @@ nxdns import /tmp/nxdns-lab/old-backup.zon --allow-delete --data-dir /tmp/nxdns- imported /tmp/nxdns-lab/old-backup.zon ``` -A restore that only puts back what is already there needs no flag at all, and -neither does one that only adds rows. The flag is about deletion specifically — -including deletion in disguise: renaming a group, or correcting a typo in an -upstream URL, changes the row's identity, so the engine sees one row gone and -one arrived. +A restore that only puts back what is already there needs no flag at all, and neither does one that only adds rows. The flag is about deletion specifically — including deletion in disguise: renaming a group, or correcting a typo in an upstream URL, changes the row's identity, so the engine sees one row gone and one arrived. -What survives a restore is worth knowing before you take an old backup out of -the drawer. Blocklist download state is kept for every source whose URL the file -still names — checksum, counters, compiled files, and the row id they are keyed -by — so restoring does not cost a re-download. Devices the server discovered -from traffic are kept whole, and one the backup names keeps the first-seen and -last-seen the database already held, so a restore never restamps your network as -newly arrived. What does go is a client the backup does not name and that was -named by hand: that row is declarative, and it is deleted with the rest. +What survives a restore is worth knowing before you take an old backup out of the drawer. Blocklist download state is kept for every source whose URL the file still names — checksum, counters, compiled files, and the row id they are keyed by — so restoring does not cost a re-download. Devices the server discovered from traffic are kept whole, and one the backup names keeps the first-seen and last-seen the database already held, so a restore never restamps your network as newly arrived. What does go is a client the backup does not name and that was named by hand: that row is declarative, and it is deleted with the rest. > The three `systemctl` lines are the only commands on this page that were not > run: this machine has no installed nxdns unit (`systemctl status nxdns` @@ -169,17 +125,10 @@ named by hand: that row is declarative, and it is deleted with the rest. ## Restoring the database file itself -Copying `config.db` back into place works too, and it is the fastest restore on -a machine that still has one. Two rules, and the second one is where restores go -wrong: +Copying `config.db` back into place works too, and it is the fastest restore on a machine that still has one. Two rules, and the second one is where restores go wrong: -1. **Take the copy from a stopped instance**, or use `nxdns export` instead. A - copy taken while nxdns is running catches the main file without the changes - sitting in its write-ahead log. -2. **Delete any stale `config.db-wal` and `config.db-shm` beside the file you - restore.** SQLite silently discards a write-ahead log that does not match the - database it sits next to. It does not warn, and it does not fail — it just - answers from the main file, so a restore quietly loses its own tail. +1. **Take the copy from a stopped instance**, or use `nxdns export` instead. A copy taken while nxdns is running catches the main file without the changes sitting in its write-ahead log. +2. **Delete any stale `config.db-wal` and `config.db-shm` beside the file you restore.** SQLite silently discards a write-ahead log that does not match the database it sits next to. It does not warn, and it does not fail — it just answers from the main file, so a restore quietly loses its own tail. ```sh systemctl stop nxdns @@ -193,15 +142,11 @@ systemctl start nxdns > Not verified on this host: these need root, an installed unit and > `/var/lib/nxdns`, none of which exist here. -None of this applies in file mode. There `config.db` is derived state — restore -the configuration file and start the service, and the first reconcile rebuilds -the database from it. +None of this applies in file mode. There `config.db` is derived state — restore the configuration file and start the service, and the first reconcile rebuilds the database from it. ## Verify a backup -The round trip is byte-stable: exporting, importing and exporting again gives -an identical file. That is the cheapest check that a backup is complete and -that it will load. +The round trip is byte-stable: exporting, importing and exporting again gives an identical file. That is the cheapest check that a backup is complete and that it will load. ```sh nxdns export --data-dir /tmp/nxdns-lab/data --out /tmp/nxdns-lab/backup2.zon @@ -214,8 +159,7 @@ wrote /tmp/nxdns-lab/backup2.zon round trip is byte-identical ``` -The same check works across data directories — export from the restored copy -and diff against the backup you restored from: +The same check works across data directories — export from the restored copy and diff against the backup you restored from: ```sh nxdns export --data-dir /tmp/nxdns-lab/data-restored --out /tmp/nxdns-lab/backup3.zon @@ -233,20 +177,12 @@ restored database exports identically Everything under the data directory other than `config.db`: - `querylog.db` — expendable history, recreated empty when absent. -- `blocklists/` — compiled snapshots. They are rebuilt from the sources named in - the configuration, so restoring the configuration is enough; the first refresh - after a restore downloads them again. +- `blocklists/` — compiled snapshots. They are rebuilt from the sources named in the configuration, so restoring the configuration is enough; the first refresh after a restore downloads them again. -The data directory layout is in -[files and directories](../reference/files-and-directories.md). +The data directory layout is in [files and directories](../reference/files-and-directories.md). ## A backup before every upgrade -There is no downgrade path. Schema migrations run forward automatically at -startup and before `export` and `import`; nothing walks them back, and `nxdns -check` does not run them at all. Take an export before installing a new binary — -see [upgrade](upgrade.md). +There is no downgrade path. Schema migrations run forward automatically at startup and before `export` and `import`; nothing walks them back, and `nxdns check` does not run them at all. Take an export before installing a new binary — see [upgrade](upgrade.md). -Every `nxdns` command on this page was executed on this host as written. The -`systemctl`, `cp`, `chown` and `chmod` lines were not: they need root and an -installed unit, and both blocks holding them say so. +Every `nxdns` command on this page was executed on this host as written. The `systemctl`, `cp`, `chown` and `chmod` lines were not: they need root and an installed unit, and both blocks holding them say so. diff --git a/docs/how-to/enable-doh-and-dot.md b/docs/how-to/enable-doh-and-dot.md index 69f9cbd..88851b6 100644 --- a/docs/how-to/enable-doh-and-dot.md +++ b/docs/how-to/enable-doh-and-dot.md @@ -1,22 +1,14 @@ # Enable DoH and DoT -nxdns can answer encrypted queries on two extra listeners: DNS over HTTPS -(`doh_server`) and DNS over TLS (`dot_server`). Both are off by default and both -need a certificate and a private key in PEM form. +nxdns can answer encrypted queries on two extra listeners: DNS over HTTPS (`doh_server`) and DNS over TLS (`dot_server`). Both are off by default and both need a certificate and a private key in PEM form. -This page uses a scratch lab under `/tmp/nxdns-lab` so the commands run without -root and without touching a real install. On a real install the files live under -`/etc/nxdns` and the data directory is `/var/lib/nxdns`; the ports are 443 and -853 rather than the unprivileged ones below. +This page uses a scratch lab under `/tmp/nxdns-lab` so the commands run without root and without touching a real install. On a real install the files live under `/etc/nxdns` and the data directory is `/var/lib/nxdns`; the ports are 443 and 853 rather than the unprivileged ones below. -Every field mentioned here is documented in -[configuration reference](../reference/configuration.md). +Every field mentioned here is documented in [configuration reference](../reference/configuration.md). ## 1. Get a certificate and key -For a LAN service the practical options are a certificate from your ACME client -(certbot, lego, caddy) for a name you control, or a self-signed pair. The lab -below uses a self-signed pair, because it needs no domain: +For a LAN service the practical options are a certificate from your ACME client (certbot, lego, caddy) for a name you control, or a self-signed pair. The lab below uses a self-signed pair, because it needs no domain: ```sh mkdir -p /tmp/nxdns-lab/etc @@ -28,16 +20,11 @@ openssl req -x509 -newkey rsa:2048 -nodes \ chmod 0600 etc/key.pem ``` -A self-signed certificate means every client has to be told to trust it, or told -to skip verification. That is why the client commands further down pass -`--insecure` and `+tls` without a CA. A real deployment uses a real certificate -and drops those flags. +A self-signed certificate means every client has to be told to trust it, or told to skip verification. That is why the client commands further down pass `--insecure` and `+tls` without a CA. A real deployment uses a real certificate and drops those flags. ## 2. Turn the listeners on -`cert_path` and `key_path` must be absolute, or relative to the process working -directory. Point both endpoints at the same pair unless you have a reason not -to: +`cert_path` and `key_path` must be absolute, or relative to the process working directory. Point both endpoints at the same pair unless you have a reason not to: ```zon .{ @@ -62,12 +49,7 @@ to: } ``` -Write that to `/tmp/nxdns-lab/etc/config.zon`. The lab runs -`nxdns run --config`, which makes that file the configuration: every start -reconciles the database onto it, so editing the file and restarting is how these -settings change here. A real install may instead run bare `nxdns run` and keep -the configuration in the database — see -[the configuration model](../explanation/configuration-model.md). +Write that to `/tmp/nxdns-lab/etc/config.zon`. The lab runs `nxdns run --config`, which makes that file the configuration: every start reconciles the database onto it, so editing the file and restarting is how these settings change here. A real install may instead run bare `nxdns run` and keep the configuration in the database — see [the configuration model](../explanation/configuration-model.md). ## 3. Check the files before starting @@ -81,17 +63,14 @@ OK upstreams[0] https://cloudflare-dns.com OK: no problems found ``` -`check` loads both endpoints' certificate and key the same way the listeners do, -so what passes here will start. An unreadable file is a failure and exits 2: +`check` loads both endpoints' certificate and key the same way the listeners do, so what passes here will start. An unreadable file is a failure and exits 2: ``` FAIL doh_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable FAIL dot_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable ``` -So is a key that does not belong to the certificate, which is the mistake worth -catching before a restart — the two files are individually valid and only their -pairing is wrong. mbedTLS writes its own line to stderr as it rejects the pair: +So is a key that does not belong to the certificate, which is the mistake worth catching before a restart — the two files are individually valid and only their pairing is wrong. mbedTLS writes its own line to stderr as it rejects the pair: ``` warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896) @@ -102,9 +81,7 @@ FAIL dot_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not b OK upstreams[0] https://cloudflare-dns.com ``` -A key readable by anyone but its owner is a warning instead. It does not change -the exit code, because the service still starts, and the summary line counts it -rather than claiming nothing was found: +A key readable by anyone but its owner is a warning instead. It does not change the exit code, because the service still starts, and the summary line counts it rather than claiming nothing was found: ``` WARN doh_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only @@ -126,9 +103,7 @@ info(nxdns): doh listener on 127.0.0.1:8443 info(nxdns): dot listener on 127.0.0.1:8853 ``` -If a certificate cannot be loaded while its endpoint is enabled, nxdns refuses -to start and exits 2 rather than serving DNS without the listener you asked -for: +If a certificate cannot be loaded while its endpoint is enabled, nxdns refuses to start and exits 2 rather than serving DNS without the listener you asked for: ``` doh_server: '/tmp/nxdns-lab/etc/cert.pem' + '/tmp/nxdns-lab/etc/key.pem': private key file is not readable @@ -138,8 +113,7 @@ run `nxdns check` to see the configuration in full ## 5. Query DoT -Recent `dig` speaks DNS over TLS with `+tls` (this page was checked with BIND -9.20.26): +Recent `dig` speaks DNS over TLS with `+tls` (this page was checked with BIND 9.20.26): ```sh dig @127.0.0.1 -p 8853 +tls example.com A +short @@ -152,13 +126,9 @@ dig @127.0.0.1 -p 8853 +tls example.com A +short ## 6. Query DoH -The only path the DoH listener serves is `/dns-query`; anything else is a 404. -It accepts both the POST form (the query as an `application/dns-message` body) -and the GET form (`?dns=` with base64url of the same bytes). +The only path the DoH listener serves is `/dns-query`; anything else is a 404. It accepts both the POST form (the query as an `application/dns-message` body) and the GET form (`?dns=` with base64url of the same bytes). -The body is a raw DNS query in wire format. Build one for `example.com A` — -header with the recursion-desired bit, one question, then the QNAME as -length-prefixed labels: +The body is a raw DNS query in wire format. Build one for `example.com A` — header with the recursion-desired bit, one question, then the QNAME as length-prefixed labels: ```sh printf '%s' '000001000001000000000000076578616d706c6503636f6d0000010001' \ @@ -186,8 +156,7 @@ http 200, 72 bytes 00000040: 04d0 0000 0000 0000 ........ ``` -The second flag byte `80` and the third answer-count field `0002` say: response, -no error, two answer records. +The second flag byte `80` and the third answer-count field `0002` say: response, no error, two answer records. The GET form takes the same bytes, base64url-encoded with the padding removed: @@ -202,16 +171,11 @@ curl -sS --insecure --http1.1 -o /tmp/nxdns-lab/get.bin \ GET http 200, 61 bytes ``` -`--http1.1` matters: without it curl offers HTTP/2 over ALPN, and the DoH -listener negotiates only what it advertises. `--insecure` is only needed for the -self-signed lab certificate. +`--http1.1` matters: without it curl offers HTTP/2 over ALPN, and the DoH listener negotiates only what it advertises. `--insecure` is only needed for the self-signed lab certificate. ## 7. Renewals -A watcher polls both files every 30 seconds and compares their modification time -and size against the pair currently loaded. When either differs it reloads and -swaps the new pair in; connections already open finish on the old certificate. -Nothing has to restart. +A watcher polls both files every 30 seconds and compares their modification time and size against the pair currently loaded. When either differs it reloads and swaps the new pair in; connections already open finish on the old certificate. Nothing has to restart. Replace the pair and wait one poll interval: @@ -232,10 +196,7 @@ info(cert_store): certificate reloaded from /tmp/nxdns-lab/etc/cert.pem ## 8. Reload immediately -To skip the wait — from an ACME deploy hook, for example — call -`POST /api/certs/reload`. It needs a session; see -[set up admin authentication](set-up-admin-authentication.md) for the login -call that fills `cookies.txt`. +To skip the wait — from an ACME deploy hook, for example — call `POST /api/certs/reload`. It needs a session; see [set up admin authentication](set-up-admin-authentication.md) for the login call that fills `cookies.txt`. ```sh curl -sS -b /tmp/nxdns-lab/cookies.txt -X POST http://127.0.0.1:8451/api/certs/reload @@ -245,9 +206,7 @@ curl -sS -b /tmp/nxdns-lab/cookies.txt -X POST http://127.0.0.1:8451/api/certs/r {"doh":{"enabled":true,"reloaded":true,"error":null},"dot":{"enabled":true,"reloaded":true,"error":null}} ``` -The route always answers 200: the per-endpoint outcome is the payload, not the -status code. A disabled endpoint reports `"enabled":false`. A reload that fails -names the reason and leaves the old certificate serving: +The route always answers 200: the per-endpoint outcome is the payload, not the status code. A disabled endpoint reports `"enabled":false`. A reload that fails names the reason and leaves the old certificate serving: ```sh chmod 000 /tmp/nxdns-lab/etc/key.pem @@ -269,8 +228,7 @@ dig @127.0.0.1 -p 8853 +tls example.com A +short 172.66.147.243 ``` -Undo it with `chmod 0600 /tmp/nxdns-lab/etc/key.pem` and reload again. The -counters are on `/metrics`: +Undo it with `chmod 0600 /tmp/nxdns-lab/etc/key.pem` and reload again. The counters are on `/metrics`: ``` nxdns_cert_reloads_total{endpoint="doh"} 3 @@ -291,10 +249,7 @@ chmod 0644 /etc/nxdns/cert.pem chmod 0600 /etc/nxdns/key.pem ``` -Under Docker the container runs as uid 65532, fixed in the image, and -`/etc/nxdns` is a read-only bind mount — the container cannot fix permissions -itself, so the host-side files must already be readable by that uid. It has no -name on the host, so chown it numerically: +Under Docker the container runs as uid 65532, fixed in the image, and `/etc/nxdns` is a read-only bind mount — the container cannot fix permissions itself, so the host-side files must already be readable by that uid. It has no name on the host, so chown it numerically: ```sh cd deploy/docker @@ -303,13 +258,8 @@ chmod 0644 etc-nxdns/cert.pem chmod 0600 etc-nxdns/key.pem ``` -**Not verified on this host:** the two `chown` blocks above. Both need root, and -the systemd one needs an `nxdns` user this development machine does not have. -Everything else on this page was executed as written. +**Not verified on this host:** the two `chown` blocks above. Both need root, and the systemd one needs an `nxdns` user this development machine does not have. Everything else on this page was executed as written. ## Ports 443 and 853 -The defaults are the standard ports, which are privileged. Under the packaged -systemd unit that is already handled: it grants `CAP_NET_BIND_SERVICE` for port -53 and the same capability covers 443 and 853. See -[install with systemd](install-with-systemd.md). +The defaults are the standard ports, which are privileged. Under the packaged systemd unit that is already handled: it grants `CAP_NET_BIND_SERVICE` for port 53 and the same capability covers 443 and 853. See [install with systemd](install-with-systemd.md). diff --git a/docs/how-to/install-with-docker.md b/docs/how-to/install-with-docker.md index cbc7aaf..17c9006 100644 --- a/docs/how-to/install-with-docker.md +++ b/docs/how-to/install-with-docker.md @@ -1,14 +1,10 @@ # Install nxdns with Docker -Runs the published nxdns image with Docker Compose. At the end a container -answers DNS on port 53 and keeps its data in a named volume. +Runs the published nxdns image with Docker Compose. At the end a container answers DNS on port 53 and keeps its data in a named volume. -The image is multi-architecture — `linux/amd64` and `linux/arm64` — so the same -tag works on a PC and on a Raspberry Pi 5. Building the image yourself is still -supported and is the last section of this page. +The image is multi-architecture — `linux/amd64` and `linux/arm64` — so the same tag works on a PC and on a Raspberry Pi 5. Building the image yourself is still supported and is the last section of this page. -For what each configuration field means, see -[the configuration reference](../reference/configuration.md). +For what each configuration field means, see [the configuration reference](../reference/configuration.md). > Verification: the failure modes, the run and the two checks in step 3 were run > on the machine that wrote an earlier revision of this page, against an image @@ -43,16 +39,9 @@ VERSION=$(curl -fsS -o /dev/null -w '%{redirect_url}' "$BASE/releases/latest" | docker pull git.mial.net/mokhtar/nxdns:$VERSION ``` -Pin a version. `:latest` exists and moves, which is what you want when you are -trying it out and not what you want on a machine your household's DNS depends -on. The lookup above asks the server for the current release rather than -hardcoding a number that goes stale one release later — Gitea redirects -`releases/latest` to the newest published release's tag page. To take a -particular version instead, set `VERSION=` yourself. +Pin a version. `:latest` exists and moves, which is what you want when you are trying it out and not what you want on a machine your household's DNS depends on. The lookup above asks the server for the current release rather than hardcoding a number that goes stale one release later — Gitea redirects `releases/latest` to the newest published release's tag page. To take a particular version instead, set `VERSION=` yourself. -Verify what you pulled before you run it. The release publishes an -`IMAGE-DIGEST.txt` asset naming the digest of the image index, and that file is -covered by the signed `SHA256SUMS.txt`: +Verify what you pulled before you run it. The release publishes an `IMAGE-DIGEST.txt` asset naming the digest of the image index, and that file is covered by the signed `SHA256SUMS.txt`: ```sh curl -fLO "$BASE/releases/download/v$VERSION/IMAGE-DIGEST.txt" @@ -65,11 +54,7 @@ docker buildx imagetools inspect git.mial.net/mokhtar/nxdns:$VERSION \ cut -d@ -f2 IMAGE-DIGEST.txt ``` -The last two have to print the same string — `IMAGE-DIGEST.txt` holds a whole -pinned reference, `name:tag@sha256:…`, so the `cut` is what reduces it to the -digest `imagetools` prints. [Verify a release](verify-a-release.md) -covers the key, the fingerprint, every failure message, and what the signature -does and does not prove. +The last two have to print the same string — `IMAGE-DIGEST.txt` holds a whole pinned reference, `name:tag@sha256:…`, so the `cut` is what reduces it to the digest `imagetools` prints. [Verify a release](verify-a-release.md) covers the key, the fingerprint, every failure message, and what the signature does and does not prove. > Not verified on this host: no image and no release are published yet, so > `docker pull` and every URL here fail today, and the `releases/latest` lookup @@ -80,10 +65,7 @@ does and does not prove. ## 2. Get the compose file and write the configuration -Every path on this page is relative to a checkout of the repository, because -that is how it was verified. Running the published image needs no checkout, -though — one file is enough. Fetch it for the version you pulled and work in -its directory instead, dropping `deploy/docker/` from the paths below: +Every path on this page is relative to a checkout of the repository, because that is how it was verified. Running the published image needs no checkout, though — one file is enough. Fetch it for the version you pulled and work in its directory instead, dropping `deploy/docker/` from the paths below: ```sh mkdir -p ~/nxdns && cd ~/nxdns @@ -94,16 +76,14 @@ curl -fLO "$BASE/raw/tag/v$VERSION/deploy/docker/compose.yaml" > `/raw/tag//` was run here against `gitea.com/gitea/tea` on > Gitea `1.27.0+dev` and returned the file with a 200. -Compose bind-mounts `deploy/docker/etc-nxdns` read-only at `/etc/nxdns`. Create -it and put the configuration in it: +Compose bind-mounts `deploy/docker/etc-nxdns` read-only at `/etc/nxdns`. Create it and put the configuration in it: ```sh mkdir -p deploy/docker/etc-nxdns $EDITOR deploy/docker/etc-nxdns/config.zon ``` -The smallest file that starts is one group named `default` and one enabled -upstream: +The smallest file that starts is one group named `default` and one enabled upstream: ```zon .{ @@ -113,15 +93,9 @@ upstream: } ``` -**The compose file ships file mode**, with -`command: ["run", "--config=/etc/nxdns/config.zon"]`. That file is the -configuration: the container reconciles its database onto it at every start, and -the admin interface answers 403 to configuration edits. To change anything, edit -the file and restart the container. It also means a fresh or recreated -`nxdns-data` volume rebuilds itself from the mounted file with no extra step. +**The compose file ships file mode**, with `command: ["run", "--config=/etc/nxdns/config.zon"]`. That file is the configuration: the container reconciles its database onto it at every start, and the admin interface answers 403 to configuration edits. To change anything, edit the file and restart the container. It also means a fresh or recreated `nxdns-data` volume rebuilds itself from the mounted file with no extra step. -The file is therefore required, and its absence is a hard failure rather than a -start with defaults: +The file is therefore required, and its absence is a hard failure rather than a start with defaults: ``` FAIL /etc/nxdns/config.zon: no such file @@ -129,9 +103,7 @@ nxdns run failed: ManagedConfigUnreadable run `nxdns check` to see the configuration in full ``` -A file that is present but rejected is a different failure with the same exit -code. No `default` group, no enabled upstream, a syntax error — `run` prints the -diagnostic and exits 2 as well. A file whose only group was named `other`: +A file that is present but rejected is a different failure with the same exit code. No `default` group, no enabled upstream, a syntax error — `run` prints the diagnostic and exits 2 as well. A file whose only group was named `other`: ``` FAIL groups: no group named 'default'; every unknown client is assigned to it @@ -139,24 +111,17 @@ nxdns run failed: MissingDefaultGroup run `nxdns check` to see the configuration in full ``` -Under `restart: unless-stopped` any of these is a restart loop — Docker has no -start limit and will retry forever. Read the lines above the failure, which name -the fault. See [Troubleshoot nxdns](troubleshoot.md). +Under `restart: unless-stopped` any of these is a restart loop — Docker has no start limit and will retry forever. Read the lines above the failure, which name the fault. See [Troubleshoot nxdns](troubleshoot.md). ### Database mode in Docker instead -Drop the `command:` line from `compose.yaml` and the container runs -`nxdns run`, with the database as the configuration and the file read by nothing. -On a fresh volume that database is empty and the container exits 2 with -`NoUsableUpstreams`, so load it once before bringing the service up: +Drop the `command:` line from `compose.yaml` and the container runs `nxdns run`, with the database as the configuration and the file read by nothing. On a fresh volume that database is empty and the container exits 2 with `NoUsableUpstreams`, so load it once before bringing the service up: ```sh docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/config.zon ``` -The file is positional; add `--allow-delete` when re-running it against a -populated volume and the diff deletes rows. Without this step, `restart: -unless-stopped` plus exit 2 is a crash loop with no way out. +The file is positional; add `--allow-delete` when re-running it against a populated volume and the diff deletes rows. Without this step, `restart: unless-stopped` plus exit 2 is a crash loop with no way out. > Not run in a container on this host, for the reason in the verification note > at the top: no image could be staged here. The `nxdns import ` and @@ -165,10 +130,7 @@ unless-stopped` plus exit 2 is a crash loop with no way out. > exited 0, the second was required after a plain `import` refused a > row-deleting file with `DestructiveImport` and exited 2. -The container runs as uid 65532, and the mount is read-only, so the container -cannot repair permissions itself. Mode 0644 works and was used here. If the -file carries a secret — `web.password`, or a `web.password_hash` from a -restored export — give it to that uid instead: +The container runs as uid 65532, and the mount is read-only, so the container cannot repair permissions itself. Mode 0644 works and was used here. If the file carries a secret — `web.password`, or a `web.password_hash` from a restored export — give it to that uid instead: ```sh chown 65532:65532 deploy/docker/etc-nxdns/config.zon @@ -190,10 +152,7 @@ NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml up -d docker compose -f deploy/docker/compose.yaml logs -f ``` -`compose.yaml` reads the image from two variables: -`${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-latest}}`. Set -`NXDNS_VERSION` to pin a release; set `NXDNS_IMAGE` to run something else -entirely, which is what the build-from-source section at the bottom does. +`compose.yaml` reads the image from two variables: `${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-latest}}`. Set `NXDNS_VERSION` to pin a release; set `NXDNS_IMAGE` to run something else entirely, which is what the build-from-source section at the bottom does. > Verified on this host with `docker compose -f deploy/docker/compose.yaml > config`, which resolves the variables without contacting a registry: no @@ -201,12 +160,7 @@ entirely, which is what the build-from-source section at the bottom does. > gives `git.mial.net/mokhtar/nxdns:0.0.1`, and `NXDNS_IMAGE=nxdns` gives > `nxdns`. -Every block on this page runs from the repository root, and none of them change -directory, so they can be pasted in order. `-f` is what makes that work: -Compose resolves the relative paths inside `compose.yaml` — the `etc-nxdns` -bind mount — against the directory holding the file, not against your shell, -and it takes the project name `docker` from that directory either way, which is -why the container is `docker-nxdns-1`. +Every block on this page runs from the repository root, and none of them change directory, so they can be pasted in order. `-f` is what makes that work: Compose resolves the relative paths inside `compose.yaml` — the `etc-nxdns` bind mount — against the directory holding the file, not against your shell, and it takes the project name `docker` from that directory either way, which is why the container is `docker-nxdns-1`. A healthy first start logs the reconcile, the authority and the bound sockets: @@ -220,8 +174,7 @@ info(nxdns): nxdns serving on udp [::]:53 tcp [::]:53 tcp 0.0.0.0:53; info(web_server): web interface listening on 0.0.0.0:8080 ``` -Every later start on an unchanged file reports `reconciled -'/etc/nxdns/config.zon': no changes` and writes nothing to the database. +Every later start on an unchanged file reports `reconciled '/etc/nxdns/config.zon': no changes` and writes nothing to the database. Confirm it answers and that the admin interface is up: @@ -240,29 +193,17 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/ > host port is occupied; free the port or edit the `ports:` list. The image > under test was built locally, not pulled: there is nothing published to pull. -The compose file publishes 53/udp, 53/tcp and 8080, keeps `/var/lib/nxdns` in -the named volume `nxdns-data`, and sets the per-namespace sysctl -`net.ipv4.ip_unprivileged_port_start=0` so uid 65532 can bind port 53 without -any capability. Uncomment the 443 and 853 mappings when you enable the DoH or -DoT listener; see [Enable DoH and DoT](enable-doh-and-dot.md). +The compose file publishes 53/udp, 53/tcp and 8080, keeps `/var/lib/nxdns` in the named volume `nxdns-data`, and sets the per-namespace sysctl `net.ipv4.ip_unprivileged_port_start=0` so uid 65532 can bind port 53 without any capability. Uncomment the 443 and 853 mappings when you enable the DoH or DoT listener; see [Enable DoH and DoT](enable-doh-and-dot.md). ## 4. Do not point the host at the container -The container resolves its own upstream DoH and DoT hostnames through the -host's DNS configuration. If you set the host's `/etc/resolv.conf` to the nxdns -container, the container's startup lookups depend on the service that is trying -to start. Point LAN clients at nxdns; leave the container's host on its own -resolver. +The container resolves its own upstream DoH and DoT hostnames through the host's DNS configuration. If you set the host's `/etc/resolv.conf` to the nxdns container, the container's startup lookups depend on the service that is trying to start. Point LAN clients at nxdns; leave the container's host on its own resolver. ## Raspberry Pi 5 -Nothing changes. The published tag is a multi-architecture index, so -`docker pull` on the Pi selects the `linux/arm64` image on its own. The -platform list is one of the things -[Verify a release](verify-a-release.md) has you check. +Nothing changes. The published tag is a multi-architecture index, so `docker pull` on the Pi selects the `linux/arm64` image on its own. The platform list is one of the things [Verify a release](verify-a-release.md) has you check. -To pull the arm64 image from an x86_64 machine — to inspect it, or to save and -copy it — name the platform: +To pull the arm64 image from an x86_64 machine — to inspect it, or to save and copy it — name the platform: ```sh docker pull --platform linux/arm64 git.mial.net/mokhtar/nxdns:$VERSION @@ -274,9 +215,7 @@ docker pull --platform linux/arm64 git.mial.net/mokhtar/nxdns:$VERSION ## Build the image from source instead -The Dockerfile does not compile anything. It assembles a filesystem around -binaries you build first, so build the admin interface and the release -artifacts from the repository root: +The Dockerfile does not compile anything. It assembles a filesystem around binaries you build first, so build the admin interface and the release artifacts from the repository root: ```sh (cd web && npm ci && npm run build) @@ -286,19 +225,11 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" DOCKER_BUILDKIT=1 docker build -t nxdns -f deploy/docker/Dockerfile . ``` -Take the version from `build.zig.zon` rather than inventing one: `verify-dist` -asserts the two agree, so a made-up string builds but fails verification. -BuildKit is required — the Dockerfile pins its builder stage to -`$BUILDPLATFORM`, which the classic builder does not define. +Take the version from `build.zig.zon` rather than inventing one: `verify-dist` asserts the two agree, so a made-up string builds but fails verification. BuildKit is required — the Dockerfile pins its builder stage to `$BUILDPLATFORM`, which the classic builder does not define. -Build `web/dist` every time, before the binaries. A stale bundle is embedded -silently and ships an admin interface that does not match its API — which is -why `dist` refuses to build against the `web/dist-placeholder` default at all. +Build `web/dist` every time, before the binaries. A stale bundle is embedded silently and ships an admin interface that does not match its API — which is why `dist` refuses to build against the `web/dist-placeholder` default at all. -The context has to be the repository root, because the Dockerfile copies -`zig-out/dist/bin` and `zig-out/dist/stage`. The result is a `scratch` image -holding the binary, a CA bundle, `/LICENSE`, `/THIRD-PARTY-NOTICES` and two -empty directories. +The context has to be the repository root, because the Dockerfile copies `zig-out/dist/bin` and `zig-out/dist/stage`. The result is a `scratch` image holding the binary, a CA bundle, `/LICENSE`, `/THIRD-PARTY-NOTICES` and two empty directories. Run that image instead of the published one by naming it: @@ -306,16 +237,13 @@ Run that image instead of the published one by naming it: NXDNS_IMAGE=nxdns docker compose -f deploy/docker/compose.yaml up -d ``` -For an arm64 image on an x86_64 machine, use buildx. The Dockerfile's builder -stage is pinned to `$BUILDPLATFORM` and only copies files, so no emulation is -involved: +For an arm64 image on an x86_64 machine, use buildx. The Dockerfile's builder stage is pinned to `$BUILDPLATFORM` and only copies files, so no emulation is involved: ```sh docker buildx build --platform linux/arm64 -t nxdns:arm64 -f deploy/docker/Dockerfile . ``` -Add `--push` or `--load` to keep the result; the default buildx driver leaves -it in the build cache. +Add `--push` or `--load` to keep the result; the default buildx driver leaves it in the build cache. > Verified on this host, except the two buildx lines. `zig build dist` was run > to completion with the version read out of `build.zig.zon` and exited 0, and diff --git a/docs/how-to/install-with-systemd.md b/docs/how-to/install-with-systemd.md index 244f38b..a45e93c 100644 --- a/docs/how-to/install-with-systemd.md +++ b/docs/how-to/install-with-systemd.md @@ -1,16 +1,10 @@ # Install nxdns with systemd -Installs nxdns as a system service on a Linux host with systemd, including a -Raspberry Pi 5. At the end the service answers DNS on port 53 and starts on -boot. +Installs nxdns as a system service on a Linux host with systemd, including a Raspberry Pi 5. At the end the service answers DNS on port 53 and starts on boot. -The normal path is to download a released tarball, verify it, and install what -is inside it. Building from source is still supported and is the last section -of this page. +The normal path is to download a released tarball, verify it, and install what is inside it. Building from source is still supported and is the last section of this page. -For what each flag does, see [the CLI reference](../reference/cli.md); for what -each configuration field means, see -[the configuration reference](../reference/configuration.md). +For what each flag does, see [the CLI reference](../reference/cli.md); for what each configuration field means, see [the configuration reference](../reference/configuration.md). > Verification: `systemd-analyze verify` was run on the machine that wrote this > page. `nxdns check`, `nxdns import`, `nxdns export` and `nxdns run` were run @@ -33,9 +27,7 @@ each configuration field means, see ## 1. Download and verify -Two static musl tarballs are published per release, one per architecture. Pick -`x86_64-linux-musl` for a normal PC or server and `aarch64-linux-musl` for a -Raspberry Pi 5. +Two static musl tarballs are published per release, one per architecture. Pick `x86_64-linux-musl` for a normal PC or server and `aarch64-linux-musl` for a Raspberry Pi 5. ```sh BASE=https://git.mial.net/mokhtar/nxdns @@ -47,15 +39,9 @@ curl -fLO "$BASE/releases/download/v$VERSION/SHA256SUMS.txt" curl -fLO "$BASE/releases/download/v$VERSION/SHA256SUMS.txt.asc" ``` -The first line asks the server which release is current instead of hardcoding a -number that goes stale one release later — Gitea redirects `releases/latest` to -the newest published release's tag page. To install a particular version -instead, set `VERSION=` yourself with the one you want; the tarball -filenames carry the version either way, so there is no version-free download -URL for them. +The first line asks the server which release is current instead of hardcoding a number that goes stale one release later — Gitea redirects `releases/latest` to the newest published release's tag page. To install a particular version instead, set `VERSION=` yourself with the one you want; the tarball filenames carry the version either way, so there is no version-free download URL for them. -Verify before you extract. The signature is over `SHA256SUMS.txt`, and -`SHA256SUMS.txt` is over the tarballs: +Verify before you extract. The signature is over `SHA256SUMS.txt`, and `SHA256SUMS.txt` is over the tarballs: ```sh gpg --verify SHA256SUMS.txt.asc SHA256SUMS.txt @@ -63,13 +49,9 @@ sha256sum -c --ignore-missing SHA256SUMS.txt tar -xzf "nxdns-$VERSION-x86_64-linux-musl.tar.gz" ``` -[Verify a release](verify-a-release.md) has the whole procedure: where the -public key comes from, what fingerprint to expect, what each failure means, and -what the signature does and does not prove. Read it once before your first -install. +[Verify a release](verify-a-release.md) has the whole procedure: where the public key comes from, what fingerprint to expect, what each failure means, and what the signature does and does not prove. Read it once before your first install. -The extracted directory `nxdns-$VERSION-x86_64-linux-musl/` holds everything -this page installs: +The extracted directory `nxdns-$VERSION-x86_64-linux-musl/` holds everything this page installs: | File | What it is | | --- | --- | @@ -92,8 +74,7 @@ cd "nxdns-$VERSION-x86_64-linux-musl" scp nxdns nxdns.service nxdns.conf target:/tmp/ ``` -For a Raspberry Pi 5, extract the `aarch64-linux-musl` tarball instead — see -[Raspberry Pi 5](#raspberry-pi-5) below. +For a Raspberry Pi 5, extract the `aarch64-linux-musl` tarball instead — see [Raspberry Pi 5](#raspberry-pi-5) below. > Not verified on this host: `target` is a placeholder for your server's > hostname, and the machine that wrote this page has no second host to copy @@ -111,9 +92,7 @@ Off the target host this prints one complaint and exits 1: nxdns.service: Command /usr/local/bin/nxdns is not executable: No such file or directory ``` -That is the ExecStart path check finding no binary yet. Any other message is a -real problem with the unit. On the target, after step 3, the same command -should print nothing. +That is the ExecStart path check finding no binary yet. Any other message is a real problem with the unit. On the target, after step 3, the same command should print nothing. > Verified on this host against `deploy/systemd/nxdns.service` in a checkout, > which is the same file the tarball ships — the path is the only difference. @@ -134,31 +113,21 @@ systemctl daemon-reload mkdir -p -m 0755 /etc/nxdns ``` -The sysusers fragment ships under the name it is installed as, so there is no -rename to get wrong. +The sysusers fragment ships under the name it is installed as, so there is no rename to get wrong. > Not verified on this host: these commands need root on a target machine. The > files they install were read at HEAD and the unit was checked with > `systemd-analyze verify`. -The service user is a static one, not `DynamicUser`: a TLS key for the DoH or -DoT listener has to be chown-able to a uid that survives a restart. +The service user is a static one, not `DynamicUser`: a TLS key for the DoH or DoT listener has to be chown-able to a uid that survives a restart. -Do not create `/var/lib/nxdns` or `/var/log/nxdns` by hand. The unit's -`StateDirectory` and `LogsDirectory` settings make systemd create them on first -start, `/var/lib/nxdns` at mode 0700 owned by `nxdns`. +Do not create `/var/lib/nxdns` or `/var/log/nxdns` by hand. The unit's `StateDirectory` and `LogsDirectory` settings make systemd create them on first start, `/var/lib/nxdns` at mode 0700 owned by `nxdns`. -`/etc/nxdns` is the one directory the `mkdir` above is for. The unit's -`ConfigurationDirectory=nxdns` also creates it, but not until the first start -in step 5, and step 4 has to write a file into it before then. systemd does not -mind finding the directory already there; it adjusts the mode and ownership to -what the unit asks for. +`/etc/nxdns` is the one directory the `mkdir` above is for. The unit's `ConfigurationDirectory=nxdns` also creates it, but not until the first start in step 5, and step 4 has to write a file into it before then. systemd does not mind finding the directory already there; it adjusts the mode and ownership to what the unit asks for. ## 4. Write the configuration -nxdns will not start with nothing to forward to. Write `/etc/nxdns/config.zon`. -The smallest file that starts is one group named `default` and one enabled -upstream: +nxdns will not start with nothing to forward to. Write `/etc/nxdns/config.zon`. The smallest file that starts is one group named `default` and one enabled upstream: ```zon .{ @@ -168,36 +137,20 @@ upstream: } ``` -That file holds a password in plain text, so restrict it as soon as you have -written it: +That file holds a password in plain text, so restrict it as soon as you have written it: ```sh chown root:nxdns /etc/nxdns/config.zon chmod 0640 /etc/nxdns/config.zon ``` -Root's umask is 022 on most distributions, so a freshly written -`/etc/nxdns/config.zon` is mode 0644 and every account on the host can read the -password out of it. The unit's `UMask=0077` does not help here: it applies to -files the service creates once it is running, and never re-chmods a file that -was written before the first start. +Root's umask is 022 on most distributions, so a freshly written `/etc/nxdns/config.zon` is mode 0644 and every account on the host can read the password out of it. The unit's `UMask=0077` does not help here: it applies to files the service creates once it is running, and never re-chmods a file that was written before the first start. -0640 with group `nxdns` rather than 0600: `/etc/nxdns` is a -`ConfigurationDirectory`, which systemd leaves owned by root, and the service -runs as `nxdns`. A root-owned 0600 file would be unreadable to it. +0640 with group `nxdns` rather than 0600: `/etc/nxdns` is a `ConfigurationDirectory`, which systemd leaves owned by root, and the service runs as `nxdns`. A root-owned 0600 file would be unreadable to it. -Keep that group read bit for good, not just for the first boot. Under -`run --config` the service reads this file on **every** start, so tightening -the mode later breaks the next restart. Under database authority it is -`nxdns import` that reads the file, as whoever runs that command, and a bare -`nxdns run` never reads it at all. +Keep that group read bit for good, not just for the first boot. Under `run --config` the service reads this file on **every** start, so tightening the mode later breaks the next restart. Under database authority it is `nxdns import` that reads the file, as whoever runs that command, and a bare `nxdns run` never reads it at all. -Do not expect `nxdns check` to catch a permissive mode here. Its only -permission warning is for a TLS private key -(`WARN doh_server.key_path: ... is mode 644; a TLS key must be readable by its -owner only`, from `checkTlsFiles` in `src/cli.zig`); it never stats the -configuration file. A mode 0644 `config.zon` passes `check` in silence, so the -`chmod` above is yours to remember. +Do not expect `nxdns check` to catch a permissive mode here. Its only permission warning is for a TLS private key (`WARN doh_server.key_path: ... is mode 644; a TLS key must be readable by its owner only`, from `checkTlsFiles` in `src/cli.zig`); it never stats the configuration file. A mode 0644 `config.zon` passes `check` in silence, so the `chmod` above is yours to remember. Check it before you start the service: @@ -205,8 +158,7 @@ Check it before you start the service: nxdns check --config /etc/nxdns/config.zon ``` -A good file prints the source it checked, one `OK` line per upstream, and -`OK: no problems found`: +A good file prints the source it checked, one `OK` line per upstream, and `OK: no problems found`: ``` checking configuration file /etc/nxdns/config.zon @@ -214,9 +166,7 @@ OK upstreams[0] https://cloudflare-dns.com OK: no problems found ``` -The upstream probe sends a real query, so this needs working DNS on the host at -the time you run it. Exit 2 means `check` found something to fix and printed -every problem it found, not only the first. +The upstream probe sends a real query, so this needs working DNS on the host at the time you run it. Exit 2 means `check` found something to fix and printed every problem it found, not only the first. Now load it into the database: @@ -229,28 +179,17 @@ info(migrations): config.db migrated from schema version 0 to 1 imported /etc/nxdns/config.zon ``` -The plaintext password is hashed into `web.password_hash` and never stored as -plaintext; `nxdns export` writes `.password = null` beside the hash. Nothing -downstream reads the plaintext again, so once step 6 confirms you can log in you -can delete the file: +The plaintext password is hashed into `web.password_hash` and never stored as plaintext; `nxdns export` writes `.password = null` beside the hash. Nothing downstream reads the plaintext again, so once step 6 confirms you can log in you can delete the file: ```sh rm /etc/nxdns/config.zon ``` -A kept file is not a backup — `nxdns export` is (see -[Back up and restore](back-up-and-restore.md)), and the export carries the -password hash rather than the password. If you keep it, leave it at 0640 -root:nxdns. +A kept file is not a backup — `nxdns export` is (see [Back up and restore](back-up-and-restore.md)), and the export carries the password hash rather than the password. If you keep it, leave it at 0640 root:nxdns. -That is the **database mode** install, which is what the packaged unit runs: -`ExecStart=/usr/local/bin/nxdns run`, no `--config`, so nothing reads a file -after this step. Change settings afterwards through the admin interface, the -API, or an export–edit–import cycle. +That is the **database mode** install, which is what the packaged unit runs: `ExecStart=/usr/local/bin/nxdns run`, no `--config`, so nothing reads a file after this step. Change settings afterwards through the admin interface, the API, or an export–edit–import cycle. -If you would rather keep `/etc/nxdns/config.zon` in git and have every restart -converge onto it, do not delete the file — go to -[Run in file mode](#run-in-file-mode) instead, and skip the `rm`. +If you would rather keep `/etc/nxdns/config.zon` in git and have every restart converge onto it, do not delete the file — go to [Run in file mode](#run-in-file-mode) instead, and skip the `rm`. > Verified on this host, with a scratch `--config` and `--data-dir` in place of > `/etc/nxdns` and `/var/lib/nxdns` — those two paths are the only difference @@ -277,20 +216,11 @@ A healthy start logs a line naming every socket it bound: info(nxdns): nxdns serving on udp [::]:53 tcp [::]:53 tcp 0.0.0.0:53; 1 upstream(s); blocklist generation 1 ``` -nxdns writes to stderr and systemd captures that into the journal; logging -needs no further configuration. Port 53 is privileged, and the unit grants -`CAP_NET_BIND_SERVICE` through `AmbientCapabilities`. +nxdns writes to stderr and systemd captures that into the journal; logging needs no further configuration. Port 53 is privileged, and the unit grants `CAP_NET_BIND_SERVICE` through `AmbientCapabilities`. -The unit does not restart the service after exit 2 or exit 64 -(`RestartPreventExitStatus=2 64`). Those are a wrong configuration and a wrong -command line, and neither clears on a retry — restarting every two seconds until -`StartLimitBurst` gives up would only bury the diagnostics that are already in -the journal. `systemctl status nxdns` shows the failed state; fix the cause and -start it again. +The unit does not restart the service after exit 2 or exit 64 (`RestartPreventExitStatus=2 64`). Those are a wrong configuration and a wrong command line, and neither clears on a retry — restarting every two seconds until `StartLimitBurst` gives up would only bury the diagnostics that are already in the journal. `systemctl status nxdns` shows the failed state; fix the cause and start it again. -If the start fails, read [Troubleshoot nxdns](troubleshoot.md). The two common -first-install failures are a port 53 already held by `systemd-resolved` and a -configuration file that does not parse. +If the start fails, read [Troubleshoot nxdns](troubleshoot.md). The two common first-install failures are a port 53 already held by `systemd-resolved` and a configuration file that does not parse. ## 6. Confirm it answers @@ -300,9 +230,7 @@ From another machine on the LAN: dig @ example.com A +short ``` -The admin interface is on port 8080 by default; log in with the password from -the configuration file. `http://:8080/api/health` reports upstream -availability and disk state without a login. +The admin interface is on port 8080 by default; log in with the password from the configuration file. `http://:8080/api/health` reports upstream availability and disk state without a login. > Not verified on this host as written: `` is a placeholder, and a > LAN client to run it from is a second machine this host does not have. What @@ -314,25 +242,15 @@ availability and disk state without a login. ## Run in file mode -In file mode `/etc/nxdns/config.zon` is the configuration: every start converges -the database onto it, and the admin interface refuses configuration edits with a -403 naming the file. Use it when you want the file in git and deployed by -Ansible. Stay in database mode when you want the UI to be the way things change. +In file mode `/etc/nxdns/config.zon` is the configuration: every start converges the database onto it, and the admin interface refuses configuration edits with a 403 naming the file. Use it when you want the file in git and deployed by Ansible. Stay in database mode when you want the UI to be the way things change. -The packaged unit is flagless on purpose — it is correct as shipped, and a -commented-out alternative `ExecStart` in a unit file is documentation -masquerading as configuration. File mode is a drop-in. +The packaged unit is flagless on purpose — it is correct as shipped, and a commented-out alternative `ExecStart` in a unit file is documentation masquerading as configuration. File mode is a drop-in. ### Adopt file mode on a box that is already running -Run these in order. **Stop first**, and do not skip that: any edit made through -the UI between an export and the restart would be silently reverted by the first -reconcile, and `nxdns check` against a live database refuses to grade it (below). +Run these in order. **Stop first**, and do not skip that: any edit made through the UI between an export and the restart would be silently reverted by the first reconcile, and `nxdns check` against a live database refuses to grade it (below). -If you are arriving here from an upgrade, the binary must already be the new -one before you export. An export written by 0.0.1 carries a `.password = ""` -line this binary refuses; see -[the order trap](upgrade.md#the-order-trap-export-with-the-new-binary-not-the-old-one). +If you are arriving here from an upgrade, the binary must already be the new one before you export. An export written by 0.0.1 carries a `.password = ""` line this binary refuses; see [the order trap](upgrade.md#the-order-trap-export-with-the-new-binary-not-the-old-one). ```sh systemctl stop nxdns @@ -360,12 +278,9 @@ systemctl daemon-reload systemctl start nxdns ``` -The empty `ExecStart=` is required. Without it systemd appends a second command -to the list rather than replacing the first, and the unit tries to run nxdns -twice. +The empty `ExecStart=` is required. Without it systemd appends a second command to the list rather than replacing the first, and the unit tries to run nxdns twice. -The first start after adoption changes nothing, because the file was rendered -from the database it is now governing: +The first start after adoption changes nothing, because the file was rendered from the database it is now governing: ``` reconciled '/etc/nxdns/config.zon': no changes @@ -373,14 +288,9 @@ info(nxdns): authority: file (/etc/nxdns/config.zon) info(nxdns): nxdns serving on udp [::]:53 tcp [::]:53 tcp 0.0.0.0:53; 1 upstream(s); blocklist generation 1 ``` -`authority: file` is the line that confirms the drop-in took. Blocklists, -compiled snapshots and client history all survive, and every later start on an -unchanged file writes nothing either. +`authority: file` is the line that confirms the drop-in took. Blocklists, compiled snapshots and client history all survive, and every later start on an unchanged file writes nothing either. -The file now carries `web.password_hash`, so restrict it the same way step 4 -does — `chown root:nxdns`, `chmod 0640`. The unit's `ReadOnlyPaths=/etc/nxdns` -denies the service write access to that directory, so the process that reads the -file cannot modify it. +The file now carries `web.password_hash`, so restrict it the same way step 4 does — `chown root:nxdns`, `chmod 0640`. The unit's `ReadOnlyPaths=/etc/nxdns` denies the service write access to that directory, so the process that reads the file cannot modify it. ### Change the configuration from now on @@ -392,11 +302,7 @@ nxdns check --config /etc/nxdns/config.zon systemctl restart nxdns ``` -Make `nxdns check --config` the precondition of any Ansible handler that -restarts nxdns. A file-mode start reads the file on **every** boot, so a bad -push that skips its handler does not fail at deploy time — it detonates at the -next power cut. Validating before restarting turns that into a failed deploy at -noon. +Make `nxdns check --config` the precondition of any Ansible handler that restarts nxdns. A file-mode start reads the file on **every** boot, so a bad push that skips its handler does not fail at deploy time — it detonates at the next power cut. Validating before restarting turns that into a failed deploy at noon. The restart prints what it changed: @@ -407,9 +313,7 @@ settings keys changed: dns.port web.port ### Leave file mode -Remove the drop-in and restart. The database already holds the last reconciled -state, so nothing else is needed and the server comes back serving the same -configuration: +Remove the drop-in and restart. The database already holds the last reconciled state, so nothing else is needed and the server comes back serving the same configuration: ```sh rm /etc/systemd/system/nxdns.service.d/file-mode.conf @@ -448,8 +352,7 @@ info(nxdns): authority: database ## Raspberry Pi 5 -The Pi 5 is aarch64. Nothing about the procedure changes except which tarball -you take: +The Pi 5 is aarch64. Nothing about the procedure changes except which tarball you take: ```sh curl -fLO "$BASE/releases/download/v$VERSION/nxdns-$VERSION-aarch64-linux-musl.tar.gz" @@ -467,12 +370,7 @@ Then follow steps 3 to 6 on the Pi. ## Build from source instead -You do not need this to install nxdns, and it gets you a binary nobody has -signed. It is here for two cases: you want to run something other than a -tagged release, or you want to build the release yourself and compare it -against the published one. For the second case, follow -[Verify a release](verify-a-release.md) rather than this section — it says what -the comparison is and is not worth. +You do not need this to install nxdns, and it gets you a binary nobody has signed. It is here for two cases: you want to run something other than a tagged release, or you want to build the release yourself and compare it against the published one. For the second case, follow [Verify a release](verify-a-release.md) rather than this section — it says what the comparison is and is not worth. Requires Zig 0.16.0 and Node.js. From the repository root: @@ -483,29 +381,18 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" -Dweb-dist=web/dist -Doptimize=ReleaseSafe ``` -The first command builds the admin interface into `web/dist`; the last one -embeds that directory in the binary. Build the interface every time, before the -binary: a stale `web/dist` ships an admin UI that does not match the API it -talks to. `dist` refuses to run against the `web/dist-placeholder` default for -exactly that reason, so there is no way to skip it by accident. +The first command builds the admin interface into `web/dist`; the last one embeds that directory in the binary. Build the interface every time, before the binary: a stale `web/dist` ships an admin UI that does not match the API it talks to. `dist` refuses to run against the `web/dist-placeholder` default for exactly that reason, so there is no way to skip it by accident. -`-Dversion-string` is required and has no default. It is what `nxdns version` -prints. Take it from `build.zig.zon` rather than inventing one: `verify-dist` -asserts that the version under build equals `.version` there, so a made-up -string like `0.0.0-local` builds but then fails verification. `-Dgit-commit` -is what distinguishes your build from the published one of the same version. +`-Dversion-string` is required and has no default. It is what `nxdns version` prints. Take it from `build.zig.zon` rather than inventing one: `verify-dist` asserts that the version under build equals `.version` there, so a made-up string like `0.0.0-local` builds but then fails verification. `-Dgit-commit` is what distinguishes your build from the published one of the same version. -What comes out under `zig-out/dist/` is the same set a release publishes, -minus the signature and the image digest: +What comes out under `zig-out/dist/` is the same set a release publishes, minus the signature and the image digest: - `bin//nxdns` — the stripped static binary, one per target - `stage/nxdns--/` — the staged payload, one per target - `nxdns--.tar.gz` — one tarball per target -- `SHA256SUMS` — the two tarball hashes. The release publishes this as - `SHA256SUMS.txt`, with a third line for the image digest appended +- `SHA256SUMS` — the two tarball hashes. The release publishes this as `SHA256SUMS.txt`, with a third line for the image digest appended -The two targets are `x86_64-linux-musl` and `aarch64-linux-musl`. Both binaries -are statically linked and need nothing installed on the target host. +The two targets are `x86_64-linux-musl` and `aarch64-linux-musl`. Both binaries are statically linked and need nothing installed on the target host. Check the result the same way the release pipeline does: @@ -514,10 +401,7 @@ zig build verify-dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse -Dweb-dist=web/dist -Doptimize=ReleaseSafe ``` -`verify-dist` extracts each archive and asserts the ELF is static and within -the size budget, that the layout and file modes are exactly what step 1 lists, -and that `nxdns version` prints what was built. It exits non-zero on any -failure. +`verify-dist` extracts each archive and asserts the ELF is static and within the size budget, that the layout and file modes are exactly what step 1 lists, and that `nxdns version` prints what was built. It exits non-zero on any failure. > Verified on this host: `zig build dist` and `zig build verify-dist` were both > run to completion with the version taken from `build.zig.zon`. `dist` @@ -529,13 +413,11 @@ failure. > `FAIL zon-version: build.zig.zon says '0.0.1', the build says '0.0.0-local'`, > which is why this section reads the version out of `build.zig.zon`. -From here, join the page at step 2 with the staged directory in place of the -extracted one: +From here, join the page at step 2 with the staged directory in place of the extracted one: ```sh cd "zig-out/dist/stage/nxdns-$VERSION-x86_64-linux-musl" scp nxdns nxdns.service nxdns.conf target:/tmp/ ``` -The aarch64 binary is built by the same command and needs no toolchain on the -Pi. +The aarch64 binary is built by the same command and needs no toolchain on the Pi. diff --git a/docs/how-to/measure-performance.md b/docs/how-to/measure-performance.md index b572392..5ffa442 100644 --- a/docs/how-to/measure-performance.md +++ b/docs/how-to/measure-performance.md @@ -1,14 +1,8 @@ # Measure performance -`tools/bench.zig` measures the three things nxdns can measure in-process: -blocklist lookup latency, cache-hit latency, and blocklist compile throughput. -Sustained query rate is not one of them — that one is end-to-end and needs a -load generator pointed at a running server. +`tools/bench.zig` measures the three things nxdns can measure in-process: blocklist lookup latency, cache-hit latency, and blocklist compile throughput. Sustained query rate is not one of them — that one is end-to-end and needs a load generator pointed at a running server. -The numbers this project treats as targets, and the numbers measured so far, -are in the [performance reference](../reference/performance.md). Why those -targets exist and why CI does not gate on them is in -[performance and testing](../explanation/performance-and-testing.md). +The numbers this project treats as targets, and the numbers measured so far, are in the [performance reference](../reference/performance.md). Why those targets exist and why CI does not gate on them is in [performance and testing](../explanation/performance-and-testing.md). ## Run the whole bench @@ -16,12 +10,9 @@ targets exist and why CI does not gate on them is in zig build bench -Doptimize=ReleaseFast ``` -That runs all three suites with the defaults: 1,000,000 domains, 200,000 -iterations per suite, seed `0x5eed`. It takes minutes, most of it generating and -loading the million-domain list. +That runs all three suites with the defaults: 1,000,000 domains, 200,000 iterations per suite, seed `0x5eed`. It takes minutes, most of it generating and loading the million-domain list. -`-Doptimize=ReleaseFast` is not optional if you want the numbers to mean -anything. A Debug build says so before it prints: +`-Doptimize=ReleaseFast` is not optional if you want the numbers to mean anything. A Debug build says so before it prints: ``` warning: Debug build; run with -Doptimize=ReleaseFast for meaningful numbers @@ -29,9 +20,7 @@ warning: Debug build; run with -Doptimize=ReleaseFast for meaningful numbers ## Run one suite, smaller -Everything after `--` goes to the harness. A suite name selects one of -`filter`, `cache`, `compile` (the default is `all`), and `--domains` / -`--iters` shrink the load: +Everything after `--` goes to the harness. A suite name selects one of `filter`, `cache`, `compile` (the default is `all`), and `--domains` / `--iters` shrink the load: ```sh zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000 @@ -47,10 +36,7 @@ filter 20000 2.38 2.76 2.88 20.32 target VmRSS < 100 MiB: PASS ``` -A reduced run is good for checking the harness works and for a rough -regression signal. It is not a result: the memory figure scales with -`--domains`, so 100,000 domains says nothing about the 1,000,000-domain memory -target. +A reduced run is good for checking the harness works and for a rough regression signal. It is not a result: the memory figure scales with `--domains`, so 100,000 domains says nothing about the 1,000,000-domain memory target. The other two suites: @@ -74,10 +60,7 @@ suite ops p50(us) p95(us) p99(us) max(us) compile 100000 wall 15.623ms, 6400464 lines/s, 100000 domains kept (informational) ``` -`--seed=N` changes the generated domains and the query order; the default is -`0x5eed`, so two runs on the same machine are comparable. `--domains` caps at -4,000,000, and the `compile` suite additionally refuses more than 2,000,000 — -the compiler's own limit. +`--seed=N` changes the generated domains and the query order; the default is `0x5eed`, so two runs on the same machine are comparable. `--domains` caps at 4,000,000, and the `compile` suite additionally refuses more than 2,000,000 — the compiler's own limit. An argument the harness does not recognise stops it before any measuring: @@ -88,36 +71,21 @@ usage: zig build bench -Doptimize=ReleaseFast -- [filter|cache|compile|all] [--d ## Read the output -- `p50`/`p95`/`p99`/`max` are per-operation microseconds, nearest-rank over - every iteration. What one operation means differs per suite: for `filter` it - is normalising a name plus evaluating it against the snapshot; for `cache` it - is building the key, getting the entry and stamping the response id. -- `blocked N/M` and `hits N/M` are sanity counters. The harness aborts if either - is zero — a suite that never hits its own path measures nothing. -- `32 regex rules` on the `filter` line is the rule set the suite loads. No - generated query matches any of them, so every operation runs all 32 programs - to their end, which is the costly case and the one worth measuring. -- Two memory figures appear on purpose. `Snapshot.memoryBytes` and - `DnsCache.memoryBytes` are the in-repo accounting of those structures; `VmRSS` - is what the kernel holds resident for the whole process, allocator slack and - code included. The truth is between them, and the memory target is judged on - `VmRSS`. -- `target ...: PASS` / `FAIL` lines appear for the targets a suite covers. On a - plain run they are informational and the exit code stays 0. +- `p50`/`p95`/`p99`/`max` are per-operation microseconds, nearest-rank over every iteration. What one operation means differs per suite: for `filter` it is normalising a name plus evaluating it against the snapshot; for `cache` it is building the key, getting the entry and stamping the response id. +- `blocked N/M` and `hits N/M` are sanity counters. The harness aborts if either is zero — a suite that never hits its own path measures nothing. +- `32 regex rules` on the `filter` line is the rule set the suite loads. No generated query matches any of them, so every operation runs all 32 programs to their end, which is the costly case and the one worth measuring. +- Two memory figures appear on purpose. `Snapshot.memoryBytes` and `DnsCache.memoryBytes` are the in-repo accounting of those structures; `VmRSS` is what the kernel holds resident for the whole process, allocator slack and code included. The truth is between them, and the memory target is judged on `VmRSS`. +- `target ...: PASS` / `FAIL` lines appear for the targets a suite covers. On a plain run they are informational and the exit code stays 0. ## Fail the run when a target is missed -`--assert` turns those lines into an exit code — 1 when any target was -exceeded, 0 otherwise. This is meant for an acceptance run on hardware you -control, not for CI: +`--assert` turns those lines into an exit code — 1 when any target was exceeded, 0 otherwise. This is meant for an acceptance run on hardware you control, not for CI: ```sh zig build bench -Doptimize=ReleaseFast -- --assert ``` -The full-scale form is the one worth asserting on, because the memory target -only means something at a million domains. On this development host the reduced -form was used to check the flag itself: +The full-scale form is the one worth asserting on, because the memory target only means something at a million domains. On this development host the reduced form was used to check the flag itself: ```sh zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000 --assert @@ -130,37 +98,22 @@ filter 20000 2.34 2.71 2.85 15.06 target VmRSS < 100 MiB: PASS ``` -**Not verified on this host at full scale:** the plain -`zig build bench -Doptimize=ReleaseFast -- --assert` above was not run during -the writing of this page — the default run takes minutes. The reduced runs -shown were all executed as written. The full-scale numbers already recorded for -this host are in the [performance reference](../reference/performance.md). +**Not verified on this host at full scale:** the plain `zig build bench -Doptimize=ReleaseFast -- --assert` above was not run during the writing of this page — the default run takes minutes. The reduced runs shown were all executed as written. The full-scale numbers already recorded for this host are in the [performance reference](../reference/performance.md). ## Measure sustained query rate -The bench harness cannot do this. Query rate is a property of the whole server -— sockets, upstreams, the query log writer — so it has to be driven from -outside, against the real binary, on the machine you care about. +The bench harness cannot do this. Query rate is a property of the whole server — sockets, upstreams, the query log writer — so it has to be driven from outside, against the real binary, on the machine you care about. -Start nxdns with real blocklists configured, then drive it from another host on -the LAN with a DNS load generator such as `dnsperf`: +Start nxdns with real blocklists configured, then drive it from another host on the LAN with a DNS load generator such as `dnsperf`: ```sh dnsperf -s 192.168.1.10 -p 53 -d queries.txt -c 20 -Q 200 -l 60 ``` -Read the client's own rate and the server's `/metrics` together: a load -generator that reports 200 qps while the server counts fewer has lost queries -somewhere, and that is the interesting number. +Read the client's own rate and the server's `/metrics` together: a load generator that reports 200 qps while the server counts fewer has lost queries somewhere, and that is the interesting number. -**Not verified on this host:** `dnsperf` is not installed here and the target -platform is a Raspberry Pi 5, not this development machine. The command above -is the shape of the measurement, not a transcript. +**Not verified on this host:** `dnsperf` is not installed here and the target platform is a Raspberry Pi 5, not this development machine. The command above is the shape of the measurement, not a transcript. ## Where to run it -The target platform is a Raspberry Pi 5. Numbers from a development x86_64 box -do not transfer — the Pi's Cortex-A76 is far slower — so a passing run here is -evidence the harness works and a baseline for spotting regressions on the -machine development happens on, and nothing more. Run `--assert` on the Pi, -where the numbers mean something. +The target platform is a Raspberry Pi 5. Numbers from a development x86_64 box do not transfer — the Pi's Cortex-A76 is far slower — so a passing run here is evidence the harness works and a baseline for spotting regressions on the machine development happens on, and nothing more. Run `--assert` on the Pi, where the numbers mean something. diff --git a/docs/how-to/set-up-admin-authentication.md b/docs/how-to/set-up-admin-authentication.md index 6203c24..c536c4e 100644 --- a/docs/how-to/set-up-admin-authentication.md +++ b/docs/how-to/set-up-admin-authentication.md @@ -1,13 +1,8 @@ # Set up admin authentication -The admin interface and its API are protected by a single operator password. -With no password set, every route is open to anything that can reach the web -port. Set one. +The admin interface and its API are protected by a single operator password. With no password set, every route is open to anything that can reach the web port. Set one. -The commands below run against the scratch lab from -[enable DoH and DoT](enable-doh-and-dot.md): data directory -`/tmp/nxdns-lab/data`, web listener on `127.0.0.1:8451`. On a real install the -data directory is `/var/lib/nxdns` and the web port is 8080. +The commands below run against the scratch lab from [enable DoH and DoT](enable-doh-and-dot.md): data directory `/tmp/nxdns-lab/data`, web listener on `127.0.0.1:8451`. On a real install the data directory is `/var/lib/nxdns` and the web port is 8080. ## 1. Set the password @@ -21,17 +16,14 @@ Put it in the configuration file, under `web`: } ``` -The plaintext is hashed with argon2id into `web.password_hash` and discarded. It -becomes no database row and appears in no log line. Setting both `password` and -`password_hash` in one file is refused: +The plaintext is hashed with argon2id into `web.password_hash` and discarded. It becomes no database row and appears in no log line. Setting both `password` and `password_hash` in one file is refused: ``` web.password: password and password_hash are both set; ambiguity in a security setting is refused import failed: PasswordAndHashBothSet ``` -Applying that file — with `nxdns import`, or with a `nxdns run --config` start — -announces the change: +Applying that file — with `nxdns import`, or with a `nxdns run --config` start — announces the change: ``` web authentication is now enabled @@ -39,9 +31,7 @@ web authentication is now enabled ### Absent, empty, and set are three different things -The two fields are optional, and the difference between leaving one out and -setting it to `""` is the difference between keeping your password and removing -it: +The two fields are optional, and the difference between leaving one out and setting it to `""` is the difference between keeping your password and removing it: | The file says | Effect on the stored password | | --- | --- | @@ -51,11 +41,7 @@ it: | `.password_hash = "$argon2id$…"` | Installs that hash, for example from an export. | | `.password_hash = ""` | **Removes the password.** Authentication is off. | -Absence has to mean "keep", because the alternative is a foot-gun with a live -round in it. An export carries the full PHC string, which is long and ugly, and -sooner or later someone trims that line out of a file before committing it — -meaning "leave the password alone". If absence meant "no password", that edit -would open the admin interface to the whole LAN without a word. +Absence has to mean "keep", because the alternative is a foot-gun with a live round in it. An export carries the full PHC string, which is long and ugly, and sooner or later someone trims that line out of a file before committing it — meaning "leave the password alone". If absence meant "no password", that edit would open the admin interface to the whole LAN without a word. So removing the password takes the explicit empty string: @@ -63,23 +49,17 @@ So removing the password takes the explicit empty string: web authentication is now disabled ``` -And an empty plaintext is refused outright, because hashing the empty string -would switch authentication *on* while making every login impossible — the login -handler rejects empty passwords: +And an empty plaintext is refused outright, because hashing the empty string would switch authentication *on* while making every login impossible — the login handler rejects empty passwords: ``` FAIL web.password: password is set to the empty string; omit the field to keep the stored password, or set password_hash = "" to disable authentication ``` -Which of steps 4 and 5 applies to your server depends on its authority. Under -`nxdns run --config FILE` the file is the password: edit it and restart, and the -API refuses the change with a 403. Under bare `nxdns run` the database holds it, -and step 4 or step 5 is how it moves. +Which of steps 4 and 5 applies to your server depends on its authority. Under `nxdns run --config FILE` the file is the password: edit it and restart, and the API refuses the change with a 403. Under bare `nxdns run` the database holds it, and step 4 or step 5 is how it moves. ## 2. Log in -Login is `POST /api/auth/login` with a JSON body. Without a session, the API -answers 401: +Login is `POST /api/auth/login` with a JSON body. Without a session, the API answers 401: ```sh curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8451/api/stats @@ -102,16 +82,13 @@ curl -sS -c /tmp/nxdns-lab/cookies.txt \ {"authenticated":true,"auth_required":true} ``` -The session token comes back in a `Set-Cookie` header, not in the body. In the -jar it looks like this (value redacted here): +The session token comes back in a `Set-Cookie` header, not in the body. In the jar it looks like this (value redacted here): ``` #HttpOnly_127.0.0.1 FALSE / FALSE 1786559938 nxdns_session ``` -The cookie is named `nxdns_session` and carries `HttpOnly; SameSite=Lax; -Path=/`. Its `Max-Age` comes from `web.session_ttl_hours`. Send it back on every -later call: +The cookie is named `nxdns_session` and carries `HttpOnly; SameSite=Lax; Path=/`. Its `Max-Age` comes from `web.session_ttl_hours`. Send it back on every later call: ```sh curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w '%{http_code}\n' \ @@ -122,8 +99,7 @@ curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w '%{http_code}\n' \ 200 ``` -A wrong password and an unknown one are the same answer, so a guess learns -nothing: +A wrong password and an unknown one are the same answer, so a guess learns nothing: ```sh curl -sS -X POST http://127.0.0.1:8451/api/auth/login \ @@ -142,9 +118,7 @@ info(web_auth): web login accepted for 127.0.0.1:34040 warning(web_auth): web login refused for 127.0.0.1:59670 ``` -Sessions live in memory only. A restart logs everyone out. Thirty-two -concurrent sessions are kept; a thirty-third login evicts the least recently -used one. +Sessions live in memory only. A restart logs everyone out. Thirty-two concurrent sessions are kept; a thirty-third login evicts the least recently used one. ## 3. Log out @@ -160,18 +134,13 @@ curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w 'stats: %{http_code}\n' \ stats: 401 ``` -Logging out with a stale cookie, or with none, answers the same way. The point -of logging out is to end up logged out, and that is where such a request -already is. +Logging out with a stale cookie, or with none, answers the same way. The point of logging out is to end up logged out, and that is where such a request already is. ## 4. Change the password on a running server -This is a database-mode procedure. In file mode `PUT /api/settings` answers 403 -naming the file; edit `web.password` there and restart instead. +This is a database-mode procedure. In file mode `PUT /api/settings` answers 403 naming the file; edit `web.password` there and restart instead. -Send the new one to `PUT /api/settings` as `web.password`. The response is the -full settings document; `password` is write-only and `password_hash` is neither -readable nor directly writable, so neither value comes back. +Send the new one to `PUT /api/settings` as `web.password`. The response is the full settings document; `password` is write-only and `password_hash` is neither readable nor directly writable, so neither value comes back. ```sh curl -sS -c /tmp/nxdns-lab/c2.txt -X POST http://127.0.0.1:8451/api/auth/login \ @@ -181,8 +150,7 @@ curl -sS -b /tmp/nxdns-lab/c2.txt -X PUT http://127.0.0.1:8451/api/settings \ -d '{"web":{"password":"a-new-password"}}' ``` -Changing the password ends every session, including the one that made the -change: +Changing the password ends every session, including the one that made the change: ```sh curl -sS -b /tmp/nxdns-lab/c2.txt -o /dev/null -w 'old session: %{http_code}\n' \ @@ -205,39 +173,26 @@ Log back in with the new password. That is the whole rotation. ## 5. Change the password without the API -If you have lost the password, the admin interface cannot help — go through the -database instead. Export, edit, import. `nxdns export` always writes -`.password = null` and carries the hash, so an exported file re-imports without -anyone knowing the password. To install a new one, put it in `.password` and -clear `.password_hash`: +If you have lost the password, the admin interface cannot help — go through the database instead. Export, edit, import. `nxdns export` always writes `.password = null` and carries the hash, so an exported file re-imports without anyone knowing the password. To install a new one, put it in `.password` and clear `.password_hash`: ```sh nxdns export --data-dir /tmp/nxdns-lab/data --out /tmp/nxdns-lab/rekeyed.zon ``` -Edit the `web` section of `/tmp/nxdns-lab/rekeyed.zon`: set `.password` to the -new value and **delete the `.password_hash` line entirely**, so the `web` block -carries one password field and not two: +Edit the `web` section of `/tmp/nxdns-lab/rekeyed.zon`: set `.password` to the new value and **delete the `.password_hash` line entirely**, so the `web` block carries one password field and not two: ```zon .password = "offline-password", ``` -Deleting the line is the part to get right. Setting `.password_hash = ""` -alongside a plaintext password does not clear the way for it — an empty string -is a present value meaning "no password", so the file then states two -contradictory things and is refused: +Deleting the line is the part to get right. Setting `.password_hash = ""` alongside a plaintext password does not clear the way for it — an empty string is a present value meaning "no password", so the file then states two contradictory things and is refused: ``` FAIL web.password: password and password_hash are both set; ambiguity in a security setting is refused import failed: PasswordAndHashBothSet ``` -Stop the server before importing. `import` rewrites the stored hash underneath a -process that read it at startup; a running server keeps verifying against the -old one, so skipping the stop leaves the new password not working until the next -restart. In the lab the server is a foreground `nxdns run`, so Ctrl-C in its -terminal stops it, and it goes back up with the same command: +Stop the server before importing. `import` rewrites the stored hash underneath a process that read it at startup; a running server keeps verifying against the old one, so skipping the stop leaves the new password not working until the next restart. In the lab the server is a foreground `nxdns run`, so Ctrl-C in its terminal stops it, and it goes back up with the same command: ```sh # Ctrl-C the `nxdns run` terminal, or `kill` its pid from another shell @@ -249,14 +204,9 @@ nxdns run --data-dir /tmp/nxdns-lab/data imported /tmp/nxdns-lab/rekeyed.zon ``` -No flag is needed: replacing a password edits a settings value and deletes no -rows. +No flag is needed: replacing a password edits a settings value and deletes no rows. -On a real install the stop and start are `systemctl stop nxdns` and -`systemctl start nxdns` around the same `import` — **not verified on this -host**, which has no installed nxdns systemd unit (`systemctl status nxdns` -answers `Unit nxdns.service could not be found.`) and where `systemctl` needs -root. See [back up and restore](back-up-and-restore.md). +On a real install the stop and start are `systemctl stop nxdns` and `systemctl start nxdns` around the same `import` — **not verified on this host**, which has no installed nxdns systemd unit (`systemctl status nxdns` answers `Unit nxdns.service could not be found.`) and where `systemctl` needs root. See [back up and restore](back-up-and-restore.md). Once it is back up the old password is refused and the new one works: @@ -288,14 +238,11 @@ nxdns export --data-dir /tmp/nxdns-lab/data | grep password .password_hash = "$argon2id$v=19$m=19456,t=2,p=1$xqzK66LgiWGvyCmCl6ZRa3GHH0nS5qZnRgVfWmeGadc$1mafhflKFIg3vcHjJaDMAXGQiOjtym2UADZsPW1xkfw", ``` -See [back up and restore](back-up-and-restore.md) for when `import` does need -`--allow-delete`. +See [back up and restore](back-up-and-restore.md) for when `import` does need `--allow-delete`. ## What happens with no password set -Authentication is off. Every route is open, and a login attempt succeeds -without minting anything — there is nothing to log in to, and a session that -authorises nothing would be a lie for the browser to store: +Authentication is off. Every route is open, and a login attempt succeeds without minting anything — there is nothing to log in to, and a session that authorises nothing would be a lie for the browser to store: ```sh curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8453/api/stats @@ -308,24 +255,13 @@ curl -sS -X POST http://127.0.0.1:8453/api/auth/login \ {"authenticated":true,"auth_required":false} ``` -`auth_required: false` is how the admin interface knows to stop showing a login -form. Treat this as a lab-only state: bind the web listener to a trusted -interface at the very least, and preferably set a password. +`auth_required: false` is how the admin interface knows to stop showing a login form. Treat this as a lab-only state: bind the web listener to a trusted interface at the very least, and preferably set a password. ## Notes -- A stored hash this build cannot parse is a 500, not a 401. Answering 401 would - tell an operator with a corrupted `web.password_hash` that their password is - wrong, and they would retype a password that can never verify. -- Requests from the box itself skip the API rate limit by default - (`web.api_localhost_exempt`). -- **If you put a reverse proxy in front of the admin interface, configure - `web.trusted_proxies` or turn `web.api_localhost_exempt` off.** A proxy on the - same box connects from loopback, so every request arrives exempt and the API - limiter — the only brake on guessing the admin password — stops applying to - anyone. Listing the proxy's address in `web.trusted_proxies` makes nxdns read - the client's address from the `X-Forwarded-For` the proxy appends, so the - limiter and the SSE connection cap bind each real client again: +- A stored hash this build cannot parse is a 500, not a 401. Answering 401 would tell an operator with a corrupted `web.password_hash` that their password is wrong, and they would retype a password that can never verify. +- Requests from the box itself skip the API rate limit by default (`web.api_localhost_exempt`). +- **If you put a reverse proxy in front of the admin interface, configure `web.trusted_proxies` or turn `web.api_localhost_exempt` off.** A proxy on the same box connects from loopback, so every request arrives exempt and the API limiter — the only brake on guessing the admin password — stops applying to anyone. Listing the proxy's address in `web.trusted_proxies` makes nxdns read the client's address from the `X-Forwarded-For` the proxy appends, so the limiter and the SSE connection cap bind each real client again: ```zig .web = .{ @@ -333,17 +269,7 @@ interface at the very least, and preferably set a password. }, ``` - The proxy must append its own entry to that header. A proxy that forwards a - client-supplied `X-Forwarded-For` unchanged is not one to trust. -- `web.session_ttl_hours`, `web.api_rate_limit_per_min` and the rest are in the - [configuration reference](../reference/configuration.md); the routes are in - the [API reference](../reference/api.md). + The proxy must append its own entry to that header. A proxy that forwards a client-supplied `X-Forwarded-For` unchanged is not one to trust. +- `web.session_ttl_hours`, `web.api_rate_limit_per_min` and the rest are in the [configuration reference](../reference/configuration.md); the routes are in the [API reference](../reference/api.md). -Every command on this page was executed on this host as written, against the -lab described at the top, except the `systemctl` stop and start named in step 5 -and marked **not verified on this host** there. That includes the whole of -steps 2 to 5, re-run for this revision: the login, logout and rate-limit -transcripts reproduced exactly as printed, the both-set refusal in step 5 was -reproduced by leaving `.password_hash = ""` in the file, and the rekey then -succeeded once that line was deleted. The cookie jar's expiry timestamp is the -one that run produced and will differ on yours. +Every command on this page was executed on this host as written, against the lab described at the top, except the `systemctl` stop and start named in step 5 and marked **not verified on this host** there. That includes the whole of steps 2 to 5, re-run for this revision: the login, logout and rate-limit transcripts reproduced exactly as printed, the both-set refusal in step 5 was reproduced by leaving `.password_hash = ""` in the file, and the rekey then succeeded once that line was deleted. The cookie jar's expiry timestamp is the one that run produced and will differ on yours. diff --git a/docs/how-to/troubleshoot.md b/docs/how-to/troubleshoot.md index c1c2da6..cf9370c 100644 --- a/docs/how-to/troubleshoot.md +++ b/docs/how-to/troubleshoot.md @@ -1,29 +1,19 @@ # Troubleshoot nxdns -Symptoms an nxdns install actually produces, what to run to identify each one, -and what to change. Every symptom on this page was reproduced on the machine -that wrote it, and every diagnosis command was run there. Two details differ -from a real install and cannot be otherwise on that machine: it has no -installed service, so the log lines were read from a foreground run instead of -`journalctl -u nxdns`, and ports 53 and 8080 were occupied, so DNS and the API -were exercised on unprivileged ports. Fixes that need root are marked. +Symptoms an nxdns install actually produces, what to run to identify each one, and what to change. Every symptom on this page was reproduced on the machine that wrote it, and every diagnosis command was run there. Two details differ from a real install and cannot be otherwise on that machine: it has no installed service, so the log lines were read from a foreground run instead of `journalctl -u nxdns`, and ports 53 and 8080 were occupied, so DNS and the API were exercised on unprivileged ports. Fixes that need root are marked. -The exit codes themselves are listed in -[the CLI reference](../reference/cli.md). +The exit codes themselves are listed in [the CLI reference](../reference/cli.md). ## The service exits with code 2 -**Symptom.** The process stops immediately. The last two lines are the error -and a pointer: +**Symptom.** The process stops immediately. The last two lines are the error and a pointer: ``` nxdns run failed: NoUsableUpstreams run `nxdns check` to see the configuration in full ``` -Exit 2 means the configuration is wrong and you can fix it. Every subcommand -uses the same definition, so a file `run` exits 2 on exits 2 from `check` and -`import` too. +Exit 2 means the configuration is wrong and you can fix it. Every subcommand uses the same definition, so a file `run` exits 2 on exits 2 from `check` and `import` too. **Diagnosis.** @@ -31,14 +21,11 @@ uses the same definition, so a file `run` exits 2 on exits 2 from `check` and nxdns check ``` -`check` prints every problem it finds, not the first, and names the source it -checked on its first line. +`check` prints every problem it finds, not the first, and names the source it checked on its first line. **Fixes by cause.** -- `NoUsableUpstreams` — the database has no enabled upstream. On a fresh - install in database mode this is simply an empty database, and the run says - what to do about it on the next line: +- `NoUsableUpstreams` — the database has no enabled upstream. On a fresh install in database mode this is simply an empty database, and the run says what to do about it on the next line: ``` nxdns run failed: NoUsableUpstreams @@ -46,22 +33,11 @@ checked on its first line. load one with `nxdns import `, or make a file the source of truth with `nxdns run --config ` ``` - Write a configuration file and take either exit: `nxdns import ` to load - it into the database once, or add `--config ` to `ExecStart` to make the - file the configuration from then on. -- `ManagedConfigUnreadable` — the service runs `run --config FILE` and that file - is missing or the process may not read it. The path is in the FAIL line above - the failure. File mode never falls back to the database, on purpose: a - fallback would turn a bad deploy into a silently stale configuration. -- `BadCertificate` — a DoH or DoT listener is enabled and its certificate or - key is unreadable, too large, unparseable, or the key does not belong to the - certificate. `run` names both paths before it exits: - `doh_server: '' + '': certificate file is not readable`. + Write a configuration file and take either exit: `nxdns import ` to load it into the database once, or add `--config ` to `ExecStart` to make the file the configuration from then on. +- `ManagedConfigUnreadable` — the service runs `run --config FILE` and that file is missing or the process may not read it. The path is in the FAIL line above the failure. File mode never falls back to the database, on purpose: a fallback would turn a bad deploy into a silently stale configuration. +- `BadCertificate` — a DoH or DoT listener is enabled and its certificate or key is unreadable, too large, unparseable, or the key does not belong to the certificate. `run` names both paths before it exits: `doh_server: '' + '': certificate file is not readable`. - `check` catches this without starting a listener. It loads both PEM files and - tests the key against the certificate through the same code `run` uses, so it - fails on exactly what `run` would fail on. Reproduced here with a self-signed - pair and the key from a second, unrelated pair: + `check` catches this without starting a listener. It loads both PEM files and tests the key against the certificate through the same code `run` uses, so it fails on exactly what `run` would fail on. Reproduced here with a self-signed pair and the key from a second, unrelated pair: ``` $ nxdns check --config config.zon @@ -77,24 +53,15 @@ checked on its first line. nxdns run failed: BadCertificate # exit 2 ``` - The `warning(tls_server)` line comes from mbedTLS on stderr and can appear - before the `checking` line, which is on stdout. A cert file containing - `not a certificate` fails the same way, with - `FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`. - An unreadable file reads - `FAIL doh_server.cert_path: '': certificate file is not readable`. + The `warning(tls_server)` line comes from mbedTLS on stderr and can appear before the `checking` line, which is on stdout. A cert file containing `not a certificate` fails the same way, with `FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`. An unreadable file reads `FAIL doh_server.cert_path: '': certificate file is not readable`. - Fix the path, the ownership, or the pair; see - [Enable DoH and DoT](enable-doh-and-dot.md). -- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a - configuration, so this only reaches a database that was edited by hand. -- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of - that family. + Fix the path, the ownership, or the pair; see [Enable DoH and DoT](enable-doh-and-dot.md). +- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a configuration, so this only reaches a database that was edited by hand. +- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of that family. ## A configuration file you just wrote is rejected -**Symptom.** `nxdns run --config`, `nxdns check --config` or `nxdns import` -prints the validation problem and stops with exit 2: +**Symptom.** `nxdns run --config`, `nxdns check --config` or `nxdns import` prints the validation problem and stops with exit 2: ``` FAIL groups: no group named 'default'; every unknown client is assigned to it @@ -118,53 +85,32 @@ nxdns run failed: NoUpstreams run `nxdns check` to see the configuration in full ``` -`NoUpstreams` from a file is not the same fault as `NoUsableUpstreams` above: -the first is a file `run` refused, the second is a database `run` accepted and -found empty. Both are exit 2. +`NoUpstreams` from a file is not the same fault as `NoUsableUpstreams` above: the first is a file `run` refused, the second is a database `run` accepted and found empty. Both are exit 2. -**Diagnosis.** Run the same file through `check`, which reports the same -problems and exits 2: +**Diagnosis.** Run the same file through `check`, which reports the same problems and exits 2: ```sh nxdns check --config /etc/nxdns/config.zon ``` -**Fix.** Correct the file the diagnostics name and start again. Nothing was -applied — a file-mode reconcile happens in one transaction that rolls back, and -a failed `import` leaves the database untouched. The exit code does not depend -on which command read the file: all three of these files were run through `run`, -`check` and `import` here, and every one of the nine combinations exited 2 with -the same diagnostic. +**Fix.** Correct the file the diagnostics name and start again. Nothing was applied — a file-mode reconcile happens in one transaction that rolls back, and a failed `import` leaves the database untouched. The exit code does not depend on which command read the file: all three of these files were run through `run`, `check` and `import` here, and every one of the nine combinations exited 2 with the same diagnostic. -Under the shipped systemd unit an exit 2 stops the service rather than -restarting it (`RestartPreventExitStatus=2 64`), so the journal holds the -diagnostics instead of drowning them in a restart loop. `systemctl start nxdns` -once the file is fixed. +Under the shipped systemd unit an exit 2 stops the service rather than restarting it (`RestartPreventExitStatus=2 64`), so the journal holds the diagnostics instead of drowning them in a restart loop. `systemctl start nxdns` once the file is fixed. -Make `nxdns check --config ` the precondition in whatever pushes the file. -In file mode every boot reads it, so an unvalidated bad push does not fail at -deploy time — it fails at the next restart, which may be a power cut at 3am. +Make `nxdns check --config ` the precondition in whatever pushes the file. In file mode every boot reads it, so an unvalidated bad push does not fail at deploy time — it fails at the next restart, which may be a power cut at 3am. ## `nxdns check` fails on a server that is running fine -**Symptom.** The service is up and answering, but `nxdns check` on the same -machine exits 2 with one long line about a write-ahead log: +**Symptom.** The service is up and answering, but `nxdns check` on the same machine exits 2 with one long line about a write-ahead log: ``` checking database /var/lib/nxdns/config.db FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line. ``` -Nothing is damaged. `check` opens `config.db` immutable so that it can never -write to it, and an immutable open ignores the write-ahead log. When that log -holds bytes, the newest settings are in it and the main file holds older ones, -so `check` refuses rather than grade stale values. +Nothing is damaged. `check` opens `config.db` immutable so that it can never write to it, and an immutable open ignores the write-ahead log. When that log holds bytes, the newest settings are in it and the main file holds older ones, so `check` refuses rather than grade stale values. -The log holds bytes after a configuration write that has not been checkpointed -yet, which on a running server means someone changed something through the web -interface or the API. A server that has only been answering queries has an empty -`config.db-wal` and `check` reads it normally — so this line comes and goes, and -its absence is not proof that nothing is running. +The log holds bytes after a configuration write that has not been checkpointed yet, which on a running server means someone changed something through the web interface or the API. A server that has only been answering queries has an empty `config.db-wal` and `check` reads it normally — so this line comes and goes, and its absence is not proof that nothing is running. **Fix.** Check the exported configuration instead of the live file: @@ -173,9 +119,7 @@ nxdns export --data-dir /var/lib/nxdns --out /tmp/current.zon nxdns check --config /tmp/current.zon ``` -`export` opens the database read/write and does see the log, so it renders the -settings that are actually in force. Stopping the service and checking again -works too: a clean shutdown checkpoints the log away. +`export` opens the database read/write and does see the log, so it renders the settings that are actually in force. Stopping the service and checking again works too: a clean shutdown checkpoints the log away. > Reproduced here on a scratch data directory rather than `/var/lib/nxdns` — > that path is the only substitution in the output above. nxdns was started on @@ -194,8 +138,7 @@ cannot bind udp [::1]:53: AddressInUse nxdns run failed: AddressInUse ``` -A bind conflict is a runtime failure, not a configuration fault, so this is -exit 1 and `nxdns check` will not find it. +A bind conflict is a runtime failure, not a configuration fault, so this is exit 1 and `nxdns check` will not find it. **Diagnosis.** @@ -205,11 +148,9 @@ ss -lntp 'sport = :53' systemctl is-active systemd-resolved ``` -On most systemd distributions the holder is `systemd-resolved`, which runs a -stub listener on `127.0.0.53:53` and on some setups binds `0.0.0.0:53`. +On most systemd distributions the holder is `systemd-resolved`, which runs a stub listener on `127.0.0.53:53` and on some setups binds `0.0.0.0:53`. -**Fix.** Turn off the stub listener and keep resolved for the host's own -lookups: +**Fix.** Turn off the stub listener and keep resolved for the host's own lookups: ```sh mkdir -p /etc/systemd/resolved.conf.d @@ -217,22 +158,18 @@ printf '[Resolve]\nDNSStubListener=no\n' > /etc/systemd/resolved.conf.d/nxdns.co systemctl restart systemd-resolved ``` -If `/etc/resolv.conf` is a symlink to `/run/systemd/resolve/stub-resolv.conf`, -repoint it at `/run/systemd/resolve/resolv.conf` so the host still resolves. +If `/etc/resolv.conf` is a symlink to `/run/systemd/resolve/stub-resolv.conf`, repoint it at `/run/systemd/resolve/resolv.conf` so the host still resolves. > Not verified on this host: this needs root, and `systemd-resolved` is > inactive here with port 53 free, so the conflict could not be reproduced > against it. The bind failure itself was reproduced by starting a second nxdns > on a port the first already held, which is the same error path. -Do not fix this by pointing the host's `/etc/resolv.conf` at nxdns when that -host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a -startup cycle, not a fix. +Do not fix this by pointing the host's `/etc/resolv.conf` at nxdns when that host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a startup cycle, not a fix. ## The container restarts in a loop -**Symptom.** `docker compose ps` shows the container restarting, and the log is -the same failure repeated. Docker has no start limit, so this goes on forever. +**Symptom.** `docker compose ps` shows the container restarting, and the log is the same failure repeated. Docker has no start limit, so this goes on forever. ``` FAIL /etc/nxdns/config.zon: not readable @@ -246,10 +183,7 @@ docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}} restarts={{.Restar stat -c '%a %u:%g %n' deploy/docker/etc-nxdns/config.zon ``` -Exit 2 naming the configuration path means the container could not read the -file the shipped `command:` makes its configuration. The container runs as uid -65532 and `/etc/nxdns` is mounted read-only, so a file at mode 0600 owned by -your own uid is unreadable to it and the container cannot repair it. +Exit 2 naming the configuration path means the container could not read the file the shipped `command:` makes its configuration. The container runs as uid 65532 and `/etc/nxdns` is mounted read-only, so a file at mode 0600 owned by your own uid is unreadable to it and the container cannot repair it. **Fix.** Either make the file world-readable, when it holds no secret: @@ -264,17 +198,11 @@ chown 65532:65532 deploy/docker/etc-nxdns/config.zon chmod 0600 deploy/docker/etc-nxdns/config.zon ``` -The 0644 path was verified against an earlier revision of this page, including -the recovery: after the `chmod` the container started and answered queries. The -`chown` needs root and was not run here. +The 0644 path was verified against an earlier revision of this page, including the recovery: after the `chmod` the container started and answered queries. The `chown` needs root and was not run here. -`FAIL /etc/nxdns/config.zon: no such file` instead of `not readable` means there -is no configuration file at all. Create `deploy/docker/etc-nxdns/config.zon` and -bring it up again; see [Install with Docker](install-with-docker.md). +`FAIL /etc/nxdns/config.zon: no such file` instead of `not readable` means there is no configuration file at all. Create `deploy/docker/etc-nxdns/config.zon` and bring it up again; see [Install with Docker](install-with-docker.md). -A container that exits 2 with `NoUsableUpstreams` is in database mode — the -`command:` line naming `--config` was removed — on a volume whose database is -still empty. Load one and bring it back up: +A container that exits 2 with `NoUsableUpstreams` is in database mode — the `command:` line naming `--config` was removed — on a volume whose database is still empty. Load one and bring it back up: ```sh docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/config.zon @@ -293,9 +221,7 @@ docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/co {"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"} ``` -This is not a fault. The service runs `nxdns run --config`, which makes that -file the configuration, and configuration writes through the API are refused so -the file and the running server cannot drift apart. +This is not a fault. The service runs `nxdns run --config`, which makes that file the configuration, and configuration writes through the API are refused so the file and the running server cannot drift apart. **Diagnosis.** The start log names the authority: @@ -315,19 +241,13 @@ nxdns check --config /etc/nxdns/config.zon systemctl restart nxdns ``` -Or, if you want the interface to be how this box is configured, leave file mode: -drop `--config` from `ExecStart` and restart. The database already holds the -last reconciled state, so nothing is lost. See -[Run in file mode](install-with-systemd.md#run-in-file-mode). +Or, if you want the interface to be how this box is configured, leave file mode: drop `--config` from `ExecStart` and restart. The database already holds the last reconciled state, so nothing is lost. See [Run in file mode](install-with-systemd.md#run-in-file-mode). -Pausing blocking, refreshing blocklists and reloading certificates are not -configuration and keep working in file mode. Deleting a client works too, unless -the file names that client's address. +Pausing blocking, refreshing blocklists and reloading certificates are not configuration and keep working in file mode. Deleting a client works too, unless the file names that client's address. ## The container cannot reach its upstreams -**Symptom.** The container starts, but every query fails and `nxdns check` -inside it reports each upstream as unreachable. +**Symptom.** The container starts, but every query fails and `nxdns check` inside it reports each upstream as unreachable. **Diagnosis.** Look at what the host resolves with: @@ -335,15 +255,11 @@ inside it reports each upstream as unreachable. cat /etc/resolv.conf ``` -**Fix.** If it points at the nxdns container, repoint it at a real resolver. -The container resolves its upstream DoH and DoT hostnames through the host's -DNS configuration, so pointing that at nxdns makes nxdns depend on itself to -start. LAN clients point at nxdns; the container's own host does not. +**Fix.** If it points at the nxdns container, repoint it at a real resolver. The container resolves its upstream DoH and DoT hostnames through the host's DNS configuration, so pointing that at nxdns makes nxdns depend on itself to start. LAN clients point at nxdns; the container's own host does not. ## The disk is filling up -**Symptom.** Writes stop but DNS keeps answering. The journal shows the -transition: +**Symptom.** Writes stop but DNS keeps answering. The journal shows the transition: ``` warning(disk_monitor): disk state ok -> critical: 33349095424 bytes free on /var/lib/nxdns @@ -361,21 +277,11 @@ curl -s http://127.0.0.1:8080/api/health {"status":"degraded","disk":{"state":"critical","free_bytes":33349079040,"db_bytes":180224,"log_bytes":0,"sample_failures":0},"upstreams":{"available":1,"total":1},"queries_dropped":0,"writer_failed":false,"refreshes_gated":1,"snapshot_generation":2} ``` -`/metrics` carries the same free, database and log byte gauges as -`nxdns_disk_free_bytes`, `nxdns_disk_db_bytes` and `nxdns_disk_log_bytes`; the -state itself is on `/api/health`, not in the metrics output. +`/metrics` carries the same free, database and log byte gauges as `nxdns_disk_free_bytes`, `nxdns_disk_db_bytes` and `nxdns_disk_log_bytes`; the state itself is on `/api/health`, not in the metrics output. -**What the state means.** The monitor samples free space and database sizes -once a minute. Below `disk.warn_free_mb` it logs the transition. Below -`disk.min_free_mb` it gates every non-essential write: the query logger holds -its batches, the client tracker stops persisting, and blocklist refreshes are -skipped and counted in `refreshes_gated`. Resolution never degrades because the -disk is full — this was verified by setting the thresholds above the free space -on the volume: the state went critical, a refresh was gated, and queries kept -being answered. +**What the state means.** The monitor samples free space and database sizes once a minute. Below `disk.warn_free_mb` it logs the transition. Below `disk.min_free_mb` it gates every non-essential write: the query logger holds its batches, the client tracker stops persisting, and blocklist refreshes are skipped and counted in `refreshes_gated`. Resolution never degrades because the disk is full — this was verified by setting the thresholds above the free space on the volume: the state went critical, a refresh was gated, and queries kept being answered. -**Fix.** Recover space — lower `logging.retention_days`, or stop the service -and delete `querylog.db` — and writes resume on the next sample. +**Fix.** Recover space — lower `logging.retention_days`, or stop the service and delete `querylog.db` — and writes resume on the next sample. ## Blocklists are not filtering @@ -387,23 +293,13 @@ and delete `querylog.db` — and writes resume on the next sample. journalctl -u nxdns | grep 'serving on' ``` -It ends in either `blocklist generation N` or -`unfiltered (no blocklist snapshot)`. +It ends in either `blocklist generation N` or `unfiltered (no blocklist snapshot)`. -**Fix.** `unfiltered` means no snapshot loaded at all; the download or compile -warning that explains it is earlier in the same start. nxdns serves anyway on -purpose — a household loses more from DNS that refuses to start than from a -window of unfiltered answers. +**Fix.** `unfiltered` means no snapshot loaded at all; the download or compile warning that explains it is earlier in the same start. nxdns serves anyway on purpose — a household loses more from DNS that refuses to start than from a window of unfiltered answers. -A generation number with nothing being blocked is a different problem: the -snapshot loaded but has no sources in it. The line -`blocklist snapshot generation 1: 0 of 0 sources loaded` says exactly that. Add -a source in the admin interface, or a `blocklist_sources` entry to the -configuration file with a `group_sources` link naming a group. +A generation number with nothing being blocked is a different problem: the snapshot loaded but has no sources in it. The line `blocklist snapshot generation 1: 0 of 0 sources loaded` says exactly that. Add a source in the admin interface, or a `blocklist_sources` entry to the configuration file with a `group_sources` link naming a group. -One name resolving while its neighbours are blocked is a third case, and -`/api/lookup` answers it directly: it reports which level of the filtering -ladder decided, and against what. +One name resolving while its neighbours are blocked is a third case, and `/api/lookup` answers it directly: it reports which level of the filtering ladder decided, and against what. ```sh curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com' @@ -413,11 +309,7 @@ curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com' {"domain":"api.ads.tvb.com","group_id":1,"local_records":false,"forward_zone":null,"blocked":false,"reason":"blocklist_exception","matched":"api.ads.tvb.com","source_url":"https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt","safe_search_rewrite":null} ``` -`blocklist_exception` means a downloaded list lifted that name with an `@@` -line, and `source_url` names the list that did it. Nothing is broken, and the -list is not overruling you: an exception cancels only what another list blocks. -Your own rule wins over it. Adding an exact block rule for the same name and -asking again reports `rule_block_exact`, `blocked` true and a null `source_url`. +`blocklist_exception` means a downloaded list lifted that name with an `@@` line, and `source_url` names the list that did it. Nothing is broken, and the list is not overruling you: an exception cancels only what another list blocks. Your own rule wins over it. Adding an exact block rule for the same name and asking again reports `rule_block_exact`, `blocked` true and a null `source_url`. ## A database stamped by a newer binary @@ -428,6 +320,4 @@ warning(migrations): config.db is at schema version 99; this nxdns binary suppor nxdns run failed: SchemaTooNew ``` -**Fix.** There is no downgrade. Import the export you took before upgrading -into a fresh data directory with the older binary; see -[Upgrade nxdns](upgrade.md). +**Fix.** There is no downgrade. Import the export you took before upgrading into a fresh data directory with the older binary; see [Upgrade nxdns](upgrade.md). diff --git a/docs/how-to/upgrade.md b/docs/how-to/upgrade.md index a76f94d..ccf3147 100644 --- a/docs/how-to/upgrade.md +++ b/docs/how-to/upgrade.md @@ -1,11 +1,8 @@ # Upgrade nxdns -Replaces a running nxdns with a newer release without losing its -configuration. The database is migrated in place on the first start of the new -binary. +Replaces a running nxdns with a newer release without losing its configuration. The database is migrated in place on the first start of the new binary. -The normal path is to download the new release, verify it, and swap the binary. -Upgrading a build you made yourself is the last section of this page. +The normal path is to download the new release, verify it, and swap the binary. Upgrading a build you made yourself is the last section of this page. > Verification: the export, the migration behaviour and the `version`/`check` > steps below were run on the machine that wrote this page, against a @@ -23,38 +20,22 @@ Upgrading a build you made yourself is the last section of this page. ## Breaking change: `run --config` now means file authority -**Read this before upgrading if anything on your box passes `--config` to -`nxdns run`** — a systemd drop-in, a wrapper script, or a `command:` in a -compose file. +**Read this before upgrading if anything on your box passes `--config` to `nxdns run`** — a systemd drop-in, a wrapper script, or a `command:` in a compose file. -`run --config FILE` used to mean *seed once*: the file was read only while the -database was still empty, and ignored on every start after that. It now means -*the file is the configuration*: every start reconciles the database onto it. +`run --config FILE` used to mean *seed once*: the file was read only while the database was still empty, and ignored on every start after that. It now means *the file is the configuration*: every start reconciles the database onto it. -For a box that was seeded once and then configured through the admin interface, -the first start after the upgrade converges the database back to that old seed -file. **Every change made through the UI since seeding is deleted.** +For a box that was seeded once and then configured through the admin interface, the first start after the upgrade converges the database back to that old seed file. **Every change made through the UI since seeding is deleted.** There are two ways out, and you pick before you restart: -- **Keep the database.** Drop the flag. `nxdns run` with no `--config` serves - the database exactly as it did before, and nothing reads a file. This is the - right answer if the UI is how you change things. -- **Adopt file mode cleanly.** Install the new binary, stop the service, export - the current database over the file path, check it, then start with the flag. - The first reconcile is then a no-op, because the file was rendered from the - database it governs. **Install the new binary first** — see the order trap - below. The full procedure is - [Adopt file mode](install-with-systemd.md#adopt-file-mode-on-a-box-that-is-already-running). +- **Keep the database.** Drop the flag. `nxdns run` with no `--config` serves the database exactly as it did before, and nothing reads a file. This is the right answer if the UI is how you change things. +- **Adopt file mode cleanly.** Install the new binary, stop the service, export the current database over the file path, check it, then start with the flag. The first reconcile is then a no-op, because the file was rendered from the database it governs. **Install the new binary first** — see the order trap below. The full procedure is [Adopt file mode](install-with-systemd.md#adopt-file-mode-on-a-box-that-is-already-running). -`nxdns check --config FILE` is unchanged: it graded that file before and it -grades that file now. +`nxdns check --config FILE` is unchanged: it graded that file before and it grades that file now. ### The order trap: export with the new binary, not the old one -Take the export **after** you have replaced the binary, with the service -stopped. Exporting first — the instinctive order, and the one step 1 of this -page tells you to take for a backup — produces a file the new binary refuses. +Take the export **after** you have replaced the binary, with the service stopped. Exporting first — the instinctive order, and the one step 1 of this page tells you to take for a backup — produces a file the new binary refuses. A 0.0.1 `nxdns export` writes both fields: @@ -63,9 +44,7 @@ A 0.0.1 `nxdns export` writes both fields: .password_hash = "$argon2id$v=19$m=19456,t=2,p=1$…", ``` -An empty `password_hash` used to mean "unset". It now means "disable -authentication", so it is a *present* value — and a file that carries both -fields states two different things about the password and is refused: +An empty `password_hash` used to mean "unset". It now means "disable authentication", so it is a *present* value — and a file that carries both fields states two different things about the password and is refused: ``` FAIL web.password: password and password_hash are both set; ambiguity in a security setting is refused @@ -73,27 +52,18 @@ FAIL web.password: password is set to the empty string; omit the field to keep t nxdns run failed: PasswordAndHashBothSet ``` -The old `nxdns check` passes that file, because the old binary agreed with the -old rule. So the failure lands at the first start after the upgrade, with the -resolver stopped and the unit refusing to retry it -(`RestartPreventExitStatus=2 64`). A new `nxdns export` writes -`.password = null` instead and has no such problem. +The old `nxdns check` passes that file, because the old binary agreed with the old rule. So the failure lands at the first start after the upgrade, with the resolver stopped and the unit refusing to retry it (`RestartPreventExitStatus=2 64`). A new `nxdns export` writes `.password = null` instead and has no such problem. -**If you already have an old export you want to adopt**, you do not need to -redo it. Delete the empty-password line and the file is valid: +**If you already have an old export you want to adopt**, you do not need to redo it. Delete the empty-password line and the file is valid: ```sh sed -i '/^ \.password = "",$/d' /etc/nxdns/config.zon nxdns check --config /etc/nxdns/config.zon ``` -Keep the `.password_hash` line — that is the password, and deleting it as well -would leave the file saying nothing about authentication, which means "keep -whatever is stored" rather than anything you would notice. +Keep the `.password_hash` line — that is the password, and deleting it as well would leave the file saying nothing about authentication, which means "keep whatever is stored" rather than anything you would notice. -The same trap has nothing to do with file mode as such: it is any 0.0.1 export -fed to the new binary, so it also applies to a restore through `nxdns import`. -Backups taken with 0.0.1 need that one line removed before they will load. +The same trap has nothing to do with file mode as such: it is any 0.0.1 export fed to the new binary, so it also applies to a restore through `nxdns import`. Backups taken with 0.0.1 need that one line removed before they will load. > Verified on this host, with one substitution stated: the repository has no > 0.0.1 binary to hand, so the old export was **simulated** by taking a current @@ -107,16 +77,9 @@ Backups taken with 0.0.1 need that one line removed before they will load. > `git show v0.0.1:src/config/export.zig` line 71 is `cfg.web.password = "";` — > not from running that binary. -A database-mode install that never passed `--config` needs nothing. Under -Docker, a fresh database-mode install must either take the new compose file or -run `import` once — see -[Database mode in Docker](install-with-docker.md#database-mode-in-docker-instead). +A database-mode install that never passed `--config` needs nothing. Under Docker, a fresh database-mode install must either take the new compose file or run `import` once — see [Database mode in Docker](install-with-docker.md#database-mode-in-docker-instead). -Two smaller renames in the same release: `nxdns import --force` is now -`--allow-delete`, and it is required only when the file's diff would delete -rows rather than whenever the database is non-empty. `nxdns check` no longer -falls back to a default file path when there is no database; it reports the -absent database and names the two ways to get one. +Two smaller renames in the same release: `nxdns import --force` is now `--allow-delete`, and it is required only when the file's diff would delete rows rather than whenever the database is non-empty. `nxdns check` no longer falls back to a default file path when there is no database; it reports the absent database and names the two ways to get one. There is no schema migration in this change. @@ -128,16 +91,9 @@ There is no downgrade path, so the export is what you fall back to: nxdns export --out /some/backup/nxdns-config.zon ``` -`/some/backup` is a stand-in for a directory you keep backups in, and the -command relies on the default `--data-dir /var/lib/nxdns` that a systemd -install has. +`/some/backup` is a stand-in for a directory you keep backups in, and the command relies on the default `--data-dir /var/lib/nxdns` that a systemd install has. -This export is a fallback, not a file to deploy. If you are adopting file mode, -take a *second* export after the binary swap and use that one — an export -written by 0.0.1 carries a `.password = ""` line the new binary refuses, as -[the order trap](#the-order-trap-export-with-the-new-binary-not-the-old-one) -explains. The same line has to come out of this backup before the new binary -will import it. +This export is a fallback, not a file to deploy. If you are adopting file mode, take a *second* export after the binary swap and use that one — an export written by 0.0.1 carries a `.password = ""` line the new binary refuses, as [the order trap](#the-order-trap-export-with-the-new-binary-not-the-old-one) explains. The same line has to come out of this backup before the new binary will import it. > Verified on this host with both paths substituted, since it has neither > `/var/lib/nxdns` nor `/some/backup`. `SCRATCH` below is a scratch directory, @@ -156,15 +112,11 @@ will import it. > The shell umask was 022, so the 0600 is `export` setting it, not the umask. > Only the two paths differ from the command above. -The file is written atomically at mode 0600 and carries -`web.password_hash`, so treat it as a secret. See -[Back up and restore](back-up-and-restore.md) for the full backup story. The -query log is deliberately not part of it. +The file is written atomically at mode 0600 and carries `web.password_hash`, so treat it as a secret. See [Back up and restore](back-up-and-restore.md) for the full backup story. The query log is deliberately not part of it. ## 2. Download and verify the new release -Read the release notes for the version you are moving to before you take it — -the `CHANGELOG.md` section for that version is the release body. +Read the release notes for the version you are moving to before you take it — the `CHANGELOG.md` section for that version is the release body. ```sh BASE=https://git.mial.net/mokhtar/nxdns @@ -179,18 +131,11 @@ sha256sum -c --ignore-missing SHA256SUMS.txt tar -xzf "nxdns-$VERSION-x86_64-linux-musl.tar.gz" ``` -The first line asks the server which release is current, so this block does not -carry a version number that goes stale — Gitea redirects `releases/latest` to -the newest published release's tag page. To move to a particular version rather -than the newest, set `VERSION=` yourself. Check it against what you are -running (`nxdns version`) before you download anything. +The first line asks the server which release is current, so this block does not carry a version number that goes stale — Gitea redirects `releases/latest` to the newest published release's tag page. To move to a particular version rather than the newest, set `VERSION=` yourself. Check it against what you are running (`nxdns version`) before you download anything. -Take `aarch64-linux-musl` for a Raspberry Pi 5. Verify every time, not only on -the first install — an upgrade is a fresh download of a fresh artifact. -[Verify a release](verify-a-release.md) is the full procedure. +Take `aarch64-linux-musl` for a Raspberry Pi 5. Verify every time, not only on the first install — an upgrade is a fresh download of a fresh artifact. [Verify a release](verify-a-release.md) is the full procedure. -Under Docker there is nothing to download: step 3 pulls the image, and the -`IMAGE-DIGEST.txt` asset is what you verify instead. +Under Docker there is nothing to download: step 3 pulls the image, and the `IMAGE-DIGEST.txt` asset is what you verify instead. > Not verified on this host: no release exists yet, so the `releases/latest` > lookup returns 404 and leaves `VERSION` empty, and every `curl` below it is a @@ -201,8 +146,7 @@ Under Docker there is nothing to download: step 3 pulls the image, and the ### systemd -Step 2 leaves the new binary in the extracted directory. Copy the one that -matches the host — `aarch64-linux-musl` for a Raspberry Pi 5: +Step 2 leaves the new binary in the extracted directory. Copy the one that matches the host — `aarch64-linux-musl` for a Raspberry Pi 5: ```sh scp "nxdns-$VERSION-x86_64-linux-musl/nxdns" target:/tmp/nxdns @@ -212,9 +156,7 @@ scp "nxdns-$VERSION-x86_64-linux-musl/nxdns" target:/tmp/nxdns > nxdns, and this host has no such second machine to copy to. There is also no > release to have extracted. -The tarball also carries `nxdns.service` and `nxdns.conf`. An upgrade does not -normally reinstall them, but compare them against what is on the target when -the release notes say the unit changed. +The tarball also carries `nxdns.service` and `nxdns.conf`. An upgrade does not normally reinstall them, but compare them against what is on the target when the release notes say the unit changed. Then, as root on the target: @@ -237,15 +179,9 @@ NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml pull NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml up -d ``` -Compose recreates the container against the same `nxdns-data` volume. What -happens to the file in `etc-nxdns` depends on the `command:` in your compose -file: with the shipped `run --config=/etc/nxdns/config.zon` the file is the -configuration and the restart reconciles onto it; without it, the database in -the volume is the configuration and the file is read by nothing. +Compose recreates the container against the same `nxdns-data` volume. What happens to the file in `etc-nxdns` depends on the `command:` in your compose file: with the shipped `run --config=/etc/nxdns/config.zon` the file is the configuration and the restart reconciles onto it; without it, the database in the volume is the configuration and the file is read by nothing. -Set `NXDNS_VERSION` on both lines, or export it. Without it the compose file -falls back to `:latest`, and `pull` and `up` could then land on different -images if a release happens between them. +Set `NXDNS_VERSION` on both lines, or export it. Without it the compose file falls back to `:latest`, and `pull` and `up` could then land on different images if a release happens between them. > Not run on this host: `pull` needs a published image, and there is none. > What was run is `docker compose -f deploy/docker/compose.yaml config`, which @@ -262,19 +198,13 @@ nxdns check --config /tmp/after-upgrade.zon dig @127.0.0.1 example.com A +short ``` -The restart in step 3 is what migrated the database, so by now the schema is -current and the service is answering. Confirming with `nxdns check` alone would -not work here, and the reason is worth knowing: `check` opens `config.db` -immutable so it can never write to it, and the migration you just performed is -sitting in `config.db-wal` waiting to be checkpointed. Rather than read around -the log and grade older settings, `check` reports it: +The restart in step 3 is what migrated the database, so by now the schema is current and the service is answering. Confirming with `nxdns check` alone would not work here, and the reason is worth knowing: `check` opens `config.db` immutable so it can never write to it, and the migration you just performed is sitting in `config.db-wal` waiting to be checkpointed. Rather than read around the log and grade older settings, `check` reports it: ``` FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line. ``` -`export` opens the database read/write and does see the log, so exporting and -then checking the export validates what is actually in force: +`export` opens the database read/write and does see the log, so exporting and then checking the export validates what is actually in force: ``` checking configuration file /tmp/after-upgrade.zon @@ -301,20 +231,15 @@ OK: no problems found ## What happens to the database -Migrations run at startup, and also before `export` and `import`, so whichever -of those you run first performs the upgrade. `nxdns check` is the exception: it -opens the database immutable and never migrates, so on a database still one -version behind it reports the mismatch and exits 2 rather than fixing it: +Migrations run at startup, and also before `export` and `import`, so whichever of those you run first performs the upgrade. `nxdns check` is the exception: it opens the database immutable and never migrates, so on a database still one version behind it reports the mismatch and exits 2 rather than fixing it: ``` FAIL /var/lib/nxdns/config.db: schema version 0, this nxdns expects 1; `nxdns run` migrates it, `check` will not ``` -That line was reproduced here against a database stamped at version 0; the path -and the version numbers are what vary. +That line was reproduced here against a database stamped at version 0; the path and the version numbers are what vary. -A fresh database is created at the current schema version; an older one is -stepped up to it. The log line names both versions: +A fresh database is created at the current schema version; an older one is stepped up to it. The log line names both versions: ``` info(migrations): config.db migrated from schema version 0 to 1 @@ -326,9 +251,7 @@ info(migrations): config.db migrated from schema version 0 to 1 > rather than nothing. Version 1 is the only schema nxdns has published, so an > upgrade from a populated older one is not a case that exists yet. -Rolling back is the case that has no answer. A database stamped by a newer -binary refuses to open, so an older binary against an upgraded data directory -fails to start: +Rolling back is the case that has no answer. A database stamped by a newer binary refuses to open, so an older binary against an upgraded data directory fails to start: ``` warning(migrations): config.db is at schema version 99; this nxdns binary supports 1 @@ -340,39 +263,26 @@ nxdns run failed: SchemaTooNew > hand and `nxdns run` was pointed at it. The two lines above are that run's > output. -That run exits 1. Recovering means importing the export you took in step 1 into -a fresh data directory with the older binary. +That run exits 1. Recovering means importing the export you took in step 1 into a fresh data directory with the older binary. ### Rolling back from file mode -Putting an older binary back needs no unit edit. The old binary accepts -`run --config` — it just reads it as the old seed-once flag — and against a -database that already holds configuration it ignores the file entirely and -serves the last state the new binary reconciled. So the service comes back up -on the configuration it was running. +Putting an older binary back needs no unit edit. The old binary accepts `run --config` — it just reads it as the old seed-once flag — and against a database that already holds configuration it ignores the file entirely and serves the last state the new binary reconciled. So the service comes back up on the configuration it was running. -The consequence is worth stating plainly: **file edits stop applying.** The old -binary will not re-read the file, so every change made to `config.zon` after the -rollback does nothing at all, silently, until the newer binary is back. If you -have to stay on the old binary, use `nxdns import` to apply file changes, or drop -the flag so the invocation matches what the binary actually does. +The consequence is worth stating plainly: **file edits stop applying.** The old binary will not re-read the file, so every change made to `config.zon` after the rollback does nothing at all, silently, until the newer binary is back. If you have to stay on the old binary, use `nxdns import` to apply file changes, or drop the flag so the invocation matches what the binary actually does. -The schema note above still governs: a database stamped by a newer binary -refuses to open, whatever mode either binary runs in. +The schema note above still governs: a database stamped by a newer binary refuses to open, whatever mode either binary runs in. ## Changing settings, not the binary -How you change a setting depends on which authority the service runs under. -`nxdns run` in `ExecStart` means the database; `nxdns run --config FILE` means -the file. The start log names it either way: +How you change a setting depends on which authority the service runs under. `nxdns run` in `ExecStart` means the database; `nxdns run --config FILE` means the file. The start log names it either way: ``` info(nxdns): authority: database info(nxdns): authority: file (/etc/nxdns/config.zon) ``` -**In file mode**, edit the file, validate it, restart. The admin interface will -refuse the change with a 403 naming the file, so there is nothing to get wrong: +**In file mode**, edit the file, validate it, restart. The admin interface will refuse the change with a 403 naming the file, so there is nothing to get wrong: ```sh $EDITOR /etc/nxdns/config.zon @@ -380,8 +290,7 @@ nxdns check --config /etc/nxdns/config.zon systemctl restart nxdns ``` -**In database mode**, change settings through the admin interface, through the -API, or with an export–edit–import cycle against a stopped server: +**In database mode**, change settings through the admin interface, through the API, or with an export–edit–import cycle against a stopped server: ```sh nxdns export --out config-backup.zon @@ -391,19 +300,14 @@ nxdns import config-backup.zon systemctl start nxdns ``` -`import` needs no flag to add rows or to edit them. It needs `--allow-delete` -only when applying the file would delete rows the database holds — including the -case where you renamed something, since changing a group's name or an upstream's -URL is a delete and an insert to the engine, not an edit. The refusal names the -tables and rolls back: +`import` needs no flag to add rows or to edit them. It needs `--allow-delete` only when applying the file would delete rows the database holds — including the case where you renamed something, since changing a group's name or an upstream's URL is a delete and an insert to the engine, not an edit. The refusal names the tables and rolls back: ``` FAIL import: this file would delete rows the database holds (upstreams 1); re-run with --allow-delete to apply it import failed: DestructiveImport ``` -Stop the server first either way. `import` rewrites configuration underneath a -process that read it at startup, and a running server picks up only some of it. +Stop the server first either way. `import` rewrites configuration underneath a process that read it at startup, and a running server picks up only some of it. > Verified on this host against a populated scratch data directory, with > `--data-dir` pointing at it — that path is the only difference from the blocks @@ -429,8 +333,7 @@ process that read it at startup, and a running server picks up only some of it. ## Upgrading to a build of your own -If you are running something you built rather than a release, step 2 is a -build instead of a download: +If you are running something you built rather than a release, step 2 is a build instead of a download: ```sh (cd web && npm ci && npm run build) @@ -439,18 +342,9 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" -Dweb-dist=web/dist -Doptimize=ReleaseSafe ``` -Rebuild `web/dist` before the binary on every upgrade. The admin interface is -embedded at build time, and an old bundle against a new API is a broken -settings page. `dist` refuses the `web/dist-placeholder` default outright, so -the only way to ship a stale bundle is to leave an old `web/dist` in place. +Rebuild `web/dist` before the binary on every upgrade. The admin interface is embedded at build time, and an old bundle against a new API is a broken settings page. `dist` refuses the `web/dist-placeholder` default outright, so the only way to ship a stale bundle is to leave an old `web/dist` in place. -The staged payload for each target is under -`zig-out/dist/stage/nxdns--/`, and step 3 continues from there -with that path in place of the extracted one. The version string has to equal -`.version` in `build.zig.zon` — `verify-dist` asserts it, so a made-up one -builds and then fails verification. What tells your build apart from the -published release of the same version is `-Dgit-commit`, which `nxdns version` -prints beside the version. +The staged payload for each target is under `zig-out/dist/stage/nxdns--/`, and step 3 continues from there with that path in place of the extracted one. The version string has to equal `.version` in `build.zig.zon` — `verify-dist` asserts it, so a made-up one builds and then fails verification. What tells your build apart from the published release of the same version is `-Dgit-commit`, which `nxdns version` prints beside the version. Under Docker, build the image and name it instead of pulling: diff --git a/docs/how-to/verify-a-release.md b/docs/how-to/verify-a-release.md index efece58..49a5eae 100644 --- a/docs/how-to/verify-a-release.md +++ b/docs/how-to/verify-a-release.md @@ -1,12 +1,8 @@ # Verify a release -Checks that a downloaded nxdns release is the one the project published and -that it arrived intact. It also gives the recipe for rebuilding the same -version from source, and says plainly what that does and does not settle. +Checks that a downloaded nxdns release is the one the project published and that it arrived intact. It also gives the recipe for rebuilding the same version from source, and says plainly what that does and does not settle. -Do this before you run the binary, not after. The whole point of the checksum -file is that it is signed, so a tampered mirror cannot hand you a matching -tarball and a matching checksum at the same time. +Do this before you run the binary, not after. The whole point of the checksum file is that it is signed, so a tampered mirror cannot hand you a matching tarball and a matching checksum at the same time. > Verification: every command on this page was run on 2026-08-09 against the > published `v0.0.1` release, from a clean directory, with a clean `GNUPGHOME` @@ -17,8 +13,7 @@ tarball and a matching checksum at the same time. ## What a release contains -Five assets, on the release page at -`https://git.mial.net/mokhtar/nxdns/releases`: +Five assets, on the release page at `https://git.mial.net/mokhtar/nxdns/releases`: | Asset | What it is | | --- | --- | @@ -28,28 +23,15 @@ Five assets, on the release page at | `SHA256SUMS.txt.asc` | A detached OpenPGP signature over `SHA256SUMS.txt` | | `IMAGE-DIGEST.txt` | The container image reference this version pushed, pinned by digest | -Each tarball holds one top-level directory, `nxdns--/`, with -six files in it: the `nxdns` binary at mode 0755, and `nxdns.service`, -`nxdns.conf`, `LICENSE`, `THIRD-PARTY-NOTICES` and `INSTALL.md` at 0644. +Each tarball holds one top-level directory, `nxdns--/`, with six files in it: the `nxdns` binary at mode 0755, and `nxdns.service`, `nxdns.conf`, `LICENSE`, `THIRD-PARTY-NOTICES` and `INSTALL.md` at 0644. -`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt` rather than the image, because the -image digest does not exist until the push has happened and cannot be computed -by the build. Signing the file that names the digest gets you the same -guarantee in one signature. +`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt` rather than the image, because the image digest does not exist until the push has happened and cannot be computed by the build. Signing the file that names the digest gets you the same guarantee in one signature. -The `.txt` on three of the five names is not decoration. Gitea decides what an -attachment may be by its file extension, and whether it accepts an -extensionless upload at all is untested against this instance, so the release -uses names it is known to accept. On disk, `zig build dist` still writes a file -called `SHA256SUMS`; the release job copies it to `SHA256SUMS.txt` and appends -the image line before signing. +The `.txt` on three of the five names is not decoration. Gitea decides what an attachment may be by its file extension, and whether it accepts an extensionless upload at all is untested against this instance, so the release uses names it is known to accept. On disk, `zig build dist` still writes a file called `SHA256SUMS`; the release job copies it to `SHA256SUMS.txt` and appends the image line before signing. ## 1. Pick a version -Every URL below takes the version from one shell variable. Ask the server -rather than typing a number that goes stale: Gitea redirects `releases/latest` -to the tag page of the newest published release — newest by publication time, -and drafts and pre-releases are excluded. +Every URL below takes the version from one shell variable. Ask the server rather than typing a number that goes stale: Gitea redirects `releases/latest` to the tag page of the newest published release — newest by publication time, and drafts and pre-releases are excluded. ```sh BASE=https://git.mial.net/mokhtar/nxdns @@ -58,8 +40,7 @@ VERSION=$(curl -fsS -o /dev/null -w '%{redirect_url}' "$BASE/releases/latest" | echo "$VERSION" ``` -To take a particular version instead, set it yourself — substitute the one you -want for the placeholder: +To take a particular version instead, set it yourself — substitute the one you want for the placeholder: ```sh VERSION= @@ -68,8 +49,7 @@ VERSION= > Verified: the two-command form, run against this repository, printed `0.0.1` > with `v0.0.1` published. -Pin the version in anything you script or automate. `latest` is convenient for -a person at a terminal and a liability in a machine that upgrades itself. +Pin the version in anything you script or automate. `latest` is convenient for a person at a terminal and a liability in a machine that upgrades itself. ## 2. Download @@ -81,8 +61,7 @@ curl -fLO "$BASE/releases/download/v$VERSION/SHA256SUMS.txt.asc" curl -fLO "$BASE/releases/download/v$VERSION/IMAGE-DIGEST.txt" ``` -`-L` is not optional: Gitea answers an asset URL with a 303 to wherever the -attachment is actually stored. +`-L` is not optional: Gitea answers an asset URL with a 303 to wherever the attachment is actually stored. For the aarch64 tarball, or for both, swap or add the filename: @@ -90,8 +69,7 @@ For the aarch64 tarball, or for both, swap or add the filename: curl -fLO "$BASE/releases/download/v$VERSION/nxdns-$VERSION-aarch64-linux-musl.tar.gz" ``` -Gitea also accepts the literal word `latest` in place of the tag, so the three -assets whose names carry no version can be fetched without one: +Gitea also accepts the literal word `latest` in place of the tag, so the three assets whose names carry no version can be fetched without one: ```sh curl -fLO "$BASE/releases/download/latest/SHA256SUMS.txt" @@ -99,10 +77,7 @@ curl -fLO "$BASE/releases/download/latest/SHA256SUMS.txt.asc" curl -fLO "$BASE/releases/download/latest/IMAGE-DIGEST.txt" ``` -That is the Gitea spelling, and it is not GitHub's. `releases/latest/download/` -— the form GitHub uses — is a 404 on Gitea; the alias goes in the tag -position, as `releases/download/latest/`. The tarball filenames contain the -version, so this alias never saves you from knowing it for those two. +That is the Gitea spelling, and it is not GitHub's. `releases/latest/download/` — the form GitHub uses — is a 404 on Gitea; the alias goes in the tag position, as `releases/download/latest/`. The tarball filenames contain the version, so this alias never saves you from knowing it for those two. > Verified against `v0.0.1`: all five assets downloaded through the versioned > path, `SHA256SUMS.txt` downloaded again through the `latest` alias and hashed @@ -110,9 +85,7 @@ version, so this alias never saves you from knowing it for those two. ## 3. Check the signature over `SHA256SUMS.txt` -Get the public key first. It is a signing subkey of the key that signs every -commit in this repository, so you can confirm the fingerprint against a clone -you already have with `git log --show-signature` or `git verify-tag v$VERSION`: +Get the public key first. It is a signing subkey of the key that signs every commit in this repository, so you can confirm the fingerprint against a clone you already have with `git log --show-signature` or `git verify-tag v$VERSION`: ``` A2061F6AB24DF2C0E92346FD1509B54946D08A95 @@ -143,32 +116,18 @@ Primary key fingerprint: A206 1F6A B24D F2C0 E923 46FD 1509 B549 46D0 8A95 Subkey fingerprint: 019D 00DF 8417 EBFD A547 1E5E F731 9CC0 24FB 5A96 ``` -The *structure* is what to read: three lines, not one. `using EDDSA key` and -`Subkey fingerprint` name the signing subkey that actually made the signature; -`Primary key fingerprint` names the certificate it hangs off, and that is the -one published above. The subkey fingerprint can change — a signing subkey is -revoked and replaced on its own — but the primary fingerprint is the -project's identity and stays. +The *structure* is what to read: three lines, not one. `using EDDSA key` and `Subkey fingerprint` name the signing subkey that actually made the signature; `Primary key fingerprint` names the certificate it hangs off, and that is the one published above. The subkey fingerprint can change — a signing subkey is revoked and replaced on its own — but the primary fingerprint is the project's identity and stays. -Exit status 0, and `Good signature`. That warning is normal and is not a -failure: it says you have not told GnuPG you believe the key belongs to the -person it claims to. +Exit status 0, and `Good signature`. That warning is normal and is not a failure: it says you have not told GnuPG you believe the key belongs to the person it claims to. -Now compare the `Primary key fingerprint` line with the fingerprint in this -page. GnuPG prints it as ten space-separated groups of four hex digits, with a -double space in the middle, while the fingerprint above is the same 40 -characters unspaced — so compare the hex digits in order and ignore the -spacing, or strip it and let the shell do it: +Now compare the `Primary key fingerprint` line with the fingerprint in this page. GnuPG prints it as ten space-separated groups of four hex digits, with a double space in the middle, while the fingerprint above is the same 40 characters unspaced — so compare the hex digits in order and ignore the spacing, or strip it and let the shell do it: ```sh gpg --verify SHA256SUMS.txt.asc SHA256SUMS.txt 2>&1 | sed -n 's/^Primary key fingerprint: //p' | tr -d ' ' ``` -That prints the 40-character form, ready to compare with -`A2061F6AB24DF2C0E92346FD1509B54946D08A95`. Do not skip the comparison — -`gpg --verify` exits 0 for a good signature from *any* key in your keyring, -including one an attacker talked you into importing. +That prints the 40-character form, ready to compare with `A2061F6AB24DF2C0E92346FD1509B54946D08A95`. Do not skip the comparison — `gpg --verify` exits 0 for a good signature from *any* key in your keyring, including one an attacker talked you into importing. A tampered `SHA256SUMS.txt` looks like this, and exits 1: @@ -197,9 +156,7 @@ nxdns--x86_64-linux-musl.tar.gz: OK IMAGE-DIGEST.txt: OK ``` -`--ignore-missing` is what makes this work when you downloaded one tarball out -of the two. Without it, `sha256sum` treats every line it cannot read as a -failure and exits 1: +`--ignore-missing` is what makes this work when you downloaded one tarball out of the two. Without it, `sha256sum` treats every line it cannot read as a failure and exits 1: ``` nxdns--x86_64-linux-musl.tar.gz: OK @@ -209,8 +166,7 @@ IMAGE-DIGEST.txt: OK sha256sum: WARNING: 1 listed file could not be read ``` -A file that is present but does not match is the case that matters, and it -says `FAILED` with no `open or read`: +A file that is present but does not match is the case that matters, and it says `FAILED` with no `open or read`: ``` nxdns--x86_64-linux-musl.tar.gz: FAILED @@ -218,9 +174,7 @@ IMAGE-DIGEST.txt: OK sha256sum: WARNING: 1 computed checksum did NOT match ``` -Check the signature before the hashes, not after. An attacker who can replace -the tarball can replace `SHA256SUMS.txt` next to it; the signature is the only -thing in the set they cannot forge. +Check the signature before the hashes, not after. An attacker who can replace the tarball can replace `SHA256SUMS.txt` next to it; the signature is the only thing in the set they cannot forge. > Verified against `v0.0.1`: with both tarballs present, `sha256sum -c` printed > three `OK` lines. The three transcripts above are the same command over @@ -234,12 +188,7 @@ thing in the set they cannot forge. tar -tvzf nxdns-$VERSION-x86_64-linux-musl.tar.gz ``` -Expect exactly one top-level directory and the six files listed above, with -mode `-rwxr-xr-x` on `nxdns` and `-rw-r--r--` on the rest, no symlinks, and no -path that begins with `/` or contains `..`. `zig build verify-dist` asserts all -of that on the extracted archive before a release is ever published, so this is -a second opinion rather than the only check — but it costs nothing and it is -the step that catches a tarball that is not the one you think it is. +Expect exactly one top-level directory and the six files listed above, with mode `-rwxr-xr-x` on `nxdns` and `-rw-r--r--` on the rest, no symlinks, and no path that begins with `/` or contains `..`. `zig build verify-dist` asserts all of that on the extracted archive before a release is ever published, so this is a second opinion rather than the only check — but it costs nothing and it is the step that catches a tarball that is not the one you think it is. Then extract: @@ -248,9 +197,7 @@ tar -xzf nxdns-$VERSION-x86_64-linux-musl.tar.gz ./nxdns-$VERSION-x86_64-linux-musl/nxdns version ``` -`version` prints the version and the git commit it was built from, then the -Zig version. The version has to match the tag you downloaded, and the commit -has to match the commit the tag points at. +`version` prints the version and the git commit it was built from, then the Zig version. The version has to match the tag you downloaded, and the commit has to match the commit the tag points at. > Verified against `v0.0.1`: both tarballs listed exactly the one directory and > six files with the stated modes, no symlinks and no absolute or `..` paths, @@ -260,16 +207,13 @@ has to match the commit the tag points at. ## 6. Verify the container image -`IMAGE-DIGEST.txt` holds one line: the image reference this version pushed, -pinned by the digest of its index, in the form +`IMAGE-DIGEST.txt` holds one line: the image reference this version pushed, pinned by the digest of its index, in the form ``` git.mial.net/mokhtar/nxdns:@sha256:<64 hex digits> ``` -`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt`, so the signature you already -checked covers that line too. Confirm the tag in the registry still resolves to -that digest: +`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt`, so the signature you already checked covers that line too. Confirm the tag in the registry still resolves to that digest: ```sh cut -d@ -f2 IMAGE-DIGEST.txt @@ -277,8 +221,7 @@ docker buildx imagetools inspect git.mial.net/mokhtar/nxdns:$VERSION \ --format '{{.Manifest.Digest}}' ``` -The two have to be the same string. A registry tag is mutable; the digest is -not, so pull the whole pinned reference rather than the tag when you care: +The two have to be the same string. A registry tag is mutable; the digest is not, so pull the whole pinned reference rather than the tag when you care: ```sh docker pull "$(cat IMAGE-DIGEST.txt)" @@ -291,12 +234,9 @@ docker buildx imagetools inspect git.mial.net/mokhtar/nxdns:$VERSION \ --format '{{range .Manifest.Manifests}}{{.Platform.OS}}/{{.Platform.Architecture}} {{end}}' ``` -`linux/amd64 linux/arm64`. The build passes `--provenance=false --sbom=false`, -so there are no `unknown/unknown` attestation entries in the list; seeing any -means the image did not come from this pipeline. +`linux/amd64 linux/arm64`. The build passes `--provenance=false --sbom=false`, so there are no `unknown/unknown` attestation entries in the list; seeing any means the image did not come from this pipeline. -The binary inside the image is the same file as the one in the matching -tarball, and the release checks that before publishing. To check it yourself: +The binary inside the image is the same file as the one in the matching tarball, and the release checks that before publishing. To check it yourself: ```sh docker create --name nxdns-verify git.mial.net/mokhtar/nxdns:$VERSION @@ -317,35 +257,18 @@ sha256sum ./nxdns-from-image ./nxdns-$VERSION-x86_64-linux-musl/nxdns It proves two things: -- The release was produced by this project's release pipeline, using a key - only that pipeline holds. -- What you have on disk is byte for byte what that pipeline uploaded. A - corrupted download, a modified mirror or a tampered proxy all break the - check. +- The release was produced by this project's release pipeline, using a key only that pipeline holds. +- What you have on disk is byte for byte what that pipeline uploaded. A corrupted download, a modified mirror or a tampered proxy all break the check. -It does not prove that the binary in the tarball was built from the source in -this repository. The machine that ran the build also held the signing key, so -a compromise of that machine produces an artifact that is signed, verifies -cleanly, and contains whatever the attacker put in it. The signature is a -statement about origin and integrity in transit. It is not a statement about -provenance from source. +It does not prove that the binary in the tarball was built from the source in this repository. The machine that ran the build also held the signing key, so a compromise of that machine produces an artifact that is signed, verifies cleanly, and contains whatever the attacker put in it. The signature is a statement about origin and integrity in transit. It is not a statement about provenance from source. -Closing that gap needs a reproducibility gate — an independent build, run -somewhere else, that lands on the same bytes — and this project does not have -one. It is a recorded deferral, not an oversight: see `specs/milestone-14.md` -ruling 12. Until it exists, nothing here claims the build is reproducible, -because nobody has measured whether it is. +Closing that gap needs a reproducibility gate — an independent build, run somewhere else, that lands on the same bytes — and this project does not have one. It is a recorded deferral, not an oversight: see `specs/milestone-14.md` ruling 12. Until it exists, nothing here claims the build is reproducible, because nobody has measured whether it is. -The signing key is a subkey rather than the primary key, which limits the -damage of the case above: a leaked release subkey is revoked on its own and -the identity, the commit signatures and everyone's existing trust in the key -survive. +The signing key is a subkey rather than the primary key, which limits the damage of the case above: a leaked release subkey is revoked on its own and the identity, the commit signatures and everyone's existing trust in the key survive. ## Rebuild it yourself -You can still build the same version from source and compare. That gets you a -binary whose provenance you know, and the comparison is worth making — read -the paragraph after the recipe before you draw a conclusion from it. +You can still build the same version from source and compare. That gets you a binary whose provenance you know, and the comparison is worth making — read the paragraph after the recipe before you draw a conclusion from it. ```sh git clone https://git.mial.net/mokhtar/nxdns @@ -358,30 +281,13 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" sha256sum zig-out/dist/nxdns-"$VERSION"-*.tar.gz ``` -`git verify-tag` is the check that the tag itself is signed by the key from -step 3, and it is the one part of this section that stands on its own: it ties -the source you just checked out to the same identity that signed the release. +`git verify-tag` is the check that the tag itself is signed by the key from step 3, and it is the one part of this section that stands on its own: it ties the source you just checked out to the same identity that signed the release. -`zig build dist` writes `zig-out/dist/`: the two tarballs, a staging directory -per target under `stage/`, the stripped binaries under `bin//`, and a -`SHA256SUMS` covering the two tarballs. The published `SHA256SUMS.txt` is that -file with a third line for `IMAGE-DIGEST.txt` appended by the release job, so -the two tarball lines should match and the local file has no third line to -compare. +`zig build dist` writes `zig-out/dist/`: the two tarballs, a staging directory per target under `stage/`, the stripped binaries under `bin//`, and a `SHA256SUMS` covering the two tarballs. The published `SHA256SUMS.txt` is that file with a third line for `IMAGE-DIGEST.txt` appended by the release job, so the two tarball lines should match and the local file has no third line to compare. -Now the caveat, and it is the whole reason this section is last. **A hash that -differs does not mean the release was tampered with.** Nothing in this project -measures whether two builds of the same commit on two different machines -produce the same bytes, and there are several ordinary reasons they would not: -a different Zig patch release, a different Node version, a different path to -the build directory, a different npm lockfile resolution. A hash that matches -is real evidence. A hash that does not match tells you only that something -about the two builds differed, and finding out what is on you. +Now the caveat, and it is the whole reason this section is last. **A hash that differs does not mean the release was tampered with.** Nothing in this project measures whether two builds of the same commit on two different machines produce the same bytes, and there are several ordinary reasons they would not: a different Zig patch release, a different Node version, a different path to the build directory, a different npm lockfile resolution. A hash that matches is real evidence. A hash that does not match tells you only that something about the two builds differed, and finding out what is on you. -If you want the comparison to mean as much as it can, match the toolchain the -release used. The Zig version is the second line of `nxdns version`, and both -it and the Node version are pinned to exact patch releases at the top of -`.gitea/workflows/gates.yml`, which is the workflow the release runs. +If you want the comparison to mean as much as it can, match the toolchain the release used. The Zig version is the second line of `nxdns version`, and both it and the Node version are pinned to exact patch releases at the top of `.gitea/workflows/gates.yml`, which is the workflow the release runs. > Verified against `v0.0.1`, and the result is the caveat above in action. The > whole recipe ran from a fresh clone: `git verify-tag v0.0.1` printed @@ -398,25 +304,14 @@ it and the Node version are pinned to exact patch releases at the top of Stop and do not run the binary. -- `sha256sum` says `FAILED` but the signature was good — you have a damaged or - substituted download. Delete it and fetch it again over a different network - before assuming anything worse. -- `gpg` says `BAD signature` — `SHA256SUMS.txt` and `SHA256SUMS.txt.asc` do not - belong together. Re-download both from the release page; a stale - `SHA256SUMS.txt.asc` left over from a previous version is the boring - explanation. -- `gpg` says `Can't check signature: No public key` — you have not imported the - key, or you imported a different one. -- The fingerprint does not match the one in step 3 — that is the case to take - seriously. Do not extract the tarball, and do not import more keys trying to - make it pass. +- `sha256sum` says `FAILED` but the signature was good — you have a damaged or substituted download. Delete it and fetch it again over a different network before assuming anything worse. +- `gpg` says `BAD signature` — `SHA256SUMS.txt` and `SHA256SUMS.txt.asc` do not belong together. Re-download both from the release page; a stale `SHA256SUMS.txt.asc` left over from a previous version is the boring explanation. +- `gpg` says `Can't check signature: No public key` — you have not imported the key, or you imported a different one. +- The fingerprint does not match the one in step 3 — that is the case to take seriously. Do not extract the tarball, and do not import more keys trying to make it pass. ## Related -- [Install with systemd](install-with-systemd.md) — where the verified tarball - goes next. +- [Install with systemd](install-with-systemd.md) — where the verified tarball goes next. - [Install with Docker](install-with-docker.md) — the published image. -- [Upgrade nxdns](upgrade.md) — the same verification, on the way to a newer - version. -- [Performance targets and what the tests prove](../explanation/performance-and-testing.md) - — the other place this project writes down what its checks do not cover. +- [Upgrade nxdns](upgrade.md) — the same verification, on the way to a newer version. +- [Performance targets and what the tests prove](../explanation/performance-and-testing.md) — the other place this project writes down what its checks do not cover. diff --git a/docs/reference/api.md b/docs/reference/api.md index e0ea5db..0344241 100644 --- a/docs/reference/api.md +++ b/docs/reference/api.md @@ -1,159 +1,74 @@ # REST API reference -nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080), -as plain HTTP. TLS termination, where an operator wants it, belongs to a reverse -proxy in front; the session cookie deliberately omits the `Secure` attribute so -the supported plain-HTTP LAN deployment works. +nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080), as plain HTTP. TLS termination, where an operator wants it, belongs to a reverse proxy in front; the session cookie deliberately omits the `Secure` attribute so the supported plain-HTTP LAN deployment works. -The machine-readable contract is `src/web/openapi.yaml`, which the running -server hands out unauthenticated at `GET /api/openapi.yaml`. Request and -response schemas for every operation live there. When this page and the YAML -disagree, the YAML wins. +The machine-readable contract is `src/web/openapi.yaml`, which the running server hands out unauthenticated at `GET /api/openapi.yaml`. Request and response schemas for every operation live there. When this page and the YAML disagree, the YAML wins. -The route table is `src/web/routes.zig`; the [Operations](#operations) table -below carries all 56 of its entries. +The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 56 of its entries. ## Conventions -- All request and response bodies are JSON (`application/json`), except - `/metrics` (Prometheus text format), `/api/openapi.yaml` (YAML) and - `/api/queries/live` (`text/event-stream`). +- All request and response bodies are JSON (`application/json`), except `/metrics` (Prometheus text format), `/api/openapi.yaml` (YAML) and `/api/queries/live` (`text/event-stream`). - Field names are snake_case, matching settings keys and SQL column names. -- Every error response carries the envelope `{"error": ""}`. The - message is operator-facing text; internal detail never reaches the wire — a - 500 body is generic and the cause goes to the server log. +- Every error response carries the envelope `{"error": ""}`. The message is operator-facing text; internal detail never reaches the wire — a 500 body is generic and the cause goes to the server log. - Request bodies are strict: an unknown field is a 400, a body over 1 MiB is a 413. -- A request whose path matches but whose method does not answers 405 with an - `Allow` header. An unknown `/api` path is a JSON 404; unknown non-`/api` paths - fall through to the embedded SPA (`index.html`), so client-side routing works. +- A request whose path matches but whose method does not answers 405 with an `Allow` header. An unknown `/api` path is a JSON 404; unknown non-`/api` paths fall through to the embedded SPA (`index.html`), so client-side routing works. - Item routes (`{id}`) match a positive integer id only. -- Mutations to groups, blocklists, rules, local records, forward zones, clients - and client prefixes take effect live. Upstreams and `/api/settings` are - restart-required. -- Every route has a policy class — `read`, `config_write` or `runtime_action` — - and in file mode the `config_write` routes are refused. See - [Configuration authority](#configuration-authority). +- Mutations to groups, blocklists, rules, local records, forward zones, clients and client prefixes take effect live. Upstreams and `/api/settings` are restart-required. +- Every route has a policy class — `read`, `config_write` or `runtime_action` — and in file mode the `config_write` routes are refused. See [Configuration authority](#configuration-authority). ## Authentication -Cookie sessions, in memory, no accounts — one operator password. Setting that -password is [set up admin -authentication](../how-to/set-up-admin-authentication.md). +Cookie sessions, in memory, no accounts — one operator password. Setting that password is [set up admin authentication](../how-to/set-up-admin-authentication.md). -- Authentication is on exactly when `web.password_hash` is set. When no password - is set, every route is open and `POST /api/auth/login` answers - `{"authenticated": true, "auth_required": false}` without setting a cookie. -- `POST /api/auth/login` takes `{"password": "..."}`. A correct password answers - 200 with a `Set-Cookie` for `nxdns_session` (`HttpOnly; SameSite=Lax; Path=/`, - `Max-Age` = the session TTL). A wrong password is a 401; a stored hash the - server cannot read is a 500, never a 401. Login attempts spend rate-limit - tokens like any other request, and argon2id verification is deliberately slow. -- Every route whose auth policy is `session` answers 401 - `{"error": "authentication required"}` without a valid cookie. -- Sessions live `web.session_ttl_hours` (default 24) from login; use does not - extend the lifetime. The table holds 32 sessions; a 33rd login evicts the - least recently used. Nothing is persisted — a server restart logs every - operator out. -- Changing the password through `PUT /api/settings` revokes every live session - immediately; the new password applies without a restart. -- `POST /api/auth/logout` ends the cookie's session and clears the cookie. The - route is `.session` like any other, so with a password configured the router - answers 401 before the handler runs when the cookie is missing, expired, - revoked or already logged out; only a live session gets the 200. With no - password configured every session route is open and logout answers 200. +- Authentication is on exactly when `web.password_hash` is set. When no password is set, every route is open and `POST /api/auth/login` answers `{"authenticated": true, "auth_required": false}` without setting a cookie. +- `POST /api/auth/login` takes `{"password": "..."}`. A correct password answers 200 with a `Set-Cookie` for `nxdns_session` (`HttpOnly; SameSite=Lax; Path=/`, `Max-Age` = the session TTL). A wrong password is a 401; a stored hash the server cannot read is a 500, never a 401. Login attempts spend rate-limit tokens like any other request, and argon2id verification is deliberately slow. +- Every route whose auth policy is `session` answers 401 `{"error": "authentication required"}` without a valid cookie. +- Sessions live `web.session_ttl_hours` (default 24) from login; use does not extend the lifetime. The table holds 32 sessions; a 33rd login evicts the least recently used. Nothing is persisted — a server restart logs every operator out. +- Changing the password through `PUT /api/settings` revokes every live session immediately; the new password applies without a restart. +- `POST /api/auth/logout` ends the cookie's session and clears the cookie. The route is `.session` like any other, so with a password configured the router answers 401 before the handler runs when the cookie is missing, expired, revoked or already logged out; only a live session gets the 200. With no password configured every session route is open and logout answers 200. ## Rate limiting -A token bucket per client address: capacity and refill are both -`web.api_rate_limit_per_min` (default 300) per minute, so a page-load burst up -to the capacity is admitted and the long-run rate holds. +A token bucket per client address: capacity and refill are both `web.api_rate_limit_per_min` (default 300) per minute, so a page-load burst up to the capacity is admitted and the long-run rate holds. -- An over-budget request answers 429 `{"error": "rate limited"}` with a - `Retry-After` header giving the seconds until a token is available (rounded - up, never zero). -- Loopback addresses (127.0.0.0/8 and ::1) are exempt while - `web.api_localhost_exempt` is true (the default). -- The address a bucket keys on is the socket peer, unless that peer is listed in - `web.trusted_proxies`. For a listed peer the address is instead the **last** - entry of the request's `X-Forwarded-For` — the entry the proxy appended, which - is the only one a client cannot write. A request from a trusted proxy with no - such header keys on the proxy itself; one whose last entry is not an IP - literal is answered 400, because the alternative is granting the proxy's own - loopback exemption to whoever sent it. Only `X-Forwarded-For` is read; - `Forwarded` (RFC 7239) and the PROXY protocol are not. -- Without `web.trusted_proxies`, a same-box reverse proxy makes every request - loopback, so the default exemption disables the limiter for all remote - clients. Set the proxy's address there, or set - `web.api_localhost_exempt = false`. -- Exempt routes, which never consult a bucket: `/metrics` and `/api/health` (a - Prometheus scrape must never see 429) and `/api/queries/live` (one long-lived - stream must not drain its address's bucket; it is bounded by the SSE - connection cap instead). -- The limiter tracks at most 4096 addresses. When the table is full and no slot - is reclaimable, requests from unknown addresses are refused with 429. +- An over-budget request answers 429 `{"error": "rate limited"}` with a `Retry-After` header giving the seconds until a token is available (rounded up, never zero). +- Loopback addresses (127.0.0.0/8 and ::1) are exempt while `web.api_localhost_exempt` is true (the default). +- The address a bucket keys on is the socket peer, unless that peer is listed in `web.trusted_proxies`. For a listed peer the address is instead the **last** entry of the request's `X-Forwarded-For` — the entry the proxy appended, which is the only one a client cannot write. A request from a trusted proxy with no such header keys on the proxy itself; one whose last entry is not an IP literal is answered 400, because the alternative is granting the proxy's own loopback exemption to whoever sent it. Only `X-Forwarded-For` is read; `Forwarded` (RFC 7239) and the PROXY protocol are not. +- Without `web.trusted_proxies`, a same-box reverse proxy makes every request loopback, so the default exemption disables the limiter for all remote clients. Set the proxy's address there, or set `web.api_localhost_exempt = false`. +- Exempt routes, which never consult a bucket: `/metrics` and `/api/health` (a Prometheus scrape must never see 429) and `/api/queries/live` (one long-lived stream must not drain its address's bucket; it is bounded by the SSE connection cap instead). +- The limiter tracks at most 4096 addresses. When the table is full and no slot is reclaimable, requests from unknown addresses are refused with 429. ## Live query stream (SSE) -`GET /api/queries/live` is server-sent events over chunked transfer, -`Content-Type: text/event-stream`, `Cache-Control: no-store`. +`GET /api/queries/live` is server-sent events over chunked transfer, `Content-Type: text/event-stream`, `Cache-Control: no-store`. -- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on - its own after a drop. -- Each query is one frame: `event: query` and a single `data:` line of JSON. The - payload carries the `GET /api/queries` row fields minus `id` (a live entry - precedes persistence): `ts`, `domain`, `client_ip`, `qtype`, `blocked`, - `block_reason`, `response_time_us`, `cache_hit`, `upstream`. -- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping - middleboxes from reaping the idle connection. -- Each subscriber buffers up to 64 entries. A client too slow for the query rate - overflows its buffer and the server ends the stream cleanly after delivering - what the buffer held — queries are never held back for a slow reader. There is - no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the - missed rows. -- Connections per client address are capped at `web.sse_max_connections_per_ip` - (default 3); over the cap is a 429. The cap binds loopback too. The server - holds at most 32 concurrent streams in total; when all slots are taken, the - answer is a 503. +- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on its own after a drop. +- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload carries the `GET /api/queries` row fields minus `id` (a live entry precedes persistence): `ts`, `domain`, `client_ip`, `qtype`, `blocked`, `block_reason`, `response_time_us`, `cache_hit`, `upstream`. +- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping middleboxes from reaping the idle connection. +- Each subscriber buffers up to 64 entries. A client too slow for the query rate overflows its buffer and the server ends the stream cleanly after delivering what the buffer held — queries are never held back for a slow reader. There is no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the missed rows. +- Connections per client address are capped at `web.sse_max_connections_per_ip` (default 3); over the cap is a 429. The cap binds loopback too. The server holds at most 32 concurrent streams in total; when all slots are taken, the answer is a 503. ## Configuration authority -Which authority is live decides whether the API may write configuration. Under -`nxdns run` the database is authority and every route behaves as it always has. -Under `nxdns run --config FILE` the file is authority, and the routes that would -edit configuration are refused: the file is the only place configuration -changes, and a restart is what applies them. +Which authority is live decides whether the API may write configuration. Under `nxdns run` the database is authority and every route behaves as it always has. Under `nxdns run --config FILE` the file is authority, and the routes that would edit configuration are refused: the file is the only place configuration changes, and a restart is what applies them. ### The refusal -A `config write` route in file mode answers **403** with the ordinary error -envelope: +A `config write` route in file mode answers **403** with the ordinary error envelope: ```json {"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"} ``` -There is no `code` field and no richer body. 403 is used for nothing else in -this API, so the status alone is the machine-readable part, and a client that -wants to know the mode in advance reads it from `GET /api/settings` rather than -probing for errors. +There is no `code` field and no richer body. 403 is used for nothing else in this API, so the status alone is the machine-readable part, and a client that wants to know the mode in advance reads it from `GET /api/settings` rather than probing for errors. -**401 comes first.** The router matches the path, spends a rate-limit token, -checks the session, and only then checks the policy. So an unauthenticated -request to a `config write` route in file mode is a 401, not a 403 — answering -403 first would tell an anonymous caller which routes exist. +**401 comes first.** The router matches the path, spends a rate-limit token, checks the session, and only then checks the policy. So an unauthenticated request to a `config write` route in file mode is a 401, not a 403 — answering 403 first would tell an anonymous caller which routes exist. -`runtime action` and `read` routes are unaffected in both modes. Pausing -blocking, refreshing blocklists, reloading certificates and logging in are -operations on a running process, not statements about configuration, so a -file-mode box still does all of them. +`runtime action` and `read` routes are unaffected in both modes. Pausing blocking, refreshing blocklists, reloading certificates and logging in are operations on a running process, not statements about configuration, so a file-mode box still does all of them. -`DELETE /api/clients/{id}` is the one route whose answer depends on the row. -Deleting a client the file does not declare is a runtime action and succeeds: -without it, a mis-identified or departed device would be immortal in file mode, -since the file can add addresses but never remove one it has never named. -Deleting a client the file *does* declare contradicts the file, and answers the -same 403. +`DELETE /api/clients/{id}` is the one route whose answer depends on the row. Deleting a client the file does not declare is a runtime action and succeeds: without it, a mis-identified or departed device would be immortal in file mode, since the file can add addresses but never remove one it has never named. Deleting a client the file *does* declare contradicts the file, and answers the same 403. ### Discovering the authority @@ -165,32 +80,20 @@ same 403. | `path` | The managed file's path, or `null` in database mode. | | `reconciled_at` | Unix seconds when this process loaded the file, or `null` in database mode. | -All three keys are always present; the two nullable ones carry `null` rather -than being omitted, so a client can read `authority.mode` without probing. +All three keys are always present; the two nullable ones carry `null` rather than being omitted, so a client can read `authority.mode` without probing. ```json {"mode": "database", "path": null, "reconciled_at": null} {"mode": "managed_file", "path": "/etc/nxdns/config.zon", "reconciled_at": 1786474016} ``` -The route requires a session, which is why the filesystem path is here rather -than on the open `/api/version` and `/api/health`. +The route requires a session, which is why the filesystem path is here rather than on the open `/api/version` and `/api/health`. -`reconciled_at` answers exactly one question: **when did this process last read -the file?** Compare it against the file's mtime to spot a restart that has not -happened yet. It is a hint and not a verdict, in both directions — a clock that -stepped, or a copy that preserved mtimes (`git checkout`, `rsync -a`), can make -a newer file look older, and the database can change without either timestamp -moving. It does not tell you whether the file and the running configuration -agree; answering that would take content hashing, which nxdns deliberately does -not do. +`reconciled_at` answers exactly one question: **when did this process last read the file?** Compare it against the file's mtime to spot a restart that has not happened yet. It is a hint and not a verdict, in both directions — a clock that stepped, or a copy that preserved mtimes (`git checkout`, `rsync -a`), can make a newer file look older, and the database can change without either timestamp moving. It does not tell you whether the file and the running configuration agree; answering that would take content hashing, which nxdns deliberately does not do. ## Operations -Auth `open` means no session is required; `session` means a valid session cookie -is required whenever a password is set. Rate limit `counted` spends a token; -`exempt` never consults the limiter. Policy `config write` is the class refused -in file mode; `read` and `runtime action` are always served. +Auth `open` means no session is required; `session` means a valid session cookie is required whenever a password is set. Rate limit `counted` spends a token; `exempt` never consults the limiter. Policy `config write` is the class refused in file mode; `read` and `runtime action` are always served. | Method | Path | Auth | Rate limit | Policy | Purpose | |---|---|---|---|---|---| @@ -251,40 +154,23 @@ in file mode; `read` and `runtime action` are always served. | PUT | `/api/settings` | session | counted | config write | Update settings | | POST | `/api/certs/reload` | session | counted | runtime action | Reload the TLS certificates from disk | -There is no `POST /api/clients`: client rows come from DNS activity or import, -never from the API. +There is no `POST /api/clients`: client rows come from DNS activity or import, never from the API. -Static assets are not routes. The router sends unmatched non-`/api` paths to the -embedded SPA before any auth or rate-limit check. +Static assets are not routes. The router sends unmatched non-`/api` paths to the embedded SPA before any auth or rate-limit check. ## Settings keys -`GET /api/settings` and `PUT /api/settings` speak the `section.field` keys of -[the configuration reference](configuration.md), with the values in their -database spelling — notably `logging.level` is `"error"`, not `"err"`. -Two keys behave differently over the API than in the file: `web.password` is -write-only (accepted on a `PUT`, never returned, hashed before storage), and -`web.password_hash` is neither readable nor directly writable, because a client -that could install a hash could install one whose password it already knows. +`GET /api/settings` and `PUT /api/settings` speak the `section.field` keys of [the configuration reference](configuration.md), with the values in their database spelling — notably `logging.level` is `"error"`, not `"err"`. Two keys behave differently over the API than in the file: `web.password` is write-only (accepted on a `PUT`, never returned, hashed before storage), and `web.password_hash` is neither readable nor directly writable, because a client that could install a hash could install one whose password it already knows. -In file mode `PUT /api/settings` is refused with the 403 above, password changes -included. The password then lives where the rest of the configuration lives: set -`web.password` in the file and restart. See -[Password and hash](configuration.md#password-and-hash). +In file mode `PUT /api/settings` is refused with the 403 above, password changes included. The password then lives where the rest of the configuration lives: set `web.password` in the file and restart. See [Password and hash](configuration.md#password-and-hash). ## Schemas -Request and response schemas for every operation live in the OpenAPI document: -`src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a -running server. +Request and response schemas for every operation live in the OpenAPI document: `src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a running server. ### Block reasons -Three places carry the same tag: `block_reason` on a `GET /api/queries` row, -`block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup` -answer. The tag names the level that decided the query, and the levels are -listed here in the order they are consulted — the first one that matches wins, -so a rule always outranks a list. +Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup` answer. The tag names the level that decided the query, and the levels are listed here in the order they are consulted — the first one that matches wins, so a rule always outranks a list. | Tag | Decided by | | --- | --- | @@ -298,10 +184,6 @@ so a rule always outranks a list. | `blocklist_domain` | A plain name in a downloaded list | | `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list | -`/api/lookup` also answers `none` when nothing matched. A query row never -carries `none`: `block_reason` is null unless the query was blocked. +`/api/lookup` also answers `none` when nothing matched. A query row never carries `none`: `block_reason` is null unless the query was blocked. -A `cname:` prefix means the decision landed on a CNAME target rather than on -the name the client asked for, so `cname:blocklist_domain` reads as "the list -blocks a name this answer redirects to". Only `/api/queries` and the live -stream show the prefix; `/api/lookup` does not follow CNAMEs. +A `cname:` prefix means the decision landed on a CNAME target rather than on the name the client asked for, so `cname:blocklist_domain` reads as "the list blocks a name this answer redirects to". Only `/api/queries` and the live stream show the prefix; `/api/lookup` does not follow CNAMEs. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 8420754..394c221 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -4,15 +4,9 @@ nxdns [options] ``` -Six subcommands: `run`, `check`, `export`, `import`, `version`, `help`. Source -of truth: `src/cli.zig`. +Six subcommands: `run`, `check`, `export`, `import`, `version`, `help`. Source of truth: `src/cli.zig`. -Every flag takes both spellings, `--flag value` and `--flag=value`. An attached -value that is empty (`--config=`) is a missing value, not an empty path. -`--allow-delete` is boolean and takes no value at all, so `--allow-delete=1` is -not a spelling of any flag this program has. A flag is rejected by the -subcommand that has no use for it: `--web-dev` outside `run` is an unknown flag, -not a no-op. +Every flag takes both spellings, `--flag value` and `--flag=value`. An attached value that is empty (`--config=`) is a missing value, not an empty path. `--allow-delete` is boolean and takes no value at all, so `--allow-delete=1` is not a spelling of any flag this program has. A flag is rejected by the subcommand that has no use for it: `--web-dev` outside `run` is an unknown flag, not a no-op. ## `run` @@ -26,27 +20,21 @@ Serves DNS until SIGINT or SIGTERM. ### Which authority the invocation selects -The presence of `--config` picks the authority, and nothing else does. There is -no default path, no probe of `/etc/nxdns`, and nothing recorded in the database: -a configuration file sitting at `/etc/nxdns/config.zon` that no flag names -changes nothing at all. +The presence of `--config` picks the authority, and nothing else does. There is no default path, no probe of `/etc/nxdns`, and nothing recorded in the database: a configuration file sitting at `/etc/nxdns/config.zon` that no flag names changes nothing at all. | Invocation | Authority | What a start does | | --- | --- | --- | | `nxdns run` | The database | Serves what `config.db` holds. Nothing reads a file. | | `nxdns run --config FILE` | FILE | Reads and validates FILE, reconciles the database onto it, then serves. | -The first log line after the migrations names the mode, so a journal says which -authority was live: +The first log line after the migrations names the mode, so a journal says which authority was live: ``` info(nxdns): authority: database info(nxdns): authority: file (/etc/nxdns/config.zon) ``` -In file mode the reconcile prints what it changed before that line — per-table -inserted (`+`), updated (`~`) and deleted (`-`) counts, the settings keys whose -values changed, and any change to whether the admin password is set: +In file mode the reconcile prints what it changed before that line — per-table inserted (`+`), updated (`~`) and deleted (`-`) counts, the settings keys whose values changed, and any change to whether the admin password is set: ``` reconciled '/etc/nxdns/config.zon': upstreams +1 ~0 -0; settings +45 ~0 -0; @@ -60,17 +48,11 @@ A start whose file matches the database writes nothing and says so: reconciled '/etc/nxdns/config.zon': no changes ``` -Blocklist state is not declarative and survives every reconcile: a source whose -URL the file still names keeps its row id, its checksum, its counters and its -compiled `.list`, `.wild` and `.allow`, so a restart in file mode -downloads nothing. Editing a source's URL is a new identity — a new row, a new id, and a -fresh download. +Blocklist state is not declarative and survives every reconcile: a source whose URL the file still names keeps its row id, its checksum, its counters and its compiled `.list`, `.wild` and `.allow`, so a restart in file mode downloads nothing. Editing a source's URL is a new identity — a new row, a new id, and a fresh download. ### Failing to start in file mode -File mode fails closed. A file that is missing, unreadable, unparseable, -oversized or invalid stops the start; nxdns never falls back to the database, -because a fallback turns a deploy typo into a silently stale configuration. +File mode fails closed. A file that is missing, unreadable, unparseable, oversized or invalid stops the start; nxdns never falls back to the database, because a fallback turns a deploy typo into a silently stale configuration. ``` FAIL /etc/nxdns/config.zon: no such file @@ -78,46 +60,21 @@ nxdns run failed: ManagedConfigUnreadable run `nxdns check` to see the configuration in full ``` -That is exit 2, and `check --config` on the same path agrees. Only path-class -open failures map that way — the file is not there, or the process may not read -it. An open that fails for a reason a retry could clear, such as -file-descriptor exhaustion or an I/O error, is exit 1: the box is wrong, not the -configuration. See [exit codes](#exit-codes). +That is exit 2, and `check --config` on the same path agrees. Only path-class open failures map that way — the file is not there, or the process may not read it. An open that fails for a reason a retry could clear, such as file-descriptor exhaustion or an I/O error, is exit 1: the box is wrong, not the configuration. See [exit codes](#exit-codes). -The exit-2 agreement covers `run --config` and `check --config`, which read the -managed file through one shared helper. It does **not** extend to `import`'s -positional argument: a missing file there is `import failed: FileNotFound`, exit -1. That is deliberate rather than an oversight — the managed file is a -declarative input an operator deploys, so its absence is a fact about the -configuration, while `import`'s argument is a path typed at a prompt, and a -mistyped path is a failed command rather than a verdict on anything. +The exit-2 agreement covers `run --config` and `check --config`, which read the managed file through one shared helper. It does **not** extend to `import`'s positional argument: a missing file there is `import failed: FileNotFound`, exit +1. That is deliberate rather than an oversight — the managed file is a declarative input an operator deploys, so its absence is a fact about the configuration, while `import`'s argument is a path typed at a prompt, and a mistyped path is a failed command rather than a verdict on anything. -Run `nxdns check --config FILE` before restarting anything that deploys a file. -It grades every declarative fault `run` would hit — read, parse, size, -validation — through the same code, which is what makes it a usable precondition -in an Ansible handler. +Run `nxdns check --config FILE` before restarting anything that deploys a file. It grades every declarative fault `run` would hit — read, parse, size, validation — through the same code, which is what makes it a usable precondition in an Ansible handler. ## `check` -Validates the configuration and probes the upstreams. Exit 0 when it found no -failures, 2 when it found one. It always reports every problem, not just the -first. What it checks, in order: +Validates the configuration and probes the upstreams. Exit 0 when it found no failures, 2 when it found one. It always reports every problem, not just the first. What it checks, in order: 1. Which source to check (see [source selection](#source-selection)). 2. Full validation — the same rules `import` enforces. -3. For each enabled DoH/DoT listener: both PEM files are read and the key is - tested against the certificate, through the same `CertStore.init` the - listeners boot with. A file that is missing, unreadable, too large or - unparseable, and a key that does not belong to the certificate, are all FAIL. - The key's permissions are a separate finding: WARN when any group or other - bit is set, which does not change the exit code. -4. A live probe: one real A query for `example.com` through every enabled - upstream, driving the same pool and failover machinery the server uses, with - the same two deadlines: `upstream.attempt_timeout_ms` bounds one try against - one upstream, `upstream.total_timeout_ms` the whole probe. A FAIL line names - the upstream and the concrete cause recorded in its health. This probe leaves - the machine, so `check` needs network access to pass. It runs from the - command line but not from unit tests. +3. For each enabled DoH/DoT listener: both PEM files are read and the key is tested against the certificate, through the same `CertStore.init` the listeners boot with. A file that is missing, unreadable, too large or unparseable, and a key that does not belong to the certificate, are all FAIL. The key's permissions are a separate finding: WARN when any group or other bit is set, which does not change the exit code. +4. A live probe: one real A query for `example.com` through every enabled upstream, driving the same pool and failover machinery the server uses, with the same two deadlines: `upstream.attempt_timeout_ms` bounds one try against one upstream, `upstream.total_timeout_ms` the whole probe. A FAIL line names the upstream and the concrete cause recorded in its health. This probe leaves the machine, so `check` needs network access to pass. It runs from the command line but not from unit tests. | Flag | Meaning | | --- | --- | @@ -126,10 +83,7 @@ first. What it checks, in order: ### Failures and warnings -Every finding carries a severity. `FAIL` is a problem that sets exit 2. `WARN` -is legal configuration that is almost certainly not what was meant — a blocklist -source no group links to, a TLS key readable beyond its owner — and never -changes an exit code, because the service starts either way. +Every finding carries a severity. `FAIL` is a problem that sets exit 2. `WARN` is legal configuration that is almost certainly not what was meant — a blocklist source no group links to, a TLS key readable beyond its owner — and never changes an exit code, because the service starts either way. The last line is a summary, and it never contradicts the lines above it: @@ -141,91 +95,55 @@ The last line is a summary, and it never contradicts the lines above it: ### Source selection -The flag decides, exactly as it does for `run`. There is no fallback and no -probing of a default path: +The flag decides, exactly as it does for `run`. There is no fallback and no probing of a default path: - `--config FILE`: grade that file, and never open the database. -- No `--config`: grade `/config.db`. It is opened immutable, so - `check` writes nothing to it; see - [what `check` does not do](#what-check-does-not-do). +- No `--config`: grade `/config.db`. It is opened immutable, so `check` writes nothing to it; see [what `check` does not do](#what-check-does-not-do). -So `nxdns check` and `nxdns check --config FILE` grade what the matching `run` -invocation would serve. That is what makes `check` a pre-restart gate rather -than an approximation of one. +So `nxdns check` and `nxdns check --config FILE` grade what the matching `run` invocation would serve. That is what makes `check` a pre-restart gate rather than an approximation of one. -A bare `check` on a box with no database says so and names both ways out of the -state, exit 2: +A bare `check` on a box with no database says so and names both ways out of the state, exit 2: ``` no config database at /var/lib/nxdns/config.db load one with `nxdns import `, or make a file the source of truth with `nxdns run --config ` ``` -The first line of output otherwise always names which source was checked. A file -larger than 4 MiB fails with `larger than 4194304 bytes`; a ZON syntax error is -reported with its line and column. +The first line of output otherwise always names which source was checked. A file larger than 4 MiB fails with `larger than 4194304 bytes`; a ZON syntax error is reported with its line and column. -A named file that is missing or unreadable is a finding like any other, not an -I/O failure that escapes the run: `FAIL : no such file` or -`FAIL : not readable`, exit 2. +A named file that is missing or unreadable is a finding like any other, not an I/O failure that escapes the run: `FAIL : no such file` or `FAIL : not readable`, exit 2. -What `check --config` cannot see is the reconcile itself. It needs no database, -so faults that only a write can produce — a disk that is full, a lock held by a -restart that started first — are invisible to it. Those are runtime failures, -exit 1, and they are not verdicts on the file. +What `check --config` cannot see is the reconcile itself. It needs no database, so faults that only a write can produce — a disk that is full, a lock held by a restart that started first — are invisible to it. Those are runtime failures, exit 1, and they are not verdicts on the file. ### What `check` does not do -`check` never writes to `config.db`. It opens the file immutable, which means -SQLite refuses every statement that would write and builds no write-ahead log, -so no `config.db-wal` and no `config.db-shm` appear beside it. It does not chmod -the file and it does not migrate the schema. +`check` never writes to `config.db`. It opens the file immutable, which means SQLite refuses every statement that would write and builds no write-ahead log, so no `config.db-wal` and no `config.db-shm` appear beside it. It does not chmod the file and it does not migrate the schema. Two consequences are worth knowing before you read a FAIL line as damage: -- A database behind this binary's schema is reported, not upgraded. Start the - service to migrate it. +- A database behind this binary's schema is reported, not upgraded. Start the service to migrate it. ``` FAIL : schema version , this nxdns expects ; `nxdns run` migrates it, `check` will not ``` -- A database with an unapplied write-ahead log cannot be graded without writing, - because the newest settings are in the log and the main file holds older ones. - `check` says so rather than reading the stale values: +- A database with an unapplied write-ahead log cannot be graded without writing, because the newest settings are in the log and the main file holds older ones. `check` says so rather than reading the stale values: ``` FAIL : uncheckpointed changes are waiting in -wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line. ``` - A running nxdns is the usual holder of that log, so this is a common answer - when checking a live server rather than a sign of damage. It depends on what - is in the log, not on whether a server is up: a configuration write that has - not been checkpointed puts bytes there, and a server that has only been - answering queries leaves `config.db-wal` empty and reads normally. Check an - `export` with `--config`, or stop the service first. + A running nxdns is the usual holder of that log, so this is a common answer when checking a live server rather than a sign of damage. It depends on what is in the log, not on whether a server is up: a configuration write that has not been checkpointed puts bytes there, and a server that has only been answering queries leaves `config.db-wal` empty and reads normally. Check an `export` with `--config`, or stop the service first. ## `export` -Writes the configuration as ZON to stdout, or atomically at mode 0600 to -`--out FILE`. `--out` paths are relative to the shell's working directory, not -to the data directory. The output is canonical: every default emitted, -deterministic ordering, no timestamps, so `export` → `import` → `export` is -byte-identical. +Writes the configuration as ZON to stdout, or atomically at mode 0600 to `--out FILE`. `--out` paths are relative to the shell's working directory, not to the data directory. The output is canonical: every default emitted, deterministic ordering, no timestamps, so `export` → `import` → `export` is byte-identical. -`export` does not create the data directory; it fails if the directory is not -there. A directory that exists without a `config.db` is not that case: the -database is opened with create semantics, so an empty `config.db` is created and -migrated, and the export is of a default configuration. +`export` does not create the data directory; it fails if the directory is not there. A directory that exists without a `config.db` is not that case: the database is opened with create semantics, so an empty `config.db` is created and migrated, and the export is of a default configuration. -The output is the same in both authority modes — nothing marks a file as -exported from a file-mode box. That is what lets `export` be the adoption tool: -the file you check is the file you deploy. +The output is the same in both authority modes — nothing marks a file as exported from a file-mode box. That is what lets `export` be the adoption tool: the file you check is the file you deploy. -`web.password` is always written as `null` and `web.password_hash` carries the -stored value, so an export re-imports without anyone knowing the password. A -`null` password is not the same statement as an empty one: see -[`web.password` and `web.password_hash`](configuration.md#password-and-hash). +`web.password` is always written as `null` and `web.password_hash` carries the stored value, so an export re-imports without anyone knowing the password. A `null` password is not the same statement as an empty one: see [`web.password` and `web.password_hash`](configuration.md#password-and-hash). | Flag | Meaning | | --- | --- | @@ -236,10 +154,7 @@ See [back up and restore](../how-to/back-up-and-restore.md). ## `import FILE` -Converges the database onto FILE in one transaction — the same reconcile a -file-mode `run` performs, done once from the command line. Prints every -validation problem; a failed import leaves the database untouched. Creates the -data directory at mode 0700 if it is missing. +Converges the database onto FILE in one transaction — the same reconcile a file-mode `run` performs, done once from the command line. Prints every validation problem; a failed import leaves the database untouched. Creates the data directory at mode 0700 if it is missing. `FILE` is positional and may appear before or after the flags. @@ -248,29 +163,20 @@ data directory at mode 0700 if it is missing. | `--data-dir DIR` | Data directory holding `config.db` (created if missing). | | `--allow-delete` | Apply a file whose diff deletes rows. | -**`import` is a stop-first operation.** It rewrites configuration underneath a -process that read it at startup, and a running server notices only some of it: -filtering picks up the imported rows at the next reload, while upstreams, -listeners and settings stay at their boot values until a restart. +**`import` is a stop-first operation.** It rewrites configuration underneath a process that read it at startup, and a running server notices only some of it: filtering picks up the imported rows at the next reload, while upstreams, listeners and settings stay at their boot values until a restart. ### The delete gate -Rows the database holds and FILE does not name are deleted. That is the point of -a declarative apply, and it is also how a mistaken `nxdns import ./wrong.zon` -empties a configured server, so it takes a flag: +Rows the database holds and FILE does not name are deleted. That is the point of a declarative apply, and it is also how a mistaken `nxdns import ./wrong.zon` empties a configured server, so it takes a flag: ``` FAIL import: this file would delete rows the database holds (upstreams 1); re-run with --allow-delete to apply it import failed: DestructiveImport ``` -That is exit 2, and the transaction rolls back. The message names every table -with a non-zero delete count, so you can tell an intended pruning from a wrong -file before applying anything. +That is exit 2, and the transaction rolls back. The message names every table with a non-zero delete count, so you can tell an intended pruning from a wrong file before applying anything. -An import that only adds rows, or only edits them, needs no flag. "Edit" here -means a change to a row nxdns can still recognise as the same row. Each table -has one column, or one tuple, that establishes identity: +An import that only adds rows, or only edits them, needs no flag. "Edit" here means a change to a row nxdns can still recognise as the same row. Each table has one column, or one tuple, that establishes identity: | Table | Identity | | --- | --- | @@ -282,43 +188,25 @@ has one column, or one tuple, that establishes identity: | `local_records` | `(name, rtype, value)` | | `rules` | `(group, pattern, kind, action)` | -Change anything else on a row — a source's name, a group's `safe_search`, a -client's group — and it is an edit, applied without a flag. Change the identity -itself, such as renaming a group or correcting a typo in an upstream URL, and -the engine sees a row that vanished and a row that appeared: that needs -`--allow-delete`. +Change anything else on a row — a source's name, a group's `safe_search`, a client's group — and it is an edit, applied without a flag. Change the identity itself, such as renaming a group or correcting a typo in an upstream URL, and the engine sees a row that vanished and a row that appeared: that needs `--allow-delete`. ### What survives an import -Runtime state is not declarative and is preserved by identity, not by luck. A -blocklist source whose URL is unchanged keeps its row id, its checksum, its -counters and its compiled files, so an import costs no downloads. Clients the -DNS path materialised from traffic are kept whole; naming one in the file -promotes that row in place, keeping its first-seen and last-seen. Observed -clients whose group the file no longer declares are moved to the `default` -group rather than deleted with it, and none of that ever trips the delete gate. +Runtime state is not declarative and is preserved by identity, not by luck. A blocklist source whose URL is unchanged keeps its row id, its checksum, its counters and its compiled files, so an import costs no downloads. Clients the DNS path materialised from traffic are kept whole; naming one in the file promotes that row in place, keeping its first-seen and last-seen. Observed clients whose group the file no longer declares are moved to the `default` group rather than deleted with it, and none of that ever trips the delete gate. ### `import` against a file-mode box -It behaves like any other import. Nothing in the database records that a file -governs it — authority lives in the invocation — so `import` neither detects nor -refuses that case. The next restart's reconcile converges the database back to -the file and its summary reports what it corrected. A source the import deleted -comes back with a new row id, which means a fresh download of the whole list. +It behaves like any other import. Nothing in the database records that a file governs it — authority lives in the invocation — so `import` neither detects nor refuses that case. The next restart's reconcile converges the database back to the file and its summary reports what it corrected. A source the import deleted comes back with a new row id, which means a fresh download of the whole list. -If a restart and an import race for the write lock, one of them simply wins: both -take `BEGIN IMMEDIATE` under a 5-second busy timeout, so the outcome is an -ordering, never a corrupted database. +If a restart and an import race for the write lock, one of them simply wins: both take `BEGIN IMMEDIATE` under a 5-second busy timeout, so the outcome is an ordering, never a corrupted database. ## `version` -Prints two lines: the nxdns version with the git commit, then the Zig version -the binary was built with. Takes no flags and no arguments. +Prints two lines: the nxdns version with the git commit, then the Zig version the binary was built with. Takes no flags and no arguments. ## `help` -Prints the usage text to stdout and exits 0. `nxdns --help` and `nxdns -h` do -the same. A usage error prints the same text to stderr and exits 64. +Prints the usage text to stdout and exits 0. `nxdns --help` and `nxdns -h` do the same. A usage error prints the same text to stderr and exits 64. ## Exit codes @@ -329,50 +217,26 @@ the same. A usage error prints the same text to stderr and exits 64. | 2 | A configuration problem the operator can fix, or a `check` that found one. | | 64 | Usage error — unknown command or flag, a flag without its value, a missing or extra argument. | -Code 2 means the same thing from every subcommand. `src/config/faults.zig` -holds the one list of errors that mean "the configuration the operator supplied -is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits -2 whichever command read it. The list is every error the validator raises, plus -`ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams`, `BadCertificate` and -`ManagedConfigUnreadable`. In practice that covers a file with a syntax error, -one larger than 4 MiB, one with no `default` group (`MissingDefaultGroup`), one -with no enabled upstream (`NoUpstreams`), a bad bind address, a bad rate limit, -an unusable certificate, `password` and `password_hash` set together, and a -`--config` path that is absent or unreadable. +Code 2 means the same thing from every subcommand. `src/config/faults.zig` holds the one list of errors that mean "the configuration the operator supplied is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits 2 whichever command read it. The list is every error the validator raises, plus `ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams`, `BadCertificate` and `ManagedConfigUnreadable`. In practice that covers a file with a syntax error, one larger than 4 MiB, one with no `default` group (`MissingDefaultGroup`), one with no enabled upstream (`NoUpstreams`), a bad bind address, a bad rate limit, an unusable certificate, `password` and `password_hash` set together, and a `--config` path that is absent or unreadable. -The last of those is the one deliberate seam. A file nxdns cannot open is a -configuration fault only when the *path* is the problem — the file is missing, -permissions deny it, a path component is not a directory. Every other open -failure, such as running out of file descriptors, is exit 1. The distinction -earns its keep under the shipped systemd unit, which stops the service on exit 2 -rather than restarting it: a transient box fault graded as a configuration fault -would take the resolver down until someone noticed. +The last of those is the one deliberate seam. A file nxdns cannot open is a configuration fault only when the *path* is the problem — the file is missing, permissions deny it, a path component is not a directory. Every other open failure, such as running out of file descriptors, is exit 1. The distinction earns its keep under the shipped systemd unit, which stops the service on exit 2 rather than restarting it: a transient box fault graded as a configuration fault would take the resolver down until someone noticed. -When `run` exits 2 it points at the diagnosis on stderr, whichever authority the -fault came from: +When `run` exits 2 it points at the diagnosis on stderr, whichever authority the fault came from: ``` run `nxdns check` to see the configuration in full ``` -On a box with no configuration at all, `run` and `check` add the line that names -both ways to get one: +On a box with no configuration at all, `run` and `check` add the line that names both ways to get one: ``` load one with `nxdns import `, or make a file the source of truth with `nxdns run --config ` ``` -`check` exits 2 for those faults and also when a probed upstream failed, when a -named configuration file is missing or unreadable, and when the database cannot -be read, is absent, or is not at this binary's schema version. Warnings never -contribute. +`check` exits 2 for those faults and also when a probed upstream failed, when a named configuration file is missing or unreadable, and when the database cannot be read, is absent, or is not at this binary's schema version. Warnings never contribute. -`import` exits 2 for those faults and for `DestructiveImport`. That last one is -deliberately not a configuration fault — it reports what applying the file would -delete, rather than anything wrong with its content — and `import` decides it -for itself; the answer to it is `--allow-delete`, not an edit. +`import` exits 2 for those faults and for `DestructiveImport`. That last one is deliberately not a configuration fault — it reports what applying the file would delete, rather than anything wrong with its content — and `import` decides it for itself; the answer to it is `--allow-delete`, not an edit. -`OutOfMemory` is exit 1 even when problems were recorded, because the report is -then incomplete. Every other error is 1. +`OutOfMemory` is exit 1 even when problems were recorded, because the report is then incomplete. Every other error is 1. Where an exit code sends you next: [troubleshoot](../how-to/troubleshoot.md). diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 6011352..bf22dfa 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -1,35 +1,20 @@ # Configuration reference -Every section, field and collection nxdns accepts, with its type, default, -unit, validation rule and the subsystem that consumes it. +Every section, field and collection nxdns accepts, with its type, default, unit, validation rule and the subsystem that consumes it. -Source of truth: `src/config/model.zig` (the model and the defaults), -`src/config/validate.zig` (the rules), `src/config/{loader,reconcile,import,export}.zig` -(the lifecycle). +Source of truth: `src/config/model.zig` (the model and the defaults), `src/config/validate.zig` (the rules), `src/config/{loader,reconcile,import,export}.zig` (the lifecycle). -For how the file, the database and `export`/`import` relate to each other, see -[the configuration model](../explanation/configuration-model.md). For the -commands that read and write configuration, see [the CLI -reference](cli.md). +For how the file, the database and `export`/`import` relate to each other, see [the configuration model](../explanation/configuration-model.md). For the commands that read and write configuration, see [the CLI reference](cli.md). ## File format -The file is ZON: a top-level anonymous struct whose fields are the sections and -collections below. Enum values are ZON enum literals (`.level = .err`, -`.response = .nxdomain`). Strings are double-quoted. The file may be at most -4 MiB (`max_config_bytes` in `src/config/loader.zig`); beyond that the error is -`ConfigTooLarge`. A syntax error is reported with its line and column. +The file is ZON: a top-level anonymous struct whose fields are the sections and collections below. Enum values are ZON enum literals (`.level = .err`, `.response = .nxdomain`). Strings are double-quoted. The file may be at most 4 MiB (`max_config_bytes` in `src/config/loader.zig`); beyond that the error is `ConfigTooLarge`. A syntax error is reported with its line and column. -Absent fields keep their defaults, both in the file and in the database. A -settings key stored in the database that the running binary does not know is -warned about and ignored, never an error. +Absent fields keep their defaults, both in the file and in the database. A settings key stored in the database that the running binary does not know is warned about and ignored, never an error. ### The `logging.level = .err` quirk -The log level `error` is a Zig keyword, so the ZON and model tag is `.err` while -the database and the settings API store the operator-facing word `"error"`. The -file says `.err`; `GET /api/settings` says `"error"`. `"err"` is not accepted as -database text, and `.error` is not a ZON tag. +The log level `error` is a Zig keyword, so the ZON and model tag is `.err` while the database and the settings API store the operator-facing word `"error"`. The file says `.err`; `GET /api/settings` says `"error"`. `"err"` is not accepted as database text, and `.error` is not a ZON tag. ## What is not in the file @@ -43,9 +28,7 @@ Storage paths are process arguments, not configuration: ## Scalar sections -The "Key" column is the settings key as stored in the database -(`section.field`); in the file the same field lives inside its section block, -for example `.dns = .{ .port = 53 }`. +The "Key" column is the settings key as stored in the database (`section.field`); in the file the same field lives inside its section block, for example `.dns = .{ .port = 53 }`. ### upstream @@ -57,16 +40,9 @@ Timeouts for talking to upstream resolvers. | `upstream.read_timeout_ms` | u32 | 3000 | ms | 100–120000 | read deadline on conditional-forward-zone exchanges (`src/local/forward_client.zig`) | | `upstream.total_timeout_ms` | u32 | 5000 | ms | 100–120000 | per-query budget of the upstream pool (`src/upstream/pool.zig`): every failover attempt together, not one of them; also the `nxdns check` probe deadline | -The two pool budgets nest. `attempt_timeout_ms` bounds one try against one -upstream; when it expires the pool records the failure and moves to the next -candidate. `total_timeout_ms` bounds the whole loop, so a query against five -unreachable upstreams costs the total budget once, not five attempt budgets in -a row. When the total expires the in-flight attempt is canceled and the query -fails with a timeout. +The two pool budgets nest. `attempt_timeout_ms` bounds one try against one upstream; when it expires the pool records the failure and moves to the next candidate. `total_timeout_ms` bounds the whole loop, so a query against five unreachable upstreams costs the total budget once, not five attempt budgets in a row. When the total expires the in-flight attempt is canceled and the query fails with a timeout. -`read_timeout_ms` is unrelated to both. It bounds a different subsystem — the -conditional-forward-zone client — so no cross-check relates it to the pool's -budgets, and it is free to sit above either of them. +`read_timeout_ms` is unrelated to both. It bounds a different subsystem — the conditional-forward-zone client — so no cross-check relates it to the pool's budgets, and it is free to sit above either of them. ### dns @@ -80,10 +56,7 @@ The plain DNS listener (UDP and TCP). | `dns.rate_limit` | u32 | 1000 | queries per window | at least 1 | per-client DNS rate limiter (`src/server/rate_limiter.zig`) | | `dns.rate_window_seconds` | u32 | 60 | seconds | 1–3600 | window of the same limiter | -`dns.bind_ipv4` and `dns.bind_ipv6` each name one socket of the dual-stack pair, -so each is required to be a literal of its own family. An IPv4 wildcard in -`dns.bind_ipv6` is refused: it would bind IPv4 as the "v6" socket and make the -real IPv4 bind fail with `AddressInUse`, silently removing the IPv6 service. +`dns.bind_ipv4` and `dns.bind_ipv6` each name one socket of the dual-stack pair, so each is required to be a literal of its own family. An IPv4 wildcard in `dns.bind_ipv6` is refused: it would bind IPv4 as the "v6" socket and make the real IPv4 bind fail with `AddressInUse`, silently removing the IPv6 service. ### blocking @@ -101,10 +74,7 @@ What a blocked query gets back. | `cache.size` | u32 | 10000 | entries | 1–1000000 | DNS answer cache capacity (`src/cache/dns_cache.zig`) | | `cache.negative_ttl_max` | u32 | 3600 | seconds | at most 86400 | cap on cached negative answers; 0 disables negative caching | -The cache's slot array is allocated in full at startup, so `cache.size` carries -a ceiling: it is a sanity bound against a typo, not a promise that the value -fits in the box's memory. There is no "off" value — to run without a cache, set -`cache.size` to 1. +The cache's slot array is allocated in full at startup, so `cache.size` carries a ceiling: it is a sanity bound against a typo, not a promise that the value fits in the box's memory. There is no "off" value — to run without a cache, set `cache.size` to 1. ### web @@ -125,8 +95,7 @@ The web interface and REST API. ### doh_server -The DNS-over-HTTPS listener (server side, for clients on the LAN). See -[enable DoH and DoT](../how-to/enable-doh-and-dot.md). +The DNS-over-HTTPS listener (server side, for clients on the LAN). See [enable DoH and DoT](../how-to/enable-doh-and-dot.md). | Key | Type | Default | Unit | Validation | Consumed by | |---|---|---|---|---|---| @@ -138,8 +107,7 @@ The DNS-over-HTTPS listener (server side, for clients on the LAN). See ### dot_server -The DNS-over-TLS listener. Same shape as `doh_server`; only the default port -differs. +The DNS-over-TLS listener. Same shape as `doh_server`; only the default port differs. | Key | Type | Default | Unit | Validation | Consumed by | |---|---|---|---|---|---| @@ -173,9 +141,7 @@ Process log and query log behavior. ### disk -Free-space thresholds for the data directory. Below them the query-log writer, -the client tracker and the blocklist scheduler are throttled -(`src/storage/disk_monitor.zig`); DNS resolution is never gated. +Free-space thresholds for the data directory. Below them the query-log writer, the client tracker and the blocklist scheduler are throttled (`src/storage/disk_monitor.zig`); DNS resolution is never gated. | Key | Type | Default | Unit | Validation | Consumed by | |---|---|---|---|---|---| @@ -191,23 +157,18 @@ the client tracker and the blocklist scheduler are throttled ## Collections -Collections are ZON lists of structs. Fields without a default are required. -Runtime columns (first/last seen timestamps, per-source download counters) are -deliberately not part of the model: import sets timestamps to the import time -and export omits them, which is what keeps the round trip byte-stable. +Collections are ZON lists of structs. Fields without a default are required. Runtime columns (first/last seen timestamps, per-source download counters) are deliberately not part of the model: import sets timestamps to the import time and export omits them, which is what keeps the round trip byte-stable. ### groups -Client groups. A group named `default` is required; every client not assigned -elsewhere lands in it, and import guarantees it keeps database id 1. +Client groups. A group named `default` is required; every client not assigned elsewhere lands in it, and import guarantees it keeps database id 1. | Field | Type | Default | Validation | |---|---|---|---| | `name` | string | required | non-empty, unique | | `safe_search` | bool | false | — | -Consumed by the filter engine (`src/filter/matcher.zig`); `safe_search` -triggers the safe-search rewrite in the query path. +Consumed by the filter engine (`src/filter/matcher.zig`); `safe_search` triggers the safe-search rewrite in the query path. ### upstreams @@ -220,23 +181,15 @@ Upstream resolvers. At least one enabled upstream is required. | `enabled` | bool | true | — | | `tls_name` | string | `""` | DoT only — a `tls_name` on an `https://` upstream is an error; when set it must be a valid domain name | -Consumed by the upstream pool (`src/upstream/pool.zig`): entries are sorted by -ascending priority and tried in order with failover. `tls_name` sets SNI and the -certificate verification name for a `tls://` upstream written as an IP literal; -empty means "verify by the URL host" (`src/upstream/dot_client.zig`). +Consumed by the upstream pool (`src/upstream/pool.zig`): entries are sorted by ascending priority and tried in order with failover. `tls_name` sets SNI and the certificate verification name for a `tls://` upstream written as an IP literal; empty means "verify by the URL host" (`src/upstream/dot_client.zig`). -A `tls://` upstream's host must be an IP literal. nxdns does not resolve an -upstream's own name — that is a bootstrap problem, and the DoT client refuses a -non-literal host on every dial — so the validator rejects the hostname form -rather than letting it fail at query time as an unreachable upstream. Write the -address and put the name in `tls_name`: +A `tls://` upstream's host must be an IP literal. nxdns does not resolve an upstream's own name — that is a bootstrap problem, and the DoT client refuses a non-literal host on every dial — so the validator rejects the hostname form rather than letting it fail at query time as an unreachable upstream. Write the address and put the name in `tls_name`: ```zig .{ .url = "tls://9.9.9.9:853", .tls_name = "dns.quad9.net" }, ``` -A DoH upstream is not affected: `https://` goes through the HTTP client, which -resolves normally, so a hostname there is the usual form. +A DoH upstream is not affected: `https://` goes through the HTTP client, which resolves normally, so a hostname there is the usual form. ### clients @@ -248,36 +201,19 @@ Known clients with a fixed group assignment. | `name` | string | `""` | — (display only, never read by the resolver) | | `group` | string | `"default"` | must name a declared group | -Consumed by the filter engine's exact address-to-group lookup -(`src/filter/matcher.zig`). +Consumed by the filter engine's exact address-to-group lookup (`src/filter/matcher.zig`). #### Learned names -A client row that carries no name of its own can still show one. Every flush -pass the client tracker takes at most 16 unnamed rows that are due, builds each -address's reverse name (`192.168.1.10` becomes `10.1.168.192.in-addr.arpa`), and -matches it against the declared [`forward_zones`](#forward_zones). On a match it -sends one PTR query to that zone's resolver and stores the answer as the row's -*learned* name. A row becomes due again 24 hours after the resolver answered or -returned NXDOMAIN, and 1 hour after any other outcome — no covering zone, a -transport failure, or a reply nxdns rejected — so declaring the missing zone or -fixing the resolver shows names within the hour. This -requires a conditional forward zone that covers the LAN's reverse space — for -example `168.192.in-addr.arpa` pointed at the router. Without such a zone nxdns -sends the reverse name to nobody, and the row stays unnamed. +A client row that carries no name of its own can still show one. Every flush pass the client tracker takes at most 16 unnamed rows that are due, builds each address's reverse name (`192.168.1.10` becomes `10.1.168.192.in-addr.arpa`), and matches it against the declared [`forward_zones`](#forward_zones). On a match it sends one PTR query to that zone's resolver and stores the answer as the row's *learned* name. A row becomes due again 24 hours after the resolver answered or returned NXDOMAIN, and 1 hour after any other outcome — no covering zone, a transport failure, or a reply nxdns rejected — so declaring the missing zone or fixing the resolver shows names within the hour. This requires a conditional forward zone that covers the LAN's reverse space — for example `168.192.in-addr.arpa` pointed at the router. Without such a zone nxdns sends the reverse name to nobody, and the row stays unnamed. -The PTR query goes to the resolver of the declared zone, wherever the operator -pointed it; nxdns does not second-guess that declaration, and it cannot stop a -LAN resolver from forwarding the query onward. +The PTR query goes to the resolver of the declared zone, wherever the operator pointed it; nxdns does not second-guess that declaration, and it cannot stop a LAN resolver from forwarding the query onward. A learned name is runtime state, like `last_seen`: - A hand-typed `name` always wins, and a named row is never asked about again. -- Learned names never appear in `nxdns export`, and `nxdns import` never sets - one. -- Each row refreshes once a day. A device rename or a DHCP lease change can - therefore display a stale name for up to 24 hours, until the next refresh - overwrites it or the router reports no name and nxdns clears it. +- Learned names never appear in `nxdns export`, and `nxdns import` never sets one. +- Each row refreshes once a day. A device rename or a DHCP lease change can therefore display a stale name for up to 24 hours, until the next refresh overwrites it or the router reports no name and nxdns clears it. ### client_prefixes @@ -289,8 +225,7 @@ Group assignment by CIDR prefix, for clients without an exact entry. | `group` | string | `"default"` | must name a declared group | | `priority` | i32 | 100 | — (ties on match are broken by lower priority) | -Consumed by the filter engine's longest-prefix match -(`src/filter/matcher.zig`). +Consumed by the filter engine's longest-prefix match (`src/filter/matcher.zig`). ### blocklist_sources @@ -303,32 +238,11 @@ Downloadable blocklists. | `enabled` | bool | true | — | | `is_suggested` | bool | false | — (web UI hint only, never read by the resolver) | -Consumed by the blocklist manager (`src/filter/manager.zig`): downloaded by the -fetcher and compiled into domain sets. A disabled source is neither downloaded -nor loaded. +Consumed by the blocklist manager (`src/filter/manager.zig`): downloaded by the fetcher and compiled into domain sets. A disabled source is neither downloaded nor loaded. -A list in Adblock Plus syntax may also carry exception lines, `@@||name^` and -`@@||name`, either of which may end in `$important`. Those become allow entries -that cancel what any attached list blocks, for the name and its subdomains. They -cancel nothing an operator decided: every rule of the table above is checked -first, so a downloaded list can reopen only a hole another downloaded list dug. -Each source reports how many it carried as `exceptions`; there is no way to write -one by hand, and no reason to want one — write an allow rule instead. +A list in Adblock Plus syntax may also carry exception lines, `@@||name^` and `@@||name`, either of which may end in `$important`. Those become allow entries that cancel what any attached list blocks, for the name and its subdomains. They cancel nothing an operator decided: every rule of the table above is checked first, so a downloaded list can reopen only a hole another downloaded list dug. Each source reports how many it carried as `exceptions`; there is no way to write one by hand, and no reason to want one — write an allow rule instead. -Two counters report what a compile skipped, and they are different facts. -`skipped_regex` counts regex lines: nxdns has a regex engine, but it takes -patterns only from the operator, so a regex line in a downloaded list is counted, -skipped and surfaced — adopt the ones you trust as `regex` rules. -`skipped_unsupported` counts lines nxdns cannot safely translate into a DNS -decision: cosmetic element hiding (`##`, `#@#`, `#?#`), rules carrying a `$` -modifier (except `$important` on an exception line, tolerated above), scheme -anchors, non-anchored `@@` forms — and, in a `domains`-format -list, a line holding more than one field before its inline comment, which usually -means the list is really a hosts file that was declared as `domains`. Neither is -an error, and the two are never one number. A large `skipped_unsupported` beside -a small `domain_count` usually means the list is written for browser extensions, -and its DNS or hosts variant will block more here. Both appear per source in the -blocklists UI and on `/api/blocklists`. +Two counters report what a compile skipped, and they are different facts. `skipped_regex` counts regex lines: nxdns has a regex engine, but it takes patterns only from the operator, so a regex line in a downloaded list is counted, skipped and surfaced — adopt the ones you trust as `regex` rules. `skipped_unsupported` counts lines nxdns cannot safely translate into a DNS decision: cosmetic element hiding (`##`, `#@#`, `#?#`), rules carrying a `$` modifier (except `$important` on an exception line, tolerated above), scheme anchors, non-anchored `@@` forms — and, in a `domains`-format list, a line holding more than one field before its inline comment, which usually means the list is really a hosts file that was declared as `domains`. Neither is an error, and the two are never one number. A large `skipped_unsupported` beside a small `domain_count` usually means the list is written for browser extensions, and its DNS or hosts variant will block more here. Both appear per source in the blocklists UI and on `/api/blocklists`. ### group_sources @@ -339,8 +253,7 @@ Which groups consult which blocklist sources. | `group` | string | required | must name a declared group | | `source_url` | string | required | must name a declared blocklist source's `url`; the (group, source_url) pair is unique | -Consumed by the filter engine when assembling each group's compiled domain sets -(`src/filter/matcher.zig`). A link to a disabled source is skipped. +Consumed by the filter engine when assembling each group's compiled domain sets (`src/filter/matcher.zig`). A link to a disabled source is skipped. ### rules @@ -353,45 +266,21 @@ Per-group allow and block overrides, checked before the blocklists. | `kind` | enum `.exact` \| `.wildcard` \| `.regex` | required | — | | `action` | enum `.allow` \| `.block` | required | — | -Pattern rules: an `.exact` pattern is a plain domain name and may not contain -`*`. A `.wildcard` pattern must contain at least one label that is exactly `*` -(`*.tracker.example`, or `*` alone), and every other label must be a legal DNS -label. `ads*.example` is not a valid wildcard; a partial label is what the -`.regex` kind is for. +Pattern rules: an `.exact` pattern is a plain domain name and may not contain `*`. A `.wildcard` pattern must contain at least one label that is exactly `*` (`*.tracker.example`, or `*` alone), and every other label must be a legal DNS label. `ads*.example` is not a valid wildcard; a partial label is what the `.regex` kind is for. -A `.regex` pattern is a regular expression matched against the whole normalized -lowercase name, unanchored unless you write `^` or `$` — the POSIX-grep -convention. It is stored exactly as you typed it, which the other two kinds are -not: lowercasing would turn `\D` into `\d`, and trimming a trailing `.` would -delete an any-byte atom. The engine (`src/filter/regex.zig`) accepts literal -bytes, `.` for any byte, character classes `[a-z0-9]` with a leading `^` for -negation, the escapes `\d` and `\w` plus `\` before any other ASCII punctuation -to make it a literal, the repetitions `*` `+` `?` `{n}` `{n,m}` `{n,}`, -alternation `|`, grouping `(...)`, and the anchors `^` and `$`. +A `.regex` pattern is a regular expression matched against the whole normalized lowercase name, unanchored unless you write `^` or `$` — the POSIX-grep convention. It is stored exactly as you typed it, which the other two kinds are not: lowercasing would turn `\D` into `\d`, and trimming a trailing `.` would delete an any-byte atom. The engine (`src/filter/regex.zig`) accepts literal bytes, `.` for any byte, character classes `[a-z0-9]` with a leading `^` for negation, the escapes `\d` and `\w` plus `\` before any other ASCII punctuation to make it a literal, the repetitions `*` `+` `?` `{n}` `{n,m}` `{n,}`, alternation `|`, grouping `(...)`, and the anchors `^` and `$`. -Everything else is refused at the edge rather than approximated, so a pattern -written for another engine fails where you can read the diagnostic instead of -silently matching names you did not mean: +Everything else is refused at the edge rather than approximated, so a pattern written for another engine fails where you can read the diagnostic instead of silently matching names you did not mean: -- backreferences, lookaround, captures, named groups, Unicode classes and the - `(?…)` prefix they share; +- backreferences, lookaround, captures, named groups, Unicode classes and the `(?…)` prefix they share; - any alphanumeric escape the list above omits — `\s`, `\b`, `\1`, `\D`; - a `]` inside a class, unless written `\]`; -- an empty pattern, and an empty branch: `ads|` is refused rather than read as a - pattern that matches every name; -- a quantifier applied straight to another quantifier: `a+?` is refused rather - than read as `(a+)?`, which matches every name. Write `(a+)?` to mean that. +- an empty pattern, and an empty branch: `ads|` is refused rather than read as a pattern that matches every name; +- a quantifier applied straight to another quantifier: `a+?` is refused rather than read as `(a+)?`, which matches every name. Write `(a+)?` to mean that. -A pattern is at most 256 bytes and compiles to at most 1024 instructions, each -limit with its own diagnostic, and one group holds at most 256 regex rules. -Groups do not capture, and the engine simulates every alternative in lockstep, -so a pattern costs at most its compiled length times the length of the name — -`(a+)+b` is as cheap here as it is expensive in a backtracking engine. +A pattern is at most 256 bytes and compiles to at most 1024 instructions, each limit with its own diagnostic, and one group holds at most 256 regex rules. Groups do not capture, and the engine simulates every alternative in lockstep, so a pattern costs at most its compiled length times the length of the name — `(a+)+b` is as cheap here as it is expensive in a backtracking engine. -Consumed by the filter engine's rule sets (`src/filter/rules.zig`), which checks -the three kinds in the order they are listed above, allow before block within -each. Regex is checked last of the three because it is the only kind that costs -more than a hash lookup or a label walk. +Consumed by the filter engine's rule sets (`src/filter/rules.zig`), which checks the three kinds in the order they are listed above, allow before block within each. Regex is checked last of the three because it is the only kind that costs more than a hash lookup or a label walk. ### local_records @@ -404,27 +293,20 @@ Local DNS answers, served without touching any upstream. | `value` | string | required | an IPv4 address for `.a`, an IPv6 address for `.aaaa`, a domain name for `.cname` | | `ttl` | u32 | 300 | 1–604800 seconds | -The (name, rtype, value) triple is unique. Consumed by the local records table -in the query path (`src/local/records.zig`). +The (name, rtype, value) triple is unique. Consumed by the local records table in the query path (`src/local/records.zig`). ### forward_zones -Zones resolved by a specific resolver instead of the configured upstreams, for -LAN or corporate domains. +Zones resolved by a specific resolver instead of the configured upstreams, for LAN or corporate domains. | Field | Type | Default | Validation | |---|---|---|---| | `zone` | string | required | a valid domain name; unique | | `resolver` | string | required | `udp://IP:port` or `tcp://IP:port`; the host must be an IP literal and the port is mandatory | -The resolver host must be an IP literal because resolving the resolver's own -name would be a bootstrap problem. Matching is longest suffix -(`src/local/forward_zones.zig`); the exchange is UDP then TCP -(`src/local/forward_client.zig`) with `upstream.read_timeout_ms` as the read -deadline. +The resolver host must be an IP literal because resolving the resolver's own name would be a bootstrap problem. Matching is longest suffix (`src/local/forward_zones.zig`); the exchange is UDP then TCP (`src/local/forward_client.zig`) with `upstream.read_timeout_ms` as the read deadline. -Reverse zones are declared the same way, and one is the prerequisite for -[learned client names](#learned-names): +Reverse zones are declared the same way, and one is the prerequisite for [learned client names](#learned-names): ```zig .forward_zones = .{ @@ -435,9 +317,7 @@ Reverse zones are declared the same way, and one is the prerequisite for ## Password and hash -Both fields are optional, and the difference between *absent* and *empty* is the -whole design. Absent means "keep whatever is stored". Empty means "there is no -password". +Both fields are optional, and the difference between *absent* and *empty* is the whole design. Absent means "keep whatever is stored". Empty means "there is no password". | The file says | What happens to the stored hash | | --- | --- | @@ -448,72 +328,35 @@ password". | `.password_hash = ""` | Cleared, which disables authentication. | | Both fields | Refused (`PasswordAndHashBothSet` — ambiguity in a security setting). | -Silence has to mean "keep", because the alternative is a trap. An operator who -exports a configuration and trims the long PHC string out of it before -committing the file to git means "leave the password alone", not "open the admin -interface to the LAN". So disabling authentication takes the explicit empty -string, and the empty *plaintext* — which would otherwise hash into a real hash -that no login can ever satisfy — is refused outright: +Silence has to mean "keep", because the alternative is a trap. An operator who exports a configuration and trims the long PHC string out of it before committing the file to git means "leave the password alone", not "open the admin interface to the LAN". So disabling authentication takes the explicit empty string, and the empty *plaintext* — which would otherwise hash into a real hash that no login can ever satisfy — is refused outright: ``` FAIL web.password: password is set to the empty string; omit the field to keep the stored password, or set password_hash = "" to disable authentication ``` -- `web.password` is operator input only. It is hashed with argon2id (OWASP - parameters: t=2, m=19 MiB, p=1, PHC encoding) into `web.password_hash` and - discarded. There is no `web.password` settings row, and `nxdns export` always - writes `.password = null`. -- `web.password_hash` is the stored argon2id PHC string. Supplying it directly, - for example from a previous export, is how a backup restores authentication - without knowing the password. +- `web.password` is operator input only. It is hashed with argon2id (OWASP parameters: t=2, m=19 MiB, p=1, PHC encoding) into `web.password_hash` and discarded. There is no `web.password` settings row, and `nxdns export` always writes `.password = null`. +- `web.password_hash` is the stored argon2id PHC string. Supplying it directly, for example from a previous export, is how a backup restores authentication without knowing the password. -A plaintext password that has not changed is verified rather than re-hashed, so -applying the same file twice leaves the same bytes in the database. That is what -keeps the export/import round trip byte-stable with a password in the file. It -costs a full argon2id computation either way — the verification is not a -shortcut, and caching the plaintext to skip it would be a security bug. +A plaintext password that has not changed is verified rather than re-hashed, so applying the same file twice leaves the same bytes in the database. That is what keeps the export/import round trip byte-stable with a password in the file. It costs a full argon2id computation either way — the verification is not a shortcut, and caching the plaintext to skip it would be a security bug. -Any change to whether a password is set is announced at startup, never left as a -count: +Any change to whether a password is set is announced at startup, never left as a count: ``` web authentication is now enabled web authentication is now disabled ``` -See [set up admin -authentication](../how-to/set-up-admin-authentication.md). +See [set up admin authentication](../how-to/set-up-admin-authentication.md). ## Validation errors -`nxdns check`, `nxdns import` and a `nxdns run --config` that reads the file -all print one `FAIL path: message` line per problem, and report every -problem rather than the first. A finding that is legal but almost certainly -unintended is prefixed `WARN` instead: it does not change the exit code, and it -is printed by all three even when nothing failed, so an accepted configuration -still says what is odd about it. +`nxdns check`, `nxdns import` and a `nxdns run --config` that reads the file all print one `FAIL path: message` line per problem, and report every problem rather than the first. A finding that is legal but almost certainly unintended is prefixed `WARN` instead: it does not change the exit code, and it is printed by all three even when nothing failed, so an accepted configuration still says what is odd about it. -A url in a diagnostic is redacted to its scheme, host and port. The userinfo, -the path, the query and the fragment are dropped, and control characters are -escaped. These lines reach the journal, and every one of those parts can carry a -credential: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the -path segment is the whole account identifier. The field path beside the message -names the entry, so `blocklist_sources[1].url` still says which one to go and -fix. +A url in a diagnostic is redacted to its scheme, host and port. The userinfo, the path, the query and the fragment are dropped, and control characters are escaped. These lines reach the journal, and every one of those parts can carry a credential: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the path segment is the whole account identifier. The field path beside the message names the entry, so `blocklist_sources[1].url` still says which one to go and fix. -What redaction cannot remove is the host, because a hostname is not a secret in -the general case — it is resolved publicly and offered as SNI on every -connection — and dropping it would leave a diagnostic that names nothing worth -reading. A vendor that puts an account identifier in the hostname therefore has -that identifier appear in any line naming the entry. NextDNS is the example: -`abcd12.dns.nextdns.io` is a profile id. +What redaction cannot remove is the host, because a hostname is not a secret in the general case — it is resolved publicly and offered as SNI on every connection — and dropping it would leave a diagnostic that names nothing worth reading. A vendor that puts an account identifier in the hostname therefore has that identifier appear in any line naming the entry. NextDNS is the example: `abcd12.dns.nextdns.io` is a profile id. -Two things limit the exposure. A hostname `tls://` upstream is rejected by the -validator (see `upstreams` above), so the DoT form of that URL can only reach -the log once, in the line rejecting it — never on every failover. And NextDNS -publishes a DoH endpoint, `https://dns.nextdns.io/abcd12`, whose identifier sits -in the path and is redacted in full; configure that form and nothing identifying -reaches the log at all. +Two things limit the exposure. A hostname `tls://` upstream is rejected by the validator (see `upstreams` above), so the DoT form of that URL can only reach the log once, in the line rejecting it — never on every failover. And NextDNS publishes a DoH endpoint, `https://dns.nextdns.io/abcd12`, whose identifier sits in the path and is redacted in full; configure that form and nothing identifying reaches the log at all. The error set is `validate.ValidateError` in `src/config/validate.zig`: @@ -550,8 +393,7 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`: | `MissingLogPath` | `logging.output = .file` with an empty or relative `file_path` | | `PasswordAndHashBothSet` | both `web.password` and `web.password_hash` are set | -Warnings are a separate set, outside `ValidateError` because they are not -failures. There is one: +Warnings are a separate set, outside `ValidateError` because they are not failures. There is one: | Warning | Raised by | | --- | --- | @@ -559,8 +401,7 @@ failures. There is one: ## Minimal working example -The smallest file that passes validation: a `default` group and one enabled -upstream. Everything else keeps its default. +The smallest file that passes validation: a `default` group and one enabled upstream. Everything else keeps its default. ```zon .{ diff --git a/docs/reference/files-and-directories.md b/docs/reference/files-and-directories.md index 4311c63..e20751d 100644 --- a/docs/reference/files-and-directories.md +++ b/docs/reference/files-and-directories.md @@ -1,39 +1,16 @@ # Files and directories -Every path nxdns reads or writes, and the mode it is created with. Source of -truth: `src/cli.zig` (`DataDir`), `src/storage/querylog_schema.zig` (the -preserved query-log databases), `src/filter/manager.zig` (the blocklist -snapshots), `src/platform/logging.zig` (the log file). +Every path nxdns reads or writes, and the mode it is created with. Source of truth: `src/cli.zig` (`DataDir`), `src/storage/querylog_schema.zig` (the preserved query-log databases), `src/filter/manager.zig` (the blocklist snapshots), `src/platform/logging.zig` (the log file). ## The data directory -Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and -`nxdns import` create it and its parents at mode 0700 when it is missing; -`nxdns check` and `nxdns export` do not create it. `export` fails if it is not -there. `check` opens it only when no `--config FILE` was given: with that flag it -grades the file and never looks at the directory at all. Without it, an absent -`config.db` is a failure naming the two ways to get one, exit 2. +Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and `nxdns import` create it and its parents at mode 0700 when it is missing; `nxdns check` and `nxdns export` do not create it. `export` fails if it is not there. `check` opens it only when no `--config FILE` was given: with that flag it grades the file and never looks at the directory at all. Without it, an absent `config.db` is a failure naming the two ways to get one, exit 2. -`run`, `import` and `export` go through `DataDir.openConfigDb`, which opens -`config.db` read/write, chmods it to 0600, enables WAL — creating -`config.db-wal` and `config.db-shm` — and then runs any pending schema -migrations. So `nxdns export` writes to the data directory, and on a database -one schema version behind it migrates it. `export` also creates an empty -`config.db` if the directory exists without one. +`run`, `import` and `export` go through `DataDir.openConfigDb`, which opens `config.db` read/write, chmods it to 0600, enables WAL — creating `config.db-wal` and `config.db-shm` — and then runs any pending schema migrations. So `nxdns export` writes to the data directory, and on a database one schema version behind it migrates it. `export` also creates an empty `config.db` if the directory exists without one. -`nxdns check` is the exception: it does not use that path at all. It opens -`config.db` immutable, which is `SQLITE_OPEN_READONLY` plus `immutable=1`, so -SQLite refuses every statement that would write and builds no wal-index. No -`config.db-wal` and no `config.db-shm` appear beside the file, the mode is left -alone, and no migration runs — a database behind this binary's schema is -reported as a failure naming `nxdns run` as the fix. A `check` against a data -directory leaves it byte-identical, and `check` reaches the database branch only -when `config.db` is already there. +`nxdns check` is the exception: it does not use that path at all. It opens `config.db` immutable, which is `SQLITE_OPEN_READONLY` plus `immutable=1`, so SQLite refuses every statement that would write and builds no wal-index. No `config.db-wal` and no `config.db-shm` appear beside the file, the mode is left alone, and no migration runs — a database behind this binary's schema is reported as a failure naming `nxdns run` as the fix. A `check` against a data directory leaves it byte-identical, and `check` reaches the database branch only when `config.db` is already there. -`immutable=1` ignores any `-wal` file, so it is refused rather than used when -one holds bytes: the newest settings would be invisible and `check` would grade -older ones from the main file. That is the "uncheckpointed changes" failure in -[the CLI reference](cli.md#what-check-does-not-do). +`immutable=1` ignores any `-wal` file, so it is refused rather than used when one holds bytes: the newest settings would be invisible and `check` would grade older ones from the main file. That is the "uncheckpointed changes" failure in [the CLI reference](cli.md#what-check-does-not-do). | Path | What it is | Mode | | --- | --- | --- | @@ -53,29 +30,15 @@ older ones from the main file. That is the "uncheckpointed changes" failure in ### The orphan sweep -The sweep decides by id, not by suffix. It matches all seven names above and -deletes those whose `` is no longer a `blocklist_sources` row, so the -compiled `.list`, `.wild` and `.allow` of a removed source go, and so do a -`.raw.tmp`, `.list.tmp`, `.wild.tmp` or `.allow.tmp` left behind by a refresh -that was killed before it could clean up. Files belonging to a source that still -has a row are never touched, whatever state they are in: the sweep holds the same -lock every refresh takes, so it never reads the directory while a refresh is -part-way through. +The sweep decides by id, not by suffix. It matches all seven names above and deletes those whose `` is no longer a `blocklist_sources` row, so the compiled `.list`, `.wild` and `.allow` of a removed source go, and so do a `.raw.tmp`, `.list.tmp`, `.wild.tmp` or `.allow.tmp` left behind by a refresh that was killed before it could clean up. Files belonging to a source that still has a row are never touched, whatever state they are in: the sweep holds the same lock every refresh takes, so it never reads the directory while a refresh is part-way through. It runs at three moments: -- at startup, before the first refresh pass — this is what collects what a - killed process left behind, and the compiled files of a source deleted while - the server was down; -- before each scheduled update pass, ahead of the disk-space gate: the sweep - only unlinks, so it is the one step here that can give a critically full disk - room back, and gating it would keep the residue that helped fill the disk; -- immediately after `DELETE /api/blocklists/{id}`, which is when an orphan is - actually created in normal operation. Without it a deleted list would keep its - megabytes until the next scheduled pass. +- at startup, before the first refresh pass — this is what collects what a killed process left behind, and the compiled files of a source deleted while the server was down; +- before each scheduled update pass, ahead of the disk-space gate: the sweep only unlinks, so it is the one step here that can give a critically full disk room back, and gating it would keep the residue that helped fill the disk; +- immediately after `DELETE /api/blocklists/{id}`, which is when an orphan is actually created in normal operation. Without it a deleted list would keep its megabytes until the next scheduled pass. -With `blocklist_update.enabled = false` there are no scheduled passes, so only -the first and the last of those three happen. +With `blocklist_update.enabled = false` there are no scheduled passes, so only the first and the last of those three happen. Each deletion is logged: @@ -83,110 +46,48 @@ Each deletion is logged: info(blocklist_manager): pruned orphaned blocklist file 9999.list ``` -A failed sweep is a warning, not an outage — leftover bytes do not justify -losing the refresh pass behind them, let alone the server. +A failed sweep is a warning, not an outage — leftover bytes do not justify losing the refresh pass behind them, let alone the server. -The temporaries of a source that still exists are cleaned by the refresh that -owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`, -`.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not. +The temporaries of a source that still exists are cleaned by the refresh that owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`, `.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not. -A `querylog.db` is moved aside when it is missing nothing but usability: -SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not -answer `ok`, or its `user_version` fingerprint does not match the schema. Only -the main file is renamed — its `-wal` and `-shm` are deleted, because a stale -WAL would be replayed into the fresh database. A missing `querylog.db` is -created without any aside file. The rename happens inside -`querylog_schema.open`, before the 0600 chmod, and that chmod names -`querylog.db` and its two sidecars only — so an aside file keeps the mode the -file had at rename time, which for a `querylog.db` nxdns itself created is 0600 -and for one an operator put there is whatever they left it at. Nothing prunes -the aside files; they accumulate until an operator removes them, and each one -holds the same browsing history the live query log holds. +A `querylog.db` is moved aside when it is missing nothing but usability: SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not answer `ok`, or its `user_version` fingerprint does not match the schema. Only the main file is renamed — its `-wal` and `-shm` are deleted, because a stale WAL would be replayed into the fresh database. A missing `querylog.db` is created without any aside file. The rename happens inside `querylog_schema.open`, before the 0600 chmod, and that chmod names `querylog.db` and its two sidecars only — so an aside file keeps the mode the file had at rename time, which for a `querylog.db` nxdns itself created is 0600 and for one an operator put there is whatever they left it at. Nothing prunes the aside files; they accumulate until an operator removes them, and each one holds the same browsing history the live query log holds. -The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash -and `querylog.db` holds the browsing history of every client on the LAN, so both -are as sensitive as each other, and a WAL file holds the same rows as the -database it belongs to. SQLite creates the main database at `0644 & ~umask`; -nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather -than being created world-readable. +The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash and `querylog.db` holds the browsing history of every client on the LAN, so both are as sensitive as each other, and a WAL file holds the same rows as the database it belongs to. SQLite creates the main database at `0644 & ~umask`; nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather than being created world-readable. ## The configuration file -There is no default path. `--config FILE` names the file, and without that flag -no file is read at all — a `config.zon` sitting in `/etc/nxdns` that no -invocation names is inert. `/etc/nxdns/config.zon` is a convention the packaging -follows, not a location nxdns probes. +There is no default path. `--config FILE` names the file, and without that flag no file is read at all — a `config.zon` sitting in `/etc/nxdns` that no invocation names is inert. `/etc/nxdns/config.zon` is a convention the packaging follows, not a location nxdns probes. -nxdns reads the file and never writes it, in either authority mode. It does not -create the file or its directory either; the systemd unit's -`ConfigurationDirectory=nxdns` creates `/etc/nxdns`, and the same unit's -`ReadOnlyPaths=/etc/nxdns` denies the service write access to it, so the file -cannot be modified by the process that reads it. +nxdns reads the file and never writes it, in either authority mode. It does not create the file or its directory either; the systemd unit's `ConfigurationDirectory=nxdns` creates `/etc/nxdns`, and the same unit's `ReadOnlyPaths=/etc/nxdns` denies the service write access to it, so the file cannot be modified by the process that reads it. -Under `run --config FILE` the file is the configuration and the database is the -runtime substrate the server reads from: every start reconciles the one onto the -other. So in that mode `config.db` is not the backup — the file is. +Under `run --config FILE` the file is the configuration and the database is the runtime substrate the server reads from: every start reconciles the one onto the other. So in that mode `config.db` is not the backup — the file is. -`nxdns export --out FILE` writes a ZON file at mode 0600 through a temporary -file and a rename. That file carries `web.password_hash`, so treat exports as -secrets. Without `--out` the export goes to stdout, where permissions are the -redirect's business. +`nxdns export --out FILE` writes a ZON file at mode 0600 through a temporary file and a rename. That file carries `web.password_hash`, so treat exports as secrets. Without `--out` the export goes to stdout, where permissions are the redirect's business. ## The log file -Only when `logging.output = .file`. The path is `logging.file_path`, default -`/var/log/nxdns/nxdns.log`, and validation requires it to be absolute. +Only when `logging.output = .file`. The path is `logging.file_path`, default `/var/log/nxdns/nxdns.log`, and validation requires it to be absolute. -**nxdns does not create the log directory.** It must exist and be writable by -the user the service runs as before the process starts; under systemd the unit's -`LogsDirectory=nxdns` does that. +**nxdns does not create the log directory.** It must exist and be writable by the user the service runs as before the process starts; under systemd the unit's `LogsDirectory=nxdns` does that. -Rotation triggers at `logging.max_size_mb` MiB and keeps `logging.max_files` -files in total, the live one included, so the highest generation on disk is -`logging.max_files - 1`. With the default 5 that is `nxdns.log` plus `nxdns.log.1` -through `nxdns.log.4`. Rotation deletes the highest generation, renames each -remaining one up by one, then renames the live file to `.1`. A `max_files` of 1 -or 0 deletes the live file instead of renaming it. The directory holding the log -file is also sampled by the disk monitor. +Rotation triggers at `logging.max_size_mb` MiB and keeps `logging.max_files` files in total, the live one included, so the highest generation on disk is `logging.max_files - 1`. With the default 5 that is `nxdns.log` plus `nxdns.log.1` through `nxdns.log.4`. Rotation deletes the highest generation, renames each remaining one up by one, then renames the live file to `.1`. A `max_files` of 1 or 0 deletes the live file instead of renaming it. The directory holding the log file is also sampled by the disk monitor. | Path | What it is | Mode | | --- | --- | --- | | `logging.file_path` (default `/var/log/nxdns/nxdns.log`) | The live process log. Opened write-only and written at a tracked offset; created when it does not exist. | `0666 & ~umask` — nxdns never chmods it | | `.1` … `.` | Rotated generations, newest first. | Inherited from the live file they were renamed from | -**Do not point external logrotate at this file.** nxdns owns the rotation of -its own log. There is no append mode in Zig 0.16, so the writer reads the file -length once when it opens the file and then writes every line at an offset it -tracks in the process. A rotator that moves the file behind it breaks that -offset, and nxdns does not notice until the next restart: +**Do not point external logrotate at this file.** nxdns owns the rotation of its own log. There is no append mode in Zig 0.16, so the writer reads the file length once when it opens the file and then writes every line at an offset it tracks in the process. A rotator that moves the file behind it breaks that offset, and nxdns does not notice until the next restart: -- With `copytruncate` the offset survives the truncation, so the next line - lands where it would have without it. The file regrows with a sparse, - NUL-filled prefix as long as the log that was just rotated away. -- With rename-and-create nxdns keeps writing to the renamed inode. The new - file stays empty, the renamed one grows without bound, and - `logging.max_size_mb` bounds nothing on disk. +- With `copytruncate` the offset survives the truncation, so the next line lands where it would have without it. The file regrows with a sparse, NUL-filled prefix as long as the log that was just rotated away. +- With rename-and-create nxdns keeps writing to the renamed inode. The new file stays empty, the renamed one grows without bound, and `logging.max_size_mb` bounds nothing on disk. -If an external rotator has to own the file, set `logging.output = .stderr` and -let the collector capture the stream instead. +If an external rotator has to own the file, set `logging.output = .stderr` and let the collector capture the stream instead. -The log file is the one path here nxdns does not set a mode on. `config.db` and -`querylog.db` are chmodded to 0600 whatever the umask; the log file is left at -whatever the umask gives it. Under the shipped unit that is 0600, because -`deploy/systemd/nxdns.service` sets `UMask=0077`. Run from a shell with the -usual `umask 022` it is 0644, and `LogsDirectory=nxdns` leaves the directory at -systemd's default 0755, so a log written there is readable by any local user. +The log file is the one path here nxdns does not set a mode on. `config.db` and `querylog.db` are chmodded to 0600 whatever the umask; the log file is left at whatever the umask gives it. Under the shipped unit that is 0600, because `deploy/systemd/nxdns.service` sets `UMask=0077`. Run from a shell with the usual `umask 022` it is 0644, and `LogsDirectory=nxdns` leaves the directory at systemd's default 0755, so a log written there is readable by any local user. -With `logging.output = .stderr` or `.syslog` nxdns writes to stderr and creates -no file; under systemd the journal captures it. +With `logging.output = .stderr` or `.syslog` nxdns writes to stderr and creates no file; under systemd the journal captures it. ## Certificate and key files -`doh_server.cert_path` / `key_path` and `dot_server.cert_path` / `key_path`, -conventionally under `/etc/nxdns`. nxdns reads them, never writes or creates -them. Both must be readable by the user nxdns runs as, and the key must belong -to the certificate: `nxdns check` loads the pair and fails when it does not. The -key should also be readable by its owner only, which `check` warns about when it -is not. A -watcher polls both files and swaps a renewed pair in without a restart. See -[enable DoH and DoT](../how-to/enable-doh-and-dot.md). +`doh_server.cert_path` / `key_path` and `dot_server.cert_path` / `key_path`, conventionally under `/etc/nxdns`. nxdns reads them, never writes or creates them. Both must be readable by the user nxdns runs as, and the key must belong to the certificate: `nxdns check` loads the pair and fails when it does not. The key should also be readable by its owner only, which `check` warns about when it is not. A watcher polls both files and swaps a renewed pair in without a restart. See [enable DoH and DoT](../how-to/enable-doh-and-dot.md). diff --git a/docs/reference/performance.md b/docs/reference/performance.md index f2e2e68..66f1350 100644 --- a/docs/reference/performance.md +++ b/docs/reference/performance.md @@ -1,9 +1,6 @@ # Performance reference -The PLAN §18 targets and the numbers measured against them. Why the targets are -these targets, and why CI does not gate on them, is [performance and -testing](../explanation/performance-and-testing.md). To reproduce the numbers, -see [measure performance](../how-to/measure-performance.md). +The PLAN §18 targets and the numbers measured against them. Why the targets are these targets, and why CI does not gate on them, is [performance and testing](../explanation/performance-and-testing.md). To reproduce the numbers, see [measure performance](../how-to/measure-performance.md). ## Targets (PLAN §18) @@ -15,25 +12,15 @@ see [measure performance](../how-to/measure-performance.md). | Memory with ~1M blocked domains < 100 MiB | `bench filter`: VmRSS with the 1M-domain snapshot loaded | | Stripped static binary ≤ 10,485,760 bytes per arch without the embedded frontend, ≤ 15,728,640 bytes with it | `zig build verify-dist`, run by the `package` gate and by the release | -The harness is `tools/bench.zig`. It measures the three targets that are -measurable in process; the qps target is end to end and the binary-size target -belongs to the packaging step. +The harness is `tools/bench.zig`. It measures the three targets that are measurable in process; the qps target is end to end and the binary-size target belongs to the packaging step. -The two size budgets are exact byte counts, not rounded mebibytes, because an -assert on a rounded number is an assert on a number nobody wrote down. -`verify-dist` checks the shipped binary against the larger budget and builds a -second time against a generated empty assets directory for the smaller one, so -the asset-free figure is a real measurement rather than an estimate. +The two size budgets are exact byte counts, not rounded mebibytes, because an assert on a rounded number is an assert on a number nobody wrote down. `verify-dist` checks the shipped binary against the larger budget and builds a second time against a generated empty assets directory for the smaller one, so the asset-free figure is a real measurement rather than an estimate. ## Measured: x86_64 development host -Date: 2026-08-13. Hardware and build: Intel Core i7-14700K, Linux 6.18, -Zig 0.16.0, `-Doptimize=ReleaseFast`, harness defaults (1,000,000 domains, -200,000 iterations per suite, seed 0x5eed). +Date: 2026-08-13. Hardware and build: Intel Core i7-14700K, Linux 6.18, Zig 0.16.0, `-Doptimize=ReleaseFast`, harness defaults (1,000,000 domains, 200,000 iterations per suite, seed 0x5eed). -This is not the target platform. The Pi 5's Cortex-A76 is far slower and these -numbers do not transfer; they establish that the harness works and set a -baseline for regressions on the machine development happens on. +This is not the target platform. The Pi 5's Cortex-A76 is far slower and these numbers do not transfer; they establish that the harness works and set a baseline for regressions on the machine development happens on. ``` suite ops p50(us) p95(us) p99(us) max(us) @@ -47,27 +34,15 @@ cache 200000 0.11 0.17 0.22 5.40 compile 1000000 wall 98.597ms, 10142224 lines/s, 1000000 domains kept (informational) ``` -Every in-process §18 target passes on this host: the filter target by about -360x, the cache target by about four orders of magnitude, the memory target by -about 3x. +Every in-process §18 target passes on this host: the filter target by about 360x, the cache target by about four orders of magnitude, the memory target by about 3x. -The filter suite loads 32 regex rules that no query in the mix matches, which is -the expensive case rather than the cheap one: the regex levels sit below every -hash and wildcard level, so a name no pattern matches is the name that runs all -32 programs to their end. Every op pays that, which is what moved the filter p95 -from 0.18 µs before regex rules existed to the 2.80 µs above. The margin against -the 1 ms target is what makes paying it on every miss an acceptable price. +The filter suite loads 32 regex rules that no query in the mix matches, which is the expensive case rather than the cheap one: the regex levels sit below every hash and wildcard level, so a name no pattern matches is the name that runs all 32 programs to their end. Every op pays that, which is what moved the filter p95 from 0.18 µs before regex rules existed to the 2.80 µs above. The margin against the 1 ms target is what makes paying it on every miss an acceptable price. ### The two memory figures -`Snapshot.memoryBytes` and `DnsCache.memoryBytes` are the in-repo accounting of -the structures themselves, which is the regression guard. VmRSS is what the -kernel holds resident for the whole process, allocator slack and code included. -The truth sits between them, and the §18 memory target is judged on VmRSS. +`Snapshot.memoryBytes` and `DnsCache.memoryBytes` are the in-repo accounting of the structures themselves, which is the regression guard. VmRSS is what the kernel holds resident for the whole process, allocator slack and code included. The truth sits between them, and the §18 memory target is judged on VmRSS. -The filter suite frees the generated list source before reading VmRSS, so its -number reflects the loaded snapshot rather than the generator. The cache suite's -VmRSS is lower because the filter suite's snapshot has been freed by then. +The filter suite frees the generated list source before reading VmRSS, so its number reflects the loaded snapshot rather than the generator. The cache suite's VmRSS is lower because the filter suite's snapshot has been freed by then. ## Raspberry Pi 5 (target platform) @@ -78,6 +53,4 @@ VmRSS is lower because the filter suite's snapshot has been freed by then. | Memory with ~1M blocked domains < 100 MiB | to be measured on hardware | | Sustained ≥ 100 qps | to be measured on hardware, end to end | -The qps target belongs to the real binary rather than the harness: it means -`nxdns run` on the Pi, driven over the LAN by a DNS load generator against real -blocklists. +The qps target belongs to the real binary rather than the harness: it means `nxdns run` on the Pi, driven over the LAN by a DNS load generator against real blocklists. diff --git a/docs/tutorial/first-run.md b/docs/tutorial/first-run.md index 780443a..0598bb1 100644 --- a/docs/tutorial/first-run.md +++ b/docs/tutorial/first-run.md @@ -1,32 +1,19 @@ # Your first nxdns -This page takes you from a checkout of the repository to a running nxdns that -answers DNS queries, blocks domains from a real blocklist, and shows you the web -interface. It runs on unprivileged ports in a scratch directory, so nothing on -your machine changes and nothing needs root. At the end you stop the server and -delete the directory. +This page takes you from a checkout of the repository to a running nxdns that answers DNS queries, blocks domains from a real blocklist, and shows you the web interface. It runs on unprivileged ports in a scratch directory, so nothing on your machine changes and nothing needs root. At the end you stop the server and delete the directory. Follow the steps in order. Each one says what it did. -Every command below was executed on x86_64 Linux with Zig 0.16.0, Node.js -24.14.1, dig 9.20.26 and curl 8.21.0. Steps 2, 4 to 11, 13 and 14 were re-run -end to end for this revision, and the transcripts are that run's output with the -tutorial directory substituted. Two things were not re-run: the browser page in -step 12 — its endpoints were exercised, the page itself was not opened — and the -`npm` build in step 1, whose `web/dist` was already on disk and is the one the -binary under test embeds. The ZON block at the end of step 14 was checked with -`nxdns check --config` rather than started. +Every command below was executed on x86_64 Linux with Zig 0.16.0, Node.js 24.14.1, dig 9.20.26 and curl 8.21.0. Steps 2, 4 to 11, 13 and 14 were re-run end to end for this revision, and the transcripts are that run's output with the tutorial directory substituted. Two things were not re-run: the browser page in step 12 — its endpoints were exercised, the page itself was not opened — and the `npm` build in step 1, whose `web/dist` was already on disk and is the one the binary under test embeds. The ZON block at the end of step 14 was checked with `nxdns check --config` rather than started. ## What you need - Zig 0.16.0 and Node.js 24, both on your `PATH`. - `dig` (from bind-tools or dnsutils) and `curl`. -- Working internet access: nxdns queries an upstream resolver over HTTPS and - downloads a 3 MB blocklist. +- Working internet access: nxdns queries an upstream resolver over HTTPS and downloads a 3 MB blocklist. - Two terminals. The server runs in the first one; you type into the second. -Build commands run from the repository root. The server and the queries use -`~/nxdns-tutorial` for everything they write. +Build commands run from the repository root. The server and the queries use `~/nxdns-tutorial` for everything they write. ## 1. Build the web interface @@ -34,9 +21,7 @@ Build commands run from the repository root. The server and the queries use cd web && npm ci && npm run build && cd .. ``` -This produces `web/dist`. Do not skip it. A plain `zig build` embeds -`web/dist-placeholder`, a one-page status stub, and you would reach step 12 and -find no admin interface there. +This produces `web/dist`. Do not skip it. A plain `zig build` embeds `web/dist-placeholder`, a one-page status stub, and you would reach step 12 and find no admin interface there. ## 2. Build nxdns @@ -44,9 +29,7 @@ find no admin interface there. zig build -Dweb-dist=web/dist ``` -The binary is `zig-out/bin/nxdns`, with `web/dist` embedded in it. SQLite and -mbedTLS are vendored and built by this command, so the first build takes a -while; later builds are cached. +The binary is `zig-out/bin/nxdns`, with `web/dist` embedded in it. SQLite and mbedTLS are vendored and built by this command, so the first build takes a while; later builds are cached. ## 3. Write a configuration file @@ -68,22 +51,11 @@ EOF Four things are set, and the rest of nxdns keeps its defaults. -DNS is on port 15353 instead of 53, and both listeners are bound to localhost. -Port 53 is privileged: binding it needs root or `CAP_NET_BIND_SERVICE`, which is -an install decision, not a first-run decision. A real install is covered in -[how-to/install-with-systemd.md](../how-to/install-with-systemd.md). +DNS is on port 15353 instead of 53, and both listeners are bound to localhost. Port 53 is privileged: binding it needs root or `CAP_NET_BIND_SERVICE`, which is an install decision, not a first-run decision. A real install is covered in [how-to/install-with-systemd.md](../how-to/install-with-systemd.md). -The `default` group and one enabled upstream are the two things nxdns will not -start without. Every client that nxdns has never seen is assigned to `default`, -and with no usable upstream there is nowhere to send a query it cannot answer -itself, so a configuration missing either one is rejected. Whichever command -reads the file says the same thing and stops the same way: `run`, `nxdns check` -and `nxdns import` all print the problem and exit 2. +The `default` group and one enabled upstream are the two things nxdns will not start without. Every client that nxdns has never seen is assigned to `default`, and with no usable upstream there is nowhere to send a query it cannot answer itself, so a configuration missing either one is rejected. Whichever command reads the file says the same thing and stops the same way: `run`, `nxdns check` and `nxdns import` all print the problem and exit 2. -This tutorial loads the file into the database once and then runs nxdns against -the database, which is what the packaged systemd unit does. There is a second -way to run nxdns, where the file itself stays the configuration; step 14 shows -what changes. +This tutorial loads the file into the database once and then runs nxdns against the database, which is what the packaged systemd unit does. There is a second way to run nxdns, where the file itself stays the configuration; step 14 shows what changes. ## 4. Check the configuration before starting @@ -97,10 +69,7 @@ OK upstreams[0] https://cloudflare-dns.com OK: no problems found ``` -`check` parses the file, validates it, and contacts each upstream to confirm it -answers. It exits 2 when it found something that has to be fixed and 0 -otherwise. It writes nothing and starts no listener, so you can run it as often -as you like. +`check` parses the file, validates it, and contacts each upstream to confirm it answers. It exits 2 when it found something that has to be fixed and 0 otherwise. It writes nothing and starts no listener, so you can run it as often as you like. ## 5. Load it into the database @@ -113,9 +82,7 @@ info(migrations): config.db migrated from schema version 0 to 1 imported /home/you/nxdns-tutorial/config.zon ``` -The data directory did not exist; nxdns created it at mode 0700 along with -`config.db`. From here the database holds the configuration, and you will change -it through the API rather than by editing the file again. +The data directory did not exist; nxdns created it at mode 0700 along with `config.db`. From here the database holds the configuration, and you will change it through the API rather than by editing the file again. ## 6. Start the server @@ -133,9 +100,7 @@ info(nxdns): nxdns serving on udp [::1]:15353 udp 127.0.0.1:15353 tcp info(web_server): web interface listening on 127.0.0.1:8080 ``` -No `--config` here, and that is the point: `authority: database` says the -database is the configuration and no file was opened at all. The file you wrote -in step 3 has done its job. +No `--config` here, and that is the point: `authority: database` says the database is the configuration and no file was opened at all. The file you wrote in step 3 has done its job. Leave this terminal running and switch to the second one. @@ -150,10 +115,7 @@ example.com. 229 IN A 172.66.147.243 example.com. 229 IN A 104.20.23.154 ``` -nxdns had no answer cached, so it forwarded the query to -`https://cloudflare-dns.com/dns-query` over HTTPS and returned what came back. -You now have a working resolver. It blocks nothing yet: the log line in step 6 -said `0 of 0 sources loaded`. +nxdns had no answer cached, so it forwarded the query to `https://cloudflare-dns.com/dns-query` over HTTPS and returned what came back. You now have a working resolver. It blocks nothing yet: the log line in step 6 said `0 of 0 sources loaded`. ## 8. Add a blocklist source @@ -167,25 +129,15 @@ curl -s -X POST http://127.0.0.1:8080/api/blocklists \ {"id":1,"url":"https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts","name":"StevenBlack hosts","enabled":true,"is_suggested":false} ``` -The API is unauthenticated here because the configuration set no `web.password`. -That is fine for a localhost tutorial and wrong for anything else; see -[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md). +The API is unauthenticated here because the configuration set no `web.password`. That is fine for a localhost tutorial and wrong for anything else; see [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md). Note the `"id":1` in the response. You need it in the next step. ## 9. Attach the source to the `default` group -A blocklist source belongs to the installation. Which groups use it is a -separate decision, which is what lets one group get a strict list and another -get none. A source that is attached to no group is downloaded and then filters -nothing. +A blocklist source belongs to the installation. Which groups use it is a separate decision, which is what lets one group get a strict list and another get none. A source that is attached to no group is downloaded and then filters nothing. -You are in that state right now, between the previous step and this one, and it -is legal rather than wrong — creating a source and attaching it afterwards is -the normal order, which is why the API accepted it without complaint. On a -stopped server `nxdns check` names such a source in a `WARN` line and still -exits 0, so a list that silently blocks nothing is something you can find out -about later. Attaching it now is what makes it take effect. +You are in that state right now, between the previous step and this one, and it is legal rather than wrong — creating a source and attaching it afterwards is the normal order, which is why the API accepted it without complaint. On a stopped server `nxdns check` names such a source in a `WARN` line and still exits 0, so a list that silently blocks nothing is something you can find out about later. Attaching it now is what makes it take effect. Ask which groups exist: @@ -197,9 +149,7 @@ curl -s http://127.0.0.1:8080/api/groups {"groups":[{"id":1,"name":"default","safe_search":false}]} ``` -The `default` group is id 1. Give it the source you just created — the request -body is the complete set of sources for that group, so sending `[1]` replaces -whatever was there: +The `default` group is id 1. Give it the source you just created — the request body is the complete set of sources for that group, so sending `[1]` replaces whatever was there: ```sh curl -s -X PUT http://127.0.0.1:8080/api/groups/1/sources \ @@ -223,30 +173,17 @@ curl -s -X POST http://127.0.0.1:8080/api/blocklists/update {"sources":[{"id":1,"state":"ok","loaded":true,"last_attempt":1786629237,"last_success":1786629238,"url":"https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts","last_error":"","domains":97648,"wildcards":0,"exceptions":0,"skipped_regex":0,"skipped_unsupported":0}]} ``` -The four zeros describe this particular download, not the hosts format. -`wildcards` counts entries covering a name and its subdomains, `exceptions` -counts the `@@` lines an Adblock Plus list uses to lift a name another list -blocks, and `skipped_regex` counts the regex lines nxdns declines to take from a -downloaded list. `skipped_unsupported` counts lines nxdns cannot translate into a -DNS decision, and a large value next to a small `domains` means the list targets -browsers rather than DNS. Only `exceptions` is Adblock-Plus-only: a hosts list -can carry regex lines, `*.`-prefixed wildcards, and bare sink addresses that -count as unsupported. This one carries none of them. +The four zeros describe this particular download, not the hosts format. `wildcards` counts entries covering a name and its subdomains, `exceptions` counts the `@@` lines an Adblock Plus list uses to lift a name another list blocks, and `skipped_regex` counts the regex lines nxdns declines to take from a downloaded list. `skipped_unsupported` counts lines nxdns cannot translate into a DNS decision, and a large value next to a small `domains` means the list targets browsers rather than DNS. Only `exceptions` is Adblock-Plus-only: a hosts list can carry regex lines, `*.`-prefixed wildcards, and bare sink addresses that count as unsupported. This one carries none of them. -The download is about 3 MB and takes a few seconds. Watch the first terminal -until this appears: +The download is about 3 MB and takes a few seconds. Watch the first terminal until this appears: ``` info(blocklist_manager): blocklist snapshot generation 6: 1 of 1 sources loaded, 2523973 bytes ``` -`1 of 1 sources loaded` is the line to wait for. nxdns builds each blocklist -snapshot in full and swaps it in atomically, so queries keep being answered from -the previous snapshot the whole time the new one is being built. After this, the -domain count in the JSON above — 97648 on the day this was run — is live. +`1 of 1 sources loaded` is the line to wait for. nxdns builds each blocklist snapshot in full and swaps it in atomically, so queries keep being answered from the previous snapshot the whole time the new one is being built. After this, the domain count in the JSON above — 97648 on the day this was run — is live. -From here on, the list is on disk under `~/nxdns-tutorial/data/blocklists`. -Restarting nxdns does not re-download it. +From here on, the list is on disk under `~/nxdns-tutorial/data/blocklists`. Restarting nxdns does not re-download it. ## 11. Watch a domain get blocked @@ -258,9 +195,7 @@ dig @127.0.0.1 -p 15353 doubleclick.net A +noall +answer doubleclick.net. 5 IN A 0.0.0.0 ``` -`0.0.0.0` with a TTL of 5 is a block, not an answer. The default block response -is the zero address and the default block TTL is 5 seconds, so a client that -caches the answer forgets it quickly after you unblock something. +`0.0.0.0` with a TTL of 5 is a block, not an answer. The default block response is the zero address and the default block TTL is 5 seconds, so a client that caches the answer forgets it quickly after you unblock something. A name that is not on the list still resolves normally: @@ -272,23 +207,11 @@ dig @127.0.0.1 -p 15353 wikipedia.org A +noall +answer wikipedia.org. 130 IN A 185.15.58.224 ``` -One thing to know before you try other names: an entry in a hosts list blocks -exactly the name it names. `doubleclick.net` is on this list, and so are -`ad.doubleclick.net` and `www.google-analytics.com`. `ads.doubleclick.net` is -not on it, and nxdns does not block it — the entry for the parent says nothing -about the child. Two things do walk up the parent chain, and neither is in play -here: a rule you write yourself, with a wildcard pattern such as -`*.doubleclick.net`, and an Adblock Plus list's `||doubleclick.net^`, which -covers the name and everything under it. This list is a hosts file, so when you -pick a domain to test against it, pick one that is literally in the file. +One thing to know before you try other names: an entry in a hosts list blocks exactly the name it names. `doubleclick.net` is on this list, and so are `ad.doubleclick.net` and `www.google-analytics.com`. `ads.doubleclick.net` is not on it, and nxdns does not block it — the entry for the parent says nothing about the child. Two things do walk up the parent chain, and neither is in play here: a rule you write yourself, with a wildcard pattern such as `*.doubleclick.net`, and an Adblock Plus list's `||doubleclick.net^`, which covers the name and everything under it. This list is a hosts file, so when you pick a domain to test against it, pick one that is literally in the file. ## 12. Open the web interface -Visit in a browser. This is the single-page application -you built in step 1, served out of the binary. The dashboard shows query and -block counts, and the Blocklists page shows the source you added with its -domain count. (The endpoints behind those two pages were checked while writing -this; the browser page itself was not opened on the verification host.) +Visit in a browser. This is the single-page application you built in step 1, served out of the binary. The dashboard shows query and block counts, and the Blocklists page shows the source you added with its domain count. (The endpoints behind those two pages were checked while writing this; the browser page itself was not opened on the verification host.) ## 13. Stop it @@ -307,18 +230,13 @@ info(blocklist_manager): blocklist snapshot generation 1: 1 of 1 sources loaded, info(nxdns): authority: database ``` -The blocklist came off disk rather than the network. The source you added, the -group it is attached to, and the query log all survived the restart because they -live in `~/nxdns-tutorial/data`, which is what `authority: database` means in -practice. +The blocklist came off disk rather than the network. The source you added, the group it is attached to, and the query log all survived the restart because they live in `~/nxdns-tutorial/data`, which is what `authority: database` means in practice. -Press Ctrl-C again to stop this second process. Nothing is listening on 15353 or -8080 now, and nothing of nxdns is running. +Press Ctrl-C again to stop this second process. Nothing is listening on 15353 or 8080 now, and nothing of nxdns is running. ## 14. See what the other mode does -You have been running in database mode. The alternative is to hand the same file -back as the configuration, which is what `--config` means: +You have been running in database mode. The alternative is to hand the same file back as the configuration, which is what `--config` means: ```sh zig-out/bin/nxdns run --data-dir ~/nxdns-tutorial/data --config ~/nxdns-tutorial/config.zon @@ -334,23 +252,11 @@ info(blocklist_manager): pruned orphaned blocklist file 1.wild info(blocklist_manager): pruned orphaned blocklist file 1.list ``` -**Read that first line.** The blocklist source is gone. That is not a bug — it is -the whole contract. The file you wrote in step 3 never mentioned a blocklist -source, and in file mode the file is the complete statement of what the -configuration is, so anything the database holds that the file does not name is -removed at every start. The reconcile said so in one line before doing it. +**Read that first line.** The blocklist source is gone. That is not a bug — it is the whole contract. The file you wrote in step 3 never mentioned a blocklist source, and in file mode the file is the complete statement of what the configuration is, so anything the database holds that the file does not name is removed at every start. The reconcile said so in one line before doing it. -The three `pruned` lines are the rest of that removal: with the row gone, the -compiled files it owned belong to nobody, so the sweep that runs at every start -deletes them. The three names are the three bodies one source compiles into — -exact domains, wildcards, and the exceptions an Adblock Plus list can lift. The -query log is untouched; what changed is the configuration, and it now matches -the file exactly. +The three `pruned` lines are the rest of that removal: with the row gone, the compiled files it owned belong to nobody, so the sweep that runs at every start deletes them. The three names are the three bodies one source compiles into — exact domains, wildcards, and the exceptions an Adblock Plus list can lift. The query log is untouched; what changed is the configuration, and it now matches the file exactly. -Neither mode is the "advanced" one. Database mode suits a box someone -administers through the web interface. File mode suits a file kept in git and -deployed by a tool, where the deployed file being what is running matters more -than clicking. To have kept the blocklist here, you would put it in the file: +Neither mode is the "advanced" one. Database mode suits a box someone administers through the web interface. File mode suits a file kept in git and deployed by a tool, where the deployed file being what is running matters more than clicking. To have kept the blocklist here, you would put it in the file: ```zon .blocklist_sources = .{ @@ -365,26 +271,17 @@ Ctrl-C to stop it. ## What you have now -A resolver that answers real queries, a real blocklist of about 98000 domains -attached to the default group, a query log, and a web interface — all inside one -directory you can delete: +A resolver that answers real queries, a real blocklist of about 98000 domains attached to the default group, a query log, and a web interface — all inside one directory you can delete: ```sh rm -rf ~/nxdns-tutorial ``` -You also saw the three things that surprise people most: a blocklist source -does nothing until a group uses it, which authority a start runs under is -printed rather than guessed, and in file mode anything the file does not name is -removed at the next start. +You also saw the three things that surprise people most: a blocklist source does nothing until a group uses it, which authority a start runs under is printed rather than guessed, and in file mode anything the file does not name is removed at the next start. ## Where to go next -- [how-to/install-with-systemd.md](../how-to/install-with-systemd.md) — the same - thing as a real service on port 53, including a Raspberry Pi 5. -- [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md) - — put a password on the web interface before it leaves localhost. -- [reference/configuration.md](../reference/configuration.md) — every field you - did not set. -- [explanation/configuration-model.md](../explanation/configuration-model.md) — - why there are two authority modes and what each one is for. +- [how-to/install-with-systemd.md](../how-to/install-with-systemd.md) — the same thing as a real service on port 53, including a Raspberry Pi 5. +- [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md) — put a password on the web interface before it leaves localhost. +- [reference/configuration.md](../reference/configuration.md) — every field you did not set. +- [explanation/configuration-model.md](../explanation/configuration-model.md) — why there are two authority modes and what each one is for. diff --git a/specs/milestone-10.md b/specs/milestone-10.md index f431415..cec8946 100644 --- a/specs/milestone-10.md +++ b/specs/milestone-10.md @@ -1,122 +1,33 @@ # Milestone 10: DoH/DoT server + cert watcher + certs/reload (PLAN Phase 9) -Goal: LAN clients resolve via DoH (RFC 8484 over HTTP/1.1 + TLS) and DoT (RFC 7858) -against local certs; certs hot-reload via a polling watcher and `POST /api/certs/reload` -(endpoint + openapi entry + contract test, deferred whole from milestone-8 ruling 2). +Goal: LAN clients resolve via DoH (RFC 8484 over HTTP/1.1 + TLS) and DoT (RFC 7858) against local certs; certs hot-reload via a polling watcher and `POST /api/certs/reload` (endpoint + openapi entry + contract test, deferred whole from milestone-8 ruling 2). -Ground truth: PLAN.md:23,155,194,232,500-503,525,547,615-616,647,664 (scope, layout, -config keys, endpoint, exit); PLAN.md:684 — HTTP/2 and DoQ are OUT; PLAN.md:297 — -"a failed reload publishes neither". Explore facts (verified this session): -- platform/tls_server.zig: ServerContext.init(gpa, cert_pem, key_pem) shared across - connections; binding is per-accept (mbedtls_ssl_setup at accept), so reload = publish - a new context pointer; the OLD context must outlive every stream set up against it. - ServerStream.reader()/writer() are real *Io.Reader/*Io.Writer; close(gpa) sends - close_notify but does NOT close the TCP stream. MBEDTLS_THREADING_C + PTHREAD are ON - (build.zig:255) — one context is safe under concurrent handshakes. -- std.http.Server.init(in: *Reader, out: *Writer) takes ARBITRARY reader/writer - (http/Server.zig:25) — web/server.zig's serveConn transfers with three lines changed. -- No usable file watching in 0.16 (inotify/fanotify are raw syscalls outside Io) — - the watcher POLLS Io.File.stat mtime+size. -- Handler.handle(self, io, which: Transport, from, query, response_buf, scratch) - (handler.zig:164); Transport {udp,tcp} drives truncation only; response_buf 512..65535; - Scratch ~7 KiB per connection. tcp_server serveConn (tcp_server.zig:265-338) is the - DoT loop verbatim modulo the stream; dns/transport.zig has prefix_len/parsePrefix/ - framePrefix/max_message_len. -- doh_client.zig: media_type "application/dns-message" (:16), contentTypeOk exported - (:171). base64 url_safe_no_pad = the RFC 8484 GET codec. http_util.queryPairs for ?dns=. +Ground truth: PLAN.md:23,155,194,232,500-503,525,547,615-616,647,664 (scope, layout, config keys, endpoint, exit); PLAN.md:684 — HTTP/2 and DoQ are OUT; PLAN.md:297 — "a failed reload publishes neither". Explore facts (verified this session): +- platform/tls_server.zig: ServerContext.init(gpa, cert_pem, key_pem) shared across connections; binding is per-accept (mbedtls_ssl_setup at accept), so reload = publish a new context pointer; the OLD context must outlive every stream set up against it. ServerStream.reader()/writer() are real *Io.Reader/*Io.Writer; close(gpa) sends close_notify but does NOT close the TCP stream. MBEDTLS_THREADING_C + PTHREAD are ON (build.zig:255) — one context is safe under concurrent handshakes. +- std.http.Server.init(in: *Reader, out: *Writer) takes ARBITRARY reader/writer (http/Server.zig:25) — web/server.zig's serveConn transfers with three lines changed. +- No usable file watching in 0.16 (inotify/fanotify are raw syscalls outside Io) — the watcher POLLS Io.File.stat mtime+size. +- Handler.handle(self, io, which: Transport, from, query, response_buf, scratch) (handler.zig:164); Transport {udp,tcp} drives truncation only; response_buf 512..65535; Scratch ~7 KiB per connection. tcp_server serveConn (tcp_server.zig:265-338) is the DoT loop verbatim modulo the stream; dns/transport.zig has prefix_len/parsePrefix/ framePrefix/max_message_len. +- doh_client.zig: media_type "application/dns-message" (:16), contentTypeOk exported (:171). base64 url_safe_no_pad = the RFC 8484 GET codec. http_util.queryPairs for ?dns=. - Fixtures: tests/fixtures self_signed cert/key via the test_fixtures anonymous import. -- Config: model.TlsEndpoint already exists (doh_server port 443, dot_server 853), - validated, settings-persisted, checked by `nxdns check`. app.zig ignores both today. +- Config: model.TlsEndpoint already exists (doh_server port 443, dot_server 853), validated, settings-persisted, checked by `nxdns check`. app.zig ignores both today. ## Rulings (binding) -1. **Two listeners**: `src/server/doh_server.zig`, `src/server/dot_server.zig` - (PLAN:194), constructed only when their endpoint is enabled. Bind failure = warn + - continue (web precedent). Both join app.zig's group; cancel semantics mirror - tcp_server's Stop enum (.canceled cancels the connection group; listener close - drains). -2. **DoH protocol**: HTTP/1.1 only (PLAN:684 excludes h2). Path `/dns-query` exactly. - `POST` with `content-type: application/dns-message` (checked via - doh_client.contentTypeOk) and `GET ?dns=`. Responses: - 200 `application/dns-message` with the reply bytes; handler `.drop` → 502 with empty - body reason? NO — `.drop` means "no answer on purpose" (malformed/refused-silent): - close the connection for POST/GET alike via 400. Errors: 404 non-/dns-query path, - 405 other methods (Allow: GET, POST), 415 wrong content-type, 400 missing/invalid - `dns` param or empty body, 413 body > dns/transport.max_message_len. No - cache-control headers (LAN, no intermediaries). Keep-alive per std.http.Server loop; - the bodyless-POST normalization from web/server.zig:443 is copied (same stdlib - assert). -3. **DoT protocol**: RFC 7858 = the tcp_server loop over ServerStream: 2-byte prefix, - parsePrefix, reject 0, read body, handle, frame + write + flush. idle_timeout race - identical (same Options shape, max_connections 64, per-conn Scratch). The TLS - handshake itself runs under the same race budget (a stalled handshake must not pin a - connection slot). -4. **Transport enum stays {udp, tcp}**: DoH/DoT call handle with `.tcp` (it only - drives truncation, which never applies to stream transports). No unused surface. -5. **ALPN, additive to platform**: ServerContext.init gains - `alpn: ?[*:null]const ?[*:0]const u8` (mbedtls_ssl_conf_alpn_protocols requires a - NULL-terminated array that OUTLIVES the config — use comptime-constant arrays). - DoH advertises ["http/1.1"], DoT ["dot"]. A client with no ALPN still connects - (mbedTLS only enforces when the client sends the extension). Existing callers pass - null; the milestone-1 spec note about "no ALPN" is superseded. -6. **Cert holder + reload**: new `src/server/cert_store.zig`. `CertStore` owns - {mutex, current: *Entry, cert_path, key_path, gpa} where Entry = {ctx: ServerContext, - refs: u32, retired: bool}. `acquire(io) *Entry` (+1 under mutex), - `release(io, entry)` (−1; free when retired and 0). `reload(io) ReloadError!void`: - read both PEM files (NUL-terminated dupes, 64 KiB cap each), ServerContext.init, on - success swap + retire the old (freed when its refs drain), on ANY failure publish - nothing (PLAN:297). Listeners acquire per connection (before accept) and release - when the connection ends — so in-flight streams keep their generation alive. - File read errors and init errors map to a typed error + a human message for the API. -7. **Cert watcher**: one task per enabled endpoint inside cert_store - (`watch(store, io)` house loop: Cancelable!void, .boot clock, warn-and-continue). - Poll every 30 s: stat cert AND key (Io.File.stat), compare mtime+size against the - last LOADED pair; any change → reload(); reload failure → warn (the old cert keeps - serving) + a `reload_failures` counter; success → info-level "certificate reloaded". - mtime+size both compared (same-second atomic renames). -8. **`POST /api/certs/reload`** (PLAN:547): handler `src/web/handlers/certs.zig` - (PLAN:232). auth = .session, rate_limit = .counted. Reloads every enabled endpoint's - store; response 200 `{"doh": {"enabled": bool, "reloaded": bool, "error": |null}, - "dot": {...}}` — per-endpoint outcome, 200 even when a reload fails (the outcome IS - the payload; nothing about the server's own state is exceptional). Disabled endpoint: - {enabled:false, reloaded:false, error:null}. WebState gains `doh_certs: ?*CertStore` - and `dot_certs: ?*CertStore` (null-defaulted, W3 pattern). -9. **openapi.yaml + contract test land together** (m8 ruling 2): the path entry with - the exact schema, and a contract-table entry + response-shape test in - web_integration_test.zig. Drift guard (b) count moves 55 → 56 — the test derives the - count from router.routes so only the yaml operation count assertion data changes. -10. **Observability**: each listener keeps tcp_server-shaped stats (connections, - tls_handshake_failures, idle_timeouts, connection_errors) + cert_store keeps - {reloads, reload_failures, last_reload_unix}. metrics.zig additively gains - `nxdns_doh_server_*`, `nxdns_dot_server_*`, `nxdns_cert_reloads_total`, - `nxdns_cert_reload_failures_total` families (counterGroup walks the structs — - follow its existing pattern). /api/health is unchanged (no new rollup this - milestone; the reload endpoint reports cert state on demand). -11. **app.zig wiring**: when `cfg.doh_server.enabled`, build its CertStore (initial - load happens in serve BEFORE the group — a bad cert at boot is exit 2 with the - validate-style message, matching `nxdns check`'s contract that enabled endpoints - have readable certs; the WATCHER handles later breakage gracefully), then the - listener; same for dot. Watcher tasks + listener tasks join the group. WebState - seams wired when web is also enabled. SIGTERM path unchanged. -12. **Integration tests** (both gated `-Dintegration`, loopback, fixtures): - DoT: std.crypto.tls.Client (no_verification) → framed A query → framed reply - (reuse resolver_integration_test's fake-upstream pattern or a local-records-only - handler). DoH POST + GET: raw HTTP/1.1 text over std.crypto.tls.Client (no - std.http.Client CA ceremony), assert status/content-type/bytes; 415/405/404/400 - matrix; keep-alive two-requests test. Reload: swap fixture cert (write a second - self-signed pair fixture), reload, NEW connection sees the new cert - (std.crypto.tls.Client exposes the peer cert? if not: assert old connections - still serve and new handshakes succeed — the observable contract), old connection - still answers. Watcher: unit-test the decision function (stat-pair compare) pure; - do not test 30 s timing. -13. **New fixture**: `tests/fixtures/self_signed_cert2.pem` + key2 (openssl one-liner, - same profile as the existing pair; README updated) for reload tests. -14. **Docs**: NOT this milestone beyond openapi.yaml (docs/api rendering is Phase 10 — - m8 ruling 3 unchanged). -15. **Zig freeze elsewhere**: dns/, filter/, cache/, storage/, existing server files - (udp/tcp/handler) untouched except: app.zig (T6), metrics.zig + routes.zig + - openapi.yaml + web_integration_test.zig + web/server.zig(WebState fields only) - (T5). handler.zig is NOT touched (ruling 4). +1. **Two listeners**: `src/server/doh_server.zig`, `src/server/dot_server.zig` (PLAN:194), constructed only when their endpoint is enabled. Bind failure = warn + continue (web precedent). Both join app.zig's group; cancel semantics mirror tcp_server's Stop enum (.canceled cancels the connection group; listener close drains). +2. **DoH protocol**: HTTP/1.1 only (PLAN:684 excludes h2). Path `/dns-query` exactly. `POST` with `content-type: application/dns-message` (checked via doh_client.contentTypeOk) and `GET ?dns=`. Responses: 200 `application/dns-message` with the reply bytes; handler `.drop` → 502 with empty body reason? NO — `.drop` means "no answer on purpose" (malformed/refused-silent): close the connection for POST/GET alike via 400. Errors: 404 non-/dns-query path, 405 other methods (Allow: GET, POST), 415 wrong content-type, 400 missing/invalid `dns` param or empty body, 413 body > dns/transport.max_message_len. No cache-control headers (LAN, no intermediaries). Keep-alive per std.http.Server loop; the bodyless-POST normalization from web/server.zig:443 is copied (same stdlib assert). +3. **DoT protocol**: RFC 7858 = the tcp_server loop over ServerStream: 2-byte prefix, parsePrefix, reject 0, read body, handle, frame + write + flush. idle_timeout race identical (same Options shape, max_connections 64, per-conn Scratch). The TLS handshake itself runs under the same race budget (a stalled handshake must not pin a connection slot). +4. **Transport enum stays {udp, tcp}**: DoH/DoT call handle with `.tcp` (it only drives truncation, which never applies to stream transports). No unused surface. +5. **ALPN, additive to platform**: ServerContext.init gains `alpn: ?[*:null]const ?[*:0]const u8` (mbedtls_ssl_conf_alpn_protocols requires a NULL-terminated array that OUTLIVES the config — use comptime-constant arrays). DoH advertises ["http/1.1"], DoT ["dot"]. A client with no ALPN still connects (mbedTLS only enforces when the client sends the extension). Existing callers pass null; the milestone-1 spec note about "no ALPN" is superseded. +6. **Cert holder + reload**: new `src/server/cert_store.zig`. `CertStore` owns {mutex, current: *Entry, cert_path, key_path, gpa} where Entry = {ctx: ServerContext, refs: u32, retired: bool}. `acquire(io) *Entry` (+1 under mutex), `release(io, entry)` (−1; free when retired and 0). `reload(io) ReloadError!void`: read both PEM files (NUL-terminated dupes, 64 KiB cap each), ServerContext.init, on success swap + retire the old (freed when its refs drain), on ANY failure publish nothing (PLAN:297). Listeners acquire per connection (before accept) and release when the connection ends — so in-flight streams keep their generation alive. File read errors and init errors map to a typed error + a human message for the API. +7. **Cert watcher**: one task per enabled endpoint inside cert_store (`watch(store, io)` house loop: Cancelable!void, .boot clock, warn-and-continue). Poll every 30 s: stat cert AND key (Io.File.stat), compare mtime+size against the last LOADED pair; any change → reload(); reload failure → warn (the old cert keeps serving) + a `reload_failures` counter; success → info-level "certificate reloaded". mtime+size both compared (same-second atomic renames). +8. **`POST /api/certs/reload`** (PLAN:547): handler `src/web/handlers/certs.zig` (PLAN:232). auth = .session, rate_limit = .counted. Reloads every enabled endpoint's store; response 200 `{"doh": {"enabled": bool, "reloaded": bool, "error": |null}, "dot": {...}}` — per-endpoint outcome, 200 even when a reload fails (the outcome IS the payload; nothing about the server's own state is exceptional). Disabled endpoint: {enabled:false, reloaded:false, error:null}. WebState gains `doh_certs: ?*CertStore` and `dot_certs: ?*CertStore` (null-defaulted, W3 pattern). +9. **openapi.yaml + contract test land together** (m8 ruling 2): the path entry with the exact schema, and a contract-table entry + response-shape test in web_integration_test.zig. Drift guard (b) count moves 55 → 56 — the test derives the count from router.routes so only the yaml operation count assertion data changes. +10. **Observability**: each listener keeps tcp_server-shaped stats (connections, tls_handshake_failures, idle_timeouts, connection_errors) + cert_store keeps {reloads, reload_failures, last_reload_unix}. metrics.zig additively gains `nxdns_doh_server_*`, `nxdns_dot_server_*`, `nxdns_cert_reloads_total`, `nxdns_cert_reload_failures_total` families (counterGroup walks the structs — follow its existing pattern). /api/health is unchanged (no new rollup this milestone; the reload endpoint reports cert state on demand). +11. **app.zig wiring**: when `cfg.doh_server.enabled`, build its CertStore (initial load happens in serve BEFORE the group — a bad cert at boot is exit 2 with the validate-style message, matching `nxdns check`'s contract that enabled endpoints have readable certs; the WATCHER handles later breakage gracefully), then the listener; same for dot. Watcher tasks + listener tasks join the group. WebState seams wired when web is also enabled. SIGTERM path unchanged. +12. **Integration tests** (both gated `-Dintegration`, loopback, fixtures): DoT: std.crypto.tls.Client (no_verification) → framed A query → framed reply (reuse resolver_integration_test's fake-upstream pattern or a local-records-only handler). DoH POST + GET: raw HTTP/1.1 text over std.crypto.tls.Client (no std.http.Client CA ceremony), assert status/content-type/bytes; 415/405/404/400 matrix; keep-alive two-requests test. Reload: swap fixture cert (write a second self-signed pair fixture), reload, NEW connection sees the new cert (std.crypto.tls.Client exposes the peer cert? if not: assert old connections still serve and new handshakes succeed — the observable contract), old connection still answers. Watcher: unit-test the decision function (stat-pair compare) pure; do not test 30 s timing. +13. **New fixture**: `tests/fixtures/self_signed_cert2.pem` + key2 (openssl one-liner, same profile as the existing pair; README updated) for reload tests. +14. **Docs**: NOT this milestone beyond openapi.yaml (docs/api rendering is Phase 10 — m8 ruling 3 unchanged). +15. **Zig freeze elsewhere**: dns/, filter/, cache/, storage/, existing server files (udp/tcp/handler) untouched except: app.zig (T6), metrics.zig + routes.zig + openapi.yaml + web_integration_test.zig + web/server.zig(WebState fields only) (T5). handler.zig is NOT touched (ruling 4). ## Sessions @@ -126,181 +37,63 @@ T1+T2 parallel first; then T3+T4+T5 parallel; then T6. ## Session T1: platform ALPN -Owns `src/platform/tls_server.zig` (+ its tests). Add the alpn parameter per ruling 5 -(extern mbedtls_ssl_conf_alpn_protocols, comptime NULL-terminated arrays, doc the -lifetime rule), update existing callers/tests with null, add one test that a client -negotiating "http/1.1" succeeds against a context advertising it (the loopback -integration test grows the assertion — std.crypto.tls.Client: check whether it can -send ALPN in 0.16; if it cannot, the unit test just proves config acceptance and the -lifetime doc stands). +Owns `src/platform/tls_server.zig` (+ its tests). Add the alpn parameter per ruling 5 (extern mbedtls_ssl_conf_alpn_protocols, comptime NULL-terminated arrays, doc the lifetime rule), update existing callers/tests with null, add one test that a client negotiating "http/1.1" succeeds against a context advertising it (the loopback integration test grows the assertion — std.crypto.tls.Client: check whether it can send ALPN in 0.16; if it cannot, the unit test just proves config acceptance and the lifetime doc stands). Acceptance: fmt/ast clean; plain + integration suites 0 failed. ### T1 As built -`ServerContext.init(gpa, cert_pem, key_pem, alpn: ?[*:null]const ?[*:0]const u8)`. -Extern `mbedtls_ssl_conf_alpn_protocols` verified against vendored 3.6.7 ssl.h:4314 -(pointer recorded, not copied — comptime-constant NULL-terminated arrays required, -documented on init). MBEDTLS_SSL_ALPN is on in the stock config. std.crypto.tls.Client -CANNOT send ALPN in 0.16 (no Options field) — negotiation untestable client-side; the -unit test proves config acceptance, the integration echo proves no-ALPN clients still -handshake against an advertising context. Servers can assert server-side via -mbedtls_ssl_get_alpn_protocol if ever needed (extern not declared — unused surface). -Mechanical ripple: tls_client_integration_test.zig:193 caller gained `, null`. +`ServerContext.init(gpa, cert_pem, key_pem, alpn: ?[*:null]const ?[*:0]const u8)`. Extern `mbedtls_ssl_conf_alpn_protocols` verified against vendored 3.6.7 ssl.h:4314 (pointer recorded, not copied — comptime-constant NULL-terminated arrays required, documented on init). MBEDTLS_SSL_ALPN is on in the stock config. std.crypto.tls.Client CANNOT send ALPN in 0.16 (no Options field) — negotiation untestable client-side; the unit test proves config acceptance, the integration echo proves no-ALPN clients still handshake against an advertising context. Servers can assert server-side via mbedtls_ssl_get_alpn_protocol if ever needed (extern not declared — unused surface). Mechanical ripple: tls_client_integration_test.zig:193 caller gained `, null`. ## Session T2: cert_store -Owns `src/server/cert_store.zig` (new). Ruling 6 (holder + refcount + reload) and -ruling 7 (watch loop + pure stat-compare decision fn) + ruling 10's counters + -`snapshotStats()`. Unit tests: acquire/release refcounting across a swap (old entry -freed only after last release), reload failure publishes nothing (bad path, bad PEM), -NUL-termination and size cap, stat-compare decision table. Uses fixtures via -test_fixtures import. +Owns `src/server/cert_store.zig` (new). Ruling 6 (holder + refcount + reload) and ruling 7 (watch loop + pure stat-compare decision fn) + ruling 10's counters + `snapshotStats()`. Unit tests: acquire/release refcounting across a swap (old entry freed only after last release), reload failure publishes nothing (bad path, bad PEM), NUL-termination and size cap, stat-compare decision table. Uses fixtures via test_fixtures import. Acceptance: fmt/ast clean; temp-root green (needs -lc + vendored mbedtls objects). ### T2 As built -16 in-file tests; temp-root 22 passed / 0 failed. Surface: -`poll_interval_s=30`, `max_pem_bytes=64KiB`, `ReloadError` (10 cases) + -`humanMessage(err) []const u8`; `FileSig{mtime_ns: i96, size}`, `Signature{cert,key}`, -pure `changed(loaded, observed) bool`; `Entry{ctx, refs, retired}`; -`CertStore.init(gpa, io, cert_path, key_path, alpn)` (boot load; typed error for T6's -exit-2), `deinit(io)` (asserts current refs==0 — join listeners first), -`acquire(io) *Entry` / `release(io, entry)` (release safe from any task; last releaser -frees a retired entry), `reload(io) ReloadError!void`, `watch(io) Cancelable!void` -(sleeps BEFORE first poll, .boot clock), `pollOnce(io)` (testable watcher pass), -`snapshotStats()` (no io; {reloads, reload_failures, last_reload_unix}). -Notes: use `&entry.ctx` for ServerStream.accept; paths are BORROWED (config outlives -the store); `reloads` counts successes only; stat failures warn but are not -reload_failures; `last_reload_unix` set at init too. +16 in-file tests; temp-root 22 passed / 0 failed. Surface: `poll_interval_s=30`, `max_pem_bytes=64KiB`, `ReloadError` (10 cases) + `humanMessage(err) []const u8`; `FileSig{mtime_ns: i96, size}`, `Signature{cert,key}`, pure `changed(loaded, observed) bool`; `Entry{ctx, refs, retired}`; `CertStore.init(gpa, io, cert_path, key_path, alpn)` (boot load; typed error for T6's exit-2), `deinit(io)` (asserts current refs==0 — join listeners first), `acquire(io) *Entry` / `release(io, entry)` (release safe from any task; last releaser frees a retired entry), `reload(io) ReloadError!void`, `watch(io) Cancelable!void` (sleeps BEFORE first poll, .boot clock), `pollOnce(io)` (testable watcher pass), `snapshotStats()` (no io; {reloads, reload_failures, last_reload_unix}). Notes: use `&entry.ctx` for ServerStream.accept; paths are BORROWED (config outlives the store); `reloads` counts successes only; stat failures warn but are not reload_failures; `last_reload_unix` set at init too. ## Session T3: dot_server (after T1+T2) -Owns `src/server/dot_server.zig` (new) + its integration test file if separate -(prefer tests inside the file, house pattern). Ruling 3: tcp_server loop shape over -ServerStream; acquire cert entry per connection, release on exit; handshake under the -race budget; stats per ruling 10; Stop enum; max_connections 64; ALPN ["dot"]. -Integration tests per ruling 12 (DoT half). +Owns `src/server/dot_server.zig` (new) + its integration test file if separate (prefer tests inside the file, house pattern). Ruling 3: tcp_server loop shape over ServerStream; acquire cert entry per connection, release on exit; handshake under the race budget; stats per ruling 10; Stop enum; max_connections 64; ALPN ["dot"]. Integration tests per ruling 12 (DoT half). Acceptance: fmt/ast clean; both suites 0 failed. ### T3 As built -`DotServer.listen(gpa, io, listen_address, h: *handler.Handler, certs: *cert_store.CertStore, options)` / -`serve(io)` / `boundAddress()` / `deinit(io)`. Options `{max_connections: u16 = 64, idle_timeout = 10s}`; -the handshake runs under the same idle budget. Deviation from tcp_server: `deinit` takes no -gpa — the allocator is stored at `listen` because `ServerStream.accept` heap-allocates one -mbedTLS ssl context per connection (documented in the file header). Stats atomics: -the four ruling-10 fields plus tcp-pattern internals `rejected_at_capacity, -rejected_at_shutdown, accept_errors`; `snapshotStats()` returns exactly the ruling-10 four. -Loop: accept → claim slot → `certs.acquire` (released on every exit) → handshake raced -against the idle budget → 2-byte-prefix loop over ServerStream reader/writer, flush per -reply, handler called with `.tcp`, `tls.close(gpa)` (close_notify) on every post-handshake -exit. A `handshook` flag ensures the ssl context closes exactly once when Select reports -expiry/cancel after a successful handshake; a timed-out handshake counts as -`tls_handshake_failures`. ALPN is set only via the store (T6). 7 unit + 4 integration -tests in-file. Teardown order: `server.deinit(io)` before `CertStore.deinit`. +`DotServer.listen(gpa, io, listen_address, h: *handler.Handler, certs: *cert_store.CertStore, options)` / `serve(io)` / `boundAddress()` / `deinit(io)`. Options `{max_connections: u16 = 64, idle_timeout = 10s}`; the handshake runs under the same idle budget. Deviation from tcp_server: `deinit` takes no gpa — the allocator is stored at `listen` because `ServerStream.accept` heap-allocates one mbedTLS ssl context per connection (documented in the file header). Stats atomics: the four ruling-10 fields plus tcp-pattern internals `rejected_at_capacity, rejected_at_shutdown, accept_errors`; `snapshotStats()` returns exactly the ruling-10 four. Loop: accept → claim slot → `certs.acquire` (released on every exit) → handshake raced against the idle budget → 2-byte-prefix loop over ServerStream reader/writer, flush per reply, handler called with `.tcp`, `tls.close(gpa)` (close_notify) on every post-handshake exit. A `handshook` flag ensures the ssl context closes exactly once when Select reports expiry/cancel after a successful handshake; a timed-out handshake counts as `tls_handshake_failures`. ALPN is set only via the store (T6). 7 unit + 4 integration tests in-file. Teardown order: `server.deinit(io)` before `CertStore.deinit`. ## Session T4: doh_server (after T1+T2) -Owns `src/server/doh_server.zig` (new). Ruling 2 in full: std.http.Server over -ServerStream reader/writer (recv buffer 8 KiB = head cap, send 4 KiB), connection -group + 64 cap + 503-free refusal is NOT applicable here (no HTTP yet at accept -overflow: over cap → close raw, cheapest honest behavior — document); per-connection -Scratch + response_buf (65535); GET base64url decode (url_safe_no_pad, reject padding/ -whitespace), POST body cap via Io.Limit; keep-alive; bodyless-POST normalization; -ALPN ["http/1.1"]; cert acquire/release per connection; stats. Integration tests per -ruling 12 (DoH half: POST, GET, 415/405/404/400, keep-alive). +Owns `src/server/doh_server.zig` (new). Ruling 2 in full: std.http.Server over ServerStream reader/writer (recv buffer 8 KiB = head cap, send 4 KiB), connection group + 64 cap + 503-free refusal is NOT applicable here (no HTTP yet at accept overflow: over cap → close raw, cheapest honest behavior — document); per-connection Scratch + response_buf (65535); GET base64url decode (url_safe_no_pad, reject padding/ whitespace), POST body cap via Io.Limit; keep-alive; bodyless-POST normalization; ALPN ["http/1.1"]; cert acquire/release per connection; stats. Integration tests per ruling 12 (DoH half: POST, GET, 415/405/404/400, keep-alive). Acceptance: fmt/ast clean; both suites 0 failed. ### T4 As built -`DohServer.listen(gpa, io, address, *Handler, *CertStore, Options)` / `serve(io)` / -`deinit(gpa, io)` / `boundAddress()` / `snapshotStats()`. Options -`{max_connections = 64, idle_timeout = 10s}` — the budget races the TLS handshake only; -requests have no timeout (web-listener precedent). Module-level -`serve(gpa, io, model.TlsEndpoint, *Handler, *CertStore)` is the bind-warn-and-continue -entry for T6. Exposed comptime `alpn_protocols` (["http/1.1"]) and `dns_query_path`. -Stats: the ruling-10 four + `bad_requests` + tcp-shaped accept counters -(`rejected_at_capacity, rejected_at_shutdown, accept_errors`) — over-capacity refusals -must stay visible. Deviations: POST cap via content-length precheck + fixed buffer + -one-byte probe for chunked bodies instead of Io.Limit (no allocation on the hot path); -`respond` can raise `HttpExpectationFailed` (100-continue), mapped to `connection_errors`; -no 413 integration test (ruling 12's matrix is 415/405/404/400). GET `?dns=` parsed by a -local validating decoder (duplicate param → 400, `%` and padding/whitespace rejected). -Oversize POST answers 413 with `keep_alive=false` (body never drained). Over the 64 cap -the raw TCP stream closes (no HTTP before a handshake). 8 unit + 4 integration tests. -T6 recipe: `CertStore.init(gpa, io, cert_path, key_path, doh_server.alpn_protocols)`; -run `doh_server.serve` and `store.watch` as group tasks; `DohServer.deinit` before -`CertStore.deinit`. +`DohServer.listen(gpa, io, address, *Handler, *CertStore, Options)` / `serve(io)` / `deinit(gpa, io)` / `boundAddress()` / `snapshotStats()`. Options `{max_connections = 64, idle_timeout = 10s}` — the budget races the TLS handshake only; requests have no timeout (web-listener precedent). Module-level `serve(gpa, io, model.TlsEndpoint, *Handler, *CertStore)` is the bind-warn-and-continue entry for T6. Exposed comptime `alpn_protocols` (["http/1.1"]) and `dns_query_path`. Stats: the ruling-10 four + `bad_requests` + tcp-shaped accept counters (`rejected_at_capacity, rejected_at_shutdown, accept_errors`) — over-capacity refusals must stay visible. Deviations: POST cap via content-length precheck + fixed buffer + one-byte probe for chunked bodies instead of Io.Limit (no allocation on the hot path); `respond` can raise `HttpExpectationFailed` (100-continue), mapped to `connection_errors`; no 413 integration test (ruling 12's matrix is 415/405/404/400). GET `?dns=` parsed by a local validating decoder (duplicate param → 400, `%` and padding/whitespace rejected). Oversize POST answers 413 with `keep_alive=false` (body never drained). Over the 64 cap the raw TCP stream closes (no HTTP before a handshake). 8 unit + 4 integration tests. T6 recipe: `CertStore.init(gpa, io, cert_path, key_path, doh_server.alpn_protocols)`; run `doh_server.serve` and `store.watch` as group tasks; `DohServer.deinit` before `CertStore.deinit`. ## Session T5: web endpoint + metrics (after T2, parallel with T3/T4) -Owns `src/web/handlers/certs.zig` (new), `src/web/routes.zig` (one entry), -`src/web/openapi.yaml` (one path), `src/web/server.zig` (two null-defaulted WebState -fields ONLY), `src/web/metrics.zig` (ruling 10 families), and the additive -web_integration_test.zig edits (contract entry + shape test + guard-b count). -Handler per ruling 8 (apply/respond split, house pattern). Unit tests for the -apply fn (both stores null; one store failing reload → error message in payload, -still 200). +Owns `src/web/handlers/certs.zig` (new), `src/web/routes.zig` (one entry), `src/web/openapi.yaml` (one path), `src/web/server.zig` (two null-defaulted WebState fields ONLY), `src/web/metrics.zig` (ruling 10 families), and the additive web_integration_test.zig edits (contract entry + shape test + guard-b count). Handler per ruling 8 (apply/respond split, house pattern). Unit tests for the apply fn (both stores null; one store failing reload → error message in payload, still 200). Acceptance: fmt/ast clean; plain suite 0 failed; drift guards green. ### T5 As built -handlers/certs.zig: apply/respond split — `applyReload(state, io) View` with -`View{doh, dot}` and `Outcome{enabled, reloaded, @"error": ?[]const u8}` (serialized as -`error`); `post` always answers 200. Route: `POST /api/certs/reload`, `.session`, -`.counted`; table 55→56; contract entry + strict shape test + drift guards green at 56. -metrics.zig: the two ruling-10 cert counters share one family pair -(`nxdns_cert_reloads_total` / `nxdns_cert_reload_failures_total`) with an -`endpoint="doh"|"dot"` label (upstream-label pattern); unwired stores omit the family; -`last_reload_unix` not exported. WebState gains `doh_certs`/`dot_certs: -?*cert_store.CertStore = null`. Deviation (accepted): deleted the m8 tripwire test in -`src/web/openapi.zig` ("the document does not promise what phase 9 owns") — it exists to -fire when this endpoint lands. Deferred to T6: `nxdns_doh_server_*` / `nxdns_dot_server_*` -listener families (T3/T4 types did not exist at T5 build time). +handlers/certs.zig: apply/respond split — `applyReload(state, io) View` with `View{doh, dot}` and `Outcome{enabled, reloaded, @"error": ?[]const u8}` (serialized as `error`); `post` always answers 200. Route: `POST /api/certs/reload`, `.session`, `.counted`; table 55→56; contract entry + strict shape test + drift guards green at 56. metrics.zig: the two ruling-10 cert counters share one family pair (`nxdns_cert_reloads_total` / `nxdns_cert_reload_failures_total`) with an `endpoint="doh"|"dot"` label (upstream-label pattern); unwired stores omit the family; `last_reload_unix` not exported. WebState gains `doh_certs`/`dot_certs: ?*cert_store.CertStore = null`. Deviation (accepted): deleted the m8 tripwire test in `src/web/openapi.zig` ("the document does not promise what phase 9 owns") — it exists to fire when this endpoint lands. Deferred to T6: `nxdns_doh_server_*` / `nxdns_dot_server_*` listener families (T3/T4 types did not exist at T5 build time). ## Session T6: app wiring + fixtures + smoke (after all) -Owns `src/app.zig`, `tests/fixtures/` (cert2/key2 + README + fixtures.zig), and -`src/tests.zig` is the ORCHESTRATOR's (report lines). Ruling 11 wiring; ruling 13 -fixture. Smoke (report transcript): boot with doh+dot enabled on unprivileged ports -with fixture certs; DoT query via a scripted TLS client (a tiny zig run or the -integration test binary is fine — no new tooling); DoH POST via curl --insecure ---http1.1 with a binary body; certs/reload via the API twice (success, then chmod the -key unreadable → error in payload, old cert still serving a new connection); SIGTERM 0. -Both suites + cross ReleaseSafe with the SPA dist. +Owns `src/app.zig`, `tests/fixtures/` (cert2/key2 + README + fixtures.zig), and `src/tests.zig` is the ORCHESTRATOR's (report lines). Ruling 11 wiring; ruling 13 fixture. Smoke (report transcript): boot with doh+dot enabled on unprivileged ports with fixture certs; DoT query via a scripted TLS client (a tiny zig run or the integration test binary is fine — no new tooling); DoH POST via curl --insecure --http1.1 with a binary body; certs/reload via the API twice (success, then chmod the key unreadable → error in payload, old cert still serving a new connection); SIGTERM 0. Both suites + cross ReleaseSafe with the SPA dist. Acceptance: full gates green; smoke transcript. ### T6 As built -app.zig realizes ruling 11 in-frame: `openCertStore` maps non-OOM `ReloadError` to -`error.BadCertificate` (exit 2, prints both paths + `cert_store.humanMessage`); -`bindDoh`/`bindDot` warn-and-continue per ruling 1; listeners + two `store.watch` loops -join the group; stores are declared before listeners so teardown runs listener-deinit → -store-deinit and the refs==0 assert holds. `dot_alpn` (["dot"]) lives in app.zig. -Deviation: `doh_server.serve` module entry is NOT used — it constructs the DohServer in -its own task frame, so WebState could never get the pointer the `nxdns_doh_server_*` -family needs; both listeners bind in app.zig's frame instead (tcp_server style). -`doh_server.serve` remains as unused pub API. **Superseded by milestone-18 ruling 1: the -unused `doh_server.serve` is deleted. Nothing ever called it, and a dead second -composition path over the listener core is exactly the divergence that milestone -collapsed. app.zig's in-frame bind is now the only way DoH comes up.** metrics.zig: `DohListenerSample` -(ruling-10 four + `bad_requests`); DoT renders `dot_server.StatsSnapshot` directly; -accept-side counters stay off the exposition; unwired listeners omit the families. -WebState gains `doh_listener`/`dot_listener` optional pointers. build.zig (out of -ownership, necessary): the exe now compiles `mbedtls_shim.c` with the threading macros — -first exe-side tls_server reference; mirrors the tests wiring. Fixtures: cert2/key2 -exported as `cert2_pem`/`key2_pem` (distinct fingerprint), currently unreferenced by -tests. Config: `model.TlsEndpoint` already existed; no model/validate/schema edits. -Smoke: DoT framed query answered from local records; DoH POST 200 -application/dns-message; reload success then key chmod 000 → `reloaded:false` with -"private key file is not readable" while the old cert keeps serving; SIGTERM 0; boot -with unreadable key exits 2. +app.zig realizes ruling 11 in-frame: `openCertStore` maps non-OOM `ReloadError` to `error.BadCertificate` (exit 2, prints both paths + `cert_store.humanMessage`); `bindDoh`/`bindDot` warn-and-continue per ruling 1; listeners + two `store.watch` loops join the group; stores are declared before listeners so teardown runs listener-deinit → store-deinit and the refs==0 assert holds. `dot_alpn` (["dot"]) lives in app.zig. Deviation: `doh_server.serve` module entry is NOT used — it constructs the DohServer in its own task frame, so WebState could never get the pointer the `nxdns_doh_server_*` family needs; both listeners bind in app.zig's frame instead (tcp_server style). `doh_server.serve` remains as unused pub API. **Superseded by milestone-18 ruling 1: the unused `doh_server.serve` is deleted. Nothing ever called it, and a dead second composition path over the listener core is exactly the divergence that milestone collapsed. app.zig's in-frame bind is now the only way DoH comes up.** metrics.zig: `DohListenerSample` (ruling-10 four + `bad_requests`); DoT renders `dot_server.StatsSnapshot` directly; accept-side counters stay off the exposition; unwired listeners omit the families. WebState gains `doh_listener`/`dot_listener` optional pointers. build.zig (out of ownership, necessary): the exe now compiles `mbedtls_shim.c` with the threading macros — first exe-side tls_server reference; mirrors the tests wiring. Fixtures: cert2/key2 exported as `cert2_pem`/`key2_pem` (distinct fingerprint), currently unreferenced by tests. Config: `model.TlsEndpoint` already existed; no model/validate/schema edits. Smoke: DoT framed query answered from local records; DoH POST 200 application/dns-message; reload success then key chmod 000 → `reloaded:false` with "private key file is not readable" while the old cert keeps serving; SIGTERM 0; boot with unreadable key exits 2. --- @@ -308,45 +101,21 @@ with unreadable key exits 2. Four rounds on one thread; round 4 returned "No findings." -Round 1 (3 important, 2 minor): doh_server leaked the TLS context when the handshake -won a race the Select reported as timeout/cancel → handshook flag (dot_server pattern). -cert_store reloads were not serialized across read→build→publish, so an older read could -publish last → `reload_mutex` held across the whole sequence (lock order: reload_mutex -before the entry mutex; accept path untouched) + a hook-pinned regression test -(`after_load_hook` seam). doh bodied-GET smuggling: std leaves unframed body bytes for -bodyless methods to be parsed as the next request → `framesUnreadableBody` guard, 400 + -close, no byte read. `.drop` now closes per ruling 2. cert2/key2 were unreferenced → -DoT reload-to-distinct-identity integration test (old connection keeps serving, new -handshake succeeds, distinct entry pointers; the 0.16 tls Client does not retain the -peer chain, so identity is proven via entry pointers). +Round 1 (3 important, 2 minor): doh_server leaked the TLS context when the handshake won a race the Select reported as timeout/cancel → handshook flag (dot_server pattern). cert_store reloads were not serialized across read→build→publish, so an older read could publish last → `reload_mutex` held across the whole sequence (lock order: reload_mutex before the entry mutex; accept path untouched) + a hook-pinned regression test (`after_load_hook` seam). doh bodied-GET smuggling: std leaves unframed body bytes for bodyless methods to be parsed as the next request → `framesUnreadableBody` guard, 400 + close, no byte read. `.drop` now closes per ruling 2. cert2/key2 were unreferenced → DoT reload-to-distinct-identity integration test (old connection keeps serving, new handshake succeeds, distinct entry pointers; the 0.16 tls Client does not retain the peer chain, so identity is proven via entry pointers). -Round 2 (2 minor): the smuggling guard ran before routing and stole 404/405 statuses → -routing decides the status, framed requests only force close. Successful POSTs ignored -the client's `Connection: close` → answer/refuse honor `head.keep_alive` in the Next -enum (the wire already did via Server.zig:628). +Round 2 (2 minor): the smuggling guard ran before routing and stole 404/405 statuses → routing decides the status, framed requests only force close. Successful POSTs ignored the client's `Connection: close` → answer/refuse honor `head.keep_alive` in the Next enum (the wire already did via Server.zig:628). -Round 3 (1 important, introduced by the round-2 reshape): bodied refusals (404/405/415) -kept keep-alive, so respond drained the whole body first — an unterminated chunked body -could pin all 64 slots with no response → `framesBody` helper; any pre-body-read refusal -closes when a body is framed; drain-free integration tests (missing terminal chunk → -404+EOF at once; undelivered content-length 4096 → 415+EOF); errorMatrix's 415 became a -zero-length wrong-type POST so its keep-alive reuse stays legitimate. +Round 3 (1 important, introduced by the round-2 reshape): bodied refusals (404/405/415) kept keep-alive, so respond drained the whole body first — an unterminated chunked body could pin all 64 slots with no response → `framesBody` helper; any pre-body-read refusal closes when a body is framed; drain-free integration tests (missing terminal chunk → 404+EOF at once; undelivered content-length 4096 → 415+EOF); errorMatrix's 415 became a zero-length wrong-type POST so its keep-alive reuse stays legitimate. -Final counts: doh_server.zig 19 tests, dot_server.zig 12, cert_store.zig 17. Final -gates: plain 1166/1279 passed, 113 skipped (integration-gated), 0 failed; integration -1275/1279, 4 skipped (live-network by design), 0 failed. +Final counts: doh_server.zig 19 tests, dot_server.zig 12, cert_store.zig 17. Final gates: plain 1166/1279 passed, 113 skipped (integration-gated), 0 failed; integration 1275/1279, 4 skipped (live-network by design), 0 failed. ## Module layout (new) -src/server/{doh_server,dot_server,cert_store}.zig, src/web/handlers/certs.zig, -tests/fixtures/{self_signed_cert2.pem,self_signed_key2.pem}. +src/server/{doh_server,dot_server,cert_store}.zig, src/web/handlers/certs.zig, tests/fixtures/{self_signed_cert2.pem,self_signed_key2.pem}. ## File ownership -T1 platform/tls_server.zig; T2 server/cert_store.zig; T3 server/dot_server.zig; -T4 server/doh_server.zig; T5 web/{handlers/certs.zig,routes.zig,openapi.yaml, -server.zig(fields),metrics.zig,web_integration_test.zig}; T6 app.zig + fixtures. -Orchestrator: src/tests.zig, spec. Parallel sessions never share a file. +T1 platform/tls_server.zig; T2 server/cert_store.zig; T3 server/dot_server.zig; T4 server/doh_server.zig; T5 web/{handlers/certs.zig,routes.zig,openapi.yaml, server.zig(fields),metrics.zig,web_integration_test.zig}; T6 app.zig + fixtures. Orchestrator: src/tests.zig, spec. Parallel sessions never share a file. ## Acceptance (milestone complete) @@ -362,7 +131,4 @@ Orchestrator: src/tests.zig, spec. Parallel sessions never share a file. ## Anti-requirements -No HTTP/2, no DoQ, no SNI/multi-cert, no client-cert auth, no session tickets, no -proxy-protocol/XFF handling, no cert generation (ACME etc.), no cache-control on DoH, -no handler.zig changes, no docs/api rendering (Phase 10), no config schema changes -(TlsEndpoint exists), no inotify. +No HTTP/2, no DoQ, no SNI/multi-cert, no client-cert auth, no session tickets, no proxy-protocol/XFF handling, no cert generation (ACME etc.), no cache-control on DoH, no handler.zig changes, no docs/api rendering (Phase 10), no config schema changes (TlsEndpoint exists), no inotify. diff --git a/specs/milestone-11.md b/specs/milestone-11.md index 49d523a..6b07885 100644 --- a/specs/milestone-11.md +++ b/specs/milestone-11.md @@ -1,255 +1,94 @@ # Milestone 11: packaging, ops and docs (PLAN Phase 10) -Goal: systemd unit, Dockerfile + compose, and the four docs (operator, architecture, -config-reference, API) — the documented deployment must work end-to-end; docs are -drift-guarded where a guard is cheap and honest. +Goal: systemd unit, Dockerfile + compose, and the four docs (operator, architecture, config-reference, API) — the documented deployment must work end-to-end; docs are drift-guarded where a guard is cheap and honest. ## Rulings (binding) -1. **Layout.** `deploy/systemd/nxdns.service` + `deploy/systemd/sysusers.conf`; - `deploy/docker/{Dockerfile,compose.yaml,.dockerignore}`; `docs/{operator.md, - architecture.md,config-reference.md,api.md}`; `README.md` at the root (the repo has - none; a portfolio repo needs a front door — short: what, why, quickstart, doc links). - PLAN.md:236 sketches docs/ subdirectories; single files need no subdirectories. +1. **Layout.** `deploy/systemd/nxdns.service` + `deploy/systemd/sysusers.conf`; `deploy/docker/{Dockerfile,compose.yaml,.dockerignore}`; `docs/{operator.md, architecture.md,config-reference.md,api.md}`; `README.md` at the root (the repo has none; a portfolio repo needs a front door — short: what, why, quickstart, doc links). PLAN.md:236 sketches docs/ subdirectories; single files need no subdirectories. -2. **API docs = hand-written `docs/api.md` + drift test.** No renderer is vendored - (redoc/scalar are exactly the dependency liability AGENTS.md refuses), and a - build-time YAML parser for rendering is scope the yaml does not justify — the yaml - itself is already served unauthenticated at `GET /api/openapi.yaml` (routes.zig:49) - and is the exhaustive contract. `docs/api.md` gives human-readable orientation: - auth model (cookie session, login flow), rate limiting, error envelope, SSE - semantics, then one line per operation (method, path, auth, one-sentence purpose) - and a pointer to the yaml for schemas. A drift test asserts every served route - appears textually in api.md (mirror of openapi.zig:34's guard). This satisfies - m8 ruling 3's deferred "docs/api rendering" as the engineering call: rendered = - readable, guarded, in-repo; not = a vendored JS bundle. +2. **API docs = hand-written `docs/api.md` + drift test.** No renderer is vendored (redoc/scalar are exactly the dependency liability AGENTS.md refuses), and a build-time YAML parser for rendering is scope the yaml does not justify — the yaml itself is already served unauthenticated at `GET /api/openapi.yaml` (routes.zig:49) and is the exhaustive contract. `docs/api.md` gives human-readable orientation: auth model (cookie session, login flow), rate limiting, error envelope, SSE semantics, then one line per operation (method, path, auth, one-sentence purpose) and a pointer to the yaml for schemas. A drift test asserts every served route appears textually in api.md (mirror of openapi.zig:34's guard). This satisfies m8 ruling 3's deferred "docs/api rendering" as the engineering call: rendered = readable, guarded, in-repo; not = a vendored JS bundle. -3. **Docs drift guards.** New `src/docs_drift_test.zig` (ORCHESTRATOR-owned, written - after the doc sessions land): (a) every route in `router.routes` appears in - docs/api.md; (b) every settings key from `model.toSettings` (the 44 keys) appears - in docs/config-reference.md; (c) every CLI subcommand name appears in - docs/operator.md. Docs embedded via a `docs_files` anonymous import added in - build.zig (test_fixtures pattern, build.zig:61). Guards are textual-containment - only — cheap, zero false authority. +3. **Docs drift guards.** New `src/docs_drift_test.zig` (ORCHESTRATOR-owned, written after the doc sessions land): (a) every route in `router.routes` appears in docs/api.md; (b) every settings key from `model.toSettings` (the 44 keys) appears in docs/config-reference.md; (c) every CLI subcommand name appears in docs/operator.md. Docs embedded via a `docs_files` anonymous import added in build.zig (test_fixtures pattern, build.zig:61). Guards are textual-containment only — cheap, zero false authority. -4. **systemd unit.** `Type=simple` (no forking, shutdown.zig:35 handles SIGTERM), - `ExecStart=/usr/local/bin/nxdns run`, stderr → journald (logging.zig:301 already - states this; `logging.output=stderr` stays the default). Static system user `nxdns` - via `deploy/systemd/sysusers.conf` (`u nxdns - "nxdns DNS sinkhole"`), NOT - DynamicUser — the TLS key must be chown-able to a stable uid ("TLS keys readable by - service user only", PLAN §19). `StateDirectory=nxdns` (0700 matches cli.zig:226), - `LogsDirectory=nxdns` (covers logging.output=file; the binary does not create the - directory, logging.zig:502), `ConfigurationDirectory=nxdns`. - `AmbientCapabilities=CAP_NET_BIND_SERVICE` + `CapabilityBoundingSet=` the same - (port 53; 443/853 covered by the same cap). Hardening: `NoNewPrivileges=yes`, - `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `PrivateDevices=yes`, - `ProtectKernelTunables/Modules/Logs=yes`, `ProtectControlGroups=yes`, - `ProtectClock=yes`, `ProtectHostname=yes`, `ProtectProc=invisible`, - `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `RestrictNamespaces=yes`, - `RestrictRealtime=yes`, `RestrictSUIDSGID=yes`, `LockPersonality=yes`, - `MemoryDenyWriteExecute=yes` (static Zig binary, no JIT), `UMask=0077`, - `SystemCallFilter=@system-service`, `SystemCallArchitectures=native`, - `Restart=on-failure`, `RestartSec=2`. No `ReadWritePaths` beyond what - StateDirectory/LogsDirectory grant. Validate with `systemd-analyze verify` if the - build host has it; report honestly if not. +4. **systemd unit.** `Type=simple` (no forking, shutdown.zig:35 handles SIGTERM), `ExecStart=/usr/local/bin/nxdns run`, stderr → journald (logging.zig:301 already states this; `logging.output=stderr` stays the default). Static system user `nxdns` via `deploy/systemd/sysusers.conf` (`u nxdns - "nxdns DNS sinkhole"`), NOT DynamicUser — the TLS key must be chown-able to a stable uid ("TLS keys readable by service user only", PLAN §19). `StateDirectory=nxdns` (0700 matches cli.zig:226), `LogsDirectory=nxdns` (covers logging.output=file; the binary does not create the directory, logging.zig:502), `ConfigurationDirectory=nxdns`. `AmbientCapabilities=CAP_NET_BIND_SERVICE` + `CapabilityBoundingSet=` the same (port 53; 443/853 covered by the same cap). Hardening: `NoNewPrivileges=yes`, `ProtectSystem=strict`, `ProtectHome=yes`, `PrivateTmp=yes`, `PrivateDevices=yes`, `ProtectKernelTunables/Modules/Logs=yes`, `ProtectControlGroups=yes`, `ProtectClock=yes`, `ProtectHostname=yes`, `ProtectProc=invisible`, `RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX`, `RestrictNamespaces=yes`, `RestrictRealtime=yes`, `RestrictSUIDSGID=yes`, `LockPersonality=yes`, `MemoryDenyWriteExecute=yes` (static Zig binary, no JIT), `UMask=0077`, `SystemCallFilter=@system-service`, `SystemCallArchitectures=native`, `Restart=on-failure`, `RestartSec=2`. No `ReadWritePaths` beyond what StateDirectory/LogsDirectory grant. Validate with `systemd-analyze verify` if the build host has it; report honestly if not. -5. **Docker.** Multi-stage: builder stage only stages `ca-certificates` (upstream - TLS verification rescans the system CA bundle, tls_client.zig:216 — a scratch - image without a bundle breaks every DoH/DoT upstream); final `FROM scratch` with - the static musl binary, `/etc/ssl/certs/ca-certificates.crt`, a nonroot numeric - `USER 65532:65532`, `VOLUME /var/lib/nxdns`, `EXPOSE 53/udp 53/tcp 8080 443 853`, - `ENTRYPOINT ["/nxdns"]`, `CMD ["run"]`. The binary is NOT built inside the - Dockerfile (the repo builds it with zig; the Dockerfile COPYes - `zig-out/cross/$TARGETARCH-…/nxdns` via a build arg or buildx TARGETARCH mapping — - keep it working for both arches). compose.yaml: ports 53:53/udp+tcp and 8080:8080 - (443/853 commented), bind-mount `./etc-nxdns:/etc/nxdns:ro`, named volume for - `/var/lib/nxdns`, `sysctls: net.ipv4.ip_unprivileged_port_start=0` so the nonroot - user binds 53 (per-netns sysctl; documented), `restart: unless-stopped`. First - boot needs a seeded `/etc/nxdns/config.zon` with a `default` group + one enabled - upstream or the container exits 2 (bootstrap.zig:38, app.zig:93) — operator.md and - a compose comment both say so. Do NOT point the host's resolv.conf at nxdns - itself for the container's own lookups. +5. **Docker.** Multi-stage: builder stage only stages `ca-certificates` (upstream TLS verification rescans the system CA bundle, tls_client.zig:216 — a scratch image without a bundle breaks every DoH/DoT upstream); final `FROM scratch` with the static musl binary, `/etc/ssl/certs/ca-certificates.crt`, a nonroot numeric `USER 65532:65532`, `VOLUME /var/lib/nxdns`, `EXPOSE 53/udp 53/tcp 8080 443 853`, `ENTRYPOINT ["/nxdns"]`, `CMD ["run"]`. The binary is NOT built inside the Dockerfile (the repo builds it with zig; the Dockerfile COPYes `zig-out/cross/$TARGETARCH-…/nxdns` via a build arg or buildx TARGETARCH mapping — keep it working for both arches). compose.yaml: ports 53:53/udp+tcp and 8080:8080 (443/853 commented), bind-mount `./etc-nxdns:/etc/nxdns:ro`, named volume for `/var/lib/nxdns`, `sysctls: net.ipv4.ip_unprivileged_port_start=0` so the nonroot user binds 53 (per-netns sysctl; documented), `restart: unless-stopped`. First boot needs a seeded `/etc/nxdns/config.zon` with a `default` group + one enabled upstream or the container exits 2 (bootstrap.zig:38, app.zig:93) — operator.md and a compose comment both say so. Do NOT point the host's resolv.conf at nxdns itself for the container's own lookups. -6. **CI.** One added job `docker` in ci.yml: after building the x86_64 exe - (ReleaseSafe, with the SPA dist like the cross job), `docker build` the image and - run a container smoke (seed a minimal config.zon; `nxdns version` + boot + one - `dig`-equivalent via the test client or `curl` on 8080/api/health; SIGTERM 0). - No registry push — no publish step exists anywhere in this repo or the infra - repo's CI, and registry credentials are an infra decision outside this repo. - Manual publishing to git.mial.net stays possible and is documented in operator.md - in one paragraph. +6. **CI.** One added job `docker` in ci.yml: after building the x86_64 exe (ReleaseSafe, with the SPA dist like the cross job), `docker build` the image and run a container smoke (seed a minimal config.zon; `nxdns version` + boot + one `dig`-equivalent via the test client or `curl` on 8080/api/health; SIGTERM 0). No registry push — no publish step exists anywhere in this repo or the infra repo's CI, and registry credentials are an infra decision outside this repo. Manual publishing to git.mial.net stays possible and is documented in operator.md in one paragraph. 7. **Docs content contracts.** - - operator.md: install (systemd path and docker path, both complete), first boot + - config.zon seeding semantics (file seeds DB once, DB is truth thereafter, - bootstrap.zig:38), auth setup (`web.password` hashed on import, never stored, - export writes ""), TLS cert/key provisioning + 0600 expectations + the reload - API/watcher, backup/restore = `nxdns export`/`import --force` (+ the 0600 export - mode and why), upgrades (schema migration = install + restart, PLAN §20.11), - data-dir layout table, exit codes (0/1/2/64, cli.zig:35), CLI reference (all six - subcommands + flags), troubleshooting (exit 2 causes, `nxdns check` semantics - incl. source-selection order cli.zig:511, disk-full degradation, port 53 - conflicts with systemd-resolved — include the disable recipe). - - architecture.md: module map (the src/ inventory), the purity rule (dns/, filter/, - local/, cache/ take bytes, no Io — AGENTS.md), std.Io injection + Threaded - backend, data flow for one query (listener → handler → filter/cache/local → - upstream → sink/logger), storage split (config.db truth / querylog.db expendable), - web stack (std.http over TLS optional, SPA embedded via web_assets, SSE), cert - hot-reload design (refcounted CertStore), failure-visibility doctrine (counters + - /metrics over log spam). Concise — a map, not a novel. - - config-reference.md: complete — every section/field with type, default, unit, - validation range, and which subsystem consumes it; collections with required - fields; DB-vs-file truth explanation; the `web.password`/`password_hash` - exclusivity; `logging.level=.err` serializes as "error" (model.zig:154). The - explorer inventory in this milestone's research is the skeleton; verify against - model.zig/validate.zig while writing, do not trust the summary blindly. + - operator.md: install (systemd path and docker path, both complete), first boot + config.zon seeding semantics (file seeds DB once, DB is truth thereafter, bootstrap.zig:38), auth setup (`web.password` hashed on import, never stored, export writes ""), TLS cert/key provisioning + 0600 expectations + the reload API/watcher, backup/restore = `nxdns export`/`import --force` (+ the 0600 export mode and why), upgrades (schema migration = install + restart, PLAN §20.11), data-dir layout table, exit codes (0/1/2/64, cli.zig:35), CLI reference (all six subcommands + flags), troubleshooting (exit 2 causes, `nxdns check` semantics incl. source-selection order cli.zig:511, disk-full degradation, port 53 conflicts with systemd-resolved — include the disable recipe). + - architecture.md: module map (the src/ inventory), the purity rule (dns/, filter/, local/, cache/ take bytes, no Io — AGENTS.md), std.Io injection + Threaded backend, data flow for one query (listener → handler → filter/cache/local → upstream → sink/logger), storage split (config.db truth / querylog.db expendable), web stack (std.http over TLS optional, SPA embedded via web_assets, SSE), cert hot-reload design (refcounted CertStore), failure-visibility doctrine (counters + /metrics over log spam). Concise — a map, not a novel. + - config-reference.md: complete — every section/field with type, default, unit, validation range, and which subsystem consumes it; collections with required fields; DB-vs-file truth explanation; the `web.password`/`password_hash` exclusivity; `logging.level=.err` serializes as "error" (model.zig:154). The explorer inventory in this milestone's research is the skeleton; verify against model.zig/validate.zig while writing, do not trust the summary blindly. - api.md: per ruling 2. - - README.md: ≤120 lines; what nxdns is, feature list (honest, shipping features - only), quickstart (docker compose path), build-from-source (zig build, node for - the SPA), doc links, license note if a LICENSE exists (do not invent one). + - README.md: ≤120 lines; what nxdns is, feature list (honest, shipping features only), quickstart (docker compose path), build-from-source (zig build, node for the SPA), doc links, license note if a LICENSE exists (do not invent one). -8. **No new runtime code.** This milestone adds zero behavior to the binary. The only - src/ change is the orchestrator's docs_drift_test.zig + its build wiring. If a doc - session finds a bug while documenting, it REPORTS it (no fix); the orchestrator - triages. +8. **No new runtime code.** This milestone adds zero behavior to the binary. The only src/ change is the orchestrator's docs_drift_test.zig + its build wiring. If a doc session finds a bug while documenting, it REPORTS it (no fix); the orchestrator triages. -9. **Pi 5 end-to-end**: the exit criterion runs on hardware this environment does not - have. The deliverable here is: both suites green, docker smoke green on x86_64, - `systemd-analyze verify` clean (or honestly reported unavailable), aarch64 binary - built and statically verified (existing cross job). The operator doc's Pi 5 recipe - is written to be executed by the user; the spec records this boundary explicitly. +9. **Pi 5 end-to-end**: the exit criterion runs on hardware this environment does not have. The deliverable here is: both suites green, docker smoke green on x86_64, `systemd-analyze verify` clean (or honestly reported unavailable), aarch64 binary built and statically verified (existing cross job). The operator doc's Pi 5 recipe is written to be executed by the user; the spec records this boundary explicitly. ## Sessions -U1, U3, U4, U5 parallel; U2 after U1 (documents the artifacts U1 produces); -orchestrator wiring (ruling 3) after U3+U5. +U1, U3, U4, U5 parallel; U2 after U1 (documents the artifacts U1 produces); orchestrator wiring (ruling 3) after U3+U5. ## Session U1: deploy artifacts + CI -Owns `deploy/systemd/nxdns.service`, `deploy/systemd/sysusers.conf`, -`deploy/docker/{Dockerfile,compose.yaml,.dockerignore}`, `.gitea/workflows/ci.yml` -(one added job). Rulings 4, 5, 6. Verify: `systemd-analyze verify` (or report -unavailable), local `docker build` + container smoke if the docker daemon is -reachable (report honestly either way), `zig build test` untouched-green. +Owns `deploy/systemd/nxdns.service`, `deploy/systemd/sysusers.conf`, `deploy/docker/{Dockerfile,compose.yaml,.dockerignore}`, `.gitea/workflows/ci.yml` (one added job). Rulings 4, 5, 6. Verify: `systemd-analyze verify` (or report unavailable), local `docker build` + container smoke if the docker daemon is reachable (report honestly either way), `zig build test` untouched-green. ## Session U2: operator.md + README.md (after U1) -Owns `docs/operator.md`, `README.md`. Rulings 7 (operator + README). Reads U1's -artifacts and the runtime/CLI facts from the code (verify against src/cli.zig, -src/app.zig, src/config/bootstrap.zig — not from memory). +Owns `docs/operator.md`, `README.md`. Rulings 7 (operator + README). Reads U1's artifacts and the runtime/CLI facts from the code (verify against src/cli.zig, src/app.zig, src/config/bootstrap.zig — not from memory). ## Session U3: config-reference.md -Owns `docs/config-reference.md`. Ruling 7. Source of truth: src/config/model.zig + -validate.zig + import/export/bootstrap. Every field, no sampling. +Owns `docs/config-reference.md`. Ruling 7. Source of truth: src/config/model.zig + validate.zig + import/export/bootstrap. Every field, no sampling. ## Session U4: architecture.md -Owns `docs/architecture.md`. Ruling 7. Reads module headers; no deep dives needed -beyond what the doc claims. +Owns `docs/architecture.md`. Ruling 7. Reads module headers; no deep dives needed beyond what the doc claims. ## Session U5: api.md -Owns `docs/api.md`. Rulings 2, 7. Source of truth: src/web/routes.zig (the served -table: method, path, auth, limiter) + openapi.yaml summaries + auth.zig/sse.zig for -the auth and SSE prose. Every route, no sampling. +Owns `docs/api.md`. Rulings 2, 7. Source of truth: src/web/routes.zig (the served table: method, path, auth, limiter) + openapi.yaml summaries + auth.zig/sse.zig for the auth and SSE prose. Every route, no sampling. ## As built -**U1** delivered per rulings 4-6 with accepted deviations: `StateDirectoryMode=0700` -(systemd defaults 0755; the binary cannot tighten a pre-existing directory) and -`User=nxdns`/`Group=nxdns` added to the unit; the Dockerfile's builder stage also maps -buildx TARGETARCH → cross-target dir and pre-chowns `/var/lib/nxdns` to 65532 (a named -volume seeded from a root-owned image dir would be unwritable on first boot); compose -gained a `build:` block; `deploy/docker/.dockerignore` is documentation-grade under -BuildKit (only a root `.dockerignore` or `Dockerfile.dockerignore` is honored — the -file's header says so). Verified: `systemd-analyze verify` clean modulo the off-host -ExecStart path (an ExecStart=/bin/true copy verifies exit 0); full local docker build + -smoke passed (binds 53 as uid 65532, /api/health ok, SIGTERM exit 0). The CI docker -job probes both 127.0.0.1 and the container IP to survive either runner topology. -Compose expects the operator-created seed at `deploy/docker/etc-nxdns/config.zon` -(minimal: a `default` group + one enabled upstream), else exit 2. +**U1** delivered per rulings 4-6 with accepted deviations: `StateDirectoryMode=0700` (systemd defaults 0755; the binary cannot tighten a pre-existing directory) and `User=nxdns`/`Group=nxdns` added to the unit; the Dockerfile's builder stage also maps buildx TARGETARCH → cross-target dir and pre-chowns `/var/lib/nxdns` to 65532 (a named volume seeded from a root-owned image dir would be unwritable on first boot); compose gained a `build:` block; `deploy/docker/.dockerignore` is documentation-grade under BuildKit (only a root `.dockerignore` or `Dockerfile.dockerignore` is honored — the file's header says so). Verified: `systemd-analyze verify` clean modulo the off-host ExecStart path (an ExecStart=/bin/true copy verifies exit 0); full local docker build + smoke passed (binds 53 as uid 65532, /api/health ok, SIGTERM exit 0). The CI docker job probes both 127.0.0.1 and the container IP to survive either runner topology. Compose expects the operator-created seed at `deploy/docker/etc-nxdns/config.zon` (minimal: a `default` group + one enabled upstream), else exit 2. -**U2** delivered docs/operator.md (397 lines; systemd + docker + Pi 5 recipes, seeding -semantics, auth, TLS, backup/restore, data-dir table, full CLI reference, exit codes, -troubleshooting incl. the systemd-resolved DNSStubListener recipe) and README.md -(72 lines, no license section — no LICENSE exists). All facts source-verified. +**U2** delivered docs/operator.md (397 lines; systemd + docker + Pi 5 recipes, seeding semantics, auth, TLS, backup/restore, data-dir table, full CLI reference, exit codes, troubleshooting incl. the systemd-resolved DNSStubListener recipe) and README.md (72 lines, no license section — no LICENSE exists). All facts source-verified. -**U3** delivered docs/config-reference.md (12 scalar sections, 9 collections, DB-vs-file -truth model, auth section, minimal + annotated examples) and surfaced five code -discrepancies during writing (see fix wave below). +**U3** delivered docs/config-reference.md (12 scalar sections, 9 collections, DB-vs-file truth model, auth section, minimal + annotated examples) and surfaced five code discrepancies during writing (see fix wave below). -**U4** delivered docs/architecture.md (module map from the //! headers, purity rule -with the honest exceptions, life-of-one-query pipeline verified against handler.zig, -storage split, web stack, CertStore design, failure-visibility doctrine). Reported one -stale comment (logging.zig:17 cited a moved cli.zig line) — orchestrator fixed the -comment to cite start.zig:724 via std.process.Init. +**U4** delivered docs/architecture.md (module map from the //! headers, purity rule with the honest exceptions, life-of-one-query pipeline verified against handler.zig, storage split, web stack, CertStore design, failure-visibility doctrine). Reported one stale comment (logging.zig:17 cited a moved cli.zig line) — orchestrator fixed the comment to cite start.zig:724 via std.process.Init. -**U5** delivered docs/api.md: all 56 operations (matches router.routes.len), auth / -rate-limit / SSE prose, error envelope, openapi.yaml pointer. No code-vs-yaml -discrepancies found. +**U5** delivered docs/api.md: all 56 operations (matches router.routes.len), auth / rate-limit / SSE prose, error envelope, openapi.yaml pointer. No code-vs-yaml discrepancies found. -**Orchestrator wiring (ruling 3)**: docs/docs.zig (embeds api.md, config-reference.md, -operator.md), `docs_files` anonymous import on the test module in build.zig, -src/docs_drift_test.zig with three containment guards (routes → api.md; toSettings -keys → config-reference.md; the six subcommand names → operator.md), tests.zig import. +**Orchestrator wiring (ruling 3)**: docs/docs.zig (embeds api.md, config-reference.md, operator.md), `docs_files` anonymous import on the test module in build.zig, src/docs_drift_test.zig with three containment guards (routes → api.md; toSettings keys → config-reference.md; the six subcommand names → operator.md), tests.zig import. -**Fix wave (orchestrator-triaged; ruling 8's no-runtime-code rule lifted for exactly -these)** — U3's five discrepancies, triaged with stdlib evidence: -1. `runtime.io_backend` DELETED end to end (model, settings handler + view, openapi, - web types/SettingsPage/settingsDiff + tests, docs). Nothing consumed it — main uses - init.io (stdlib Threaded, start.zig:724), and 0.16's std.Io.Evented has stubbed - networking (Uring.zig netConnectIp → error.NetworkDown), so PLAN decision E's - "io_uring via flag" is not deliverable at this tag. Re-add when std ships working - evented net. Old DB rows warn-and-ignore via fromSettings' unknown-key path. -2. `upstream.connect_timeout_ms` DELETED. No call site; the pool races the whole - attempt against total_timeout (app.zig sets it from totalTimeout); Threaded panics - on IpAddress.ConnectOptions.timeout != .none; std.http.Client has no knob. The - validate cross-check is now total >= read only. -3. `cache.size` KEPT: 0 is clean documented disabled behavior (put short-circuits; - in-file test "a cache of zero entries stores nothing"). Doc row corrected. -4. `dns.bind_ipv6` TIGHTENED: checkBind generalized to a BindFamily enum; dns.bind_ipv6 - requires an IPv6 literal (an IPv4 wildcard there made the v4 bind AddressInUse get - swallowed with a false "dual-stack" log — silent IPv6 loss). web/TLS binds stay .any. -5. doh/dot `readTimeout(.{})` sites KEPT: they are test fixtures; the real idle budget - is the ruled 10s Options default. The doc claim was wrong and was removed. -Settings key counts after deletion: toSettings emits 43 (incl. web.password_hash); -the API-visible restart-required set is 42 (was 44). +**Fix wave (orchestrator-triaged; ruling 8's no-runtime-code rule lifted for exactly these)** — U3's five discrepancies, triaged with stdlib evidence: +1. `runtime.io_backend` DELETED end to end (model, settings handler + view, openapi, web types/SettingsPage/settingsDiff + tests, docs). Nothing consumed it — main uses init.io (stdlib Threaded, start.zig:724), and 0.16's std.Io.Evented has stubbed networking (Uring.zig netConnectIp → error.NetworkDown), so PLAN decision E's "io_uring via flag" is not deliverable at this tag. Re-add when std ships working evented net. Old DB rows warn-and-ignore via fromSettings' unknown-key path. +2. `upstream.connect_timeout_ms` DELETED. No call site; the pool races the whole attempt against total_timeout (app.zig sets it from totalTimeout); Threaded panics on IpAddress.ConnectOptions.timeout != .none; std.http.Client has no knob. The validate cross-check is now total >= read only. +3. `cache.size` KEPT: 0 is clean documented disabled behavior (put short-circuits; in-file test "a cache of zero entries stores nothing"). Doc row corrected. +4. `dns.bind_ipv6` TIGHTENED: checkBind generalized to a BindFamily enum; dns.bind_ipv6 requires an IPv6 literal (an IPv4 wildcard there made the v4 bind AddressInUse get swallowed with a false "dual-stack" log — silent IPv6 loss). web/TLS binds stay .any. +5. doh/dot `readTimeout(.{})` sites KEPT: they are test fixtures; the real idle budget is the ruled 10s Options default. The doc claim was wrong and was removed. Settings key counts after deletion: toSettings emits 43 (incl. web.password_hash); the API-visible restart-required set is 42 (was 44). ## Review (Codex, as built) Three rounds on one thread; round 3 returned "No findings." -Round 1 (5 important): stale-DB bind_ipv6 rows bypassed the new validate check at boot -→ app.zig parseBind now enforces the IP family on both dns binds (BadBindAddress, -exit 2, remedy in the message; the v4 side had the symmetric hole); app.zig's tests -were not collected by tests.zig at all — the import line was added and the new test -runs. operator.md's TLS recipe assumed the nxdns host user → Docker path now chowns -65532:65532 numerically host-side (read-only bind mount). PLAN.md still promised the -io_uring flag and connect_timeout_ms → synced (Io bullet records the drop with stdlib -evidence; decision E row; example config). Drift guards were maskable → api.md guard -anchors the full "| METHOD | `pattern` |" row per operation; operator.md guard anchors -the "### `name" reference headings. +Round 1 (5 important): stale-DB bind_ipv6 rows bypassed the new validate check at boot → app.zig parseBind now enforces the IP family on both dns binds (BadBindAddress, exit 2, remedy in the message; the v4 side had the symmetric hole); app.zig's tests were not collected by tests.zig at all — the import line was added and the new test runs. operator.md's TLS recipe assumed the nxdns host user → Docker path now chowns 65532:65532 numerically host-side (read-only bind mount). PLAN.md still promised the io_uring flag and connect_timeout_ms → synced (Io bullet records the drop with stdlib evidence; decision E row; example config). Drift guards were maskable → api.md guard anchors the full "| METHOD | `pattern` |" row per operation; operator.md guard anchors the "### `name" reference headings. -Round 2 (1 important, 1 minor): seed-permission guidance covered only web.password → -now web.password or web.password_hash (a restored export); PLAN.md's "stubs all -networking" overstated 0.16's Uring — now names the stubbed operations precisely. +Round 2 (1 important, 1 minor): seed-permission guidance covered only web.password → now web.password or web.password_hash (a restored export); PLAN.md's "stubs all networking" overstated 0.16's Uring — now names the stubbed operations precisely. -Final gates: plain 1171/1284 passed, 113 skipped (integration-gated), 0 failed; -integration 1280/1284, 4 skipped (live-network by design), 0 failed; cross ReleaseSafe -with the SPA dist 18/18; web suite 121/121 with format/lint/typecheck clean. +Final gates: plain 1171/1284 passed, 113 skipped (integration-gated), 0 failed; integration 1280/1284, 4 skipped (live-network by design), 0 failed; cross ReleaseSafe with the SPA dist 18/18; web suite 121/121 with format/lint/typecheck clean. ## Module layout (new) -deploy/systemd/{nxdns.service,sysusers.conf}, deploy/docker/{Dockerfile,compose.yaml, -.dockerignore}, docs/{operator,architecture,config-reference,api}.md, README.md, -src/docs_drift_test.zig (orchestrator). +deploy/systemd/{nxdns.service,sysusers.conf}, deploy/docker/{Dockerfile,compose.yaml, .dockerignore}, docs/{operator,architecture,config-reference,api}.md, README.md, src/docs_drift_test.zig (orchestrator). ## File ownership -U1 deploy/* + ci.yml; U2 docs/operator.md + README.md; U3 docs/config-reference.md; -U4 docs/architecture.md; U5 docs/api.md; orchestrator src/docs_drift_test.zig, -build.zig (docs_files module), src/tests.zig. +U1 deploy/* + ci.yml; U2 docs/operator.md + README.md; U3 docs/config-reference.md; U4 docs/architecture.md; U5 docs/api.md; orchestrator src/docs_drift_test.zig, build.zig (docs_files module), src/tests.zig. ## Acceptance (milestone complete) @@ -268,6 +107,5 @@ build.zig (docs_files module), src/tests.zig. - No vendored API-doc renderer (redoc/scalar/swagger-ui), no YAML parser. - No registry publish step; no k3s manifests (the infra repo owns deployment there). -- No SIGHUP/reload feature, no env-var config, no new CLI flags — document what - exists; report gaps instead of filling them. +- No SIGHUP/reload feature, no env-var config, no new CLI flags — document what exists; report gaps instead of filling them. - No LICENSE invention; no badges or marketing prose in README. diff --git a/specs/milestone-12.md b/specs/milestone-12.md index 3576280..7946229 100644 --- a/specs/milestone-12.md +++ b/specs/milestone-12.md @@ -1,83 +1,31 @@ # Milestone 12: performance measurement pass + aarch64 test execution -Goal: close the two gaps the post-Phase-10 completeness sweep found — PLAN §18 has no -measurement infrastructure (deferred at specs/milestone-7.md:113 and never delivered), -and aarch64 tests are cross-built but never executed (PLAN.md:145 promised qemu). +Goal: close the two gaps the post-Phase-10 completeness sweep found — PLAN §18 has no measurement infrastructure (deferred at specs/milestone-7.md:113 and never delivered), and aarch64 tests are cross-built but never executed (PLAN.md:145 promised qemu). ## Rulings (binding) -1. **Bench harness = `tools/bench.zig` + `zig build bench`.** Not in src/ — src/ is - the shipped product; tests.zig aggregates everything shippable and the bench must - be a separate compilation anyway (one-file-one-module rule; matcher.zig and - dns_cache.zig already belong to the test compilation). Module wiring per the fuzz - pattern (build.zig:68-92): modules rooted at src/filter/matcher.zig and - src/cache/dns_cache.zig — their relative import closures come along; no - sqlite/mbedTLS linking needed. `if (b.args) |args| run.addArgs(args)`. +1. **Bench harness = `tools/bench.zig` + `zig build bench`.** Not in src/ — src/ is the shipped product; tests.zig aggregates everything shippable and the bench must be a separate compilation anyway (one-file-one-module rule; matcher.zig and dns_cache.zig already belong to the test compilation). Module wiring per the fuzz pattern (build.zig:68-92): modules rooted at src/filter/matcher.zig and src/cache/dns_cache.zig — their relative import closures come along; no sqlite/mbedTLS linking needed. `if (b.args) |args| run.addArgs(args)`. 2. **What it measures** (PLAN §18 targets): - - `filter`: `matcher.normalize` + `Snapshot.evaluate` per op against a Snapshot - built from a generated, sorted `d{d:0>7}.example.com` list body (~1M lines, - in-memory; DomainSet.max_count 4M and compiler.max_domains 2M leave headroom) - plus a small wild body. Mixed case ratios: hit, miss, parent-walk. Target - p95 < 1 ms. - - `cache`: `buildKey` + `DnsCache.get` + `packet.setId` (the handler's hit path; - TTL aging happens inside get) on a ~10k-entry cache prefilled with the fuzz - corpus response. Mixed hit/miss. Target p95 < 5 ms. - - `compile`: `compiler.compile` over a generated 1M-line hosts body — wall time, - informational (no §18 target; no datapoint exists today). - - Memory: `/proc/self/status` VmRSS (nothing in-repo wraps it; read it directly) - plus the in-repo accounting (`Snapshot.memoryBytes`, `DnsCache.memoryBytes`). - Target < 100 MB RSS with ~1M domains loaded. - - Timing: `std.Io.Clock.awake.now(io)` / `durationTo` — std.time.Timer does not - exist at 0.16. Percentiles: collect per-op nanos in a preallocated []u64, sort - (std.mem.sort), report p50/p95/p99/max. No percentile helper exists; write it - in the tool. + - `filter`: `matcher.normalize` + `Snapshot.evaluate` per op against a Snapshot built from a generated, sorted `d{d:0>7}.example.com` list body (~1M lines, in-memory; DomainSet.max_count 4M and compiler.max_domains 2M leave headroom) plus a small wild body. Mixed case ratios: hit, miss, parent-walk. Target p95 < 1 ms. + - `cache`: `buildKey` + `DnsCache.get` + `packet.setId` (the handler's hit path; TTL aging happens inside get) on a ~10k-entry cache prefilled with the fuzz corpus response. Mixed hit/miss. Target p95 < 5 ms. + - `compile`: `compiler.compile` over a generated 1M-line hosts body — wall time, informational (no §18 target; no datapoint exists today). + - Memory: `/proc/self/status` VmRSS (nothing in-repo wraps it; read it directly) plus the in-repo accounting (`Snapshot.memoryBytes`, `DnsCache.memoryBytes`). Target < 100 MB RSS with ~1M domains loaded. + - Timing: `std.Io.Clock.awake.now(io)` / `durationTo` — std.time.Timer does not exist at 0.16. Percentiles: collect per-op nanos in a preallocated []u64, sort (std.mem.sort), report p50/p95/p99/max. No percentile helper exists; write it in the tool. -3. **Assertion policy.** Default run is informational (prints a table). `--assert` - exits non-zero when a §18 target is exceeded — for the Pi 5 run, NOT for CI: - required CI stays deterministic (AGENTS.md) and perf assertions on shared runners - are flake generators. CI does not run the bench at all this milestone. +3. **Assertion policy.** Default run is informational (prints a table). `--assert` exits non-zero when a §18 target is exceeded — for the Pi 5 run, NOT for CI: required CI stays deterministic (AGENTS.md) and perf assertions on shared runners are flake generators. CI does not run the bench at all this milestone. -4. **Bench flags**: `--domains=N` (default 1_000_000), `--iters=N` (default 200_000), - `--seed=N` (default fixed), `--assert`, subcommands `filter|cache|compile|all` - (default all). Debug-mode runs print a one-line warning recommending - `-Doptimize=ReleaseFast`. +4. **Bench flags**: `--domains=N` (default 1_000_000), `--iters=N` (default 200_000), `--seed=N` (default fixed), `--assert`, subcommands `filter|cache|compile|all` (default all). Debug-mode runs print a one-line warning recommending `-Doptimize=ReleaseFast`. -5. **docs/performance.md**: the §18 targets table; measured numbers from THIS x86_64 - host (dated, hardware named honestly, marked as not-the-target-platform); the - in-memory accounting numbers; the one-command Pi 5 recipe - (`zig build bench -Doptimize=ReleaseFast -- --assert`); a sentence on why CI does - not gate on perf. The Pi 5 rows stay "to be measured on hardware". +5. **docs/performance.md**: the §18 targets table; measured numbers from THIS x86_64 host (dated, hardware named honestly, marked as not-the-target-platform); the in-memory accounting numbers; the one-command Pi 5 recipe (`zig build bench -Doptimize=ReleaseFast -- --assert`); a sentence on why CI does not gate on perf. The Pi 5 rows stay "to be measured on hardware". -6. **aarch64 execution = second test artifact, `zig build test-aarch64 -fqemu`.** - Per stdlib evidence: Run steps try qemu only when enable_qemu (the `-fqemu` CLI - flag; a *Build field, no per-step override) and exec `qemu-aarch64` bare from - PATH — so CI installs `qemu-user` (NOT qemu-user-static, which ships the - `-static` name Zig will not find; binfmt is unnecessary). Static musl means no - sysroot/--libc-runtimes. build.zig: resolve aarch64-linux-musl, addTest with the - same wiring as the native tests artifact (sqlite/mbedtls helpers are already - target-parameterized), set `.linkage = .static` on the artifact (TestOptions has - no linkage field), `run.skip_foreign_checks = true`, leave - failing_to_execute_foreign_is_an_error true so a missing qemu is loud. Fuzz - artifacts excluded. +6. **aarch64 execution = second test artifact, `zig build test-aarch64 -fqemu`.** Per stdlib evidence: Run steps try qemu only when enable_qemu (the `-fqemu` CLI flag; a *Build field, no per-step override) and exec `qemu-aarch64` bare from PATH — so CI installs `qemu-user` (NOT qemu-user-static, which ships the `-static` name Zig will not find; binfmt is unnecessary). Static musl means no sysroot/--libc-runtimes. build.zig: resolve aarch64-linux-musl, addTest with the same wiring as the native tests artifact (sqlite/mbedtls helpers are already target-parameterized), set `.linkage = .static` on the artifact (TestOptions has no linkage field), `run.skip_foreign_checks = true`, leave failing_to_execute_foreign_is_an_error true so a missing qemu is loud. Fuzz artifacts excluded. -7. **qemu scope: plain suite only, blocking.** No `-Dintegration` under qemu: the - integration tests are multithreaded loopback TLS with wall-clock budgets, and - qemu-user's 5-20x slowdown makes them a flake source — required CI stays - deterministic. The plain suite (the aggregator's tests without the 12 - fuzz-artifact tests: 1159 pass + 113 integration-gated skips; all pure - DNS/filter/cache logic included) - is the portable-correctness signal aarch64 needs. Recorded here as the - engineering call closing PLAN.md:145's promise. +7. **qemu scope: plain suite only, blocking.** No `-Dintegration` under qemu: the integration tests are multithreaded loopback TLS with wall-clock budgets, and qemu-user's 5-20x slowdown makes them a flake source — required CI stays deterministic. The plain suite (the aggregator's tests without the 12 fuzz-artifact tests: 1159 pass + 113 integration-gated skips; all pure DNS/filter/cache logic included) is the portable-correctness signal aarch64 needs. Recorded here as the engineering call closing PLAN.md:145's promise. -8. **CI additions**: one `test-aarch64` job (setup-zig, apt qemu-user, - `zig build test-aarch64 -fqemu`); plus the missing §18 size assert — the cross - job gains a second build WITHOUT `-Dweb-dist` (placeholder dist) and asserts the - stripped copies < 10 MiB per arch (the < 15 MiB with-assets assert already - exists). Job graph otherwise untouched. +8. **CI additions**: one `test-aarch64` job (setup-zig, apt qemu-user, `zig build test-aarch64 -fqemu`); plus the missing §18 size assert — the cross job gains a second build WITHOUT `-Dweb-dist` (placeholder dist) and asserts the stripped copies < 10 MiB per arch (the < 15 MiB with-assets assert already exists). Job graph otherwise untouched. -9. **No src/ changes.** matcher/dns_cache/compiler public APIs are used as-is; if - the bench needs something they do not expose, report — do not extend them. +9. **No src/ changes.** matcher/dns_cache/compiler public APIs are used as-is; if the bench needs something they do not expose, report — do not extend them. ## Sessions @@ -85,70 +33,29 @@ V1 then V2 (both edit build.zig — sequenced, not parallel). ## Session V1: bench harness + performance doc -Owns tools/bench.zig, build.zig (the bench step only), docs/performance.md. -Rulings 1-5. Runs the bench on this host (ReleaseFast) and writes the measured -numbers into the doc. +Owns tools/bench.zig, build.zig (the bench step only), docs/performance.md. Rulings 1-5. Runs the bench on this host (ReleaseFast) and writes the measured numbers into the doc. ### V1 As built -Delivered tools/bench.zig, the build.zig bench step, and docs/performance.md. One -deviation from ruling 1's wiring: Zig 0.16 rejects separate modules rooted at -matcher.zig and dns_cache.zig ("file exists in multiple modules" — their closures -share model.zig/types.zig; verified with a minimal repro). Instead a b.addWriteFiles -stage copies src/ plus a generated 7-line aggregator root (bench_core.zig) into one -`core` module — the same staging trick the web assets use; a build.zig comment -records why. The fuzz-corpus response is a byte-for-byte copy with provenance -comment (importing corpus.zig recreates the module conflict; the corpus documents -copies as house style). /proc/self/status reads use readerStreaming (procfs stats -size 0, readFileAlloc returns empty). Measured on the build host (i7-14700K, -ReleaseFast, defaults: 1M domains, 200k iters): filter p95 0.18 µs (target < 1 ms), -cache p95 0.14 µs (target < 5 ms), VmRSS 31.8 MiB with the 1M-domain snapshot -(target < 100 MiB), Snapshot.memoryBytes 28.0 MiB, compile 96 ms ≈ 10.4M lines/s -(informational). --assert exit paths verified both ways (a 4M-domain run exceeds -the 1M-scoped RSS target and exits 1; that was an exit-path exercise, not a target -miss). No src/ changes; matcher/dns_cache/compiler APIs sufficed. Cache suite fixes -entries at 10k (the handler-default Config.size); --domains shapes filter and -compile only. Both suites 0 failed after the build.zig edit. +Delivered tools/bench.zig, the build.zig bench step, and docs/performance.md. One deviation from ruling 1's wiring: Zig 0.16 rejects separate modules rooted at matcher.zig and dns_cache.zig ("file exists in multiple modules" — their closures share model.zig/types.zig; verified with a minimal repro). Instead a b.addWriteFiles stage copies src/ plus a generated 7-line aggregator root (bench_core.zig) into one `core` module — the same staging trick the web assets use; a build.zig comment records why. The fuzz-corpus response is a byte-for-byte copy with provenance comment (importing corpus.zig recreates the module conflict; the corpus documents copies as house style). /proc/self/status reads use readerStreaming (procfs stats size 0, readFileAlloc returns empty). Measured on the build host (i7-14700K, ReleaseFast, defaults: 1M domains, 200k iters): filter p95 0.18 µs (target < 1 ms), cache p95 0.14 µs (target < 5 ms), VmRSS 31.8 MiB with the 1M-domain snapshot (target < 100 MiB), Snapshot.memoryBytes 28.0 MiB, compile 96 ms ≈ 10.4M lines/s (informational). --assert exit paths verified both ways (a 4M-domain run exceeds the 1M-scoped RSS target and exits 1; that was an exit-path exercise, not a target miss). No src/ changes; matcher/dns_cache/compiler APIs sufficed. Cache suite fixes entries at 10k (the handler-default Config.size); --domains shapes filter and compile only. Both suites 0 failed after the build.zig edit. ## Session V2: aarch64 test execution + CI (after V1) -Owns build.zig (the test-aarch64 artifact/step only), .gitea/workflows/ci.yml -(the new job + the cross-job size-assert addition). Rulings 6-8. Runs -`zig build test-aarch64 -fqemu` locally if qemu-aarch64 is installable/present; -reports honestly if not. +Owns build.zig (the test-aarch64 artifact/step only), .gitea/workflows/ci.yml (the new job + the cross-job size-assert addition). Rulings 6-8. Runs `zig build test-aarch64 -fqemu` locally if qemu-aarch64 is installable/present; reports honestly if not. ### V2 As built -build.zig gained the test-aarch64 block after the bench block: aarch64-linux-musl -addTest with wiring identical to the native tests artifact, `.linkage = .static` on -the artifact, `run.skip_foreign_checks = true`, failing-to-execute left loud. Stdlib -mechanics re-verified (Run.zig:71/78/226, Build.zig:73, system.zig:111 — bare -`qemu-aarch64` from PATH). ci.yml gained the test-aarch64 job (apt qemu-user, -`zig build test-aarch64 -fqemu`) and the cross job builds a second no-dist pair to -`--prefix zig-out/nodist` (existing with-assets steps byte-identical) with a +build.zig gained the test-aarch64 block after the bench block: aarch64-linux-musl addTest with wiring identical to the native tests artifact, `.linkage = .static` on the artifact, `run.skip_foreign_checks = true`, failing-to-execute left loud. Stdlib mechanics re-verified (Run.zig:71/78/226, Build.zig:73, system.zig:111 — bare `qemu-aarch64` from PATH). ci.yml gained the test-aarch64 job (apt qemu-user, `zig build test-aarch64 -fqemu`) and the cross job builds a second no-dist pair to `--prefix zig-out/nodist` (existing with-assets steps byte-identical) with a < 10 MiB stripped assert; the new assert skips the static-linkage recheck (proven -on the same-config with-assets build). Local verification: real qemu execution — -1159 pass / 113 skip / 0 failed (native count minus the 12 excluded fuzz tests); -no-dist stripped sizes 5,880,336 (x86_64) and 5,212,640 (aarch64) bytes. No --Dintegration under qemu per ruling 7. +on the same-config with-assets build). Local verification: real qemu execution — 1159 pass / 113 skip / 0 failed (native count minus the 12 excluded fuzz tests); no-dist stripped sizes 5,880,336 (x86_64) and 5,212,640 (aarch64) bytes. No -Dintegration under qemu per ruling 7. ## Review (Codex, as built) Two rounds on one thread; round 2 returned "No findings." -Round 1 (2 important, 2 minor): bench.zig's explicit body.deinit left the earlier -errdefer armed on an undefined list (double-free on any later error) → clearAndFree -keeps the list valid. Ubuntu's qemu-user recommends qemu-user-binfmt, which apt -installs by default — binfmt registration would bypass the -fqemu path CI intends -to exercise → --no-install-recommends. performance.md's headroom claim overstated -the memory margin (3x, not orders of magnitude) → corrected. Ruling 7 quoted the -native aggregate test count instead of the aarch64 artifact's own (1159 + 113 -skips, fuzz excluded) → corrected. +Round 1 (2 important, 2 minor): bench.zig's explicit body.deinit left the earlier errdefer armed on an undefined list (double-free on any later error) → clearAndFree keeps the list valid. Ubuntu's qemu-user recommends qemu-user-binfmt, which apt installs by default — binfmt registration would bypass the -fqemu path CI intends to exercise → --no-install-recommends. performance.md's headroom claim overstated the memory margin (3x, not orders of magnitude) → corrected. Ruling 7 quoted the native aggregate test count instead of the aarch64 artifact's own (1159 + 113 skips, fuzz excluded) → corrected. -Final gates: plain 1171/1284 passed, 113 skipped (integration-gated), 0 failed; -integration 1280/1284, 4 skipped (live-network by design), 0 failed; qemu aarch64 -run 1159 passed, 113 skipped, 0 failed; cross ReleaseSafe with dist 18/18; no-dist -stripped sizes 5.9/5.2 MB under the 10 MiB assert. +Final gates: plain 1171/1284 passed, 113 skipped (integration-gated), 0 failed; integration 1280/1284, 4 skipped (live-network by design), 0 failed; qemu aarch64 run 1159 passed, 113 skipped, 0 failed; cross ReleaseSafe with dist 18/18; no-dist stripped sizes 5.9/5.2 MB under the 10 MiB assert. ## Module layout (new) @@ -156,8 +63,7 @@ tools/bench.zig, docs/performance.md. ## File ownership -V1 tools/bench.zig + docs/performance.md + build.zig(bench); V2 build.zig(aarch64) + -ci.yml. Orchestrator: spec sync, review, commit. +V1 tools/bench.zig + docs/performance.md + build.zig(bench); V2 build.zig(aarch64) + ci.yml. Orchestrator: spec sync, review, commit. ## Acceptance (milestone complete) @@ -174,6 +80,4 @@ ci.yml. Orchestrator: spec sync, review, commit. - No perf assertions in CI; no bench in the test step. - No qemu integration suite; no binfmt setup; no qemu-user-static. -- No new pub API on matcher/dns_cache/compiler; no synthetic-load DNS server - benchmark (the ≥100 qps target is end-to-end on the Pi — the operator recipe - covers it via the real binary, not a harness). +- No new pub API on matcher/dns_cache/compiler; no synthetic-load DNS server benchmark (the ≥100 qps target is end-to-end on the Pi — the operator recipe covers it via the real binary, not a harness). diff --git a/specs/milestone-13.md b/specs/milestone-13.md index 2115451..c98f883 100644 --- a/specs/milestone-13.md +++ b/specs/milestone-13.md @@ -1,76 +1,30 @@ # Milestone 13: restructure the documentation to Diátaxis -Goal: replace the four mixed-mode documents with the four Diátaxis modes, add the -missing tutorial, and prove every instruction by running it. +Goal: replace the four mixed-mode documents with the four Diátaxis modes, add the missing tutorial, and prove every instruction by running it. ## Rulings (binding) -1. **Four directories plus an index.** `docs/tutorial/`, `docs/how-to/`, - `docs/reference/`, `docs/explanation/`, and `docs/README.md` — the index names - the four modes, says who each is for, and links every page. The old - `docs/{operator,architecture,config-reference,api,performance}.md` are deleted - by the orchestrator once the new pages exist; no redirect stubs (greenfield - repo, AGENTS.md forbids compatibility leftovers). +1. **Four directories plus an index.** `docs/tutorial/`, `docs/how-to/`, `docs/reference/`, `docs/explanation/`, and `docs/README.md` — the index names the four modes, says who each is for, and links every page. The old `docs/{operator,architecture,config-reference,api,performance}.md` are deleted by the orchestrator once the new pages exist; no redirect stubs (greenfield repo, AGENTS.md forbids compatibility leftovers). -2. **A document serves one mode.** The defect being fixed is `operator.md`, which - interleaves install procedure, CLI reference, exit-code tables and - troubleshooting in 418 lines. Reference pages state what is; how-to pages state - what to do; explanation states why; the tutorial teaches. When porting text, - move a paragraph to the mode it belongs to rather than keeping it where it was. +2. **A document serves one mode.** The defect being fixed is `operator.md`, which interleaves install procedure, CLI reference, exit-code tables and troubleshooting in 418 lines. Reference pages state what is; how-to pages state what to do; explanation states why; the tutorial teaches. When porting text, move a paragraph to the mode it belongs to rather than keeping it where it was. -3. **Every command block in `tutorial/` and `how-to/` is executed verbatim by the - session that writes it**, on this host, before the session reports. A command - that cannot run here (needs a Raspberry Pi, root, a domain, hardware TLS) is - marked in the page itself as not verified on this host, with the reason. This - ruling exists because milestone 11's docs were written from source-reading - alone: the first real blocklist download aborted the process (commit 35f2324) - and a stale SPA bundle crashed the settings page — both would have surfaced if - the documented paths had been run. No session may report a page complete on the - strength of having read the code. This ruling constrains **command lines**; - a pasted **output line** may carry a placeholder where the literal text - would go stale, as milestone-17 ruling 8 does with `nxdns `. +3. **Every command block in `tutorial/` and `how-to/` is executed verbatim by the session that writes it**, on this host, before the session reports. A command that cannot run here (needs a Raspberry Pi, root, a domain, hardware TLS) is marked in the page itself as not verified on this host, with the reason. This ruling exists because milestone 11's docs were written from source-reading alone: the first real blocklist download aborted the process (commit 35f2324) and a stale SPA bundle crashed the settings page — both would have surfaced if the documented paths had been run. No session may report a page complete on the strength of having read the code. This ruling constrains **command lines**; a pasted **output line** may carry a placeholder where the literal text would go stale, as milestone-17 ruling 8 does with `nxdns `. -4. **Content is verified against `src/`, not copied from the old pages.** Defaults, - flags, exit codes, paths and route names come from the code at HEAD. Report any - discrepancy found; do not fix source in this milestone (docs-only, ruling 9). +4. **Content is verified against `src/`, not copied from the old pages.** Defaults, flags, exit codes, paths and route names come from the code at HEAD. Report any discrepancy found; do not fix source in this milestone (docs-only, ruling 9). -5. **The tutorial has one guaranteed outcome.** `docs/tutorial/first-run.md` takes - a reader from a clean checkout to a running nxdns that answers a query and - blocks a domain from a real blocklist, on unprivileged ports in a scratch - directory, then stops it cleanly. It teaches by doing and states what the reader - will have at the end. No branching, no options menu, no "you may also". +5. **The tutorial has one guaranteed outcome.** `docs/tutorial/first-run.md` takes a reader from a clean checkout to a running nxdns that answers a query and blocks a domain from a real blocklist, on unprivileged ports in a scratch directory, then stops it cleanly. It teaches by doing and states what the reader will have at the end. No branching, no options menu, no "you may also". 6. **File map.** - `docs/tutorial/first-run.md` — new. - - `docs/how-to/`: `install-with-systemd.md` (incl. the Raspberry Pi 5 aarch64 - binary), `install-with-docker.md`, `upgrade.md`, `troubleshoot.md`, - `enable-doh-and-dot.md`, `set-up-admin-authentication.md`, - `back-up-and-restore.md`, `measure-performance.md`. - - `docs/reference/`: `configuration.md` (every section, field, default, range), - `api.md` (all 56 operations), `cli.md` (six subcommands, every flag, exit - codes), `files-and-directories.md` (data dir layout, file modes), - `performance.md` (targets + measured numbers). - - `docs/explanation/`: `architecture.md`, `configuration-model.md` (the file - seeds the database once, the database is truth, export/import round trip), - `performance-and-testing.md` (why the targets exist, why CI does not gate on - them, what hermetic tests do and do not prove). + - `docs/how-to/`: `install-with-systemd.md` (incl. the Raspberry Pi 5 aarch64 binary), `install-with-docker.md`, `upgrade.md`, `troubleshoot.md`, `enable-doh-and-dot.md`, `set-up-admin-authentication.md`, `back-up-and-restore.md`, `measure-performance.md`. + - `docs/reference/`: `configuration.md` (every section, field, default, range), `api.md` (all 56 operations), `cli.md` (six subcommands, every flag, exit codes), `files-and-directories.md` (data dir layout, file modes), `performance.md` (targets + measured numbers). + - `docs/explanation/`: `architecture.md`, `configuration-model.md` (the file seeds the database once, the database is truth, export/import round trip), `performance-and-testing.md` (why the targets exist, why CI does not gate on them, what hermetic tests do and do not prove). -7. **Drift guards repointed and kept honest.** `docs/docs.zig` embeds - `reference/api.md`, `reference/configuration.md`, `reference/cli.md`. - `src/docs_drift_test.zig` keeps its three guards against those paths: every - served route as a full `| METHOD | \`pattern\` |` row; every `toSettings` key - verbatim; every subcommand as its own heading. `reference/cli.md` gives each - subcommand a `## \`name\`` heading so the guard anchors on structure, not prose. +7. **Drift guards repointed and kept honest.** `docs/docs.zig` embeds `reference/api.md`, `reference/configuration.md`, `reference/cli.md`. `src/docs_drift_test.zig` keeps its three guards against those paths: every served route as a full `| METHOD | \`pattern\` |` row; every `toSettings` key verbatim; every subcommand as its own heading. `reference/cli.md` gives each subcommand a `## \`name\`` heading so the guard anchors on structure, not prose. -8. **Style.** Plain sentences. No banned vocabulary (leverage, seamless, robust as - filler, rule-of-three padding, "it's important to note"). No badges, no - marketing. Code blocks are copy-pasteable and use the scratch paths the - tutorial establishes, not `/var/lib/nxdns`, unless the page is about a real - install. +8. **Style.** Plain sentences. No banned vocabulary (leverage, seamless, robust as filler, rule-of-three padding, "it's important to note"). No badges, no marketing. Code blocks are copy-pasteable and use the scratch paths the tutorial establishes, not `/var/lib/nxdns`, unless the page is about a real install. -9. **Docs-only.** The only non-docs edits are the orchestrator's: `docs/docs.zig`, - `src/docs_drift_test.zig`, `README.md` links, and PLAN.md's repo-layout line. - No behavior changes. +9. **Docs-only.** The only non-docs edits are the orchestrator's: `docs/docs.zig`, `src/docs_drift_test.zig`, `README.md` links, and PLAN.md's repo-layout line. No behavior changes. ## Sessions @@ -78,226 +32,101 @@ X1, X2, X3, X4, X5 run in parallel — every session owns distinct files. ## Session X1: index + tutorial + README links -Owns `docs/README.md`, `docs/tutorial/first-run.md`, `README.md` (links section -only). Rulings 1, 5, 3 (the tutorial is executed end to end). +Owns `docs/README.md`, `docs/tutorial/first-run.md`, `README.md` (links section only). Rulings 1, 5, 3 (the tutorial is executed end to end). ## Session X2: how-to, install and operations -Owns `docs/how-to/{install-with-systemd,install-with-docker,upgrade,troubleshoot}.md`. -Rulings 2, 3, 4. +Owns `docs/how-to/{install-with-systemd,install-with-docker,upgrade,troubleshoot}.md`. Rulings 2, 3, 4. ## Session X3: how-to, security and measurement -Owns `docs/how-to/{enable-doh-and-dot,set-up-admin-authentication,back-up-and-restore,measure-performance}.md`. -Rulings 2, 3, 4. +Owns `docs/how-to/{enable-doh-and-dot,set-up-admin-authentication,back-up-and-restore,measure-performance}.md`. Rulings 2, 3, 4. ## Session X4: reference -Owns `docs/reference/{configuration,api,cli,files-and-directories,performance}.md`. -Rulings 2, 4, 7 (heading convention). +Owns `docs/reference/{configuration,api,cli,files-and-directories,performance}.md`. Rulings 2, 4, 7 (heading convention). ## Session X5: explanation -Owns `docs/explanation/{architecture,configuration-model,performance-and-testing}.md`. -Rulings 2, 4. +Owns `docs/explanation/{architecture,configuration-model,performance-and-testing}.md`. Rulings 2, 4. ## File ownership -X1 docs/README.md + docs/tutorial/* + README.md; X2 and X3 disjoint files under -docs/how-to/; X4 docs/reference/*; X5 docs/explanation/*. Orchestrator: -docs/docs.zig, src/docs_drift_test.zig, PLAN.md, deletion of the old pages, spec. +X1 docs/README.md + docs/tutorial/* + README.md; X2 and X3 disjoint files under docs/how-to/; X4 docs/reference/*; X5 docs/explanation/*. Orchestrator: docs/docs.zig, src/docs_drift_test.zig, PLAN.md, deletion of the old pages, spec. ## Delivered -Seventeen pages: `docs/README.md`, one tutorial, eight how-to guides, five -reference pages, three explanation pages. The five old documents are deleted. +Seventeen pages: `docs/README.md`, one tutorial, eight how-to guides, five reference pages, three explanation pages. The five old documents are deleted. -The orchestrator repointed `docs/docs.zig` at `reference/{api,configuration,cli}.md` -and `src/docs_drift_test.zig` at the same three, moved the subcommand anchor from -`### \`name` to `## \`name`, fixed the README Quickstart link that pointed at the -deleted `operator.md`, and corrected PLAN.md's repo-layout line and its §13.2 -OpenAPI paragraph (which still promised a renderer and contract tests that were -never built). +The orchestrator repointed `docs/docs.zig` at `reference/{api,configuration,cli}.md` and `src/docs_drift_test.zig` at the same three, moved the subcommand anchor from `### \`name` to `## \`name`, fixed the README Quickstart link that pointed at the deleted `operator.md`, and corrected PLAN.md's repo-layout line and its §13.2 OpenAPI paragraph (which still promised a renderer and contract tests that were never built). -All three guards were proven able to fail: deleting one API table row, one -settings key row and one `## \`export` heading each produced a named build -failure. The settings guard was strengthened during review — it searched for a -bare key, which prose and the annotated example could satisfy after a field's row -was deleted; it now anchors on `| \`key\` |`. +All three guards were proven able to fail: deleting one API table row, one settings key row and one `## \`export` heading each produced a named build failure. The settings guard was strengthened during review — it searched for a bare key, which prose and the annotated example could satisfy after a field's row was deleted; it now anchors on `| \`key\` |`. -Ruling 3 held. The tutorial ran end to end twice, the second time after the review -fixes. DoH and DoT answered real queries, `POST /api/certs/reload` returned both -its success and its failure payload, the export/import round trip was -byte-identical, a Docker image was built and its container served DNS, and the -offline password change was proven (old password 401, new 200). What could not run -here — root-only steps, a second LAN host, a Raspberry Pi, a registry push, the -full-scale bench, `dnsperf` — is marked in the page that documents it, with the -reason. +Ruling 3 held. The tutorial ran end to end twice, the second time after the review fixes. DoH and DoT answered real queries, `POST /api/certs/reload` returned both its success and its failure payload, the export/import round trip was byte-identical, a Docker image was built and its container served DNS, and the offline password change was proven (old password 401, new 200). What could not run here — root-only steps, a second LAN host, a Raspberry Pi, a registry push, the full-scale bench, `dnsperf` — is marked in the page that documents it, with the reason. -Five review rounds: 15 findings, then 8, then 1, then 1, then clean. One round-1 -finding was rejected on evidence (`troubleshoot.md` already stated the verified -exit codes; the contradiction was in `reference/cli.md`). +Five review rounds: 15 findings, then 8, then 1, then 1, then clean. One round-1 finding was rejected on evidence (`troubleshoot.md` already stated the verified exit codes; the contradiction was in `reference/cli.md`). ## Fix wave (supersedes ruling 9) -Ruling 9 kept the documentation wave docs-only so that writing pages could not -churn behaviour underneath itself. That job is done and committed at 16c9de2. -The nine discrepancies the wave found are now fixed inside milestone 13, on the -user's direction; milestone 14 stays packaging and publishing. +Ruling 9 kept the documentation wave docs-only so that writing pages could not churn behaviour underneath itself. That job is done and committed at 16c9de2. The nine discrepancies the wave found are now fixed inside milestone 13, on the user's direction; milestone 14 stays packaging and publishing. Binding rulings for the fix wave: -F-a. **One definition of "the operator's configuration is wrong."** The -divergence in D1 exists because `app.isConfigFault` and `cli.failureExitCode` -each carry their own list. Neither list is the fix. Add `src/config/faults.zig` -with exactly: +F-a. **One definition of "the operator's configuration is wrong."** The divergence in D1 exists because `app.isConfigFault` and `cli.failureExitCode` each carry their own list. Neither list is the fix. Add `src/config/faults.zig` with exactly: ```zig pub fn isConfigFault(err: anyerror) bool ``` -covering the seed and validation errors (`ParseZon`, `ConfigTooLarge`, -`MissingDefaultGroup`, `NoUpstreams`, `NoUsableUpstreams`, `BadBindAddress`, -`BadRateLimit`, `BadCertificate`, `PasswordAndHashBothSet`, and every -`validate.ValidateError`). `app.zig` and `cli.zig` both call it and keep no -private list. `run`, `check` and `import` then agree: a rejected configuration -file is exit 2 from every subcommand. +covering the seed and validation errors (`ParseZon`, `ConfigTooLarge`, `MissingDefaultGroup`, `NoUpstreams`, `NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit`, `BadCertificate`, `PasswordAndHashBothSet`, and every `validate.ValidateError`). `app.zig` and `cli.zig` both call it and keep no private list. `run`, `check` and `import` then agree: a rejected configuration file is exit 2 from every subcommand. -F-b. **Diagnostics carry severity.** `validate.Diagnostics` gains a severity per -item — `.fail` or `.warn`. `writeAll` prints `FAIL `/`WARN ` accordingly. -Counting splits: failures set exit 2, warnings never change an exit code. This -is the API D2 and D5 both need, so it is pinned here rather than invented twice. +F-b. **Diagnostics carry severity.** `validate.Diagnostics` gains a severity per item — `.fail` or `.warn`. `writeAll` prints `FAIL `/`WARN ` accordingly. Counting splits: failures set exit 2, warnings never change an exit code. This is the API D2 and D5 both need, so it is pinned here rather than invented twice. -F-c. **`check` never writes.** It opens `config.db` read-only and does not -migrate. A database behind the current schema is reported, not upgraded. If a -read-only open is impossible for a database needing WAL recovery, report that as -a failure naming `nxdns run` as the fix — do not silently fall back to a -writable open. +F-c. **`check` never writes.** It opens `config.db` read-only and does not migrate. A database behind the current schema is reported, not upgraded. If a read-only open is impossible for a database needing WAL recovery, report that as a failure naming `nxdns run` as the fix — do not silently fall back to a writable open. -F-d. **`check` proves what it claims.** It parses the certificate and the key and -verifies they pair, through the same code the server uses, so a green `check` -cannot be followed by `run` exiting 2 on `BadCertificate`. +F-d. **`check` proves what it claims.** It parses the certificate and the key and verifies they pair, through the same code the server uses, so a green `check` cannot be followed by `run` exiting 2 on `BadCertificate`. -F-e. **Documentation follows behaviour in the same wave.** Every page that -documents the old behaviour is corrected: exit codes in `reference/cli.md`, -`how-to/troubleshoot.md`, `how-to/install-with-docker.md`, `tutorial/first-run.md`; -the certificate gap in `how-to/troubleshoot.md` and `how-to/enable-doh-and-dot.md`; -the "check writes" note in `reference/files-and-directories.md`. The three drift -guards stay green. Ruling 3 still binds: a changed command is re-run here. +F-e. **Documentation follows behaviour in the same wave.** Every page that documents the old behaviour is corrected: exit codes in `reference/cli.md`, `how-to/troubleshoot.md`, `how-to/install-with-docker.md`, `tutorial/first-run.md`; the certificate gap in `how-to/troubleshoot.md` and `how-to/enable-doh-and-dot.md`; the "check writes" note in `reference/files-and-directories.md`. The three drift guards stay green. Ruling 3 still binds: a changed command is re-run here. F-f. **Every fix ships with a test that fails without it.** ## The nine discrepancies -1. `nxdns run` exits 1 for seed-file errors (`ParseZon`, `MissingDefaultGroup`, - `NoUpstreams`) while `check` and `import` exit 2 for the same file, because - `app.isConfigFault` lists only `NoUsableUpstreams`, `BadBindAddress`, - `BadRateLimit`, `BadCertificate`. -2. `nxdns check` prints `OK: no problems found` and exits 0 after emitting WARN - lines. -3. `nxdns check` never parses a certificate or tests that the key matches it — it - checks path readability and key mode only. A green `check` is followed by - `run` exiting 2 with `BadCertificate` on a mismatched pair. -4. `check --config ` exits 1; the implicit path prints `nothing to - check` and exits 2. -5. A blocklist source attached to no group is silently inert, and `validate.zig` - has no diagnostic for it. -6. `nxdns check` says it validates without writing, but the `config.db` branch - chmods, enables WAL and runs migrations. -7. `pruneOrphans` matches only `.list`/`.wild`, so an orphaned `.raw.tmp` is never - swept. -8. `isEmpty` counts auto-materialised client rows, so a server that has answered - one query ignores a seed file placed afterwards. +1. `nxdns run` exits 1 for seed-file errors (`ParseZon`, `MissingDefaultGroup`, `NoUpstreams`) while `check` and `import` exit 2 for the same file, because `app.isConfigFault` lists only `NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit`, `BadCertificate`. +2. `nxdns check` prints `OK: no problems found` and exits 0 after emitting WARN lines. +3. `nxdns check` never parses a certificate or tests that the key matches it — it checks path readability and key mode only. A green `check` is followed by `run` exiting 2 with `BadCertificate` on a mismatched pair. +4. `check --config ` exits 1; the implicit path prints `nothing to check` and exits 2. +5. A blocklist source attached to no group is silently inert, and `validate.zig` has no diagnostic for it. +6. `nxdns check` says it validates without writing, but the `config.db` branch chmods, enables WAL and runs migrations. +7. `pruneOrphans` matches only `.list`/`.wild`, so an orphaned `.raw.tmp` is never swept. +8. `isEmpty` counts auto-materialised client rows, so a server that has answered one query ignores a seed file placed afterwards. 9. PLAN §18 and the docs say 10 MB / 15 MB; CI asserts 10 MiB / 15 MiB. ## Fix wave delivered -All nine are closed. Each shipped with a test its author watched fail with the -implementation reverted — ruling F-f was enforced by demanding the observed -failure output, not an assertion that a test would fail. +All nine are closed. Each shipped with a test its author watched fail with the implementation reverted — ruling F-f was enforced by demanding the observed failure output, not an assertion that a test would fail. -D1 is one `src/config/faults.zig` deriving the fault set by comptime reflection -over `validate.ValidateError`, with no exclusion list; `run`, `check` and -`import` all exit 2 on the same rejected file, for `MissingDefaultGroup` and for -`ParseZon`. D2 and D5 rest on `validate.Diagnostics` gaining `.fail`/`.warn`: -`check` prints `OK: no failures found, 1 warning` and exits 0. D3 proves the -certificate pair through `cert_store.CertStore.init`, the same code the -listeners use, so a green `check` cannot be followed by `run` exiting 2 on -`BadCertificate`. D4 fixes the missing-file exit at the root read. D6 opens -`config.db` with `OpenMode.immutable` and no migrate; a stale `-wal` is reported -naming `nxdns run`, never read past. D7 sweeps all five suffixes. D8 counts -`hand_edited = 1` only. D9 is MiB everywhere CI asserts MiB. +D1 is one `src/config/faults.zig` deriving the fault set by comptime reflection over `validate.ValidateError`, with no exclusion list; `run`, `check` and `import` all exit 2 on the same rejected file, for `MissingDefaultGroup` and for `ParseZon`. D2 and D5 rest on `validate.Diagnostics` gaining `.fail`/`.warn`: `check` prints `OK: no failures found, 1 warning` and exits 0. D3 proves the certificate pair through `cert_store.CertStore.init`, the same code the listeners use, so a green `check` cannot be followed by `run` exiting 2 on `BadCertificate`. D4 fixes the missing-file exit at the root read. D6 opens `config.db` with `OpenMode.immutable` and no migrate; a stale `-wal` is reported naming `nxdns run`, never read past. D7 sweeps all five suffixes. D8 counts `hand_edited = 1` only. D9 is MiB everywhere CI asserts MiB. Four defects the wave found that were not among the nine: -1. **`pruneOrphans` had no production caller.** D7's widened matching was - unreachable at runtime. `Manager.sweepOrphans` now runs at startup, before - each interval pass, and on `DELETE /api/blocklists/:id`. -2. **An import destroyed a device's observed timestamps.** `first_seen` and - `last_seen` are runtime state, not configuration, and the model carries no - field for either — so a configured row was inserted with the import clock in - both columns. They now follow the address: a device the database already knew - keeps them, only an unseen address takes the import's clock, and a client the - file omits is removed with its history. Verified live, not only in tests. -3. **A blocklist or upstream url reached the log whole.** A signed url or an - `?apikey=` query persisted in journald. `src/safe_url.zig` exports `redact`, - which keeps scheme, host and port and drops userinfo, path, query and - fragment. It scans rather than parses, deliberately: `error.BadUrl` is one of the - failures these very lines report, so the inputs a parser refuses are exactly - the ones that must still redact. Twenty-two call sites across `manager.zig`, - `validate.zig`, `cli.zig` and `app.zig`. -4. **`src/main.zig` built both runner writers in positional mode.** With stderr - redirected to a regular file, runner output pwrote over what `std.log` had - already written at offset 0, and `2>>` was silently broken because pwrite - ignores `O_APPEND`. Both writers now use `writerStreaming`. +1. **`pruneOrphans` had no production caller.** D7's widened matching was unreachable at runtime. `Manager.sweepOrphans` now runs at startup, before each interval pass, and on `DELETE /api/blocklists/:id`. +2. **An import destroyed a device's observed timestamps.** `first_seen` and `last_seen` are runtime state, not configuration, and the model carries no field for either — so a configured row was inserted with the import clock in both columns. They now follow the address: a device the database already knew keeps them, only an unseen address takes the import's clock, and a client the file omits is removed with its history. Verified live, not only in tests. +3. **A blocklist or upstream url reached the log whole.** A signed url or an `?apikey=` query persisted in journald. `src/safe_url.zig` exports `redact`, which keeps scheme, host and port and drops userinfo, path, query and fragment. It scans rather than parses, deliberately: `error.BadUrl` is one of the failures these very lines report, so the inputs a parser refuses are exactly the ones that must still redact. Twenty-two call sites across `manager.zig`, `validate.zig`, `cli.zig` and `app.zig`. +4. **`src/main.zig` built both runner writers in positional mode.** With stderr redirected to a regular file, runner output pwrote over what `std.log` had already written at offset 0, and `2>>` was silently broken because pwrite ignores `O_APPEND`. Both writers now use `writerStreaming`. -Two review findings were answered against the reviewer rather than by it, both -with evidence rather than argument: +Two review findings were answered against the reviewer rather than by it, both with evidence rather than argument: -- **A failed stderr write must not stop the server.** The reviewer wanted - `seedFromFile` to propagate an output failure as a runtime failure. Implementing - that proposal and running the suite showed it replacing - `error.MissingDefaultGroup` with `error.WriteFailed` — a broken stderr would - hide why the seed file was refused. The discards stay, and the reasoning now - sits above them, with the derived half of the claim labelled as derived. -- **The log-injection hole was already closed for `std.log`.** `logging.zig:342` - escapes control bytes in every log message, so the forged-line scenario the - reviewer described could not happen through that path. The live gap was - `cli.zig`'s stdout. Escaping stays in `safe_url` as well as the sink, because - `validate.Diagnostics` builds `Problem.message` as an allocated string that - `web/handlers/mutations.zig` returns as a 400 body — a channel no log sink can - escape. +- **A failed stderr write must not stop the server.** The reviewer wanted `seedFromFile` to propagate an output failure as a runtime failure. Implementing that proposal and running the suite showed it replacing `error.MissingDefaultGroup` with `error.WriteFailed` — a broken stderr would hide why the seed file was refused. The discards stay, and the reasoning now sits above them, with the derived half of the claim labelled as derived. +- **The log-injection hole was already closed for `std.log`.** `logging.zig:342` escapes control bytes in every log message, so the forged-line scenario the reviewer described could not happen through that path. The live gap was `cli.zig`'s stdout. Escaping stays in `safe_url` as well as the sink, because `validate.Diagnostics` builds `Problem.message` as an allocated string that `web/handlers/mutations.zig` returns as a 400 body — a channel no log sink can escape. -Redaction ended stricter than it started. `redact` prints scheme, host and port -only: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the path -segment is the whole account identifier, so keeping the path kept the credential. -The rule has no exemption for `nxdns check`'s stdout, which is the output an -operator pastes into a bug report. Log lines identify a source by row id and name -instead, and the two duplicate diagnostics now name the other entry -(`duplicate of upstreams[0]`) rather than quoting a url that no longer shows why -the two collide. +Redaction ended stricter than it started. `redact` prints scheme, host and port only: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the path segment is the whole account identifier, so keeping the path kept the credential. The rule has no exemption for `nxdns check`'s stdout, which is the output an operator pastes into a bug report. Log lines identify a source by row id and name instead, and the two duplicate diagnostics now name the other entry (`duplicate of upstreams[0]`) rather than quoting a url that no longer shows why the two collide. -One credential redaction cannot remove, pinned as a test and documented rather -than hidden: a NextDNS **DoT** upstream is `tls://abcd12.dns.nextdns.io`, which -carries the same identifier in the hostname. Removing it would leave no host and -no actionable line. A hostname is resolved publicly and offered as SNI in any -case, so it is not private the way a query string is, and dropping every host -would cost every operator a diagnostic to cover one vendor's choice. The -reference page names the mitigation the operator controls instead: NextDNS also -publishes a DoH endpoint whose identifier sits in the path and is redacted whole. +One credential redaction cannot remove, pinned as a test and documented rather than hidden: a NextDNS **DoT** upstream is `tls://abcd12.dns.nextdns.io`, which carries the same identifier in the hostname. Removing it would leave no host and no actionable line. A hostname is resolved publicly and offered as SNI in any case, so it is not private the way a query string is, and dropping every host would cost every operator a diagnostic to cover one vendor's choice. The reference page names the mitigation the operator controls instead: NextDNS also publishes a DoH endpoint whose identifier sits in the path and is redacted whole. -Four rounds of review hardened `redact` and each one found what the last missed. -Three of the four were the same root cause, which is worth naming because the -first two fixes treated it as bad luck: **the scanner tried to identify a host -inside text that is not a url, and guessed.** It guessed the query was safe to cut -before the userinfo, leaving `user:pa55` as the host. It guessed a `\` was not a -separator. And it guessed which side of an `@` was the host when a late delimiter -made both readings available. +Four rounds of review hardened `redact` and each one found what the last missed. Three of the four were the same root cause, which is worth naming because the first two fixes treated it as bad luck: **the scanner tried to identify a host inside text that is not a url, and guessed.** It guessed the query was safe to cut before the userinfo, leaving `user:pa55` as the host. It guessed a `\` was not a separator. And it guessed which side of an `@` was the host when a late delimiter made both readings available. -That third one was recorded here, in an earlier revision of this section, as an -accepted cost: an `@` in a backslash path prints "a misleading host, never a -credential". **That claim was false and is retracted.** Running the shipped code -disproved it: +That third one was recorded here, in an earlier revision of this section, as an accepted cost: an `@` in a backslash path prints "a misleading host, never a credential". **That claim was false and is retracted.** Running the shipped code disproved it: ``` https://lists.example?token=prefix@hunter2 -> https://hunter2 @@ -305,277 +134,79 @@ https://lists.example#f@hunter2 -> https://hunter2 https:\\lists.example\p@hunter2 -> https://hunter2 ``` -A query string is the most likely place in a url for a token, so the text the -scanner promoted to "host" was the secret itself. The note claiming otherwise is -why three subsequent rounds passed over it. +A query string is the most likely place in a url for a token, so the text the scanner promoted to "host" was the secret itself. The note claiming otherwise is why three subsequent rounds passed over it. -The fix is a rule rather than a fourth special case: **the scan no longer guesses, -it declines.** The `/` cut now runs first and unconditionally, which is safe for -the reason the old ordering missed — everything after the `/` is dropped anyway, -so an `@` there never needed to be userinfo to stay out of the log. Where two -readings genuinely survive, the authority is omitted whole and `format` prints -`(ambiguous authority omitted)`, which is prose rather than a placeholder host so -an operator reads it as a statement about the line. `SafeUrl.authority` is -therefore `?[]const u8`: empty and `null` are different answers, one saying the -url names no authority and the other saying it names one that cannot be resolved. -The same rule caught a case nobody had raised — `https:a@hunter2`, where RFC 3986 -reads `hunter2` as a path segment and WHATWG reads it as the host. The -disagreement between two parsers is itself the evidence of ambiguity. +The fix is a rule rather than a fourth special case: **the scan no longer guesses, it declines.** The `/` cut now runs first and unconditionally, which is safe for the reason the old ordering missed — everything after the `/` is dropped anyway, so an `@` there never needed to be userinfo to stay out of the log. Where two readings genuinely survive, the authority is omitted whole and `format` prints `(ambiguous authority omitted)`, which is prose rather than a placeholder host so an operator reads it as a statement about the line. `SafeUrl.authority` is therefore `?[]const u8`: empty and `null` are different answers, one saying the url names no authority and the other saying it names one that cannot be resolved. The same rule caught a case nobody had raised — `https:a@hunter2`, where RFC 3986 reads `hunter2` as a path segment and WHATWG reads it as the host. The disagreement between two parsers is itself the evidence of ambiguity. -A redesign was considered and rejected: parse with `std.Uri.parse` first and print -nothing but the scheme when the parse fails. It reaches the same "do not guess" -place, but it also discards the host for every url malformed in a harmless way, -and `redact` exists to be callable from the `error.BadUrl` paths that report -exactly those. The ordering fix gets the property without the cost. +A redesign was considered and rejected: parse with `std.Uri.parse` first and print nothing but the scheme when the parse fails. It reaches the same "do not guess" place, but it also discards the host for every url malformed in a harmless way, and `redact` exists to be callable from the `error.BadUrl` paths that report exactly those. The ordering fix gets the property without the cost. -Escaping is owned by the type that introduces the delimiter: `quoteText` writes -its own quotes and escapes `'` inside them, so a caller cannot reopen the hole by -adding quotes of its own. Escaping inside a plain helper would have left the -defect one caller away, which on a third review round is not a fix. +Escaping is owned by the type that introduces the delimiter: `quoteText` writes its own quotes and escapes `'` inside them, so a caller cannot reopen the hole by adding quotes of its own. Escaping inside a plain helper would have left the defect one caller away, which on a third review round is not a fix. -`SafeUrl` was then found to break that same rule from the other side. Its `format` -hardcoded the `none` delimiter, so it never escaped `'` — while eight call sites -wrapped it in `'{f}'` of their own. `https://ho'st/x` redacts to `https://ho'st`, -which inside a caller's quotes reads as `'ho'` followed by loose text. Two -independent findings converged on it: this review, and the `/metrics` work, which -hit the same shape with `"` instead of `'`. +`SafeUrl` was then found to break that same rule from the other side. Its `format` hardcoded the `none` delimiter, so it never escaped `'` — while eight call sites wrapped it in `'{f}'` of their own. `https://ho'st/x` redacts to `https://ho'st`, which inside a caller's quotes reads as `'ho'` followed by loose text. Two independent findings converged on it: this review, and the `/metrics` work, which hit the same shape with `"` instead of `'`. -`redactQuoted` closes it, and the choice between the two is a stated rule rather -than per-call-site judgement: **use the quoted form whenever anything follows the -url on the line**, because a redacted authority can still hold a space, a `:` and a -`'`, and unquoted it can impersonate whatever comes next — `upstream {f} failed: -{t}` with an authority of `ok failed: Timeout` reports a failure that did not -happen. Bare `redact` is for the two cases where that cannot arise: the url ends -the line (`cli.zig`'s `OK upstreams[N]`, `context.zig`'s missing-id line), or the -caller owns the escaping for a delimiter of its own (`metrics.zig`). Ten sites -moved to the quoted form. The output is byte-identical for any url without a `'`, -so no documented output changed. +`redactQuoted` closes it, and the choice between the two is a stated rule rather than per-call-site judgement: **use the quoted form whenever anything follows the url on the line**, because a redacted authority can still hold a space, a `:` and a `'`, and unquoted it can impersonate whatever comes next — `upstream {f} failed: {t}` with an authority of `ok failed: Timeout` reports a failure that did not happen. Bare `redact` is for the two cases where that cannot arise: the url ends the line (`cli.zig`'s `OK upstreams[N]`, `context.zig`'s missing-id line), or the caller owns the escaping for a delimiter of its own (`metrics.zig`). Ten sites moved to the quoted form. The output is byte-identical for any url without a `'`, so no documented output changed. -One property is asserted rather than assumed, because it is what makes a `\` in the -output always this file's and never the operator's: a `\` ends an authority, so -unlike a source name it can never reach the value to be doubled. +One property is asserted rather than assumed, because it is what makes a `\` in the output always this file's and never the operator's: a `\` ends an authority, so unlike a source name it can never reach the value to be doubled. -Round five found two more, both in the same scanner, which is now five rounds and -five findings. Both were confirmed by running the code before being fixed. +Round five found two more, both in the same scanner, which is now five rounds and five findings. Both were confirmed by running the code before being fixed. -The first is the ambiguity rule applied to only half its cases. `https:a@hunter2` -was withheld, but `https:hunter2` — the same opaque path with no `@` in it — -printed whole, because the check sat *after* the `@` lookup and a url with no `@` -returned before reaching it. The reading is ambiguous either way; the `@` was never -what made it so. The check now runs before the `@` lookup. +The first is the ambiguity rule applied to only half its cases. `https:a@hunter2` was withheld, but `https:hunter2` — the same opaque path with no `@` in it — printed whole, because the check sat *after* the `@` lookup and a url with no `@` returned before reaching it. The reading is ambiguous either way; the `@` was never what made it so. The check now runs before the `@` lookup. -Moving it exposed the reason it had been placed there: `localhost` satisfies the -scheme production, so a rule keyed on the production alone withholds -`localhost:8080/x`, which is the shape an operator on a LAN is most likely to -write. A port is not an opaque path, so the digits after the colon are what -separate the two. This is checked against the trimmed authority, not the raw one, -or `localhost:8080?x` would fail the digit test on the query. +Moving it exposed the reason it had been placed there: `localhost` satisfies the scheme production, so a rule keyed on the production alone withholds `localhost:8080/x`, which is the shape an operator on a LAN is most likely to write. A port is not an opaque path, so the digits after the colon are what separate the two. This is checked against the trimmed authority, not the raw one, or `localhost:8080?x` would fail the digit test on the query. -The second: a network-path reference (`//lists.example/hosts.txt`) redacted to the -empty string, because the unconditional `/` cut lands at byte zero. Not a leak — -nothing was printed — but the line named no source at all, and the authority in it -is not in doubt. A leading run of `/` is now consumed the way a scheme delimiter's -is, so the userinfo in `//user:pa55@lists.example/x` is dropped rather than the -whole authority withheld. An ambiguous one such as `//a?b@c` is still withheld: the -branch settles where the authority starts, not that every reading of it is -resolved. +The second: a network-path reference (`//lists.example/hosts.txt`) redacted to the empty string, because the unconditional `/` cut lands at byte zero. Not a leak — nothing was printed — but the line named no source at all, and the authority in it is not in doubt. A leading run of `/` is now consumed the way a scheme delimiter's is, so the userinfo in `//user:pa55@lists.example/x` is dropped rather than the whole authority withheld. An ambiguous one such as `//a?b@c` is still withheld: the branch settles where the authority starts, not that every reading of it is resolved. -Round six found three more in the same function, all the same class again: a -separator run that was read as an authority delimiter when it does not settle one. +Round six found three more in the same function, all the same class again: a separator run that was read as an authority delimiter when it does not settle one. -`https:/hunter2` and `https:///hunter2` printed the path segment as the host. The -scan accepted a delimiter of *any* number of separators, and the comment recording -why was accurate when it was written and stale by the time it was read: the -tolerance existed so `https:/user:pass@host/list` would find an authority instead -of printing its userinfo. That reason expired when the `/` cut moved ahead of the -userinfo lookup in round four — the authority of a url with no `://` now ends at -its first `/`, so it holds no userinfo to print. Only a run of exactly two -introduces an authority; one leaves an absolute path and three or more is an empty -authority to RFC 3986 and a host to WHATWG. The four inputs that motivated the old -tolerance are still safe, now by being withheld rather than resolved, and that is -asserted where the old behaviour used to be. +`https:/hunter2` and `https:///hunter2` printed the path segment as the host. The scan accepted a delimiter of *any* number of separators, and the comment recording why was accurate when it was written and stale by the time it was read: the tolerance existed so `https:/user:pass@host/list` would find an authority instead of printing its userinfo. That reason expired when the `/` cut moved ahead of the userinfo lookup in round four — the authority of a url with no `://` now ends at its first `/`, so it holds no userinfo to print. Only a run of exactly two introduces an authority; one leaves an absolute path and three or more is an empty authority to RFC 3986 and a host to WHATWG. The four inputs that motivated the old tolerance are still safe, now by being withheld rather than resolved, and that is asserted where the old behaviour used to be. -`///lists.example/x` had the same defect in the network-path branch added one -round earlier, which consumed the whole run. Exactly two there too. +`///lists.example/x` had the same defect in the network-path branch added one round earlier, which consumed the whole run. Exactly two there too. -The third retracts something this section claimed one round ago. `isPort` was -introduced so a schemeless `localhost:8080` would keep resolving, on the reasoning -that a port is not an opaque path. It is not that simple: `https:123456` is an -opaque path whose digits are as much a token as any other text, and the exception -printed it whole. The reviewer proposed excluding known schemes from the -exception; the exception is gone instead, because a list of known schemes is the -kind of thing that goes stale silently and this file already has one rule that -covers it. `localhost:8080/x` is now withheld, which costs nothing an operator -needs: a url reaching this without a scheme is one the validator is rejecting, and -the field path beside it names which. An `IP:port` is unaffected — a leading digit -fails the scheme production, so `10.0.0.2:8080` and `[::1]:853` still resolve. +The third retracts something this section claimed one round ago. `isPort` was introduced so a schemeless `localhost:8080` would keep resolving, on the reasoning that a port is not an opaque path. It is not that simple: `https:123456` is an opaque path whose digits are as much a token as any other text, and the exception printed it whole. The reviewer proposed excluding known schemes from the exception; the exception is gone instead, because a list of known schemes is the kind of thing that goes stale silently and this file already has one rule that covers it. `localhost:8080/x` is now withheld, which costs nothing an operator needs: a url reaching this without a scheme is one the validator is rejecting, and the field path beside it names which. An `IP:port` is unaffected — a leading digit fails the scheme production, so `10.0.0.2:8080` and `[::1]:853` still resolve. -Round seven took the same rule one step further. "Exactly two separators" still -accepted `\\`, `/\` and `\/`, and WHATWG converts a `\` to a `/` only for a -*special* scheme — so `https:\\hunter2` has the same split reading as the cases -above, and `tls:\\hunter2` has no reading at all under which `hunter2` is a host, -`tls` not being special. RFC 3986 gives `\` no meaning anywhere. Only `//` opens -an authority now. A `\` still ends one, and still anchors the scheme scan so a -backslash-pasted url is reported by its scheme, but it opens nothing. +Round seven took the same rule one step further. "Exactly two separators" still accepted `\\`, `/\` and `\/`, and WHATWG converts a `\` to a `/` only for a *special* scheme — so `https:\\hunter2` has the same split reading as the cases above, and `tls:\\hunter2` has no reading at all under which `hunter2` is a host, `tls` not being special. RFC 3986 gives `\` no meaning anywhere. Only `//` opens an authority now. A `\` still ends one, and still anchors the scheme scan so a backslash-pasted url is reported by its scheme, but it opens nothing. -Round seven also caught a stale comment, which is the second time in two rounds -that a comment outlived the reason it recorded. Both said something true when -written and false when read, and both were load-bearing — the first is why the -any-length tolerance survived three rounds after its justification expired. +Round seven also caught a stale comment, which is the second time in two rounds that a comment outlived the reason it recorded. Both said something true when written and false when read, and both were load-bearing — the first is why the any-length tolerance survived three rounds after its justification expired. -Seven rounds, nine findings, one function. Every one was a place where the scan -committed to a reading the text did not support. What finally holds is not a -sharper scan but a smaller claim: the authority is printed only where exactly one -reading survives, and withheld everywhere else. Concretely, an authority is read -only after a literal `//`, whether a scheme introduced it or not. +Seven rounds, nine findings, one function. Every one was a place where the scan committed to a reading the text did not support. What finally holds is not a sharper scan but a smaller claim: the authority is printed only where exactly one reading survives, and withheld everywhere else. Concretely, an authority is read only after a literal `//`, whether a scheme introduced it or not. -The cost is paid by malformed input alone, and it is paid in diagnostic detail -rather than in safety: a backslash-pasted or schemeless url now reports its scheme -and no host. Every such url is one the validator is already rejecting, and the -field path or row id beside it names which one. A sweep of eighteen shapes -carrying `hunter2`, `abcd12`, `s3cr3t`, `token`, `pa55` and `123456` finds none of -them in any output. +The cost is paid by malformed input alone, and it is paid in diagnostic detail rather than in safety: a backslash-pasted or schemeless url now reports its scheme and no host. Every such url is one the validator is already rejecting, and the field path or row id beside it names which one. A sweep of eighteen shapes carrying `hunter2`, `abcd12`, `s3cr3t`, `token`, `pa55` and `123456` finds none of them in any output. -Round eight returned no critical and no important finding, and two minor ones that -cut in opposite directions. A run of three or more slashes was reported as an -authority the url does not have, when it is contested rather than absent — RFC -3986 reads an empty authority, WHATWG resolving against a special-scheme base -reads a host. `SafeUrl` exists to keep those two answers apart, so it is now -withheld rather than reported as empty. +Round eight returned no critical and no important finding, and two minor ones that cut in opposite directions. A run of three or more slashes was reported as an authority the url does not have, when it is contested rather than absent — RFC 3986 reads an empty authority, WHATWG resolving against a special-scheme base reads a host. `SafeUrl` exists to keep those two answers apart, so it is now withheld rather than reported as empty. -The other corrected a claim this file made about the very thing it was withholding. -`localhost:8080` is **not** contested: `localhost` is not one of WHATWG's six -special schemes, so both standards read a scheme and an opaque path, and the -host-and-port an operator meant is a reading no parser offers. It is still -withheld — the text after the colon is a path segment under every reading — but the -marker calls it an authority that could not be resolved when the honest answer is -that there is none. +The other corrected a claim this file made about the very thing it was withholding. `localhost:8080` is **not** contested: `localhost` is not one of WHATWG's six special schemes, so both standards read a scheme and an opaque path, and the host-and-port an operator meant is a reading no parser offers. It is still withheld — the text after the colon is a path segment under every reading — but the marker calls it an authority that could not be resolved when the honest answer is that there is none. -That wording defect was first recorded rather than fixed, on the grounds that -telling the two apart needs a scheme list and a scheme list is what the previous -round had just removed. Round nine rejected that and was right to: the list round -six removed described *what nxdns supports*, so it went stale whenever a transport -was added. WHATWG's special schemes are a closed set fixed by the URL Standard — -`ftp`, `file`, `http`, `https`, `ws`, `wss` — which never described nxdns and -cannot go stale with it. Conflating the two was the error, and knowingly shipping -a diagnostic that contradicts its own type's contract is the tech debt this -project does not take on. +That wording defect was first recorded rather than fixed, on the grounds that telling the two apart needs a scheme list and a scheme list is what the previous round had just removed. Round nine rejected that and was right to: the list round six removed described *what nxdns supports*, so it went stale whenever a transport was added. WHATWG's special schemes are a closed set fixed by the URL Standard — `ftp`, `file`, `http`, `https`, `ws`, `wss` — which never described nxdns and cannot go stale with it. Conflating the two was the error, and knowingly shipping a diagnostic that contradicts its own type's contract is the tech debt this project does not take on. -So `redact` now distinguishes them. Only a special scheme can disagree with RFC -3986 about text no `//` introduced, so `https:hunter2` is withheld as contested -and `localhost:8080`, `mailto:ops@example.com` and `tls:\\host` are withheld as -naming no authority at all. Round nine's other finding was the same rule missing -from the schemeless branch: `\\lists.example\path` and `/\lists.example/path` were -reported as absent when they are contested. A run is settled when it is one -separator, or when the reading that looks for a host finds none — -`\\?\C:\lists\hosts.txt` ends its authority at the `?` under both — and contested -otherwise. +So `redact` now distinguishes them. Only a special scheme can disagree with RFC 3986 about text no `//` introduced, so `https:hunter2` is withheld as contested and `localhost:8080`, `mailto:ops@example.com` and `tls:\\host` are withheld as naming no authority at all. Round nine's other finding was the same rule missing from the schemeless branch: `\\lists.example\path` and `/\lists.example/path` were reported as absent when they are contested. A run is settled when it is one separator, or when the reading that looks for a host finds none — `\\?\C:\lists\hosts.txt` ends its authority at the `?` under both — and contested otherwise. -Round ten returned nothing critical and nothing important, four minor findings and -a nit. Two were taken: the nit, which was a comment claiming -`https:\\dns.nextdns.io\abcd12` "names a host" when the file's own reasoning is -that it is contested; and the rendering question — `tls:\\host` prints `tls://` -although the input held no `//`. That one is now contractual rather than -accidental. `format` renders `scheme://authority` canonically and does not quote -the input's syntax; nothing about which bytes separated the scheme survives -redaction, and nothing should, because the input is not meant to be reconstructible -from the output. +Round ten returned nothing critical and nothing important, four minor findings and a nit. Two were taken: the nit, which was a comment claiming `https:\\dns.nextdns.io\abcd12` "names a host" when the file's own reasoning is that it is contested; and the rendering question — `tls:\\host` prints `tls://` although the input held no `//`. That one is now contractual rather than accidental. `format` renders `scheme://authority` canonically and does not quote the input's syntax; nothing about which bytes separated the scheme survives redaction, and nothing should, because the input is not meant to be reconstructible from the output. -**Three findings were declined, and the reason is checked rather than asserted.** -`https:/user@/x`, `\path@hunter2` and `file:secret` are classified as contested -when both readings in fact find no host. Running them shows why that is tolerable: -each prints `(ambiguous authority omitted)`, so each says *less* than it could and -none prints the path segment. The mechanisms are named in the file — emptiness -tested before userinfo removal, a leading run of one reaching the late-delimiter -rule, and `file` having its own WHATWG parsing states that `isSpecialScheme` does -not model. Fixing them means modelling more of two standards for inputs no -accepted configuration can hold: every url this program takes carries `http`, -`https`, `tls`, `udp` or `tcp` and a `//`. All three are pinned by a test, because -the failure that would matter is the opposite one — if any of them ever starts -naming a host, that test fails. +**Three findings were declined, and the reason is checked rather than asserted.** `https:/user@/x`, `\path@hunter2` and `file:secret` are classified as contested when both readings in fact find no host. Running them shows why that is tolerable: each prints `(ambiguous authority omitted)`, so each says *less* than it could and none prints the path segment. The mechanisms are named in the file — emptiness tested before userinfo removal, a leading run of one reaching the late-delimiter rule, and `file` having its own WHATWG parsing states that `isSpecialScheme` does not model. Fixing them means modelling more of two standards for inputs no accepted configuration can hold: every url this program takes carries `http`, `https`, `tls`, `udp` or `tcp` and a `//`. All three are pinned by a test, because the failure that would matter is the opposite one — if any of them ever starts naming a host, that test fails. -That is where the review loop was stopped, on a judgement rather than on an empty -round. Rounds eight, nine and ten each returned nothing critical and nothing -important, and the findings had moved from "this prints a secret" to "this claims -more than it knows about a url no operator can configure". Round ten opened a new -sub-class of its own — WHATWG's `file:` state machine — which is the signal that -the remaining work is unbounded and no longer about safety. +That is where the review loop was stopped, on a judgement rather than on an empty round. Rounds eight, nine and ten each returned nothing critical and nothing important, and the findings had moved from "this prints a secret" to "this claims more than it knows about a url no operator can configure". Round ten opened a new sub-class of its own — WHATWG's `file:` state machine — which is the signal that the remaining work is unbounded and no longer about safety. -Ten rounds, sixteen findings, one function. A sweep of twenty-five shapes carrying -`hunter2`, `abcd12`, `s3cr3t`, `token`, `pa55`, `123456`, `hosts.txt` and -`example.com` finds none of them in any output. +Ten rounds, sixteen findings, one function. A sweep of twenty-five shapes carrying `hunter2`, `abcd12`, `s3cr3t`, `token`, `pa55`, `123456`, `hosts.txt` and `example.com` finds none of them in any output. -The last gap was found by sweeping every log site in the tree rather than -trusting the review's file list: `src/upstream/dot_client.zig` printed the -upstream url whole at four sites and `pool.zig` at a fifth. Three review rounds -and four agents had redacted urls across `manager.zig`, `validate.zig`, `cli.zig` -and `app.zig` without anyone asking which other modules logged the same values. +The last gap was found by sweeping every log site in the tree rather than trusting the review's file list: `src/upstream/dot_client.zig` printed the upstream url whole at four sites and `pool.zig` at a fifth. Three review rounds and four agents had redacted urls across `manager.zig`, `validate.zig`, `cli.zig` and `app.zig` without anyone asking which other modules logged the same values. -Deliberately not changed: `POST /api/blocklists` still ignores the -`SourceInNoGroup` warning. The web flow creates a source before any group link -can exist, so the warning is structural at that moment and a 400 would be wrong. -The three CLI paths carry it instead. +Deliberately not changed: `POST /api/blocklists` still ignores the `SourceInNoGroup` warning. The web flow creates a source before any group link can exist, so the warning is structural at that moment and a 400 would be wrong. The three CLI paths carry it instead. -Claims in the wave that no test backs, recorded rather than buried: the two -`app.zig` `log.warn` redactions, because a `std.log` line is not observable from -a unit test under the default runner; `cli.zig`'s `not a usable DoH url` -redaction, argued unreachable by any credential-carrying url because -`Endpoint.parse` rejects `@?#` in the authority first; the two probe lines that -need a reachable upstream; and the WAL guard's residual window, which is argued -rather than observed. +Claims in the wave that no test backs, recorded rather than buried: the two `app.zig` `log.warn` redactions, because a `std.log` line is not observable from a unit test under the default runner; `cli.zig`'s `not a usable DoH url` redaction, argued unreachable by any credential-carrying url because `Endpoint.parse` rejects `@?#` in the authority first; the two probe lines that need a reachable upstream; and the WAL guard's residual window, which is argued rather than observed. ## Round four -The `/metrics` family was the last place a credential still reached an -unauthenticated reader. `GET /metrics` is `.auth = .open` in `src/web/routes.zig` -and `web.bind` defaults to `0.0.0.0`, so an unauthenticated `curl` printed -`nxdns_upstream_up{url="https://dns.nextdns.io/abcd12"} 1` with HTTP 200. Every -other redaction in this wave was on a path that already required a session or a -shell on the host; this one was not. +The `/metrics` family was the last place a credential still reached an unauthenticated reader. `GET /metrics` is `.auth = .open` in `src/web/routes.zig` and `web.bind` defaults to `0.0.0.0`, so an unauthenticated `curl` printed `nxdns_upstream_up{url="https://dns.nextdns.io/abcd12"} 1` with HTTP 200. Every other redaction in this wave was on a path that already required a session or a shell on the host; this one was not. -Redacting it needed a second escaping layer rather than a call to `redact`. -`redact` cuts the authority at `/@?#\` and escapes every control byte, so it can -emit neither a raw newline nor a lone backslash — but `"` is in none of those cut -sets, so `https://ho"st/x` arrives at the label as `https://ho"st` and closes the -label value, letting the rest of the string write label pairs of its own. -`writeLabelValue` therefore runs *over* the redacted text. The ordering also -matters in the other direction: doubling the backslash turns the two characters -`redact` writes for a control byte into an unambiguous `\\n`, so a parser reads a -backslash rather than a newline. +Redacting it needed a second escaping layer rather than a call to `redact`. `redact` cuts the authority at `/@?#\` and escapes every control byte, so it can emit neither a raw newline nor a lone backslash — but `"` is in none of those cut sets, so `https://ho"st/x` arrives at the label as `https://ho"st` and closes the label value, letting the rest of the string write label pairs of its own. `writeLabelValue` therefore runs *over* the redacted text. The ordering also matters in the other direction: doubling the backslash turns the two characters `redact` writes for a control byte into an unambiguous `\\n`, so a parser reads a backslash rather than a newline. -The redaction then introduced a defect of its own, caught before it shipped: two -upstreams on one host redact to one label set, so `nxdns_upstream_up` rendered -twice with identical labels in a single exposition. Two NextDNS profiles is a -realistic configuration. The fix labels each sample with its array position in -`renderUpstreams`, so uniqueness holds by construction and a hand-built `Sample` -cannot forge a collision. The index is positional and therefore stable only while -the pool order is — `pool.Snapshot` carries no row id, and threading one through -the pool and the repository was judged out of scope for a review round. The -caveat is in the file. It only bites when two upstreams share an origin; for -distinct origins the url carries the identity and a reorder is harmless. +The redaction then introduced a defect of its own, caught before it shipped: two upstreams on one host redact to one label set, so `nxdns_upstream_up` rendered twice with identical labels in a single exposition. Two NextDNS profiles is a realistic configuration. The fix labels each sample with its array position in `renderUpstreams`, so uniqueness holds by construction and a hand-built `Sample` cannot forge a collision. The index is positional and therefore stable only while the pool order is — `pool.Snapshot` carries no row id, and threading one through the pool and the repository was judged out of scope for a review round. The caveat is in the file. It only bites when two upstreams share an origin; for distinct origins the url carries the identity and a reorder is harmless. -Claim scoping on that defect, since the three statements are not one: the -duplicate exposition text **was** reproduced, out of the same `render` function -`/metrics` calls. It was **not** reproduced over HTTP from a pre-fix server — every -live run had the index in place. And "a scrape carrying it is rejected" is -**reasoned, not observed**; nothing was fed to a Prometheus. Whether a scraper -rejects the pair or keeps the last sample, the down upstream is lost, which is the -part the fix rests on. +Claim scoping on that defect, since the three statements are not one: the duplicate exposition text **was** reproduced, out of the same `render` function `/metrics` calls. It was **not** reproduced over HTTP from a pre-fix server — every live run had the index in place. And "a scrape carrying it is rejected" is **reasoned, not observed**; nothing was fed to a Prometheus. Whether a scraper rejects the pair or keeps the last sample, the down upstream is lost, which is the part the fix rests on. -`src/storage/repositories/context.zig` printed a blocklist url whole and a group -name raw on its two `NotFound` paths. Both are invariant-failure paths that fire -rarely, which is why they outlived three rounds of sweeping. +`src/storage/repositories/context.zig` printed a blocklist url whole and a group name raw on its two `NotFound` paths. Both are invariant-failure paths that fire rarely, which is why they outlived three rounds of sweeping. -One test file was never running. `src/config/faults.zig` was absent from -`src/tests.zig`, and Zig collects tests only from the root module's explicit -import list — a transitively imported file contributes none, even when its values -are used, which was confirmed against a scratch project rather than assumed. The -five reflection guards in that file are what catch a future `ValidateError` -variant being added and left unclassified; they had never executed. Its runtime -behaviour was covered — `cli.zig` has a test that classifies through it — so the -gap was in the regression guard, not in what shipped. A sweep of the tree found no -other unlisted file with tests; `dns/dns.zig` is a re-export barrel with none, and -its own comment documents this rule. +One test file was never running. `src/config/faults.zig` was absent from `src/tests.zig`, and Zig collects tests only from the root module's explicit import list — a transitively imported file contributes none, even when its values are used, which was confirmed against a scratch project rather than assumed. The five reflection guards in that file are what catch a future `ValidateError` variant being added and left unclassified; they had never executed. Its runtime behaviour was covered — `cli.zig` has a test that classifies through it — so the gap was in the regression guard, not in what shipped. A sweep of the tree found no other unlisted file with tests; `dns/dns.zig` is a re-export barrel with none, and its own comment documents this rule. ## Acceptance (milestone complete) @@ -592,9 +223,7 @@ its own comment documents this rule. ## Anti-requirements -- No new documentation subjects (no metrics-families reference, no security-model - page) — this milestone moves and splits existing content plus the tutorial. +- No new documentation subjects (no metrics-families reference, no security-model page) — this milestone moves and splits existing content plus the tutorial. - No redirect stubs or `docs/legacy/`. -- No source fixes; report discrepancies instead. (Held for the documentation - wave. Superseded by the fix wave above, which closes the nine it reported.) +- No source fixes; report discrepancies instead. (Held for the documentation wave. Superseded by the fix wave above, which closes the nine it reported.) - No doc generator, no site builder, no MkDocs. diff --git a/specs/milestone-14.md b/specs/milestone-14.md index 8f2ff6f..0b19f47 100644 --- a/specs/milestone-14.md +++ b/specs/milestone-14.md @@ -1,9 +1,6 @@ # Milestone 14: build, package and publish releases -Goal: turn a signed git tag into a published, verifiable release — two static -musl tarballs and one multi-architecture container image on the self-hosted -Gitea at `git.mial.net`, with a licence, third-party notices, a changelog, a -detached signature, and documentation that leads with download-and-verify. +Goal: turn a signed git tag into a published, verifiable release — two static musl tarballs and one multi-architecture container image on the self-hosted Gitea at `git.mial.net`, with a licence, third-party notices, a changelog, a detached signature, and documentation that leads with download-and-verify. First tag: `v0.0.1`. @@ -11,89 +8,49 @@ First tag: `v0.0.1`. ### 1. PLAN.md is amended first -`PLAN.md:41` lists "Prebuilt binaries / published Docker images / project -website" under §2.2 *Out of Scope (permanent scope decisions, not deferrals)*. -`AGENTS.md` makes PLAN the source of truth, so this milestone is invalid until -that line is amended. The orchestrator edits PLAN before any session starts: +`PLAN.md:41` lists "Prebuilt binaries / published Docker images / project website" under §2.2 *Out of Scope (permanent scope decisions, not deferrals)*. `AGENTS.md` makes PLAN the source of truth, so this milestone is invalid until that line is amended. The orchestrator edits PLAN before any session starts: -- §2.2: remove prebuilt binaries and published container images. **The project - website stays out of scope.** -- Add a §on publication: tag-triggered releases, the artifact set, the signing - model, and the two deferrals in ruling 12. -- `PLAN.md:571`: drop "build date" from the `nxdns version` line. The version - string and the git commit identify a build exactly, and a date is one more - input that a reproducible build would have to pin. `src/version.zig` is - already correct; PLAN was wrong. -- `PLAN.md:638`: state exact byte limits — 15,728,640 with embedded assets, - 10,485,760 without — resolving milestone-13 discrepancy 9 in favour of what - CI already asserts. +- §2.2: remove prebuilt binaries and published container images. **The project website stays out of scope.** +- Add a §on publication: tag-triggered releases, the artifact set, the signing model, and the two deferrals in ruling 12. +- `PLAN.md:571`: drop "build date" from the `nxdns version` line. The version string and the git commit identify a build exactly, and a date is one more input that a reproducible build would have to pin. `src/version.zig` is already correct; PLAN was wrong. +- `PLAN.md:638`: state exact byte limits — 15,728,640 with embedded assets, 10,485,760 without — resolving milestone-13 discrepancy 9 in favour of what CI already asserts. ### 2. The version lives in the tag, and in exactly one other place -The tag is authoritative. `v0.0.1` means `-Dversion-string=0.0.1`; -`-Dgit-commit` is the tag's peeled commit. +The tag is authoritative. `v0.0.1` means `-Dversion-string=0.0.1`; `-Dgit-commit` is the tag's peeled commit. -`build.zig.zon:3` holds `.version`, which Zig requires and nothing reads. It is -bumped in the commit before each tag, and `verify-dist` fails when it disagrees -with the version under build. Two strings, one assert, no third copy anywhere. +`build.zig.zon:3` holds `.version`, which Zig requires and nothing reads. It is bumped in the commit before each tag, and `verify-dist` fails when it disagrees with the version under build. Two strings, one assert, no third copy anywhere. -Tags are **never moved**. A tag that produced a bad release is abandoned; the -fix ships as the next patch version. Only `vMAJOR.MINOR.PATCH` is accepted — -the release workflow rejects any tag carrying a pre-release suffix. +Tags are **never moved**. A tag that produced a bad release is abandoned; the fix ships as the next patch version. Only `vMAJOR.MINOR.PATCH` is accepted — the release workflow rejects any tag carrying a pre-release suffix. ### 3. Licence and third-party notices -`LICENSE` holds the English text of EUPL-1.2 verbatim, with -`Copyright (c) 2026 Mokhtar Mial`. `README.md` gains a short notice naming the -licence and the SPDX identifier `EUPL-1.2`. **No per-file SPDX headers.** +`LICENSE` holds the English text of EUPL-1.2 verbatim, with `Copyright (c) 2026 Mokhtar Mial`. `README.md` gains a short notice naming the licence and the SPDX identifier `EUPL-1.2`. **No per-file SPDX headers.** -`THIRD-PARTY-NOTICES` is assembled by `zig build dist` from a committed, -reviewed inventory under `licenses/`. It is not scraped from the dependency -tree at build time: a generated notices file that nobody reads rots silently -into a false statement. +`THIRD-PARTY-NOTICES` is assembled by `zig build dist` from a committed, reviewed inventory under `licenses/`. It is not scraped from the dependency tree at build time: a generated notices file that nobody reads rots silently into a false statement. -The inventory must cover everything the shipped artifacts actually contain, not -the direct dependency list: +The inventory must cover everything the shipped artifacts actually contain, not the direct dependency list: - **musl libc** (MIT) — statically linked into every binary. - **Zig standard library and compiler-rt** (MIT) — likewise. - **SQLite 3.53.4** — public domain, no obligation, listed for completeness. -- **Mbed TLS 3.6.7** — with an explicit line recording that it is taken under - the Apache-2.0 option of its dual `Apache-2.0 OR GPL-2.0-or-later` licence, - followed by the **full Apache-2.0 text**. Naming the choice is good practice; - shipping the text is the actual obligation. -- **Project Everest and p256-m** — compiled in at `build.zig:394` even though - the stock config leaves both drivers disabled. -- **The web bundle's runtime closure** — the transitive set, not the four direct - entries in `web/package.json`. Tailwind is a devDependency whose generated CSS - ships, so "production dependencies" understates it. +- **Mbed TLS 3.6.7** — with an explicit line recording that it is taken under the Apache-2.0 option of its dual `Apache-2.0 OR GPL-2.0-or-later` licence, followed by the **full Apache-2.0 text**. Naming the choice is good practice; shipping the text is the actual obligation. +- **Project Everest and p256-m** — compiled in at `build.zig:394` even though the stock config leaves both drivers disabled. +- **The web bundle's runtime closure** — the transitive set, not the four direct entries in `web/package.json`. Tailwind is a devDependency whose generated CSS ships, so "production dependencies" understates it. -A CI guard records the identity of the dependency sets (`build.zig.zon` -dependencies, and the npm closure) and fails when either changes without a -matching change under `licenses/`. It detects drift; it does not derive the -inventory. +A CI guard records the identity of the dependency sets (`build.zig.zon` dependencies, and the npm closure) and fails when either changes without a matching change under `licenses/`. It detects drift; it does not derive the inventory. -The **container image carries `/LICENSE` and `/THIRD-PARTY-NOTICES` too**. -Distributing the image is distribution, and the obligations do not live in the -tarball. +The **container image carries `/LICENSE` and `/THIRD-PARTY-NOTICES` too**. Distributing the image is distribution, and the obligations do not live in the tarball. ### 4. `zig build dist` replaces `zig build cross` -The `cross` step is deleted. `dist` is the single command that produces -everything releasable, and it runs on a laptop exactly as it runs on the runner. +The `cross` step is deleted. `dist` is the single command that produces everything releasable, and it runs on a laptop exactly as it runs on the runner. -Inputs: `-Dversion-string` (required — no default), `-Dgit-commit`, -`-Dweb-dist`, `-Doptimize=ReleaseSafe`. +Inputs: `-Dversion-string` (required — no default), `-Dgit-commit`, `-Dweb-dist`, `-Doptimize=ReleaseSafe`. -**`dist` fails when `-Dweb-dist` resolves to `web/dist-placeholder`.** The -default at `build.zig:24` is the placeholder, so a release built without the -flag would silently ship a placeholder admin page. There is no override flag; -an escape hatch here is a foot-gun with a safety label on it. +**`dist` fails when `-Dweb-dist` resolves to `web/dist-placeholder`.** The default at `build.zig:24` is the placeholder, so a release built without the flag would silently ship a placeholder admin page. There is no override flag; an escape hatch here is a foot-gun with a safety label on it. -Per triple (`x86_64-linux-musl`, `aarch64-linux-musl`), `dist` builds -ReleaseSafe with `.linkage = .static` and `.strip = true`. Stripping uses -`std.Build.Module.strip` (`-fstrip`, `Module.zig:545`), which removes the -`objcopy` and `binutils-aarch64-linux-gnu` dependency from the runner. +Per triple (`x86_64-linux-musl`, `aarch64-linux-musl`), `dist` builds ReleaseSafe with `.linkage = .static` and `.strip = true`. Stripping uses `std.Build.Module.strip` (`-fstrip`, `Module.zig:545`), which removes the `objcopy` and `binutils-aarch64-linux-gnu` dependency from the runner. It stages one directory per triple, `nxdns--/`, holding: @@ -106,9 +63,7 @@ It stages one directory per triple, `nxdns--/`, holding: | `THIRD-PARTY-NOTICES` | 0644 | | `INSTALL.md` | 0644 | -`deploy/systemd/sysusers.conf` is renamed to `nxdns.conf` — the name it is -installed under at `/usr/lib/sysusers.d/nxdns.conf`. A file that changes name -during install is a step an operator can get wrong. +`deploy/systemd/sysusers.conf` is renamed to `nxdns.conf` — the name it is installed under at `/usr/lib/sysusers.d/nxdns.conf`. A file that changes name during install is a step an operator can get wrong. Archiving runs as two `b.addSystemCommand` steps, never one: @@ -118,270 +73,130 @@ tar --format=gnu --sort=name --mtime=@0 --owner=0 --group=0 \ gzip -n -9 .tar ``` -`b.addSystemCommand` executes argv directly and does not interpret `|`, and a -shell wrapper without `pipefail` would report only `gzip`'s exit status while a -failed `tar` passed silently. `gzip -n` is required because `--mtime=@0` -normalises the tar member times but not the timestamp gzip writes into its own -header. The environment sets `LC_ALL=C` and `TZ=UTC`. +`b.addSystemCommand` executes argv directly and does not interpret `|`, and a shell wrapper without `pipefail` would report only `gzip`'s exit status while a failed `tar` passed silently. `gzip -n` is required because `--mtime=@0` normalises the tar member times but not the timestamp gzip writes into its own header. The environment sets `LC_ALL=C` and `TZ=UTC`. -Both steps declare the staging directory as a build input and the archive as an -output, so Zig's cache cannot serve a stale artifact. +Both steps declare the staging directory as a build input and the archive as an output, so Zig's cache cannot serve a stale artifact. -`dist` also emits a checksum file covering **only the two tarballs**. It cannot -cover the image: the digest does not exist until buildx has pushed, which -happens later and elsewhere. The release job appends that line (ruling 7). +`dist` also emits a checksum file covering **only the two tarballs**. It cannot cover the image: the digest does not exist until buildx has pushed, which happens later and elsewhere. The release job appends that line (ruling 7). ### 5. `zig build verify-dist` asserts what CI shell asserts today -The 50 lines of shell at `ci.yml:129` and `:168` move into a build step, so a -developer can run the release checks on a laptop. Logic that only runs in CI is -the brittleness this milestone exists to remove. +The 50 lines of shell at `ci.yml:129` and `:168` move into a build step, so a developer can run the release checks on a laptop. Logic that only runs in CI is the brittleness this milestone exists to remove. It operates on the **extracted** archive, not the staging directory: -- ELF header: correct `e_machine` per triple, **no `PT_INTERP`, no `DT_NEEDED`**. - Matching the string `statically linked` from `file(1)` is not a static-linkage - test. +- ELF header: correct `e_machine` per triple, **no `PT_INTERP`, no `DT_NEEDED`**. Matching the string `statically linked` from `file(1)` is not a static-linkage test. - Stripped binary ≤ 15,728,640 bytes. -- Archive layout: exactly one top-level directory, the exact file allowlist from - ruling 4, expected modes, no symlinks, no path traversal. -- `nxdns version` prints the version under build and the git commit. Native - architecture only — the aarch64 binary needs qemu and is skipped without - `-fqemu`. +- Archive layout: exactly one top-level directory, the exact file allowlist from ruling 4, expected modes, no symlinks, no path traversal. +- `nxdns version` prints the version under build and the git commit. Native architecture only — the aarch64 binary needs qemu and is skipped without `-fqemu`. - `build.zig.zon` `.version` equals the version under build. -The asset-free budget (10,485,760 bytes) gets **its own build against a -generated empty assets directory**, not against the placeholder. Ruling 4 makes -the placeholder unbuildable, and re-admitting it through a back door for one -size check would defeat the point. +The asset-free budget (10,485,760 bytes) gets **its own build against a generated empty assets directory**, not against the placeholder. Ruling 4 makes the placeholder unbuildable, and re-admitting it through a back door for one size check would defeat the point. -**A second `zig build` invocation does not inherit the first one's `-D` options.** -`zig build dist -Dversion-string=X` followed by a bare `zig build verify-dist` -verifies a *differently configured* build. Every caller — the workflows, the -documentation, and this spec's own acceptance list — passes the same -`-Dversion-string`, `-Dgit-commit`, `-Dweb-dist` and `-Doptimize` to both. +**A second `zig build` invocation does not inherit the first one's `-D` options.** `zig build dist -Dversion-string=X` followed by a bare `zig build verify-dist` verifies a *differently configured* build. Every caller — the workflows, the documentation, and this spec's own acceptance list — passes the same `-Dversion-string`, `-Dgit-commit`, `-Dweb-dist` and `-Doptimize` to both. ### 6. Container image `deploy/docker/Dockerfile`: -- Pin the base by digest: - `alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce`. -- Add `--platform=$BUILDPLATFORM` to the builder stage. It only copies files, so - pinning it to the build host means the arm64 image needs no qemu. -- **Delete `apk add --no-cache ca-certificates` (`Dockerfile:15`).** Verified: - the Alpine base already ships `/etc/ssl/certs/ca-certificates.crt` (179,359 - bytes) from `ca-certificates-bundle`. Removing it removes the only network - fetch in the image build. -- Keep the `mkdir` and `chown 65532` — `/var/lib/nxdns` must exist with that - ownership so Docker copies it onto a fresh named volume. +- Pin the base by digest: `alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce`. +- Add `--platform=$BUILDPLATFORM` to the builder stage. It only copies files, so pinning it to the build host means the arm64 image needs no qemu. +- **Delete `apk add --no-cache ca-certificates` (`Dockerfile:15`).** Verified: the Alpine base already ships `/etc/ssl/certs/ca-certificates.crt` (179,359 bytes) from `ca-certificates-bundle`. Removing it removes the only network fetch in the image build. +- Keep the `mkdir` and `chown 65532` — `/var/lib/nxdns` must exist with that ownership so Docker copies it onto a fresh named volume. - Add `/LICENSE` and `/THIRD-PARTY-NOTICES` (ruling 3). -- OCI labels: `source`, `revision`, `version`, `licenses=EUPL-1.2`, `created`, - `title`, `description`. +- OCI labels: `source`, `revision`, `version`, `licenses=EUPL-1.2`, `created`, `title`, `description`. -`deploy/docker/.dockerignore` is renamed to `deploy/docker/Dockerfile.dockerignore`. -The build context is the repository root, so Docker never reads the current -path, and the whole worktree — `.git`, `node_modules`, `zig-cache` — is being -sent to the daemon today. +`deploy/docker/.dockerignore` is renamed to `deploy/docker/Dockerfile.dockerignore`. The build context is the repository root, so Docker never reads the current path, and the whole worktree — `.git`, `node_modules`, `zig-cache` — is being sent to the daemon today. -buildx runs with `--provenance=false --sbom=false`. Recent buildx adds -provenance attestations by default, which create `unknown/unknown` platform -entries and change the index digest; Gitea's OCI 1.1 support is unverified -(go-gitea#25846) and this is not the milestone to find out. `SOURCE_DATE_EPOCH` -comes from the tag date. +buildx runs with `--provenance=false --sbom=false`. Recent buildx adds provenance attestations by default, which create `unknown/unknown` platform entries and change the index digest; Gitea's OCI 1.1 support is unverified (go-gitea#25846) and this is not the milestone to find out. `SOURCE_DATE_EPOCH` comes from the tag date. -`deploy/docker/compose.yaml` references the published image rather than -building one. +`deploy/docker/compose.yaml` references the published image rather than building one. -**Verify before publishing:** the binary inside each image is byte-identical to -the binary in the matching tarball. +**Verify before publishing:** the binary inside each image is byte-identical to the binary in the matching tarball. ### 7. Workflows -`.gitea/workflows/gates.yml`, `on: workflow_call`, holds every blocking check: -`test` (with `-Dintegration`), `test-aarch64` (`-fqemu`), `frontend`, `package` -(`dist` + `verify-dist` + the asset-free size check) and `container` (build the -image and run the existing smoke test from `ci.yml:219`). Moving only the three -test jobs would drop the packaging and container checks precisely when they -matter most. +`.gitea/workflows/gates.yml`, `on: workflow_call`, holds every blocking check: `test` (with `-Dintegration`), `test-aarch64` (`-fqemu`), `frontend`, `package` (`dist` + `verify-dist` + the asset-free size check) and `container` (build the image and run the existing smoke test from `ci.yml:219`). Moving only the three test jobs would drop the packaging and container checks precisely when they matter most. -`ci.yml` calls it on `push` and `pull_request` for **`master`**. `master` is -canonical: `origin/main` is deleted, and Gitea's default branch is changed to -match. Today `ci.yml:5` watches `main` while work happens on `master`, so -milestone 13 has never run through CI. +`ci.yml` calls it on `push` and `pull_request` for **`master`**. `master` is canonical: `origin/main` is deleted, and Gitea's default branch is changed to match. Today `ci.yml:5` watches `main` while work happens on `master`, so milestone 13 has never run through CI. `.gitea/workflows/release.yml`, `on: push: tags: ['v*']`, in this order: -1. Checkout with `fetch-depth: 0` and tags. The default shallow clone breaks - ancestry checks, previous-tag lookup and changelog generation. +1. Checkout with `fetch-depth: 0` and tags. The default shallow clone breaks ancestry checks, previous-tag lookup and changelog generation. 2. Reject any tag that is not exactly `vMAJOR.MINOR.PATCH`. -3. `git verify-tag`, requiring an annotated tag and requiring the signature's - fingerprint to equal a fingerprint pinned in the workflow. A bare - `verify-tag` proves only that *some* imported key signed it. +3. `git verify-tag`, requiring an annotated tag and requiring the signature's fingerprint to equal a fingerprint pinned in the workflow. A bare `verify-tag` proves only that *some* imported key signed it. 4. Assert the tag's commit is an ancestor of `origin/master`. 5. Assert no published release exists for this tag. Delete any leftover draft. -6. Assert the version is greater than the highest published release version, so - a late-finishing older tag cannot move `latest` backwards. +6. Assert the version is greater than the highest published release version, so a late-finishing older tag cannot move `latest` backwards. 7. Gates, blocking, via `needs:`. 8. Build the web UI; `zig build dist`; `zig build verify-dist`. 9. Extract the `CHANGELOG.md` section matching the version. **Fail when absent.** -10. buildx build and push **`:` only**. Capture the index digest from - `--metadata-file`, validate it as `sha256:<64 hex>`, and confirm the pushed - tag resolves to that digest with exactly the two intended platforms. -11. Write the digest file, append its checksum line, verify the assembled - checksum file. -12. Sign the checksum file (ruling 8). Verify the signature locally before - uploading it. +10. buildx build and push **`:` only**. Capture the index digest from `--metadata-file`, validate it as `sha256:<64 hex>`, and confirm the pushed tag resolves to that digest with exactly the two intended platforms. +11. Write the digest file, append its checksum line, verify the assembled checksum file. +12. Sign the checksum file (ruling 8). Verify the signature locally before uploading it. 13. Create the release as a **draft**; upload the assets. -14. Move `:latest` and verify it resolves to the built digest. Re-check the - monotonic-version invariant here: step 6 ran before the gates, and proves - nothing about which of two in-flight tags finishes last. -15. **Publish the draft last.** Publication is the one irreversible act, so it - goes after everything that can still fail. +14. Move `:latest` and verify it resolves to the built digest. Re-check the monotonic-version invariant here: step 6 ran before the gates, and proves nothing about which of two in-flight tags finishes last. +15. **Publish the draft last.** Publication is the one irreversible act, so it goes after everything that can still fail. -Amended after review. The original order published at 14 and moved `:latest` at -15, which deadlocks: a failure while moving `:latest` leaves a published release, -and ruling 9 makes a re-run refuse a published release. Nothing could repair it. -The cost of the corrected order is a short window where `:latest` serves the new -image before the release page is public. That is recoverable by a re-run; the -deadlock was not. +Amended after review. The original order published at 14 and moved `:latest` at 15, which deadlocks: a failure while moving `:latest` leaves a published release, and ruling 9 makes a re-run refuse a published release. Nothing could repair it. The cost of the corrected order is a short window where `:latest` serves the new image before the release page is public. That is recoverable by a re-run; the deadlock was not. -Every action in `release.yml` is pinned to a full commit SHA. -`actions/checkout@v4` and `mlugg/setup-zig@v2` are mutable tags on another -party's server, and a compromise upstream would run on the runner holding the -signing subkey and the registry token. `ci.yml` may keep moving tags; it holds -no secrets. +Every action in `release.yml` is pinned to a full commit SHA. `actions/checkout@v4` and `mlugg/setup-zig@v2` are mutable tags on another party's server, and a compromise upstream would run on the runner holding the signing subkey and the registry token. `ci.yml` may keep moving tags; it holds no secrets. ### 8. Signing **Two different keys are involved, and the original ruling conflated them.** -- The **tag-signing key** is the human's. `git tag -s` uses it, and step 3 checks - it. What gets pinned in `release.yml` is the **primary certificate - fingerprint**, which `git verify-tag --raw` emits as the **last** field of the - `VALIDSIG` line. Field 3 is whichever key actually made the signature — the - signing subkey once one exists. Pinning field 3 would mean that creating the - release subkey below silently blocks every future release, with an error - message that reads like a forged tag. Reproduced against a real keyring during - review. -- The **artifact-signing subkey** is the runner's, and it signs the checksum - file. It is a dedicated GPG signing subkey of the author's existing key. A - leaked subkey is revoked on its own; the identity, the commit signature history - and everyone's existing trust survive. +- The **tag-signing key** is the human's. `git tag -s` uses it, and step 3 checks it. What gets pinned in `release.yml` is the **primary certificate fingerprint**, which `git verify-tag --raw` emits as the **last** field of the `VALIDSIG` line. Field 3 is whichever key actually made the signature — the signing subkey once one exists. Pinning field 3 would mean that creating the release subkey below silently blocks every future release, with an error message that reads like a forged tag. Reproduced against a real keyring during review. +- The **artifact-signing subkey** is the runner's, and it signs the checksum file. It is a dedicated GPG signing subkey of the author's existing key. A leaked subkey is revoked on its own; the identity, the commit signature history and everyone's existing trust survive. -Pinning the primary fingerprint means adding or rotating a signing subkey is a -non-event for verification. +Pinning the primary fingerprint means adding or rotating a signing subkey is a non-event for verification. -Every required secret is validated in the **guard job**, before the gates and -before any registry push. Validating a fingerprint inside the signing step means -a placeholder value burns an immutable version tag before it fails. +Every required secret is validated in the **guard job**, before the gates and before any registry push. Validating a fingerprint inside the signing step means a placeholder value burns an immutable version tag before it fails. -Implementation requirements: a temporary `GNUPGHOME`; assert the imported -material contains no primary secret key; `--local-user !` -so GPG cannot fall back to another key; batch and loopback pinentry; verify the -produced signature before upload; scrub `GNUPGHOME` and `DOCKER_CONFIG` and kill -the agent on every exit path. The registry token is passed by -`--password-stdin` into a temporary `DOCKER_CONFIG`, and `persist-credentials` -is off. +Implementation requirements: a temporary `GNUPGHOME`; assert the imported material contains no primary secret key; `--local-user !` so GPG cannot fall back to another key; batch and loopback pinentry; verify the produced signature before upload; scrub `GNUPGHOME` and `DOCKER_CONFIG` and kill the agent on every exit path. The registry token is passed by `--password-stdin` into a temporary `DOCKER_CONFIG`, and `persist-credentials` is off. -Secrets: `RELEASE_GPG_SUBKEY`, `RELEASE_GPG_PASSPHRASE`, `REGISTRY_TOKEN`. The -built-in `GITEA_TOKEN` cannot publish to the package registry, which is why the -third exists; it can still create the release and upload assets. +Secrets: `RELEASE_GPG_SUBKEY`, `RELEASE_GPG_PASSPHRASE`, `REGISTRY_TOKEN`. The built-in `GITEA_TOKEN` cannot publish to the package registry, which is why the third exists; it can still create the release and upload assets. -**Stated honestly, in `docs/how-to/verify-a-release.md`:** the signature proves -the artifact came from this pipeline and reached the operator unaltered. It does -not prove the binary matches the source, because the machine that built it also -held the key. Ruling 12 defers the control that closes that gap. Do not write a -stronger claim than that. +**Stated honestly, in `docs/how-to/verify-a-release.md`:** the signature proves the artifact came from this pipeline and reached the operator unaltered. It does not prove the binary matches the source, because the machine that built it also held the key. Ruling 12 defers the control that closes that gap. Do not write a stronger claim than that. ### 9. Failure and recovery are specified, not improvised -- The draft is the unit of work for the *release record*. The registry is not - covered by it: the version tag becomes publicly pullable at step 10. Only the - release page and its assets stay hidden until step 15. -- The existence check happens **before** the push, not after. Gitea's container - tags are mutable — immutability is a workflow invariant, not a registry - guarantee — so a push-then-compare would already have overwritten the tag it - claims to refuse. Read the existing digest with - `HEAD /v2/mokhtar/nxdns/manifests/`, sending an `Accept` header for - the index media types and following the `401` bearer challenge with the PAT. - Absent is `404`; present returns `Docker-Content-Digest`. -- A re-run deletes an existing **draft** and repeats. It refuses to touch a - **published** release. -- The registry version tag is immutable, and the workflow never writes it twice. - A re-run that finds the tag present pushes nothing: it adopts the pushed - digest and asserts the *contents* of that image against the artifacts it just - built, which is the check that matters and the only one available. Comparing a - rebuilt index digest was the first design and is not implementable — buildx - cannot report an index digest without pushing, and cross-machine - reproducibility is deferred (ruling 12), so the rebuilt digest is expected to - differ even when nothing changed. See recorded deviation 10. -- `:latest` moves last, so a failure between the image push and publication - leaves the version tag pushed and `latest` untouched. That is recoverable by - re-running. -- If a tag is burned — the pipeline itself is broken and the fix is on `master` — - the release is abandoned and reissued as the next patch version. This is the - terminal path, and it must be written down before `v0.0.1`, not discovered - during it. +- The draft is the unit of work for the *release record*. The registry is not covered by it: the version tag becomes publicly pullable at step 10. Only the release page and its assets stay hidden until step 15. +- The existence check happens **before** the push, not after. Gitea's container tags are mutable — immutability is a workflow invariant, not a registry guarantee — so a push-then-compare would already have overwritten the tag it claims to refuse. Read the existing digest with `HEAD /v2/mokhtar/nxdns/manifests/`, sending an `Accept` header for the index media types and following the `401` bearer challenge with the PAT. Absent is `404`; present returns `Docker-Content-Digest`. +- A re-run deletes an existing **draft** and repeats. It refuses to touch a **published** release. +- The registry version tag is immutable, and the workflow never writes it twice. A re-run that finds the tag present pushes nothing: it adopts the pushed digest and asserts the *contents* of that image against the artifacts it just built, which is the check that matters and the only one available. Comparing a rebuilt index digest was the first design and is not implementable — buildx cannot report an index digest without pushing, and cross-machine reproducibility is deferred (ruling 12), so the rebuilt digest is expected to differ even when nothing changed. See recorded deviation 10. +- `:latest` moves last, so a failure between the image push and publication leaves the version tag pushed and `latest` untouched. That is recoverable by re-running. +- If a tag is burned — the pipeline itself is broken and the fix is on `master` — the release is abandoned and reissued as the next patch version. This is the terminal path, and it must be written down before `v0.0.1`, not discovered during it. ### 10. Release notes -`CHANGELOG.md` at the repository root, Keep a Changelog format, with an -`## [Unreleased]` section maintained as work happens. The release body is that -section, plus a generated appendix: `git log --oneline` since the previous tag -inside a collapsed `
`, a compare link, the tarball hashes and the image -digest. +`CHANGELOG.md` at the repository root, Keep a Changelog format, with an `## [Unreleased]` section maintained as work happens. The release body is that section, plus a generated appendix: `git log --oneline` since the previous tag inside a collapsed `
`, a compare link, the tarball hashes and the image digest. -The nullable value is "the previous reachable **published release**", never "the -previous git tag". An abandoned tag from ruling 9 must not become the comparison -base, and `git describe` would pick exactly that. +The nullable value is "the previous reachable **published release**", never "the previous git tag". An abandoned tag from ruling 9 must not become the comparison base, and `git describe` would pick exactly that. -With no published base — the first release, and equally the case where `v0.0.1` -was abandoned and `v0.0.2` becomes the first published one — the log is -`git log --oneline ` and the compare link is omitted. It is **not** a range -with an empty left side: `..v0.0.1` resolves against `HEAD` and produces a wrong -or empty appendix rather than "all history". +With no published base — the first release, and equally the case where `v0.0.1` was abandoned and `v0.0.2` becomes the first published one — the log is `git log --oneline ` and the compare link is omitted. It is **not** a range with an empty left side: `..v0.0.1` resolves against `HEAD` and produces a wrong or empty appendix rather than "all history". The `v0.0.1` section is hand-written. ### 11. Documentation -`install-with-systemd.md`, `install-with-docker.md` and `upgrade.md` lead with -download-and-verify; the existing build-from-source steps move to a later -section of the same page. New `docs/how-to/verify-a-release.md` gives the -verification commands and the rebuild recipe. +`install-with-systemd.md`, `install-with-docker.md` and `upgrade.md` lead with download-and-verify; the existing build-from-source steps move to a later section of the same page. New `docs/how-to/verify-a-release.md` gives the verification commands and the rebuild recipe. -**Milestone 13 ruling 3 applies unchanged**: every command block in `tutorial/` -and `how-to/` is executed on this host by the session that writes it, or marked -in-page as unverified with the reason. +**Milestone 13 ruling 3 applies unchanged**: every command block in `tutorial/` and `how-to/` is executed on this host by the session that writes it, or marked in-page as unverified with the reason. -Two rot hazards the drift test cannot see. `src/docs_drift_test.zig` guards only -`reference/{api,configuration,cli}.md`; the how-to pages are unguarded: +Two rot hazards the drift test cannot see. `src/docs_drift_test.zig` guards only `reference/{api,configuration,cli}.md`; the how-to pages are unguarded: -- The transcripts hardcode `nxdns 0.1.0-dev` in four pages. After `v0.0.1` they - are wrong and nothing fails. Use a version-neutral placeholder, and add a - guard that rejects a stale literal release version in the docs. -- A download URL with the version in the path goes stale at `v0.0.2`. Use a - `latest` download form if Gitea provides one, or a placeholder the reader - substitutes. Probe first (ruling 13); do not guess. +- The transcripts hardcode `nxdns 0.1.0-dev` in four pages. After `v0.0.1` they are wrong and nothing fails. Use a version-neutral placeholder, and add a guard that rejects a stale literal release version in the docs. +- A download URL with the version in the path goes stale at `v0.0.2`. Use a `latest` download form if Gitea provides one, or a placeholder the reader substitutes. Probe first (ruling 13); do not guess. -Sweep the whole repository for surfaces the rewrite would otherwise miss: -`README.md`, `Dockerfile` comments, `compose.yaml`, `explanation/performance-and-testing.md`, -and any remaining `zig build cross` or source-only-distribution text. +Sweep the whole repository for surfaces the rewrite would otherwise miss: `README.md`, `Dockerfile` comments, `compose.yaml`, `explanation/performance-and-testing.md`, and any remaining `zig build cross` or source-only-distribution text. ### 12. Deferred, deliberately Recorded in PLAN so they are decisions rather than oversights: -- **The reproducibility gate** — build twice in two directory paths and assert - identical hashes. Deferred until `v0.0.1` proves the pipeline. Until it exists, - no document may describe the build as reproducible. Do the cheap parts now - regardless: pin Node to an exact patch in `ci.yml` and `web/package.json` - (both float at `24` today), `gzip -n`, `LC_ALL=C`, `TZ=UTC`. -- **cosign signatures on the image** — Gitea Actions has no OIDC identity token, - so keyless signing is impossible and only a key-based signature is available. - Whether Gitea's registry accepts a cosign signature manifest is unverified. - Spike it before committing to it. +- **The reproducibility gate** — build twice in two directory paths and assert identical hashes. Deferred until `v0.0.1` proves the pipeline. Until it exists, no document may describe the build as reproducible. Do the cheap parts now regardless: pin Node to an exact patch in `ci.yml` and `web/package.json` (both float at `24` today), `gzip -n`, `LC_ALL=C`, `TZ=UTC`. +- **cosign signatures on the image** — Gitea Actions has no OIDC identity token, so keyless signing is impossible and only a key-based signature is available. Whether Gitea's registry accepts a cosign signature manifest is unverified. Spike it before committing to it. ### 13. Manual prerequisites, done before the first tag @@ -389,70 +204,37 @@ None of these are code, and all of them block `v0.0.1`: The order matters, and the original list got parts of it wrong. Corrected: -1. Create the artifact-signing subkey, export it with `--export-secret-subkeys`, - and publish the public key to `keys.openpgp.org`. Separately, identify which - key actually signs your tags — after step 1 that is normally the new signing - subkey, whose **primary** fingerprint is what gets pinned. -2. Pin the primary fingerprint in `release.yml`, replacing the placeholder. The - guard fails closed on the placeholder, so no tag can succeed before this. -3. **Store** the secrets: `RELEASE_GPG_SUBKEY`, `RELEASE_GPG_PASSPHRASE`, and a - registry personal access token with package write scope as `REGISTRY_TOKEN`. - Creating the token is not storing it. -4. `app.ini`: add `.asc` and `.sig` to `[attachment] ALLOWED_TYPES`, and raise - `MAX_FILES` from 5. Restart Gitea and confirm the *effective* settings through - `/api/v1/settings/attachment` before trusting them. Verified against the live - instance on 2026-08-05: `max_size` is 100 MB (ample), `max_files` is 5 (this - release has exactly five assets, no headroom), and `.asc` is absent, so the - signature is rejected today. The implementation names the assets - `SHA256SUMS.txt`, `SHA256SUMS.txt.asc` and `IMAGE-DIGEST.txt`, so `.txt` and - `.asc` are the two extensions that must be allowed. -5. **Probe: does the runner support `docker buildx` with the `docker-container` - driver, and can it extract a foreign-platform image?** The old `docker` job - only proved plain `docker build`, and the arm64 image-versus-tarball check - needs more than that. -6. Land the workflows on `master` **while `main` still exists**, and let the - reusable-gates workflow run green once, so Gitea emits its actual - status-check context names. -7. Configure branch protection on `master` using **those observed names**. The - reusable-workflow refactor changes them; keeping the old required checks can - make merges impossible or leave the intended gates non-required. +1. Create the artifact-signing subkey, export it with `--export-secret-subkeys`, and publish the public key to `keys.openpgp.org`. Separately, identify which key actually signs your tags — after step 1 that is normally the new signing subkey, whose **primary** fingerprint is what gets pinned. +2. Pin the primary fingerprint in `release.yml`, replacing the placeholder. The guard fails closed on the placeholder, so no tag can succeed before this. +3. **Store** the secrets: `RELEASE_GPG_SUBKEY`, `RELEASE_GPG_PASSPHRASE`, and a registry personal access token with package write scope as `REGISTRY_TOKEN`. Creating the token is not storing it. +4. `app.ini`: add `.asc` and `.sig` to `[attachment] ALLOWED_TYPES`, and raise `MAX_FILES` from 5. Restart Gitea and confirm the *effective* settings through `/api/v1/settings/attachment` before trusting them. Verified against the live instance on 2026-08-05: `max_size` is 100 MB (ample), `max_files` is 5 (this release has exactly five assets, no headroom), and `.asc` is absent, so the signature is rejected today. The implementation names the assets `SHA256SUMS.txt`, `SHA256SUMS.txt.asc` and `IMAGE-DIGEST.txt`, so `.txt` and `.asc` are the two extensions that must be allowed. +5. **Probe: does the runner support `docker buildx` with the `docker-container` driver, and can it extract a foreign-platform image?** The old `docker` job only proved plain `docker build`, and the arm64 image-versus-tarball check needs more than that. +6. Land the workflows on `master` **while `main` still exists**, and let the reusable-gates workflow run green once, so Gitea emits its actual status-check context names. +7. Configure branch protection on `master` using **those observed names**. The reusable-workflow refactor changes them; keeping the old required checks can make merges impossible or leave the intended gates non-required. 8. Change Gitea's default branch to `master`. 9. Delete `origin/main` **last** — Gitea refuses to delete the default branch. -10. Merge the licence, changelog and `build.zig.zon` version commit, and let - `master` go green. -11. Dry-run the release path. A tag-triggered workflow cannot be exercised - without pushing *some* tag, so use a disposable tag or a scratch repository. - **Never use `v0.0.1` as the dry run** and then expect to reuse it: tags are - never moved (ruling 2), so a burned dry-run tag is spent. +10. Merge the licence, changelog and `build.zig.zon` version commit, and let `master` go green. +11. Dry-run the release path. A tag-triggered workflow cannot be exercised without pushing *some* tag, so use a disposable tag or a scratch repository. **Never use `v0.0.1` as the dry run** and then expect to reuse it: tags are never moved (ruling 2), so a burned dry-run tag is spent. ## Sessions -S1–S5 run in parallel. S1 and S2 share one interface, fixed here so neither -blocks: S2 owns everything under `licenses/` and the licence texts; S1 owns the -build step that assembles them into `THIRD-PARTY-NOTICES`. The contract is -`zig build dist -Dversion-string=X -Dgit-commit=Y -Dweb-dist=web/dist` writing -to `zig-out/dist/`. +S1–S5 run in parallel. S1 and S2 share one interface, fixed here so neither blocks: S2 owns everything under `licenses/` and the licence texts; S1 owns the build step that assembles them into `THIRD-PARTY-NOTICES`. The contract is `zig build dist -Dversion-string=X -Dgit-commit=Y -Dweb-dist=web/dist` writing to `zig-out/dist/`. ### Session S1: build system -Owns `build.zig`. Rulings 2, 4, 5. Deletes `cross`, adds `dist` and -`verify-dist`, moves the CI shell asserts into the build graph. +Owns `build.zig`. Rulings 2, 4, 5. Deletes `cross`, adds `dist` and `verify-dist`, moves the CI shell asserts into the build graph. ### Session S2: licence, notices, changelog -Owns `LICENSE`, `licenses/`, `CHANGELOG.md`, `README.md` (notice and links). -Ruling 3, ruling 10. Compiles the inventory by auditing what the artifacts -actually contain, and writes the CI drift guard for the dependency sets. +Owns `LICENSE`, `licenses/`, `CHANGELOG.md`, `README.md` (notice and links). Ruling 3, ruling 10. Compiles the inventory by auditing what the artifacts actually contain, and writes the CI drift guard for the dependency sets. ### Session S3: container -Owns `deploy/docker/*`, `deploy/systemd/*` (the `sysusers.conf` rename). -Ruling 6. +Owns `deploy/docker/*`, `deploy/systemd/*` (the `sysusers.conf` rename). Ruling 6. ### Session S4: workflows -Owns `.gitea/workflows/*`. Rulings 7, 8, 9. Must not edit `build.zig` — it -consumes the step contract above. +Owns `.gitea/workflows/*`. Rulings 7, 8, 9. Must not edit `build.zig` — it consumes the step contract above. ### Session S5: documentation @@ -460,146 +242,41 @@ Owns `docs/**`. Ruling 11, and milestone 13 ruling 3. ### Orchestrator -`PLAN.md` (ruling 1), this spec, deletion of `origin/main`, the manual -prerequisites in ruling 13, and the `v0.0.1` tag. +`PLAN.md` (ruling 1), this spec, deletion of `origin/main`, the manual prerequisites in ruling 13, and the `v0.0.1` tag. ## Recorded (implementation) -Accepted deviations and corrections from integration. The five rulings amended -above (7, 8, 9, 10, 13, plus the option-inheritance note in 5 and the scoping of -the last acceptance line) were all wrong as first written; each amendment says -what it replaced and why. +Accepted deviations and corrections from integration. The five rulings amended above (7, 8, 9, 10, 13, plus the option-inheritance note in 5 and the scoping of the last acceptance line) were all wrong as first written; each amendment says what it replaced and why. -1. **`build.zig.zon` said `0.1.0`.** It traced to the first build-baseline - commit — a scaffold default never bumped. Ruling 2 makes `verify-dist` assert - it against the version under build, so the CI packaging gate could never have - passed. Set to `0.0.1`, matching the first tag. -2. **Assets carry a `.txt` extension**: `SHA256SUMS.txt`, `SHA256SUMS.txt.asc`, - `IMAGE-DIGEST.txt`. This resolves ruling 13's extensionless-asset probe by - construction — `.txt` is already in the live `ALLOWED_TYPES`, so only `.asc` - still has to be added. -3. **`INSTALL.md` was added** at the repository root. `dist` hard-requires it in - the staged payload and `verify-dist` asserts its mode. -4. **The notices preamble moved to `licenses/preamble.txt`.** The tarball and the - image ship different sets, and the hardcoded preamble scoped itself to "a - single static executable" — which the image is not. The image additionally - redistributes Alpine's Mozilla CA bundle (`MPL-2.0 AND MIT`, 179,359 bytes); - the tarball does not. -5. **Vite joined Tailwind in the inventory.** Both are devDependencies whose - generated output ships inside the binary. The original inventory applied that - rule to one of them and stopped. -6. **Node is pinned to `24.19.0`** in `gates.yml`, `release.yml` and - `web/package.json` — the exact-patch pin ruling 12 asks for. -7. **`gates.yml` is SHA-pinned too.** Ruling 7 pinned only `release.yml`, but - `release.yml` calls `gates.yml`, and those jobs share the runner host and - docker daemon with the job holding the signing subkey. `live-tls.yml` keeps - moving tags; it references no secret. -8. **Determinism measured better than claimed.** Two `dist` runs with separate - cache directories and separate prefixes produced byte-identical tarballs — a - genuine recompile, not a cache replay. The documentation still claims only - same-directory determinism, because ruling 12's gate does not exist yet and an - unguarded property decays. The stronger result is recorded here, not promised - to operators. -9. **Test count moved from 1461 to 1481**: twelve licence-drift tests, then eight - more from the review pass below. Skip counts are unchanged. +1. **`build.zig.zon` said `0.1.0`.** It traced to the first build-baseline commit — a scaffold default never bumped. Ruling 2 makes `verify-dist` assert it against the version under build, so the CI packaging gate could never have passed. Set to `0.0.1`, matching the first tag. +2. **Assets carry a `.txt` extension**: `SHA256SUMS.txt`, `SHA256SUMS.txt.asc`, `IMAGE-DIGEST.txt`. This resolves ruling 13's extensionless-asset probe by construction — `.txt` is already in the live `ALLOWED_TYPES`, so only `.asc` still has to be added. +3. **`INSTALL.md` was added** at the repository root. `dist` hard-requires it in the staged payload and `verify-dist` asserts its mode. +4. **The notices preamble moved to `licenses/preamble.txt`.** The tarball and the image ship different sets, and the hardcoded preamble scoped itself to "a single static executable" — which the image is not. The image additionally redistributes Alpine's Mozilla CA bundle (`MPL-2.0 AND MIT`, 179,359 bytes); the tarball does not. +5. **Vite joined Tailwind in the inventory.** Both are devDependencies whose generated output ships inside the binary. The original inventory applied that rule to one of them and stopped. +6. **Node is pinned to `24.19.0`** in `gates.yml`, `release.yml` and `web/package.json` — the exact-patch pin ruling 12 asks for. +7. **`gates.yml` is SHA-pinned too.** Ruling 7 pinned only `release.yml`, but `release.yml` calls `gates.yml`, and those jobs share the runner host and docker daemon with the job holding the signing subkey. `live-tls.yml` keeps moving tags; it references no secret. +8. **Determinism measured better than claimed.** Two `dist` runs with separate cache directories and separate prefixes produced byte-identical tarballs — a genuine recompile, not a cache replay. The documentation still claims only same-directory determinism, because ruling 12's gate does not exist yet and an unguarded property decays. The stronger result is recorded here, not promised to operators. +9. **Test count moved from 1461 to 1481**: twelve licence-drift tests, then eight more from the review pass below. Skip counts are unchanged. -The rest came out of an adversarial review of the finished implementation. Each -was reproduced before it was fixed. +The rest came out of an adversarial review of the finished implementation. Each was reproduced before it was fixed. -10. **The version tag was pushed before the immutability check ran.** The first - implementation pushed `:$VERSION` and then compared the resulting digest - with the pre-push one — a check that reports a violation it just caused. On - a re-run that produced different bytes the tag was already overwritten and - the original image lost. Replaced by the probe-then-adopt design ruling 9 - now describes: a real `HEAD /v2/…/manifests/`, and an existing tag - is adopted rather than rebuilt. Exercised against a fake registry covering - `404`, `200`, the `401` bearer challenge, `500`, and a `200` with no - `Docker-Content-Digest`. -11. **The guard proved nothing about the signing key.** It checked that - `RELEASE_GPG_SUBKEY` and `RELEASE_GPG_PASSPHRASE` were non-empty. A - public-only export, an export missing the pinned subkey, and a placeholder - passphrase all passed it and failed for the first time in the signing step — - after the image push. The guard now imports the material, asserts the pinned - subkey is present as a secret key, and signs a throwaway file with the - passphrase. -12. **`jq … | grep … || true` swallowed a malformed API response.** The `|| true` - exists so grep's no-match is not fatal; it covers the whole pipeline, so a - `200` carrying a JSON object instead of an array read as "no published - releases" — the one wrong answer that moves `:latest` backwards. The payload - is now type-checked before it is read. -13. **The monotonic re-check was not the concurrency backstop it claimed to be.** - Two concurrent releases are both drafts while they run, so neither appears in - the other's published list and both pass. The workflow `concurrency:` group - is the only thing that serialises them. A second check was added that does - close it: `:latest`'s own version label is read from the registry - immediately before the tag moves, so the invariant is checked against the - state being mutated. -14. **Publication could deadlock the tag.** A `PATCH` that Gitea committed but - whose response was lost left the release public and the step failed, after - which the guard refused every re-run. The step now re-reads the release and - treats an already-published one as success. -15. **The secret-scrub backstop killed the wrong gpg-agent.** `gpgconf --kill` - acts on the agent of the `GNUPGHOME` it is pointed at, and the bare call - killed the runner's default agent while every leaked temporary home's agent - kept running with the key cached. It now kills each home in its own home, - and the guard job has the same `if: always()` backstop the publish job had. -16. **The Zig dependency guard could be silenced by pasting.** Unlike the npm - half, nothing tied `build.zig.zon` back to the inventory, so a Mbed TLS bump - plus the suggested identity paste left the notices claiming the old version. - Each dependency now has to appear in the inventory at the version its URL - names. The Zig toolchain version and the entries vendored inside Mbed TLS are - checked the same way. -17. **The container base image was not an input to any guard.** It is the source - of the CA bundle the image redistributes. `deploy/docker/Dockerfile` is now - embedded in the `licenses_files` module, its digest-pinned `FROM` is a - recorded identity section, and the CA bundle entry must name the Alpine - release that `FROM` pins. -18. **A tree-shaken package that started shipping would have gone unnoticed.** - `cookie-es`, `isbot`, `seroval` and `seroval-plugins` are in the lockfile - closure and in no shipped byte. If application code imported one, no - lockfile, version or dependency set would change — only the bundle. The - frontend gate now recomputes the set of packages in `web/dist` from a - `--sourcemap` build and diffs it against a recorded section, and the drift - test requires every name in that section to be inventoried and refuses one - that is still listed as not shipped. -19. **The recorded npm licence token was parsed and discarded**, so a package - that relicensed passed as long as its version had not moved. Shipped - packages must now all carry the licence the inventory's texts assume; - `npm_not_shipped` is exempt, and `isbot` is Unlicense. -20. **The full-text licence checks were marker probes.** They prove the right - document is present but survive most of it being deleted. The Apache-2.0 and - MPL-2.0 texts are now pinned by SHA-256. -21. **Zig's compiler-rt contains code ported from LLVM's.** Zig's `LICENSE` is - bare MIT naming only "Zig contributors". Reviewed: the ports carry - Apache-2.0 WITH LLVM-exception, and that exception waives Apache §4(a), - §4(b) and §4(d) for portions embedded in object form — the only form nxdns - ships — so no further notice is owed. Recorded in the Zig inventory note - rather than left as an unexamined gap. -22. **`npx` was replaced by the installed binary** in the new frontend gate. `npx` - downloads a package it cannot find locally, so a wrong working directory - would have turned a licence check into an unpinned fetch. Reproduced: it - fetched `vite@8.2.0` over the pinned `8.1.5`. +10. **The version tag was pushed before the immutability check ran.** The first implementation pushed `:$VERSION` and then compared the resulting digest with the pre-push one — a check that reports a violation it just caused. On a re-run that produced different bytes the tag was already overwritten and the original image lost. Replaced by the probe-then-adopt design ruling 9 now describes: a real `HEAD /v2/…/manifests/`, and an existing tag is adopted rather than rebuilt. Exercised against a fake registry covering `404`, `200`, the `401` bearer challenge, `500`, and a `200` with no `Docker-Content-Digest`. +11. **The guard proved nothing about the signing key.** It checked that `RELEASE_GPG_SUBKEY` and `RELEASE_GPG_PASSPHRASE` were non-empty. A public-only export, an export missing the pinned subkey, and a placeholder passphrase all passed it and failed for the first time in the signing step — after the image push. The guard now imports the material, asserts the pinned subkey is present as a secret key, and signs a throwaway file with the passphrase. +12. **`jq … | grep … || true` swallowed a malformed API response.** The `|| true` exists so grep's no-match is not fatal; it covers the whole pipeline, so a `200` carrying a JSON object instead of an array read as "no published releases" — the one wrong answer that moves `:latest` backwards. The payload is now type-checked before it is read. +13. **The monotonic re-check was not the concurrency backstop it claimed to be.** Two concurrent releases are both drafts while they run, so neither appears in the other's published list and both pass. The workflow `concurrency:` group is the only thing that serialises them. A second check was added that does close it: `:latest`'s own version label is read from the registry immediately before the tag moves, so the invariant is checked against the state being mutated. +14. **Publication could deadlock the tag.** A `PATCH` that Gitea committed but whose response was lost left the release public and the step failed, after which the guard refused every re-run. The step now re-reads the release and treats an already-published one as success. +15. **The secret-scrub backstop killed the wrong gpg-agent.** `gpgconf --kill` acts on the agent of the `GNUPGHOME` it is pointed at, and the bare call killed the runner's default agent while every leaked temporary home's agent kept running with the key cached. It now kills each home in its own home, and the guard job has the same `if: always()` backstop the publish job had. +16. **The Zig dependency guard could be silenced by pasting.** Unlike the npm half, nothing tied `build.zig.zon` back to the inventory, so a Mbed TLS bump plus the suggested identity paste left the notices claiming the old version. Each dependency now has to appear in the inventory at the version its URL names. The Zig toolchain version and the entries vendored inside Mbed TLS are checked the same way. +17. **The container base image was not an input to any guard.** It is the source of the CA bundle the image redistributes. `deploy/docker/Dockerfile` is now embedded in the `licenses_files` module, its digest-pinned `FROM` is a recorded identity section, and the CA bundle entry must name the Alpine release that `FROM` pins. +18. **A tree-shaken package that started shipping would have gone unnoticed.** `cookie-es`, `isbot`, `seroval` and `seroval-plugins` are in the lockfile closure and in no shipped byte. If application code imported one, no lockfile, version or dependency set would change — only the bundle. The frontend gate now recomputes the set of packages in `web/dist` from a `--sourcemap` build and diffs it against a recorded section, and the drift test requires every name in that section to be inventoried and refuses one that is still listed as not shipped. +19. **The recorded npm licence token was parsed and discarded**, so a package that relicensed passed as long as its version had not moved. Shipped packages must now all carry the licence the inventory's texts assume; `npm_not_shipped` is exempt, and `isbot` is Unlicense. +20. **The full-text licence checks were marker probes.** They prove the right document is present but survive most of it being deleted. The Apache-2.0 and MPL-2.0 texts are now pinned by SHA-256. +21. **Zig's compiler-rt contains code ported from LLVM's.** Zig's `LICENSE` is bare MIT naming only "Zig contributors". Reviewed: the ports carry Apache-2.0 WITH LLVM-exception, and that exception waives Apache §4(a), §4(b) and §4(d) for portions embedded in object form — the only form nxdns ships — so no further notice is owed. Recorded in the Zig inventory note rather than left as an unexamined gap. +22. **`npx` was replaced by the installed binary** in the new frontend gate. `npx` downloads a package it cannot find locally, so a wrong working directory would have turned a licence check into an unpinned fetch. Reproduced: it fetched `vite@8.2.0` over the pinned `8.1.5`. -23. **`actions/checkout` destroys the annotated tag object.** Found by the first - live dry run, not by review: on a tag ref, checkout fetches the *commit* SHA - into `refs/tags/`, so the signed tag reads as lightweight and the guard - refuses it as unannotated. Both jobs that read the tag object — signature - verification in the guard, the tagger date in the publish job — now force- - refetch `refs/tags/$TAG` from origin first. The same run also proved the - fail-closed secret guard for real: the first dry-run attempt ran with no - secrets configured (they were on the wrong repository) and stopped in the - guard with nothing built or pushed. +23. **`actions/checkout` destroys the annotated tag object.** Found by the first live dry run, not by review: on a tag ref, checkout fetches the *commit* SHA into `refs/tags/`, so the signed tag reads as lightweight and the guard refuses it as unannotated. Both jobs that read the tag object — signature verification in the guard, the tagger date in the publish job — now force- refetch `refs/tags/$TAG` from origin first. The same run also proved the fail-closed secret guard for real: the first dry-run attempt ran with no secrets configured (they were on the wrong repository) and stopped in the guard with nothing built or pushed. -24. **Publication orchestration moved out of workflow shell into - `tools/release.zig`.** Ruling 5 already moved the packaging asserts out of - CI shell for one reason — "checks that only exist inside a workflow file are - the brittleness this exists to remove" — and the release job was the larger - half of the same problem, left in place. Three live failures came out of it, - and each was found by executing the workflow, which is the most expensive - place to find anything: `actions/checkout` replacing the annotated tag object - (deviation 23), the refetch that fixed it having no credentials because - `persist-credentials` is off, and the multiline armored subkey escaping the - runner's log masker, which masks per line. +24. **Publication orchestration moved out of workflow shell into `tools/release.zig`.** Ruling 5 already moved the packaging asserts out of CI shell for one reason — "checks that only exist inside a workflow file are the brittleness this exists to remove" — and the release job was the larger half of the same problem, left in place. Three live failures came out of it, and each was found by executing the workflow, which is the most expensive place to find anything: `actions/checkout` replacing the annotated tag object (deviation 23), the refetch that fixed it having no credentials because `persist-credentials` is off, and the multiline armored subkey escaping the runner's log masker, which masks per line. Twelve subcommands, one per step group: `guard-tag`, `guard-ancestry`, `guard-releases`, `resolve`, `changelog`, `image`, @@ -627,55 +304,18 @@ was reproduced before it was fixed. writes it to a mode-600 file inside the temporary `GNUPGHOME`. A single-line secret is one the masker can actually mask. -25. **The first signing subkey was leaked into a job log and rotated.** Dry-run - attempt 3 failed inside the credential-less refetch, and the runner printed - the failing step's env block; the multiline armored `RELEASE_GPG_SUBKEY` - escaped the per-line masker while the single-line passphrase was masked. - Exposure: the passphrase-protected secret subkey only — the passphrase and - the primary key were never on the runner. Response: both runs that ever saw - the secret were deleted (verified 404 via the API and absent from - `actions_log` on disk), subkey `B281CECC…` was revoked with the primary, - and its replacement `019D00DF…` is the pinned `RELEASE_SIGNING_FPR`. The - base64 contract in deviation 24 is the preventive half of this record. +25. **The first signing subkey was leaked into a job log and rotated.** Dry-run attempt 3 failed inside the credential-less refetch, and the runner printed the failing step's env block; the multiline armored `RELEASE_GPG_SUBKEY` escaped the per-line masker while the single-line passphrase was masked. Exposure: the passphrase-protected secret subkey only — the passphrase and the primary key were never on the runner. Response: both runs that ever saw the secret were deleted (verified 404 via the API and absent from `actions_log` on disk), subkey `B281CECC…` was revoked with the primary, and its replacement `019D00DF…` is the pinned `RELEASE_SIGNING_FPR`. The base64 contract in deviation 24 is the preventive half of this record. -26. **The image is named by the public registry host, never the server URL.** - Attempt 4 reached the registry and failed at `docker login gitea:3000`: - inside the cluster `GITHUB_SERVER_URL` is `http://gitea:3000`, docker - refuses plain-http registries, and an image named `gitea:3000/…` would be - unpullable from anywhere that matters — a wrong name that would have been - written into the released `IMAGE-DIGEST.txt`. `release.yml` now pins - `REGISTRY_HOST: git.mial.net`; the tool uses it for docker and image - naming, and keeps the internal URL for the manifest probe (same registry, - no TLS dependency in the tool). The old shell had the identical latent bug; - no run ever reached it. +26. **The image is named by the public registry host, never the server URL.** Attempt 4 reached the registry and failed at `docker login gitea:3000`: inside the cluster `GITHUB_SERVER_URL` is `http://gitea:3000`, docker refuses plain-http registries, and an image named `gitea:3000/…` would be unpullable from anywhere that matters — a wrong name that would have been written into the released `IMAGE-DIGEST.txt`. `release.yml` now pins `REGISTRY_HOST: git.mial.net`; the tool uses it for docker and image naming, and keeps the internal URL for the manifest probe (same registry, no TLS dependency in the tool). The old shell had the identical latent bug; no run ever reached it. ### Not verified, and why -- **The workflows' validation history.** Before any live run, `release.yml` and - `gates.yml` were validated by YAML parse and `bash -n`, plus two steps lifted - out and run directly: the registry probe against a fake registry (five - response shapes) and the bundled-package check against the real `web/` build, - proven able to fail. The live dry run then superseded this: attempt 5 - published `v0.0.0` end to end — guard, gates, image push to both platforms, - binary-identity assertion, signing, draft, `:latest`, publication — and the - assets verified from a clean directory (checksums OK, signature good under - the rotated subkey). The throwaway release, tag and registry versions were - deleted afterwards. - Everything else that talks to the registry or the Gitea API — `buildx build - --push`, `imagetools`, draft creation, asset upload, publication, the - adopt-an-existing-tag path — is unexercised. -- **`RELEASE_SIGNING_FPR` is still the placeholder.** By design: the guard fails - closed on it. It also means the release workflow cannot succeed as committed - until manual prerequisite 2 is done. -- **The aarch64 binary was never executed.** No `qemu-aarch64` on the build host, - so `verify-dist`'s aarch64 version check legitimately skips and `test-aarch64` - could not run. The binary is checked statically: ELF class, no `PT_INTERP`, no - `DT_NEEDED`, size. -- **The multi-architecture image build is unverified.** Only the native amd64 - image was built and run. +- **The workflows' validation history.** Before any live run, `release.yml` and `gates.yml` were validated by YAML parse and `bash -n`, plus two steps lifted out and run directly: the registry probe against a fake registry (five response shapes) and the bundled-package check against the real `web/` build, proven able to fail. The live dry run then superseded this: attempt 5 published `v0.0.0` end to end — guard, gates, image push to both platforms, binary-identity assertion, signing, draft, `:latest`, publication — and the assets verified from a clean directory (checksums OK, signature good under the rotated subkey). The throwaway release, tag and registry versions were deleted afterwards. Everything else that talks to the registry or the Gitea API — `buildx build --push`, `imagetools`, draft creation, asset upload, publication, the adopt-an-existing-tag path — is unexercised. +- **`RELEASE_SIGNING_FPR` is still the placeholder.** By design: the guard fails closed on it. It also means the release workflow cannot succeed as committed until manual prerequisite 2 is done. +- **The aarch64 binary was never executed.** No `qemu-aarch64` on the build host, so `verify-dist`'s aarch64 version check legitimately skips and `test-aarch64` could not run. The binary is checked statically: ELF class, no `PT_INTERP`, no `DT_NEEDED`, size. +- **The multi-architecture image build is unverified.** Only the native amd64 image was built and run. - **`shellcheck` was not run** — not installed on the build host. -- **The local Node is 24.14.1, not the pinned 24.19.0.** There is no `.npmrc`, so - `engines` does not hard-fail, and the frontend gates ran under the older patch. +- **The local Node is 24.14.1, not the pinned 24.19.0.** There is no `.npmrc`, so `engines` does not hard-fail, and the frontend gates ran under the older patch. ## Acceptance (milestone complete) @@ -742,8 +382,7 @@ was reproduced before it was fixed. - No GoReleaser, no nfpm, no `.deb` or `.rpm`, no Homebrew, no AUR. - No cosign, no SBOM, no SLSA provenance, no in-toto attestations. -- No `:edge` image, no rolling dev build, no pre-release tags, no floating - `0.0` or `0` image tags. +- No `:edge` image, no rolling dev build, no pre-release tags, no floating `0.0` or `0` image tags. - No third architecture, no non-Linux target, no glibc build. - No project website, no documentation site generator. - No changelog generation from commit messages, and no Conventional Commits. diff --git a/specs/milestone-15.md b/specs/milestone-15.md index ac0665d..24ac264 100644 --- a/specs/milestone-15.md +++ b/specs/milestone-15.md @@ -1,44 +1,24 @@ # Milestone 15: make a green run mean a real pass -Goal: repair the verification pipeline — CI that watches the branch we push to, -a test run whose output contains no false failure text, guards that make the -test list and the embedded frontend complete by construction, and coverage for -the seams the audit found unguarded. This is phase 1 of `TECH_DEBT.md`; nothing -in phases 2-5 can be validated until this lands. +Goal: repair the verification pipeline — CI that watches the branch we push to, a test run whose output contains no false failure text, guards that make the test list and the embedded frontend complete by construction, and coverage for the seams the audit found unguarded. This is phase 1 of `TECH_DEBT.md`; nothing in phases 2-5 can be validated until this lands. -Origin: `TECH_DEBT.md` Theme 1 (lines 17-52), audited at `1ff727f`, every -finding adversarially verified. +Origin: `TECH_DEBT.md` Theme 1 (lines 17-52), audited at `1ff727f`, every finding adversarially verified. ## Sequencing (binding) -This milestone lands **before** milestone 14 is implemented. Ruling 1 pulls the -`ci.yml` branch switch forward from milestone-14 §7; when milestone 14's S4 -later restructures the workflows into `gates.yml`/`release.yml`, it rebases on -what this milestone leaves behind. One more overlap is binding: milestone 14's -S1 rewrites `build.zig` (deletes `cross`, adds `dist`/`verify-dist`) — it must -rebase on and **preserve** the configure-time machinery this milestone adds -there (ruling 4's import guard, ruling 5's stamp check, ruling 6b's staged -`core` module). Milestone 14's spec stands as written otherwise. +This milestone lands **before** milestone 14 is implemented. Ruling 1 pulls the `ci.yml` branch switch forward from milestone-14 §7; when milestone 14's S4 later restructures the workflows into `gates.yml`/`release.yml`, it rebases on what this milestone leaves behind. One more overlap is binding: milestone 14's S1 rewrites `build.zig` (deletes `cross`, adds `dist`/`verify-dist`) — it must rebase on and **preserve** the configure-time machinery this milestone adds there (ruling 4's import guard, ruling 5's stamp check, ruling 6b's staged `core` module). Milestone 14's spec stands as written otherwise. ## Rulings (binding) ### 1. CI watches `master` -`.gitea/workflows/ci.yml:1-11` triggers on `push`/`pull_request` for `main`. -Day-to-day pushes go to `origin/master` (`branch.master.merge` confirms), so -milestone 13+ never ran through any gate. Change both `branches:` lists to -`[master]`. Nothing else in the file changes — the `gates.yml` restructure is -milestone-14 work. +`.gitea/workflows/ci.yml:1-11` triggers on `push`/`pull_request` for `main`. Day-to-day pushes go to `origin/master` (`branch.master.merge` confirms), so milestone 13+ never ran through any gate. Change both `branches:` lists to `[master]`. Nothing else in the file changes — the `gates.yml` restructure is milestone-14 work. -Orchestrator, after merge: delete `origin/main`, set the Gitea default branch -to `master`, push, and confirm a run starts. (Milestone-14 §7 already rules -`master` canonical; this executes the branch part now.) +Orchestrator, after merge: delete `origin/main`, set the Gitea default branch to `master`, push, and confirm a run starts. (Milestone-14 §7 already rules `master` canonical; this executes the branch part now.) ### 2. The live-network suite runs on a schedule -`.gitea/workflows/live-tls.yml` runs on `workflow_dispatch` only. Both failure -modes it exists to catch have each happened once (35f2324; the milestone-4 cert -drift). Add: +`.gitea/workflows/live-tls.yml` runs on `workflow_dispatch` only. Both failure modes it exists to catch have each happened once (35f2324; the milestone-4 cert drift). Add: ```yaml on: @@ -47,225 +27,83 @@ on: - cron: "0 5 * * 1" ``` -Weekly, Monday 05:00 UTC. It stays non-blocking — this does not conflict with -the milestone-1 ruling that live tests never gate a push. Everything else in -the file is untouched. +Weekly, Monday 05:00 UTC. It stays non-blocking — this does not conflict with the milestone-1 ruling that live tests never gate a push. Everything else in the file is untouched. ### 3. The teardown abort is root-caused, or documented where it lives -Reproduction, verified at HEAD: `zig build test` exits 0 but prints as its -last line `failed command: .../test ... --listen=-`. The same binary exits 0 -in stdio mode (`1242 passed; 115 skipped; 0 failed`) and exits 134 (SIGABRT) -under `--listen=-`. The abort happens after all tests pass, in teardown, only -under zig's IPC runner. No comment anywhere in the repository records this; -the knowledge lives in one contributor's memory and `TECH_DEBT.md:24-25`. +Reproduction, verified at HEAD: `zig build test` exits 0 but prints as its last line `failed command: .../test ... --listen=-`. The same binary exits 0 in stdio mode (`1242 passed; 115 skipped; 0 failed`) and exits 134 (SIGABRT) under `--listen=-`. The abort happens after all tests pass, in teardown, only under zig's IPC runner. No comment anywhere in the repository records this; the knowledge lives in one contributor's memory and `TECH_DEBT.md:24-25`. -The cost is not the abort — it is that the one string that should mean failure -prints on every success, and everyone learns to ignore failure text. +The cost is not the abort — it is that the one string that should mean failure prints on every success, and everyone learns to ignore failure text. The session must attempt a root cause, in this order: -1. Reproduce: run the cached test binary directly with and without - `--listen=-`. Capture the abort backtrace (`ulimit -c` / gdb / the zig - `--debug-...` runner flags — whatever this host offers). -2. Bisect the mbedTLS linkage: `addMbedtlsThreadingMacros` (build.zig), the - shim (`src/platform/mbedtls_shim.c`), and the two `extern fn` link checks in - `src/tests.zig:121-123` are the suspects. A minimal reproducer (empty test - file + mbedTLS link + `--listen`) decides whether the abort is ours or an - upstream zig 0.16 runner defect. -3. Outcome A — the cause is ours: fix it. Acceptance is a `zig build test` run - whose output contains no `failed command` line. -4. Outcome B — the cause is upstream: do not patch around it. Document it in - two places: a comment on the test wiring in `build.zig` (at the - `b.addTest` block, lines 48-69) stating the exact mechanism and the upstream - reference, and a paragraph in `AGENTS.md` stating that `failed command` + - exit 0 after a full pass is this known artifact and that any *other* - failure text is real. +1. Reproduce: run the cached test binary directly with and without `--listen=-`. Capture the abort backtrace (`ulimit -c` / gdb / the zig `--debug-...` runner flags — whatever this host offers). +2. Bisect the mbedTLS linkage: `addMbedtlsThreadingMacros` (build.zig), the shim (`src/platform/mbedtls_shim.c`), and the two `extern fn` link checks in `src/tests.zig:121-123` are the suspects. A minimal reproducer (empty test file + mbedTLS link + `--listen`) decides whether the abort is ours or an upstream zig 0.16 runner defect. +3. Outcome A — the cause is ours: fix it. Acceptance is a `zig build test` run whose output contains no `failed command` line. +4. Outcome B — the cause is upstream: do not patch around it. Document it in two places: a comment on the test wiring in `build.zig` (at the `b.addTest` block, lines 48-69) stating the exact mechanism and the upstream reference, and a paragraph in `AGENTS.md` stating that `failed command` + exit 0 after a full pass is this known artifact and that any *other* failure text is real. -Either way: no check is loosened, no output is filtered, and the spec is -updated afterward to record which outcome held (per the spec-update rule). +Either way: no check is loosened, no output is filtered, and the spec is updated afterward to record which outcome held (per the spec-update rule). -**Outcome recorded (implementation): B — upstream, and this ruling's own -diagnosis was wrong.** The label is not an abort and has nothing to do with -mbedTLS. In Zig 0.16.0, `std/Build/Step/Run.zig:1540` sets -`result_failed_command` unconditionally on every spawn and nothing clears it -on success; `compiler/build_runner.zig:1381` prints a step's diagnostics -whenever the child wrote to stderr, "no matter the result", and `:1515` then -emits the `failed command:` label because the field is non-null. Any Run -step that succeeds while writing one byte to stderr gets the label; our -warning-path tests log through the real sink. Minimal reproducer: one -passing test containing a `std.debug.print` shows the label; the same test -without the print shows nothing. The old "exits 134 under `--listen=-`" -observation was a manual-invocation artifact: the IPC runner panics with -`EndOfStream` (test_runner.zig:88) reading a closed stdin when no build -runner sits on the other end. Documented in the `b.addTest` comment in -build.zig and in AGENTS.md ("Reading `zig build test` output"). No matching -upstream issue found; the 0.16.0 source lines are the reference. +**Outcome recorded (implementation): B — upstream, and this ruling's own diagnosis was wrong.** The label is not an abort and has nothing to do with mbedTLS. In Zig 0.16.0, `std/Build/Step/Run.zig:1540` sets `result_failed_command` unconditionally on every spawn and nothing clears it on success; `compiler/build_runner.zig:1381` prints a step's diagnostics whenever the child wrote to stderr, "no matter the result", and `:1515` then emits the `failed command:` label because the field is non-null. Any Run step that succeeds while writing one byte to stderr gets the label; our warning-path tests log through the real sink. Minimal reproducer: one passing test containing a `std.debug.print` shows the label; the same test without the print shows nothing. The old "exits 134 under `--listen=-`" observation was a manual-invocation artifact: the IPC runner panics with `EndOfStream` (test_runner.zig:88) reading a closed stdin when no build runner sits on the other end. Documented in the `b.addTest` comment in build.zig and in AGENTS.md ("Reading `zig build test` output"). No matching upstream issue found; the 0.16.0 source lines are the reference. ### 4. The test import list gets a completeness guard -`src/tests.zig:3-119` is a hand-maintained `comptime` block of 115 -`_ = @import("...");` lines. Zig collects tests only from the root module; a -forgotten import silently drops a file's tests (verified empirically). The -list stays hand-written — generation via a staged copy of `src/` would point -diagnostics at cache paths — but it becomes complete by construction: +`src/tests.zig:3-119` is a hand-maintained `comptime` block of 115 `_ = @import("...");` lines. Zig collects tests only from the root module; a forgotten import silently drops a file's tests (verified empirically). The list stays hand-written — generation via a staged copy of `src/` would point diagnostics at cache paths — but it becomes complete by construction: -In `build.zig`, at configure time, walk `src/` recursively. For every `*.zig` -file except `src/tests.zig` itself, require that `src/tests.zig` contains a -**line** which, after whitespace trim, is exactly `_ = @import("");` -where `` is the file's path relative to `src/`. A line match, not a -substring search: a commented-out import (`// _ = @import(...)`) trims to a -line that starts with `//` and does not match, and a path that is a prefix of -another (`db.zig` vs `db2.zig`) cannot false-match because the full line is -compared. Duplicate matching lines are an error too (they hide a botched -merge). On the first missing file, fail the configure with an error that -names it: +In `build.zig`, at configure time, walk `src/` recursively. For every `*.zig` file except `src/tests.zig` itself, require that `src/tests.zig` contains a **line** which, after whitespace trim, is exactly `_ = @import("");` where `` is the file's path relative to `src/`. A line match, not a substring search: a commented-out import (`// _ = @import(...)`) trims to a line that starts with `//` and does not match, and a path that is a prefix of another (`db.zig` vs `db2.zig`) cannot false-match because the full line is compared. Duplicate matching lines are an error too (they hide a botched merge). On the first missing file, fail the configure with an error that names it: ``` src/tests.zig is missing `_ = @import("filter/new_module.zig");` ``` -No allowlist. A file with no tests still gets imported — the import is free, -and the rule stays exceptionless. The walk runs on the host in every -invocation, including the aarch64 configure. +No allowlist. A file with no tests still gets imported — the import is free, and the rule stays exceptionless. The walk runs on the host in every invocation, including the aarch64 configure. ### 5. `web/dist` gets a freshness stamp -A stale `web/dist` has already shipped a crashing settings page once -(`docs/explanation/performance-and-testing.md:154-163` admits the hole). -Mechanism — one implementation, in JavaScript, because the frontend CI job has -Node and no Zig: +A stale `web/dist` has already shipped a crashing settings page once (`docs/explanation/performance-and-testing.md:154-163` admits the hole). Mechanism — one implementation, in JavaScript, because the frontend CI job has Node and no Zig: - New file `web/scripts/stamp-dist.mjs`. Two modes: - - default (write): hash the input set, write the hex digest to - `web/dist/.src-hash`. - - `--check`: recompute, compare with `web/dist/.src-hash`, exit 1 with - `web/dist is stale: rebuild the frontend (npm run build)` on mismatch or - missing stamp. -- The script resolves every path (input files and `web/dist/.src-hash`) from - its own location via `import.meta.url`, never from `process.cwd()`: write - mode runs from `web/` (npm script) and check mode runs from the repo root - (build.zig system command), and both must hash the same set. The - acceptance run exercises both working directories. -- Input set, sorted by path, SHA-256 over `path ++ "\x00" ++ contents ++ - "\x00"` per file: every file under `web/src/` and `web/public/`, plus - `web/index.html`, `web/package.json`, `web/package-lock.json`, - `web/vite.config.ts`, `web/tsconfig.json`, `web/tsconfig.app.json`, - `web/tsconfig.node.json`. Node's `crypto` module; no new dependency. -- `web/package.json`: `"build"` becomes `"vite build && node - scripts/stamp-dist.mjs"`. -- `build.zig`: when the resolved `-Dweb-dist` value is exactly `web/dist`, the - asset pipeline (`webAssetsIndex`) gains a dependency on - `b.addSystemCommand(&.{ "node", "web/scripts/stamp-dist.mjs", "--check" })`. - The default `web/dist-placeholder` path and any other explicit path skip the - check; the `web-dist` option description says so. + - default (write): hash the input set, write the hex digest to `web/dist/.src-hash`. + - `--check`: recompute, compare with `web/dist/.src-hash`, exit 1 with `web/dist is stale: rebuild the frontend (npm run build)` on mismatch or missing stamp. +- The script resolves every path (input files and `web/dist/.src-hash`) from its own location via `import.meta.url`, never from `process.cwd()`: write mode runs from `web/` (npm script) and check mode runs from the repo root (build.zig system command), and both must hash the same set. The acceptance run exercises both working directories. +- Input set, sorted by path, SHA-256 over `path ++ "\x00" ++ contents ++ "\x00"` per file: every file under `web/src/` and `web/public/`, plus `web/index.html`, `web/package.json`, `web/package-lock.json`, `web/vite.config.ts`, `web/tsconfig.json`, `web/tsconfig.app.json`, `web/tsconfig.node.json`. Node's `crypto` module; no new dependency. +- `web/package.json`: `"build"` becomes `"vite build && node scripts/stamp-dist.mjs"`. +- `build.zig`: when the resolved `-Dweb-dist` value is exactly `web/dist`, the asset pipeline (`webAssetsIndex`) gains a dependency on `b.addSystemCommand(&.{ "node", "web/scripts/stamp-dist.mjs", "--check" })`. The default `web/dist-placeholder` path and any other explicit path skip the check; the `web-dist` option description says so. ### 6. Three new fuzz surfaces, corpus replay only -All targets follow the existing registration shape (one `test "fuzz ..."` + -one `fn target(ctx, smith: *Smith)`), attach to `test_step`, and run as corpus -replay under plain `zig build test` — once per corpus entry plus once on empty -input. CI never passes `--fuzz` (blocked by upstream zig 0.16 defects; the -audit confirmed corpus replay is the sanctioned CI smoke). +All targets follow the existing registration shape (one `test "fuzz ..."` + one `fn target(ctx, smith: *Smith)`), attach to `test_step`, and run as corpus replay under plain `zig build test` — once per corpus entry plus once on empty input. CI never passes `--fuzz` (blocked by upstream zig 0.16 defects; the audit confirmed corpus replay is the sanctioned CI smoke). -**6a. `stripEcs`** — the only attacker-facing packet-rewriting entry point, -absent from the fuzz targets. Add a target to `tests/fuzz/dns_fuzz.zig`. -Derive input exactly as `parseTarget` (lines 57-89) already does: it ends with -a validated `Packet` and its `OptRecord`, which are `stripEcs`'s second and -third parameters. Respect the three entry assertions -(`src/dns/edns.zig:196-204`): pass the same `bytes` the packet was parsed -from, and a separate stack `out` buffer — an assertion trip from a violated -precondition is not a finding. Assert on `.rewritten` output: it re-parses, -`findOptRecord` + `parseOpt` succeed, no ECS option (code 8) remains, and -header counts survive. No build.zig change — the module exists. +**6a. `stripEcs`** — the only attacker-facing packet-rewriting entry point, absent from the fuzz targets. Add a target to `tests/fuzz/dns_fuzz.zig`. Derive input exactly as `parseTarget` (lines 57-89) already does: it ends with a validated `Packet` and its `OptRecord`, which are `stripEcs`'s second and third parameters. Respect the three entry assertions (`src/dns/edns.zig:196-204`): pass the same `bytes` the packet was parsed from, and a separate stack `out` buffer — an assertion trip from a violated precondition is not a finding. Assert on `.rewritten` output: it re-parses, `findOptRecord` + `parseOpt` succeed, no ECS option (code 8) remains, and header counts survive. No build.zig change — the module exists. -**6b. `compiler.compile`** — the streaming -`takeDelimiter`/`StreamTooLong`/discard loop (`src/filter/compiler.zig:64-78`) -is never fuzzed, and its `error.EndOfStream`-during-discard arm (line 73) has -no test at all. `compiler.zig` imports `../dns/`, so a module rooted under -`src/filter/` fails with `ImportOutsideModulePath`. Use the staged-copy -aggregator pattern from `bench_core` (`build.zig:112-135`) — `bench_core` -already exposes `compiler` at line 124; reusing the same staged tree is -allowed. New file `tests/fuzz/compiler_fuzz.zig`, module import name `core`. -The target feeds `compile()` via `std.Io.Reader.fixed` over smith bytes with a -deliberately small reader buffer, into discarding writers, and asserts it -returns without panic and that `counts` are internally consistent. Corpus -(inline, `sliceInput` idiom): a line longer than `max_line_len` (4096) with -and without a trailing `\n`, so both the discard path and the -EndOfStream-during-discard arm replay. Add a plain unit test for the line-73 -arm alongside the existing compiler tests. +**6b. `compiler.compile`** — the streaming `takeDelimiter`/`StreamTooLong`/discard loop (`src/filter/compiler.zig:64-78`) is never fuzzed, and its `error.EndOfStream`-during-discard arm (line 73) has no test at all. `compiler.zig` imports `../dns/`, so a module rooted under `src/filter/` fails with `ImportOutsideModulePath`. Use the staged-copy aggregator pattern from `bench_core` (`build.zig:112-135`) — `bench_core` already exposes `compiler` at line 124; reusing the same staged tree is allowed. New file `tests/fuzz/compiler_fuzz.zig`, module import name `core`. The target feeds `compile()` via `std.Io.Reader.fixed` over smith bytes with a deliberately small reader buffer, into discarding writers, and asserts it returns without panic and that `counts` are internally consistent. Corpus (inline, `sliceInput` idiom): a line longer than `max_line_len` (4096) with and without a trailing `\n`, so both the discard path and the EndOfStream-during-discard arm replay. Add a plain unit test for the line-73 arm alongside the existing compiler tests. -**6c. `http_util`** — the third untrusted-byte family, unfuzzed. -`src/web/http_util.zig` imports only `std`, so the fuzz module roots at a new -file `tests/fuzz/http_util_fuzz.zig` with `addImport("http_util", )` — no aggregator. Targets: `parsePath`, -`decodeInPlace` (both `PlusRule` values), `queryValue`. Invariants to assert: -split-before-decode and decode-only-shrinks (result length ≤ input length; -result is a prefix-aliased slice of the buffer). **Corrected during -implementation:** the original phrasing "a decoded segment never gains a `/`" -is false — `%2F` legitimately decodes to a literal `/` inside its segment -(http_util.zig's own tests assert it), and the fuzz target caught that on its -first run. The property that holds is a count: segmentation is decided by the -raw bytes, so each decoded segment pairs with its raw `/`-delimited chunk in -order, the segment counts match, and each segment is no longer than its -chunk. A decode-then-split implementation still fails this. -`dnsParam`/`decodeDnsValue` stay -private in `doh_server.zig` and stay unfuzzed — recorded, out of scope. +**6c. `http_util`** — the third untrusted-byte family, unfuzzed. `src/web/http_util.zig` imports only `std`, so the fuzz module roots at a new file `tests/fuzz/http_util_fuzz.zig` with `addImport("http_util", )` — no aggregator. Targets: `parsePath`, `decodeInPlace` (both `PlusRule` values), `queryValue`. Invariants to assert: split-before-decode and decode-only-shrinks (result length ≤ input length; result is a prefix-aliased slice of the buffer). **Corrected during implementation:** the original phrasing "a decoded segment never gains a `/`" is false — `%2F` legitimately decodes to a literal `/` inside its segment (http_util.zig's own tests assert it), and the fuzz target caught that on its first run. The property that holds is a count: segmentation is decided by the raw bytes, so each decoded segment pairs with its raw `/`-delimited chunk in order, the segment counts match, and each segment is no longer than its chunk. A decode-then-split implementation still fails this. `dnsParam`/`decodeDnsValue` stay private in `doh_server.zig` and stay unfuzzed — recorded, out of scope. ### 7. The multi-read fetch path gets a real fixture route -The exact seam of 35k-killer 35f2324 is still never driven end to end: every -fixture reply is one `request.respond` of a ~130-byte body -(`src/filter/filter_integration_test.zig:414`), and the post-fix unit tests -pre-buffer readers. Two thresholds matter and they are different: a body over -16 KiB (`fetcher.min_transfer_buf`, fetcher.zig:24) forces the fetcher's -`pumpBody` loop to iterate; a body over the fixture's 8192-byte write buffer -forces the fixture to flush in parts. +The exact seam of 35k-killer 35f2324 is still never driven end to end: every fixture reply is one `request.respond` of a ~130-byte body (`src/filter/filter_integration_test.zig:414`), and the post-fix unit tests pre-buffer readers. Two thresholds matter and they are different: a body over 16 KiB (`fetcher.min_transfer_buf`, fetcher.zig:24) forces the fetcher's `pumpBody` loop to iterate; a body over the fixture's 8192-byte write buffer forces the fixture to flush in parts. -- Add a route `chunked` to the `Route` enum (line 354). Its `respond` arm uses - `respondStreaming`, writes a well-formed blocklist body of at least 24 KiB - in at least three writes with an explicit `flush()` between each, then ends - the stream. `keep_alive = false`, like every other arm (the comment at lines - 409-411 is load-bearing). -- Add one `-Dintegration` test that drives the full `refreshOnce` path through - this route and asserts the parsed domain counts. -- Prove it can fail: during development, reintroduce the - `readSliceShort(self.transfer_buf)` aliasing pattern locally and confirm the - new test dies where the old suite stayed green. Record the proof in the - session notes; do not commit the revert. **Proof recorded - (implementation):** with the aliasing reintroduced, the new test failed - (`expected 1500, found 1238`) while all 1378 other tests stayed green. On - this body the bug silently dropped 262 domains rather than crashing — the - count assertion does the real work, not the crash. +- Add a route `chunked` to the `Route` enum (line 354). Its `respond` arm uses `respondStreaming`, writes a well-formed blocklist body of at least 24 KiB in at least three writes with an explicit `flush()` between each, then ends the stream. `keep_alive = false`, like every other arm (the comment at lines 409-411 is load-bearing). +- Add one `-Dintegration` test that drives the full `refreshOnce` path through this route and asserts the parsed domain counts. +- Prove it can fail: during development, reintroduce the `readSliceShort(self.transfer_buf)` aliasing pattern locally and confirm the new test dies where the old suite stayed green. Record the proof in the session notes; do not commit the revert. **Proof recorded (implementation):** with the aliasing reintroduced, the new test failed (`expected 1500, found 1238`) while all 1378 other tests stayed green. On this body the bug silently dropped 262 domains rather than crashing — the count assertion does the real work, not the crash. ### 8. The rotation failure paths get tests -`src/platform/logging.zig` codifies an exactly-one-sink-error contract across -four functions (stated at lines 455-460) and its own test section admits the -rotation failure paths are uncovered (lines 604-612). Injection seam: +`src/platform/logging.zig` codifies an exactly-one-sink-error contract across four functions (stated at lines 455-460) and its own test section admits the rotation failure paths are uncovered (lines 604-612). Injection seam: ```zig var rotate_fault: enum { none, fail_delete, fail_rename } = .none; ``` -file-private, read by `deleteLocked` (line 590) and `renameLocked` (line 597) -as their first statement (`if (rotate_fault == .fail_delete) return -error.RotateFailed;`), compiled only under `@import("builtin").is_test`. Two -new tests, each following the established shape of the failed-open test at -lines 883-909 (save and restore `state.*`, hold `std.debug.lockStderr`): +file-private, read by `deleteLocked` (line 590) and `renameLocked` (line 597) as their first statement (`if (rotate_fault == .fail_delete) return error.RotateFailed;`), compiled only under `@import("builtin").is_test`. Two new tests, each following the established shape of the failed-open test at lines 883-909 (save and restore `state.*`, hold `std.debug.lockStderr`): -- `fail_delete`: `prepareFileLocked` returns false, `sink_errors` rose by - exactly one, `rotate_pending` stays set, the file stays closed. +- `fail_delete`: `prepareFileLocked` returns false, `sink_errors` rose by exactly one, `rotate_pending` stays set, the file stays closed. - `fail_rename`: same assertions through the rename step. ### 9. The CLI drift guard derives its needles -`src/docs_drift_test.zig:51` hardcodes the subcommand list; its two sibling -guards derive theirs (`routes.table`, `model.toSettings`). The source of truth -today is three parallel copies: the `if (eql(...))` parse chain -(`cli.zig:91-102`), the `main.zig:69-78` dispatch switch, and `usage_text` -(cli.zig:334). Note the spelling trap: tags are `export_`/`import_`, argv -strings are `export`/`import`. +`src/docs_drift_test.zig:51` hardcodes the subcommand list; its two sibling guards derive theirs (`routes.table`, `model.toSettings`). The source of truth today is three parallel copies: the `if (eql(...))` parse chain (`cli.zig:91-102`), the `main.zig:69-78` dispatch switch, and `usage_text` (cli.zig:334). Note the spelling trap: tags are `export_`/`import_`, argv strings are `export`/`import`. - Add to `cli.zig`: @@ -281,61 +119,41 @@ strings are `export`/`import`. }; ``` - plus a comptime assert that `command_names.len == - @typeInfo(Command).@"union".fields.len`, so a new tag without a table entry - fails the compile. -- `parseArgs` matches the command word by iterating `command_names` (the - per-command argument parsing that follows stays as-is). -- `docs_drift_test.zig` iterates `cli.command_names` for its `## `{s}`` - needles; the hardcoded list is deleted. -- New test in `cli.zig`: every `command_names[i].name` appears in - `usage_text`. -- `main.zig`'s dispatch switch stays — it is exhaustive over the union and the - compiler already guards it. + plus a comptime assert that `command_names.len == @typeInfo(Command).@"union".fields.len`, so a new tag without a table entry fails the compile. +- `parseArgs` matches the command word by iterating `command_names` (the per-command argument parsing that follows stays as-is). +- `docs_drift_test.zig` iterates `cli.command_names` for its `## `{s}`` needles; the hardcoded list is deleted. +- New test in `cli.zig`: every `command_names[i].name` appears in `usage_text`. +- `main.zig`'s dispatch switch stays — it is exhaustive over the union and the compiler already guards it. ## Sessions -S1-S4 run in parallel; no two sessions write the same file. The S1/S3 -interface is fixed here so neither blocks: S3 writes the fuzz source files -named in ruling 6; S1 wires them in `build.zig` with the module shapes named -there (6b gets `core` via the staged-copy pattern; 6c gets `http_util` rooted -at `src/web/http_util.zig`); both artifacts attach to `test_step` exactly like -`fuzz_tests` at build.zig:92. +S1-S4 run in parallel; no two sessions write the same file. The S1/S3 interface is fixed here so neither blocks: S3 writes the fuzz source files named in ruling 6; S1 wires them in `build.zig` with the module shapes named there (6b gets `core` via the staged-copy pattern; 6c gets `http_util` rooted at `src/web/http_util.zig`); both artifacts attach to `test_step` exactly like `fuzz_tests` at build.zig:92. ### Session S1: build system and stamp -Owns `build.zig`, `AGENTS.md`, `web/scripts/stamp-dist.mjs`, -`web/package.json`. Rulings 3, 4, 5, and the wiring half of 6. +Owns `build.zig`, `AGENTS.md`, `web/scripts/stamp-dist.mjs`, `web/package.json`. Rulings 3, 4, 5, and the wiring half of 6. ### Session S2: workflows -Owns `.gitea/workflows/ci.yml`, `.gitea/workflows/live-tls.yml`. Rulings 1 -(file edit only), 2. +Owns `.gitea/workflows/ci.yml`, `.gitea/workflows/live-tls.yml`. Rulings 1 (file edit only), 2. ### Session S3: fuzz targets and fixture -Owns `tests/fuzz/dns_fuzz.zig`, `tests/fuzz/compiler_fuzz.zig` (new), -`tests/fuzz/http_util_fuzz.zig` (new), -`src/filter/filter_integration_test.zig`, and the compiler unit-test addition -in `src/filter/compiler.zig`. Rulings 6 (target half), 7. +Owns `tests/fuzz/dns_fuzz.zig`, `tests/fuzz/compiler_fuzz.zig` (new), `tests/fuzz/http_util_fuzz.zig` (new), `src/filter/filter_integration_test.zig`, and the compiler unit-test addition in `src/filter/compiler.zig`. Rulings 6 (target half), 7. ### Session S4: guards and seams -Owns `src/cli.zig`, `src/docs_drift_test.zig`, `src/platform/logging.zig`. -Rulings 8, 9. +Owns `src/cli.zig`, `src/docs_drift_test.zig`, `src/platform/logging.zig`. Rulings 8, 9. ### Orchestrator -Ruling 1's remote operations (delete `origin/main`, flip the Gitea default, -confirm a run starts), the ruling-3 outcome recorded back into this spec, and -striking the closed Theme-1 findings in `TECH_DEBT.md`. +Ruling 1's remote operations (delete `origin/main`, flip the Gitea default, confirm a run starts), the ruling-3 outcome recorded back into this spec, and striking the closed Theme-1 findings in `TECH_DEBT.md`. ## Module layout New files: -- `web/scripts/stamp-dist.mjs` — dist freshness stamp, write and `--check` - modes. +- `web/scripts/stamp-dist.mjs` — dist freshness stamp, write and `--check` modes. - `tests/fuzz/compiler_fuzz.zig` — compiler streaming-loop target. - `tests/fuzz/http_util_fuzz.zig` — HTTP parser targets. @@ -377,13 +195,10 @@ New files: ## Anti-requirements -- No `gates.yml`, no `release.yml`, no `dist`/`verify-dist` steps — that is - milestone 14, which rebases on this milestone. +- No `gates.yml`, no `release.yml`, no `dist`/`verify-dist` steps — that is milestone 14, which rebases on this milestone. - No `--fuzz` anywhere in CI. - No fuzzing or relocation of `dnsParam`/`decodeDnsValue`. - No generated `tests.zig`; the hand list stays, the guard makes it complete. -- No frontend changes beyond `web/scripts/stamp-dist.mjs` and the one-line - `build` script edit. -- No filtering, wrapping, or suppression of test-runner output to hide the - `failed command` line. +- No frontend changes beyond `web/scripts/stamp-dist.mjs` and the one-line `build` script edit. +- No filtering, wrapping, or suppression of test-runner output to hide the `failed command` line. - No new dependencies, Zig or npm. diff --git a/specs/milestone-16.md b/specs/milestone-16.md index a0006b7..efae4a3 100644 --- a/specs/milestone-16.md +++ b/specs/milestone-16.md @@ -1,402 +1,169 @@ # Milestone 16: contained behavioral fixes -Goal: fix the verified bugs and silent failures from `TECH_DEBT.md` themes 3, -5, 6 and 7 — each one localized, each with a test that fails before and -passes after. No refactors: the duplication work is milestone 18, and a -refactor that must simultaneously fix behavior is how mirrored copies diverge -further. +Goal: fix the verified bugs and silent failures from `TECH_DEBT.md` themes 3, 5, 6 and 7 — each one localized, each with a test that fails before and passes after. No refactors: the duplication work is milestone 18, and a refactor that must simultaneously fix behavior is how mirrored copies diverge further. -**PROVISIONAL.** Written before milestone 15 was built. m15 touches -`build.zig`, workflows, fuzz files, `filter_integration_test.zig`, `cli.zig`, -`docs_drift_test.zig` and `logging.zig` — re-verify line references in those -files before starting. +**PROVISIONAL.** Written before milestone 15 was built. m15 touches `build.zig`, workflows, fuzz files, `filter_integration_test.zig`, `cli.zig`, `docs_drift_test.zig` and `logging.zig` — re-verify line references in those files before starting. ## Rulings (binding) ### 1. `loadSource` propagates cancellation -`src/filter/manager.zig:541-544` and `:547-550`: two identical catch sites -fold `error.Canceled` into `loadFailure(...)`, recording a bogus "Canceled" -load-failed status, consuming the one-shot cancellation, and publishing a -snapshot with the source excluded. Ten other catch sites in the file -propagate it correctly. Add `if (err == error.Canceled) return -error.Canceled;` beside the existing OutOfMemory arm at both sites — -`Manager.Error` already has the member. Unit test: a reader that fails with -Canceled during `loadSource` makes the whole call return Canceled and writes -no status. +`src/filter/manager.zig:541-544` and `:547-550`: two identical catch sites fold `error.Canceled` into `loadFailure(...)`, recording a bogus "Canceled" load-failed status, consuming the one-shot cancellation, and publishing a snapshot with the source excluded. Ten other catch sites in the file propagate it correctly. Add `if (err == error.Canceled) return error.Canceled;` beside the existing OutOfMemory arm at both sites — `Manager.Error` already has the member. Unit test: a reader that fails with Canceled during `loadSource` makes the whole call return Canceled and writes no status. ### 2. `commitStatus` never drops an outcome silently -`src/filter/manager.zig:1237-1250`: the id-match loop falls through with no -log when a source has no status entry (reachable via a startupPass / -web-insert race; milestone-5 policy says every non-ok state is recorded and -surfaced). Add a `log.warn` after the loop naming the source id. Unit test: -committing a status for an unknown id emits the warning and does not crash. +`src/filter/manager.zig:1237-1250`: the id-match loop falls through with no log when a source has no status entry (reachable via a startupPass / web-insert race; milestone-5 policy says every non-ok state is recorded and surfaced). Add a `log.warn` after the loop naming the source id. Unit test: committing a status for an unknown id emits the warning and does not crash. ### 3. Downloads and compiles leave the writer lock -`src/filter/manager.zig:609-624`: `refreshAll` holds `writer_lock` across -every download (300 s budget each, `app.zig:85`) and every compile; every -web mutation ends in `Manager.reload` on the same lock, so an unrelated rule -save parks uncancelably — on a request path with no timeout, occupying one -of 64 web slots — until the pass ends. The download-to-tmp flow already -exists (`download` writes `.raw.tmp` via `compiledName`, with -`deleteQuietly` defers at :646-648); the lock is just taken too early. +`src/filter/manager.zig:609-624`: `refreshAll` holds `writer_lock` across every download (300 s budget each, `app.zig:85`) and every compile; every web mutation ends in `Manager.reload` on the same lock, so an unrelated rule save parks uncancelably — on a request path with no timeout, occupying one of 64 web slots — until the pass ends. The download-to-tmp flow already exists (`download` writes `.raw.tmp` via `compiledName`, with `deleteQuietly` defers at :646-648); the lock is just taken too early. Restructure: -- New `refresh_lock: std.Io.Mutex` on Manager, serializing refresh passes - against each other only. +- New `refresh_lock: std.Io.Mutex` on Manager, serializing refresh passes against each other only. - Split `refreshOne` (:626-729): the fetch/detect/compile stages (:650, :659, :668) run under `refresh_lock` alone; the publish stage (:711), the status commit and the final `reloadLocked` run under `writer_lock`. -- `refreshAll` takes `refresh_lock` for the pass, and `writer_lock` only - per-source around publish plus once for the final reload. `refreshSource` - (:582-586) follows the same split. -- The lock-ordering rule, documented on both mutex fields: `refresh_lock` is - never acquired while holding `writer_lock`. -- `refresh_lock` serializes blocklist-**directory maintenance** too, not - just refresh passes: `pruneOrphans` (:1040, invariant comment at +- `refreshAll` takes `refresh_lock` for the pass, and `writer_lock` only per-source around publish plus once for the final reload. `refreshSource` (:582-586) follows the same split. +- The lock-ordering rule, documented on both mutex fields: `refresh_lock` is never acquired while holding `writer_lock`. +- `refresh_lock` serializes blocklist-**directory maintenance** too, not just refresh passes: `pruneOrphans` (:1040, invariant comment at :1130-1136) today assumes every temp-file writer holds `writer_lock`; - once downloads write `.raw.tmp`/`.list.tmp`/`.wild.tmp` under - `refresh_lock` alone, a concurrent source deletion (blocklists.zig - delete → reload → pruneFiles) could sweep an in-flight refresh's temp - files. `pruneOrphans` therefore takes `refresh_lock` first (then - `writer_lock` if it needs it — same order as everywhere), and its - invariant comment is rewritten. Regression test: a source deletion - during a stalled refresh must not delete the stalled refresh's temp - files. + once downloads write `.raw.tmp`/`.list.tmp`/`.wild.tmp` under `refresh_lock` alone, a concurrent source deletion (blocklists.zig delete → reload → pruneFiles) could sweep an in-flight refresh's temp files. `pruneOrphans` therefore takes `refresh_lock` first (then `writer_lock` if it needs it — same order as everywhere), and its invariant comment is rewritten. Regression test: a source deletion during a stalled refresh must not delete the stalled refresh's temp files. -Integration test (`filter_integration_test.zig`): with a refresh pass parked -on a deliberately stalled fixture route, a concurrent rule mutation -completes without waiting for the pass. DNS serving was never affected and -stays covered by the existing tests. +Integration test (`filter_integration_test.zig`): with a refresh pass parked on a deliberately stalled fixture route, a concurrent rule mutation completes without waiting for the pass. DNS serving was never affected and stays covered by the existing tests. ### 4. Truncated responses are never cached -`src/cache/dns_cache.zig:106-131`: `classify` reads `rcode` and `ancount` -and never `flags.tc`, so a TC=1 answer from a misbehaving upstream is cached -for up to `max_ttl_seconds` (86 400 s) and re-served — TC bit intact, even -over TCP, which can loop retrying clients (RFC 2181 §9). Add `if -(p.header.flags.tc) return null;` (`flags.tc` is `src/dns/header.zig:20`). +`src/cache/dns_cache.zig:106-131`: `classify` reads `rcode` and `ancount` and never `flags.tc`, so a TC=1 answer from a misbehaving upstream is cached for up to `max_ttl_seconds` (86 400 s) and re-served — TC bit intact, even over TCP, which can loop retrying clients (RFC 2181 §9). Add `if (p.header.flags.tc) return null;` (`flags.tc` is `src/dns/header.zig:20`). -`transport.validateResponse` is deliberately untouched: rejecting TC there -would change failover semantics for every exchange, and the forward client -legitimately reads TC for its UDP-to-TCP retry (`forward_client.zig:178`). -Record: considered, rejected. Unit test beside the existing classify tests: -a TC=1 NOERROR response classifies as null. +`transport.validateResponse` is deliberately untouched: rejecting TC there would change failover semantics for every exchange, and the forward client legitimately reads TC for its UDP-to-TCP retry (`forward_client.zig:178`). Record: considered, rejected. Unit test beside the existing classify tests: a TC=1 NOERROR response classifies as null. ### 5. The format sniffer stops eating hosts files with `##` banners -`src/filter/parsers.zig:54-73`: `hasAbpMarker` runs before `isComment`, and -`isElementHiding` (:86-93) matches `##` (and `#@#`, `#?#`, `#$#`, `#%#`) -unanchored with no comment guard — unlike the `$` branch two lines later, -which is guarded. A hosts file with a `##` banner parses whole as ABP: -`0.0.0.0` enters the domain set and hosts lines with inline URL comments are -silently dropped (real blocking loss, reproduced with the URLhaus banner -style). The milestone-5 as-built note (specs/milestone-5.md:1677-1678) -already *claims* the separators are matched anchored — the code never was. -Guarding the branch with `!isComment` is a circular no-op (`isComment` -consults `isElementHiding`); do not attempt it. +`src/filter/parsers.zig:54-73`: `hasAbpMarker` runs before `isComment`, and `isElementHiding` (:86-93) matches `##` (and `#@#`, `#?#`, `#$#`, `#%#`) unanchored with no comment guard — unlike the `$` branch two lines later, which is guarded. A hosts file with a `##` banner parses whole as ABP: `0.0.0.0` enters the domain set and hosts lines with inline URL comments are silently dropped (real blocking loss, reproduced with the URLhaus banner style). The milestone-5 as-built note (specs/milestone-5.md:1677-1678) already *claims* the separators are matched anchored — the code never was. Guarding the branch with `!isComment` is a circular no-op (`isComment` consults `isElementHiding`); do not attempt it. -Fix — a positional predicate in `isElementHiding`: a separator at offset `p` -counts as element hiding only when +Fix — a positional predicate in `isElementHiding`: a separator at offset `p` counts as element hiding only when -- `p == 0` and the character after the separator is not whitespace, `#`, or - end of line (a generic rule `##.ad` counts; a banner `## Title`, `####`, - or bare `##` does not), or -- `p > 0` and `line[p - 1]` is not whitespace and not `#` (a rule - `example.com##.ad` counts; prose `see ## below` does not). +- `p == 0` and the character after the separator is not whitespace, `#`, or end of line (a generic rule `##.ad` counts; a banner `## Title`, `####`, or bare `##` does not), or +- `p > 0` and `line[p - 1]` is not whitespace and not `#` (a rule `example.com##.ad` counts; prose `see ## below` does not). -Required test cases, in `parsers.zig`: the URLhaus banner style sniffs as -hosts; `##.ad-banner` sniffs as ABP; `example.com##.ad` sniffs as ABP; -`#@#exception` after a domain sniffs as ABP; a `#`-initial line containing -`##` sniffs as a comment. The fuzz target (`blocklist_fuzz.zig`) already -covers `detectFormat` for crashes. +Required test cases, in `parsers.zig`: the URLhaus banner style sniffs as hosts; `##.ad-banner` sniffs as ABP; `example.com##.ad` sniffs as ABP; `#@#exception` after a domain sniffs as ABP; a `#`-initial line containing `##` sniffs as a comment. The fuzz target (`blocklist_fuzz.zig`) already covers `detectFormat` for crashes. ### 6. VACUUM respects the disk monitor -`src/storage/retention.zig`: `runOnce` (:78) does prune → checkpoint → -(every 7th pass, :94-95) VACUUM, and takes no monitor — the word does not -appear in the file — while its two sibling tasks gate on -`disk_monitor.writesAllowed()` (logger.zig:386 as a parameter, -manager.zig:1058 as a field). VACUUM is the most expensive write the -program makes and fires unconditionally on the exact filesystem the monitor -watches, then fails SQLITE_FULL with no retry for seven daily passes. +`src/storage/retention.zig`: `runOnce` (:78) does prune → checkpoint → (every 7th pass, :94-95) VACUUM, and takes no monitor — the word does not appear in the file — while its two sibling tasks gate on `disk_monitor.writesAllowed()` (logger.zig:386 as a parameter, manager.zig:1058 as a field). VACUUM is the most expensive write the program makes and fires unconditionally on the exact filesystem the monitor watches, then fails SQLITE_FULL with no retry for seven daily passes. -`runOnce` and `run` gain `monitor: ?*disk_monitor.Monitor` (the logger's -parameter shape). Only the VACUUM step gates: prune and checkpoint keep -running — they free space. A skipped vacuum increments a new -`Stats.vacuums_gated` and retries on the *next* pass (drop the modulo-only -trigger for a `passes_since_vacuum >= vacuum_every_passes` counter that -resets on success). `app.zig:561` passes the monitor like :560 does for the -logger. Tests: extend the cadence tests (:223) with a gated pass — the -vacuum is skipped, counted, and runs on the next allowed pass. +`runOnce` and `run` gain `monitor: ?*disk_monitor.Monitor` (the logger's parameter shape). Only the VACUUM step gates: prune and checkpoint keep running — they free space. A skipped vacuum increments a new `Stats.vacuums_gated` and retries on the *next* pass (drop the modulo-only trigger for a `passes_since_vacuum >= vacuum_every_passes` counter that resets on success). `app.zig:561` passes the monitor like :560 does for the logger. Tests: extend the cadence tests (:223) with a gated pass — the vacuum is skipped, counted, and runs on the next allowed pass. ### 7. An oversized Cookie header degrades loudly, and the session survives -`src/web/server.zig:612-620` (`copyHeader`): a header value over the buffer -(cookie buffer = `http_util.max_cookie_len` = 1024, http_util.zig:35) -returns `""` with no log — under the documented reverse-proxy-on-shared- -domain deployment (foreign cookies riding along), every request silently -401s and the operator sees an unexplained login loop. +`src/web/server.zig:612-620` (`copyHeader`): a header value over the buffer (cookie buffer = `http_util.max_cookie_len` = 1024, http_util.zig:35) returns `""` with no log — under the documented reverse-proxy-on-shared- domain deployment (foreign cookies riding along), every request silently 401s and the operator sees an unexplained login loop. -`copyHeader` stays generic. The cookie call site (:510) changes: when the -raw header value exceeds the buffer, extract only the session pair by -running `http_util.cookieValue` (:200 — it already parses pairs and returns -a slice of the original header) with the existing session-cookie name -constant against the full value, copy that pair into the buffer as -`=`, and emit one `log.debug` with the dropped size (the -`.web_server` scope at :55; no secret is logged). If even the pair does not -fit, keep the empty result but still log. Integration test -(`web_integration_test.zig`): a request with a 2 KiB cookie header whose -session pair is valid is authenticated; the same header without the pair -401s. +`copyHeader` stays generic. The cookie call site (:510) changes: when the raw header value exceeds the buffer, extract only the session pair by running `http_util.cookieValue` (:200 — it already parses pairs and returns a slice of the original header) with the existing session-cookie name constant against the full value, copy that pair into the buffer as `=`, and emit one `log.debug` with the dropped size (the `.web_server` scope at :55; no secret is logged). If even the pair does not fit, keep the empty result but still log. Integration test (`web_integration_test.zig`): a request with a 2 KiB cookie header whose session pair is valid is authenticated; the same header without the pair 401s. ### 8. Live view detects fatal SSE rejections -`web/src/features/live/useLiveQueries.ts`: per the WHATWG spec a non-200 -response fails an EventSource permanently after one error event, so the -3-consecutive-errors threshold (:29, :125-133) is unreachable on exactly the -429/401 paths it was built for — the UI shows "Reconnecting…" forever and -the capped state and session probe are dead. `FakeEventSource` has no -readyState, so tests pass — the mocked-network class again. +`web/src/features/live/useLiveQueries.ts`: per the WHATWG spec a non-200 response fails an EventSource permanently after one error event, so the 3-consecutive-errors threshold (:29, :125-133) is unreachable on exactly the 429/401 paths it was built for — the UI shows "Reconnecting…" forever and the capped state and session probe are dead. `FakeEventSource` has no readyState, so tests pass — the mocked-network class again. -- `EventSourceLike` (:9-13) gains `readyState: number`; export `const - EVENT_SOURCE_CLOSED = 2`. The browser `EventSource` satisfies it - structurally. -- In the error handler: `readyState === EVENT_SOURCE_CLOSED` is a permanent - failure — close, set `capped`, run the session probe immediately, - bypassing the counter. Transient errors (browser auto-retry pending) keep - the existing counter path. -- `fakeEventSource.ts` gains `readyState` with the real lifecycle - (CONNECTING → OPEN on `emit("open")`, CLOSED on `close()` and on - `failFatal()`, a new test helper). -- New tests: a fatal rejection (single error event, readyState CLOSED) - reaches `capped` and calls `probeSession`; a transient error still takes - three to trip. +- `EventSourceLike` (:9-13) gains `readyState: number`; export `const EVENT_SOURCE_CLOSED = 2`. The browser `EventSource` satisfies it structurally. +- In the error handler: `readyState === EVENT_SOURCE_CLOSED` is a permanent failure — close, set `capped`, run the session probe immediately, bypassing the counter. Transient errors (browser auto-retry pending) keep the existing counter path. +- `fakeEventSource.ts` gains `readyState` with the real lifecycle (CONNECTING → OPEN on `emit("open")`, CLOSED on `close()` and on `failFatal()`, a new test helper). +- New tests: a fatal rejection (single error event, readyState CLOSED) reaches `capped` and calls `probeSession`; a transient error still takes three to trip. ### 9. A timed-out DoH handshake is a handshake failure -`src/server/doh_server.zig:311-315` counts a handshake-race timeout as -`idle_timeouts`; DoT (:313-317) counts it as `tls_handshake_failures`, -which is the recorded spec ruling. Since DoH requests have no other timer, -its `idle_timeouts` metric can only ever mean handshake stalls — the same -exported name with disjoint semantics per listener. Align DoH's -`.timed_out` arm to `tls_handshake_failures`. +`src/server/doh_server.zig:311-315` counts a handshake-race timeout as `idle_timeouts`; DoT (:313-317) counts it as `tls_handshake_failures`, which is the recorded spec ruling. Since DoH requests have no other timer, its `idle_timeouts` metric can only ever mean handshake stalls — the same exported name with disjoint semantics per listener. Align DoH's `.timed_out` arm to `tls_handshake_failures`. ### 10. Idle DoH keep-alive connections are reclaimed -`src/server/doh_server.zig:70`: `idle_timeout` bounds only the handshake; -its own doc comment admits requests have none. DoH stubs hold keep-alives by -design, and with no TCP keepalive a vanished peer pins one of 64 slots until -restart — while DoT on the same LAN reclaims after 10 s. +`src/server/doh_server.zig:70`: `idle_timeout` bounds only the handshake; its own doc comment admits requests have none. DoH stubs hold keep-alives by design, and with no TCP keepalive a vanished peer pins one of 64 slots until restart — while DoT on the same LAN reclaims after 10 s. -Extend the existing race to `receiveHead` (:333), the wait for the next -request on a keep-alive connection, using the DoT out-param precedent -(readPrefix's `out_len`): a small wrapper writes the received `Request` (or -its error) through a pointer so the raced function stays `anyerror!void`. -On `.timed_out`: bump `idle_timeouts` (now truthfully named again after -ruling 9) and close the connection. Preserve the existing error triage at +Extend the existing race to `receiveHead` (:333), the wait for the next request on a keep-alive connection, using the DoT out-param precedent (readPrefix's `out_len`): a small wrapper writes the received `Request` (or its error) through a pointer so the raced function stays `anyerror!void`. On `.timed_out`: bump `idle_timeouts` (now truthfully named again after ruling 9) and close the connection. Preserve the existing error triage at :334-346 (`HttpConnectionClosing` and `ReadFailed` return uncounted; the -three structural errors count `bad_requests`/`connection_errors` as today). -The body read and `handleRequest` stay untimed, like the web listener. +three structural errors count `bad_requests`/`connection_errors` as today). The body read and `handleRequest` stay untimed, like the web listener. -New integration-gated test mirroring the DoT idle test -(dot_server.zig:1098): a client that completes the handshake and one request -then goes quiet is closed after a short idle budget; `idle_timeouts == 1`, -`tls_handshake_failures == 0`. DoH currently has no idle test at all. +New integration-gated test mirroring the DoT idle test (dot_server.zig:1098): a client that completes the handshake and one request then goes quiet is closed after a short idle budget; `idle_timeouts == 1`, `tls_handshake_failures == 0`. DoH currently has no idle test at all. ### 11. SSE shutdown does not wait for a heartbeat -`src/web/sse.zig`: the Hub has no shutdown signal, so graceful drain parks -up to 15 s (`heartbeat_interval`, web/handlers/live.zig:32) per idle -subscriber — the web `beginShutdown` (web/server.zig:593-605) shuts down -sockets but nothing wakes a task inside `Hub.wait`. Measured: the SSE -integration test budgets 40 s for exactly this -(web_integration_test.zig:74-75). +`src/web/sse.zig`: the Hub has no shutdown signal, so graceful drain parks up to 15 s (`heartbeat_interval`, web/handlers/live.zig:32) per idle subscriber — the web `beginShutdown` (web/server.zig:593-605) shuts down sockets but nothing wakes a task inside `Hub.wait`. Measured: the SSE integration test budgets 40 s for exactly this (web_integration_test.zig:74-75). -- `Wake` (:33) gains `.closed`. Hub gains `closing: bool` and `pub fn - close(self: *Hub, io: std.Io) void`: under the mutex, set the flag and - `event.set` every active slot. `wait` returns `.closed` immediately when - the flag is set. +- `Wake` (:33) gains `.closed`. Hub gains `closing: bool` and `pub fn close(self: *Hub, io: std.Io) void`: under the mutex, set the flag and `event.set` every active slot. `wait` returns `.closed` immediately when the flag is set. - `live.zig:114`: `.closed => return`. -- `web/server.zig` `deinit` (:349): call the hub's close (via the state's - hub pointer) immediately before `beginShutdown` (:360). -- Test: a subscriber parked in `wait` returns `.closed` promptly after - `close`; the W10 integration teardown no longer stalls (tighten - `sse_budget` only if the heartbeat assertion itself allows it). +- `web/server.zig` `deinit` (:349): call the hub's close (via the state's hub pointer) immediately before `beginShutdown` (:360). +- Test: a subscriber parked in `wait` returns `.closed` promptly after `close`; the W10 integration teardown no longer stalls (tighten `sse_budget` only if the heartbeat assertion itself allows it). ### 12. The API limiter's sweep runs -`src/web/api_limiter.zig:210`: `sweep` exists, preallocates -`stale_keys` so it never allocates, is tested — and has no production -caller; only the DNS limiter got scheduled. Once 4096 distinct addresses -have been seen, the table stays full forever and every unknown-address -request pays an O(4096) eviction scan under the limiter mutex. +`src/web/api_limiter.zig:210`: `sweep` exists, preallocates `stale_keys` so it never allocates, is tested — and has no production caller; only the DNS limiter got scheduled. Once 4096 distinct addresses have been seen, the table stays full forever and every unknown-address request pays an O(4096) eviction scan under the limiter mutex. -`runMaintenance` (app.zig:709) gains an `api_limiter: ?*ApiLimiter` -parameter, wired at :565 from the instance built at :380-388. In the loop, -beside the two existing tasks: `_ = api.sweep(io, Clock.awake.now(io));` — -note it locks itself, unlike the DNS limiter. Test: the maintenance-loop -integration coverage asserts a stale bucket disappears. +`runMaintenance` (app.zig:709) gains an `api_limiter: ?*ApiLimiter` parameter, wired at :565 from the instance built at :380-388. In the loop, beside the two existing tasks: `_ = api.sweep(io, Clock.awake.now(io));` — note it locks itself, unlike the DNS limiter. Test: the maintenance-loop integration coverage asserts a stale bucket disappears. ### 13. UDP/53 and TCP/53 stats reach /metrics -`src/server/udp_server.zig:38-49` and `tcp_server.zig:54-60`: both stats -structs are written and read by nothing outside their own integration -tests — `dropped_no_slot`/`dropped_oversize` have no metric, no API field, -and errors log below the default level, while the DoT/DoH siblings export -equivalent counters. The module doc sells "dropped and counted"; the count -is unobservable. +`src/server/udp_server.zig:38-49` and `tcp_server.zig:54-60`: both stats structs are written and read by nothing outside their own integration tests — `dropped_no_slot`/`dropped_oversize` have no metric, no API field, and errors log below the default level, while the DoT/DoH siblings export equivalent counters. The module doc sells "dropped and counted"; the count is unobservable. -- Both gain `pub const Snapshot` + `pub fn snapshotStats` in the exact DoT - shape (dot_server.zig:61, :232). Field names stay as-is (the - `accepted`/`connections` unification is milestone-18 work). -- `WebState` (src/web/server.zig) gains `udp_listeners: []const - *udp_server.UdpServer = &.{}` and `tcp_listeners: []const - *tcp_server.TcpServer = &.{}` — S4 adds the fields with exactly these - names and types; S2 consumes them (the app builds four listeners: - udp6/udp4/tcp6/tcp4, app.zig:506-528). -- `metrics.zig`: sum each family across its listeners into one - `nxdns_udp_server_*_total` / `nxdns_tcp_server_*_total` group via the - existing `counterGroup` derivation; extend the name-assertion test - (:722-743). +- Both gain `pub const Snapshot` + `pub fn snapshotStats` in the exact DoT shape (dot_server.zig:61, :232). Field names stay as-is (the `accepted`/`connections` unification is milestone-18 work). +- `WebState` (src/web/server.zig) gains `udp_listeners: []const *udp_server.UdpServer = &.{}` and `tcp_listeners: []const *tcp_server.TcpServer = &.{}` — S4 adds the fields with exactly these names and types; S2 consumes them (the app builds four listeners: udp6/udp4/tcp6/tcp4, app.zig:506-528). +- `metrics.zig`: sum each family across its listeners into one `nxdns_udp_server_*_total` / `nxdns_tcp_server_*_total` group via the existing `counterGroup` derivation; extend the name-assertion test (:722-743). - `app.zig` wires the slices. ### 14. Forward-client counters survive the query -`src/local/forward_client.zig:44-57`: `Stats` (queries, udp_truncated, -foreign_datagrams, failures) is instrumented, documented as preventing "an -unrecorded failure mode" — and the only production caller builds a -stack-local client per query (handler.zig:394) and drops it, so the spoofing -signal does not exist. +`src/local/forward_client.zig:44-57`: `Stats` (queries, udp_truncated, foreign_datagrams, failures) is instrumented, documented as preventing "an unrecorded failure mode" — and the only production caller builds a stack-local client per query (handler.zig:394) and drops it, so the spoofing signal does not exist. -`Handler.Stats` (handler.zig:130-154) gains `forward_udp_truncated`, -`forward_foreign_datagrams`, `forward_failures` (atomic, like the other 17; -queries are already counted by `forward_zone_answers`). After the exchange -in `viaForwardZone`, fetchAdd the client's counters into them. The metrics -reflection (`dns_stat_fields`, metrics.zig:48) picks the new fields up -without an edit — assert the three new `nxdns_dns_*_total` names in the -metrics test. `ForwardClient.Stats` itself stays plain and per-instance. +`Handler.Stats` (handler.zig:130-154) gains `forward_udp_truncated`, `forward_foreign_datagrams`, `forward_failures` (atomic, like the other 17; queries are already counted by `forward_zone_answers`). After the exchange in `viaForwardZone`, fetchAdd the client's counters into them. The metrics reflection (`dns_stat_fields`, metrics.zig:48) picks the new fields up without an edit — assert the three new `nxdns_dns_*_total` names in the metrics test. `ForwardClient.Stats` itself stays plain and per-instance. ### 15. The TLS server keeps the concrete transport cause -`src/platform/tls_server.zig:418-438`: both BIO callbacks discard the -stashed cause — `net_reader.err` / `net_writer.err` (which hold -Reset/Timeout/**Canceled**, std Io/net.zig:1260/:1324) are never read -anywhere in the file — so a routine idle-budget cancel surfaces as a warn -"mbedtls_ssl_read failed" and peer resets are indistinguishable from -timeouts. The client side solved exactly this (dot_client.zig -`concreteRead`/`concreteWrite`, :303-318). +`src/platform/tls_server.zig:418-438`: both BIO callbacks discard the stashed cause — `net_reader.err` / `net_writer.err` (which hold Reset/Timeout/**Canceled**, std Io/net.zig:1260/:1324) are never read anywhere in the file — so a routine idle-budget cancel surfaces as a warn "mbedtls_ssl_read failed" and peer resets are indistinguishable from timeouts. The client side solved exactly this (dot_client.zig `concreteRead`/`concreteWrite`, :303-318). -- `ServerStream` gains `recv_cause: ?anyerror = null`, `send_cause: - ?anyerror = null`; `bioRecv`/`bioSend` stash - `self.net_reader.err`/`self.net_writer.err` on failure before returning - the mbedtls code. -- `ReadError` (:216-223) widens with `Canceled`; the read path - (`readIntoBuffer`, :334) maps a stashed `error.Canceled` to it instead of - `TlsFailed`. Callers' race harnesses already treat cancellation - separately. -- Unit tests in the file's existing stub style: a reader whose `err` is - Canceled yields `error.Canceled`; a Reset yields `TlsFailed` with the - cause stashed for logging (ruling 16). +- `ServerStream` gains `recv_cause: ?anyerror = null`, `send_cause: ?anyerror = null`; `bioRecv`/`bioSend` stash `self.net_reader.err`/`self.net_writer.err` on failure before returning the mbedtls code. +- `ReadError` (:216-223) widens with `Canceled`; the read path (`readIntoBuffer`, :334) maps a stashed `error.Canceled` to it instead of `TlsFailed`. Callers' race harnesses already treat cancellation separately. +- Unit tests in the file's existing stub style: a reader whose `err` is Canceled yields `error.Canceled`; a Reset yields `TlsFailed` with the cause stashed for logging (ruling 16). ### 16. Peer misbehavior logs at debug, like the read path already rules -`src/platform/tls_server.zig`: the read path deliberately logs a peer TCP -drop at debug (:362-364, "a louder level would be a log-spam vector"), then -`close` warns on close_notify against the same dead socket (:311) and -`handshake` warns per probe (:287). `.tls_server` is not in the dedup scope -set (logging.zig:103-108), so peer-driven warns can evict genuine warnings -from the rotating log. +`src/platform/tls_server.zig`: the read path deliberately logs a peer TCP drop at debug (:362-364, "a louder level would be a log-spam vector"), then `close` warns on close_notify against the same dead socket (:311) and `handshake` warns per probe (:287). `.tls_server` is not in the dedup scope set (logging.zig:103-108), so peer-driven warns can evict genuine warnings from the rotating log. -- `report` (:474-479) gains a level parameter. The close_notify site passes - debug when `peer_closed` is set or the stashed cause (ruling 15) is a - peer-class error; the handshake site likewise. Local/config failures keep - warn. -- Add `.tls_server` to `isDedupScope` (logging.zig:103-108) and to its - membership test (:637-644). +- `report` (:474-479) gains a level parameter. The close_notify site passes debug when `peer_closed` is set or the stashed cause (ruling 15) is a peer-class error; the handshake site likewise. Local/config failures keep warn. +- Add `.tls_server` to `isDedupScope` (logging.zig:103-108) and to its membership test (:637-644). -**Scope widened during implementation (orchestrator ruling):** the level -choice also applies to the `ssl_read` and `ssl_write` report sites, not only -the two named ones. Ruling 15's "cause stashed for logging" wording requires -the read site to consume the level, and an unconditional warn on the write -half would keep the same spam vector open through `close` → `flush` against -a dead peer. Peer-class causes are decided by pure helpers (`isPeerCause`, -`causeLevel`); `check` still passes warn. +**Scope widened during implementation (orchestrator ruling):** the level choice also applies to the `ssl_read` and `ssl_write` report sites, not only the two named ones. Ruling 15's "cause stashed for logging" wording requires the read site to consume the level, and an unconditional warn on the write half would keep the same spam vector open through `close` → `flush` against a dead peer. Peer-class causes are decided by pure helpers (`isPeerCause`, `causeLevel`); `check` still passes warn. ### 17. The query log heals its own gaps -`web/src/features/queries/QueryLogPage.tsx`: the hand-rolled accumulation -(`extra`, `cursorOverride`, `generation`, :79-98) develops a silent -mid-table row gap when the base page refetches after 30 s staleness — the -newest-100 boundary moves up while `extra` starts below the old cursor, and -the stale override means load-more never heals it. On a live DNS server the -trigger is routine. +`web/src/features/queries/QueryLogPage.tsx`: the hand-rolled accumulation (`extra`, `cursorOverride`, `generation`, :79-98) develops a silent mid-table row gap when the base page refetches after 30 s staleness — the newest-100 boundary moves up while `extra` starts below the old cursor, and the stale override means load-more never heals it. On a live DNS server the trigger is routine. -Migrate to `useInfiniteQuery`: the endpoint already maps onto it -(`getNextPageParam: (last) => last.next_before ?? undefined`; keyset -pagination confirmed server-side, handlers/queries.zig:103-114). A -background refetch then refetches all pages in order — consistent, no gap. -The three behaviors the old code carried by hand: staleness discard is -handled by the query itself; keep the `isPlaceholderData` disable on the -button; the 401 branch is already covered by the global cache-level -`handleUnauthorized` (queryClient.ts:29-30). Delete `extra`, -`cursorOverride`, `generation`, `loadingMore`, `moreError`. Rewrite the five -load-more tests; add one for the gap scenario: base refetch with new rows -between page renders leaves no discontinuity. +Migrate to `useInfiniteQuery`: the endpoint already maps onto it (`getNextPageParam: (last) => last.next_before ?? undefined`; keyset pagination confirmed server-side, handlers/queries.zig:103-114). A background refetch then refetches all pages in order — consistent, no gap. The three behaviors the old code carried by hand: staleness discard is handled by the query itself; keep the `isPlaceholderData` disable on the button; the 401 branch is already covered by the global cache-level `handleUnauthorized` (queryClient.ts:29-30). Delete `extra`, `cursorOverride`, `generation`, `loadingMore`, `moreError`. Rewrite the five load-more tests; add one for the gap scenario: base refetch with new rows between page renders leaves no discontinuity. ### 18. Password hashing leaves the config lock -`src/web/handlers/settings.zig`: `applyPut` holds `config_lock` from :286 -to :341, and the argon2id call (t=2, m=19 MiB, :373-391) sits inside it at +`src/web/handlers/settings.zig`: `applyPut` holds `config_lock` from :286 to :341, and the argon2id call (t=2, m=19 MiB, :373-391) sits inside it at :312 — every settings GET (:407-409 takes the same lock) and every mutation -stalls for the hash duration on the Pi 5. The login path already does it -right: copy under lock, hash unlocked, re-check generation under lock -(auth.zig:47-71), and the LiveHash generation check (auth.zig:73-79) -already closes the concurrent-install race. +stalls for the hash duration on the Pi 5. The login path already does it right: copy under lock, hash unlocked, re-check generation under lock (auth.zig:47-71), and the LiveHash generation check (auth.zig:73-79) already closes the concurrent-install race. -Move the password-length check and the `hashPassword` call before the lock -acquisition; everything from `loadConfig` on stays inside. The hash input -is the parsed patch only, so nothing under the lock is needed. Tests: the -existing applyPut suite passes unchanged; add one that actually -distinguishes the new behavior — "both complete" already held before the -fix, the GET merely waited out the hash. A stall seam under -`builtin.is_test` (the milestone-15 rotation-seam shape) parks -`hashPassword` on an event; the test starts a password PUT, waits until -the hash is parked, completes a settings GET **while the hash is still -parked**, then releases the seam and joins the PUT. +Move the password-length check and the `hashPassword` call before the lock acquisition; everything from `loadConfig` on stays inside. The hash input is the parsed patch only, so nothing under the lock is needed. Tests: the existing applyPut suite passes unchanged; add one that actually distinguishes the new behavior — "both complete" already held before the fix, the GET merely waited out the hash. A stall seam under `builtin.is_test` (the milestone-15 rotation-seam shape) parks `hashPassword` on an event; the test starts a password PUT, waits until the hash is parked, completes a settings GET **while the hash is still parked**, then releases the seam and joins the PUT. ## Sessions -S1-S5 run in parallel. One cross-session interface, fixed here: S4 adds the -`WebState.udp_listeners`/`tcp_listeners` fields exactly as ruling 13 names -them; S2 wires and consumes them without touching `web/server.zig`. +S1-S5 run in parallel. One cross-session interface, fixed here: S4 adds the `WebState.udp_listeners`/`tcp_listeners` fields exactly as ruling 13 names them; S2 wires and consumes them without touching `web/server.zig`. ### Session S1: filter -Owns `src/filter/manager.zig`, `src/filter/parsers.zig`, -`src/filter/filter_integration_test.zig`. Rulings 1, 2, 3, 5. +Owns `src/filter/manager.zig`, `src/filter/parsers.zig`, `src/filter/filter_integration_test.zig`. Rulings 1, 2, 3, 5. ### Session S2: cache, retention, counters, wiring -Owns `src/cache/dns_cache.zig`, `src/storage/retention.zig`, `src/app.zig`, -`src/server/udp_server.zig`, `src/server/tcp_server.zig`, -`src/local/forward_client.zig`, `src/server/handler.zig`, -`src/web/metrics.zig`, and the storage/server integration tests those -touch. Rulings 4, 6, 12, 13 (except the WebState fields), 14. +Owns `src/cache/dns_cache.zig`, `src/storage/retention.zig`, `src/app.zig`, `src/server/udp_server.zig`, `src/server/tcp_server.zig`, `src/local/forward_client.zig`, `src/server/handler.zig`, `src/web/metrics.zig`, and the storage/server integration tests those touch. Rulings 4, 6, 12, 13 (except the WebState fields), 14. ### Session S3: TLS listeners -Owns `src/platform/tls_server.zig`, `src/platform/logging.zig`, -`src/server/doh_server.zig`. Rulings 9, 10, 15, 16. +Owns `src/platform/tls_server.zig`, `src/platform/logging.zig`, `src/server/doh_server.zig`. Rulings 9, 10, 15, 16. ### Session S4: web server -Owns `src/web/server.zig`, `src/web/sse.zig`, `src/web/handlers/live.zig`, -`src/web/handlers/settings.zig`, `src/web/web_integration_test.zig`, plus -the ruling-13 field additions. Rulings 7, 11, 18. +Owns `src/web/server.zig`, `src/web/sse.zig`, `src/web/handlers/live.zig`, `src/web/handlers/settings.zig`, `src/web/web_integration_test.zig`, plus the ruling-13 field additions. Rulings 7, 11, 18. ### Session S5: frontend @@ -405,8 +172,7 @@ Owns `web/src/features/live/*`, `web/src/features/queries/*`. Rulings 8, ### Orchestrator -Strikes the closed findings in `TECH_DEBT.md`; updates the stale -milestone-5 as-built line if ruling 5 changed its truth value. +Strikes the closed findings in `TECH_DEBT.md`; updates the stale milestone-5 as-built line if ruling 5 changed its truth value. ## Acceptance (milestone complete) @@ -447,8 +213,7 @@ milestone-5 as-built line if ruling 5 changed its truth value. ## Anti-requirements -- No listener-core extraction, no shared repo helpers, no shared transport - helpers — milestone 18. +- No listener-core extraction, no shared repo helpers, no shared transport helpers — milestone 18. - No stats field renames (`accepted` vs `connections`) — milestone 18. - No config key renames and no trusted-proxy setting — milestone 17. - No `useInfiniteQuery` migration anywhere but the query log. diff --git a/specs/milestone-17.md b/specs/milestone-17.md index f3e90d1..261a043 100644 --- a/specs/milestone-17.md +++ b/specs/milestone-17.md @@ -1,419 +1,134 @@ # Milestone 17: contract repairs -Goal: make the operator-facing contract true. Four decided items — a real -per-query deadline, the upstream editor milestone 8 promised, an opt-in -trusted-proxy setting, and a field-level contract guard between the server -and the frontend — plus BADVERS, the validator holes, and the doc claims -the code contradicts. `TECH_DEBT.md` Theme 4. All four decisions are made; -do not re-litigate them. +Goal: make the operator-facing contract true. Four decided items — a real per-query deadline, the upstream editor milestone 8 promised, an opt-in trusted-proxy setting, and a field-level contract guard between the server and the frontend — plus BADVERS, the validator holes, and the doc claims the code contradicts. `TECH_DEBT.md` Theme 4. All four decisions are made; do not re-litigate them. -**PROVISIONAL.** Written before milestones 15-16 were built. m15 touches -`docs_drift_test.zig` and `cli.zig`; m16 touches `pool.zig`'s neighbors, -the listeners and the settings handler. Re-verify line references before -each session starts. +**PROVISIONAL.** Written before milestones 15-16 were built. m15 touches `docs_drift_test.zig` and `cli.zig`; m16 touches `pool.zig`'s neighbors, the listeners and the settings handler. Re-verify line references before each session starts. ## Rulings (binding) ### 1. `total_timeout_ms` becomes a real per-query deadline -Today the key feeds `Pool.attempt_timeout` (app.zig:283-288 → -pool.zig:110) and bounds one attempt; with N upstreams down a query takes -N × 5000 ms. The reference docs contradict each other (configuration.md -says "per-query budget", cli.md says "per-attempt") and PLAN promises a -total budget. Decision: the total budget becomes real. +Today the key feeds `Pool.attempt_timeout` (app.zig:283-288 → pool.zig:110) and bounds one attempt; with N upstreams down a query takes N × 5000 ms. The reference docs contradict each other (configuration.md says "per-query budget", cli.md says "per-attempt") and PLAN promises a total budget. Decision: the total budget becomes real. -- `model.Upstream` gains `attempt_timeout_ms: u32 = 2500`; - `total_timeout_ms` keeps its name and default (5000) and becomes the - whole-exchange deadline. `read_timeout_ms` is untouched (it feeds the - forward-zone client only, app.zig:412). -- `pool.zig`: `attempt` keeps its race, now budgeted by the new key. The - two-pass failover loop is extracted into a private - `exchangeLoopLen(self, io, query, response_buf) transport.ExchangeError!usize` - (the `exchangeLen` precedent, pool.zig:690-693: `Io.concurrent` stores - the future's return value, so the raced loop returns a length and - `exchange` rebuilds the slice). A new outer race wraps that helper with - the total budget, using the select idiom already in the file. The outer - cancel unwinds an in-flight attempt — the `entry.busy` lock is taken - cancelably on purpose (:169-174) and released by defer. Outer expiry - returns `error.Timeout`, **unconditionally**: the fast-failure paths - (all upstreams disabled → `ConnectFailed`) finish long before the - budget, so no attempted-marker is needed, and a test pins that an - all-disabled pool still fails immediately rather than waiting out the - deadline. -- `validate.zig`: `checkTimeout` both keys; the cross-check becomes - `attempt_timeout_ms <= total_timeout_ms` (the old `total >= read` check, - which related knobs of different subsystems, is deleted). -- Settings plumbing for the new key (the reflective walk carries it; the - hand-written mirrors do not): `expected_keys` (model.zig:449, sorted), - the round-trip test with a non-default value, the `SettingsView` - upstream struct in web_integration_test.zig:528, `types.ts` - `Settings.upstream`, `openapi.yaml` Settings + SettingsPatch, - SettingsPage `SECTIONS[0]`, the configuration.md row (drift-guarded), - and the cli.md/configuration.md prose — both now say: total is the - per-query budget, attempt bounds one upstream try. -- Tests: a pool test with two stalling upstreams proves the exchange - returns within the total budget, not 2 × attempt; the existing - per-attempt tests keep passing against the new key. +- `model.Upstream` gains `attempt_timeout_ms: u32 = 2500`; `total_timeout_ms` keeps its name and default (5000) and becomes the whole-exchange deadline. `read_timeout_ms` is untouched (it feeds the forward-zone client only, app.zig:412). +- `pool.zig`: `attempt` keeps its race, now budgeted by the new key. The two-pass failover loop is extracted into a private `exchangeLoopLen(self, io, query, response_buf) transport.ExchangeError!usize` (the `exchangeLen` precedent, pool.zig:690-693: `Io.concurrent` stores the future's return value, so the raced loop returns a length and `exchange` rebuilds the slice). A new outer race wraps that helper with the total budget, using the select idiom already in the file. The outer cancel unwinds an in-flight attempt — the `entry.busy` lock is taken cancelably on purpose (:169-174) and released by defer. Outer expiry returns `error.Timeout`, **unconditionally**: the fast-failure paths (all upstreams disabled → `ConnectFailed`) finish long before the budget, so no attempted-marker is needed, and a test pins that an all-disabled pool still fails immediately rather than waiting out the deadline. +- `validate.zig`: `checkTimeout` both keys; the cross-check becomes `attempt_timeout_ms <= total_timeout_ms` (the old `total >= read` check, which related knobs of different subsystems, is deleted). +- Settings plumbing for the new key (the reflective walk carries it; the hand-written mirrors do not): `expected_keys` (model.zig:449, sorted), the round-trip test with a non-default value, the `SettingsView` upstream struct in web_integration_test.zig:528, `types.ts` `Settings.upstream`, `openapi.yaml` Settings + SettingsPatch, SettingsPage `SECTIONS[0]`, the configuration.md row (drift-guarded), and the cli.md/configuration.md prose — both now say: total is the per-query budget, attempt bounds one upstream try. +- Tests: a pool test with two stalling upstreams proves the exchange returns within the total budget, not 2 × attempt; the existing per-attempt tests keep passing against the new key. ### 2. The validator closes its two verified holes -- **Hostname `tls://` upstreams.** `Endpoint.parse` accepts any host - text; the DoT client then refuses non-literals on every dial - (`resolveAddress`, dot_client.zig:74-80), so a hostname `tls://` config - validates clean and fails at query time as a peer fault. Milestone 3 - explicitly ordered this check carried into the validator and it was - dropped. Correction to the audit: the function to mirror is **not** - `parseResolver` itself (validate.zig:257 — that is the forward-zone - resolver check); the upstream block in `checkCollections` - (validate.zig:603-639) gains, for `.dot` endpoints, - `net.IpAddress.parse(endpoint.host, endpoint.port) catch` → a new - `error.UpstreamHostNotIpLiteral` diagnostic ("tls:// upstreams take an - IP literal host"). `faults.zig` picks the variant up automatically. - Fix the broken example at configuration.md:348 — the hostname NextDNS - `tls://` config can never complete an exchange; show an IP-literal - `tls://` example and move the hostname form to `https://`. -- **Boot-allocation bounds.** `query_log_buffer_max` is floor-checked - only (an Entry is ~420 bytes; `maxInt(u32)` asks for ~1.8 TB at boot), - and `cache.size` is not validated at all (grep confirms). Both get the - file's ceiling-only idiom: `1..1_000_000` with the "must be at most" - message shape. No full memory-fit guarantee — out of reach, not needed. +- **Hostname `tls://` upstreams.** `Endpoint.parse` accepts any host text; the DoT client then refuses non-literals on every dial (`resolveAddress`, dot_client.zig:74-80), so a hostname `tls://` config validates clean and fails at query time as a peer fault. Milestone 3 explicitly ordered this check carried into the validator and it was dropped. Correction to the audit: the function to mirror is **not** `parseResolver` itself (validate.zig:257 — that is the forward-zone resolver check); the upstream block in `checkCollections` (validate.zig:603-639) gains, for `.dot` endpoints, `net.IpAddress.parse(endpoint.host, endpoint.port) catch` → a new `error.UpstreamHostNotIpLiteral` diagnostic ("tls:// upstreams take an IP literal host"). `faults.zig` picks the variant up automatically. Fix the broken example at configuration.md:348 — the hostname NextDNS `tls://` config can never complete an exchange; show an IP-literal `tls://` example and move the hostname form to `https://`. +- **Boot-allocation bounds.** `query_log_buffer_max` is floor-checked only (an Entry is ~420 bytes; `maxInt(u32)` asks for ~1.8 TB at boot), and `cache.size` is not validated at all (grep confirms). Both get the file's ceiling-only idiom: `1..1_000_000` with the "must be at most" message shape. No full memory-fit guarantee — out of reach, not needed. -**Recorded (implementation):** three consequences this ruling did not call -out. (a) `cache.size = 0` was a documented "disables caching" mode; the -`1..` floor removes it — configuration.md now says there is no off value -(use 1). (b) The configuration.md NextDNS-DoT redaction passage lost its -premise (a hostname `tls://` URL is now unconfigurable) and was rewritten. -(c) Three config fixtures used `tls://dot.example:853` and now use an IP -literal with `tls_name`; the back-up-and-restore.md export transcript grew -to `head -10` with real re-captured output. Also: `Pool.init` takes a named -`pool.Timeouts { attempt, total }` struct, not two positional durations — -two same-typed adjacent parameters would be silently swappable. +**Recorded (implementation):** three consequences this ruling did not call out. (a) `cache.size = 0` was a documented "disables caching" mode; the `1..` floor removes it — configuration.md now says there is no off value (use 1). (b) The configuration.md NextDNS-DoT redaction passage lost its premise (a hostname `tls://` URL is now unconfigurable) and was rewritten. (c) Three config fixtures used `tls://dot.example:853` and now use an IP literal with `tls_name`; the back-up-and-restore.md export transcript grew to `head -10` with real re-captured output. Also: `Pool.init` takes a named `pool.Timeouts { attempt, total }` struct, not two positional durations — two same-typed adjacent parameters would be silently swappable. ### 3. The upstream editor is built -Milestone 8 ruling 9 requires upstream CRUD in the UI ("the Settings page -must edit them"); milestone 9 dropped the obligation when the SPA was -built. The entire data layer exists with zero importers -(`upstreamsQuery`, three mutation factories at queries.ts:251-264, four -api.ts wrappers). Decision: build it. Placement deviates from m8's -literal text: a dedicated **Upstreams page** (nav entry), matching the -house resource-page pattern — record the deviation beside m8 ruling 9. +Milestone 8 ruling 9 requires upstream CRUD in the UI ("the Settings page must edit them"); milestone 9 dropped the obligation when the SPA was built. The entire data layer exists with zero importers (`upstreamsQuery`, three mutation factories at queries.ts:251-264, four api.ts wrappers). Decision: build it. Placement deviates from m8's literal text: a dedicated **Upstreams page** (nav entry), matching the house resource-page pattern — record the deviation beside m8 ruling 9. -- New `web/src/features/upstreams/UpstreamsPage.tsx` + - `UpstreamForm.tsx`, modeled on the blocklists pair (the closest - analogue; copy its named idioms: editing-state-as-row, two mutation - instances of one factory so row-toggle state never bleeds into the - form, `mutateAsync` submit, full-row resend on toggle because **PUT is - a replace**, `key={editing?.id ?? "add"}` remount, `window.confirm` - delete, split error routing). -- Fields: url, priority (number), enabled (toggle), tls_name. 400s carry - the real validator's text (the handler validates via - `mutations.checkUpstream`); surface the three 409 conflicts verbatim - ("an upstream with that url already exists", "the last enabled - upstream cannot be disabled/removed"). -- Every mutation response carries `restart_required: true` and today - nothing raises the banner outside SettingsPage. Generalize the restart - banner copy to "Changes saved. Restart nxdns to apply." and raise it on - upstream create/update/delete success. -- Wiring is three edits: `createRoute` in routes.tsx, the `routeTree` - entry, the AppShell nav item. The health table stays on the Dashboard - (health rows have no `id` — url is the only join key — and reflect the - running pool, so a new upstream appears there only after restart; the - page states this beside the restart banner). -- Dead-layer cleanup in the same stroke (the same finding): delete the - seven unused single-row `get*` wrappers (correction: seven, not five — - `getGroup` and `getUpstream` are shadowed by longer names in grep), - plus `getMetrics`, `getOpenapiYaml`, and `ruleUpdateMutation`. The - editor reads rows from the list query per house idiom, so none gains a - caller. Git history keeps them. -- Page tests in the stubbed-fetch house style: list render, add, toggle - resends the full row, delete confirms, a 409 renders inline, the - banner raises. +- New `web/src/features/upstreams/UpstreamsPage.tsx` + `UpstreamForm.tsx`, modeled on the blocklists pair (the closest analogue; copy its named idioms: editing-state-as-row, two mutation instances of one factory so row-toggle state never bleeds into the form, `mutateAsync` submit, full-row resend on toggle because **PUT is a replace**, `key={editing?.id ?? "add"}` remount, `window.confirm` delete, split error routing). +- Fields: url, priority (number), enabled (toggle), tls_name. 400s carry the real validator's text (the handler validates via `mutations.checkUpstream`); surface the three 409 conflicts verbatim ("an upstream with that url already exists", "the last enabled upstream cannot be disabled/removed"). +- Every mutation response carries `restart_required: true` and today nothing raises the banner outside SettingsPage. Generalize the restart banner copy to "Changes saved. Restart nxdns to apply." and raise it on upstream create/update/delete success. +- Wiring is three edits: `createRoute` in routes.tsx, the `routeTree` entry, the AppShell nav item. The health table stays on the Dashboard (health rows have no `id` — url is the only join key — and reflect the running pool, so a new upstream appears there only after restart; the page states this beside the restart banner). +- Dead-layer cleanup in the same stroke (the same finding): delete the seven unused single-row `get*` wrappers (correction: seven, not five — `getGroup` and `getUpstream` are shadowed by longer names in grep), plus `getMetrics`, `getOpenapiYaml`, and `ruleUpdateMutation`. The editor reads rows from the list query per house idiom, so none gains a caller. Git history keeps them. +- Page tests in the stubbed-fetch house style: list render, add, toggle resends the full row, delete confirms, a 409 renders inline, the banner raises. ### 4. Trusted proxies restore real client identity -Behind the documented same-box reverse proxy every request is loopback: -`localhost_exempt = true` (default) then disables the API limiter — the -only brake on argon2 brute force — and all remote users share one -address's 3-stream SSE cap. No X-Forwarded-For handling exists anywhere -(verified repo-wide). Decision: opt-in trusted-proxy setting. +Behind the documented same-box reverse proxy every request is loopback: `localhost_exempt = true` (default) then disables the API limiter — the only brake on argon2 brute force — and all remote users share one address's 3-stream SSE cap. No X-Forwarded-For handling exists anywhere (verified repo-wide). Decision: opt-in trusted-proxy setting. -- New setting `web.trusted_proxies: []const u8 = ""` — a comma-separated - list of IP literals in one string (the settings codec supports scalars - only; a list type is a compile error at model.zig:358). Empty means - off: behavior today. -- Validation: each comma-separated element must parse as an IP literal → - new `error.BadTrustedProxy` diagnostic in the web block of - `checkScalars`. -- Mechanics: when the socket peer is in the trusted set, the effective - client address is the **last** valid IP literal in the request's - `X-Forwarded-For` (the entry our proxy appended); a **missing** header - falls back to the socket peer. Implementation follows the - copy-before-dispatch constraint (http_util.zig:10-14): a new - `max_xff_len = 256` budget, a new Conn buffer, and a new `Request` - field `client_addr: address.NetAddress` computed once in - `handleRequest`. The copy keeps the **tail**, not the head: a new - bounded suffix-copy helper beside `copyHeader`, because `copyHeader` - returns empty on overflow (server.zig:608) and an empty result would - fall back to the socket peer — loopback behind the intended same-box - proxy, which `api_localhost_exempt = true` (model.zig:89, the default) - then exempts. That is the exact bypass this ruling closes: a client - ships an oversized XFF through the proxy and regains the exemption. - The proxy appends `, ` last, so the real entry always fits - in the 256-byte tail. Fail closed on the residue: header present but - no valid last IP literal in the tail means the trusted proxy violated - its contract — respond 400, never fall back to the exemptible peer. - Only a misconfigured proxy can trigger that arm; a spoofed prefix - cannot, because the proxy's appended entry always terminates the - chain. `bucketLimit` (server.zig:201-207) and the - SSE acquire/release in `live.zig:82-87` both read `client_addr` — the - SSE pair must use the same key or releases leak. `api_limiter.Config` - is untouched: `WebState.web` already carries every web field - (app.zig:472), so the trust check reads `state.web`. -- With trust configured, a proxied remote address is not loopback, so - the exemption no longer swallows it — the limiter and the per-address - SSE cap bind per real client. Requests arriving at the socket from - non-trusted, non-loopback peers are unaffected. -- Settings plumbing checklist (same list as ruling 1, plus the - hand-written web mirrors the fact-finding flagged): `expected_keys`, - round-trip test, `WebView` + `view()` in handlers/settings.zig:207-248, - `restart_required_keys` balance test, `SettingsView` web struct in the - integration test, `types.ts` `Settings.web`, openapi web schemas - (including the `required` array), SettingsPage web section, the - configuration.md row, the api.md rate-limiting section, and a warning - in the proxy/authentication how-to: when proxying without - `trusted_proxies`, set `api_localhost_exempt = false`. -- Tests: unit tests on the effective-address derivation (trusted peer + - XFF chain, untrusted peer + spoofed XFF ignored, trusted peer + no - XFF, trusted peer + an XFF over 256 bytes whose valid last entry is - still honored from the tail, trusted peer + a present-but-invalid - chain → 400); an integration test proving a proxied remote address is - rate-limited while the proxy itself stays exempt. +- New setting `web.trusted_proxies: []const u8 = ""` — a comma-separated list of IP literals in one string (the settings codec supports scalars only; a list type is a compile error at model.zig:358). Empty means off: behavior today. +- Validation: each comma-separated element must parse as an IP literal → new `error.BadTrustedProxy` diagnostic in the web block of `checkScalars`. +- Mechanics: when the socket peer is in the trusted set, the effective client address is the **last** valid IP literal in the request's `X-Forwarded-For` (the entry our proxy appended); a **missing** header falls back to the socket peer. Implementation follows the copy-before-dispatch constraint (http_util.zig:10-14): a new `max_xff_len = 256` budget, a new Conn buffer, and a new `Request` field `client_addr: address.NetAddress` computed once in `handleRequest`. The copy keeps the **tail**, not the head: a new bounded suffix-copy helper beside `copyHeader`, because `copyHeader` returns empty on overflow (server.zig:608) and an empty result would fall back to the socket peer — loopback behind the intended same-box proxy, which `api_localhost_exempt = true` (model.zig:89, the default) then exempts. That is the exact bypass this ruling closes: a client ships an oversized XFF through the proxy and regains the exemption. The proxy appends `, ` last, so the real entry always fits in the 256-byte tail. Fail closed on the residue: header present but no valid last IP literal in the tail means the trusted proxy violated its contract — respond 400, never fall back to the exemptible peer. Only a misconfigured proxy can trigger that arm; a spoofed prefix cannot, because the proxy's appended entry always terminates the chain. `bucketLimit` (server.zig:201-207) and the SSE acquire/release in `live.zig:82-87` both read `client_addr` — the SSE pair must use the same key or releases leak. `api_limiter.Config` is untouched: `WebState.web` already carries every web field (app.zig:472), so the trust check reads `state.web`. +- With trust configured, a proxied remote address is not loopback, so the exemption no longer swallows it — the limiter and the per-address SSE cap bind per real client. Requests arriving at the socket from non-trusted, non-loopback peers are unaffected. +- Settings plumbing checklist (same list as ruling 1, plus the hand-written web mirrors the fact-finding flagged): `expected_keys`, round-trip test, `WebView` + `view()` in handlers/settings.zig:207-248, `restart_required_keys` balance test, `SettingsView` web struct in the integration test, `types.ts` `Settings.web`, openapi web schemas (including the `required` array), SettingsPage web section, the configuration.md row, the api.md rate-limiting section, and a warning in the proxy/authentication how-to: when proxying without `trusted_proxies`, set `api_localhost_exempt = false`. +- Tests: unit tests on the effective-address derivation (trusted peer + XFF chain, untrusted peer + spoofed XFF ignored, trusted peer + no XFF, trusted peer + an XFF over 256 bytes whose valid last entry is still honored from the tail, trusted peer + a present-but-invalid chain → 400); an integration test proving a proxied remote address is rate-limited while the proxy itself stays exempt. -**Recorded (implementation of ruling 4):** `Request.client_addr` is -`std.Io.net.IpAddress`, not `address.NetAddress` — `http_util.zig` is a -module root for the fuzz target and cannot import nxdns modules; any future -ruling that wants a domain type on `Request` hits this. `clientAddr` reads -only the final chain entry, never scanning backwards: an earlier entry is -client-controlled, so a fallback would hand the request an attacker-chosen -identity. Known residual, out of this ruling's list: the login log line in -`handlers/auth.zig` still prints the socket peer, which reads as loopback -behind a trusted proxy — carried to milestone 19. +**Recorded (implementation of ruling 4):** `Request.client_addr` is `std.Io.net.IpAddress`, not `address.NetAddress` — `http_util.zig` is a module root for the fuzz target and cannot import nxdns modules; any future ruling that wants a domain type on `Request` hits this. `clientAddr` reads only the final chain entry, never scanning backwards: an earlier entry is client-controlled, so a fallback would hand the request an attacker-chosen identity. Known residual, out of this ruling's list: the login log line in `handlers/auth.zig` still prints the socket peer, which reads as loopback behind a trusted proxy — carried to milestone 19. ### 5. A field-level contract guard between server and frontend -The REST contract lives in three hand-synced copies. The Zig side is -genuinely well guarded (56-entry contract table parsing live responses -with `.ignore_unknown_fields = false`; route-level openapi guards — which -live in `web_integration_test.zig:1494-1590` and `openapi.zig`, not in -`docs_drift_test.zig` as the audit said). **TypeScript is guarded by -nothing**: every frontend test stubs fetch, and types.ts is strictly -narrower than the wire in places (literal unions like -`Health.status: "ok" | "degraded"`), so re-parsing into Zig structs can -never catch an out-of-union string. Decision: one integration layer -asserting the frontend's consumed shapes against real responses. +The REST contract lives in three hand-synced copies. The Zig side is genuinely well guarded (56-entry contract table parsing live responses with `.ignore_unknown_fields = false`; route-level openapi guards — which live in `web_integration_test.zig:1494-1590` and `openapi.zig`, not in `docs_drift_test.zig` as the audit said). **TypeScript is guarded by nothing**: every frontend test stubs fetch, and types.ts is strictly narrower than the wire in places (literal unions like `Health.status: "ok" | "degraded"`), so re-parsing into Zig structs can never catch an out-of-union string. Decision: one integration layer asserting the frontend's consumed shapes against real responses. Mechanism — captured samples validated by `tsc`, no new dependencies: -- A new `-Dintegration` test beside the contract walk drives the real - `Env` server through every route whose contract-table kind is - `.json` **and** which the frontend consumes through an `api.ts` wrapper - — the GETs **and** the JSON-returning POST/PUT wrappers (login, the - group/client/rule/upstream/blocklist writes, refresh, pause, the - settings PUT), driven with deterministic seed payloads so the mutating - responses are stable too (plus one sample per shared error response - class). GET-only would leave every write-path response contract - unguarded. Explicitly excluded: - the `.raw` routes (`/metrics`, `/api/openapi.yaml` — not JSON, and - ruling 3 deletes their unused wrappers), the `.sse` route, and the - `.none` deletes. The endpoint-to-type mapping is derived from `api.ts`, - not invented. Captured bodies are **canonicalized**: keys sorted, every - number replaced with 0 (type-preserving; strings and booleans kept — - they come from the deterministic seed). Canonicalization is what makes - the output byte-stable across runs. -- The canonical samples render into a committed file - `web/src/lib/contractSamples.gen.ts`: for each endpoint, - `export const sample_: = ;` with the matching - import from `types.ts`. TypeScript object literals get excess-property - checking, so a server field missing from types.ts, a types.ts field - missing from the wire, and an out-of-union literal all fail - `npm run typecheck` — which already runs in CI's frontend job. -**Recorded (implementation):** canonicalization gained two rules beyond -"keys sorted, numbers→0": repeated array elements dedupe by canonical text -(60 identical zeroed timeseries buckets collapse to one; differing rows all -stay), and two build-identity strings (`Version.git_commit`, -`Version.zig_version`) redact to `""` so the golden is not pinned to -one machine — the list is the named constant `volatile_string_keys`. The -renderer emits prettier's own fixpoint output, because CI runs -`format:check` on the tree. 43 samples: 38 endpoint + 5 error classes. The -regen command is documented in AGENTS.md. +- A new `-Dintegration` test beside the contract walk drives the real `Env` server through every route whose contract-table kind is `.json` **and** which the frontend consumes through an `api.ts` wrapper — the GETs **and** the JSON-returning POST/PUT wrappers (login, the group/client/rule/upstream/blocklist writes, refresh, pause, the settings PUT), driven with deterministic seed payloads so the mutating responses are stable too (plus one sample per shared error response class). GET-only would leave every write-path response contract unguarded. Explicitly excluded: the `.raw` routes (`/metrics`, `/api/openapi.yaml` — not JSON, and ruling 3 deletes their unused wrappers), the `.sse` route, and the `.none` deletes. The endpoint-to-type mapping is derived from `api.ts`, not invented. Captured bodies are **canonicalized**: keys sorted, every number replaced with 0 (type-preserving; strings and booleans kept — they come from the deterministic seed). Canonicalization is what makes the output byte-stable across runs. +- The canonical samples render into a committed file `web/src/lib/contractSamples.gen.ts`: for each endpoint, `export const sample_: = ;` with the matching import from `types.ts`. TypeScript object literals get excess-property checking, so a server field missing from types.ts, a types.ts field missing from the wire, and an out-of-union literal all fail `npm run typecheck` — which already runs in CI's frontend job. **Recorded (implementation):** canonicalization gained two rules beyond "keys sorted, numbers→0": repeated array elements dedupe by canonical text (60 identical zeroed timeseries buckets collapse to one; differing rows all stay), and two build-identity strings (`Version.git_commit`, `Version.zig_version`) redact to `""` so the golden is not pinned to one machine — the list is the named constant `volatile_string_keys`. The renderer emits prettier's own fixpoint output, because CI runs `format:check` on the tree. 43 samples: 38 endpoint + 5 error classes. The regen command is documented in AGENTS.md. -- Embedding: an anonymous-import root must be Zig, not TypeScript, so a - wrapper `web/src/lib/contract_samples.zig` exports - `pub const bytes = @embedFile("contractSamples.gen.ts");` and build.zig - registers it — the exact `docs/docs.zig` pattern. The Zig test - byte-compares the embedded bytes against the freshly rendered content - and fails on mismatch. -- Regeneration is explicit, not write-beside-the-embed (embedded bytes - carry no source-tree path): a build option `-Dcontract-samples-out` - (absolute path, wired through `build_options`) makes the test write the - fresh content there instead of comparing. The pinned command, stated in - the failure message and the docs: - `zig build test -Dintegration -Dcontract-samples-out="$PWD/web/src/lib/contractSamples.gen.ts"`. - Golden-file discipline: the server side keeps the samples current; - `tsc` keeps types.ts honest against them. -- `types.ts` gains `ErrorEnvelope` matching the shared error responses - (verify the exact envelope shape against the handlers before writing - it) so the error samples are typed too. -- Recorded residual risk: the server↔openapi seam stays route-level - only. The yaml is served verbatim and never parsed; field-level yaml - drift remains possible and is accepted — revisit only if it bites. +- Embedding: an anonymous-import root must be Zig, not TypeScript, so a wrapper `web/src/lib/contract_samples.zig` exports `pub const bytes = @embedFile("contractSamples.gen.ts");` and build.zig registers it — the exact `docs/docs.zig` pattern. The Zig test byte-compares the embedded bytes against the freshly rendered content and fails on mismatch. +- Regeneration is explicit, not write-beside-the-embed (embedded bytes carry no source-tree path): a build option `-Dcontract-samples-out` (absolute path, wired through `build_options`) makes the test write the fresh content there instead of comparing. The pinned command, stated in the failure message and the docs: `zig build test -Dintegration -Dcontract-samples-out="$PWD/web/src/lib/contractSamples.gen.ts"`. Golden-file discipline: the server side keeps the samples current; `tsc` keeps types.ts honest against them. +- `types.ts` gains `ErrorEnvelope` matching the shared error responses (verify the exact envelope shape against the handlers before writing it) so the error samples are typed too. +- Recorded residual risk: the server↔openapi seam stays route-level only. The yaml is served verbatim and never parsed; field-level yaml drift remains possible and is accepted — revisit only if it bites. ### 6. BADVERS becomes expressible -RFC 6891 §6.1.3: a version-1 EDNS query must get RCODE 16 (BADVERS) with -version 0 in the reply OPT. Nothing checks `opt.version` anywhere -(verified), and the write path cannot express it: `addOptEcho` hardcodes -`extended_rcode = 0` (packet.zig:354) and `synthesize` takes a 4-bit -`types.Rcode`. Negligible operational impact; the mechanism is the -finding. +RFC 6891 §6.1.3: a version-1 EDNS query must get RCODE 16 (BADVERS) with version 0 in the reply OPT. Nothing checks `opt.version` anywhere (verified), and the write path cannot express it: `addOptEcho` hardcodes `extended_rcode = 0` (packet.zig:354) and `synthesize` takes a 4-bit `types.Rcode`. Negligible operational impact; the mechanism is the finding. -- `edns.zig` gains the write-side splitter beside the read-side - `extendedRcode` (:272): a 12-bit rcode splits into the header's four - bits and the OPT's upper eight. `extendedRcode` stays — it is the - read-side pair (milestone 19 assumes it survives). -- `packet.zig` gains `addOptWithRcode(request_opt, do_bit, - extended_rcode: u8)`; `addOptEcho` becomes a call with 0. -- `handler.zig`: the version check sits directly after the successful - `parseOpt` (:227-233), before the opcode check — `if (o.version != 0)` - → a BADVERS synthesis (header `.no_error` + OPT `extended_rcode = 1, - version = 0`), counted in a new `Handler.Stats.badvers` (the metrics - reflection exports it; extend the name test). -- Tests: a version-1 query yields composite rcode 16 (assert via - `edns.extendedRcode`) and version 0 in the reply OPT; a version-0 - query is unaffected. +- `edns.zig` gains the write-side splitter beside the read-side `extendedRcode` (:272): a 12-bit rcode splits into the header's four bits and the OPT's upper eight. `extendedRcode` stays — it is the read-side pair (milestone 19 assumes it survives). +- `packet.zig` gains `addOptWithRcode(request_opt, do_bit, extended_rcode: u8)`; `addOptEcho` becomes a call with 0. +- `handler.zig`: the version check sits directly after the successful `parseOpt` (:227-233), before the opcode check — `if (o.version != 0)` → a BADVERS synthesis (header `.no_error` + OPT `extended_rcode = 1, version = 0`), counted in a new `Handler.Stats.badvers` (the metrics reflection exports it; extend the name test). +- Tests: a version-1 query yields composite rcode 16 (assert via `edns.extendedRcode`) and version 0 in the reply OPT; a version-0 query is unaffected. -**Recorded (implementation):** the write-side splitter is -`edns.splitRcode` with `pub const badvers: u12 = 16`; the BADVERS reply -echoes the question when `qdcount == 1`, following the neighboring -FORMERR sites, so a client can bind the reply to its query; a malformed -OPT whose version byte is 1 answers FORMERR, because `parseOpt` fails -before the version check can read the byte. +**Recorded (implementation):** the write-side splitter is `edns.splitRcode` with `pub const badvers: u12 = 16`; the BADVERS reply echoes the question when `qdcount == 1`, following the neighboring FORMERR sites, so a client can bind the reply to its query; a malformed OPT whose version byte is 1 answers FORMERR, because `parseOpt` fails before the version check can read the byte. ### 7. The stale phase comments state the as-built contract -- `rate_limiter.zig:5-6` ("Phase 7 decides the locking"): rewrite to the - as-built truth — the handler serializes with `Handler.limiter_mutex` - (handler.zig:124), uncancelable on the query path, cancelable in the - `runMaintenance` sweep (app.zig:702-731 already states the rationale - to adopt). -- `shutdown.zig` `trigger` doc ("what a Phase 8 restart endpoint would - call"): no restart endpoint exists or is planned — m8 shipped - `restart_required` echoes instead. Rewrite to "what a test uses". -- `pool.zig:83-84`: tense only — the health endpoint exists; "Feeds - `GET /api/upstream/health`." +- `rate_limiter.zig:5-6` ("Phase 7 decides the locking"): rewrite to the as-built truth — the handler serializes with `Handler.limiter_mutex` (handler.zig:124), uncancelable on the query path, cancelable in the `runMaintenance` sweep (app.zig:702-731 already states the rationale to adopt). +- `shutdown.zig` `trigger` doc ("what a Phase 8 restart endpoint would call"): no restart endpoint exists or is planned — m8 shipped `restart_required` echoes instead. Rewrite to "what a test uses". +- `pool.zig:83-84`: tense only — the health endpoint exists; "Feeds `GET /api/upstream/health`." ### 8. Doc transcripts stop hardcoding the version -Four sites print `0.1.0-dev` (tutorial/first-run.md:108, -install-with-docker.md:126, install-with-systemd.md:206, -upgrade.md:142 — three how-to plus one tutorial, two different -transcript shapes). Correct today; silently wrong at the first tag. -Milestone 14 mandates a placeholder and a guard but names neither; the -milestone-13 executed-transcript rule is in tension. +Four sites print `0.1.0-dev` (tutorial/first-run.md:108, install-with-docker.md:126, install-with-systemd.md:206, upgrade.md:142 — three how-to plus one tutorial, two different transcript shapes). Correct today; silently wrong at the first tag. Milestone 14 mandates a placeholder and a guard but names neither; the milestone-13 executed-transcript rule is in tension. -- Ruling on the tension, explicit: milestone-13 ruling 3 constrains - **command lines**; pasted **output lines** may carry placeholders. - The orchestrator adds one clarifying sentence to milestone-13.md - beside ruling 3. -- The four output lines replace the version with `` (e.g. - `nxdns serving on ...`). -- The guard: `docs/docs.zig` additionally embeds the tutorial and - how-to pages; a new test in `docs_drift_test.zig` asserts no embedded - doc page contains the string `0.1.0-dev` and that the four transcript - pages contain `nxdns `. (Note m15 rewrote this file's CLI - test — rebase, don't collide.) +- Ruling on the tension, explicit: milestone-13 ruling 3 constrains **command lines**; pasted **output lines** may carry placeholders. The orchestrator adds one clarifying sentence to milestone-13.md beside ruling 3. +- The four output lines replace the version with `` (e.g. `nxdns serving on ...`). +- The guard: `docs/docs.zig` additionally embeds the tutorial and how-to pages; a new test in `docs_drift_test.zig` asserts no embedded doc page contains the string `0.1.0-dev` and that the four transcript pages contain `nxdns `. (Note m15 rewrote this file's CLI test — rebase, don't collide.) ### 9. The log docs stop claiming append mode -`files-and-directories.md:143` claims the log file is "opened for -append". It is not: `.mode = .write_only` plus `seekTo(state.file_pos)` -per line, offset seeded once at open — the source comment at -logging.zig:502 says so outright ("0.16.0 has no append mode"). Under -external logrotate the divergence is destructive (copytruncate → sparse -NUL-prefixed file; rename+create → unbounded writes to the renamed -inode, `max_size_mb` unenforceable). +`files-and-directories.md:143` claims the log file is "opened for append". It is not: `.mode = .write_only` plus `seekTo(state.file_pos)` per line, offset seeded once at open — the source comment at logging.zig:502 says so outright ("0.16.0 has no append mode"). Under external logrotate the divergence is destructive (copytruncate → sparse NUL-prefixed file; rename+create → unbounded writes to the renamed inode, `max_size_mb` unenforceable). -Docs only: correct the row (positional writes, offset tracked -in-process), and add an operator warning — nxdns owns rotation for this -file; do not point external logrotate at it. A per-line stat re-check -was considered and rejected: a syscall per log line on the hot path to -defend against a misconfiguration the docs now forbid. Record that -decision here. +Docs only: correct the row (positional writes, offset tracked in-process), and add an operator warning — nxdns owns rotation for this file; do not point external logrotate at it. A per-line stat re-check was considered and rejected: a syscall per log line on the hot path to defend against a misconfiguration the docs now forbid. Record that decision here. ## Sessions -Dependency graph: S1 → S2 → S5 (they share model.zig, validate.zig, -types.ts, openapi.yaml, SettingsPage.tsx, web_integration_test.zig — the -later session rebases on the earlier). S3, S4, S6 run in parallel with -the chain and each other. +Dependency graph: S1 → S2 → S5 (they share model.zig, validate.zig, types.ts, openapi.yaml, SettingsPage.tsx, web_integration_test.zig — the later session rebases on the earlier). S3, S4, S6 run in parallel with the chain and each other. ### Session S1: deadline and validator -Owns `src/upstream/pool.zig`, `src/config/model.zig`, -`src/config/validate.zig`, `src/app.zig`, plus the ruling-1 plumbing -trail (types.ts, openapi.yaml, SettingsPage SECTIONS[0], the -integration-test SettingsView, configuration.md, cli.md). Rulings 1, 2, -and the pool comment from ruling 7. +Owns `src/upstream/pool.zig`, `src/config/model.zig`, `src/config/validate.zig`, `src/app.zig`, plus the ruling-1 plumbing trail (types.ts, openapi.yaml, SettingsPage SECTIONS[0], the integration-test SettingsView, configuration.md, cli.md). Rulings 1, 2, and the pool comment from ruling 7. ### Session S2: trusted proxy (after S1) -Owns `src/web/server.zig`, `src/web/http_util.zig`, -`src/web/handlers/live.zig`, `src/web/handlers/settings.zig`, and its -plumbing trail through the same shared files S1 touched. Ruling 4. +Owns `src/web/server.zig`, `src/web/http_util.zig`, `src/web/handlers/live.zig`, `src/web/handlers/settings.zig`, and its plumbing trail through the same shared files S1 touched. Ruling 4. ### Session S3: upstream editor -Owns `web/src/features/upstreams/*` (new), `web/src/routes.tsx`, -`web/src/shell/AppShell.tsx`, `web/src/lib/api.ts`, -`web/src/lib/queries.ts`, `web/src/features/settings/RestartBanner.tsx` -and `restartBanner.ts`, `specs/milestone-8.md` (the deviation note). -Ruling 3. +Owns `web/src/features/upstreams/*` (new), `web/src/routes.tsx`, `web/src/shell/AppShell.tsx`, `web/src/lib/api.ts`, `web/src/lib/queries.ts`, `web/src/features/settings/RestartBanner.tsx` and `restartBanner.ts`, `specs/milestone-8.md` (the deviation note). Ruling 3. ### Session S4: BADVERS -Owns `src/dns/packet.zig`, `src/dns/edns.zig`, `src/server/handler.zig`, -the metrics name test in `src/web/metrics.zig`. Ruling 6. +Owns `src/dns/packet.zig`, `src/dns/edns.zig`, `src/server/handler.zig`, the metrics name test in `src/web/metrics.zig`. Ruling 6. ### Session S5: contract samples (after S1 and S2) -Owns the new generator test in `src/web/web_integration_test.zig`, -`build.zig` (the anonymous import), `src/tests.zig` (if a new file needs -an entry), `web/src/lib/contractSamples.gen.ts` (committed golden), -`web/src/lib/types.ts` (`ErrorEnvelope` only). Ruling 5. +Owns the new generator test in `src/web/web_integration_test.zig`, `build.zig` (the anonymous import), `src/tests.zig` (if a new file needs an entry), `web/src/lib/contractSamples.gen.ts` (committed golden), `web/src/lib/types.ts` (`ErrorEnvelope` only). Ruling 5. ### Session S6: docs and comments -Owns `src/server/rate_limiter.zig` and `src/server/shutdown.zig` -(headers only), `docs/docs.zig`, `src/docs_drift_test.zig`, the four -transcript pages, `docs/reference/files-and-directories.md`, -`specs/milestone-13.md` (the clarifying sentence). Rulings 7 (except the -pool comment), 8, 9. +Owns `src/server/rate_limiter.zig` and `src/server/shutdown.zig` (headers only), `docs/docs.zig`, `src/docs_drift_test.zig`, the four transcript pages, `docs/reference/files-and-directories.md`, `specs/milestone-13.md` (the clarifying sentence). Rulings 7 (except the pool comment), 8, 9. ### Orchestrator -Strikes the closed findings in `TECH_DEBT.md`; confirms PLAN's -total-budget promise now matches the code; records the m8 placement -deviation. +Strikes the closed findings in `TECH_DEBT.md`; confirms PLAN's total-budget promise now matches the code; records the m8 placement deviation. ## Module layout -New files: `web/src/features/upstreams/UpstreamsPage.tsx`, -`web/src/features/upstreams/UpstreamForm.tsx`, -`web/src/lib/contractSamples.gen.ts` (generated, committed). +New files: `web/src/features/upstreams/UpstreamsPage.tsx`, `web/src/features/upstreams/UpstreamForm.tsx`, `web/src/lib/contractSamples.gen.ts` (generated, committed). -Deleted surface: the seven unused `get*` wrappers, `getMetrics`, -`getOpenapiYaml`, `ruleUpdateMutation`. +Deleted surface: the seven unused `get*` wrappers, `getMetrics`, `getOpenapiYaml`, `ruleUpdateMutation`. ## Acceptance (milestone complete) @@ -459,10 +174,8 @@ Deleted surface: the seven unused `get*` wrappers, `getMetrics`, ## Anti-requirements - No restart endpoint — the `restart_required` echo model stands. -- No hostname resolution for `tls://` upstreams — the validator rejects; - the client's own check stays as defense in depth. -- No openapi codegen and no runtime schema validation library — the - contract guard is captured samples + tsc, nothing more. +- No hostname resolution for `tls://` upstreams — the validator rejects; the client's own check stays as defense in depth. +- No openapi codegen and no runtime schema validation library — the contract guard is captured samples + tsc, nothing more. - No proxy protocol (PROXY/haproxy) support; X-Forwarded-For only. - No per-line stat re-check in the logger (ruling 9 records why). - No upstream health join in the editor beyond the stated restart note. diff --git a/specs/milestone-18.md b/specs/milestone-18.md index c2c4f47..84a3aa6 100644 --- a/specs/milestone-18.md +++ b/specs/milestone-18.md @@ -1,95 +1,35 @@ # Milestone 18: collapse the duplicated infrastructure -Goal: the copy-paste infrastructure from `TECH_DEBT.md` Theme 2 becomes -shared code — the listener core first (the audit's only high finding, four -hand-synced copies of the most invariant-heavy concurrency code in the -repository, drift already live), then the repository memory-safety -choreography, the web CRUD shells, the transport scaffolding, the name -normalization, the line iterator, and the frontend class constants. After -this milestone, a fix in any of these families lands once. +Goal: the copy-paste infrastructure from `TECH_DEBT.md` Theme 2 becomes shared code — the listener core first (the audit's only high finding, four hand-synced copies of the most invariant-heavy concurrency code in the repository, drift already live), then the repository memory-safety choreography, the web CRUD shells, the transport scaffolding, the name normalization, the line iterator, and the frontend class constants. After this milestone, a fix in any of these families lands once. -**PROVISIONAL.** Written before milestones 15-17 were built. m16 touches the -listeners (DoH receiveHead race, stats export), the manager (refresh_lock) -and the metrics tests; m17 touches pool.zig and the handlers. Re-verify -every line reference and re-diff the copies before each session starts. +**PROVISIONAL.** Written before milestones 15-17 were built. m16 touches the listeners (DoH receiveHead race, stats export), the manager (refresh_lock) and the metrics tests; m17 touches pool.zig and the handlers. Re-verify every line reference and re-diff the copies before each session starts. ## Rulings (binding) ### 1. One listener core -`tcp_server.zig`, `dot_server.zig`, `doh_server.zig` and `web/server.zig` -each carry private copies of: the `State`/`ConnState`/`Stop`/`Claim` enums, -`retry_delay`, `bump`, the slot pool with `claim`/`finish` (the same -three-step close dance), `beginShutdown` (identical body, different log -text), `serve` with the cancel-protection dance, `acceptLoop` with the same -error mapping, `decideClaim` in two shapes plus `firstFree`, and — in the -three DNS listeners — the byte-identical `Outcome`/`Result`/`race`/`expire` -select harness. `readPrefix`/`readBody`/`writeReply` are byte-identical -between tcp and dot. The predicted failure mode already fired once: the -milestone-10 review hand-ported the `handshook` TLS-context-leak fix from -dot to doh. Six unit tests are duplicated between tcp and dot, three more -between doh and web. +`tcp_server.zig`, `dot_server.zig`, `doh_server.zig` and `web/server.zig` each carry private copies of: the `State`/`ConnState`/`Stop`/`Claim` enums, `retry_delay`, `bump`, the slot pool with `claim`/`finish` (the same three-step close dance), `beginShutdown` (identical body, different log text), `serve` with the cancel-protection dance, `acceptLoop` with the same error mapping, `decideClaim` in two shapes plus `firstFree`, and — in the three DNS listeners — the byte-identical `Outcome`/`Result`/`race`/`expire` select harness. `readPrefix`/`readBody`/`writeReply` are byte-identical between tcp and dot. The predicted failure mode already fired once: the milestone-10 review hand-ported the `handshook` TLS-context-leak fix from dot to doh. Six unit tests are duplicated between tcp and dot, three more between doh and web. New file `src/server/listener.zig`: - The four enums, `retry_delay`, `bump` — plain shared declarations. -- The race harness: `Outcome`, `Result`, `race`, `expire` — one copy, - generic over the raced function (the existing `comptime f: anytype` shape - is already generic; it only needs to move). -- The framing helpers `readPrefix`, `readBody`, `writeReply` (tcp/dot - consumers; doh speaks HTTP). -- `pub fn Core(comptime Cfg: type) type` generating: the conns array with - `Cfg.ConnPayload` per slot, mutex, `run_state`, `shutdown_begun`, - `claim`/`finish`/`beginShutdown`/`decideClaim`/`firstFree`, the accept - loop, and the `serve` skeleton with the cancel-protection dance. `Cfg` - supplies exactly four things: the per-slot payload type (`ConnPayload` — - the payloads genuinely differ per listener, TLS context and buffers), - the per-connection serve function (`serveConn`), the read/write buffer - sizes, and the at-capacity behavior (close for the DNS listeners; the - web listener's `refuse` 503 for web). -- The Core does **not** own the TLS lifecycle. Certificate pin and - release, TLS-context ownership, the handshake, close_notify, and - cleanup after a late race completion form one ordered sequence - (doh_server.zig:294-330, dot_server.zig:288-323) and stay inside each - listener's `serveConn`. What is shared is a helper, - `listener.handshakeStage`, that owns the `handshook` exactly-once flag - and its late-success cleanup contract — the exact defect the - milestone-10 review hand-ported — with a doc comment stating the - caller's required ordering (pin → handshake via the helper → serve → - TLS close → release, cleanup on every early exit). Both TLS listeners - adopt it; the leak class closes there, not in the Core. +- The race harness: `Outcome`, `Result`, `race`, `expire` — one copy, generic over the raced function (the existing `comptime f: anytype` shape is already generic; it only needs to move). +- The framing helpers `readPrefix`, `readBody`, `writeReply` (tcp/dot consumers; doh speaks HTTP). +- `pub fn Core(comptime Cfg: type) type` generating: the conns array with `Cfg.ConnPayload` per slot, mutex, `run_state`, `shutdown_begun`, `claim`/`finish`/`beginShutdown`/`decideClaim`/`firstFree`, the accept loop, and the `serve` skeleton with the cancel-protection dance. `Cfg` supplies exactly four things: the per-slot payload type (`ConnPayload` — the payloads genuinely differ per listener, TLS context and buffers), the per-connection serve function (`serveConn`), the read/write buffer sizes, and the at-capacity behavior (close for the DNS listeners; the web listener's `refuse` 503 for web). +- The Core does **not** own the TLS lifecycle. Certificate pin and release, TLS-context ownership, the handshake, close_notify, and cleanup after a late race completion form one ordered sequence (doh_server.zig:294-330, dot_server.zig:288-323) and stay inside each listener's `serveConn`. What is shared is a helper, `listener.handshakeStage`, that owns the `handshook` exactly-once flag and its late-success cleanup contract — the exact defect the milestone-10 review hand-ported — with a doc comment stating the caller's required ordering (pin → handshake via the helper → serve → TLS close → release, cleanup on every early exit). Both TLS listeners adopt it; the leak class closes there, not in the Core. - The shared unit tests move here; the per-file duplicates are deleted. Unifications the extraction forces, all sanctioned: -- Counter name: `connections` everywhere. tcp and web rename `accepted`. - This renames the m16-added metric family - `nxdns_tcp_server_accepted_total` → `nxdns_tcp_server_connections_total` - — greenfield, allowed; update the metrics name test and the docs - reference. Core stats are one shared struct; `tls_handshake_failures`, - `bad_requests`, `requests` stay listener-specific beside it, and the - exported snapshots stay flat so /metrics output is unchanged except the - tcp rename. -- `deinit` shape: all four store the allocator at `listen` (dot's - milestone-10 deviation becomes the rule) and take `(self, io)`. -- The dead `pub fn serve` at doh_server.zig:687-707 (documented as - intentional in specs/milestone-10.md:286-289) is deleted; note the - update beside that spec line. +- Counter name: `connections` everywhere. tcp and web rename `accepted`. This renames the m16-added metric family `nxdns_tcp_server_accepted_total` → `nxdns_tcp_server_connections_total` — greenfield, allowed; update the metrics name test and the docs reference. Core stats are one shared struct; `tls_handshake_failures`, `bad_requests`, `requests` stay listener-specific beside it, and the exported snapshots stay flat so /metrics output is unchanged except the tcp rename. +- `deinit` shape: all four store the allocator at `listen` (dot's milestone-10 deviation becomes the rule) and take `(self, io)`. +- The dead `pub fn serve` at doh_server.zig:687-707 (documented as intentional in specs/milestone-10.md:286-289) is deleted; note the update beside that spec line. -The per-listener files keep what genuinely differs: the connection payload -(TLS context and buffers), the serve-one-connection logic, DoH's HTTP -handling, and the web listener's request router. Every existing listener -integration test must pass unchanged apart from renamed counters. +The per-listener files keep what genuinely differs: the connection payload (TLS context and buffers), the serve-one-connection logic, DoH's HTTP handling, and the web listener's request router. Every existing listener integration test must pass unchanged apart from renamed counters. ### 2. The repository list choreography exists once -The `prepare → ArrayList → errdefer out.deinit → errdefer freeX → -columnTextAlloc → append` shape is hand-rolled 19 times across the seven -repository files, with the load-bearing errdefer ordering repeated 18 times -and 26 hand-written `freeX` loops. The milestone-4 spec's own reference -sample (specs/milestone-4.md:1263-1272) declares the two errdefers in the -**reverse, use-after-free order** — every implementation silently corrected -it; the next repo copied from the spec is a landmine. +The `prepare → ArrayList → errdefer out.deinit → errdefer freeX → columnTextAlloc → append` shape is hand-rolled 19 times across the seven repository files, with the load-bearing errdefer ordering repeated 18 times and 26 hand-written `freeX` loops. The milestone-4 spec's own reference sample (specs/milestone-4.md:1263-1272) declares the two errdefers in the **reverse, use-after-free order** — every implementation silently corrected it; the next repo copied from the spec is a landmine. `src/storage/repositories/crud.zig` (today: `execStrict` only) gains: @@ -105,38 +45,16 @@ pub fn listRows( pub fn freeRows(comptime Row: type, gpa: Allocator, items: []const Row) void ``` -`listRows` owns the errdefer choreography — written once, with the ordering -comment. `freeRows` frees every `[]const u8` **and every `?[]const u8`** -field by comptime reflection — `SourceRow.checksum` is an allocated -optional slice (sources_repo.zig:91-104) and its current destructor frees -the non-null payload explicitly; a shallow slice-only reflection leaks it. -Any other owning field shape is a `@compileError`, so a future row cannot -silently leak. The allocation-failure tests must keep a case with a -non-null checksum. -The per-row `readRow` functions stay in the repos and keep their per-column -errdefers. All 19 list functions become one-line delegations; the `freeX` -wrappers stay as thin `pub` shims where callers use them, or are deleted -where `freeRows` is called directly. Every `checkAllAllocationFailures` -test stays and must pass — they now exercise the shared helper from 20 -angles. +`listRows` owns the errdefer choreography — written once, with the ordering comment. `freeRows` frees every `[]const u8` **and every `?[]const u8`** field by comptime reflection — `SourceRow.checksum` is an allocated optional slice (sources_repo.zig:91-104) and its current destructor frees the non-null payload explicitly; a shallow slice-only reflection leaks it. Any other owning field shape is a `@compileError`, so a future row cannot silently leak. The allocation-failure tests must keep a case with a non-null checksum. The per-row `readRow` functions stay in the repos and keep their per-column errdefers. All 19 list functions become one-line delegations; the `freeX` wrappers stay as thin `pub` shims where callers use them, or are deleted where `freeRows` is called directly. Every `checkAllAllocationFailures` test stays and must pass — they now exercise the shared helper from 20 angles. -Correct the milestone-4 sample in place and note the correction in that -spec. +Correct the milestone-4 sample in place and note the correction in that spec. ### 3. The web CRUD shells are generated; the decisions stay hand-written -Seven handler files repeat the 4-line `configDb` switch 40 times in three -forms, and five of them repeat identical `list`/`get`/`remove` shells. The -four reload flavors (plain; local.zig's publish-under-lock; blocklists' -pruneFiles; groups' read-back-under-lock) are genuinely different and are -**not** unified. +Seven handler files repeat the 4-line `configDb` switch 40 times in three forms, and five of them repeat identical `list`/`get`/`remove` shells. The four reload flavors (plain; local.zig's publish-under-lock; blocklists' pruneFiles; groups' read-back-under-lock) are genuinely different and are **not** unified. -- `mutations.zig` gains `pub fn requireConfigDb(state: *server.WebState) - error{NoConfigDb}!*db.Db`. The 40 switch sites become one-line `catch` - arms producing the same three response forms (the Failure payload is the - same constant text everywhere). -- `mutations.zig` gains a comptime resource descriptor generating only the - identical trio: +- `mutations.zig` gains `pub fn requireConfigDb(state: *server.WebState) error{NoConfigDb}!*db.Db`. The 40 switch sites become one-line `catch` arms producing the same three response forms (the Failure payload is the same constant text everywhere). +- `mutations.zig` gains a comptime resource descriptor generating only the identical trio: ```zig // desc is an anonymous struct literal; `anytype` is not legal as a @@ -150,184 +68,67 @@ pruneFiles; groups' read-back-under-lock) are genuinely different and are // []const u8 ("upstreams", the JSON list key). ``` - yielding `list`/`get`/`remove` handlers in the exact current shape. - Adopted by upstreams, groups, blocklists, clients, rules and the two - local.zig sets where the shell matches; `settings.zig` (get + applyPut - only) does not adopt. All `applyCreate`/`applyUpdate`/`applyDelete` - bodies and the per-resource decision functions (countEnabledExcept, - updateLocked/deleteLocked, pruneFiles, applyReplacePrefixes, publish, - RuleView, applyPut) stay hand-written. + yielding `list`/`get`/`remove` handlers in the exact current shape. Adopted by upstreams, groups, blocklists, clients, rules and the two local.zig sets where the shell matches; `settings.zig` (get + applyPut only) does not adopt. All `applyCreate`/`applyUpdate`/`applyDelete` bodies and the per-resource decision functions (countEnabledExcept, updateLocked/deleteLocked, pruneFiles, applyReplacePrefixes, publish, RuleView, applyPut) stay hand-written. -Acceptance is a grep: the 4-line switch appears zero times outside -`mutations.zig`. +Acceptance is a grep: the 4-line switch appears zero times outside `mutations.zig`. ### 4. The transport scaffolding is shared, and DoH gets its missing unwrap -- The byte-identical `Outcome` + `expire` + select-race body in - `pool.zig:253-274/310-317` and `forward_client.zig:195-215/260-267` - moves to `transport.zig` as one generic exchange-race helper (the raced - function, the budget and one comment word are the only differences - today). `manager.zig`'s `fetchWithin` may adopt it if it generalizes - without contortion; not required. -- `mapPhase` (duplicated byte-for-byte, dot_client.zig:283 / - forward_client.zig:293) moves to `transport.zig` as `pub`. -- The cancel-protected close helpers (four near-copies: - forward_client.zig:281/287, dot_client.zig:271/277) become one - `pub fn closeBlocked(io: std.Io, target: anytype) void` in transport.zig - (`target.close(io)` under swapped protection). The inline copies in - logging.zig and metrics.zig are out of scope — they are not transport - code. -- `doh_client.zig` gains the stashed-cause unwrap the other two transports - carry (`concreteRead`/`concreteWrite` precedent, dot_client.zig:303-318): - today its `mapError` never reads the http client's stashed cause behind - `ReadFailed`/`WriteFailed`, so rare mid-exchange local errors (e.g. - ENOBUFS) are recorded as peer faults against a healthy upstream — a - stated spec-invariant violation, fixed once for DoT and left in DoH. - Port the unwrap and the corresponding stub-based tests. -- `sendFailure`/`receiveFailure` stay per-file: the unwrapping genuinely - differs by stream type. +- The byte-identical `Outcome` + `expire` + select-race body in `pool.zig:253-274/310-317` and `forward_client.zig:195-215/260-267` moves to `transport.zig` as one generic exchange-race helper (the raced function, the budget and one comment word are the only differences today). `manager.zig`'s `fetchWithin` may adopt it if it generalizes without contortion; not required. +- `mapPhase` (duplicated byte-for-byte, dot_client.zig:283 / forward_client.zig:293) moves to `transport.zig` as `pub`. +- The cancel-protected close helpers (four near-copies: forward_client.zig:281/287, dot_client.zig:271/277) become one `pub fn closeBlocked(io: std.Io, target: anytype) void` in transport.zig (`target.close(io)` under swapped protection). The inline copies in logging.zig and metrics.zig are out of scope — they are not transport code. +- `doh_client.zig` gains the stashed-cause unwrap the other two transports carry (`concreteRead`/`concreteWrite` precedent, dot_client.zig:303-318): today its `mapError` never reads the http client's stashed cause behind `ReadFailed`/`WriteFailed`, so rare mid-exchange local errors (e.g. ENOBUFS) are recorded as peer faults against a healthy upstream — a stated spec-invariant violation, fixed once for DoT and left in DoH. Port the unwrap and the corresponding stub-based tests. +- `sendFailure`/`receiveFailure` stay per-file: the unwrapping genuinely differs by stream type. ### 5. `normalizeName` lives beside `fromText` -The copies in `records.zig:197` and `forward_zones.zig:129` are -byte-identical (only doc comments differ). Move the function to -`src/dns/name.zig` as `pub fn normalizeText(text: []const u8, buf: -*[types.max_name_len]u8) error{BadName}![]const u8`, next to `fromText` -(both callers already import the module; it stays pure — no allocation, no -Io). Both files delete their copies and their private `NameError`. +The copies in `records.zig:197` and `forward_zones.zig:129` are byte-identical (only doc comments differ). Move the function to `src/dns/name.zig` as `pub fn normalizeText(text: []const u8, buf: *[types.max_name_len]u8) error{BadName}![]const u8`, next to `fromText` (both callers already import the module; it stays pure — no allocation, no Io). Both files delete their copies and their private `NameError`. -The other three variants are **not** unified — their policies differ on -purpose (rules.zig:195 rejects controls and space but skips `fromText` -because patterns hold `*`; compiler.zig:155 adds a two-label minimum and -counter-based reporting; dns_cache.zig:59 validates nothing because its -input is asserted). Each of the three gains a one-line comment pointing at -`name.normalizeText` and naming its own policy difference, so the next -reader knows the divergence is intentional. +The other three variants are **not** unified — their policies differ on purpose (rules.zig:195 rejects controls and space but skips `fromText` because patterns hold `*`; compiler.zig:155 adds a two-label minimum and counter-based reporting; dns_cache.zig:59 validates nothing because its input is asserted). Each of the three gains a one-line comment pointing at `name.normalizeText` and naming its own policy difference, so the next reader knows the divergence is intentional. ### 6. One bounded-line iterator -The subtle `takeDelimiter`/`StreamTooLong`/`discardDelimiterInclusive` loop -exists twice (`compiler.zig:64-87`, `manager.zig collectSample:1423-1441`), -both correct, both carrying the infinite-loop-hazard comment and a -regression test. Their seven behavioral differences (termination, counting, -trimming, filtering, exit style, error set, arm order) all live *outside* -the hazardous core. New shared iterator in `src/filter/parsers.zig`: +The subtle `takeDelimiter`/`StreamTooLong`/`discardDelimiterInclusive` loop exists twice (`compiler.zig:64-87`, `manager.zig collectSample:1423-1441`), both correct, both carrying the infinite-loop-hazard comment and a regression test. Their seven behavioral differences (termination, counting, trimming, filtering, exit style, error set, arm order) all live *outside* the hazardous core. New shared iterator in `src/filter/parsers.zig`: ```zig pub const LineEvent = union(enum) { line: []const u8, long_line }; pub fn nextBoundedLine(r: *std.Io.Reader, max_len: usize) error{ReadFailed}!?LineEvent ``` -encapsulating the take/discard dance (including the -EndOfStream-during-discard arm and the large-reader-buffer length check). -The limit is a **parameter** — `parsers.zig` must not import `compiler.zig` -(compiler imports parsers, so that is a cycle, and parsers.zig is a -standalone std-only fuzz-module root); both callers pass -`compiler.max_line_len` from their side. `compile` keeps its `long_lines` counting -and `\r` handling on top; `collectSample` keeps its trim/filter/count-limit -on top. Both regression tests must still pass; the m15 compiler fuzz target -now exercises the shared core. +encapsulating the take/discard dance (including the EndOfStream-during-discard arm and the large-reader-buffer length check). The limit is a **parameter** — `parsers.zig` must not import `compiler.zig` (compiler imports parsers, so that is a cycle, and parsers.zig is a standalone std-only fuzz-module root); both callers pass `compiler.max_line_len` from their side. `compile` keeps its `long_lines` counting and `\r` handling on top; `collectSample` keeps its trim/filter/count-limit on top. Both regression tests must still pass; the m15 compiler fuzz target now exercises the shared core. ### 7. `cli.zig` stops re-spelling app policy -- The DoH buffer sizes: `app.zig:89-90` names - `doh_request_buf_len = 1024` / `doh_transfer_buf_len = 4096` (private); - `cli.zig:885-886` re-spells them as bare literals, so changing the - constants would leave `nxdns check` probing different buffers than - `nxdns run` uses — undermining the probe's stated purpose. The constants - move to `doh_client.zig` as `pub const default_request_buf_len` / - `default_transfer_buf_len`; both `app.zig` and `cli.zig` use them. No - deeper unification of `probeUpstreams` vs `Upstreams.build` — the - one-at-a-time probe and the slab build are intentionally different - shapes. -- The ZON failure channels: `check` prints the multi-line `zon_diag` - rendering inline in a single `FAIL` line (cli.zig:721-729), embedding - newlines mid-record and contradicting the one-line-per-problem promise - (docs/reference/configuration.md:332); `import` routes through - `reportParseFailure` (config/import.zig:167 — currently **private**). - Make `reportParseFailure` `pub`; `check` builds a local - `validate.Diagnostics`, feeds the parse failure through it, and renders - each problem as its own `FAIL` line like its other diagnostics. - Acceptance: a config with a multi-line ZON error yields one `FAIL` line - per parser message from both `check` and `import`. +- The DoH buffer sizes: `app.zig:89-90` names `doh_request_buf_len = 1024` / `doh_transfer_buf_len = 4096` (private); `cli.zig:885-886` re-spells them as bare literals, so changing the constants would leave `nxdns check` probing different buffers than `nxdns run` uses — undermining the probe's stated purpose. The constants move to `doh_client.zig` as `pub const default_request_buf_len` / `default_transfer_buf_len`; both `app.zig` and `cli.zig` use them. No deeper unification of `probeUpstreams` vs `Upstreams.build` — the one-at-a-time probe and the slab build are intentionally different shapes. +- The ZON failure channels: `check` prints the multi-line `zon_diag` rendering inline in a single `FAIL` line (cli.zig:721-729), embedding newlines mid-record and contradicting the one-line-per-problem promise (docs/reference/configuration.md:332); `import` routes through `reportParseFailure` (config/import.zig:167 — currently **private**). Make `reportParseFailure` `pub`; `check` builds a local `validate.Diagnostics`, feeds the parse failure through it, and renders each problem as its own `FAIL` line like its other diagnostics. Acceptance: a config with a multi-line ZON error yields one `FAIL` line per parser message from both `check` and `import`. ### 8. `build.zig` wires a test suite once -The host (lines 47-69) and aarch64 (150-179) test suites repeat twelve -wiring lines; the aarch64 triple is re-spelled at :152 instead of read from -`cross_targets` (:10-13). Extract `fn addTestSuite(b, target, optimize, -options, web_assets) *Step.Compile` mirroring the existing `addExecutable` -helper; the aarch64 call resolves its target from `cross_targets[1]` (or a -named constant both use). The two blocks' three real differences -(`linkage = .static`, `skip_foreign_checks`) stay at the call sites. The -m15 fuzz-suite wiring repeats the same shape three times — fold those calls -into the helper too if the anonymous-import needs line up; otherwise leave -them and say so. +The host (lines 47-69) and aarch64 (150-179) test suites repeat twelve wiring lines; the aarch64 triple is re-spelled at :152 instead of read from `cross_targets` (:10-13). Extract `fn addTestSuite(b, target, optimize, options, web_assets) *Step.Compile` mirroring the existing `addExecutable` helper; the aarch64 call resolves its target from `cross_targets[1]` (or a named constant both use). The two blocks' three real differences (`linkage = .static`, `skip_foreign_checks`) stay at the call sites. The m15 fuzz-suite wiring repeats the same shape three times — fold those calls into the helper too if the anonymous-import needs line up; otherwise leave them and say so. ### 9. One Smith encoder -`sliceInput` is byte-identical in `blocklist_fuzz.zig:160` and -`corpus.zig:123` (blocklist_fuzz cannot import corpus.zig — corpus imports -the `dns` module its build target lacks; that is why it was copied). New -dependency-free `tests/fuzz/smith_encode.zig` holding `sliceInput` and its -length self-test; `corpus.zig` and `blocklist_fuzz.zig` import it by -relative path (each fuzz module compiles its own copy — source-level dedup -is the goal). `pairInput` stays in blocklist_fuzz, `sliceIntInput` stays in -corpus — both build on the shared one. The m15 fuzz files -(`compiler_fuzz.zig`, `http_util_fuzz.zig`) adopt it where they inlined -the idiom. +`sliceInput` is byte-identical in `blocklist_fuzz.zig:160` and `corpus.zig:123` (blocklist_fuzz cannot import corpus.zig — corpus imports the `dns` module its build target lacks; that is why it was copied). New dependency-free `tests/fuzz/smith_encode.zig` holding `sliceInput` and its length self-test; `corpus.zig` and `blocklist_fuzz.zig` import it by relative path (each fuzz module compiles its own copy — source-level dedup is the goal). `pairInput` stays in blocklist_fuzz, `sliceIntInput` stays in corpus — both build on the shared one. The m15 fuzz files (`compiler_fuzz.zig`, `http_util_fuzz.zig`) adopt it where they inlined the idiom. ### 10. Frontend: one class vocabulary, shared form plumbing -15 of 30 non-test `.tsx` files re-declare Tailwind class constants in two -naming conventions; the primary-button literal appears at 7 sites, the -input literal at 6, the focus-visible fragment 39 times across 17 files — -and the two inputs that *dropped* the focus ring (PrefixesEditor.tsx:13, -GroupsPage.tsx:17, the only unfocusable inputs in the app) silently violate -the milestone-9 accessibility floor. The project's own precedent -(InlineError was hoisted during milestone 9) is house style left -unapplied. +15 of 30 non-test `.tsx` files re-declare Tailwind class constants in two naming conventions; the primary-button literal appears at 7 sites, the input literal at 6, the focus-visible fragment 39 times across 17 files — and the two inputs that *dropped* the focus ring (PrefixesEditor.tsx:13, GroupsPage.tsx:17, the only unfocusable inputs in the app) silently violate the milestone-9 accessibility floor. The project's own precedent (InlineError was hoisted during milestone 9) is house style left unapplied. -- New `web/src/ui/classes.ts` exporting the named constants (camelCase, one - convention): `inputClass`, `buttonClass`, `primaryButtonClass`, - `thClass`, `tdClass`, `formCardClass`, `tableWrapClass`, - `retryButtonClass` — taken verbatim from the current majority literals - (all carrying the focus ring). All 15 declaring files adopt; the two - ring-less inputs are fixed by adoption. Acceptance is a grep: the three - top literals appear only in `ui/classes.ts`, and every - input/button/select in `web/src` carries the focus-visible fragment (the - one sanctioned variant is PauseWidget's negative offset). -- New `web/src/ui/useCrudForm.ts`: the mutation trio + `FormState` + - `openForm`/`onSubmit`/`onDelete` plumbing that RecordsTab and ZonesTab - share byte-for-byte, parameterized over the entity and its three - mutation factories. Both tabs adopt it; their field JSX and tables stay - local. Other pages may adopt where the shape fits; none is forced. -- The shadowing `InlineError` in DashboardPage.tsx:36: extend - `lib/InlineError.tsx` with an optional `onRetry` prop rendering the - retry button; DashboardPage imports it and the local copy is deleted. +- New `web/src/ui/classes.ts` exporting the named constants (camelCase, one convention): `inputClass`, `buttonClass`, `primaryButtonClass`, `thClass`, `tdClass`, `formCardClass`, `tableWrapClass`, `retryButtonClass` — taken verbatim from the current majority literals (all carrying the focus ring). All 15 declaring files adopt; the two ring-less inputs are fixed by adoption. Acceptance is a grep: the three top literals appear only in `ui/classes.ts`, and every input/button/select in `web/src` carries the focus-visible fragment (the one sanctioned variant is PauseWidget's negative offset). +- New `web/src/ui/useCrudForm.ts`: the mutation trio + `FormState` + `openForm`/`onSubmit`/`onDelete` plumbing that RecordsTab and ZonesTab share byte-for-byte, parameterized over the entity and its three mutation factories. Both tabs adopt it; their field JSX and tables stay local. Other pages may adopt where the shape fits; none is forced. +- The shadowing `InlineError` in DashboardPage.tsx:36: extend `lib/InlineError.tsx` with an optional `onRetry` prop rendering the retry button; DashboardPage imports it and the local copy is deleted. ## Sessions -S1-S7 run in parallel; no two sessions write the same file. Interfaces -fixed here: S4 makes the two DoH buffer constants `pub` in -`doh_client.zig`; S5 consumes them from `cli.zig`/`app.zig` — wait for S4's -commit of that one declaration or agree the exact names above and build -against them. +S1-S7 run in parallel; no two sessions write the same file. Interfaces fixed here: S4 makes the two DoH buffer constants `pub` in `doh_client.zig`; S5 consumes them from `cli.zig`/`app.zig` — wait for S4's commit of that one declaration or agree the exact names above and build against them. ### Session S1: listener core -Owns `src/server/listener.zig` (new), `src/server/tcp_server.zig`, -`src/server/dot_server.zig`, `src/server/doh_server.zig`, -`src/web/server.zig`, `src/web/metrics.zig`, and the listener integration -tests (`tcp_server_integration_test.zig`, -`udp_server_integration_test.zig`, `src/web/server_integration_test.zig`, -`resolver_integration_test.zig`, `phase7_integration_test.zig` as needed). -Ruling 1. +Owns `src/server/listener.zig` (new), `src/server/tcp_server.zig`, `src/server/dot_server.zig`, `src/server/doh_server.zig`, `src/web/server.zig`, `src/web/metrics.zig`, and the listener integration tests (`tcp_server_integration_test.zig`, `udp_server_integration_test.zig`, `src/web/server_integration_test.zig`, `resolver_integration_test.zig`, `phase7_integration_test.zig` as needed). Ruling 1. ### Session S2: storage -Owns `src/storage/repositories/*` and the milestone-4 spec correction. -Ruling 2. +Owns `src/storage/repositories/*` and the milestone-4 spec correction. Ruling 2. ### Session S3: web handlers @@ -335,19 +136,11 @@ Owns `src/web/handlers/*`, `src/web/web_integration_test.zig`. Ruling 3. ### Session S4: transport -Owns `src/upstream/transport.zig`, `src/upstream/pool.zig`, -`src/upstream/dot_client.zig`, `src/upstream/doh_client.zig`, -`src/local/forward_client.zig`. Ruling 4, and the `pub` constants half of -ruling 7. +Owns `src/upstream/transport.zig`, `src/upstream/pool.zig`, `src/upstream/dot_client.zig`, `src/upstream/doh_client.zig`, `src/local/forward_client.zig`. Ruling 4, and the `pub` constants half of ruling 7. ### Session S5: names, lines, cli -Owns `src/dns/name.zig`, `src/local/records.zig`, -`src/local/forward_zones.zig`, `src/filter/rules.zig`, -`src/filter/compiler.zig`, `src/filter/manager.zig`, -`src/filter/parsers.zig`, `src/cache/dns_cache.zig` (comment only), -`src/cli.zig`, `src/config/import.zig`, `src/app.zig`. Rulings 5, 6, and -the consumer half of 7. +Owns `src/dns/name.zig`, `src/local/records.zig`, `src/local/forward_zones.zig`, `src/filter/rules.zig`, `src/filter/compiler.zig`, `src/filter/manager.zig`, `src/filter/parsers.zig`, `src/cache/dns_cache.zig` (comment only), `src/cli.zig`, `src/config/import.zig`, `src/app.zig`. Rulings 5, 6, and the consumer half of 7. ### Session S6: build and fuzz @@ -359,19 +152,13 @@ Owns `web/src/**`. Ruling 10. ### Orchestrator -Strikes the closed findings in `TECH_DEBT.md`; records the tcp metric -rename in the docs reference; updates the milestone-10 deviation note -(ruling 1) and the milestone-4 sample (ruling 2). +Strikes the closed findings in `TECH_DEBT.md`; records the tcp metric rename in the docs reference; updates the milestone-10 deviation note (ruling 1) and the milestone-4 sample (ruling 2). ## Module layout -New files: `src/server/listener.zig`, `tests/fuzz/smith_encode.zig`, -`web/src/ui/classes.ts`, `web/src/ui/useCrudForm.ts`. +New files: `src/server/listener.zig`, `tests/fuzz/smith_encode.zig`, `web/src/ui/classes.ts`, `web/src/ui/useCrudForm.ts`. -Deleted surface: the four per-listener copies of the shared machinery, the -dead `doh_server.serve`, the two `normalizeName` copies and their -`NameError`s, the duplicated listener unit tests, the DashboardPage -`InlineError`. +Deleted surface: the four per-listener copies of the shared machinery, the dead `doh_server.serve`, the two `normalizeName` copies and their `NameError`s, the duplicated listener unit tests, the DashboardPage `InlineError`. ## Acceptance (milestone complete) @@ -408,143 +195,57 @@ dead `doh_server.serve`, the two `normalizeName` copies and their ## Recorded (implementation) -Accepted deviations and findings from the built milestone. Each was -reviewed and accepted at integration; the rulings above stand except as -recorded here. +Accepted deviations and findings from the built milestone. Each was reviewed and accepted at integration; the rulings above stand except as recorded here. ### Ruling 1 (S1) -- `Cfg` supplies more than the four listed members: `Owner`, - `ConnPayload`, `serveConn`, `read_buffer_len`, `write_buffer_len`, - plus `log` (the owner's `std.log` scope) and `name`, because the - accept loop and shutdown log and the text had to stay byte-identical. - Two optional decls: `refuse` (absent means close the stream; the web - listener supplies its 503) and `initPayload`/`deinitPayload`, which - exist only for the web arena's create-in-listen / destroy-in-deinit - lifecycle. -- The read/write staging buffers live in the core's `Conn`, not in - `ConnPayload` — all four listeners have exactly one of each and - differ only in size. The web listener's `recv_buf`/`send_buf` are now - `read_buf`/`write_buf`. -- Stats layout: core counters live at `server.core.stats.*` - (`listener.CoreStats`); listener-specific counters stay on the owner - at `server.stats.*`. `tcp_server.Stats` is an alias of `CoreStats`. - Exported snapshots stay flat, so /metrics output is unchanged except - the tcp rename. `idle_timeouts` sits in `CoreStats` per the ruling, - which gives the web listener a counter it never bumps; it exports no - family, so nothing is visible. -- The docs-reference update for the rename has no target: - `nxdns_tcp_server_accepted_total` appears in no file under docs/ or - web/. Only the metrics name test changed. -- 18 duplicated unit tests were deleted (6 tcp, 6 dot, 3 doh, 3 web) - and replaced by 6 shared claim-rule tests in `listener.zig`. - `web/server_integration_test.zig`'s `withServer` now returns a local - plain-`u64` `Counters` struct instead of `server.Stats`. -- File-ownership breach: S1 edited two files owned by other sessions, - both mechanical fallout of the sanctioned `deinit` shape change — - three call sites in `src/app.zig` (S5) and one line in - `src/web/web_integration_test.zig` (S3). Both owners reviewed and - kept the edits. S1 also added the required `src/tests.zig` import - line for the new file. +- `Cfg` supplies more than the four listed members: `Owner`, `ConnPayload`, `serveConn`, `read_buffer_len`, `write_buffer_len`, plus `log` (the owner's `std.log` scope) and `name`, because the accept loop and shutdown log and the text had to stay byte-identical. Two optional decls: `refuse` (absent means close the stream; the web listener supplies its 503) and `initPayload`/`deinitPayload`, which exist only for the web arena's create-in-listen / destroy-in-deinit lifecycle. +- The read/write staging buffers live in the core's `Conn`, not in `ConnPayload` — all four listeners have exactly one of each and differ only in size. The web listener's `recv_buf`/`send_buf` are now `read_buf`/`write_buf`. +- Stats layout: core counters live at `server.core.stats.*` (`listener.CoreStats`); listener-specific counters stay on the owner at `server.stats.*`. `tcp_server.Stats` is an alias of `CoreStats`. Exported snapshots stay flat, so /metrics output is unchanged except the tcp rename. `idle_timeouts` sits in `CoreStats` per the ruling, which gives the web listener a counter it never bumps; it exports no family, so nothing is visible. +- The docs-reference update for the rename has no target: `nxdns_tcp_server_accepted_total` appears in no file under docs/ or web/. Only the metrics name test changed. +- 18 duplicated unit tests were deleted (6 tcp, 6 dot, 3 doh, 3 web) and replaced by 6 shared claim-rule tests in `listener.zig`. `web/server_integration_test.zig`'s `withServer` now returns a local plain-`u64` `Counters` struct instead of `server.Stats`. +- File-ownership breach: S1 edited two files owned by other sessions, both mechanical fallout of the sanctioned `deinit` shape change — three call sites in `src/app.zig` (S5) and one line in `src/web/web_integration_test.zig` (S3). Both owners reviewed and kept the edits. S1 also added the required `src/tests.zig` import line for the new file. ### Ruling 2 (S2) -- `listRowsBound(comptime Row, database, gpa, comptime sql, args: - anytype, comptime readRow)` exists beside `listRows` for the bound - queries; `freeRow` exists beside `freeRows`. Unknown owning field - shapes are a `@compileError` as ruled. The milestone-4 sample - correction was made by S2, not the orchestrator. +- `listRowsBound(comptime Row, database, gpa, comptime sql, args: anytype, comptime readRow)` exists beside `listRows` for the bound queries; `freeRow` exists beside `freeRows`. Unknown owning field shapes are a `@compileError` as ruled. The milestone-4 sample correction was made by S2, not the orchestrator. ### Ruling 3 (S3) -- `plural` is a separate descriptor member: the "listing X" log context - differs from the JSON envelope key for three resources - (`local_records`, `client_prefixes`, `forward_zones`), so deriving one - from the other would change three log strings. -- `view` is an optional descriptor member: rules and local records map - rows through `RuleView`/`RecordView`; without it neither could adopt - without changing its response body. When absent, the row serializes - as-is with no copy. -- `remove` has two comptime-detected arities: three handlers' decision - functions read rows and take an `Allocator` before the id, three do - not. The generator validates the full signature of whichever shape it - finds. +- `plural` is a separate descriptor member: the "listing X" log context differs from the JSON envelope key for three resources (`local_records`, `client_prefixes`, `forward_zones`), so deriving one from the other would change three log strings. +- `view` is an optional descriptor member: rules and local records map rows through `RuleView`/`RecordView`; without it neither could adopt without changing its response body. When absent, the row serializes as-is with no copy. +- `remove` has two comptime-detected arities: three handlers' decision functions read rows and take an `Allocator` before the id, three do not. The generator validates the full signature of whichever shape it finds. ### Ruling 4 (S4) -- The ruling's line numbers were stale after m17 (pool.zig's race sites - were at 184-206 and 306-335). -- `closeBlocked` branches at comptime on the close method's parameter - count: `tls_client.TlsStream.close()` takes no `Io` (it owns the one - it was built with), while the other three closes take `io`. -- "The stashed-cause accessor" is three accessors, one per collapse - point: the send phase reads `req.connection.?.stream_writer.err`, - `receiveHead` reads `Connection.getReadError()`, and the body read - consults `Response.bodyErr()` first (HTTP framing faults) then the - connection. -- `Connection.getReadError` can panic (it reads `stream_reader.err.?`). - The unwrap guards the plain-connection no-cause case; the TLS case - relies on std's documented contract that a cause exists after - `error.ReadFailed`. -- `raceWithin` requires the raced function's return type to be exactly - `ExchangeError!T` at comptime. Stricter than the ruling asked; it is - what keeps every failure path inside the race group. Consequence: - `manager.zig`'s `fetchWithin` (optional per the ruling) cannot adopt - it as written — its raced function has a different error set. -- The acceptance line "`fn expire(` production copies are gone" is - scoped to ruling 4's transport files. `filter/manager.zig`, - `storage/logger.zig` and `server/listener.zig` keep their own — none - is transport code. +- The ruling's line numbers were stale after m17 (pool.zig's race sites were at 184-206 and 306-335). +- `closeBlocked` branches at comptime on the close method's parameter count: `tls_client.TlsStream.close()` takes no `Io` (it owns the one it was built with), while the other three closes take `io`. +- "The stashed-cause accessor" is three accessors, one per collapse point: the send phase reads `req.connection.?.stream_writer.err`, `receiveHead` reads `Connection.getReadError()`, and the body read consults `Response.bodyErr()` first (HTTP framing faults) then the connection. +- `Connection.getReadError` can panic (it reads `stream_reader.err.?`). The unwrap guards the plain-connection no-cause case; the TLS case relies on std's documented contract that a cause exists after `error.ReadFailed`. +- `raceWithin` requires the raced function's return type to be exactly `ExchangeError!T` at comptime. Stricter than the ruling asked; it is what keeps every failure path inside the race group. Consequence: `manager.zig`'s `fetchWithin` (optional per the ruling) cannot adopt it as written — its raced function has a different error set. +- The acceptance line "`fn expire(` production copies are gone" is scoped to ruling 4's transport files. `filter/manager.zig`, `storage/logger.zig` and `server/listener.zig` keep their own — none is transport code. ### Rulings 5, 6, 7 (S5, S4) -- `nextBoundedLine` returns `.long_line` from the - EndOfStream-during-discard arm, and the next call returns `null`. The - two originals disagreed there (`compile` counted then broke; - `collectSample` returned without counting); this shape preserves both - behaviors — `compile` counts exactly as before, `collectSample` - ignores the event and sees `null`. Safe because - `discardDelimiterInclusive` drains the stream before reporting - `EndOfStream`. -- The length check runs on the raw line for both callers, as the ruling - placed it. One input changes classification: a line of exactly - `max_line_len + 1` bytes ending in `\r` was compiled before and now - counts as `long_lines`. Accepted as the intended reading. -- `check`'s per-message `FAIL` lines render import's path constant - (`FAIL config: `), not the file name. `checkImpl` - prints the file path immediately above and `check` examines one - source per run, and this keeps `check` and `import` rendering the - same failure identically — the acceptance criterion. +- `nextBoundedLine` returns `.long_line` from the EndOfStream-during-discard arm, and the next call returns `null`. The two originals disagreed there (`compile` counted then broke; `collectSample` returned without counting); this shape preserves both behaviors — `compile` counts exactly as before, `collectSample` ignores the event and sees `null`. Safe because `discardDelimiterInclusive` drains the stream before reporting `EndOfStream`. +- The length check runs on the raw line for both callers, as the ruling placed it. One input changes classification: a line of exactly `max_line_len + 1` bytes ending in `\r` was compiled before and now counts as `long_lines`. Accepted as the intended reading. +- `check`'s per-message `FAIL` lines render import's path constant (`FAIL config: `), not the file name. `checkImpl` prints the file path immediately above and `check` examines one source per run, and this keeps `check` and `import` rendering the same failure identically — the acceptance criterion. ### Rulings 8, 9 (S6) -- The fuzz-suite wiring did not fold into `addTestSuite` (the ruling's - own fallback): it lives in a separate `addFuzzSuite` helper, with a - shared `sourceModule` helper. The aarch64 target resolves from - `cross_targets[1]` behind a comptime prefix guard. -- `pairInput` moved into `smith_encode.zig` rather than staying in - blocklist_fuzz: m15's fuzz files had made it a two-copy duplicate, - which is the condition ruling 9 exists to remove. +- The fuzz-suite wiring did not fold into `addTestSuite` (the ruling's own fallback): it lives in a separate `addFuzzSuite` helper, with a shared `sourceModule` helper. The aarch64 target resolves from `cross_targets[1]` behind a comptime prefix guard. +- `pairInput` moved into `smith_encode.zig` rather than staying in blocklist_fuzz: m15's fuzz files had made it a two-copy duplicate, which is the condition ruling 9 exists to remove. ### Ruling 10 (S7) -- `ui/classes.ts` exports 17 constants, not the 8 listed — the extra - ones are the focus-ring fragment and compositions the 15 adopting - files needed to drop their literals without re-spelling anything. -- Seven ring-less focusable controls were fixed by adoption, not the - two the ruling named. The acceptance grep ("every input/button/select - carries the focus-visible fragment") was taken as the invariant over - the anti-requirement's count of two, which undercounted. +- `ui/classes.ts` exports 17 constants, not the 8 listed — the extra ones are the focus-ring fragment and compositions the 15 adopting files needed to drop their literals without re-spelling anything. +- Seven ring-less focusable controls were fixed by adoption, not the two the ruling named. The acceptance grep ("every input/button/select carries the focus-visible fragment") was taken as the invariant over the anti-requirement's count of two, which undercounted. ## Anti-requirements -- No behavioral changes: this milestone moves code. The only sanctioned - behavior deltas are the tcp counter rename, DoH's cause unwrap, the two - restored focus rings, and the `check` ZON rendering — each named above. +- No behavioral changes: this milestone moves code. The only sanctioned behavior deltas are the tcp counter rename, DoH's cause unwrap, the two restored focus rings, and the `check` ZON rendering — each named above. - No unification of the three intentionally-divergent normalize policies. - No unification of the four reload flavors. -- No `web/src/ui/` component library beyond `classes.ts` and - `useCrudForm.ts` — no button/table/dialog components in this milestone. -- No listener behavior additions (timeouts, keepalive) — m16 finished - those. +- No `web/src/ui/` component library beyond `classes.ts` and `useCrudForm.ts` — no button/table/dialog components in this milestone. +- No listener behavior additions (timeouts, keepalive) — m16 finished those. - No renaming of exported metric families beyond the one tcp rename. diff --git a/specs/milestone-19.md b/specs/milestone-19.md index 2fc3b92..ae2d726 100644 --- a/specs/milestone-19.md +++ b/specs/milestone-19.md @@ -1,212 +1,92 @@ # Milestone 19: hygiene sweep -Goal: remove the dead surface, unify the re-hardcoded constants, close the -small silent failures, and fix the frontend state hazards — the residue of -`TECH_DEBT.md` Theme 8 plus the remaining lows, after the behavioral fixes -(m16), the contract work (m17) and the duplication refactors (m18) have -landed. +Goal: remove the dead surface, unify the re-hardcoded constants, close the small silent failures, and fix the frontend state hazards — the residue of `TECH_DEBT.md` Theme 8 plus the remaining lows, after the behavioral fixes (m16), the contract work (m17) and the duplication refactors (m18) have landed. -**PROVISIONAL.** This spec was written before milestones 16-18 were built. It -assumes: m17 shipped BADVERS (so `edns.extendedRcode` has a production -caller), m17 built the upstream editor (so the queries.ts upstream factories -have consumers and are not deleted here), and m18 shipped the frontend `ui/` -extraction and deleted doh_server's dead `serve()`. Before starting this -milestone, re-verify every line reference and update this spec where the -earlier milestones moved things. +**PROVISIONAL.** This spec was written before milestones 16-18 were built. It assumes: m17 shipped BADVERS (so `edns.extendedRcode` has a production caller), m17 built the upstream editor (so the queries.ts upstream factories have consumers and are not deleted here), and m18 shipped the frontend `ui/` extraction and deleted doh_server's dead `serve()`. Before starting this milestone, re-verify every line reference and update this spec where the earlier milestones moved things. ## Rulings (binding) ### 1. The dead ECS parse surface is deleted -`src/dns/edns.zig`: delete `parseEcs` (:111), `Ecs` (:98-105), `EcsError` -(:107), `ecs_family_ipv4`/`ecs_family_ipv6` (:95-96), and their in-file tests -(:371-:469 block, nine tests). `stripEcs` filters by option code without -decoding, forward mode keys on raw bytes — no production path decodes ECS. +`src/dns/edns.zig`: delete `parseEcs` (:111), `Ecs` (:98-105), `EcsError` (:107), `ecs_family_ipv4`/`ecs_family_ipv6` (:95-96), and their in-file tests (:371-:469 block, nine tests). `stripEcs` filters by option code without decoding, forward mode keys on raw bytes — no production path decodes ECS. -Trap the audit missed: `tests/fuzz/dns_fuzz.zig:86` calls `edns.parseEcs` -inside a fuzz target that the default `test` step builds. Delete that line in -the same change or the build breaks. +Trap the audit missed: `tests/fuzz/dns_fuzz.zig:86` calls `edns.parseEcs` inside a fuzz target that the default `test` step builds. Delete that line in the same change or the build breaks. -`extendedRcode` (:272) is **kept**: milestone 17 gave it the BADVERS caller. -If m17 diverged and BADVERS was not built, delete it too and note the -divergence here. +`extendedRcode` (:272) is **kept**: milestone 17 gave it the BADVERS caller. If m17 diverged and BADVERS was not built, delete it too and note the divergence here. -Precedent: `ecsPayload` was built and removed within milestone 7 when it lost -its only caller (specs/milestone-7.md:361). Git history keeps the code. +Precedent: `ecsPayload` was built and removed within milestone 7 when it lost its only caller (specs/milestone-7.md:361). Git history keeps the code. ### 2. `writeSettings` returns an error union -`src/web/handlers/settings.zig:349` returns `?db.Error`, so every exit is a -value return and the `errdefer` at :354 never fires — it implies protection -it cannot provide. The real safety is two explicit `tx.rollback()` calls -(:358, :363 — the audit said three; it is two). +`src/web/handlers/settings.zig:349` returns `?db.Error`, so every exit is a value return and the `errdefer` at :354 never fires — it implies protection it cannot provide. The real safety is two explicit `tx.rollback()` calls (:358, :363 — the audit said three; it is two). -Change the return type to `db.Error!void`. The errdefer becomes live -(`db.Tx.rollback` at storage/db.zig:814 is idempotent and documented safe in -an errdefer), and the two manual rollback-then-return arms collapse into -plain `try`/error returns. The one caller (`applyPut`, :327) switches from -`if (writeSettings(...)) |err|` to `catch`; its hash-free-on-failure behavior -is unchanged. +Change the return type to `db.Error!void`. The errdefer becomes live (`db.Tx.rollback` at storage/db.zig:814 is idempotent and documented safe in an errdefer), and the two manual rollback-then-return arms collapse into plain `try`/error returns. The one caller (`applyPut`, :327) switches from `if (writeSettings(...)) |err|` to `catch`; its hash-free-on-failure behavior is unchanged. ### 3. The logger's field widths become the single source -`src/storage/logger.zig:44-48` holds four file-private constants -(`max_domain_len = 253`, `max_client_len = 45`, `max_reason_len = 32`, -`max_upstream_len = 64`). Make all four `pub`. Then: +`src/storage/logger.zig:44-48` holds four file-private constants (`max_domain_len = 253`, `max_client_len = 45`, `max_reason_len = 32`, `max_upstream_len = 64`). Make all four `pub`. Then: -- `src/server/handler.zig:67` deletes its local `max_reason_len = 32`; the - comptime guard at :69-75 checks `logger.max_reason_len`, so the guard - finally proves what its comment claims. -- `src/server/handler.zig:79` deletes its local `max_ip_text = 45` and uses - `logger.max_client_len` (the :82 `max_resolver_text` derivation follows). +- `src/server/handler.zig:67` deletes its local `max_reason_len = 32`; the comptime guard at :69-75 checks `logger.max_reason_len`, so the guard finally proves what its comment claims. +- `src/server/handler.zig:79` deletes its local `max_ip_text = 45` and uses `logger.max_client_len` (the :82 `max_resolver_text` derivation follows). - `src/server/clients.zig:32` likewise. -- `src/web/handlers/queries.zig:27-30` — the fifth copy the audit missed — - adopts `logger.max_domain_len` for the domain width. Its `max_client_len` - is **64**, not 45; the query log's client column is written by the logger - and can never exceed 45 bytes, so it adopts `logger.max_client_len` too. - If a test proves a wider value ever reaches that field, stop and record - the divergence instead of forcing it. +- `src/web/handlers/queries.zig:27-30` — the fifth copy the audit missed — adopts `logger.max_domain_len` for the domain width. Its `max_client_len` is **64**, not 45; the query log's client column is written by the logger and can never exceed 45 bytes, so it adopts `logger.max_client_len` too. If a test proves a wider value ever reaches that field, stop and record the divergence instead of forcing it. -Acceptance is a grep: the literals 253, 45 and 32 appear in these roles only -in logger.zig. +Acceptance is a grep: the literals 253, 45 and 32 appear in these roles only in logger.zig. ### 4. `track()` reports fullness; one mutex acquisition per query -`src/server/handler.zig:255-258` takes the tracker mutex twice per query: -`tracker.track(io, from)` and then `tracker.snapshotStats(io)` — a full -struct copy — solely to mirror `dropped_full` into the handler's atomic -`tracker_full` (:150-153). +`src/server/handler.zig:255-258` takes the tracker mutex twice per query: `tracker.track(io, from)` and then `tracker.snapshotStats(io)` — a full struct copy — solely to mirror `dropped_full` into the handler's atomic `tracker_full` (:150-153). -Change `Tracker.track`/`trackAt` (`src/server/clients.zig:89-97`) to return -`u64`: the current `dropped_full`, read under the mutex they already hold. -The handler stores that return value. `snapshotStats` stays for /metrics. +Change `Tracker.track`/`trackAt` (`src/server/clients.zig:89-97`) to return `u64`: the current `dropped_full`, read under the mutex they already hold. The handler stores that return value. `snapshotStats` stays for /metrics. ### 5. Log truncation gets a marker and a counter -`src/platform/logging.zig:285-288`: `mw.print(format, args) catch {}` on a -fixed 2048-byte writer, then the partial buffer is used with no marker — the -module's own never-both-discarded-and-silent invariant, violated in its own -kitchen. On `error.WriteFailed`: overwrite the final three bytes of the -buffered message with `...` (the safe_url.zig marker, src/safe_url.zig:11-14) -and increment a new `state.stats.lines_truncated`, sibling to -`lines_deduped` (:295). One test: a message over 2048 bytes ends in `...` -and bumps the counter by one. +`src/platform/logging.zig:285-288`: `mw.print(format, args) catch {}` on a fixed 2048-byte writer, then the partial buffer is used with no marker — the module's own never-both-discarded-and-silent invariant, violated in its own kitchen. On `error.WriteFailed`: overwrite the final three bytes of the buffered message with `...` (the safe_url.zig marker, src/safe_url.zig:11-14) and increment a new `state.stats.lines_truncated`, sibling to `lines_deduped` (:295). One test: a message over 2048 bytes ends in `...` and bumps the counter by one. ### 6. TLS errors are classified by name, not by name prefix -`src/filter/fetcher.zig:169-170` and `src/upstream/doh_client.zig:160-161` -classify TLS failures with `startsWith(@errorName(err), "Tls")` / -`"Certificate"` — a std rename silently downgrades TLS failures into generic -buckets. Verified reachable set from `std.http.Client` (0.16.0) **today**: -exactly `error.TlsInitializationFailed` and -`error.CertificateBundleLoadFailure`. But this milestone lands after -milestone 18, whose ruling 4 makes the DoH path unwrap the client's stashed -read cause — and that cause set includes the record-layer members -(`TlsAlert`, `TlsBadRecordMac`, `TlsDecodeError`, ...) from -`std.crypto.tls.Client`. A two-member switch written against today's -surface would downgrade those to generic receive failures. +`src/filter/fetcher.zig:169-170` and `src/upstream/doh_client.zig:160-161` classify TLS failures with `startsWith(@errorName(err), "Tls")` / `"Certificate"` — a std rename silently downgrades TLS failures into generic buckets. Verified reachable set from `std.http.Client` (0.16.0) **today**: exactly `error.TlsInitializationFailed` and `error.CertificateBundleLoadFailure`. But this milestone lands after milestone 18, whose ruling 4 makes the DoH path unwrap the client's stashed read cause — and that cause set includes the record-layer members (`TlsAlert`, `TlsBadRecordMac`, `TlsDecodeError`, ...) from `std.crypto.tls.Client`. A two-member switch written against today's surface would downgrade those to generic receive failures. -In both files: replace the prefix match with an exact switch → `error.TlsFailed` -naming the two top-level members **plus**, on whichever paths m18's cause -unwrap surfaces, the `Tls*` members of the unwrapped read-cause set — -enumerated at implementation time from the pinned std sources against the -m18 code as landed, not from this spec's list. The `else` falls through to -the existing per-phase mapping. Rewrite both tests (fetcher.zig:311-313, -doh_client.zig:261-263): they currently assert on `CertificateExpired`, -which nothing can produce, and omit `CertificateBundleLoadFailure`; the new -tests use only names from the verified reachable set, including a stashed -record-layer cause on the unwrapped path. +In both files: replace the prefix match with an exact switch → `error.TlsFailed` naming the two top-level members **plus**, on whichever paths m18's cause unwrap surfaces, the `Tls*` members of the unwrapped read-cause set — enumerated at implementation time from the pinned std sources against the m18 code as landed, not from this spec's list. The `else` falls through to the existing per-phase mapping. Rewrite both tests (fetcher.zig:311-313, doh_client.zig:261-263): they currently assert on `CertificateExpired`, which nothing can produce, and omit `CertificateBundleLoadFailure`; the new tests use only names from the verified reachable set, including a stashed record-layer cause on the unwrapped path. -If m18 already extracted a shared `mapError` into transport.zig, apply this -ruling to the shared copy instead and note it here. +If m18 already extracted a shared `mapError` into transport.zig, apply this ruling to the shared copy instead and note it here. ### 7. The transcribed mbedTLS values get an init-time check -`src/platform/tls_server.zig:501-511` hand-transcribes four config enums and -six error codes from the pinned Mbed TLS 3.6.7 headers with no drift guard — -unlike the sizes and alignment, which the shim reports. A renumbered -`close_notify` after a version bump would silently invert truncation -detection (:286, :351). +`src/platform/tls_server.zig:501-511` hand-transcribes four config enums and six error codes from the pinned Mbed TLS 3.6.7 headers with no drift guard — unlike the sizes and alignment, which the shim reports. A renumbered `close_notify` after a version bump would silently invert truncation detection (:286, :351). -`src/platform/mbedtls_shim.c` gains ten getters in the existing shape -(`int nx_const_ssl_transport_stream(void) { return MBEDTLS_SSL_TRANSPORT_STREAM; }` -etc. — the needed headers are already included). `tls_server.zig` verifies -all ten against the transcribed constants next to the existing alignment -assert at :78, and the :657 test block asserts them too. Cheap insurance, -one-time cost. +`src/platform/mbedtls_shim.c` gains ten getters in the existing shape (`int nx_const_ssl_transport_stream(void) { return MBEDTLS_SSL_TRANSPORT_STREAM; }` etc. — the needed headers are already included). `tls_server.zig` verifies all ten against the transcribed constants next to the existing alignment assert at :78, and the :657 test block asserts them too. Cheap insurance, one-time cost. ### 8. The reload lock invariant is written down -Fourteen call sites in four handler files (groups, rules, clients, -blocklists — the audit said five handlers; it is four files, fourteen sites) -call `mutations.reload` *after* releasing `config_lock`. That is safe only -because the production `reload_fn` re-reads the database under the manager's -own writer lock — a cross-module fact stated nowhere. `local.zig` is the -deliberate opposite (publish under the lock; the ordering rationale lives at -local.zig:80-85). +Fourteen call sites in four handler files (groups, rules, clients, blocklists — the audit said five handlers; it is four files, fourteen sites) call `mutations.reload` *after* releasing `config_lock`. That is safe only because the production `reload_fn` re-reads the database under the manager's own writer lock — a cross-module fact stated nowhere. `local.zig` is the deliberate opposite (publish under the lock; the ordering rationale lives at local.zig:80-85). -Extend the doc comment on `mutations.reload` -(src/web/handlers/mutations.zig:122-125): the `reload_fn` contract is that it -re-reads all state from the database itself; it must never accept pre-read -rows (contrast `swapLocalTables`, which is the pre-read shape and is why -local.zig holds the lock). A future signature change to `reload` that adds -row-passing breaks fourteen call sites' correctness — the comment must say -exactly that. Documentation only; no locking change. +Extend the doc comment on `mutations.reload` (src/web/handlers/mutations.zig:122-125): the `reload_fn` contract is that it re-reads all state from the database itself; it must never accept pre-read rows (contrast `swapLocalTables`, which is the pre-read shape and is why local.zig holds the lock). A future signature change to `reload` that adds row-passing breaks fourteen call sites' correctness — the comment must say exactly that. Documentation only; no locking change. ### 9. `web_dev_dir` moves into WebState -`src/app.zig:605-609` holds the `--web-dev` directory in a module-level -mutable global because the fallback handler "has no closure" — but -`serveWebDev` (:613-620) receives `*WebState` and discards it. Add -`dev_dir: []const u8 = ""` to `WebState` (src/web/server.zig, beside -`version`), set it at the composition root, read it in `serveWebDev`, delete -the global and its apologia comment. +`src/app.zig:605-609` holds the `--web-dev` directory in a module-level mutable global because the fallback handler "has no closure" — but `serveWebDev` (:613-620) receives `*WebState` and discards it. Add `dev_dir: []const u8 = ""` to `WebState` (src/web/server.zig, beside `version`), set it at the composition root, read it in `serveWebDev`, delete the global and its apologia comment. ### 10. The private key PEM is wiped -`src/server/cert_store.zig:326` frees the key PEM without zeroization, and -`readPem` (:345-362) uses `readFileAllocOptions`, whose grow-as-you-read can -leave intermediate copies in freed pages — a wipe at the call site cannot -reach them. Split the key path: a `readKeyPem` that reads through a single -fixed `max_pem_bytes + 1` buffer (no reallocation), copies to an exact-size -allocation, and `std.crypto.secureZero`s the big buffer before returning; the -caller wipes the returned slice before `gpa.free`. The idiom precedent is -auth.zig:228. The certificate PEM path stays as-is — it is public material. -Runs at boot and on real renewals only (:311-312 stats gate the poll). +`src/server/cert_store.zig:326` frees the key PEM without zeroization, and `readPem` (:345-362) uses `readFileAllocOptions`, whose grow-as-you-read can leave intermediate copies in freed pages — a wipe at the call site cannot reach them. Split the key path: a `readKeyPem` that reads through a single fixed `max_pem_bytes + 1` buffer (no reallocation), copies to an exact-size allocation, and `std.crypto.secureZero`s the big buffer before returning; the caller wipes the returned slice before `gpa.free`. The idiom precedent is auth.zig:228. The certificate PEM path stays as-is — it is public material. Runs at boot and on real renewals only (:311-312 stats gate the poll). ### 11. Header-array overflow asserts -`src/web/http_util.zig:335` maps a too-long `extra_headers` onto -`error.OutOfMemory` — a future eighth header would surface as mysterious -OOM-labeled drops. Every one of the 62 call sites passes a compile-time -count, maximum 3 (static.zig:174, +1 for content-type = 4 of 8). Replace the -check with `std.debug.assert(extra_headers.len + 1 <= headers.len);` so the -mistake fails loudly in tests. +`src/web/http_util.zig:335` maps a too-long `extra_headers` onto `error.OutOfMemory` — a future eighth header would surface as mysterious OOM-labeled drops. Every one of the 62 call sites passes a compile-time count, maximum 3 (static.zig:174, +1 for content-type = 4 of 8). Replace the check with `std.debug.assert(extra_headers.len + 1 <= headers.len);` so the mistake fails loudly in tests. ### 12. Shipped `.gz` siblings are verified; orphans are rejected -`tools/gen_web_assets.zig:71-77` embeds a dist-shipped `.gz` sibling with -zero content checks, and an orphan `.gz` (no base file) is silently embedded -as unreachable `application/octet-stream` bytes. Exposure is latent (no -frontend compression plugin today) — close it while it is cheap: +`tools/gen_web_assets.zig:71-77` embeds a dist-shipped `.gz` sibling with zero content checks, and an orphan `.gz` (no base file) is silently embedded as unreachable `application/octet-stream` bytes. Exposure is latent (no frontend compression plugin today) — close it while it is cheap: -- Sibling: decompress with `std.compress.flate.Decompress` (container - `.gzip`; the tool already uses the Compress side at :133) and byte-compare - against the base file; `std.process.fatal` on mismatch, matching the - tool's existing error style (:73-75). +- Sibling: decompress with `std.compress.flate.Decompress` (container `.gzip`; the tool already uses the Compress side at :133) and byte-compare against the base file; `std.process.fatal` on mismatch, matching the tool's existing error style (:73-75). - Orphan: `std.process.fatal` naming the path, instead of indexing it. -Extend the embedded-dist test (`src/web/static.zig:413-439`) to decompress -each `.gz` entry and compare with its base — it currently checks magic bytes -and size only. +Extend the embedded-dist test (`src/web/static.zig:413-439`) to decompress each `.gz` entry and compare with its base — it currently checks magic bytes and size only. ### 13. The Dockerfile arch default comes from the host -`deploy/docker/Dockerfile:21`: `${TARGETARCH:-amd64}` — under the legacy -builder TARGETARCH is empty, so a plain `docker build` on the Pi 5 packages -the x86_64 binary and dies at `docker run` with exec-format, far from the -mistake. Replace the default with a `uname -m` mapping resolved before the -case: +`deploy/docker/Dockerfile:21`: `${TARGETARCH:-amd64}` — under the legacy builder TARGETARCH is empty, so a plain `docker build` on the Pi 5 packages the x86_64 binary and dies at `docker run` with exec-format, far from the mistake. Replace the default with a `uname -m` mapping resolved before the case: ```dockerfile RUN arch="${TARGETARCH:-}"; \ @@ -221,10 +101,7 @@ The existing unsupported-arch arm stays fail-fast. ### 14. Frontend: the settings registry is typed -`web/src/features/settings/SettingsPage.tsx` — `SectionDef` exists (:21) but -`FieldDef.key` is `string` (:16), driving `as Record` casts -at :217, :226, :261. A typo'd key compiles and breaks the field. Make the -pair generic: +`web/src/features/settings/SettingsPage.tsx` — `SectionDef` exists (:21) but `FieldDef.key` is `string` (:16), driving `as Record` casts at :217, :226, :261. A typo'd key compiles and breaks the field. Make the pair generic: ```ts interface FieldDef { @@ -235,45 +112,19 @@ interface SectionDef { section: S; title: string; fiel function defineSection(def: SectionDef) { return def; } ``` -`SECTIONS` (:35-118, 12 sections) becomes a tuple of `defineSection(...)` -calls, `TLS_FIELDS` (:27) becomes `FieldDef<"doh_server" | "dot_server">[]` -compatible, and the three `Record` casts are deleted. The -per-branch value casts inside `FieldRow` (:141, :160, :174, :198) may stay — -they are narrowing on `kind`, not drift holes. Acceptance: renaming a -Settings field makes `tsc` fail on the registry. +`SECTIONS` (:35-118, 12 sections) becomes a tuple of `defineSection(...)` calls, `TLS_FIELDS` (:27) becomes `FieldDef<"doh_server" | "dot_server">[]` compatible, and the three `Record` casts are deleted. The per-branch value casts inside `FieldRow` (:141, :160, :174, :198) may stay — they are narrowing on `kind`, not drift holes. Acceptance: renaming a Settings field makes `tsc` fail on the registry. ### 15. Frontend: the settings baseline is frozen -Same file, :220 — `buildSettingsPatch(data.settings, edited, ...)` diffs the -mount-time clone against **live** query data, so a background refetch makes -out-of-band changes appear as user edits and Save silently reverts them. Add -a `baseline` state initialized with the same `structuredClone` and reset in -the same `onSuccess` (:234-241) where `edited` resets; the patch becomes -`buildSettingsPatch(baseline, edited, ...)`. `settingsDiff.ts` is unchanged. +Same file, :220 — `buildSettingsPatch(data.settings, edited, ...)` diffs the mount-time clone against **live** query data, so a background refetch makes out-of-band changes appear as user edits and Save silently reverts them. Add a `baseline` state initialized with the same `structuredClone` and reset in the same `onSuccess` (:234-241) where `edited` resets; the patch becomes `buildSettingsPatch(baseline, edited, ...)`. `settingsDiff.ts` is unchanged. ### 16. Frontend: refresh status leaves the query cache -`web/src/features/blocklists/BlocklistsPage.tsx:33` reads the refresh -snapshot with non-subscribing `getQueryData` from a cache entry that has no -queryFn, no subscriber, and the default 5-minute gcTime — after which the -page claims no refresh ever ran. This is client UI state. New module -`web/src/features/blocklists/refreshStore.ts` in the exact shape of -`settings/restartBanner.ts` (module-level value + listener Set + -`useSyncExternalStore` hook), holding `SourceStatus[] | null`. The mutation -(`blocklistsUpdateNowMutation`, lib/queries.ts:156) writes the store instead -of `setQueryData`; the page subscribes via the hook; the -`queryKeys.blocklistSources` entry and its doc comment are deleted. Note the -prefix-invalidation side effect disappears with it (invalidateBlocklistWorld -invalidating `["blocklists"]` used to mark the entry stale — harmless, since -the entry could never refetch). +`web/src/features/blocklists/BlocklistsPage.tsx:33` reads the refresh snapshot with non-subscribing `getQueryData` from a cache entry that has no queryFn, no subscriber, and the default 5-minute gcTime — after which the page claims no refresh ever ran. This is client UI state. New module `web/src/features/blocklists/refreshStore.ts` in the exact shape of `settings/restartBanner.ts` (module-level value + listener Set + `useSyncExternalStore` hook), holding `SourceStatus[] | null`. The mutation (`blocklistsUpdateNowMutation`, lib/queries.ts:156) writes the store instead of `setQueryData`; the page subscribes via the hook; the `queryKeys.blocklistSources` entry and its doc comment are deleted. Note the prefix-invalidation side effect disappears with it (invalidateBlocklistWorld invalidating `["blocklists"]` used to mark the entry stale — harmless, since the entry could never refetch). ### 17. Frontend: one default-group helper -Four encodings of "group 1", two semantics (GroupsPage.tsx:14 named -constant; PrefixesEditor.tsx:21 and LookupPage.tsx:133 byte-identical -prefer-id-1 expressions; RulesPage.tsx:24 first-of-list — which, because the -API orders by name (groups_repo.zig:148), preselects the alphabetically -first group, not the default). New module `web/src/lib/defaultGroup.ts`: +Four encodings of "group 1", two semantics (GroupsPage.tsx:14 named constant; PrefixesEditor.tsx:21 and LookupPage.tsx:133 byte-identical prefer-id-1 expressions; RulesPage.tsx:24 first-of-list — which, because the API orders by name (groups_repo.zig:148), preselects the alphabetically first group, not the default). New module `web/src/lib/defaultGroup.ts`: ```ts export const DEFAULT_GROUP_ID = 1; @@ -282,59 +133,25 @@ export function defaultGroupId(groups: readonly Group[]): number { } ``` -All four sites adopt it; RulesPage's preselect bug goes away with the -adoption. +All four sites adopt it; RulesPage's preselect bug goes away with the adoption. ### 18. Frontend: the dashboard primes without throwing -`web/src/routes.tsx:97-102` — the dashboard loader's `Promise.all` over four -`ensureQueryData` calls means one failing endpoint blanks the whole page with -`RouteError` on cold navigation, defeating the page's own per-widget -degrade (DashboardPage.tsx:68-96, which uses `useQuery` precisely to allow -it). Change **only the dashboard loader** to `Promise.allSettled`. The five -other `Promise.all` loaders stay: their pages use `useSuspenseQuery` and have -no granular fallback to preserve. +`web/src/routes.tsx:97-102` — the dashboard loader's `Promise.all` over four `ensureQueryData` calls means one failing endpoint blanks the whole page with `RouteError` on cold navigation, defeating the page's own per-widget degrade (DashboardPage.tsx:68-96, which uses `useQuery` precisely to allow it). Change **only the dashboard loader** to `Promise.allSettled`. The five other `Promise.all` loaders stay: their pages use `useSuspenseQuery` and have no granular fallback to preserve. ### 19. Frontend: the query-key table is complete -`web/src/lib/queries.ts` — eight raw literals, all in this file: seven -`["lookup"]` prefix-invalidations (:99, :125, :132, :159, :167, :189, :211) -and one `["groups"]` (:133). Add `lookupAll: ["lookup"] as const` to -`queryKeys`, use it at the seven sites, and use `queryKeys.groups` at :133. -Acceptance: no `["lookup"]`/`["groups"]` literal outside the table. +`web/src/lib/queries.ts` — eight raw literals, all in this file: seven `["lookup"]` prefix-invalidations (:99, :125, :132, :159, :167, :189, :211) and one `["groups"]` (:133). Add `lookupAll: ["lookup"] as const` to `queryKeys`, use it at the seven sites, and use `queryKeys.groups` at :133. Acceptance: no `["lookup"]`/`["groups"]` literal outside the table. ### 20. Frontend: the form catch swallows only the expected class -`web/src/features/blocklists/BlocklistForm.tsx:21-33` — the bare `catch {}` -wraps both the `await onSubmit` and the reset `setState` calls; anything but -the mutation rejection dies silently with no console trace. House precedent -is auth/store.tsx:85-89 (swallow only the 401 it expects, rethrow the rest), -pinned by two tests. Restructure: `try { await onSubmit(...) } catch (error) -{ if (error instanceof ApiError) return; throw error; }`, resets after the -try. The page's `` keeps rendering the mutation error as today. +`web/src/features/blocklists/BlocklistForm.tsx:21-33` — the bare `catch {}` wraps both the `await onSubmit` and the reset `setState` calls; anything but the mutation rejection dies silently with no console trace. House precedent is auth/store.tsx:85-89 (swallow only the 401 it expects, rethrow the rest), pinned by two tests. Restructure: `try { await onSubmit(...) } catch (error) { if (error instanceof ApiError) return; throw error; }`, resets after the try. The page's `` keeps rendering the mutation error as today. ### Carried in from milestone 17: the stale phase-comment sweep -Milestone 17 ruling 7 rewrote the "Phase 7/8" headers in three files; its -repo-wide acceptance grep then found 24 more stale phase references in -files its sessions did not own, recorded here for this milestone: -safesearch.zig:58, forward_client.zig:7, dns_cache.zig:10/:42/:435, -cli.zig:332, filter_integration_test.zig:1391, logger.zig:178/:303, -retention.zig:7/:137/:151, disk_monitor.zig:7, querylog_schema.zig:71, -db.zig:272, manager.zig:292/:613/:1145, and the six repository files -(groups_repo.zig:7, rules_repo.zig:15, sources_repo.zig:9, -upstreams_repo.zig:7, local_repo.zig:3, clients_repo.zig:12-13). Each -comment is rewritten to state the as-built truth it gestures at (not -deleted, unless it says nothing beyond the phase number). Line numbers -are hints from m17-time HEAD; re-grep before acting. Acceptance: "Phase -7" and "Phase 8" appear in no source doc comment repo-wide; spec files -and TECH_DEBT.md are exempt. The session owning each file in the plan -below picks up its share; files owned by no session fall to S4. +Milestone 17 ruling 7 rewrote the "Phase 7/8" headers in three files; its repo-wide acceptance grep then found 24 more stale phase references in files its sessions did not own, recorded here for this milestone: safesearch.zig:58, forward_client.zig:7, dns_cache.zig:10/:42/:435, cli.zig:332, filter_integration_test.zig:1391, logger.zig:178/:303, retention.zig:7/:137/:151, disk_monitor.zig:7, querylog_schema.zig:71, db.zig:272, manager.zig:292/:613/:1145, and the six repository files (groups_repo.zig:7, rules_repo.zig:15, sources_repo.zig:9, upstreams_repo.zig:7, local_repo.zig:3, clients_repo.zig:12-13). Each comment is rewritten to state the as-built truth it gestures at (not deleted, unless it says nothing beyond the phase number). Line numbers are hints from m17-time HEAD; re-grep before acting. Acceptance: "Phase 7" and "Phase 8" appear in no source doc comment repo-wide; spec files and TECH_DEBT.md are exempt. The session owning each file in the plan below picks up its share; files owned by no session fall to S4. -Also carried from milestone 17 (ruling 4 residual): the login log line in -`src/web/handlers/auth.zig` prints the socket peer, which reads as -loopback for every login behind a trusted proxy; it should print -`request.client_addr`. +Also carried from milestone 17 (ruling 4 residual): the login log line in `src/web/handlers/auth.zig` prints the socket peer, which reads as loopback for every login behind a trusted proxy; it should print `request.client_addr`. ## Sessions @@ -342,18 +159,11 @@ S1-S4 run in parallel; no two sessions write the same file. ### Session S1: backend surface -Owns `src/dns/edns.zig`, `tests/fuzz/dns_fuzz.zig`, -`src/web/handlers/settings.zig`, `src/web/handlers/mutations.zig`, -`src/web/http_util.zig`, `src/server/cert_store.zig`, `src/app.zig`, -`src/web/server.zig`. Rulings 1, 2, 8, 9, 10, 11. +Owns `src/dns/edns.zig`, `tests/fuzz/dns_fuzz.zig`, `src/web/handlers/settings.zig`, `src/web/handlers/mutations.zig`, `src/web/http_util.zig`, `src/server/cert_store.zig`, `src/app.zig`, `src/web/server.zig`. Rulings 1, 2, 8, 9, 10, 11. ### Session S2: constants, counters, classification -Owns `src/storage/logger.zig`, `src/server/handler.zig`, -`src/server/clients.zig`, `src/web/handlers/queries.zig`, -`src/platform/logging.zig`, `src/filter/fetcher.zig`, -`src/upstream/doh_client.zig`, `src/platform/tls_server.zig`, -`src/platform/mbedtls_shim.c`. Rulings 3, 4, 5, 6, 7. +Owns `src/storage/logger.zig`, `src/server/handler.zig`, `src/server/clients.zig`, `src/web/handlers/queries.zig`, `src/platform/logging.zig`, `src/filter/fetcher.zig`, `src/upstream/doh_client.zig`, `src/platform/tls_server.zig`, `src/platform/mbedtls_shim.c`. Rulings 3, 4, 5, 6, 7. ### Session S3: frontend @@ -361,13 +171,11 @@ Owns everything under `web/src/`. Rulings 14-20. ### Session S4: tools and deploy -Owns `tools/gen_web_assets.zig`, `src/web/static.zig`, -`deploy/docker/Dockerfile`. Rulings 12, 13. +Owns `tools/gen_web_assets.zig`, `src/web/static.zig`, `deploy/docker/Dockerfile`. Rulings 12, 13. ### Orchestrator -Re-verifies this spec's line references against post-m18 reality before S1-S4 -start; strikes the closed findings in `TECH_DEBT.md` after. +Re-verifies this spec's line references against post-m18 reality before S1-S4 start; strikes the closed findings in `TECH_DEBT.md` after. ## Module layout @@ -376,9 +184,7 @@ New files: - `web/src/features/blocklists/refreshStore.ts` — refresh snapshot store. - `web/src/lib/defaultGroup.ts` — default-group constant and helper. -Deleted surface: `parseEcs`/`Ecs`/`EcsError`/`ecs_family_*` (edns.zig), the -`web_dev_dir` global (app.zig), the `queryKeys.blocklistSources` entry -(queries.ts). +Deleted surface: `parseEcs`/`Ecs`/`EcsError`/`ecs_family_*` (edns.zig), the `web_dev_dir` global (app.zig), the `queryKeys.blocklistSources` entry (queries.ts). ## Acceptance (milestone complete) @@ -437,110 +243,44 @@ Accepted deviations and findings from the built milestone. ### Ruling 1 (S1) -- The edns.zig test block held six `parseEcs` tests, not nine. A seventh - use lived inside "encodeOpt round-trips through record parse and - parseOpt", which is not a parseEcs test; its last two lines now compare - the found option's bytes directly. -- `extendedRcode` stays, with a nuance the ruling's premise missed: the - BADVERS production path uses the write-side `splitRcode`; extendedRcode - is the read-side verifier whose callers are the two BADVERS wire-format - tests (handler.zig, packet.zig). Deleting it would weaken those tests. -- The write-fault seam for the rollback test has no exported control - surface — the test lives in the same file and needs none. -- The ruling-8 doc comment names `local.zig`'s `publish` function instead - of a line number, so it cannot go stale. +- The edns.zig test block held six `parseEcs` tests, not nine. A seventh use lived inside "encodeOpt round-trips through record parse and parseOpt", which is not a parseEcs test; its last two lines now compare the found option's bytes directly. +- `extendedRcode` stays, with a nuance the ruling's premise missed: the BADVERS production path uses the write-side `splitRcode`; extendedRcode is the read-side verifier whose callers are the two BADVERS wire-format tests (handler.zig, packet.zig). Deleting it would weaken those tests. +- The write-fault seam for the rollback test has no exported control surface — the test lives in the same file and needs none. +- The ruling-8 doc comment names `local.zig`'s `publish` function instead of a line number, so it cannot go stale. ### Rulings 3-7 (S2) -- Ownership amendment: `src/web/metrics.zig` (owned by no session) was - granted to S2 mid-flight. Ruling 5's new counter raises the /metrics - sample count 29 → 30 (`nxdns_log_lines_truncated_total`); the - hard-coded assertion became fully derived - (`1 + dns_stat_fields.len + @typeInfo(LoggerCounters)... + - @typeInfo(logging.Stats)...`), so a new counter in any of those - structs extends the exposition and the assertion together. -- Ruling 6 lands asymmetrically, following the ruling body over the - stale acceptance bullet: doh_client.zig names eleven members (the two - collapsed top-level ones plus the nine `Tls*` members of the unwrapped - read-cause set); fetcher.zig names only the two, because it has no - cause unwrap and no record-layer member can reach it. Both files carry - a comment pointing at the other. -- Addition beyond the ruling: an exact switch does not catch a std - rename — `error.X` in an expression names a member into existence, so - a renamed member leaves the switch compiling and matching nothing. - Comptime guards close this for real: doh_client.zig asserts every - member of `std.crypto.tls.Client.ReadError` maps to `TlsFailed`, and - both files assert the two collapsed names still exist in - `std.http.Client.RequestError`. -- The ruling's "stashed record-layer cause through the unwrap" test is - not constructible: the stub connection is `.plain` on purpose, and a - plain connection's stashed error type has no `Tls*` member. The - record-layer set is instead tested by an `inline for` over the std - error set against `mapError` — all nine members, not one sample. -- Ruling 3 behavioral note: queries.zig's client width dropped 64 → 45 - as directed; a `?client=` filter of 46-64 bytes now returns 400 - instead of matching nothing. `src/filter/wildcard.zig`'s - `max_pattern_len = 253` is a rule-pattern limit, not a logger role, - and stays. +- Ownership amendment: `src/web/metrics.zig` (owned by no session) was granted to S2 mid-flight. Ruling 5's new counter raises the /metrics sample count 29 → 30 (`nxdns_log_lines_truncated_total`); the hard-coded assertion became fully derived (`1 + dns_stat_fields.len + @typeInfo(LoggerCounters)... + @typeInfo(logging.Stats)...`), so a new counter in any of those structs extends the exposition and the assertion together. +- Ruling 6 lands asymmetrically, following the ruling body over the stale acceptance bullet: doh_client.zig names eleven members (the two collapsed top-level ones plus the nine `Tls*` members of the unwrapped read-cause set); fetcher.zig names only the two, because it has no cause unwrap and no record-layer member can reach it. Both files carry a comment pointing at the other. +- Addition beyond the ruling: an exact switch does not catch a std rename — `error.X` in an expression names a member into existence, so a renamed member leaves the switch compiling and matching nothing. Comptime guards close this for real: doh_client.zig asserts every member of `std.crypto.tls.Client.ReadError` maps to `TlsFailed`, and both files assert the two collapsed names still exist in `std.http.Client.RequestError`. +- The ruling's "stashed record-layer cause through the unwrap" test is not constructible: the stub connection is `.plain` on purpose, and a plain connection's stashed error type has no `Tls*` member. The record-layer set is instead tested by an `inline for` over the std error set against `mapError` — all nine members, not one sample. +- Ruling 3 behavioral note: queries.zig's client width dropped 64 → 45 as directed; a `?client=` filter of 46-64 bytes now returns 400 instead of matching nothing. `src/filter/wildcard.zig`'s `max_pattern_len = 253` is a rule-pattern limit, not a logger role, and stays. ### Rulings 14-20 (S3) -- Ruling 14: the three `Record` consumption casts - collapse into one documented cast inside `sectionValues` rather than - zero — iterating the heterogeneous registry erases the per-section - correlation TypeScript needs, and `TlsListenerSettings` (a named - interface, no implicit index signature) blocks the assignment route - with types.ts off-limits. Four casts before, one after. -- Ruling 20: the catch body lives in an exported `swallowMutationError` - helper because jsdom never fires `unhandledrejection`, so a rethrow - from an async submit handler cannot be asserted directly; the helper - is tested directly, mirroring how auth/store.tsx's logout pair is - tested. -- The refresh store notifies synchronously inside `onSuccess`, one flush - before react-query's success dispatch; one pre-existing assertion - moved from `getByText` to `await findByText`. The sub-frame transient - was judged cheaper than batching through notifyManager. -- `lib/queries.ts` imports `features/blocklists/refreshStore.ts` (lib → - features) because the spec directs the mutation factory to write the - store; recorded as sanctioned layering. -- `SourceStatusSection`'s prop is `SourceStatus[] | null` to match the - store; BlocklistsPage tests clear the module-level store in - `beforeEach`. -- Ruling 18: post-m18 there are four other `Promise.all` loaders, not - five. +- Ruling 14: the three `Record` consumption casts collapse into one documented cast inside `sectionValues` rather than zero — iterating the heterogeneous registry erases the per-section correlation TypeScript needs, and `TlsListenerSettings` (a named interface, no implicit index signature) blocks the assignment route with types.ts off-limits. Four casts before, one after. +- Ruling 20: the catch body lives in an exported `swallowMutationError` helper because jsdom never fires `unhandledrejection`, so a rethrow from an async submit handler cannot be asserted directly; the helper is tested directly, mirroring how auth/store.tsx's logout pair is tested. +- The refresh store notifies synchronously inside `onSuccess`, one flush before react-query's success dispatch; one pre-existing assertion moved from `getByText` to `await findByText`. The sub-frame transient was judged cheaper than batching through notifyManager. +- `lib/queries.ts` imports `features/blocklists/refreshStore.ts` (lib → features) because the spec directs the mutation factory to write the store; recorded as sanctioned layering. +- `SourceStatusSection`'s prop is `SourceStatus[] | null` to match the store; BlocklistsPage tests clear the module-level store in `beforeEach`. +- Ruling 18: post-m18 there are four other `Promise.all` loaders, not five. ### Rulings 12, 13 and carry-ins (S4) -- The undecodable-sibling message reports `decompress.err orelse err`, - because `std.Io.Reader` collapses every decode failure into - `error.ReadFailed` and stashes the real one. -- The Dockerfile keeps its `&&` chain instead of the spec sketch's `;` - separators, so any failed step still aborts the layer; the case arms - are untouched. -- The phase-comment sweep leaves `PLAN.md:605`/`:609` alone — they are - the section headings that define Phase 7 and Phase 8, not stale - comments. -- The orchestrator added the host-fallback sentence to - docs/how-to/install-with-docker.md. +- The undecodable-sibling message reports `decompress.err orelse err`, because `std.Io.Reader` collapses every decode failure into `error.ReadFailed` and stashes the real one. +- The Dockerfile keeps its `&&` chain instead of the spec sketch's `;` separators, so any failed step still aborts the layer; the case arms are untouched. +- The phase-comment sweep leaves `PLAN.md:605`/`:609` alone — they are the section headings that define Phase 7 and Phase 8, not stale comments. +- The orchestrator added the host-fallback sentence to docs/how-to/install-with-docker.md. ### Residual (named, not fixed) -`fetcher.zig` hands the collapsed `ReadFailed`/`WriteFailed` to -`mapError` with no cause unwrap, so a blocklist download canceled at -shutdown returns `error.ReceiveFailed` instead of `error.Canceled` — the -class m18 ruling 4 fixed for DoH. Consequence: a wrong source status and -a spurious warn line at shutdown. Fixing it needs `req`/`resp` plumbing -at four call sites; it deserves its own ruling. +`fetcher.zig` hands the collapsed `ReadFailed`/`WriteFailed` to `mapError` with no cause unwrap, so a blocklist download canceled at shutdown returns `error.ReceiveFailed` instead of `error.Canceled` — the class m18 ruling 4 fixed for DoH. Consequence: a wrong source status and a spurious warn line at shutdown. Fixing it needs `req`/`resp` plumbing at four call sites; it deserves its own ruling. ## Anti-requirements -- No deletion of `extendedRcode` (m17 gave it a caller) and no deletion of - the upstream mutation factories (m17 built their consumer). +- No deletion of `extendedRcode` (m17 gave it a caller) and no deletion of the upstream mutation factories (m17 built their consumer). - No locking change in the reload handlers — ruling 8 is documentation. - No new frontend dependencies; the store is hand-rolled like restartBanner. -- No compression plugin for the frontend build; ruling 12 guards the seam, - it does not start using it. -- No `useInfiniteQuery` migration and no QueryLogPage work — that was - milestone 16. -- No renaming of settings keys or API fields — typing the registry must not - change the wire format. +- No compression plugin for the frontend build; ruling 12 guards the seam, it does not start using it. +- No `useInfiniteQuery` migration and no QueryLogPage work — that was milestone 16. +- No renaming of settings keys or API fields — typing the registry must not change the wire format. diff --git a/specs/milestone-20.md b/specs/milestone-20.md index dd000e0..6c57e5f 100644 --- a/specs/milestone-20.md +++ b/specs/milestone-20.md @@ -1,185 +1,64 @@ # Milestone 20: declarative configuration for IaC -Goal: a config file an operator can keep in git and deploy with Ansible, where -the file is the sole declarative source of truth, converged at every boot — -without re-downloading every blocklist on every boot, and without the UI -silently diverging from the file. Two authority modes, selected by the -presence of one flag: bare `run` serves the DB; `run --config=` makes -the file authority. +Goal: a config file an operator can keep in git and deploy with Ansible, where the file is the sole declarative source of truth, converged at every boot — without re-downloading every blocklist on every boot, and without the UI silently diverging from the file. Two authority modes, selected by the presence of one flag: bare `run` serves the DB; `run --config=` makes the file authority. -Design finalized 2026-08-09 after three adversarial rounds (red team ops 1-16 -and debt 1-14; Codex cross-validation F1-F12; a Codex round on the premise -revision). All findings are folded in below or declined with written reasons; -all file:line anchors were re-verified at the pre-implementation HEAD -(7039a9f). A premise revision replaced the earlier `--config-source` mode -enum with presence-of-`--config` and deleted the persisted authority marker; -rulings 1, 6 and 8 record the reasons — do not reintroduce either. +Design finalized 2026-08-09 after three adversarial rounds (red team ops 1-16 and debt 1-14; Codex cross-validation F1-F12; a Codex round on the premise revision). All findings are folded in below or declined with written reasons; all file:line anchors were re-verified at the pre-implementation HEAD (7039a9f). A premise revision replaced the earlier `--config-source` mode enum with presence-of-`--config` and deleted the persisted authority marker; rulings 1, 6 and 8 record the reasons — do not reintroduce either. ## Implementation contract (read first) - Read `AGENTS.md`, then this spec whole, before session work starts. -- Session order: R1 → R2 sequential; R3 and R4 parallel to both (Sessions, - below). File ownership is write-exclusivity; the interfaces between - sessions (`reconcile.Summary`, `WebState.authority` + `reconciled_at`, - `RouteInfo.policy`) are fixed in this spec and are not renegotiable - mid-build. -- Gates: `zig build test` and `zig build test -Dintegration` with 0 failed, - plus the drift-guard regenerations the Tests section names. Skip counts are - reported with their reasons (plain-suite skips are integration-gated; the 4 - integration skips are the live-network TLS tests excluded by milestone-1 - design). One `-Dlive` run covers the real download path (Tests). -- No `std.log.err` in new code. No secrets in logs — url redaction goes - through `src/safe_url.zig` as everywhere else. -- Every fix or behaviour claim lands with a test the author watched fail - against the reverted implementation (milestone-13 ruling F-f applies). -- Doc pages touched by R4 follow milestone-13 ruling 3: every command block - in `tutorial/` and `how-to/` is executed on this host by the session that - writes it, or marked in-page as unverified with the reason. -- The final commit is GPG-signed by the user (`git commit -S`, lowercase, - single line); stage the work and hand the command over. +- Session order: R1 → R2 sequential; R3 and R4 parallel to both (Sessions, below). File ownership is write-exclusivity; the interfaces between sessions (`reconcile.Summary`, `WebState.authority` + `reconciled_at`, `RouteInfo.policy`) are fixed in this spec and are not renegotiable mid-build. +- Gates: `zig build test` and `zig build test -Dintegration` with 0 failed, plus the drift-guard regenerations the Tests section names. Skip counts are reported with their reasons (plain-suite skips are integration-gated; the 4 integration skips are the live-network TLS tests excluded by milestone-1 design). One `-Dlive` run covers the real download path (Tests). +- No `std.log.err` in new code. No secrets in logs — url redaction goes through `src/safe_url.zig` as everywhere else. +- Every fix or behaviour claim lands with a test the author watched fail against the reverted implementation (milestone-13 ruling F-f applies). +- Doc pages touched by R4 follow milestone-13 ruling 3: every command block in `tutorial/` and `how-to/` is executed on this host by the session that writes it, or marked in-page as unverified with the reason. +- The final commit is GPG-signed by the user (`git commit -S`, lowercase, single line); stage the work and hand the command over. ## Rulings (binding) ### 1. Authority is the invocation: `--config` present means the file governs -`nxdns run` — the DB is authority, today's appliance behaviour. -`nxdns run --config=/etc/nxdns/config.zon` — the file is authority. The -flag's presence selects the mode; there is no mode enum and no default path. -`check` keeps its existing surface under the same rule: bare `check` grades -the DB, `check --config ` grades the file. The DB-exists-wins heuristic -in `checkImpl` (cli.zig:559-588) and `CheckArgs.config_explicit` are deleted. +`nxdns run` — the DB is authority, today's appliance behaviour. `nxdns run --config=/etc/nxdns/config.zon` — the file is authority. The flag's presence selects the mode; there is no mode enum and no default path. `check` keeps its existing surface under the same rule: bare `check` grades the DB, `check --config ` grades the file. The DB-exists-wins heuristic in `checkImpl` (cli.zig:559-588) and `CheckArgs.config_explicit` are deleted. -Two principles, separated deliberately, because conflating them is what -produced the rejected `--config-source` enum: +Two principles, separated deliberately, because conflating them is what produced the rejected `--config-source` enum: -- **Authority must be explicit in the invocation.** An operator reads - `ExecStart` and knows which authority is live. Probing `/etc/nxdns` for a - file and switching behaviour on its existence is ambient magic — the same - class of heuristic that made seed-once bootstrap a source of doc lies (the - compose comment, the first-run tutorial) — and stays banned: a file on disk - that no flag names changes nothing. -- **A mode enum is the wrong shape for a two-state choice a path already - expresses.** `--config-source=db` names an implementation, not an operator - intent, and a mode flag beside a path flag manufactures invalid - combinations (path without mode, mode without path) that then need pairing - rules and usage errors to defend. Presence-of-path has no invalid - combinations and nothing to defend. +- **Authority must be explicit in the invocation.** An operator reads `ExecStart` and knows which authority is live. Probing `/etc/nxdns` for a file and switching behaviour on its existence is ambient magic — the same class of heuristic that made seed-once bootstrap a source of doc lies (the compose comment, the first-run tutorial) — and stays banned: a file on disk that no flag names changes nothing. +- **A mode enum is the wrong shape for a two-state choice a path already expresses.** `--config-source=db` names an implementation, not an operator intent, and a mode flag beside a path flag manufactures invalid combinations (path without mode, mode without path) that then need pairing rules and usage errors to defend. Presence-of-path has no invalid combinations and nothing to defend. -Breaking change, named: `run --config ` today means seed-once; the same -syntax now means file authority — reconcile on every boot, UI config writes -rejected. For an operator who seeded once and then configured through the UI, -the first post-upgrade restart converges the DB to that old seed file, -deleting the UI edits. The upgrade doc's breaking-changes section leads with -this and gives the two exits (ruling 9): drop the flag, or re-export to the -file path first. `check --config` keeps its meaning exactly. Greenfield rules -apply — the seed-once interface does not survive for compatibility — but the -break is loud in the docs, never silent-by-omission. +Breaking change, named: `run --config ` today means seed-once; the same syntax now means file authority — reconcile on every boot, UI config writes rejected. For an operator who seeded once and then configured through the UI, the first post-upgrade restart converges the DB to that old seed file, deleting the UI edits. The upgrade doc's breaking-changes section leads with this and gives the two exits (ruling 9): drop the flag, or re-export to the file path first. `check --config` keeps its meaning exactly. Greenfield rules apply — the seed-once interface does not survive for compatibility — but the break is loud in the docs, never silent-by-omission. -A rename (`--managed-config`) that would make the old invocation fail loudly -was considered twice and declined: it trades a worse name and a permanent -asymmetry with `check --config` against a one-time hazard whose exposed -population is the pre-change install base of a project whose first release is -days old. The hazard is real and the docs lead with it; the interface does -not carry the scar. This is a judgment, recorded so it is revisited only with -new facts (a real install base would be one). +A rename (`--managed-config`) that would make the old invocation fail loudly was considered twice and declined: it trades a worse name and a permanent asymmetry with `check --config` against a one-time hazard whose exposed population is the pre-change install base of a project whose first release is days old. The hazard is real and the docs lead with it; the interface does not carry the scar. This is a judgment, recorded so it is revisited only with new facts (a real install base would be one). ### 2. File mode fails closed, and *declarative* failure is exit 2 -File mode contract: the file is the sole declarative source; the DB stays the -runtime substrate and the effective-config read path. Startup sequence, -replacing the `seedFromFile` call at app.zig:196: +File mode contract: the file is the sole declarative source; the DB stays the runtime substrate and the effective-config read path. Startup sequence, replacing the `seedFromFile` call at app.zig:196: 1. `DataDir.open` → open config DB → `migrate` (unchanged). -2. Read the file (existing 4 MiB cap), ZON parse (arena, never freed — keep - the import.zig discipline), `validate.validate`. -3. Reconcile into the DB in one `BEGIN IMMEDIATE` transaction, `errdefer` - rollback (ruling 3). +2. Read the file (existing 4 MiB cap), ZON parse (arena, never freed — keep the import.zig discipline), `validate.validate`. +3. Reconcile into the DB in one `BEGIN IMMEDIATE` transaction, `errdefer` rollback (ruling 3). 4. `config_export.readConfig` (app.zig:206) → serve, unchanged from there on. -A missing, unreadable, or invalid file fails startup. Never fall back to the -DB: a fallback turns a deploy typo into a silently stale config. +A missing, unreadable, or invalid file fails startup. Never fall back to the DB: a fallback turns a deploy typo into a silently stale config. -Exit codes keep the 2am contract — exit 2 means "your config is wrong, run -`nxdns check`"; exit 1 means "the box is wrong". `faults.isConfigFault` -deliberately excludes `FileNotFound`/`AccessDenied` (faults.zig:107-108), and -that stays true in general. The file-mode loader is the seam, and it maps -**path-class open failures only**: `FileNotFound`, `AccessDenied`, -`PermissionDenied`, `NotDir`, `IsDir`, `SymLinkLoop`, `NameTooLong`, -`BadPathName` become `error.ManagedConfigUnreadable` (message includes the -path), which joins the `ConfigFault` set beside `ParseZon`/`ConfigTooLarge`. -Every other member of `ReadFileAllocError` — `SystemResources`, -`ProcessFdQuotaExceeded`, `SystemFdQuotaExceeded`, I/O errors, `OutOfMemory` — -propagates unmapped, exit 1: those are box faults a retry can clear, and once -ruling 9 adds `RestartPreventExitStatus=2 64`, mapping them to exit 2 would -stop the unit permanently on a transient fault. The mapping is an explicit -named error set in the loader, switched exhaustively with -`else => |other| return other`, so a std error added in a Zig bump defaults to -exit 1 rather than silently to exit 2 — the same closed-set discipline -faults.zig already documents. The mapping lives in **one shared helper** used -by both `run` and `check`; two copies would let the two grade the same -unreadable file differently, exactly the divergence faults.zig:1-8 exists to -abolish. +Exit codes keep the 2am contract — exit 2 means "your config is wrong, run `nxdns check`"; exit 1 means "the box is wrong". `faults.isConfigFault` deliberately excludes `FileNotFound`/`AccessDenied` (faults.zig:107-108), and that stays true in general. The file-mode loader is the seam, and it maps **path-class open failures only**: `FileNotFound`, `AccessDenied`, `PermissionDenied`, `NotDir`, `IsDir`, `SymLinkLoop`, `NameTooLong`, `BadPathName` become `error.ManagedConfigUnreadable` (message includes the path), which joins the `ConfigFault` set beside `ParseZon`/`ConfigTooLarge`. Every other member of `ReadFileAllocError` — `SystemResources`, `ProcessFdQuotaExceeded`, `SystemFdQuotaExceeded`, I/O errors, `OutOfMemory` — propagates unmapped, exit 1: those are box faults a retry can clear, and once ruling 9 adds `RestartPreventExitStatus=2 64`, mapping them to exit 2 would stop the unit permanently on a transient fault. The mapping is an explicit named error set in the loader, switched exhaustively with `else => |other| return other`, so a std error added in a Zig bump defaults to exit 1 rather than silently to exit 2 — the same closed-set discipline faults.zig already documents. The mapping lives in **one shared helper** used by both `run` and `check`; two copies would let the two grade the same unreadable file differently, exactly the divergence faults.zig:1-8 exists to abolish. -**Scope of the check/run agreement claim**: `check --config=` -grades exactly the declarative faults `run` would hit — read (path-class), -parse, size, validate — through the same shared loader helper. Reconcile-time -faults (FK violations, `SQLITE_BUSY` from a restart race, disk full) are -runtime faults, exit 1, and structurally invisible to `check`, which needs no -DB. The acceptance encodes the scoped claim, not "check passing guarantees run -converges". The one *systematic* gap the red team found — a group removal -tripping the un-cascaded `clients.group_id` FK against observed rows `check` -cannot see — is eliminated in the engine itself (ruling 3's reassign rule), -not papered over in `check`. +**Scope of the check/run agreement claim**: `check --config=` grades exactly the declarative faults `run` would hit — read (path-class), parse, size, validate — through the same shared loader helper. Reconcile-time faults (FK violations, `SQLITE_BUSY` from a restart race, disk full) are runtime faults, exit 1, and structurally invisible to `check`, which needs no DB. The acceptance encodes the scoped claim, not "check passing guarantees run converges". The one *systematic* gap the red team found — a group removal tripping the un-cascaded `clients.group_id` FK against observed rows `check` cannot see — is eliminated in the engine itself (ruling 3's reassign rule), not papered over in `check`. -Reconcile-time diagnostics include the SQLite error (`SQLITE_FULL` by name -when that is the cause) so a full SD card reads as "disk", not as a bare -exit 1. Diagnostics render to `r.err` and flush immediately, keeping the -current `seedFromFile` discipline (app.zig:137-178) — `serve` never returns, -so a buffered error line is a lost error line. +Reconcile-time diagnostics include the SQLite error (`SQLITE_FULL` by name when that is the cause) so a full SD card reads as "disk", not as a bare exit 1. Diagnostics render to `r.err` and flush immediately, keeping the current `seedFromFile` discipline (app.zig:137-178) — `serve` never returns, so a buffered error line is a lost error line. -Because every boot revalidates the file, a latent file error is no longer a -first-boot-only hazard: a bad push that skips its restart handler detonates at -the next power blip. Two mitigations, both in ruling 9: the shipped unit gains -`RestartPreventExitStatus=2 64` (retrying a config fault every 2 s is pure -loop; the journal holds the diagnostics), and the deployment docs mandate -`nxdns check --config=` as the pre-restart gate in any Ansible -handler. +Because every boot revalidates the file, a latent file error is no longer a first-boot-only hazard: a bad push that skips its restart handler detonates at the next power blip. Two mitigations, both in ruling 9: the shipped unit gains `RestartPreventExitStatus=2 64` (retrying a config fault every 2 s is pure loop; the journal holds the diagnostics), and the deployment docs mandate `nxdns check --config=` as the pre-restart gate in any Ansible handler. -Db mode: steps 2-3 are skipped entirely; the DB is truth exactly as today. -Bare `check` on a box with no `config.db` exits 2: `no config database at ` -plus the ruling-6 hint line — the deleted heuristic's "nothing to check" -branch (cli.zig:582-587) is replaced, not dropped. +Db mode: steps 2-3 are skipped entirely; the DB is truth exactly as today. Bare `check` on a box with no `config.db` exits 2: `no config database at ` plus the ruling-6 hint line — the deleted heuristic's "nothing to check" branch (cli.zig:582-587) is replaced, not dropped. ### 3. Reconcile engine: replace declarative state, preserve runtime state by stable identity -New module `src/config/reconcile.zig`, replacing `applyToDb`'s wipe+reinsert. -The defect it exists to fix: today's import deletes and reinserts -`blocklist_sources` including runtime columns (checksum, `last_updated`, -counters — config_schema.zig:46-56), and compiled blocklists are keyed by -source row id (`.list`/`.wild`). A naive re-import every boot would -force a full re-download and recompile of every blocklist on every restart. +New module `src/config/reconcile.zig`, replacing `applyToDb`'s wipe+reinsert. The defect it exists to fix: today's import deletes and reinserts `blocklist_sources` including runtime columns (checksum, `last_updated`, counters — config_schema.zig:46-56), and compiled blocklists are keyed by source row id (`.list`/`.wild`). A naive re-import every boot would force a full re-download and recompile of every blocklist on every restart. -Contract, per table: match rows by identity key ⇒ UPDATE declarative columns -in place (row id survives); present in DB but absent from the file ⇒ DELETE; -present in file but not DB ⇒ INSERT. Never wipe. One transaction. The -`clients` table refines the match rule with promotion (below): an observed row -whose IP the file declares is matched and updated, never deleted. +Contract, per table: match rows by identity key ⇒ UPDATE declarative columns in place (row id survives); present in DB but absent from the file ⇒ DELETE; present in file but not DB ⇒ INSERT. Never wipe. One transaction. The `clients` table refines the match rule with promotion (below): an observed row whose IP the file declares is matched and updated, never deleted. -**Identity matching is on canonical forms.** Import already canonicalizes -client IPs and prefixes before insert (import.zig:226-237; `FD00:0:0:0:0:0:0:1` -stores as `fd00::1`) and the schema documents the columns as canonical text. -The engine canonicalizes file values *before* matching; matching the raw file -string against the canonical column would churn ids under an unchanged -non-canonical file, violating ruling 5 in a way an exported-config test -(export emits canonical forms) cannot catch. The idempotence test includes a -non-canonical file. +**Identity matching is on canonical forms.** Import already canonicalizes client IPs and prefixes before insert (import.zig:226-237; `FD00:0:0:0:0:0:0:1` stores as `fd00::1`) and the schema documents the columns as canonical text. The engine canonicalizes file values *before* matching; matching the raw file string against the canonical column would churn ids under an unchanged non-canonical file, violating ruling 5 in a way an exported-config test (export emits canonical forms) cannot catch. The idempotence test includes a non-canonical file. -**Writes only on difference.** A matched row whose declarative columns already -equal the file's values gets no UPDATE. This is stronger than byte-stability: -reconciling an unchanged file performs **zero writes**, so a no-op boot needs -no WAL headroom and a querylog-full SD card cannot brick a file-mode restart -that db mode would survive. Testable: the second reconcile returns an all-zero -summary and `sqlite3_total_changes` does not move. +**Writes only on difference.** A matched row whose declarative columns already equal the file's values gets no UPDATE. This is stronger than byte-stability: reconciling an unchanged file performs **zero writes**, so a no-op boot needs no WAL headroom and a querylog-full SD card cannot brick a file-mode restart that db mode would survive. Testable: the second reconcile returns an all-zero summary and `sqlite3_total_changes` does not move. | Table | Identity key | Declarative columns | Runtime-owned (preserved on match) | |---|---|---|---| @@ -191,77 +70,21 @@ summary and `sqlite3_total_changes` does not move. | `group_sources` | `(group_id, source_id)`, resolved via the name/url maps | whole row | — | | `settings` | key | value via `settings_repo.putSetting` upsert, on difference only | delete keys not produced by `toSettings`, **except** `web.password_hash`, which the reconciler owns directly (ruling 4) | -**Pass order (binding).** `delete_order` alone under-specifies the engine: -inserts need parents before children, and a group DELETE cascades through -`rules`, `client_prefixes`, `group_sources` (config_schema.zig:35, :60, :67), -which could silently undo child-table work and corrupt the diff counts if -deletes ran first or interleaved. The order is: +**Pass order (binding).** `delete_order` alone under-specifies the engine: inserts need parents before children, and a group DELETE cascades through `rules`, `client_prefixes`, `group_sources` (config_schema.zig:35, :60, :67), which could silently undo child-table work and corrupt the diff counts if deletes ran first or interleaved. The order is: -- **Phase A — upserts, parents first**: `groups`, `blocklist_sources`, then - the referrers (`clients`, `client_prefixes`, `rules`, `group_sources`, - `upstreams`, `local_records`, `forward_zones`, `settings`), with the - name→id / url→id maps built after the parent passes. Client promotion - happens here, in the `clients` pass. -- **Phase B — delete-absent, child first**, in `delete_order` - (config_schema.zig:99). Because every declarative child of a dying group is - itself absent from the file (validate guarantees file rules/prefixes - reference file groups), the child passes have already deleted and counted - them by the time the parent DELETE runs; the FK cascades become a safety - net, never the accountant. -- Immediately before the `groups` delete pass: observed clients - (`hand_edited=0`) whose `group_id` belongs to a dying group are reassigned - to the default group (id 1). `clients.group_id` has **no** ON DELETE clause - (config_schema.zig:26) — without this step, removing or renaming a group - that observed devices had been assigned to trips the FK mid-transaction: - exit 1, restart loop, and a `check` that said the file was fine. The - reassignment is the semantics we want anyway: the operator un-declared the - group, not the devices. +- **Phase A — upserts, parents first**: `groups`, `blocklist_sources`, then the referrers (`clients`, `client_prefixes`, `rules`, `group_sources`, `upstreams`, `local_records`, `forward_zones`, `settings`), with the name→id / url→id maps built after the parent passes. Client promotion happens here, in the `clients` pass. +- **Phase B — delete-absent, child first**, in `delete_order` (config_schema.zig:99). Because every declarative child of a dying group is itself absent from the file (validate guarantees file rules/prefixes reference file groups), the child passes have already deleted and counted them by the time the parent DELETE runs; the FK cascades become a safety net, never the accountant. +- Immediately before the `groups` delete pass: observed clients (`hand_edited=0`) whose `group_id` belongs to a dying group are reassigned to the default group (id 1). `clients.group_id` has **no** ON DELETE clause (config_schema.zig:26) — without this step, removing or renaming a group that observed devices had been assigned to trips the FK mid-transaction: exit 1, restart loop, and a `check` that said the file was fine. The reassignment is the semantics we want anyway: the operator un-declared the group, not the devices. -Consequences that make the fix complete: an unchanged URL keeps id **and** -checksum + counters + `last_updated` together, so `loadSource` never sees -`.never_fetched`, `needsRefresh` does not fire spuriously (preserving -`last_updated` matters — checksum alone is not enough), `sweepOrphans` never -orphans `.list`/`.wild`, and a restart costs zero downloads. A changed -URL is a new identity: new row, new id, fresh download — consistent with -artifacts keyed by id. **Accepted trade, stated**: the old row's compiled -`.list`/`.wild` are orphaned at commit and swept before the -new source's first fetch succeeds, so a URL edit whose new host is down leaves -that list unenforced until a fetch lands. Deferring the sweep until a -successor fetch would need a cross-artifact lifecycle for an event that is -rare, operator-initiated, and bounded by the scheduler's retry — not worth the -machinery on a household box. The REST `updateSource` keeps runtime columns -across a URL edit; in file mode that route is rejected (ruling 7), so the -divergence is unreachable; in db mode the reconciler only runs via explicit -`import`. +Consequences that make the fix complete: an unchanged URL keeps id **and** checksum + counters + `last_updated` together, so `loadSource` never sees `.never_fetched`, `needsRefresh` does not fire spuriously (preserving `last_updated` matters — checksum alone is not enough), `sweepOrphans` never orphans `.list`/`.wild`, and a restart costs zero downloads. A changed URL is a new identity: new row, new id, fresh download — consistent with artifacts keyed by id. **Accepted trade, stated**: the old row's compiled `.list`/`.wild` are orphaned at commit and swept before the new source's first fetch succeeds, so a URL edit whose new host is down leaves that list unenforced until a fetch lands. Deferring the sweep until a successor fetch would need a cross-artifact lifecycle for an event that is rare, operator-initiated, and bounded by the scheduler's retry — not worth the machinery on a household box. The REST `updateSource` keeps runtime columns across a URL edit; in file mode that route is rejected (ruling 7), so the divergence is unreachable; in db mode the reconciler only runs via explicit `import`. -Related clock fix, same subsystem: `needsRefresh` is wall-clock arithmetic -(`now - last >= interval`, manager.zig:1201-1202) and the Pi has no RTC. A -fetch stamped while the clock was ahead (pre-NTP boot, restored image) -suspends refresh until real time catches the future timestamp — and this -design's idempotence would faithfully preserve the poison forever, having -deleted the wipe that used to be the accidental reset lever. The engine's -sibling fix: `needsRefresh` treats `last_updated > now` as refresh-due. One -comparison, with a test. +Related clock fix, same subsystem: `needsRefresh` is wall-clock arithmetic (`now - last >= interval`, manager.zig:1201-1202) and the Pi has no RTC. A fetch stamped while the clock was ahead (pre-NTP boot, restored image) suspends refresh until real time catches the future timestamp — and this design's idempotence would faithfully preserve the poison forever, having deleted the wipe that used to be the accidental reset lever. The engine's sibling fix: `needsRefresh` treats `last_updated > now` as refresh-due. One comparison, with a test. -Client promotion in place makes the `saved_clients` lift/merge/restore -scaffolding (import.zig:268-361, including `merge_observed_timestamps_sql`) -unnecessary: that machinery exists only because the wipe destroyed -`first_seen`/`last_seen` and had to smuggle them across. With no wipe, the -semantics it encodes — observed history survives declaration — are a plain -UPDATE that never touches the timestamp columns. The scaffolding dies with -the wipe (Deletions); its tests' semantics move to the promotion tests. +Client promotion in place makes the `saved_clients` lift/merge/restore scaffolding (import.zig:268-361, including `merge_observed_timestamps_sql`) unnecessary: that machinery exists only because the wipe destroyed `first_seen`/`last_seen` and had to smuggle them across. With no wipe, the semantics it encodes — observed history survives declaration — are a plain UPDATE that never touches the timestamp columns. The scaffolding dies with the wipe (Deletions); its tests' semantics move to the promotion tests. -**Ordering invariant**: reconcile commits before `manager.reload` -(app.zig:400) runs and before the scheduler's `sweepOrphans` can fire, so -preserved ids and checksums are visible to the filter layer before any -pruning. This holds by construction — reconcile completes inside `serve()` -before the manager exists — and is enforced *behaviorally*, not by a -statement-order unit test with nothing to grip: the restart-no-redownload -integration test (Tests, below) fails if anything between reconcile and -reload re-orders or wipes. No `serve()` restructuring is chartered for this. +**Ordering invariant**: reconcile commits before `manager.reload` (app.zig:400) runs and before the scheduler's `sweepOrphans` can fire, so preserved ids and checksums are visible to the filter layer before any pruning. This holds by construction — reconcile completes inside `serve()` before the manager exists — and is enforced *behaviorally*, not by a statement-order unit test with nothing to grip: the restart-no-redownload integration test (Tests, below) fails if anything between reconcile and reload re-orders or wipes. No `serve()` restructuring is chartered for this. -The engine returns a summary (ruling 8's input and the cross-session -contract): +The engine returns a summary (ruling 8's input and the cross-session contract): ```zig pub const TableCounts = struct { inserted: u32, updated: u32, deleted: u32 }; @@ -274,499 +97,138 @@ pub const Summary = struct { }; ``` -`updated` counts only rows actually written (writes-on-difference); -observed-client preservation counts nothing; promotion and reassignment count -as `updated` on `clients`. `deleted` on `clients` means exactly one thing: a -formerly declared (`hand_edited=1`) row absent from the file. An unchanged -file yields an all-zero summary. +`updated` counts only rows actually written (writes-on-difference); observed-client preservation counts nothing; promotion and reassignment count as `updated` on `clients`. `deleted` on `clients` means exactly one thing: a formerly declared (`hand_edited=1`) row absent from the file. An unchanged file yields an all-zero summary. -New repo verbs carry the engine (`sources_repo.upsertByUrl`, -per-table `deleteWhereNotIn`-style helpers), not new call ordering. `applyToDb` -and its wipe loop are deleted; import.zig keeps its parse/validate/diagnostics -plumbing. +New repo verbs carry the engine (`sources_repo.upsertByUrl`, per-table `deleteWhereNotIn`-style helpers), not new call ordering. `applyToDb` and its wipe loop are deleted; import.zig keeps its parse/validate/diagnostics plumbing. ### 4. Password: the file overrides when it speaks, and only then -`model.Web` changes to `password: ?[]const u8 = null` and -`password_hash: ?[]const u8 = null`. The settings bridge splits its skip -policy by direction — `isSkipped` becomes two functions, because encode and -decode need different sets: +`model.Web` changes to `password: ?[]const u8 = null` and `password_hash: ?[]const u8 = null`. The settings bridge splits its skip policy by direction — `isSkipped` becomes two functions, because encode and decode need different sets: -- `isEncodeSkipped` = {`web.password`, `web.password_hash`}: `toSettings` - stops emitting `web.password_hash` (the reconciler owns that settings row - directly) and continues to never emit `web.password`. -- `isDecodeSkipped` = {`web.password`} only: `fromSettings` **still loads** - `web.password_hash` from the settings table — the reconciler owning the - write does not mean the read path stops seeing it. Skipping it on decode - would leave `cfg.web.password_hash` null on every read path - (`config_export.readConfig` at export.zig:52, `mutations.loadConfig`), turn - `authEnabled` false, and silently disable auth in both modes. -- `decodeValue` gains an `.optional => try decodeValue(child, text)` arm - (model.zig:408-421 has none today; an unskipped optional field is currently - a `@compileError`). Absent key ⇒ default `null`; present ⇒ non-null. +- `isEncodeSkipped` = {`web.password`, `web.password_hash`}: `toSettings` stops emitting `web.password_hash` (the reconciler owns that settings row directly) and continues to never emit `web.password`. +- `isDecodeSkipped` = {`web.password`} only: `fromSettings` **still loads** `web.password_hash` from the settings table — the reconciler owning the write does not mean the read path stops seeing it. Skipping it on decode would leave `cfg.web.password_hash` null on every read path (`config_export.readConfig` at export.zig:52, `mutations.loadConfig`), turn `authEnabled` false, and silently disable auth in both modes. +- `decodeValue` gains an `.optional => try decodeValue(child, text)` arm (model.zig:408-421 has none today; an unskipped optional field is currently a `@compileError`). Absent key ⇒ default `null`; present ⇒ non-null. -`auth.authEnabled` becomes `(web.password_hash orelse "").len != 0` -(auth.zig:62-63); app.zig:475's `.live_hash = .init(...)` unwraps with -`orelse ""`. The cases: +`auth.authEnabled` becomes `(web.password_hash orelse "").len != 0` (auth.zig:62-63); app.zig:475's `.live_hash = .init(...)` unwraps with `orelse ""`. The cases: -- **Both set**: existing error (`PasswordAndHashBothSet`, import.zig:250, - validate.zig:395 — the check ports to `!= null` on both fields). -- **`password` present and empty**: rejected by `validate` with a new - diagnostic naming the remedy — `password_hash = ""` is how auth is disabled - declaratively. Without this rejection, `.password = ""` would hash the - empty string into a non-empty PHC (`authEnabled` true) while auth.zig:90 - refuses every empty-password login: auth on, unreachable. A declarative - fault, exit 2, so ruling 2's check/run agreement holds. -- **`password` set (non-empty plaintext)**: verify against the stored - `web.password_hash` with argon2; on match keep the stored hash, on mismatch - or absent stored hash, hash fresh. The justification is ruling 5 alone: - hashing unconditionally generates a fresh salt per apply and breaks - byte-stability. It is **not** a cost saving — argon2 verification recomputes - the full function (same t=2, m=19 MiB) with the stored salt, so - verify-and-keep costs exactly what hashing costs. Do not "optimize" the - verify away with any cached-plaintext scheme; that would be a security bug. -- **`password_hash` set**: written verbatim. An explicit `password_hash = ""` - is the declarative way to disable auth (auth.zig's documented empty-hash - state). -- **Neither set**: the stored hash is untouched. This is the deliberate - carve-out from file-as-sole-truth, because the naive alternative is a trap - the red team walked straight into: `toSettings` today always emits - `web.password_hash` with default `""` (model.zig:534, :93), so an operator - who hand-trims the ugly PHC string out of an exported file — meaning "keep - the current password" — would silently reconcile `""` over the stored hash - and open the admin UI to the LAN (`authEnabled` is `len != 0`, auth.zig:63). - Silence must mean "keep", and disabling auth must require the explicit - empty string. +- **Both set**: existing error (`PasswordAndHashBothSet`, import.zig:250, validate.zig:395 — the check ports to `!= null` on both fields). +- **`password` present and empty**: rejected by `validate` with a new diagnostic naming the remedy — `password_hash = ""` is how auth is disabled declaratively. Without this rejection, `.password = ""` would hash the empty string into a non-empty PHC (`authEnabled` true) while auth.zig:90 refuses every empty-password login: auth on, unreachable. A declarative fault, exit 2, so ruling 2's check/run agreement holds. +- **`password` set (non-empty plaintext)**: verify against the stored `web.password_hash` with argon2; on match keep the stored hash, on mismatch or absent stored hash, hash fresh. The justification is ruling 5 alone: hashing unconditionally generates a fresh salt per apply and breaks byte-stability. It is **not** a cost saving — argon2 verification recomputes the full function (same t=2, m=19 MiB) with the stored salt, so verify-and-keep costs exactly what hashing costs. Do not "optimize" the verify away with any cached-plaintext scheme; that would be a security bug. +- **`password_hash` set**: written verbatim. An explicit `password_hash = ""` is the declarative way to disable auth (auth.zig's documented empty-hash state). +- **Neither set**: the stored hash is untouched. This is the deliberate carve-out from file-as-sole-truth, because the naive alternative is a trap the red team walked straight into: `toSettings` today always emits `web.password_hash` with default `""` (model.zig:534, :93), so an operator who hand-trims the ugly PHC string out of an exported file — meaning "keep the current password" — would silently reconcile `""` over the stored hash and open the admin UI to the LAN (`authEnabled` is `len != 0`, auth.zig:63). Silence must mean "keep", and disabling auth must require the explicit empty string. -**Export's canonical form**: `password = null`, `password_hash = `. A non-null `password` is never emitted — the current -`cfg.web.password = "";` at export.zig:71 ported literally would make every -export carry a present-empty password beside a stored hash, tripping -`PasswordAndHashBothSet` on re-import: export's own output failing its own -rule, breaking ruling 9's adoption walkthrough and ruling 5's round trip. The -comment at export.zig:67-70 and the header sample change with it. With -`password = null`, the empty-plaintext rejection above does not fire and the -"hash written verbatim" branch keeps idempotence. +**Export's canonical form**: `password = null`, `password_hash = `. A non-null `password` is never emitted — the current `cfg.web.password = "";` at export.zig:71 ported literally would make every export carry a present-empty password beside a stored hash, tripping `PasswordAndHashBothSet` on re-import: export's own output failing its own rule, breaking ruling 9's adoption walkthrough and ruling 5's round trip. The comment at export.zig:67-70 and the header sample change with it. With `password = null`, the empty-plaintext rejection above does not fire and the "hash written verbatim" branch keeps idempotence. -Any auth transition (enabled/disabled/rotated) is reported in the summary and -printed by ruling 8 — an auth change is never a silent line item in a count. +Any auth transition (enabled/disabled/rotated) is reported in the summary and printed by ruling 8 — an auth change is never a silent line item in a count. ### 5. Idempotence is the invariant -Reconciling the same file twice produces a byte-identical database — ids, -checksums, timestamps, `created_at`, password hash, the whole settings -table — **and** the second pass performs zero writes (all-zero summary, -`total_changes` unmoved). The unit test asserts -byte-stability via the `dump()` helper (import.zig:469), which is rewritten to -iterate an explicit all-tables list (`config_schema.table_names`, all ten -content-bearing tables) because its current driver, `content_tables`, is -deleted (ruling 6). Anything that churns under an unchanged file — including a -*non-canonical but equivalent* file — is a bug in the engine, by definition. -(This invariant is what forces ruling 3's rules-multiset, canonical matching, -and writes-on-difference, and ruling 4's password handling — the places a -naive design silently violates it. It is also half of why ruling 8 persists -no authority state at all.) +Reconciling the same file twice produces a byte-identical database — ids, checksums, timestamps, `created_at`, password hash, the whole settings table — **and** the second pass performs zero writes (all-zero summary, `total_changes` unmoved). The unit test asserts byte-stability via the `dump()` helper (import.zig:469), which is rewritten to iterate an explicit all-tables list (`config_schema.table_names`, all ten content-bearing tables) because its current driver, `content_tables`, is deleted (ruling 6). Anything that churns under an unchanged file — including a *non-canonical but equivalent* file — is a bug in the engine, by definition. (This invariant is what forces ruling 3's rules-multiset, canonical matching, and writes-on-difference, and ruling 4's password handling — the places a naive design silently violates it. It is also half of why ruling 8 persists no authority state at all.) ### 6. `import` becomes a thin wrapper over reconcile; the guard becomes diff-gated -`nxdns import` is db mode's one-shot apply and the restore tool, reimplemented -on the reconcile engine. Reconcile is non-destructive of *runtime* state, but -deletion of declarative rows absent from the file is still a first-class -outcome (ruling 3) — a mistaken `nxdns import ./wrong.zon` against a -configured DB would still remove every group, rule, upstream, and source not -in that file. So the guard is **retargeted, not deleted**: emptiness-gating -(`isEmpty`) becomes diff-gating. +`nxdns import` is db mode's one-shot apply and the restore tool, reimplemented on the reconcile engine. Reconcile is non-destructive of *runtime* state, but deletion of declarative rows absent from the file is still a first-class outcome (ruling 3) — a mistaken `nxdns import ./wrong.zon` against a configured DB would still remove every group, rule, upstream, and source not in that file. So the guard is **retargeted, not deleted**: emptiness-gating (`isEmpty`) becomes diff-gating. -- After the reconcile passes, still inside the same `BEGIN IMMEDIATE` — - preserving import.zig:199-203's deliberate check-inside-the-write-lock - property, no TOCTOU — if any table's `deleted != 0` and no override flag, - roll back and fail exit 2 with the per-table delete counts in the message. - Observed-client reassignment is not declarative data and never trips the - gate (and under ruling 3's promotion rule, observed rows are never deleted - by reconcile at all). -- The flag is `--allow-delete` (`Options.allow_delete`), the renamed - `--force`; the error is `error.DestructiveImport`, the renamed - `DatabaseNotEmpty`, exit-2-mapped where cli.zig:533 maps today. This is - strictly better than the emptiness guard: additive and edit-only re-imports - stop needing a flag at all, and the flag now names what it permits. - "Edit-only" means edits to declarative columns on a matched identity; an - edit that *changes an identity column* — a group name, a source or upstream - url, a prefix, a zone, a rule tuple, a local-record identity — is a delete - plus an insert to the engine (ruling 3) and needs the flag. cli.md states - the distinction. -- `import.isEmpty` and `content_tables` still die — the diff-gate needs no - table list. +- After the reconcile passes, still inside the same `BEGIN IMMEDIATE` — preserving import.zig:199-203's deliberate check-inside-the-write-lock property, no TOCTOU — if any table's `deleted != 0` and no override flag, roll back and fail exit 2 with the per-table delete counts in the message. Observed-client reassignment is not declarative data and never trips the gate (and under ruling 3's promotion rule, observed rows are never deleted by reconcile at all). +- The flag is `--allow-delete` (`Options.allow_delete`), the renamed `--force`; the error is `error.DestructiveImport`, the renamed `DatabaseNotEmpty`, exit-2-mapped where cli.zig:533 maps today. This is strictly better than the emptiness guard: additive and edit-only re-imports stop needing a flag at all, and the flag now names what it permits. "Edit-only" means edits to declarative columns on a matched identity; an edit that *changes an identity column* — a group name, a source or upstream url, a prefix, a zone, a rule tuple, a local-record identity — is a delete plus an insert to the engine (ruling 3) and needs the flag. cli.md states the distinction. +- `import.isEmpty` and `content_tables` still die — the diff-gate needs no table list. -`import` does not detect a file-managed DB, because nothing records one: -authority lives in the invocation (ruling 1) and the DB carries no marker -(ruling 8). On a box whose unit runs file mode, `import` behaves like any -other import — the diff-gate guards deletion, and the next boot's reconcile -converges the DB back to the file, its summary reporting what it corrected. -The docs own this story plainly: `import` is a **stop-first operation**, on a -file-mode box doubly so — against a running instance its effect is partial -(the runtime divergence below) and lasts only until the next restart. The -file-authority contract is stated with the same precision everywhere: the -file is the sole declarative source, **converged at every boot** — not a -lock on the database between boots. A detect-and-warn variant was -designed and deleted (ruling 8) — a per-invocation warning cannot *prevent* -db-side writes anyway, the CLI user is root on their own box, and the -asymmetry with the web layer's 403 is intentional: the web UI has -anonymous-ish LAN users, the CLI has the operator. +`import` does not detect a file-managed DB, because nothing records one: authority lives in the invocation (ruling 1) and the DB carries no marker (ruling 8). On a box whose unit runs file mode, `import` behaves like any other import — the diff-gate guards deletion, and the next boot's reconcile converges the DB back to the file, its summary reporting what it corrected. The docs own this story plainly: `import` is a **stop-first operation**, on a file-mode box doubly so — against a running instance its effect is partial (the runtime divergence below) and lasts only until the next restart. The file-authority contract is stated with the same precision everywhere: the file is the sole declarative source, **converged at every boot** — not a lock on the database between boots. A detect-and-warn variant was designed and deleted (ruling 8) — a per-invocation warning cannot *prevent* db-side writes anyway, the CLI user is root on their own box, and the asymmetry with the web layer's 403 is intentional: the web UI has anonymous-ish LAN users, the CLI has the operator. -What a mid-run import against a *running* file-mode instance actually does — -documented, because the divergence is partial, not merely deferred: -`Manager.reload` re-reads `blocklist_sources`, `groups`, `group_sources`, -`rules`, `clients`, `client_prefixes` from the DB at runtime -(manager.zig:405-489) and the scheduler runs `sweepOrphans` before every pass -(manager.zig:1095, :1115), so filtering follows the imported rows on the next -reload and can unlink the compiled `.list`/`.wild` of sources the -import deleted — while upstreams, listeners, and settings stay at boot -values, and the web UI shows the imported state under the file-authority -banner. The next restart's reconcile re-inserts deleted sources from the file -with new ids: a full re-download, the exact cost this design exists to -prevent. That is why the docs name the restart requirement and the -re-download cost. The settings envelope's `reconciled_at` cannot see a CLI -import — ruling 7's weakened claim covers exactly this. +What a mid-run import against a *running* file-mode instance actually does — documented, because the divergence is partial, not merely deferred: `Manager.reload` re-reads `blocklist_sources`, `groups`, `group_sources`, `rules`, `clients`, `client_prefixes` from the DB at runtime (manager.zig:405-489) and the scheduler runs `sweepOrphans` before every pass (manager.zig:1095, :1115), so filtering follows the imported rows on the next reload and can unlink the compiled `.list`/`.wild` of sources the import deleted — while upstreams, listeners, and settings stay at boot values, and the web UI shows the imported state under the file-authority banner. The next restart's reconcile re-inserts deleted sources from the file with new ids: a full re-download, the exact cost this design exists to prevent. That is why the docs name the restart requirement and the re-download cost. The settings envelope's `reconciled_at` cannot see a CLI import — ruling 7's weakened claim covers exactly this. -The startup-vs-import race needs no code: both paths take `BEGIN IMMEDIATE` -under the 5 s busy timeout (db.zig:252), so the outcome is ordering, not -corruption — an import racing a restart may be reverted by the reconcile that -wins the lock second. Documented, not engineered around. +The startup-vs-import race needs no code: both paths take `BEGIN IMMEDIATE` under the 5 s busy timeout (db.zig:252), so the outcome is ordering, not corruption — an import racing a restart may be reverted by the reconcile that wins the lock second. Documented, not engineered around. -First-run story in db mode, after bootstrap dies: a fresh empty DB fails -validation naturally (`NoUsableUpstreams`, exit 2). The remediation hint — -one fixed line naming `nxdns import` and `run --config` — is owned by -**cli.zig's db-source fault renderer** (one arm, beside the exit-code mapping; -Zig errors carry no text and validate.zig must stay mode-blind), and the same -line serves bare `check` with no DB (ruling 2). One location, R2's charter. +First-run story in db mode, after bootstrap dies: a fresh empty DB fails validation naturally (`NoUsableUpstreams`, exit 2). The remediation hint — one fixed line naming `nxdns import` and `run --config` — is owned by **cli.zig's db-source fault renderer** (one arm, beside the exit-code mapping; Zig errors carry no text and validate.zig must stay mode-blind), and the same line serves bare `check` with no DB (ruling 2). One location, R2's charter. -`export` is unchanged in role: the diagnostic/capture tool in both modes, and -the file-mode adoption tool (its canonical password form changes per -ruling 4). +`export` is unchanged in role: the diagnostic/capture tool in both modes, and the file-mode adoption tool (its canonical password form changes per ruling 4). ### 7. Web layer: policy as data, rejected after auth, plain 403 -`WebState` gains `authority: union(enum) { database, managed_file: []const u8 }` -and `reconciled_at: ?i64`, built where `WebState` is assembled (app.zig:468). -The `managed_file` path slice is owned by `serve`'s arena, which outlives -`WebState` — R2 provides it, R3 consumes it, neither copies. `reconciled_at` -is stamped by `serve` immediately after the reconcile commits — same process, -same frame, a local — and is `null` in db mode, which never reconciles. +`WebState` gains `authority: union(enum) { database, managed_file: []const u8 }` and `reconciled_at: ?i64`, built where `WebState` is assembled (app.zig:468). The `managed_file` path slice is owned by `serve`'s arena, which outlives `WebState` — R2 provides it, R3 consumes it, neither copies. `reconciled_at` is stamped by `serve` immediately after the reconcile commits — same process, same frame, a local — and is `null` in db mode, which never reconciles. -`RouteInfo` gains `policy: enum { read, config_write, runtime_action }` beside -`auth` and `rate_limit` — policy-as-data, matching the table's existing style. -No default value: all 56 route entries state their class explicitly, and -router.zig's `test_table` (:198) gains the field too. Classification is -per-route, not per-prefix: `POST /api/blocklists/update` is a -`runtime_action`; its CRUD siblings are `config_write`. +`RouteInfo` gains `policy: enum { read, config_write, runtime_action }` beside `auth` and `rate_limit` — policy-as-data, matching the table's existing style. No default value: all 56 route entries state their class explicitly, and router.zig's `test_table` (:198) gains the field too. Classification is per-route, not per-prefix: `POST /api/blocklists/update` is a `runtime_action`; its CRUD siblings are `config_write`. -- Config writes, rejected in file mode: all group / blocklist / rule / - local-record / forward-zone / upstream / client-prefix mutations, - `PUT /api/settings` (password changes go through the file; its - `live_hash.installAndRevoke` side effect never fires in file mode), and - client PUT — naming or regrouping an observed client is declarative drift - (open question 1). -- Client DELETE is a **runtime action**: deleting an observed - (`hand_edited=0`) row discards runtime state the file never declared — - without this, a mis-identified or departed device's row is immortal in file - mode, since the file can only promote IPs, never remove them. Deleting a - *declared* client contradicts the file: the handler answers the same 403 - envelope. This is the one policy decision that needs a row read; it lives - in the client handler, not the router. (After ruling 3's promotion, a - declared IP's row *is* declared — DELETE answers 403, the intended - reading.) -- Runtime actions, always live: pause, blocklist refresh, cert reload, - login/logout. +- Config writes, rejected in file mode: all group / blocklist / rule / local-record / forward-zone / upstream / client-prefix mutations, `PUT /api/settings` (password changes go through the file; its `live_hash.installAndRevoke` side effect never fires in file mode), and client PUT — naming or regrouping an observed client is declarative drift (open question 1). +- Client DELETE is a **runtime action**: deleting an observed (`hand_edited=0`) row discards runtime state the file never declared — without this, a mis-identified or departed device's row is immortal in file mode, since the file can only promote IPs, never remove them. Deleting a *declared* client contradicts the file: the handler answers the same 403 envelope. This is the one policy decision that needs a row read; it lives in the client handler, not the router. (After ruling 3's promotion, a declared IP's row *is* declared — DELETE answers 403, the intended reading.) +- Runtime actions, always live: pause, blocklist refresh, cert reload, login/logout. -Enforcement lives in `router.dispatch` **after** `check_auth`, before the -handler — match → rate limit → auth → policy. Pre-auth rejection would leak -route existence; the codebase answers 401 first and this design keeps that. +Enforcement lives in `router.dispatch` **after** `check_auth`, before the handler — match → rate limit → auth → policy. Pre-auth rejection would leak route existence; the codebase answers 401 first and this design keeps that. -Rejection is **403**, body the existing single-field envelope: -`{"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"}`. -Not 409 — that status already means constraint conflict four ways in -openapi.yaml — and **no `code` field**: 403 is unused today, so the status -alone is machine-readable; the UI learns authority declaratively from -`GET /api/settings`, not by probing errors; and the single-property `Error` -schema, golden contract samples, and `api.ts` stay untouched. An optional field -with no consumer is machinery, not a contract. +Rejection is **403**, body the existing single-field envelope: `{"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"}`. Not 409 — that status already means constraint conflict four ways in openapi.yaml — and **no `code` field**: 403 is unused today, so the status alone is machine-readable; the UI learns authority declaratively from `GET /api/settings`, not by probing errors; and the single-property `Error` schema, golden contract samples, and `api.ts` stay untouched. An optional field with no consumer is machinery, not a contract. -The envelope must survive its own path: `respondError` today builds into a -fixed 512-byte buffer and **silently downgrades to `text/plain`** on overflow -(http_util.zig:315-327) — a long managed-file path (nested bind mounts) would -demote the documented JSON envelope. Fix at the root: `respondError` gets the -arena treatment `respondJson` already uses (`Writer.Allocating` over -`request.arena`, :337-339) and the `respondPlain` silent-downgrade path is -deleted — which fixes every long error message, not just this one. -`src/web/http_util.zig` joins R3's ownership. +The envelope must survive its own path: `respondError` today builds into a fixed 512-byte buffer and **silently downgrades to `text/plain`** on overflow (http_util.zig:315-327) — a long managed-file path (nested bind mounts) would demote the documented JSON envelope. Fix at the root: `respondError` gets the arena treatment `respondJson` already uses (`Writer.Allocating` over `request.arena`, :337-339) and the `respondPlain` silent-downgrade path is deleted — which fixes every long error message, not just this one. `src/web/http_util.zig` joins R3's ownership. -Authority discovery: the `GET /api/settings` envelope gains -`authority: {mode, path, reconciled_at}` — an authenticated route, so the -filesystem path never leaks through open `/api/version`/`/api/health`. All -three come from `WebState` (live truth); `path` is present in file mode only -and `reconciled_at` is nullable — `null` in db mode. Its meaning is exactly -"**this process loaded the file at T**" — restart-pending detection: an mtime -newer than `reconciled_at` means the running process has not loaded the -current file. The comparison is one-directional and non-authoritative — a -stepped clock (pre-NTP boot stamping the future, ruling 3's clock fix -territory) or a preserved mtime (`git checkout`, `rsync -a`) can make a newer -file look older, and the DB can move without either timestamp moving -(ruling 6's `import`, this ruling's `runtime_action` routes). It does not -answer "is the file what the server uses"; answering that would take content -hashing, which the anti-requirements refuse. The UI renders a read-only -banner (RestartBanner slot precedent) and disables mutation controls, with -the 403 as backstop. +Authority discovery: the `GET /api/settings` envelope gains `authority: {mode, path, reconciled_at}` — an authenticated route, so the filesystem path never leaks through open `/api/version`/`/api/health`. All three come from `WebState` (live truth); `path` is present in file mode only and `reconciled_at` is nullable — `null` in db mode. Its meaning is exactly "**this process loaded the file at T**" — restart-pending detection: an mtime newer than `reconciled_at` means the running process has not loaded the current file. The comparison is one-directional and non-authoritative — a stepped clock (pre-NTP boot stamping the future, ruling 3's clock fix territory) or a preserved mtime (`git checkout`, `rsync -a`) can make a newer file look older, and the DB can move without either timestamp moving (ruling 6's `import`, this ruling's `runtime_action` routes). It does not answer "is the file what the server uses"; answering that would take content hashing, which the anti-requirements refuse. The UI renders a read-only banner (RestartBanner slot precedent) and disables mutation controls, with the 403 as backstop. ### 8. The operator can see what happened -- `logStartup` (after `logging.install`) logs the authority: - `authority: database` / `authority: file (/etc/nxdns/config.zon)`. -- In file mode, the reconcile summary (ruling 3's `Summary`) prints through - the Runner at startup, matching `seedFromFile`'s existing output discipline: - per-table inserted/updated/deleted counts, the changed settings **keys** - (never values), and the auth transition when there is one. It is the answer - to "what did that restart change" without opening sqlite. -- **Authority is never persisted.** The DB carries no record of which mode - wrote it: authority lives in the invocation (ruling 1), per-process state - (`WebState.authority`, `reconciled_at`) serves the API (ruling 7), and the - journal holds the history. A persisted marker was designed twice and - deleted twice. A per-boot `reconciled_at` settings row breaks ruling 5 - outright — `dump()` iterates the settings table, so any row rewritten per - reconcile forfeits byte-identity, and `putSetting` moves `total_changes`, - forfeiting zero-writes. The write-on-difference `authority.mode`/`path` - variant survived idempotence but cost a reserved key namespace, a - `fromSettings` decode filter, and a reconciler sweep exemption — three - mechanisms whose only consumer was one `import` warning (ruling 6). State - that exists to power a courtesy message is not worth a namespace. The - settings sweep's exemption list is therefore exactly one key: - `web.password_hash` (ruling 4). +- `logStartup` (after `logging.install`) logs the authority: `authority: database` / `authority: file (/etc/nxdns/config.zon)`. +- In file mode, the reconcile summary (ruling 3's `Summary`) prints through the Runner at startup, matching `seedFromFile`'s existing output discipline: per-table inserted/updated/deleted counts, the changed settings **keys** (never values), and the auth transition when there is one. It is the answer to "what did that restart change" without opening sqlite. +- **Authority is never persisted.** The DB carries no record of which mode wrote it: authority lives in the invocation (ruling 1), per-process state (`WebState.authority`, `reconciled_at`) serves the API (ruling 7), and the journal holds the history. A persisted marker was designed twice and deleted twice. A per-boot `reconciled_at` settings row breaks ruling 5 outright — `dump()` iterates the settings table, so any row rewritten per reconcile forfeits byte-identity, and `putSetting` moves `total_changes`, forfeiting zero-writes. The write-on-difference `authority.mode`/`path` variant survived idempotence but cost a reserved key namespace, a `fromSettings` decode filter, and a reconciler sweep exemption — three mechanisms whose only consumer was one `import` warning (ruling 6). State that exists to power a courtesy message is not worth a namespace. The settings sweep's exemption list is therefore exactly one key: `web.password_hash` (ruling 4). -No `std.log.err` in any new code, per the standing spec rule for new code -(the pre-existing calls in db.zig/tls_server.zig are out of scope). +No `std.log.err` in any new code, per the standing spec rule for new code (the pre-existing calls in db.zig/tls_server.zig are out of scope). ### 9. Deployment and migration -- **systemd**: the shipped unit stays flagless (db default) and gains two - lines. `RestartPreventExitStatus=2 64` — a config fault or usage error must - not restart-loop every 2 s until StartLimitBurst; the journal holds the - diagnostics and the fix is a file edit, not a retry. (Correct in db mode - too: exit 2 means the config is wrong in either mode. And it is why - ruling 2 keeps exit 2 to *declarative* faults only — a transient box fault - mapped to exit 2 would stop the unit permanently.) And - `ReadOnlyPaths=/etc/nxdns` — `ConfigurationDirectory=nxdns` makes systemd - create the directory owned by the service user, so without this line the - source-of-truth file is writable by the very process whose mutation routes - file mode exists to disable; nxdns never writes `/etc/nxdns` in either mode, - so the base unit ships the enforcement rather than recommending it. Docs - show a drop-in for file mode: `ExecStart=` reset plus - `--config=/etc/nxdns/config.zon`. -- **Ansible / config-management**: the docs' deployment guidance mandates - `nxdns check --config=` as the handler precondition — validate - the pushed file *before* restarting, so a typo is a failed deploy at noon, - not a dead resolver at the next 3am power blip. -- **docker**: compose ships file mode as its example — - `command: ["run", "--config=/etc/nxdns/config.zon"]` (resolving old open - question 4; the `:ro` mount at compose.yaml:15 already suggests it). This keeps the - fresh-install and volume-loss stories working after bootstrap dies: a - recreated `nxdns-data` volume reconciles from the mounted file on next - start, which is *better* than the old seed-once self-heal. The binary's - default stays db. The seed-once comment (compose.yaml:9-12) is rewritten. - For db-mode-in-docker, the docs show the recovery one-liner - (`docker compose run --rm nxdns import /etc/nxdns/config.zon` — the file is - positional, `ImportArgs.file` at cli.zig:61, plus `--allow-delete` when the - diff deletes) — without it, `restart: unless-stopped` plus exit 2 is an - infinite crash loop (docker has no start limit) with no documented way out. -- **Adopt file mode** on a UI-configured box: **stop the service first**, then - `nxdns export --out /etc/nxdns/config.zon` → - `nxdns check --config=/etc/nxdns/config.zon` → add the flag → start. - Stop-first is load-bearing twice: any UI edit landing between a live - export and the restart would be silently reverted by the first reconcile, - and `check` refuses to grade a live database (immutable open, `WalPending` - against the steady-state WAL, db.zig:222-226) so the pre-flight gate only - works stopped. (Sync note, found in R4: the spec originally claimed - `export` itself refuses against a live instance — it does not; `export` - opens read/write and succeeds. The claim was corrected to name `check`.) - Stopped, - the first reconcile's summary is all-zero and writes nothing — blocklist - state, compiled files, and client history all survive. Every unchanged boot - after it writes nothing either. -- **Leave file mode**: drop the flag (remove the drop-in), restart. The DB - already holds the last reconciled state; nothing else needed. -- **Binary downgrade from file mode**: no unit edit is needed — the old - binary accepts `run --config` with seed-once semantics, and against the - already-configured DB it ignores the file and serves the last-reconciled - state. The rollback note states the consequence: file edits stop applying - until the binary is upgraded again. -- **Restore from backup**: db mode is stop → restore `config.db` → start, - with the WAL caveat db.zig itself documents: take backups only from a - stopped instance (or use `export`), and on restore delete any stale - `config.db-wal`/`config.db-shm` beside the target — a mismatched WAL is - silently discarded by SQLite, which turns "restore" into "lose the tail". - Restoring an *exported file* into a populated DB via `import` is exactly - the deleting case: back-up-and-restore.md documents `--allow-delete` there - (ruling 6) — the flag choreography is rewritten, not removed. In file mode - `config.db` is not the config backup — the file is; restore = redeploy the - file. The query-log DB restores independently in both modes. +- **systemd**: the shipped unit stays flagless (db default) and gains two lines. `RestartPreventExitStatus=2 64` — a config fault or usage error must not restart-loop every 2 s until StartLimitBurst; the journal holds the diagnostics and the fix is a file edit, not a retry. (Correct in db mode too: exit 2 means the config is wrong in either mode. And it is why ruling 2 keeps exit 2 to *declarative* faults only — a transient box fault mapped to exit 2 would stop the unit permanently.) And `ReadOnlyPaths=/etc/nxdns` — `ConfigurationDirectory=nxdns` makes systemd create the directory owned by the service user, so without this line the source-of-truth file is writable by the very process whose mutation routes file mode exists to disable; nxdns never writes `/etc/nxdns` in either mode, so the base unit ships the enforcement rather than recommending it. Docs show a drop-in for file mode: `ExecStart=` reset plus `--config=/etc/nxdns/config.zon`. +- **Ansible / config-management**: the docs' deployment guidance mandates `nxdns check --config=` as the handler precondition — validate the pushed file *before* restarting, so a typo is a failed deploy at noon, not a dead resolver at the next 3am power blip. +- **docker**: compose ships file mode as its example — `command: ["run", "--config=/etc/nxdns/config.zon"]` (resolving old open question 4; the `:ro` mount at compose.yaml:15 already suggests it). This keeps the fresh-install and volume-loss stories working after bootstrap dies: a recreated `nxdns-data` volume reconciles from the mounted file on next start, which is *better* than the old seed-once self-heal. The binary's default stays db. The seed-once comment (compose.yaml:9-12) is rewritten. For db-mode-in-docker, the docs show the recovery one-liner (`docker compose run --rm nxdns import /etc/nxdns/config.zon` — the file is positional, `ImportArgs.file` at cli.zig:61, plus `--allow-delete` when the diff deletes) — without it, `restart: unless-stopped` plus exit 2 is an infinite crash loop (docker has no start limit) with no documented way out. +- **Adopt file mode** on a UI-configured box: **stop the service first**, then `nxdns export --out /etc/nxdns/config.zon` → `nxdns check --config=/etc/nxdns/config.zon` → add the flag → start. Stop-first is load-bearing twice: any UI edit landing between a live export and the restart would be silently reverted by the first reconcile, and `check` refuses to grade a live database (immutable open, `WalPending` against the steady-state WAL, db.zig:222-226) so the pre-flight gate only works stopped. (Sync note, found in R4: the spec originally claimed `export` itself refuses against a live instance — it does not; `export` opens read/write and succeeds. The claim was corrected to name `check`.) Stopped, the first reconcile's summary is all-zero and writes nothing — blocklist state, compiled files, and client history all survive. Every unchanged boot after it writes nothing either. +- **Leave file mode**: drop the flag (remove the drop-in), restart. The DB already holds the last reconciled state; nothing else needed. +- **Binary downgrade from file mode**: no unit edit is needed — the old binary accepts `run --config` with seed-once semantics, and against the already-configured DB it ignores the file and serves the last-reconciled state. The rollback note states the consequence: file edits stop applying until the binary is upgraded again. +- **Restore from backup**: db mode is stop → restore `config.db` → start, with the WAL caveat db.zig itself documents: take backups only from a stopped instance (or use `export`), and on restore delete any stale `config.db-wal`/`config.db-shm` beside the target — a mismatched WAL is silently discarded by SQLite, which turns "restore" into "lose the tail". Restoring an *exported file* into a populated DB via `import` is exactly the deleting case: back-up-and-restore.md documents `--allow-delete` there (ruling 6) — the flag choreography is rewritten, not removed. In file mode `config.db` is not the config backup — the file is; restore = redeploy the file. The query-log DB restores independently in both modes. -Migration honesty, replacing the earlier blanket claim: a db-mode install -that never passed `--config` needs nothing. Anyone whose unit, wrapper, or -compose `command` carries `run --config` (the documented seed-once invocation -in first-run.md and three how-to labs) gets file authority at the first -post-upgrade start — the seed file becomes the config, and UI edits made -since seeding are deleted by the first reconcile. The upgrade doc's -breaking-changes section leads with this and gives the two exits: drop the -flag to keep the DB, or re-export to the file path first to adopt file mode -cleanly (the walkthrough above). `check --config` keeps its meaning. Docker -db-mode fresh installs must use the new compose or run `import` once. No -schema migration. +Migration honesty, replacing the earlier blanket claim: a db-mode install that never passed `--config` needs nothing. Anyone whose unit, wrapper, or compose `command` carries `run --config` (the documented seed-once invocation in first-run.md and three how-to labs) gets file authority at the first post-upgrade start — the seed file becomes the config, and UI edits made since seeding are deleted by the first reconcile. The upgrade doc's breaking-changes section leads with this and gives the two exits: drop the flag to keep the DB, or re-export to the file path first to adopt file mode cleanly (the walkthrough above). `check --config` keeps its meaning. Docker db-mode fresh installs must use the new compose or run `import` once. No schema migration. ### 10. Documentation is part of the change -`explanation/configuration-model.md` is rewritten around the two-mode model -(its "why the database wins" argument becomes mode-scoped, not superseded). -Seed-once claims are corrected in: the first-run tutorial, install-with-systemd, -install-with-docker, upgrade (breaking-changes + rollback sections, -ruling 9), back-up-and-restore (new restore semantics, WAL sidecars, -`--allow-delete`), set-up-admin-authentication (the ruling 4 -absence/empty-string rule), troubleshoot, cli.md -(`run`/`check`/`import`/`export`, `--allow-delete`), configuration.md -(optional password fields), files-and-directories.md, api.md (settings -envelope, 403, per-route rejectability), the compose.yaml comment, -README/INSTALL. +`explanation/configuration-model.md` is rewritten around the two-mode model (its "why the database wins" argument becomes mode-scoped, not superseded). Seed-once claims are corrected in: the first-run tutorial, install-with-systemd, install-with-docker, upgrade (breaking-changes + rollback sections, ruling 9), back-up-and-restore (new restore semantics, WAL sidecars, `--allow-delete`), set-up-admin-authentication (the ruling 4 absence/empty-string rule), troubleshoot, cli.md (`run`/`check`/`import`/`export`, `--allow-delete`), configuration.md (optional password fields), files-and-directories.md, api.md (settings envelope, 403, per-route rejectability), the compose.yaml comment, README/INSTALL. ## Deletions (complete list) `config/bootstrap.zig` (+2 tests, S7 cases 15-18, + app.zig import/call sites :35/:142/:146 and tests :1124/:1237) · `import.isEmpty` (+6 tests, + the -storage_integration_test.zig call sites :763/:795/:894, + the clients_repo.zig -doc comments :6/:129) · `config_schema.content_tables` (+its test; `dump()` -re-pointed to the new `table_names` list) · `app.seedFromFile` (+its app.zig -tests) · `CheckArgs.config_explicit` · the `checkImpl` source heuristic · -`applyToDb`'s wipe loop, including the `saved_clients` lift/merge/restore -scaffolding (import.zig:268-361 — superseded by ruling 3's -promotion-in-place) · `toSettings`'s `web.password_hash` emission (+its -key-list test row; the model.zig:665-670 "skipped in both directions" test is -rewritten for the encode/decode split) · `http_util.respondPlain`'s -silent-downgrade path (`respondError` rebuilt on the request arena). +storage_integration_test.zig call sites :763/:795/:894, + the clients_repo.zig doc comments :6/:129) · `config_schema.content_tables` (+its test; `dump()` re-pointed to the new `table_names` list) · `app.seedFromFile` (+its app.zig tests) · `CheckArgs.config_explicit` · the `checkImpl` source heuristic · `applyToDb`'s wipe loop, including the `saved_clients` lift/merge/restore scaffolding (import.zig:268-361 — superseded by ruling 3's promotion-in-place) · `toSettings`'s `web.password_hash` emission (+its key-list test row; the model.zig:665-670 "skipped in both directions" test is rewritten for the encode/decode split) · `http_util.respondPlain`'s silent-downgrade path (`respondError` rebuilt on the request arena). -Renamed, not deleted (ruling 6): `Options.force` → `Options.allow_delete` -(`--force` → `--allow-delete`), `error.DatabaseNotEmpty` → -`error.DestructiveImport` (cli.zig:533 arm retargeted, faults.zig exclusion -and tests follow the name). +Renamed, not deleted (ruling 6): `Options.force` → `Options.allow_delete` (`--force` → `--allow-delete`), `error.DatabaseNotEmpty` → `error.DestructiveImport` (cli.zig:533 arm retargeted, faults.zig exclusion and tests follow the name). ## Sessions -R1 lands first and R2 rewires onto it and carries every deletion — they are -ordered, not parallel (the old plan had R1 deleting `bootstrap.zig` out from -under R2-owned app.zig call sites). R1 is **additive engine plus one -coordinated model change**, not purely additive: the `model.Web` optional -fields force same-session edits at every site that reads them, or the tree -stops building. R3 and R4 run parallel to both; their interfaces -(`reconcile.Summary`, `WebState.authority` + `reconciled_at`, -`RouteInfo.policy`) are fixed here. +R1 lands first and R2 rewires onto it and carries every deletion — they are ordered, not parallel (the old plan had R1 deleting `bootstrap.zig` out from under R2-owned app.zig call sites). R1 is **additive engine plus one coordinated model change**, not purely additive: the `model.Web` optional fields force same-session edits at every site that reads them, or the tree stops building. R3 and R4 run parallel to both; their interfaces (`reconcile.Summary`, `WebState.authority` + `reconciled_at`, `RouteInfo.policy`) are fixed here. ### Session R1: reconcile engine + the coordinated model change -Owns `src/config/reconcile.zig` (engine + `Summary` as specified), the new -repo verbs in `src/storage/repositories/`, the `needsRefresh` clock clamp in -`src/filter/manager.zig`, registration in `src/tests.zig`. Rulings 3, 4, 5. +Owns `src/config/reconcile.zig` (engine + `Summary` as specified), the new repo verbs in `src/storage/repositories/`, the `needsRefresh` clock clamp in `src/filter/manager.zig`, registration in `src/tests.zig`. Rulings 3, 4, 5. -Owns `src/config/model.zig` whole: the optional `Web.password`/`password_hash` -fields, the `isEncodeSkipped`/`isDecodeSkipped` split, `decodeValue`'s -`.optional` arm, and the bridge tests. And the sites the -optional fields break, which no session previously owned: `validate.zig` -(:395 both-set port, the new empty-password rejection, test :1966-1968), -`auth.zig` (`authEnabled` orelse), `export.zig` (canonical `password = null`, -comment and header sample), `src/web/handlers/settings.zig` (`newPassword`'s -`orelse` chain at :184-186, and `FieldType` collapsing `?T` so `Partial`'s -`?FieldType(...)` at :114 does not generate a double optional — landing this -in R1 keeps R3's `web/` work unblocked). R1 also makes the one-line -`orelse ""` edit at app.zig:475 so the tree builds; app.zig otherwise stays -R2's, and R2 absorbs that line into its startup rewire. +Owns `src/config/model.zig` whole: the optional `Web.password`/`password_hash` fields, the `isEncodeSkipped`/`isDecodeSkipped` split, `decodeValue`'s `.optional` arm, and the bridge tests. And the sites the optional fields break, which no session previously owned: `validate.zig` (:395 both-set port, the new empty-password rejection, test :1966-1968), `auth.zig` (`authEnabled` orelse), `export.zig` (canonical `password = null`, comment and header sample), `src/web/handlers/settings.zig` (`newPassword`'s `orelse` chain at :184-186, and `FieldType` collapsing `?T` so `Partial`'s `?FieldType(...)` at :114 does not generate a double optional — landing this in R1 keeps R3's `web/` work unblocked). R1 also makes the one-line `orelse ""` edit at app.zig:475 so the tree builds; app.zig otherwise stays R2's, and R2 absorbs that line into its startup rewire. ### Session R2: CLI + startup + deletions -Owns `src/cli.zig`, `src/app.zig`, `src/config/faults.zig`, -`src/config/import.zig` (thin-wrapper rewrite, diff-gate, `--allow-delete`), -the shared loader-fault mapping helper (ruling 2, used by `run` and `check`), -and the entire Deletions list plus the renames. Rulings 1, 2, 6, 8. Threads authority and -`reconciled_at` into `WebState` (arena-owned path) but does not touch the -router. Owns the cli.zig hint-line renderer arm. +Owns `src/cli.zig`, `src/app.zig`, `src/config/faults.zig`, `src/config/import.zig` (thin-wrapper rewrite, diff-gate, `--allow-delete`), the shared loader-fault mapping helper (ruling 2, used by `run` and `check`), and the entire Deletions list plus the renames. Rulings 1, 2, 6, 8. Threads authority and `reconciled_at` into `WebState` (arena-owned path) but does not touch the router. Owns the cli.zig hint-line renderer arm. ### Session R3: web enforcement + UI -Owns `src/web/router.zig` (dispatch policy step, `test_table` gains the -policy column), `src/web/routes.zig` (all 56 entries state `policy` -explicitly — no default), `src/web/http_util.zig` (`respondError` arena -rewrite, `respondPlain` downgrade deletion), `src/web/openapi.yaml`, the -client-DELETE observed/declared branch in the clients handler, `web/` -frontend (banner, disabled controls, settings envelope with nullable -`reconciled_at`). Ruling 7. Consumes `WebState.authority`/`reconciled_at` as -fixed above. +Owns `src/web/router.zig` (dispatch policy step, `test_table` gains the policy column), `src/web/routes.zig` (all 56 entries state `policy` explicitly — no default), `src/web/http_util.zig` (`respondError` arena rewrite, `respondPlain` downgrade deletion), `src/web/openapi.yaml`, the client-DELETE observed/declared branch in the clients handler, `web/` frontend (banner, disabled controls, settings envelope with nullable `reconciled_at`). Ruling 7. Consumes `WebState.authority`/`reconciled_at` as fixed above. ### Session R4: deployment + docs -Owns `deploy/**`, `docs/**`, `README.md`, `INSTALL`, compose file + comment, -the systemd unit's two new lines. Rulings 9, 10. +Owns `deploy/**`, `docs/**`, `README.md`, `INSTALL`, compose file + comment, the systemd unit's two new lines. Rulings 9, 10. ### Orchestrator -This spec, R1→R2 sequencing, cross-session integration, the drift-guard -regenerations that span sessions (contract samples, openapi route counts, -api.md/cli.md rows), and the `-Dlive` acceptance run. +This spec, R1→R2 sequencing, cross-session integration, the drift-guard regenerations that span sessions (contract samples, openapi route counts, api.md/cli.md rows), and the `-Dlive` acceptance run. ## Tests -- **Reconcile unit tests** (R1, `:memory:` + migrate): idempotence via the - rewritten `dump()` byte-stability across all tables after applying the same - file twice — including a file carrying a plaintext password **and a - non-canonical but equivalent file** (`FD00::1`-style) — plus the zero-writes - assertion (all-zero `Summary`, `total_changes` unmoved) on the second pass; - per-table preservation via `sources_repo.SourceRow` after seeding stats with - `updateSourceStats`; URL change ⇒ new id; source removal; observed-client - survival, **promote-on-declare (asserting `first_seen`/`last_seen` survive - the promotion and the row id is stable)**, and **reassign-to-default when - their group is removed or renamed** (the un-cascaded FK case); - `safe_search` edit on an existing group converges; rules `created_at` - stability including duplicate tuples; password verify-keeps-hash, - mismatch-rehashes, absent-keeps-stored, explicit-empty-disables; rollback - on mid-tx failure; default-group id 1 pin; `needsRefresh` - future-`last_updated` clamp. -- **Model/bridge tests** (R1, model.zig test neighbourhood at :659): - `decodeValue` on an optional field (absent ⇒ - null, present ⇒ value); `web.password_hash` decodes from settings but is - not encoded (the encode/decode split); `validate` rejects present-and-empty - `password` with the `password_hash = ""` remedy in the diagnostic; export ⇒ - import round trip with the canonical `password = null` form. -- **Import gate tests** (R2): a file whose diff deletes rows, without - `--allow-delete` ⇒ rollback, exit 2, per-table delete counts in the - message; with the flag ⇒ applied; additive/edit-only import needs no flag. -- **Loader-fault mapping** (R2, on the shared helper directly): each - path-class open error ⇒ `ManagedConfigUnreadable`; a non-path member of the - set propagates unmapped. -- **Restart-no-redownload** in filter_integration_test.zig: reconcile, then a - `Manager` restart reuses `.list`/`.wild` with no refetch (fixtures - already assert reuse by id) — this is also the behavioral enforcement of - ruling 3's ordering invariant. Plus **one `-Dlive` run of the real download - path** — hermetic suites have hidden a process-killing bug on the real - network path before (MEMORY), so the acceptance includes the real path once. -- **Router**: policy classification unit tests via `test_table`/`matchPath`, - no sockets; the contract table in web_integration_test.zig gains a `policy` - column and the existing 1:1 coverage assertion widens so classification - cannot drift; one socketed test per class in file mode (config write → 403, - runtime action → 2xx, read → 200) plus observed-vs-declared client DELETE, - via `EnvOptions` authority; `respondError` with a message longer than the - old 512-byte buffer stays `application/json`. Route count stays 56 — no new - routes. -- **CLI**: `parseArgs` tests: `run` without `--config` selects db authority; - `run --config=` selects file authority and the path lands in the run - args; `--allow-delete` parsing; `usage_text` test; exit-2 for a missing - managed file and for bare `check` with no DB (hint line present) — via the - `Captured` runner (in-process; remember the buffered-writer caveat — flush - through a real `File.Writer` where the test asserts delivery). -- **Drift guards knowingly tripped and regenerated**: openapi.yaml (settings - envelope, 403 responses), golden contract samples (`-Dcontract-samples-out`), - api.md rows, cli.md sections, docs_drift_test, `toSettings` key-list test. +- **Reconcile unit tests** (R1, `:memory:` + migrate): idempotence via the rewritten `dump()` byte-stability across all tables after applying the same file twice — including a file carrying a plaintext password **and a non-canonical but equivalent file** (`FD00::1`-style) — plus the zero-writes assertion (all-zero `Summary`, `total_changes` unmoved) on the second pass; per-table preservation via `sources_repo.SourceRow` after seeding stats with `updateSourceStats`; URL change ⇒ new id; source removal; observed-client survival, **promote-on-declare (asserting `first_seen`/`last_seen` survive the promotion and the row id is stable)**, and **reassign-to-default when their group is removed or renamed** (the un-cascaded FK case); `safe_search` edit on an existing group converges; rules `created_at` stability including duplicate tuples; password verify-keeps-hash, mismatch-rehashes, absent-keeps-stored, explicit-empty-disables; rollback on mid-tx failure; default-group id 1 pin; `needsRefresh` future-`last_updated` clamp. +- **Model/bridge tests** (R1, model.zig test neighbourhood at :659): `decodeValue` on an optional field (absent ⇒ null, present ⇒ value); `web.password_hash` decodes from settings but is not encoded (the encode/decode split); `validate` rejects present-and-empty `password` with the `password_hash = ""` remedy in the diagnostic; export ⇒ import round trip with the canonical `password = null` form. +- **Import gate tests** (R2): a file whose diff deletes rows, without `--allow-delete` ⇒ rollback, exit 2, per-table delete counts in the message; with the flag ⇒ applied; additive/edit-only import needs no flag. +- **Loader-fault mapping** (R2, on the shared helper directly): each path-class open error ⇒ `ManagedConfigUnreadable`; a non-path member of the set propagates unmapped. +- **Restart-no-redownload** in filter_integration_test.zig: reconcile, then a `Manager` restart reuses `.list`/`.wild` with no refetch (fixtures already assert reuse by id) — this is also the behavioral enforcement of ruling 3's ordering invariant. Plus **one `-Dlive` run of the real download path** — hermetic suites have hidden a process-killing bug on the real network path before (MEMORY), so the acceptance includes the real path once. +- **Router**: policy classification unit tests via `test_table`/`matchPath`, no sockets; the contract table in web_integration_test.zig gains a `policy` column and the existing 1:1 coverage assertion widens so classification cannot drift; one socketed test per class in file mode (config write → 403, runtime action → 2xx, read → 200) plus observed-vs-declared client DELETE, via `EnvOptions` authority; `respondError` with a message longer than the old 512-byte buffer stays `application/json`. Route count stays 56 — no new routes. +- **CLI**: `parseArgs` tests: `run` without `--config` selects db authority; `run --config=` selects file authority and the path lands in the run args; `--allow-delete` parsing; `usage_text` test; exit-2 for a missing managed file and for bare `check` with no DB (hint line present) — via the `Captured` runner (in-process; remember the buffered-writer caveat — flush through a real `File.Writer` where the test asserts delivery). +- **Drift guards knowingly tripped and regenerated**: openapi.yaml (settings envelope, 403 responses), golden contract samples (`-Dcontract-samples-out`), api.md rows, cli.md sections, docs_drift_test, `toSettings` key-list test. ## Acceptance (design complete when implemented) @@ -854,68 +316,31 @@ api.md/cli.md rows), and the `-Dlive` acceptance run. ## Anti-requirements - No file watcher, no inotify, no SIGHUP reload — restart is the reload. -- No UI write-back to the file, no partial/merge authority, no per-table - hybrid modes, no multi-file config. -- No persisted authority state, no config hashing, no content-based change - detection — authority lives in the invocation; last-load time is - per-process state in the settings envelope, honest about what it can and - cannot answer (ruling 7). -- No deferred deletion of a replaced source's compiled artifacts — the URL-edit - gap is an accepted trade (ruling 3), bounded by the scheduler retry. +- No UI write-back to the file, no partial/merge authority, no per-table hybrid modes, no multi-file config. +- No persisted authority state, no config hashing, no content-based change detection — authority lives in the invocation; last-load time is per-process state in the settings envelope, honest about what it can and cannot answer (ruling 7). +- No deferred deletion of a replaced source's compiled artifacts — the URL-edit gap is an accepted trade (ruling 3), bounded by the scheduler retry. - No etag/conditional GET for blocklist fetches. -- No `code` field or richer error envelope — the status is the machine - contract. +- No `code` field or richer error envelope — the status is the machine contract. - No rule-identity schema change (no synthetic rule key column). -- No preservation of rule `created_at` across pattern edits — an edited rule - is a new rule. -- No CLI-side blocking or detection of `import` against a file-mode box's - DB — the diff-gate guards deletion; the next boot converges and its - summary reports what it corrected. +- No preservation of rule `created_at` across pattern edits — an edited rule is a new rule. +- No CLI-side blocking or detection of `import` against a file-mode box's DB — the diff-gate guards deletion; the next boot converges and its summary reports what it corrected. - No fallback from file mode to db mode under any failure. ## Resolved defaults (were open questions; the user may overrule before R3/R4) -1. **UI naming and regrouping of observed (`hand_edited=0`) clients in file - mode: rejected as declarative drift** (client PUT stays `config_write`, - ruling 7). The household user adds the client to the file instead. The - alternative — carving naming/grouping out as a runtime action — - reintroduces two-way merge for one table, which the anti-requirements - refuse. Affects R3. -2. **`export` output is byte-identical in both modes** — no file-mode - annotation. An annotation would make export → file → adopt produce a - different file than the one checked, for a label the operator already has - in the unit file. Affects R4's round-trip docs. -3. **File mode ships as a documented systemd drop-in**, not a second - commented `ExecStart` in the packaged unit. The packaged unit stays - flagless and correct by itself; a commented alternative line in a unit - file is a doc pretending to be config. Affects R4. +1. **UI naming and regrouping of observed (`hand_edited=0`) clients in file mode: rejected as declarative drift** (client PUT stays `config_write`, ruling 7). The household user adds the client to the file instead. The alternative — carving naming/grouping out as a runtime action — reintroduces two-way merge for one table, which the anti-requirements refuse. Affects R3. +2. **`export` output is byte-identical in both modes** — no file-mode annotation. An annotation would make export → file → adopt produce a different file than the one checked, for a label the operator already has in the unit file. Affects R4's round-trip docs. +3. **File mode ships as a documented systemd drop-in**, not a second commented `ExecStart` in the packaged unit. The packaged unit stays flagless and correct by itself; a commented alternative line in a unit file is a doc pretending to be config. Affects R4. ## Cross-validation findings rejected -Round 1: none — all twelve findings (F1-F12) were confirmed against the repo -and are folded into the rulings above. The F1 remedy has since been -superseded: the premise revision deleted the persisted marker entirely -(ruling 8) instead of keeping it write-on-difference. +Round 1: none — all twelve findings (F1-F12) were confirmed against the repo and are folded into the rulings above. The F1 remedy has since been superseded: the premise revision deleted the persisted marker entirely (ruling 8) instead of keeping it write-on-difference. -Round 2 (on the premise revision) returned three important findings and one -minor. Folded: the docker recovery one-liner used a `--config` flag `import` -does not have (fixed in ruling 9, file is positional); `import` is now -stated everywhere as a stop-first operation and the file-authority contract -as "converged at every boot" (ruling 6); identity-column edits are named as -delete-plus-insert needing `--allow-delete` (ruling 6). Declined: renaming -`run --config` to `--managed-config` — reasons recorded in ruling 1. +Round 2 (on the premise revision) returned three important findings and one minor. Folded: the docker recovery one-liner used a `--config` flag `import` does not have (fixed in ruling 9, file is positional); `import` is now stated everywhere as a stop-first operation and the file-authority contract as "converged at every boot" (ruling 6); identity-column edits are named as delete-plus-insert needing `--allow-delete` (ruling 6). Declined: renaming `run --config` to `--managed-config` — reasons recorded in ruling 1. ## Red-team findings rejected -None rejected outright — every finding checked out against the repo. Two -proposed remedies were declined while their findings were accepted: +None rejected outright — every finding checked out against the repo. Two proposed remedies were declined while their findings were accepted: -- **ops 10's mitigation** (keep the old compiled artifact until the successor - URL's first successful fetch): declined — a cross-artifact lifecycle for a - rare, operator-initiated event on a household box; the trade is now stated - in ruling 3 and the anti-requirements instead. -- **ops 14's db-mode warning** ("a config file exists but authority is - database"): declined — db mode is given no file path, so warning would mean - probing a well-known location, which is precisely the ambient inference - ruling 1 bans; the accepted half (drift visibility) is served by the - settings envelope's `authority` block and per-process `reconciled_at`. +- **ops 10's mitigation** (keep the old compiled artifact until the successor URL's first successful fetch): declined — a cross-artifact lifecycle for a rare, operator-initiated event on a household box; the trade is now stated in ruling 3 and the anti-requirements instead. +- **ops 14's db-mode warning** ("a config file exists but authority is database"): declined — db mode is given no file path, so warning would mean probing a well-known location, which is precisely the ambient inference ruling 1 bans; the accepted half (drift visibility) is served by the settings envelope's `authority` block and per-process `reconciled_at`. diff --git a/specs/milestone-21.md b/specs/milestone-21.md index feb7491..033b99f 100644 --- a/specs/milestone-21.md +++ b/specs/milestone-21.md @@ -1,129 +1,49 @@ # Milestone 21: ABP exceptions from lists, regex rules for operators -Goal: honor `@@||domain^` exception lines in downloaded blocklists as allow -entries scoped below operator rules and above list blocks (tier 1), and add a -`regex` rule kind for operator rules backed by a homegrown linear-time engine -(tier 2). Nothing else from the ABP syntax enters scope. +Goal: honor `@@||domain^` exception lines in downloaded blocklists as allow entries scoped below operator rules and above list blocks (tier 1), and add a `regex` rule kind for operator rules backed by a homegrown linear-time engine (tier 2). Nothing else from the ABP syntax enters scope. -Design written 2026-08-09 against HEAD `ffc3ca6`; every anchor re-verified -2026-08-11 against `a8e0fe4` after milestone 20 landed. The milestone amends -PLAN §2.2, which currently rules regex out permanently; the amendment is part -of session S3, not a side effect. +Design written 2026-08-09 against HEAD `ffc3ca6`; every anchor re-verified 2026-08-11 against `a8e0fe4` after milestone 20 landed. The milestone amends PLAN §2.2, which currently rules regex out permanently; the amendment is part of session S3, not a side effect. ## Implementation contract (read first) - Read `AGENTS.md`, then this spec whole, before session work starts. -- Pure core stays pure: `src/filter/` files that are fuzz-module roots import - only `std` (`src/filter/parsers.zig:1-14`). The new `src/filter/regex.zig` - obeys the same constraint. -- Every new `src/**.zig` file must be listed in `src/tests.zig` - (`build.zig:494-539` fatals otherwise). -- Frozen DDL is frozen: schema changes are new migration steps - (`src/storage/config_schema.zig:1-6`, `src/storage/migrations.zig:21-22`). -- After any API shape change, regenerate the contract samples - (`web/src/lib/contractSamples.gen.ts`; procedure in AGENTS.md). +- Pure core stays pure: `src/filter/` files that are fuzz-module roots import only `std` (`src/filter/parsers.zig:1-14`). The new `src/filter/regex.zig` obeys the same constraint. +- Every new `src/**.zig` file must be listed in `src/tests.zig` (`build.zig:494-539` fatals otherwise). +- Frozen DDL is frozen: schema changes are new migration steps (`src/storage/config_schema.zig:1-6`, `src/storage/migrations.zig:21-22`). +- After any API shape change, regenerate the contract samples (`web/src/lib/contractSamples.gen.ts`; procedure in AGENTS.md). ## Rulings (binding) ### 1. Exception lines are `@@||name^` and nothing else, plus one modifier -`src/filter/parser_abp.zig:22` currently maps every `@@` line to -`.unsupported`. After this milestone, a line is an exception when it is -`@@||name^` or `@@||name` (same trailing-`^` and `rule_tokens` treatment as the -block anchor at parser_abp.zig:26-34), optionally suffixed with the literal -`$important` — that suffix is the common form in AdGuard-authored lists and -changes nothing about the meaning here, because list exceptions already sit -below every operator rule. Any other `@@` form (`@@name` without the anchor, -any other `$` modifier, a path, a scheme) stays `.unsupported`. The -parser-header policy paragraph (parser_abp.zig:5-6) is rewritten to state the -new rule and its precedence justification: a list exception can cancel only -list blocks, never an operator decision, so no downloaded list can open an -allow hole the operator did not open. +`src/filter/parser_abp.zig:22` currently maps every `@@` line to `.unsupported`. After this milestone, a line is an exception when it is `@@||name^` or `@@||name` (same trailing-`^` and `rule_tokens` treatment as the block anchor at parser_abp.zig:26-34), optionally suffixed with the literal `$important` — that suffix is the common form in AdGuard-authored lists and changes nothing about the meaning here, because list exceptions already sit below every operator rule. Any other `@@` form (`@@name` without the anchor, any other `$` modifier, a path, a scheme) stays `.unsupported`. The parser-header policy paragraph (parser_abp.zig:5-6) is rewritten to state the new rule and its precedence justification: a list exception can cancel only list blocks, never an operator decision, so no downloaded list can open an allow hole the operator did not open. ### 2. Precedence: list exceptions sit between operator rules and list blocks -`matcher.Snapshot.evaluate` (`src/filter/matcher.zig:286-338`) gains one level -between the operator wildcard-block walk (level 4) and the blocklist domain -probe (level 5): for each attached source, an exception match — full name or -parent walk, apex covered — returns `blocked = false`, reason -`.blocklist_exception`, `matched` = the matching entry, `source` = the source -index. The doc comment at matcher.zig:269-285 and PLAN §3.10 (PLAN.md:108-117) -are both updated with the new level. Regex rules (ruling 6) slot in as levels -after the wildcard rules and before list exceptions, allow before block, so -the full order is: exact allow, exact block, wildcard allow, wildcard block, -regex allow, regex block, list exception, list domain, list wildcard. -"Tie-break at same specificity: allow wins" is preserved. +`matcher.Snapshot.evaluate` (`src/filter/matcher.zig:286-338`) gains one level between the operator wildcard-block walk (level 4) and the blocklist domain probe (level 5): for each attached source, an exception match — full name or parent walk, apex covered — returns `blocked = false`, reason `.blocklist_exception`, `matched` = the matching entry, `source` = the source index. The doc comment at matcher.zig:269-285 and PLAN §3.10 (PLAN.md:108-117) are both updated with the new level. Regex rules (ruling 6) slot in as levels after the wildcard rules and before list exceptions, allow before block, so the full order is: exact allow, exact block, wildcard allow, wildcard block, regex allow, regex block, list exception, list domain, list wildcard. "Tie-break at same specificity: allow wins" is preserved. ### 3. The compiled-source format grows a third body, checksum-compatibly -`compiler.compile` (`src/filter/compiler.zig:50-56`) takes a third writer -(`allow_w`) and `Counts` (compiler.zig:23-35) gains `exceptions: u32 = 0`. -Exception candidates go through the existing `addCandidate` path into a third -`Entries` and emit as a sorted, deduplicated `.allow` body. The shared SHA-256 -covers the bodies in order list, wild, allow — because SHA-256 of -`list ++ wild ++ ""` equals the current SHA-256 of `list ++ wild`, every -already-published checksum stays valid, and `Manager.loadSource` -(`src/filter/manager.zig:543-598`) treats a missing `.allow` file as an -empty body. No refetch is forced by upgrading. `bodyChecksum` -(manager.zig:1572) follows the same order; `source_file_suffixes` -(manager.zig:1589) gains `.allow.tmp` and `.allow` with longest-suffix-first -order preserved; the on-disk header (manager.zig:221-241) gains -`# exceptions {d}` after the `# wildcards` line and the pinning test at -manager.zig:1914-1946 is extended, not weakened. +`compiler.compile` (`src/filter/compiler.zig:50-56`) takes a third writer (`allow_w`) and `Counts` (compiler.zig:23-35) gains `exceptions: u32 = 0`. Exception candidates go through the existing `addCandidate` path into a third `Entries` and emit as a sorted, deduplicated `.allow` body. The shared SHA-256 covers the bodies in order list, wild, allow — because SHA-256 of `list ++ wild ++ ""` equals the current SHA-256 of `list ++ wild`, every already-published checksum stays valid, and `Manager.loadSource` (`src/filter/manager.zig:543-598`) treats a missing `.allow` file as an empty body. No refetch is forced by upgrading. `bodyChecksum` (manager.zig:1572) follows the same order; `source_file_suffixes` (manager.zig:1589) gains `.allow.tmp` and `.allow` with longest-suffix-first order preserved; the on-disk header (manager.zig:221-241) gains `# exceptions {d}` after the `# wildcards` line and the pinning test at manager.zig:1914-1946 is extended, not weakened. -The "checksum over the `.list` body followed by the `.wild` body" sentence -exists in THREE places, not the one this ruling first named: `compiler.zig:38-39`, -`manager.zig:218` and `sources_repo.zig:100`. All three move together, or the -next reader trusts a stale one. +The "checksum over the `.list` body followed by the `.wild` body" sentence exists in THREE places, not the one this ruling first named: `compiler.zig:38-39`, `manager.zig:218` and `sources_repo.zig:100`. All three move together, or the next reader trusts a stale one. ### 4. Exception counts persist and surface -Migration step 3 (`ddl_v3`, appended at `src/storage/migrations.zig:23-26`): -`ALTER TABLE blocklist_sources ADD COLUMN exception_count INTEGER NOT NULL -DEFAULT 0;`. `sources_repo.SourceRow` and `updateSourceStats` -(`src/storage/repositories/sources_repo.zig:80-93,159-169`) carry it, and the -checksum doc line at sources_repo.zig:100 is updated alongside -`bodyChecksum`'s per ruling 3; -`SourceStatus` rehydration (`manager.zig:1458-1502`) restores it alongside the -existing three counts; `StatusView` (`src/web/handlers/blocklists.zig:52-78`) -gains `exceptions: u32`; the blocklists UI shows it where `skipped_regex` -already shows. +Migration step 3 (`ddl_v3`, appended at `src/storage/migrations.zig:23-26`): `ALTER TABLE blocklist_sources ADD COLUMN exception_count INTEGER NOT NULL DEFAULT 0;`. `sources_repo.SourceRow` and `updateSourceStats` (`src/storage/repositories/sources_repo.zig:80-93,159-169`) carry it, and the checksum doc line at sources_repo.zig:100 is updated alongside `bodyChecksum`'s per ruling 3; `SourceStatus` rehydration (`manager.zig:1458-1502`) restores it alongside the existing three counts; `StatusView` (`src/web/handlers/blocklists.zig:52-78`) gains `exceptions: u32`; the blocklists UI shows it where `skipped_regex` already shows. -`skipped_regex` shows in TWO tables, and `exceptions` follows it into both: -`SourceStatus.skipped_regex` (`web/src/lib/types.ts:192`) renders at -`SourceStatusSection.tsx:112`, and `Blocklist.skipped_regex_count` -(`types.ts:163`) renders at `BlocklistsPage.tsx:188`. S1 therefore owns -`BlocklistsPage.tsx` as well. +`skipped_regex` shows in TWO tables, and `exceptions` follows it into both: `SourceStatus.skipped_regex` (`web/src/lib/types.ts:192`) renders at `SourceStatusSection.tsx:112`, and `Blocklist.skipped_regex_count` (`types.ts:163`) renders at `BlocklistsPage.tsx:188`. S1 therefore owns `BlocklistsPage.tsx` as well. ### 5. The regex engine is a Pike VM, linear-time by construction, `std` only -New file `src/filter/regex.zig`. Syntax: literal bytes, `.`, character -classes `[...]` with ranges and leading-`^` negation, escapes -`\. \\ \- \d \w`, repetition `* + ? {n} {n,m}`, alternation `|`, -non-capturing grouping `(...)`, anchors `^` and `$`. No backreferences, no -lookaround, no captures. +New file `src/filter/regex.zig`. Syntax: literal bytes, `.`, character classes `[...]` with ranges and leading-`^` negation, escapes `\. \\ \- \d \w`, repetition `* + ? {n} {n,m}`, alternation `|`, non-capturing grouping `(...)`, anchors `^` and `$`. No backreferences, no lookaround, no captures. **Two syntax amendments from S2's review**, both widening what is accepted: -- `{n,}` is legal. It lowers to `{n}` followed by `*`, stays linear, and an - over-large `n` still reports `PatternTooComplex`. Operators write this form; - rejecting it buys no safety. -- `\` before any ASCII punctuation yields that literal, not only the five - escapes listed above — `\*`, `\/` and `\+` occur in Pi-hole-style patterns. - This can only narrow a pattern to a literal, never silently change its - meaning. Escapes that WOULD change meaning stay rejected: any alphanumeric - escape outside `\d` and `\w` (`\s`, `\b`, `\1`, `\D`, `\p{L}`) is - `BadPattern`. +- `{n,}` is legal. It lowers to `{n}` followed by `*`, stays linear, and an over-large `n` still reports `PatternTooComplex`. Operators write this form; rejecting it buys no safety. +- `\` before any ASCII punctuation yields that literal, not only the five escapes listed above — `\*`, `\/` and `\+` occur in Pi-hole-style patterns. This can only narrow a pattern to a literal, never silently change its meaning. Escapes that WOULD change meaning stay rejected: any alphanumeric escape outside `\d` and `\w` (`\s`, `\b`, `\1`, `\D`, `\p{L}`) is `BadPattern`. -**One rejection the review added:** a quantifier applied directly to another -quantifier is `BadPattern`. `a+?` previously compiled as `(a+)?`, which -matches every name — a block rule written in conventional lazy syntax would -have sinkholed the whole LAN instead of being refused. Parenthesised forms -such as `(a+)?` stay legal and keep their meaning. Matching is unanchored unless anchors are written -(POSIX-grep convention, matching Pi-hole user expectations). Input is the -normalized lowercase name, ≤ `types.max_name_len` bytes. Hard limits, each a -distinct error: pattern ≤ 256 bytes (`PatternTooLong`), compiled program -≤ 1024 instructions (`PatternTooComplex`). Public API: +**One rejection the review added:** a quantifier applied directly to another quantifier is `BadPattern`. `a+?` previously compiled as `(a+)?`, which matches every name — a block rule written in conventional lazy syntax would have sinkholed the whole LAN instead of being refused. Parenthesised forms such as `(a+)?` stay legal and keep their meaning. Matching is unanchored unless anchors are written (POSIX-grep convention, matching Pi-hole user expectations). Input is the normalized lowercase name, ≤ `types.max_name_len` bytes. Hard limits, each a distinct error: pattern ≤ 256 bytes (`PatternTooLong`), compiled program ≤ 1024 instructions (`PatternTooComplex`). Public API: ```zig pub const Error = error{ OutOfMemory, BadPattern, PatternTooLong, PatternTooComplex }; @@ -132,166 +52,44 @@ pub fn compile(gpa: Allocator, pattern: []const u8) Error!Program; pub fn matches(prog: *const Program, input: []const u8) bool; ``` -`matches` is a Pike VM: two thread lists, each program counter admitted at -most once per input position, worst case O(program × input) with zero -allocation at match time. +`matches` is a Pike VM: two thread lists, each program counter admitted at most once per input position, worst case O(program × input) with zero allocation at match time. -**Amended in S2.** This ruling first said the thread lists live in the -`Program`, sized at compile time. They do not, and must not: the runtime is -`std.Io.Threaded`, so several query threads evaluate one shared snapshot at -once. Scratch inside a shared `Program` is a data race, and reaching it -through `*const Program` would need a `@constCast` that is undefined -behaviour on a genuinely const program. All VM scratch — both thread lists, -the admission marks and the closure stack — is instead a fixed array on the -caller's stack, sized by the compile-time `max_program_len` constant. Zero -allocation at match time is preserved, the published signature is unchanged, -and a `Program` becomes safe to share across threads, which the original -wording would have prevented. The engine is a fuzz-module root like parsers.zig and imports only -`std`. +**Amended in S2.** This ruling first said the thread lists live in the `Program`, sized at compile time. They do not, and must not: the runtime is `std.Io.Threaded`, so several query threads evaluate one shared snapshot at once. Scratch inside a shared `Program` is a data race, and reaching it through `*const Program` would need a `@constCast` that is undefined behaviour on a genuinely const program. All VM scratch — both thread lists, the admission marks and the closure stack — is instead a fixed array on the caller's stack, sized by the compile-time `max_program_len` constant. Zero allocation at match time is preserved, the published signature is unchanged, and a `Program` becomes safe to share across threads, which the original wording would have prevented. The engine is a fuzz-module root like parsers.zig and imports only `std`. ### 6. `regex` is a third rule kind, validated at the edge, memoized by the cache -Migration step 4 (`ddl_v4`): the 12-step rebuild of `rules` with -`CHECK(kind IN ('exact','wildcard','regex'))` — the frozen v1 DDL -(`src/storage/config_schema.zig:65-72`) cannot be edited. The rebuilt table -keeps the name `rules`: `config_schema.table_names` -(config_schema.zig:112-119) and the invariant tests at -config_schema.zig:120-127 and migrations.zig:356 assert the schema's table -set, and a rename would fail both. `model.RuleKind` -(`src/config/model.zig:259-275`) gains `.regex`; the exhaustive switches in -`config/validate.zig:1099-1130` (compile the pattern, report -`"... is not a valid regex pattern"` through the existing error path at -validate.zig:891-902) and `src/filter/rules.zig:62-68` extend. `RuleSet` -(`rules.zig:28-36`) grows `regex_allow` and `regex_block` slices holding -compiled `Program`s plus their pattern texts (for `Decision.matched`); -`bucketOf` (rules.zig:133-143) becomes a six-bucket layout, and the fixed -`var spans: [4]std.ArrayList(Span)` at rules.zig:55 widens with it; -`patternIsValid` (`validate.zig:1099-1103`) is declared -`error{OutOfMemory}!bool`, so a regex compile's `BadPattern`, -`PatternTooLong` and `PatternTooComplex` must either fold into `false` or -widen that error set together with its caller at validate.zig:893 — the -diagnostic text itself needs no edit, because validate.zig:898 already -interpolates `rule.kind.toDb()` into "{f} is not a valid {s} pattern"; -`max_regex_per_group: usize = 256` with `TooManyRegexRules` mirroring -`max_wildcards_per_group` (rules.zig:26). A pattern that fails to compile is -`error.BadPattern` at snapshot build, never skipped (rules.zig:41-49 doc -holds). Reason tags `rule_allow_regex` and `rule_block_regex` join -`matcher.Reason` (matcher.zig:24-32); `/api/lookup` and the query log pick -them up automatically via `@tagName` (`src/web/handlers/lookup.zig:89`; -`max_reason_len = 32` in `src/storage/logger.zig` fits both at 16 chars). -Regex evaluation runs only after every hash and wildcard level missed, and -answers are memoized by the existing DNS cache like every other decision, so -the per-query cost lands on cache misses only. +Migration step 4 (`ddl_v4`): the 12-step rebuild of `rules` with `CHECK(kind IN ('exact','wildcard','regex'))` — the frozen v1 DDL (`src/storage/config_schema.zig:65-72`) cannot be edited. The rebuilt table keeps the name `rules`: `config_schema.table_names` (config_schema.zig:112-119) and the invariant tests at config_schema.zig:120-127 and migrations.zig:356 assert the schema's table set, and a rename would fail both. `model.RuleKind` (`src/config/model.zig:259-275`) gains `.regex`; the exhaustive switches in `config/validate.zig:1099-1130` (compile the pattern, report `"... is not a valid regex pattern"` through the existing error path at validate.zig:891-902) and `src/filter/rules.zig:62-68` extend. `RuleSet` (`rules.zig:28-36`) grows `regex_allow` and `regex_block` slices holding compiled `Program`s plus their pattern texts (for `Decision.matched`); `bucketOf` (rules.zig:133-143) becomes a six-bucket layout, and the fixed `var spans: [4]std.ArrayList(Span)` at rules.zig:55 widens with it; `patternIsValid` (`validate.zig:1099-1103`) is declared `error{OutOfMemory}!bool`, so a regex compile's `BadPattern`, `PatternTooLong` and `PatternTooComplex` must either fold into `false` or widen that error set together with its caller at validate.zig:893 — the diagnostic text itself needs no edit, because validate.zig:898 already interpolates `rule.kind.toDb()` into "{f} is not a valid {s} pattern"; `max_regex_per_group: usize = 256` with `TooManyRegexRules` mirroring `max_wildcards_per_group` (rules.zig:26). A pattern that fails to compile is `error.BadPattern` at snapshot build, never skipped (rules.zig:41-49 doc holds). Reason tags `rule_allow_regex` and `rule_block_regex` join `matcher.Reason` (matcher.zig:24-32); `/api/lookup` and the query log pick them up automatically via `@tagName` (`src/web/handlers/lookup.zig:89`; `max_reason_len = 32` in `src/storage/logger.zig` fits both at 16 chars). Regex evaluation runs only after every hash and wildcard level missed, and answers are memoized by the existing DNS cache like every other decision, so the per-query cost lands on cache misses only. ### 7. The web contract names the third kind everywhere it names the first two -`src/web/handlers/rules.zig`: `toInput` accepts `"regex"`; the 400 string at -rules.zig:42 becomes `"kind must be 'exact', 'wildcard' or 'regex'"`. -`src/web/openapi.yaml:1948,1962,1976`: all three `enum: [exact, wildcard]` become -`[exact, wildcard, regex]`. `web/src/lib/types.ts:195`: -`RuleKind = "exact" | "wildcard" | "regex"`; the rules page kind selector -gains the option. Milestone 23 replaced the native `` with a React Aria wrapper, so that is now a data edit, not JSX: append to `KIND_OPTIONS` at `RulesPage.tsx:15-18`, and extend the option-list assertion at `RulesPage.test.tsx:119`. Any new test that opens the selector depends on the `CSS.escape` polyfill in `web/vitest.setup.ts`. Contract samples are regenerated with the AGENTS.md command (`zig build test -Dintegration -Dcontract-samples-out=...`); no npm script generates them. `nxdns export` / `import` round-trip the new kind with no extra work once `RuleKind.toDb/fromDb` extend — the enum round-trip test at model.zig:782 is extended to prove it. ### 8. PLAN amendments land with the code, in S3 -PLAN.md:36 (§2.2) is rewritten: operator regex rules are in scope, backed by -the linear-time engine of ruling 5; regex lines in downloaded lists stay -counted and skipped; `$` modifiers (except the `$important` suffix of ruling -1), partial-segment wildcards, and browser-syntax honoring stay permanently -out. PLAN.md:26 (§2.1 filtering sentence), PLAN.md:106 (§3.9), PLAN.md:108-117 -(§3.10 precedence) and PLAN.md:700 (decision B) are updated to match rulings -2 and 6. PLAN.md:99 (§3.8) documents exactly two compiled bodies and names -`.wild` as the second; ruling 3's third body makes it stale, so S1 -updates that line when it lands the `.allow` body. PLAN.md:73 ("No -TOML/regex/HTTP packages needed") stays true and stays as written — a -homegrown engine adds no package. In-code echoes of the old §2.2 move with it: -`src/filter/wildcard.zig:6-7,20-23`, `src/filter/parsers.zig:25`, -`src/filter/parser_abp.zig:5-6`, `src/filter/compiler.zig:129-131`. +PLAN.md:36 (§2.2) is rewritten: operator regex rules are in scope, backed by the linear-time engine of ruling 5; regex lines in downloaded lists stay counted and skipped; `$` modifiers (except the `$important` suffix of ruling 1), partial-segment wildcards, and browser-syntax honoring stay permanently out. PLAN.md:26 (§2.1 filtering sentence), PLAN.md:106 (§3.9), PLAN.md:108-117 (§3.10 precedence) and PLAN.md:700 (decision B) are updated to match rulings 2 and 6. PLAN.md:99 (§3.8) documents exactly two compiled bodies and names `.wild` as the second; ruling 3's third body makes it stale, so S1 updates that line when it lands the `.allow` body. PLAN.md:73 ("No TOML/regex/HTTP packages needed") stays true and stays as written — a homegrown engine adds no package. In-code echoes of the old §2.2 move with it: `src/filter/wildcard.zig:6-7,20-23`, `src/filter/parsers.zig:25`, `src/filter/parser_abp.zig:5-6`, `src/filter/compiler.zig:129-131`. ### 9. Milestone 20's authority modes bound where rules are written -m20 classifies POST/PUT/DELETE `/api/rules` as `.config_write` -(`src/web/routes.zig:94-97`), and under `.managed_file` authority the router -answers 403 (`src/web/router.zig:173-178`). Everything in this milestone -works in both modes, but the write path differs: in `.database` mode regex -rules arrive through the API; in `.managed_file` mode they arrive through the -config file and `src/config/reconcile.zig` (`reconcileRules`, -reconcile.zig:566-598), whose enum comparison carries `.regex` with no code -change. S3 owns a reconcile test proving a file-declared regex rule converges -into the table. Every S3 API acceptance check runs a server in `.database` -authority (no `--config`). The `SELECT *` dump helpers -(`src/config/import.zig:167-186`, `src/config/reconcile.zig:981`) will emit -the new `exception_count` column after `ddl_v3`; golden dump assertions in -those suites are updated in S1, which owns that migration. Reconcile's -runtime-column protection (reconcile.zig:11-15,435) covers `exception_count` -with no change — the column survives reconciles untouched. +m20 classifies POST/PUT/DELETE `/api/rules` as `.config_write` (`src/web/routes.zig:94-97`), and under `.managed_file` authority the router answers 403 (`src/web/router.zig:173-178`). Everything in this milestone works in both modes, but the write path differs: in `.database` mode regex rules arrive through the API; in `.managed_file` mode they arrive through the config file and `src/config/reconcile.zig` (`reconcileRules`, reconcile.zig:566-598), whose enum comparison carries `.regex` with no code change. S3 owns a reconcile test proving a file-declared regex rule converges into the table. Every S3 API acceptance check runs a server in `.database` authority (no `--config`). The `SELECT *` dump helpers (`src/config/import.zig:167-186`, `src/config/reconcile.zig:981`) will emit the new `exception_count` column after `ddl_v3`; golden dump assertions in those suites are updated in S1, which owns that migration. Reconcile's runtime-column protection (reconcile.zig:11-15,435) covers `exception_count` with no change — the column survives reconciles untouched. ### 10. Fuzz invariants move, never lapse -`tests/fuzz/blocklist_fuzz.zig` header invariant "covers_apex only on -`.wildcard`" (its stated form at :4-28) becomes "only on `.wildcard` or -`.exception`". `tests/fuzz/compiler_fuzz.zig:43` reads `Format` from -`compile`'s parameter list by index — the new `allow_w` parameter appends -after `wild_w`, leaving index 2 valid; the session touching `compile` runs the -fuzz suite and fixes that line if the assumption fails. A new -`tests/fuzz/regex_fuzz.zig` target asserts: `compile` on arbitrary bytes -never crashes and either errors or produces a program within the ruling-5 -limits; `matches` terminates and its VM step count never exceeds -program length × (input length + 1); compile-then-match is deterministic. +`tests/fuzz/blocklist_fuzz.zig` header invariant "covers_apex only on `.wildcard`" (its stated form at :4-28) becomes "only on `.wildcard` or `.exception`". `tests/fuzz/compiler_fuzz.zig:43` reads `Format` from `compile`'s parameter list by index — the new `allow_w` parameter appends after `wild_w`, leaving index 2 valid; the session touching `compile` runs the fuzz suite and fixes that line if the assumption fails. A new `tests/fuzz/regex_fuzz.zig` target asserts: `compile` on arbitrary bytes never crashes and either errors or produces a program within the ruling-5 limits; `matches` terminates and its VM step count never exceeds program length × (input length + 1); compile-then-match is deterministic. ## Sessions -Three sessions. S1 and S2 run in parallel — they share no files. S3 starts -after both land. +Three sessions. S1 and S2 run in parallel — they share no files. S3 starts after both land. ### Session S1: list exceptions end to end (tier 1) -Owns: `src/filter/parsers.zig`, `src/filter/parser_abp.zig`, -`src/filter/compiler.zig`, `src/filter/manager.zig`, `src/filter/matcher.zig`, -`src/filter/domain_set.zig` (only if a helper is needed; expected untouched), -`src/filter/filter_integration_test.zig`, `src/storage/migrations.zig` -(step 3 only), `src/storage/repositories/sources_repo.zig`, -`src/web/handlers/blocklists.zig`, `src/web/handlers/lookup.zig` (doc -sentence only), `src/web/openapi.yaml` (StatusView shape only), -`web/src/features/blocklists/*` (which includes `BlocklistsPage.tsx` per -ruling 4), `web/src/lib/types.ts` (source-stat fields -only), `web/src/lib/contractSamples.gen.ts`, `PLAN.md` (line 99 only, per -ruling 8 — S3 owns every other PLAN edit), -`tests/fuzz/blocklist_fuzz.zig`, `tests/fuzz/compiler_fuzz.zig`, and the -dump-golden assertions in `src/config/import.zig` and -`src/config/reconcile.zig` test suites only (ruling 9's `exception_count` -fallout from `ddl_v3`; S1 touches no reconcile logic). +Owns: `src/filter/parsers.zig`, `src/filter/parser_abp.zig`, `src/filter/compiler.zig`, `src/filter/manager.zig`, `src/filter/matcher.zig`, `src/filter/domain_set.zig` (only if a helper is needed; expected untouched), `src/filter/filter_integration_test.zig`, `src/storage/migrations.zig` (step 3 only), `src/storage/repositories/sources_repo.zig`, `src/web/handlers/blocklists.zig`, `src/web/handlers/lookup.zig` (doc sentence only), `src/web/openapi.yaml` (StatusView shape only), `web/src/features/blocklists/*` (which includes `BlocklistsPage.tsx` per ruling 4), `web/src/lib/types.ts` (source-stat fields only), `web/src/lib/contractSamples.gen.ts`, `PLAN.md` (line 99 only, per ruling 8 — S3 owns every other PLAN edit), `tests/fuzz/blocklist_fuzz.zig`, `tests/fuzz/compiler_fuzz.zig`, and the dump-golden assertions in `src/config/import.zig` and `src/config/reconcile.zig` test suites only (ruling 9's `exception_count` fallout from `ddl_v3`; S1 touches no reconcile logic). -- S1.1 `Kind.exception` in parsers.zig; parser_abp emits it per ruling 1; - parser_hosts and parser_domains never emit it (no change beyond the enum). +- S1.1 `Kind.exception` in parsers.zig; parser_abp emits it per ruling 1; parser_hosts and parser_domains never emit it (no change beyond the enum). - S1.2 compiler third body per ruling 3; `Counts.exceptions`. -- S1.3 manager: suffixes, header line, `bodyChecksum`, `loadSource` - missing-file-is-empty, `Snapshot.Compiled.allow_body`, - `prepareRefresh`/`publishRefresh`/`applyLoadOutcomes` carry the count. - `rejectedWithoutEntries` (manager.zig:1563-1570) treats a compile with only - exceptions as loadable, not rejected. -- S1.4 matcher: `SourceSets.exceptions`, `Reason.blocklist_exception`, - the new evaluate level per ruling 2, `memoryBytes` includes the new sets. -- S1.5 storage + API + UI per ruling 4. `migrations.zig:349` asserts - `target_version == 2`; S1 moves it to 3 (S3 moves it to 4). The spec did - not name that test; it fails otherwise. -- S1.6 tests: parser cases (`@@||x^`, `@@||x`, `@@||x^$important`, - `@@||x^$third-party` → unsupported, `@@x` → unsupported); compiler - three-body + checksum-compat cases (empty allow body reproduces the old - digest byte for byte); manager header pin extended; matcher precedence - cases: operator block beats list exception, list exception beats list - domain and list wildcard, exception parent walk covers apex and - subdomains; an integration case through - `filter_integration_test.zig` with a real ABP fixture carrying `@@` lines. +- S1.3 manager: suffixes, header line, `bodyChecksum`, `loadSource` missing-file-is-empty, `Snapshot.Compiled.allow_body`, `prepareRefresh`/`publishRefresh`/`applyLoadOutcomes` carry the count. `rejectedWithoutEntries` (manager.zig:1563-1570) treats a compile with only exceptions as loadable, not rejected. +- S1.4 matcher: `SourceSets.exceptions`, `Reason.blocklist_exception`, the new evaluate level per ruling 2, `memoryBytes` includes the new sets. +- S1.5 storage + API + UI per ruling 4. `migrations.zig:349` asserts `target_version == 2`; S1 moves it to 3 (S3 moves it to 4). The spec did not name that test; it fails otherwise. +- S1.6 tests: parser cases (`@@||x^`, `@@||x`, `@@||x^$important`, `@@||x^$third-party` → unsupported, `@@x` → unsupported); compiler three-body + checksum-compat cases (empty allow body reproduces the old digest byte for byte); manager header pin extended; matcher precedence cases: operator block beats list exception, list exception beats list domain and list wildcard, exception parent walk covers apex and subdomains; an integration case through `filter_integration_test.zig` with a real ABP fixture carrying `@@` lines. Acceptance (S1): - [x] `zig build test` passes; fuzz targets build and run. @@ -312,15 +110,10 @@ Acceptance (S1): ### Session S2: the regex engine (tier 2, engine only) -Owns: `src/filter/regex.zig` (new), `tests/fuzz/regex_fuzz.zig` (new), -`src/tests.zig` (one added line), `build.zig` (fuzz-suite wiring for the new -target only). +Owns: `src/filter/regex.zig` (new), `tests/fuzz/regex_fuzz.zig` (new), `src/tests.zig` (one added line), `build.zig` (fuzz-suite wiring for the new target only). - S2.1 the engine per ruling 5: parser → AST → NFA program → Pike VM. -- S2.2 unit tests in-file: every syntax form; anchored and unanchored - matching; negated classes; `{n,m}` bounds; each error case; the - pathological backtracker-killers (`(a+)+b` against `aaaaaaaaaaaaaaaaaaaaX`, - nested alternation) complete within the step bound. +- S2.2 unit tests in-file: every syntax form; anchored and unanchored matching; negated classes; `{n,m}` bounds; each error case; the pathological backtracker-killers (`(a+)+b` against `aaaaaaaaaaaaaaaaaaaaX`, nested alternation) complete within the step bound. - S2.3 the fuzz target per ruling 10. Acceptance (S2): @@ -334,32 +127,15 @@ Acceptance (S2): ### Session S3: the regex rule kind, wired through (needs S1 + S2) -Owns: `src/storage/migrations.zig` (step 4), `src/storage/config_schema.zig` -(comment only if needed), `src/config/model.zig`, `src/config/validate.zig`, -`src/filter/rules.zig`, `src/filter/matcher.zig`, -`src/storage/repositories/rules_repo.zig`, `src/web/handlers/rules.zig`, -`src/web/handlers/mutations.zig` (only if `checkRule` needs the kind), -`src/web/openapi.yaml`, `web/src/features/rules/*`, `web/src/lib/types.ts`, -`web/src/lib/contractSamples.gen.ts`, `src/config/reconcile.zig` (test per -ruling 9), `PLAN.md`, `src/filter/wildcard.zig` (comments), -`src/filter/parsers.zig` (comment), `src/filter/parser_abp.zig` (comment), -`src/filter/compiler.zig` (comment), `tools/bench.zig`. +Owns: `src/storage/migrations.zig` (step 4), `src/storage/config_schema.zig` (comment only if needed), `src/config/model.zig`, `src/config/validate.zig`, `src/filter/rules.zig`, `src/filter/matcher.zig`, `src/storage/repositories/rules_repo.zig`, `src/web/handlers/rules.zig`, `src/web/handlers/mutations.zig` (only if `checkRule` needs the kind), `src/web/openapi.yaml`, `web/src/features/rules/*`, `web/src/lib/types.ts`, `web/src/lib/contractSamples.gen.ts`, `src/config/reconcile.zig` (test per ruling 9), `PLAN.md`, `src/filter/wildcard.zig` (comments), `src/filter/parsers.zig` (comment), `src/filter/parser_abp.zig` (comment), `src/filter/compiler.zig` (comment), `tools/bench.zig`. - S3.1 migration step 4 per ruling 6; repo and model layers. - S3.2 validate at both edges (config import, API) per rulings 6 and 7. - S3.3 `RuleSet` six buckets; matcher levels per ruling 2; reasons. - S3.4 web + UI + openapi + samples per ruling 7. - S3.5 PLAN and comment amendments per ruling 8. -- S3.6 bench: the filter suite in `tools/bench.zig` gains a variant with 32 - regex rules loaded; the existing p95 < 1 ms assertion covers it. The line to - change is `tools/bench.zig:158`, which currently reads `.rules = &.{}` — the - filter bench loads no rules at all today, so the variant is new coverage - rather than an edit to an existing rule set. S3 also moves - `migrations.zig:349` from 3 to 4. -- S3.7 tests: rule CRUD with kind `regex` through the API including the 400 - for a bad pattern at insert time; precedence cases regex-allow over - regex-block, wildcard over regex, regex over list entries; export/import - round trip; a migration test upgrading a v3 database. +- S3.6 bench: the filter suite in `tools/bench.zig` gains a variant with 32 regex rules loaded; the existing p95 < 1 ms assertion covers it. The line to change is `tools/bench.zig:158`, which currently reads `.rules = &.{}` — the filter bench loads no rules at all today, so the variant is new coverage rather than an edit to an existing rule set. S3 also moves `migrations.zig:349` from 3 to 4. +- S3.7 tests: rule CRUD with kind `regex` through the API including the 400 for a bad pattern at insert time; precedence cases regex-allow over regex-block, wildcard over regex, regex over list entries; export/import round trip; a migration test upgrading a v3 database. Acceptance (S3): - [x] `zig build test` and `cd web && npm test` pass. @@ -380,11 +156,7 @@ Acceptance (S3): ### Orchestrator -Verify S1 and S2 acceptance before starting S3. After S3: run the full gate -set (`zig build test`, `test-aarch64` if qemu present, `npm test`, -`npm run assert-bundled`), then a live smoke against a scratch server: load -one real ABP list with `@@` lines, add one regex rule, verify both over dig -and `/api/lookup`. Record deviations in `## Recorded (implementation)`. +Verify S1 and S2 acceptance before starting S3. After S3: run the full gate set (`zig build test`, `test-aarch64` if qemu present, `npm test`, `npm run assert-bundled`), then a live smoke against a scratch server: load one real ABP list with `@@` lines, add one regex rule, verify both over dig and `/api/lookup`. Record deviations in `## Recorded (implementation)`. ## Module layout @@ -419,227 +191,56 @@ Deleted surface: none. ## Recorded (anchor re-verification, 2026-08-12) -The spec was written against `ffc3ca6` and verified against `a8e0fe4`. It was -re-verified a third time after milestones 22 and 23 landed (TypeScript 7, -Tailwind removed, StyleX and React Aria). Findings, all folded into the -rulings above: +The spec was written against `ffc3ca6` and verified against `a8e0fe4`. It was re-verified a third time after milestones 22 and 23 landed (TypeScript 7, Tailwind removed, StyleX and React Aria). Findings, all folded into the rulings above: -- **No Zig file changed** between `a8e0fe4` and this re-verification. Every - Zig anchor holds; six ranges are off by a line or two but still contain what - the spec names. The schema is still at version 2, so migration steps 3 and 4 - are genuinely new. -- The rules-page kind selector is no longer a ``. It is a `KIND_OPTIONS` array feeding `web/src/ui/Select.tsx`, a React Aria wrapper (ruling 7 rewritten). +- `migrations.zig:349` pins `target_version` and is unnamed by the original spec. S1 moves it to 3, S3 to 4. +- `exception_count` has two UI sites, not one, so S1 owns `BlocklistsPage.tsx` (ruling 4 rewritten). - The checksum sentence has three copies, not one (ruling 3 rewritten). -- `patternIsValid`'s error set cannot carry the engine's three error tags as - written (ruling 6 rewritten). -- `PLAN.md:99` documents two compiled bodies and goes stale with ruling 3 - (ruling 8 rewritten). -- `tools/bench.zig:158` reads `.rules = &.{}`, so the filter bench loads no - rules today. -- `web/vitest.setup.ts` polyfills `CSS.escape`; without it, a test that opens - the React Aria selector throws under jsdom. -- `npm run build` gained `scripts/assert-css-layers.mjs`, which fails on any - rule outside a cascade layer. A pure StyleX change cannot trip it. +- `patternIsValid`'s error set cannot carry the engine's three error tags as written (ruling 6 rewritten). +- `PLAN.md:99` documents two compiled bodies and goes stale with ruling 3 (ruling 8 rewritten). +- `tools/bench.zig:158` reads `.rules = &.{}`, so the filter bench loads no rules today. +- `web/vitest.setup.ts` polyfills `CSS.escape`; without it, a test that opens the React Aria selector throws under jsdom. +- `npm run build` gained `scripts/assert-css-layers.mjs`, which fails on any rule outside a cascade layer. A pure StyleX change cannot trip it. ## Recorded (implementation) -Deviations and findings from the build, the Codex review rounds and the live -smoke. Everything here is folded into the code; nothing is outstanding. +Deviations and findings from the build, the Codex review rounds and the live smoke. Everything here is folded into the code; nothing is outstanding. -- **Regex scratch is on the caller's stack, not in `Program`.** `std.Io.Threaded` - means several query threads share one snapshot, so the Pike VM's two thread - lists cannot live in the compiled program. `matches` takes its scratch from - the caller's frame, which keeps `Program` immutable and shareable and keeps - the match path allocation-free. -- **`{n,}` is legal and `\` + any ASCII punctuation is legal.** Ruling 5 named - neither. Quantifier-on-quantifier (`a+?` read as `(a+)?`) is `BadPattern`: - the first Codex round found `a+?` compiling to something that matched - everything. -- **Embedded whitespace was accepted on the anchored ABP forms.** A line such - as `||good.example bad.example^` reached the compiler, which lowercases and - length-checks but does not reject a space, and wrote an entry only a query - carrying the same space could match. `compiler.zig` passes `.wildcard` and - `.exception` text to `addCandidate` whole, which is why the space survived - there. `parser_abp.isNameCandidate` now refuses whitespace and control bytes - on those two paths. +- **Regex scratch is on the caller's stack, not in `Program`.** `std.Io.Threaded` means several query threads share one snapshot, so the Pike VM's two thread lists cannot live in the compiled program. `matches` takes its scratch from the caller's frame, which keeps `Program` immutable and shareable and keeps the match path allocation-free. +- **`{n,}` is legal and `\` + any ASCII punctuation is legal.** Ruling 5 named neither. Quantifier-on-quantifier (`a+?` read as `(a+)?`) is `BadPattern`: the first Codex round found `a+?` compiling to something that matched everything. +- **Embedded whitespace was accepted on the anchored ABP forms.** A line such as `||good.example bad.example^` reached the compiler, which lowercases and length-checks but does not reject a space, and wrote an entry only a query carrying the same space could match. `compiler.zig` passes `.wildcard` and `.exception` text to `addCandidate` whole, which is why the space survived there. `parser_abp.isNameCandidate` now refuses whitespace and control bytes on those two paths. - The bare-name path deliberately does **not** use it. `compiler.zig:93-98` - tokenizes `.domain` text on whitespace and adds each field separately, so a - bare line carrying a space was never broken — it produced two valid entries. - Since `detectFormat` assigns one format per source, a mostly-ABP list that - also carries hosts-style lines depends on exactly that tokenizer to keep - them working. The first attempt at this fix applied the helper to all three - paths and silently dropped that fallback; the second Codex pass caught it. - Two tests now pin it: a parser test that the bare form stays `.domain`, and - a compiler test that an ABP-classified list carrying `0.0.0.0 ads.example` - still emits `ads.example`. The third pass pointed out that the parser test - alone would pass even if the compiler stopped tokenizing, which is the - behaviour the fallback actually depends on. -- **The `.allow` body left stale enumerations behind it.** Adding a third - compiled body — and with it a fourth temporary, `.allow.tmp` — updated the - production code but not every place that lists the file names. The third review pass found four: two `manager.zig` tests that - spell the names by hand, the orphan-sweep fixture in - `filter_integration_test.zig`, and `PLAN.md` §3.13 and §5. The - `manager.zig` table-driven test was worse than stale — it iterates - `source_file_suffixes` itself, so deleting an entry changes the code and the - test's expectations together and everything still passes. The whole repo was - then swept for the pattern rather than the four instances patched, which - turned up seven more — including the reload-cancellation test, whose fixture - wrote no `.allow` file at all, so the third of `loadSource`'s three read - sites was never exercised. A fourth pass then found test 10e comparing only - `.list` and `.wild` across a restart, so a restart that rewrote the exception - body alone would have stayed green. A `comptime` assertion on - `source_file_suffixes.len` now breaks the build when a suffix is added or - removed without updating the hand-written tests. -- **A pre-existing flake in the required suite.** `src/cli.zig:1538` asserted - `std.mem.count(u8, text, "OK") == 0` over output that embeds the temporary - directory path. `std.testing.tmpDir` names that directory with base64 over - 12 random bytes, whose 64-symbol alphabet includes uppercase: 15 adjacent - positions each carry `OK` with probability 1/4096, so about one run in 273 - fails a test that has nothing to do with naming. It surfaced during - this milestone's watched-fail injections. The assertion now checks that no - line *starts* with a verdict, which covers both `OK:` and - `OK upstreams[...]` and cannot match a path segment. Reproduced and fixed - outside the milestone's scope because a randomly failing required gate - devalues every green run after it. -- **`PLAN.md` still described the seed-once config model.** Seven sites said or - implied that the first start seeds the database from `/etc/nxdns/config.zon`, - which milestone 20 replaced with the two authority modes selected by the - presence of `--config`. One of them listed a `config/bootstrap.zig` that does - not exist — the module is `loader.zig` plus `reconcile.zig`. The claim was - checked against `src/app.zig:339`, not just against the docs. This is - milestone-20 drift found while fixing the milestone-21 echoes in the same - document, and corrected because PLAN is the source of truth a later session - builds from. -- **`PLAN.md` §12.1 held a config sample nobody could load.** It described an - `.upstream.servers` field that never existed and omitted the required - `.groups` and `.upstreams`. The section now points at - `docs/reference/configuration.md` and `nxdns export` and keeps only a - skeleton: a second copy of the schema is what produced the drift, so the - copy is gone rather than corrected. -- **`PLAN.md` §7.1 claimed blocklist entries never parent-walk.** Two of the - three list levels do: wildcard entries match every proper parent, and - exception entries walk the candidate chain, which is why - `@@||good.ads.example^` also lifts `y.good.ads.example`. Only domain entries - match the query name alone. -- **`src/config/model.zig`'s header described the retired bootstrap** and - omitted `exception_count` from its list of runtime columns. The first attempt - at correcting it introduced a new error — it called import a wholesale - replacement set against reconciliation — which the sixth pass caught. - `import.zig:3` is explicit that import has been a thin wrapper over - `reconcile.zig` since milestone 20, so there is one declarative write path, - not two, and it preserves the runtime state of every row the input still - names (`reconcile.zig:1139`). A seventh pass then corrected two more claims - in the same header: `Config` is the whole shape of a config file but not the - only shape the repositories accept (the API writes through `RuleInput`, - `ClientInput`, `ClientEdit`), and compiled-body reuse turns on the preserved - source id and checksum rather than the counters — `loadSource` names the - files after the id and accepts them only against the stored checksum. -- **`PLAN.md` §11.2 presented the v1 DDL as the live schema.** It predates - three migrations, so it lacks `upstreams.tls_name` and - `blocklist_sources.exception_count` and its `kind` CHECK admits only - `exact` and `wildcard` — implementing against it would produce a database - that rejects every regex rule this milestone added. The section is now - labelled the v1 baseline and points at `config_schema.zig` and - `migrations.zig`, with the three steps named. -- **`INSTALL.md` said the service reads `config.zon` "on the first start".** - Neither authority mode behaves that way: under `run --config` the service - reads it on every start, and under database authority `nxdns import` reads it - while a bare `nxdns run` never does. The sentence justified a file mode, so - an operator tightening permissions after first boot would have broken the - next file-mode restart. More milestone-20 drift. The claim had a second copy - in `docs/how-to/install-with-systemd.md`, found only because the seventh pass - looked for it after the first copy was fixed. -- **`PLAN.md` §7.1 and the Phase 5 summary missed this milestone's own - additions.** The evaluation sequence went straight from operator rules to - blocklist domains, omitting the exception level, and the matcher was still - enumerated as exact/parent/wildcard with no regex. Ruling 8 required these - echoes and S3 did not reach them. -- **The UI trimmed regex patterns.** `RulesPage.onSubmit` applied - `pattern.trim()` to every kind, so a regex created in the UI did not store - the bytes an identical `POST /api/rules` would. It now trims only `exact` - and `wildcard`, where the server normalizes anyway. No client-side - whitespace rejection was added: `" foo|bar"` still has a live `bar` branch, - so refusing it would over-reject, and the server stays the authority on - pattern validity. -- **The rule pattern field opted into mobile autocapitalization.** An - autocapitalized regex validates and then silently never matches, because - query names are lowercase and a regex is never normalized. The field now - sets `autoCapitalize="none"`, `autoCorrect="off"`, `spellCheck={false}`. -- **`RuleSet.build` received the snapshot arena for its temporaries.** An - arena reclaims only its most recent allocation, so every `defer …deinit` - inside `build` was a silent no-op and the scratch survived until snapshot - teardown, uncounted by `memoryBytes()`. `build` now takes a permanent and a - scratch allocator, and `matcher.zig` passes `arena` and `gpa` respectively. - Regex programs compile into scratch and are copied across by a new - `Program.clone`, which keeps `regex.zig` a single-allocator engine and - leaves `compile`'s signature — and therefore `config/validate.zig` and the - fuzz target — untouched. + The bare-name path deliberately does **not** use it. `compiler.zig:93-98` tokenizes `.domain` text on whitespace and adds each field separately, so a bare line carrying a space was never broken — it produced two valid entries. Since `detectFormat` assigns one format per source, a mostly-ABP list that also carries hosts-style lines depends on exactly that tokenizer to keep them working. The first attempt at this fix applied the helper to all three paths and silently dropped that fallback; the second Codex pass caught it. Two tests now pin it: a parser test that the bare form stays `.domain`, and a compiler test that an ABP-classified list carrying `0.0.0.0 ads.example` still emits `ads.example`. The third pass pointed out that the parser test alone would pass even if the compiler stopped tokenizing, which is the behaviour the fallback actually depends on. +- **The `.allow` body left stale enumerations behind it.** Adding a third compiled body — and with it a fourth temporary, `.allow.tmp` — updated the production code but not every place that lists the file names. The third review pass found four: two `manager.zig` tests that spell the names by hand, the orphan-sweep fixture in `filter_integration_test.zig`, and `PLAN.md` §3.13 and §5. The `manager.zig` table-driven test was worse than stale — it iterates `source_file_suffixes` itself, so deleting an entry changes the code and the test's expectations together and everything still passes. The whole repo was then swept for the pattern rather than the four instances patched, which turned up seven more — including the reload-cancellation test, whose fixture wrote no `.allow` file at all, so the third of `loadSource`'s three read sites was never exercised. A fourth pass then found test 10e comparing only `.list` and `.wild` across a restart, so a restart that rewrote the exception body alone would have stayed green. A `comptime` assertion on `source_file_suffixes.len` now breaks the build when a suffix is added or removed without updating the hand-written tests. +- **A pre-existing flake in the required suite.** `src/cli.zig:1538` asserted `std.mem.count(u8, text, "OK") == 0` over output that embeds the temporary directory path. `std.testing.tmpDir` names that directory with base64 over 12 random bytes, whose 64-symbol alphabet includes uppercase: 15 adjacent positions each carry `OK` with probability 1/4096, so about one run in 273 fails a test that has nothing to do with naming. It surfaced during this milestone's watched-fail injections. The assertion now checks that no line *starts* with a verdict, which covers both `OK:` and `OK upstreams[...]` and cannot match a path segment. Reproduced and fixed outside the milestone's scope because a randomly failing required gate devalues every green run after it. +- **`PLAN.md` still described the seed-once config model.** Seven sites said or implied that the first start seeds the database from `/etc/nxdns/config.zon`, which milestone 20 replaced with the two authority modes selected by the presence of `--config`. One of them listed a `config/bootstrap.zig` that does not exist — the module is `loader.zig` plus `reconcile.zig`. The claim was checked against `src/app.zig:339`, not just against the docs. This is milestone-20 drift found while fixing the milestone-21 echoes in the same document, and corrected because PLAN is the source of truth a later session builds from. +- **`PLAN.md` §12.1 held a config sample nobody could load.** It described an `.upstream.servers` field that never existed and omitted the required `.groups` and `.upstreams`. The section now points at `docs/reference/configuration.md` and `nxdns export` and keeps only a skeleton: a second copy of the schema is what produced the drift, so the copy is gone rather than corrected. +- **`PLAN.md` §7.1 claimed blocklist entries never parent-walk.** Two of the three list levels do: wildcard entries match every proper parent, and exception entries walk the candidate chain, which is why `@@||good.ads.example^` also lifts `y.good.ads.example`. Only domain entries match the query name alone. +- **`src/config/model.zig`'s header described the retired bootstrap** and omitted `exception_count` from its list of runtime columns. The first attempt at correcting it introduced a new error — it called import a wholesale replacement set against reconciliation — which the sixth pass caught. `import.zig:3` is explicit that import has been a thin wrapper over `reconcile.zig` since milestone 20, so there is one declarative write path, not two, and it preserves the runtime state of every row the input still names (`reconcile.zig:1139`). A seventh pass then corrected two more claims in the same header: `Config` is the whole shape of a config file but not the only shape the repositories accept (the API writes through `RuleInput`, `ClientInput`, `ClientEdit`), and compiled-body reuse turns on the preserved source id and checksum rather than the counters — `loadSource` names the files after the id and accepts them only against the stored checksum. +- **`PLAN.md` §11.2 presented the v1 DDL as the live schema.** It predates three migrations, so it lacks `upstreams.tls_name` and `blocklist_sources.exception_count` and its `kind` CHECK admits only `exact` and `wildcard` — implementing against it would produce a database that rejects every regex rule this milestone added. The section is now labelled the v1 baseline and points at `config_schema.zig` and `migrations.zig`, with the three steps named. +- **`INSTALL.md` said the service reads `config.zon` "on the first start".** Neither authority mode behaves that way: under `run --config` the service reads it on every start, and under database authority `nxdns import` reads it while a bare `nxdns run` never does. The sentence justified a file mode, so an operator tightening permissions after first boot would have broken the next file-mode restart. More milestone-20 drift. The claim had a second copy in `docs/how-to/install-with-systemd.md`, found only because the seventh pass looked for it after the first copy was fixed. +- **`PLAN.md` §7.1 and the Phase 5 summary missed this milestone's own additions.** The evaluation sequence went straight from operator rules to blocklist domains, omitting the exception level, and the matcher was still enumerated as exact/parent/wildcard with no regex. Ruling 8 required these echoes and S3 did not reach them. +- **The UI trimmed regex patterns.** `RulesPage.onSubmit` applied `pattern.trim()` to every kind, so a regex created in the UI did not store the bytes an identical `POST /api/rules` would. It now trims only `exact` and `wildcard`, where the server normalizes anyway. No client-side whitespace rejection was added: `" foo|bar"` still has a live `bar` branch, so refusing it would over-reject, and the server stays the authority on pattern validity. +- **The rule pattern field opted into mobile autocapitalization.** An autocapitalized regex validates and then silently never matches, because query names are lowercase and a regex is never normalized. The field now sets `autoCapitalize="none"`, `autoCorrect="off"`, `spellCheck={false}`. +- **`RuleSet.build` received the snapshot arena for its temporaries.** An arena reclaims only its most recent allocation, so every `defer …deinit` inside `build` was a silent no-op and the scratch survived until snapshot teardown, uncounted by `memoryBytes()`. `build` now takes a permanent and a scratch allocator, and `matcher.zig` passes `arena` and `gpa` respectively. Regex programs compile into scratch and are copied across by a new `Program.clone`, which keeps `regex.zig` a single-allocator engine and leaves `compile`'s signature — and therefore `config/validate.zig` and the fuzz target — untouched. - Measured on 16 groups each holding 256 regex rules of 254 bytes, - 4096 wildcards and 2048 exact rules, with no blocklist sources: - arena capacity fell from 135,662,440 to 12,686,214 bytes against an - unchanged `memoryBytes()` of 11,058,791, so the ratio of real to reported - went from 12.27× to 1.15×. The hidden footprint per snapshot fell from - 118.83 MiB to 1.55 MiB, so 117.28 MiB went away. A reload holds two - snapshots, so it was carrying twice that. - `memoryBytes()` needed no change: the formula was always right about what - the `RuleSet` retains, and the divergence was arena capacity the formula - does not claim to describe. The residual 1.15× is arena node headers and - page rounding, which `memoryBytes` documents itself as excluding. + Measured on 16 groups each holding 256 regex rules of 254 bytes, 4096 wildcards and 2048 exact rules, with no blocklist sources: arena capacity fell from 135,662,440 to 12,686,214 bytes against an unchanged `memoryBytes()` of 11,058,791, so the ratio of real to reported went from 12.27× to 1.15×. The hidden footprint per snapshot fell from 118.83 MiB to 1.55 MiB, so 117.28 MiB went away. A reload holds two snapshots, so it was carrying twice that. `memoryBytes()` needed no change: the formula was always right about what the `RuleSet` retains, and the divergence was arena capacity the formula does not claim to describe. The residual 1.15× is arena node headers and page rounding, which `memoryBytes` documents itself as excluding. - The guard is `the build's temporaries stay out of the permanent allocator`, - which asserts `arena.queryCapacity() < 2 * set.memoryBytes()` and passes - `scratch` as `testing.allocator`, so a permanent allocation wrongly taken - from scratch also fails as a leak. It was watched failing with the split - reverted. -- **An interactive `--fuzz` session cannot run on zig 0.16.0.** Building any - fuzz target with `-ffuzz` fails inside the stock - `/usr/lib/zig/compiler/test_runner.zig:566`, which passes a - `*builtin.StackTrace` to `debug.writeStackTrace` where a - `*const debug.StackTrace` is wanted — two distinct struct declarations. The - failure is entirely inside the toolchain and reproduces on `compiler-fuzz`, - a target this milestone did not touch. Corpus replay under `zig build test` - is unaffected and remains the evidence for the step-bound property. -- **`api.md` gained a block-reason table.** The milestone added three reason - tags and no doc page enumerated any of them. The table lists all nine in - evaluation order plus `none` and the `cname:` prefix. -- **Two comment sites still counted three temporaries.** `manager.zig` line 40 - (the `refresh_lock` invariant) and line 1293 (`pruneOrphans`) both omitted - `.allow.tmp`. Each presents itself as exhaustive, so a maintainer could have - added an `.allow.tmp` writer outside `refresh_lock` and let the orphan sweep - delete it mid-compile. `sourceFileId` and `source_file_suffixes` already - matched all four. -- **The rule pattern placeholder named only two kinds.** Ruling 7 requires the - web contract to name the third kind everywhere it names the first two, and - the placeholder read `ads.example.com or *.example.com`. It now carries a - regex example too. -- **Two pre-existing tutorial errors surfaced during the docs sweep.** - `tutorial/first-run.md` step 14 claimed the compiled list stays on disk when - it is in fact pruned (real output: `pruned orphaned blocklist file 1.allow`, - `1.wild`, `1.list`), and step 11 named `ads.doubleclick.net`, which - StevenBlack no longer carries. + The guard is `the build's temporaries stay out of the permanent allocator`, which asserts `arena.queryCapacity() < 2 * set.memoryBytes()` and passes `scratch` as `testing.allocator`, so a permanent allocation wrongly taken from scratch also fails as a leak. It was watched failing with the split reverted. +- **An interactive `--fuzz` session cannot run on zig 0.16.0.** Building any fuzz target with `-ffuzz` fails inside the stock `/usr/lib/zig/compiler/test_runner.zig:566`, which passes a `*builtin.StackTrace` to `debug.writeStackTrace` where a `*const debug.StackTrace` is wanted — two distinct struct declarations. The failure is entirely inside the toolchain and reproduces on `compiler-fuzz`, a target this milestone did not touch. Corpus replay under `zig build test` is unaffected and remains the evidence for the step-bound property. +- **`api.md` gained a block-reason table.** The milestone added three reason tags and no doc page enumerated any of them. The table lists all nine in evaluation order plus `none` and the `cname:` prefix. +- **Two comment sites still counted three temporaries.** `manager.zig` line 40 (the `refresh_lock` invariant) and line 1293 (`pruneOrphans`) both omitted `.allow.tmp`. Each presents itself as exhaustive, so a maintainer could have added an `.allow.tmp` writer outside `refresh_lock` and let the orphan sweep delete it mid-compile. `sourceFileId` and `source_file_suffixes` already matched all four. +- **The rule pattern placeholder named only two kinds.** Ruling 7 requires the web contract to name the third kind everywhere it names the first two, and the placeholder read `ads.example.com or *.example.com`. It now carries a regex example too. +- **Two pre-existing tutorial errors surfaced during the docs sweep.** `tutorial/first-run.md` step 14 claimed the compiled list stays on disk when it is in fact pruned (real output: `pruned orphaned blocklist file 1.allow`, `1.wild`, `1.list`), and step 11 named `ads.doubleclick.net`, which StevenBlack no longer carries. ## Anti-requirements -- No `$` modifier support beyond tolerating `$important` on exception lines. - `$dnstype`, `$dnsrewrite`, `$client`, `$denyallow` and every browser - modifier stay unsupported and counted. -- No regex from downloaded lists. `.regex` lines stay counted and skipped; - `skipped_regex` keeps its meaning. The engine exists for operator rules - only. -- No partial-segment wildcards (`ads*.example.com`) — writing one as a rule - stays rejected; the regex kind covers the need. -- No backreferences, lookaround, captures, named groups or Unicode classes in - the engine, ever. A pattern needing them is rejected, not approximated. +- No `$` modifier support beyond tolerating `$important` on exception lines. `$dnstype`, `$dnsrewrite`, `$client`, `$denyallow` and every browser modifier stay unsupported and counted. +- No regex from downloaded lists. `.regex` lines stay counted and skipped; `skipped_regex` keeps its meaning. The engine exists for operator rules only. +- No partial-segment wildcards (`ads*.example.com`) — writing one as a rule stays rejected; the regex kind covers the need. +- No backreferences, lookaround, captures, named groups or Unicode classes in the engine, ever. A pattern needing them is rejected, not approximated. - No PCRE2, RE2 or any external regex dependency. -- No exception-rule UI editor: exceptions come from lists; operators write - allow rules. -- No re-download forced by the upgrade; checksum compatibility (ruling 3) is - a requirement, not an optimization. +- No exception-rule UI editor: exceptions come from lists; operators write allow rules. +- No re-download forced by the upgrade; checksum compatibility (ruling 3) is a requirement, not an optimization. diff --git a/specs/milestone-22.md b/specs/milestone-22.md index 9c6c803..88063cd 100644 --- a/specs/milestone-22.md +++ b/specs/milestone-22.md @@ -1,63 +1,32 @@ # Milestone 22: TypeScript 7 -Goal: move the web toolchain from TypeScript 6.0.3 to TypeScript 7.0.x (the -native compiler, GA 2026-07-08). One devDependency bump; every web gate and -the release pipeline stay green. +Goal: move the web toolchain from TypeScript 6.0.3 to TypeScript 7.0.x (the native compiler, GA 2026-07-08). One devDependency bump; every web gate and the release pipeline stay green. -Design written 2026-08-12 against HEAD `91a0aa9`. TypeScript 7.0.2 is -`typescript@latest` on npm at spec time; the milestone pins whatever 7.0.x is -current when the session runs. +Design written 2026-08-12 against HEAD `91a0aa9`. TypeScript 7.0.2 is `typescript@latest` on npm at spec time; the milestone pins whatever 7.0.x is current when the session runs. ## Implementation contract (read first) - Read `AGENTS.md`, then this spec whole, before session work starts. -- `web/src/lib/contractSamples.gen.ts` is generated; it is type-checked by - `tsc -b` but never hand-edited. If TS7 rejects it, the fix goes in the - generator (`AGENTS.md` regeneration procedure), not the file. -- No license work: `typescript` is a devDependency, absent from both the - runtime closure and the bundled set in `licenses/dependency-identity.txt`. - The Zig drift guard (`src/licenses_drift_test.zig`) must still pass — - it recomputes from `web/package-lock.json` and dev-only changes are - invisible to it. +- `web/src/lib/contractSamples.gen.ts` is generated; it is type-checked by `tsc -b` but never hand-edited. If TS7 rejects it, the fix goes in the generator (`AGENTS.md` regeneration procedure), not the file. +- No license work: `typescript` is a devDependency, absent from both the runtime closure and the bundled set in `licenses/dependency-identity.txt`. The Zig drift guard (`src/licenses_drift_test.zig`) must still pass — it recomputes from `web/package-lock.json` and dev-only changes are invisible to it. ## Rulings (binding) ### 1. The bump is one line plus the lockfile -`web/package.json:45` `"typescript": "6.0.3"` becomes the current 7.0.x, -exact pin (repo convention: no ranges, milestone-14 ruling 12 pins by exact -version everywhere). `npm install` updates `web/package-lock.json`. Nothing -else in `dependencies` or `devDependencies` moves in this milestone. +`web/package.json:45` `"typescript": "6.0.3"` becomes the current 7.0.x, exact pin (repo convention: no ranges, milestone-14 ruling 12 pins by exact version everywhere). `npm install` updates `web/package-lock.json`. Nothing else in `dependencies` or `devDependencies` moves in this milestone. ### 2. The tsconfigs are already TS7-clean; verify, do not churn -Checked against TS7's removed options (recorded here so the session does not -re-derive it): `tsconfig.app.json` and `tsconfig.node.json` use -`moduleResolution: "bundler"`, `module: "esnext"`, targets `es2022`/`es2023`, -`paths` without `baseUrl`, `verbatimModuleSyntax`, `isolatedModules`, -`noEmit`. None of TS7's removals (`target: es5`, `downlevelIteration`, -`moduleResolution: node10`/`classic`, `module: amd/umd/system/none`, -`baseUrl`-as-alias) are present. `tsconfig.json` is a two-entry project -reference; TS7 supports `tsc -b` and project references. The session changes -a tsconfig only if `tsc -b` errors demand it, and records any such change in -`## Recorded (implementation)`. +Checked against TS7's removed options (recorded here so the session does not re-derive it): `tsconfig.app.json` and `tsconfig.node.json` use `moduleResolution: "bundler"`, `module: "esnext"`, targets `es2022`/`es2023`, `paths` without `baseUrl`, `verbatimModuleSyntax`, `isolatedModules`, `noEmit`. None of TS7's removals (`target: es5`, `downlevelIteration`, `moduleResolution: node10`/`classic`, `module: amd/umd/system/none`, `baseUrl`-as-alias) are present. `tsconfig.json` is a two-entry project reference; TS7 supports `tsc -b` and project references. The session changes a tsconfig only if `tsc -b` errors demand it, and records any such change in `## Recorded (implementation)`. ### 3. Known TS7 edge: `skipLibCheck` no longer hides parse-level .d.ts errors -Both tsconfigs set `skipLibCheck: true`. Under TS7 that still skips type -checking of `.d.ts` files but not parse-level errors in them. If a dependency -`.d.ts` fails to parse, the fix is a dependency patch bump or an upstream -issue reference recorded in the spec — never a copied-and-edited local -`.d.ts`. +Both tsconfigs set `skipLibCheck: true`. Under TS7 that still skips type checking of `.d.ts` files but not parse-level errors in them. If a dependency `.d.ts` fails to parse, the fix is a dependency patch bump or an upstream issue reference recorded in the spec — never a copied-and-edited local `.d.ts`. ### 4. Nothing else in the toolchain consumes the TS compiler -Verified at spec time: `oxlint` (own parser), `vite`/`vitest` -(esbuild transpile, no type checking), `prettier`, and -`@vitejs/plugin-react` run no `tsc` and load no `typescript` API. The only -consumer is the `typecheck` script (`web/package.json:12`, `tsc -b`), run in -CI at `.gitea/workflows/gates.yml:113-114`. TS7 has no stable programmatic -API until 7.1; nothing in this repo needs one. +Verified at spec time: `oxlint` (own parser), `vite`/`vitest` (esbuild transpile, no type checking), `prettier`, and `@vitejs/plugin-react` run no `tsc` and load no `typescript` API. The only consumer is the `typecheck` script (`web/package.json:12`, `tsc -b`), run in CI at `.gitea/workflows/gates.yml:113-114`. TS7 has no stable programmatic API until 7.1; nothing in this repo needs one. ## Sessions @@ -65,16 +34,11 @@ One session. ### Session S1: the bump -Owns: `web/package.json`, `web/package-lock.json`, and — only if ruling 2 -forces it — `web/tsconfig.app.json`, `web/tsconfig.node.json`, -`web/tsconfig.json`. +Owns: `web/package.json`, `web/package-lock.json`, and — only if ruling 2 forces it — `web/tsconfig.app.json`, `web/tsconfig.node.json`, `web/tsconfig.json`. - S1.1 bump per ruling 1; `npm install`. -- S1.2 run, in order: `npm run format:check`, `npm run lint`, - `npm run typecheck`, `npm test`, `npm run build`, `npm run assert-bundled` - (all from `web/`). -- S1.3 `zig build test` (the licenses drift guard and the contract-samples - guard run inside it). +- S1.2 run, in order: `npm run format:check`, `npm run lint`, `npm run typecheck`, `npm test`, `npm run build`, `npm run assert-bundled` (all from `web/`). +- S1.3 `zig build test` (the licenses drift guard and the contract-samples guard run inside it). Acceptance (S1): - [ ] `web/node_modules/.bin/tsc --version` reports 7.0.x. @@ -95,14 +59,8 @@ New files: none. Deleted surface: none. ## Anti-requirements -- No other dependency bumps ride along — not oxlint, not vite, not vitest, - not `@types/*`. -- No tsconfig flag additions or "modernization" beyond what a TS7 error - forces. -- No typescript-eslint, ts-jest, or any tool needing the TS programmatic API - (unavailable until 7.1, and unwanted regardless — oxlint is the linter). -- No `tsgo` binary references anywhere: the binary is `tsc`, scripts stay as - they are. -- No edits to `web/src/**` — a source change to satisfy the new compiler is a - finding to record and fix, not silently absorb, unless `tsc -b` fails - without it; then it is recorded in `## Recorded (implementation)`. +- No other dependency bumps ride along — not oxlint, not vite, not vitest, not `@types/*`. +- No tsconfig flag additions or "modernization" beyond what a TS7 error forces. +- No typescript-eslint, ts-jest, or any tool needing the TS programmatic API (unavailable until 7.1, and unwanted regardless — oxlint is the linter). +- No `tsgo` binary references anywhere: the binary is `tsc`, scripts stay as they are. +- No edits to `web/src/**` — a source change to satisfy the new compiler is a finding to record and fix, not silently absorb, unless `tsc -b` fails without it; then it is recorded in `## Recorded (implementation)`. diff --git a/specs/milestone-23.md b/specs/milestone-23.md index 8871b25..9d2145a 100644 --- a/specs/milestone-23.md +++ b/specs/milestone-23.md @@ -1,153 +1,57 @@ # Milestone 23: StyleX and React Aria replace Tailwind -Goal: every styled element in `web/src` renders through StyleX -(`@stylexjs/unplugin` at build time); Dialog, AlertDialog, Tabs and Select -come from `react-aria-components`; Tailwind is removed. TanStack Router and -TanStack Query stay (decided 2026-08-12 after review; the router swap is out -of scope permanently, not deferred). +Goal: every styled element in `web/src` renders through StyleX (`@stylexjs/unplugin` at build time); Dialog, AlertDialog, Tabs and Select come from `react-aria-components`; Tailwind is removed. TanStack Router and TanStack Query stay (decided 2026-08-12 after review; the router swap is out of scope permanently, not deferred). -Design written 2026-08-12 against HEAD `91a0aa9`. Needs milestone 22 landed -(the StyleX and RAC `.d.ts` surfaces are checked by the TS7 `tsc -b`). +Design written 2026-08-12 against HEAD `91a0aa9`. Needs milestone 22 landed (the StyleX and RAC `.d.ts` surfaces are checked by the TS7 `tsc -b`). -Inventory this spec rests on: 522 `className=` occurrences across 31 -`web/src` files; 17 shared class constants in `web/src/ui/classes.ts`; -`web/src/styles.css` is the single line `@import "tailwindcss"` (so dark mode -is Tailwind 4's default `prefers-color-scheme` media strategy — no class -toggle exists); the Tailwind plugin sits at `web/vite.config.ts:3,8`; -`tailwindcss` and `@tailwindcss/vite` are devDependencies -(`web/package.json:34,44`) and appear in neither the runtime closure nor the -bundled set of `licenses/dependency-identity.txt`. +Inventory this spec rests on: 522 `className=` occurrences across 31 `web/src` files; 17 shared class constants in `web/src/ui/classes.ts`; `web/src/styles.css` is the single line `@import "tailwindcss"` (so dark mode is Tailwind 4's default `prefers-color-scheme` media strategy — no class toggle exists); the Tailwind plugin sits at `web/vite.config.ts:3,8`; `tailwindcss` and `@tailwindcss/vite` are devDependencies (`web/package.json:34,44`) and appear in neither the runtime closure nor the bundled set of `licenses/dependency-identity.txt`. ## Implementation contract (read first) - Read `AGENTS.md`, then this spec whole, before session work starts. -- `web/src/lib/contractSamples.gen.ts` and `web/src/lib/types.ts` are API - contract surfaces; this milestone is styling-only and touches neither. -- Every session leaves all gates green: `npm run format:check`, `npm run - lint`, `npm run typecheck`, `npm test`, `npm run build`, - `npm run assert-bundled` (from `web/`), and `zig build test`. The release - gate (`zig build dist` + `verify-dist`, `.gitea/workflows/gates.yml`) keeps - its byte budgets; they are not tight — `web/dist` is 440,924 bytes raw - against the 15 MiB binary budget — but `verify-dist` remains the arbiter. -- `assert-bundled` fails whenever the bundled npm package set changes. That - failure is the license workflow trigger, not an obstacle: review each new - package into `licenses/inventory.zon`, then record the new set in - `licenses/dependency-identity.txt` (procedure in that file's header). - Never update the recorded list without the inventory review. +- `web/src/lib/contractSamples.gen.ts` and `web/src/lib/types.ts` are API contract surfaces; this milestone is styling-only and touches neither. +- Every session leaves all gates green: `npm run format:check`, `npm run lint`, `npm run typecheck`, `npm test`, `npm run build`, `npm run assert-bundled` (from `web/`), and `zig build test`. The release gate (`zig build dist` + `verify-dist`, `.gitea/workflows/gates.yml`) keeps its byte budgets; they are not tight — `web/dist` is 440,924 bytes raw against the 15 MiB binary budget — but `verify-dist` remains the arbiter. +- `assert-bundled` fails whenever the bundled npm package set changes. That failure is the license workflow trigger, not an obstacle: review each new package into `licenses/inventory.zon`, then record the new set in `licenses/dependency-identity.txt` (procedure in that file's header). Never update the recorded list without the inventory review. ## Rulings (binding) ### 1. Build integration is `@stylexjs/unplugin`, proven before any page moves -`stylex.vite()` from `@stylexjs/unplugin` goes into `web/vite.config.ts` -alongside the existing plugins (Tailwind stays until S3). Configuration per -the official Vite guide (stylexjs.com/docs/learn/installation/vite): -`useCSSLayers: true`, `runtimeInjection: false`, `dev` keyed off the Vite -mode. The generated CSS injects into an existing CSS asset, so -`web/src/styles.css` (imported by `main.tsx:8`) remains the CSS entry for the -whole milestone; the dev/HMR virtual-module wiring from that guide is part of -S1. The unmaintained community `vite-plugin-stylex` is forbidden. S1's -acceptance proves dev server, `vitest run` and `vite build` all handle a -StyleX component before S2 or S3 convert anything — this is a gate, not an -assumption. +`stylex.vite()` from `@stylexjs/unplugin` goes into `web/vite.config.ts` alongside the existing plugins (Tailwind stays until S3). Configuration per the official Vite guide (stylexjs.com/docs/learn/installation/vite): `useCSSLayers: true`, `runtimeInjection: false`, `dev` keyed off the Vite mode. The generated CSS injects into an existing CSS asset, so `web/src/styles.css` (imported by `main.tsx:8`) remains the CSS entry for the whole milestone; the dev/HMR virtual-module wiring from that guide is part of S1. The unmaintained community `vite-plugin-stylex` is forbidden. S1's acceptance proves dev server, `vitest run` and `vite build` all handle a StyleX component before S2 or S3 convert anything — this is a gate, not an assumption. ### 2. Tokens live in `web/src/ui/tokens.stylex.ts` -`stylex.defineVars` with role names, not raw shades: surface, surface-raised, -border, border-strong, text, text-muted (the zinc ramp today), primary and -primary-text (blue-600/white), danger, danger-surface, danger-border (the red -ramp), focus (blue-600). Each var carries its dark value under -`@media (prefers-color-scheme: dark)` inside `defineVars` — same media -strategy the app has now. No theme toggle, no `data-theme` attribute. +`stylex.defineVars` with role names, not raw shades: surface, surface-raised, border, border-strong, text, text-muted (the zinc ramp today), primary and primary-text (blue-600/white), danger, danger-surface, danger-border (the red ramp), focus (blue-600). Each var carries its dark value under `@media (prefers-color-scheme: dark)` inside `defineVars` — same media strategy the app has now. No theme toggle, no `data-theme` attribute. ### 3. `web/src/ui/styles.ts` replaces `ui/classes.ts` -The 17 exports of `web/src/ui/classes.ts` (focusRing, insetFocusRing, input, -smallInput, button, smallButton, largeButton, primaryButton, -largePrimaryButton, rowButton, linkButton, dangerLinkButton, retryButton, th, -td, tableWrap, formCard) become `stylex.create` style objects in -`web/src/ui/styles.ts`, composed at call sites via -`{...stylex.props(styles.button, extra)}`. The milestone-9 accessibility -floor restated by milestone-18 ruling 10 carries over verbatim: every -interactive element renders the focus ring (`:focus-visible` outline, 2px, -offset 2, focus token; the inset variant for controls flush against panel -edges). `ui/classes.ts` is deleted in S3 when its last importer converts. +The 17 exports of `web/src/ui/classes.ts` (focusRing, insetFocusRing, input, smallInput, button, smallButton, largeButton, primaryButton, largePrimaryButton, rowButton, linkButton, dangerLinkButton, retryButton, th, td, tableWrap, formCard) become `stylex.create` style objects in `web/src/ui/styles.ts`, composed at call sites via `{...stylex.props(styles.button, extra)}`. The milestone-9 accessibility floor restated by milestone-18 ruling 10 carries over verbatim: every interactive element renders the focus ring (`:focus-visible` outline, 2px, offset 2, focus token; the inset variant for controls flush against panel edges). `ui/classes.ts` is deleted in S3 when its last importer converts. ### 4. React Aria scope is exactly four components — end state, not a phase -From `react-aria-components`: Dialog + Modal (replacing the hand-rolled -`role="dialog"` overlay in `web/src/features/clients/ClientEditDialog.tsx:27-28`), -AlertDialog composition (ruling 5), Tabs (replacing the button-pair tab -switcher in `web/src/features/local/LocalDnsPage.tsx`), Select (replacing the -7 files carrying native ``: LookupPage, QueryLogPage, RecordsTab, RulesPage, SettingsPage, PrefixesEditor, ClientEditDialog). Everything else — buttons, text inputs, tables, links, banners — stays native HTML styled with StyleX. A native styled `