mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 13 Activity

docs: unwrap hand-wrapped prose repo-wide
Gates / frontend (push) Successful in 1m2s
Details
Gates / test (push) Successful in 1m38s
Details
Gates / package (push) Successful in 5m5s
Details
Gates / test-aarch64 (push) Successful in 6m30s
Details
Gates / container (push) Successful in 15s
Details
CI / gates (push) Successful in 13m30s
Details

Browse Source
This commit is contained in:
mokhtar
2026-08-15 16:27:36 +02:00
parent 50b8fd5c61
commit 5b3d1cd65c
48 changed files with 2691 additions and 11699 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/explanation/architecture.md
+28 -154
View File
@@ -1,15 +1,8 @@
# Architecture
nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary
that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against
blocklists, and serves an embedded admin SPA over HTTP. This page maps the
source tree and explains the few design rules that hold everywhere, and why
they are the rules.
nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against blocklists, and serves an embedded admin SPA over HTTP. This page maps the source tree and explains the few design rules that hold everywhere, and why they are the rules.
For what the configuration fields, API routes and CLI flags actually are, see
[reference/configuration.md](../reference/configuration.md),
[reference/api.md](../reference/api.md) and
[reference/cli.md](../reference/cli.md). This page does not repeat them.
For what the configuration fields, API routes and CLI flags actually are, see [reference/configuration.md](../reference/configuration.md), [reference/api.md](../reference/api.md) and [reference/cli.md](../reference/cli.md). This page does not repeat them.
## Module map
@@ -39,8 +32,7 @@ Directories:
| `src/web/` | The admin HTTP layer: `server.zig` (listener), `router.zig`/`routes.zig`, one file per resource under `handlers/`, `auth.zig` (sessions), `sse.zig` (live query fanout), `static.zig` (embedded SPA), `metrics.zig` (Prometheus), `openapi.zig` (served contract), `api_limiter.zig`, `http_util.zig`. |
| `src/platform/` | OS and TLS edges: IP address values, the `std.log` sink (`logging.zig`), `statfs.zig` (free-space query via libc), client TLS over `std.crypto.tls` (`tls_client.zig`), server TLS over vendored Mbed TLS (`tls_server.zig`). |
The SPA source lives in `web/` at the repo root; the build embeds its `dist/`
output as the `web_assets` module (`-Dweb-dist`).
The SPA source lives in `web/` at the repo root; the build embeds its `dist/` output as the `web_assets` module (`-Dweb-dist`).
```
main.zig ── cli.zig ── app.zig (composition root)
@@ -58,48 +50,23 @@ main.zig ── cli.zig ── app.zig (composition root)
## The purity rule
`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no
`std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs
a timestamp takes it as a parameter — the cache, the rate limiter and the
pause flag all work this way, so every decision is testable without a backend.
`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no `std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs a timestamp takes it as a parameter — the cache, the rate limiter and the pause flag all work this way, so every decision is testable without a backend.
The point is not purity for its own sake. A decision that depends on a hidden
clock or a hidden socket can only be tested by arranging the world around it;
one that takes the clock as an argument is tested by passing a number. The
whole filtering and caching pipeline can therefore be exercised in the plain,
network-free test suite, which is what makes that suite worth gating CI on.
The point is not purity for its own sake. A decision that depends on a hidden clock or a hidden socket can only be tested by arranging the world around it; one that takes the clock as an argument is tested by passing a number. The whole filtering and caching pipeline can therefore be exercised in the plain, network-free test suite, which is what makes that suite worth gating CI on.
The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads
lists, `filter/manager.zig` owns the compiled files, the DB columns and the
snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN
resolver. Those three files are the only ones under those four directories
that take a `std.Io`. The decision path a query takes through them allocates
nothing and opens nothing.
The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads lists, `filter/manager.zig` owns the compiled files, the DB columns and the snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN resolver. Those three files are the only ones under those four directories that take a `std.Io`. The decision path a query takes through them allocates nothing and opens nothing.
The honest cost of the exceptions shows up in
[performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is
where the one production crash came from, precisely because it is the file the
pure suite cannot reach.
The honest cost of the exceptions shows up in [performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is where the one production crash came from, precisely because it is the file the pure suite cannot reach.
## std.Io injection
There is one `std.Io` in the process. `main` receives it through
`std.process.Init` — on the standard start path this is the Threaded backend
(`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to
`cli.Runner`, from which `app.zig` threads it into every collaborator as a
parameter. No module constructs its own event loop or reads an ambient clock;
tests build their own `std.Io.Threaded` instance and pass it the same way.
There is one `std.Io` in the process. `main` receives it through `std.process.Init` — on the standard start path this is the Threaded backend (`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to `cli.Runner`, from which `app.zig` threads it into every collaborator as a parameter. No module constructs its own event loop or reads an ambient clock; tests build their own `std.Io.Threaded` instance and pass it the same way.
The one deliberate exception is `storage/db.zig`: SQLite performs its own file
I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's
VFS to route through `std.Io` would be a large amount of C-boundary code to
make one dependency match a convention it does not need.
The one deliberate exception is `storage/db.zig`: SQLite performs its own file I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's VFS to route through `std.Io` would be a large amount of C-boundary code to make one dependency match a convention it does not need.
## Life of one query
The pipeline lives in `src/server/handler.zig` — `Handler.handle` does
validation and setup, then `Context.run` decides the answer. Its order is
PLAN §4; the stages below are the code's actual call chain:
The pipeline lives in `src/server/handler.zig` — `Handler.handle` does validation and setup, then `Context.run` decides the answer. Its order is PLAN §4; the stages below are the code's actual call chain:
```
UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig)
@@ -132,139 +99,46 @@ UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig)
└─► async logger ─► querylog.db
```
Local records win over forward zones, and both win over filtering: a name
nxdns answers itself never reaches a blocklist. Pause suspends filtering only;
local records, forward zones, cache, upstream and the query log keep running,
which is what makes pause safe to hand to a household member. A question whose
class is not IN bypasses local answers, filtering and the cache entirely and
goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and
declines to cache answers it does not model.
Local records win over forward zones, and both win over filtering: a name nxdns answers itself never reaches a blocklist. Pause suspends filtering only; local records, forward zones, cache, upstream and the query log keep running, which is what makes pause safe to hand to a household member. A question whose class is not IN bypasses local answers, filtering and the cache entirely and goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and declines to cache answers it does not model.
`handle` returns no error union. Every failure is either a DNS response the
client can act on or a counted drop, because there is no caller above it that
could do anything useful with a Zig error. Two consequences are worth knowing:
answers synthesized from a safe-search rewrite are never cached (the rewrite is
per group, and the cache is not), and a SERVFAIL reply short-circuits before the
query sink, so it appears in the counters but not in the query log.
`handle` returns no error union. Every failure is either a DNS response the client can act on or a counted drop, because there is no caller above it that could do anything useful with a Zig error. Two consequences are worth knowing: answers synthesized from a safe-search rewrite are never cached (the rewrite is per group, and the cache is not), and a SERVFAIL reply short-circuits before the query sink, so it appears in the counters but not in the query log.
The query path never waits on the database. `QuerySink` copies the entry, the
SSE hub gets it first, and one writer task owns the `querylog.db` handle behind
an `std.Io.Queue`. A slow disk delays logging, never resolution.
The query path never waits on the database. `QuerySink` copies the entry, the SSE hub gets it first, and one writer task owns the `querylog.db` handle behind an `std.Io.Queue`. A slow disk delays logging, never resolution.
Two smaller decisions in the same spirit: the upstream pool makes a second
pass that ignores backoff, so "every endpoint is in backoff" degrades to
trying anyway rather than to a blanket SERVFAIL; and a handler that has no
filter snapshot yet answers unfiltered rather than refusing. Both prefer a
working resolver over a correct-looking failure.
Two smaller decisions in the same spirit: the upstream pool makes a second pass that ignores backoff, so "every endpoint is in backoff" degrades to trying anyway rather than to a blanket SERVFAIL; and a handler that has no filter snapshot yet answers unfiltered rather than refusing. Both prefer a working resolver over a correct-looking failure.
## Storage
Two databases with opposite contracts, in one data directory (see
[reference/files-and-directories.md](../reference/files-and-directories.md)).
Two databases with opposite contracts, in one data directory (see [reference/files-and-directories.md](../reference/files-and-directories.md)).
**`config.db` is what the server reads.** Its schema is versioned: `migrations.zig` holds
an ordered list of steps, step 1 being the verbatim DDL from
`config_schema.zig`, each applied inside one transaction. `nxdns import`
replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed
import changes nothing; `nxdns export` renders it back as canonical ZON,
byte-identical across round trips.
**`config.db` is what the server reads.** Its schema is versioned: `migrations.zig` holds an ordered list of steps, step 1 being the verbatim DDL from `config_schema.zig`, each applied inside one transaction. `nxdns import` replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed import changes nothing; `nxdns export` renders it back as canonical ZON, byte-identical across round trips.
Which of the file and the database is *authoritative* is chosen by the
invocation, not by state: bare `nxdns run` serves the database, and
`nxdns run --config FILE` makes the file authoritative and reconciles the
database onto it at every start. `reconcile.zig` is that convergence, matching
rows by identity and writing only differences, so runtime state — blocklist
checksums, compiled snapshots, client history — survives. Why it works that way
is [configuration-model.md](configuration-model.md).
Which of the file and the database is *authoritative* is chosen by the invocation, not by state: bare `nxdns run` serves the database, and `nxdns run --config FILE` makes the file authoritative and reconciles the database onto it at every start. `reconcile.zig` is that convergence, matching rows by identity and writing only differences, so runtime state — blocklist checksums, compiled snapshots, client history — survives. Why it works that way is [configuration-model.md](configuration-model.md).
**`querylog.db` is expendable.** It is never migrated. Its schema carries a
fingerprint derived from the DDL text, and at open, a missing, corrupt,
non-database, `quick_check`-failing or fingerprint-mismatched file is moved
aside and recreated empty — the old file is kept under a new name rather than
deleted, so an operator can still look at it. Retention deletes old rows daily
and periodically rewrites the file to reclaim space.
**`querylog.db` is expendable.** It is never migrated. Its schema carries a fingerprint derived from the DDL text, and at open, a missing, corrupt, non-database, `quick_check`-failing or fingerprint-mismatched file is moved aside and recreated empty — the old file is kept under a new name rather than deleted, so an operator can still look at it. Retention deletes old rows daily and periodically rewrites the file to reclaim space.
The split exists so that the churn of the second database can never endanger
the first. Query logs are high-volume, disposable, and the thing most likely
to be corrupted by a power cut on an SD card; configuration is small,
irreplaceable, and the thing an operator would have to reconstruct by hand.
Giving them one file would force the careful contract onto the noisy data or
the loose contract onto the valuable data.
The split exists so that the churn of the second database can never endanger the first. Query logs are high-volume, disposable, and the thing most likely to be corrupted by a power cut on an SD card; configuration is small, irreplaceable, and the thing an operator would have to reconstruct by hand. Giving them one file would force the careful contract onto the noisy data or the loose contract onto the valuable data.
## Web stack
`web/server.zig` runs one `std.http.Server` per connection over its own accept
loop, with a fixed set of pre-allocated connection slots, optionally behind
TLS. Over capacity it answers 503 rather than queueing without bound — the
admin UI is not the product, and it must not be able to starve DNS.
`web/server.zig` runs one `std.http.Server` per connection over its own accept loop, with a fixed set of pre-allocated connection slots, optionally behind TLS. Over capacity it answers 503 rather than queueing without bound — the admin UI is not the product, and it must not be able to starve DNS.
The SPA is embedded at build time: `static.zig` serves the `web_assets` module
— bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where
it paid off — via a linear scan with no filesystem access at runtime. (The one
exception is `nxdns run --web-dev DIR`, which serves from disk with no cache
headers, for developing the SPA against a running server.) `GET
/api/queries/live` is server-sent events over chunked transfer, fed by the
same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`)
matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML
is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest
by tests that assert every served route appears in it.
The SPA is embedded at build time: `static.zig` serves the `web_assets` module — bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where it paid off — via a linear scan with no filesystem access at runtime. (The one exception is `nxdns run --web-dev DIR`, which serves from disk with no cache headers, for developing the SPA against a running server.) `GET /api/queries/live` is server-sent events over chunked transfer, fed by the same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`) matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest by tests that assert every served route appears in it.
Authentication (`web/auth.zig`): the operator's password is verified against
an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import
and never stored). A successful login mints a 256-bit token carried in a
cookie; the in-memory session table holds only SHA-256 digests of tokens,
compared in constant time, capped at 32 sessions with LRU eviction. Nothing is
persisted, so a restart logs everyone out — for a household LAN that is a
feature, not a gap. Unauthenticated by design: the monitoring endpoints
(health, version, metrics), the served OpenAPI contract, login itself, and the
static SPA assets, which the router hands to the SPA fallback before any auth
check. Everything else requires the cookie, and the API has its own
token-bucket rate limiter. See
[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
Authentication (`web/auth.zig`): the operator's password is verified against an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import and never stored). A successful login mints a 256-bit token carried in a cookie; the in-memory session table holds only SHA-256 digests of tokens, compared in constant time, capped at 32 sessions with LRU eviction. Nothing is persisted, so a restart logs everyone out — for a household LAN that is a feature, not a gap. Unauthenticated by design: the monitoring endpoints (health, version, metrics), the served OpenAPI contract, login itself, and the static SPA assets, which the router hands to the SPA fallback before any auth check. Everything else requires the cookie, and the API has its own token-bucket rate limiter. See [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
## DoH, DoT and certificate hot-reload
`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and
`server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server
TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as
`std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether
they are encrypted.
`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and `server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as `std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether they are encrypted.
Certificates hot-reload through `server/cert_store.zig`. One refcounted
`CertStore` per endpoint owns the published TLS context generation; listeners
`acquire` it per connection and `release` it when the connection ends, so a
reload never frees a context mid-handshake. Reload publishes nothing on
failure: both PEM files are read and a whole new context is built before
anything swaps, and any failure leaves the old generation serving. A watcher
polls mtime and size of both files every 30 seconds
(`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same
path on demand and reports the per-endpoint outcome as its payload.
Certificates hot-reload through `server/cert_store.zig`. One refcounted `CertStore` per endpoint owns the published TLS context generation; listeners `acquire` it per connection and `release` it when the connection ends, so a reload never frees a context mid-handshake. Reload publishes nothing on failure: both PEM files are read and a whole new context is built before anything swaps, and any failure leaves the old generation serving. A watcher polls mtime and size of both files every 30 seconds (`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same path on demand and reports the per-endpoint outcome as its payload.
The requirement driving all of this is that a certbot renewal must not need a
restart and must not be able to break DNS. A half-swapped context or a
free-while-in-use would do exactly that, so the store is built so neither is
representable. See
[how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md).
The requirement driving all of this is that a certbot renewal must not need a restart and must not be able to break DNS. A half-swapped context or a free-while-in-use would do exactly that, so the store is built so neither is representable. See [how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md).
## Failure visibility
Every failure mode must be visible, and the surface is counters, not log lines
(AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they
are only read after someone already suspects a problem, and on an SD card they
are a way to fill a disk.
Every failure mode must be visible, and the surface is counters, not log lines (AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they are only read after someone already suspects a problem, and on an SD card they are a way to fill a disk.
So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP,
REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and
forward-zone answers, safe-search rewrites, paused and unfiltered queries, and
the tracker-full condition. Listeners count dropped datagrams instead of
queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text
0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers
200: "degraded" is a fact about the box, not a failed request, and a monitor
that cannot distinguish the two is worse than no monitor.
So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP, REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and forward-zone answers, safe-search rewrites, paused and unfiltered queries, and the tracker-full condition. Listeners count dropped datagrams instead of queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text 0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers 200: "degraded" is a fact about the box, not a failed request, and a monitor that cannot distinguish the two is worse than no monitor.
The disk monitor classifies free space against thresholds and gates
non-essential writes; the query logger holds its batches while writes are
disallowed rather than dropping them silently or writing until the filesystem
fills. `std.log` is reserved for failures nobody else records, with
upstream-error deduplication so a flapping resolver cannot fill a disk with
identical lines.
The disk monitor classifies free space against thresholds and gates non-essential writes; the query logger holds its batches while writes are disallowed rather than dropping them silently or writing until the filesystem fills. `std.log` is reserved for failures nobody else records, with upstream-error deduplication so a flapping resolver cannot fill a disk with identical lines.
docs/explanation/configuration-model.md
+39 -165
View File
@@ -1,13 +1,8 @@
# The configuration model
nxdns is configured two ways — a ZON file and a web UI — and only one of them
can be the truth at a time. This page explains how that choice is made, what
each mode is for, and why the design leaves the fewest ways to lose an
operator's work.
nxdns is configured two ways — a ZON file and a web UI — and only one of them can be the truth at a time. This page explains how that choice is made, what each mode is for, and why the design leaves the fewest ways to lose an operator's work.
For the fields themselves see
[reference/configuration.md](../reference/configuration.md); for the commands
and their exit codes see [reference/cli.md](../reference/cli.md).
For the fields themselves see [reference/configuration.md](../reference/configuration.md); for the commands and their exit codes see [reference/cli.md](../reference/cli.md).
## The rule
@@ -18,37 +13,19 @@ nxdns run the database is the truth
nxdns run --config /etc/nxdns/config.zon the file is the truth
```
That is the entire selection mechanism. There is no mode setting, no default
file path, and nothing recorded in the database about which mode last wrote it.
A `config.zon` that exists but that no invocation names changes nothing at all.
That is the entire selection mechanism. There is no mode setting, no default file path, and nothing recorded in the database about which mode last wrote it. A `config.zon` that exists but that no invocation names changes nothing at all.
Two properties fall out of that, and both were chosen on purpose.
**An operator can read `ExecStart` and know which authority is live.** The
alternative — probe a well-known path, and behave differently depending on
whether a file happens to be there — is ambient magic. It is also the exact
class of rule that produced years of documentation lies in this project: the
old design read the file only while the database was empty, which meant the same
command did two different things depending on state nobody could see from the
command line, and every page that described it eventually described it wrongly.
**An operator can read `ExecStart` and know which authority is live.** The alternative — probe a well-known path, and behave differently depending on whether a file happens to be there — is ambient magic. It is also the exact class of rule that produced years of documentation lies in this project: the old design read the file only while the database was empty, which meant the same command did two different things depending on state nobody could see from the command line, and every page that described it eventually described it wrongly.
**A path already expresses a two-state choice, so a mode flag beside it would
be redundant and worse.** An earlier draft had `--config-source=db|file`. A mode
flag next to a path flag manufactures combinations that cannot mean anything —
a path with no mode, a mode with no path — and each one then needs a pairing
rule and a usage error to defend it. Presence-of-path has no invalid
combinations, so there is nothing to defend.
**A path already expresses a two-state choice, so a mode flag beside it would be redundant and worse.** An earlier draft had `--config-source=db|file`. A mode flag next to a path flag manufactures combinations that cannot mean anything — a path with no mode, a mode with no path — and each one then needs a pairing rule and a usage error to defend it. Presence-of-path has no invalid combinations, so there is nothing to defend.
## Database mode
`nxdns run`. `config.db` holds the configuration; the UI, the API and `nxdns
import` write to it; nothing reads a file. This is the appliance: someone sets
the box up once, and afterwards the household member who wants to unblock one
domain clicks a button.
`nxdns run`. `config.db` holds the configuration; the UI, the API and `nxdns import` write to it; nothing reads a file. This is the appliance: someone sets the box up once, and afterwards the household member who wants to unblock one domain clicks a button.
A fresh install in this mode starts from an empty database, which fails
validation on its own terms — there is nowhere to forward a query to — and says
what to do about it:
A fresh install in this mode starts from an empty database, which fails validation on its own terms — there is nowhere to forward a query to — and says what to do about it:
```
nxdns run failed: NoUsableUpstreams
@@ -58,188 +35,85 @@ load one with `nxdns import <file>`, or make a file the source of truth with `nx
## File mode
`nxdns run --config FILE`. The file is the sole declarative source, and the
database becomes the runtime substrate: every start reads the file, validates
it, converges the database onto it, and serves from there. Configuration writes
through the API are refused with a 403.
`nxdns run --config FILE`. The file is the sole declarative source, and the database becomes the runtime substrate: every start reads the file, validates it, converges the database onto it, and serves from there. Configuration writes through the API are refused with a 403.
This is the mode for a file kept in git and pushed by Ansible. What it buys is
that the deployed file is what is running — not "was imported once", not
"was imported unless someone clicked something since".
This is the mode for a file kept in git and pushed by Ansible. What it buys is that the deployed file is what is running — not "was imported once", not "was imported unless someone clicked something since".
Three properties make it usable rather than merely correct.
**It fails closed.** A file that is missing, unreadable, unparseable, oversized
or invalid stops the start. nxdns never falls back to the database, because a
fallback turns a deploy typo into a configuration that is silently months old
and looks fine. That failure is exit 2, so `nxdns check --config FILE` is a real
pre-restart gate: validate the pushed file in the handler, and a typo is a
failed deploy at noon rather than a dead resolver at the next power cut.
**It fails closed.** A file that is missing, unreadable, unparseable, oversized or invalid stops the start. nxdns never falls back to the database, because a fallback turns a deploy typo into a configuration that is silently months old and looks fine. That failure is exit 2, so `nxdns check --config FILE` is a real pre-restart gate: validate the pushed file in the handler, and a typo is a failed deploy at noon rather than a dead resolver at the next power cut.
**It converges rather than replaces.** Reconciling matches rows by identity and
writes only what differs. A source whose URL has not changed keeps its row id,
its checksum, its counters and its compiled blocklist files — so a restart in
file mode downloads nothing, which is the difference between a design that is
tolerable to restart and one that costs three minutes and 100 MB every time.
**It converges rather than replaces.** Reconciling matches rows by identity and writes only what differs. A source whose URL has not changed keeps its row id, its checksum, its counters and its compiled blocklist files — so a restart in file mode downloads nothing, which is the difference between a design that is tolerable to restart and one that costs three minutes and 100 MB every time.
**An unchanged file writes nothing at all.** Not "writes the same bytes" —
performs zero write statements, and reports it:
**An unchanged file writes nothing at all.** Not "writes the same bytes" — performs zero write statements, and reports it:
```
reconciled '/etc/nxdns/config.zon': no changes
```
That matters beyond elegance. A box whose SD card is full of query log can still
restart in file mode, because a no-op reconcile needs no write-ahead-log
headroom.
That matters beyond elegance. A box whose SD card is full of query log can still restart in file mode, because a no-op reconcile needs no write-ahead-log headroom.
**Converged at every boot is not a lock between boots.** Nothing stops `nxdns
import` or a `runtime action` route from moving the database while the server
runs. The contract is that the next start puts it back, and says what it
corrected.
**Converged at every boot is not a lock between boots.** Nothing stops `nxdns import` or a `runtime action` route from moving the database while the server runs. The contract is that the next start puts it back, and says what it corrected.
## What the file cannot take away
The file is authoritative over configuration. It is not authoritative over
things it has no vocabulary for, and reconciling has to preserve those or the
mode is unusable.
The file is authoritative over configuration. It is not authoritative over things it has no vocabulary for, and reconciling has to preserve those or the mode is unusable.
- **Blocklist download state.** Checksums, fetch timestamps and domain counts
belong to the network, not the operator. They survive on every matched row.
- **Client history.** Devices nxdns saw on the wire are kept whole. Naming one
in the file promotes that row in place — it keeps its first-seen and
last-seen and its row id, and counts as an update rather than a delete and an
insert.
- **Devices whose group is un-declared.** Remove a group from the file and the
observed clients assigned to it move to `default`. The operator un-declared
the group, not the devices.
- **The password, when the file does not mention it.** See
[the password](#the-password).
- **Blocklist download state.** Checksums, fetch timestamps and domain counts belong to the network, not the operator. They survive on every matched row.
- **Client history.** Devices nxdns saw on the wire are kept whole. Naming one in the file promotes that row in place — it keeps its first-seen and last-seen and its row id, and counts as an update rather than a delete and an insert.
- **Devices whose group is un-declared.** Remove a group from the file and the observed clients assigned to it move to `default`. The operator un-declared the group, not the devices.
- **The password, when the file does not mention it.** See [the password](#the-password).
The one thing identity cannot survive is a change to identity itself. Edit a
source's URL and the engine sees one row gone and one row arrived: new id, fresh
download, and the old compiled files swept. That is consistent — artifacts are
keyed by row id — and it is why the CLI asks for `--allow-delete` when a diff
deletes anything.
The one thing identity cannot survive is a change to identity itself. Edit a source's URL and the engine sees one row gone and one row arrived: new id, fresh download, and the old compiled files swept. That is consistent — artifacts are keyed by row id — and it is why the CLI asks for `--allow-delete` when a diff deletes anything.
## Why the database is the substrate in both modes
Even in file mode the database is where the server reads its effective
configuration from. That is not a leftover; it is what lets one read path serve
both modes, and it is what makes the runtime state above have somewhere to live.
Even in file mode the database is where the server reads its effective configuration from. That is not a leftover; it is what lets one read path serve both modes, and it is what makes the runtime state above have somewhere to live.
If the file were read directly on every query path there would be no place to
keep a checksum, and no way for the UI to show anything. If both file and
database were authoritative there would be a two-way merge, and a merge cannot
work here: when the UI writes a rule and the file still says otherwise, nothing
in the system knows which of two statements is the newer intention. Every
resolution silently destroys work someone meant to keep.
If the file were read directly on every query path there would be no place to keep a checksum, and no way for the UI to show anything. If both file and database were authoritative there would be a two-way merge, and a merge cannot work here: when the UI writes a rule and the file still says otherwise, nothing in the system knows which of two statements is the newer intention. Every resolution silently destroys work someone meant to keep.
So the modes are exclusive, and the failure mode of each is loud. In database
mode, editing the file does nothing, which you find out the first time you look
at the UI. In file mode, the UI refuses the edit to your face with a message
naming the file to edit instead.
So the modes are exclusive, and the failure mode of each is loud. In database mode, editing the file does nothing, which you find out the first time you look at the UI. In file mode, the UI refuses the edit to your face with a message naming the file to edit instead.
## The round trip
The file stays useful as an editing surface in both modes — text is diffable,
reviewable and easy to back up — because `export` renders the database into the
same shape `import` and file mode read:
The file stays useful as an editing surface in both modes — text is diffable, reviewable and easy to back up — because `export` renders the database into the same shape `import` and file mode read:
```
nxdns export → canonical ZON → edit → nxdns import → config.db
```
`nxdns export` writes canonical ZON: a fixed two-line header, every default
emitted, deterministic ordering from the model's field order and the
repositories' `ORDER BY` clauses, and no timestamps or hostnames anywhere.
Runtime columns are absent from the configuration model on purpose, so two
exports taken from a live, busy server are identical. `export → import →
export` is byte-identical, and a test asserts it.
`nxdns export` writes canonical ZON: a fixed two-line header, every default emitted, deterministic ordering from the model's field order and the repositories' `ORDER BY` clauses, and no timestamps or hostnames anywhere. Runtime columns are absent from the configuration model on purpose, so two exports taken from a live, busy server are identical. `export → import → export` is byte-identical, and a test asserts it.
Byte-stability is not cosmetic. It is what makes an exported file usable in
version control and what makes a diff of two exports mean something: any
difference is a configuration change, never noise from when the export ran.
Byte-stability is not cosmetic. It is what makes an exported file usable in version control and what makes a diff of two exports mean something: any difference is a configuration change, never noise from when the export ran.
The output is also identical in both modes — nothing marks a file as coming
from a file-mode box. That is deliberate, because it is what makes export the
adoption tool: the file you check is the file you deploy, byte for byte. The
label an operator wants is in the unit file, where they put it.
The output is also identical in both modes — nothing marks a file as coming from a file-mode box. That is deliberate, because it is what makes export the adoption tool: the file you check is the file you deploy, byte for byte. The label an operator wants is in the unit file, where they put it.
Reconciling the same file twice produces a byte-identical database — ids,
checksums, `created_at`, the password hash, the whole settings table — including
when the file is written in a non-canonical but equivalent form, such as
`FD00:0:0:0:0:0:0:1` for an address stored as `fd00::1`. Anything that churns
under an unchanged file is a bug in the engine by definition. That single
invariant is what forces most of the design above: matching on canonical forms,
writing only on difference, treating duplicate rule tuples as a multiset, and
verifying a password rather than re-hashing it.
Reconciling the same file twice produces a byte-identical database — ids, checksums, `created_at`, the password hash, the whole settings table — including when the file is written in a non-canonical but equivalent form, such as `FD00:0:0:0:0:0:0:1` for an address stored as `fd00::1`. Anything that churns under an unchanged file is a bug in the engine by definition. That single invariant is what forces most of the design above: matching on canonical forms, writing only on difference, treating duplicate rule tuples as a multiset, and verifying a password rather than re-hashing it.
## Unknown keys, and why the asymmetry is deliberate
An unknown key in the **database** is warned about and ignored. An unknown key
in the **file** is a hard error.
An unknown key in the **database** is warned about and ignored. An unknown key in the **file** is a hard error.
They are different situations. A settings row the running binary does not
recognise is almost always an older binary reading a database written by a newer
one — a downgrade, or a rollback after a bad upgrade. Refusing to start there
would mean a downgrade bricks the config database, and the operator would have
to hand-edit SQLite to recover. Warning and ignoring means the downgrade works,
the unknown setting sits inert, and the upgrade back picks it up again.
They are different situations. A settings row the running binary does not recognise is almost always an older binary reading a database written by a newer one — a downgrade, or a rollback after a bad upgrade. Refusing to start there would mean a downgrade bricks the config database, and the operator would have to hand-edit SQLite to recover. Warning and ignoring means the downgrade works, the unknown setting sits inert, and the upgrade back picks it up again.
A key in a file, by contrast, is something a human just typed. The likeliest
cause is a typo, and the second likeliest is a field that no longer exists.
Silently ignoring it would mean the setting the operator believes they applied
was never applied — exactly the silent-divergence failure the whole model is
built to avoid. So the ZON parser rejects unknown fields with a line and column.
A key in a file, by contrast, is something a human just typed. The likeliest cause is a typo, and the second likeliest is a field that no longer exists. Silently ignoring it would mean the setting the operator believes they applied was never applied — exactly the silent-divergence failure the whole model is built to avoid. So the ZON parser rejects unknown fields with a line and column.
The tolerance has a boundary worth stating plainly: it covers unknown *keys*,
not unparseable *values*. A known key whose stored text does not decode into its
type is an error, not a warning.
The tolerance has a boundary worth stating plainly: it covers unknown *keys*, not unparseable *values*. A known key whose stored text does not decode into its type is an error, not a warning.
## The password
`web.password` is a write-only input. It is never a stored value. A non-empty
one is hashed with argon2id (PHC encoding, OWASP argon2id parameters) into
`web.password_hash`, and the plaintext is cleared before anything is written.
There is no settings row that can hold it: the model skips `web.password` in
both directions of the settings bridge, so the plaintext has nowhere to go even
by accident.
`web.password` is a write-only input. It is never a stored value. A non-empty one is hashed with argon2id (PHC encoding, OWASP argon2id parameters) into `web.password_hash`, and the plaintext is cleared before anything is written. There is no settings row that can hold it: the model skips `web.password` in both directions of the settings bridge, so the plaintext has nowhere to go even by accident.
Both fields are optional, and this is the one deliberate carve-out from
file-as-sole-truth: **a file that mentions neither leaves the stored hash
alone.**
Both fields are optional, and this is the one deliberate carve-out from file-as-sole-truth: **a file that mentions neither leaves the stored hash alone.**
The reason is a trap the design walked into once. An export carries the full PHC
string, which is long and ugly, and an operator committing that file to git will
sooner or later delete the line — meaning "keep the current password". If
absence meant "no password", that edit would reconcile an empty hash over the
stored one and open the admin UI to the entire LAN, silently, because
authentication is on exactly when the hash is non-empty. Silence has to mean
keep. Disabling authentication takes the explicit `password_hash = ""`.
The reason is a trap the design walked into once. An export carries the full PHC string, which is long and ugly, and an operator committing that file to git will sooner or later delete the line — meaning "keep the current password". If absence meant "no password", that edit would reconcile an empty hash over the stored one and open the admin UI to the entire LAN, silently, because authentication is on exactly when the hash is non-empty. Silence has to mean keep. Disabling authentication takes the explicit `password_hash = ""`.
The other end of the same problem is `password = ""`. Hashing the empty string
produces a perfectly valid hash, so authentication would be *on* — while the
login handler refuses every empty password, so it could never be satisfied.
Auth on and unreachable is worse than either alternative, so that file is
refused at validation with a diagnostic naming the remedy.
The other end of the same problem is `password = ""`. Hashing the empty string produces a perfectly valid hash, so authentication would be *on* — while the login handler refuses every empty password, so it could never be satisfied. Auth on and unreachable is worse than either alternative, so that file is refused at validation with a diagnostic naming the remedy.
A plaintext password that has not changed is verified against the stored hash
and kept rather than re-hashed. That is byte-stability, not a saving:
verification recomputes the same argon2id function with the stored salt and
costs exactly what hashing costs. Hashing unconditionally would generate a fresh
salt on every start and break the invariant above.
A plaintext password that has not changed is verified against the stored hash and kept rather than re-hashed. That is byte-stability, not a saving: verification recomputes the same argon2id function with the stored salt and costs exactly what hashing costs. Hashing unconditionally would generate a fresh salt on every start and break the invariant above.
Export's canonical form is therefore `password = null` beside the stored
`password_hash`. Writing an empty *string* there instead would make every export
carry a present-but-empty password next to a hash — tripping the both-set rule
on re-import, so export's own output would fail export's own contract.
Export's canonical form is therefore `password = null` beside the stored `password_hash`. Writing an empty *string* there instead would make every export carry a present-but-empty password next to a hash — tripping the both-set rule on re-import, so export's own output would fail export's own contract.
## What is not configuration
Storage paths are process arguments, not configuration fields: `--data-dir`,
`--config`, `--web-dev`. They cannot live in the file, because the file is found
by way of them — a path that told you where to find the thing that told you the
path would be circular. They are also the settings a supervisor (systemd,
Docker) owns rather than the operator's policy about DNS. See
[reference/files-and-directories.md](../reference/files-and-directories.md).
Storage paths are process arguments, not configuration fields: `--data-dir`, `--config`, `--web-dev`. They cannot live in the file, because the file is found by way of them — a path that told you where to find the thing that told you the path would be circular. They are also the settings a supervisor (systemd, Docker) owns rather than the operator's policy about DNS. See [reference/files-and-directories.md](../reference/files-and-directories.md).
docs/explanation/performance-and-testing.md
+32 -158
View File
@@ -1,12 +1,8 @@
# Performance targets and what the tests prove
Two related questions: why the performance numbers are the numbers, and why CI
does not enforce them — and then, less comfortably, what a green test suite
here does and does not tell you.
Two related questions: why the performance numbers are the numbers, and why CI does not enforce them — and then, less comfortably, what a green test suite here does and does not tell you.
For the targets and the measured results as data, see
[reference/performance.md](../reference/performance.md); to run the bench
yourself, [how-to/measure-performance.md](../how-to/measure-performance.md).
For the targets and the measured results as data, see [reference/performance.md](../reference/performance.md); to run the bench yourself, [how-to/measure-performance.md](../how-to/measure-performance.md).
## Where the targets come from
@@ -16,197 +12,75 @@ PLAN §18 sets five:
- blocklist lookup p95 < 1 ms;
- cached response p95 < 5 ms;
- memory with ~1M blocked domains < 100 MiB;
- stripped static binary ≤ 10,485,760 bytes per arch, ≤ 15,728,640 bytes with
the embedded frontend.
- stripped static binary ≤ 10,485,760 bytes per arch, ≤ 15,728,640 bytes with the embedded frontend.
They are household-scale numbers, and they are deliberately unambitious. 100
qps is far more than a house generates; the point of the target is not speed
but that a Pi 5 with an SD card never becomes the reason the internet feels
broken. The latency targets exist for the same reason: DNS sits in front of
every connection anyone makes, so the failure people notice is not throughput
but a stall. The memory target is what keeps a 1M-entry blocklist from
competing with everything else on a 4 GB board. The binary-size target is
about what a static single-binary deployment is for — if it does not fit on a
constrained box and copy over a slow link in one step, the packaging decision
has not paid for itself.
They are household-scale numbers, and they are deliberately unambitious. 100 qps is far more than a house generates; the point of the target is not speed but that a Pi 5 with an SD card never becomes the reason the internet feels broken. The latency targets exist for the same reason: DNS sits in front of every connection anyone makes, so the failure people notice is not throughput but a stall. The memory target is what keeps a 1M-entry blocklist from competing with everything else on a 4 GB board. The binary-size target is about what a static single-binary deployment is for — if it does not fit on a constrained box and copy over a slow link in one step, the packaging decision has not paid for itself.
`tools/bench.zig` (`zig build bench`) measures the three that are measurable
in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry
snapshot), `cache` (key build plus cache get plus id patch), and `compile`
(the blocklist compiler over a 1M-line body, informational — there is no §18
target for it because no prior datapoint exists). Memory comes from
`/proc/self/status` VmRSS. The qps target is not in the harness at all: it is
end-to-end against the real binary with a DNS load generator, because a
harness number for "queries per second" would measure the harness.
`tools/bench.zig` (`zig build bench`) measures the three that are measurable in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry snapshot), `cache` (key build plus cache get plus id patch), and `compile` (the blocklist compiler over a 1M-line body, informational — there is no §18 target for it because no prior datapoint exists). Memory comes from `/proc/self/status` VmRSS. The qps target is not in the harness at all: it is end-to-end against the real binary with a DNS load generator, because a harness number for "queries per second" would measure the harness.
The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product,
and `src/tests.zig` aggregates everything shippable.
The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product, and `src/tests.zig` aggregates everything shippable.
## Why CI does not gate on performance
Required CI stays deterministic (AGENTS.md). Latency assertions on shared
runners measure the runner's noisy neighbours; the same commit passes and
fails depending on what else the host is doing. A gate that flakes does not
protect anything — it trains people to re-run the job, and once re-running is
routine, a real regression gets re-run too. The flaky gate is worse than no
gate, because it also consumes the attention a real gate would need.
Required CI stays deterministic (AGENTS.md). Latency assertions on shared runners measure the runner's noisy neighbours; the same commit passes and fails depending on what else the host is doing. A gate that flakes does not protect anything — it trains people to re-run the job, and once re-running is routine, a real regression gets re-run too. The flaky gate is worse than no gate, because it also consumes the attention a real gate would need.
So the bench defaults to informational, and `--assert` — which exits non-zero
on a missed target — exists for hardware you control. Run it on the Pi, where
the numbers describe the machine the software actually has to run on. The
x86_64 development-host numbers in
[reference/performance.md](../reference/performance.md) are a regression
baseline for the machine development happens on, not a claim about the target
platform; a Cortex-A76 is far slower and those numbers do not transfer.
So the bench defaults to informational, and `--assert` — which exits non-zero on a missed target — exists for hardware you control. Run it on the Pi, where the numbers describe the machine the software actually has to run on. The x86_64 development-host numbers in [reference/performance.md](../reference/performance.md) are a regression baseline for the machine development happens on, not a claim about the target platform; a Cortex-A76 is far slower and those numbers do not transfer.
CI does gate on the one performance property that *is* deterministic: binary
size. The `package` job builds the release artifacts and `zig build verify-dist`
asserts both §18 budgets against them. Size is a function of the input, not of
the runner's mood, so it is exactly the kind of thing a shared runner can
measure honestly.
CI does gate on the one performance property that *is* deterministic: binary size. The `package` job builds the release artifacts and `zig build verify-dist` asserts both §18 budgets against them. Size is a function of the input, not of the runner's mood, so it is exactly the kind of thing a shared runner can measure honestly.
The budgets are asserted as exact byte counts, and the asset-free budget gets
its own build against a generated empty assets directory rather than against
`web/dist-placeholder`. The placeholder is not buildable by `dist` at all —
that is the guard against a release shipping a stub admin page — and letting it
back in through a size check would have defeated the guard for the sake of one
number.
The budgets are asserted as exact byte counts, and the asset-free budget gets its own build against a generated empty assets directory rather than against `web/dist-placeholder`. The placeholder is not buildable by `dist` at all — that is the guard against a release shipping a stub admin page — and letting it back in through a size check would have defeated the guard for the sake of one number.
## What the test suite is
Every blocking check lives in `.gitea/workflows/gates.yml`, which is a
`workflow_call` workflow with nothing in it but jobs. `ci.yml` calls it on push
and pull request for `master`, and `release.yml` calls it before it builds
anything publishable. That shape exists for one reason: a check that lived in
`ci.yml` alone would be a check a release could skip.
Every blocking check lives in `.gitea/workflows/gates.yml`, which is a `workflow_call` workflow with nothing in it but jobs. `ci.yml` calls it on push and pull request for `master`, and `release.yml` calls it before it builds anything publishable. That shape exists for one reason: a check that lived in `ci.yml` alone would be a check a release could skip.
Five jobs, all required:
- `test` — the Zig suite with `-Dintegration`.
- `test-aarch64` — the same suite cross-built for aarch64 and executed under
qemu-user, plain tier only.
- `test-aarch64` — the same suite cross-built for aarch64 and executed under qemu-user, plain tier only.
- `frontend` — format, lint, typecheck, the vitest cases, build.
- `package` — `zig build dist` and `zig build verify-dist`, which is where the
size budgets, the ELF static-linkage assert and the archive layout checks
are.
- `container` — builds the image, asserts the binary inside it is byte-identical
to the one in the matching tarball, and smoke-tests it by booting the
container and polling `/api/health`.
- `package` — `zig build dist` and `zig build verify-dist`, which is where the size budgets, the ELF static-linkage assert and the archive layout checks are.
- `container` — builds the image, asserts the binary inside it is byte-identical to the one in the matching tarball, and smoke-tests it by booting the container and polling `/api/health`.
The Zig suite has three tiers, gated by build flags:
- **plain `zig build test`** — pure logic. No sockets, no threads, no clock
budgets. This is the tier the purity rule
([architecture.md](architecture.md)) exists to make possible.
- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:`
databases, temp directories. Nothing leaves the host.
- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT
handshakes against real resolvers). Four tests, and they run in a
manual-dispatch workflow, never on push or pull request.
- **plain `zig build test`** — pure logic. No sockets, no threads, no clock budgets. This is the tier the purity rule ([architecture.md](architecture.md)) exists to make possible.
- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:` databases, temp directories. Nothing leaves the host.
- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT handshakes against real resolvers). Four tests, and they run in a manual-dispatch workflow, never on push or pull request.
The aarch64 job runs the plain tier only. The integration tests are
multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown
turns those budgets into a flake source — the same reasoning that keeps the
bench out of CI. What aarch64 needs to prove is portable correctness of the
DNS, filter and cache logic, and the plain tier is exactly that.
The aarch64 job runs the plain tier only. The integration tests are multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown turns those budgets into a flake source — the same reasoning that keeps the bench out of CI. What aarch64 needs to prove is portable correctness of the DNS, filter and cache logic, and the plain tier is exactly that.
At the time of writing, plain `zig build test` is 1175 of 1288 passing with
113 skipped and 0 failed, the skips being the integration-gated tests.
Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with
`-Dintegration` (the 4 skips are the live-network tests) and 1159 passing
under qemu, 0 failed in each.
At the time of writing, plain `zig build test` is 1175 of 1288 passing with 113 skipped and 0 failed, the skips being the integration-gated tests. Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with `-Dintegration` (the 4 skips are the live-network tests) and 1159 passing under qemu, 0 failed in each.
## What it does not prove
The suite is hermetic by design. That is the right default: it is fast, it is
deterministic, it can gate merges. But hermetic and correct are different
properties, and the gap has already cost this project twice.
The suite is hermetic by design. That is the right default: it is fast, it is deterministic, it can gate merges. But hermetic and correct are different properties, and the gap has already cost this project twice.
**The blocklist download aborted the process on first real use.** The fetcher
constructed the HTTP response reader over `transfer_buf` and then read *into*
that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's
already-buffered bytes into the caller's destination — so source and
destination were the same allocation, and Zig's `@memcpy` requires them not to
overlap. It aborts.
**The blocklist download aborted the process on first real use.** The fetcher constructed the HTTP response reader over `transfer_buf` and then read *into* that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's already-buffered bytes into the caller's destination — so source and destination were the same allocation, and Zig's `@memcpy` requires them not to overlap. It aborts.
The reason no test caught it is precise and instructive. The copy length is
zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is
fine. Bytes only accumulate in the reader's own buffer when a read comes back
short of filling the destination and the loop goes round again — that is, when
the body arrives in more than one stream call. The loopback fixture answers
every request with one small in-memory body that lands in a single read, and
the one over-size test never streams a byte, because the fetcher refuses an
oversized `content-length` on the response head. Every test in the suite was
on the zero-length-memcpy side of the branch. The first real download — a
multi-megabyte list over TLS across the WAN, arriving in many TCP segments —
was on the other side, and took the process down. The fix (commit 35f2324)
streams the body straight into the caller's writer, so the reader's buffer is
never a destination slice, and it came with four regression tests that put a
fully-buffered reader into exactly the state the old code could not survive.
The reason no test caught it is precise and instructive. The copy length is zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is fine. Bytes only accumulate in the reader's own buffer when a read comes back short of filling the destination and the loop goes round again — that is, when the body arrives in more than one stream call. The loopback fixture answers every request with one small in-memory body that lands in a single read, and the one over-size test never streams a byte, because the fetcher refuses an oversized `content-length` on the response head. Every test in the suite was on the zero-length-memcpy side of the branch. The first real download — a multi-megabyte list over TLS across the WAN, arriving in many TCP segments — was on the other side, and took the process down. The fix (commit 35f2324) streams the body straight into the caller's writer, so the reader's buffer is never a destination slice, and it came with four regression tests that put a fully-buffered reader into exactly the state the old code could not survive.
**A stale embedded SPA bundle shipped a settings page that crashed on load,
while 121 web tests passed.** `web/dist/` is gitignored and
`-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The
frontend tests ran against the sources, in jsdom, and were green; the binary
carried an older build. The tests were testing something the artifact did not
contain.
**A stale embedded SPA bundle shipped a settings page that crashed on load, while 121 web tests passed.** `web/dist/` is gitignored and `-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The frontend tests ran against the sources, in jsdom, and were green; the binary carried an older build. The tests were testing something the artifact did not contain.
Note what these two have in common. Neither was a logic bug that a better unit
test would have caught. One lived in the seam between the pure core and its
one I/O edge; the other lived in the seam between two build systems. Hermetic
tests are constructed to exclude exactly those seams — that is what makes them
hermetic.
Note what these two have in common. Neither was a logic bug that a better unit test would have caught. One lived in the seam between the pure core and its one I/O edge; the other lived in the seam between two build systems. Hermetic tests are constructed to exclude exactly those seams — that is what makes them hermetic.
## The lesson, and where it now lives
Green hermetic tests are a floor, not a ceiling. They prove the logic is
consistent with itself. They cannot prove the program works, because the
things they deliberately exclude — real network reads, real TLS, real file
sizes, real build artifacts — are where a program meets reality.
Green hermetic tests are a floor, not a ceiling. They prove the logic is consistent with itself. They cannot prove the program works, because the things they deliberately exclude — real network reads, real TLS, real file sizes, real build artifacts — are where a program meets reality.
The response is not to make CI non-deterministic. It is to require that the
real paths get exercised by a human before work is called done. That is now
ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and
the how-to pages is executed verbatim, on the host, by the session that writes
it, and a command that cannot run there is marked in the page as unverified
with the reason. Documentation written from source-reading alone is how both
of these shipped; documentation that has been run is a second, independent
test suite that exercises precisely the paths the hermetic one skips.
The response is not to make CI non-deterministic. It is to require that the real paths get exercised by a human before work is called done. That is now ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and the how-to pages is executed verbatim, on the host, by the session that writes it, and a command that cannot run there is marked in the page as unverified with the reason. Documentation written from source-reading alone is how both of these shipped; documentation that has been run is a second, independent test suite that exercises precisely the paths the hermetic one skips.
Two honest gaps remain, stated so nobody has to rediscover them:
- Nothing in the suite drives a multi-read HTTP body through the fetcher end
to end. The regression tests cover `pumpBody` directly over a pre-buffered
reader; the loopback fixture still sends one small body per connection.
- There is no freshness check on `web/dist`. CI cannot embed a stale bundle,
because the jobs that pass `-Dweb-dist` rebuild the frontend immediately
beforehand. A local build can, and will do it without a warning.
- Nothing in the suite drives a multi-read HTTP body through the fetcher end to end. The regression tests cover `pumpBody` directly over a pre-buffered reader; the loopback fixture still sends one small body per connection.
- There is no freshness check on `web/dist`. CI cannot embed a stale bundle, because the jobs that pass `-Dweb-dist` rebuild the frontend immediately beforehand. A local build can, and will do it without a warning.
## What a signed release does not prove either
The same distinction applies one level out, to the artifacts. A release is
signed, and the signature is worth having: it says the artifact came from this
project's pipeline and reached you unaltered. It does not say the binary was
built from the source in this repository, because the machine that ran the
build also held the signing key. An attacker with that machine produces
something that verifies cleanly and contains whatever they put in it.
The same distinction applies one level out, to the artifacts. A release is signed, and the signature is worth having: it says the artifact came from this project's pipeline and reached you unaltered. It does not say the binary was built from the source in this repository, because the machine that ran the build also held the signing key. An attacker with that machine produces something that verifies cleanly and contains whatever they put in it.
The control that closes that gap is a reproducibility gate — an independent
build, in a different directory on a different machine, landing on the same
bytes. It does not exist. It is a recorded deferral (`specs/milestone-14.md`
ruling 12), not something nobody thought of, and until it exists no document
here describes the build as reproducible: nobody has measured whether it is.
The cheap inputs to reproducibility are already in place — `gzip -n`,
`--mtime=@0`, `LC_ALL=C`, `TZ=UTC`, exact Zig and Node pins — which makes the
gate cheap to add later and proves nothing on its own.
The control that closes that gap is a reproducibility gate — an independent build, in a different directory on a different machine, landing on the same bytes. It does not exist. It is a recorded deferral (`specs/milestone-14.md` ruling 12), not something nobody thought of, and until it exists no document here describes the build as reproducible: nobody has measured whether it is. The cheap inputs to reproducibility are already in place — `gzip -n`, `--mtime=@0`, `LC_ALL=C`, `TZ=UTC`, exact Zig and Node pins — which makes the gate cheap to add later and proves nothing on its own.
What the release pipeline is required to hold to is narrower: two runs of
`zig build dist` on the same commit **in the same directory** produce
byte-identical tarballs. Same-directory determinism is a much weaker property
than reproducibility, and conflating the two is exactly the kind of claim this
page exists to refuse.
What the release pipeline is required to hold to is narrower: two runs of `zig build dist` on the same commit **in the same directory** produce byte-identical tarballs. Same-directory determinism is a much weaker property than reproducibility, and conflating the two is exactly the kind of claim this page exists to refuse.
[Verify a release](../how-to/verify-a-release.md) states the same limits where
an operator will actually meet them, and gives the rebuild-and-compare recipe
with the caveat that a differing hash is not evidence of tampering while this
gap is open.
[Verify a release](../how-to/verify-a-release.md) states the same limits where an operator will actually meet them, and gives the rebuild-and-compare recipe with the caveat that a differing hash is not evidence of tampering while this gap is open.