milestone 13: restructure docs to diataxis, tutorial, every command executed
This commit is contained in:
@@ -0,0 +1,264 @@
|
||||
# Architecture
|
||||
|
||||
nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary
|
||||
that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against
|
||||
blocklists, and serves an embedded admin SPA over HTTP. This page maps the
|
||||
source tree and explains the few design rules that hold everywhere, and why
|
||||
they are the rules.
|
||||
|
||||
For what the configuration fields, API routes and CLI flags actually are, see
|
||||
[reference/configuration.md](../reference/configuration.md),
|
||||
[reference/api.md](../reference/api.md) and
|
||||
[reference/cli.md](../reference/cli.md). This page does not repeat them.
|
||||
|
||||
## Module map
|
||||
|
||||
Top-level files:
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `src/main.zig` | Process shell: writers, argv, dispatch, exit code. |
|
||||
| `src/cli.zig` | Every command body (`run`, `check`, `export`, `import`, `version`, `help`); takes its writers as parameters so tests capture output without a process. |
|
||||
| `src/app.zig` | The composition root: everything `nxdns run` owns, built in order. Nothing else constructs a collaborator. |
|
||||
| `src/version.zig` | Build-time version strings. |
|
||||
| `src/tests.zig` | Test root; imports each file directly. |
|
||||
| `src/docs_drift_test.zig` | Guards that keep the reference pages in step with the code. |
|
||||
|
||||
Directories:
|
||||
|
||||
| Directory | Role |
|
||||
|---|---|
|
||||
| `src/dns/` | Pure DNS wire format: header, names, questions, records, whole packets, EDNS(0)/ECS (`edns.zig`), enums and limits (`types.zig`). No allocation, no `std.Io` beyond writing to a caller's writer. |
|
||||
| `src/filter/` | Blocklist pipeline: line parsers (hosts, domains, ABP), the compiler that turns a downloaded list into `.list`/`.wild` bodies, `domain_set.zig` (exact-match set, no Bloom filter), `matcher.zig` (the immutable snapshot every query evaluates against), per-group `rules.zig`, `wildcard.zig`, `safesearch.zig`, blocked-response synthesis (`response.zig`). Two I/O edges live here too: `fetcher.zig` (HTTP download) and `manager.zig` (files + DB + snapshot swap). |
|
||||
| `src/local/` | Local DNS records and conditional forward zones: immutable lookup tables built once from DB rows (`records.zig`, `forward_zones.zig`), plus the plain UDP/TCP client for LAN resolvers (`forward_client.zig`). |
|
||||
| `src/cache/` | `dns_cache.zig`: bounded in-memory TTL cache of whole response messages, keyed by the question. The clock arrives as a parameter. |
|
||||
| `src/upstream/` | Upstream resolution: shared vocabulary and the `Client` interface (`transport.zig`), DoH client (RFC 8484), DoT client (RFC 7858), per-endpoint health and backoff (`health.zig`), and `pool.zig` — priority-ordered failover that is itself a `transport.Client`, so the handler sees one interface. |
|
||||
| `src/server/` | The serving side: UDP/TCP/DoH/DoT listeners, `handler.zig` (the whole query pipeline), `cert_store.zig` (refcounted TLS cert holder), `rate_limiter.zig`, `pause.zig`, `clients.zig` (client auto-materialisation), `local_tables.zig` (published local-answer tables), `query_sink.zig` (log and SSE fanout), `shutdown.zig` (SIGINT/SIGTERM into one `std.Io.Event`). |
|
||||
| `src/storage/` | SQLite ownership: `db.zig` is the only file that calls SQLite, `config_schema.zig` + `migrations.zig` for `config.db`, `querylog_schema.zig` (open-or-recreate), async query `logger.zig`, `retention.zig`, `disk_monitor.zig`, and one repository per table under `repositories/`. |
|
||||
| `src/config/` | The one configuration model (`model.zig`), the pure validator (`validate.zig`), `import.zig`/`export.zig` (ZON to and from `config.db`, byte-stable round trip), `bootstrap.zig` (first-start seeding — a policy wrapper over import). |
|
||||
| `src/web/` | The admin HTTP layer: `server.zig` (listener), `router.zig`/`routes.zig`, one file per resource under `handlers/`, `auth.zig` (sessions), `sse.zig` (live query fanout), `static.zig` (embedded SPA), `metrics.zig` (Prometheus), `openapi.zig` (served contract), `api_limiter.zig`, `http_util.zig`. |
|
||||
| `src/platform/` | OS and TLS edges: IP address values, the `std.log` sink (`logging.zig`), `statfs.zig` (free-space query via libc), client TLS over `std.crypto.tls` (`tls_client.zig`), server TLS over vendored Mbed TLS (`tls_server.zig`). |
|
||||
|
||||
The SPA source lives in `web/` at the repo root; the build embeds its `dist/`
|
||||
output as the `web_assets` module (`-Dweb-dist`).
|
||||
|
||||
```
|
||||
main.zig ── cli.zig ── app.zig (composition root)
|
||||
│ injects std.Io + collaborators
|
||||
┌──────────────────────┴───────────────────────┐
|
||||
│ server/ web/ upstream/ storage/ │ I/O edge
|
||||
│ platform/ config/{import,export,bootstrap} │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ dns/ filter/* local/* cache/ │ pure core:
|
||||
│ config/{model,validate} │ bytes in, bytes out
|
||||
└──────────────────────────────────────────────┘
|
||||
* except filter/{fetcher,manager}.zig and local/forward_client.zig,
|
||||
which are those directories' named I/O edges
|
||||
```
|
||||
|
||||
## The purity rule
|
||||
|
||||
`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no
|
||||
`std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs
|
||||
a timestamp takes it as a parameter — the cache, the rate limiter and the
|
||||
pause flag all work this way, so every decision is testable without a backend.
|
||||
|
||||
The point is not purity for its own sake. A decision that depends on a hidden
|
||||
clock or a hidden socket can only be tested by arranging the world around it;
|
||||
one that takes the clock as an argument is tested by passing a number. The
|
||||
whole filtering and caching pipeline can therefore be exercised in the plain,
|
||||
network-free test suite, which is what makes that suite worth gating CI on.
|
||||
|
||||
The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads
|
||||
lists, `filter/manager.zig` owns the compiled files, the DB columns and the
|
||||
snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN
|
||||
resolver. Those three files are the only ones under those four directories
|
||||
that take a `std.Io`. The decision path a query takes through them allocates
|
||||
nothing and opens nothing.
|
||||
|
||||
The honest cost of the exceptions shows up in
|
||||
[performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is
|
||||
where the one production crash came from, precisely because it is the file the
|
||||
pure suite cannot reach.
|
||||
|
||||
## std.Io injection
|
||||
|
||||
There is one `std.Io` in the process. `main` receives it through
|
||||
`std.process.Init` — on the standard start path this is the Threaded backend
|
||||
(`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to
|
||||
`cli.Runner`, from which `app.zig` threads it into every collaborator as a
|
||||
parameter. No module constructs its own event loop or reads an ambient clock;
|
||||
tests build their own `std.Io.Threaded` instance and pass it the same way.
|
||||
|
||||
The one deliberate exception is `storage/db.zig`: SQLite performs its own file
|
||||
I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's
|
||||
VFS to route through `std.Io` would be a large amount of C-boundary code to
|
||||
make one dependency match a convention it does not need.
|
||||
|
||||
## Life of one query
|
||||
|
||||
The pipeline lives in `src/server/handler.zig` — `Handler.handle` does
|
||||
validation and setup, then `Context.run` decides the answer. Its order is
|
||||
PLAN §4; the stages below are the code's actual call chain:
|
||||
|
||||
```
|
||||
UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig)
|
||||
└──────┴──────┴────┘
|
||||
│ raw query bytes, listener-owned buffers
|
||||
▼
|
||||
handler.handle
|
||||
├─ header parse (too short / QR set → counted drop)
|
||||
├─ rate limit (over budget → REFUSED)
|
||||
├─ packet + EDNS validation (FORMERR / NOTIMP, qdcount != 1 → FORMERR)
|
||||
├─ one clock read, client tracking, snapshot + local-table acquire
|
||||
├─ group lookup (snapshot.groupForClient)
|
||||
│
|
||||
└─ Context.run
|
||||
├─ qclass != IN ──► upstream, unfiltered and uncached
|
||||
├─ local records ────────────────► authoritative answer
|
||||
├─ forward zones ─► cache ─► LAN resolver ─► cache put ─► answer
|
||||
│
|
||||
└─ upstream path (filtering off while paused)
|
||||
├─ filter snapshot evaluate ─► blocked? synthesized block reply
|
||||
├─ safe-search rewrite (per group)
|
||||
├─ cache get ─► hit? answer
|
||||
├─ upstream pool: priority failover across DoH/DoT endpoints
|
||||
├─ CNAME uncloak: walk the answer's chain, re-evaluate each target
|
||||
└─ cache put (skipped for safe-search answers)
|
||||
▼
|
||||
reply: UDP size check ─► TC=1 if over the limit ─► listener sends
|
||||
│
|
||||
└─► QuerySink ─► SSE hub (GET /api/queries/live)
|
||||
└─► async logger ─► querylog.db
|
||||
```
|
||||
|
||||
Local records win over forward zones, and both win over filtering: a name
|
||||
nxdns answers itself never reaches a blocklist. Pause suspends filtering only;
|
||||
local records, forward zones, cache, upstream and the query log keep running,
|
||||
which is what makes pause safe to hand to a household member. A question whose
|
||||
class is not IN bypasses local answers, filtering and the cache entirely and
|
||||
goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and
|
||||
declines to cache answers it does not model.
|
||||
|
||||
`handle` returns no error union. Every failure is either a DNS response the
|
||||
client can act on or a counted drop, because there is no caller above it that
|
||||
could do anything useful with a Zig error. Two consequences are worth knowing:
|
||||
answers synthesized from a safe-search rewrite are never cached (the rewrite is
|
||||
per group, and the cache is not), and a SERVFAIL reply short-circuits before the
|
||||
query sink, so it appears in the counters but not in the query log.
|
||||
|
||||
The query path never waits on the database. `QuerySink` copies the entry, the
|
||||
SSE hub gets it first, and one writer task owns the `querylog.db` handle behind
|
||||
an `std.Io.Queue`. A slow disk delays logging, never resolution.
|
||||
|
||||
Two smaller decisions in the same spirit: the upstream pool makes a second
|
||||
pass that ignores backoff, so "every endpoint is in backoff" degrades to
|
||||
trying anyway rather than to a blanket SERVFAIL; and a handler that has no
|
||||
filter snapshot yet answers unfiltered rather than refusing. Both prefer a
|
||||
working resolver over a correct-looking failure.
|
||||
|
||||
## Storage
|
||||
|
||||
Two databases with opposite contracts, in one data directory (see
|
||||
[reference/files-and-directories.md](../reference/files-and-directories.md)).
|
||||
|
||||
**`config.db` is the truth.** Its schema is versioned: `migrations.zig` holds
|
||||
an ordered list of steps, step 1 being the verbatim DDL from
|
||||
`config_schema.zig`, each applied inside one transaction. `nxdns import`
|
||||
replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed
|
||||
import changes nothing; `nxdns export` renders it back as canonical ZON,
|
||||
byte-identical across round trips. A config file seeds this database exactly
|
||||
once at first start. Why it works that way is
|
||||
[configuration-model.md](configuration-model.md).
|
||||
|
||||
**`querylog.db` is expendable.** It is never migrated. Its schema carries a
|
||||
fingerprint derived from the DDL text, and at open, a missing, corrupt,
|
||||
non-database, `quick_check`-failing or fingerprint-mismatched file is moved
|
||||
aside and recreated empty — the old file is kept under a new name rather than
|
||||
deleted, so an operator can still look at it. Retention deletes old rows daily
|
||||
and periodically rewrites the file to reclaim space.
|
||||
|
||||
The split exists so that the churn of the second database can never endanger
|
||||
the first. Query logs are high-volume, disposable, and the thing most likely
|
||||
to be corrupted by a power cut on an SD card; configuration is small,
|
||||
irreplaceable, and the thing an operator would have to reconstruct by hand.
|
||||
Giving them one file would force the careful contract onto the noisy data or
|
||||
the loose contract onto the valuable data.
|
||||
|
||||
## Web stack
|
||||
|
||||
`web/server.zig` runs one `std.http.Server` per connection over its own accept
|
||||
loop, with a fixed set of pre-allocated connection slots, optionally behind
|
||||
TLS. Over capacity it answers 503 rather than queueing without bound — the
|
||||
admin UI is not the product, and it must not be able to starve DNS.
|
||||
|
||||
The SPA is embedded at build time: `static.zig` serves the `web_assets` module
|
||||
— bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where
|
||||
it paid off — via a linear scan with no filesystem access at runtime. (The one
|
||||
exception is `nxdns run --web-dev DIR`, which serves from disk with no cache
|
||||
headers, for developing the SPA against a running server.) `GET
|
||||
/api/queries/live` is server-sent events over chunked transfer, fed by the
|
||||
same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`)
|
||||
matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML
|
||||
is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest
|
||||
by tests that assert every served route appears in it.
|
||||
|
||||
Authentication (`web/auth.zig`): the operator's password is verified against
|
||||
an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import
|
||||
and never stored). A successful login mints a 256-bit token carried in a
|
||||
cookie; the in-memory session table holds only SHA-256 digests of tokens,
|
||||
compared in constant time, capped at 32 sessions with LRU eviction. Nothing is
|
||||
persisted, so a restart logs everyone out — for a household LAN that is a
|
||||
feature, not a gap. Unauthenticated by design: the monitoring endpoints
|
||||
(health, version, metrics), the served OpenAPI contract, login itself, and the
|
||||
static SPA assets, which the router hands to the SPA fallback before any auth
|
||||
check. Everything else requires the cookie, and the API has its own
|
||||
token-bucket rate limiter. See
|
||||
[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
|
||||
|
||||
## DoH, DoT and certificate hot-reload
|
||||
|
||||
`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and
|
||||
`server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server
|
||||
TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as
|
||||
`std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether
|
||||
they are encrypted.
|
||||
|
||||
Certificates hot-reload through `server/cert_store.zig`. One refcounted
|
||||
`CertStore` per endpoint owns the published TLS context generation; listeners
|
||||
`acquire` it per connection and `release` it when the connection ends, so a
|
||||
reload never frees a context mid-handshake. Reload publishes nothing on
|
||||
failure: both PEM files are read and a whole new context is built before
|
||||
anything swaps, and any failure leaves the old generation serving. A watcher
|
||||
polls mtime and size of both files every 30 seconds
|
||||
(`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same
|
||||
path on demand and reports the per-endpoint outcome as its payload.
|
||||
|
||||
The requirement driving all of this is that a certbot renewal must not need a
|
||||
restart and must not be able to break DNS. A half-swapped context or a
|
||||
free-while-in-use would do exactly that, so the store is built so neither is
|
||||
representable. See
|
||||
[how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md).
|
||||
|
||||
## Failure visibility
|
||||
|
||||
Every failure mode must be visible, and the surface is counters, not log lines
|
||||
(AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they
|
||||
are only read after someone already suspects a problem, and on an SD card they
|
||||
are a way to fill a disk.
|
||||
|
||||
So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP,
|
||||
REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and
|
||||
forward-zone answers, safe-search rewrites, paused and unfiltered queries, and
|
||||
the tracker-full condition. Listeners count dropped datagrams instead of
|
||||
queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text
|
||||
0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers
|
||||
200: "degraded" is a fact about the box, not a failed request, and a monitor
|
||||
that cannot distinguish the two is worse than no monitor.
|
||||
|
||||
The disk monitor classifies free space against thresholds and gates
|
||||
non-essential writes; the query logger holds its batches while writes are
|
||||
disallowed rather than dropping them silently or writing until the filesystem
|
||||
fills. `std.log` is reserved for failures nobody else records, with
|
||||
upstream-error deduplication so a flapping resolver cannot fill a disk with
|
||||
identical lines.
|
||||
@@ -0,0 +1,178 @@
|
||||
# The configuration model
|
||||
|
||||
nxdns is configured two ways — a ZON file and a web UI — and only one of them
|
||||
can be the truth. This page explains which, and why that choice is the one
|
||||
that leaves the fewest ways to lose an operator's work.
|
||||
|
||||
For the fields themselves see
|
||||
[reference/configuration.md](../reference/configuration.md); for the commands
|
||||
and their exit codes see [reference/cli.md](../reference/cli.md).
|
||||
|
||||
## The rule
|
||||
|
||||
The database is the truth. The file is a seed.
|
||||
|
||||
`config.db` in the data directory holds the running configuration. The ZON
|
||||
file (default `/etc/nxdns/config.zon`, overridable with `--config`) is read on
|
||||
`nxdns run` only while the database is still empty: if the file exists and the
|
||||
database holds no configuration, it is imported. Once the import has put rows
|
||||
in, the file is not opened again.
|
||||
|
||||
The condition is the state of the database, not a one-shot flag. A `run` whose
|
||||
seed fails — unreadable file, parse error, failed validation, a constraint
|
||||
violation inside the import transaction — leaves the database empty, so the
|
||||
next `run` reads the file again. That is what makes fixing a typo and starting
|
||||
again work.
|
||||
|
||||
`src/config/bootstrap.zig` is that policy and nothing else — a wrapper over
|
||||
the same import path `nxdns import` uses. Three outcomes:
|
||||
|
||||
- no file: the database is used as it is;
|
||||
- file present, database empty: seed it;
|
||||
- file present, database already configured: skip, without reading the file.
|
||||
|
||||
A file that exists but is unreadable, unparseable or invalid fails the start,
|
||||
with every problem printed. nxdns never falls back to silent defaults over a
|
||||
file an operator wrote — a resolver that boots "successfully" with a
|
||||
configuration nobody chose is the worst outcome available, because it looks
|
||||
like it worked.
|
||||
|
||||
## Why the database wins
|
||||
|
||||
The alternative designs all lose data.
|
||||
|
||||
If the file were the truth, the admin UI could not write. Every change would
|
||||
be an SSH session and a restart, which defeats the reason the UI exists: the
|
||||
household member who wants to unblock one domain is not going to edit ZON.
|
||||
|
||||
If both were the truth, they would disagree. The UI writes a rule; the file
|
||||
still says otherwise; the next restart either silently reverts the rule or
|
||||
silently ignores the file. Both are silent, and both destroy work someone
|
||||
intended to keep. There is no merge rule that fixes this, because the system
|
||||
cannot know which of two conflicting statements is the newer intention.
|
||||
|
||||
So the file's authority ends the moment the database has content. Editing
|
||||
`config.zon` after first boot does nothing — no partial effect, no warning
|
||||
that some fields took and others did not. That is a blunt rule, and it is the
|
||||
point: the failure mode is "my edit did nothing", which is visible the first
|
||||
time you look at the UI, rather than "my edit was applied and then quietly
|
||||
undone next Tuesday".
|
||||
|
||||
The emptiness check is a real query over the content tables, not a flag: the
|
||||
database counts as configured when any content table has rows, or when the
|
||||
default group has been altered. One consequence is worth knowing, because it
|
||||
is not obvious: clients are auto-materialised when they first send a query, so
|
||||
a server that has answered even one query is "configured" and will ignore a
|
||||
seed file placed there afterwards.
|
||||
|
||||
## The round trip
|
||||
|
||||
Losing the file as an editing surface would be a real loss — text is
|
||||
diffable, reviewable and easy to back up — so the file is kept as a
|
||||
*rendering* of the database rather than a rival to it. That is what
|
||||
`export`/`import` are for:
|
||||
|
||||
```
|
||||
nxdns export → canonical ZON → edit → nxdns import --force → config.db
|
||||
```
|
||||
|
||||
`nxdns export` renders the database as canonical ZON: a fixed two-line
|
||||
header, every default emitted, deterministic ordering from the model's field
|
||||
order and the repositories' `ORDER BY` clauses, and no timestamps or hostnames
|
||||
anywhere. Runtime columns — first-seen, last-seen, per-source counters — are
|
||||
absent from the configuration model on purpose, so two exports taken from a
|
||||
live, busy server are identical. The round trip `export → import → export` is
|
||||
byte-identical, and a test asserts it.
|
||||
|
||||
Byte-stability is not cosmetic. It is what makes an exported file usable in
|
||||
version control and what makes a diff of two exports mean something: any
|
||||
difference is a configuration change, never noise from when the export ran.
|
||||
|
||||
`nxdns import` replaces the whole database content in one transaction. Not a
|
||||
merge, not a patch: delete every content table in foreign-key-safe order, then
|
||||
insert what the file says. A failed import — bad syntax, failed validation, a
|
||||
constraint violation halfway through — leaves the database exactly as it was,
|
||||
because everything happens inside a single `BEGIN IMMEDIATE`.
|
||||
|
||||
Without `--force`, import refuses a database that already has content. That
|
||||
check runs *inside* the transaction, after the lock is taken, so it cannot be
|
||||
raced by a concurrent write. The effect is that a plain `import` can never
|
||||
clobber a configured server by accident, and clobbering it deliberately takes
|
||||
one visible extra word on the command line. See
|
||||
[how-to/back-up-and-restore.md](../how-to/back-up-and-restore.md).
|
||||
|
||||
## One model, three surfaces
|
||||
|
||||
`src/config/model.zig` defines exactly one `Config` type. The ZON parser
|
||||
produces it, the database reader produces it, the validator consumes it, the
|
||||
composition root consumes it, and the settings API derives its key list and
|
||||
its patch struct from its type information rather than mirroring the fields by
|
||||
hand. Adding a field in one place therefore cannot leave the other surfaces
|
||||
behind; the alternative — a file schema, a DB schema and an API schema kept in
|
||||
step by discipline — is the standard way configuration systems rot.
|
||||
|
||||
The model has two shapes of field, and the difference is structural rather
|
||||
than stylistic. Struct-typed fields are scalar sections (`dns`, `cache`,
|
||||
`web`, …) and live in a single key/value `settings` table as
|
||||
`section.field` text rows. Slice-typed fields are collections (groups,
|
||||
upstreams, clients, rules, local records, forward zones, …) and each gets its
|
||||
own table with foreign keys. `toSettings` and `fromSettings` are the two
|
||||
halves of the scalar bridge, generated by an `inline for` over the model, so
|
||||
the key list is a consequence of the type rather than a second list to
|
||||
maintain.
|
||||
|
||||
## Unknown keys, and why the asymmetry is deliberate
|
||||
|
||||
An unknown key in the **database** is warned about and ignored. An unknown key
|
||||
in the **file** is a hard error.
|
||||
|
||||
They are different situations. A settings row the running binary does not
|
||||
recognise is almost always an older binary reading a database written by a
|
||||
newer one — a downgrade, or a rollback after a bad upgrade. Refusing to start
|
||||
there would mean a downgrade bricks the config database, and the operator
|
||||
would have to hand-edit SQLite to recover. Warning and ignoring means the
|
||||
downgrade works, the unknown setting sits inert, and the upgrade back picks it
|
||||
up again.
|
||||
|
||||
A key in a file, by contrast, is something a human just typed. The likeliest
|
||||
cause is a typo, and the second likeliest is a field that no longer exists.
|
||||
Silently ignoring it would mean the setting the operator believes they applied
|
||||
was never applied — exactly the silent-divergence failure the whole model is
|
||||
built to avoid. So the ZON parser rejects unknown fields with a line and
|
||||
column.
|
||||
|
||||
The tolerance has a boundary worth stating plainly: it covers unknown *keys*,
|
||||
not unparseable *values*. A known key whose stored text does not decode into
|
||||
its type is an error, not a warning.
|
||||
|
||||
## The password
|
||||
|
||||
`web.password` is a write-only input. It is never a stored value.
|
||||
|
||||
At import time, a non-empty `web.password` is hashed with argon2id (PHC
|
||||
encoding, OWASP argon2id parameters) into `web.password_hash`, and the
|
||||
plaintext field is cleared before anything is written. There is no settings
|
||||
row that can hold it: the model explicitly skips `web.password` in both
|
||||
directions of the settings bridge, so the plaintext has nowhere to go even by
|
||||
accident. `nxdns export` always writes `.password = ""` and carries the hash
|
||||
instead — which is also what makes the round trip stable, since an export that
|
||||
tried to reproduce a plaintext it never had could not be byte-identical.
|
||||
|
||||
Setting both `password` and `password_hash` in one file is an error rather
|
||||
than a precedence rule. The two say different things about what the password
|
||||
is, and picking a winner would mean the operator's other statement was
|
||||
silently discarded. The file has to say one thing.
|
||||
|
||||
The practical shape of a password change is therefore export, set `.password`
|
||||
to the new value, clear `.password_hash`, import with `--force`. The procedure
|
||||
is in
|
||||
[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
|
||||
|
||||
## What is not configuration
|
||||
|
||||
Storage paths are process arguments, not configuration fields: `--data-dir`,
|
||||
`--config`, `--web-dev`. They cannot live in the file, because the file is
|
||||
found by way of them — a path that told you where to find the thing that told
|
||||
you the path would be circular. They are also the settings a supervisor
|
||||
(systemd, Docker) owns rather than the operator's policy about DNS. See
|
||||
[reference/files-and-directories.md](../reference/files-and-directories.md).
|
||||
@@ -0,0 +1,161 @@
|
||||
# Performance targets and what the tests prove
|
||||
|
||||
Two related questions: why the performance numbers are the numbers, and why CI
|
||||
does not enforce them — and then, less comfortably, what a green test suite
|
||||
here does and does not tell you.
|
||||
|
||||
For the targets and the measured results as data, see
|
||||
[reference/performance.md](../reference/performance.md); to run the bench
|
||||
yourself, [how-to/measure-performance.md](../how-to/measure-performance.md).
|
||||
|
||||
## Where the targets come from
|
||||
|
||||
PLAN §18 sets five:
|
||||
|
||||
- sustained ≥ 100 qps on a Raspberry Pi 5;
|
||||
- blocklist lookup p95 < 1 ms;
|
||||
- cached response p95 < 5 ms;
|
||||
- memory with ~1M blocked domains < 100 MB;
|
||||
- stripped static binary < 10 MB per arch, < 15 MB with the embedded frontend.
|
||||
|
||||
They are household-scale numbers, and they are deliberately unambitious. 100
|
||||
qps is far more than a house generates; the point of the target is not speed
|
||||
but that a Pi 5 with an SD card never becomes the reason the internet feels
|
||||
broken. The latency targets exist for the same reason: DNS sits in front of
|
||||
every connection anyone makes, so the failure people notice is not throughput
|
||||
but a stall. The memory target is what keeps a 1M-entry blocklist from
|
||||
competing with everything else on a 4 GB board. The binary-size target is
|
||||
about what a static single-binary deployment is for — if it does not fit on a
|
||||
constrained box and copy over a slow link in one step, the packaging decision
|
||||
has not paid for itself.
|
||||
|
||||
`tools/bench.zig` (`zig build bench`) measures the three that are measurable
|
||||
in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry
|
||||
snapshot), `cache` (key build plus cache get plus id patch), and `compile`
|
||||
(the blocklist compiler over a 1M-line body, informational — there is no §18
|
||||
target for it because no prior datapoint exists). Memory comes from
|
||||
`/proc/self/status` VmRSS. The qps target is not in the harness at all: it is
|
||||
end-to-end against the real binary with a DNS load generator, because a
|
||||
harness number for "queries per second" would measure the harness.
|
||||
|
||||
The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product,
|
||||
and `src/tests.zig` aggregates everything shippable.
|
||||
|
||||
## Why CI does not gate on performance
|
||||
|
||||
Required CI stays deterministic (AGENTS.md). Latency assertions on shared
|
||||
runners measure the runner's noisy neighbours; the same commit passes and
|
||||
fails depending on what else the host is doing. A gate that flakes does not
|
||||
protect anything — it trains people to re-run the job, and once re-running is
|
||||
routine, a real regression gets re-run too. The flaky gate is worse than no
|
||||
gate, because it also consumes the attention a real gate would need.
|
||||
|
||||
So the bench defaults to informational, and `--assert` — which exits non-zero
|
||||
on a missed target — exists for hardware you control. Run it on the Pi, where
|
||||
the numbers describe the machine the software actually has to run on. The
|
||||
x86_64 development-host numbers in
|
||||
[reference/performance.md](../reference/performance.md) are a regression
|
||||
baseline for the machine development happens on, not a claim about the target
|
||||
platform; a Cortex-A76 is far slower and those numbers do not transfer.
|
||||
|
||||
CI does gate on the one performance property that *is* deterministic: binary
|
||||
size. The `cross` job strips the release binaries and asserts them under the
|
||||
§18 budgets. Size is a function of the input, not of the runner's mood, so it
|
||||
is exactly the kind of thing a shared runner can measure honestly.
|
||||
|
||||
## What the test suite is
|
||||
|
||||
The blocking CI (Gitea Actions, `.gitea/workflows/ci.yml`) runs five jobs, all
|
||||
required: the Zig suite with `-Dintegration`; the same suite cross-built for
|
||||
aarch64 and executed under qemu-user; the frontend (format, lint, typecheck,
|
||||
121 vitest cases, build); the cross-build with the two stripped-size asserts;
|
||||
and a Docker smoke run that boots the image and polls `/api/health`.
|
||||
|
||||
The Zig suite has three tiers, gated by build flags:
|
||||
|
||||
- **plain `zig build test`** — pure logic. No sockets, no threads, no clock
|
||||
budgets. This is the tier the purity rule
|
||||
([architecture.md](architecture.md)) exists to make possible.
|
||||
- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:`
|
||||
databases, temp directories. Nothing leaves the host.
|
||||
- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT
|
||||
handshakes against real resolvers). Four tests, and they run in a
|
||||
manual-dispatch workflow, never on push or pull request.
|
||||
|
||||
The aarch64 job runs the plain tier only. The integration tests are
|
||||
multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown
|
||||
turns those budgets into a flake source — the same reasoning that keeps the
|
||||
bench out of CI. What aarch64 needs to prove is portable correctness of the
|
||||
DNS, filter and cache logic, and the plain tier is exactly that.
|
||||
|
||||
At the time of writing, plain `zig build test` is 1175 of 1288 passing with
|
||||
113 skipped and 0 failed, the skips being the integration-gated tests.
|
||||
Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with
|
||||
`-Dintegration` (the 4 skips are the live-network tests) and 1159 passing
|
||||
under qemu, 0 failed in each.
|
||||
|
||||
## What it does not prove
|
||||
|
||||
The suite is hermetic by design. That is the right default: it is fast, it is
|
||||
deterministic, it can gate merges. But hermetic and correct are different
|
||||
properties, and the gap has already cost this project twice.
|
||||
|
||||
**The blocklist download aborted the process on first real use.** The fetcher
|
||||
constructed the HTTP response reader over `transfer_buf` and then read *into*
|
||||
that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's
|
||||
already-buffered bytes into the caller's destination — so source and
|
||||
destination were the same allocation, and Zig's `@memcpy` requires them not to
|
||||
overlap. It aborts.
|
||||
|
||||
The reason no test caught it is precise and instructive. The copy length is
|
||||
zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is
|
||||
fine. Bytes only accumulate in the reader's own buffer when a read comes back
|
||||
short of filling the destination and the loop goes round again — that is, when
|
||||
the body arrives in more than one stream call. The loopback fixture answers
|
||||
every request with one small in-memory body that lands in a single read, and
|
||||
the one over-size test never streams a byte, because the fetcher refuses an
|
||||
oversized `content-length` on the response head. Every test in the suite was
|
||||
on the zero-length-memcpy side of the branch. The first real download — a
|
||||
multi-megabyte list over TLS across the WAN, arriving in many TCP segments —
|
||||
was on the other side, and took the process down. The fix (commit 35f2324)
|
||||
streams the body straight into the caller's writer, so the reader's buffer is
|
||||
never a destination slice, and it came with four regression tests that put a
|
||||
fully-buffered reader into exactly the state the old code could not survive.
|
||||
|
||||
**A stale embedded SPA bundle shipped a settings page that crashed on load,
|
||||
while 121 web tests passed.** `web/dist/` is gitignored and
|
||||
`-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The
|
||||
frontend tests ran against the sources, in jsdom, and were green; the binary
|
||||
carried an older build. The tests were testing something the artifact did not
|
||||
contain.
|
||||
|
||||
Note what these two have in common. Neither was a logic bug that a better unit
|
||||
test would have caught. One lived in the seam between the pure core and its
|
||||
one I/O edge; the other lived in the seam between two build systems. Hermetic
|
||||
tests are constructed to exclude exactly those seams — that is what makes them
|
||||
hermetic.
|
||||
|
||||
## The lesson, and where it now lives
|
||||
|
||||
Green hermetic tests are a floor, not a ceiling. They prove the logic is
|
||||
consistent with itself. They cannot prove the program works, because the
|
||||
things they deliberately exclude — real network reads, real TLS, real file
|
||||
sizes, real build artifacts — are where a program meets reality.
|
||||
|
||||
The response is not to make CI non-deterministic. It is to require that the
|
||||
real paths get exercised by a human before work is called done. That is now
|
||||
ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and
|
||||
the how-to pages is executed verbatim, on the host, by the session that writes
|
||||
it, and a command that cannot run there is marked in the page as unverified
|
||||
with the reason. Documentation written from source-reading alone is how both
|
||||
of these shipped; documentation that has been run is a second, independent
|
||||
test suite that exercises precisely the paths the hermetic one skips.
|
||||
|
||||
Two honest gaps remain, stated so nobody has to rediscover them:
|
||||
|
||||
- Nothing in the suite drives a multi-read HTTP body through the fetcher end
|
||||
to end. The regression tests cover `pumpBody` directly over a pre-buffered
|
||||
reader; the loopback fixture still sends one small body per connection.
|
||||
- There is no freshness check on `web/dist`. CI cannot embed a stale bundle,
|
||||
because the jobs that pass `-Dweb-dist` rebuild the frontend immediately
|
||||
beforehand. A local build can, and will do it without a warning.
|
||||
Reference in New Issue
Block a user