milestone 13: restructure docs to diataxis, tutorial, every command executed
This commit is contained in:
@@ -0,0 +1,264 @@
|
||||
# Architecture
|
||||
|
||||
nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary
|
||||
that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against
|
||||
blocklists, and serves an embedded admin SPA over HTTP. This page maps the
|
||||
source tree and explains the few design rules that hold everywhere, and why
|
||||
they are the rules.
|
||||
|
||||
For what the configuration fields, API routes and CLI flags actually are, see
|
||||
[reference/configuration.md](../reference/configuration.md),
|
||||
[reference/api.md](../reference/api.md) and
|
||||
[reference/cli.md](../reference/cli.md). This page does not repeat them.
|
||||
|
||||
## Module map
|
||||
|
||||
Top-level files:
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `src/main.zig` | Process shell: writers, argv, dispatch, exit code. |
|
||||
| `src/cli.zig` | Every command body (`run`, `check`, `export`, `import`, `version`, `help`); takes its writers as parameters so tests capture output without a process. |
|
||||
| `src/app.zig` | The composition root: everything `nxdns run` owns, built in order. Nothing else constructs a collaborator. |
|
||||
| `src/version.zig` | Build-time version strings. |
|
||||
| `src/tests.zig` | Test root; imports each file directly. |
|
||||
| `src/docs_drift_test.zig` | Guards that keep the reference pages in step with the code. |
|
||||
|
||||
Directories:
|
||||
|
||||
| Directory | Role |
|
||||
|---|---|
|
||||
| `src/dns/` | Pure DNS wire format: header, names, questions, records, whole packets, EDNS(0)/ECS (`edns.zig`), enums and limits (`types.zig`). No allocation, no `std.Io` beyond writing to a caller's writer. |
|
||||
| `src/filter/` | Blocklist pipeline: line parsers (hosts, domains, ABP), the compiler that turns a downloaded list into `.list`/`.wild` bodies, `domain_set.zig` (exact-match set, no Bloom filter), `matcher.zig` (the immutable snapshot every query evaluates against), per-group `rules.zig`, `wildcard.zig`, `safesearch.zig`, blocked-response synthesis (`response.zig`). Two I/O edges live here too: `fetcher.zig` (HTTP download) and `manager.zig` (files + DB + snapshot swap). |
|
||||
| `src/local/` | Local DNS records and conditional forward zones: immutable lookup tables built once from DB rows (`records.zig`, `forward_zones.zig`), plus the plain UDP/TCP client for LAN resolvers (`forward_client.zig`). |
|
||||
| `src/cache/` | `dns_cache.zig`: bounded in-memory TTL cache of whole response messages, keyed by the question. The clock arrives as a parameter. |
|
||||
| `src/upstream/` | Upstream resolution: shared vocabulary and the `Client` interface (`transport.zig`), DoH client (RFC 8484), DoT client (RFC 7858), per-endpoint health and backoff (`health.zig`), and `pool.zig` — priority-ordered failover that is itself a `transport.Client`, so the handler sees one interface. |
|
||||
| `src/server/` | The serving side: UDP/TCP/DoH/DoT listeners, `handler.zig` (the whole query pipeline), `cert_store.zig` (refcounted TLS cert holder), `rate_limiter.zig`, `pause.zig`, `clients.zig` (client auto-materialisation), `local_tables.zig` (published local-answer tables), `query_sink.zig` (log and SSE fanout), `shutdown.zig` (SIGINT/SIGTERM into one `std.Io.Event`). |
|
||||
| `src/storage/` | SQLite ownership: `db.zig` is the only file that calls SQLite, `config_schema.zig` + `migrations.zig` for `config.db`, `querylog_schema.zig` (open-or-recreate), async query `logger.zig`, `retention.zig`, `disk_monitor.zig`, and one repository per table under `repositories/`. |
|
||||
| `src/config/` | The one configuration model (`model.zig`), the pure validator (`validate.zig`), `import.zig`/`export.zig` (ZON to and from `config.db`, byte-stable round trip), `bootstrap.zig` (first-start seeding — a policy wrapper over import). |
|
||||
| `src/web/` | The admin HTTP layer: `server.zig` (listener), `router.zig`/`routes.zig`, one file per resource under `handlers/`, `auth.zig` (sessions), `sse.zig` (live query fanout), `static.zig` (embedded SPA), `metrics.zig` (Prometheus), `openapi.zig` (served contract), `api_limiter.zig`, `http_util.zig`. |
|
||||
| `src/platform/` | OS and TLS edges: IP address values, the `std.log` sink (`logging.zig`), `statfs.zig` (free-space query via libc), client TLS over `std.crypto.tls` (`tls_client.zig`), server TLS over vendored Mbed TLS (`tls_server.zig`). |
|
||||
|
||||
The SPA source lives in `web/` at the repo root; the build embeds its `dist/`
|
||||
output as the `web_assets` module (`-Dweb-dist`).
|
||||
|
||||
```
|
||||
main.zig ── cli.zig ── app.zig (composition root)
|
||||
│ injects std.Io + collaborators
|
||||
┌──────────────────────┴───────────────────────┐
|
||||
│ server/ web/ upstream/ storage/ │ I/O edge
|
||||
│ platform/ config/{import,export,bootstrap} │
|
||||
├──────────────────────────────────────────────┤
|
||||
│ dns/ filter/* local/* cache/ │ pure core:
|
||||
│ config/{model,validate} │ bytes in, bytes out
|
||||
└──────────────────────────────────────────────┘
|
||||
* except filter/{fetcher,manager}.zig and local/forward_client.zig,
|
||||
which are those directories' named I/O edges
|
||||
```
|
||||
|
||||
## The purity rule
|
||||
|
||||
`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no
|
||||
`std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs
|
||||
a timestamp takes it as a parameter — the cache, the rate limiter and the
|
||||
pause flag all work this way, so every decision is testable without a backend.
|
||||
|
||||
The point is not purity for its own sake. A decision that depends on a hidden
|
||||
clock or a hidden socket can only be tested by arranging the world around it;
|
||||
one that takes the clock as an argument is tested by passing a number. The
|
||||
whole filtering and caching pipeline can therefore be exercised in the plain,
|
||||
network-free test suite, which is what makes that suite worth gating CI on.
|
||||
|
||||
The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads
|
||||
lists, `filter/manager.zig` owns the compiled files, the DB columns and the
|
||||
snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN
|
||||
resolver. Those three files are the only ones under those four directories
|
||||
that take a `std.Io`. The decision path a query takes through them allocates
|
||||
nothing and opens nothing.
|
||||
|
||||
The honest cost of the exceptions shows up in
|
||||
[performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is
|
||||
where the one production crash came from, precisely because it is the file the
|
||||
pure suite cannot reach.
|
||||
|
||||
## std.Io injection
|
||||
|
||||
There is one `std.Io` in the process. `main` receives it through
|
||||
`std.process.Init` — on the standard start path this is the Threaded backend
|
||||
(`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to
|
||||
`cli.Runner`, from which `app.zig` threads it into every collaborator as a
|
||||
parameter. No module constructs its own event loop or reads an ambient clock;
|
||||
tests build their own `std.Io.Threaded` instance and pass it the same way.
|
||||
|
||||
The one deliberate exception is `storage/db.zig`: SQLite performs its own file
|
||||
I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's
|
||||
VFS to route through `std.Io` would be a large amount of C-boundary code to
|
||||
make one dependency match a convention it does not need.
|
||||
|
||||
## Life of one query
|
||||
|
||||
The pipeline lives in `src/server/handler.zig` — `Handler.handle` does
|
||||
validation and setup, then `Context.run` decides the answer. Its order is
|
||||
PLAN §4; the stages below are the code's actual call chain:
|
||||
|
||||
```
|
||||
UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig)
|
||||
└──────┴──────┴────┘
|
||||
│ raw query bytes, listener-owned buffers
|
||||
▼
|
||||
handler.handle
|
||||
├─ header parse (too short / QR set → counted drop)
|
||||
├─ rate limit (over budget → REFUSED)
|
||||
├─ packet + EDNS validation (FORMERR / NOTIMP, qdcount != 1 → FORMERR)
|
||||
├─ one clock read, client tracking, snapshot + local-table acquire
|
||||
├─ group lookup (snapshot.groupForClient)
|
||||
│
|
||||
└─ Context.run
|
||||
├─ qclass != IN ──► upstream, unfiltered and uncached
|
||||
├─ local records ────────────────► authoritative answer
|
||||
├─ forward zones ─► cache ─► LAN resolver ─► cache put ─► answer
|
||||
│
|
||||
└─ upstream path (filtering off while paused)
|
||||
├─ filter snapshot evaluate ─► blocked? synthesized block reply
|
||||
├─ safe-search rewrite (per group)
|
||||
├─ cache get ─► hit? answer
|
||||
├─ upstream pool: priority failover across DoH/DoT endpoints
|
||||
├─ CNAME uncloak: walk the answer's chain, re-evaluate each target
|
||||
└─ cache put (skipped for safe-search answers)
|
||||
▼
|
||||
reply: UDP size check ─► TC=1 if over the limit ─► listener sends
|
||||
│
|
||||
└─► QuerySink ─► SSE hub (GET /api/queries/live)
|
||||
└─► async logger ─► querylog.db
|
||||
```
|
||||
|
||||
Local records win over forward zones, and both win over filtering: a name
|
||||
nxdns answers itself never reaches a blocklist. Pause suspends filtering only;
|
||||
local records, forward zones, cache, upstream and the query log keep running,
|
||||
which is what makes pause safe to hand to a household member. A question whose
|
||||
class is not IN bypasses local answers, filtering and the cache entirely and
|
||||
goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and
|
||||
declines to cache answers it does not model.
|
||||
|
||||
`handle` returns no error union. Every failure is either a DNS response the
|
||||
client can act on or a counted drop, because there is no caller above it that
|
||||
could do anything useful with a Zig error. Two consequences are worth knowing:
|
||||
answers synthesized from a safe-search rewrite are never cached (the rewrite is
|
||||
per group, and the cache is not), and a SERVFAIL reply short-circuits before the
|
||||
query sink, so it appears in the counters but not in the query log.
|
||||
|
||||
The query path never waits on the database. `QuerySink` copies the entry, the
|
||||
SSE hub gets it first, and one writer task owns the `querylog.db` handle behind
|
||||
an `std.Io.Queue`. A slow disk delays logging, never resolution.
|
||||
|
||||
Two smaller decisions in the same spirit: the upstream pool makes a second
|
||||
pass that ignores backoff, so "every endpoint is in backoff" degrades to
|
||||
trying anyway rather than to a blanket SERVFAIL; and a handler that has no
|
||||
filter snapshot yet answers unfiltered rather than refusing. Both prefer a
|
||||
working resolver over a correct-looking failure.
|
||||
|
||||
## Storage
|
||||
|
||||
Two databases with opposite contracts, in one data directory (see
|
||||
[reference/files-and-directories.md](../reference/files-and-directories.md)).
|
||||
|
||||
**`config.db` is the truth.** Its schema is versioned: `migrations.zig` holds
|
||||
an ordered list of steps, step 1 being the verbatim DDL from
|
||||
`config_schema.zig`, each applied inside one transaction. `nxdns import`
|
||||
replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed
|
||||
import changes nothing; `nxdns export` renders it back as canonical ZON,
|
||||
byte-identical across round trips. A config file seeds this database exactly
|
||||
once at first start. Why it works that way is
|
||||
[configuration-model.md](configuration-model.md).
|
||||
|
||||
**`querylog.db` is expendable.** It is never migrated. Its schema carries a
|
||||
fingerprint derived from the DDL text, and at open, a missing, corrupt,
|
||||
non-database, `quick_check`-failing or fingerprint-mismatched file is moved
|
||||
aside and recreated empty — the old file is kept under a new name rather than
|
||||
deleted, so an operator can still look at it. Retention deletes old rows daily
|
||||
and periodically rewrites the file to reclaim space.
|
||||
|
||||
The split exists so that the churn of the second database can never endanger
|
||||
the first. Query logs are high-volume, disposable, and the thing most likely
|
||||
to be corrupted by a power cut on an SD card; configuration is small,
|
||||
irreplaceable, and the thing an operator would have to reconstruct by hand.
|
||||
Giving them one file would force the careful contract onto the noisy data or
|
||||
the loose contract onto the valuable data.
|
||||
|
||||
## Web stack
|
||||
|
||||
`web/server.zig` runs one `std.http.Server` per connection over its own accept
|
||||
loop, with a fixed set of pre-allocated connection slots, optionally behind
|
||||
TLS. Over capacity it answers 503 rather than queueing without bound — the
|
||||
admin UI is not the product, and it must not be able to starve DNS.
|
||||
|
||||
The SPA is embedded at build time: `static.zig` serves the `web_assets` module
|
||||
— bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where
|
||||
it paid off — via a linear scan with no filesystem access at runtime. (The one
|
||||
exception is `nxdns run --web-dev DIR`, which serves from disk with no cache
|
||||
headers, for developing the SPA against a running server.) `GET
|
||||
/api/queries/live` is server-sent events over chunked transfer, fed by the
|
||||
same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`)
|
||||
matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML
|
||||
is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest
|
||||
by tests that assert every served route appears in it.
|
||||
|
||||
Authentication (`web/auth.zig`): the operator's password is verified against
|
||||
an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import
|
||||
and never stored). A successful login mints a 256-bit token carried in a
|
||||
cookie; the in-memory session table holds only SHA-256 digests of tokens,
|
||||
compared in constant time, capped at 32 sessions with LRU eviction. Nothing is
|
||||
persisted, so a restart logs everyone out — for a household LAN that is a
|
||||
feature, not a gap. Unauthenticated by design: the monitoring endpoints
|
||||
(health, version, metrics), the served OpenAPI contract, login itself, and the
|
||||
static SPA assets, which the router hands to the SPA fallback before any auth
|
||||
check. Everything else requires the cookie, and the API has its own
|
||||
token-bucket rate limiter. See
|
||||
[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
|
||||
|
||||
## DoH, DoT and certificate hot-reload
|
||||
|
||||
`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and
|
||||
`server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server
|
||||
TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as
|
||||
`std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether
|
||||
they are encrypted.
|
||||
|
||||
Certificates hot-reload through `server/cert_store.zig`. One refcounted
|
||||
`CertStore` per endpoint owns the published TLS context generation; listeners
|
||||
`acquire` it per connection and `release` it when the connection ends, so a
|
||||
reload never frees a context mid-handshake. Reload publishes nothing on
|
||||
failure: both PEM files are read and a whole new context is built before
|
||||
anything swaps, and any failure leaves the old generation serving. A watcher
|
||||
polls mtime and size of both files every 30 seconds
|
||||
(`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same
|
||||
path on demand and reports the per-endpoint outcome as its payload.
|
||||
|
||||
The requirement driving all of this is that a certbot renewal must not need a
|
||||
restart and must not be able to break DNS. A half-swapped context or a
|
||||
free-while-in-use would do exactly that, so the store is built so neither is
|
||||
representable. See
|
||||
[how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md).
|
||||
|
||||
## Failure visibility
|
||||
|
||||
Every failure mode must be visible, and the surface is counters, not log lines
|
||||
(AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they
|
||||
are only read after someone already suspects a problem, and on an SD card they
|
||||
are a way to fill a disk.
|
||||
|
||||
So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP,
|
||||
REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and
|
||||
forward-zone answers, safe-search rewrites, paused and unfiltered queries, and
|
||||
the tracker-full condition. Listeners count dropped datagrams instead of
|
||||
queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text
|
||||
0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers
|
||||
200: "degraded" is a fact about the box, not a failed request, and a monitor
|
||||
that cannot distinguish the two is worse than no monitor.
|
||||
|
||||
The disk monitor classifies free space against thresholds and gates
|
||||
non-essential writes; the query logger holds its batches while writes are
|
||||
disallowed rather than dropping them silently or writing until the filesystem
|
||||
fills. `std.log` is reserved for failures nobody else records, with
|
||||
upstream-error deduplication so a flapping resolver cannot fill a disk with
|
||||
identical lines.
|
||||
Reference in New Issue
Block a user