mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 13 Activity

docs: unwrap hand-wrapped prose repo-wide
Gates / frontend (push) Successful in 1m2s
Details
Gates / test (push) Successful in 1m38s
Details
Gates / package (push) Successful in 5m5s
Details
Gates / test-aarch64 (push) Successful in 6m30s
Details
Gates / container (push) Successful in 15s
Details
CI / gates (push) Successful in 13m30s
Details

Browse Source
This commit is contained in:
mokhtar
2026-08-15 16:27:36 +02:00
parent 50b8fd5c61
commit 5b3d1cd65c
48 changed files with 2691 additions and 11699 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/README.md
+22 -46
View File
@@ -1,8 +1,6 @@
# nxdns documentation
The pages are split by what you are trying to do, following
[Diátaxis](https://diataxis.fr/). Each page serves one of four purposes, and
knowing which one you want is the fastest way to the right page.
The pages are split by what you are trying to do, following [Diátaxis](https://diataxis.fr/). Each page serves one of four purposes, and knowing which one you want is the fastest way to the right page.
| Mode | For | Read it when |
| --- | --- | --- |
@@ -13,62 +11,40 @@ knowing which one you want is the fastest way to the right page.
## Tutorial
A lesson, not a procedure: one path with one outcome, on a scratch directory you
can delete afterwards.
A lesson, not a procedure: one path with one outcome, on a scratch directory you can delete afterwards.
- [tutorial/first-run.md](tutorial/first-run.md) — build nxdns, resolve a name,
block a domain from a real blocklist, open the web interface, stop cleanly.
- [tutorial/first-run.md](tutorial/first-run.md) — build nxdns, resolve a name, block a domain from a real blocklist, open the web interface, stop cleanly.
## How-to guides
Steps for a goal you already have. They assume you know what nxdns is.
- [how-to/verify-a-release.md](how-to/verify-a-release.md) — check the
signature and the checksums before you run anything, and what they prove.
- [how-to/install-with-systemd.md](how-to/install-with-systemd.md) — a real
install as a system service, including the Raspberry Pi 5 aarch64 binary.
- [how-to/install-with-docker.md](how-to/install-with-docker.md) — the published
container image and the compose file.
- [how-to/upgrade.md](how-to/upgrade.md) — move to a new release without losing
state.
- [how-to/troubleshoot.md](how-to/troubleshoot.md) — what to do when it does not
answer, does not block, or will not start.
- [how-to/enable-doh-and-dot.md](how-to/enable-doh-and-dot.md) — serve encrypted
DNS with certificates.
- [how-to/set-up-admin-authentication.md](how-to/set-up-admin-authentication.md)
— put a password on the web interface and the API.
- [how-to/back-up-and-restore.md](how-to/back-up-and-restore.md) — export and
import the configuration, and what to copy.
- [how-to/measure-performance.md](how-to/measure-performance.md) — run the
benchmark harness on your own hardware.
- [how-to/verify-a-release.md](how-to/verify-a-release.md) — check the signature and the checksums before you run anything, and what they prove.
- [how-to/install-with-systemd.md](how-to/install-with-systemd.md) — a real install as a system service, including the Raspberry Pi 5 aarch64 binary.
- [how-to/install-with-docker.md](how-to/install-with-docker.md) — the published container image and the compose file.
- [how-to/upgrade.md](how-to/upgrade.md) — move to a new release without losing state.
- [how-to/troubleshoot.md](how-to/troubleshoot.md) — what to do when it does not answer, does not block, or will not start.
- [how-to/enable-doh-and-dot.md](how-to/enable-doh-and-dot.md) — serve encrypted DNS with certificates.
- [how-to/set-up-admin-authentication.md](how-to/set-up-admin-authentication.md) — put a password on the web interface and the API.
- [how-to/back-up-and-restore.md](how-to/back-up-and-restore.md) — export and import the configuration, and what to copy.
- [how-to/measure-performance.md](how-to/measure-performance.md) — run the benchmark harness on your own hardware.
## Reference
Descriptions of what is there. No procedures, no advice.
- [reference/configuration.md](reference/configuration.md) — every
configuration section, field, default and range.
- [reference/api.md](reference/api.md) — every REST route, authentication and
the event stream.
- [reference/cli.md](reference/cli.md) — the six subcommands, every flag, every
exit code.
- [reference/files-and-directories.md](reference/files-and-directories.md) — the
data directory layout and file modes.
- [reference/performance.md](reference/performance.md) — the targets and the
measured numbers.
- [reference/configuration.md](reference/configuration.md) — every configuration section, field, default and range.
- [reference/api.md](reference/api.md) — every REST route, authentication and the event stream.
- [reference/cli.md](reference/cli.md) — the six subcommands, every flag, every exit code.
- [reference/files-and-directories.md](reference/files-and-directories.md) — the data directory layout and file modes.
- [reference/performance.md](reference/performance.md) — the targets and the measured numbers.
## Explanation
Background. Nothing here is needed to operate nxdns; it is here so the decisions
are inspectable.
Background. Nothing here is needed to operate nxdns; it is here so the decisions are inspectable.
- [explanation/architecture.md](explanation/architecture.md) — the module map
and the design it comes from.
- [explanation/configuration-model.md](explanation/configuration-model.md) — why
there are two authority modes, how each one is selected, and what each is for.
- [explanation/performance-and-testing.md](explanation/performance-and-testing.md)
— why the targets exist, why CI does not gate on them, and what the hermetic
tests do and do not prove.
- [explanation/architecture.md](explanation/architecture.md) — the module map and the design it comes from.
- [explanation/configuration-model.md](explanation/configuration-model.md) — why there are two authority modes, how each one is selected, and what each is for.
- [explanation/performance-and-testing.md](explanation/performance-and-testing.md) — why the targets exist, why CI does not gate on them, and what the hermetic tests do and do not prove.
Scope and per-milestone contracts live outside this directory, in
[../PLAN.md](../PLAN.md) and [../specs/](../specs/).
Scope and per-milestone contracts live outside this directory, in [../PLAN.md](../PLAN.md) and [../specs/](../specs/).
docs/explanation/architecture.md
+28 -154
View File
@@ -1,15 +1,8 @@
# Architecture
nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary
that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against
blocklists, and serves an embedded admin SPA over HTTP. This page maps the
source tree and explains the few design rules that hold everywhere, and why
they are the rules.
nxdns is a self-hosted DNS sinkhole for a household LAN: one static Zig binary that answers DNS on UDP/TCP 53 (optionally DoH and DoT), filters against blocklists, and serves an embedded admin SPA over HTTP. This page maps the source tree and explains the few design rules that hold everywhere, and why they are the rules.
For what the configuration fields, API routes and CLI flags actually are, see
[reference/configuration.md](../reference/configuration.md),
[reference/api.md](../reference/api.md) and
[reference/cli.md](../reference/cli.md). This page does not repeat them.
For what the configuration fields, API routes and CLI flags actually are, see [reference/configuration.md](../reference/configuration.md), [reference/api.md](../reference/api.md) and [reference/cli.md](../reference/cli.md). This page does not repeat them.
## Module map
@@ -39,8 +32,7 @@ Directories:
| `src/web/` | The admin HTTP layer: `server.zig` (listener), `router.zig`/`routes.zig`, one file per resource under `handlers/`, `auth.zig` (sessions), `sse.zig` (live query fanout), `static.zig` (embedded SPA), `metrics.zig` (Prometheus), `openapi.zig` (served contract), `api_limiter.zig`, `http_util.zig`. |
| `src/platform/` | OS and TLS edges: IP address values, the `std.log` sink (`logging.zig`), `statfs.zig` (free-space query via libc), client TLS over `std.crypto.tls` (`tls_client.zig`), server TLS over vendored Mbed TLS (`tls_server.zig`). |
The SPA source lives in `web/` at the repo root; the build embeds its `dist/`
output as the `web_assets` module (`-Dweb-dist`).
The SPA source lives in `web/` at the repo root; the build embeds its `dist/` output as the `web_assets` module (`-Dweb-dist`).
```
main.zig ── cli.zig ── app.zig (composition root)
@@ -58,48 +50,23 @@ main.zig ── cli.zig ── app.zig (composition root)
## The purity rule
`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no
`std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs
a timestamp takes it as a parameter — the cache, the rate limiter and the
pause flag all work this way, so every decision is testable without a backend.
`dns/`, `filter/`, `local/` and `cache/` take bytes and return bytes: no `std.Io`, no sockets, no clocks hidden inside (AGENTS.md). Anything that needs a timestamp takes it as a parameter — the cache, the rate limiter and the pause flag all work this way, so every decision is testable without a backend.
The point is not purity for its own sake. A decision that depends on a hidden
clock or a hidden socket can only be tested by arranging the world around it;
one that takes the clock as an argument is tested by passing a number. The
whole filtering and caching pipeline can therefore be exercised in the plain,
network-free test suite, which is what makes that suite worth gating CI on.
The point is not purity for its own sake. A decision that depends on a hidden clock or a hidden socket can only be tested by arranging the world around it; one that takes the clock as an argument is tested by passing a number. The whole filtering and caching pipeline can therefore be exercised in the plain, network-free test suite, which is what makes that suite worth gating CI on.
The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads
lists, `filter/manager.zig` owns the compiled files, the DB columns and the
snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN
resolver. Those three files are the only ones under those four directories
that take a `std.Io`. The decision path a query takes through them allocates
nothing and opens nothing.
The exceptions are deliberate, few, and named: `filter/fetcher.zig` downloads lists, `filter/manager.zig` owns the compiled files, the DB columns and the snapshot swap, and `local/forward_client.zig` speaks UDP/TCP to a LAN resolver. Those three files are the only ones under those four directories that take a `std.Io`. The decision path a query takes through them allocates nothing and opens nothing.
The honest cost of the exceptions shows up in
[performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is
where the one production crash came from, precisely because it is the file the
pure suite cannot reach.
The honest cost of the exceptions shows up in [performance-and-testing.md](performance-and-testing.md): `fetcher.zig` is where the one production crash came from, precisely because it is the file the pure suite cannot reach.
## std.Io injection
There is one `std.Io` in the process. `main` receives it through
`std.process.Init` — on the standard start path this is the Threaded backend
(`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to
`cli.Runner`, from which `app.zig` threads it into every collaborator as a
parameter. No module constructs its own event loop or reads an ambient clock;
tests build their own `std.Io.Threaded` instance and pass it the same way.
There is one `std.Io` in the process. `main` receives it through `std.process.Init` — on the standard start path this is the Threaded backend (`std.Io.Threaded`, constructed in the stdlib's start code) — and hands it to `cli.Runner`, from which `app.zig` threads it into every collaborator as a parameter. No module constructs its own event loop or reads an ambient clock; tests build their own `std.Io.Threaded` instance and pass it the same way.
The one deliberate exception is `storage/db.zig`: SQLite performs its own file
I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's
VFS to route through `std.Io` would be a large amount of C-boundary code to
make one dependency match a convention it does not need.
The one deliberate exception is `storage/db.zig`: SQLite performs its own file I/O through its VFS, so that file takes no `std.Io` at all. Wrapping SQLite's VFS to route through `std.Io` would be a large amount of C-boundary code to make one dependency match a convention it does not need.
## Life of one query
The pipeline lives in `src/server/handler.zig` — `Handler.handle` does
validation and setup, then `Context.run` decides the answer. Its order is
PLAN §4; the stages below are the code's actual call chain:
The pipeline lives in `src/server/handler.zig` — `Handler.handle` does validation and setup, then `Context.run` decides the answer. Its order is PLAN §4; the stages below are the code's actual call chain:
```
UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig)
@@ -132,139 +99,46 @@ UDP/53 TCP/53 DoH DoT (src/server/{udp,tcp,doh,dot}_server.zig)
└─► async logger ─► querylog.db
```
Local records win over forward zones, and both win over filtering: a name
nxdns answers itself never reaches a blocklist. Pause suspends filtering only;
local records, forward zones, cache, upstream and the query log keep running,
which is what makes pause safe to hand to a household member. A question whose
class is not IN bypasses local answers, filtering and the cache entirely and
goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and
declines to cache answers it does not model.
Local records win over forward zones, and both win over filtering: a name nxdns answers itself never reaches a blocklist. Pause suspends filtering only; local records, forward zones, cache, upstream and the query log keep running, which is what makes pause safe to hand to a household member. A question whose class is not IN bypasses local answers, filtering and the cache entirely and goes straight upstream — nxdns has no opinion about CHAOS or HESIOD names and declines to cache answers it does not model.
`handle` returns no error union. Every failure is either a DNS response the
client can act on or a counted drop, because there is no caller above it that
could do anything useful with a Zig error. Two consequences are worth knowing:
answers synthesized from a safe-search rewrite are never cached (the rewrite is
per group, and the cache is not), and a SERVFAIL reply short-circuits before the
query sink, so it appears in the counters but not in the query log.
`handle` returns no error union. Every failure is either a DNS response the client can act on or a counted drop, because there is no caller above it that could do anything useful with a Zig error. Two consequences are worth knowing: answers synthesized from a safe-search rewrite are never cached (the rewrite is per group, and the cache is not), and a SERVFAIL reply short-circuits before the query sink, so it appears in the counters but not in the query log.
The query path never waits on the database. `QuerySink` copies the entry, the
SSE hub gets it first, and one writer task owns the `querylog.db` handle behind
an `std.Io.Queue`. A slow disk delays logging, never resolution.
The query path never waits on the database. `QuerySink` copies the entry, the SSE hub gets it first, and one writer task owns the `querylog.db` handle behind an `std.Io.Queue`. A slow disk delays logging, never resolution.
Two smaller decisions in the same spirit: the upstream pool makes a second
pass that ignores backoff, so "every endpoint is in backoff" degrades to
trying anyway rather than to a blanket SERVFAIL; and a handler that has no
filter snapshot yet answers unfiltered rather than refusing. Both prefer a
working resolver over a correct-looking failure.
Two smaller decisions in the same spirit: the upstream pool makes a second pass that ignores backoff, so "every endpoint is in backoff" degrades to trying anyway rather than to a blanket SERVFAIL; and a handler that has no filter snapshot yet answers unfiltered rather than refusing. Both prefer a working resolver over a correct-looking failure.
## Storage
Two databases with opposite contracts, in one data directory (see
[reference/files-and-directories.md](../reference/files-and-directories.md)).
Two databases with opposite contracts, in one data directory (see [reference/files-and-directories.md](../reference/files-and-directories.md)).
**`config.db` is what the server reads.** Its schema is versioned: `migrations.zig` holds
an ordered list of steps, step 1 being the verbatim DDL from
`config_schema.zig`, each applied inside one transaction. `nxdns import`
replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed
import changes nothing; `nxdns export` renders it back as canonical ZON,
byte-identical across round trips.
**`config.db` is what the server reads.** Its schema is versioned: `migrations.zig` holds an ordered list of steps, step 1 being the verbatim DDL from `config_schema.zig`, each applied inside one transaction. `nxdns import` replaces the whole content atomically under `BEGIN IMMEDIATE`, so a failed import changes nothing; `nxdns export` renders it back as canonical ZON, byte-identical across round trips.
Which of the file and the database is *authoritative* is chosen by the
invocation, not by state: bare `nxdns run` serves the database, and
`nxdns run --config FILE` makes the file authoritative and reconciles the
database onto it at every start. `reconcile.zig` is that convergence, matching
rows by identity and writing only differences, so runtime state — blocklist
checksums, compiled snapshots, client history — survives. Why it works that way
is [configuration-model.md](configuration-model.md).
Which of the file and the database is *authoritative* is chosen by the invocation, not by state: bare `nxdns run` serves the database, and `nxdns run --config FILE` makes the file authoritative and reconciles the database onto it at every start. `reconcile.zig` is that convergence, matching rows by identity and writing only differences, so runtime state — blocklist checksums, compiled snapshots, client history — survives. Why it works that way is [configuration-model.md](configuration-model.md).
**`querylog.db` is expendable.** It is never migrated. Its schema carries a
fingerprint derived from the DDL text, and at open, a missing, corrupt,
non-database, `quick_check`-failing or fingerprint-mismatched file is moved
aside and recreated empty — the old file is kept under a new name rather than
deleted, so an operator can still look at it. Retention deletes old rows daily
and periodically rewrites the file to reclaim space.
**`querylog.db` is expendable.** It is never migrated. Its schema carries a fingerprint derived from the DDL text, and at open, a missing, corrupt, non-database, `quick_check`-failing or fingerprint-mismatched file is moved aside and recreated empty — the old file is kept under a new name rather than deleted, so an operator can still look at it. Retention deletes old rows daily and periodically rewrites the file to reclaim space.
The split exists so that the churn of the second database can never endanger
the first. Query logs are high-volume, disposable, and the thing most likely
to be corrupted by a power cut on an SD card; configuration is small,
irreplaceable, and the thing an operator would have to reconstruct by hand.
Giving them one file would force the careful contract onto the noisy data or
the loose contract onto the valuable data.
The split exists so that the churn of the second database can never endanger the first. Query logs are high-volume, disposable, and the thing most likely to be corrupted by a power cut on an SD card; configuration is small, irreplaceable, and the thing an operator would have to reconstruct by hand. Giving them one file would force the careful contract onto the noisy data or the loose contract onto the valuable data.
## Web stack
`web/server.zig` runs one `std.http.Server` per connection over its own accept
loop, with a fixed set of pre-allocated connection slots, optionally behind
TLS. Over capacity it answers 503 rather than queueing without bound — the
admin UI is not the product, and it must not be able to starve DNS.
`web/server.zig` runs one `std.http.Server` per connection over its own accept loop, with a fixed set of pre-allocated connection slots, optionally behind TLS. Over capacity it answers 503 rather than queueing without bound — the admin UI is not the product, and it must not be able to starve DNS.
The SPA is embedded at build time: `static.zig` serves the `web_assets` module
— bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where
it paid off — via a linear scan with no filesystem access at runtime. (The one
exception is `nxdns run --web-dev DIR`, which serves from disk with no cache
headers, for developing the SPA against a running server.) `GET
/api/queries/live` is server-sent events over chunked transfer, fed by the
same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`)
matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML
is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest
by tests that assert every served route appears in it.
The SPA is embedded at build time: `static.zig` serves the `web_assets` module — bytes, content type, strong ETag, and a pre-compressed `.gz` sibling where it paid off — via a linear scan with no filesystem access at runtime. (The one exception is `nxdns run --web-dev DIR`, which serves from disk with no cache headers, for developing the SPA against a running server.) `GET /api/queries/live` is server-sent events over chunked transfer, fed by the same `QuerySink` the logger reads. Routing is a flat table (`routes.zig`) matched linearly; a few dozen routes do not justify a trie. The OpenAPI YAML is hand-written, embedded and served at `GET /api/openapi.yaml`, kept honest by tests that assert every served route appears in it.
Authentication (`web/auth.zig`): the operator's password is verified against
an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import
and never stored). A successful login mints a 256-bit token carried in a
cookie; the in-memory session table holds only SHA-256 digests of tokens,
compared in constant time, capped at 32 sessions with LRU eviction. Nothing is
persisted, so a restart logs everyone out — for a household LAN that is a
feature, not a gap. Unauthenticated by design: the monitoring endpoints
(health, version, metrics), the served OpenAPI contract, login itself, and the
static SPA assets, which the router hands to the SPA fallback before any auth
check. Everything else requires the cookie, and the API has its own
token-bucket rate limiter. See
[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
Authentication (`web/auth.zig`): the operator's password is verified against an argon2id PHC string (`web.password_hash`; the plaintext is hashed on import and never stored). A successful login mints a 256-bit token carried in a cookie; the in-memory session table holds only SHA-256 digests of tokens, compared in constant time, capped at 32 sessions with LRU eviction. Nothing is persisted, so a restart logs everyone out — for a household LAN that is a feature, not a gap. Unauthenticated by design: the monitoring endpoints (health, version, metrics), the served OpenAPI contract, login itself, and the static SPA assets, which the router hands to the SPA fallback before any auth check. Everything else requires the cookie, and the API has its own token-bucket rate limiter. See [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
## DoH, DoT and certificate hot-reload
`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and
`server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server
TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as
`std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether
they are encrypted.
`server/doh_server.zig` (RFC 8484 over HTTP/1.1 and TLS) and `server/dot_server.zig` (RFC 7858) mirror the plain listeners' shape. Server TLS terminates in Mbed TLS (`platform/tls_server.zig`), exposing plaintext as `std.Io.Reader`/`std.Io.Writer`, so the listeners above it do not know whether they are encrypted.
Certificates hot-reload through `server/cert_store.zig`. One refcounted
`CertStore` per endpoint owns the published TLS context generation; listeners
`acquire` it per connection and `release` it when the connection ends, so a
reload never frees a context mid-handshake. Reload publishes nothing on
failure: both PEM files are read and a whole new context is built before
anything swaps, and any failure leaves the old generation serving. A watcher
polls mtime and size of both files every 30 seconds
(`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same
path on demand and reports the per-endpoint outcome as its payload.
Certificates hot-reload through `server/cert_store.zig`. One refcounted `CertStore` per endpoint owns the published TLS context generation; listeners `acquire` it per connection and `release` it when the connection ends, so a reload never frees a context mid-handshake. Reload publishes nothing on failure: both PEM files are read and a whole new context is built before anything swaps, and any failure leaves the old generation serving. A watcher polls mtime and size of both files every 30 seconds (`cert_store.poll_interval_s`); `POST /api/certs/reload` triggers the same path on demand and reports the per-endpoint outcome as its payload.
The requirement driving all of this is that a certbot renewal must not need a
restart and must not be able to break DNS. A half-swapped context or a
free-while-in-use would do exactly that, so the store is built so neither is
representable. See
[how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md).
The requirement driving all of this is that a certbot renewal must not need a restart and must not be able to break DNS. A half-swapped context or a free-while-in-use would do exactly that, so the store is built so neither is representable. See [how-to/enable-doh-and-dot.md](../how-to/enable-doh-and-dot.md).
## Failure visibility
Every failure mode must be visible, and the surface is counters, not log lines
(AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they
are only read after someone already suspects a problem, and on an SD card they
are a way to fill a disk.
Every failure mode must be visible, and the surface is counters, not log lines (AGENTS.md). Log lines are a bad primitive for this: they are unbounded, they are only read after someone already suspects a problem, and on an SD card they are a way to fill a disk.
So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP,
REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and
forward-zone answers, safe-search rewrites, paused and unfiltered queries, and
the tracker-full condition. Listeners count dropped datagrams instead of
queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text
0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers
200: "degraded" is a fact about the box, not a failed request, and a monitor
that cannot distinguish the two is worse than no monitor.
So the handler counts every outcome in atomics — drops, FORMERR, NOTIMP, REFUSED, SERVFAIL, blocked, uncloak-blocked, truncated, cache hits, local and forward-zone answers, safe-search rewrites, paused and unfiltered queries, and the tracker-full condition. Listeners count dropped datagrams instead of queueing them unboundedly. `GET /metrics` renders all of it as Prometheus text 0.0.4, and `GET /api/health` rolls it up for a monitor. Health always answers 200: "degraded" is a fact about the box, not a failed request, and a monitor that cannot distinguish the two is worse than no monitor.
The disk monitor classifies free space against thresholds and gates
non-essential writes; the query logger holds its batches while writes are
disallowed rather than dropping them silently or writing until the filesystem
fills. `std.log` is reserved for failures nobody else records, with
upstream-error deduplication so a flapping resolver cannot fill a disk with
identical lines.
The disk monitor classifies free space against thresholds and gates non-essential writes; the query logger holds its batches while writes are disallowed rather than dropping them silently or writing until the filesystem fills. `std.log` is reserved for failures nobody else records, with upstream-error deduplication so a flapping resolver cannot fill a disk with identical lines.
docs/explanation/configuration-model.md
+39 -165
View File
@@ -1,13 +1,8 @@
# The configuration model
nxdns is configured two ways — a ZON file and a web UI — and only one of them
can be the truth at a time. This page explains how that choice is made, what
each mode is for, and why the design leaves the fewest ways to lose an
operator's work.
nxdns is configured two ways — a ZON file and a web UI — and only one of them can be the truth at a time. This page explains how that choice is made, what each mode is for, and why the design leaves the fewest ways to lose an operator's work.
For the fields themselves see
[reference/configuration.md](../reference/configuration.md); for the commands
and their exit codes see [reference/cli.md](../reference/cli.md).
For the fields themselves see [reference/configuration.md](../reference/configuration.md); for the commands and their exit codes see [reference/cli.md](../reference/cli.md).
## The rule
@@ -18,37 +13,19 @@ nxdns run the database is the truth
nxdns run --config /etc/nxdns/config.zon the file is the truth
```
That is the entire selection mechanism. There is no mode setting, no default
file path, and nothing recorded in the database about which mode last wrote it.
A `config.zon` that exists but that no invocation names changes nothing at all.
That is the entire selection mechanism. There is no mode setting, no default file path, and nothing recorded in the database about which mode last wrote it. A `config.zon` that exists but that no invocation names changes nothing at all.
Two properties fall out of that, and both were chosen on purpose.
**An operator can read `ExecStart` and know which authority is live.** The
alternative — probe a well-known path, and behave differently depending on
whether a file happens to be there — is ambient magic. It is also the exact
class of rule that produced years of documentation lies in this project: the
old design read the file only while the database was empty, which meant the same
command did two different things depending on state nobody could see from the
command line, and every page that described it eventually described it wrongly.
**An operator can read `ExecStart` and know which authority is live.** The alternative — probe a well-known path, and behave differently depending on whether a file happens to be there — is ambient magic. It is also the exact class of rule that produced years of documentation lies in this project: the old design read the file only while the database was empty, which meant the same command did two different things depending on state nobody could see from the command line, and every page that described it eventually described it wrongly.
**A path already expresses a two-state choice, so a mode flag beside it would
be redundant and worse.** An earlier draft had `--config-source=db|file`. A mode
flag next to a path flag manufactures combinations that cannot mean anything —
a path with no mode, a mode with no path — and each one then needs a pairing
rule and a usage error to defend it. Presence-of-path has no invalid
combinations, so there is nothing to defend.
**A path already expresses a two-state choice, so a mode flag beside it would be redundant and worse.** An earlier draft had `--config-source=db|file`. A mode flag next to a path flag manufactures combinations that cannot mean anything — a path with no mode, a mode with no path — and each one then needs a pairing rule and a usage error to defend it. Presence-of-path has no invalid combinations, so there is nothing to defend.
## Database mode
`nxdns run`. `config.db` holds the configuration; the UI, the API and `nxdns
import` write to it; nothing reads a file. This is the appliance: someone sets
the box up once, and afterwards the household member who wants to unblock one
domain clicks a button.
`nxdns run`. `config.db` holds the configuration; the UI, the API and `nxdns import` write to it; nothing reads a file. This is the appliance: someone sets the box up once, and afterwards the household member who wants to unblock one domain clicks a button.
A fresh install in this mode starts from an empty database, which fails
validation on its own terms — there is nowhere to forward a query to — and says
what to do about it:
A fresh install in this mode starts from an empty database, which fails validation on its own terms — there is nowhere to forward a query to — and says what to do about it:
```
nxdns run failed: NoUsableUpstreams
@@ -58,188 +35,85 @@ load one with `nxdns import <file>`, or make a file the source of truth with `nx
## File mode
`nxdns run --config FILE`. The file is the sole declarative source, and the
database becomes the runtime substrate: every start reads the file, validates
it, converges the database onto it, and serves from there. Configuration writes
through the API are refused with a 403.
`nxdns run --config FILE`. The file is the sole declarative source, and the database becomes the runtime substrate: every start reads the file, validates it, converges the database onto it, and serves from there. Configuration writes through the API are refused with a 403.
This is the mode for a file kept in git and pushed by Ansible. What it buys is
that the deployed file is what is running — not "was imported once", not
"was imported unless someone clicked something since".
This is the mode for a file kept in git and pushed by Ansible. What it buys is that the deployed file is what is running — not "was imported once", not "was imported unless someone clicked something since".
Three properties make it usable rather than merely correct.
**It fails closed.** A file that is missing, unreadable, unparseable, oversized
or invalid stops the start. nxdns never falls back to the database, because a
fallback turns a deploy typo into a configuration that is silently months old
and looks fine. That failure is exit 2, so `nxdns check --config FILE` is a real
pre-restart gate: validate the pushed file in the handler, and a typo is a
failed deploy at noon rather than a dead resolver at the next power cut.
**It fails closed.** A file that is missing, unreadable, unparseable, oversized or invalid stops the start. nxdns never falls back to the database, because a fallback turns a deploy typo into a configuration that is silently months old and looks fine. That failure is exit 2, so `nxdns check --config FILE` is a real pre-restart gate: validate the pushed file in the handler, and a typo is a failed deploy at noon rather than a dead resolver at the next power cut.
**It converges rather than replaces.** Reconciling matches rows by identity and
writes only what differs. A source whose URL has not changed keeps its row id,
its checksum, its counters and its compiled blocklist files — so a restart in
file mode downloads nothing, which is the difference between a design that is
tolerable to restart and one that costs three minutes and 100 MB every time.
**It converges rather than replaces.** Reconciling matches rows by identity and writes only what differs. A source whose URL has not changed keeps its row id, its checksum, its counters and its compiled blocklist files — so a restart in file mode downloads nothing, which is the difference between a design that is tolerable to restart and one that costs three minutes and 100 MB every time.
**An unchanged file writes nothing at all.** Not "writes the same bytes" —
performs zero write statements, and reports it:
**An unchanged file writes nothing at all.** Not "writes the same bytes" — performs zero write statements, and reports it:
```
reconciled '/etc/nxdns/config.zon': no changes
```
That matters beyond elegance. A box whose SD card is full of query log can still
restart in file mode, because a no-op reconcile needs no write-ahead-log
headroom.
That matters beyond elegance. A box whose SD card is full of query log can still restart in file mode, because a no-op reconcile needs no write-ahead-log headroom.
**Converged at every boot is not a lock between boots.** Nothing stops `nxdns
import` or a `runtime action` route from moving the database while the server
runs. The contract is that the next start puts it back, and says what it
corrected.
**Converged at every boot is not a lock between boots.** Nothing stops `nxdns import` or a `runtime action` route from moving the database while the server runs. The contract is that the next start puts it back, and says what it corrected.
## What the file cannot take away
The file is authoritative over configuration. It is not authoritative over
things it has no vocabulary for, and reconciling has to preserve those or the
mode is unusable.
The file is authoritative over configuration. It is not authoritative over things it has no vocabulary for, and reconciling has to preserve those or the mode is unusable.
- **Blocklist download state.** Checksums, fetch timestamps and domain counts
belong to the network, not the operator. They survive on every matched row.
- **Client history.** Devices nxdns saw on the wire are kept whole. Naming one
in the file promotes that row in place — it keeps its first-seen and
last-seen and its row id, and counts as an update rather than a delete and an
insert.
- **Devices whose group is un-declared.** Remove a group from the file and the
observed clients assigned to it move to `default`. The operator un-declared
the group, not the devices.
- **The password, when the file does not mention it.** See
[the password](#the-password).
- **Blocklist download state.** Checksums, fetch timestamps and domain counts belong to the network, not the operator. They survive on every matched row.
- **Client history.** Devices nxdns saw on the wire are kept whole. Naming one in the file promotes that row in place — it keeps its first-seen and last-seen and its row id, and counts as an update rather than a delete and an insert.
- **Devices whose group is un-declared.** Remove a group from the file and the observed clients assigned to it move to `default`. The operator un-declared the group, not the devices.
- **The password, when the file does not mention it.** See [the password](#the-password).
The one thing identity cannot survive is a change to identity itself. Edit a
source's URL and the engine sees one row gone and one row arrived: new id, fresh
download, and the old compiled files swept. That is consistent — artifacts are
keyed by row id — and it is why the CLI asks for `--allow-delete` when a diff
deletes anything.
The one thing identity cannot survive is a change to identity itself. Edit a source's URL and the engine sees one row gone and one row arrived: new id, fresh download, and the old compiled files swept. That is consistent — artifacts are keyed by row id — and it is why the CLI asks for `--allow-delete` when a diff deletes anything.
## Why the database is the substrate in both modes
Even in file mode the database is where the server reads its effective
configuration from. That is not a leftover; it is what lets one read path serve
both modes, and it is what makes the runtime state above have somewhere to live.
Even in file mode the database is where the server reads its effective configuration from. That is not a leftover; it is what lets one read path serve both modes, and it is what makes the runtime state above have somewhere to live.
If the file were read directly on every query path there would be no place to
keep a checksum, and no way for the UI to show anything. If both file and
database were authoritative there would be a two-way merge, and a merge cannot
work here: when the UI writes a rule and the file still says otherwise, nothing
in the system knows which of two statements is the newer intention. Every
resolution silently destroys work someone meant to keep.
If the file were read directly on every query path there would be no place to keep a checksum, and no way for the UI to show anything. If both file and database were authoritative there would be a two-way merge, and a merge cannot work here: when the UI writes a rule and the file still says otherwise, nothing in the system knows which of two statements is the newer intention. Every resolution silently destroys work someone meant to keep.
So the modes are exclusive, and the failure mode of each is loud. In database
mode, editing the file does nothing, which you find out the first time you look
at the UI. In file mode, the UI refuses the edit to your face with a message
naming the file to edit instead.
So the modes are exclusive, and the failure mode of each is loud. In database mode, editing the file does nothing, which you find out the first time you look at the UI. In file mode, the UI refuses the edit to your face with a message naming the file to edit instead.
## The round trip
The file stays useful as an editing surface in both modes — text is diffable,
reviewable and easy to back up — because `export` renders the database into the
same shape `import` and file mode read:
The file stays useful as an editing surface in both modes — text is diffable, reviewable and easy to back up — because `export` renders the database into the same shape `import` and file mode read:
```
nxdns export → canonical ZON → edit → nxdns import → config.db
```
`nxdns export` writes canonical ZON: a fixed two-line header, every default
emitted, deterministic ordering from the model's field order and the
repositories' `ORDER BY` clauses, and no timestamps or hostnames anywhere.
Runtime columns are absent from the configuration model on purpose, so two
exports taken from a live, busy server are identical. `export → import →
export` is byte-identical, and a test asserts it.
`nxdns export` writes canonical ZON: a fixed two-line header, every default emitted, deterministic ordering from the model's field order and the repositories' `ORDER BY` clauses, and no timestamps or hostnames anywhere. Runtime columns are absent from the configuration model on purpose, so two exports taken from a live, busy server are identical. `export → import → export` is byte-identical, and a test asserts it.
Byte-stability is not cosmetic. It is what makes an exported file usable in
version control and what makes a diff of two exports mean something: any
difference is a configuration change, never noise from when the export ran.
Byte-stability is not cosmetic. It is what makes an exported file usable in version control and what makes a diff of two exports mean something: any difference is a configuration change, never noise from when the export ran.
The output is also identical in both modes — nothing marks a file as coming
from a file-mode box. That is deliberate, because it is what makes export the
adoption tool: the file you check is the file you deploy, byte for byte. The
label an operator wants is in the unit file, where they put it.
The output is also identical in both modes — nothing marks a file as coming from a file-mode box. That is deliberate, because it is what makes export the adoption tool: the file you check is the file you deploy, byte for byte. The label an operator wants is in the unit file, where they put it.
Reconciling the same file twice produces a byte-identical database — ids,
checksums, `created_at`, the password hash, the whole settings table — including
when the file is written in a non-canonical but equivalent form, such as
`FD00:0:0:0:0:0:0:1` for an address stored as `fd00::1`. Anything that churns
under an unchanged file is a bug in the engine by definition. That single
invariant is what forces most of the design above: matching on canonical forms,
writing only on difference, treating duplicate rule tuples as a multiset, and
verifying a password rather than re-hashing it.
Reconciling the same file twice produces a byte-identical database — ids, checksums, `created_at`, the password hash, the whole settings table — including when the file is written in a non-canonical but equivalent form, such as `FD00:0:0:0:0:0:0:1` for an address stored as `fd00::1`. Anything that churns under an unchanged file is a bug in the engine by definition. That single invariant is what forces most of the design above: matching on canonical forms, writing only on difference, treating duplicate rule tuples as a multiset, and verifying a password rather than re-hashing it.
## Unknown keys, and why the asymmetry is deliberate
An unknown key in the **database** is warned about and ignored. An unknown key
in the **file** is a hard error.
An unknown key in the **database** is warned about and ignored. An unknown key in the **file** is a hard error.
They are different situations. A settings row the running binary does not
recognise is almost always an older binary reading a database written by a newer
one — a downgrade, or a rollback after a bad upgrade. Refusing to start there
would mean a downgrade bricks the config database, and the operator would have
to hand-edit SQLite to recover. Warning and ignoring means the downgrade works,
the unknown setting sits inert, and the upgrade back picks it up again.
They are different situations. A settings row the running binary does not recognise is almost always an older binary reading a database written by a newer one — a downgrade, or a rollback after a bad upgrade. Refusing to start there would mean a downgrade bricks the config database, and the operator would have to hand-edit SQLite to recover. Warning and ignoring means the downgrade works, the unknown setting sits inert, and the upgrade back picks it up again.
A key in a file, by contrast, is something a human just typed. The likeliest
cause is a typo, and the second likeliest is a field that no longer exists.
Silently ignoring it would mean the setting the operator believes they applied
was never applied — exactly the silent-divergence failure the whole model is
built to avoid. So the ZON parser rejects unknown fields with a line and column.
A key in a file, by contrast, is something a human just typed. The likeliest cause is a typo, and the second likeliest is a field that no longer exists. Silently ignoring it would mean the setting the operator believes they applied was never applied — exactly the silent-divergence failure the whole model is built to avoid. So the ZON parser rejects unknown fields with a line and column.
The tolerance has a boundary worth stating plainly: it covers unknown *keys*,
not unparseable *values*. A known key whose stored text does not decode into its
type is an error, not a warning.
The tolerance has a boundary worth stating plainly: it covers unknown *keys*, not unparseable *values*. A known key whose stored text does not decode into its type is an error, not a warning.
## The password
`web.password` is a write-only input. It is never a stored value. A non-empty
one is hashed with argon2id (PHC encoding, OWASP argon2id parameters) into
`web.password_hash`, and the plaintext is cleared before anything is written.
There is no settings row that can hold it: the model skips `web.password` in
both directions of the settings bridge, so the plaintext has nowhere to go even
by accident.
`web.password` is a write-only input. It is never a stored value. A non-empty one is hashed with argon2id (PHC encoding, OWASP argon2id parameters) into `web.password_hash`, and the plaintext is cleared before anything is written. There is no settings row that can hold it: the model skips `web.password` in both directions of the settings bridge, so the plaintext has nowhere to go even by accident.
Both fields are optional, and this is the one deliberate carve-out from
file-as-sole-truth: **a file that mentions neither leaves the stored hash
alone.**
Both fields are optional, and this is the one deliberate carve-out from file-as-sole-truth: **a file that mentions neither leaves the stored hash alone.**
The reason is a trap the design walked into once. An export carries the full PHC
string, which is long and ugly, and an operator committing that file to git will
sooner or later delete the line — meaning "keep the current password". If
absence meant "no password", that edit would reconcile an empty hash over the
stored one and open the admin UI to the entire LAN, silently, because
authentication is on exactly when the hash is non-empty. Silence has to mean
keep. Disabling authentication takes the explicit `password_hash = ""`.
The reason is a trap the design walked into once. An export carries the full PHC string, which is long and ugly, and an operator committing that file to git will sooner or later delete the line — meaning "keep the current password". If absence meant "no password", that edit would reconcile an empty hash over the stored one and open the admin UI to the entire LAN, silently, because authentication is on exactly when the hash is non-empty. Silence has to mean keep. Disabling authentication takes the explicit `password_hash = ""`.
The other end of the same problem is `password = ""`. Hashing the empty string
produces a perfectly valid hash, so authentication would be *on* — while the
login handler refuses every empty password, so it could never be satisfied.
Auth on and unreachable is worse than either alternative, so that file is
refused at validation with a diagnostic naming the remedy.
The other end of the same problem is `password = ""`. Hashing the empty string produces a perfectly valid hash, so authentication would be *on* — while the login handler refuses every empty password, so it could never be satisfied. Auth on and unreachable is worse than either alternative, so that file is refused at validation with a diagnostic naming the remedy.
A plaintext password that has not changed is verified against the stored hash
and kept rather than re-hashed. That is byte-stability, not a saving:
verification recomputes the same argon2id function with the stored salt and
costs exactly what hashing costs. Hashing unconditionally would generate a fresh
salt on every start and break the invariant above.
A plaintext password that has not changed is verified against the stored hash and kept rather than re-hashed. That is byte-stability, not a saving: verification recomputes the same argon2id function with the stored salt and costs exactly what hashing costs. Hashing unconditionally would generate a fresh salt on every start and break the invariant above.
Export's canonical form is therefore `password = null` beside the stored
`password_hash`. Writing an empty *string* there instead would make every export
carry a present-but-empty password next to a hash — tripping the both-set rule
on re-import, so export's own output would fail export's own contract.
Export's canonical form is therefore `password = null` beside the stored `password_hash`. Writing an empty *string* there instead would make every export carry a present-but-empty password next to a hash — tripping the both-set rule on re-import, so export's own output would fail export's own contract.
## What is not configuration
Storage paths are process arguments, not configuration fields: `--data-dir`,
`--config`, `--web-dev`. They cannot live in the file, because the file is found
by way of them — a path that told you where to find the thing that told you the
path would be circular. They are also the settings a supervisor (systemd,
Docker) owns rather than the operator's policy about DNS. See
[reference/files-and-directories.md](../reference/files-and-directories.md).
Storage paths are process arguments, not configuration fields: `--data-dir`, `--config`, `--web-dev`. They cannot live in the file, because the file is found by way of them — a path that told you where to find the thing that told you the path would be circular. They are also the settings a supervisor (systemd, Docker) owns rather than the operator's policy about DNS. See [reference/files-and-directories.md](../reference/files-and-directories.md).
docs/explanation/performance-and-testing.md
+32 -158
View File
@@ -1,12 +1,8 @@
# Performance targets and what the tests prove
Two related questions: why the performance numbers are the numbers, and why CI
does not enforce them — and then, less comfortably, what a green test suite
here does and does not tell you.
Two related questions: why the performance numbers are the numbers, and why CI does not enforce them — and then, less comfortably, what a green test suite here does and does not tell you.
For the targets and the measured results as data, see
[reference/performance.md](../reference/performance.md); to run the bench
yourself, [how-to/measure-performance.md](../how-to/measure-performance.md).
For the targets and the measured results as data, see [reference/performance.md](../reference/performance.md); to run the bench yourself, [how-to/measure-performance.md](../how-to/measure-performance.md).
## Where the targets come from
@@ -16,197 +12,75 @@ PLAN §18 sets five:
- blocklist lookup p95 < 1 ms;
- cached response p95 < 5 ms;
- memory with ~1M blocked domains < 100 MiB;
- stripped static binary ≤ 10,485,760 bytes per arch, ≤ 15,728,640 bytes with
the embedded frontend.
- stripped static binary ≤ 10,485,760 bytes per arch, ≤ 15,728,640 bytes with the embedded frontend.
They are household-scale numbers, and they are deliberately unambitious. 100
qps is far more than a house generates; the point of the target is not speed
but that a Pi 5 with an SD card never becomes the reason the internet feels
broken. The latency targets exist for the same reason: DNS sits in front of
every connection anyone makes, so the failure people notice is not throughput
but a stall. The memory target is what keeps a 1M-entry blocklist from
competing with everything else on a 4 GB board. The binary-size target is
about what a static single-binary deployment is for — if it does not fit on a
constrained box and copy over a slow link in one step, the packaging decision
has not paid for itself.
They are household-scale numbers, and they are deliberately unambitious. 100 qps is far more than a house generates; the point of the target is not speed but that a Pi 5 with an SD card never becomes the reason the internet feels broken. The latency targets exist for the same reason: DNS sits in front of every connection anyone makes, so the failure people notice is not throughput but a stall. The memory target is what keeps a 1M-entry blocklist from competing with everything else on a 4 GB board. The binary-size target is about what a static single-binary deployment is for — if it does not fit on a constrained box and copy over a slow link in one step, the packaging decision has not paid for itself.
`tools/bench.zig` (`zig build bench`) measures the three that are measurable
in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry
snapshot), `cache` (key build plus cache get plus id patch), and `compile`
(the blocklist compiler over a 1M-line body, informational — there is no §18
target for it because no prior datapoint exists). Memory comes from
`/proc/self/status` VmRSS. The qps target is not in the harness at all: it is
end-to-end against the real binary with a DNS load generator, because a
harness number for "queries per second" would measure the harness.
`tools/bench.zig` (`zig build bench`) measures the three that are measurable in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry snapshot), `cache` (key build plus cache get plus id patch), and `compile` (the blocklist compiler over a 1M-line body, informational — there is no §18 target for it because no prior datapoint exists). Memory comes from `/proc/self/status` VmRSS. The qps target is not in the harness at all: it is end-to-end against the real binary with a DNS load generator, because a harness number for "queries per second" would measure the harness.
The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product,
and `src/tests.zig` aggregates everything shippable.
The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product, and `src/tests.zig` aggregates everything shippable.
## Why CI does not gate on performance
Required CI stays deterministic (AGENTS.md). Latency assertions on shared
runners measure the runner's noisy neighbours; the same commit passes and
fails depending on what else the host is doing. A gate that flakes does not
protect anything — it trains people to re-run the job, and once re-running is
routine, a real regression gets re-run too. The flaky gate is worse than no
gate, because it also consumes the attention a real gate would need.
Required CI stays deterministic (AGENTS.md). Latency assertions on shared runners measure the runner's noisy neighbours; the same commit passes and fails depending on what else the host is doing. A gate that flakes does not protect anything — it trains people to re-run the job, and once re-running is routine, a real regression gets re-run too. The flaky gate is worse than no gate, because it also consumes the attention a real gate would need.
So the bench defaults to informational, and `--assert` — which exits non-zero
on a missed target — exists for hardware you control. Run it on the Pi, where
the numbers describe the machine the software actually has to run on. The
x86_64 development-host numbers in
[reference/performance.md](../reference/performance.md) are a regression
baseline for the machine development happens on, not a claim about the target
platform; a Cortex-A76 is far slower and those numbers do not transfer.
So the bench defaults to informational, and `--assert` — which exits non-zero on a missed target — exists for hardware you control. Run it on the Pi, where the numbers describe the machine the software actually has to run on. The x86_64 development-host numbers in [reference/performance.md](../reference/performance.md) are a regression baseline for the machine development happens on, not a claim about the target platform; a Cortex-A76 is far slower and those numbers do not transfer.
CI does gate on the one performance property that *is* deterministic: binary
size. The `package` job builds the release artifacts and `zig build verify-dist`
asserts both §18 budgets against them. Size is a function of the input, not of
the runner's mood, so it is exactly the kind of thing a shared runner can
measure honestly.
CI does gate on the one performance property that *is* deterministic: binary size. The `package` job builds the release artifacts and `zig build verify-dist` asserts both §18 budgets against them. Size is a function of the input, not of the runner's mood, so it is exactly the kind of thing a shared runner can measure honestly.
The budgets are asserted as exact byte counts, and the asset-free budget gets
its own build against a generated empty assets directory rather than against
`web/dist-placeholder`. The placeholder is not buildable by `dist` at all —
that is the guard against a release shipping a stub admin page — and letting it
back in through a size check would have defeated the guard for the sake of one
number.
The budgets are asserted as exact byte counts, and the asset-free budget gets its own build against a generated empty assets directory rather than against `web/dist-placeholder`. The placeholder is not buildable by `dist` at all — that is the guard against a release shipping a stub admin page — and letting it back in through a size check would have defeated the guard for the sake of one number.
## What the test suite is
Every blocking check lives in `.gitea/workflows/gates.yml`, which is a
`workflow_call` workflow with nothing in it but jobs. `ci.yml` calls it on push
and pull request for `master`, and `release.yml` calls it before it builds
anything publishable. That shape exists for one reason: a check that lived in
`ci.yml` alone would be a check a release could skip.
Every blocking check lives in `.gitea/workflows/gates.yml`, which is a `workflow_call` workflow with nothing in it but jobs. `ci.yml` calls it on push and pull request for `master`, and `release.yml` calls it before it builds anything publishable. That shape exists for one reason: a check that lived in `ci.yml` alone would be a check a release could skip.
Five jobs, all required:
- `test` — the Zig suite with `-Dintegration`.
- `test-aarch64` — the same suite cross-built for aarch64 and executed under
qemu-user, plain tier only.
- `test-aarch64` — the same suite cross-built for aarch64 and executed under qemu-user, plain tier only.
- `frontend` — format, lint, typecheck, the vitest cases, build.
- `package` — `zig build dist` and `zig build verify-dist`, which is where the
size budgets, the ELF static-linkage assert and the archive layout checks
are.
- `container` — builds the image, asserts the binary inside it is byte-identical
to the one in the matching tarball, and smoke-tests it by booting the
container and polling `/api/health`.
- `package` — `zig build dist` and `zig build verify-dist`, which is where the size budgets, the ELF static-linkage assert and the archive layout checks are.
- `container` — builds the image, asserts the binary inside it is byte-identical to the one in the matching tarball, and smoke-tests it by booting the container and polling `/api/health`.
The Zig suite has three tiers, gated by build flags:
- **plain `zig build test`** — pure logic. No sockets, no threads, no clock
budgets. This is the tier the purity rule
([architecture.md](architecture.md)) exists to make possible.
- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:`
databases, temp directories. Nothing leaves the host.
- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT
handshakes against real resolvers). Four tests, and they run in a
manual-dispatch workflow, never on push or pull request.
- **plain `zig build test`** — pure logic. No sockets, no threads, no clock budgets. This is the tier the purity rule ([architecture.md](architecture.md)) exists to make possible.
- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:` databases, temp directories. Nothing leaves the host.
- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT handshakes against real resolvers). Four tests, and they run in a manual-dispatch workflow, never on push or pull request.
The aarch64 job runs the plain tier only. The integration tests are
multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown
turns those budgets into a flake source — the same reasoning that keeps the
bench out of CI. What aarch64 needs to prove is portable correctness of the
DNS, filter and cache logic, and the plain tier is exactly that.
The aarch64 job runs the plain tier only. The integration tests are multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown turns those budgets into a flake source — the same reasoning that keeps the bench out of CI. What aarch64 needs to prove is portable correctness of the DNS, filter and cache logic, and the plain tier is exactly that.
At the time of writing, plain `zig build test` is 1175 of 1288 passing with
113 skipped and 0 failed, the skips being the integration-gated tests.
Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with
`-Dintegration` (the 4 skips are the live-network tests) and 1159 passing
under qemu, 0 failed in each.
At the time of writing, plain `zig build test` is 1175 of 1288 passing with 113 skipped and 0 failed, the skips being the integration-gated tests. Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with `-Dintegration` (the 4 skips are the live-network tests) and 1159 passing under qemu, 0 failed in each.
## What it does not prove
The suite is hermetic by design. That is the right default: it is fast, it is
deterministic, it can gate merges. But hermetic and correct are different
properties, and the gap has already cost this project twice.
The suite is hermetic by design. That is the right default: it is fast, it is deterministic, it can gate merges. But hermetic and correct are different properties, and the gap has already cost this project twice.
**The blocklist download aborted the process on first real use.** The fetcher
constructed the HTTP response reader over `transfer_buf` and then read *into*
that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's
already-buffered bytes into the caller's destination — so source and
destination were the same allocation, and Zig's `@memcpy` requires them not to
overlap. It aborts.
**The blocklist download aborted the process on first real use.** The fetcher constructed the HTTP response reader over `transfer_buf` and then read *into* that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's already-buffered bytes into the caller's destination — so source and destination were the same allocation, and Zig's `@memcpy` requires them not to overlap. It aborts.
The reason no test caught it is precise and instructive. The copy length is
zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is
fine. Bytes only accumulate in the reader's own buffer when a read comes back
short of filling the destination and the loop goes round again — that is, when
the body arrives in more than one stream call. The loopback fixture answers
every request with one small in-memory body that lands in a single read, and
the one over-size test never streams a byte, because the fetcher refuses an
oversized `content-length` on the response head. Every test in the suite was
on the zero-length-memcpy side of the branch. The first real download — a
multi-megabyte list over TLS across the WAN, arriving in many TCP segments —
was on the other side, and took the process down. The fix (commit 35f2324)
streams the body straight into the caller's writer, so the reader's buffer is
never a destination slice, and it came with four regression tests that put a
fully-buffered reader into exactly the state the old code could not survive.
The reason no test caught it is precise and instructive. The copy length is zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is fine. Bytes only accumulate in the reader's own buffer when a read comes back short of filling the destination and the loop goes round again — that is, when the body arrives in more than one stream call. The loopback fixture answers every request with one small in-memory body that lands in a single read, and the one over-size test never streams a byte, because the fetcher refuses an oversized `content-length` on the response head. Every test in the suite was on the zero-length-memcpy side of the branch. The first real download — a multi-megabyte list over TLS across the WAN, arriving in many TCP segments — was on the other side, and took the process down. The fix (commit 35f2324) streams the body straight into the caller's writer, so the reader's buffer is never a destination slice, and it came with four regression tests that put a fully-buffered reader into exactly the state the old code could not survive.
**A stale embedded SPA bundle shipped a settings page that crashed on load,
while 121 web tests passed.** `web/dist/` is gitignored and
`-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The
frontend tests ran against the sources, in jsdom, and were green; the binary
carried an older build. The tests were testing something the artifact did not
contain.
**A stale embedded SPA bundle shipped a settings page that crashed on load, while 121 web tests passed.** `web/dist/` is gitignored and `-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The frontend tests ran against the sources, in jsdom, and were green; the binary carried an older build. The tests were testing something the artifact did not contain.
Note what these two have in common. Neither was a logic bug that a better unit
test would have caught. One lived in the seam between the pure core and its
one I/O edge; the other lived in the seam between two build systems. Hermetic
tests are constructed to exclude exactly those seams — that is what makes them
hermetic.
Note what these two have in common. Neither was a logic bug that a better unit test would have caught. One lived in the seam between the pure core and its one I/O edge; the other lived in the seam between two build systems. Hermetic tests are constructed to exclude exactly those seams — that is what makes them hermetic.
## The lesson, and where it now lives
Green hermetic tests are a floor, not a ceiling. They prove the logic is
consistent with itself. They cannot prove the program works, because the
things they deliberately exclude — real network reads, real TLS, real file
sizes, real build artifacts — are where a program meets reality.
Green hermetic tests are a floor, not a ceiling. They prove the logic is consistent with itself. They cannot prove the program works, because the things they deliberately exclude — real network reads, real TLS, real file sizes, real build artifacts — are where a program meets reality.
The response is not to make CI non-deterministic. It is to require that the
real paths get exercised by a human before work is called done. That is now
ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and
the how-to pages is executed verbatim, on the host, by the session that writes
it, and a command that cannot run there is marked in the page as unverified
with the reason. Documentation written from source-reading alone is how both
of these shipped; documentation that has been run is a second, independent
test suite that exercises precisely the paths the hermetic one skips.
The response is not to make CI non-deterministic. It is to require that the real paths get exercised by a human before work is called done. That is now ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and the how-to pages is executed verbatim, on the host, by the session that writes it, and a command that cannot run there is marked in the page as unverified with the reason. Documentation written from source-reading alone is how both of these shipped; documentation that has been run is a second, independent test suite that exercises precisely the paths the hermetic one skips.
Two honest gaps remain, stated so nobody has to rediscover them:
- Nothing in the suite drives a multi-read HTTP body through the fetcher end
to end. The regression tests cover `pumpBody` directly over a pre-buffered
reader; the loopback fixture still sends one small body per connection.
- There is no freshness check on `web/dist`. CI cannot embed a stale bundle,
because the jobs that pass `-Dweb-dist` rebuild the frontend immediately
beforehand. A local build can, and will do it without a warning.
- Nothing in the suite drives a multi-read HTTP body through the fetcher end to end. The regression tests cover `pumpBody` directly over a pre-buffered reader; the loopback fixture still sends one small body per connection.
- There is no freshness check on `web/dist`. CI cannot embed a stale bundle, because the jobs that pass `-Dweb-dist` rebuild the frontend immediately beforehand. A local build can, and will do it without a warning.
## What a signed release does not prove either
The same distinction applies one level out, to the artifacts. A release is
signed, and the signature is worth having: it says the artifact came from this
project's pipeline and reached you unaltered. It does not say the binary was
built from the source in this repository, because the machine that ran the
build also held the signing key. An attacker with that machine produces
something that verifies cleanly and contains whatever they put in it.
The same distinction applies one level out, to the artifacts. A release is signed, and the signature is worth having: it says the artifact came from this project's pipeline and reached you unaltered. It does not say the binary was built from the source in this repository, because the machine that ran the build also held the signing key. An attacker with that machine produces something that verifies cleanly and contains whatever they put in it.
The control that closes that gap is a reproducibility gate — an independent
build, in a different directory on a different machine, landing on the same
bytes. It does not exist. It is a recorded deferral (`specs/milestone-14.md`
ruling 12), not something nobody thought of, and until it exists no document
here describes the build as reproducible: nobody has measured whether it is.
The cheap inputs to reproducibility are already in place — `gzip -n`,
`--mtime=@0`, `LC_ALL=C`, `TZ=UTC`, exact Zig and Node pins — which makes the
gate cheap to add later and proves nothing on its own.
The control that closes that gap is a reproducibility gate — an independent build, in a different directory on a different machine, landing on the same bytes. It does not exist. It is a recorded deferral (`specs/milestone-14.md` ruling 12), not something nobody thought of, and until it exists no document here describes the build as reproducible: nobody has measured whether it is. The cheap inputs to reproducibility are already in place — `gzip -n`, `--mtime=@0`, `LC_ALL=C`, `TZ=UTC`, exact Zig and Node pins — which makes the gate cheap to add later and proves nothing on its own.
What the release pipeline is required to hold to is narrower: two runs of
`zig build dist` on the same commit **in the same directory** produce
byte-identical tarballs. Same-directory determinism is a much weaker property
than reproducibility, and conflating the two is exactly the kind of claim this
page exists to refuse.
What the release pipeline is required to hold to is narrower: two runs of `zig build dist` on the same commit **in the same directory** produce byte-identical tarballs. Same-directory determinism is a much weaker property than reproducibility, and conflating the two is exactly the kind of claim this page exists to refuse.
[Verify a release](../how-to/verify-a-release.md) states the same limits where
an operator will actually meet them, and gives the rebuild-and-compare recipe
with the caveat that a differing hash is not evidence of tampering while this
gap is open.
[Verify a release](../how-to/verify-a-release.md) states the same limits where an operator will actually meet them, and gives the rebuild-and-compare recipe with the caveat that a differing hash is not evidence of tampering while this gap is open.
docs/how-to/back-up-and-restore.md
+26 -90
View File
@@ -1,27 +1,15 @@
# Back up and restore
The configuration is the only state worth keeping. `nxdns export` writes it out
as a ZON file and `nxdns import` writes one back. The query log is deliberately
not part of a backup: it is expendable history, and if it is missing it gets
recreated empty.
The configuration is the only state worth keeping. `nxdns export` writes it out as a ZON file and `nxdns import` writes one back. The query log is deliberately not part of a backup: it is expendable history, and if it is missing it gets recreated empty.
**Which authority the service runs under decides what the backup *is*.** Check
the start log:
**Which authority the service runs under decides what the backup *is*.** Check the start log:
- `authority: database` — `config.db` holds the configuration. Back it up with
`nxdns export`, and restore with `nxdns import` or by replacing the database
file.
- `authority: file (<path>)` — that file holds the configuration, and it is
already a text file you can keep in git. **The file is the backup.** Restoring
means putting the file back and restarting; the database rebuilds itself from
it. `config.db` is a cache of the file in this mode, not the thing to preserve.
- `authority: database` — `config.db` holds the configuration. Back it up with `nxdns export`, and restore with `nxdns import` or by replacing the database file.
- `authority: file (<path>)` — that file holds the configuration, and it is already a text file you can keep in git. **The file is the backup.** Restoring means putting the file back and restarting; the database rebuilds itself from it. `config.db` is a cache of the file in this mode, not the thing to preserve.
The rest of this page covers database mode unless it says otherwise.
The commands below use the scratch lab from
[enable DoH and DoT](enable-doh-and-dot.md), data directory
`/tmp/nxdns-lab/data`. On a real install drop `--data-dir` and the default
`/var/lib/nxdns` applies.
The commands below use the scratch lab from [enable DoH and DoT](enable-doh-and-dot.md), data directory `/tmp/nxdns-lab/data`. On a real install drop `--data-dir` and the default `/var/lib/nxdns` applies.
## Back up
@@ -35,8 +23,7 @@ nxdns export --data-dir /tmp/nxdns-lab/data --out /tmp/nxdns-lab/backup.zon
wrote /tmp/nxdns-lab/backup.zon
```
The write is a temp file plus a rename, so an interrupted export leaves no
half-written backup, and the result is mode 0600:
The write is a temp file plus a rename, so an interrupted export leaves no half-written backup, and the result is mode 0600:
```sh
stat -c '%a %n' /tmp/nxdns-lab/backup.zon
@@ -57,15 +44,9 @@ grep password /tmp/nxdns-lab/backup.zon
.password_hash = "$argon2id$v=19$m=19456,t=2,p=1$hOjjnTrTZ6kU8XrwQuJ7ZFC3B5LumaB4hRe7kBbzJ6Q$YsC2rCTDKv96eEwPhs+D6vbDliLogEZph8EkagKSuy8",
```
Treat backups as secrets. `.password` is always exported as `null` — the
plaintext is never stored anywhere — so the file re-imports without anyone
knowing the password. `null` and `""` are different statements here: `null`
means the file says nothing about the password, while an empty `password_hash`
would disable authentication. See
[Password and hash](../reference/configuration.md#password-and-hash).
Treat backups as secrets. `.password` is always exported as `null` — the plaintext is never stored anywhere — so the file re-imports without anyone knowing the password. `null` and `""` are different statements here: `null` means the file says nothing about the password, while an empty `password_hash` would disable authentication. See [Password and hash](../reference/configuration.md#password-and-hash).
Without `--out` the export goes to stdout, where the file mode is your
redirect's problem:
Without `--out` the export goes to stdout, where the file mode is your redirect's problem:
```sh
nxdns export --data-dir /tmp/nxdns-lab/data | head -10
@@ -84,15 +65,11 @@ nxdns export --data-dir /tmp/nxdns-lab/data | head -10
.bind_ipv4 = "127.0.0.1",
```
Runtime facts are left out on purpose: client first-seen and last-seen times,
rule creation times, per-source domain counts and checksums. They are things a
running server produces, not configuration, and including them would make two
exports taken minutes apart differ.
Runtime facts are left out on purpose: client first-seen and last-seen times, rule creation times, per-source domain counts and checksums. They are things a running server produces, not configuration, and including them would make two exports taken minutes apart differ.
## Restore onto a fresh data directory
This is the normal restore: new machine, new disk, empty data directory.
Nothing to overwrite, so nothing to authorise.
This is the normal restore: new machine, new disk, empty data directory. Nothing to overwrite, so nothing to authorise.
```sh
nxdns import /tmp/nxdns-lab/backup.zon --data-dir /tmp/nxdns-lab/data-restored
@@ -103,15 +80,11 @@ info(migrations): config.db migrated from schema version 0 to 1
imported /tmp/nxdns-lab/backup.zon
```
The migration line is expected: `import` creates and migrates the database
before writing to it.
The migration line is expected: `import` creates and migrates the database before writing to it.
## Restore over an existing database
**Stop the server first.** `import` rewrites configuration underneath a process
that read it at startup, and a running server picks up only part of it —
filtering follows the new rows at the next reload, while upstreams, listeners
and settings stay at their boot values until a restart.
**Stop the server first.** `import` rewrites configuration underneath a process that read it at startup, and a running server picks up only part of it — filtering follows the new rows at the next reload, while upstreams, listeners and settings stay at their boot values until a restart.
```sh
systemctl stop nxdns
@@ -119,10 +92,7 @@ nxdns import /var/backups/nxdns-config.zon
systemctl start nxdns
```
`import` converges the database onto the file. Rows the file still names are
matched and updated in place; rows it no longer names are deleted. That last
part is what a restore of an old backup does to everything added since, so it
takes a flag:
`import` converges the database onto the file. Rows the file still names are matched and updated in place; rows it no longer names are deleted. That last part is what a restore of an old backup does to everything added since, so it takes a flag:
```sh
nxdns import /tmp/nxdns-lab/old-backup.zon --data-dir /tmp/nxdns-lab/data
@@ -133,10 +103,7 @@ FAIL import: this file would delete rows the database holds (upstreams 1); re-ru
import failed: DestructiveImport
```
That exits 2 and rolls the transaction back, so nothing is half-applied. The
message names every table that would lose rows, which is usually enough to tell
an intended restore from the wrong file. Say `--allow-delete` when deleting is
what you mean:
That exits 2 and rolls the transaction back, so nothing is half-applied. The message names every table that would lose rows, which is usually enough to tell an intended restore from the wrong file. Say `--allow-delete` when deleting is what you mean:
```sh
nxdns import /tmp/nxdns-lab/old-backup.zon --allow-delete --data-dir /tmp/nxdns-lab/data
@@ -146,20 +113,9 @@ nxdns import /tmp/nxdns-lab/old-backup.zon --allow-delete --data-dir /tmp/nxdns-
imported /tmp/nxdns-lab/old-backup.zon
```
A restore that only puts back what is already there needs no flag at all, and
neither does one that only adds rows. The flag is about deletion specifically —
including deletion in disguise: renaming a group, or correcting a typo in an
upstream URL, changes the row's identity, so the engine sees one row gone and
one arrived.
A restore that only puts back what is already there needs no flag at all, and neither does one that only adds rows. The flag is about deletion specifically — including deletion in disguise: renaming a group, or correcting a typo in an upstream URL, changes the row's identity, so the engine sees one row gone and one arrived.
What survives a restore is worth knowing before you take an old backup out of
the drawer. Blocklist download state is kept for every source whose URL the file
still names — checksum, counters, compiled files, and the row id they are keyed
by — so restoring does not cost a re-download. Devices the server discovered
from traffic are kept whole, and one the backup names keeps the first-seen and
last-seen the database already held, so a restore never restamps your network as
newly arrived. What does go is a client the backup does not name and that was
named by hand: that row is declarative, and it is deleted with the rest.
What survives a restore is worth knowing before you take an old backup out of the drawer. Blocklist download state is kept for every source whose URL the file still names — checksum, counters, compiled files, and the row id they are keyed by — so restoring does not cost a re-download. Devices the server discovered from traffic are kept whole, and one the backup names keeps the first-seen and last-seen the database already held, so a restore never restamps your network as newly arrived. What does go is a client the backup does not name and that was named by hand: that row is declarative, and it is deleted with the rest.
> The three `systemctl` lines are the only commands on this page that were not
> run: this machine has no installed nxdns unit (`systemctl status nxdns`
@@ -169,17 +125,10 @@ named by hand: that row is declarative, and it is deleted with the rest.
## Restoring the database file itself
Copying `config.db` back into place works too, and it is the fastest restore on
a machine that still has one. Two rules, and the second one is where restores go
wrong:
Copying `config.db` back into place works too, and it is the fastest restore on a machine that still has one. Two rules, and the second one is where restores go wrong:
1. **Take the copy from a stopped instance**, or use `nxdns export` instead. A
copy taken while nxdns is running catches the main file without the changes
sitting in its write-ahead log.
2. **Delete any stale `config.db-wal` and `config.db-shm` beside the file you
restore.** SQLite silently discards a write-ahead log that does not match the
database it sits next to. It does not warn, and it does not fail — it just
answers from the main file, so a restore quietly loses its own tail.
1. **Take the copy from a stopped instance**, or use `nxdns export` instead. A copy taken while nxdns is running catches the main file without the changes sitting in its write-ahead log.
2. **Delete any stale `config.db-wal` and `config.db-shm` beside the file you restore.** SQLite silently discards a write-ahead log that does not match the database it sits next to. It does not warn, and it does not fail — it just answers from the main file, so a restore quietly loses its own tail.
```sh
systemctl stop nxdns
@@ -193,15 +142,11 @@ systemctl start nxdns
> Not verified on this host: these need root, an installed unit and
> `/var/lib/nxdns`, none of which exist here.
None of this applies in file mode. There `config.db` is derived state — restore
the configuration file and start the service, and the first reconcile rebuilds
the database from it.
None of this applies in file mode. There `config.db` is derived state — restore the configuration file and start the service, and the first reconcile rebuilds the database from it.
## Verify a backup
The round trip is byte-stable: exporting, importing and exporting again gives
an identical file. That is the cheapest check that a backup is complete and
that it will load.
The round trip is byte-stable: exporting, importing and exporting again gives an identical file. That is the cheapest check that a backup is complete and that it will load.
```sh
nxdns export --data-dir /tmp/nxdns-lab/data --out /tmp/nxdns-lab/backup2.zon
@@ -214,8 +159,7 @@ wrote /tmp/nxdns-lab/backup2.zon
round trip is byte-identical
```
The same check works across data directories — export from the restored copy
and diff against the backup you restored from:
The same check works across data directories — export from the restored copy and diff against the backup you restored from:
```sh
nxdns export --data-dir /tmp/nxdns-lab/data-restored --out /tmp/nxdns-lab/backup3.zon
@@ -233,20 +177,12 @@ restored database exports identically
Everything under the data directory other than `config.db`:
- `querylog.db` — expendable history, recreated empty when absent.
- `blocklists/` — compiled snapshots. They are rebuilt from the sources named in
the configuration, so restoring the configuration is enough; the first refresh
after a restore downloads them again.
- `blocklists/` — compiled snapshots. They are rebuilt from the sources named in the configuration, so restoring the configuration is enough; the first refresh after a restore downloads them again.
The data directory layout is in
[files and directories](../reference/files-and-directories.md).
The data directory layout is in [files and directories](../reference/files-and-directories.md).
## A backup before every upgrade
There is no downgrade path. Schema migrations run forward automatically at
startup and before `export` and `import`; nothing walks them back, and `nxdns
check` does not run them at all. Take an export before installing a new binary —
see [upgrade](upgrade.md).
There is no downgrade path. Schema migrations run forward automatically at startup and before `export` and `import`; nothing walks them back, and `nxdns check` does not run them at all. Take an export before installing a new binary — see [upgrade](upgrade.md).
Every `nxdns` command on this page was executed on this host as written. The
`systemctl`, `cp`, `chown` and `chmod` lines were not: they need root and an
installed unit, and both blocks holding them say so.
Every `nxdns` command on this page was executed on this host as written. The `systemctl`, `cp`, `chown` and `chmod` lines were not: they need root and an installed unit, and both blocks holding them say so.
docs/how-to/enable-doh-and-dot.md
+23 -73
View File
@@ -1,22 +1,14 @@
# Enable DoH and DoT
nxdns can answer encrypted queries on two extra listeners: DNS over HTTPS
(`doh_server`) and DNS over TLS (`dot_server`). Both are off by default and both
need a certificate and a private key in PEM form.
nxdns can answer encrypted queries on two extra listeners: DNS over HTTPS (`doh_server`) and DNS over TLS (`dot_server`). Both are off by default and both need a certificate and a private key in PEM form.
This page uses a scratch lab under `/tmp/nxdns-lab` so the commands run without
root and without touching a real install. On a real install the files live under
`/etc/nxdns` and the data directory is `/var/lib/nxdns`; the ports are 443 and
853 rather than the unprivileged ones below.
This page uses a scratch lab under `/tmp/nxdns-lab` so the commands run without root and without touching a real install. On a real install the files live under `/etc/nxdns` and the data directory is `/var/lib/nxdns`; the ports are 443 and 853 rather than the unprivileged ones below.
Every field mentioned here is documented in
[configuration reference](../reference/configuration.md).
Every field mentioned here is documented in [configuration reference](../reference/configuration.md).
## 1. Get a certificate and key
For a LAN service the practical options are a certificate from your ACME client
(certbot, lego, caddy) for a name you control, or a self-signed pair. The lab
below uses a self-signed pair, because it needs no domain:
For a LAN service the practical options are a certificate from your ACME client (certbot, lego, caddy) for a name you control, or a self-signed pair. The lab below uses a self-signed pair, because it needs no domain:
```sh
mkdir -p /tmp/nxdns-lab/etc
@@ -28,16 +20,11 @@ openssl req -x509 -newkey rsa:2048 -nodes \
chmod 0600 etc/key.pem
```
A self-signed certificate means every client has to be told to trust it, or told
to skip verification. That is why the client commands further down pass
`--insecure` and `+tls` without a CA. A real deployment uses a real certificate
and drops those flags.
A self-signed certificate means every client has to be told to trust it, or told to skip verification. That is why the client commands further down pass `--insecure` and `+tls` without a CA. A real deployment uses a real certificate and drops those flags.
## 2. Turn the listeners on
`cert_path` and `key_path` must be absolute, or relative to the process working
directory. Point both endpoints at the same pair unless you have a reason not
to:
`cert_path` and `key_path` must be absolute, or relative to the process working directory. Point both endpoints at the same pair unless you have a reason not to:
```zon
.{
@@ -62,12 +49,7 @@ to:
}
```
Write that to `/tmp/nxdns-lab/etc/config.zon`. The lab runs
`nxdns run --config`, which makes that file the configuration: every start
reconciles the database onto it, so editing the file and restarting is how these
settings change here. A real install may instead run bare `nxdns run` and keep
the configuration in the database — see
[the configuration model](../explanation/configuration-model.md).
Write that to `/tmp/nxdns-lab/etc/config.zon`. The lab runs `nxdns run --config`, which makes that file the configuration: every start reconciles the database onto it, so editing the file and restarting is how these settings change here. A real install may instead run bare `nxdns run` and keep the configuration in the database — see [the configuration model](../explanation/configuration-model.md).
## 3. Check the files before starting
@@ -81,17 +63,14 @@ OK upstreams[0] https://cloudflare-dns.com
OK: no problems found
```
`check` loads both endpoints' certificate and key the same way the listeners do,
so what passes here will start. An unreadable file is a failure and exits 2:
`check` loads both endpoints' certificate and key the same way the listeners do, so what passes here will start. An unreadable file is a failure and exits 2:
```
FAIL doh_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable
FAIL dot_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable
```
So is a key that does not belong to the certificate, which is the mistake worth
catching before a restart — the two files are individually valid and only their
pairing is wrong. mbedTLS writes its own line to stderr as it rejects the pair:
So is a key that does not belong to the certificate, which is the mistake worth catching before a restart — the two files are individually valid and only their pairing is wrong. mbedTLS writes its own line to stderr as it rejects the pair:
```
warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
@@ -102,9 +81,7 @@ FAIL dot_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not b
OK upstreams[0] https://cloudflare-dns.com
```
A key readable by anyone but its owner is a warning instead. It does not change
the exit code, because the service still starts, and the summary line counts it
rather than claiming nothing was found:
A key readable by anyone but its owner is a warning instead. It does not change the exit code, because the service still starts, and the summary line counts it rather than claiming nothing was found:
```
WARN doh_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only
@@ -126,9 +103,7 @@ info(nxdns): doh listener on 127.0.0.1:8443
info(nxdns): dot listener on 127.0.0.1:8853
```
If a certificate cannot be loaded while its endpoint is enabled, nxdns refuses
to start and exits 2 rather than serving DNS without the listener you asked
for:
If a certificate cannot be loaded while its endpoint is enabled, nxdns refuses to start and exits 2 rather than serving DNS without the listener you asked for:
```
doh_server: '/tmp/nxdns-lab/etc/cert.pem' + '/tmp/nxdns-lab/etc/key.pem': private key file is not readable
@@ -138,8 +113,7 @@ run `nxdns check` to see the configuration in full
## 5. Query DoT
Recent `dig` speaks DNS over TLS with `+tls` (this page was checked with BIND
9.20.26):
Recent `dig` speaks DNS over TLS with `+tls` (this page was checked with BIND 9.20.26):
```sh
dig @127.0.0.1 -p 8853 +tls example.com A +short
@@ -152,13 +126,9 @@ dig @127.0.0.1 -p 8853 +tls example.com A +short
## 6. Query DoH
The only path the DoH listener serves is `/dns-query`; anything else is a 404.
It accepts both the POST form (the query as an `application/dns-message` body)
and the GET form (`?dns=` with base64url of the same bytes).
The only path the DoH listener serves is `/dns-query`; anything else is a 404. It accepts both the POST form (the query as an `application/dns-message` body) and the GET form (`?dns=` with base64url of the same bytes).
The body is a raw DNS query in wire format. Build one for `example.com A` —
header with the recursion-desired bit, one question, then the QNAME as
length-prefixed labels:
The body is a raw DNS query in wire format. Build one for `example.com A` — header with the recursion-desired bit, one question, then the QNAME as length-prefixed labels:
```sh
printf '%s' '000001000001000000000000076578616d706c6503636f6d0000010001' \
@@ -186,8 +156,7 @@ http 200, 72 bytes
00000040: 04d0 0000 0000 0000 ........
```
The second flag byte `80` and the third answer-count field `0002` say: response,
no error, two answer records.
The second flag byte `80` and the third answer-count field `0002` say: response, no error, two answer records.
The GET form takes the same bytes, base64url-encoded with the padding removed:
@@ -202,16 +171,11 @@ curl -sS --insecure --http1.1 -o /tmp/nxdns-lab/get.bin \
GET http 200, 61 bytes
```
`--http1.1` matters: without it curl offers HTTP/2 over ALPN, and the DoH
listener negotiates only what it advertises. `--insecure` is only needed for the
self-signed lab certificate.
`--http1.1` matters: without it curl offers HTTP/2 over ALPN, and the DoH listener negotiates only what it advertises. `--insecure` is only needed for the self-signed lab certificate.
## 7. Renewals
A watcher polls both files every 30 seconds and compares their modification time
and size against the pair currently loaded. When either differs it reloads and
swaps the new pair in; connections already open finish on the old certificate.
Nothing has to restart.
A watcher polls both files every 30 seconds and compares their modification time and size against the pair currently loaded. When either differs it reloads and swaps the new pair in; connections already open finish on the old certificate. Nothing has to restart.
Replace the pair and wait one poll interval:
@@ -232,10 +196,7 @@ info(cert_store): certificate reloaded from /tmp/nxdns-lab/etc/cert.pem
## 8. Reload immediately
To skip the wait — from an ACME deploy hook, for example — call
`POST /api/certs/reload`. It needs a session; see
[set up admin authentication](set-up-admin-authentication.md) for the login
call that fills `cookies.txt`.
To skip the wait — from an ACME deploy hook, for example — call `POST /api/certs/reload`. It needs a session; see [set up admin authentication](set-up-admin-authentication.md) for the login call that fills `cookies.txt`.
```sh
curl -sS -b /tmp/nxdns-lab/cookies.txt -X POST http://127.0.0.1:8451/api/certs/reload
@@ -245,9 +206,7 @@ curl -sS -b /tmp/nxdns-lab/cookies.txt -X POST http://127.0.0.1:8451/api/certs/r
{"doh":{"enabled":true,"reloaded":true,"error":null},"dot":{"enabled":true,"reloaded":true,"error":null}}
```
The route always answers 200: the per-endpoint outcome is the payload, not the
status code. A disabled endpoint reports `"enabled":false`. A reload that fails
names the reason and leaves the old certificate serving:
The route always answers 200: the per-endpoint outcome is the payload, not the status code. A disabled endpoint reports `"enabled":false`. A reload that fails names the reason and leaves the old certificate serving:
```sh
chmod 000 /tmp/nxdns-lab/etc/key.pem
@@ -269,8 +228,7 @@ dig @127.0.0.1 -p 8853 +tls example.com A +short
172.66.147.243
```
Undo it with `chmod 0600 /tmp/nxdns-lab/etc/key.pem` and reload again. The
counters are on `/metrics`:
Undo it with `chmod 0600 /tmp/nxdns-lab/etc/key.pem` and reload again. The counters are on `/metrics`:
```
nxdns_cert_reloads_total{endpoint="doh"} 3
@@ -291,10 +249,7 @@ chmod 0644 /etc/nxdns/cert.pem
chmod 0600 /etc/nxdns/key.pem
```
Under Docker the container runs as uid 65532, fixed in the image, and
`/etc/nxdns` is a read-only bind mount — the container cannot fix permissions
itself, so the host-side files must already be readable by that uid. It has no
name on the host, so chown it numerically:
Under Docker the container runs as uid 65532, fixed in the image, and `/etc/nxdns` is a read-only bind mount — the container cannot fix permissions itself, so the host-side files must already be readable by that uid. It has no name on the host, so chown it numerically:
```sh
cd deploy/docker
@@ -303,13 +258,8 @@ chmod 0644 etc-nxdns/cert.pem
chmod 0600 etc-nxdns/key.pem
```
**Not verified on this host:** the two `chown` blocks above. Both need root, and
the systemd one needs an `nxdns` user this development machine does not have.
Everything else on this page was executed as written.
**Not verified on this host:** the two `chown` blocks above. Both need root, and the systemd one needs an `nxdns` user this development machine does not have. Everything else on this page was executed as written.
## Ports 443 and 853
The defaults are the standard ports, which are privileged. Under the packaged
systemd unit that is already handled: it grants `CAP_NET_BIND_SERVICE` for port
53 and the same capability covers 443 and 853. See
[install with systemd](install-with-systemd.md).
The defaults are the standard ports, which are privileged. Under the packaged systemd unit that is already handled: it grants `CAP_NET_BIND_SERVICE` for port 53 and the same capability covers 443 and 853. See [install with systemd](install-with-systemd.md).
docs/how-to/install-with-docker.md
+29 -101
View File
@@ -1,14 +1,10 @@
# Install nxdns with Docker
Runs the published nxdns image with Docker Compose. At the end a container
answers DNS on port 53 and keeps its data in a named volume.
Runs the published nxdns image with Docker Compose. At the end a container answers DNS on port 53 and keeps its data in a named volume.
The image is multi-architecture — `linux/amd64` and `linux/arm64` — so the same
tag works on a PC and on a Raspberry Pi 5. Building the image yourself is still
supported and is the last section of this page.
The image is multi-architecture — `linux/amd64` and `linux/arm64` — so the same tag works on a PC and on a Raspberry Pi 5. Building the image yourself is still supported and is the last section of this page.
For what each configuration field means, see
[the configuration reference](../reference/configuration.md).
For what each configuration field means, see [the configuration reference](../reference/configuration.md).
> Verification: the failure modes, the run and the two checks in step 3 were run
> on the machine that wrote an earlier revision of this page, against an image
@@ -43,16 +39,9 @@ VERSION=$(curl -fsS -o /dev/null -w '%{redirect_url}' "$BASE/releases/latest" |
docker pull git.mial.net/mokhtar/nxdns:$VERSION
```
Pin a version. `:latest` exists and moves, which is what you want when you are
trying it out and not what you want on a machine your household's DNS depends
on. The lookup above asks the server for the current release rather than
hardcoding a number that goes stale one release later — Gitea redirects
`releases/latest` to the newest published release's tag page. To take a
particular version instead, set `VERSION=<version>` yourself.
Pin a version. `:latest` exists and moves, which is what you want when you are trying it out and not what you want on a machine your household's DNS depends on. The lookup above asks the server for the current release rather than hardcoding a number that goes stale one release later — Gitea redirects `releases/latest` to the newest published release's tag page. To take a particular version instead, set `VERSION=<version>` yourself.
Verify what you pulled before you run it. The release publishes an
`IMAGE-DIGEST.txt` asset naming the digest of the image index, and that file is
covered by the signed `SHA256SUMS.txt`:
Verify what you pulled before you run it. The release publishes an `IMAGE-DIGEST.txt` asset naming the digest of the image index, and that file is covered by the signed `SHA256SUMS.txt`:
```sh
curl -fLO "$BASE/releases/download/v$VERSION/IMAGE-DIGEST.txt"
@@ -65,11 +54,7 @@ docker buildx imagetools inspect git.mial.net/mokhtar/nxdns:$VERSION \
cut -d@ -f2 IMAGE-DIGEST.txt
```
The last two have to print the same string — `IMAGE-DIGEST.txt` holds a whole
pinned reference, `name:tag@sha256:…`, so the `cut` is what reduces it to the
digest `imagetools` prints. [Verify a release](verify-a-release.md)
covers the key, the fingerprint, every failure message, and what the signature
does and does not prove.
The last two have to print the same string — `IMAGE-DIGEST.txt` holds a whole pinned reference, `name:tag@sha256:…`, so the `cut` is what reduces it to the digest `imagetools` prints. [Verify a release](verify-a-release.md) covers the key, the fingerprint, every failure message, and what the signature does and does not prove.
> Not verified on this host: no image and no release are published yet, so
> `docker pull` and every URL here fail today, and the `releases/latest` lookup
@@ -80,10 +65,7 @@ does and does not prove.
## 2. Get the compose file and write the configuration
Every path on this page is relative to a checkout of the repository, because
that is how it was verified. Running the published image needs no checkout,
though — one file is enough. Fetch it for the version you pulled and work in
its directory instead, dropping `deploy/docker/` from the paths below:
Every path on this page is relative to a checkout of the repository, because that is how it was verified. Running the published image needs no checkout, though — one file is enough. Fetch it for the version you pulled and work in its directory instead, dropping `deploy/docker/` from the paths below:
```sh
mkdir -p ~/nxdns && cd ~/nxdns
@@ -94,16 +76,14 @@ curl -fLO "$BASE/raw/tag/v$VERSION/deploy/docker/compose.yaml"
> `<repo>/raw/tag/<tag>/<path>` was run here against `gitea.com/gitea/tea` on
> Gitea `1.27.0+dev` and returned the file with a 200.
Compose bind-mounts `deploy/docker/etc-nxdns` read-only at `/etc/nxdns`. Create
it and put the configuration in it:
Compose bind-mounts `deploy/docker/etc-nxdns` read-only at `/etc/nxdns`. Create it and put the configuration in it:
```sh
mkdir -p deploy/docker/etc-nxdns
$EDITOR deploy/docker/etc-nxdns/config.zon
```
The smallest file that starts is one group named `default` and one enabled
upstream:
The smallest file that starts is one group named `default` and one enabled upstream:
```zon
.{
@@ -113,15 +93,9 @@ upstream:
}
```
**The compose file ships file mode**, with
`command: ["run", "--config=/etc/nxdns/config.zon"]`. That file is the
configuration: the container reconciles its database onto it at every start, and
the admin interface answers 403 to configuration edits. To change anything, edit
the file and restart the container. It also means a fresh or recreated
`nxdns-data` volume rebuilds itself from the mounted file with no extra step.
**The compose file ships file mode**, with `command: ["run", "--config=/etc/nxdns/config.zon"]`. That file is the configuration: the container reconciles its database onto it at every start, and the admin interface answers 403 to configuration edits. To change anything, edit the file and restart the container. It also means a fresh or recreated `nxdns-data` volume rebuilds itself from the mounted file with no extra step.
The file is therefore required, and its absence is a hard failure rather than a
start with defaults:
The file is therefore required, and its absence is a hard failure rather than a start with defaults:
```
FAIL /etc/nxdns/config.zon: no such file
@@ -129,9 +103,7 @@ nxdns run failed: ManagedConfigUnreadable
run `nxdns check` to see the configuration in full
```
A file that is present but rejected is a different failure with the same exit
code. No `default` group, no enabled upstream, a syntax error — `run` prints the
diagnostic and exits 2 as well. A file whose only group was named `other`:
A file that is present but rejected is a different failure with the same exit code. No `default` group, no enabled upstream, a syntax error — `run` prints the diagnostic and exits 2 as well. A file whose only group was named `other`:
```
FAIL groups: no group named 'default'; every unknown client is assigned to it
@@ -139,24 +111,17 @@ nxdns run failed: MissingDefaultGroup
run `nxdns check` to see the configuration in full
```
Under `restart: unless-stopped` any of these is a restart loop — Docker has no
start limit and will retry forever. Read the lines above the failure, which name
the fault. See [Troubleshoot nxdns](troubleshoot.md).
Under `restart: unless-stopped` any of these is a restart loop — Docker has no start limit and will retry forever. Read the lines above the failure, which name the fault. See [Troubleshoot nxdns](troubleshoot.md).
### Database mode in Docker instead
Drop the `command:` line from `compose.yaml` and the container runs
`nxdns run`, with the database as the configuration and the file read by nothing.
On a fresh volume that database is empty and the container exits 2 with
`NoUsableUpstreams`, so load it once before bringing the service up:
Drop the `command:` line from `compose.yaml` and the container runs `nxdns run`, with the database as the configuration and the file read by nothing. On a fresh volume that database is empty and the container exits 2 with `NoUsableUpstreams`, so load it once before bringing the service up:
```sh
docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/config.zon
```
The file is positional; add `--allow-delete` when re-running it against a
populated volume and the diff deletes rows. Without this step, `restart:
unless-stopped` plus exit 2 is a crash loop with no way out.
The file is positional; add `--allow-delete` when re-running it against a populated volume and the diff deletes rows. Without this step, `restart: unless-stopped` plus exit 2 is a crash loop with no way out.
> Not run in a container on this host, for the reason in the verification note
> at the top: no image could be staged here. The `nxdns import <file>` and
@@ -165,10 +130,7 @@ unless-stopped` plus exit 2 is a crash loop with no way out.
> exited 0, the second was required after a plain `import` refused a
> row-deleting file with `DestructiveImport` and exited 2.
The container runs as uid 65532, and the mount is read-only, so the container
cannot repair permissions itself. Mode 0644 works and was used here. If the
file carries a secret — `web.password`, or a `web.password_hash` from a
restored export — give it to that uid instead:
The container runs as uid 65532, and the mount is read-only, so the container cannot repair permissions itself. Mode 0644 works and was used here. If the file carries a secret — `web.password`, or a `web.password_hash` from a restored export — give it to that uid instead:
```sh
chown 65532:65532 deploy/docker/etc-nxdns/config.zon
@@ -190,10 +152,7 @@ NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml up -d
docker compose -f deploy/docker/compose.yaml logs -f
```
`compose.yaml` reads the image from two variables:
`${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-latest}}`. Set
`NXDNS_VERSION` to pin a release; set `NXDNS_IMAGE` to run something else
entirely, which is what the build-from-source section at the bottom does.
`compose.yaml` reads the image from two variables: `${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-latest}}`. Set `NXDNS_VERSION` to pin a release; set `NXDNS_IMAGE` to run something else entirely, which is what the build-from-source section at the bottom does.
> Verified on this host with `docker compose -f deploy/docker/compose.yaml
> config`, which resolves the variables without contacting a registry: no
@@ -201,12 +160,7 @@ entirely, which is what the build-from-source section at the bottom does.
> gives `git.mial.net/mokhtar/nxdns:0.0.1`, and `NXDNS_IMAGE=nxdns` gives
> `nxdns`.
Every block on this page runs from the repository root, and none of them change
directory, so they can be pasted in order. `-f` is what makes that work:
Compose resolves the relative paths inside `compose.yaml` — the `etc-nxdns`
bind mount — against the directory holding the file, not against your shell,
and it takes the project name `docker` from that directory either way, which is
why the container is `docker-nxdns-1`.
Every block on this page runs from the repository root, and none of them change directory, so they can be pasted in order. `-f` is what makes that work: Compose resolves the relative paths inside `compose.yaml` — the `etc-nxdns` bind mount — against the directory holding the file, not against your shell, and it takes the project name `docker` from that directory either way, which is why the container is `docker-nxdns-1`.
A healthy first start logs the reconcile, the authority and the bound sockets:
@@ -220,8 +174,7 @@ info(nxdns): nxdns <version> serving on udp [::]:53 tcp [::]:53 tcp 0.0.0.0:53;
info(web_server): web interface listening on 0.0.0.0:8080
```
Every later start on an unchanged file reports `reconciled
'/etc/nxdns/config.zon': no changes` and writes nothing to the database.
Every later start on an unchanged file reports `reconciled '/etc/nxdns/config.zon': no changes` and writes nothing to the database.
Confirm it answers and that the admin interface is up:
@@ -240,29 +193,17 @@ curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/
> host port is occupied; free the port or edit the `ports:` list. The image
> under test was built locally, not pulled: there is nothing published to pull.
The compose file publishes 53/udp, 53/tcp and 8080, keeps `/var/lib/nxdns` in
the named volume `nxdns-data`, and sets the per-namespace sysctl
`net.ipv4.ip_unprivileged_port_start=0` so uid 65532 can bind port 53 without
any capability. Uncomment the 443 and 853 mappings when you enable the DoH or
DoT listener; see [Enable DoH and DoT](enable-doh-and-dot.md).
The compose file publishes 53/udp, 53/tcp and 8080, keeps `/var/lib/nxdns` in the named volume `nxdns-data`, and sets the per-namespace sysctl `net.ipv4.ip_unprivileged_port_start=0` so uid 65532 can bind port 53 without any capability. Uncomment the 443 and 853 mappings when you enable the DoH or DoT listener; see [Enable DoH and DoT](enable-doh-and-dot.md).
## 4. Do not point the host at the container
The container resolves its own upstream DoH and DoT hostnames through the
host's DNS configuration. If you set the host's `/etc/resolv.conf` to the nxdns
container, the container's startup lookups depend on the service that is trying
to start. Point LAN clients at nxdns; leave the container's host on its own
resolver.
The container resolves its own upstream DoH and DoT hostnames through the host's DNS configuration. If you set the host's `/etc/resolv.conf` to the nxdns container, the container's startup lookups depend on the service that is trying to start. Point LAN clients at nxdns; leave the container's host on its own resolver.
## Raspberry Pi 5
Nothing changes. The published tag is a multi-architecture index, so
`docker pull` on the Pi selects the `linux/arm64` image on its own. The
platform list is one of the things
[Verify a release](verify-a-release.md) has you check.
Nothing changes. The published tag is a multi-architecture index, so `docker pull` on the Pi selects the `linux/arm64` image on its own. The platform list is one of the things [Verify a release](verify-a-release.md) has you check.
To pull the arm64 image from an x86_64 machine — to inspect it, or to save and
copy it — name the platform:
To pull the arm64 image from an x86_64 machine — to inspect it, or to save and copy it — name the platform:
```sh
docker pull --platform linux/arm64 git.mial.net/mokhtar/nxdns:$VERSION
@@ -274,9 +215,7 @@ docker pull --platform linux/arm64 git.mial.net/mokhtar/nxdns:$VERSION
## Build the image from source instead
The Dockerfile does not compile anything. It assembles a filesystem around
binaries you build first, so build the admin interface and the release
artifacts from the repository root:
The Dockerfile does not compile anything. It assembles a filesystem around binaries you build first, so build the admin interface and the release artifacts from the repository root:
```sh
(cd web && npm ci && npm run build)
@@ -286,19 +225,11 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)"
DOCKER_BUILDKIT=1 docker build -t nxdns -f deploy/docker/Dockerfile .
```
Take the version from `build.zig.zon` rather than inventing one: `verify-dist`
asserts the two agree, so a made-up string builds but fails verification.
BuildKit is required — the Dockerfile pins its builder stage to
`$BUILDPLATFORM`, which the classic builder does not define.
Take the version from `build.zig.zon` rather than inventing one: `verify-dist` asserts the two agree, so a made-up string builds but fails verification. BuildKit is required — the Dockerfile pins its builder stage to `$BUILDPLATFORM`, which the classic builder does not define.
Build `web/dist` every time, before the binaries. A stale bundle is embedded
silently and ships an admin interface that does not match its API — which is
why `dist` refuses to build against the `web/dist-placeholder` default at all.
Build `web/dist` every time, before the binaries. A stale bundle is embedded silently and ships an admin interface that does not match its API — which is why `dist` refuses to build against the `web/dist-placeholder` default at all.
The context has to be the repository root, because the Dockerfile copies
`zig-out/dist/bin` and `zig-out/dist/stage`. The result is a `scratch` image
holding the binary, a CA bundle, `/LICENSE`, `/THIRD-PARTY-NOTICES` and two
empty directories.
The context has to be the repository root, because the Dockerfile copies `zig-out/dist/bin` and `zig-out/dist/stage`. The result is a `scratch` image holding the binary, a CA bundle, `/LICENSE`, `/THIRD-PARTY-NOTICES` and two empty directories.
Run that image instead of the published one by naming it:
@@ -306,16 +237,13 @@ Run that image instead of the published one by naming it:
NXDNS_IMAGE=nxdns docker compose -f deploy/docker/compose.yaml up -d
```
For an arm64 image on an x86_64 machine, use buildx. The Dockerfile's builder
stage is pinned to `$BUILDPLATFORM` and only copies files, so no emulation is
involved:
For an arm64 image on an x86_64 machine, use buildx. The Dockerfile's builder stage is pinned to `$BUILDPLATFORM` and only copies files, so no emulation is involved:
```sh
docker buildx build --platform linux/arm64 -t nxdns:arm64 -f deploy/docker/Dockerfile .
```
Add `--push` or `--load` to keep the result; the default buildx driver leaves
it in the build cache.
Add `--push` or `--load` to keep the result; the default buildx driver leaves it in the build cache.
> Verified on this host, except the two buildx lines. `zig build dist` was run
> to completion with the version read out of `build.zig.zon` and exited 0, and
docs/how-to/install-with-systemd.md
+50 -168
View File
@@ -1,16 +1,10 @@
# Install nxdns with systemd
Installs nxdns as a system service on a Linux host with systemd, including a
Raspberry Pi 5. At the end the service answers DNS on port 53 and starts on
boot.
Installs nxdns as a system service on a Linux host with systemd, including a Raspberry Pi 5. At the end the service answers DNS on port 53 and starts on boot.
The normal path is to download a released tarball, verify it, and install what
is inside it. Building from source is still supported and is the last section
of this page.
The normal path is to download a released tarball, verify it, and install what is inside it. Building from source is still supported and is the last section of this page.
For what each flag does, see [the CLI reference](../reference/cli.md); for what
each configuration field means, see
[the configuration reference](../reference/configuration.md).
For what each flag does, see [the CLI reference](../reference/cli.md); for what each configuration field means, see [the configuration reference](../reference/configuration.md).
> Verification: `systemd-analyze verify` was run on the machine that wrote this
> page. `nxdns check`, `nxdns import`, `nxdns export` and `nxdns run` were run
@@ -33,9 +27,7 @@ each configuration field means, see
## 1. Download and verify
Two static musl tarballs are published per release, one per architecture. Pick
`x86_64-linux-musl` for a normal PC or server and `aarch64-linux-musl` for a
Raspberry Pi 5.
Two static musl tarballs are published per release, one per architecture. Pick `x86_64-linux-musl` for a normal PC or server and `aarch64-linux-musl` for a Raspberry Pi 5.
```sh
BASE=https://git.mial.net/mokhtar/nxdns
@@ -47,15 +39,9 @@ curl -fLO "$BASE/releases/download/v$VERSION/SHA256SUMS.txt"
curl -fLO "$BASE/releases/download/v$VERSION/SHA256SUMS.txt.asc"
```
The first line asks the server which release is current instead of hardcoding a
number that goes stale one release later — Gitea redirects `releases/latest` to
the newest published release's tag page. To install a particular version
instead, set `VERSION=<version>` yourself with the one you want; the tarball
filenames carry the version either way, so there is no version-free download
URL for them.
The first line asks the server which release is current instead of hardcoding a number that goes stale one release later — Gitea redirects `releases/latest` to the newest published release's tag page. To install a particular version instead, set `VERSION=<version>` yourself with the one you want; the tarball filenames carry the version either way, so there is no version-free download URL for them.
Verify before you extract. The signature is over `SHA256SUMS.txt`, and
`SHA256SUMS.txt` is over the tarballs:
Verify before you extract. The signature is over `SHA256SUMS.txt`, and `SHA256SUMS.txt` is over the tarballs:
```sh
gpg --verify SHA256SUMS.txt.asc SHA256SUMS.txt
@@ -63,13 +49,9 @@ sha256sum -c --ignore-missing SHA256SUMS.txt
tar -xzf "nxdns-$VERSION-x86_64-linux-musl.tar.gz"
```
[Verify a release](verify-a-release.md) has the whole procedure: where the
public key comes from, what fingerprint to expect, what each failure means, and
what the signature does and does not prove. Read it once before your first
install.
[Verify a release](verify-a-release.md) has the whole procedure: where the public key comes from, what fingerprint to expect, what each failure means, and what the signature does and does not prove. Read it once before your first install.
The extracted directory `nxdns-$VERSION-x86_64-linux-musl/` holds everything
this page installs:
The extracted directory `nxdns-$VERSION-x86_64-linux-musl/` holds everything this page installs:
| File | What it is |
| --- | --- |
@@ -92,8 +74,7 @@ cd "nxdns-$VERSION-x86_64-linux-musl"
scp nxdns nxdns.service nxdns.conf target:/tmp/
```
For a Raspberry Pi 5, extract the `aarch64-linux-musl` tarball instead — see
[Raspberry Pi 5](#raspberry-pi-5) below.
For a Raspberry Pi 5, extract the `aarch64-linux-musl` tarball instead — see [Raspberry Pi 5](#raspberry-pi-5) below.
> Not verified on this host: `target` is a placeholder for your server's
> hostname, and the machine that wrote this page has no second host to copy
@@ -111,9 +92,7 @@ Off the target host this prints one complaint and exits 1:
nxdns.service: Command /usr/local/bin/nxdns is not executable: No such file or directory
```
That is the ExecStart path check finding no binary yet. Any other message is a
real problem with the unit. On the target, after step 3, the same command
should print nothing.
That is the ExecStart path check finding no binary yet. Any other message is a real problem with the unit. On the target, after step 3, the same command should print nothing.
> Verified on this host against `deploy/systemd/nxdns.service` in a checkout,
> which is the same file the tarball ships — the path is the only difference.
@@ -134,31 +113,21 @@ systemctl daemon-reload
mkdir -p -m 0755 /etc/nxdns
```
The sysusers fragment ships under the name it is installed as, so there is no
rename to get wrong.
The sysusers fragment ships under the name it is installed as, so there is no rename to get wrong.
> Not verified on this host: these commands need root on a target machine. The
> files they install were read at HEAD and the unit was checked with
> `systemd-analyze verify`.
The service user is a static one, not `DynamicUser`: a TLS key for the DoH or
DoT listener has to be chown-able to a uid that survives a restart.
The service user is a static one, not `DynamicUser`: a TLS key for the DoH or DoT listener has to be chown-able to a uid that survives a restart.
Do not create `/var/lib/nxdns` or `/var/log/nxdns` by hand. The unit's
`StateDirectory` and `LogsDirectory` settings make systemd create them on first
start, `/var/lib/nxdns` at mode 0700 owned by `nxdns`.
Do not create `/var/lib/nxdns` or `/var/log/nxdns` by hand. The unit's `StateDirectory` and `LogsDirectory` settings make systemd create them on first start, `/var/lib/nxdns` at mode 0700 owned by `nxdns`.
`/etc/nxdns` is the one directory the `mkdir` above is for. The unit's
`ConfigurationDirectory=nxdns` also creates it, but not until the first start
in step 5, and step 4 has to write a file into it before then. systemd does not
mind finding the directory already there; it adjusts the mode and ownership to
what the unit asks for.
`/etc/nxdns` is the one directory the `mkdir` above is for. The unit's `ConfigurationDirectory=nxdns` also creates it, but not until the first start in step 5, and step 4 has to write a file into it before then. systemd does not mind finding the directory already there; it adjusts the mode and ownership to what the unit asks for.
## 4. Write the configuration
nxdns will not start with nothing to forward to. Write `/etc/nxdns/config.zon`.
The smallest file that starts is one group named `default` and one enabled
upstream:
nxdns will not start with nothing to forward to. Write `/etc/nxdns/config.zon`. The smallest file that starts is one group named `default` and one enabled upstream:
```zon
.{
@@ -168,36 +137,20 @@ upstream:
}
```
That file holds a password in plain text, so restrict it as soon as you have
written it:
That file holds a password in plain text, so restrict it as soon as you have written it:
```sh
chown root:nxdns /etc/nxdns/config.zon
chmod 0640 /etc/nxdns/config.zon
```
Root's umask is 022 on most distributions, so a freshly written
`/etc/nxdns/config.zon` is mode 0644 and every account on the host can read the
password out of it. The unit's `UMask=0077` does not help here: it applies to
files the service creates once it is running, and never re-chmods a file that
was written before the first start.
Root's umask is 022 on most distributions, so a freshly written `/etc/nxdns/config.zon` is mode 0644 and every account on the host can read the password out of it. The unit's `UMask=0077` does not help here: it applies to files the service creates once it is running, and never re-chmods a file that was written before the first start.
0640 with group `nxdns` rather than 0600: `/etc/nxdns` is a
`ConfigurationDirectory`, which systemd leaves owned by root, and the service
runs as `nxdns`. A root-owned 0600 file would be unreadable to it.
0640 with group `nxdns` rather than 0600: `/etc/nxdns` is a `ConfigurationDirectory`, which systemd leaves owned by root, and the service runs as `nxdns`. A root-owned 0600 file would be unreadable to it.
Keep that group read bit for good, not just for the first boot. Under
`run --config` the service reads this file on **every** start, so tightening
the mode later breaks the next restart. Under database authority it is
`nxdns import` that reads the file, as whoever runs that command, and a bare
`nxdns run` never reads it at all.
Keep that group read bit for good, not just for the first boot. Under `run --config` the service reads this file on **every** start, so tightening the mode later breaks the next restart. Under database authority it is `nxdns import` that reads the file, as whoever runs that command, and a bare `nxdns run` never reads it at all.
Do not expect `nxdns check` to catch a permissive mode here. Its only
permission warning is for a TLS private key
(`WARN doh_server.key_path: ... is mode 644; a TLS key must be readable by its
owner only`, from `checkTlsFiles` in `src/cli.zig`); it never stats the
configuration file. A mode 0644 `config.zon` passes `check` in silence, so the
`chmod` above is yours to remember.
Do not expect `nxdns check` to catch a permissive mode here. Its only permission warning is for a TLS private key (`WARN doh_server.key_path: ... is mode 644; a TLS key must be readable by its owner only`, from `checkTlsFiles` in `src/cli.zig`); it never stats the configuration file. A mode 0644 `config.zon` passes `check` in silence, so the `chmod` above is yours to remember.
Check it before you start the service:
@@ -205,8 +158,7 @@ Check it before you start the service:
nxdns check --config /etc/nxdns/config.zon
```
A good file prints the source it checked, one `OK` line per upstream, and
`OK: no problems found`:
A good file prints the source it checked, one `OK` line per upstream, and `OK: no problems found`:
```
checking configuration file /etc/nxdns/config.zon
@@ -214,9 +166,7 @@ OK upstreams[0] https://cloudflare-dns.com
OK: no problems found
```
The upstream probe sends a real query, so this needs working DNS on the host at
the time you run it. Exit 2 means `check` found something to fix and printed
every problem it found, not only the first.
The upstream probe sends a real query, so this needs working DNS on the host at the time you run it. Exit 2 means `check` found something to fix and printed every problem it found, not only the first.
Now load it into the database:
@@ -229,28 +179,17 @@ info(migrations): config.db migrated from schema version 0 to 1
imported /etc/nxdns/config.zon
```
The plaintext password is hashed into `web.password_hash` and never stored as
plaintext; `nxdns export` writes `.password = null` beside the hash. Nothing
downstream reads the plaintext again, so once step 6 confirms you can log in you
can delete the file:
The plaintext password is hashed into `web.password_hash` and never stored as plaintext; `nxdns export` writes `.password = null` beside the hash. Nothing downstream reads the plaintext again, so once step 6 confirms you can log in you can delete the file:
```sh
rm /etc/nxdns/config.zon
```
A kept file is not a backup — `nxdns export` is (see
[Back up and restore](back-up-and-restore.md)), and the export carries the
password hash rather than the password. If you keep it, leave it at 0640
root:nxdns.
A kept file is not a backup — `nxdns export` is (see [Back up and restore](back-up-and-restore.md)), and the export carries the password hash rather than the password. If you keep it, leave it at 0640 root:nxdns.
That is the **database mode** install, which is what the packaged unit runs:
`ExecStart=/usr/local/bin/nxdns run`, no `--config`, so nothing reads a file
after this step. Change settings afterwards through the admin interface, the
API, or an export–edit–import cycle.
That is the **database mode** install, which is what the packaged unit runs: `ExecStart=/usr/local/bin/nxdns run`, no `--config`, so nothing reads a file after this step. Change settings afterwards through the admin interface, the API, or an export–edit–import cycle.
If you would rather keep `/etc/nxdns/config.zon` in git and have every restart
converge onto it, do not delete the file — go to
[Run in file mode](#run-in-file-mode) instead, and skip the `rm`.
If you would rather keep `/etc/nxdns/config.zon` in git and have every restart converge onto it, do not delete the file — go to [Run in file mode](#run-in-file-mode) instead, and skip the `rm`.
> Verified on this host, with a scratch `--config` and `--data-dir` in place of
> `/etc/nxdns` and `/var/lib/nxdns` — those two paths are the only difference
@@ -277,20 +216,11 @@ A healthy start logs a line naming every socket it bound:
info(nxdns): nxdns <version> serving on udp [::]:53 tcp [::]:53 tcp 0.0.0.0:53; 1 upstream(s); blocklist generation 1
```
nxdns writes to stderr and systemd captures that into the journal; logging
needs no further configuration. Port 53 is privileged, and the unit grants
`CAP_NET_BIND_SERVICE` through `AmbientCapabilities`.
nxdns writes to stderr and systemd captures that into the journal; logging needs no further configuration. Port 53 is privileged, and the unit grants `CAP_NET_BIND_SERVICE` through `AmbientCapabilities`.
The unit does not restart the service after exit 2 or exit 64
(`RestartPreventExitStatus=2 64`). Those are a wrong configuration and a wrong
command line, and neither clears on a retry — restarting every two seconds until
`StartLimitBurst` gives up would only bury the diagnostics that are already in
the journal. `systemctl status nxdns` shows the failed state; fix the cause and
start it again.
The unit does not restart the service after exit 2 or exit 64 (`RestartPreventExitStatus=2 64`). Those are a wrong configuration and a wrong command line, and neither clears on a retry — restarting every two seconds until `StartLimitBurst` gives up would only bury the diagnostics that are already in the journal. `systemctl status nxdns` shows the failed state; fix the cause and start it again.
If the start fails, read [Troubleshoot nxdns](troubleshoot.md). The two common
first-install failures are a port 53 already held by `systemd-resolved` and a
configuration file that does not parse.
If the start fails, read [Troubleshoot nxdns](troubleshoot.md). The two common first-install failures are a port 53 already held by `systemd-resolved` and a configuration file that does not parse.
## 6. Confirm it answers
@@ -300,9 +230,7 @@ From another machine on the LAN:
dig @<server-ip> example.com A +short
```
The admin interface is on port 8080 by default; log in with the password from
the configuration file. `http://<server-ip>:8080/api/health` reports upstream
availability and disk state without a login.
The admin interface is on port 8080 by default; log in with the password from the configuration file. `http://<server-ip>:8080/api/health` reports upstream availability and disk state without a login.
> Not verified on this host as written: `<server-ip>` is a placeholder, and a
> LAN client to run it from is a second machine this host does not have. What
@@ -314,25 +242,15 @@ availability and disk state without a login.
## Run in file mode
In file mode `/etc/nxdns/config.zon` is the configuration: every start converges
the database onto it, and the admin interface refuses configuration edits with a
403 naming the file. Use it when you want the file in git and deployed by
Ansible. Stay in database mode when you want the UI to be the way things change.
In file mode `/etc/nxdns/config.zon` is the configuration: every start converges the database onto it, and the admin interface refuses configuration edits with a 403 naming the file. Use it when you want the file in git and deployed by Ansible. Stay in database mode when you want the UI to be the way things change.
The packaged unit is flagless on purpose — it is correct as shipped, and a
commented-out alternative `ExecStart` in a unit file is documentation
masquerading as configuration. File mode is a drop-in.
The packaged unit is flagless on purpose — it is correct as shipped, and a commented-out alternative `ExecStart` in a unit file is documentation masquerading as configuration. File mode is a drop-in.
### Adopt file mode on a box that is already running
Run these in order. **Stop first**, and do not skip that: any edit made through
the UI between an export and the restart would be silently reverted by the first
reconcile, and `nxdns check` against a live database refuses to grade it (below).
Run these in order. **Stop first**, and do not skip that: any edit made through the UI between an export and the restart would be silently reverted by the first reconcile, and `nxdns check` against a live database refuses to grade it (below).
If you are arriving here from an upgrade, the binary must already be the new
one before you export. An export written by 0.0.1 carries a `.password = ""`
line this binary refuses; see
[the order trap](upgrade.md#the-order-trap-export-with-the-new-binary-not-the-old-one).
If you are arriving here from an upgrade, the binary must already be the new one before you export. An export written by 0.0.1 carries a `.password = ""` line this binary refuses; see [the order trap](upgrade.md#the-order-trap-export-with-the-new-binary-not-the-old-one).
```sh
systemctl stop nxdns
@@ -360,12 +278,9 @@ systemctl daemon-reload
systemctl start nxdns
```
The empty `ExecStart=` is required. Without it systemd appends a second command
to the list rather than replacing the first, and the unit tries to run nxdns
twice.
The empty `ExecStart=` is required. Without it systemd appends a second command to the list rather than replacing the first, and the unit tries to run nxdns twice.
The first start after adoption changes nothing, because the file was rendered
from the database it is now governing:
The first start after adoption changes nothing, because the file was rendered from the database it is now governing:
```
reconciled '/etc/nxdns/config.zon': no changes
@@ -373,14 +288,9 @@ info(nxdns): authority: file (/etc/nxdns/config.zon)
info(nxdns): nxdns <version> serving on udp [::]:53 tcp [::]:53 tcp 0.0.0.0:53; 1 upstream(s); blocklist generation 1
```
`authority: file` is the line that confirms the drop-in took. Blocklists,
compiled snapshots and client history all survive, and every later start on an
unchanged file writes nothing either.
`authority: file` is the line that confirms the drop-in took. Blocklists, compiled snapshots and client history all survive, and every later start on an unchanged file writes nothing either.
The file now carries `web.password_hash`, so restrict it the same way step 4
does — `chown root:nxdns`, `chmod 0640`. The unit's `ReadOnlyPaths=/etc/nxdns`
denies the service write access to that directory, so the process that reads the
file cannot modify it.
The file now carries `web.password_hash`, so restrict it the same way step 4 does — `chown root:nxdns`, `chmod 0640`. The unit's `ReadOnlyPaths=/etc/nxdns` denies the service write access to that directory, so the process that reads the file cannot modify it.
### Change the configuration from now on
@@ -392,11 +302,7 @@ nxdns check --config /etc/nxdns/config.zon
systemctl restart nxdns
```
Make `nxdns check --config` the precondition of any Ansible handler that
restarts nxdns. A file-mode start reads the file on **every** boot, so a bad
push that skips its handler does not fail at deploy time — it detonates at the
next power cut. Validating before restarting turns that into a failed deploy at
noon.
Make `nxdns check --config` the precondition of any Ansible handler that restarts nxdns. A file-mode start reads the file on **every** boot, so a bad push that skips its handler does not fail at deploy time — it detonates at the next power cut. Validating before restarting turns that into a failed deploy at noon.
The restart prints what it changed:
@@ -407,9 +313,7 @@ settings keys changed: dns.port web.port
### Leave file mode
Remove the drop-in and restart. The database already holds the last reconciled
state, so nothing else is needed and the server comes back serving the same
configuration:
Remove the drop-in and restart. The database already holds the last reconciled state, so nothing else is needed and the server comes back serving the same configuration:
```sh
rm /etc/systemd/system/nxdns.service.d/file-mode.conf
@@ -448,8 +352,7 @@ info(nxdns): authority: database
## Raspberry Pi 5
The Pi 5 is aarch64. Nothing about the procedure changes except which tarball
you take:
The Pi 5 is aarch64. Nothing about the procedure changes except which tarball you take:
```sh
curl -fLO "$BASE/releases/download/v$VERSION/nxdns-$VERSION-aarch64-linux-musl.tar.gz"
@@ -467,12 +370,7 @@ Then follow steps 3 to 6 on the Pi.
## Build from source instead
You do not need this to install nxdns, and it gets you a binary nobody has
signed. It is here for two cases: you want to run something other than a
tagged release, or you want to build the release yourself and compare it
against the published one. For the second case, follow
[Verify a release](verify-a-release.md) rather than this section — it says what
the comparison is and is not worth.
You do not need this to install nxdns, and it gets you a binary nobody has signed. It is here for two cases: you want to run something other than a tagged release, or you want to build the release yourself and compare it against the published one. For the second case, follow [Verify a release](verify-a-release.md) rather than this section — it says what the comparison is and is not worth.
Requires Zig 0.16.0 and Node.js. From the repository root:
@@ -483,29 +381,18 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)"
-Dweb-dist=web/dist -Doptimize=ReleaseSafe
```
The first command builds the admin interface into `web/dist`; the last one
embeds that directory in the binary. Build the interface every time, before the
binary: a stale `web/dist` ships an admin UI that does not match the API it
talks to. `dist` refuses to run against the `web/dist-placeholder` default for
exactly that reason, so there is no way to skip it by accident.
The first command builds the admin interface into `web/dist`; the last one embeds that directory in the binary. Build the interface every time, before the binary: a stale `web/dist` ships an admin UI that does not match the API it talks to. `dist` refuses to run against the `web/dist-placeholder` default for exactly that reason, so there is no way to skip it by accident.
`-Dversion-string` is required and has no default. It is what `nxdns version`
prints. Take it from `build.zig.zon` rather than inventing one: `verify-dist`
asserts that the version under build equals `.version` there, so a made-up
string like `0.0.0-local` builds but then fails verification. `-Dgit-commit`
is what distinguishes your build from the published one of the same version.
`-Dversion-string` is required and has no default. It is what `nxdns version` prints. Take it from `build.zig.zon` rather than inventing one: `verify-dist` asserts that the version under build equals `.version` there, so a made-up string like `0.0.0-local` builds but then fails verification. `-Dgit-commit` is what distinguishes your build from the published one of the same version.
What comes out under `zig-out/dist/` is the same set a release publishes,
minus the signature and the image digest:
What comes out under `zig-out/dist/` is the same set a release publishes, minus the signature and the image digest:
- `bin/<triple>/nxdns` — the stripped static binary, one per target
- `stage/nxdns-<version>-<triple>/` — the staged payload, one per target
- `nxdns-<version>-<triple>.tar.gz` — one tarball per target
- `SHA256SUMS` — the two tarball hashes. The release publishes this as
`SHA256SUMS.txt`, with a third line for the image digest appended
- `SHA256SUMS` — the two tarball hashes. The release publishes this as `SHA256SUMS.txt`, with a third line for the image digest appended
The two targets are `x86_64-linux-musl` and `aarch64-linux-musl`. Both binaries
are statically linked and need nothing installed on the target host.
The two targets are `x86_64-linux-musl` and `aarch64-linux-musl`. Both binaries are statically linked and need nothing installed on the target host.
Check the result the same way the release pipeline does:
@@ -514,10 +401,7 @@ zig build verify-dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse
-Dweb-dist=web/dist -Doptimize=ReleaseSafe
```
`verify-dist` extracts each archive and asserts the ELF is static and within
the size budget, that the layout and file modes are exactly what step 1 lists,
and that `nxdns version` prints what was built. It exits non-zero on any
failure.
`verify-dist` extracts each archive and asserts the ELF is static and within the size budget, that the layout and file modes are exactly what step 1 lists, and that `nxdns version` prints what was built. It exits non-zero on any failure.
> Verified on this host: `zig build dist` and `zig build verify-dist` were both
> run to completion with the version taken from `build.zig.zon`. `dist`
@@ -529,13 +413,11 @@ failure.
> `FAIL zon-version: build.zig.zon says '0.0.1', the build says '0.0.0-local'`,
> which is why this section reads the version out of `build.zig.zon`.
From here, join the page at step 2 with the staged directory in place of the
extracted one:
From here, join the page at step 2 with the staged directory in place of the extracted one:
```sh
cd "zig-out/dist/stage/nxdns-$VERSION-x86_64-linux-musl"
scp nxdns nxdns.service nxdns.conf target:/tmp/
```
The aarch64 binary is built by the same command and needs no toolchain on the
Pi.
The aarch64 binary is built by the same command and needs no toolchain on the Pi.
docs/how-to/measure-performance.md
+20 -67
View File
@@ -1,14 +1,8 @@
# Measure performance
`tools/bench.zig` measures the three things nxdns can measure in-process:
blocklist lookup latency, cache-hit latency, and blocklist compile throughput.
Sustained query rate is not one of them — that one is end-to-end and needs a
load generator pointed at a running server.
`tools/bench.zig` measures the three things nxdns can measure in-process: blocklist lookup latency, cache-hit latency, and blocklist compile throughput. Sustained query rate is not one of them — that one is end-to-end and needs a load generator pointed at a running server.
The numbers this project treats as targets, and the numbers measured so far,
are in the [performance reference](../reference/performance.md). Why those
targets exist and why CI does not gate on them is in
[performance and testing](../explanation/performance-and-testing.md).
The numbers this project treats as targets, and the numbers measured so far, are in the [performance reference](../reference/performance.md). Why those targets exist and why CI does not gate on them is in [performance and testing](../explanation/performance-and-testing.md).
## Run the whole bench
@@ -16,12 +10,9 @@ targets exist and why CI does not gate on them is in
zig build bench -Doptimize=ReleaseFast
```
That runs all three suites with the defaults: 1,000,000 domains, 200,000
iterations per suite, seed `0x5eed`. It takes minutes, most of it generating and
loading the million-domain list.
That runs all three suites with the defaults: 1,000,000 domains, 200,000 iterations per suite, seed `0x5eed`. It takes minutes, most of it generating and loading the million-domain list.
`-Doptimize=ReleaseFast` is not optional if you want the numbers to mean
anything. A Debug build says so before it prints:
`-Doptimize=ReleaseFast` is not optional if you want the numbers to mean anything. A Debug build says so before it prints:
```
warning: Debug build; run with -Doptimize=ReleaseFast for meaningful numbers
@@ -29,9 +20,7 @@ warning: Debug build; run with -Doptimize=ReleaseFast for meaningful numbers
## Run one suite, smaller
Everything after `--` goes to the harness. A suite name selects one of
`filter`, `cache`, `compile` (the default is `all`), and `--domains` /
`--iters` shrink the load:
Everything after `--` goes to the harness. A suite name selects one of `filter`, `cache`, `compile` (the default is `all`), and `--domains` / `--iters` shrink the load:
```sh
zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000
@@ -47,10 +36,7 @@ filter 20000 2.38 2.76 2.88 20.32
target VmRSS < 100 MiB: PASS
```
A reduced run is good for checking the harness works and for a rough
regression signal. It is not a result: the memory figure scales with
`--domains`, so 100,000 domains says nothing about the 1,000,000-domain memory
target.
A reduced run is good for checking the harness works and for a rough regression signal. It is not a result: the memory figure scales with `--domains`, so 100,000 domains says nothing about the 1,000,000-domain memory target.
The other two suites:
@@ -74,10 +60,7 @@ suite ops p50(us) p95(us) p99(us) max(us)
compile 100000 wall 15.623ms, 6400464 lines/s, 100000 domains kept (informational)
```
`--seed=N` changes the generated domains and the query order; the default is
`0x5eed`, so two runs on the same machine are comparable. `--domains` caps at
4,000,000, and the `compile` suite additionally refuses more than 2,000,000 —
the compiler's own limit.
`--seed=N` changes the generated domains and the query order; the default is `0x5eed`, so two runs on the same machine are comparable. `--domains` caps at 4,000,000, and the `compile` suite additionally refuses more than 2,000,000 — the compiler's own limit.
An argument the harness does not recognise stops it before any measuring:
@@ -88,36 +71,21 @@ usage: zig build bench -Doptimize=ReleaseFast -- [filter|cache|compile|all] [--d
## Read the output
- `p50`/`p95`/`p99`/`max` are per-operation microseconds, nearest-rank over
every iteration. What one operation means differs per suite: for `filter` it
is normalising a name plus evaluating it against the snapshot; for `cache` it
is building the key, getting the entry and stamping the response id.
- `blocked N/M` and `hits N/M` are sanity counters. The harness aborts if either
is zero — a suite that never hits its own path measures nothing.
- `32 regex rules` on the `filter` line is the rule set the suite loads. No
generated query matches any of them, so every operation runs all 32 programs
to their end, which is the costly case and the one worth measuring.
- Two memory figures appear on purpose. `Snapshot.memoryBytes` and
`DnsCache.memoryBytes` are the in-repo accounting of those structures; `VmRSS`
is what the kernel holds resident for the whole process, allocator slack and
code included. The truth is between them, and the memory target is judged on
`VmRSS`.
- `target ...: PASS` / `FAIL` lines appear for the targets a suite covers. On a
plain run they are informational and the exit code stays 0.
- `p50`/`p95`/`p99`/`max` are per-operation microseconds, nearest-rank over every iteration. What one operation means differs per suite: for `filter` it is normalising a name plus evaluating it against the snapshot; for `cache` it is building the key, getting the entry and stamping the response id.
- `blocked N/M` and `hits N/M` are sanity counters. The harness aborts if either is zero — a suite that never hits its own path measures nothing.
- `32 regex rules` on the `filter` line is the rule set the suite loads. No generated query matches any of them, so every operation runs all 32 programs to their end, which is the costly case and the one worth measuring.
- Two memory figures appear on purpose. `Snapshot.memoryBytes` and `DnsCache.memoryBytes` are the in-repo accounting of those structures; `VmRSS` is what the kernel holds resident for the whole process, allocator slack and code included. The truth is between them, and the memory target is judged on `VmRSS`.
- `target ...: PASS` / `FAIL` lines appear for the targets a suite covers. On a plain run they are informational and the exit code stays 0.
## Fail the run when a target is missed
`--assert` turns those lines into an exit code — 1 when any target was
exceeded, 0 otherwise. This is meant for an acceptance run on hardware you
control, not for CI:
`--assert` turns those lines into an exit code — 1 when any target was exceeded, 0 otherwise. This is meant for an acceptance run on hardware you control, not for CI:
```sh
zig build bench -Doptimize=ReleaseFast -- --assert
```
The full-scale form is the one worth asserting on, because the memory target
only means something at a million domains. On this development host the reduced
form was used to check the flag itself:
The full-scale form is the one worth asserting on, because the memory target only means something at a million domains. On this development host the reduced form was used to check the flag itself:
```sh
zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000 --assert
@@ -130,37 +98,22 @@ filter 20000 2.34 2.71 2.85 15.06
target VmRSS < 100 MiB: PASS
```
**Not verified on this host at full scale:** the plain
`zig build bench -Doptimize=ReleaseFast -- --assert` above was not run during
the writing of this page — the default run takes minutes. The reduced runs
shown were all executed as written. The full-scale numbers already recorded for
this host are in the [performance reference](../reference/performance.md).
**Not verified on this host at full scale:** the plain `zig build bench -Doptimize=ReleaseFast -- --assert` above was not run during the writing of this page — the default run takes minutes. The reduced runs shown were all executed as written. The full-scale numbers already recorded for this host are in the [performance reference](../reference/performance.md).
## Measure sustained query rate
The bench harness cannot do this. Query rate is a property of the whole server
— sockets, upstreams, the query log writer — so it has to be driven from
outside, against the real binary, on the machine you care about.
The bench harness cannot do this. Query rate is a property of the whole server — sockets, upstreams, the query log writer — so it has to be driven from outside, against the real binary, on the machine you care about.
Start nxdns with real blocklists configured, then drive it from another host on
the LAN with a DNS load generator such as `dnsperf`:
Start nxdns with real blocklists configured, then drive it from another host on the LAN with a DNS load generator such as `dnsperf`:
```sh
dnsperf -s 192.168.1.10 -p 53 -d queries.txt -c 20 -Q 200 -l 60
```
Read the client's own rate and the server's `/metrics` together: a load
generator that reports 200 qps while the server counts fewer has lost queries
somewhere, and that is the interesting number.
Read the client's own rate and the server's `/metrics` together: a load generator that reports 200 qps while the server counts fewer has lost queries somewhere, and that is the interesting number.
**Not verified on this host:** `dnsperf` is not installed here and the target
platform is a Raspberry Pi 5, not this development machine. The command above
is the shape of the measurement, not a transcript.
**Not verified on this host:** `dnsperf` is not installed here and the target platform is a Raspberry Pi 5, not this development machine. The command above is the shape of the measurement, not a transcript.
## Where to run it
The target platform is a Raspberry Pi 5. Numbers from a development x86_64 box
do not transfer — the Pi's Cortex-A76 is far slower — so a passing run here is
evidence the harness works and a baseline for spotting regressions on the
machine development happens on, and nothing more. Run `--assert` on the Pi,
where the numbers mean something.
The target platform is a Raspberry Pi 5. Numbers from a development x86_64 box do not transfer — the Pi's Cortex-A76 is far slower — so a passing run here is evidence the harness works and a baseline for spotting regressions on the machine development happens on, and nothing more. Run `--assert` on the Pi, where the numbers mean something.
docs/how-to/set-up-admin-authentication.md
+32 -106
View File
@@ -1,13 +1,8 @@
# Set up admin authentication
The admin interface and its API are protected by a single operator password.
With no password set, every route is open to anything that can reach the web
port. Set one.
The admin interface and its API are protected by a single operator password. With no password set, every route is open to anything that can reach the web port. Set one.
The commands below run against the scratch lab from
[enable DoH and DoT](enable-doh-and-dot.md): data directory
`/tmp/nxdns-lab/data`, web listener on `127.0.0.1:8451`. On a real install the
data directory is `/var/lib/nxdns` and the web port is 8080.
The commands below run against the scratch lab from [enable DoH and DoT](enable-doh-and-dot.md): data directory `/tmp/nxdns-lab/data`, web listener on `127.0.0.1:8451`. On a real install the data directory is `/var/lib/nxdns` and the web port is 8080.
## 1. Set the password
@@ -21,17 +16,14 @@ Put it in the configuration file, under `web`:
}
```
The plaintext is hashed with argon2id into `web.password_hash` and discarded. It
becomes no database row and appears in no log line. Setting both `password` and
`password_hash` in one file is refused:
The plaintext is hashed with argon2id into `web.password_hash` and discarded. It becomes no database row and appears in no log line. Setting both `password` and `password_hash` in one file is refused:
```
web.password: password and password_hash are both set; ambiguity in a security setting is refused
import failed: PasswordAndHashBothSet
```
Applying that file — with `nxdns import`, or with a `nxdns run --config` start —
announces the change:
Applying that file — with `nxdns import`, or with a `nxdns run --config` start — announces the change:
```
web authentication is now enabled
@@ -39,9 +31,7 @@ web authentication is now enabled
### Absent, empty, and set are three different things
The two fields are optional, and the difference between leaving one out and
setting it to `""` is the difference between keeping your password and removing
it:
The two fields are optional, and the difference between leaving one out and setting it to `""` is the difference between keeping your password and removing it:
| The file says | Effect on the stored password |
| --- | --- |
@@ -51,11 +41,7 @@ it:
| `.password_hash = "$argon2id$…"` | Installs that hash, for example from an export. |
| `.password_hash = ""` | **Removes the password.** Authentication is off. |
Absence has to mean "keep", because the alternative is a foot-gun with a live
round in it. An export carries the full PHC string, which is long and ugly, and
sooner or later someone trims that line out of a file before committing it —
meaning "leave the password alone". If absence meant "no password", that edit
would open the admin interface to the whole LAN without a word.
Absence has to mean "keep", because the alternative is a foot-gun with a live round in it. An export carries the full PHC string, which is long and ugly, and sooner or later someone trims that line out of a file before committing it — meaning "leave the password alone". If absence meant "no password", that edit would open the admin interface to the whole LAN without a word.
So removing the password takes the explicit empty string:
@@ -63,23 +49,17 @@ So removing the password takes the explicit empty string:
web authentication is now disabled
```
And an empty plaintext is refused outright, because hashing the empty string
would switch authentication *on* while making every login impossible — the login
handler rejects empty passwords:
And an empty plaintext is refused outright, because hashing the empty string would switch authentication *on* while making every login impossible — the login handler rejects empty passwords:
```
FAIL web.password: password is set to the empty string; omit the field to keep the stored password, or set password_hash = "" to disable authentication
```
Which of steps 4 and 5 applies to your server depends on its authority. Under
`nxdns run --config FILE` the file is the password: edit it and restart, and the
API refuses the change with a 403. Under bare `nxdns run` the database holds it,
and step 4 or step 5 is how it moves.
Which of steps 4 and 5 applies to your server depends on its authority. Under `nxdns run --config FILE` the file is the password: edit it and restart, and the API refuses the change with a 403. Under bare `nxdns run` the database holds it, and step 4 or step 5 is how it moves.
## 2. Log in
Login is `POST /api/auth/login` with a JSON body. Without a session, the API
answers 401:
Login is `POST /api/auth/login` with a JSON body. Without a session, the API answers 401:
```sh
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8451/api/stats
@@ -102,16 +82,13 @@ curl -sS -c /tmp/nxdns-lab/cookies.txt \
{"authenticated":true,"auth_required":true}
```
The session token comes back in a `Set-Cookie` header, not in the body. In the
jar it looks like this (value redacted here):
The session token comes back in a `Set-Cookie` header, not in the body. In the jar it looks like this (value redacted here):
```
#HttpOnly_127.0.0.1 FALSE / FALSE 1786559938 nxdns_session <redacted>
```
The cookie is named `nxdns_session` and carries `HttpOnly; SameSite=Lax;
Path=/`. Its `Max-Age` comes from `web.session_ttl_hours`. Send it back on every
later call:
The cookie is named `nxdns_session` and carries `HttpOnly; SameSite=Lax; Path=/`. Its `Max-Age` comes from `web.session_ttl_hours`. Send it back on every later call:
```sh
curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w '%{http_code}\n' \
@@ -122,8 +99,7 @@ curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w '%{http_code}\n' \
200
```
A wrong password and an unknown one are the same answer, so a guess learns
nothing:
A wrong password and an unknown one are the same answer, so a guess learns nothing:
```sh
curl -sS -X POST http://127.0.0.1:8451/api/auth/login \
@@ -142,9 +118,7 @@ info(web_auth): web login accepted for 127.0.0.1:34040
warning(web_auth): web login refused for 127.0.0.1:59670
```
Sessions live in memory only. A restart logs everyone out. Thirty-two
concurrent sessions are kept; a thirty-third login evicts the least recently
used one.
Sessions live in memory only. A restart logs everyone out. Thirty-two concurrent sessions are kept; a thirty-third login evicts the least recently used one.
## 3. Log out
@@ -160,18 +134,13 @@ curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w 'stats: %{http_code}\n' \
stats: 401
```
Logging out with a stale cookie, or with none, answers the same way. The point
of logging out is to end up logged out, and that is where such a request
already is.
Logging out with a stale cookie, or with none, answers the same way. The point of logging out is to end up logged out, and that is where such a request already is.
## 4. Change the password on a running server
This is a database-mode procedure. In file mode `PUT /api/settings` answers 403
naming the file; edit `web.password` there and restart instead.
This is a database-mode procedure. In file mode `PUT /api/settings` answers 403 naming the file; edit `web.password` there and restart instead.
Send the new one to `PUT /api/settings` as `web.password`. The response is the
full settings document; `password` is write-only and `password_hash` is neither
readable nor directly writable, so neither value comes back.
Send the new one to `PUT /api/settings` as `web.password`. The response is the full settings document; `password` is write-only and `password_hash` is neither readable nor directly writable, so neither value comes back.
```sh
curl -sS -c /tmp/nxdns-lab/c2.txt -X POST http://127.0.0.1:8451/api/auth/login \
@@ -181,8 +150,7 @@ curl -sS -b /tmp/nxdns-lab/c2.txt -X PUT http://127.0.0.1:8451/api/settings \
-d '{"web":{"password":"a-new-password"}}'
```
Changing the password ends every session, including the one that made the
change:
Changing the password ends every session, including the one that made the change:
```sh
curl -sS -b /tmp/nxdns-lab/c2.txt -o /dev/null -w 'old session: %{http_code}\n' \
@@ -205,39 +173,26 @@ Log back in with the new password. That is the whole rotation.
## 5. Change the password without the API
If you have lost the password, the admin interface cannot help — go through the
database instead. Export, edit, import. `nxdns export` always writes
`.password = null` and carries the hash, so an exported file re-imports without
anyone knowing the password. To install a new one, put it in `.password` and
clear `.password_hash`:
If you have lost the password, the admin interface cannot help — go through the database instead. Export, edit, import. `nxdns export` always writes `.password = null` and carries the hash, so an exported file re-imports without anyone knowing the password. To install a new one, put it in `.password` and clear `.password_hash`:
```sh
nxdns export --data-dir /tmp/nxdns-lab/data --out /tmp/nxdns-lab/rekeyed.zon
```
Edit the `web` section of `/tmp/nxdns-lab/rekeyed.zon`: set `.password` to the
new value and **delete the `.password_hash` line entirely**, so the `web` block
carries one password field and not two:
Edit the `web` section of `/tmp/nxdns-lab/rekeyed.zon`: set `.password` to the new value and **delete the `.password_hash` line entirely**, so the `web` block carries one password field and not two:
```zon
.password = "offline-password",
```
Deleting the line is the part to get right. Setting `.password_hash = ""`
alongside a plaintext password does not clear the way for it — an empty string
is a present value meaning "no password", so the file then states two
contradictory things and is refused:
Deleting the line is the part to get right. Setting `.password_hash = ""` alongside a plaintext password does not clear the way for it — an empty string is a present value meaning "no password", so the file then states two contradictory things and is refused:
```
FAIL web.password: password and password_hash are both set; ambiguity in a security setting is refused
import failed: PasswordAndHashBothSet
```
Stop the server before importing. `import` rewrites the stored hash underneath a
process that read it at startup; a running server keeps verifying against the
old one, so skipping the stop leaves the new password not working until the next
restart. In the lab the server is a foreground `nxdns run`, so Ctrl-C in its
terminal stops it, and it goes back up with the same command:
Stop the server before importing. `import` rewrites the stored hash underneath a process that read it at startup; a running server keeps verifying against the old one, so skipping the stop leaves the new password not working until the next restart. In the lab the server is a foreground `nxdns run`, so Ctrl-C in its terminal stops it, and it goes back up with the same command:
```sh
# Ctrl-C the `nxdns run` terminal, or `kill` its pid from another shell
@@ -249,14 +204,9 @@ nxdns run --data-dir /tmp/nxdns-lab/data
imported /tmp/nxdns-lab/rekeyed.zon
```
No flag is needed: replacing a password edits a settings value and deletes no
rows.
No flag is needed: replacing a password edits a settings value and deletes no rows.
On a real install the stop and start are `systemctl stop nxdns` and
`systemctl start nxdns` around the same `import` — **not verified on this
host**, which has no installed nxdns systemd unit (`systemctl status nxdns`
answers `Unit nxdns.service could not be found.`) and where `systemctl` needs
root. See [back up and restore](back-up-and-restore.md).
On a real install the stop and start are `systemctl stop nxdns` and `systemctl start nxdns` around the same `import` — **not verified on this host**, which has no installed nxdns systemd unit (`systemctl status nxdns` answers `Unit nxdns.service could not be found.`) and where `systemctl` needs root. See [back up and restore](back-up-and-restore.md).
Once it is back up the old password is refused and the new one works:
@@ -288,14 +238,11 @@ nxdns export --data-dir /tmp/nxdns-lab/data | grep password
.password_hash = "$argon2id$v=19$m=19456,t=2,p=1$xqzK66LgiWGvyCmCl6ZRa3GHH0nS5qZnRgVfWmeGadc$1mafhflKFIg3vcHjJaDMAXGQiOjtym2UADZsPW1xkfw",
```
See [back up and restore](back-up-and-restore.md) for when `import` does need
`--allow-delete`.
See [back up and restore](back-up-and-restore.md) for when `import` does need `--allow-delete`.
## What happens with no password set
Authentication is off. Every route is open, and a login attempt succeeds
without minting anything — there is nothing to log in to, and a session that
authorises nothing would be a lie for the browser to store:
Authentication is off. Every route is open, and a login attempt succeeds without minting anything — there is nothing to log in to, and a session that authorises nothing would be a lie for the browser to store:
```sh
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8453/api/stats
@@ -308,24 +255,13 @@ curl -sS -X POST http://127.0.0.1:8453/api/auth/login \
{"authenticated":true,"auth_required":false}
```
`auth_required: false` is how the admin interface knows to stop showing a login
form. Treat this as a lab-only state: bind the web listener to a trusted
interface at the very least, and preferably set a password.
`auth_required: false` is how the admin interface knows to stop showing a login form. Treat this as a lab-only state: bind the web listener to a trusted interface at the very least, and preferably set a password.
## Notes
- A stored hash this build cannot parse is a 500, not a 401. Answering 401 would
tell an operator with a corrupted `web.password_hash` that their password is
wrong, and they would retype a password that can never verify.
- Requests from the box itself skip the API rate limit by default
(`web.api_localhost_exempt`).
- **If you put a reverse proxy in front of the admin interface, configure
`web.trusted_proxies` or turn `web.api_localhost_exempt` off.** A proxy on the
same box connects from loopback, so every request arrives exempt and the API
limiter — the only brake on guessing the admin password — stops applying to
anyone. Listing the proxy's address in `web.trusted_proxies` makes nxdns read
the client's address from the `X-Forwarded-For` the proxy appends, so the
limiter and the SSE connection cap bind each real client again:
- A stored hash this build cannot parse is a 500, not a 401. Answering 401 would tell an operator with a corrupted `web.password_hash` that their password is wrong, and they would retype a password that can never verify.
- Requests from the box itself skip the API rate limit by default (`web.api_localhost_exempt`).
- **If you put a reverse proxy in front of the admin interface, configure `web.trusted_proxies` or turn `web.api_localhost_exempt` off.** A proxy on the same box connects from loopback, so every request arrives exempt and the API limiter — the only brake on guessing the admin password — stops applying to anyone. Listing the proxy's address in `web.trusted_proxies` makes nxdns read the client's address from the `X-Forwarded-For` the proxy appends, so the limiter and the SSE connection cap bind each real client again:
```zig
.web = .{
@@ -333,17 +269,7 @@ interface at the very least, and preferably set a password.
},
```
The proxy must append its own entry to that header. A proxy that forwards a
client-supplied `X-Forwarded-For` unchanged is not one to trust.
- `web.session_ttl_hours`, `web.api_rate_limit_per_min` and the rest are in the
[configuration reference](../reference/configuration.md); the routes are in
the [API reference](../reference/api.md).
The proxy must append its own entry to that header. A proxy that forwards a client-supplied `X-Forwarded-For` unchanged is not one to trust.
- `web.session_ttl_hours`, `web.api_rate_limit_per_min` and the rest are in the [configuration reference](../reference/configuration.md); the routes are in the [API reference](../reference/api.md).
Every command on this page was executed on this host as written, against the
lab described at the top, except the `systemctl` stop and start named in step 5
and marked **not verified on this host** there. That includes the whole of
steps 2 to 5, re-run for this revision: the login, logout and rate-limit
transcripts reproduced exactly as printed, the both-set refusal in step 5 was
reproduced by leaving `.password_hash = ""` in the file, and the rekey then
succeeded once that line was deleted. The cookie jar's expiry timestamp is the
one that run produced and will differ on yours.
Every command on this page was executed on this host as written, against the lab described at the top, except the `systemctl` stop and start named in step 5 and marked **not verified on this host** there. That includes the whole of steps 2 to 5, re-run for this revision: the login, logout and rate-limit transcripts reproduced exactly as printed, the both-set refusal in step 5 was reproduced by leaving `.password_hash = ""` in the file, and the rekey then succeeded once that line was deleted. The cookie jar's expiry timestamp is the one that run produced and will differ on yours.
docs/how-to/troubleshoot.md
+49 -159
View File
@@ -1,29 +1,19 @@
# Troubleshoot nxdns
Symptoms an nxdns install actually produces, what to run to identify each one,
and what to change. Every symptom on this page was reproduced on the machine
that wrote it, and every diagnosis command was run there. Two details differ
from a real install and cannot be otherwise on that machine: it has no
installed service, so the log lines were read from a foreground run instead of
`journalctl -u nxdns`, and ports 53 and 8080 were occupied, so DNS and the API
were exercised on unprivileged ports. Fixes that need root are marked.
Symptoms an nxdns install actually produces, what to run to identify each one, and what to change. Every symptom on this page was reproduced on the machine that wrote it, and every diagnosis command was run there. Two details differ from a real install and cannot be otherwise on that machine: it has no installed service, so the log lines were read from a foreground run instead of `journalctl -u nxdns`, and ports 53 and 8080 were occupied, so DNS and the API were exercised on unprivileged ports. Fixes that need root are marked.
The exit codes themselves are listed in
[the CLI reference](../reference/cli.md).
The exit codes themselves are listed in [the CLI reference](../reference/cli.md).
## The service exits with code 2
**Symptom.** The process stops immediately. The last two lines are the error
and a pointer:
**Symptom.** The process stops immediately. The last two lines are the error and a pointer:
```
nxdns run failed: NoUsableUpstreams
run `nxdns check` to see the configuration in full
```
Exit 2 means the configuration is wrong and you can fix it. Every subcommand
uses the same definition, so a file `run` exits 2 on exits 2 from `check` and
`import` too.
Exit 2 means the configuration is wrong and you can fix it. Every subcommand uses the same definition, so a file `run` exits 2 on exits 2 from `check` and `import` too.
**Diagnosis.**
@@ -31,14 +21,11 @@ uses the same definition, so a file `run` exits 2 on exits 2 from `check` and
nxdns check
```
`check` prints every problem it finds, not the first, and names the source it
checked on its first line.
`check` prints every problem it finds, not the first, and names the source it checked on its first line.
**Fixes by cause.**
- `NoUsableUpstreams` — the database has no enabled upstream. On a fresh
install in database mode this is simply an empty database, and the run says
what to do about it on the next line:
- `NoUsableUpstreams` — the database has no enabled upstream. On a fresh install in database mode this is simply an empty database, and the run says what to do about it on the next line:
```
nxdns run failed: NoUsableUpstreams
@@ -46,22 +33,11 @@ checked on its first line.
load one with `nxdns import <file>`, or make a file the source of truth with `nxdns run --config <file>`
```
Write a configuration file and take either exit: `nxdns import <file>` to load
it into the database once, or add `--config <file>` to `ExecStart` to make the
file the configuration from then on.
- `ManagedConfigUnreadable` — the service runs `run --config FILE` and that file
is missing or the process may not read it. The path is in the FAIL line above
the failure. File mode never falls back to the database, on purpose: a
fallback would turn a bad deploy into a silently stale configuration.
- `BadCertificate` — a DoH or DoT listener is enabled and its certificate or
key is unreadable, too large, unparseable, or the key does not belong to the
certificate. `run` names both paths before it exits:
`doh_server: '<cert>' + '<key>': certificate file is not readable`.
Write a configuration file and take either exit: `nxdns import <file>` to load it into the database once, or add `--config <file>` to `ExecStart` to make the file the configuration from then on.
- `ManagedConfigUnreadable` — the service runs `run --config FILE` and that file is missing or the process may not read it. The path is in the FAIL line above the failure. File mode never falls back to the database, on purpose: a fallback would turn a bad deploy into a silently stale configuration.
- `BadCertificate` — a DoH or DoT listener is enabled and its certificate or key is unreadable, too large, unparseable, or the key does not belong to the certificate. `run` names both paths before it exits: `doh_server: '<cert>' + '<key>': certificate file is not readable`.
`check` catches this without starting a listener. It loads both PEM files and
tests the key against the certificate through the same code `run` uses, so it
fails on exactly what `run` would fail on. Reproduced here with a self-signed
pair and the key from a second, unrelated pair:
`check` catches this without starting a listener. It loads both PEM files and tests the key against the certificate through the same code `run` uses, so it fails on exactly what `run` would fail on. Reproduced here with a self-signed pair and the key from a second, unrelated pair:
```
$ nxdns check --config config.zon
@@ -77,24 +53,15 @@ checked on its first line.
nxdns run failed: BadCertificate # exit 2
```
The `warning(tls_server)` line comes from mbedTLS on stderr and can appear
before the `checking` line, which is on stdout. A cert file containing
`not a certificate` fails the same way, with
`FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`.
An unreadable file reads
`FAIL doh_server.cert_path: '<path>': certificate file is not readable`.
The `warning(tls_server)` line comes from mbedTLS on stderr and can appear before the `checking` line, which is on stdout. A cert file containing `not a certificate` fails the same way, with `FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`. An unreadable file reads `FAIL doh_server.cert_path: '<path>': certificate file is not readable`.
Fix the path, the ownership, or the pair; see
[Enable DoH and DoT](enable-doh-and-dot.md).
- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a
configuration, so this only reaches a database that was edited by hand.
- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of
that family.
Fix the path, the ownership, or the pair; see [Enable DoH and DoT](enable-doh-and-dot.md).
- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a configuration, so this only reaches a database that was edited by hand.
- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of that family.
## A configuration file you just wrote is rejected
**Symptom.** `nxdns run --config`, `nxdns check --config` or `nxdns import`
prints the validation problem and stops with exit 2:
**Symptom.** `nxdns run --config`, `nxdns check --config` or `nxdns import` prints the validation problem and stops with exit 2:
```
FAIL groups: no group named 'default'; every unknown client is assigned to it
@@ -118,53 +85,32 @@ nxdns run failed: NoUpstreams
run `nxdns check` to see the configuration in full
```
`NoUpstreams` from a file is not the same fault as `NoUsableUpstreams` above:
the first is a file `run` refused, the second is a database `run` accepted and
found empty. Both are exit 2.
`NoUpstreams` from a file is not the same fault as `NoUsableUpstreams` above: the first is a file `run` refused, the second is a database `run` accepted and found empty. Both are exit 2.
**Diagnosis.** Run the same file through `check`, which reports the same
problems and exits 2:
**Diagnosis.** Run the same file through `check`, which reports the same problems and exits 2:
```sh
nxdns check --config /etc/nxdns/config.zon
```
**Fix.** Correct the file the diagnostics name and start again. Nothing was
applied — a file-mode reconcile happens in one transaction that rolls back, and
a failed `import` leaves the database untouched. The exit code does not depend
on which command read the file: all three of these files were run through `run`,
`check` and `import` here, and every one of the nine combinations exited 2 with
the same diagnostic.
**Fix.** Correct the file the diagnostics name and start again. Nothing was applied — a file-mode reconcile happens in one transaction that rolls back, and a failed `import` leaves the database untouched. The exit code does not depend on which command read the file: all three of these files were run through `run`, `check` and `import` here, and every one of the nine combinations exited 2 with the same diagnostic.
Under the shipped systemd unit an exit 2 stops the service rather than
restarting it (`RestartPreventExitStatus=2 64`), so the journal holds the
diagnostics instead of drowning them in a restart loop. `systemctl start nxdns`
once the file is fixed.
Under the shipped systemd unit an exit 2 stops the service rather than restarting it (`RestartPreventExitStatus=2 64`), so the journal holds the diagnostics instead of drowning them in a restart loop. `systemctl start nxdns` once the file is fixed.
Make `nxdns check --config <file>` the precondition in whatever pushes the file.
In file mode every boot reads it, so an unvalidated bad push does not fail at
deploy time — it fails at the next restart, which may be a power cut at 3am.
Make `nxdns check --config <file>` the precondition in whatever pushes the file. In file mode every boot reads it, so an unvalidated bad push does not fail at deploy time — it fails at the next restart, which may be a power cut at 3am.
## `nxdns check` fails on a server that is running fine
**Symptom.** The service is up and answering, but `nxdns check` on the same
machine exits 2 with one long line about a write-ahead log:
**Symptom.** The service is up and answering, but `nxdns check` on the same machine exits 2 with one long line about a write-ahead log:
```
checking database /var/lib/nxdns/config.db
FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
```
Nothing is damaged. `check` opens `config.db` immutable so that it can never
write to it, and an immutable open ignores the write-ahead log. When that log
holds bytes, the newest settings are in it and the main file holds older ones,
so `check` refuses rather than grade stale values.
Nothing is damaged. `check` opens `config.db` immutable so that it can never write to it, and an immutable open ignores the write-ahead log. When that log holds bytes, the newest settings are in it and the main file holds older ones, so `check` refuses rather than grade stale values.
The log holds bytes after a configuration write that has not been checkpointed
yet, which on a running server means someone changed something through the web
interface or the API. A server that has only been answering queries has an empty
`config.db-wal` and `check` reads it normally — so this line comes and goes, and
its absence is not proof that nothing is running.
The log holds bytes after a configuration write that has not been checkpointed yet, which on a running server means someone changed something through the web interface or the API. A server that has only been answering queries has an empty `config.db-wal` and `check` reads it normally — so this line comes and goes, and its absence is not proof that nothing is running.
**Fix.** Check the exported configuration instead of the live file:
@@ -173,9 +119,7 @@ nxdns export --data-dir /var/lib/nxdns --out /tmp/current.zon
nxdns check --config /tmp/current.zon
```
`export` opens the database read/write and does see the log, so it renders the
settings that are actually in force. Stopping the service and checking again
works too: a clean shutdown checkpoints the log away.
`export` opens the database read/write and does see the log, so it renders the settings that are actually in force. Stopping the service and checking again works too: a clean shutdown checkpoints the log away.
> Reproduced here on a scratch data directory rather than `/var/lib/nxdns` —
> that path is the only substitution in the output above. nxdns was started on
@@ -194,8 +138,7 @@ cannot bind udp [::1]:53: AddressInUse
nxdns run failed: AddressInUse
```
A bind conflict is a runtime failure, not a configuration fault, so this is
exit 1 and `nxdns check` will not find it.
A bind conflict is a runtime failure, not a configuration fault, so this is exit 1 and `nxdns check` will not find it.
**Diagnosis.**
@@ -205,11 +148,9 @@ ss -lntp 'sport = :53'
systemctl is-active systemd-resolved
```
On most systemd distributions the holder is `systemd-resolved`, which runs a
stub listener on `127.0.0.53:53` and on some setups binds `0.0.0.0:53`.
On most systemd distributions the holder is `systemd-resolved`, which runs a stub listener on `127.0.0.53:53` and on some setups binds `0.0.0.0:53`.
**Fix.** Turn off the stub listener and keep resolved for the host's own
lookups:
**Fix.** Turn off the stub listener and keep resolved for the host's own lookups:
```sh
mkdir -p /etc/systemd/resolved.conf.d
@@ -217,22 +158,18 @@ printf '[Resolve]\nDNSStubListener=no\n' > /etc/systemd/resolved.conf.d/nxdns.co
systemctl restart systemd-resolved
```
If `/etc/resolv.conf` is a symlink to `/run/systemd/resolve/stub-resolv.conf`,
repoint it at `/run/systemd/resolve/resolv.conf` so the host still resolves.
If `/etc/resolv.conf` is a symlink to `/run/systemd/resolve/stub-resolv.conf`, repoint it at `/run/systemd/resolve/resolv.conf` so the host still resolves.
> Not verified on this host: this needs root, and `systemd-resolved` is
> inactive here with port 53 free, so the conflict could not be reproduced
> against it. The bind failure itself was reproduced by starting a second nxdns
> on a port the first already held, which is the same error path.
Do not fix this by pointing the host's `/etc/resolv.conf` at nxdns when that
host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a
startup cycle, not a fix.
Do not fix this by pointing the host's `/etc/resolv.conf` at nxdns when that host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a startup cycle, not a fix.
## The container restarts in a loop
**Symptom.** `docker compose ps` shows the container restarting, and the log is
the same failure repeated. Docker has no start limit, so this goes on forever.
**Symptom.** `docker compose ps` shows the container restarting, and the log is the same failure repeated. Docker has no start limit, so this goes on forever.
```
FAIL /etc/nxdns/config.zon: not readable
@@ -246,10 +183,7 @@ docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}} restarts={{.Restar
stat -c '%a %u:%g %n' deploy/docker/etc-nxdns/config.zon
```
Exit 2 naming the configuration path means the container could not read the
file the shipped `command:` makes its configuration. The container runs as uid
65532 and `/etc/nxdns` is mounted read-only, so a file at mode 0600 owned by
your own uid is unreadable to it and the container cannot repair it.
Exit 2 naming the configuration path means the container could not read the file the shipped `command:` makes its configuration. The container runs as uid 65532 and `/etc/nxdns` is mounted read-only, so a file at mode 0600 owned by your own uid is unreadable to it and the container cannot repair it.
**Fix.** Either make the file world-readable, when it holds no secret:
@@ -264,17 +198,11 @@ chown 65532:65532 deploy/docker/etc-nxdns/config.zon
chmod 0600 deploy/docker/etc-nxdns/config.zon
```
The 0644 path was verified against an earlier revision of this page, including
the recovery: after the `chmod` the container started and answered queries. The
`chown` needs root and was not run here.
The 0644 path was verified against an earlier revision of this page, including the recovery: after the `chmod` the container started and answered queries. The `chown` needs root and was not run here.
`FAIL /etc/nxdns/config.zon: no such file` instead of `not readable` means there
is no configuration file at all. Create `deploy/docker/etc-nxdns/config.zon` and
bring it up again; see [Install with Docker](install-with-docker.md).
`FAIL /etc/nxdns/config.zon: no such file` instead of `not readable` means there is no configuration file at all. Create `deploy/docker/etc-nxdns/config.zon` and bring it up again; see [Install with Docker](install-with-docker.md).
A container that exits 2 with `NoUsableUpstreams` is in database mode — the
`command:` line naming `--config` was removed — on a volume whose database is
still empty. Load one and bring it back up:
A container that exits 2 with `NoUsableUpstreams` is in database mode — the `command:` line naming `--config` was removed — on a volume whose database is still empty. Load one and bring it back up:
```sh
docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/config.zon
@@ -293,9 +221,7 @@ docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/co
{"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"}
```
This is not a fault. The service runs `nxdns run --config`, which makes that
file the configuration, and configuration writes through the API are refused so
the file and the running server cannot drift apart.
This is not a fault. The service runs `nxdns run --config`, which makes that file the configuration, and configuration writes through the API are refused so the file and the running server cannot drift apart.
**Diagnosis.** The start log names the authority:
@@ -315,19 +241,13 @@ nxdns check --config /etc/nxdns/config.zon
systemctl restart nxdns
```
Or, if you want the interface to be how this box is configured, leave file mode:
drop `--config` from `ExecStart` and restart. The database already holds the
last reconciled state, so nothing is lost. See
[Run in file mode](install-with-systemd.md#run-in-file-mode).
Or, if you want the interface to be how this box is configured, leave file mode: drop `--config` from `ExecStart` and restart. The database already holds the last reconciled state, so nothing is lost. See [Run in file mode](install-with-systemd.md#run-in-file-mode).
Pausing blocking, refreshing blocklists and reloading certificates are not
configuration and keep working in file mode. Deleting a client works too, unless
the file names that client's address.
Pausing blocking, refreshing blocklists and reloading certificates are not configuration and keep working in file mode. Deleting a client works too, unless the file names that client's address.
## The container cannot reach its upstreams
**Symptom.** The container starts, but every query fails and `nxdns check`
inside it reports each upstream as unreachable.
**Symptom.** The container starts, but every query fails and `nxdns check` inside it reports each upstream as unreachable.
**Diagnosis.** Look at what the host resolves with:
@@ -335,15 +255,11 @@ inside it reports each upstream as unreachable.
cat /etc/resolv.conf
```
**Fix.** If it points at the nxdns container, repoint it at a real resolver.
The container resolves its upstream DoH and DoT hostnames through the host's
DNS configuration, so pointing that at nxdns makes nxdns depend on itself to
start. LAN clients point at nxdns; the container's own host does not.
**Fix.** If it points at the nxdns container, repoint it at a real resolver. The container resolves its upstream DoH and DoT hostnames through the host's DNS configuration, so pointing that at nxdns makes nxdns depend on itself to start. LAN clients point at nxdns; the container's own host does not.
## The disk is filling up
**Symptom.** Writes stop but DNS keeps answering. The journal shows the
transition:
**Symptom.** Writes stop but DNS keeps answering. The journal shows the transition:
```
warning(disk_monitor): disk state ok -> critical: 33349095424 bytes free on /var/lib/nxdns
@@ -361,21 +277,11 @@ curl -s http://127.0.0.1:8080/api/health
{"status":"degraded","disk":{"state":"critical","free_bytes":33349079040,"db_bytes":180224,"log_bytes":0,"sample_failures":0},"upstreams":{"available":1,"total":1},"queries_dropped":0,"writer_failed":false,"refreshes_gated":1,"snapshot_generation":2}
```
`/metrics` carries the same free, database and log byte gauges as
`nxdns_disk_free_bytes`, `nxdns_disk_db_bytes` and `nxdns_disk_log_bytes`; the
state itself is on `/api/health`, not in the metrics output.
`/metrics` carries the same free, database and log byte gauges as `nxdns_disk_free_bytes`, `nxdns_disk_db_bytes` and `nxdns_disk_log_bytes`; the state itself is on `/api/health`, not in the metrics output.
**What the state means.** The monitor samples free space and database sizes
once a minute. Below `disk.warn_free_mb` it logs the transition. Below
`disk.min_free_mb` it gates every non-essential write: the query logger holds
its batches, the client tracker stops persisting, and blocklist refreshes are
skipped and counted in `refreshes_gated`. Resolution never degrades because the
disk is full — this was verified by setting the thresholds above the free space
on the volume: the state went critical, a refresh was gated, and queries kept
being answered.
**What the state means.** The monitor samples free space and database sizes once a minute. Below `disk.warn_free_mb` it logs the transition. Below `disk.min_free_mb` it gates every non-essential write: the query logger holds its batches, the client tracker stops persisting, and blocklist refreshes are skipped and counted in `refreshes_gated`. Resolution never degrades because the disk is full — this was verified by setting the thresholds above the free space on the volume: the state went critical, a refresh was gated, and queries kept being answered.
**Fix.** Recover space — lower `logging.retention_days`, or stop the service
and delete `querylog.db` — and writes resume on the next sample.
**Fix.** Recover space — lower `logging.retention_days`, or stop the service and delete `querylog.db` — and writes resume on the next sample.
## Blocklists are not filtering
@@ -387,23 +293,13 @@ and delete `querylog.db` — and writes resume on the next sample.
journalctl -u nxdns | grep 'serving on'
```
It ends in either `blocklist generation N` or
`unfiltered (no blocklist snapshot)`.
It ends in either `blocklist generation N` or `unfiltered (no blocklist snapshot)`.
**Fix.** `unfiltered` means no snapshot loaded at all; the download or compile
warning that explains it is earlier in the same start. nxdns serves anyway on
purpose — a household loses more from DNS that refuses to start than from a
window of unfiltered answers.
**Fix.** `unfiltered` means no snapshot loaded at all; the download or compile warning that explains it is earlier in the same start. nxdns serves anyway on purpose — a household loses more from DNS that refuses to start than from a window of unfiltered answers.
A generation number with nothing being blocked is a different problem: the
snapshot loaded but has no sources in it. The line
`blocklist snapshot generation 1: 0 of 0 sources loaded` says exactly that. Add
a source in the admin interface, or a `blocklist_sources` entry to the
configuration file with a `group_sources` link naming a group.
A generation number with nothing being blocked is a different problem: the snapshot loaded but has no sources in it. The line `blocklist snapshot generation 1: 0 of 0 sources loaded` says exactly that. Add a source in the admin interface, or a `blocklist_sources` entry to the configuration file with a `group_sources` link naming a group.
One name resolving while its neighbours are blocked is a third case, and
`/api/lookup` answers it directly: it reports which level of the filtering
ladder decided, and against what.
One name resolving while its neighbours are blocked is a third case, and `/api/lookup` answers it directly: it reports which level of the filtering ladder decided, and against what.
```sh
curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com'
@@ -413,11 +309,7 @@ curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com'
{"domain":"api.ads.tvb.com","group_id":1,"local_records":false,"forward_zone":null,"blocked":false,"reason":"blocklist_exception","matched":"api.ads.tvb.com","source_url":"https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt","safe_search_rewrite":null}
```
`blocklist_exception` means a downloaded list lifted that name with an `@@`
line, and `source_url` names the list that did it. Nothing is broken, and the
list is not overruling you: an exception cancels only what another list blocks.
Your own rule wins over it. Adding an exact block rule for the same name and
asking again reports `rule_block_exact`, `blocked` true and a null `source_url`.
`blocklist_exception` means a downloaded list lifted that name with an `@@` line, and `source_url` names the list that did it. Nothing is broken, and the list is not overruling you: an exception cancels only what another list blocks. Your own rule wins over it. Adding an exact block rule for the same name and asking again reports `rule_block_exact`, `blocked` true and a null `source_url`.
## A database stamped by a newer binary
@@ -428,6 +320,4 @@ warning(migrations): config.db is at schema version 99; this nxdns binary suppor
nxdns run failed: SchemaTooNew
```
**Fix.** There is no downgrade. Import the export you took before upgrading
into a fresh data directory with the older binary; see
[Upgrade nxdns](upgrade.md).
**Fix.** There is no downgrade. Import the export you took before upgrading into a fresh data directory with the older binary; see [Upgrade nxdns](upgrade.md).
docs/how-to/upgrade.md
+45 -151
View File
@@ -1,11 +1,8 @@
# Upgrade nxdns
Replaces a running nxdns with a newer release without losing its
configuration. The database is migrated in place on the first start of the new
binary.
Replaces a running nxdns with a newer release without losing its configuration. The database is migrated in place on the first start of the new binary.
The normal path is to download the new release, verify it, and swap the binary.
Upgrading a build you made yourself is the last section of this page.
The normal path is to download the new release, verify it, and swap the binary. Upgrading a build you made yourself is the last section of this page.
> Verification: the export, the migration behaviour and the `version`/`check`
> steps below were run on the machine that wrote this page, against a
@@ -23,38 +20,22 @@ Upgrading a build you made yourself is the last section of this page.
## Breaking change: `run --config` now means file authority
**Read this before upgrading if anything on your box passes `--config` to
`nxdns run`** — a systemd drop-in, a wrapper script, or a `command:` in a
compose file.
**Read this before upgrading if anything on your box passes `--config` to `nxdns run`** — a systemd drop-in, a wrapper script, or a `command:` in a compose file.
`run --config FILE` used to mean *seed once*: the file was read only while the
database was still empty, and ignored on every start after that. It now means
*the file is the configuration*: every start reconciles the database onto it.
`run --config FILE` used to mean *seed once*: the file was read only while the database was still empty, and ignored on every start after that. It now means *the file is the configuration*: every start reconciles the database onto it.
For a box that was seeded once and then configured through the admin interface,
the first start after the upgrade converges the database back to that old seed
file. **Every change made through the UI since seeding is deleted.**
For a box that was seeded once and then configured through the admin interface, the first start after the upgrade converges the database back to that old seed file. **Every change made through the UI since seeding is deleted.**
There are two ways out, and you pick before you restart:
- **Keep the database.** Drop the flag. `nxdns run` with no `--config` serves
the database exactly as it did before, and nothing reads a file. This is the
right answer if the UI is how you change things.
- **Adopt file mode cleanly.** Install the new binary, stop the service, export
the current database over the file path, check it, then start with the flag.
The first reconcile is then a no-op, because the file was rendered from the
database it governs. **Install the new binary first** — see the order trap
below. The full procedure is
[Adopt file mode](install-with-systemd.md#adopt-file-mode-on-a-box-that-is-already-running).
- **Keep the database.** Drop the flag. `nxdns run` with no `--config` serves the database exactly as it did before, and nothing reads a file. This is the right answer if the UI is how you change things.
- **Adopt file mode cleanly.** Install the new binary, stop the service, export the current database over the file path, check it, then start with the flag. The first reconcile is then a no-op, because the file was rendered from the database it governs. **Install the new binary first** — see the order trap below. The full procedure is [Adopt file mode](install-with-systemd.md#adopt-file-mode-on-a-box-that-is-already-running).
`nxdns check --config FILE` is unchanged: it graded that file before and it
grades that file now.
`nxdns check --config FILE` is unchanged: it graded that file before and it grades that file now.
### The order trap: export with the new binary, not the old one
Take the export **after** you have replaced the binary, with the service
stopped. Exporting first — the instinctive order, and the one step 1 of this
page tells you to take for a backup — produces a file the new binary refuses.
Take the export **after** you have replaced the binary, with the service stopped. Exporting first — the instinctive order, and the one step 1 of this page tells you to take for a backup — produces a file the new binary refuses.
A 0.0.1 `nxdns export` writes both fields:
@@ -63,9 +44,7 @@ A 0.0.1 `nxdns export` writes both fields:
.password_hash = "$argon2id$v=19$m=19456,t=2,p=1$…",
```
An empty `password_hash` used to mean "unset". It now means "disable
authentication", so it is a *present* value — and a file that carries both
fields states two different things about the password and is refused:
An empty `password_hash` used to mean "unset". It now means "disable authentication", so it is a *present* value — and a file that carries both fields states two different things about the password and is refused:
```
FAIL web.password: password and password_hash are both set; ambiguity in a security setting is refused
@@ -73,27 +52,18 @@ FAIL web.password: password is set to the empty string; omit the field to keep t
nxdns run failed: PasswordAndHashBothSet
```
The old `nxdns check` passes that file, because the old binary agreed with the
old rule. So the failure lands at the first start after the upgrade, with the
resolver stopped and the unit refusing to retry it
(`RestartPreventExitStatus=2 64`). A new `nxdns export` writes
`.password = null` instead and has no such problem.
The old `nxdns check` passes that file, because the old binary agreed with the old rule. So the failure lands at the first start after the upgrade, with the resolver stopped and the unit refusing to retry it (`RestartPreventExitStatus=2 64`). A new `nxdns export` writes `.password = null` instead and has no such problem.
**If you already have an old export you want to adopt**, you do not need to
redo it. Delete the empty-password line and the file is valid:
**If you already have an old export you want to adopt**, you do not need to redo it. Delete the empty-password line and the file is valid:
```sh
sed -i '/^ \.password = "",$/d' /etc/nxdns/config.zon
nxdns check --config /etc/nxdns/config.zon
```
Keep the `.password_hash` line — that is the password, and deleting it as well
would leave the file saying nothing about authentication, which means "keep
whatever is stored" rather than anything you would notice.
Keep the `.password_hash` line — that is the password, and deleting it as well would leave the file saying nothing about authentication, which means "keep whatever is stored" rather than anything you would notice.
The same trap has nothing to do with file mode as such: it is any 0.0.1 export
fed to the new binary, so it also applies to a restore through `nxdns import`.
Backups taken with 0.0.1 need that one line removed before they will load.
The same trap has nothing to do with file mode as such: it is any 0.0.1 export fed to the new binary, so it also applies to a restore through `nxdns import`. Backups taken with 0.0.1 need that one line removed before they will load.
> Verified on this host, with one substitution stated: the repository has no
> 0.0.1 binary to hand, so the old export was **simulated** by taking a current
@@ -107,16 +77,9 @@ Backups taken with 0.0.1 need that one line removed before they will load.
> `git show v0.0.1:src/config/export.zig` line 71 is `cfg.web.password = "";` —
> not from running that binary.
A database-mode install that never passed `--config` needs nothing. Under
Docker, a fresh database-mode install must either take the new compose file or
run `import` once — see
[Database mode in Docker](install-with-docker.md#database-mode-in-docker-instead).
A database-mode install that never passed `--config` needs nothing. Under Docker, a fresh database-mode install must either take the new compose file or run `import` once — see [Database mode in Docker](install-with-docker.md#database-mode-in-docker-instead).
Two smaller renames in the same release: `nxdns import --force` is now
`--allow-delete`, and it is required only when the file's diff would delete
rows rather than whenever the database is non-empty. `nxdns check` no longer
falls back to a default file path when there is no database; it reports the
absent database and names the two ways to get one.
Two smaller renames in the same release: `nxdns import --force` is now `--allow-delete`, and it is required only when the file's diff would delete rows rather than whenever the database is non-empty. `nxdns check` no longer falls back to a default file path when there is no database; it reports the absent database and names the two ways to get one.
There is no schema migration in this change.
@@ -128,16 +91,9 @@ There is no downgrade path, so the export is what you fall back to:
nxdns export --out /some/backup/nxdns-config.zon
```
`/some/backup` is a stand-in for a directory you keep backups in, and the
command relies on the default `--data-dir /var/lib/nxdns` that a systemd
install has.
`/some/backup` is a stand-in for a directory you keep backups in, and the command relies on the default `--data-dir /var/lib/nxdns` that a systemd install has.
This export is a fallback, not a file to deploy. If you are adopting file mode,
take a *second* export after the binary swap and use that one — an export
written by 0.0.1 carries a `.password = ""` line the new binary refuses, as
[the order trap](#the-order-trap-export-with-the-new-binary-not-the-old-one)
explains. The same line has to come out of this backup before the new binary
will import it.
This export is a fallback, not a file to deploy. If you are adopting file mode, take a *second* export after the binary swap and use that one — an export written by 0.0.1 carries a `.password = ""` line the new binary refuses, as [the order trap](#the-order-trap-export-with-the-new-binary-not-the-old-one) explains. The same line has to come out of this backup before the new binary will import it.
> Verified on this host with both paths substituted, since it has neither
> `/var/lib/nxdns` nor `/some/backup`. `SCRATCH` below is a scratch directory,
@@ -156,15 +112,11 @@ will import it.
> The shell umask was 022, so the 0600 is `export` setting it, not the umask.
> Only the two paths differ from the command above.
The file is written atomically at mode 0600 and carries
`web.password_hash`, so treat it as a secret. See
[Back up and restore](back-up-and-restore.md) for the full backup story. The
query log is deliberately not part of it.
The file is written atomically at mode 0600 and carries `web.password_hash`, so treat it as a secret. See [Back up and restore](back-up-and-restore.md) for the full backup story. The query log is deliberately not part of it.
## 2. Download and verify the new release
Read the release notes for the version you are moving to before you take it —
the `CHANGELOG.md` section for that version is the release body.
Read the release notes for the version you are moving to before you take it — the `CHANGELOG.md` section for that version is the release body.
```sh
BASE=https://git.mial.net/mokhtar/nxdns
@@ -179,18 +131,11 @@ sha256sum -c --ignore-missing SHA256SUMS.txt
tar -xzf "nxdns-$VERSION-x86_64-linux-musl.tar.gz"
```
The first line asks the server which release is current, so this block does not
carry a version number that goes stale — Gitea redirects `releases/latest` to
the newest published release's tag page. To move to a particular version rather
than the newest, set `VERSION=<version>` yourself. Check it against what you are
running (`nxdns version`) before you download anything.
The first line asks the server which release is current, so this block does not carry a version number that goes stale — Gitea redirects `releases/latest` to the newest published release's tag page. To move to a particular version rather than the newest, set `VERSION=<version>` yourself. Check it against what you are running (`nxdns version`) before you download anything.
Take `aarch64-linux-musl` for a Raspberry Pi 5. Verify every time, not only on
the first install — an upgrade is a fresh download of a fresh artifact.
[Verify a release](verify-a-release.md) is the full procedure.
Take `aarch64-linux-musl` for a Raspberry Pi 5. Verify every time, not only on the first install — an upgrade is a fresh download of a fresh artifact. [Verify a release](verify-a-release.md) is the full procedure.
Under Docker there is nothing to download: step 3 pulls the image, and the
`IMAGE-DIGEST.txt` asset is what you verify instead.
Under Docker there is nothing to download: step 3 pulls the image, and the `IMAGE-DIGEST.txt` asset is what you verify instead.
> Not verified on this host: no release exists yet, so the `releases/latest`
> lookup returns 404 and leaves `VERSION` empty, and every `curl` below it is a
@@ -201,8 +146,7 @@ Under Docker there is nothing to download: step 3 pulls the image, and the
### systemd
Step 2 leaves the new binary in the extracted directory. Copy the one that
matches the host — `aarch64-linux-musl` for a Raspberry Pi 5:
Step 2 leaves the new binary in the extracted directory. Copy the one that matches the host — `aarch64-linux-musl` for a Raspberry Pi 5:
```sh
scp "nxdns-$VERSION-x86_64-linux-musl/nxdns" target:/tmp/nxdns
@@ -212,9 +156,7 @@ scp "nxdns-$VERSION-x86_64-linux-musl/nxdns" target:/tmp/nxdns
> nxdns, and this host has no such second machine to copy to. There is also no
> release to have extracted.
The tarball also carries `nxdns.service` and `nxdns.conf`. An upgrade does not
normally reinstall them, but compare them against what is on the target when
the release notes say the unit changed.
The tarball also carries `nxdns.service` and `nxdns.conf`. An upgrade does not normally reinstall them, but compare them against what is on the target when the release notes say the unit changed.
Then, as root on the target:
@@ -237,15 +179,9 @@ NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml pull
NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml up -d
```
Compose recreates the container against the same `nxdns-data` volume. What
happens to the file in `etc-nxdns` depends on the `command:` in your compose
file: with the shipped `run --config=/etc/nxdns/config.zon` the file is the
configuration and the restart reconciles onto it; without it, the database in
the volume is the configuration and the file is read by nothing.
Compose recreates the container against the same `nxdns-data` volume. What happens to the file in `etc-nxdns` depends on the `command:` in your compose file: with the shipped `run --config=/etc/nxdns/config.zon` the file is the configuration and the restart reconciles onto it; without it, the database in the volume is the configuration and the file is read by nothing.
Set `NXDNS_VERSION` on both lines, or export it. Without it the compose file
falls back to `:latest`, and `pull` and `up` could then land on different
images if a release happens between them.
Set `NXDNS_VERSION` on both lines, or export it. Without it the compose file falls back to `:latest`, and `pull` and `up` could then land on different images if a release happens between them.
> Not run on this host: `pull` needs a published image, and there is none.
> What was run is `docker compose -f deploy/docker/compose.yaml config`, which
@@ -262,19 +198,13 @@ nxdns check --config /tmp/after-upgrade.zon
dig @127.0.0.1 example.com A +short
```
The restart in step 3 is what migrated the database, so by now the schema is
current and the service is answering. Confirming with `nxdns check` alone would
not work here, and the reason is worth knowing: `check` opens `config.db`
immutable so it can never write to it, and the migration you just performed is
sitting in `config.db-wal` waiting to be checkpointed. Rather than read around
the log and grade older settings, `check` reports it:
The restart in step 3 is what migrated the database, so by now the schema is current and the service is answering. Confirming with `nxdns check` alone would not work here, and the reason is worth knowing: `check` opens `config.db` immutable so it can never write to it, and the migration you just performed is sitting in `config.db-wal` waiting to be checkpointed. Rather than read around the log and grade older settings, `check` reports it:
```
FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
```
`export` opens the database read/write and does see the log, so exporting and
then checking the export validates what is actually in force:
`export` opens the database read/write and does see the log, so exporting and then checking the export validates what is actually in force:
```
checking configuration file /tmp/after-upgrade.zon
@@ -301,20 +231,15 @@ OK: no problems found
## What happens to the database
Migrations run at startup, and also before `export` and `import`, so whichever
of those you run first performs the upgrade. `nxdns check` is the exception: it
opens the database immutable and never migrates, so on a database still one
version behind it reports the mismatch and exits 2 rather than fixing it:
Migrations run at startup, and also before `export` and `import`, so whichever of those you run first performs the upgrade. `nxdns check` is the exception: it opens the database immutable and never migrates, so on a database still one version behind it reports the mismatch and exits 2 rather than fixing it:
```
FAIL /var/lib/nxdns/config.db: schema version 0, this nxdns expects 1; `nxdns run` migrates it, `check` will not
```
That line was reproduced here against a database stamped at version 0; the path
and the version numbers are what vary.
That line was reproduced here against a database stamped at version 0; the path and the version numbers are what vary.
A fresh database is created at the current schema version; an older one is
stepped up to it. The log line names both versions:
A fresh database is created at the current schema version; an older one is stepped up to it. The log line names both versions:
```
info(migrations): config.db migrated from schema version 0 to 1
@@ -326,9 +251,7 @@ info(migrations): config.db migrated from schema version 0 to 1
> rather than nothing. Version 1 is the only schema nxdns has published, so an
> upgrade from a populated older one is not a case that exists yet.
Rolling back is the case that has no answer. A database stamped by a newer
binary refuses to open, so an older binary against an upgraded data directory
fails to start:
Rolling back is the case that has no answer. A database stamped by a newer binary refuses to open, so an older binary against an upgraded data directory fails to start:
```
warning(migrations): config.db is at schema version 99; this nxdns binary supports 1
@@ -340,39 +263,26 @@ nxdns run failed: SchemaTooNew
> hand and `nxdns run` was pointed at it. The two lines above are that run's
> output.
That run exits 1. Recovering means importing the export you took in step 1 into
a fresh data directory with the older binary.
That run exits 1. Recovering means importing the export you took in step 1 into a fresh data directory with the older binary.
### Rolling back from file mode
Putting an older binary back needs no unit edit. The old binary accepts
`run --config` — it just reads it as the old seed-once flag — and against a
database that already holds configuration it ignores the file entirely and
serves the last state the new binary reconciled. So the service comes back up
on the configuration it was running.
Putting an older binary back needs no unit edit. The old binary accepts `run --config` — it just reads it as the old seed-once flag — and against a database that already holds configuration it ignores the file entirely and serves the last state the new binary reconciled. So the service comes back up on the configuration it was running.
The consequence is worth stating plainly: **file edits stop applying.** The old
binary will not re-read the file, so every change made to `config.zon` after the
rollback does nothing at all, silently, until the newer binary is back. If you
have to stay on the old binary, use `nxdns import` to apply file changes, or drop
the flag so the invocation matches what the binary actually does.
The consequence is worth stating plainly: **file edits stop applying.** The old binary will not re-read the file, so every change made to `config.zon` after the rollback does nothing at all, silently, until the newer binary is back. If you have to stay on the old binary, use `nxdns import` to apply file changes, or drop the flag so the invocation matches what the binary actually does.
The schema note above still governs: a database stamped by a newer binary
refuses to open, whatever mode either binary runs in.
The schema note above still governs: a database stamped by a newer binary refuses to open, whatever mode either binary runs in.
## Changing settings, not the binary
How you change a setting depends on which authority the service runs under.
`nxdns run` in `ExecStart` means the database; `nxdns run --config FILE` means
the file. The start log names it either way:
How you change a setting depends on which authority the service runs under. `nxdns run` in `ExecStart` means the database; `nxdns run --config FILE` means the file. The start log names it either way:
```
info(nxdns): authority: database
info(nxdns): authority: file (/etc/nxdns/config.zon)
```
**In file mode**, edit the file, validate it, restart. The admin interface will
refuse the change with a 403 naming the file, so there is nothing to get wrong:
**In file mode**, edit the file, validate it, restart. The admin interface will refuse the change with a 403 naming the file, so there is nothing to get wrong:
```sh
$EDITOR /etc/nxdns/config.zon
@@ -380,8 +290,7 @@ nxdns check --config /etc/nxdns/config.zon
systemctl restart nxdns
```
**In database mode**, change settings through the admin interface, through the
API, or with an export–edit–import cycle against a stopped server:
**In database mode**, change settings through the admin interface, through the API, or with an export–edit–import cycle against a stopped server:
```sh
nxdns export --out config-backup.zon
@@ -391,19 +300,14 @@ nxdns import config-backup.zon
systemctl start nxdns
```
`import` needs no flag to add rows or to edit them. It needs `--allow-delete`
only when applying the file would delete rows the database holds — including the
case where you renamed something, since changing a group's name or an upstream's
URL is a delete and an insert to the engine, not an edit. The refusal names the
tables and rolls back:
`import` needs no flag to add rows or to edit them. It needs `--allow-delete` only when applying the file would delete rows the database holds — including the case where you renamed something, since changing a group's name or an upstream's URL is a delete and an insert to the engine, not an edit. The refusal names the tables and rolls back:
```
FAIL import: this file would delete rows the database holds (upstreams 1); re-run with --allow-delete to apply it
import failed: DestructiveImport
```
Stop the server first either way. `import` rewrites configuration underneath a
process that read it at startup, and a running server picks up only some of it.
Stop the server first either way. `import` rewrites configuration underneath a process that read it at startup, and a running server picks up only some of it.
> Verified on this host against a populated scratch data directory, with
> `--data-dir` pointing at it — that path is the only difference from the blocks
@@ -429,8 +333,7 @@ process that read it at startup, and a running server picks up only some of it.
## Upgrading to a build of your own
If you are running something you built rather than a release, step 2 is a
build instead of a download:
If you are running something you built rather than a release, step 2 is a build instead of a download:
```sh
(cd web && npm ci && npm run build)
@@ -439,18 +342,9 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)"
-Dweb-dist=web/dist -Doptimize=ReleaseSafe
```
Rebuild `web/dist` before the binary on every upgrade. The admin interface is
embedded at build time, and an old bundle against a new API is a broken
settings page. `dist` refuses the `web/dist-placeholder` default outright, so
the only way to ship a stale bundle is to leave an old `web/dist` in place.
Rebuild `web/dist` before the binary on every upgrade. The admin interface is embedded at build time, and an old bundle against a new API is a broken settings page. `dist` refuses the `web/dist-placeholder` default outright, so the only way to ship a stale bundle is to leave an old `web/dist` in place.
The staged payload for each target is under
`zig-out/dist/stage/nxdns-<version>-<triple>/`, and step 3 continues from there
with that path in place of the extracted one. The version string has to equal
`.version` in `build.zig.zon` — `verify-dist` asserts it, so a made-up one
builds and then fails verification. What tells your build apart from the
published release of the same version is `-Dgit-commit`, which `nxdns version`
prints beside the version.
The staged payload for each target is under `zig-out/dist/stage/nxdns-<version>-<triple>/`, and step 3 continues from there with that path in place of the extracted one. The version string has to equal `.version` in `build.zig.zon` — `verify-dist` asserts it, so a made-up one builds and then fails verification. What tells your build apart from the published release of the same version is `-Dgit-commit`, which `nxdns version` prints beside the version.
Under Docker, build the image and name it instead of pulling:
docs/how-to/verify-a-release.md
+44 -149
View File
@@ -1,12 +1,8 @@
# Verify a release
Checks that a downloaded nxdns release is the one the project published and
that it arrived intact. It also gives the recipe for rebuilding the same
version from source, and says plainly what that does and does not settle.
Checks that a downloaded nxdns release is the one the project published and that it arrived intact. It also gives the recipe for rebuilding the same version from source, and says plainly what that does and does not settle.
Do this before you run the binary, not after. The whole point of the checksum
file is that it is signed, so a tampered mirror cannot hand you a matching
tarball and a matching checksum at the same time.
Do this before you run the binary, not after. The whole point of the checksum file is that it is signed, so a tampered mirror cannot hand you a matching tarball and a matching checksum at the same time.
> Verification: every command on this page was run on 2026-08-09 against the
> published `v0.0.1` release, from a clean directory, with a clean `GNUPGHOME`
@@ -17,8 +13,7 @@ tarball and a matching checksum at the same time.
## What a release contains
Five assets, on the release page at
`https://git.mial.net/mokhtar/nxdns/releases`:
Five assets, on the release page at `https://git.mial.net/mokhtar/nxdns/releases`:
| Asset | What it is |
| --- | --- |
@@ -28,28 +23,15 @@ Five assets, on the release page at
| `SHA256SUMS.txt.asc` | A detached OpenPGP signature over `SHA256SUMS.txt` |
| `IMAGE-DIGEST.txt` | The container image reference this version pushed, pinned by digest |
Each tarball holds one top-level directory, `nxdns-<version>-<triple>/`, with
six files in it: the `nxdns` binary at mode 0755, and `nxdns.service`,
`nxdns.conf`, `LICENSE`, `THIRD-PARTY-NOTICES` and `INSTALL.md` at 0644.
Each tarball holds one top-level directory, `nxdns-<version>-<triple>/`, with six files in it: the `nxdns` binary at mode 0755, and `nxdns.service`, `nxdns.conf`, `LICENSE`, `THIRD-PARTY-NOTICES` and `INSTALL.md` at 0644.
`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt` rather than the image, because the
image digest does not exist until the push has happened and cannot be computed
by the build. Signing the file that names the digest gets you the same
guarantee in one signature.
`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt` rather than the image, because the image digest does not exist until the push has happened and cannot be computed by the build. Signing the file that names the digest gets you the same guarantee in one signature.
The `.txt` on three of the five names is not decoration. Gitea decides what an
attachment may be by its file extension, and whether it accepts an
extensionless upload at all is untested against this instance, so the release
uses names it is known to accept. On disk, `zig build dist` still writes a file
called `SHA256SUMS`; the release job copies it to `SHA256SUMS.txt` and appends
the image line before signing.
The `.txt` on three of the five names is not decoration. Gitea decides what an attachment may be by its file extension, and whether it accepts an extensionless upload at all is untested against this instance, so the release uses names it is known to accept. On disk, `zig build dist` still writes a file called `SHA256SUMS`; the release job copies it to `SHA256SUMS.txt` and appends the image line before signing.
## 1. Pick a version
Every URL below takes the version from one shell variable. Ask the server
rather than typing a number that goes stale: Gitea redirects `releases/latest`
to the tag page of the newest published release — newest by publication time,
and drafts and pre-releases are excluded.
Every URL below takes the version from one shell variable. Ask the server rather than typing a number that goes stale: Gitea redirects `releases/latest` to the tag page of the newest published release — newest by publication time, and drafts and pre-releases are excluded.
```sh
BASE=https://git.mial.net/mokhtar/nxdns
@@ -58,8 +40,7 @@ VERSION=$(curl -fsS -o /dev/null -w '%{redirect_url}' "$BASE/releases/latest" |
echo "$VERSION"
```
To take a particular version instead, set it yourself — substitute the one you
want for the placeholder:
To take a particular version instead, set it yourself — substitute the one you want for the placeholder:
```sh
VERSION=<version>
@@ -68,8 +49,7 @@ VERSION=<version>
> Verified: the two-command form, run against this repository, printed `0.0.1`
> with `v0.0.1` published.
Pin the version in anything you script or automate. `latest` is convenient for
a person at a terminal and a liability in a machine that upgrades itself.
Pin the version in anything you script or automate. `latest` is convenient for a person at a terminal and a liability in a machine that upgrades itself.
## 2. Download
@@ -81,8 +61,7 @@ curl -fLO "$BASE/releases/download/v$VERSION/SHA256SUMS.txt.asc"
curl -fLO "$BASE/releases/download/v$VERSION/IMAGE-DIGEST.txt"
```
`-L` is not optional: Gitea answers an asset URL with a 303 to wherever the
attachment is actually stored.
`-L` is not optional: Gitea answers an asset URL with a 303 to wherever the attachment is actually stored.
For the aarch64 tarball, or for both, swap or add the filename:
@@ -90,8 +69,7 @@ For the aarch64 tarball, or for both, swap or add the filename:
curl -fLO "$BASE/releases/download/v$VERSION/nxdns-$VERSION-aarch64-linux-musl.tar.gz"
```
Gitea also accepts the literal word `latest` in place of the tag, so the three
assets whose names carry no version can be fetched without one:
Gitea also accepts the literal word `latest` in place of the tag, so the three assets whose names carry no version can be fetched without one:
```sh
curl -fLO "$BASE/releases/download/latest/SHA256SUMS.txt"
@@ -99,10 +77,7 @@ curl -fLO "$BASE/releases/download/latest/SHA256SUMS.txt.asc"
curl -fLO "$BASE/releases/download/latest/IMAGE-DIGEST.txt"
```
That is the Gitea spelling, and it is not GitHub's. `releases/latest/download/`
— the form GitHub uses — is a 404 on Gitea; the alias goes in the tag
position, as `releases/download/latest/`. The tarball filenames contain the
version, so this alias never saves you from knowing it for those two.
That is the Gitea spelling, and it is not GitHub's. `releases/latest/download/` — the form GitHub uses — is a 404 on Gitea; the alias goes in the tag position, as `releases/download/latest/`. The tarball filenames contain the version, so this alias never saves you from knowing it for those two.
> Verified against `v0.0.1`: all five assets downloaded through the versioned
> path, `SHA256SUMS.txt` downloaded again through the `latest` alias and hashed
@@ -110,9 +85,7 @@ version, so this alias never saves you from knowing it for those two.
## 3. Check the signature over `SHA256SUMS.txt`
Get the public key first. It is a signing subkey of the key that signs every
commit in this repository, so you can confirm the fingerprint against a clone
you already have with `git log --show-signature` or `git verify-tag v$VERSION`:
Get the public key first. It is a signing subkey of the key that signs every commit in this repository, so you can confirm the fingerprint against a clone you already have with `git log --show-signature` or `git verify-tag v$VERSION`:
```
A2061F6AB24DF2C0E92346FD1509B54946D08A95
@@ -143,32 +116,18 @@ Primary key fingerprint: A206 1F6A B24D F2C0 E923 46FD 1509 B549 46D0 8A95
Subkey fingerprint: 019D 00DF 8417 EBFD A547 1E5E F731 9CC0 24FB 5A96
```
The *structure* is what to read: three lines, not one. `using EDDSA key` and
`Subkey fingerprint` name the signing subkey that actually made the signature;
`Primary key fingerprint` names the certificate it hangs off, and that is the
one published above. The subkey fingerprint can change — a signing subkey is
revoked and replaced on its own — but the primary fingerprint is the
project's identity and stays.
The *structure* is what to read: three lines, not one. `using EDDSA key` and `Subkey fingerprint` name the signing subkey that actually made the signature; `Primary key fingerprint` names the certificate it hangs off, and that is the one published above. The subkey fingerprint can change — a signing subkey is revoked and replaced on its own — but the primary fingerprint is the project's identity and stays.
Exit status 0, and `Good signature`. That warning is normal and is not a
failure: it says you have not told GnuPG you believe the key belongs to the
person it claims to.
Exit status 0, and `Good signature`. That warning is normal and is not a failure: it says you have not told GnuPG you believe the key belongs to the person it claims to.
Now compare the `Primary key fingerprint` line with the fingerprint in this
page. GnuPG prints it as ten space-separated groups of four hex digits, with a
double space in the middle, while the fingerprint above is the same 40
characters unspaced — so compare the hex digits in order and ignore the
spacing, or strip it and let the shell do it:
Now compare the `Primary key fingerprint` line with the fingerprint in this page. GnuPG prints it as ten space-separated groups of four hex digits, with a double space in the middle, while the fingerprint above is the same 40 characters unspaced — so compare the hex digits in order and ignore the spacing, or strip it and let the shell do it:
```sh
gpg --verify SHA256SUMS.txt.asc SHA256SUMS.txt 2>&1 |
sed -n 's/^Primary key fingerprint: //p' | tr -d ' '
```
That prints the 40-character form, ready to compare with
`A2061F6AB24DF2C0E92346FD1509B54946D08A95`. Do not skip the comparison —
`gpg --verify` exits 0 for a good signature from *any* key in your keyring,
including one an attacker talked you into importing.
That prints the 40-character form, ready to compare with `A2061F6AB24DF2C0E92346FD1509B54946D08A95`. Do not skip the comparison — `gpg --verify` exits 0 for a good signature from *any* key in your keyring, including one an attacker talked you into importing.
A tampered `SHA256SUMS.txt` looks like this, and exits 1:
@@ -197,9 +156,7 @@ nxdns-<version>-x86_64-linux-musl.tar.gz: OK
IMAGE-DIGEST.txt: OK
```
`--ignore-missing` is what makes this work when you downloaded one tarball out
of the two. Without it, `sha256sum` treats every line it cannot read as a
failure and exits 1:
`--ignore-missing` is what makes this work when you downloaded one tarball out of the two. Without it, `sha256sum` treats every line it cannot read as a failure and exits 1:
```
nxdns-<version>-x86_64-linux-musl.tar.gz: OK
@@ -209,8 +166,7 @@ IMAGE-DIGEST.txt: OK
sha256sum: WARNING: 1 listed file could not be read
```
A file that is present but does not match is the case that matters, and it
says `FAILED` with no `open or read`:
A file that is present but does not match is the case that matters, and it says `FAILED` with no `open or read`:
```
nxdns-<version>-x86_64-linux-musl.tar.gz: FAILED
@@ -218,9 +174,7 @@ IMAGE-DIGEST.txt: OK
sha256sum: WARNING: 1 computed checksum did NOT match
```
Check the signature before the hashes, not after. An attacker who can replace
the tarball can replace `SHA256SUMS.txt` next to it; the signature is the only
thing in the set they cannot forge.
Check the signature before the hashes, not after. An attacker who can replace the tarball can replace `SHA256SUMS.txt` next to it; the signature is the only thing in the set they cannot forge.
> Verified against `v0.0.1`: with both tarballs present, `sha256sum -c` printed
> three `OK` lines. The three transcripts above are the same command over
@@ -234,12 +188,7 @@ thing in the set they cannot forge.
tar -tvzf nxdns-$VERSION-x86_64-linux-musl.tar.gz
```
Expect exactly one top-level directory and the six files listed above, with
mode `-rwxr-xr-x` on `nxdns` and `-rw-r--r--` on the rest, no symlinks, and no
path that begins with `/` or contains `..`. `zig build verify-dist` asserts all
of that on the extracted archive before a release is ever published, so this is
a second opinion rather than the only check — but it costs nothing and it is
the step that catches a tarball that is not the one you think it is.
Expect exactly one top-level directory and the six files listed above, with mode `-rwxr-xr-x` on `nxdns` and `-rw-r--r--` on the rest, no symlinks, and no path that begins with `/` or contains `..`. `zig build verify-dist` asserts all of that on the extracted archive before a release is ever published, so this is a second opinion rather than the only check — but it costs nothing and it is the step that catches a tarball that is not the one you think it is.
Then extract:
@@ -248,9 +197,7 @@ tar -xzf nxdns-$VERSION-x86_64-linux-musl.tar.gz
./nxdns-$VERSION-x86_64-linux-musl/nxdns version
```
`version` prints the version and the git commit it was built from, then the
Zig version. The version has to match the tag you downloaded, and the commit
has to match the commit the tag points at.
`version` prints the version and the git commit it was built from, then the Zig version. The version has to match the tag you downloaded, and the commit has to match the commit the tag points at.
> Verified against `v0.0.1`: both tarballs listed exactly the one directory and
> six files with the stated modes, no symlinks and no absolute or `..` paths,
@@ -260,16 +207,13 @@ has to match the commit the tag points at.
## 6. Verify the container image
`IMAGE-DIGEST.txt` holds one line: the image reference this version pushed,
pinned by the digest of its index, in the form
`IMAGE-DIGEST.txt` holds one line: the image reference this version pushed, pinned by the digest of its index, in the form
```
git.mial.net/mokhtar/nxdns:<version>@sha256:<64 hex digits>
```
`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt`, so the signature you already
checked covers that line too. Confirm the tag in the registry still resolves to
that digest:
`SHA256SUMS.txt` covers `IMAGE-DIGEST.txt`, so the signature you already checked covers that line too. Confirm the tag in the registry still resolves to that digest:
```sh
cut -d@ -f2 IMAGE-DIGEST.txt
@@ -277,8 +221,7 @@ docker buildx imagetools inspect git.mial.net/mokhtar/nxdns:$VERSION \
--format '{{.Manifest.Digest}}'
```
The two have to be the same string. A registry tag is mutable; the digest is
not, so pull the whole pinned reference rather than the tag when you care:
The two have to be the same string. A registry tag is mutable; the digest is not, so pull the whole pinned reference rather than the tag when you care:
```sh
docker pull "$(cat IMAGE-DIGEST.txt)"
@@ -291,12 +234,9 @@ docker buildx imagetools inspect git.mial.net/mokhtar/nxdns:$VERSION \
--format '{{range .Manifest.Manifests}}{{.Platform.OS}}/{{.Platform.Architecture}} {{end}}'
```
`linux/amd64 linux/arm64`. The build passes `--provenance=false --sbom=false`,
so there are no `unknown/unknown` attestation entries in the list; seeing any
means the image did not come from this pipeline.
`linux/amd64 linux/arm64`. The build passes `--provenance=false --sbom=false`, so there are no `unknown/unknown` attestation entries in the list; seeing any means the image did not come from this pipeline.
The binary inside the image is the same file as the one in the matching
tarball, and the release checks that before publishing. To check it yourself:
The binary inside the image is the same file as the one in the matching tarball, and the release checks that before publishing. To check it yourself:
```sh
docker create --name nxdns-verify git.mial.net/mokhtar/nxdns:$VERSION
@@ -317,35 +257,18 @@ sha256sum ./nxdns-from-image ./nxdns-$VERSION-x86_64-linux-musl/nxdns
It proves two things:
- The release was produced by this project's release pipeline, using a key
only that pipeline holds.
- What you have on disk is byte for byte what that pipeline uploaded. A
corrupted download, a modified mirror or a tampered proxy all break the
check.
- The release was produced by this project's release pipeline, using a key only that pipeline holds.
- What you have on disk is byte for byte what that pipeline uploaded. A corrupted download, a modified mirror or a tampered proxy all break the check.
It does not prove that the binary in the tarball was built from the source in
this repository. The machine that ran the build also held the signing key, so
a compromise of that machine produces an artifact that is signed, verifies
cleanly, and contains whatever the attacker put in it. The signature is a
statement about origin and integrity in transit. It is not a statement about
provenance from source.
It does not prove that the binary in the tarball was built from the source in this repository. The machine that ran the build also held the signing key, so a compromise of that machine produces an artifact that is signed, verifies cleanly, and contains whatever the attacker put in it. The signature is a statement about origin and integrity in transit. It is not a statement about provenance from source.
Closing that gap needs a reproducibility gate — an independent build, run
somewhere else, that lands on the same bytes — and this project does not have
one. It is a recorded deferral, not an oversight: see `specs/milestone-14.md`
ruling 12. Until it exists, nothing here claims the build is reproducible,
because nobody has measured whether it is.
Closing that gap needs a reproducibility gate — an independent build, run somewhere else, that lands on the same bytes — and this project does not have one. It is a recorded deferral, not an oversight: see `specs/milestone-14.md` ruling 12. Until it exists, nothing here claims the build is reproducible, because nobody has measured whether it is.
The signing key is a subkey rather than the primary key, which limits the
damage of the case above: a leaked release subkey is revoked on its own and
the identity, the commit signatures and everyone's existing trust in the key
survive.
The signing key is a subkey rather than the primary key, which limits the damage of the case above: a leaked release subkey is revoked on its own and the identity, the commit signatures and everyone's existing trust in the key survive.
## Rebuild it yourself
You can still build the same version from source and compare. That gets you a
binary whose provenance you know, and the comparison is worth making — read
the paragraph after the recipe before you draw a conclusion from it.
You can still build the same version from source and compare. That gets you a binary whose provenance you know, and the comparison is worth making — read the paragraph after the recipe before you draw a conclusion from it.
```sh
git clone https://git.mial.net/mokhtar/nxdns
@@ -358,30 +281,13 @@ zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)"
sha256sum zig-out/dist/nxdns-"$VERSION"-*.tar.gz
```
`git verify-tag` is the check that the tag itself is signed by the key from
step 3, and it is the one part of this section that stands on its own: it ties
the source you just checked out to the same identity that signed the release.
`git verify-tag` is the check that the tag itself is signed by the key from step 3, and it is the one part of this section that stands on its own: it ties the source you just checked out to the same identity that signed the release.
`zig build dist` writes `zig-out/dist/`: the two tarballs, a staging directory
per target under `stage/`, the stripped binaries under `bin/<triple>/`, and a
`SHA256SUMS` covering the two tarballs. The published `SHA256SUMS.txt` is that
file with a third line for `IMAGE-DIGEST.txt` appended by the release job, so
the two tarball lines should match and the local file has no third line to
compare.
`zig build dist` writes `zig-out/dist/`: the two tarballs, a staging directory per target under `stage/`, the stripped binaries under `bin/<triple>/`, and a `SHA256SUMS` covering the two tarballs. The published `SHA256SUMS.txt` is that file with a third line for `IMAGE-DIGEST.txt` appended by the release job, so the two tarball lines should match and the local file has no third line to compare.
Now the caveat, and it is the whole reason this section is last. **A hash that
differs does not mean the release was tampered with.** Nothing in this project
measures whether two builds of the same commit on two different machines
produce the same bytes, and there are several ordinary reasons they would not:
a different Zig patch release, a different Node version, a different path to
the build directory, a different npm lockfile resolution. A hash that matches
is real evidence. A hash that does not match tells you only that something
about the two builds differed, and finding out what is on you.
Now the caveat, and it is the whole reason this section is last. **A hash that differs does not mean the release was tampered with.** Nothing in this project measures whether two builds of the same commit on two different machines produce the same bytes, and there are several ordinary reasons they would not: a different Zig patch release, a different Node version, a different path to the build directory, a different npm lockfile resolution. A hash that matches is real evidence. A hash that does not match tells you only that something about the two builds differed, and finding out what is on you.
If you want the comparison to mean as much as it can, match the toolchain the
release used. The Zig version is the second line of `nxdns version`, and both
it and the Node version are pinned to exact patch releases at the top of
`.gitea/workflows/gates.yml`, which is the workflow the release runs.
If you want the comparison to mean as much as it can, match the toolchain the release used. The Zig version is the second line of `nxdns version`, and both it and the Node version are pinned to exact patch releases at the top of `.gitea/workflows/gates.yml`, which is the workflow the release runs.
> Verified against `v0.0.1`, and the result is the caveat above in action. The
> whole recipe ran from a fresh clone: `git verify-tag v0.0.1` printed
@@ -398,25 +304,14 @@ it and the Node version are pinned to exact patch releases at the top of
Stop and do not run the binary.
- `sha256sum` says `FAILED` but the signature was good — you have a damaged or
substituted download. Delete it and fetch it again over a different network
before assuming anything worse.
- `gpg` says `BAD signature` — `SHA256SUMS.txt` and `SHA256SUMS.txt.asc` do not
belong together. Re-download both from the release page; a stale
`SHA256SUMS.txt.asc` left over from a previous version is the boring
explanation.
- `gpg` says `Can't check signature: No public key` — you have not imported the
key, or you imported a different one.
- The fingerprint does not match the one in step 3 — that is the case to take
seriously. Do not extract the tarball, and do not import more keys trying to
make it pass.
- `sha256sum` says `FAILED` but the signature was good — you have a damaged or substituted download. Delete it and fetch it again over a different network before assuming anything worse.
- `gpg` says `BAD signature` — `SHA256SUMS.txt` and `SHA256SUMS.txt.asc` do not belong together. Re-download both from the release page; a stale `SHA256SUMS.txt.asc` left over from a previous version is the boring explanation.
- `gpg` says `Can't check signature: No public key` — you have not imported the key, or you imported a different one.
- The fingerprint does not match the one in step 3 — that is the case to take seriously. Do not extract the tarball, and do not import more keys trying to make it pass.
## Related
- [Install with systemd](install-with-systemd.md) — where the verified tarball
goes next.
- [Install with systemd](install-with-systemd.md) — where the verified tarball goes next.
- [Install with Docker](install-with-docker.md) — the published image.
- [Upgrade nxdns](upgrade.md) — the same verification, on the way to a newer
version.
- [Performance targets and what the tests prove](../explanation/performance-and-testing.md)
— the other place this project writes down what its checks do not cover.
- [Upgrade nxdns](upgrade.md) — the same verification, on the way to a newer version.
- [Performance targets and what the tests prove](../explanation/performance-and-testing.md) — the other place this project writes down what its checks do not cover.
docs/reference/api.md
+46 -164
View File
@@ -1,159 +1,74 @@
# REST API reference
nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080),
as plain HTTP. TLS termination, where an operator wants it, belongs to a reverse
proxy in front; the session cookie deliberately omits the `Secure` attribute so
the supported plain-HTTP LAN deployment works.
nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080), as plain HTTP. TLS termination, where an operator wants it, belongs to a reverse proxy in front; the session cookie deliberately omits the `Secure` attribute so the supported plain-HTTP LAN deployment works.
The machine-readable contract is `src/web/openapi.yaml`, which the running
server hands out unauthenticated at `GET /api/openapi.yaml`. Request and
response schemas for every operation live there. When this page and the YAML
disagree, the YAML wins.
The machine-readable contract is `src/web/openapi.yaml`, which the running server hands out unauthenticated at `GET /api/openapi.yaml`. Request and response schemas for every operation live there. When this page and the YAML disagree, the YAML wins.
The route table is `src/web/routes.zig`; the [Operations](#operations) table
below carries all 56 of its entries.
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 56 of its entries.
## Conventions
- All request and response bodies are JSON (`application/json`), except
`/metrics` (Prometheus text format), `/api/openapi.yaml` (YAML) and
`/api/queries/live` (`text/event-stream`).
- All request and response bodies are JSON (`application/json`), except `/metrics` (Prometheus text format), `/api/openapi.yaml` (YAML) and `/api/queries/live` (`text/event-stream`).
- Field names are snake_case, matching settings keys and SQL column names.
- Every error response carries the envelope `{"error": "<message>"}`. The
message is operator-facing text; internal detail never reaches the wire — a
500 body is generic and the cause goes to the server log.
- Every error response carries the envelope `{"error": "<message>"}`. The message is operator-facing text; internal detail never reaches the wire — a 500 body is generic and the cause goes to the server log.
- Request bodies are strict: an unknown field is a 400, a body over 1 MiB is a
413.
- A request whose path matches but whose method does not answers 405 with an
`Allow` header. An unknown `/api` path is a JSON 404; unknown non-`/api` paths
fall through to the embedded SPA (`index.html`), so client-side routing works.
- A request whose path matches but whose method does not answers 405 with an `Allow` header. An unknown `/api` path is a JSON 404; unknown non-`/api` paths fall through to the embedded SPA (`index.html`), so client-side routing works.
- Item routes (`{id}`) match a positive integer id only.
- Mutations to groups, blocklists, rules, local records, forward zones, clients
and client prefixes take effect live. Upstreams and `/api/settings` are
restart-required.
- Every route has a policy class — `read`, `config_write` or `runtime_action` —
and in file mode the `config_write` routes are refused. See
[Configuration authority](#configuration-authority).
- Mutations to groups, blocklists, rules, local records, forward zones, clients and client prefixes take effect live. Upstreams and `/api/settings` are restart-required.
- Every route has a policy class — `read`, `config_write` or `runtime_action` — and in file mode the `config_write` routes are refused. See [Configuration authority](#configuration-authority).
## Authentication
Cookie sessions, in memory, no accounts — one operator password. Setting that
password is [set up admin
authentication](../how-to/set-up-admin-authentication.md).
Cookie sessions, in memory, no accounts — one operator password. Setting that password is [set up admin authentication](../how-to/set-up-admin-authentication.md).
- Authentication is on exactly when `web.password_hash` is set. When no password
is set, every route is open and `POST /api/auth/login` answers
`{"authenticated": true, "auth_required": false}` without setting a cookie.
- `POST /api/auth/login` takes `{"password": "..."}`. A correct password answers
200 with a `Set-Cookie` for `nxdns_session` (`HttpOnly; SameSite=Lax; Path=/`,
`Max-Age` = the session TTL). A wrong password is a 401; a stored hash the
server cannot read is a 500, never a 401. Login attempts spend rate-limit
tokens like any other request, and argon2id verification is deliberately slow.
- Every route whose auth policy is `session` answers 401
`{"error": "authentication required"}` without a valid cookie.
- Sessions live `web.session_ttl_hours` (default 24) from login; use does not
extend the lifetime. The table holds 32 sessions; a 33rd login evicts the
least recently used. Nothing is persisted — a server restart logs every
operator out.
- Changing the password through `PUT /api/settings` revokes every live session
immediately; the new password applies without a restart.
- `POST /api/auth/logout` ends the cookie's session and clears the cookie. The
route is `.session` like any other, so with a password configured the router
answers 401 before the handler runs when the cookie is missing, expired,
revoked or already logged out; only a live session gets the 200. With no
password configured every session route is open and logout answers 200.
- Authentication is on exactly when `web.password_hash` is set. When no password is set, every route is open and `POST /api/auth/login` answers `{"authenticated": true, "auth_required": false}` without setting a cookie.
- `POST /api/auth/login` takes `{"password": "..."}`. A correct password answers 200 with a `Set-Cookie` for `nxdns_session` (`HttpOnly; SameSite=Lax; Path=/`, `Max-Age` = the session TTL). A wrong password is a 401; a stored hash the server cannot read is a 500, never a 401. Login attempts spend rate-limit tokens like any other request, and argon2id verification is deliberately slow.
- Every route whose auth policy is `session` answers 401 `{"error": "authentication required"}` without a valid cookie.
- Sessions live `web.session_ttl_hours` (default 24) from login; use does not extend the lifetime. The table holds 32 sessions; a 33rd login evicts the least recently used. Nothing is persisted — a server restart logs every operator out.
- Changing the password through `PUT /api/settings` revokes every live session immediately; the new password applies without a restart.
- `POST /api/auth/logout` ends the cookie's session and clears the cookie. The route is `.session` like any other, so with a password configured the router answers 401 before the handler runs when the cookie is missing, expired, revoked or already logged out; only a live session gets the 200. With no password configured every session route is open and logout answers 200.
## Rate limiting
A token bucket per client address: capacity and refill are both
`web.api_rate_limit_per_min` (default 300) per minute, so a page-load burst up
to the capacity is admitted and the long-run rate holds.
A token bucket per client address: capacity and refill are both `web.api_rate_limit_per_min` (default 300) per minute, so a page-load burst up to the capacity is admitted and the long-run rate holds.
- An over-budget request answers 429 `{"error": "rate limited"}` with a
`Retry-After` header giving the seconds until a token is available (rounded
up, never zero).
- Loopback addresses (127.0.0.0/8 and ::1) are exempt while
`web.api_localhost_exempt` is true (the default).
- The address a bucket keys on is the socket peer, unless that peer is listed in
`web.trusted_proxies`. For a listed peer the address is instead the **last**
entry of the request's `X-Forwarded-For` — the entry the proxy appended, which
is the only one a client cannot write. A request from a trusted proxy with no
such header keys on the proxy itself; one whose last entry is not an IP
literal is answered 400, because the alternative is granting the proxy's own
loopback exemption to whoever sent it. Only `X-Forwarded-For` is read;
`Forwarded` (RFC 7239) and the PROXY protocol are not.
- Without `web.trusted_proxies`, a same-box reverse proxy makes every request
loopback, so the default exemption disables the limiter for all remote
clients. Set the proxy's address there, or set
`web.api_localhost_exempt = false`.
- Exempt routes, which never consult a bucket: `/metrics` and `/api/health` (a
Prometheus scrape must never see 429) and `/api/queries/live` (one long-lived
stream must not drain its address's bucket; it is bounded by the SSE
connection cap instead).
- The limiter tracks at most 4096 addresses. When the table is full and no slot
is reclaimable, requests from unknown addresses are refused with 429.
- An over-budget request answers 429 `{"error": "rate limited"}` with a `Retry-After` header giving the seconds until a token is available (rounded up, never zero).
- Loopback addresses (127.0.0.0/8 and ::1) are exempt while `web.api_localhost_exempt` is true (the default).
- The address a bucket keys on is the socket peer, unless that peer is listed in `web.trusted_proxies`. For a listed peer the address is instead the **last** entry of the request's `X-Forwarded-For` — the entry the proxy appended, which is the only one a client cannot write. A request from a trusted proxy with no such header keys on the proxy itself; one whose last entry is not an IP literal is answered 400, because the alternative is granting the proxy's own loopback exemption to whoever sent it. Only `X-Forwarded-For` is read; `Forwarded` (RFC 7239) and the PROXY protocol are not.
- Without `web.trusted_proxies`, a same-box reverse proxy makes every request loopback, so the default exemption disables the limiter for all remote clients. Set the proxy's address there, or set `web.api_localhost_exempt = false`.
- Exempt routes, which never consult a bucket: `/metrics` and `/api/health` (a Prometheus scrape must never see 429) and `/api/queries/live` (one long-lived stream must not drain its address's bucket; it is bounded by the SSE connection cap instead).
- The limiter tracks at most 4096 addresses. When the table is full and no slot is reclaimable, requests from unknown addresses are refused with 429.
## Live query stream (SSE)
`GET /api/queries/live` is server-sent events over chunked transfer,
`Content-Type: text/event-stream`, `Cache-Control: no-store`.
`GET /api/queries/live` is server-sent events over chunked transfer, `Content-Type: text/event-stream`, `Cache-Control: no-store`.
- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on
its own after a drop.
- Each query is one frame: `event: query` and a single `data:` line of JSON. The
payload carries the `GET /api/queries` row fields minus `id` (a live entry
precedes persistence): `ts`, `domain`, `client_ip`, `qtype`, `blocked`,
`block_reason`, `response_time_us`, `cache_hit`, `upstream`.
- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping
middleboxes from reaping the idle connection.
- Each subscriber buffers up to 64 entries. A client too slow for the query rate
overflows its buffer and the server ends the stream cleanly after delivering
what the buffer held — queries are never held back for a slow reader. There is
no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the
missed rows.
- Connections per client address are capped at `web.sse_max_connections_per_ip`
(default 3); over the cap is a 429. The cap binds loopback too. The server
holds at most 32 concurrent streams in total; when all slots are taken, the
answer is a 503.
- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on its own after a drop.
- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload carries the `GET /api/queries` row fields minus `id` (a live entry precedes persistence): `ts`, `domain`, `client_ip`, `qtype`, `blocked`, `block_reason`, `response_time_us`, `cache_hit`, `upstream`.
- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping middleboxes from reaping the idle connection.
- Each subscriber buffers up to 64 entries. A client too slow for the query rate overflows its buffer and the server ends the stream cleanly after delivering what the buffer held — queries are never held back for a slow reader. There is no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the missed rows.
- Connections per client address are capped at `web.sse_max_connections_per_ip` (default 3); over the cap is a 429. The cap binds loopback too. The server holds at most 32 concurrent streams in total; when all slots are taken, the answer is a 503.
## Configuration authority
Which authority is live decides whether the API may write configuration. Under
`nxdns run` the database is authority and every route behaves as it always has.
Under `nxdns run --config FILE` the file is authority, and the routes that would
edit configuration are refused: the file is the only place configuration
changes, and a restart is what applies them.
Which authority is live decides whether the API may write configuration. Under `nxdns run` the database is authority and every route behaves as it always has. Under `nxdns run --config FILE` the file is authority, and the routes that would edit configuration are refused: the file is the only place configuration changes, and a restart is what applies them.
### The refusal
A `config write` route in file mode answers **403** with the ordinary error
envelope:
A `config write` route in file mode answers **403** with the ordinary error envelope:
```json
{"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"}
```
There is no `code` field and no richer body. 403 is used for nothing else in
this API, so the status alone is the machine-readable part, and a client that
wants to know the mode in advance reads it from `GET /api/settings` rather than
probing for errors.
There is no `code` field and no richer body. 403 is used for nothing else in this API, so the status alone is the machine-readable part, and a client that wants to know the mode in advance reads it from `GET /api/settings` rather than probing for errors.
**401 comes first.** The router matches the path, spends a rate-limit token,
checks the session, and only then checks the policy. So an unauthenticated
request to a `config write` route in file mode is a 401, not a 403 — answering
403 first would tell an anonymous caller which routes exist.
**401 comes first.** The router matches the path, spends a rate-limit token, checks the session, and only then checks the policy. So an unauthenticated request to a `config write` route in file mode is a 401, not a 403 — answering 403 first would tell an anonymous caller which routes exist.
`runtime action` and `read` routes are unaffected in both modes. Pausing
blocking, refreshing blocklists, reloading certificates and logging in are
operations on a running process, not statements about configuration, so a
file-mode box still does all of them.
`runtime action` and `read` routes are unaffected in both modes. Pausing blocking, refreshing blocklists, reloading certificates and logging in are operations on a running process, not statements about configuration, so a file-mode box still does all of them.
`DELETE /api/clients/{id}` is the one route whose answer depends on the row.
Deleting a client the file does not declare is a runtime action and succeeds:
without it, a mis-identified or departed device would be immortal in file mode,
since the file can add addresses but never remove one it has never named.
Deleting a client the file *does* declare contradicts the file, and answers the
same 403.
`DELETE /api/clients/{id}` is the one route whose answer depends on the row. Deleting a client the file does not declare is a runtime action and succeeds: without it, a mis-identified or departed device would be immortal in file mode, since the file can add addresses but never remove one it has never named. Deleting a client the file *does* declare contradicts the file, and answers the same 403.
### Discovering the authority
@@ -165,32 +80,20 @@ same 403.
| `path` | The managed file's path, or `null` in database mode. |
| `reconciled_at` | Unix seconds when this process loaded the file, or `null` in database mode. |
All three keys are always present; the two nullable ones carry `null` rather
than being omitted, so a client can read `authority.mode` without probing.
All three keys are always present; the two nullable ones carry `null` rather than being omitted, so a client can read `authority.mode` without probing.
```json
{"mode": "database", "path": null, "reconciled_at": null}
{"mode": "managed_file", "path": "/etc/nxdns/config.zon", "reconciled_at": 1786474016}
```
The route requires a session, which is why the filesystem path is here rather
than on the open `/api/version` and `/api/health`.
The route requires a session, which is why the filesystem path is here rather than on the open `/api/version` and `/api/health`.
`reconciled_at` answers exactly one question: **when did this process last read
the file?** Compare it against the file's mtime to spot a restart that has not
happened yet. It is a hint and not a verdict, in both directions — a clock that
stepped, or a copy that preserved mtimes (`git checkout`, `rsync -a`), can make
a newer file look older, and the database can change without either timestamp
moving. It does not tell you whether the file and the running configuration
agree; answering that would take content hashing, which nxdns deliberately does
not do.
`reconciled_at` answers exactly one question: **when did this process last read the file?** Compare it against the file's mtime to spot a restart that has not happened yet. It is a hint and not a verdict, in both directions — a clock that stepped, or a copy that preserved mtimes (`git checkout`, `rsync -a`), can make a newer file look older, and the database can change without either timestamp moving. It does not tell you whether the file and the running configuration agree; answering that would take content hashing, which nxdns deliberately does not do.
## Operations
Auth `open` means no session is required; `session` means a valid session cookie
is required whenever a password is set. Rate limit `counted` spends a token;
`exempt` never consults the limiter. Policy `config write` is the class refused
in file mode; `read` and `runtime action` are always served.
Auth `open` means no session is required; `session` means a valid session cookie is required whenever a password is set. Rate limit `counted` spends a token; `exempt` never consults the limiter. Policy `config write` is the class refused in file mode; `read` and `runtime action` are always served.
| Method | Path | Auth | Rate limit | Policy | Purpose |
|---|---|---|---|---|---|
@@ -251,40 +154,23 @@ in file mode; `read` and `runtime action` are always served.
| PUT | `/api/settings` | session | counted | config write | Update settings |
| POST | `/api/certs/reload` | session | counted | runtime action | Reload the TLS certificates from disk |
There is no `POST /api/clients`: client rows come from DNS activity or import,
never from the API.
There is no `POST /api/clients`: client rows come from DNS activity or import, never from the API.
Static assets are not routes. The router sends unmatched non-`/api` paths to the
embedded SPA before any auth or rate-limit check.
Static assets are not routes. The router sends unmatched non-`/api` paths to the embedded SPA before any auth or rate-limit check.
## Settings keys
`GET /api/settings` and `PUT /api/settings` speak the `section.field` keys of
[the configuration reference](configuration.md), with the values in their
database spelling — notably `logging.level` is `"error"`, not `"err"`.
Two keys behave differently over the API than in the file: `web.password` is
write-only (accepted on a `PUT`, never returned, hashed before storage), and
`web.password_hash` is neither readable nor directly writable, because a client
that could install a hash could install one whose password it already knows.
`GET /api/settings` and `PUT /api/settings` speak the `section.field` keys of [the configuration reference](configuration.md), with the values in their database spelling — notably `logging.level` is `"error"`, not `"err"`. Two keys behave differently over the API than in the file: `web.password` is write-only (accepted on a `PUT`, never returned, hashed before storage), and `web.password_hash` is neither readable nor directly writable, because a client that could install a hash could install one whose password it already knows.
In file mode `PUT /api/settings` is refused with the 403 above, password changes
included. The password then lives where the rest of the configuration lives: set
`web.password` in the file and restart. See
[Password and hash](configuration.md#password-and-hash).
In file mode `PUT /api/settings` is refused with the 403 above, password changes included. The password then lives where the rest of the configuration lives: set `web.password` in the file and restart. See [Password and hash](configuration.md#password-and-hash).
## Schemas
Request and response schemas for every operation live in the OpenAPI document:
`src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a
running server.
Request and response schemas for every operation live in the OpenAPI document: `src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a running server.
### Block reasons
Three places carry the same tag: `block_reason` on a `GET /api/queries` row,
`block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup`
answer. The tag names the level that decided the query, and the levels are
listed here in the order they are consulted — the first one that matches wins,
so a rule always outranks a list.
Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup` answer. The tag names the level that decided the query, and the levels are listed here in the order they are consulted — the first one that matches wins, so a rule always outranks a list.
| Tag | Decided by |
| --- | --- |
@@ -298,10 +184,6 @@ so a rule always outranks a list.
| `blocklist_domain` | A plain name in a downloaded list |
| `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list |
`/api/lookup` also answers `none` when nothing matched. A query row never
carries `none`: `block_reason` is null unless the query was blocked.
`/api/lookup` also answers `none` when nothing matched. A query row never carries `none`: `block_reason` is null unless the query was blocked.
A `cname:` prefix means the decision landed on a CNAME target rather than on
the name the client asked for, so `cname:blocklist_domain` reads as "the list
blocks a name this answer redirects to". Only `/api/queries` and the live
stream show the prefix; `/api/lookup` does not follow CNAMEs.
A `cname:` prefix means the decision landed on a CNAME target rather than on the name the client asked for, so `cname:blocklist_domain` reads as "the list blocks a name this answer redirects to". Only `/api/queries` and the live stream show the prefix; `/api/lookup` does not follow CNAMEs.
docs/reference/cli.md
+48 -184
View File
@@ -4,15 +4,9 @@
nxdns <command> [options]
```
Six subcommands: `run`, `check`, `export`, `import`, `version`, `help`. Source
of truth: `src/cli.zig`.
Six subcommands: `run`, `check`, `export`, `import`, `version`, `help`. Source of truth: `src/cli.zig`.
Every flag takes both spellings, `--flag value` and `--flag=value`. An attached
value that is empty (`--config=`) is a missing value, not an empty path.
`--allow-delete` is boolean and takes no value at all, so `--allow-delete=1` is
not a spelling of any flag this program has. A flag is rejected by the
subcommand that has no use for it: `--web-dev` outside `run` is an unknown flag,
not a no-op.
Every flag takes both spellings, `--flag value` and `--flag=value`. An attached value that is empty (`--config=`) is a missing value, not an empty path. `--allow-delete` is boolean and takes no value at all, so `--allow-delete=1` is not a spelling of any flag this program has. A flag is rejected by the subcommand that has no use for it: `--web-dev` outside `run` is an unknown flag, not a no-op.
## `run`
@@ -26,27 +20,21 @@ Serves DNS until SIGINT or SIGTERM.
### Which authority the invocation selects
The presence of `--config` picks the authority, and nothing else does. There is
no default path, no probe of `/etc/nxdns`, and nothing recorded in the database:
a configuration file sitting at `/etc/nxdns/config.zon` that no flag names
changes nothing at all.
The presence of `--config` picks the authority, and nothing else does. There is no default path, no probe of `/etc/nxdns`, and nothing recorded in the database: a configuration file sitting at `/etc/nxdns/config.zon` that no flag names changes nothing at all.
| Invocation | Authority | What a start does |
| --- | --- | --- |
| `nxdns run` | The database | Serves what `config.db` holds. Nothing reads a file. |
| `nxdns run --config FILE` | FILE | Reads and validates FILE, reconciles the database onto it, then serves. |
The first log line after the migrations names the mode, so a journal says which
authority was live:
The first log line after the migrations names the mode, so a journal says which authority was live:
```
info(nxdns): authority: database
info(nxdns): authority: file (/etc/nxdns/config.zon)
```
In file mode the reconcile prints what it changed before that line — per-table
inserted (`+`), updated (`~`) and deleted (`-`) counts, the settings keys whose
values changed, and any change to whether the admin password is set:
In file mode the reconcile prints what it changed before that line — per-table inserted (`+`), updated (`~`) and deleted (`-`) counts, the settings keys whose values changed, and any change to whether the admin password is set:
```
reconciled '/etc/nxdns/config.zon': upstreams +1 ~0 -0; settings +45 ~0 -0;
@@ -60,17 +48,11 @@ A start whose file matches the database writes nothing and says so:
reconciled '/etc/nxdns/config.zon': no changes
```
Blocklist state is not declarative and survives every reconcile: a source whose
URL the file still names keeps its row id, its checksum, its counters and its
compiled `<id>.list`, `<id>.wild` and `<id>.allow`, so a restart in file mode
downloads nothing. Editing a source's URL is a new identity — a new row, a new id, and a
fresh download.
Blocklist state is not declarative and survives every reconcile: a source whose URL the file still names keeps its row id, its checksum, its counters and its compiled `<id>.list`, `<id>.wild` and `<id>.allow`, so a restart in file mode downloads nothing. Editing a source's URL is a new identity — a new row, a new id, and a fresh download.
### Failing to start in file mode
File mode fails closed. A file that is missing, unreadable, unparseable,
oversized or invalid stops the start; nxdns never falls back to the database,
because a fallback turns a deploy typo into a silently stale configuration.
File mode fails closed. A file that is missing, unreadable, unparseable, oversized or invalid stops the start; nxdns never falls back to the database, because a fallback turns a deploy typo into a silently stale configuration.
```
FAIL /etc/nxdns/config.zon: no such file
@@ -78,46 +60,21 @@ nxdns run failed: ManagedConfigUnreadable
run `nxdns check` to see the configuration in full
```
That is exit 2, and `check --config` on the same path agrees. Only path-class
open failures map that way — the file is not there, or the process may not read
it. An open that fails for a reason a retry could clear, such as
file-descriptor exhaustion or an I/O error, is exit 1: the box is wrong, not the
configuration. See [exit codes](#exit-codes).
That is exit 2, and `check --config` on the same path agrees. Only path-class open failures map that way — the file is not there, or the process may not read it. An open that fails for a reason a retry could clear, such as file-descriptor exhaustion or an I/O error, is exit 1: the box is wrong, not the configuration. See [exit codes](#exit-codes).
The exit-2 agreement covers `run --config` and `check --config`, which read the
managed file through one shared helper. It does **not** extend to `import`'s
positional argument: a missing file there is `import failed: FileNotFound`, exit
1. That is deliberate rather than an oversight — the managed file is a
declarative input an operator deploys, so its absence is a fact about the
configuration, while `import`'s argument is a path typed at a prompt, and a
mistyped path is a failed command rather than a verdict on anything.
The exit-2 agreement covers `run --config` and `check --config`, which read the managed file through one shared helper. It does **not** extend to `import`'s positional argument: a missing file there is `import failed: FileNotFound`, exit
1. That is deliberate rather than an oversight — the managed file is a declarative input an operator deploys, so its absence is a fact about the configuration, while `import`'s argument is a path typed at a prompt, and a mistyped path is a failed command rather than a verdict on anything.
Run `nxdns check --config FILE` before restarting anything that deploys a file.
It grades every declarative fault `run` would hit — read, parse, size,
validation — through the same code, which is what makes it a usable precondition
in an Ansible handler.
Run `nxdns check --config FILE` before restarting anything that deploys a file. It grades every declarative fault `run` would hit — read, parse, size, validation — through the same code, which is what makes it a usable precondition in an Ansible handler.
## `check`
Validates the configuration and probes the upstreams. Exit 0 when it found no
failures, 2 when it found one. It always reports every problem, not just the
first. What it checks, in order:
Validates the configuration and probes the upstreams. Exit 0 when it found no failures, 2 when it found one. It always reports every problem, not just the first. What it checks, in order:
1. Which source to check (see [source selection](#source-selection)).
2. Full validation — the same rules `import` enforces.
3. For each enabled DoH/DoT listener: both PEM files are read and the key is
tested against the certificate, through the same `CertStore.init` the
listeners boot with. A file that is missing, unreadable, too large or
unparseable, and a key that does not belong to the certificate, are all FAIL.
The key's permissions are a separate finding: WARN when any group or other
bit is set, which does not change the exit code.
4. A live probe: one real A query for `example.com` through every enabled
upstream, driving the same pool and failover machinery the server uses, with
the same two deadlines: `upstream.attempt_timeout_ms` bounds one try against
one upstream, `upstream.total_timeout_ms` the whole probe. A FAIL line names
the upstream and the concrete cause recorded in its health. This probe leaves
the machine, so `check` needs network access to pass. It runs from the
command line but not from unit tests.
3. For each enabled DoH/DoT listener: both PEM files are read and the key is tested against the certificate, through the same `CertStore.init` the listeners boot with. A file that is missing, unreadable, too large or unparseable, and a key that does not belong to the certificate, are all FAIL. The key's permissions are a separate finding: WARN when any group or other bit is set, which does not change the exit code.
4. A live probe: one real A query for `example.com` through every enabled upstream, driving the same pool and failover machinery the server uses, with the same two deadlines: `upstream.attempt_timeout_ms` bounds one try against one upstream, `upstream.total_timeout_ms` the whole probe. A FAIL line names the upstream and the concrete cause recorded in its health. This probe leaves the machine, so `check` needs network access to pass. It runs from the command line but not from unit tests.
| Flag | Meaning |
| --- | --- |
@@ -126,10 +83,7 @@ first. What it checks, in order:
### Failures and warnings
Every finding carries a severity. `FAIL` is a problem that sets exit 2. `WARN`
is legal configuration that is almost certainly not what was meant — a blocklist
source no group links to, a TLS key readable beyond its owner — and never
changes an exit code, because the service starts either way.
Every finding carries a severity. `FAIL` is a problem that sets exit 2. `WARN` is legal configuration that is almost certainly not what was meant — a blocklist source no group links to, a TLS key readable beyond its owner — and never changes an exit code, because the service starts either way.
The last line is a summary, and it never contradicts the lines above it:
@@ -141,91 +95,55 @@ The last line is a summary, and it never contradicts the lines above it:
### Source selection
The flag decides, exactly as it does for `run`. There is no fallback and no
probing of a default path:
The flag decides, exactly as it does for `run`. There is no fallback and no probing of a default path:
- `--config FILE`: grade that file, and never open the database.
- No `--config`: grade `<data-dir>/config.db`. It is opened immutable, so
`check` writes nothing to it; see
[what `check` does not do](#what-check-does-not-do).
- No `--config`: grade `<data-dir>/config.db`. It is opened immutable, so `check` writes nothing to it; see [what `check` does not do](#what-check-does-not-do).
So `nxdns check` and `nxdns check --config FILE` grade what the matching `run`
invocation would serve. That is what makes `check` a pre-restart gate rather
than an approximation of one.
So `nxdns check` and `nxdns check --config FILE` grade what the matching `run` invocation would serve. That is what makes `check` a pre-restart gate rather than an approximation of one.
A bare `check` on a box with no database says so and names both ways out of the
state, exit 2:
A bare `check` on a box with no database says so and names both ways out of the state, exit 2:
```
no config database at /var/lib/nxdns/config.db
load one with `nxdns import <file>`, or make a file the source of truth with `nxdns run --config <file>`
```
The first line of output otherwise always names which source was checked. A file
larger than 4 MiB fails with `larger than 4194304 bytes`; a ZON syntax error is
reported with its line and column.
The first line of output otherwise always names which source was checked. A file larger than 4 MiB fails with `larger than 4194304 bytes`; a ZON syntax error is reported with its line and column.
A named file that is missing or unreadable is a finding like any other, not an
I/O failure that escapes the run: `FAIL <path>: no such file` or
`FAIL <path>: not readable`, exit 2.
A named file that is missing or unreadable is a finding like any other, not an I/O failure that escapes the run: `FAIL <path>: no such file` or `FAIL <path>: not readable`, exit 2.
What `check --config` cannot see is the reconcile itself. It needs no database,
so faults that only a write can produce — a disk that is full, a lock held by a
restart that started first — are invisible to it. Those are runtime failures,
exit 1, and they are not verdicts on the file.
What `check --config` cannot see is the reconcile itself. It needs no database, so faults that only a write can produce — a disk that is full, a lock held by a restart that started first — are invisible to it. Those are runtime failures, exit 1, and they are not verdicts on the file.
### What `check` does not do
`check` never writes to `config.db`. It opens the file immutable, which means
SQLite refuses every statement that would write and builds no write-ahead log,
so no `config.db-wal` and no `config.db-shm` appear beside it. It does not chmod
the file and it does not migrate the schema.
`check` never writes to `config.db`. It opens the file immutable, which means SQLite refuses every statement that would write and builds no write-ahead log, so no `config.db-wal` and no `config.db-shm` appear beside it. It does not chmod the file and it does not migrate the schema.
Two consequences are worth knowing before you read a FAIL line as damage:
- A database behind this binary's schema is reported, not upgraded. Start the
service to migrate it.
- A database behind this binary's schema is reported, not upgraded. Start the service to migrate it.
```
FAIL <db>: schema version <n>, this nxdns expects <m>; `nxdns run` migrates it, `check` will not
```
- A database with an unapplied write-ahead log cannot be graded without writing,
because the newest settings are in the log and the main file holds older ones.
`check` says so rather than reading the stale values:
- A database with an unapplied write-ahead log cannot be graded without writing, because the newest settings are in the log and the main file holds older ones. `check` says so rather than reading the stale values:
```
FAIL <db>: uncheckpointed changes are waiting in <db>-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
```
A running nxdns is the usual holder of that log, so this is a common answer
when checking a live server rather than a sign of damage. It depends on what
is in the log, not on whether a server is up: a configuration write that has
not been checkpointed puts bytes there, and a server that has only been
answering queries leaves `config.db-wal` empty and reads normally. Check an
`export` with `--config`, or stop the service first.
A running nxdns is the usual holder of that log, so this is a common answer when checking a live server rather than a sign of damage. It depends on what is in the log, not on whether a server is up: a configuration write that has not been checkpointed puts bytes there, and a server that has only been answering queries leaves `config.db-wal` empty and reads normally. Check an `export` with `--config`, or stop the service first.
## `export`
Writes the configuration as ZON to stdout, or atomically at mode 0600 to
`--out FILE`. `--out` paths are relative to the shell's working directory, not
to the data directory. The output is canonical: every default emitted,
deterministic ordering, no timestamps, so `export` → `import` → `export` is
byte-identical.
Writes the configuration as ZON to stdout, or atomically at mode 0600 to `--out FILE`. `--out` paths are relative to the shell's working directory, not to the data directory. The output is canonical: every default emitted, deterministic ordering, no timestamps, so `export` → `import` → `export` is byte-identical.
`export` does not create the data directory; it fails if the directory is not
there. A directory that exists without a `config.db` is not that case: the
database is opened with create semantics, so an empty `config.db` is created and
migrated, and the export is of a default configuration.
`export` does not create the data directory; it fails if the directory is not there. A directory that exists without a `config.db` is not that case: the database is opened with create semantics, so an empty `config.db` is created and migrated, and the export is of a default configuration.
The output is the same in both authority modes — nothing marks a file as
exported from a file-mode box. That is what lets `export` be the adoption tool:
the file you check is the file you deploy.
The output is the same in both authority modes — nothing marks a file as exported from a file-mode box. That is what lets `export` be the adoption tool: the file you check is the file you deploy.
`web.password` is always written as `null` and `web.password_hash` carries the
stored value, so an export re-imports without anyone knowing the password. A
`null` password is not the same statement as an empty one: see
[`web.password` and `web.password_hash`](configuration.md#password-and-hash).
`web.password` is always written as `null` and `web.password_hash` carries the stored value, so an export re-imports without anyone knowing the password. A `null` password is not the same statement as an empty one: see [`web.password` and `web.password_hash`](configuration.md#password-and-hash).
| Flag | Meaning |
| --- | --- |
@@ -236,10 +154,7 @@ See [back up and restore](../how-to/back-up-and-restore.md).
## `import FILE`
Converges the database onto FILE in one transaction — the same reconcile a
file-mode `run` performs, done once from the command line. Prints every
validation problem; a failed import leaves the database untouched. Creates the
data directory at mode 0700 if it is missing.
Converges the database onto FILE in one transaction — the same reconcile a file-mode `run` performs, done once from the command line. Prints every validation problem; a failed import leaves the database untouched. Creates the data directory at mode 0700 if it is missing.
`FILE` is positional and may appear before or after the flags.
@@ -248,29 +163,20 @@ data directory at mode 0700 if it is missing.
| `--data-dir DIR` | Data directory holding `config.db` (created if missing). |
| `--allow-delete` | Apply a file whose diff deletes rows. |
**`import` is a stop-first operation.** It rewrites configuration underneath a
process that read it at startup, and a running server notices only some of it:
filtering picks up the imported rows at the next reload, while upstreams,
listeners and settings stay at their boot values until a restart.
**`import` is a stop-first operation.** It rewrites configuration underneath a process that read it at startup, and a running server notices only some of it: filtering picks up the imported rows at the next reload, while upstreams, listeners and settings stay at their boot values until a restart.
### The delete gate
Rows the database holds and FILE does not name are deleted. That is the point of
a declarative apply, and it is also how a mistaken `nxdns import ./wrong.zon`
empties a configured server, so it takes a flag:
Rows the database holds and FILE does not name are deleted. That is the point of a declarative apply, and it is also how a mistaken `nxdns import ./wrong.zon` empties a configured server, so it takes a flag:
```
FAIL import: this file would delete rows the database holds (upstreams 1); re-run with --allow-delete to apply it
import failed: DestructiveImport
```
That is exit 2, and the transaction rolls back. The message names every table
with a non-zero delete count, so you can tell an intended pruning from a wrong
file before applying anything.
That is exit 2, and the transaction rolls back. The message names every table with a non-zero delete count, so you can tell an intended pruning from a wrong file before applying anything.
An import that only adds rows, or only edits them, needs no flag. "Edit" here
means a change to a row nxdns can still recognise as the same row. Each table
has one column, or one tuple, that establishes identity:
An import that only adds rows, or only edits them, needs no flag. "Edit" here means a change to a row nxdns can still recognise as the same row. Each table has one column, or one tuple, that establishes identity:
| Table | Identity |
| --- | --- |
@@ -282,43 +188,25 @@ has one column, or one tuple, that establishes identity:
| `local_records` | `(name, rtype, value)` |
| `rules` | `(group, pattern, kind, action)` |
Change anything else on a row — a source's name, a group's `safe_search`, a
client's group — and it is an edit, applied without a flag. Change the identity
itself, such as renaming a group or correcting a typo in an upstream URL, and
the engine sees a row that vanished and a row that appeared: that needs
`--allow-delete`.
Change anything else on a row — a source's name, a group's `safe_search`, a client's group — and it is an edit, applied without a flag. Change the identity itself, such as renaming a group or correcting a typo in an upstream URL, and the engine sees a row that vanished and a row that appeared: that needs `--allow-delete`.
### What survives an import
Runtime state is not declarative and is preserved by identity, not by luck. A
blocklist source whose URL is unchanged keeps its row id, its checksum, its
counters and its compiled files, so an import costs no downloads. Clients the
DNS path materialised from traffic are kept whole; naming one in the file
promotes that row in place, keeping its first-seen and last-seen. Observed
clients whose group the file no longer declares are moved to the `default`
group rather than deleted with it, and none of that ever trips the delete gate.
Runtime state is not declarative and is preserved by identity, not by luck. A blocklist source whose URL is unchanged keeps its row id, its checksum, its counters and its compiled files, so an import costs no downloads. Clients the DNS path materialised from traffic are kept whole; naming one in the file promotes that row in place, keeping its first-seen and last-seen. Observed clients whose group the file no longer declares are moved to the `default` group rather than deleted with it, and none of that ever trips the delete gate.
### `import` against a file-mode box
It behaves like any other import. Nothing in the database records that a file
governs it — authority lives in the invocation — so `import` neither detects nor
refuses that case. The next restart's reconcile converges the database back to
the file and its summary reports what it corrected. A source the import deleted
comes back with a new row id, which means a fresh download of the whole list.
It behaves like any other import. Nothing in the database records that a file governs it — authority lives in the invocation — so `import` neither detects nor refuses that case. The next restart's reconcile converges the database back to the file and its summary reports what it corrected. A source the import deleted comes back with a new row id, which means a fresh download of the whole list.
If a restart and an import race for the write lock, one of them simply wins: both
take `BEGIN IMMEDIATE` under a 5-second busy timeout, so the outcome is an
ordering, never a corrupted database.
If a restart and an import race for the write lock, one of them simply wins: both take `BEGIN IMMEDIATE` under a 5-second busy timeout, so the outcome is an ordering, never a corrupted database.
## `version`
Prints two lines: the nxdns version with the git commit, then the Zig version
the binary was built with. Takes no flags and no arguments.
Prints two lines: the nxdns version with the git commit, then the Zig version the binary was built with. Takes no flags and no arguments.
## `help`
Prints the usage text to stdout and exits 0. `nxdns --help` and `nxdns -h` do
the same. A usage error prints the same text to stderr and exits 64.
Prints the usage text to stdout and exits 0. `nxdns --help` and `nxdns -h` do the same. A usage error prints the same text to stderr and exits 64.
## Exit codes
@@ -329,50 +217,26 @@ the same. A usage error prints the same text to stderr and exits 64.
| 2 | A configuration problem the operator can fix, or a `check` that found one. |
| 64 | Usage error — unknown command or flag, a flag without its value, a missing or extra argument. |
Code 2 means the same thing from every subcommand. `src/config/faults.zig`
holds the one list of errors that mean "the configuration the operator supplied
is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits
2 whichever command read it. The list is every error the validator raises, plus
`ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams`, `BadCertificate` and
`ManagedConfigUnreadable`. In practice that covers a file with a syntax error,
one larger than 4 MiB, one with no `default` group (`MissingDefaultGroup`), one
with no enabled upstream (`NoUpstreams`), a bad bind address, a bad rate limit,
an unusable certificate, `password` and `password_hash` set together, and a
`--config` path that is absent or unreadable.
Code 2 means the same thing from every subcommand. `src/config/faults.zig` holds the one list of errors that mean "the configuration the operator supplied is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits 2 whichever command read it. The list is every error the validator raises, plus `ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams`, `BadCertificate` and `ManagedConfigUnreadable`. In practice that covers a file with a syntax error, one larger than 4 MiB, one with no `default` group (`MissingDefaultGroup`), one with no enabled upstream (`NoUpstreams`), a bad bind address, a bad rate limit, an unusable certificate, `password` and `password_hash` set together, and a `--config` path that is absent or unreadable.
The last of those is the one deliberate seam. A file nxdns cannot open is a
configuration fault only when the *path* is the problem — the file is missing,
permissions deny it, a path component is not a directory. Every other open
failure, such as running out of file descriptors, is exit 1. The distinction
earns its keep under the shipped systemd unit, which stops the service on exit 2
rather than restarting it: a transient box fault graded as a configuration fault
would take the resolver down until someone noticed.
The last of those is the one deliberate seam. A file nxdns cannot open is a configuration fault only when the *path* is the problem — the file is missing, permissions deny it, a path component is not a directory. Every other open failure, such as running out of file descriptors, is exit 1. The distinction earns its keep under the shipped systemd unit, which stops the service on exit 2 rather than restarting it: a transient box fault graded as a configuration fault would take the resolver down until someone noticed.
When `run` exits 2 it points at the diagnosis on stderr, whichever authority the
fault came from:
When `run` exits 2 it points at the diagnosis on stderr, whichever authority the fault came from:
```
run `nxdns check` to see the configuration in full
```
On a box with no configuration at all, `run` and `check` add the line that names
both ways to get one:
On a box with no configuration at all, `run` and `check` add the line that names both ways to get one:
```
load one with `nxdns import <file>`, or make a file the source of truth with `nxdns run --config <file>`
```
`check` exits 2 for those faults and also when a probed upstream failed, when a
named configuration file is missing or unreadable, and when the database cannot
be read, is absent, or is not at this binary's schema version. Warnings never
contribute.
`check` exits 2 for those faults and also when a probed upstream failed, when a named configuration file is missing or unreadable, and when the database cannot be read, is absent, or is not at this binary's schema version. Warnings never contribute.
`import` exits 2 for those faults and for `DestructiveImport`. That last one is
deliberately not a configuration fault — it reports what applying the file would
delete, rather than anything wrong with its content — and `import` decides it
for itself; the answer to it is `--allow-delete`, not an edit.
`import` exits 2 for those faults and for `DestructiveImport`. That last one is deliberately not a configuration fault — it reports what applying the file would delete, rather than anything wrong with its content — and `import` decides it for itself; the answer to it is `--allow-delete`, not an edit.
`OutOfMemory` is exit 1 even when problems were recorded, because the report is
then incomplete. Every other error is 1.
`OutOfMemory` is exit 1 even when problems were recorded, because the report is then incomplete. Every other error is 1.
Where an exit code sends you next: [troubleshoot](../how-to/troubleshoot.md).
docs/reference/configuration.md
+55 -214
View File
@@ -1,35 +1,20 @@
# Configuration reference
Every section, field and collection nxdns accepts, with its type, default,
unit, validation rule and the subsystem that consumes it.
Every section, field and collection nxdns accepts, with its type, default, unit, validation rule and the subsystem that consumes it.
Source of truth: `src/config/model.zig` (the model and the defaults),
`src/config/validate.zig` (the rules), `src/config/{loader,reconcile,import,export}.zig`
(the lifecycle).
Source of truth: `src/config/model.zig` (the model and the defaults), `src/config/validate.zig` (the rules), `src/config/{loader,reconcile,import,export}.zig` (the lifecycle).
For how the file, the database and `export`/`import` relate to each other, see
[the configuration model](../explanation/configuration-model.md). For the
commands that read and write configuration, see [the CLI
reference](cli.md).
For how the file, the database and `export`/`import` relate to each other, see [the configuration model](../explanation/configuration-model.md). For the commands that read and write configuration, see [the CLI reference](cli.md).
## File format
The file is ZON: a top-level anonymous struct whose fields are the sections and
collections below. Enum values are ZON enum literals (`.level = .err`,
`.response = .nxdomain`). Strings are double-quoted. The file may be at most
4 MiB (`max_config_bytes` in `src/config/loader.zig`); beyond that the error is
`ConfigTooLarge`. A syntax error is reported with its line and column.
The file is ZON: a top-level anonymous struct whose fields are the sections and collections below. Enum values are ZON enum literals (`.level = .err`, `.response = .nxdomain`). Strings are double-quoted. The file may be at most 4 MiB (`max_config_bytes` in `src/config/loader.zig`); beyond that the error is `ConfigTooLarge`. A syntax error is reported with its line and column.
Absent fields keep their defaults, both in the file and in the database. A
settings key stored in the database that the running binary does not know is
warned about and ignored, never an error.
Absent fields keep their defaults, both in the file and in the database. A settings key stored in the database that the running binary does not know is warned about and ignored, never an error.
### The `logging.level = .err` quirk
The log level `error` is a Zig keyword, so the ZON and model tag is `.err` while
the database and the settings API store the operator-facing word `"error"`. The
file says `.err`; `GET /api/settings` says `"error"`. `"err"` is not accepted as
database text, and `.error` is not a ZON tag.
The log level `error` is a Zig keyword, so the ZON and model tag is `.err` while the database and the settings API store the operator-facing word `"error"`. The file says `.err`; `GET /api/settings` says `"error"`. `"err"` is not accepted as database text, and `.error` is not a ZON tag.
## What is not in the file
@@ -43,9 +28,7 @@ Storage paths are process arguments, not configuration:
## Scalar sections
The "Key" column is the settings key as stored in the database
(`section.field`); in the file the same field lives inside its section block,
for example `.dns = .{ .port = 53 }`.
The "Key" column is the settings key as stored in the database (`section.field`); in the file the same field lives inside its section block, for example `.dns = .{ .port = 53 }`.
### upstream
@@ -57,16 +40,9 @@ Timeouts for talking to upstream resolvers.
| `upstream.read_timeout_ms` | u32 | 3000 | ms | 100–120000 | read deadline on conditional-forward-zone exchanges (`src/local/forward_client.zig`) |
| `upstream.total_timeout_ms` | u32 | 5000 | ms | 100–120000 | per-query budget of the upstream pool (`src/upstream/pool.zig`): every failover attempt together, not one of them; also the `nxdns check` probe deadline |
The two pool budgets nest. `attempt_timeout_ms` bounds one try against one
upstream; when it expires the pool records the failure and moves to the next
candidate. `total_timeout_ms` bounds the whole loop, so a query against five
unreachable upstreams costs the total budget once, not five attempt budgets in
a row. When the total expires the in-flight attempt is canceled and the query
fails with a timeout.
The two pool budgets nest. `attempt_timeout_ms` bounds one try against one upstream; when it expires the pool records the failure and moves to the next candidate. `total_timeout_ms` bounds the whole loop, so a query against five unreachable upstreams costs the total budget once, not five attempt budgets in a row. When the total expires the in-flight attempt is canceled and the query fails with a timeout.
`read_timeout_ms` is unrelated to both. It bounds a different subsystem — the
conditional-forward-zone client — so no cross-check relates it to the pool's
budgets, and it is free to sit above either of them.
`read_timeout_ms` is unrelated to both. It bounds a different subsystem — the conditional-forward-zone client — so no cross-check relates it to the pool's budgets, and it is free to sit above either of them.
### dns
@@ -80,10 +56,7 @@ The plain DNS listener (UDP and TCP).
| `dns.rate_limit` | u32 | 1000 | queries per window | at least 1 | per-client DNS rate limiter (`src/server/rate_limiter.zig`) |
| `dns.rate_window_seconds` | u32 | 60 | seconds | 1–3600 | window of the same limiter |
`dns.bind_ipv4` and `dns.bind_ipv6` each name one socket of the dual-stack pair,
so each is required to be a literal of its own family. An IPv4 wildcard in
`dns.bind_ipv6` is refused: it would bind IPv4 as the "v6" socket and make the
real IPv4 bind fail with `AddressInUse`, silently removing the IPv6 service.
`dns.bind_ipv4` and `dns.bind_ipv6` each name one socket of the dual-stack pair, so each is required to be a literal of its own family. An IPv4 wildcard in `dns.bind_ipv6` is refused: it would bind IPv4 as the "v6" socket and make the real IPv4 bind fail with `AddressInUse`, silently removing the IPv6 service.
### blocking
@@ -101,10 +74,7 @@ What a blocked query gets back.
| `cache.size` | u32 | 10000 | entries | 1–1000000 | DNS answer cache capacity (`src/cache/dns_cache.zig`) |
| `cache.negative_ttl_max` | u32 | 3600 | seconds | at most 86400 | cap on cached negative answers; 0 disables negative caching |
The cache's slot array is allocated in full at startup, so `cache.size` carries
a ceiling: it is a sanity bound against a typo, not a promise that the value
fits in the box's memory. There is no "off" value — to run without a cache, set
`cache.size` to 1.
The cache's slot array is allocated in full at startup, so `cache.size` carries a ceiling: it is a sanity bound against a typo, not a promise that the value fits in the box's memory. There is no "off" value — to run without a cache, set `cache.size` to 1.
### web
@@ -125,8 +95,7 @@ The web interface and REST API.
### doh_server
The DNS-over-HTTPS listener (server side, for clients on the LAN). See
[enable DoH and DoT](../how-to/enable-doh-and-dot.md).
The DNS-over-HTTPS listener (server side, for clients on the LAN). See [enable DoH and DoT](../how-to/enable-doh-and-dot.md).
| Key | Type | Default | Unit | Validation | Consumed by |
|---|---|---|---|---|---|
@@ -138,8 +107,7 @@ The DNS-over-HTTPS listener (server side, for clients on the LAN). See
### dot_server
The DNS-over-TLS listener. Same shape as `doh_server`; only the default port
differs.
The DNS-over-TLS listener. Same shape as `doh_server`; only the default port differs.
| Key | Type | Default | Unit | Validation | Consumed by |
|---|---|---|---|---|---|
@@ -173,9 +141,7 @@ Process log and query log behavior.
### disk
Free-space thresholds for the data directory. Below them the query-log writer,
the client tracker and the blocklist scheduler are throttled
(`src/storage/disk_monitor.zig`); DNS resolution is never gated.
Free-space thresholds for the data directory. Below them the query-log writer, the client tracker and the blocklist scheduler are throttled (`src/storage/disk_monitor.zig`); DNS resolution is never gated.
| Key | Type | Default | Unit | Validation | Consumed by |
|---|---|---|---|---|---|
@@ -191,23 +157,18 @@ the client tracker and the blocklist scheduler are throttled
## Collections
Collections are ZON lists of structs. Fields without a default are required.
Runtime columns (first/last seen timestamps, per-source download counters) are
deliberately not part of the model: import sets timestamps to the import time
and export omits them, which is what keeps the round trip byte-stable.
Collections are ZON lists of structs. Fields without a default are required. Runtime columns (first/last seen timestamps, per-source download counters) are deliberately not part of the model: import sets timestamps to the import time and export omits them, which is what keeps the round trip byte-stable.
### groups
Client groups. A group named `default` is required; every client not assigned
elsewhere lands in it, and import guarantees it keeps database id 1.
Client groups. A group named `default` is required; every client not assigned elsewhere lands in it, and import guarantees it keeps database id 1.
| Field | Type | Default | Validation |
|---|---|---|---|
| `name` | string | required | non-empty, unique |
| `safe_search` | bool | false | — |
Consumed by the filter engine (`src/filter/matcher.zig`); `safe_search`
triggers the safe-search rewrite in the query path.
Consumed by the filter engine (`src/filter/matcher.zig`); `safe_search` triggers the safe-search rewrite in the query path.
### upstreams
@@ -220,23 +181,15 @@ Upstream resolvers. At least one enabled upstream is required.
| `enabled` | bool | true | — |
| `tls_name` | string | `""` | DoT only — a `tls_name` on an `https://` upstream is an error; when set it must be a valid domain name |
Consumed by the upstream pool (`src/upstream/pool.zig`): entries are sorted by
ascending priority and tried in order with failover. `tls_name` sets SNI and the
certificate verification name for a `tls://` upstream written as an IP literal;
empty means "verify by the URL host" (`src/upstream/dot_client.zig`).
Consumed by the upstream pool (`src/upstream/pool.zig`): entries are sorted by ascending priority and tried in order with failover. `tls_name` sets SNI and the certificate verification name for a `tls://` upstream written as an IP literal; empty means "verify by the URL host" (`src/upstream/dot_client.zig`).
A `tls://` upstream's host must be an IP literal. nxdns does not resolve an
upstream's own name — that is a bootstrap problem, and the DoT client refuses a
non-literal host on every dial — so the validator rejects the hostname form
rather than letting it fail at query time as an unreachable upstream. Write the
address and put the name in `tls_name`:
A `tls://` upstream's host must be an IP literal. nxdns does not resolve an upstream's own name — that is a bootstrap problem, and the DoT client refuses a non-literal host on every dial — so the validator rejects the hostname form rather than letting it fail at query time as an unreachable upstream. Write the address and put the name in `tls_name`:
```zig
.{ .url = "tls://9.9.9.9:853", .tls_name = "dns.quad9.net" },
```
A DoH upstream is not affected: `https://` goes through the HTTP client, which
resolves normally, so a hostname there is the usual form.
A DoH upstream is not affected: `https://` goes through the HTTP client, which resolves normally, so a hostname there is the usual form.
### clients
@@ -248,36 +201,19 @@ Known clients with a fixed group assignment.
| `name` | string | `""` | — (display only, never read by the resolver) |
| `group` | string | `"default"` | must name a declared group |
Consumed by the filter engine's exact address-to-group lookup
(`src/filter/matcher.zig`).
Consumed by the filter engine's exact address-to-group lookup (`src/filter/matcher.zig`).
#### Learned names
A client row that carries no name of its own can still show one. Every flush
pass the client tracker takes at most 16 unnamed rows that are due, builds each
address's reverse name (`192.168.1.10` becomes `10.1.168.192.in-addr.arpa`), and
matches it against the declared [`forward_zones`](#forward_zones). On a match it
sends one PTR query to that zone's resolver and stores the answer as the row's
*learned* name. A row becomes due again 24 hours after the resolver answered or
returned NXDOMAIN, and 1 hour after any other outcome — no covering zone, a
transport failure, or a reply nxdns rejected — so declaring the missing zone or
fixing the resolver shows names within the hour. This
requires a conditional forward zone that covers the LAN's reverse space — for
example `168.192.in-addr.arpa` pointed at the router. Without such a zone nxdns
sends the reverse name to nobody, and the row stays unnamed.
A client row that carries no name of its own can still show one. Every flush pass the client tracker takes at most 16 unnamed rows that are due, builds each address's reverse name (`192.168.1.10` becomes `10.1.168.192.in-addr.arpa`), and matches it against the declared [`forward_zones`](#forward_zones). On a match it sends one PTR query to that zone's resolver and stores the answer as the row's *learned* name. A row becomes due again 24 hours after the resolver answered or returned NXDOMAIN, and 1 hour after any other outcome — no covering zone, a transport failure, or a reply nxdns rejected — so declaring the missing zone or fixing the resolver shows names within the hour. This requires a conditional forward zone that covers the LAN's reverse space — for example `168.192.in-addr.arpa` pointed at the router. Without such a zone nxdns sends the reverse name to nobody, and the row stays unnamed.
The PTR query goes to the resolver of the declared zone, wherever the operator
pointed it; nxdns does not second-guess that declaration, and it cannot stop a
LAN resolver from forwarding the query onward.
The PTR query goes to the resolver of the declared zone, wherever the operator pointed it; nxdns does not second-guess that declaration, and it cannot stop a LAN resolver from forwarding the query onward.
A learned name is runtime state, like `last_seen`:
- A hand-typed `name` always wins, and a named row is never asked about again.
- Learned names never appear in `nxdns export`, and `nxdns import` never sets
one.
- Each row refreshes once a day. A device rename or a DHCP lease change can
therefore display a stale name for up to 24 hours, until the next refresh
overwrites it or the router reports no name and nxdns clears it.
- Learned names never appear in `nxdns export`, and `nxdns import` never sets one.
- Each row refreshes once a day. A device rename or a DHCP lease change can therefore display a stale name for up to 24 hours, until the next refresh overwrites it or the router reports no name and nxdns clears it.
### client_prefixes
@@ -289,8 +225,7 @@ Group assignment by CIDR prefix, for clients without an exact entry.
| `group` | string | `"default"` | must name a declared group |
| `priority` | i32 | 100 | — (ties on match are broken by lower priority) |
Consumed by the filter engine's longest-prefix match
(`src/filter/matcher.zig`).
Consumed by the filter engine's longest-prefix match (`src/filter/matcher.zig`).
### blocklist_sources
@@ -303,32 +238,11 @@ Downloadable blocklists.
| `enabled` | bool | true | — |
| `is_suggested` | bool | false | — (web UI hint only, never read by the resolver) |
Consumed by the blocklist manager (`src/filter/manager.zig`): downloaded by the
fetcher and compiled into domain sets. A disabled source is neither downloaded
nor loaded.
Consumed by the blocklist manager (`src/filter/manager.zig`): downloaded by the fetcher and compiled into domain sets. A disabled source is neither downloaded nor loaded.
A list in Adblock Plus syntax may also carry exception lines, `@@||name^` and
`@@||name`, either of which may end in `$important`. Those become allow entries
that cancel what any attached list blocks, for the name and its subdomains. They
cancel nothing an operator decided: every rule of the table above is checked
first, so a downloaded list can reopen only a hole another downloaded list dug.
Each source reports how many it carried as `exceptions`; there is no way to write
one by hand, and no reason to want one — write an allow rule instead.
A list in Adblock Plus syntax may also carry exception lines, `@@||name^` and `@@||name`, either of which may end in `$important`. Those become allow entries that cancel what any attached list blocks, for the name and its subdomains. They cancel nothing an operator decided: every rule of the table above is checked first, so a downloaded list can reopen only a hole another downloaded list dug. Each source reports how many it carried as `exceptions`; there is no way to write one by hand, and no reason to want one — write an allow rule instead.
Two counters report what a compile skipped, and they are different facts.
`skipped_regex` counts regex lines: nxdns has a regex engine, but it takes
patterns only from the operator, so a regex line in a downloaded list is counted,
skipped and surfaced — adopt the ones you trust as `regex` rules.
`skipped_unsupported` counts lines nxdns cannot safely translate into a DNS
decision: cosmetic element hiding (`##`, `#@#`, `#?#`), rules carrying a `$`
modifier (except `$important` on an exception line, tolerated above), scheme
anchors, non-anchored `@@` forms — and, in a `domains`-format
list, a line holding more than one field before its inline comment, which usually
means the list is really a hosts file that was declared as `domains`. Neither is
an error, and the two are never one number. A large `skipped_unsupported` beside
a small `domain_count` usually means the list is written for browser extensions,
and its DNS or hosts variant will block more here. Both appear per source in the
blocklists UI and on `/api/blocklists`.
Two counters report what a compile skipped, and they are different facts. `skipped_regex` counts regex lines: nxdns has a regex engine, but it takes patterns only from the operator, so a regex line in a downloaded list is counted, skipped and surfaced — adopt the ones you trust as `regex` rules. `skipped_unsupported` counts lines nxdns cannot safely translate into a DNS decision: cosmetic element hiding (`##`, `#@#`, `#?#`), rules carrying a `$` modifier (except `$important` on an exception line, tolerated above), scheme anchors, non-anchored `@@` forms — and, in a `domains`-format list, a line holding more than one field before its inline comment, which usually means the list is really a hosts file that was declared as `domains`. Neither is an error, and the two are never one number. A large `skipped_unsupported` beside a small `domain_count` usually means the list is written for browser extensions, and its DNS or hosts variant will block more here. Both appear per source in the blocklists UI and on `/api/blocklists`.
### group_sources
@@ -339,8 +253,7 @@ Which groups consult which blocklist sources.
| `group` | string | required | must name a declared group |
| `source_url` | string | required | must name a declared blocklist source's `url`; the (group, source_url) pair is unique |
Consumed by the filter engine when assembling each group's compiled domain sets
(`src/filter/matcher.zig`). A link to a disabled source is skipped.
Consumed by the filter engine when assembling each group's compiled domain sets (`src/filter/matcher.zig`). A link to a disabled source is skipped.
### rules
@@ -353,45 +266,21 @@ Per-group allow and block overrides, checked before the blocklists.
| `kind` | enum `.exact` \| `.wildcard` \| `.regex` | required | — |
| `action` | enum `.allow` \| `.block` | required | — |
Pattern rules: an `.exact` pattern is a plain domain name and may not contain
`*`. A `.wildcard` pattern must contain at least one label that is exactly `*`
(`*.tracker.example`, or `*` alone), and every other label must be a legal DNS
label. `ads*.example` is not a valid wildcard; a partial label is what the
`.regex` kind is for.
Pattern rules: an `.exact` pattern is a plain domain name and may not contain `*`. A `.wildcard` pattern must contain at least one label that is exactly `*` (`*.tracker.example`, or `*` alone), and every other label must be a legal DNS label. `ads*.example` is not a valid wildcard; a partial label is what the `.regex` kind is for.
A `.regex` pattern is a regular expression matched against the whole normalized
lowercase name, unanchored unless you write `^` or `$` — the POSIX-grep
convention. It is stored exactly as you typed it, which the other two kinds are
not: lowercasing would turn `\D` into `\d`, and trimming a trailing `.` would
delete an any-byte atom. The engine (`src/filter/regex.zig`) accepts literal
bytes, `.` for any byte, character classes `[a-z0-9]` with a leading `^` for
negation, the escapes `\d` and `\w` plus `\` before any other ASCII punctuation
to make it a literal, the repetitions `*` `+` `?` `{n}` `{n,m}` `{n,}`,
alternation `|`, grouping `(...)`, and the anchors `^` and `$`.
A `.regex` pattern is a regular expression matched against the whole normalized lowercase name, unanchored unless you write `^` or `$` — the POSIX-grep convention. It is stored exactly as you typed it, which the other two kinds are not: lowercasing would turn `\D` into `\d`, and trimming a trailing `.` would delete an any-byte atom. The engine (`src/filter/regex.zig`) accepts literal bytes, `.` for any byte, character classes `[a-z0-9]` with a leading `^` for negation, the escapes `\d` and `\w` plus `\` before any other ASCII punctuation to make it a literal, the repetitions `*` `+` `?` `{n}` `{n,m}` `{n,}`, alternation `|`, grouping `(...)`, and the anchors `^` and `$`.
Everything else is refused at the edge rather than approximated, so a pattern
written for another engine fails where you can read the diagnostic instead of
silently matching names you did not mean:
Everything else is refused at the edge rather than approximated, so a pattern written for another engine fails where you can read the diagnostic instead of silently matching names you did not mean:
- backreferences, lookaround, captures, named groups, Unicode classes and the
`(?…)` prefix they share;
- backreferences, lookaround, captures, named groups, Unicode classes and the `(?…)` prefix they share;
- any alphanumeric escape the list above omits — `\s`, `\b`, `\1`, `\D`;
- a `]` inside a class, unless written `\]`;
- an empty pattern, and an empty branch: `ads|` is refused rather than read as a
pattern that matches every name;
- a quantifier applied straight to another quantifier: `a+?` is refused rather
than read as `(a+)?`, which matches every name. Write `(a+)?` to mean that.
- an empty pattern, and an empty branch: `ads|` is refused rather than read as a pattern that matches every name;
- a quantifier applied straight to another quantifier: `a+?` is refused rather than read as `(a+)?`, which matches every name. Write `(a+)?` to mean that.
A pattern is at most 256 bytes and compiles to at most 1024 instructions, each
limit with its own diagnostic, and one group holds at most 256 regex rules.
Groups do not capture, and the engine simulates every alternative in lockstep,
so a pattern costs at most its compiled length times the length of the name —
`(a+)+b` is as cheap here as it is expensive in a backtracking engine.
A pattern is at most 256 bytes and compiles to at most 1024 instructions, each limit with its own diagnostic, and one group holds at most 256 regex rules. Groups do not capture, and the engine simulates every alternative in lockstep, so a pattern costs at most its compiled length times the length of the name — `(a+)+b` is as cheap here as it is expensive in a backtracking engine.
Consumed by the filter engine's rule sets (`src/filter/rules.zig`), which checks
the three kinds in the order they are listed above, allow before block within
each. Regex is checked last of the three because it is the only kind that costs
more than a hash lookup or a label walk.
Consumed by the filter engine's rule sets (`src/filter/rules.zig`), which checks the three kinds in the order they are listed above, allow before block within each. Regex is checked last of the three because it is the only kind that costs more than a hash lookup or a label walk.
### local_records
@@ -404,27 +293,20 @@ Local DNS answers, served without touching any upstream.
| `value` | string | required | an IPv4 address for `.a`, an IPv6 address for `.aaaa`, a domain name for `.cname` |
| `ttl` | u32 | 300 | 1–604800 seconds |
The (name, rtype, value) triple is unique. Consumed by the local records table
in the query path (`src/local/records.zig`).
The (name, rtype, value) triple is unique. Consumed by the local records table in the query path (`src/local/records.zig`).
### forward_zones
Zones resolved by a specific resolver instead of the configured upstreams, for
LAN or corporate domains.
Zones resolved by a specific resolver instead of the configured upstreams, for LAN or corporate domains.
| Field | Type | Default | Validation |
|---|---|---|---|
| `zone` | string | required | a valid domain name; unique |
| `resolver` | string | required | `udp://IP:port` or `tcp://IP:port`; the host must be an IP literal and the port is mandatory |
The resolver host must be an IP literal because resolving the resolver's own
name would be a bootstrap problem. Matching is longest suffix
(`src/local/forward_zones.zig`); the exchange is UDP then TCP
(`src/local/forward_client.zig`) with `upstream.read_timeout_ms` as the read
deadline.
The resolver host must be an IP literal because resolving the resolver's own name would be a bootstrap problem. Matching is longest suffix (`src/local/forward_zones.zig`); the exchange is UDP then TCP (`src/local/forward_client.zig`) with `upstream.read_timeout_ms` as the read deadline.
Reverse zones are declared the same way, and one is the prerequisite for
[learned client names](#learned-names):
Reverse zones are declared the same way, and one is the prerequisite for [learned client names](#learned-names):
```zig
.forward_zones = .{
@@ -435,9 +317,7 @@ Reverse zones are declared the same way, and one is the prerequisite for
## Password and hash
Both fields are optional, and the difference between *absent* and *empty* is the
whole design. Absent means "keep whatever is stored". Empty means "there is no
password".
Both fields are optional, and the difference between *absent* and *empty* is the whole design. Absent means "keep whatever is stored". Empty means "there is no password".
| The file says | What happens to the stored hash |
| --- | --- |
@@ -448,72 +328,35 @@ password".
| `.password_hash = ""` | Cleared, which disables authentication. |
| Both fields | Refused (`PasswordAndHashBothSet` — ambiguity in a security setting). |
Silence has to mean "keep", because the alternative is a trap. An operator who
exports a configuration and trims the long PHC string out of it before
committing the file to git means "leave the password alone", not "open the admin
interface to the LAN". So disabling authentication takes the explicit empty
string, and the empty *plaintext* — which would otherwise hash into a real hash
that no login can ever satisfy — is refused outright:
Silence has to mean "keep", because the alternative is a trap. An operator who exports a configuration and trims the long PHC string out of it before committing the file to git means "leave the password alone", not "open the admin interface to the LAN". So disabling authentication takes the explicit empty string, and the empty *plaintext* — which would otherwise hash into a real hash that no login can ever satisfy — is refused outright:
```
FAIL web.password: password is set to the empty string; omit the field to keep the stored password, or set password_hash = "" to disable authentication
```
- `web.password` is operator input only. It is hashed with argon2id (OWASP
parameters: t=2, m=19 MiB, p=1, PHC encoding) into `web.password_hash` and
discarded. There is no `web.password` settings row, and `nxdns export` always
writes `.password = null`.
- `web.password_hash` is the stored argon2id PHC string. Supplying it directly,
for example from a previous export, is how a backup restores authentication
without knowing the password.
- `web.password` is operator input only. It is hashed with argon2id (OWASP parameters: t=2, m=19 MiB, p=1, PHC encoding) into `web.password_hash` and discarded. There is no `web.password` settings row, and `nxdns export` always writes `.password = null`.
- `web.password_hash` is the stored argon2id PHC string. Supplying it directly, for example from a previous export, is how a backup restores authentication without knowing the password.
A plaintext password that has not changed is verified rather than re-hashed, so
applying the same file twice leaves the same bytes in the database. That is what
keeps the export/import round trip byte-stable with a password in the file. It
costs a full argon2id computation either way — the verification is not a
shortcut, and caching the plaintext to skip it would be a security bug.
A plaintext password that has not changed is verified rather than re-hashed, so applying the same file twice leaves the same bytes in the database. That is what keeps the export/import round trip byte-stable with a password in the file. It costs a full argon2id computation either way — the verification is not a shortcut, and caching the plaintext to skip it would be a security bug.
Any change to whether a password is set is announced at startup, never left as a
count:
Any change to whether a password is set is announced at startup, never left as a count:
```
web authentication is now enabled
web authentication is now disabled
```
See [set up admin
authentication](../how-to/set-up-admin-authentication.md).
See [set up admin authentication](../how-to/set-up-admin-authentication.md).
## Validation errors
`nxdns check`, `nxdns import` and a `nxdns run --config` that reads the file
all print one `FAIL path: message` line per problem, and report every
problem rather than the first. A finding that is legal but almost certainly
unintended is prefixed `WARN` instead: it does not change the exit code, and it
is printed by all three even when nothing failed, so an accepted configuration
still says what is odd about it.
`nxdns check`, `nxdns import` and a `nxdns run --config` that reads the file all print one `FAIL path: message` line per problem, and report every problem rather than the first. A finding that is legal but almost certainly unintended is prefixed `WARN` instead: it does not change the exit code, and it is printed by all three even when nothing failed, so an accepted configuration still says what is odd about it.
A url in a diagnostic is redacted to its scheme, host and port. The userinfo,
the path, the query and the fragment are dropped, and control characters are
escaped. These lines reach the journal, and every one of those parts can carry a
credential: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the
path segment is the whole account identifier. The field path beside the message
names the entry, so `blocklist_sources[1].url` still says which one to go and
fix.
A url in a diagnostic is redacted to its scheme, host and port. The userinfo, the path, the query and the fragment are dropped, and control characters are escaped. These lines reach the journal, and every one of those parts can carry a credential: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the path segment is the whole account identifier. The field path beside the message names the entry, so `blocklist_sources[1].url` still says which one to go and fix.
What redaction cannot remove is the host, because a hostname is not a secret in
the general case — it is resolved publicly and offered as SNI on every
connection — and dropping it would leave a diagnostic that names nothing worth
reading. A vendor that puts an account identifier in the hostname therefore has
that identifier appear in any line naming the entry. NextDNS is the example:
`abcd12.dns.nextdns.io` is a profile id.
What redaction cannot remove is the host, because a hostname is not a secret in the general case — it is resolved publicly and offered as SNI on every connection — and dropping it would leave a diagnostic that names nothing worth reading. A vendor that puts an account identifier in the hostname therefore has that identifier appear in any line naming the entry. NextDNS is the example: `abcd12.dns.nextdns.io` is a profile id.
Two things limit the exposure. A hostname `tls://` upstream is rejected by the
validator (see `upstreams` above), so the DoT form of that URL can only reach
the log once, in the line rejecting it — never on every failover. And NextDNS
publishes a DoH endpoint, `https://dns.nextdns.io/abcd12`, whose identifier sits
in the path and is redacted in full; configure that form and nothing identifying
reaches the log at all.
Two things limit the exposure. A hostname `tls://` upstream is rejected by the validator (see `upstreams` above), so the DoT form of that URL can only reach the log once, in the line rejecting it — never on every failover. And NextDNS publishes a DoH endpoint, `https://dns.nextdns.io/abcd12`, whose identifier sits in the path and is redacted in full; configure that form and nothing identifying reaches the log at all.
The error set is `validate.ValidateError` in `src/config/validate.zig`:
@@ -550,8 +393,7 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
| `MissingLogPath` | `logging.output = .file` with an empty or relative `file_path` |
| `PasswordAndHashBothSet` | both `web.password` and `web.password_hash` are set |
Warnings are a separate set, outside `ValidateError` because they are not
failures. There is one:
Warnings are a separate set, outside `ValidateError` because they are not failures. There is one:
| Warning | Raised by |
| --- | --- |
@@ -559,8 +401,7 @@ failures. There is one:
## Minimal working example
The smallest file that passes validation: a `default` group and one enabled
upstream. Everything else keeps its default.
The smallest file that passes validation: a `default` group and one enabled upstream. Everything else keeps its default.
```zon
.{
docs/reference/files-and-directories.md
+28 -127
View File
@@ -1,39 +1,16 @@
# Files and directories
Every path nxdns reads or writes, and the mode it is created with. Source of
truth: `src/cli.zig` (`DataDir`), `src/storage/querylog_schema.zig` (the
preserved query-log databases), `src/filter/manager.zig` (the blocklist
snapshots), `src/platform/logging.zig` (the log file).
Every path nxdns reads or writes, and the mode it is created with. Source of truth: `src/cli.zig` (`DataDir`), `src/storage/querylog_schema.zig` (the preserved query-log databases), `src/filter/manager.zig` (the blocklist snapshots), `src/platform/logging.zig` (the log file).
## The data directory
Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and
`nxdns import` create it and its parents at mode 0700 when it is missing;
`nxdns check` and `nxdns export` do not create it. `export` fails if it is not
there. `check` opens it only when no `--config FILE` was given: with that flag it
grades the file and never looks at the directory at all. Without it, an absent
`config.db` is a failure naming the two ways to get one, exit 2.
Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and `nxdns import` create it and its parents at mode 0700 when it is missing; `nxdns check` and `nxdns export` do not create it. `export` fails if it is not there. `check` opens it only when no `--config FILE` was given: with that flag it grades the file and never looks at the directory at all. Without it, an absent `config.db` is a failure naming the two ways to get one, exit 2.
`run`, `import` and `export` go through `DataDir.openConfigDb`, which opens
`config.db` read/write, chmods it to 0600, enables WAL — creating
`config.db-wal` and `config.db-shm` — and then runs any pending schema
migrations. So `nxdns export` writes to the data directory, and on a database
one schema version behind it migrates it. `export` also creates an empty
`config.db` if the directory exists without one.
`run`, `import` and `export` go through `DataDir.openConfigDb`, which opens `config.db` read/write, chmods it to 0600, enables WAL — creating `config.db-wal` and `config.db-shm` — and then runs any pending schema migrations. So `nxdns export` writes to the data directory, and on a database one schema version behind it migrates it. `export` also creates an empty `config.db` if the directory exists without one.
`nxdns check` is the exception: it does not use that path at all. It opens
`config.db` immutable, which is `SQLITE_OPEN_READONLY` plus `immutable=1`, so
SQLite refuses every statement that would write and builds no wal-index. No
`config.db-wal` and no `config.db-shm` appear beside the file, the mode is left
alone, and no migration runs — a database behind this binary's schema is
reported as a failure naming `nxdns run` as the fix. A `check` against a data
directory leaves it byte-identical, and `check` reaches the database branch only
when `config.db` is already there.
`nxdns check` is the exception: it does not use that path at all. It opens `config.db` immutable, which is `SQLITE_OPEN_READONLY` plus `immutable=1`, so SQLite refuses every statement that would write and builds no wal-index. No `config.db-wal` and no `config.db-shm` appear beside the file, the mode is left alone, and no migration runs — a database behind this binary's schema is reported as a failure naming `nxdns run` as the fix. A `check` against a data directory leaves it byte-identical, and `check` reaches the database branch only when `config.db` is already there.
`immutable=1` ignores any `-wal` file, so it is refused rather than used when
one holds bytes: the newest settings would be invisible and `check` would grade
older ones from the main file. That is the "uncheckpointed changes" failure in
[the CLI reference](cli.md#what-check-does-not-do).
`immutable=1` ignores any `-wal` file, so it is refused rather than used when one holds bytes: the newest settings would be invisible and `check` would grade older ones from the main file. That is the "uncheckpointed changes" failure in [the CLI reference](cli.md#what-check-does-not-do).
| Path | What it is | Mode |
| --- | --- | --- |
@@ -53,29 +30,15 @@ older ones from the main file. That is the "uncheckpointed changes" failure in
### The orphan sweep
The sweep decides by id, not by suffix. It matches all seven names above and
deletes those whose `<id>` is no longer a `blocklist_sources` row, so the
compiled `.list`, `.wild` and `.allow` of a removed source go, and so do a
`.raw.tmp`, `.list.tmp`, `.wild.tmp` or `.allow.tmp` left behind by a refresh
that was killed before it could clean up. Files belonging to a source that still
has a row are never touched, whatever state they are in: the sweep holds the same
lock every refresh takes, so it never reads the directory while a refresh is
part-way through.
The sweep decides by id, not by suffix. It matches all seven names above and deletes those whose `<id>` is no longer a `blocklist_sources` row, so the compiled `.list`, `.wild` and `.allow` of a removed source go, and so do a `.raw.tmp`, `.list.tmp`, `.wild.tmp` or `.allow.tmp` left behind by a refresh that was killed before it could clean up. Files belonging to a source that still has a row are never touched, whatever state they are in: the sweep holds the same lock every refresh takes, so it never reads the directory while a refresh is part-way through.
It runs at three moments:
- at startup, before the first refresh pass — this is what collects what a
killed process left behind, and the compiled files of a source deleted while
the server was down;
- before each scheduled update pass, ahead of the disk-space gate: the sweep
only unlinks, so it is the one step here that can give a critically full disk
room back, and gating it would keep the residue that helped fill the disk;
- immediately after `DELETE /api/blocklists/{id}`, which is when an orphan is
actually created in normal operation. Without it a deleted list would keep its
megabytes until the next scheduled pass.
- at startup, before the first refresh pass — this is what collects what a killed process left behind, and the compiled files of a source deleted while the server was down;
- before each scheduled update pass, ahead of the disk-space gate: the sweep only unlinks, so it is the one step here that can give a critically full disk room back, and gating it would keep the residue that helped fill the disk;
- immediately after `DELETE /api/blocklists/{id}`, which is when an orphan is actually created in normal operation. Without it a deleted list would keep its megabytes until the next scheduled pass.
With `blocklist_update.enabled = false` there are no scheduled passes, so only
the first and the last of those three happen.
With `blocklist_update.enabled = false` there are no scheduled passes, so only the first and the last of those three happen.
Each deletion is logged:
@@ -83,110 +46,48 @@ Each deletion is logged:
info(blocklist_manager): pruned orphaned blocklist file 9999.list
```
A failed sweep is a warning, not an outage — leftover bytes do not justify
losing the refresh pass behind them, let alone the server.
A failed sweep is a warning, not an outage — leftover bytes do not justify losing the refresh pass behind them, let alone the server.
The temporaries of a source that still exists are cleaned by the refresh that
owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`,
`.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not.
The temporaries of a source that still exists are cleaned by the refresh that owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`, `.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not.
A `querylog.db` is moved aside when it is missing nothing but usability:
SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not
answer `ok`, or its `user_version` fingerprint does not match the schema. Only
the main file is renamed — its `-wal` and `-shm` are deleted, because a stale
WAL would be replayed into the fresh database. A missing `querylog.db` is
created without any aside file. The rename happens inside
`querylog_schema.open`, before the 0600 chmod, and that chmod names
`querylog.db` and its two sidecars only — so an aside file keeps the mode the
file had at rename time, which for a `querylog.db` nxdns itself created is 0600
and for one an operator put there is whatever they left it at. Nothing prunes
the aside files; they accumulate until an operator removes them, and each one
holds the same browsing history the live query log holds.
A `querylog.db` is moved aside when it is missing nothing but usability: SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not answer `ok`, or its `user_version` fingerprint does not match the schema. Only the main file is renamed — its `-wal` and `-shm` are deleted, because a stale WAL would be replayed into the fresh database. A missing `querylog.db` is created without any aside file. The rename happens inside `querylog_schema.open`, before the 0600 chmod, and that chmod names `querylog.db` and its two sidecars only — so an aside file keeps the mode the file had at rename time, which for a `querylog.db` nxdns itself created is 0600 and for one an operator put there is whatever they left it at. Nothing prunes the aside files; they accumulate until an operator removes them, and each one holds the same browsing history the live query log holds.
The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash
and `querylog.db` holds the browsing history of every client on the LAN, so both
are as sensitive as each other, and a WAL file holds the same rows as the
database it belongs to. SQLite creates the main database at `0644 & ~umask`;
nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather
than being created world-readable.
The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash and `querylog.db` holds the browsing history of every client on the LAN, so both are as sensitive as each other, and a WAL file holds the same rows as the database it belongs to. SQLite creates the main database at `0644 & ~umask`; nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather than being created world-readable.
## The configuration file
There is no default path. `--config FILE` names the file, and without that flag
no file is read at all — a `config.zon` sitting in `/etc/nxdns` that no
invocation names is inert. `/etc/nxdns/config.zon` is a convention the packaging
follows, not a location nxdns probes.
There is no default path. `--config FILE` names the file, and without that flag no file is read at all — a `config.zon` sitting in `/etc/nxdns` that no invocation names is inert. `/etc/nxdns/config.zon` is a convention the packaging follows, not a location nxdns probes.
nxdns reads the file and never writes it, in either authority mode. It does not
create the file or its directory either; the systemd unit's
`ConfigurationDirectory=nxdns` creates `/etc/nxdns`, and the same unit's
`ReadOnlyPaths=/etc/nxdns` denies the service write access to it, so the file
cannot be modified by the process that reads it.
nxdns reads the file and never writes it, in either authority mode. It does not create the file or its directory either; the systemd unit's `ConfigurationDirectory=nxdns` creates `/etc/nxdns`, and the same unit's `ReadOnlyPaths=/etc/nxdns` denies the service write access to it, so the file cannot be modified by the process that reads it.
Under `run --config FILE` the file is the configuration and the database is the
runtime substrate the server reads from: every start reconciles the one onto the
other. So in that mode `config.db` is not the backup — the file is.
Under `run --config FILE` the file is the configuration and the database is the runtime substrate the server reads from: every start reconciles the one onto the other. So in that mode `config.db` is not the backup — the file is.
`nxdns export --out FILE` writes a ZON file at mode 0600 through a temporary
file and a rename. That file carries `web.password_hash`, so treat exports as
secrets. Without `--out` the export goes to stdout, where permissions are the
redirect's business.
`nxdns export --out FILE` writes a ZON file at mode 0600 through a temporary file and a rename. That file carries `web.password_hash`, so treat exports as secrets. Without `--out` the export goes to stdout, where permissions are the redirect's business.
## The log file
Only when `logging.output = .file`. The path is `logging.file_path`, default
`/var/log/nxdns/nxdns.log`, and validation requires it to be absolute.
Only when `logging.output = .file`. The path is `logging.file_path`, default `/var/log/nxdns/nxdns.log`, and validation requires it to be absolute.
**nxdns does not create the log directory.** It must exist and be writable by
the user the service runs as before the process starts; under systemd the unit's
`LogsDirectory=nxdns` does that.
**nxdns does not create the log directory.** It must exist and be writable by the user the service runs as before the process starts; under systemd the unit's `LogsDirectory=nxdns` does that.
Rotation triggers at `logging.max_size_mb` MiB and keeps `logging.max_files`
files in total, the live one included, so the highest generation on disk is
`logging.max_files - 1`. With the default 5 that is `nxdns.log` plus `nxdns.log.1`
through `nxdns.log.4`. Rotation deletes the highest generation, renames each
remaining one up by one, then renames the live file to `.1`. A `max_files` of 1
or 0 deletes the live file instead of renaming it. The directory holding the log
file is also sampled by the disk monitor.
Rotation triggers at `logging.max_size_mb` MiB and keeps `logging.max_files` files in total, the live one included, so the highest generation on disk is `logging.max_files - 1`. With the default 5 that is `nxdns.log` plus `nxdns.log.1` through `nxdns.log.4`. Rotation deletes the highest generation, renames each remaining one up by one, then renames the live file to `.1`. A `max_files` of 1 or 0 deletes the live file instead of renaming it. The directory holding the log file is also sampled by the disk monitor.
| Path | What it is | Mode |
| --- | --- | --- |
| `logging.file_path` (default `/var/log/nxdns/nxdns.log`) | The live process log. Opened write-only and written at a tracked offset; created when it does not exist. | `0666 & ~umask` — nxdns never chmods it |
| `<file_path>.1` … `<file_path>.<max_files - 1>` | Rotated generations, newest first. | Inherited from the live file they were renamed from |
**Do not point external logrotate at this file.** nxdns owns the rotation of
its own log. There is no append mode in Zig 0.16, so the writer reads the file
length once when it opens the file and then writes every line at an offset it
tracks in the process. A rotator that moves the file behind it breaks that
offset, and nxdns does not notice until the next restart:
**Do not point external logrotate at this file.** nxdns owns the rotation of its own log. There is no append mode in Zig 0.16, so the writer reads the file length once when it opens the file and then writes every line at an offset it tracks in the process. A rotator that moves the file behind it breaks that offset, and nxdns does not notice until the next restart:
- With `copytruncate` the offset survives the truncation, so the next line
lands where it would have without it. The file regrows with a sparse,
NUL-filled prefix as long as the log that was just rotated away.
- With rename-and-create nxdns keeps writing to the renamed inode. The new
file stays empty, the renamed one grows without bound, and
`logging.max_size_mb` bounds nothing on disk.
- With `copytruncate` the offset survives the truncation, so the next line lands where it would have without it. The file regrows with a sparse, NUL-filled prefix as long as the log that was just rotated away.
- With rename-and-create nxdns keeps writing to the renamed inode. The new file stays empty, the renamed one grows without bound, and `logging.max_size_mb` bounds nothing on disk.
If an external rotator has to own the file, set `logging.output = .stderr` and
let the collector capture the stream instead.
If an external rotator has to own the file, set `logging.output = .stderr` and let the collector capture the stream instead.
The log file is the one path here nxdns does not set a mode on. `config.db` and
`querylog.db` are chmodded to 0600 whatever the umask; the log file is left at
whatever the umask gives it. Under the shipped unit that is 0600, because
`deploy/systemd/nxdns.service` sets `UMask=0077`. Run from a shell with the
usual `umask 022` it is 0644, and `LogsDirectory=nxdns` leaves the directory at
systemd's default 0755, so a log written there is readable by any local user.
The log file is the one path here nxdns does not set a mode on. `config.db` and `querylog.db` are chmodded to 0600 whatever the umask; the log file is left at whatever the umask gives it. Under the shipped unit that is 0600, because `deploy/systemd/nxdns.service` sets `UMask=0077`. Run from a shell with the usual `umask 022` it is 0644, and `LogsDirectory=nxdns` leaves the directory at systemd's default 0755, so a log written there is readable by any local user.
With `logging.output = .stderr` or `.syslog` nxdns writes to stderr and creates
no file; under systemd the journal captures it.
With `logging.output = .stderr` or `.syslog` nxdns writes to stderr and creates no file; under systemd the journal captures it.
## Certificate and key files
`doh_server.cert_path` / `key_path` and `dot_server.cert_path` / `key_path`,
conventionally under `/etc/nxdns`. nxdns reads them, never writes or creates
them. Both must be readable by the user nxdns runs as, and the key must belong
to the certificate: `nxdns check` loads the pair and fails when it does not. The
key should also be readable by its owner only, which `check` warns about when it
is not. A
watcher polls both files and swaps a renewed pair in without a restart. See
[enable DoH and DoT](../how-to/enable-doh-and-dot.md).
`doh_server.cert_path` / `key_path` and `dot_server.cert_path` / `key_path`, conventionally under `/etc/nxdns`. nxdns reads them, never writes or creates them. Both must be readable by the user nxdns runs as, and the key must belong to the certificate: `nxdns check` loads the pair and fails when it does not. The key should also be readable by its owner only, which `check` warns about when it is not. A watcher polls both files and swaps a renewed pair in without a restart. See [enable DoH and DoT](../how-to/enable-doh-and-dot.md).
docs/reference/performance.md
+10 -37
View File
@@ -1,9 +1,6 @@
# Performance reference
The PLAN §18 targets and the numbers measured against them. Why the targets are
these targets, and why CI does not gate on them, is [performance and
testing](../explanation/performance-and-testing.md). To reproduce the numbers,
see [measure performance](../how-to/measure-performance.md).
The PLAN §18 targets and the numbers measured against them. Why the targets are these targets, and why CI does not gate on them, is [performance and testing](../explanation/performance-and-testing.md). To reproduce the numbers, see [measure performance](../how-to/measure-performance.md).
## Targets (PLAN §18)
@@ -15,25 +12,15 @@ see [measure performance](../how-to/measure-performance.md).
| Memory with ~1M blocked domains < 100 MiB | `bench filter`: VmRSS with the 1M-domain snapshot loaded |
| Stripped static binary ≤ 10,485,760 bytes per arch without the embedded frontend, ≤ 15,728,640 bytes with it | `zig build verify-dist`, run by the `package` gate and by the release |
The harness is `tools/bench.zig`. It measures the three targets that are
measurable in process; the qps target is end to end and the binary-size target
belongs to the packaging step.
The harness is `tools/bench.zig`. It measures the three targets that are measurable in process; the qps target is end to end and the binary-size target belongs to the packaging step.
The two size budgets are exact byte counts, not rounded mebibytes, because an
assert on a rounded number is an assert on a number nobody wrote down.
`verify-dist` checks the shipped binary against the larger budget and builds a
second time against a generated empty assets directory for the smaller one, so
the asset-free figure is a real measurement rather than an estimate.
The two size budgets are exact byte counts, not rounded mebibytes, because an assert on a rounded number is an assert on a number nobody wrote down. `verify-dist` checks the shipped binary against the larger budget and builds a second time against a generated empty assets directory for the smaller one, so the asset-free figure is a real measurement rather than an estimate.
## Measured: x86_64 development host
Date: 2026-08-13. Hardware and build: Intel Core i7-14700K, Linux 6.18,
Zig 0.16.0, `-Doptimize=ReleaseFast`, harness defaults (1,000,000 domains,
200,000 iterations per suite, seed 0x5eed).
Date: 2026-08-13. Hardware and build: Intel Core i7-14700K, Linux 6.18, Zig 0.16.0, `-Doptimize=ReleaseFast`, harness defaults (1,000,000 domains, 200,000 iterations per suite, seed 0x5eed).
This is not the target platform. The Pi 5's Cortex-A76 is far slower and these
numbers do not transfer; they establish that the harness works and set a
baseline for regressions on the machine development happens on.
This is not the target platform. The Pi 5's Cortex-A76 is far slower and these numbers do not transfer; they establish that the harness works and set a baseline for regressions on the machine development happens on.
```
suite ops p50(us) p95(us) p99(us) max(us)
@@ -47,27 +34,15 @@ cache 200000 0.11 0.17 0.22 5.40
compile 1000000 wall 98.597ms, 10142224 lines/s, 1000000 domains kept (informational)
```
Every in-process §18 target passes on this host: the filter target by about
360x, the cache target by about four orders of magnitude, the memory target by
about 3x.
Every in-process §18 target passes on this host: the filter target by about 360x, the cache target by about four orders of magnitude, the memory target by about 3x.
The filter suite loads 32 regex rules that no query in the mix matches, which is
the expensive case rather than the cheap one: the regex levels sit below every
hash and wildcard level, so a name no pattern matches is the name that runs all
32 programs to their end. Every op pays that, which is what moved the filter p95
from 0.18 µs before regex rules existed to the 2.80 µs above. The margin against
the 1 ms target is what makes paying it on every miss an acceptable price.
The filter suite loads 32 regex rules that no query in the mix matches, which is the expensive case rather than the cheap one: the regex levels sit below every hash and wildcard level, so a name no pattern matches is the name that runs all 32 programs to their end. Every op pays that, which is what moved the filter p95 from 0.18 µs before regex rules existed to the 2.80 µs above. The margin against the 1 ms target is what makes paying it on every miss an acceptable price.
### The two memory figures
`Snapshot.memoryBytes` and `DnsCache.memoryBytes` are the in-repo accounting of
the structures themselves, which is the regression guard. VmRSS is what the
kernel holds resident for the whole process, allocator slack and code included.
The truth sits between them, and the §18 memory target is judged on VmRSS.
`Snapshot.memoryBytes` and `DnsCache.memoryBytes` are the in-repo accounting of the structures themselves, which is the regression guard. VmRSS is what the kernel holds resident for the whole process, allocator slack and code included. The truth sits between them, and the §18 memory target is judged on VmRSS.
The filter suite frees the generated list source before reading VmRSS, so its
number reflects the loaded snapshot rather than the generator. The cache suite's
VmRSS is lower because the filter suite's snapshot has been freed by then.
The filter suite frees the generated list source before reading VmRSS, so its number reflects the loaded snapshot rather than the generator. The cache suite's VmRSS is lower because the filter suite's snapshot has been freed by then.
## Raspberry Pi 5 (target platform)
@@ -78,6 +53,4 @@ VmRSS is lower because the filter suite's snapshot has been freed by then.
| Memory with ~1M blocked domains < 100 MiB | to be measured on hardware |
| Sustained ≥ 100 qps | to be measured on hardware, end to end |
The qps target belongs to the real binary rather than the harness: it means
`nxdns run` on the Pi, driven over the LAN by a DNS load generator against real
blocklists.
The qps target belongs to the real binary rather than the harness: it means `nxdns run` on the Pi, driven over the LAN by a DNS load generator against real blocklists.
docs/tutorial/first-run.md
+36 -139
View File
@@ -1,32 +1,19 @@
# Your first nxdns
This page takes you from a checkout of the repository to a running nxdns that
answers DNS queries, blocks domains from a real blocklist, and shows you the web
interface. It runs on unprivileged ports in a scratch directory, so nothing on
your machine changes and nothing needs root. At the end you stop the server and
delete the directory.
This page takes you from a checkout of the repository to a running nxdns that answers DNS queries, blocks domains from a real blocklist, and shows you the web interface. It runs on unprivileged ports in a scratch directory, so nothing on your machine changes and nothing needs root. At the end you stop the server and delete the directory.
Follow the steps in order. Each one says what it did.
Every command below was executed on x86_64 Linux with Zig 0.16.0, Node.js
24.14.1, dig 9.20.26 and curl 8.21.0. Steps 2, 4 to 11, 13 and 14 were re-run
end to end for this revision, and the transcripts are that run's output with the
tutorial directory substituted. Two things were not re-run: the browser page in
step 12 — its endpoints were exercised, the page itself was not opened — and the
`npm` build in step 1, whose `web/dist` was already on disk and is the one the
binary under test embeds. The ZON block at the end of step 14 was checked with
`nxdns check --config` rather than started.
Every command below was executed on x86_64 Linux with Zig 0.16.0, Node.js 24.14.1, dig 9.20.26 and curl 8.21.0. Steps 2, 4 to 11, 13 and 14 were re-run end to end for this revision, and the transcripts are that run's output with the tutorial directory substituted. Two things were not re-run: the browser page in step 12 — its endpoints were exercised, the page itself was not opened — and the `npm` build in step 1, whose `web/dist` was already on disk and is the one the binary under test embeds. The ZON block at the end of step 14 was checked with `nxdns check --config` rather than started.
## What you need
- Zig 0.16.0 and Node.js 24, both on your `PATH`.
- `dig` (from bind-tools or dnsutils) and `curl`.
- Working internet access: nxdns queries an upstream resolver over HTTPS and
downloads a 3 MB blocklist.
- Working internet access: nxdns queries an upstream resolver over HTTPS and downloads a 3 MB blocklist.
- Two terminals. The server runs in the first one; you type into the second.
Build commands run from the repository root. The server and the queries use
`~/nxdns-tutorial` for everything they write.
Build commands run from the repository root. The server and the queries use `~/nxdns-tutorial` for everything they write.
## 1. Build the web interface
@@ -34,9 +21,7 @@ Build commands run from the repository root. The server and the queries use
cd web && npm ci && npm run build && cd ..
```
This produces `web/dist`. Do not skip it. A plain `zig build` embeds
`web/dist-placeholder`, a one-page status stub, and you would reach step 12 and
find no admin interface there.
This produces `web/dist`. Do not skip it. A plain `zig build` embeds `web/dist-placeholder`, a one-page status stub, and you would reach step 12 and find no admin interface there.
## 2. Build nxdns
@@ -44,9 +29,7 @@ find no admin interface there.
zig build -Dweb-dist=web/dist
```
The binary is `zig-out/bin/nxdns`, with `web/dist` embedded in it. SQLite and
mbedTLS are vendored and built by this command, so the first build takes a
while; later builds are cached.
The binary is `zig-out/bin/nxdns`, with `web/dist` embedded in it. SQLite and mbedTLS are vendored and built by this command, so the first build takes a while; later builds are cached.
## 3. Write a configuration file
@@ -68,22 +51,11 @@ EOF
Four things are set, and the rest of nxdns keeps its defaults.
DNS is on port 15353 instead of 53, and both listeners are bound to localhost.
Port 53 is privileged: binding it needs root or `CAP_NET_BIND_SERVICE`, which is
an install decision, not a first-run decision. A real install is covered in
[how-to/install-with-systemd.md](../how-to/install-with-systemd.md).
DNS is on port 15353 instead of 53, and both listeners are bound to localhost. Port 53 is privileged: binding it needs root or `CAP_NET_BIND_SERVICE`, which is an install decision, not a first-run decision. A real install is covered in [how-to/install-with-systemd.md](../how-to/install-with-systemd.md).
The `default` group and one enabled upstream are the two things nxdns will not
start without. Every client that nxdns has never seen is assigned to `default`,
and with no usable upstream there is nowhere to send a query it cannot answer
itself, so a configuration missing either one is rejected. Whichever command
reads the file says the same thing and stops the same way: `run`, `nxdns check`
and `nxdns import` all print the problem and exit 2.
The `default` group and one enabled upstream are the two things nxdns will not start without. Every client that nxdns has never seen is assigned to `default`, and with no usable upstream there is nowhere to send a query it cannot answer itself, so a configuration missing either one is rejected. Whichever command reads the file says the same thing and stops the same way: `run`, `nxdns check` and `nxdns import` all print the problem and exit 2.
This tutorial loads the file into the database once and then runs nxdns against
the database, which is what the packaged systemd unit does. There is a second
way to run nxdns, where the file itself stays the configuration; step 14 shows
what changes.
This tutorial loads the file into the database once and then runs nxdns against the database, which is what the packaged systemd unit does. There is a second way to run nxdns, where the file itself stays the configuration; step 14 shows what changes.
## 4. Check the configuration before starting
@@ -97,10 +69,7 @@ OK upstreams[0] https://cloudflare-dns.com
OK: no problems found
```
`check` parses the file, validates it, and contacts each upstream to confirm it
answers. It exits 2 when it found something that has to be fixed and 0
otherwise. It writes nothing and starts no listener, so you can run it as often
as you like.
`check` parses the file, validates it, and contacts each upstream to confirm it answers. It exits 2 when it found something that has to be fixed and 0 otherwise. It writes nothing and starts no listener, so you can run it as often as you like.
## 5. Load it into the database
@@ -113,9 +82,7 @@ info(migrations): config.db migrated from schema version 0 to 1
imported /home/you/nxdns-tutorial/config.zon
```
The data directory did not exist; nxdns created it at mode 0700 along with
`config.db`. From here the database holds the configuration, and you will change
it through the API rather than by editing the file again.
The data directory did not exist; nxdns created it at mode 0700 along with `config.db`. From here the database holds the configuration, and you will change it through the API rather than by editing the file again.
## 6. Start the server
@@ -133,9 +100,7 @@ info(nxdns): nxdns <version> serving on udp [::1]:15353 udp 127.0.0.1:15353 tcp
info(web_server): web interface listening on 127.0.0.1:8080
```
No `--config` here, and that is the point: `authority: database` says the
database is the configuration and no file was opened at all. The file you wrote
in step 3 has done its job.
No `--config` here, and that is the point: `authority: database` says the database is the configuration and no file was opened at all. The file you wrote in step 3 has done its job.
Leave this terminal running and switch to the second one.
@@ -150,10 +115,7 @@ example.com. 229 IN A 172.66.147.243
example.com. 229 IN A 104.20.23.154
```
nxdns had no answer cached, so it forwarded the query to
`https://cloudflare-dns.com/dns-query` over HTTPS and returned what came back.
You now have a working resolver. It blocks nothing yet: the log line in step 6
said `0 of 0 sources loaded`.
nxdns had no answer cached, so it forwarded the query to `https://cloudflare-dns.com/dns-query` over HTTPS and returned what came back. You now have a working resolver. It blocks nothing yet: the log line in step 6 said `0 of 0 sources loaded`.
## 8. Add a blocklist source
@@ -167,25 +129,15 @@ curl -s -X POST http://127.0.0.1:8080/api/blocklists \
{"id":1,"url":"https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts","name":"StevenBlack hosts","enabled":true,"is_suggested":false}
```
The API is unauthenticated here because the configuration set no `web.password`.
That is fine for a localhost tutorial and wrong for anything else; see
[how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
The API is unauthenticated here because the configuration set no `web.password`. That is fine for a localhost tutorial and wrong for anything else; see [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md).
Note the `"id":1` in the response. You need it in the next step.
## 9. Attach the source to the `default` group
A blocklist source belongs to the installation. Which groups use it is a
separate decision, which is what lets one group get a strict list and another
get none. A source that is attached to no group is downloaded and then filters
nothing.
A blocklist source belongs to the installation. Which groups use it is a separate decision, which is what lets one group get a strict list and another get none. A source that is attached to no group is downloaded and then filters nothing.
You are in that state right now, between the previous step and this one, and it
is legal rather than wrong — creating a source and attaching it afterwards is
the normal order, which is why the API accepted it without complaint. On a
stopped server `nxdns check` names such a source in a `WARN` line and still
exits 0, so a list that silently blocks nothing is something you can find out
about later. Attaching it now is what makes it take effect.
You are in that state right now, between the previous step and this one, and it is legal rather than wrong — creating a source and attaching it afterwards is the normal order, which is why the API accepted it without complaint. On a stopped server `nxdns check` names such a source in a `WARN` line and still exits 0, so a list that silently blocks nothing is something you can find out about later. Attaching it now is what makes it take effect.
Ask which groups exist:
@@ -197,9 +149,7 @@ curl -s http://127.0.0.1:8080/api/groups
{"groups":[{"id":1,"name":"default","safe_search":false}]}
```
The `default` group is id 1. Give it the source you just created — the request
body is the complete set of sources for that group, so sending `[1]` replaces
whatever was there:
The `default` group is id 1. Give it the source you just created — the request body is the complete set of sources for that group, so sending `[1]` replaces whatever was there:
```sh
curl -s -X PUT http://127.0.0.1:8080/api/groups/1/sources \
@@ -223,30 +173,17 @@ curl -s -X POST http://127.0.0.1:8080/api/blocklists/update
{"sources":[{"id":1,"state":"ok","loaded":true,"last_attempt":1786629237,"last_success":1786629238,"url":"https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts","last_error":"","domains":97648,"wildcards":0,"exceptions":0,"skipped_regex":0,"skipped_unsupported":0}]}
```
The four zeros describe this particular download, not the hosts format.
`wildcards` counts entries covering a name and its subdomains, `exceptions`
counts the `@@` lines an Adblock Plus list uses to lift a name another list
blocks, and `skipped_regex` counts the regex lines nxdns declines to take from a
downloaded list. `skipped_unsupported` counts lines nxdns cannot translate into a
DNS decision, and a large value next to a small `domains` means the list targets
browsers rather than DNS. Only `exceptions` is Adblock-Plus-only: a hosts list
can carry regex lines, `*.`-prefixed wildcards, and bare sink addresses that
count as unsupported. This one carries none of them.
The four zeros describe this particular download, not the hosts format. `wildcards` counts entries covering a name and its subdomains, `exceptions` counts the `@@` lines an Adblock Plus list uses to lift a name another list blocks, and `skipped_regex` counts the regex lines nxdns declines to take from a downloaded list. `skipped_unsupported` counts lines nxdns cannot translate into a DNS decision, and a large value next to a small `domains` means the list targets browsers rather than DNS. Only `exceptions` is Adblock-Plus-only: a hosts list can carry regex lines, `*.`-prefixed wildcards, and bare sink addresses that count as unsupported. This one carries none of them.
The download is about 3 MB and takes a few seconds. Watch the first terminal
until this appears:
The download is about 3 MB and takes a few seconds. Watch the first terminal until this appears:
```
info(blocklist_manager): blocklist snapshot generation 6: 1 of 1 sources loaded, 2523973 bytes
```
`1 of 1 sources loaded` is the line to wait for. nxdns builds each blocklist
snapshot in full and swaps it in atomically, so queries keep being answered from
the previous snapshot the whole time the new one is being built. After this, the
domain count in the JSON above — 97648 on the day this was run — is live.
`1 of 1 sources loaded` is the line to wait for. nxdns builds each blocklist snapshot in full and swaps it in atomically, so queries keep being answered from the previous snapshot the whole time the new one is being built. After this, the domain count in the JSON above — 97648 on the day this was run — is live.
From here on, the list is on disk under `~/nxdns-tutorial/data/blocklists`.
Restarting nxdns does not re-download it.
From here on, the list is on disk under `~/nxdns-tutorial/data/blocklists`. Restarting nxdns does not re-download it.
## 11. Watch a domain get blocked
@@ -258,9 +195,7 @@ dig @127.0.0.1 -p 15353 doubleclick.net A +noall +answer
doubleclick.net. 5 IN A 0.0.0.0
```
`0.0.0.0` with a TTL of 5 is a block, not an answer. The default block response
is the zero address and the default block TTL is 5 seconds, so a client that
caches the answer forgets it quickly after you unblock something.
`0.0.0.0` with a TTL of 5 is a block, not an answer. The default block response is the zero address and the default block TTL is 5 seconds, so a client that caches the answer forgets it quickly after you unblock something.
A name that is not on the list still resolves normally:
@@ -272,23 +207,11 @@ dig @127.0.0.1 -p 15353 wikipedia.org A +noall +answer
wikipedia.org. 130 IN A 185.15.58.224
```
One thing to know before you try other names: an entry in a hosts list blocks
exactly the name it names. `doubleclick.net` is on this list, and so are
`ad.doubleclick.net` and `www.google-analytics.com`. `ads.doubleclick.net` is
not on it, and nxdns does not block it — the entry for the parent says nothing
about the child. Two things do walk up the parent chain, and neither is in play
here: a rule you write yourself, with a wildcard pattern such as
`*.doubleclick.net`, and an Adblock Plus list's `||doubleclick.net^`, which
covers the name and everything under it. This list is a hosts file, so when you
pick a domain to test against it, pick one that is literally in the file.
One thing to know before you try other names: an entry in a hosts list blocks exactly the name it names. `doubleclick.net` is on this list, and so are `ad.doubleclick.net` and `www.google-analytics.com`. `ads.doubleclick.net` is not on it, and nxdns does not block it — the entry for the parent says nothing about the child. Two things do walk up the parent chain, and neither is in play here: a rule you write yourself, with a wildcard pattern such as `*.doubleclick.net`, and an Adblock Plus list's `||doubleclick.net^`, which covers the name and everything under it. This list is a hosts file, so when you pick a domain to test against it, pick one that is literally in the file.
## 12. Open the web interface
Visit <http://127.0.0.1:8080> in a browser. This is the single-page application
you built in step 1, served out of the binary. The dashboard shows query and
block counts, and the Blocklists page shows the source you added with its
domain count. (The endpoints behind those two pages were checked while writing
this; the browser page itself was not opened on the verification host.)
Visit <http://127.0.0.1:8080> in a browser. This is the single-page application you built in step 1, served out of the binary. The dashboard shows query and block counts, and the Blocklists page shows the source you added with its domain count. (The endpoints behind those two pages were checked while writing this; the browser page itself was not opened on the verification host.)
## 13. Stop it
@@ -307,18 +230,13 @@ info(blocklist_manager): blocklist snapshot generation 1: 1 of 1 sources loaded,
info(nxdns): authority: database
```
The blocklist came off disk rather than the network. The source you added, the
group it is attached to, and the query log all survived the restart because they
live in `~/nxdns-tutorial/data`, which is what `authority: database` means in
practice.
The blocklist came off disk rather than the network. The source you added, the group it is attached to, and the query log all survived the restart because they live in `~/nxdns-tutorial/data`, which is what `authority: database` means in practice.
Press Ctrl-C again to stop this second process. Nothing is listening on 15353 or
8080 now, and nothing of nxdns is running.
Press Ctrl-C again to stop this second process. Nothing is listening on 15353 or 8080 now, and nothing of nxdns is running.
## 14. See what the other mode does
You have been running in database mode. The alternative is to hand the same file
back as the configuration, which is what `--config` means:
You have been running in database mode. The alternative is to hand the same file back as the configuration, which is what `--config` means:
```sh
zig-out/bin/nxdns run --data-dir ~/nxdns-tutorial/data --config ~/nxdns-tutorial/config.zon
@@ -334,23 +252,11 @@ info(blocklist_manager): pruned orphaned blocklist file 1.wild
info(blocklist_manager): pruned orphaned blocklist file 1.list
```
**Read that first line.** The blocklist source is gone. That is not a bug — it is
the whole contract. The file you wrote in step 3 never mentioned a blocklist
source, and in file mode the file is the complete statement of what the
configuration is, so anything the database holds that the file does not name is
removed at every start. The reconcile said so in one line before doing it.
**Read that first line.** The blocklist source is gone. That is not a bug — it is the whole contract. The file you wrote in step 3 never mentioned a blocklist source, and in file mode the file is the complete statement of what the configuration is, so anything the database holds that the file does not name is removed at every start. The reconcile said so in one line before doing it.
The three `pruned` lines are the rest of that removal: with the row gone, the
compiled files it owned belong to nobody, so the sweep that runs at every start
deletes them. The three names are the three bodies one source compiles into —
exact domains, wildcards, and the exceptions an Adblock Plus list can lift. The
query log is untouched; what changed is the configuration, and it now matches
the file exactly.
The three `pruned` lines are the rest of that removal: with the row gone, the compiled files it owned belong to nobody, so the sweep that runs at every start deletes them. The three names are the three bodies one source compiles into — exact domains, wildcards, and the exceptions an Adblock Plus list can lift. The query log is untouched; what changed is the configuration, and it now matches the file exactly.
Neither mode is the "advanced" one. Database mode suits a box someone
administers through the web interface. File mode suits a file kept in git and
deployed by a tool, where the deployed file being what is running matters more
than clicking. To have kept the blocklist here, you would put it in the file:
Neither mode is the "advanced" one. Database mode suits a box someone administers through the web interface. File mode suits a file kept in git and deployed by a tool, where the deployed file being what is running matters more than clicking. To have kept the blocklist here, you would put it in the file:
```zon
.blocklist_sources = .{
@@ -365,26 +271,17 @@ Ctrl-C to stop it.
## What you have now
A resolver that answers real queries, a real blocklist of about 98000 domains
attached to the default group, a query log, and a web interface — all inside one
directory you can delete:
A resolver that answers real queries, a real blocklist of about 98000 domains attached to the default group, a query log, and a web interface — all inside one directory you can delete:
```sh
rm -rf ~/nxdns-tutorial
```
You also saw the three things that surprise people most: a blocklist source
does nothing until a group uses it, which authority a start runs under is
printed rather than guessed, and in file mode anything the file does not name is
removed at the next start.
You also saw the three things that surprise people most: a blocklist source does nothing until a group uses it, which authority a start runs under is printed rather than guessed, and in file mode anything the file does not name is removed at the next start.
## Where to go next
- [how-to/install-with-systemd.md](../how-to/install-with-systemd.md) — the same
thing as a real service on port 53, including a Raspberry Pi 5.
- [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md)
— put a password on the web interface before it leaves localhost.
- [reference/configuration.md](../reference/configuration.md) — every field you
did not set.
- [explanation/configuration-model.md](../explanation/configuration-model.md) —
why there are two authority modes and what each one is for.
- [how-to/install-with-systemd.md](../how-to/install-with-systemd.md) — the same thing as a real service on port 53, including a Raspberry Pi 5.
- [how-to/set-up-admin-authentication.md](../how-to/set-up-admin-authentication.md) — put a password on the web interface before it leaves localhost.
- [reference/configuration.md](../reference/configuration.md) — every field you did not set.
- [explanation/configuration-model.md](../explanation/configuration-model.md) — why there are two authority modes and what each one is for.