milestone 13 discrepancies: redact credentials from urls in logs, metrics and cli output

docs/explanation/configuration-model.md

@@ -59,11 +59,35 @@ time you look at the UI, rather than "my edit was applied and then quietly
 undone next Tuesday".
 
 The emptiness check is a real query over the content tables, not a flag: the
-database counts as configured when any content table has rows, or when the
-default group has been altered. One consequence is worth knowing, because it
-is not obvious: clients are auto-materialised when they first send a query, so
-a server that has answered even one query is "configured" and will ignore a
-seed file placed there afterwards.
+database counts as configured when a content table holds rows, or when the
+default group has been altered. What it measures is intent, not activity, and
+the client table is where those two come apart. Clients are auto-materialised
+when they first send a query — the DNS path writes a row per device it sees —
+and such a row records what the network did, not what an operator decided. It
+carries `hand_edited = 0`, the emptiness check counts only the `hand_edited = 1`
+rows, and `export` omits them. So answering queries never turns an unconfigured
+database into a configured one; naming a device does.
+
+Counting traffic here would have been a quiet trap: a server that resolved one
+name would have declared itself configured and ignored a seed file placed
+afterwards, and the operator would have had no line of output saying why.
+
+The same distinction survives an import. `import` replaces the content in one
+transaction, which empties `clients` along with every other table, so every
+client row is saved before the wipe. The materialised ones are put back after it
+unchanged, and restoring a backup does not make the server forget the devices it
+has met.
+
+An address the imported file names belongs to the file — the operator's
+statement wins over the discovered row — with one carve-out. First-seen and
+last-seen are not configuration: they record when a device was heard from, the
+configuration model has no field for either, and an import is not a query. So
+they follow the address rather than the row. If the database already knew that
+address, its two timestamps are carried onto the new row; only an address the
+database has never seen takes the import's clock. Without that, re-importing a
+backup would stamp every device the operator had bothered to name as though it
+had just arrived — and those are exactly the devices whose history is worth
+something.
 
 ## The round trip
 
@@ -90,11 +114,17 @@ difference is a configuration change, never noise from when the export ran.
 
 `nxdns import` replaces the whole database content in one transaction. Not a
 merge, not a patch: delete every content table in foreign-key-safe order, then
-insert what the file says. A failed import — bad syntax, failed validation, a
-constraint violation halfway through — leaves the database exactly as it was,
-because everything happens inside a single `BEGIN IMMEDIATE`.
+insert what the file says — with the one exception described above. Every client
+row is lifted out first. The materialised ones are put back rather than
+recreated from a file that never held them, and the observed timestamps of an
+address the file *does* name are merged onto its new row. A client the file
+leaves out is gone, history included: nothing puts a `hand_edited = 1` row back.
+A failed import — bad syntax, failed
+validation, a constraint violation halfway through — leaves the database exactly
+as it was, because everything happens inside a single `BEGIN IMMEDIATE`.
 
-Without `--force`, import refuses a database that already has content. That
+Without `--force`, import refuses a database that already holds configuration.
+That
 check runs *inside* the transaction, after the lock is taken, so it cannot be
 raced by a concurrent write. The effect is that a plain `import` can never
 clobber a configured server by accident, and clobbering it deliberately takes

docs/explanation/performance-and-testing.md

@@ -15,8 +15,8 @@ PLAN §18 sets five:
 - sustained ≥ 100 qps on a Raspberry Pi 5;
 - blocklist lookup p95 < 1 ms;
 - cached response p95 < 5 ms;
-- memory with ~1M blocked domains < 100 MB;
-- stripped static binary < 10 MB per arch, < 15 MB with the embedded frontend.
+- memory with ~1M blocked domains < 100 MiB;
+- stripped static binary < 10 MiB per arch, < 15 MiB with the embedded frontend.
 
 They are household-scale numbers, and they are deliberately unambitious. 100
 qps is far more than a house generates; the point of the target is not speed

docs/how-to/back-up-and-restore.md

@@ -88,8 +88,8 @@ before writing to it.
 
 ## Restore over an existing database
 
-`import` refuses a database that already has content, so a plain `import` can
-never clobber a configured server by accident:
+`import` refuses a database that already holds configuration, so a plain
+`import` can never clobber a configured server by accident:
 
 ```sh
 nxdns import /tmp/nxdns-lab/backup.zon --data-dir /tmp/nxdns-lab/data
@@ -112,6 +112,13 @@ imported /tmp/nxdns-lab/backup.zon
 Stop the server first. `import` replaces the whole configuration underneath a
 process that has already read it, and a running server will not notice.
 
+What `--force` does to the client list is worth knowing before you restore an
+old backup. A client the backup does not name is removed, and its first-seen
+and last-seen go with it — restoring a month-old file drops the devices you
+named since. Devices the server discovered from traffic are kept, and a device
+the backup does name keeps the first-seen and last-seen the database already
+held, so a restore does not restamp your whole network as newly arrived.
+
 On a real install that is the systemd unit:
 
 ```sh
@@ -185,8 +192,9 @@ The data directory layout is in
 ## A backup before every upgrade
 
 There is no downgrade path. Schema migrations run forward automatically at
-startup and before `check`, `export` and `import`; nothing walks them back. Take
-an export before installing a new binary — see [upgrade](upgrade.md).
+startup and before `export` and `import`; nothing walks them back, and `nxdns
+check` does not run them at all. Take an export before installing a new binary —
+see [upgrade](upgrade.md).
 
 Every command on this page was executed on this host as written, except the
 `systemctl` block marked **Not verified on this host** above.

docs/how-to/enable-doh-and-dot.md

@@ -75,23 +75,40 @@ nxdns check --data-dir /tmp/nxdns-lab/data --config /tmp/nxdns-lab/etc/config.zo
 
 ```
 checking configuration file /tmp/nxdns-lab/etc/config.zon
-OK https://cloudflare-dns.com/dns-query
+OK upstreams[0] https://cloudflare-dns.com
 OK: no problems found
 ```
 
-`check` reads both endpoints' certificate and key. An unreadable file is a
-failure and exits 2:
+`check` loads both endpoints' certificate and key the same way the listeners do,
+so what passes here will start. An unreadable file is a failure and exits 2:
 
 ```
-FAIL doh_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem' is not readable
-FAIL dot_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem' is not readable
+FAIL doh_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable
+FAIL dot_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable
 ```
 
-A key readable by anyone but its owner is a warning, and does not change the
-exit code, because the service still starts:
+So is a key that does not belong to the certificate, which is the mistake worth
+catching before a restart — the two files are individually valid and only their
+pairing is wrong. mbedTLS writes its own line to stderr as it rejects the pair:
+
+```
+warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
+warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
+checking configuration file /tmp/nxdns-lab/etc/mismatch.zon
+FAIL doh_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not belong to the certificate
+FAIL dot_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not belong to the certificate
+OK upstreams[0] https://cloudflare-dns.com
+```
+
+A key readable by anyone but its owner is a warning instead. It does not change
+the exit code, because the service still starts, and the summary line counts it
+rather than claiming nothing was found:
 
 ```
 WARN doh_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only
+WARN dot_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only
+OK upstreams[0] https://cloudflare-dns.com
+OK: no failures found, 2 warnings
 ```
 
 ## 4. Start and confirm the listeners

docs/how-to/install-with-docker.md

@@ -73,13 +73,28 @@ database has nothing to forward to. The log is
 `no configuration file at '/etc/nxdns/config.zon'; using the database as it is`
 followed by `nxdns run failed: NoUsableUpstreams`.
 
-A file that is present but rejected is a different failure and a different exit
+A file that is present but rejected is a different failure with the same exit
 code. No `default` group, no enabled upstream, a syntax error — `run` prints the
-diagnostic and exits **1**, for example `nxdns run failed: MissingDefaultGroup`.
-Both were run here against this image: a seed file whose only group was named
-`other` exited 1, and an empty `/etc/nxdns` exited 2 with `NoUsableUpstreams`.
-Under `restart: unless-stopped` either one is a restart loop, so read the exit
-code from `docker inspect` to tell them apart; see
+diagnostic and exits 2 as well. Both were run here against this image. A seed
+file whose only group was named `other`:
+
+```
+FAIL groups: no group named 'default'; every unknown client is assigned to it
+nxdns run failed: MissingDefaultGroup
+run `nxdns check` to see the configuration in full
+```
+
+and an empty `/etc/nxdns`:
+
+```
+info(config_bootstrap): no configuration file at '/etc/nxdns/config.zon'; using the database as it is
+nxdns run failed: NoUsableUpstreams
+run `nxdns check` to see the configuration in full
+```
+
+Under `restart: unless-stopped` either one is a restart loop, and the exit code
+alone no longer tells them apart: read the lines above the failure, which either
+name the diagnostic in the file or say there was no file at all. See
 [Troubleshoot nxdns](troubleshoot.md).
 
 The container runs as uid 65532, and the mount is read-only, so the container

docs/how-to/install-with-systemd.md

@@ -152,7 +152,7 @@ A good file prints the source it checked, one `OK` line per upstream, and
 
 ```
 checking configuration file /etc/nxdns/config.zon
-OK https://cloudflare-dns.com/dns-query
+OK upstreams[0] https://cloudflare-dns.com
 OK: no problems found
 ```
 

docs/how-to/set-up-admin-authentication.md

@@ -228,7 +228,7 @@ nxdns export --data-dir /tmp/nxdns-lab/data | grep password
         .password_hash = "$argon2id$v=19$m=19456,t=2,p=1$kvlRj1tdGul3MlfbvzLncLKWirNpJRJ3howFA9/ysgg$7elW7PPQ3WXHwI4YOmOpZ/1KNEQo7ZDLRJhnYOPMjqw",
 ```
 
-`--force` is required because the database already has content. See
+`--force` is required because the database already holds configuration. See
 [back up and restore](back-up-and-restore.md).
 
 ## What happens with no password set

docs/how-to/troubleshoot.md

@@ -21,8 +21,9 @@ nxdns run failed: NoUsableUpstreams
 run `nxdns check` to see the configuration in full
 ```
 
-Exit 2 is reserved for a small set of faults `run` raises itself:
-`NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit` and `BadCertificate`.
+Exit 2 means the configuration is wrong and you can fix it. Every subcommand
+uses the same definition, so a file `run` exits 2 on exits 2 from `check` and
+`import` too.
 
 **Diagnosis.**
 
@@ -45,18 +46,18 @@ checked on its first line.
   certificate. `run` names both paths before it exits:
   `doh_server: '<cert>' + '<key>': certificate file is not readable`.
 
-  **`check` does not catch most of this.** It tests only that each file is
-  readable, and warns when the key is readable beyond its owner; it never opens
-  the PEM. Parsing and the key/certificate pairing happen when `run` builds the
-  TLS context, so `check` can print `OK: no problems found` on a configuration
-  `run` then refuses. Reproduced here with a self-signed pair and the key from a
-  second, unrelated pair:
+  `check` catches this without starting a listener. It loads both PEM files and
+  tests the key against the certificate through the same code `run` uses, so it
+  fails on exactly what `run` would fail on. Reproduced here with a self-signed
+  pair and the key from a second, unrelated pair:
 
   ```
   $ nxdns check --config config.zon
+  warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
   checking configuration file config.zon
-  OK https://cloudflare-dns.com/dns-query
-  OK: no problems found                                             # exit 0
+  FAIL doh_server.key_path: 'mismatched-key.pem': private key does not belong to the certificate
+  OK upstreams[0] https://cloudflare-dns.com
+                                                                    # exit 2
 
   $ nxdns run --config config.zon --data-dir ./data
   warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
@@ -64,59 +65,103 @@ checked on its first line.
   nxdns run failed: BadCertificate                                  # exit 2
   ```
 
-  A cert file containing `not a certificate` behaves the same way — `check`
-  exits 0, `run` exits 2 with `certificate PEM could not be parsed`. So a
-  successful `check` means the paths and permissions are right, not that the
-  certificate is usable; the only test of that is starting the service. Fix the
-  path, the ownership, or the pair; see
+  The `warning(tls_server)` line comes from mbedTLS on stderr and can appear
+  before the `checking` line, which is on stdout. A cert file containing
+  `not a certificate` fails the same way, with
+  `FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`.
+  An unreadable file reads
+  `FAIL doh_server.cert_path: '<path>': certificate file is not readable`.
+
+  Fix the path, the ownership, or the pair; see
   [Enable DoH and DoT](enable-doh-and-dot.md).
 - `BadRateLimit` — a rate limit or window is zero. `import` refuses such a
   configuration, so this only reaches a database that was edited by hand.
 - `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of
   that family.
 
-## The service exits with code 1 on a seed file you just wrote
+## A seed file you just wrote is rejected
 
 **Symptom.** A first start against an empty database prints the validation
-problem and stops, but with exit code 1, not 2:
+problem and stops with exit 2:
 
 ```
-groups: no group named 'default'; every unknown client is assigned to it
+FAIL groups: no group named 'default'; every unknown client is assigned to it
 nxdns run failed: MissingDefaultGroup
+run `nxdns check` to see the configuration in full
 ```
 
 A syntax error behaves the same way:
 
 ```
-config: 2:42: error: expected ',' after initializer
+FAIL config: 3:16: error: expected ',' after initializer
 nxdns run failed: ParseZon
+run `nxdns check` to see the configuration in full
 ```
 
 So does a seed file whose upstream list is empty or all disabled:
 
 ```
-upstreams: at least one upstream must be enabled
+FAIL upstreams: at least one upstream must be enabled
 nxdns run failed: NoUpstreams
+run `nxdns check` to see the configuration in full
 ```
 
 `NoUpstreams` from a seed file is not the same fault as `NoUsableUpstreams`
 above: the first is a file `run` refused, the second is a database `run`
-accepted and found empty. Only the second is exit 2.
+accepted and found empty. Both are exit 2.
 
-**Diagnosis.** Run the same file through `check`, which reports it as a
-configuration problem and exits 2:
+**Diagnosis.** Run the same file through `check`, which reports the same
+problems and exits 2:
 
 ```sh
 nxdns check --config /etc/nxdns/config.zon
 ```
 
 **Fix.** Correct the file the diagnostics name and start again. The database is
-still empty after a failed seed, so the next start re-reads the file. Note that
-`nxdns check` and `nxdns import` of the same bad file exit 2 while `nxdns run`
-exits 1 — the exit code differs by command, the diagnostics do not. All three
-commands were run here against a file missing its `default` group, one with a
-syntax error and one with no enabled upstream, and every pair came out that
-way.
+still empty after a failed seed, so the next start re-reads the file. The exit
+code no longer depends on which command read the file: all three of these files
+were run through `run`, `check` and `import` here, and every one of the nine
+combinations exited 2 with the same diagnostic.
+
+## `nxdns check` fails on a server that is running fine
+
+**Symptom.** The service is up and answering, but `nxdns check` on the same
+machine exits 2 with one long line about a write-ahead log:
+
+```
+checking database /var/lib/nxdns/config.db
+FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
+```
+
+Nothing is damaged. `check` opens `config.db` immutable so that it can never
+write to it, and an immutable open ignores the write-ahead log. When that log
+holds bytes, the newest settings are in it and the main file holds older ones,
+so `check` refuses rather than grade stale values.
+
+The log holds bytes after a configuration write that has not been checkpointed
+yet, which on a running server means someone changed something through the web
+interface or the API. A server that has only been answering queries has an empty
+`config.db-wal` and `check` reads it normally — so this line comes and goes, and
+its absence is not proof that nothing is running.
+
+**Fix.** Check the exported configuration instead of the live file:
+
+```sh
+nxdns export --data-dir /var/lib/nxdns --out /tmp/current.zon
+nxdns check --config /tmp/current.zon
+```
+
+`export` opens the database read/write and does see the log, so it renders the
+settings that are actually in force. Stopping the service and checking again
+works too: a clean shutdown checkpoints the log away.
+
+> Reproduced here on a scratch data directory rather than `/var/lib/nxdns` —
+> that path is the only substitution in the output above. nxdns was started on
+> unprivileged ports; `config.db-wal` was 0 bytes and `check` exited 0; one
+> `POST /api/blocklists` took it to 8272 bytes and `check` then printed the line
+> above and exited 2; `export` from the same live directory succeeded and its
+> output checked clean; and after a clean shutdown `check --data-dir` exited 0
+> again.
 
 ## Port 53 is already taken
 

docs/how-to/upgrade.md

@@ -108,43 +108,63 @@ configuration.
 
 ```sh
 nxdns version
-nxdns check
+nxdns export --out /tmp/after-upgrade.zon
+nxdns check --config /tmp/after-upgrade.zon
 dig @127.0.0.1 example.com A +short
 ```
 
-`nxdns check` with no `--config` checks the database, which is what you want
-after an upgrade — it names its source on the first line and migrates a
-database that is one schema version behind before checking it:
+The restart in step 3 is what migrated the database, so by now the schema is
+current and the service is answering. Confirming with `nxdns check` alone would
+not work here, and the reason is worth knowing: `check` opens `config.db`
+immutable so it can never write to it, and the migration you just performed is
+sitting in `config.db-wal` waiting to be checkpointed. Rather than read around
+the log and grade older settings, `check` reports it:
 
 ```
-checking database /var/lib/nxdns/config.db
-OK https://cloudflare-dns.com/dns-query
+FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
+```
+
+`export` opens the database read/write and does see the log, so exporting and
+then checking the export validates what is actually in force:
+
+```
+checking configuration file /tmp/after-upgrade.zon
+OK upstreams[0] https://cloudflare-dns.com
 OK: no problems found
 ```
 
-> Verified on this host for the first two commands, with `--data-dir` pointing
-> at the scratch data directory instead of `/var/lib/nxdns`:
+> Verified on this host for the first three commands, with `--data-dir`
+> pointing at a scratch data directory instead of `/var/lib/nxdns` — that path
+> is the only difference from the blocks above:
 >
 > ```
 > $ nxdns version
 > nxdns 0.1.0-dev (unknown)
 > zig 0.16.0
-> $ nxdns check --data-dir $SCRATCH/data
-> checking database /…/scratchpad/data/config.db
-> OK https://cloudflare-dns.com/dns-query
-> OK: no problems found
 > ```
 >
-> The database path on the first line is the only difference from the block
-> above. The `dig` line was not run in this round: nothing is listening on
-> 127.0.0.1:53 here, and port 53 needs root.
+> Against a running server whose database had just been migrated and seeded,
+> `nxdns check --data-dir` printed the uncheckpointed-log line above and exited
+> 2, while `nxdns export --out` followed by `nxdns check --config` on the result
+> exited 0 with `OK: no problems found`. The `dig` line was not run in this
+> round: nothing is listening on 127.0.0.1:53 here, and port 53 needs root.
 
 ## What happens to the database
 
-Migrations run at startup, and also before `check`, `export` and `import`, so
-whichever of those you run first performs the upgrade. A fresh database is
-created at the current schema version; an older one is stepped up to it. The
-log line names both versions:
+Migrations run at startup, and also before `export` and `import`, so whichever
+of those you run first performs the upgrade. `nxdns check` is the exception: it
+opens the database immutable and never migrates, so on a database still one
+version behind it reports the mismatch and exits 2 rather than fixing it:
+
+```
+FAIL /var/lib/nxdns/config.db: schema version 0, this nxdns expects 2; `nxdns run` migrates it, `check` will not
+```
+
+That line was reproduced here against a database stamped at version 0; the path
+and the version numbers are what vary.
+
+A fresh database is created at the current schema version; an older one is
+stepped up to it. The log line names both versions:
 
 ```
 info(migrations): config.db migrated from schema version 0 to 2
@@ -194,9 +214,11 @@ nxdns import config-backup.zon --force
 systemctl start nxdns
 ```
 
-`--force` is required here. A plain `import` into a database that already has
-content fails with `import failed: DatabaseNotEmpty` and exits 2, so it cannot
-clobber a configured server by accident.
+`--force` is required here. A plain `import` into a database that already holds
+configuration fails with `import failed: DatabaseNotEmpty` and exits 2, so it
+cannot clobber a configured server by accident. What counts is what an operator
+set: client rows the DNS path materialised from traffic never trigger the
+refusal on their own.
 
 > Verified on this host for the two `nxdns` lines, against a populated scratch
 > data directory:

docs/reference/cli.md

@@ -23,21 +23,24 @@ Serves DNS until SIGINT or SIGTERM.
 | `--config FILE` | Seed configuration file (default `/etc/nxdns/config.zon`). Read only when the database has never been configured. |
 | `--web-dev DIR` | Serve the web interface from DIR instead of the embedded assets, with no cache headers. Development only. |
 
-Seeding failures are not treated as configuration faults here: a seed file that
-is unparseable, oversized or invalid prints its diagnostics and exits 1, not 2.
-See [exit codes](#exit-codes).
+A seed file that is unparseable, oversized or invalid prints its diagnostics and
+exits 2 — the same code `check` and `import` give for the same file. See
+[exit codes](#exit-codes).
 
 ## `check`
 
-Validates the configuration and probes the upstreams. Exit 0 when clean, 2 when
-it found problems. It always reports every problem, not just the first. What it
-checks, in order:
+Validates the configuration and probes the upstreams. Exit 0 when it found no
+failures, 2 when it found one. It always reports every problem, not just the
+first. What it checks, in order:
 
 1. Which source to check (see [source selection](#source-selection)).
 2. Full validation — the same rules `import` enforces.
-3. For each enabled DoH/DoT listener: certificate and key are readable (FAIL if
-   not), and the key's permissions are owner-only (WARN if any group or other
-   bit is set; the WARN does not change the exit code).
+3. For each enabled DoH/DoT listener: both PEM files are read and the key is
+   tested against the certificate, through the same `CertStore.init` the
+   listeners boot with. A file that is missing, unreadable, too large or
+   unparseable, and a key that does not belong to the certificate, are all FAIL.
+   The key's permissions are a separate finding: WARN when any group or other
+   bit is set, which does not change the exit code.
 4. A live probe: one real A query for `example.com` through every enabled
    upstream, driving the same pool and failover machinery the server uses, with
    `upstream.total_timeout_ms` as the per-attempt deadline. A FAIL line names
@@ -50,13 +53,28 @@ checks, in order:
 | `--data-dir DIR` | Data directory to look for `config.db` in. |
 | `--config FILE` | Check this file instead of the database. |
 
+### Failures and warnings
+
+Every finding carries a severity. `FAIL` is a problem that sets exit 2. `WARN`
+is legal configuration that is almost certainly not what was meant — a blocklist
+source no group links to, a TLS key readable beyond its owner — and never
+changes an exit code, because the service starts either way.
+
+The last line is a summary, and it never contradicts the lines above it:
+
+| What was found | Last line | Exit |
+| --- | --- | --- |
+| Nothing | `OK: no problems found` | 0 |
+| Warnings only | `OK: no failures found, 2 warnings` — `1 warning` in the singular | 0 |
+| At least one failure | No summary line; the FAIL lines are the report | 2 |
+
 ### Source selection
 
 - `--config FILE` given explicitly: check that file, nothing else.
 - Otherwise, if `<data-dir>/config.db` exists: check the database — the right
-  default, since the database is the truth on a configured server. Pending
-  schema migrations are applied first, so a `check` immediately after an upgrade
-  works.
+  default, since the database is the truth on a configured server. It is opened
+  immutable, so `check` writes nothing to it; see
+  [what `check` does not do](#what-check-does-not-do).
 - Otherwise, if the default config file path exists: check it.
 - Otherwise: "nothing to check", exit 2.
 
@@ -64,10 +82,42 @@ The first line of output always names which source was checked. A file larger
 than 4 MiB fails with `larger than 4194304 bytes`; a ZON syntax error is
 reported with its line and column.
 
-A missing file is treated differently depending on how the path was reached. An
-explicit `--config FILE` that does not exist is an I/O failure: `check failed:
-FileNotFound` on stderr, exit 1. A default path that does not exist is one of
-the cases above and produces "nothing to check", exit 2.
+A named file that is missing or unreadable is a finding like any other, not an
+I/O failure that escapes the run: `FAIL <path>: no such file` or
+`FAIL <path>: not readable`, exit 2. That is the same code the implicit path
+gives when there is nothing to check, so naming the file does not change what a
+missing file costs.
+
+### What `check` does not do
+
+`check` never writes to `config.db`. It opens the file immutable, which means
+SQLite refuses every statement that would write and builds no write-ahead log,
+so no `config.db-wal` and no `config.db-shm` appear beside it. It does not chmod
+the file and it does not migrate the schema.
+
+Two consequences are worth knowing before you read a FAIL line as damage:
+
+- A database behind this binary's schema is reported, not upgraded. Start the
+  service to migrate it.
+
+  ```
+  FAIL <db>: schema version <n>, this nxdns expects <m>; `nxdns run` migrates it, `check` will not
+  ```
+
+- A database with an unapplied write-ahead log cannot be graded without writing,
+  because the newest settings are in the log and the main file holds older ones.
+  `check` says so rather than reading the stale values:
+
+  ```
+  FAIL <db>: uncheckpointed changes are waiting in <db>-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
+  ```
+
+  A running nxdns is the usual holder of that log, so this is a common answer
+  when checking a live server rather than a sign of damage. It depends on what
+  is in the log, not on whether a server is up: a configuration write that has
+  not been checkpointed puts bytes there, and a server that has only been
+  answering queries leaves `config.db-wal` empty and reads normally. Check an
+  `export` with `--config`, or stop the service first.
 
 ## `export`
 
@@ -92,17 +142,24 @@ See [back up and restore](../how-to/back-up-and-restore.md).
 ## `import FILE`
 
 Validates FILE and replaces the whole configuration with it in one transaction.
-Prints every validation problem on failure; a failed import leaves the database
-untouched. Refuses a database that already has content unless `--force` is
-given (`DatabaseNotEmpty`, exit 2). Creates the data directory at mode 0700 if
-it is missing.
+Prints every validation problem; a failed import leaves the database untouched.
+Refuses a database that already holds configuration unless `--force` is given
+(`DatabaseNotEmpty`, exit 2). The check measures what an operator set, not what
+the network did: client rows the DNS path materialised from traffic never
+trigger the refusal on their own. Creates the data directory at mode 0700 if it
+is missing.
+
+Client history survives the replacement. An address the database already knew
+keeps its first-seen and last-seen even when FILE names it; only an address it
+has never seen takes the import's clock. A client FILE leaves out is removed,
+history included.
 
 `FILE` is positional and may appear before or after the flags.
 
 | Flag | Meaning |
 | --- | --- |
 | `--data-dir DIR` | Data directory holding `config.db` (created if missing). |
-| `--force` | Replace a database that already has content. |
+| `--force` | Replace a database that already holds configuration. |
 
 ## `version`
 
@@ -120,31 +177,37 @@ the same. A usage error prints the same text to stderr and exits 64.
 | --- | --- |
 | 0 | Success. |
 | 1 | Runtime failure — I/O, database, out of memory. A partial diagnostic report caused by an allocation failure is a runtime failure, not a verdict on the configuration. |
-| 2 | A configuration problem the operator can fix, or a `check` that found one. Which faults qualify differs per subcommand; see below. |
+| 2 | A configuration problem the operator can fix, or a `check` that found one. |
 | 64 | Usage error — unknown command or flag, a flag without its value, a missing or extra argument. |
 
-Code 2 is not a single rule shared by every subcommand. `check` and `import`
-classify configuration faults; `run` maps only four errors to it and lets
-everything else out as a runtime failure.
+Code 2 means the same thing from every subcommand. `src/config/faults.zig`
+holds the one list of errors that mean "the configuration the operator supplied
+is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits
+2 whichever command read it. The list is every error the validator raises, plus
+`ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams` and `BadCertificate`. In
+practice that covers a seed file with a syntax error, one larger than 4 MiB, one
+with no `default` group (`MissingDefaultGroup`), one with no enabled upstream
+(`NoUpstreams`), a bad bind address, a bad rate limit, an unusable certificate,
+and `password` and `password_hash` set together.
 
-`check` exits 2 when validation reported at least one problem, when a
-certificate or key named by an enabled listener is unreadable, when a probed
-upstream failed, when the file is larger than 4 MiB, when the file has a ZON
-syntax error, and when there was nothing to check. Any other error escaping the
-run — a missing explicit `--config` file, an unreadable database — prints
-`check failed: <Error>` and exits 1.
+When `run` exits 2 it points at the diagnosis on stderr, whether the fault came
+from the seed file or from the database it loaded:
 
-`import` exits 2 when validation recorded at least one problem, and for
-`DatabaseNotEmpty`, `ConfigTooLarge`, `ParseZon` and `PasswordAndHashBothSet`.
-`OutOfMemory` is 1 even when problems were recorded, because the report is then
-incomplete. Every other error is 1.
+```
+run `nxdns check` to see the configuration in full
+```
 
-`run` exits 2 for `NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit` and
-`BadCertificate`, and points the operator at `nxdns check` on stderr when it
-does. Nothing else is remapped. In particular, seeding the database from
-`--config` on first start happens before that classification applies to it: a
-seed file that is unparseable (`ParseZon`), larger than 4 MiB
-(`ConfigTooLarge`), or rejected by validation prints its diagnostics to stderr
-and exits 1. The same file given to `check` or `import` exits 2.
+`check` exits 2 for those faults and also when a probed upstream failed, when a
+named configuration file is missing or unreadable, when the database cannot be
+read or is not at this binary's schema version, and when there was nothing to
+check. Warnings never contribute.
+
+`import` exits 2 for those faults and for `DatabaseNotEmpty`. That last one is
+deliberately not a configuration fault — it reports the state of the database
+rather than the content of a file — and `import` decides it for itself; the
+answer to it is `--force`, not an edit.
+
+`OutOfMemory` is exit 1 even when problems were recorded, because the report is
+then incomplete. Every other error is 1.
 
 Where an exit code sends you next: [troubleshoot](../how-to/troubleshoot.md).

docs/reference/configuration.md

@@ -115,7 +115,7 @@ The DNS-over-HTTPS listener (server side, for clients on the LAN). See
 | `doh_server.enabled` | bool | false | — | — | gates the DoH listener (`src/server/doh_server.zig`) |
 | `doh_server.bind` | string | `"0.0.0.0"` | IP address | must parse as an IP address of either family | DoH listener bind |
 | `doh_server.port` | u16 | 443 | port | 1–65535 (0 is refused, enabled or not) | DoH listener port |
-| `doh_server.cert_path` | string | `"/etc/nxdns/cert.pem"` | path | non-empty when enabled | certificate loaded into the hot-reloading `CertStore`; readability is checked by `nxdns check`, not by the validator |
+| `doh_server.cert_path` | string | `"/etc/nxdns/cert.pem"` | path | non-empty when enabled | certificate loaded into the hot-reloading `CertStore`; the file itself is loaded and paired with the key by `nxdns check`, not by the validator |
 | `doh_server.key_path` | string | `"/etc/nxdns/key.pem"` | path | non-empty when enabled | private key for the same; `nxdns check` warns when it is readable beyond its owner |
 
 ### dot_server
@@ -169,7 +169,7 @@ the client tracker and the blocklist scheduler are throttled
 | Key | Type | Default | Unit | Validation | Consumed by |
 |---|---|---|---|---|---|
 | `blocklist_update.enabled` | bool | true | — | — | blocklist refresh scheduler (`src/filter/manager.zig`); when false the scheduler stops after the startup pass and only a manual refresh runs |
-| `blocklist_update.interval_hours` | u16 | 24 | hours | at least 1 | sleep between refresh passes and the per-source staleness test |
+| `blocklist_update.interval_hours` | u16 | 24 | hours | at least 1 | sleep between refresh passes and the per-source staleness test; each pass is preceded by the [orphan sweep](files-and-directories.md#the-orphan-sweep), which runs even when the pass itself is skipped for disk space |
 
 ## Collections
 
@@ -329,9 +329,35 @@ authentication](../how-to/set-up-admin-authentication.md).
 
 ## Validation errors
 
-`nxdns check` and `nxdns import` print one `path: message` line per problem and
-report every problem, not just the first. The error set is
-`validate.ValidateError` in `src/config/validate.zig`:
+`nxdns check`, `nxdns import` and the `nxdns run` that seeds a database from the
+file all print one `FAIL path: message` line per problem, and report every
+problem rather than the first. A finding that is legal but almost certainly
+unintended is prefixed `WARN` instead: it does not change the exit code, and it
+is printed by all three even when nothing failed, so an accepted configuration
+still says what is odd about it.
+
+A url in a diagnostic is redacted to its scheme, host and port. The userinfo,
+the path, the query and the fragment are dropped, and control characters are
+escaped. These lines reach the journal, and every one of those parts can carry a
+credential: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the
+path segment is the whole account identifier. The field path beside the message
+names the entry, so `blocklist_sources[1].url` still says which one to go and
+fix.
+
+One credential this cannot remove: a NextDNS **DoT** upstream is
+`tls://abcd12.dns.nextdns.io`, which carries the same identifier in the hostname.
+Stripping it would leave no host at all and no line worth reading. So if you use
+NextDNS over DoT, your profile id appears in the log.
+
+The host is kept because a hostname is not a secret in the general case — it is
+resolved publicly and offered as SNI on every connection — and dropping it would
+cost every operator a diagnostic to cover one vendor's choice. If that trade is
+wrong for you, it is yours to make rather than ours: NextDNS also publishes a DoH
+endpoint, `https://dns.nextdns.io/abcd12`, whose identifier sits in the path and
+is redacted in full. Configure that form instead and nothing identifying reaches
+the log.
+
+The error set is `validate.ValidateError` in `src/config/validate.zig`:
 
 | Error | Raised by |
 | --- | --- |
@@ -363,6 +389,13 @@ report every problem, not just the first. The error set is
 | `MissingLogPath` | `logging.output = .file` with an empty or relative `file_path` |
 | `PasswordAndHashBothSet` | both `web.password` and `web.password_hash` are set |
 
+Warnings are a separate set, outside `ValidateError` because they are not
+failures. There is one:
+
+| Warning | Raised by |
+| --- | --- |
+| `SourceInNoGroup` | a `blocklist_sources` entry that no `group_sources` link names; it downloads and blocks nothing until a group uses it |
+
 ## Minimal working example
 
 The smallest file that passes validation: a `default` group and one enabled

docs/reference/files-and-directories.md

@@ -16,19 +16,31 @@ never looks at the directory, and without one `check` falls back to the default
 configuration file, or prints "nothing to check" and exits 2 when neither source
 exists.
 
-No subcommand opens the directory read-only. `run`, `import`, `export` and a
-`check` that resolved to the database all go through the same
-`DataDir.openConfigDb`, which opens `config.db` read/write, chmods it to 0600,
-enables WAL — creating `config.db-wal` and `config.db-shm` — and then runs any
-pending schema migrations. So `nxdns export` and `nxdns check` write to the data
-directory, and on a database one schema version behind they migrate it. `export`
-additionally creates an empty `config.db` if the directory exists without one;
-`check` reaches the database branch only when `config.db` is already there.
+`run`, `import` and `export` go through `DataDir.openConfigDb`, which opens
+`config.db` read/write, chmods it to 0600, enables WAL — creating
+`config.db-wal` and `config.db-shm` — and then runs any pending schema
+migrations. So `nxdns export` writes to the data directory, and on a database
+one schema version behind it migrates it. `export` also creates an empty
+`config.db` if the directory exists without one.
+
+`nxdns check` is the exception: it does not use that path at all. It opens
+`config.db` immutable, which is `SQLITE_OPEN_READONLY` plus `immutable=1`, so
+SQLite refuses every statement that would write and builds no wal-index. No
+`config.db-wal` and no `config.db-shm` appear beside the file, the mode is left
+alone, and no migration runs — a database behind this binary's schema is
+reported as a failure naming `nxdns run` as the fix. A `check` against a data
+directory leaves it byte-identical, and `check` reaches the database branch only
+when `config.db` is already there.
+
+`immutable=1` ignores any `-wal` file, so it is refused rather than used when
+one holds bytes: the newest settings would be invisible and `check` would grade
+older ones from the main file. That is the "uncheckpointed changes" failure in
+[the CLI reference](cli.md#what-check-does-not-do).
 
 | Path | What it is | Mode |
 | --- | --- | --- |
 | `config.db` | The configuration database — the single source of truth, including `web.password_hash`. | 0600 |
-| `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created when WAL is enabled, inheriting the main file's permissions. | 0600 |
+| `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created by `run`, `import` and `export` when WAL is enabled, inheriting the main file's permissions. `check` creates neither. | 0600 |
 | `querylog.db` | The query log: every domain every client asked for. Expendable — if it is missing or unusable it is recreated empty. | 0600 |
 | `querylog.db-wal`, `querylog.db-shm` | WAL sidecars for `querylog.db`. | 0600 |
 | `querylog.db.corrupt-<unix-seconds>` | A `querylog.db` that could not be used, moved aside before an empty one was created in its place. Kept, never overwritten. | Whatever the renamed file had — no chmod reaches it |
@@ -38,11 +50,45 @@ additionally creates an empty `config.db` if the directory exists without one;
 | `blocklists/<id>.wild` | Wildcard entries for the same source. | 0600 |
 | `blocklists/<id>.raw.tmp`, `<id>.list.tmp`, `<id>.wild.tmp` | Transient refresh state: the downloaded body and the two compile outputs before they are published by rename. | 0600 |
 
-`<id>` is the `blocklist_sources` row id. A `.list` or `.wild` file whose id is
-no longer a `blocklist_sources` row is deleted by the orphan sweep; files of a
-live source are left alone whatever their state, and the sweep matches only the
-two published suffixes, so a `.tmp` file left by an interrupted refresh is not
-swept — the next refresh of that source overwrites it.
+`<id>` is the `blocklist_sources` row id.
+
+### The orphan sweep
+
+The sweep decides by id, not by suffix. It matches all five names above and
+deletes those whose `<id>` is no longer a `blocklist_sources` row, so the
+compiled `.list` and `.wild` of a removed source go, and so do a `.raw.tmp`,
+`.list.tmp` or `.wild.tmp` left behind by a refresh that was killed before it
+could clean up. Files belonging to a source that still has a row are never
+touched, whatever state they are in: the sweep holds the same lock every refresh
+takes, so it never reads the directory while a refresh is part-way through.
+
+It runs at three moments:
+
+- at startup, before the first refresh pass — this is what collects what a
+  killed process left behind, and the compiled files of a source deleted while
+  the server was down;
+- before each scheduled update pass, ahead of the disk-space gate: the sweep
+  only unlinks, so it is the one step here that can give a critically full disk
+  room back, and gating it would keep the residue that helped fill the disk;
+- immediately after `DELETE /api/blocklists/{id}`, which is when an orphan is
+  actually created in normal operation. Without it a deleted list would keep its
+  megabytes until the next scheduled pass.
+
+With `blocklist_update.enabled = false` there are no scheduled passes, so only
+the first and the last of those three happen.
+
+Each deletion is logged:
+
+```
+info(blocklist_manager): pruned orphaned blocklist file 9999.list
+```
+
+A failed sweep is a warning, not an outage — leftover bytes do not justify
+losing the refresh pass behind them, let alone the server.
+
+The temporaries of a source that still exists are cleaned by the refresh that
+owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`,
+`.list.tmp` and `.wild.tmp` as it finishes, successfully or not.
 
 A `querylog.db` is moved aside when it is missing nothing but usability:
 SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not
@@ -111,7 +157,9 @@ no file; under systemd the journal captures it.
 
 `doh_server.cert_path` / `key_path` and `dot_server.cert_path` / `key_path`,
 conventionally under `/etc/nxdns`. nxdns reads them, never writes or creates
-them. Both must be readable by the user nxdns runs as; the key should be
-readable by its owner only, which `nxdns check` warns about when it is not. A
+them. Both must be readable by the user nxdns runs as, and the key must belong
+to the certificate: `nxdns check` loads the pair and fails when it does not. The
+key should also be readable by its owner only, which `check` warns about when it
+is not. A
 watcher polls both files and swaps a renewed pair in without a restart. See
 [enable DoH and DoT](../how-to/enable-doh-and-dot.md).

docs/reference/performance.md

@@ -12,8 +12,8 @@ see [measure performance](../how-to/measure-performance.md).
 | Sustained ≥ 100 qps on Raspberry Pi 5 | End-to-end against the real binary on the Pi; not a harness number |
 | Blocklist lookup p95 < 1 ms | `bench filter`: `matcher.normalize` + `Snapshot.evaluate` per op |
 | Cached response p95 < 5 ms | `bench cache`: `buildKey` + `DnsCache.get` + `packet.setId` per op |
-| Memory with ~1M blocked domains < 100 MB | `bench filter`: VmRSS with the 1M-domain snapshot loaded |
-| Stripped static binary < 10 MB per arch (< 15 MB with the embedded frontend) | CI size assert on the `cross` artifacts |
+| Memory with ~1M blocked domains < 100 MiB | `bench filter`: VmRSS with the 1M-domain snapshot loaded |
+| Stripped static binary < 10 MiB per arch (< 15 MiB with the embedded frontend) | CI size assert on the `cross` artifacts |
 
 The harness is `tools/bench.zig`. It measures the three targets that are
 measurable in process; the qps target is end to end and the binary-size target
@@ -61,7 +61,7 @@ VmRSS is lower because the filter suite's snapshot has been freed by then.
 | --- | --- |
 | Blocklist lookup p95 < 1 ms | to be measured on hardware |
 | Cached response p95 < 5 ms | to be measured on hardware |
-| Memory with ~1M blocked domains < 100 MB | to be measured on hardware |
+| Memory with ~1M blocked domains < 100 MiB | to be measured on hardware |
 | Sustained ≥ 100 qps | to be measured on hardware, end to end |
 
 The qps target belongs to the real binary rather than the harness: it means

docs/tutorial/first-run.md

@@ -71,9 +71,9 @@ an install decision, not a first-run decision. A real install is covered in
 The `default` group and one enabled upstream are the two things nxdns will not
 start without. Every client that nxdns has never seen is assigned to `default`,
 and with no usable upstream there is nowhere to send a query it cannot answer
-itself, so a configuration missing either one is rejected. Seeding happens
-inside `run`, so `run` prints the problem and exits 1; `nxdns check` and `nxdns
-import` reject the same file with exit code 2.
+itself, so a configuration missing either one is rejected. Whichever command
+reads the file says the same thing and stops the same way: `run`, `nxdns check`
+and `nxdns import` all print the problem and exit 2.
 
 ## 4. Check the configuration before starting
 
@@ -83,13 +83,14 @@ zig-out/bin/nxdns check --data-dir ~/nxdns-tutorial/data --config ~/nxdns-tutori
 
 ```
 checking configuration file /home/you/nxdns-tutorial/config.zon
-OK https://cloudflare-dns.com/dns-query
+OK upstreams[0] https://cloudflare-dns.com
 OK: no problems found
 ```
 
 `check` parses the file, validates it, and contacts each upstream to confirm it
-answers. It exits 0 when there is nothing to fix and 2 when there is. It does
-not start any listener, so you can run it as often as you like.
+answers. It exits 2 when it found something that has to be fixed and 0
+otherwise. It writes nothing and starts no listener, so you can run it as often
+as you like.
 
 ## 5. Start the server
 
@@ -158,6 +159,13 @@ separate decision, which is what lets one group get a strict list and another
 get none. A source that is attached to no group is downloaded and then filters
 nothing.
 
+You are in that state right now, between the previous step and this one, and it
+is legal rather than wrong — creating a source and attaching it afterwards is
+the normal order, which is why the API accepted it without complaint. On a
+stopped server `nxdns check` names such a source in a `WARN` line and still
+exits 0, so a list that silently blocks nothing is something you can find out
+about later. Attaching it now is what makes it take effect.
+
 Ask which groups exist:
 
 ```sh