diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index f32179f..c1b8966 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -107,8 +107,8 @@ jobs: npm ci npm run build - # ReleaseSafe because the < 15 MB budget (PLAN §18) is for release - # binaries; a Debug build strips to ~25 MB and can never meet it. + # ReleaseSafe because the < 15 MiB budget (PLAN §18) is for release + # binaries; a Debug build strips to roughly 25 MiB and can never meet it. - name: Build static musl executables run: zig build cross -Dweb-dist=web/dist -Doptimize=ReleaseSafe @@ -153,7 +153,7 @@ jobs: size=$(stat -c %s "$binary.stripped") echo "$triple: stripped size $size bytes" if [ "$size" -ge "$size_limit" ]; then - echo "stripped executable exceeds the 15 MB budget: $binary" + echo "stripped executable exceeds the 15 MiB budget: $binary" exit 1 fi done diff --git a/PLAN.md b/PLAN.md index f74249d..ec5ba8f 100644 --- a/PLAN.md +++ b/PLAN.md @@ -76,7 +76,7 @@ Verified: 0.16.0 ships `std.crypto.tls.Client` only. There is no server-side TLS - **DB is truth. Config file format is ZON** (`std.zon` parse + stringify — typed parsing into config structs, exact round-trip, stdlib-maintained, comments supported). No TOML: a third-party parser plus a hand-written serializer is two failure surfaces in the correctness-critical bootstrap/round-trip path, bought for syntax familiarity. - First start: if DB empty and `/etc/nxdns/config.zon` exists, validate → seed DB. Subsequent starts ignore the file. -- `nxdns export [--out file.zon]` dumps DB state as canonical ZON. `nxdns import ` validates + replaces DB contents (`--force` if DB non-empty). Export/import = backup + host migration, **not** upgrades (§3.7). +- `nxdns export [--out file.zon]` dumps DB state as canonical ZON. `nxdns import ` validates + replaces DB contents (`--force` if the DB holds configuration; client rows materialised from traffic do not count, and survive the replacement. First-seen/last-seen are runtime state, not configuration: they follow the address, so an import never restamps a device the DB already knew). Export/import = backup + host migration, **not** upgrades (§3.7). - No file watcher, no auto-regeneration. ### 3.6 Storage Layout (Decision H) @@ -130,7 +130,7 @@ IPv4 + IPv6 full parity for: client identity, rate limiting, logging, group assi - `/etc/nxdns/config.zon` — bootstrap (first start only). - `/var/lib/nxdns/config.db`, `/var/lib/nxdns/querylog.db` -- `/var/lib/nxdns/blocklists/*.list|*.wild` +- `/var/lib/nxdns/blocklists/*.list|*.wild` (plus `*.raw.tmp|*.list.tmp|*.wild.tmp` during a refresh) - `/var/log/nxdns/nxdns.log` — only in file output mode; default is stderr → journald. ### 3.14 Frontend Stack (Decision I) @@ -521,7 +521,7 @@ Periodic delete of rows older than `retention_days`; scheduled checkpoint/VACUUM ### 12.2 Validation -At least one upstream; ports in range; cert+key readable if DoH/DoT server enabled; resolver URLs parseable. `nxdns check` runs the validator + probes upstreams. +At least one upstream; ports in range; resolver URLs parseable. `nxdns check` runs the validator, probes upstreams, and loads the cert+key pair of each enabled DoH/DoT server through the same `CertStore.init` the listeners use. ### 12.3 Settings Semantics @@ -565,7 +565,7 @@ Requirements: responsive desktop/mobile; route loaders for initial fetch; TanSta ## 15. CLI - `nxdns run` — start the server. -- `nxdns check` — validate config, probe upstreams, test cert readability; nonzero exit on failure. +- `nxdns check` — validate config, probe upstreams, load each enabled listener's certificate and verify its key pairs with it; exit 2 on failure, 0 with warnings. - `nxdns export [--out file.zon]` - `nxdns import [--force]` - `nxdns version` — app version, Zig version string, build date, git commit. @@ -634,8 +634,8 @@ Exit: documented deployment works end-to-end on the Pi 5. - Sustained ≥ 100 qps on Raspberry Pi 5. - Blocklist lookup p95 < 1 ms. - Cached response p95 < 5 ms. -- Memory with ~1M blocked domains < 100 MB. -- Stripped static binary < 10 MB per arch (excluding embedded frontend assets; < 15 MB with them). +- Memory with ~1M blocked domains < 100 MiB. +- Stripped static binary < 10 MiB per arch (excluding embedded frontend assets; < 15 MiB with them). --- diff --git a/docs/explanation/configuration-model.md b/docs/explanation/configuration-model.md index f3d68bb..938932e 100644 --- a/docs/explanation/configuration-model.md +++ b/docs/explanation/configuration-model.md @@ -59,11 +59,35 @@ time you look at the UI, rather than "my edit was applied and then quietly undone next Tuesday". The emptiness check is a real query over the content tables, not a flag: the -database counts as configured when any content table has rows, or when the -default group has been altered. One consequence is worth knowing, because it -is not obvious: clients are auto-materialised when they first send a query, so -a server that has answered even one query is "configured" and will ignore a -seed file placed there afterwards. +database counts as configured when a content table holds rows, or when the +default group has been altered. What it measures is intent, not activity, and +the client table is where those two come apart. Clients are auto-materialised +when they first send a query — the DNS path writes a row per device it sees — +and such a row records what the network did, not what an operator decided. It +carries `hand_edited = 0`, the emptiness check counts only the `hand_edited = 1` +rows, and `export` omits them. So answering queries never turns an unconfigured +database into a configured one; naming a device does. + +Counting traffic here would have been a quiet trap: a server that resolved one +name would have declared itself configured and ignored a seed file placed +afterwards, and the operator would have had no line of output saying why. + +The same distinction survives an import. `import` replaces the content in one +transaction, which empties `clients` along with every other table, so every +client row is saved before the wipe. The materialised ones are put back after it +unchanged, and restoring a backup does not make the server forget the devices it +has met. + +An address the imported file names belongs to the file — the operator's +statement wins over the discovered row — with one carve-out. First-seen and +last-seen are not configuration: they record when a device was heard from, the +configuration model has no field for either, and an import is not a query. So +they follow the address rather than the row. If the database already knew that +address, its two timestamps are carried onto the new row; only an address the +database has never seen takes the import's clock. Without that, re-importing a +backup would stamp every device the operator had bothered to name as though it +had just arrived — and those are exactly the devices whose history is worth +something. ## The round trip @@ -90,11 +114,17 @@ difference is a configuration change, never noise from when the export ran. `nxdns import` replaces the whole database content in one transaction. Not a merge, not a patch: delete every content table in foreign-key-safe order, then -insert what the file says. A failed import — bad syntax, failed validation, a -constraint violation halfway through — leaves the database exactly as it was, -because everything happens inside a single `BEGIN IMMEDIATE`. +insert what the file says — with the one exception described above. Every client +row is lifted out first. The materialised ones are put back rather than +recreated from a file that never held them, and the observed timestamps of an +address the file *does* name are merged onto its new row. A client the file +leaves out is gone, history included: nothing puts a `hand_edited = 1` row back. +A failed import — bad syntax, failed +validation, a constraint violation halfway through — leaves the database exactly +as it was, because everything happens inside a single `BEGIN IMMEDIATE`. -Without `--force`, import refuses a database that already has content. That +Without `--force`, import refuses a database that already holds configuration. +That check runs *inside* the transaction, after the lock is taken, so it cannot be raced by a concurrent write. The effect is that a plain `import` can never clobber a configured server by accident, and clobbering it deliberately takes diff --git a/docs/explanation/performance-and-testing.md b/docs/explanation/performance-and-testing.md index 8b5ad4c..12f95d6 100644 --- a/docs/explanation/performance-and-testing.md +++ b/docs/explanation/performance-and-testing.md @@ -15,8 +15,8 @@ PLAN §18 sets five: - sustained ≥ 100 qps on a Raspberry Pi 5; - blocklist lookup p95 < 1 ms; - cached response p95 < 5 ms; -- memory with ~1M blocked domains < 100 MB; -- stripped static binary < 10 MB per arch, < 15 MB with the embedded frontend. +- memory with ~1M blocked domains < 100 MiB; +- stripped static binary < 10 MiB per arch, < 15 MiB with the embedded frontend. They are household-scale numbers, and they are deliberately unambitious. 100 qps is far more than a house generates; the point of the target is not speed diff --git a/docs/how-to/back-up-and-restore.md b/docs/how-to/back-up-and-restore.md index 899d83c..4339b04 100644 --- a/docs/how-to/back-up-and-restore.md +++ b/docs/how-to/back-up-and-restore.md @@ -88,8 +88,8 @@ before writing to it. ## Restore over an existing database -`import` refuses a database that already has content, so a plain `import` can -never clobber a configured server by accident: +`import` refuses a database that already holds configuration, so a plain +`import` can never clobber a configured server by accident: ```sh nxdns import /tmp/nxdns-lab/backup.zon --data-dir /tmp/nxdns-lab/data @@ -112,6 +112,13 @@ imported /tmp/nxdns-lab/backup.zon Stop the server first. `import` replaces the whole configuration underneath a process that has already read it, and a running server will not notice. +What `--force` does to the client list is worth knowing before you restore an +old backup. A client the backup does not name is removed, and its first-seen +and last-seen go with it — restoring a month-old file drops the devices you +named since. Devices the server discovered from traffic are kept, and a device +the backup does name keeps the first-seen and last-seen the database already +held, so a restore does not restamp your whole network as newly arrived. + On a real install that is the systemd unit: ```sh @@ -185,8 +192,9 @@ The data directory layout is in ## A backup before every upgrade There is no downgrade path. Schema migrations run forward automatically at -startup and before `check`, `export` and `import`; nothing walks them back. Take -an export before installing a new binary — see [upgrade](upgrade.md). +startup and before `export` and `import`; nothing walks them back, and `nxdns +check` does not run them at all. Take an export before installing a new binary — +see [upgrade](upgrade.md). Every command on this page was executed on this host as written, except the `systemctl` block marked **Not verified on this host** above. diff --git a/docs/how-to/enable-doh-and-dot.md b/docs/how-to/enable-doh-and-dot.md index bbc589b..5a798af 100644 --- a/docs/how-to/enable-doh-and-dot.md +++ b/docs/how-to/enable-doh-and-dot.md @@ -75,23 +75,40 @@ nxdns check --data-dir /tmp/nxdns-lab/data --config /tmp/nxdns-lab/etc/config.zo ``` checking configuration file /tmp/nxdns-lab/etc/config.zon -OK https://cloudflare-dns.com/dns-query +OK upstreams[0] https://cloudflare-dns.com OK: no problems found ``` -`check` reads both endpoints' certificate and key. An unreadable file is a -failure and exits 2: +`check` loads both endpoints' certificate and key the same way the listeners do, +so what passes here will start. An unreadable file is a failure and exits 2: ``` -FAIL doh_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem' is not readable -FAIL dot_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem' is not readable +FAIL doh_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable +FAIL dot_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable ``` -A key readable by anyone but its owner is a warning, and does not change the -exit code, because the service still starts: +So is a key that does not belong to the certificate, which is the mistake worth +catching before a restart — the two files are individually valid and only their +pairing is wrong. mbedTLS writes its own line to stderr as it rejects the pair: + +``` +warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896) +warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896) +checking configuration file /tmp/nxdns-lab/etc/mismatch.zon +FAIL doh_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not belong to the certificate +FAIL dot_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not belong to the certificate +OK upstreams[0] https://cloudflare-dns.com +``` + +A key readable by anyone but its owner is a warning instead. It does not change +the exit code, because the service still starts, and the summary line counts it +rather than claiming nothing was found: ``` WARN doh_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only +WARN dot_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only +OK upstreams[0] https://cloudflare-dns.com +OK: no failures found, 2 warnings ``` ## 4. Start and confirm the listeners diff --git a/docs/how-to/install-with-docker.md b/docs/how-to/install-with-docker.md index e1552b0..3783934 100644 --- a/docs/how-to/install-with-docker.md +++ b/docs/how-to/install-with-docker.md @@ -73,13 +73,28 @@ database has nothing to forward to. The log is `no configuration file at '/etc/nxdns/config.zon'; using the database as it is` followed by `nxdns run failed: NoUsableUpstreams`. -A file that is present but rejected is a different failure and a different exit +A file that is present but rejected is a different failure with the same exit code. No `default` group, no enabled upstream, a syntax error — `run` prints the -diagnostic and exits **1**, for example `nxdns run failed: MissingDefaultGroup`. -Both were run here against this image: a seed file whose only group was named -`other` exited 1, and an empty `/etc/nxdns` exited 2 with `NoUsableUpstreams`. -Under `restart: unless-stopped` either one is a restart loop, so read the exit -code from `docker inspect` to tell them apart; see +diagnostic and exits 2 as well. Both were run here against this image. A seed +file whose only group was named `other`: + +``` +FAIL groups: no group named 'default'; every unknown client is assigned to it +nxdns run failed: MissingDefaultGroup +run `nxdns check` to see the configuration in full +``` + +and an empty `/etc/nxdns`: + +``` +info(config_bootstrap): no configuration file at '/etc/nxdns/config.zon'; using the database as it is +nxdns run failed: NoUsableUpstreams +run `nxdns check` to see the configuration in full +``` + +Under `restart: unless-stopped` either one is a restart loop, and the exit code +alone no longer tells them apart: read the lines above the failure, which either +name the diagnostic in the file or say there was no file at all. See [Troubleshoot nxdns](troubleshoot.md). The container runs as uid 65532, and the mount is read-only, so the container diff --git a/docs/how-to/install-with-systemd.md b/docs/how-to/install-with-systemd.md index b0da3c1..f3bbd63 100644 --- a/docs/how-to/install-with-systemd.md +++ b/docs/how-to/install-with-systemd.md @@ -152,7 +152,7 @@ A good file prints the source it checked, one `OK` line per upstream, and ``` checking configuration file /etc/nxdns/config.zon -OK https://cloudflare-dns.com/dns-query +OK upstreams[0] https://cloudflare-dns.com OK: no problems found ``` diff --git a/docs/how-to/set-up-admin-authentication.md b/docs/how-to/set-up-admin-authentication.md index 604673a..e878846 100644 --- a/docs/how-to/set-up-admin-authentication.md +++ b/docs/how-to/set-up-admin-authentication.md @@ -228,7 +228,7 @@ nxdns export --data-dir /tmp/nxdns-lab/data | grep password .password_hash = "$argon2id$v=19$m=19456,t=2,p=1$kvlRj1tdGul3MlfbvzLncLKWirNpJRJ3howFA9/ysgg$7elW7PPQ3WXHwI4YOmOpZ/1KNEQo7ZDLRJhnYOPMjqw", ``` -`--force` is required because the database already has content. See +`--force` is required because the database already holds configuration. See [back up and restore](back-up-and-restore.md). ## What happens with no password set diff --git a/docs/how-to/troubleshoot.md b/docs/how-to/troubleshoot.md index 593613a..d25ae07 100644 --- a/docs/how-to/troubleshoot.md +++ b/docs/how-to/troubleshoot.md @@ -21,8 +21,9 @@ nxdns run failed: NoUsableUpstreams run `nxdns check` to see the configuration in full ``` -Exit 2 is reserved for a small set of faults `run` raises itself: -`NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit` and `BadCertificate`. +Exit 2 means the configuration is wrong and you can fix it. Every subcommand +uses the same definition, so a file `run` exits 2 on exits 2 from `check` and +`import` too. **Diagnosis.** @@ -45,18 +46,18 @@ checked on its first line. certificate. `run` names both paths before it exits: `doh_server: '' + '': certificate file is not readable`. - **`check` does not catch most of this.** It tests only that each file is - readable, and warns when the key is readable beyond its owner; it never opens - the PEM. Parsing and the key/certificate pairing happen when `run` builds the - TLS context, so `check` can print `OK: no problems found` on a configuration - `run` then refuses. Reproduced here with a self-signed pair and the key from a - second, unrelated pair: + `check` catches this without starting a listener. It loads both PEM files and + tests the key against the certificate through the same code `run` uses, so it + fails on exactly what `run` would fail on. Reproduced here with a self-signed + pair and the key from a second, unrelated pair: ``` $ nxdns check --config config.zon + warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896) checking configuration file config.zon - OK https://cloudflare-dns.com/dns-query - OK: no problems found # exit 0 + FAIL doh_server.key_path: 'mismatched-key.pem': private key does not belong to the certificate + OK upstreams[0] https://cloudflare-dns.com + # exit 2 $ nxdns run --config config.zon --data-dir ./data warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896) @@ -64,59 +65,103 @@ checked on its first line. nxdns run failed: BadCertificate # exit 2 ``` - A cert file containing `not a certificate` behaves the same way — `check` - exits 0, `run` exits 2 with `certificate PEM could not be parsed`. So a - successful `check` means the paths and permissions are right, not that the - certificate is usable; the only test of that is starting the service. Fix the - path, the ownership, or the pair; see + The `warning(tls_server)` line comes from mbedTLS on stderr and can appear + before the `checking` line, which is on stdout. A cert file containing + `not a certificate` fails the same way, with + `FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`. + An unreadable file reads + `FAIL doh_server.cert_path: '': certificate file is not readable`. + + Fix the path, the ownership, or the pair; see [Enable DoH and DoT](enable-doh-and-dot.md). - `BadRateLimit` — a rate limit or window is zero. `import` refuses such a configuration, so this only reaches a database that was edited by hand. - `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of that family. -## The service exits with code 1 on a seed file you just wrote +## A seed file you just wrote is rejected **Symptom.** A first start against an empty database prints the validation -problem and stops, but with exit code 1, not 2: +problem and stops with exit 2: ``` -groups: no group named 'default'; every unknown client is assigned to it +FAIL groups: no group named 'default'; every unknown client is assigned to it nxdns run failed: MissingDefaultGroup +run `nxdns check` to see the configuration in full ``` A syntax error behaves the same way: ``` -config: 2:42: error: expected ',' after initializer +FAIL config: 3:16: error: expected ',' after initializer nxdns run failed: ParseZon +run `nxdns check` to see the configuration in full ``` So does a seed file whose upstream list is empty or all disabled: ``` -upstreams: at least one upstream must be enabled +FAIL upstreams: at least one upstream must be enabled nxdns run failed: NoUpstreams +run `nxdns check` to see the configuration in full ``` `NoUpstreams` from a seed file is not the same fault as `NoUsableUpstreams` above: the first is a file `run` refused, the second is a database `run` -accepted and found empty. Only the second is exit 2. +accepted and found empty. Both are exit 2. -**Diagnosis.** Run the same file through `check`, which reports it as a -configuration problem and exits 2: +**Diagnosis.** Run the same file through `check`, which reports the same +problems and exits 2: ```sh nxdns check --config /etc/nxdns/config.zon ``` **Fix.** Correct the file the diagnostics name and start again. The database is -still empty after a failed seed, so the next start re-reads the file. Note that -`nxdns check` and `nxdns import` of the same bad file exit 2 while `nxdns run` -exits 1 — the exit code differs by command, the diagnostics do not. All three -commands were run here against a file missing its `default` group, one with a -syntax error and one with no enabled upstream, and every pair came out that -way. +still empty after a failed seed, so the next start re-reads the file. The exit +code no longer depends on which command read the file: all three of these files +were run through `run`, `check` and `import` here, and every one of the nine +combinations exited 2 with the same diagnostic. + +## `nxdns check` fails on a server that is running fine + +**Symptom.** The service is up and answering, but `nxdns check` on the same +machine exits 2 with one long line about a write-ahead log: + +``` +checking database /var/lib/nxdns/config.db +FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line. +``` + +Nothing is damaged. `check` opens `config.db` immutable so that it can never +write to it, and an immutable open ignores the write-ahead log. When that log +holds bytes, the newest settings are in it and the main file holds older ones, +so `check` refuses rather than grade stale values. + +The log holds bytes after a configuration write that has not been checkpointed +yet, which on a running server means someone changed something through the web +interface or the API. A server that has only been answering queries has an empty +`config.db-wal` and `check` reads it normally — so this line comes and goes, and +its absence is not proof that nothing is running. + +**Fix.** Check the exported configuration instead of the live file: + +```sh +nxdns export --data-dir /var/lib/nxdns --out /tmp/current.zon +nxdns check --config /tmp/current.zon +``` + +`export` opens the database read/write and does see the log, so it renders the +settings that are actually in force. Stopping the service and checking again +works too: a clean shutdown checkpoints the log away. + +> Reproduced here on a scratch data directory rather than `/var/lib/nxdns` — +> that path is the only substitution in the output above. nxdns was started on +> unprivileged ports; `config.db-wal` was 0 bytes and `check` exited 0; one +> `POST /api/blocklists` took it to 8272 bytes and `check` then printed the line +> above and exited 2; `export` from the same live directory succeeded and its +> output checked clean; and after a clean shutdown `check --data-dir` exited 0 +> again. ## Port 53 is already taken diff --git a/docs/how-to/upgrade.md b/docs/how-to/upgrade.md index 5e1d43e..2adb15f 100644 --- a/docs/how-to/upgrade.md +++ b/docs/how-to/upgrade.md @@ -108,43 +108,63 @@ configuration. ```sh nxdns version -nxdns check +nxdns export --out /tmp/after-upgrade.zon +nxdns check --config /tmp/after-upgrade.zon dig @127.0.0.1 example.com A +short ``` -`nxdns check` with no `--config` checks the database, which is what you want -after an upgrade — it names its source on the first line and migrates a -database that is one schema version behind before checking it: +The restart in step 3 is what migrated the database, so by now the schema is +current and the service is answering. Confirming with `nxdns check` alone would +not work here, and the reason is worth knowing: `check` opens `config.db` +immutable so it can never write to it, and the migration you just performed is +sitting in `config.db-wal` waiting to be checkpointed. Rather than read around +the log and grade older settings, `check` reports it: ``` -checking database /var/lib/nxdns/config.db -OK https://cloudflare-dns.com/dns-query +FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line. +``` + +`export` opens the database read/write and does see the log, so exporting and +then checking the export validates what is actually in force: + +``` +checking configuration file /tmp/after-upgrade.zon +OK upstreams[0] https://cloudflare-dns.com OK: no problems found ``` -> Verified on this host for the first two commands, with `--data-dir` pointing -> at the scratch data directory instead of `/var/lib/nxdns`: +> Verified on this host for the first three commands, with `--data-dir` +> pointing at a scratch data directory instead of `/var/lib/nxdns` — that path +> is the only difference from the blocks above: > > ``` > $ nxdns version > nxdns 0.1.0-dev (unknown) > zig 0.16.0 -> $ nxdns check --data-dir $SCRATCH/data -> checking database /…/scratchpad/data/config.db -> OK https://cloudflare-dns.com/dns-query -> OK: no problems found > ``` > -> The database path on the first line is the only difference from the block -> above. The `dig` line was not run in this round: nothing is listening on -> 127.0.0.1:53 here, and port 53 needs root. +> Against a running server whose database had just been migrated and seeded, +> `nxdns check --data-dir` printed the uncheckpointed-log line above and exited +> 2, while `nxdns export --out` followed by `nxdns check --config` on the result +> exited 0 with `OK: no problems found`. The `dig` line was not run in this +> round: nothing is listening on 127.0.0.1:53 here, and port 53 needs root. ## What happens to the database -Migrations run at startup, and also before `check`, `export` and `import`, so -whichever of those you run first performs the upgrade. A fresh database is -created at the current schema version; an older one is stepped up to it. The -log line names both versions: +Migrations run at startup, and also before `export` and `import`, so whichever +of those you run first performs the upgrade. `nxdns check` is the exception: it +opens the database immutable and never migrates, so on a database still one +version behind it reports the mismatch and exits 2 rather than fixing it: + +``` +FAIL /var/lib/nxdns/config.db: schema version 0, this nxdns expects 2; `nxdns run` migrates it, `check` will not +``` + +That line was reproduced here against a database stamped at version 0; the path +and the version numbers are what vary. + +A fresh database is created at the current schema version; an older one is +stepped up to it. The log line names both versions: ``` info(migrations): config.db migrated from schema version 0 to 2 @@ -194,9 +214,11 @@ nxdns import config-backup.zon --force systemctl start nxdns ``` -`--force` is required here. A plain `import` into a database that already has -content fails with `import failed: DatabaseNotEmpty` and exits 2, so it cannot -clobber a configured server by accident. +`--force` is required here. A plain `import` into a database that already holds +configuration fails with `import failed: DatabaseNotEmpty` and exits 2, so it +cannot clobber a configured server by accident. What counts is what an operator +set: client rows the DNS path materialised from traffic never trigger the +refusal on their own. > Verified on this host for the two `nxdns` lines, against a populated scratch > data directory: diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 9287ea0..62e4b33 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -23,21 +23,24 @@ Serves DNS until SIGINT or SIGTERM. | `--config FILE` | Seed configuration file (default `/etc/nxdns/config.zon`). Read only when the database has never been configured. | | `--web-dev DIR` | Serve the web interface from DIR instead of the embedded assets, with no cache headers. Development only. | -Seeding failures are not treated as configuration faults here: a seed file that -is unparseable, oversized or invalid prints its diagnostics and exits 1, not 2. -See [exit codes](#exit-codes). +A seed file that is unparseable, oversized or invalid prints its diagnostics and +exits 2 — the same code `check` and `import` give for the same file. See +[exit codes](#exit-codes). ## `check` -Validates the configuration and probes the upstreams. Exit 0 when clean, 2 when -it found problems. It always reports every problem, not just the first. What it -checks, in order: +Validates the configuration and probes the upstreams. Exit 0 when it found no +failures, 2 when it found one. It always reports every problem, not just the +first. What it checks, in order: 1. Which source to check (see [source selection](#source-selection)). 2. Full validation — the same rules `import` enforces. -3. For each enabled DoH/DoT listener: certificate and key are readable (FAIL if - not), and the key's permissions are owner-only (WARN if any group or other - bit is set; the WARN does not change the exit code). +3. For each enabled DoH/DoT listener: both PEM files are read and the key is + tested against the certificate, through the same `CertStore.init` the + listeners boot with. A file that is missing, unreadable, too large or + unparseable, and a key that does not belong to the certificate, are all FAIL. + The key's permissions are a separate finding: WARN when any group or other + bit is set, which does not change the exit code. 4. A live probe: one real A query for `example.com` through every enabled upstream, driving the same pool and failover machinery the server uses, with `upstream.total_timeout_ms` as the per-attempt deadline. A FAIL line names @@ -50,13 +53,28 @@ checks, in order: | `--data-dir DIR` | Data directory to look for `config.db` in. | | `--config FILE` | Check this file instead of the database. | +### Failures and warnings + +Every finding carries a severity. `FAIL` is a problem that sets exit 2. `WARN` +is legal configuration that is almost certainly not what was meant — a blocklist +source no group links to, a TLS key readable beyond its owner — and never +changes an exit code, because the service starts either way. + +The last line is a summary, and it never contradicts the lines above it: + +| What was found | Last line | Exit | +| --- | --- | --- | +| Nothing | `OK: no problems found` | 0 | +| Warnings only | `OK: no failures found, 2 warnings` — `1 warning` in the singular | 0 | +| At least one failure | No summary line; the FAIL lines are the report | 2 | + ### Source selection - `--config FILE` given explicitly: check that file, nothing else. - Otherwise, if `/config.db` exists: check the database — the right - default, since the database is the truth on a configured server. Pending - schema migrations are applied first, so a `check` immediately after an upgrade - works. + default, since the database is the truth on a configured server. It is opened + immutable, so `check` writes nothing to it; see + [what `check` does not do](#what-check-does-not-do). - Otherwise, if the default config file path exists: check it. - Otherwise: "nothing to check", exit 2. @@ -64,10 +82,42 @@ The first line of output always names which source was checked. A file larger than 4 MiB fails with `larger than 4194304 bytes`; a ZON syntax error is reported with its line and column. -A missing file is treated differently depending on how the path was reached. An -explicit `--config FILE` that does not exist is an I/O failure: `check failed: -FileNotFound` on stderr, exit 1. A default path that does not exist is one of -the cases above and produces "nothing to check", exit 2. +A named file that is missing or unreadable is a finding like any other, not an +I/O failure that escapes the run: `FAIL : no such file` or +`FAIL : not readable`, exit 2. That is the same code the implicit path +gives when there is nothing to check, so naming the file does not change what a +missing file costs. + +### What `check` does not do + +`check` never writes to `config.db`. It opens the file immutable, which means +SQLite refuses every statement that would write and builds no write-ahead log, +so no `config.db-wal` and no `config.db-shm` appear beside it. It does not chmod +the file and it does not migrate the schema. + +Two consequences are worth knowing before you read a FAIL line as damage: + +- A database behind this binary's schema is reported, not upgraded. Start the + service to migrate it. + + ``` + FAIL : schema version , this nxdns expects ; `nxdns run` migrates it, `check` will not + ``` + +- A database with an unapplied write-ahead log cannot be graded without writing, + because the newest settings are in the log and the main file holds older ones. + `check` says so rather than reading the stale values: + + ``` + FAIL : uncheckpointed changes are waiting in -wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line. + ``` + + A running nxdns is the usual holder of that log, so this is a common answer + when checking a live server rather than a sign of damage. It depends on what + is in the log, not on whether a server is up: a configuration write that has + not been checkpointed puts bytes there, and a server that has only been + answering queries leaves `config.db-wal` empty and reads normally. Check an + `export` with `--config`, or stop the service first. ## `export` @@ -92,17 +142,24 @@ See [back up and restore](../how-to/back-up-and-restore.md). ## `import FILE` Validates FILE and replaces the whole configuration with it in one transaction. -Prints every validation problem on failure; a failed import leaves the database -untouched. Refuses a database that already has content unless `--force` is -given (`DatabaseNotEmpty`, exit 2). Creates the data directory at mode 0700 if -it is missing. +Prints every validation problem; a failed import leaves the database untouched. +Refuses a database that already holds configuration unless `--force` is given +(`DatabaseNotEmpty`, exit 2). The check measures what an operator set, not what +the network did: client rows the DNS path materialised from traffic never +trigger the refusal on their own. Creates the data directory at mode 0700 if it +is missing. + +Client history survives the replacement. An address the database already knew +keeps its first-seen and last-seen even when FILE names it; only an address it +has never seen takes the import's clock. A client FILE leaves out is removed, +history included. `FILE` is positional and may appear before or after the flags. | Flag | Meaning | | --- | --- | | `--data-dir DIR` | Data directory holding `config.db` (created if missing). | -| `--force` | Replace a database that already has content. | +| `--force` | Replace a database that already holds configuration. | ## `version` @@ -120,31 +177,37 @@ the same. A usage error prints the same text to stderr and exits 64. | --- | --- | | 0 | Success. | | 1 | Runtime failure — I/O, database, out of memory. A partial diagnostic report caused by an allocation failure is a runtime failure, not a verdict on the configuration. | -| 2 | A configuration problem the operator can fix, or a `check` that found one. Which faults qualify differs per subcommand; see below. | +| 2 | A configuration problem the operator can fix, or a `check` that found one. | | 64 | Usage error — unknown command or flag, a flag without its value, a missing or extra argument. | -Code 2 is not a single rule shared by every subcommand. `check` and `import` -classify configuration faults; `run` maps only four errors to it and lets -everything else out as a runtime failure. +Code 2 means the same thing from every subcommand. `src/config/faults.zig` +holds the one list of errors that mean "the configuration the operator supplied +is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits +2 whichever command read it. The list is every error the validator raises, plus +`ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams` and `BadCertificate`. In +practice that covers a seed file with a syntax error, one larger than 4 MiB, one +with no `default` group (`MissingDefaultGroup`), one with no enabled upstream +(`NoUpstreams`), a bad bind address, a bad rate limit, an unusable certificate, +and `password` and `password_hash` set together. -`check` exits 2 when validation reported at least one problem, when a -certificate or key named by an enabled listener is unreadable, when a probed -upstream failed, when the file is larger than 4 MiB, when the file has a ZON -syntax error, and when there was nothing to check. Any other error escaping the -run — a missing explicit `--config` file, an unreadable database — prints -`check failed: ` and exits 1. +When `run` exits 2 it points at the diagnosis on stderr, whether the fault came +from the seed file or from the database it loaded: -`import` exits 2 when validation recorded at least one problem, and for -`DatabaseNotEmpty`, `ConfigTooLarge`, `ParseZon` and `PasswordAndHashBothSet`. -`OutOfMemory` is 1 even when problems were recorded, because the report is then -incomplete. Every other error is 1. +``` +run `nxdns check` to see the configuration in full +``` -`run` exits 2 for `NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit` and -`BadCertificate`, and points the operator at `nxdns check` on stderr when it -does. Nothing else is remapped. In particular, seeding the database from -`--config` on first start happens before that classification applies to it: a -seed file that is unparseable (`ParseZon`), larger than 4 MiB -(`ConfigTooLarge`), or rejected by validation prints its diagnostics to stderr -and exits 1. The same file given to `check` or `import` exits 2. +`check` exits 2 for those faults and also when a probed upstream failed, when a +named configuration file is missing or unreadable, when the database cannot be +read or is not at this binary's schema version, and when there was nothing to +check. Warnings never contribute. + +`import` exits 2 for those faults and for `DatabaseNotEmpty`. That last one is +deliberately not a configuration fault — it reports the state of the database +rather than the content of a file — and `import` decides it for itself; the +answer to it is `--force`, not an edit. + +`OutOfMemory` is exit 1 even when problems were recorded, because the report is +then incomplete. Every other error is 1. Where an exit code sends you next: [troubleshoot](../how-to/troubleshoot.md). diff --git a/docs/reference/configuration.md b/docs/reference/configuration.md index 49b1bbf..4b976a3 100644 --- a/docs/reference/configuration.md +++ b/docs/reference/configuration.md @@ -115,7 +115,7 @@ The DNS-over-HTTPS listener (server side, for clients on the LAN). See | `doh_server.enabled` | bool | false | — | — | gates the DoH listener (`src/server/doh_server.zig`) | | `doh_server.bind` | string | `"0.0.0.0"` | IP address | must parse as an IP address of either family | DoH listener bind | | `doh_server.port` | u16 | 443 | port | 1–65535 (0 is refused, enabled or not) | DoH listener port | -| `doh_server.cert_path` | string | `"/etc/nxdns/cert.pem"` | path | non-empty when enabled | certificate loaded into the hot-reloading `CertStore`; readability is checked by `nxdns check`, not by the validator | +| `doh_server.cert_path` | string | `"/etc/nxdns/cert.pem"` | path | non-empty when enabled | certificate loaded into the hot-reloading `CertStore`; the file itself is loaded and paired with the key by `nxdns check`, not by the validator | | `doh_server.key_path` | string | `"/etc/nxdns/key.pem"` | path | non-empty when enabled | private key for the same; `nxdns check` warns when it is readable beyond its owner | ### dot_server @@ -169,7 +169,7 @@ the client tracker and the blocklist scheduler are throttled | Key | Type | Default | Unit | Validation | Consumed by | |---|---|---|---|---|---| | `blocklist_update.enabled` | bool | true | — | — | blocklist refresh scheduler (`src/filter/manager.zig`); when false the scheduler stops after the startup pass and only a manual refresh runs | -| `blocklist_update.interval_hours` | u16 | 24 | hours | at least 1 | sleep between refresh passes and the per-source staleness test | +| `blocklist_update.interval_hours` | u16 | 24 | hours | at least 1 | sleep between refresh passes and the per-source staleness test; each pass is preceded by the [orphan sweep](files-and-directories.md#the-orphan-sweep), which runs even when the pass itself is skipped for disk space | ## Collections @@ -329,9 +329,35 @@ authentication](../how-to/set-up-admin-authentication.md). ## Validation errors -`nxdns check` and `nxdns import` print one `path: message` line per problem and -report every problem, not just the first. The error set is -`validate.ValidateError` in `src/config/validate.zig`: +`nxdns check`, `nxdns import` and the `nxdns run` that seeds a database from the +file all print one `FAIL path: message` line per problem, and report every +problem rather than the first. A finding that is legal but almost certainly +unintended is prefixed `WARN` instead: it does not change the exit code, and it +is printed by all three even when nothing failed, so an accepted configuration +still says what is odd about it. + +A url in a diagnostic is redacted to its scheme, host and port. The userinfo, +the path, the query and the fragment are dropped, and control characters are +escaped. These lines reach the journal, and every one of those parts can carry a +credential: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the +path segment is the whole account identifier. The field path beside the message +names the entry, so `blocklist_sources[1].url` still says which one to go and +fix. + +One credential this cannot remove: a NextDNS **DoT** upstream is +`tls://abcd12.dns.nextdns.io`, which carries the same identifier in the hostname. +Stripping it would leave no host at all and no line worth reading. So if you use +NextDNS over DoT, your profile id appears in the log. + +The host is kept because a hostname is not a secret in the general case — it is +resolved publicly and offered as SNI on every connection — and dropping it would +cost every operator a diagnostic to cover one vendor's choice. If that trade is +wrong for you, it is yours to make rather than ours: NextDNS also publishes a DoH +endpoint, `https://dns.nextdns.io/abcd12`, whose identifier sits in the path and +is redacted in full. Configure that form instead and nothing identifying reaches +the log. + +The error set is `validate.ValidateError` in `src/config/validate.zig`: | Error | Raised by | | --- | --- | @@ -363,6 +389,13 @@ report every problem, not just the first. The error set is | `MissingLogPath` | `logging.output = .file` with an empty or relative `file_path` | | `PasswordAndHashBothSet` | both `web.password` and `web.password_hash` are set | +Warnings are a separate set, outside `ValidateError` because they are not +failures. There is one: + +| Warning | Raised by | +| --- | --- | +| `SourceInNoGroup` | a `blocklist_sources` entry that no `group_sources` link names; it downloads and blocks nothing until a group uses it | + ## Minimal working example The smallest file that passes validation: a `default` group and one enabled diff --git a/docs/reference/files-and-directories.md b/docs/reference/files-and-directories.md index 0d40aa7..e6d8109 100644 --- a/docs/reference/files-and-directories.md +++ b/docs/reference/files-and-directories.md @@ -16,19 +16,31 @@ never looks at the directory, and without one `check` falls back to the default configuration file, or prints "nothing to check" and exits 2 when neither source exists. -No subcommand opens the directory read-only. `run`, `import`, `export` and a -`check` that resolved to the database all go through the same -`DataDir.openConfigDb`, which opens `config.db` read/write, chmods it to 0600, -enables WAL — creating `config.db-wal` and `config.db-shm` — and then runs any -pending schema migrations. So `nxdns export` and `nxdns check` write to the data -directory, and on a database one schema version behind they migrate it. `export` -additionally creates an empty `config.db` if the directory exists without one; -`check` reaches the database branch only when `config.db` is already there. +`run`, `import` and `export` go through `DataDir.openConfigDb`, which opens +`config.db` read/write, chmods it to 0600, enables WAL — creating +`config.db-wal` and `config.db-shm` — and then runs any pending schema +migrations. So `nxdns export` writes to the data directory, and on a database +one schema version behind it migrates it. `export` also creates an empty +`config.db` if the directory exists without one. + +`nxdns check` is the exception: it does not use that path at all. It opens +`config.db` immutable, which is `SQLITE_OPEN_READONLY` plus `immutable=1`, so +SQLite refuses every statement that would write and builds no wal-index. No +`config.db-wal` and no `config.db-shm` appear beside the file, the mode is left +alone, and no migration runs — a database behind this binary's schema is +reported as a failure naming `nxdns run` as the fix. A `check` against a data +directory leaves it byte-identical, and `check` reaches the database branch only +when `config.db` is already there. + +`immutable=1` ignores any `-wal` file, so it is refused rather than used when +one holds bytes: the newest settings would be invisible and `check` would grade +older ones from the main file. That is the "uncheckpointed changes" failure in +[the CLI reference](cli.md#what-check-does-not-do). | Path | What it is | Mode | | --- | --- | --- | | `config.db` | The configuration database — the single source of truth, including `web.password_hash`. | 0600 | -| `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created when WAL is enabled, inheriting the main file's permissions. | 0600 | +| `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created by `run`, `import` and `export` when WAL is enabled, inheriting the main file's permissions. `check` creates neither. | 0600 | | `querylog.db` | The query log: every domain every client asked for. Expendable — if it is missing or unusable it is recreated empty. | 0600 | | `querylog.db-wal`, `querylog.db-shm` | WAL sidecars for `querylog.db`. | 0600 | | `querylog.db.corrupt-` | A `querylog.db` that could not be used, moved aside before an empty one was created in its place. Kept, never overwritten. | Whatever the renamed file had — no chmod reaches it | @@ -38,11 +50,45 @@ additionally creates an empty `config.db` if the directory exists without one; | `blocklists/.wild` | Wildcard entries for the same source. | 0600 | | `blocklists/.raw.tmp`, `.list.tmp`, `.wild.tmp` | Transient refresh state: the downloaded body and the two compile outputs before they are published by rename. | 0600 | -`` is the `blocklist_sources` row id. A `.list` or `.wild` file whose id is -no longer a `blocklist_sources` row is deleted by the orphan sweep; files of a -live source are left alone whatever their state, and the sweep matches only the -two published suffixes, so a `.tmp` file left by an interrupted refresh is not -swept — the next refresh of that source overwrites it. +`` is the `blocklist_sources` row id. + +### The orphan sweep + +The sweep decides by id, not by suffix. It matches all five names above and +deletes those whose `` is no longer a `blocklist_sources` row, so the +compiled `.list` and `.wild` of a removed source go, and so do a `.raw.tmp`, +`.list.tmp` or `.wild.tmp` left behind by a refresh that was killed before it +could clean up. Files belonging to a source that still has a row are never +touched, whatever state they are in: the sweep holds the same lock every refresh +takes, so it never reads the directory while a refresh is part-way through. + +It runs at three moments: + +- at startup, before the first refresh pass — this is what collects what a + killed process left behind, and the compiled files of a source deleted while + the server was down; +- before each scheduled update pass, ahead of the disk-space gate: the sweep + only unlinks, so it is the one step here that can give a critically full disk + room back, and gating it would keep the residue that helped fill the disk; +- immediately after `DELETE /api/blocklists/{id}`, which is when an orphan is + actually created in normal operation. Without it a deleted list would keep its + megabytes until the next scheduled pass. + +With `blocklist_update.enabled = false` there are no scheduled passes, so only +the first and the last of those three happen. + +Each deletion is logged: + +``` +info(blocklist_manager): pruned orphaned blocklist file 9999.list +``` + +A failed sweep is a warning, not an outage — leftover bytes do not justify +losing the refresh pass behind them, let alone the server. + +The temporaries of a source that still exists are cleaned by the refresh that +owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`, +`.list.tmp` and `.wild.tmp` as it finishes, successfully or not. A `querylog.db` is moved aside when it is missing nothing but usability: SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not @@ -111,7 +157,9 @@ no file; under systemd the journal captures it. `doh_server.cert_path` / `key_path` and `dot_server.cert_path` / `key_path`, conventionally under `/etc/nxdns`. nxdns reads them, never writes or creates -them. Both must be readable by the user nxdns runs as; the key should be -readable by its owner only, which `nxdns check` warns about when it is not. A +them. Both must be readable by the user nxdns runs as, and the key must belong +to the certificate: `nxdns check` loads the pair and fails when it does not. The +key should also be readable by its owner only, which `check` warns about when it +is not. A watcher polls both files and swaps a renewed pair in without a restart. See [enable DoH and DoT](../how-to/enable-doh-and-dot.md). diff --git a/docs/reference/performance.md b/docs/reference/performance.md index 9b79ede..1b67c70 100644 --- a/docs/reference/performance.md +++ b/docs/reference/performance.md @@ -12,8 +12,8 @@ see [measure performance](../how-to/measure-performance.md). | Sustained ≥ 100 qps on Raspberry Pi 5 | End-to-end against the real binary on the Pi; not a harness number | | Blocklist lookup p95 < 1 ms | `bench filter`: `matcher.normalize` + `Snapshot.evaluate` per op | | Cached response p95 < 5 ms | `bench cache`: `buildKey` + `DnsCache.get` + `packet.setId` per op | -| Memory with ~1M blocked domains < 100 MB | `bench filter`: VmRSS with the 1M-domain snapshot loaded | -| Stripped static binary < 10 MB per arch (< 15 MB with the embedded frontend) | CI size assert on the `cross` artifacts | +| Memory with ~1M blocked domains < 100 MiB | `bench filter`: VmRSS with the 1M-domain snapshot loaded | +| Stripped static binary < 10 MiB per arch (< 15 MiB with the embedded frontend) | CI size assert on the `cross` artifacts | The harness is `tools/bench.zig`. It measures the three targets that are measurable in process; the qps target is end to end and the binary-size target @@ -61,7 +61,7 @@ VmRSS is lower because the filter suite's snapshot has been freed by then. | --- | --- | | Blocklist lookup p95 < 1 ms | to be measured on hardware | | Cached response p95 < 5 ms | to be measured on hardware | -| Memory with ~1M blocked domains < 100 MB | to be measured on hardware | +| Memory with ~1M blocked domains < 100 MiB | to be measured on hardware | | Sustained ≥ 100 qps | to be measured on hardware, end to end | The qps target belongs to the real binary rather than the harness: it means diff --git a/docs/tutorial/first-run.md b/docs/tutorial/first-run.md index 6389edf..7185433 100644 --- a/docs/tutorial/first-run.md +++ b/docs/tutorial/first-run.md @@ -71,9 +71,9 @@ an install decision, not a first-run decision. A real install is covered in The `default` group and one enabled upstream are the two things nxdns will not start without. Every client that nxdns has never seen is assigned to `default`, and with no usable upstream there is nowhere to send a query it cannot answer -itself, so a configuration missing either one is rejected. Seeding happens -inside `run`, so `run` prints the problem and exits 1; `nxdns check` and `nxdns -import` reject the same file with exit code 2. +itself, so a configuration missing either one is rejected. Whichever command +reads the file says the same thing and stops the same way: `run`, `nxdns check` +and `nxdns import` all print the problem and exit 2. ## 4. Check the configuration before starting @@ -83,13 +83,14 @@ zig-out/bin/nxdns check --data-dir ~/nxdns-tutorial/data --config ~/nxdns-tutori ``` checking configuration file /home/you/nxdns-tutorial/config.zon -OK https://cloudflare-dns.com/dns-query +OK upstreams[0] https://cloudflare-dns.com OK: no problems found ``` `check` parses the file, validates it, and contacts each upstream to confirm it -answers. It exits 0 when there is nothing to fix and 2 when there is. It does -not start any listener, so you can run it as often as you like. +answers. It exits 2 when it found something that has to be fixed and 0 +otherwise. It writes nothing and starts no listener, so you can run it as often +as you like. ## 5. Start the server @@ -158,6 +159,13 @@ separate decision, which is what lets one group get a strict list and another get none. A source that is attached to no group is downloaded and then filters nothing. +You are in that state right now, between the previous step and this one, and it +is legal rather than wrong — creating a source and attaching it afterwards is +the normal order, which is why the API accepted it without complaint. On a +stopped server `nxdns check` names such a source in a `WARN` line and still +exits 0, so a list that silently blocks nothing is something you can find out +about later. Attaching it now is what makes it take effect. + Ask which groups exist: ```sh diff --git a/specs/milestone-13.md b/specs/milestone-13.md index 24b3448..a60d1a4 100644 --- a/specs/milestone-13.md +++ b/specs/milestone-13.md @@ -136,9 +136,56 @@ Five review rounds: 15 findings, then 8, then 1, then 1, then clean. One round-1 finding was rejected on evidence (`troubleshoot.md` already stated the verified exit codes; the contradiction was in `reference/cli.md`). -## Discrepancies found, not fixed +## Fix wave (supersedes ruling 9) -Docs-only per ruling 9. These are source-side and belong to a later milestone: +Ruling 9 kept the documentation wave docs-only so that writing pages could not +churn behaviour underneath itself. That job is done and committed at 16c9de2. +The nine discrepancies the wave found are now fixed inside milestone 13, on the +user's direction; milestone 14 stays packaging and publishing. + +Binding rulings for the fix wave: + +F-a. **One definition of "the operator's configuration is wrong."** The +divergence in D1 exists because `app.isConfigFault` and `cli.failureExitCode` +each carry their own list. Neither list is the fix. Add `src/config/faults.zig` +with exactly: + +```zig +pub fn isConfigFault(err: anyerror) bool +``` + +covering the seed and validation errors (`ParseZon`, `ConfigTooLarge`, +`MissingDefaultGroup`, `NoUpstreams`, `NoUsableUpstreams`, `BadBindAddress`, +`BadRateLimit`, `BadCertificate`, `PasswordAndHashBothSet`, and every +`validate.ValidateError`). `app.zig` and `cli.zig` both call it and keep no +private list. `run`, `check` and `import` then agree: a rejected configuration +file is exit 2 from every subcommand. + +F-b. **Diagnostics carry severity.** `validate.Diagnostics` gains a severity per +item — `.fail` or `.warn`. `writeAll` prints `FAIL `/`WARN ` accordingly. +Counting splits: failures set exit 2, warnings never change an exit code. This +is the API D2 and D5 both need, so it is pinned here rather than invented twice. + +F-c. **`check` never writes.** It opens `config.db` read-only and does not +migrate. A database behind the current schema is reported, not upgraded. If a +read-only open is impossible for a database needing WAL recovery, report that as +a failure naming `nxdns run` as the fix — do not silently fall back to a +writable open. + +F-d. **`check` proves what it claims.** It parses the certificate and the key and +verifies they pair, through the same code the server uses, so a green `check` +cannot be followed by `run` exiting 2 on `BadCertificate`. + +F-e. **Documentation follows behaviour in the same wave.** Every page that +documents the old behaviour is corrected: exit codes in `reference/cli.md`, +`how-to/troubleshoot.md`, `how-to/install-with-docker.md`, `tutorial/first-run.md`; +the certificate gap in `how-to/troubleshoot.md` and `how-to/enable-doh-and-dot.md`; +the "check writes" note in `reference/files-and-directories.md`. The three drift +guards stay green. Ruling 3 still binds: a changed command is re-run here. + +F-f. **Every fix ships with a test that fails without it.** + +## The nine discrepancies 1. `nxdns run` exits 1 for seed-file errors (`ParseZon`, `MissingDefaultGroup`, `NoUpstreams`) while `check` and `import` exit 2 for the same file, because @@ -161,6 +208,373 @@ Docs-only per ruling 9. These are source-side and belong to a later milestone: one query ignores a seed file placed afterwards. 9. PLAN §18 and the docs say 10 MB / 15 MB; CI asserts 10 MiB / 15 MiB. +## Fix wave delivered + +All nine are closed. Each shipped with a test its author watched fail with the +implementation reverted — ruling F-f was enforced by demanding the observed +failure output, not an assertion that a test would fail. + +D1 is one `src/config/faults.zig` deriving the fault set by comptime reflection +over `validate.ValidateError`, with no exclusion list; `run`, `check` and +`import` all exit 2 on the same rejected file, for `MissingDefaultGroup` and for +`ParseZon`. D2 and D5 rest on `validate.Diagnostics` gaining `.fail`/`.warn`: +`check` prints `OK: no failures found, 1 warning` and exits 0. D3 proves the +certificate pair through `cert_store.CertStore.init`, the same code the +listeners use, so a green `check` cannot be followed by `run` exiting 2 on +`BadCertificate`. D4 fixes the missing-file exit at the root read. D6 opens +`config.db` with `OpenMode.immutable` and no migrate; a stale `-wal` is reported +naming `nxdns run`, never read past. D7 sweeps all five suffixes. D8 counts +`hand_edited = 1` only. D9 is MiB everywhere CI asserts MiB. + +Four defects the wave found that were not among the nine: + +1. **`pruneOrphans` had no production caller.** D7's widened matching was + unreachable at runtime. `Manager.sweepOrphans` now runs at startup, before + each interval pass, and on `DELETE /api/blocklists/:id`. +2. **An import destroyed a device's observed timestamps.** `first_seen` and + `last_seen` are runtime state, not configuration, and the model carries no + field for either — so a configured row was inserted with the import clock in + both columns. They now follow the address: a device the database already knew + keeps them, only an unseen address takes the import's clock, and a client the + file omits is removed with its history. Verified live, not only in tests. +3. **A blocklist or upstream url reached the log whole.** A signed url or an + `?apikey=` query persisted in journald. `src/safe_url.zig` exports `redact`, + which keeps scheme, host and port and drops userinfo, path, query and + fragment. It scans rather than parses, deliberately: `error.BadUrl` is one of the + failures these very lines report, so the inputs a parser refuses are exactly + the ones that must still redact. Twenty-two call sites across `manager.zig`, + `validate.zig`, `cli.zig` and `app.zig`. +4. **`src/main.zig` built both runner writers in positional mode.** With stderr + redirected to a regular file, runner output pwrote over what `std.log` had + already written at offset 0, and `2>>` was silently broken because pwrite + ignores `O_APPEND`. Both writers now use `writerStreaming`. + +Two review findings were answered against the reviewer rather than by it, both +with evidence rather than argument: + +- **A failed stderr write must not stop the server.** The reviewer wanted + `seedFromFile` to propagate an output failure as a runtime failure. Implementing + that proposal and running the suite showed it replacing + `error.MissingDefaultGroup` with `error.WriteFailed` — a broken stderr would + hide why the seed file was refused. The discards stay, and the reasoning now + sits above them, with the derived half of the claim labelled as derived. +- **The log-injection hole was already closed for `std.log`.** `logging.zig:342` + escapes control bytes in every log message, so the forged-line scenario the + reviewer described could not happen through that path. The live gap was + `cli.zig`'s stdout. Escaping stays in `safe_url` as well as the sink, because + `validate.Diagnostics` builds `Problem.message` as an allocated string that + `web/handlers/mutations.zig` returns as a 400 body — a channel no log sink can + escape. + +Redaction ended stricter than it started. `redact` prints scheme, host and port +only: a NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, where the path +segment is the whole account identifier, so keeping the path kept the credential. +The rule has no exemption for `nxdns check`'s stdout, which is the output an +operator pastes into a bug report. Log lines identify a source by row id and name +instead, and the two duplicate diagnostics now name the other entry +(`duplicate of upstreams[0]`) rather than quoting a url that no longer shows why +the two collide. + +One credential redaction cannot remove, pinned as a test and documented rather +than hidden: a NextDNS **DoT** upstream is `tls://abcd12.dns.nextdns.io`, which +carries the same identifier in the hostname. Removing it would leave no host and +no actionable line. A hostname is resolved publicly and offered as SNI in any +case, so it is not private the way a query string is, and dropping every host +would cost every operator a diagnostic to cover one vendor's choice. The +reference page names the mitigation the operator controls instead: NextDNS also +publishes a DoH endpoint whose identifier sits in the path and is redacted whole. + +Four rounds of review hardened `redact` and each one found what the last missed. +Three of the four were the same root cause, which is worth naming because the +first two fixes treated it as bad luck: **the scanner tried to identify a host +inside text that is not a url, and guessed.** It guessed the query was safe to cut +before the userinfo, leaving `user:pa55` as the host. It guessed a `\` was not a +separator. And it guessed which side of an `@` was the host when a late delimiter +made both readings available. + +That third one was recorded here, in an earlier revision of this section, as an +accepted cost: an `@` in a backslash path prints "a misleading host, never a +credential". **That claim was false and is retracted.** Running the shipped code +disproved it: + +``` +https://lists.example?token=prefix@hunter2 -> https://hunter2 +https://lists.example#f@hunter2 -> https://hunter2 +https:\\lists.example\p@hunter2 -> https://hunter2 +``` + +A query string is the most likely place in a url for a token, so the text the +scanner promoted to "host" was the secret itself. The note claiming otherwise is +why three subsequent rounds passed over it. + +The fix is a rule rather than a fourth special case: **the scan no longer guesses, +it declines.** The `/` cut now runs first and unconditionally, which is safe for +the reason the old ordering missed — everything after the `/` is dropped anyway, +so an `@` there never needed to be userinfo to stay out of the log. Where two +readings genuinely survive, the authority is omitted whole and `format` prints +`(ambiguous authority omitted)`, which is prose rather than a placeholder host so +an operator reads it as a statement about the line. `SafeUrl.authority` is +therefore `?[]const u8`: empty and `null` are different answers, one saying the +url names no authority and the other saying it names one that cannot be resolved. +The same rule caught a case nobody had raised — `https:a@hunter2`, where RFC 3986 +reads `hunter2` as a path segment and WHATWG reads it as the host. The +disagreement between two parsers is itself the evidence of ambiguity. + +A redesign was considered and rejected: parse with `std.Uri.parse` first and print +nothing but the scheme when the parse fails. It reaches the same "do not guess" +place, but it also discards the host for every url malformed in a harmless way, +and `redact` exists to be callable from the `error.BadUrl` paths that report +exactly those. The ordering fix gets the property without the cost. + +Escaping is owned by the type that introduces the delimiter: `quoteText` writes +its own quotes and escapes `'` inside them, so a caller cannot reopen the hole by +adding quotes of its own. Escaping inside a plain helper would have left the +defect one caller away, which on a third review round is not a fix. + +`SafeUrl` was then found to break that same rule from the other side. Its `format` +hardcoded the `none` delimiter, so it never escaped `'` — while eight call sites +wrapped it in `'{f}'` of their own. `https://ho'st/x` redacts to `https://ho'st`, +which inside a caller's quotes reads as `'ho'` followed by loose text. Two +independent findings converged on it: this review, and the `/metrics` work, which +hit the same shape with `"` instead of `'`. + +`redactQuoted` closes it, and the choice between the two is a stated rule rather +than per-call-site judgement: **use the quoted form whenever anything follows the +url on the line**, because a redacted authority can still hold a space, a `:` and a +`'`, and unquoted it can impersonate whatever comes next — `upstream {f} failed: +{t}` with an authority of `ok failed: Timeout` reports a failure that did not +happen. Bare `redact` is for the two cases where that cannot arise: the url ends +the line (`cli.zig`'s `OK upstreams[N]`, `context.zig`'s missing-id line), or the +caller owns the escaping for a delimiter of its own (`metrics.zig`). Ten sites +moved to the quoted form. The output is byte-identical for any url without a `'`, +so no documented output changed. + +One property is asserted rather than assumed, because it is what makes a `\` in the +output always this file's and never the operator's: a `\` ends an authority, so +unlike a source name it can never reach the value to be doubled. + +Round five found two more, both in the same scanner, which is now five rounds and +five findings. Both were confirmed by running the code before being fixed. + +The first is the ambiguity rule applied to only half its cases. `https:a@hunter2` +was withheld, but `https:hunter2` — the same opaque path with no `@` in it — +printed whole, because the check sat *after* the `@` lookup and a url with no `@` +returned before reaching it. The reading is ambiguous either way; the `@` was never +what made it so. The check now runs before the `@` lookup. + +Moving it exposed the reason it had been placed there: `localhost` satisfies the +scheme production, so a rule keyed on the production alone withholds +`localhost:8080/x`, which is the shape an operator on a LAN is most likely to +write. A port is not an opaque path, so the digits after the colon are what +separate the two. This is checked against the trimmed authority, not the raw one, +or `localhost:8080?x` would fail the digit test on the query. + +The second: a network-path reference (`//lists.example/hosts.txt`) redacted to the +empty string, because the unconditional `/` cut lands at byte zero. Not a leak — +nothing was printed — but the line named no source at all, and the authority in it +is not in doubt. A leading run of `/` is now consumed the way a scheme delimiter's +is, so the userinfo in `//user:pa55@lists.example/x` is dropped rather than the +whole authority withheld. An ambiguous one such as `//a?b@c` is still withheld: the +branch settles where the authority starts, not that every reading of it is +resolved. + +Round six found three more in the same function, all the same class again: a +separator run that was read as an authority delimiter when it does not settle one. + +`https:/hunter2` and `https:///hunter2` printed the path segment as the host. The +scan accepted a delimiter of *any* number of separators, and the comment recording +why was accurate when it was written and stale by the time it was read: the +tolerance existed so `https:/user:pass@host/list` would find an authority instead +of printing its userinfo. That reason expired when the `/` cut moved ahead of the +userinfo lookup in round four — the authority of a url with no `://` now ends at +its first `/`, so it holds no userinfo to print. Only a run of exactly two +introduces an authority; one leaves an absolute path and three or more is an empty +authority to RFC 3986 and a host to WHATWG. The four inputs that motivated the old +tolerance are still safe, now by being withheld rather than resolved, and that is +asserted where the old behaviour used to be. + +`///lists.example/x` had the same defect in the network-path branch added one +round earlier, which consumed the whole run. Exactly two there too. + +The third retracts something this section claimed one round ago. `isPort` was +introduced so a schemeless `localhost:8080` would keep resolving, on the reasoning +that a port is not an opaque path. It is not that simple: `https:123456` is an +opaque path whose digits are as much a token as any other text, and the exception +printed it whole. The reviewer proposed excluding known schemes from the +exception; the exception is gone instead, because a list of known schemes is the +kind of thing that goes stale silently and this file already has one rule that +covers it. `localhost:8080/x` is now withheld, which costs nothing an operator +needs: a url reaching this without a scheme is one the validator is rejecting, and +the field path beside it names which. An `IP:port` is unaffected — a leading digit +fails the scheme production, so `10.0.0.2:8080` and `[::1]:853` still resolve. + +Round seven took the same rule one step further. "Exactly two separators" still +accepted `\\`, `/\` and `\/`, and WHATWG converts a `\` to a `/` only for a +*special* scheme — so `https:\\hunter2` has the same split reading as the cases +above, and `tls:\\hunter2` has no reading at all under which `hunter2` is a host, +`tls` not being special. RFC 3986 gives `\` no meaning anywhere. Only `//` opens +an authority now. A `\` still ends one, and still anchors the scheme scan so a +backslash-pasted url is reported by its scheme, but it opens nothing. + +Round seven also caught a stale comment, which is the second time in two rounds +that a comment outlived the reason it recorded. Both said something true when +written and false when read, and both were load-bearing — the first is why the +any-length tolerance survived three rounds after its justification expired. + +Seven rounds, nine findings, one function. Every one was a place where the scan +committed to a reading the text did not support. What finally holds is not a +sharper scan but a smaller claim: the authority is printed only where exactly one +reading survives, and withheld everywhere else. Concretely, an authority is read +only after a literal `//`, whether a scheme introduced it or not. + +The cost is paid by malformed input alone, and it is paid in diagnostic detail +rather than in safety: a backslash-pasted or schemeless url now reports its scheme +and no host. Every such url is one the validator is already rejecting, and the +field path or row id beside it names which one. A sweep of eighteen shapes +carrying `hunter2`, `abcd12`, `s3cr3t`, `token`, `pa55` and `123456` finds none of +them in any output. + +Round eight returned no critical and no important finding, and two minor ones that +cut in opposite directions. A run of three or more slashes was reported as an +authority the url does not have, when it is contested rather than absent — RFC +3986 reads an empty authority, WHATWG resolving against a special-scheme base +reads a host. `SafeUrl` exists to keep those two answers apart, so it is now +withheld rather than reported as empty. + +The other corrected a claim this file made about the very thing it was withholding. +`localhost:8080` is **not** contested: `localhost` is not one of WHATWG's six +special schemes, so both standards read a scheme and an opaque path, and the +host-and-port an operator meant is a reading no parser offers. It is still +withheld — the text after the colon is a path segment under every reading — but the +marker calls it an authority that could not be resolved when the honest answer is +that there is none. + +That wording defect was first recorded rather than fixed, on the grounds that +telling the two apart needs a scheme list and a scheme list is what the previous +round had just removed. Round nine rejected that and was right to: the list round +six removed described *what nxdns supports*, so it went stale whenever a transport +was added. WHATWG's special schemes are a closed set fixed by the URL Standard — +`ftp`, `file`, `http`, `https`, `ws`, `wss` — which never described nxdns and +cannot go stale with it. Conflating the two was the error, and knowingly shipping +a diagnostic that contradicts its own type's contract is the tech debt this +project does not take on. + +So `redact` now distinguishes them. Only a special scheme can disagree with RFC +3986 about text no `//` introduced, so `https:hunter2` is withheld as contested +and `localhost:8080`, `mailto:ops@example.com` and `tls:\\host` are withheld as +naming no authority at all. Round nine's other finding was the same rule missing +from the schemeless branch: `\\lists.example\path` and `/\lists.example/path` were +reported as absent when they are contested. A run is settled when it is one +separator, or when the reading that looks for a host finds none — +`\\?\C:\lists\hosts.txt` ends its authority at the `?` under both — and contested +otherwise. + +Round ten returned nothing critical and nothing important, four minor findings and +a nit. Two were taken: the nit, which was a comment claiming +`https:\\dns.nextdns.io\abcd12` "names a host" when the file's own reasoning is +that it is contested; and the rendering question — `tls:\\host` prints `tls://` +although the input held no `//`. That one is now contractual rather than +accidental. `format` renders `scheme://authority` canonically and does not quote +the input's syntax; nothing about which bytes separated the scheme survives +redaction, and nothing should, because the input is not meant to be reconstructible +from the output. + +**Three findings were declined, and the reason is checked rather than asserted.** +`https:/user@/x`, `\path@hunter2` and `file:secret` are classified as contested +when both readings in fact find no host. Running them shows why that is tolerable: +each prints `(ambiguous authority omitted)`, so each says *less* than it could and +none prints the path segment. The mechanisms are named in the file — emptiness +tested before userinfo removal, a leading run of one reaching the late-delimiter +rule, and `file` having its own WHATWG parsing states that `isSpecialScheme` does +not model. Fixing them means modelling more of two standards for inputs no +accepted configuration can hold: every url this program takes carries `http`, +`https`, `tls`, `udp` or `tcp` and a `//`. All three are pinned by a test, because +the failure that would matter is the opposite one — if any of them ever starts +naming a host, that test fails. + +That is where the review loop was stopped, on a judgement rather than on an empty +round. Rounds eight, nine and ten each returned nothing critical and nothing +important, and the findings had moved from "this prints a secret" to "this claims +more than it knows about a url no operator can configure". Round ten opened a new +sub-class of its own — WHATWG's `file:` state machine — which is the signal that +the remaining work is unbounded and no longer about safety. + +Ten rounds, sixteen findings, one function. A sweep of twenty-five shapes carrying +`hunter2`, `abcd12`, `s3cr3t`, `token`, `pa55`, `123456`, `hosts.txt` and +`example.com` finds none of them in any output. + +The last gap was found by sweeping every log site in the tree rather than +trusting the review's file list: `src/upstream/dot_client.zig` printed the +upstream url whole at four sites and `pool.zig` at a fifth. Three review rounds +and four agents had redacted urls across `manager.zig`, `validate.zig`, `cli.zig` +and `app.zig` without anyone asking which other modules logged the same values. + +Deliberately not changed: `POST /api/blocklists` still ignores the +`SourceInNoGroup` warning. The web flow creates a source before any group link +can exist, so the warning is structural at that moment and a 400 would be wrong. +The three CLI paths carry it instead. + +Claims in the wave that no test backs, recorded rather than buried: the two +`app.zig` `log.warn` redactions, because a `std.log` line is not observable from +a unit test under the default runner; `cli.zig`'s `not a usable DoH url` +redaction, argued unreachable by any credential-carrying url because +`Endpoint.parse` rejects `@?#` in the authority first; the two probe lines that +need a reachable upstream; and the WAL guard's residual window, which is argued +rather than observed. + +## Round four + +The `/metrics` family was the last place a credential still reached an +unauthenticated reader. `GET /metrics` is `.auth = .open` in `src/web/routes.zig` +and `web.bind` defaults to `0.0.0.0`, so an unauthenticated `curl` printed +`nxdns_upstream_up{url="https://dns.nextdns.io/abcd12"} 1` with HTTP 200. Every +other redaction in this wave was on a path that already required a session or a +shell on the host; this one was not. + +Redacting it needed a second escaping layer rather than a call to `redact`. +`redact` cuts the authority at `/@?#\` and escapes every control byte, so it can +emit neither a raw newline nor a lone backslash — but `"` is in none of those cut +sets, so `https://ho"st/x` arrives at the label as `https://ho"st` and closes the +label value, letting the rest of the string write label pairs of its own. +`writeLabelValue` therefore runs *over* the redacted text. The ordering also +matters in the other direction: doubling the backslash turns the two characters +`redact` writes for a control byte into an unambiguous `\\n`, so a parser reads a +backslash rather than a newline. + +The redaction then introduced a defect of its own, caught before it shipped: two +upstreams on one host redact to one label set, so `nxdns_upstream_up` rendered +twice with identical labels in a single exposition. Two NextDNS profiles is a +realistic configuration. The fix labels each sample with its array position in +`renderUpstreams`, so uniqueness holds by construction and a hand-built `Sample` +cannot forge a collision. The index is positional and therefore stable only while +the pool order is — `pool.Snapshot` carries no row id, and threading one through +the pool and the repository was judged out of scope for a review round. The +caveat is in the file. It only bites when two upstreams share an origin; for +distinct origins the url carries the identity and a reorder is harmless. + +Claim scoping on that defect, since the three statements are not one: the +duplicate exposition text **was** reproduced, out of the same `render` function +`/metrics` calls. It was **not** reproduced over HTTP from a pre-fix server — every +live run had the index in place. And "a scrape carrying it is rejected" is +**reasoned, not observed**; nothing was fed to a Prometheus. Whether a scraper +rejects the pair or keeps the last sample, the down upstream is lost, which is the +part the fix rests on. + +`src/storage/repositories/context.zig` printed a blocklist url whole and a group +name raw on its two `NotFound` paths. Both are invariant-failure paths that fire +rarely, which is why they outlived three rounds of sweeping. + +One test file was never running. `src/config/faults.zig` was absent from +`src/tests.zig`, and Zig collects tests only from the root module's explicit +import list — a transitively imported file contributes none, even when its values +are used, which was confirmed against a scratch project rather than assumed. The +five reflection guards in that file are what catch a future `ValidateError` +variant being added and left unclassified; they had never executed. Its runtime +behaviour was covered — `cli.zig` has a test that classifies through it — so the +gap was in the regression guard, not in what shipped. A sweep of the tree found no +other unlisted file with tests; `dns/dns.zig` is a re-export barrel with none, and +its own comment documents this rule. + ## Acceptance (milestone complete) - [x] The five old `docs/*.md` files are gone; the four directories and the index @@ -168,9 +582,9 @@ Docs-only per ruling 9. These are source-side and belong to a later milestone: - [x] Every command in `tutorial/` and `how-to/` was executed on this host, or is marked in-page as unverified with a reason. - [x] `zig build test` passes with the three drift guards pointing at the new - reference pages: 16/16 steps, 1175 pass, 113 skip (integration-gated), 0 failed. -- [x] `zig build test -Dintegration` passes: 1284 pass, 4 skip (the live-network - TLS tests excluded by milestone-1 design), 0 failed. No src/ behavior change. + reference pages: 16/16 steps, 1254 pass, 115 skip (integration-gated), 0 failed. +- [x] `zig build test -Dintegration` passes: 16/16 steps, 1365 pass, 4 skip (the + live-network TLS tests excluded by milestone-1 design), 0 failed. - [x] No page mixes modes: no procedure in `reference/`, no field table in any `how-to/` or `tutorial/` page. @@ -179,5 +593,6 @@ Docs-only per ruling 9. These are source-side and belong to a later milestone: - No new documentation subjects (no metrics-families reference, no security-model page) — this milestone moves and splits existing content plus the tutorial. - No redirect stubs or `docs/legacy/`. -- No source fixes; report discrepancies instead. +- No source fixes; report discrepancies instead. (Held for the documentation + wave. Superseded by the fix wave above, which closes the nine it reported.) - No doc generator, no site builder, no MkDocs. diff --git a/src/app.zig b/src/app.zig index c0c1893..29d5e68 100644 --- a/src/app.zig +++ b/src/app.zig @@ -44,6 +44,7 @@ const doh_client = @import("upstream/doh_client.zig"); const doh_server = @import("server/doh_server.zig"); const dot_client = @import("upstream/dot_client.zig"); const dot_server = @import("server/dot_server.zig"); +const faults = @import("config/faults.zig"); const fetcher = @import("filter/fetcher.zig"); const forward_zones = @import("local/forward_zones.zig"); const handler = @import("server/handler.zig"); @@ -60,6 +61,7 @@ const pool_mod = @import("upstream/pool.zig"); const query_sink = @import("server/query_sink.zig"); const rate_limiter = @import("server/rate_limiter.zig"); const retention_mod = @import("storage/retention.zig"); +const safe_url = @import("safe_url.zig"); const shutdown = @import("server/shutdown.zig"); const sse = @import("web/sse.zig"); const static = @import("web/static.zig"); @@ -87,24 +89,14 @@ const download_budget_s = 300; const doh_request_buf_len = 1024; const doh_transfer_buf_len = 4096; -/// A configuration fault the operator can fix, as opposed to a runtime one. -/// These are the only failures this file raises itself; everything else comes -/// out of a collaborator. -const ConfigError = error{ - NoUsableUpstreams, - BadBindAddress, - BadRateLimit, - BadCertificate, -}; - pub fn run(runner: cli.Runner, args: cli.RunArgs) u8 { const code = serve(runner, args) catch |err| code: { runner.err.print("nxdns run failed: {s}\n", .{@errorName(err)}) catch {}; - if (isConfigFault(err)) { + const mapped = failureExitCode(err); + if (mapped == cli.exit_check) { runner.err.writeAll("run `nxdns check` to see the configuration in full\n") catch {}; - break :code cli.exit_check; } - break :code cli.exit_runtime; + break :code mapped; }; // Output the operator never received is not output, so a failed flush @@ -114,15 +106,75 @@ pub fn run(runner: cli.Runner, args: cli.RunArgs) u8 { return code; } -fn isConfigFault(err: anyerror) bool { - return switch (err) { - error.NoUsableUpstreams, - error.BadBindAddress, - error.BadRateLimit, - error.BadCertificate, - => true, - else => false, - }; +/// The one classification, from `config/faults.zig`. This file keeps no list of +/// its own: `run` exiting 1 on a seed file `check` and `import` exit 2 on was +/// exactly the cost of the second list that used to be here. +fn failureExitCode(err: anyerror) u8 { + return if (faults.isConfigFault(err)) cli.exit_check else cli.exit_runtime; +} + +/// First run only: the file seeds an empty database and is ignored forever +/// after. Its diagnostics are the operator's one chance to see what the file +/// said, so they are printed the way `check` and `import` print them. +/// +/// Printed on the way out either way. A seed file can be accepted and still +/// carry warnings — a blocklist source in no group is downloaded and compiled +/// into nothing — and a warning that only appears when the start fails is a +/// warning nobody ever reads: the start it describes is the one that worked. +/// `check` reported it and `run` did not, which left the same file graded two +/// ways. +/// +/// The runner's error writer, not `std.log`: this runs before +/// `logging.install`, and one rendering of a diagnostic across `run`, `check` +/// and `import` is the point of `Diagnostics.writeAll`. +/// +/// Flushed here rather than left to `run`'s exit flush. That writer is buffered +/// (`main` gives it 4 KiB) and `serve` does not return for as long as the +/// service runs, so a line left in the buffer reaches the operator when the +/// process stops — days after the start it describes. A failure path flushes +/// anyway because it returns immediately; the successful start is the one that +/// needs this. +fn seedFromFile( + r: cli.Runner, + config_db: *db.Db, + dir: std.Io.Dir, + config_path: []const u8, +) bootstrap.Error!bootstrap.Outcome { + var diags: validate.Diagnostics = .init(r.gpa); + defer diags.deinit(); + + const result = bootstrap.bootstrap(r.io, r.gpa, config_db, dir, config_path, &diags); + + // Neither discard is an oversight, and the two answer different questions. + // + // On a rejected seed file, `result` is returned untouched: the operator gets + // the reason the start failed, never a writer error standing in front of it. + // A broken stderr is not why the configuration was refused. + // + // On a seed that worked, a failure here does not stop the start. The trade + // is one lost warning line against a household with no name resolution, and + // `run` before this point is the only stretch of this program where an + // output failure could take DNS down at all — ruling 4 already says nothing + // after it is fatal. Nor could the failure be reported: this writer *is* the + // error channel, and `logging.install` has not run yet, so `std.log` resolves + // to the same stderr a diagnostic about it would have to travel down. + // + // It is not lost from the process either. A failed drain consumes nothing, + // so whatever the buffer held it still holds — that half is observed, in + // "a broken error writer does not stop a first start that succeeded" below, + // which reads the retained warning back out of the same writer. + // + // What happens to those bytes afterwards is derived, not watched, and is + // labelled so deliberately. `Io.Writer.defaultFlush` drains while `end != 0` + // and `run`'s exit flush maps a failure to exit 1, so a stderr still broken + // at shutdown should carry the condition out in the exit code, and one that + // recovered should deliver the line late. No test drives `run` that far. + // Two limits come with the derivation: an empty buffer flushes clean and + // reports nothing at all, and a failure that recovers ends at exit 0 with a + // line the operator reads days after the start it describes. + diags.writeAll(r.err) catch {}; + r.err.flush() catch {}; + return result; } fn serve(r: cli.Runner, args: cli.RunArgs) !u8 { @@ -141,18 +193,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 { defer config_db.close(); _ = try migrations.migrate(&config_db); - { - // First run only: the file seeds an empty database and is ignored - // forever after. Its diagnostics are the operator's one chance to see - // why a config file was rejected, so they are printed like `check` - // prints them. - var diags: validate.Diagnostics = .init(gpa); - defer diags.deinit(); - _ = bootstrap.bootstrap(io, gpa, &config_db, std.Io.Dir.cwd(), paths.config, &diags) catch |err| { - diags.writeAll(r.err) catch {}; - return err; - }; - } + _ = try seedFromFile(r, &config_db, std.Io.Dir.cwd(), paths.config); // Every string in `cfg` points into this arena, and the pool's endpoints, // the handler's records and the monitor's paths all keep such strings. It @@ -717,7 +758,7 @@ const Upstreams = struct { http: *std.http.Client, bundle: *Certificate.Bundle, bundle_lock: *std.Io.RwLock, - ) (Allocator.Error || ConfigError)!Upstreams { + ) (Allocator.Error || error{NoUsableUpstreams})!Upstreams { var enabled: usize = 0; for (servers) |server| { if (server.enabled) enabled += 1; @@ -748,7 +789,10 @@ const Upstreams = struct { if (!server.enabled) continue; const endpoint = transport.Endpoint.parse(server.url) catch { - log.warn("upstream '{s}' is not an https:// or tls:// endpoint; skipped", .{server.url}); + log.warn( + "upstream {f} is not an https:// or tls:// endpoint; skipped", + .{safe_url.redactQuoted(server.url)}, + ); continue; }; @@ -762,7 +806,10 @@ const Upstreams = struct { self.doh_buf[base..][0..doh_request_buf_len], self.doh_buf[base + doh_request_buf_len ..][0..doh_transfer_buf_len], ) catch { - log.warn("upstream '{s}' is not a usable DoH url; skipped", .{server.url}); + log.warn( + "upstream {f} is not a usable DoH url; skipped", + .{safe_url.redactQuoted(server.url)}, + ); continue; }; doh_count += 1; @@ -888,6 +935,280 @@ test "parseBind refuses a bind address of the wrong family" { try std.testing.expect(std.mem.containsAtLeast(u8, printed, 1, "is not an IPv4 address")); } +test "run maps a rejected configuration to exit 2 and everything else to exit 1" { + try std.testing.expectEqual(cli.exit_check, failureExitCode(error.MissingDefaultGroup)); + try std.testing.expectEqual(cli.exit_check, failureExitCode(error.ParseZon)); + try std.testing.expectEqual(cli.exit_check, failureExitCode(error.NoUsableUpstreams)); + try std.testing.expectEqual(cli.exit_check, failureExitCode(error.BadCertificate)); + try std.testing.expectEqual(cli.exit_runtime, failureExitCode(error.AccessDenied)); + try std.testing.expectEqual(cli.exit_runtime, failureExitCode(error.OutOfMemory)); +} + +test "run, check and import agree on a seed file with no default group" { + const config_import = @import("config/import.zig"); + + var threaded: std.Io.Threaded = .init(std.testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = std.testing.allocator; + + // The file from discrepancy D1: parseable, one upstream, no group named + // 'default'. + const source: [:0]const u8 = + \\.{ + \\ .groups = .{ .{ .name = "kids" } }, + \\ .upstreams = .{ .{ .url = "https://dns.example/dns-query" } }, + \\} + ; + + // `run`: `serve` seeds through `bootstrap`, which is a wrapper over this + // exact call, so this is the error `run` classifies. + var database = try db.Db.open(":memory:", .{ .mode = .memory }); + defer database.close(); + try db.applyPragmas(&database, .{}); + _ = try migrations.migrate(&database); + + var diags: validate.Diagnostics = .init(gpa); + defer diags.deinit(); + try std.testing.expectError( + error.MissingDefaultGroup, + config_import.importSource(io, gpa, &database, source, .{}, &diags), + ); + try std.testing.expectEqual(cli.exit_check, failureExitCode(error.MissingDefaultGroup)); + + // `import`: `cli.failureExitCode` reaches exit 2 by either route — the + // recorded failures, or the classification `run` just used. + try std.testing.expect(diags.failureCount() != 0); + try std.testing.expect(faults.isConfigFault(error.MissingDefaultGroup)); + + // `check`: the same file, through the code `nxdns check` runs. + var arena_state: std.heap.ArenaAllocator = .init(gpa); + defer arena_state.deinit(); + const cfg = try std.zon.parse.fromSliceAlloc(model.Config, arena_state.allocator(), source, null, .{}); + + var out_buf: [2048]u8 = undefined; + var err_buf: [256]u8 = undefined; + var out: Writer = .fixed(&out_buf); + var err_writer: Writer = .fixed(&err_buf); + const r: cli.Runner = .{ .io = io, .gpa = gpa, .out = &out, .err = &err_writer }; + + try std.testing.expectEqual(cli.exit_check, try cli.checkConfig(r, cfg, false)); +} + +test "a configuration whose blocklist source is in no group imports and checks clean" { + const config_import = @import("config/import.zig"); + + var threaded: std.Io.Threaded = .init(std.testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = std.testing.allocator; + + // A source is created before it is attached — `docs/tutorial/first-run.md` + // POSTs the blocklist and then PUTs the group's sources — so an unattached + // source is a legal intermediate state on every write path. It warns; it + // never fails. + const source: [:0]const u8 = + \\.{ + \\ .groups = .{ .{ .name = "default" } }, + \\ .upstreams = .{ .{ .url = "https://dns.example/dns-query" } }, + \\ .blocklist_sources = .{ .{ .url = "https://lists.example/hosts.txt", .name = "ads" } }, + \\} + ; + + var database = try db.Db.open(":memory:", .{ .mode = .memory }); + defer database.close(); + try db.applyPragmas(&database, .{}); + _ = try migrations.migrate(&database); + + var diags: validate.Diagnostics = .init(gpa); + defer diags.deinit(); + try config_import.importSource(io, gpa, &database, source, .{}, &diags); + try std.testing.expectEqual(@as(usize, 0), diags.failureCount()); + try std.testing.expectEqual(@as(usize, 1), diags.warningCount()); + + var arena_state: std.heap.ArenaAllocator = .init(gpa); + defer arena_state.deinit(); + const cfg = try std.zon.parse.fromSliceAlloc(model.Config, arena_state.allocator(), source, null, .{}); + + var check_diags: validate.Diagnostics = .init(gpa); + defer check_diags.deinit(); + // No error means no subcommand may reject it; the warning is report-only. + try validate.validate(cfg, &check_diags); + try std.testing.expectEqual(@as(usize, 0), check_diags.failureCount()); + try std.testing.expectEqual(@as(usize, 1), check_diags.warningCount()); +} + +test "a first start that seeds from a file prints the warnings the file earned" { + // D5, second half. The seed file is read once in the life of a database, so + // a warning it earns is printed on that start or never. `run` printed + // diagnostics only when the file was rejected, which made a successful first + // start the one place the finding could not surface. + var threaded: std.Io.Threaded = .init(std.testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = std.testing.allocator; + + var tmp = std.testing.tmpDir(.{}); + defer tmp.cleanup(); + try tmp.dir.writeFile(io, .{ .sub_path = "config.zon", .data = + \\.{ + \\ .groups = .{ .{ .name = "default" } }, + \\ .upstreams = .{ .{ .url = "https://dns.example/dns-query" } }, + \\ .blocklist_sources = .{ .{ .url = "https://lists.example/hosts.txt", .name = "ads" } }, + \\} + }); + + var database = try db.Db.open(":memory:", .{ .mode = .memory }); + defer database.close(); + try db.applyPragmas(&database, .{}); + _ = try migrations.migrate(&database); + + // A buffered file writer, the shape `main` builds over stderr, and not + // `Writer.fixed`: a fixed writer's flush is a no-op, so it counts a line + // still sitting in the buffer as delivered. Reading the file back is the + // only way to ask what the operator can actually see. + var out_buf: [64]u8 = undefined; + var out: Writer = .fixed(&out_buf); + var err_file = try tmp.dir.createFile(io, "stderr.txt", .{}); + defer err_file.close(io); + var err_buf: [4096]u8 = undefined; + var err_writer = err_file.writer(io, &err_buf); + const r: cli.Runner = .{ .io = io, .gpa = gpa, .out = &out, .err = &err_writer.interface }; + + // The file is valid, so the start succeeds and the database is seeded. + try std.testing.expectEqual( + bootstrap.Outcome.seeded, + try seedFromFile(r, &database, tmp.dir, "config.zon"), + ); + try std.testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM upstreams")); + + // Nothing flushes here on purpose. In production `serve` runs from this + // point until the service stops, so a line that has not reached the file by + // now is a line the operator does not get for days. + const printed = try tmp.dir.readFileAlloc(io, "stderr.txt", gpa, .limited(8192)); + defer gpa.free(printed); + + try std.testing.expect(std.mem.containsAtLeast(u8, printed, 1, "WARN blocklist_sources[0]: ")); + try std.testing.expect(std.mem.containsAtLeast(u8, printed, 1, "belongs to no group")); + // A warning is not a rejection: nothing here claims the start failed. + try std.testing.expectEqual(@as(usize, 0), std.mem.count(u8, printed, "FAIL")); +} + +/// A broken stderr, in the shape `main` builds: a buffered `File.Writer`, with +/// its drain switched to the mode that fails. `Writer.fixed` cannot stand in — +/// its flush is `noopFlush`, so it has no failure to report and its `written()` +/// counts a line still sitting in the buffer as delivered. +/// +/// The file stays empty for as long as the mode is `.failure`, which is what +/// lets a test tell "the writer really failed" from "the writer worked". +fn brokenErrWriter(io: std.Io, file: std.Io.File, buffer: []u8) std.Io.File.Writer { + var w = file.writer(io, buffer); + w.mode = .failure; + return w; +} + +test "a broken error writer does not replace the reason a seed file was rejected" { + // The operator has to see why seeding failed, and a broken stderr is not + // that reason. + var threaded: std.Io.Threaded = .init(std.testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = std.testing.allocator; + + var tmp = std.testing.tmpDir(.{}); + defer tmp.cleanup(); + // No group named 'default': rejected, and it records a FAIL line on the way. + try tmp.dir.writeFile(io, .{ .sub_path = "config.zon", .data = + \\.{ + \\ .groups = .{ .{ .name = "kids" } }, + \\ .upstreams = .{ .{ .url = "https://dns.example/dns-query" } }, + \\} + }); + + var database = try db.Db.open(":memory:", .{ .mode = .memory }); + defer database.close(); + try db.applyPragmas(&database, .{}); + _ = try migrations.migrate(&database); + + var out_buf: [64]u8 = undefined; + var out: Writer = .fixed(&out_buf); + var err_file = try tmp.dir.createFile(io, "stderr.txt", .{}); + defer err_file.close(io); + // Eight bytes: no FAIL line fits, so `writeAll` must drain mid-line and is + // itself the call that fails. The test below covers the other discard, where + // the line fits and only the flush fails. + var err_buf: [8]u8 = undefined; + var err_writer = brokenErrWriter(io, err_file, &err_buf); + const r: cli.Runner = .{ .io = io, .gpa = gpa, .out = &out, .err = &err_writer.interface }; + + try std.testing.expectError( + error.MissingDefaultGroup, + seedFromFile(r, &database, tmp.dir, "config.zon"), + ); + + // Empty, so the writer did fail — without this the assertion above would + // hold just as well against a writer that worked. + const printed = try tmp.dir.readFileAlloc(io, "stderr.txt", gpa, .limited(8192)); + defer gpa.free(printed); + try std.testing.expectEqual(@as(usize, 0), printed.len); +} + +test "a broken error writer does not stop a first start that succeeded" { + // The call this file makes: a DNS server for a household does not refuse to + // resolve because stderr is broken. What it must not do is drop the warning + // on the floor, so the second half checks the buffer still holds it. + var threaded: std.Io.Threaded = .init(std.testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = std.testing.allocator; + + var tmp = std.testing.tmpDir(.{}); + defer tmp.cleanup(); + // Valid, and it earns one warning: the source belongs to no group. + try tmp.dir.writeFile(io, .{ .sub_path = "config.zon", .data = + \\.{ + \\ .groups = .{ .{ .name = "default" } }, + \\ .upstreams = .{ .{ .url = "https://dns.example/dns-query" } }, + \\ .blocklist_sources = .{ .{ .url = "https://lists.example/hosts.txt", .name = "ads" } }, + \\} + }); + + var database = try db.Db.open(":memory:", .{ .mode = .memory }); + defer database.close(); + try db.applyPragmas(&database, .{}); + _ = try migrations.migrate(&database); + + var out_buf: [64]u8 = undefined; + var out: Writer = .fixed(&out_buf); + var err_file = try tmp.dir.createFile(io, "stderr.txt", .{}); + defer err_file.close(io); + // 4 KiB, the buffer `main` gives the real stderr writer: the warning fits, + // so `writeAll` succeeds into the buffer and the flush is what fails. That + // is the shape production hits. + var err_buf: [4096]u8 = undefined; + var err_writer = brokenErrWriter(io, err_file, &err_buf); + const r: cli.Runner = .{ .io = io, .gpa = gpa, .out = &out, .err = &err_writer.interface }; + + try std.testing.expectEqual( + bootstrap.Outcome.seeded, + try seedFromFile(r, &database, tmp.dir, "config.zon"), + ); + try std.testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM upstreams")); + + // Nothing reached the file, so the flush really did fail. + const undelivered = try tmp.dir.readFileAlloc(io, "stderr.txt", gpa, .limited(8192)); + defer gpa.free(undelivered); + try std.testing.expectEqual(@as(usize, 0), undelivered.len); + + // And the warning is still buffered rather than dropped: this is the same + // writer, and these are the bytes `run`'s exit flush meets on the way out. + err_writer.mode = .positional; + try err_writer.interface.flush(); + const printed = try tmp.dir.readFileAlloc(io, "stderr.txt", gpa, .limited(8192)); + defer gpa.free(printed); + try std.testing.expect(std.mem.containsAtLeast(u8, printed, 1, "WARN blocklist_sources[0]: ")); +} + fn reportBind(r: cli.Runner, which: []const u8, addr: net.IpAddress, err: anyerror) anyerror { r.err.print("cannot bind {s} {f}: {s}\n", .{ which, addr, @errorName(err) }) catch {}; return err; diff --git a/src/cli.zig b/src/cli.zig index e7b26de..50932a4 100644 --- a/src/cli.zig +++ b/src/cli.zig @@ -20,9 +20,11 @@ const tls = std.crypto.tls; const app = @import("app.zig"); const config_export = @import("config/export.zig"); +const faults = @import("config/faults.zig"); const import = @import("config/import.zig"); const model = @import("config/model.zig"); const validate = @import("config/validate.zig"); +const cert_store = @import("server/cert_store.zig"); const db = @import("storage/db.zig"); const migrations = @import("storage/migrations.zig"); const querylog_schema = @import("storage/querylog_schema.zig"); @@ -30,6 +32,7 @@ const doh_client = @import("upstream/doh_client.zig"); const dot_client = @import("upstream/dot_client.zig"); const pool = @import("upstream/pool.zig"); const transport = @import("upstream/transport.zig"); +const safe_url = @import("safe_url.zig"); const version = @import("version.zig"); pub const exit_ok: u8 = 0; @@ -436,7 +439,7 @@ pub fn runImport(r: Runner, args: ImportArgs) u8 { // should need one run to see the whole list. diags.writeAll(r.err) catch {}; r.err.print("import failed: {s}\n", .{@errorName(e)}) catch {}; - return finish(r, failureExitCode(e, diags.problems.items.len)); + return finish(r, failureExitCode(e, diags.failureCount())); }; return finish(r, exit_ok); } @@ -458,32 +461,42 @@ fn importImpl(r: Runner, args: ImportArgs, diags: *validate.Diagnostics) !void { .{ .force = args.force }, diags, ); + + // A configuration that imports cleanly can still have recorded warnings — a + // blocklist source in no group is the one that found this. `validate` + // returns nothing for a warning, so printing diagnostics on the failure path + // alone made `import` the command that read the finding and threw it away, + // while `check` printed it from the same file. Only warnings can be here: + // any recorded failure returned above. + try diags.writeAll(r.out); try r.out.print("imported {s}\n", .{args.file}); } /// A configuration the operator can fix exits 2; everything else is a runtime /// failure. `validate` records a diagnostic for every error it returns and then -/// returns the first one, so a non-empty diagnostics list is the reliable -/// discriminator; the named errors below are the config faults that never reach -/// the validator. +/// returns the first one, so a recorded failure is the reliable discriminator; +/// `config/faults.zig` classifies the errors that never reach the validator. +/// That file is the only list — this function keeps none of its own, which is +/// what stops `run`, `check` and `import` drifting apart again (D1). /// -/// `error.OutOfMemory` is matched first, before the list is consulted. Both -/// recording paths — `validate` and import's per-line rendering of a ZON syntax -/// error — add one problem at a time and can run out of memory partway, which -/// leaves problems recorded for a run whose real outcome is a resource failure. -/// A partial report is not a verdict on the configuration, so the runtime exit -/// code wins. -fn failureExitCode(e: anyerror, problems: usize) u8 { +/// `error.DatabaseNotEmpty` is the one exception, and it is deliberate: it +/// reports the state of the database rather than the content of a file, so it +/// is not a configuration fault, and `import` alone decides it is exit 2. +/// +/// `error.OutOfMemory` is matched first, before anything else is consulted. +/// Both recording paths — `validate` and import's per-line rendering of a ZON +/// syntax error — add one problem at a time and can run out of memory partway, +/// which leaves problems recorded for a run whose real outcome is a resource +/// failure. A partial report is not a verdict on the configuration, so the +/// runtime exit code wins. +/// +/// `failures` counts recorded failures, never warnings: a warning never changes +/// an exit code (F-b). +fn failureExitCode(e: anyerror, failures: usize) u8 { if (e == error.OutOfMemory) return exit_runtime; - if (problems != 0) return exit_check; - return switch (e) { - error.DatabaseNotEmpty, - error.ConfigTooLarge, - error.ParseZon, - error.PasswordAndHashBothSet, - => exit_check, - else => exit_runtime, - }; + if (failures != 0) return exit_check; + if (e == error.DatabaseNotEmpty) return exit_check; + return if (faults.isConfigFault(e)) exit_check else exit_runtime; } // --------------------------------------------------------------------------- @@ -520,21 +533,10 @@ fn checkImpl(r: Runner, args: CheckArgs, probe: bool) !u8 { return checkFile(r, arena, args.paths.config, probe); } - const config_db_path = try std.fs.path.join(arena, &.{ args.paths.data_dir, config_db_name }); + const config_db_path = try std.fs.path.joinZ(arena, &.{ args.paths.data_dir, config_db_name }); if (try pathExists(r.io, config_db_path)) { try r.out.print("checking database {s}\n", .{config_db_path}); - - var data = try DataDir.open(r.io, r.gpa, args.paths.data_dir, false); - defer data.close(r.io, r.gpa); - - // A `check` on a database one schema version behind must still work, - // which is what an operator runs right after an upgrade. - var database = try data.openConfigDb(r.io); - defer database.close(); - _ = try migrations.migrate(&database); - - const cfg = try config_export.readConfig(&database, arena); - return checkConfig(r, cfg, probe); + return checkDatabase(r, arena, config_db_path, probe); } if (try pathExists(r.io, args.paths.config)) { @@ -550,6 +552,121 @@ fn checkImpl(r: Runner, args: CheckArgs, probe: bool) !u8 { return exit_check; } +/// `check` reads `config.db` and writes nothing to it (F-c): no create, no +/// chmod, no `applyPragmas` — which is what would turn WAL on and leave +/// `config.db-wal` and `config.db-shm` behind — and above all no `migrate`. A +/// database behind this binary's schema is reported here; upgrading it is +/// `nxdns run`'s job, and a command that claims to validate without writing may +/// not commit a schema step. +/// +/// `.immutable` is the enforcement, not a convention: SQLite refuses every +/// statement that would write, and the mode's `immutable=1` also stops the +/// pager building a wal-index, so no `config.db-wal` and no `config.db-shm` +/// appear beside the file. `.read_only` alone creates both and cannot delete +/// them on close, which is how a "validate without writing" command left two +/// files behind. +fn checkDatabase(r: Runner, arena: Allocator, path: [:0]const u8, probe: bool) !u8 { + var database = db.Db.open(path, .{ .mode = .{ .immutable = r.io } }) catch |e| switch (e) { + error.OutOfMemory => return error.OutOfMemory, + error.WalPending => return walPendingFailure(r, path), + else => return unreadable(r, path, e), + }; + defer database.close(); + + const reading = try readDatabase(&database, arena); + + // `immutable=1` takes no lock — that is what keeps it from building a + // wal-index and leaving two sidecars behind — so a writer was free to append + // to the log or checkpoint into the main file for the whole of the read + // above. Proved before one word of it is reported: a stale answer and a torn + // read both look exactly like an ordinary finding, which is how this failure + // stays invisible. + database.verifyImmutable() catch |e| switch (e) { + error.OutOfMemory => return error.OutOfMemory, + error.WalPending => return walPendingFailure(r, path), + else => return unreadable(r, path, e), + }; + + switch (reading) { + .version_unreadable => |e| { + try r.out.print("FAIL {s}: the schema version cannot be read ({s})\n", .{ path, @errorName(e) }); + return exit_check; + }, + // Naming both numbers is the point: "at 1, expects 2" tells an operator + // to run `nxdns run`, where a bare SQLite complaint about a missing + // column tells them nothing. + .version_mismatch => |stamped| { + const fix = if (stamped < migrations.target_version) + "`nxdns run` migrates it, `check` will not" + else + "it was written by a newer nxdns"; + try r.out.print( + "FAIL {s}: schema version {d}, this nxdns expects {d}; {s}\n", + .{ path, stamped, migrations.target_version, fix }, + ); + return exit_check; + }, + .config_unreadable => { + // The schema version is already known good, so whatever this is, + // the SQLite message is the only thing that narrows it down. + // `verifyImmutable` makes no SQLite call, so this is still the + // message from the read. + var buf: [256]u8 = undefined; + try r.out.print("FAIL {s}: cannot be read ({s})\n", .{ path, database.lastError(&buf) }); + return exit_check; + }, + .config => |cfg| return checkConfig(r, cfg, probe), + } +} + +/// Everything read out of `config.db`, held rather than reported, because a +/// value from an unlocked read is only worth reporting once `verifyImmutable` +/// has said the files it came from stood still. A wrong schema-version line is +/// as misleading as a wrong setting. +const DbReading = union(enum) { + config: model.Config, + version_unreadable: anyerror, + version_mismatch: u32, + config_unreadable, +}; + +/// Reads only, so an immutable handle serves it. +fn readDatabase(database: *db.Db, arena: Allocator) error{OutOfMemory}!DbReading { + const stamped = migrations.readVersion(database) catch |e| switch (e) { + error.OutOfMemory => return error.OutOfMemory, + else => return .{ .version_unreadable = e }, + }; + if (stamped != migrations.target_version) return .{ .version_mismatch = stamped }; + + const cfg = config_export.readConfig(database, arena) catch |e| switch (e) { + error.OutOfMemory => return error.OutOfMemory, + else => return .config_unreadable, + }; + return .{ .config = cfg }; +} + +/// One wording for `error.WalPending`, whether the log was already there when +/// the read opened or arrived while it ran: from the operator's side those are +/// the same situation, a writer holding changes this read cannot see. +/// +/// `immutable=1` ignores the write-ahead log, so the newest committed settings +/// would be invisible and `check` would quietly grade the older ones in the main +/// file. The guard is deliberately conservative — a live writer, an interrupted +/// process and a checkpointed log that was simply kept all look the same from +/// outside — so the line says what to do and does not claim anything is damaged. +fn walPendingFailure(r: Runner, path: []const u8) !u8 { + try r.out.print( + "FAIL {s}: uncheckpointed changes are waiting in {s}{s}, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.\n", + .{ path, path, db.wal_suffix }, + ); + return exit_check; +} + +fn unreadable(r: Runner, path: []const u8, e: anyerror) !u8 { + try r.out.print("FAIL {s}: cannot be opened for reading ({s})\n", .{ path, @errorName(e) }); + return exit_check; +} + fn pathExists(io: std.Io, path: []const u8) std.Io.Dir.AccessError!bool { std.Io.Dir.cwd().access(io, path, .{}) catch |e| switch (e) { error.FileNotFound => return false, @@ -583,6 +700,19 @@ fn checkFile(r: Runner, arena: Allocator, path: []const u8, probe: bool) !u8 { try r.out.print("FAIL {s}: larger than {d} bytes\n", .{ path, import.max_config_bytes }); return exit_check; }, + // D4: a named file that is missing or unreadable is the same + // operator-fixable condition as one that fails to parse, so it is + // reported as a finding rather than escaping as a runtime failure. The + // implicit path already exits 2 when it finds nothing to check; naming + // the file must not change the code. + error.FileNotFound => { + try r.out.print("FAIL {s}: no such file\n", .{path}); + return exit_check; + }, + error.AccessDenied, error.PermissionDenied => { + try r.out.print("FAIL {s}: not readable\n", .{path}); + return exit_check; + }, else => |other| return other, }; @@ -602,48 +732,83 @@ fn checkFile(r: Runner, arena: Allocator, path: []const u8, probe: bool) !u8 { return checkConfig(r, cfg, probe); } +/// What one part of `check` found. Failures set exit 2; warnings are printed, +/// counted for the summary, and never change an exit code (F-b) — the service +/// starts either way. +const Tally = struct { + failures: usize = 0, + warnings: usize = 0, + + fn plus(self: Tally, other: Tally) Tally { + return .{ + .failures = self.failures + other.failures, + .warnings = self.warnings + other.warnings, + }; + } +}; + /// The half of `check` that has a `Config` already: validate, report every /// diagnostic, then the certificate and upstream checks. With TLS disabled and /// `probe` false it touches neither the filesystem nor the network, which is /// what makes it unit-testable. +/// +/// The summary never contradicts the lines above it (D2): a run that printed +/// WARN lines says so instead of claiming no problems were found, and still +/// exits 0. pub fn checkConfig(r: Runner, cfg: model.Config, probe: bool) !u8 { var diags: validate.Diagnostics = .init(r.gpa); defer diags.deinit(); - // The returned error is `problems[0].err` — one of the lines about to be - // printed — so it carries nothing the report does not. Only an allocation - // failure means the report itself is incomplete. + // The returned error is the first recorded failure — one of the lines about + // to be printed — so it carries nothing the report does not. Only an + // allocation failure means the report itself is incomplete. validate.validate(cfg, &diags) catch |e| switch (e) { error.OutOfMemory => return error.OutOfMemory, else => {}, }; + // `writeAll` prints the "FAIL "/"WARN " prefix itself; the lines below add + // their own because they are not diagnostics. try diags.writeAll(r.out); - var failures = diags.problems.items.len; - failures += try checkCertificates(r, cfg); - if (probe) failures += try probeUpstreams(r, cfg); + var tally: Tally = .{ .failures = diags.failureCount(), .warnings = diags.warningCount() }; + tally = tally.plus(try checkCertificates(r, cfg)); + if (probe) tally.failures += try probeUpstreams(r, cfg); - if (failures != 0) return exit_check; + if (tally.failures != 0) return exit_check; + if (tally.warnings != 0) { + try r.out.print("OK: no failures found, {d} warning{s}\n", .{ + tally.warnings, + if (tally.warnings == 1) "" else "s", + }); + return exit_ok; + } try r.out.writeAll("OK: no problems found\n"); return exit_ok; } -fn checkCertificates(r: Runner, cfg: model.Config) !usize { - return try checkTlsFiles(r, cfg.doh_server, "doh_server") + - try checkTlsFiles(r, cfg.dot_server, "dot_server"); +fn checkCertificates(r: Runner, cfg: model.Config) !Tally { + const doh = try checkTlsFiles(r, cfg.doh_server, "doh_server"); + const dot = try checkTlsFiles(r, cfg.dot_server, "dot_server"); + return doh.plus(dot); } -/// An unreadable certificate or key fails the run; a key readable by anyone -/// beyond its owner is a warning (PLAN §19) and leaves the exit code alone, -/// because the service still starts. -fn checkTlsFiles(r: Runner, endpoint: model.TlsEndpoint, comptime section: []const u8) !usize { - if (!endpoint.enabled) return 0; - var failures: usize = 0; - - if (!try pathReadable(r.io, endpoint.cert_path)) { - try r.out.print("FAIL " ++ section ++ ".cert_path: '{s}' is not readable\n", .{endpoint.cert_path}); - failures += 1; - } +/// Proves the pair rather than the paths (D3). `CertStore.init` is the load the +/// listeners boot with: it reads both PEM files and builds a +/// `tls_server.ServerContext`, which is where Mbed TLS parses the chain, parses +/// the key and checks that the key belongs to the leaf. Testing readability +/// alone let `check` exit 0 on a certificate and key that do not pair, seconds +/// before `run` exited 2 on `BadCertificate`. +/// +/// No listener is bound and nothing is published: the context is built and +/// freed. `alpn` is null because ALPN is negotiated per connection and plays no +/// part in loading a pair; every other input is the server's. +/// +/// A key readable by anyone beyond its owner stays a warning (PLAN §19) and is +/// reported even when the pair itself fails, because it is a separate finding +/// about a file that exists. +fn checkTlsFiles(r: Runner, endpoint: model.TlsEndpoint, comptime section: []const u8) !Tally { + if (!endpoint.enabled) return .{}; + var tally: Tally = .{}; if (try pathReadable(r.io, endpoint.key_path)) { const stat = try std.Io.Dir.cwd().statFile(r.io, endpoint.key_path, .{}); @@ -653,13 +818,46 @@ fn checkTlsFiles(r: Runner, endpoint: model.TlsEndpoint, comptime section: []con "WARN " ++ section ++ ".key_path: '{s}' is mode {o}; a TLS key must be readable by its owner only\n", .{ endpoint.key_path, mode }, ); + tally.warnings += 1; } - } else { - try r.out.print("FAIL " ++ section ++ ".key_path: '{s}' is not readable\n", .{endpoint.key_path}); - failures += 1; } - return failures; + var store = cert_store.CertStore.init( + r.gpa, + r.io, + endpoint.cert_path, + endpoint.key_path, + null, + ) catch |e| { + const at: struct { field: []const u8, path: []const u8 } = switch (e) { + error.OutOfMemory => return error.OutOfMemory, + // Not a verdict on the configuration: the platform's entropy source + // failed, which is the same failure `run` would hit. + error.EntropyFailed => return error.EntropyFailed, + error.CertUnreadable, + error.CertTooLarge, + error.CertParse, + // Mbed TLS rejected the server configuration built from this pair, + // so the certificate is what the operator has to look at. + error.ConfigFailed, + => .{ .field = "cert_path", .path = endpoint.cert_path }, + error.KeyUnreadable, + error.KeyTooLarge, + error.KeyParse, + error.KeyMismatch, + => .{ .field = "key_path", .path = endpoint.key_path }, + }; + try r.out.print("FAIL " ++ section ++ ".{s}: '{s}': {s}\n", .{ + at.field, + at.path, + cert_store.humanMessage(e), + }); + tally.failures += 1; + return tally; + }; + store.deinit(r.io); + + return tally; } /// One `Pool` per upstream, never one pool over all of them. The pool's job is @@ -697,11 +895,19 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize { // backoff, so the value only has to be a value. const seed: u64 = @truncate(@as(u96, @bitCast(std.Io.Clock.real.now(r.io).nanoseconds))); - for (cfg.upstreams) |server| { + // Every line below names the upstream by its index into the configuration + // as well as by its redacted url, and the index is the config index rather + // than a count of the upstreams probed — a disabled entry still occupies + // one. The url alone no longer identifies an entry: `safe_url.redact` drops + // the path, and two upstreams on one host commonly differ only there (a + // NextDNS profile is `https://dns.nextdns.io/`). `upstreams[N]` is + // the same name `config/validate.zig` gives the entry, so a FAIL line here + // and a FAIL line from the validator point at the same place. + for (cfg.upstreams, 0..) |server, i| { if (!server.enabled) continue; const endpoint = transport.Endpoint.parse(server.url) catch { - try r.out.print("FAIL {s}: not an https:// or tls:// endpoint\n", .{server.url}); + try r.out.print("FAIL upstreams[{d}] {f}: not an https:// or tls:// endpoint\n", .{ i, safe_url.redactQuoted(server.url) }); failures += 1; continue; }; @@ -713,7 +919,7 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize { const client: transport.Client = switch (endpoint.scheme) { .doh => doh: { doh = doh_client.DohClient.init(&http, endpoint, &request_buf, &transfer_buf) catch { - try r.out.print("FAIL {s}: not a usable DoH url\n", .{server.url}); + try r.out.print("FAIL upstreams[{d}] {f}: not a usable DoH url\n", .{ i, safe_url.redactQuoted(server.url) }); failures += 1; continue; }; @@ -740,7 +946,7 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize { var single: pool.Pool = .init(&entries, .{}, attempt_timeout, seed); if (single.exchange(r.io, probe_query, response_buf)) |_| { - try r.out.print("OK {s}\n", .{server.url}); + try r.out.print("OK upstreams[{d}] {f}\n", .{ i, safe_url.redact(server.url) }); } else |_| { // The concrete cause lives in the entry's health, which is where the // pool put it; `@errorName` of the pool's return value would only @@ -748,7 +954,7 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize { var snapshots: [1]pool.Snapshot = undefined; const taken = try single.snapshot(r.io, &snapshots); const detail = if (taken == 1) snapshots[0].last_error else "no detail recorded"; - try r.out.print("FAIL {s}: {s}\n", .{ server.url, detail }); + try r.out.print("FAIL upstreams[{d}] {f}: {s}\n", .{ i, safe_url.redactQuoted(server.url), detail }); failures += 1; } } @@ -1019,6 +1225,391 @@ test "an allocation failure while recording diagnostics exits 1, not 2" { try testing.expect(saw_partial_report); } -// `runCheck` against real paths, `runExport` and `runImport` all need a real -// data directory, and the upstream probe leaves the machine. Those cases are -// S7's: `src/storage/storage_integration_test.zig` cases 20-22. +test "failureExitCode keeps no list of its own and classifies through config/faults.zig" { + // D1: every one of these reached `import` from a rejected seed file and was + // classified as a runtime failure by the private list this function used to + // carry, while `check` called the same file a configuration fault. + try testing.expectEqual(exit_check, failureExitCode(error.MissingDefaultGroup, 0)); + try testing.expectEqual(exit_check, failureExitCode(error.NoUpstreams, 0)); + try testing.expectEqual(exit_check, failureExitCode(error.BadUpstreamUrl, 0)); + try testing.expectEqual(exit_check, failureExitCode(error.NoUsableUpstreams, 0)); + try testing.expectEqual(exit_check, failureExitCode(error.BadCertificate, 0)); + + // Every member of the shared classification, so a variant added to + // `ValidateError` cannot exit 1 from `import` while exiting 2 from `run`. + inline for (@typeInfo(validate.ValidateError).error_set.?) |member| { + const err = @field(anyerror, member.name); + const expected: u8 = if (faults.isConfigFault(err)) exit_check else exit_runtime; + try testing.expectEqual(expected, failureExitCode(err, 0)); + } + + // The one config-shaped exit 2 `cli` still decides for itself: it reports + // the state of the database, not the content of a file. + try testing.expect(!faults.isConfigFault(error.DatabaseNotEmpty)); + try testing.expectEqual(exit_check, failureExitCode(error.DatabaseNotEmpty, 0)); +} + +const fixtures = @import("test_fixtures"); + +/// A tmp directory holding the fixture PEM pair and a data directory, addressed +/// by cwd-relative paths the same way an operator's configuration names them. +/// Must not move after `init`: the slices point into the buffers. +const CheckEnv = struct { + tmp: testing.TmpDir, + cert_path_buf: [160]u8, + key_path_buf: [160]u8, + data_dir_buf: [160]u8, + missing_path_buf: [160]u8, + cert_path: []const u8, + key_path: []const u8, + data_dir: []const u8, + /// A path inside the tmp directory that is never created. + missing_path: []const u8, + + fn init(env: *CheckEnv) !void { + env.tmp = testing.tmpDir(.{}); + errdefer env.tmp.cleanup(); + env.cert_path = try env.path(&env.cert_path_buf, "cert.pem"); + env.key_path = try env.path(&env.key_path_buf, "key.pem"); + env.data_dir = try env.path(&env.data_dir_buf, "data"); + env.missing_path = try env.path(&env.missing_path_buf, "no-such-config.zon"); + } + + fn deinit(env: *CheckEnv) void { + env.tmp.cleanup(); + } + + fn path(env: *const CheckEnv, buf: []u8, name: []const u8) ![]const u8 { + return std.fmt.bufPrint(buf, ".zig-cache/tmp/{s}/{s}", .{ env.tmp.sub_path, name }); + } + + /// The key lands at 0600, so only a test that asks for the permission + /// warning gets one. + fn writePair(env: *CheckEnv, io: std.Io, key_pem: []const u8) !void { + try env.tmp.dir.writeFile(io, .{ .sub_path = "cert.pem", .data = fixtures.cert_pem }); + try env.tmp.dir.writeFile(io, .{ .sub_path = "key.pem", .data = key_pem }); + try env.tmp.dir.setFilePermissions(io, "key.pem", .fromMode(0o600), .{}); + } + + /// Valid but for whatever the test broke about the TLS pair. + fn tlsConfig(env: *const CheckEnv) model.Config { + return .{ + .groups = &.{.{ .name = "default" }}, + .upstreams = &.{.{ .url = "https://dns.example/dns-query" }}, + .doh_server = .{ + .enabled = true, + .cert_path = env.cert_path, + .key_path = env.key_path, + }, + }; + } + + fn expectAbsent(env: *CheckEnv, io: std.Io, name: []const u8) !void { + env.tmp.dir.access(io, name, .{}) catch |e| switch (e) { + error.FileNotFound => return, + else => |other| return other, + }; + std.debug.print("expected '{s}' not to exist\n", .{name}); + return error.TestUnexpectedResult; + } +}; + +test "check fails a certificate and a key that do not pair" { + // D3: readability alone passed this configuration, and `run` then exited 2 + // on `BadCertificate` seconds later. + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + try env.writePair(r.io, fixtures.mismatched_key_pem); + + try testing.expectEqual(exit_check, try checkConfig(r, env.tlsConfig(), false)); + + const text = captured.out.written(); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "FAIL doh_server.key_path")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "does not belong to the certificate")); + try testing.expectEqual(@as(usize, 0), std.mem.count(u8, text, "OK:")); +} + +test "check fails a certificate file that does not parse" { + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + try env.writePair(r.io, fixtures.key_pem); + try env.tmp.dir.writeFile(r.io, .{ .sub_path = "cert.pem", .data = "not a certificate\n" }); + + try testing.expectEqual(exit_check, try checkConfig(r, env.tlsConfig(), false)); + + const text = captured.out.written(); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "FAIL doh_server.cert_path")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "could not be parsed")); +} + +test "check passes a certificate and a key that pair" { + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + try env.writePair(r.io, fixtures.key_pem); + + try testing.expectEqual(exit_ok, try checkConfig(r, env.tlsConfig(), false)); + try testing.expectEqualStrings("OK: no problems found\n", captured.out.written()); +} + +test "a check whose only findings are warnings exits 0 and says so" { + // D2: the summary used to read "OK: no problems found" directly under the + // WARN line it was contradicting. + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + try env.writePair(r.io, fixtures.key_pem); + try env.tmp.dir.setFilePermissions(r.io, "key.pem", .fromMode(0o644), .{}); + + // A warning never changes an exit code: the service still starts. + try testing.expectEqual(exit_ok, try checkConfig(r, env.tlsConfig(), false)); + + const text = captured.out.written(); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "WARN doh_server.key_path")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "OK: no failures found, 1 warning\n")); + try testing.expectEqual(@as(usize, 0), std.mem.count(u8, text, "no problems found")); +} + +test "check --config naming a missing file is a reported failure at exit 2" { + // D4: this escaped `checkImpl` as `check failed: FileNotFound` at exit 1, + // while the implicit path exits 2 for the same operator-fixable condition. + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + + const code = runCheck(r, .{ + .paths = .{ .config = env.missing_path }, + .config_explicit = true, + }, false); + + try testing.expectEqual(exit_check, code); + const text = captured.out.written(); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "FAIL")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "no such file")); + try testing.expectEqualStrings("", captured.err.written()); +} + +test "check reads config.db without writing to it" { + // D6: the database branch opened read/write, chmod'ed 0600, turned WAL on — + // which is what creates the two sidecars — and committed migration steps, + // from a command whose contract is that it validates without writing. + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + + { + var data = try DataDir.open(r.io, r.gpa, env.data_dir, true); + defer data.close(r.io, r.gpa); + var database = try data.openConfigDb(r.io); + defer database.close(); + _ = try migrations.migrate(&database); + } + + // A mode `openConfigDb` would overwrite, so its chmod cannot hide. + try env.tmp.dir.setFilePermissions(r.io, "data/config.db", .fromMode(0o644), .{}); + const before = try env.tmp.dir.statFile(r.io, "data/config.db", .{}); + + _ = runCheck(r, .{ .paths = .{ .data_dir = env.data_dir } }, false); + try testing.expect(std.mem.containsAtLeast(u8, captured.out.written(), 1, "checking database")); + + // Byte for byte the database the writer left, at the mode the writer left: + // no migration step committed, no chmod, no `PRAGMA journal_mode`. + const after = try env.tmp.dir.statFile(r.io, "data/config.db", .{}); + try testing.expectEqual(before.size, after.size); + try testing.expectEqual(before.mtime.nanoseconds, after.mtime.nanoseconds); + try testing.expectEqual( + @as(@TypeOf(after.permissions.toMode()), 0o644), + after.permissions.toMode() & 0o777, + ); + + // Nothing beside it either. A `.read_only` open recreates the wal-index of + // a database whose header says WAL and cannot delete it on close, which + // left `config.db-wal` and `config.db-shm` behind; `.immutable` builds no + // wal-index at all. + try env.expectAbsent(r.io, "data/config.db-wal"); + try env.expectAbsent(r.io, "data/config.db-shm"); +} + +test "check refuses a database whose write-ahead log still holds changes" { + // `immutable=1` ignores the log, so grading the main file would silently + // report settings the operator already replaced. + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + + { + var data = try DataDir.open(r.io, r.gpa, env.data_dir, true); + defer data.close(r.io, r.gpa); + var database = try data.openConfigDb(r.io); + defer database.close(); + _ = try migrations.migrate(&database); + } + // Bytes are all the guard reads, and it never gets as far as opening this + // as a log: `db.Db.open` refuses on the size alone. + try env.tmp.dir.writeFile(r.io, .{ + .sub_path = "data/config.db" ++ db.wal_suffix, + .data = "uncheckpointed frames", + }); + + const code = runCheck(r, .{ .paths = .{ .data_dir = env.data_dir } }, false); + + try testing.expectEqual(exit_check, code); + const text = captured.out.written(); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "FAIL")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "uncheckpointed changes")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "config.db" ++ db.wal_suffix)); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns run")); + // Not a claim about damage: an operator reading this must not reach for a + // recovery tool. + try testing.expectEqual(@as(usize, 0), std.mem.count(u8, text, "corrupt")); +} + +test "check reports a database behind the schema rather than migrating it" { + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + + // Zero bytes is a valid, empty SQLite database: schema version 0, which is + // exactly what an upgrade leaves behind when a step has not run yet. + _ = try std.Io.Dir.cwd().createDirPathStatus(r.io, env.data_dir, .fromMode(0o700)); + try env.tmp.dir.writeFile(r.io, .{ .sub_path = "data/config.db", .data = "" }); + + const code = runCheck(r, .{ .paths = .{ .data_dir = env.data_dir } }, false); + + try testing.expectEqual(exit_check, code); + const text = captured.out.written(); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "FAIL")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns run")); + + // Both real numbers, not a generic SQLite complaint: an empty database is + // at version 0, and the expected one comes from the migration list rather + // than a literal, so adding a step cannot make this line lie. + var expected_buf: [96]u8 = undefined; + const expected = try std.fmt.bufPrint( + &expected_buf, + "schema version 0, this nxdns expects {d};", + .{migrations.target_version}, + ); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, expected)); + + // Not upgraded: the schema `migrate` would have committed is still absent. + const after = try env.tmp.dir.statFile(r.io, "data/config.db", .{}); + try testing.expectEqual(@as(u64, 0), after.size); + try env.expectAbsent(r.io, "data/config.db-wal"); + try env.expectAbsent(r.io, "data/config.db-shm"); +} + +test "the upstream probe redacts a url it cannot parse, without leaving the machine" { + // No socket is opened on this branch: `Endpoint.parse` rejects the `@` + // before the loop builds a client, so the leak is reachable in a required + // test rather than only behind a live probe. + // + // It is also the only probe branch a credential can reach. `Endpoint.parse` + // refuses `@`, `?` and `#` in the authority and `?`/`#` in the path, so a + // url carrying userinfo or a query never gets as far as `DohClient.init` + // and its "not a usable DoH url" line. That line's redaction is defensive. + // + // The two upstreams are the NextDNS shape, where the profile id in the path + // is the account credential and is the only thing telling two entries + // apart. Both halves of the contract are asserted at once: neither the + // userinfo nor the profile id may reach stdout, and what is left has to + // still say which of the two entries the operator must go and fix. The + // disabled entry ahead of them is there because the index has to be the + // index into `upstreams`, not a count of the entries probed. + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + + const cfg: model.Config = .{ + .groups = &.{.{ .name = "default" }}, + .upstreams = &.{ + .{ .url = "https://dns.example/dns-query", .enabled = false }, + .{ .url = "https://lists:hunter2@dns.nextdns.io/abcd12" }, + .{ .url = "https://lists:hunter2@dns.nextdns.io/efgh34" }, + }, + }; + + try testing.expectEqual(@as(usize, 2), try probeUpstreams(r, cfg)); + + const text = captured.out.written(); + try testing.expectEqualStrings( + "FAIL upstreams[1] 'https://dns.nextdns.io': not an https:// or tls:// endpoint\n" ++ + "FAIL upstreams[2] 'https://dns.nextdns.io': not an https:// or tls:// endpoint\n", + text, + ); + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "hunter2")); + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "abcd12")); + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "efgh34")); +} + +test "a successful import prints the warnings the file earned" { + // D5, second half. `check` printed this WARN and `import` recorded it and + // threw it away, because diagnostics were written on the failure path only. + // The operator who imported the file learned nothing about the source they + // had just added, which blocks nothing until a group links it. + var env: CheckEnv = undefined; + try env.init(); + defer env.deinit(); + + var captured: Captured = .init(testing.allocator); + defer captured.deinit(); + const r = captured.runner(); + + try env.tmp.dir.writeFile(r.io, .{ .sub_path = "config.zon", .data = + \\.{ + \\ .groups = .{ .{ .name = "default" } }, + \\ .upstreams = .{ .{ .url = "https://dns.example/dns-query" } }, + \\ .blocklist_sources = .{ .{ .url = "https://lists.example/hosts.txt", .name = "ads" } }, + \\} + }); + + var file_buf: [160]u8 = undefined; + const file = try env.path(&file_buf, "config.zon"); + const code = runImport(r, .{ .paths = .{ .data_dir = env.data_dir }, .file = file }); + + // A warning never changes an exit code, and the import really happened. + try testing.expectEqual(exit_ok, code); + const text = captured.out.written(); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "WARN blocklist_sources[0]: ")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "belongs to no group")); + try testing.expect(std.mem.containsAtLeast(u8, text, 1, "imported ")); + try testing.expectEqualStrings("", captured.err.written()); +} + +// `runExport` needs a real data directory and the upstream probe leaves the +// machine. Those cases are S7's: +// `src/storage/storage_integration_test.zig` cases 20-22. diff --git a/src/config/bootstrap.zig b/src/config/bootstrap.zig index 56e15f1..0ed3591 100644 --- a/src/config/bootstrap.zig +++ b/src/config/bootstrap.zig @@ -7,7 +7,10 @@ //! The policy is three lines long: //! //! - no file → normal steady state, keep the database as it is; -//! - database already configured → the file is ignored, as PLAN §3.5 requires; +//! - database already configured → the file is ignored, as PLAN §3.5 requires. +//! Configured means an operator put something there. A database that has only +//! answered queries is not configured, however many client rows the DNS path +//! materialised into it, and `import.isEmpty` is where that line is drawn; //! - otherwise → import it, and a file that is unreadable, unparseable or //! invalid is an error. The operator wrote that file and meant it; starting //! with silent defaults instead is the exact failure mode PLAN §1.3 exists to @@ -55,6 +58,84 @@ pub fn bootstrap( return .seeded; } -// Every path through `bootstrap` starts with a filesystem access, so all three -// outcomes are exercised in `src/storage/storage_integration_test.zig` (S7) -// against real files. There is nothing here that an in-memory test could reach. +// --------------------------------------------------------------------------- +// tests +// --------------------------------------------------------------------------- +// +// All three outcomes are exercised end to end in +// `src/storage/storage_integration_test.zig` (S7) against a real data directory. +// What the two cases below add is the one distinction that decides which outcome +// an operator gets, and it is too important to leave behind a `-Dintegration` +// flag: whether the database has been *configured*, not whether it has been +// *used*. + +const testing = std.testing; + +const clients_repo = @import("../storage/repositories/clients_repo.zig"); +const migrations = @import("../storage/migrations.zig"); + +const seed_source = + \\.{ + \\ .groups = .{ .{ .name = "default" } }, + \\ .upstreams = .{ .{ .url = "https://dns.example/dns-query" } }, + \\} +; + +/// Unparseable on purpose: a call that succeeds proves the file was never read. +const broken_source = ".{ .groups = "; + +fn openMigrated() !db.Db { + var database = try db.Db.open(":memory:", .{ .mode = .memory }); + errdefer database.close(); + try db.applyPragmas(&database, .{}); + _ = try migrations.migrate(&database); + return database; +} + +test "a server that has answered queries still seeds from its configuration file" { + const io = testing.io; + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + try tmp.dir.writeFile(io, .{ .sub_path = "config.zon", .data = seed_source }); + + var database = try openMigrated(); + defer database.close(); + + // The unattended first boot: the server came up on defaults, answered + // traffic, and the operator dropped a config file in afterwards. + try clients_repo.upsertSeen(&database, "192.168.1.5", 1700000000); + + var diags: validate.Diagnostics = .init(testing.allocator); + defer diags.deinit(); + + const outcome = try bootstrap(io, testing.allocator, &database, tmp.dir, "config.zon", &diags); + try testing.expectEqual(Outcome.seeded, outcome); + try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM upstreams")); + // Seeding did not cost the operator the device list they had been watching. + try testing.expectEqual(@as(i64, 1), try clients_repo.countClients(&database)); +} + +test "a client the operator has customised keeps the configuration file out" { + const io = testing.io; + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + try tmp.dir.writeFile(io, .{ .sub_path = "config.zon", .data = broken_source }); + + var database = try openMigrated(); + defer database.close(); + + try clients_repo.upsertSeen(&database, "192.168.1.5", 1700000000); + const id = try database.queryInt("SELECT id FROM clients WHERE ip = '192.168.1.5'"); + try clients_repo.updateClient(&database, id, .{ .name = "tv", .group_id = 1 }); + + var diags: validate.Diagnostics = .init(testing.allocator); + defer diags.deinit(); + + const outcome = try bootstrap(io, testing.allocator, &database, tmp.dir, "config.zon", &diags); + try testing.expectEqual(Outcome.db_already_configured, outcome); + try testing.expectEqual(@as(usize, 0), diags.problems.items.len); + // The name and the flag the operator set are still theirs. + try testing.expectEqual(@as(i64, 1), try database.queryInt( + "SELECT count(*) FROM clients WHERE name = 'tv' AND hand_edited = 1", + )); +} diff --git a/src/config/faults.zig b/src/config/faults.zig new file mode 100644 index 0000000..dbec8a1 --- /dev/null +++ b/src/config/faults.zig @@ -0,0 +1,115 @@ +//! One definition of "the operator's configuration is wrong". +//! +//! `run`, `check` and `import` all sort a failure into two buckets: the +//! configuration is wrong and the operator can fix it (exit 2, `nxdns check` +//! is the next step), or something else broke (exit 1). Each subcommand used to +//! carry its own list of which errors meant which, and the lists disagreed — +//! the same seed file exited 1 from `run` and 2 from `check`. There is one list +//! now, and it is this file. Nothing else may keep a second one. + +const std = @import("std"); + +const validate = @import("validate.zig"); + +/// `ValidateError` enters as a whole set rather than variant by variant, so a +/// variant added to the validator cannot silently fall through to exit 1. The +/// four extras are the configuration faults raised outside the validator: the +/// ZON reader (`ParseZon`), the seed-file size limit (`ConfigTooLarge`), the +/// composition root's upstream build (`NoUsableUpstreams`) and its certificate +/// load (`BadCertificate`). +/// +/// Not here on purpose: `error.DatabaseNotEmpty`, which reports the state of +/// the database rather than the content of a file, and is the one config-shaped +/// exit 2 `cli` decides for itself. +const ConfigFault = validate.ValidateError || error{ + ParseZon, + ConfigTooLarge, + NoUsableUpstreams, + BadCertificate, +}; + +const faults: []const anyerror = blk: { + const set = @typeInfo(ConfigFault).error_set.?; + var list: [set.len]anyerror = undefined; + for (set, 0..) |member, i| list[i] = @field(anyerror, member.name); + const frozen = list; + break :blk &frozen; +}; + +/// True when `err` means the configuration the operator supplied is wrong. +/// Linear over a set of about forty errors, on failure paths only. +pub fn isConfigFault(err: anyerror) bool { + for (faults) |fault| { + if (err == fault) return true; + } + return false; +} + +// --------------------------------------------------------------------------- +// Tests +// --------------------------------------------------------------------------- + +const testing = std.testing; + +test "every ValidateError variant is a configuration fault, with no exceptions" { + // The guard on the derivation: a variant added to `ValidateError` and left + // out of the classification fails here rather than exiting 1 in the field. + // + // No member is excused. This file used to subtract `error.OutOfMemory` + // here, which made the rule "every ValidateError is a fault, except one" — + // a private exclusion list of exactly the kind this file exists to abolish. + // `validate.ValidateError` no longer carries an allocation failure, so the + // rule is literal again. + inline for (@typeInfo(validate.ValidateError).error_set.?) |member| { + const err = @field(anyerror, member.name); + if (!isConfigFault(err)) { + std.debug.print("isConfigFault(error.{s}) is false, expected true\n", .{member.name}); + return error.TestUnexpectedResult; + } + } +} + +test "an allocation failure is not a member of the validator's verdict" { + // The root of it: `faults.zig` can only be exception-free while + // `ValidateError` holds nothing that is not a verdict on the file. + inline for (@typeInfo(validate.ValidateError).error_set.?) |member| { + if (std.mem.eql(u8, member.name, "OutOfMemory")) { + std.debug.print("ValidateError carries error.OutOfMemory\n", .{}); + return error.TestUnexpectedResult; + } + } + // It is still reachable from `validate`, just not as a finding: the + // allocator can fail and the caller has to handle it. + comptime var reachable = false; + inline for (@typeInfo(validate.Error).error_set.?) |member| { + if (comptime std.mem.eql(u8, member.name, "OutOfMemory")) reachable = true; + } + try testing.expect(reachable); +} + +test "the faults raised outside the validator are configuration faults" { + try testing.expect(isConfigFault(error.ParseZon)); + try testing.expect(isConfigFault(error.ConfigTooLarge)); + try testing.expect(isConfigFault(error.NoUsableUpstreams)); + try testing.expect(isConfigFault(error.BadCertificate)); +} + +test "the seed-file errors that used to exit 1 from run are configuration faults" { + // D1 verbatim: these three reached `run` from a rejected seed file and were + // classified as runtime failures. + try testing.expect(isConfigFault(error.ParseZon)); + try testing.expect(isConfigFault(error.MissingDefaultGroup)); + try testing.expect(isConfigFault(error.NoUpstreams)); +} + +test "a runtime failure is not a configuration fault" { + try testing.expect(!isConfigFault(error.OutOfMemory)); + try testing.expect(!isConfigFault(error.AccessDenied)); + try testing.expect(!isConfigFault(error.FileNotFound)); + try testing.expect(!isConfigFault(error.AddressInUse)); + // A state conflict, not a bad file: `import` refuses to overwrite a + // configured database and decides that exit code itself. + try testing.expect(!isConfigFault(error.DatabaseNotEmpty)); + // Only ever a warning, so it never reaches an exit code by this route. + try testing.expect(!isConfigFault(error.SourceInNoGroup)); +} diff --git a/src/config/import.zig b/src/config/import.zig index e02116e..2b0f5c4 100644 --- a/src/config/import.zig +++ b/src/config/import.zig @@ -1,4 +1,13 @@ -//! `nxdns import`: a ZON file becomes the whole content of `config.db`. +//! `nxdns import`: a ZON file becomes the whole configuration of `config.db`. +//! +//! Configuration, not content: the `hand_edited = 0` client rows the DNS path +//! materialises from live traffic are runtime state, they are absent from an +//! export, and an import carries them across rather than deleting them. +//! +//! `clients.first_seen` and `clients.last_seen` are runtime state on every client +//! row, configured ones included, and the config model carries neither. So an +//! address the database already knew keeps both across an import, and only an +//! address it has never seen takes the import's clock. //! //! The order is the specification. Nothing reaches the database until the file //! has been read, parsed and validated, and every write happens inside one @@ -44,21 +53,31 @@ const canonical_buf_len = 64; // emptiness // --------------------------------------------------------------------------- -/// A database is "never configured" when the migrations have run and nothing -/// else has. The migrations themselves create `schema_version` and seed -/// `groups(1, 'default')`, so "no rows anywhere" is the wrong test. +/// A database is "never configured" when the migrations have run and the +/// operator has added nothing. The migrations themselves create +/// `schema_version` and seed `groups(1, 'default')`, so "no rows anywhere" is +/// the wrong test. /// -/// True iff every table in `config_schema.content_tables` is empty, `groups` -/// holds exactly one row, and that row is the seeded `(1, 'default', 0)`. +/// True iff no table in `config_schema.content_tables` holds a row the operator +/// put there, `groups` holds exactly one row, and that row is the seeded +/// `(1, 'default', 0)`. /// -/// The client count here includes auto-materialised rows: a server that has -/// answered one query is configured enough that a bootstrap file must not -/// overwrite it. +/// `clients` is the one table a row can reach without an operator: the DNS path +/// materialises `hand_edited = 0` rows straight from live traffic (PLAN §7.2). +/// Counting those made emptiness a function of traffic — a server that had +/// answered a single query silently ignored the seed file its operator dropped +/// next to it. So only `hand_edited = 1` rows count, which is the predicate +/// `clients_repo.listClients` already exports by: this database is empty exactly +/// when its export is empty. pub fn isEmpty(database: *db.Db) db.Error!bool { // `inline for` over a comptime table list: every statement below is a // compile-time string, so no table name is ever concatenated at run time. inline for (config_schema.content_tables) |table| { - if (try database.queryInt("SELECT count(*) FROM " ++ table) != 0) return false; + const count_sql = comptime if (std.mem.eql(u8, table, "clients")) + "SELECT count(*) FROM clients WHERE hand_edited = 1" + else + "SELECT count(*) FROM " ++ table; + if (try database.queryInt(count_sql) != 0) return false; } if (try database.queryInt("SELECT count(*) FROM groups") != 1) return false; const seeded = try database.queryInt( @@ -179,6 +198,8 @@ pub fn applyToDb( // writes are one atomic unit. if (!options.force and !try isEmpty(database)) return error.DatabaseNotEmpty; + try liftSavedClients(database); + inline for (config_schema.delete_order) |table| { try database.exec("DELETE FROM " ++ table ++ ";"); } @@ -203,6 +224,8 @@ pub fn applyToDb( canonical.ip = try canonicalIp(client.ip, &buf); try clients_repo.insertClient(database, canonical, ctx); } + try restoreSavedClients(database, group_ids.get("default").?); + for (cfg.client_prefixes) |entry| { var buf: [canonical_buf_len]u8 = undefined; var canonical = entry; @@ -238,6 +261,102 @@ pub fn applyToDb( try tx.commit(); } +const lift_saved_clients_sql: [:0]const u8 = + \\DROP TABLE IF EXISTS temp.saved_clients; + \\CREATE TEMP TABLE saved_clients AS + \\ SELECT ip, name, hand_edited, first_seen, last_seen FROM clients; +; + +/// Carries what the wipe must not destroy over the wipe that follows: the +/// auto-materialised client rows themselves, and the observed timestamps of +/// every client row whatever its flag. +/// +/// A `hand_edited = 0` row is runtime state, not configuration: the DNS path +/// wrote it from live traffic and `clients_repo.listClients` already keeps it out +/// of an export. Replacing the *configuration* must therefore not delete it — +/// but it cannot survive in place either, because `clients.group_id` references +/// `groups(id)` with no cascade, and the wipe empties `groups`. So the rows step +/// aside into the temp database and come back once `groups` holds `default` +/// again. +/// +/// The table takes every row, not only the `hand_edited = 0` ones, because +/// `first_seen` and `last_seen` are runtime state on a configured row too — the +/// tracker keeps writing `last_seen` on the clients the operator named. Saving +/// only the materialised rows would keep the observation history of the devices +/// nobody named and destroy it for the devices somebody did. `hand_edited` rides +/// along so the restore can tell the two apart. +/// +/// `IF EXISTS` because a rolled-back import must not poison the next one. +fn liftSavedClients(database: *db.Db) Error!void { + return database.exec(lift_saved_clients_sql); +} + +const merge_observed_timestamps_sql: [:0]const u8 = + \\UPDATE clients AS c + \\ SET first_seen = m.first_seen, last_seen = m.last_seen + \\ FROM temp.saved_clients m + \\ WHERE m.ip = c.ip +; + +const restore_materialised_clients_sql = + \\INSERT INTO clients (ip, name, group_id, hand_edited, first_seen, last_seen) + \\SELECT m.ip, m.name, ?1, 0, m.first_seen, m.last_seen + \\ FROM temp.saved_clients m + \\ WHERE m.hand_edited = 0 + \\ AND NOT EXISTS (SELECT 1 FROM clients c WHERE c.ip = m.ip) +; + +/// Puts back what `liftSavedClients` set aside, in two steps that must stay in +/// this order: the restore drops the temp table on its way out, so the merge +/// cannot follow it. +/// +/// **The merge.** An address the config declares belongs to the config — but +/// `first_seen` and `last_seen` are not the config's to state. The model carries +/// neither field, so `insertClient` writes `now` into both as a placeholder for a +/// device it knows nothing about. When the database already held that address, the +/// placeholder is the worse of the two values and both columns come from the saved +/// row instead. Everything else on the row stays the config's: the name, the +/// group, and `hand_edited = 1`. +/// +/// Not "the earlier `first_seen` and the later `last_seen`": the placeholder is +/// the wall clock at import time and every real observation predates it, so +/// "later" would resolve to the placeholder every time and stamp each named +/// device as seen at the moment of the import. An operator restoring a backup +/// would read that as liveness. `first_seen` and `last_seen` mean "when a query +/// from this address arrived", `pruneStale` and the API both read them that way, +/// and an import is not a query. Taking both from the saved row also keeps +/// `first_seen <= last_seen`, which `upsertSeen` guarantees pairwise. +/// +/// Only the config's own clients are in the table at this point, so the update +/// needs no filter of its own, and the saved row's flag does not enter into it: a +/// device keeps its history whether the previous row was materialised or +/// configured. A `--force` re-import of the same file is the case that matters — +/// it deletes and rewrites every configured client, and without the merge each +/// one would come back claiming it was first seen at the moment of the import. +/// +/// **The restore.** Only `hand_edited = 0` rows come back. A configured client is +/// configuration, so a file that leaves its address out has removed that client +/// and the row must stay gone; its history was saved for the merge, not for a +/// resurrection. `WHERE NOT EXISTS` rather than `INSERT OR IGNORE`: the only +/// materialised row worth skipping is one whose address the config claims — the +/// operator naming a device the server had already discovered, handled by the +/// merge above — and every other constraint failure stays loud. +/// +/// The rows return to `default`, the group `upsertSeen` materialises into. +/// `first_seen` and `last_seen` cross unchanged; the row id does not, because the +/// config's clients went in first and hold the low ids now. Nothing references +/// `clients.id` inside `config.db`, and §3.6 keeps the query log out of it. +fn restoreSavedClients(database: *db.Db, default_group_id: i64) Error!void { + try database.exec(merge_observed_timestamps_sql); + { + var stmt = try database.prepare(restore_materialised_clients_sql); + defer stmt.deinit(); + try stmt.bindInt(1, default_group_id); + try stmt.exec(); + } + return database.exec("DROP TABLE temp.saved_clients;"); +} + /// `default` goes in first and takes rowid 1. §11.2 seeds group 1 as `default` /// and §7.2's fallback assignment depends on it; letting an import renumber it /// would silently move every unassigned client. @@ -303,7 +422,7 @@ fn canonicalPrefix(text: []const u8, buf: []u8) error{BadClientPrefix}![]const u } /// argon2id with the OWASP parameters (t=2, m=19 MiB, p=1) rather than the -/// 64 MiB `interactive_2id`, because PLAN §18 budgets under 100 MB total on a +/// 64 MiB `interactive_2id`, because PLAN §18 budgets under 100 MiB total on a /// Pi 5. /// /// `strHash`'s error set reaches beyond this module's (it carries @@ -419,16 +538,45 @@ test "isEmpty is false once a settings row exists" { try testing.expect(!try isEmpty(&database)); } -test "isEmpty is false once an auto-materialized client exists" { +test "isEmpty ignores the client rows live traffic materialises" { var database = try openMigrated(); defer database.close(); - try database.exec( - \\INSERT INTO clients (ip, name, group_id, hand_edited, first_seen, last_seen) - \\VALUES ('192.168.1.5', NULL, 1, 0, 1, 1); + + // The real §7.2 write path, not hand-written SQL: what makes these rows + // ignorable is that `upsertSeen` is the thing that wrote them. + try clients_repo.upsertSeen(&database, "192.168.1.5", 1700000000); + try clients_repo.upsertSeen(&database, "192.168.1.6", 1700000100); + + try testing.expectEqual(@as(i64, 2), try clients_repo.countClients(&database)); + try testing.expect(try isEmpty(&database)); +} + +test "isEmpty is false once a client carries operator intent" { + var database = try openMigrated(); + defer database.close(); + + _ = try clients_repo.insertClientRow( + &database, + .{ .ip = "192.168.1.7", .name = "printer", .group_id = 1 }, + 1700000000, ); try testing.expect(!try isEmpty(&database)); } +test "isEmpty is false once an operator edits a materialised client" { + var database = try openMigrated(); + defer database.close(); + + try clients_repo.upsertSeen(&database, "192.168.1.5", 1700000000); + try testing.expect(try isEmpty(&database)); + + // A PUT through the API is what turns a discovered device into policy, and + // that policy is exactly what a seed would replace. + const id = try database.queryInt("SELECT id FROM clients WHERE ip = '192.168.1.5'"); + try clients_repo.updateClient(&database, id, .{ .name = "tv", .group_id = 1 }); + try testing.expect(!try isEmpty(&database)); +} + test "isEmpty is false once the seeded group is changed" { var database = try openMigrated(); defer database.close(); @@ -459,6 +607,206 @@ test "importSource seeds a migrated database and group 'default' keeps id 1" { ); } +test "an import keeps the materialised clients it found and lets the config claim an address" { + var threaded: std.Io.Threaded = .init(testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + + var database = try openMigrated(); + defer database.close(); + + // Two devices the server discovered. `full_source` names the second one. + // The first is seen twice, so `first_seen` and `last_seen` differ and the + // assertions below cannot pass by carrying one column into both. + try clients_repo.upsertSeen(&database, "192.168.1.5", 1700000000); + try clients_repo.upsertSeen(&database, "192.168.1.5", 1700000100); + try clients_repo.upsertSeen(&database, "fd00::1", 1700000200); + + try importText(io, &database, full_source, .{}); + + try testing.expectEqual(@as(i64, 2), try clients_repo.countClients(&database)); + + // The device the config says nothing about keeps its flag, its group and + // both timestamps: a seed is not a reason to forget when a device appeared. + var stmt = try database.prepare( + "SELECT hand_edited, group_id, first_seen, last_seen FROM clients WHERE ip = '192.168.1.5'", + ); + defer stmt.deinit(); + try testing.expect(try stmt.step()); + try testing.expectEqual(@as(i64, 0), stmt.columnInt(0)); + try testing.expectEqual(@as(i64, 1), stmt.columnInt(1)); + try testing.expectEqual(@as(i64, 1700000000), stmt.columnInt(2)); + try testing.expectEqual(@as(i64, 1700000100), stmt.columnInt(3)); + + // The one the config names belongs to the config: named, in `kids`, and + // hand-edited, so a later prune leaves it alone. + try testing.expectEqual(@as(i64, 1), try database.queryInt( + \\SELECT count(*) FROM clients c JOIN groups g ON g.id = c.group_id + \\ WHERE c.ip = 'fd00::1' AND c.hand_edited = 1 AND c.name = 'tablet' AND g.name = 'kids' + )); +} + +test "the config claiming a discovered address keeps that device's observed timestamps" { + var threaded: std.Io.Threaded = .init(testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + + var database = try openMigrated(); + defer database.close(); + + // The device is seen twice, so the two timestamps differ and neither + // assertion below can pass by carrying one column into the other. + // `full_source` names this address. + try clients_repo.upsertSeen(&database, "fd00::1", 1700000200); + try clients_repo.upsertSeen(&database, "fd00::1", 1700000900); + + try importText(io, &database, full_source, .{}); + + var stmt = try database.prepare( + "SELECT hand_edited, name, first_seen, last_seen FROM clients WHERE ip = 'fd00::1'", + ); + defer stmt.deinit(); + try testing.expect(try stmt.step()); + // The row is the config's: named, hand-edited. + try testing.expectEqual(@as(i64, 1), stmt.columnInt(0)); + try testing.expectEqualStrings("tablet", stmt.columnText(1)); + // The observation history is the tracker's. The import clock is `now`, so + // both columns would hold a value far above these if the import had written + // its own. + try testing.expectEqual(@as(i64, 1700000200), stmt.columnInt(2)); + try testing.expectEqual(@as(i64, 1700000900), stmt.columnInt(3)); +} + +test "a failed import leaves the observed timestamps exactly as they were" { + var threaded: std.Io.Threaded = .init(testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = testing.allocator; + + var database = try openMigrated(); + defer database.close(); + + // One address the config below claims, one it says nothing about. + try clients_repo.upsertSeen(&database, "fd00::1", 1700000200); + try clients_repo.upsertSeen(&database, "fd00::1", 1700000900); + try clients_repo.upsertSeen(&database, "192.168.1.5", 1700000000); + + const before = try dump(&database, gpa); + defer gpa.free(before); + + // The clients go in, the timestamps merge, and then two identical local + // records violate `UNIQUE(name, rtype, value)`. Everything the import wrote + // must go with the transaction. + const broken: model.Config = .{ + .groups = &.{.{ .name = "default" }}, + .upstreams = &.{.{ .url = "https://dns.example/dns-query" }}, + .clients = &.{.{ .ip = "fd00::1", .name = "tablet" }}, + .local_records = &.{ + .{ .name = "dup.lan", .rtype = .a, .value = "10.0.0.1" }, + .{ .name = "dup.lan", .rtype = .a, .value = "10.0.0.1" }, + }, + }; + try testing.expectError(error.Constraint, applyToDb(io, gpa, &database, broken, 42, .{})); + + const after = try dump(&database, gpa); + defer gpa.free(after); + try testing.expectEqualStrings(before, after); +} + +test "a forced re-import replaces the configured clients and keeps the materialised ones" { + var threaded: std.Io.Threaded = .init(testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + + var database = try openMigrated(); + defer database.close(); + try importText(io, &database, full_source, .{}); + try clients_repo.upsertSeen(&database, "10.0.0.9", 1700000200); + // The device the old file named keeps querying, so its row carries real + // observation history when the wipe reaches it. + try clients_repo.upsertSeen(&database, "fd00::1", 1700003000); + + try importText(io, &database, minimal_source, .{ .force = true }); + + // `full_source`'s hand-edited client went with the rest of the old + // configuration; the discovered one did not. + try testing.expectEqual(@as(i64, 1), try clients_repo.countClients(&database)); + try testing.expectEqual(@as(i64, 1700000200), try database.queryInt( + "SELECT first_seen FROM clients WHERE ip = '10.0.0.9' AND hand_edited = 0", + )); + // The lift saves a configured client's history so the merge can hand it back + // to the same address. It must never become a reason to resurrect a client + // the new file leaves out: dropping a client from the file removes it. + try testing.expectEqual(@as(i64, 0), try database.queryInt( + "SELECT count(*) FROM clients WHERE ip = 'fd00::1'", + )); + try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM upstreams")); +} + +test "a forced re-import keeps the observed timestamps of a client the config names" { + var threaded: std.Io.Threaded = .init(testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + + var database = try openMigrated(); + defer database.close(); + + // The device appears in traffic, the operator's file names it, and it keeps + // querying afterwards. The row is now `hand_edited = 1` and carries a real + // `first_seen` and a later `last_seen`. + try clients_repo.upsertSeen(&database, "fd00::1", 1700000200); + try importText(io, &database, full_source, .{}); + try clients_repo.upsertSeen(&database, "fd00::1", 1700005000); + + try importText(io, &database, full_source, .{ .force = true }); + + var stmt = try database.prepare( + "SELECT hand_edited, first_seen, last_seen FROM clients WHERE ip = 'fd00::1'", + ); + defer stmt.deinit(); + try testing.expect(try stmt.step()); + try testing.expectEqual(@as(i64, 1), stmt.columnInt(0)); + // Re-importing the same file is not an observation of the device. + try testing.expectEqual(@as(i64, 1700000200), stmt.columnInt(1)); + try testing.expectEqual(@as(i64, 1700005000), stmt.columnInt(2)); +} + +test "a failed forced import leaves every client timestamp exactly as it was" { + var threaded: std.Io.Threaded = .init(testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = testing.allocator; + + var database = try openMigrated(); + defer database.close(); + + // A configured client with observation history, and a materialised one. + try importText(io, &database, full_source, .{}); + try clients_repo.upsertSeen(&database, "fd00::1", 1700005000); + try clients_repo.upsertSeen(&database, "10.0.0.9", 1700000200); + + const before = try dump(&database, gpa); + defer gpa.free(before); + + const broken: model.Config = .{ + .groups = &.{.{ .name = "default" }}, + .upstreams = &.{.{ .url = "https://dns.example/dns-query" }}, + .clients = &.{.{ .ip = "fd00::1", .name = "tablet" }}, + .local_records = &.{ + .{ .name = "dup.lan", .rtype = .a, .value = "10.0.0.1" }, + .{ .name = "dup.lan", .rtype = .a, .value = "10.0.0.1" }, + }, + }; + try testing.expectError( + error.Constraint, + applyToDb(io, gpa, &database, broken, 42, .{ .force = true }), + ); + + const after = try dump(&database, gpa); + defer gpa.free(after); + try testing.expectEqualStrings(before, after); +} + test "applyToDb without force refuses a configured database and changes nothing" { var threaded: std.Io.Threaded = .init(testing.allocator, .{}); defer threaded.deinit(); diff --git a/src/config/validate.zig b/src/config/validate.zig index fcbf101..29e98fb 100644 --- a/src/config/validate.zig +++ b/src/config/validate.zig @@ -13,6 +13,29 @@ //! Parsers are reused, never reimplemented: `transport.Endpoint.parse` for //! upstream URLs, `NetAddress.parse` / `Prefix.parse` for addresses, and //! `dns.name.fromText` for every domain-shaped string. +//! +//! **No url is ever printed whole.** Every diagnostic that names a url renders +//! it through `safe_url.redact`, which drops the userinfo, the query and the +//! fragment. A diagnostic goes to stderr from `nxdns run` and `nxdns check`, +//! which under systemd is journald, and a url an operator supplied can carry a +//! password in its userinfo or an api key in its query. That the validator is +//! *rejecting* the url changes nothing: the line is written either way, and +//! `error.BadUrl` inputs are exactly the ones a parser cannot help redact — +//! which is why `redact` scans rather than parses. Do not add a tenth +//! diagnostic that prints `{s}` of a url. +//! +//! **No operator-supplied text is ever quoted by a format string.** A group +//! name, a client address, a rule pattern and a blocklist source name all come +//! out of the configuration file or out of a database row, and none of the four +//! is restricted to a printable character set on the way in. A `\n` in one of +//! them forges a whole diagnostic line and a `'` in one of them closes the +//! quotes a format string put around it, which forges a field. Both reach an +//! operator: `cli.zig` prints these messages to stdout, and +//! `web/handlers/mutations.zig` returns `Problem.message` as the body of a 400. +//! So every such value prints as `{f}` of `safe_url.quoteText`, which writes its +//! own quotes and escapes what would close them — the quote and its escape stay +//! in one place, where they cannot drift apart. Do not write `'{s}'` in this +//! file. const std = @import("std"); const Allocator = std.mem.Allocator; @@ -21,12 +44,22 @@ const Writer = std.Io.Writer; const model = @import("model.zig"); const address = @import("../platform/address.zig"); const dns_name = @import("../dns/name.zig"); +const safe_url = @import("../safe_url.zig"); const transport = @import("../upstream/transport.zig"); const Config = model.Config; const NetAddress = address.NetAddress; const Prefix = address.Prefix; +/// Every way this validator can say "the operator's configuration is wrong", +/// and nothing else. `config/faults.zig` consumes the set whole and calls every +/// member a configuration fault, so a member that is not a verdict on the +/// configuration would make that classification a lie. +/// +/// `OutOfMemory` is therefore not here. The validator allocates diagnostic text +/// and can fail to, but a failed allocation says nothing about the file — it +/// says the report is incomplete — so it travels in `Error` below and exits 1 +/// like every other resource failure. pub const ValidateError = error{ NoUpstreams, BadUpstreamUrl, @@ -65,13 +98,25 @@ pub const ValidateError = error{ MissingKeyPath, MissingLogPath, PasswordAndHashBothSet, - OutOfMemory, }; +/// What `validate` returns: a verdict on the configuration, or the allocation +/// failure that stopped it reaching one. Callers that must tell the two apart — +/// every one of them, because they map to different exit codes — match +/// `error.OutOfMemory` first and treat the rest as findings. +pub const Error = ValidateError || Allocator.Error; + /// What a `Problem` can carry: every semantic `ValidateError`, plus the one /// syntactic failure (`ParseZon`) that `config/import.zig` routes through the -/// same channel so a syntax error's line/column reaches the operator's output. -pub const ProblemError = ValidateError || error{ParseZon}; +/// same channel so a syntax error's line/column reaches the operator's output, +/// plus the warnings — which are never returned by `validate` and so are not +/// `ValidateError` members. +pub const ProblemError = ValidateError || error{ ParseZon, SourceInNoGroup }; + +/// `.fail` rejects the configuration and is what an exit code is computed from. +/// `.warn` reports something legal that is almost certainly not what the +/// operator meant; it is printed and it never changes an exit code. +pub const Severity = enum { fail, warn }; pub const Problem = struct { /// Dotted path into the config, e.g. "upstreams[2].url". Owned by `Diagnostics`. @@ -79,6 +124,7 @@ pub const Problem = struct { /// Human-readable, e.g. "unknown group 'kids'". Owned by `Diagnostics`. message: []const u8, err: ProblemError, + severity: Severity, }; pub const Diagnostics = struct { @@ -97,9 +143,10 @@ pub const Diagnostics = struct { self.problems.deinit(self.gpa); } - /// Both the path and the message are formatted, because a path carries the - /// offending element's index ("clients[1].ip") and a message carries the - /// offending value. `Diagnostics` owns both strings from here on. + /// Records a failure. Both the path and the message are formatted, because a + /// path carries the offending element's index ("clients[1].ip") and a + /// message carries the offending value. `Diagnostics` owns both strings from + /// here on. pub fn add( self: *Diagnostics, err: ProblemError, @@ -107,18 +154,79 @@ pub const Diagnostics = struct { path_args: anytype, comptime message_fmt: []const u8, message_args: anytype, + ) error{OutOfMemory}!void { + return self.record(.fail, err, path_fmt, path_args, message_fmt, message_args); + } + + /// Records a warning: the same rendering, and no effect on any exit code. + pub fn addWarning( + self: *Diagnostics, + err: ProblemError, + comptime path_fmt: []const u8, + path_args: anytype, + comptime message_fmt: []const u8, + message_args: anytype, + ) error{OutOfMemory}!void { + return self.record(.warn, err, path_fmt, path_args, message_fmt, message_args); + } + + fn record( + self: *Diagnostics, + severity: Severity, + err: ProblemError, + comptime path_fmt: []const u8, + path_args: anytype, + comptime message_fmt: []const u8, + message_args: anytype, ) error{OutOfMemory}!void { const path = try std.fmt.allocPrint(self.gpa, path_fmt, path_args); errdefer self.gpa.free(path); const message = try std.fmt.allocPrint(self.gpa, message_fmt, message_args); errdefer self.gpa.free(message); - try self.problems.append(self.gpa, .{ .path = path, .message = message, .err = err }); + try self.problems.append(self.gpa, .{ + .path = path, + .message = message, + .err = err, + .severity = severity, + }); } - /// One "path: message" line per problem. + /// How many problems reject the configuration. This is the number an exit + /// code is computed from; `warningCount` is reported and ignored. + pub fn failureCount(self: *const Diagnostics) usize { + return self.count(.fail); + } + + pub fn warningCount(self: *const Diagnostics) usize { + return self.count(.warn); + } + + fn count(self: *const Diagnostics, severity: Severity) usize { + var total: usize = 0; + for (self.problems.items) |problem| { + if (problem.severity == severity) total += 1; + } + return total; + } + + /// The first problem that rejects the configuration, for a caller that + /// reports one line rather than the whole list. A warning is never it. + pub fn firstFailure(self: *const Diagnostics) ?Problem { + for (self.problems.items) |problem| { + if (problem.severity == .fail) return problem; + } + return null; + } + + /// One "FAIL path: message" or "WARN path: message" line per problem, in the + /// order they were recorded. pub fn writeAll(self: *const Diagnostics, w: *Writer) Writer.Error!void { for (self.problems.items) |problem| { - try w.print("{s}: {s}\n", .{ problem.path, problem.message }); + const label = switch (problem.severity) { + .fail => "FAIL", + .warn => "WARN", + }; + try w.print("{s} {s}: {s}\n", .{ label, problem.path, problem.message }); } } }; @@ -170,25 +278,30 @@ pub fn parseResolver(text: []const u8) ResolverError!Resolver { // validate // --------------------------------------------------------------------------- -/// Collects EVERY problem into `diags`, then returns the first one's `err`. +/// Collects EVERY problem into `diags`, then returns the first failure's `err`. /// Checks run in a fixed order — the scalar sections in `Config` declaration /// order, then the collections in `Config` declaration order — so the returned /// error is deterministic for a given config. -pub fn validate(cfg: Config, diags: *Diagnostics) ValidateError!void { +/// +/// A configuration whose only problems are warnings is a valid configuration: +/// this returns nothing, and the warnings are in `diags` for the caller to +/// print. +pub fn validate(cfg: Config, diags: *Diagnostics) Error!void { var arena_state: std.heap.ArenaAllocator = .init(diags.gpa); defer arena_state.deinit(); const scratch = arena_state.allocator(); // The caller may hand in a `Diagnostics` that already holds problems from // another layer (import's ParseZon lines), so the returned error is the - // first problem THIS call recorded — always a `ValidateError`, which makes + // first failure THIS call recorded — always a `ValidateError`, which makes // the cast checked-safe. const first = diags.problems.items.len; try checkScalars(cfg, diags); try checkCollections(cfg, diags, scratch); - if (diags.problems.items.len == first) return; - return @errorCast(diags.problems.items[first].err); + for (diags.problems.items[first..]) |problem| { + if (problem.severity == .fail) return @errorCast(problem.err); + } } const min_timeout_ms = 100; @@ -291,8 +404,8 @@ fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void { error.MissingLogPath, "logging.file_path", .{}, - "output is 'file' so file_path must be a non-empty absolute path, got '{s}'", - .{cfg.logging.file_path}, + "output is 'file' so file_path must be a non-empty absolute path, got {f}", + .{safe_url.quoteText(cfg.logging.file_path)}, ); } @@ -351,15 +464,15 @@ fn checkBind( comptime family: BindFamily, ) error{OutOfMemory}!void { const addr = NetAddress.parse(text) catch { - try diags.add(error.BadBindAddress, path, .{}, "'{s}' is not an IP address", .{text}); + try diags.add(error.BadBindAddress, path, .{}, "{f} is not an IP address", .{safe_url.quoteText(text)}); return; }; switch (family) { .ip4 => if (std.meta.activeTag(addr) != NetAddress.ip4) { - try diags.add(error.BadBindAddress, path, .{}, "'{s}' is not an IPv4 address", .{text}); + try diags.add(error.BadBindAddress, path, .{}, "{f} is not an IPv4 address", .{safe_url.quoteText(text)}); }, .ip6 => if (std.meta.activeTag(addr) != NetAddress.ip6) { - try diags.add(error.BadBindAddress, path, .{}, "'{s}' is not an IPv6 address", .{text}); + try diags.add(error.BadBindAddress, path, .{}, "{f} is not an IPv6 address", .{safe_url.quoteText(text)}); }, .any => {}, } @@ -394,12 +507,25 @@ fn checkTlsEndpoint( } } -const StringSet = std.StringHashMapUnmanaged(void); +/// Each key mapped to the index of the entry that introduced it. The index is +/// there for the duplicate diagnostics: a message quoting the value cannot name +/// the entry a duplicate collides with, and for a url it cannot even show the +/// collision, because `safe_url.redact` drops the component two otherwise +/// identical urls differ in. +const IndexSet = std.StringHashMapUnmanaged(usize); -/// True when `key` was already present. `key` must outlive `set`. -fn markSeen(set: *StringSet, scratch: Allocator, key: []const u8) error{OutOfMemory}!bool { +/// The index of the entry that first carried `key`, or null when `key` is new. +/// `key` must outlive `set`. +fn firstSeen( + set: *IndexSet, + scratch: Allocator, + key: []const u8, + index: usize, +) error{OutOfMemory}!?usize { const gop = try set.getOrPut(scratch, key); - return gop.found_existing; + if (gop.found_existing) return gop.value_ptr.*; + gop.value_ptr.* = index; + return null; } /// Canonical text for an address, allocated from `scratch`. RFC 5952 form for @@ -430,8 +556,8 @@ fn checkTlsName( error.TlsNameOnNonTlsUpstream, "upstreams[{d}].tls_name", .{index}, - "tls_name is only for a tls:// upstream; '{s}' verifies by its url host", - .{server.url}, + "tls_name is only for a tls:// upstream; {f} verifies by its url host", + .{safe_url.redactQuoted(server.url)}, ); return; } @@ -441,25 +567,25 @@ fn checkTlsName( error.BadTlsName, "upstreams[{d}].tls_name", .{index}, - "'{s}' is not a valid domain name", - .{server.tls_name}, + "{f} is not a valid domain name", + .{safe_url.quoteText(server.tls_name)}, ); }; } fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{OutOfMemory}!void { - var group_names: StringSet = .empty; + var group_names: IndexSet = .empty; var has_default = false; for (cfg.groups, 0..) |group, i| { if (group.name.len == 0) { try diags.add(error.EmptyGroupName, "groups[{d}].name", .{i}, "group name is empty", .{}); - } else if (try markSeen(&group_names, scratch, group.name)) { + } else if (try firstSeen(&group_names, scratch, group.name, i)) |_| { try diags.add( error.DuplicateGroupName, "groups[{d}].name", .{i}, - "duplicate group name '{s}'", - .{group.name}, + "duplicate group name {f}", + .{safe_url.quoteText(group.name)}, ); } if (std.mem.eql(u8, group.name, "default")) has_default = true; @@ -474,7 +600,7 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ ); } - var upstream_urls: StringSet = .empty; + var upstream_urls: IndexSet = .empty; var enabled_upstreams: usize = 0; for (cfg.upstreams, 0..) |server, i| { // The scheme decides whether `tls_name` is meaningful, so the parse @@ -487,17 +613,17 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ error.BadUpstreamUrl, "upstreams[{d}].url", .{i}, - "'{s}' is not an https:// or tls:// endpoint", - .{server.url}, + "{f} is not an https:// or tls:// endpoint", + .{safe_url.redactQuoted(server.url)}, ); } - if (try markSeen(&upstream_urls, scratch, server.url)) { + if (try firstSeen(&upstream_urls, scratch, server.url, i)) |first| { try diags.add( error.DuplicateUpstreamUrl, "upstreams[{d}].url", .{i}, - "duplicate upstream url '{s}'", - .{server.url}, + "duplicate of upstreams[{d}]", + .{first}, ); } if (server.enabled) enabled_upstreams += 1; @@ -512,17 +638,17 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ ); } - var client_ips: StringSet = .empty; + var client_ips: IndexSet = .empty; for (cfg.clients, 0..) |client, i| { if (NetAddress.parse(client.ip)) |addr| { const text = try canonical(scratch, addr); - if (try markSeen(&client_ips, scratch, text)) { + if (try firstSeen(&client_ips, scratch, text, i)) |_| { try diags.add( error.DuplicateClientIp, "clients[{d}].ip", .{i}, - "duplicate client ip '{s}' (canonical form '{s}')", - .{ client.ip, text }, + "duplicate client ip {f} (canonical form {f})", + .{ safe_url.quoteText(client.ip), safe_url.quoteText(text) }, ); } } else |_| { @@ -530,24 +656,24 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ error.BadClientIp, "clients[{d}].ip", .{i}, - "'{s}' is not an IP address", - .{client.ip}, + "{f} is not an IP address", + .{safe_url.quoteText(client.ip)}, ); } try checkGroupRef(diags, &group_names, client.group, "clients[{d}].group", .{i}); } - var client_prefixes: StringSet = .empty; + var client_prefixes: IndexSet = .empty; for (cfg.client_prefixes, 0..) |entry, i| { if (Prefix.parse(entry.prefix)) |prefix| { const text = try canonical(scratch, prefix); - if (try markSeen(&client_prefixes, scratch, text)) { + if (try firstSeen(&client_prefixes, scratch, text, i)) |_| { try diags.add( error.DuplicateClientPrefix, "client_prefixes[{d}].prefix", .{i}, - "duplicate client prefix '{s}' (canonical form '{s}')", - .{ entry.prefix, text }, + "duplicate client prefix {f} (canonical form {f})", + .{ safe_url.quoteText(entry.prefix), safe_url.quoteText(text) }, ); } } else |_| { @@ -555,31 +681,31 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ error.BadClientPrefix, "client_prefixes[{d}].prefix", .{i}, - "'{s}' is not a CIDR prefix", - .{entry.prefix}, + "{f} is not a CIDR prefix", + .{safe_url.quoteText(entry.prefix)}, ); } try checkGroupRef(diags, &group_names, entry.group, "client_prefixes[{d}].group", .{i}); } - var source_urls: StringSet = .empty; + var source_urls: IndexSet = .empty; for (cfg.blocklist_sources, 0..) |source, i| { if (!sourceUrlIsValid(source.url)) { try diags.add( error.BadSourceUrl, "blocklist_sources[{d}].url", .{i}, - "'{s}' is not an http:// or https:// url with a host", - .{source.url}, + "{f} is not an http:// or https:// url with a host", + .{safe_url.redactQuoted(source.url)}, ); } - if (try markSeen(&source_urls, scratch, source.url)) { + if (try firstSeen(&source_urls, scratch, source.url, i)) |first| { try diags.add( error.DuplicateSourceUrl, "blocklist_sources[{d}].url", .{i}, - "duplicate source url '{s}'", - .{source.url}, + "duplicate of blocklist_sources[{d}]", + .{first}, ); } if (source.name.len == 0) { @@ -593,30 +719,50 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ } } - var group_source_pairs: StringSet = .empty; + var group_source_pairs: IndexSet = .empty; + var linked_sources: IndexSet = .empty; for (cfg.group_sources, 0..) |link, i| { try checkGroupRef(diags, &group_names, link.group, "group_sources[{d}].group", .{i}); + _ = try firstSeen(&linked_sources, scratch, link.source_url, i); if (!source_urls.contains(link.source_url)) { try diags.add( error.UnknownSource, "group_sources[{d}].source_url", .{i}, - "unknown blocklist source '{s}'", - .{link.source_url}, + "unknown blocklist source {f}", + .{safe_url.redactQuoted(link.source_url)}, ); } const key = try std.fmt.allocPrint(scratch, "{s}\x00{s}", .{ link.group, link.source_url }); - if (try markSeen(&group_source_pairs, scratch, key)) { + if (try firstSeen(&group_source_pairs, scratch, key, i)) |_| { try diags.add( error.DuplicateGroupSource, "group_sources[{d}]", .{i}, - "duplicate link from group '{s}' to source '{s}'", - .{ link.group, link.source_url }, + "duplicate link from group {f} to source {f}", + .{ safe_url.quoteText(link.group), safe_url.redactQuoted(link.source_url) }, ); } } + // A source no group links to is downloaded, parsed and compiled into + // nothing: `filter/compiler.zig` builds one set per group out of the + // `group_sources` links, so an unlinked source blocks not one domain. The + // operator sees a healthy source with a domain count and a filter that does + // nothing, which is exactly the shape of the tutorial session that found + // this. Legal configuration — a list may be staged before it is attached — + // so it warns and leaves the exit code alone. + for (cfg.blocklist_sources, 0..) |source, i| { + if (linked_sources.contains(source.url)) continue; + try diags.addWarning( + error.SourceInNoGroup, + "blocklist_sources[{d}]", + .{i}, + "source {f} {f} belongs to no group, so nothing it lists is blocked; link it from group_sources", + .{ safe_url.quoteText(source.name), safe_url.redactQuoted(source.url) }, + ); + } + for (cfg.rules, 0..) |rule, i| { try checkGroupRef(diags, &group_names, rule.group, "rules[{d}].group", .{i}); if (!try patternIsValid(scratch, rule.pattern, rule.kind)) { @@ -624,21 +770,21 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ error.BadRulePattern, "rules[{d}].pattern", .{i}, - "'{s}' is not a valid {s} pattern", - .{ rule.pattern, rule.kind.toDb() }, + "{f} is not a valid {s} pattern", + .{ safe_url.quoteText(rule.pattern), rule.kind.toDb() }, ); } } - var local_records: StringSet = .empty; + var local_records: IndexSet = .empty; for (cfg.local_records, 0..) |record, i| { _ = dns_name.fromText(record.name) catch { try diags.add( error.BadLocalRecordName, "local_records[{d}].name", .{i}, - "'{s}' is not a valid domain name", - .{record.name}, + "{f} is not a valid domain name", + .{safe_url.quoteText(record.name)}, ); }; if (!recordValueIsValid(record.rtype, record.value)) { @@ -646,8 +792,8 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ error.BadLocalRecordValue, "local_records[{d}].value", .{i}, - "'{s}' is not a valid {s} value", - .{ record.value, record.rtype.toDb() }, + "{f} is not a valid {s} value", + .{ safe_url.quoteText(record.value), record.rtype.toDb() }, ); } if (record.ttl < 1 or record.ttl > max_record_ttl_seconds) { @@ -664,35 +810,35 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ "{s}\x00{s}\x00{s}", .{ record.name, record.rtype.toDb(), record.value }, ); - if (try markSeen(&local_records, scratch, key)) { + if (try firstSeen(&local_records, scratch, key, i)) |_| { try diags.add( error.DuplicateLocalRecord, "local_records[{d}]", .{i}, - "duplicate local record '{s}' {s} '{s}'", - .{ record.name, record.rtype.toDb(), record.value }, + "duplicate local record {f} {s} {f}", + .{ safe_url.quoteText(record.name), record.rtype.toDb(), safe_url.quoteText(record.value) }, ); } } - var zones: StringSet = .empty; + var zones: IndexSet = .empty; for (cfg.forward_zones, 0..) |zone, i| { _ = dns_name.fromText(zone.zone) catch { try diags.add( error.BadForwardZone, "forward_zones[{d}].zone", .{i}, - "'{s}' is not a valid domain name", - .{zone.zone}, + "{f} is not a valid domain name", + .{safe_url.quoteText(zone.zone)}, ); }; - if (try markSeen(&zones, scratch, zone.zone)) { + if (try firstSeen(&zones, scratch, zone.zone, i)) |_| { try diags.add( error.DuplicateForwardZone, "forward_zones[{d}].zone", .{i}, - "duplicate forward zone '{s}'", - .{zone.zone}, + "duplicate forward zone {f}", + .{safe_url.quoteText(zone.zone)}, ); } _ = parseResolver(zone.resolver) catch { @@ -700,8 +846,8 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ error.BadResolverUrl, "forward_zones[{d}].resolver", .{i}, - "'{s}' is not a udp:// or tcp:// resolver with an IP literal and a port", - .{zone.resolver}, + "{f} is not a udp:// or tcp:// resolver with an IP literal and a port", + .{safe_url.redactQuoted(zone.resolver)}, ); }; } @@ -709,13 +855,19 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{ fn checkGroupRef( diags: *Diagnostics, - group_names: *const StringSet, + group_names: *const IndexSet, name: []const u8, comptime path_fmt: []const u8, path_args: anytype, ) error{OutOfMemory}!void { if (group_names.contains(name)) return; - try diags.add(error.UnknownGroup, path_fmt, path_args, "unknown group '{s}'", .{name}); + try diags.add( + error.UnknownGroup, + path_fmt, + path_args, + "unknown group {f}", + .{safe_url.quoteText(name)}, + ); } pub const Authority = struct { @@ -887,21 +1039,26 @@ fn expectProblem(cfg: Config, expected: ValidateError, expected_path: []const u8 var diags: Diagnostics = .init(testing.allocator); defer diags.deinit(); try testing.expectError(expected, validate(cfg, &diags)); - try testing.expect(diags.problems.items.len >= 1); - try testing.expectEqualStrings(expected_path, diags.problems.items[0].path); - try testing.expectEqual(expected, diags.problems.items[0].err); + const failure = diags.firstFailure() orelse return error.TestExpectedFailure; + try testing.expectEqualStrings(expected_path, failure.path); + try testing.expectEqual(expected, failure.err); } +/// Clean means no failures AND no warnings: a warning nobody asked for is a +/// false alarm in `nxdns check`, so it has to be as hard to add by accident as +/// a failure is. fn expectClean(cfg: Config) !void { var diags: Diagnostics = .init(testing.allocator); defer diags.deinit(); - validate(cfg, &diags) catch |err| { + const result = validate(cfg, &diags); + const failed = if (result) |_| false else |_| true; + if (failed or diags.problems.items.len != 0) { var buf: [4096]u8 = undefined; var w: Writer = .fixed(&buf); diags.writeAll(&w) catch {}; std.debug.print("unexpected problems:\n{s}", .{w.buffered()}); - return err; - }; + } + try result; try testing.expectEqual(@as(usize, 0), diags.problems.items.len); } @@ -1081,9 +1238,377 @@ test "a source url is accepted over https and over http" { .{ .url = "https://[fd00::2]:8443/hosts.txt", .name = "literal with port" }, .{ .url = "https://lists.example", .name = "no path" }, }; + // Each one is linked, because an unlinked source is a warning of its own. + cfg.group_sources = &.{ + .{ .group = "default", .source_url = "https://lists.example/hosts.txt" }, + .{ .group = "default", .source_url = "http://nas.lan:8080/lists/hosts.txt?v=2" }, + .{ .group = "default", .source_url = "https://[fd00::1]/hosts.txt" }, + .{ .group = "default", .source_url = "https://[fd00::2]:8443/hosts.txt" }, + .{ .group = "default", .source_url = "https://lists.example" }, + }; try expectClean(cfg); } +/// Renders every diagnostic `cfg` produces and fails if the secret survived +/// anywhere in the text, naming the line that leaked it. +/// +/// Over the whole report rather than one message, deliberately: a url reaches +/// nine diagnostics here, and a fix that redacts the line a test names while a +/// neighbouring line still prints the password is not a fix. +fn expectNoSecret(cfg: Config, secret: []const u8) !void { + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + validate(cfg, &diags) catch |e| switch (e) { + error.OutOfMemory => return e, + else => {}, + }; + try testing.expect(diags.problems.items.len != 0); + + for (diags.problems.items) |problem| { + if (std.mem.containsAtLeast(u8, problem.message, 1, secret)) { + std.debug.print( + "'{s}' reached a diagnostic: {s}: {s}\n", + .{ secret, problem.path, problem.message }, + ); + return error.SecretInDiagnostic; + } + } +} + +test "no diagnostic prints a url's userinfo or query" { + // Observed on the real path: a rejected source url put the operator's + // password on stderr, which under systemd is journald, where it stays. + // Rejecting the url is not a defence — the line is written either way. + const leaky_source = "http://lists:hunter2@127.0.0.1:9/hosts.txt"; + + var bad_source = baseConfig(); + bad_source.blocklist_sources = &.{.{ .url = leaky_source, .name = "ads" }}; + try expectNoSecret(bad_source, "hunter2"); + + // The same url twice: reported by `DuplicateSourceUrl`, and by the + // no-group warning that a legal one earns. + var duplicate_source = baseConfig(); + duplicate_source.blocklist_sources = &.{ + .{ .url = "https://token:hunter2@lists.example/hosts.txt", .name = "a" }, + .{ .url = "https://token:hunter2@lists.example/hosts.txt", .name = "b" }, + }; + try expectNoSecret(duplicate_source, "hunter2"); + + // An api key in the query, through the link that names an undeclared source + // and the duplicate link beside it. + var link = baseConfig(); + link.group_sources = &.{ + .{ .group = "default", .source_url = "https://lists.example/h.txt?apikey=hunter2" }, + .{ .group = "default", .source_url = "https://lists.example/h.txt?apikey=hunter2" }, + }; + try expectNoSecret(link, "hunter2"); + + // Upstreams carry the same shapes and print through the same channel. + var bad_upstream = baseConfig(); + bad_upstream.upstreams = &.{.{ .url = "ftp://user:hunter2@dns.example/" }}; + try expectNoSecret(bad_upstream, "hunter2"); + + var duplicate_upstream = baseConfig(); + duplicate_upstream.upstreams = &.{ + .{ .url = "https://dns.example/dns-query?key=hunter2" }, + .{ .url = "https://dns.example/dns-query?key=hunter2" }, + }; + try expectNoSecret(duplicate_upstream, "hunter2"); + + // A `tls_name` on a DoH upstream reports the url it does not belong to. + var tls_name = baseConfig(); + tls_name.upstreams = &.{.{ + .url = "https://dns.example/dns-query?key=hunter2", + .tls_name = "dns.example", + }}; + try expectNoSecret(tls_name, "hunter2"); + + // A resolver may not carry userinfo at all, which means the line that + // reports one is the line that would print it. + var resolver = baseConfig(); + resolver.forward_zones = &.{.{ .zone = "corp.lan", .resolver = "udp://user:hunter2@192.168.1.1:53" }}; + try expectNoSecret(resolver, "hunter2"); +} + +/// Renders every diagnostic `cfg` produces and fails if any control byte +/// survived into the text, naming the line that carried it. +/// +/// Over the whole report rather than one message, for the reason +/// `expectNoSecret` is: one operator-supplied name reaches several diagnostics, +/// and escaping the line a test names while a neighbouring line writes the raw +/// byte is not a fix. +fn expectNoControlByte(cfg: Config) !void { + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + validate(cfg, &diags) catch |e| switch (e) { + error.OutOfMemory => return e, + else => {}, + }; + try testing.expect(diags.problems.items.len != 0); + + for (diags.problems.items) |problem| { + for (problem.message) |byte| { + if (byte >= 0x20 and byte != 0x7f) continue; + std.debug.print( + "a control byte reached a diagnostic: {s}: {s}\n", + .{ problem.path, problem.message }, + ); + return error.ControlByteInDiagnostic; + } + } +} + +test "no diagnostic prints an operator's control characters" { + // Every text field an operator can write, carrying the byte that ends a + // line. A diagnostic goes to `nxdns check`'s stdout and to the body of a + // 400 from `POST /api/blocklists`, and neither is a `std.log` line, so + // nothing downstream escapes what these lines print. + const forged = "ads\n2026-01-01 ERROR nxdns: forged\x1b[2K"; + + var groups = baseConfig(); + groups.groups = &.{ .{ .name = "default" }, .{ .name = forged }, .{ .name = forged } }; + try expectNoControlByte(groups); + + var unknown_group = baseConfig(); + unknown_group.clients = &.{.{ .ip = "192.168.1.10", .group = forged }}; + try expectNoControlByte(unknown_group); + + var client = baseConfig(); + client.clients = &.{.{ .ip = forged }}; + try expectNoControlByte(client); + + var prefix = baseConfig(); + prefix.client_prefixes = &.{.{ .prefix = forged }}; + try expectNoControlByte(prefix); + + // The reported line: a name is printed by a warning that a legal + // configuration earns, so this one needs no invalid value to reach output. + var source = baseConfig(); + source.blocklist_sources = &.{.{ .url = "https://lists.example/hosts.txt", .name = forged }}; + try expectNoControlByte(source); + + var link = baseConfig(); + link.group_sources = &.{.{ .group = forged, .source_url = "https://lists.example/hosts.txt" }}; + try expectNoControlByte(link); + + // A `*` is what makes this pattern invalid for an exact rule, and so what + // makes it reach the diagnostic: a label may hold any byte on the wire, so + // `dns.name.fromText` accepts a newline inside one and a control character + // alone does not reject a pattern. + var rule = baseConfig(); + rule.rules = &.{.{ .group = "default", .pattern = "*" ++ forged, .kind = .exact, .action = .block }}; + try expectNoControlByte(rule); + + var record = baseConfig(); + record.local_records = &.{.{ .name = forged, .rtype = .a, .value = forged }}; + try expectNoControlByte(record); + + var duplicate_record = baseConfig(); + duplicate_record.local_records = &.{ + .{ .name = forged, .rtype = .cname, .value = "nas.lan" }, + .{ .name = forged, .rtype = .cname, .value = "nas.lan" }, + }; + try expectNoControlByte(duplicate_record); + + var zone = baseConfig(); + zone.forward_zones = &.{ + .{ .zone = forged, .resolver = "udp://192.168.1.1:53" }, + .{ .zone = forged, .resolver = "udp://192.168.1.1:53" }, + }; + try expectNoControlByte(zone); + + // The empty label is what rejects this name, for the reason the rule + // pattern above needs a `*`. + var tls_name = baseConfig(); + tls_name.upstreams = &.{.{ .url = "tls://1.1.1.1:853", .tls_name = ".." ++ forged }}; + try expectNoControlByte(tls_name); + + var bind = baseConfig(); + bind.dns.bind_ipv4 = forged; + try expectNoControlByte(bind); + + var log_path = baseConfig(); + log_path.logging.output = .file; + log_path.logging.file_path = forged; + try expectNoControlByte(log_path); +} + +test "a source name cannot forge a line or close the quotes around it" { + // The name is operator-supplied text out of the configuration file or out of + // a database row, printed beside a url. Unescaped, the `'` closes the quotes + // the message put around it and the `\n` ends the line, so the warning below + // would print a second line naming a url this source does not have. + var cfg = baseConfig(); + cfg.blocklist_sources = &.{.{ + .url = "https://lists.example/hosts.txt", + .name = "ads'\n2026-01-01 ERROR forged (https://decoy.example)", + }}; + + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + // Legal configuration: the name is ugly, not invalid. + try validate(cfg, &diags); + try testing.expectEqual(@as(usize, 1), diags.warningCount()); + + const problem = diags.problems.items[0]; + // The whole message, so the assertion fails in both directions: it fails if + // an escape is dropped, and it fails if the quotes are written twice. + try testing.expectEqualStrings( + "source 'ads\\'\\n2026-01-01 ERROR forged (https://decoy.example)' " ++ + "'https://lists.example' belongs to no group, so nothing it lists is blocked; " ++ + "link it from group_sources", + problem.message, + ); + + // What the operator sees: one line, which is the property the escaping + // exists for. `writeAll` ends each problem with the only newline in it. + var buf: [512]u8 = undefined; + var w: Writer = .fixed(&buf); + try diags.writeAll(&w); + try testing.expectEqual(@as(usize, 1), std.mem.count(u8, w.buffered(), "\n")); + try testing.expect(std.mem.endsWith(u8, w.buffered(), "link it from group_sources\n")); +} + +test "a redacted diagnostic still names the source the operator has to fix" { + // Redaction that costs the operator the identity of the offending entry + // would trade one failure for another. The message keeps the scheme, the + // host and the port — where the source points — and nothing else: the two + // sources here sit on one host and differ only in the path, so the message + // cannot tell them apart and is not asked to. What names the entry is + // `path`, which carries the index into the config, and that is the whole + // reason the url may be reduced to what does not carry a credential. + var cfg = baseConfig(); + cfg.blocklist_sources = &.{ + .{ .url = "http://127.0.0.1:9/good.txt", .name = "ok" }, + .{ .url = "http://lists:hunter2@127.0.0.1:9/subscribe/tok3n/hosts.txt", .name = "leaky" }, + }; + cfg.group_sources = &.{ + .{ .group = "default", .source_url = "http://127.0.0.1:9/good.txt" }, + .{ .group = "default", .source_url = "http://lists:hunter2@127.0.0.1:9/subscribe/tok3n/hosts.txt" }, + }; + + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + try testing.expectError(error.BadSourceUrl, validate(cfg, &diags)); + + const problem = diags.firstFailure() orelse return error.TestExpectedFailure; + try testing.expectEqualStrings("blocklist_sources[1].url", problem.path); + // The whole message, so the assertion fails in both directions: it fails if + // the userinfo or the path token comes back, and it fails if redaction + // over-reaches and stops saying where the source points. + try testing.expectEqualStrings( + "'http://127.0.0.1:9' is not an http:// or https:// url with a host", + problem.message, + ); + try testing.expect(!std.mem.containsAtLeast(u8, problem.message, 1, "hunter2")); + try testing.expect(!std.mem.containsAtLeast(u8, problem.message, 1, "tok3n")); +} + +test "a duplicate url names the entry it duplicates" { + // Two NextDNS profiles differ only in the path, which `safe_url.redact` + // drops, so a message quoting the url prints the same text for a colliding + // pair and for an unrelated one. The index is the evidence instead, and it + // is the first entry's index: `upstreams[2]` collides with `upstreams[0]` + // here, and nothing derived from the loop counter can produce that. + var cfg = baseConfig(); + cfg.upstreams = &.{ + .{ .url = "https://dns.nextdns.io/abcd12" }, + .{ .url = "https://dns.nextdns.io/efgh34" }, + .{ .url = "https://dns.nextdns.io/abcd12" }, + }; + + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + try testing.expectError(error.DuplicateUpstreamUrl, validate(cfg, &diags)); + + try testing.expectEqual(@as(usize, 1), diags.failureCount()); + const problem = diags.firstFailure() orelse return error.TestExpectedFailure; + try testing.expectEqualStrings("upstreams[2].url", problem.path); + try testing.expectEqualStrings("duplicate of upstreams[0]", problem.message); +} + +test "a duplicate source url names the entry it duplicates" { + var cfg = baseConfig(); + cfg.blocklist_sources = &.{ + .{ .url = "https://lists.example/d/tok3n/hosts.txt", .name = "first" }, + .{ .url = "https://lists.example/d/other/hosts.txt", .name = "second" }, + .{ .url = "https://lists.example/d/tok3n/hosts.txt", .name = "third" }, + }; + cfg.group_sources = &.{ + .{ .group = "default", .source_url = "https://lists.example/d/tok3n/hosts.txt" }, + .{ .group = "default", .source_url = "https://lists.example/d/other/hosts.txt" }, + }; + + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + try testing.expectError(error.DuplicateSourceUrl, validate(cfg, &diags)); + + try testing.expectEqual(@as(usize, 1), diags.failureCount()); + const problem = diags.firstFailure() orelse return error.TestExpectedFailure; + try testing.expectEqualStrings("blocklist_sources[2].url", problem.path); + try testing.expectEqualStrings("duplicate of blocklist_sources[0]", problem.message); + // The token the url carries is what the old wording printed, twice. + try testing.expect(!std.mem.containsAtLeast(u8, problem.message, 1, "tok3n")); +} + +test "a blocklist source in no group is one warning and no failure" { + var cfg = baseConfig(); + cfg.blocklist_sources = &.{.{ .url = "https://lists.example/hosts.txt", .name = "stevenblack" }}; + + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + // Legal configuration: no error, so no subcommand may fail on it. + try validate(cfg, &diags); + try testing.expectEqual(@as(usize, 0), diags.failureCount()); + try testing.expectEqual(@as(usize, 1), diags.warningCount()); + try testing.expectEqual(@as(?Problem, null), diags.firstFailure()); + + const problem = diags.problems.items[0]; + try testing.expectEqual(Severity.warn, problem.severity); + try testing.expectEqual(ProblemError.SourceInNoGroup, problem.err); + try testing.expectEqualStrings("blocklist_sources[0]", problem.path); + try testing.expect(std.mem.containsAtLeast(u8, problem.message, 1, "stevenblack")); + + var buf: [512]u8 = undefined; + var w: Writer = .fixed(&buf); + try diags.writeAll(&w); + try testing.expect(std.mem.startsWith(u8, w.buffered(), "WARN blocklist_sources[0]: ")); +} + +test "a source linked from any group is not warned about" { + var cfg = baseConfig(); + cfg.groups = &.{ .{ .name = "default" }, .{ .name = "kids" } }; + cfg.blocklist_sources = &.{ + .{ .url = "https://lists.example/a.txt", .name = "a" }, + .{ .url = "https://lists.example/b.txt", .name = "b" }, + }; + cfg.group_sources = &.{.{ .group = "kids", .source_url = "https://lists.example/a.txt" }}; + + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + try validate(cfg, &diags); + try testing.expectEqual(@as(usize, 1), diags.warningCount()); + try testing.expectEqualStrings("blocklist_sources[1]", diags.problems.items[0].path); +} + +test "a warning is reported next to a failure without becoming one" { + var cfg = baseConfig(); + cfg.dns.port = 0; + cfg.blocklist_sources = &.{.{ .url = "https://lists.example/hosts.txt", .name = "ads" }}; + + var diags: Diagnostics = .init(testing.allocator); + defer diags.deinit(); + try testing.expectError(error.BadPort, validate(cfg, &diags)); + try testing.expectEqual(@as(usize, 1), diags.failureCount()); + try testing.expectEqual(@as(usize, 1), diags.warningCount()); + + var buf: [512]u8 = undefined; + var w: Writer = .fixed(&buf); + try diags.writeAll(&w); + try testing.expect(std.mem.startsWith(u8, w.buffered(), "FAIL dns.port: ")); + try testing.expect(std.mem.containsAtLeast(u8, w.buffered(), 1, "\nWARN blocklist_sources[0]: ")); +} + test "error.DuplicateSourceUrl" { var cfg = baseConfig(); cfg.blocklist_sources = &.{ @@ -1274,12 +1799,13 @@ test "a config with five distinct problems yields five diagnostics and the first defer diags.deinit(); try testing.expectError(error.BadPort, validate(cfg, &diags)); try testing.expectEqual(@as(usize, 5), diags.problems.items.len); + try testing.expectEqual(@as(usize, 5), diags.failureCount()); try testing.expectEqualStrings("dns.port", diags.problems.items[0].path); var buf: [1024]u8 = undefined; var w: Writer = .fixed(&buf); try diags.writeAll(&w); - try testing.expect(std.mem.startsWith(u8, w.buffered(), "dns.port: ")); + try testing.expect(std.mem.startsWith(u8, w.buffered(), "FAIL dns.port: ")); try testing.expectEqual(@as(usize, 5), std.mem.count(u8, w.buffered(), "\n")); } diff --git a/src/filter/filter_integration_test.zig b/src/filter/filter_integration_test.zig index fdefd75..4a52a73 100644 --- a/src/filter/filter_integration_test.zig +++ b/src/filter/filter_integration_test.zig @@ -1041,17 +1041,78 @@ test "10: pruneOrphans deletes the files of a deleted source and leaves live one defer dir.close(io); try dir.writeFile(io, .{ .sub_path = "9999.list", .data = "gone.example.com\n" }); try dir.writeFile(io, .{ .sub_path = "9999.wild", .data = "" }); - // A refresh that is still running owns its temporaries, so they are not - // orphans and must survive. + // Id 9999 has no `blocklist_sources` row, so no refresh can be writing for + // it: these temporaries are what a refresh that died mid-write leaves + // behind, and the sweep is the only thing that will ever remove them. try dir.writeFile(io, .{ .sub_path = "9999.raw.tmp", .data = "" }); + try dir.writeFile(io, .{ .sub_path = "9999.list.tmp", .data = "" }); + try dir.writeFile(io, .{ .sub_path = "9999.wild.tmp", .data = "" }); + + // The live source does have a row, so its temporary is a refresh in + // progress and must survive a sweep that runs beside it. + var live_tmp_buf: [64]u8 = undefined; + const live_tmp = try std.fmt.bufPrint(&live_tmp_buf, "{d}.raw.tmp", .{id}); + try dir.writeFile(io, .{ .sub_path = live_tmp, .data = "" }); try env.mgr.pruneOrphans(io); var live_buf: [64]u8 = undefined; try dir.access(io, try std.fmt.bufPrint(&live_buf, "{d}.list", .{id}), .{}); - try dir.access(io, "9999.raw.tmp", .{}); + try dir.access(io, live_tmp, .{}); try testing.expectError(error.FileNotFound, dir.access(io, "9999.list", .{})); try testing.expectError(error.FileNotFound, dir.access(io, "9999.wild", .{})); + try testing.expectError(error.FileNotFound, dir.access(io, "9999.raw.tmp", .{})); + try testing.expectError(error.FileNotFound, dir.access(io, "9999.list.tmp", .{})); + try testing.expectError(error.FileNotFound, dir.access(io, "9999.wild.tmp", .{})); +} + +test "10b: the scheduler sweeps orphans on its own, with no operator call" { + if (!build_options.integration) return error.SkipZigTest; + + const gpa = testing.allocator; + const env = try Env.create(gpa); + defer env.destroy(); + const io = env.io(); + + const list_body = "aaa.example.com\n"; + const id = try seedSource(&env.database, source_url); + try publishFixtureFiles(env, id, list_body, ""); + // Dated now, so the startup pass finds nothing to download: this case + // asserts the sweep, and `source_url` points at a port nobody is serving. + try sources_repo.updateSourceStats(&env.database, id, .{ + .last_updated = std.Io.Clock.real.now(io).toSeconds(), + .domain_count = 1, + .wildcard_count = 0, + .skipped_regex_count = 0, + .checksum = &bodyChecksum(list_body, ""), + }); + + var dir = try env.blocklistDir(); + defer dir.close(io); + // What a source deleted while the server was down leaves, and what a + // process killed mid-refresh leaves. Nothing else in the tree removes + // either. + try dir.writeFile(io, .{ .sub_path = "9999.list", .data = "gone.example.com\n" }); + try dir.writeFile(io, .{ .sub_path = "9999.wild", .data = "" }); + try dir.writeFile(io, .{ .sub_path = "9999.raw.tmp", .data = "" }); + + // `runScheduler` is the entry point `app.zig` hands to `Io.Group`, and the + // only one the server ever calls. A disabled update stops it after the + // startup pass, so the production path runs to completion here with no + // interval to wait out. + env.mgr.update.enabled = false; + try env.mgr.runScheduler(io); + + try testing.expectError(error.FileNotFound, dir.access(io, "9999.list", .{})); + try testing.expectError(error.FileNotFound, dir.access(io, "9999.wild", .{})); + try testing.expectError(error.FileNotFound, dir.access(io, "9999.raw.tmp", .{})); + + // The live source kept its files and is still filtering: the sweep did not + // take the snapshot the same pass had just published. + var live_buf: [64]u8 = undefined; + try dir.access(io, try std.fmt.bufPrint(&live_buf, "{d}.list", .{id}), .{}); + const decision, _ = try env.evaluate("aaa.example.com"); + try testing.expect(decision.blocked); } // --------------------------------------------------------------------------- diff --git a/src/filter/manager.zig b/src/filter/manager.zig index 7983c5a..1c63f87 100644 --- a/src/filter/manager.zig +++ b/src/filter/manager.zig @@ -39,6 +39,7 @@ const std = @import("std"); const Allocator = std.mem.Allocator; const model = @import("../config/model.zig"); +const safe_url = @import("../safe_url.zig"); const db = @import("../storage/db.zig"); const clients_repo = @import("../storage/repositories/clients_repo.zig"); const groups_repo = @import("../storage/repositories/groups_repo.zig"); @@ -60,6 +61,10 @@ pub const max_error_len: usize = 128; /// `SourceStatus.url` is fixed-size so a copied status borrows nothing. A /// blocklist url longer than this is truncated in the status only; the row /// keeps it whole. +/// +/// The log form of a url is bounded separately by `safe_url.max_len`. The two +/// numbers agree today and answer different questions; neither follows the +/// other. pub const max_url_len: usize = 255; /// A compiled body larger than this is refused at load. A source that reaches @@ -80,6 +85,38 @@ const sample_buf_len: usize = parsers.sample_lines * (compiler.max_line_len + 1) /// `` is at most 20 characters and the longest suffix is `.list.tmp`. const name_buf_len: usize = 48; +/// How one blocklist source is named in a log line: by its row id and its name, +/// which are its own identity, and by its redacted url, which says where it +/// points and nothing more. +/// +/// The url used to carry the identity here on its own. It cannot: `safe_url` +/// drops the path, because a path segment is a place an operator's token lives, +/// and two sources on one host are told apart by exactly that path. The id and +/// the name are on the row every one of these lines already holds, they are +/// what the API and the web UI show, and neither can leak what the url holds. +/// The name is escaped for the same reason the url is — both are database text +/// and a newline in either would forge a log line. It carries its own quotes, +/// out of `safe_url.quoteText`, because a quote this format string added would +/// be a quote the name could close: `ads' (https://decoy.example) --` would then +/// read as a source pointing somewhere it does not. +const SourceLabel = struct { + id: i64, + name: []const u8, + url: []const u8, + + fn of(row: sources_repo.SourceRow) SourceLabel { + return .{ .id = row.id, .name = row.name, .url = row.url }; + } + + pub fn format(self: SourceLabel, w: *std.Io.Writer) std.Io.Writer.Error!void { + try w.print("source {d} {f} {f}", .{ + self.id, + safe_url.quoteText(self.name), + safe_url.redactQuoted(self.url), + }); + } +}; + pub const Paths = struct { /// ``, owned by the caller and left open for the manager's life. dir: std.Io.Dir, @@ -520,7 +557,7 @@ pub const Manager = struct { // `replace` calls — a new `.list` beside an old `.wild` — is caught // here and refreshed, not served as a half-updated list. if (!std.mem.eql(u8, stored, &bodyChecksum(list_body, wild_body))) { - log.warn("blocklist {s}: compiled files do not match the stored checksum", .{row.url}); + log.warn("blocklist {f}: compiled files do not match the stored checksum", .{SourceLabel.of(row)}); return .{ .failed = .{ .state = .load_failed, .text = "ChecksumMismatch" } }; } @@ -610,7 +647,7 @@ pub const Manager = struct { defer self.deleteQuietly(io, dir, list_tmp); defer self.deleteQuietly(io, dir, wild_tmp); - self.download(io, dir, raw_name, row.url) catch |err| switch (err) { + self.download(io, dir, raw_name, row) catch |err| switch (err) { error.OutOfMemory => return error.OutOfMemory, error.Canceled => return error.Canceled, else => { @@ -692,13 +729,16 @@ pub const Manager = struct { } /// The body goes to a temporary file, never to memory: `max_body_bytes` is - /// 64 MB and the memory budget has no room for it beside two snapshots. + /// 64 MiB and the memory budget has no room for it beside two snapshots. + /// + /// It takes the whole row rather than the url alone because its two log + /// lines name the source by its id and name, which only the row carries. fn download( self: *Manager, io: std.Io, dir: std.Io.Dir, raw_name: []const u8, - url: []const u8, + row: sources_repo.SourceRow, ) !void { const file = try dir.createFile(io, raw_name, .{ .permissions = .fromMode(0o600) }); defer file.close(io); @@ -707,14 +747,14 @@ pub const Manager = struct { defer self.gpa.free(buffer); var fw = file.writer(io, buffer); - const result = self.fetchWithin(io, url, &fw.interface) catch |err| { + const result = self.fetchWithin(io, row.url, &fw.interface) catch |err| { // `fetcher.Error.Unexpected` is what a failing sink surfaces as; // the concrete cause is on this writer, which the fetcher does not // own. if (fw.err) |cause| return cause; if (err == error.HttpStatus) { if (self.fetcher.last_status) |status| { - log.warn("blocklist {s}: http status {d}", .{ url, @intFromEnum(status) }); + log.warn("blocklist {f}: http status {d}", .{ SourceLabel.of(row), @intFromEnum(status) }); } } return err; @@ -724,7 +764,7 @@ pub const Manager = struct { // buffer this function is about to drop. try file.sync(io); - log.debug("blocklist {s}: downloaded {d} bytes", .{ url, result.bytes_read }); + log.debug("blocklist {f}: downloaded {d} bytes", .{ SourceLabel.of(row), result.bytes_read }); } /// `std.http.Client` has no per-request deadline, so the whole exchange @@ -906,7 +946,7 @@ pub const Manager = struct { err: anyerror, ) void { _ = self; - log.warn("blocklist {s}: download failed: {s}", .{ row.url, @errorName(err) }); + log.warn("blocklist {f}: download failed: {s}", .{ SourceLabel.of(row), @errorName(err) }); status.fail(.fetch_failed, @errorName(err)); } @@ -917,7 +957,7 @@ pub const Manager = struct { err: anyerror, ) void { _ = self; - log.warn("blocklist {s}: compile failed: {s}", .{ row.url, @errorName(err) }); + log.warn("blocklist {f}: compile failed: {s}", .{ SourceLabel.of(row), @errorName(err) }); status.fail(.compile_failed, @errorName(err)); } @@ -934,7 +974,7 @@ pub const Manager = struct { "NoValidEntries invalid={d} unsupported={d} long_lines={d}", .{ counts.invalid, counts.skipped_unsupported, counts.long_lines }, ) catch "NoValidEntries"; - log.warn("blocklist {s}: {s}", .{ row.url, text }); + log.warn("blocklist {f}: {s}", .{ SourceLabel.of(row), text }); status.fail(.no_valid_entries, text); } @@ -954,6 +994,13 @@ pub const Manager = struct { /// `update.enabled == false` stops after the startup pass; manual refresh /// through `refreshAll` still works. pub fn runScheduler(self: *Manager, io: std.Io) std.Io.Cancelable!void { + // Ahead of the pass, not after it. This is the sweep that collects what + // a killed process left behind: a `.raw.tmp` as large as the body the + // dead refresh was writing, and the compiled files of a source deleted + // while the server was down. Both are bytes the pass below is about to + // ask the same filesystem for. + try self.sweepOrphans(io); + self.startupPass(io) catch |err| switch (err) { error.Canceled => return error.Canceled, else => log.warn("blocklist startup pass failed: {s}", .{@errorName(err)}), @@ -968,6 +1015,11 @@ pub const Manager = struct { }; while (true) { try interval.sleep(io); + // Ahead of the gate as well as ahead of the pass: the sweep only + // unlinks, so it is the one thing here that can give a critically + // full disk room back, and gating it would keep the residue that + // helped fill the disk in the first place. + try self.sweepOrphans(io); if (self.refreshGated()) continue; self.refreshAll(io) catch |err| switch (err) { error.Canceled => return error.Canceled, @@ -976,6 +1028,21 @@ pub const Manager = struct { } } + /// `pruneOrphans` with its failure absorbed. Leftover bytes under + /// `/blocklists/` are not an outage, and a sweep that could not + /// read the directory must not cost the household the refresh pass behind + /// it — let alone the server. Cancellation is the one outcome that + /// propagates, because it means shutdown. + /// + /// Taken from outside every `*Locked` body: `pruneOrphans` takes + /// `writer_lock` itself and the mutex is not reentrant. + fn sweepOrphans(self: *Manager, io: std.Io) std.Io.Cancelable!void { + self.pruneOrphans(io) catch |err| switch (err) { + error.Canceled => return error.Canceled, + else => log.warn("pruning orphaned blocklist files failed: {s}", .{@errorName(err)}), + }; + } + /// The §11.6 gate, consulted by scheduled passes only (ruling 17). A /// download writes tens of megabytes into the blocklist directory and the /// compile writes as much again, which is exactly the "non-essential write" @@ -1045,12 +1112,26 @@ pub const Manager = struct { // orphans // ----------------------------------------------------------------------- - /// Deletes `.list` and `.wild` files whose id is no longer a - /// `blocklist_sources` row. Files of a live source are left alone, + /// Deletes the compiled files and the leftover temporaries whose id is no + /// longer a `blocklist_sources` row. Files of a live source are left alone, /// whatever their state. + /// + /// Three callers, and between them they cover every way an orphan is made: + /// `runScheduler` sweeps once before its startup pass — the residue of a + /// process that was killed mid-refresh, and of a source deleted while the + /// server was down — and again before each scheduled pass; the + /// `DELETE /api/blocklists/{id}` handler sweeps as soon as it has removed + /// the row, so the directory follows the table an operator can see instead + /// of waiting out `blocklist_update.interval_hours`. + /// + /// It is safe to call on a fresh install: `openDir` creates + /// `/blocklists/` if nothing has yet, and an empty directory + /// sweeps to nothing. pub fn pruneOrphans(self: *Manager, io: std.Io) Error!void { - // A refresh in flight owns the temporaries of a live source; the sweep - // must not run beside one and decide from a half-written directory. + // Every path that writes a temporary holds this lock too, so the sweep + // never reads a directory a refresh is halfway through. The temporaries + // it can see therefore belong to a finished or a dead refresh, and only + // those of a source with no row are removed. self.writer_lock.lockUncancelable(io); defer self.writer_lock.unlock(io); @@ -1079,14 +1160,14 @@ pub const Manager = struct { }, } orelse break; if (entry.kind != .file) continue; - const id = compiledId(entry.name) orelse continue; + const id = sourceFileId(entry.name) orelse continue; if (containsId(rows.items, id)) continue; try doomed.append(self.gpa, try self.gpa.dupe(u8, entry.name)); } for (doomed.items) |name| { self.deleteQuietly(io, dir, name); - log.info("pruned orphaned compiled file {s}", .{name}); + log.info("pruned orphaned blocklist file {s}", .{name}); } } @@ -1253,7 +1334,7 @@ const LoadOutcome = union(enum) { }; fn loadFailure(row: sources_repo.SourceRow, file_name: []const u8, err: anyerror) LoadOutcome { - log.warn("blocklist {s}: reading {s} failed: {s}", .{ row.url, file_name, @errorName(err) }); + log.warn("blocklist {f}: reading {s} failed: {s}", .{ SourceLabel.of(row), file_name, @errorName(err) }); return .{ .failed = .{ .state = .load_failed, .text = @errorName(err) } }; } @@ -1394,17 +1475,27 @@ fn compiledName(buf: *[name_buf_len]u8, id: i64, suffix: []const u8) []const u8 return std.fmt.bufPrint(buf, "{d}{s}", .{ id, suffix }) catch unreachable; } -/// The source id a compiled file belongs to, or null when the name is not one -/// of ours. Temporary files are deliberately not matched: they belong to a -/// refresh that may still be running. -fn compiledId(file_name: []const u8) ?i64 { - const stem = if (std.mem.endsWith(u8, file_name, ".list")) - file_name[0 .. file_name.len - ".list".len] - else if (std.mem.endsWith(u8, file_name, ".wild")) - file_name[0 .. file_name.len - ".wild".len] - else - return null; - return std.fmt.parseInt(i64, stem, 10) catch null; +/// Every name `compiledName` can produce, longest suffix first so `.list.tmp` +/// is never read as `.list`. +const source_file_suffixes = [_][]const u8{ ".list.tmp", ".wild.tmp", ".raw.tmp", ".list", ".wild" }; + +/// The source id a file under the blocklist directory belongs to, or null when +/// the name is not one of ours. +/// +/// The three temporaries count. A refresh that dies between writing one and +/// renaming it leaves a file no later refresh reuses and no `defer` reaches, so +/// excluding them from the sweep means nothing ever removes them. Matching them +/// is safe because `pruneOrphans` holds `writer_lock` for its whole body: every +/// path that creates a temporary runs under that same lock, so no refresh is in +/// flight while the sweep reads the directory, and a temporary the sweep sees +/// belonging to a source that still has a row is kept regardless. +fn sourceFileId(file_name: []const u8) ?i64 { + for (source_file_suffixes) |suffix| { + if (!std.mem.endsWith(u8, file_name, suffix)) continue; + const stem = file_name[0 .. file_name.len - suffix.len]; + return std.fmt.parseInt(i64, stem, 10) catch null; + } + return null; } fn containsId(rows: []const sources_repo.SourceRow, id: i64) bool { @@ -1566,6 +1657,62 @@ test "the header writer produces the documented text" { ++ "# sha256 " ++ "0" ** 64 ++ "\n", w.buffered()); } +test "the log label names a source without printing what its url carries" { + // Every `log.warn` in this file formats its subject through `SourceLabel`, + // so this is the text of those lines. A `std.log` line is not observable + // from a unit test under the default runner; the label is. + var buf: [1024]u8 = undefined; + + const row: sources_repo.SourceRow = .{ + .id = 3, + .url = "https://lists.example/download/token/hunter2/hosts.txt?apikey=s3cr3t", + .name = "ads", + .enabled = true, + .last_updated = null, + .domain_count = 0, + .wildcard_count = 0, + .skipped_regex_count = 0, + .checksum = null, + }; + const printed = try std.fmt.bufPrint(&buf, "blocklist {f}: download failed: {s}", .{ + SourceLabel.of(row), + @errorName(error.ConnectFailed), + }); + try testing.expectEqualStrings( + "blocklist source 3 'ads' 'https://lists.example': download failed: ConnectFailed", + printed, + ); + try testing.expect(!std.mem.containsAtLeast(u8, printed, 1, "hunter2")); + try testing.expect(!std.mem.containsAtLeast(u8, printed, 1, "s3cr3t")); + + // The row is database text, and a path that writes it does not have to + // validate as strictly as the config validator. Neither column may end the + // line and start one of the operator's choosing. + var forged = row; + forged.name = "ads\n2026-01-01 ERROR forged"; + forged.url = "https://lists.example\n2026-01-01 ERROR forged/hosts.txt"; + const escaped = try std.fmt.bufPrint(&buf, "blocklist {f}", .{SourceLabel.of(forged)}); + try testing.expectEqualStrings( + "blocklist source 3 'ads\\n2026-01-01 ERROR forged'" ++ + " 'https://lists.example\\n2026-01-01 ERROR forged'", + escaped, + ); + try testing.expect(!std.mem.containsAtLeast(u8, escaped, 1, "\n")); + + // A name is operator-supplied and reaches the row through the API, so it + // can close the quote this label puts around it and open a decoy that reads + // as the url of a second source. The quote it would close is escaped, and + // the escape is unambiguous because a `\` is escaped too. + var decoy = row; + decoy.name = "ads' (https://decoy.example) --"; + decoy.url = "https://lists.example/hosts.txt"; + const quoted = try std.fmt.bufPrint(&buf, "blocklist {f}", .{SourceLabel.of(decoy)}); + try testing.expectEqualStrings( + "blocklist source 3 'ads\\' (https://decoy.example) --' 'https://lists.example'", + quoted, + ); +} + test "stripHeader returns the body of a compiled file" { const file = "# nxdns blocklist\n" ++ @@ -1615,13 +1762,28 @@ test "compiledName spells the four file names of a source" { try testing.expectEqualStrings("42.list.tmp", compiledName(&buf, 42, ".list.tmp")); } -test "compiledId matches compiled files and nothing else" { - try testing.expectEqual(@as(?i64, 7), compiledId("7.list")); - try testing.expectEqual(@as(?i64, 7), compiledId("7.wild")); - try testing.expectEqual(@as(?i64, null), compiledId("7.list.tmp")); - try testing.expectEqual(@as(?i64, null), compiledId("7.raw.tmp")); - try testing.expectEqual(@as(?i64, null), compiledId("notes.list")); - try testing.expectEqual(@as(?i64, null), compiledId("README")); +test "sourceFileId matches every name a refresh writes, including the temporaries" { + try testing.expectEqual(@as(?i64, 7), sourceFileId("7.list")); + try testing.expectEqual(@as(?i64, 7), sourceFileId("7.wild")); + + // A temporary left by a refresh that died belongs to its source id, so the + // sweep can tell whether that source still has a row. + try testing.expectEqual(@as(?i64, 7), sourceFileId("7.raw.tmp")); + try testing.expectEqual(@as(?i64, 7), sourceFileId("7.list.tmp")); + try testing.expectEqual(@as(?i64, 7), sourceFileId("7.wild.tmp")); + + try testing.expectEqual(@as(?i64, null), sourceFileId("notes.list")); + try testing.expectEqual(@as(?i64, null), sourceFileId("notes.raw.tmp")); + try testing.expectEqual(@as(?i64, null), sourceFileId("7.tmp")); + try testing.expectEqual(@as(?i64, null), sourceFileId("7.raw")); + try testing.expectEqual(@as(?i64, null), sourceFileId("README")); +} + +test "every name compiledName writes is a name the sweep can attribute" { + var buf: [name_buf_len]u8 = undefined; + for (source_file_suffixes) |suffix| { + try testing.expectEqual(@as(?i64, 42), sourceFileId(compiledName(&buf, 42, suffix))); + } } test "a failed refresh keeps the fields of the compiled files still serving" { diff --git a/src/main.zig b/src/main.zig index ff5d66f..a95f2f5 100644 --- a/src/main.zig +++ b/src/main.zig @@ -18,13 +18,35 @@ comptime { _ = @import("cli.zig"); } +/// Streaming, never positional, and this is load-bearing rather than a default +/// worth changing back. +/// +/// `File.writer` builds a `.positional` writer: it keeps an offset of its own, +/// starting at zero, and pwrites there. `std.log` writes to the same descriptor +/// directly and shares the descriptor's offset. With stderr redirected to a +/// regular file the two disagree about where the end is, so the runner's first +/// flush lands on top of whatever the log already wrote at the start and the +/// operator loses those lines. `writerStreaming` uses the descriptor's own +/// offset, so both writers append. +/// +/// A terminal and a pipe are unaffected either way, because neither is seekable +/// and the positional writer falls back to streaming on them. `2>file` is where +/// it shows, and `nxdns run 2>logfile` is an ordinary way to run this program. +/// `>>` needs it too: pwrite ignores `O_APPEND`. +/// +/// Takes the `File` rather than naming stdout and stderr inside, so a test can +/// hand it a real file and read back what an operator would have. +fn runnerWriter(file: std.Io.File, io: std.Io, buffer: []u8) std.Io.File.Writer { + return file.writerStreaming(io, buffer); +} + pub fn main(init: std.process.Init) u8 { // Both `File.Writer` values are self-referential and must not move, so they // stay in these `var` slots for the whole of `main`. var out_buffer: [4096]u8 = undefined; var err_buffer: [4096]u8 = undefined; - var out = std.Io.File.stdout().writer(init.io, &out_buffer); - var err = std.Io.File.stderr().writer(init.io, &err_buffer); + var out = runnerWriter(std.Io.File.stdout(), init.io, &out_buffer); + var err = runnerWriter(std.Io.File.stderr(), init.io, &err_buffer); const runner: cli.Runner = .{ .io = init.io, @@ -55,3 +77,39 @@ pub fn main(init: std.process.Init) u8 { .help => cli.runHelp(runner), }; } + +// --------------------------------------------------------------------------- +// tests +// --------------------------------------------------------------------------- + +test "a runner writer appends after what std.log already put in a redirected file" { + // Reproduced live before this test existed: `nxdns run --config + // 2> file` lost `info(migrations): config.db migrated from schema version 0 + // to 2`, because the runner's error writer pwrote its first flush at offset + // zero, over the line the log had already written. Down a pipe the same run + // kept both lines, which is why a terminal never showed this. + var threaded: std.Io.Threaded = .init(std.testing.allocator, .{}); + defer threaded.deinit(); + const io = threaded.io(); + const gpa = std.testing.allocator; + + var tmp = std.testing.tmpDir(.{}); + defer tmp.cleanup(); + var file = try tmp.dir.createFile(io, "stderr.txt", .{}); + defer file.close(io); + + // What the log sink does: a direct write on the shared descriptor, which + // moves the descriptor's offset. A positional writer does not see it. + const log_line = "info(migrations): config.db migrated from schema version 0 to 2\n"; + try file.writeStreamingAll(io, log_line); + + const runner_line = "nxdns run failed: MissingDefaultGroup\n"; + var buffer: [4096]u8 = undefined; + var w = runnerWriter(file, io, &buffer); + try w.interface.writeAll(runner_line); + try w.interface.flush(); + + const seen = try tmp.dir.readFileAlloc(io, "stderr.txt", gpa, .limited(4096)); + defer gpa.free(seen); + try std.testing.expectEqualStrings(log_line ++ runner_line, seen); +} diff --git a/src/safe_url.zig b/src/safe_url.zig new file mode 100644 index 0000000..12f32ab --- /dev/null +++ b/src/safe_url.zig @@ -0,0 +1,947 @@ +//! The one redaction an operator-supplied string passes through before it +//! reaches a log line or an operator-facing diagnostic. +//! +//! It lives at the root rather than under `filter/` because both the blocklist +//! manager and the configuration validator print urls, and a `config/` module +//! importing `filter/` would be the wrong dependency direction. It takes bytes +//! and returns bytes: no `Io`, no allocator, no failure mode. + +const std = @import("std"); + +/// The longest text one `SafeUrl` or one `QuotedText` prints, ahead of the `...` +/// that marks a truncation. An operator-supplied url or name has no length +/// limit and a log line must have one. A `QuotedText`'s two quotes fall outside +/// the count: they are the delimiter rather than the text, and a truncated name +/// still closes the one it opened. +/// +/// The count is of printed characters, not of source bytes: an escape sequence +/// costs the two or four characters it prints, not the one byte it stands for. +/// A string of control characters therefore truncates early instead of printing +/// several times its length. +/// +/// This is deliberately a second number rather than a reuse of the blocklist +/// manager's `max_url_len`, which sizes the inline url copy inside +/// `SourceStatus`. The two answer different questions — how wide one log line +/// may be, and how large a status value a caller may keep is — and they agree +/// on 255 only because that width suits both. Neither may be changed on the +/// other's account. +pub const max_len: usize = 255; + +/// The only form of a url that may be written to a log. `format` prints the +/// scheme, the host and the port; it drops the userinfo, the path, the query +/// and the fragment; it escapes every control character; and it bounds the +/// result at `max_len`. +/// +/// Where the line falls, and why it falls there. A blocklist source url is +/// operator-supplied and nothing on the way in restricts it to a bare public +/// path: `?apikey=…`, a signed url whose signature is a query parameter, +/// `https://user:pass@host/list.txt` and `https://host/d//hosts.txt` all +/// validate, import and store. Userinfo, path, query and fragment are the four +/// components where a credential can legally live, and a log line outlives the +/// process — journald and the configured log file keep it, and neither is as +/// protected as the database row the url came from. The rule against writing a +/// secret to a log is absolute, so all four go. +/// +/// Dropping the path costs the url the one job it used to do here: telling two +/// sources on one host apart. That job was never the url's. A caller names the +/// source it reports on from the source's own identity — `filter/manager.zig` +/// prints the row id and the name beside this — and an identity read from the +/// row cannot leak what the url holds. +/// +/// What it still prints, deliberately: the scheme, the host and the port, +/// because an operator reading a failure has to know where the source points; +/// and, for a url that names no scheme at all, whatever precedes the first `/` +/// once the userinfo is removed, because nothing in such a string distinguishes +/// a host from anything else and that text is the closest thing to one. A +/// string that names a scheme without a `//` after it gets no such treatment, +/// whether or not it holds an `@` — see `redact`, where the text after such a +/// scheme is a path segment under at least one reading and is never printed. +/// +/// The host is therefore the one place a credential still survives this, and it +/// is not hypothetical: NextDNS identifies an account by a profile id, which its +/// DoH url carries in the path — `https://dns.nextdns.io/abcd12`, dropped here — +/// and its DoT url carries in the hostname — `tls://abcd12.dns.nextdns.io`, +/// kept. Nothing can be done about the second without printing no host at all, +/// which would leave every line unactionable. An operator whose provider puts a +/// secret in the hostname has published it to every resolver on the way to it +/// long before it reaches this function. +/// +/// It scans and never parses, so it cannot fail. `error.BadUrl` — the error +/// `std.Uri.parse` raises — is one of the failures reported through here, so +/// the inputs a parser refuses are exactly the inputs this must still redact. +/// Only `://` introduces an authority, though: a run of one separator, of three +/// or more, or of two that are not both `/`, leaves text that RFC 3986 reads as a +/// path and WHATWG may read as a host, and `redact` prints no host it cannot +/// settle. A `\` still ends an authority and still anchors the scheme scan — see +/// `isSeparator` — it just does not open one. +/// +/// Scanning a string a parser refuses means some of those strings have more +/// than one reading, and the readings disagree about which side of an `@` the +/// host is on. `https://lists.example?token=prefix@hunter2` is one: RFC 3986 +/// ends the authority at the `?`, so the host is `lists.example` and `hunter2` +/// is part of an api key; read the `@` as a userinfo delimiter instead and the +/// host is `hunter2`. No scan resolves that, and for a while this one chose the +/// second reading and printed `https://hunter2` — a query-string secret printed +/// as a host, which is the exact leak this file exists to stop. `authority` is +/// therefore `null` on such a url and `format` prints +/// `(ambiguous authority omitted)` in place of a host. Withholding the +/// authority is always available and never wrong; choosing a side is wrong half +/// the time, and the half it is wrong in is the half that leaks. +pub const SafeUrl = struct { + /// The scheme without its `:`, or empty when the url carries no scheme + /// delimiter. + scheme: []const u8, + /// The authority without its userinfo — the host and, when present, the + /// port — or `null` when the url admits two readings of where the host is. + /// Empty and `null` are different answers: empty says the url names no + /// authority, `null` says it names one this cannot resolve. + authority: ?[]const u8, + + /// Unquoted. A caller that prints this inside a delimiter of its own must + /// use `redactQuoted` instead, or handle the delimiter itself the way + /// `web/metrics.zig` does for a Prometheus label value. + /// + /// This renders `scheme://authority` canonically; it does not quote the + /// input's own syntax. A scheme is always followed by `://`, so + /// `tls:\\host` prints `tls://` and `https://?apikey=…` prints `https://`, + /// though only the second contained a `//`. The `://` marks where a host + /// would go, and the operator's remedy is the same either way: the url names + /// no host this can print. Nothing about which bytes separated the scheme + /// from the rest survives redaction, and nothing should — the input is not + /// reproducible from this and is not meant to be. + pub fn format(self: SafeUrl, w: *std.Io.Writer) std.Io.Writer.Error!void { + return self.write(w, .none); + } + + /// A truncation ends the value and returns, so a caller that has written an + /// opening delimiter still gets to write the closing one. + fn write(self: SafeUrl, w: *std.Io.Writer, delimiter: Delimiter) std.Io.Writer.Error!void { + var budget: usize = max_len; + if (!try writeEscaped(w, self.scheme, &budget, delimiter)) return w.writeAll("..."); + if (self.scheme.len != 0 and !try writeEscaped(w, "://", &budget, delimiter)) + return w.writeAll("..."); + const authority = self.authority orelse ambiguous_authority; + if (!try writeEscaped(w, authority, &budget, delimiter)) return w.writeAll("..."); + } +}; + +/// A redacted url in the form a caller may print inside a log line or a +/// diagnostic sentence, where it needs a delimiter to keep a host from running +/// into the words around it. +/// +/// **`format` writes the quotes**, for the reason `QuotedText` states: a +/// delimiter a caller adds is a delimiter the value can close. Redaction does +/// not make that go away. A `'` is neither a component separator nor a control +/// character, so it survives the scan into the authority — `https://ho'st/x` +/// redacts to `https://ho'st`, and inside a caller's own `'…'` that reads as +/// `'ho'` followed by loose text. The quote and its escape therefore live here, +/// together, and a caller adds none of its own. +/// +/// A caller sizing a fixed buffer needs `max_len + 3` for the value, as +/// `SafeUrl` does, plus the two quotes. +pub const QuotedUrl = struct { + url: SafeUrl, + + pub fn format(self: QuotedUrl, w: *std.Io.Writer) std.Io.Writer.Error!void { + try w.writeByte('\''); + try self.url.write(w, .single_quote); + try w.writeByte('\''); + } +}; + +/// What `format` prints where a host would go when it cannot say which text is +/// the host. It is prose rather than a placeholder host because an operator has +/// to read it as a statement about the line and not as an address: `https://` +/// alone already means "this url names no authority", and the two call for +/// different actions. +/// +/// It spends the same printing budget the authority would have, so the bound +/// `web/metrics.zig` sizes its buffer against still holds. +/// +/// An operator can write a source url whose redaction is this same text, since +/// a url that is not a url prints as itself. That collision costs nothing: it +/// makes one line say less about a source than it could, and it cannot make a +/// credential read as a host, which is the direction that matters. +const ambiguous_authority = "(ambiguous authority omitted)"; + +/// An operator-supplied string that is not a url — a blocklist source name — in +/// the only form it may be written to a log. It holds no credential by design, +/// so nothing is dropped from it; it comes out of a database row the same way a +/// url does, so a control character in it can forge a log line the same way, +/// and `format` escapes and bounds it for that reason alone. +/// +/// **`format` writes the quotes.** A caller printing a name inside a log line +/// has to delimit it, or a name with a space in it runs into the words around +/// it; and a delimiter a caller adds is a delimiter the name can close. `ads' +/// (https://decoy.example) --` inside a caller's quotes produces a line naming a +/// url no source has. So the quote and its escape live in one place, here, +/// where they cannot drift apart. A caller adds none of its own. +pub const QuotedText = struct { + text: []const u8, + + pub fn format(self: QuotedText, w: *std.Io.Writer) std.Io.Writer.Error!void { + var budget: usize = max_len; + try w.writeByte('\''); + if (!try writeEscaped(w, self.text, &budget, .single_quote)) try w.writeAll("..."); + try w.writeByte('\''); + } +}; + +/// The `SafeUrl` of `url`. Both fields borrow from `url`, which every caller +/// holds for the length of the call it prints in. +pub fn redact(url: []const u8) SafeUrl { + const first_sep = std.mem.indexOfAny(u8, url, separators) orelse url.len; + const first_colon = std.mem.indexOfScalar(u8, url, ':') orelse url.len; + + // A scheme delimiter is a colon immediately before the first separator of + // the whole string. Anchoring on the first separator is what keeps `a/b:/c` + // from reading as a scheme. It anchors on a `\` too, so a url pasted with + // backslashes still has its scheme recognised and echoed, even though a `\` + // no longer opens an authority. + var scheme: []const u8 = ""; + var rest = url; + var delimited = false; + if (first_colon + 1 == first_sep and isScheme(url[0..first_colon])) { + scheme = url[0..first_colon]; + var after = first_sep; + while (after < url.len and isSeparator(url[after])) after += 1; + // Exactly `//` introduces an authority, and nothing else does. One + // separator leaves an absolute path — RFC 3986 reads `https:/hunter2` as + // the path `/hunter2`, WHATWG reads `hunter2` as the host — and three or + // more is an empty authority to the first and a host to the second. + // + // A run of two that is not two slashes depends on the scheme. WHATWG + // converts a `\` to a `/` only for a *special* scheme, so `https:\\host` + // is contested the same way, while `tls:\\host` — `tls` is not special — + // has no reading at all under which `host` is a host. RFC 3986 gives `\` + // no meaning anywhere. + // + // So the two answers are different answers, and `authority` carries the + // difference: `null` where the readings disagree, empty where they agree + // the url names no authority. Either way the path segment stays out of + // the log, which is the property that matters; this decides only what the + // line then claims about it. + // + // An earlier revision accepted any number of separators, to keep + // `https:/user:pass@host/list` from printing its userinfo. That reason + // expired when the `/` cut moved ahead of the userinfo lookup: the + // authority of a url with no `://` now ends at its first `/`, so it holds + // no userinfo to print. Withholding it is both safe and the honest + // answer. + if (after - first_sep != 2 or url[first_sep] != '/' or url[first_sep + 1] != '/') + return .{ .scheme = scheme, .authority = contestedOrAbsent(scheme, url[after..]) }; + rest = url[after..]; + delimited = true; + } + + // A network-path reference names an authority and no scheme (RFC 3986 + // §4.2). Without this the `/` cut below lands at byte zero, the authority is + // empty, and the line reports nothing at all — including for + // `//user:pa55@lists.example/x`, where the host is not in doubt. + // Exactly `//` here too, and for the same reason. RFC 3986 reads an authority + // after `//` and nothing else — `///hunter2/x` is an empty authority and the + // path `/hunter2/x`, and `\\hunter2\x` is a path outright. WHATWG resolves a + // reference against a base, and against a special-scheme base its + // ignore-slashes state reads `hunter2` as the host in both. A run that is not + // exactly `//` is therefore contested, not settled, and it is withheld rather + // than reported as an authority the url does not have. + // + // A run of one is settled: both readings make it a path. So is any run whose + // candidate authority is empty — `\\?\C:\lists\hosts.txt` ends the authority + // at its `?` under the reading that looks for one, so neither finds a host + // and there is nothing to contest. + if (!delimited) { + var run: usize = 0; + while (run < rest.len and isSeparator(rest[run])) run += 1; + if (run >= 2) { + if (candidateAuthority(rest[run..]).len == 0) + return .{ .scheme = scheme, .authority = "" }; + if (run != 2 or rest[0] != '/' or rest[1] != '/') + return .{ .scheme = scheme, .authority = null }; + rest = rest[2..]; + delimited = true; + } + } + + // The authority ends at the first `/`. Everything from there on is path, + // query or fragment, and none of the three is printed — so an `@` after + // that slash is not userinfo by any reading, and no longer needs to be one + // to stay out of the log. + const authority = rest[0 .. std.mem.indexOfScalar(u8, rest, '/') orelse rest.len]; + + // A scheme with no separator after it is an opaque path under RFC 3986 and, + // for a special scheme, a host under WHATWG — so `https:hunter2` is either a + // path segment, which is where a token lives, or a host. The disagreement is + // the whole of the evidence, exactly as it is for `https:a@hunter2`, and this + // check runs before the `@` lookup so both readings reach it. + // + // No exception is made for a suffix that looks like a port. An earlier + // revision took the digits in `localhost:8080` for one, which also let + // `https:123456` through, whose digits are an opaque path and as much a token + // as any other text. Neither is resolved now. + // + // Note what the two cases are not. `https:` is a WHATWG special scheme, so + // `https:hunter2` is contested — an opaque path to RFC 3986, a host to + // WHATWG. `localhost:` is not special, so `localhost:8080` is a scheme and an + // opaque path to *both*, and what an operator meant by it — a host and a + // port — is a reading no parser offers. It is withheld all the same, because + // the text after the colon is a path segment under every reading and a path + // segment is never printed. + // + // So the two get the same treatment and different answers: `https:hunter2` is + // withheld as contested, `localhost:8080` as naming no authority at all. + // + // An `IP:port` is unaffected: a leading digit fails the scheme production, so + // `10.0.0.2:8080` and `[::1]:853` resolve. + if (!delimited) { + const trimmed = cut(authority); + const colon = std.mem.indexOfScalar(u8, trimmed, ':') orelse trimmed.len; + if (colon != trimmed.len and isScheme(trimmed[0..colon])) return .{ + .scheme = scheme, + .authority = contestedOrAbsent(trimmed[0..colon], trimmed[colon + 1 ..]), + }; + } + + // With no `@` there is no userinfo to remove and no side to choose. A `?`, + // a `#` or a `\` ends the authority; each of the three ends it under every + // reading of the text before it. + const at = std.mem.lastIndexOfScalar(u8, authority, '@') orelse + return .{ .scheme = scheme, .authority = cut(authority) }; + + // A `?`, a `#` or a `\` in front of that `@` makes the authority ambiguous, + // and the two readings put the host on opposite sides of the `@`. On + // `https://lists.example?token=prefix@hunter2` the text after it is an api + // key; on `https://user:pa55?@host` the text before it is a password. Both + // readings are available on both urls and nothing in either string tells + // them apart, so neither side may be printed. + if (std.mem.indexOfAny(u8, authority[0..at], "?#\\") != null) + return .{ .scheme = scheme, .authority = null }; + + // An `@` is a userinfo delimiter only inside an authority. The scan knows one + // is there in exactly three cases: a scheme delimiter introduced it, a `//` + // did, or the string names no scheme at all, where this file's rule is that + // the text before the first `/` is the closest thing to a host. The remaining + // case — a scheme with no separator after it — was withheld above, before the + // `@` was looked for, because it is ambiguous whether or not an `@` is in it. + return .{ .scheme = scheme, .authority = cut(authority[at + 1 ..]) }; +} + + +/// `text` up to the first `?`, `#` or `\`, each of which ends an authority. A +/// `\` is here rather than in `redact`'s `/` cut because the `/` cut runs before +/// the userinfo is located and a `\` before an `@` is not a separator under +/// every reading — `https://user:pa55\@host` is ambiguous, not hierarchical. +fn cut(text: []const u8) []const u8 { + return text[0 .. std.mem.indexOfAny(u8, text, "?#\\") orelse text.len]; +} + +/// The `QuotedText` of `text`, which it borrows for the length of the call that +/// prints it. The result prints its own quotes; see `QuotedText`. +pub fn quoteText(text: []const u8) QuotedText { + return .{ .text = text }; +} + +/// The `redact` of `url`, quoted. +/// +/// The rule for choosing between the two, so it does not have to be rediscovered +/// per call site: **use this whenever anything follows the url on the line.** A +/// redacted authority can still hold a space, a `:` and a `'`, so unquoted it can +/// impersonate whatever comes next — `upstream {f} failed: {t}` with a url whose +/// authority is `ok failed: Timeout` reports a failure that did not happen. +/// +/// `redact` is for the two cases where that cannot arise: the url ends the line, +/// or the caller owns the escaping for a delimiter of its own, as +/// `web/metrics.zig` does for a Prometheus label value. +/// +/// The result prints its own quotes; see `QuotedUrl`. +pub fn redactQuoted(url: []const u8) QuotedUrl { + return .{ .url = redact(url) }; +} + +/// What ends a url's components. `/` is RFC 3986's. `\` is here because the +/// strings this scan exists for are the ones a parser refuses: +/// `https:\\dns.nextdns.io\abcd12` carries an account identifier after a `\`, +/// and a scan that read only `/` printed the whole of it. +/// +/// It ends a component; it does not open an authority. What that url *names* is +/// contested — path text to RFC 3986, which gives `\` no meaning, and a host to +/// WHATWG, which reads `\` as `/` for a special scheme — so `redact` withholds +/// it. The `\` still matters here because both readings agree the account +/// identifier after it is not part of any host. +/// `null` when the readings disagree about whether `after` holds a host, empty +/// when they agree it holds none. Both withhold; they differ in what the line +/// claims, and `SafeUrl.authority` documents that as a real distinction. +/// +/// Known imprecision, in the safe direction. Three shapes return `null` where +/// both readings in fact find no host, so the line says "could not resolve" +/// where "names none" is the truth: +/// +/// - `https:/user@/x` — emptiness is decided before the userinfo is removed, +/// so `user@` counts as a candidate host when the host after it is empty. +/// - `\path@hunter2` — a leading run of one is settled as a path under both +/// readings, but reaches the late-delimiter rule instead of returning here. +/// - `file:secret` — `file` is in `special_schemes`, but WHATWG gives it its +/// own parsing states in which that input is a local path with no host. +/// +/// Each prints *less* than it could, never more; none prints the path segment. +/// Fixing them means modelling more of two standards for inputs no accepted +/// configuration can hold: every url this program takes carries `http`, `https`, +/// `tls`, `udp` or `tcp` and a `//`, so all three shapes reach a line only as +/// something the validator is already rejecting by field path. +/// +/// Only a WHATWG special scheme reads an authority out of text a `//` did not +/// introduce, so only a special scheme can disagree with RFC 3986 here. And +/// nothing is contested when the reading that looks for a host finds none — +/// `https:` alone names no authority under either. +fn contestedOrAbsent(scheme: []const u8, after: []const u8) ?[]const u8 { + if (candidateAuthority(after).len == 0) return ""; + return if (isSpecialScheme(scheme)) null else ""; +} + +/// The host a WHATWG-style reading would take out of `after`, used only to tell +/// an empty one from a non-empty one. +fn candidateAuthority(after: []const u8) []const u8 { + return cut(after[0 .. std.mem.indexOfScalar(u8, after, '/') orelse after.len]); +} + +/// WHATWG's special schemes: the ones whose urls it reads an authority into +/// without a `//`, and whose backslashes it converts to slashes. +/// +/// This is a closed set fixed by the URL Standard, not a list of what this +/// program supports. That is the difference between it and the known-scheme list +/// an earlier revision removed: this one cannot go stale when nxdns learns a new +/// transport, because it never described nxdns in the first place. +const special_schemes = [_][]const u8{ "ftp", "file", "http", "https", "ws", "wss" }; + +fn isSpecialScheme(scheme: []const u8) bool { + for (special_schemes) |special| { + if (std.ascii.eqlIgnoreCase(scheme, special)) return true; + } + return false; +} + +const separators = "/\\"; + +fn isSeparator(c: u8) bool { + return std.mem.indexOfScalar(u8, separators, c) != null; +} + +/// Whether `text` is a url scheme: an ASCII letter followed by letters, digits, +/// `+`, `-` and `.` — RFC 3986's production, which is what a scheme delimiter +/// has to look like before the text in front of it may be dropped as one. +fn isScheme(text: []const u8) bool { + if (text.len == 0) return false; + if (!std.ascii.isAlphabetic(text[0])) return false; + for (text[1..]) |c| { + if (std.ascii.isAlphanumeric(c)) continue; + if (c == '+' or c == '-' or c == '.') continue; + return false; + } + return true; +} + +/// Writes `text` with every control character escaped, spending at most +/// `budget` printed characters and never splitting an escape sequence. Returns +/// whether the whole of `text` was written. +/// +/// The control escapes are the ones `platform/logging.zig` uses on a whole log +/// message, byte for byte, and that is deliberate. `delimiter` adds the one +/// escape that sink has no reason to make. +/// +/// Two layers escape the same bytes here and neither is redundant. Do not +/// delete this one on the grounds that the log sink already covers it: that +/// sink covers every `std.log` line and nothing else, and a redacted url does +/// not only reach a sink. `config/validate.zig` builds `Problem.message` as an +/// allocated string; `web/handlers/mutations.zig` prints that message into the +/// body of the 400 from `POST /api/blocklists`, and `cli.zig` prints it to the +/// stdout of an interactive `nxdns check`. Control bytes baked into that string +/// reach an HTTP response and an operator's terminal, neither of which any log +/// sink is in a position to escape. A url has to arrive safe rather than be +/// made safe by whatever it is written to. +/// +/// The cost is that a url inside a log line is escaped twice: `\n` in the row +/// prints as `\\n` in journald and as `\n` on stdout. One notation across both +/// is what keeps that legible. +/// +/// A `\` is escaped for the same reason it is there: `\n` in the output then +/// means the byte this function replaced and `\\n` means two characters an +/// operator typed. It is what makes `\'` unambiguous as well. +fn writeEscaped( + w: *std.Io.Writer, + text: []const u8, + budget: *usize, + delimiter: Delimiter, +) std.Io.Writer.Error!bool { + const hex = "0123456789abcdef"; + for (text) |byte| { + var hex_buf: [4]u8 = undefined; + const escape: ?[]const u8 = switch (byte) { + '\\' => "\\\\", + '\n' => "\\n", + '\r' => "\\r", + '\t' => "\\t", + '\'' => if (delimiter == .single_quote) "\\'" else null, + 0x00...0x08, 0x0b, 0x0c, 0x0e...0x1f, 0x7f => blk: { + hex_buf = .{ '\\', 'x', hex[byte >> 4], hex[byte & 0x0f] }; + break :blk &hex_buf; + }, + else => null, + }; + if (escape) |seq| { + if (budget.* < seq.len) return false; + try w.writeAll(seq); + budget.* -= seq.len; + } else { + if (budget.* < 1) return false; + try w.writeByte(byte); + budget.* -= 1; + } + } + return true; +} + +/// The character the caller of `writeEscaped` wraps the escaped text in, which +/// is therefore the one character beyond the control set that has to be escaped +/// inside it. `.none` is a value nothing wraps: a `SafeUrl` is printed bare, and +/// escaping a `'` in a host would say a `'` there means something it does not. +const Delimiter = enum { none, single_quote }; + +// --------------------------------------------------------------------------- +// tests +// --------------------------------------------------------------------------- + +const testing = std.testing; + +fn expectRedacted(expected: []const u8, url: []const u8) !void { + var buf: [8 * max_len]u8 = undefined; + try testing.expectEqualStrings(expected, try std.fmt.bufPrint(&buf, "{f}", .{redact(url)})); +} + +test "redact drops the query, the fragment and the userinfo" { + // The credential shapes a source url can hold: an api key in the query, a + // signed url whose signature is a query parameter, and userinfo. + try expectRedacted( + "https://lists.example", + "https://lists.example/hosts.txt?apikey=s3cr3t", + ); + try expectRedacted( + "https://cdn.example", + "https://cdn.example/l/hosts.txt?Expires=1700000000&Signature=abc123&Key-Pair-Id=K2", + ); + try expectRedacted( + "https://lists.example", + "https://user:pa55@lists.example/hosts.txt", + ); + try expectRedacted( + "https://lists.example:8443", + "https://token@lists.example:8443/hosts.txt?t=1#frag", + ); +} + +test "redact drops a credential carried in a path segment" { + // The path is a place a token lives — a per-subscriber download url is the + // common shape — and the rule against writing a secret to a log does not + // bend for the component it sits in. + try expectRedacted( + "https://lists.example", + "https://lists.example/download/token/hunter2/hosts.txt", + ); + // NextDNS: the path segment is the account identifier, so this is the shape + // the rule exists for rather than an invented one. + try expectRedacted("https://dns.nextdns.io", "https://dns.nextdns.io/abcd12"); + try expectRedacted("https://dns.nextdns.io", "https://dns.nextdns.io/abcd12/mydevice"); + // Its DoT form puts the same id in the hostname, where redaction cannot + // reach it. Pinned so the limit stays visible rather than being discovered. + try expectRedacted("tls://abcd12.dns.nextdns.io", "tls://abcd12.dns.nextdns.io"); + try expectRedacted("https://lists.example", "https://lists.example/hunter2"); + // An `@` in the path is not userinfo, and no longer has to be told apart + // from it: the path goes either way. + try expectRedacted("https://lists.example", "https://lists.example/@who/hosts.txt"); +} + +test "redact keeps everything an operator needs to know where a source points" { + try expectRedacted("https://lists.example", "https://lists.example/hosts.txt"); + try expectRedacted("http://10.0.0.2:8080", "http://10.0.0.2:8080/a/b.txt"); + try expectRedacted("https://lists.example", "https://lists.example?apikey=s3cr3t"); + // No host: what is left still names the scheme. + try expectRedacted("https://", "https://?apikey=s3cr3t"); +} + +test "redact holds on the malformed urls a parser refuses" { + // These reach the log through `error.BadUrl`, so scanning has to hold where + // `std.Uri.parse` gives up. A scheme delimiter of one separator, of three, + // and of none at all: each once carried the userinfo into the log, because + // the scan looked for `://` and found no authority without it. + // + // Each is now withheld rather than resolved. An earlier revision read the + // text after a run of any length as the authority, which drops the userinfo + // on these four but prints the path segment of `https:/hunter2` as a host. + // Only a run of exactly two says an authority is there; what these hold after + // one, or after three, is a path to RFC 3986 and a host to WHATWG. The + // property the line asserts is unchanged — no userinfo reaches the log — and + // it now holds by withholding rather than by resolving. + try expectRedacted("https://(ambiguous authority omitted)", "https:/user:hunter2@host/list"); + try expectRedacted("https://(ambiguous authority omitted)", "https:///user:hunter2@host/list"); + try expectRedacted("HTTPS://(ambiguous authority omitted)", "HTTPS:/user:hunter2@host/list"); + try expectRedacted("https://(ambiguous authority omitted)", "https:\\user:hunter2@host\\list"); + // A separator of none at all is withheld for the same reason. + try expectRedacted("(ambiguous authority omitted)", "https:user:hunter2@host/list"); + try expectRedacted("", "?apikey=s3cr3t"); + try expectRedacted("not a url", "not a url"); + try expectRedacted("", ""); + // A colon that is not a scheme delimiter does not make one. + try expectRedacted("lists.example", "lists.example/a:/b"); + try expectRedacted("", "/download/token/hunter2/hosts.txt"); +} + +test "redact treats a backslash as a hierarchical separator" { + // A url typed or pasted with backslashes reaches these lines through + // `error.BadUrl`, so the scan has to cut on one. NextDNS again, because the + // path segment it carries is the whole account identifier. + // A `\` ends an authority but never opens one. WHATWG converts it to a `/` + // only for a special scheme, and RFC 3986 gives it no meaning at all, so none + // of these names a host that both readings agree on — and `tls:\\host` has no + // reading at all that makes it one. The account identifier stays out of the + // line either way, which is the property this test is for. + try expectRedacted("https://(ambiguous authority omitted)", "https:\\\\dns.nextdns.io\\abcd12"); + try expectRedacted("https://(ambiguous authority omitted)", "https:/\\dns.nextdns.io\\abcd12"); + try expectRedacted("https://(ambiguous authority omitted)", "https:\\/dns.nextdns.io\\abcd12"); + try expectRedacted("https://(ambiguous authority omitted)", "https:\\dns.nextdns.io\\abcd12"); + try expectRedacted("tls://", "tls:\\\\abcd12.dns.nextdns.io"); + // A token in a backslash path goes the way a token in a `/` path goes. + try expectRedacted( + "https://lists.example", + "https://lists.example\\download\\token\\hunter2\\hosts.txt", + ); + // The `\\` does not open an authority, so this is reported by its scheme + // alone. What the assertion is really for is that the token in front of the + // `@` does not reach the line, and withholding delivers that at least as + // well as resolving did. + try expectRedacted( + "https://(ambiguous authority omitted)", + "https:\\\\token@lists.example:8443\\hosts.txt?t=1#frag", + ); + // A backslash separator does not make a scheme out of a colon that is not + // one, exactly as a `/` does not. + try expectRedacted("lists.example", "lists.example\\a:\\b"); + try expectRedacted("", "\\download\\token\\hunter2\\hosts.txt"); +} + +test "redact resolves a backslash authority only where one reading survives" { + // A `\` after the userinfo ends the authority under the WHATWG reading and + // is an illegal host byte under RFC 3986's, so both readings agree that + // nothing after it is a host. Cutting there prints less than either. + try expectRedacted("https://host", "https://user@host\\list"); + // But only once a `//` has established that an authority is there at all. A + // leading `\\` does not, so the userinfo is withheld with everything else + // rather than cut out of a host that was never settled. + try expectRedacted( + "https://(ambiguous authority omitted)", + "https:\\\\user:hunter2@host\\list", + ); + + // A `\` in front of the `@` is the ambiguous shape instead, and used to + // print the text on one side of it: `https:\\lists.example\path@evil` gave + // `https://evil`. See "redact omits an authority it cannot resolve". + try expectRedacted( + "https://(ambiguous authority omitted)", + "https:\\\\lists.example\\path@evil", + ); + + // A Windows path is not a url and leaves nothing that names a host. The + // line it appears on still carries the source's row id and name. + try expectRedacted("", "\\\\?\\C:\\lists\\hosts.txt"); +} + +test "redact omits an authority it cannot resolve" { + // The shape this exists for: a query parameter whose value holds an `@`. + // The text after that `@` is the query, which is where an api key lives, and + // a scan that read it as the end of a userinfo printed the key where the + // host goes — having already dropped the real host. + try expectRedacted( + "https://(ambiguous authority omitted)", + "https://lists.example?token=prefix@hunter2", + ); + try expectRedacted( + "https://(ambiguous authority omitted)", + "https://lists.example?user=a@b.example&key=hunter2", + ); + try expectRedacted( + "https://(ambiguous authority omitted)", + "https://lists.example#f@hunter2", + ); + try expectRedacted( + "https://(ambiguous authority omitted)", + "https:\\\\lists.example\\p@hunter2", + ); + + // The mirror image, which the previous ordering fixed and this keeps fixed: + // the text before the `@` is a credential just as often, so neither side may + // be printed. Both of these once printed `user:pa55` as the host. + try expectRedacted( + "https://(ambiguous authority omitted)", + "https://user:pa55?@host/x", + ); + try expectRedacted( + "https://(ambiguous authority omitted)", + "https://user:pa55#@host/x", + ); + try expectRedacted( + "https://(ambiguous authority omitted)", + "https://user:pa55\\@host/x", + ); + try expectRedacted( + "https://(ambiguous authority omitted)", + "https:\\\\user:pa55?@host\\x", + ); + + // A url naming a scheme without a separator after it has no authority by + // RFC 3986 and one by the WHATWG parser, so the `@` in it is a path + // separator under the first reading and a userinfo delimiter under the + // second. `hunter2` is a path segment or a host and no scan can say which. + try expectRedacted("(ambiguous authority omitted)", "https:a@hunter2"); + try expectRedacted("(ambiguous authority omitted)", "https:user:hunter2@host/list"); + // `user` is not one of WHATWG's special schemes, so both readings make this a + // scheme and an opaque path and neither finds a host. It names no authority + // rather than one that cannot be resolved, and the password is withheld by + // the same rule either way. + try expectRedacted("", "user:pa55@lists.example/hosts.txt?apikey=s3cr3t"); + + // A url with no scheme at all prints the marker on its own, which is still + // not a url and still not a host. + try expectRedacted("(ambiguous authority omitted)", "lists.example?token=a@hunter2"); +} + +test "redact tells an omitted authority apart from an absent one" { + // Three outcomes an operator has to be able to tell apart, because the + // action each calls for differs: a host, no host, and a host this cannot + // name. Only the third withholds anything. + try expectRedacted("https://lists.example", "https://lists.example/hosts.txt"); + try expectRedacted("https://", "https://?apikey=s3cr3t"); + try expectRedacted("https://(ambiguous authority omitted)", "https://?u=a@hunter2"); + + try expectRedacted("lists.example", "lists.example/hosts.txt"); + try expectRedacted("", "?apikey=s3cr3t"); + try expectRedacted("(ambiguous authority omitted)", "?u=a@hunter2"); +} + +test "redact still resolves an authority whose delimiters follow the userinfo" { + // The ambiguity is an `@` after a `?`, a `#` or a `\`, not an `@` at all. + // Where the delimiters fall the way a url puts them, the host is not in + // doubt and withholding it would cost an operator the line's whole point. + try expectRedacted("https://lists.example", "https://user@lists.example?apikey=s3cr3t"); + try expectRedacted("https://lists.example", "https://user:pa55@lists.example#frag"); + try expectRedacted("https://lists.example:8443", "https://token@lists.example:8443\\hosts.txt"); + try expectRedacted("https://lists.example", "https://user@lists.example/x?u=a@b"); + // No `@` in the authority: a `?` still ends it, and nothing is ambiguous. + try expectRedacted("https://lists.example", "https://lists.example?apikey=s3cr3t"); +} + +test "redact escapes the control characters that would forge a log line" { + // A row can be written by a path that does not validate as strictly as the + // config validator, so the manager prints whatever the column holds. A + // newline in it would end the line and start one of the operator's + // choosing. + try expectRedacted( + "https://lists.example\\n2026-01-01 ERROR forged", + "https://lists.example\n2026-01-01 ERROR forged/hosts.txt", + ); + try expectRedacted("https://a\\rb", "https://a\rb/x"); + try expectRedacted("https://a\\tb", "https://a\tb/x"); + try expectRedacted("https://a\\x00b", "https://a\x00b/x"); + try expectRedacted("https://a\\x7fb", "https://a\x7fb/x"); + // An ESC would reach a terminal as a control sequence on the one path the + // log sink does not cover: `cli.zig` prints a diagnostic to stdout. + try expectRedacted("https://a\\x1bb", "https://a\x1bb/x"); + // A literal backslash ends the authority, so `redact` cannot print one at + // all. The doubling that keeps an escape sequence unambiguous is exercised + // where a backslash does survive: `quoteText`, below. + try expectRedacted("https://a", "https://a\\nb/x"); +} + +test "redact bounds the line it prints at max_len" { + const long_host = "h" ** (2 * max_len); + var buf: [8 * max_len]u8 = undefined; + + const printed = try std.fmt.bufPrint(&buf, "{f}", .{redact("https://" ++ long_host ++ "/x")}); + try testing.expectEqualStrings(("https://" ++ long_host)[0..max_len] ++ "...", printed); + + // The bound counts what is printed, so an escape cannot spend four + // characters of a log line per byte of url. + const control_host = "\n" ** max_len; + const escaped = try std.fmt.bufPrint(&buf, "{f}", .{redact("https://" ++ control_host ++ "/x")}); + try testing.expect(escaped.len <= max_len + 3); + try testing.expect(std.mem.endsWith(u8, escaped, "...")); + try testing.expect(!std.mem.containsAtLeast(u8, escaped, 1, "\n")); + + // A scheme long enough on its own truncates inside the scheme rather than + // printing it whole and starting on the host. + const long_scheme = "s" ** (2 * max_len); + const truncated = try std.fmt.bufPrint(&buf, "{f}", .{redact(long_scheme ++ ":/host")}); + try testing.expectEqualStrings(long_scheme[0..max_len] ++ "...", truncated); +} + +test "quoteText escapes and bounds an operator-supplied name" { + var buf: [8 * max_len]u8 = undefined; + + try testing.expectEqualStrings( + "'ads and trackers'", + try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads and trackers")}), + ); + try testing.expectEqualStrings( + "'ads\\n2026-01-01 ERROR forged'", + try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads\n2026-01-01 ERROR forged")}), + ); + // A name cannot close the quote around it and write what follows as though + // it were another field of the line. + try testing.expectEqualStrings( + "'ads\\' (https://decoy.example) --'", + try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads' (https://decoy.example) --")}), + ); + // Nor by escaping the escape: a `\` before the quote is doubled first, so + // `\'` in the output is this function's and never the operator's. + try testing.expectEqualStrings( + "'ads\\\\\\' (https://decoy.example) --'", + try std.fmt.bufPrint(&buf, "{f}", .{quoteText("ads\\' (https://decoy.example) --")}), + ); + // Nor by running past `max_len`: the truncation closes the quote too. + const long_name = "n" ** (2 * max_len); + try testing.expectEqualStrings( + "'" ++ long_name[0..max_len] ++ "...'", + try std.fmt.bufPrint(&buf, "{f}", .{quoteText(long_name)}), + ); + try testing.expectEqualStrings( + "'" ++ "\\'" ** (max_len / 2) ++ "...'", + try std.fmt.bufPrint(&buf, "{f}", .{quoteText("'" ** max_len)}), + ); +} + +test "redact withholds a scheme whose separator is missing, with or without an @" { + // RFC 3986 reads `hunter2` as an opaque path and a path segment is where a + // token lives; WHATWG inserts the missing `//` for a special scheme and reads + // it as the host. An earlier revision withheld this only when an `@` was + // present, so the plainer shape printed the path whole. + try expectRedacted("(ambiguous authority omitted)", "https:hunter2"); + try expectRedacted("(ambiguous authority omitted)", "https:a@hunter2"); + try expectRedacted("(ambiguous authority omitted)", "https:123456"); + + // Only a WHATWG special scheme can disagree with RFC 3986 here, because only + // a special scheme reads an authority out of text no `//` introduced. For + // every other scheme both readings say the same thing — a scheme and an + // opaque path — so these name no authority rather than an unresolved one. + // Withheld either way; what differs is what the line then claims. + try expectRedacted("", "mailto:ops@example.com"); + try expectRedacted("", "localhost:8080/hosts.txt"); + try expectRedacted("", "localhost:8080@evil/x"); + + // An `IP:port` is unaffected, because a leading digit fails the scheme + // production and there is nothing to disagree about. + try expectRedacted("10.0.0.2:8080", "10.0.0.2:8080/hosts.txt"); + try expectRedacted("[::1]:853", "[::1]:853/x"); + try expectRedacted("lists.example", "lists.example/hosts.txt"); +} + +test "redact takes exactly two separators as an authority delimiter" { + // One separator leaves an absolute path: RFC 3986 reads `https:/hunter2` as + // the path `/hunter2`, WHATWG reads `hunter2` as the host. Three or more is + // an empty authority to the first and a host to the second. An earlier + // revision accepted any run and printed the path segment as the host. + try expectRedacted("https://(ambiguous authority omitted)", "https:/hunter2"); + try expectRedacted("https://(ambiguous authority omitted)", "https:///hunter2"); + try expectRedacted("localhost://", "localhost:/hunter2"); + try expectRedacted("https://lists.example", "https://lists.example/hosts.txt"); + + // The userinfo this tolerance was introduced to protect is protected by + // withholding instead. The `/` cut runs ahead of the userinfo lookup now, so + // the authority of a url with no `://` ends before any `@` it holds. + try expectRedacted("https://(ambiguous authority omitted)", "https:/user:pass@host/list"); +} + +test "redact resolves a network-path reference instead of reporting nothing" { + // The `/` cut lands at byte zero here, so without the `//` branch every one + // of these redacted to the empty string and the line named no source at all. + try expectRedacted("lists.example", "//lists.example/hosts.txt"); + // Exactly two. A longer run is contested — an empty authority to RFC 3986, + // and `lists.example` as the host to WHATWG resolving against a + // special-scheme base — so it is withheld rather than reported as a url that + // names no authority. Those are different answers and this type keeps them + // apart. + try expectRedacted("(ambiguous authority omitted)", "///lists.example/x"); + // The authority is not in doubt, so the userinfo is dropped rather than the + // whole of it withheld. + try expectRedacted("lists.example", "//user:pa55@lists.example/hosts.txt"); + // An ambiguous one is still withheld: the branch says where the authority + // starts, not that every reading of it is settled. + try expectRedacted("(ambiguous authority omitted)", "//a?b@c"); + // A single leading `/` is a path, not an authority, and still names none. + try expectRedacted("", "/path/only"); +} + +test "the shapes redact over-withholds on print less, never more" { + // The three imprecisions `contestedOrAbsent` documents. Each says "could not + // resolve" where both readings in fact find no host, so each prints less than + // it could. Pinned because the failure that matters is the other direction: + // if one of these ever starts naming a host, this test says so. + try expectRedacted("https://(ambiguous authority omitted)", "https:/user@/x"); + try expectRedacted("https://(ambiguous authority omitted)", "https:/user:hunter2@/x"); + try expectRedacted("(ambiguous authority omitted)", "\\path@hunter2"); + try expectRedacted("(ambiguous authority omitted)", "file:secret"); + try expectRedacted("file://(ambiguous authority omitted)", "file:\\secret"); + // `file` still resolves where a `//` settles it, which is why the imprecision + // is in the classification and not in the scan. + try expectRedacted("file://host", "file://host/secret"); +} + +test "redactQuoted writes its own quotes and closes them in every exit" { + var buf: [8 * max_len]u8 = undefined; + + try testing.expectEqualStrings( + "'https://lists.example'", + try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://lists.example/hosts.txt")}), + ); + // The omitted-authority value is prose with spaces in it, which is the case + // the quotes exist for: unquoted it runs into the words of the sentence + // around it. + try testing.expectEqualStrings( + "'https://(ambiguous authority omitted)'", + try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://lists.example?token=prefix@hunter2")}), + ); + // A truncation returns early, and the closing quote still has to be written + // or the rest of the line reads as part of the value. + const long_host = "h" ** (2 * max_len); + try testing.expectEqualStrings( + "'https://" ++ ("h" ** (max_len - "https://".len)) ++ "...'", + try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://" ++ long_host ++ "/x")}), + ); +} + +test "a redacted authority cannot close the quote a caller would have added" { + var buf: [8 * max_len]u8 = undefined; + + // A `'` is neither a component separator nor a control character, so it + // survives redaction into the authority. `redact` leaves it, which is why a + // caller may not supply the quotes itself. + try testing.expectEqualStrings( + "https://ho'st", + try std.fmt.bufPrint(&buf, "{f}", .{redact("https://ho'st/x")}), + ); + try testing.expectEqualStrings( + "'https://ho\\'st'", + try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://ho'st/x")}), + ); + // The forging shape, whole: an operator-supplied url that ends the value and + // writes what follows as though the line had said it. + try testing.expectEqualStrings( + "'https://ho\\' is fine; upstreams[9] '", + try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://ho' is fine; upstreams[9] /x")}), + ); + // The escape-the-escape shape that `QuotedText` has to defend against does + // not arise here, and not because the escaper is different — it is the same + // one. A `\` ends an authority, so it never reaches the value to be doubled. + // This is asserted rather than assumed: it is the property that makes a + // single `\` in the output always this file's and never the operator's. + try testing.expectEqualStrings( + "'https://ho'", + try std.fmt.bufPrint(&buf, "{f}", .{redactQuoted("https://ho\\'st/x")}), + ); +} diff --git a/src/server/udp_server.zig b/src/server/udp_server.zig index ef78930..5f78e5d 100644 --- a/src/server/udp_server.zig +++ b/src/server/udp_server.zig @@ -77,7 +77,7 @@ pub const UdpServer = struct { /// handler, so nothing larger than 4096 bytes leaves this socket. /// /// Cost: 4096 + 65535 + the scratch below ≈ 74 KiB per slot, so the - /// default 64 slots hold ≈ 4.6 MiB. The PLAN §18 budget is 100 MB with + /// default 64 slots hold ≈ 4.6 MiB. The PLAN §18 budget is 100 MiB with /// ~1M blocked domains, so this pool takes about 5% of it. reply: [transport.max_message_len]u8, /// The handler's per-query working memory. It belongs to the slot so @@ -297,7 +297,7 @@ test "a slot's reply buffer holds a whole DNS message" { test "the default slot pool stays inside the memory budget" { // 4096 + 65535 + scratch ≈ 74 KiB per slot; 64 slots ≈ 4.6 MiB, against the - // 100 MB of PLAN §18. + // 100 MiB of PLAN §18. const options: Options = .{}; const pool_bytes = @sizeOf(UdpServer.Slot) * @as(usize, options.max_in_flight); try testing.expect(pool_bytes < 8 * 1024 * 1024); diff --git a/src/storage/db.zig b/src/storage/db.zig index 975a4aa..9f3bb68 100644 --- a/src/storage/db.zig +++ b/src/storage/db.zig @@ -1,12 +1,17 @@ //! The whole SQLite surface nxdns owns (PLAN Decision G). Nothing above this //! file calls SQLite directly. //! -//! **This file takes no `std.Io`.** It is the one deliberate exception to -//! Decision E. SQLite performs its own file I/O through its VFS; routing it -//! through `std.Io` would mean writing a custom SQLite VFS — a large, +//! **SQLite's own file I/O takes no `std.Io`.** It is the one deliberate +//! exception to Decision E. SQLite performs its own file I/O through its VFS; +//! routing it through `std.Io` would mean writing a custom SQLite VFS — a large, //! security-sensitive component bought for nothing at household scale. Every //! other storage file that touches the filesystem takes `io: std.Io`. //! +//! The exception covers SQLite, not nxdns. The one place this file does its own +//! filesystem calls — the probes guarding `OpenMode.immutable`, at the open and +//! again at `Db.verifyImmutable` — follows the ordinary rule and takes an +//! `io: std.Io`, which is why that mode carries one. +//! //! The C API is declared by hand below. No `@cImport` — the handles stay //! opaque, matching `src/platform/tls_server.zig`'s Mbed TLS approach. @@ -135,6 +140,11 @@ pub const Error = error{ SqliteError, OutOfMemory, Unexpected, + /// Not a SQLite result code. An `OpenMode.immutable` read would have + /// answered from a stale main file, because `-wal` holds bytes or the + /// files moved while the read ran. Raised by the open and again by + /// `Db.verifyImmutable`. See `OpenMode.immutable`. + WalPending, }; /// Maps a primary SQLite result code to `Error`. `SQLITE_NOMEM` becomes @@ -190,19 +200,73 @@ fn check(code: c_int) Error!void { return mapCode(code); } -pub const OpenMode = enum { read_write_create, read_write_existing, read_only, memory }; +pub const OpenMode = union(enum) { + read_write_create, + read_write_existing, + read_only, + memory, + /// Read a database that this process promises not to change, and that no + /// writer may be touching: `SQLITE_OPEN_READONLY` plus the `immutable=1` URI + /// parameter, which makes the pager treat the file like a temp file — no + /// locking, no rollback journal, no wal-index — so **no `-wal` and no `-shm` + /// appear beside it**. + /// + /// This mode exists for `nxdns check` (milestone-13 ruling F-c), which must + /// validate without writing. `.read_only` alone is not enough, and this is + /// measured, not assumed: reading a database whose header says WAL makes + /// SQLite build the wal-index, so `config.db-wal` (0 bytes) and + /// `config.db-shm` (32 KiB) are created, and a read-only connection cannot + /// remove them on close. A command that claims to write nothing must not + /// leave two files behind. Do not "simplify" this back to `.read_only`. + /// + /// What `immutable=1` costs: SQLite then **ignores any `-wal` file**. The + /// newest committed rows live there, so an immutable read of a database with + /// an un-checkpointed WAL would answer from stale data and say nothing — a + /// worse failure than the two sidecar files it removes. `Db.open` therefore + /// refuses this mode with `error.WalPending` whenever `-wal` exists and + /// is not empty; the caller reports that as a failure and names `nxdns run`, + /// which opens read-write and checkpoints, as the fix. + /// + /// The guard lives inside `open` rather than in a helper callers are trusted + /// to call, because the failure it prevents is silent: a caller that forgets + /// a helper gets a plausible wrong answer, and nothing anywhere reports it. + /// + /// **The open is half the guard.** `immutable=1` takes no lock, so nothing + /// keeps a writer out for the duration of the read, and a check made only + /// before the read can say only that the log was empty *then*. + /// `Db.verifyImmutable` makes the other half, and a caller that grades what + /// it read without calling it is back to the stale answer this mode exists + /// to refuse. + /// + /// A zero-length `-wal` does not block the open: it holds no frames, so the + /// main file is complete. That is the exact leftover the pre-F-c `check` + /// used to create. + /// + /// The `std.Io` is for that probe — the one filesystem call nxdns itself + /// makes in this file. Passing it is what makes the guard unskippable. + immutable: std.Io, +}; pub const OpenOptions = struct { mode: OpenMode = .read_write_create, busy_timeout_ms: c_int = 5000, }; +/// SQLite's name for the write-ahead log beside ``. Exported so a caller +/// reporting `error.WalPending` can name the file without hard-coding SQLite's +/// naming convention, which this file owns. +pub const wal_suffix = "-wal"; + /// One SQLite connection. /// /// A `Db` must not move once a `Stmt` prepared from it is alive: every `Stmt` /// holds a `*Db`. pub const Db = struct { handle: *c.Sqlite3, + /// Set by `OpenMode.immutable` and null in every other mode: what the + /// database file and its `-wal` looked like when the read began, for + /// `verifyImmutable` to compare against when it ends. + immutable_guard: ?ImmutableGuard = null, /// Every mode carries `FULLMUTEX` (serialized mode). Phase 6's query logger /// and Phase 8's API handlers share one handle across `std.Io` tasks, and a @@ -220,47 +284,59 @@ pub const Db = struct { .read_write_create, .memory => base | open_flag.readwrite | open_flag.create, .read_write_existing => base | open_flag.readwrite, .read_only => base | open_flag.readonly, + // Its own function: the URI buffer is 12 KiB, and every other open + // in the process would carry it in this frame. + .immutable => |io| return openImmutable(io, path, options.busy_timeout_ms), }; const filename: [:0]const u8 = switch (options.mode) { .memory => ":memory:", else => path, }; - var handle: ?*c.Sqlite3 = null; - const rc = c.sqlite3_open_v2(filename.ptr, &handle, flags, null); - if (rc != result.ok) { - // sqlite3_open_v2 allocates a handle even on failure. Read the - // message from it, then close it; dropping it leaks on every - // failed open. - // Logged at `warn`, not `err`: the failure itself reaches the - // caller as a typed error, and this line only carries the message - // that would otherwise die with the handle. - if (handle) |h| { - log.warn("sqlite3_open_v2 failed for '{s}': {s} (code {d}/{d})", .{ - filename, - std.mem.span(c.sqlite3_errmsg(h)), - rc & 0xff, - c.sqlite3_extended_errcode(h), - }); - _ = c.sqlite3_close_v2(h); - } else { - log.warn("sqlite3_open_v2 failed for '{s}': {s} (code {d})", .{ - filename, - std.mem.span(c.sqlite3_errstr(rc)), - rc, - }); - } - return mapCode(rc); - } - const h = handle orelse return error.SqliteError; + const h = try openHandle(filename, flags); + return applyBusyTimeout(h, options.busy_timeout_ms); + } - // A silently ignored busy timeout is how a contended WAL database turns - // into random SQLITE_BUSY failures under load. - check(c.sqlite3_busy_timeout(h, options.busy_timeout_ms)) catch |e| { - _ = c.sqlite3_close_v2(h); - return e; - }; - return .{ .handle = h }; + /// Proves that the files an `OpenMode.immutable` read answered from stood + /// still while it ran, and fails the read with `error.WalPending` when they + /// did not. Call it after the last read and before anything read is + /// reported. + /// + /// `immutable=1` takes no lock at all — that is what stops the pager + /// building a wal-index, and the price is that nothing keeps a writer out. + /// The probe at open time can only say the log was empty at that instant: a + /// writer that appends one frame the instant after leaves the read answering + /// from the older pages of the main file, silently, which is the whole + /// failure `OpenMode.immutable`'s guard exists to prevent. + /// + /// Two things are compared, because a writer can hide in either: + /// + /// - `-wal` holding bytes now. A log that appeared, one that grew, and + /// one written into the empty file the open accepted all land here. + /// - `` itself moving — size, inode, mtime or ctime. This is the + /// checkpoint the log cannot show: a writer that checkpointed into the main + /// file and truncated its log back to nothing leaves both stats saying "no + /// frames" while the pages the read saw have been replaced. + /// + /// Best effort, and honestly so: a filesystem with coarse timestamps can + /// hide a rewrite that lands on the same byte count in the same tick. That + /// cannot be fixed from outside SQLite's locking, and taking a lock is the + /// one thing this mode may not do. What it closes is the window a single + /// stat before the read leaves open for the whole of the read. + /// + /// Two stats and nothing else: no `-wal` or `-shm` is created, and neither + /// sidecar is removed or truncated. Calling this on a handle opened in any + /// other mode is a caller bug. + pub fn verifyImmutable(self: *Db) Error!void { + const guard = self.immutable_guard orelse unreachable; + const now = try markImmutable(guard.io, guard.path); + if (!std.meta.eql(now, guard.mark)) { + log.warn( + "'{s}' changed while it was being read without a lock; the read is not trustworthy", + .{guard.path}, + ); + return error.WalPending; + } } pub fn close(self: *Db) void { @@ -355,6 +431,199 @@ pub const Db = struct { } }; +fn openHandle(filename: [:0]const u8, flags: c_int) Error!*c.Sqlite3 { + var handle: ?*c.Sqlite3 = null; + const rc = c.sqlite3_open_v2(filename.ptr, &handle, flags, null); + if (rc != result.ok) { + // sqlite3_open_v2 allocates a handle even on failure. Read the + // message from it, then close it; dropping it leaks on every + // failed open. + // Logged at `warn`, not `err`: the failure itself reaches the + // caller as a typed error, and this line only carries the message + // that would otherwise die with the handle. + if (handle) |h| { + log.warn("sqlite3_open_v2 failed for '{s}': {s} (code {d}/{d})", .{ + filename, + std.mem.span(c.sqlite3_errmsg(h)), + rc & 0xff, + c.sqlite3_extended_errcode(h), + }); + _ = c.sqlite3_close_v2(h); + } else { + log.warn("sqlite3_open_v2 failed for '{s}': {s} (code {d})", .{ + filename, + std.mem.span(c.sqlite3_errstr(rc)), + rc, + }); + } + return mapCode(rc); + } + return handle orelse error.SqliteError; +} + +/// A silently ignored busy timeout is how a contended WAL database turns into +/// random SQLITE_BUSY failures under load. +fn applyBusyTimeout(h: *c.Sqlite3, busy_timeout_ms: c_int) Error!Db { + check(c.sqlite3_busy_timeout(h, busy_timeout_ms)) catch |e| { + _ = c.sqlite3_close_v2(h); + return e; + }; + return .{ .handle = h }; +} + +// --------------------------------------------------------------------------- +// OpenMode.immutable +// --------------------------------------------------------------------------- + +const uri_scheme = "file:"; +const uri_immutable_query = "?immutable=1"; + +/// Worst case: every byte of the longest path the platform accepts becomes +/// `%HH`. +const immutable_uri_buf_len = + uri_scheme.len + 3 * std.Io.Dir.max_path_bytes + uri_immutable_query.len + 1; + +fn openImmutable(io: std.Io, path: [:0]const u8, busy_timeout_ms: c_int) Error!Db { + const mark = try markImmutable(io, path); + + var buf: [immutable_uri_buf_len]u8 = undefined; + const uri = try immutableUri(&buf, path); + + // `uri` without `open_flag.uri` would be opened as a filename spelled + // "file:...", creating nothing and finding nothing. + const flags = open_flag.exrescode | open_flag.fullmutex | + open_flag.readonly | open_flag.uri; + const h = try openHandle(uri, flags); + var database = try applyBusyTimeout(h, busy_timeout_ms); + database.immutable_guard = .{ .io = io, .path = path, .mark = mark }; + return database; +} + +/// What `OpenMode.immutable` recorded at the start of a read so that +/// `Db.verifyImmutable` can prove nothing moved by the end of it. +pub const ImmutableGuard = struct { + io: std.Io, + /// Borrowed. Must outlive the `Db`, which every caller satisfies by owning + /// the path for at least as long as the connection it opened with it. + path: []const u8, + mark: FileMark, +}; + +/// The main database file at one instant, in the fields an outside observer can +/// compare cheaply. `atime` is deliberately absent: reading the file changes it, +/// so comparing it would report every read as a change. +/// +/// All zero when the file does not exist, which is itself a state worth +/// comparing — a database replaced by an unlink is a database that moved. +const FileMark = struct { + present: bool, + size: u64, + inode: std.Io.File.INode, + mtime_ns: i96, + ctime_ns: i96, +}; + +/// The state an immutable read must find unchanged, or `error.WalPending` when +/// `-wal` already holds bytes. +/// +/// The `-wal` rule is deliberately conservative: any non-empty log fails. +/// Deciding whether it really holds committed frames means running WAL recovery +/// — checksums, salt, the wal-index — which is the writing that +/// `OpenMode.immutable` exists to avoid. A live writer, a crash, and a +/// checkpointed-but-retained log all land here, and refusing to answer is the +/// right side to err on: the alternative is a stale answer nobody can see is +/// stale. A zero-length log holds no frames, so the main file is complete and it +/// passes. +/// +/// A failed stat is not "no WAL": it means this cannot be known, so it stays a +/// failure. +fn markImmutable(io: std.Io, path: []const u8) Error!FileMark { + var buf: [std.Io.Dir.max_path_bytes + wal_suffix.len]u8 = undefined; + const sidecar = std.fmt.bufPrint(&buf, "{s}{s}", .{ path, wal_suffix }) catch + return error.TooBig; + + if (try statOrAbsent(io, sidecar)) |wal| { + if (wal.size > 0) return error.WalPending; + } + + const main = try statOrAbsent(io, path) orelse return .{ + .present = false, + .size = 0, + .inode = 0, + .mtime_ns = 0, + .ctime_ns = 0, + }; + return .{ + .present = true, + .size = main.size, + .inode = main.inode, + .mtime_ns = main.mtime.nanoseconds, + .ctime_ns = main.ctime.nanoseconds, + }; +} + +fn statOrAbsent(io: std.Io, path: []const u8) Error!?std.Io.Dir.Stat { + return std.Io.Dir.cwd().statFile(io, path, .{}) catch |e| switch (e) { + error.FileNotFound => return null, + else => { + log.warn("cannot stat '{s}': {t}", .{ path, e }); + return error.IoErr; + }, + }; +} + +/// `file:` + percent-encoded `path` + `?immutable=1`. +/// +/// The encoding is load-bearing, not cosmetic. `?` opens SQLite's query section +/// and `#` its fragment, so an unencoded data directory named `dns?db` would +/// silently open a *different* file; `%` must be encoded because SQLite decodes +/// `%HH` on its way back to a filename. `--data-dir` is operator input, so all +/// three are reachable. +/// +/// Everything outside the unreserved set (`A-Z a-z 0-9 - . _ ~ /`) is encoded, +/// which is always safe: SQLite decodes every escape in the path before handing +/// the name to its VFS, so the bytes it opens are the bytes passed in. +/// +/// `/` stays literal to keep diagnostics readable, with one exception. SQLite +/// reads `file://…` as a URI authority and rejects any authority but the empty +/// one or `localhost` (`sqlite3ParseUri`), so a path beginning with `//` — legal +/// POSIX — has its second slash encoded. +fn immutableUri(buf: []u8, path: []const u8) error{TooBig}![:0]const u8 { + var out: usize = 0; + try appendSlice(buf, &out, uri_scheme); + for (path, 0..) |ch, i| { + const opens_authority = i == 1 and ch == '/' and path[0] == '/'; + if (isUriUnreserved(ch) and !opens_authority) { + try appendByte(buf, &out, ch); + } else { + const hex = "0123456789ABCDEF"; + try appendByte(buf, &out, '%'); + try appendByte(buf, &out, hex[ch >> 4]); + try appendByte(buf, &out, hex[ch & 0xf]); + } + } + try appendSlice(buf, &out, uri_immutable_query); + try appendByte(buf, &out, 0); + return buf[0 .. out - 1 :0]; +} + +fn isUriUnreserved(ch: u8) bool { + return switch (ch) { + 'a'...'z', 'A'...'Z', '0'...'9', '-', '.', '_', '~', '/' => true, + else => false, + }; +} + +fn appendByte(buf: []u8, out: *usize, ch: u8) error{TooBig}!void { + if (out.* == buf.len) return error.TooBig; + buf[out.*] = ch; + out.* += 1; +} + +fn appendSlice(buf: []u8, out: *usize, bytes: []const u8) error{TooBig}!void { + for (bytes) |ch| try appendByte(buf, out, ch); +} + /// One prepared statement. /// /// There is deliberately **no prepared-statement cache in this milestone**. @@ -735,6 +1004,235 @@ test "a row-producing statement reports its row through step" { try testing.expect(try stmt.step()); } +// --------------------------------------------------------------------------- +// OpenMode.immutable +// --------------------------------------------------------------------------- + +/// `std.testing.tmpDir` creates its directory under `.zig-cache/tmp/` relative to +/// the process working directory, which is also how SQLite's VFS resolves the +/// filename it is handed (`queries_repo.zig:721`). +const tmp_prefix = ".zig-cache/tmp/"; +const sub_path_len = @typeInfo(@FieldType(testing.TmpDir, "sub_path")).array.len; +const test_io = testing.io; + +fn tmpPath(buf: []u8, tmp: *const testing.TmpDir, name: []const u8) ![:0]const u8 { + return std.fmt.bufPrintZ(buf, "{s}{s}/{s}", .{ tmp_prefix, &tmp.sub_path, name }); +} + +/// A file database in WAL mode holding one row, `id = marker`. Closing the last +/// connection checkpoints and unlinks both sidecars, but the header keeps saying +/// WAL — which is what makes a later `.read_only` open recreate them. +fn writeWalDatabase(path: [:0]const u8, marker: i64) !void { + var database = try Db.open(path, .{ .mode = .read_write_create }); + defer database.close(); + try applyPragmas(&database, .{}); + try database.exec("CREATE TABLE t (id INTEGER PRIMARY KEY);"); + var stmt = try database.prepare("INSERT INTO t (id) VALUES (?1)"); + defer stmt.deinit(); + try stmt.bindInt(1, marker); + try stmt.exec(); +} + +/// A second writer doing what a running nxdns does: it appends to the +/// write-ahead log and, being the last connection, checkpoints into the main +/// file and unlinks both sidecars on close. The blob is what makes the main +/// file grow by whole pages, so a test that watches for the change does not rest +/// on the filesystem's timestamp resolution. +fn checkpointOver(path: [:0]const u8) !void { + var database = try Db.open(path, .{ .mode = .read_write_existing }); + defer database.close(); + try applyPragmas(&database, .{}); + try database.exec("INSERT INTO t (id) VALUES (8);"); + try database.exec("CREATE TABLE bulk (v TEXT);"); + try database.exec("INSERT INTO bulk (v) VALUES (hex(randomblob(30000)));"); +} + +fn expectAbsent(dir: std.Io.Dir, name: []const u8) !void { + dir.access(test_io, name, .{}) catch |e| switch (e) { + error.FileNotFound => return, + else => |other| return other, + }; + std.debug.print("sidecar '{s}' exists and must not\n", .{name}); + return error.SidecarPresent; +} + +test "immutableUri encodes what would otherwise change which file is opened" { + var buf: [256]u8 = undefined; + + try testing.expectEqualStrings( + "file:/var/lib/nxdns/config.db?immutable=1", + try immutableUri(&buf, "/var/lib/nxdns/config.db"), + ); + // '?' would start SQLite's query section, '#' its fragment, '%' an escape. + try testing.expectEqualStrings( + "file:/data%3Fdir/config.db?immutable=1", + try immutableUri(&buf, "/data?dir/config.db"), + ); + try testing.expectEqualStrings( + "file:/data%23dir/config.db?immutable=1", + try immutableUri(&buf, "/data#dir/config.db"), + ); + try testing.expectEqualStrings( + "file:/data%25dir/config.db?immutable=1", + try immutableUri(&buf, "/data%dir/config.db"), + ); + try testing.expectEqualStrings( + "file:/a%20b/c%3Fd%23e%25f.db?immutable=1", + try immutableUri(&buf, "/a b/c?d#e%f.db"), + ); + // A relative path stays relative: SQLite's VFS resolves it against the + // working directory, exactly as a bare filename would be. + try testing.expectEqualStrings( + "file:config.db?immutable=1", + try immutableUri(&buf, "config.db"), + ); + // A leading "//" would be read as a URI authority and rejected. + try testing.expectEqualStrings( + "file:/%2Fnet/share/config.db?immutable=1", + try immutableUri(&buf, "//net/share/config.db"), + ); + // Only the authority position is special: "//" further in stays literal. + try testing.expectEqualStrings( + "file:/net//share/config.db?immutable=1", + try immutableUri(&buf, "/net//share/config.db"), + ); + // Non-ASCII bytes survive the round trip because SQLite decodes them back. + try testing.expectEqualStrings( + "file:/caf%C3%A9/config.db?immutable=1", + try immutableUri(&buf, "/café/config.db"), + ); + + var small: [16]u8 = undefined; + try testing.expectError(error.TooBig, immutableUri(&small, "/var/lib/nxdns/config.db")); +} + +test "an immutable open of a WAL database creates no -wal and no -shm" { + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + var path_buf: [tmp_prefix.len + sub_path_len + 32]u8 = undefined; + const path = try tmpPath(&path_buf, &tmp, "config.db"); + + try writeWalDatabase(path, 7); + try expectAbsent(tmp.dir, "config.db-wal"); + try expectAbsent(tmp.dir, "config.db-shm"); + + { + var database = try Db.open(path, .{ .mode = .{ .immutable = test_io } }); + defer database.close(); + try testing.expectEqual(@as(i64, 7), try database.queryInt("SELECT id FROM t")); + // The point of the mode: a `.read_only` open creates both of these here, + // and cannot delete them on close. + try expectAbsent(tmp.dir, "config.db-wal"); + try expectAbsent(tmp.dir, "config.db-shm"); + } + try expectAbsent(tmp.dir, "config.db-wal"); + try expectAbsent(tmp.dir, "config.db-shm"); +} + +test "an immutable open of a path holding URI metacharacters opens the intended file" { + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + var decoy_buf: [tmp_prefix.len + sub_path_len + 32]u8 = undefined; + var path_buf: [tmp_prefix.len + sub_path_len + 64]u8 = undefined; + + // Unencoded, SQLite cuts the filename at the '?' and opens "/d". That + // file exists here and holds a different database, so the failure without + // percent-encoding is a wrong answer, not an error. + try tmp.dir.createDirPath(test_io, "d?x#y%z"); + try writeWalDatabase(try tmpPath(&decoy_buf, &tmp, "d"), 99); + const path = try tmpPath(&path_buf, &tmp, "d?x#y%z/config.db"); + try writeWalDatabase(path, 7); + + var database = try Db.open(path, .{ .mode = .{ .immutable = test_io } }); + defer database.close(); + try testing.expectEqual(@as(i64, 7), try database.queryInt("SELECT id FROM t")); +} + +test "an immutable open refuses a database whose -wal holds bytes" { + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + var path_buf: [tmp_prefix.len + sub_path_len + 32]u8 = undefined; + const path = try tmpPath(&path_buf, &tmp, "config.db"); + try writeWalDatabase(path, 7); + + // What an unclean shutdown leaves behind, written directly so the case does + // not depend on when SQLite decides to checkpoint. + try tmp.dir.writeFile(test_io, .{ .sub_path = "config.db" ++ wal_suffix, .data = &([_]u8{0x37} ** 32) }); + try testing.expectError(error.WalPending, Db.open(path, .{ .mode = .{ .immutable = test_io } })); + + // A zero-length `-wal` holds no frames, so the main file is complete: the + // exact leftover the pre-F-c `check` created must not block a check. + try tmp.dir.writeFile(test_io, .{ .sub_path = "config.db" ++ wal_suffix, .data = "" }); + var database = try Db.open(path, .{ .mode = .{ .immutable = test_io } }); + defer database.close(); + try testing.expectEqual(@as(i64, 7), try database.queryInt("SELECT id FROM t")); +} + +test "an immutable read refuses a -wal that arrives while it is in flight" { + // The probe at open time can only say the log was empty *then*. + // `immutable=1` takes no lock, so a writer is free to arrive one instant + // later, and the read goes on answering from the older pages of the main + // file with nothing anywhere reporting it. + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + var path_buf: [tmp_prefix.len + sub_path_len + 32]u8 = undefined; + const path = try tmpPath(&path_buf, &tmp, "config.db"); + try writeWalDatabase(path, 7); + + var database = try Db.open(path, .{ .mode = .{ .immutable = test_io } }); + defer database.close(); + try testing.expectEqual(@as(i64, 7), try database.queryInt("SELECT id FROM t")); + // Nothing has moved yet, so the read stands. + try database.verifyImmutable(); + + // A zero-length log still holds no frames: the rule at the end of the read + // is the rule at the start of it. + try tmp.dir.writeFile(test_io, .{ .sub_path = "config.db" ++ wal_suffix, .data = "" }); + try database.verifyImmutable(); + + // Frames, now, in the log the open accepted as empty. + try tmp.dir.writeFile(test_io, .{ + .sub_path = "config.db" ++ wal_suffix, + .data = &([_]u8{0x37} ** 32), + }); + try testing.expectError(error.WalPending, database.verifyImmutable()); + // Repeatable: reporting the race is all it does. + try testing.expectError(error.WalPending, database.verifyImmutable()); + + // And it repairs nothing. The operator's log is byte for byte what was + // written, and no wal-index appeared beside it. + const wal = try tmp.dir.statFile(test_io, "config.db" ++ wal_suffix, .{}); + try testing.expectEqual(@as(u64, 32), wal.size); + try expectAbsent(tmp.dir, "config.db-shm"); +} + +test "an immutable read refuses a main file checkpointed under it" { + // The case a `-wal` probe cannot see at either end: a writer checkpointed + // into the main file and, closing, unlinked its log again. Both stats say + // "no frames" while the pages the read answered from have been replaced. + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + var path_buf: [tmp_prefix.len + sub_path_len + 32]u8 = undefined; + const path = try tmpPath(&path_buf, &tmp, "config.db"); + try writeWalDatabase(path, 7); + + var database = try Db.open(path, .{ .mode = .{ .immutable = test_io } }); + defer database.close(); + try testing.expectEqual(@as(i64, 7), try database.queryInt("SELECT id FROM t")); + try database.verifyImmutable(); + + const before = try tmp.dir.statFile(test_io, "config.db", .{}); + try checkpointOver(path); + const after = try tmp.dir.statFile(test_io, "config.db", .{}); + + // The premise of the test, proved rather than assumed: the main file really + // did move, and the log really is gone again. + try testing.expect(after.size != before.size); + try expectAbsent(tmp.dir, "config.db" ++ wal_suffix); + + try testing.expectError(error.WalPending, database.verifyImmutable()); +} + test "a duplicate insert into a UNIQUE column returns error.Constraint" { var db = try openMemory(); defer db.close(); diff --git a/src/storage/migrations.zig b/src/storage/migrations.zig index 1b471b7..3bb7c5f 100644 --- a/src/storage/migrations.zig +++ b/src/storage/migrations.zig @@ -31,6 +31,9 @@ const ddl_v2: [:0]const u8 = \\ALTER TABLE upstreams ADD COLUMN tls_name TEXT NOT NULL DEFAULT ''; ; +/// The schema version this binary expects. A database `readVersion` reports +/// below this needs `nxdns run` to migrate it; above it is `error.SchemaTooNew` +/// and needs a newer nxdns. pub const target_version: u32 = steps[steps.len - 1].version; comptime { @@ -102,9 +105,16 @@ pub fn migrateSteps(database: *db.Db, list: []const Step) Error!u32 { return target; } +/// The schema version stamped in `database`, compared against `target_version`. +/// /// `0` when `schema_version` does not exist yet. Zero rows or more than one row /// is `error.SchemaCorrupt` — the version of a database is never guessed. -fn readVersion(database: *db.Db) Error!u32 { +/// +/// Reads only, so it works on a connection opened `.read_only` or +/// `.immutable`. That is what it is public for: `nxdns check` may not migrate +/// (ruling F-c), and "at version 1, this binary expects 2" tells an operator +/// what to do where a bare SQLite error message does not. +pub fn readVersion(database: *db.Db) Error!u32 { const present = try database.queryInt( "SELECT count(*) FROM sqlite_schema WHERE type='table' AND name='schema_version'", ); @@ -302,6 +312,47 @@ test "a failing step after step 2 rolls back the whole upgrade from version 1" { try testing.expectEqual(@as(u32, 1), try readVersion(&database)); } +test "readVersion reports 0 before a migration and target_version after it" { + var database = try openMigrated(); + defer database.close(); + + try testing.expectEqual(@as(u32, 0), try readVersion(&database)); + _ = try migrate(&database); + try testing.expectEqual(target_version, try readVersion(&database)); +} + +/// `.zig-cache/tmp/` is where `std.testing.tmpDir` puts its directories, and +/// SQLite's VFS resolves filenames against the same working directory +/// (`db.zig`'s immutable tests). +const tmp_prefix = ".zig-cache/tmp/"; +const sub_path_len = @typeInfo(@FieldType(testing.TmpDir, "sub_path")).array.len; + +test "readVersion reads a file database through an immutable open, writing nothing" { + var tmp = testing.tmpDir(.{}); + defer tmp.cleanup(); + var path_buf: [tmp_prefix.len + sub_path_len + 32]u8 = undefined; + const path = try std.fmt.bufPrintZ(&path_buf, "{s}{s}/config.db", .{ tmp_prefix, &tmp.sub_path }); + + // A database an older nxdns left at version 1. `check` must report that, not + // migrate it (ruling F-c). + { + var database = try db.Db.open(path, .{ .mode = .read_write_create }); + defer database.close(); + try db.applyPragmas(&database, .{}); + const first = [_]Step{.{ .version = 1, .sql = config_schema.ddl_v1 }}; + try testing.expectEqual(@as(u32, 1), try migrateSteps(&database, &first)); + } + + var database = try db.Db.open(path, .{ .mode = .{ .immutable = testing.io } }); + defer database.close(); + try testing.expectEqual(@as(u32, 1), try readVersion(&database)); + try testing.expectEqual(@as(u32, 2), target_version); + + // A write through this connection is refused by SQLite, not by convention. + try testing.expectError(error.ReadOnly, database.exec("DELETE FROM schema_version;")); + try testing.expectEqual(@as(u32, 1), try readVersion(&database)); +} + test "delete_order and content_tables name exactly the tables the schema creates" { var database = try openMigrated(); defer database.close(); diff --git a/src/storage/repositories/clients_repo.zig b/src/storage/repositories/clients_repo.zig index d32ebb2..46b0ce3 100644 --- a/src/storage/repositories/clients_repo.zig +++ b/src/storage/repositories/clients_repo.zig @@ -2,8 +2,11 @@ //! //! `listClients` returns only `hand_edited = 1` rows. A client the server //! materialised from live traffic is runtime state, not configuration, and must -//! not appear in an export. `countClients` counts **all** rows, because S5's -//! "has this database ever been configured" predicate needs the true count. +//! not appear in an export. `hand_edited` is the only marker of operator intent +//! in this table, so it also decides what `import.isEmpty` counts: a database +//! carrying nothing but materialised rows has never been configured, and a seed +//! file must still be able to fill it. `countClients` counts **all** rows and is +//! a test helper — it deliberately does not answer that question. //! //! The import path is list / insert / deleteAll / count, plus the two runtime //! calls `upsertSeen` and `pruneStale` that the Phase 7 client tracker owns. @@ -134,7 +137,8 @@ pub fn deleteAllClients(database: *db.Db) db.Error!void { return database.exec("DELETE FROM clients;"); } -/// Counts every row, including the ones `listClients` filters out. +/// Counts every row, including the materialised ones `listClients` filters out. +/// Used by tests; `import.isEmpty` counts operator intent instead. pub fn countClients(database: *db.Db) db.Error!i64 { return database.queryInt("SELECT count(*) FROM clients"); } diff --git a/src/storage/repositories/context.zig b/src/storage/repositories/context.zig index 9aeca72..248a599 100644 --- a/src/storage/repositories/context.zig +++ b/src/storage/repositories/context.zig @@ -10,6 +10,8 @@ const std = @import("std"); +const safe_url = @import("../../safe_url.zig"); + const log = std.log.scoped(.repositories); /// Group name → `groups.id`, or blocklist source URL → `blocklist_sources.id`. @@ -17,6 +19,42 @@ pub const IdMap = std.StringHashMapUnmanaged(i64); const no_ids: IdMap = .empty; +/// The line either lookup writes when the caller's map lacks a name, as a value +/// rather than a format string at each call site. +/// +/// Two reasons, in that order. A blocklist source url is operator-supplied and +/// nothing on the way in stops it carrying a credential in its userinfo, its +/// path or its query, so it reaches the log through `safe_url.redact` and +/// through nothing else. And a `std.log` line is not readable from a unit test +/// under the default test runner, which installs its own `std_options`; the +/// tests below read this value instead of stderr. +const MissingId = union(enum) { + /// A group name, out of the configuration file or a `groups` row. + group: []const u8, + /// A blocklist source url, out of the configuration file or a + /// `blocklist_sources` row. + source: []const u8, + + pub fn format(self: MissingId, w: *std.Io.Writer) std.Io.Writer.Error!void { + switch (self) { + // A group name holds no credential, so nothing is dropped from it. + // It goes through `quoteText` for what that does to any + // operator-supplied string: it delimits a name with a space in it, + // escapes a `'` that would otherwise close the delimiter, and bounds + // a name of any length. The quotes are the type's own, so this + // format string adds none. + .group => |name| try w.print("no group id for {f}", .{safe_url.quoteText(name)}), + // Redaction costs this line the component that told two sources on + // one host apart, and there is no row id to name the source by + // instead: the id the other call sites print is the one this call + // failed to find. The scheme, the host and the port are what is + // left. The rule against writing a secret to a log does not bend for + // a line that would read better with one. + .source => |url| try w.print("no blocklist source id for {f}", .{safe_url.redact(url)}), + } + } +}; + pub const InsertContext = struct { /// Unix epoch seconds, from `std.Io.Clock.real.now(io).toSeconds()`. now: i64 = 0, @@ -29,14 +67,16 @@ pub const InsertContext = struct { /// `NotFound` is a member of `db.Error`, so it needs no wider error set. pub fn groupId(self: InsertContext, name: []const u8) error{NotFound}!i64 { return self.group_ids.get(name) orelse { - log.warn("no group id for '{s}'", .{name}); + log.warn("{f}", .{MissingId{ .group = name }}); return error.NotFound; }; } + /// `error.NotFound` means what it means in `groupId`, for the map keyed by + /// source url. pub fn sourceId(self: InsertContext, url: []const u8) error{NotFound}!i64 { return self.source_ids.get(url) orelse { - log.warn("no blocklist source id for '{s}'", .{url}); + log.warn("{f}", .{MissingId{ .source = url }}); return error.NotFound; }; } @@ -44,6 +84,48 @@ pub const InsertContext = struct { const testing = std.testing; +fn expectLine(expected: []const u8, missing: MissingId) !void { + var buf: [8 * safe_url.max_len]u8 = undefined; + try testing.expectEqualStrings(expected, try std.fmt.bufPrint(&buf, "{f}", .{missing})); +} + +test "a missing source id names where the url points and not what it carries" { + // The three components a blocklist url carries a credential in, on the one + // line that used to print all three: an api key in the query, userinfo, and + // a token in a path segment. + try expectLine( + "no blocklist source id for https://lists.example", + .{ .source = "https://lists.example/hosts.txt?apikey=s3cr3t" }, + ); + try expectLine( + "no blocklist source id for https://lists.example:8443", + .{ .source = "https://user:pa55@lists.example:8443/hosts.txt" }, + ); + try expectLine( + "no blocklist source id for https://lists.example", + .{ .source = "https://lists.example/download/token/hunter2/hosts.txt" }, + ); + // What an operator still gets: the scheme, the host and the port. + try expectLine( + "no blocklist source id for http://10.0.0.2:8080", + .{ .source = "http://10.0.0.2:8080/a/hosts.txt" }, + ); +} + +test "a missing group id quotes, escapes and bounds the name" { + try expectLine("no group id for 'kids'", .{ .group = "kids" }); + // A name is database text as well as file text, so a newline in it would end + // this line and start one of the operator's choosing. + try expectLine( + "no group id for 'ads\\n2026-01-01 ERROR forged'", + .{ .group = "ads\n2026-01-01 ERROR forged" }, + ); + // Nor can a name close the quote around it. + try expectLine("no group id for 'kids\\' --'", .{ .group = "kids' --" }); + const long_name = "n" ** (2 * safe_url.max_len); + try expectLine("no group id for '" ++ long_name[0..safe_url.max_len] ++ "...'", .{ .group = long_name }); +} + test "an InsertContext with no maps reports a missing id rather than trapping" { const ctx: InsertContext = .{}; try testing.expectError(error.NotFound, ctx.groupId("default")); diff --git a/src/tests.zig b/src/tests.zig index 7782593..3f614a8 100644 --- a/src/tests.zig +++ b/src/tests.zig @@ -4,6 +4,7 @@ comptime { _ = @import("main.zig"); _ = @import("app.zig"); _ = @import("version.zig"); + _ = @import("safe_url.zig"); _ = @import("dns/types.zig"); _ = @import("dns/header.zig"); _ = @import("dns/name.zig"); @@ -31,6 +32,7 @@ comptime { _ = @import("storage/db.zig"); _ = @import("config/model.zig"); _ = @import("config/validate.zig"); + _ = @import("config/faults.zig"); _ = @import("storage/config_schema.zig"); _ = @import("storage/migrations.zig"); _ = @import("storage/querylog_schema.zig"); diff --git a/src/upstream/dot_client.zig b/src/upstream/dot_client.zig index 0a101c3..007318e 100644 --- a/src/upstream/dot_client.zig +++ b/src/upstream/dot_client.zig @@ -16,11 +16,59 @@ const net = std.Io.net; const tls = std.crypto.tls; const Certificate = std.crypto.Certificate; +const safe_url = @import("../safe_url.zig"); const transport = @import("transport.zig"); const tls_client = @import("../platform/tls_client.zig"); const log = std.log.scoped(.dot_client); +/// One diagnostic line about one client, as a value rather than a format string +/// repeated at each call site. +/// +/// Two reasons, in that order. The url is redacted in exactly one place, so a +/// line added later cannot print it whole — the defect this type closes was four +/// call sites each formatting `endpoint.url` with `{s}`, missed by three review +/// rounds because each looked like the three beside it. And a `std.log` line is +/// not readable from a unit test under the default test runner, which installs +/// its own `std_options`; the tests below read this value instead of stderr. +const Diagnostic = struct { + endpoint: transport.Endpoint, + detail: Detail, + + const Detail = union(enum) { + /// `resolveAddress` refused the host. + not_an_ip_literal, + connect_failed: anyerror, + handshake_failed: Handshake, + bundle_load_failed: anyerror, + + const Handshake = struct { + verify_name: []const u8, + cause: anyerror, + }; + }; + + pub fn format(self: Diagnostic, w: *std.Io.Writer) std.Io.Writer.Error!void { + try w.print("dot upstream {f}: ", .{safe_url.redactQuoted(self.endpoint.url)}); + switch (self.detail) { + // The redacted url ends in the host and the port, so naming the host + // again would add nothing but an unredacted copy of it. + .not_an_ip_literal => try w.writeAll("host is not an IP literal"), + .connect_failed => |err| try w.print("connect failed: {s}", .{@errorName(err)}), + // `verify_name` is a host name rather than a url, so it carries no + // component redaction could drop. It goes through `quoteText` for + // what that does to any operator-supplied string: it delimits it, + // escapes it and bounds it. + .handshake_failed => |hs| try w.print("TLS handshake as {f} failed: {s} ({t})", .{ + safe_url.quoteText(hs.verify_name), + @errorName(hs.cause), + tls_client.classify(hs.cause), + }), + .bundle_load_failed => |err| try w.print("CA bundle load failed: {s}", .{@errorName(err)}), + } + } +}; + pub const ResolveError = error{ConnectFailed}; /// DoT endpoints take IP literals. Name resolution for upstreams is out of @@ -89,6 +137,10 @@ pub const DotClient = struct { return .{ .ptr = self, .exchangeFn = exchangeFn }; } + fn diagnose(self: *const DotClient, detail: Diagnostic.Detail) Diagnostic { + return .{ .endpoint = self.endpoint, .detail = detail }; + } + fn exchangeFn( ptr: *anyopaque, io: std.Io, @@ -119,20 +171,14 @@ pub const DotClient = struct { if (query.len > transport.max_message_len) return error.BufferTooSmall; const address = resolveAddress(self.endpoint) catch |err| { - log.warn("dot upstream {s}: host \"{s}\" is not an IP literal", .{ - self.endpoint.url, - self.endpoint.host, - }); + log.warn("{f}", .{self.diagnose(.not_an_ip_literal)}); return err; }; try self.ensureBundle(io); var stream = address.connect(io, .{ .mode = .stream }) catch |err| { - log.debug("dot upstream {s}: connect failed: {s}", .{ - self.endpoint.url, - @errorName(err), - }); + log.debug("{f}", .{self.diagnose(.{ .connect_failed = err })}); return mapPhase(err, error.ConnectFailed); }; defer closeStream(io, &stream); @@ -155,12 +201,10 @@ pub const DotClient = struct { .stream_write_buffer = self.buffers.stream_write, }) catch |err| { const cause = concreteHandshake(&tls_stream, err); - log.warn("dot upstream {s}: TLS handshake as \"{s}\" failed: {s} ({t})", .{ - self.endpoint.url, - self.verify_name, - @errorName(cause), - tls_client.classify(cause), - }); + log.warn("{f}", .{self.diagnose(.{ .handshake_failed = .{ + .verify_name = self.verify_name, + .cause = cause, + } })}); return mapPhase(cause, error.TlsFailed); }; defer closeTls(io, &tls_stream); @@ -214,10 +258,7 @@ pub const DotClient = struct { self.bundle.rescan(self.gpa, io, std.Io.Clock.real.now(io)) catch |err| { self.bundle.deinit(self.gpa); self.bundle.* = .empty; - log.warn("dot upstream {s}: CA bundle load failed: {s}", .{ - self.endpoint.url, - @errorName(err), - }); + log.warn("{f}", .{self.diagnose(.{ .bundle_load_failed = err })}); return mapPhase(err, error.TlsFailed); }; } @@ -286,6 +327,102 @@ fn receiveFailure(stream: *tls_client.TlsStream, err: anyerror) transport.Exchan const testing = std.testing; +fn expectDiagnostic(expected: []const u8, url: []const u8, detail: Diagnostic.Detail) !void { + var buf: [8 * safe_url.max_len]u8 = undefined; + const line = try std.fmt.bufPrint(&buf, "{f}", .{Diagnostic{ + .endpoint = .{ .scheme = .dot, .url = url, .host = "host.example", .port = 853, .path = "/" }, + .detail = detail, + }}); + try testing.expectEqualStrings(expected, line); +} + +test "every diagnostic line redacts the url it names the upstream by" { + // The endpoints are built by hand rather than parsed, on purpose. + // `Endpoint.parse` refuses `@`, `?` and `#` in the authority and refuses a + // `.dot` path other than "/", so no url reaching this client through it can + // carry a credential in a component `redact` drops. That is a property of a + // parser one file away, not of this file, and these lines used to print + // whatever `endpoint.url` held. The redaction is what keeps the parser's + // rules from being load-bearing here. + try expectDiagnostic( + "dot upstream 'tls://dns.example': host is not an IP literal", + "tls://user:hunter2@dns.example/abcd12", + .not_an_ip_literal, + ); + try expectDiagnostic( + "dot upstream 'tls://dns.example:8853': connect failed: ConnectionRefused", + "tls://dns.example:8853/abcd12", + .{ .connect_failed = error.ConnectionRefused }, + ); + try expectDiagnostic( + "dot upstream 'tls://dns.example': CA bundle load failed: FileNotFound", + "tls://dns.example/abcd12?apikey=s3cr3t", + .{ .bundle_load_failed = error.FileNotFound }, + ); + try expectDiagnostic( + "dot upstream 'tls://dns.example': TLS handshake as 'one.one.one.one' failed: " ++ + "CertificateHostMismatch (certificate)", + "tls://token@dns.example/abcd12", + .{ .handshake_failed = .{ + .verify_name = "one.one.one.one", + .cause = error.CertificateHostMismatch, + } }, + ); +} + +test "a diagnostic escapes and bounds the operator-supplied text it prints" { + // A control byte in either field would forge a second record on the one + // output `std.log`'s own sink does not reach, and an unbounded host or + // verification name would write an unbounded line. + try expectDiagnostic( + "dot upstream 'tls://dns.example\\n2026-01-01 ERROR forged': host is not an IP literal", + "tls://dns.example\n2026-01-01 ERROR forged", + .not_an_ip_literal, + ); + try expectDiagnostic( + "dot upstream 'tls://dns.example': TLS handshake as 'a\\nb' failed: TlsAlert (handshake)", + "tls://dns.example", + .{ .handshake_failed = .{ .verify_name = "a\nb", .cause = error.TlsAlert } }, + ); + + // Doubling every operator-supplied field writes the same line, and each + // field is built from the three byte costs at once: a printable byte spends + // one character of the budget, `\n` spends two and `\x00` spends four. That + // expansion is why `safe_url.max_len` counts printed characters rather than + // source bytes, so the bound holds against the widest escape rather than in + // spite of it. + const long = "h\n\x00" ** (2 * safe_url.max_len); + const longer = long ** 2; + var short_buf: [16 * safe_url.max_len]u8 = undefined; + var long_buf: [16 * safe_url.max_len]u8 = undefined; + try testing.expectEqualStrings( + try std.fmt.bufPrint(&short_buf, "{f}", .{Diagnostic{ + .endpoint = .{ .scheme = .dot, .url = "tls://" ++ long, .host = long, .port = 853, .path = "/" }, + .detail = .{ .handshake_failed = .{ .verify_name = long, .cause = error.TlsAlert } }, + }}), + try std.fmt.bufPrint(&long_buf, "{f}", .{Diagnostic{ + .endpoint = .{ .scheme = .dot, .url = "tls://" ++ longer, .host = longer, .port = 853, .path = "/" }, + .detail = .{ .handshake_failed = .{ .verify_name = longer, .cause = error.TlsAlert } }, + }}), + ); +} + +test "a diagnostic keeps what a parsed DoT url carries" { + // The limit, pinned so it stays visible: a NextDNS DoT upstream is + // `tls://abcd12.dns.nextdns.io`, whose hostname is the whole account + // identifier. Redaction cannot remove it without leaving no host and an + // unactionable line. See `safe_url.SafeUrl`. + var buf: [8 * safe_url.max_len]u8 = undefined; + const line = try std.fmt.bufPrint(&buf, "{f}", .{Diagnostic{ + .endpoint = try .parse("tls://abcd12.dns.nextdns.io"), + .detail = .not_an_ip_literal, + }}); + try testing.expectEqualStrings( + "dot upstream 'tls://abcd12.dns.nextdns.io': host is not an IP literal", + line, + ); +} + test "resolveAddress accepts IP literals" { const v4 = try resolveAddress(try .parse("tls://1.1.1.1:853")); try testing.expectEqual(@as(u16, 853), v4.ip4.port); diff --git a/src/upstream/pool.zig b/src/upstream/pool.zig index ec863de..64e171e 100644 --- a/src/upstream/pool.zig +++ b/src/upstream/pool.zig @@ -40,10 +40,32 @@ const std = @import("std"); const health = @import("health.zig"); +const safe_url = @import("../safe_url.zig"); const transport = @import("transport.zig"); const log = std.log.scoped(.upstream); +/// The one line `exchange` writes about a failed attempt, as a value. +/// +/// It is a value rather than a format string at the call site for the same +/// reason `dot_client.Diagnostic` is: a `std.log` line is not readable from a +/// unit test under the default test runner, so the test below reads this +/// instead of stderr. An upstream url is operator-supplied and a DoH one carries +/// its credential in the path — `https://dns.nextdns.io/abcd12` is a whole +/// NextDNS account identifier — so it reaches the line through +/// `safe_url.redactQuoted`. Quoted rather than bare because the error name +/// follows it: a redacted authority may still hold a space and a `:`, so +/// unquoted, a url ending `ok failed: Timeout` would report a failure that did +/// not happen. +const AttemptFailure = struct { + endpoint: transport.Endpoint, + err: transport.ExchangeError, + + pub fn format(self: AttemptFailure, w: *std.Io.Writer) std.Io.Writer.Error!void { + try w.print("upstream {f} failed: {t}", .{ safe_url.redactQuoted(self.endpoint.url), self.err }); + } +}; + pub const Entry = struct { endpoint: transport.Endpoint, client: transport.Client, @@ -61,6 +83,11 @@ pub const Entry = struct { /// A copy of one entry's health, taken under the mutex. Feeds /// `GET /api/upstream/health` in Phase 8. pub const Snapshot = struct { + /// Whole, not redacted. `GET /api/upstream/health` returns this to a session + /// that `GET /api/upstreams` already serves the same url to in full, so + /// redacting here would hide nothing from that reader and would make two + /// responses of one API disagree. A consumer reachable without a session has + /// to redact it itself. url: []const u8, enabled: bool, available: bool, @@ -165,7 +192,7 @@ pub const Pool = struct { const response = result catch |err| switch (transport.group(err)) { .peer_fault => { - log.debug("upstream {s} failed: {t}", .{ entry.endpoint.url, err }); + log.debug("{f}", .{AttemptFailure{ .endpoint = entry.endpoint, .err = err }}); self.recordFailure(io, entry, completed_at, err); last_fault = err; continue; @@ -385,6 +412,38 @@ const test_cfg: health.Config = .{ const test_timeout: std.Io.Clock.Duration = .{ .raw = .fromSeconds(10), .clock = .awake }; +fn expectFailureLine(expected: []const u8, url: []const u8, err: transport.ExchangeError) !void { + var buf: [8 * safe_url.max_len]u8 = undefined; + const line = try std.fmt.bufPrint(&buf, "{f}", .{AttemptFailure{ + .endpoint = try .parse(url), + .err = err, + }}); + try testing.expectEqualStrings(expected, line); +} + +test "the failed-attempt line names an upstream by a url carrying no credential" { + // A NextDNS DoH upstream puts the whole account identifier in the path, and + // this line ran at `debug` on every peer fault, so a debug-level operator + // persisted it to the journal once per failure. + try expectFailureLine( + "upstream 'https://dns.nextdns.io' failed: Timeout", + "https://dns.nextdns.io/abcd12", + error.Timeout, + ); + try expectFailureLine( + "upstream 'https://cdn.example:8443' failed: TlsFailed", + "https://cdn.example:8443/d/hunter2/dns-query", + error.TlsFailed, + ); + // What it still says, because an operator reading a failover has to know + // which upstream failed: the scheme, the host and the port. + try expectFailureLine( + "upstream 'tls://9.9.9.9:853' failed: ConnectFailed", + "tls://9.9.9.9:853", + error.ConnectFailed, + ); +} + test "Pool satisfies the Client interface" { var threaded: std.Io.Threaded = .init(testing.allocator, .{}); defer threaded.deinit(); diff --git a/src/web/handlers/blocklists.zig b/src/web/handlers/blocklists.zig index 9359e7e..67c07d0 100644 --- a/src/web/handlers/blocklists.zig +++ b/src/web/handlers/blocklists.zig @@ -9,6 +9,10 @@ //! refresh downloads and compiles before the response is written: 202 is //! "accepted and done as far as this connection is concerned", and the status //! table in the body is what tells the operator which sources actually landed. +//! +//! `DELETE /api/blocklists/{id}` removes the row, reloads, and then sweeps the +//! compiled files that row named, so `/blocklists/` follows the table +//! the operator is looking at rather than the scheduler's next pass. const std = @import("std"); const Allocator = std.mem.Allocator; @@ -130,7 +134,27 @@ pub fn applyDelete(state: *server.WebState, io: std.Io, id: i64) ?Failure { state.config_lock.unlock(io); written catch |err| return mutations.dbFailure(err, url_conflict); - return mutations.reload(state, io); + const failure = mutations.reload(state, io); + pruneFiles(state, io); + return failure; +} + +/// Removes the compiled files the deleted row leaves behind. +/// +/// This is the moment an orphan is made during normal operation, and the only +/// other sweep is the scheduler's — up to `blocklist_update.interval_hours` +/// away. Without this call a deleted list keeps its megabytes on disk for a day. +/// +/// After the reload and never part of the response: the row is gone and the +/// snapshot has stopped enforcing the list, so bytes still on disk are not a +/// failed delete. `Manager.pruneOrphans` takes the manager's writer lock, which +/// the reload above has already taken and released — nothing here holds it, and +/// `state.config_lock` was released before either. +fn pruneFiles(state: *server.WebState, io: std.Io) void { + const manager = state.manager orelse return; + manager.pruneOrphans(io) catch |err| { + log.warn("pruning the deleted blocklist's files failed: {s}", .{@errorName(err)}); + }; } /// Refreshes every enabled source, then applies the result (ruling 12). diff --git a/src/web/handlers/mutations.zig b/src/web/handlers/mutations.zig index 5bb2ea6..d989ce9 100644 --- a/src/web/handlers/mutations.zig +++ b/src/web/handlers/mutations.zig @@ -184,8 +184,12 @@ fn rebuildFailed(what: []const u8, cause: []const u8) Failure { const skeleton_group = "default"; const skeleton_upstream: model.UpstreamServer = .{ .url = "https://dns.example/dns-query" }; -/// Runs the shipped validator over `cfg` and returns the first problem's text, +/// Runs the shipped validator over `cfg` and returns the first failure's text, /// or null when the candidate is valid. The text is arena-allocated. +/// +/// Failures only: a warning describes a configuration that is legal, and a row +/// this API is about to store cannot be rejected for one. The blocklist source +/// a POST creates is in no group yet — a warning by design, and never a 400. pub fn firstProblem(arena: Allocator, cfg: model.Config) error{OutOfMemory}!?[]const u8 { var diags: validate.Diagnostics = .init(arena); defer diags.deinit(); @@ -194,8 +198,7 @@ pub fn firstProblem(arena: Allocator, cfg: model.Config) error{OutOfMemory}!?[]c error.OutOfMemory => return error.OutOfMemory, else => {}, }; - if (diags.problems.items.len == 0) return null; - const problem = diags.problems.items[0]; + const problem = diags.firstFailure() orelse return null; return try std.fmt.allocPrint(arena, "{s}: {s}", .{ problem.path, problem.message }); } diff --git a/src/web/metrics.zig b/src/web/metrics.zig index 34a8e70..8a6103f 100644 --- a/src/web/metrics.zig +++ b/src/web/metrics.zig @@ -2,7 +2,9 @@ //! //! Two halves, so that neither needs the other to be testable: `collect` walks //! the live collaborators and copies every number into a `Sample`, and `render` -//! turns a `Sample` into text. Nothing is computed during rendering. +//! turns a `Sample` into text. Nothing is computed during rendering, with one +//! exception: an upstream url is redacted where its label is written rather +//! than where it is copied. `writeUrlLabel` carries the reasoning. //! //! Three rules the collection half obeys: //! @@ -32,6 +34,7 @@ const logging = @import("../platform/logging.zig"); const pool_mod = @import("../upstream/pool.zig"); const rate_limiter = @import("../server/rate_limiter.zig"); const retention_mod = @import("../storage/retention.zig"); +const safe_url = @import("../safe_url.zig"); const server = @import("server.zig"); /// The exposition format version, as the 0.0.4 specification writes it. @@ -96,6 +99,9 @@ pub const DohListenerSample = struct { /// One upstream, with every string owned by the caller's arena. pub const UpstreamSample = struct { + /// The configured url, whole. It reaches the exposition only through + /// `writeUrlLabel`, which redacts it; a reader of this field is reading a + /// credential. url: []const u8, enabled: bool, available: bool, @@ -377,16 +383,19 @@ fn endpointValue( fn renderUpstreams(w: *std.Io.Writer, list: []const UpstreamSample) std.Io.Writer.Error!void { try labeledHead(w, "nxdns_upstream_up", "1 while an upstream is enabled and healthy.", "gauge"); - for (list) |entry| try labeledValue(w, "nxdns_upstream_up", entry.url, @intFromBool(entry.available)); + for (list, 0..) |entry, i| { + try labeledValue(w, "nxdns_upstream_up", i, entry.url, @intFromBool(entry.available)); + } try labeledHead(w, "nxdns_upstream_enabled", "1 while an upstream is enabled by configuration.", "gauge"); - for (list) |entry| try labeledValue(w, "nxdns_upstream_enabled", entry.url, @intFromBool(entry.enabled)); + for (list, 0..) |entry, i| { + try labeledValue(w, "nxdns_upstream_enabled", i, entry.url, @intFromBool(entry.enabled)); + } try labeledHead(w, "nxdns_upstream_success_rate", "Share of recent exchanges that succeeded.", "gauge"); - for (list) |entry| { - try w.writeAll("nxdns_upstream_success_rate{url=\""); - try writeLabelValue(w, entry.url); - try w.print("\"}} {d:.4}\n", .{entry.success_rate}); + for (list, 0..) |entry, i| { + try writeUpstreamLabels(w, "nxdns_upstream_success_rate", i, entry.url); + try w.print(" {d:.4}\n", .{entry.success_rate}); } try labeledHead( @@ -395,15 +404,19 @@ fn renderUpstreams(w: *std.Io.Writer, list: []const UpstreamSample) std.Io.Write "Failures since an upstream last answered.", "gauge", ); - for (list) |entry| { - try labeledValue(w, "nxdns_upstream_consecutive_failures", entry.url, entry.consecutive_failures); + for (list, 0..) |entry, i| { + try labeledValue(w, "nxdns_upstream_consecutive_failures", i, entry.url, entry.consecutive_failures); } try labeledHead(w, "nxdns_upstream_successes_total", "Exchanges an upstream answered.", "counter"); - for (list) |entry| try labeledValue(w, "nxdns_upstream_successes_total", entry.url, entry.total_successes); + for (list, 0..) |entry, i| { + try labeledValue(w, "nxdns_upstream_successes_total", i, entry.url, entry.total_successes); + } try labeledHead(w, "nxdns_upstream_failures_total", "Exchanges an upstream failed.", "counter"); - for (list) |entry| try labeledValue(w, "nxdns_upstream_failures_total", entry.url, entry.total_failures); + for (list, 0..) |entry, i| { + try labeledValue(w, "nxdns_upstream_failures_total", i, entry.url, entry.total_failures); + } } /// Every field of a plain counter struct, under one prefix. @@ -438,12 +451,93 @@ fn labeledHead( fn labeledValue( w: *std.Io.Writer, name: []const u8, + index: usize, url: []const u8, value: u64, ) std.Io.Writer.Error!void { - try w.print("{s}{{url=\"", .{name}); - try writeLabelValue(w, url); - try w.print("\"}} {d}\n", .{value}); + try writeUpstreamLabels(w, name, index, url); + try w.print(" {d}\n", .{value}); +} + +/// The label set every upstream family shares, up to and including the closing +/// brace. One definition, because six families have to agree on it exactly: +/// Prometheus identifies a series by its name and its whole label set, so a +/// family that labelled its samples differently would be a different series. +/// +/// `index` is the upstream's position in the pool, in the priority order `Pool` +/// sorts on. It is here because the url alone stopped identifying a series once +/// it was redacted: two upstreams on one host — the shape a NextDNS account with +/// two profiles takes — both print `https://dns.nextdns.io`, and two samples of +/// one name with one label set is a duplicate series a scrape must not contain. +/// The position is read from the rendered slice rather than carried in +/// `UpstreamSample`, so no caller can build two samples that claim one index. +/// +/// What the index is not: a durable key, and the difference is an operator's to +/// know. `Pool.Snapshot` carries no row id — threading one out of the repository +/// through the pool to reach here is a larger change than the defect warrants — +/// so the position is all there is. Removing `upstreams[0]` renumbers every +/// upstream after it, and one upstream's history then continues under the label +/// its neighbour used to carry. +/// +/// What bounds that: the index is only load-bearing when two upstreams share an +/// origin, which is the case it was added for. Where origins differ, `url` +/// carries the identity on its own and a reorder moves nothing that a query +/// grouping on `url` can see. So group on `url`, and read `index` as the +/// disambiguator between upstreams that group would otherwise merge. +fn writeUpstreamLabels( + w: *std.Io.Writer, + name: []const u8, + index: usize, + url: []const u8, +) std.Io.Writer.Error!void { + try w.print("{s}{{index=\"{d}\",url=\"", .{ name, index }); + try writeUrlLabel(w, url); + try w.writeAll("\"}"); +} + +/// The one place an upstream url becomes exposition text. +/// +/// `/metrics` is `.auth = .open` in `web/routes.zig` and `web.bind` defaults to +/// `0.0.0.0`, so a url in a label is readable by anything on the LAN without a +/// session, and a Prometheus that scrapes it keeps that string for as long as it +/// keeps the series. A NextDNS DoH upstream is `https://dns.nextdns.io/abcd12`, +/// where the path segment is the whole account identifier, so the label prints +/// what `safe_url.redact` leaves: the scheme, the host and the port. +/// +/// The redaction sits here rather than in `collect` because this is where the +/// open endpoint writes the value. A `Sample` built anywhere else renders +/// through this function too, so the guarantee cannot be one caller away. +/// `UpstreamSample.url` stays whole for the same reason it is safe to: nothing +/// but this function reads it, and the session-authenticated +/// `GET /api/upstream/health` reports the same pool with the same urls whole. +/// +/// **`redact` output is not safe to interpolate into a label value, and this +/// function is the reason it never has to be.** Do not delete the second layer +/// on the grounds that the first one escapes. +/// +/// What `redact` gives: it escapes every control character and bounds its +/// output, so it emits neither a raw newline nor a lone backslash. What it does +/// not give: it leaves `"` alone. A `"` is legal in the text `redact` keeps, and +/// it is the one character that ends a label value — so a host holding one +/// closes the label and lets the rest of the string write label pairs of its +/// own. That is a defect of this call site, not of `redact`: a `"` needs no +/// escape in a log line, which is what `redact` was written for. +/// +/// So `writeLabelValue` runs over the redacted text rather than instead of it, +/// and the order is the whole point. It also doubles a backslash `redact` wrote, +/// which is what keeps `\n` in a host from reading as a newline to a parser: the +/// four bytes `\x1b` arrive at a scrape as `\\x1b`. +/// +/// A label value's escapes are not a shell's. `quoteText`, which the log lines +/// use, answers a different question — its delimiter is `'` and it writes its +/// own quotes — and it is not the tool here. +fn writeUrlLabel(w: *std.Io.Writer, url: []const u8) std.Io.Writer.Error!void { + // `SafeUrl.format` prints at most `max_len` characters, plus the `...` that + // marks a truncation. The buffer is that bound, so the write cannot fail. + var buf: [safe_url.max_len + 3]u8 = undefined; + var redacted: std.Io.Writer = .fixed(&buf); + try redacted.print("{f}", .{safe_url.redact(url)}); + try writeLabelValue(w, redacted.buffered()); } /// The three characters the exposition format reserves inside a label value. @@ -552,22 +646,24 @@ test "a full sample renders the whole exposition, byte for byte" { try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_blocklist_generation 4\n")); try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_disk_free_bytes 1000\n")); try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_disk_sample_failures_total 1\n")); + // The label set carries the redaction, so the path of the configured url is + // already gone from the golden text. try testing.expect(std.mem.containsAtLeast( u8, text, 1, - "nxdns_upstream_up{url=\"https://dns.example/dns-query\"} 1\n", + "nxdns_upstream_up{index=\"0\",url=\"https://dns.example\"} 1\n", )); try testing.expect(std.mem.containsAtLeast( u8, text, 1, - "nxdns_upstream_success_rate{url=\"https://dns.example/dns-query\"} 0.9000\n", + "nxdns_upstream_success_rate{index=\"0\",url=\"https://dns.example\"} 0.9000\n", )); try testing.expect(std.mem.endsWith( u8, text, - "nxdns_upstream_failures_total{url=\"https://dns.example/dns-query\"} 1\n", + "nxdns_upstream_failures_total{index=\"0\",url=\"https://dns.example\"} 1\n", )); } @@ -671,15 +767,281 @@ test "cert reload counters render per endpoint, only for the wired stores" { try testing.expect(std.mem.containsAtLeast(u8, both, 1, "nxdns_cert_reload_failures_total{endpoint=\"dot\"} 0\n")); } +/// Reads a sample line's label set the way a scrape does and returns the number +/// of label pairs, or null if the line is not one a parser accepts. +/// +/// A checker rather than a golden string, because the input that exercises it is +/// a url no parser accepts, and `safe_url.redact` is entitled to change what it +/// prints for one of those. What may not change is the property: an +/// operator-supplied byte must not close a label value early, end the line, or +/// leave an escape sequence behind that means something else to a parser. Only +/// the format's three escapes are accepted for that reason — a `\x1b` `redact` +/// wrote reaches here as `\\x1b`, whose backslash is escaped and whose `x1b` is +/// three ordinary characters. +fn labelPairs(line: []const u8) ?usize { + var i = (std.mem.indexOfScalar(u8, line, '{') orelse return null) + 1; + var pairs: usize = 0; + while (true) { + const eq = std.mem.indexOfScalarPos(u8, line, i, '=') orelse return null; + if (eq == i) return null; + for (line[i..eq]) |c| if (!std.ascii.isAlphanumeric(c) and c != '_') return null; + if (eq + 1 >= line.len or line[eq + 1] != '"') return null; + + i = eq + 2; + while (true) { + if (i >= line.len) return null; + if (line[i] == '"') break; + if (line[i] != '\\') { + i += 1; + continue; + } + if (i + 1 >= line.len) return null; + switch (line[i + 1]) { + '\\', '"', 'n' => i += 2, + else => return null, + } + } + + pairs += 1; + i += 1; + if (i >= line.len) return null; + if (line[i] == '}') return pairs; + if (line[i] != ',') return null; + i += 1; + } +} + +/// `name{labels}` — what Prometheus identifies a series by. Null for a sample +/// that carries no label set. +fn seriesKey(line: []const u8) ?[]const u8 { + const close = std.mem.lastIndexOfScalar(u8, line, '}') orelse return null; + return line[0 .. close + 1]; +} + test "a label value escapes the characters the format reserves" { + // Every shape an operator-supplied url can take that reaches the label with + // a character the format reserves. The assertion is the property, not the + // text: these are urls no parser accepts, and `safe_url.redact` may change + // what it prints for one of them without changing what this test protects. + const hostile = [_][]const u8{ + "https://a\"b/dns-query", + "https://a\nb/dns-query", + "https://a\\b/dns-query", + "https://a\x1bb/dns-query", + "https://user:pa55@h\"ost/dns-query", + "https://\"}{=,\"/dns-query", + // The shape `safe_url.redact` is being hardened against in this same + // wave: a `?` before the last `@`. What it prints is that fix's to + // decide; that the label holds it safely is this one's. + "https://lists.example?token=prefix@hunter2", + }; + + for (hostile) |url| { + const upstream_list = [_]UpstreamSample{.{ + .url = url, + .enabled = true, + .available = false, + .consecutive_failures = 2, + .total_successes = 0, + .total_failures = 2, + .success_rate = 0, + }}; + const text = try renderToString(testing.allocator, .{ .upstreams = &upstream_list }); + defer testing.allocator.free(text); + + // The url wrote no line of its own, and lost none: every line is a + // comment or a sample, and the six families contribute six samples. + var samples: usize = 0; + var lines = std.mem.splitScalar(u8, text, '\n'); + while (lines.next()) |line| { + if (line.len == 0 or std.mem.startsWith(u8, line, "# ")) continue; + try testing.expect(std.mem.startsWith(u8, line, "nxdns_")); + if (!std.mem.startsWith(u8, line, "nxdns_upstream_")) continue; + samples += 1; + // Both labels are there, and both values close where they opened. + try testing.expectEqual(@as(?usize, 2), labelPairs(line)); + } + try testing.expectEqual(@as(usize, 6), samples); + } +} + +test "two upstreams on one host stay two series" { + // Redaction costs the url the job of telling two upstreams apart: a NextDNS + // account with two profiles is two urls on one host, and both print + // `https://dns.nextdns.io`. Two samples of one name with one label set is a + // duplicate series, which is a broken scrape rather than a hidden one. + const upstream_list = [_]UpstreamSample{ + .{ + .url = "https://dns.nextdns.io/abcd12", + .enabled = true, + .available = true, + .consecutive_failures = 0, + .total_successes = 5, + .total_failures = 0, + .success_rate = 1, + }, + .{ + .url = "https://dns.nextdns.io/efgh34", + .enabled = true, + .available = false, + .consecutive_failures = 3, + .total_successes = 9, + .total_failures = 3, + .success_rate = 0.75, + }, + }; + const text = try renderToString(testing.allocator, .{ .upstreams = &upstream_list }); + defer testing.allocator.free(text); + + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_up{index=\"0\",url=\"https://dns.nextdns.io\"} 1\n", + )); + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_up{index=\"1\",url=\"https://dns.nextdns.io\"} 0\n", + )); + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_successes_total{index=\"1\",url=\"https://dns.nextdns.io\"} 9\n", + )); + + // Distinguishable, not merely present. `nxdns_upstream_up` is the family + // this matters most in: one of these two upstreams is down and the other is + // up, and a reader has to be able to see which. Under one shared label set + // the two samples say 1 and 0 of the same series, so a scrape either reports + // whichever it read last or rejects the pair — and the down upstream is + // invisible either way, on the endpoint an operator watches to find out. + var up_keys: [2][]const u8 = undefined; + var up_values: [2][]const u8 = undefined; + var found: usize = 0; + var up_lines = std.mem.splitScalar(u8, text, '\n'); + while (up_lines.next()) |line| { + if (!std.mem.startsWith(u8, line, "nxdns_upstream_up{")) continue; + try testing.expect(found < up_keys.len); + const key = seriesKey(line).?; + up_keys[found] = key; + up_values[found] = line[key.len + 1 ..]; + found += 1; + } + try testing.expectEqual(@as(usize, 2), found); + try testing.expect(!std.mem.eql(u8, up_keys[0], up_keys[1])); + try testing.expectEqualStrings("1", up_values[0]); + try testing.expectEqualStrings("0", up_values[1]); + + // No two samples in the scrape share a series key, whatever the urls were. + var keys: [32][]const u8 = undefined; + var count: usize = 0; + var lines = std.mem.splitScalar(u8, text, '\n'); + while (lines.next()) |line| { + if (line.len == 0 or std.mem.startsWith(u8, line, "# ")) continue; + const key = seriesKey(line) orelse continue; + for (keys[0..count]) |seen| try testing.expect(!std.mem.eql(u8, seen, key)); + keys[count] = key; + count += 1; + } + try testing.expectEqual(@as(usize, 12), count); +} + +test "an upstream url is redacted before it reaches an open endpoint's label" { + // `/metrics` is `.auth = .open`, so every label here is readable without a + // session by anything that can reach the bind address. A NextDNS DoH + // upstream carries the whole account identifier in its path, and a scraper + // keeps a label for as long as it keeps the series. + const upstream_list = [_]UpstreamSample{ + .{ + .url = "https://dns.nextdns.io/abcd12", + .enabled = true, + .available = true, + .consecutive_failures = 0, + .total_successes = 3, + .total_failures = 0, + .success_rate = 1, + }, + .{ + .url = "https://user:hunter2@dns.example:8443/dns-query?apikey=s3cr3t#frag", + .enabled = false, + .available = false, + .consecutive_failures = 4, + .total_successes = 0, + .total_failures = 4, + .success_rate = 0, + }, + }; + const text = try renderToString(testing.allocator, .{ .upstreams = &upstream_list }); + defer testing.allocator.free(text); + + // The four components a credential can live in, none of them exposed. + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "abcd12")); + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "hunter2")); + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "s3cr3t")); + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "frag")); + try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "dns-query")); + + // Every family carries the label, so none of the six may keep the whole url. + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_up{index=\"0\",url=\"https://dns.nextdns.io\"} 1\n", + )); + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_enabled{index=\"0\",url=\"https://dns.nextdns.io\"} 1\n", + )); + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_success_rate{index=\"0\",url=\"https://dns.nextdns.io\"} 1.0000\n", + )); + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_successes_total{index=\"0\",url=\"https://dns.nextdns.io\"} 3\n", + )); + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_failures_total{index=\"0\",url=\"https://dns.nextdns.io\"} 0\n", + )); + + // The scheme, the host and the port stay: an operator reading a scrape has + // to know which upstream a series is about. + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_enabled{index=\"1\",url=\"https://dns.example:8443\"} 0\n", + )); + try testing.expect(std.mem.containsAtLeast( + u8, + text, + 1, + "nxdns_upstream_consecutive_failures{index=\"1\",url=\"https://dns.example:8443\"} 4\n", + )); +} + +test "a url longer than the redaction bound cannot run past it" { + const long_host = "h" ** (4 * safe_url.max_len); const upstream_list = [_]UpstreamSample{.{ - .url = "https://dns.example/a\"b\\c", + .url = "https://" ++ long_host ++ "/dns-query", .enabled = true, - .available = false, - .consecutive_failures = 2, + .available = true, + .consecutive_failures = 0, .total_successes = 0, - .total_failures = 2, - .success_rate = 0, + .total_failures = 0, + .success_rate = 1, }}; const text = try renderToString(testing.allocator, .{ .upstreams = &upstream_list }); defer testing.allocator.free(text); @@ -688,7 +1050,8 @@ test "a label value escapes the characters the format reserves" { u8, text, 1, - "nxdns_upstream_up{url=\"https://dns.example/a\\\"b\\\\c\"} 0\n", + "nxdns_upstream_up{index=\"0\",url=\"" ++ + ("https://" ++ long_host)[0..safe_url.max_len] ++ "...\"} 1\n", )); } diff --git a/src/web/web_integration_test.zig b/src/web/web_integration_test.zig index df3c2f7..b03fe0a 100644 --- a/src/web/web_integration_test.zig +++ b/src/web/web_integration_test.zig @@ -1129,6 +1129,93 @@ test "W10 a rule mutation reloads the snapshot and the change is live" { try bounded(env.io(), default_budget, mutationReloads, .{ env.io(), env }); } +// --------------------------------------------------------------------------- +// deleting a source takes its compiled files with it (m13 ruling F-f) +// --------------------------------------------------------------------------- + +/// The id in a `201 Created` body from `/api/blocklists`. +fn createdId(body: []const u8) !i64 { + const marker = "\"id\":"; + const at = std.mem.indexOf(u8, body, marker) orelse return error.TestNoId; + const rest = body[at + marker.len ..]; + const end = std.mem.indexOfNone(u8, rest, "0123456789") orelse rest.len; + return std.fmt.parseInt(i64, rest[0..end], 10); +} + +fn writeCompiled(io: std.Io, dir: std.Io.Dir, id: i64, body: []const u8) !void { + var buf: [64]u8 = undefined; + try dir.writeFile(io, .{ + .sub_path = try std.fmt.bufPrint(&buf, "{d}.list", .{id}), + .data = body, + }); + try dir.writeFile(io, .{ + .sub_path = try std.fmt.bufPrint(&buf, "{d}.wild", .{id}), + .data = "", + }); +} + +fn accessCompiled(io: std.Io, dir: std.Io.Dir, id: i64) !void { + var buf: [64]u8 = undefined; + return dir.access(io, try std.fmt.bufPrint(&buf, "{d}.list", .{id}), .{}); +} + +fn deleteSweepsCompiledFiles(io: std.Io, env: *Env) anyerror!void { + var conn: Conn = undefined; + try conn.connect(io, env.addr); + defer conn.close(io); + + var body_buf: [4096]u8 = undefined; + try conn.request("POST", "/api/blocklists", null, "{\"url\":\"https://doomed.test/a.txt\",\"name\":\"doomed\"}"); + var response = try conn.receive(&body_buf); + try testing.expectEqual(@as(u16, 201), response.status); + const doomed = try createdId(response.body); + + try conn.request("POST", "/api/blocklists", null, "{\"url\":\"https://kept.test/b.txt\",\"name\":\"kept\"}"); + response = try conn.receive(&body_buf); + try testing.expectEqual(@as(u16, 201), response.status); + const kept = try createdId(response.body); + + // The files a refresh would have produced for each row. Neither row carries + // a checksum, so the reload the delete runs treats both as never fetched + // and reads neither — this case is about the directory, not the snapshot. + _ = try env.tmp.dir.createDirPathStatus(io, "blocklists", .fromMode(0o700)); + var dir = try env.tmp.dir.openDir(io, "blocklists", .{ .iterate = true }); + defer dir.close(io); + try writeCompiled(io, dir, doomed, "doomed.example\n"); + try writeCompiled(io, dir, kept, "kept.example\n"); + + var target_buf: [64]u8 = undefined; + const target = try std.fmt.bufPrint(&target_buf, "/api/blocklists/{d}", .{doomed}); + try conn.request("DELETE", target, null, null); + response = try conn.receive(&body_buf); + try testing.expectEqual(@as(u16, 204), response.status); + + // The row is gone, so its files are orphans; without a sweep on this path + // they would sit here until a restart or the scheduler's next pass. + var name_buf: [64]u8 = undefined; + try testing.expectError(error.FileNotFound, dir.access( + io, + try std.fmt.bufPrint(&name_buf, "{d}.list", .{doomed}), + .{}, + )); + try testing.expectError(error.FileNotFound, dir.access( + io, + try std.fmt.bufPrint(&name_buf, "{d}.wild", .{doomed}), + .{}, + )); + try accessCompiled(io, dir, kept); +} + +test "W10 deleting a blocklist deletes its compiled files and spares the others" { + if (!build_options.integration) return error.SkipZigTest; + + const gpa = testing.allocator; + var env = try Env.create(gpa, .{}); + defer env.destroy(); + + try bounded(env.io(), default_budget, deleteSweepsCompiledFiles, .{ env.io(), env }); +} + // --------------------------------------------------------------------------- // pause via the API changes a real handler decision (ruling 15) // ---------------------------------------------------------------------------