mokhtar/nxdns
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

milestone 13 discrepancies: redact credentials from urls in logs, metrics and cli output

Browse Source
This commit is contained in:
mokhtar
2026-08-07 00:45:17 +02:00
parent 1ff727feb8
commit 8c3328562e
39 changed files with 5734 additions and 510 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/how-to/troubleshoot.md
+74 -29
View File
@@ -21,8 +21,9 @@ nxdns run failed: NoUsableUpstreams
run `nxdns check` to see the configuration in full
```
Exit 2 is reserved for a small set of faults `run` raises itself:
`NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit` and `BadCertificate`.
Exit 2 means the configuration is wrong and you can fix it. Every subcommand
uses the same definition, so a file `run` exits 2 on exits 2 from `check` and
`import` too.
**Diagnosis.**
@@ -45,18 +46,18 @@ checked on its first line.
certificate. `run` names both paths before it exits:
`doh_server: '<cert>' + '<key>': certificate file is not readable`.
**`check` does not catch most of this.** It tests only that each file is
readable, and warns when the key is readable beyond its owner; it never opens
the PEM. Parsing and the key/certificate pairing happen when `run` builds the
TLS context, so `check` can print `OK: no problems found` on a configuration
`run` then refuses. Reproduced here with a self-signed pair and the key from a
second, unrelated pair:
`check` catches this without starting a listener. It loads both PEM files and
tests the key against the certificate through the same code `run` uses, so it
fails on exactly what `run` would fail on. Reproduced here with a self-signed
pair and the key from a second, unrelated pair:
```
$ nxdns check --config config.zon
warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
checking configuration file config.zon
OK https://cloudflare-dns.com/dns-query
OK: no problems found # exit 0
FAIL doh_server.key_path: 'mismatched-key.pem': private key does not belong to the certificate
OK upstreams[0] https://cloudflare-dns.com
# exit 2
$ nxdns run --config config.zon --data-dir ./data
warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
@@ -64,59 +65,103 @@ checked on its first line.
nxdns run failed: BadCertificate # exit 2
```
A cert file containing `not a certificate` behaves the same way — `check`
exits 0, `run` exits 2 with `certificate PEM could not be parsed`. So a
successful `check` means the paths and permissions are right, not that the
certificate is usable; the only test of that is starting the service. Fix the
path, the ownership, or the pair; see
The `warning(tls_server)` line comes from mbedTLS on stderr and can appear
before the `checking` line, which is on stdout. A cert file containing
`not a certificate` fails the same way, with
`FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`.
An unreadable file reads
`FAIL doh_server.cert_path: '<path>': certificate file is not readable`.
Fix the path, the ownership, or the pair; see
[Enable DoH and DoT](enable-doh-and-dot.md).
- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a
configuration, so this only reaches a database that was edited by hand.
- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of
that family.
## The service exits with code 1 on a seed file you just wrote
## A seed file you just wrote is rejected
**Symptom.** A first start against an empty database prints the validation
problem and stops, but with exit code 1, not 2:
problem and stops with exit 2:
```
groups: no group named 'default'; every unknown client is assigned to it
FAIL groups: no group named 'default'; every unknown client is assigned to it
nxdns run failed: MissingDefaultGroup
run `nxdns check` to see the configuration in full
```
A syntax error behaves the same way:
```
config: 2:42: error: expected ',' after initializer
FAIL config: 3:16: error: expected ',' after initializer
nxdns run failed: ParseZon
run `nxdns check` to see the configuration in full
```
So does a seed file whose upstream list is empty or all disabled:
```
upstreams: at least one upstream must be enabled
FAIL upstreams: at least one upstream must be enabled
nxdns run failed: NoUpstreams
run `nxdns check` to see the configuration in full
```
`NoUpstreams` from a seed file is not the same fault as `NoUsableUpstreams`
above: the first is a file `run` refused, the second is a database `run`
accepted and found empty. Only the second is exit 2.
accepted and found empty. Both are exit 2.
**Diagnosis.** Run the same file through `check`, which reports it as a
configuration problem and exits 2:
**Diagnosis.** Run the same file through `check`, which reports the same
problems and exits 2:
```sh
nxdns check --config /etc/nxdns/config.zon
```
**Fix.** Correct the file the diagnostics name and start again. The database is
still empty after a failed seed, so the next start re-reads the file. Note that
`nxdns check` and `nxdns import` of the same bad file exit 2 while `nxdns run`
exits 1 — the exit code differs by command, the diagnostics do not. All three
commands were run here against a file missing its `default` group, one with a
syntax error and one with no enabled upstream, and every pair came out that
way.
still empty after a failed seed, so the next start re-reads the file. The exit
code no longer depends on which command read the file: all three of these files
were run through `run`, `check` and `import` here, and every one of the nine
combinations exited 2 with the same diagnostic.
## `nxdns check` fails on a server that is running fine
**Symptom.** The service is up and answering, but `nxdns check` on the same
machine exits 2 with one long line about a write-ahead log:
```
checking database /var/lib/nxdns/config.db
FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
```
Nothing is damaged. `check` opens `config.db` immutable so that it can never
write to it, and an immutable open ignores the write-ahead log. When that log
holds bytes, the newest settings are in it and the main file holds older ones,
so `check` refuses rather than grade stale values.
The log holds bytes after a configuration write that has not been checkpointed
yet, which on a running server means someone changed something through the web
interface or the API. A server that has only been answering queries has an empty
`config.db-wal` and `check` reads it normally — so this line comes and goes, and
its absence is not proof that nothing is running.
**Fix.** Check the exported configuration instead of the live file:
```sh
nxdns export --data-dir /var/lib/nxdns --out /tmp/current.zon
nxdns check --config /tmp/current.zon
```
`export` opens the database read/write and does see the log, so it renders the
settings that are actually in force. Stopping the service and checking again
works too: a clean shutdown checkpoints the log away.
> Reproduced here on a scratch data directory rather than `/var/lib/nxdns` —
> that path is the only substitution in the output above. nxdns was started on
> unprivileged ports; `config.db-wal` was 0 bytes and `check` exited 0; one
> `POST /api/blocklists` took it to 8272 bytes and `check` then printed the line
> above and exited 2; `export` from the same live directory succeeded and its
> output checked clean; and after a clean shutdown `check --data-dir` exited 0
> again.
## Port 53 is already taken