mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 13 Activity

docs: unwrap hand-wrapped prose repo-wide
Gates / frontend (push) Successful in 1m2s
Details
Gates / test (push) Successful in 1m38s
Details
Gates / package (push) Successful in 5m5s
Details
Gates / test-aarch64 (push) Successful in 6m30s
Details
Gates / container (push) Successful in 15s
Details
CI / gates (push) Successful in 13m30s
Details

Browse Source
This commit is contained in:
mokhtar
2026-08-15 16:27:36 +02:00
parent 50b8fd5c61
commit 5b3d1cd65c
48 changed files with 2691 additions and 11699 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/how-to/troubleshoot.md
+49 -159
View File
@@ -1,29 +1,19 @@
# Troubleshoot nxdns
Symptoms an nxdns install actually produces, what to run to identify each one,
and what to change. Every symptom on this page was reproduced on the machine
that wrote it, and every diagnosis command was run there. Two details differ
from a real install and cannot be otherwise on that machine: it has no
installed service, so the log lines were read from a foreground run instead of
`journalctl -u nxdns`, and ports 53 and 8080 were occupied, so DNS and the API
were exercised on unprivileged ports. Fixes that need root are marked.
Symptoms an nxdns install actually produces, what to run to identify each one, and what to change. Every symptom on this page was reproduced on the machine that wrote it, and every diagnosis command was run there. Two details differ from a real install and cannot be otherwise on that machine: it has no installed service, so the log lines were read from a foreground run instead of `journalctl -u nxdns`, and ports 53 and 8080 were occupied, so DNS and the API were exercised on unprivileged ports. Fixes that need root are marked.
The exit codes themselves are listed in
[the CLI reference](../reference/cli.md).
The exit codes themselves are listed in [the CLI reference](../reference/cli.md).
## The service exits with code 2
**Symptom.** The process stops immediately. The last two lines are the error
and a pointer:
**Symptom.** The process stops immediately. The last two lines are the error and a pointer:
```
nxdns run failed: NoUsableUpstreams
run `nxdns check` to see the configuration in full
```
Exit 2 means the configuration is wrong and you can fix it. Every subcommand
uses the same definition, so a file `run` exits 2 on exits 2 from `check` and
`import` too.
Exit 2 means the configuration is wrong and you can fix it. Every subcommand uses the same definition, so a file `run` exits 2 on exits 2 from `check` and `import` too.
**Diagnosis.**
@@ -31,14 +21,11 @@ uses the same definition, so a file `run` exits 2 on exits 2 from `check` and
nxdns check
```
`check` prints every problem it finds, not the first, and names the source it
checked on its first line.
`check` prints every problem it finds, not the first, and names the source it checked on its first line.
**Fixes by cause.**
- `NoUsableUpstreams` — the database has no enabled upstream. On a fresh
install in database mode this is simply an empty database, and the run says
what to do about it on the next line:
- `NoUsableUpstreams` — the database has no enabled upstream. On a fresh install in database mode this is simply an empty database, and the run says what to do about it on the next line:
```
nxdns run failed: NoUsableUpstreams
@@ -46,22 +33,11 @@ checked on its first line.
load one with `nxdns import <file>`, or make a file the source of truth with `nxdns run --config <file>`
```
Write a configuration file and take either exit: `nxdns import <file>` to load
it into the database once, or add `--config <file>` to `ExecStart` to make the
file the configuration from then on.
- `ManagedConfigUnreadable` — the service runs `run --config FILE` and that file
is missing or the process may not read it. The path is in the FAIL line above
the failure. File mode never falls back to the database, on purpose: a
fallback would turn a bad deploy into a silently stale configuration.
- `BadCertificate` — a DoH or DoT listener is enabled and its certificate or
key is unreadable, too large, unparseable, or the key does not belong to the
certificate. `run` names both paths before it exits:
`doh_server: '<cert>' + '<key>': certificate file is not readable`.
Write a configuration file and take either exit: `nxdns import <file>` to load it into the database once, or add `--config <file>` to `ExecStart` to make the file the configuration from then on.
- `ManagedConfigUnreadable` — the service runs `run --config FILE` and that file is missing or the process may not read it. The path is in the FAIL line above the failure. File mode never falls back to the database, on purpose: a fallback would turn a bad deploy into a silently stale configuration.
- `BadCertificate` — a DoH or DoT listener is enabled and its certificate or key is unreadable, too large, unparseable, or the key does not belong to the certificate. `run` names both paths before it exits: `doh_server: '<cert>' + '<key>': certificate file is not readable`.
`check` catches this without starting a listener. It loads both PEM files and
tests the key against the certificate through the same code `run` uses, so it
fails on exactly what `run` would fail on. Reproduced here with a self-signed
pair and the key from a second, unrelated pair:
`check` catches this without starting a listener. It loads both PEM files and tests the key against the certificate through the same code `run` uses, so it fails on exactly what `run` would fail on. Reproduced here with a self-signed pair and the key from a second, unrelated pair:
```
$ nxdns check --config config.zon
@@ -77,24 +53,15 @@ checked on its first line.
nxdns run failed: BadCertificate # exit 2
```
The `warning(tls_server)` line comes from mbedTLS on stderr and can appear
before the `checking` line, which is on stdout. A cert file containing
`not a certificate` fails the same way, with
`FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`.
An unreadable file reads
`FAIL doh_server.cert_path: '<path>': certificate file is not readable`.
The `warning(tls_server)` line comes from mbedTLS on stderr and can appear before the `checking` line, which is on stdout. A cert file containing `not a certificate` fails the same way, with `FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed`. An unreadable file reads `FAIL doh_server.cert_path: '<path>': certificate file is not readable`.
Fix the path, the ownership, or the pair; see
[Enable DoH and DoT](enable-doh-and-dot.md).
- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a
configuration, so this only reaches a database that was edited by hand.
- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of
that family.
Fix the path, the ownership, or the pair; see [Enable DoH and DoT](enable-doh-and-dot.md).
- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a configuration, so this only reaches a database that was edited by hand.
- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of that family.
## A configuration file you just wrote is rejected
**Symptom.** `nxdns run --config`, `nxdns check --config` or `nxdns import`
prints the validation problem and stops with exit 2:
**Symptom.** `nxdns run --config`, `nxdns check --config` or `nxdns import` prints the validation problem and stops with exit 2:
```
FAIL groups: no group named 'default'; every unknown client is assigned to it
@@ -118,53 +85,32 @@ nxdns run failed: NoUpstreams
run `nxdns check` to see the configuration in full
```
`NoUpstreams` from a file is not the same fault as `NoUsableUpstreams` above:
the first is a file `run` refused, the second is a database `run` accepted and
found empty. Both are exit 2.
`NoUpstreams` from a file is not the same fault as `NoUsableUpstreams` above: the first is a file `run` refused, the second is a database `run` accepted and found empty. Both are exit 2.
**Diagnosis.** Run the same file through `check`, which reports the same
problems and exits 2:
**Diagnosis.** Run the same file through `check`, which reports the same problems and exits 2:
```sh
nxdns check --config /etc/nxdns/config.zon
```
**Fix.** Correct the file the diagnostics name and start again. Nothing was
applied — a file-mode reconcile happens in one transaction that rolls back, and
a failed `import` leaves the database untouched. The exit code does not depend
on which command read the file: all three of these files were run through `run`,
`check` and `import` here, and every one of the nine combinations exited 2 with
the same diagnostic.
**Fix.** Correct the file the diagnostics name and start again. Nothing was applied — a file-mode reconcile happens in one transaction that rolls back, and a failed `import` leaves the database untouched. The exit code does not depend on which command read the file: all three of these files were run through `run`, `check` and `import` here, and every one of the nine combinations exited 2 with the same diagnostic.
Under the shipped systemd unit an exit 2 stops the service rather than
restarting it (`RestartPreventExitStatus=2 64`), so the journal holds the
diagnostics instead of drowning them in a restart loop. `systemctl start nxdns`
once the file is fixed.
Under the shipped systemd unit an exit 2 stops the service rather than restarting it (`RestartPreventExitStatus=2 64`), so the journal holds the diagnostics instead of drowning them in a restart loop. `systemctl start nxdns` once the file is fixed.
Make `nxdns check --config <file>` the precondition in whatever pushes the file.
In file mode every boot reads it, so an unvalidated bad push does not fail at
deploy time — it fails at the next restart, which may be a power cut at 3am.
Make `nxdns check --config <file>` the precondition in whatever pushes the file. In file mode every boot reads it, so an unvalidated bad push does not fail at deploy time — it fails at the next restart, which may be a power cut at 3am.
## `nxdns check` fails on a server that is running fine
**Symptom.** The service is up and answering, but `nxdns check` on the same
machine exits 2 with one long line about a write-ahead log:
**Symptom.** The service is up and answering, but `nxdns check` on the same machine exits 2 with one long line about a write-ahead log:
```
checking database /var/lib/nxdns/config.db
FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.
```
Nothing is damaged. `check` opens `config.db` immutable so that it can never
write to it, and an immutable open ignores the write-ahead log. When that log
holds bytes, the newest settings are in it and the main file holds older ones,
so `check` refuses rather than grade stale values.
Nothing is damaged. `check` opens `config.db` immutable so that it can never write to it, and an immutable open ignores the write-ahead log. When that log holds bytes, the newest settings are in it and the main file holds older ones, so `check` refuses rather than grade stale values.
The log holds bytes after a configuration write that has not been checkpointed
yet, which on a running server means someone changed something through the web
interface or the API. A server that has only been answering queries has an empty
`config.db-wal` and `check` reads it normally — so this line comes and goes, and
its absence is not proof that nothing is running.
The log holds bytes after a configuration write that has not been checkpointed yet, which on a running server means someone changed something through the web interface or the API. A server that has only been answering queries has an empty `config.db-wal` and `check` reads it normally — so this line comes and goes, and its absence is not proof that nothing is running.
**Fix.** Check the exported configuration instead of the live file:
@@ -173,9 +119,7 @@ nxdns export --data-dir /var/lib/nxdns --out /tmp/current.zon
nxdns check --config /tmp/current.zon
```
`export` opens the database read/write and does see the log, so it renders the
settings that are actually in force. Stopping the service and checking again
works too: a clean shutdown checkpoints the log away.
`export` opens the database read/write and does see the log, so it renders the settings that are actually in force. Stopping the service and checking again works too: a clean shutdown checkpoints the log away.
> Reproduced here on a scratch data directory rather than `/var/lib/nxdns` —
> that path is the only substitution in the output above. nxdns was started on
@@ -194,8 +138,7 @@ cannot bind udp [::1]:53: AddressInUse
nxdns run failed: AddressInUse
```
A bind conflict is a runtime failure, not a configuration fault, so this is
exit 1 and `nxdns check` will not find it.
A bind conflict is a runtime failure, not a configuration fault, so this is exit 1 and `nxdns check` will not find it.
**Diagnosis.**
@@ -205,11 +148,9 @@ ss -lntp 'sport = :53'
systemctl is-active systemd-resolved
```
On most systemd distributions the holder is `systemd-resolved`, which runs a
stub listener on `127.0.0.53:53` and on some setups binds `0.0.0.0:53`.
On most systemd distributions the holder is `systemd-resolved`, which runs a stub listener on `127.0.0.53:53` and on some setups binds `0.0.0.0:53`.
**Fix.** Turn off the stub listener and keep resolved for the host's own
lookups:
**Fix.** Turn off the stub listener and keep resolved for the host's own lookups:
```sh
mkdir -p /etc/systemd/resolved.conf.d
@@ -217,22 +158,18 @@ printf '[Resolve]\nDNSStubListener=no\n' > /etc/systemd/resolved.conf.d/nxdns.co
systemctl restart systemd-resolved
```
If `/etc/resolv.conf` is a symlink to `/run/systemd/resolve/stub-resolv.conf`,
repoint it at `/run/systemd/resolve/resolv.conf` so the host still resolves.
If `/etc/resolv.conf` is a symlink to `/run/systemd/resolve/stub-resolv.conf`, repoint it at `/run/systemd/resolve/resolv.conf` so the host still resolves.
> Not verified on this host: this needs root, and `systemd-resolved` is
> inactive here with port 53 free, so the conflict could not be reproduced
> against it. The bind failure itself was reproduced by starting a second nxdns
> on a port the first already held, which is the same error path.
Do not fix this by pointing the host's `/etc/resolv.conf` at nxdns when that
host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a
startup cycle, not a fix.
Do not fix this by pointing the host's `/etc/resolv.conf` at nxdns when that host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a startup cycle, not a fix.
## The container restarts in a loop
**Symptom.** `docker compose ps` shows the container restarting, and the log is
the same failure repeated. Docker has no start limit, so this goes on forever.
**Symptom.** `docker compose ps` shows the container restarting, and the log is the same failure repeated. Docker has no start limit, so this goes on forever.
```
FAIL /etc/nxdns/config.zon: not readable
@@ -246,10 +183,7 @@ docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}} restarts={{.Restar
stat -c '%a %u:%g %n' deploy/docker/etc-nxdns/config.zon
```
Exit 2 naming the configuration path means the container could not read the
file the shipped `command:` makes its configuration. The container runs as uid
65532 and `/etc/nxdns` is mounted read-only, so a file at mode 0600 owned by
your own uid is unreadable to it and the container cannot repair it.
Exit 2 naming the configuration path means the container could not read the file the shipped `command:` makes its configuration. The container runs as uid 65532 and `/etc/nxdns` is mounted read-only, so a file at mode 0600 owned by your own uid is unreadable to it and the container cannot repair it.
**Fix.** Either make the file world-readable, when it holds no secret:
@@ -264,17 +198,11 @@ chown 65532:65532 deploy/docker/etc-nxdns/config.zon
chmod 0600 deploy/docker/etc-nxdns/config.zon
```
The 0644 path was verified against an earlier revision of this page, including
the recovery: after the `chmod` the container started and answered queries. The
`chown` needs root and was not run here.
The 0644 path was verified against an earlier revision of this page, including the recovery: after the `chmod` the container started and answered queries. The `chown` needs root and was not run here.
`FAIL /etc/nxdns/config.zon: no such file` instead of `not readable` means there
is no configuration file at all. Create `deploy/docker/etc-nxdns/config.zon` and
bring it up again; see [Install with Docker](install-with-docker.md).
`FAIL /etc/nxdns/config.zon: no such file` instead of `not readable` means there is no configuration file at all. Create `deploy/docker/etc-nxdns/config.zon` and bring it up again; see [Install with Docker](install-with-docker.md).
A container that exits 2 with `NoUsableUpstreams` is in database mode — the
`command:` line naming `--config` was removed — on a volume whose database is
still empty. Load one and bring it back up:
A container that exits 2 with `NoUsableUpstreams` is in database mode — the `command:` line naming `--config` was removed — on a volume whose database is still empty. Load one and bring it back up:
```sh
docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/config.zon
@@ -293,9 +221,7 @@ docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/co
{"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"}
```
This is not a fault. The service runs `nxdns run --config`, which makes that
file the configuration, and configuration writes through the API are refused so
the file and the running server cannot drift apart.
This is not a fault. The service runs `nxdns run --config`, which makes that file the configuration, and configuration writes through the API are refused so the file and the running server cannot drift apart.
**Diagnosis.** The start log names the authority:
@@ -315,19 +241,13 @@ nxdns check --config /etc/nxdns/config.zon
systemctl restart nxdns
```
Or, if you want the interface to be how this box is configured, leave file mode:
drop `--config` from `ExecStart` and restart. The database already holds the
last reconciled state, so nothing is lost. See
[Run in file mode](install-with-systemd.md#run-in-file-mode).
Or, if you want the interface to be how this box is configured, leave file mode: drop `--config` from `ExecStart` and restart. The database already holds the last reconciled state, so nothing is lost. See [Run in file mode](install-with-systemd.md#run-in-file-mode).
Pausing blocking, refreshing blocklists and reloading certificates are not
configuration and keep working in file mode. Deleting a client works too, unless
the file names that client's address.
Pausing blocking, refreshing blocklists and reloading certificates are not configuration and keep working in file mode. Deleting a client works too, unless the file names that client's address.
## The container cannot reach its upstreams
**Symptom.** The container starts, but every query fails and `nxdns check`
inside it reports each upstream as unreachable.
**Symptom.** The container starts, but every query fails and `nxdns check` inside it reports each upstream as unreachable.
**Diagnosis.** Look at what the host resolves with:
@@ -335,15 +255,11 @@ inside it reports each upstream as unreachable.
cat /etc/resolv.conf
```
**Fix.** If it points at the nxdns container, repoint it at a real resolver.
The container resolves its upstream DoH and DoT hostnames through the host's
DNS configuration, so pointing that at nxdns makes nxdns depend on itself to
start. LAN clients point at nxdns; the container's own host does not.
**Fix.** If it points at the nxdns container, repoint it at a real resolver. The container resolves its upstream DoH and DoT hostnames through the host's DNS configuration, so pointing that at nxdns makes nxdns depend on itself to start. LAN clients point at nxdns; the container's own host does not.
## The disk is filling up
**Symptom.** Writes stop but DNS keeps answering. The journal shows the
transition:
**Symptom.** Writes stop but DNS keeps answering. The journal shows the transition:
```
warning(disk_monitor): disk state ok -> critical: 33349095424 bytes free on /var/lib/nxdns
@@ -361,21 +277,11 @@ curl -s http://127.0.0.1:8080/api/health
{"status":"degraded","disk":{"state":"critical","free_bytes":33349079040,"db_bytes":180224,"log_bytes":0,"sample_failures":0},"upstreams":{"available":1,"total":1},"queries_dropped":0,"writer_failed":false,"refreshes_gated":1,"snapshot_generation":2}
```
`/metrics` carries the same free, database and log byte gauges as
`nxdns_disk_free_bytes`, `nxdns_disk_db_bytes` and `nxdns_disk_log_bytes`; the
state itself is on `/api/health`, not in the metrics output.
`/metrics` carries the same free, database and log byte gauges as `nxdns_disk_free_bytes`, `nxdns_disk_db_bytes` and `nxdns_disk_log_bytes`; the state itself is on `/api/health`, not in the metrics output.
**What the state means.** The monitor samples free space and database sizes
once a minute. Below `disk.warn_free_mb` it logs the transition. Below
`disk.min_free_mb` it gates every non-essential write: the query logger holds
its batches, the client tracker stops persisting, and blocklist refreshes are
skipped and counted in `refreshes_gated`. Resolution never degrades because the
disk is full — this was verified by setting the thresholds above the free space
on the volume: the state went critical, a refresh was gated, and queries kept
being answered.
**What the state means.** The monitor samples free space and database sizes once a minute. Below `disk.warn_free_mb` it logs the transition. Below `disk.min_free_mb` it gates every non-essential write: the query logger holds its batches, the client tracker stops persisting, and blocklist refreshes are skipped and counted in `refreshes_gated`. Resolution never degrades because the disk is full — this was verified by setting the thresholds above the free space on the volume: the state went critical, a refresh was gated, and queries kept being answered.
**Fix.** Recover space — lower `logging.retention_days`, or stop the service
and delete `querylog.db` — and writes resume on the next sample.
**Fix.** Recover space — lower `logging.retention_days`, or stop the service and delete `querylog.db` — and writes resume on the next sample.
## Blocklists are not filtering
@@ -387,23 +293,13 @@ and delete `querylog.db` — and writes resume on the next sample.
journalctl -u nxdns | grep 'serving on'
```
It ends in either `blocklist generation N` or
`unfiltered (no blocklist snapshot)`.
It ends in either `blocklist generation N` or `unfiltered (no blocklist snapshot)`.
**Fix.** `unfiltered` means no snapshot loaded at all; the download or compile
warning that explains it is earlier in the same start. nxdns serves anyway on
purpose — a household loses more from DNS that refuses to start than from a
window of unfiltered answers.
**Fix.** `unfiltered` means no snapshot loaded at all; the download or compile warning that explains it is earlier in the same start. nxdns serves anyway on purpose — a household loses more from DNS that refuses to start than from a window of unfiltered answers.
A generation number with nothing being blocked is a different problem: the
snapshot loaded but has no sources in it. The line
`blocklist snapshot generation 1: 0 of 0 sources loaded` says exactly that. Add
a source in the admin interface, or a `blocklist_sources` entry to the
configuration file with a `group_sources` link naming a group.
A generation number with nothing being blocked is a different problem: the snapshot loaded but has no sources in it. The line `blocklist snapshot generation 1: 0 of 0 sources loaded` says exactly that. Add a source in the admin interface, or a `blocklist_sources` entry to the configuration file with a `group_sources` link naming a group.
One name resolving while its neighbours are blocked is a third case, and
`/api/lookup` answers it directly: it reports which level of the filtering
ladder decided, and against what.
One name resolving while its neighbours are blocked is a third case, and `/api/lookup` answers it directly: it reports which level of the filtering ladder decided, and against what.
```sh
curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com'
@@ -413,11 +309,7 @@ curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com'
{"domain":"api.ads.tvb.com","group_id":1,"local_records":false,"forward_zone":null,"blocked":false,"reason":"blocklist_exception","matched":"api.ads.tvb.com","source_url":"https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt","safe_search_rewrite":null}
```
`blocklist_exception` means a downloaded list lifted that name with an `@@`
line, and `source_url` names the list that did it. Nothing is broken, and the
list is not overruling you: an exception cancels only what another list blocks.
Your own rule wins over it. Adding an exact block rule for the same name and
asking again reports `rule_block_exact`, `blocked` true and a null `source_url`.
`blocklist_exception` means a downloaded list lifted that name with an `@@` line, and `source_url` names the list that did it. Nothing is broken, and the list is not overruling you: an exception cancels only what another list blocks. Your own rule wins over it. Adding an exact block rule for the same name and asking again reports `rule_block_exact`, `blocked` true and a null `source_url`.
## A database stamped by a newer binary
@@ -428,6 +320,4 @@ warning(migrations): config.db is at schema version 99; this nxdns binary suppor
nxdns run failed: SchemaTooNew
```
**Fix.** There is no downgrade. Import the export you took before upgrading
into a fresh data directory with the older binary; see
[Upgrade nxdns](upgrade.md).
**Fix.** There is no downgrade. Import the export you took before upgrading into a fresh data directory with the older binary; see [Upgrade nxdns](upgrade.md).