milestone 13: restructure docs to diataxis, tutorial, every command executed
This commit is contained in:
@@ -0,0 +1,295 @@
|
||||
# Troubleshoot nxdns
|
||||
|
||||
Symptoms an nxdns install actually produces, what to run to identify each one,
|
||||
and what to change. Every symptom on this page was reproduced on the machine
|
||||
that wrote it, and every diagnosis command was run there. Two details differ
|
||||
from a real install and cannot be otherwise on that machine: it has no
|
||||
installed service, so the log lines were read from a foreground run instead of
|
||||
`journalctl -u nxdns`, and ports 53 and 8080 were occupied, so DNS and the API
|
||||
were exercised on unprivileged ports. Fixes that need root are marked.
|
||||
|
||||
The exit codes themselves are listed in
|
||||
[the CLI reference](../reference/cli.md).
|
||||
|
||||
## The service exits with code 2
|
||||
|
||||
**Symptom.** The process stops immediately. The last two lines are the error
|
||||
and a pointer:
|
||||
|
||||
```
|
||||
nxdns run failed: NoUsableUpstreams
|
||||
run `nxdns check` to see the configuration in full
|
||||
```
|
||||
|
||||
Exit 2 is reserved for a small set of faults `run` raises itself:
|
||||
`NoUsableUpstreams`, `BadBindAddress`, `BadRateLimit` and `BadCertificate`.
|
||||
|
||||
**Diagnosis.**
|
||||
|
||||
```sh
|
||||
nxdns check
|
||||
```
|
||||
|
||||
`check` prints every problem it finds, not the first, and names the source it
|
||||
checked on its first line.
|
||||
|
||||
**Fixes by cause.**
|
||||
|
||||
- `NoUsableUpstreams` — the database has no enabled upstream. On a fresh
|
||||
install this means the seed file was missing or in the wrong place; the start
|
||||
log says `no configuration file at '/etc/nxdns/config.zon'; using the
|
||||
database as it is`. Write the seed file and start again against the still
|
||||
empty database, or `nxdns import <file> --force`.
|
||||
- `BadCertificate` — a DoH or DoT listener is enabled and its certificate or
|
||||
key is unreadable, too large, unparseable, or the key does not belong to the
|
||||
certificate. `run` names both paths before it exits:
|
||||
`doh_server: '<cert>' + '<key>': certificate file is not readable`.
|
||||
|
||||
**`check` does not catch most of this.** It tests only that each file is
|
||||
readable, and warns when the key is readable beyond its owner; it never opens
|
||||
the PEM. Parsing and the key/certificate pairing happen when `run` builds the
|
||||
TLS context, so `check` can print `OK: no problems found` on a configuration
|
||||
`run` then refuses. Reproduced here with a self-signed pair and the key from a
|
||||
second, unrelated pair:
|
||||
|
||||
```
|
||||
$ nxdns check --config config.zon
|
||||
checking configuration file config.zon
|
||||
OK https://cloudflare-dns.com/dns-query
|
||||
OK: no problems found # exit 0
|
||||
|
||||
$ nxdns run --config config.zon --data-dir ./data
|
||||
warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
|
||||
doh_server: 'cert.pem' + 'mismatched-key.pem': private key does not belong to the certificate
|
||||
nxdns run failed: BadCertificate # exit 2
|
||||
```
|
||||
|
||||
A cert file containing `not a certificate` behaves the same way — `check`
|
||||
exits 0, `run` exits 2 with `certificate PEM could not be parsed`. So a
|
||||
successful `check` means the paths and permissions are right, not that the
|
||||
certificate is usable; the only test of that is starting the service. Fix the
|
||||
path, the ownership, or the pair; see
|
||||
[Enable DoH and DoT](enable-doh-and-dot.md).
|
||||
- `BadRateLimit` — a rate limit or window is zero. `import` refuses such a
|
||||
configuration, so this only reaches a database that was edited by hand.
|
||||
- `BadBindAddress` — `dns.bind_ipv4` or `dns.bind_ipv6` is not an address of
|
||||
that family.
|
||||
|
||||
## The service exits with code 1 on a seed file you just wrote
|
||||
|
||||
**Symptom.** A first start against an empty database prints the validation
|
||||
problem and stops, but with exit code 1, not 2:
|
||||
|
||||
```
|
||||
groups: no group named 'default'; every unknown client is assigned to it
|
||||
nxdns run failed: MissingDefaultGroup
|
||||
```
|
||||
|
||||
A syntax error behaves the same way:
|
||||
|
||||
```
|
||||
config: 2:42: error: expected ',' after initializer
|
||||
nxdns run failed: ParseZon
|
||||
```
|
||||
|
||||
So does a seed file whose upstream list is empty or all disabled:
|
||||
|
||||
```
|
||||
upstreams: at least one upstream must be enabled
|
||||
nxdns run failed: NoUpstreams
|
||||
```
|
||||
|
||||
`NoUpstreams` from a seed file is not the same fault as `NoUsableUpstreams`
|
||||
above: the first is a file `run` refused, the second is a database `run`
|
||||
accepted and found empty. Only the second is exit 2.
|
||||
|
||||
**Diagnosis.** Run the same file through `check`, which reports it as a
|
||||
configuration problem and exits 2:
|
||||
|
||||
```sh
|
||||
nxdns check --config /etc/nxdns/config.zon
|
||||
```
|
||||
|
||||
**Fix.** Correct the file the diagnostics name and start again. The database is
|
||||
still empty after a failed seed, so the next start re-reads the file. Note that
|
||||
`nxdns check` and `nxdns import` of the same bad file exit 2 while `nxdns run`
|
||||
exits 1 — the exit code differs by command, the diagnostics do not. All three
|
||||
commands were run here against a file missing its `default` group, one with a
|
||||
syntax error and one with no enabled upstream, and every pair came out that
|
||||
way.
|
||||
|
||||
## Port 53 is already taken
|
||||
|
||||
**Symptom.** The process exits 1, having named the socket it could not have:
|
||||
|
||||
```
|
||||
cannot bind udp [::1]:53: AddressInUse
|
||||
nxdns run failed: AddressInUse
|
||||
```
|
||||
|
||||
A bind conflict is a runtime failure, not a configuration fault, so this is
|
||||
exit 1 and `nxdns check` will not find it.
|
||||
|
||||
**Diagnosis.**
|
||||
|
||||
```sh
|
||||
ss -lnup 'sport = :53'
|
||||
ss -lntp 'sport = :53'
|
||||
systemctl is-active systemd-resolved
|
||||
```
|
||||
|
||||
On most systemd distributions the holder is `systemd-resolved`, which runs a
|
||||
stub listener on `127.0.0.53:53` and on some setups binds `0.0.0.0:53`.
|
||||
|
||||
**Fix.** Turn off the stub listener and keep resolved for the host's own
|
||||
lookups:
|
||||
|
||||
```sh
|
||||
mkdir -p /etc/systemd/resolved.conf.d
|
||||
printf '[Resolve]\nDNSStubListener=no\n' > /etc/systemd/resolved.conf.d/nxdns.conf
|
||||
systemctl restart systemd-resolved
|
||||
```
|
||||
|
||||
If `/etc/resolv.conf` is a symlink to `/run/systemd/resolve/stub-resolv.conf`,
|
||||
repoint it at `/run/systemd/resolve/resolv.conf` so the host still resolves.
|
||||
|
||||
> Not verified on this host: this needs root, and `systemd-resolved` is
|
||||
> inactive here with port 53 free, so the conflict could not be reproduced
|
||||
> against it. The bind failure itself was reproduced by starting a second nxdns
|
||||
> on a port the first already held, which is the same error path.
|
||||
|
||||
Do not fix this by pointing the host's `/etc/resolv.conf` at nxdns when that
|
||||
host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a
|
||||
startup cycle, not a fix.
|
||||
|
||||
## The container restarts in a loop
|
||||
|
||||
**Symptom.** `docker compose ps` shows the container restarting, and the log is
|
||||
one line repeated:
|
||||
|
||||
```
|
||||
nxdns run failed: AccessDenied
|
||||
```
|
||||
|
||||
**Diagnosis.**
|
||||
|
||||
```sh
|
||||
docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}} restarts={{.RestartCount}}' docker-nxdns-1
|
||||
stat -c '%a %u:%g %n' deploy/docker/etc-nxdns/config.zon
|
||||
```
|
||||
|
||||
Exit 1 with `AccessDenied` means the container could not read the seed file.
|
||||
The container runs as uid 65532 and `/etc/nxdns` is mounted read-only, so a
|
||||
file at mode 0600 owned by your own uid is unreadable to it and the container
|
||||
cannot repair it.
|
||||
|
||||
**Fix.** Either make the file world-readable, when it holds no secret:
|
||||
|
||||
```sh
|
||||
chmod 0644 deploy/docker/etc-nxdns/config.zon
|
||||
```
|
||||
|
||||
or give it to the container's uid:
|
||||
|
||||
```sh
|
||||
chown 65532:65532 deploy/docker/etc-nxdns/config.zon
|
||||
chmod 0600 deploy/docker/etc-nxdns/config.zon
|
||||
```
|
||||
|
||||
The 0644 path was verified here, including the recovery: after the `chmod` the
|
||||
container started and answered queries. The `chown` needs root and was not run
|
||||
here.
|
||||
|
||||
A container that exits 2 instead — `nxdns run failed: NoUsableUpstreams` after
|
||||
`no configuration file at '/etc/nxdns/config.zon'` — has no seed file at all on
|
||||
a fresh volume. Create `deploy/docker/etc-nxdns/config.zon` and bring it up
|
||||
again; see [Install with Docker](install-with-docker.md).
|
||||
|
||||
## The container cannot reach its upstreams
|
||||
|
||||
**Symptom.** The container starts, but every query fails and `nxdns check`
|
||||
inside it reports each upstream as unreachable.
|
||||
|
||||
**Diagnosis.** Look at what the host resolves with:
|
||||
|
||||
```sh
|
||||
cat /etc/resolv.conf
|
||||
```
|
||||
|
||||
**Fix.** If it points at the nxdns container, repoint it at a real resolver.
|
||||
The container resolves its upstream DoH and DoT hostnames through the host's
|
||||
DNS configuration, so pointing that at nxdns makes nxdns depend on itself to
|
||||
start. LAN clients point at nxdns; the container's own host does not.
|
||||
|
||||
## The disk is filling up
|
||||
|
||||
**Symptom.** Writes stop but DNS keeps answering. The journal shows the
|
||||
transition:
|
||||
|
||||
```
|
||||
warning(disk_monitor): disk state ok -> critical: 33349095424 bytes free on /var/lib/nxdns
|
||||
```
|
||||
|
||||
**Diagnosis.**
|
||||
|
||||
```sh
|
||||
curl -s http://127.0.0.1:8080/api/health
|
||||
```
|
||||
|
||||
`/api/health` needs no login and reports the state and what has been gated:
|
||||
|
||||
```json
|
||||
{"status":"degraded","disk":{"state":"critical","free_bytes":33349079040,"db_bytes":180224,"log_bytes":0,"sample_failures":0},"upstreams":{"available":1,"total":1},"queries_dropped":0,"writer_failed":false,"refreshes_gated":1,"snapshot_generation":2}
|
||||
```
|
||||
|
||||
`/metrics` carries the same free, database and log byte gauges as
|
||||
`nxdns_disk_free_bytes`, `nxdns_disk_db_bytes` and `nxdns_disk_log_bytes`; the
|
||||
state itself is on `/api/health`, not in the metrics output.
|
||||
|
||||
**What the state means.** The monitor samples free space and database sizes
|
||||
once a minute. Below `disk.warn_free_mb` it logs the transition. Below
|
||||
`disk.min_free_mb` it gates every non-essential write: the query logger holds
|
||||
its batches, the client tracker stops persisting, and blocklist refreshes are
|
||||
skipped and counted in `refreshes_gated`. Resolution never degrades because the
|
||||
disk is full — this was verified by setting the thresholds above the free space
|
||||
on the volume: the state went critical, a refresh was gated, and queries kept
|
||||
being answered.
|
||||
|
||||
**Fix.** Recover space — lower `logging.retention_days`, or stop the service
|
||||
and delete `querylog.db` — and writes resume on the next sample.
|
||||
|
||||
## Blocklists are not filtering
|
||||
|
||||
**Symptom.** Domains that should be blocked resolve normally.
|
||||
|
||||
**Diagnosis.** Read the startup line:
|
||||
|
||||
```sh
|
||||
journalctl -u nxdns | grep 'serving on'
|
||||
```
|
||||
|
||||
It ends in either `blocklist generation N` or
|
||||
`unfiltered (no blocklist snapshot)`.
|
||||
|
||||
**Fix.** `unfiltered` means no snapshot loaded at all; the download or compile
|
||||
warning that explains it is earlier in the same start. nxdns serves anyway on
|
||||
purpose — a household loses more from DNS that refuses to start than from a
|
||||
window of unfiltered answers.
|
||||
|
||||
A generation number with nothing being blocked is a different problem: the
|
||||
snapshot loaded but has no sources in it. The line
|
||||
`blocklist snapshot generation 1: 0 of 0 sources loaded` says exactly that. Add
|
||||
a source in the admin interface, or in the seed file before the first start.
|
||||
|
||||
## A database stamped by a newer binary
|
||||
|
||||
**Symptom.** After putting an older binary back, it will not start:
|
||||
|
||||
```
|
||||
warning(migrations): config.db is at schema version 99; this nxdns binary supports 2
|
||||
nxdns run failed: SchemaTooNew
|
||||
```
|
||||
|
||||
**Fix.** There is no downgrade. Import the export you took before upgrading
|
||||
into a fresh data directory with the older binary; see
|
||||
[Upgrade nxdns](upgrade.md).
|
||||
Reference in New Issue
Block a user