nxdns/docs/how-to/troubleshoot.md

Troubleshoot nxdns

Symptoms an nxdns install actually produces, what to run to identify each one, and what to change. Every symptom on this page was reproduced on the machine that wrote it, and every diagnosis command was run there. Two details differ from a real install and cannot be otherwise on that machine: it has no installed service, so the log lines were read from a foreground run instead of journalctl -u nxdns, and ports 53 and 8080 were occupied, so DNS and the API were exercised on unprivileged ports. Fixes that need root are marked.

The exit codes themselves are listed in the CLI reference.

The service exits with code 2

Symptom. The process stops immediately. The last two lines are the error and a pointer:

nxdns run failed: NoUsableUpstreams
run `nxdns check` to see the configuration in full

Exit 2 means the configuration is wrong and you can fix it. Every subcommand uses the same definition, so a file run exits 2 on exits 2 from check and import too.

Diagnosis.

nxdns check

check prints every problem it finds, not the first, and names the source it checked on its first line.

Fixes by cause.

• NoUsableUpstreams — the database has no enabled upstream. On a fresh install in database mode this is simply an empty database, and the run says what to do about it on the next line:

  nxdns run failed: NoUsableUpstreams
  run `nxdns check` to see the configuration in full
  load one with `nxdns import <file>`, or make a file the source of truth with `nxdns run --config <file>`
  

  Write a configuration file and take either exit: nxdns import <file> to load it into the database once, or add --config <file> to ExecStart to make the file the configuration from then on.

• ManagedConfigUnreadable — the service runs run --config FILE and that file is missing or the process may not read it. The path is in the FAIL line above the failure. File mode never falls back to the database, on purpose: a fallback would turn a bad deploy into a silently stale configuration.

• BadCertificate — a DoH or DoT listener is enabled and its certificate or key is unreadable, too large, unparseable, or the key does not belong to the certificate. run names both paths before it exits: doh_server: '<cert>' + '<key>': certificate file is not readable.

  check catches this without starting a listener. It loads both PEM files and tests the key against the certificate through the same code run uses, so it fails on exactly what run would fail on. Reproduced here with a self-signed pair and the key from a second, unrelated pair:

  $ nxdns check --config config.zon
  warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
  checking configuration file config.zon
  FAIL doh_server.key_path: 'mismatched-key.pem': private key does not belong to the certificate
  OK upstreams[0] https://cloudflare-dns.com
                                                                    # exit 2
  
  $ nxdns run --config config.zon --data-dir ./data
  warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
  doh_server: 'cert.pem' + 'mismatched-key.pem': private key does not belong to the certificate
  nxdns run failed: BadCertificate                                  # exit 2
  

  The warning(tls_server) line comes from mbedTLS on stderr and can appear before the checking line, which is on stdout. A cert file containing not a certificate fails the same way, with FAIL doh_server.cert_path: 'junk.pem': certificate PEM could not be parsed. An unreadable file reads FAIL doh_server.cert_path: '<path>': certificate file is not readable.

  Fix the path, the ownership, or the pair; see Enable DoH and DoT.

• BadRateLimit — a rate limit or window is zero. import refuses such a configuration, so this only reaches a database that was edited by hand.

• BadBindAddress — dns.bind_ipv4 or dns.bind_ipv6 is not an address of that family.

A configuration file you just wrote is rejected

Symptom. nxdns run --config, nxdns check --config or nxdns import prints the validation problem and stops with exit 2:

FAIL groups: no group named 'default'; every unknown client is assigned to it
nxdns run failed: MissingDefaultGroup
run `nxdns check` to see the configuration in full

A syntax error behaves the same way:

FAIL config: 3:16: error: expected ',' after initializer
nxdns run failed: ParseZon
run `nxdns check` to see the configuration in full

So does a file whose upstream list is empty or all disabled:

FAIL upstreams: at least one upstream must be enabled
nxdns run failed: NoUpstreams
run `nxdns check` to see the configuration in full

NoUpstreams from a file is not the same fault as NoUsableUpstreams above: the first is a file run refused, the second is a database run accepted and found empty. Both are exit 2.

Diagnosis. Run the same file through check, which reports the same problems and exits 2:

nxdns check --config /etc/nxdns/config.zon

Fix. Correct the file the diagnostics name and start again. Nothing was applied — a file-mode reconcile happens in one transaction that rolls back, and a failed import leaves the database untouched. The exit code does not depend on which command read the file: all three of these files were run through run, check and import here, and every one of the nine combinations exited 2 with the same diagnostic.

Under the shipped systemd unit an exit 2 stops the service rather than restarting it (RestartPreventExitStatus=2 64), so the journal holds the diagnostics instead of drowning them in a restart loop. systemctl start nxdns once the file is fixed.

Make nxdns check --config <file> the precondition in whatever pushes the file. In file mode every boot reads it, so an unvalidated bad push does not fail at deploy time — it fails at the next restart, which may be a power cut at 3am.

nxdns check fails on a server that is running fine

Symptom. The service is up and answering, but nxdns check on the same machine exits 2 with one long line about a write-ahead log:

checking database /var/lib/nxdns/config.db
FAIL /var/lib/nxdns/config.db: uncheckpointed changes are waiting in /var/lib/nxdns/config.db-wal, and reading without writing would answer from the older settings in the main file; `nxdns run` applies them. A running nxdns normally holds this log, which is the usual reason to see this line.

Nothing is damaged. check opens config.db immutable so that it can never write to it, and an immutable open ignores the write-ahead log. When that log holds bytes, the newest settings are in it and the main file holds older ones, so check refuses rather than grade stale values.

The log holds bytes after a configuration write that has not been checkpointed yet, which on a running server means someone changed something through the web interface or the API. A server that has only been answering queries has an empty config.db-wal and check reads it normally — so this line comes and goes, and its absence is not proof that nothing is running.

Fix. Check the exported configuration instead of the live file:

nxdns export --data-dir /var/lib/nxdns --out /tmp/current.zon
nxdns check --config /tmp/current.zon

export opens the database read/write and does see the log, so it renders the settings that are actually in force. Stopping the service and checking again works too: a clean shutdown checkpoints the log away.

Reproduced here on a scratch data directory rather than /var/lib/nxdns — that path is the only substitution in the output above. nxdns was started on unprivileged ports; config.db-wal was 0 bytes and check exited 0; one POST /api/blocklists took it to 8272 bytes and check then printed the line above and exited 2; export from the same live directory succeeded and its output checked clean; and after a clean shutdown check --data-dir exited 0 again.

Port 53 is already taken

Symptom. The process exits 1, having named the socket it could not have:

cannot bind udp [::1]:53: AddressInUse
nxdns run failed: AddressInUse

A bind conflict is a runtime failure, not a configuration fault, so this is exit 1 and nxdns check will not find it.

Diagnosis.

ss -lnup 'sport = :53'
ss -lntp 'sport = :53'
systemctl is-active systemd-resolved

On most systemd distributions the holder is systemd-resolved, which runs a stub listener on 127.0.0.53:53 and on some setups binds 0.0.0.0:53.

Fix. Turn off the stub listener and keep resolved for the host's own lookups:

mkdir -p /etc/systemd/resolved.conf.d
printf '[Resolve]\nDNSStubListener=no\n' > /etc/systemd/resolved.conf.d/nxdns.conf
systemctl restart systemd-resolved

If /etc/resolv.conf is a symlink to /run/systemd/resolve/stub-resolv.conf, repoint it at /run/systemd/resolve/resolv.conf so the host still resolves.

Not verified on this host: this needs root, and systemd-resolved is inactive here with port 53 free, so the conflict could not be reproduced against it. The bind failure itself was reproduced by starting a second nxdns on a port the first already held, which is the same error path.

Do not fix this by pointing the host's /etc/resolv.conf at nxdns when that host is where nxdns resolves its own upstream DoH and DoT hostnames. That is a startup cycle, not a fix.

The container restarts in a loop

Symptom. docker compose ps shows the container restarting, and the log is the same failure repeated. Docker has no start limit, so this goes on forever.

FAIL /etc/nxdns/config.zon: not readable
nxdns run failed: ManagedConfigUnreadable

Diagnosis.

docker inspect -f '{{.State.Status}} exit={{.State.ExitCode}} restarts={{.RestartCount}}' docker-nxdns-1
stat -c '%a %u:%g %n' deploy/docker/etc-nxdns/config.zon

Exit 2 naming the configuration path means the container could not read the file the shipped command: makes its configuration. The container runs as uid 65532 and /etc/nxdns is mounted read-only, so a file at mode 0600 owned by your own uid is unreadable to it and the container cannot repair it.

Fix. Either make the file world-readable, when it holds no secret:

chmod 0644 deploy/docker/etc-nxdns/config.zon

or give it to the container's uid:

chown 65532:65532 deploy/docker/etc-nxdns/config.zon
chmod 0600 deploy/docker/etc-nxdns/config.zon

The 0644 path was verified against an earlier revision of this page, including the recovery: after the chmod the container started and answered queries. The chown needs root and was not run here.

FAIL /etc/nxdns/config.zon: no such file instead of not readable means there is no configuration file at all. Create deploy/docker/etc-nxdns/config.zon and bring it up again; see Install with Docker.

A container that exits 2 with NoUsableUpstreams is in database mode — the command: line naming --config was removed — on a volume whose database is still empty. Load one and bring it back up:

docker compose -f deploy/docker/compose.yaml run --rm nxdns import /etc/nxdns/config.zon

Not re-run on this host: staging a release image needs zig build dist, which could not run here while the web bundle was mid-rebuild by other work in the same checkout. The failure text quoted above is what the same binary prints outside a container, which was reproduced here, with the container's paths.

The admin interface refuses an edit with 403

Symptom. Saving anything in the admin interface fails, and the API answers:

{"error":"configuration is managed by /etc/nxdns/config.zon; edit the file and restart"}

This is not a fault. The service runs nxdns run --config, which makes that file the configuration, and configuration writes through the API are refused so the file and the running server cannot drift apart.

Diagnosis. The start log names the authority:

journalctl -u nxdns | grep 'authority:'

info(nxdns): authority: file (/etc/nxdns/config.zon)

Fix. Edit the file, validate it, restart:

$EDITOR /etc/nxdns/config.zon
nxdns check --config /etc/nxdns/config.zon
systemctl restart nxdns

Or, if you want the interface to be how this box is configured, leave file mode: drop --config from ExecStart and restart. The database already holds the last reconciled state, so nothing is lost. See Run in file mode.

Pausing blocking, refreshing blocklists and reloading certificates are not configuration and keep working in file mode. Deleting a client works too, unless the file names that client's address.

The container cannot reach its upstreams

Symptom. The container starts, but every query fails and nxdns check inside it reports each upstream as unreachable.

Diagnosis. Look at what the host resolves with:

cat /etc/resolv.conf

Fix. If it points at the nxdns container, repoint it at a real resolver. The container resolves its upstream DoH and DoT hostnames through the host's DNS configuration, so pointing that at nxdns makes nxdns depend on itself to start. LAN clients point at nxdns; the container's own host does not.

SERVFAIL in bursts, only under concurrent load

Symptom. Single queries answer normally, but a burst of unique names — a page load, a device waking up, a dig loop — returns SERVFAIL for some of them, and the upstream itself is healthy on /api/health.

Diagnosis. Scrape the queue counters:

curl -s http://127.0.0.1:8080/metrics | grep nxdns_upstream_

nxdns_upstream_in_flight against nxdns_upstream_slots is how much of an upstream's concurrency is in use right now, and nxdns_upstream_queued_total counts exchanges that had to wait for a slot (with nxdns_upstream_queued_seconds_total for how long they waited in total). Both queue counters are approximate — they are sampled when a query is admitted, not measured as a queue length. A queued_total climbing with each burst means queries are waiting on the upstream rather than failing at it, and a query that waits past upstream.total_timeout_ms is given up on in the queue and answered SERVFAIL. nxdns_upstream_budget_exhausted_total counts those give-ups pool-wide — queries whose own budget ran out, in the queue or mid-attempt, before any upstream answered. It carries no url label on purpose: running out of budget is a fact about the pool, so the exhausted attempt is never charged to an upstream's health and is never attributed in the query log, although the row can still name the last endpoint whose success or recorded failure preceded it.

nxdns_upstream_reuse_recoveries_total is the other half of the picture: it counts DoT connections that went stale between exchanges and were redialed. A few are normal — a resolver is free to close an idle connection. One per query means the connection is never being reused, and every query is paying a full TLS handshake.

Fix. Raise upstream.total_timeout_ms if the queue drains but drains too slowly for the budget. Otherwise the queue is telling you the upstream is slow: an upstream answering in a few milliseconds does not fill eight concurrent slots at household query rates, so sustained queueing points at the resolver you configured, and a faster one is the fix. Adding a second upstream helps only with saturation, not with latency: failover is strict priority, not load spreading. A query does take the next upstream when the first one's slots are all busy — priority orders the candidates that can be admitted right now — but once every eligible upstream is full it blocks on the highest-priority one, and it still sends one query to one upstream at a time. The per-upstream slot count is compiled, not configured, so there is no knob to widen one upstream either.

The disk is filling up

Symptom. Writes stop but DNS keeps answering. The journal shows the transition:

warning(disk_monitor): disk state ok -> critical: 33349095424 bytes free on /var/lib/nxdns

Diagnosis.

curl -s http://127.0.0.1:8080/api/health

/api/health needs no login. It answers with five condition objects, and status is degraded when any one of them is in a degrading state:

{"status":"degraded","protection":{"state":"active","until":null},"upstreams":{"state":"ok","available":1,"total":1},"query_history":{"state":"recording","dropped_total":0,"last_drop_s":null},"diagnostics":{"state":"recording","active_warnings":1,"active_errors":0},"disk":{"state":"critical","free_bytes":33349079040}}

The disk monitor's warn state is reported here as low, because warn reads as a log level rather than as a quantity of disk.

/metrics carries the free, database and log byte gauges as nxdns_disk_free_bytes, nxdns_disk_db_bytes and nxdns_disk_log_bytes, and counts gated refreshes as nxdns_blocklist_refreshes_gated_total. The disk state itself is on /api/health, not in the metrics output.

What the state means. The monitor samples free space and database sizes once a minute. Below disk.warn_free_mb it logs the transition. Below disk.min_free_mb it gates every non-essential write: the query logger holds its batches, the client tracker stops persisting, and blocklist refreshes are skipped and counted in nxdns_blocklist_refreshes_gated_total. A gate that has already cost query rows shows as query_history.state = "losing". Resolution never degrades because the disk is full — this was verified by setting the thresholds above the free space on the volume: the state went critical, a refresh was gated, and queries kept being answered.

Fix. Recover space — lower logging.retention_days, or stop the service and delete querylog.db — and writes resume on the next sample.

Blocklists are not filtering

Symptom. Domains that should be blocked resolve normally.

Diagnosis. Read the startup line:

journalctl -u nxdns | grep 'serving on'

It ends in either blocklist generation N or unfiltered (no blocklist snapshot).

Fix. unfiltered means no snapshot loaded at all; the download or compile warning that explains it is earlier in the same start. nxdns serves anyway on purpose — a household loses more from DNS that refuses to start than from a window of unfiltered answers.

A generation number with nothing being blocked is a different problem: the snapshot loaded but has no sources in it. The line blocklist snapshot generation 1: 0 of 0 sources loaded says exactly that. Add a source in the admin interface, or a blocklist_sources entry to the configuration file with a group_sources link naming a group.

One name resolving while its neighbours are blocked is a third case, and /api/lookup answers it directly: it reports which level of the filtering ladder decided, and against what.

curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com'

{"domain":"api.ads.tvb.com","group_id":1,"local_records":false,"forward_zone":null,"blocked":false,"reason":"blocklist_exception","matched":"api.ads.tvb.com","source_url":"https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt","safe_search_rewrite":null}

blocklist_exception means a downloaded list lifted that name with an @@ line, and source_url names the list that did it. Nothing is broken, and the list is not overruling you: an exception cancels only what another list blocks. Your own rule wins over it. Adding an exact block rule for the same name and asking again reports rule_block_exact, blocked true and a null source_url.

A database stamped by a newer binary

Symptom. After putting an older binary back, it will not start:

warning(migrations): config.db is at schema version 99; this nxdns binary supports 1
nxdns run failed: SchemaTooNew

Fix. There is no downgrade. Import the export you took before upgrading into a fresh data directory with the older binary; see Upgrade nxdns.