mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 25 Activity

upstream: an unnamed connect errno fails over, local-resource exchanges are counted, doh faults carry the http status

Browse Source
This commit is contained in:
mokhtar
2026-10-20 10:11:34 +02:00
parent 2697adfa27
commit 8efe504c51
8 changed files with 191 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/how-to/troubleshoot.md
+22
View File
@@ -273,6 +273,28 @@ curl -s http://127.0.0.1:8080/metrics | grep nxdns_upstream_
**Fix.** Raise `upstream.total_timeout_ms` if the queue drains but drains too slowly for the budget. Otherwise the queue is telling you the upstream is slow: an upstream answering in a few milliseconds does not fill eight concurrent slots at household query rates, so sustained queueing points at the resolver you configured, and a faster one is the fix. Adding a second upstream helps only with saturation, not with latency: failover is strict priority, not load spreading. A query does take the next upstream when the first one's slots are all busy — priority orders the candidates that can be admitted right now — but once every eligible upstream is full it blocks on the highest-priority one, and it still sends one query to one upstream at a time. The per-upstream slot count is compiled, not configured, so there is no knob to widen one upstream either.
## SERVFAIL rows that name no upstream
**Symptom.** The query log shows SERVFAIL rows whose `upstream` is empty, and `/api/health` shows every upstream healthy.
**Diagnosis.** A failed exchange names no upstream only when no attributable attempt happened, and two pool-wide counters say why:
```sh
curl -s http://127.0.0.1:8080/metrics | grep -E 'nxdns_upstream_(budget_exhausted|local_resource)_total'
```
`nxdns_upstream_budget_exhausted_total` is a query whose own budget ran out first; the previous section covers it. `nxdns_upstream_local_resource_total` is an exchange this process ended on its own resources: out of memory, out of file descriptors, or a system error the runtime has no name for. Neither is charged to an upstream, because no upstream caused it, and a local failure does not fail over, because the next upstream would hit the same wall. Both count exchanges, not rows without an upstream: when an earlier upstream had already failed in the same exchange, the row names that one.
**Fix.** For `local_resource_total`, look at the process's limits: `grep -E 'Max open files' /proc/$(pidof nxdns)/limits` against `ls /proc/$(pidof nxdns)/fd | wc -l`, and `VmRSS` in `/proc/$(pidof nxdns)/status` against the memory the host has. A connect that fails for a reason the runtime cannot name — a blackhole route to the upstream is one — is not counted here: it is charged to that upstream as `ConnectFailed (cause Unexpected)`, and the query fails over.
## A DoH upstream fails with `HttpStatus (status 505 HTTP Version Not Supported)`
**Symptom.** Diagnostics shows the upstream failing with that cause, and no query ever succeeds against it.
**Diagnosis.** The server refuses HTTP/1.1. nxdns speaks DoH over HTTP/1.1 only, and some public resolvers (Quad9's `https://dns.quad9.net/dns-query`, for one) require HTTP/2. Other statuses name themselves the same way: a 403 or 404 is usually a wrong URL path.
**Fix.** Use that provider's DoT endpoint instead (for Quad9, `tls://9.9.9.9:853` with `tls_name = "dns.quad9.net"`), or a DoH upstream that accepts HTTP/1.1.
## The disk is filling up
**Symptom.** Writes stop but DNS keeps answering. The journal shows the transition: