nxdns/docs/how-to/enable-doh-and-dot.md

Enable DoH and DoT

nxdns can answer encrypted queries on two extra listeners: DNS over HTTPS (doh_server) and DNS over TLS (dot_server). Both are off by default and both need a certificate and a private key in PEM form.

This page uses a scratch lab under /tmp/nxdns-lab so the commands run without root and without touching a real install. On a real install the files live under /etc/nxdns and the data directory is /var/lib/nxdns; the ports are 443 and 853 rather than the unprivileged ones below.

Every field mentioned here is documented in configuration reference.

1. Get a certificate and key

For a LAN service the practical options are a certificate from your ACME client (certbot, lego, caddy) for a name you control, or a self-signed pair. The lab below uses a self-signed pair, because it needs no domain:

mkdir -p /tmp/nxdns-lab/etc
cd /tmp/nxdns-lab
openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout etc/key.pem -out etc/cert.pem -days 365 \
  -subj "/CN=nxdns.lan" \
  -addext "subjectAltName=DNS:nxdns.lan,DNS:localhost,IP:127.0.0.1"
chmod 0600 etc/key.pem

A self-signed certificate means every client has to be told to trust it, or told to skip verification. That is why the client commands further down pass --insecure and +tls without a CA. A real deployment uses a real certificate and drops those flags.

2. Turn the listeners on

cert_path and key_path must be absolute, or relative to the process working directory. Point both endpoints at the same pair unless you have a reason not to:

.{
    .groups = .{.{ .name = "default" }},
    .upstreams = .{.{ .url = "https://cloudflare-dns.com/dns-query" }},
    .dns = .{ .bind_ipv4 = "127.0.0.1", .bind_ipv6 = "::1", .port = 15400 },
    .web = .{ .bind = "127.0.0.1", .port = 8451, .password = "lab-password" },
    .doh_server = .{
        .enabled = true,
        .bind = "127.0.0.1",
        .port = 8443,
        .cert_path = "/tmp/nxdns-lab/etc/cert.pem",
        .key_path = "/tmp/nxdns-lab/etc/key.pem",
    },
    .dot_server = .{
        .enabled = true,
        .bind = "127.0.0.1",
        .port = 8853,
        .cert_path = "/tmp/nxdns-lab/etc/cert.pem",
        .key_path = "/tmp/nxdns-lab/etc/key.pem",
    },
}

Write that to /tmp/nxdns-lab/etc/config.zon. The configuration file seeds an empty database and is then ignored; to change these settings on a server that already has a database, edit them through the API or through export/import — see the configuration model.

3. Check the files before starting

nxdns check --data-dir /tmp/nxdns-lab/data --config /tmp/nxdns-lab/etc/config.zon

checking configuration file /tmp/nxdns-lab/etc/config.zon
OK upstreams[0] https://cloudflare-dns.com
OK: no problems found

check loads both endpoints' certificate and key the same way the listeners do, so what passes here will start. An unreadable file is a failure and exits 2:

FAIL doh_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable
FAIL dot_server.cert_path: '/tmp/nxdns-lab/etc/cert.pem': certificate file is not readable

So is a key that does not belong to the certificate, which is the mistake worth catching before a restart — the two files are individually valid and only their pairing is wrong. mbedTLS writes its own line to stderr as it rejects the pair:

warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
warning(tls_server): mbedtls_pk_check_pair failed: RSA - Key failed to pass the validity check of the library (-16896)
checking configuration file /tmp/nxdns-lab/etc/mismatch.zon
FAIL doh_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not belong to the certificate
FAIL dot_server.key_path: '/tmp/nxdns-lab/etc/other.pem': private key does not belong to the certificate
OK upstreams[0] https://cloudflare-dns.com

A key readable by anyone but its owner is a warning instead. It does not change the exit code, because the service still starts, and the summary line counts it rather than claiming nothing was found:

WARN doh_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only
WARN dot_server.key_path: '/tmp/nxdns-lab/etc/key.pem' is mode 644; a TLS key must be readable by its owner only
OK upstreams[0] https://cloudflare-dns.com
OK: no failures found, 2 warnings

4. Start and confirm the listeners

nxdns run --data-dir /tmp/nxdns-lab/data --config /tmp/nxdns-lab/etc/config.zon

Two lines in the log say the listeners bound:

info(nxdns): doh listener on 127.0.0.1:8443
info(nxdns): dot listener on 127.0.0.1:8853

If a certificate cannot be loaded while its endpoint is enabled, nxdns refuses to start and exits 2 rather than serving DNS without the listener you asked for:

doh_server: '/tmp/nxdns-lab/etc/cert.pem' + '/tmp/nxdns-lab/etc/key.pem': private key file is not readable
nxdns run failed: BadCertificate
run `nxdns check` to see the configuration in full

5. Query DoT

Recent dig speaks DNS over TLS with +tls (this page was checked with BIND 9.20.26):

dig @127.0.0.1 -p 8853 +tls example.com A +short

172.66.147.243
104.20.23.154

6. Query DoH

The only path the DoH listener serves is /dns-query; anything else is a 404. It accepts both the POST form (the query as an application/dns-message body) and the GET form (?dns= with base64url of the same bytes).

The body is a raw DNS query in wire format. Build one for example.com A — header with the recursion-desired bit, one question, then the QNAME as length-prefixed labels:

printf '%s' '000001000001000000000000076578616d706c6503636f6d0000010001' \
  | xxd -r -p > /tmp/nxdns-lab/query.bin

POST it:

curl -sS --insecure --http1.1 \
  -H 'content-type: application/dns-message' \
  --data-binary @/tmp/nxdns-lab/query.bin \
  --output /tmp/nxdns-lab/answer.bin \
  -w 'http %{http_code}, %{size_download} bytes\n' \
  https://127.0.0.1:8443/dns-query
xxd /tmp/nxdns-lab/answer.bin

http 200, 72 bytes
00000000: 0000 8180 0001 0002 0000 0001 0765 7861  .............exa
00000010: 6d70 6c65 0363 6f6d 0000 0100 01c0 0c00  mple.com........
00000020: 0100 0100 0000 0400 04ac 4293 f3c0 0c00  ..........B.....
00000030: 0100 0100 0000 0400 0468 1417 9a00 0029  .........h.....)
00000040: 04d0 0000 0000 0000                      ........

The second flag byte 80 and the third answer-count field 0002 say: response, no error, two answer records.

The GET form takes the same bytes, base64url-encoded with the padding removed:

Q=$(base64 -w0 /tmp/nxdns-lab/query.bin | tr '+/' '-_' | tr -d '=')
curl -sS --insecure --http1.1 -o /tmp/nxdns-lab/get.bin \
  -w 'GET http %{http_code}, %{size_download} bytes\n' \
  "https://127.0.0.1:8443/dns-query?dns=$Q"

GET http 200, 61 bytes

--http1.1 matters: without it curl offers HTTP/2 over ALPN, and the DoH listener negotiates only what it advertises. --insecure is only needed for the self-signed lab certificate.

7. Renewals

A watcher polls both files every 30 seconds and compares their modification time and size against the pair currently loaded. When either differs it reloads and swaps the new pair in; connections already open finish on the old certificate. Nothing has to restart.

Replace the pair and wait one poll interval:

openssl req -x509 -newkey rsa:2048 -nodes \
  -keyout /tmp/nxdns-lab/etc/key.pem -out /tmp/nxdns-lab/etc/cert.pem -days 365 \
  -subj "/CN=nxdns.lan" \
  -addext "subjectAltName=DNS:nxdns.lan,DNS:localhost,IP:127.0.0.1"
chmod 0600 /tmp/nxdns-lab/etc/key.pem

Within 30 seconds the log says so, once per endpoint that watches the file:

info(cert_store): certificate reloaded from /tmp/nxdns-lab/etc/cert.pem
info(cert_store): certificate reloaded from /tmp/nxdns-lab/etc/cert.pem

8. Reload immediately

To skip the wait — from an ACME deploy hook, for example — call POST /api/certs/reload. It needs a session; see set up admin authentication for the login call that fills cookies.txt.

curl -sS -b /tmp/nxdns-lab/cookies.txt -X POST http://127.0.0.1:8451/api/certs/reload

{"doh":{"enabled":true,"reloaded":true,"error":null},"dot":{"enabled":true,"reloaded":true,"error":null}}

The route always answers 200: the per-endpoint outcome is the payload, not the status code. A disabled endpoint reports "enabled":false. A reload that fails names the reason and leaves the old certificate serving:

chmod 000 /tmp/nxdns-lab/etc/key.pem
curl -sS -b /tmp/nxdns-lab/cookies.txt -X POST http://127.0.0.1:8451/api/certs/reload

{"doh":{"enabled":true,"reloaded":false,"error":"private key file is not readable"},"dot":{"enabled":true,"reloaded":false,"error":"private key file is not readable"}}

The listeners keep working through that failure:

dig @127.0.0.1 -p 8853 +tls example.com A +short

104.20.23.154
172.66.147.243

Undo it with chmod 0600 /tmp/nxdns-lab/etc/key.pem and reload again. The counters are on /metrics:

nxdns_cert_reloads_total{endpoint="doh"} 3
nxdns_cert_reloads_total{endpoint="dot"} 3
nxdns_cert_reload_failures_total{endpoint="doh"} 2
nxdns_cert_reload_failures_total{endpoint="dot"} 2

File ownership on a real install

The key must be readable by the user nxdns runs as, and by nobody else.

Under systemd the service runs as the nxdns user:

chown nxdns:nxdns /etc/nxdns/cert.pem /etc/nxdns/key.pem
chmod 0644 /etc/nxdns/cert.pem
chmod 0600 /etc/nxdns/key.pem

Under Docker the container runs as uid 65532, fixed in the image, and /etc/nxdns is a read-only bind mount — the container cannot fix permissions itself, so the host-side files must already be readable by that uid. It has no name on the host, so chown it numerically:

cd deploy/docker
chown 65532:65532 etc-nxdns/cert.pem etc-nxdns/key.pem
chmod 0644 etc-nxdns/cert.pem
chmod 0600 etc-nxdns/key.pem

Not verified on this host: the two chown blocks above. Both need root, and the systemd one needs an nxdns user this development machine does not have. Everything else on this page was executed as written.

Ports 443 and 853

The defaults are the standard ports, which are privileged. Under the packaged systemd unit that is already handled: it grants CAP_NET_BIND_SERVICE for port 53 and the same capability covers 443 and 853. See install with systemd.