mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 25 Activity

docs: rewrite for people running nxdns; query-history refusals name their remedy and a refused start gives no wrong hints

Browse Source
This commit is contained in:
mokhtar
2026-10-21 17:03:00 +02:00
parent fbc8fd7882
commit c21e5a9542
31 changed files with 1186 additions and 3877 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/files-and-directories.md
+47 -75
View File
@@ -1,113 +1,85 @@
# Files and directories
Every path nxdns reads or writes, and the mode it is created with. Source of truth: `src/cli.zig` (`DataDir`), `src/storage/querylog_schema.zig` (the preserved query-log databases), `src/filter/manager.zig` (the blocklist snapshots), `src/platform/logging.zig` (the log file).
Every file nxdns reads or writes, and who may read it.
## The data directory
Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and `nxdns import` create it and its parents at mode 0700 when it is missing; `nxdns check` and `nxdns export` do not create it. `export` fails if it is not there. `check` opens it only when no `--config FILE` was given: with that flag it grades the file and never looks at the directory at all. Without it, an absent `config.db` is a failure naming the two ways to get one, exit 2.
`run`, `import` and `export` go through `DataDir.openConfigDb`, which opens `config.db` read/write, chmods it to 0600, enables WAL — creating `config.db-wal` and `config.db-shm` — and then runs any pending schema migrations. So `nxdns export` writes to the data directory, and on a database one schema version behind it migrates it. `export` also creates an empty `config.db` if the directory exists without one.
`nxdns check` is the exception: it does not use that path at all. It opens `config.db` immutable, which is `SQLITE_OPEN_READONLY` plus `immutable=1`, so SQLite refuses every statement that would write and builds no wal-index. No `config.db-wal` and no `config.db-shm` appear beside the file, the mode is left alone, and no migration runs — a database behind this binary's schema is reported as a failure naming `nxdns run` as the fix. A `check` against a data directory leaves it byte-identical, and `check` reaches the database branch only when `config.db` is already there.
`immutable=1` ignores any `-wal` file, so it is refused rather than used when one holds bytes: the newest settings would be invisible and `check` would grade older ones from the main file. That is the "uncheckpointed changes" failure in [the CLI reference](cli.md#what-check-does-not-do).
`/var/lib/nxdns` by default, or the directory `--data-dir` names. `nxdns run` and `nxdns import` create it at mode 0700 when it is missing.
| Path | What it is | Mode |
| --- | --- | --- |
| `config.db` | The configuration database, including `web.password_hash`. The source of truth in database mode; in file mode it is the runtime substrate the file is reconciled onto (see [the configuration file](#the-configuration-file)). | 0600 |
| `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created by `run`, `import` and `export` when WAL is enabled, inheriting the main file's permissions. `check` creates neither. | 0600 |
| `querylog.db` | The query log: every domain every client asked for. Missing or damaged, it is recreated empty; an older schema is migrated in place, and a schema this build cannot use refuses the startup. See [the query-log lifecycle](query-log-lifecycle.md). | 0600 |
| `querylog.db-wal`, `querylog.db-shm` | WAL sidecars for `querylog.db`. | 0600 |
| `querylog.db.<reason>-<unix-seconds>` | A damaged `querylog.db`, moved aside before an empty one was created in its place. Kept, never overwritten. `<reason>` is one of `corrupt`, `not-a-database` or `quick-check-failed`; see [why a query log is moved aside](#why-a-query-log-is-moved-aside). | Whatever the renamed file had — no chmod reaches it |
| `querylog.db.<reason>-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 1 and rises until the name is free. Two recreates within one second is the case it exists for. | The same |
| `querylog.db.pre-migrate-<unix-seconds>` | A complete copy of `querylog.db` taken immediately before a schema migration. Exactly one survives: a successful migration deletes every other one, and a later successful start retries that cleanup. Written by `VACUUM INTO`, so it holds the committed database including anything still only in the write-ahead log, and it needs no sidecars of its own. | 0600 |
| `querylog.db.pre-migrate-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 2. | The same |
| `blocklists/` | Compiled blocklist snapshots, one subdirectory of the data directory. | 0700 |
| `blocklists/<id>.list` | Exact domains for blocklist source `<id>`, one per line, behind a header. | 0600 |
| `blocklists/<id>.wild` | Wildcard entries for the same source. | 0600 |
| `blocklists/<id>.allow` | Exception entries for the same source: the names its `@@` lines lift. Absent on a source compiled before exceptions were honoured, which reads as empty. | 0600 |
| `blocklists/<id>.raw.tmp`, `<id>.list.tmp`, `<id>.wild.tmp`, `<id>.allow.tmp` | Transient refresh state: the downloaded body and the three compile outputs before they are published by rename. | 0600 |
| `config.db` | The configuration, including the password hash. In database mode this is the configuration; in file mode nxdns rebuilds it from the file at every start. | 0600 |
| `querylog.db` | The query history: every name every device asked for. See [The query history database](#the-query-history-database). | 0600 |
| `config.db-wal`, `config.db-shm`, `querylog.db-wal`, `querylog.db-shm` | SQLite's working files for each database. They hold the same data, and belong with their database whenever you copy or move it. | 0600 |
| `querylog.db.pre-migrate-<unix-seconds>` | A copy of the query history taken just before an upgrade converted it. The latest one is kept. | 0600 |
| `querylog.db.<reason>-<unix-seconds>` | A damaged query history, moved aside; see below. | as it was |
| `blocklists/` | The downloaded blocklists, compiled. | 0700 |
| `blocklists/<id>.list`, `<id>.wild`, `<id>.allow` | One blocklist's domains, wildcard entries and exceptions. `<id>` is the list's number in nxdns. | 0600 |
| `blocklists/<id>.*.tmp` | A download in progress. | 0600 |
`<id>` is the `blocklist_sources` row id.
The databases are private because they are sensitive: `config.db` holds the password hash, and `querylog.db` and its copies hold the browsing history of every device on your network. nxdns restricts them to 0600 whatever the umask.
### The orphan sweep
The sweep decides by id, not by suffix. It matches all seven names above and deletes those whose `<id>` is no longer a `blocklist_sources` row, so the compiled `.list`, `.wild` and `.allow` of a removed source go, and so do a `.raw.tmp`, `.list.tmp`, `.wild.tmp` or `.allow.tmp` left behind by a refresh that was killed before it could clean up. Files belonging to a source that still has a row are never touched, whatever state they are in: the sweep holds the same lock every refresh takes, so it never reads the directory while a refresh is part-way through.
It runs at three moments:
- at startup, before the first refresh pass — this is what collects what a killed process left behind, and the compiled files of a source deleted while the server was down;
- before each scheduled update pass, ahead of the disk-space gate: the sweep only unlinks, so it is the one step here that can give a critically full disk room back, and gating it would keep the residue that helped fill the disk;
- immediately after `DELETE /api/blocklists/{id}`, which is when an orphan is actually created in normal operation. Without it a deleted list would keep its megabytes until the next scheduled pass.
With `blocklist_update.enabled = false` there are no scheduled passes, so only the first and the last of those three happen.
Each deletion is logged:
nxdns deletes a blocklist's files when you remove the list, and at startup it deletes any left behind by a list removed while it was down or by a download that was interrupted:
```
info(blocklist_manager): pruned orphaned blocklist file 9999.list
```
A failed sweep is a warning, not an outage — leftover bytes do not justify losing the refresh pass behind them, let alone the server.
`nxdns check` only reads the data directory, and changes nothing in it.
The temporaries of a source that still exists are cleaned by the refresh that owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`, `.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not.
## The query history database
### Why a query log is moved aside
At each start nxdns checks `querylog.db` before it serves anything.
A `querylog.db` is moved aside only when it is genuinely damaged, and the name it is given says which of the three cases it hit:
- **Current**: it opens.
- **From an older release**: nxdns copies it to `querylog.db.pre-migrate-<unix-seconds>`, then converts it in one step. If converting fails, the file keeps its old contents and nxdns refuses to start.
- **Missing**: nxdns creates an empty one.
- **Damaged**: nxdns renames it to `querylog.db.<reason>-<unix-seconds>`, deletes its `-wal` and `-shm` files, and starts an empty one. `<reason>` is `corrupt`, `not-a-database` or `quick-check-failed`. Nothing deletes these copies; remove them when you no longer want the history in them.
- **Written by a newer release, or too old to convert**: nxdns refuses to start, exits 2 and leaves the file as it is. The message says what to do:
| `<reason>` | What happened |
| --- | --- |
| `corrupt` | SQLite reported the file as damaged. |
| `not-a-database` | The file is not a SQLite database at all. |
| `quick-check-failed` | `PRAGMA quick_check` did not answer `ok`. |
```
refusing to open querylog database '/var/lib/nxdns/querylog.db': it is stamped 2, and this build supports schema versions 1 to 1 (SchemaTooNew). The file is left exactly as it is; a newer nxdns wrote it. Run that release again, put back the querylog.db.pre-migrate-* copy an upgrade left beside it, or move the file aside, together with its -wal and -shm files, to start with an empty query history
```
There is no fourth case. A healthy file carrying a schema version this build cannot use is neither migrated nor renamed: the startup refuses and the file stays where it is, which is [the query-log lifecycle](query-log-lifecycle.md). You can still find a `querylog.db.schema-changed-<unix-seconds>` in a data directory, because 0.0.13 and older produced one on any schema change — and still do, if you downgrade to one of them. This build never writes that name.
To start with an empty history, stop nxdns and move the three files aside:
Only the main file is renamed — its `-wal` and `-shm` are deleted, because a stale WAL would be replayed into the fresh database. A missing `querylog.db` is created without any aside file. The rename happens inside `querylog_schema.open`, before the 0600 chmod, and that chmod names `querylog.db` and its two sidecars only — so an aside file keeps the mode the file had at rename time, which for a `querylog.db` nxdns itself created is 0600 and for one an operator put there is whatever they left it at. Nothing prunes the aside files; they accumulate until an operator removes them, and each one holds the same browsing history the live query log holds.
```sh
cd /var/lib/nxdns
mkdir -p old-querylog
mv querylog.db querylog.db-wal querylog.db-shm old-querylog/
```
### Why the databases are 0600
`mv` complains about a `-wal` or `-shm` file that is not there; that is fine.
The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash and `querylog.db` holds the browsing history of every client on the LAN, so both are as sensitive as each other, and a WAL file holds the same rows as the database it belongs to. SQLite creates the main database at `0644 & ~umask`; nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather than being created world-readable.
A healthy query history is never replaced or deleted. A release that cannot convert the old history says `resets your query history` in its release notes, and explains how to keep the old file.
A `pre-migrate` backup holds that same browsing history, so it is chmodded 0600 the way the live file is: SQLite's `VACUUM INTO` creates it at `0644 & ~umask` and nxdns restricts it immediately afterwards. A backup it cannot restrict is a failed backup — the partial file is deleted and the startup refuses with `MigrationBackupFailed`, rather than leaving a world-readable copy behind.
### Going back to 0.0.13 or older
The aside files are the exception: they get no chmod at all, and an aside keeps whatever mode it had at rename time. For a `querylog.db` nxdns itself created that is 0600; for one an operator put there it is whatever they left it at.
Releases up to 0.0.13 do not recognise the current query history. They rename it to `querylog.db.schema-changed-<unix-seconds>` and start an empty one. To get the history back after returning to a newer release, stop nxdns, then in the data directory:
1. Move the empty `querylog.db` out of the way.
2. Delete `querylog.db-wal` and `querylog.db-shm`. Left in place, they would corrupt the history you restore.
3. Rename `querylog.db.schema-changed-<unix-seconds>` to `querylog.db`.
4. Start nxdns.
## The configuration file
There is no default path. `--config FILE` names the file, and without that flag no file is read at all — a `config.zon` sitting in `/etc/nxdns` that no invocation names is inert. `/etc/nxdns/config.zon` is a convention the packaging follows, not a location nxdns probes.
nxdns reads a configuration file only when `--config` names it, and never writes it. `/etc/nxdns/config.zon` is the conventional place, and the packaged systemd unit creates `/etc/nxdns` and keeps the service from writing to it.
nxdns reads the file and never writes it, in either authority mode. It does not create the file or its directory either; the systemd unit's `ConfigurationDirectory=nxdns` creates `/etc/nxdns`, and the same unit's `ReadOnlyPaths=/etc/nxdns` denies the service write access to it, so the file cannot be modified by the process that reads it.
Under `run --config FILE` the file is the configuration and the database is the runtime substrate the server reads from: every start reconciles the one onto the other. So in that mode `config.db` is not the backup — the file is.
`nxdns export --out FILE` writes a ZON file at mode 0600 through a temporary file and a rename. That file carries `web.password_hash`, so treat exports as secrets. Without `--out` the export goes to stdout, where permissions are the redirect's business.
An export, `nxdns export --out FILE`, is created at mode 0600 because it carries the password hash.
## The log file
Only when `logging.output = .file`. The path is `logging.file_path`, default `/var/log/nxdns/nxdns.log`, and validation requires it to be absolute.
Only when `logging.output = .file`. The file is `logging.file_path`, `/var/log/nxdns/nxdns.log` by default.
**nxdns does not create the log directory.** It must exist and be writable by the user the service runs as before the process starts; under systemd the unit's `LogsDirectory=nxdns` does that.
nxdns does not create the log directory; under systemd the packaged unit does. It rotates the file itself at `logging.max_size_mb` and keeps `logging.max_files` files, so with the defaults that is `nxdns.log` and `nxdns.log.1` to `nxdns.log.4`.
Rotation triggers at `logging.max_size_mb` MiB and keeps `logging.max_files` files in total, the live one included, so the highest generation on disk is `logging.max_files - 1`. With the default 5 that is `nxdns.log` plus `nxdns.log.1` through `nxdns.log.4`. Rotation deletes the highest generation, renames each remaining one up by one, then renames the live file to `.1`. A `max_files` of 1 or 0 deletes the live file instead of renaming it. The directory holding the log file is also sampled by the disk monitor.
Do not rotate this file with logrotate. nxdns keeps writing where it left off, so a rotated file either fills with blank space or stops growing while the old one grows without limit. If another tool must own the log, set `logging.output = .stderr` and let journald or Docker collect it.
| Path | What it is | Mode |
| --- | --- | --- |
| `logging.file_path` (default `/var/log/nxdns/nxdns.log`) | The live process log. Opened write-only and written at a tracked offset; created when it does not exist. | `0666 & ~umask` — nxdns never chmods it |
| `<file_path>.1` … `<file_path>.<max_files - 1>` | Rotated generations, newest first. | Inherited from the live file they were renamed from |
nxdns does not set the log file's mode. Under the packaged unit, which sets `UMask=0077`, it is 0600; started from a shell with the usual umask it is readable by every local user.
**Do not point external logrotate at this file.** nxdns owns the rotation of its own log. There is no append mode in Zig 0.16, so the writer reads the file length once when it opens the file and then writes every line at an offset it tracks in the process. A rotator that moves the file behind it breaks that offset, and nxdns does not notice until the next restart:
With `logging.output = .stderr` or `.syslog`, nxdns writes to standard error and creates no file.
- With `copytruncate` the offset survives the truncation, so the next line lands where it would have without it. The file regrows with a sparse, NUL-filled prefix as long as the log that was just rotated away.
- With rename-and-create nxdns keeps writing to the renamed inode. The new file stays empty, the renamed one grows without bound, and `logging.max_size_mb` bounds nothing on disk.
## Certificates and keys
If an external rotator has to own the file, set `logging.output = .stderr` and let the collector capture the stream instead.
The log file is the one path here nxdns does not set a mode on. `config.db` and `querylog.db` are chmodded to 0600 whatever the umask; the log file is left at whatever the umask gives it. Under the shipped unit that is 0600, because `deploy/systemd/nxdns.service` sets `UMask=0077`. Run from a shell with the usual `umask 022` it is 0644, and `LogsDirectory=nxdns` leaves the directory at systemd's default 0755, so a log written there is readable by any local user.
With `logging.output = .stderr` or `.syslog` nxdns writes to stderr and creates no file; under systemd the journal captures it.
## Certificate and key files
`doh_server.cert_path` / `key_path` and `dot_server.cert_path` / `key_path`, conventionally under `/etc/nxdns`. nxdns reads them, never writes or creates them. Both must be readable by the user nxdns runs as, and the key must belong to the certificate: `nxdns check` loads the pair and fails when it does not. The key should also be readable by its owner only, which `check` warns about when it is not. A watcher polls both files and swaps a renewed pair in without a restart. See [enable DoH and DoT](../how-to/enable-doh-and-dot.md).
The files `doh_server.cert_path`, `doh_server.key_path`, `dot_server.cert_path` and `dot_server.key_path` name, `/etc/nxdns/cert.pem` and `/etc/nxdns/key.pem` by default. nxdns only reads them. The user nxdns runs as must be able to read both, and nobody else should be able to read the key. nxdns picks up a renewed pair within 30 seconds; see [Enable DoH and DoT](../how-to/enable-doh-and-dot.md).