storage: version querylog.db and migrate it in place, never reset a healthy file

querylog.db carries a schema version; migrations run at startup as one transaction after a vacuumed 0600 backup, and every failure refuses startup (exit 2, no systemd restart loop) instead of starting empty. corruption is the only automatic recreate left. the cut gate now requires a fixture-proven migration or an explicit versioned break with restore instructions, and locks shipped migration files and fixtures byte-for-byte.

docs/README.md

@@ -37,6 +37,7 @@ Descriptions of what is there. No procedures, no advice.
 - [reference/api.md](reference/api.md) — every REST route, authentication and the event stream.
 - [reference/cli.md](reference/cli.md) — the six subcommands, every flag, every exit code.
 - [reference/files-and-directories.md](reference/files-and-directories.md) — the data directory layout and file modes.
+- [reference/query-log-lifecycle.md](reference/query-log-lifecycle.md) — how `querylog.db` is versioned, migrated, backed up and, rarely, recreated.
 - [reference/performance.md](reference/performance.md) — the targets and the measured numbers.
 
 ## Explanation

docs/explanation/architecture.md

@@ -27,7 +27,7 @@ Directories:
 | `src/cache/` | `dns_cache.zig`: bounded in-memory TTL cache of whole response messages, keyed by the question. The clock arrives as a parameter. |
 | `src/upstream/` | Upstream resolution: shared vocabulary and the `Client` interface (`transport.zig`), DoH client (RFC 8484), DoT client (RFC 7858), per-endpoint health and backoff (`health.zig`), and `pool.zig` — priority-ordered failover that is itself a `transport.Client`, so the handler sees one interface. |
 | `src/server/` | The serving side: UDP/TCP/DoH/DoT listeners, `handler.zig` (the whole query pipeline), `cert_store.zig` (refcounted TLS cert holder), `rate_limiter.zig`, `pause.zig`, `clients.zig` (client auto-materialisation), `local_tables.zig` (published local-answer tables), `query_sink.zig` (log and SSE fanout), `shutdown.zig` (SIGINT/SIGTERM into one `std.Io.Event`). |
-| `src/storage/` | SQLite ownership: `db.zig` is the only file that calls SQLite, `config_schema.zig` + `migrations.zig` for `config.db`, `querylog_schema.zig` (open-or-recreate), async query `logger.zig`, `retention.zig`, `disk_monitor.zig`, and one repository per table under `repositories/`. |
+| `src/storage/` | SQLite ownership: `db.zig` is the only file that calls SQLite, `config_schema.zig` + `migrations.zig` for `config.db`, `querylog_schema.zig` + `querylog_versions.zig` + `querylog_migrations.zig` for `querylog.db`, async query `logger.zig`, `retention.zig`, `disk_monitor.zig`, and one repository per table under `repositories/`. |
 | `src/config/` | The one configuration model (`model.zig`), the pure validator (`validate.zig`), `import.zig`/`export.zig` (ZON to and from `config.db`, byte-stable round trip), `loader.zig` (read/parse/validate a named file, with the shared fault mapping), `reconcile.zig` (converge the database onto a parsed config by row identity). |
 | `src/web/` | The admin HTTP layer: `server.zig` (listener), `router.zig`/`routes.zig`, one file per resource under `handlers/`, `auth.zig` (sessions), `sse.zig` (live query fanout), `static.zig` (embedded SPA), `metrics.zig` (Prometheus), `openapi.zig` (served contract), `api_limiter.zig`, `http_util.zig`. |
 | `src/platform/` | OS and TLS edges: IP address values, the `std.log` sink (`logging.zig`), `statfs.zig` (free-space query via libc), client TLS over `std.crypto.tls` (`tls_client.zig`), server TLS over vendored Mbed TLS (`tls_server.zig`). |
@@ -115,7 +115,7 @@ Two databases with opposite contracts, in one data directory (see [reference/fil
 
 Which of the file and the database is *authoritative* is chosen by the invocation, not by state: bare `nxdns run` serves the database, and `nxdns run --config FILE` makes the file authoritative and reconciles the database onto it at every start. `reconcile.zig` is that convergence, matching rows by identity and writing only differences, so runtime state — blocklist checksums, compiled snapshots, client history — survives. Why it works that way is [configuration-model.md](configuration-model.md).
 
-**`querylog.db` is expendable.** It is never migrated. Its schema carries a fingerprint derived from the DDL text, and at open, a missing, corrupt, non-database, `quick_check`-failing or fingerprint-mismatched file is moved aside and recreated empty — the old file is kept under a new name rather than deleted, so an operator can still look at it. Retention deletes old rows daily and periodically rewrites the file to reclaim space.
+**`querylog.db` is the expendable one, but its history is not thrown away.** It carries a logical version in `PRAGMA user_version`, and an older supported version is migrated in place at open: one transaction, behind one `querylog.db.pre-migrate-<epoch>` backup, of which only the newest is kept. Only real damage recreates the file — missing, corrupt, not a database, or failing `quick_check` — and then the old file is kept under a new name rather than deleted, so an operator can still look at it. A healthy file this build cannot read is neither migrated nor moved aside: the startup refuses and says why, because losing months of history to a rollback is worse than a server that will not start. Retention deletes old rows daily and periodically rewrites the file to reclaim space. The whole contract is [reference/query-log-lifecycle.md](../reference/query-log-lifecycle.md).
 
 The split exists so that the churn of the second database can never endanger the first. Query logs are high-volume, disposable, and the thing most likely to be corrupted by a power cut on an SD card; configuration is small, irreplaceable, and the thing an operator would have to reconstruct by hand. Giving them one file would force the careful contract onto the noisy data or the loose contract onto the valuable data.
 

docs/how-to/troubleshoot.md

@@ -339,3 +339,37 @@ nxdns run failed: SchemaTooNew
 ```
 
 **Fix.** There is no downgrade. Import the export you took before upgrading into a fresh data directory with the older binary; see [Upgrade nxdns](upgrade.md).
+
+## The server refuses to start over querylog.db
+
+**Symptom.** The process stops at startup naming the query log, and the error is one of four names:
+
+```
+error(querylog_schema): refusing to open querylog database '/var/lib/nxdns/querylog.db': it is stamped 7, and this build supports schema versions 1 to 1 (SchemaTooNew). The file is left exactly as it is; see docs/how-to/troubleshoot.md, "The server refuses to start over querylog.db"
+nxdns run failed: SchemaTooNew
+```
+
+This is a refusal, not damage. nxdns will not replace a healthy query log to get itself started, so the file is left exactly as it was — schema, rows, coverage watermark and version stamp all unchanged — and the startup fails instead. All four exit 2, the code that means an operator has to act, because none of them resolves on a retry — the shipped systemd unit's `RestartPreventExitStatus=2 64` stops the unit on the first refusal instead of restart-looping it. `systemctl status nxdns` shows the refusal. `nxdns check` does not grade the query log at all, so it will not reproduce any of these.
+
+[The query-log lifecycle](../reference/query-log-lifecycle.md) is the full contract behind this page.
+
+**Fixes by name.**
+
+- `SchemaTooNew` — the file was stamped by a newer nxdns than the one you are running, which normally means a binary was rolled back. Put the newer release back and start it: the file is exactly as that release left it. If you mean to stay on the older release, that release cannot read this file, so restore the `querylog.db.pre-migrate-<unix-seconds>` copy the upgrade left beside it — stop the server, move `querylog.db` and its `querylog.db-wal` and `querylog.db-shm` out of the way, rename the backup to `querylog.db`, and start. Starting empty is also an option: with the server stopped, move `querylog.db` and both sidecars aside and the next start creates a fresh log.
+- `SchemaUnsupported` — the stamp is not a version this build can reach. Either the file predates 0.0.12, or it came from somewhere else, or a release since deliberately broke the schema; the changelog section for the release you are running says so when it is the third case. There is no migration path, by contract. Keep the file if the history matters — copy it somewhere and read it with the `sqlite3` shell — and if starting with an empty log is acceptable, stop the server, move `querylog.db`, `querylog.db-wal` and `querylog.db-shm` out of the data directory by hand, and start again.
+- `MigrationBackupFailed` — a migration was due and the pre-migration backup could not be written, so nothing was migrated. The line above names the destination and the reason, which is almost always a full or read-only data directory. Free space or fix the permissions and start again.
+- `MigrationFailed` — read the log line above it, because two different states wear this one name.
+
+**Which `MigrationFailed` you have.** The distinction is in the line the migration logged, and it decides whether you do anything at all:
+
+- Before the commit: `querylog migration 1 -> 2 failed before commit (...); the database is unchanged`. Nothing was applied. The file still carries its old version and every row, and this run's backup was deleted because the original is intact. Restarting will attempt the same migration and fail the same way, so this needs the underlying cause — the log line names it — or a report.
+- After the commit: `querylog migration 1 -> 2 COMMITTED and the database IS at version 2, but the connection could not be restored: ...; the backup '...' is kept and the next start will open the migrated file normally`. The migration DID complete. Only that one startup is refused, the pre-migration backup is kept, and the next start opens the migrated file on the ordinary current-version path. Start the server again.
+
+In neither case does the server start with an empty log on its own. Recreating a query log automatically is reserved for real corruption; see [why a query log is moved aside](../reference/files-and-directories.md#why-a-query-log-is-moved-aside).
+
+> Not reproduced against a running service: the four refusals are covered by the
+> test suite rather than by a hand-driven install, and the released chain has no
+> migration step in it yet, so no upgrade produces a `pre-migrate` backup today.
+> The messages above are the ones `src/storage/querylog_schema.zig` and
+> `src/storage/querylog_migrations.zig` emit, with a data directory path and
+> example version numbers filled in.

docs/how-to/upgrade.md

@@ -263,7 +263,7 @@ nxdns run failed: SchemaTooNew
 > hand and `nxdns run` was pointed at it. The two lines above are that run's
 > output.
 
-That run exits 1. Recovering means importing the export you took in step 1 into a fresh data directory with the older binary.
+That run exits 2, and the shipped unit stops rather than restart-loops it. Recovering means importing the export you took in step 1 into a fresh data directory with the older binary.
 
 ### Rolling back from file mode
 

docs/reference/cli.md

@@ -214,7 +214,7 @@ Prints the usage text to stdout and exits 0. `nxdns --help` and `nxdns -h` do th
 | --- | --- |
 | 0 | Success. |
 | 1 | Runtime failure — I/O, database, out of memory. A partial diagnostic report caused by an allocation failure is a runtime failure, not a verdict on the configuration. |
-| 2 | A configuration problem the operator can fix, or a `check` that found one. |
+| 2 | A configuration problem the operator can fix, a `check` that found one, or a deliberate refusal to run that no retry will clear. |
 | 64 | Usage error — unknown command or flag, a flag without its value, a missing or extra argument. |
 
 Code 2 means the same thing from every subcommand. `src/config/faults.zig` holds the one list of errors that mean "the configuration the operator supplied is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits 2 whichever command read it. The list is every error the validator raises, plus `ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams`, `BadCertificate` and `ManagedConfigUnreadable`. In practice that covers a file with a syntax error, one larger than 4 MiB, one with no `default` group (`MissingDefaultGroup`), one with no enabled upstream (`NoUpstreams`), a bad bind address, a bad rate limit, an unusable certificate, `password` and `password_hash` set together, and a `--config` path that is absent or unreadable.
@@ -237,6 +237,8 @@ load one with `nxdns import <file>`, or make a file the source of truth with `nx
 
 `import` exits 2 for those faults and for `DestructiveImport`. That last one is deliberately not a configuration fault — it reports what applying the file would delete, rather than anything wrong with its content — and `import` decides it for itself; the answer to it is `--allow-delete`, not an edit.
 
+`run` also exits 2 on the four query-log schema refusals — `SchemaTooNew`, `SchemaUnsupported`, `MigrationFailed` and `MigrationBackupFailed` — which are in the same list. They are not a verdict on a file the operator wrote, but they share the property exit 2 exists to signal: the server is refusing on purpose, an operator has to act, and a restart will only repeat the refusal. Exit 1 would put them under the unit's `Restart=on-failure` and loop them. See [the server refuses to start over querylog.db](../how-to/troubleshoot.md#the-server-refuses-to-start-over-querylogdb).
+
 `OutOfMemory` is exit 1 even when problems were recorded, because the report is then incomplete. Every other error is 1.
 
 Where an exit code sends you next: [troubleshoot](../how-to/troubleshoot.md).

docs/reference/files-and-directories.md

@@ -16,10 +16,12 @@ Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and `nx
 | --- | --- | --- |
 | `config.db` | The configuration database, including `web.password_hash`. The source of truth in database mode; in file mode it is the runtime substrate the file is reconciled onto (see [the configuration file](#the-configuration-file)). | 0600 |
 | `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created by `run`, `import` and `export` when WAL is enabled, inheriting the main file's permissions. `check` creates neither. | 0600 |
-| `querylog.db` | The query log: every domain every client asked for. Expendable — if it is missing or unusable it is recreated empty. | 0600 |
+| `querylog.db` | The query log: every domain every client asked for. Missing or damaged, it is recreated empty; an older schema is migrated in place, and a schema this build cannot use refuses the startup. See [the query-log lifecycle](query-log-lifecycle.md). | 0600 |
 | `querylog.db-wal`, `querylog.db-shm` | WAL sidecars for `querylog.db`. | 0600 |
-| `querylog.db.<reason>-<unix-seconds>` | A `querylog.db` this build could not use, moved aside before an empty one was created in its place. Kept, never overwritten. `<reason>` is one of `corrupt`, `not-a-database`, `quick-check-failed` or `schema-changed`; see [why a query log is moved aside](#why-a-query-log-is-moved-aside). | Whatever the renamed file had — no chmod reaches it |
+| `querylog.db.<reason>-<unix-seconds>` | A damaged `querylog.db`, moved aside before an empty one was created in its place. Kept, never overwritten. `<reason>` is one of `corrupt`, `not-a-database` or `quick-check-failed`; see [why a query log is moved aside](#why-a-query-log-is-moved-aside). | Whatever the renamed file had — no chmod reaches it |
 | `querylog.db.<reason>-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 1 and rises until the name is free. Two recreates within one second is the case it exists for. | The same |
+| `querylog.db.pre-migrate-<unix-seconds>` | A complete copy of `querylog.db` taken immediately before a schema migration. Exactly one survives: a successful migration deletes every other one, and a later successful start retries that cleanup. Written by `VACUUM INTO`, so it holds the committed database including anything still only in the write-ahead log, and it needs no sidecars of its own. | 0600 |
+| `querylog.db.pre-migrate-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 2. | The same |
 | `blocklists/` | Compiled blocklist snapshots, one subdirectory of the data directory. | 0700 |
 | `blocklists/<id>.list` | Exact domains for blocklist source `<id>`, one per line, behind a header. | 0600 |
 | `blocklists/<id>.wild` | Wildcard entries for the same source. | 0600 |
@@ -52,14 +54,15 @@ The temporaries of a source that still exists are cleaned by the refresh that ow
 
 ### Why a query log is moved aside
 
-A `querylog.db` is moved aside when it is missing nothing but usability, and the name it is given says which of the four cases it hit:
+A `querylog.db` is moved aside only when it is genuinely damaged, and the name it is given says which of the three cases it hit:
 
 | `<reason>` | What happened |
 | --- | --- |
 | `corrupt` | SQLite reported the file as damaged. |
 | `not-a-database` | The file is not a SQLite database at all. |
 | `quick-check-failed` | `PRAGMA quick_check` did not answer `ok`. |
-| `schema-changed` | Nothing is wrong with the file. Its `user_version` fingerprint does not match this build's schema, so this build cannot read it. Upgrades that touch the query-log schema produce this one, and the file they set aside is a healthy database. |
+
+There is no fourth case. A healthy file carrying a schema version this build cannot use is neither migrated nor renamed: the startup refuses and the file stays where it is, which is [the query-log lifecycle](query-log-lifecycle.md). You can still find a `querylog.db.schema-changed-<unix-seconds>` in a data directory, because 0.0.13 and older produced one on any schema change — and still do, if you downgrade to one of them. This build never writes that name.
 
 Only the main file is renamed — its `-wal` and `-shm` are deleted, because a stale WAL would be replayed into the fresh database. A missing `querylog.db` is created without any aside file. The rename happens inside `querylog_schema.open`, before the 0600 chmod, and that chmod names `querylog.db` and its two sidecars only — so an aside file keeps the mode the file had at rename time, which for a `querylog.db` nxdns itself created is 0600 and for one an operator put there is whatever they left it at. Nothing prunes the aside files; they accumulate until an operator removes them, and each one holds the same browsing history the live query log holds.
 
@@ -67,6 +70,10 @@ Only the main file is renamed — its `-wal` and `-shm` are deleted, because a s
 
 The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash and `querylog.db` holds the browsing history of every client on the LAN, so both are as sensitive as each other, and a WAL file holds the same rows as the database it belongs to. SQLite creates the main database at `0644 & ~umask`; nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather than being created world-readable.
 
+A `pre-migrate` backup holds that same browsing history, so it is chmodded 0600 the way the live file is: SQLite's `VACUUM INTO` creates it at `0644 & ~umask` and nxdns restricts it immediately afterwards. A backup it cannot restrict is a failed backup — the partial file is deleted and the startup refuses with `MigrationBackupFailed`, rather than leaving a world-readable copy behind.
+
+The aside files are the exception: they get no chmod at all, and an aside keeps whatever mode it had at rename time. For a `querylog.db` nxdns itself created that is 0600; for one an operator put there it is whatever they left it at.
+
 ## The configuration file
 
 There is no default path. `--config FILE` names the file, and without that flag no file is read at all — a `config.zon` sitting in `/etc/nxdns` that no invocation names is inert. `/etc/nxdns/config.zon` is a convention the packaging follows, not a location nxdns probes.

docs/reference/query-log-lifecycle.md

@@ -0,0 +1,70 @@
+# The query log's lifecycle
+
+What happens to `querylog.db` when nxdns opens it: how the file is versioned, when it is migrated, when the server refuses to start over it, and the one case in which it is still replaced. Source of truth: `src/storage/querylog_versions.zig` (the version metadata), `src/storage/querylog_schema.zig` (the open path) and `src/storage/querylog_migrations.zig` (the migration runner).
+
+The rule this page exists to state: **a healthy `querylog.db` is never replaced and never moved aside.** A schema this build cannot use refuses the startup instead. Your query history is not the server's to discard.
+
+## The version stamp
+
+Every `querylog.db` carries a logical schema version in SQLite's `PRAGMA user_version`. It is a small counter — 1 in this release — and not a hash of anything. A file created by this build is stamped as it is created.
+
+Two other values matter, both in `querylog_versions.zig`:
+
+| Constant | Today | What it means |
+| --- | --- | --- |
+| `current_version` | 1 | The version this build creates and reads. |
+| `minimum_supported_version` | 1 | The oldest stamped version this build can migrate up to `current_version`. |
+| `legacy_fingerprint` | 1975011655 | The `user_version` the 0.0.12 and 0.0.13 binaries wrote: a CRC32 of their schema text, under the older policy where a mismatch meant "replace the file". |
+
+`legacy_fingerprint` is frozen forever. Those two releases stamped a hash rather than a version, so this build recognises that one literal number as "version 1" and restamps the file as 1 on the first open. The restamp runs in its own transaction; if it fails, the old stamp and every row stay exactly as they were and the startup refuses.
+
+## What an open does
+
+nxdns opens `querylog.db` once at startup, before it serves anything, and no second process shares a data directory. On a file that is readable and passes `PRAGMA quick_check`, the stamp decides:
+
+| Stamp | What happens |
+| --- | --- |
+| `current_version` | Opens. Nothing is migrated. |
+| `legacy_fingerprint` | Read as version 1: restamped to 1, then treated as version 1 by the rows above and below. |
+| Between `minimum_supported_version` and `current_version` | Migrated in place, then opens. |
+| Above `current_version`, up to 1000000 | REFUSE: `SchemaTooNew`. |
+| Anything else — 0, a negative, another fingerprint, a version below the minimum | REFUSE: `SchemaUnsupported`. |
+
+A refusal changes nothing. The schema, the rows, the coverage watermark and the stamp are all left as they are, no file is set aside, no new file is created, and `nxdns run` exits. The log line names the path, the stamp it found, the range this build supports and [the troubleshooting section](../how-to/troubleshoot.md#the-server-refuses-to-start-over-querylogdb).
+
+## Migrating in place
+
+A migration is one backup and one transaction.
+
+1. **Back up.** `VACUUM INTO` writes a complete copy — including anything still only in the write-ahead log — to `querylog.db.pre-migrate-<unix-seconds>` beside the database. If that name is taken, `-2`, `-3` and so on are tried. A backup that cannot be written is `MigrationBackupFailed`, and the partial copy is deleted; an older backup beside it survives.
+2. **Migrate.** `BEGIN IMMEDIATE`, re-read the stamp under the lock, run every step, run `PRAGMA foreign_key_check`, stamp the new version, `COMMIT`. One transaction covers the whole chain, so the file is either at the old version or at the new one and never in between.
+3. **Clean up.** Every other `querylog.db.pre-migrate-*` beside the file is deleted. **One backup is kept**: the one this migration just took. A later successful start retries that cleanup if it failed.
+
+If a step fails before the commit, the transaction rolls back, this run's backup is deleted, and the startup refuses with `MigrationFailed`. The database keeps the version and the rows it had.
+
+If the commit succeeds and something after it fails, the log says so plainly — the migration DID complete and the file IS at the new version. The backup is kept, the startup still refuses with `MigrationFailed`, and the next start opens the migrated file normally.
+
+## Corruption is the only automatic recreate
+
+Four conditions still create a fresh, empty `querylog.db`: the file is missing, SQLite reports it as corrupt, it is not a SQLite database at all, or `PRAGMA quick_check` does not answer `ok`. Except for the missing case, the unusable file is renamed to `querylog.db.<reason>-<unix-seconds>` and kept. See [why a query log is moved aside](files-and-directories.md#why-a-query-log-is-moved-aside).
+
+Every other failure — a lock held elsewhere, a permission problem, a full disk, a version this build cannot reach — propagates and leaves the file alone.
+
+## Downgrading
+
+**Downgrading to 0.0.13 or older resets your query log.** Those binaries predate this contract: they compare `user_version` against a hash of their own schema text, find this build's version stamp instead, and treat that as a mismatch — so they rename `querylog.db` to `querylog.db.schema-changed-<unix-seconds>` and start an empty log. Nothing is destroyed, but the live log is empty until you put the aside file back, and the restamp that provoked it takes no backup of its own.
+
+To recover, go back to a migration-aware release, stop the server, then, in the data directory:
+
+1. Move the empty `querylog.db` the old binary created out of the way.
+2. Delete its `querylog.db-wal` and `querylog.db-shm`. This is not optional: replaying the empty file's write-ahead log into the restored history would corrupt it.
+3. Rename `querylog.db.schema-changed-<unix-seconds>` back to `querylog.db`.
+4. Start the server.
+
+Downgrading between two migration-aware releases is safe in the sense that matters: a build that finds a stamp above its own `current_version` refuses to start with `SchemaTooNew` and touches nothing. Go forward again, or restore the `pre-migrate` backup the upgrade left.
+
+## Breaking the schema on purpose
+
+A release may still break the query-log schema outright rather than migrate it. That is allowed, and it is never silent. Such a release raises `current_version`, sets `minimum_supported_version` to the same value, and ships no migration step — so files from before the break classify as below the minimum and `open` refuses them with `SchemaUnsupported` rather than replacing them. The release notes carry the phrase `resets your query history` and a `Restoring your query history` section, and the cut gate refuses to build the release without both.
+
+So the contract is: a break is always versioned, always refused at startup with the file intact, and always disclosed in the changelog.