query log batching: one transaction per flush interval, not per query

CHANGELOG.md

@@ -4,14 +4,21 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
 
 Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
 
-## [Unreleased]
+## [0.0.7] - 2026-08-20
+
+Operational failures get a page of their own, and the query log stops wearing out the disk it lives on: the deployed Pi was writing half a gigabyte a day to store two megabytes of query rows, one transaction per query. Both came out of running 0.0.6 on real hardware.
 
 ### Added
 
 - **A diagnostics page.** Operational failures now land in one curated log instead of only journald: blocklist download failures, certificate reload failures, disk pressure, query-log writer and maintenance failures, upstream exchange and history failures, client tracking failures, listener and configuration problems at boot, and the query-log recreation an upgrade causes. One entry per failing subject — an entry opens on the first failure, counts repeats, and closes itself when the subject recovers; nothing needs dismissing. Each entry says what it means for the service and what to do about it. `GET /api/diagnostics` serves the log, `GET /api/health` reports the active counts and degrades while the diagnostics store itself cannot write, and `/metrics` gains `nxdns_diagnostics_active_warnings`, `nxdns_diagnostics_active_errors` and `nxdns_diagnostics_write_failures_total`. Resolved entries can be purged when you decide the history has served its purpose — one entry from its row or its detail page, or the whole resolved history at once with "Purge all resolved" (`DELETE /api/diagnostics/{id}` and `DELETE /api/diagnostics`). An entry that is still failing is the current state of the box, not history, so it has no purge action and the API answers 409.
 
+### Changed
+
+- **The query log commits once a minute instead of once a query.** The writer batched for 100 milliseconds, which at a household's query rate means almost every query got a transaction of its own — and a transaction costs the disk far more than the row it carries. On the deployed Pi that came to roughly 0.5 GiB of writes a day to store 2.3 MB of query rows, the kind of write volume that kills an SD card. The batch window is now `logging.query_log_flush_interval_s`: 60 seconds by default (the same minute Pi-hole's `DBinterval` defaults to, for the same reason), anything from 0 to 3600, editable on the settings page. Batches are still capped at 100 rows, so a burst is committed as soon as it fills one rather than waiting out the window, and the in-memory queue, its drop-oldest backpressure and retention are untouched. The price is two kinds of lag: a crash costs about one interval of query history — more if the writer was held back by a full disk or a slow write — and every query-log-backed view — the query-log page, the dashboard totals, the timeseries — is about one interval behind. The live page is not affected; it is fed before the queue. Set the key to `0` for the old write-immediately behavior.
+
 ### Fixed
 
+- **Shutdown no longer races the last query rows to the disk.** The query-log writer was stopped by the same cancellation that stopped the DNS listeners, so whether the batch it was holding reached the database depended on which happened to land first, the cancellation or the queue closing. Shutdown now stops and joins the listeners and every other query producer first, then closes the queue, then waits for the writer to finish emptying it — the held batch and everything still queued get written. If free space is below the critical threshold and the disk monitor will not let that final write through, the rows are counted as dropped instead of holding the exit open indefinitely.
 - **An upstream success rate no longer rounds up to 100.0% while failures stand.** One decimal place cannot hold 12,696 successes out of 12,698 attempts: it rounded to `100.0%`, so the row claimed perfect reliability next to a failure count of 2. Neither end of the scale is reachable by rounding any more — `100.0%` needs an actual absence of failures and `0.0%` an actual absence of successes, and a rate a hair off either end shows `99.9%` or `0.1%` instead.
 - **A query log set aside by a schema change is no longer named `corrupt`.** Every recreate wrote the old file to `querylog.db.corrupt-<unix seconds>`, whatever sent it there — including the fingerprint mismatch an upgrade causes, where the file is a healthy database this build simply cannot read. The name is the only account of the reason that outlives the log line, so it read as an accusation and invited operators to delete an intact file. The name now says which of the four cases it hit: `querylog.db.corrupt-…`, `.not-a-database-…`, `.quick-check-failed-…` or `.schema-changed-…`. The 0.0.6 upgrade produces `schema-changed`. Nothing else about the recreate changed, and no existing aside file is renamed.
 

PLAN.md

@@ -503,7 +503,7 @@ CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
 ### 11.4 Query Logger
 
 - In-memory buffer, mutex guarded, hard cap `query_log_buffer_max` (default 10000).
-- Flush: batch size (default 100) or max interval (default 100ms).
+- Flush: batch size (100, comptime) or max interval `query_log_flush_interval_s` (default 60s, 0–3600, 0 = do not wait). One transaction per interval: at household query rates a per-query commit costs orders of magnitude more disk writes than the rows are worth. The interval is also roughly what a crash costs, while the writer is healthy and the disk gate is open — a gated or lock-delayed batch is older, so it is a normal case, not a bound.
 - Privacy transforms (hide_domains / hide_client_ips) applied before persist + SSE fanout.
 - Backpressure: buffer full → drop oldest unflushed entry, increment monotonic `queries_dropped` (exposed in `/api/health` + `/metrics`). SSE fanout precedes buffer insert, so live viewers still see dropped-from-persistence entries.
 

admin/src/features/settings/SettingsPage.test.tsx

@@ -33,6 +33,7 @@ function baseSettings(): Settings {
 			level: "info",
 			retention_days: 30,
 			query_log_buffer_max: 10000,
+			query_log_flush_interval_s: 60,
 			hide_domains: false,
 			hide_client_ips: false,
 			output: "stderr",

admin/src/features/settings/SettingsPage.tsx

@@ -116,6 +116,7 @@ const SECTIONS: readonly AnySectionDef[] = [
 			{ key: "level", kind: ["error", "warn", "info", "debug"] },
 			{ key: "retention_days", kind: "number" },
 			{ key: "query_log_buffer_max", kind: "number" },
+			{ key: "query_log_flush_interval_s", kind: "number" },
 			{ key: "hide_domains", kind: "boolean" },
 			{ key: "hide_client_ips", kind: "boolean" },
 			{ key: "output", kind: ["stderr", "syslog", "file"] },

admin/src/features/settings/authority.test.tsx

@@ -38,6 +38,7 @@ function baseSettings(): Settings {
 			level: "info",
 			retention_days: 30,
 			query_log_buffer_max: 10000,
+			query_log_flush_interval_s: 60,
 			hide_domains: false,
 			hide_client_ips: false,
 			output: "stderr",

admin/src/lib/contractSamples.gen.ts

@@ -557,6 +557,7 @@ export const sample_get_settings: SettingsEnvelope = {
 		"logging.level",
 		"logging.retention_days",
 		"logging.query_log_buffer_max",
+		"logging.query_log_flush_interval_s",
 		"logging.hide_domains",
 		"logging.hide_client_ips",
 		"logging.output",
@@ -618,6 +619,7 @@ export const sample_get_settings: SettingsEnvelope = {
 			max_size_mb: 0,
 			output: "stderr",
 			query_log_buffer_max: 0,
+			query_log_flush_interval_s: 0,
 			retention_days: 0,
 		},
 		upstream: {
@@ -680,6 +682,7 @@ export const sample_put_settings: SettingsEnvelope = {
 		"logging.level",
 		"logging.retention_days",
 		"logging.query_log_buffer_max",
+		"logging.query_log_flush_interval_s",
 		"logging.hide_domains",
 		"logging.hide_client_ips",
 		"logging.output",
@@ -741,6 +744,7 @@ export const sample_put_settings: SettingsEnvelope = {
 			max_size_mb: 0,
 			output: "stderr",
 			query_log_buffer_max: 0,
+			query_log_flush_interval_s: 0,
 			retention_days: 0,
 		},
 		upstream: {

admin/src/lib/settingsDiff.test.ts

@@ -25,6 +25,7 @@ function baseSettings(): Settings {
 			level: "info",
 			retention_days: 30,
 			query_log_buffer_max: 10000,
+			query_log_flush_interval_s: 60,
 			hide_domains: false,
 			hide_client_ips: false,
 			output: "stderr",

admin/src/lib/types.ts

@@ -469,6 +469,7 @@ export interface Settings {
 		level: "error" | "warn" | "info" | "debug";
 		retention_days: number;
 		query_log_buffer_max: number;
+		query_log_flush_interval_s: number;
 		hide_domains: boolean;
 		hide_client_ips: boolean;
 		output: "stderr" | "syslog" | "file";

docs/reference/configuration.md

@@ -132,6 +132,7 @@ Process log and query log behavior.
 | `logging.level` | enum `.err` \| `.warn` \| `.info` \| `.debug` | `.info` | — | one of the four tags; stored as `"error"` / `"warn"` / `"info"` / `"debug"` | log threshold (`src/platform/logging.zig`) |
 | `logging.retention_days` | u16 | 30 | days | at least 1 | query-log pruning cutoff (`src/storage/retention.zig`) and the client tracker's last-seen cutoff (`src/server/clients.zig`) |
 | `logging.query_log_buffer_max` | u32 | 10000 | entries | 1–1000000 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`) |
+| `logging.query_log_flush_interval_s` | u16 | 60 | seconds | 0–3600 | how long the query-log writer gathers entries before committing them in one transaction (`src/storage/logger.zig`); see the note below |
 | `logging.hide_domains` | bool | false | — | — | the query log stores a hidden marker instead of the domain |
 | `logging.hide_client_ips` | bool | false | — | — | the query log stores a hidden marker instead of the client address |
 | `logging.output` | enum `.stderr` \| `.syslog` \| `.file` | `.stderr` | — | one of the three tags | log sink selection (`src/platform/logging.zig`); `.stderr` and `.syslog` both write to stderr (journald captures it), `.file` rotates |
@@ -139,6 +140,19 @@ Process log and query log behavior.
 | `logging.max_size_mb` | u32 | 50 | MiB | at least 1 | rotation trigger for the log file |
 | `logging.max_files` | u8 | 5 | files | at least 1 | log files kept in total, the live one included, so the highest rotated generation is `max_files - 1`; the default 5 keeps `nxdns.log` plus `nxdns.log.1` through `nxdns.log.4`, and a value of 1 keeps only the live file, which rotation deletes rather than renames |
 
+#### What `query_log_flush_interval_s` costs and buys
+
+The query-log writer commits one transaction per interval instead of one per query. At a household's query rate — a few queries a second at most, often a fraction of one — a per-query commit writes hundreds of times more bytes to the disk than the rows themselves occupy, because every commit rewrites the WAL frames, the WAL index and the page headers whatever the row size. That write amplification is what wears out an SD card, and the default of 60 seconds is what stops it. It matches Pi-hole's `DBinterval`, which defaults to the same minute for the same reason.
+
+What it costs:
+
+- **Crash-loss window.** A process that dies takes roughly `interval` seconds of query history with it. That is the normal case, not a guaranteed maximum: a batch the disk monitor is holding back (free space below the critical threshold) or one waiting on a database write lock can be considerably older when the process dies. Power loss can additionally lose recent committed transactions, because `querylog.db` runs with WAL and `synchronous=NORMAL` — that was already true at any interval, and setting `0` does not buy per-query durability. Query history is the least valuable data on this box: nothing else depends on it, and it is deleted by retention anyway.
+- **Staleness.** Every read backed by the query log — the query-log page, the dashboard totals, the timeseries — lags about `interval` seconds behind, and further behind while writes are gated or slow. The live view does not lag: it is fed from the SSE hub before the queue, so queries appear there the moment they are answered.
+
+`0` means "do not wait": the writer commits the entry that woke it together with whatever is already queued, up to 100 rows. Use it when you want the query-log page to be current to the second and you do not care what that costs the disk.
+
+Two things do not change with the interval: a batch is capped at 100 rows, so a burst is committed as soon as it fills one rather than waiting out the window, and shutdown writes what the writer is holding instead of waiting for the interval to end.
+
 ### disk
 
 Free-space thresholds for the data directory. Below them the query-log writer, the client tracker and the blocklist scheduler are throttled (`src/storage/disk_monitor.zig`); DNS resolution is never gated.
@@ -384,6 +398,7 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
 | `BadTtl` | `blocking.ttl`, `cache.negative_ttl_max`, a record `ttl`, `web.session_ttl_hours` or `blocklist_update.interval_hours` outside its range |
 | `BadCacheSize` | `cache.size` outside 1–1000000 |
 | `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 1–1000000 |
+| `BadFlushInterval` | `logging.query_log_flush_interval_s` above 3600 |
 | `BadLogRotation` | `logging.max_size_mb` or `logging.max_files` below 1 |
 | `BadDiskThresholds` | a threshold below 1, or `min_free_mb` above `warn_free_mb` |
 | `BadRateLimit` | `dns.rate_limit`, `dns.rate_window_seconds`, `web.api_rate_limit_per_min` or `web.sse_max_connections_per_ip` out of range |

specs/querylog-batching.md

@@ -0,0 +1,25 @@
+# Query-log batching: one transaction per interval, not per query
+
+Measured on the deployed Pi: ~0.5 GiB/day of process writes to persist ~2.3 MB/day of query rows, because the writer's 100 ms batch window degenerates to one transaction per query at household rates (~0.25 qps). Write wear killed the previous SD card. Fix: widen the window, Pi-hole precedent (FTL DBinterval=60).
+
+## Contract
+
+- New config field `logging.query_log_flush_interval_s: u16`, default 60, valid 0–3600, dedicated validation classification (not folded into an existing one).
+  - `0` means: do not wait for more entries; immediately flush the entry plus whatever is already queued, up to `flush_batch` rows. It is NOT per-query power-loss durability — `synchronous=NORMAL` never promised that; say so in the reference.
+- Writer loop (`src/storage/logger.zig`): block on `getOne`, then deadline = now + interval on the **`.boot` clock** (matching upstream history; `.awake` would stretch the window across suspend), fill until deadline or `flush_batch` (100) rows, one transaction per flush; if the queue holds more, keep flushing in `flush_batch` chunks.
+- Shutdown (the Codex-found race, fix required): today `Logger.shutdown` closes the queue and the app then cancels the task group containing the writer, so cancellation can beat the close-observation and drop the held batch. New order: stop and join query producers first, close the queue, await the writer outside the cancelled group (give the writer a lifetime separate from the producers' group). If the disk gate forbids the final write, drain and count the entries as dropped — never hang shutdown, never lose them uncounted.
+- Unchanged on purpose: queue cap `query_log_buffer_max` + drop-oldest backpressure, `flush_batch`, pragmas, wal_autocheckpoint, retention, domains interning. One commit per minute makes those second-order.
+
+## Propagation (config field checklist)
+
+Model key + round-trip drift guards, validation + validation reference, settings API view (`src/web/handlers/settings.zig`), openapi.yaml, admin types + settings control, contract samples regenerated, docs config reference, CHANGELOG.
+
+## Documentation wording
+
+- Crash-loss window: up to `interval` seconds of query history on process failure; power loss can additionally lose recent committed transactions (WAL + synchronous=NORMAL). Query history is the least valuable data on the box.
+- Staleness: every query-log-backed read (query-log page, totals, timeseries) lags up to `interval` seconds. The live view is unaffected — it is fed from the hub before the queue.
+
+## Acceptance
+
+- [ ] Deterministic tests: interval batching (entries within the window land in one transaction), flush_batch early flush, 0-sentinel immediate flush, shutdown drains a held batch and the queue (the race sequence: producers stopped → queue closed → writer awaited), disk-gated final drain counts drops.
+- [ ] Suites: `zig build test` and `-Dintegration` 0 failed; admin typecheck/vitest/prettier clean; goldens regenerated.

src/app.zig

@@ -921,11 +921,31 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
     // while a task could still touch it.
     var group: std.Io.Group = .init;
 
+    // The gate every non-essential write consults. Reading it before the
+    // monitor's own task has sampled is safe: a fresh `Monitor` publishes `.ok`
+    // (disk_monitor.zig:63), so nothing is refused for want of a sample.
+    const gate: ?*disk_monitor.Monitor = &monitor;
+
+    // The query-log writer is deliberately *not* in `group`, and starts before
+    // every producer. Inside the group its life would end with the same
+    // `cancel` that stops the producers, and cancellation would race the
+    // queue's close: whichever landed first decided whether the batch the
+    // writer was holding reached the database or was counted as dropped. Given
+    // its own future, it outlives the producers by construction, and the
+    // teardown below can close the queue with nobody left to fill it and then
+    // wait for the writer to finish emptying it.
+    var writer_future = try io.concurrent(
+        logger_mod.Logger.runWriter,
+        .{ &query_logger, io, &querylog_writer_db, gate },
+    );
+
     // Ruling 4's shutdown order, on the one path every exit from here takes:
-    // the logger sees a closed queue and drains what it holds rather than
-    // losing it to cancellation (ruling 22), then every task stops, and only
-    // then does the final flush run — with no recording task left that could
-    // add a cell after it.
+    // every producer stops and is joined, then the queue closes, then the
+    // writer is awaited — so the last batch is written rather than raced — and
+    // only then does the final history flush run, with no recording task left
+    // that could add a cell after it. A writer the disk gate will not let write
+    // counts its batch as dropped instead of holding the exit open
+    // (`logger.zig`), so this wait always ends.
     //
     // A `defer` and not straight-line code after `shutdown.wait`, because a
     // `concurrent` spawn below can fail with the DNS listeners already
@@ -933,25 +953,12 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
     // signal gets. The `querylog_history_db` this flush writes through is
     // declared above, so its `close` runs after it.
     defer {
-        query_logger.shutdown(io);
         group.cancel(io);
+        query_logger.shutdown(io);
+        writer_future.await(io) catch {};
         history.flushOnce(io, &querylog_history_db, upstream_history_repo.flush);
     }
 
-    // The gate every non-essential write consults. Reading it before the
-    // monitor's own task has sampled is safe: a fresh `Monitor` publishes `.ok`
-    // (disk_monitor.zig:63), so nothing is refused for want of a sample.
-    const gate: ?*disk_monitor.Monitor = &monitor;
-
-    // The writer starts before the listeners, and that order is the deferred
-    // drain's precondition: a listener that is already accepting queries
-    // enqueues log entries, and `Logger.shutdown` only closes the queue —
-    // someone has to be on the other end to write what it hands over. Spawned
-    // after the listeners, a `concurrent` failure in between would leave those
-    // entries with no consumer and `group.cancel` nothing to drain, which is
-    // exactly the loss the teardown above exists to prevent.
-    try group.concurrent(io, logger_mod.Logger.runWriter, .{ &query_logger, io, &querylog_writer_db, gate });
-
     if (udp6) |*s| try group.concurrent(io, udp_server.UdpServer.serve, .{ s, io });
     if (udp4) |*s| try group.concurrent(io, udp_server.UdpServer.serve, .{ s, io });
     if (tcp6) |*s| try group.concurrent(io, tcp_server.TcpServer.serve, .{ s, io });

src/config/model.zig

@@ -231,6 +231,10 @@ pub const Logging = struct {
     level: LogLevel = .info,
     retention_days: u16 = 30,
     query_log_buffer_max: u32 = 10000,
+    /// How long the query-log writer gathers entries before it commits them.
+    /// `0` does not wait at all: it flushes the entry that woke the writer plus
+    /// whatever is already queued.
+    query_log_flush_interval_s: u16 = 60,
     hide_domains: bool = false,
     hide_client_ips: bool = false,
     output: LogOutput = .stderr,
@@ -565,6 +569,7 @@ const expected_keys = [_][]const u8{
     "logging.max_size_mb",
     "logging.output",
     "logging.query_log_buffer_max",
+    "logging.query_log_flush_interval_s",
     "logging.retention_days",
     "upstream.attempt_timeout_ms",
     "upstream.read_timeout_ms",
@@ -657,6 +662,7 @@ test "toSettings and fromSettings round-trip a non-default config" {
             .level = .err,
             .retention_days = 41,
             .query_log_buffer_max = 43,
+            .query_log_flush_interval_s = 44,
             .hide_domains = true,
             .hide_client_ips = true,
             .output = .file,

src/config/validate.zig

@@ -93,6 +93,7 @@ pub const ValidateError = error{
     BadTtl,
     BadCacheSize,
     BadRetention,
+    BadFlushInterval,
     BadLogRotation,
     BadDiskThresholds,
     BadRateLimit,
@@ -335,6 +336,11 @@ const max_rate_window_seconds = 3_600;
 /// the box, and nxdns does not try to know that.
 const max_boot_entries = 1_000_000;
 
+/// The ceiling on the query-log flush window. An hour of queries is already
+/// more history than a crash is allowed to cost; beyond that the setting stops
+/// being a batching knob and becomes a way to lose a working day of rows.
+const max_flush_interval_s = 3_600;
+
 fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void {
     const up = cfg.upstream;
     try checkTimeout(diags, up.attempt_timeout_ms, "upstream.attempt_timeout_ms");
@@ -464,6 +470,16 @@ fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void {
             .{ max_boot_entries, cfg.logging.query_log_buffer_max },
         );
     }
+    // No floor: 0 is the documented "do not wait" setting, not a mistake.
+    if (cfg.logging.query_log_flush_interval_s > max_flush_interval_s) {
+        try diags.add(
+            error.BadFlushInterval,
+            "logging.query_log_flush_interval_s",
+            .{},
+            "must be at most {d}, got {d}",
+            .{ max_flush_interval_s, cfg.logging.query_log_flush_interval_s },
+        );
+    }
     if (cfg.logging.max_size_mb < 1) {
         try diags.add(error.BadLogRotation, "logging.max_size_mb", .{}, "must be at least 1", .{});
     }
@@ -1946,6 +1962,22 @@ test "error.BadRetention" {
     try expectProblem(huge_buffer, error.BadRetention, "logging.query_log_buffer_max");
 }
 
+test "error.BadFlushInterval" {
+    var cfg = baseConfig();
+    cfg.logging.query_log_flush_interval_s = 3601;
+    try expectProblem(cfg, error.BadFlushInterval, "logging.query_log_flush_interval_s");
+
+    // 0 is the "do not wait" setting and 3600 is the ceiling itself: both are
+    // legal, and a floor check would reject the first.
+    var immediate = baseConfig();
+    immediate.logging.query_log_flush_interval_s = 0;
+    try expectClean(immediate);
+
+    var edge = baseConfig();
+    edge.logging.query_log_flush_interval_s = max_flush_interval_s;
+    try expectClean(edge);
+}
+
 test "error.BadLogRotation" {
     var cfg = baseConfig();
     cfg.logging.max_files = 0;

src/storage/logger.zig

@@ -13,9 +13,19 @@
 //! order; `QuerySink` calls them separately so both of its consumers see the
 //! one transformed entry.
 //!
+//! One transaction per `logging.query_log_flush_interval_s`, not one per query.
+//! At household rates a per-query commit costs orders of magnitude more disk
+//! writes than the rows are worth — a minute of batching is what keeps an SD
+//! card alive. The window is also what a crash costs: while the writer is
+//! healthy and the disk gate is open, a process that dies loses about one
+//! interval of query history. It is not a ceiling — a batch the gate is
+//! holding, or one waiting on a write lock, is older than that — and nothing
+//! here promises one, because query history is the cheapest data on the box.
+//!
 //! Log rows are expendable. A full queue drops the oldest unflushed entry, a
 //! failed batch is dropped whole, and a disk that crossed the critical
-//! threshold holds batches back indefinitely. Each of the three has a counter.
+//! threshold holds batches back until it recovers — or, if the process is
+//! already stopping, drops what it holds. Each of the three has a counter.
 //! A writer that cannot prepare its statements closes the queue and marks
 //! `writer_failed`, so the loss is visible rather than silent.
 
@@ -31,10 +41,10 @@ const queries_repo = @import("repositories/queries_repo.zig");
 /// and the two names collide inside the struct.
 const scope = std.log.scoped(.query_logger);
 
-/// Flush tuning is comptime: §12.1 defines no configuration keys for it and a
-/// household deployment has no reason to tune it.
+/// Rows per transaction. Comptime: the window, not the batch size, is what an
+/// operator has a reason to move, and a batch this size already amortizes the
+/// commit at any household rate.
 pub const flush_batch = 100;
-pub const flush_interval_ms = 100;
 
 /// What `hide_domains` and `hide_client_ips` store instead of the real value.
 pub const hidden_marker = "hidden";
@@ -161,6 +171,14 @@ fn emptyAsNull(value: []const u8) ?[]const u8 {
 
 const EntryQueue = std.Io.Queue(Entry);
 
+/// A database failure is not in here: a batch the database refuses is dropped,
+/// counted and reported where it happens, and the writer carries on.
+/// `GatedAtShutdown` is the one condition `flush` cannot settle by itself —
+/// the disk gate is shut and the process is stopping, so the batch in hand is
+/// lost and so is everything still queued behind it. Only `runWriter` can see
+/// both, so it does the counting and files the single episode.
+pub const FlushError = std.Io.Cancelable || error{GatedAtShutdown};
+
 /// What the flush interval race can produce. `Select` demands that each field
 /// type match its task's return type exactly.
 const Outcome = union(enum) {
@@ -178,6 +196,10 @@ pub const Logger = struct {
     /// closed and every entry counts as dropped from that point, so a caller
     /// that sees this must not expect rows.
     writer_failed: std.atomic.Value(bool),
+    /// Set by `shutdown`, read by the disk gate. The gate holds a batch for as
+    /// long as the disk stays critical, which is right while the process is
+    /// running and a hang once it is stopping — nothing will ever release it.
+    draining: std.atomic.Value(bool),
     /// Wired by the composition root after `init`, following the
     /// `gate: ?*disk_monitor.Monitor` idiom. Null in every unit test here.
     diagnostics: ?*events.Store = null,
@@ -195,6 +217,7 @@ pub const Logger = struct {
             .rows_written = .init(0),
             .batches_gated = .init(0),
             .writer_failed = .init(false),
+            .draining = .init(false),
         };
     }
 
@@ -246,6 +269,11 @@ pub const Logger = struct {
     /// whole life. Returns when `shutdown` closes the queue and the last batch
     /// is flushed, or when the task is canceled.
     ///
+    /// It must outlive the producers rather than share their lifetime: a
+    /// cancellation that races the close decides at random whether the batch in
+    /// hand is written or counted as dropped. `app.zig` spawns this outside the
+    /// group it cancels for exactly that reason.
+    ///
     /// `monitor` is the §11.6 gate. Null disables gating.
     pub fn runWriter(
         self: *Logger,
@@ -262,7 +290,7 @@ pub const Logger = struct {
             // life of the process, so the row stays active, which is the truth.
             self.reportWrite(io, "writer", "preparing the batch statements failed", @errorName(err), 0);
             self.queue.close(io);
-            self.dropRemaining(io);
+            _ = self.dropRemaining(io);
             return;
         };
         defer writer.deinit();
@@ -275,10 +303,7 @@ pub const Logger = struct {
                 error.Closed => return,
                 error.Canceled => |e| return e,
             };
-            const deadline: std.Io.Clock.Timestamp = .fromNow(io, .{
-                .raw = .fromMilliseconds(flush_interval_ms),
-                .clock = .awake,
-            });
+            const deadline = self.flushDeadline(io);
             // `n` is live across both calls: entries already taken off the
             // queue are lost if either one is canceled, so they must count.
             var n: usize = 1;
@@ -286,17 +311,55 @@ pub const Logger = struct {
                 self.countDropped(n);
                 return err;
             };
-            self.flush(io, &writer, batch[0..n], monitor) catch |err| {
+            self.flush(io, &writer, batch[0..n], monitor) catch |err| switch (err) {
+                error.Canceled => |e| {
                     self.countDropped(n);
-                return err;
+                    return e;
+                },
+                // Nothing will open the gate now. Everything still queued is
+                // lost with the batch in hand, so it is counted here and
+                // announced once — a per-chunk report would write to the very
+                // disk that is out of space, dozens of times, on the way out.
+                error.GatedAtShutdown => {
+                    self.countDropped(n);
+                    const lost = n + self.dropRemaining(io);
+                    scope.warn(
+                        "query log: {d} rows dropped at shutdown, the disk gate was closed",
+                        .{lost},
+                    );
+                    self.reportWrite(
+                        io,
+                        "batch",
+                        "query log rows were dropped at shutdown",
+                        "DiskCritical",
+                        lost,
+                    );
+                    return;
+                },
             };
         }
     }
 
-    /// Counts every entry left in a closed queue as dropped. The drain is
-    /// uncancelable: a cancellation racing the writer's own failure would
-    /// otherwise abandon the buffered entries without counting them.
-    fn dropRemaining(self: *Logger, io: std.Io) void {
+    /// When the batch that starts now must be committed. `.boot` and not
+    /// `.awake`: a suspended box would otherwise stretch the window by however
+    /// long it slept, and the rows are already in memory waiting.
+    ///
+    /// An interval of 0 yields a deadline that has already passed, which is
+    /// exactly the documented sentinel — `fill` then takes what is queued and
+    /// returns without waiting for anything.
+    fn flushDeadline(self: *const Logger, io: std.Io) std.Io.Clock.Timestamp {
+        return .fromNow(io, .{
+            .raw = .fromSeconds(self.cfg.query_log_flush_interval_s),
+            .clock = .boot,
+        });
+    }
+
+    /// Counts every entry left in a closed queue as dropped, and returns how
+    /// many. The drain is uncancelable: a cancellation racing the writer's own
+    /// failure would otherwise abandon the buffered entries without counting
+    /// them.
+    fn dropRemaining(self: *Logger, io: std.Io) usize {
+        var total: usize = 0;
         var leftover: [flush_batch]Entry = undefined;
         while (true) {
             const n = self.queue.getUncancelable(io, &leftover, 0) catch |err| switch (err) {
@@ -304,18 +367,24 @@ pub const Logger = struct {
             };
             if (n == 0) break;
             self.countDropped(n);
+            total += n;
         }
+        return total;
     }
 
-    /// Closes the queue. `log` drops from here on and `runWriter` returns once
-    /// it has flushed what was left.
+    /// Closes the queue and tells the writer to stop waiting on anything that
+    /// may never arrive. `log` drops from here on, and `runWriter` returns once
+    /// it has flushed what was left — a blocked `getOne` on a closed queue
+    /// returns immediately, so the interval is never waited out at shutdown.
     ///
-    /// A writer held by the disk gate keeps holding: it flushes when the disk
-    /// recovers, and the `group.cancel` that follows this call in `app.zig`
-    /// stops a writer that will not wait. A canceled
-    /// writer counts the batch it holds under `queries_dropped`.
+    /// The close comes first: a writer that sees `draining` set must be able to
+    /// drain the queue to the end, and only a closed queue reports its end.
+    ///
+    /// Every producer must be stopped and joined before this is called
+    /// (`app.zig`): an entry enqueued after the close is a dropped entry.
     pub fn shutdown(self: *Logger, io: std.Io) void {
         self.queue.close(io);
+        self.draining.store(true, .release);
     }
 
     /// Fills `batch` behind the entry already in slot 0, until it is full or
@@ -387,7 +456,7 @@ pub const Logger = struct {
         writer: *queries_repo.BatchWriter,
         entries: []const Entry,
         monitor: ?*disk_monitor.Monitor,
-    ) std.Io.Cancelable!void {
+    ) FlushError!void {
         if (entries.len == 0) return;
 
         if (monitor) |m| {
@@ -396,6 +465,13 @@ pub const Logger = struct {
                 .clock = .awake,
             };
             while (!m.writesAllowed()) {
+                // Waiting for the disk to recover is right while the process
+                // runs and wrong once it is stopping: nothing is going to free
+                // space during shutdown, so the batch is lost either way and
+                // the only choice left is between losing it counted and
+                // hanging the exit. The counting and the one report belong to
+                // `runWriter`, which knows the rest of the queue is lost too.
+                if (self.draining.load(.acquire)) return error.GatedAtShutdown;
                 _ = self.batches_gated.fetchAdd(1, .monotonic);
                 try pause.sleep(io);
             }
@@ -691,7 +767,7 @@ test "log after shutdown drops instead of blocking" {
     try testing.expectEqual(@as(u64, 1), logger.queries_dropped.load(.monotonic));
 }
 
-test "the writer drains every entry and shutdown ends it" {
+test "shutdown writes the batch the writer holds and the rest of the queue" {
     var threaded: std.Io.Threaded = .init(testing.allocator, .{});
     defer threaded.deinit();
     const io = threaded.io();
@@ -700,7 +776,9 @@ test "the writer drains every entry and shutdown ends it" {
     defer database.close();
 
     var buf: [512]Entry = undefined;
-    var logger: Logger = .init(.{}, &buf);
+    // An hour: nothing here can be explained by the window expiring. Every row
+    // that lands does so because the close released it.
+    var logger: Logger = .init(.{ .query_log_flush_interval_s = 3600 }, &buf);
 
     var future = try io.concurrent(Logger.runWriter, .{
         &logger,
@@ -714,6 +792,10 @@ test "the writer drains every entry and shutdown ends it" {
         const written = try std.fmt.bufPrint(name, "d{d}.example", .{i % 10});
         logger.log(io, sampleEntry(@intCast(i), written));
     }
+
+    // The app.zig order: the only producer is this task and it is done, so the
+    // close cannot lose an entry, and the writer is awaited rather than
+    // canceled.
     logger.shutdown(io);
     try future.await(io);
 
@@ -723,7 +805,57 @@ test "the writer drains every entry and shutdown ends it" {
     try testing.expectEqual(@as(i64, 10), try queries_repo.countDomains(&database));
 }
 
-test "the writer flushes an entry once the interval passes" {
+test "entries that arrive inside one window reach the database in one batch" {
+    var threaded: std.Io.Threaded = .init(testing.allocator, .{});
+    defer threaded.deinit();
+    const io = threaded.io();
+
+    var database = try openLog();
+    defer database.close();
+    var writer = try queries_repo.BatchWriter.init(&database);
+    defer writer.deinit();
+
+    var buf: [16]Entry = undefined;
+    var logger: Logger = .init(.{ .query_log_flush_interval_s = 3600 }, &buf);
+
+    for (0..6) |i| logger.log(io, sampleEntry(@intCast(i), "batched.example"));
+    // No further producer exists, so the close is what ends the fill — the same
+    // thing that ends it at shutdown.
+    logger.shutdown(io);
+
+    var batch: [flush_batch]Entry = undefined;
+    batch[0] = try logger.queue.getOne(io);
+    var n: usize = 1;
+    try logger.fill(io, &batch, logger.flushDeadline(io), &n);
+    try testing.expectEqual(@as(usize, 6), n);
+
+    // One `writeBatch` call, which is one transaction (`queries_repo.zig`).
+    try logger.flush(io, &writer, batch[0..n], null);
+    try testing.expectEqual(@as(u64, 6), logger.rows_written.load(.monotonic));
+    try testing.expectEqual(@as(i64, 6), try queries_repo.countRows(&database));
+}
+
+test "a zero interval takes what is queued and waits for nothing" {
+    var threaded: std.Io.Threaded = .init(testing.allocator, .{});
+    defer threaded.deinit();
+    const io = threaded.io();
+
+    var buf: [8]Entry = undefined;
+    var logger: Logger = .init(.{ .query_log_flush_interval_s = 0 }, &buf);
+
+    logger.log(io, sampleEntry(1, "now.example"));
+    logger.log(io, sampleEntry(2, "now.example"));
+
+    var batch: [flush_batch]Entry = undefined;
+    batch[0] = try logger.queue.getOne(io);
+    var n: usize = 1;
+    // The queue is open and the batch has room: any non-zero interval blocks
+    // here until it expires. Zero returns with what was already queued.
+    try logger.fill(io, &batch, logger.flushDeadline(io), &n);
+    try testing.expectEqual(@as(usize, 2), n);
+}
+
+test "the writer holds an entry for the length of the flush interval" {
     var threaded: std.Io.Threaded = .init(testing.allocator, .{});
     defer threaded.deinit();
     const io = threaded.io();
@@ -732,7 +864,7 @@ test "the writer flushes an entry once the interval passes" {
     defer database.close();
 
     var buf: [8]Entry = undefined;
-    var logger: Logger = .init(.{}, &buf);
+    var logger: Logger = .init(.{ .query_log_flush_interval_s = 1 }, &buf);
 
     var future = try io.concurrent(Logger.runWriter, .{
         &logger,
@@ -743,20 +875,71 @@ test "the writer flushes an entry once the interval passes" {
 
     logger.log(io, sampleEntry(1, "only.example"));
 
+    // Sampled, not asserted, while the writer runs: an assertion that fails
+    // here would return before the writer is stopped, and `Threaded.deinit`
+    // then waits on a task nothing will ever end.
+    const quarter: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(250), .clock = .awake };
+    try quarter.sleep(io);
+    const written_at_a_quarter = logger.rows_written.load(.monotonic);
+
+    // Three times the interval. A flush that has not happened by then is a
+    // failure, not slowness.
     const poll: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(5), .clock = .awake };
     var waited: usize = 0;
-    while (logger.rows_written.load(.monotonic) == 0) : (waited += 1) {
-        // Ten times the interval; a flush that has not happened by then is a
-        // failure, not slowness.
-        try testing.expect(waited < 200);
+    while (logger.rows_written.load(.monotonic) == 0 and waited < 600) : (waited += 1) {
         try poll.sleep(io);
     }
 
     logger.shutdown(io);
     try future.await(io);
+
+    // A writer that commits per query has already written at a quarter of the
+    // window; this one has not.
+    try testing.expectEqual(@as(u64, 0), written_at_a_quarter);
+    try testing.expect(waited < 600);
+    try testing.expectEqual(@as(u64, 1), logger.rows_written.load(.monotonic));
     try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(&database));
 }
 
+test "a full batch flushes without waiting for the interval" {
+    var threaded: std.Io.Threaded = .init(testing.allocator, .{});
+    defer threaded.deinit();
+    const io = threaded.io();
+
+    var database = try openLog();
+    defer database.close();
+
+    var buf: [256]Entry = undefined;
+    var logger: Logger = .init(.{ .query_log_flush_interval_s = 3600 }, &buf);
+
+    var future = try io.concurrent(Logger.runWriter, .{
+        &logger,
+        io,
+        &database,
+        @as(?*disk_monitor.Monitor, null),
+    });
+
+    for (0..150) |i| logger.log(io, sampleEntry(@intCast(i), "burst.example"));
+
+    // The window is an hour away, so only a full batch can release a flush.
+    const poll: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(5), .clock = .awake };
+    var waited: usize = 0;
+    while (logger.rows_written.load(.monotonic) < flush_batch and waited < 400) : (waited += 1) {
+        try poll.sleep(io);
+    }
+    // Sampled with the writer still running, asserted once it has stopped.
+    const written_before_shutdown = logger.rows_written.load(.monotonic);
+
+    logger.shutdown(io);
+    try future.await(io);
+
+    // Exactly one batch went out early, and the 50 behind it waited for the
+    // close rather than for the hour.
+    try testing.expect(waited < 400);
+    try testing.expectEqual(@as(u64, flush_batch), written_before_shutdown);
+    try testing.expectEqual(@as(i64, 150), try queries_repo.countRows(&database));
+}
+
 test "a gated flush holds the batch until the disk recovers" {
     var threaded: std.Io.Threaded = .init(testing.allocator, .{});
     defer threaded.deinit();
@@ -867,7 +1050,9 @@ test "a canceled writer counts the batch it was holding" {
     defer database.close();
 
     var buf: [8]Entry = undefined;
-    var logger: Logger = .init(.{}, &buf);
+    // Zero interval: the writer reaches the gate with what is queued instead of
+    // waiting a minute for a producer that no longer exists.
+    var logger: Logger = .init(.{ .query_log_flush_interval_s = 0 }, &buf);
 
     var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
     monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
@@ -898,6 +1083,67 @@ test "a canceled writer counts the batch it was holding" {
     try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
 }
 
+test "a disk-gated writer drops what it holds at shutdown instead of hanging" {
+    var threaded: std.Io.Threaded = .init(testing.allocator, .{});
+    defer threaded.deinit();
+    const io = threaded.io();
+
+    var database = try openLog();
+    defer database.close();
+
+    var fx: events_fixture.Fixture = .{};
+    try fx.init(io, 1000);
+    defer fx.deinit();
+
+    var buf: [512]Entry = undefined;
+    var logger: Logger = .init(.{ .query_log_flush_interval_s = 0 }, &buf);
+    logger.diagnostics = &fx.store;
+
+    var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
+    monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
+
+    logger.log(io, sampleEntry(1, "held.example"));
+    logger.log(io, sampleEntry(2, "held.example"));
+
+    var future = try io.concurrent(Logger.runWriter, .{
+        &logger,
+        io,
+        &database,
+        @as(?*disk_monitor.Monitor, &monitor),
+    });
+
+    const poll: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(5), .clock = .awake };
+    var waited: usize = 0;
+    while (logger.batches_gated.load(.monotonic) == 0 and waited < 400) : (waited += 1) {
+        try poll.sleep(io);
+    }
+
+    // Two and a half chunks wait behind the batch the gate is holding, so a
+    // drain that reported per chunk would file three episodes' worth of writes
+    // to the disk that is out of space.
+    const queued = 250;
+    for (0..queued) |i| logger.log(io, sampleEntry(@intCast(i + 3), "queued.example"));
+
+    // The disk never recovers. The wait still ends, and every entry is counted.
+    logger.shutdown(io);
+    try future.await(io);
+
+    try testing.expect(waited < 400);
+    try testing.expectEqual(@as(u64, queued + 2), logger.queries_dropped.load(.monotonic));
+    try testing.expectEqual(@as(u64, 0), logger.rows_written.load(.monotonic));
+    try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
+
+    // One episode, reported once, and its detail carries the whole loss rather
+    // than the size of whichever chunk was last.
+    try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
+    try testing.expectEqual(@as(i64, 1), try fx.count("SELECT occurrences FROM operational_events"));
+    try testing.expectEqualStrings("batch", try fx.text("SELECT subject_key FROM operational_events"));
+    try testing.expectEqualStrings(
+        "query log rows were dropped at shutdown: DiskCritical (252 rows)",
+        try fx.text("SELECT detail FROM operational_events"),
+    );
+}
+
 test "an empty batch touches neither the database nor the counters" {
     var threaded: std.Io.Threaded = .init(testing.allocator, .{});
     defer threaded.deinit();

src/storage/phase6_integration_test.zig

@@ -228,7 +228,11 @@ test "S8 case 2: a single entry reaches the file once the flush interval passes"
     defer log_db.deinit();
 
     var queue_buf: [8]logger.Entry = undefined;
-    var query_log: logger.Logger = .init(.{}, &queue_buf);
+    // A one-second window, not the 60-second default: this case is about the
+    // entry reaching the file on the interval alone, with nothing to close the
+    // queue for it.
+    const flush_interval_s = 1;
+    var query_log: logger.Logger = .init(.{ .query_log_flush_interval_s = flush_interval_s }, &queue_buf);
 
     var future = try io.concurrent(logger.Logger.runWriter, .{
         &query_log,
@@ -239,9 +243,9 @@ test "S8 case 2: a single entry reaches the file once the flush interval passes"
 
     query_log.log(io, entryAt(1, "only.example"));
 
-    // Ten flush intervals of headroom: a row that has not landed by then is a
+    // Three flush intervals of headroom: a row that has not landed by then is a
     // failure of the interval race, not a slow machine.
-    const limit = 10 * logger.flush_interval_ms / 5;
+    const limit = 3 * flush_interval_s * 1000 / 5;
     try awaitCount(&query_log.rows_written, 1, limit);
     try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(log_db.database()));
 
@@ -268,7 +272,9 @@ test "S8 case 3: a full queue drops the oldest entries and the newest survive" {
     monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
 
     var queue_buf: [8]logger.Entry = undefined;
-    var query_log: logger.Logger = .init(.{}, &queue_buf);
+    // Zero interval: this case is about the queue cap and the gate, so the
+    // writer takes what is queued and goes straight to the gate.
+    var query_log: logger.Logger = .init(.{ .query_log_flush_interval_s = 0 }, &queue_buf);
 
     // The whole burst is enqueued before the writer starts. A writer already
     // draining the queue would take entries out of it mid-burst and make the
@@ -286,8 +292,9 @@ test "S8 case 3: a full queue drops the oldest entries and the newest survive" {
     try awaitCount(&query_log.batches_gated, 1, 200);
     try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(log_db.database()));
 
-    // Un-gate before the shutdown: a writer held by the disk gate holds its
-    // batch, and `shutdown` alone would never release it.
+    // Un-gate before the shutdown: a writer still held by the gate counts its
+    // batch as dropped rather than writing it, which is a different case (the
+    // logger's own suite covers it).
     monitor.state_raw.store(@intFromEnum(disk_monitor.State.ok), .monotonic);
     try awaitCount(&query_log.rows_written, 8, 300);
     query_log.shutdown(io);
@@ -403,7 +410,9 @@ test "S8 case 6: a critical disk gates the flushes and recovery releases them" {
     try testing.expect(monitor.gauges().db_bytes > 0);
 
     var queue_buf: [64]logger.Entry = undefined;
-    var query_log: logger.Logger = .init(.{}, &queue_buf);
+    // Zero interval: the case is the gate's hold and release, so the writer
+    // takes the five queued entries straight to it.
+    var query_log: logger.Logger = .init(.{ .query_log_flush_interval_s = 0 }, &queue_buf);
     for (0..5) |i| query_log.log(io, entryAt(@intCast(i), "gated.example"));
 
     var future = try io.concurrent(logger.Logger.runWriter, .{

src/web/handlers/settings.zig

@@ -203,6 +203,7 @@ const LoggingView = struct {
     level: []const u8,
     retention_days: u16,
     query_log_buffer_max: u32,
+    query_log_flush_interval_s: u16,
     hide_domains: bool,
     hide_client_ips: bool,
     output: []const u8,
@@ -263,6 +264,7 @@ pub fn view(cfg: model.Config) View {
             .level = cfg.logging.level.toDb(),
             .retention_days = cfg.logging.retention_days,
             .query_log_buffer_max = cfg.logging.query_log_buffer_max,
+            .query_log_flush_interval_s = cfg.logging.query_log_flush_interval_s,
             .hide_domains = cfg.logging.hide_domains,
             .hide_client_ips = cfg.logging.hide_client_ips,
             .output = cfg.logging.output.toDb(),

src/web/openapi.yaml

@@ -2503,13 +2503,14 @@ components:
               enum: [strip, forward]
         logging:
           type: object
-          required: [level, retention_days, query_log_buffer_max, hide_domains, hide_client_ips, output, file_path, max_size_mb, max_files]
+          required: [level, retention_days, query_log_buffer_max, query_log_flush_interval_s, hide_domains, hide_client_ips, output, file_path, max_size_mb, max_files]
           properties:
             level:
               type: string
               enum: [error, warn, info, debug]
             retention_days: { type: integer }
             query_log_buffer_max: { type: integer }
+            query_log_flush_interval_s: { type: integer }
             hide_domains: { type: boolean }
             hide_client_ips: { type: boolean }
             output:
@@ -2651,6 +2652,7 @@ components:
             level: { type: string }
             retention_days: { type: integer }
             query_log_buffer_max: { type: integer }
+            query_log_flush_interval_s: { type: integer }
             hide_domains: { type: boolean }
             hide_client_ips: { type: boolean }
             output: { type: string }

src/web/web_integration_test.zig

@@ -598,6 +598,7 @@ const SettingsView = struct {
             level: []const u8,
             retention_days: u16,
             query_log_buffer_max: u32,
+            query_log_flush_interval_s: u16,
             hide_domains: bool,
             hide_client_ips: bool,
             output: []const u8,