diff --git a/CHANGELOG.md b/CHANGELOG.md index a1d67cd..df253de 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -37,6 +37,7 @@ Query provenance: every logged query becomes exactly explainable — what the po - **Upgrading resets your query history.** The `query_log` table gains the provenance columns below, and `querylog.db` is never migrated (it holds expendable log rows, so a schema change replaces the file instead of upgrading it). On the first start after the upgrade the old file is set aside as `querylog.db.schema-changed-` and a fresh one is created. Nothing else is touched: `config.db` keeps your configuration and your diagnostics history. The recreate files a resolved `query_log.recreated` diagnostics entry naming the file that was kept and the timestamp the new history begins at, and a new `querylog_meta` table records that coverage start, so the dashboard can say "history is available from ..." instead of charting an empty range as zero. The set-aside file is a working SQLite database and can be deleted once you have decided you do not want it. - **`logging.query_log_buffer_max` now accepts 1 to 37449, down from 1 to 1000000.** The queued entry carries every new provenance field by value and is about four times as wide as before — 1792 bytes against 432 — so the meaningful bound is bytes rather than entries. The ceiling is computed at compile time from the width of the entry so that the queue's worst case stays within 64 MiB, and it moves whenever that width does. The default of 10000 is unchanged and costs about 17 MiB. A configuration above the new ceiling is rejected at startup with the ceiling in the message. - **Group and blocklist source names are now capped at 64 bytes.** Both are copied into every query-log row that mentions them, so an unbounded name was an unbounded cost per row. A longer name is rejected as `GroupNameTooLong` or `SourceNameTooLong`. +- **The query log returns to SQLite's default checkpoint cadence.** 0.0.8 stretched `wal_autocheckpoint` on every read-write `querylog.db` connection from the 1000-page default to 8192 pages, on the expectation that it would cut about 130 MiB a day of checkpoint writeback on the deployed Pi. Field measurement on that Pi showed no measurable effect on daily disk writes, so all it bought was a roughly five-hour power-loss durability window in place of the default's ~40 minutes. No pragma is issued any more: the cadence is SQLite's 1000 pages, about 4 MiB, and the ~40-minute boundary is back. ### Fixed diff --git a/specs/querylog-autocheckpoint.md b/specs/querylog-autocheckpoint.md deleted file mode 100644 index 55e2908..0000000 --- a/specs/querylog-autocheckpoint.md +++ /dev/null @@ -1,43 +0,0 @@ -# querylog.db: wal_autocheckpoint = 8192 - -One constant. The v0.0.7 batching cut process writes from ~0.5 to 0.281 GiB/day (measured over a 10 h process lifetime on the Pi); ~130 MiB/day of the remainder is autocheckpoint writeback — SQLite's 1000-page default trips every ~40 min and rewrites the same hot index/interior pages into the main db each time. At 8192 pages (32 MiB at the 4096-byte page size) the cadence drops to ~5 h, cutting those in-place rewrites ~8x, expected total ≈190 MiB/day. The previous SD card died of write wear; the current card's endurance is unknown, which argues for cutting known writes, not against it. Codex approved the decision and this shape (thread 01a0205d). - -## Decision - -`PRAGMA wal_autocheckpoint = 8192` on every read-write querylog.db connection. Hardcoded constant, no config knob, no checkpoint task. `synchronous=NORMAL` and the daily retention `wal_checkpoint(TRUNCATE)` (queries_repo.zig:169, called from the retention pass) stay as they are. config.db — including the diagnostics store's connection, which `app.zig:416` opens via `openConfigDb` despite the variable name `events_db` — keeps the SQLite default. There are exactly two database files; nothing named events.db exists. - -## Durability contract (goes in the constant's comment, stated precisely) - -- Commit never fsyncs at `synchronous=NORMAL`; the checkpoint's fsync is the only guaranteed durability boundary. This change moves that boundary from ~40 min to ~5 h of querylog data (query rows + upstream-history minutes) under power loss or kernel panic. Typical loss stays far smaller (kernel writeback), but that is not a guarantee. -- Process crash or clean stop loses nothing committed, at any threshold. Consistency is never at risk: recovery replays the longest valid WAL prefix atomically. -- 32 MiB is an expectation, not a cap: a pinned reader snapshot stops a passive checkpoint partway and the WAL overshoots until the reader finishes; the daily TRUNCATE is the backstop that shrinks the file. - -## Implementation - -1. **src/storage/db.zig** — `Pragmas` (:753) gains `wal_autocheckpoint_pages: ?i32 = null`. `applyPragmas` (:762), when non-null: `PRAGMA wal_autocheckpoint = N;` then read back via the pragma's own return and fail loudly on mismatch — mirror the `foreign_keys` set-and-verify at :781-783. Default null leaves every existing `.{}` caller (config.db sites, tests) untouched with zero diffs. db.zig stays generic; it must not know the word querylog. -2. **src/storage/querylog_schema.zig** — owns the constant (the module already owns querylog policy: fingerprint, DDL, recreate classification): `pub const wal_autocheckpoint_pages: i32 = 8192;` carrying the durability contract above as its comment. Passed at both production `applyPragmas` sites: the probe path (:129) and `createFresh` (:253). The third `applyPragmas` in that file (:273) is inside an in-memory test and stays `.{}` deliberately. -3. **src/cli.zig** — `reopenQuerylogDb` (:354) passes the constant. These three sites are the only read-write querylog connections by construction — every open flows through `querylog_schema.open` or `DataDir.reopenQuerylogDb`. - -## Tests - -- Unit, db.zig, in-memory (the pragma reads back per-connection regardless of journal mode): default `Pragmas` leaves `PRAGMA wal_autocheckpoint` at 1000; a set value reads back. -- File-backed storage integration test through the real helpers: `openQuerylogDb` and `reopenQuerylogDb` connections both read back 8192; an `openConfigDb` connection reads 1000. -- The read-back inside `applyPragmas` makes misapplication loud at startup, complementing both. - -## Docs - -- CHANGELOG Unreleased, Changed: the checkpoint cadence change, the measured why, and the widened power-loss window stated per the durability contract (not as an unconditional bound). -- One short paragraph appended to specs/querylog-batching.md linking here. - -## Rejected (do not relitigate without new facts) - -- Config knob: nobody tunes this twice; scope is small on purpose. -- Periodic checkpoint task: reproduces autocheckpoint with more moving parts (cadence state, busy handling, shutdown, diagnostics). -- `wal_autocheckpoint=0` + daily TRUNCATE only: unbounded intraday WAL growth under reader pinning; strictly worse. -- `journal_size_limit`: redundant with the daily TRUNCATE, and rejecting it needs no claim about passive checkpoints never truncating (after a completed checkpoint resets the WAL, the limit does truncate on the next write — the knob is merely surplus here). -- Touching config.db policy or differentiating reader vs writer querylog connections: readers cannot trip checkpoints, the pragma is inert on them; uniformity is simpler. - -## Gates - -1. `zig build test` and `-Dintegration` 0 failed; fmt clean. -2. Field verification on the released build (the Pi deploys releases, not branches, so this necessarily follows the cut — owner-ordered 2026-08-21): confirm the WAL resets normally, writeback falls materially, and no batches drop. What to measure and over what window is the deployment side's call; a bad result reverts the constant in a follow-up patch release. diff --git a/specs/querylog-batching.md b/specs/querylog-batching.md index 330c822..dbd4731 100644 --- a/specs/querylog-batching.md +++ b/specs/querylog-batching.md @@ -19,10 +19,6 @@ Model key + round-trip drift guards, validation + validation reference, settings - Crash-loss window: up to `interval` seconds of query history on process failure; power loss can additionally lose recent committed transactions (WAL + synchronous=NORMAL). Query history is the least valuable data on the box. - Staleness: every query-log-backed read (query-log page, totals, timeseries) lags up to `interval` seconds. The live view is unaffected — it is fed from the hub before the queue. -## Follow-up - -The "unchanged on purpose" line above no longer holds for `wal_autocheckpoint`. Batching landed and the Pi measured 0.281 GiB/day, of which ~130 MiB is autocheckpoint writeback — second-order next to one transaction per query, first-order next to one per minute. specs/querylog-autocheckpoint.md raises the threshold to 8192 pages on every read-write `querylog.db` connection and states the durability contract that comes with it. Nothing else in this spec changes. - ## Acceptance - [ ] Deterministic tests: interval batching (entries within the window land in one transaction), flush_batch early flush, 0-sentinel immediate flush, shutdown drains a held batch and the queue (the race sequence: producers stopped → queue closed → writer awaited), disk-gated final drain counts drops. diff --git a/src/cli.zig b/src/cli.zig index 061eaa6..dbe7d3c 100644 --- a/src/cli.zig +++ b/src/cli.zig @@ -351,9 +351,7 @@ pub const DataDir = struct { _ = io; var database = try db.Db.open(self.querylog_db_path, .{ .mode = .read_write_existing }); errdefer database.close(); - try db.applyPragmas(&database, .{ - .wal_autocheckpoint_pages = querylog_schema.wal_autocheckpoint_pages, - }); + try db.applyPragmas(&database, .{}); return database; } diff --git a/src/storage/db.zig b/src/storage/db.zig index f576f82..e500462 100644 --- a/src/storage/db.zig +++ b/src/storage/db.zig @@ -755,9 +755,6 @@ pub const Pragmas = struct { journal_wal: bool = true, synchronous_normal: bool = true, foreign_keys: bool = true, - /// Null leaves SQLite's 1000-page default. A caller that sets it owns the - /// durability consequences, which depend on what the database holds. - wal_autocheckpoint_pages: ?i32 = null, }; /// MUST be called before any transaction is opened: `PRAGMA foreign_keys` is a @@ -789,16 +786,6 @@ pub fn applyPragmas(self: *Db, p: Pragmas) Error!void { return error.SqliteError; } } - if (p.wal_autocheckpoint_pages) |pages| { - var buf: [64]u8 = undefined; - const sql = std.fmt.bufPrintZ(&buf, "PRAGMA wal_autocheckpoint = {d};", .{pages}) catch unreachable; - try self.exec(sql); - const applied = try self.queryInt("PRAGMA wal_autocheckpoint"); - if (applied != pages) { - log.warn("PRAGMA wal_autocheckpoint = {d} reported {d}", .{ pages, applied }); - return error.SqliteError; - } - } } /// A write transaction. @@ -1052,18 +1039,6 @@ test "applyPragmas succeeds and foreign_keys reads back as 1" { try testing.expectEqual(@as(i64, 1), try db.queryInt("PRAGMA foreign_keys")); } -test "applyPragmas leaves wal_autocheckpoint at the default unless a page count is given" { - var db = try openMemory(); - defer db.close(); - try applyPragmas(&db, .{}); - try testing.expectEqual(@as(i64, 1000), try db.queryInt("PRAGMA wal_autocheckpoint")); - - var configured = try openMemory(); - defer configured.close(); - try applyPragmas(&configured, .{ .wal_autocheckpoint_pages = 8192 }); - try testing.expectEqual(@as(i64, 8192), try configured.queryInt("PRAGMA wal_autocheckpoint")); -} - test "open on a directory path returns error.CantOpen and leaks no handle" { var i: usize = 0; while (i < 1000) : (i += 1) { diff --git a/src/storage/querylog_schema.zig b/src/storage/querylog_schema.zig index 6bc82ec..d2b86a8 100644 --- a/src/storage/querylog_schema.zig +++ b/src/storage/querylog_schema.zig @@ -92,31 +92,6 @@ pub const fingerprint: i32 = blk: { const set_user_version = std.fmt.comptimePrint("PRAGMA user_version = {d};", .{fingerprint}); -/// The WAL checkpoint threshold for every read-write `querylog.db` connection, -/// in pages (32 MiB at the 4096-byte page size). SQLite's 1000-page default -/// trips every ~40 min under this workload and rewrites the same hot index and -/// interior pages into the main database each time; 8192 stretches that to ~5 h -/// and cuts those in-place rewrites ~8x, which is SD-card write wear this -/// household appliance does not need to spend. -/// -/// The durability consequence, stated precisely: -/// -/// - Commit never fsyncs at `synchronous = NORMAL`; the checkpoint's fsync is -/// the only guaranteed durability boundary. This moves that boundary from -/// ~40 min to ~5 h of querylog data under power loss or kernel panic. -/// Typical loss stays far smaller because of kernel writeback, but that is -/// not a guarantee. -/// - Process crash or clean stop loses nothing committed, at any threshold. -/// Consistency is never at risk: recovery replays the longest valid WAL -/// prefix atomically. -/// - 32 MiB is an expectation, not a cap: a pinned reader snapshot stops a -/// passive checkpoint partway and the WAL overshoots until that reader -/// finishes. The daily retention `wal_checkpoint(TRUNCATE)` is the backstop -/// that shrinks the file. -/// -/// `config.db` keeps the SQLite default: it holds configuration, not a log. -pub const wal_autocheckpoint_pages: i32 = 8192; - /// Long enough for any path this program will be handed, plus the aside suffix. /// A longer path is `error.NameTooLong`, which is what the filesystem calls /// would have returned anyway. @@ -166,7 +141,7 @@ pub fn open(io: std.Io, dir: std.Io.Dir, path: [:0]const u8) Error!OpenResult { break :probe recreatable(e) orelse return e; const opened = &handle.?; - db.applyPragmas(opened, .{ .wal_autocheckpoint_pages = wal_autocheckpoint_pages }) catch |e| + db.applyPragmas(opened, .{}) catch |e| break :probe recreatable(e) orelse return e; const healthy = quickCheck(opened) catch |e| @@ -290,7 +265,7 @@ fn deleteSidecars(io: std.Io, dir: std.Io.Dir, path: []const u8) Error!void { fn createFresh(path: [:0]const u8) db.Error!db.Db { var database = try db.Db.open(path, .{ .mode = .read_write_create }); errdefer database.close(); - try db.applyPragmas(&database, .{ .wal_autocheckpoint_pages = wal_autocheckpoint_pages }); + try db.applyPragmas(&database, .{}); var tx = try db.Tx.begin(&database); errdefer tx.rollback(); diff --git a/src/storage/storage_integration_test.zig b/src/storage/storage_integration_test.zig index 621bd1f..e0e4e62 100644 --- a/src/storage/storage_integration_test.zig +++ b/src/storage/storage_integration_test.zig @@ -600,36 +600,6 @@ test "S7 case 23: a locked querylog propagates Busy and is never destroyed" { try testing.expectEqual(@as(usize, 0), asides_after.items.items.len); } -// --------------------------------------------------------------------------- -// the querylog checkpoint threshold (specs/querylog-autocheckpoint.md) -// --------------------------------------------------------------------------- - -test "every querylog connection carries the raised wal_autocheckpoint; config.db keeps the default" { - if (!build_options.integration) return error.SkipZigTest; - - var f: Fixture = .init(); - defer f.deinit(); - - var data = try openMigrated(&f, "data"); - defer data.deinit(); - - var opened = try data.dir.openQuerylogDb(io); - defer opened.database.close(); - try testing.expectEqual( - @as(i64, querylog_schema.wal_autocheckpoint_pages), - try opened.database.queryInt("PRAGMA wal_autocheckpoint"), - ); - - var reopened = try data.dir.reopenQuerylogDb(io); - defer reopened.close(); - try testing.expectEqual( - @as(i64, querylog_schema.wal_autocheckpoint_pages), - try reopened.queryInt("PRAGMA wal_autocheckpoint"), - ); - - try testing.expectEqual(@as(i64, 1000), try data.database.queryInt("PRAGMA wal_autocheckpoint")); -} - // --------------------------------------------------------------------------- // case 8-10: config.db, permissions and the schema stamp // ---------------------------------------------------------------------------