30 lines
3.4 KiB
Markdown
30 lines
3.4 KiB
Markdown
# Query-log batching: one transaction per interval, not per query
|
||
|
||
Measured on the deployed Pi: ~0.5 GiB/day of process writes to persist ~2.3 MB/day of query rows, because the writer's 100 ms batch window degenerates to one transaction per query at household rates (~0.25 qps). Write wear killed the previous SD card. Fix: widen the window, Pi-hole precedent (FTL DBinterval=60).
|
||
|
||
## Contract
|
||
|
||
- New config field `logging.query_log_flush_interval_s: u16`, default 60, valid 0–3600, dedicated validation classification (not folded into an existing one).
|
||
- `0` means: do not wait for more entries; immediately flush the entry plus whatever is already queued, up to `flush_batch` rows. It is NOT per-query power-loss durability — `synchronous=NORMAL` never promised that; say so in the reference.
|
||
- Writer loop (`src/storage/logger.zig`): block on `getOne`, then deadline = now + interval on the **`.boot` clock** (matching upstream history; `.awake` would stretch the window across suspend), fill until deadline or `flush_batch` (100) rows, one transaction per flush; if the queue holds more, keep flushing in `flush_batch` chunks.
|
||
- Shutdown (the Codex-found race, fix required): today `Logger.shutdown` closes the queue and the app then cancels the task group containing the writer, so cancellation can beat the close-observation and drop the held batch. New order: stop and join query producers first, close the queue, await the writer outside the cancelled group (give the writer a lifetime separate from the producers' group). If the disk gate forbids the final write, drain and count the entries as dropped — never hang shutdown, never lose them uncounted.
|
||
- Unchanged on purpose: queue cap `query_log_buffer_max` + drop-oldest backpressure, `flush_batch`, pragmas, wal_autocheckpoint, retention, domains interning. One commit per minute makes those second-order.
|
||
|
||
## Propagation (config field checklist)
|
||
|
||
Model key + round-trip drift guards, validation + validation reference, settings API view (`src/web/handlers/settings.zig`), openapi.yaml, admin types + settings control, contract samples regenerated, docs config reference, CHANGELOG.
|
||
|
||
## Documentation wording
|
||
|
||
- Crash-loss window: up to `interval` seconds of query history on process failure; power loss can additionally lose recent committed transactions (WAL + synchronous=NORMAL). Query history is the least valuable data on the box.
|
||
- Staleness: every query-log-backed read (query-log page, totals, timeseries) lags up to `interval` seconds. The live view is unaffected — it is fed from the hub before the queue.
|
||
|
||
## Follow-up
|
||
|
||
The "unchanged on purpose" line above no longer holds for `wal_autocheckpoint`. Batching landed and the Pi measured 0.281 GiB/day, of which ~130 MiB is autocheckpoint writeback — second-order next to one transaction per query, first-order next to one per minute. specs/querylog-autocheckpoint.md raises the threshold to 8192 pages on every read-write `querylog.db` connection and states the durability contract that comes with it. Nothing else in this spec changes.
|
||
|
||
## Acceptance
|
||
|
||
- [ ] Deterministic tests: interval batching (entries within the window land in one transaction), flush_batch early flush, 0-sentinel immediate flush, shutdown drains a held batch and the queue (the race sequence: producers stopped → queue closed → writer awaited), disk-gated final drain counts drops.
|
||
- [ ] Suites: `zig build test` and `-Dintegration` 0 failed; admin typecheck/vitest/prettier clean; goldens regenerated.
|