overview: one endpoint, live projections and a response cache (m36)

specs/milestone-36.md

@@ -0,0 +1,417 @@
+# Milestone 36: Overview performance — combined endpoint, projections, cache
+
+Replace the five per-panel stats endpoints with one `GET /api/overview` served
+from materialized projections in `querylog.db` plus an in-memory response
+cache, so Overview cost stops growing with query-log size.
+
+## Motivation (measured)
+
+Today each Overview load runs five separate scans of every raw row in the
+window, serialized on `WebState.querylog_lock`, re-polled every 30 s. Measured
+x86 ReleaseSafe (bench at scratchpad `statsbench2/`, production-like skew;
+Pi ≈ 3–3.5× slower):
+
+| Rows | 30d, five scans (today) | 30d, one combined scan | 30d, projections |
+|-----:|------------------------:|-----------------------:|-----------------:|
+| 1M   | ~2.2 s                  | 264 ms                 | 41 ms            |
+| 3M   | ~6.6 s                  | 793 ms                 | 38 ms            |
+| 5M   | ~11.8 s                 | 1,346 ms               | 40 ms            |
+
+Projection maintenance costs +10% per 100-row insert batch, and ~1.5 MB of
+disk in the bench — a size bounded by retained buckets × distinct
+client/type/route keys, independent of raw query volume. Production is on a ~100k rows/day growth
+curve (≈3M rows at 30-day retention), so the projection path is the design
+target, not a contingency. Both computations were cross-checked for identical
+output in the bench.
+
+Design ruling (owner + Codex consultation, 2026-08-27): stay on SQLite;
+projections live in the same file as the raw rows and are updated in the same
+transaction, so SQLite's transaction is the coherence mechanism — no second
+file, no epoch protocol. The DDL change re-fingerprints `querylog.db`; the
+existing rename-aside path handles old files (one-time history reset,
+disclosed in the changelog). No backfill migration.
+
+## Sessions
+
+Four sessions. A first. B and C after A, in parallel (disjoint files). D after B.
+
+- A: storage — projection schema, writer maintenance, retention, new read path.
+  A does NOT delete the five existing aggregate functions — `stats.zig` still
+  calls them until B lands, and A must leave `zig build test` green.
+- B: web — `/api/overview` handler, response cache, removal of the five old
+  endpoints, OpenAPI/contract regeneration.
+- C: admin — one overview query, types, component/data plumbing, tests.
+- D: storage cleanup — delete the five now-unreferenced aggregate functions.
+
+---
+
+## Session A: storage
+
+### A.1 Schema (src/storage/querylog_schema.zig)
+
+Append four projection tables to `ddl`. Grain: 30-minute buckets, `bucket` =
+floor-to-grid of the row timestamp: `@divFloor(timestamp, 1800) * 1800` in Zig
+and the equivalent floor semantics in any SQL (SQLite integer `/` truncates
+toward zero, which differs on negative timestamps — use floor everywhere, as
+`window()` does). 1800 divides every serving
+width ≥ 30 min (1800, 3600, 21600), which is what makes one grain serve the
+24h, 7d and 30d windows exactly. The 1h window (60 s buckets) is NOT served
+from projections (A.4).
+
+```sql
+CREATE TABLE bucket_totals (
+    bucket INTEGER PRIMARY KEY,
+    queries INTEGER NOT NULL,
+    blocked INTEGER NOT NULL,
+    cached INTEGER NOT NULL,
+    rt_sum INTEGER NOT NULL,     -- sum(response_time_us) over timed rows
+    rt_count INTEGER NOT NULL    -- count(response_time_us)
+) WITHOUT ROWID;
+
+CREATE TABLE bucket_clients (
+    bucket INTEGER NOT NULL,
+    client_ip TEXT NOT NULL,
+    queries INTEGER NOT NULL,
+    PRIMARY KEY (bucket, client_ip)
+) WITHOUT ROWID;
+
+CREATE TABLE bucket_types (
+    bucket INTEGER NOT NULL,
+    qtype INTEGER NOT NULL,      -- -1 encodes a NULL qtype, losslessly
+    count INTEGER NOT NULL,
+    PRIMARY KEY (bucket, qtype)
+) WITHOUT ROWID;
+
+CREATE TABLE bucket_routes (
+    bucket INTEGER NOT NULL,
+    route_kind TEXT NOT NULL,
+    source_present INTEGER NOT NULL,  -- 0: source NULL; 1: source = source_text
+    source_text TEXT NOT NULL,        -- '' when source_present = 0
+    count INTEGER NOT NULL,
+    PRIMARY KEY (bucket, route_kind, source_present, source_text),
+    CHECK (source_present IN (0, 1)),
+    CHECK (source_present = 1 OR source_text = '')
+) WITHOUT ROWID;
+```
+
+Column semantics match the existing aggregates exactly: `blocked` counts
+`blocked <> 0`; `cached` counts `cache_hit = 1`; routes' `source` is the
+existing CASE (`upstream` rows → `upstream`, `forward_zone` rows →
+`forward_zone`, else NULL). The fingerprint moves automatically; do not touch
+the fingerprint machinery.
+
+### A.2 Writer maintenance (src/storage/repositories/queries_repo.zig)
+
+`BatchWriter.writeBatch` updates all four projections inside the same
+transaction that inserts the raw rows:
+
+- Aggregate the batch in Zig first, producing per-key deltas; then one UPSERT
+  per touched key:
+  `INSERT ... ON CONFLICT(...) DO UPDATE SET queries = queries + excluded.queries, ...`.
+  The aggregation must accept any slice length — `writeBatch`'s API does not
+  enforce the logger's 100-row batching, so no fixed-size arrays sized to it.
+  `BatchWriter` currently owns no allocator: `init` gains one, owned for the
+  writer's life, used only for the per-batch delta maps; scratch is freed (or
+  a retained map cleared) at the end of every `writeBatch`, and
+  `error.OutOfMemory` fails the batch before the transaction opens — no
+  hidden global allocator, no implicit size cap, no quadratic rescanning.
+- No per-row SQL, no triggers.
+- Failure contract: any failed projection statement rolls the whole
+  transaction back — raw rows and projections together — resets every
+  projection statement, and leaves the writer usable for the next batch
+  (`resetAll` discipline as for the raw statements today). Fault-injection
+  acceptance: a batch whose projection update fails leaves the database
+  unchanged, and the next batch succeeds.
+- The bench measured this at 0.39 ms vs 0.34 ms per batch — acceptance is
+  correctness, not speed.
+
+### A.3 Retention (src/storage/repositories/queries_repo.zig, prune path)
+
+In the same transaction as `pruneOlderThan(cutoff)`'s raw delete:
+
+1. Delete projection rows with `bucket < floor(cutoff / 1800) * 1800` from all
+   four tables.
+2. If `cutoff` is not on a bucket boundary, recompute the straddling bucket
+   (`floor(cutoff/1800)*1800`) from the remaining raw rows and replace its
+   projection rows in all four tables. Never approximate.
+
+Failure atomicity: a failure during the projection delete or the
+straddling-bucket replacement rolls back the raw delete, the watermark
+advance and every projection change together — one transaction, tested by
+fault injection.
+
+Implementation note (Session A, recorded post-build): the bucket_totals
+recompute carries `HAVING count(*) > 0` — a bare SQL aggregate always yields
+one row, and an emptied straddling bucket must disappear, not persist as
+zeros.
+
+### A.4 Read path (src/storage/repositories/queries_repo.zig)
+
+One function producing the whole Overview payload for a window, from one
+already-open read transaction (the caller owns transaction + lock, as today):
+
+```zig
+pub const Overview = struct {
+    totals: StatsTotals,
+    buckets: []const Bucket,          // bucket_count entries, zero-filled
+    clients: ClientsBreakdown,        // top-8 + other, as today
+    types: []const TypeCount,         // sorted as stats_types_sql sorts
+    routes: []const RouteCount,       // sorted as stats_routes_sql sorts
+};
+
+pub fn overview(
+    database: *db.Db,
+    arena: Allocator,
+    since: i64,
+    bucket_seconds: u32,
+    bucket_count: u32,
+) db.Error!Overview
+```
+
+Storage owns these scalars — no import of any web module. `until` is derived
+as `since + bucket_seconds * bucket_count` with the same overflow checks as
+`timeseries`. Preconditions, checked before path selection and tested:
+`bucket_seconds != 0` and `bucket_count != 0` (else `error.Misuse`, matching
+the existing clients contract); on the projection path
+(`bucket_seconds >= 1800`) additionally `since` a multiple of 1800 and
+`bucket_seconds % 1800 == 0`, else `error.Misuse`. The handler's `window()`
+guarantees all of them.
+
+Two implementations behind one entry point, chosen by `bucket_seconds`:
+
+- `bucket_seconds >= 1800` (24h, 7d, 30d): read the four projection tables
+  over `[since, until)`, aggregating 30-min rows up to the serving width in
+  Zig. `distinct_clients` comes from grouping `bucket_clients` by `client_ip`
+  over the window — never from summing per-bucket counts.
+  `avg_response_time_us` = `sum(rt_sum) / sum(rt_count)`, null when
+  `rt_count` sums to 0. Top-8 clients ranked by window total desc, ties by
+  `client_ip` asc (BINARY), residual summed into `other` — identical cut
+  semantics to `statsClients`.
+- `bucket_seconds < 1800` (1h): one single pass over the raw rows in the
+  window (one SELECT of the needed columns, stepped once), aggregating
+  everything in Zig. Memory bound: O(distinct clients + distinct qtypes +
+  distinct routes) in the window — explicitly permitted; this is a household
+  LAN and the same bound the arena-returning aggregates already carry. This
+  replaces today's five scans and the clients rank+bucket double scan. Same
+  output contracts.
+
+Sort orders and tie-breaks must reproduce the existing SQL orderings exactly
+(types: count desc, null last within tie, qtype asc; routes: count desc,
+route_kind asc, null source last, source asc) — the goldens' byte-stability
+argument carries over. Note: `RouteKind`'s enum declaration order is not
+alphabetical; "route_kind asc" means the stored text's byte order, so any Zig
+comparator orders by `@tagName` bytes, never by enum ordinal (Session A's
+accumulator already does; mutation-tested).
+
+### A.5 Acceptance criteria
+
+- [ ] `zig build test` green.
+- [ ] Property test: after an arbitrary interleaving of batches and prunes
+      (including a prune cutoff off the bucket grid), every projection table
+      equals a from-scratch recomputation from `query_log`.
+- [ ] Equivalence test: `overview()` output (both paths) equals a test-only
+      oracle over the same window on the same data — including empty windows,
+      NULL qtype, NULL source on an `upstream` row, ties in ranking, and a
+      window whose last bucket is in progress. The oracle is a copy of the
+      five existing SQL aggregates living in the test file, so it survives
+      Session D's deletion of the production functions.
+- [ ] Fingerprint test updated (table/index count assertions in
+      querylog_schema tests).
+
+---
+
+## Session B: web
+
+### B.1 Endpoint (src/web/handlers/overview.zig, replacing stats.zig's five)
+
+`GET /api/overview?period=1h|24h|30d|7d` (same grammar, default 24h, same 400
+text). One read transaction under `WebState.querylog_lock` covering the
+aggregate and `coverage.read` — one snapshot, no cross-panel skew. Response:
+
+```json
+{
+  "period": "24h", "since": ..., "until": ..., "bucket_seconds": 1800,
+  "totals": { "queries": n, "blocked": n, "clients": n, "avg_response_time_us": n|null },
+  "buckets": [ { "ts": ..., "queries": n, "blocked": n, "cached": n }, ... ],
+  "clients": [ { "client": "ip", "buckets": [n, ...] }, ... ],
+  "other": [n, ...],
+  "types": [ { "qtype": n|null, "count": n }, ... ],
+  "routes": [ { "route": "...", "source": "..."|null, "count": n }, ... ],
+  "coverage": { ... }
+}
+```
+
+Field shapes and semantics are exactly today's five bodies merged; `Period`,
+`window()`, `max_buckets` move to (or stay importable from) the new handler.
+503 when the query log is unavailable; 500 logging unchanged. Route metadata
+identical to the removed endpoints: same authentication (`.session`), same
+rate-limit class (`.counted`), same authority policy (`.read`).
+
+Remove `GET /api/stats`, `/api/stats/timeseries`, `/api/stats/types`,
+`/api/stats/routes`, `/api/stats/clients` and their routes.
+
+Contract surface (B owns all of it): add the new path and schema to the
+OpenAPI document, remove the five old operations and their schemas, update
+every drift guard that lists them, add a contract sample for
+`/api/overview`, and regenerate `admin/src/lib/contractSamples.gen.ts`
+(reserved for B — Session C must not touch it). Update the API listings in
+`docs/` and `PLAN.md` that name the five endpoints or the querylog layout.
+
+### B.2 Response cache (src/web/server.zig WebState + overview.zig)
+
+Per-period cached response body, invalidated by data change or window roll.
+Key: `(period, window.until, data_version)` where `data_version` is `PRAGMA
+data_version` on the web task's connection (it changes when any other
+connection — logger, retention — commits).
+
+The entire cache decision happens under `querylog_lock`; nothing touches the
+shared connection or the slots outside it. Exact sequence per request:
+
+1. Acquire `querylog_lock` — ONCE. `server.QuerylogRead.open` acquires this
+   lock itself, so the overview handler must not call it after step 1: B
+   refactors the scope into a lock-owning wrapper plus a
+   locked-caller variant (for example `QuerylogRead.openLocked`, documented
+   as requiring the lock), and the overview path uses the locked-caller
+   variant for step 4. A literal "lock, then QuerylogRead.open" deadlocks.
+2. Sample `PRAGMA data_version` (inside the lock — the shared connection may
+   otherwise have a foreign transaction open, and the slots need the mutual
+   exclusion anyway).
+3. Hit (`slot.period == period and slot.until == window.until and
+   slot.data_version == sampled`): copy the stored bytes into the request
+   arena, release the lock, respond. The copy is what makes a concurrent
+   rebuild's free-and-replace safe.
+4. Miss: open the read transaction, build the body, commit. Publish to the
+   slot ONLY after a successful commit, keyed by the version sampled in
+   step 2 (a commit landing during the build bumps `data_version`, so the
+   next request rebuilds — stale-under-new-key is impossible). A failed
+   commit or build publishes nothing and responds 500 as today.
+5. Copy to the request arena, release the lock, write the socket. The lock
+   never spans a socket write (existing discipline).
+
+Because the check happens only under the lock, `querylog_lock` is the
+single-flight: a second request for the same key waits and then hits.
+
+Storage: one slot per period (4 slots) in `WebState`; body bytes allocated
+from `WebState.gpa`, replaced on rebuild (free old, install new), freed in
+`deinit`. No capacity limit beyond the allocator — a body is bounded by the
+fixed bucket counts plus the household client/type/route cardinality.
+
+No adaptive polling and no combined-endpoint staging: with projections + this
+cache a rebuild is ~40 ms x86 / ~0.13 s Pi, so the admin's existing 30 s
+cadence is fine.
+
+### B.3 Acceptance criteria
+
+- [ ] `zig build test` green; handler tests ported from stats.zig (period
+      grammar, window math, one-snapshot behavior) plus: cache hit returns
+      byte-identical body; a logger commit (data_version bump) invalidates;
+      a window roll invalidates; a retention prune committed through another
+      connection invalidates (both the aggregates and the cached
+      `coverage.available_since` are replaced); a failed read-transaction
+      commit neither installs nor replaces a cache entry.
+- [ ] `curl /api/overview?period=30d` on a seeded scratch instance returns all
+      panels consistent (breakdowns sum to totals on a quiet database).
+- [ ] The five old routes return 404.
+
+---
+
+## Session C: admin
+
+### C.1 Data layer
+
+- `admin/src/lib/types.ts`: one `Overview` type mirroring B.1; remove the five
+  per-panel response types.
+- `admin/src/lib/api.ts`: `getOverview(period)`; remove the five getters.
+- `admin/src/lib/queries.ts`: `overviewQuery(period)` with
+  `refetchInterval: 30_000` and key `["overview", period]`; remove the five
+  stats query factories and their keys.
+
+### C.2 Overview page
+
+`admin/src/features/overview/overviewWindow.ts` and the chart components
+consume the single query: one `useQuery` where five ran in parallel. Loading,
+error and coverage handling collapse to one page-level surface (one spinner
+state, one error state for the whole Overview); the per-panel shells,
+layout, copy, chart dimensions and accessibility attributes stay exactly as
+they are. Chart components (TimeseriesChart, ClientChart, Donut) keep their
+props — adapt the mapping layer, not the charts. C must not touch
+`contractSamples.gen.ts` (B owns its regeneration).
+
+### C.3 Acceptance criteria
+
+- [ ] `npm test` green in admin/ (mock the one endpoint; port the five-query
+      tests).
+- [ ] `npm run typecheck` green — the build alone does not run tsc. B and C
+      are file-disjoint but type-coupled through `contractSamples.gen.ts`:
+      C's typecheck/test/build gates run (or re-run) AFTER B has regenerated
+      that file. Implementation may proceed in parallel; the green gate is
+      sequenced.
+- [ ] `npm run build` green; bundle-size assertion still passes.
+- [ ] Manual: scratch instance renders all Overview panels from the new
+      endpoint on all four periods.
+
+---
+
+## Session D: storage cleanup
+
+After B is merged and green: delete `statsTotals`, `timeseries`, `statsTypes`,
+`statsRoutes`, `statsClients` and their SQL constants from
+`queries_repo.zig` — nothing references them once `stats.zig` is gone. The
+test-only oracle from A.5 stays. Acceptance: `zig build test` green, no dead
+stats SQL remains in production code.
+
+---
+
+## Module layout
+
+- `src/web/handlers/overview.zig` — new; replaces `src/web/handlers/stats.zig`
+  (deleted by B).
+- `src/storage/querylog_schema.zig` — projection DDL appended.
+- `src/storage/repositories/queries_repo.zig` — writer maintenance, retention
+  integration, `overview()` read path (A); five old aggregates deleted (D).
+- Admin files per C.1/C.2.
+
+## File ownership
+
+- A: `src/storage/*` (old aggregates left in place), plus the mechanical
+  allocator-plumbing at every `BatchWriter.init` call site outside storage
+  (`src/web/handlers/*`, `src/web/web_integration_test.zig`, any test using
+  the writer) — A runs before B, so this is sequential, not shared,
+  ownership; A's gate is the full `zig build test`.
+- B: `src/web/*`, plus: `src/app.zig` (cache cleanup at the composition
+  root), `src/tests.zig` (handler import swap), the OpenAPI document and its
+  drift guards, contract samples including
+  `admin/src/lib/contractSamples.gen.ts`, `docs/**` API listings, `PLAN.md`
+  stale sections, and the Overview/API sections of `specs/ui-redesign.md`
+  (which still mandates the five endpoints and must be amended, not obeyed).
+- C: `admin/*` EXCEPT `admin/src/lib/contractSamples.gen.ts`.
+- D: `src/storage/repositories/queries_repo.zig` (sequential, after B).
+- Orchestrator: `CHANGELOG.md` (hand-written, per release process).
+B and C run in parallel; the one shared-tree exception above is reserved to B.
+
+## Acceptance criteria (milestone complete)
+
+- [ ] `zig build test` and `zig build test -Dintegration` green (the
+      integration suite carries the live route walk, contract-sample
+      comparison, concurrent querylog reads and OpenAPI guards); admin
+      `npm test`/`typecheck`/`build` green.
+- [ ] Scratch-instance smoke: seeded data + live digs; Overview correct on all
+      periods; old endpoints gone.
+- [ ] Changelog discloses: schema change resets query history (rename-aside),
+      five endpoints replaced by `/api/overview`.
+- [ ] Release gate (`zig build cut` fingerprint check) satisfied.
+
+## Anti-requirements
+
+- No second database file, no epoch/validity protocol, no ATTACH.
+- No backfill migration, no rebuild command, no catch-up cursor — projections
+  are born with the file and maintained transactionally; that is the whole
+  coherence story.
+- No connection pool, no adaptive polling, no DuckDB.
+- No new indexes on `query_log`, no triggers, no per-row projection SQL.
+- Do not change the 1h/24h/7d/30d period grammar, bucket widths or counts.
+- No HTTP-level caching of any kind: no ETag, no `Cache-Control`, no
+  stale-while-revalidate, no background refresh, and never cache a 500/503
+  body. The cache is exactly the in-process design of B.2.
+- No visual redesign, no chart-prop changes, no cache configuration knobs,
+  no cache metrics. This milestone changes data acquisition and storage only.