milestone 30: overview as a dashboard, explicit health contract, period aggregations

specs/milestone-30.md

@@ -0,0 +1,146 @@
+# Milestone 30: Overview replacement
+
+Redesign step 4 of specs/ui-redesign.md (§Overview, §API changes /api/health, §Deletions), amended by the owner's 2026-08-22 ruling: the dashboard becomes Overview **with Pi-hole's dashboard layout** (stat tiles, two charts, two donuts — no status or issues sections; see S3/S2). The health contract becomes explicit — nothing may degrade the rollup without appearing in the response. Codex design review folded in (thread 01a028b1); corrections marked where they changed the shape.
+
+**One amendment to the accepted design, made here and recorded in the changelog**: ui-redesign.md:113/203 lists an `upstream_history` health condition. After this milestone deletes the dashboard's upstream table, the upstream-minute history subsystem has no product consumer at all — a writer whose only reader is its own failure signal. The no-versioning rule forbids keeping it as a stub, so this milestone deletes the subsystem (S1.2) and the health contract has **five** conditions, not six.
+
+## Sessions
+
+S1 (done): Zig — health contract, subsystem deletion, contracts, docs. S3 (after S1): Zig — the three period aggregations. S2 (after S3): admin — the Pi-hole Overview, sidebar pause control, Diagnostics badge/strip, deletions, smoke. Sequential; **the milestone lands as one atomic commit** (Codex: S1 alone breaks the generated contract sample's TS assignment and every Health consumer — its red admin typecheck is an intra-milestone state, never a landed one).
+
+---
+
+## Session S1: the explicit health contract
+
+### S1.1 `GET /api/health` (src/web/handlers/health.zig)
+
+New body: `status` plus five condition objects. The degrading set is exactly (Codex: name it, don't gesture at "not ok"): protection `unavailable`, upstreams `unavailable`, query_history `losing`|`failed`, diagnostics `unavailable`, disk `low`|`critical`. Nothing else reaches `degraded()` — this kills the hidden `history_flush_failing` input (health.zig:65-73/177-178) along with its subsystem.
+
+| object | shape | state rule |
+| --- | --- | --- |
+| `protection` | `{state: "active"\|"paused"\|"unavailable", until: ?i64}` | Precedence `unavailable` → `paused` → `active` (Codex). `unavailable` ⇔ no current filter snapshot exists (the condition under which handler.zig answers with `snapshot_unavailable` provenance) — pause state is irrelevant then, `until` null. Else `paused` ⇔ the pause is live now (indefinite → until null; timed and unexpired → until set). **An expired timed pause is `active`** (Codex). Paused does NOT degrade — operator choice, surfaced not alarmed; unavailable degrades. |
+| `upstreams` | `{state: "ok"\|"unavailable", available, total}` | unavailable ⇔ available == 0. `total` = enabled routing candidates (the pool is built from enabled upstreams only — Codex); S2 labels it "of N enabled". |
+| `query_history` | `{state: "recording"\|"losing"\|"failed", dropped_total, last_drop_s: ?i64}` | failed ⇔ `writer_failed`. losing ⇔ a logger-owned **gating-episode state** (Codex critical: `draining` is a shutdown flag, false during normal disk gating): logger marks the episode active while the disk gate holds writes, flips to losing once a drop occurs during that episode, clears when the gate reopens. Event-driven; no time window. Else recording. Queue-overflow drops outside a gating episode do not flip the state (cumulative ≠ current); they surface via `dropped_total` + `last_drop_s` (atomic stamped per drop, null until one happens). |
+| `diagnostics` | `{state: "recording"\|"unavailable", active_warnings, active_errors}` | current rule (`!present or write_failed`) |
+| `disk` | `{state: "ok"\|"low"\|"critical", free_bytes}` | disk_monitor states; **warn→low renamed at the serialization boundary**; `db_bytes`/`log_bytes`/`sample_failures` leave the body (Codex: sample failures already have a Prometheus counter and active `disk.probe` diagnostics — same ruling as refreshes_gated) |
+
+All current top-level extras die: `queries_dropped`/`writer_failed` fold into `query_history`; `refreshes_gated` leaves health entirely (its Prometheus counter remains the record — Codex: table/prose disagreed, this is the ruling); `snapshot_generation` dies (no real admin consumer — Codex; generation stays in Prometheus, and protection `unavailable` is the UI-facing fact).
+
+logger.zig work (S1-owned): the gating-episode state + `last_drop_s` atomic, stamped at the drop sites (logger.zig:687 area). **Episode start must be race-free** (Codex: runWriter fills a batch before checking the gate, so a naïve "mark on gate check" misclassifies gate-caused overflow drops as ordinary ones): the episode begins when the first pending batch observes the closed gate — before further timed filling — or via an equivalent race-free gate observation. Rollup tests rewritten per condition: each degrading state degrades alone and is visible in the body; paused does not degrade; expired pause reports active; protection unavailable wins over paused.
+
+### S1.2 Delete the upstream-minute history subsystem
+
+Extent (verified by exploration; the build agent must hit every item):
+
+- src/upstream/history.zig (whole file) and src/storage/repositories/upstream_history_repo.zig (whole file).
+- querylog_schema.zig: `upstream_targets` (:74), `upstream_minute` (:79), `idx_upstream_minute_ts` (:90), expected-object list :341-342. **This changes the schema fingerprint → querylog.db resets via the m28 aside mechanism; changelog states it.**
+- app.zig wiring: :52,75,549-553 init/diagnostics/pool.history, :639 reopen path, :800 web state, :925-939 shutdown flush, :954 run task.
+- pool.zig: `history` field :52,138-141 and `recordHistory` :377-386 (history-only). **`entry.health.recordSuccess/recordFailure` (:345,:364) and `recordDiagnostics` (:390+) are live routing/backoff/diagnostics state and must survive**; callsites :259,:269 keep those calls.
+- retention.zig: history prune :19,134-135 + tests :457-505.
+- Handler src/web/handlers/upstream_health.zig, route routes.zig:77, server.zig:43, openapi.yaml :501 path + :2122 enum, goldens.
+- health.zig history signal (dies with the rollup rewrite anyway).
+- metrics.zig: the four `nxdns_upstream_history_*` metrics :34,221,391-411 + tests :819-828.
+- events.zig: the **emitter** enum entry dies, but the read side keeps the legacy code (Codex critical: rows survive, the list endpoint passes stored codes through, and dropping the union member would make real responses violate the contract; unresolvable active legacy episodes would also pin `untracked_active_count` and the SQL slow path forever). Contract: `upstream_history.write` stays in the openapi/TS event-code union and in eventCopy.ts, marked legacy; on store init/upgrade, any **active** `upstream_history.write` episode is resolved once, **with the open time as its resolution time** (not old `last_seen` — Codex: else the episode can be pruned in the same init), and **before the active-mirror load/count and pruning** (or the mirror is recomputed after) so `untracked_active_count` and the SQL fast path settle correctly. Test: an m29 database with an active flush-failure episode opens with that episode resolved and listable, `untracked_active_count` at 0, and the fast path in use.
+- retention.zig: also `Retention.Stats.upstream_rows_pruned`, its atomic counter, and `nxdns_retention_upstream_rows_pruned_total` (Codex: they would remain forever zero).
+- tests.zig :80,:84; affected web_integration_test.zig / resolver_integration_test.zig / pool.zig:937-980 tests.
+- `stats.periodParam` stays (stats uses it).
+- Sweep (S1.4 gate): case-insensitive grep over src/ docs/ PLAN.md for `upstream[_ -]history`, `upstream_minute`, `upstream_targets`, `upstream_rows_pruned`, `/api/upstream/health` — known stale spots include src/cli.zig, querylog_schema.zig WAL commentary, src/upstream/health.zig, metrics comments (the read-side legacy event code is the one allowed survivor).
+
+### S1.3 Contracts and docs
+
+openapi.yaml: Health schema rewritten (five objects, enums exact), upstream/health path+schema removed. Goldens regenerated; drift guards updated (route count −1; the health guard covers the nested objects). Docs are S1-owned (Codex): a prose sweep, not just cited lines — PLAN.md (upstream-history/API mentions :328,489-500,563 plus the summary and page/manual-check passages), docs/reference/api.md, docs/how-to/troubleshoot.md (old flat health JSON), docs/tutorial/first-run.md and docs/reference/configuration.md ("Dashboard" naming → Overview), stale comments in pool.zig. **specs/ui-redesign.md is S1-owned too** (Codex): amend §§event sources, API changes, recreation, deletion cost, and build sequence to the five-condition contract and the subsystem deletion, with a dated amendment note. CHANGELOG (S1 half): health contract break (pre-v0.1), subsystem + endpoint removal, querylog.db reset, warn→low, the design amendment.
+
+Destructive-transition acceptance (Codex: generic fingerprint tests do not pin this): a test opens an m29-schema querylog.db, asserts it is renamed aside, recreated without the two tables, coverage restarts, and `query_log.recreated` is emitted. S2 updates the `query_log.recreated` eventCopy wording, which currently says the database "was unreadable" — false for a planned schema change; the copy must encompass both causes.
+
+### S1.4 Acceptance (S1)
+
+- [ ] Rollup tests per S1.1, incl. gating-episode losing-state transitions (gate holds without a drop → recording; drop during episode → losing; gate reopens → recording; writer death → failed) and `last_drop_s` stamping on a forced queue overflow.
+- [ ] `zig build test` + `-Dintegration` 0 failed; goldens green; fmt clean; the S1.2 case-insensitive sweep finds nothing **except the allowed read-side legacy survivors**: the `upstream_history.write` code in the event union/openapi/read path and their tests (Codex: state the exception so the gate is passable). Admin is expectedly red until S2 — an intra-milestone state only; nothing is committed before S2's gates pass.
+
+
+## Session S3: period aggregations (Zig)
+
+**Owner ruling 2026-08-22**: after reviewing the built status-row Overview on a wide monitor, the owner replaced the three-section design with Pi-hole's dashboard layout ("just copy pi-hole's dashboard's layout; get rid of the current status section"). S1's health contract is unchanged. S3 adds the aggregations; S2 describes the final admin state. Build order: S1 (done) → S3 → S2.
+
+All three aggregations read the query log over the same `[since,until)` Span the stats endpoints use (`stats.periodParam`, floor-aligned buckets), return `period/since/until/coverage` like the existing stats bodies, and honor `hide_domains`/`hide_client_ips` transforms where fields are sensitive. Pure SQL over m28's provenance columns — **no revival of the upstream-minute subsystem, no new writers, no new state.**
+
+**Per-response atomicity** (Codex): each response — the three new endpoints AND the existing totals/timeseries — reads its aggregation and its `coverage` inside one SQLite read transaction, so retention cannot prune between the aggregate and the watermark, and the clients endpoint ranks and buckets from one database state. **Transaction ownership** (Codex critical): WebState shares one querylog_db connection across concurrent HTTP tasks with no mutex — SQLite serialized mode protects single calls, not transactions; concurrent BEGINs fail and foreign reads can interleave inside a transaction. S3 adds a `querylog_lock` covering every web-layer access to that connection (or separate read connections — implementer's choice, stated in the report), and the read transaction uses **deferred BEGIN**, not the existing `db.Tx` BEGIN IMMEDIATE (which would block the logger/retention writers); if a read-transaction abstraction is added, db.zig joins S3 ownership. Tests: concurrent stats/query requests under load; a prune racing a response never yields pre-prune data tagged with a post-prune `available_since`.
+
+### S3.1 `GET /api/stats/types` — query-type breakdown
+
+`{period, since, until, coverage, types: [{qtype: ?u16, count: u64}]}` — GROUP BY qtype. **No `name` field** (Codex: the only qtype-name mapping lives in admin qtype.ts; a Zig copy would drift — the UI labels codes with its existing `qtypeName()`). `query_log.qtype` is nullable: null groups into its own row (`qtype: null`), never silently dropped. Ordering: count DESC, then qtype ASC with null last (deterministic for goldens/colors). No zero rows.
+
+### S3.2 `GET /api/stats/routes` — how queries were answered
+
+`{period, since, until, coverage, routes: [{route: RouteKind-wire-string, source: ?string, count: u64}]}`. **Grouping key exact** (Codex: `source_name` is blocklist provenance, NOT the answering upstream): `upstream` rows group by `query_log.upstream`; `forward_zone` rows by `query_log.forward_zone`; `blocked`/`cache`/`local`/`rejected` rows have `source: null`. A null upstream/forward_zone identity on those route kinds is its own `source: null` row (UI labels it "Unknown"). Ordering: count DESC, then route ASC (the stored wire string, i.e. alphabetical: blocked, cache, forward_zone, local, rejected, upstream — not enum declaration order), then source ASC nulls last. This feeds the "Upstream servers" donut: blocked/cache/local/rejected shares by route kind, each upstream and forward zone by name.
+
+### S3.3 `GET /api/stats/clients` — per-client timeseries
+
+`{period, since, until, bucket_seconds, coverage, clients: [{client: string, buckets: [u64]}], other: [u64]}` — **`bucket_seconds`, the established field name** (Codex). Top 8 clients ranked by total in-window count DESC then client ASC (deterministic cut); everything else sums into `other`, which is **always present and bucket-count-sized**, including empty windows and ≤8-client windows. Client strings arrive as stored — redaction is write-time (logger substitutes the hidden marker before the row exists), so the read path has no transform; hidden rows aggregate as one client named `hidden` (as-built, tested). Bucket alignment identical to `/api/stats/timeseries` so the charts share an x-axis; zero-filled; every series length equals the bucket count.
+
+### S3.4 Contracts
+
+Routes registered (session auth, stats rate-limit class), openapi paths + schemas, drift guards extended to the three bodies. Contract-sample seed traffic must exercise ≥2 qtypes, ≥2 route kinds and ≥2 clients. **contractSamples.gen.ts regeneration happens in S3** (Codex: the committed golden is TypeScript and byte-compared by W10 during integration; S3 cannot pass its own gate without it) — the handwritten-TS typecheck stays red until S2, same intra-milestone rule as S1.
+
+Handler tests per endpoint: empty window, populated matrix, redaction transforms, coverage flag, null-qtype row, null-source row, the top-8 boundary (9th client folds into other), tie-order determinism. Empty-window bodies are exact: `types`/`routes` return empty arrays, `clients` returns zero-filled buckets with an empty `clients` list. **Conservation tests** (Codex, controlled-state only — see S2.2's window-coherence limit): over one shared span, `sum(types.count) == totals.queries`, `sum(routes.count) == totals.queries`, and per bucket `sum(named client series) + other == timeseries.queries` — catches null-loss, route omission, and bad partitioning.
+
+### S3.5 Acceptance (S3)
+
+- [ ] `zig build test` + `-Dintegration` 0 failed (W10 included); goldens green; fmt clean. Admin handwritten-TS red stays expected until S2.
+
+## Session S2 (revised): Overview as Pi-hole's dashboard (admin)
+
+### S2.1 Shell
+
+- Nav: "Overview" first; root index redirects to `/overview`. **The header ProtectionIndicator is deleted** — no protection display in the header.
+- **PauseControl moves to the sidebar bottom, directly above the version label — in BOTH sidebar renderings** (desktop rail and the mobile drawer, each of which has the version footer; Codex): same control, reading `useProtection()`, hidden while protection is unavailable/unknown, menu closes when state leaves active, mutation invalidates healthQuery. Existing PauseControl tests move with it plus placement tests for both renderings; the 390px smoke exercises the drawer. **The control states the pause with itself** (live-smoke gap, 2026-08-22: with the indicator and the status row both deleted, a paused resolver left no trace in the DOM outside `/diagnostics`, and "Resume" names an action without naming the state it would end): a state line rendered with the control in both renderings — `Paused until HH:MM` when `until` is set, `Paused` when it is null, nothing at all while active, since the button already says Pause. Text, never colour alone; the same `formatClock` and the same `protection` reading as the health strip, so the two cannot disagree, and `useProtection`'s expiry refetch retires both together. Tested for both paused shapes and for its absence while active.
+- **Diagnostics nav item badge**: from healthQuery — `active_warnings + active_errors` as a count; degraded health `status` with zero events shows "!" so no degraded state is invisible; **a failed health poll shows a neutral "!" with accessible text "Health unavailable"** (Codex: unknown must not be unbadged — only initial loading may be); hidden only when health data exists, the latest poll succeeded, and status is ok with zero events — **"fresh" means exactly that, never TanStack staleness** (Codex: healthQuery's staleTime is 0, so `isStale` would badge every gap between polls). Count/shape+text, never color alone. Tests: 0 hidden, N shown, degraded-zero-events "!", failed-poll "!", initial-loading hidden.
+- **Diagnostics page health strip**: the five S1.1 conditions rendered compactly at the top; healthy quiet, degraded highlighted. **Full load contract migrates from the deleted StatusSection** (Codex): visible loading before first health response; on refetch failure an error row with Retry, cached conditions marked stale, never presented as current; recovery clears the staleness. **Exact link matrix, restated** (Codex: no dangling reference): protection unavailable → `/blocklists`; upstreams unavailable → `/upstreams`; query history losing → in-page filter `component=disk`; failed → `component=query_log`; disk low/critical → `component=disk`; diagnostics unavailable → no link, copy "Diagnostics are not being recorded. Check free disk space and the configuration database's permissions." In-page filter links **set `component` and clear incompatible active filters** (severity/state stay default). The drops secondary text ("N queries dropped, last at HH:MM"; "N queries dropped" when `last_drop_s` null) lives on the query-history entry.
+
+### S2.2 The page — Pi-hole's layout with our data
+
+**Period is URL state**: `/overview?period=1h|24h|7d|30d` validated in the route search, default `24h`; the picker navigates (functional search update); deep-link and reload tests (Codex: the accepted time-scoping contract keeps shareable period state; the layout ruling did not revoke it).
+
+Top to bottom, edge-to-edge grid, no status/issues sections:
+
+1. **Four stat tiles**, neutral chrome (owner ruling: no colored accents; emphasis via value typography): Queries, Blocked (count + %), Clients, Avg response. Footer links: Queries → `/activity?mode=history&since&until` (returned bounds); Blocked → same + `blocked=true`; Clients → `/clients`; Avg response → none. Period picker in the section header governs every panel.
+2. **Queries over time** — the existing stacked chart, full width.
+3. **Client activity over time** — new stacked chart from `/api/stats/clients`: one series per named client + "other", same x-axis, legend with client names. **Series are labelled by registered name where one exists** (owner, 2026-08-22): the frontend resolves address → name at render time through the existing `useClientNames()` / `clientLabel()` lookup the query tables already use — hand-typed `name` first, reverse-DNS `learned_name` behind it, the bare address when the clients list knows neither — and the legend and the hidden table carry the same resolved string, so the graphic and the accessible surface never name one client differently. **Colour keys on the stored address regardless**, so registering or renaming a client never repaints its series. `clientsQuery` joins the loader's fire-and-forget set and polls on the same 30s cadence as the tables. Tests: named client renders its name, unregistered renders its address, and the swatch of a named series is still the colour of its address.
+4. **Query types** and **Upstream servers** donuts from `/api/stats/types` / `/api/stats/routes`. Two-column row **only above a named breakpoint (1280px); stacked below** (Codex). **Stacked, the ring and its legend are centred in the panel** (owner, 2026-08-22: full-page width with the ring pinned left reads as a mistake); above the breakpoint each donut is one of a pair and stays left-anchored, in line with the panels above it. Labels: qtype via the existing `qtypeName()` (one home, `TYPE<n>` fallback, "Unknown" for null); routes via route-kind labels ("Blocked", "Cache", "Local", "Rejected") and source names ("Unknown" for null source on upstream/forward-zone rows). New reusable SVG donut, **non-focusable SVG + visible legend + a visually-hidden table carrying every label/count** (Codex: pin the pattern, no delegated a11y choice). **Empty windows render "No queries in this period."** in both donuts and the client chart — never a blank panel or a division by zero (tested). Legend/table identity and React keys are the composite `(route, source)` / qtype value — display names may collide ("Unknown" twice, same source name on two kinds), so ambiguous entries carry secondary route-kind text. **Series/slice colors key on semantic identity** — qtype value, client string, and for routes: fixed colors for the source-less route-kind slices (Blocked/Cache/Local/Rejected) while upstream/forward-zone slices key on the full `(route, source)` pair (Codex: keying on route kind alone would merge adjacent upstream slices); "other" fixed — so a rank change between refreshes never recolors an entry. The donut SVG carries `aria-hidden="true"` and `focusable="false"` (Codex: non-focusable alone does not leave the accessibility tree; the legend + hidden table are the accessible surface).
+
+**Window coherence, five requests** (totals, timeseries, clients, types, routes). This is **window coherence, not data-snapshot coherence** (Codex: matching identities cannot prove a common database state — live inserts between requests may shift counts slightly between panels, and that is accepted; conservation holds in controlled tests only). The page holds one **coherent window** identified by `(period, since, until, coverage.available_since)` (coverage joins the identity — retention advancing between requests must not mix pre/post-prune windows). Window selection comparator (Codex): newer `until` wins; for equal bounds, newer `available_since`; tested where a bucket boundary and a coverage advance cross. a response is a member only if all four fields match. Rendering is per-panel against the window: panels whose data matches render; a panel whose request is loading shows its own loading state; a panel whose request failed shows its own error+Retry — **other panels keep rendering the coherent window** (a failed donut never blanks the charts). Laggards: every response behind the window is refetched **once per endpoint-keyed mismatch episode** (episode key = endpoint + window identity); a retry token invalidates all stale completions, including across period changes; previous-period placeholder data never participates. If an endpoint stays behind after its retry, that panel shows the error state; the page never renders two windows at once. One CoverageNotice, derived from the window's watermark. Tests: one lagging endpoint refetched once then coherent; still-behind terminal error per panel; period change with in-flight stale completions; coverage-advance mismatch; per-panel failure isolation.
+
+### S2.3 Pause beside blocked queries — unchanged
+
+RelatedActions on both detail surfaces, shared control, hidden when unavailable/unknown.
+
+### S2.4 Migration and smoke
+
+- Delete: StatusSection, statusRows, IssuesSection, ProtectionIndicator (+ their tests — behaviors re-pinned on the sidebar control/badge/health strip or declared dead in the report; "unknown never renders active" re-pins on the sidebar PauseControl and badge). **The `/overview` route loader is replaced** (Codex): drop the active-diagnostics prefetch, prefetch the five stats endpoints + health; sweep stale loader comments/imports and stale ownership comments in PauseControl, protection.ts, OverviewPage, ActivitySection.
+- types.ts for the three new endpoints (hand-written halves); healthQuery consumers updated (badge, strip, control).
+- Tests per S2.1/S2.2 lists plus root redirect.
+- Smoke on the real binary (smoke28 config, traffic across ≥2 clients and ≥2 qtypes): screenshots at 1280/1920/2560/390 — healthy Overview fully populated (tiles, both charts, both donuts); paused via the sidebar control (desktop + drawer); Diagnostics badge + health strip with a tripped blocklist-refresh diagnostic (a dead forward zone produces SERVFAIL but no diagnostic); the Queries tile link landing on the exact window; `/overview?period=1h` deep link. Prefix m30smoke-.
+
+### S2.5 Acceptance (S2)
+
+- [ ] Admin gates clean; `zig build -Dadmin-dist=admin/dist` succeeds; zig suites untouched-green.
+- [ ] Screenshots per S2.4.
+
+## File ownership
+
+S1: done (src/**, PLAN.md, docs/**, specs/ui-redesign.md, CHANGELOG S1 half). S3: src/web/** (three handlers, routes.zig, openapi.yaml, goldens incl. contractSamples.gen.ts regeneration, web_integration_test.zig), src/storage/repositories/queries_repo.zig (aggregations), **PLAN.md and docs/reference/api.md again** (Codex: PLAN still says health is surfaced on Overview with status+issues; both need the three new routes and the Overview description fixed), CHANGELOG (S3 lines). S2: admin/src/**, CHANGELOG (S2 half), specs/ui-redesign.md re-amendment covering **Overview, shell placement, API additions, time scoping, deletions, and build sequence** (dated note; the Pi-hole ruling supersedes the three-section prose). Sequential S3 → S2; one commit.
+
+## Anti-requirements
+
+- No new health inputs beyond the five objects; no polling-cadence changes.
+- No revival of the upstream-minute subsystem — the routes breakdown is a query-log aggregation.
+- No charting dependency; SVG only, donuts included.
+- No colored tile accents (owner ruling); color carries state on values only.
+- No Zig qtype-name mapping (the admin's qtype.ts stays the one home).
+- No task-shaped configuration pages (step 5); no contract-closure sweep (step 6).
+- Paused protection must not degrade `status`.
+
+## Acceptance (milestone complete)
+
+- [ ] All suites green (zig, integration, goldens, admin); one signed commit; screenshots per S2.4.
+- [ ] CHANGELOG documents the health contract break, the subsystem removal + querylog.db reset, warn→low, the three new stats endpoints, the Pi-hole Overview, and the ui-redesign amendments.

specs/ui-redesign.md

@@ -2,6 +2,10 @@
 
 Author: Codex (gpt-5.6-sol, extra-high effort), 2026-08-19. **Not accepted yet.** Untracked on purpose until Mokhtar rules on the open questions at the end.
 
+**Amendment, 2026-08-22 (milestone 30).** Step 4 deletes the upstream-minute history subsystem outright rather than keeping its health condition. Once Overview loses the upstream table, the subsystem has no product consumer at all — a writer whose only reader is its own failure signal — and the no-versioning rule forbids leaving it as a stub. So `/api/health` carries **five** conditions, not six: there is no `upstream_history` object. The `upstream_history.write` event code survives as **legacy on the read side only**: stored rows keep their code, the list endpoint passes it through, and the event store resolves any still-open episode once at init. The passages below are amended in place.
+
+**Second amendment, 2026-08-22 (milestone 30, owner ruling after reviewing the built page on a wide monitor).** Overview is **not** three sections. It takes Pi-hole's dashboard layout: four stat tiles, two full-width charts, two breakdown donuts, and no current-state readout at all. The status rows and the active-issues list are withdrawn from Overview — the five health conditions become a compact strip at the top of the **Diagnostics** page, where the episodes that explain them already live, and the open-episode count becomes a badge on the Diagnostics navigation item. The shell's protection indicator is deleted outright; the Pause control moves to the foot of the sidebar, in both the desktop rail and the mobile drawer. Three period aggregations are added to feed the new panels. The §Overview, §API changes, §Time scoping, §Deletions and §Build sequence passages below are amended in place; where the older three-section prose survives elsewhere, this note supersedes it.
+
 Answers that shaped it: the server and API may change; the surface-ownership split is right; file-mode configuration pages are read-only; diagnostics are curated structured events in the vein of Pi-hole's; time scoping is per workflow; a past query must be explainable exactly; Query Log and Live merge.
 
 ## Navigation
@@ -44,32 +48,31 @@ Keeps primary navigation because identifying and naming unknown devices is an op
 
 ## Overview
 
-Three sections, nothing else.
+One question, answered over a period the reader chooses: what did the resolver do. No current-state readout — that moved to Diagnostics (§Diagnostics) — and no active-issues list.
 
-**1. Current status.** Five current facts, each conveyed by text and icon as well as colour. Healthy rows stay quiet; degraded rows link to the diagnostic or configuration surface that explains them.
+**Period is URL state.** `/overview?period=1h|24h|7d|30d`, validated in the route search, defaulting to 24h. The picker navigates, so a view of the page is a link.
 
-| Status | Shows | Why it belongs |
-| --- | --- | --- |
-| Protection | Active, paused until a timestamp, or unavailable, with Pause/Resume | Confirms filtering is in force, and carries the valid runtime action |
-| Upstreams | available / configured, now | Confirms DNS can leave the network |
-| Query history | Recording, losing rows, or writer failed | Says whether Activity can be trusted |
-| Diagnostics | Recording or unavailable | A failure reporter that cannot record failures must itself be visible |
-| Storage | ok / low / critical, free bytes | Says whether writes are safe, and explains write gating |
+Top to bottom, edge to edge:
 
-The shell carries a small global "Protection active/paused" indicator linking back to Overview. The controls themselves stay on Overview and beside blocked-query details.
+1. **Four stat tiles**, neutral chrome throughout — no coloured accents; emphasis is typographic. Queries, Blocked (count and rate), Clients, Average response. Each tile carries the way into the rows behind its number: Queries and Blocked open Activity for exactly the bounds the stats response returned, Clients opens the clients page, and Average response has nothing to open.
+2. **Queries over time** — the existing query-volume timeline, split blocked/cached/other, full width.
+3. **Client activity over time** — one stacked series per named client plus "other", on the same bucket alignment as the timeline so the two charts share an x-axis. A client registered under a name is labelled by it, with the same precedence the query tables apply and the address kept as the title; colour keys on the address, so naming a client never repaints its series.
+4. **Query types** and **Upstream servers** — two donuts, side by side above 1280px and stacked below, with the ring and its legend centred in the panel while stacked and left-anchored once they are a pair. Types are labelled by the admin's own `qtypeName()`; routes by route-kind labels and by the answering resolver or zone. Each donut's SVG is decoration (`aria-hidden`, `focusable="false"`); a visible legend and a visually hidden table are the accessible surface. An empty window says "No queries in this period." rather than drawing nothing.
 
-**2. Active issues.** Severity, short title, affected object, how long it has been active, link to the detail. When none exist, one restrained line: "No active operational issues." Resolved failures never appear here, and healthy subsystems never get permanent green cards.
+Colours key on semantic identity — the qtype value, the client string, the `(route, source)` pair — so a rank change between two polls never repaints an entry. Charts stay lightweight SVG; no charting dependency.
 
-**3. Activity over a period.** The existing 1h / 24h / 7d / 30d control. Every value uses exactly the returned `[since, until)` window: queries, blocked count and rate, distinct clients, average response time, one query-volume timeline split blocked/cached/other, and "Open activity for this period" carrying the exact bounds. The timeline stays the existing lightweight SVG; no charting dependency.
+**Window coherence, five requests.** Totals, timeseries, clients, types and routes are separate calls, and the page holds one window identified by `(period, since, until, coverage.available_since)` — the watermark joins the identity because retention advancing mid-page changes what the same span can answer for. A response is a member only if all four fields match. Rendering is per panel: a member renders, a panel still in flight shows its own loading state, a panel whose request failed shows its own error and Retry, and the members keep rendering throughout — a failed donut never blanks the charts. A response behind the window is refetched once per endpoint-keyed episode and, if it stays behind, that panel alone shows an error. This is window coherence, not data-snapshot coherence: live inserts between requests may shift counts slightly between panels, and that is accepted. One coverage notice for the page, from the window's watermark.
 
-If the selected period predates available data, the section says "Query history is available from …" rather than charting the missing span as zero.
+**The shell.** The header carries no protection display at all. The Pause/Resume control sits at the foot of the sidebar, above the version label, in both the desktop rail and the mobile drawer; it is the only global runtime action, and it belongs to the resolver rather than to any page. It still appears beside the detail of a query that was blocked. The control states a pause with itself — "Paused until 14:05", or "Paused" when the pause has no end — because "Resume" names an action without naming the state it would end, and with the indicator and the status rows both gone the sidebar is the only place a page other than Diagnostics can carry that fact. An active resolver gets no line; the button says Pause, which is the whole message. The line and the health strip read one `protection` condition through one clock format, so they cannot disagree. The Diagnostics navigation item carries a badge: the open-episode count, or a neutral "!" when the rollup is degraded with nothing open and when the latest health poll failed — an unknown must never read as healthy. It is hidden only when health data exists, the latest poll succeeded, and the rollup is ok with nothing open.
 
-Removed from Overview: the historical upstream table, the "last failure" text, per-upstream period rates, the database and log byte breakdown, the standalone cache card.
+Removed from Overview: the historical upstream table, the "last failure" text, per-upstream period rates, the database and log byte breakdown, the standalone cache card, the status rows, the active-issues list.
 
 ## Diagnostics
 
 Not a journald viewer, and it does not subscribe to `std.log`. Producers emit a finite set of typed events at the failure boundary.
 
+**The health strip (amended 2026-08-22).** The page opens with the five `/api/health` conditions rendered compactly: Protection, Upstreams, Query history, Diagnostics, Storage, each stating its state in words and an icon as well as colour. A healthy condition is quiet; a degraded one is highlighted and offers the way out. The link matrix is exact — protection `unavailable` to Blocklists, upstreams `unavailable` to Upstreams, query history `losing` to this page filtered to `component=disk`, `failed` to `component=query_log`, disk `low` or `critical` to `component=disk`, and diagnostics `unavailable` to nothing at all, because the surface a link would filter is the thing that is broken. An in-page filter link sets `component` and clears the time bounds, which could otherwise hide the very episodes it points at; severity and state keep whatever the reader chose. Dropped-row counts ride the query-history entry. Loading: a visible state before the first reading; on a refetch failure an error row with Retry, with the conditions on screen marked as the last reading that arrived rather than the current state, cleared when a poll succeeds again. This is the surface the withdrawn Overview status rows became.
+
 ### Event model
 
 One row is one failure episode.
@@ -110,7 +113,6 @@ If the store itself cannot write, an atomic `event_store_failed` state appears i
 | Certificate stat or reload failure | `certificate.reload`, keyed by `doh`/`dot` | files readable and reload succeeds |
 | Query writer init or batch failure | `query_log.write`, keyed by `writer`/`batch`/`queue` | writer starts, or a batch succeeds without drops |
 | Query retention prune/checkpoint/vacuum failure | `query_log.maintenance`, keyed by operation | that operation succeeds |
-| Upstream-history flush failure | `upstream_history.write`, singleton | next flush succeeds |
 | Client-name selection or persistence failure | `client_names.storage`, keyed by operation | next pass succeeds |
 | Client materialisation or pruning failure | `clients.storage`, keyed by operation | next pass succeeds |
 | Upstream exchange failure | `upstream.exchange`, keyed by upstream URL | next successful exchange |
@@ -118,7 +120,7 @@ If the store itself cannot write, an atomic `event_store_failed` state appears i
 | Query-log recreation | `query_log.recreated`, one-shot | inserted resolved |
 | Configuration warning leaving a capability skipped | `configuration.load`, keyed by setting | clean load after restart |
 
-Sixteen codes. Codex proposed a seventeenth, `api.storage`, for a storage failure that produced an HTTP 500 — cut on review, because it breaks this section's own exclusion rule: a 500 already answered its caller. Do not merge the remaining codes to shrink the count either. A merged code forces `subject_key` to carry what the code no longer says.
+Fourteen codes emitted. A fifteenth, `upstream_history.write`, is legacy and read-side only: milestone 30 deleted its producer and its emitter enum member, and it survives in the documented wire union — fifteen values in all — so stored rows stay readable and in-contract. Codex proposed a sixteenth, `api.storage`, for a storage failure that produced an HTTP 500 — cut on review, because it breaks this section's own exclusion rule: a 500 already answered its caller. Do not merge the remaining codes to shrink the count either. A merged code forces `subject_key` to carry what the code no longer says.
 
 An upstream event describes a consecutive failure episode, not one row per retry. One timeout followed by success is one resolved episode.
 
@@ -180,7 +182,7 @@ Live SSE events carry the same provenance shape without a persisted `id`. A froz
 
 One contract: unix seconds UTC, `since` inclusive, `until` exclusive, point data qualifies on `since <= ts < until`, diagnostic episodes qualify when their active interval overlaps the range, current state is labelled "Now" and no historical selector touches it.
 
-URLs: `/overview?period=24h` with the server returning the exact aligned bounds; `/activity?mode=history&since=…&until=…` with `domain`, `client`, `blocked` and the other filters in the URL; `/diagnostics?since=…&until=…&severity=…&component=…`; `/activity?mode=live` with Follow/Freeze as ephemeral UI state. Investigation links always carry absolute bounds, so a viewed incident does not drift as time passes. Timestamps display in the browser's timezone; URLs and APIs stay timezone-independent.
+URLs: `/overview?period=24h`, validated in the route search and defaulting to 24h, with the server returning the exact aligned bounds and every panel of the page judged against one window identity that includes the coverage watermark; `/activity?mode=history&since=…&until=…` with `domain`, `client`, `blocked` and the other filters in the URL; `/diagnostics?since=…&until=…&severity=…&component=…`; `/activity?mode=live` with Follow/Freeze as ephemeral UI state. Investigation links always carry absolute bounds, so a viewed incident does not drift as time passes. Timestamps display in the browser's timezone; URLs and APIs stay timezone-independent.
 
 ## File mode
 
@@ -200,7 +202,7 @@ Database mode uses the same information architecture with real edit actions, plu
 
 `GET /api/config/status` → `{authority, path, reconciled_at, restart_pending}`. `restart_pending` is process state: database-mode mutations that need a restart set it, a successful restart clears it.
 
-`GET /api/health` becomes explicit about every condition that contributes to degradation — `protection`, `upstreams`, `query_history`, `upstream_history`, `diagnostics`, `disk`, each an object with its own state. The current hidden `history_flush_failing` contribution is eliminated: nothing may degrade the rollup without appearing in the response.
+`GET /api/health` becomes explicit about every condition that contributes to degradation — `protection`, `upstreams`, `query_history`, `diagnostics`, `disk`, each an object with its own state. Nothing may degrade the rollup without appearing in the response, so the hidden `history_flush_failing` contribution goes, and so does the subsystem behind it. The degrading set is exactly: protection `unavailable`, upstreams `unavailable`, query history `losing` or `failed`, diagnostics `unavailable`, disk `low` or `critical`. A paused protection is surfaced, never alarmed. The disk monitor's `warn` is renamed `low` at the serialization boundary. `queries_dropped`, `writer_failed`, `refreshes_gated` and `snapshot_generation` leave the body; the first two fold into `query_history`, and the last two stay in Prometheus.
 
 `GET /api/diagnostics?state=&severity=&component=&since=&until=&limit=&before=` returns `{events[], next_before, active:{warnings, errors}}`. `GET /api/diagnostics/{id}` returns one event or 404 after retention. No acknowledgement, dismissal, generic-action or raw-log endpoints.
 
@@ -208,6 +210,8 @@ Database mode uses the same information architecture with real edit actions, plu
 
 `GET /api/stats` and `/api/stats/timeseries` add `complete` and `available_since`.
 
+**Three period aggregations (added 2026-08-22)** to feed the new Overview panels, all taking the same `period` parameter and reporting over the same aligned window, and all reading their rows and their coverage watermark inside one deferred SQLite read transaction. `GET /api/stats/types` → `{period, since, until, coverage, types:[{qtype, count}]}`, the numeric type only — naming types stays the admin's job, and a second table in the server would drift out of agreement with it — with the rows that recorded no type kept as their own `null` group. `GET /api/stats/routes` → `{period, since, until, coverage, routes:[{route, source, count}]}`, grouping `upstream` rows by the answering resolver and `forward_zone` rows by the zone, with blocked, cache, local and rejected carrying no source. `GET /api/stats/clients` → `{period, since, until, bucket_seconds, coverage, clients:[{client, buckets}], other}`, bucketed exactly as `/api/stats/timeseries`, the eight busiest clients named and everything else summed into `other`, which is always present and always bucket-count-sized. No new writers and no new state: all three are pure reads over the query log's provenance columns.
+
 Existing mutation endpoints stay specific. Diagnostics introduces no generic "perform remediation" endpoint; it invokes the existing blocklist-refresh and certificate-reload operations.
 
 ## Schema changes
@@ -249,7 +253,7 @@ No new provenance table, no key/value store — household retention makes nullab
 
 `transformed()` must hide `matched`, `cname_target` and `safe_search_target` under `hide_domains`, not only `domain`.
 
-On recreation: query rows, provenance and upstream-minute history reset together as today; the old file stays aside; `config.db` diagnostics survive; a resolved `query_log.recreated` event records the reason, the aside path and the new coverage start; Overview and Activity report the incomplete range instead of charting zero.
+On recreation: query rows and provenance reset together; the old file stays aside; `config.db` diagnostics survive; a resolved `query_log.recreated` event records the reason, the aside path and the new coverage start; Overview and Activity report the incomplete range instead of charting zero.
 
 ## Deletions and their cost
 
@@ -258,7 +262,9 @@ On recreation: query rows, provenance and upstream-minute history reset together
 | Separate Query Log and Live pages | separate bookmarks; both modes remain in Activity |
 | Standalone Lookup page | a top-level bookmark; testing remains under Activity |
 | Top-level Groups, Blocklists, Rules, Local DNS, Upstreams, Settings | direct resource navigation; all capabilities remain under task-shaped configuration |
-| Historical upstream table on Overview | at-a-glance period rates; availability stays, failures move to Diagnostics |
+| Historical upstream table on Overview, and the upstream-minute history subsystem behind it | at-a-glance period rates, and the ranged per-upstream counts entirely; availability stays on the Diagnostics health strip, failures are Diagnostics episodes. `querylog.db` resets on the schema change |
+| Overview status rows and active-issues list (2026-08-22) | a current-state readout on the landing page; the five conditions move to the Diagnostics health strip and the open count to the Diagnostics nav badge |
+| Shell protection indicator (2026-08-22) | protection stated on every page; the sidebar Pause control carries it, its label for the action and its state line for a pause, and the health strip states it in full |
 | "last failure · 9h ago" text | nothing actionable; the episode becomes a diagnostic |
 | Detailed DB/log byte gauges | exact component sizes stay in Prometheus; free space stays on Overview |
 | Standalone cache card | one prominent number; cache stays in the timeline and metrics |
@@ -279,7 +285,7 @@ Each step leaves the app working and shippable, and updates its OpenAPI contract
 
    Parsed-SERVFAIL logging reverses ruling 20: `handler.zig:507` counts today rather than logging. The `Context` exists at every `servFail` site that follows question parsing. Pre-parse failures correctly stay counters.
 3. **Activity consolidation.** The unified History/Live surface, URL filters, freeze/follow, live detail, current-policy test, historical detail links. Query Log, Live and Lookup routes and code are removed in the same change. Route, SSE, accessibility, reconnect and bounded-buffer tests.
-4. **Overview replacement.** Current status, active diagnostics, one coherent activity section. New health contract and completeness states. The upstream-history table, stale-failure text, detailed DiskCard and cache card go.
+4. **Overview replacement.** New five-condition health contract and completeness states, the three period aggregations, and — per the 2026-08-22 ruling — Pi-hole's dashboard layout: four stat tiles, the query-volume and per-client charts, the types and routes donuts, all against one coherent window. The five conditions become the Diagnostics health strip and the nav badge; the shell indicator dies and Pause moves to the sidebar foot. The upstream table, stale-failure text, detailed DiskCard and cache card go, and the upstream-minute history subsystem goes with the table — `src/upstream/history.zig`, its repository, its two `querylog.db` tables, `GET /api/upstream/health` and its four Prometheus metrics. Deleting the tables changes the query-log fingerprint, so this step resets `querylog.db` the same way step 2 does.
 5. **Task-shaped configuration and file mode.** `/api/config/status` and server-owned `restart_pending`. Protection, Resolution and System in both read-only and editable forms. Clients and its detail route redesigned. Old configuration routes replaced atomically; global banner and disabled forms removed.
 6. **Contract closure.** Remove obsolete queries, types, stores, CSS, tests and route fixtures. Regenerate contract samples, update OpenAPI and reference docs, add cross-surface acceptance tests for investigation links, file authority, query-log recreation, active-event recovery and time bounds. Zig, frontend, integration, accessibility and byte-budget checks; no new dependency.