# Milestone 30: Overview replacement

Redesign step 4 of specs/ui-redesign.md (§Overview, §API changes /api/health, §Deletions), amended by the owner's 2026-08-22 ruling: the dashboard becomes Overview **with Pi-hole's dashboard layout** (stat tiles, two charts, two donuts — no status or issues sections; see S3/S2). The health contract becomes explicit — nothing may degrade the rollup without appearing in the response. Codex design review folded in (thread 01a028b1); corrections marked where they changed the shape.

**One amendment to the accepted design, made here and recorded in the changelog**: ui-redesign.md:113/203 lists an `upstream_history` health condition. After this milestone deletes the dashboard's upstream table, the upstream-minute history subsystem has no product consumer at all — a writer whose only reader is its own failure signal. The no-versioning rule forbids keeping it as a stub, so this milestone deletes the subsystem (S1.2) and the health contract has **five** conditions, not six.

## Sessions

S1 (done): Zig — health contract, subsystem deletion, contracts, docs. S3 (after S1): Zig — the three period aggregations. S2 (after S3): admin — the Pi-hole Overview, sidebar pause control, Diagnostics badge/strip, deletions, smoke. Sequential; **the milestone lands as one atomic commit** (Codex: S1 alone breaks the generated contract sample's TS assignment and every Health consumer — its red admin typecheck is an intra-milestone state, never a landed one).

---

## Session S1: the explicit health contract

### S1.1 `GET /api/health` (src/web/handlers/health.zig)

New body: `status` plus five condition objects. The degrading set is exactly (Codex: name it, don't gesture at "not ok"): protection `unavailable`, upstreams `unavailable`, query_history `losing`|`failed`, diagnostics `unavailable`, disk `low`|`critical`. Nothing else reaches `degraded()` — this kills the hidden `history_flush_failing` input (health.zig:65-73/177-178) along with its subsystem.

| object | shape | state rule |
| --- | --- | --- |
| `protection` | `{state: "active"\|"paused"\|"unavailable", until: ?i64}` | Precedence `unavailable` → `paused` → `active` (Codex). `unavailable` ⇔ no current filter snapshot exists (the condition under which handler.zig answers with `snapshot_unavailable` provenance) — pause state is irrelevant then, `until` null. Else `paused` ⇔ the pause is live now (indefinite → until null; timed and unexpired → until set). **An expired timed pause is `active`** (Codex). Paused does NOT degrade — operator choice, surfaced not alarmed; unavailable degrades. |
| `upstreams` | `{state: "ok"\|"unavailable", available, total}` | unavailable ⇔ available == 0. `total` = enabled routing candidates (the pool is built from enabled upstreams only — Codex); S2 labels it "of N enabled". |
| `query_history` | `{state: "recording"\|"losing"\|"failed", dropped_total, last_drop_s: ?i64}` | failed ⇔ `writer_failed`. losing ⇔ a logger-owned **gating-episode state** (Codex critical: `draining` is a shutdown flag, false during normal disk gating): logger marks the episode active while the disk gate holds writes, flips to losing once a drop occurs during that episode, clears when the gate reopens. Event-driven; no time window. Else recording. Queue-overflow drops outside a gating episode do not flip the state (cumulative ≠ current); they surface via `dropped_total` + `last_drop_s` (atomic stamped per drop, null until one happens). |
| `diagnostics` | `{state: "recording"\|"unavailable", active_warnings, active_errors}` | current rule (`!present or write_failed`) |
| `disk` | `{state: "ok"\|"low"\|"critical", free_bytes}` | disk_monitor states; **warn→low renamed at the serialization boundary**; `db_bytes`/`log_bytes`/`sample_failures` leave the body (Codex: sample failures already have a Prometheus counter and active `disk.probe` diagnostics — same ruling as refreshes_gated) |

All current top-level extras die: `queries_dropped`/`writer_failed` fold into `query_history`; `refreshes_gated` leaves health entirely (its Prometheus counter remains the record — Codex: table/prose disagreed, this is the ruling); `snapshot_generation` dies (no real admin consumer — Codex; generation stays in Prometheus, and protection `unavailable` is the UI-facing fact).

logger.zig work (S1-owned): the gating-episode state + `last_drop_s` atomic, stamped at the drop sites (logger.zig:687 area). **Episode start must be race-free** (Codex: runWriter fills a batch before checking the gate, so a naïve "mark on gate check" misclassifies gate-caused overflow drops as ordinary ones): the episode begins when the first pending batch observes the closed gate — before further timed filling — or via an equivalent race-free gate observation. Rollup tests rewritten per condition: each degrading state degrades alone and is visible in the body; paused does not degrade; expired pause reports active; protection unavailable wins over paused.

### S1.2 Delete the upstream-minute history subsystem

Extent (verified by exploration; the build agent must hit every item):

- src/upstream/history.zig (whole file) and src/storage/repositories/upstream_history_repo.zig (whole file).
- querylog_schema.zig: `upstream_targets` (:74), `upstream_minute` (:79), `idx_upstream_minute_ts` (:90), expected-object list :341-342. **This changes the schema fingerprint → querylog.db resets via the m28 aside mechanism; changelog states it.**
- app.zig wiring: :52,75,549-553 init/diagnostics/pool.history, :639 reopen path, :800 web state, :925-939 shutdown flush, :954 run task.
- pool.zig: `history` field :52,138-141 and `recordHistory` :377-386 (history-only). **`entry.health.recordSuccess/recordFailure` (:345,:364) and `recordDiagnostics` (:390+) are live routing/backoff/diagnostics state and must survive**; callsites :259,:269 keep those calls.
- retention.zig: history prune :19,134-135 + tests :457-505.
- Handler src/web/handlers/upstream_health.zig, route routes.zig:77, server.zig:43, openapi.yaml :501 path + :2122 enum, goldens.
- health.zig history signal (dies with the rollup rewrite anyway).
- metrics.zig: the four `nxdns_upstream_history_*` metrics :34,221,391-411 + tests :819-828.
- events.zig: the **emitter** enum entry dies, but the read side keeps the legacy code (Codex critical: rows survive, the list endpoint passes stored codes through, and dropping the union member would make real responses violate the contract; unresolvable active legacy episodes would also pin `untracked_active_count` and the SQL slow path forever). Contract: `upstream_history.write` stays in the openapi/TS event-code union and in eventCopy.ts, marked legacy; on store init/upgrade, any **active** `upstream_history.write` episode is resolved once, **with the open time as its resolution time** (not old `last_seen` — Codex: else the episode can be pruned in the same init), and **before the active-mirror load/count and pruning** (or the mirror is recomputed after) so `untracked_active_count` and the SQL fast path settle correctly. Test: an m29 database with an active flush-failure episode opens with that episode resolved and listable, `untracked_active_count` at 0, and the fast path in use.
- retention.zig: also `Retention.Stats.upstream_rows_pruned`, its atomic counter, and `nxdns_retention_upstream_rows_pruned_total` (Codex: they would remain forever zero).
- tests.zig :80,:84; affected web_integration_test.zig / resolver_integration_test.zig / pool.zig:937-980 tests.
- `stats.periodParam` stays (stats uses it).
- Sweep (S1.4 gate): case-insensitive grep over src/ docs/ PLAN.md for `upstream[_ -]history`, `upstream_minute`, `upstream_targets`, `upstream_rows_pruned`, `/api/upstream/health` — known stale spots include src/cli.zig, querylog_schema.zig WAL commentary, src/upstream/health.zig, metrics comments (the read-side legacy event code is the one allowed survivor).

### S1.3 Contracts and docs

openapi.yaml: Health schema rewritten (five objects, enums exact), upstream/health path+schema removed. Goldens regenerated; drift guards updated (route count −1; the health guard covers the nested objects). Docs are S1-owned (Codex): a prose sweep, not just cited lines — PLAN.md (upstream-history/API mentions :328,489-500,563 plus the summary and page/manual-check passages), docs/reference/api.md, docs/how-to/troubleshoot.md (old flat health JSON), docs/tutorial/first-run.md and docs/reference/configuration.md ("Dashboard" naming → Overview), stale comments in pool.zig. **specs/ui-redesign.md is S1-owned too** (Codex): amend §§event sources, API changes, recreation, deletion cost, and build sequence to the five-condition contract and the subsystem deletion, with a dated amendment note. CHANGELOG (S1 half): health contract break (pre-v0.1), subsystem + endpoint removal, querylog.db reset, warn→low, the design amendment.

Destructive-transition acceptance (Codex: generic fingerprint tests do not pin this): a test opens an m29-schema querylog.db, asserts it is renamed aside, recreated without the two tables, coverage restarts, and `query_log.recreated` is emitted. S2 updates the `query_log.recreated` eventCopy wording, which currently says the database "was unreadable" — false for a planned schema change; the copy must encompass both causes.

### S1.4 Acceptance (S1)

- [ ] Rollup tests per S1.1, incl. gating-episode losing-state transitions (gate holds without a drop → recording; drop during episode → losing; gate reopens → recording; writer death → failed) and `last_drop_s` stamping on a forced queue overflow.
- [ ] `zig build test` + `-Dintegration` 0 failed; goldens green; fmt clean; the S1.2 case-insensitive sweep finds nothing **except the allowed read-side legacy survivors**: the `upstream_history.write` code in the event union/openapi/read path and their tests (Codex: state the exception so the gate is passable). Admin is expectedly red until S2 — an intra-milestone state only; nothing is committed before S2's gates pass.


## Session S3: period aggregations (Zig)

**Owner ruling 2026-08-22**: after reviewing the built status-row Overview on a wide monitor, the owner replaced the three-section design with Pi-hole's dashboard layout ("just copy pi-hole's dashboard's layout; get rid of the current status section"). S1's health contract is unchanged. S3 adds the aggregations; S2 describes the final admin state. Build order: S1 (done) → S3 → S2.

All three aggregations read the query log over the same `[since,until)` Span the stats endpoints use (`stats.periodParam`, floor-aligned buckets), return `period/since/until/coverage` like the existing stats bodies, and honor `hide_domains`/`hide_client_ips` transforms where fields are sensitive. Pure SQL over m28's provenance columns — **no revival of the upstream-minute subsystem, no new writers, no new state.**

**Per-response atomicity** (Codex): each response — the three new endpoints AND the existing totals/timeseries — reads its aggregation and its `coverage` inside one SQLite read transaction, so retention cannot prune between the aggregate and the watermark, and the clients endpoint ranks and buckets from one database state. **Transaction ownership** (Codex critical): WebState shares one querylog_db connection across concurrent HTTP tasks with no mutex — SQLite serialized mode protects single calls, not transactions; concurrent BEGINs fail and foreign reads can interleave inside a transaction. S3 adds a `querylog_lock` covering every web-layer access to that connection (or separate read connections — implementer's choice, stated in the report), and the read transaction uses **deferred BEGIN**, not the existing `db.Tx` BEGIN IMMEDIATE (which would block the logger/retention writers); if a read-transaction abstraction is added, db.zig joins S3 ownership. Tests: concurrent stats/query requests under load; a prune racing a response never yields pre-prune data tagged with a post-prune `available_since`.

### S3.1 `GET /api/stats/types` — query-type breakdown

`{period, since, until, coverage, types: [{qtype: ?u16, count: u64}]}` — GROUP BY qtype. **No `name` field** (Codex: the only qtype-name mapping lives in admin qtype.ts; a Zig copy would drift — the UI labels codes with its existing `qtypeName()`). `query_log.qtype` is nullable: null groups into its own row (`qtype: null`), never silently dropped. Ordering: count DESC, then qtype ASC with null last (deterministic for goldens/colors). No zero rows.

### S3.2 `GET /api/stats/routes` — how queries were answered

`{period, since, until, coverage, routes: [{route: RouteKind-wire-string, source: ?string, count: u64}]}`. **Grouping key exact** (Codex: `source_name` is blocklist provenance, NOT the answering upstream): `upstream` rows group by `query_log.upstream`; `forward_zone` rows by `query_log.forward_zone`; `blocked`/`cache`/`local`/`rejected` rows have `source: null`. A null upstream/forward_zone identity on those route kinds is its own `source: null` row (UI labels it "Unknown"). Ordering: count DESC, then route ASC (the stored wire string, i.e. alphabetical: blocked, cache, forward_zone, local, rejected, upstream — not enum declaration order), then source ASC nulls last. This feeds the "Upstream servers" donut: blocked/cache/local/rejected shares by route kind, each upstream and forward zone by name.

### S3.3 `GET /api/stats/clients` — per-client timeseries

`{period, since, until, bucket_seconds, coverage, clients: [{client: string, buckets: [u64]}], other: [u64]}` — **`bucket_seconds`, the established field name** (Codex). Top 8 clients ranked by total in-window count DESC then client ASC (deterministic cut); everything else sums into `other`, which is **always present and bucket-count-sized**, including empty windows and ≤8-client windows. Client strings arrive as stored — redaction is write-time (logger substitutes the hidden marker before the row exists), so the read path has no transform; hidden rows aggregate as one client named `hidden` (as-built, tested). Bucket alignment identical to `/api/stats/timeseries` so the charts share an x-axis; zero-filled; every series length equals the bucket count.

### S3.4 Contracts

Routes registered (session auth, stats rate-limit class), openapi paths + schemas, drift guards extended to the three bodies. Contract-sample seed traffic must exercise ≥2 qtypes, ≥2 route kinds and ≥2 clients. **contractSamples.gen.ts regeneration happens in S3** (Codex: the committed golden is TypeScript and byte-compared by W10 during integration; S3 cannot pass its own gate without it) — the handwritten-TS typecheck stays red until S2, same intra-milestone rule as S1.

Handler tests per endpoint: empty window, populated matrix, redaction transforms, coverage flag, null-qtype row, null-source row, the top-8 boundary (9th client folds into other), tie-order determinism. Empty-window bodies are exact: `types`/`routes` return empty arrays, `clients` returns zero-filled buckets with an empty `clients` list. **Conservation tests** (Codex, controlled-state only — see S2.2's window-coherence limit): over one shared span, `sum(types.count) == totals.queries`, `sum(routes.count) == totals.queries`, and per bucket `sum(named client series) + other == timeseries.queries` — catches null-loss, route omission, and bad partitioning.

### S3.5 Acceptance (S3)

- [ ] `zig build test` + `-Dintegration` 0 failed (W10 included); goldens green; fmt clean. Admin handwritten-TS red stays expected until S2.

## Session S2 (revised): Overview as Pi-hole's dashboard (admin)

### S2.1 Shell

- Nav: "Overview" first; root index redirects to `/overview`. **The header ProtectionIndicator is deleted** — no protection display in the header.
- **PauseControl moves to the sidebar bottom, directly above the version label — in BOTH sidebar renderings** (desktop rail and the mobile drawer, each of which has the version footer; Codex): same control, reading `useProtection()`, hidden while protection is unavailable/unknown, menu closes when state leaves active, mutation invalidates healthQuery. Existing PauseControl tests move with it plus placement tests for both renderings; the 390px smoke exercises the drawer. **The control states the pause with itself** (live-smoke gap, 2026-08-22: with the indicator and the status row both deleted, a paused resolver left no trace in the DOM outside `/diagnostics`, and "Resume" names an action without naming the state it would end): a state line rendered with the control in both renderings — `Paused until HH:MM` when `until` is set, `Paused` when it is null, nothing at all while active, since the button already says Pause. Text, never colour alone; the same `formatClock` and the same `protection` reading as the health strip, so the two cannot disagree, and `useProtection`'s expiry refetch retires both together. Tested for both paused shapes and for its absence while active.
- **Diagnostics nav item badge**: from healthQuery — `active_warnings + active_errors` as a count; degraded health `status` with zero events shows "!" so no degraded state is invisible; **a failed health poll shows a neutral "!" with accessible text "Health unavailable"** (Codex: unknown must not be unbadged — only initial loading may be); hidden only when health data exists, the latest poll succeeded, and status is ok with zero events — **"fresh" means exactly that, never TanStack staleness** (Codex: healthQuery's staleTime is 0, so `isStale` would badge every gap between polls). Count/shape+text, never color alone. Tests: 0 hidden, N shown, degraded-zero-events "!", failed-poll "!", initial-loading hidden.
- **Diagnostics page health strip**: the five S1.1 conditions rendered compactly at the top; healthy quiet, degraded highlighted. **Full load contract migrates from the deleted StatusSection** (Codex): visible loading before first health response; on refetch failure an error row with Retry, cached conditions marked stale, never presented as current; recovery clears the staleness. **Exact link matrix, restated** (Codex: no dangling reference): protection unavailable → `/blocklists`; upstreams unavailable → `/upstreams`; query history losing → in-page filter `component=disk`; failed → `component=query_log`; disk low/critical → `component=disk`; diagnostics unavailable → no link, copy "Diagnostics are not being recorded. Check free disk space and the configuration database's permissions." In-page filter links **set `component` and clear incompatible active filters** (severity/state stay default). The drops secondary text ("N queries dropped, last at HH:MM"; "N queries dropped" when `last_drop_s` null) lives on the query-history entry.

### S2.2 The page — Pi-hole's layout with our data

**Period is URL state**: `/overview?period=1h|24h|7d|30d` validated in the route search, default `24h`; the picker navigates (functional search update); deep-link and reload tests (Codex: the accepted time-scoping contract keeps shareable period state; the layout ruling did not revoke it).

Top to bottom, edge-to-edge grid, no status/issues sections:

1. **Four stat tiles**, neutral chrome (owner ruling: no colored accents; emphasis via value typography): Queries, Blocked (count + %), Clients, Avg response. Footer links: Queries → `/activity?mode=history&since&until` (returned bounds); Blocked → same + `blocked=true`; Clients → `/clients`; Avg response → none. Period picker in the section header governs every panel.
2. **Queries over time** — the existing stacked chart, full width.
3. **Client activity over time** — new stacked chart from `/api/stats/clients`: one series per named client + "other", same x-axis, legend with client names. **Series are labelled by registered name where one exists** (owner, 2026-08-22): the frontend resolves address → name at render time through the existing `useClientNames()` / `clientLabel()` lookup the query tables already use — hand-typed `name` first, reverse-DNS `learned_name` behind it, the bare address when the clients list knows neither — and the legend and the hidden table carry the same resolved string, so the graphic and the accessible surface never name one client differently. **Colour keys on the stored address regardless**, so registering or renaming a client never repaints its series. `clientsQuery` joins the loader's fire-and-forget set and polls on the same 30s cadence as the tables. Tests: named client renders its name, unregistered renders its address, and the swatch of a named series is still the colour of its address.
4. **Query types** and **Upstream servers** donuts from `/api/stats/types` / `/api/stats/routes`. Two-column row **only above a named breakpoint (1280px); stacked below** (Codex). **Stacked, the ring and its legend are centred in the panel** (owner, 2026-08-22: full-page width with the ring pinned left reads as a mistake); above the breakpoint each donut is one of a pair and stays left-anchored, in line with the panels above it. Labels: qtype via the existing `qtypeName()` (one home, `TYPE<n>` fallback, "Unknown" for null); routes via route-kind labels ("Blocked", "Cache", "Local", "Rejected") and source names ("Unknown" for null source on upstream/forward-zone rows). New reusable SVG donut, **non-focusable SVG + visible legend + a visually-hidden table carrying every label/count** (Codex: pin the pattern, no delegated a11y choice). **Empty windows render "No queries in this period."** in both donuts and the client chart — never a blank panel or a division by zero (tested). Legend/table identity and React keys are the composite `(route, source)` / qtype value — display names may collide ("Unknown" twice, same source name on two kinds), so ambiguous entries carry secondary route-kind text. **Series/slice colors key on semantic identity** — qtype value, client string, and for routes: fixed colors for the source-less route-kind slices (Blocked/Cache/Local/Rejected) while upstream/forward-zone slices key on the full `(route, source)` pair (Codex: keying on route kind alone would merge adjacent upstream slices); "other" fixed — so a rank change between refreshes never recolors an entry. The donut SVG carries `aria-hidden="true"` and `focusable="false"` (Codex: non-focusable alone does not leave the accessibility tree; the legend + hidden table are the accessible surface).

**Window coherence, five requests** (totals, timeseries, clients, types, routes). This is **window coherence, not data-snapshot coherence** (Codex: matching identities cannot prove a common database state — live inserts between requests may shift counts slightly between panels, and that is accepted; conservation holds in controlled tests only). The page holds one **coherent window** identified by `(period, since, until, coverage.available_since)` (coverage joins the identity — retention advancing between requests must not mix pre/post-prune windows). Window selection comparator (Codex): newer `until` wins; for equal bounds, newer `available_since`; tested where a bucket boundary and a coverage advance cross. a response is a member only if all four fields match. Rendering is per-panel against the window: panels whose data matches render; a panel whose request is loading shows its own loading state; a panel whose request failed shows its own error+Retry — **other panels keep rendering the coherent window** (a failed donut never blanks the charts). Laggards: every response behind the window is refetched **once per endpoint-keyed mismatch episode** (episode key = endpoint + window identity); a retry token invalidates all stale completions, including across period changes; previous-period placeholder data never participates. If an endpoint stays behind after its retry, that panel shows the error state; the page never renders two windows at once. One CoverageNotice, derived from the window's watermark. Tests: one lagging endpoint refetched once then coherent; still-behind terminal error per panel; period change with in-flight stale completions; coverage-advance mismatch; per-panel failure isolation.

### S2.3 Pause beside blocked queries — unchanged

RelatedActions on both detail surfaces, shared control, hidden when unavailable/unknown.

### S2.4 Migration and smoke

- Delete: StatusSection, statusRows, IssuesSection, ProtectionIndicator (+ their tests — behaviors re-pinned on the sidebar control/badge/health strip or declared dead in the report; "unknown never renders active" re-pins on the sidebar PauseControl and badge). **The `/overview` route loader is replaced** (Codex): drop the active-diagnostics prefetch, prefetch the five stats endpoints + health; sweep stale loader comments/imports and stale ownership comments in PauseControl, protection.ts, OverviewPage, ActivitySection.
- types.ts for the three new endpoints (hand-written halves); healthQuery consumers updated (badge, strip, control).
- Tests per S2.1/S2.2 lists plus root redirect.
- Smoke on the real binary (smoke28 config, traffic across ≥2 clients and ≥2 qtypes): screenshots at 1280/1920/2560/390 — healthy Overview fully populated (tiles, both charts, both donuts); paused via the sidebar control (desktop + drawer); Diagnostics badge + health strip with a tripped blocklist-refresh diagnostic (a dead forward zone produces SERVFAIL but no diagnostic); the Queries tile link landing on the exact window; `/overview?period=1h` deep link. Prefix m30smoke-.

### S2.5 Acceptance (S2)

- [ ] Admin gates clean; `zig build -Dadmin-dist=admin/dist` succeeds; zig suites untouched-green.
- [ ] Screenshots per S2.4.

## File ownership

S1: done (src/**, PLAN.md, docs/**, specs/ui-redesign.md, CHANGELOG S1 half). S3: src/web/** (three handlers, routes.zig, openapi.yaml, goldens incl. contractSamples.gen.ts regeneration, web_integration_test.zig), src/storage/repositories/queries_repo.zig (aggregations), **PLAN.md and docs/reference/api.md again** (Codex: PLAN still says health is surfaced on Overview with status+issues; both need the three new routes and the Overview description fixed), CHANGELOG (S3 lines). S2: admin/src/**, CHANGELOG (S2 half), specs/ui-redesign.md re-amendment covering **Overview, shell placement, API additions, time scoping, deletions, and build sequence** (dated note; the Pi-hole ruling supersedes the three-section prose). Sequential S3 → S2; one commit.

## Anti-requirements

- No new health inputs beyond the five objects; no polling-cadence changes.
- No revival of the upstream-minute subsystem — the routes breakdown is a query-log aggregation.
- No charting dependency; SVG only, donuts included.
- No colored tile accents (owner ruling); color carries state on values only.
- No Zig qtype-name mapping (the admin's qtype.ts stays the one home).
- No task-shaped configuration pages (step 5); no contract-closure sweep (step 6).
- Paused protection must not degrade `status`.

## Acceptance (milestone complete)

- [ ] All suites green (zig, integration, goldens, admin); one signed commit; screenshots per S2.4.
- [ ] CHANGELOG documents the health contract break, the subsystem removal + querylog.db reset, warn→low, the three new stats endpoints, the Pi-hole Overview, and the ui-redesign amendments.
