milestone 27: diagnostics — operational failures land in one curated log, resolved history purgeable

specs/ui-redesign.md

@@ -0,0 +1,304 @@
+# UI redesign proposal
+
+Author: Codex (gpt-5.6-sol, extra-high effort), 2026-08-19. **Not accepted yet.** Untracked on purpose until Mokhtar rules on the open questions at the end.
+
+Answers that shaped it: the server and API may change; the surface-ownership split is right; file-mode configuration pages are read-only; diagnostics are curated structured events in the vein of Pi-hole's; time scoping is per workflow; a past query must be explainable exactly; Query Log and Live merge.
+
+## Navigation
+
+Five primary items. Configuration expands to three task-shaped subpages and holds no landing route of its own.
+
+| Navigation | Route | Operator question | Replaces |
+| --- | --- | --- | --- |
+| Overview | `/overview` | Is DNS healthy and protecting the household now, and what happened in this period? | Dashboard |
+| Activity | `/activity` | What requests are happening or happened, and why did nxdns handle them that way? | Query Log, Live, Lookup |
+| Clients | `/clients` | Who is this address, which policy applies, and what has it been querying? | Clients |
+| Diagnostics | `/diagnostics` | What is failing or has failed, what is affected, what should I do? | new |
+| Protection | `/configuration/protection` | What policy governs each group, and which rules and lists produce it? | Groups, Blocklists, Rules |
+| Resolution | `/configuration/resolution` | Where does nxdns answer or forward permitted names? | Local DNS, Upstreams |
+| System | `/configuration/system` | What service, storage, logging, TLS and web settings is this process running with? | Settings |
+
+Secondary routes, reached from those surfaces rather than the nav: `/activity/queries/:id`, `/activity/test`, `/clients/:id`, `/diagnostics/:id`.
+
+No current page survives unchanged. Login, logout and pause survive functionally, restyled into the new shell.
+
+### Activity
+
+Two modes over the same columns and filters. History is persisted queries with keyset pagination and an absolute range. Live is follow-by-default with Freeze/Resume over the existing bounded 500-row buffer.
+
+Columns: Time, Domain, Client, Type, Result, Route, Duration. Rule matches, source URLs and upstream errors live in the detail view, never on every row.
+
+Domain testing stays as an Activity action labelled "Current policy simulation". It must never read as an explanation of a historical query.
+
+### Protection
+
+Group-centred: group list, selected group detail, effective safe-search setting, assigned blocklist sources, rules scoped to the group, client count linking to matching clients. A Sources tab holds the shared blocklist catalogue and the "Update now" runtime action.
+
+### Resolution
+
+Three tabs: upstream pool, local records, forward zones.
+
+### Clients
+
+Keeps primary navigation because identifying and naming unknown devices is an operational job, not configuration. The *learned* marker appears only here, beside the name. Prefix assignments live here as "Network assignments".
+
+## Overview
+
+Three sections, nothing else.
+
+**1. Current status.** Five current facts, each conveyed by text and icon as well as colour. Healthy rows stay quiet; degraded rows link to the diagnostic or configuration surface that explains them.
+
+| Status | Shows | Why it belongs |
+| --- | --- | --- |
+| Protection | Active, paused until a timestamp, or unavailable, with Pause/Resume | Confirms filtering is in force, and carries the valid runtime action |
+| Upstreams | available / configured, now | Confirms DNS can leave the network |
+| Query history | Recording, losing rows, or writer failed | Says whether Activity can be trusted |
+| Diagnostics | Recording or unavailable | A failure reporter that cannot record failures must itself be visible |
+| Storage | ok / low / critical, free bytes | Says whether writes are safe, and explains write gating |
+
+The shell carries a small global "Protection active/paused" indicator linking back to Overview. The controls themselves stay on Overview and beside blocked-query details.
+
+**2. Active issues.** Severity, short title, affected object, how long it has been active, link to the detail. When none exist, one restrained line: "No active operational issues." Resolved failures never appear here, and healthy subsystems never get permanent green cards.
+
+**3. Activity over a period.** The existing 1h / 24h / 7d / 30d control. Every value uses exactly the returned `[since, until)` window: queries, blocked count and rate, distinct clients, average response time, one query-volume timeline split blocked/cached/other, and "Open activity for this period" carrying the exact bounds. The timeline stays the existing lightweight SVG; no charting dependency.
+
+If the selected period predates available data, the section says "Query history is available from …" rather than charting the missing span as zero.
+
+Removed from Overview: the historical upstream table, the "last failure" text, per-upstream period rates, the database and log byte breakdown, the standalone cache card.
+
+## Diagnostics
+
+Not a journald viewer, and it does not subscribe to `std.log`. Producers emit a finite set of typed events at the failure boundary.
+
+### Event model
+
+One row is one failure episode.
+
+| Field | Meaning |
+| --- | --- |
+| `id` | Durable identifier |
+| `code` | Fixed machine-readable kind |
+| `subject_key` | Internal stable identity, may hold a full URL, never serialized |
+| `subject_label` | Bounded, redacted, operator-facing identity |
+| `severity` | `warning` or `error` |
+| `first_seen` / `last_seen` | Episode bounds |
+| `occurrences` | Deduplicated report count |
+| `resolved_at` | Null while active |
+| `detail` | Bounded last error or current condition |
+
+No `info` severity. Normal starts, refreshes and reloads do not become entries; a success resolves its prior failure.
+
+Active dedup key is `(code, subject_key)`. A repeat updates `last_seen`, `occurrences`, severity and detail. A success resolves the row. A later failure opens a new episode rather than reopening the old one. Severity records the worst state reached. There is no acknowledgement or manual dismissal: active means the component has not demonstrated recovery. One-shot material events, such as a query-log recreation, are inserted already resolved.
+
+### Storage
+
+Events live in `config.db` as runtime state, excluded from export, import and file reconciliation. Active rows are never pruned. Resolved rows keep 90 days, with a hard cap of the newest 5,000. `detail` caps at 512 bytes. Pruning runs at startup and from existing maintenance; no new scheduler.
+
+One table, one repository, one fixed event-code enum. A small serialized event store owns a dedicated `config.db` connection; background producers report synchronously through its mutex. This is the existing pattern, not a new one: `app.zig:333` opens a connection for migration and reconciliation and `app.zig:544` opens a separate `web_config_db`, and the background producers (fetcher, disk monitor, logger writer) have no other safe path into `config.db`.
+
+If the store itself cannot write, an atomic `event_store_failed` state appears in `/api/health` and journald. It clears on the next successful write — no periodic probe. The store receives a write whenever anything fails or recovers, and a flag left set while nothing needs writing costs nothing.
+
+### Event sources
+
+| Source | Event identity | Recovery |
+| --- | --- | --- |
+| Disk warn/critical transitions | `disk.space`, singleton | next `ok` sample |
+| Failed `statvfs` or directory sizing | `disk.probe`, keyed by operation/path | next successful probe |
+| Per-source download, HTTP, parse, compile or file-read failure | `blocklist.refresh`, keyed by source URL | that source refreshes |
+| Initial snapshot or whole-pass failure | `blocklist.snapshot`, singleton | a snapshot publishes |
+| Blocklist file cleanup failure | `blocklist.storage`, keyed by operation | that operation succeeds |
+| Certificate stat or reload failure | `certificate.reload`, keyed by `doh`/`dot` | files readable and reload succeeds |
+| Query writer init or batch failure | `query_log.write`, keyed by `writer`/`batch`/`queue` | writer starts, or a batch succeeds without drops |
+| Query retention prune/checkpoint/vacuum failure | `query_log.maintenance`, keyed by operation | that operation succeeds |
+| Upstream-history flush failure | `upstream_history.write`, singleton | next flush succeeds |
+| Client-name selection or persistence failure | `client_names.storage`, keyed by operation | next pass succeeds |
+| Client materialisation or pruning failure | `clients.storage`, keyed by operation | next pass succeeds |
+| Upstream exchange failure | `upstream.exchange`, keyed by upstream URL | next successful exchange |
+| Enabled DoH/DoT listener that cannot start | `listener.start`, keyed by endpoint | successful start after restart |
+| Query-log recreation | `query_log.recreated`, one-shot | inserted resolved |
+| Configuration warning leaving a capability skipped | `configuration.load`, keyed by setting | clean load after restart |
+
+Sixteen codes. Codex proposed a seventeenth, `api.storage`, for a storage failure that produced an HTTP 500 — cut on review, because it breaks this section's own exclusion rule: a 500 already answered its caller. Do not merge the remaining codes to shrink the count either. A merged code forces `subject_key` to carry what the code no longer says.
+
+An upstream event describes a consecutive failure episode, not one row per retry. One timeout followed by success is one resolved episode.
+
+Outside Diagnostics: invalid requests, conflicts, rate limits and failed logins already answered to their caller; expected reverse-DNS outcomes; individual TLS handshake failures already counted; development asset-server warnings; fatal startup failures that stop the UI existing. Disk-gated skips do not duplicate — the active disk event explains the cause and counters keep the totals.
+
+### From event to remediation
+
+The detail shows current or resolved state; first seen, last seen, occurrences, resolution time; impact in operator language; the last bounded underlying error; the exact next action; how nxdns will verify recovery; links to the relevant configuration and activity window.
+
+The frontend uses one exhaustive `switch` over the fixed codes for titles, impact, remediation and routes. No generic action schema, no plugin mechanism.
+
+## Query provenance
+
+Stays in the expendable `querylog.db`. Does not go into the operational-events table.
+
+`block_reason` is replaced by the fuller `policy_reason`. New columns:
+
+| Column | Type | Purpose |
+| --- | --- | --- |
+| `qclass` | INTEGER NOT NULL | explains filtering bypass for non-IN questions |
+| `rcode` | INTEGER NOT NULL | the client-visible result, including SERVFAIL |
+| `group_id` | INTEGER | group at query time, not a foreign key |
+| `group_name` | TEXT | historical label, survives a rename |
+| `policy_action` | TEXT NOT NULL | `not_evaluated`, `allow`, `block` |
+| `policy_reason` | TEXT NOT NULL | pipeline or matcher reason |
+| `matched` | TEXT | exact rule pattern or list entry |
+| `source_id` | INTEGER | blocklist source at query time |
+| `source_name` | TEXT | historical source label |
+| `cname_target` | TEXT | target that caused an uncloaked block |
+| `safe_search_target` | TEXT | name used for the rewrite |
+| `route_kind` | TEXT NOT NULL | `blocked`, `local`, `forward_zone`, `upstream` |
+| `forward_zone` | TEXT | exact matched zone |
+
+`upstream` changes from the unhelpful `"pool"` marker to the actual configured upstream or forward resolver **on the exchange that actually happened**. On a cache hit `route_kind` is `cache` and `upstream` is NULL. Credentials are redacted at the serialization boundary.
+
+Codex wanted the upstream recorded on cache hits too. Cut on review: it would widen every `src/cache/dns_cache.zig` entry to carry an upstream label, and it states a half-truth, because on a cache hit no upstream answered. The interface change it does need is real and worth doing — `handler.zig:79-83` records ruling 20, which deliberately keeps the pool's answering endpoint out of `transport.Client`'s reach. Exposing it touches `src/upstream/`, not the pure core. Do not fall back to `"pool"`: the exact upstream on a SERVFAIL row is the single most useful correlation this redesign adds.
+
+`policy_reason` is a closed enum: `local_record`, `forward_zone`, `non_in_class`, `paused`, `snapshot_unavailable`, `no_match`, plus the existing rule allow/block and blocklist exception/domain/wildcard reasons.
+
+For a CNAME-uncloaked block, the policy fields describe the target's decision and `cname_target` preserves the target responsible.
+
+Every syntactically parsed request that receives a response is logged, including synthesized SERVFAIL. Requests too malformed to identify a question stay counters, not fabricated rows.
+
+No response payloads, answer RR sets, EDNS data or packet bytes are stored. The record explains nxdns's own decision, not the external resolver's answer.
+
+Privacy transforms apply to every new domain-bearing field, not only `domain`: with `hide_domains` on, matched names, CNAME targets and safe-search targets hide consistently.
+
+A one-row `querylog_meta (created_at INTEGER NOT NULL)` table lets the stats and query APIs return a conservative `available_since`, which distinguishes "zero queries" from "history does not exist".
+
+### Historical query detail
+
+Ordered explanation: request (time, domain, client, type, class); group (historical id and name); policy (evaluated or not, allow/block, exact matched rule or list candidate, historical source, whether filtering was paused or unavailable); rewrites (safe-search target, CNAME target); route (local, forward zone, cache, or selected upstream); response (rcode, duration); related actions (test the domain against current policy, view the client, view activity for the same domain or client, view diagnostics in a five-minute window around the query).
+
+Historical and current facts are visually separated. A rule, source or group that no longer exists stays visible as a historical value and is not linked to a different current object.
+
+Live SSE events carry the same provenance shape without a persisted `id`. A frozen live row shows its in-memory detail; no correlation id is invented to link it to a row SQLite has not written.
+
+## Time scoping
+
+One contract: unix seconds UTC, `since` inclusive, `until` exclusive, point data qualifies on `since <= ts < until`, diagnostic episodes qualify when their active interval overlaps the range, current state is labelled "Now" and no historical selector touches it.
+
+URLs: `/overview?period=24h` with the server returning the exact aligned bounds; `/activity?mode=history&since=…&until=…` with `domain`, `client`, `blocked` and the other filters in the URL; `/diagnostics?since=…&until=…&severity=…&component=…`; `/activity?mode=live` with Follow/Freeze as ephemeral UI state. Investigation links always carry absolute bounds, so a viewed incident does not drift as time passes. Timestamps display in the browser's timezone; URLs and APIs stay timezone-independent.
+
+## File mode
+
+One persistent authority line in the shell:
+
+> File-managed · `/etc/nxdns/config.zon` · loaded 19 Aug 2026, 08:42
+
+It sits in the Configuration sub-navigation and appears elsewhere as a compact lock indicator. No full-width banner on every route. The wording is "running configuration loaded from", not "file contents" — the server cannot prove a since-edited file still matches the running process.
+
+A file-managed page uses definition lists for scalars and tables or cards for collections, with human labels and the exact ZON key shown secondarily (`logging.retention_days`). No text inputs, no checkboxes, no Add/Edit/Delete, no disabled form shells, no simulated Save. A short page note says where edits happen and that a restart may be needed.
+
+Runtime actions stay ordinary enabled buttons: pause/resume, update blocklists now, reload certificates, delete an observed undeclared client, login/logout.
+
+Database mode uses the same information architecture with real edit actions, plus a server-owned `restart_pending` boolean. The current client-only restart banner and its local store are removed, so a browser refresh cannot erase the warning.
+
+## API changes
+
+`GET /api/config/status` → `{authority, path, reconciled_at, restart_pending}`. `restart_pending` is process state: database-mode mutations that need a restart set it, a successful restart clears it.
+
+`GET /api/health` becomes explicit about every condition that contributes to degradation — `protection`, `upstreams`, `query_history`, `upstream_history`, `diagnostics`, `disk`, each an object with its own state. The current hidden `history_flush_failing` contribution is eliminated: nothing may degrade the rollup without appearing in the response.
+
+`GET /api/diagnostics?state=&severity=&component=&since=&until=&limit=&before=` returns `{events[], next_before, active:{warnings, errors}}`. `GET /api/diagnostics/{id}` returns one event or 404 after retention. No acknowledgement, dismissal, generic-action or raw-log endpoints.
+
+`GET /api/queries` keeps keyset pagination and its filters; rows gain `rcode`, `route_kind`, `policy_action` and the short policy reason the table needs, and the body gains `coverage: {complete, available_since}`. `GET /api/queries/{id}` returns nested `request` / `policy` / `route` / `response` provenance. `GET /api/queries/live` sends the same object without `id`.
+
+`GET /api/stats` and `/api/stats/timeseries` add `complete` and `available_since`.
+
+Existing mutation endpoints stay specific. Diagnostics introduces no generic "perform remediation" endpoint; it invokes the existing blocklist-refresh and certificate-reload operations.
+
+## Schema changes
+
+`config.db`:
+
+```sql
+CREATE TABLE operational_events (
+    id INTEGER PRIMARY KEY,
+    code TEXT NOT NULL,
+    subject_key TEXT NOT NULL,
+    subject_label TEXT NOT NULL,
+    severity TEXT NOT NULL CHECK (severity IN ('warning', 'error')),
+    first_seen INTEGER NOT NULL,
+    last_seen INTEGER NOT NULL,
+    occurrences INTEGER NOT NULL CHECK (occurrences > 0),
+    resolved_at INTEGER,
+    detail TEXT NOT NULL DEFAULT '',
+    CHECK (resolved_at IS NULL OR resolved_at >= first_seen)
+);
+
+CREATE UNIQUE INDEX idx_operational_events_active
+    ON operational_events(code, subject_key)
+    WHERE resolved_at IS NULL;
+
+CREATE INDEX idx_operational_events_last_seen
+    ON operational_events(last_seen DESC);
+```
+
+Absent from the configuration table lists and the reconciliation delete order.
+
+`querylog.db`: `querylog_meta` and the provenance columns above. No new index — Codex proposed `idx_query_log_rcode`, cut on review, because every rcode question the UI asks is time-scoped and `idx_query_log_ts` already bounds the scan. 700k rows on a Pi 5 do not need a second index for a rare filter, and every index taxes the hot insert path in `logger.zig`.
+
+No new provenance table, no key/value store — household retention makes nullable columns cheaper than a normalized graph of decision objects.
+
+### Entry buffer widths
+
+`logger.zig`'s `Entry` uses fixed buffers sized by `max_reason_len = 32` and `max_upstream_len = 64`, and travels through the `Io.Queue` by value. The new fields roughly triple it: `matched` holds a full pattern, and `cname_target`, `safe_search_target` and `forward_zone` each hold up to 253 bytes. That is fine at `flush_batch = 100`, but the widths are part of this contract and must be set explicitly, not left to whatever the first implementation picks.
+
+`transformed()` must hide `matched`, `cname_target` and `safe_search_target` under `hide_domains`, not only `domain`.
+
+On recreation: query rows, provenance and upstream-minute history reset together as today; the old file stays aside; `config.db` diagnostics survive; a resolved `query_log.recreated` event records the reason, the aside path and the new coverage start; Overview and Activity report the incomplete range instead of charting zero.
+
+## Deletions and their cost
+
+| Deleted | Lost |
+| --- | --- |
+| Separate Query Log and Live pages | separate bookmarks; both modes remain in Activity |
+| Standalone Lookup page | a top-level bookmark; testing remains under Activity |
+| Top-level Groups, Blocklists, Rules, Local DNS, Upstreams, Settings | direct resource navigation; all capabilities remain under task-shaped configuration |
+| Historical upstream table on Overview | at-a-glance period rates; availability stays, failures move to Diagnostics |
+| "last failure · 9h ago" text | nothing actionable; the episode becomes a diagnostic |
+| Detailed DB/log byte gauges | exact component sizes stay in Prometheus; free space stays on Overview |
+| Standalone cache card | one prominent number; cache stays in the timeline and metrics |
+| Ephemeral blocklist `SourceStatusSection` | transient success detail after navigation; durable counters and diagnostics remain |
+| Disabled configuration forms in file mode | the illusion that fields can be edited |
+| Global read-only banner | repeated warning text; authority stays visible once in the shell |
+| Client-only restart banner state | nothing reliable; server-owned `restart_pending` replaces it |
+| Old route aliases and redirects | existing bookmarks break; no permanent duplicate routing layer |
+
+## Build sequence
+
+Each step leaves the app working and shippable, and updates its OpenAPI contract, generated sample, TypeScript types and deterministic tests before landing.
+
+1. **Diagnostics vertical slice.** `operational_events` schema, repository, serialized store, retention, health state, every typed emitter, API, UI, navigation. Recovery paths and deterministic failure-injection tests per event code. The rest of the UI stays intact.
+2. **Query provenance vertical slice.** Query-log fingerprint and schema, metadata table, provenance capture in the handler and logger, parsed-SERVFAIL logging, detail and coverage APIs, the historical detail route, and the `query_log.recreated` emission. Existing list summary fields stay so the current pages keep working.
+
+   **This step destroys the existing query history.** The provenance DDL edit changes the fingerprint, so `querylog_schema.open` recreates the file and keeps the old one aside as `querylog.db.schema-changed-<unix seconds>`. Acceptable pre-v0.1, and `available_since` carries the story in the UI, but it is a consequence of this step and must be stated in its spec and its changelog entry. Its acceptance tests cover the recreate, the aside name and the coverage sequence — which is also the natural first `query_log.recreated` emission.
+
+   Parsed-SERVFAIL logging reverses ruling 20: `handler.zig:507` counts today rather than logging. The `Context` exists at every `servFail` site that follows question parsing. Pre-parse failures correctly stay counters.
+3. **Activity consolidation.** The unified History/Live surface, URL filters, freeze/follow, live detail, current-policy test, historical detail links. Query Log, Live and Lookup routes and code are removed in the same change. Route, SSE, accessibility, reconnect and bounded-buffer tests.
+4. **Overview replacement.** Current status, active diagnostics, one coherent activity section. New health contract and completeness states. The upstream-history table, stale-failure text, detailed DiskCard and cache card go.
+5. **Task-shaped configuration and file mode.** `/api/config/status` and server-owned `restart_pending`. Protection, Resolution and System in both read-only and editable forms. Clients and its detail route redesigned. Old configuration routes replaced atomically; global banner and disabled forms removed.
+6. **Contract closure.** Remove obsolete queries, types, stores, CSS, tests and route fixtures. Regenerate contract samples, update OpenAPI and reference docs, add cross-surface acceptance tests for investigation links, file authority, query-log recreation, active-event recovery and time bounds. Zig, frontend, integration, accessibility and byte-budget checks; no new dependency.
+
+## Codex's least-certain calls
+
+- **Clients in primary navigation.** It earns the slot if identifying unknown devices and checking their group is routine. If Mokhtar almost always reaches a client from a query, Clients moves under Protection and leaves the nav.
+- **Recording the exact selected upstream per query.** It materially improves correlating SERVFAIL queries with upstream events, but needs the pool exchange result to expose the selected target. Drop back to `"pool"` only if that interface change proves invasive and exact resolver identity never changes an action.
+- **90-day / 5,000-event retention.** Conservative fixed bounds, not settings. Change only after measuring real row size and event rate on the Pi; do not add configurable retention pre-emptively.
+
+## Rulings (Fable review, 2026-08-20)
+
+Verdict: build it, with the four cuts folded in above and the rulings below.
+
+**Diagnostics live in `config.db`. Accepted.** The `querylog.db` alternative is self-refuting: that file is recreated on any schema edit or corruption, so the recreation event dies with the thing it describes. A third database needs either its own migration discipline or a recreate policy that loses the events — the same problem with more files. The write-traffic objection is overstated twice: event volume is failure-rate volume, deduplicated, with no `info` severity, so it is near zero in steady state; and `config.db` already takes operational writes, because client rows are materialized from traffic (PLAN §3.5, `clients.last_seen`, `learned_name`). Pre-v0.1 the table is one edit to `ddl_v1` with no migration step. Keeping it out of `delete_order` and `table_names` is enforceable — `config_schema.zig` has tests pinning those lists.
+
+**`group_name`, `source_name` and `matched` stay TEXT. Accepted, and my inconsistency objection was weaker than I put it.** The `domains` table exists because a domain appears on every row, runs to 253 bytes, and feeds `GROUP BY` stats. None of that holds here. The closer precedent is `client_ip`, which is TEXT with the comment "not a FK: log rows are immutable facts" (`querylog_schema.zig:37`). `group_name` is short and mostly `default`; `source_name` and `matched` are non-NULL only on blocked rows; `matched` has cardinality high enough that normalizing buys nothing. Keep both halves of each id/name pair: the id links to the same object across a rename, the name survives a delete, and the detail view needs both.
+
+**Provenance capture does not violate the pure core.** Verified against the code. The policy decision is made in `src/server/handler.zig`, which is already impure and already holds `Io`, the clock and the log call. `matcher.Decision` (`matcher.zig:39-51`) already returns `reason`, `matched` and `source`; the handler throws `matched` and `source` away at `handler.zig:501`. Capture is mostly widening `logger.zig`'s `Entry` and `LogFields`, not threading state through `dns/`, `filter/`, `local/` or `cache/`.
+
+**Six milestones under the harness, one per step.** Do not fold step 6 into step 5: the closure sweep regenerates the contract samples and adds the cross-surface acceptance tests, and it deserves its own verify gate. If anything needs splitting it is step 1, which touches about ten subsystems — store, API and UI first, then the emitters. An event store with three emitters is already shippable and honest.
+
+**One caution, not a blocker.** Diagnostics will be an empty page most of the year. The "No active operational issues" line has to make empty read as healthy, not broken.