milestone 28: query provenance — every logged query is exactly explainable

query rows gain qclass, rcode, group, policy action and reason, the
matched rule or list entry with its source, cname and safe-search
targets, route kind, forward zone, and the resolver that actually
answered — the pool and local markers die. servfails are logged and
name the resolver that lost; post-parse protocol refusals become rows.
a detail page at /queries/:id renders the ordered explanation, and
coverage watermarks distinguish an empty history from a missing one.

the schema fingerprint changes: existing query history is recreated
with the old file kept aside and the reset filed as a resolved
diagnostic. fixes an oversized udp reply being rebuilt as noerror,
which handed clients a truncated nxdomain as success.

specs/milestone-28.md

@@ -0,0 +1,210 @@
+# Milestone 28: query provenance
+
+Redesign step 2 of specs/ui-redesign.md ("Query provenance", "Schema changes", "Entry buffer widths", "API changes", rulings). Every logged query becomes exactly explainable: what policy decided, what matched, where the answer came from, and what the client saw. The existing Query Log/Live/Lookup pages keep working; their replacement is step 3 (milestone 29). Codex spec review folded in (thread 01a02643); its corrections are marked where they changed a ruling.
+
+**This milestone destroys existing query history.** The DDL edit changes the CRC fingerprint (querylog_schema.zig:71-78), so `open` recreates the file and sets the old one aside as `querylog.db.schema-changed-<unix seconds>`. Acceptable pre-v0.1. The changelog entry must say so, and the recreate is the natural first `query_log.recreated` diagnostics emission.
+
+## Sessions
+
+S1 (storage) and S2 (upstream identity) run in parallel — disjoint files. S3 (handler capture) needs both. S4 (web API + contracts) needs S3. S5 (admin) needs S4.
+
+---
+
+## Session S1: querylog schema, repo, logger, coverage watermark
+
+### S1.1 DDL (src/storage/querylog_schema.zig)
+
+`query_log` drops `block_reason` and gains, after the existing columns:
+
+```sql
+qclass INTEGER NOT NULL,
+rcode INTEGER NOT NULL,
+group_id INTEGER,
+group_name TEXT,
+policy_action TEXT NOT NULL,
+policy_reason TEXT NOT NULL,
+matched TEXT,
+source_id INTEGER,
+source_name TEXT,
+cname_target TEXT,
+safe_search_target TEXT,
+route_kind TEXT NOT NULL,
+forward_zone TEXT
+```
+
+Existing columns (`blocked`, `cache_hit`, `upstream`, …) stay — stats and the step-3 filters still use them. No new index (ruling: `idx_query_log_ts` bounds every time-scoped question; the insert path pays for indexes).
+
+New singleton table, in the same DDL string (Codex: enforce one row):
+
+```sql
+CREATE TABLE querylog_meta (
+    id INTEGER PRIMARY KEY CHECK (id = 1),
+    created_at INTEGER NOT NULL,
+    available_since INTEGER NOT NULL
+);
+```
+
+`createFresh` inserts the row with `created_at = now` and `available_since = now + 1` (conservative: an old row logged in the same second as recreation must not let `since = now` claim completeness). **`available_since` is a monotonic coverage watermark, not a constant** (Codex): pruning advances it to the retention cutoff **in the same transaction as the delete** — the repo exposes one transactional `pruneOlderThan(cutoff)` that deletes and advances together; a failure of either rolls back both. It never moves backward. `queries_repo` gains `availableSince() i64`.
+
+The object-count test (querylog_schema.zig:295-318) updates to 5 tables. TEXT id/name pairs are deliberate (ruling: `client_ip` precedent, "log rows are immutable facts"); ids are NOT foreign keys.
+
+### S1.2 Closed enums — neutral module `src/storage/provenance.zig` (Codex: logger already imports queries_repo, so enums owned by logger would cycle)
+
+Imported by logger, queries_repo, handler and web code. Stored as `@tagName` TEXT; the read path parses text back to the enum and treats an unknown value as a data error, not a passthrough string.
+
+- `PolicyAction`: `not_evaluated`, `allow`, `block`.
+- `PolicyReason`: the nine serializable `matcher.Reason` tags (matcher.zig:24-37 minus `none`) plus `local_record`, `forward_zone`, `non_in_class`, `paused`, `snapshot_unavailable`, `no_match`, `protocol_error`. Matcher→PolicyReason conversion is an exhaustive switch. The CNAME case is NOT a `cname:` string prefix any more: `cname_target` non-NULL carries that fact, and `policy_reason` holds the target's own reason.
+- `RouteKind`: `blocked`, `local`, `forward_zone`, `upstream`, `cache`, `rejected`.
+
+`protocol_error` + `rejected` cover post-parse protocol refusals (BADVERS, NOTIMP, EDNS FORMERR) — see S3.4; this extends the redesign's enum list, recorded here as an amendment required by its own "every syntactically parsed request that receives a response is logged" rule.
+
+`blockReason()`, `cname_reason_prefix` and the comptime width proof (handler.zig:66-74, 816-823) die in S3.
+
+### S1.3 Entry widening (src/storage/logger.zig)
+
+`Entry` stays a by-value fixed-buffer struct through `Io.Queue`. New fields with explicit widths:
+
+| field | width / type |
+| --- | --- |
+| `qclass` | `u16` |
+| `rcode` | `u16`, validated ≤ 0xFFF (12-bit EDNS extended RCODE, edns.zig:219; Codex: u8 cannot hold it) |
+| `group_id` | `?i64` |
+| `group_name` | buffer sized by the new `max_group_name_len` (S1.5) |
+| `policy_action` | `provenance.PolicyAction` |
+| `policy_reason` | `provenance.PolicyReason` |
+| `matched` | buffer sized by the rule maximum — the regex engine accepts 256-byte patterns (regex.zig:48), so 256 bytes with a **`u16` length** (the generic `copyInto` returns `u8`; widen it or add a u16 variant — 256 does not fit u8) |
+| `source_id` | `?i64` |
+| `source_name` | buffer sized by the new `max_source_name_len` (S1.5) |
+| `cname_target` | 253-byte buffer |
+| `safe_search_target` | 253-byte buffer |
+| `forward_zone` | 253-byte buffer |
+| `upstream` | widened: sized for the maximum redacted `scheme://host:port` form (host up to 253 bytes), `u16` length — the current 64-byte buffer silently truncates a long valid DoH hostname (Codex); the endpoint host bound gets validated where endpoints are parsed |
+
+`Entry.Fields` gains the borrowed equivalents with `""`/null defaults. `reason_buf`/`max_reason_len` are removed with `block_reason`. The truncation test (logger.zig:610-625) updates.
+
+`transformed()` (logger.zig:234-239) additionally rewrites `matched`, `cname_target`, `safe_search_target` to `hidden_marker` under `hide_domains`. `forward_zone`, `group_name`, `source_name` are configuration labels, not query-derived, and stay visible.
+
+### S1.4 Entry memory budget (Codex: validation permits 1,000,000 queued entries, validate.zig:464, and sse.zig:54 embeds 2,048 entries)
+
+The widened `@sizeOf(Entry)` gets a documented byte budget: `query_log_buffer_max`'s validation upper bound is recomputed so the queue's worst case stays ≤ 64 MiB (`max = 64 MiB / @sizeOf(Entry)`, computed at comptime, stated in the validation reference and CHANGELOG since the accepted range shrinks). Tests assert the default and the new maximum fit the budget, and the SSE hub comment states its embedded-entry cost.
+
+### S1.5 Name caps — neutral module `src/config/limits.zig` (Codex: logger→validate for caps plus validate→logger for `@sizeOf(Entry)` is a cycle)
+
+No length cap exists today for group or source names (validate.zig:721, :853 reject only empty). New `config/limits.zig` owns `max_group_name_len = 64` and `max_source_name_len = 64`; validate.zig enforces them (boundary tests at 64 and 65 bytes, new classification entries per the existing pattern); logger.zig sizes its buffers from them and **exports the computed `query_log_buffer_max` ceiling** (S1.4), which validate.zig imports.
+
+### S1.6 Repo (src/storage/repositories/queries_repo.zig)
+
+- `Row`/`insert_row_sql`/`BatchWriter` bind the new columns; `Sql.capacity` (:262-265) updated.
+- `QueryRow` (read) gains `qclass: u16`, `rcode: u16`, `route_kind`, `policy_action`, `policy_reason` (parsed enums, serialized as strings); loses `block_reason`. NULL→`""` convention unchanged for text.
+- New `detailById(id) ?QueryDetail`: full provenance row joined with domains, for `GET /api/queries/{id}`.
+- `availableSince()` and the transactional prune-plus-advance per S1.1; retention.zig calls the combined operation.
+- `stats_totals_sql`/`timeseries_sql` unchanged.
+
+### S1.7 `query_log.recreated` emission
+
+The existing emission site (app.zig:638) already fires on recreate; it gains the **initial coverage start** (the fresh `available_since`) in its detail alongside reason and aside filename (ui-redesign.md:252 requires the new coverage start). Verify open order (events store vs querylog open) and carry the `OpenResult` rather than reordering database opens if needed.
+
+### S1.8 Acceptance (S1)
+
+- [ ] Fingerprint tests updated; recreate test proves aside name, `querylog_meta` singleton row, and the recreated-event detail carrying the coverage start (end-to-end recreate→coverage test).
+- [ ] Watermark tests: prune advances `available_since` to the cutoff atomically; a failed delete, a failed watermark update, and a failed commit each leave both untouched (rollback proven); it never regresses.
+- [ ] Round-trip test: an `Entry` with every provenance field set survives queue → `toRow` → insert → `detailById` intact (modulo NULL mapping); includes a 256-byte `matched` boundary case.
+- [ ] `transformed()` tests split by flag (Codex: the client is governed by `hide_client_ips`, not `hide_domains`): `hide_domains` hides domain, matched, cname_target, safe_search_target and preserves client_ip; `hide_client_ips` hides client_ip and preserves the rest; both flags leave group/source/zone names.
+- [ ] Buffer-budget tests per S1.4; name-cap boundary tests per S1.5.
+- [ ] `zig build test` 0 failed.
+
+---
+
+## Session S2: selected-resolver identity (every `transport.Client` implementor)
+
+Reverses ruling 20 for the exchange that actually happened. `transport.Client` (transport.zig:325-343) gains a per-call out-parameter: `exchangeFn(ptr, io, query, response_buf, selected: *?[]const u8)`.
+
+**Contract (Codex critical): the identity survives failure.** Each implementation sets `selected.*` to the resolver it is about to attempt, before the attempt; after an all-failed exchange it names the last attempted resolver. The handler consumes it on success AND on the error path — a SERVFAIL row carrying its resolver is the single most useful correlation this redesign adds (ui-redesign.md:157). The slice must stay valid for the query's duration: `Pool` uses `entry.endpoint.url` (Endpoint-owned, stable); the pointer is per-call, threaded into `exchangeLoopLen` (pool.zig:203-266) — never a `*Pool` field (concurrent queries race).
+
+`Pool` reports the **raw** URL; redaction happens in the handler when formatting into the Entry (S3) — `safe_url.redact` is a formatter, so the pool test asserts the raw selected identity and the credential end-to-end test lives in S3/S4 (Codex).
+
+Callers initialize the output to null before the call. `ForwardClient` (forward_client.zig) holds only a parsed `Resolver`, so it gains **owned identity storage** — the formatted resolver text lives in the client and the out-parameter borrows it (Codex: this is real behavior, not a mechanical discard; the "discards only" framing was wrong). Fakes set explicit stable identities.
+
+S2 owns every implementor, fake and call site: pool.zig, forward_client.zig, transport.zig fakes (:635-660), doh_client_live_test.zig:47 and dot_client_live_test.zig:60 (imported by tests.zig — they break compilation if missed, Codex), handler.zig's inline fake (:1074) and its call site (:457, pass-and-ignore — S3 consumes it), doh_server.zig, dot_server.zig, udp/tcp/resolver/phase7 integration tests, web_integration_test.zig. This overlaps S1 on zero files; the handler/web edits are signature-level and complete before S3/S4 start (sequential).
+
+- [ ] Pool tests: winning endpoint reported; failover reports the answerer, not the first attempt; all-failed reports the last attempted; a timeout mid-flight reports the in-flight resolver.
+- [ ] `zig build test` 0 failed.
+
+## Session S3: handler capture (src/server/handler.zig + server tests)
+
+After S1+S2. All provenance assembled in `Context` and passed through `LogFields` → `Entry.Fields`:
+
+- `qclass` from `ctx.q.qclass`; `group_id`/`group_name` from `snapshot.groups[ctx.group]` when snapshot non-null, else null/"" with `policy_reason = snapshot_unavailable` on the unfiltered path.
+- Path mapping: qclass≠IN → `not_evaluated`/`non_in_class`, route `upstream`; pause → `not_evaluated`/`paused`; local → `allow`/`local_record`, route `local`; forward zone → `allow`/`forward_zone`, route `forward_zone` + zone name; cache hits → route `cache`, upstream NULL; upstream answers → route `upstream`, upstream = S2's selected identity redacted via `safe_url` into a Context-local buffer before `Entry.init` — `"pool"`/`pool_upstream` die; allowed matcher decisions (`rule_allow_*`, `blocklist_exception`) → `allow` with exact reason, matched and source captured (today discarded at :442, Codex); no match → `allow`/`no_match`; blocked → `block`/matcher reason, route `blocked`.
+- **`upstream` is non-null only for attempted upstream or forward-zone exchanges, including their failures** (Codex). It is NULL for local, blocked, cache and rejected routes — the `"local"` marker (handler.zig:392) dies with `"pool"`. Asserted per route in the table tests.
+- `matched` + `source_id`/`source_name` from `matcher.Decision` and `snapshot.sources[decision.source.?]`. **Copy `matched` and the uncloak target into Context-local buffers before the uncloak loop continues** — the scratch buffers are reused per chain step (matcher.zig:733-736).
+- CNAME-uncloaked block: policy fields describe the target's decision; `cname_target` holds the target name. `Context.uncloak` (:727-742) widens its return to the target's full decision + name.
+- Safe search: `safe_search_target` = the rewrite target; policy stays `allow`.
+
+### S3.1 rcode capture and the truncation bug (Codex)
+
+`reply`'s UDP truncation rebuild (:522-529) currently rewrites every oversized response to NOERROR — an existing defect: an oversized NXDOMAIN reaches the client as success. Fix here: parse the source rcode before rebuilding and preserve it **via `splitRcode` into both the header and the response OPT** (edns.zig:219-225; header-only preservation loses the upper 8 bits, Codex), then parse the final bytes once for logging. No per-path "known rcode" plumbing: the logged rcode is always derived from the final packet + OPT. Tests: oversized NXDOMAIN keeps NXDOMAIN+TC; an oversized extended-RCODE response keeps the full 12-bit value on the wire and in the log.
+
+### S3.2 servFail logging
+
+Every `servFail` site (8, all inside Context) now replies AND logs: `rcode = servfail`, upstream = the last attempted resolver when the failure came from an exchange, policy/route fields as far as the pipeline got.
+
+### S3.3 Post-parse protocol refusals
+
+BADVERS (:245), NOTIMP (:253) and bad-EDNS FORMERR (:231) answer an identifiable question but precede `Context`. Construct the logging context as soon as one question is parsed and log these as `not_evaluated`/`protocol_error`, route `rejected`, with the actual rcode. Pre-question failures (rate-limit REFUSED, unparseable, qdcount≠1) stay counters — unchanged.
+
+### S3.4 Acceptance (S3)
+
+- [ ] Table-driven provenance tests, one asserted row per path: non-IN, paused, no-snapshot, local, forward-zone, forward-zone cache hit, upstream cache hit, upstream answer (exact redacted URL asserted), rule allow, blocklist exception (source id+name), no-match, rule block, blocklist block with source id+name, CNAME-uncloaked block (cname_target + target's reason), safe-search rewrite, each servFail flavor (exchange-failure case asserts the last-attempted resolver), BADVERS, NOTIMP, bad-EDNS.
+- [ ] Credential tests (Codex: `Endpoint.parse` rejects `@`, so userinfo cannot come through production config): a handler test with an injected userinfo-bearing identity proves redaction before `Entry.init`; the production-config cross-surface sweep (row, SSE, detail all secret-free, using an accepted credential-bearing DoH path shape) lives in S4.
+- [ ] Truncation-rcode regression tests per S3.1.
+- [ ] `pool_upstream` and the `"pool"` marker are gone from provenance producers and serializers (scoped grep — logging fixtures elsewhere are out of scope, Codex).
+- [ ] `zig build test` 0 failed.
+
+## Session S4: web API + contracts (src/web/, openapi.yaml, contract samples)
+
+- List `QueryRow` serialization gains `qclass`, `rcode`, `route_kind`, `policy_action`, `policy_reason`; `block_reason` is gone (S5 updates the admin in the same milestone; `blocked`/`cache_hit`/`upstream` survive, satisfying "existing summary fields stay").
+- New `GET /api/queries/{id}` → 200 nested `{request:{time,domain,client,qtype,qclass}, group:{id,name}, policy:{action,reason,matched,source_id,source_name}, rewrites:{cname_target,safe_search_target}, route:{kind,forward_zone,upstream}, response:{rcode,duration_us}}`; 404 unknown/pruned; 503 no querylog. Route added; the openapi path-count guard (web_integration_test.zig:~2775) and the route-table cardinality assertion (routes.zig:154) update together.
+- **One shared full-provenance DTO** (Codex): define `Provenance` (the nested body above) once; `QueryDetail = {id} + Provenance` and the SSE event = `Provenance` exactly — not "QueryRow minus id" (ui-redesign.md:177 requires full live detail). The list row stays a separate summary projection. The live.zig lockstep test compares field names AND types against the shared DTO.
+- `GET /api/queries` body gains `coverage: {complete: bool, available_since: i64}`; `complete = (filter.since != null and filter.since >= available_since)` with `available_since` the S1 watermark. `/api/stats` and `/api/stats/timeseries` gain the same pair, judged against the period's aligned `since`. Documented in openapi.
+- openapi.yaml: all schemas updated; `/api/queries/{id}` documented; the three enums enumerated. New focused drift guards for QueryRow, QueryDetail (nested objects included), the coverage object and the three enum value sets against `provenance.zig` — comparing field names, types, nullability and requiredness, not names alone (the path-count guard protects none of that, Codex).
+- Contract samples regenerated (justfile goldens recipe); add `get_query_detail` sample; drift test green.
+- [ ] Web integration: detail 200/404/503; coverage fields present and correct across a since-bounded and an unbounded request; keyset walk green. `zig build test` + `-Dintegration` 0 failed.
+
+## Session S5: admin (admin/src)
+
+Scope deliberately thin — Activity is milestone 29:
+
+- `lib/types.ts`: `QueryRow` updated; new `Provenance` + `QueryDetail = {id} & Provenance`; `QueriesPage` gains `coverage`; `LiveQueryEvent = Provenance`. **`LiveRow` becomes a discriminated union** (Codex: a reconnect gap-fetch returns summary `QueryRow`s, which cannot fabricate full provenance): `{kind:"streamed", event: LiveQueryEvent}` | `{kind:"recovered", row: QueryRow}` — recovered rows keep their `id` and link to `/queries/$id`; a shared summary projection feeds the flat `QueryCells` from either arm. ringBuffer.ts and useLiveQueries.ts updated; tests cover both arms and assert the streamed projection drops no field silently.
+- QueryLogPage/Live status cell switches from `block_reason` to `policy_reason`; no other column changes.
+- New route `/queries/$id` + detail page modeled on `diagnosticDetailRoute`: the ordered explanation (request, group, policy, rewrites, route, response), historical facts visually separated from current-state links, related actions (lookup the domain, filtered query-log links). Navigation is a real keyboard-focusable link in the row; row-wide pointer click is an enhancement only (Codex).
+- Coverage: whenever a response says `complete == false`, show "Query history is available from …" against the effective lower bound (not only when the watermark postdates the window, Codex). This touches the current dashboard's stats consumers — permitted: the anti-requirement below bans an Overview redesign, not this notice.
+- [ ] vitest: detail page renders each section from a fixture; hidden-domain fixture renders the marker; coverage-line tests (watermark inside the window and after it); ring-buffer projection test. Typecheck/prettier/oxlint clean.
+
+---
+
+## File ownership
+
+| Files | Session |
+| --- | --- |
+| storage/querylog_schema.zig, storage/provenance.zig (new), config/limits.zig (new), storage/logger.zig, storage/repositories/queries_repo.zig, storage/retention.zig, config/validate.zig, storage tests, app.zig (recreated-event detail) | S1 |
+| upstream/* (incl. both live tests), local/forward_client.zig, server transport fakes + signature fixes in handler.zig/doh/dot/integration/web tests | S2 |
+| server/handler.zig + server tests (behavioral) | S3 |
+| web/*, openapi.yaml, web_integration_test.zig, contract samples | S4 |
+| admin/src/* | S5 |
+
+S1/S2 are disjoint. S2's mechanical signature edits in S3/S4 territory land before those sessions start.
+
+## Anti-requirements
+
+- No response payloads, RR sets, EDNS payloads or packet bytes stored.
+- No new querylog index; no normalized provenance tables.
+- No upstream label on cache hits.
+- No route aliases; no Activity consolidation (step 3); no Overview redesign (step 4) — the coverage notice on existing pages is in scope.
+- No config knobs for any of this.
+
+## Acceptance (milestone complete)
+
+- [ ] `zig build test` and `-Dintegration` 0 failed; `zig fmt --check` clean; admin typecheck/vitest/oxlint/prettier clean; goldens drift test green.
+- [ ] Live smoke on the real binary: a blocked, an allowed-with-match, a cached, an upstream, a forward-zone and a SERVFAIL query each produce a correct detail page; screenshots taken.
+- [ ] CHANGELOG Unreleased entry states the history reset, the aside filename pattern, the truncation-rcode fix, and the shrunk `query_log_buffer_max` range.