nxdns/specs/milestone-28.md

Milestone 28: query provenance

Redesign step 2 of specs/ui-redesign.md ("Query provenance", "Schema changes", "Entry buffer widths", "API changes", rulings). Every logged query becomes exactly explainable: what policy decided, what matched, where the answer came from, and what the client saw. The existing Query Log/Live/Lookup pages keep working; their replacement is step 3 (milestone 29). Codex spec review folded in (thread 01a02643); its corrections are marked where they changed a ruling.

This milestone destroys existing query history. The DDL edit changes the CRC fingerprint (querylog_schema.zig:71-78), so open recreates the file and sets the old one aside as querylog.db.schema-changed-<unix seconds>. Acceptable pre-v0.1. The changelog entry must say so, and the recreate is the natural first query_log.recreated diagnostics emission.

Sessions

S1 (storage) and S2 (upstream identity) run in parallel — disjoint files. S3 (handler capture) needs both. S4 (web API + contracts) needs S3. S5 (admin) needs S4.

---

Session S1: querylog schema, repo, logger, coverage watermark

S1.1 DDL (src/storage/querylog_schema.zig)

query_log drops block_reason and gains, after the existing columns:

qclass INTEGER NOT NULL,
rcode INTEGER NOT NULL,
group_id INTEGER,
group_name TEXT,
policy_action TEXT NOT NULL,
policy_reason TEXT NOT NULL,
matched TEXT,
source_id INTEGER,
source_name TEXT,
cname_target TEXT,
safe_search_target TEXT,
route_kind TEXT NOT NULL,
forward_zone TEXT

Existing columns (blocked, cache_hit, upstream, …) stay — stats and the step-3 filters still use them. No new index (ruling: idx_query_log_ts bounds every time-scoped question; the insert path pays for indexes).

New singleton table, in the same DDL string (Codex: enforce one row):

CREATE TABLE querylog_meta (
    id INTEGER PRIMARY KEY CHECK (id = 1),
    created_at INTEGER NOT NULL,
    available_since INTEGER NOT NULL
);

createFresh inserts the row with created_at = now and available_since = now + 1 (conservative: an old row logged in the same second as recreation must not let since = now claim completeness). available_since is a monotonic coverage watermark, not a constant (Codex): pruning advances it to the retention cutoff in the same transaction as the delete — the repo exposes one transactional pruneOlderThan(cutoff) that deletes and advances together; a failure of either rolls back both. It never moves backward. queries_repo gains availableSince() i64.

The object-count test (querylog_schema.zig:295-318) updates to 5 tables. TEXT id/name pairs are deliberate (ruling: client_ip precedent, "log rows are immutable facts"); ids are NOT foreign keys.

S1.2 Closed enums — neutral module src/storage/provenance.zig (Codex: logger already imports queries_repo, so enums owned by logger would cycle)

Imported by logger, queries_repo, handler and web code. Stored as @tagName TEXT; the read path parses text back to the enum and treats an unknown value as a data error, not a passthrough string.

• PolicyAction: not_evaluated, allow, block.
• PolicyReason: the nine serializable matcher.Reason tags (matcher.zig:24-37 minus none) plus local_record, forward_zone, non_in_class, paused, snapshot_unavailable, no_match, protocol_error. Matcher→PolicyReason conversion is an exhaustive switch. The CNAME case is NOT a cname: string prefix any more: cname_target non-NULL carries that fact, and policy_reason holds the target's own reason.
• RouteKind: blocked, local, forward_zone, upstream, cache, rejected.

protocol_error + rejected cover post-parse protocol refusals (BADVERS, NOTIMP, EDNS FORMERR) — see S3.4; this extends the redesign's enum list, recorded here as an amendment required by its own "every syntactically parsed request that receives a response is logged" rule.

blockReason(), cname_reason_prefix and the comptime width proof (handler.zig:66-74, 816-823) die in S3.

S1.3 Entry widening (src/storage/logger.zig)

Entry stays a by-value fixed-buffer struct through Io.Queue. New fields with explicit widths:

field	width / type
qclass	u16
rcode	u16, validated ≤ 0xFFF (12-bit EDNS extended RCODE, edns.zig:219; Codex: u8 cannot hold it)
group_id	?i64
group_name	buffer sized by the new max_group_name_len (S1.5)
policy_action	provenance.PolicyAction
policy_reason	provenance.PolicyReason
matched	buffer sized by the rule maximum — the regex engine accepts 256-byte patterns (regex.zig:48), so 256 bytes with a u16 length (the generic copyInto returns u8; widen it or add a u16 variant — 256 does not fit u8)
source_id	?i64
source_name	buffer sized by the new max_source_name_len (S1.5)
cname_target	253-byte buffer
safe_search_target	253-byte buffer
forward_zone	253-byte buffer
upstream	widened: sized for the maximum redacted scheme://host:port form (host up to 253 bytes), u16 length — the current 64-byte buffer silently truncates a long valid DoH hostname (Codex); the endpoint host bound gets validated where endpoints are parsed

Entry.Fields gains the borrowed equivalents with ""/null defaults. reason_buf/max_reason_len are removed with block_reason. The truncation test (logger.zig:610-625) updates.

transformed() (logger.zig:234-239) additionally rewrites matched, cname_target, safe_search_target to hidden_marker under hide_domains. forward_zone, group_name, source_name are configuration labels, not query-derived, and stay visible.

S1.4 Entry memory budget (Codex: validation permits 1,000,000 queued entries, validate.zig:464, and sse.zig:54 embeds 2,048 entries)

The widened @sizeOf(Entry) gets a documented byte budget: query_log_buffer_max's validation upper bound is recomputed so the queue's worst case stays ≤ 64 MiB (max = 64 MiB / @sizeOf(Entry), computed at comptime, stated in the validation reference and CHANGELOG since the accepted range shrinks). Tests assert the default and the new maximum fit the budget, and the SSE hub comment states its embedded-entry cost.

S1.5 Name caps — neutral module src/config/limits.zig (Codex: logger→validate for caps plus validate→logger for @sizeOf(Entry) is a cycle)

No length cap exists today for group or source names (validate.zig:721, :853 reject only empty). New config/limits.zig owns max_group_name_len = 64 and max_source_name_len = 64; validate.zig enforces them (boundary tests at 64 and 65 bytes, new classification entries per the existing pattern); logger.zig sizes its buffers from them and exports the computed query_log_buffer_max ceiling (S1.4), which validate.zig imports.

S1.6 Repo (src/storage/repositories/queries_repo.zig)

• Row/insert_row_sql/BatchWriter bind the new columns; Sql.capacity (:262-265) updated.
• QueryRow (read) gains qclass: u16, rcode: u16, route_kind, policy_action, policy_reason (parsed enums, serialized as strings); loses block_reason. NULL→"" convention unchanged for text.
• New detailById(id) ?QueryDetail: full provenance row joined with domains, for GET /api/queries/{id}.
• availableSince() and the transactional prune-plus-advance per S1.1; retention.zig calls the combined operation.
• stats_totals_sql/timeseries_sql unchanged.

S1.7 query_log.recreated emission

The existing emission site (app.zig:638) already fires on recreate; it gains the initial coverage start (the fresh available_since) in its detail alongside reason and aside filename (ui-redesign.md:252 requires the new coverage start). Verify open order (events store vs querylog open) and carry the OpenResult rather than reordering database opens if needed.

S1.8 Acceptance (S1)

• Fingerprint tests updated; recreate test proves aside name, querylog_meta singleton row, and the recreated-event detail carrying the coverage start (end-to-end recreate→coverage test).
• Watermark tests: prune advances available_since to the cutoff atomically; a failed delete, a failed watermark update, and a failed commit each leave both untouched (rollback proven); it never regresses.
• Round-trip test: an Entry with every provenance field set survives queue → toRow → insert → detailById intact (modulo NULL mapping); includes a 256-byte matched boundary case.
• transformed() tests split by flag (Codex: the client is governed by hide_client_ips, not hide_domains): hide_domains hides domain, matched, cname_target, safe_search_target and preserves client_ip; hide_client_ips hides client_ip and preserves the rest; both flags leave group/source/zone names.
• Buffer-budget tests per S1.4; name-cap boundary tests per S1.5.
• zig build test 0 failed.

---

Session S2: selected-resolver identity (every transport.Client implementor)

Reverses ruling 20 for the exchange that actually happened. transport.Client (transport.zig:325-343) gains a per-call out-parameter: exchangeFn(ptr, io, query, response_buf, selected: *?[]const u8).

Contract (Codex critical): the identity survives failure. Each implementation sets selected.* to the resolver it is about to attempt, before the attempt; after an all-failed exchange it names the last attempted resolver. The handler consumes it on success AND on the error path — a SERVFAIL row carrying its resolver is the single most useful correlation this redesign adds (ui-redesign.md:157). The slice must stay valid for the query's duration: Pool uses entry.endpoint.url (Endpoint-owned, stable); the pointer is per-call, threaded into exchangeLoopLen (pool.zig:203-266) — never a *Pool field (concurrent queries race).

Pool reports the raw URL; redaction happens in the handler when formatting into the Entry (S3) — safe_url.redact is a formatter, so the pool test asserts the raw selected identity and the credential end-to-end test lives in S3/S4 (Codex).

Callers initialize the output to null before the call. ForwardClient (forward_client.zig) holds only a parsed Resolver, so it gains owned identity storage — the formatted resolver text lives in the client and the out-parameter borrows it (Codex: this is real behavior, not a mechanical discard; the "discards only" framing was wrong). Fakes set explicit stable identities.

S2 owns every implementor, fake and call site: pool.zig, forward_client.zig, transport.zig fakes (:635-660), doh_client_live_test.zig:47 and dot_client_live_test.zig:60 (imported by tests.zig — they break compilation if missed, Codex), handler.zig's inline fake (:1074) and its call site (:457, pass-and-ignore — S3 consumes it), doh_server.zig, dot_server.zig, udp/tcp/resolver/phase7 integration tests, web_integration_test.zig. This overlaps S1 on zero files; the handler/web edits are signature-level and complete before S3/S4 start (sequential).

• Pool tests: winning endpoint reported; failover reports the answerer, not the first attempt; all-failed reports the last attempted; a timeout mid-flight reports the in-flight resolver.
• zig build test 0 failed.

Session S3: handler capture (src/server/handler.zig + server tests)

After S1+S2. All provenance assembled in Context and passed through LogFields → Entry.Fields:

• qclass from ctx.q.qclass; group_id/group_name from snapshot.groups[ctx.group] when snapshot non-null, else null/"" with policy_reason = snapshot_unavailable on the unfiltered path.
• Path mapping: qclass≠IN → not_evaluated/non_in_class, route upstream; pause → not_evaluated/paused; local → allow/local_record, route local; forward zone → allow/forward_zone, route forward_zone + zone name; cache hits → route cache, upstream NULL; upstream answers → route upstream, upstream = S2's selected identity redacted via safe_url into a Context-local buffer before Entry.init — "pool"/pool_upstream die; allowed matcher decisions (rule_allow_*, blocklist_exception) → allow with exact reason, matched and source captured (today discarded at :442, Codex); no match → allow/no_match; blocked → block/matcher reason, route blocked.
• upstream is non-null only for attempted upstream or forward-zone exchanges, including their failures (Codex). It is NULL for local, blocked, cache and rejected routes — the "local" marker (handler.zig:392) dies with "pool". Asserted per route in the table tests.
• matched + source_id/source_name from matcher.Decision and snapshot.sources[decision.source.?]. Copy matched and the uncloak target into Context-local buffers before the uncloak loop continues — the scratch buffers are reused per chain step (matcher.zig:733-736).
• CNAME-uncloaked block: policy fields describe the target's decision; cname_target holds the target name. Context.uncloak (:727-742) widens its return to the target's full decision + name.
• Safe search: safe_search_target = the rewrite target; policy stays allow.

S3.1 rcode capture and the truncation bug (Codex)

reply's UDP truncation rebuild (:522-529) currently rewrites every oversized response to NOERROR — an existing defect: an oversized NXDOMAIN reaches the client as success. Fix here: parse the source rcode before rebuilding and preserve it via splitRcode into both the header and the response OPT (edns.zig:219-225; header-only preservation loses the upper 8 bits, Codex), then parse the final bytes once for logging. No per-path "known rcode" plumbing: the logged rcode is always derived from the final packet + OPT. Tests: oversized NXDOMAIN keeps NXDOMAIN+TC; an oversized extended-RCODE response keeps the full 12-bit value on the wire and in the log.

S3.2 servFail logging

Every servFail site (8, all inside Context) now replies AND logs: rcode = servfail, upstream = the last attempted resolver when the failure came from an exchange, policy/route fields as far as the pipeline got.

S3.3 Post-parse protocol refusals

BADVERS (:245), NOTIMP (:253) and bad-EDNS FORMERR (:231) answer an identifiable question but precede Context. Construct the logging context as soon as one question is parsed and log these as not_evaluated/protocol_error, route rejected, with the actual rcode. Pre-question failures (rate-limit REFUSED, unparseable, qdcount≠1) stay counters — unchanged.

S3.4 Acceptance (S3)

• Table-driven provenance tests, one asserted row per path: non-IN, paused, no-snapshot, local, forward-zone, forward-zone cache hit, upstream cache hit, upstream answer (exact redacted URL asserted), rule allow, blocklist exception (source id+name), no-match, rule block, blocklist block with source id+name, CNAME-uncloaked block (cname_target + target's reason), safe-search rewrite, each servFail flavor (exchange-failure case asserts the last-attempted resolver), BADVERS, NOTIMP, bad-EDNS.
• Credential tests (Codex: Endpoint.parse rejects @, so userinfo cannot come through production config): a handler test with an injected userinfo-bearing identity proves redaction before Entry.init; the production-config cross-surface sweep (row, SSE, detail all secret-free, using an accepted credential-bearing DoH path shape) lives in S4.
• Truncation-rcode regression tests per S3.1.
• pool_upstream and the "pool" marker are gone from provenance producers and serializers (scoped grep — logging fixtures elsewhere are out of scope, Codex).
• zig build test 0 failed.

Session S4: web API + contracts (src/web/, openapi.yaml, contract samples)

• List QueryRow serialization gains qclass, rcode, route_kind, policy_action, policy_reason; block_reason is gone (S5 updates the admin in the same milestone; blocked/cache_hit/upstream survive, satisfying "existing summary fields stay").
• New GET /api/queries/{id} → 200 nested {request:{time,domain,client,qtype,qclass}, group:{id,name}, policy:{action,reason,matched,source_id,source_name}, rewrites:{cname_target,safe_search_target}, route:{kind,forward_zone,upstream}, response:{rcode,duration_us}}; 404 unknown/pruned; 503 no querylog. Route added; the openapi path-count guard (web_integration_test.zig:~2775) and the route-table cardinality assertion (routes.zig:154) update together.
• One shared full-provenance DTO (Codex): define Provenance (the nested body above) once; QueryDetail = {id} + Provenance and the SSE event = Provenance exactly — not "QueryRow minus id" (ui-redesign.md:177 requires full live detail). The list row stays a separate summary projection. The live.zig lockstep test compares field names AND types against the shared DTO.
• GET /api/queries body gains coverage: {complete: bool, available_since: i64}; complete = (filter.since != null and filter.since >= available_since) with available_since the S1 watermark. /api/stats and /api/stats/timeseries gain the same pair, judged against the period's aligned since. Documented in openapi.
• openapi.yaml: all schemas updated; /api/queries/{id} documented; the three enums enumerated. New focused drift guards for QueryRow, QueryDetail (nested objects included), the coverage object and the three enum value sets against provenance.zig — comparing field names, types, nullability and requiredness, not names alone (the path-count guard protects none of that, Codex).
• Contract samples regenerated (justfile goldens recipe); add get_query_detail sample; drift test green.
• Web integration: detail 200/404/503; coverage fields present and correct across a since-bounded and an unbounded request; keyset walk green. zig build test + -Dintegration 0 failed.

Session S5: admin (admin/src)

Scope deliberately thin — Activity is milestone 29:

• lib/types.ts: QueryRow updated; new Provenance + QueryDetail = {id} & Provenance; QueriesPage gains coverage; LiveQueryEvent = Provenance. LiveRow becomes a discriminated union (Codex: a reconnect gap-fetch returns summary QueryRows, which cannot fabricate full provenance): {kind:"streamed", event: LiveQueryEvent} | {kind:"recovered", row: QueryRow} — recovered rows keep their id and link to /queries/$id; a shared summary projection feeds the flat QueryCells from either arm. ringBuffer.ts and useLiveQueries.ts updated; tests cover both arms and assert the streamed projection drops no field silently.
• QueryLogPage/Live status cell switches from block_reason to policy_reason; no other column changes.
• New route /queries/$id + detail page modeled on diagnosticDetailRoute: the ordered explanation (request, group, policy, rewrites, route, response), historical facts visually separated from current-state links, related actions (lookup the domain, filtered query-log links). Navigation is a real keyboard-focusable link in the row; row-wide pointer click is an enhancement only (Codex).
• Coverage: whenever a response says complete == false, show "Query history is available from …" against the effective lower bound (not only when the watermark postdates the window, Codex). This touches the current dashboard's stats consumers — permitted: the anti-requirement below bans an Overview redesign, not this notice.
• vitest: detail page renders each section from a fixture; hidden-domain fixture renders the marker; coverage-line tests (watermark inside the window and after it); ring-buffer projection test. Typecheck/prettier/oxlint clean.

---

File ownership

Files	Session
storage/querylog_schema.zig, storage/provenance.zig (new), config/limits.zig (new), storage/logger.zig, storage/repositories/queries_repo.zig, storage/retention.zig, config/validate.zig, storage tests, app.zig (recreated-event detail)	S1
upstream/* (incl. both live tests), local/forward_client.zig, server transport fakes + signature fixes in handler.zig/doh/dot/integration/web tests	S2
server/handler.zig + server tests (behavioral)	S3
web/*, openapi.yaml, web_integration_test.zig, contract samples	S4
admin/src/*	S5

S1/S2 are disjoint. S2's mechanical signature edits in S3/S4 territory land before those sessions start.

Anti-requirements

• No response payloads, RR sets, EDNS payloads or packet bytes stored.
• No new querylog index; no normalized provenance tables.
• No upstream label on cache hits.
• No route aliases; no Activity consolidation (step 3); no Overview redesign (step 4) — the coverage notice on existing pages is in scope.
• No config knobs for any of this.

Acceptance (milestone complete)

• zig build test and -Dintegration 0 failed; zig fmt --check clean; admin typecheck/vitest/oxlint/prettier clean; goldens drift test green.
• Live smoke on the real binary: a blocked, an allowed-with-match, a cached, an upstream, a forward-zone and a SERVFAIL query each produce a correct detail page; screenshots taken.
• CHANGELOG Unreleased entry states the history reset, the aside filename pattern, the truncation-rcode fix, and the shrunk query_log_buffer_max range.