14 Commits
Author SHA1 Message Date
mokhtar 8cc85b487a build: bump version to 0.0.14
Gates / frontend (push) Successful in 1m47s
Gates / test (push) Successful in 2m32s
Gates / test-aarch64 (push) Successful in 8m30s
Gates / package (push) Successful in 4m37s
Gates / container (push) Successful in 10s
CI / gates (push) Successful in 29m58s
Release / guard (push) Successful in 34s
Gates / frontend (push) Successful in 1m43s
Gates / test (push) Successful in 2m26s
Gates / test-aarch64 (push) Successful in 7m42s
Gates / package (push) Successful in 59s
Gates / container (push) Successful in 14s
Release / gates (push) Successful in 11m28s
Release / publish (push) Successful in 7m4s
2026-08-28 17:57:09 +02:00
mokhtar 0f01c2fbd7 storage: version querylog.db and migrate it in place, never reset a healthy file
Gates / frontend (push) Successful in 2m8s
Gates / test (push) Successful in 2m46s
Gates / test-aarch64 (push) Successful in 8m38s
Gates / package (push) Successful in 4m39s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 31m58s
querylog.db carries a schema version; migrations run at startup as one transaction after a vacuumed 0600 backup, and every failure refuses startup (exit 2, no systemd restart loop) instead of starting empty. corruption is the only automatic recreate left. the cut gate now requires a fixture-proven migration or an explicit versioned break with restore instructions, and locks shipped migration files and fixtures byte-for-byte.
2026-08-28 17:56:19 +02:00
mokhtar c9701fae85 build: bump version to 0.0.13
Gates / frontend (push) Successful in 1m35s
Gates / test (push) Successful in 2m23s
Gates / test-aarch64 (push) Successful in 8m13s
Gates / package (push) Successful in 4m15s
Gates / container (push) Successful in 10s
CI / gates (push) Successful in 23m19s
Release / guard (push) Successful in 34s
Gates / frontend (push) Successful in 1m39s
Gates / test (push) Successful in 2m19s
Gates / test-aarch64 (push) Successful in 7m25s
Gates / package (push) Successful in 48s
Release / publish (push) Successful in 6m36s
Gates / container (push) Successful in 17s
Release / gates (push) Successful in 10m52s
2026-08-27 21:46:08 +02:00
mokhtar a3aa7febb4 upstream: one absolute per-query budget across queueing and failover
Gates / frontend (push) Successful in 1m46s
Gates / test (push) Successful in 2m32s
Gates / package (push) Successful in 4m20s
Gates / test-aarch64 (push) Successful in 8m15s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 30m33s
waiting for a slot now spends the query budget; truncated attempts that
expire fault the budget, not the upstream, and are never attributed.
admission sweeps in priority order before blocking. forward zones spend
read_timeout_ms once across udp, truncation and tcp. adds
nxdns_upstream_budget_exhausted_total and a 64-upstream validation limit.
2026-08-27 21:10:43 +02:00
mokhtar d2e12ae0e2 build: bump version to 0.0.12
Gates / test (push) Successful in 2m20s
Gates / frontend (push) Successful in 1m33s
Gates / test-aarch64 (push) Successful in 8m13s
Gates / package (push) Successful in 4m33s
Gates / container (push) Successful in 10s
CI / gates (push) Successful in 27m59s
Release / guard (push) Successful in 1m45s
Gates / frontend (push) Successful in 1m59s
Gates / test (push) Successful in 2m24s
Gates / test-aarch64 (push) Successful in 7m33s
Gates / package (push) Successful in 45s
Gates / container (push) Successful in 14s
Release / gates (push) Successful in 25m53s
Release / publish (push) Successful in 8m32s
2026-08-27 17:49:06 +02:00
mokhtar 6a0630c288 overview: one endpoint, live projections and a response cache (m36)
Gates / frontend (push) Successful in 2m6s
Gates / test (push) Successful in 2m57s
Gates / test-aarch64 (push) Successful in 8m31s
Gates / package (push) Successful in 4m19s
Gates / container (push) Failing after 2s
CI / gates (push) Failing after 26m21s
2026-08-27 17:48:20 +02:00
mokhtar a261dc2aa3 build: bump version to 0.0.11
Gates / test (push) Successful in 2m12s
Gates / test-aarch64 (push) Successful in 7m1s
Gates / package (push) Successful in 42s
Release / gates (push) Successful in 10m8s
Gates / container (push) Successful in 10s
Release / publish (push) Successful in 10m22s
Gates / test-aarch64 (push) Successful in 9m19s
Gates / package (push) Successful in 10m49s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 25m8s
Gates / frontend (push) Successful in 1m42s
Gates / frontend (push) Successful in 3m55s
Gates / test (push) Successful in 4m41s
Release / guard (push) Successful in 1m48s
2026-08-24 18:41:46 +02:00
mokhtar 90b85ef954 changelog: 0.0.11
Gates / test (push) Successful in 2m6s
Gates / test-aarch64 (push) Successful in 7m31s
Gates / frontend (push) Successful in 1m36s
Gates / package (push) Failing after 16m49s
Gates / container (push) Skipped
CI / gates (push) Failing after 35m55s
2026-08-24 18:40:29 +02:00
mokhtar ad26aca198 admin: draw the overview charts with visx
the hand-written scale, tick, stacking and arc math is replaced by visx 4.0.0
primitives; rendering, colours and themes stay the app's own. all four charts
share one hover treatment: the client chart gains the tooltip and dimming the
query timeline had, the donuts gain both, an open tooltip follows a data
refresh instead of going stale, and it retires when the window rolls. the
timeline's third series is named allowed instead of other, and the client
chart's other aggregate disappears from a window where it counted nothing.
licenses gain the isc text for the bundled d3 modules.
2026-08-24 18:40:17 +02:00
mokhtar 08d756cc87 build: bump version to 0.0.10
Gates / frontend (push) Successful in 2m1s
Gates / test (push) Successful in 3m6s
Gates / test-aarch64 (push) Successful in 9m28s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 14s
CI / gates (push) Successful in 22m4s
Release / guard (push) Successful in 1m32s
Gates / frontend (push) Successful in 1m28s
Gates / test (push) Successful in 1m53s
Gates / package (push) Successful in 23s
Gates / container (push) Successful in 17s
Release / gates (push) Successful in 9m33s
Gates / test-aarch64 (push) Successful in 6m58s
Release / publish (push) Successful in 10m46s
2026-08-24 00:05:33 +02:00
mokhtar 44ecc2c4ae changelog: 0.0.10
Gates / frontend (push) Successful in 1m31s
Gates / test (push) Successful in 1m53s
Gates / test-aarch64 (push) Successful in 7m2s
Gates / package (push) Successful in 5m44s
Gates / container (push) Successful in 9s
CI / gates (push) Successful in 48m46s
2026-08-24 00:05:21 +02:00
mokhtar ce143d1d87 db-mode config changes apply live in-process
Gates / frontend (push) Successful in 1m43s
Gates / test (push) Successful in 2m14s
Gates / test-aarch64 (push) Successful in 8m3s
Gates / package (push) Successful in 5m42s
Gates / container (push) Successful in 54s
CI / gates (push) Successful in 50m24s
settings and upstream writes now follow a prepare, commit, publish, retire
contract: candidates are built and validated before the database transaction,
published as infallible pointer swaps, and old generations retire after their
readers drain. per-query policy values snapshot once per query; upstream pool,
cache, rate limiter, sessions, api limiter, log sink, blocklist scheduler and
the query-log queue each gained one named live operation. restart_required
shrinks from every scalar key to the bind keys and web.enabled; the admin ui
drops its restart notices for everything else. file mode is unchanged.
2026-08-24 00:04:28 +02:00
mokhtar f7f4c8be09 admin: only the content region scrolls on wide screens
the shell grid grew past the viewport and scrolled the document,
carrying the sidebar with it. the shell is now viewport-height at the
wide breakpoint with main as the sole scroll container; the nav list
scrolls inside the pinned rail; router scroll restoration targets the
inner scroller so navigation resets it and back/forward restores it.
2026-08-23 15:17:51 +02:00
mokhtar fe71efe335 cut: schema gate — refuse to release an undisclosed querylog schema change
the gate recomputes the previous release tag's ddl fingerprint from the
remote peeled object and compares it against the tree's; a change must
be disclosed by 'resets your query history' in the version's changelog
section. the 0.0.9 reset shipped with an announcement claiming no
schema change; this makes the impact mechanical instead of remembered.
2026-08-23 15:07:27 +02:00
123 changed files with 20293 additions and 4393 deletions
+17
View File
@@ -32,6 +32,23 @@ Any *other* failure text is real. Trust the summary line: `zig build test` exiti
One trap: running a cached test binary by hand with `--listen=-` aborts with `internal test runner failure: EndOfStream`. That is not a teardown bug; the IPC runner is talking to a closed stdin because no build runner is on the other end. Run the binary with no arguments to get the plain stdio report.
## Debug-mode miscompile: a `bool` live across an atomic read-modify-write
In Debug the x86_64 self-hosted backend is the default, and zig 0.16.0's atomic read-modify-write lowering there does not invalidate a `bool` the register allocator is still tracking in EFLAGS. The `bool` silently becomes the flags the `lock xadd` left behind. Release modes go through LLVM and are unaffected, so this can only ever break `zig build test`, never a shipped binary.
The shape to avoid is a comparison whose result stays live across `fetchAdd`/`fetchSub`/`@atomicRmw` and is then branched on:
```zig
const idle = old.refs == 0; // sete 0x50(%rsp) -- correct
_ = self.published.fetchAdd(1, .monotonic); // lock xadd %rdi,(%rsi)
// sete 0x51(%rsp) -- bogus, reads EFLAGS from the xadd
return if (idle) old else null; // branches on 0x51, not 0x50
```
That function returns `null` for every input. `fetchAdd` is the only trigger: a plain `+= 1`, an atomic `load`, and an atomic `store` in the same slot all compile correctly, and inserting any call (including `std.debug.print`) between the comparison and the branch forces a spill that hides it. Build the same file with `-fllvm` or `-OReleaseSafe` to confirm a suspected instance.
`Owner.published` in `src/upstream/owner.zig` is a plain `u64` under the owner's mutex for this reason. Do not "modernize" it to `std.atomic.Value(u64)`.
## Regenerating the contract samples
`admin/src/lib/contractSamples.gen.ts` is a committed golden of canonicalized API responses, byte-compared against the live server by a `-Dintegration` test and type-checked by `tsc`. After a deliberate API contract change, regenerate it with:
+54
View File
@@ -4,6 +4,60 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
## [0.0.14] - 2026-08-28
Schema changes stop costing you your query history. querylog.db is now version-stamped and migrated in place; the server refuses to start rather than ever reset a healthy file, and the release tooling refuses to ship a schema change that is neither migratable nor explicitly disclosed with recovery steps. Three releases (0.0.6, 0.0.9, 0.0.12) each discarded the log on upgrade; this ends that.
### Changed
- **querylog.db is migrated in place.** The file now carries a schema version, and a release that changes the schema ships a migration that runs at startup: one consistent backup (`querylog.db.pre-migrate-<timestamp>`, mode 0600, only the most recent kept), then every step and the version stamp in a single transaction. A failure before the commit rolls back and leaves your file exactly as it was.
- **The server refuses instead of resetting.** A querylog.db it cannot use — newer than the binary, older than 0.0.12, or mid-migration failure — is left untouched and the server exits with a clear message instead of setting the file aside and starting an empty log. The exit code (2) tells systemd not to restart-loop a deliberate refusal. Corruption is the only case that still sets a file aside automatically.
- **The release gate now enforces the contract.** A schema change cannot be tagged unless it either ships a working migration (proven in CI against a frozen fixture of the previous schema, with shipped migration files locked byte-for-byte once released) or explicitly declares a break — which requires a version bump the server refuses on, a reset disclosure, and step-by-step restore instructions in this file.
**One hazard to know when downgrading.** The first start under this release restamps querylog.db from the old fingerprint to version 1 (contents untouched). If you later downgrade to 0.0.13 or older, that binary treats the new stamp as a schema mismatch, moves your file aside as `querylog.db.schema-changed-<timestamp>`, and starts an empty log. To recover: return to 0.0.14 or newer, stop the server, move the empty `querylog.db` away and delete its `querylog.db-wal` and `querylog.db-shm` files (leaving them would corrupt the restored file), rename the `.schema-changed-<timestamp>` file back to `querylog.db`, and start.
## [0.0.13] - 2026-08-27
The upstream query budget becomes one honest deadline. A busy network no longer blames a healthy standby for running out of time, and a query burst no longer queues invisibly until everything answers SERVFAIL at once.
### Fixed
- **The per-query upstream budget is now one absolute deadline, spent by everything that blocks.** Waiting for a free slot on a saturated upstream now spends the query's `upstream.total_timeout_ms` budget just like the exchange itself, instead of being invisible to it — under a burst, queries used to wait out their whole budget in the queue and then start attempts they could never finish. An attempt near the end of the budget runs truncated, and when a truncated attempt runs out of time that is evidence about the budget, not the upstream: it no longer counts against that upstream's health or success rate, and the query log no longer names an upstream that was given no fair chance. A field incident produced 279 rows blaming a standby whose health counters read zero for zero; those rows now attribute nothing.
- **A query no longer blocks behind a saturated upstream while another has capacity.** Admission sweeps the upstreams in priority order and takes the first free slot; priority now means the order among upstreams that can be admitted right now, and a query blocks only when nothing has capacity — on the highest-priority eligible upstream, bounded by the remaining budget.
- **Conditional forward zones spend `upstream.read_timeout_ms` once per query.** A UDP attempt, a truncated answer and the TCP retry now share the one budget instead of taking a fresh one each, so a slow zone resolver can no longer stretch a single query to several times the configured timeout.
### Added
- **`nxdns_upstream_budget_exhausted_total`.** A pool-wide counter of queries whose budget ran out — in the queue or mid-attempt — before any upstream answered. It carries no per-upstream label on purpose: running out of budget is a fact about the pool.
- **A configuration with more than 64 enabled upstreams is rejected at validation** with a clear message, instead of tripping an internal limit at startup.
## [0.0.12] - 2026-08-27
Overview stops re-reading the whole query log. One endpoint, one snapshot, pre-aggregated buckets — a 30-day view now costs the same on a month of history as on a day of it. Read the upgrade note first: it resets your query history.
### Changed
- **Upgrading resets your query history.** The query-log schema gains the aggregate tables described below, and `querylog.db` is never migrated: the first start after the upgrade sets the old file aside (kept on disk next to the new one, named with the reason) and begins a fresh log. Settings, groups, blocklists and every other configuration are untouched.
- **The Overview is served by one endpoint, `GET /api/overview`.** It replaces `GET /api/stats`, `/api/stats/timeseries`, `/api/stats/types`, `/api/stats/routes` and `/api/stats/clients`, which are gone. The five panels now come from a single database snapshot, so they can no longer disagree with each other, and the page shows one loading and one error state instead of five.
- **Query statistics are pre-aggregated as they are written.** The query log now maintains 30-minute aggregate tables in the same transaction that stores the rows, and the 24-hour, 7-day and 30-day views read those instead of scanning every logged query. The cost of opening the Overview no longer grows with the size of the log: measured at three million rows, the 30-day view went from roughly eight-tenths of a second of scanning to under fifty milliseconds, at the price of about ten percent on each background write batch and ~1.5 MB of disk. The server also keeps the most recent response per period in memory and serves repeat polls from it while nothing has changed — until new queries land, retention prunes, or the period's time window rolls forward — so on a quiet network most of the steady 30-second refreshes do no database work at all.
The Overview charts move to visx and grow up: one hover treatment across all four, honest labels, and maintained d3 math under the app's own rendering.
### Changed
- **The Overview charts are drawn with visx.** The hand-written chart layout code is replaced by visx 4.0.0 primitives — maintained d3 math for the scales, ticks, stacking and arcs — while the rendering, colours and themes stay the app's own. The charts read as before, with four behaviour improvements: all four charts now share the same hover treatment (the per-client chart and both donuts gain the tooltip and dimming the query timeline already had, so pointing at a ring segment names it, its count and its share), an open tooltip follows a data refresh instead of showing stale counts, and it retires cleanly when the time window rolls. The admin bundle grows by about 68 KB and stays under its size budget.
- **The query timeline's third series is called "Allowed".** What the chart called "Other" is every query that was neither blocked nor served from cache — answered upstream, from a local record or a forward zone — so it is now named for what it is rather than for the subtraction that produces it. It stays on the chart even in a window where nothing was allowed, alongside Blocked and Cached: all three name a kind of answer a query can get, and a period where every query was blocked or cached is worth seeing.
- **The client chart drops "Other" in a window where it counted nothing.** That series aggregates the clients outside the top eight, so when it counts nothing there is nothing being aggregated, and a legend entry, a tooltip row and a table column that exist only to say "zero" are noise. The named clients stay even at zero, because a client that went quiet is a fact about the window.
## [0.0.10] - 2026-08-24
Configuration goes live: when the database owns the configuration, saving a setting reconfigures the running process instead of asking for a restart. The restart-required set shrinks to the listen sockets and the admin interface switch.
### Changed
- **Almost every settings change now applies while the server runs.** When the database owns the configuration, saving a setting takes effect immediately — the blocking response, upstream timeouts, cache size, rate limits, session lifetime, log level and destination, blocklist update schedule, privacy flags, disk thresholds and the query-log buffer all reconfigure the running process, exactly as Pi-hole and AdGuard Home do. Nothing is written to the database unless the running server already accepted it, so the API can never report a value the process refused. The restart-required set shrinks from every scalar key to the twelve that genuinely need one: listen addresses and ports, and turning the admin interface itself on or off. The admin pages drop their restart notices for everything else, and an upstream edit — the loudest offender — now applies to the next query. A configuration file still works the way it always has: edit the file, restart the process.
- **The release cut refuses to ship an undisclosed query-log schema change.** `zig build cut` now compares the `querylog.db` schema fingerprint of the previous release tag against this tree's, and when they differ it requires the changelog section for the version being cut to state that the upgrade discards the stored query history. 0.0.9 changed the schema and its announcement did not mention it; the file is never migrated, so that upgrade silently threw every logged query away.
## [0.0.9] - 2026-08-22
Query provenance: every logged query becomes exactly explainable — what the policy decided, what matched, where the answer came from and what the client saw. The handler records all of it as the reply goes out, `query_log` stores it, and a detail page reads one query back in the order the pipeline decided it. Read the upgrade note below first: it resets your query history.
+9 -7
View File
@@ -85,13 +85,13 @@ Verified: 0.16.0 ships `std.crypto.tls.Client` only. There is no server-side TLS
Two SQLite files with opposite write profiles, isolated from each other:
- **`config.db`** — small, precious, rarely written: groups, clients, prefixes, upstreams, blocklist source metadata, rules, local records, forward zones, settings, schema version.
- **`querylog.db`** — high-churn, large, expendable: query log + its own private `domains` dimension table. Client identity stored as **IP text**, not a FK into config — log rows are immutable facts and must not point at mutable config rows. If `querylog.db` is missing or corrupt at startup, rename aside, recreate, keep serving. Log loss is not an outage.
- **`querylog.db`** — high-churn, large, expendable: query log + its own private `domains` dimension table. Client identity stored as **IP text**, not a FK into config — log rows are immutable facts and must not point at mutable config rows. If `querylog.db` is missing or corrupt at startup, rename aside, recreate, keep serving — corruption only; a healthy file whose schema this build cannot use refuses the startup instead (§3.7).
- No cross-DB references. Retention/VACUUM churn never touches `config.db`; config backup is a copy of a tiny file.
### 3.7 Upgrades: Auto-Migration (Decision J)
- `config.db`: numbered, sequential SQL migration steps compiled into the binary. At startup: read schema version row, apply newer steps inside a transaction, continue. Operator upgrade = install binary, restart. Before v0.1 the list holds one step — the baseline of §11.2, edited in place — because nxdns has no installs and a step exists only to reconcile a database somebody already has.
- `querylog.db`: **no migrations.** On schema mismatch: rename aside, recreate fresh.
- `querylog.db`: a logical version in `PRAGMA user_version`, migrated **in place** at startup by the same shape of compiled step list, inside one transaction and behind one `querylog.db.pre-migrate-<epoch>` backup (only the newest is kept). A healthy file is never renamed aside: a version this build cannot reach refuses the startup with instructions, and only corruption recreates. A deliberate break is still allowed, but it must be versioned, refused at startup, and disclosed in the changelog — the cut gate enforces that. See `docs/reference/query-log-lifecycle.md`.
### 3.8 Blocklist Storage (Decision A)
@@ -238,7 +238,7 @@ src/
web/
server.zig router.zig auth.zig sse.zig static.zig metrics.zig openapi.zig
handlers/
auth.zig stats.zig queries.zig clients.zig groups.zig blocklists.zig
auth.zig overview.zig queries.zig clients.zig groups.zig blocklists.zig
rules.zig local.zig lookup.zig pause.zig settings.zig
upstream_health.zig certs.zig health.zig version.zig
@@ -484,6 +484,8 @@ CREATE INDEX idx_query_log_client ON query_log(client_ip);
CREATE INDEX idx_query_log_domain ON query_log(domain_id);
```
The sketch above is the original shape; `src/storage/querylog_schema.zig` is the authority, and the provenance columns milestone 28 added are not repeated here. Beside the raw rows the file carries four projection tables — `bucket_totals`, `bucket_clients`, `bucket_types`, `bucket_routes` — on a 30-minute grain, which is what `GET /api/overview` reads for the 24h, 7d and 30d windows instead of scanning every row. They are maintained by the batch writer and by retention inside the same transaction as the raw rows, so SQLite's transaction is the whole coherence story: no second file, no backfill, no rebuild command. The 1h window is narrower than the grain and takes one raw scan.
### 11.4 Query Logger
- In-memory buffer, mutex guarded, hard cap `query_log_buffer_max` (default 10000).
@@ -493,7 +495,7 @@ CREATE INDEX idx_query_log_domain ON query_log(domain_id);
### 11.5 Retention
Periodic delete of rows older than `retention_days`; scheduled checkpoint/VACUUM on `querylog.db` only.
Periodic delete of rows older than `retention_days`, dropping the projection buckets behind the cutoff and recomputing the straddling one in the same transaction; scheduled checkpoint/VACUUM on `querylog.db` only.
### 11.6 Disk Discipline (cloudflared lesson)
@@ -536,7 +538,7 @@ Scalars in `settings(key, value)`; ordered/structured items in dedicated tables.
### 13.1 Endpoints
- `POST /api/auth/login`, `POST /api/auth/logout`
- `GET /api/stats?period=…`, `GET /api/stats/timeseries?period=…`, `GET /api/stats/types?period=…`, `GET /api/stats/routes?period=…`, `GET /api/stats/clients?period=…`
- `GET /api/overview?period=…` — every Overview panel in one response over one read transaction
- `GET /api/queries` (filter + paginate), `GET /api/queries/live` (SSE, per-IP cap)
- `GET/PUT /api/clients/{id}`
- `GET/POST/PUT/DELETE /api/groups…`, `/api/blocklists…`, `/api/rules…`, `/api/local-records…`, `/api/forward-zones…`
@@ -664,7 +666,7 @@ The project publishes released binaries and container images from its own Gitea
6. Disk-fill degrades gracefully; no silent log-flood failure mode.
7. Web UI + API provide full admin functionality; OpenAPI contract tests green.
8. `nxdns export` round-trips via `nxdns import`.
9. Query logging, stats, SSE live stream work; querylog.db corruption self-heals.
9. Query logging, the overview, SSE live stream work; querylog.db corruption self-heals.
10. Local DoH + DoT endpoints serve LAN clients.
11. Schema upgrade = install + restart (migration test proves it).
12. All suites green in Gitea CI for both targets.
@@ -692,5 +694,5 @@ The project publishes released binaries and container images from its own Gitea
| G | SQLite vendored amalgamation + own thin wrapper |
| H | Two DBs: `config.db` (precious) + `querylog.db` (expendable, self-contained, client IP as text) |
| I | Frontend embedded in binary; dev flag serves from disk; static musl release builds |
| J | Auto-migration for `config.db` at startup; `querylog.db` recreated on mismatch |
| J | Auto-migration at startup for both databases; `querylog.db` recreated only when corrupt |
| — | Safe-search per-group; Prometheus `/metrics` in scope; CI on self-hosted Gitea Actions |
+508 -3
View File
@@ -11,6 +11,12 @@
"@stylexjs/stylex": "0.19.0",
"@tanstack/react-query": "5.101.4",
"@tanstack/react-router": "1.170.18",
"@visx/axis": "4.0.0",
"@visx/grid": "4.0.0",
"@visx/group": "4.0.0",
"@visx/scale": "4.0.0",
"@visx/shape": "4.0.0",
"@visx/tooltip": "4.0.0",
"react": "19.2.8",
"react-aria-components": "1.20.0",
"react-dom": "19.2.8"
@@ -1619,6 +1625,84 @@
"assertion-error": "^2.0.1"
}
},
"node_modules/@types/d3-array": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/@types/d3-array/-/d3-array-3.0.3.tgz",
"integrity": "sha512-Reoy+pKnvsksN0lQUlcH6dOGjRZ/3WRwXR//m+/8lt1BXeI4xyaUZoqULNjyXXRuh0Mj4LNpkCvhUpQlY3X5xQ==",
"license": "MIT"
},
"node_modules/@types/d3-color": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/@types/d3-color/-/d3-color-3.1.0.tgz",
"integrity": "sha512-HKuicPHJuvPgCD+np6Se9MQvS6OCbJmOjGvylzMJRlDwUXjKTTXs6Pwgk79O09Vj/ho3u1ofXnhFOaEWWPrlwA==",
"license": "MIT"
},
"node_modules/@types/d3-delaunay": {
"version": "6.0.1",
"resolved": "https://registry.npmjs.org/@types/d3-delaunay/-/d3-delaunay-6.0.1.tgz",
"integrity": "sha512-tLxQ2sfT0p6sxdG75c6f/ekqxjyYR0+LwPrsO1mbC9YDBzPJhs2HbJJRrn8Ez1DBoHRo2yx7YEATI+8V1nGMnQ==",
"license": "MIT"
},
"node_modules/@types/d3-format": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/@types/d3-format/-/d3-format-3.0.1.tgz",
"integrity": "sha512-5KY70ifCCzorkLuIkDe0Z9YTf9RR2CjBX1iaJG+rgM/cPP+sO+q9YdQ9WdhQcgPj1EQiJ2/0+yUkkziTG6Lubg==",
"license": "MIT"
},
"node_modules/@types/d3-geo": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/@types/d3-geo/-/d3-geo-3.1.0.tgz",
"integrity": "sha512-856sckF0oP/diXtS4jNsiQw/UuK5fQG8l/a9VVLeSouf1/PPbBE1i1W852zVwKwYCBkFJJB7nCFTbk6UMEXBOQ==",
"license": "MIT",
"dependencies": {
"@types/geojson": "*"
}
},
"node_modules/@types/d3-interpolate": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/@types/d3-interpolate/-/d3-interpolate-3.0.1.tgz",
"integrity": "sha512-jx5leotSeac3jr0RePOH1KdR9rISG91QIE4Q2PYTu4OymLTZfA3SrnURSLzKH48HmXVUru50b8nje4E79oQSQw==",
"license": "MIT",
"dependencies": {
"@types/d3-color": "*"
}
},
"node_modules/@types/d3-path": {
"version": "3.1.1",
"resolved": "https://registry.npmjs.org/@types/d3-path/-/d3-path-3.1.1.tgz",
"integrity": "sha512-VMZBYyQvbGmWyWVea0EHs/BwLgxc+MKi1zLDCONksozI4YJMcTt8ZEuIR4Sb1MMTE8MMW49v0IwI5+b7RmfWlg==",
"license": "MIT"
},
"node_modules/@types/d3-scale": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/@types/d3-scale/-/d3-scale-4.0.2.tgz",
"integrity": "sha512-Yk4htunhPAwN0XGlIwArRomOjdoBFXC3+kCxK2Ubg7I9shQlVSJy/pG/Ht5ASN+gdMIalpk8TJ5xV74jFsetLA==",
"license": "MIT",
"dependencies": {
"@types/d3-time": "*"
}
},
"node_modules/@types/d3-shape": {
"version": "3.1.7",
"resolved": "https://registry.npmjs.org/@types/d3-shape/-/d3-shape-3.1.7.tgz",
"integrity": "sha512-VLvUQ33C+3J+8p+Daf+nYSOsjB4GXp19/S/aGo60m9h1v6XaxjiT82lKVWJCfzhtuZ3yD7i/TPeC/fuKLLOSmg==",
"license": "MIT",
"dependencies": {
"@types/d3-path": "*"
}
},
"node_modules/@types/d3-time": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/@types/d3-time/-/d3-time-3.0.0.tgz",
"integrity": "sha512-sZLCdHvBUcNby1cB6Fd3ZBrABbjz3v1Vm90nysCQ6Vt7vd6e/h9Lt7SiJUoEX0l4Dzc7P5llKyhqSi1ycSf1Hg==",
"license": "MIT"
},
"node_modules/@types/d3-time-format": {
"version": "2.1.0",
"resolved": "https://registry.npmjs.org/@types/d3-time-format/-/d3-time-format-2.1.0.tgz",
"integrity": "sha512-/myT3I7EwlukNOX2xVdMzb8FRgNzRMpsZddwst9Ld/VFe6LyJyRp0s32l/V9XoUzk+Gqu56F/oGk6507+8BxrA==",
"license": "MIT"
},
"node_modules/@types/deep-eql": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/@types/deep-eql/-/deep-eql-4.0.2.tgz",
@@ -1633,6 +1717,12 @@
"dev": true,
"license": "MIT"
},
"node_modules/@types/geojson": {
"version": "7946.0.16",
"resolved": "https://registry.npmjs.org/@types/geojson/-/geojson-7946.0.16.tgz",
"integrity": "sha512-6C8nqWur3j98U6+lXDfTUWIfgvZU+EumvpHKcYjujKH7woYyLj2sUmff0tRhrqM7BohUw7Pz3ZB1jj2gW9Fvmg==",
"license": "MIT"
},
"node_modules/@types/node": {
"version": "26.1.1",
"resolved": "https://registry.npmjs.org/@types/node/-/node-26.1.1.tgz",
@@ -1647,7 +1737,7 @@
"version": "19.2.17",
"resolved": "https://registry.npmjs.org/@types/react/-/react-19.2.17.tgz",
"integrity": "sha512-MXfmqaVPEVgkBT/aY0aGCkRWWtByiYQXo3xdQ8r5RzuFrPiRn8Gar2tQdXSUQ2GKV3bkXckek89V8wQBY2Q/Aw==",
"dev": true,
"devOptional": true,
"license": "MIT",
"dependencies": {
"csstype": "^3.2.2"
@@ -1657,7 +1747,7 @@
"version": "19.2.3",
"resolved": "https://registry.npmjs.org/@types/react-dom/-/react-dom-19.2.3.tgz",
"integrity": "sha512-jp2L/eY6fn+KgVVQAOqYItbF0VY/YApe5Mz2F0aykSO8gx31bYCZyvSeYxCHKvzHG5eZjc+zyaS5BrBWya2+kQ==",
"dev": true,
"devOptional": true,
"license": "MIT",
"peerDependencies": {
"@types/react": "^19.2.0"
@@ -2003,6 +2093,211 @@
"node": ">=16.20.0"
}
},
"node_modules/@visx/axis": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/axis/-/axis-4.0.0.tgz",
"integrity": "sha512-cSPNO9Aic2UX49AmIeJJ9TQrrAkvUHpHOGqimUAXhbg07VI3HoHDKcXZdrjvQz+g0LhubrUkjj8Gwju2WZ3dtg==",
"license": "MIT",
"dependencies": {
"@visx/group": "4.0.0",
"@visx/point": "4.0.0",
"@visx/scale": "4.0.0",
"@visx/shape": "4.0.0",
"@visx/text": "4.0.0",
"classnames": "^2.3.1"
},
"peerDependencies": {
"@types/react": "^18.0.0 || ^19.0.0",
"react": "^18.0.0 || ^19.0.0"
},
"peerDependenciesMeta": {
"@types/react": {
"optional": true
}
}
},
"node_modules/@visx/bounds": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/bounds/-/bounds-4.0.0.tgz",
"integrity": "sha512-3wAfN5fg2bxg/5f3MlKWzGDTKU2hdKkxq8bVWXpPZQV0UiVc0myuU1qa34c7tN3hg5SDe6MJi/bae2eTQDgEiQ==",
"license": "MIT",
"peerDependencies": {
"@types/react": "^18.0.0 || ^19.0.0",
"@types/react-dom": "^18.0.0 || ^19.0.0",
"react": "^18.0.0 || ^19.0.0",
"react-dom": "^18.0.0 || ^19.0.0"
},
"peerDependenciesMeta": {
"@types/react": {
"optional": true
},
"@types/react-dom": {
"optional": true
}
}
},
"node_modules/@visx/curve": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/curve/-/curve-4.0.0.tgz",
"integrity": "sha512-bXrOSd2BzVCxR7kBE7gPTSU4plxzYN75w7erdHYabwn3vzP+xFk1o0gg3NOHEPJzt5tXqf0Ecm5YqHACOnipTA==",
"license": "MIT",
"dependencies": {
"@visx/vendor": "4.0.0"
}
},
"node_modules/@visx/grid": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/grid/-/grid-4.0.0.tgz",
"integrity": "sha512-BUnznqYo8jyn3zXRWZiF0zwg1VQl6w5kYqZ+XMY+cPXtAHZZbwGXBChMpPefH5Lj1+P1b4Pf3QihKG8zjL9PCw==",
"license": "MIT",
"dependencies": {
"@visx/curve": "4.0.0",
"@visx/group": "4.0.0",
"@visx/point": "4.0.0",
"@visx/scale": "4.0.0",
"@visx/shape": "4.0.0",
"classnames": "^2.3.1"
},
"peerDependencies": {
"@types/react": "^18.0.0 || ^19.0.0",
"react": "^18.0.0 || ^19.0.0"
},
"peerDependenciesMeta": {
"@types/react": {
"optional": true
}
}
},
"node_modules/@visx/group": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/group/-/group-4.0.0.tgz",
"integrity": "sha512-HGAq8BCqn5x0t94CJOLJeKtWss9LFpF3HY69HbuO1PlR+B9c4aVT9xbjfudmU8wGQZLT6JNt+PJvxbO2aJ+1aA==",
"license": "MIT",
"dependencies": {
"classnames": "^2.3.1"
},
"peerDependencies": {
"@types/react": "^18.0.0 || ^19.0.0",
"react": "^18.0.0 || ^19.0.0"
},
"peerDependenciesMeta": {
"@types/react": {
"optional": true
}
}
},
"node_modules/@visx/point": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/point/-/point-4.0.0.tgz",
"integrity": "sha512-hwJ9UlIjg3iV4iiJ3RN5SNTXama+zGaNe57sXSrSvPiqhkDXypsYBdDSlAnHmKbMzHP0zFmgB1UJciB/QUyigw==",
"license": "MIT"
},
"node_modules/@visx/scale": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/scale/-/scale-4.0.0.tgz",
"integrity": "sha512-Q3VaSLKlsrxxSl9CwS2D3Mvz4/4kIp/6NMKRY8Im0Au1hrQh2egU806p0+JwN0ccuGGDe03jyAxX4BSRAH8Plg==",
"license": "MIT",
"dependencies": {
"@visx/vendor": "4.0.0"
}
},
"node_modules/@visx/shape": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/shape/-/shape-4.0.0.tgz",
"integrity": "sha512-X0FP3OFjQhc5/6Vj2aY7cK2JwKYI0H0hQM7cdNRWy/ZMMlrh5KpEdqx0HVo0KoTeTlkUvO+kydMLnAuuCRFIFw==",
"license": "MIT",
"dependencies": {
"@visx/curve": "4.0.0",
"@visx/group": "4.0.0",
"@visx/scale": "4.0.0",
"@visx/vendor": "4.0.0",
"classnames": "^2.3.1"
},
"peerDependencies": {
"@types/react": "^18.0.0 || ^19.0.0",
"react": "^18.0.0 || ^19.0.0"
},
"peerDependenciesMeta": {
"@types/react": {
"optional": true
}
}
},
"node_modules/@visx/text": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/text/-/text-4.0.0.tgz",
"integrity": "sha512-39f2goSaKy5Mqn89lRDGSJ1IMr49wplkbIFHsFI8cPnAPbguiJxYZ+ulizd7Vcsq2z/hytuGv3Lmk165zUiUYQ==",
"license": "MIT",
"dependencies": {
"classnames": "^2.3.1",
"reduce-css-calc": "^1.3.0"
},
"peerDependencies": {
"@types/react": "^18.0.0 || ^19.0.0",
"react": "^18.0.0 || ^19.0.0"
},
"peerDependenciesMeta": {
"@types/react": {
"optional": true
}
}
},
"node_modules/@visx/tooltip": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/tooltip/-/tooltip-4.0.0.tgz",
"integrity": "sha512-+6J2h5qPvniT0pQtxQrLYGErPRVHr3pRfcMkNqTxD/HgofDGsL6vEBkjMR535MF1hkWYQ9XzOiWP6ERN4TF+mg==",
"license": "MIT",
"dependencies": {
"@visx/bounds": "4.0.0",
"classnames": "^2.3.1",
"react-use-measure": "^2.0.4"
},
"peerDependencies": {
"@types/react": "^18.0.0 || ^19.0.0",
"@types/react-dom": "^18.0.0 || ^19.0.0",
"react": "^18.0.0 || ^19.0.0",
"react-dom": "^18.0.0 || ^19.0.0"
},
"peerDependenciesMeta": {
"@types/react": {
"optional": true
},
"@types/react-dom": {
"optional": true
}
}
},
"node_modules/@visx/vendor": {
"version": "4.0.0",
"resolved": "https://registry.npmjs.org/@visx/vendor/-/vendor-4.0.0.tgz",
"integrity": "sha512-LoBNzWjTXzBfu095BQxB14IwZQHHOLs8ZMzM6t2FL4ZGORaACgLcmXRRLhFo0VmRR8L7iNTM4HHc1j6yFvzsAA==",
"license": "MIT and ISC",
"dependencies": {
"@types/d3-array": "3.0.3",
"@types/d3-color": "3.1.0",
"@types/d3-delaunay": "6.0.1",
"@types/d3-format": "3.0.1",
"@types/d3-geo": "3.1.0",
"@types/d3-interpolate": "3.0.1",
"@types/d3-path": "3.1.1",
"@types/d3-scale": "4.0.2",
"@types/d3-shape": "3.1.7",
"@types/d3-time": "3.0.0",
"@types/d3-time-format": "2.1.0",
"d3-array": "3.2.1",
"d3-color": "3.1.0",
"d3-delaunay": "6.0.2",
"d3-format": "3.1.0",
"d3-geo": "3.1.0",
"d3-interpolate": "3.0.1",
"d3-path": "3.1.0",
"d3-scale": "4.0.2",
"d3-shape": "3.2.0",
"d3-time": "3.1.0",
"d3-time-format": "4.1.0",
"internmap": "2.0.3"
}
},
"node_modules/@vitejs/plugin-react": {
"version": "6.0.4",
"resolved": "https://registry.npmjs.org/@vitejs/plugin-react/-/plugin-react-6.0.4.tgz",
@@ -2211,6 +2506,12 @@
"node": ">=12"
}
},
"node_modules/balanced-match": {
"version": "0.4.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-0.4.2.tgz",
"integrity": "sha512-STw03mQKnGUYtoNjmowo4F2cRmIIxYEGiMsjjwla/u5P1lxadj/05WkNaFjNiKTgJkj8KiXbgAiRTmcQRwQNtg==",
"license": "MIT"
},
"node_modules/baseline-browser-mapping": {
"version": "2.11.12",
"resolved": "https://registry.npmjs.org/baseline-browser-mapping/-/baseline-browser-mapping-2.11.12.tgz",
@@ -2299,6 +2600,12 @@
"node": ">=18"
}
},
"node_modules/classnames": {
"version": "2.5.1",
"resolved": "https://registry.npmjs.org/classnames/-/classnames-2.5.1.tgz",
"integrity": "sha512-saHYOzhIQs6wy2sVxTM6bUDsQO4F50V9RQ22qBpEdCW+I+/Wmke2HOl6lS6dTpdxVhb88/I6+Hs+438c3lfUow==",
"license": "MIT"
},
"node_modules/client-only": {
"version": "0.0.1",
"resolved": "https://registry.npmjs.org/client-only/-/client-only-0.0.1.tgz",
@@ -2351,9 +2658,136 @@
"version": "3.2.3",
"resolved": "https://registry.npmjs.org/csstype/-/csstype-3.2.3.tgz",
"integrity": "sha512-z1HGKcYy2xA8AGQfwrn0PAy+PB7X/GSj3UVJW9qKyn43xWa+gl5nXmU4qqLMRzWVLFC8KusUX8T/0kCiOYpAIQ==",
"dev": true,
"devOptional": true,
"license": "MIT"
},
"node_modules/d3-array": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/d3-array/-/d3-array-3.2.1.tgz",
"integrity": "sha512-gUY/qeHq/yNqqoCKNq4vtpFLdoCdvyNpWoC/KNjhGbhDuQpAM9sIQQKkXSNpXa9h5KySs/gzm7R88WkUutgwWQ==",
"license": "ISC",
"dependencies": {
"internmap": "1 - 2"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-color": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-color/-/d3-color-3.1.0.tgz",
"integrity": "sha512-zg/chbXyeBtMQ1LbD/WSoW2DpC3I0mpmPdW+ynRTj/x2DAWYrIY7qeZIHidozwV24m4iavr15lNwIwLxRmOxhA==",
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/d3-delaunay": {
"version": "6.0.2",
"resolved": "https://registry.npmjs.org/d3-delaunay/-/d3-delaunay-6.0.2.tgz",
"integrity": "sha512-IMLNldruDQScrcfT+MWnazhHbDJhcRJyOEBAJfwQnHle1RPh6WDuLvxNArUju2VSMSUuKlY5BGHRJ2cYyoFLQQ==",
"license": "ISC",
"dependencies": {
"delaunator": "5"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-format": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-format/-/d3-format-3.1.0.tgz",
"integrity": "sha512-YyUI6AEuY/Wpt8KWLgZHsIU86atmikuoOmCfommt0LYHiQSPjvX2AcFc38PX0CBpr2RCyZhjex+NS/LPOv6YqA==",
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/d3-geo": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-geo/-/d3-geo-3.1.0.tgz",
"integrity": "sha512-JEo5HxXDdDYXCaWdwLRt79y7giK8SbhZJbFWXqbRTolCHFI5jRqteLzCsq51NKbUoX0PjBVSohxrx+NoOUujYA==",
"license": "ISC",
"dependencies": {
"d3-array": "2.5.0 - 3"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-interpolate": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/d3-interpolate/-/d3-interpolate-3.0.1.tgz",
"integrity": "sha512-3bYs1rOD33uo8aqJfKP3JWPAibgw8Zm2+L9vBKEHJ2Rg+viTR7o5Mmv5mZcieN+FRYaAOWX5SJATX6k1PWz72g==",
"license": "ISC",
"dependencies": {
"d3-color": "1 - 3"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-path": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-path/-/d3-path-3.1.0.tgz",
"integrity": "sha512-p3KP5HCf/bvjBSSKuXid6Zqijx7wIfNW+J/maPs+iwR35at5JCbLUT0LzF1cnjbCHWhqzQTIN2Jpe8pRebIEFQ==",
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/d3-scale": {
"version": "4.0.2",
"resolved": "https://registry.npmjs.org/d3-scale/-/d3-scale-4.0.2.tgz",
"integrity": "sha512-GZW464g1SH7ag3Y7hXjf8RoUuAFIqklOAq3MRl4OaWabTFJY9PN/E1YklhXLh+OQ3fM9yS2nOkCoS+WLZ6kvxQ==",
"license": "ISC",
"dependencies": {
"d3-array": "2.10.0 - 3",
"d3-format": "1 - 3",
"d3-interpolate": "1.2.0 - 3",
"d3-time": "2.1.1 - 3",
"d3-time-format": "2 - 4"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-shape": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/d3-shape/-/d3-shape-3.2.0.tgz",
"integrity": "sha512-SaLBuwGm3MOViRq2ABk3eLoxwZELpH6zhl3FbAoJ7Vm1gofKx6El1Ib5z23NUEhF9AsGl7y+dzLe5Cw2AArGTA==",
"license": "ISC",
"dependencies": {
"d3-path": "^3.1.0"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-time": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/d3-time/-/d3-time-3.1.0.tgz",
"integrity": "sha512-VqKjzBLejbSMT4IgbmVgDjpkYrNWUYJnbCGo874u7MMKIWsILRX+OpX/gTk8MqjpT1A/c6HY2dCA77ZN0lkQ2Q==",
"license": "ISC",
"dependencies": {
"d3-array": "2 - 3"
},
"engines": {
"node": ">=12"
}
},
"node_modules/d3-time-format": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/d3-time-format/-/d3-time-format-4.1.0.tgz",
"integrity": "sha512-dJxPBlzC7NugB2PDLwo9Q8JiTR3M3e4/XANkreKSUxF8vvXKqm1Yfq4Q5dl8budlunRVlUUaDUgFt7eA8D6NLg==",
"license": "ISC",
"dependencies": {
"d3-time": "1 - 3"
},
"engines": {
"node": ">=12"
}
},
"node_modules/data-urls": {
"version": "7.0.0",
"resolved": "https://registry.npmjs.org/data-urls/-/data-urls-7.0.0.tgz",
@@ -2393,6 +2827,15 @@
"dev": true,
"license": "MIT"
},
"node_modules/delaunator": {
"version": "5.1.0",
"resolved": "https://registry.npmjs.org/delaunator/-/delaunator-5.1.0.tgz",
"integrity": "sha512-AGrQ4QSgssa1NGmWmLPqN5NY2KajF5MqxetNEO+o0n3ZwZZeTmt7bBnvzHWrmkZFxGgr4HdyFgelzgi06otLuQ==",
"license": "ISC",
"dependencies": {
"robust-predicates": "^3.0.2"
}
},
"node_modules/dequal": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/dequal/-/dequal-2.0.3.tgz",
@@ -2533,6 +2976,15 @@
"node": "^20.19.0 || ^22.12.0 || >=24.0.0"
}
},
"node_modules/internmap": {
"version": "2.0.3",
"resolved": "https://registry.npmjs.org/internmap/-/internmap-2.0.3.tgz",
"integrity": "sha512-5Hh7Y1wQbvY5ooGgPbDaL5iYLAPzMTUrjMulskHLH6wnv/A+1q5rgEaiuqEjB+oxGXIVZs1FF+R/KPN3ZSQYYg==",
"license": "ISC",
"engines": {
"node": ">=12"
}
},
"node_modules/invariant": {
"version": "2.2.4",
"resolved": "https://registry.npmjs.org/invariant/-/invariant-2.2.4.tgz",
@@ -2946,6 +3398,12 @@
"@jridgewell/sourcemap-codec": "^1.5.5"
}
},
"node_modules/math-expression-evaluator": {
"version": "1.4.0",
"resolved": "https://registry.npmjs.org/math-expression-evaluator/-/math-expression-evaluator-1.4.0.tgz",
"integrity": "sha512-4vRUvPyxdO8cWULGTh9dZWL2tZK6LDBvj+OGHBER7poH9Qdt7kXEoj20wiz4lQUbUXQZFjPbe5mVDo9nutizCw==",
"license": "MIT"
},
"node_modules/mdn-data": {
"version": "2.27.1",
"resolved": "https://registry.npmjs.org/mdn-data/-/mdn-data-2.27.1.tgz",
@@ -3254,6 +3712,47 @@
"react": "^16.8.0 || ^17.0.0-rc.1 || ^18.0.0 || ^19.0.0-rc.1"
}
},
"node_modules/react-use-measure": {
"version": "2.1.7",
"resolved": "https://registry.npmjs.org/react-use-measure/-/react-use-measure-2.1.7.tgz",
"integrity": "sha512-KrvcAo13I/60HpwGO5jpW7E9DfusKyLPLvuHlUyP5zqnmAPhNc6qTRjUQrdTADl0lpPpDVU2/Gg51UlOGHXbdg==",
"license": "MIT",
"peerDependencies": {
"react": ">=16.13",
"react-dom": ">=16.13"
},
"peerDependenciesMeta": {
"react-dom": {
"optional": true
}
}
},
"node_modules/reduce-css-calc": {
"version": "1.3.0",
"resolved": "https://registry.npmjs.org/reduce-css-calc/-/reduce-css-calc-1.3.0.tgz",
"integrity": "sha512-0dVfwYVOlf/LBA2ec4OwQ6p3X9mYxn/wOl2xTcLwjnPYrkgEfPx3VI4eGCH3rQLlPISG5v9I9bkZosKsNRTRKA==",
"license": "MIT",
"dependencies": {
"balanced-match": "^0.4.2",
"math-expression-evaluator": "^1.2.14",
"reduce-function-call": "^1.0.1"
}
},
"node_modules/reduce-function-call": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/reduce-function-call/-/reduce-function-call-1.0.3.tgz",
"integrity": "sha512-Hl/tuV2VDgWgCSEeWMLwxLZqX7OK59eU1guxXsRKTAyeYimivsKdtcV4fu3r710tpG5GmDKDhQ0HSZLExnNmyQ==",
"license": "MIT",
"dependencies": {
"balanced-match": "^1.0.0"
}
},
"node_modules/reduce-function-call/node_modules/balanced-match": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/balanced-match/-/balanced-match-1.0.2.tgz",
"integrity": "sha512-3oSeUO0TMV67hN1AmbXsK4yaqU7tjiHlbxRDZOpH0KW9+CeX4bRAaX0Anxt0tx2MrpRpWwQaPwIlISEJhYU5Pw==",
"license": "MIT"
},
"node_modules/require-from-string": {
"version": "2.0.2",
"resolved": "https://registry.npmjs.org/require-from-string/-/require-from-string-2.0.2.tgz",
@@ -3264,6 +3763,12 @@
"node": ">=0.10.0"
}
},
"node_modules/robust-predicates": {
"version": "3.0.3",
"resolved": "https://registry.npmjs.org/robust-predicates/-/robust-predicates-3.0.3.tgz",
"integrity": "sha512-NS3levdsRIUOmiJ8FZWCP7LG3QpJyrs/TE0Zpf1yvZu8cAJJ6QMW92H1c7kWpdIHo8RvmLxN/o2JXTKHp74lUA==",
"license": "Unlicense"
},
"node_modules/rolldown": {
"version": "1.1.5",
"resolved": "https://registry.npmjs.org/rolldown/-/rolldown-1.1.5.tgz",
+6
View File
@@ -28,6 +28,12 @@
"@stylexjs/stylex": "0.19.0",
"@tanstack/react-query": "5.101.4",
"@tanstack/react-router": "1.170.18",
"@visx/axis": "4.0.0",
"@visx/grid": "4.0.0",
"@visx/group": "4.0.0",
"@visx/scale": "4.0.0",
"@visx/shape": "4.0.0",
"@visx/tooltip": "4.0.0",
"react": "19.2.8",
"react-aria-components": "1.20.0",
"react-dom": "19.2.8"
@@ -1,5 +1,5 @@
import { fireEvent, screen, waitFor, within } from "@testing-library/react";
import { DATABASE, renderPage, stubApi, type Call } from "./testFixtures";
import { DATABASE, contentArea, renderPage, stubApi, type Call } from "./testFixtures";
/**
* Resolution in database mode: the upstream pool, the local records and the
@@ -37,7 +37,7 @@ test("the upstream pool is the default tab and lists every field", async () => {
expect((screen.getByLabelText("udp://1.1.1.1:53 enabled") as HTMLInputElement).checked).toBe(true);
expect((screen.getByLabelText("tls://9.9.9.9:853 enabled") as HTMLInputElement).checked).toBe(false);
expect(screen.getByRole("heading", { name: "Add upstream" })).toBeTruthy();
expect(screen.getByText(/takes effect at the next restart/)).toBeTruthy();
expect(screen.getByText(/applies to the next query/)).toBeTruthy();
});
test("adding an upstream posts every field", async () => {
@@ -56,24 +56,18 @@ test("adding an upstream posts every field", async () => {
});
});
test("an upstream write re-reads the config status, and the shell states the pending restart", async () => {
// The client never decides a restart is owed: the server sets the flag, and
// the mutation's invalidation is only what makes the page ask again.
let restartPending = false;
await openResolution(undefined, {
responses: { "GET /api/config/status": () => ({ ...DATABASE, restart_pending: restartPending }) },
onWrite: () => {
restartPending = true;
return null;
},
});
test("an upstream write applies live, so the tab says nothing about a restart", async () => {
// The server rebuilds the pool on the write and echoes `restart_required:
// false`, so `restart_pending` stays down and silence is the whole report.
await openResolution(undefined, { responses: { "GET /api/config/status": () => DATABASE } });
await screen.findByRole("heading", { name: "Add upstream" });
expect(screen.queryByText(/Restart nxdns to apply them/)).toBeNull();
fireEvent.change(screen.getByLabelText("URL"), { target: { value: "udp://8.8.8.8:53" } });
fireEvent.click(screen.getByRole("button", { name: "Add upstream" }));
await screen.findByText(/Saved changes are not running yet\. Restart nxdns to apply them\./);
await waitFor(() => expect(writes()).toHaveLength(1));
expect(screen.queryByText(/Restart nxdns to apply them/)).toBeNull();
expect(contentArea().textContent).not.toMatch(/restart/i);
});
test("toggling enabled resends the whole row", async () => {
@@ -1,16 +1,16 @@
import { act, fireEvent, screen, waitFor, within } from "@testing-library/react";
import { queryKeys } from "@/lib/queries";
import type { Settings, SettingsPatch } from "@/lib/types";
import { DATABASE, MANAGED_FILE, baseSettings, renderPage, stubApi } from "./testFixtures";
import { DATABASE, MANAGED_FILE, RESTART_REQUIRED_KEYS, baseSettings, renderPage, stubApi } from "./testFixtures";
/**
* System in database mode: the settings form, its diff contract, and the
* certificate reload that is a runtime action under both authorities.
*/
// `logging.level` is enum-backed, so the list covers both field renderings: an
// input whose label carries the mark, and a `Select` that cannot.
const RESTART_KEYS = ["dns.port", "web.port", "logging.level"];
// What the server actually reports: the listener binds and `web.enabled`.
// Every other key applies live, so it carries no mark at all.
const RESTART_KEYS = RESTART_REQUIRED_KEYS;
let stored: Settings;
let putBodies: SettingsPatch[];
@@ -32,10 +32,10 @@ function applyPatch(patch: SettingsPatch): void {
}
}
/** Mirrors settings.zig: a patch touching only `web.password` applies live. */
function needsRestart(patch: SettingsPatch): boolean {
/** Mirrors apply.zig's table: only a listed key leaves the server owing a restart. */
function needsRestart(patch: SettingsPatch, keys: readonly string[]): boolean {
return Object.entries(patch).some(([section, fields]) =>
Object.keys(fields as Record<string, unknown>).some((key) => !(section === "web" && key === "password")),
Object.keys(fields as Record<string, unknown>).some((key) => keys.includes(`${section}.${key}`)),
);
}
@@ -50,11 +50,11 @@ afterEach(() => {
vi.unstubAllGlobals();
});
async function openSystem() {
async function openSystem(restartKeys: readonly string[] = RESTART_KEYS) {
stubApi(DATABASE, {
responses: {
"GET /api/config/status": () => ({ ...DATABASE, restart_pending: restartPending }),
"GET /api/settings": () => ({ settings: stored, restart_required: RESTART_KEYS }),
"GET /api/settings": () => ({ settings: stored, restart_required: restartKeys }),
},
onWrite: (call) => {
if (call.url !== "/api/settings") return null;
@@ -62,8 +62,8 @@ async function openSystem() {
putBodies.push(patch);
if (putResponse !== null) return putResponse();
applyPatch(patch);
if (needsRestart(patch)) restartPending = true;
return json({ settings: stored, restart_required: RESTART_KEYS });
if (needsRestart(patch, restartKeys)) restartPending = true;
return json({ settings: stored, restart_required: restartKeys });
},
});
const router = await renderPage("/configuration/system", "System");
@@ -103,15 +103,50 @@ test("a changed field enables Save and the PUT body is exactly the diff", async
test("a restart-required key is marked as one, from the envelope's list", async () => {
await openSystem();
expect(within(screen.getByRole("group", { name: "DNS" })).getByText("needs restart")).toBeTruthy();
// `rate_limit` is not on the list, so it carries no mark.
// The DNS binds and the port are the section's whole share of the list;
// `rate_limit` and `rate_window_seconds` apply live and carry no mark.
const dns = screen.getByRole("group", { name: "DNS" });
expect(within(dns).getAllByText("needs restart")).toHaveLength(3);
const cache = screen.getByRole("group", { name: "Cache" });
expect(within(cache).queryByText("needs restart")).toBeNull();
});
test("an enum-backed key on the list is marked too, not only text and number fields", async () => {
test("every key the server applies live is drawn without restart messaging", async () => {
await openSystem();
// Silence is the report for a live key: no mark on the field, and editing
// one owes nothing afterwards either.
for (const title of ["Upstream", "Blocking", "Cache", "EDNS", "Logging", "Disk", "Blocklist Update"]) {
const section = screen.getByRole("group", { name: title });
expect(within(section).queryByText("needs restart")).toBeNull();
}
const logging = screen.getByRole("group", { name: "Logging" });
fireEvent.change(within(logging).getByLabelText("retention_days"), { target: { value: "14" } });
fireEvent.click(saveButton());
await waitFor(() => expect(putBodies).toHaveLength(1));
expect(putBodies[0]).toEqual({ logging: { retention_days: 14 } });
await waitFor(() => expect(saveButton().disabled).toBe(true));
expect(restartNotice()).toBeNull();
});
test("a port edit still owes a restart, and the shell says so", async () => {
await openSystem();
const dns = screen.getByRole("group", { name: "DNS" });
fireEvent.change(within(dns).getByLabelText(/^port/), { target: { value: "5353" } });
fireEvent.click(saveButton());
await waitFor(() => expect(putBodies).toHaveLength(1));
await screen.findByText(/Saved changes are not running yet/);
});
test("an enum-backed key on the list is marked too, not only text and number fields", async () => {
// No shipped restart-required key is enum-backed, but the list is the
// server's to change, so the `Select` rendering is pinned against one.
await openSystem(["logging.level"]);
const logging = screen.getByRole("group", { name: "Logging" });
// `logging.level` is a Select and `logging.output` is not on the list, so
// exactly one mark belongs to this section.
@@ -176,7 +211,7 @@ test("password flow: note shown, confirm required, PUT sends web.password, no re
expect(restartNotice()).toBeNull();
});
test("a mixed patch makes the server owe a restart, and the shell says so", async () => {
test("a patch of live keys alone leaves the server owing nothing", async () => {
await openSystem();
const web = screen.getByRole("group", { name: "Web" });
@@ -187,7 +222,8 @@ test("a mixed patch makes the server owe a restart, and the shell says so", asyn
await waitFor(() => expect(putBodies).toHaveLength(1));
expect(putBodies[0]).toEqual({ web: { session_ttl_hours: 48, password: "hunter2" } });
await screen.findByText(/Saved changes are not running yet/);
await waitFor(() => expect(saveButton().disabled).toBe(true));
expect(restartNotice()).toBeNull();
});
test("the form is disabled while the PUT is pending and re-enabled after success", async () => {
@@ -13,7 +13,7 @@ import QueryPanel from "./QueryPanel";
import UpstreamForm from "./UpstreamForm";
import { styles as config } from "./styles";
const INTRO = "The pool builds its clients at startup, so an edit here takes effect at the next restart.";
const INTRO = "The pool is rebuilt as you save, so an edit here applies to the next query.";
const styles = stylex.create({
url: {
@@ -99,14 +99,17 @@ test("a failed status is announced by the shell on a page that is not configurat
stubApi(DATABASE, {
responses: {
"GET /api/config/status": new Response(JSON.stringify({ error: "gone" }), { status: 404 }),
"GET /api/stats?period=24h": {
"GET /api/overview?period=24h": {
period: "24h",
since: 0,
until: 86400,
queries: 0,
blocked: 0,
clients: 0,
avg_response_time_us: null,
bucket_seconds: 1800,
totals: { queries: 0, blocked: 0, clients: 0, avg_response_time_us: null },
buckets: [],
clients: [],
other: [],
types: [],
routes: [],
coverage: { complete: true, available_since: 0 },
},
},
@@ -14,6 +14,7 @@ import { AuthProvider } from "@/auth/store";
import { health } from "@/lib/healthFixture";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import { sample_get_settings } from "@/lib/contractSamples.gen";
import type { ConfigStatus, Settings } from "@/lib/types";
export const CONFIG_PATH = "/etc/nxdns/config.zon";
@@ -33,6 +34,13 @@ export const MANAGED_FILE: ConfigStatus = {
restart_pending: false,
};
/**
* The keys `/api/settings` still reports as restart-required, taken from the
* committed contract sample so a server-side change to the set fails the tests
* that pin it rather than passing against a stale copy.
*/
export const RESTART_REQUIRED_KEYS: readonly string[] = sample_get_settings.restart_required;
export function baseSettings(): Settings {
return {
upstream: { attempt_timeout_ms: 2500, read_timeout_ms: 3000, total_timeout_ms: 5000 },
@@ -197,7 +205,7 @@ function defaultResponses(status: ConfigStatus): Record<string, unknown> {
"GET /api/upstreams": { upstreams: UPSTREAMS },
"GET /api/local-records": { local_records: LOCAL_RECORDS },
"GET /api/forward-zones": { forward_zones: FORWARD_ZONES },
"GET /api/settings": { settings: baseSettings(), restart_required: ["dns.port", "web.port"] },
"GET /api/settings": { settings: baseSettings(), restart_required: RESTART_REQUIRED_KEYS },
};
}
@@ -0,0 +1,277 @@
import { fireEvent, render as renderBare, screen, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { createQueryClient } from "@/lib/queryClient";
import { formatTime } from "@/lib/format";
import ClientChart, { type ClientChartData } from "./ClientChart";
import { OTHER_KEY, clientKey, seriesColor } from "./seriesColors";
const SINCE = 1_700_000_000;
const BUCKET = 1800;
function clients(named: { client: string; buckets: number[] }[], other: number[]): ClientChartData {
return { since: SINCE, bucket_seconds: BUCKET, clients: named, other };
}
const TWO_BUCKETS = clients(
[
{ client: "192.0.2.30", buckets: [10, 1] },
{ client: "192.0.2.31", buckets: [20, 1] },
],
[5, 1],
);
/**
* The chart looks a client's registered name up, so it needs a query client. No
* client is registered in these fixtures, which is what leaves the addresses on
* screen as the labels.
*/
function render(data: ClientChartData) {
const client = createQueryClient();
const tree = (next: ClientChartData) => (
<QueryClientProvider client={client}>
<ClientChart data={next} />
</QueryClientProvider>
);
const result = renderBare(tree(data));
return { ...result, rerender: (next: ClientChartData) => result.rerender(tree(next)) };
}
beforeEach(() => {
vi.stubGlobal(
"fetch",
vi.fn(async () =>
Promise.resolve(
new Response(JSON.stringify({ clients: [] }), {
status: 200,
headers: { "content-type": "application/json" },
}),
),
),
);
});
afterEach(() => vi.unstubAllGlobals());
function overlayRects(container: HTMLElement): SVGRectElement[] {
return Array.from(container.querySelectorAll<SVGRectElement>('rect[fill="transparent"]'));
}
test("the data table is the SVG's accessible equivalent", () => {
render(TWO_BUCKETS);
expect(screen.getByRole("img", { name: /client activity over time/i })).toBeTruthy();
const table = screen.getByRole("table", { name: "Queries per client per time bucket" });
expect(within(table).getAllByRole("row").length).toBe(3);
});
/**
* The x-axis is this response's own `since` plus one bucket width per column.
* The timeseries endpoint aligns its buckets with these, which is what lets the
* two charts stack above one another without either knowing about the other.
*/
test("the columns are timestamped from this response's own window", () => {
render(TWO_BUCKETS);
const rows = within(screen.getByRole("table")).getAllByRole("rowheader");
expect(rows.map((row) => row.textContent)).toEqual([formatTime(SINCE), formatTime(SINCE + BUCKET)]);
});
/**
* Every series here is a disjoint part of the whole, so the scale has to come
* from the tallest column's own sum. Taking it from the largest single value
* instead would run the tallest column off the top of the plot: the axis has to
* reach 35 here, not 20.
*/
test("the value scale covers the tallest column's total, not its largest series", () => {
const { container } = render(clients([{ client: "192.0.2.30", buckets: [20] }], [15]));
const labels = Array.from(container.querySelectorAll(".visx-axis-left text")).map((label) => label.textContent);
expect(labels[labels.length - 1]).toBe("35");
});
test("the series are drawn in the colour of the client's address, and Other in its own", () => {
const { container } = render(clients([{ client: "192.0.2.30", buckets: [10] }], [5]));
const fills = Array.from(container.querySelectorAll("rect"))
.map((rect) => rect.getAttribute("fill"))
.filter((fill) => fill !== "transparent");
expect(fills).toEqual([seriesColor(clientKey("192.0.2.30")), seriesColor(OTHER_KEY)]);
});
test("a window with no queries says so instead of drawing an empty grid", () => {
const { container } = render(clients([{ client: "192.0.2.30", buckets: [0, 0] }], [0, 0]));
expect(screen.getByText("No queries in this period.")).toBeTruthy();
expect(container.querySelector("svg")).toBeNull();
});
test("no bucket at all says the same thing", () => {
render(clients([], []));
expect(screen.getByText("No queries in this period.")).toBeTruthy();
});
test("hover text is the tooltip alone, never a bare SVG title", () => {
const { container } = render(TWO_BUCKETS);
expect(container.querySelectorAll("title")).toHaveLength(0);
});
test("each bucket's hit target spans the full plot height", () => {
const { container } = render(TWO_BUCKETS);
const rects = overlayRects(container);
expect(rects).toHaveLength(2);
for (const rect of rects) {
expect(rect.getAttribute("y")).toBe("8");
expect(rect.getAttribute("height")).toBe("210");
}
});
/**
* A band scale spends a gap after the last column as well as between them, so a
* full-step hit target on the last bucket would reach into the right margin.
*/
test("the last hit target stops at the plot's right edge", () => {
const { container } = render(TWO_BUCKETS);
const last = overlayRects(container).at(-1) as SVGRectElement;
const right = Number(last.getAttribute("x")) + Number(last.getAttribute("width"));
// 640 fallback width, less the 44px left and 8px right margins.
expect(right).toBeCloseTo(44 + 588, 6);
});
/**
* The window refreshes every half minute under an open tooltip. The tooltip
* holds a bucket index and reads the numbers out of the render it is drawing, so
* a refresh that keeps the same buckets updates it rather than leaving last
* minute's counts on screen.
*/
test("a refresh in the same window retells the hovered bucket with the new counts", () => {
const { container, rerender } = render(TWO_BUCKETS);
fireEvent.mouseOver(overlayRects(container)[0]);
expect(
Array.from((container.querySelector("dl") as HTMLElement).querySelectorAll("dd")).map((dd) => dd.textContent),
).toEqual(["35", "10", "20", "5"]);
rerender(
clients(
[
{ client: "192.0.2.30", buckets: [11, 1] },
{ client: "192.0.2.31", buckets: [22, 1] },
],
[6, 1],
),
);
expect(
Array.from((container.querySelector("dl") as HTMLElement).querySelectorAll("dd")).map((dd) => dd.textContent),
).toEqual(["39", "11", "22", "6"]);
});
/**
* A rolling window is the case a stored copy gets wrong: the bucket the pointer
* was over is gone, so index 0 now names a different span.
*/
test("a refresh that rolls the window takes the tooltip down instead of relabelling it", () => {
const { container, rerender } = render(TWO_BUCKETS);
fireEvent.mouseOver(overlayRects(container)[0]);
expect(container.querySelectorAll("dl")).toHaveLength(1);
rerender({ ...TWO_BUCKETS, since: SINCE + BUCKET });
expect(container.querySelectorAll("dl")).toHaveLength(0);
const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
expect(stacks.every((group) => group.getAttribute("opacity") === "1")).toBe(true);
});
/** The placement the hand-rolled tooltip had, restored over visx's 10px defaults. */
test("the tooltip is offset 8px from the chart top and from the bucket it names", () => {
const { container } = render(TWO_BUCKETS);
fireEvent.mouseOver(overlayRects(container)[0]);
const tooltip = container.querySelector(".visx-tooltip") as HTMLElement;
expect(tooltip.style.transform).toBe("translate(200px, 8px)");
});
/**
* `withBoundingRects` measures its node once, on mount, so the buckets must not
* share one. jsdom reports every rect as zero, so the mount is what this can
* observe, not the measurement itself.
*/
test("each bucket gets its own tooltip mount, so each is measured for itself", () => {
const { container } = render(TWO_BUCKETS);
const rects = overlayRects(container);
fireEvent.mouseOver(rects[0]);
const first = container.querySelector(".visx-tooltip");
fireEvent.mouseOver(rects[1]);
expect(first).not.toBeNull();
expect(container.querySelector(".visx-tooltip")).not.toBe(first);
});
test("pointing at a bucket names its total and every series, and dims the rest", () => {
const { container } = render(TWO_BUCKETS);
fireEvent.mouseOver(overlayRects(container)[0]);
const tooltip = container.querySelector("dl") as HTMLElement;
expect(tooltip.previousElementSibling?.textContent).toBe(formatTime(SINCE));
expect(Array.from(tooltip.querySelectorAll("dt")).map((dt) => dt.textContent)).toEqual([
"Queries",
"192.0.2.30",
"192.0.2.31",
"Other",
]);
expect(Array.from(tooltip.querySelectorAll("dd")).map((dd) => dd.textContent)).toEqual(["35", "10", "20", "5"]);
const swatches = Array.from(tooltip.querySelectorAll("dt span")).map((span) => span.getAttribute("style"));
expect(swatches[0]).toContain(seriesColor(clientKey("192.0.2.30")));
expect(swatches[2]).toContain(seriesColor(OTHER_KEY));
const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
expect(stacks.map((group) => group.getAttribute("opacity"))).toEqual(["1", "0.55"]);
});
test("leaving the chart takes the tooltip and the dimming with it", () => {
const { container } = render(TWO_BUCKETS);
fireEvent.mouseOver(overlayRects(container)[0]);
expect(container.querySelectorAll("dl")).toHaveLength(1);
fireEvent.mouseOut(container.querySelector("svg") as SVGSVGElement);
expect(container.querySelectorAll("dl")).toHaveLength(0);
const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
expect(stacks.every((group) => group.getAttribute("opacity") === "1")).toBe(true);
});
/**
* "Other" is everything outside the top eight clients. In a window where it
* counted nothing there is no eighth client to aggregate, and the entry would
* appear in the legend, the stack, the tooltip and the table saying only that it
* is empty. The named clients stay at zero: a client that went quiet is a fact.
*/
test("a window where Other counted nothing drops it from every surface", () => {
const { container } = render(clients([{ client: "192.0.2.30", buckets: [10, 4] }], [0, 0]));
expect(Array.from(container.querySelectorAll("ul li")).map((item) => item.textContent)).toEqual(["192.0.2.30"]);
expect(
within(screen.getByRole("table"))
.getAllByRole("columnheader")
.map((cell) => cell.textContent),
).toEqual(["Time", "192.0.2.30"]);
fireEvent.mouseOver(overlayRects(container)[0]);
const terms = Array.from((container.querySelector("dl") as HTMLElement).querySelectorAll("dt"));
expect(terms.map((term) => term.textContent)).toEqual(["Queries", "192.0.2.30"]);
});
test("one query outside the named clients is enough to keep Other", () => {
const { container } = render(clients([{ client: "192.0.2.30", buckets: [10, 4] }], [0, 1]));
expect(Array.from(container.querySelectorAll("ul li")).map((item) => item.textContent)).toEqual([
"192.0.2.30",
"Other",
]);
});
+133 -160
View File
@@ -2,56 +2,42 @@
* Client activity over the same window as the query-volume chart: one stacked
* series per named client, plus everything outside the top eight as "Other".
*
* The x-axis is the timeseries endpoint's own bucket alignment, so the two
* charts stack directly above one another and a spike in one is at the same
* horizontal position in the other. Colour keys on the client string, so a
* client that changes rank between polls keeps its colour.
* The x-axis is derived from this response's own `since` and `bucket_seconds`,
* which the API aligns with the timeseries endpoint's buckets, so the two charts
* stack directly above one another and a spike in one is at the same horizontal
* position in the other. Colour keys on the client string, so a client that
* changes rank between polls keeps its colour.
*/
import { useEffect, useRef, useState } from "react";
import * as stylex from "@stylexjs/stylex";
import { Group } from "@visx/group";
import { BarStack } from "@visx/shape";
import { formatTime } from "@/lib/format";
import type { StatsClients } from "@/lib/types";
import type { OverviewClientSeries } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { layoutStacked } from "./chartLayout";
import { clientLabel, useClientNames, type ClientNames } from "@/features/clients/clientNames";
import {
BucketOverlay,
CHART_HEIGHT,
ChartFrame,
ChartRoot,
ChartTooltip,
EmptyChart,
StackSegment,
bandScale,
labelTickValues,
plotArea,
slotCenter,
useActiveIndex,
useMeasuredWidth,
valueScale,
valueTicks,
type TooltipContent,
} from "./chartKit";
import { OTHER_KEY, clientKey, seriesColor } from "./seriesColors";
const CHART_HEIGHT = 240;
const FALLBACK_WIDTH = 640;
const styles = stylex.create({
empty: {
display: "flex",
alignItems: "center",
justifyContent: "center",
height: CHART_HEIGHT,
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "dashed",
borderColor: colors.borderStrong,
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
root: {
position: "relative",
},
gridLine: {
stroke: colors.border,
},
axisLine: {
stroke: colors.borderStrong,
},
axisLabel: {
fill: colors.textMuted,
fontSize: "10px",
},
/** The hairline separating touching segments is the page ground, not a colour. */
segment: {
stroke: colors.surface,
},
legend: {
marginTop: "0.5rem",
display: "flex",
@@ -80,31 +66,6 @@ const styles = stylex.create({
swatchColor: (color: string) => ({ backgroundColor: color }),
});
function useContainerWidth(): [React.RefObject<HTMLDivElement | null>, number] {
const ref = useRef<HTMLDivElement>(null);
const [width, setWidth] = useState(0);
useEffect(() => {
const el = ref.current;
if (el === null) return;
setWidth(el.clientWidth);
if (typeof ResizeObserver === "undefined") return;
const observer = new ResizeObserver(() => setWidth(el.clientWidth));
observer.observe(el);
return () => observer.disconnect();
}, []);
return [ref, width];
}
const compact = new Intl.NumberFormat(undefined, { notation: "compact" });
function formatTick(ts: number, bucketSeconds: number): string {
const date = new Date(ts * 1000);
if (bucketSeconds >= 86_400) {
return new Intl.DateTimeFormat(undefined, { month: "short", day: "numeric" }).format(date);
}
return new Intl.DateTimeFormat(undefined, { hour: "numeric", minute: "2-digit" }).format(date);
}
interface Series {
key: string;
label: string;
@@ -115,12 +76,13 @@ interface Series {
}
/**
* "Other" last, so it sits at the top of every column rather than under a
* client, and always present: the response always carries the series, and a
* legend that dropped it on a quiet period would make the reader think the
* chart's clients were all of them.
* "Other" last, so it sits at the top of every column rather than under a client,
* and dropped entirely when it counted nothing across the window: an aggregation
* bucket that aggregated nothing is a legend entry, a stack key, a tooltip row
* and a table column all saying zero. The named clients stay at zero, because a
* client that went quiet is something the reader wants to see.
*/
function seriesOf(data: StatsClients, names: ClientNames): Series[] {
function seriesOf(data: ClientChartData, names: ClientNames): Series[] {
const named = data.clients.map((client) => ({
key: clientKey(client.client),
// The name if the client is registered under one, the address otherwise —
@@ -131,119 +93,130 @@ function seriesOf(data: StatsClients, names: ClientNames): Series[] {
color: seriesColor(clientKey(client.client)),
buckets: client.buckets,
}));
if (data.other.every((count) => count === 0)) return named;
return [
...named,
{ key: OTHER_KEY, label: "Other", address: null, color: seriesColor(OTHER_KEY), buckets: data.other },
];
}
export default function ClientChart({ data }: { data: StatsClients }) {
const [containerRef, measuredWidth] = useContainerWidth();
/**
* The slice of the Overview body this chart draws. Declared here rather than
* taken whole, so what the chart reads is stated where it is read.
*/
export interface ClientChartData {
since: number;
bucket_seconds: number;
clients: OverviewClientSeries[];
other: number[];
}
/** One column: the timestamp plus one entry per series, keyed by the series key. */
type Column = { ts: number } & Record<string, number>;
export default function ClientChart({ data }: { data: ClientChartData }) {
const [containerRef, width] = useMeasuredWidth();
// A hover survives a re-render only while it still names the same bucket at
// the same place: a poll that rolls the window, or a resize, retires it.
const hovered = useActiveIndex(`${data.since}:${data.bucket_seconds}:${data.other.length}:${width}`);
const names = useClientNames();
const width = measuredWidth > 0 ? measuredWidth : FALLBACK_WIDTH;
const series = seriesOf(data, names);
const bucketCount = data.other.length;
const columns: Column[] = Array.from({ length: bucketCount }, (_, i) => {
const column: Column = { ts: data.since + i * data.bucket_seconds };
for (const one of series) column[one.key] = one.buckets[i] ?? 0;
return column;
});
if (bucketCount === 0) {
return (
<div ref={containerRef} {...stylex.props(styles.empty)}>
No queries in this period.
</div>
);
if (bucketCount === 0 || columns.every((column) => series.every((one) => column[one.key] === 0))) {
return <EmptyChart containerRef={containerRef} />;
}
const columns = Array.from({ length: bucketCount }, (_, i) => ({
ts: data.since + i * data.bucket_seconds,
values: series.map((one) => one.buckets[i] ?? 0),
}));
if (columns.every((column) => column.values.every((value) => value === 0))) {
return (
<div ref={containerRef} {...stylex.props(styles.empty)}>
No queries in this period.
</div>
);
const timestamps = columns.map((column) => column.ts);
const totals = columns.map((column) => series.reduce((sum, one) => sum + column[one.key], 0));
const plot = plotArea(width);
const xScale = bandScale(timestamps, plot);
// Every series here is a disjoint part of the whole rather than a highlighted
// subset of a separately reported total, so the tallest column's own sum is
// the scale.
const yScale = valueScale(Math.max(...totals), [plot.bottom, plot.y]);
const yTicks = valueTicks(yScale);
const colorOf = new Map(series.map((one) => [one.key, one.color]));
function tooltipOf(index: number): TooltipContent {
return {
title: formatTime(columns[index].ts),
rows: [
{ key: "queries", label: "Queries", value: String(totals[index]) },
...series.map((one) => ({
key: one.key,
label: one.label,
color: one.color,
value: String(columns[index][one.key]),
})),
],
};
}
const layout = layoutStacked(columns, width, CHART_HEIGHT);
const baseline = layout.plot.y + layout.plot.height;
return (
<div ref={containerRef} {...stylex.props(styles.root)}>
<ChartRoot containerRef={containerRef}>
<svg
role="img"
aria-label={`Client activity over time, ${bucketCount} buckets, ${series.length} series`}
width="100%"
height={CHART_HEIGHT}
viewBox={`0 0 ${width} ${CHART_HEIGHT}`}
onMouseLeave={hovered.clear}
>
{layout.yTicks.map((tick) => (
<g key={tick.value}>
<line
x1={layout.plot.x}
x2={layout.plot.x + layout.plot.width}
y1={tick.y}
y2={tick.y}
{...stylex.props(styles.gridLine)}
<ChartFrame
plot={plot}
yScale={yScale}
yTicks={yTicks}
xScale={xScale}
xTickValues={labelTickValues(timestamps, plot.width)}
bucketSeconds={data.bucket_seconds}
/>
<text
x={layout.plot.x - 6}
y={tick.y}
textAnchor="end"
dominantBaseline="middle"
{...stylex.props(styles.axisLabel, shared.tabularNums)}
<BarStack<Column, string>
data={columns}
keys={series.map((one) => one.key)}
x={(column) => column.ts}
xScale={xScale}
yScale={yScale}
color={(key) => colorOf.get(key) ?? seriesColor(key)}
>
{compact.format(tick.value)}
</text>
</g>
))}
<line
x1={layout.plot.x}
x2={layout.plot.x + layout.plot.width}
y1={baseline}
y2={baseline}
{...stylex.props(styles.axisLine)}
{(stacks) =>
columns.map((column, index) => (
<Group
key={column.ts}
opacity={hovered.index === null || hovered.index === index ? 1 : 0.55}
>
{stacks.map((stack) => {
const bar = stack.bars[index];
return (
<StackSegment
key={stack.key}
x={bar.x}
y={bar.y}
width={bar.width}
height={bar.height}
fill={bar.color}
/>
{layout.xTicks.map((tick) => (
<text
key={tick.ts}
x={tick.x}
y={baseline + 14}
textAnchor="middle"
{...stylex.props(styles.axisLabel)}
>
{formatTick(tick.ts, data.bucket_seconds)}
</text>
))}
{layout.columns.map((column) => (
<g key={column.ts}>
{column.segments.map((rect, index) =>
rect.height <= 0 ? null : (
<rect
key={series[index].key}
x={rect.x}
y={rect.y}
width={rect.width}
height={rect.height}
fill={series[index].color}
strokeWidth={rect.width > 3 ? 1 : 0}
{...stylex.props(styles.segment)}
/>
),
)}
<rect
x={column.slot.x}
y={column.slot.y}
width={column.slot.width}
height={column.slot.height}
fill="transparent"
>
<title>
{`${formatTime(column.ts)}: ${column.total} ${column.total === 1 ? "query" : "queries"}`}
</title>
</rect>
</g>
))}
);
})}
</Group>
))
}
</BarStack>
<BucketOverlay plot={plot} values={timestamps} xScale={xScale} onEnter={hovered.show} />
</svg>
{hovered.index !== null && (
<ChartTooltip
index={hovered.index}
content={tooltipOf(hovered.index)}
left={slotCenter(xScale, timestamps[hovered.index], plot)}
/>
)}
<ul {...stylex.props(styles.legend)}>
{series.map((one) => (
<li key={one.key} title={one.address ?? undefined} {...stylex.props(styles.legendItem)}>
@@ -269,14 +242,14 @@ export default function ClientChart({ data }: { data: StatsClients }) {
{columns.map((column) => (
<tr key={column.ts}>
<th scope="row">{formatTime(column.ts)}</th>
{column.values.map((value, index) => (
<td key={series[index].key}>{value}</td>
{series.map((one) => (
<td key={one.key}>{column[one.key]}</td>
))}
</tr>
))}
</tbody>
</table>
</div>
</div>
</ChartRoot>
);
}
+200
View File
@@ -0,0 +1,200 @@
import { fireEvent, render, screen, within } from "@testing-library/react";
import Donut, { type DonutSlice } from "./Donut";
function slice(key: string, value: number, color = "#000000"): DonutSlice {
return { key, label: key.toUpperCase(), value, color };
}
function ring(container: HTMLElement): SVGPathElement[] {
return Array.from(container.querySelectorAll("svg path"));
}
/** The x radius of every `A` command in a path, in the order they are drawn. */
function arcRadii(d: string): number[] {
return Array.from(d.matchAll(/A(-?[\d.]+),/g)).map((match) => Number(match[1]));
}
function draw(slices: DonutSlice[]) {
return render(<Donut slices={slices} caption="Queries by DNS type" unit="Queries" />);
}
test("a breakdown of nothing says so rather than dividing by zero", () => {
const { container } = draw([slice("a", 0), slice("b", 0)]);
expect(screen.getByText("No queries in this period.")).toBeTruthy();
expect(container.querySelector("svg")).toBeNull();
});
test("an empty breakdown says the same thing", () => {
draw([]);
expect(screen.getByText("No queries in this period.")).toBeTruthy();
});
/** A legend entry reading 0% is noise, and a zero slice has no arc to draw. */
test("zero-valued entries are dropped from the ring, the legend and the table", () => {
const { container } = draw([slice("a", 3), slice("z", 0), slice("b", 1)]);
expect(ring(container)).toHaveLength(2);
expect(screen.queryByText("Z")).toBeNull();
expect(
within(screen.getByRole("table"))
.getAllByRole("rowheader")
.map((row) => row.textContent),
).toEqual(["A", "B"]);
});
/**
* The caller has already ranked the slices, so the ring must not re-sort them:
* the first arc opens at twelve o'clock and each one starts where the last
* ended, running clockwise.
*
* The fixture runs 1 then 3 on purpose. d3's default pie sort is by value
* descending, so a fixture that happened to be given in descending order would
* pass with the sort left on and prove nothing.
*/
test("the ring starts at twelve o'clock and runs clockwise in the order given", () => {
const { container } = draw([slice("a", 1, "#112233"), slice("b", 3, "#445566")]);
const paths = ring(container);
// The small slice is drawn first because it was given first.
expect(paths[0].getAttribute("fill")).toBe("#112233");
expect(paths[0].getAttribute("d")?.startsWith("M0,-90")).toBe(true);
// A quarter turn clockwise from the top is three o'clock, where the second
// slice picks up.
expect(paths[1].getAttribute("fill")).toBe("#445566");
expect(paths[1].getAttribute("d")?.startsWith("M90,0")).toBe(true);
});
/**
* A whole-circle arc from a point back to itself draws nothing at all, so the
* one-entry case has to come out as a closed ring: two half arcs out and two
* back.
*/
test("a single entry is a closed ring, not a zero-length arc", () => {
const { container } = draw([slice("only", 7)]);
const paths = ring(container);
expect(paths).toHaveLength(1);
const d = paths[0].getAttribute("d") ?? "";
expect(d.match(/A/g)).toHaveLength(4);
// Two half arcs out at the outer radius and two back at the inner one: a
// 180px ring 36px thick.
expect(arcRadii(d)).toEqual([90, 90, 54, 54]);
});
test("shares are of the drawn total, in the legend and in the hidden table alike", () => {
draw([slice("a", 3), slice("b", 1)]);
expect(screen.getAllByText("75.0%")).toHaveLength(2);
expect(screen.getAllByText("25.0%")).toHaveLength(2);
});
/**
* The colour is the caller's, never recomputed from the key here: the page owns
* the identity-to-colour mapping and two donuts of one page must agree with it.
*/
test("each arc is filled with the colour its slice carries", () => {
const { container } = draw([slice("a", 3, "#112233"), slice("b", 1, "#445566")]);
expect(ring(container).map((path) => path.getAttribute("fill"))).toEqual(["#112233", "#445566"]);
});
/**
* Colour is a pure function of identity and so cannot rule out two slices of one
* panel sharing a hue. The stroke is what stops neighbours from merging into one
* shape.
*/
test("every arc is outlined in the panel's own surface colour", () => {
const { container } = draw([slice("a", 3), slice("b", 1)]);
for (const path of ring(container)) {
expect(path.getAttribute("stroke-width")).toBe("1");
expect(path.getAttribute("stroke")).toMatch(/^var\(--/);
}
});
test("the ring is decoration; the legend and the hidden table are the accessible surface", () => {
const { container } = draw([slice("a", 3), slice("b", 1)]);
const svg = container.querySelector("svg");
expect(svg?.getAttribute("aria-hidden")).toBe("true");
expect(svg?.getAttribute("focusable")).toBe("false");
expect(screen.getByRole("table", { name: "Queries by DNS type" })).toBeTruthy();
});
/**
* The ring gets the bar charts' hover treatment: the shared tooltip names the
* slice and repeats what the legend says about it, and the slices it is not
* naming dim out of the way. The ring stays `aria-hidden` — the tooltip is a
* pointer affordance, and the legend beside it is still the readable copy.
*/
test("pointing at a slice names it and dims the rest", () => {
const { container } = draw([slice("a", 3, "#112233"), slice("b", 1, "#445566")]);
const paths = ring(container);
fireEvent.mouseOver(paths[0]);
const tooltip = container.querySelector("dl") as HTMLElement;
expect(tooltip.previousElementSibling?.textContent).toBe("A");
expect(Array.from(tooltip.querySelectorAll("dt")).map((term) => term.textContent)).toEqual(["Queries", "Share"]);
expect(Array.from(tooltip.querySelectorAll("dd")).map((value) => value.textContent)).toEqual(["3", "75.0%"]);
// The same share the legend and the hidden table already print for this slice.
expect(screen.getAllByText("75.0%")).toHaveLength(3);
expect(paths.map((path) => path.getAttribute("opacity"))).toEqual(["1", "0.55"]);
expect(container.querySelector("svg")?.getAttribute("aria-hidden")).toBe("true");
// The tooltip points at the middle of the arc, which the component computes
// from the slice values rather than from the drawn path. Slice A is three
// quarters of the ring, so its midpoint is at 135 degrees, on a circle of
// radius 72 — (140.9, 140.9) from the ring's top-left corner, plus the 8px
// the tooltip stands off by. Nothing else here would catch that arithmetic
// drifting away from the ring the Pie actually draws.
const tooltipBox = container.querySelector(".visx-tooltip") as HTMLElement;
expect(tooltipBox.style.transform).toBe("translate(149px, 149px)");
});
test("leaving the ring takes the tooltip and the dimming with it", () => {
const { container } = draw([slice("a", 3), slice("b", 1)]);
fireEvent.mouseOver(ring(container)[0]);
expect(container.querySelectorAll("dl")).toHaveLength(1);
fireEvent.mouseOut(container.querySelector("svg") as SVGSVGElement);
expect(container.querySelectorAll("dl")).toHaveLength(0);
expect(ring(container).map((path) => path.getAttribute("opacity"))).toEqual(["1", "1"]);
});
/**
* The same slice can change width between refreshes, so a mount measured at the
* old width would be placed at the wrong one. This is the donut's half of the
* remount the bar charts do.
*/
test("a refresh keeps the hovered slice current and remeasures it", () => {
const { container, rerender } = draw([slice("a", 3), slice("b", 1)]);
fireEvent.mouseOver(ring(container)[0]);
const first = container.querySelector(".visx-tooltip");
expect(Array.from(first?.querySelectorAll("dd") ?? []).map((value) => value.textContent)).toEqual(["3", "75.0%"]);
rerender(<Donut slices={[slice("a", 3000), slice("b", 1000)]} caption="Queries by DNS type" unit="Queries" />);
const second = container.querySelector(".visx-tooltip");
expect(Array.from(second?.querySelectorAll("dd") ?? []).map((value) => value.textContent)).toEqual([
"3,000",
"75.0%",
]);
expect(second).not.toBe(first);
});
/** A slice that is gone cannot be described, so the hover goes with it. */
test("a refresh that changes the slices takes the tooltip down", () => {
const { container, rerender } = draw([slice("a", 3), slice("b", 1)]);
fireEvent.mouseOver(ring(container)[0]);
expect(container.querySelectorAll("dl")).toHaveLength(1);
rerender(<Donut slices={[slice("c", 3), slice("b", 1)]} caption="Queries by DNS type" unit="Queries" />);
expect(container.querySelectorAll("dl")).toHaveLength(0);
});
+119 -34
View File
@@ -13,12 +13,26 @@
*/
import * as stylex from "@stylexjs/stylex";
import { Group } from "@visx/group";
import { Pie } from "@visx/shape";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { layoutDonut, type DonutSlice } from "./donutLayout";
import { ChartTooltip, useActiveIndex, type TooltipContent } from "./chartKit";
export interface DonutSlice {
/** Semantic identity: the React key, the colour key and the legend's identity. */
key: string;
label: string;
/** Disambiguates entries whose labels collide — two "Unknown"s, one name on two route kinds. */
secondary?: string;
value: number;
color: string;
}
const SIZE = 180;
const THICKNESS = 36;
const OUTER_RADIUS = SIZE / 2;
const INNER_RADIUS = OUTER_RADIUS - THICKNESS;
const numberFormat = new Intl.NumberFormat();
@@ -54,8 +68,12 @@ const styles = stylex.create({
justifyContent: { default: "center", [TWO_COLUMN]: "flex-start" },
gap: "1.25rem",
},
/** The tooltip is placed against the ring's own box, so slice coordinates can
* be used unchanged rather than measured against the whole panel. */
ring: {
position: "relative",
flexShrink: 0,
lineHeight: 0,
},
/**
* Capped and left-anchored. Without the cap the row justifies across whatever
@@ -115,6 +133,21 @@ function sharePercent(share: number): string {
return `${(share * 100).toFixed(1)}%`;
}
/**
* Where a slice's tooltip points: the middle of its arc, in the ring box's own
* coordinates. This restates the `Pie` configuration below — clockwise from
* twelve o'clock, caller order, no pad angle — and the two must agree.
*/
function sliceAnchor(drawn: DonutSlice[], index: number, total: number): { left: number; top: number } {
const before = drawn.slice(0, index).reduce((sum, slice) => sum + slice.value, 0);
const middle = (2 * Math.PI * (before + drawn[index].value / 2)) / total;
const radius = (OUTER_RADIUS + INNER_RADIUS) / 2;
return {
left: OUTER_RADIUS + Math.sin(middle) * radius,
top: OUTER_RADIUS - Math.cos(middle) * radius,
};
}
export default function Donut({
slices,
caption,
@@ -126,56 +159,108 @@ export default function Donut({
/** The column header for the counted thing, e.g. "Queries". */
unit: string;
}) {
const layout = layoutDonut(slices, SIZE, THICKNESS);
// Zero-valued entries have no arc to draw and a legend entry reading 0 is
// noise, so they are dropped before anything is measured or drawn.
const drawn = slices.filter((slice) => slice.value > 0);
const total = drawn.reduce((sum, slice) => sum + slice.value, 0);
// A hover names a slice by position, so it survives only while the ring is
// still made of the same slices in the same order.
const hovered = useActiveIndex(drawn.map((slice) => slice.key).join(","));
if (layout.total === 0) {
function tooltipOf(index: number): TooltipContent {
const slice = drawn[index];
return {
title: slice.secondary === undefined ? slice.label : `${slice.label} (${slice.secondary})`,
rows: [
{ key: "value", label: unit, color: slice.color, value: numberFormat.format(slice.value) },
{ key: "share", label: "Share", value: sharePercent(slice.value / total) },
],
};
}
if (total === 0) {
return <div {...stylex.props(styles.empty)}>No queries in this period.</div>;
}
return (
<div {...stylex.props(styles.body)}>
<div {...stylex.props(styles.ring)}>
{/* The ring stays out of the accessibility tree even though it is now
a pointer target: the tooltip repeats what the legend beside it
already says in text, so nothing here is the only copy. */}
<svg
aria-hidden="true"
focusable="false"
width={SIZE}
height={SIZE}
viewBox={`0 0 ${SIZE} ${SIZE}`}
{...stylex.props(styles.ring)}
onMouseLeave={hovered.clear}
>
{layout.arcs.map((arc) => (
// The stroke is what keeps a shared hue from lying. Colour is a pure
// function of identity, so two neighbouring slices can come out the
// same; outlined in the panel's own colour they still read as two
// shapes rather than merging into one. Attributes rather than a
// class, as the client chart's segments are, so the separation is
// visible to a test and not only to a stylesheet.
{/* Arc paths are generated around the origin, and `Pie`'s own
`top`/`left` group is skipped when it is given a render prop, so
the ring is centred here instead. */}
<Group top={OUTER_RADIUS} left={OUTER_RADIUS}>
<Pie
data={drawn}
pieValue={(slice) => slice.value}
// The caller has already ranked the slices, so d3's own sort is off
// in both of its forms and the ring runs clockwise from twelve
// o'clock in the order given. @visx/shape 4.0.0 already disables
// it by default — but that default exists because d3-shape v3
// changed sortValues to descending under it, so saying it here is
// what keeps a future bump from silently re-ranking the ring.
pieSort={null}
pieSortValues={null}
outerRadius={OUTER_RADIUS}
innerRadius={INNER_RADIUS}
>
{({ arcs, path }) =>
arcs.map((arc, index) => (
// The stroke is what keeps a shared hue from lying. Colour is a
// pure function of identity, so two neighbouring slices can come
// out the same; outlined in the panel's own colour they still
// read as two shapes rather than merging into one. Attributes
// rather than a class, as the client chart's segments are, so
// the separation is visible to a test and not only to a
// stylesheet.
<path
key={arc.slice.key}
d={arc.d}
fill={arc.slice.color}
fillRule="evenodd"
key={arc.data.key}
d={path(arc) ?? ""}
fill={arc.data.color}
stroke={colors.surfaceRaised}
strokeWidth={1}
opacity={hovered.index === null || hovered.index === index ? 1 : 0.55}
onMouseEnter={() => hovered.show(index)}
/>
))}
))
}
</Pie>
</Group>
</svg>
<ul {...stylex.props(styles.legend)}>
{layout.arcs.map((arc) => (
<li key={arc.slice.key} {...stylex.props(styles.legendItem)}>
<span
aria-hidden="true"
{...stylex.props(styles.swatch, styles.swatchColor(arc.slice.color))}
{hovered.index !== null && (
<ChartTooltip
index={hovered.index}
content={tooltipOf(hovered.index)}
{...sliceAnchor(drawn, hovered.index, total)}
/>
)}
</div>
<ul {...stylex.props(styles.legend)}>
{drawn.map((slice) => (
<li key={slice.key} {...stylex.props(styles.legendItem)}>
<span aria-hidden="true" {...stylex.props(styles.swatch, styles.swatchColor(slice.color))} />
<span {...stylex.props(styles.label)}>
{arc.slice.label}
{arc.slice.secondary !== undefined && (
<span {...stylex.props(styles.secondary)}>{arc.slice.secondary}</span>
{slice.label}
{slice.secondary !== undefined && (
<span {...stylex.props(styles.secondary)}>{slice.secondary}</span>
)}
</span>
<span {...stylex.props(styles.count, shared.tabularNums)}>
{numberFormat.format(arc.slice.value)}
{numberFormat.format(slice.value)}
</span>
<span {...stylex.props(styles.share, shared.tabularNums)}>
{sharePercent(slice.value / total)}
</span>
<span {...stylex.props(styles.share, shared.tabularNums)}>{sharePercent(arc.share)}</span>
</li>
))}
</ul>
@@ -190,15 +275,15 @@ export default function Donut({
</tr>
</thead>
<tbody>
{layout.arcs.map((arc) => (
<tr key={arc.slice.key}>
{drawn.map((slice) => (
<tr key={slice.key}>
<th scope="row">
{arc.slice.secondary === undefined
? arc.slice.label
: `${arc.slice.label} (${arc.slice.secondary})`}
{slice.secondary === undefined
? slice.label
: `${slice.label} (${slice.secondary})`}
</th>
<td>{arc.slice.value}</td>
<td>{sharePercent(arc.share)}</td>
<td>{slice.value}</td>
<td>{sharePercent(slice.value / total)}</td>
</tr>
))}
</tbody>
@@ -0,0 +1,133 @@
/**
* The part of Overview that does not wait for anything: the heading, the period
* picker, and the pulsing body the page shows while the window is in flight.
*
* It lives apart from `OverviewPage` so the route's pending component can render
* the identical surface while the page chunk loads. Importing the page itself
* would pull the charts into the main bundle, and a second hand-written copy of
* the frame would drift. Nothing here imports a chart.
*/
import * as stylex from "@stylexjs/stylex";
import { useNavigate, useSearch } from "@tanstack/react-router";
import type { Period } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { DEFAULT_PERIOD, PERIODS } from "./period";
const styles = stylex.create({
page: {
display: "flex",
flexDirection: "column",
gap: "1rem",
},
headingRow: {
display: "flex",
flexWrap: "wrap",
alignItems: "center",
justifyContent: "space-between",
gap: "0.75rem",
},
heading: {
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
},
periodGroup: {
display: "flex",
gap: "0.25rem",
},
period: {
borderStyle: "none",
borderRadius: "0.25rem",
paddingInline: "0.625rem",
paddingBlock: "0.25rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
/** The pressed fill is heavier than `surfaceHover`, so a hover cannot mimic it. */
periodSelected: {
backgroundColor: {
default: "oklch(92% 0.004 286.32)",
"@media (prefers-color-scheme: dark)": "oklch(37% 0.013 285.805)",
},
color: colors.text,
fontWeight: 500,
},
periodIdle: {
backgroundColor: { default: "transparent", ":hover": colors.surfaceHover },
color: colors.textSecondary,
},
loading: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
});
export function PeriodPicker({ period, onChange }: { period: Period; onChange: (period: Period) => void }) {
return (
<div role="group" aria-label="Period" {...stylex.props(styles.periodGroup)}>
{PERIODS.map((option) => (
<button
key={option}
type="button"
aria-pressed={option === period}
onClick={() => onChange(option)}
{...stylex.props(
styles.period,
option === period ? styles.periodSelected : styles.periodIdle,
shared.focusRing,
)}
>
{option}
</button>
))}
</div>
);
}
export function OverviewLoading() {
return (
<p role="status" {...stylex.props(styles.loading, shared.pulse)}>
Loading
</p>
);
}
export function OverviewFrame({
period,
onChange,
children,
}: {
period: Period;
onChange: (period: Period) => void;
children: React.ReactNode;
}) {
return (
<div {...stylex.props(styles.page)}>
<div {...stylex.props(styles.headingRow)}>
<h1 {...stylex.props(styles.heading)}>Overview</h1>
<PeriodPicker period={period} onChange={onChange} />
</div>
{children}
</div>
);
}
/**
* The route's pending surface. The picker stays live because it only writes the
* search parameter, which the route already re-reads on its own.
*/
export function OverviewPending() {
const period = useSearch({ from: "/shell/overview" }).period ?? DEFAULT_PERIOD;
const navigate = useNavigate({ from: "/overview" });
return (
<OverviewFrame
period={period}
onChange={(next) => void navigate({ search: (prev) => ({ ...prev, period: next }) })}
>
<OverviewLoading />
</OverviewFrame>
);
}
@@ -14,66 +14,34 @@ import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import { clientKey, seriesColor } from "./seriesColors";
import { clientKey, qtypeKey, seriesColor } from "./seriesColors";
import { health } from "@/lib/healthFixture";
import type { Health, StatsClients, StatsRoutes, StatsTimeseries, StatsTotals, StatsTypes } from "@/lib/types";
import type { Health, Overview } from "@/lib/types";
const SINCE = Date.UTC(2026, 0, 1, 0, 0) / 1000;
const UNTIL = Date.UTC(2026, 0, 2, 0, 0) / 1000;
const COVERAGE = { complete: true, available_since: SINCE };
const TOTALS: StatsTotals = {
period: "24h",
since: SINCE,
until: UNTIL,
queries: 1000,
blocked: 250,
clients: 7,
avg_response_time_us: 2345,
coverage: COVERAGE,
};
const SERIES: StatsTimeseries = {
const OVERVIEW: Overview = {
period: "24h",
since: SINCE,
until: UNTIL,
bucket_seconds: 1800,
totals: { queries: 1000, blocked: 250, clients: 7, avg_response_time_us: 2345 },
buckets: [
{ ts: SINCE, queries: 60, blocked: 20, cached: 10 },
{ ts: SINCE + 1800, queries: 40, blocked: 0, cached: 0 },
],
coverage: COVERAGE,
};
const CLIENTS: StatsClients = {
period: "24h",
since: SINCE,
until: UNTIL,
bucket_seconds: 1800,
clients: [
{ client: "192.0.2.30", buckets: [40, 20] },
{ client: "192.0.2.31", buckets: [20, 20] },
],
other: [0, 0],
coverage: COVERAGE,
};
const TYPES: StatsTypes = {
period: "24h",
since: SINCE,
until: UNTIL,
types: [
{ qtype: 1, count: 600 },
{ qtype: 28, count: 300 },
{ qtype: null, count: 100 },
],
coverage: COVERAGE,
};
const ROUTES: StatsRoutes = {
period: "24h",
since: SINCE,
until: UNTIL,
routes: [
{ route: "upstream", source: "https://dns.example/dns-query", count: 500 },
{ route: "blocked", source: null, count: 250 },
@@ -83,22 +51,27 @@ const ROUTES: StatsRoutes = {
coverage: COVERAGE,
};
/** The same shapes an hour wide, so a period change is observable in every panel. */
const HOUR = {
totals: { ...TOTALS, period: "1h", since: UNTIL - 3600, queries: 12, blocked: 3, clients: 2 } as StatsTotals,
timeseries: { ...SERIES, period: "1h", since: UNTIL - 3600, bucket_seconds: 60, buckets: [] } as StatsTimeseries,
clients: { ...CLIENTS, period: "1h", since: UNTIL - 3600, clients: [], other: [] } as StatsClients,
types: { ...TYPES, period: "1h", since: UNTIL - 3600, types: [] } as StatsTypes,
routes: { ...ROUTES, period: "1h", since: UNTIL - 3600, routes: [] } as StatsRoutes,
/** The same shape an hour wide and empty, so a period change is observable. */
const HOUR: Overview = {
...OVERVIEW,
period: "1h",
since: UNTIL - 3600,
bucket_seconds: 60,
totals: { queries: 12, blocked: 3, clients: 2, avg_response_time_us: 2345 },
buckets: [],
clients: [],
other: [],
types: [],
routes: [],
};
let healthBody: Health;
let failing: Set<string>;
let failing: boolean;
/** The registered clients, as `/api/clients` answers them. */
let registered: { ip: string; name: string; learned_name: string }[];
let coverageComplete: boolean;
/** Paths held in flight, so a test can look at the page while one is pending. */
let delayed: Map<string, Promise<void>>;
/** Held in flight, so a test can look at the page while the request is pending. */
let delayed: Promise<void> | null;
function json(payload: unknown, status = 200): Response {
return new Response(JSON.stringify(payload), { status, headers: { "content-type": "application/json" } });
@@ -110,27 +83,18 @@ function withCoverage<T extends { coverage: typeof COVERAGE }>(body: T): T {
beforeEach(() => {
healthBody = health();
failing = new Set();
failing = false;
registered = [];
coverageComplete = true;
delayed = new Map();
delayed = null;
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
const hour = url.includes("period=1h");
for (const [path, body] of [
["/api/stats/timeseries", hour ? HOUR.timeseries : SERIES],
["/api/stats/clients", hour ? HOUR.clients : CLIENTS],
["/api/stats/types", hour ? HOUR.types : TYPES],
["/api/stats/routes", hour ? HOUR.routes : ROUTES],
["/api/stats", hour ? HOUR.totals : TOTALS],
] as const) {
if (!url.startsWith(path)) continue;
if (failing.has(path)) return json({ error: "endpoint unavailable" }, 400);
const held = delayed.get(path);
if (held !== undefined) await held;
return json(withCoverage(body));
if (url.startsWith("/api/overview")) {
if (failing) return json({ error: "endpoint unavailable" }, 400);
if (delayed !== null) await delayed;
return json(withCoverage(url.includes("period=1h") ? HOUR : OVERVIEW));
}
if (url === "/api/clients") {
return json({
@@ -203,25 +167,39 @@ test("every donut arc is outlined, so two slices of one hue still read as two",
}
});
test("a slow endpoint does not hold the page back: the panels that answered render beside it", async () => {
// Through the real route, which is the point: the loader starts the five
// requests and awaits none of them. If it awaited, the router would hold the
// whole page until the slowest answered and this would time out on the tiles.
test("the page builds a donut slice's colour from the entry's identity", async () => {
// `Donut` renders the colour it is handed and never recomputes one, so the
// mapping from identity to hue is the page's job and is pinned here.
renderApp();
await screen.findByText("1,000");
await waitFor(() => expect(within(panel("Query types")).getAllByText("A")).toHaveLength(2));
const item = within(panel("Query types")).getAllByText("A")[0].closest("li") as HTMLElement;
const swatch = item.querySelector("span[aria-hidden]") as HTMLElement;
expect(swatch.getAttribute("style")).toContain(seriesColor(qtypeKey(1)));
});
test("a request in flight leaves the heading and the picker usable behind one loading surface", async () => {
// Through the real route, which is the point: the loader starts the request
// and awaits it nowhere. If it awaited, the router would hold the whole page —
// heading and period picker included — until the response landed.
let release = () => {};
delayed.set("/api/stats/routes", new Promise<void>((resolve) => (release = resolve)));
delayed = new Promise<void>((resolve) => (release = resolve));
renderApp();
// The tiles and both charts are readable while the routes request is still
// in flight, and the panel waiting on it says so for itself.
await screen.findByText("1,000");
expect(within(panel("Queries over time")).getAllByText("Blocked").length).toBeGreaterThan(0);
expect(within(panel("Client activity over time")).getAllByText("192.0.2.30")).toHaveLength(2);
expect(within(panel("Query types")).getAllByText("A")).toHaveLength(2);
expect(within(panel("Upstream servers")).getByRole("status").textContent).toBe("Loading…");
await screen.findByRole("heading", { name: "Overview", level: 1 });
expect(screen.getByRole("button", { name: "1h" })).toBeTruthy();
// One loading state for the whole page, not one per panel.
const loading = await screen.findByText("Loading…");
expect(loading.getAttribute("role")).toBe("status");
expect(screen.getAllByText("Loading…")).toHaveLength(1);
expect(screen.queryByRole("heading", { name: "Query types" })).toBeNull();
release();
await waitFor(() => expect(within(panel("Upstream servers")).queryByRole("status")).toBeNull());
delayed = null;
await screen.findByText("1,000");
expect(screen.queryByText("Loading…")).toBeNull();
});
test("a registered client is named in the chart, an unregistered one keeps its address", async () => {
@@ -259,9 +237,10 @@ test("naming a client does not recolour its series", async () => {
expect(swatch.getAttribute("style")).toContain(seriesColor(clientKey("192.0.2.30")));
});
test("the client chart names Other even in a period where it counted nothing", async () => {
// The fixture's other series is all zeroes. Dropping it from the legend there
// would tell the reader the two named clients were every client.
test("the client chart drops Other in a period where it counted nothing", async () => {
// The fixture's other series is all zeroes. An aggregation bucket that
// aggregated nothing is a legend entry and a table column that say only that
// they are empty; the named clients stay, because a quiet client is a fact.
renderApp();
await screen.findByRole("heading", { name: "Client activity over time" });
@@ -269,8 +248,8 @@ test("the client chart names Other even in a period where it counted nothing", a
expect(chart).toBeTruthy();
// Twice each: the legend swatch and the column header of the table a screen
// reader gets instead of the graphic.
await waitFor(() => expect(within(chart as HTMLElement).getAllByText("Other")).toHaveLength(2));
expect(within(chart as HTMLElement).getAllByText("192.0.2.30")).toHaveLength(2);
await waitFor(() => expect(within(chart as HTMLElement).getAllByText("192.0.2.30")).toHaveLength(2));
expect(within(chart as HTMLElement).queryAllByText("Other")).toHaveLength(0);
});
test("the page is four tiles, two charts and two donuts — no status or issues sections", async () => {
@@ -379,17 +358,24 @@ test("the picker rescopes every panel and writes the period into the url", async
expect(screen.queryByText("1,000")).toBeNull();
});
test("one failing panel keeps its own error and leaves the rest of the page standing", async () => {
failing.add("/api/stats/routes");
test("a failed request is one error for the whole page, stated once and retryable", async () => {
failing = true;
renderApp();
await screen.findByText("1,000");
await waitFor(() => expect(within(panel("Upstream servers")).getByText("endpoint unavailable")).toBeTruthy());
expect(within(panel("Upstream servers")).getByRole("button", { name: "Retry" })).toBeTruthy();
// A failed donut never blanks the charts.
expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
expect(screen.getByRole("img", { name: /client activity over time/i })).toBeTruthy();
await screen.findByText("endpoint unavailable");
// One statement of the failure, not one per panel: there is a single request
// behind every panel, so a second copy would only repeat this sentence.
expect(screen.getAllByText("endpoint unavailable")).toHaveLength(1);
expect(screen.getAllByRole("button", { name: "Retry" })).toHaveLength(1);
// The heading and the picker survive it, so the reader can rescope or retry.
expect(screen.getByRole("heading", { name: "Overview", level: 1 })).toBeTruthy();
expect(screen.getByRole("button", { name: "1h" })).toBeTruthy();
expect(screen.queryByText("Something went wrong")).toBeNull();
failing = false;
fireEvent.click(screen.getByRole("button", { name: "Retry" }));
await screen.findByText("1,000");
expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
});
test("an incomplete window states its watermark once for the whole page", async () => {
+32 -113
View File
@@ -8,9 +8,9 @@
* The period is URL state, so a view is a link: `/overview?period=1h` opens
* exactly what the sender was reading.
*
* Every panel reads the same window (`overviewWindow.ts`) and renders on its
* own. A donut whose request failed shows its own error while the charts keep
* their data, and no two panels ever describe different spans.
* One request feeds every panel (`overviewWindow.ts`), so the page has one
* loading state and one error state rather than six: there is no longer a
* partial answer to render, and nothing left for a panel to disagree about.
*/
import * as stylex from "@stylexjs/stylex";
@@ -18,16 +18,16 @@ import { useNavigate, useSearch } from "@tanstack/react-router";
import CoverageNotice from "@/lib/CoverageNotice";
import InlineError from "@/lib/InlineError";
import { qtypeName } from "@/features/provenance/qtype";
import type { Period, StatsRoutes, StatsTypes } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import type { Overview, OverviewRouteRow, OverviewTypeRow } from "@/lib/types";
import { colors } from "@/ui/tokens.stylex";
import ClientChart from "./ClientChart";
import Donut from "./Donut";
import StatTiles from "./StatTiles";
import TimeseriesChart from "./TimeseriesChart";
import type { DonutSlice } from "./donutLayout";
import type { DonutSlice } from "./Donut";
import { OverviewFrame, OverviewLoading } from "./OverviewFrame";
import { useOverviewWindow, type Panel } from "./overviewWindow";
import { DEFAULT_PERIOD, PERIODS } from "./period";
import { DEFAULT_PERIOD } from "./period";
import { qtypeKey, routeKey, seriesColor } from "./seriesColors";
/**
@@ -48,48 +48,6 @@ const ROUTE_LABELS = {
} as const;
const styles = stylex.create({
page: {
display: "flex",
flexDirection: "column",
gap: "1rem",
},
headingRow: {
display: "flex",
flexWrap: "wrap",
alignItems: "center",
justifyContent: "space-between",
gap: "0.75rem",
},
heading: {
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
},
periodGroup: {
display: "flex",
gap: "0.25rem",
},
period: {
borderStyle: "none",
borderRadius: "0.25rem",
paddingInline: "0.625rem",
paddingBlock: "0.25rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
/** The pressed fill is heavier than `surfaceHover`, so a hover cannot mimic it. */
periodSelected: {
backgroundColor: {
default: "oklch(92% 0.004 286.32)",
"@media (prefers-color-scheme: dark)": "oklch(37% 0.013 285.805)",
},
color: colors.text,
fontWeight: 500,
},
periodIdle: {
backgroundColor: { default: "transparent", ":hover": colors.surfaceHover },
color: colors.textSecondary,
},
panel: {
borderRadius: "0.25rem",
borderWidth: 1,
@@ -110,54 +68,20 @@ const styles = stylex.create({
gap: "1rem",
gridTemplateColumns: { default: "minmax(0, 1fr)", [TWO_COLUMN]: "repeat(2, minmax(0, 1fr))" },
},
loading: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
});
function PeriodPicker({ period, onChange }: { period: Period; onChange: (period: Period) => void }) {
return (
<div role="group" aria-label="Period" {...stylex.props(styles.periodGroup)}>
{PERIODS.map((option) => (
<button
key={option}
type="button"
aria-pressed={option === period}
onClick={() => onChange(option)}
{...stylex.props(
styles.period,
option === period ? styles.periodSelected : styles.periodIdle,
shared.focusRing,
)}
>
{option}
</button>
))}
</div>
);
}
/**
* One panel's three states. Loading and error are the panel's own: a failure
* here never reaches past this box, which is what keeps a failed donut from
* blanking the charts beside it.
* The page's three states. The heading and the period picker stay put through
* all three, so the reader can rescope or retry without waiting for anything.
*/
function PanelBody<T>({ panel, children }: { panel: Panel<T>; children: (data: T) => React.ReactNode }) {
function PageBody({ panel, children }: { panel: Panel<Overview>; children: (data: Overview) => React.ReactNode }) {
if (panel.status === "error") return <InlineError error={panel.error} onRetry={panel.retry} />;
if (panel.status === "loading") {
return (
<p role="status" {...stylex.props(styles.loading, shared.pulse)}>
Loading
</p>
);
}
if (panel.status === "loading") return <OverviewLoading />;
return <>{children(panel.data)}</>;
}
function typeSlices(data: StatsTypes): DonutSlice[] {
return data.types.map((row) => ({
function typeSlices(types: OverviewTypeRow[]): DonutSlice[] {
return types.map((row) => ({
key: qtypeKey(row.qtype),
label: row.qtype === null ? "Unknown" : qtypeName(row.qtype),
value: row.count,
@@ -171,8 +95,8 @@ function typeSlices(data: StatsTypes): DonutSlice[] {
* appear under two kinds and two rows can both be "Unknown". The four
* source-less kinds are their own label and need no qualifier.
*/
function routeSlices(data: StatsRoutes): DonutSlice[] {
return data.routes.map((row) => {
function routeSlices(routes: OverviewRouteRow[]): DonutSlice[] {
return routes.map((row) => {
const named = row.route === "upstream" || row.route === "forward_zone";
return {
key: routeKey(row.route, row.source),
@@ -190,33 +114,31 @@ export default function OverviewPage() {
const overview = useOverviewWindow(period);
return (
<div {...stylex.props(styles.page)}>
<div {...stylex.props(styles.headingRow)}>
<h1 {...stylex.props(styles.heading)}>Overview</h1>
<PeriodPicker
<OverviewFrame
period={period}
onChange={(next) => void navigate({ search: (prev) => ({ ...prev, period: next }) })}
/>
</div>
>
<PageBody panel={overview}>
{(data) => (
<>
<StatTiles stats={{ since: data.since, until: data.until, ...data.totals }} />
<PanelBody panel={overview.totals}>{(totals) => <StatTiles stats={totals} />}</PanelBody>
{/* One notice for the page: every panel is judged against the same window,
so a second copy would only repeat this sentence. */}
{overview.coverage !== null && <CoverageNotice coverage={overview.coverage} />}
{/* One notice for the page: every panel came out of this one
response, so a second copy would only repeat this sentence. */}
<CoverageNotice coverage={data.coverage} />
<section aria-labelledby="overview-queries" {...stylex.props(styles.panel)}>
<h2 id="overview-queries" {...stylex.props(styles.panelHeading)}>
Queries over time
</h2>
<PanelBody panel={overview.timeseries}>{(data) => <TimeseriesChart data={data} />}</PanelBody>
<TimeseriesChart data={data} />
</section>
<section aria-labelledby="overview-clients" {...stylex.props(styles.panel)}>
<h2 id="overview-clients" {...stylex.props(styles.panelHeading)}>
Client activity over time
</h2>
<PanelBody panel={overview.clients}>{(data) => <ClientChart data={data} />}</PanelBody>
<ClientChart data={data} />
</section>
<div {...stylex.props(styles.donutRow)}>
@@ -224,25 +146,22 @@ export default function OverviewPage() {
<h2 id="overview-types" {...stylex.props(styles.panelHeading)}>
Query types
</h2>
<PanelBody panel={overview.types}>
{(data) => <Donut slices={typeSlices(data)} caption="Queries by DNS type" unit="Queries" />}
</PanelBody>
<Donut slices={typeSlices(data.types)} caption="Queries by DNS type" unit="Queries" />
</section>
<section aria-labelledby="overview-routes" {...stylex.props(styles.panel)}>
<h2 id="overview-routes" {...stylex.props(styles.panelHeading)}>
Upstream servers
</h2>
<PanelBody panel={overview.routes}>
{(data) => (
<Donut
slices={routeSlices(data)}
slices={routeSlices(data.routes)}
caption="Queries by how they were answered"
unit="Queries"
/>
)}
</PanelBody>
</section>
</div>
</div>
</>
)}
</PageBody>
</OverviewFrame>
);
}
+9 -3
View File
@@ -5,7 +5,7 @@
* typographic, so the eye ranks the figures rather than the panels, and a tile
* never implies a state it is not reporting.
*
* The Activity links carry the bounds the **stats response** returned, not
* The Activity links carry the bounds the **overview response** returned, not
* bounds computed here — a client-computed window would send the reader to a
* slightly different span than the one they were just reading.
*/
@@ -13,7 +13,7 @@
import * as stylex from "@stylexjs/stylex";
import { Link } from "@tanstack/react-router";
import { formatMicros } from "@/lib/format";
import type { StatsTotals } from "@/lib/types";
import type { OverviewTotals } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
@@ -101,7 +101,13 @@ function Tile({
);
}
export default function StatTiles({ stats }: { stats: StatsTotals }) {
/** The window's totals with the bounds they were measured over. */
export interface StatTilesData extends OverviewTotals {
since: number;
until: number;
}
export default function StatTiles({ stats }: { stats: StatTilesData }) {
const window = {
mode: "history" as const,
since: stats.since,
@@ -1,25 +1,25 @@
import { render, screen, within } from "@testing-library/react";
import { fireEvent, render, screen, within } from "@testing-library/react";
import * as stylex from "@stylexjs/stylex";
import type { StatsTimeseries } from "@/lib/types";
import { formatTime } from "@/lib/format";
import type { Bucket } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import TimeseriesChart from "./TimeseriesChart";
import TimeseriesChart, { type TimeseriesData } from "./TimeseriesChart";
const SINCE = 1_700_000_000;
function timeseries(bucketCount: number): StatsTimeseries {
return {
period: "24h",
since: SINCE,
until: SINCE + bucketCount * 1800,
bucket_seconds: 1800,
coverage: { complete: true, available_since: SINCE },
buckets: Array.from({ length: bucketCount }, (_, i) => ({
function timeseries(buckets: Bucket[]): TimeseriesData {
return { since: SINCE, bucket_seconds: 1800, buckets };
}
function counting(bucketCount: number): TimeseriesData {
return timeseries(
Array.from({ length: bucketCount }, (_, i) => ({
ts: SINCE + i * 1800,
queries: i + 1,
blocked: 1,
cached: 1,
})),
};
);
}
/** The element wearing the shared hidden style, found by its compiled classes. */
@@ -29,8 +29,13 @@ function hiddenElement(container: HTMLElement): Element | null {
return container.querySelector(classes.map((name) => `.${name}`).join(""));
}
/** The transparent per-bucket hit targets, in bucket order. */
function overlayRects(container: HTMLElement): SVGRectElement[] {
return Array.from(container.querySelectorAll<SVGRectElement>('rect[fill="transparent"]'));
}
test("the data table is the SVG's accessible equivalent", () => {
render(<TimeseriesChart data={timeseries(3)} />);
render(<TimeseriesChart data={counting(3)} />);
expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
const table = screen.getByRole("table", { name: "Queries per time bucket" });
@@ -44,9 +49,339 @@ test("the data table is the SVG's accessible equivalent", () => {
* push the document's scroll height a screen past the app shell.
*/
test("the hidden data table is clipped by a block wrapper, not by the table itself", () => {
const { container } = render(<TimeseriesChart data={timeseries(48)} />);
const { container } = render(<TimeseriesChart data={counting(48)} />);
const hidden = hiddenElement(container);
expect(hidden?.tagName).toBe("DIV");
expect(hidden?.querySelector("table")).not.toBeNull();
});
/**
* "Allowed" is what the reported total leaves over, and the three counts come
* from separate columns that a partial write can leave inconsistent. A negative
* remainder would draw a segment upside down.
*/
test("allowed is the remainder of the reported total, clamped at zero", () => {
const { container } = render(
<TimeseriesChart data={timeseries([{ ts: SINCE, queries: 10, blocked: 8, cached: 5 }])} />,
);
const row = within(screen.getByRole("table")).getAllByRole("row")[1];
expect(
within(row)
.getAllByRole("cell")
.map((cell) => cell.textContent),
).toEqual(["10", "8", "5", "0"]);
// Blocked and cached are drawn; the empty "allowed" segment is not.
expect(container.querySelectorAll('rect[fill="#3b82f6"]')).toHaveLength(0);
// The scale comes from the reported total, not from the stack's own sum.
// Scaling to the sum would reach 13 here and leave the bar four fifths of the
// way up a plot whose own numbers say it is full.
const ticks = Array.from(container.querySelectorAll(".visx-axis-left text")).map((tick) => tick.textContent);
expect(ticks[ticks.length - 1]).toBe("10");
});
test("a window with no queries says so instead of drawing an empty grid", () => {
const { container } = render(
<TimeseriesChart
data={timeseries([
{ ts: SINCE, queries: 0, blocked: 0, cached: 0 },
{ ts: SINCE + 1800, queries: 0, blocked: 0, cached: 0 },
])}
/>,
);
expect(screen.getByText("No queries in this period.")).toBeTruthy();
expect(container.querySelector("svg")).toBeNull();
});
test("no bucket at all says the same thing", () => {
render(<TimeseriesChart data={timeseries([])} />);
expect(screen.getByText("No queries in this period.")).toBeTruthy();
});
/**
* The three category colours are fixed constants of this chart rather than
* anything derived from `seriesColors`, which would paint "other" grey.
*/
test("the segments are drawn in this chart's own category colours", () => {
const { container } = render(
<TimeseriesChart data={timeseries([{ ts: SINCE, queries: 100, blocked: 40, cached: 10 }])} />,
);
const fills = Array.from(container.querySelectorAll("rect"))
.map((rect) => rect.getAttribute("fill"))
.filter((fill) => fill !== "transparent");
expect(fills).toEqual(["#ef4444", "#059669", "#3b82f6"]);
});
/**
* The graphic carries no `<title>`: it duplicated the tooltip, the browser
* showed it after its own delay and out of the app's styling, and the hidden
* table is already the accessible equivalent.
*/
test("hover text is the tooltip alone, never a bare SVG title", () => {
const { container } = render(<TimeseriesChart data={counting(3)} />);
expect(container.querySelectorAll("title")).toHaveLength(0);
});
/** The hit target is the whole column slot, including the space above a short stack. */
test("each bucket's hit target spans the full plot height", () => {
const { container } = render(<TimeseriesChart data={counting(3)} />);
const rects = overlayRects(container);
expect(rects).toHaveLength(3);
for (const rect of rects) {
expect(rect.getAttribute("y")).toBe("8");
expect(rect.getAttribute("height")).toBe("210");
}
});
/**
* A band scale spends a gap after the last column as well as between them, so a
* full-step hit target on the last bucket would reach into the right margin and
* catch pointers that are past the plot entirely.
*/
test("the last hit target stops at the plot's right edge", () => {
const { container } = render(<TimeseriesChart data={counting(3)} />);
const last = overlayRects(container).at(-1) as SVGRectElement;
const right = Number(last.getAttribute("x")) + Number(last.getAttribute("width"));
// 640 fallback width, less the 44px left and 8px right margins.
expect(right).toBeCloseTo(44 + 588, 6);
});
test("pointing at a bucket names its total and every series, and dims the rest", () => {
const { container } = render(
<TimeseriesChart
data={timeseries([
{ ts: SINCE, queries: 100, blocked: 40, cached: 10 },
{ ts: SINCE + 1800, queries: 50, blocked: 5, cached: 5 },
])}
/>,
);
fireEvent.mouseOver(overlayRects(container)[0]);
const tooltip = container.querySelector("dl") as HTMLElement;
expect(tooltip.previousElementSibling?.textContent).toBe(formatTime(SINCE));
const values = Array.from(tooltip.querySelectorAll("dd")).map((dd) => dd.textContent);
expect(values).toEqual(["100", "40", "10", "50"]);
const terms = Array.from(tooltip.querySelectorAll("dt")).map((dt) => dt.textContent);
expect(terms).toEqual(["Queries", "Blocked", "Cached", "Allowed"]);
// One swatch per series, in the colour the segment is drawn in.
const swatches = Array.from(tooltip.querySelectorAll("dt span")).map((span) => span.getAttribute("style"));
expect(swatches[0]).toContain("#ef4444");
expect(swatches[1]).toContain("#059669");
expect(swatches[2]).toContain("#3b82f6");
const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
expect(stacks.map((group) => group.getAttribute("opacity"))).toEqual(["1", "0.55"]);
});
/**
* The tooltip sits 8px down from the chart's top edge and 8px to the side of the
* slot it names — the placement the hand-rolled tooltip had, restored over
* visx's own 10px defaults.
*/
test("the tooltip is offset 8px from the chart top and from the bucket it names", () => {
const { container } = render(<TimeseriesChart data={counting(2)} />);
fireEvent.mouseOver(overlayRects(container)[0]);
// The first slot spans 44 to 44 + step, so its centre is 191.5 and the
// tooltip sits 8px right of it. visx rounds the placement to whole pixels.
const tooltip = container.querySelector(".visx-tooltip") as HTMLElement;
expect(tooltip.style.transform).toBe("translate(200px, 8px)");
});
/**
* `withBoundingRects` measures its node once, on mount. Sharing one mount across
* buckets would place a wide bucket's tooltip with a narrow bucket's measured
* width, which flips or clips it at the right-hand edge of the plot. Remounting
* per bucket is what forces a fresh measurement; jsdom reports every rect as
* zero, so the mount is what this can observe, not the measurement itself.
*/
test("each bucket gets its own tooltip mount, so each is measured for itself", () => {
const { container } = render(<TimeseriesChart data={counting(2)} />);
const rects = overlayRects(container);
fireEvent.mouseOver(rects[0]);
const first = container.querySelector(".visx-tooltip");
fireEvent.mouseOver(rects[1]);
const second = container.querySelector(".visx-tooltip");
expect(first).not.toBeNull();
expect(second).not.toBe(first);
});
/**
* The window refreshes every half minute under an open tooltip. The tooltip
* holds a bucket index and reads the numbers out of the render it is drawing, so
* a refresh that keeps the same buckets updates it rather than leaving last
* minute's counts on screen.
*/
test("a refresh in the same window retells the hovered bucket with the new counts", () => {
const before = timeseries([
{ ts: SINCE, queries: 100, blocked: 40, cached: 10 },
{ ts: SINCE + 1800, queries: 50, blocked: 5, cached: 5 },
]);
const { container, rerender } = render(<TimeseriesChart data={before} />);
fireEvent.mouseOver(overlayRects(container)[0]);
expect(
Array.from((container.querySelector("dl") as HTMLElement).querySelectorAll("dd")).map((dd) => dd.textContent),
).toEqual(["100", "40", "10", "50"]);
rerender(
<TimeseriesChart
data={timeseries([
{ ts: SINCE, queries: 120, blocked: 60, cached: 10 },
{ ts: SINCE + 1800, queries: 50, blocked: 5, cached: 5 },
])}
/>,
);
const values = Array.from((container.querySelector("dl") as HTMLElement).querySelectorAll("dd")).map(
(dd) => dd.textContent,
);
expect(values).toEqual(["120", "60", "10", "50"]);
});
/**
* A rolling window is the case a stored copy gets wrong: the bucket the pointer
* was over is gone, so index 0 now names a different span. The tooltip goes away
* rather than describing a bucket that is no longer drawn, and nothing stays
* dimmed behind it. Rolling back to the earlier window must not bring it back
* either: the selection is deleted when the window moves, not held aside.
*/
test("a refresh that rolls the window takes the tooltip down instead of relabelling it", () => {
const { container, rerender } = render(
<TimeseriesChart
data={timeseries([
{ ts: SINCE, queries: 100, blocked: 40, cached: 10 },
{ ts: SINCE + 1800, queries: 50, blocked: 5, cached: 5 },
])}
/>,
);
fireEvent.mouseOver(overlayRects(container)[0]);
expect(container.querySelectorAll("dl")).toHaveLength(1);
rerender(
<TimeseriesChart
data={{
...timeseries([
{ ts: SINCE + 1800, queries: 50, blocked: 5, cached: 5 },
{ ts: SINCE + 3600, queries: 70, blocked: 7, cached: 7 },
]),
since: SINCE + 1800,
}}
/>,
);
expect(container.querySelectorAll("dl")).toHaveLength(0);
const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
expect(stacks.every((group) => group.getAttribute("opacity") === "1")).toBe(true);
rerender(
<TimeseriesChart
data={timeseries([
{ ts: SINCE, queries: 100, blocked: 40, cached: 10 },
{ ts: SINCE + 1800, queries: 50, blocked: 5, cached: 5 },
])}
/>,
);
expect(container.querySelectorAll("dl")).toHaveLength(0);
});
/**
* The same bucket can change width between refreshes — a count crossing a digit
* boundary, a client name arriving — and a mount measured at the old width is
* placed at the wrong one. A content change therefore remounts the tooltip, the
* same way moving between buckets does.
*/
test("a bucket whose numbers change is remounted, so it is measured again", () => {
const { container, rerender } = render(
<TimeseriesChart data={timeseries([{ ts: SINCE, queries: 9, blocked: 4, cached: 1 }])} />,
);
fireEvent.mouseOver(overlayRects(container)[0]);
const first = container.querySelector(".visx-tooltip");
rerender(<TimeseriesChart data={timeseries([{ ts: SINCE, queries: 1000, blocked: 4, cached: 1 }])} />);
const second = container.querySelector(".visx-tooltip");
expect(first).not.toBeNull();
expect(second).not.toBeNull();
expect(second).not.toBe(first);
});
test("leaving the chart takes the tooltip and the dimming with it", () => {
const { container } = render(<TimeseriesChart data={counting(3)} />);
fireEvent.mouseOver(overlayRects(container)[0]);
expect(container.querySelectorAll("dl")).toHaveLength(1);
fireEvent.mouseOut(container.querySelector("svg") as SVGSVGElement);
expect(container.querySelectorAll("dl")).toHaveLength(0);
const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
expect(stacks.every((group) => group.getAttribute("opacity") === "1")).toBe(true);
});
/**
* The third series is every query neither blocked nor served from cache. It was
* called "Other", which named the arithmetic rather than the thing.
*/
test("the remainder series is called Allowed everywhere it surfaces", () => {
const { container } = render(
<TimeseriesChart data={timeseries([{ ts: SINCE, queries: 10, blocked: 2, cached: 3 }])} />,
);
const legend = Array.from(container.querySelectorAll("ul li")).map((item) => item.textContent);
expect(legend).toEqual(["Blocked", "Cached", "Allowed"]);
expect(
within(screen.getByRole("table"))
.getAllByRole("columnheader")
.map((cell) => cell.textContent),
).toEqual(["Time", "Queries", "Blocked", "Cached", "Allowed"]);
fireEvent.mouseOver(overlayRects(container)[0]);
const terms = Array.from((container.querySelector("dl") as HTMLElement).querySelectorAll("dt"));
expect(terms.map((term) => term.textContent)).toEqual(["Queries", "Blocked", "Cached", "Allowed"]);
expect(screen.queryByText("Other")).toBeNull();
});
/**
* A window where everything was blocked or served from cache has no allowed
* queries, and that is worth reading rather than hiding: an absent series would
* say the same thing as a series nobody looked at. The three categories are all
* real answers a query can get, so none of them is dropped for counting zero.
* The client chart's "Other" is dropped at zero, but that one aggregates clients
* beyond the top eight rather than naming a kind of answer.
*/
test("a window with nothing allowed keeps the series at zero", () => {
const { container } = render(
<TimeseriesChart
data={timeseries([
{ ts: SINCE, queries: 4, blocked: 4, cached: 0 },
{ ts: SINCE + 1800, queries: 6, blocked: 6, cached: 0 },
])}
/>,
);
const legend = Array.from(container.querySelectorAll("ul li")).map((item) => item.textContent);
expect(legend).toEqual(["Blocked", "Cached", "Allowed"]);
fireEvent.mouseOver(overlayRects(container)[0]);
const tooltip = container.querySelector("dl") as HTMLElement;
expect(Array.from(tooltip.querySelectorAll("dt")).map((term) => term.textContent)).toEqual([
"Queries",
"Blocked",
"Cached",
"Allowed",
]);
expect(Array.from(tooltip.querySelectorAll("dd")).map((value) => value.textContent)).toEqual(["4", "4", "0", "0"]);
});
+148 -240
View File
@@ -1,107 +1,49 @@
import { useEffect, useRef, useState } from "react";
import * as stylex from "@stylexjs/stylex";
import { Group } from "@visx/group";
import { BarStack } from "@visx/shape";
import { formatTime } from "@/lib/format";
import type { StatsTimeseries } from "@/lib/types";
import type { Bucket } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { isEmptyTimeseries, layoutTimeseries, type BarLayout } from "./chartLayout";
import {
BucketOverlay,
CHART_HEIGHT,
ChartFrame,
ChartRoot,
ChartTooltip,
EmptyChart,
StackSegment,
bandScale,
labelTickValues,
plotArea,
slotCenter,
useActiveIndex,
useMeasuredWidth,
valueScale,
valueTicks,
type TooltipContent,
} from "./chartKit";
// Series colors validated for CVD separation and 3:1 surface contrast in both
// modes (Tailwind red-500 / blue-500 / emerald-600; same hex light and dark).
// The third key stays `other` — it is the colour key and the response field, and
// renaming it would repaint the series. Only what the reader sees is "Allowed".
const SERIES = [
{ key: "blocked", label: "Blocked", color: "#ef4444" },
{ key: "cached", label: "Cached", color: "#059669" },
{ key: "other", label: "Other", color: "#3b82f6" },
{ key: "other", label: "Allowed", color: "#3b82f6" },
] as const;
const CHART_HEIGHT = 240;
const FALLBACK_WIDTH = 640;
type Series = (typeof SERIES)[number];
type SeriesKey = Series["key"];
const SERIES_COLOR: Record<SeriesKey, string> = {
blocked: SERIES[0].color,
cached: SERIES[1].color,
other: SERIES[2].color,
};
const styles = stylex.create({
empty: {
display: "flex",
alignItems: "center",
justifyContent: "center",
height: CHART_HEIGHT,
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "dashed",
borderColor: colors.borderStrong,
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
chartRoot: {
position: "relative",
},
tooltip: {
pointerEvents: "none",
position: "absolute",
top: "0.5rem",
zIndex: 10,
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.borderStrong,
backgroundColor: colors.surfaceRaised,
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
fontSize: "0.75rem",
lineHeight: "1rem",
boxShadow: "0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1)",
},
/** Dynamic: the tooltip flips to whichever side of the bar has room. */
tooltipLeft: (left: number) => ({ left, right: null }),
tooltipRight: (right: number) => ({ left: null, right }),
tooltipTitle: {
fontWeight: 500,
},
tooltipList: {
display: "flex",
flexDirection: "column",
gap: "0.125rem",
marginTop: "0.25rem",
},
tooltipRow: {
display: "flex",
alignItems: "center",
justifyContent: "space-between",
gap: "1rem",
},
tooltipTerm: {
display: "flex",
alignItems: "center",
gap: "0.375rem",
color: colors.textMuted,
},
swatch: {
display: "inline-block",
borderRadius: "0.125rem",
},
/** Dynamic: the swatch takes the series colour the SVG bars are drawn in. */
swatchColor: (color: string) => ({ backgroundColor: color }),
swatchSmall: {
width: "0.5rem",
height: "0.5rem",
},
swatchLarge: {
width: "0.625rem",
height: "0.625rem",
},
gridLine: {
stroke: colors.border,
},
axisLine: {
stroke: colors.borderStrong,
},
axisLabel: {
fill: colors.textMuted,
fontSize: "10px",
},
/** The hairline separating touching segments is the page ground, not a colour. */
segment: {
stroke: colors.surface,
},
legend: {
marginTop: "0.5rem",
display: "flex",
@@ -117,177 +59,141 @@ const styles = stylex.create({
alignItems: "center",
gap: "0.375rem",
},
swatch: {
display: "inline-block",
width: "0.625rem",
height: "0.625rem",
borderRadius: "0.125rem",
},
/** Dynamic: the swatch takes the series colour the SVG bars are drawn in. */
swatchColor: (color: string) => ({ backgroundColor: color }),
});
function useContainerWidth(): [React.RefObject<HTMLDivElement | null>, number] {
const ref = useRef<HTMLDivElement>(null);
const [width, setWidth] = useState(0);
useEffect(() => {
const el = ref.current;
if (el === null) return;
setWidth(el.clientWidth);
if (typeof ResizeObserver === "undefined") return;
const observer = new ResizeObserver(() => setWidth(el.clientWidth));
observer.observe(el);
return () => observer.disconnect();
}, []);
return [ref, width];
interface Column {
ts: number;
queries: number;
blocked: number;
cached: number;
/** queries - blocked - cached, clamped at 0. */
other: number;
}
const compact = new Intl.NumberFormat(undefined, { notation: "compact" });
function formatTick(ts: number, bucketSeconds: number): string {
const date = new Date(ts * 1000);
if (bucketSeconds >= 86_400) {
return new Intl.DateTimeFormat(undefined, { month: "short", day: "numeric" }).format(date);
}
return new Intl.DateTimeFormat(undefined, { hour: "numeric", minute: "2-digit" }).format(date);
function columnsOf(buckets: Bucket[]): Column[] {
return buckets.map((bucket) => ({
ts: bucket.ts,
queries: bucket.queries,
blocked: bucket.blocked,
cached: bucket.cached,
other: Math.max(0, bucket.queries - bucket.blocked - bucket.cached),
}));
}
function barSummary(bar: BarLayout): string {
return `${formatTime(bar.bucket.ts)}: ${bar.bucket.queries} queries, ${bar.bucket.blocked} blocked, ${bar.bucket.cached} cached`;
function tooltipOf(column: Column): TooltipContent {
return {
title: formatTime(column.ts),
rows: [
{ key: "queries", label: "Queries", value: String(column.queries) },
...SERIES.map((series) => ({
key: series.key,
label: series.label,
color: series.color,
value: String(column[series.key]),
})),
],
};
}
function Tooltip({ bar, chartWidth }: { bar: BarLayout; chartWidth: number }) {
const centerX = bar.slot.x + bar.slot.width / 2;
const leftHalf = centerX < chartWidth / 2;
const side = leftHalf
? styles.tooltipLeft(Math.min(centerX + 8, chartWidth - 160))
: styles.tooltipRight(chartWidth - centerX + 8);
return (
<div {...stylex.props(styles.tooltip, side)}>
<div {...stylex.props(styles.tooltipTitle)}>{formatTime(bar.bucket.ts)}</div>
<dl {...stylex.props(styles.tooltipList)}>
<div {...stylex.props(styles.tooltipRow)}>
<dt {...stylex.props(styles.tooltipTerm)}>Queries</dt>
<dd {...stylex.props(shared.tabularNums)}>{bar.bucket.queries}</dd>
</div>
{SERIES.map((series) => (
<div key={series.key} {...stylex.props(styles.tooltipRow)}>
<dt {...stylex.props(styles.tooltipTerm)}>
<span
aria-hidden="true"
{...stylex.props(styles.swatch, styles.swatchSmall, styles.swatchColor(series.color))}
/>
{series.label}
</dt>
<dd {...stylex.props(shared.tabularNums)}>
{series.key === "other" ? bar.other : bar.bucket[series.key]}
</dd>
</div>
))}
</dl>
</div>
);
/**
* The slice of the Overview body this chart draws. Declared here rather than
* taken whole, so what the chart reads is stated where it is read.
*/
export interface TimeseriesData {
since: number;
bucket_seconds: number;
buckets: Bucket[];
}
export default function TimeseriesChart({ data }: { data: StatsTimeseries }) {
const [containerRef, measuredWidth] = useContainerWidth();
const [hovered, setHovered] = useState<number | null>(null);
const width = measuredWidth > 0 ? measuredWidth : FALLBACK_WIDTH;
export default function TimeseriesChart({ data }: { data: TimeseriesData }) {
const [containerRef, width] = useMeasuredWidth();
// A hover survives a re-render only while it still names the same bucket at
// the same place: a poll that rolls the window, or a resize, retires it.
const hovered = useActiveIndex(`${data.since}:${data.bucket_seconds}:${data.buckets.length}:${width}`);
if (data.buckets.length === 0 || isEmptyTimeseries(data.buckets)) {
return (
<div ref={containerRef} {...stylex.props(styles.empty)}>
No queries in this period.
</div>
);
if (data.buckets.length === 0 || data.buckets.every((bucket) => bucket.queries === 0)) {
return <EmptyChart containerRef={containerRef} />;
}
const layout = layoutTimeseries(data.buckets, width, CHART_HEIGHT);
const baseline = layout.plot.y + layout.plot.height;
const hoveredBar = hovered !== null ? layout.bars[hovered] : undefined;
const columns = columnsOf(data.buckets);
const timestamps = columns.map((column) => column.ts);
const plot = plotArea(width);
const xScale = bandScale(timestamps, plot);
// The scale is the reported total rather than the stack's own sum: blocked
// and cached are parts of `queries`, which a clamped `other` can undercount.
const yScale = valueScale(Math.max(...columns.map((column) => column.queries)), [plot.bottom, plot.y]);
const yTicks = valueTicks(yScale);
return (
<div ref={containerRef} {...stylex.props(styles.chartRoot)}>
<ChartRoot containerRef={containerRef}>
<svg
role="img"
aria-label={`Queries over time, ${data.buckets.length} buckets: blocked, cached and other queries per bucket`}
aria-label={`Queries over time, ${data.buckets.length} buckets: ${SERIES.map((series) => series.label.toLowerCase()).join(", ")} queries per bucket`}
width="100%"
height={CHART_HEIGHT}
viewBox={`0 0 ${width} ${CHART_HEIGHT}`}
onMouseLeave={() => setHovered(null)}
onMouseLeave={hovered.clear}
>
{layout.yTicks.map((tick) => (
<g key={tick.value}>
<line
x1={layout.plot.x}
x2={layout.plot.x + layout.plot.width}
y1={tick.y}
y2={tick.y}
{...stylex.props(styles.gridLine)}
<ChartFrame
plot={plot}
yScale={yScale}
yTicks={yTicks}
xScale={xScale}
xTickValues={labelTickValues(timestamps, plot.width)}
bucketSeconds={data.bucket_seconds}
/>
<text
x={layout.plot.x - 6}
y={tick.y}
textAnchor="end"
dominantBaseline="middle"
{...stylex.props(styles.axisLabel, shared.tabularNums)}
<BarStack<Column, SeriesKey>
data={columns}
keys={SERIES.map((series) => series.key)}
x={(column) => column.ts}
xScale={xScale}
yScale={yScale}
color={(key) => SERIES_COLOR[key]}
>
{compact.format(tick.value)}
</text>
</g>
))}
<line
x1={layout.plot.x}
x2={layout.plot.x + layout.plot.width}
y1={baseline}
y2={baseline}
{...stylex.props(styles.axisLine)}
/>
{layout.xTicks.map((tick) => (
<text
key={tick.ts}
x={tick.x}
y={baseline + 14}
textAnchor="middle"
{...stylex.props(styles.axisLabel)}
{(stacks) =>
columns.map((column, index) => (
<Group
key={column.ts}
opacity={hovered.index === null || hovered.index === index ? 1 : 0.55}
>
{formatTick(tick.ts, data.bucket_seconds)}
</text>
))}
{layout.bars.map((bar, i) => (
<g key={bar.bucket.ts} opacity={hovered === null || hovered === i ? 1 : 0.55}>
{SERIES.map((series) => {
const rect = bar.segments[series.key];
if (rect.height <= 0) return null;
{stacks.map((stack) => {
const bar = stack.bars[index];
return (
<rect
key={series.key}
x={rect.x}
y={rect.y}
width={rect.width}
height={rect.height}
fill={series.color}
strokeWidth={rect.width > 3 ? 1 : 0}
{...stylex.props(styles.segment)}
<StackSegment
key={stack.key}
x={bar.x}
y={bar.y}
width={bar.width}
height={bar.height}
fill={bar.color}
/>
);
})}
</g>
))}
{layout.bars.map((bar, i) => (
<rect
key={bar.bucket.ts}
x={bar.slot.x}
y={bar.slot.y}
width={bar.slot.width}
height={bar.slot.height}
fill="transparent"
onMouseEnter={() => setHovered(i)}
>
<title>{barSummary(bar)}</title>
</rect>
))}
</Group>
))
}
</BarStack>
<BucketOverlay plot={plot} values={timestamps} xScale={xScale} onEnter={hovered.show} />
</svg>
{hoveredBar !== undefined && <Tooltip bar={hoveredBar} chartWidth={width} />}
{hovered.index !== null && (
<ChartTooltip
index={hovered.index}
content={tooltipOf(columns[hovered.index])}
left={slotCenter(xScale, timestamps[hovered.index], plot)}
/>
)}
<ul {...stylex.props(styles.legend)}>
{SERIES.map((series) => (
<li key={series.key} {...stylex.props(styles.legendItem)}>
<span
aria-hidden="true"
{...stylex.props(styles.swatch, styles.swatchLarge, styles.swatchColor(series.color))}
/>
<span aria-hidden="true" {...stylex.props(styles.swatch, styles.swatchColor(series.color))} />
{series.label}
</li>
))}
@@ -299,24 +205,26 @@ export default function TimeseriesChart({ data }: { data: StatsTimeseries }) {
<tr>
<th scope="col">Time</th>
<th scope="col">Queries</th>
<th scope="col">Blocked</th>
<th scope="col">Cached</th>
<th scope="col">Other</th>
{SERIES.map((series) => (
<th key={series.key} scope="col">
{series.label}
</th>
))}
</tr>
</thead>
<tbody>
{layout.bars.map((bar) => (
<tr key={bar.bucket.ts}>
<th scope="row">{formatTime(bar.bucket.ts)}</th>
<td>{bar.bucket.queries}</td>
<td>{bar.bucket.blocked}</td>
<td>{bar.bucket.cached}</td>
<td>{bar.other}</td>
{columns.map((column) => (
<tr key={column.ts}>
<th scope="row">{formatTime(column.ts)}</th>
<td>{column.queries}</td>
{SERIES.map((series) => (
<td key={series.key}>{column[series.key]}</td>
))}
</tr>
))}
</tbody>
</table>
</div>
</div>
</ChartRoot>
);
}
@@ -0,0 +1,149 @@
import { render } from "@testing-library/react";
import {
CHART_HEIGHT,
ChartFrame,
bandPaddingInner,
bandScale,
labelTickValues,
plotArea,
useMeasuredWidth,
valueScale,
valueTicks,
} from "./chartKit";
describe("useMeasuredWidth", () => {
/**
* jsdom lays nothing out, so `clientWidth` is 0 there and a chart scaled to it
* would draw a zero-width plot. Every chart test depends on this fallback.
*/
test("an unmeasurable container falls back to 640", () => {
function Probe() {
const [ref, width] = useMeasuredWidth();
return <div ref={ref} data-testid="probe" data-width={width} />;
}
const { getByTestId } = render(<Probe />);
expect(getByTestId("probe").getAttribute("data-width")).toBe("640");
});
});
describe("valueScale", () => {
test("the data maximum is nicened outward and the ticks land on round numbers", () => {
const scale = valueScale(1780, [240, 0]);
expect(scale.domain()).toEqual([0, 1800]);
expect(scale.ticks(5)).toEqual([0, 500, 1000, 1500]);
expect(valueTicks(scale)).toEqual([0, 500, 1000, 1500]);
});
});
describe("bandPaddingInner", () => {
/**
* The gap is a constant number of pixels, so the fraction it takes of one slot
* has to fall as the slots get wider. 24 hourly columns in a 1388px plot is
* the 24h window on a full-width panel.
*/
test("the fraction is two pixels per column of the plot's width", () => {
expect(bandPaddingInner(24, 1388)).toBeCloseTo((2 * 24) / 1388, 12);
expect(bandPaddingInner(24, 1388)).toBeCloseTo(0.034582, 6);
});
/**
* Past the cap the gap would be most of the slot and the columns would vanish
* into slivers, so it stops at half the slot and the bars stay visible.
*/
test("the gap never takes more than half a slot", () => {
expect(bandPaddingInner(1000, 100)).toBe(0.5);
});
test("no columns and no width mean no gap to compute", () => {
expect(bandPaddingInner(0, 640)).toBe(0);
expect(bandPaddingInner(24, 0)).toBe(0);
});
});
describe("labelTickValues", () => {
/** A week of hourly buckets in a panel the width of a laptop window. */
test("labels thin out to one every 21 buckets at 748px of plot", () => {
const timestamps = Array.from({ length: 168 }, (_, i) => i * 3600);
expect(labelTickValues(timestamps, 748)).toEqual([0, 75600, 151200, 226800, 302400, 378000, 453600, 529200]);
});
test("every bucket is labelled when they all fit", () => {
expect(labelTickValues([0, 3600, 7200], 748)).toEqual([0, 3600, 7200]);
});
});
/** The frame on its own, at the width the charts draw at in jsdom. */
function renderFrame() {
const plot = plotArea(640);
const timestamps = [0, 3600, 7200];
const yScale = valueScale(100, [plot.bottom, plot.y]);
const yTicks = valueTicks(yScale);
const { container } = render(
<svg width={640} height={CHART_HEIGHT}>
<ChartFrame
plot={plot}
yScale={yScale}
yTicks={yTicks}
xScale={bandScale(timestamps, plot)}
xTickValues={timestamps}
bucketSeconds={3600}
/>
</svg>,
);
return { container, plot, yTicks };
}
describe("ChartFrame", () => {
test("the value axis is bare: no axis line, no tick marks, labels 6px left of the plot", () => {
const { container, plot, yTicks } = renderFrame();
const axis = container.querySelector(".visx-axis-left") as SVGGElement;
expect(axis.getAttribute("transform")).toBe(`translate(${plot.x}, 0)`);
expect(axis.querySelector("line")).toBeNull();
const labels = Array.from(axis.querySelectorAll("text"));
expect(labels.map((label) => label.textContent)).toEqual(yTicks.map(String));
for (const label of labels) {
expect(label.getAttribute("x")).toBe("0");
expect(label.getAttribute("dx")).toBe("-6px");
expect(label.getAttribute("text-anchor")).toBe("end");
// The label sits on the tick, centred: visx's own 0.25em nudge is
// cancelled so it does not double up with the middle baseline.
expect(label.getAttribute("dy")).toBe("0");
expect(label.getAttribute("dominant-baseline")).toBe("middle");
}
});
/**
* The time axis keeps its baseline and drops its tick marks, and the labels
* are placed by the group transform plus `dy` rather than by the font-size
* guess visx would otherwise make.
*/
test("the time axis keeps only its baseline, and its labels sit 16px below it", () => {
const { container, plot } = renderFrame();
const axis = container.querySelector(".visx-axis-bottom") as SVGGElement;
expect(axis.getAttribute("transform")).toBe(`translate(0, ${plot.bottom})`);
const lines = Array.from(axis.querySelectorAll("line"));
expect(lines).toHaveLength(1);
expect(lines[0].getAttribute("class")).toContain("visx-axis-line");
for (const label of Array.from(axis.querySelectorAll("text"))) {
expect(label.getAttribute("y")).toBe("0");
expect(label.getAttribute("dy")).toBe("16px");
expect(label.getAttribute("text-anchor")).toBe("middle");
}
});
test("every grid line has a labelled tick on it and no tick floats without a line", () => {
const { container } = renderFrame();
const gridY = Array.from(container.querySelectorAll(".visx-rows line")).map((line) => line.getAttribute("y1"));
const labelY = Array.from(container.querySelectorAll(".visx-axis-left text")).map((label) =>
label.getAttribute("y"),
);
expect(gridY.length).toBeGreaterThan(0);
expect(gridY).toEqual(labelY);
});
});
+498
View File
@@ -0,0 +1,498 @@
/**
* The plumbing the two bar charts on Overview share: the width measurement, the
* scales, the axis and grid chrome, the segment separator, the per-bucket hit
* target and the tooltip.
*
* Geometry is visx's; the chrome is ours. visx's own axis defaults draw tick
* marks, an axis line on both axes and Arial 10px in #222, none of which this
* app wants, so every wrapper here turns those off and dresses the result in
* StyleX tokens instead.
*/
import { useEffect, useRef, useState } from "react";
import * as stylex from "@stylexjs/stylex";
import { AxisBottom, AxisLeft, type TickRendererProps } from "@visx/axis";
import { GridRows } from "@visx/grid";
import { scaleBand, scaleLinear } from "@visx/scale";
import { TooltipWithBounds } from "@visx/tooltip";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
export const CHART_HEIGHT = 240;
export const MARGIN = { top: 8, right: 8, bottom: 22, left: 44 } as const;
/** The width a chart draws at before a measurement exists — jsdom, and the first paint. */
const FALLBACK_WIDTH = 640;
/** How many pixels one x-axis label needs to itself before the labels thin out. */
const MIN_X_LABEL_PX = 90;
/** The gap between two touching columns, in pixels of the drawn plot. */
const BAR_GAP_PX = 2;
const Y_TICK_COUNT = 5;
export interface Plot {
x: number;
y: number;
width: number;
height: number;
/** The y of the baseline: the bottom edge of the plot. */
bottom: number;
}
export function plotArea(width: number): Plot {
const height = Math.max(0, CHART_HEIGHT - MARGIN.top - MARGIN.bottom);
return {
x: MARGIN.left,
y: MARGIN.top,
width: Math.max(0, width - MARGIN.left - MARGIN.right),
height,
bottom: MARGIN.top + height,
};
}
/**
* The container's width, measured on mount and on every resize. Deliberately
* `useEffect` rather than `useLayoutEffect`: the first paint draws at the
* fallback width and the measured width lands a frame later, which is the
* timing the charts have always had.
*/
export function useMeasuredWidth(): [React.RefObject<HTMLDivElement | null>, number] {
const ref = useRef<HTMLDivElement>(null);
const [width, setWidth] = useState(0);
useEffect(() => {
const el = ref.current;
if (el === null) return;
setWidth(el.clientWidth);
if (typeof ResizeObserver === "undefined") return;
const observer = new ResizeObserver(() => setWidth(el.clientWidth));
observer.observe(el);
return () => observer.disconnect();
}, []);
return [ref, width > 0 ? width : FALLBACK_WIDTH];
}
/**
* The value axis. `nice: true` rounds the domain outward, so the `max` passed
* here is the data's own maximum and not a pre-rounded one.
*/
export function valueScale(max: number, range: [number, number]) {
return scaleLinear<number>({ domain: [0, max], range, nice: true });
}
export type ValueScale = ReturnType<typeof valueScale>;
/** The tick values the grid and the value axis both draw. */
export function valueTicks(scale: ValueScale): number[] {
return scale.ticks(Y_TICK_COUNT);
}
/**
* How wide the gap between two columns is, as a fraction of one column's slot.
* A constant would make the gap grow with the column: 30 daily columns and 60
* minute columns are drawn at the same pixel gap, so the fraction has to be
* computed from how many columns share the plot.
*/
export function bandPaddingInner(bucketCount: number, plotWidth: number): number {
if (bucketCount === 0 || plotWidth <= 0) return 0;
return Math.min(0.5, (BAR_GAP_PX * bucketCount) / plotWidth);
}
/** The time axis: one band per bucket, in the order the buckets were given. */
export function bandScale(values: number[], plot: Plot) {
return scaleBand<number>({
domain: values,
range: [plot.x, plot.x + plot.width],
paddingInner: bandPaddingInner(values.length, plot.width),
});
}
export type BandScale = ReturnType<typeof bandScale>;
/**
* The subset of bucket timestamps that get an x-axis label. A 30-day window is
* 30 columns and a 1-hour window is 60, so at narrow widths the labels have to
* thin out rather than overprint each other.
*/
export function labelTickValues(values: number[], plotWidth: number): number[] {
if (values.length === 0 || plotWidth <= 0) return values.slice(0, 1);
const step = Math.max(1, Math.ceil((values.length * MIN_X_LABEL_PX) / plotWidth));
return values.filter((_, i) => i % step === 0);
}
const compact = new Intl.NumberFormat(undefined, { notation: "compact" });
export function formatBucketTime(ts: number, bucketSeconds: number): string {
const date = new Date(ts * 1000);
if (bucketSeconds >= 86_400) {
return new Intl.DateTimeFormat(undefined, { month: "short", day: "numeric" }).format(date);
}
return new Intl.DateTimeFormat(undefined, { hour: "numeric", minute: "2-digit" }).format(date);
}
const styles = stylex.create({
empty: {
display: "flex",
alignItems: "center",
justifyContent: "center",
height: CHART_HEIGHT,
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "dashed",
borderColor: colors.borderStrong,
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
root: {
position: "relative",
},
axisLabel: {
fill: colors.textMuted,
fontSize: "10px",
},
/** The hairline separating touching segments is the page ground, not a colour. */
segment: {
stroke: colors.surface,
},
tooltip: {
pointerEvents: "none",
position: "absolute",
zIndex: 10,
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.borderStrong,
backgroundColor: colors.surfaceRaised,
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
fontSize: "0.75rem",
lineHeight: "1rem",
boxShadow: "0 1px 3px 0 rgb(0 0 0 / 0.1), 0 1px 2px -1px rgb(0 0 0 / 0.1)",
},
tooltipTitle: {
fontWeight: 500,
},
tooltipList: {
display: "flex",
flexDirection: "column",
gap: "0.125rem",
marginTop: "0.25rem",
},
tooltipRow: {
display: "flex",
alignItems: "center",
justifyContent: "space-between",
gap: "1rem",
},
tooltipTerm: {
display: "flex",
alignItems: "center",
gap: "0.375rem",
color: colors.textMuted,
},
swatch: {
display: "inline-block",
width: "0.5rem",
height: "0.5rem",
borderRadius: "0.125rem",
},
/** Dynamic: the swatch takes the series colour the SVG bars are drawn in. */
swatchColor: (color: string) => ({ backgroundColor: color }),
});
/** Both charts' "nothing here" state, which is also where the ref to measure lives. */
export function EmptyChart({ containerRef }: { containerRef: React.RefObject<HTMLDivElement | null> }) {
return (
<div ref={containerRef} {...stylex.props(styles.empty)}>
No queries in this period.
</div>
);
}
export function ChartRoot({
containerRef,
children,
}: {
containerRef: React.RefObject<HTMLDivElement | null>;
children: React.ReactNode;
}) {
return (
<div ref={containerRef} {...stylex.props(styles.root)}>
{children}
</div>
);
}
/**
* visx's `Ticks` derives a label's `y` from a guessed font size, which is wrong
* here because the size comes from a stylesheet it cannot read. The label is
* therefore placed by the axis group's own transform plus the `dy` the caller
* passes, and the guessed `y` is dropped.
*/
function TickLabel({ x, dx, dy, textAnchor, dominantBaseline, formattedValue }: TickRendererProps) {
return (
<text
x={x}
y={0}
dx={dx}
dy={dy}
textAnchor={textAnchor}
dominantBaseline={dominantBaseline}
{...stylex.props(styles.axisLabel)}
>
{formattedValue}
</text>
);
}
/** As `TickLabel`, but keeping the scaled tick position the vertical axis puts in `y`. */
function ValueTickLabel({ x, y, dx, dy, textAnchor, dominantBaseline, formattedValue }: TickRendererProps) {
return (
<text
x={x}
y={y}
dx={dx}
dy={dy}
textAnchor={textAnchor}
dominantBaseline={dominantBaseline}
{...stylex.props(styles.axisLabel, shared.tabularNums)}
>
{formattedValue}
</text>
);
}
/**
* The grid and the two axes. The grid lines and the value labels are computed
* from one tick array, so every drawn line has a number beside it and no number
* floats without one.
*/
export function ChartFrame({
plot,
yScale,
yTicks,
xScale,
xTickValues,
bucketSeconds,
}: {
plot: Plot;
yScale: ValueScale;
yTicks: number[];
xScale: BandScale;
xTickValues: number[];
bucketSeconds: number;
}) {
return (
<>
<GridRows scale={yScale} tickValues={yTicks} left={plot.x} width={plot.width} stroke={colors.border} />
<AxisLeft
scale={yScale}
left={plot.x}
tickValues={yTicks}
numTicks={Y_TICK_COUNT}
hideAxisLine
hideTicks
tickLength={0}
tickFormat={(value) => compact.format(Number(value))}
// `dy` overrides AxisLeft's own 0.25em nudge, which would double up
// with the middle baseline this app centres its value labels on.
tickLabelProps={{ dx: "-6px", dy: 0, textAnchor: "end", dominantBaseline: "middle" }}
tickComponent={ValueTickLabel}
/>
<AxisBottom
scale={xScale}
top={plot.bottom}
tickValues={xTickValues}
hideTicks
tickLength={0}
stroke={colors.borderStrong}
tickFormat={(value) => formatBucketTime(Number(value), bucketSeconds)}
tickLabelProps={{ dy: "16px", textAnchor: "middle" }}
tickComponent={TickLabel}
/>
</>
);
}
/**
* One segment of a stacked column. The separator is drawn only once the column
* is wide enough for two neighbouring segments to read as two shapes; below
* that it would be most of the bar.
*/
export function StackSegment({
x,
y,
width,
height,
fill,
}: {
x: number;
y: number;
width: number;
height: number;
fill: string;
}) {
if (height <= 0) return null;
return (
<rect
x={x}
y={y}
width={width}
height={height}
fill={fill}
strokeWidth={width > 3 ? 1 : 0}
{...stylex.props(styles.segment)}
/>
);
}
/**
* The transparent hit targets: one per bucket, spanning the whole plot height so
* that pointing at the empty space above a short column still selects it.
*
* A slot is a whole band step, gap included, so that no pixel between two
* columns belongs to neither. The last slot is clipped to the plot's right edge:
* the band scale spends the trailing gap on nothing, and a full-step rect there
* would reach into the right margin.
*/
export function BucketOverlay({
plot,
values,
xScale,
onEnter,
}: {
plot: Plot;
values: number[];
xScale: BandScale;
onEnter: (index: number) => void;
}) {
const step = xScale.step();
const right = plot.x + plot.width;
return (
<>
{values.map((value, index) => {
const x = xScale(value) ?? plot.x;
return (
<rect
key={value}
x={x}
y={plot.y}
width={Math.max(0, Math.min(step, right - x))}
height={plot.height}
fill="transparent"
onMouseEnter={() => onEnter(index)}
/>
);
})}
</>
);
}
/** The x a bucket's tooltip points at: the centre of its slot, not of its narrower bar. */
export function slotCenter(xScale: BandScale, value: number, plot: Plot): number {
return (xScale(value) ?? plot.x) + xScale.step() / 2;
}
/**
* Which item the pointer is on, and nothing else — a bucket in the bar charts, a
* slice in the donuts.
*
* Deliberately not `useTooltip`: holding the hovered item's numbers and screen
* position in state decouples them from the data. The window refreshes every
* half minute and the container can be resized under an open tooltip, and either
* one leaves a stored copy describing an item that has moved or changed. Only
* the index is kept, so the caller reads the values and the coordinates out of
* the render it is currently drawing.
*
* `windowKey` is what makes a changing set safe: when the items themselves
* change, index `i` no longer names the one the pointer was over, so the
* selection made against the old key is dropped and the tooltip goes away.
*/
export function useActiveIndex(windowKey: string) {
const [selection, setSelection] = useState<{ index: number; key: string } | null>(null);
// A selection belongs to the window it was made in. Deleting it as soon as the
// key moves on is what stops a returning key — a resize back to a previous
// width, a poll that restores a bucket count — from reviving a hover the
// pointer never made.
if (selection !== null && selection.key !== windowKey) setSelection(null);
return {
index: selection !== null && selection.key === windowKey ? selection.index : null,
show: (index: number) => setSelection({ index, key: windowKey }),
clear: () => setSelection(null),
};
}
export interface TooltipRow {
key: string;
label: string;
/** Omitted for a row that names no drawn shape, such as a bucket's total. */
color?: string;
value: string;
}
/** What a tooltip says, whichever chart opened it: a heading and its rows. */
export interface TooltipContent {
title: string;
rows: TooltipRow[];
}
/**
* `TooltipWithBounds` positions itself with an inline transform and drops it
* when `unstyled` is set, so the default look is replaced by handing it an empty
* style object rather than by turning styling off.
*/
const NO_INLINE_STYLE = {};
/** The gap the tooltip keeps from its anchor point. */
const TOOLTIP_OFFSET = 8;
export function ChartTooltip({
content,
index,
left,
top = 0,
}: {
content: TooltipContent;
/** Which item the tooltip names; part of what forces a fresh measurement. */
index: number;
left: number;
top?: number;
}) {
// `withBoundingRects` measures once, in `componentDidMount`, and never again,
// so every content change needs its own mount to be measured at its own size.
// The index alone is not enough: a refresh can widen the same item's text past
// a digit boundary, or resolve a client's name, and the stale width would place
// it wrongly at the right edge.
const measureKey = [index, content.title, ...content.rows.map((row) => `${row.label}=${row.value}`)].join("|");
return (
<TooltipWithBounds
key={measureKey}
left={left}
top={top}
offsetLeft={TOOLTIP_OFFSET}
offsetTop={TOOLTIP_OFFSET}
style={NO_INLINE_STYLE}
className={stylex.props(styles.tooltip).className}
>
<div {...stylex.props(styles.tooltipTitle)}>{content.title}</div>
<dl {...stylex.props(styles.tooltipList)}>
{content.rows.map((row) => (
<div key={row.key} {...stylex.props(styles.tooltipRow)}>
<dt {...stylex.props(styles.tooltipTerm)}>
{row.color !== undefined && (
<span
aria-hidden="true"
{...stylex.props(styles.swatch, styles.swatchColor(row.color))}
/>
)}
{row.label}
</dt>
<dd {...stylex.props(shared.tabularNums)}>{row.value}</dd>
</div>
))}
</dl>
</TooltipWithBounds>
);
}
@@ -1,93 +0,0 @@
import type { Bucket } from "@/lib/types";
import { MARGIN, isEmptyTimeseries, layoutTimeseries, niceTicks } from "./chartLayout";
function bucket(ts: number, queries: number, blocked = 0, cached = 0): Bucket {
return { ts, queries, blocked, cached };
}
describe("niceTicks", () => {
test("zero max yields a single zero tick", () => {
expect(niceTicks(0)).toEqual([0]);
});
test("picks a 1/2/5 step and extends past max", () => {
expect(niceTicks(7)).toEqual([0, 2, 4, 6, 8]);
expect(niceTicks(100)).toEqual([0, 50, 100]);
expect(niceTicks(1234)).toEqual([0, 500, 1000, 1500]);
});
});
describe("isEmptyTimeseries", () => {
test("true for no buckets and for all-zero buckets", () => {
expect(isEmptyTimeseries([])).toBe(true);
expect(isEmptyTimeseries([bucket(0, 0), bucket(60, 0)])).toBe(true);
});
test("false when any bucket has queries", () => {
expect(isEmptyTimeseries([bucket(0, 0), bucket(60, 3)])).toBe(false);
});
});
describe("layoutTimeseries", () => {
test("segment heights are proportional and stack to the queries total", () => {
const layout = layoutTimeseries([bucket(0, 100, 40, 10), bucket(60, 50, 0, 0)], 480, 240);
const plotHeight = 240 - MARGIN.top - MARGIN.bottom;
const baseline = MARGIN.top + plotHeight;
const [first, second] = layout.bars;
expect(layout.scaleMax).toBe(100);
expect(first.other).toBe(50);
expect(first.segments.blocked.height).toBeCloseTo(plotHeight * 0.4);
expect(first.segments.cached.height).toBeCloseTo(plotHeight * 0.1);
expect(first.segments.other.height).toBeCloseTo(plotHeight * 0.5);
expect(first.segments.blocked.y + first.segments.blocked.height).toBeCloseTo(baseline);
expect(first.segments.cached.y + first.segments.cached.height).toBeCloseTo(first.segments.blocked.y);
expect(first.segments.other.y + first.segments.other.height).toBeCloseTo(first.segments.cached.y);
expect(first.segments.other.y).toBeCloseTo(MARGIN.top);
expect(second.segments.other.height).toBeCloseTo(plotHeight * 0.5);
});
test("clamps other at zero when blocked + cached exceed queries", () => {
const layout = layoutTimeseries([bucket(0, 10, 8, 5)], 480, 240);
expect(layout.bars[0].other).toBe(0);
expect(layout.bars[0].segments.other.height).toBe(0);
});
test("zero data still lays out zero-height bars on a unit scale", () => {
const layout = layoutTimeseries([bucket(0, 0), bucket(60, 0)], 480, 240);
expect(layout.scaleMax).toBe(1);
expect(layout.bars).toHaveLength(2);
for (const bar of layout.bars) {
expect(bar.segments.blocked.height).toBe(0);
expect(bar.segments.cached.height).toBe(0);
expect(bar.segments.other.height).toBe(0);
}
expect(layout.yTicks).toEqual([{ value: 0, y: MARGIN.top + (240 - MARGIN.top - MARGIN.bottom) }]);
});
test("single bucket fills the plot width minus the gap", () => {
const layout = layoutTimeseries([bucket(0, 5, 1, 1)], 480, 240);
const plotWidth = 480 - MARGIN.left - MARGIN.right;
const bar = layout.bars[0];
expect(bar.slot.width).toBeCloseTo(plotWidth);
expect(bar.segments.blocked.width).toBeCloseTo(plotWidth - 2);
expect(bar.segments.blocked.x).toBeCloseTo(MARGIN.left + 1);
expect(layout.xTicks).toEqual([{ ts: 0, x: MARGIN.left + plotWidth / 2 }]);
});
test("x ticks thin out when buckets outnumber the label budget", () => {
const buckets = Array.from({ length: 168 }, (_, i) => bucket(i * 3600, i));
const layout = layoutTimeseries(buckets, 800, 240);
expect(layout.xTicks.length).toBeLessThan(buckets.length / 10);
expect(layout.xTicks[0].ts).toBe(0);
const xs = layout.xTicks.map((tick) => tick.x);
expect([...xs].sort((a, b) => a - b)).toEqual(xs);
});
test("empty bucket list yields no bars and no x ticks", () => {
const layout = layoutTimeseries([], 480, 240);
expect(layout.bars).toEqual([]);
expect(layout.xTicks).toEqual([]);
expect(layout.scaleMax).toBe(1);
});
});
-182
View File
@@ -1,182 +0,0 @@
import type { Bucket } from "@/lib/types";
export interface Rect {
x: number;
y: number;
width: number;
height: number;
}
export interface BarLayout {
bucket: Bucket;
/** queries - blocked - cached, clamped at 0. */
other: number;
slot: Rect;
segments: {
blocked: Rect;
cached: Rect;
other: Rect;
};
}
export interface ChartLayout {
width: number;
height: number;
plot: Rect;
scaleMax: number;
bars: BarLayout[];
yTicks: { value: number; y: number }[];
xTicks: { ts: number; x: number }[];
}
export const MARGIN = { top: 8, right: 8, bottom: 22, left: 44 } as const;
const BAR_GAP = 2;
const MIN_X_LABEL_PX = 90;
/** Tick values from 0 upward in a 1/2/5 step, extended until the last tick covers `max`. */
export function niceTicks(max: number, targetCount = 4): number[] {
if (max <= 0) return [0];
const rawStep = max / targetCount;
const magnitude = Math.pow(10, Math.floor(Math.log10(rawStep)));
const normalized = rawStep / magnitude;
const step = (normalized <= 1 ? 1 : normalized <= 2 ? 2 : normalized <= 5 ? 5 : 10) * magnitude;
const ticks: number[] = [];
for (let value = 0; ; value += step) {
ticks.push(value);
if (value >= max) break;
}
return ticks;
}
export function isEmptyTimeseries(buckets: Bucket[]): boolean {
return buckets.every((bucket) => bucket.queries === 0);
}
function plotRect(width: number, height: number): Rect {
return {
x: MARGIN.left,
y: MARGIN.top,
width: Math.max(0, width - MARGIN.left - MARGIN.right),
height: Math.max(0, height - MARGIN.top - MARGIN.bottom),
};
}
/**
* How many columns share one x-axis label. A 30-day window is 30 columns and a
* 1-hour window is 60, so at narrow widths the labels have to thin out rather
* than overprint each other.
*/
function labelStepFor(count: number, plotWidth: number): number {
if (count === 0 || plotWidth <= 0) return 1;
return Math.max(1, Math.ceil((count * MIN_X_LABEL_PX) / plotWidth));
}
/** One stacked column: the series values in the order the caller stacks them. */
export interface StackedColumn {
ts: number;
total: number;
slot: Rect;
segments: Rect[];
}
export interface StackedLayout {
width: number;
height: number;
plot: Rect;
scaleMax: number;
columns: StackedColumn[];
yTicks: { value: number; y: number }[];
xTicks: { ts: number; x: number }[];
}
/**
* The same geometry as the query-volume chart, for an arbitrary number of
* series. The scale comes from the tallest column's own total, because every
* series here is a disjoint part of the whole rather than a highlighted subset
* of a separately reported total.
*/
export function layoutStacked(
columns: { ts: number; values: number[] }[],
width: number,
height: number,
): StackedLayout {
const plot = plotRect(width, height);
const totals = columns.map((column) => column.values.reduce((sum, value) => sum + value, 0));
const tickValues = niceTicks(Math.max(0, ...totals));
const scaleMax = Math.max(tickValues[tickValues.length - 1], 1);
const baseline = plot.y + plot.height;
const toHeight = (value: number) => (value / scaleMax) * plot.height;
const slotWidth = columns.length > 0 ? plot.width / columns.length : 0;
const barWidth = Math.max(1, slotWidth - BAR_GAP);
const laidOut: StackedColumn[] = columns.map((column, i) => {
const slotX = plot.x + i * slotWidth;
const barX = slotX + (slotWidth - barWidth) / 2;
let top = baseline;
const segments = column.values.map((value) => {
const segmentHeight = toHeight(value);
top -= segmentHeight;
return { x: barX, y: top, width: barWidth, height: segmentHeight };
});
return {
ts: column.ts,
total: totals[i],
slot: { x: slotX, y: plot.y, width: slotWidth, height: plot.height },
segments,
};
});
const step = labelStepFor(columns.length, plot.width);
return {
width,
height,
plot,
scaleMax,
columns: laidOut,
yTicks: tickValues.map((value) => ({ value, y: baseline - toHeight(value) })),
xTicks: laidOut
.filter((_, i) => i % step === 0)
.map((column) => ({ ts: column.ts, x: column.slot.x + column.slot.width / 2 })),
};
}
export function layoutTimeseries(buckets: Bucket[], width: number, height: number): ChartLayout {
const plot = plotRect(width, height);
const maxQueries = buckets.reduce((max, bucket) => Math.max(max, bucket.queries), 0);
const tickValues = niceTicks(maxQueries);
const scaleMax = Math.max(tickValues[tickValues.length - 1], 1);
const baseline = plot.y + plot.height;
const toHeight = (value: number) => (value / scaleMax) * plot.height;
const slotWidth = buckets.length > 0 ? plot.width / buckets.length : 0;
const barWidth = Math.max(1, slotWidth - BAR_GAP);
const bars: BarLayout[] = buckets.map((bucket, i) => {
const slotX = plot.x + i * slotWidth;
const barX = slotX + (slotWidth - barWidth) / 2;
const other = Math.max(0, bucket.queries - bucket.blocked - bucket.cached);
const blockedH = toHeight(bucket.blocked);
const cachedH = toHeight(bucket.cached);
const otherH = toHeight(other);
return {
bucket,
other,
slot: { x: slotX, y: plot.y, width: slotWidth, height: plot.height },
segments: {
blocked: { x: barX, y: baseline - blockedH, width: barWidth, height: blockedH },
cached: { x: barX, y: baseline - blockedH - cachedH, width: barWidth, height: cachedH },
other: { x: barX, y: baseline - blockedH - cachedH - otherH, width: barWidth, height: otherH },
},
};
});
const yTicks = tickValues.map((value) => ({ value, y: baseline - toHeight(value) }));
const labelStep = labelStepFor(buckets.length, plot.width);
const xTicks = bars
.filter((_, i) => i % labelStep === 0)
.map((bar) => ({ ts: bar.bucket.ts, x: bar.slot.x + bar.slot.width / 2 }));
return { width, height, plot, scaleMax, bars, yTicks, xTicks };
}
@@ -1,47 +0,0 @@
import { layoutDonut, type DonutSlice } from "./donutLayout";
function slice(key: string, value: number): DonutSlice {
return { key, label: key, value, color: "#000000" };
}
test("an empty breakdown has no total and no arcs to draw", () => {
expect(layoutDonut([], 100, 20)).toEqual({ size: 100, total: 0, arcs: [] });
});
test("a breakdown of nothing but zeroes is empty, not a division by zero", () => {
const layout = layoutDonut([slice("a", 0), slice("b", 0)], 100, 20);
expect(layout.total).toBe(0);
expect(layout.arcs).toEqual([]);
});
test("zero-valued entries are dropped rather than legended at 0%", () => {
const layout = layoutDonut([slice("a", 3), slice("b", 0), slice("c", 1)], 100, 20);
expect(layout.arcs.map((arc) => arc.slice.key)).toEqual(["a", "c"]);
expect(layout.total).toBe(4);
});
test("shares are of the drawn total and add up to one", () => {
const layout = layoutDonut([slice("a", 3), slice("b", 1)], 100, 20);
expect(layout.arcs.map((arc) => arc.share)).toEqual([0.75, 0.25]);
});
test("slices keep the order they were ranked in, starting at twelve o'clock", () => {
const layout = layoutDonut([slice("a", 1), slice("b", 1)], 100, 20);
expect(layout.arcs[0].d.startsWith("M 50.000 0.000")).toBe(true);
// The second slice begins where the first ended, half a turn round.
expect(layout.arcs[1].d.startsWith("M 50.000 100.000")).toBe(true);
});
test("a slice over half the ring takes the large-arc flag", () => {
const layout = layoutDonut([slice("a", 9), slice("b", 1)], 100, 20);
expect(layout.arcs[0].d).toContain("A 50 50 0 1 1");
expect(layout.arcs[1].d).toContain("A 50 50 0 0 1");
});
test("a single entry is a closed ring, not a zero-length arc that draws nothing", () => {
const layout = layoutDonut([slice("only", 7)], 100, 20);
expect(layout.arcs).toHaveLength(1);
expect(layout.arcs[0].share).toBe(1);
// Two half arcs out and two back: a lone `A` from a point to itself is a no-op.
expect(layout.arcs[0].d.match(/A /g)).toHaveLength(4);
});
@@ -1,95 +0,0 @@
/**
* Annulus geometry for the two breakdown donuts. Pure, so the arithmetic that
* decides whether a slice closes correctly is testable without a DOM.
*/
export interface DonutSlice {
/** Semantic identity: the React key, the colour key and the legend's identity. */
key: string;
label: string;
/** Disambiguates entries whose labels collide — two "Unknown"s, one name on two route kinds. */
secondary?: string;
value: number;
color: string;
}
export interface DonutArc {
slice: DonutSlice;
/** Of the whole, 0 to 1. */
share: number;
d: string;
}
export interface DonutLayout {
size: number;
total: number;
arcs: DonutArc[];
}
const START_ANGLE = -Math.PI / 2;
function point(center: number, radius: number, angle: number): string {
return `${(center + radius * Math.cos(angle)).toFixed(3)} ${(center + radius * Math.sin(angle)).toFixed(3)}`;
}
/**
* A whole-circle slice cannot be drawn as one arc — start and end coincide, and
* the renderer draws nothing at all — so the ring is two half arcs.
*/
function fullRing(center: number, outer: number, inner: number): string {
const top = `${center} ${center - outer}`;
const bottom = `${center} ${center + outer}`;
const innerTop = `${center} ${center - inner}`;
const innerBottom = `${center} ${center + inner}`;
return [
`M ${top}`,
`A ${outer} ${outer} 0 0 1 ${bottom}`,
`A ${outer} ${outer} 0 0 1 ${top}`,
`M ${innerTop}`,
`A ${inner} ${inner} 0 0 0 ${innerBottom}`,
`A ${inner} ${inner} 0 0 0 ${innerTop}`,
"Z",
].join(" ");
}
/**
* Slices in the order given — the caller has already ranked them — starting at
* twelve o'clock and running clockwise. Zero-valued slices are dropped: they
* have no arc to draw, and a legend entry reading 0 is noise.
*/
export function layoutDonut(slices: DonutSlice[], size: number, thickness: number): DonutLayout {
const drawn = slices.filter((slice) => slice.value > 0);
const total = drawn.reduce((sum, slice) => sum + slice.value, 0);
if (total <= 0) return { size, total: 0, arcs: [] };
const center = size / 2;
const outer = center;
const inner = Math.max(0, center - thickness);
if (drawn.length === 1) {
return {
size,
total,
arcs: [{ slice: drawn[0], share: 1, d: fullRing(center, outer, inner) }],
};
}
let angle = START_ANGLE;
const arcs = drawn.map((slice) => {
const share = slice.value / total;
const sweep = share * Math.PI * 2;
const end = angle + sweep;
const large = sweep > Math.PI ? 1 : 0;
const d = [
`M ${point(center, outer, angle)}`,
`A ${outer} ${outer} 0 ${large} 1 ${point(center, outer, end)}`,
`L ${point(center, inner, end)}`,
`A ${inner} ${inner} 0 ${large} 0 ${point(center, inner, angle)}`,
"Z",
].join(" ");
angle = end;
return { slice, share, d };
});
return { size, total, arcs };
}
@@ -1,74 +1,41 @@
/**
* Window coherence across the five Overview requests, migrated from the
* two-request `activityWindow` this replaces. Every behaviour that hook pinned
* is pinned here — the identity, the one retry per mismatch episode, the
* terminal error, the discarded previous-period pair and the stale completion
* that must not speak — now over five endpoints and with the watermark in the
* identity, plus the per-panel isolation the layout added.
* The hook over the single `/api/overview` request.
*
* The five-endpoint build reconciled five window identities here — the retry per
* mismatch episode, the terminal "different window" error, the orphaned stale
* completion. One request cannot disagree with itself, so those behaviours have
* no subject left and are gone rather than ported. What survived the collapse is
* pinned below: the three states, and the one rule a single request still does
* not settle — that a `keepPreviousData` body from the period the reader left
* must never render under the new period's label.
*/
import { render, screen, waitFor } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { createQueryClient } from "@/lib/queryClient";
import type { Coverage, Period } from "@/lib/types";
import {
newerWindow,
sameWindow,
useOverviewWindow,
windowIdOf,
OVERVIEW_ENDPOINTS,
type OverviewEndpoint,
} from "./overviewWindow";
import type { Period } from "@/lib/types";
import { useOverviewWindow } from "./overviewWindow";
const SINCE = Date.UTC(2026, 0, 1, 0, 0) / 1000;
const UNTIL = Date.UTC(2026, 0, 2, 0, 0) / 1000;
const COVERAGE: Coverage = { complete: true, available_since: SINCE };
/** Where each endpoint's body currently ends, and what watermark it admits. */
interface Bounds {
until: number;
availableSince: number;
}
let failing: boolean;
let calls: number;
const PATHS: Record<OverviewEndpoint, string> = {
totals: "/api/stats?period=",
timeseries: "/api/stats/timeseries?period=",
clients: "/api/stats/clients?period=",
types: "/api/stats/types?period=",
routes: "/api/stats/routes?period=",
};
let bounds: Record<OverviewEndpoint, Bounds>;
let failing: Set<OverviewEndpoint>;
let calls: Record<OverviewEndpoint, number>;
/** Endpoints that answer for the page's window from their second call onward. */
let catchUp: Set<OverviewEndpoint>;
/** Held to keep one answer in flight while the test moves the page on. */
let hold: { promise: Promise<void>; release: () => void } | null;
function endpointOf(url: string): OverviewEndpoint | null {
// Longest prefix first: `/api/stats?` and `/api/stats/…` share a stem.
for (const endpoint of ["timeseries", "clients", "types", "routes", "totals"] as const) {
if (url.startsWith(PATHS[endpoint])) return endpoint;
}
return null;
}
function body(endpoint: OverviewEndpoint, period: Period): unknown {
const { until, availableSince } = bounds[endpoint];
const shared = { period, since: SINCE, until, coverage: { ...COVERAGE, available_since: availableSince } };
switch (endpoint) {
case "totals":
return { ...shared, queries: 10, blocked: 2, clients: 1, avg_response_time_us: 1000 };
case "timeseries":
return { ...shared, bucket_seconds: 3600, buckets: [] };
case "clients":
return { ...shared, bucket_seconds: 3600, clients: [], other: [] };
case "types":
return { ...shared, types: [] };
case "routes":
return { ...shared, routes: [] };
}
function body(period: Period): unknown {
return {
period,
since: SINCE,
until: UNTIL,
bucket_seconds: 1800,
totals: { queries: 10, blocked: 2, clients: 1, avg_response_time_us: 1000 },
buckets: [],
clients: [],
other: [],
types: [],
routes: [],
coverage: { complete: true, available_since: SINCE },
};
}
function json(payload: unknown, status = 200): Response {
@@ -76,55 +43,36 @@ function json(payload: unknown, status = 200): Response {
}
beforeEach(() => {
bounds = {
totals: { until: UNTIL, availableSince: SINCE },
timeseries: { until: UNTIL, availableSince: SINCE },
clients: { until: UNTIL, availableSince: SINCE },
types: { until: UNTIL, availableSince: SINCE },
routes: { until: UNTIL, availableSince: SINCE },
};
failing = new Set();
catchUp = new Set();
hold = null;
calls = { totals: 0, timeseries: 0, clients: 0, types: 0, routes: 0 };
failing = false;
calls = 0;
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
const endpoint = endpointOf(url);
if (endpoint === null) return json({ error: "not stubbed" }, 404);
calls[endpoint] += 1;
if (failing.has(endpoint)) return json({ error: "endpoint unavailable" }, 400);
if (catchUp.has(endpoint) && calls[endpoint] >= 2)
bounds[endpoint] = { until: UNTIL, availableSince: SINCE };
if (!url.startsWith("/api/overview")) return json({ error: "not stubbed" }, 404);
calls += 1;
if (failing) return json({ error: "endpoint unavailable" }, 400);
const period = (new URLSearchParams(url.split("?")[1]).get("period") ?? "24h") as Period;
// Built before the wait, so a held answer carries what its own request
// would have returned rather than what the page has moved on to.
const payload = json(body(endpoint, period));
if (hold !== null && endpoint === "routes" && calls.routes === 2) await hold.promise;
return payload;
return json(body(period));
}),
);
});
afterEach(() => vi.unstubAllGlobals());
/** The last retry the hook handed out, so a test can spend it. */
let lastRetry: () => void;
function Probe({ period }: { period: Period }) {
const overview = useOverviewWindow(period);
return (
<ul>
{OVERVIEW_ENDPOINTS.map((endpoint) => {
const panel = overview[endpoint];
const panel = useOverviewWindow(period);
if (panel.status === "error") lastRetry = panel.retry;
const detail =
panel.status === "ready"
? `${panel.data.period}@${panel.data.until}/${panel.data.coverage.available_since}`
? `${panel.data.period}@${panel.data.until}`
: panel.status === "error"
? (panel.error as Error).message
: "";
return <li key={endpoint}>{`${endpoint}:${panel.status}:${detail}`}</li>;
})}
</ul>
);
return <p>{`${panel.status}:${detail}`}</p>;
}
function renderProbe(period: Period = "24h") {
@@ -144,149 +92,36 @@ function renderProbe(period: Period = "24h") {
};
}
function line(endpoint: OverviewEndpoint): string {
const item = screen.getAllByRole("listitem").find((element) => element.textContent?.startsWith(`${endpoint}:`));
if (item === undefined) throw new Error(`no probe line for ${endpoint}`);
return item.textContent ?? "";
function line(): string {
return screen.getByRole("paragraph").textContent ?? "";
}
test("the window identity is the period, both bounds and the watermark together", () => {
const base = { period: "24h" as const, since: SINCE, until: UNTIL, availableSince: SINCE };
expect(sameWindow(base, { ...base })).toBe(true);
expect(sameWindow(base, { ...base, period: "1h" })).toBe(false);
expect(sameWindow(base, { ...base, since: SINCE - 1 })).toBe(false);
expect(sameWindow(base, { ...base, until: UNTIL + 1 })).toBe(false);
// The bounds agree and the answers still describe different windows: a prune
// between the two requests moved what the same span can be answered for.
expect(sameWindow(base, { ...base, availableSince: SINCE + 60 })).toBe(false);
});
test("the newer until wins, and for equal bounds the later watermark does", () => {
const base = { period: "24h" as const, since: SINCE, until: UNTIL, availableSince: SINCE };
expect(newerWindow(base, { ...base, until: UNTIL + 60 }).until).toBe(UNTIL + 60);
expect(newerWindow({ ...base, until: UNTIL + 60 }, base).until).toBe(UNTIL + 60);
expect(newerWindow(base, { ...base, availableSince: SINCE + 60 }).availableSince).toBe(SINCE + 60);
// A newer watermark does not outrank an older window's later bound.
expect(newerWindow({ ...base, until: UNTIL + 60 }, { ...base, availableSince: SINCE + 60 }).until).toBe(UNTIL + 60);
});
test("windowIdOf reads the four fields off any of the five bodies", () => {
expect(windowIdOf({ period: "7d", since: 1, until: 2, coverage: { complete: false, available_since: 3 } })).toEqual(
{
period: "7d",
since: 1,
until: 2,
availableSince: 3,
},
);
});
test("five responses for one window render as five ready panels", async () => {
test("the page is loading until the body for the selected period arrives", async () => {
renderProbe();
await waitFor(() => expect(line("totals")).toContain("ready"));
for (const endpoint of OVERVIEW_ENDPOINTS) {
expect(line(endpoint)).toBe(`${endpoint}:ready:24h@${UNTIL}/${SINCE}`);
}
expect(line()).toBe("loading:");
await waitFor(() => expect(line()).toBe(`ready:24h@${UNTIL}`));
});
test("one endpoint behind a bucket boundary is refetched once and then agrees", async () => {
// Behind on its first answer, caught up by the time the hook asks again.
bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
catchUp.add("routes");
test("a failed request is one error for the whole page, with a retry that refetches", async () => {
failing = true;
renderProbe();
await waitFor(() => expect(line("routes")).toContain("ready"));
expect(calls.routes).toBe(2);
expect(calls.totals).toBe(1);
});
await waitFor(() => expect(line()).toBe("error:endpoint unavailable"));
const spent = calls;
test("a laggard that stays behind fails its own panel and leaves the rest rendering", async () => {
bounds.types = { until: UNTIL - 3600, availableSince: SINCE };
renderProbe();
await waitFor(() => expect(line("types")).toContain("error"));
expect(line("types")).toContain("different window");
// One retry, not a loop.
expect(calls.types).toBe(2);
for (const endpoint of ["totals", "timeseries", "clients", "routes"] as const) {
expect(line(endpoint)).toContain("ready");
}
});
test("a failed request degrades its own panel; the charts keep the window", async () => {
failing.add("routes");
renderProbe();
await waitFor(() => expect(line("routes")).toContain("error"));
expect(line("routes")).toContain("endpoint unavailable");
expect(line("timeseries")).toContain("ready");
expect(line("totals")).toContain("ready");
});
test("a watermark that advanced mid-page is a mismatch, not a mixed window", async () => {
// Same bounds, later watermark: retention pruned between the two responses.
bounds.clients = { until: UNTIL, availableSince: SINCE + 600 };
renderProbe();
await waitFor(() => expect(line("clients")).toContain(`/${SINCE + 600}`));
// The page adopts the later watermark, so the four older answers are the
// laggards and each gets its one retry rather than rendering beside it.
await waitFor(() => expect(calls.totals).toBe(2));
expect(line("clients")).toContain("ready");
failing = false;
lastRetry();
await waitFor(() => expect(line()).toBe(`ready:24h@${UNTIL}`));
expect(calls).toBeGreaterThan(spent);
});
test("a retained previous-period body never renders under the new period's label", async () => {
const { rerenderWith } = renderProbe("24h");
await waitFor(() => expect(line("totals")).toBe(`totals:ready:24h@${UNTIL}/${SINCE}`));
await waitFor(() => expect(line()).toBe(`ready:24h@${UNTIL}`));
rerenderWith("1h");
// Whatever `keepPreviousData` is holding, no panel may claim it answers 1h.
await waitFor(() => expect(line("totals")).toBe(`totals:ready:1h@${UNTIL}/${SINCE}`));
for (const endpoint of OVERVIEW_ENDPOINTS) expect(line(endpoint)).toContain("1h@");
});
test("a period change buys the new window its own retry", async () => {
bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
const { rerenderWith } = renderProbe("24h");
await waitFor(() => expect(line("routes")).toContain("error"));
const spent = calls.routes;
rerenderWith("1h");
// The mismatch persists under the new period, and the episode key changed
// with it: the retry the abandoned period spent is not the new one's.
await waitFor(() => expect(calls.routes).toBeGreaterThan(spent));
await waitFor(() => expect(line("routes")).toContain("error"));
});
test("a retry in flight when the period changes cannot spend the window's retry later", async () => {
// The stale completion the tokens exist to orphan: routes lags under 24h, the
// hook issues its one retry, and the reader picks 1h before that retry lands.
bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
let release = () => {};
hold = { promise: new Promise<void>((resolve) => (release = resolve)), release: () => release() };
const { rerenderWith } = renderProbe("24h");
await waitFor(() => expect(calls.routes).toBe(2));
bounds.routes = { until: UNTIL, availableSince: SINCE };
rerenderWith("1h");
await waitFor(() => expect(line("routes")).toContain("1h@"));
// The abandoned retry lands now, under a period it was never asked for.
hold.release();
hold = null;
await waitFor(() => expect(line("routes")).toContain("ready"));
// Back to the window it was issued for, still lagging. The stale completion
// must not have marked this episode spent: the panel gets a real retry before
// it is allowed to reach the terminal error.
bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
rerenderWith("24h");
// The cached lagging body is there to render immediately, and the panel must
// not state the terminal error off it: that error means "retried and still
// behind", and this visit has not retried anything yet. An abandoned
// completion recording the episode as spent is what would produce it here.
expect(line("routes")).toContain("loading");
await waitFor(() => expect(line("routes")).toContain("different window"));
// `keepPreviousData` is holding the 24h body. It is a complete answer and
// still the wrong one to draw under "1h", so the page waits.
expect(line()).toBe("loading:");
await waitFor(() => expect(line()).toBe(`ready:1h@${UNTIL}`));
});
+22 -237
View File
@@ -1,249 +1,34 @@
/**
* One period, five requests, one window.
* One period, one request, one window.
*
* Totals, the timeline, the per-client series and the two breakdowns are
* separate calls, so a refresh that straddles a bucket boundary — or a retention
* pass that advances the watermark mid-page — can answer them for different
* windows. Rendering them side by side anyway would put a headline count above
* charts of a different span, a mixed page that looks exactly like a real one.
* The five per-panel endpoints this replaces could each answer for a different
* span, so the page had to reconcile five window identities, retry the laggards
* and fail the ones that stayed behind. `GET /api/overview` answers every panel
* out of a single read transaction: the totals, both timelines and both
* breakdowns describe the same span and the same database state by construction,
* and none of that reconciliation has anything left to reconcile.
*
* This is **window** coherence, not data-snapshot coherence: matching bounds
* cannot prove a common database state, and live inserts between requests may
* still shift counts slightly between panels. What it does guarantee is that no
* two panels ever describe different spans.
*
* Rendering is per panel. A panel whose request is still in flight shows its own
* loading state and a panel whose request failed shows its own error, while the
* panels that match the window keep rendering — a failed donut never blanks the
* charts.
* What remains is the one rule a single request does not settle by itself.
* `keepPreviousData` holds the body of the period the reader just left — a
* complete, self-consistent answer, and still the wrong one to draw under the
* new label — so a body is a member of this window only while its own `period`
* is the selected one. Until then the page is loading.
*/
import { useCallback, useEffect, useRef, useState } from "react";
import { keepPreviousData, useQuery, type UseQueryResult } from "@tanstack/react-query";
import { statsClientsQuery, statsQuery, statsRoutesQuery, statsTypesQuery, timeseriesQuery } from "@/lib/queries";
import type {
Coverage,
Period,
StatsClients,
StatsRoutes,
StatsTimeseries,
StatsTotals,
StatsTypes,
} from "@/lib/types";
export const OVERVIEW_ENDPOINTS = ["totals", "timeseries", "clients", "types", "routes"] as const;
export type OverviewEndpoint = (typeof OVERVIEW_ENDPOINTS)[number];
interface EndpointBodies {
totals: StatsTotals;
timeseries: StatsTimeseries;
clients: StatsClients;
types: StatsTypes;
routes: StatsRoutes;
}
/**
* What makes two responses the same window. `available_since` joins the bounds
* because retention advancing between requests changes what the same `[since,
* until)` can answer for, and mixing a pre-prune answer with a post-prune one is
* the failure the bounds alone would not catch.
*/
export interface WindowId {
period: Period;
since: number;
until: number;
availableSince: number;
}
/** The four fields every window-bounded stats body carries. */
interface Bounded {
period: Period;
since: number;
until: number;
coverage: Coverage;
}
export function windowIdOf(body: Bounded): WindowId {
return {
period: body.period,
since: body.since,
until: body.until,
availableSince: body.coverage.available_since,
};
}
export function sameWindow(a: WindowId, b: WindowId): boolean {
return a.period === b.period && a.since === b.since && a.until === b.until && a.availableSince === b.availableSince;
}
/**
* Which of two candidate windows the page adopts: the one that reaches further
* forward in time, and for identical bounds the one that admits the later
* watermark. Both rules pick the answer a laggard has to catch up to.
*/
export function newerWindow(a: WindowId, b: WindowId): WindowId {
if (b.until !== a.until) return b.until > a.until ? b : a;
return b.availableSince > a.availableSince ? b : a;
}
function keyOf(id: WindowId): string {
return `${id.period}|${id.since}|${id.until}|${id.availableSince}`;
}
import { useCallback } from "react";
import { keepPreviousData, useQuery } from "@tanstack/react-query";
import { overviewQuery } from "@/lib/queries";
import type { Overview, Period } from "@/lib/types";
export type Panel<T> =
{ status: "loading" } | { status: "error"; error: unknown; retry: () => void } | { status: "ready"; data: T };
export interface OverviewWindow {
/** Null until one response for the selected period has arrived. */
window: WindowId | null;
/** The adopted window's watermark, for the page's single coverage notice. */
coverage: Coverage | null;
totals: Panel<StatsTotals>;
timeseries: Panel<StatsTimeseries>;
clients: Panel<StatsClients>;
types: Panel<StatsTypes>;
routes: Panel<StatsRoutes>;
}
export function useOverviewWindow(period: Period): Panel<Overview> {
const query = useQuery({ ...overviewQuery(period), placeholderData: keepPreviousData });
const { refetch } = query;
const retry = useCallback(() => void refetch(), [refetch]);
/**
* A laggard that stayed behind after its one retry. Not an `ApiError`: nothing
* failed, the endpoint simply never caught up, and `InlineError` renders the
* message verbatim.
*/
export const MISMATCH = new Error("This panel is for a different window than the rest of the page. Try again.");
export function useOverviewWindow(period: Period): OverviewWindow {
const queries: { [K in OverviewEndpoint]: UseQueryResult<EndpointBodies[K]> } = {
totals: useQuery({ ...statsQuery(period), placeholderData: keepPreviousData }),
timeseries: useQuery({ ...timeseriesQuery(period), placeholderData: keepPreviousData }),
clients: useQuery({ ...statsClientsQuery(period), placeholderData: keepPreviousData }),
types: useQuery({ ...statsTypesQuery(period), placeholderData: keepPreviousData }),
routes: useQuery({ ...statsRoutesQuery(period), placeholderData: keepPreviousData }),
};
// A `keepPreviousData` placeholder for the period just left is a complete,
// self-consistent body — and still the wrong one to show under the new label,
// so it is neither a candidate for the window nor a member of it.
const answers = new Map<OverviewEndpoint, WindowId>();
for (const endpoint of OVERVIEW_ENDPOINTS) {
const data = queries[endpoint].data;
if (data !== undefined && data.period === period) answers.set(endpoint, windowIdOf(data));
}
let window: WindowId | null = null;
for (const id of answers.values()) window = window === null ? id : newerWindow(window, id);
// The effect below runs on what the responses say, not on how many times they
// arrived: a poll that returns byte-identical data must not restart the retry
// bookkeeping. The refetchers ride a ref for the same reason — TanStack hands
// back a fresh function identity on some renders, and depending on it would
// re-enter the effect with nothing changed.
const answersKey = OVERVIEW_ENDPOINTS.map((endpoint) => {
const id = answers.get(endpoint);
return id === undefined ? "" : keyOf(id);
}).join("~");
const latest = useRef({ answers, refetch: queries });
latest.current = { answers, refetch: queries };
// Which mismatch episode each endpoint has already spent its retry on, keyed
// by endpoint and window identity so a new window buys a new attempt.
const retriedFor = useRef(new Map<OverviewEndpoint, string>());
// Which retry each endpoint is waiting on. Per endpoint, because one shared
// counter would let a second endpoint's retry silence the first's completion;
// bumped on every retry issued, so a completion from a window or a period the
// page has left can neither clear an error the current one reached nor spend
// the current window's one retry.
const tokens = useRef(new Map<OverviewEndpoint, number>());
// State, not a ref: a retry that returns byte-identical data changes nothing
// else a render could see, and the panel still has to reach its error.
const [landedFor, setLandedFor] = useState(new Map<OverviewEndpoint, string>());
// Leaving a period ends every episode it opened. A retry issued for the old
// period can still be in flight, and without this its completion would land
// under the new one holding a token the map still honours: it would record an
// episode as spent, so a return to that window would reach the terminal error
// without the retry that error is supposed to follow. Bumping the tokens
// orphans those answers, and the cleared maps let the new window start clean.
const [lastPeriod, setLastPeriod] = useState(period);
if (lastPeriod !== period) {
setLastPeriod(period);
for (const endpoint of OVERVIEW_ENDPOINTS) {
tokens.current.set(endpoint, (tokens.current.get(endpoint) ?? 0) + 1);
}
retriedFor.current.clear();
setLandedFor(new Map());
}
const windowKey = window === null ? null : keyOf(window);
useEffect(() => {
if (windowKey === null) return;
for (const [endpoint, identity] of latest.current.answers) {
if (keyOf(identity) === windowKey) {
retriedFor.current.delete(endpoint);
continue;
}
const episode = `${endpoint}|${windowKey}`;
if (retriedFor.current.get(endpoint) === episode) continue;
retriedFor.current.set(endpoint, episode);
const token = (tokens.current.get(endpoint) ?? 0) + 1;
tokens.current.set(endpoint, token);
const landed = () => {
if (tokens.current.get(endpoint) !== token) return;
setLandedFor((previous) => new Map(previous).set(endpoint, episode));
};
void latest.current.refetch[endpoint].refetch().then(landed, landed);
}
}, [answersKey, windowKey]);
const retry = useCallback((endpoint: OverviewEndpoint) => {
retriedFor.current.delete(endpoint);
tokens.current.set(endpoint, (tokens.current.get(endpoint) ?? 0) + 1);
setLandedFor((previous) => {
const next = new Map(previous);
next.delete(endpoint);
return next;
});
void latest.current.refetch[endpoint].refetch();
}, []);
function panelOf<K extends OverviewEndpoint>(endpoint: K): Panel<EndpointBodies[K]> {
const query = queries[endpoint];
const onRetry = () => retry(endpoint);
if (query.isError) return { status: "error", error: query.error, retry: onRetry };
const data = query.data;
if (
data !== undefined &&
windowKey !== null &&
data.period === period &&
keyOf(windowIdOf(data)) === windowKey
) {
return { status: "ready", data };
}
if (windowKey !== null && landedFor.get(endpoint) === `${endpoint}|${windowKey}`) {
return { status: "error", error: MISMATCH, retry: onRetry };
}
if (query.isError) return { status: "error", error: query.error, retry };
if (query.data !== undefined && query.data.period === period) return { status: "ready", data: query.data };
return { status: "loading" };
}
const panels = {
totals: panelOf("totals"),
timeseries: panelOf("timeseries"),
clients: panelOf("clients"),
types: panelOf("types"),
routes: panelOf("routes"),
};
// The notice describes the window, so any member of it can supply the
// watermark: whichever panel arrived says the same thing about coverage.
let coverage: Coverage | null = null;
for (const endpoint of OVERVIEW_ENDPOINTS) {
const panel = panels[endpoint];
if (panel.status === "ready") {
coverage = panel.data.coverage;
break;
}
}
return { window, coverage, ...panels };
}
+3 -3
View File
@@ -1,4 +1,4 @@
import { ApiError, deleteGroup, getQueries, getStats, listGroups, login, putGroupSources } from "@/lib/api";
import { ApiError, deleteGroup, getOverview, getQueries, listGroups, login, putGroupSources } from "@/lib/api";
function jsonResponse(payload: unknown, status = 200, headers: Record<string, string> = {}): Response {
return new Response(JSON.stringify(payload), {
@@ -50,7 +50,7 @@ test("falls back to a status message on a non-JSON error body", async () => {
test("parses Retry-After on 429", async () => {
fetchMock.mockResolvedValue(jsonResponse({ error: "rate limited" }, 429, { "Retry-After": "17" }));
const failure = await getStats("1h").catch((e: unknown) => e);
const failure = await getOverview("1h").catch((e: unknown) => e);
expect(failure).toBeInstanceOf(ApiError);
expect((failure as ApiError).status).toBe(429);
expect((failure as ApiError).retryAfter).toBe(17);
@@ -58,7 +58,7 @@ test("parses Retry-After on 429", async () => {
test("ignores a malformed Retry-After header", async () => {
fetchMock.mockResolvedValue(jsonResponse({ error: "rate limited" }, 429, { "Retry-After": "soon" }));
const failure = await getStats().catch((e: unknown) => e);
const failure = await getOverview().catch((e: unknown) => e);
expect((failure as ApiError).retryAfter).toBeUndefined();
});
+4 -13
View File
@@ -23,6 +23,7 @@ import type {
LoginResponse,
LogoutResponse,
LookupResult,
Overview,
PausePost,
PauseState,
Period,
@@ -35,11 +36,6 @@ import type {
SettingsEnvelope,
SettingsPatch,
SourceStatus,
StatsClients,
StatsRoutes,
StatsTimeseries,
StatsTotals,
StatsTypes,
Upstream,
UpstreamEcho,
UpstreamInput,
@@ -112,7 +108,7 @@ export const login = (body: LoginRequest): Promise<LoginResponse> =>
request("/api/auth/login", { method: "POST", body });
export const logout = (): Promise<LogoutResponse> => request("/api/auth/logout", { method: "POST", body: {} });
// Query log + stats
// Query log + overview
export const getQueries = (filter: QueriesFilter = {}): Promise<QueriesPage> =>
request(`/api/queries${qs({ ...filter })}`);
@@ -123,13 +119,8 @@ export const getQueryDetail = (id: number): Promise<QueryDetail> => request(`/ap
/** `EventSource` URL for the live stream; not a fetch route. */
export const liveQueriesUrl = "/api/queries/live";
export const getStats = (period?: Period): Promise<StatsTotals> => request(`/api/stats${qs({ period })}`);
export const getStatsTimeseries = (period?: Period): Promise<StatsTimeseries> =>
request(`/api/stats/timeseries${qs({ period })}`);
export const getStatsTypes = (period?: Period): Promise<StatsTypes> => request(`/api/stats/types${qs({ period })}`);
export const getStatsRoutes = (period?: Period): Promise<StatsRoutes> => request(`/api/stats/routes${qs({ period })}`);
export const getStatsClients = (period?: Period): Promise<StatsClients> =>
request(`/api/stats/clients${qs({ period })}`);
/** Every Overview panel for one window, from one read transaction. */
export const getOverview = (period?: Period): Promise<Overview> => request(`/api/overview${qs({ period })}`);
export const getLookup = (domain: string, groupId?: number): Promise<LookupResult> =>
request(`/api/lookup${qs({ domain, group_id: groupId })}`);
+34 -143
View File
@@ -29,6 +29,7 @@ import type {
LoginResponse,
LogoutResponse,
LookupResult,
Overview,
PauseState,
QueriesPage,
QueryDetail,
@@ -36,11 +37,6 @@ import type {
RuleEcho,
SettingsEnvelope,
SourceStatus,
StatsClients,
StatsRoutes,
StatsTimeseries,
StatsTotals,
StatsTypes,
Upstream,
UpstreamEcho,
Version,
@@ -441,7 +437,7 @@ export const sample_create_upstream: UpstreamEcho = {
enabled: true,
id: 0,
priority: 0,
restart_required: true,
restart_required: false,
tls_name: "",
url: "https://dns2.example/dns-query",
};
@@ -458,7 +454,7 @@ export const sample_update_upstream: UpstreamEcho = {
enabled: true,
id: 0,
priority: 0,
restart_required: true,
restart_required: false,
tls_name: "",
url: "https://dns.example/dns-query",
};
@@ -588,39 +584,6 @@ export const sample_get_query_detail: QueryDetail = {
},
};
export const sample_get_stats: StatsTotals = {
avg_response_time_us: null,
blocked: 0,
clients: 0,
coverage: {
available_since: 0,
complete: true,
},
period: "1h",
queries: 0,
since: 0,
until: 0,
};
export const sample_get_stats_timeseries: StatsTimeseries = {
bucket_seconds: 0,
buckets: [
{
blocked: 0,
cached: 0,
queries: 0,
ts: 0,
},
],
coverage: {
available_since: 0,
complete: true,
},
period: "1h",
since: 0,
until: 0,
};
export const sample_get_pause: PauseState = {
paused: false,
until: null,
@@ -633,51 +596,18 @@ export const sample_post_pause: PauseState = {
export const sample_get_settings: SettingsEnvelope = {
restart_required: [
"upstream.attempt_timeout_ms",
"upstream.read_timeout_ms",
"upstream.total_timeout_ms",
"dns.bind_ipv4",
"dns.bind_ipv6",
"dns.port",
"dns.rate_limit",
"dns.rate_window_seconds",
"blocking.response",
"blocking.ttl",
"cache.size",
"cache.negative_ttl_max",
"web.enabled",
"web.bind",
"web.port",
"web.session_ttl_hours",
"web.api_rate_limit_per_min",
"web.api_localhost_exempt",
"web.sse_max_connections_per_ip",
"web.trusted_proxies",
"doh_server.enabled",
"doh_server.bind",
"doh_server.port",
"doh_server.cert_path",
"doh_server.key_path",
"dot_server.enabled",
"dot_server.bind",
"dot_server.port",
"dot_server.cert_path",
"dot_server.key_path",
"edns.ecs_mode",
"logging.level",
"logging.retention_days",
"logging.query_log_buffer_max",
"logging.query_log_flush_interval_s",
"logging.hide_domains",
"logging.hide_client_ips",
"logging.output",
"logging.file_path",
"logging.max_size_mb",
"logging.max_files",
"disk.min_free_mb",
"disk.warn_free_mb",
"blocklist_update.enabled",
"blocklist_update.interval_hours",
],
settings: {
blocking: {
@@ -753,51 +683,18 @@ export const sample_get_settings: SettingsEnvelope = {
export const sample_put_settings: SettingsEnvelope = {
restart_required: [
"upstream.attempt_timeout_ms",
"upstream.read_timeout_ms",
"upstream.total_timeout_ms",
"dns.bind_ipv4",
"dns.bind_ipv6",
"dns.port",
"dns.rate_limit",
"dns.rate_window_seconds",
"blocking.response",
"blocking.ttl",
"cache.size",
"cache.negative_ttl_max",
"web.enabled",
"web.bind",
"web.port",
"web.session_ttl_hours",
"web.api_rate_limit_per_min",
"web.api_localhost_exempt",
"web.sse_max_connections_per_ip",
"web.trusted_proxies",
"doh_server.enabled",
"doh_server.bind",
"doh_server.port",
"doh_server.cert_path",
"doh_server.key_path",
"dot_server.enabled",
"dot_server.bind",
"dot_server.port",
"dot_server.cert_path",
"dot_server.key_path",
"edns.ecs_mode",
"logging.level",
"logging.retention_days",
"logging.query_log_buffer_max",
"logging.query_log_flush_interval_s",
"logging.hide_domains",
"logging.hide_client_ips",
"logging.output",
"logging.file_path",
"logging.max_size_mb",
"logging.max_files",
"disk.min_free_mb",
"disk.warn_free_mb",
"blocklist_update.enabled",
"blocklist_update.interval_hours",
],
settings: {
blocking: {
@@ -903,31 +800,35 @@ export const sample_error_not_found: ErrorEnvelope = {
error: "not found",
};
export const sample_get_stats_types: StatsTypes = {
coverage: {
available_since: 0,
complete: true,
},
period: "1h",
since: 0,
types: [
export const sample_get_overview: Overview = {
bucket_seconds: 0,
buckets: [
{
count: 0,
qtype: 0,
blocked: 0,
cached: 0,
queries: 0,
ts: 0,
},
],
clients: [
{
count: 0,
qtype: null,
buckets: [0],
client: "192.0.2.30",
},
{
buckets: [0],
client: "192.0.2.31",
},
{
buckets: [0],
client: "192.0.2.32",
},
],
until: 0,
};
export const sample_get_stats_routes: StatsRoutes = {
coverage: {
available_since: 0,
complete: true,
},
other: [0],
period: "1h",
routes: [
{
@@ -972,32 +873,22 @@ export const sample_get_stats_routes: StatsRoutes = {
},
],
since: 0,
until: 0,
};
export const sample_get_stats_clients: StatsClients = {
bucket_seconds: 0,
clients: [
totals: {
avg_response_time_us: 0,
blocked: 0,
clients: 0,
queries: 0,
},
types: [
{
buckets: [0],
client: "192.0.2.30",
count: 0,
qtype: 0,
},
{
buckets: [0],
client: "192.0.2.31",
},
{
buckets: [0],
client: "192.0.2.32",
count: 0,
qtype: null,
},
],
coverage: {
available_since: 0,
complete: true,
},
other: [0],
period: "1h",
since: 0,
until: 0,
};
+8 -39
View File
@@ -21,11 +21,7 @@ import type {
export const queryKeys = {
health: ["health"] as const,
version: ["version"] as const,
stats: (period: Period) => ["stats", period] as const,
timeseries: (period: Period) => ["stats", "timeseries", period] as const,
statsTypes: (period: Period) => ["stats", "types", period] as const,
statsRoutes: (period: Period) => ["stats", "routes", period] as const,
statsClients: (period: Period) => ["stats", "clients", period] as const,
overview: (period: Period) => ["overview", period] as const,
queriesInfinite: (filter: QueriesFilter) => ["queries", "infinite", filter] as const,
queryDetail: (id: number) => ["queries", "detail", id] as const,
diagnosticsInfinite: (filter: DiagnosticsFilter) => ["diagnostics", "infinite", filter] as const,
@@ -54,34 +50,10 @@ export const healthQuery = () =>
export const versionQuery = () =>
queryOptions({ queryKey: queryKeys.version, queryFn: api.getVersion, staleTime: Infinity });
export const statsQuery = (period: Period = "24h") =>
queryOptions({ queryKey: queryKeys.stats(period), queryFn: () => api.getStats(period), refetchInterval: 30_000 });
export const timeseriesQuery = (period: Period = "24h") =>
export const overviewQuery = (period: Period = "24h") =>
queryOptions({
queryKey: queryKeys.timeseries(period),
queryFn: () => api.getStatsTimeseries(period),
refetchInterval: 30_000,
});
export const statsTypesQuery = (period: Period = "24h") =>
queryOptions({
queryKey: queryKeys.statsTypes(period),
queryFn: () => api.getStatsTypes(period),
refetchInterval: 30_000,
});
export const statsRoutesQuery = (period: Period = "24h") =>
queryOptions({
queryKey: queryKeys.statsRoutes(period),
queryFn: () => api.getStatsRoutes(period),
refetchInterval: 30_000,
});
export const statsClientsQuery = (period: Period = "24h") =>
queryOptions({
queryKey: queryKeys.statsClients(period),
queryFn: () => api.getStatsClients(period),
queryKey: queryKeys.overview(period),
queryFn: () => api.getOverview(period),
refetchInterval: 30_000,
});
@@ -328,14 +300,11 @@ export const clientPrefixesPutMutation = (qc: QueryClient) => ({
},
});
// The pool builds its clients at startup, so every upstream write leaves the
// server owing a restart. The flag it sets lives on /api/config/status, and the
// shell notice reads it there — hence the second invalidation.
// An upstream write rebuilds the pool in the running process, so the row list
// is the only thing it changes: no restart is owed and /api/config/status says
// the same thing after the write as before it.
function invalidateUpstreams(qc: QueryClient): Promise<unknown> {
return Promise.all([
qc.invalidateQueries({ queryKey: queryKeys.upstreams }),
qc.invalidateQueries({ queryKey: queryKeys.configStatus }),
]);
return qc.invalidateQueries({ queryKey: queryKeys.upstreams });
}
export const upstreamCreateMutation = (qc: QueryClient) => ({
+24 -39
View File
@@ -291,15 +291,13 @@ export interface DiagnosticsFilter {
before?: number;
}
export interface StatsTotals {
period: Period;
since: number;
until: number;
/** The window's four headline numbers. */
export interface OverviewTotals {
queries: number;
blocked: number;
/** Distinct clients seen in the window, not a sum of per-bucket counts. */
clients: number;
avg_response_time_us: number | null;
coverage: Coverage;
}
export interface Bucket {
@@ -309,67 +307,53 @@ export interface Bucket {
cached: number;
}
export interface StatsTimeseries {
period: Period;
since: number;
until: number;
bucket_seconds: number;
buckets: Bucket[];
coverage: Coverage;
}
/**
* One DNS type's share of the window. `qtype` is the numeric code as logged:
* naming it is the admin's job (`features/provenance/qtype.ts`), and a row whose
* type was never recorded keeps its own `null` group rather than disappearing.
*/
export interface StatsTypeRow {
export interface OverviewTypeRow {
qtype: number | null;
count: number;
}
export interface StatsTypes {
period: Period;
since: number;
until: number;
types: StatsTypeRow[];
coverage: Coverage;
}
/**
* How the window's queries were answered. `source` names the answering upstream
* on `upstream` rows and the zone on `forward_zone` rows; every other route kind
* carries null, as does a row whose identity was not recorded.
*/
export interface StatsRouteRow {
export interface OverviewRouteRow {
route: RouteKind;
source: string | null;
count: number;
}
export interface StatsRoutes {
period: Period;
since: number;
until: number;
routes: StatsRouteRow[];
coverage: Coverage;
}
/** One client's per-bucket counts, aligned to `StatsTimeseries`'s buckets. */
export interface StatsClientSeries {
/** One client's per-bucket counts, aligned to `Overview.buckets`. */
export interface OverviewClientSeries {
client: string;
buckets: number[];
}
export interface StatsClients {
/**
* Everything the Overview page draws, for one window, from one request. The
* server answers all six panels out of a single read transaction, so the
* headline totals, the two timelines and the two breakdowns are guaranteed to
* describe the same span *and* the same database state — a coherence the five
* endpoints this replaces could not offer.
*/
export interface Overview {
period: Period;
since: number;
until: number;
bucket_seconds: number;
totals: OverviewTotals;
buckets: Bucket[];
/** The eight busiest clients in the window, ranked by total count. */
clients: StatsClientSeries[];
clients: OverviewClientSeries[];
/** Everything outside the top eight. Always present and always bucket-count-sized. */
other: number[];
types: OverviewTypeRow[];
routes: OverviewRouteRow[];
coverage: Coverage;
}
@@ -550,7 +534,7 @@ export interface UpstreamEcho {
priority: number;
enabled: boolean;
tls_name: string;
restart_required: true;
restart_required: false;
}
export interface PauseState {
@@ -646,8 +630,9 @@ export interface SettingsEnvelope {
* authenticated route, never one of the open ones.
*
* `restart_pending` is true once the server has committed a change only a
* restart applies (an upstream write, a settings key). Nothing but process
* exit clears it, so a browser reload cannot dismiss it.
* restart applies. Every settings key applies live except the listener binds
* and `web.enabled`, so those are the only writes that raise it. Nothing but
* process exit clears it, so a browser reload cannot dismiss it.
*/
export interface ConfigStatus {
authority: "database" | "managed_file";
+17 -16
View File
@@ -32,18 +32,15 @@ import {
groupsQuery,
healthQuery,
localRecordsQuery,
overviewQuery,
queriesInfiniteQuery,
queryDetailQuery,
rulesQuery,
settingsQuery,
statsClientsQuery,
statsQuery,
statsRoutesQuery,
statsTypesQuery,
timeseriesQuery,
upstreamsQuery,
} from "@/lib/queries";
import { DEFAULT_PERIOD, parsePeriod } from "@/features/overview/period";
import { OverviewPending } from "@/features/overview/OverviewFrame";
import {
validateGroupId,
validateProtectionSearch,
@@ -168,26 +165,28 @@ const overviewRoute = createRoute({
}),
loaderDeps: ({ search }): { period: Period } => ({ period: search.period ?? DEFAULT_PERIOD }),
/**
* Started here, awaited nowhere. Every panel reads these with `useQuery` and
* owns its own loading and error surface, so awaiting would trade that whole
* contract for one blocking navigation: the page would sit on the slowest of
* five requests and then appear complete, instead of the four that answered
* rendering beside the one still in flight. The rejections are caught only to
* keep them from going unhandled; the panels state them.
* Started here, awaited nowhere. The page reads these with `useQuery` and owns
* its own loading and error surface, so awaiting would trade that contract for
* one blocking navigation: nothing at all until the request answered, rather
* than the heading and the period picker while it is in flight. The rejections
* are caught only to keep them from going unhandled; the page states them.
*/
loader: ({ context, deps }) => {
const start = (promise: Promise<unknown>) => void promise.catch(() => {});
start(context.queryClient.ensureQueryData(healthQuery()));
start(context.queryClient.ensureQueryData(statsQuery(deps.period)));
start(context.queryClient.ensureQueryData(timeseriesQuery(deps.period)));
start(context.queryClient.ensureQueryData(statsClientsQuery(deps.period)));
start(context.queryClient.ensureQueryData(overviewQuery(deps.period)));
// The registered names the client chart labels its series with. Started here
// so the lookup is not a second round trip after the page chunk lands.
start(context.queryClient.ensureQueryData(clientsQuery()));
start(context.queryClient.ensureQueryData(statsTypesQuery(deps.period)));
start(context.queryClient.ensureQueryData(statsRoutesQuery(deps.period)));
},
component: lazyRouteComponent(() => import("@/features/overview/OverviewPage")),
/**
* The page's own loading surface, rendered while its chunk is still in flight.
* The default pending component would put a second, differently-placed
* "Loading…" before it, which reads as a stutter rather than one wait.
* `OverviewFrame` is a separate module so this import leaves the charts lazy.
*/
pendingComponent: OverviewPending,
});
/**
@@ -425,6 +424,8 @@ export function createAppRouter(history?: RouterHistory, queryClient: QueryClien
context: { queryClient },
defaultPreload: "intent",
defaultPreloadStaleTime: 0,
scrollRestoration: true,
scrollToTopSelectors: ["#main-content"],
defaultPendingComponent: RoutePending,
defaultErrorComponent: RouteError,
});
+11 -30
View File
@@ -19,44 +19,16 @@ const RECONCILED_AT = 1754899200;
const DATABASE: ConfigStatus = { authority: "database", path: null, reconciled_at: null, restart_pending: false };
const RESPONSES: Record<string, unknown> = {
"/api/stats?period=24h": {
period: "24h",
since: 0,
until: 86400,
queries: 0,
blocked: 0,
clients: 0,
avg_response_time_us: null,
coverage: { complete: true, available_since: 0 },
},
"/api/stats/timeseries?period=24h": {
"/api/overview?period=24h": {
period: "24h",
since: 0,
until: 86400,
bucket_seconds: 1800,
totals: { queries: 0, blocked: 0, clients: 0, avg_response_time_us: null },
buckets: [],
coverage: { complete: true, available_since: 0 },
},
"/api/stats/clients?period=24h": {
period: "24h",
since: 0,
until: 86400,
bucket_seconds: 1800,
clients: [],
other: [],
coverage: { complete: true, available_since: 0 },
},
"/api/stats/types?period=24h": {
period: "24h",
since: 0,
until: 86400,
types: [],
coverage: { complete: true, available_since: 0 },
},
"/api/stats/routes?period=24h": {
period: "24h",
since: 0,
until: 86400,
routes: [],
coverage: { complete: true, available_since: 0 },
},
@@ -146,6 +118,15 @@ test("shell renders the overview route with all nav links", async () => {
}
});
test("main carries the ids the router scrolls and restores", async () => {
renderShell();
await screen.findByRole("heading", { name: "Overview" });
const main = screen.getByRole("main");
expect(main.id).toBe("main-content");
expect(main.getAttribute("data-scroll-restoration-id")).toBe("main");
});
test("the three configuration pages sit under a labelled group, after the rest", async () => {
renderShell();
await screen.findByRole("heading", { name: "Overview" });
+12 -2
View File
@@ -112,6 +112,8 @@ const styles = stylex.create({
},
shell: {
minHeight: "100dvh",
height: { default: null, [WIDE]: "100dvh" },
overflow: { default: null, [WIDE]: "hidden" },
backgroundColor: colors.surface,
color: colors.text,
display: { default: "block", [WIDE]: "grid" },
@@ -120,6 +122,8 @@ const styles = stylex.create({
sidebar: {
display: { default: "none", [WIDE]: "flex" },
flexDirection: { default: null, [WIDE]: "column" },
minHeight: { default: null, [WIDE]: 0 },
overflow: { default: null, [WIDE]: "hidden" },
borderRightWidth: 1,
borderRightStyle: "solid",
borderRightColor: colors.border,
@@ -133,11 +137,14 @@ const styles = stylex.create({
},
sidebarNav: {
flex: 1,
minHeight: { default: null, [WIDE]: 0 },
overflowY: { default: null, [WIDE]: "auto" },
paddingInline: "0.5rem",
},
column: {
display: "flex",
minHeight: "100dvh",
minHeight: { default: "100dvh", [WIDE]: 0 },
minWidth: { default: null, [WIDE]: 0 },
flexDirection: "column",
},
header: {
@@ -177,6 +184,9 @@ const styles = stylex.create({
},
main: {
flex: 1,
minHeight: { default: null, [WIDE]: 0 },
minWidth: { default: null, [WIDE]: 0 },
overflowY: { default: null, [WIDE]: "auto" },
padding: "1rem",
},
});
@@ -335,7 +345,7 @@ export default function AppShell() {
<SidebarFooter />
</div>
)}
<main {...stylex.props(styles.main)}>
<main id="main-content" data-scroll-restoration-id="main" {...stylex.props(styles.main)}>
<Outlet />
</main>
</div>
+23
View File
@@ -60,3 +60,26 @@ if (globalThis.CSS === undefined) {
* than per call; a test that genuinely never resolves still fails, only later.
*/
configure({ asyncUtilTimeout: 5000 });
/**
* jsdom implements no `Element.prototype.scrollTo`, and its `window.scrollTo` is
* a stub that logs "Not implemented". The router's scroll restoration calls both
* on every navigation, so without these the shell throws into its error boundary
* in tests while working in a browser. jsdom has no layout, so a scroll is a
* position assignment and nothing more.
*/
if (typeof Element.prototype.scrollTo !== "function") {
Element.prototype.scrollTo = function scrollTo(...args: unknown[]) {
const options = (typeof args[0] === "object" ? args[0] : { left: args[0], top: args[1] }) as ScrollToOptions;
if (typeof options.top === "number") this.scrollTop = options.top;
if (typeof options.left === "number") this.scrollLeft = options.left;
} as typeof Element.prototype.scrollTo;
}
window.scrollTo = function scrollTo(...args: unknown[]) {
const options = (typeof args[0] === "object" ? args[0] : { left: args[0], top: args[1] }) as ScrollToOptions;
if (typeof options.top === "number")
Object.defineProperty(window, "scrollY", { value: options.top, configurable: true });
if (typeof options.left === "number")
Object.defineProperty(window, "scrollX", { value: options.left, configurable: true });
} as typeof window.scrollTo;
+20 -1
View File
@@ -275,7 +275,21 @@ pub fn build(b: *std.Build) void {
// needs the operator's terminal so `git commit -S` can reach pinentry —
// none of which a workflow supplies and all of which a Run step passes
// through.
// The cut's schema gate compares the querylog fingerprint of the previous
// release against this tree's. It must read that number from the file the
// server uses, never from a copy: a duplicated DDL or a duplicated hash
// would let the gate pass a schema change it no longer describes. Only
// `fingerprint` and `fingerprintOf` are referenced, both of which are
// comptime-computable text hashing, so no SQLite symbol is pulled in and
// the host tool needs no library.
const querylog_schema_mod = b.createModule(.{
.root_source_file = b.path("src/storage/querylog_schema.zig"),
.target = b.graph.host,
.optimize = optimize,
});
const cut_tool = hostTool(b, "cut");
cut_tool.root_module.addImport("querylog_schema", querylog_schema_mod);
const cut_run = b.addRunArtifact(cut_tool);
// It pushes commits and tags, so it must never be answered from the run
// cache, and it must run at the build root whatever directory `zig build`
@@ -298,7 +312,12 @@ pub fn build(b: *std.Build) void {
.optimize = optimize,
}),
});
test_step.dependOn(&b.addRunArtifact(cut_tests).step);
cut_tests.root_module.addImport("querylog_schema", querylog_schema_mod);
const cut_tests_run = b.addRunArtifact(cut_tests);
// The schema-gate round trip reads `src/storage/querylog_schema.zig` off
// disk, so the test binary has to run at the build root.
cut_tests_run.setCwd(b.path("."));
test_step.dependOn(&cut_tests_run.step);
addDist(b, options, admin_assets, .{
.version = version_option,
+1 -1
View File
@@ -1,6 +1,6 @@
.{
.name = .nxdns,
.version = "0.0.9",
.version = "0.0.14",
.minimum_zig_version = "0.16.0",
.paths = .{""},
.fingerprint = 0x3307b311dded1d91,
+1
View File
@@ -37,6 +37,7 @@ Descriptions of what is there. No procedures, no advice.
- [reference/api.md](reference/api.md) — every REST route, authentication and the event stream.
- [reference/cli.md](reference/cli.md) — the six subcommands, every flag, every exit code.
- [reference/files-and-directories.md](reference/files-and-directories.md) — the data directory layout and file modes.
- [reference/query-log-lifecycle.md](reference/query-log-lifecycle.md) — how `querylog.db` is versioned, migrated, backed up and, rarely, recreated.
- [reference/performance.md](reference/performance.md) — the targets and the measured numbers.
## Explanation
+2 -2
View File
@@ -27,7 +27,7 @@ Directories:
| `src/cache/` | `dns_cache.zig`: bounded in-memory TTL cache of whole response messages, keyed by the question. The clock arrives as a parameter. |
| `src/upstream/` | Upstream resolution: shared vocabulary and the `Client` interface (`transport.zig`), DoH client (RFC 8484), DoT client (RFC 7858), per-endpoint health and backoff (`health.zig`), and `pool.zig` — priority-ordered failover that is itself a `transport.Client`, so the handler sees one interface. |
| `src/server/` | The serving side: UDP/TCP/DoH/DoT listeners, `handler.zig` (the whole query pipeline), `cert_store.zig` (refcounted TLS cert holder), `rate_limiter.zig`, `pause.zig`, `clients.zig` (client auto-materialisation), `local_tables.zig` (published local-answer tables), `query_sink.zig` (log and SSE fanout), `shutdown.zig` (SIGINT/SIGTERM into one `std.Io.Event`). |
| `src/storage/` | SQLite ownership: `db.zig` is the only file that calls SQLite, `config_schema.zig` + `migrations.zig` for `config.db`, `querylog_schema.zig` (open-or-recreate), async query `logger.zig`, `retention.zig`, `disk_monitor.zig`, and one repository per table under `repositories/`. |
| `src/storage/` | SQLite ownership: `db.zig` is the only file that calls SQLite, `config_schema.zig` + `migrations.zig` for `config.db`, `querylog_schema.zig` + `querylog_versions.zig` + `querylog_migrations.zig` for `querylog.db`, async query `logger.zig`, `retention.zig`, `disk_monitor.zig`, and one repository per table under `repositories/`. |
| `src/config/` | The one configuration model (`model.zig`), the pure validator (`validate.zig`), `import.zig`/`export.zig` (ZON to and from `config.db`, byte-stable round trip), `loader.zig` (read/parse/validate a named file, with the shared fault mapping), `reconcile.zig` (converge the database onto a parsed config by row identity). |
| `src/web/` | The admin HTTP layer: `server.zig` (listener), `router.zig`/`routes.zig`, one file per resource under `handlers/`, `auth.zig` (sessions), `sse.zig` (live query fanout), `static.zig` (embedded SPA), `metrics.zig` (Prometheus), `openapi.zig` (served contract), `api_limiter.zig`, `http_util.zig`. |
| `src/platform/` | OS and TLS edges: IP address values, the `std.log` sink (`logging.zig`), `statfs.zig` (free-space query via libc), client TLS over `std.crypto.tls` (`tls_client.zig`), server TLS over vendored Mbed TLS (`tls_server.zig`). |
@@ -115,7 +115,7 @@ Two databases with opposite contracts, in one data directory (see [reference/fil
Which of the file and the database is *authoritative* is chosen by the invocation, not by state: bare `nxdns run` serves the database, and `nxdns run --config FILE` makes the file authoritative and reconciles the database onto it at every start. `reconcile.zig` is that convergence, matching rows by identity and writing only differences, so runtime state — blocklist checksums, compiled snapshots, client history — survives. Why it works that way is [configuration-model.md](configuration-model.md).
**`querylog.db` is expendable.** It is never migrated. Its schema carries a fingerprint derived from the DDL text, and at open, a missing, corrupt, non-database, `quick_check`-failing or fingerprint-mismatched file is moved aside and recreated empty — the old file is kept under a new name rather than deleted, so an operator can still look at it. Retention deletes old rows daily and periodically rewrites the file to reclaim space.
**`querylog.db` is the expendable one, but its history is not thrown away.** It carries a logical version in `PRAGMA user_version`, and an older supported version is migrated in place at open: one transaction, behind one `querylog.db.pre-migrate-<epoch>` backup, of which only the newest is kept. Only real damage recreates the file — missing, corrupt, not a database, or failing `quick_check` — and then the old file is kept under a new name rather than deleted, so an operator can still look at it. A healthy file this build cannot read is neither migrated nor moved aside: the startup refuses and says why, because losing months of history to a rollback is worse than a server that will not start. Retention deletes old rows daily and periodically rewrites the file to reclaim space. The whole contract is [reference/query-log-lifecycle.md](../reference/query-log-lifecycle.md).
The split exists so that the churn of the second database can never endanger the first. Query logs are high-volume, disposable, and the thing most likely to be corrupted by a power cut on an SD card; configuration is small, irreplaceable, and the thing an operator would have to reconstruct by hand. Giving them one file would force the careful contract onto the noisy data or the loose contract onto the valuable data.
+10 -10
View File
@@ -62,7 +62,7 @@ Which of steps 4 and 5 applies to your server depends on its authority. Under `n
Login is `POST /api/auth/login` with a JSON body. Without a session, the API answers 401:
```sh
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8451/api/stats
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8451/api/overview
```
```
@@ -92,7 +92,7 @@ The cookie is named `nxdns_session` and carries `HttpOnly; SameSite=Lax; Path=/`
```sh
curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w '%{http_code}\n' \
http://127.0.0.1:8451/api/stats
http://127.0.0.1:8451/api/overview
```
```
@@ -125,13 +125,13 @@ Sessions live in memory only. A restart logs everyone out. Thirty-two concurrent
```sh
curl -sS -b /tmp/nxdns-lab/cookies.txt -c /tmp/nxdns-lab/cookies.txt \
-X POST http://127.0.0.1:8451/api/auth/logout
curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w 'stats: %{http_code}\n' \
http://127.0.0.1:8451/api/stats
curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w 'overview: %{http_code}\n' \
http://127.0.0.1:8451/api/overview
```
```
{"authenticated":false}
stats: 401
overview: 401
```
Logging out with a stale cookie, or with none, answers the same way. The point of logging out is to end up logged out, and that is where such a request already is.
@@ -154,7 +154,7 @@ Changing the password ends every session, including the one that made the change
```sh
curl -sS -b /tmp/nxdns-lab/c2.txt -o /dev/null -w 'old session: %{http_code}\n' \
http://127.0.0.1:8451/api/stats
http://127.0.0.1:8451/api/overview
curl -sS -X POST http://127.0.0.1:8451/api/auth/login \
-H 'content-type: application/json' -d '{"password":"lab-password"}' \
-w ' (old password)\n'
@@ -217,14 +217,14 @@ curl -sS -X POST http://127.0.0.1:8451/api/auth/login \
curl -sS -c /tmp/nxdns-lab/c5.txt -X POST http://127.0.0.1:8451/api/auth/login \
-H 'content-type: application/json' -d '{"password":"offline-password"}' \
-w ' (new password, http %{http_code})\n'
curl -sS -b /tmp/nxdns-lab/c5.txt -o /dev/null -w 'stats: %{http_code}\n' \
http://127.0.0.1:8451/api/stats
curl -sS -b /tmp/nxdns-lab/c5.txt -o /dev/null -w 'overview: %{http_code}\n' \
http://127.0.0.1:8451/api/overview
```
```
{"error":"invalid password"} (old password, http 401)
{"authenticated":true,"auth_required":true} (new password, http 200)
stats: 200
overview: 200
```
The next export shows the new hash and a null `password` again:
@@ -245,7 +245,7 @@ See [back up and restore](back-up-and-restore.md) for when `import` does need `-
Authentication is off. Every route is open, and a login attempt succeeds without minting anything — there is nothing to log in to, and a session that authorises nothing would be a lie for the browser to store:
```sh
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8453/api/stats
curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8453/api/overview
curl -sS -X POST http://127.0.0.1:8453/api/auth/login \
-H 'content-type: application/json' -d '{"password":"anything"}'
```
+36 -2
View File
@@ -267,11 +267,11 @@ cat /etc/resolv.conf
curl -s http://127.0.0.1:8080/metrics | grep nxdns_upstream_
```
`nxdns_upstream_in_flight` against `nxdns_upstream_slots` is how much of an upstream's concurrency is in use right now, and `nxdns_upstream_queued_total` counts exchanges that had to wait for a slot (with `nxdns_upstream_queued_seconds_total` for how long they waited in total). Both queue counters are approximate — they are sampled when a query is admitted, not measured as a queue length. A `queued_total` climbing with each burst means queries are waiting on the upstream rather than failing at it, and a query that waits past `upstream.total_timeout_ms` is canceled in the queue and answered SERVFAIL.
`nxdns_upstream_in_flight` against `nxdns_upstream_slots` is how much of an upstream's concurrency is in use right now, and `nxdns_upstream_queued_total` counts exchanges that had to wait for a slot (with `nxdns_upstream_queued_seconds_total` for how long they waited in total). Both queue counters are approximate — they are sampled when a query is admitted, not measured as a queue length. A `queued_total` climbing with each burst means queries are waiting on the upstream rather than failing at it, and a query that waits past `upstream.total_timeout_ms` is given up on in the queue and answered SERVFAIL. `nxdns_upstream_budget_exhausted_total` counts those give-ups pool-wide — queries whose own budget ran out, in the queue or mid-attempt, before any upstream answered. It carries no `url` label on purpose: running out of budget is a fact about the pool, so the exhausted attempt is never charged to an upstream's health and is never attributed in the query log, although the row can still name the last endpoint whose success or recorded failure preceded it.
`nxdns_upstream_reuse_recoveries_total` is the other half of the picture: it counts DoT connections that went stale between exchanges and were redialed. A few are normal — a resolver is free to close an idle connection. One per query means the connection is never being reused, and every query is paying a full TLS handshake.
**Fix.** Raise `upstream.total_timeout_ms` if the queue drains but drains too slowly for the budget. Otherwise the queue is telling you the upstream is slow: an upstream answering in a few milliseconds does not fill eight concurrent slots at household query rates, so sustained queueing points at the resolver you configured, and a faster one is the fix. Adding a second upstream is not: failover is strict priority, not load spreading — a query waits for a slot on the first available upstream and only reaches the next one when that upstream fails or is in backoff, so a second entry adds no concurrent capacity to the first. The per-upstream slot count is compiled, not configured, so there is no knob to widen one upstream either.
**Fix.** Raise `upstream.total_timeout_ms` if the queue drains but drains too slowly for the budget. Otherwise the queue is telling you the upstream is slow: an upstream answering in a few milliseconds does not fill eight concurrent slots at household query rates, so sustained queueing points at the resolver you configured, and a faster one is the fix. Adding a second upstream helps only with saturation, not with latency: failover is strict priority, not load spreading. A query does take the next upstream when the first one's slots are all busy — priority orders the candidates that can be admitted right now — but once every eligible upstream is full it blocks on the highest-priority one, and it still sends one query to one upstream at a time. The per-upstream slot count is compiled, not configured, so there is no knob to widen one upstream either.
## The disk is filling up
@@ -339,3 +339,37 @@ nxdns run failed: SchemaTooNew
```
**Fix.** There is no downgrade. Import the export you took before upgrading into a fresh data directory with the older binary; see [Upgrade nxdns](upgrade.md).
## The server refuses to start over querylog.db
**Symptom.** The process stops at startup naming the query log, and the error is one of four names:
```
error(querylog_schema): refusing to open querylog database '/var/lib/nxdns/querylog.db': it is stamped 7, and this build supports schema versions 1 to 1 (SchemaTooNew). The file is left exactly as it is; see docs/how-to/troubleshoot.md, "The server refuses to start over querylog.db"
nxdns run failed: SchemaTooNew
```
This is a refusal, not damage. nxdns will not replace a healthy query log to get itself started, so the file is left exactly as it was — schema, rows, coverage watermark and version stamp all unchanged — and the startup fails instead. All four exit 2, the code that means an operator has to act, because none of them resolves on a retry — the shipped systemd unit's `RestartPreventExitStatus=2 64` stops the unit on the first refusal instead of restart-looping it. `systemctl status nxdns` shows the refusal. `nxdns check` does not grade the query log at all, so it will not reproduce any of these.
[The query-log lifecycle](../reference/query-log-lifecycle.md) is the full contract behind this page.
**Fixes by name.**
- `SchemaTooNew` — the file was stamped by a newer nxdns than the one you are running, which normally means a binary was rolled back. Put the newer release back and start it: the file is exactly as that release left it. If you mean to stay on the older release, that release cannot read this file, so restore the `querylog.db.pre-migrate-<unix-seconds>` copy the upgrade left beside it — stop the server, move `querylog.db` and its `querylog.db-wal` and `querylog.db-shm` out of the way, rename the backup to `querylog.db`, and start. Starting empty is also an option: with the server stopped, move `querylog.db` and both sidecars aside and the next start creates a fresh log.
- `SchemaUnsupported` — the stamp is not a version this build can reach. Either the file predates 0.0.12, or it came from somewhere else, or a release since deliberately broke the schema; the changelog section for the release you are running says so when it is the third case. There is no migration path, by contract. Keep the file if the history matters — copy it somewhere and read it with the `sqlite3` shell — and if starting with an empty log is acceptable, stop the server, move `querylog.db`, `querylog.db-wal` and `querylog.db-shm` out of the data directory by hand, and start again.
- `MigrationBackupFailed` — a migration was due and the pre-migration backup could not be written, so nothing was migrated. The line above names the destination and the reason, which is almost always a full or read-only data directory. Free space or fix the permissions and start again.
- `MigrationFailed` — read the log line above it, because two different states wear this one name.
**Which `MigrationFailed` you have.** The distinction is in the line the migration logged, and it decides whether you do anything at all:
- Before the commit: `querylog migration 1 -> 2 failed before commit (...); the database is unchanged`. Nothing was applied. The file still carries its old version and every row, and this run's backup was deleted because the original is intact. Restarting will attempt the same migration and fail the same way, so this needs the underlying cause — the log line names it — or a report.
- After the commit: `querylog migration 1 -> 2 COMMITTED and the database IS at version 2, but the connection could not be restored: ...; the backup '...' is kept and the next start will open the migrated file normally`. The migration DID complete. Only that one startup is refused, the pre-migration backup is kept, and the next start opens the migrated file on the ordinary current-version path. Start the server again.
In neither case does the server start with an empty log on its own. Recreating a query log automatically is reserved for real corruption; see [why a query log is moved aside](../reference/files-and-directories.md#why-a-query-log-is-moved-aside).
> Not reproduced against a running service: the four refusals are covered by the
> test suite rather than by a hand-driven install, and the released chain has no
> migration step in it yet, so no upgrade produces a `pre-migrate` backup today.
> The messages above are the ones `src/storage/querylog_schema.zig` and
> `src/storage/querylog_migrations.zig` emit, with a data directory path and
> example version numbers filled in.
+1 -1
View File
@@ -263,7 +263,7 @@ nxdns run failed: SchemaTooNew
> hand and `nxdns run` was pointed at it. The two lines above are that run's
> output.
That run exits 1. Recovering means importing the export you took in step 1 into a fresh data directory with the older binary.
That run exits 2, and the shipped unit stops rather than restart-loops it. Recovering means importing the export you took in step 1 into a fresh data directory with the older binary.
### Rolling back from file mode
+6 -10
View File
@@ -15,7 +15,7 @@ The route table is `src/web/routes.zig`; the [Operations](#operations) table bel
413.
- A request whose path matches but whose method does not answers 405 with an `Allow` header. An unknown `/api` path is a JSON 404; unknown non-`/api` paths fall through to the embedded SPA (`index.html`), so client-side routing works.
- Item routes (`{id}`) match a positive integer id only.
- Mutations to groups, blocklists, rules, local records, forward zones, clients and client prefixes take effect live. Upstreams and `/api/settings` are restart-required, and once one of them is written `GET /api/config/status` reports `restart_pending: true`.
- Mutations take effect live, upstreams and `/api/settings` included: a write rebuilds or reconfigures the owner it belongs to in-process. The exceptions are the settings keys that create or destroy a socket — the DNS, web, DoH and DoT bind addresses, ports and enabled flags. Writing one of those commits the row and reports `restart_pending: true` on `GET /api/config/status`; the settings envelope lists exactly those keys under `restart_required`.
- Every route has a policy class — `read`, `config_write` or `runtime_action` — and in file mode the `config_write` routes are refused. See [Configuration authority](#configuration-authority).
## Authentication
@@ -90,7 +90,7 @@ All four keys are always present; the two nullable ones carry `null` rather than
The route requires a session, which is why the filesystem path is here rather than on the open `/api/version` and `/api/health`.
`restart_pending` is per-process state and nothing but process exit clears it. It rises when this server writes an upstream or a settings key — the changes the running process cannot apply — and it is never persisted, so a `false` read after a restart means the restart happened, not that the flag was cleared. In file mode it stays false: those writes are refused before any handler runs.
`restart_pending` is per-process state and nothing but process exit clears it. It rises for exactly one kind of change: a settings key that creates or destroys a socket — the DNS, web, DoH and DoT bind addresses, ports and enabled flags. Every other write, upstreams included, is applied in-process and leaves the flag alone. It is never persisted, so a `false` read after a restart means the restart happened, not that the flag was cleared. In file mode it stays false: those writes are refused before any handler runs.
`reconciled_at` answers exactly one question: **when did this process last read the file?** Compare it against the file's mtime to spot a restart that has not happened yet. It is a hint and not a verdict, in both directions — a clock that stepped, or a copy that preserved mtimes (`git checkout`, `rsync -a`), can make a newer file look older, and the database can change without either timestamp moving. It does not tell you whether the file and the running configuration agree; answering that would take content hashing, which nxdns deliberately does not do.
@@ -109,11 +109,7 @@ Auth `open` means no session is required; `session` means a valid session cookie
| GET | `/api/queries` | session | counted | read | Query log rows for the Activity page |
| GET | `/api/queries/{id}` | session | counted | read | One query, fully explained |
| GET | `/api/queries/live` | session | exempt | read | Live query stream (server-sent events) |
| GET | `/api/stats` | session | counted | read | Totals for a period |
| GET | `/api/stats/timeseries` | session | counted | read | Bucketed counts for a period |
| GET | `/api/stats/types` | session | counted | read | Query-type breakdown for a period |
| GET | `/api/stats/routes` | session | counted | read | How the period's queries were answered |
| GET | `/api/stats/clients` | session | counted | read | Per-client bucketed counts for a period |
| GET | `/api/overview` | session | counted | read | Everything the Overview page draws, for one period |
| GET | `/api/lookup` | session | counted | read | Explain a domain |
| GET | `/api/diagnostics` | session | counted | read | Operational event log |
| DELETE | `/api/diagnostics` | session | counted | runtime action | Purge every resolved event |
@@ -173,7 +169,7 @@ Static assets are not routes. The router sends unmatched non-`/api` paths to the
## Settings keys
The envelope both operations answer with is `{settings, restart_required}`: the stored values, and the list of keys a restart applies. It says nothing about the live authority or a restart already owed — those are per-process facts, and `GET /api/config/status` is their one home.
The envelope both operations answer with is `{settings, restart_required}`: the stored values, and the list of keys a restart applies — the bind addresses, ports and enabled flags of the four listeners, and nothing else. Every other key is applied by the PUT that changes it. It says nothing about the live authority or a restart already owed — those are per-process facts, and `GET /api/config/status` is their one home.
`GET /api/settings` and `PUT /api/settings` speak the `section.field` keys of [the configuration reference](configuration.md), with the values in their database spelling — notably `logging.level` is `"error"`, not `"err"`. Two keys behave differently over the API than in the file: `web.password` is write-only (accepted on a `PUT`, never returned, hashed before storage), and `web.password_hash` is neither readable nor directly writable, because a client that could install a hash could install one whose password it already knows.
@@ -221,6 +217,6 @@ A non-empty `rewrites.cname_target` on a query detail means the decision landed
### Coverage
Every window-bounded read — `GET /api/queries` and the five `GET /api/stats*` endpoints — answers with a `coverage` object: `available_since` is the oldest instant the query log is still complete for, and `complete` is true only when the window the request asked about starts at or after it. Retention deletes rows and advances the watermark in one transaction, so a client can tell an empty window from a pruned one instead of charting the gap as zero. A request with no lower bound at all asks about the whole of history, and is never complete.
Every window-bounded read — `GET /api/queries` and `GET /api/overview` — answers with a `coverage` object: `available_since` is the oldest instant the query log is still complete for, and `complete` is true only when the window the request asked about starts at or after it. Retention deletes rows and advances the watermark in one transaction, so a client can tell an empty window from a pruned one instead of charting the gap as zero. A request with no lower bound at all asks about the whole of history, and is never complete.
Each of these responses reads its rows and its watermark inside one SQLite read transaction, so retention cannot prune between the two and hand back pre-prune rows tagged with a post-prune `available_since`. Coherence stops there: two separate requests are two separate reads, and queries logged between them can move the counts.
Each of these responses reads its rows and its watermark inside one SQLite read transaction, so retention cannot prune between the two and hand back pre-prune rows tagged with a post-prune `available_since`. `GET /api/overview` puts every Overview panel inside that one transaction, so its totals and its four breakdowns describe one database state. Coherence stops there: two separate requests are two separate reads, and queries logged between them can move the counts.
+3 -1
View File
@@ -214,7 +214,7 @@ Prints the usage text to stdout and exits 0. `nxdns --help` and `nxdns -h` do th
| --- | --- |
| 0 | Success. |
| 1 | Runtime failure — I/O, database, out of memory. A partial diagnostic report caused by an allocation failure is a runtime failure, not a verdict on the configuration. |
| 2 | A configuration problem the operator can fix, or a `check` that found one. |
| 2 | A configuration problem the operator can fix, a `check` that found one, or a deliberate refusal to run that no retry will clear. |
| 64 | Usage error — unknown command or flag, a flag without its value, a missing or extra argument. |
Code 2 means the same thing from every subcommand. `src/config/faults.zig` holds the one list of errors that mean "the configuration the operator supplied is wrong", and `run`, `check` and `import` all ask it, so a rejected file exits 2 whichever command read it. The list is every error the validator raises, plus `ParseZon`, `ConfigTooLarge`, `NoUsableUpstreams`, `BadCertificate` and `ManagedConfigUnreadable`. In practice that covers a file with a syntax error, one larger than 4 MiB, one with no `default` group (`MissingDefaultGroup`), one with no enabled upstream (`NoUpstreams`), a bad bind address, a bad rate limit, an unusable certificate, `password` and `password_hash` set together, and a `--config` path that is absent or unreadable.
@@ -237,6 +237,8 @@ load one with `nxdns import <file>`, or make a file the source of truth with `nx
`import` exits 2 for those faults and for `DestructiveImport`. That last one is deliberately not a configuration fault — it reports what applying the file would delete, rather than anything wrong with its content — and `import` decides it for itself; the answer to it is `--allow-delete`, not an edit.
`run` also exits 2 on the four query-log schema refusals — `SchemaTooNew`, `SchemaUnsupported`, `MigrationFailed` and `MigrationBackupFailed` — which are in the same list. They are not a verdict on a file the operator wrote, but they share the property exit 2 exists to signal: the server is refusing on purpose, an operator has to act, and a restart will only repeat the refusal. Exit 1 would put them under the unit's `Restart=on-failure` and loop them. See [the server refuses to start over querylog.db](../how-to/troubleshoot.md#the-server-refuses-to-start-over-querylogdb).
`OutOfMemory` is exit 1 even when problems were recorded, because the report is then incomplete. Every other error is 1.
Where an exit code sends you next: [troubleshoot](../how-to/troubleshoot.md).
+5 -3
View File
@@ -37,12 +37,14 @@ Timeouts for talking to upstream resolvers.
| Key | Type | Default | Unit | Validation | Consumed by |
|---|---|---|---|---|---|
| `upstream.attempt_timeout_ms` | u32 | 2500 | ms | 100120000, and not above `total_timeout_ms` | deadline on one attempt against one upstream inside the pool's failover loop (`src/upstream/pool.zig`), the whole attempt including the connect |
| `upstream.read_timeout_ms` | u32 | 3000 | ms | 100120000 | read deadline on conditional-forward-zone exchanges (`src/local/forward_client.zig`) |
| `upstream.read_timeout_ms` | u32 | 3000 | ms | 100120000 | budget for a whole conditional-forward-zone exchange (`src/local/forward_client.zig`): the UDP attempt, a TC=1 fallback and the TCP retry together |
| `upstream.total_timeout_ms` | u32 | 5000 | ms | 100120000 | per-query budget of the upstream pool (`src/upstream/pool.zig`): every failover attempt together, not one of them; also the `nxdns check` probe deadline |
The two pool budgets nest. `attempt_timeout_ms` bounds one try against one upstream; when it expires the pool records the failure and moves to the next candidate. `total_timeout_ms` bounds the whole loop, so a query against five unreachable upstreams costs the total budget once, not five attempt budgets in a row. When the total expires the in-flight attempt is canceled and the query fails with a timeout.
`read_timeout_ms` is unrelated to both. It bounds a different subsystem — the conditional-forward-zone client — so no cross-check relates it to the pool's budgets, and it is free to sit above either of them.
Budget semantics. The deadline is an instant, computed once when the query enters the pool, and every blocking step spends against it — waiting for a free slot on an upstream as much as the exchange itself. An attempt therefore runs against `min(now + attempt_timeout_ms, deadline)`, which near the end of the budget is *truncated*: shorter than the configured attempt. Attribution follows from that. A truncated attempt that expires is evidence about the budget, not about the upstream, so it leaves that upstream's health and success rate untouched and the query log's `upstream` field unchanged — the row may still name the endpoint of the preceding attributable attempt, and is null only when there was none — and it increments the pool-wide `nxdns_upstream_budget_exhausted_total` metric. An attempt that expires on its full budget, or that fails outright (a refused connection, a bad answer, the peer's own timeout), is evidence about the upstream: it is recorded against that upstream's health and names it in the query log. Either way the client is answered SERVFAIL. Forward-zone queries are not affected — they have one configured resolver, so a timeout there always names it.
`read_timeout_ms` is unrelated to both pool budgets. It bounds a different subsystem — the conditional-forward-zone client — so no cross-check relates it to the pool's budgets, and it is free to sit above either of them. Within that subsystem it is one budget for the whole exchange: a query that goes out over UDP, comes back truncated and is retried over TCP has the two legs and the fallback share `read_timeout_ms`, never one each.
### dns
@@ -318,7 +320,7 @@ Zones resolved by a specific resolver instead of the configured upstreams, for L
| `zone` | string | required | a valid domain name; unique |
| `resolver` | string | required | `udp://IP:port` or `tcp://IP:port`; the host must be an IP literal and the port is mandatory |
The resolver host must be an IP literal because resolving the resolver's own name would be a bootstrap problem. Matching is longest suffix (`src/local/forward_zones.zig`); the exchange is UDP then TCP (`src/local/forward_client.zig`) with `upstream.read_timeout_ms` as the read deadline.
The resolver host must be an IP literal because resolving the resolver's own name would be a bootstrap problem. Matching is longest suffix (`src/local/forward_zones.zig`); the exchange is UDP then TCP (`src/local/forward_client.zig`), with `upstream.read_timeout_ms` bounding the whole exchange rather than each leg.
Reverse zones are declared the same way, and one is the prerequisite for [learned client names](#learned-names):
+11 -4
View File
@@ -16,10 +16,12 @@ Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and `nx
| --- | --- | --- |
| `config.db` | The configuration database, including `web.password_hash`. The source of truth in database mode; in file mode it is the runtime substrate the file is reconciled onto (see [the configuration file](#the-configuration-file)). | 0600 |
| `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created by `run`, `import` and `export` when WAL is enabled, inheriting the main file's permissions. `check` creates neither. | 0600 |
| `querylog.db` | The query log: every domain every client asked for. Expendable — if it is missing or unusable it is recreated empty. | 0600 |
| `querylog.db` | The query log: every domain every client asked for. Missing or damaged, it is recreated empty; an older schema is migrated in place, and a schema this build cannot use refuses the startup. See [the query-log lifecycle](query-log-lifecycle.md). | 0600 |
| `querylog.db-wal`, `querylog.db-shm` | WAL sidecars for `querylog.db`. | 0600 |
| `querylog.db.<reason>-<unix-seconds>` | A `querylog.db` this build could not use, moved aside before an empty one was created in its place. Kept, never overwritten. `<reason>` is one of `corrupt`, `not-a-database`, `quick-check-failed` or `schema-changed`; see [why a query log is moved aside](#why-a-query-log-is-moved-aside). | Whatever the renamed file had — no chmod reaches it |
| `querylog.db.<reason>-<unix-seconds>` | A damaged `querylog.db`, moved aside before an empty one was created in its place. Kept, never overwritten. `<reason>` is one of `corrupt`, `not-a-database` or `quick-check-failed`; see [why a query log is moved aside](#why-a-query-log-is-moved-aside). | Whatever the renamed file had — no chmod reaches it |
| `querylog.db.<reason>-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 1 and rises until the name is free. Two recreates within one second is the case it exists for. | The same |
| `querylog.db.pre-migrate-<unix-seconds>` | A complete copy of `querylog.db` taken immediately before a schema migration. Exactly one survives: a successful migration deletes every other one, and a later successful start retries that cleanup. Written by `VACUUM INTO`, so it holds the committed database including anything still only in the write-ahead log, and it needs no sidecars of its own. | 0600 |
| `querylog.db.pre-migrate-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 2. | The same |
| `blocklists/` | Compiled blocklist snapshots, one subdirectory of the data directory. | 0700 |
| `blocklists/<id>.list` | Exact domains for blocklist source `<id>`, one per line, behind a header. | 0600 |
| `blocklists/<id>.wild` | Wildcard entries for the same source. | 0600 |
@@ -52,14 +54,15 @@ The temporaries of a source that still exists are cleaned by the refresh that ow
### Why a query log is moved aside
A `querylog.db` is moved aside when it is missing nothing but usability, and the name it is given says which of the four cases it hit:
A `querylog.db` is moved aside only when it is genuinely damaged, and the name it is given says which of the three cases it hit:
| `<reason>` | What happened |
| --- | --- |
| `corrupt` | SQLite reported the file as damaged. |
| `not-a-database` | The file is not a SQLite database at all. |
| `quick-check-failed` | `PRAGMA quick_check` did not answer `ok`. |
| `schema-changed` | Nothing is wrong with the file. Its `user_version` fingerprint does not match this build's schema, so this build cannot read it. Upgrades that touch the query-log schema produce this one, and the file they set aside is a healthy database. |
There is no fourth case. A healthy file carrying a schema version this build cannot use is neither migrated nor renamed: the startup refuses and the file stays where it is, which is [the query-log lifecycle](query-log-lifecycle.md). You can still find a `querylog.db.schema-changed-<unix-seconds>` in a data directory, because 0.0.13 and older produced one on any schema change — and still do, if you downgrade to one of them. This build never writes that name.
Only the main file is renamed — its `-wal` and `-shm` are deleted, because a stale WAL would be replayed into the fresh database. A missing `querylog.db` is created without any aside file. The rename happens inside `querylog_schema.open`, before the 0600 chmod, and that chmod names `querylog.db` and its two sidecars only — so an aside file keeps the mode the file had at rename time, which for a `querylog.db` nxdns itself created is 0600 and for one an operator put there is whatever they left it at. Nothing prunes the aside files; they accumulate until an operator removes them, and each one holds the same browsing history the live query log holds.
@@ -67,6 +70,10 @@ Only the main file is renamed — its `-wal` and `-shm` are deleted, because a s
The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash and `querylog.db` holds the browsing history of every client on the LAN, so both are as sensitive as each other, and a WAL file holds the same rows as the database it belongs to. SQLite creates the main database at `0644 & ~umask`; nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather than being created world-readable.
A `pre-migrate` backup holds that same browsing history, so it is chmodded 0600 the way the live file is: SQLite's `VACUUM INTO` creates it at `0644 & ~umask` and nxdns restricts it immediately afterwards. A backup it cannot restrict is a failed backup — the partial file is deleted and the startup refuses with `MigrationBackupFailed`, rather than leaving a world-readable copy behind.
The aside files are the exception: they get no chmod at all, and an aside keeps whatever mode it had at rename time. For a `querylog.db` nxdns itself created that is 0600; for one an operator put there it is whatever they left it at.
## The configuration file
There is no default path. `--config FILE` names the file, and without that flag no file is read at all — a `config.zon` sitting in `/etc/nxdns` that no invocation names is inert. `/etc/nxdns/config.zon` is a convention the packaging follows, not a location nxdns probes.
+70
View File
@@ -0,0 +1,70 @@
# The query log's lifecycle
What happens to `querylog.db` when nxdns opens it: how the file is versioned, when it is migrated, when the server refuses to start over it, and the one case in which it is still replaced. Source of truth: `src/storage/querylog_versions.zig` (the version metadata), `src/storage/querylog_schema.zig` (the open path) and `src/storage/querylog_migrations.zig` (the migration runner).
The rule this page exists to state: **a healthy `querylog.db` is never replaced and never moved aside.** A schema this build cannot use refuses the startup instead. Your query history is not the server's to discard.
## The version stamp
Every `querylog.db` carries a logical schema version in SQLite's `PRAGMA user_version`. It is a small counter — 1 in this release — and not a hash of anything. A file created by this build is stamped as it is created.
Two other values matter, both in `querylog_versions.zig`:
| Constant | Today | What it means |
| --- | --- | --- |
| `current_version` | 1 | The version this build creates and reads. |
| `minimum_supported_version` | 1 | The oldest stamped version this build can migrate up to `current_version`. |
| `legacy_fingerprint` | 1975011655 | The `user_version` the 0.0.12 and 0.0.13 binaries wrote: a CRC32 of their schema text, under the older policy where a mismatch meant "replace the file". |
`legacy_fingerprint` is frozen forever. Those two releases stamped a hash rather than a version, so this build recognises that one literal number as "version 1" and restamps the file as 1 on the first open. The restamp runs in its own transaction; if it fails, the old stamp and every row stay exactly as they were and the startup refuses.
## What an open does
nxdns opens `querylog.db` once at startup, before it serves anything, and no second process shares a data directory. On a file that is readable and passes `PRAGMA quick_check`, the stamp decides:
| Stamp | What happens |
| --- | --- |
| `current_version` | Opens. Nothing is migrated. |
| `legacy_fingerprint` | Read as version 1: restamped to 1, then treated as version 1 by the rows above and below. |
| Between `minimum_supported_version` and `current_version` | Migrated in place, then opens. |
| Above `current_version`, up to 1000000 | REFUSE: `SchemaTooNew`. |
| Anything else — 0, a negative, another fingerprint, a version below the minimum | REFUSE: `SchemaUnsupported`. |
A refusal changes nothing. The schema, the rows, the coverage watermark and the stamp are all left as they are, no file is set aside, no new file is created, and `nxdns run` exits. The log line names the path, the stamp it found, the range this build supports and [the troubleshooting section](../how-to/troubleshoot.md#the-server-refuses-to-start-over-querylogdb).
## Migrating in place
A migration is one backup and one transaction.
1. **Back up.** `VACUUM INTO` writes a complete copy — including anything still only in the write-ahead log — to `querylog.db.pre-migrate-<unix-seconds>` beside the database. If that name is taken, `-2`, `-3` and so on are tried. A backup that cannot be written is `MigrationBackupFailed`, and the partial copy is deleted; an older backup beside it survives.
2. **Migrate.** `BEGIN IMMEDIATE`, re-read the stamp under the lock, run every step, run `PRAGMA foreign_key_check`, stamp the new version, `COMMIT`. One transaction covers the whole chain, so the file is either at the old version or at the new one and never in between.
3. **Clean up.** Every other `querylog.db.pre-migrate-*` beside the file is deleted. **One backup is kept**: the one this migration just took. A later successful start retries that cleanup if it failed.
If a step fails before the commit, the transaction rolls back, this run's backup is deleted, and the startup refuses with `MigrationFailed`. The database keeps the version and the rows it had.
If the commit succeeds and something after it fails, the log says so plainly — the migration DID complete and the file IS at the new version. The backup is kept, the startup still refuses with `MigrationFailed`, and the next start opens the migrated file normally.
## Corruption is the only automatic recreate
Four conditions still create a fresh, empty `querylog.db`: the file is missing, SQLite reports it as corrupt, it is not a SQLite database at all, or `PRAGMA quick_check` does not answer `ok`. Except for the missing case, the unusable file is renamed to `querylog.db.<reason>-<unix-seconds>` and kept. See [why a query log is moved aside](files-and-directories.md#why-a-query-log-is-moved-aside).
Every other failure — a lock held elsewhere, a permission problem, a full disk, a version this build cannot reach — propagates and leaves the file alone.
## Downgrading
**Downgrading to 0.0.13 or older resets your query log.** Those binaries predate this contract: they compare `user_version` against a hash of their own schema text, find this build's version stamp instead, and treat that as a mismatch — so they rename `querylog.db` to `querylog.db.schema-changed-<unix-seconds>` and start an empty log. Nothing is destroyed, but the live log is empty until you put the aside file back, and the restamp that provoked it takes no backup of its own.
To recover, go back to a migration-aware release, stop the server, then, in the data directory:
1. Move the empty `querylog.db` the old binary created out of the way.
2. Delete its `querylog.db-wal` and `querylog.db-shm`. This is not optional: replaying the empty file's write-ahead log into the restored history would corrupt it.
3. Rename `querylog.db.schema-changed-<unix-seconds>` back to `querylog.db`.
4. Start the server.
Downgrading between two migration-aware releases is safe in the sense that matters: a build that finds a stamp above its own `current_version` refuses to start with `SchemaTooNew` and touches nothing. Go forward again, or restore the `pre-migrate` backup the upgrade left.
## Breaking the schema on purpose
A release may still break the query-log schema outright rather than migrate it. That is allowed, and it is never silent. Such a release raises `current_version`, sets `minimum_supported_version` to the same value, and ships no migration step — so files from before the break classify as below the minimum and `open` refuses them with `SchemaUnsupported` rather than replacing them. The release notes carry the phrase `resets your query history` and a `Restoring your query history` section, and the cut gate refuses to build the release without both.
So the contract is: a break is always versioned, always refused at startup with the file intact, and always disclosed in the changelog.
+21
View File
@@ -0,0 +1,21 @@
(MIT)
Copyright (c) 2013 Julian Gruber &lt;julian@juliangruber.com&gt;
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
of the Software, and to permit persons to whom the Software is furnished to do
so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+21
View File
@@ -0,0 +1,21 @@
The MIT License (MIT)
Copyright (c) 2018 Jed Watson
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+25
View File
@@ -0,0 +1,25 @@
The nine ISC-licensed d3 modules the admin UI bundles carry the same permission
notice under different copyright years. The notices are reproduced together
here, one line per package, followed by the ISC text they share.
d3-array Copyright 2010-2022 Mike Bostock
d3-color Copyright 2010-2022 Mike Bostock
d3-format Copyright 2010-2021 Mike Bostock
d3-interpolate Copyright 2010-2021 Mike Bostock
d3-path Copyright 2015-2022 Mike Bostock
d3-scale Copyright 2010-2021 Mike Bostock
d3-shape Copyright 2010-2022 Mike Bostock
d3-time Copyright 2010-2022 Mike Bostock
internmap Copyright 2021 Mike Bostock
Permission to use, copy, modify, and/or distribute this software for any purpose
with or without fee is hereby granted, provided that the above copyright notice
and this permission notice appear in all copies.
THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES WITH
REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY SPECIAL, DIRECT,
INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS
OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER
TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF
THIS SOFTWARE.
+70
View File
@@ -52,20 +52,67 @@ sqlite url=https://sqlite.org/2026/sqlite-amalgamation-3530400.zip hash=N-V-__8A
@tanstack/react-store 0.9.3 MIT
@tanstack/router-core 1.171.15 MIT
@tanstack/store 0.9.3 MIT
@types/d3-array 3.0.3 MIT
@types/d3-color 3.1.0 MIT
@types/d3-delaunay 6.0.1 MIT
@types/d3-format 3.0.1 MIT
@types/d3-geo 3.1.0 MIT
@types/d3-interpolate 3.0.1 MIT
@types/d3-path 3.1.1 MIT
@types/d3-scale 4.0.2 MIT
@types/d3-shape 3.1.7 MIT
@types/d3-time 3.0.0 MIT
@types/d3-time-format 2.1.0 MIT
@types/geojson 7946.0.16 MIT
@types/react 19.2.17 MIT
@types/react-dom 19.2.3 MIT
@visx/axis 4.0.0 MIT
@visx/bounds 4.0.0 MIT
@visx/curve 4.0.0 MIT
@visx/grid 4.0.0 MIT
@visx/group 4.0.0 MIT
@visx/point 4.0.0 MIT
@visx/scale 4.0.0 MIT
@visx/shape 4.0.0 MIT
@visx/text 4.0.0 MIT
@visx/tooltip 4.0.0 MIT
@visx/vendor 4.0.0 MIT and ISC
aria-hidden 1.2.6 MIT
balanced-match 0.4.2 MIT
classnames 2.5.1 MIT
client-only 0.0.1 MIT
clsx 2.1.1 MIT
cookie-es 3.1.1 MIT
css-mediaquery 0.1.2 BSD
csstype 3.2.3 MIT
d3-array 3.2.1 ISC
d3-color 3.1.0 ISC
d3-delaunay 6.0.2 ISC
d3-format 3.1.0 ISC
d3-geo 3.1.0 ISC
d3-interpolate 3.0.1 ISC
d3-path 3.1.0 ISC
d3-scale 4.0.2 ISC
d3-shape 3.2.0 ISC
d3-time 3.1.0 ISC
d3-time-format 4.1.0 ISC
delaunator 5.1.0 ISC
internmap 2.0.3 ISC
invariant 2.2.4 MIT
isbot 5.2.1 Unlicense
js-tokens 4.0.0 MIT
loose-envify 1.4.0 MIT
math-expression-evaluator 1.4.0 MIT
react 19.2.8 MIT
react-aria 3.51.0 Apache-2.0
react-aria-components 1.20.0 Apache-2.0
react-dom 19.2.8 MIT
react-stately 3.49.0 Apache-2.0
react-use-measure 2.1.7 MIT
reduce-css-calc 1.3.0 MIT
reduce-function-call 1.0.3 MIT
balanced-match 1.0.2 MIT
robust-predicates 3.0.3 Unlicense
scheduler 0.27.0 MIT
seroval 1.5.6 MIT
seroval-plugins 1.5.6 MIT
@@ -90,11 +137,34 @@ alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695
@tanstack/react-store
@tanstack/router-core
@tanstack/store
@visx/axis
@visx/bounds
@visx/grid
@visx/group
@visx/point
@visx/scale
@visx/shape
@visx/text
@visx/tooltip
balanced-match
classnames
clsx
d3-array
d3-color
d3-format
d3-interpolate
d3-path
d3-scale
d3-shape
d3-time
internmap
math-expression-evaluator
react
react-aria
react-aria-components
react-dom
react-stately
reduce-css-calc
reduce-function-call
scheduler
use-sync-external-store
+61
View File
@@ -71,6 +71,25 @@
// a `sources` list — no guard would raise it, which is why it is written down
// here.
//
// Twenty-three more joined the not-shipped list with visx, and the sourcemap
// build settled which. Fourteen are types only and emit no runtime code at all:
// @types/d3-array, @types/d3-color, @types/d3-delaunay, @types/d3-format,
// @types/d3-geo, @types/d3-interpolate, @types/d3-path, @types/d3-scale,
// @types/d3-shape, @types/d3-time, @types/d3-time-format, @types/geojson,
// @types/react and @types/react-dom, with csstype under them. They are in the
// runtime closure rather than among the devDependencies only because the @visx
// packages declare them as ordinary dependencies. @visx/curve is the curve
// factory the line and area shapes take, which these charts do not draw.
// @visx/vendor is the one worth stating plainly: it is a re-export shim over the
// d3 modules, npm records it as `MIT and ISC` because of what it re-exports, and
// the bundler resolves straight through it to the d3 packages themselves — so
// its own bytes never reach admin/dist and the d3 entry below is what covers the
// code that does. d3-delaunay, d3-geo, d3-time-format, delaunator and
// robust-predicates are the map and Voronoi half of that shim, which no chart
// here imports; the first four are ISC and robust-predicates is Unlicense.
// react-use-measure is the resize hook @visx/responsive uses, and this app keeps
// its own, so nothing pulls it in.
//
// Of the remaining devDependencies, none puts a byte in admin/dist: @vitejs/
// plugin-react and typescript only transform our own sources (react-refresh is
// dev-server only, and tslib is optional and unused), lightningcss only
@@ -155,6 +174,48 @@
.note = "Bundled into the admin UI JavaScript. Separate entry from the other TanStack packages: same MIT text, different copyright line.",
.file = "tanstack-store-mit.txt",
},
.{
.component = "visx (@visx/axis, @visx/bounds, @visx/grid, @visx/group, @visx/point, @visx/scale, @visx/shape, @visx/text, @visx/tooltip)",
.version = "@visx/axis 4.0.0, @visx/bounds 4.0.0, @visx/grid 4.0.0, @visx/group 4.0.0, @visx/point 4.0.0, @visx/scale 4.0.0, @visx/shape 4.0.0, @visx/text 4.0.0, @visx/tooltip 4.0.0",
.note = "The chart primitives the Overview charts are drawn with — scales, stacked bars, pie arcs, axes, grid and tooltip — bundled into the admin UI JavaScript. Nine of the eleven @visx packages in the closure ship; @visx/curve and @visx/vendor do not (see the note at the head of this file). All nine carry a byte-identical MIT text. In the tarballs and in the image.",
.file = "visx-mit.txt",
},
.{
.component = "d3 (d3-array, d3-color, d3-format, d3-interpolate, d3-path, d3-scale, d3-shape, d3-time, internmap)",
.version = "d3-array 3.2.1, d3-color 3.1.0, d3-format 3.1.0, d3-interpolate 3.0.1, d3-path 3.1.0, d3-scale 4.0.2, d3-shape 3.2.0, d3-time 3.1.0, internmap 2.0.3",
.note = "The scale, colour, number-format and path arithmetic behind visx, bundled into the admin UI JavaScript. They arrive through @visx/vendor, which re-exports them without contributing bytes of its own, so these nine are what the bundle actually carries. The first ISC dependencies this project has taken: Mokhtar Mial accepted ISC inbound for nxdns on 2026-08-24, which is the decision that let them ship. ISC asks only that the copyright notice and the permission notice travel with the copies; the nine notices differ in their copyright years alone, so the file below reproduces every notice above the single shared permission text. In the tarballs and in the image.",
.file = "d3-isc.txt",
},
.{
.component = "classnames",
.version = "classnames 2.5.1",
.note = "The class-name joiner visx uses to merge its own chart class names with a caller's. Bundled into the admin UI JavaScript. Separate entry from the other MIT packages: same MIT text, different copyright line.",
.file = "classnames-mit.txt",
},
.{
.component = "balanced-match",
.version = "balanced-match 0.4.2, balanced-match 1.0.2",
.note = "The bracket matcher reduce-function-call calls when it takes a CSS calc() expression apart. Bundled into the admin UI JavaScript. Two versions are installed, one under reduce-function-call and one at the root; both carry this text. Separate entry from the other MIT packages: same MIT text, different copyright line.",
.file = "balanced-match-mit.txt",
},
.{
.component = "math-expression-evaluator",
.version = "math-expression-evaluator 1.4.0",
.note = "The arithmetic evaluator reduce-css-calc reduces a calc() expression with, reached from @visx/text's width measurement. Bundled into the admin UI JavaScript. Separate entry from the other MIT packages: same MIT text, different copyright line.",
.file = "math-expression-evaluator-mit.txt",
},
.{
.component = "reduce-css-calc",
.version = "reduce-css-calc 1.3.0",
.note = "Resolves CSS calc() expressions for @visx/text. Bundled into the admin UI JavaScript. Separate entry from the other MIT packages: same MIT text, different copyright line.",
.file = "reduce-css-calc-mit.txt",
},
.{
.component = "reduce-function-call",
.version = "reduce-function-call 1.0.3",
.note = "The function-call walker reduce-css-calc is built on. Bundled into the admin UI JavaScript. Separate entry from reduce-css-calc: same MIT text, different copyright line.",
.file = "reduce-function-call-mit.txt",
},
.{
.component = "Vite",
.version = "vite 8.1.5",
+7
View File
@@ -55,6 +55,13 @@ pub const texts: []const Text = &.{
.{ .name = "clsx-mit.txt", .body = @embedFile("clsx-mit.txt") },
.{ .name = "stylex-mit.txt", .body = @embedFile("stylex-mit.txt") },
.{ .name = "styleq-mit.txt", .body = @embedFile("styleq-mit.txt") },
.{ .name = "visx-mit.txt", .body = @embedFile("visx-mit.txt") },
.{ .name = "d3-isc.txt", .body = @embedFile("d3-isc.txt") },
.{ .name = "classnames-mit.txt", .body = @embedFile("classnames-mit.txt") },
.{ .name = "balanced-match-mit.txt", .body = @embedFile("balanced-match-mit.txt") },
.{ .name = "math-expression-evaluator-mit.txt", .body = @embedFile("math-expression-evaluator-mit.txt") },
.{ .name = "reduce-css-calc-mit.txt", .body = @embedFile("reduce-css-calc-mit.txt") },
.{ .name = "reduce-function-call-mit.txt", .body = @embedFile("reduce-function-call-mit.txt") },
.{ .name = "vite-mit.txt", .body = @embedFile("vite-mit.txt") },
.{ .name = "rolldown-mit.txt", .body = @embedFile("rolldown-mit.txt") },
.{ .name = "mozilla-ca-bundle-mpl-2.0.txt", .body = @embedFile("mozilla-ca-bundle-mpl-2.0.txt") },
@@ -0,0 +1,22 @@
The MIT License (MIT)
Copyright (c) 2015 Ankit G.
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+20
View File
@@ -0,0 +1,20 @@
The MIT License (MIT)
Copyright (c) 2014 Maxime Thirouin & Joakim Bengtson
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+20
View File
@@ -0,0 +1,20 @@
The MIT License (MIT)
Copyright (c) Maxime Thirouin
Permission is hereby granted, free of charge, to any person obtaining a copy of
this software and associated documentation files (the "Software"), to deal in
the Software without restriction, including without limitation the rights to
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of
the Software, and to permit persons to whom the Software is furnished to do so,
subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR
COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER
IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2017-2018 Harrison Shoff
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+264
View File
@@ -0,0 +1,264 @@
# Milestone 34: hot-apply — tiers A and B
DB-mode config changes apply live, in-process, for every key that does not create a socket. The comptime "every key is restart-required" blanket is replaced by an explicit per-key apply table. Design: rev 3 of the hot-reload analysis; spec hardened through two Codex review rounds (thread 01a02efc). Listener rebind and `web.enabled` lifecycle are milestone 35.
## The write contract (every session honors it)
Prepare → commit → publish → retire:
1. **Prepare**: under the serialized config-mutation path, build and validate everything the change needs, from the FINAL MERGED config — ONE candidate per affected owner, never one per key. Prepared resources are heap-stable and owned (a candidate outlives the request arena). Nothing is published. Any failure: clean up every prepared resource, error to the client, NO db write.
2. **Commit**: the DB transaction, only after every prepare succeeded. Commit failure: clean up all prepared resources.
3. **Publish**: infallible, I/O-free operations — handle swaps, pointer swaps, stores under a lock. Closing files, joining tasks, freeing memory belong to retire, not publish.
4. **Retire**: old generations/handles are drained, closed, and freed after their readers release.
Signatures: every operation that touches a `std.Io` primitive (Mutex, RwLock, Condition, concurrent, sleep) takes `io: std.Io`. The spec's named signatures include it; a builder adds it wherever else the primitive demands it.
## Sessions
Six sessions, strictly sequential (S1 → … → S6); each session owns the tree while it runs and ends with `zig build test` green (S6 adds the admin suite).
**Honesty rule**: `restart_pending`/`restart_required` keep firing exactly as today until S5 swaps the mechanism atomically. No earlier session removes a restart signal.
---
## Session S1: per-query policy snapshots (tier A)
### S1.1 DNS policy snapshot
`src/server/handler.zig`: the per-query reads — blocking response + ttl, `ecs_mode`, `forward_read_timeout`, `negative_ttl_max` — move into one `Policy` struct behind `std.Io.RwLock` (the filter discipline, manager.zig:438/856). Each query copies the `Policy` ONCE at query start (shared lock only for the copy); `ecs_mode` at :782/:829/:835 reads the copy. Publish: `pub fn setPolicy(h: *Handler, io: std.Io, p: Policy)`.
### S1.2 Web trusted proxies
`src/web/server.zig` ~:531: `trusted_proxies` becomes an owned immutable generation in `WebState`, pointer-replaced under an RwLock (io-threaded), old generation retired after readers release. Prepare copies out of the request arena.
### S1.3 Logger config split
`src/storage/logger.zig`: `hide_domains` + `hide_client_ips` (applied on the PRODUCER path ~:449) are ONE privacy policy: both pack into a single atomic (one u8), stored together by `setPrivacy` and loaded ONCE per entry — a producer can never observe a mixed policy that redacts the domain but exposes the client, or vice versa. `query_log_flush_interval_s` (~:609) is an independent `.monotonic` atomic with `setFlushInterval`.
### S1.4 Disk thresholds
`src/storage/disk_monitor.zig`: `min_free_mb`/`warn_free_mb` are one invariant pair (warn ≥ min): both u32s packed into ONE atomic u64; readers unpack a single load.
### S1 implementation notes (post-build)
- `Context`'s provenance method `policy(...)` renamed `notePolicy` (collision with the new per-query field).
- Pure-atomic setters (`setPrivacy`, `setFlushInterval`, `setThresholds`) take no `io` — they touch no Io primitive.
- `LiveProxies.install` RETURNS the retired generation; the caller frees it in retire.
- The trusted-proxies read hold ends at the client-address verdict, BEFORE router dispatch — a hold spanning dispatch would let a settings PUT deadlock on the shared lock its own request holds.
- The flush-interval test covers short→long only; long→short is unobservable without waking a writer already parked on its old deadline (S5's PUT test inherits this bound).
- S1 setters have no production caller until S5 wires the PUT flow — test-only until then, by design.
### S1.5 Acceptance criteria
- [ ] One query is internally consistent across a concurrent `setPolicy` (both ecs/blocking reads agree).
- [ ] trusted_proxies replaced under concurrent request-path reads; testing allocator clean.
- [ ] Privacy flip with FORCED producer interleaving: pre-flip entries unredacted, post-flip redacted.
- [ ] Threshold reader vs concurrent pair-stores: every observed pair satisfies warn ≥ min.
- [ ] Flush-interval change observed on the writer's next cycle via the existing timing seam.
- [ ] `zig build test` green.
---
## Session S2: upstream extraction + generation owner + lock hygiene
### S2.1 Extract the upstream composition out of app.zig
`Upstreams`, `build`, `deinit`, and what they need from `ConfigLoad` (all private in app.zig ~:1158) move to `src/upstream/owner.zig`; app.zig imports it, never the reverse. `build` returns the generation plus a `BuildReport`. `ConfigLoad.note` is NOT just log output — it writes operational-event rows and retains canonical keys for `finalize` (app.zig:367). Contract: at BOOT, app.zig replays the successful `BuildReport` through the exact `ConfigLoad.note` path before `finalize`, so diagnostics are unchanged; at RUNTIME, candidate preparation is side-effect-free (no event rows) until commit, and the report is RECONCILED in RETIRE (after the pointer publish — event rows are SQLite I/O, banned from publish): retire reconciles SCOPED, not via `ConfigLoad.finalize` (finalize calls `Store.resolveExcept`, events.zig:402, which resolves EVERY active configuration.load event outside its kept set — at runtime that would falsely resolve unrelated boot warnings). Each generation OWNS its report keys (stored in the generation), and the retire payload is STABLE: prepare copies the LIVE generation's report keys (under the owner mutex) into the PUT-owned apply state, so reconciliation never depends on the old generation's lifetime and NEVER runs from a query's `release` path — it runs on the PUT task in retire. Retire: emit the new report's events, then individually resolve exactly `copied_previous_keys new_report_keys`. S5 tests through the real PUT path: introducing a malformed upstream raises the warning; fixing it resolves it; an UNRELATED active configuration warning survives the whole introduce/fix cycle.
### S2.2 UpstreamOwner
Discipline: CertStore's (cert_store.zig:199/:237) — a mutex held briefly for refcounted borrows.
- `pub fn acquire(o: *Owner, io: std.Io) *Generation` — lock, ++refs, return.
- `pub fn release(o: *Owner, io: std.Io, g: *Generation)` — lock, --refs; a release that drops a RETIRED generation to zero deinits it (Upstreams.deinit needs io).
- `pub fn replace(o: *Owner, io: std.Io, prepared: *Generation) ?*Generation` — lock, swap the live pointer, mark old retired; if the old generation's refs are ALREADY zero, return it for the caller to retire immediately (the CertStore refs==0 pattern) — otherwise null and the last release retires it.
- `pub fn deinit(o: *Owner, io: std.Io)` — shutdown teardown of the live generation, called by serve() after listeners and metrics readers have stopped.
- Prepared generations are heap-allocated and OWN every configuration string (URL text — transport.Endpoint borrows it, transport.zig:51 — plus tls_name, host, path): each generation carries its own arena covering every borrowed row string until retirement. Candidates built from request-arena rows copy into that arena at prepare.
- Timeouts are baked into the generation; a timeout change is a replace.
- Query path: `Handler` stores `*Owner` (replacing the boot-time type-erased client, handler.zig:91). Per exchange: acquire → generation's `transport.Client` → exchange → copy the resolver identity from `selected` (borrowed from the endpoint, transport.zig:350) into the PER-QUERY `Context.upstream_buf` (handler.zig:453 — never a Handler-owned buffer; Handler is shared across concurrent queries) → release.
- Metrics/health: `WebState` stores `*Owner` (replacing `*Pool`, web/server.zig:144); a scrape acquires for its duration.
### S2.3 Cache and rate-limiter lock hygiene
- handler.zig:799 and app.zig:1124: cache pointer loads move inside `cache_mutex`; then `replaceCache` swaps under it (entries lost — accepted).
- handler.zig:189: same for the rate limiter; `replaceRateLimiter` under its lock (windows reset — accepted).
### S2 implementation notes (post-build)
- The retire-time scoped reconciler is DEFERRED TO S5 (S2 has no runtime replace caller; uncalled code fails the values). S2 delivers its precondition: generations own copyable report keys. **S5 must implement the reconciler + its tests (including unrelated-warning-survives).**
- `Generation` gains a test-only `borrowing`/`borrowingPool` payload so ~90 existing test sites keep their fake clients without heap generations; production always takes the built path.
- One skipped-upstream log string reworded to unify with the event detail; the event detail (what the criterion asserts) is byte-identical.
- Integration-gated call sites were mechanically touched (comptime-dead without -Dintegration; would have broken that build).
### S2.4 Acceptance criteria
- [ ] Concurrent exchanges vs `replace`: in-flight completes on G1; G1 deinits only after last release (or immediately when refs==0 at replace); new exchanges on G2; allocator clean.
- [ ] A replace while NO reader holds G1 retires G1 via the replace return path (refs==0 branch covered).
- [ ] Generation string ownership: build a candidate from a transient arena, free the arena, exchange still reads valid url/tls_name (allocator-poisoning test).
- [ ] Metrics scrape concurrent with replace: clean.
- [ ] Cache and rate-limiter swaps under concurrent use: clean.
- [ ] Boot diagnostics unchanged: the events ConfigLoad.note wrote before the extraction are written identically (assert on the events store, not stdout).
- [ ] Restart signals untouched this session.
- [ ] `zig build test` green.
---
## Session S3: in-place reconfigure, retention, certs, sink, scheduler
### S3.1 Sessions TTL
Slots gain `issued_at` beside `expires_at` (auth.zig:311). `setTtl(io, ttl)` recomputes each live slot: `expires_at = issued_at + ttl` under the existing mutex. Semantics are SERVER-SIDE: shortening takes effect for every session; lengthening is bounded by the browser cookie's original Max-Age (the cookie is not refreshed — sessions do not become sliding; state this in a comment). Login cookie Max-Age (handlers/auth.zig:127) reads the LIVE ttl.
### S3.2 ApiLimiter
ALL config reads move under the existing mutex (`check` reads `localhost_exempt` pre-lock, api_limiter.zig:135). `setLimits(io, now, limits)`: refill each bucket THROUGH `now` at the old rate, clamp tokens to the new capacity, then install the new rate — no retroactive refill at the new rate, and never move a bucket's clock backward (a bucket already newer than `now` is clamped only). SSE per-IP counts untouched.
### S3.3 Retention days
Two consumers (retention.zig:103, clients.zig:228): one shared `.monotonic` atomic u32 owned by app-level state, read per pass by both; `setRetentionDays` stores.
### S3.4 Certificate paths
`doh_server.cert_path/key_path`, `dot_server.cert_path/key_path` on an ENABLED endpoint: CertStore's paths (borrowed boot slices, reread by reload at cert_store.zig:111, read outside `mutex` by reload/pollOnce at :226/:268) become owned, replaceable strings. The whole apply is serialized under the store's existing `reload_mutex` — the SAME mutex reload holds across load+publication: prepare (under reload_mutex) loads the candidate cert+key from the new paths; publish (still under reload_mutex, taking the generation `mutex` only for the swap — lock order: reload_mutex outer, mutex inner, matching reload today) swaps paths and generation together. `pollOnce` currently reads the paths BEFORE taking reload_mutex (cert_store.zig:268): it now takes reload_mutex around its path reads and calls a non-locking `reloadLocked` helper (reload becomes reload_mutex-lock + `reloadLocked`) so acquisition is never recursive. With that, no concurrent path borrow can outlive an apply. Connections pinning the old cert finish on it. `POST /api/certs/reload` rereads the owned paths.
On a DISABLED endpoint no CertStore exists (handlers/certs.zig:31 — `doh_certs`/`dot_certs` are null): the change is DB-ONLY, and the key's table entry says so; the paths are validated when milestone 35 implements enable. This is stated in the table note and the API docs.
### S3.5 Log sink
Three disjoint cases, decided from the merged config — no reuse marker, no lock spanning prepare and publish:
- **File target changed** (the merged config has output=file AND the path differs from the current file target, or output switches TO file): prepare opens the NEW target (fallible, closing the TOCTOU in logging.zig:228's close-then-reopen) into an owned `PreparedSink` that carries the COMPLETE target state: the open file, its MEASURED length as the new `file_pos`, and `rotate_pending = false` (the handle couples to both fields, logging.zig:196 — inheriting the old position corrupts writes; inheriting a pending rotation rotates the new target spuriously). Publish installs `{file, file_pos, rotate_pending}` + config atomically under the sink's existing lock; the DETACHED old handle closes in retire. Races can only touch the OLD state, replaced wholesale.
- **File target removed** (output switches FROM file to stderr/syslog): nothing to prepare; publish installs `{file = null, file_pos = 0, rotate_pending = false}` + config atomically; the detached file closes in retire.
- **File target unchanged** (every remaining case: output stays stderr/syslog, or output=file with the same path — level, limits, or a file_path change while output is non-file): publish updates ONLY the config fields under the sink lock and NEVER touches the handle or its position/rotation state, which stay owned by the existing rotation/recovery machinery. A same-path config apply does not repair a broken handle (the existing per-write recovery does). No prepare-time handle inspection exists, so there is nothing to race. (A file_path change while output is stderr still updates the config so a later output=file switch — its own apply — opens the right target.)
The disk monitor's measured directory derives from the final merged `output + file_path` on EVERY log_sink apply: output `file` → the file's directory (installed even if only output changed); output stderr/syslog → cleared (monitor stops measuring a log dir). `Monitor.setLogDir(io, owned_path_or_null)`; `sample` borrows `log_dir_path` across directory I/O (disk_monitor.zig:112), so the path is a pinned generation — sample acquires (refcount or lock held for the borrow), setLogDir swaps, old path freed after the borrow releases. Owned path prepared pre-commit.
### S3.6 Wakeable blocklist scheduler
manager.zig:1493 restructured into a wakeable loop parked on a condition:
- The startup refresh pass runs exactly as today, including when disabled.
- `setSchedule(io, enabled, interval_hours)` stores under the scheduler's mutex with a version counter (no lost wakes between check and park) and signals.
- Anchor rule: the anchor is the completion time of the last refresh pass that RAN — success or failure both advance it; a disk-gate skip ALSO advances it (today's semantics: the scheduled slot is skipped, not retried early). Next refresh = anchor + interval; if that is in the past at set/enable time, refresh immediately.
- Disabled parks; the task exits only on shutdown, exactly as today.
- Production wake primitive: `std.Io.Event.waitTimeout` (Condition has no timed wait in 0.16). The Event is STICKY after `set` — the reset sequence prevents both spinning and lost wakes: under the scheduler's mutex the loop reads the version, RESETS the event, recomputes its deadline, releases the mutex, then waits; `setSchedule` (under the same mutex) bumps the version, then sets the event. A set that lands between the loop's reset and its wait completes the wait immediately; the version recheck decides whether anything changed.
- Test seam: an injectable clock/step seam — validated intervals are ≥ 1 h; acceptance tests must not sleep real time.
### S3 implementation notes (post-build)
- The disabled-endpoint cert criterion (DB-only, no store call, no-restart response) is PURELY a PUT concern — **deferred to S5's obligations** beside the S2 reconciler.
- `model.retentionSeconds` deleted (dead once both consumers read the shared cell); `RetentionDays.seconds()` is the single conversion point.
- A disabled scheduler PARKS; the seam models shutdown (`shutdown_at_first_park` → error.Canceled) — the loop's only exit is shutdown.
- `publishPathChange` reads no clock (loaded_at captured at prepare).
- `Monitor.deinit(io)` added to free an installed log-dir generation (no-op at boot — arena-borrowed).
- `setLimits` takes the full limiter `Config` (the three fields ARE the config; a twin struct would be invented generality).
- Sink race criteria proven as the two reachable interleavings (publish shares the stderr lock with rotation/closure — no third ordering exists); the monitor criterion is a real two-task race.
### S3.7 Acceptance criteria
- [ ] TTL: shortening expires an over-age live session immediately; lengthening extends server-side validity; a fresh login's cookie Max-Age reflects the live ttl.
- [ ] Limiter: no pre-lock config read remains; refill-through-now at old rate proven with a controlled clock; capacity cut clamps; SSE counts survive.
- [ ] Retention: both consumers observe a change on their next pass.
- [ ] Certs: bad candidate refused at prepare, store untouched; good change serves the new cert on the next handshake (-Dintegration loopback); a concurrent reload during an apply is serialized (test drives both under the seam).
- [ ] Disabled-endpoint cert change: DB row changes, no store call, response marks no restart.
- [ ] Sink: publish does no open/close (close observed in retire); bad path refused at prepare; new file receives lines; monitor samples the new dir; no use-after-free under concurrent sample; a same-path config-only apply concurrent with a forced rotation AND with a write-failure closure changes only config fields, never the handle; a target-change apply racing rotation swaps cleanly (old handle closed in retire); switching to a PRE-EXISTING nonempty file starts at its measured length; switching away from a target with `rotate_pending` set does not rotate the new target; a `file → stderr` apply detaches the handle (closed in retire) and clears position/rotation state.
- [ ] Scheduler via the seam: shortened interval → next at new cadence; disable parks; re-enable anchors per rule; startup pass runs when disabled; failed pass advances the anchor; gate-skip advances the anchor.
- [ ] `zig build test` green.
---
## Session S4: logger queue controller
### S4.1 Controller scope
`logging.query_log_buffer_max`. A stable controller owns the logger generation: buffer, `Logger`, writer future, and the writer's DB/gate dependencies (writer holds prepared statements for its run, logger.zig:514; future owned by serve today, app.zig:897). serve() creates the controller and delegates; shutdown ordering through the controller is EXACTLY today's safe order — quiesce producers → close queue / set draining → the writer drains the CLOSED queue → await the future (logger.zig:632, app.zig:921; never "drain then close": an empty writer blocks in getOne until close).
Facade: `QuerySink` (query_sink.zig:18), metrics (web/metrics.zig:207) and health (web/handlers/health.zig:241, via web/server.zig:150) hold the CONTROLLER, not `*Logger`. The controller owns what must be continuous across swaps: counters, gate-episode state, `last_drop_s`, diagnostics references, AND the S1 privacy/flush atomics (setters address the controller and survive resize). `writer_failed` reflects the LIVE writer: a successful resize publish clears it (the new writer prepared cleanly); a retired writer's failures still land in the drop/diagnostic counters.
Producer read-side protocol: a producer's `log()` acquires the live generation through the facade with a refcount (or a lock held through transform + enqueue) — a producer can never enqueue into a queue that retire has closed; retire waits for old-generation producer borrows to release before closing the old queue. This is what makes "no resize drop window" true, not an aspiration.
### S4.2 Resize — all fallibility before commit
1. **Prepare** (fallible): validate against the comptime ceiling (37449, startup's message); allocate the new buffer and Logger state; OPEN A NEW querylog DB connection for the new generation — one connection per writer for its whole life (logger.zig:514); two writers must never share `querylog_writer_db`. The connection factory: the canonical writer-connection opener (today `cli.DataDir.reopenQuerylogDb` + `db.applyPragmas`, cli.zig:350) MOVES into `src/storage` (a pub fn beside `querylog_schema.open` taking the data-dir handle + path); cli.zig delegates to it, and the controller receives the factory inputs at construction. The controller owns each generation's connection and closes it after that generation's writer is joined — and SPAWN the new writer PARKED: the spawn (`io.concurrent` can fail, Io.zig:2352) and statement preparation (runWriter can fail, logger.zig:524) happen NOW; the parked writer signals ready and waits on an activation gate. Any failure: tear down, close the new connection; nothing changed. If a PREVIOUS resize's retired generation has not finished retiring, prepare REFUSES with a distinct error ("previous resize still draining") — at most one retired generation exists, which bounds writers, connections, and buffers.
2. **Commit** the DB row.
3. **Publish** (infallible): swap the facade's live-generation pointer and open the activation gate. No waiting.
4. **Retire** (controller-owned reaper): wait for old-producer borrows to release → close the old queue WITHOUT setting the process-shutdown `draining` flag (a distinct retirement mode: with `draining` set, a gate-held writer takes the GatedAtShutdown path and DROPS its batch, logger.zig:642/:726 — retirement instead lets it flush when the gate reopens) → the old writer drains the closed queue → await its future → close its DB connection → free logger + buffer.
Reaper ownership: the reaper task is spawned ONCE at controller construction (boot — fallibility there is fine) and lives for the controller's life, processing retirements signaled by publish; publish never spawns. `Future.await` is not thread-safe (Io.zig:1198), so the reaper is the SOLE joiner of retired writers. Process shutdown, in order: quiesce producers → CLOSE the live queue (a writer blocked in getOne wakes only on close, logger.zig:642) → set `draining` on EVERY outstanding generation (live and retired — turning a gate-parked retired writer onto the counted GatedAtShutdown path) → join the LIVE writer → signal and join the REAPER (which finishes joining any retired writer). The f1a85d3 ordering holds within each join.
Accounting: every entry accepted into the old queue is written when the gate allows, or counted by the existing gate/shutdown machinery; entries after publish land in the new queue. No resize-specific drop path exists.
### S4 implementation notes (post-build)
- The controller is a new file, `src/storage/logger_controller.zig`, not a second half of `logger.zig` (2190 lines before this session, and generations/retirement/the reaper are a separate concern from `Entry` and the writer loop). Add it to the module layout.
- `Logger` keeps its exact public surface, so every f1a85d3 test passes unchanged. Continuity is achieved by SUMMING rather than by sharing cells: `Controller.sample` folds `base` (finals of joined generations) + the live generation + the outstanding retired one, under the controller mutex, and folds a retired generation's finals into `base` before freeing it. `last_drop_s` is a max, the gate episode is the worst of the outstanding generations, `writer_failed` is the live one's.
- `Logger.runWriter` is split: it prepares its statements and calls the new `pub fn runPrepared(io, writer, monitor)`. A resize generation prepares separately so a statement failure is a refused settings change. `Logger.retire(io)` is the close-without-`draining` retirement mode.
- The BOOT generation deliberately takes the plain `runWriter` path, not the parked one: a boot that cannot prepare must still serve DNS with `writer_failed` set, which is today's behaviour. Only a resize can afford to refuse.
- The connection factory is `querylog_schema.reopen(io, dir, path)`; `cli.DataDir.reopenQuerylogDb` delegates. It takes the dir handle to make `open`'s resolution pairing explicit at every call site, and does not dereference it.
- The setters are `Controller.setPrivacy(io, p)` / `setFlushInterval(io, s)` and take `io` (they lock), unlike S1's pure-atomic pair. They store to every outstanding generation; each generation still keeps ONE packed privacy word, so a producer's single load stays the mixed-policy guarantee.
- `Controller.prepare` returns typed errors; `sizeMessage(err, requested, buf)` renders `config/validate.zig`'s exact wording for S5's client error.
- `Controller.retirementPending(io)` is the "previous resize still draining" predicate, exposed for S5 and the tests.
- Test sites keep their stack `Logger`s through `logger_controller.Borrowed` — S2's `upstream_owner.Borrowed` pattern.
- **S5 obligations unchanged**: the S2 retire-time scoped reconciler and the S3 disabled-endpoint cert criterion. S4 adds none.
### S4.3 Acceptance criteria
- [ ] Resize under forced concurrent producers: no deadlock; exact accounting — old-queue entries all written (or gate-counted), new-queue entries written, produced == written + counted; no resize-specific drops.
- [ ] Resize with the disk gate CLOSED: publish completes immediately; old writer retires after the gate reopens; nothing lost beyond what the gate itself counts.
- [ ] Prepare failure (spawn or statement prep) leaves the running logger untouched and writes no DB row.
- [ ] Invalid size refused with startup's message.
- [ ] Counters/gate state/`last_drop_s` continuous across a swap (metrics read before and after agree modulo new writes).
- [ ] Privacy interleaving test pauses a producer between its two field decisions across a concurrent `setPrivacy`: a mixed policy (domain redacted, client exposed, or the reverse) is impossible.
- [ ] Shutdown while a retirement is gate-blocked: clean join, the retired writer's held batch is counted by GatedAtShutdown, no deadlock, no double-await.
- [ ] f1a85d3 deadlock regression tests pass unchanged.
- [ ] `zig build test` green.
---
## Session S5: the apply table and the settings write path
### S5.1 The key table
`src/web/handlers/settings.zig`: delete the comptime `restart_required_keys` generation (:68-95) and `touchesRestartRequiredKey` (:187-200). The table maps EVERY settings key to a CONCRETE operation enum — `dns_policy`, `trusted_proxies`, `logger_privacy`, `logger_flush`, `disk_thresholds`, `upstream_generation`, `cache`, `rate_limiter`, `sessions_ttl`, `api_limiter`, `retention`, `certs_doh`, `certs_dot`, `log_sink`, `scheduler`, `logger_queue`, `bind`, `web_lifecycle` — one entry per key, no second unguarded switch; the broad class (live/subsystem/bind/web_lifecycle) is DERIVED from the operation. `bind` (dns/web/doh/dot bind + port + doh/dot enabled) and `web_lifecycle` (`web.enabled`) set `restart_pending` exactly as today; milestone 35 executes them.
Comptime test: walk `@typeInfo(model.Config)` as the old generator did; every scalar key appears exactly once (replaces :549).
### S5.2 The PUT flow
validate → group changed keys by OPERATION → prepare one candidate per affected owner from the final merged config → any failure cleans up all prepared candidates, errors, no DB write → commit (failure: clean up) → publish each prepared candidate → retire. `web.password` keeps its existing path. Upstream create/update/delete (handlers/upstreams.zig): build the candidate generation from the HYPOTHETICAL post-mutation row set before the repository write; commit; publish. The direct `restart_pending` sets (:58/:90/:114) are deleted HERE. `restart_required` in upstream responses (upstreams.zig:156) becomes `false`; update the OpenAPI `UpstreamEcho` pin (openapi.yaml:2749), the `/api/settings` description that says every scalar setting requires restart (openapi.yaml:1699), the API reference claim that all upstream/settings mutations do (docs/reference/api.md:18), generated types, and contract samples.
### S5.3 Config status
`GET /api/config/status`: `restart_pending` remains, fed only by `bind`/`web_lifecycle` operations. The settings envelope's `restart_required` list (:538) shrinks to those keys.
### S5 implementation notes (post-build)
- The table and the four phases live in a new file, `src/web/handlers/apply.zig`, not in `settings.zig`: the upstream RESOURCE handlers need the same prepare/publish/retire, so putting it in the settings handler would have made one handler the other's library. `settings.zig` re-exports `restart_required_keys` and nothing else of it. Add it to the module layout.
- Operations are derived from the VALUES, not from the keys the patch named (`changedOperations(before, after)`). The admin form submits every field it read, and treating that as eighteen applies would resize the query log on every save — and refuse the second save with `PreviousResizeDraining`. A key rewritten to what it already was changes nothing, applies nothing, and owes no restart.
- `app.zig` now heap-allocates the DNS cache and rate limiter and frees them THROUGH the handler (`replaceCache(io, null)` at teardown): after a `cache.size` apply, what the handler holds is not what boot built. Both are built inside one block whose errdefers end with it, so a boot that fails between the two creations frees them and a boot that fails later leaves them to the handler's teardown defers.
- The OpenAPI overview and the three upstream mutation descriptions said restart-required. They now say the change is live, which is what the runtime does and what the `restart_required: false` pin already promised. `docs/reference/api.md` was already correct. `contractSamples.gen.ts` is generated from live responses rather than from the document, so no regeneration followed.
- `WebState` gains `upstream_build` (the shared HTTP client, certificate bundle and its lock) and `retention_days`. Without the first, an upstream mutation is a database row and nothing else, which is what a handler test's borrowed owner wants.
- `Owner.published` is a plain `u64` under the owner mutex, with `publishedCount(io)`, not an atomic: it is the observable the "one candidate per owner" tests read. An atomic counter in the same struct reproducibly perturbed the S2 test "a replace with no reader holding the live generation retires it through the return path" into failing (~1 run in 1); a mutex-guarded field is both the discipline the rest of `Owner` follows and stable. **That S2 test's sensitivity to unrelated timing is worth a look on its own.**
- `mutations.Resource` now accepts a `remove` that returns `error{OutOfMemory}!?Failure`, because a delete that builds a candidate allocates. Both shapes are still checked exactly.
- The upstream note rendering moved to `upstream_owner.Rendered`, shared by `app.zig`'s boot replay and the runtime reconciler, so a warning raised at boot and the same warning raised by a write are byte-identical.
- Reconciliation reads the new report by RE-ACQUIRING the owner rather than keeping the published pointer: a concurrent second write could retire and free that generation the moment its refs hit zero.
- The S2 reconciler test drives its rebuilds through `/api/upstreams`, not `/api/settings`: a settings PUT validates the whole stored configuration and refuses a row malformed enough to produce a build finding, so the finding can never be reached from there.
- `logging.file_path` must be absolute, so the integration environment resolves its tmp dir with `Dir.realPath`.
- `Plan.prepare` abandons the partly built plan itself on a PROPAGATED error (`errdefer self.abandon(io)`), and only returns a `Failure` for the caller to abandon. An `OutOfMemory` out of the log-directory step used to bypass the caller's `abandon`, leaking whatever earlier owners had built and — because `CertStore` holds `reload_mutex` from its prepare to its publish — wedging every later certificate change and reload.
- `Controller.prepare` takes the merged `model.Logging`, not an entry count. Seeding the candidate from the LIVE privacy and flush values was wrong for the one PUT that changes the buffer size and a privacy flag together: publish applies the privacy to the generation being retired, so the replacement went live writing what the operator had just asked to hide.
- The `api_limiter` reading is taken in prepare (`Plan.limiter_now`) and used in publish. A clock is I/O, and publish is I/O-free. That reading is STALE by publish time whenever a request served in between refilled a bucket past it, so `setLimits` is monotonic per bucket: a bucket already newer than `now` is clamped to the new capacity and keeps its clock. Rewinding it would let the next request buy the same interval a second time, at the new rate.
- `metrics.collect` loads `handler.cache` and `handler.limiter` INSIDE the mutexes that guard them, the same way the request path does: `Plan.retire` frees the displaced object as soon as the swap returns, so a pointer read before the lock can be freed under the reader.
- `CertStore.publishPathChange` frees the displaced entry and the old paths BEFORE bumping `reloads`. A `bool` live across an atomic read-modify-write is the Debug-backend miscompile AGENTS.md documents.
- Two S5.4 criteria are proven one step short of the wording. `certs_doh`/`certs_dot` assert the store reloaded and the owned paths moved, not a TLS handshake against the new certificate; `dns_policy` asserts `policySnapshot`, which is the read a query performs, not a query. Both are the collaborator's own observable, and neither gap is a claim about a path that is untested.
### S5.4 Acceptance criteria — one real-PUT test per OPERATION
Every operation proven through the real route dispatch → prepare → commit → publish path:
- [ ] dns_policy (blocking.ttl live on next query — integration), trusted_proxies, logger_privacy, logger_flush, disk_thresholds, cache (size change swaps, next lookup misses), rate_limiter, sessions_ttl, api_limiter, retention, certs_doh AND certs_dot separately (enabled: handshake serves new cert — integration; disabled: DB-only), log_sink (+ monitor re-point on path change AND on output change both directions), scheduler (via seam), logger_queue (accounting per S4), upstream_generation (row mutation: next exchange uses it, response `restart_required: false`; timeout change: generation counter +1).
- [ ] web_lifecycle: a `web.enabled` PUT commits the DB row, sets `restart_pending`, and executes NOTHING (milestone 35 executes it).
- [ ] A multi-key PUT touching one owner twice builds ONE candidate (generation counter delta == 1).
- [ ] An upstream PUT while G1 is held by an in-flight exchange: publish + diagnostics reconciliation complete on the PUT task; G1 retires later via release; events correct throughout.
- [ ] A PUT mixing a live key with a failing prepare writes NOTHING and changes nothing.
- [ ] A `bind` key PUT still sets `restart_pending`; nothing else can.
- [ ] Contract samples for `/api/config/status`, settings envelope, upstream responses regenerate; route-count pin passes.
- [ ] `zig build test` and `zig build test -Dintegration` green.
---
## Session S6: admin SPA
- Per-field "needs restart" tags (SettingsForm.tsx:119-123/:209-223/:267) render only for keys the API lists (bind + web_lifecycle). Banner logic untouched. Upstream flows lose restart messaging; regenerated types carry `restart_required: false`. Live fields say nothing — silence is success. Regenerate goldens (`just goldens`).
- [ ] From `admin/`: `npm run typecheck && npm test && npm run lint && npm run format:check && npm run build && npm run assert-bundled` green; bundle under ceiling.
- [ ] A test: live-key edit renders no restart affordance; a port edit still does.
### S6 implementation notes (post-build)
- `SettingsForm` needed no change at all. It already built its mark set from `envelope.restart_required` and nothing else, so the twelve-key list arrived and the other fields went quiet on their own. The session's real work was the copy and the tests that had pinned the old answer.
- The tests now take the key list from `sample_get_settings.restart_required` in the committed contract sample, re-exported as `RESTART_REQUIRED_KEYS` from the configuration fixtures, rather than from a hand-written array. A server-side change to the set now fails the tests that pin it instead of passing against a stale copy.
- One test keeps a hand-written list, `["logging.level"]`: no shipped restart-required key is enum-backed, so the `Select` rendering — where the mark reaches a screen reader through `aria-describedby`, not through the label — has no key left to exercise it. The list is the server's to change and the form must render any key it names, so the branch and its test stay, with the override stated in the test.
- `invalidateUpstreams` dropped its `/api/config/status` invalidation. An upstream write rebuilds the pool in the running process, so the status answer after the write is the answer before it, and re-reading it only asserted a claim that is no longer true.
- `just goldens` was not run: S5 regenerated the samples and this session changed no server response.
---
## Module layout (new/changed)
`src/upstream/owner.zig` (new); `src/storage/logger_controller.zig` (new); `src/storage/querylog_schema.zig`; `src/cli.zig`; `src/server/handler.zig`; `src/server/query_sink.zig`; `src/web/server.zig`; `src/storage/logger.zig`; `src/storage/disk_monitor.zig`; `src/storage/retention.zig`; `src/server/clients.zig`; `src/platform/logging.zig`; `src/filter/manager.zig`; `src/server/cert_store.zig`; `src/web/auth.zig` + `src/web/handlers/auth.zig`; `src/web/api_limiter.zig`; `src/web/handlers/apply.zig` (new); `src/web/handlers/settings.zig`; `src/web/handlers/upstreams.zig`; `src/web/handlers/certs.zig`; `src/web/metrics.zig`; `src/web/handlers/health.zig`; `src/web/openapi.yaml`; `docs/reference/api.md`; `src/app.zig`; `admin/src/features/configuration/*`.
## File ownership
Strictly sequential; each session owns the tree while it runs.
## Acceptance criteria (milestone complete)
- [ ] Every settings key except `bind`/`web_lifecycle` applies live through its table-declared operation, each proven by a real-PUT test (S5.4).
- [ ] `restart_pending` only from `bind`/`web_lifecycle`; no API response or doc claims a restart for anything else.
- [ ] No fallible or I/O work in any publish; every prepare/commit failure leaves DB and runtime unchanged.
- [ ] Full gates: `zig build test`, `zig build test -Dintegration`, admin suite from `admin/` incl. `assert-bundled`, `zig fmt --check build.zig src tools`, bundle ceiling.
- [ ] Querylog schema fingerprint unchanged (cut's schema-gate takes the equal branch).
## Anti-requirements
- No generic hot-swap framework; one named operation per owner.
- No listener rebind, no `web.enabled` execution, no `applied: false` surface, no `restart_pending` removal — milestone 35.
- No config file watcher; file mode unchanged. No new config keys; no DB schema changes. Sessions do not become sliding.
+190
View File
@@ -0,0 +1,190 @@
# Milestone 35: visx charts
Replace the hand-rolled chart layout math in the admin Overview with visx primitives, and unify the chart components' duplicated plumbing. Owner decision 2026-08-24 after a measured bake-off (recharts +377,355 pre-gzip bytes, nivo +332,686, visx +74,382 against ~103,000 bytes of budget room; visx is the only candidate that fits the 800,000-byte gate). The charts must read as the same charts afterward: same palette, same geometry, same accessible surfaces.
## Sessions
One session. Admin SPA plus the license records; no Zig source, no API changes.
---
## Session 1: port the Overview charts to visx
### 1.1 Dependencies and records
Add to `admin/package.json` under `dependencies` (NOT devDependencies), pinned exact (no `^`): `@visx/scale@4.0.0`, `@visx/shape@4.0.0`, `@visx/group@4.0.0`, `@visx/axis@4.0.0`, `@visx/grid@4.0.0`, `@visx/tooltip@4.0.0`. Nothing else — in particular NOT `@visx/responsive` (the app keeps its own resize hook) and NOT `@visx/text`, `@visx/legend`, `@visx/xychart`.
Three separate record obligations, each updated from its own evidence, not from this spec's package list:
1. The sourcemap-derived bundled set that `npm run assert-bundled` checks — regenerate/extend it from the actual build output (expect the transitive `@visx/{bounds,curve,point,text,vendor}` to appear).
2. The lockfile-derived npm runtime closure that the Zig drift gate checks — update from the lockfile.
3. `licenses/inventory.zon` + `licenses/dependency-identity.txt` + license-text files under `licenses/` for every shipped package. The direct `@visx` packages are MIT; `@visx/vendor` is `MIT AND ISC` (it re-exports ISC-licensed d3 modules) — record that exact expression and include the ISC text.
### 1.2 Shared chart plumbing — new `admin/src/features/overview/chartKit.tsx`
Owns what is today duplicated or divergent between `TimeseriesChart.tsx` and `ClientChart.tsx`:
- `useMeasuredWidth()` — extracted once from the two near-identical hooks. Contract, matching both current hooks: reads `clientWidth` in the mount `useEffect` (NOT `useLayoutEffect` — keep the current timing), observes subsequent changes via `ResizeObserver`, disconnects on cleanup, and falls back to 640 when the measured width is 0 (jsdom). Covered by a test for the zero-width fallback.
- A y-scale factory: exported function returning `scaleLinear({ domain: [0, max], range, nice: true })`. The `[0, max]` values in 1.3 are INPUT data domains; nicening mutates them. Consumers pass `numTicks={5}` to `AxisLeft` as a hint. Kit test, exact: input domain `[0, 1780]`, range `[240, 0]` → effective domain `[0, 1800]` and `scale.ticks(5)` equal to `[0, 500, 1000, 1500]`.
- Axis wrappers over `AxisBottom`/`AxisLeft` applying the app's tick text styles. Current layout constants are preserved exactly unless named here: chart height 240, margins `{top: 8, right: 8, bottom: 22, left: 44}`. Axis chrome matches today's rendering, not visx defaults: `AxisLeft` hides its axis line and tick marks entirely; `AxisBottom` keeps only the existing baseline — no tick marks. Y labels keep the existing compact-number formatting; x labels keep the existing time formatter driven by `bucket_seconds`. X-label thinning preserves the current algorithm verbatim: `step = max(1, ceil(bucketCount * 90 / plotWidth))`, ticks at indices where `i % step === 0` — kit test with this exact fixture: 168 hourly buckets (ts = 0, 3600, ...) at `plotWidth = 748``step = 21`, tick indices `[0, 21, 42, 63, 84, 105, 126, 147]`, tick timestamps `[0, 75600, 151200, 226800, 302400, 378000, 453600, 529200]`. Exact axis attributes (in the spec, not implementer-chosen): `AxisLeft hideAxisLine hideTicks` with tick labels 6px left of the plot edge (current position); `AxisBottom hideTicks tickLength={0}` with `tickLabelProps` `dy="16px"` and `textAnchor="middle"` — the sanctioned baseline+14 → baseline+16 change. Kit test asserts the axis group transform plus the label `y`/`dy` attributes that produce baseline + 16px, and the y-label -6px offset.
- `GridRows` and `AxisLeft` MUST share tick positions: compute `yScale.ticks(5)` once and pass the same array as `tickValues` to both (each labeled tick owns its grid line, as today). Test: grid-line y positions equal labeled-tick y positions.
- `GridRows`, horizontal only, stroked with the app's existing grid color token.
- One tooltip: `useTooltip` + `TooltipWithBounds`, app tokens, no transition. Content contract for both bar charts: the bucket's formatted time, the total, and one row per displayed series with its label, swatch color, and value.
### 1.3 The three charts
Port in place; file names, props, and data contracts unchanged, with one exception below (`DonutSlice`). Parents and existing tests keep working.
- `TimeseriesChart.tsx``scaleBand` (x) + the kit y-scale + `BarStack` for blocked/cached/other. Y domain: `[0, max(bucket.queries)]`, and `other = max(0, queries - blocked - cached)` per bucket (test the `blocked + cached > queries` clamp). Band gap: the current chart draws a fixed ~2px gap regardless of bucket count; a constant `paddingInner` cannot do that, so compute it per render: `paddingInner = min(0.5, 2 * bucketCount / plotWidth)`. Segment separator: 1px stroke of the `colors.surface` token, applied only when the bar is wider than 3px (current behavior, TimeseriesChart.tsx:249). Keep the empty-state ("No queries in this period.") exactly; coverage messaging is OWNED BY OverviewPage (via useOverviewWindow), not this component — do not add coverage logic here.
- `ClientChart.tsx` — same band/linear scales and `BarStack`, per-client series plus Other. Y domain: `[0, max over buckets of (sum of all client values + other)]` (test it). X domain derived independently from its own props — `data.since + index * data.bucket_seconds` for `data.other.length` buckets — which is equivalent to the timeseries' domain from `buckets[].ts` because the API aligns them; do NOT add cross-chart props or page-level scale coordination. Identical margins/ranges to the timeseries. Gains the shared tooltip and dimming (below).
- `Donut.tsx``@visx/shape`'s `Pie` for arc generation, our own `<path>` emission. Pie config pins the deleted donutLayout contract: filter `value <= 0` slices before the Pie, sorting disabled (preserve caller order), no pad angle, clockwise from twelve o'clock, outer radius 90, inner radius 54, shares computed from the drawn positive total, and the single-positive-slice full ring must render. Keep the 1px `colors.surfaceRaised` arc outline (an existing test pins it). `DonutSlice` moves: `export interface DonutSlice` from `Donut.tsx`, and `OverviewPage.tsx:28` imports it from `./Donut`.
Interaction contract (both bar charts, identical): a full-plot-height transparent hit target per bucket; hovered bucket shows the shared tooltip; non-active buckets dim to opacity 0.55; tooltip and dimming clear on pointer leave; ALL bare SVG `<title>` hover text is removed from BOTH charts (the timeseries currently has both a tooltip and `<title>` at TimeseriesChart.tsx:269 — the `<title>`s go).
Accessibility contract (matches current tests, do not "improve" it): the two bar-chart SVGs KEEP `role="img"` and their `aria-label` (asserted in TimeseriesChart.test.tsx:32 and OverviewPage.test.tsx:382), with the visually-hidden tables as the detailed equivalent; only the donut SVG is `aria-hidden` + `focusable="false"` with the visible legend and hidden table as its surface.
Color contract: data-series colors are exempt from the token rule, and each chart keeps ITS existing source — they are not unified onto one mapping. TimeseriesChart keeps its fixed category constants exactly as today (including blue `#3b82f6` for Other — NOT `seriesColor(OTHER_KEY)`, which is gray). ClientChart uses `seriesColor(clientKey)` / `seriesColor(OTHER_KEY)`. Donut renders `fill={slice.color}` from its prop contract — never recomputed from `slice.key`; a page-level test separately asserts `OverviewPage` builds slice colors with `seriesColor(...)`. Never rank/order-based anywhere. Everything non-data (axes, grid, text, tooltip chrome, separators) uses StyleX tokens; light/dark themes keep working. No animation anywhere.
### 1.4 Deletions
`chartLayout.ts`, `chartLayout.test.ts`, `donutLayout.ts`, `donutLayout.test.ts` are deleted. Their SEMANTIC contracts do not die with them — they move (see 1.5). Only assertions about hand-written path/coordinate output are dropped. Any still-needed helper with no visx equivalent moves into `chartKit.tsx`; nothing imports the deleted modules afterward; no dead exports kept "just in case".
### 1.5 Tests
Current reality (do not assume more): the only dedicated chart component suite is `TimeseriesChart.test.tsx`; ClientChart and Donut are covered via `OverviewPage.test.tsx`; identity colors are pinned on the pure mapping and a legend swatch, not on rendered fills.
- Preserve and port: `TimeseriesChart.test.tsx`, the page-level suites (`OverviewPage.test.tsx` incl. the one-coverage-notice test), the color-mapping tests.
- New dedicated suites for `ClientChart` and `Donut`.
- Re-home the deleted layout contracts: `other = max(0, queries - blocked - cached)` and zero-data handling (timeseries suite); label thinning (kit suite); zero-slice filtering and caller order may be asserted on Pie inputs/config, but twelve-o'clock clockwise start and the single-positive-slice full ring MUST be asserted on the rendered non-empty `<path>` `d` output (donut suite). Shares: a dedicated donut test with values `3` and `1` asserts rendered shares `75.0%` and `25.0%` in both the legend and the hidden table.
- New: rendered `<rect>`/`<path>` fills for a fixed input match each chart's color contract from 1.3 (timeseries constants, ClientChart `seriesColor`, donut `slice.color`).
- New: both bar charts show the shared tooltip with the 1.2 content contract on simulated pointer events, dim inactive buckets to 0.55, clear on pointer leave; neither chart contains an SVG `<title>`. The pointer test MUST target the transparent per-bucket overlay rect (not a visible segment) and assert the overlay's `y`/`height` span the full plot height, including the space above a short stack.
- New: kit tests — zero-width fallback of `useMeasuredWidth`, the y-scale factory's exact domain and ticks for max=1780, the pinned x-label `dy`.
### 1.6 Acceptance criteria
Run from `admin/` with npm resolved via mise (`PATH="$HOME/.local/share/mise/shims:$PATH"` or `mise exec -- npm ...`):
- [ ] `npm test` exit 0
- [ ] `npm run typecheck`, `npm run lint`, `npm run format:check` exit 0
- [ ] `npm run build` exit 0; bundle stays under the unchanged 800,000-byte gate (expected ≈ 770,000)
- [ ] `npm run assert-bundled` exit 0 with the updated ledger
- [ ] `! rg 'chartLayout|donutLayout' admin/src` succeeds, and `test ! -e admin/src/features/overview/chartLayout.ts && test ! -e admin/src/features/overview/chartLayout.test.ts && test ! -e admin/src/features/overview/donutLayout.ts && test ! -e admin/src/features/overview/donutLayout.test.ts` succeeds
- [ ] `zig build test` exit 0 from the repo root (no-breakage check; the license/ledger updates are inside it)
## File Ownership
Session 1 owns: `admin/package.json`, `admin/package-lock.json`, the bundled-set and runtime-closure ledgers, `licenses/inventory.zon`, `licenses/dependency-identity.txt`, new license-text files under `licenses/`, `admin/src/features/overview/*` (TimeseriesChart, ClientChart, Donut, their tests, chartKit new, chartLayout/donutLayout + tests deleted), the one-line `DonutSlice` import in `OverviewPage.tsx`, `specs/milestone-35.md` (implementation notes).
## Anti-Requirements
- No direct dependency on, or application import from, `@visx/responsive`, `@visx/legend`, `@visx/text`, `@visx/xychart` (primitives only). `@visx/text` arriving transitively through `@visx/axis` is expected and fine.
- No animation, no react-spring, no transition on the tooltip.
- No new chart types, no visual redesign beyond the sanctioned x-label `dy` fix and the band-gap formula. The charts should read as the same charts.
- No budget raise; no changes to `assert-bundle-size.mjs`.
- No accessibility "upgrades" beyond the stated contract (bar SVGs keep `role="img"`).
- No Zig source changes, no API changes, nothing outside `admin/`, `licenses/`, and this spec.
### Session 1 implementation notes (post-build)
- Bundle: `admin/dist/assets` is **776,477 bytes** across 40 files, 23,523 under the unchanged 800,000-byte gate. `assert-bundle-size.mjs` untouched.
- `DonutSlice` now lives in `Donut.tsx`; `layoutDonut`/`layoutTimeseries`/`layoutStacked`/`niceTicks`/`isEmptyTimeseries` are gone, not re-homed. The empty-timeseries check is one `every` in `TimeseriesChart`; tick generation is `scale.ticks(5)`.
- `chartKit.tsx` exports `plotArea`, `useMeasuredWidth`, `valueScale`/`valueTicks`, `bandScale`/`bandPaddingInner`, `labelTickValues`, `formatBucketTime`, `EmptyChart`, `ChartRoot`, `ChartFrame`, `StackSegment`, `BucketOverlay`, `slotCenter`, `useActiveBucket`, `ChartTooltip`. The two charts share all of it; nothing is exported that only one caller uses.
- **Divergence — the bundled set.** 1.1 predicted `@visx/{bounds,curve,point,text,vendor}` transitively. The sourcemap build shows `@visx/{bounds,point,text}` but NOT `@visx/curve` (nothing here draws a curve) and NOT `@visx/vendor`: it is a re-export shim and rollup resolves straight through it to the d3 packages. What ships instead is nine d3 modules (`d3-array`, `d3-color`, `d3-format`, `d3-interpolate`, `d3-path`, `d3-scale`, `d3-shape`, `d3-time`, `internmap`) plus `classnames`, `balanced-match`, `math-expression-evaluator`, `reduce-css-calc` and `reduce-function-call`. The ledger was regenerated from that evidence, as 1.1 directs.
- **Divergence — a Zig source change was unavoidable.** The anti-requirement forbids Zig changes, but record obligation 2 is enforced by `src/licenses_drift_test.zig`, and `zig build test` (1.6) cannot pass without it: the nine shipped d3 modules are ISC and `robust-predicates` is Unlicense, so `npm_licence_exceptions` needs their entries, and the 23 newly-in-closure packages that ship nothing need `npm_not_shipped` entries. Only those two tables changed; no production Zig was touched.
- **ISC is new to this project.** `licenses/d3-isc.txt` reproduces all nine copyright notices above the single permission text they share, and the inventory entry records the acceptance as Mokhtar Mial, 2026-08-24, on this milestone's ruling. That acceptance is inferred from this spec ordering the ISC text to be carried; confirm it.
- **Known gap in the drift guard.** `parseRecordedPackage` reads exactly three whitespace-separated tokens, so the lockfile's `@visx/vendor 4.0.0 MIT and ISC` is checked as `MIT` and its ISC half passes unreviewed. `@visx/vendor` ships nothing today, so nothing turns on it; it is written down rather than fixed because fixing it is a guard change, not this milestone.
- Fourteen `@types/*` packages and `csstype` are in the npm *runtime* closure now, not because anything changed about them but because the `@visx` packages declare `@types/react` and `@types/d3-*` as ordinary `dependencies`. They emit no runtime code and are recorded as not shipped.
- Axis geometry: `AxisBottom` also takes `tickLength={0}` (1.2 named it) and `AxisLeft` takes it too, which 1.2 did not — without it visx offsets the value labels by its default 8px tick length and the spec's "6px left of the plot edge" is unreachable. `AxisLeft` further takes `dy: 0` to cancel visx's own `0.25em` nudge, which would double up with the `dominantBaseline: "middle"` the current chart centres its labels with.
- Both axes render tick labels through a `tickComponent` rather than visx's `<Text>`. `Ticks` derives a label's `y` from a *guessed* font size (`Math.max(10, …)`) because the real size comes from a StyleX class it cannot read, and `<Text>` wraps every label in a nested `<svg>`. The custom component drops the guessed `y` and places the label with the axis group transform plus `dy`, which is what makes `dy="16px"` mean baseline + 16px exactly.
- `Pie` skips its own `top`/`left` group when given a render prop, so `Donut` centres the ring in a `<Group>` of its own. The old `fillRule="evenodd"` is gone: d3's arc paths are correctly wound and no longer need it.
- Band gap: `paddingInner = min(0.5, 2 * n / plotWidth)` yields a gap of `step - bandwidth` ≈ 2.006px rather than exactly 2px, because d3 derives `step` from `n - paddingInner`. Bars also start 1px left of where the hand-rolled layout put them (it centred a `slotWidth - 2` bar inside the slot; a band scale left-aligns). Both are sub-pixel-scale and the charts read the same.
- Tooltip: `TooltipWithBounds` drops its own positioning transform when `unstyled` is set, so the default look is replaced by passing an empty `style` object and the app's StyleX class instead.
- Tests: **507** pass across 49 files (was 458 across 46). New suites `chartKit.test.tsx` (10), `ClientChart.test.tsx` (15), `Donut.test.tsx` (9); `TimeseriesChart.test.tsx` grew from 2 to 16; `OverviewPage.test.tsx` gained the donut-slice-colour test. The counts above 493 came from the two review passes.
- Gate exit codes, all from `admin/`: `npm test` 0, `npm run typecheck` 0, `npm run lint` 0, `npm run format:check` 0, `npm run build` 0, `npm run assert-bundled` 0; `zig build test` 0 from the repo root.
### Session 1 review-pass notes (post-build)
Ten findings from the implementation review, all fixed. Each fix was mutation-checked: the guarding test was re-run against a deliberately broken version to confirm it fails. One finding did not survive that check and is reported as such.
- **Tooltip lifecycle (behaviour change).** `useTooltip` is gone from both charts. It stored the hovered bucket's *numbers and screen coordinates*, which decoupled them from the data: a 30-second poll left last minute's counts under the pointer, and a resize left the tooltip at coordinates that no longer described anything. Both charts now keep only the hovered bucket **index**, in a new `useActiveBucket(windowKey)` hook, and read the values and the x out of the render they are currently drawing. `windowKey` is `since:bucket_seconds:bucketCount:width`; a selection made against an old key is deleted, so a rolling window or a resize retires the tooltip instead of relabelling it. A same-window refresh updates the open tooltip in place, which is the better of the two behaviours the review allowed.
- **Tooltip is keyed by bucket.** `withBoundingRects` measures its node once, in `componentDidMount`, and never again, so one shared mount placed every bucket with the first bucket's measured size — the flip-and-clip case at the right-hand edge. `key={bucket}` gives each bucket its own mount and its own measurement. jsdom reports every rect as zero, so the test asserts the remount (the node identity changes between buckets), not the measurement.
- **Tooltip offsets (behaviour change).** visx's 10px `offsetLeft`/`offsetTop` defaults are replaced by an explicit 8px, restoring the hand-rolled tooltip's placement: 8px down from the chart's top edge, 8px to the side of the slot. `top` is now 0 with the offset supplying the 8, rather than `plot.y` coincidentally being 8.
- **Overlay clamp (behaviour change).** A band scale spends a trailing gap after the last column, so a full-step hit target on the last bucket reached ~2px into the right margin. The last slot is now clipped to `plot.x + plot.width`. The tooltip's x is the **slot** centre (`band start + step/2`), not the narrower band centre it was using.
- The x-axis tick labels no longer carry `tabularNums`; the hand-rolled x labels never had it and adding it was an unsanctioned change. The y-axis labels keep it, as they always had it.
- Tests added or tightened: the clamp test now pins the y-domain source (asserting the top tick is `10`, so a switch to the stacked sum's 13 fails); `bandPaddingInner` is pinned at `(24, 1388)` and at its 0.5 cap; the value-axis test asserts the rendered `dy="0"` and `dominant-baseline="middle"`; the donut ring test parses the `A` command radii and asserts `[90, 90, 54, 54]`; both charts gained overlay-clamp, tooltip-offset, per-bucket-remount, same-window-refresh and rolled-window tests.
- **One finding could not be closed as stated.** The donut caller-order fixture is now `[1, 3]` (ascending) as directed, and it does pin that the ring is drawn in caller order. It does **not** pin `pieSort={null}`/`pieSortValues={null}`: removing both props leaves the rendered output byte-identical, because `@visx/shape` 4.0.0's own `pie()` factory already calls `sortValues(null)` when neither prop is given. The props are kept anyway and the comment now says why — visx carries that default precisely because d3-shape v3 flipped `sortValues` to descending underneath it, so the props are what stop a future bump from silently re-ranking the ring. No test can distinguish them at this version.
### Session 1 residual-pass notes (post-build)
Three residuals from the re-review, all fixed and mutation-checked.
- **A retired selection is deleted, not masked (behaviour change).** `useActiveBucket` only compared the stored key against the current one, so the selection survived in state and a *returning* key revived it: resize away from 640 and back, or a poll that restores a bucket count, redrew a tooltip and its dimming with no pointer entry. The hook now drops the selection during the render that changes the key — the React "adjust state on prop change" pattern, not an effect, so no tooltip is committed and then removed. The rolled-window test rolls away and back and asserts nothing returns.
- **The tooltip remeasures on content change as well as on bucket change.** `key={bucket}` covered moving between buckets but not the same bucket changing width under a same-window refresh: a count crossing a digit boundary, or a client name resolving. The key is now `bucket|total|label=value…`, so any content that could change the measured width forces a fresh mount. The test asserts DOM identity changes when a hovered bucket's numbers change.
- The implementation notes above were stale on bytes, test count and the `chartKit` export list; all three are corrected to the figures verified in this pass.
- Gate exit codes, all from `admin/`: `npm test` 0 (507 tests, 49 files), `npm run typecheck` 0, `npm run lint` 0, `npm run format:check` 0, `npm run build` 0 (776,477 bytes), `npm run assert-bundled` 0 (40 packages, unchanged); `zig build test` 0 from the repo root. No commits.
### Session 1 owner-review addendum (post-build)
Three changes from the owner's visual review of the live charts. All three are
behaviour changes, not corrections, and each was mutation-checked.
- **The client chart drops "Other" when it counted nothing.** `ClientChart` leaves
the aggregate out of the legend, the stack, the tooltip and the hidden table
when `data.other` is all zeroes; the named clients stay at zero. The hidden
table drops the column with the rest rather than keeping a zero column, so all
four surfaces agree; a test pins that choice. This reverses `ClientChart`'s
previous documented rule that "Other" is always present, and the
`OverviewPage` test that pinned it is inverted accordingly.
**`TimeseriesChart` does not do this** — see the ruling below.
- **`TimeseriesChart`'s third series is labelled "Allowed".** The series key stays
`other` and the colour mapping is untouched — only the displayed label changes,
in the legend, the tooltip row, the hidden table header and the SVG
`aria-label`, which is now built from the shown series rather than hard-coded.
- **The donuts gain the bar charts' hover treatment.** Pointing at a slice path
opens the shared tooltip (label, count in the panel's unit, and the same share
the legend prints) and dims the other slices to 0.55; leaving the ring clears
both. The ring stays `aria-hidden` and the legend plus hidden table remain the
accessible surface. The slice path is the hit target; no overlay was added.
Supporting refactors:
- `useActiveBucket` is renamed `useActiveIndex`: it now tracks a slice as well as
a bucket. Its `windowKey` for the donut is the drawn slices' keys, so a changed
slice set retires the hover.
- `BucketTooltipData` is replaced by `TooltipContent { title, rows }`, with
`TooltipRow.value` a string and `TooltipRow.color` optional. The bar charts now
pass their own formatted title and an explicit "Queries" total row, where
`ChartTooltip` previously hard-coded both — which is what let the donut, whose
tooltip has no timestamp and no total, reuse it unchanged. `ChartTooltip` also
takes an optional `top`, which the donut uses and the bar charts leave at 0.
- `Donut.sliceAnchor()` computes a slice's mid-arc point from the slice values.
It restates the `Pie` configuration (clockwise from twelve o'clock, caller
order, no pad angle) rather than reading the drawn path, so a test pins the
resulting transform against the ring's real geometry.
- Tests: **515** pass across 49 files (was 507). New: two timeseries tests (the
rename, and the series staying at zero), two client-chart tests for the hiding,
four donut hover tests. Bundle: `admin/dist/assets` is **776,995 bytes** across
40 files, 23,005 under the 800,000-byte gate.
- Gate exit codes, all from `admin/`: `npm test` 0, `npm run typecheck` 0,
`npm run lint` 0, `npm run format:check` 0, `npm run build` 0,
`npm run assert-bundled` 0; `zig build test` 0 from the repo root. No commits.
**Ruling (2026-08-24).** The review's item 1 originally applied the hiding to both
bar charts. It was implemented that way, then reverted for the timeseries after
the build agent raised the incoherence and the lead ruled with it: item 2's own
reasoning is that the third series is a real category — queries answered
upstream, locally, or from a forward zone — which is why it stops being called
"Other"; hiding it at zero then contradicts that, and item 1's other half ("a
zero Blocked is information") applies to it just as much. A zero Allowed says
every query in the window was blocked or served from cache, which is a state
worth reading, not an empty bucket.
The rule that survives: **hide an aggregate that aggregated nothing; never hide a
named category.** `ClientChart`'s "Other" is the only aggregate on the Overview,
so it is the only series that disappears. `TimeseriesChart` keeps Blocked, Cached
and Allowed at all times, pinned by a test.
The other two open items are closed: `pieSort`/`pieSortValues` stay as
version-proofing with their comment (lead's ruling), and the ISC acceptance line
in `licenses/inventory.zon` rests with the owner.
+417
View File
@@ -0,0 +1,417 @@
# Milestone 36: Overview performance — combined endpoint, projections, cache
Replace the five per-panel stats endpoints with one `GET /api/overview` served
from materialized projections in `querylog.db` plus an in-memory response
cache, so Overview cost stops growing with query-log size.
## Motivation (measured)
Today each Overview load runs five separate scans of every raw row in the
window, serialized on `WebState.querylog_lock`, re-polled every 30 s. Measured
x86 ReleaseSafe (bench at scratchpad `statsbench2/`, production-like skew;
Pi ≈ 33.5× slower):
| Rows | 30d, five scans (today) | 30d, one combined scan | 30d, projections |
|-----:|------------------------:|-----------------------:|-----------------:|
| 1M | ~2.2 s | 264 ms | 41 ms |
| 3M | ~6.6 s | 793 ms | 38 ms |
| 5M | ~11.8 s | 1,346 ms | 40 ms |
Projection maintenance costs +10% per 100-row insert batch, and ~1.5 MB of
disk in the bench — a size bounded by retained buckets × distinct
client/type/route keys, independent of raw query volume. Production is on a ~100k rows/day growth
curve (≈3M rows at 30-day retention), so the projection path is the design
target, not a contingency. Both computations were cross-checked for identical
output in the bench.
Design ruling (owner + Codex consultation, 2026-08-27): stay on SQLite;
projections live in the same file as the raw rows and are updated in the same
transaction, so SQLite's transaction is the coherence mechanism — no second
file, no epoch protocol. The DDL change re-fingerprints `querylog.db`; the
existing rename-aside path handles old files (one-time history reset,
disclosed in the changelog). No backfill migration.
## Sessions
Four sessions. A first. B and C after A, in parallel (disjoint files). D after B.
- A: storage — projection schema, writer maintenance, retention, new read path.
A does NOT delete the five existing aggregate functions — `stats.zig` still
calls them until B lands, and A must leave `zig build test` green.
- B: web — `/api/overview` handler, response cache, removal of the five old
endpoints, OpenAPI/contract regeneration.
- C: admin — one overview query, types, component/data plumbing, tests.
- D: storage cleanup — delete the five now-unreferenced aggregate functions.
---
## Session A: storage
### A.1 Schema (src/storage/querylog_schema.zig)
Append four projection tables to `ddl`. Grain: 30-minute buckets, `bucket` =
floor-to-grid of the row timestamp: `@divFloor(timestamp, 1800) * 1800` in Zig
and the equivalent floor semantics in any SQL (SQLite integer `/` truncates
toward zero, which differs on negative timestamps — use floor everywhere, as
`window()` does). 1800 divides every serving
width ≥ 30 min (1800, 3600, 21600), which is what makes one grain serve the
24h, 7d and 30d windows exactly. The 1h window (60 s buckets) is NOT served
from projections (A.4).
```sql
CREATE TABLE bucket_totals (
bucket INTEGER PRIMARY KEY,
queries INTEGER NOT NULL,
blocked INTEGER NOT NULL,
cached INTEGER NOT NULL,
rt_sum INTEGER NOT NULL, -- sum(response_time_us) over timed rows
rt_count INTEGER NOT NULL -- count(response_time_us)
) WITHOUT ROWID;
CREATE TABLE bucket_clients (
bucket INTEGER NOT NULL,
client_ip TEXT NOT NULL,
queries INTEGER NOT NULL,
PRIMARY KEY (bucket, client_ip)
) WITHOUT ROWID;
CREATE TABLE bucket_types (
bucket INTEGER NOT NULL,
qtype INTEGER NOT NULL, -- -1 encodes a NULL qtype, losslessly
count INTEGER NOT NULL,
PRIMARY KEY (bucket, qtype)
) WITHOUT ROWID;
CREATE TABLE bucket_routes (
bucket INTEGER NOT NULL,
route_kind TEXT NOT NULL,
source_present INTEGER NOT NULL, -- 0: source NULL; 1: source = source_text
source_text TEXT NOT NULL, -- '' when source_present = 0
count INTEGER NOT NULL,
PRIMARY KEY (bucket, route_kind, source_present, source_text),
CHECK (source_present IN (0, 1)),
CHECK (source_present = 1 OR source_text = '')
) WITHOUT ROWID;
```
Column semantics match the existing aggregates exactly: `blocked` counts
`blocked <> 0`; `cached` counts `cache_hit = 1`; routes' `source` is the
existing CASE (`upstream` rows → `upstream`, `forward_zone` rows →
`forward_zone`, else NULL). The fingerprint moves automatically; do not touch
the fingerprint machinery.
### A.2 Writer maintenance (src/storage/repositories/queries_repo.zig)
`BatchWriter.writeBatch` updates all four projections inside the same
transaction that inserts the raw rows:
- Aggregate the batch in Zig first, producing per-key deltas; then one UPSERT
per touched key:
`INSERT ... ON CONFLICT(...) DO UPDATE SET queries = queries + excluded.queries, ...`.
The aggregation must accept any slice length — `writeBatch`'s API does not
enforce the logger's 100-row batching, so no fixed-size arrays sized to it.
`BatchWriter` currently owns no allocator: `init` gains one, owned for the
writer's life, used only for the per-batch delta maps; scratch is freed (or
a retained map cleared) at the end of every `writeBatch`, and
`error.OutOfMemory` fails the batch before the transaction opens — no
hidden global allocator, no implicit size cap, no quadratic rescanning.
- No per-row SQL, no triggers.
- Failure contract: any failed projection statement rolls the whole
transaction back — raw rows and projections together — resets every
projection statement, and leaves the writer usable for the next batch
(`resetAll` discipline as for the raw statements today). Fault-injection
acceptance: a batch whose projection update fails leaves the database
unchanged, and the next batch succeeds.
- The bench measured this at 0.39 ms vs 0.34 ms per batch — acceptance is
correctness, not speed.
### A.3 Retention (src/storage/repositories/queries_repo.zig, prune path)
In the same transaction as `pruneOlderThan(cutoff)`'s raw delete:
1. Delete projection rows with `bucket < floor(cutoff / 1800) * 1800` from all
four tables.
2. If `cutoff` is not on a bucket boundary, recompute the straddling bucket
(`floor(cutoff/1800)*1800`) from the remaining raw rows and replace its
projection rows in all four tables. Never approximate.
Failure atomicity: a failure during the projection delete or the
straddling-bucket replacement rolls back the raw delete, the watermark
advance and every projection change together — one transaction, tested by
fault injection.
Implementation note (Session A, recorded post-build): the bucket_totals
recompute carries `HAVING count(*) > 0` — a bare SQL aggregate always yields
one row, and an emptied straddling bucket must disappear, not persist as
zeros.
### A.4 Read path (src/storage/repositories/queries_repo.zig)
One function producing the whole Overview payload for a window, from one
already-open read transaction (the caller owns transaction + lock, as today):
```zig
pub const Overview = struct {
totals: StatsTotals,
buckets: []const Bucket, // bucket_count entries, zero-filled
clients: ClientsBreakdown, // top-8 + other, as today
types: []const TypeCount, // sorted as stats_types_sql sorts
routes: []const RouteCount, // sorted as stats_routes_sql sorts
};
pub fn overview(
database: *db.Db,
arena: Allocator,
since: i64,
bucket_seconds: u32,
bucket_count: u32,
) db.Error!Overview
```
Storage owns these scalars — no import of any web module. `until` is derived
as `since + bucket_seconds * bucket_count` with the same overflow checks as
`timeseries`. Preconditions, checked before path selection and tested:
`bucket_seconds != 0` and `bucket_count != 0` (else `error.Misuse`, matching
the existing clients contract); on the projection path
(`bucket_seconds >= 1800`) additionally `since` a multiple of 1800 and
`bucket_seconds % 1800 == 0`, else `error.Misuse`. The handler's `window()`
guarantees all of them.
Two implementations behind one entry point, chosen by `bucket_seconds`:
- `bucket_seconds >= 1800` (24h, 7d, 30d): read the four projection tables
over `[since, until)`, aggregating 30-min rows up to the serving width in
Zig. `distinct_clients` comes from grouping `bucket_clients` by `client_ip`
over the window — never from summing per-bucket counts.
`avg_response_time_us` = `sum(rt_sum) / sum(rt_count)`, null when
`rt_count` sums to 0. Top-8 clients ranked by window total desc, ties by
`client_ip` asc (BINARY), residual summed into `other` — identical cut
semantics to `statsClients`.
- `bucket_seconds < 1800` (1h): one single pass over the raw rows in the
window (one SELECT of the needed columns, stepped once), aggregating
everything in Zig. Memory bound: O(distinct clients + distinct qtypes +
distinct routes) in the window — explicitly permitted; this is a household
LAN and the same bound the arena-returning aggregates already carry. This
replaces today's five scans and the clients rank+bucket double scan. Same
output contracts.
Sort orders and tie-breaks must reproduce the existing SQL orderings exactly
(types: count desc, null last within tie, qtype asc; routes: count desc,
route_kind asc, null source last, source asc) — the goldens' byte-stability
argument carries over. Note: `RouteKind`'s enum declaration order is not
alphabetical; "route_kind asc" means the stored text's byte order, so any Zig
comparator orders by `@tagName` bytes, never by enum ordinal (Session A's
accumulator already does; mutation-tested).
### A.5 Acceptance criteria
- [ ] `zig build test` green.
- [ ] Property test: after an arbitrary interleaving of batches and prunes
(including a prune cutoff off the bucket grid), every projection table
equals a from-scratch recomputation from `query_log`.
- [ ] Equivalence test: `overview()` output (both paths) equals a test-only
oracle over the same window on the same data — including empty windows,
NULL qtype, NULL source on an `upstream` row, ties in ranking, and a
window whose last bucket is in progress. The oracle is a copy of the
five existing SQL aggregates living in the test file, so it survives
Session D's deletion of the production functions.
- [ ] Fingerprint test updated (table/index count assertions in
querylog_schema tests).
---
## Session B: web
### B.1 Endpoint (src/web/handlers/overview.zig, replacing stats.zig's five)
`GET /api/overview?period=1h|24h|30d|7d` (same grammar, default 24h, same 400
text). One read transaction under `WebState.querylog_lock` covering the
aggregate and `coverage.read` — one snapshot, no cross-panel skew. Response:
```json
{
"period": "24h", "since": ..., "until": ..., "bucket_seconds": 1800,
"totals": { "queries": n, "blocked": n, "clients": n, "avg_response_time_us": n|null },
"buckets": [ { "ts": ..., "queries": n, "blocked": n, "cached": n }, ... ],
"clients": [ { "client": "ip", "buckets": [n, ...] }, ... ],
"other": [n, ...],
"types": [ { "qtype": n|null, "count": n }, ... ],
"routes": [ { "route": "...", "source": "..."|null, "count": n }, ... ],
"coverage": { ... }
}
```
Field shapes and semantics are exactly today's five bodies merged; `Period`,
`window()`, `max_buckets` move to (or stay importable from) the new handler.
503 when the query log is unavailable; 500 logging unchanged. Route metadata
identical to the removed endpoints: same authentication (`.session`), same
rate-limit class (`.counted`), same authority policy (`.read`).
Remove `GET /api/stats`, `/api/stats/timeseries`, `/api/stats/types`,
`/api/stats/routes`, `/api/stats/clients` and their routes.
Contract surface (B owns all of it): add the new path and schema to the
OpenAPI document, remove the five old operations and their schemas, update
every drift guard that lists them, add a contract sample for
`/api/overview`, and regenerate `admin/src/lib/contractSamples.gen.ts`
(reserved for B — Session C must not touch it). Update the API listings in
`docs/` and `PLAN.md` that name the five endpoints or the querylog layout.
### B.2 Response cache (src/web/server.zig WebState + overview.zig)
Per-period cached response body, invalidated by data change or window roll.
Key: `(period, window.until, data_version)` where `data_version` is `PRAGMA
data_version` on the web task's connection (it changes when any other
connection — logger, retention — commits).
The entire cache decision happens under `querylog_lock`; nothing touches the
shared connection or the slots outside it. Exact sequence per request:
1. Acquire `querylog_lock` — ONCE. `server.QuerylogRead.open` acquires this
lock itself, so the overview handler must not call it after step 1: B
refactors the scope into a lock-owning wrapper plus a
locked-caller variant (for example `QuerylogRead.openLocked`, documented
as requiring the lock), and the overview path uses the locked-caller
variant for step 4. A literal "lock, then QuerylogRead.open" deadlocks.
2. Sample `PRAGMA data_version` (inside the lock — the shared connection may
otherwise have a foreign transaction open, and the slots need the mutual
exclusion anyway).
3. Hit (`slot.period == period and slot.until == window.until and
slot.data_version == sampled`): copy the stored bytes into the request
arena, release the lock, respond. The copy is what makes a concurrent
rebuild's free-and-replace safe.
4. Miss: open the read transaction, build the body, commit. Publish to the
slot ONLY after a successful commit, keyed by the version sampled in
step 2 (a commit landing during the build bumps `data_version`, so the
next request rebuilds — stale-under-new-key is impossible). A failed
commit or build publishes nothing and responds 500 as today.
5. Copy to the request arena, release the lock, write the socket. The lock
never spans a socket write (existing discipline).
Because the check happens only under the lock, `querylog_lock` is the
single-flight: a second request for the same key waits and then hits.
Storage: one slot per period (4 slots) in `WebState`; body bytes allocated
from `WebState.gpa`, replaced on rebuild (free old, install new), freed in
`deinit`. No capacity limit beyond the allocator — a body is bounded by the
fixed bucket counts plus the household client/type/route cardinality.
No adaptive polling and no combined-endpoint staging: with projections + this
cache a rebuild is ~40 ms x86 / ~0.13 s Pi, so the admin's existing 30 s
cadence is fine.
### B.3 Acceptance criteria
- [ ] `zig build test` green; handler tests ported from stats.zig (period
grammar, window math, one-snapshot behavior) plus: cache hit returns
byte-identical body; a logger commit (data_version bump) invalidates;
a window roll invalidates; a retention prune committed through another
connection invalidates (both the aggregates and the cached
`coverage.available_since` are replaced); a failed read-transaction
commit neither installs nor replaces a cache entry.
- [ ] `curl /api/overview?period=30d` on a seeded scratch instance returns all
panels consistent (breakdowns sum to totals on a quiet database).
- [ ] The five old routes return 404.
---
## Session C: admin
### C.1 Data layer
- `admin/src/lib/types.ts`: one `Overview` type mirroring B.1; remove the five
per-panel response types.
- `admin/src/lib/api.ts`: `getOverview(period)`; remove the five getters.
- `admin/src/lib/queries.ts`: `overviewQuery(period)` with
`refetchInterval: 30_000` and key `["overview", period]`; remove the five
stats query factories and their keys.
### C.2 Overview page
`admin/src/features/overview/overviewWindow.ts` and the chart components
consume the single query: one `useQuery` where five ran in parallel. Loading,
error and coverage handling collapse to one page-level surface (one spinner
state, one error state for the whole Overview); the per-panel shells,
layout, copy, chart dimensions and accessibility attributes stay exactly as
they are. Chart components (TimeseriesChart, ClientChart, Donut) keep their
props — adapt the mapping layer, not the charts. C must not touch
`contractSamples.gen.ts` (B owns its regeneration).
### C.3 Acceptance criteria
- [ ] `npm test` green in admin/ (mock the one endpoint; port the five-query
tests).
- [ ] `npm run typecheck` green — the build alone does not run tsc. B and C
are file-disjoint but type-coupled through `contractSamples.gen.ts`:
C's typecheck/test/build gates run (or re-run) AFTER B has regenerated
that file. Implementation may proceed in parallel; the green gate is
sequenced.
- [ ] `npm run build` green; bundle-size assertion still passes.
- [ ] Manual: scratch instance renders all Overview panels from the new
endpoint on all four periods.
---
## Session D: storage cleanup
After B is merged and green: delete `statsTotals`, `timeseries`, `statsTypes`,
`statsRoutes`, `statsClients` and their SQL constants from
`queries_repo.zig` — nothing references them once `stats.zig` is gone. The
test-only oracle from A.5 stays. Acceptance: `zig build test` green, no dead
stats SQL remains in production code.
---
## Module layout
- `src/web/handlers/overview.zig` — new; replaces `src/web/handlers/stats.zig`
(deleted by B).
- `src/storage/querylog_schema.zig` — projection DDL appended.
- `src/storage/repositories/queries_repo.zig` — writer maintenance, retention
integration, `overview()` read path (A); five old aggregates deleted (D).
- Admin files per C.1/C.2.
## File ownership
- A: `src/storage/*` (old aggregates left in place), plus the mechanical
allocator-plumbing at every `BatchWriter.init` call site outside storage
(`src/web/handlers/*`, `src/web/web_integration_test.zig`, any test using
the writer) — A runs before B, so this is sequential, not shared,
ownership; A's gate is the full `zig build test`.
- B: `src/web/*`, plus: `src/app.zig` (cache cleanup at the composition
root), `src/tests.zig` (handler import swap), the OpenAPI document and its
drift guards, contract samples including
`admin/src/lib/contractSamples.gen.ts`, `docs/**` API listings, `PLAN.md`
stale sections, and the Overview/API sections of `specs/ui-redesign.md`
(which still mandates the five endpoints and must be amended, not obeyed).
- C: `admin/*` EXCEPT `admin/src/lib/contractSamples.gen.ts`.
- D: `src/storage/repositories/queries_repo.zig` (sequential, after B).
- Orchestrator: `CHANGELOG.md` (hand-written, per release process).
B and C run in parallel; the one shared-tree exception above is reserved to B.
## Acceptance criteria (milestone complete)
- [ ] `zig build test` and `zig build test -Dintegration` green (the
integration suite carries the live route walk, contract-sample
comparison, concurrent querylog reads and OpenAPI guards); admin
`npm test`/`typecheck`/`build` green.
- [ ] Scratch-instance smoke: seeded data + live digs; Overview correct on all
periods; old endpoints gone.
- [ ] Changelog discloses: schema change resets query history (rename-aside),
five endpoints replaced by `/api/overview`.
- [ ] Release gate (`zig build cut` fingerprint check) satisfied.
## Anti-requirements
- No second database file, no epoch/validity protocol, no ATTACH.
- No backfill migration, no rebuild command, no catch-up cursor — projections
are born with the file and maintained transactionally; that is the whole
coherence story.
- No connection pool, no adaptive polling, no DuckDB.
- No new indexes on `query_log`, no triggers, no per-row projection SQL.
- Do not change the 1h/24h/7d/30d period grammar, bucket widths or counts.
- No HTTP-level caching of any kind: no ETag, no `Cache-Control`, no
stale-while-revalidate, no background refresh, and never cache a 500/503
body. The cache is exactly the in-process design of B.2.
- No visual redesign, no chart-prop changes, no cache configuration knobs,
no cache metrics. This milestone changes data acquisition and storage only.
+337
View File
@@ -0,0 +1,337 @@
# Milestone 37: Upstream failover budget — deadline ownership and honest attribution
Make the pool's failover loop own one deadline (admission included), revive the
standby under primary saturation, and stop blaming endpoints for budget
exhaustion.
## The defect (field-verified on the Pi, 0.0.11)
With defaults attempt=2500 ms / total=5000 ms and two upstreams:
1. `Pool.exchangeLoopLen` (pool.zig:277) never computes remaining time. The
semaphore wait consumes the total budget invisibly: under a traffic burst
the primary's slots saturate, a new request queues behind them while the
standby sits idle with free slots, and the outer `raceWithin(total)` cancels
whatever finally runs. 155 requests died at the 5 s cap; 706 of 717
SERVFAILs came from one bursty client. On an idle pool a fast standby works
today — the queue is the killer, not the timer arithmetic alone.
2. The reporting contradicts itself. `selected` is written before the attempt
runs (pool.zig:348), so the query log blamed the standby on 279 rows;
cancelled attempts are excluded from health (pool.zig:363), so its counters
read 0/0. Nothing records "the request exhausted its own budget."
Design reviewed and converged with Codex (thread of 2026-08-27). Defaults do
not change.
## Sessions
Three, strictly sequential: A (transport vocabulary) → B (pool) → C
(handler, forward client, surfaces).
---
## Session A: transport vocabulary
### A.1 `error.BudgetExhausted` and a fourth group (src/upstream/transport.zig)
- Add `BudgetExhausted` to `ExchangeError`.
- Add `.budget_exhausted` to `Group`; `group()` maps the new error to it. The
switch stays exhaustive with no `else` — every switch over `Group` breaks at
compile time until it handles the new group, which is the point. The
production switches are `pool.zig` (:353), `handler.zig` (:677, :735) and
`forward_client.zig` (:124). Session A owns a mechanical placeholder arm at
each (pool and forward_client: propagate the error without recording
health; handler: `=> ctx.servFail()`), plus any test switches the compiler
flags, so A's `zig build test` passes. B and C then own the real behavior
at their sites. (health.zig has no `group` call; the doh/dot occurrences
are tests, not switches.)
### A.2 Timer-origin race (src/upstream/transport.zig)
`raceWithin` collapses "the expiry task won" and "the raced operation itself
returned error.Timeout" into one `error.Timeout`. Add:
```zig
pub const RaceOutcome = enum { completed, expired };
pub fn raceUntilTagged(
io: std.Io,
expiry_at: std.Io.Clock.Timestamp,
outcome: *RaceOutcome,
comptime f: anytype,
args: anytype,
) ExchangeError!RacedPayload(f)
```
The expiry parameter is an ABSOLUTE timestamp, not a duration: a duration
computed from `deadline.toDurationFromNow()` and then slept re-anchors at
"now", drifting past the total deadline and misclassifying a nominally full
attempt. The pool computes `expiry_at = min(now + attempt, total_deadline)`
and passes the timestamp. On the expiry side the function sets `outcome.* =
.expired` and returns `error.Timeout`; on completion it sets `.completed` and
returns the raced result (which may itself be `error.Timeout` from the leaf —
that is a completed peer timeout, not an expiry). `raceWithin` keeps its
public API and behavior but delegates to the same internal harness (compute
the absolute deadline, discard the outcome) — one `Select` harness in the
file, not two copies.
### A.3 Acceptance
- [ ] `zig build test` green.
- [ ] Unit tests: tagged race distinguishes leaf `error.Timeout` (completed)
from expiry (`expired`); the untagged wrapper is unchanged behavior.
---
## Session B: the pool (src/upstream/pool.zig)
### B.1 One deadline owns the loop
- `Pool.exchange` establishes `deadline = std.Io.Timeout{ .duration =
self.timeouts.total }.toDeadline(io)` (`.awake` clock, as the loop uses
today) and passes it down. The equal-deadline outer `raceWithin` at
pool.zig:267 is REMOVED — the loop owns the deadline; two timers aimed at
the same instant race each other and let the outer cancellation bypass the
loop's classification. No replacement watchdog (the outer race never bounded
uncancelable health writes anyway; a later-firing watchdog is scope creep).
- Every blocking step consumes the deadline:
- Admission: through the ownership-safe protocol of B.2 — never a bare
`Semaphore.wait` raced against an expiry. Racing the wait with
`Select.cancelDiscard` can leak a permit: the wait may have decremented
the count in the same instant the expiry wins, and the discarded success
never reaches the caller's release-defer. If no time remains before
admission, return `error.BudgetExhausted` without waiting.
- Attempt: raced via `raceUntilTagged` against
`expiry_at = min(now + timeouts.attempt, total_deadline)`.
- `total < attempt` is already rejected by validate.zig; `total == attempt`
(and any admission overhead) simply yields truncated attempts, handled by
B.3.
### B.2 Admission without head-of-line blocking
Zig 0.16's `Semaphore` has neither try-acquire nor a timed wait, so the pool
gains a local admission helper mirroring the standard semaphore's own
mutex/decrement/condition protocol (never by patching `../zig`, never by
spinning):
- `tryAcquire()` — take a permit if one is immediately available, else fail
without blocking.
- `acquireUntil(deadline)` — timed acquisition; returns Acquired (holding a
permit), Expired (holding none), or `error.Canceled` (holding none).
`std.Io.Condition` has no timed wait in 0.16, so this is a NEW pool-local
primitive, specified exactly: state lives under one mutex (permit count +
waiter bookkeeping); the wait itself may race `Condition.wait` against a
sleep via `Select`, because a permit is only ever taken under the mutex
AFTER the race resolves — a discarded wake is a lost notification, not a
lost permit. To keep that lost notification from stranding another waiter,
an exiting waiter that may have absorbed a signal (expiry or cancellation
path) re-signals the condition before returning. Permit conservation is by
construction: the decrement and the "did I win" decision happen under the
same mutex. Every take is non-blocking (the mutex's `tryLock`): a lock
miss reads as "no permit now", and `acquireUntil` re-enters the
absolute-deadline race on each miss, so contention on the admission mutex
never carries a call past its deadline (review round 2026-08-27). Tests:
an expiry/acquisition tie leaves the permit count exact; a cancelled
waiter holds nothing and a peer waiter still wakes; a contended admission
mutex does not carry `acquireUntil` past its deadline.
Loop semantics — priority means ordering among immediately admissible
candidates:
1. Pass one, first sweep in priority order: `tryAcquire` on each available
entry; the first immediate success is attempted. A saturated entry is
skipped while another eligible entry has capacity.
2. If the attempted entry fails (peer fault), the sweep continues from the
next entry, still by `tryAcquire`; previously skipped saturated entries
are re-tried by `tryAcquire` on each subsequent sweep step (a slot may
have freed).
3. Only when no eligible entry has an immediate permit does the loop block:
`acquireUntil(remaining deadline)` on the highest-priority eligible
entry. Expired → `error.BudgetExhausted`. The no-head-of-line guarantee
is deliberately scoped to capacity observed during the sweep: once
blocked, a lower-priority slot freeing does not wake this waiter
(any-entry wakeups need multi-wait machinery this milestone does not
buy). Record this bound in the admission helper's doc comment.
4. Pass two (backoff probing) keeps today's in-order probing but admits
through the same helper bounded by the remaining deadline — it
deliberately retains blocking, one entry at a time, because probing a
backed-off entry is already a last resort.
5. The post-admission health recheck survives the refactor: after acquiring
a permit by either path in pass one, re-read health against a fresh
`now`; an entry that entered backoff while this task waited is released
(permit returned) and the sweep resumes. The existing regression test
for this recheck is retained.
6. Before any attempt starts — immediate admission included — the loop
re-reads the clock; a deadline already passed returns
`error.BudgetExhausted` with the permit returned and no endpoint named,
so an instantly-completing leaf can never manufacture evidence after
exhaustion (review round 2026-08-27). Test: an exchange whose deadline
is already gone starts no attempt.
7. The compiled pool bound (`Pool.max_entries`, 64) is enforced at config
validation: more than 64 ENABLED upstreams is `TooManyUpstreams` at path
`upstreams`, so a valid config can never trip the pool assert (review
round 2026-08-27). Tests: 64 enabled passes, 65 fails, 65 listed with
64 disabled passes.
8. Queue accounting keeps its meaning: `queued_total` and
`queued_seconds_total` count only the blocking `acquireUntil` path,
recorded whether it ends in acquisition, expiry or cancellation; a
`tryAcquire` — hit or miss — never counts as queued. Acceptance test
retained.
### B.3 Classification (uses A.2's tagged race)
| Attempt outcome | Budget it ran with | Health | `selected` | Loop action |
| --- | --- | --- | --- | --- |
| success | any | success | this endpoint | return answer |
| expiry (`expired`) | full `attempt` | failure | this endpoint | continue failover |
| expiry (`expired`) | truncated | untouched | unchanged | return `error.BudgetExhausted` |
| completed peer fault (incl. leaf Timeout) | any, even truncated | failure | this endpoint | continue failover |
| local_resource | — | untouched | unchanged | return err (as today) |
| cancellation | — | untouched | unchanged | return `error.Canceled` |
| wait exhausts deadline | — | untouched | unchanged (null if nothing ran) | return `error.BudgetExhausted` |
| completed attempt returns `error.BudgetExhausted` (a leaf may emit it once it exists) | any | untouched | unchanged | return it; counter increments once at the outer pool |
A truncated expiry is a censored observation: the pool did not grant the
configured observation interval, so it is evidence about the pool's budget,
never about the peer. No minimum-attempt floor exists.
### B.4 `selected` = last attributable endpoint
Assign `selected.*` only in the success row and the two health-recording
failure rows above — after the attempt completes, not before it starts. This
attribution rule is POOL-SPECIFIC: leaf clients (DoH, DoT, ForwardClient,
test fakes) keep their write-before-attempt behavior — they have one
endpoint and record no health, so "the endpoint I tried" is honest there.
The `Client.exchange` doc comment in transport.zig is amended to state both
contracts: implementations may write before each attempt; `Pool` documents
its stricter last-attributable rule on `Pool.exchange` itself. The pool-level
tests at pool.zig:748-771 (selected-before-attempt) and :910 invert into the
new contract's tests; leaf-client tests are untouched.
### B.5 The counter
`Pool` gains one pool-level counter, `budget_exhausted_total` (atomic u64,
incremented once per exchange that returns `error.BudgetExhausted`, never per
endpoint). `Pool.snapshot` returns per-entry rows and cannot carry a
pool-wide number without duplicating it — so the counter is exposed through a
separate getter, `Pool.budgetExhaustedTotal()`, and B owns the mechanical
plumbing at every existing snapshot call site its change touches so B's own
gate passes before C. No per-stage split (queue vs attempt) —
fixed-cardinality stage labels are deferred until an operator needs them.
### B.6 Acceptance (the decisive regressions first)
- [ ] Fast standby at shipped defaults: entry 0 stalls its full attempt
budget, entry 1 answers instantly, attempt = total/2 → entry 1 answers.
- [ ] Saturated primary, free standby: entry 0's slots all held by stalled
exchanges, entry 1 free and fast → entry 1 answers well inside the
deadline. This is the Pi reproduction; it must FAIL against the current
code and pass after B.2.
- [ ] Truncated expiry mutates no health, returns BudgetExhausted, leaves
`selected` at the last attributable endpoint.
- [ ] Truncated attempt failing with ConnectionRefused mutates health and
updates `selected`.
- [ ] Queue-only exhaustion → BudgetExhausted with `selected == null`.
- [ ] All-backoff pass two runs under the same deadline.
- [ ] External cancellation returns Canceled, counts no budget, mutates
nothing.
- [ ] `budget_exhausted_total` increments once per exhausted exchange.
- [ ] Existing invariants hold: cancelled waiter returns its permit; two
stalling upstreams cost ≤ total, not one budget each.
---
## Session C: handler, forward client, surfaces
### C.1 Handler (src/server/handler.zig)
Both `transport.group` switches (:677, :735) gain `.budget_exhausted =>
ctx.servFail()` — SERVFAIL on the wire, same as peer faults; no rcode fits
better. No per-query warn log (spam); the counter is the record.
### C.2 Forward client (src/local/forward_client.zig)
One outer budget for the whole exchange: wrap UDP attempt → truncation
fallback → TCP in a single tagged race against `read_timeout` (the UDP
receive's internal deadline and the TCP `raceWithin` at :217 collapse into
the one outer bound — remove the fresh TCP budget). Every timeout here
concerns the single configured resolver, so expiry stays `error.Timeout`
(peer evidence), never BudgetExhausted — the budget/peer distinction is pool
policy. Test: truncated-UDP-then-stalled-TCP completes or expires within one
`read_timeout`, not two. Update the doc comment on `read_timeout_ms` in the
config (validate.zig:360 note and the settings description) to say it bounds
the whole forward-zone exchange. The key is NOT renamed (anti-requirement).
### C.3 Surfaces
- Metrics: `nxdns_upstream_budget_exhausted_total` (pool-wide) wherever
`nxdns_upstream_*` counters render, read via `Pool.budgetExhaustedTotal()`.
- `/api/health` is NOT changed (decision, not omission): the counter is an
operator metric, it never changes health status, and adding it to the API
would drag openapi.yaml, admin types, fixtures, contract samples and the
admin gates into a milestone that owes them nothing. Metrics only.
- docs: `docs/reference/configuration.md` — every statement describing
`read_timeout_ms` as a per-read or per-attempt bound is rewritten to the
whole-exchange contract; the upstream timeouts section gains one paragraph
on budget semantics (deadline, truncation, attribution). The stale contract
text in `src/config/model.zig` (the setting's doc comment) and the note at
`src/config/validate.zig:360` are updated to match.
### C.4 Acceptance
- [ ] `zig build test` and `zig build test -Dintegration` green.
- [ ] Metrics test covers the new counter's rendering.
- [ ] Forward-client single-budget test per C.2.
---
## File ownership
- A: `src/upstream/transport.zig`, plus mechanical placeholder arms at the
broken `Group` switches (`src/upstream/pool.zig`, `src/server/handler.zig`,
`src/local/forward_client.zig`, test switches the compiler flags) —
sequential ownership, A runs alone; C takes forward_client.zig and
handler.zig over later in sequence.
- B: `src/upstream/pool.zig` (+ the `Client.exchange` doc contract in
transport.zig and snapshot-caller plumbing for the new getter — B runs
alone after A).
- C: `src/server/handler.zig`, `src/local/forward_client.zig`, the metrics
rendering files, `src/config/model.zig` + `src/config/validate.zig` doc
text, `docs/reference/configuration.md`.
- Orchestrator: CHANGELOG.md.
## Acceptance criteria (milestone complete)
- [ ] All session criteria; both suites green; `zig fmt --check` clean.
- [x] The saturated-primary regression demonstrably fails on pre-milestone
code and passes after. Evidence (orchestrator-run mutation check,
2026-08-27): with the pass-one `tryAcquire` sweep disabled in
`Pool.admit` — restoring pre-fix head-of-line blocking — the test
"a saturated primary defers to a standby that has capacity" fails with
`error.BudgetExhausted` out of the blocking admission path, and four
queue-accounting/attribution tests fail with it (5 failed, seed
0x91ce5df5). Sweep restored: 30/30 steps, 1892/2067 passed, 0 failed.
- [ ] Changelog: failover now works under primary saturation; SERVFAILs from
budget exhaustion are counted, not blamed on an upstream; for
upstream-pool queries the query-log `upstream` field now names only
endpoints whose outcome was recorded (may be null) — forward-zone
queries keep naming their single configured resolver as before.
## Anti-requirements
- No default timeout changes.
- No parallel/racing fan-out to multiple upstreams. Priority failover stays,
with priority defined as ordering among immediately admissible candidates
(B.2) — a saturated higher-priority entry defers to an admissible
lower-priority one; it does not outrank an idle standby by blocking on it.
- No `/api/health` or admin changes; the counter surfaces in metrics only.
- No patching of the vendored/system Zig stdlib; the admission helper is
pool-local.
- No rename of `upstream.read_timeout_ms` (doc fix only).
- No minimum-attempt floor constant.
- No per-stage split of the budget counter; no per-query budget log lines.
- No watchdog replacing the removed outer race.
- No fake upstream identity (e.g. "budget") in the query log; no new
query-log column.
- No changes to backoff policy, slot counts, or health scoring beyond the
attribution rules above.
+192
View File
@@ -0,0 +1,192 @@
# Milestone 38: querylog schema migrations
Stop the recurring query-history loss: schema changes migrate querylog.db in place; the automatic reset survives only for real corruption; explicit breaks stay possible but must be versioned, refused by `open`, and ship recovery instructions.
Owner rulings (2026-08-28): baseline is the 0.0.12/0.0.13 schema — nothing older is migratable; breaking changes remain allowed but must be explicit with clear changelog instructions; keep only the most recent pre-migration backup.
## Sessions
A (storage framework) first. B (cut gate) needs A's modules. C (docs) after A (documents A's behavior; shares no files with B). The orchestrator writes the changelog.
---
## Session A: migration framework in storage
### A.1 Version metadata module (pure, no SQLite)
New file `src/storage/querylog_versions.zig` — importable by `tools/cut.zig` without linking SQLite. ONLY comptime data:
- `pub const current_version: i32 = 1;`
- `pub const minimum_supported_version: i32 = 1;` — files stamped below this refuse. An EXPLICIT BREAK in a future release is expressed here: bump `current_version`, set `minimum_supported_version = current_version`, ship no step. The chain then cannot reach the new version from below the minimum and `open` refuses the old file — a break is always versioned, always refused at runtime, never silent.
- `pub const legacy_fingerprint: i32 = 1975011655;` — the literal `user_version` stamp the 0.0.12/0.0.13 binaries wrote (CRC32 of their DDL text). FROZEN literal, derived from nothing; comment cites v0.0.12.
- `pub const version_floor_guard: i32 = 1_000_000;`
- Comptime asserts: `minimum_supported_version >= 1`; `minimum_supported_version <= current_version`; `current_version <= version_floor_guard`; `legacy_fingerprint` outside `[0, version_floor_guard]`; `step_sql.len == current_version - minimum_supported_version`.
- `pub const step_sql: []const [:0]const u8 = &.{};` — step i migrates version `minimum_supported_version + i` to `+ i + 1`; each entry is `@embedFile("migrations/v<from>.sql")`. EMPTY this milestone.
- **Steps are SQL-only. There are no migration hooks.** A rebuild that m36-style projections would need is expressible as plain SQL (the recompute statements are SQL); a future change that truly cannot be SQL must amend this design explicitly in its own spec. This keeps every shipped migration byte-comparable (B.2 Gate 2) with no mutable code path.
- Shipped step files `src/storage/migrations/v<from>.sql` and fixtures (B.1) are immutable once released; the cut gate byte-compares them against the previous tag.
### A.2 Runner module and the rebuild rule
New file `src/storage/querylog_migrations.zig` (SQLite side): the runner and the equivalence oracle.
- `pub fn migrateSteps(database: *db.Db, sql: []const [:0]const u8, from: i32, target: i32) (db.Error || error{TransactionViolation})!void` — runs the steps and the final `PRAGMA user_version = target` stamp inside the caller's already-open transaction. SLICING CONTRACT: `sql` is exactly the `[from, target)` suffix — `sql[0]` migrates `from -> from + 1`; asserted: `sql.len == @intCast(target - from)`. Production callers slice `step_sql[from - minimum_supported_version ..]`. While steps execute, the runner installs SQLite's authorizer (`sqlite3_set_authorizer`; expose a scoped install/clear pair on the db wrapper) denying `SQLITE_TRANSACTION` — a step cannot BEGIN/COMMIT/ROLLBACK at all, which is the only reliable guard (a step containing `COMMIT; BEGIN IMMEDIATE;` would pass a post-step autocommit check while breaking atomicity; that exact bypass is a required negative test, and the test must also assert the authorizer is cleared after the rejection: the rollback succeeds and the SAME connection can then execute transaction statements normally — a leaked authorizer would block cleanup and strand the connection inside the migration transaction). The authorizer is cleared on every exit path. Belt: the post-step `sqlite3_get_autocommit(db) == 0` check stays. `migrateSteps`'s error set is `(db.Error || error{TransactionViolation})`; `runMigration` maps `TransactionViolation` to `error.MigrationFailed`. The no-transaction-statements rule is also in the step-authoring doc comment.
- `pub fn runMigration(io: std.Io, dir: std.Io.Dir, path: [:0]const u8, database: *db.Db, sql: []const [:0]const u8, from: i32, target: i32) Error!void` — the full orchestration seam: backup (A.4 step 1), transaction + `migrateSteps` + commit (step 2), failure handling (step 3), retention (step 4). `open` calls it with production metadata; synthetic tests call it directly with test chains, so the REAL backup/collision/retention/error paths are what the tests prove.
- **Rebuild rule** (doc comment on `step_sql`): a step that changes a table's shape must produce a table whose stored CREATE text is byte-identical to the fresh DDL's. The RUNNER brackets every migration with: `PRAGMA foreign_keys = OFF` and `PRAGMA legacy_alter_table = ON` BEFORE `BEGIN IMMEDIATE` (with `foreign_keys` on — which `db.applyPragmas` enables — a rename of a referenced parent rewrites child tables' FK text to `<t>_old`, corrupting them the moment the old table drops; `legacy_alter_table` alone does not prevent that), and restores both pragmas on EVERY exit path, success or failure (they are connection-global and non-transactional). Before COMMIT the runner runs `PRAGMA foreign_key_check` and fails the migration on any row. Step sequence: `ALTER TABLE <t> RENAME TO <t>_old`, `CREATE TABLE <t> ...` pasted VERBATIM from the target `querylog_schema.ddl`, `INSERT INTO <t> SELECT ...` mapping, `DROP TABLE <t>_old`, recreate EVERY dependent object of `<t>` verbatim from the target DDL — indexes AND triggers (both dropped with `<t>_old`). Views are NOT dropped by the rename or the drop (with `legacy_alter_table` on they keep naming `<t>`), so a step DROPs each view over `<t>` FIRST and recreates it verbatim LAST — recreating without the drop fails with "view already exists". `ALTER TABLE ... ADD/RENAME COLUMN` on a kept table is forbidden — SQLite rewrites stored CREATE text under it and the oracle's text layer would rightly fail.
### A.3 The open path (rework `querylog_schema.open`)
`open` owns the file exclusively: nxdns opens querylog.db once at startup before serving, and no other process shares a data dir (existing deployment contract; restate in `open`'s doc comment — the backup-then-lock sequence relies on it).
The version-handling half of `open` is factored as `openVersioned(io, dir, path, handle, plan) Error!void` where `handle: *?db.Db` is an optional SLOT: `openVersioned` closes and nulls it on every error path, so the caller's `errdefer` no-ops and single-close is structural rather than a convention (as built 2026-08-28; the post-commit test asserts `handle == null`). `plan: Plan = .{ .minimum: i32, .current: i32, .legacy_fingerprint: i32, .step_sql: []const [:0]const u8 }`. Production `open` passes the constant plan from `querylog_versions`; tests inject synthetic plans, which is what makes classification, migration, the post-commit mapping, and the sole-close ownership all testable through the REAL open path even while the production chain is empty. Classification itself stays a pure function of `(stamped, plan)`.
Classify a healthy existing file: read `PRAGMA user_version` as `stamped`, map to a logical version FIRST, mutate NOTHING during classification:
| condition | logical version | action |
| --- | --- | --- |
| `stamped == legacy_fingerprint` | 1 | classify version 1 by the rows below; if it lands on "current" or "supported older", first restamp to 1 (one transaction, A.5 error mapping), then proceed |
| `stamped == current_version` | stamped | open as today |
| `minimum_supported_version <= v < current_version` | v | migrate via `runMigration` |
| `current_version < v <= version_floor_guard` | v | REFUSE: `error.SchemaTooNew` |
| anything else (0, negatives, other fingerprints, below minimum) | — | REFUSE: `error.SchemaUnsupported` |
The order matters: after a future explicit break raises the minimum above 1, a legacy-fingerprint file maps to version 1, classifies as below-minimum, and refuses WITHOUT the restamp — an unsupported file is never modified.
REFUSE: the canonical file stays in place, logically untouched (schema, rows, watermark, stamp unchanged — WAL/SHM sidecar bytes may change from the probe; not a violation), nothing set aside, no new file, `open` errors, the server does not start. The log line names the path, the stamped value, the supported range, and `docs/how-to/troubleshoot.md` ("The server refuses to start over querylog.db").
Recreate lanes `missing`, `not_a_database`, `corrupt`, `quick_check_failed` unchanged. `RecreateReason.fingerprint_mismatch` and the `schema-changed` aside tag are DELETED.
Fresh files: after executing `ddl`, stamp `PRAGMA user_version = current_version` (the stamp is already a separate statement; the DDL text does not change this milestone, so `querylog_schema.fingerprint` does not move).
Backup retention has two passes with different authority. A migration's step 4 KNOWS the newest backup — this run's exact filename — and deletes every other `querylog.db.pre-migrate-*`; it is the primary mechanism. A plain successful open at current version runs a CONSERVATIVE retry for cleanups that once failed: parse `<epoch>` and the optional `-N` collision suffix from each name, delete only files whose epoch is STRICTLY below the maximum, keep every file tied at the maximum epoch, and never delete a name that does not parse. This pass EXPLICITLY assumes forward-moving wall clock between migrations (record the assumption in its doc comment): under a clock rollback an older high-epoch name could outrank a genuinely newer backup, which is why the authoritative exact-name pass in step 4 is the primary mechanism and this pass is only the retry for its failures.
### A.4 Running a migration (`runMigration`)
1. **Backup.** `VACUUM INTO` on the live connection (no open transaction) to `querylog.db.pre-migrate-<epoch>` in the database's directory. Destination must not pre-exist: on collision retry `-<epoch>-2`, `-3`, … The path enters the statement through an SQL string-literal quoting helper (double every `'`), never raw interpolation. On failure: delete the partial destination just created (only that file; an older valid backup survives), REFUSE with `error.MigrationBackupFailed`.
2. **One transaction.** `BEGIN IMMEDIATE`; re-read `user_version` under the lock. If it no longer equals `from`: ROLLBACK, delete this run's backup, REFUSE with `error.MigrationFailed` (exclusive ownership makes this outside interference). Otherwise `migrateSteps(db, sql, from, target)` — every step and the stamp in this one transaction — then COMMIT once.
3. **On PRE-COMMIT failure:** ROLLBACK, delete this run's backup, REFUSE with `error.MigrationFailed`, log the failing step index. Canonical file keeps its logical state. Never fall through to recreate.
3b. **On POST-COMMIT failure** (pragma restore or anything after a successful COMMIT): the file IS at `target` and that is said plainly in the log; the backup is KEPT (never deleted on this path). `runMigration` does NOT close the borrowed connection — it returns the distinct internal error `error.MigrationCommittedButUnclean`, and `querylog_schema.open`, which owns the handle and already has the sole error-path close, performs that one close and surfaces `error.MigrationFailed` to its caller. The next start takes the current-version lane cleanly. No post-commit path may claim the file unchanged or delete the backup.
4. **On success:** best-effort delete of every OTHER `querylog.db.pre-migrate-*` (keep this run's). Deletion errors warn and do not fail startup; A.3's every-open retention retries later. Log one line naming `from -> target` and the kept backup.
### A.5 Legacy restamp error mapping
The fingerprint→1 restamp is this milestone's only real mutation of operator data. It runs in one transaction; any failure (statement or commit) maps to `error.MigrationFailed`, rolls back, and leaves the legacy stamp and every row intact — REFUSE semantics, never recreate. Session A adds a test-only fault-injection seam to the db wrapper (`src/storage/db.zig`, following its existing `ReadTx.commit` injection style): one SQL-substring-matched one-shot seam on `Db.exec` covers statement and commit alike (both restamp statements pass through `Db.exec`), and the same seam drives the post-commit pragma-restore failure. Refusal paths log at `err`, which the test runner treats as failure, so `querylog_migrations.expected_failures` (begin/end/capturing, modelled on `db.read_tx_faults`) captures EXPECTED refusal logs per test; an unexpected refusal elsewhere still fails its test (as built 2026-08-28). Acceptance tests: the restamp forced to fail at (a) the statement and (b) the commit each leave `user_version == legacy_fingerprint` and the rows readable by a subsequent successful open.
### A.6 Schema equivalence oracle
`pub fn schemaEquivalent(gpa: std.mem.Allocator, a: *db.Db, b: *db.Db) (db.Error || std.mem.Allocator.Error)!bool` in `querylog_migrations.zig`. Two layers, both must agree:
1. **Textual, exact:** for every non-`sqlite_` object in `sqlite_schema` (tables, indexes, views, triggers), compare `(type, name, tbl_name, sql)` with `sql` compared byte-for-byte. No normalization: the A.2 rebuild rule guarantees a migrated table carries the verbatim fresh CREATE text, and a fresh file trivially does. This layer sees CHECK constraints, foreign keys, WITHOUT ROWID, partial-index predicates, trigger/view bodies.
2. **Structural belt:** per table, `PRAGMA table_xinfo` rows and `pragma_table_list` `wr`/`strict` flags; per table, `PRAGMA foreign_key_list`; per index, `PRAGMA index_xinfo` plus `index_list` `unique`/`origin`/`partial` flags.
Sort object and row lists before comparison. Negative tests: dropped `CHECK (rcode BETWEEN 0 AND 4095)`; dropped `REFERENCES domains(id)`; dropped `WITHOUT ROWID`; added column; and a table rebuilt via `ALTER TABLE ... RENAME` WITHOUT the verbatim-text rule compares UNEQUAL (proves the text layer catches SQLite's rename rewrite).
### A.7 Acceptance criteria
- [ ] Fresh file stamps `user_version = 1`, opens as current.
- [ ] A file stamped `1975011655` opens, restamps to 1, keeps every row; second open takes the current lane.
- [ ] `SchemaTooNew` and `SchemaUnsupported` refuse: schema dump, row count, watermark, stamp unchanged after refusal; no aside, no new file. One byte-hash variant on a checkpointed, sidecar-free fixture.
- [ ] Legacy-below-minimum ordering: with a test-local metadata view where minimum > 1 (drive the classification helper directly with injected constants — classification must be a pure function of `(stamped, minimum, current)` for exactly this reason), a legacy-fingerprint stamp classifies as REFUSE and no restamp happens.
- [ ] A.5 restamp-failure test.
- [ ] Synthetic chain through `runMigration` (1→3, two SQL steps, the second using the full A.2 rebuild sequence on a real table): backup exists, is a valid db, contains pre-migration rows; `user_version` lands on 3; rows survived; the rebuilt table's CREATE text equals the injected target text.
- [ ] Referenced-parent rebuild: a synthetic step rebuilds `domains` (referenced by `query_log`); after the migration, `query_log`'s stored FK text still says `REFERENCES domains(id)` (not `domains_old`), `PRAGMA foreign_key_check` is empty, and both pragmas read their defaults (`foreign_keys` per `applyPragmas`, `legacy_alter_table` off) after success AND after a forced failure.
- [ ] Mid-chain failure (step 2's SQL errors): canonical file logically unchanged (still version 1, rows intact), this run's backup deleted, an older backup preserved, `error.MigrationFailed`.
- [ ] `legacy_alter_table` pragma is OFF after both success and failure paths.
- [ ] Post-commit failure branch, driven through `openVersioned` with an injected synthetic plan (not by calling `runMigration` directly): force the pragma restore to fail after a successful COMMIT (fault seam) and assert: the file is at the target version with the migrated schema, the backup remains, the connection is closed exactly once (by the open path), that startup refuses with `error.MigrationFailed`, and the NEXT `openVersioned` under the same plan succeeds through the current-version lane.
- [ ] Backup retention: two successful synthetic migrations leave exactly one `pre-migrate-*`, the newer (step-4 authority, exact name). A directory seeded with an older epoch, a newest epoch, and a `-2` suffix tied at the newest epoch has a plain successful open delete only the older epoch — both max-epoch ties survive; an unparseable `pre-migrate-*` name survives untouched.
- [ ] Backup consistency: a row committed but not checkpointed (WAL-only) is present in the backup.
- [ ] `PRAGMA user_version` transactionality: set inside a transaction, ROLLBACK, original value observed.
- [ ] Oracle: fresh==fresh true; every A.6 negative test false; a `runMigration`-migrated file vs a fresh file at the target schema true.
- [ ] Grep scoped to `src/` and `tools/`: the `fingerprint_mismatch` identifier and the `schema-changed` aside-tag string are gone from active code (docs, specs, and changelog legitimately keep the words — the downgrade recovery text names the aside). Both suites green.
---
## Session B: cut gate inversion + fixture proof
### B.1 Fixtures
- `src/storage/testdata/querylog-v1-schema.sql` — the version-1 DDL frozen verbatim (today's `querylog_schema.ddl` text; the stamp is NOT part of it — the loader applies `PRAGMA user_version = 1`).
- `src/storage/testdata/querylog-v1-data.sql` — representative COHERENT content: query_log rows covering every `route_kind` and the NULL variants (qtype, cache_hit, response_time_us, upstream, forward_zone), matching `domains` rows, a non-default `available_since`, and `bucket_*` projection rows consistent with the raw rows. A fixture-validity test loads it and runs the projection-coherence oracle BEFORE any migration, so an incoherent fixture fails on its own.
- Immutable once shipped (header comment). From here on, every supported logical version in `[minimum_supported_version, current_version]` has a fixture pair — the current version's pair is the next migration's starting fixture, and an explicit break ships the new baseline pair.
The **fixture proof tests** (appended to `querylog_migrations.zig` by Session B, sequenced after A):
1. For EVERY starting version in `[minimum_supported_version, current_version)`: load that version's fixture pair, stamp it, run the real production chain, assert `schemaEquivalent` against a fresh-`ddl` db, every row survived, `available_since` preserved, projection coherence holds. Empty today; load-bearing without edits the day the chain grows.
2. The CURRENT version's fixture pair, stamped `current_version`, opens on the current lane, is `schemaEquivalent` to a fresh-`ddl` db, and passes projection coherence — the pair whose existence Gate 2 requires is thereby proven coherent, since the `[minimum, current)` loop never exercises it.
3. The legacy-stamp variant: a v1-fixture file stamped `1975011655` — while `minimum_supported_version == 1` it opens, restamps, and passes the same assertions as (2); the test is written against the classification helper's injected constants so that when a future break raises the minimum above 1, its companion assertion (legacy stamp + minimum > 1 REFUSES with `error.SchemaUnsupported`, file untouched) is already in the suite.
### B.2 The gate in tools/cut.zig
`cut` imports `querylog_versions` (pure, no SQLite — the link contract is why A.1 is separate). Two INDEPENDENT gates replace the disclose-a-reset gate. Let `prev_version` be the previous tag's `current_version` (parse `git show <tag>:src/storage/querylog_versions.zig` with the existing simple-extraction style; a tag predating the module means 1).
**Gate 1 — schema text.** Fingerprint the previous tag's DDL text vs the tree's. If changed, require ONE of:
- **Migration lane:** `current_version > prev_version` AND `prev_version >= minimum_supported_version` (the previous release's files are actually reachable — an explicit break can never wear this lane) AND the chain covers `[prev_version, current_version)` (with contiguous per-step files, that is `step_sql.len == current_version - minimum_supported_version` plus the fixture/file checks of Gate 2).
- **Explicit-break lane:** `current_version > prev_version` AND `minimum_supported_version == current_version` AND the changelog section contains BOTH "resets your query history" AND a `### Restoring your query history` heading with a non-empty body.
- Neither: FAIL.
**Gate 2 — migration metadata.** Runs INDEPENDENTLY of Gate 1 (catches data-only migrations and prefix edits when the DDL is unchanged):
- Every `src/storage/migrations/v<from>.sql` present at the previous tag: byte-identical in the tree; missing: FAIL.
- Every `src/storage/testdata/querylog-v*-{schema,data}.sql` present at the previous tag: byte-identical; missing: FAIL.
- A fixture pair exists for every version in `[minimum_supported_version, current_version]`: else FAIL.
- `current_version < prev_version`: FAIL (never regresses).
- `current_version > prev_version` with neither a new step file nor a break (`minimum == current`): FAIL.
- `current_version > prev_version` via new step(s) — REGARDLESS of whether the DDL fingerprint moved (data-only migrations included): the changelog section must contain "migrates your query log in place"; else FAIL.
- Let `prev_minimum` be the previous tag's `minimum_supported_version` (module absent at tag: 1). `minimum_supported_version < prev_minimum`: FAIL. `minimum_supported_version > prev_minimum` is ONLY acceptable as the full explicit break — `minimum == current` AND `current_version > prev_version` AND the break-lane changelog requirements — REGARDLESS of the DDL fingerprint; any other raise: FAIL (a release must never silently drop supported schemas).
- The tree's `legacy_fingerprint` is not the literal `1975011655`: FAIL (the legacy anchor is frozen forever; editing it strands unupgraded 0.0.12/0.0.13 files).
### B.3 Acceptance criteria
- [ ] Gate unit tests (pure functions over injected inputs, house style): unchanged schema + unchanged metadata passes; migration lane passes; explicit-break lane passes; changed schema with neither FAILS; break metadata (`minimum == current`) presented with the migration phrase FAILS Gate 1's migration lane; version bump with short chain FAILS; edited shipped step FAILS despite a version append; edited fixture FAILS; deleted step file FAILS; missing target-version fixture pair FAILS; version regression FAILS; version bump with no step and no break FAILS; data-only step (unchanged DDL) without the migration phrase FAILS; minimum regression FAILS; minimum raised without the full break FAILS (unchanged DDL variant included); edited `legacy_fingerprint` FAILS; previous tag without `querylog_versions.zig` maps to `prev_version == 1` and `prev_minimum == 1`.
- [ ] Fixture-validity test and fixture proof loop pass in the plain suite.
- [ ] `zig build cut` compiles; both suites green.
---
## Session C: docs (after A)
- `docs/how-to/troubleshoot.md`: new section "The server refuses to start over querylog.db" — `SchemaTooNew` (downgraded binary: return to the newer release, or restore the matching `pre-migrate` backup), `SchemaUnsupported` (file predates 0.0.12 or is foreign: not migratable; how to set it aside by hand if starting empty is acceptable), `MigrationFailed`/`MigrationBackupFailed` (the server never starts empty on its own; before the migration committed the file is untouched, and in the rare committed-but-unclean case the log says the migration DID complete, the backup is kept, and the next start simply proceeds).
- `docs/reference/` page on the query-log lifecycle: version stamp, in-place migration, one kept backup, the honest downgrade contract (downgrading to 0.0.13 or older RESETS the log — those binaries predate this contract; migration-aware binaries refuse cleanly), corruption as the only automatic recreate, the explicit-break contract (versioned, refused at startup, changelog carries restore instructions).
- Update the documents that still state the old contract: `PLAN.md`, `docs/explanation/architecture.md`, `specs/release-cut.md` — surgical edits to the stale sentences only.
Acceptance: prose accurate against A/B behavior, unwrapped lines, both suites still green.
---
## Module Layout
- `src/storage/querylog_versions.zig` — NEW: pure version/step metadata (cut-importable, no hooks by design).
- `src/storage/querylog_migrations.zig` — NEW: `migrateSteps`, `runMigration`, `schemaEquivalent`, fixture proof tests.
- `src/storage/migrations/` — one immutable SQL file per shipped step. NOT created this milestone (empty chain; git carries no empty directory) — the first real step creates it.
- `src/storage/querylog_schema.zig` — open-path rework, stamp change, lane deletions, every-open retention.
- `src/storage/testdata/querylog-v1-schema.sql`, `querylog-v1-data.sql` — NEW frozen fixtures.
- `src/storage/querylog_fixtures.zig` — NEW (Session B, as built): fixture loading and the survival oracle — full-content comparison against a pristine copy, each value encoded type-tag + byte-length + bytes so the comparison is injective (review round 2026-08-28).
- `tools/cut.zig` — two-gate rework.
- Session C's doc files.
## File Ownership
A: both new storage modules, `migrations/` dir, `querylog_schema.zig`, callers touched by lane deletion. B (after A): `tools/cut.zig`, `testdata/`, appends tests to `querylog_migrations.zig`, and makes the projection-coherence checker in `queries_repo.zig` `pub` (export-only edit — the checker is currently private to that file, which no session otherwise owns; B's fixture tests need it). C (after A): docs, `PLAN.md`, `specs/release-cut.md`. Orchestrator: CHANGELOG.md, spec sync.
A also owns the fault-injection seam addition in `src/storage/db.zig` (A.5).
## Changelog requirement (orchestrator)
This milestone's own changelog entry must disclose the one hazard neither gate can see: opening querylog.db under this release restamps it from the legacy fingerprint to version 1, so a LATER downgrade to 0.0.13 or older treats the numeric stamp as a fingerprint mismatch, renames the file to a `.schema-changed-<epoch>` aside, and starts an empty log. The restamp itself creates NO backup, so the accurate recovery is: return to a migration-aware release; stop the server; move the empty downgrade-created `querylog.db` out of the way AND delete its `querylog.db-wal`/`querylog.db-shm` sidecars (replaying the empty file's sidecars into the restored history would corrupt it — the recreate code documents this); move the downgrade-created `.schema-changed-<epoch>` aside back to `querylog.db`; start. The entry states the hazard and exactly that procedure.
## Acceptance Criteria (Milestone Complete)
- [ ] No code path recreates or sets aside a healthy querylog.db (grep proves the lane gone).
- [ ] A 0.0.13-created file (v1 schema + `1975011655` stamp) opens under the new binary with every row intact.
- [ ] Refusals and pre-commit migration failures leave the file logically untouched; a post-commit `MigrationFailed` leaves it successfully migrated to `target` (backup kept) and only refuses that one startup; the restamp is this milestone's only real mutation and its failure refuses without loss.
- [ ] The cut gate refuses: a schema change with neither lane, any edit to shipped steps or fixtures, a data-only migration without disclosure, and an explicit break without versioning + restore instructions.
- [ ] Both suites green, fmt clean.
## Anti-Requirements
- NO migration steps for pre-0.0.12 schemas (refusal with instructions is the contract).
- NO real chain step this milestone; synthetic chains live in tests only.
- NO migration hooks — steps are SQL files, period; a future need amends the design in its own spec.
- NO generic column-intersection salvage.
- NO `ALTER TABLE ADD/RENAME COLUMN` on kept tables in future steps (rebuild rule; recorded in doc comments, machine-enforced only via the oracle's exact-text layer).
- NO admin UI/API surface for migrations; startup log lines are the interface.
- NO config knob for backup retention.
- NO change to config.db handling.
+17
View File
@@ -54,3 +54,20 @@ Pure functions unit-tested: semver validation (accept/reject table incl. leading
- [ ] `just --list` shows the recipes; `just verify` passes locally.
- [ ] `zig build cut -- patch` derives the next version and refuses in preflight on a dirty tree or a missing changelog section, mutating nothing; `zig build cut -- 0.0.9` and `-- banana` refuse naming the three kinds.
- [ ] `zig build test` and `-Dintegration` 0 failed; `zig fmt --check` clean.
## Addendum: the schema gate (post-0.0.9)
> Superseded by milestone 38. The addendum below records the gate as it was first built, when `querylog.db` was never migrated. The server now versions and migrates that file in place (`docs/reference/query-log-lifecycle.md`), and the single disclose-a-reset check described here was replaced by the two independent gates of `specs/milestone-38.md` §B.2.
0.0.9 changed the `query_log` DDL and its announcement said nothing about it. At the time `querylog.db` was never migrated: the server stamped `PRAGMA user_version` with a CRC32 of the DDL text, and on a mismatch it renamed the file aside and created an empty one, so the first start after such a release destroyed the operator's query history. Nothing in the cut noticed, because nothing in the cut had ever read the schema.
`schema-gate` is a read-only preflight check beside the others. It compares releases, not commits:
1. `git ls-remote --tags origin`, and the highest `vMAJOR.MINOR.PATCH` strictly below the version being cut is the previous release. Strictly below, because a rerun may already see the tag it is cutting. What is kept is the OBJECT ID origin published for that tag — the peeled `^{}` commit where there is one — not the tag name: a local tag of the same name can be stale or replaced, and reading its tree would compare against a schema origin never shipped, which passes silently whenever that schema happens to match this one. No such tag PASSES trivially — a first release has nothing to compare against.
2. `git show <oid>:src/storage/querylog_schema.zig`, and `extractDdl` recovers the `ddl` constant from that source the way the compiler reads a multiline string: the lines after `pub const ddl: [:0]const u8 =` that begin with `\\`, stripped of indentation and the `\\`, joined with newlines, ending at the `;`. Blank lines and `//` comments may appear before, between and after the `\\` lines and contribute nothing, exactly as the compiler treats them. A test applies the same function to the file on disk and asserts the result fingerprints to `querylog_schema.fingerprint` — that equality is what makes the text scan trustworthy.
3. The old DDL goes through `querylog_schema.fingerprintOf`, factored out of the comptime `fingerprint` so the gate and the server share one hash rather than two copies of one expression. The tool imports the schema module (build.zig, `querylog_schema_mod`); only these two decls are referenced, so no SQLite symbol comes with them.
4. Equal fingerprints PASS. Different fingerprints require the `## [<v>]` changelog section to contain the literal phrase `resets your query history`; present PASSES, absent is a soft FAIL naming both fingerprints, the phrase and what the change costs.
Every step that cannot answer — the `ls-remote`, the `git show`, the extraction, an unreadable CHANGELOG.md — is a soft FAIL naming the step. A gate that does not know whether the schema moved must never report that it did not.
Fixing a FAIL is a sentence in the changelog, not a flag: there is no override, because the only thing the gate asks for is that the release notes be true.
+5 -5
View File
@@ -54,14 +54,14 @@ One question, answered over a period the reader chooses: what did the resolver d
Top to bottom, edge to edge:
1. **Four stat tiles**, neutral chrome throughout — no coloured accents; emphasis is typographic. Queries, Blocked (count and rate), Clients, Average response. Each tile carries the way into the rows behind its number: Queries and Blocked open Activity for exactly the bounds the stats response returned, Clients opens the clients page, and Average response has nothing to open.
1. **Four stat tiles**, neutral chrome throughout — no coloured accents; emphasis is typographic. Queries, Blocked (count and rate), Clients, Average response. Each tile carries the way into the rows behind its number: Queries and Blocked open Activity for exactly the bounds the overview response returned, Clients opens the clients page, and Average response has nothing to open.
2. **Queries over time** — the existing query-volume timeline, split blocked/cached/other, full width.
3. **Client activity over time** — one stacked series per named client plus "other", on the same bucket alignment as the timeline so the two charts share an x-axis. A client registered under a name is labelled by it, with the same precedence the query tables apply and the address kept as the title; colour keys on the address, so naming a client never repaints its series.
4. **Query types** and **Upstream servers** — two donuts, side by side above 1280px and stacked below, with the ring and its legend centred in the panel while stacked and left-anchored once they are a pair. Types are labelled by the admin's own `qtypeName()`; routes by route-kind labels and by the answering resolver or zone. Each donut's SVG is decoration (`aria-hidden`, `focusable="false"`); a visible legend and a visually hidden table are the accessible surface. An empty window says "No queries in this period." rather than drawing nothing.
Colours key on semantic identity — the qtype value, the client string, the `(route, source)` pair — so a rank change between two polls never repaints an entry. Charts stay lightweight SVG; no charting dependency.
**Window coherence, five requests.** Totals, timeseries, clients, types and routes are separate calls, and the page holds one window identified by `(period, since, until, coverage.available_since)` — the watermark joins the identity because retention advancing mid-page changes what the same span can answer for. A response is a member only if all four fields match. Rendering is per panel: a member renders, a panel still in flight shows its own loading state, a panel whose request failed shows its own error and Retry, and the members keep rendering throughout — a failed donut never blanks the charts. A response behind the window is refetched once per endpoint-keyed episode and, if it stays behind, that panel alone shows an error. This is window coherence, not data-snapshot coherence: live inserts between requests may shift counts slightly between panels, and that is accepted. One coverage notice for the page, from the window's watermark.
**One request, one snapshot (superseded 2026-08-27 by milestone 36; the paragraph below replaces the original five-request window-coherence design).** The page makes one call, `GET /api/overview?period=…`, whose body carries totals, timeseries, clients, types, routes and coverage from a single read transaction — data-snapshot coherence, so the panels cannot disagree and no reconciliation layer exists. Loading and error are page-level: one loading surface, one error with Retry for the whole Overview. The per-panel shells, layout, copy and accessibility surfaces are unchanged. One coverage notice for the page, from the response's watermark. A `keepPreviousData` body whose own `period` is not the selected one keeps the page in loading.
**The shell.** The header carries no protection display at all. The Pause/Resume control sits at the foot of the sidebar, above the version label, in both the desktop rail and the mobile drawer; it is the only global runtime action, and it belongs to the resolver rather than to any page. It still appears beside the detail of a query that was blocked. The control states a pause with itself — "Paused until 14:05", or "Paused" when the pause has no end — because "Resume" names an action without naming the state it would end, and with the indicator and the status rows both gone the sidebar is the only place a page other than Diagnostics can carry that fact. An active resolver gets no line; the button says Pause, which is the whole message. The line and the health strip read one `protection` condition through one clock format, so they cannot disagree. The Diagnostics navigation item carries a badge: the open-episode count, or a neutral "!" when the rollup is degraded with nothing open and when the latest health poll failed — an unknown must never read as healthy. It is hidden only when health data exists, the latest poll succeeded, and the rollup is ok with nothing open.
@@ -168,7 +168,7 @@ No response payloads, answer RR sets, EDNS data or packet bytes are stored. The
Privacy transforms apply to every new domain-bearing field, not only `domain`: with `hide_domains` on, matched names, CNAME targets and safe-search targets hide consistently.
A one-row `querylog_meta (created_at INTEGER NOT NULL)` table lets the stats and query APIs return a conservative `available_since`, which distinguishes "zero queries" from "history does not exist".
A one-row `querylog_meta (created_at INTEGER NOT NULL)` table lets the overview and query APIs return a conservative `available_since`, which distinguishes "zero queries" from "history does not exist".
### Historical query detail
@@ -208,9 +208,9 @@ Database mode uses the same information architecture with real edit actions, plu
`GET /api/queries` keeps keyset pagination and its filters; rows gain `rcode`, `route_kind`, `policy_action` and the short policy reason the table needs, and the body gains `coverage: {complete, available_since}`. `GET /api/queries/{id}` returns nested `request` / `policy` / `route` / `response` provenance. `GET /api/queries/live` sends the same object without `id`.
`GET /api/stats` and `/api/stats/timeseries` add `complete` and `available_since`.
**Superseded by milestone 36 (2026-08-27).** This section originally specified five per-panel endpoints — `GET /api/stats`, `/api/stats/timeseries`, `/api/stats/types`, `/api/stats/routes` and `/api/stats/clients`. Five requests could promise a shared window but never a shared snapshot, and each one scanned every raw row in it. They are replaced by a single `GET /api/overview?period=1h|24h|7d|30d`, which returns `{period, since, until, bucket_seconds, totals:{queries, blocked, clients, avg_response_time_us}, buckets:[{ts, queries, blocked, cached}], clients:[{client, buckets}], other, types:[{qtype, count}], routes:[{route, source, count}], coverage:{complete, available_since}}` — every field with the semantics the five bodies gave it, over one deferred SQLite read transaction, so the breakdowns and the coverage watermark describe one database state. The 24h, 7d and 30d windows are served from 30-minute projection tables maintained transactionally beside the raw rows; the 1h window takes one raw scan. Per-period response caching keyed on `(window.until, PRAGMA data_version)` lives in the web layer.
**Three period aggregations (added 2026-08-22)** to feed the new Overview panels, all taking the same `period` parameter and reporting over the same aligned window, and all reading their rows and their coverage watermark inside one deferred SQLite read transaction. `GET /api/stats/types``{period, since, until, coverage, types:[{qtype, count}]}`, the numeric type only — naming types stays the admin's job, and a second table in the server would drift out of agreement with it — with the rows that recorded no type kept as their own `null` group. `GET /api/stats/routes``{period, since, until, coverage, routes:[{route, source, count}]}`, grouping `upstream` rows by the answering resolver and `forward_zone` rows by the zone, with blocked, cache, local and rejected carrying no source. `GET /api/stats/clients``{period, since, until, bucket_seconds, coverage, clients:[{client, buckets}], other}`, bucketed exactly as `/api/stats/timeseries`, the eight busiest clients named and everything else summed into `other`, which is always present and always bucket-count-sized. No new writers and no new state: all three are pure reads over the query log's provenance columns.
The panel semantics the five endpoints defined all carry over unchanged: types are the numeric type only — naming types stays the admin's job, and a second table in the server would drift out of agreement with it — with the rows that recorded no type kept as their own `null` group; routes group `upstream` rows by the answering resolver and `forward_zone` rows by the zone, with blocked, cache, local and rejected carrying no source; clients name the eight busiest and sum everything else into `other`, which is always present and always bucket-count-sized.
Existing mutation endpoints stay specific. Diagnostics introduces no generic "perform remediation" endpoint; it invokes the existing blocklist-refresh and certificate-reload operations.
+277 -361
View File
@@ -28,7 +28,6 @@ const Allocator = std.mem.Allocator;
const Certificate = std.crypto.Certificate;
const Writer = std.Io.Writer;
const net = std.Io.net;
const tls = std.crypto.tls;
const api_limiter = @import("web/api_limiter.zig");
const auth = @import("web/auth.zig");
@@ -40,9 +39,7 @@ const config_export = @import("config/export.zig");
const db = @import("storage/db.zig");
const disk_monitor = @import("storage/disk_monitor.zig");
const dns_cache = @import("cache/dns_cache.zig");
const doh_client = @import("upstream/doh_client.zig");
const doh_server = @import("server/doh_server.zig");
const dot_client = @import("upstream/dot_client.zig");
const dot_server = @import("server/dot_server.zig");
const events = @import("storage/events.zig");
const faults = @import("config/faults.zig");
@@ -53,26 +50,26 @@ const http_util = @import("web/http_util.zig");
const loader = @import("config/loader.zig");
const local_records = @import("local/records.zig");
const local_tables = @import("server/local_tables.zig");
const logger_mod = @import("storage/logger.zig");
const logger_controller = @import("storage/logger_controller.zig");
const logging = @import("platform/logging.zig");
const manager_mod = @import("filter/manager.zig");
const migrations = @import("storage/migrations.zig");
const model = @import("config/model.zig");
const pause = @import("server/pause.zig");
const pool_mod = @import("upstream/pool.zig");
const queries_repo = @import("storage/repositories/queries_repo.zig");
const query_sink = @import("server/query_sink.zig");
const querylog_migrations = @import("storage/querylog_migrations.zig");
const querylog_schema = @import("storage/querylog_schema.zig");
const rate_limiter = @import("server/rate_limiter.zig");
const reconcile = @import("config/reconcile.zig");
const retention_mod = @import("storage/retention.zig");
const safe_url = @import("safe_url.zig");
const shutdown = @import("server/shutdown.zig");
const sse = @import("web/sse.zig");
const static = @import("web/static.zig");
const tcp_server = @import("server/tcp_server.zig");
const transport = @import("upstream/transport.zig");
const udp_server = @import("server/udp_server.zig");
const upstream_owner = @import("upstream/owner.zig");
const validate = @import("config/validate.zig");
const version = @import("version.zig");
const web_server = @import("web/server.zig");
@@ -89,11 +86,6 @@ const maintenance_interval_s = 60;
/// takes longer than this is not going to finish at all.
const download_budget_s = 300;
/// Per DoH upstream. The sizes live in `doh_client.zig` so that `nxdns check`
/// probes the buffers `nxdns run` serves with.
const doh_request_buf_len = doh_client.default_request_buf_len;
const doh_transfer_buf_len = doh_client.default_transfer_buf_len;
pub fn run(runner: cli.Runner, args: cli.RunArgs) u8 {
const code = serve(runner, args) catch |err| code: {
runner.err.print("nxdns run failed: {s}\n", .{@errorName(err)}) catch {};
@@ -220,17 +212,13 @@ fn reconcileFromFileAt(
return result;
}
/// An upstream's identity is its url: the whole url is the key, and the
/// redaction is the label, because a url can carry an account token.
/// Boot's replay of one upstream finding. The rendering is the owner's, shared
/// with the runtime reconciler so a warning raised at boot and the same warning
/// raised by a settings PUT are byte-identical.
fn noteUpstream(config_load: *ConfigLoad, url: []const u8, message: []const u8) void {
var label_buf: [events.Store.max_subject_label_len]u8 = undefined;
const label = std.fmt.bufPrint(&label_buf, "{f}", .{safe_url.redact(url)}) catch &label_buf;
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&detail_buf, "upstream {f} {s}", .{
safe_url.redactQuoted(url),
message,
}) catch &detail_buf;
config_load.note(url, label, detail);
var rendered: upstream_owner.Rendered = .{};
rendered.render(.{ .url = url, .message = message });
config_load.note(url, rendered.label, rendered.detail);
}
/// Returns the moment the transaction committed, which is what the settings
@@ -529,49 +517,50 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
defer bundle.deinit(gpa);
var bundle_lock: std.Io.RwLock = .init;
var upstreams = try Upstreams.build(io, gpa, cfg.upstreams, &dns_http, &bundle, &bundle_lock, &config_load);
defer upstreams.deinit(io, gpa);
var pool: pool_mod.Pool = .init(
upstreams.active(),
.{},
.{
const upstream_generation = try upstream_owner.build(.{
.gpa = gpa,
.io = io,
.servers = cfg.upstreams,
.http = &dns_http,
.bundle = &bundle,
.bundle_lock = &bundle_lock,
.timeouts = .{
.attempt = .{ .raw = model.attemptTimeout(cfg.upstream), .clock = .awake },
.total = .{ .raw = model.totalTimeout(cfg.upstream), .clock = .awake },
},
@truncate(@as(u96, @bitCast(std.Io.Clock.real.now(io).nanoseconds))),
);
.seed = @truncate(@as(u96, @bitCast(std.Io.Clock.real.now(io).nanoseconds))),
.diagnostics = event_store,
});
pool.diagnostics = event_store;
// `build` writes no event rows, so that a settings PUT can prepare a
// candidate that is never published without leaving a trace. Boot has no
// such candidate: this generation is the one that serves, and replaying its
// report through the collector is what keeps the `configuration.load`
// episodes — and the keys `finalize` below spares — exactly what they were
// when this composition lived in this file.
for (upstream_generation.report().notes) |finding| {
noteUpstream(&config_load, finding.url, finding.message);
}
const upstream_count = upstream_generation.activeCount();
var upstreams: upstream_owner.Owner = .init(upstream_generation);
defer upstreams.deinit(io);
// -----------------------------------------------------------------------
// per-query state
// -----------------------------------------------------------------------
var cache: dns_cache.DnsCache = try .init(gpa, cfg.cache);
defer cache.deinit();
var limiter: rate_limiter.RateLimiter = try .init(gpa, .{
.limit = cfg.dns.rate_limit,
.window_seconds = cfg.dns.rate_window_seconds,
});
defer limiter.deinit();
var paused: pause.Pause = .{};
var tracker: clients.Tracker = .init(cfg.logging.retention_days);
// One cell for both retention consumers, owned here so a settings apply
// moves the daily query-log prune and the stale-client prune together.
var retention_days: retention_mod.RetentionDays = .init(cfg.logging.retention_days);
var tracker: clients.Tracker = .init(&retention_days);
tracker.diagnostics = event_store;
// Naming rides the tracker's pass, on the tracker's task and connection
// (milestone-25 ruling 1), and reads the live forward zones.
var client_names_resolver: client_names.Resolver = .init(&tables);
client_names_resolver.diagnostics = event_store;
// The queue holds waiting tasks in intrusive lists, so neither the buffer
// nor the `Logger` may move once a task has touched either.
const queue_buf = try gpa.alloc(logger_mod.Entry, cfg.logging.query_log_buffer_max);
defer gpa.free(queue_buf);
var query_logger: logger_mod.Logger = .init(cfg.logging, queue_buf);
query_logger.diagnostics = event_store;
// Milestone 8 fans every logged query out to the SSE hub as well. The hub
// exists only when the web interface does (ruling 6) — without it the sink
// costs the query path one null check. Its rings are ~900 KiB, so it lives
@@ -584,18 +573,20 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
created.init();
hub = created;
}
var sink: query_sink.QuerySink = .init(&query_logger, hub);
// -----------------------------------------------------------------------
// disk, retention and the remaining connections (ruling 21)
// -----------------------------------------------------------------------
const data_path = try arena.dupeZ(u8, paths.data_dir);
const log_dir_path: ?[:0]const u8 = if (cfg.logging.output == .file)
try arena.dupeZ(u8, std.fs.path.dirname(cfg.logging.file_path) orelse ".")
const log_dir_path: ?[:0]const u8 = if (logging.logDirname(cfg.logging)) |dir|
try arena.dupeZ(u8, dir)
else
null;
var monitor: disk_monitor.Monitor = .init(cfg.disk, data.dir, data_path, log_dir_path);
// Frees whatever log-directory generation a settings apply installed; boot's
// path is borrowed from the arena and owned by nobody here.
defer monitor.deinit(io);
// Ruling 17. The scheduler consults it before every scheduled pass; the
// startup `reload` below is an operator action and stays ungated.
@@ -618,13 +609,31 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// policy and the right one here too.
monitor.sample(io, event_store, boot_now_s);
var retention: retention_mod.Retention = .init(cfg.logging);
var retention: retention_mod.Retention = .init(&retention_days);
var querylog_opened = try data.openQuerylogDb(io);
var querylog_writer_db = querylog_opened.database;
defer querylog_writer_db.close();
reportQuerylogRecreated(event_store, io, boot_now_s, &querylog_opened, &querylog_writer_db);
// The controller adopts that first connection and owns the query logger
// from here: the buffer, the `Logger`, the writer task and the connection
// are one generation, and `logging.query_log_buffer_max` can replace all
// four while the server runs (milestone 34 §S4). Its writer starts now and
// parks on an empty queue, which is where it would be anyway — no producer
// exists until the listeners below start serving.
var log_controller: logger_controller.Controller = undefined;
try log_controller.init(io, .{
.gpa = gpa,
.database = querylog_writer_db,
.source = .{ .dir = std.Io.Dir.cwd(), .path = data.querylog_db_path },
.logging = cfg.logging,
.monitor = &monitor,
.diagnostics = event_store,
});
defer log_controller.deinit(io);
var sink: query_sink.QuerySink = .init(&log_controller, hub);
var querylog_retention_db = try data.reopenQuerylogDb(io);
defer querylog_retention_db.close();
var tracker_db = try data.openConfigDb(io);
@@ -683,20 +692,56 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// listeners
// -----------------------------------------------------------------------
// Both are on the heap and both are freed through the handler rather than
// through this frame: a `cache.size` or `dns.rate_limit` change swaps in a
// replacement built with this same `gpa` and frees what it displaced, so
// what the handler holds at shutdown is not necessarily what boot built.
// Both are built inside one block so their errdefers end with it: after
// the block the handler is the sole owner, and the teardown defers below
// are what free them. An errdefer that outlived the block would free the
// same object those defers free.
const both = blk: {
const c = try gpa.create(dns_cache.DnsCache);
errdefer gpa.destroy(c);
c.* = try .init(gpa, cfg.cache);
errdefer c.deinit();
const l = try gpa.create(rate_limiter.RateLimiter);
errdefer gpa.destroy(l);
l.* = try .init(gpa, .{
.limit = cfg.dns.rate_limit,
.window_seconds = cfg.dns.rate_window_seconds,
});
break :blk .{ .cache = c, .limiter = l };
};
const cache = both.cache;
const limiter = both.limiter;
var h: handler.Handler = .{
.upstream = pool.client(),
.upstream = &upstreams,
.policy = .{
.blocking = .{ .mode = cfg.blocking.response, .ttl = cfg.blocking.ttl },
.ecs_mode = cfg.edns.ecs_mode,
.forward_read_timeout = .{ .raw = model.readTimeout(cfg.upstream), .clock = .awake },
.negative_ttl_max = cfg.cache.negative_ttl_max,
},
.manager = &manager,
.local_tables = &tables,
.cache = &cache,
.negative_ttl_max = cfg.cache.negative_ttl_max,
.limiter = &limiter,
.cache = cache,
.limiter = limiter,
.sink = &sink,
.pause = &paused,
.tracker = &tracker,
};
// These run after `group.cancel` below, so no query is inside either table.
defer if (h.replaceCache(io, null)) |live| {
live.deinit();
gpa.destroy(live);
};
defer if (h.replaceRateLimiter(io, null)) |live| {
live.deinit();
gpa.destroy(live);
};
// -----------------------------------------------------------------------
// DoH/DoT listeners (milestone-10 ruling 11)
@@ -770,9 +815,16 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// The live hash may own a gpa replacement after a settings PUT; this defer
// runs after `group.cancel` below, so no web task can still read it.
defer web_state.live_hash.deinit(gpa);
// Same argument as the live hash: a settings PUT may have installed an
// owned generation, and this runs after `group.cancel`.
defer web_state.proxies.deinit(gpa);
// The Overview response cache owns its bodies from `gpa`. Same argument
// again: no web task can still be reading a slot once the group is cancelled.
defer web_state.overview_cache.deinit(gpa);
if (cfg.web.enabled) web_state = .{
.gpa = gpa,
.web = cfg.web,
.proxies = .init(cfg.web.trusted_proxies),
.authority = authority,
.reconciled_at = reconciled_at,
.live_hash = .init(cfg.web.password_hash orelse ""),
@@ -781,11 +833,17 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
.tracker = &tracker,
.client_names = &client_names_resolver,
.manager = &manager,
.pool = &pool,
.upstreams = &upstreams,
.upstream_build = .{
.http = &dns_http,
.bundle = &bundle,
.bundle_lock = &bundle_lock,
},
.monitor = &monitor,
.local_tables = &tables,
.logger = &query_logger,
.logger = &log_controller,
.retention = &retention,
.retention_days = &retention_days,
.sessions = if (sessions) |*s| s else null,
.limiter = if (web_limiter) |*l| l else null,
.hub = hub,
@@ -894,25 +952,20 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// (disk_monitor.zig:63), so nothing is refused for want of a sample.
const gate: ?*disk_monitor.Monitor = &monitor;
// The query-log writer is deliberately *not* in `group`, and starts before
// every producer. Inside the group its life would end with the same
// `cancel` that stops the producers, and cancellation would race the
// queue's close: whichever landed first decided whether the batch the
// writer was holding reached the database or was counted as dropped. Given
// its own future, it outlives the producers by construction, and the
// teardown below can close the queue with nobody left to fill it and then
// wait for the writer to finish emptying it.
var writer_future = try io.concurrent(
logger_mod.Logger.runWriter,
.{ &query_logger, io, &querylog_writer_db, gate },
);
// The query-log writers are deliberately *not* in `group`, and the live one
// started before every producer. Inside the group their lives would end
// with the same `cancel` that stops the producers, and cancellation would
// race the queue's close: whichever landed first decided whether the batch
// a writer was holding reached the database or was counted as dropped. The
// controller owns their futures instead, so they outlive the producers by
// construction.
//
// Ruling 4's shutdown order, on the one path every exit from here takes:
// every producer stops and is joined, then the queue closes, then the
// writer is awaited — so the last batch is written rather than raced. A
// writer the disk gate will not let write counts its batch as dropped
// instead of holding the exit open (`logger.zig`), so this wait always
// ends.
// every producer stops and is joined, then `Controller.shutdown` closes
// each queue and awaits each writer — so the last batch is written rather
// than raced. A writer the disk gate will not let write counts its batch as
// dropped instead of holding the exit open (`logger.zig`), so this wait
// always ends.
//
// A `defer` and not straight-line code after `shutdown.wait`, because a
// `concurrent` spawn below can fail with the DNS listeners already
@@ -920,8 +973,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// signal gets.
defer {
group.cancel(io);
query_logger.shutdown(io);
writer_future.await(io) catch {};
log_controller.shutdown(io);
}
if (udp6) |*s| try group.concurrent(io, udp_server.UdpServer.serve, .{ s, io });
@@ -946,7 +998,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// this box exists for — keeps serving.
if (cfg.web.enabled) try group.concurrent(io, web_server.serve, .{ &web_state, io });
logStartup(io, authority, &manager, upstreams.active().len, .{
logStartup(io, authority, &manager, upstream_count, .{
.udp6 = if (udp6) |*s| s.boundAddress() else null,
.udp4 = if (udp4) |*s| s.boundAddress() else null,
.tcp6 = if (tcp6) |*s| s.boundAddress() else null,
@@ -1121,18 +1173,21 @@ fn maintenanceOnce(
api: ?*api_limiter.ApiLimiter,
io: std.Io,
) std.Io.Cancelable!void {
if (h.cache) |cache| {
// Both pointers are read inside the mutex that guards their replacement,
// the same discipline the query path follows: a load taken before the lock
// could sweep a table `replaceCache`/`replaceRateLimiter` has just freed.
{
const now_s = std.Io.Clock.real.now(io).toSeconds();
try h.cache_mutex.lock(io);
_ = cache.sweep(now_s);
h.cache_mutex.unlock(io);
defer h.cache_mutex.unlock(io);
if (h.cache) |cache| _ = cache.sweep(now_s);
}
if (h.limiter) |limiter| {
{
const now = std.Io.Clock.awake.now(io);
try h.limiter_mutex.lock(io);
_ = limiter.sweep(now);
h.limiter_mutex.unlock(io);
defer h.limiter_mutex.unlock(io);
if (h.limiter) |limiter| _ = limiter.sweep(now);
}
// The API limiter takes its own mutex, unlike the two above, which are the
@@ -1140,215 +1195,6 @@ fn maintenanceOnce(
if (api) |limiter| _ = limiter.sweep(io, std.Io.Clock.awake.now(io));
}
// ---------------------------------------------------------------------------
// upstreams
// ---------------------------------------------------------------------------
/// The pool's entries and everything they point into.
///
/// Every enabled upstream gets `pool_mod.slots_per_entry` leaf clients, one per
/// slot of its entry, so that many exchanges can be in flight against it at
/// once. `Slot.client` is a type-erased pointer into `doh` or `dot`, each of
/// those clients borrows a slice of `doh_buf`/`dot_buf`, and each entry borrows
/// a run of `slot_storage` and one counter of `recovery_counters` — so every
/// allocation here lives exactly as long as the pool does, and none of them is
/// ever resized. One slot is used by one task at a time, which is why the
/// buffers are per client and not shared the way `cli.probeUpstreams` shares
/// them.
const Upstreams = struct {
entries: []pool_mod.Entry,
used: usize,
/// Sliced per entry into `Entry.slots`, never pointing into the client
/// arrays: `Pool.init` sorts entries and the slices have to survive it.
slot_storage: []pool_mod.Slot,
/// One per enabled upstream, and the reason it is a separate allocation:
/// `Pool.init` sorts entries by value, so a counter living inside an entry
/// would be pointed at by the wrong upstream's clients after the sort.
recovery_counters: []std.atomic.Value(u64),
doh: []doh_client.DohClient,
dot: []dot_client.DotClient,
/// How much of `doh`/`dot` was actually initialized. A malformed or skipped
/// upstream leaves the tail of an over-allocated array undefined, and both
/// `deinit` and `build`'s failure paths iterate only the initialized
/// prefix — reading a `DotClient` that was never built, or closing a
/// session that was never opened, is what these two counts prevent.
doh_used: usize,
dot_used: usize,
doh_buf: []u8,
dot_buf: []u8,
/// A disabled upstream is left out entirely; a malformed one warns and is
/// skipped, because one bad row in a table of four must not take DNS down.
/// No usable row at all is a configuration fault.
fn build(
io: std.Io,
gpa: Allocator,
servers: []const model.UpstreamServer,
http: *std.http.Client,
bundle: *Certificate.Bundle,
bundle_lock: *std.Io.RwLock,
config_load: *ConfigLoad,
) (Allocator.Error || error{NoUsableUpstreams})!Upstreams {
var enabled: usize = 0;
for (servers) |server| {
if (server.enabled) enabled += 1;
}
if (enabled == 0) return error.NoUsableUpstreams;
const chunk = tls.Client.min_buffer_len;
const slots = pool_mod.slots_per_entry;
const leaf_clients = enabled * slots;
var self: Upstreams = .{
.entries = try gpa.alloc(pool_mod.Entry, enabled),
.used = 0,
.slot_storage = &.{},
.recovery_counters = &.{},
.doh = &.{},
.dot = &.{},
.doh_used = 0,
.dot_used = 0,
.doh_buf = &.{},
.dot_buf = &.{},
};
errdefer self.deinit(io, gpa);
self.slot_storage = try gpa.alloc(pool_mod.Slot, leaf_clients);
self.recovery_counters = try gpa.alloc(std.atomic.Value(u64), enabled);
for (self.recovery_counters) |*counter| counter.* = .init(0);
self.doh = try gpa.alloc(doh_client.DohClient, leaf_clients);
self.dot = try gpa.alloc(dot_client.DotClient, leaf_clients);
self.doh_buf = try gpa.alloc(u8, leaf_clients * (doh_request_buf_len + doh_transfer_buf_len));
self.dot_buf = try gpa.alloc(u8, leaf_clients * 4 * chunk);
for (servers) |server| {
if (!server.enabled) continue;
const endpoint = transport.Endpoint.parse(server.url) catch {
log.warn(
"upstream {f} is not an https:// or tls:// endpoint; skipped",
.{safe_url.redactQuoted(server.url)},
);
noteUpstream(config_load, server.url, "not an https:// or tls:// endpoint; skipped");
continue;
};
const entry_slots = self.slot_storage[self.used * slots ..][0..slots];
switch (endpoint.scheme) {
.doh => if (!self.wireDoh(http, endpoint, entry_slots)) {
log.warn(
"upstream {f} is not a usable DoH url; skipped",
.{safe_url.redactQuoted(server.url)},
);
noteUpstream(config_load, server.url, "not a usable DoH url; skipped");
continue;
},
.dot => self.wireDot(gpa, endpoint, server.tls_name, bundle, bundle_lock, entry_slots),
}
self.entries[self.used] = .{
.endpoint = endpoint,
.slots = entry_slots,
.priority = server.priority,
.enabled = true,
.health = .init,
.sem = .{ .permits = entry_slots.len },
.reuse_recoveries = &self.recovery_counters[self.used],
};
self.used += 1;
}
if (self.used == 0) return error.NoUsableUpstreams;
return self;
}
/// One `DohClient` per slot, all sharing the one `std.http.Client`: its
/// connection pool already serves concurrent requests, and a `DohClient`'s
/// only mutable state is the two buffers this gives each slot its own of.
///
/// False means the url is not a usable DoH url, which `DohClient.init`
/// decides from the url alone — so it fails on the first slot or on none.
/// `doh_used` still advances per client rather than per entry: it means
/// "initialized", and a skipped entry's clients are simply never reached.
fn wireDoh(
self: *Upstreams,
http: *std.http.Client,
endpoint: transport.Endpoint,
slots: []pool_mod.Slot,
) bool {
for (slots) |*slot| {
const index = self.doh_used;
const base = index * (doh_request_buf_len + doh_transfer_buf_len);
self.doh[index] = doh_client.DohClient.init(
http,
endpoint,
self.doh_buf[base..][0..doh_request_buf_len],
self.doh_buf[base + doh_request_buf_len ..][0..doh_transfer_buf_len],
) catch return false;
self.doh_used = index + 1;
slot.* = .{ .client = self.doh[index].client() };
}
return true;
}
/// One `DotClient` per slot, each with its own four TLS buffers and all
/// sharing the trust store. Every client of one entry reports its stale-reuse
/// recoveries through that entry's counter.
fn wireDot(
self: *Upstreams,
gpa: Allocator,
endpoint: transport.Endpoint,
tls_name: []const u8,
bundle: *Certificate.Bundle,
bundle_lock: *std.Io.RwLock,
slots: []pool_mod.Slot,
) void {
const chunk = tls.Client.min_buffer_len;
const recoveries = &self.recovery_counters[self.used];
for (slots) |*slot| {
const index = self.dot_used;
const base = index * 4 * chunk;
self.dot[index] = dot_client.DotClient.init(
endpoint,
tls_name,
gpa,
bundle,
bundle_lock,
recoveries,
.{
.tls_read = self.dot_buf[base..][0..chunk],
.tls_write = self.dot_buf[base + chunk ..][0..chunk],
.stream_read = self.dot_buf[base + 2 * chunk ..][0..chunk],
.stream_write = self.dot_buf[base + 3 * chunk ..][0..chunk],
},
);
self.dot_used = index + 1;
slot.* = .{ .client = self.dot[index].client() };
}
}
/// The prefix `Pool.init` is given. The rest of `entries` is allocated but
/// never filled, which is what keeps `deinit` able to free the whole block.
fn active(self: *Upstreams) []pool_mod.Entry {
return self.entries[0..self.used];
}
/// Connections first, memory second: a `DotClient` holds a socket its
/// buffers belong to, so nothing it points at may be freed before it is
/// closed.
fn deinit(self: *Upstreams, io: std.Io, gpa: Allocator) void {
for (self.dot[0..self.dot_used]) |*client| client.close(io);
gpa.free(self.dot_buf);
gpa.free(self.doh_buf);
gpa.free(self.dot);
gpa.free(self.doh);
gpa.free(self.recovery_counters);
gpa.free(self.slot_storage);
gpa.free(self.entries);
self.* = undefined;
}
};
// ---------------------------------------------------------------------------
// listeners
// ---------------------------------------------------------------------------
@@ -1471,9 +1317,9 @@ test "the recreated detail names the aside and the new coverage start" {
var buf: [events.Store.max_detail_len]u8 = undefined;
try std.testing.expectEqualStrings(
"previous file kept as 'querylog.db.schema-changed-1700000000'; " ++
"previous file kept as 'querylog.db.quick-check-failed-1700000000'; " ++
"query history is available from 1700000001",
recreatedDetail(&buf, "querylog.db.schema-changed-1700000000", 1700000001),
recreatedDetail(&buf, "querylog.db.quick-check-failed-1700000000", 1700000001),
);
// A fresh file that will not answer is a separate failure; the line still
@@ -1592,7 +1438,7 @@ const m29_ddl: [:0]const u8 =
\\VALUES (1, unixepoch(), unixepoch() + 1);
;
test "an m29 query log is set aside and recreated without the upstream-history tables" {
test "an m29 query log refuses the startup and is left exactly as it is" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
@@ -1603,10 +1449,6 @@ test "an m29 query log is set aside and recreated without the upstream-history t
var path_buf: [256]u8 = undefined;
const path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/querylog.db", .{tmp.sub_path});
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
// The fixture is only worth anything while it is still a *different*
// schema from this build's, and one that carries the deleted tables.
try testing.expect(!std.mem.eql(u8, m29_ddl, querylog_schema.ddl));
@@ -1616,9 +1458,8 @@ test "an m29 query log is set aside and recreated without the upstream-history t
// edit to the literal cannot satisfy by changing what it is compared to.
try testing.expectEqual(m29_fingerprint, @as(i32, @bitCast(std.hash.Crc32.hash(m29_ddl))));
// A healthy m29 file, stamped with the fingerprint m29's own DDL produced
// and backdated so its coverage promise is visibly the older one.
const m29_coverage = blk: {
// A healthy m29 file, stamped with the fingerprint m29's own DDL produced.
{
var m29 = try db.Db.open(path, .{ .mode = .read_write_create });
defer m29.close();
try db.applyPragmas(&m29, .{});
@@ -1636,43 +1477,47 @@ test "an m29 query log is set aside and recreated without the upstream-history t
"PRAGMA user_version = {d};",
.{m29_fingerprint},
));
break :blk try m29.queryInt("SELECT available_since FROM querylog_meta");
};
try testing.expectEqual(m29_available_since, m29_coverage);
var opened = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer opened.database.close();
// Set aside under the name that says the file was healthy and this build
// moved, and still on disk for an operator who wants it.
try testing.expectEqual(querylog_schema.RecreateReason.fingerprint_mismatch, opened.recreated.?);
try testing.expect(std.mem.indexOf(u8, opened.aside(), ".schema-changed-") != null);
try tmp.dir.access(io, std.fs.path.basename(opened.aside()), .{});
// The two tables are gone from the file this process will write to.
for ([_][]const u8{ "upstream_targets", "upstream_minute", "idx_upstream_minute_ts" }) |name| {
var stmt = try opened.database.prepare("SELECT count(*) FROM sqlite_schema WHERE name = ?1");
defer stmt.deinit();
try stmt.bindText(1, name);
try testing.expect(try stmt.step());
try testing.expectEqual(@as(i64, 0), stmt.columnInt(0));
}
// Coverage restarts: the new file does not inherit the replaced one's
// promise about what it can answer. Strictly newer, not merely not-older —
// a recreation that copied the watermark across would pass the weaker test.
const coverage = try queries_repo.availableSince(&opened.database);
try testing.expect(coverage > m29_coverage);
// m29 predates the version stamp entirely: its `user_version` is a CRC of a
// schema no migration chain starts from, so the only honest answer is to
// refuse and say so. The pre-0.0.12 contract — set it aside and start empty
// — is gone.
querylog_migrations.expected_failures.begin();
defer querylog_migrations.expected_failures.end();
try testing.expectError(
error.SchemaUnsupported,
querylog_schema.open(io, std.Io.Dir.cwd(), path),
);
reportQuerylogRecreated(&fx.store, io, 2000, &opened, &opened.database);
try testing.expectEqualStrings("query_log.recreated", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings(
"fingerprint_mismatch",
try fx.text("SELECT subject_key FROM operational_events"),
// Nothing was renamed, nothing was created, and the file still answers for
// itself: the operator can downgrade and keep the history.
var entries: usize = 0;
var it = tmp.dir.iterate();
while (try it.next(io)) |entry| {
try testing.expect(std.mem.startsWith(u8, entry.name, "querylog.db"));
try testing.expect(std.mem.indexOfScalar(u8, entry.name[10..], '.') == null);
entries += 1;
}
try testing.expect(entries >= 1);
var reopened = try db.Db.open(path, .{ .mode = .read_write_existing });
defer reopened.close();
try testing.expectEqual(
@as(i64, m29_fingerprint),
try reopened.queryInt("PRAGMA user_version"),
);
try testing.expectEqual(
m29_available_since,
try reopened.queryInt("SELECT available_since FROM querylog_meta"),
);
try testing.expectEqual(
@as(i64, 1),
try reopened.queryInt("SELECT count(*) FROM upstream_targets"),
);
}
test "a fingerprint recreate files a resolved event naming the real aside and watermark" {
test "a recreate files a resolved event naming the real aside and watermark" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
@@ -1694,27 +1539,17 @@ test "a fingerprint recreate files a resolved event naming the real aside and wa
created.database.close();
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
// A healthy file this build's DDL no longer matches, which is what an
// upgrade that edits the schema produces.
{
var stamped = try db.Db.open(path, .{ .mode = .read_write_existing });
defer stamped.close();
var sql_buf: [64]u8 = undefined;
try stamped.exec(try std.fmt.bufPrintZ(
&sql_buf,
"PRAGMA user_version = {d};",
.{querylog_schema.fingerprint +% 1},
));
}
// Real damage: corruption is the only thing that recreates now.
try tmp.dir.writeFile(io, .{ .sub_path = "querylog.db", .data = "not a database at all" });
var recreated = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer recreated.database.close();
try testing.expectEqual(querylog_schema.RecreateReason.fingerprint_mismatch, recreated.recreated.?);
try testing.expectEqual(querylog_schema.RecreateReason.not_a_database, recreated.recreated.?);
reportQuerylogRecreated(&fx.store, io, 2000, &recreated, &recreated.database);
try testing.expectEqualStrings("query_log.recreated", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("fingerprint_mismatch", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("not_a_database", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("warning", try fx.text("SELECT severity FROM operational_events"));
// One-shot: already over when it is filed, so it never becomes an open
// episode `/api/health` counts.
@@ -1767,18 +1602,17 @@ test "a recreate under a long data directory keeps the watermark and a usable na
{
var created = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer created.database.close();
var sql_buf: [64]u8 = undefined;
try created.database.exec(try std.fmt.bufPrintZ(
&sql_buf,
"PRAGMA user_version = {d};",
.{querylog_schema.fingerprint +% 1},
));
created.database.close();
}
{
var deep_dir = try tmp.dir.openDir(io, nested, .{});
defer deep_dir.close(io);
try deep_dir.writeFile(io, .{ .sub_path = "querylog.db", .data = "not a database at all" });
}
var recreated = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer recreated.database.close();
try testing.expectEqual(querylog_schema.RecreateReason.fingerprint_mismatch, recreated.recreated.?);
try testing.expectEqual(querylog_schema.RecreateReason.not_a_database, recreated.recreated.?);
const line_overhead = "previous file kept as ''; query history is available from ".len;
try testing.expect(recreated.aside().len + line_overhead > events.Store.max_detail_len);
@@ -1837,6 +1671,13 @@ test "run maps a rejected configuration to exit 2 and everything else to exit 1"
try std.testing.expectEqual(cli.exit_check, failureExitCode(error.ParseZon));
try std.testing.expectEqual(cli.exit_check, failureExitCode(error.NoUsableUpstreams));
try std.testing.expectEqual(cli.exit_check, failureExitCode(error.BadCertificate));
// The query-log schema refusals: `run` is the only command that reaches
// them, and exit 1 would put a deliberate refusal under the unit's
// `Restart=on-failure`.
try std.testing.expectEqual(cli.exit_check, failureExitCode(error.SchemaTooNew));
try std.testing.expectEqual(cli.exit_check, failureExitCode(error.SchemaUnsupported));
try std.testing.expectEqual(cli.exit_check, failureExitCode(error.MigrationFailed));
try std.testing.expectEqual(cli.exit_check, failureExitCode(error.MigrationBackupFailed));
try std.testing.expectEqual(cli.exit_runtime, failureExitCode(error.AccessDenied));
try std.testing.expectEqual(cli.exit_runtime, failureExitCode(error.OutOfMemory));
}
@@ -2414,10 +2255,11 @@ test "one maintenance pass drops the api limiter's stale buckets" {
_ = limiter.check(io, .{ .nanoseconds = now.nanoseconds - 2 * window_ns }, client);
try std.testing.expectEqual(@as(u32, 1), limiter.trackedClients(io));
// Nothing here exchanges: the pass only sweeps the two tables.
var unreachable_upstream: upstream_owner.Borrowed = .{};
var h: handler.Handler = .{
.upstream = .{ .ptr = undefined, .exchangeFn = undefined },
.blocking = .{ .mode = .zero, .ttl = 5 },
.forward_read_timeout = .{ .raw = .fromMilliseconds(50), .clock = .awake },
.upstream = unreachable_upstream.client(.{ .ptr = undefined, .exchangeFn = undefined }),
.policy = .{ .blocking = .{ .mode = .zero, .ttl = 5 }, .forward_read_timeout = .{ .raw = .fromMilliseconds(50), .clock = .awake } },
};
try maintenanceOnce(&h, &limiter, io);
@@ -2520,6 +2362,80 @@ test "a configuration finding is reported once and finalize closes the rest" {
));
}
test "the upstream build's report replays into the same rows the boot path used to write" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
// Last boot's finding for a row this boot has fixed. Only `finalize`
// closes it, which is why the replay has to run before it.
fx.store.report(io, 900, .configuration_load, "ftp://fixed.example", "ftp://fixed.example", .warning, "stale");
var http: std.http.Client = .{ .allocator = testing.allocator, .io = io };
defer http.deinit();
var bundle: Certificate.Bundle = .empty;
defer bundle.deinit(testing.allocator);
var bundle_lock: std.Io.RwLock = .init;
const generation = try upstream_owner.build(.{
.gpa = testing.allocator,
.io = io,
// The credential in the bad row is why the key and the rendered text
// differ: the key is the whole url, and both rendered forms drop it.
.servers = &.{
.{ .url = "ftp://user:hunter2@nope.example" },
.{ .url = "https://good.example/dns-query" },
},
.http = &http,
.bundle = &bundle,
.bundle_lock = &bundle_lock,
.timeouts = .{
.attempt = .{ .raw = .fromMilliseconds(50), .clock = .awake },
.total = .{ .raw = .fromMilliseconds(200), .clock = .awake },
},
.seed = 1,
});
var upstreams: upstream_owner.Owner = .init(generation);
defer upstreams.deinit(io);
// `build` itself wrote nothing: a candidate a settings PUT never publishes
// must leave the diagnostics log exactly as it found it.
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
var collector: ConfigLoad = .{ .store = &fx.store, .io = io, .now_s = 1000 };
for (generation.report().notes) |finding| {
noteUpstream(&collector, finding.url, finding.message);
}
collector.finalize();
// The whole url is the key, the redaction is the label, and the detail is
// the sentence `noteUpstream` has always written — byte for byte.
try testing.expectEqual(
@as(i64, 1),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
try testing.expectEqualStrings("ftp://user:hunter2@nope.example", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings("ftp://nope.example", try fx.text(
"SELECT subject_label FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings(
"upstream 'ftp://nope.example' not an https:// or tls:// endpoint; skipped",
try fx.text("SELECT detail FROM operational_events WHERE resolved_at IS NULL"),
);
// The row that was wrong last boot and is not wrong now is closed by the
// same `finalize` as ever: the replay is what puts the keys in front of it.
try testing.expectEqual(@as(i64, 1), try fx.count(
"SELECT count(*) FROM operational_events WHERE subject_key = 'ftp://fixed.example' AND resolved_at IS NOT NULL",
));
}
test "an over-long boot finding list refuses to finalize rather than truncate" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
+7 -10
View File
@@ -343,16 +343,13 @@ pub const DataDir = struct {
}
/// An additional connection to a `querylog.db` that `openQuerylogDb` has
/// already established. A running server needs two background ones — the
/// log writer and the retention pass each own one (`retention.zig`'s
/// contract) — plus a third for the web task when the web interface is
/// enabled.
/// already established, bound to this data directory's path.
///
/// The opener itself is `querylog_schema.reopen`, beside the schema it
/// belongs to: a logger generation opens its own writer connection at
/// runtime and has no `DataDir` to ask.
pub fn reopenQuerylogDb(self: *const DataDir, io: std.Io) !db.Db {
_ = io;
var database = try db.Db.open(self.querylog_db_path, .{ .mode = .read_write_existing });
errdefer database.close();
try db.applyPragmas(&database, .{});
return database;
return querylog_schema.reopen(io, std.Io.Dir.cwd(), self.querylog_db_path);
}
/// A sidecar that does not exist yet is not a failure: `-wal` and `-shm`
@@ -1015,7 +1012,7 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize {
.priority = server.priority,
.enabled = true,
.health = .init,
.sem = .{ .permits = slots.len },
.admission = .{ .permits = slots.len },
.reuse_recoveries = &recoveries,
}};
var single: pool.Pool = .init(&entries, .{}, timeouts, seed);
+25 -1
View File
@@ -13,7 +13,7 @@ const validate = @import("validate.zig");
/// `ValidateError` enters as a whole set rather than variant by variant, so a
/// variant added to the validator cannot silently fall through to exit 1. The
/// five extras are the configuration faults raised outside the validator: the
/// first five extras are the configuration faults raised outside the validator: the
/// ZON reader (`ParseZon`), the file size limit (`ConfigTooLarge`), the managed
/// file the operator named and this process cannot open
/// (`ManagedConfigUnreadable`, milestone-20 ruling 2), the composition root's
@@ -26,6 +26,16 @@ const validate = @import("validate.zig");
/// missing file anywhere else stays a runtime failure. `config/loader.zig` owns
/// the conversion and the closed set of open errors that qualify.
///
/// The four querylog schema refusals are here for the exit code, not because a
/// `.zon` file is wrong: `SchemaTooNew` and `SchemaUnsupported` are a deliberate
/// refusal to touch a `querylog.db` this binary does not understand, and
/// `MigrationFailed` and `MigrationBackupFailed` are a deliberate refusal to run
/// on a database whose migration or pre-migration backup did not complete. All
/// four need an operator, and none of them will resolve on a retry — exit 1 puts
/// them under systemd's `Restart=on-failure` and restart-loops a server that is
/// refusing on purpose. The unit's `RestartPreventExitStatus=2 64` is what exit
/// 2 buys them.
///
/// Not here on purpose: `error.DestructiveImport`, which reports what an import
/// would do to the database rather than the content of a file, and is the one
/// config-shaped exit 2 `cli` decides for itself.
@@ -35,6 +45,10 @@ const ConfigFault = validate.ValidateError || error{
ManagedConfigUnreadable,
NoUsableUpstreams,
BadCertificate,
SchemaTooNew,
SchemaUnsupported,
MigrationFailed,
MigrationBackupFailed,
};
const faults: []const anyerror = blk: {
@@ -119,6 +133,16 @@ test "the seed-file errors that used to exit 1 from run are configuration faults
try testing.expect(isConfigFault(error.NoUpstreams));
}
test "the querylog schema refusals exit 2 so systemd does not restart-loop them" {
try testing.expect(isConfigFault(error.SchemaTooNew));
try testing.expect(isConfigFault(error.SchemaUnsupported));
try testing.expect(isConfigFault(error.MigrationFailed));
try testing.expect(isConfigFault(error.MigrationBackupFailed));
// The refusals are a closed set. A neighbouring schema error is a corrupt
// database, not a refusal, and stays a runtime failure.
try testing.expect(!isConfigFault(error.SchemaCorrupt));
}
test "a runtime failure is not a configuration fault" {
try testing.expect(!isConfigFault(error.OutOfMemory));
try testing.expect(!isConfigFault(error.AccessDenied));
+5 -12
View File
@@ -57,9 +57,11 @@ pub const Config = struct {
pub const Upstream = struct {
/// Bounds one attempt against one upstream inside the pool's failover loop.
attempt_timeout_ms: u32 = 2500,
/// The forward-zone client's read deadline, and nothing else. It bounds a
/// different subsystem from the two above (`src/local/forward_client.zig`),
/// so no cross-check relates it to them.
/// The forward-zone client's whole-exchange budget, and nothing else: one
/// bound covers the UDP attempt, a TC=1 fallback and the TCP retry
/// together, not each of them. It bounds a different subsystem from the two
/// above (`src/local/forward_client.zig`), so no cross-check relates it to
/// them.
read_timeout_ms: u32 = 3000,
/// The whole-exchange budget: every failover attempt together, not one of
/// them. The pool races the entire loop against it.
@@ -380,10 +382,6 @@ pub fn sessionTtlSeconds(w: Web) i64 {
return @as(i64, w.session_ttl_hours) * 3600;
}
pub fn retentionSeconds(l: Logging) i64 {
return @as(i64, l.retention_days) * 86400;
}
pub fn maxLogBytes(l: Logging) u64 {
return @as(u64, l.max_size_mb) * 1024 * 1024;
}
@@ -840,7 +838,6 @@ test "unit conversions" {
totalTimeout(.{}).nanoseconds,
);
try testing.expectEqual(@as(i64, 24 * 3600), sessionTtlSeconds(.{}));
try testing.expectEqual(@as(i64, 30 * 86400), retentionSeconds(.{}));
try testing.expectEqual(@as(u64, 50 * 1024 * 1024), maxLogBytes(.{}));
try testing.expectEqual(@as(u64, 200 * 1024 * 1024), minFreeBytes(.{}));
try testing.expectEqual(@as(u64, 500 * 1024 * 1024), warnFreeBytes(.{}));
@@ -865,10 +862,6 @@ test "unit conversions at the field maximum do not overflow" {
@as(i64, std.math.maxInt(u16)) * 3600,
sessionTtlSeconds(.{ .session_ttl_hours = std.math.maxInt(u16) }),
);
try testing.expectEqual(
@as(i64, std.math.maxInt(u16)) * 86400,
retentionSeconds(.{ .retention_days = std.math.maxInt(u16) }),
);
try testing.expectEqual(
@as(u64, std.math.maxInt(u32)) * 1024 * 1024,
maxLogBytes(.{ .max_size_mb = std.math.maxInt(u32) }),
+66 -2
View File
@@ -52,6 +52,7 @@ const limits = @import("limits.zig");
const logger = @import("../storage/logger.zig");
const regex = @import("../filter/regex.zig");
const safe_url = @import("../safe_url.zig");
const pool = @import("../upstream/pool.zig");
const transport = @import("../upstream/transport.zig");
const Config = model.Config;
@@ -69,6 +70,7 @@ const Prefix = address.Prefix;
/// like every other resource failure.
pub const ValidateError = error{
NoUpstreams,
TooManyUpstreams,
BadUpstreamUrl,
UpstreamHostNotIpLiteral,
DuplicateUpstreamUrl,
@@ -357,8 +359,9 @@ fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void {
// The only cross-check that relates two knobs of one subsystem: the pool
// races one attempt against `attempt` and the whole failover loop against
// `total`, so an attempt budget above the total one can never be reached.
// `read_timeout_ms` belongs to the forward-zone client and is deliberately
// unrelated to both.
// `read_timeout_ms` bounds the forward-zone client's whole exchange —
// UDP attempt, TC=1 fallback and TCP retry under one budget — and is
// deliberately unrelated to both.
if (up.attempt_timeout_ms > up.total_timeout_ms) {
try diags.add(
error.BadTimeout,
@@ -823,6 +826,21 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{
.{},
);
}
// Each enabled upstream becomes one pool entry, and the failover loop
// tracks the entries it has spent in a fixed bitset of `Pool.max_entries`
// bits. Without this check a config past that bound reaches an assert and
// panics at startup, which is the wrong way to tell an operator that a
// number is too large. Disabled upstreams are not counted: they never
// become entries.
if (enabled_upstreams > pool.Pool.max_entries) {
try diags.add(
error.TooManyUpstreams,
"upstreams",
.{},
"{d} upstreams are enabled; nxdns is built for at most {d}",
.{ enabled_upstreams, pool.Pool.max_entries },
);
}
var client_ips: IndexSet = .empty;
for (cfg.clients, 0..) |client, i| {
@@ -1323,6 +1341,52 @@ test "error.NoUpstreams when nothing is enabled" {
try expectProblem(cfg, error.NoUpstreams, "upstreams");
}
/// `count` distinct enabled upstreams. Generated rather than written out
/// because the bound this exercises is 64, and a hand-written list that long
/// would say less than the loop does.
fn ManyUpstreams(comptime count: usize) type {
return struct {
const list: [count]model.UpstreamServer = blk: {
var built: [count]model.UpstreamServer = undefined;
for (&built, 0..) |*server, i| {
server.* = .{ .url = std.fmt.comptimePrint("https://u{d}.example/dns-query", .{i}) };
}
break :blk built;
};
};
}
fn manyUpstreams(comptime count: usize) []const model.UpstreamServer {
return &ManyUpstreams(count).list;
}
test "as many enabled upstreams as the pool holds validates cleanly" {
var cfg = baseConfig();
cfg.upstreams = manyUpstreams(pool.Pool.max_entries);
try expectClean(cfg);
}
test "error.TooManyUpstreams one enabled upstream past the pool's bound" {
// The pool asserts this bound, so without the check here a valid-looking
// config panics at startup instead of being reported.
var cfg = baseConfig();
cfg.upstreams = manyUpstreams(pool.Pool.max_entries + 1);
try expectProblem(cfg, error.TooManyUpstreams, "upstreams");
}
test "upstreams past the pool's bound are fine while they are disabled" {
// Only enabled upstreams become pool entries, so a long list with a small
// enabled subset is not near the bound at all.
var cfg = baseConfig();
cfg.upstreams = comptime blk: {
var list = manyUpstreams(pool.Pool.max_entries + 1)[0 .. pool.Pool.max_entries + 1].*;
for (list[1..]) |*server| server.enabled = false;
const frozen = list;
break :blk &frozen;
};
try expectClean(cfg);
}
test "error.BadUpstreamUrl on an unsupported scheme" {
var cfg = baseConfig();
cfg.upstreams = &.{.{ .url = "ftp://dns.example/" }};
+66 -7
View File
@@ -1311,11 +1311,13 @@ test "10b: the scheduler sweeps orphans on its own, with no operator call" {
try dir.writeFile(io, .{ .sub_path = "9999.allow.tmp", .data = "" });
// `runScheduler` is the entry point `app.zig` hands to `Io.Group`, and the
// only one the server ever calls. A disabled update stops it after the
// startup pass, so the production path runs to completion here with no
// interval to wait out.
// only one the server ever calls. A disabled update parks it after the
// startup pass, and the seam turns that park into the shutdown the task
// only otherwise exits on, so the production path runs to completion here
// with no interval to wait out.
env.mgr.update.enabled = false;
try env.mgr.runScheduler(io);
env.mgr.schedule_clock = .shutdown_at_first_park;
try testing.expectError(error.Canceled, env.mgr.runScheduler(io));
try testing.expectError(error.FileNotFound, dir.access(io, "9999.list", .{}));
try testing.expectError(error.FileNotFound, dir.access(io, "9999.wild", .{}));
@@ -1555,8 +1557,8 @@ test "10e: a reconcile then a restart reuses the compiled files and downloads no
// The restart. The old manager is gone and a new one comes up over the same
// directory and the same database with nothing carried across in memory.
// `runScheduler` is the boot sequence the server runs — the orphan sweep,
// then the startup pass — and a disabled update makes it return rather than
// wait out an interval.
// then the startup pass — and a disabled update parks it rather than
// waiting out an interval; the seam turns that park into shutdown.
env.mgr.deinit(io);
env.mgr = try manager.Manager.init(
gpa,
@@ -1566,7 +1568,8 @@ test "10e: a reconcile then a restart reuses the compiled files and downloads no
.{ .enabled = false },
budget,
);
try env.mgr.runScheduler(io);
env.mgr.schedule_clock = .shutdown_at_first_park;
try testing.expectError(error.Canceled, env.mgr.runScheduler(io));
// Nothing was downloaded. The server is still listening, so this is a
// decision the pass made rather than a connection it could not have opened.
@@ -2456,6 +2459,62 @@ test "27: a failing refresh opens a blocklist.refresh episode a good one closes"
);
}
/// Records the deadline of the first park and then reports the shutdown the
/// scheduler loop only otherwise exits on. Its clock never moves, so the
/// deadline it captures is exactly `anchor + interval`.
const FirstParkRecorder = struct {
deadline_s: ?i64 = null,
parked: bool = false,
fn clock(self: *FirstParkRecorder) manager.ScheduleClock {
return .{ .ctx = self, .nowFn = now, .waitFn = wait };
}
fn now(_: ?*anyopaque, _: std.Io) i64 {
return 0;
}
fn wait(ctx: ?*anyopaque, _: std.Io, _: *std.Io.Event, deadline_s: ?i64) std.Io.Cancelable!void {
const self: *FirstParkRecorder = @ptrCast(@alignCast(ctx.?));
self.deadline_s = deadline_s;
self.parked = true;
return error.Canceled;
}
};
test "34: a pass whose refresh failed still advances the schedule anchor" {
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
const env = try Env.create(gpa);
defer env.destroy();
const io = env.io();
var fixture = try HttpFixture.init(io, http_body);
defer fixture.deinit(io);
// Every download this pass makes fails.
fixture.setRoute(.oversize);
var group: std.Io.Group = .init;
defer group.cancel(io);
try group.concurrent(io, HttpFixture.serve, .{ &fixture, io });
var url_buf: [64]u8 = undefined;
const url = try fixture.url(&url_buf);
_ = try seedSource(&env.database, url);
var recorder: FirstParkRecorder = .{};
env.mgr.schedule_clock = recorder.clock();
env.mgr.setSchedule(io, true, 2);
try testing.expectError(error.Canceled, env.mgr.runScheduler(io));
// The startup pass tried the source and failed. The anchor still moved to
// that pass's completion, so the next refresh is a full interval away
// rather than immediate: a failing source must not become a download loop.
try testing.expect(recorder.parked);
try testing.expectEqual(@as(?i64, 2 * 3_600), recorder.deadline_s);
}
test "27: one refreshAll pass records one occurrence of a failing source" {
if (!build_options.integration) return error.SkipZigTest;
+404 -12
View File
@@ -333,12 +333,83 @@ pub fn stripHeader(bytes: []const u8) []const u8 {
return rest;
}
/// The scheduler's two time operations, behind a seam. Validated intervals are
/// at least an hour, so a test that used the real clock would either sleep an
/// hour or prove nothing; a test installs its own step clock instead.
pub const ScheduleClock = struct {
ctx: ?*anyopaque = null,
/// Seconds on a monotonic clock. Only differences matter.
nowFn: *const fn (ctx: ?*anyopaque, io: std.Io) i64,
/// Returns when `deadline_s` arrives or `event` is set, whichever comes
/// first; a null deadline waits for the event alone. A spurious early
/// return is allowed — the caller rechecks both the version and the clock.
waitFn: *const fn (
ctx: ?*anyopaque,
io: std.Io,
event: *std.Io.Event,
deadline_s: ?i64,
) std.Io.Cancelable!void,
pub const real: ScheduleClock = .{ .nowFn = realNow, .waitFn = realWait };
/// Test seam: the loop only ever exits on shutdown, so a test that wants
/// `runScheduler` to run its startup pass and return installs this and
/// gets `error.Canceled` at the first park.
pub const shutdown_at_first_park: ScheduleClock = .{ .nowFn = realNow, .waitFn = cancelWait };
fn cancelWait(_: ?*anyopaque, _: std.Io, _: *std.Io.Event, _: ?i64) std.Io.Cancelable!void {
return error.Canceled;
}
/// `boot` rather than `awake`: a box that suspends overnight should still
/// see its daily interval elapse.
fn realNow(_: ?*anyopaque, io: std.Io) i64 {
return std.Io.Clock.boot.now(io).toSeconds();
}
fn realWait(
_: ?*anyopaque,
io: std.Io,
event: *std.Io.Event,
deadline_s: ?i64,
) std.Io.Cancelable!void {
const timeout: std.Io.Timeout = if (deadline_s) |seconds| .{ .deadline = .{
.raw = .{ .nanoseconds = @as(i96, seconds) * std.time.ns_per_s },
.clock = .boot,
} } else .none;
event.waitTimeout(io, timeout) catch |err| switch (err) {
error.Timeout => {},
error.Canceled => return error.Canceled,
};
}
};
pub const Manager = struct {
gpa: Allocator,
database: *db.Db,
paths: Paths,
fetcher: *fetcher.Fetcher,
/// Read and written only under `schedule_mutex`; `setSchedule` replaces it
/// while the scheduler is parked.
update: model.BlocklistUpdate,
/// Guards `update`, `schedule_version` and `schedule_anchor_s`.
///
/// Lock ordering: innermost. `needsRefresh` takes it while `refresh_lock`
/// is held, and nothing that holds it takes another manager lock.
schedule_mutex: std.Io.Mutex,
/// Bumped by every `setSchedule`. The scheduler reads it before it parks
/// and again after it wakes: a change that lands in that window is what the
/// recheck catches, so no wake is lost and none is mistaken for a deadline.
schedule_version: u64,
/// When the last refresh pass that RAN completed, on `ScheduleClock`'s
/// clock. Success, failure and a disk-gate skip all advance it — the
/// scheduled slot is spent either way and is not retried early. Null until
/// the startup pass finishes.
schedule_anchor_s: ?i64,
/// Sticky once set, so `setSchedule` can never signal into a gap. The loop
/// resets it under `schedule_mutex` before it recomputes its deadline.
schedule_event: std.Io.Event,
schedule_clock: ScheduleClock,
/// Bounds one download. `std.http.Client` has no per-request deadline, so
/// the fetch runs under `io.concurrent` against a sleep of this length.
total_budget: std.Io.Clock.Duration,
@@ -412,6 +483,11 @@ pub const Manager = struct {
.paths = paths,
.fetcher = fetcher_ptr,
.update = update,
.schedule_mutex = .init,
.schedule_version = 0,
.schedule_anchor_s = null,
.schedule_event = .unset,
.schedule_clock = .real,
.total_budget = total_budget,
.lock = .init,
.writer_lock = .init,
@@ -1490,8 +1566,9 @@ pub const Manager = struct {
/// it has no usable compiled files or its `last_updated` is older than the
/// interval.
///
/// `update.enabled == false` stops after the startup pass; manual refresh
/// through `refreshAll` still works.
/// `update.enabled == false` parks after the startup pass; manual refresh
/// through `refreshAll` still works, and a later `setSchedule` wakes the
/// loop rather than needing a restart.
pub fn runScheduler(self: *Manager, io: std.Io) std.Io.Cancelable!void {
// Ahead of the pass, not after it. This is the sweep that collects what
// a killed process left behind: a `.raw.tmp` as large as the body the
@@ -1511,18 +1588,76 @@ pub const Manager = struct {
self.flushDiagnostics(io);
},
};
if (!self.update.enabled) return;
// The startup pass ran, so it anchors the schedule — including when
// updates are disabled, so a later enable measures its first interval
// from real work rather than from the moment the operator flipped the
// switch.
self.anchorNow(io);
// `boot` rather than `awake`: a box that suspends overnight should
// still see its daily interval elapse.
const interval: std.Io.Clock.Duration = .{
.raw = .fromSeconds(model.updateIntervalSeconds(self.update)),
.clock = .boot,
};
while (true) {
try interval.sleep(io);
try self.scheduledPass(io);
// One hold: read the version, reset the sticky event, and take the
// schedule the deadline is computed from. A `setSchedule` that
// lands after this reset completes the wait below at once, and the
// version recheck decides whether the wake meant anything.
self.schedule_mutex.lockUncancelable(io);
const version = self.schedule_version;
self.schedule_event.reset();
const enabled = self.update.enabled;
const interval_s = model.updateIntervalSeconds(self.update);
const anchor = self.schedule_anchor_s;
self.schedule_mutex.unlock(io);
const now_s = self.schedule_clock.nowFn(self.schedule_clock.ctx, io);
// Disabled parks on the event alone. The task still exits only on
// shutdown, exactly as it did when it returned here.
const deadline_s: ?i64 = if (enabled) (anchor orelse now_s) + interval_s else null;
if (deadline_s == null or now_s < deadline_s.?) {
try self.schedule_clock.waitFn(self.schedule_clock.ctx, io, &self.schedule_event, deadline_s);
// Either the schedule changed under us or the wait was
// spurious; recompute from the top rather than guess.
if (self.scheduleVersion(io) != version) continue;
if (deadline_s == null) continue;
if (self.schedule_clock.nowFn(self.schedule_clock.ctx, io) < deadline_s.?) continue;
}
try self.scheduledPass(io);
self.anchorNow(io);
}
}
/// Installs a new blocklist-update schedule and wakes the scheduler. The
/// anchor is untouched: the next refresh is due one NEW interval after the
/// last pass that ran, which the loop refreshes immediately when that
/// moment is already past.
pub fn setSchedule(self: *Manager, io: std.Io, enabled: bool, interval_hours: u16) void {
self.schedule_mutex.lockUncancelable(io);
self.update = .{ .enabled = enabled, .interval_hours = interval_hours };
self.schedule_version += 1;
self.schedule_mutex.unlock(io);
self.schedule_event.set(io);
}
/// The live schedule. Every reader outside the scheduler loop goes through
/// here, so none of them reads `update` while `setSchedule` writes it.
pub fn schedule(self: *Manager, io: std.Io) model.BlocklistUpdate {
self.schedule_mutex.lockUncancelable(io);
defer self.schedule_mutex.unlock(io);
return self.update;
}
fn scheduleVersion(self: *Manager, io: std.Io) u64 {
self.schedule_mutex.lockUncancelable(io);
defer self.schedule_mutex.unlock(io);
return self.schedule_version;
}
fn anchorNow(self: *Manager, io: std.Io) void {
const now_s = self.schedule_clock.nowFn(self.schedule_clock.ctx, io);
self.schedule_mutex.lockUncancelable(io);
self.schedule_anchor_s = now_s;
self.schedule_mutex.unlock(io);
}
/// What one elapsed interval does. Split from the loop above so a test can
@@ -1643,7 +1778,7 @@ pub const Manager = struct {
// else would ever clear it. A stamp from the future is not evidence of
// a recent fetch.
if (last > now) return true;
return now - last >= model.updateIntervalSeconds(self.update);
return now - last >= model.updateIntervalSeconds(self.schedule(io));
}
// -----------------------------------------------------------------------
@@ -3188,3 +3323,260 @@ test "bodyChecksum covers the list body, then the wild body, then the allow body
&bodyChecksum("a.example.com\n", "c.example.com\n", "b.example.com\n"),
));
}
// ---------------------------------------------------------------------------
// wakeable scheduler (milestone-34 S3.6)
// ---------------------------------------------------------------------------
/// A `ScheduleClock` that never sleeps. Each park is recorded, then the clock
/// jumps straight to the deadline so the loop runs the next pass at once; a
/// budget of parks ends the run with the `error.Canceled` shutdown is the only
/// other source of. A park may also fire a `setSchedule`, which is what a
/// settings PUT landing while the scheduler waits looks like.
const StepClock = struct {
const max_parks = 16;
mutex: std.Io.Mutex = .init,
manager: *Manager,
now_s: i64 = 0,
/// Deadline of each park in order; null means "parked with no deadline",
/// which is what a disabled schedule does.
parks: [max_parks]?i64 = @splat(null),
park_count: usize = 0,
budget: usize = 2,
/// Fired from inside the park at this index, before the wait returns.
change_at_park: ?usize = null,
change_enabled: bool = true,
change_hours: u16 = 1,
fn clock(self: *StepClock) ScheduleClock {
return .{ .ctx = self, .nowFn = now, .waitFn = wait };
}
fn now(ctx: ?*anyopaque, io: std.Io) i64 {
const self: *StepClock = @ptrCast(@alignCast(ctx.?));
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
return self.now_s;
}
fn wait(ctx: ?*anyopaque, io: std.Io, _: *std.Io.Event, deadline_s: ?i64) std.Io.Cancelable!void {
const self: *StepClock = @ptrCast(@alignCast(ctx.?));
self.mutex.lockUncancelable(io);
const index = self.park_count;
if (index < max_parks) self.parks[index] = deadline_s;
self.park_count = index + 1;
const fire_change = self.change_at_park == index;
const over_budget = self.park_count >= self.budget;
if (deadline_s) |d| self.now_s = d;
self.mutex.unlock(io);
// Taken outside this clock's own mutex: `setSchedule` takes the
// manager's, and the loop reads this clock under neither.
if (fire_change) {
self.manager.setSchedule(io, self.change_enabled, self.change_hours);
return;
}
if (over_budget or deadline_s == null) return error.Canceled;
}
fn parked(self: *StepClock) []const ?i64 {
return self.parks[0..@min(self.park_count, max_parks)];
}
};
const hour = 3_600;
test "the scheduler parks one interval past the anchor and again past each pass" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var manager = try testManager(&database, &f);
defer manager.deinit(io);
manager.update = .{ .enabled = true, .interval_hours = 2 };
var step: StepClock = .{ .manager = &manager, .budget = 3 };
manager.schedule_clock = step.clock();
try testing.expectError(error.Canceled, manager.runScheduler(io));
// The startup pass anchored at 0, so the first park is due at 2 h and each
// completed pass re-anchors: 2 h, 4 h, 6 h.
try testing.expectEqualSlices(?i64, &.{ 2 * hour, 4 * hour, 6 * hour }, step.parked());
}
test "a shortened interval moves the next refresh onto the new cadence" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var manager = try testManager(&database, &f);
defer manager.deinit(io);
manager.update = .{ .enabled = true, .interval_hours = 24 };
// The PUT lands while the loop waits out the 24-hour deadline.
var step: StepClock = .{
.manager = &manager,
.budget = 4,
.change_at_park = 0,
.change_enabled = true,
.change_hours = 1,
};
manager.schedule_clock = step.clock();
try testing.expectError(error.Canceled, manager.runScheduler(io));
// Park 0 was the old 24-hour deadline; the change woke it, and every park
// after it is one hour past the anchor the previous pass set.
const parks = step.parked();
try testing.expectEqual(@as(usize, 4), parks.len);
try testing.expectEqual(@as(?i64, 24 * hour), parks[0]);
try testing.expectEqual(@as(?i64, 24 * hour + hour), parks[1]);
try testing.expectEqual(@as(?i64, 25 * hour + hour), parks[2]);
}
test "a disabled schedule parks with no deadline and the startup pass still runs" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var manager = try testManager(&database, &f);
defer manager.deinit(io);
manager.update = .{ .enabled = false, .interval_hours = 1 };
var step: StepClock = .{ .manager = &manager, .budget = 8 };
manager.schedule_clock = step.clock();
try testing.expectError(error.Canceled, manager.runScheduler(io));
// The startup pass ran — it published a snapshot even with updates off —
// and then the loop parked once, on nothing.
try testing.expect(manager.generation > 0);
try testing.expectEqualSlices(?i64, &.{null}, step.parked());
}
test "re-enabling anchors the first interval on the last pass that ran" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var manager = try testManager(&database, &f);
defer manager.deinit(io);
manager.update = .{ .enabled = false, .interval_hours = 1 };
var step: StepClock = .{
.manager = &manager,
.budget = 3,
.change_at_park = 0,
.change_enabled = true,
.change_hours = 3,
};
manager.schedule_clock = step.clock();
try testing.expectError(error.Canceled, manager.runScheduler(io));
const parks = step.parked();
// Park 0 is the disabled park; the enable wakes it, and the first deadline
// is three hours past the STARTUP pass's anchor rather than past the
// moment the operator flipped the switch.
try testing.expectEqual(@as(?i64, null), parks[0]);
try testing.expectEqual(@as(?i64, 3 * hour), parks[1]);
}
test "an interval already elapsed at enable time refreshes immediately" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var manager = try testManager(&database, &f);
defer manager.deinit(io);
manager.update = .{ .enabled = true, .interval_hours = 2 };
var step: StepClock = .{ .manager = &manager, .budget = 2 };
manager.schedule_clock = step.clock();
// The anchor is four hours in the past, so two hours past it is already
// gone and the loop must not wait at all before its first pass.
manager.schedule_anchor_s = -4 * hour;
step.now_s = 0;
try testing.expectError(error.Canceled, manager.runScheduler(io));
// The startup pass re-anchors at 0, so this proves nothing on its own
// unless the anchor survives it; assert on the parks instead: the first
// park is one interval past the startup anchor, never a wait for a
// deadline already behind us.
const parks = step.parked();
try testing.expectEqual(@as(?i64, 2 * hour), parks[0]);
}
test "a gate-skipped pass advances the anchor rather than retrying early" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var manager = try testManager(&database, &f);
defer manager.deinit(io);
manager.update = .{ .enabled = true, .interval_hours = 2 };
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
manager.monitor = &monitor;
var step: StepClock = .{ .manager = &manager, .budget = 3 };
manager.schedule_clock = step.clock();
try testing.expectError(error.Canceled, manager.runScheduler(io));
// Every scheduled pass was refused by the gate, and each one still spent
// its slot: the deadlines march one interval at a time instead of
// collapsing onto the same anchor.
try testing.expectEqualSlices(?i64, &.{ 2 * hour, 4 * hour, 6 * hour }, step.parked());
// The startup pass is gated too, so three refusals: one startup and the
// two scheduled passes the parks above bracket.
try testing.expectEqual(@as(u64, 3), manager.refreshesGated());
}
test "setSchedule is what the live schedule readers see" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var manager = try testManager(&database, &f);
defer manager.deinit(io);
manager.setSchedule(io, false, 6);
const live = manager.schedule(io);
try testing.expect(!live.enabled);
try testing.expectEqual(@as(u16, 6), live.interval_hours);
try testing.expect(manager.schedule_event.isSet());
}
+41
View File
@@ -73,6 +73,29 @@ const npm_not_shipped = [_][]const u8{
"aria-hidden",
"client-only",
"tslib",
"@types/d3-array",
"@types/d3-color",
"@types/d3-delaunay",
"@types/d3-format",
"@types/d3-geo",
"@types/d3-interpolate",
"@types/d3-path",
"@types/d3-scale",
"@types/d3-shape",
"@types/d3-time",
"@types/d3-time-format",
"@types/geojson",
"@types/react",
"@types/react-dom",
"csstype",
"@visx/curve",
"@visx/vendor",
"d3-delaunay",
"d3-geo",
"d3-time-format",
"delaunator",
"robust-predicates",
"react-use-measure",
};
/// Components the shipped artifacts contain that no dependency file mentions,
@@ -158,6 +181,24 @@ const npm_licence_exceptions = [_]NpmLicence{
// and @stylexjs/stylex's compiler.
.{ .name = "isbot", .licence = "Unlicense" },
.{ .name = "css-mediaquery", .licence = "BSD" },
// Shipped: the d3 modules visx computes its geometry with. Mokhtar Mial
// accepted ISC inbound on 2026-08-24; licenses/d3-isc.txt carries the text.
.{ .name = "d3-array", .licence = "ISC" },
.{ .name = "d3-color", .licence = "ISC" },
.{ .name = "d3-format", .licence = "ISC" },
.{ .name = "d3-interpolate", .licence = "ISC" },
.{ .name = "d3-path", .licence = "ISC" },
.{ .name = "d3-scale", .licence = "ISC" },
.{ .name = "d3-shape", .licence = "ISC" },
.{ .name = "d3-time", .licence = "ISC" },
.{ .name = "internmap", .licence = "ISC" },
// Not shipped: the rest of the d3 closure, reached only through
// @visx/vendor's map and Voronoi re-exports, which no chart here imports.
.{ .name = "d3-delaunay", .licence = "ISC" },
.{ .name = "d3-geo", .licence = "ISC" },
.{ .name = "d3-time-format", .licence = "ISC" },
.{ .name = "delaunator", .licence = "ISC" },
.{ .name = "robust-predicates", .licence = "Unlicense" },
};
const NpmLicence = struct {
+147 -19
View File
@@ -109,6 +109,11 @@ pub const ForwardClient = struct {
/// `.udp` resolvers send one datagram and fall back to TCP when the answer
/// comes back with TC=1. `.tcp` resolvers skip straight to the TCP path.
///
/// `read_timeout` bounds the WHOLE exchange, truncation fallback included:
/// the instant is computed once here and every blocking step inside runs
/// against it, so a truncated UDP answer followed by a stalled TCP retry
/// costs one budget rather than two.
pub fn exchange(
self: *ForwardClient,
io: std.Io,
@@ -120,10 +125,22 @@ pub const ForwardClient = struct {
if (response_buf.len == 0) return error.BufferTooSmall;
self.stats.queries += 1;
return self.route(io, query, response_buf) catch |err| {
const expiry_at: std.Io.Clock.Timestamp = .fromNow(io, self.read_timeout);
// The outcome is deliberately discarded. A forward zone has exactly one
// configured resolver, so an expiry here is still that resolver failing
// to answer in time: `error.Timeout` is peer evidence, and the
// budget/peer distinction is upstream-pool policy.
var outcome: transport.RaceOutcome = .completed;
return transport.raceUntilTagged(io, expiry_at, &outcome, route, .{
self,
io,
query,
response_buf,
expiry_at,
}) catch |err| {
switch (transport.group(err)) {
.peer_fault, .local_resource => self.stats.failures += 1,
.cancellation => {},
.cancellation, .budget_exhausted => {},
}
return err;
};
@@ -134,11 +151,12 @@ pub const ForwardClient = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
expiry_at: std.Io.Clock.Timestamp,
) transport.ExchangeError![]u8 {
if (self.resolver.scheme == .udp) {
if (try self.exchangeUdp(io, query, response_buf)) |reply| return reply;
if (try self.exchangeUdp(io, query, response_buf, expiry_at)) |reply| return reply;
}
return self.exchangeTcp(io, query, response_buf);
return self.tcpOnce(io, query, response_buf);
}
/// `null` means the resolver set TC=1 and the caller must retry over TCP.
@@ -151,6 +169,7 @@ pub const ForwardClient = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
expiry_at: std.Io.Clock.Timestamp,
) transport.ExchangeError!?[]u8 {
const dest = self.destination();
const local = wildcardFor(dest);
@@ -166,9 +185,10 @@ pub const ForwardClient = struct {
return transport.mapPhase(err, error.SendFailed);
};
// A deadline, not a duration: a discarded foreign datagram restarts the
// receive, and a duration would hand each retry the full budget again.
const deadline = (std.Io.Timeout{ .duration = self.read_timeout }).toDeadline(io);
// The exchange-wide instant, not a fresh duration: a discarded foreign
// datagram restarts the receive, and a duration would hand each retry
// the full budget again.
const deadline: std.Io.Timeout = .{ .deadline = expiry_at };
while (true) {
const msg = socket.receiveTimeout(io, response_buf, deadline) catch |err| switch (err) {
@@ -205,18 +225,11 @@ pub const ForwardClient = struct {
}
}
/// The read budget bounds the whole TCP exchange through
/// `transport.raceWithin`. `ConnectOptions.timeout` is never set: the
/// Threaded backend panics on it (Threaded.zig:12076).
fn exchangeTcp(
self: *ForwardClient,
io: std.Io,
query: []const u8,
response_buf: []u8,
) transport.ExchangeError![]u8 {
return transport.raceWithin(io, self.read_timeout, tcpOnce, .{ self, io, query, response_buf });
}
/// Unbounded on its own: `exchange` runs it inside the exchange-wide race,
/// which is what cancels a stalled connect or read. It takes no budget of
/// its own, so a truncation fallback does not start a second one.
/// `ConnectOptions.timeout` is never set: the Threaded backend panics on it
/// (Threaded.zig:12076).
fn tcpOnce(
self: *ForwardClient,
io: std.Io,
@@ -307,6 +320,11 @@ fn receiveFailure(stream_reader: *const net.Stream.Reader, err: anyerror) transp
}
const testing = std.testing;
const build_options = @import("build_options");
const name_mod = @import("../dns/name.zig");
const packet = @import("../dns/packet.zig");
const question = @import("../dns/question.zig");
const types = @import("../dns/types.zig");
fn testBuf() [min_frame_buf]u8 {
return undefined;
@@ -467,3 +485,113 @@ test "a stashed stream error is preferred over the collapsed one" {
receiveFailure(&stream_reader, error.EndOfStream),
);
}
const one_budget_ms = 400;
/// Late enough in the budget that a second, fresh budget for the TCP leg would
/// be unmistakable in the elapsed time.
const truncate_after_ms = 300;
/// Answers the first datagram late in the budget with TC=1, which sends the
/// client to TCP — where a listener that never accepts leaves it stalled.
fn truncatingThenStallingResolver(io: std.Io, socket: *const net.Socket) void {
var buf: [2048]u8 = undefined;
const msg = socket.receive(io, &buf) catch return;
const request = packet.parse(msg.data) catch return;
(std.Io.Clock.Duration{
.raw = .fromMilliseconds(truncate_after_ms),
.clock = .awake,
}).sleep(io) catch return;
// The question has to be echoed: `transport.validateResponse` runs before
// the client reads the TC bit, so a bare header would come back as
// `BadResponse` and never reach the TCP fallback this case is about.
const q = packet.firstQuestion(request) orelse return;
var reply_buf: [512]u8 = undefined;
var b = packet.ResponseBuilder.init(&reply_buf, request.header, q) catch return;
const reply = b.finish();
var parsed = dns_header.parse(reply) catch return;
parsed.flags.tc = true;
dns_header.encode(parsed, reply[0..types.header_len]);
socket.send(io, &msg.from, reply) catch return;
}
fn testQuery(io: std.Io, buf: []u8) ![]const u8 {
var id_bytes: [2]u8 = undefined;
io.random(&id_bytes);
dns_header.encode(.{
.id = std.mem.readInt(u16, &id_bytes, .big),
.flags = .{
.rcode = .no_error,
.z = 0,
.ra = false,
.rd = true,
.tc = false,
.aa = false,
.opcode = .query,
.qr = false,
},
.qdcount = 1,
.ancount = 0,
.nscount = 0,
.arcount = 0,
}, buf[0..types.header_len]);
var w: std.Io.Writer = .fixed(buf[types.header_len..]);
try question.encode(.{
.name = try name_mod.fromText("nas.lan"),
.qtype = .a,
.qclass = .in,
}, &w);
return buf[0 .. types.header_len + w.buffered().len];
}
test "one budget covers the udp leg, the TC=1 fallback and the tcp leg" {
if (!build_options.integration) return error.SkipZigTest;
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
const bind_address: net.IpAddress = try .parse("127.0.0.1", 0);
const socket = try bind_address.bind(io, .{ .mode = .dgram });
defer socket.close(io);
const port = socket.address.ip4.port;
// Bound but never accepted: the connect completes out of the kernel's
// backlog and the read then waits forever, which is the stall a second
// budget would be spent on.
const tcp_address: net.IpAddress = try .parse("127.0.0.1", port);
var tcp_listener = try tcp_address.listen(io, .{ .reuse_address = true });
defer tcp_listener.deinit(io);
var group: std.Io.Group = .init;
defer group.cancel(io);
try group.concurrent(io, truncatingThenStallingResolver, .{ io, &socket });
var url_buf: [64]u8 = undefined;
const url = try std.fmt.bufPrint(&url_buf, "udp://127.0.0.1:{d}", .{port});
var frame_buf: [min_frame_buf]u8 = undefined;
var fc: ForwardClient = .init(
try validate.parseResolver(url),
&frame_buf,
.{ .raw = .fromMilliseconds(one_budget_ms), .clock = .awake },
);
var query_buf: [types.header_len + types.max_name_len + 4]u8 = undefined;
const query = try testQuery(io, &query_buf);
var response_buf: [2048]u8 = undefined;
const started = std.Io.Clock.awake.now(io);
try testing.expectError(error.Timeout, fc.exchange(io, query, &response_buf));
const elapsed = started.durationTo(std.Io.Clock.awake.now(io)).nanoseconds;
// Two budgets would spend 300 ms on the UDP leg and then a fresh 400 ms on
// the stalled TCP leg. The bound sits between one budget and that sum, so
// the double-budget shape cannot pass.
try testing.expect(elapsed < @as(i96, one_budget_ms + truncate_after_ms / 2) * std.time.ns_per_ms);
try testing.expectEqual(@as(u64, 1), fc.stats.udp_truncated);
try testing.expectEqual(@as(u64, 1), fc.stats.failures);
}
+556
View File
@@ -252,6 +252,200 @@ fn installWithMaxBytes(io: std.Io, cfg: model.Logging, max_bytes: u64) void {
if (state.output == .file) openFileLocked();
}
// ---------------------------------------------------------------------------
// Hot apply (milestone-34 S3.5)
// ---------------------------------------------------------------------------
/// Which of the three disjoint shapes a `logging` apply takes. The case is
/// decided from the FINAL MERGED config against the live sink, and it depends
/// only on `output` and `file_path` — fields nothing but an apply writes.
/// Rotation and write-failure recovery move `file`, `file_pos` and
/// `rotate_pending`, never these two, so a case decided in one lock hold is
/// still the right case in the next.
pub const ApplyCase = enum {
/// The merged config wants a file, and it is not the file that is open:
/// the path differs, or output is switching TO file.
target_changed,
/// Output is switching away from file. Nothing to open.
target_removed,
/// Everything else — output stays stderr/syslog, or output stays file on
/// the SAME path. Only config fields move; the handle and its position and
/// rotation state stay with the rotation machinery that owns them.
target_unchanged,
};
/// The complete new target state for a `target_changed` apply. The handle
/// couples to both other fields: inheriting the old `file_pos` would write
/// past the new file's end, and inheriting a pending rotation would rotate the
/// new target on its first line.
pub const PreparedSink = struct {
file: std.Io.File,
file_pos: u64,
rotate_pending: bool = false,
};
pub const PrepareError = error{
/// `file_path` does not fit in the sink's path buffer, so the sink could
/// not name the file it was told to write.
PathTooLong,
/// The new target could not be opened, created, or measured.
TargetUnopenable,
};
/// A validated `logging` apply, owning everything publish needs. Publish takes
/// no borrow from the request arena, so this outlives the request that built
/// it.
pub const PreparedApply = struct {
case: ApplyCase,
threshold: std.log.Level,
output: model.LogOutput,
path_buf: [std.Io.Dir.max_path_bytes]u8,
path_len: usize,
max_files: u8,
max_bytes: u64,
sink: ?PreparedSink,
pub fn path(self: *const PreparedApply) []const u8 {
return self.path_buf[0..self.path_len];
}
};
/// Prepare: everything fallible happens here, and nothing is published. A
/// `target_changed` apply opens the NEW file and MEASURES it — the open is the
/// step that can fail, and doing it here closes the close-then-reopen window
/// `installWithMaxBytes` has, where a bad new path leaves no sink at all.
pub fn prepareApply(io: std.Io, cfg: model.Logging) PrepareError!PreparedApply {
return prepareApplyWithMaxBytes(io, cfg, model.maxLogBytes(cfg));
}
/// Test-only entry point, mirroring `installForTest`.
pub fn prepareApplyForTest(io: std.Io, cfg: model.Logging, max_bytes: u64) PrepareError!PreparedApply {
return prepareApplyWithMaxBytes(io, cfg, max_bytes);
}
fn prepareApplyWithMaxBytes(io: std.Io, cfg: model.Logging, max_bytes: u64) PrepareError!PreparedApply {
if (cfg.file_path.len > std.Io.Dir.max_path_bytes) return error.PathTooLong;
var prepared: PreparedApply = .{
.case = undefined,
.threshold = toStdLevel(cfg.level),
.output = cfg.output,
.path_buf = undefined,
.path_len = cfg.file_path.len,
.max_files = cfg.max_files,
.max_bytes = max_bytes,
.sink = null,
};
@memcpy(prepared.path_buf[0..prepared.path_len], cfg.file_path);
prepared.case = classifyApply(cfg);
if (prepared.case == .target_changed) {
prepared.sink = try openTarget(io, prepared.path());
}
return prepared;
}
fn classifyApply(cfg: model.Logging) ApplyCase {
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
const currently_file = state.output == .file;
if (cfg.output != .file) return if (currently_file) .target_removed else .target_unchanged;
if (!currently_file) return .target_changed;
return if (std.mem.eql(u8, state.path(), cfg.file_path)) .target_unchanged else .target_changed;
}
/// Opens `p` and measures it, without touching the live sink.
fn openTarget(io: std.Io, p: []const u8) PrepareError!PreparedSink {
if (p.len == 0) return error.TargetUnopenable;
const prev = io.swapCancelProtection(.blocked);
defer _ = io.swapCancelProtection(prev);
const dir: std.Io.Dir = .cwd();
const file = dir.openFile(io, p, .{ .mode = .write_only }) catch |open_err| switch (open_err) {
error.FileNotFound => dir.createFile(io, p, .{ .truncate = false }) catch
return error.TargetUnopenable,
else => return error.TargetUnopenable,
};
errdefer file.close(io);
// A pre-existing nonempty target is appended to, so the new position is
// its measured length rather than zero.
const length = file.length(io) catch return error.TargetUnopenable;
return .{ .file = file, .file_pos = length, .rotate_pending = false };
}
/// Publish: infallible and I/O-free, one hold of the sink lock. Returns the
/// DETACHED old handle, which `retireApply` closes — closing a file is retire
/// work, and doing it here would put a syscall inside the publish.
pub fn publishApply(prepared: PreparedApply) ?std.Io.File {
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
state.threshold = prepared.threshold;
state.output = prepared.output;
state.path_len = prepared.path_len;
@memcpy(state.path_buf[0..state.path_len], prepared.path());
state.max_files = prepared.max_files;
state.max_bytes = prepared.max_bytes;
switch (prepared.case) {
// The handle and its position and rotation state are not this apply's
// to move; a broken handle is repaired by the existing per-write
// recovery, not by a config change.
.target_unchanged => return null,
.target_changed => {
const detached = state.file;
const sink = prepared.sink.?;
state.file = sink.file;
state.file_pos = sink.file_pos;
state.rotate_pending = sink.rotate_pending;
return detached;
},
.target_removed => {
const detached = state.file;
state.file = null;
state.file_pos = 0;
state.rotate_pending = false;
return detached;
},
}
}
/// Retire: closes the handle `publishApply` detached, after no writer can
/// reach it — the swap happened under the sink lock, so any writer that held
/// it has already returned.
pub fn retireApply(io: std.Io, detached: ?std.Io.File) void {
const file = detached orelse return;
const prev = io.swapCancelProtection(.blocked);
defer _ = io.swapCancelProtection(prev);
file.close(io);
}
/// Discards a prepared apply that will not be published, because its commit
/// failed or a sibling owner's prepare did.
pub fn abortApply(io: std.Io, prepared: PreparedApply) void {
const sink = prepared.sink orelse return;
const prev = io.swapCancelProtection(.blocked);
defer _ = io.swapCancelProtection(prev);
sink.file.close(io);
}
/// The directory the disk monitor should measure for `cfg`: the log file's
/// directory when output is `file`, and null otherwise — with output on
/// stderr or syslog there is no log file to run out of room for.
///
/// A path with no directory component measures the working directory, which is
/// where a bare filename lands.
pub fn logDirname(cfg: model.Logging) ?[]const u8 {
if (cfg.output != .file) return null;
if (cfg.file_path.len == 0) return null;
return std.fs.path.dirname(cfg.file_path) orelse ".";
}
/// Flushes and closes the file, and restores pass-through stderr formatting.
pub fn deinstall() void {
var stderr_buf: [64]u8 = undefined;
@@ -1099,3 +1293,365 @@ test "a message over the buffer is marked and counted" {
const colon = std.mem.indexOf(u8, written, ": ").?;
try testing.expectEqual(max_message_bytes, written[colon + 2 ..].len - 1);
}
// ---------------------------------------------------------------------------
// hot apply (milestone-34 S3.5)
// ---------------------------------------------------------------------------
/// The sink is one process-wide `state` behind the stderr lock, so an apply
/// test has to save it, drive the apply, and put it back. Nothing here holds
/// the lock across an apply call: `classifyApply` and `publishApply` take it
/// themselves.
const ApplyFixture = struct {
threaded: std.Io.Threaded,
tmp: testing.TmpDir,
saved: State,
fn init(self: *ApplyFixture) void {
self.threaded = .init(testing.allocator, .{});
self.tmp = testing.tmpDir(.{});
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
self.saved = state;
state = .{};
state.io = self.threaded.io();
}
fn deinit(self: *ApplyFixture) void {
{
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
if (state.file) |f| f.close(self.threaded.io());
state = self.saved;
}
self.tmp.cleanup();
self.threaded.deinit();
}
fn io(self: *ApplyFixture) std.Io {
return self.threaded.io();
}
fn path(self: *ApplyFixture, buf: []u8, name: []const u8) []const u8 {
return std.fmt.bufPrint(buf, ".zig-cache/tmp/{s}/{s}", .{ self.tmp.sub_path, name }) catch unreachable;
}
/// Puts the sink on `p` with a real open handle, the way a running server
/// with `output = file` sits.
fn openOn(self: *ApplyFixture, p: []const u8) void {
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
state.installed = true;
state.output = .file;
state.path_len = p.len;
@memcpy(state.path_buf[0..p.len], p);
state.max_bytes = 1 << 20;
state.max_files = 3;
openFileLocked();
_ = self;
}
fn snapshot(self: *ApplyFixture) State {
_ = self;
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
return state;
}
fn setHandleState(self: *ApplyFixture, file: ?std.Io.File, file_pos: u64, rotate_pending: bool) void {
_ = self;
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
state.file = file;
state.file_pos = file_pos;
state.rotate_pending = rotate_pending;
}
/// Writes one record through the live handle, as `emitFileLocked` does.
fn writeThroughSink(self: *ApplyFixture, text: []const u8) !void {
_ = self;
var stderr_buf: [64]u8 = undefined;
_ = std.debug.lockStderr(&stderr_buf);
defer std.debug.unlockStderr();
try writeLineLocked(state.file.?, text);
}
fn read(self: *ApplyFixture, buf: []u8, name: []const u8) ![]u8 {
return self.tmp.dir.readFileAlloc(self.io(), name, testing.allocator, .limited(buf.len)) catch |err| return err;
}
};
fn fileCfg(p: []const u8) model.Logging {
return .{ .output = .file, .file_path = p, .level = .info, .max_files = 3, .max_size_mb = 1 };
}
test "a bad target path is refused at prepare and the live sink is untouched" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
var buf: [160]u8 = undefined;
const live = fx.path(&buf, "nxdns.log");
fx.openOn(live);
const before = fx.snapshot();
try testing.expect(before.file != null);
// A directory that does not exist: the open and the create both fail.
var bad_buf: [200]u8 = undefined;
const bad = fx.path(&bad_buf, "no-such-dir/nxdns.log");
try testing.expectError(
error.TargetUnopenable,
prepareApplyForTest(fx.io(), fileCfg(bad), 1 << 20),
);
const after = fx.snapshot();
try testing.expectEqual(before.file.?.handle, after.file.?.handle);
try testing.expectEqualStrings(live, after.path());
}
test "a target change publishes without closing, and retire closes the old handle" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
var first_buf: [160]u8 = undefined;
var second_buf: [160]u8 = undefined;
const first = fx.path(&first_buf, "first.log");
const second = fx.path(&second_buf, "second.log");
fx.openOn(first);
const before = fx.snapshot();
const prepared = try prepareApplyForTest(fx.io(), fileCfg(second), 1 << 20);
try testing.expectEqual(ApplyCase.target_changed, prepared.case);
const detached = publishApply(prepared);
const after = fx.snapshot();
// Publish swapped the handle and left the old one OPEN: the old descriptor
// still writes, which it could not if publish had closed it.
try testing.expectEqual(before.file.?.handle, detached.?.handle);
try testing.expect(after.file.?.handle != detached.?.handle);
try testing.expectEqualStrings(second, after.path());
try testing.expectEqual(@as(u64, 0), after.file_pos);
try testing.expect(!after.rotate_pending);
var old_writer_buf: [64]u8 = undefined;
var ow = detached.?.writer(fx.io(), &old_writer_buf);
try ow.interface.writeAll("still open\n");
try ow.interface.flush();
retireApply(fx.io(), detached);
// The new target receives lines.
try fx.writeThroughSink("1 info: on the new target\n");
var read_buf: [256]u8 = undefined;
const contents = try fx.read(&read_buf, "second.log");
defer testing.allocator.free(contents);
try testing.expectEqualStrings("1 info: on the new target\n", contents);
}
test "switching to a pre-existing nonempty file starts at its measured length" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
const existing = "already here\n";
try fx.tmp.dir.writeFile(fx.io(), .{ .sub_path = "kept.log", .data = existing });
var first_buf: [160]u8 = undefined;
var kept_buf: [160]u8 = undefined;
fx.openOn(fx.path(&first_buf, "first.log"));
const kept = fx.path(&kept_buf, "kept.log");
const prepared = try prepareApplyForTest(fx.io(), fileCfg(kept), 1 << 20);
try testing.expectEqual(@as(u64, existing.len), prepared.sink.?.file_pos);
retireApply(fx.io(), publishApply(prepared));
try testing.expectEqual(@as(u64, existing.len), fx.snapshot().file_pos);
// Inheriting the old position would have overwritten the existing bytes.
try fx.writeThroughSink("appended\n");
var read_buf: [256]u8 = undefined;
const contents = try fx.read(&read_buf, "kept.log");
defer testing.allocator.free(contents);
try testing.expectEqualStrings(existing ++ "appended\n", contents);
}
test "a target change does not inherit a pending rotation" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
var first_buf: [160]u8 = undefined;
var second_buf: [160]u8 = undefined;
fx.openOn(fx.path(&first_buf, "first.log"));
const second = fx.path(&second_buf, "second.log");
// A rotation the old target owed and never completed: the handle is closed
// and the rotation is still pending.
const stale = fx.snapshot().file.?;
stale.close(fx.io());
fx.setHandleState(null, 0, true);
const prepared = try prepareApplyForTest(fx.io(), fileCfg(second), 1 << 20);
try testing.expect(!prepared.sink.?.rotate_pending);
retireApply(fx.io(), publishApply(prepared));
const after = fx.snapshot();
try testing.expect(!after.rotate_pending);
try testing.expect(after.file != null);
try testing.expectEqual(@as(u64, 0), after.file_pos);
}
test "a file to stderr apply detaches the handle and clears position and rotation" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
var buf: [160]u8 = undefined;
const live = fx.path(&buf, "nxdns.log");
fx.openOn(live);
fx.setHandleState(fx.snapshot().file, 4_096, true);
const before = fx.snapshot();
const prepared = try prepareApplyForTest(fx.io(), .{
.output = .stderr,
.file_path = live,
.level = .warn,
}, 1 << 20);
try testing.expectEqual(ApplyCase.target_removed, prepared.case);
const detached = publishApply(prepared);
const after = fx.snapshot();
try testing.expectEqual(before.file.?.handle, detached.?.handle);
try testing.expectEqual(@as(?std.Io.File, null), after.file);
try testing.expectEqual(@as(u64, 0), after.file_pos);
try testing.expect(!after.rotate_pending);
try testing.expectEqual(model.LogOutput.stderr, after.output);
// The path still moves, so a later switch back to file opens the right one.
try testing.expectEqualStrings(live, after.path());
retireApply(fx.io(), detached);
}
test "a same-path apply changes only config fields, whatever the handle is doing" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
var buf: [160]u8 = undefined;
const live = fx.path(&buf, "nxdns.log");
fx.openOn(live);
// Publish takes the same lock a rotation and a write-failure closure hold,
// so the only reachable interleavings are "before publish" and "after".
// Both leave the handle state the apply must not touch; these are the two
// states each of them leaves behind.
const handle_states = [_]struct { file: bool, pos: u64, pending: bool }{
// Mid-rotation: handle closed, rotation owed.
.{ .file = false, .pos = 0, .pending = true },
// Healthy and part-written.
.{ .file = true, .pos = 8_192, .pending = false },
};
const open_handle = fx.snapshot().file.?;
for (handle_states) |want| {
fx.setHandleState(if (want.file) open_handle else null, want.pos, want.pending);
const prepared = try prepareApplyForTest(fx.io(), .{
.output = .file,
.file_path = live,
.level = .debug,
.max_files = 9,
.max_size_mb = 7,
}, 4_242);
try testing.expectEqual(ApplyCase.target_unchanged, prepared.case);
// Nothing detached, so retire has nothing to close.
try testing.expectEqual(@as(?std.Io.File, null), publishApply(prepared));
const after = fx.snapshot();
try testing.expectEqual(want.pos, after.file_pos);
try testing.expectEqual(want.pending, after.rotate_pending);
try testing.expectEqual(want.file, after.file != null);
// The config fields did move.
try testing.expectEqual(std.log.Level.debug, after.threshold);
try testing.expectEqual(@as(u8, 9), after.max_files);
try testing.expectEqual(@as(u64, 4_242), after.max_bytes);
}
fx.setHandleState(open_handle, 0, false);
}
test "a file_path change while output is stderr updates the config and touches no handle" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
var buf: [160]u8 = undefined;
const later = fx.path(&buf, "later.log");
const prepared = try prepareApplyForTest(fx.io(), .{
.output = .syslog,
.file_path = later,
.level = .info,
}, 1 << 20);
try testing.expectEqual(ApplyCase.target_unchanged, prepared.case);
try testing.expectEqual(@as(?std.Io.File, null), publishApply(prepared));
const after = fx.snapshot();
try testing.expectEqual(@as(?std.Io.File, null), after.file);
try testing.expectEqualStrings(later, after.path());
// The later switch to file opens exactly that path.
const to_file = try prepareApplyForTest(fx.io(), fileCfg(later), 1 << 20);
try testing.expectEqual(ApplyCase.target_changed, to_file.case);
retireApply(fx.io(), publishApply(to_file));
try testing.expect(fx.snapshot().file != null);
}
test "an aborted apply closes the target it opened" {
var fx: ApplyFixture = undefined;
fx.init();
defer fx.deinit();
var buf: [160]u8 = undefined;
const target = fx.path(&buf, "never.log");
// The commit failed, so the prepared target must not leak its descriptor.
const prepared = try prepareApplyForTest(fx.io(), fileCfg(target), 1 << 20);
abortApply(fx.io(), prepared);
const after = fx.snapshot();
try testing.expectEqual(@as(?std.Io.File, null), after.file);
try testing.expectEqual(model.LogOutput.stderr, after.output);
}
test "logDirname follows output and file_path in both directions" {
try testing.expectEqualStrings("/var/log/nxdns", logDirname(.{
.output = .file,
.file_path = "/var/log/nxdns/nxdns.log",
}).?);
// A bare filename lands in the working directory.
try testing.expectEqualStrings(".", logDirname(.{
.output = .file,
.file_path = "nxdns.log",
}).?);
// Output away from file stops the measurement whatever the path says.
try testing.expectEqual(@as(?[]const u8, null), logDirname(.{
.output = .stderr,
.file_path = "/var/log/nxdns/nxdns.log",
}));
try testing.expectEqual(@as(?[]const u8, null), logDirname(.{
.output = .syslog,
.file_path = "/var/log/nxdns/nxdns.log",
}));
}
+274 -8
View File
@@ -109,10 +109,13 @@ pub const CertStore = struct {
pub const Kind = enum { doh, dot };
gpa: std.mem.Allocator,
/// Borrowed from the config; must outlive the store.
cert_path: []const u8,
/// Borrowed from the config; must outlive the store.
key_path: []const u8,
/// Owned. A settings apply replaces both paths, so a borrowed config slice
/// would dangle the moment the row it came from went away. Read and
/// written only under `reload_mutex`, which every apply and every reload
/// holds for its whole read-build-publish sequence.
cert_path: []u8,
/// Owned; see `cert_path`.
key_path: []u8,
/// Passed through to every `ServerContext.init`; the same lifetime rule
/// applies — a comptime-constant NULL-terminated array, never memory that
/// can go away before the store.
@@ -136,7 +139,8 @@ pub const CertStore = struct {
/// Test seam: runs inside `reload` between a successful `load` and the
/// publish, i.e. inside `reload_mutex`. Lets a test occupy the window
/// where an unserialized reload could be overtaken. Must not call
/// `reload` synchronously (that would self-deadlock on `reload_mutex`).
/// `reload`, `pollOnce` or `preparePathChange` synchronously — all four
/// take `reload_mutex`, which is not reentrant.
after_load_hook: ?ReloadHook,
/// Set by the composition root right after `init`, with `diagnostics`.
@@ -168,10 +172,17 @@ pub const CertStore = struct {
alpn: ?[*:null]const ?[*:0]const u8,
) ReloadError!CertStore {
const first = try load(gpa, io, cert_path, key_path, alpn);
errdefer {
first.entry.ctx.deinit(gpa);
gpa.destroy(first.entry);
}
const owned_cert = try gpa.dupe(u8, cert_path);
errdefer gpa.free(owned_cert);
const owned_key = try gpa.dupe(u8, key_path);
return .{
.gpa = gpa,
.cert_path = cert_path,
.key_path = key_path,
.cert_path = owned_cert,
.key_path = owned_key,
.alpn = alpn,
.mutex = .init,
.reload_mutex = .init,
@@ -193,6 +204,8 @@ pub const CertStore = struct {
std.debug.assert(current.refs == 0);
self.mutex.unlock(io);
self.destroyEntry(current);
self.gpa.free(self.cert_path);
self.gpa.free(self.key_path);
self.* = undefined;
}
@@ -226,7 +239,14 @@ pub const CertStore = struct {
pub fn reload(self: *CertStore, io: std.Io) ReloadError!void {
self.reload_mutex.lockUncancelable(io);
defer self.reload_mutex.unlock(io);
return self.reloadLocked(io);
}
/// `reload`'s body, for callers that already hold `reload_mutex` —
/// `pollOnce` does, because it must read `cert_path`/`key_path` under the
/// same lock an apply replaces them under. `std.Io.Mutex` is not
/// reentrant, so this exists rather than a recursive `reload` call.
fn reloadLocked(self: *CertStore, io: std.Io) ReloadError!void {
const next = load(self.gpa, io, self.cert_path, self.key_path, self.alpn) catch |err| {
_ = self.reload_failures.fetchAdd(1, .monotonic);
return err;
@@ -247,6 +267,93 @@ pub const CertStore = struct {
self.last_reload_unix.store(std.Io.Clock.real.now(io).toSeconds(), .monotonic);
}
/// A candidate certificate loaded from new paths, not yet published.
/// Holding one means holding `reload_mutex`: exactly one of
/// `publishPathChange` or `abortPathChange` must follow, and it releases
/// the lock.
pub const PreparedPaths = struct {
cert_path: []u8,
key_path: []u8,
loaded: Loaded,
/// Read at prepare so publish reads no clock: publish must touch
/// nothing outside memory it already owns.
loaded_at_unix: i64,
};
/// Prepare half of a `doh_server`/`dot_server` cert-path change: takes
/// `reload_mutex` and loads the certificate and key from the NEW paths.
/// Nothing is published, so a failure leaves the store exactly as it was —
/// the old certificate keeps serving and the caller writes no DB row. The
/// lock is released on failure and held on success, which is what makes
/// the whole apply serialized against `reload` and `pollOnce`.
pub fn preparePathChange(
self: *CertStore,
io: std.Io,
cert_path: []const u8,
key_path: []const u8,
) ReloadError!PreparedPaths {
self.reload_mutex.lockUncancelable(io);
errdefer self.reload_mutex.unlock(io);
const owned_cert = try self.gpa.dupe(u8, cert_path);
errdefer self.gpa.free(owned_cert);
const owned_key = try self.gpa.dupe(u8, key_path);
errdefer self.gpa.free(owned_key);
const next = load(self.gpa, io, cert_path, key_path, self.alpn) catch |err| {
_ = self.reload_failures.fetchAdd(1, .monotonic);
return err;
};
return .{
.cert_path = owned_cert,
.key_path = owned_key,
.loaded = next,
.loaded_at_unix = std.Io.Clock.real.now(io).toSeconds(),
};
}
/// Publish half: infallible and I/O-free. The paths and the generation
/// they were loaded from are installed together — the generation `mutex`
/// is taken only for that swap, inside `reload_mutex`, the same lock order
/// `reload` uses. Releases `reload_mutex`.
///
/// Connections that pinned the old generation finish on the old
/// certificate; the old entry is freed once its last reader releases.
pub fn publishPathChange(self: *CertStore, io: std.Io, prepared: PreparedPaths) void {
const old_cert_path = self.cert_path;
const old_key_path = self.key_path;
self.mutex.lockUncancelable(io);
const old = self.current;
self.cert_path = prepared.cert_path;
self.key_path = prepared.key_path;
self.current = prepared.loaded.entry;
self.loaded = prepared.loaded.sig;
old.retired = true;
const free_old = old.refs == 0;
self.mutex.unlock(io);
// The branch runs before the counter bump, never after: a `bool` still
// live across an atomic read-modify-write is the zig 0.16.0 Debug
// miscompile AGENTS.md documents.
if (free_old) self.destroyEntry(old);
self.gpa.free(old_cert_path);
self.gpa.free(old_key_path);
_ = self.reloads.fetchAdd(1, .monotonic);
self.last_reload_unix.store(prepared.loaded_at_unix, .monotonic);
self.reload_mutex.unlock(io);
}
/// Discards a prepared candidate — the commit that would have published it
/// failed, or a sibling owner's prepare did. Releases `reload_mutex`.
pub fn abortPathChange(self: *CertStore, io: std.Io, prepared: PreparedPaths) void {
self.gpa.free(prepared.cert_path);
self.gpa.free(prepared.key_path);
self.destroyEntry(prepared.loaded.entry);
self.reload_mutex.unlock(io);
}
/// Sleep first: `init` just loaded the files this poll would compare
/// against. `.boot` so a suspended box still sees the interval elapse.
pub fn watch(self: *CertStore, io: std.Io) std.Io.Cancelable!void {
@@ -265,7 +372,15 @@ pub const CertStore = struct {
/// changed, and the old one keeps serving either way. A failed reload
/// warns and counts (`reload_failures`); the signature stays at the loaded
/// pair, so every subsequent poll retries until the files parse.
///
/// The whole pass runs under `reload_mutex`: the paths it stats are the
/// ones an apply replaces, and a poll that read a path outside the lock
/// could stat a freed slice or reload a pair that was never published
/// together.
pub fn pollOnce(self: *CertStore, io: std.Io, now_s: i64) void {
self.reload_mutex.lockUncancelable(io);
defer self.reload_mutex.unlock(io);
const cert_sig = statSig(io, self.cert_path) catch |err| {
log.warn("stat {s} failed; keeping the loaded certificate", .{self.cert_path});
self.reportReload(io, now_s, "stat of the certificate failed", @errorName(err));
@@ -290,7 +405,7 @@ pub const CertStore = struct {
return;
}
if (self.reload(io)) {
if (self.reloadLocked(io)) {
log.info("certificate reloaded from {s}", .{self.cert_path});
if (self.diagnostics) |store| store.resolve(io, now_s, .certificate_reload, @tagName(self.kind));
} else |err| {
@@ -993,3 +1108,154 @@ test "an unchanged poll closes the episode a transient stat failure opened" {
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
// ---------------------------------------------------------------------------
// path apply (milestone-34 S3.4)
// ---------------------------------------------------------------------------
test "a bad candidate is refused at prepare and the store keeps serving" {
var env: TestEnv = undefined;
try env.init();
defer env.deinit();
const io = env.io();
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
const before = store.acquire(io);
store.release(io, before);
const before_paths_cert = store.cert_path;
try env.tmp.dir.writeFile(io, .{ .sub_path = "bad.pem", .data = "not a certificate" });
var bad_buf: [128]u8 = undefined;
const bad_path = try std.fmt.bufPrint(&bad_buf, ".zig-cache/tmp/{s}/bad.pem", .{env.tmp.sub_path});
try testing.expectError(
error.CertParse,
store.preparePathChange(io, bad_path, env.key_path),
);
// Nothing published: same generation, same paths, and `reload_mutex` was
// released — a second prepare would deadlock otherwise.
const after = store.acquire(io);
store.release(io, after);
try testing.expectEqual(before, after);
try testing.expectEqual(before_paths_cert.ptr, store.cert_path.ptr);
try testing.expectEqualStrings(env.cert_path, store.cert_path);
try testing.expectEqual(@as(u64, 0), store.snapshotStats().reloads);
try testing.expectEqual(@as(u64, 1), store.snapshotStats().reload_failures);
// A missing candidate path is refused the same way.
try testing.expectError(
error.CertUnreadable,
store.preparePathChange(io, "./nxdns-no-such-cert-4a11.pem", env.key_path),
);
}
test "a published path change installs the new pair and retires the old generation" {
var env: TestEnv = undefined;
try env.init();
defer env.deinit();
const io = env.io();
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
// A second, byte-different copy of the same valid pair under new names.
const grown = try std.mem.concat(testing.allocator, u8, &.{ fixtures.cert_pem, "\n" });
defer testing.allocator.free(grown);
try env.tmp.dir.writeFile(io, .{ .sub_path = "next-cert.pem", .data = grown });
try env.tmp.dir.writeFile(io, .{ .sub_path = "next-key.pem", .data = fixtures.key_pem });
var cert_buf: [128]u8 = undefined;
var key_buf: [128]u8 = undefined;
const next_cert = try std.fmt.bufPrint(&cert_buf, ".zig-cache/tmp/{s}/next-cert.pem", .{env.tmp.sub_path});
const next_key = try std.fmt.bufPrint(&key_buf, ".zig-cache/tmp/{s}/next-key.pem", .{env.tmp.sub_path});
// A connection pinned to the old generation finishes on it.
const pinned = store.acquire(io);
const prepared = try store.preparePathChange(io, next_cert, next_key);
store.publishPathChange(io, prepared);
try testing.expectEqualStrings(next_cert, store.cert_path);
try testing.expectEqualStrings(next_key, store.key_path);
try testing.expectEqual(@as(u64, 1), store.snapshotStats().reloads);
try testing.expect(pinned.retired);
const serving = store.acquire(io);
try testing.expect(serving != pinned);
store.release(io, serving);
store.release(io, pinned);
// The watcher now measures the new pair, so an untouched pair polls clean
// and a rewritten one reloads.
store.pollOnce(io, 1_000);
try testing.expectEqual(@as(u64, 1), store.snapshotStats().reloads);
try env.tmp.dir.writeFile(io, .{ .sub_path = "next-cert.pem", .data = fixtures.cert_pem });
store.pollOnce(io, 1_100);
try testing.expectEqual(@as(u64, 2), store.snapshotStats().reloads);
}
test "an aborted path change frees the candidate and leaves the store untouched" {
var env: TestEnv = undefined;
try env.init();
defer env.deinit();
const io = env.io();
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
const before = store.acquire(io);
store.release(io, before);
// The commit this candidate was built for failed; the testing allocator
// proves the abort frees everything the prepare took.
const prepared = try store.preparePathChange(io, env.cert_path, env.key_path);
store.abortPathChange(io, prepared);
const after = store.acquire(io);
store.release(io, after);
try testing.expectEqual(before, after);
try testing.expectEqual(@as(u64, 0), store.snapshotStats().reloads);
// `reload_mutex` came back, so the store still reloads.
try store.reload(io);
}
test "a reload racing a path apply is serialized behind it" {
var env: TestEnv = undefined;
try env.init();
defer env.deinit();
const io = env.io();
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
const grown = try std.mem.concat(testing.allocator, u8, &.{ fixtures.cert_pem, "\n" });
defer testing.allocator.free(grown);
try env.tmp.dir.writeFile(io, .{ .sub_path = "next-cert.pem", .data = grown });
try env.tmp.dir.writeFile(io, .{ .sub_path = "next-key.pem", .data = fixtures.key_pem });
var cert_buf: [128]u8 = undefined;
var key_buf: [128]u8 = undefined;
const next_cert = try std.fmt.bufPrint(&cert_buf, ".zig-cache/tmp/{s}/next-cert.pem", .{env.tmp.sub_path});
const next_key = try std.fmt.bufPrint(&key_buf, ".zig-cache/tmp/{s}/next-key.pem", .{env.tmp.sub_path});
// Prepare holds `reload_mutex` across the whole apply.
const prepared = try store.preparePathChange(io, next_cert, next_key);
try testing.expect(!store.reload_mutex.tryLock());
// The concurrent reload cannot start, so it cannot publish the OLD paths
// over the new generation.
var racing = try io.concurrent(CertStore.reload, .{ &store, io });
store.publishPathChange(io, prepared);
try racing.await(io);
// Two publications, and the last word is the apply's pair: the racing
// reload reread the paths the apply installed.
try testing.expectEqual(@as(u64, 2), store.snapshotStats().reloads);
try testing.expectEqualStrings(next_cert, store.cert_path);
store.mutex.lockUncancelable(io);
const final = store.loaded;
store.mutex.unlock(io);
try testing.expectEqual(@as(u64, grown.len), final.cert.size);
}
+59 -18
View File
@@ -27,6 +27,7 @@ const db = @import("../storage/db.zig");
const disk_monitor = @import("../storage/disk_monitor.zig");
const events = @import("../storage/events.zig");
const logger = @import("../storage/logger.zig");
const retention = @import("../storage/retention.zig");
const log = std.log.scoped(.clients);
@@ -59,7 +60,7 @@ pub const Tracker = struct {
/// Guards `pending`, `count`, `passes` and `stats`. Every field below is
/// written under it, so a reader takes it too; see `snapshotStats`.
mutex: std.Io.Mutex,
retention_days: u16,
retention_days: *const retention.RetentionDays,
pending: [max_pending]Pending,
count: u32,
passes: u64,
@@ -70,8 +71,9 @@ pub const Tracker = struct {
/// `retention_days` is `logging.retention_days`, the same knob the query log
/// prunes by (milestone-7 ruling 16). A client silent for that long is as
/// uninteresting as a query that old.
pub fn init(retention_days: u16) Tracker {
/// uninteresting as a query that old — so both consumers share ONE cell
/// and a settings apply moves them together.
pub fn init(retention_days: *const retention.RetentionDays) Tracker {
return .{
.mutex = .init,
.retention_days = retention_days,
@@ -225,7 +227,7 @@ pub const Tracker = struct {
self.mutex.unlock(io);
if (due) {
const cutoff = now_s - @as(i64, self.retention_days) * 86_400;
const cutoff = now_s - self.retention_days.seconds();
if (clients_repo.pruneStale(database, cutoff)) |deleted| {
self.mutex.lockUncancelable(io);
self.stats.pruned += deleted;
@@ -328,7 +330,8 @@ test "a client tracked twice before a flush yields one row at the later time" {
var database = try openMigrated();
defer database.close();
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
// A table with room reports no drops.
try testing.expectEqual(@as(u64, 0), tracker.trackAt(io, parsed("192.168.1.10"), 1700000000));
try testing.expectEqual(@as(u64, 0), tracker.trackAt(io, parsed("192.168.1.10"), 1700000030));
@@ -354,7 +357,8 @@ test "distinct clients each get a row and ipv6 text is canonical" {
var database = try openMigrated();
defer database.close();
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000000);
_ = tracker.trackAt(io, parsed("192.168.1.11"), 1700000001);
_ = tracker.trackAt(io, parsed("fd00:0:0:0:0:0:0:1"), 1700000002);
@@ -378,7 +382,8 @@ test "a full table drops further clients and counts them" {
var database = try openMigrated();
defer database.close();
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
for (0..Tracker.max_pending) |i| {
var octets: [4]u8 = undefined;
std.mem.writeInt(u32, &octets, @intCast(i), .big);
@@ -421,7 +426,8 @@ test "a flush touches a hand-edited row without changing what the operator set"
\\VALUES ('192.168.1.10', 'laptop', 2, 1, 1690000000, 1690000000);
);
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000000);
tracker.flushOnce(io, &database, true, null);
@@ -449,7 +455,8 @@ test "a gated pass writes nothing and keeps the pending clients" {
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
try testing.expect(!monitor.writesAllowed());
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000000);
tracker.flushOnce(io, &database, monitor.writesAllowed(), null);
@@ -477,7 +484,8 @@ test "a failing upsert counts and leaves the client to be tracked again" {
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000000);
_ = tracker.trackAt(io, parsed("192.168.1.11"), 1700000000);
tracker.flushOnce(io, &database, true, null);
@@ -507,7 +515,8 @@ test "the pass that comes due prunes the clients that went quiet" {
try clients_repo.upsertSeen(&database, "10.0.0.1", now - 40 * day);
try clients_repo.upsertSeen(&database, "10.0.0.2", now - 29 * day);
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
// Every pass before the due one leaves both rows alone.
for (0..Tracker.prune_every_passes - 1) |_| {
tracker.flushOnce(io, &database, true, null);
@@ -534,7 +543,8 @@ test "a shorter retention prunes what the default keeps" {
const now = std.Io.Clock.real.now(io).toSeconds();
try clients_repo.upsertSeen(&database, "10.0.0.1", now - 3 * 86_400);
var tracker: Tracker = .init(1);
var days: retention.RetentionDays = .init(1);
var tracker: Tracker = .init(&days);
tracker.passes = Tracker.prune_every_passes - 1;
tracker.flushOnce(io, &database, true, null);
@@ -542,6 +552,31 @@ test "a shorter retention prunes what the default keeps" {
try testing.expectEqual(@as(u64, 1), tracker.snapshotStats(io).pruned);
}
test "setRetentionDays changes the cutoff the next prune pass uses" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
const now = std.Io.Clock.real.now(io).toSeconds();
try clients_repo.upsertSeen(&database, "10.0.0.1", now - 3 * 86_400);
var days: retention.RetentionDays = .init(7);
var tracker: Tracker = .init(&days);
tracker.passes = Tracker.prune_every_passes - 1;
tracker.flushOnce(io, &database, true, null);
try testing.expectEqual(@as(i64, 1), try clients_repo.countClients(&database));
// The shared cell, not a copy taken at construction.
days.setRetentionDays(1);
tracker.passes = Tracker.prune_every_passes - 1;
tracker.flushOnce(io, &database, true, null);
try testing.expectEqual(@as(i64, 0), try clients_repo.countClients(&database));
try testing.expectEqual(@as(u64, 1), tracker.snapshotStats(io).pruned);
}
test "the run loop flushes on its interval and returns on cancel" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -550,7 +585,8 @@ test "the run loop flushes on its interval and returns on cancel" {
var database = try openMigrated();
defer database.close();
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000000);
var future = try io.concurrent(Tracker.run, .{
@@ -622,7 +658,8 @@ test "the drain lands before any exchange, and attempts stop at the cap" {
names.exchange_fn = CountingExchange.exchange;
CountingExchange.reset(&database);
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
const pending = clients_repo.max_per_pass + 4;
for (0..pending) |i| {
_ = tracker.trackAt(io, .{ .ip4 = .{ 192, 168, 2, @intCast(i) } }, 1700000000);
@@ -656,7 +693,8 @@ test "a row the due pass prunes is never asked about" {
const now = std.Io.Clock.real.now(io).toSeconds();
try clients_repo.upsertSeen(&database, "192.168.1.10", now - 40 * 86_400);
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
tracker.passes = Tracker.prune_every_passes - 1;
tracker.flushOnce(io, &database, true, &names);
@@ -681,7 +719,8 @@ test "a gated pass attempts no naming either" {
CountingExchange.reset(&database);
try clients_repo.upsertSeen(&database, "192.168.1.10", 1700000000);
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
tracker.flushOnce(io, &database, false, &names);
try testing.expectEqual(@as(usize, 0), CountingExchange.calls);
@@ -700,7 +739,8 @@ test "a failing materialise opens one episode per pass and a clean pass closes i
try fx.init(io, 1000);
defer fx.deinit();
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
tracker.diagnostics = &fx.store;
try database.exec(
@@ -742,7 +782,8 @@ test "a failing prune opens its own episode the next due pass closes" {
try fx.init(io, 1000);
defer fx.deinit();
var tracker: Tracker = .init(30);
var days: retention.RetentionDays = .init(30);
var tracker: Tracker = .init(&days);
tracker.diagnostics = &fx.store;
// One pass short of due, so the pass below is the pruning one.
tracker.passes = Tracker.prune_every_passes - 1;
+5 -3
View File
@@ -36,6 +36,7 @@ const listener = @import("listener.zig");
const model = @import("../config/model.zig");
const tls_server = @import("../platform/tls_server.zig");
const transport = @import("../upstream/transport.zig");
const upstream_owner = @import("../upstream/owner.zig");
pub const dns_query_path = "/dns-query";
@@ -619,6 +620,7 @@ const Harness = struct {
store: cert_store.CertStore,
tables: local_tables_mod.LocalTables,
upstream: FailingUpstream,
upstream_owner: upstream_owner.Borrowed,
h: handler.Handler,
server: DohServer,
group: std.Io.Group,
@@ -646,10 +648,10 @@ const Harness = struct {
errdefer hx.tables.deinit(testing.allocator);
hx.upstream = .{};
hx.upstream_owner = .{};
hx.h = .{
.upstream = hx.upstream.client(),
.blocking = test_blocking,
.forward_read_timeout = test_forward_timeout,
.upstream = hx.upstream_owner.client(hx.upstream.client()),
.policy = .{ .blocking = test_blocking, .forward_read_timeout = test_forward_timeout },
.local_tables = &hx.tables,
};
+103 -9
View File
@@ -28,6 +28,7 @@ const handler = @import("handler.zig");
const listener = @import("listener.zig");
const tls_server = @import("../platform/tls_server.zig");
const transport = @import("../upstream/transport.zig");
const upstream_owner = @import("../upstream/owner.zig");
/// Plaintext staging for `ServerStream`: the framing bytes and the decrypted
/// record tail pass through here, while whole messages go straight to
@@ -312,11 +313,10 @@ const forward_timeout: std.Io.Clock.Duration = .{
/// An upstream and nothing else optional: no filtering, no cache, no log. The
/// listener is what these tests exercise, so the handler is the same bare one
/// its own tests use.
fn bareHandler(client: transport.Client) handler.Handler {
fn bareHandler(up: *upstream_owner.Owner) handler.Handler {
return .{
.upstream = client,
.blocking = blocking,
.forward_read_timeout = forward_timeout,
.upstream = up,
.policy = .{ .blocking = blocking, .forward_read_timeout = forward_timeout },
};
}
@@ -567,7 +567,8 @@ test "dot: two framed queries share one TLS connection" {
defer env.deinit(io);
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: std.Io.net.IpAddress = try .parse("127.0.0.1", 0);
var server = try DotServer.listen(gpa, io, listen_address, &h, &env.store, .{ .max_connections = 2 });
@@ -607,7 +608,8 @@ test "dot: a transport EOF without close_notify is a connection error, not a cra
defer env.deinit(io);
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: std.Io.net.IpAddress = try .parse("127.0.0.1", 0);
var server = try DotServer.listen(gpa, io, listen_address, &h, &env.store, .{ .max_connections = 2 });
@@ -645,7 +647,8 @@ test "dot: plain TCP bytes fail the handshake and are counted" {
defer env.deinit(io);
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: std.Io.net.IpAddress = try .parse("127.0.0.1", 0);
var server = try DotServer.listen(gpa, io, listen_address, &h, &env.store, .{ .max_connections = 2 });
@@ -731,7 +734,8 @@ test "dot: a reload serves new handshakes without breaking the old connection" {
defer env.deinit(io);
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: std.Io.net.IpAddress = try .parse("127.0.0.1", 0);
var server = try DotServer.listen(gpa, io, listen_address, &h, &env.store, .{ .max_connections = 2 });
@@ -759,6 +763,95 @@ test "dot: a reload serves new handshakes without breaking the old connection" {
};
}
/// S3.4 across a live listener: a cert PATH change — new files, not rewritten
/// ones — serves the new certificate on the next handshake while the
/// connection pinned to the old generation finishes on it.
fn dotPathChangeServesNewCert(io: std.Io, address_: std.Io.net.IpAddress, env: *CertEnv) anyerror!void {
var first: TestTls = undefined;
try first.connect(io, address_);
defer first.close(io);
try first.sendQuery();
try expectAnswersQuery(try first.readReply());
const old_entry = env.store.acquire(io);
defer env.store.release(io, old_entry);
try env.tmp.dir.writeFile(io, .{ .sub_path = "cert2.pem", .data = fixtures.cert2_pem });
try env.tmp.dir.writeFile(io, .{ .sub_path = "key2.pem", .data = fixtures.key2_pem });
var cert_buf: [128]u8 = undefined;
var key_buf: [128]u8 = undefined;
const next_cert = try std.fmt.bufPrint(&cert_buf, ".zig-cache/tmp/{s}/cert2.pem", .{env.tmp.sub_path});
const next_key = try std.fmt.bufPrint(&key_buf, ".zig-cache/tmp/{s}/key2.pem", .{env.tmp.sub_path});
const prepared = try env.store.preparePathChange(io, next_cert, next_key);
env.store.publishPathChange(io, prepared);
const new_entry = env.store.acquire(io);
defer env.store.release(io, new_entry);
try testing.expect(old_entry != new_entry);
var second: TestTls = undefined;
try second.connect(io, address_);
defer second.close(io);
try second.sendQuery();
try expectAnswersQuery(try second.readReply());
try first.sendQuery();
try expectAnswersQuery(try first.readReply());
try second.client.end();
try second.net_writer.interface.flush();
var second_tail: [1]u8 = undefined;
try testing.expectError(error.EndOfStream, second.client.reader.readSliceAll(&second_tail));
try first.client.end();
try first.net_writer.interface.flush();
var first_tail: [1]u8 = undefined;
try testing.expectError(error.EndOfStream, first.client.reader.readSliceAll(&first_tail));
}
test "dot: a cert path change serves the new certificate on the next handshake" {
const build_options = @import("build_options");
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
var threaded: std.Io.Threaded = .init(gpa, .{});
defer threaded.deinit();
const io = threaded.io();
var env: CertEnv = undefined;
try env.init(io);
defer env.deinit(io);
var fake: FakeUpstream = .{ .reply = response_bytes };
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: std.Io.net.IpAddress = try .parse("127.0.0.1", 0);
var server = try DotServer.listen(gpa, io, listen_address, &h, &env.store, .{ .max_connections = 2 });
const server_address = server.boundAddress();
var group: std.Io.Group = .init;
try group.concurrent(io, DotServer.serve, .{ &server, io });
try bounded(io, dotPathChangeServesNewCert, .{ io, server_address, &env });
const stats = server.snapshotStats();
try testing.expectEqual(@as(u64, 2), stats.connections);
try testing.expectEqual(@as(u64, 0), stats.tls_handshake_failures);
try testing.expectEqual(@as(u64, 0), stats.connection_errors);
const store_stats = env.store.snapshotStats();
try testing.expectEqual(@as(u64, 1), store_stats.reloads);
try testing.expectEqual(@as(u64, 0), store_stats.reload_failures);
server.deinit(io);
group.await(io) catch |err| switch (err) {
error.Canceled => unreachable,
};
}
test "dot: an idle connection is closed with close_notify and counted" {
const build_options = @import("build_options");
if (!build_options.integration) return error.SkipZigTest;
@@ -773,7 +866,8 @@ test "dot: an idle connection is closed with close_notify and counted" {
defer env.deinit(io);
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: std.Io.net.IpAddress = try .parse("127.0.0.1", 0);
var server = try DotServer.listen(gpa, io, listen_address, &h, &env.store, .{
+542 -143
View File
File diff suppressed because it is too large Load Diff
+43 -21
View File
@@ -32,6 +32,7 @@ const forward_zones = @import("../local/forward_zones.zig");
const handler = @import("handler.zig");
const header = @import("../dns/header.zig");
const local_tables = @import("local_tables.zig");
const logger_controller = @import("../storage/logger_controller.zig");
const logger_mod = @import("../storage/logger.zig");
const provenance = @import("../storage/provenance.zig");
const manager = @import("../filter/manager.zig");
@@ -47,8 +48,10 @@ const rate_limiter = @import("rate_limiter.zig");
const record = @import("../dns/record.zig");
const records = @import("../local/records.zig");
const response = @import("../filter/response.zig");
const retention_mod = @import("../storage/retention.zig");
const shutdown = @import("shutdown.zig");
const transport = @import("../upstream/transport.zig");
const upstream_owner = @import("../upstream/owner.zig");
const types = @import("../dns/types.zig");
const udp_server = @import("udp_server.zig");
@@ -79,11 +82,10 @@ const zone_ttl: u32 = 120;
/// The handler every case starts from: an upstream, the blocking options and
/// the empty local tables. Each case wires in the collaborators it exercises.
fn baseHandler(client: transport.Client) handler.Handler {
fn baseHandler(up: *upstream_owner.Owner) handler.Handler {
return .{
.upstream = client,
.blocking = blocking,
.forward_read_timeout = forward_timeout,
.upstream = up,
.policy = .{ .blocking = blocking, .forward_read_timeout = forward_timeout },
};
}
@@ -265,6 +267,11 @@ fn fixtureManager(m: *manager.Manager, snapshot: *matcher.Snapshot) void {
.paths = undefined,
.fetcher = undefined,
.update = .{},
.schedule_mutex = .init,
.schedule_version = 0,
.schedule_anchor_s = null,
.schedule_event = .unset,
.schedule_clock = .real,
.total_budget = forward_timeout,
.lock = .init,
.writer_lock = .init,
@@ -317,10 +324,12 @@ test "S7 case 1: a blocked domain is answered with the zero address and logged"
var queue_buf: [log_queue_len]logger_mod.Entry = undefined;
var lg: logger_mod.Logger = .init(.{}, &queue_buf);
var sink: query_sink.QuerySink = .init(&lg, null);
var log_owner: logger_controller.Borrowed = .{};
var sink: query_sink.QuerySink = .init(log_owner.over(&lg), null);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.manager = &mgr;
h.sink = &sink;
@@ -374,7 +383,8 @@ test "S7 case 2: an allow rule beats the blocklist and the upstream answers" {
fixtureManager(&mgr, &snapshot);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.manager = &mgr;
var loop = try Loop.bind(gpa, io, &h);
@@ -411,7 +421,8 @@ test "S7 case 3: a local record answers authoritatively without an upstream" {
defer table.deinit(gpa);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
var tables: local_tables.LocalTables = .{ .records = table };
h.local_tables = &tables;
@@ -479,12 +490,13 @@ test "S7 case 4: a forward zone reaches its resolver, bypasses the blocklist and
defer cache.deinit();
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.manager = &mgr;
var tables: local_tables.LocalTables = .{ .zones = zones };
h.local_tables = &tables;
h.cache = &cache;
h.negative_ttl_max = 3600;
h.policy.negative_ttl_max = 3600;
var loop = try Loop.bind(gpa, io, &h);
defer loop.stop(gpa, io);
@@ -533,12 +545,14 @@ test "S7 case 5: a cached answer comes back with a fresh id, an aged ttl and a l
var queue_buf: [log_queue_len]logger_mod.Entry = undefined;
var lg: logger_mod.Logger = .init(.{}, &queue_buf);
var sink: query_sink.QuerySink = .init(&lg, null);
var log_owner: logger_controller.Borrowed = .{};
var sink: query_sink.QuerySink = .init(log_owner.over(&lg), null);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.cache = &cache;
h.negative_ttl_max = 3600;
h.policy.negative_ttl_max = 3600;
h.sink = &sink;
var loop = try Loop.bind(gpa, io, &h);
@@ -622,10 +636,12 @@ test "S7 case 6: a cname into a blocked target blocks the original question" {
var queue_buf: [log_queue_len]logger_mod.Entry = undefined;
var lg: logger_mod.Logger = .init(.{}, &queue_buf);
var sink: query_sink.QuerySink = .init(&lg, null);
var log_owner: logger_controller.Borrowed = .{};
var sink: query_sink.QuerySink = .init(log_owner.over(&lg), null);
var fake: FakeUpstream = .{ .reply = .{ .cname = "tracker.example.org" } };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.manager = &mgr;
h.sink = &sink;
@@ -681,7 +697,8 @@ test "S7 case 7: safe search answers the original question with a cname to the t
fixtureManager(&mgr, &snapshot);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.manager = &mgr;
var loop = try Loop.bind(gpa, io, &h);
@@ -733,10 +750,12 @@ test "S7 case 8: the third query inside the window is refused" {
var queue_buf: [log_queue_len]logger_mod.Entry = undefined;
var lg: logger_mod.Logger = .init(.{}, &queue_buf);
var sink: query_sink.QuerySink = .init(&lg, null);
var log_owner: logger_controller.Borrowed = .{};
var sink: query_sink.QuerySink = .init(log_owner.over(&lg), null);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.limiter = &limiter;
h.sink = &sink;
@@ -785,7 +804,8 @@ test "S7 case 9: pause lifts filtering and unpause restores it" {
paused.pauseFor(std.Io.Clock.real.now(io).toSeconds(), null);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.manager = &mgr;
h.pause = &paused;
@@ -831,10 +851,12 @@ test "S7 case 10: the querying client is materialised as a row" {
try db.applyPragmas(&database, .{});
_ = try migrations.migrate(&database);
var tracker: clients.Tracker = .init(30);
var retention_days: retention_mod.RetentionDays = .init(30);
var tracker: clients.Tracker = .init(&retention_days);
var fake: FakeUpstream = .{ .reply = .a };
var h = baseHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = baseHandler(h_owner.client(fake.client()));
h.tracker = &tracker;
var loop = try Loop.bind(gpa, io, &h);
+25 -9
View File
@@ -13,24 +13,36 @@
const std = @import("std");
const logger = @import("../storage/logger.zig");
const logger_controller = @import("../storage/logger_controller.zig");
const sse = @import("../web/sse.zig");
pub const QuerySink = struct {
logger: *logger.Logger,
/// The controller, not a `Logger`: `logging.query_log_buffer_max` can
/// change while the server runs, and the generation a producer enqueues
/// into has to be the one that is live at that moment.
controller: *logger_controller.Controller,
/// Null when `web.enabled` is false: nothing subscribes, so nothing needs
/// a hub, and the DNS path pays one null check.
hub: ?*sse.Hub,
pub fn init(query_logger: *logger.Logger, hub: ?*sse.Hub) QuerySink {
return .{ .logger = query_logger, .hub = hub };
pub fn init(controller: *logger_controller.Controller, hub: ?*sse.Hub) QuerySink {
return .{ .controller = controller, .hub = hub };
}
/// Transforms once, publishes, then enqueues. Never blocks the query path
/// and never fails: both consumers drop rather than wait.
///
/// The borrow spans both halves. A resize that lands between them would
/// otherwise leave this entry going into a queue retirement has already
/// closed, and that is exactly the drop window the controller exists to
/// make impossible.
pub fn log(self: *QuerySink, io: std.Io, entry: logger.Entry) void {
const transformed = self.logger.transformed(entry);
const generation = self.controller.acquire(io);
defer self.controller.release(io, generation);
const transformed = generation.logger.transformed(entry);
if (self.hub) |hub| hub.publish(io, transformed);
self.logger.logTransformed(io, transformed);
generation.logger.logTransformed(io, transformed);
}
};
@@ -60,7 +72,8 @@ test "the sink publishes and logs the same entry" {
var queue_buf: [4]logger.Entry = undefined;
var query_logger: logger.Logger = .init(.{}, &queue_buf);
var sink: QuerySink = .init(&query_logger, hub);
var owner: logger_controller.Borrowed = .{};
var sink: QuerySink = .init(owner.over(&query_logger), hub);
const id = hub.subscribe(io).?;
defer hub.unsubscribe(io, id);
@@ -87,7 +100,8 @@ test "fanout does not depend on the entry reaching the queue" {
var queue_buf: [4]logger.Entry = undefined;
var query_logger: logger.Logger = .init(.{}, &queue_buf);
var sink: QuerySink = .init(&query_logger, hub);
var owner: logger_controller.Borrowed = .{};
var sink: QuerySink = .init(owner.over(&query_logger), hub);
const id = hub.subscribe(io).?;
defer hub.unsubscribe(io, id);
@@ -115,7 +129,8 @@ test "the privacy transforms run once, before both consumers" {
.{ .hide_domains = true, .hide_client_ips = true },
&queue_buf,
);
var sink: QuerySink = .init(&query_logger, hub);
var owner: logger_controller.Borrowed = .{};
var sink: QuerySink = .init(owner.over(&query_logger), hub);
const id = hub.subscribe(io).?;
defer hub.unsubscribe(io, id);
@@ -138,7 +153,8 @@ test "a sink without a hub still logs" {
var queue_buf: [4]logger.Entry = undefined;
var query_logger: logger.Logger = .init(.{}, &queue_buf);
var sink: QuerySink = .init(&query_logger, null);
var owner: logger_controller.Borrowed = .{};
var sink: QuerySink = .init(owner.over(&query_logger), null);
sink.log(io, sampleEntry(5, "nohub.example"));
+8 -6
View File
@@ -26,6 +26,7 @@ const types = @import("../dns/types.zig");
const health = @import("../upstream/health.zig");
const pool = @import("../upstream/pool.zig");
const transport = @import("../upstream/transport.zig");
const upstream_owner = @import("../upstream/owner.zig");
const testing = std.testing;
@@ -42,11 +43,10 @@ const forward_timeout: std.Io.Clock.Duration = .{
/// An upstream and nothing else optional: no filtering, no cache, no log. The
/// listeners and the pool are what this test exercises, so the handler is the
/// same bare one its own tests use.
fn bareHandler(client: transport.Client) handler.Handler {
fn bareHandler(up: *upstream_owner.Owner) handler.Handler {
return .{
.upstream = client,
.blocking = blocking,
.forward_read_timeout = forward_timeout,
.upstream = up,
.policy = .{ .blocking = blocking, .forward_read_timeout = forward_timeout },
};
}
@@ -176,7 +176,7 @@ const EntryStorage = struct {
.priority = priority,
.enabled = true,
.health = .init,
.sem = .{ .permits = self.slots.len },
.admission = .{ .permits = self.slots.len },
.reuse_recoveries = &self.recoveries,
};
}
@@ -275,7 +275,9 @@ test "the whole resolver answers over udp and tcp and fails over to a healthy up
};
var upstreams: pool.Pool = .init(&entries, test_cfg, pool_timeouts, 1);
var h = bareHandler(upstreams.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(upstreams.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var udp = try udp_server.UdpServer.bind(gpa, io, listen_address, &h, .{ .max_in_flight = 4 });
+16 -10
View File
@@ -21,6 +21,7 @@ const header = @import("../dns/header.zig");
const packet = @import("../dns/packet.zig");
const types = @import("../dns/types.zig");
const transport = @import("../upstream/transport.zig");
const upstream_owner = @import("../upstream/owner.zig");
const testing = std.testing;
@@ -37,11 +38,10 @@ const forward_timeout: std.Io.Clock.Duration = .{
/// An upstream and nothing else optional: no filtering, no cache, no log. The
/// listener is what these tests exercise, so the handler is the same bare one
/// its own tests use.
fn bareHandler(client: transport.Client) handler.Handler {
fn bareHandler(up: *upstream_owner.Owner) handler.Handler {
return .{
.upstream = client,
.blocking = blocking,
.forward_read_timeout = forward_timeout,
.upstream = up,
.policy = .{ .blocking = blocking, .forward_read_timeout = forward_timeout },
};
}
@@ -175,7 +175,8 @@ test "two length-prefixed queries share one connection" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try tcp_server.TcpServer.listen(gpa, io, listen_address, &h, .{ .max_connections = 2 });
@@ -205,7 +206,8 @@ test "the claimed slot records the connecting client" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try tcp_server.TcpServer.listen(gpa, io, listen_address, &h, .{ .max_connections = 2 });
@@ -282,7 +284,8 @@ test "a canceled serve does not wait for a live connection" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
// The idle budget is the whole time a drain would have to wait out, so it
@@ -344,7 +347,8 @@ test "an idle connection is closed and counted" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try tcp_server.TcpServer.listen(gpa, io, listen_address, &h, .{
@@ -377,7 +381,8 @@ test "a zero-length message is a connection error" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try tcp_server.TcpServer.listen(gpa, io, listen_address, &h, .{
@@ -429,7 +434,8 @@ test "deinit ends a serve loop that is blocked on accept" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try tcp_server.TcpServer.listen(gpa, io, listen_address, &h, .{ .max_connections = 2 });
+12 -8
View File
@@ -20,6 +20,7 @@ const header = @import("../dns/header.zig");
const packet = @import("../dns/packet.zig");
const types = @import("../dns/types.zig");
const transport = @import("../upstream/transport.zig");
const upstream_owner = @import("../upstream/owner.zig");
const testing = std.testing;
@@ -36,11 +37,10 @@ const forward_timeout: std.Io.Clock.Duration = .{
/// An upstream and nothing else optional: no filtering, no cache, no log. The
/// listener is what these tests exercise, so the handler is the same bare one
/// its own tests use.
fn bareHandler(client: transport.Client) handler.Handler {
fn bareHandler(up: *upstream_owner.Owner) handler.Handler {
return .{
.upstream = client,
.blocking = blocking,
.forward_read_timeout = forward_timeout,
.upstream = up,
.policy = .{ .blocking = blocking, .forward_read_timeout = forward_timeout },
};
}
@@ -112,7 +112,8 @@ test "a udp query is answered on the loopback" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try udp_server.UdpServer.bind(gpa, io, listen_address, &h, .{ .max_in_flight = 4 });
@@ -150,7 +151,8 @@ test "a runt datagram is dropped and no reply is sent" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try udp_server.UdpServer.bind(gpa, io, listen_address, &h, .{ .max_in_flight = 4 });
@@ -185,7 +187,8 @@ test "an oversize datagram arrives truncated and is dropped" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try udp_server.UdpServer.bind(gpa, io, listen_address, &h, .{ .max_in_flight = 4 });
@@ -226,7 +229,8 @@ test "deinit ends a serve loop that is blocked on receive" {
const io = threaded.io();
var fake: FakeUpstream = .{ .reply = response_bytes };
var h = bareHandler(fake.client());
var h_owner: upstream_owner.Borrowed = .{};
var h = bareHandler(h_owner.client(fake.client()));
const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
var server = try udp_server.UdpServer.bind(gpa, io, listen_address, &h, .{ .max_in_flight = 4 });
+183
View File
@@ -54,9 +54,24 @@ pub const c = struct {
pub extern fn sqlite3_column_count(stmt: *c.Stmt) c_int;
pub extern fn sqlite3_column_type(stmt: *c.Stmt, col: c_int) c_int;
pub extern fn sqlite3_column_int64(stmt: *c.Stmt, col: c_int) i64;
pub extern fn sqlite3_column_double(stmt: *c.Stmt, col: c_int) f64;
pub extern fn sqlite3_column_text(stmt: *c.Stmt, col: c_int) ?[*]const u8;
pub extern fn sqlite3_column_bytes(stmt: *c.Stmt, col: c_int) c_int;
pub extern fn sqlite3_set_authorizer(db: *Sqlite3, xAuth: ?*const AuthCallback, user_data: ?*anyopaque) c_int;
pub extern fn sqlite3_get_autocommit(db: *Sqlite3) c_int;
pub extern fn sqlite3_last_insert_rowid(db: *Sqlite3) i64;
/// `int (*)(void*, int, const char*, const char*, const char*, const char*)`.
/// The four name arguments are NULL for actions that do not use them, which
/// `SQLITE_TRANSACTION` is except for its operation name.
pub const AuthCallback = fn (
user_data: ?*anyopaque,
action: c_int,
arg1: ?[*:0]const u8,
arg2: ?[*:0]const u8,
database: ?[*:0]const u8,
trigger_or_view: ?[*:0]const u8,
) callconv(.c) c_int;
pub extern fn sqlite3_changes(db: *Sqlite3) c_int;
pub extern fn sqlite3_total_changes(db: *Sqlite3) c_int;
};
@@ -105,6 +120,20 @@ pub const open_flag = struct {
pub const exrescode: c_int = 0x2000000;
};
/// The authorizer verdicts and the one action code nxdns denies, from the
/// vendored `sqlite3.h` (3.53.4). `SQLITE_DENY` fails the *prepare* with
/// `SQLITE_AUTH`, which is what makes it a real guard: the statement never
/// runs at all.
pub const auth = struct {
pub const deny: c_int = 1;
pub const transaction: c_int = 22;
/// `SAVEPOINT`, `RELEASE` and `ROLLBACK TO` report under this code, not
/// under `transaction`. A savepoint at the outermost level opens a real
/// transaction and `RELEASE` commits it, so a step using one splits the
/// migration exactly as a bare `COMMIT` would.
pub const savepoint: c_int = 32;
};
/// Column type codes returned by `sqlite3_column_type`.
pub const column_type = struct {
pub const integer: c_int = 1;
@@ -367,6 +396,7 @@ pub const Db = struct {
/// message is read back through `sqlite3_errmsg`, so there is no
/// `sqlite3_free` obligation.
pub fn exec(self: *Db, sql: [:0]const u8) Error!void {
if (execFaultTripped(sql)) return error.Internal;
return check(c.sqlite3_exec(self.handle, sql.ptr, null, null, null));
}
@@ -424,6 +454,38 @@ pub const Db = struct {
return value;
}
/// False while a transaction is open on this connection. The migration
/// runner's belt check: a step that somehow ended the runner's transaction
/// must not be allowed to look like a success.
pub fn inTransaction(self: *Db) bool {
return c.sqlite3_get_autocommit(self.handle) == 0;
}
/// Denies every transaction statement — `BEGIN`, `COMMIT`, `ROLLBACK`,
/// `SAVEPOINT`, `RELEASE` — until `clearAuthorizer` runs.
///
/// `guard` must outlive the installed window: SQLite keeps the pointer.
/// Install and clear are a scoped pair; the migration runner clears on
/// every exit path, because a leaked authorizer would go on denying the
/// ROLLBACK that cleans up after the very statement it rejected.
pub fn denyTransactions(self: *Db, guard: *TransactionGuard) Error!void {
guard.* = .{};
return check(c.sqlite3_set_authorizer(self.handle, transactionAuthorizer, guard));
}
/// Never fails in a way the caller can act on: passing a null callback only
/// clears state SQLite already holds. A failure is logged and swallowed so
/// this stays usable in `defer`.
pub fn clearAuthorizer(self: *Db) void {
const rc = c.sqlite3_set_authorizer(self.handle, null, null);
if (rc != result.ok) {
log.err("sqlite3_set_authorizer(null) returned {s} (code {d})", .{
std.mem.span(c.sqlite3_errstr(rc)),
rc,
});
}
}
pub fn lastInsertRowid(self: *Db) i64 {
return c.sqlite3_last_insert_rowid(self.handle);
}
@@ -442,6 +504,34 @@ pub const Db = struct {
}
};
/// Records whether the authorizer installed by `Db.denyTransactions` actually
/// rejected anything. The rejection reaches the caller as `error.Auth`, which
/// is indistinguishable from any other authorization failure; this flag is what
/// lets the migration runner name the real cause.
pub const TransactionGuard = struct {
denied: bool = false,
};
fn transactionAuthorizer(
user_data: ?*anyopaque,
action: c_int,
arg1: ?[*:0]const u8,
arg2: ?[*:0]const u8,
database: ?[*:0]const u8,
trigger_or_view: ?[*:0]const u8,
) callconv(.c) c_int {
_ = arg2;
_ = database;
_ = trigger_or_view;
if (action != auth.transaction and action != auth.savepoint) return result.ok;
const guard: *TransactionGuard = @ptrCast(@alignCast(user_data.?));
guard.denied = true;
log.warn("migration step attempted a transaction statement: {s}", .{
if (arg1) |op| std.mem.span(op) else "(unnamed)",
});
return auth.deny;
}
fn openHandle(filename: [:0]const u8, flags: c_int) Error!*c.Sqlite3 {
var handle: ?*c.Sqlite3 = null;
const rc = c.sqlite3_open_v2(filename.ptr, &handle, flags, null);
@@ -963,6 +1053,45 @@ pub const read_tx_faults = if (builtin.is_test) struct {
}
} else struct {};
/// Fails one chosen `Db.exec` so a test can drive a failure SQLite itself will
/// not produce on demand. Test builds only, same shape as `read_tx_seam`.
///
/// The migration paths this exists for — the legacy restamp and the
/// post-commit pragma restore — run statements that always succeed against a
/// healthy file, and their recovery behaviour is the whole point of the
/// milestone. Matching on the SQL text rather than counting calls keeps a test
/// naming the statement it means.
const exec_seam = if (builtin.is_test) struct {
var fail_matching: ?[]const u8 = null;
} else struct {};
fn execFaultTripped(sql: []const u8) bool {
if (!builtin.is_test) return false;
const needle = exec_seam.fail_matching orelse return false;
if (std.mem.indexOf(u8, sql, needle) == null) return false;
exec_seam.fail_matching = null;
return true;
}
/// The seam's controls, for tests in this file and in the storage layer.
pub const exec_faults = if (builtin.is_test) struct {
/// Arms the next `Db.exec` whose SQL contains `needle` to fail with
/// `error.Internal` before the statement reaches SQLite. One shot: it
/// disarms itself when it trips. Pair it with `defer disarm()` so a test
/// that never trips the fault cannot leak it into the next one.
pub fn failNextMatching(needle: []const u8) void {
exec_seam.fail_matching = needle;
}
pub fn disarm() void {
exec_seam.fail_matching = null;
}
pub fn armed() bool {
return exec_seam.fail_matching != null;
}
} else struct {};
const testing = std.testing;
fn openMemory() Error!Db {
@@ -1479,3 +1608,57 @@ test "a duplicate insert into a UNIQUE column returns error.Constraint" {
try stmt.bindText(1, "only");
try testing.expectError(error.Constraint, stmt.step());
}
test "the transaction authorizer denies transaction statements and clears cleanly" {
var db = try openMemory();
defer db.close();
try db.exec("CREATE TABLE t (id INTEGER PRIMARY KEY);");
try db.exec("BEGIN IMMEDIATE;");
try testing.expect(db.inTransaction());
var guard: TransactionGuard = .{};
try db.denyTransactions(&guard);
// Ordinary work still runs: only the transaction statements are refused.
try db.exec("INSERT INTO t (id) VALUES (1);");
try testing.expect(!guard.denied);
try testing.expectError(error.Auth, db.exec("COMMIT;"));
try testing.expect(guard.denied);
// The deny happens at prepare, so the transaction is still open.
try testing.expect(db.inTransaction());
// SAVEPOINT and its RELEASE report under a different action code, and they
// are the same bypass: at the outermost level they are a transaction under
// another name, and inside one they can still discard the migration's work.
guard.denied = false;
try testing.expectError(error.Auth, db.exec("SAVEPOINT half_a_migration;"));
try testing.expect(guard.denied);
guard.denied = false;
try testing.expectError(error.Auth, db.exec("RELEASE half_a_migration;"));
try testing.expect(guard.denied);
try testing.expect(db.inTransaction());
db.clearAuthorizer();
// The same connection is usable again — a leaked authorizer would strand it
// inside the transaction by denying this too.
try db.exec("ROLLBACK;");
try testing.expect(!db.inTransaction());
try testing.expectEqual(@as(i64, 0), try db.queryInt("SELECT count(*) FROM t"));
}
test "the exec fault seam fires once, on the statement it names" {
var db = try openMemory();
defer db.close();
try db.exec("CREATE TABLE t (id INTEGER PRIMARY KEY);");
exec_faults.failNextMatching("COMMIT");
defer exec_faults.disarm();
try db.exec("BEGIN IMMEDIATE;");
try db.exec("INSERT INTO t (id) VALUES (1);");
try testing.expectError(error.Internal, db.exec("COMMIT;"));
try testing.expect(!exec_faults.armed());
// Disarmed: the retry is a real COMMIT.
try db.exec("COMMIT;");
try testing.expectEqual(@as(i64, 1), try db.queryInt("SELECT count(*) FROM t"));
}
+269 -9
View File
@@ -33,11 +33,65 @@ pub fn classify(free_bytes: u64, cfg: model.Disk) State {
return .ok;
}
/// `min_free_mb` and `warn_free_mb` are one invariant pair — `classify` reads
/// both and reports the more severe verdict — so they live in one atomic word
/// and a reader unpacks a single load. Two atomics would let a sample land
/// between the two stores and classify against half of one configuration and
/// half of another.
fn packThresholds(d: model.Disk) u64 {
return (@as(u64, d.min_free_mb) << 32) | d.warn_free_mb;
}
fn unpackThresholds(bits: u64) model.Disk {
return .{
.min_free_mb = @truncate(bits >> 32),
.warn_free_mb = @truncate(bits),
};
}
/// Where the measured log directory comes from. `sample` borrows the path
/// across a directory scan, so the path cannot simply be replaced under it: a
/// reader pins a generation for the whole borrow and `setLogDir` retires the
/// old one, which is freed by whichever of the two — the last reader or the
/// setter — finds it retired with no refs.
pub const LogDirSource = union(enum) {
/// The path `init` was given, borrowed from the config. It outlives the
/// process, so a reader still holding it after a swap is safe and it needs
/// no pin. Null means logs do not go to a file.
boot: ?[:0]const u8,
/// Every generation `setLogDir` installs. Null means the same as above.
installed: ?*LogDir,
};
/// A reader's hold on the log directory for the length of one scan. `pinned`
/// is null for the boot source, which nothing frees.
pub const LogDirBorrow = struct {
path: ?[:0]const u8,
pinned: ?*LogDir,
};
pub const LogDir = struct {
path: [:0]const u8,
refs: u32 = 0,
retired: bool = false,
/// Non-null exactly for heap generations, and the allocator that frees
/// them.
gpa: ?std.mem.Allocator = null,
fn destroy(self: *LogDir) void {
const gpa = self.gpa orelse return;
gpa.free(self.path);
gpa.destroy(self);
}
};
pub const Monitor = struct {
cfg: model.Disk,
thresholds_packed: std.atomic.Value(u64),
data_dir: std.Io.Dir,
data_path: [:0]const u8,
log_dir_path: ?[:0]const u8,
/// Guards `log_dir` and every generation's `refs`/`retired`.
log_dir_mutex: std.Io.Mutex,
log_dir: LogDirSource,
state_raw: std.atomic.Value(u8),
free_bytes: std.atomic.Value(u64),
@@ -57,10 +111,11 @@ pub const Monitor = struct {
log_dir_path: ?[:0]const u8,
) Monitor {
return .{
.cfg = cfg,
.thresholds_packed = .init(packThresholds(cfg)),
.data_dir = data_dir,
.data_path = data_path,
.log_dir_path = log_dir_path,
.log_dir_mutex = .init,
.log_dir = .{ .boot = log_dir_path },
.state_raw = .init(@intFromEnum(State.ok)),
.free_bytes = .init(0),
.db_bytes = .init(0),
@@ -69,6 +124,90 @@ pub const Monitor = struct {
};
}
/// The live threshold pair, from one load: `warn >= min` holds for every
/// value this ever returns, whatever a concurrent `setThresholds` does.
pub fn thresholds(self: *const Monitor) model.Disk {
return unpackThresholds(self.thresholds_packed.load(.monotonic));
}
pub fn setThresholds(self: *Monitor, d: model.Disk) void {
self.thresholds_packed.store(packThresholds(d), .monotonic);
}
/// Pins the log directory for one scan. Every borrow is matched by a
/// `releaseLogDir`, which is what lets `setLogDir` free a generation the
/// moment no scan is reading its path.
pub fn acquireLogDir(self: *Monitor, io: std.Io) LogDirBorrow {
self.log_dir_mutex.lockUncancelable(io);
defer self.log_dir_mutex.unlock(io);
switch (self.log_dir) {
.boot => |path| return .{ .path = path, .pinned = null },
.installed => |maybe| {
const gen = maybe orelse return .{ .path = null, .pinned = null };
gen.refs += 1;
return .{ .path = gen.path, .pinned = gen };
},
}
}
pub fn releaseLogDir(self: *Monitor, io: std.Io, borrow: LogDirBorrow) void {
const gen = borrow.pinned orelse return;
self.log_dir_mutex.lockUncancelable(io);
std.debug.assert(gen.refs > 0);
gen.refs -= 1;
const free_it = gen.retired and gen.refs == 0;
self.log_dir_mutex.unlock(io);
if (free_it) gen.destroy();
}
/// Prepare half of a log-directory change: allocates the owned path and
/// its generation node before any commit, so publish cannot fail. `path`
/// null means logs no longer go to a file and nothing is measured.
pub fn prepareLogDir(
gpa: std.mem.Allocator,
path: ?[]const u8,
) std.mem.Allocator.Error!?*LogDir {
const p = path orelse return null;
const owned = try gpa.dupeZ(u8, p);
errdefer gpa.free(owned);
const gen = try gpa.create(LogDir);
gen.* = .{ .path = owned, .gpa = gpa };
return gen;
}
/// Discards a generation `prepareLogDir` built that will not be published.
pub fn destroyPreparedLogDir(prepared: ?*LogDir) void {
if (prepared) |gen| gen.destroy();
}
/// Publish half: infallible and I/O-free. The old generation is retired
/// and freed here when no scan holds it, or by the last release otherwise.
pub fn setLogDir(self: *Monitor, io: std.Io, prepared: ?*LogDir) void {
self.log_dir_mutex.lockUncancelable(io);
const old: ?*LogDir = switch (self.log_dir) {
.boot => null,
.installed => |maybe| maybe,
};
self.log_dir = .{ .installed = prepared };
var free_old = false;
if (old) |gen| {
gen.retired = true;
free_old = gen.refs == 0;
}
self.log_dir_mutex.unlock(io);
if (free_old) old.?.destroy();
}
/// Frees any installed log-directory generation. Every scan must have
/// released first, which shutdown ordering guarantees.
pub fn deinit(self: *Monitor, io: std.Io) void {
self.setLogDir(io, null);
}
pub fn state(self: *const Monitor) State {
return @enumFromInt(self.state_raw.load(.monotonic));
}
@@ -109,7 +248,9 @@ pub const Monitor = struct {
probeFailed(store, io, now_s, "data_dir", "sizing the data directory failed", err);
}
if (self.log_dir_path) |path| {
const borrow = self.acquireLogDir(io);
defer self.releaseLogDir(io, borrow);
if (borrow.path) |path| {
if (self.sumLogDir(io, path)) |bytes| {
self.log_bytes.store(bytes, .monotonic);
if (store) |s| s.resolve(io, now_s, .disk_probe, "log_dir");
@@ -120,7 +261,7 @@ pub const Monitor = struct {
}
}
self.publish(io, store, now_s, classify(free, self.cfg), free);
self.publish(io, store, now_s, classify(free, self.thresholds()), free);
}
/// Sample first, then sleep: a process that starts on a full disk must not
@@ -466,7 +607,7 @@ test "a threshold above the real free space drives the state to critical" {
try testing.expectEqual(State.critical, monitor.state());
try testing.expect(!monitor.writesAllowed());
monitor.cfg = .{ .min_free_mb = 0, .warn_free_mb = 0 };
monitor.setThresholds(.{ .min_free_mb = 0, .warn_free_mb = 0 });
monitor.sample(io, null, 0);
try testing.expectEqual(State.ok, monitor.state());
try testing.expect(monitor.writesAllowed());
@@ -509,7 +650,7 @@ test "a disk transition records an episode per severity and closes it on recover
monitor.sample(io, &fx.store, 1060);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
monitor.cfg = .{ .min_free_mb = 0, .warn_free_mb = 0 };
monitor.setThresholds(.{ .min_free_mb = 0, .warn_free_mb = 0 });
monitor.sample(io, &fx.store, 1120);
try testing.expectEqual(State.ok, monitor.state());
try testing.expectEqual(
@@ -551,7 +692,9 @@ test "a failed probe opens an episode the next clean pass closes" {
try tmp.dir.createDirPath(io, "logs");
var path_buf: [256]u8 = undefined;
monitor.log_dir_path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/logs", .{tmp.sub_path});
const good_dir = try std.fmt.bufPrint(&path_buf, ".zig-cache/tmp/{s}/logs", .{tmp.sub_path});
monitor.setLogDir(io, try Monitor.prepareLogDir(testing.allocator, good_dir));
defer monitor.deinit(io);
monitor.sample(io, &fx.store, 1100);
try testing.expectEqual(
@@ -560,6 +703,37 @@ test "a failed probe opens an episode the next clean pass closes" {
);
}
/// Alternates between two pairs that each satisfy `warn >= min`, so any
/// observed pair violating it can only have been torn out of two stores.
fn storeThresholdPairs(monitor: *Monitor, rounds: usize) void {
for (0..rounds) |i| {
monitor.setThresholds(if (i % 2 == 0)
.{ .min_free_mb = 1, .warn_free_mb = 2 }
else
.{ .min_free_mb = 3_000_000, .warn_free_mb = 4_000_000 });
}
}
test "a threshold reader never observes a pair from two different stores" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var monitor: Monitor = .init(.{ .min_free_mb = 1, .warn_free_mb = 2 }, std.Io.Dir.cwd(), ".", null);
const rounds = 20_000;
var writer = try io.concurrent(storeThresholdPairs, .{ &monitor, rounds });
// Recorded, not asserted, while the writer runs: an assertion that returned
// here would leave `Threaded.deinit` joining a task nothing ends.
var torn = false;
for (0..rounds) |_| {
const pair = monitor.thresholds();
if (pair.warn_free_mb < pair.min_free_mb) torn = true;
}
writer.await(io);
try testing.expect(!torn);
}
test "every emit site is inert when the store is absent" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -574,3 +748,89 @@ test "every emit site is inert when the store is absent" {
monitor.sample(io, null, 0);
try testing.expectEqual(@as(u64, 1), monitor.sample_failures.load(.monotonic));
}
// ---------------------------------------------------------------------------
// setLogDir (milestone-34 S3.5)
// ---------------------------------------------------------------------------
test "setLogDir re-points the measurement and frees the retired generation" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
try tmp.dir.createDirPath(io, "first");
try tmp.dir.createDirPath(io, "second");
try tmp.dir.writeFile(io, .{ .sub_path = "first/nxdns.log", .data = "aaaa" });
try tmp.dir.writeFile(io, .{ .sub_path = "second/nxdns.log", .data = "bbbbbbbb" });
var first_buf: [160]u8 = undefined;
var second_buf: [160]u8 = undefined;
const first = try std.fmt.bufPrint(&first_buf, ".zig-cache/tmp/{s}/first", .{tmp.sub_path});
const second = try std.fmt.bufPrint(&second_buf, ".zig-cache/tmp/{s}/second", .{tmp.sub_path});
var monitor: Monitor = .init(.{ .min_free_mb = 0, .warn_free_mb = 0 }, tmp.dir, ".", null);
defer monitor.deinit(io);
// Boot measures nothing.
monitor.sample(io, null, 1_000);
try testing.expectEqual(@as(u64, 0), monitor.gauges().log_bytes);
monitor.setLogDir(io, try Monitor.prepareLogDir(testing.allocator, first));
monitor.sample(io, null, 1_100);
try testing.expectEqual(@as(u64, 4), monitor.gauges().log_bytes);
// The retired generation is freed here; the testing allocator says so.
monitor.setLogDir(io, try Monitor.prepareLogDir(testing.allocator, second));
monitor.sample(io, null, 1_200);
try testing.expectEqual(@as(u64, 8), monitor.gauges().log_bytes);
// Output moved away from file: nothing is measured, and the gauge keeps
// its last reading rather than claiming zero bytes of logs.
monitor.setLogDir(io, null);
monitor.sample(io, null, 1_300);
try testing.expectEqual(@as(u64, 8), monitor.gauges().log_bytes);
}
test "a prepared log directory that is never published is freed by the caller" {
const prepared = try Monitor.prepareLogDir(testing.allocator, "/var/log/nxdns");
Monitor.destroyPreparedLogDir(prepared);
try testing.expectEqual(@as(?*LogDir, null), try Monitor.prepareLogDir(testing.allocator, null));
}
test "a sample borrowing a log directory survives a concurrent setLogDir" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
try tmp.dir.createDirPath(io, "logs");
try tmp.dir.writeFile(io, .{ .sub_path = "logs/nxdns.log", .data = "aaaa" });
var path_buf: [160]u8 = undefined;
const logs = try std.fmt.bufPrint(&path_buf, ".zig-cache/tmp/{s}/logs", .{tmp.sub_path});
var monitor: Monitor = .init(.{ .min_free_mb = 0, .warn_free_mb = 0 }, tmp.dir, ".", null);
defer monitor.deinit(io);
monitor.setLogDir(io, try Monitor.prepareLogDir(testing.allocator, logs));
const Racer = struct {
fn sample(m: *Monitor, sio: std.Io) void {
for (0..200) |i| m.sample(sio, null, @intCast(1_000 + i));
}
fn repoint(m: *Monitor, sio: std.Io, p: []const u8) void {
for (0..200) |i| {
const prepared = Monitor.prepareLogDir(testing.allocator, if (i % 2 == 0) p else null) catch return;
m.setLogDir(sio, prepared);
}
}
};
var group: std.Io.Group = .init;
defer group.cancel(io);
try group.concurrent(io, Racer.sample, .{ &monitor, io });
try group.concurrent(io, Racer.repoint, .{ &monitor, io, logs });
try group.await(io);
}
+214 -16
View File
@@ -30,6 +30,7 @@
//! `writer_failed`, so the loss is visible rather than silent.
const std = @import("std");
const Allocator = std.mem.Allocator;
const builtin = @import("builtin");
const db = @import("db.zig");
@@ -399,8 +400,23 @@ const discard_stall = if (builtin.is_test) struct {
}
};
/// The §11.4 privacy policy: one value, never two. A producer decides the
/// domain fields and the client field from ONE load of `privacy_packed`, so no
/// entry can leave `transformed` with the domain redacted and the client
/// exposed, or the reverse, because a `setPrivacy` landed between the two
/// decisions.
pub const Privacy = packed struct(u8) {
hide_domains: bool = false,
hide_client_ips: bool = false,
_reserved: u6 = 0,
};
pub const Logger = struct {
cfg: model.Logging,
/// Both privacy flags in one atomic byte, loaded once per entry.
privacy_packed: std.atomic.Value(u8),
/// Independent of the privacy policy: it governs when a batch commits, not
/// what a row contains, so nothing pairs the two.
flush_interval_s: std.atomic.Value(u16),
queue: EntryQueue,
queries_dropped: std.atomic.Value(u64),
/// When the newest drop happened, in unix seconds; 0 means none yet. Read
@@ -434,7 +450,11 @@ pub const Logger = struct {
/// touched it.
pub fn init(cfg: model.Logging, queue_buf: []Entry) Logger {
return .{
.cfg = cfg,
.privacy_packed = .init(@bitCast(Privacy{
.hide_domains = cfg.hide_domains,
.hide_client_ips = cfg.hide_client_ips,
})),
.flush_interval_s = .init(cfg.query_log_flush_interval_s),
.queue = .init(queue_buf),
.queries_dropped = .init(0),
.last_drop_s = .init(0),
@@ -464,8 +484,9 @@ pub const Logger = struct {
/// configuration labels the operator wrote, identical on every row that
/// hits them, and they say nothing about which name a client looked up.
pub fn transformed(self: *const Logger, entry: Entry) Entry {
const policy = self.privacy();
var out = entry;
if (self.cfg.hide_domains) {
if (policy.hide_domains) {
out.setDomain(hidden_marker);
// Only where there is something to hide: an empty field means the
// query had no such value, and writing a marker would claim it did.
@@ -473,10 +494,24 @@ pub const Logger = struct {
if (out.cname_len != 0) out.setCnameTarget(hidden_marker);
if (out.safe_search_len != 0) out.setSafeSearchTarget(hidden_marker);
}
if (self.cfg.hide_client_ips) out.setClientIp(hidden_marker);
if (policy.hide_client_ips) out.setClientIp(hidden_marker);
return out;
}
/// The live policy, from one load. Every producer decision about one entry
/// must come from a single call to this.
pub fn privacy(self: *const Logger) Privacy {
return @bitCast(self.privacy_packed.load(.monotonic));
}
pub fn setPrivacy(self: *Logger, p: Privacy) void {
self.privacy_packed.store(@bitCast(p), .monotonic);
}
pub fn setFlushInterval(self: *Logger, seconds: u16) void {
self.flush_interval_s.store(seconds, .monotonic);
}
/// `log` without the transforms, for a caller that already applied them.
pub fn logTransformed(self: *Logger, io: std.Io, entry: Entry) void {
self.enqueue(io, entry);
@@ -521,13 +556,17 @@ pub const Logger = struct {
/// group it cancels for exactly that reason.
///
/// `monitor` is the §11.6 gate. Null disables gating.
///
/// `gpa` belongs to the `BatchWriter` for that writer's whole life; it
/// allocates the projection deltas of one batch and nothing else.
pub fn runWriter(
self: *Logger,
io: std.Io,
gpa: Allocator,
database: *db.Db,
monitor: ?*disk_monitor.Monitor,
) std.Io.Cancelable!void {
var writer = queries_repo.BatchWriter.init(database) catch |err| {
var writer = queries_repo.BatchWriter.init(gpa, database) catch |err| {
scope.warn("query logger: preparing the batch statements failed: {s}", .{@errorName(err)});
// Without a writer there is no consumer, so leaving the queue open
// would silently swallow every later entry.
@@ -540,7 +579,22 @@ pub const Logger = struct {
return;
};
defer writer.deinit();
return self.runPrepared(io, &writer, monitor);
}
/// The writer loop over statements someone else prepared.
///
/// `runWriter` prepares and then calls this. A logger generation created by
/// a resize prepares separately, before anything is published, so that a
/// statement failure is refused at prepare time instead of silently killing
/// the writer of a queue producers are already filling
/// (`logger_controller.zig`).
pub fn runPrepared(
self: *Logger,
io: std.Io,
writer: *queries_repo.BatchWriter,
monitor: ?*disk_monitor.Monitor,
) std.Io.Cancelable!void {
var batch: [flush_batch]Entry = undefined;
while (true) {
// A closed queue hands over its buffered elements before it reports
@@ -564,7 +618,7 @@ pub const Logger = struct {
self.countDropped(io, n, at);
return err;
};
self.flush(io, &writer, batch[0..n], monitor) catch |err| switch (err) {
self.flush(io, writer, batch[0..n], monitor) catch |err| switch (err) {
error.Canceled => |e| {
self.countDropped(io, n, at);
return e;
@@ -606,7 +660,7 @@ pub const Logger = struct {
/// returns without waiting for anything.
fn flushDeadline(self: *const Logger, io: std.Io) std.Io.Clock.Timestamp {
return .fromNow(io, .{
.raw = .fromSeconds(self.cfg.query_log_flush_interval_s),
.raw = .fromSeconds(self.flush_interval_s.load(.monotonic)),
.clock = .boot,
});
}
@@ -644,6 +698,18 @@ pub const Logger = struct {
self.draining.store(true, .release);
}
/// `shutdown` without `draining`: this generation is being replaced, not
/// the process stopped.
///
/// The flag is what turns a gate-held batch into a counted loss
/// (`flush`'s `GatedAtShutdown`), and a retired writer must not take that
/// path — the disk can still recover, and the rows it is holding are still
/// going to be written when it does. Every producer of this generation
/// must have released it before the close, exactly as at shutdown.
pub fn retire(self: *Logger, io: std.Io) void {
self.queue.close(io);
}
/// Fills `batch` behind the entry already in slot 0, until it is full or
/// `deadline` passes. `n` counts the slots that hold an entry, and stays
/// accurate on the cancellation path so the caller can count what is lost.
@@ -1073,7 +1139,7 @@ test "an entry with every provenance field set survives the queue, toRow, insert
var database = try openLog();
defer database.close();
var writer = try queries_repo.BatchWriter.init(&database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -1240,6 +1306,80 @@ test "log hides only the field its switch names" {
try testing.expectEqualStrings("192.0.2.10", untouched.clientIp());
}
/// The rendezvous that forces the flip to land BETWEEN two producer entries
/// rather than whenever the scheduler feels like it.
const PrivacyFlip = struct {
logger: *Logger,
/// Set by the producer once its pre-flip entry is enqueued.
before_done: std.Io.Event = .unset,
/// Set by the flipper once `setPrivacy` has returned.
flipped: std.Io.Event = .unset,
fn run(self: *PrivacyFlip, io: std.Io) void {
self.before_done.wait(io) catch return;
self.logger.setPrivacy(.{ .hide_domains = true, .hide_client_ips = true });
self.flipped.set(io);
}
};
test "a privacy flip redacts every entry after it and none before it" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var buf: [4]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
var flip: PrivacyFlip = .{ .logger = &logger };
var future = try io.concurrent(PrivacyFlip.run, .{ &flip, io });
logger.log(io, sampleEntry(1, "before.example"));
flip.before_done.set(io);
try flip.flipped.wait(io);
logger.log(io, sampleEntry(2, "after.example"));
future.await(io);
const before = try logger.queue.getOne(io);
try testing.expectEqualStrings("before.example", before.domain());
try testing.expectEqualStrings("192.0.2.10", before.clientIp());
const after = try logger.queue.getOne(io);
try testing.expectEqualStrings(hidden_marker, after.domain());
try testing.expectEqualStrings(hidden_marker, after.clientIp());
}
fn flipPrivacyRepeatedly(logger: *Logger, rounds: usize) void {
for (0..rounds) |i| {
logger.setPrivacy(if (i % 2 == 0)
.{}
else
.{ .hide_domains = true, .hide_client_ips = true });
}
}
test "no producer observes a privacy policy that redacts one field and not the other" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var buf: [4]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
const source = sampleEntry(1, "tracker.example");
const rounds = 20_000;
var flipper = try io.concurrent(flipPrivacyRepeatedly, .{ &logger, rounds });
// Recorded, not asserted, while the flipper runs: an assertion that
// returned here would leave `Threaded.deinit` joining a task nothing ends.
var mixed = false;
for (0..rounds) |_| {
const out = logger.transformed(source);
const domain_hidden = std.mem.eql(u8, out.domain(), hidden_marker);
const client_hidden = std.mem.eql(u8, out.clientIp(), hidden_marker);
if (domain_hidden != client_hidden) mixed = true;
}
flipper.await(io);
try testing.expect(!mixed);
}
test "the split halves reproduce log byte for byte" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -1331,6 +1471,7 @@ test "shutdown writes the batch the writer holds and the rest of the queue" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
@@ -1367,7 +1508,7 @@ test "entries that arrive inside one window reach the database in one batch" {
var database = try openLog();
defer database.close();
var writer = try queries_repo.BatchWriter.init(&database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [16]Entry = undefined;
@@ -1424,6 +1565,7 @@ test "the writer holds an entry for the length of the flush interval" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
@@ -1476,6 +1618,7 @@ test "a full batch flushes without waiting for the interval" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
@@ -1507,6 +1650,58 @@ test "a full batch flushes without waiting for the interval" {
try testing.expectEqual(@as(i64, 150), try queries_repo.countRows(&database));
}
test "the writer's next cycle uses the interval set since its last one" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var buf: [8]Entry = undefined;
// Zero: every cycle commits what it has and goes straight back to `getOne`,
// so the first entry proves the writer is running and parked between cycles.
var logger: Logger = .init(.{ .query_log_flush_interval_s = 0 }, &buf);
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
// See the note in "shutdown writes the batch the writer holds": this must
// run before the deferred `database.close`.
defer {
logger.shutdown(io);
future.await(io) catch {};
}
logger.log(io, sampleEntry(1, "first.example"));
const poll: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(5), .clock = .awake };
var waited: usize = 0;
while (logger.rows_written.load(.monotonic) == 0 and waited < 400) : (waited += 1) {
try poll.sleep(io);
}
// An hour, installed while the writer is parked: the cycle the next entry
// starts must wait it out instead of committing at once.
logger.setFlushInterval(3600);
logger.log(io, sampleEntry(2, "second.example"));
const quarter: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(250), .clock = .awake };
try quarter.sleep(io);
const written_under_the_new_interval = logger.rows_written.load(.monotonic);
logger.shutdown(io);
try future.await(io);
try testing.expect(waited < 400);
// The first entry, and only the first: the second is still held.
try testing.expectEqual(@as(u64, 1), written_under_the_new_interval);
// The close releases it, which is what makes the hold a hold and not a loss.
try testing.expectEqual(@as(i64, 2), try queries_repo.countRows(&database));
}
test "a gated flush holds the batch until the disk recovers" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -1514,7 +1709,7 @@ test "a gated flush holds the batch until the disk recovers" {
var database = try openLog();
defer database.close();
var writer = try queries_repo.BatchWriter.init(&database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -1568,7 +1763,7 @@ test "a failing batch is dropped whole and the writer stays usable" {
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
var writer = try queries_repo.BatchWriter.init(&database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -1603,7 +1798,7 @@ test "a writer that cannot prepare closes the queue and counts every entry" {
for (0..3) |i| logger.log(io, sampleEntry(@intCast(i), "early.example"));
try logger.runWriter(io, &database, null);
try logger.runWriter(io, testing.allocator, &database, null);
try testing.expect(logger.writer_failed.load(.acquire));
try testing.expectEqual(@as(u64, 3), logger.queries_dropped.load(.monotonic));
@@ -1656,6 +1851,7 @@ test "the gating episode opens on the gate, turns losing on a drop, and clears o
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
testing.allocator,
&database,
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -1837,6 +2033,7 @@ test "a canceled writer counts the batch it was holding" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
testing.allocator,
&database,
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -1886,6 +2083,7 @@ test "a disk-gated writer drops what it holds at shutdown instead of hanging" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
testing.allocator,
&database,
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -1935,7 +2133,7 @@ test "an empty batch touches neither the database nor the counters" {
var database = try openLog();
defer database.close();
var writer = try queries_repo.BatchWriter.init(&database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -1969,7 +2167,7 @@ test "a dropped batch opens an error episode the next good batch closes" {
try fx.init(io, 1000);
defer fx.deinit();
var writer = try queries_repo.BatchWriter.init(&database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -2012,7 +2210,7 @@ test "a writer that cannot prepare leaves an episode no recovery path claims" {
var logger: Logger = .init(.{}, &buf);
logger.diagnostics = &fx.store;
try logger.runWriter(io, &database, null);
try logger.runWriter(io, testing.allocator, &database, null);
try testing.expectEqualStrings("writer", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
@@ -2023,7 +2221,7 @@ test "a writer that cannot prepare leaves an episode no recovery path claims" {
// The writer returned, so nothing can ever close this. A second run finds
// the queue closed and adds no second episode.
try logger.runWriter(io, &database, null);
try logger.runWriter(io, testing.allocator, &database, null);
try testing.expectEqual(
@as(i64, 1),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
File diff suppressed because it is too large Load Diff
+11 -4
View File
@@ -27,6 +27,7 @@ const disk_monitor = @import("disk_monitor.zig");
const logger = @import("logger.zig");
const queries_repo = @import("repositories/queries_repo.zig");
const querylog_schema = @import("querylog_schema.zig");
const querylog_versions = @import("querylog_versions.zig");
const retention = @import("retention.zig");
const testing = std.testing;
@@ -144,7 +145,7 @@ fn awaitCount(counter: *const std.atomic.Value(u64), target: u64, limit: usize)
}
fn writeRows(database: *db.Db, timestamps: []const i64, domain: []const u8) !void {
var writer = try queries_repo.BatchWriter.init(database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, database);
defer writer.deinit();
var rows: [16]queries_repo.Row = undefined;
@@ -208,6 +209,7 @@ test "S8 case 1: the logger writes a real querylog.db end to end" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, null),
});
@@ -232,7 +234,7 @@ test "S8 case 1: the logger writes a real querylog.db end to end" {
try testing.expectEqual(@as(i64, 250), try queries_repo.countRows(log_db.database()));
try testing.expectEqual(@as(i64, 10), try queries_repo.countDomains(log_db.database()));
try testing.expectEqual(
@as(i64, querylog_schema.fingerprint),
@as(i64, querylog_versions.current_version),
try log_db.database().queryInt("PRAGMA user_version"),
);
}
@@ -256,6 +258,7 @@ test "S8 case 2: a single entry reaches the file once the flush interval passes"
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, null),
});
@@ -311,6 +314,7 @@ test "S8 case 3: a full queue drops the oldest entries and the newest survive" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -359,6 +363,7 @@ test "S8 case 4: the privacy transforms reach the stored rows" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, null),
});
@@ -406,7 +411,8 @@ test "S8 case 5: a retention pass prunes the old rows and truncates the write-ah
try testing.expectEqual(@as(i64, 4), try queries_repo.countRows(log_db.database()));
try testing.expect(try f.sizeOf("querylog.db-wal") > 0);
var pass: retention.Retention = .init(.{ .retention_days = 30 });
var pass_days: retention.RetentionDays = .init(30);
var pass: retention.Retention = .init(&pass_days);
pass.runOnce(io, log_db.database(), null, null);
try testing.expectEqual(@as(u64, 1), pass.snapshotStats().passes);
@@ -458,6 +464,7 @@ test "S8 case 6: a critical disk gates the flushes and recovery releases them" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -473,7 +480,7 @@ test "S8 case 6: a critical disk gates the flushes and recovery releases them" {
try testing.expectEqual(@as(u64, 0), query_log.rows_written.load(.monotonic));
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(log_db.database()));
monitor.cfg = .{ .min_free_mb = 0, .warn_free_mb = 0 };
monitor.setThresholds(.{ .min_free_mb = 0, .warn_free_mb = 0 });
monitor.sample(io, null, 0);
try testing.expectEqual(disk_monitor.State.ok, monitor.state());
try testing.expect(monitor.writesAllowed());
+535
View File
@@ -0,0 +1,535 @@
//! The shipped `querylog.db` fixtures, and the proof that every supported
//! schema version reaches the current one with the operator's rows intact.
//!
//! A file of its own, not a section of `querylog_migrations.zig`, because of the
//! link contract that split `querylog_versions.zig` out in the first place. The
//! assertions here need `repositories/queries_repo.zig`'s projection-coherence
//! oracle, and that file reaches across `src/` for the config and filter types;
//! importing it from `querylog_migrations.zig` would pull all of it into the
//! module `tools/cut.zig` builds `querylog_schema.zig` as, where those paths lie
//! outside the module root and do not compile.
//!
//! The fixtures themselves — `testdata/querylog-v<N>-{schema,data}.sql` — are
//! immutable once released. Every version in `[minimum_supported_version,
//! current_version]` has a pair: the current version's pair is the next
//! migration's starting point, and an explicit break ships the new baseline.
const std = @import("std");
const db = @import("db.zig");
const migrations = @import("querylog_migrations.zig");
const queries_repo = @import("repositories/queries_repo.zig");
const querylog_schema = @import("querylog_schema.zig");
const versions = @import("querylog_versions.zig");
const testing = std.testing;
/// A temporary directory and the `querylog.db` path inside it. Deliberately not
/// `querylog_migrations.zig`'s test harness: that one builds synthetic schemas,
/// while everything here starts from the shipped fixture files.
const Harness = struct {
threaded: std.Io.Threaded,
tmp: std.testing.TmpDir,
buf: [256]u8 = undefined,
fn init() Harness {
return .{
.threaded = .init(testing.allocator, .{}),
.tmp = testing.tmpDir(.{ .iterate = true }),
};
}
fn deinit(self: *Harness) void {
self.tmp.cleanup();
self.threaded.deinit();
}
fn io(self: *Harness) std.Io {
return self.threaded.io();
}
fn path(self: *Harness) [:0]const u8 {
return std.fmt.bufPrintZ(&self.buf, ".zig-cache/tmp/{s}/querylog.db", .{self.tmp.sub_path}) catch
unreachable;
}
fn openLive(self: *Harness) !db.Db {
var database = try db.Db.open(self.path(), .{ .mode = .read_write_existing });
errdefer database.close();
try db.applyPragmas(&database, .{});
return database;
}
};
/// One shipped schema version's frozen pair. Both halves are immutable once
/// released — the release cut byte-compares them against the previous tag — and
/// every version in `[minimum_supported_version, current_version]` must have a
/// pair, which the cut gate also enforces.
const Fixture = struct {
version: i32,
schema: [:0]const u8,
data: [:0]const u8,
};
const fixtures = [_]Fixture{
.{
.version = 1,
.schema = @embedFile("testdata/querylog-v1-schema.sql"),
.data = @embedFile("testdata/querylog-v1-data.sql"),
},
};
fn fixtureFor(version: i32) ?Fixture {
for (fixtures) |fixture| {
if (fixture.version == version) return fixture;
}
return null;
}
/// Writes a fixture pair to `path` and stamps it. `stamp` is a parameter rather
/// than `fixture.version` because the legacy-fingerprint file is the same
/// version-1 bytes under a different stamp.
fn writeFixture(path: [:0]const u8, fixture: Fixture, stamp: i32) !void {
var database = try db.Db.open(path, .{ .mode = .read_write_create });
defer database.close();
try db.applyPragmas(&database, .{});
try database.exec(fixture.schema);
try database.exec(fixture.data);
var buf: [64]u8 = undefined;
const sql = std.fmt.bufPrintZ(&buf, "PRAGMA user_version = {d};", .{stamp}) catch unreachable;
try database.exec(sql);
}
/// A second path in the harness's directory, for the reference databases the
/// assertions below compare against.
fn sidePath(h: *Harness, buf: []u8, name: []const u8) [:0]const u8 {
return std.fmt.bufPrintZ(buf, ".zig-cache/tmp/{s}/{s}", .{ h.tmp.sub_path, name }) catch unreachable;
}
/// A database holding nothing but the current `ddl`, which is what a file
/// created by this build is.
fn openFreshCurrent(h: *Harness, buf: []u8) !db.Db {
var database = try db.Db.open(sidePath(h, buf, "fresh.db"), .{ .mode = .read_write_create });
errdefer database.close();
try db.applyPragmas(&database, .{});
try database.exec(querylog_schema.ddl);
return database;
}
/// An untouched load of `fixture`, to compare a migrated or opened file against
/// rather than restating the fixture's contents in the assertions.
fn openPristine(h: *Harness, buf: []u8, fixture: Fixture) !db.Db {
const path = sidePath(h, buf, "pristine.db");
try writeFixture(path, fixture, fixture.version);
var database = try db.Db.open(path, .{ .mode = .read_write_existing });
errdefer database.close();
try db.applyPragmas(&database, .{});
return database;
}
/// Every row of every table, as one canonical text. Exact equality is only the
/// right question for a file no migration has reshaped; a migrated file is
/// checked by the counts and the watermark instead.
fn dumpContent(gpa: std.mem.Allocator, database: *db.Db, out: *std.ArrayList(u8)) !void {
var tables: std.ArrayList([]u8) = .empty;
defer freeOwned(gpa, &tables);
{
var stmt = try database.prepare(
\\SELECT name FROM sqlite_schema
\\WHERE type = 'table' AND name NOT LIKE 'sqlite\_%' ESCAPE '\'
\\ORDER BY name
);
defer stmt.deinit();
// Copied out before the per-table statements step: a borrowed
// `columnText` would not survive them.
while (try stmt.step()) try tables.append(gpa, try stmt.columnTextAlloc(gpa, 0));
}
// The lines are sorted here rather than by the query: the four `bucket_*`
// tables are WITHOUT ROWID, so `ORDER BY rowid` is not available to all of
// them and no single column list is.
var lines: std.ArrayList([]u8) = .empty;
defer freeOwned(gpa, &lines);
for (tables.items) |table| {
var sql_buf: [256]u8 = undefined;
const sql = std.fmt.bufPrint(&sql_buf, "SELECT * FROM \"{s}\"", .{table}) catch unreachable;
var stmt = try database.prepare(sql);
defer stmt.deinit();
while (try stmt.step()) {
var line: std.ArrayList(u8) = .empty;
errdefer line.deinit(gpa);
try line.print(gpa, "R|{s}", .{table});
var col: c_int = 0;
const columns: c_int = db.c.sqlite3_column_count(stmt.handle);
while (col < columns) : (col += 1) {
try line.print(gpa, "|{s}", .{stmt.columnTextOrNull(col) orelse "<null>"});
}
try lines.append(gpa, try line.toOwnedSlice(gpa));
}
}
std.mem.sortUnstable([]u8, lines.items, {}, struct {
fn lessThan(_: void, a: []u8, b: []u8) bool {
return std.mem.lessThan(u8, a, b);
}
}.lessThan);
for (lines.items) |line| {
try out.appendSlice(gpa, line);
try out.append(gpa, '\n');
}
}
/// One value, encoded so that no two different values can produce the same
/// bytes: a type tag, the byte length, then the bytes themselves.
///
/// Nothing here is a sentinel and nothing is escaped, which is the point. A
/// serialization that wrote NULL as `<null>` cannot tell a NULL apart from the
/// six-character string of the same name, and one that separated values with
/// `|` cannot tell `a|b` in one column from `a` and `b` in two — so a migration
/// that turned a NULL `upstream` into text, or shifted a value from one column
/// into its neighbour, would compare EQUAL to the original. The length prefix
/// makes the stream uniquely decodable, so equal encodings mean equal rows.
///
/// A float travels as its bit pattern rather than as printed digits: the
/// question here is whether the value survived, not whether it rounds the same.
fn writeValue(gpa: std.mem.Allocator, stmt: *db.Stmt, col: c_int, out: *std.ArrayList(u8)) !void {
switch (db.c.sqlite3_column_type(stmt.handle, col)) {
db.column_type.null_value => try out.print(gpa, "n0:", .{}),
db.column_type.integer => {
var buf: [24]u8 = undefined;
const text = std.fmt.bufPrint(&buf, "{d}", .{stmt.columnInt(col)}) catch unreachable;
try out.print(gpa, "i{d}:{s}", .{ text.len, text });
},
db.column_type.float => {
const bits: u64 = @bitCast(db.c.sqlite3_column_double(stmt.handle, col));
var buf: [24]u8 = undefined;
const text = std.fmt.bufPrint(&buf, "{d}", .{bits}) catch unreachable;
try out.print(gpa, "f{d}:{s}", .{ text.len, text });
},
db.column_type.blob => {
// `columnText` on a blob hands back the same bytes SQLite stores,
// which is what this compares; it is not read as text.
const bytes = stmt.columnText(col);
try out.print(gpa, "b{d}:{s}", .{ bytes.len, bytes });
},
else => {
const bytes = stmt.columnText(col);
try out.print(gpa, "t{d}:{s}", .{ bytes.len, bytes });
},
}
}
/// One query's rows, in the order the query returns them, tagged with `label` so
/// that a difference names the relation it came from. The column count is part
/// of each row for the same reason the lengths are part of each value.
fn dumpQuery(
gpa: std.mem.Allocator,
database: *db.Db,
label: []const u8,
sql: [:0]const u8,
out: *std.ArrayList(u8),
) !void {
var stmt = try database.prepare(sql);
defer stmt.deinit();
while (try stmt.step()) {
const columns: c_int = db.c.sqlite3_column_count(stmt.handle);
try out.print(gpa, "{s}:{d}:", .{ label, columns });
var col: c_int = 0;
while (col < columns) : (col += 1) {
try writeValue(gpa, &stmt, col, out);
}
try out.append(gpa, '\n');
}
}
/// The operator's data as the application means it, in an order a migration
/// cannot permute.
///
/// `q.*` rather than a column list on purpose: a migration that adds a column
/// must show that column here, and a hand-written list would quietly stop
/// covering the table the day it grows. `d.domain` rides along so that the text
/// a row names is compared, not only the id it happens to hold.
const logical_relations = [_]struct { label: []const u8, sql: [:0]const u8 }{
.{
.label = "query_log",
.sql =
\\SELECT q.*, d.domain FROM query_log q
\\JOIN domains d ON d.id = q.domain_id
\\ORDER BY q.id
,
},
.{ .label = "domains", .sql = "SELECT * FROM domains ORDER BY id" },
.{ .label = "querylog_meta", .sql = "SELECT * FROM querylog_meta ORDER BY id" },
};
fn dumpLogical(gpa: std.mem.Allocator, database: *db.Db, out: *std.ArrayList(u8)) !void {
for (logical_relations) |relation| {
try dumpQuery(gpa, database, relation.label, relation.sql, out);
}
}
fn freeOwned(gpa: std.mem.Allocator, list: *std.ArrayList([]u8)) void {
for (list.items) |item| gpa.free(item);
list.deinit(gpa);
}
fn expectSameContent(a: *db.Db, b: *db.Db) !void {
var text_a: std.ArrayList(u8) = .empty;
defer text_a.deinit(testing.allocator);
var text_b: std.ArrayList(u8) = .empty;
defer text_b.deinit(testing.allocator);
try dumpContent(testing.allocator, a, &text_a);
try dumpContent(testing.allocator, b, &text_b);
try testing.expectEqualStrings(text_a.items, text_b.items);
}
fn rowCount(database: *db.Db, table: []const u8) !i64 {
var buf: [128]u8 = undefined;
return database.queryInt(std.fmt.bufPrint(&buf, "SELECT count(*) FROM \"{s}\"", .{table}) catch unreachable);
}
/// What must hold after a fixture has come through `openVersioned`, whatever
/// path it took: every row the operator had is still there with the same
/// CONTENT, and the projections still agree with the raw rows.
///
/// Counting rows and checking one watermark is what this used to do, and a
/// migration that shifted a timestamp, dropped a `qtype` or crossed two rows'
/// `client_ip` values passed it. The comparison is therefore the full logical
/// content of the three operator tables, `available_since` included as one
/// column of `querylog_meta` among the rest. The counts stay because a count
/// difference is the failure worth naming plainly.
///
/// The relations are named rather than derived because they are the ones holding
/// operator data; the `bucket_*` projections are derived from them, which
/// `expectProjectionsMatchRecompute` is the right check for. A future migration
/// that renames a table amends this alongside the step that does it. So does one
/// that renumbers `id` values: this asserts they survive, which every rebuild
/// written to the A.2 rule does.
fn expectFixtureSurvived(opened: *db.Db, pristine: *db.Db) !void {
for ([_][]const u8{ "query_log", "domains", "querylog_meta" }) |table| {
try testing.expectEqual(try rowCount(pristine, table), try rowCount(opened, table));
}
var opened_text: std.ArrayList(u8) = .empty;
defer opened_text.deinit(testing.allocator);
var pristine_text: std.ArrayList(u8) = .empty;
defer pristine_text.deinit(testing.allocator);
try dumpLogical(testing.allocator, opened, &opened_text);
try dumpLogical(testing.allocator, pristine, &pristine_text);
try testing.expectEqualStrings(pristine_text.items, opened_text.items);
try queries_repo.expectProjectionsMatchRecompute(opened);
}
test "the survival comparison tells a NULL apart from text that looks like one" {
// The mutation a count-and-watermark check misses entirely, and a
// sentinel-string serialization misses just as completely: one column of one
// row stops being NULL and becomes the very text the sentinel used. Every
// count, the watermark and the projections all still agree.
const fixture = fixtureFor(1) orelse return error.MissingFixture;
var h: Harness = .init();
defer h.deinit();
try writeFixture(h.path(), fixture, fixture.version);
var side: [256]u8 = undefined;
var pristine = try openPristine(&h, &side, fixture);
defer pristine.close();
var corrupted = try h.openLive();
defer corrupted.close();
try testing.expectEqual(@as(i64, 0), corrupted.changes());
try corrupted.exec(
\\UPDATE query_log SET upstream = '<null>'
\\WHERE id = (SELECT min(id) FROM query_log WHERE upstream IS NULL)
);
// The fixture has to actually carry a NULL `upstream` for this to be a test
// of anything.
try testing.expectEqual(@as(i64, 1), corrupted.changes());
// Everything the old check looked at still agrees, which is why it passed.
for ([_][]const u8{ "query_log", "domains", "querylog_meta" }) |table| {
try testing.expectEqual(try rowCount(&pristine, table), try rowCount(&corrupted, table));
}
try testing.expectEqual(
try pristine.queryInt("SELECT available_since FROM querylog_meta WHERE id = 1"),
try corrupted.queryInt("SELECT available_since FROM querylog_meta WHERE id = 1"),
);
try queries_repo.expectProjectionsMatchRecompute(&corrupted);
// The dumps are compared here rather than through `expectFixtureSurvived`
// so that a PASSING run stays silent: `expectEqualStrings` prints the whole
// diff before it returns its error, and this is the comparison that function
// makes.
var corrupted_text: std.ArrayList(u8) = .empty;
defer corrupted_text.deinit(testing.allocator);
var pristine_text: std.ArrayList(u8) = .empty;
defer pristine_text.deinit(testing.allocator);
try dumpLogical(testing.allocator, &corrupted, &corrupted_text);
try dumpLogical(testing.allocator, &pristine, &pristine_text);
try testing.expect(!std.mem.eql(u8, corrupted_text.items, pristine_text.items));
}
test "every supported version ships a fixture pair" {
// The cut gate enforces this against a release; the suite enforces it
// against a commit, so a version bump that forgot its fixtures fails here
// long before anyone reaches for `zig build cut`.
var version = versions.minimum_supported_version;
while (version <= versions.current_version) : (version += 1) {
try testing.expect(fixtureFor(version) != null);
}
}
test "each shipped fixture pair is coherent before any migration touches it" {
for (fixtures) |fixture| {
var h: Harness = .init();
defer h.deinit();
try writeFixture(h.path(), fixture, fixture.version);
var database = try h.openLive();
defer database.close();
try testing.expectEqual(
@as(i64, fixture.version),
try database.queryInt("PRAGMA user_version"),
);
// An incoherent fixture has to fail as a fixture, not later as a
// migration that appears to have corrupted the projections.
try queries_repo.expectProjectionsMatchRecompute(&database);
try testing.expectEqual(@as(i64, 0), try database.queryInt("SELECT count(*) FROM pragma_foreign_key_check"));
}
}
test "every fixture below the current version migrates to it through the shipped chain" {
// Empty while the chain is: `minimum_supported_version == current_version`
// today. It is written as the loop so that the day a step ships, the
// fixture it starts from is proved through the REAL production plan with no
// edit to this test.
var version = versions.minimum_supported_version;
while (version < versions.current_version) : (version += 1) {
const fixture = fixtureFor(version) orelse return error.MissingFixture;
var h: Harness = .init();
defer h.deinit();
try writeFixture(h.path(), fixture, version);
var side: [256]u8 = undefined;
var pristine = try openPristine(&h, &side, fixture);
defer pristine.close();
var result = try querylog_schema.open(h.io(), std.Io.Dir.cwd(), h.path());
defer result.database.close();
try testing.expectEqual(@as(?querylog_schema.RecreateReason, null), result.recreated);
try testing.expectEqual(
@as(i64, versions.current_version),
try result.database.queryInt("PRAGMA user_version"),
);
var fresh_buf: [256]u8 = undefined;
var fresh = try openFreshCurrent(&h, &fresh_buf);
defer fresh.close();
try testing.expect(try migrations.schemaEquivalent(testing.allocator, &result.database, &fresh));
try expectFixtureSurvived(&result.database, &pristine);
}
}
test "the current version's fixture pair opens on the current lane unchanged" {
// The loop above never reaches this pair, and Gate 2 of the cut requires it
// to exist. This is what proves it is a real, coherent file rather than one
// shipped to satisfy a gate.
const fixture = fixtureFor(versions.current_version) orelse return error.MissingFixture;
var h: Harness = .init();
defer h.deinit();
try writeFixture(h.path(), fixture, versions.current_version);
var side: [256]u8 = undefined;
var pristine = try openPristine(&h, &side, fixture);
defer pristine.close();
var result = try querylog_schema.open(h.io(), std.Io.Dir.cwd(), h.path());
defer result.database.close();
try testing.expectEqual(@as(?querylog_schema.RecreateReason, null), result.recreated);
try testing.expectEqual(
@as(i64, versions.current_version),
try result.database.queryInt("PRAGMA user_version"),
);
var fresh_buf: [256]u8 = undefined;
var fresh = try openFreshCurrent(&h, &fresh_buf);
defer fresh.close();
try testing.expect(try migrations.schemaEquivalent(testing.allocator, &result.database, &fresh));
try expectFixtureSurvived(&result.database, &pristine);
// No migration ran, so nothing reshaped anything: byte-for-byte the rows
// that were loaded.
try expectSameContent(&result.database, &pristine);
}
test "a fixture carrying the 0.0.12 fingerprint restamps, and refuses once the minimum rises" {
const fixture = fixtureFor(1) orelse return error.MissingFixture;
// Version 1 is the legacy fingerprint's logical version, so the pair only
// has anything to say while 1 is still supported.
if (versions.minimum_supported_version <= 1 and versions.current_version >= 1) {
var h: Harness = .init();
defer h.deinit();
try writeFixture(h.path(), fixture, versions.legacy_fingerprint);
var side: [256]u8 = undefined;
var pristine = try openPristine(&h, &side, fixture);
defer pristine.close();
var result = try querylog_schema.open(h.io(), std.Io.Dir.cwd(), h.path());
defer result.database.close();
try testing.expectEqual(@as(?querylog_schema.RecreateReason, null), result.recreated);
try testing.expectEqual(
@as(i64, versions.current_version),
try result.database.queryInt("PRAGMA user_version"),
);
var fresh_buf: [256]u8 = undefined;
var fresh = try openFreshCurrent(&h, &fresh_buf);
defer fresh.close();
try testing.expect(try migrations.schemaEquivalent(testing.allocator, &result.database, &fresh));
try expectFixtureSurvived(&result.database, &pristine);
}
// The companion, already in the suite for the day a break raises the
// minimum above 1: the same bytes under the same stamp are then a file this
// build cannot reach, and it is refused without being touched.
var h: Harness = .init();
defer h.deinit();
try writeFixture(h.path(), fixture, versions.legacy_fingerprint);
const after_break: querylog_schema.Plan = .{
.minimum = 2,
.current = 2,
.legacy_fingerprint = versions.legacy_fingerprint,
.step_sql = &.{},
};
try testing.expect(querylog_schema.classify(versions.legacy_fingerprint, after_break).action ==
.refuse_unsupported);
try testing.expect(!querylog_schema.classify(versions.legacy_fingerprint, after_break).restamp);
var handle: ?db.Db = try h.openLive();
defer if (handle) |*open_db| open_db.close();
migrations.expected_failures.begin();
defer migrations.expected_failures.end();
try testing.expectError(
error.SchemaUnsupported,
querylog_schema.openVersioned(h.io(), std.Io.Dir.cwd(), h.path(), &handle, after_break),
);
var check = try h.openLive();
defer check.close();
try testing.expectEqual(
@as(i64, versions.legacy_fingerprint),
try check.queryInt("PRAGMA user_version"),
);
var pristine_buf: [256]u8 = undefined;
var pristine = try openPristine(&h, &pristine_buf, fixture);
defer pristine.close();
try expectSameContent(&check, &pristine);
}
File diff suppressed because it is too large Load Diff
+811 -43
View File
@@ -1,14 +1,15 @@
//! The `querylog.db` schema and its open-or-recreate policy.
//! The `querylog.db` schema and its open policy.
//!
//! `querylog.db` is never migrated (PLAN §3.7). It holds expendable log rows,
//! so a schema change replaces the file instead of upgrading it. The
//! replacement trigger is a fingerprint derived from the DDL text itself, so
//! editing the schema below automatically invalidates every existing file — the
//! policy cannot drift out of sync with the SQL.
//! `querylog.db` carries a logical schema version in `PRAGMA user_version`
//! (`querylog_versions.zig`), and a file stamped below the current version is
//! MIGRATED in place. A healthy file is never replaced and never set aside: a
//! version this build cannot reach refuses the startup with instructions
//! instead, because the operator's query history is not this program's to
//! discard.
//!
//! **Recreating is destructive, so the predicate is a positive whitelist.** Only
//! a missing file, `error.Corrupt`, `error.NotADb`, a failed `PRAGMA
//! quick_check` and a fingerprint mismatch recreate. Every other error
//! a missing file, `error.Corrupt`, `error.NotADb` and a failed `PRAGMA
//! quick_check` recreate — genuine corruption, nothing else. Every other error
//! propagates and the file on disk is not touched. `error.Busy` / `error.Locked`
//! mean another process holds the write lock — waiting is right, deleting is
//! catastrophic. `error.OutOfMemory` is this process's problem. `error.CantOpen`
@@ -19,13 +20,25 @@
const std = @import("std");
const db = @import("db.zig");
const migrations = @import("querylog_migrations.zig");
const versions = @import("querylog_versions.zig");
const log = std.log.scoped(.querylog_schema);
/// See `querylog_migrations.fail`: `err`, unless a test has said it is causing
/// this refusal on purpose.
fn fail(comptime fmt: []const u8, args: anytype) void {
if (migrations.expected_failures.capturing()) {
log.warn(fmt, args);
} else {
log.err(fmt, args);
}
}
/// PLAN §11.3, plus the coverage watermark of milestone 28. Multi-statement
/// text — it goes through `db.Db.exec`, never through `prepare`.
///
/// The trailing INSERT seeds `querylog_meta`, which is part of the schema
/// The INSERT seeds `querylog_meta`, which is part of the schema
/// rather than a later step: a `query_log` with no watermark beside it cannot
/// answer whether an empty result means "no queries" or "no history", and every
/// database this program reads from is created by executing this string.
@@ -36,6 +49,13 @@ const log = std.log.scoped(.querylog_schema);
/// logged in the same second the file was created is not evidence that the
/// second is completely covered, and the watermark's whole job is to be
/// conservative. From there it only ever advances, in `queries_repo.pruneOlderThan`.
///
/// The four `bucket_*` tables are the Overview projections (milestone 36), on a
/// 30-minute grain that divides every serving width the API offers. They carry
/// no history of their own: they are born with the file and maintained in the
/// same transaction as every insert and every prune, so SQLite's transaction is
/// the only coherence mechanism there is. There is no backfill path — a file
/// whose projections could disagree with its rows cannot exist.
pub const ddl: [:0]const u8 =
\\CREATE TABLE domains (
\\ id INTEGER PRIMARY KEY,
@@ -78,8 +98,50 @@ pub const ddl: [:0]const u8 =
\\);
\\INSERT INTO querylog_meta (id, created_at, available_since)
\\VALUES (1, unixepoch(), unixepoch() + 1);
\\
\\CREATE TABLE bucket_totals (
\\ bucket INTEGER PRIMARY KEY,
\\ queries INTEGER NOT NULL,
\\ blocked INTEGER NOT NULL,
\\ cached INTEGER NOT NULL,
\\ rt_sum INTEGER NOT NULL, -- sum(response_time_us) over timed rows
\\ rt_count INTEGER NOT NULL -- count(response_time_us)
\\) WITHOUT ROWID;
\\
\\CREATE TABLE bucket_clients (
\\ bucket INTEGER NOT NULL,
\\ client_ip TEXT NOT NULL,
\\ queries INTEGER NOT NULL,
\\ PRIMARY KEY (bucket, client_ip)
\\) WITHOUT ROWID;
\\
\\CREATE TABLE bucket_types (
\\ bucket INTEGER NOT NULL,
\\ qtype INTEGER NOT NULL, -- -1 encodes a NULL qtype, losslessly
\\ count INTEGER NOT NULL,
\\ PRIMARY KEY (bucket, qtype)
\\) WITHOUT ROWID;
\\
\\CREATE TABLE bucket_routes (
\\ bucket INTEGER NOT NULL,
\\ route_kind TEXT NOT NULL,
\\ source_present INTEGER NOT NULL, -- 0: source NULL; 1: source = source_text
\\ source_text TEXT NOT NULL, -- '' when source_present = 0
\\ count INTEGER NOT NULL,
\\ PRIMARY KEY (bucket, route_kind, source_present, source_text),
\\ CHECK (source_present IN (0, 1)),
\\ CHECK (source_present = 1 OR source_text = '')
\\) WITHOUT ROWID;
;
/// The fingerprint of an arbitrary DDL text. `tools/cut.zig` calls this at
/// runtime on the DDL of the previous release tag, so the release gate and the
/// server compute the same number from the same function rather than from two
/// copies of one expression.
pub fn fingerprintOf(text: []const u8) i32 {
return @bitCast(std.hash.Crc32.hash(text));
}
/// `PRAGMA user_version` is a signed 32-bit field. Deriving the fingerprint from
/// the DDL means editing the schema automatically invalidates every existing
/// file — which is exactly the policy.
@@ -87,17 +149,24 @@ pub const fingerprint: i32 = blk: {
// Covers the CRC lookup-table generation in std.hash.crc, which evaluates
// under this scope's quota and overflows the 1000 default (and 100k).
@setEvalBranchQuota(2_000_000);
break :blk @bitCast(std.hash.Crc32.hash(ddl));
break :blk fingerprintOf(ddl);
};
const set_user_version = std.fmt.comptimePrint("PRAGMA user_version = {d};", .{fingerprint});
/// What a fresh file is stamped with. The logical version, not the fingerprint:
/// from this release on, `user_version` is a version number.
const set_user_version = std.fmt.comptimePrint(
"PRAGMA user_version = {d};",
.{versions.current_version},
);
/// Long enough for any path this program will be handed, plus the aside suffix.
/// A longer path is `error.NameTooLong`, which is what the filesystem calls
/// would have returned anyway.
const path_buf_len = 4096 + 64;
pub const RecreateReason = enum { missing, corrupt, not_a_database, quick_check_failed, fingerprint_mismatch };
/// Corruption, and nothing else. A healthy file with a version this build does
/// not handle refuses the startup; it is never recreated and never set aside.
pub const RecreateReason = enum { missing, corrupt, not_a_database, quick_check_failed };
pub const OpenResult = struct {
database: db.Db,
@@ -117,10 +186,75 @@ pub const OpenResult = struct {
}
};
pub const Error = db.Error || error{AsideNameCollision} ||
std.Io.Dir.RenamePreserveError || std.Io.Dir.DeleteFileError || std.Io.Dir.AccessError;
pub const Error = db.Error || error{
AsideNameCollision,
/// The file's version is above this build's. A downgrade, almost always.
SchemaTooNew,
/// The file's version is one this build cannot migrate from: older than
/// `minimum_supported_version`, or not a stamp nxdns ever wrote.
SchemaUnsupported,
MigrationFailed,
MigrationBackupFailed,
NameTooLong,
} || std.Io.Dir.RenamePreserveError || std.Io.Dir.DeleteFileError || std.Io.Dir.AccessError;
/// Opens `path`, recreating it if and only if it is genuinely unusable.
/// The version metadata `openVersioned` works against. Production passes
/// `production_plan`; tests inject synthetic chains, which is what makes
/// migration, the post-commit branch and the ownership of the handle testable
/// through the real open path while the shipped chain is still empty.
pub const Plan = struct {
minimum: i32,
current: i32,
legacy_fingerprint: i32,
step_sql: []const [:0]const u8,
};
pub const production_plan: Plan = .{
.minimum = versions.minimum_supported_version,
.current = versions.current_version,
.legacy_fingerprint = versions.legacy_fingerprint,
.step_sql = versions.step_sql,
};
/// What a stamped `user_version` means. A pure function of the stamp and the
/// plan's three numbers — no file, no clock, no mutation.
pub const Action = enum { open_current, migrate, refuse_too_new, refuse_unsupported };
pub const Classification = struct {
/// The stamp mapped onto the version line. Equal to the stamp except for
/// the legacy fingerprint, which IS version 1.
logical: i32,
action: Action,
/// The legacy fingerprint must be replaced by its logical number before the
/// file is used — but only on a lane that accepts the file. An unsupported
/// file is never modified.
restamp: bool,
};
/// The classification table. The order is load-bearing: the legacy fingerprint
/// becomes version 1 FIRST, and only then is version 1 judged against the
/// plan's range. After a future explicit break raises the minimum above 1, a
/// legacy-stamped file therefore classifies as below-minimum and refuses
/// without ever being restamped.
pub fn classify(stamped: i32, plan: Plan) Classification {
const legacy = stamped == plan.legacy_fingerprint;
const logical: i32 = if (legacy) 1 else stamped;
const action: Action = if (logical == plan.current)
.open_current
else if (logical >= plan.minimum and logical < plan.current)
.migrate
else if (logical > plan.current and logical <= versions.version_floor_guard)
.refuse_too_new
else
.refuse_unsupported;
const accepted = action == .open_current or action == .migrate;
return .{ .logical = logical, .action = action, .restamp = legacy and accepted };
}
/// Opens `path`, recreating it if and only if it is genuinely unusable, and
/// migrating it if and only if it carries an older supported version.
///
/// `path` is resolved twice by two different mechanisms: `dir`-relative for the
/// filesystem calls, and process-cwd-relative by SQLite's VFS, which knows
@@ -131,7 +265,7 @@ pub fn open(io: std.Io, dir: std.Io.Dir, path: [:0]const u8) Error!OpenResult {
var handle: ?db.Db = null;
errdefer if (handle) |*h| h.close();
const reason: ?RecreateReason = probe: {
const cause: RecreateReason = probe: {
dir.access(io, path, .{}) catch |e| switch (e) {
error.FileNotFound => break :probe .missing,
else => |other| return other,
@@ -148,15 +282,10 @@ pub fn open(io: std.Io, dir: std.Io.Dir, path: [:0]const u8) Error!OpenResult {
break :probe recreatable(e) orelse return e;
if (!healthy) break :probe .quick_check_failed;
const stamped = opened.queryInt("PRAGMA user_version") catch |e|
break :probe recreatable(e) orelse return e;
if (stamped != fingerprint) break :probe .fingerprint_mismatch;
break :probe null;
try openVersioned(io, dir, path, &handle, production_plan);
return .{ .database = handle.?, .recreated = null };
};
const cause = reason orelse return .{ .database = handle.?, .recreated = null };
// Close first, so SQLite checkpoints and drops `-wal`/`-shm` where it can.
if (handle) |*h| h.close();
handle = null;
@@ -191,6 +320,154 @@ pub fn open(io: std.Io, dir: std.Io.Dir, path: [:0]const u8) Error!OpenResult {
return result;
}
/// The version half of `open`, against an injectable `plan`.
///
/// `handle` is the slot holding the healthy, pragma-applied connection to
/// `path`. On success the connection stays in it, at `plan.current`. On every
/// failure this function closes the connection and sets the slot to null, so
/// the caller's own error-path close cannot double-close it — including the
/// committed-but-unclean path, which is the one place the handle must be
/// dropped even though the file on disk is fine.
///
/// **nxdns owns `path` exclusively.** It opens `querylog.db` once at startup,
/// before it serves anything, and no second process shares a data directory —
/// the standing deployment contract. The backup-then-lock sequence in
/// `querylog_migrations.runMigration` relies on it: between the `VACUUM INTO`
/// and the `BEGIN IMMEDIATE` there is no lock, and the re-read of
/// `user_version` under the lock is what turns a violation of that contract
/// into a refusal instead of a corrupted migration.
pub fn openVersioned(
io: std.Io,
dir: std.Io.Dir,
path: [:0]const u8,
handle: *?db.Db,
plan: Plan,
) Error!void {
const database = &(handle.*.?);
errdefer closeSlot(handle);
const stamped64 = try database.queryInt("PRAGMA user_version");
const stamped = std.math.cast(i32, stamped64) orelse {
// `user_version` is a signed 32-bit field, so this cannot come from
// SQLite. Refusing is the same answer any other foreign stamp gets.
refusalLog(path, stamped64, plan, "SchemaUnsupported");
return error.SchemaUnsupported;
};
const verdict = classify(stamped, plan);
switch (verdict.action) {
.refuse_too_new => {
refusalLog(path, stamped64, plan, "SchemaTooNew");
return error.SchemaTooNew;
},
.refuse_unsupported => {
refusalLog(path, stamped64, plan, "SchemaUnsupported");
return error.SchemaUnsupported;
},
.open_current, .migrate => {},
}
if (verdict.restamp) try restampLegacy(database, path, verdict.logical);
switch (verdict.action) {
.migrate => {
const first = @as(usize, @intCast(verdict.logical - plan.minimum));
migrations.runMigration(
io,
dir,
path,
database,
plan.step_sql[first..],
verdict.logical,
plan.current,
) catch |e| switch (e) {
// Two different states of the FILE — migrated and kept with its
// backup, or logically untouched — and one shared state of the
// CONNECTION: its pragmas are not what `applyPragmas`
// guarantees, so it must not serve. Closing it here is the
// single close either path gets. After a commit the next start
// opens the migrated file on the current-version lane; after a
// failure it retries the migration from the top.
error.MigrationCommittedButUnclean, error.MigrationFailedUnclean => {
closeSlot(handle);
return error.MigrationFailed;
},
error.MigrationBackupFailed => return error.MigrationBackupFailed,
error.MigrationFailed, error.NameTooLong => return error.MigrationFailed,
else => |other| return other,
};
},
.open_current => migrations.pruneBackupsConservative(io, dir, path),
else => unreachable,
}
}
/// Replaces the 0.0.12/0.0.13 fingerprint stamp with the logical version it
/// stands for. This is the milestone's only real mutation of operator data, so
/// it runs in its own transaction and any failure leaves the legacy stamp and
/// every row exactly as they were — a refusal, never a recreate.
fn restampLegacy(database: *db.Db, path: []const u8, logical: i32) Error!void {
var stamp_buf: [64]u8 = undefined;
const stamp = std.fmt.bufPrintZ(&stamp_buf, "PRAGMA user_version = {d};", .{logical}) catch
unreachable; // an i32 and a fixed prefix cannot overrun 64 bytes
restamp: {
var tx = db.Tx.begin(database) catch break :restamp;
database.exec(stamp) catch {
tx.rollback();
break :restamp;
};
tx.commit() catch {
tx.rollback();
break :restamp;
};
log.info("querylog database '{s}' carried the 0.0.12 schema fingerprint; " ++
"restamped as schema version {d}", .{ path, logical });
return;
}
var buf: [256]u8 = undefined;
fail("cannot restamp querylog database '{s}' as schema version {d}: {s}; " ++
"the file is unchanged", .{ path, logical, database.lastError(&buf) });
return error.MigrationFailed;
}
fn closeSlot(handle: *?db.Db) void {
if (handle.*) |*h| h.close();
handle.* = null;
}
fn refusalLog(path: []const u8, stamped: i64, plan: Plan, name: []const u8) void {
fail("refusing to open querylog database '{s}': it is stamped {d}, and this build " ++
"supports schema versions {d} to {d} ({s}). The file is left exactly as it is; " ++
"see docs/how-to/troubleshoot.md, \"The server refuses to start over querylog.db\"", .{
path, stamped, plan.minimum, plan.current, name,
});
}
/// An additional connection to a `querylog.db` that `open` has already
/// established, with the pragmas every connection to the file needs.
///
/// The canonical opener for every background connection: the log writer, the
/// retention pass and the web task each own one (`retention.zig`'s contract),
/// and a logger generation opens one per writer for that writer's whole life
/// (`logger_controller.zig`) — two writers must never share a handle.
///
/// `dir` and `path` follow `open`'s resolution rule, and `dir` participates in
/// it the same way: the caller passes either an absolute path with `dir` open
/// on its parent, or `std.Io.Dir.cwd()` with a cwd-relative path. Nothing here
/// touches the directory itself — the file already exists by contract — so the
/// handle is present to make the pairing explicit at every call site rather
/// than to be dereferenced.
pub fn reopen(io: std.Io, dir: std.Io.Dir, path: [:0]const u8) db.Error!db.Db {
_ = io;
_ = dir;
var database = try db.Db.open(path, .{ .mode = .read_write_existing });
errdefer database.close();
try db.applyPragmas(&database, .{});
return database;
}
/// The whitelist. `null` means "propagate, do not touch the file".
fn recreatable(e: db.Error) ?RecreateReason {
return switch (e) {
@@ -213,16 +490,15 @@ fn quickCheck(database: *db.Db) db.Error!bool {
/// What the aside file's name calls the reason it was set aside.
///
/// The name is the only account of the reason an operator gets: the log line
/// naming it scrolls away, the file stays for months. `fingerprint_mismatch` is
/// a database with nothing wrong with it — this build's DDL moved — so calling
/// its file "corrupt" invites the operator to delete evidence of a healthy file.
/// naming it scrolls away, the file stays for months. Every tag here names real
/// damage, which is the whole set of reasons left — a healthy file whose
/// version this build cannot handle refuses the startup and is not renamed.
fn asideTag(reason: RecreateReason) []const u8 {
return switch (reason) {
.missing => unreachable, // there is no file to rename
.corrupt => "corrupt",
.not_a_database => "not-a-database",
.quick_check_failed => "quick-check-failed",
.fingerprint_mismatch => "schema-changed",
};
}
@@ -280,6 +556,10 @@ const testing = std.testing;
test "fingerprint matches a fresh hash of the DDL" {
try testing.expectEqual(fingerprint, @as(i32, @bitCast(std.hash.Crc32.hash(ddl))));
// The runtime entry point the release gate uses is the same function the
// comptime constant is built from.
try testing.expectEqual(fingerprint, fingerprintOf(ddl));
try testing.expect(fingerprintOf(ddl[0 .. ddl.len - 1]) != fingerprint);
}
test "ddl creates the query-log tables and every index" {
@@ -289,13 +569,23 @@ test "ddl creates the query-log tables and every index" {
try database.exec(ddl);
try testing.expectEqual(
@as(i64, 3),
@as(i64, 7),
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
);
// The three explicit indexes plus `domains.domain`'s autoindex, and
// nothing else: the four projection tables are WITHOUT ROWID, so each
// one's PRIMARY KEY *is* its storage rather than a second b-tree to keep
// in step on every insert.
try testing.expectEqual(
@as(i64, 4),
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='index'"),
);
const objects = [_][]const u8{
"domains", "query_log",
"idx_query_log_ts", "idx_query_log_client",
"idx_query_log_domain", "querylog_meta",
"bucket_totals", "bucket_clients",
"bucket_types", "bucket_routes",
};
for (objects) |name| {
var stmt = try database.prepare("SELECT count(*) FROM sqlite_schema WHERE name = ?1");
@@ -369,11 +659,14 @@ test "querylog_meta is seeded with one row the schema will not let a second join
try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM querylog_meta"));
}
test "the user_version statement stamps the fingerprint" {
test "the user_version statement stamps the current schema version" {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try database.exec(set_user_version);
try testing.expectEqual(@as(i64, fingerprint), try database.queryInt("PRAGMA user_version"));
try testing.expectEqual(
@as(i64, versions.current_version),
try database.queryInt("PRAGMA user_version"),
);
}
// The behaviour these two cases describe — a resource error leaves the file on
@@ -402,7 +695,6 @@ test "the aside name says why, and a healthy file is never called corrupt" {
try testing.expectEqualStrings("corrupt", asideTag(.corrupt));
try testing.expectEqualStrings("not-a-database", asideTag(.not_a_database));
try testing.expectEqualStrings("quick-check-failed", asideTag(.quick_check_failed));
try testing.expectEqualStrings("schema-changed", asideTag(.fingerprint_mismatch));
}
test "recreatable selects exactly two of db.Error's members" {
@@ -446,7 +738,7 @@ test "a recreate returns the aside name by value and a fresh create returns none
try tmp.dir.access(io, kept, .{});
}
test "a recreate resets coverage to the new file and keeps the old one aside" {
test "a corrupt file is recreated, coverage restarts, and the old one is kept aside" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
@@ -463,21 +755,14 @@ test "a recreate resets coverage to the new file and keeps the old one aside" {
try created.database.exec("INSERT INTO domains (domain) VALUES ('old.example');");
created.database.close();
// A healthy file this build's DDL no longer matches — the case milestone
// 28's own schema edit produces on every upgrade.
{
var stamped = try db.Db.open(path, .{ .mode = .read_write_existing });
defer stamped.close();
var sql_buf: [64]u8 = undefined;
try stamped.exec(try std.fmt.bufPrintZ(&sql_buf, "PRAGMA user_version = {d};", .{fingerprint +% 1}));
}
// Real damage, which is now the only thing that recreates.
try tmp.dir.writeFile(io, .{ .sub_path = "querylog.db", .data = "not a database at all" });
var recreated = try open(io, std.Io.Dir.cwd(), path);
defer recreated.database.close();
try testing.expectEqual(RecreateReason.fingerprint_mismatch, recreated.recreated.?);
// The name says the file was healthy and this build moved, not that it rotted.
try testing.expect(std.mem.indexOf(u8, recreated.aside(), ".schema-changed-") != null);
try testing.expectEqual(RecreateReason.not_a_database, recreated.recreated.?);
try testing.expect(std.mem.indexOf(u8, recreated.aside(), ".not-a-database-") != null);
try tmp.dir.access(io, std.fs.path.basename(recreated.aside()), .{});
// Exactly one meta row, and coverage starts at the recreate rather than
@@ -516,3 +801,486 @@ test "a clean reopen reports no recreate and no aside" {
try testing.expectEqual(@as(?RecreateReason, null), second.recreated);
try testing.expectEqualStrings("", second.aside());
}
test "classification is a pure function of the stamp and the plan's three numbers" {
const plan: Plan = .{
.minimum = 1,
.current = 3,
.legacy_fingerprint = versions.legacy_fingerprint,
.step_sql = &.{},
};
try testing.expectEqual(Action.open_current, classify(3, plan).action);
try testing.expectEqual(Action.migrate, classify(1, plan).action);
try testing.expectEqual(Action.migrate, classify(2, plan).action);
try testing.expectEqual(Action.refuse_too_new, classify(4, plan).action);
try testing.expectEqual(Action.refuse_too_new, classify(versions.version_floor_guard, plan).action);
// Above the floor guard is not a version this project ever wrote.
try testing.expectEqual(
Action.refuse_unsupported,
classify(versions.version_floor_guard + 1, plan).action,
);
for ([_]i32{ 0, -1, -1_000_000, 603440875 }) |foreign| {
try testing.expectEqual(Action.refuse_unsupported, classify(foreign, plan).action);
try testing.expect(!classify(foreign, plan).restamp);
}
// The legacy fingerprint IS version 1, and being version 1 is what decides
// its lane.
const legacy = classify(versions.legacy_fingerprint, plan);
try testing.expectEqual(@as(i32, 1), legacy.logical);
try testing.expectEqual(Action.migrate, legacy.action);
try testing.expect(legacy.restamp);
}
test "a legacy stamp below a raised minimum refuses without a restamp" {
// What a future explicit break looks like from this side: the minimum has
// moved past 1, so the 0.0.12 file is no longer reachable. The ORDER is the
// point — mapping to 1 first and judging second is what stops the restamp
// from mutating a file this build will refuse anyway.
const after_break: Plan = .{
.minimum = 3,
.current = 3,
.legacy_fingerprint = versions.legacy_fingerprint,
.step_sql = &.{},
};
const legacy = classify(versions.legacy_fingerprint, after_break);
try testing.expectEqual(@as(i32, 1), legacy.logical);
try testing.expectEqual(Action.refuse_unsupported, legacy.action);
try testing.expect(!legacy.restamp);
// And versions 1 and 2, which the break dropped, refuse the same way.
try testing.expectEqual(Action.refuse_unsupported, classify(1, after_break).action);
try testing.expectEqual(Action.refuse_unsupported, classify(2, after_break).action);
try testing.expectEqual(Action.open_current, classify(3, after_break).action);
}
test "the production plan classifies a fresh stamp as current" {
try testing.expectEqual(
Action.open_current,
classify(versions.current_version, production_plan).action,
);
try testing.expect(classify(versions.legacy_fingerprint, production_plan).restamp);
}
/// The five lines every file-backed test below opens with.
const Fixture = struct {
threaded: std.Io.Threaded,
tmp: std.testing.TmpDir,
buf: [256]u8 = undefined,
fn init() Fixture {
return .{
.threaded = .init(testing.allocator, .{}),
.tmp = testing.tmpDir(.{ .iterate = true }),
};
}
fn deinit(self: *Fixture) void {
self.tmp.cleanup();
self.threaded.deinit();
}
fn io(self: *Fixture) std.Io {
return self.threaded.io();
}
fn path(self: *Fixture) [:0]const u8 {
return std.fmt.bufPrintZ(&self.buf, ".zig-cache/tmp/{s}/querylog.db", .{self.tmp.sub_path}) catch
unreachable;
}
fn stamp(self: *Fixture, value: i32) !void {
var database = try db.Db.open(self.path(), .{ .mode = .read_write_existing });
defer database.close();
var sql: [64]u8 = undefined;
try database.exec(try std.fmt.bufPrintZ(&sql, "PRAGMA user_version = {d};", .{value}));
}
fn liveHandle(self: *Fixture) !?db.Db {
var database = try db.Db.open(self.path(), .{ .mode = .read_write_existing });
errdefer database.close();
try db.applyPragmas(&database, .{});
return database;
}
fn countMatching(self: *Fixture, prefix: []const u8) !usize {
var found: usize = 0;
var it = self.tmp.dir.iterate();
while (try it.next(self.io())) |entry| {
if (std.mem.startsWith(u8, entry.name, prefix)) found += 1;
}
return found;
}
fn expectModeOfOnlyMatch(self: *Fixture, prefix: []const u8, expected: std.posix.mode_t) !void {
var it = self.tmp.dir.iterate();
while (try it.next(self.io())) |entry| {
if (!std.mem.startsWith(u8, entry.name, prefix)) continue;
const stat = try self.tmp.dir.statFile(self.io(), entry.name, .{});
const mode = stat.permissions.toMode() & 0o777;
if (mode != expected) {
std.debug.print("mode of '{s}' is {o}, expected {o}\n", .{ entry.name, mode, expected });
return error.TestUnexpectedResult;
}
return;
}
std.debug.print("no file starting with '{s}'\n", .{prefix});
return error.TestUnexpectedResult;
}
};
test "a fresh file is stamped with the current schema version" {
var f: Fixture = .init();
defer f.deinit();
var created = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer created.database.close();
try testing.expectEqual(RecreateReason.missing, created.recreated.?);
try testing.expectEqual(
@as(i64, versions.current_version),
try created.database.queryInt("PRAGMA user_version"),
);
}
test "a 0.0.13 file is restamped as version 1 and keeps every row" {
var f: Fixture = .init();
defer f.deinit();
{
var created = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer created.database.close();
try created.database.exec("INSERT INTO domains (domain) VALUES ('kept.example');");
}
// Exactly what 0.0.12 and 0.0.13 wrote: the CRC of their DDL, which is this
// build's DDL unchanged.
try f.stamp(versions.legacy_fingerprint);
try testing.expectEqual(versions.legacy_fingerprint, fingerprint);
{
var upgraded = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer upgraded.database.close();
try testing.expectEqual(@as(?RecreateReason, null), upgraded.recreated);
try testing.expectEqual(@as(i64, 1), try upgraded.database.queryInt("PRAGMA user_version"));
try testing.expectEqual(
@as(i64, 1),
try upgraded.database.queryInt("SELECT count(*) FROM domains WHERE domain = 'kept.example'"),
);
}
// Nothing was set aside on the way, and the second start is an ordinary
// current-version open.
try testing.expectEqual(@as(usize, 0), try f.countMatching("querylog.db.schema"));
var again = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer again.database.close();
try testing.expectEqual(@as(?RecreateReason, null), again.recreated);
try testing.expectEqual(@as(i64, 1), try again.database.queryInt("PRAGMA user_version"));
}
test "a version this build cannot handle refuses and leaves the file alone" {
var f: Fixture = .init();
defer f.deinit();
{
var created = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer created.database.close();
try created.database.exec("INSERT INTO domains (domain) VALUES ('kept.example');");
}
const watermark = blk: {
var probe = try db.Db.open(f.path(), .{ .mode = .read_write_existing });
defer probe.close();
break :blk try probe.queryInt("SELECT available_since FROM querylog_meta");
};
migrations.expected_failures.begin();
defer migrations.expected_failures.end();
const lanes = [_]struct { stamp: i32, expected: anyerror }{
.{ .stamp = versions.current_version + 1, .expected = error.SchemaTooNew },
.{ .stamp = versions.version_floor_guard, .expected = error.SchemaTooNew },
.{ .stamp = 0, .expected = error.SchemaUnsupported },
.{ .stamp = -3, .expected = error.SchemaUnsupported },
.{ .stamp = 603440875, .expected = error.SchemaUnsupported },
};
for (lanes) |lane| {
try f.stamp(lane.stamp);
try testing.expectError(lane.expected, open(f.io(), std.Io.Dir.cwd(), f.path()));
// Schema, rows, watermark and stamp all as they were, and nothing new
// beside the file.
var probe = try db.Db.open(f.path(), .{ .mode = .read_write_existing });
defer probe.close();
try testing.expectEqual(@as(i64, lane.stamp), try probe.queryInt("PRAGMA user_version"));
try testing.expectEqual(
@as(i64, 1),
try probe.queryInt("SELECT count(*) FROM domains WHERE domain = 'kept.example'"),
);
try testing.expectEqual(
watermark,
try probe.queryInt("SELECT available_since FROM querylog_meta"),
);
try testing.expectEqual(@as(usize, 0), try f.countMatching("querylog.db."));
}
}
test "a restamp that fails at the statement or at the commit refuses without loss" {
for ([_][]const u8{ "PRAGMA user_version = 1;", "COMMIT;" }) |failing| {
var f: Fixture = .init();
defer f.deinit();
{
var created = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer created.database.close();
try created.database.exec("INSERT INTO domains (domain) VALUES ('kept.example');");
}
try f.stamp(versions.legacy_fingerprint);
migrations.expected_failures.begin();
defer migrations.expected_failures.end();
db.exec_faults.failNextMatching(failing);
defer db.exec_faults.disarm();
try testing.expectError(
error.MigrationFailed,
open(f.io(), std.Io.Dir.cwd(), f.path()),
);
try testing.expect(!db.exec_faults.armed());
// The legacy stamp and every row are exactly as they were: this is a
// refusal, and a refusal never costs the operator anything.
{
var probe = try db.Db.open(f.path(), .{ .mode = .read_write_existing });
defer probe.close();
try testing.expectEqual(
@as(i64, versions.legacy_fingerprint),
try probe.queryInt("PRAGMA user_version"),
);
}
// And the next start, with nothing injected, does the restamp properly.
var recovered = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer recovered.database.close();
try testing.expectEqual(@as(i64, 1), try recovered.database.queryInt("PRAGMA user_version"));
try testing.expectEqual(
@as(i64, 1),
try recovered.database.queryInt("SELECT count(*) FROM domains WHERE domain = 'kept.example'"),
);
}
}
/// A one-step chain from the real current version to one above it. Nothing in
/// the shipped chain can exercise migration while `step_sql` is empty, so the
/// open path's migration lanes are driven through this instead.
const synthetic_plan: Plan = .{
.minimum = versions.current_version,
.current = versions.current_version + 1,
.legacy_fingerprint = versions.legacy_fingerprint,
.step_sql = &.{"CREATE TABLE migration_marker (id INTEGER PRIMARY KEY);"},
};
test "a post-commit failure keeps the migration, keeps the backup, and refuses once" {
var f: Fixture = .init();
defer f.deinit();
{
var created = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer created.database.close();
try created.database.exec("INSERT INTO domains (domain) VALUES ('kept.example');");
}
{
var handle = try f.liveHandle();
errdefer if (handle) |*h| h.close();
migrations.expected_failures.begin();
defer migrations.expected_failures.end();
// The first statement of the pragma restore, which runs only after a
// successful COMMIT.
db.exec_faults.failNextMatching("legacy_alter_table = OFF");
defer db.exec_faults.disarm();
try testing.expectError(
error.MigrationFailed,
openVersioned(f.io(), std.Io.Dir.cwd(), f.path(), &handle, synthetic_plan),
);
// Closed exactly once, by the open path: the slot it was handed is
// empty, so no caller can close it again.
try testing.expect(handle == null);
}
// The file IS migrated. The log said so, and this is what it meant.
{
var probe = try db.Db.open(f.path(), .{ .mode = .read_write_existing });
defer probe.close();
try testing.expectEqual(
@as(i64, synthetic_plan.current),
try probe.queryInt("PRAGMA user_version"),
);
try testing.expectEqual(
@as(i64, 1),
try probe.queryInt("SELECT count(*) FROM sqlite_schema WHERE name = 'migration_marker'"),
);
try testing.expectEqual(
@as(i64, 1),
try probe.queryInt("SELECT count(*) FROM domains WHERE domain = 'kept.example'"),
);
}
try testing.expectEqual(@as(usize, 1), try f.countMatching("querylog.db.pre-migrate-"));
// The backup is the whole query history at the moment of the migration, so
// it carries the live file's mode and not SQLite's `0644 & ~umask`.
try f.expectModeOfOnlyMatch("querylog.db.pre-migrate-", 0o600);
// The next start is ordinary: the current-version lane, no second
// migration, and the one backup still there for the operator.
var handle = try f.liveHandle();
defer if (handle) |*h| h.close();
try openVersioned(f.io(), std.Io.Dir.cwd(), f.path(), &handle, synthetic_plan);
try testing.expect(handle != null);
try testing.expectEqual(
@as(i64, synthetic_plan.current),
try handle.?.queryInt("PRAGMA user_version"),
);
try testing.expectEqual(@as(usize, 1), try f.countMatching("querylog.db.pre-migrate-"));
}
/// The same shape as `synthetic_plan`, with a step SQLite refuses to prepare. It
/// drives the pre-commit failure path without any fault seam, leaving the seam
/// free for the pragma restore.
const failing_plan: Plan = .{
.minimum = versions.current_version,
.current = versions.current_version + 1,
.legacy_fingerprint = versions.legacy_fingerprint,
.step_sql = &.{"CREATE TABLE migration_marker (id INTEGER PRIMARY KEY) NOT A STATEMENT;"},
};
test "a pre-commit failure whose pragma restore also fails closes the connection" {
var f: Fixture = .init();
defer f.deinit();
{
var created = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer created.database.close();
try created.database.exec("INSERT INTO domains (domain) VALUES ('kept.example');");
}
// The runner's own answer first: a failed restore is a DIFFERENT error from
// a failed migration, because the two leave the connection in different
// states even though they leave the file in the same one.
{
var handle = try f.liveHandle();
defer if (handle) |*h| h.close();
migrations.expected_failures.begin();
defer migrations.expected_failures.end();
db.exec_faults.failNextMatching("legacy_alter_table = OFF");
defer db.exec_faults.disarm();
try testing.expectError(error.MigrationFailedUnclean, migrations.runMigration(
f.io(),
std.Io.Dir.cwd(),
f.path(),
&handle.?,
failing_plan.step_sql,
failing_plan.minimum,
failing_plan.current,
));
try testing.expect(!db.exec_faults.armed());
}
// And the open path's answer: the handle is closed, exactly as it is after a
// post-commit restore failure. A connection that may still hold
// `foreign_keys = OFF` never reaches the server.
{
var handle = try f.liveHandle();
errdefer if (handle) |*h| h.close();
migrations.expected_failures.begin();
defer migrations.expected_failures.end();
db.exec_faults.failNextMatching("legacy_alter_table = OFF");
defer db.exec_faults.disarm();
try testing.expectError(
error.MigrationFailed,
openVersioned(f.io(), std.Io.Dir.cwd(), f.path(), &handle, failing_plan),
);
try testing.expect(handle == null);
}
// The contrast that makes the rule visible, back at the runner, where the
// connection survives to be inspected: the same failing step with the
// restore working is a plain `MigrationFailed`, and that error promises the
// pragmas `applyPragmas` guarantees.
{
var handle = try f.liveHandle();
defer if (handle) |*h| h.close();
migrations.expected_failures.begin();
defer migrations.expected_failures.end();
try testing.expectError(error.MigrationFailed, migrations.runMigration(
f.io(),
std.Io.Dir.cwd(),
f.path(),
&handle.?,
failing_plan.step_sql,
failing_plan.minimum,
failing_plan.current,
));
try testing.expectEqual(@as(i64, 1), try handle.?.queryInt("PRAGMA foreign_keys"));
try testing.expectEqual(@as(i64, 0), try handle.?.queryInt("PRAGMA legacy_alter_table"));
}
// No path committed anything, and every run deleted its own backup.
var probe = try db.Db.open(f.path(), .{ .mode = .read_write_existing });
defer probe.close();
try testing.expectEqual(
@as(i64, failing_plan.minimum),
try probe.queryInt("PRAGMA user_version"),
);
try testing.expectEqual(
@as(i64, 1),
try probe.queryInt("SELECT count(*) FROM domains WHERE domain = 'kept.example'"),
);
try testing.expectEqual(@as(usize, 0), try f.countMatching("querylog.db.pre-migrate-"));
}
test "a plain open retries a retention cleanup that once failed" {
var f: Fixture = .init();
defer f.deinit();
{
var created = try open(f.io(), std.Io.Dir.cwd(), f.path());
created.database.close();
}
// What a migration whose step-4 cleanup failed leaves behind: an older
// epoch, the newest epoch, and a same-second collision name tied with it.
const seeded = [_][]const u8{
"querylog.db.pre-migrate-1600000000",
"querylog.db.pre-migrate-1700000000",
"querylog.db.pre-migrate-1700000000-2",
"querylog.db.pre-migrate-handwritten",
};
for (seeded) |name| {
try f.tmp.dir.writeFile(f.io(), .{ .sub_path = name, .data = "x" });
}
var opened = try open(f.io(), std.Io.Dir.cwd(), f.path());
defer opened.database.close();
try testing.expectEqual(@as(?RecreateReason, null), opened.recreated);
// Only the strictly older epoch goes: both files tied at the newest epoch
// survive, because this pass cannot tell which of them a migration made,
// and a name it did not write is never its to delete.
try testing.expect(!try exists(&f, seeded[0]));
for (seeded[1..]) |name| try testing.expect(try exists(&f, name));
}
fn exists(f: *Fixture, name: []const u8) !bool {
f.tmp.dir.access(f.io(), name, .{}) catch |e| switch (e) {
error.FileNotFound => return false,
else => return e,
};
return true;
}
+89
View File
@@ -0,0 +1,89 @@
//! The `querylog.db` schema version chain: comptime metadata and nothing else.
//!
//! Separate from `querylog_migrations.zig` so `tools/cut.zig` can import it
//! without linking SQLite. Nothing in this file may reach for `db.zig`, for a
//! C symbol, or for an allocator — the release gate reads these constants at
//! build time, and a dependency here would drag the whole storage layer into
//! the cut tool.
//!
//! **There are no migration hooks.** A step is a SQL file, period. Every
//! shipped step is therefore byte-comparable against the previous tag, which is
//! what lets the cut gate prove a released migration was never edited. A future
//! change that genuinely cannot be expressed in SQL must amend this design in
//! its own spec rather than adding a code path here.
const std = @import("std");
/// The version a file created by this build carries in `PRAGMA user_version`.
pub const current_version: i32 = 1;
/// The oldest stamped version this build can reach `current_version` from.
/// A file stamped below this refuses to open.
///
/// An EXPLICIT BREAK in a future release is expressed here and only here: bump
/// `current_version`, set `minimum_supported_version = current_version`, and
/// ship no step. The chain then cannot reach the new version from below the
/// minimum, so `open` refuses the old file by the ordinary rules. A break is
/// always versioned, always refused at runtime, and never silent.
pub const minimum_supported_version: i32 = 1;
/// The literal `user_version` the 0.0.12 and 0.0.13 binaries stamped: the CRC32
/// of their DDL text, under the pre-migration policy where a stamp mismatch
/// meant "replace the file".
///
/// FROZEN. It is derived from nothing at build time on purpose — recomputing it
/// from today's DDL would silently stop recognising the files it exists to
/// recognise the moment the schema moves. Editing it strands every 0.0.12 and
/// 0.0.13 file that has not yet been opened by a migration-aware build, which
/// is why the cut gate fails on any change to this line.
pub const legacy_fingerprint: i32 = 1975011655;
/// Logical versions live far below any plausible CRC32 stamp. A value above
/// this is not a version this project ever wrote, so it classifies as
/// unsupported rather than as a from-the-future schema.
pub const version_floor_guard: i32 = 1_000_000;
/// One entry per shipped step: `step_sql[i]` migrates version
/// `minimum_supported_version + i` to `minimum_supported_version + i + 1`.
/// Each entry is `@embedFile("migrations/v<from>.sql")`, and each such file is
/// immutable once released.
///
/// **Step-authoring rules** (the runner enforces the first, the equivalence
/// oracle catches violations of the rest):
///
/// - A step contains no transaction statement. No `BEGIN`, no `COMMIT`, no
/// `ROLLBACK`, no `SAVEPOINT`: the runner wraps the whole chain in one
/// transaction and installs an authorizer that denies them outright.
/// - A step that changes a table's shape must REBUILD it, so that the CREATE
/// text SQLite stores ends up byte-identical to the fresh DDL's:
/// `DROP` every view over `<t>` first; `ALTER TABLE <t> RENAME TO <t>_old`;
/// `CREATE TABLE <t> ...` pasted verbatim from `querylog_schema.ddl`;
/// `INSERT INTO <t> SELECT ... FROM <t>_old`; `DROP TABLE <t>_old`; recreate
/// every index and trigger of `<t>` verbatim; recreate the dropped views
/// verbatim last.
/// - `ALTER TABLE ... ADD COLUMN` and `ALTER TABLE ... RENAME COLUMN` on a kept
/// table are forbidden. SQLite rewrites the stored CREATE text under them,
/// and the oracle's exact-text layer would rightly call the result unequal.
pub const step_sql: []const [:0]const u8 = &.{};
comptime {
std.debug.assert(minimum_supported_version >= 1);
std.debug.assert(minimum_supported_version <= current_version);
std.debug.assert(current_version <= version_floor_guard);
std.debug.assert(legacy_fingerprint < 0 or legacy_fingerprint > version_floor_guard);
std.debug.assert(step_sql.len == @as(usize, @intCast(current_version - minimum_supported_version)));
}
test "the chain covers exactly the supported range" {
try std.testing.expectEqual(
@as(usize, @intCast(current_version - minimum_supported_version)),
step_sql.len,
);
}
test "the legacy anchor is the literal 0.0.12 stamp" {
// Not `fingerprintOf(ddl)`. The number is a historical fact about released
// binaries, so a test that recomputed it would move with the schema and
// prove nothing.
try std.testing.expectEqual(@as(i32, 1975011655), legacy_fingerprint);
}
File diff suppressed because it is too large Load Diff
+78 -18
View File
@@ -14,7 +14,6 @@ const std = @import("std");
const db = @import("db.zig");
const disk_monitor = @import("disk_monitor.zig");
const events = @import("events.zig");
const model = @import("../config/model.zig");
const queries_repo = @import("repositories/queries_repo.zig");
const log = std.log.scoped(.retention);
@@ -50,15 +49,42 @@ const Counters = struct {
vacuums_gated: std.atomic.Value(u64) = .init(0),
};
/// `logging.retention_days`, shared by its two consumers — this pass and
/// `server/clients.zig`'s stale-client prune. One cell rather than a copy in
/// each: the two must never prune to different cutoffs, and a settings apply
/// stores once. Owned by app-level state and outlives both readers.
///
/// `.monotonic` is enough: the value stands alone and orders nothing else, and
/// each consumer reads it once per pass.
pub const RetentionDays = struct {
value: std.atomic.Value(u32),
pub fn init(days: u16) RetentionDays {
return .{ .value = .init(days) };
}
pub fn get(self: *const RetentionDays) u32 {
return self.value.load(.monotonic);
}
pub fn setRetentionDays(self: *RetentionDays, days: u16) void {
self.value.store(days, .monotonic);
}
pub fn seconds(self: *const RetentionDays) i64 {
return @as(i64, self.get()) * std.time.s_per_day;
}
};
pub const Retention = struct {
cfg: model.Logging,
days: *const RetentionDays,
counters: Counters,
/// Passes since the last vacuum that succeeded. Plain rather than atomic:
/// only the retention task reads or writes it, and no consumer reports it.
passes_since_vacuum: u32,
pub fn init(cfg: model.Logging) Retention {
return .{ .cfg = cfg, .counters = .{}, .passes_since_vacuum = 0 };
pub fn init(days: *const RetentionDays) Retention {
return .{ .days = days, .counters = .{}, .passes_since_vacuum = 0 };
}
/// The counters, read one at a time. A scrape that lands mid-pass can see a
@@ -100,7 +126,7 @@ pub const Retention = struct {
) void {
add(&self.counters.passes, 1);
const now = std.Io.Clock.real.now(io).toSeconds();
const cutoff = now - model.retentionSeconds(self.cfg);
const cutoff = now - self.days.seconds();
// Diagnostics retention rides this pass rather than a schedule of its
// own: one daily housekeeping task, and a box restarted every night
@@ -225,7 +251,7 @@ fn openLog() !db.Db {
}
fn writeRows(database: *db.Db, timestamps: []const i64) !void {
var writer = try queries_repo.BatchWriter.init(database);
var writer = try queries_repo.BatchWriter.init(testing.allocator, database);
defer writer.deinit();
var rows: [8]queries_repo.Row = undefined;
for (timestamps, rows[0..timestamps.len]) |timestamp, *row| {
@@ -268,7 +294,8 @@ test "a pass prunes the rows past the retention window and keeps the rest" {
const day = 86_400;
try writeRows(&database, &.{ now - 40 * day, now - 31 * day, now - 29 * day, now - 60 });
var retention: Retention = .init(.{ .retention_days = 30 });
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 2), try queries_repo.countRows(&database));
@@ -292,17 +319,41 @@ test "the cutoff follows retention_days" {
// window of the other.
try writeRows(&database, &.{now - 3 * day});
var keeps: Retention = .init(.{ .retention_days = 7 });
var keeps_days: RetentionDays = .init(7);
var keeps: Retention = .init(&keeps_days);
keeps.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 0), keeps.snapshotStats().rows_pruned);
var prunes: Retention = .init(.{ .retention_days = 1 });
var prunes_days: RetentionDays = .init(1);
var prunes: Retention = .init(&prunes_days);
prunes.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 1), prunes.snapshotStats().rows_pruned);
}
test "setRetentionDays changes the cutoff the next pass prunes by" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const now = std.Io.Clock.real.now(io).toSeconds();
try writeRows(&database, &.{now - 3 * 86_400});
var days: RetentionDays = .init(7);
var pass: Retention = .init(&days);
pass.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(&database));
days.setRetentionDays(1);
pass.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 1), pass.snapshotStats().rows_pruned);
}
test "the seventh pass vacuums and the six before it do not" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -311,7 +362,8 @@ test "the seventh pass vacuums and the six before it do not" {
var database = try openLog();
defer database.close();
var retention: Retention = .init(.{});
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
for (0..6) |_| {
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(u64, 0), retention.snapshotStats().vacuums);
@@ -340,7 +392,8 @@ test "a gated pass skips the vacuum, counts it, and vacuums on the next pass" {
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
var gated: Retention = .init(.{});
var gated_days: RetentionDays = .init(30);
var gated: Retention = .init(&gated_days);
for (0..vacuum_every_passes) |_| gated.runOnce(io, &database, &monitor, null);
// Prune and checkpoint ran on every pass; only the vacuum was refused.
@@ -371,7 +424,8 @@ test "a warn state still allows the vacuum" {
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.warn), .monotonic);
var retention: Retention = .init(.{});
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
for (0..vacuum_every_passes) |_| retention.runOnce(io, &database, &monitor, null);
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().vacuums);
@@ -386,7 +440,8 @@ test "a pass over an empty database still counts" {
var database = try openLog();
defer database.close();
var retention: Retention = .init(.{});
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().passes);
@@ -410,7 +465,8 @@ test "a failing prune counts the pass and leaves the rows alone" {
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
var retention: Retention = .init(.{ .retention_days = 30 });
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(&database));
@@ -435,7 +491,8 @@ test "the next pass retries what the failed one could not do" {
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
var retention: Retention = .init(.{ .retention_days = 30 });
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 2), try queries_repo.countRows(&database));
@@ -466,7 +523,8 @@ test "a failing prune opens a maintenance episode the next clean pass closes" {
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
var retention: Retention = .init(.{});
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
retention.runOnce(io, &database, null, &fx.store);
// Only the prune failed; the checkpoint succeeded, and a success writes no
@@ -501,7 +559,8 @@ test "a gated vacuum is a maintenance failure the next ungated pass closes" {
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
var retention: Retention = .init(.{});
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
for (0..vacuum_every_passes) |_| retention.runOnce(io, &database, &monitor, &fx.store);
try testing.expectEqualStrings("vacuum", try fx.text(
@@ -536,7 +595,8 @@ test "a pass prunes the diagnostics store once" {
fx.store.reportResolved(io, stale, .query_log_recreated, "one-shot", "one-shot", .warning, "aside kept");
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
var retention: Retention = .init(.{});
var retention_days: RetentionDays = .init(30);
var retention: Retention = .init(&retention_days);
retention.runOnce(io, &database, null, &fx.store);
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
+37 -22
View File
@@ -28,7 +28,9 @@ const model = @import("../config/model.zig");
const validate = @import("../config/validate.zig");
const db = @import("db.zig");
const migrations = @import("migrations.zig");
const querylog_migrations = @import("querylog_migrations.zig");
const querylog_schema = @import("querylog_schema.zig");
const querylog_versions = @import("querylog_versions.zig");
const testing = std.testing;
@@ -354,7 +356,7 @@ test "S7 case 1: querylog open on a fresh directory creates the schema" {
try testing.expectEqual(querylog_schema.RecreateReason.missing, result.recreated.?);
try testing.expectEqual(
@as(i64, querylog_schema.fingerprint),
@as(i64, querylog_versions.current_version),
try result.database.queryInt("PRAGMA user_version"),
);
try testing.expectEqual(
@@ -387,7 +389,7 @@ test "S7 case 2: reopening a healthy querylog recreates nothing" {
try testing.expectEqual(@as(usize, 0), asides.items.items.len);
}
test "S7 case 3: a wrong user_version recreates and keeps the old file aside" {
test "S7 case 3: a version this build cannot handle refuses and touches nothing" {
if (!build_options.integration) return error.SkipZigTest;
var f: Fixture = .init();
@@ -396,29 +398,42 @@ test "S7 case 3: a wrong user_version recreates and keeps the old file aside" {
var buf: [path_buf_len]u8 = undefined;
const path = try f.pathZ(&buf, "querylog.db");
try stampUserVersion(path, querylog_schema.fingerprint +% 1);
querylog_migrations.expected_failures.begin();
defer querylog_migrations.expected_failures.end();
// Both refusal lanes, against a checkpointed file with no sidecars beside
// it: a stamp above this build's version (a downgrade) and a stamp that is
// not a version at all (a pre-0.0.12 fingerprint, or a foreign file).
const refusals = [_]struct { stamp: i32, expected: anyerror }{
.{ .stamp = querylog_versions.current_version + 1, .expected = error.SchemaTooNew },
.{ .stamp = 0, .expected = error.SchemaUnsupported },
.{ .stamp = -7, .expected = error.SchemaUnsupported },
.{ .stamp = 603440875, .expected = error.SchemaUnsupported },
};
for (refusals) |lane| {
try stampUserVersion(path, lane.stamp);
try testing.expect(!try f.exists("querylog.db-wal"));
const original = try f.read("querylog.db");
defer testing.allocator.free(original);
var result = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer result.database.close();
try testing.expectEqual(
querylog_schema.RecreateReason.fingerprint_mismatch,
result.recreated.?,
try testing.expectError(
lane.expected,
querylog_schema.open(io, std.Io.Dir.cwd(), path),
);
// Byte-identical, not merely "still readable": nothing was rewritten,
// no aside was made, and no fresh database was created beside it.
const after = try f.read("querylog.db");
defer testing.allocator.free(after);
try testing.expectEqualSlices(u8, original, after);
var asides = try collectAsides(&f);
defer asides.deinit();
try testing.expectEqual(@as(usize, 1), asides.items.items.len);
// The file was healthy: this build's schema moved, the database did not rot.
// An operator who reads "corrupt" here deletes a file that was never broken.
try testing.expect(std.mem.startsWith(u8, asides.items.items[0], "querylog.db.schema-changed-"));
const kept = try f.read(asides.items.items[0]);
defer testing.allocator.free(kept);
try testing.expectEqualSlices(u8, original, kept);
try testing.expectEqual(@as(usize, 0), asides.items.items.len);
}
}
test "S7 case 4: a garbage file recreates and the garbage is preserved" {
@@ -468,11 +483,11 @@ test "S7 case 5: two recreates in the same second produce two distinct aside fil
var round: usize = 0;
while (round < 2) : (round += 1) {
try stampUserVersion(path, querylog_schema.fingerprint +% 1);
try f.write("querylog.db", "not a database at all");
var result = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer result.database.close();
try testing.expectEqual(
querylog_schema.RecreateReason.fingerprint_mismatch,
querylog_schema.RecreateReason.not_a_database,
result.recreated.?,
);
}
@@ -492,7 +507,7 @@ test "S7 case 6: a stale write-ahead log is removed before the fresh database is
var buf: [path_buf_len]u8 = undefined;
const path = try f.pathZ(&buf, "querylog.db");
try stampUserVersion(path, querylog_schema.fingerprint +% 1);
try f.write("querylog.db", "not a database at all");
// Existence alone proves nothing: the fresh database turns WAL on again and
// writes its own `-wal`. The marker is what distinguishes the stale file
@@ -503,7 +518,7 @@ test "S7 case 6: a stale write-ahead log is removed before the fresh database is
var result = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer result.database.close();
try testing.expectEqual(
querylog_schema.RecreateReason.fingerprint_mismatch,
querylog_schema.RecreateReason.not_a_database,
result.recreated.?,
);
@@ -591,7 +606,7 @@ test "S7 case 23: a locked querylog propagates Busy and is never destroyed" {
defer reopened.database.close();
try testing.expectEqual(@as(?querylog_schema.RecreateReason, null), reopened.recreated);
try testing.expectEqual(
@as(i64, querylog_schema.fingerprint),
@as(i64, querylog_versions.current_version),
try reopened.database.queryInt("PRAGMA user_version"),
);
+88
View File
@@ -0,0 +1,88 @@
-- FROZEN FIXTURE. Representative content for a querylog.db at schema version 1,
-- loaded on top of `querylog-v1-schema.sql`.
--
-- IMMUTABLE once released, for the same reason as its schema half: the release
-- gate byte-compares it against the previous tag. A future schema version ships
-- a NEW pair rather than editing this one.
--
-- The rows are chosen to be hard on a migration rather than realistic: every
-- `route_kind`, the NULL variants of `qtype`, `cache_hit`, `response_time_us`,
-- `upstream` and `forward_zone`, a non-IN qclass with a non-zero rcode, the
-- group and source id/name pairs, `cname_target`/`safe_search_target`, an
-- `available_since` that has been advanced away from its DDL default, and
-- timestamps that straddle two 30-minute projection buckets.
--
-- The `bucket_*` rows below are the recomputation of the raw rows, transcribed
-- from the same SQL `queries_repo`'s coherence oracle recomputes with. A
-- fixture-validity test runs that oracle over this file BEFORE any migration,
-- so an incoherent transcription fails on its own rather than as a migration
-- bug.
UPDATE querylog_meta SET created_at = 1699998000, available_since = 1699998600 WHERE id = 1;
INSERT INTO domains (id, domain) VALUES
(1, 'ads.example'),
(2, 'news.example'),
(3, 'chat.example'),
(4, 'printer.lan'),
(5, 'nas.lan'),
(6, 'cdn.example');
INSERT INTO query_log (
id, timestamp, domain_id, client_ip, qtype, blocked, response_time_us,
cache_hit, upstream, qclass, rcode, group_id, group_name, policy_action,
policy_reason, matched, source_id, source_name, cname_target,
safe_search_target, route_kind, forward_zone
) VALUES
(1, 1699999260, 1, '10.0.0.1', 1, 1, NULL, 0, NULL, 1, 0, 7, 'kids',
'block', 'blocklist_domain', 'ads.example', 3, 'stevenblack', NULL, NULL,
'blocked', NULL),
(2, 1699999320, 2, '10.0.0.1', 28, 0, 1500, 0, '9.9.9.9:853', 1, 0, NULL,
NULL, 'allow', 'no_match', NULL, NULL, NULL, NULL, NULL, 'upstream', NULL),
(3, 1699999380, 2, '10.0.0.2', 1, 0, 90, 1, NULL, 1, 0, NULL, NULL,
'allow', 'no_match', NULL, NULL, NULL, NULL, NULL, 'cache', NULL),
(4, 1699999440, 3, '10.0.0.2', NULL, 0, NULL, NULL, NULL, 3, 4, NULL, NULL,
'not_evaluated', 'non_in_class', NULL, NULL, NULL, NULL, NULL, 'rejected',
NULL),
(5, 1699999500, 4, '10.0.0.3', 1, 0, 200, 0, NULL, 1, 0, NULL, NULL,
'not_evaluated', 'local_record', NULL, NULL, NULL, NULL, NULL, 'local',
NULL),
(6, 1700000700, 5, '10.0.0.3', 15, 0, 3400, 0, NULL, 1, 0, NULL, NULL,
'not_evaluated', 'forward_zone', NULL, NULL, NULL, NULL, NULL,
'forward_zone', 'lan.example'),
(7, 1700001060, 1, '10.0.0.1', 1, 1, NULL, 0, NULL, 1, 0, 7, 'kids',
'block', 'blocklist_wildcard', '*.ads.example', 3, 'stevenblack', NULL,
NULL, 'blocked', NULL),
(8, 1700001120, 6, '10.0.0.4', 65, 0, 2500, 1, '1.1.1.1:853', 1, 0, NULL,
NULL, 'allow', 'no_match', NULL, NULL, NULL, 'edge.cdn.example',
'forcesafesearch.example', 'upstream', NULL);
INSERT INTO bucket_totals (bucket, queries, blocked, cached, rt_sum, rt_count) VALUES
(1699999200, 6, 1, 1, 5190, 4),
(1700001000, 2, 1, 1, 2500, 1);
INSERT INTO bucket_clients (bucket, client_ip, queries) VALUES
(1699999200, '10.0.0.1', 2),
(1699999200, '10.0.0.2', 2),
(1699999200, '10.0.0.3', 2),
(1700001000, '10.0.0.1', 1),
(1700001000, '10.0.0.4', 1);
-- qtype -1 is the lossless encoding of the NULL qtype on row 4.
INSERT INTO bucket_types (bucket, qtype, count) VALUES
(1699999200, -1, 1),
(1699999200, 1, 3),
(1699999200, 15, 1),
(1699999200, 28, 1),
(1700001000, 1, 1),
(1700001000, 65, 1);
INSERT INTO bucket_routes (bucket, route_kind, source_present, source_text, count) VALUES
(1699999200, 'blocked', 0, '', 1),
(1699999200, 'cache', 0, '', 1),
(1699999200, 'forward_zone', 1, 'lan.example', 1),
(1699999200, 'local', 0, '', 1),
(1699999200, 'rejected', 0, '', 1),
(1699999200, 'upstream', 1, '9.9.9.9:853', 1),
(1700001000, 'blocked', 0, '', 1),
(1700001000, 'upstream', 1, '1.1.1.1:853', 1);
+88
View File
@@ -0,0 +1,88 @@
-- FROZEN FIXTURE. querylog.db schema version 1: byte-for-byte the `ddl` text of
-- `src/storage/querylog_schema.zig`, which is the schema the 0.0.12 and 0.0.13
-- binaries created and the one version 1 names.
--
-- IMMUTABLE once released. The release gate byte-compares this file against the
-- previous tag and fails the cut on any edit, because it is the starting point
-- every future migration is proved against: editing it would prove a migration
-- against a file no operator ever had. A new schema version ships a NEW pair.
--
-- The `PRAGMA user_version` stamp is deliberately NOT part of this file. The
-- loader applies it, which is what lets one fixture serve both the version-1
-- stamp and the 0.0.12/0.0.13 legacy fingerprint.
CREATE TABLE domains (
id INTEGER PRIMARY KEY,
domain TEXT NOT NULL UNIQUE
);
CREATE TABLE query_log (
id INTEGER PRIMARY KEY,
timestamp INTEGER NOT NULL,
domain_id INTEGER NOT NULL REFERENCES domains(id),
client_ip TEXT NOT NULL, -- text, not a FK: log rows are immutable facts
qtype INTEGER,
blocked INTEGER NOT NULL,
response_time_us INTEGER,
cache_hit INTEGER,
upstream TEXT,
qclass INTEGER NOT NULL,
rcode INTEGER NOT NULL,
group_id INTEGER, -- text/id pairs, not FKs: a renamed
group_name TEXT, -- group must not rewrite history
policy_action TEXT NOT NULL,
policy_reason TEXT NOT NULL,
matched TEXT,
source_id INTEGER,
source_name TEXT,
cname_target TEXT,
safe_search_target TEXT,
route_kind TEXT NOT NULL,
forward_zone TEXT,
CHECK (rcode BETWEEN 0 AND 4095) -- twelve bits (RFC 6891 6.1.3)
);
CREATE INDEX idx_query_log_ts ON query_log(timestamp);
CREATE INDEX idx_query_log_client ON query_log(client_ip);
CREATE INDEX idx_query_log_domain ON query_log(domain_id);
CREATE TABLE querylog_meta (
id INTEGER PRIMARY KEY CHECK (id = 1), -- one row, enforced by the schema
created_at INTEGER NOT NULL,
available_since INTEGER NOT NULL
);
INSERT INTO querylog_meta (id, created_at, available_since)
VALUES (1, unixepoch(), unixepoch() + 1);
CREATE TABLE bucket_totals (
bucket INTEGER PRIMARY KEY,
queries INTEGER NOT NULL,
blocked INTEGER NOT NULL,
cached INTEGER NOT NULL,
rt_sum INTEGER NOT NULL, -- sum(response_time_us) over timed rows
rt_count INTEGER NOT NULL -- count(response_time_us)
) WITHOUT ROWID;
CREATE TABLE bucket_clients (
bucket INTEGER NOT NULL,
client_ip TEXT NOT NULL,
queries INTEGER NOT NULL,
PRIMARY KEY (bucket, client_ip)
) WITHOUT ROWID;
CREATE TABLE bucket_types (
bucket INTEGER NOT NULL,
qtype INTEGER NOT NULL, -- -1 encodes a NULL qtype, losslessly
count INTEGER NOT NULL,
PRIMARY KEY (bucket, qtype)
) WITHOUT ROWID;
CREATE TABLE bucket_routes (
bucket INTEGER NOT NULL,
route_kind TEXT NOT NULL,
source_present INTEGER NOT NULL, -- 0: source NULL; 1: source = source_text
source_text TEXT NOT NULL, -- '' when source_present = 0
count INTEGER NOT NULL,
PRIMARY KEY (bucket, route_kind, source_present, source_text),
CHECK (source_present IN (0, 1)),
CHECK (source_present = 1 OR source_text = '')
) WITHOUT ROWID;

Some files were not shown because too many files have changed in this diff Show More