9 Commits
Author SHA1 Message Date
mokhtar 324704b53f build: bump version to 0.0.7
Gates / package (push) Successful in 41s
Gates / test (push) Successful in 1m35s
Gates / test-aarch64 (push) Successful in 6m48s
Gates / frontend (push) Successful in 1m22s
Gates / container (push) Successful in 9s
CI / gates (push) Successful in 23m18s
Gates / frontend (push) Successful in 1m8s
Gates / package (push) Successful in 40s
Release / guard (push) Successful in 17s
Gates / test (push) Successful in 1m35s
Gates / test-aarch64 (push) Successful in 6m55s
Gates / container (push) Successful in 9s
Release / gates (push) Successful in 9m23s
Release / publish (push) Successful in 4m26s
2026-08-20 22:34:13 +02:00
mokhtar addf24f92c query log batching: one transaction per flush interval, not per query
Gates / frontend (push) Successful in 1m18s
Gates / test (push) Successful in 2m46s
Gates / test-aarch64 (push) Successful in 7m33s
Gates / package (push) Successful in 5m34s
Gates / container (push) Successful in 17s
CI / gates (push) Successful in 16m16s
Gates / frontend (push) Successful in 1m8s
Gates / container (push) Successful in 9s
Release / gates (push) Successful in 9m15s
Release / guard (push) Successful in 19s
Gates / test (push) Successful in 1m34s
Gates / test-aarch64 (push) Successful in 6m46s
Gates / package (push) Successful in 39s
Release / publish (push) Failing after 4m7s
2026-08-20 20:57:11 +02:00
mokhtar 037f209179 milestone 27: diagnostics — operational failures land in one curated log, resolved history purgeable
Gates / frontend (push) Successful in 1m33s
Gates / test (push) Successful in 1m48s
Gates / test-aarch64 (push) Successful in 7m10s
Gates / package (push) Successful in 5m31s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 14m51s
2026-08-20 20:05:59 +02:00
mokhtar 3dd8214ef2 dashboard: a success rate never rounds up to 100.0% while failures stand
Gates / frontend (push) Successful in 1m27s
Gates / test (push) Successful in 1m53s
Gates / test-aarch64 (push) Successful in 7m7s
Gates / package (push) Successful in 3m57s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 13m15s
2026-08-19 17:54:02 +02:00
mokhtar 64c0d723a6 querylog: the aside file's name says why, so a schema change is not called corrupt 2026-08-17 20:24:21 +02:00
mokhtar 377f00a35f changelog: 0.0.6 releases today
Release / guard (push) Successful in 22s
Gates / frontend (push) Successful in 1m30s
Gates / test (push) Successful in 1m50s
Gates / frontend (push) Successful in 1m9s
Gates / test-aarch64 (push) Successful in 6m55s
Gates / test (push) Successful in 1m34s
Gates / test-aarch64 (push) Successful in 6m9s
Gates / package (push) Successful in 3m49s
Gates / package (push) Successful in 32s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 21m28s
Gates / container (push) Successful in 9s
Release / gates (push) Successful in 20m7s
Release / publish (push) Successful in 4m26s
2026-08-17 18:22:54 +02:00
mokhtar e0a7cd8a6b milestone 26: upstream health answers for the selected period 2026-08-17 18:21:56 +02:00
mokhtar 3ed9a57822 changelog: 0.0.5 releases today
Gates / test (push) Successful in 1m44s
Gates / frontend (push) Successful in 1m2s
Gates / test-aarch64 (push) Successful in 5m55s
Gates / package (push) Successful in 28s
Gates / container (push) Successful in 16s
CI / gates (push) Successful in 20m25s
Release / guard (push) Successful in 18s
Gates / frontend (push) Successful in 1m5s
Gates / test-aarch64 (push) Successful in 6m33s
Gates / test (push) Successful in 1m31s
Gates / package (push) Successful in 3m41s
Gates / container (push) Successful in 9s
Release / gates (push) Successful in 19m27s
Release / publish (push) Successful in 5m43s
2026-08-16 23:55:19 +02:00
mokhtar 0601098ab0 admin: drop the learned tag from the query tables 2026-08-16 23:55:10 +02:00
74 changed files with 12001 additions and 349 deletions
+39 -1
View File
@@ -4,7 +4,45 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
## [Unreleased]
## [0.0.7] - 2026-08-20
Operational failures get a page of their own, and the query log stops wearing out the disk it lives on: the deployed Pi was writing half a gigabyte a day to store two megabytes of query rows, one transaction per query. Both came out of running 0.0.6 on real hardware.
### Added
- **A diagnostics page.** Operational failures now land in one curated log instead of only journald: blocklist download failures, certificate reload failures, disk pressure, query-log writer and maintenance failures, upstream exchange and history failures, client tracking failures, listener and configuration problems at boot, and the query-log recreation an upgrade causes. One entry per failing subject — an entry opens on the first failure, counts repeats, and closes itself when the subject recovers; nothing needs dismissing. Each entry says what it means for the service and what to do about it. `GET /api/diagnostics` serves the log, `GET /api/health` reports the active counts and degrades while the diagnostics store itself cannot write, and `/metrics` gains `nxdns_diagnostics_active_warnings`, `nxdns_diagnostics_active_errors` and `nxdns_diagnostics_write_failures_total`. Resolved entries can be purged when you decide the history has served its purpose — one entry from its row or its detail page, or the whole resolved history at once with "Purge all resolved" (`DELETE /api/diagnostics/{id}` and `DELETE /api/diagnostics`). An entry that is still failing is the current state of the box, not history, so it has no purge action and the API answers 409.
### Changed
- **The query log commits once a minute instead of once a query.** The writer batched for 100 milliseconds, which at a household's query rate means almost every query got a transaction of its own — and a transaction costs the disk far more than the row it carries. On the deployed Pi that came to roughly 0.5 GiB of writes a day to store 2.3 MB of query rows, the kind of write volume that kills an SD card. The batch window is now `logging.query_log_flush_interval_s`: 60 seconds by default (the same minute Pi-hole's `DBinterval` defaults to, for the same reason), anything from 0 to 3600, editable on the settings page. Batches are still capped at 100 rows, so a burst is committed as soon as it fills one rather than waiting out the window, and the in-memory queue, its drop-oldest backpressure and retention are untouched. The price is two kinds of lag: a crash costs about one interval of query history — more if the writer was held back by a full disk or a slow write — and every query-log-backed view — the query-log page, the dashboard totals, the timeseries — is about one interval behind. The live page is not affected; it is fed before the queue. Set the key to `0` for the old write-immediately behavior.
### Fixed
- **Shutdown no longer races the last query rows to the disk.** The query-log writer was stopped by the same cancellation that stopped the DNS listeners, so whether the batch it was holding reached the database depended on which happened to land first, the cancellation or the queue closing. Shutdown now stops and joins the listeners and every other query producer first, then closes the queue, then waits for the writer to finish emptying it — the held batch and everything still queued get written. If free space is below the critical threshold and the disk monitor will not let that final write through, the rows are counted as dropped instead of holding the exit open indefinitely.
- **An upstream success rate no longer rounds up to 100.0% while failures stand.** One decimal place cannot hold 12,696 successes out of 12,698 attempts: it rounded to `100.0%`, so the row claimed perfect reliability next to a failure count of 2. Neither end of the scale is reachable by rounding any more — `100.0%` needs an actual absence of failures and `0.0%` an actual absence of successes, and a rate a hair off either end shows `99.9%` or `0.1%` instead.
- **A query log set aside by a schema change is no longer named `corrupt`.** Every recreate wrote the old file to `querylog.db.corrupt-<unix seconds>`, whatever sent it there — including the fingerprint mismatch an upgrade causes, where the file is a healthy database this build simply cannot read. The name is the only account of the reason that outlives the log line, so it read as an accusation and invited operators to delete an intact file. The name now says which of the four cases it hit: `querylog.db.corrupt-…`, `.not-a-database-…`, `.quick-check-failed-…` or `.schema-changed-…`. The 0.0.6 upgrade produces `schema-changed`. Nothing else about the recreate changed, and no existing aside file is renamed.
## [0.0.6] - 2026-08-17
The period picker now scopes the whole dashboard. The upstream table was the last widget that ignored it, and fixing that meant recording upstream outcomes over time instead of counting them since boot. Read the query-log note below before you upgrade.
### Added
- Four metrics for the new upstream-history recorder: `nxdns_upstream_history_flushes_total`, `nxdns_upstream_history_flush_failures_total`, `nxdns_upstream_history_rows_dropped_total` and the `nxdns_upstream_history_pending` gauge. While a flush to the database keeps failing, `GET /api/health` reports `degraded`; it recovers on the next flush that succeeds.
### Changed
- **Upstream health answers for the selected period.** The dashboard's upstream table used to print counters accumulated since process start beside a success rate taken over the last 32 exchanges, which is how "63 failures" and "100.0% success rate" ended up in the same row under a period picker that scoped nothing there. Every upstream outcome is now aggregated into its wall-clock minute and written to `querylog.db`, and `GET /api/upstream/health?period=…` serves the selected window: attempts, failures, success rate, and the last failure with its error name, all inside the period, from 31 days of history. A window with no attempts reports no success rate at all instead of a perfect one, and the table shows an em-dash. The in-memory health state that drives failover and backoff is unchanged, as are its `/metrics` series.
- **`GET /api/upstream/health` changed shape.** Gone from each upstream: `consecutive_failures`, `total_successes`, `total_failures`, the last-32 `success_rate`, `last_error` and `last_error_age_s`. Each upstream keeps `url`, `enabled` and `available` and gains a `period` object with the ranged numbers; the body gains `period`, `since`, `until` and a `complete` flag that says whether any outcome was known to be dropped inside the window. The removed counters are still exported by `/metrics` under their existing names. On the dashboard the "Right now" section is gone with them: the upstream table rejoined the ranged part of the page, and the disk card, the one live widget left, is titled "Storage now".
- **The query log is recreated on upgrade.** Recording upstream history added two tables to the `querylog.db` schema, and its fingerprint check refuses a database that does not match the shipped definition. On first start this version renames the existing `querylog.db` aside as `querylog.db.corrupt-<unix seconds>` in the data directory and creates a fresh one, so query history and stats restart empty. The renamed file is left in place rather than deleted, so removing it is your call. `config.db` is untouched: no configuration is lost.
## [0.0.5] - 2026-08-16
One rendering fix on the 0.0.4 feature, caught the day it shipped.
### Changed
- The query tables no longer repeat the *learned* tag on every row: in the live page and the query log a learned name is just muted, with the address still in the row's tooltip. The clients page keeps the tag, where it appears once per client and says something.
## [0.0.4] - 2026-08-16
+41 -3
View File
@@ -325,7 +325,7 @@ Per-group boolean. Rewrites known engine domains to their safe-search CNAME targ
- Schemes: `https://…` → DoH, `tls://host:853` → DoT.
- Ordered by priority; sequential attempt; per-upstream failure counters; exponential backoff with jitter; success resets.
- `UpstreamHealth` per upstream: last_success_at, last_error_at, last_error_message, rolling success rate, consecutive failures, backoff-until. Exposed via `GET /api/upstream/health`, dashboard, `/metrics`, and `nxdns check`.
- `UpstreamHealth` per upstream: last_success_at, last_error_at, last_error_message, rolling success rate, consecutive failures, backoff-until. This is routing state: it drives failover and backoff, and is exposed through `/metrics` and `nxdns check`. `GET /api/upstream/health?period=…` exposes none of it except the live `enabled`/`available` pair; its counts, success rate and last failure are ranged aggregates read from the per-minute upstream history in `querylog.db`, so the dashboard's period scopes them like every other number on the page.
- DoH client: `std.http.Client` with `content-type/accept: application/dns-message`; strict status + payload checks.
- `platform/tls_client.zig` enforces per-connection read/write deadlines, classifies TLS errors explicitly, retries with backoff. Integration tests cover timeout/hang scenarios so compiler upgrades can't silently regress them.
- Connect, read, and total-budget timeouts each configurable.
@@ -435,8 +435,28 @@ CREATE TABLE forward_zones (
);
CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT NOT NULL);
CREATE TABLE operational_events (
id INTEGER PRIMARY KEY,
code TEXT NOT NULL,
subject_key TEXT NOT NULL,
subject_label TEXT NOT NULL,
severity TEXT NOT NULL CHECK (severity IN ('warning', 'error')),
first_seen INTEGER NOT NULL,
last_seen INTEGER NOT NULL,
occurrences INTEGER NOT NULL CHECK (occurrences > 0),
resolved_at INTEGER,
detail TEXT NOT NULL DEFAULT '',
CHECK (resolved_at IS NULL OR resolved_at >= first_seen)
);
CREATE UNIQUE INDEX idx_operational_events_active
ON operational_events(code, subject_key) WHERE resolved_at IS NULL;
CREATE INDEX idx_operational_events_last_seen
ON operational_events(last_seen DESC);
```
`operational_events` is the one table here that is **not** configuration. It is the diagnostics log of `src/storage/events.zig`: one row per failure episode, opened on the first failure and resolved when the same subject succeeds again. It is deliberately absent from `config_schema.table_names` and `config_schema.delete_order`, so `nxdns export` never emits it and `nxdns import` never wipes it.
### 11.3 querylog.db Schema
```sql
@@ -460,12 +480,30 @@ CREATE TABLE query_log (
CREATE INDEX idx_query_log_ts ON query_log(timestamp);
CREATE INDEX idx_query_log_client ON query_log(client_ip);
CREATE INDEX idx_query_log_domain ON query_log(domain_id);
CREATE TABLE upstream_targets (
id INTEGER PRIMARY KEY,
url TEXT NOT NULL UNIQUE -- the historical identity: config.db ids cannot cross database files
);
CREATE TABLE upstream_minute (
upstream_id INTEGER NOT NULL REFERENCES upstream_targets(id),
minute_ts INTEGER NOT NULL,
successes INTEGER NOT NULL,
failures INTEGER NOT NULL,
last_failure_ts INTEGER,
last_error TEXT,
PRIMARY KEY (upstream_id, minute_ts),
CHECK (successes >= 0),
CHECK (failures >= 0)
) WITHOUT ROWID;
CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
```
### 11.4 Query Logger
- In-memory buffer, mutex guarded, hard cap `query_log_buffer_max` (default 10000).
- Flush: batch size (default 100) or max interval (default 100ms).
- Flush: batch size (100, comptime) or max interval `query_log_flush_interval_s` (default 60s, 03600, 0 = do not wait). One transaction per interval: at household query rates a per-query commit costs orders of magnitude more disk writes than the rows are worth. The interval is also roughly what a crash costs, while the writer is healthy and the disk gate is open — a gated or lock-delayed batch is older, so it is a normal case, not a bound.
- Privacy transforms (hide_domains / hide_client_ips) applied before persist + SSE fanout.
- Backpressure: buffer full → drop oldest unflushed entry, increment monotonic `queries_dropped` (exposed in `/api/health` + `/metrics`). SSE fanout precedes buffer insert, so live viewers still see dropped-from-persistence entries.
@@ -522,7 +560,7 @@ Scalars in `settings(key, value)`; ordered/structured items in dedicated tables.
- `GET /api/lookup?domain=…&group_id=…`
- `GET/POST /api/pause`
- `GET/PUT /api/settings`
- `GET /api/upstream/health`
- `GET /api/upstream/health?period=…`
- `POST /api/certs/reload`
- `GET /api/health` — overall + disk + upstream + queries_dropped rollup
- `GET /metrics` — Prometheus text exposition: query counters (total/blocked/cached), per-upstream health, cache stats, queries_dropped, disk gauges
+5 -1
View File
@@ -3,6 +3,11 @@
* with the same precedence the Clients page applies: a hand-typed `name` wins,
* the reverse-DNS `learned_name` stands in muted behind it, and an address with
* neither — including one the loaded list has never seen — stays bare.
*
* The muted colour is the whole of the affordance here. The Clients page pairs
* it with an outlined "learned" tag, and keeps it: one mention per client is
* information. Repeating that tag down every row of a query table is noise, so
* the tables carry the name alone.
*/
import { useMemo } from "react";
@@ -43,7 +48,6 @@ export function ClientName({ ip, names }: { ip: string; names: ClientNames }) {
return (
<span title={ip} {...stylex.props(shared.learnedName)}>
{client.learned_name}
<span {...stylex.props(shared.learnedTag)}>learned</span>
</span>
);
}
@@ -5,6 +5,9 @@ import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
/** Wall clock at import; the upstream fixtures date their failures against it. */
const NOW_S = Math.floor(Date.now() / 1000);
const RESPONSES: Record<string, unknown> = {
"/api/stats?period=24h": {
period: "24h",
@@ -58,32 +61,67 @@ const RESPONSES: Record<string, unknown> = {
writer_failed: false,
refreshes_gated: 0,
snapshot_generation: 3,
diagnostics: { state: "recording", active_warnings: 1, active_errors: 0 },
},
"/api/upstream/health": {
"/api/upstream/health?period=24h": {
period: "24h",
since: NOW_S - 86_400,
until: NOW_S,
available: 1,
total: 2,
complete: true,
upstreams: [
{
url: "https://dns.example/dns-query",
enabled: true,
available: false,
consecutive_failures: 4,
total_successes: 90,
total_failures: 10,
success_rate: 0.9,
last_error: "timeout",
period: {
attempts: 100,
successes: 90,
failures: 10,
success_rate: 0.9,
// 3h30m before the fixture's now, far from a unit boundary.
last_failure_at: NOW_S - 12_600,
last_failure_error: "timeout",
},
},
{
url: "udp://9.9.9.9:53",
enabled: true,
available: true,
consecutive_failures: 0,
total_successes: 100,
total_failures: 0,
success_rate: 1,
last_error: "",
period: {
attempts: 100,
successes: 100,
failures: 0,
success_rate: 1,
last_failure_at: null,
last_failure_error: null,
},
},
],
},
"/api/upstream/health?period=1h": {
period: "1h",
since: NOW_S - 3600,
until: NOW_S,
available: 1,
total: 2,
total: 1,
complete: true,
upstreams: [
{
url: "https://dns.example/dns-query",
enabled: true,
available: true,
period: {
attempts: 7,
successes: 6,
failures: 1,
success_rate: 6 / 7,
last_failure_at: NOW_S - 300,
last_failure_error: "timeout",
},
},
],
},
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
};
@@ -142,7 +180,7 @@ test("dashboard renders stats, chart, disk card, upstream table and health banne
expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
expect(screen.getByText("Blocked", { selector: "li" })).toBeTruthy();
expect(screen.getByText("Disk")).toBeTruthy();
expect(screen.getByText("Storage now")).toBeTruthy();
expect(screen.getByText("warn")).toBeTruthy();
expect(screen.getAllByText("400.0 MiB").length).toBeGreaterThan(0);
expect(screen.getByText("12.0 MiB")).toBeTruthy();
@@ -155,10 +193,32 @@ test("dashboard renders stats, chart, disk card, upstream table and health banne
expect(screen.getByText("https://dns.example/dns-query")).toBeTruthy();
expect(screen.getByText("90.0%")).toBeTruthy();
expect(screen.getByText("100.0%")).toBeTruthy();
expect(screen.getByText("timeout")).toBeTruthy();
expect(screen.getByText("timeout · 3h ago")).toBeTruthy();
expect(screen.getByText("1/2 available")).toBeTruthy();
});
test("live state is labeled on its own card, not by a section that disowns the picker", async () => {
renderDashboard();
await screen.findByRole("heading", { name: "Dashboard" });
expect(screen.getByText("Storage now")).toBeTruthy();
expect(screen.queryByRole("region", { name: "Right now" })).toBeNull();
expect(screen.queryByText("Right now")).toBeNull();
expect(screen.queryByText("Snapshot state; the period above does not apply.")).toBeNull();
});
test("the period picker rescopes the upstream table", async () => {
renderDashboard();
await screen.findByRole("heading", { name: "Dashboard" });
await screen.findByText("90.0%");
fireEvent.click(screen.getByRole("button", { name: "1h" }));
await screen.findByText("85.7%");
expect(screen.getByRole("columnheader", { name: "Selected period · 1h" })).toBeTruthy();
expect(screen.queryByText("90.0%")).toBeNull();
});
test("period picker refetches stats and shows the empty chart state", async () => {
renderDashboard();
await screen.findByRole("heading", { name: "Dashboard" });
@@ -173,7 +233,7 @@ test("period picker refetches stats and shows the empty chart state", async () =
});
test("one failing endpoint degrades its own widget on cold navigation", async () => {
failing.add("/api/upstream/health");
failing.add("/api/upstream/health?period=24h");
renderDashboard();
await screen.findByRole("heading", { name: "Dashboard" });
@@ -184,6 +244,6 @@ test("one failing endpoint degrades its own widget on cold navigation", async ()
expect(screen.getByText("1,000")).toBeTruthy();
expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
expect(screen.getByText("Disk")).toBeTruthy();
expect(screen.getByText("Storage now")).toBeTruthy();
expect(screen.queryByText("https://dns.example/dns-query")).toBeNull();
});
@@ -66,7 +66,7 @@ const styles = stylex.create({
"@media (prefers-color-scheme: dark)": "oklch(27.4% 0.006 286.033)",
},
},
/** The chart takes two thirds beside the disk card from `lg`, one column below. */
/** The chart takes two thirds beside the storage card from `lg`, one column below. */
panelGrid: {
display: "grid",
gap: "1rem",
@@ -123,7 +123,7 @@ export default function DashboardPage() {
const stats = useQuery({ ...statsQuery(period), placeholderData: keepPreviousData });
const timeseries = useQuery({ ...timeseriesQuery(period), placeholderData: keepPreviousData });
const health = useQuery(healthQuery());
const upstreamHealth = useQuery(upstreamHealthQuery());
const upstreamHealth = useQuery({ ...upstreamHealthQuery(period), placeholderData: keepPreviousData });
return (
<section {...stylex.props(styles.page)}>
+3 -1
View File
@@ -74,8 +74,10 @@ function stateStyle(state: Health["disk"]["state"]) {
export default function DiskCard({ disk }: { disk: Health["disk"] }) {
return (
<section {...stylex.props(styles.card)}>
{/* Live state, unlike the ranged widgets around it; the title says so
rather than a section rule the picker would have to disown. */}
<h2 {...stylex.props(styles.heading)}>
Disk
Storage now
<span {...stylex.props(styles.badge, stateStyle(disk.state))}>{disk.state}</span>
</h2>
<dl {...stylex.props(styles.list)}>
@@ -0,0 +1,180 @@
import { render, screen, within } from "@testing-library/react";
import type { UpstreamHealth, UpstreamHealthEntry, UpstreamPeriodStats } from "@/lib/types";
import UpstreamHealthTable from "./UpstreamHealthTable";
const NOW_S = 1_700_000_000;
beforeEach(() => {
vi.spyOn(Date, "now").mockReturnValue(NOW_S * 1000);
});
afterEach(() => {
vi.restoreAllMocks();
});
const ZERO: UpstreamPeriodStats = {
attempts: 0,
successes: 0,
failures: 0,
success_rate: null,
last_failure_at: null,
last_failure_error: null,
};
function period(overrides: Partial<UpstreamPeriodStats> = {}): UpstreamPeriodStats {
return {
attempts: 100,
successes: 90,
failures: 10,
success_rate: 0.9,
// 3h30m ago, far from a unit boundary.
last_failure_at: NOW_S - 12_600,
last_failure_error: "Timeout",
...overrides,
};
}
function entry(overrides: Partial<UpstreamHealthEntry> = {}): UpstreamHealthEntry {
return {
url: "https://dns.example/dns-query",
enabled: true,
available: true,
period: period(),
...overrides,
};
}
function renderTable(upstreams: UpstreamHealthEntry[], overrides: Partial<UpstreamHealth> = {}) {
const health: UpstreamHealth = {
period: "24h",
since: NOW_S - 86_400,
until: NOW_S,
available: upstreams.filter((upstream) => upstream.available).length,
total: upstreams.length,
complete: true,
upstreams,
...overrides,
};
render(<UpstreamHealthTable health={health} />);
}
function rowOf(url: string): HTMLElement {
const cell = screen.getByText(url);
const row = cell.closest("tr");
if (row === null) throw new Error(`no row for ${url}`);
return row;
}
test("the ranged columns sit under a header naming the selected period", () => {
renderTable([entry()]);
expect(screen.getByRole("columnheader", { name: "Selected period · 24h" })).toBeTruthy();
for (const name of ["Upstream", "Status now", "Attempts", "Failures", "Success rate", "Last failure"]) {
expect(screen.getByRole("columnheader", { name })).toBeTruthy();
}
// The unranged yes/no pair the ranged table replaced.
expect(screen.queryByRole("columnheader", { name: "Enabled" })).toBeNull();
expect(screen.queryByRole("columnheader", { name: "Available" })).toBeNull();
});
test("status now is one word from live state, not from the window", () => {
renderTable([
entry({ url: "https://a.example/dns-query" }),
entry({ url: "https://b.example/dns-query", available: false }),
entry({ url: "https://c.example/dns-query", enabled: false, available: false }),
]);
expect(within(rowOf("https://a.example/dns-query")).getByText("Available")).toBeTruthy();
expect(within(rowOf("https://b.example/dns-query")).getByText("Backing off")).toBeTruthy();
expect(within(rowOf("https://c.example/dns-query")).getByText("Disabled")).toBeTruthy();
});
test("last failure pairs the error name with its age, em-dash when the window holds none", () => {
renderTable([
entry({ url: "https://a.example/dns-query" }),
entry({
url: "https://b.example/dns-query",
period: period({ last_failure_at: null, last_failure_error: null }),
}),
]);
expect(within(rowOf("https://a.example/dns-query")).getByText("Timeout · 3h ago")).toBeTruthy();
expect(within(rowOf("https://b.example/dns-query")).getByText("—")).toBeTruthy();
});
test("a window with no attempts renders em-dashes and never a perfect rate", () => {
renderTable([entry({ period: ZERO })]);
const cells = within(rowOf("https://dns.example/dns-query")).getAllByRole("cell");
expect(cells.map((cell) => cell.textContent)).toEqual([
"https://dns.example/dns-query",
"Available",
"0",
"0",
"—",
"—",
]);
expect(screen.queryByText("100.0%")).toBeNull();
expect(screen.queryByText("0.0%")).toBeNull();
});
test("the card says so when every upstream was idle in the window", () => {
renderTable([entry({ url: "https://a.example/dns-query", period: ZERO }), entry({ period: ZERO })]);
expect(screen.getByText("No upstream attempts in this period.")).toBeTruthy();
});
test("one upstream with attempts keeps the idle message away", () => {
renderTable([entry({ url: "https://a.example/dns-query", period: ZERO }), entry()]);
expect(screen.queryByText("No upstream attempts in this period.")).toBeNull();
});
test("an incomplete window carries a note; a complete one claims nothing", () => {
renderTable([entry()], { complete: false });
expect(screen.getByText(/history incomplete/i)).toBeTruthy();
});
test("a complete window shows no completeness text at all", () => {
renderTable([entry()], { complete: true });
expect(screen.queryByText(/history incomplete/i)).toBeNull();
expect(screen.queryByText(/complete/i)).toBeNull();
});
test("an empty pool says so instead of drawing a table", () => {
renderTable([]);
expect(screen.getByText("No upstreams configured.")).toBeTruthy();
expect(screen.queryByRole("table")).toBeNull();
});
test("a rate a hair under perfect never rounds up to 100.0% while failures stand", () => {
// The real row that produced this: 12,698 attempts, 2 failures, 99.984%.
renderTable([
entry({
period: period({ attempts: 12_698, successes: 12_696, failures: 2, success_rate: 12_696 / 12_698 }),
}),
]);
expect(screen.queryByText("100.0%")).toBeNull();
expect(screen.getByText("99.9%")).toBeTruthy();
});
test("a rate a hair above nothing never rounds down to 0.0% while successes stand", () => {
renderTable([
entry({
period: period({ attempts: 12_698, successes: 2, failures: 12_696, success_rate: 2 / 12_698 }),
}),
]);
expect(screen.queryByText("0.0%")).toBeNull();
expect(screen.getByText("0.1%")).toBeTruthy();
});
test("a window with no failures at all still reads 100.0%", () => {
renderTable([entry({ period: period({ attempts: 500, successes: 500, failures: 0, success_rate: 1 }) })]);
expect(screen.getByText("100.0%")).toBeTruthy();
});
@@ -1,8 +1,11 @@
import * as stylex from "@stylexjs/stylex";
import type { UpstreamHealth } from "@/lib/types";
import { formatAge } from "@/lib/format";
import type { UpstreamHealth, UpstreamHealthEntry, UpstreamPeriodStats } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
const numberFormat = new Intl.NumberFormat();
const styles = stylex.create({
card: {
borderRadius: "0.25rem",
@@ -34,6 +37,13 @@ const styles = stylex.create({
lineHeight: "1.25rem",
color: colors.textMuted,
},
note: {
paddingInline: "1rem",
paddingBlock: "0.5rem",
fontSize: "0.75rem",
lineHeight: "1rem",
color: colors.textMuted,
},
tableWrap: {
overflowX: "auto",
},
@@ -43,6 +53,24 @@ const styles = stylex.create({
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
/**
* The two live columns are left outside the span: everything under it answers
* for the selected window, and nothing else on this card does.
*/
groupRow: {
fontSize: "0.75rem",
lineHeight: "1rem",
color: colors.textMuted,
},
groupHead: {
borderBottomWidth: 1,
borderBottomStyle: "solid",
borderBottomColor: colors.border,
paddingInline: "1rem",
paddingBottom: "0.25rem",
textAlign: "center",
fontWeight: 500,
},
headRow: {
borderBottomWidth: 1,
borderBottomStyle: "solid",
@@ -85,12 +113,45 @@ const styles = stylex.create({
},
});
function YesNo({ value, badValue }: { value: boolean; badValue: boolean }) {
const bad = value === badValue;
return <span {...stylex.props(bad && styles.bad)}>{value ? "yes" : "no"}</span>;
/** Live pool state in one word. Configuration first: a disabled upstream is not backing off. */
function statusNow(upstream: UpstreamHealthEntry): "Available" | "Backing off" | "Disabled" {
if (!upstream.enabled) return "Disabled";
return upstream.available ? "Available" : "Backing off";
}
/**
* `success_rate` is null exactly when the window holds no attempt, and that must
* not read as perfect reliability — hence the em-dash rather than `100.0%`.
*
* One decimal place cannot hold 12,696 of 12,698: it rounds to `100.0%`, and the
* row then claims perfection beside a failure count of 2. Neither endpoint may
* be reached by rounding — only by actually having no failure, or no success.
*/
function successRate(period: UpstreamPeriodStats): string {
if (period.success_rate === null) return "—";
const rounded = period.success_rate * 100;
if (rounded > 99.9 && period.failures > 0) return "99.9%";
if (rounded < 0.1 && period.successes > 0) return "0.1%";
return `${rounded.toFixed(1)}%`;
}
/**
* The age is formatted once, when the row renders; nothing here ticks. It is
* measured against the browser's clock rather than the response's `until`, so a
* cached response ages visibly instead of freezing at the moment it was served.
*/
function lastFailure(period: UpstreamPeriodStats, nowSeconds: number): string {
if (period.last_failure_at === null) return "—";
const age = formatAge(Math.max(0, nowSeconds - period.last_failure_at));
const error = period.last_failure_error;
return error === null || error === "" ? age : `${error} · ${age}`;
}
export default function UpstreamHealthTable({ health }: { health: UpstreamHealth }) {
const nowSeconds = Math.floor(Date.now() / 1000);
const idle = health.upstreams.length > 0 && health.upstreams.every(({ period }) => period.attempts === 0);
return (
<section {...stylex.props(styles.card)}>
<h2 {...stylex.props(styles.heading)}>
@@ -105,15 +166,21 @@ export default function UpstreamHealthTable({ health }: { health: UpstreamHealth
<div {...stylex.props(styles.tableWrap)}>
<table {...stylex.props(styles.table)}>
<thead>
<tr {...stylex.props(styles.groupRow)}>
<td colSpan={2} />
<th scope="colgroup" colSpan={4} {...stylex.props(styles.groupHead)}>
Selected period · {health.period}
</th>
</tr>
<tr {...stylex.props(styles.headRow)}>
<th scope="col" {...stylex.props(styles.th)}>
URL
Upstream
</th>
<th scope="col" {...stylex.props(styles.th)}>
Enabled
Status now
</th>
<th scope="col" {...stylex.props(styles.th)}>
Available
<th scope="col" {...stylex.props(styles.th, styles.thRight)}>
Attempts
</th>
<th scope="col" {...stylex.props(styles.th, styles.thRight)}>
Failures
@@ -122,35 +189,53 @@ export default function UpstreamHealthTable({ health }: { health: UpstreamHealth
Success rate
</th>
<th scope="col" {...stylex.props(styles.th)}>
Last error
Last failure
</th>
</tr>
</thead>
<tbody>
{health.upstreams.map((upstream) => (
<tr key={upstream.url} {...stylex.props(styles.row)}>
<td {...stylex.props(styles.cell, styles.small, shared.mono)}>{upstream.url}</td>
<td {...stylex.props(styles.cell)}>
<YesNo value={upstream.enabled} badValue={false} />
</td>
<td {...stylex.props(styles.cell)}>
<YesNo value={upstream.available} badValue={false} />
</td>
<td {...stylex.props(styles.cell, styles.cellRight, shared.tabularNums)}>
{upstream.total_failures}
</td>
<td {...stylex.props(styles.cell, styles.cellRight, shared.tabularNums)}>
{(upstream.success_rate * 100).toFixed(1)}%
</td>
<td {...stylex.props(styles.cell, styles.small, styles.muted)}>
{upstream.last_error || "—"}
</td>
</tr>
))}
{health.upstreams.map((upstream) => {
const status = statusNow(upstream);
return (
<tr key={upstream.url} {...stylex.props(styles.row)}>
<td {...stylex.props(styles.cell, styles.small, shared.mono)}>
{upstream.url}
</td>
<td {...stylex.props(styles.cell)}>
<span
{...stylex.props(
status === "Backing off" && styles.bad,
status === "Disabled" && styles.muted,
)}
>
{status}
</span>
</td>
<td {...stylex.props(styles.cell, styles.cellRight, shared.tabularNums)}>
{numberFormat.format(upstream.period.attempts)}
</td>
<td {...stylex.props(styles.cell, styles.cellRight, shared.tabularNums)}>
{numberFormat.format(upstream.period.failures)}
</td>
<td {...stylex.props(styles.cell, styles.cellRight, shared.tabularNums)}>
{successRate(upstream.period)}
</td>
<td {...stylex.props(styles.cell, styles.small, styles.muted)}>
{lastFailure(upstream.period, nowSeconds)}
</td>
</tr>
);
})}
</tbody>
</table>
</div>
)}
{idle && <p {...stylex.props(styles.note)}>No upstream attempts in this period.</p>}
{!health.complete && (
<p {...stylex.props(styles.note)}>
History incomplete: outcomes were dropped in this window, so these counts are a lower bound.
</p>
)}
</section>
);
}
@@ -0,0 +1,218 @@
import { cleanup, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import { DIAGNOSTIC_CODES, type DiagnosticEvent } from "@/lib/types";
import { EVENT_COPY } from "./eventCopy";
const NOW_S = Math.floor(Date.now() / 1000);
function event(overrides: Partial<DiagnosticEvent> = {}): DiagnosticEvent {
return {
id: 42,
code: "blocklist.refresh",
component: "blocklist",
subject: "StevenBlack",
severity: "warning",
first_seen: NOW_S - 7200,
last_seen: NOW_S - 600,
occurrences: 4,
resolved_at: null,
detail: "download failed: ConnectionTimedOut",
...overrides,
};
}
/** A 204: what `DELETE /api/diagnostics/{id}` answers on a purge. */
const NO_CONTENT = Symbol("204");
let responses: Record<string, unknown>;
let requested: string[];
beforeEach(() => {
requested = [];
responses = {
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
};
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL, init?: { method?: string }) => {
const url = String(input);
const method = init?.method ?? "GET";
const key = method === "GET" ? url : `${method} ${url}`;
requested.push(key);
const payload = responses[key];
if (payload === undefined)
return new Response(JSON.stringify({ error: "no such event" }), {
status: 404,
headers: { "content-type": "application/json" },
});
if (payload === NO_CONTENT) return new Response(null, { status: 204 });
return new Response(JSON.stringify(payload), {
status: 200,
headers: { "content-type": "application/json" },
});
}),
);
});
afterEach(() => {
vi.unstubAllGlobals();
});
/**
* `retry` is off in the failure test: the shared client backs 5xx off for
* seconds, which the render assertions would sit through for nothing.
*/
function renderDetail(id: number, { retry = true } = {}) {
const queryClient = createQueryClient();
if (!retry) {
const defaults = queryClient.getDefaultOptions();
queryClient.setDefaultOptions({ ...defaults, queries: { ...defaults.queries, retry: false } });
}
const router = createAppRouter(createMemoryHistory({ initialEntries: [`/diagnostics/${id}`] }), queryClient);
render(
<AuthProvider>
<QueryClientProvider client={queryClient}>
<RouterProvider router={router} />
</QueryClientProvider>
</AuthProvider>,
);
return router;
}
test("an open episode shows its facts, its copy and the error the server sent", async () => {
responses["/api/diagnostics/42"] = event();
renderDetail(42);
await screen.findByRole("heading", { name: "Blocklist source failed to update" });
expect(screen.getByText("Warning")).toBeTruthy();
expect(screen.getByText("StevenBlack")).toBeTruthy();
expect(screen.getByText("Active for 2h")).toBeTruthy();
expect(screen.getByText("Not yet — still failing")).toBeTruthy();
expect(screen.getByText("4")).toBeTruthy();
expect(screen.getByText("blocklist.refresh")).toBeTruthy();
expect(screen.getByText(EVENT_COPY["blocklist.refresh"].impact)).toBeTruthy();
expect(screen.getByText(EVENT_COPY["blocklist.refresh"].remediation)).toBeTruthy();
expect(screen.getByText("download failed: ConnectionTimedOut")).toBeTruthy();
expect(screen.getByRole("link", { name: "Go to Blocklists" }).getAttribute("href")).toBe("/blocklists");
});
test("a resolved episode states how long it lasted, not how long it has run", async () => {
responses["/api/diagnostics/7"] = event({ id: 7, resolved_at: NOW_S - 3600 });
renderDetail(7);
await screen.findByRole("heading", { name: "Blocklist source failed to update" });
expect(screen.getByText("Resolved after 1h")).toBeTruthy();
expect(screen.queryByText("Not yet — still failing")).toBeNull();
});
test("an open episode offers no purge", async () => {
responses["/api/diagnostics/42"] = event();
renderDetail(42);
await screen.findByRole("heading", { name: "Blocklist source failed to update" });
expect(screen.queryByRole("button", { name: "Purge" })).toBeNull();
});
test("purging a resolved episode asks first, then returns to the list", async () => {
responses["/api/diagnostics/7"] = event({ id: 7, resolved_at: NOW_S - 3600 });
responses["DELETE /api/diagnostics/7"] = NO_CONTENT;
responses["/api/diagnostics?state=active"] = { events: [], next_before: null, active: { warnings: 0, errors: 0 } };
responses["/api/diagnostics?state=resolved"] = {
events: [],
next_before: null,
active: { warnings: 0, errors: 0 },
};
const router = renderDetail(7);
await screen.findByRole("heading", { name: "Blocklist source failed to update" });
fireEvent.click(screen.getByRole("button", { name: "Purge" }));
const dialog = await screen.findByRole("alertdialog");
expect(dialog.textContent).toContain("Purge this resolved event? Its history is gone for good.");
fireEvent.click(within(dialog).getByRole("button", { name: "Cancel" }));
expect(requested).not.toContain("DELETE /api/diagnostics/7");
fireEvent.click(screen.getByRole("button", { name: "Purge" }));
fireEvent.click(within(await screen.findByRole("alertdialog")).getByRole("button", { name: "Purge" }));
await waitFor(() => expect(requested).toContain("DELETE /api/diagnostics/7"));
// The row it was showing no longer exists, so the page it navigates to is
// the list rather than a 404 of its own.
await waitFor(() => expect(router.state.location.pathname).toBe("/diagnostics"));
});
test("a refused purge stays on the event and shows why", async () => {
responses["/api/diagnostics/7"] = event({ id: 7, resolved_at: NOW_S - 3600 });
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL, init?: { method?: string }) => {
if (init?.method === "DELETE")
return new Response(JSON.stringify({ error: "the event is still active" }), {
status: 409,
headers: { "content-type": "application/json" },
});
return new Response(JSON.stringify(responses[String(input)] ?? {}), {
status: 200,
headers: { "content-type": "application/json" },
});
}),
);
const router = renderDetail(7, { retry: false });
await screen.findByRole("heading", { name: "Blocklist source failed to update" });
fireEvent.click(screen.getByRole("button", { name: "Purge" }));
fireEvent.click(within(await screen.findByRole("alertdialog")).getByRole("button", { name: "Purge" }));
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("the event is still active");
expect(router.state.location.pathname).toBe("/diagnostics/7");
});
test("every code renders its own title, impact and remediation", async () => {
for (const [index, code] of DIAGNOSTIC_CODES.entries()) {
const id = 100 + index;
responses[`/api/diagnostics/${id}`] = event({ id, code, component: code.slice(0, code.indexOf(".")) });
renderDetail(id);
const copy = EVENT_COPY[code];
await screen.findByRole("heading", { name: copy.title });
expect(screen.getByText(copy.impact), code).toBeTruthy();
expect(screen.getByText(copy.remediation), code).toBeTruthy();
screen.getByText(code);
cleanup();
}
});
test("an event retention has removed shows the server's message, not an empty page", async () => {
renderDetail(999);
await screen.findByText("no such event");
expect(screen.getByRole("link", { name: "← All diagnostics" })).toBeTruthy();
});
test("an unavailable store reports the failure instead of loading forever", async () => {
responses["/api/diagnostics/42"] = event();
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) =>
String(input).startsWith("/api/diagnostics/")
? new Response(JSON.stringify({ error: "store unavailable" }), {
status: 503,
headers: { "content-type": "application/json" },
})
: new Response(JSON.stringify(responses[String(input)] ?? {}), {
status: 200,
headers: { "content-type": "application/json" },
}),
),
);
renderDetail(42, { retry: false });
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("The server is starting or degraded.");
expect(screen.queryByText("Loading event…")).toBeNull();
expect(screen.getByRole("link", { name: "← All diagnostics" })).toBeTruthy();
});
@@ -0,0 +1,226 @@
import { useState } from "react";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Link, useNavigate, useParams } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import InlineError from "@/lib/InlineError";
import { formatDuration, formatTime } from "@/lib/format";
import { diagnosticPurgeMutation, diagnosticQuery } from "@/lib/queries";
import ConfirmDialog from "@/ui/ConfirmDialog";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import SeverityBadge from "./SeverityBadge";
import { componentLabel, copyFor } from "./eventCopy";
const styles = stylex.create({
back: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.primaryOnSurface,
textDecorationLine: "none",
},
headingRow: {
marginTop: "0.5rem",
display: "flex",
alignItems: "center",
flexWrap: "wrap",
gap: "0.5rem",
},
heading: {
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
},
purgeAction: {
marginInlineStart: "auto",
},
subject: {
marginTop: "0.25rem",
color: colors.textSecondary,
wordBreak: "break-all",
},
panel: {
marginTop: "1rem",
maxWidth: "48rem",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
backgroundColor: colors.surfaceRaised,
padding: "1rem",
},
facts: {
display: "grid",
gap: "0.5rem 1rem",
gridTemplateColumns: {
default: "auto",
"@media (min-width: 640px)": "max-content 1fr",
},
margin: 0,
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
term: {
color: colors.textMuted,
},
value: {
margin: 0,
},
sectionHeading: {
marginTop: "1.5rem",
fontSize: "1.125rem",
lineHeight: "1.75rem",
fontWeight: 600,
},
prose: {
marginTop: "0.5rem",
maxWidth: "48rem",
fontSize: "0.875rem",
lineHeight: "1.5rem",
},
detail: {
marginTop: "0.5rem",
maxWidth: "48rem",
overflowX: "auto",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
padding: "0.75rem",
fontSize: "0.8125rem",
lineHeight: "1.25rem",
whiteSpace: "pre-wrap",
wordBreak: "break-all",
},
links: {
marginTop: "1rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
link: {
color: colors.primaryOnSurface,
},
loading: {
marginTop: "1rem",
color: colors.textMuted,
},
});
export default function DiagnosticDetailPage() {
const { id } = useParams({ from: "/shell/diagnostics/$id" });
const eventId = Number(id);
const { data, error, isPending, refetch } = useQuery(diagnosticQuery(eventId));
const navigate = useNavigate();
const queryClient = useQueryClient();
const purge = useMutation(diagnosticPurgeMutation(queryClient));
const [confirming, setConfirming] = useState(false);
function confirmPurge() {
setConfirming(false);
// The row this page is about is gone, so staying here would show the
// 404 the purge itself caused.
purge.mutate(eventId, { onSuccess: () => void navigate({ to: "/diagnostics" }) });
}
if (isPending) {
return (
<p {...stylex.props(styles.loading, shared.pulse)} role="status">
Loading event
</p>
);
}
if (data === undefined) {
return (
<section>
<Link to="/diagnostics" {...stylex.props(styles.back, shared.focusRing)}>
All diagnostics
</Link>
<InlineError error={error} onRetry={() => void refetch()} />
</section>
);
}
const copy = copyFor(data.code);
const resolvedAt = data.resolved_at;
const span = (resolvedAt ?? Math.floor(Date.now() / 1000)) - data.first_seen;
return (
<section>
<Link to="/diagnostics" {...stylex.props(styles.back, shared.focusRing)}>
All diagnostics
</Link>
<div {...stylex.props(styles.headingRow)}>
<h1 {...stylex.props(styles.heading)}>{copy.title}</h1>
<SeverityBadge severity={data.severity} />
{/* Only history can be purged: an open episode is the current state of the box. */}
{resolvedAt !== null && (
<button
type="button"
onClick={() => setConfirming(true)}
disabled={purge.isPending}
{...stylex.props(styles.purgeAction, shared.dangerLinkButton, shared.focusRing)}
>
Purge
</button>
)}
</div>
<p {...stylex.props(styles.subject)}>{data.subject}</p>
<InlineError error={purge.error} />
<div {...stylex.props(styles.panel)}>
<dl {...stylex.props(styles.facts)}>
<dt {...stylex.props(styles.term)}>State</dt>
<dd {...stylex.props(styles.value)}>
{resolvedAt === null
? `Active for ${formatDuration(span)}`
: `Resolved after ${formatDuration(span)}`}
</dd>
<dt {...stylex.props(styles.term)}>First seen</dt>
<dd {...stylex.props(styles.value)}>{formatTime(data.first_seen)}</dd>
<dt {...stylex.props(styles.term)}>Last seen</dt>
<dd {...stylex.props(styles.value)}>{formatTime(data.last_seen)}</dd>
<dt {...stylex.props(styles.term)}>Occurrences</dt>
<dd {...stylex.props(styles.value, shared.tabularNums)}>{data.occurrences}</dd>
<dt {...stylex.props(styles.term)}>Resolved</dt>
<dd {...stylex.props(styles.value)}>
{data.resolved_at === null ? "Not yet — still failing" : formatTime(data.resolved_at)}
</dd>
<dt {...stylex.props(styles.term)}>Component</dt>
<dd {...stylex.props(styles.value)}>{componentLabel(data.component)}</dd>
<dt {...stylex.props(styles.term)}>Code</dt>
<dd {...stylex.props(styles.value, shared.mono)}>{data.code}</dd>
</dl>
</div>
<h2 {...stylex.props(styles.sectionHeading)}>Impact</h2>
<p {...stylex.props(styles.prose)}>{copy.impact}</p>
<h2 {...stylex.props(styles.sectionHeading)}>What to do</h2>
<p {...stylex.props(styles.prose)}>{copy.remediation}</p>
<h2 {...stylex.props(styles.sectionHeading)}>Last error</h2>
{data.detail === "" ? (
<p {...stylex.props(styles.prose)}>The server recorded no error text for this event.</p>
) : (
<pre {...stylex.props(styles.detail, shared.mono)}>{data.detail}</pre>
)}
{copy.link !== undefined && (
<p {...stylex.props(styles.links)}>
<Link to={copy.link.to} {...stylex.props(styles.link, shared.focusRing)}>
Go to {copy.link.label}
</Link>
</p>
)}
<ConfirmDialog
isOpen={confirming}
title="Purge event"
message="Purge this resolved event? Its history is gone for good."
confirmLabel="Purge"
onConfirm={confirmPurge}
onCancel={() => setConfirming(false)}
/>
</section>
);
}
@@ -0,0 +1,339 @@
import { fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import type { DiagnosticEvent, DiagnosticsPage } from "@/lib/types";
// Ages are rendered against the wall clock, so the fixtures are anchored to it
// rather than to a frozen instant: faking time here would fight the query
// client's own timers for no gain.
const NOW_S = Math.floor(Date.now() / 1000);
function event(id: number, overrides: Partial<DiagnosticEvent> = {}): DiagnosticEvent {
return {
id,
code: "blocklist.refresh",
component: "blocklist",
subject: "StevenBlack",
severity: "warning",
first_seen: NOW_S - 3600,
last_seen: NOW_S - 300,
occurrences: 3,
resolved_at: null,
detail: "download failed: ConnectionTimedOut",
...overrides,
};
}
function page(events: DiagnosticEvent[], nextBefore: number | null = null): DiagnosticsPage {
return { events, next_before: nextBefore, active: { warnings: 1, errors: 1 } };
}
const ACTIVE = page([
event(42),
event(41, {
code: "upstream.exchange",
component: "upstream",
subject: "tls://dns.example:853",
severity: "error",
occurrences: 1,
}),
]);
const RESOLVED = page([
event(30, { code: "disk.space", component: "disk", subject: "data", resolved_at: NOW_S - 7200 }),
]);
/** A stubbed response that carries a non-200 status instead of a payload. */
class Failure {
constructor(
readonly status: number,
readonly body: unknown,
) {}
}
function fail(status: number, message: string): Failure {
return new Failure(status, { error: message });
}
/** A 204: what `DELETE /api/diagnostics/{id}` answers on a purge. */
const NO_CONTENT = Symbol("204");
let responses: Record<string, unknown>;
let requested: string[];
beforeEach(() => {
requested = [];
responses = {
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
"/api/diagnostics?state=active": ACTIVE,
"/api/diagnostics?state=resolved": RESOLVED,
};
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL, init?: { method?: string }) => {
const url = String(input);
const method = init?.method ?? "GET";
// Reads stay keyed by url alone, so the assertions below read as the
// request line they are; writes carry their method.
const key = method === "GET" ? url : `${method} ${url}`;
requested.push(key);
const payload = responses[key];
if (payload === undefined) return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
if (payload === NO_CONTENT) return new Response(null, { status: 204 });
if (payload instanceof Failure) {
return new Response(JSON.stringify(payload.body), {
status: payload.status,
headers: { "content-type": "application/json" },
});
}
return new Response(JSON.stringify(payload), {
status: 200,
headers: { "content-type": "application/json" },
});
}),
);
});
afterEach(() => {
vi.unstubAllGlobals();
});
/**
* `retry` is off in the failure tests: the shared client backs 5xx off for
* seconds, which the render assertions would sit through for nothing.
*/
function renderRoute(path = "/diagnostics", { retry = true } = {}) {
const queryClient = createQueryClient();
if (!retry) {
const defaults = queryClient.getDefaultOptions();
queryClient.setDefaultOptions({ ...defaults, queries: { ...defaults.queries, retry: false } });
}
const router = createAppRouter(createMemoryHistory({ initialEntries: [path] }), queryClient);
render(
<AuthProvider>
<QueryClientProvider client={queryClient}>
<RouterProvider router={router} />
</QueryClientProvider>
</AuthProvider>,
);
return router;
}
/** A RAC Select names its trigger with the current value and then the label. */
function trigger(label: string): HTMLElement {
return screen.getByRole("button", { name: new RegExp(`${label}$`) });
}
async function pick(label: string, option: string) {
fireEvent.click(trigger(label));
fireEvent.click(await screen.findByRole("option", { name: option }));
await waitFor(() => expect(screen.queryByRole("listbox")).toBeNull());
}
test("active episodes come first, each with its title, subject, age and count", async () => {
renderRoute();
await screen.findByRole("heading", { name: "Diagnostics" });
const active = screen.getByText("Blocklist source failed to update").closest("li")!;
expect(within(active).getByText("Warning")).toBeTruthy();
expect(within(active).getByText("StevenBlack")).toBeTruthy();
expect(within(active).getByText(/Active for 1h · 3 occurrences/)).toBeTruthy();
const failing = screen.getByText("Upstream failing").closest("li")!;
expect(within(failing).getByText("Error")).toBeTruthy();
expect(within(failing).getByText(/1 occurrence(?!s)/)).toBeTruthy();
// The resolved history is a separate section, below the active list.
const table = within(screen.getByRole("table"));
expect(table.getByText("Disk space low")).toBeTruthy();
expect(screen.getByText(/Showing 1 resolved entry — end of history/)).toBeTruthy();
});
test("nothing open reads as good news, not as a broken page", async () => {
responses["/api/diagnostics?state=active"] = page([]);
renderRoute();
await screen.findByRole("heading", { name: "Diagnostics" });
const healthy = await screen.findByText("No active operational issues.");
expect(healthy.getAttribute("role")).toBe("status");
// Quiet: no alert anywhere on the page, and no empty table standing in.
expect(screen.queryByRole("alert")).toBeNull();
});
test("a filter lands in the url and refetches both sections through it", async () => {
responses["/api/diagnostics?severity=error&state=active"] = page([
event(41, { code: "upstream.exchange", component: "upstream", severity: "error" }),
]);
responses["/api/diagnostics?severity=error&state=resolved"] = page([]);
const router = renderRoute();
await screen.findByRole("heading", { name: "Diagnostics" });
await pick("Severity", "Errors");
await waitFor(() => expect(router.state.location.search).toEqual({ severity: "error" }));
await waitFor(() => expect(screen.queryByText("Blocklist source failed to update")).toBeNull());
expect(requested).toContain("/api/diagnostics?severity=error&state=active");
expect(requested).toContain("/api/diagnostics?severity=error&state=resolved");
});
test("the state filter hides the section it excludes", async () => {
const router = renderRoute();
await screen.findByRole("heading", { name: "Diagnostics" });
await pick("Show", "Active only");
await waitFor(() => expect(router.state.location.search).toEqual({ state: "active" }));
expect(screen.queryByRole("heading", { name: "Resolved" })).toBeNull();
expect(screen.getByRole("heading", { name: "Active" })).toBeTruthy();
});
test("a url written by hand starts on the filters it names", async () => {
responses["/api/diagnostics?component=disk&state=resolved"] = RESOLVED;
renderRoute("/diagnostics?state=resolved&component=disk");
await screen.findByRole("heading", { name: "Diagnostics" });
await screen.findByText("Disk space low");
expect(screen.queryByRole("heading", { name: "Active" })).toBeNull();
expect(requested).toContain("/api/diagnostics?component=disk&state=resolved");
});
test("load more appends the next page of resolved history", async () => {
responses["/api/diagnostics?state=resolved"] = page(
[event(30, { code: "disk.space", component: "disk", subject: "data", resolved_at: NOW_S - 7200 })],
30,
);
responses["/api/diagnostics?state=resolved&before=30"] = page([
event(12, {
code: "certificate.reload",
component: "certificate",
subject: "doh",
resolved_at: NOW_S - 90_000,
}),
]);
renderRoute();
await screen.findByText("Disk space low");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await screen.findByText("TLS certificate reload failed");
expect(screen.getByText(/Showing 2 resolved entries — end of history/)).toBeTruthy();
});
test("an unavailable store reports the failure instead of loading forever", async () => {
responses["/api/diagnostics?state=active"] = fail(503, "store unavailable");
renderRoute("/diagnostics", { retry: false });
await screen.findByRole("heading", { name: "Diagnostics" });
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("The server is starting or degraded.");
expect(screen.queryByText("Loading diagnostics…")).toBeNull();
// The resolved section answered, so it still renders its own history.
expect(screen.getByText("Disk space low")).toBeTruthy();
});
test("a failed history query reports the failure and retries on demand", async () => {
responses["/api/diagnostics?state=resolved"] = fail(500, "diagnostics store read failed");
renderRoute("/diagnostics", { retry: false });
await screen.findByRole("heading", { name: "Diagnostics" });
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("diagnostics store read failed");
expect(screen.queryByText("Loading history…")).toBeNull();
responses["/api/diagnostics?state=resolved"] = RESOLVED;
fireEvent.click(screen.getByRole("button", { name: "Retry" }));
await screen.findByText("Disk space low");
expect(screen.queryByRole("alert")).toBeNull();
});
test("only the resolved history offers a purge", async () => {
renderRoute();
await screen.findByRole("heading", { name: "Diagnostics" });
// An episode still failing is the state of the box, not history: no purge
// affordance anywhere on its card.
const active = screen.getByText("Blocklist source failed to update").closest("li")!;
expect(within(active).queryByRole("button", { name: "Purge" })).toBeNull();
const row = screen.getByText("Disk space low").closest("tr")!;
expect(within(row).getByRole("button", { name: "Purge" })).toBeTruthy();
expect(screen.getByRole("button", { name: "Purge all resolved" })).toBeTruthy();
});
test("with no resolved history there is nothing to purge in bulk", async () => {
responses["/api/diagnostics?state=resolved"] = page([]);
renderRoute();
await screen.findByRole("heading", { name: "Diagnostics" });
await screen.findByText("Nothing has failed and recovered in the retained window.");
expect(screen.queryByRole("button", { name: "Purge all resolved" })).toBeNull();
});
test("purging one row asks first, then sends the DELETE and refetches the lists", async () => {
responses["DELETE /api/diagnostics/30"] = NO_CONTENT;
renderRoute();
await screen.findByText("Disk space low");
fireEvent.click(within(screen.getByText("Disk space low").closest("tr")!).getByRole("button", { name: "Purge" }));
const dialog = await screen.findByRole("alertdialog");
expect(dialog.textContent).toContain("Purge this resolved event? Its history is gone for good.");
fireEvent.click(within(dialog).getByRole("button", { name: "Cancel" }));
expect(requested).not.toContain("DELETE /api/diagnostics/30");
fireEvent.click(within(screen.getByText("Disk space low").closest("tr")!).getByRole("button", { name: "Purge" }));
fireEvent.click(within(await screen.findByRole("alertdialog")).getByRole("button", { name: "Purge" }));
await waitFor(() => expect(requested).toContain("DELETE /api/diagnostics/30"));
// The invalidation covers both sections: the page the row left and the
// active list, whose `active` counts come from the same table.
await waitFor(() =>
expect(requested.filter((url) => url === "/api/diagnostics?state=resolved").length).toBeGreaterThan(1),
);
await waitFor(() =>
expect(requested.filter((url) => url === "/api/diagnostics?state=active").length).toBeGreaterThan(1),
);
});
test("purging the whole history asks first and sends one DELETE", async () => {
responses["DELETE /api/diagnostics"] = { purged: 1 };
renderRoute();
await screen.findByText("Disk space low");
fireEvent.click(screen.getByRole("button", { name: "Purge all resolved" }));
const dialog = await screen.findByRole("alertdialog");
expect(dialog.textContent).toContain("Purge all resolved events? Active events are kept.");
// What the server will answer once the purge has landed; the refetch the
// mutation triggers is what has to pick it up.
responses["/api/diagnostics?state=resolved"] = page([]);
fireEvent.click(within(dialog).getByRole("button", { name: "Purge all" }));
await waitFor(() => expect(requested).toContain("DELETE /api/diagnostics"));
await waitFor(() => expect(screen.queryByText("Disk space low")).toBeNull());
expect(screen.getByText("Blocklist source failed to update")).toBeTruthy();
});
test("a refused purge reports the server's reason and keeps the row", async () => {
responses["DELETE /api/diagnostics/30"] = fail(409, "the event is still active; it can be purged once it resolves");
renderRoute("/diagnostics", { retry: false });
await screen.findByText("Disk space low");
fireEvent.click(within(screen.getByText("Disk space low").closest("tr")!).getByRole("button", { name: "Purge" }));
fireEvent.click(within(await screen.findByRole("alertdialog")).getByRole("button", { name: "Purge" }));
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("the event is still active");
expect(screen.getByText("Disk space low")).toBeTruthy();
});
test("an episode links to its own detail page", async () => {
responses["/api/diagnostics/42"] = event(42);
renderRoute();
const link = await screen.findByRole("link", { name: "Blocklist source failed to update" });
expect(link.getAttribute("href")).toBe("/diagnostics/42");
});
@@ -0,0 +1,498 @@
import { useState } from "react";
import {
useInfiniteQuery,
useMutation,
useQueryClient,
type InfiniteData,
type UseInfiniteQueryResult,
} from "@tanstack/react-query";
import { Link, useNavigate, useSearch } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import * as api from "@/lib/api";
import InlineError from "@/lib/InlineError";
import { formatDuration, formatTime } from "@/lib/format";
import { diagnosticPurgeMutation, diagnosticsInfiniteQuery, diagnosticsPurgeResolvedMutation } from "@/lib/queries";
import type {
DiagnosticEvent,
DiagnosticSeverity,
DiagnosticState,
DiagnosticsFilter,
DiagnosticsPage as Page,
} from "@/lib/types";
import ConfirmDialog from "@/ui/ConfirmDialog";
import Select from "@/ui/Select";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import SeverityBadge from "./SeverityBadge";
import { DIAGNOSTIC_COMPONENTS, componentLabel, copyFor } from "./eventCopy";
const DARK = "@media (prefers-color-scheme: dark)";
const STATE_OPTIONS = [
{ value: "all", label: "Active and resolved" },
{ value: "active", label: "Active only" },
{ value: "resolved", label: "Resolved only" },
];
const SEVERITY_OPTIONS = [
{ value: "any", label: "Any severity" },
{ value: "warning", label: "Warnings" },
{ value: "error", label: "Errors" },
];
const COMPONENT_OPTIONS = [
{ value: "any", label: "All components" },
...DIAGNOSTIC_COMPONENTS.map((component) => ({ value: component, label: componentLabel(component) })),
];
const styles = stylex.create({
heading: {
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
},
intro: {
marginTop: "0.25rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
maxWidth: "48rem",
},
filterGrid: {
marginTop: "1rem",
display: "grid",
gap: "0.75rem",
gridTemplateColumns: {
default: "repeat(1, minmax(0, 1fr))",
"@media (min-width: 640px)": "repeat(3, minmax(0, 1fr))",
},
maxWidth: "48rem",
},
sectionHeading: {
marginTop: "1.5rem",
fontSize: "1.125rem",
lineHeight: "1.75rem",
fontWeight: 600,
},
sectionHeadingRow: {
display: "flex",
alignItems: "baseline",
flexWrap: "wrap",
justifyContent: "space-between",
gap: "0.75rem",
},
/**
* Nothing open is the normal state of a working install, so it gets one
* quiet muted line — no border, no icon, no alert role. A panel here would
* read as a broken page rather than as good news.
*/
healthy: {
marginTop: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textSecondary,
},
empty: {
marginTop: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
cardList: {
marginTop: "0.75rem",
display: "flex",
flexDirection: "column",
gap: "0.5rem",
listStyleType: "none",
padding: 0,
},
card: {
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
backgroundColor: colors.surfaceRaised,
paddingInline: "0.75rem",
paddingBlock: "0.625rem",
},
cardTop: {
display: "flex",
alignItems: "baseline",
flexWrap: "wrap",
gap: "0.5rem",
},
cardTitle: {
fontWeight: 500,
color: colors.primaryOnSurface,
textDecorationLine: "none",
},
subject: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textSecondary,
wordBreak: "break-all",
},
meta: {
marginTop: "0.25rem",
fontSize: "0.75rem",
lineHeight: "1rem",
color: colors.textMuted,
},
tableWrap: {
marginTop: "0.75rem",
overflowX: "auto",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
},
table: {
width: "100%",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
head: {
backgroundColor: { default: "oklch(98.5% 0 none)", [DARK]: "oklch(21% 0.006 285.885)" },
textAlign: "left",
},
th: {
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
fontWeight: 500,
color: colors.textSecondary,
whiteSpace: "nowrap",
},
row: {
borderTopWidth: { default: 1, ":first-child": 0 },
borderTopStyle: "solid",
borderTopColor: colors.border,
},
cell: {
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
},
nowrap: {
whiteSpace: "nowrap",
},
rowLink: {
color: colors.primaryOnSurface,
textDecorationLine: "none",
},
footer: {
marginTop: "0.75rem",
display: "flex",
alignItems: "center",
gap: "0.75rem",
},
note: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
moreError: {
marginTop: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.dangerText,
},
});
type Section = UseInfiniteQueryResult<InfiniteData<Page, unknown>, Error>;
/** The two enum filters, narrowed from the picker's string rather than cast. */
function asState(value: string): DiagnosticState | undefined {
return value === "active" || value === "resolved" ? value : undefined;
}
function asSeverity(value: string): DiagnosticSeverity | undefined {
return value === "warning" || value === "error" ? value : undefined;
}
function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
function occurrenceText(count: number): string {
return `${count} ${count === 1 ? "occurrence" : "occurrences"}`;
}
function rowsOf(section: Section): DiagnosticEvent[] {
return (section.data?.pages ?? []).flatMap((page) => page.events);
}
/**
* The cursor comes from the newest page on screen, not from `hasNextPage`:
* while placeholder data stands in for a filter change the query state is
* empty, and the button would flash away and back.
*/
function hasMore(section: Section): boolean {
const pages = section.data?.pages ?? [];
const last = pages[pages.length - 1];
return last !== undefined && last.next_before !== null;
}
function MoreButton({ section }: { section: Section }) {
const more = hasMore(section);
// A 401 is already redirecting via the cache-level handleUnauthorized.
const isUnauthorized = section.error instanceof api.ApiError && section.error.status === 401;
const failed = section.isFetchNextPageError && !isUnauthorized ? errorMessage(section.error) : null;
if (!more && failed === null) return null;
return (
<>
<div {...stylex.props(styles.footer)}>
{more && (
<button
type="button"
onClick={() => {
if (section.isFetchingNextPage || section.isPlaceholderData) return;
void section.fetchNextPage();
}}
disabled={section.isFetchingNextPage || section.isPlaceholderData}
{...stylex.props(shared.button, shared.focusRing)}
>
{section.isFetchingNextPage ? "Loading…" : "Load more"}
</button>
)}
</div>
{failed !== null && (
<p role="alert" {...stylex.props(styles.moreError)}>
Failed to load more: {failed}
</p>
)}
</>
);
}
function ActiveCard({ event, now }: { event: DiagnosticEvent; now: number }) {
const copy = copyFor(event.code);
return (
<li {...stylex.props(styles.card)}>
<div {...stylex.props(styles.cardTop)}>
<SeverityBadge severity={event.severity} />
<Link
to="/diagnostics/$id"
params={{ id: String(event.id) }}
{...stylex.props(styles.cardTitle, shared.focusRing)}
>
{copy.title}
</Link>
<span {...stylex.props(styles.subject)}>{event.subject}</span>
</div>
<p {...stylex.props(styles.meta)}>
Active for {formatDuration(now - event.first_seen)} · {occurrenceText(event.occurrences)} · last failure{" "}
{formatTime(event.last_seen)}
</p>
</li>
);
}
function HistoryRow({ event, onPurge, busy }: { event: DiagnosticEvent; onPurge: () => void; busy: boolean }) {
const copy = copyFor(event.code);
return (
<tr {...stylex.props(styles.row)}>
<td {...stylex.props(styles.cell)}>
<SeverityBadge severity={event.severity} />
</td>
<td {...stylex.props(styles.cell)}>
<Link
to="/diagnostics/$id"
params={{ id: String(event.id) }}
{...stylex.props(styles.rowLink, shared.focusRing)}
>
{copy.title}
</Link>
</td>
<td {...stylex.props(styles.cell)}>{event.subject}</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>{formatTime(event.first_seen)}</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>
{event.resolved_at === null ? "—" : formatTime(event.resolved_at)}
</td>
<td {...stylex.props(styles.cell, styles.nowrap, shared.tabularNums)}>{event.occurrences}</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>
<button
type="button"
onClick={onPurge}
disabled={busy}
{...stylex.props(shared.dangerLinkButton, shared.focusRing)}
>
Purge
</button>
</td>
</tr>
);
}
export default function DiagnosticsPage() {
const search = useSearch({ from: "/shell/diagnostics" });
const navigate = useNavigate({ from: "/diagnostics" });
const state = search.state ?? "all";
const base: DiagnosticsFilter = {};
if (search.severity !== undefined) base.severity = search.severity;
if (search.component !== undefined) base.component = search.component;
const active = useInfiniteQuery(diagnosticsInfiniteQuery({ ...base, state: "active" }, state !== "resolved"));
const history = useInfiniteQuery(diagnosticsInfiniteQuery({ ...base, state: "resolved" }, state !== "active"));
const queryClient = useQueryClient();
const purgeOne = useMutation(diagnosticPurgeMutation(queryClient));
const purgeAll = useMutation(diagnosticsPurgeResolvedMutation(queryClient));
// `null` is "no dialog"; the id is which row it is about, and `"all"` the
// whole history. One piece of state, so the two dialogs cannot both be open.
const [pendingPurge, setPendingPurge] = useState<number | "all" | null>(null);
const activeRows = rowsOf(active);
const historyRows = rowsOf(history);
const now = Math.floor(Date.now() / 1000);
const purging = purgeOne.isPending || purgeAll.isPending;
function setSearch(patch: Partial<typeof search>) {
void navigate({ search: (prev) => ({ ...prev, ...patch }) });
}
function confirmPurge() {
if (pendingPurge === null) return;
if (pendingPurge === "all") {
purgeAll.mutate();
} else {
purgeOne.mutate(pendingPurge);
}
setPendingPurge(null);
}
return (
<section>
<h1 {...stylex.props(styles.heading)}>Diagnostics</h1>
<p {...stylex.props(styles.intro)}>
Operational failures, one entry per subject that failed. An entry opens on the first failure, counts
repeats, and closes when the subject recovers.
</p>
<div {...stylex.props(styles.filterGrid)}>
<Select
variant="compactField"
label="Show"
value={state}
onChange={(value) => setSearch({ state: asState(value) })}
options={STATE_OPTIONS}
/>
<Select
variant="compactField"
label="Severity"
value={search.severity ?? "any"}
onChange={(value) => setSearch({ severity: asSeverity(value) })}
options={SEVERITY_OPTIONS}
/>
<Select
variant="compactField"
label="Component"
value={search.component ?? "any"}
onChange={(value) => setSearch({ component: value === "any" ? undefined : value })}
options={COMPONENT_OPTIONS}
/>
</div>
{state !== "resolved" && (
<>
<h2 {...stylex.props(styles.sectionHeading)}>Active</h2>
{active.status === "error" ? (
<InlineError error={active.error} onRetry={() => void active.refetch()} />
) : active.data === undefined ? (
<p {...stylex.props(styles.empty, shared.pulse)} role="status">
Loading diagnostics
</p>
) : activeRows.length === 0 ? (
<p {...stylex.props(styles.healthy)} role="status">
No active operational issues.
</p>
) : (
<>
<ul {...stylex.props(styles.cardList)}>
{activeRows.map((event) => (
<ActiveCard key={event.id} event={event} now={now} />
))}
</ul>
<MoreButton section={active} />
</>
)}
</>
)}
{state !== "active" && (
<>
<div {...stylex.props(styles.sectionHeading, styles.sectionHeadingRow)}>
<h2>Resolved</h2>
{historyRows.length > 0 && (
<button
type="button"
onClick={() => setPendingPurge("all")}
disabled={purging}
{...stylex.props(shared.dangerLinkButton, shared.focusRing)}
>
Purge all resolved
</button>
)}
</div>
{history.status === "error" ? (
<InlineError error={history.error} onRetry={() => void history.refetch()} />
) : history.data === undefined ? (
<p {...stylex.props(styles.empty, shared.pulse)} role="status">
Loading history
</p>
) : historyRows.length === 0 ? (
<p {...stylex.props(styles.empty)}>Nothing has failed and recovered in the retained window.</p>
) : (
<>
<div {...stylex.props(styles.tableWrap)}>
<table {...stylex.props(styles.table)}>
<thead {...stylex.props(styles.head)}>
<tr>
<th {...stylex.props(styles.th)}>Severity</th>
<th {...stylex.props(styles.th)}>Event</th>
<th {...stylex.props(styles.th)}>Subject</th>
<th {...stylex.props(styles.th)}>Started</th>
<th {...stylex.props(styles.th)}>Resolved</th>
<th {...stylex.props(styles.th)}>Occurrences</th>
<th {...stylex.props(styles.th)}>
<span {...stylex.props(shared.srOnly)}>Actions</span>
</th>
</tr>
</thead>
<tbody>
{historyRows.map((event) => (
<HistoryRow
key={event.id}
event={event}
busy={purging}
onPurge={() => setPendingPurge(event.id)}
/>
))}
</tbody>
</table>
</div>
<p {...stylex.props(styles.footer, styles.note)}>
Showing {historyRows.length} resolved {historyRows.length === 1 ? "entry" : "entries"}
{hasMore(history) ? "" : " — end of history"}
</p>
<MoreButton section={history} />
</>
)}
<InlineError error={purgeOne.error ?? purgeAll.error} />
</>
)}
<ConfirmDialog
isOpen={pendingPurge !== null}
title={pendingPurge === "all" ? "Purge resolved history" : "Purge event"}
message={
pendingPurge === "all"
? "Purge all resolved events? Active events are kept."
: "Purge this resolved event? Its history is gone for good."
}
confirmLabel={pendingPurge === "all" ? "Purge all" : "Purge"}
onConfirm={confirmPurge}
onCancel={() => setPendingPurge(null)}
/>
</section>
);
}
@@ -0,0 +1,42 @@
/**
* The severity chip both diagnostics views carry. The word is the affordance —
* colour alone would leave the severity unreadable to a screen reader and to
* anyone who does not separate the amber from the red.
*/
import * as stylex from "@stylexjs/stylex";
import type { DiagnosticSeverity } from "@/lib/types";
import { colors } from "@/ui/tokens.stylex";
const styles = stylex.create({
badge: {
display: "inline-block",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
paddingInline: "0.375rem",
paddingBlock: "0.125rem",
fontSize: "0.75rem",
lineHeight: "1rem",
fontWeight: 500,
whiteSpace: "nowrap",
},
warning: {
borderColor: colors.warnBorder,
backgroundColor: colors.warnSurface,
color: colors.warnText,
},
error: {
borderColor: colors.dangerBorder,
backgroundColor: colors.dangerSurface,
color: colors.dangerText,
},
});
export default function SeverityBadge({ severity }: { severity: DiagnosticSeverity }) {
return (
<span {...stylex.props(styles.badge, severity === "error" ? styles.error : styles.warning)}>
{severity === "error" ? "Error" : "Warning"}
</span>
);
}
@@ -0,0 +1,50 @@
import { DIAGNOSTIC_CODES } from "@/lib/types";
import { DIAGNOSTIC_COMPONENTS, EVENT_COPY, componentLabel, copyFor } from "./eventCopy";
test("the enum holds the fifteen codes the store defines", () => {
expect(DIAGNOSTIC_CODES).toHaveLength(15);
expect(new Set(DIAGNOSTIC_CODES).size).toBe(15);
});
test("every code has copy, and no copy belongs to a code that does not exist", () => {
for (const code of DIAGNOSTIC_CODES) {
const copy = EVENT_COPY[code];
expect(copy, code).toBeDefined();
expect(copy.title.length, code).toBeGreaterThan(0);
expect(copy.impact.length, code).toBeGreaterThan(0);
expect(copy.remediation.length, code).toBeGreaterThan(0);
}
expect(Object.keys(EVENT_COPY).sort()).toEqual([...DIAGNOSTIC_CODES].sort());
});
test("titles are distinct, so two open episodes never read as the same event", () => {
const titles = DIAGNOSTIC_CODES.map((code) => EVENT_COPY[code].title);
expect(new Set(titles).size).toBe(titles.length);
});
test("a code this build has never heard of falls back to the code itself", () => {
// The server is the authority on the enum; a newer one can send a sixteenth.
const copy = copyFor("nonsense.code" as (typeof DIAGNOSTIC_CODES)[number]);
expect(copy.title).toBe("nonsense.code");
expect(copy.remediation.length).toBeGreaterThan(0);
});
test("the component options are the code prefixes, deduplicated and in enum order", () => {
expect(DIAGNOSTIC_COMPONENTS).toEqual([
"disk",
"blocklist",
"certificate",
"query_log",
"upstream_history",
"upstream",
"client_names",
"clients",
"listener",
"configuration",
]);
});
test("component labels read as prose without inventing a name", () => {
expect(componentLabel("query_log")).toBe("Query log");
expect(componentLabel("disk")).toBe("Disk");
});
+155
View File
@@ -0,0 +1,155 @@
/**
* What each event code means to the operator, in three fixed fields: what the
* episode is (`title`), what it costs while it stays open (`impact`), and what
* to do about it (`remediation`). The server sends a code and an error string;
* every word of explanation the page shows comes from here.
*
* The record is exhaustive over `DiagnosticCode` by type, and a test walks
* `DIAGNOSTIC_CODES` to prove it at runtime too. A sixteenth code added to the
* enum fails `tsc` here before it can reach the page as a bare dotted string.
*
* `link` points at the configuration surface that governs the failure. Those
* are today's routes; the navigation restructure re-points them.
*/
import { DIAGNOSTIC_CODES, type DiagnosticCode } from "@/lib/types";
/** The literal paths keep `link.to` assignable to a typed router `Link`. */
export type CopyLinkPath = "/settings" | "/blocklists" | "/upstreams" | "/clients";
export interface EventCopy {
title: string;
impact: string;
remediation: string;
link?: { to: CopyLinkPath; label: string };
}
/**
* The copy for a code, with a floor under it. `tsc` proves the record covers
* the union, but a server one release ahead can send a code this build has
* never heard of; showing the raw code beats rendering "undefined".
*/
export function copyFor(code: DiagnosticCode): EventCopy {
return (
EVENT_COPY[code] ?? {
title: code,
impact: "This build has no description for this event code.",
remediation: "The error detail below is the whole of what the server reported.",
}
);
}
const SETTINGS = { to: "/settings", label: "Settings" } as const;
const BLOCKLISTS = { to: "/blocklists", label: "Blocklists" } as const;
const UPSTREAMS = { to: "/upstreams", label: "Upstreams" } as const;
const CLIENTS = { to: "/clients", label: "Clients" } as const;
/**
* The component filter's options, derived from the codes rather than listed
* again: the server matches `component` against the part of `code` before the
* dot, so any list written by hand here could drift from the enum.
*/
export const DIAGNOSTIC_COMPONENTS: readonly string[] = [
...new Set(DIAGNOSTIC_CODES.map((code) => code.slice(0, code.indexOf(".")))),
];
/** `query_log` → "Query log". Display only; the filter sends the raw component. */
export function componentLabel(component: string): string {
const spaced = component.replaceAll("_", " ");
return spaced.charAt(0).toUpperCase() + spaced.slice(1);
}
export const EVENT_COPY: Record<DiagnosticCode, EventCopy> = {
"disk.space": {
title: "Disk space low",
impact: "Below the critical threshold nxdns stops blocklist updates and query log flushes to protect the disk.",
remediation: "Free space on the data volume, or lower the retention window so the query log holds fewer days.",
link: SETTINGS,
},
"disk.probe": {
title: "Disk usage probe failed",
impact: "Free space is unknown, so the low-disk guard cannot act until a probe succeeds.",
remediation: "Check that the data and log directories exist and that the service user can read them.",
link: SETTINGS,
},
"blocklist.refresh": {
title: "Blocklist source failed to update",
impact: "The source keeps serving its last good snapshot, so blocking continues but the list ages.",
remediation: "Check the source url and the machine's internet access, then update the lists again.",
link: BLOCKLISTS,
},
"blocklist.snapshot": {
title: "Filter snapshot failed to publish",
impact: "The resolver keeps the snapshot it already holds; blocklist edits do not take effect until one publishes.",
remediation: "Check free disk space and the data directory's permissions, then update the lists again.",
link: BLOCKLISTS,
},
"blocklist.storage": {
title: "Blocklist storage operation failed",
impact: "Cached list files or their database rows are out of step; a later pass can redownload what is missing.",
remediation: "Check free disk space and the data directory's permissions.",
link: BLOCKLISTS,
},
"certificate.reload": {
title: "TLS certificate reload failed",
impact: "The endpoint keeps serving the certificate it already loaded, which expires on its own schedule.",
remediation:
"Check the certificate and key paths, and that renewal writes both files the service user can read.",
link: SETTINGS,
},
"query_log.write": {
title: "Query log write failed",
impact: "Queries are resolved and answered as usual, but they are not being recorded.",
remediation: "Check free disk space and the log database's permissions, then restart nxdns.",
link: SETTINGS,
},
"query_log.maintenance": {
title: "Query log maintenance failed",
impact: "Old rows are not being trimmed, so the log database grows past its retention window.",
remediation: "Check free disk space; the next maintenance pass retries on its own.",
link: SETTINGS,
},
"query_log.recreated": {
title: "Query log recreated",
impact: "The old log database was unreadable and was moved aside; the history it held is not in the new one.",
remediation: "Keep or delete the aside file named below. Nothing else is required — logging is running.",
link: SETTINGS,
},
"upstream_history.write": {
title: "Upstream history write failed",
impact: "Resolution is unaffected; the per-upstream success and failure aggregates lose the affected window.",
remediation: "Check free disk space and the configuration database's permissions.",
link: UPSTREAMS,
},
"upstream.exchange": {
title: "Upstream failing",
impact: "Queries fall through to the remaining upstreams; answers are slower while this one backs off.",
remediation: "Check the upstream's reachability and its TLS name. Remove it if it stays down.",
link: UPSTREAMS,
},
"client_names.storage": {
title: "Client name storage failed",
impact: "Learned reverse-DNS names are not persisted, so clients can show as bare addresses after a restart.",
remediation: "Check free disk space and the configuration database's permissions.",
link: CLIENTS,
},
"clients.storage": {
title: "Client record storage failed",
impact: "New clients may not appear in the list and stale ones may not be pruned.",
remediation: "Check free disk space and the configuration database's permissions.",
link: CLIENTS,
},
"listener.start": {
title: "Encrypted DNS listener failed to start",
impact: "That endpoint is not accepting queries. Plain DNS on port 53 is unaffected.",
remediation:
"Check the bind address, the port, and the certificate paths, then restart nxdns. The episode closes on a clean start.",
link: SETTINGS,
},
"configuration.load": {
title: "Configuration problem at startup",
impact: "The setting named below was rejected or replaced by its default for this run.",
remediation: "Correct the setting and restart nxdns. The episode closes on a clean start.",
link: SETTINGS,
},
};
+4 -7
View File
@@ -131,14 +131,11 @@ test("resolves each row's client to its display name, keeping the IP as the tool
expect(named.getAttribute("title")).toBe("192.0.2.10");
expect(screen.queryByText("pi.lan")).toBeNull();
// The cell holds the learned name followed by the tag, so the match is on
// the containing span rather than on a bare text node.
const learned = screen.getByText(
(content, element) => element?.tagName === "SPAN" && content.startsWith("laptop.lan"),
);
// A learned name reads muted and nothing more here: the "learned" tag would
// repeat on every row of the table, so the Clients page carries it instead.
const learned = screen.getByText("laptop.lan");
expect(learned.getAttribute("title")).toBe("192.0.2.11");
// The affordance is text, not colour, so a screen reader announces it too.
expect(within(learned).getByText("learned")).toBeTruthy();
expect(within(learned.closest("tr")!).queryByText("learned")).toBeNull();
// A known client with neither name, and a client the loaded list has never
// seen, both fall back to the bare address with no tooltip standing in.
@@ -137,14 +137,11 @@ test("resolves each row's client to its display name, keeping the IP as the tool
expect(named.getAttribute("title")).toBe("192.0.2.10");
expect(screen.queryByText("pi.lan")).toBeNull();
// The cell holds the learned name followed by the tag, so the match is on
// the containing span rather than on a bare text node.
const learned = screen.getByText(
(content, element) => element?.tagName === "SPAN" && content.startsWith("laptop.lan"),
);
// A learned name reads muted and nothing more here: the "learned" tag would
// repeat on every row of the table, so the Clients page carries it instead.
const learned = screen.getByText("laptop.lan");
expect(learned.getAttribute("title")).toBe("192.0.2.11");
// The affordance is text, not colour, so a screen reader announces it too.
expect(within(learned).getByText("learned")).toBeTruthy();
expect(within(learned.closest("tr")!).queryByText("learned")).toBeNull();
// A known client with neither name, and a client the loaded list has never
// seen, both fall back to the bare address with no tooltip standing in.
@@ -33,6 +33,7 @@ function baseSettings(): Settings {
level: "info",
retention_days: 30,
query_log_buffer_max: 10000,
query_log_flush_interval_s: 60,
hide_domains: false,
hide_client_ips: false,
output: "stderr",
@@ -116,6 +116,7 @@ const SECTIONS: readonly AnySectionDef[] = [
{ key: "level", kind: ["error", "warn", "info", "debug"] },
{ key: "retention_days", kind: "number" },
{ key: "query_log_buffer_max", kind: "number" },
{ key: "query_log_flush_interval_s", kind: "number" },
{ key: "hide_domains", kind: "boolean" },
{ key: "hide_client_ips", kind: "boolean" },
{ key: "output", kind: ["stderr", "syslog", "file"] },
@@ -38,6 +38,7 @@ function baseSettings(): Settings {
level: "info",
retention_days: 30,
query_log_buffer_max: 10000,
query_log_flush_interval_s: 60,
hide_domains: false,
hide_client_ips: false,
output: "stderr",
+20 -1
View File
@@ -6,6 +6,10 @@ import type {
ClientEdit,
ClientPrefix,
ClientPrefixInput,
DiagnosticEvent,
DiagnosticsFilter,
DiagnosticsPage,
DiagnosticsPurge,
ForwardZone,
ForwardZoneInput,
Group,
@@ -118,7 +122,22 @@ export const getStatsTimeseries = (period?: Period): Promise<StatsTimeseries> =>
export const getLookup = (domain: string, groupId?: number): Promise<LookupResult> =>
request(`/api/lookup${qs({ domain, group_id: groupId })}`);
export const getUpstreamHealth = (): Promise<UpstreamHealth> => request("/api/upstream/health");
export const getUpstreamHealth = (period?: Period): Promise<UpstreamHealth> =>
request(`/api/upstream/health${qs({ period })}`);
// Diagnostics
export const getDiagnostics = (filter: DiagnosticsFilter = {}): Promise<DiagnosticsPage> =>
request(`/api/diagnostics${qs({ ...filter })}`);
export const getDiagnostic = (id: number): Promise<DiagnosticEvent> => request(`/api/diagnostics/${id}`);
/** Purges one resolved event. An event still active answers 409, an unknown id 404. */
export const purgeDiagnostic = (id: number): Promise<void> => request(`/api/diagnostics/${id}`, { method: "DELETE" });
/** Purges the whole resolved history; active events are never touched. */
export const purgeResolvedDiagnostics = (): Promise<DiagnosticsPurge> =>
request("/api/diagnostics", { method: "DELETE" });
// Groups
+75 -5
View File
@@ -16,6 +16,9 @@ import type {
BlocklistEcho,
Client,
ClientPrefix,
DiagnosticEvent,
DiagnosticsPage,
DiagnosticsPurge,
ErrorEnvelope,
ForwardZone,
Group,
@@ -39,6 +42,11 @@ import type {
} from "@/lib/types";
export const sample_get_health: Health = {
diagnostics: {
active_errors: 0,
active_warnings: 0,
state: "recording",
},
disk: {
db_bytes: 0,
free_bytes: 0,
@@ -73,6 +81,57 @@ export const sample_logout: LogoutResponse = {
authenticated: false,
};
export const sample_get_diagnostics: DiagnosticsPage = {
active: {
errors: 0,
warnings: 0,
},
events: [
{
code: "upstream_history.write",
component: "upstream_history",
detail: "Busy",
first_seen: 0,
id: 0,
last_seen: 0,
occurrences: 0,
resolved_at: 0,
severity: "warning",
subject: "history",
},
{
code: "blocklist.refresh",
component: "blocklist",
detail: "download failed: ConnectionTimedOut",
first_seen: 0,
id: 0,
last_seen: 0,
occurrences: 0,
resolved_at: null,
severity: "warning",
subject: "StevenBlack",
},
],
next_before: null,
};
export const sample_get_diagnostic: DiagnosticEvent = {
code: "blocklist.refresh",
component: "blocklist",
detail: "download failed: ConnectionTimedOut",
first_seen: 0,
id: 0,
last_seen: 0,
occurrences: 0,
resolved_at: null,
severity: "warning",
subject: "StevenBlack",
};
export const sample_purge_diagnostics: DiagnosticsPurge = {
purged: 0,
};
export const sample_create_blocklist: BlocklistEcho = {
enabled: false,
id: 0,
@@ -333,16 +392,23 @@ export const sample_update_upstream: UpstreamEcho = {
export const sample_get_upstream_health: UpstreamHealth = {
available: 0,
complete: true,
period: "24h",
since: 0,
total: 0,
until: 0,
upstreams: [
{
available: true,
consecutive_failures: 0,
enabled: true,
last_error: "",
success_rate: 0,
total_failures: 0,
total_successes: 0,
period: {
attempts: 0,
failures: 0,
last_failure_at: null,
last_failure_error: null,
success_rate: null,
successes: 0,
},
url: "https://dns.example/dns-query",
},
],
@@ -491,6 +557,7 @@ export const sample_get_settings: SettingsEnvelope = {
"logging.level",
"logging.retention_days",
"logging.query_log_buffer_max",
"logging.query_log_flush_interval_s",
"logging.hide_domains",
"logging.hide_client_ips",
"logging.output",
@@ -552,6 +619,7 @@ export const sample_get_settings: SettingsEnvelope = {
max_size_mb: 0,
output: "stderr",
query_log_buffer_max: 0,
query_log_flush_interval_s: 0,
retention_days: 0,
},
upstream: {
@@ -614,6 +682,7 @@ export const sample_put_settings: SettingsEnvelope = {
"logging.level",
"logging.retention_days",
"logging.query_log_buffer_max",
"logging.query_log_flush_interval_s",
"logging.hide_domains",
"logging.hide_client_ips",
"logging.output",
@@ -675,6 +744,7 @@ export const sample_put_settings: SettingsEnvelope = {
max_size_mb: 0,
output: "stderr",
query_log_buffer_max: 0,
query_log_flush_interval_s: 0,
retention_days: 0,
},
upstream: {
+22 -1
View File
@@ -1,4 +1,4 @@
import { formatBytes, formatMicros, formatTime } from "@/lib/format";
import { formatAge, formatBytes, formatDuration, formatMicros, formatTime } from "@/lib/format";
test("formatTime renders unix seconds in the given locale and zone", () => {
// 2024-01-01T00:00:00Z; ICU emits U+202F before AM/PM in recent Node.
@@ -15,6 +15,27 @@ test("formatBytes humanizes with binary units", () => {
expect(formatBytes(2 * 1024 ** 4)).toBe("2.0 TiB");
});
test("formatAge steps up a unit at each boundary and truncates", () => {
expect(formatAge(0)).toBe("0s ago");
expect(formatAge(59)).toBe("59s ago");
expect(formatAge(60)).toBe("1m ago");
expect(formatAge(3599)).toBe("59m ago");
expect(formatAge(3600)).toBe("1h ago");
expect(formatAge(10800)).toBe("3h ago");
expect(formatAge(86399)).toBe("23h ago");
expect(formatAge(86400)).toBe("1d ago");
expect(formatAge(400000)).toBe("4d ago");
});
test("formatDuration is the same span without the 'ago', and never negative", () => {
expect(formatDuration(0)).toBe("0s");
expect(formatDuration(59)).toBe("59s");
expect(formatDuration(3600)).toBe("1h");
expect(formatDuration(86400)).toBe("1d");
// Clock skew between the server's timestamps and the browser's clock.
expect(formatDuration(-5)).toBe("0s");
});
test("formatMicros renders milliseconds with one decimal", () => {
expect(formatMicros(0)).toBe("0.0 ms");
expect(formatMicros(1234)).toBe("1.2 ms");
+30
View File
@@ -21,6 +21,36 @@ export function formatBytes(bytes: number): string {
return `${value.toFixed(1)} ${unit}`;
}
const AGE_UNITS = [
{ seconds: 86400, suffix: "d" },
{ seconds: 3600, suffix: "h" },
{ seconds: 60, suffix: "m" },
] as const;
/**
* Seconds of elapsed time → a coarse "3h ago". Truncating and single-unit on
* purpose: this labels a snapshot the caller renders once, so a reader must not
* take it for a live count. Nothing re-renders it as it ages.
*/
export function formatAge(seconds: number): string {
for (const unit of AGE_UNITS) {
if (seconds >= unit.seconds) return `${Math.floor(seconds / unit.seconds)}${unit.suffix} ago`;
}
return `${Math.floor(seconds)}s ago`;
}
/**
* Seconds of elapsed time → a coarse "3h", the same single truncated unit as
* `formatAge` without the "ago". For a span the caller labels itself, as in
* "active for 3h". A negative span reads "0s": clock skew is not a duration.
*/
export function formatDuration(seconds: number): string {
for (const unit of AGE_UNITS) {
if (seconds >= unit.seconds) return `${Math.floor(seconds / unit.seconds)}${unit.suffix}`;
}
return `${Math.max(0, Math.floor(seconds))}s`;
}
/** Microseconds → milliseconds with one decimal, e.g. 1234 → "1.2 ms". */
export function formatMicros(micros: number): string {
return `${(micros / 1000).toFixed(1)} ms`;
+51 -3
View File
@@ -5,6 +5,8 @@ import type {
BlocklistInput,
ClientEdit,
ClientPrefixInput,
DiagnosticsFilter,
DiagnosticsPage,
ForwardZoneInput,
GroupInput,
LocalRecordInput,
@@ -23,7 +25,11 @@ export const queryKeys = {
stats: (period: Period) => ["stats", period] as const,
timeseries: (period: Period) => ["stats", "timeseries", period] as const,
queriesInfinite: (filter: QueriesFilter) => ["queries", "infinite", filter] as const,
upstreamHealth: ["upstream-health"] as const,
diagnosticsInfinite: (filter: DiagnosticsFilter) => ["diagnostics", "infinite", filter] as const,
diagnostic: (id: number) => ["diagnostics", "event", id] as const,
/** Prefix of every diagnostics entry, page and detail alike; the purge target. */
diagnosticsAll: ["diagnostics"] as const,
upstreamHealth: (period: Period) => ["upstream-health", period] as const,
lookup: (domain: string, groupId?: number) => ["lookup", domain, groupId ?? null] as const,
/** Prefix of every `lookup` entry; the invalidation target after any verdict input changes. */
lookupAll: ["lookup"] as const,
@@ -69,8 +75,37 @@ export const queriesInfiniteQuery = (filter: QueriesFilter = {}) =>
placeholderData: keepPreviousData,
});
export const upstreamHealthQuery = () =>
queryOptions({ queryKey: queryKeys.upstreamHealth, queryFn: api.getUpstreamHealth, refetchInterval: 30_000 });
// Keyset pagination on `next_before`, exactly as the query log pages
// (handlers/diagnostics.zig copies the /api/queries contract). The active view
// polls on healthQuery's cadence because an episode opening is the same news a
// health banner carries; a resolved-history page is settled and does not poll.
// `enabled` belongs to the factory rather than to a spread at the call site:
// spreading the options object loses the page-param type, and the Diagnostics
// page turns one of its two sections off whenever a filter excludes it.
export const diagnosticsInfiniteQuery = (filter: DiagnosticsFilter = {}, enabled = true) =>
infiniteQueryOptions({
enabled,
queryKey: queryKeys.diagnosticsInfinite(filter),
queryFn: ({ pageParam }: { pageParam: number | undefined }) =>
api.getDiagnostics(pageParam === undefined ? filter : { ...filter, before: pageParam }),
initialPageParam: undefined as number | undefined,
getNextPageParam: (last: DiagnosticsPage) => last.next_before ?? undefined,
placeholderData: keepPreviousData,
refetchInterval: filter.state === "active" ? 10_000 : undefined,
});
export const diagnosticQuery = (id: number) =>
queryOptions({ queryKey: queryKeys.diagnostic(id), queryFn: () => api.getDiagnostic(id) });
// The period is part of the key: the upstream aggregates are ranged like the
// stats ones, so the picker has to refetch them rather than reuse a cached
// window under a new label.
export const upstreamHealthQuery = (period: Period = "24h") =>
queryOptions({
queryKey: queryKeys.upstreamHealth(period),
queryFn: () => api.getUpstreamHealth(period),
refetchInterval: 30_000,
});
export const lookupQuery = (domain: string, groupId?: number) =>
queryOptions({ queryKey: queryKeys.lookup(domain, groupId), queryFn: () => api.getLookup(domain, groupId) });
@@ -105,6 +140,19 @@ export const settingsQuery = () => queryOptions({ queryKey: queryKeys.settings,
// Group membership and names feed lookup verdicts and the group columns on
// clients, prefixes and rules, hence the wide invalidation on group mutations.
// Both purges invalidate the whole `diagnostics` prefix rather than one page
// key: the resolved list, the active list (whose `active` counts ride along) and
// the detail query of the row just deleted all describe the table that changed.
export const diagnosticPurgeMutation = (qc: QueryClient) => ({
mutationFn: (id: number) => api.purgeDiagnostic(id),
onSuccess: () => qc.invalidateQueries({ queryKey: queryKeys.diagnosticsAll }),
});
export const diagnosticsPurgeResolvedMutation = (qc: QueryClient) => ({
mutationFn: () => api.purgeResolvedDiagnostics(),
onSuccess: () => qc.invalidateQueries({ queryKey: queryKeys.diagnosticsAll }),
});
function invalidateGroupWorld(qc: QueryClient): Promise<unknown> {
return Promise.all([
qc.invalidateQueries({ queryKey: queryKeys.groups }),
+1
View File
@@ -25,6 +25,7 @@ function baseSettings(): Settings {
level: "info",
retention_days: 30,
query_log_buffer_max: 10000,
query_log_flush_interval_s: 60,
hide_domains: false,
hide_client_ips: false,
output: "stderr",
+107 -6
View File
@@ -28,6 +28,16 @@ export interface Health {
writer_failed: boolean;
refreshes_gated: number;
snapshot_generation: number | null;
/**
* The diagnostics store's own state, not a summary of what it holds:
* `unavailable` means the store is missing or its last write failed, so the
* counts below are the last ones it managed to observe.
*/
diagnostics: {
state: "recording" | "unavailable";
active_warnings: number;
active_errors: number;
};
}
export interface Version {
@@ -81,6 +91,77 @@ export interface QueriesFilter {
until?: number;
}
/**
* The fifteen operational event codes, in the order `src/storage/events.zig`
* declares them. A value, not only a type, because the copy map has to be
* proven exhaustive at runtime as well as by `tsc`.
*/
export const DIAGNOSTIC_CODES = [
"disk.space",
"disk.probe",
"blocklist.refresh",
"blocklist.snapshot",
"blocklist.storage",
"certificate.reload",
"query_log.write",
"query_log.maintenance",
"query_log.recreated",
"upstream_history.write",
"upstream.exchange",
"client_names.storage",
"clients.storage",
"listener.start",
"configuration.load",
] as const;
export type DiagnosticCode = (typeof DIAGNOSTIC_CODES)[number];
export type DiagnosticSeverity = "warning" | "error";
/** Which episodes a query selects. `all` is the server's default. */
export type DiagnosticState = "active" | "resolved" | "all";
export interface DiagnosticEvent {
id: number;
code: DiagnosticCode;
/** The part of `code` before the dot, repeated by the server for filtering. */
component: string;
/** The display identity of what failed; redacted where it derives from a url. */
subject: string;
severity: DiagnosticSeverity;
first_seen: number;
last_seen: number;
occurrences: number;
/** Null while the episode is still open. */
resolved_at: number | null;
detail: string;
}
export interface DiagnosticsPage {
events: DiagnosticEvent[];
next_before: number | null;
/** Episodes open right now, whatever this page filtered to. */
active: {
warnings: number;
errors: number;
};
}
/** `DELETE /api/diagnostics` — how many resolved events the purge removed. */
export interface DiagnosticsPurge {
purged: number;
}
export interface DiagnosticsFilter {
state?: DiagnosticState;
severity?: DiagnosticSeverity;
component?: string;
since?: number;
until?: number;
limit?: number;
before?: number;
}
export interface StatsTotals {
period: Period;
since: number;
@@ -119,21 +200,40 @@ export interface LookupResult {
safe_search_rewrite: string | null;
}
export interface UpstreamPeriodStats {
attempts: number;
successes: number;
failures: number;
/** successes/attempts, 0 to 1; null when attempts is 0 — no observations is not perfect reliability. */
success_rate: number | null;
/** The newest failure inside the window, unix seconds; null when the window holds none. */
last_failure_at: number | null;
/** The error name belonging to last_failure_at; null exactly when it is. */
last_failure_error: string | null;
}
export interface UpstreamHealthEntry {
url: string;
/** Live configuration, not history. */
enabled: boolean;
/** Live state; false while the upstream is backing off. */
available: boolean;
consecutive_failures: number;
total_successes: number;
total_failures: number;
success_rate: number;
last_error: string;
period: UpstreamPeriodStats;
}
export interface UpstreamHealth {
upstreams: UpstreamHealthEntry[];
period: Period;
since: number;
until: number;
available: number;
total: number;
/**
* No capacity drops known in this process within the selected window; up to about a minute of
* the newest outcomes may not have flushed yet, and outcomes lost in an unclean shutdown are
* not detectable.
*/
complete: boolean;
upstreams: UpstreamHealthEntry[];
}
export interface Group {
@@ -369,6 +469,7 @@ export interface Settings {
level: "error" | "warn" | "info" | "debug";
retention_days: number;
query_log_buffer_max: number;
query_log_flush_interval_s: number;
hide_domains: boolean;
hide_client_ips: boolean;
output: "stderr" | "syslog" | "file";
+51
View File
@@ -12,10 +12,13 @@ import {
import AppShell from "@/shell/AppShell";
import { ApiError } from "@/lib/api";
import { createQueryClient } from "@/lib/queryClient";
import type { DiagnosticSeverity, DiagnosticState, DiagnosticsFilter } from "@/lib/types";
import {
blocklistsQuery,
clientPrefixesQuery,
clientsQuery,
diagnosticQuery,
diagnosticsInfiniteQuery,
forwardZonesQuery,
groupsQuery,
healthQuery,
@@ -214,6 +217,52 @@ const lookupRoute = createRoute({
component: lazyRouteComponent(() => import("@/features/lookup/LookupPage")),
});
/**
* The three filters live in the url so an episode can be linked to as it was
* read. Anything else in the search object is dropped: an unknown value would
* reach the api as a query parameter the handler rejects with a 400.
*/
const diagnosticsRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/diagnostics",
validateSearch: (
search: Record<string, unknown>,
): { state?: DiagnosticState; severity?: DiagnosticSeverity; component?: string } => {
const state = search["state"];
const severity = search["severity"];
const component = search["component"];
return {
state: state === "active" || state === "resolved" ? state : undefined,
severity: severity === "warning" || severity === "error" ? severity : undefined,
component: typeof component === "string" && component !== "" ? component : undefined,
};
},
loaderDeps: ({ search }) => search,
// allSettled: the two sections render their own state, and the resolved
// history failing must not replace the active list with the error page.
loader: ({ context, deps }) => {
const base: DiagnosticsFilter = {};
if (deps.severity !== undefined) base.severity = deps.severity;
if (deps.component !== undefined) base.component = deps.component;
return Promise.allSettled([
context.queryClient.ensureInfiniteQueryData(diagnosticsInfiniteQuery({ ...base, state: "active" })),
context.queryClient.ensureInfiniteQueryData(diagnosticsInfiniteQuery({ ...base, state: "resolved" })),
]);
},
component: lazyRouteComponent(() => import("@/features/diagnostics/DiagnosticsPage")),
});
const diagnosticDetailRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/diagnostics/$id",
// Swallowed on purpose: an event retention has removed is a 404 the page
// itself explains, with the way back to the list. The whole-page error
// component would state it as a request failure instead.
loader: ({ context, params }) =>
context.queryClient.ensureQueryData(diagnosticQuery(Number(params.id))).catch(() => undefined),
component: lazyRouteComponent(() => import("@/features/diagnostics/DiagnosticDetailPage")),
});
const settingsRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/settings",
@@ -234,6 +283,8 @@ const routeTree = rootRoute.addChildren([
localDnsRoute,
upstreamsRoute,
lookupRoute,
diagnosticsRoute,
diagnosticDetailRoute,
settingsRoute,
]),
]);
+2
View File
@@ -16,6 +16,7 @@ const NAV_LABELS = [
"Local DNS",
"Upstreams",
"Lookup",
"Diagnostics",
"Settings",
];
@@ -39,6 +40,7 @@ const RESPONSES: Record<string, unknown> = {
writer_failed: false,
refreshes_gated: 0,
snapshot_generation: null,
diagnostics: { state: "recording", active_warnings: 0, active_errors: 0 },
},
"/api/upstream/health": { upstreams: [], available: 1, total: 1 },
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
+1
View File
@@ -26,6 +26,7 @@ const NAV_ITEMS = [
{ to: "/local-dns", label: "Local DNS" },
{ to: "/upstreams", label: "Upstreams" },
{ to: "/lookup", label: "Lookup" },
{ to: "/diagnostics", label: "Diagnostics" },
{ to: "/settings", label: "Settings" },
] as const;
+4 -2
View File
@@ -169,8 +169,10 @@ export const styles = stylex.create({
},
/**
* A learned name is runtime state, not something the operator typed, so it
* reads muted and carries an outlined "learned" tag. The tag is real text —
* a screen reader announces it — because colour alone is not an affordance.
* reads muted wherever it appears. The Clients page adds the outlined tag,
* where the client is named once; the query tables leave it off rather than
* repeat it on every row. The tag is real text — a screen reader announces
* it — because colour alone is not an affordance.
*/
learnedName: {
color: colors.textMuted,
+1 -1
View File
@@ -1,6 +1,6 @@
.{
.name = .nxdns,
.version = "0.0.4",
.version = "0.0.7",
.minimum_zig_version = "0.16.0",
.paths = .{""},
.fingerprint = 0x3307b311dded1d91,
+7 -1
View File
@@ -4,7 +4,7 @@ nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080), a
The machine-readable contract is `src/web/openapi.yaml`, which the running server hands out unauthenticated at `GET /api/openapi.yaml`. Request and response schemas for every operation live there. When this page and the YAML disagree, the YAML wins.
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 56 of its entries.
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 60 of its entries.
## Conventions
@@ -109,6 +109,10 @@ Auth `open` means no session is required; `session` means a valid session cookie
| GET | `/api/stats/timeseries` | session | counted | read | Bucketed counts for a period |
| GET | `/api/lookup` | session | counted | read | Explain a domain |
| GET | `/api/upstream/health` | session | counted | read | Upstream pool health |
| GET | `/api/diagnostics` | session | counted | read | Operational event log |
| DELETE | `/api/diagnostics` | session | counted | runtime action | Purge every resolved event |
| GET | `/api/diagnostics/{id}` | session | counted | read | One operational event |
| DELETE | `/api/diagnostics/{id}` | session | counted | runtime action | Purge one resolved event |
| GET | `/api/groups` | session | counted | read | List groups |
| POST | `/api/groups` | session | counted | config write | Create a group |
| GET | `/api/groups/{id}` | session | counted | read | Read a group |
@@ -156,6 +160,8 @@ Auth `open` means no session is required; `session` means a valid session cookie
There is no `POST /api/clients`: client rows come from DNS activity or import, never from the API.
The two diagnostics deletes purge history only. `DELETE /api/diagnostics/{id}` answers 204 for a resolved event, 409 for one that is still active — an open episode is the current state of the box, not history — and 404 for an id no row holds. `DELETE /api/diagnostics` removes every resolved event and answers `{"purged": N}`, leaving the active ones. Events still resolve on their own; these only decide when the resolved rows go.
Static assets are not routes. The router sends unmatched non-`/api` paths to the embedded SPA before any auth or rate-limit check.
## Settings keys
+15
View File
@@ -132,6 +132,7 @@ Process log and query log behavior.
| `logging.level` | enum `.err` \| `.warn` \| `.info` \| `.debug` | `.info` | — | one of the four tags; stored as `"error"` / `"warn"` / `"info"` / `"debug"` | log threshold (`src/platform/logging.zig`) |
| `logging.retention_days` | u16 | 30 | days | at least 1 | query-log pruning cutoff (`src/storage/retention.zig`) and the client tracker's last-seen cutoff (`src/server/clients.zig`) |
| `logging.query_log_buffer_max` | u32 | 10000 | entries | 11000000 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`) |
| `logging.query_log_flush_interval_s` | u16 | 60 | seconds | 03600 | how long the query-log writer gathers entries before committing them in one transaction (`src/storage/logger.zig`); see the note below |
| `logging.hide_domains` | bool | false | — | — | the query log stores a hidden marker instead of the domain |
| `logging.hide_client_ips` | bool | false | — | — | the query log stores a hidden marker instead of the client address |
| `logging.output` | enum `.stderr` \| `.syslog` \| `.file` | `.stderr` | — | one of the three tags | log sink selection (`src/platform/logging.zig`); `.stderr` and `.syslog` both write to stderr (journald captures it), `.file` rotates |
@@ -139,6 +140,19 @@ Process log and query log behavior.
| `logging.max_size_mb` | u32 | 50 | MiB | at least 1 | rotation trigger for the log file |
| `logging.max_files` | u8 | 5 | files | at least 1 | log files kept in total, the live one included, so the highest rotated generation is `max_files - 1`; the default 5 keeps `nxdns.log` plus `nxdns.log.1` through `nxdns.log.4`, and a value of 1 keeps only the live file, which rotation deletes rather than renames |
#### What `query_log_flush_interval_s` costs and buys
The query-log writer commits one transaction per interval instead of one per query. At a household's query rate — a few queries a second at most, often a fraction of one — a per-query commit writes hundreds of times more bytes to the disk than the rows themselves occupy, because every commit rewrites the WAL frames, the WAL index and the page headers whatever the row size. That write amplification is what wears out an SD card, and the default of 60 seconds is what stops it. It matches Pi-hole's `DBinterval`, which defaults to the same minute for the same reason.
What it costs:
- **Crash-loss window.** A process that dies takes roughly `interval` seconds of query history with it. That is the normal case, not a guaranteed maximum: a batch the disk monitor is holding back (free space below the critical threshold) or one waiting on a database write lock can be considerably older when the process dies. Power loss can additionally lose recent committed transactions, because `querylog.db` runs with WAL and `synchronous=NORMAL` — that was already true at any interval, and setting `0` does not buy per-query durability. Query history is the least valuable data on this box: nothing else depends on it, and it is deleted by retention anyway.
- **Staleness.** Every read backed by the query log — the query-log page, the dashboard totals, the timeseries — lags about `interval` seconds behind, and further behind while writes are gated or slow. The live view does not lag: it is fed from the SSE hub before the queue, so queries appear there the moment they are answered.
`0` means "do not wait": the writer commits the entry that woke it together with whatever is already queued, up to 100 rows. Use it when you want the query-log page to be current to the second and you do not care what that costs the disk.
Two things do not change with the interval: a batch is capped at 100 rows, so a burst is committed as soon as it fills one rather than waiting out the window, and shutdown writes what the writer is holding instead of waiting for the interval to end.
### disk
Free-space thresholds for the data directory. Below them the query-log writer, the client tracker and the blocklist scheduler are throttled (`src/storage/disk_monitor.zig`); DNS resolution is never gated.
@@ -384,6 +398,7 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
| `BadTtl` | `blocking.ttl`, `cache.negative_ttl_max`, a record `ttl`, `web.session_ttl_hours` or `blocklist_update.interval_hours` outside its range |
| `BadCacheSize` | `cache.size` outside 11000000 |
| `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 11000000 |
| `BadFlushInterval` | `logging.query_log_flush_interval_s` above 3600 |
| `BadLogRotation` | `logging.max_size_mb` or `logging.max_files` below 1 |
| `BadDiskThresholds` | a threshold below 1, or `min_free_mb` above `warn_free_mb` |
| `BadRateLimit` | `dns.rate_limit`, `dns.rate_window_seconds`, `web.api_rate_limit_per_min` or `web.sse_max_connections_per_ip` out of range |
+16 -3
View File
@@ -18,8 +18,8 @@ Default `/var/lib/nxdns`, overridable with `--data-dir DIR`. `nxdns run` and `nx
| `config.db-wal`, `config.db-shm` | SQLite write-ahead log and shared-memory index for `config.db`. Created by `run`, `import` and `export` when WAL is enabled, inheriting the main file's permissions. `check` creates neither. | 0600 |
| `querylog.db` | The query log: every domain every client asked for. Expendable — if it is missing or unusable it is recreated empty. | 0600 |
| `querylog.db-wal`, `querylog.db-shm` | WAL sidecars for `querylog.db`. | 0600 |
| `querylog.db.corrupt-<unix-seconds>` | A `querylog.db` that could not be used, moved aside before an empty one was created in its place. Kept, never overwritten. | Whatever the renamed file had — no chmod reaches it |
| `querylog.db.corrupt-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 1 and rises until the name is free. Two recreates within one second is the case it exists for. | The same |
| `querylog.db.<reason>-<unix-seconds>` | A `querylog.db` this build could not use, moved aside before an empty one was created in its place. Kept, never overwritten. `<reason>` is one of `corrupt`, `not-a-database`, `quick-check-failed` or `schema-changed`; see [why a query log is moved aside](#why-a-query-log-is-moved-aside). | Whatever the renamed file had — no chmod reaches it |
| `querylog.db.<reason>-<unix-seconds>-<n>` | The same, when the plain name is taken — `<n>` counts from 1 and rises until the name is free. Two recreates within one second is the case it exists for. | The same |
| `blocklists/` | Compiled blocklist snapshots, one subdirectory of the data directory. | 0700 |
| `blocklists/<id>.list` | Exact domains for blocklist source `<id>`, one per line, behind a header. | 0600 |
| `blocklists/<id>.wild` | Wildcard entries for the same source. | 0600 |
@@ -50,7 +50,20 @@ A failed sweep is a warning, not an outage — leftover bytes do not justify los
The temporaries of a source that still exists are cleaned by the refresh that owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`, `.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not.
A `querylog.db` is moved aside when it is missing nothing but usability: SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not answer `ok`, or its `user_version` fingerprint does not match the schema. Only the main file is renamed — its `-wal` and `-shm` are deleted, because a stale WAL would be replayed into the fresh database. A missing `querylog.db` is created without any aside file. The rename happens inside `querylog_schema.open`, before the 0600 chmod, and that chmod names `querylog.db` and its two sidecars only — so an aside file keeps the mode the file had at rename time, which for a `querylog.db` nxdns itself created is 0600 and for one an operator put there is whatever they left it at. Nothing prunes the aside files; they accumulate until an operator removes them, and each one holds the same browsing history the live query log holds.
### Why a query log is moved aside
A `querylog.db` is moved aside when it is missing nothing but usability, and the name it is given says which of the four cases it hit:
| `<reason>` | What happened |
| --- | --- |
| `corrupt` | SQLite reported the file as damaged. |
| `not-a-database` | The file is not a SQLite database at all. |
| `quick-check-failed` | `PRAGMA quick_check` did not answer `ok`. |
| `schema-changed` | Nothing is wrong with the file. Its `user_version` fingerprint does not match this build's schema, so this build cannot read it. Upgrades that touch the query-log schema produce this one, and the file they set aside is a healthy database. |
Only the main file is renamed — its `-wal` and `-shm` are deleted, because a stale WAL would be replayed into the fresh database. A missing `querylog.db` is created without any aside file. The rename happens inside `querylog_schema.open`, before the 0600 chmod, and that chmod names `querylog.db` and its two sidecars only — so an aside file keeps the mode the file had at rename time, which for a `querylog.db` nxdns itself created is 0600 and for one an operator put there is whatever they left it at. Nothing prunes the aside files; they accumulate until an operator removes them, and each one holds the same browsing history the live query log holds.
### Why the databases are 0600
The 0600 modes are not cosmetic. `config.db` holds the argon2id password hash and `querylog.db` holds the browsing history of every client on the LAN, so both are as sensitive as each other, and a WAL file holds the same rows as the database it belongs to. SQLite creates the main database at `0644 & ~umask`; nxdns chmods it to 0600 before enabling WAL, so the sidecars inherit 0600 rather than being created world-readable.
+315
View File
@@ -0,0 +1,315 @@
# Milestone 26: upstream health honors the dashboard period
The dashboard's upstream numbers must respect the selected period. Today `GET /api/upstream/health` serves process-lifetime counters and a last-32-exchanges window beside a period picker that scopes everything else on the page; "63 failures, 100.0% success rate" side by side was the result. This milestone records per-minute upstream outcome history in `querylog.db` and serves ranged aggregates from it, keeping the in-memory health state for routing, backoff and `/metrics` only.
Design authority: the Codex design review of 2026-08-17 (adopted whole). Where this spec deviates from it, the deviation is named in the ruling that makes it.
## Implementation contract (read first)
- Verify every stdlib claim against `/home/mokhtar/app/zig` at tag 0.16.0. Pre-0.16 knowledge is stale.
- m13 ruling F-f binds every session: every behavior ships with a test the author watched fail — run the assertion before the code, or with the code reverted, and say so in the report.
- No new `std.log.err`. Counters and the health rollup are the failure surface; each new failure path logs at most one `warn` per pass.
- `admin/src/lib/contractSamples.gen.ts` is regenerated with the AGENTS.md command, never hand-edited, in the same session that changes `src/web/openapi.yaml`.
- Do not commit. The orchestrator commits after review, spec sync, and the user's screenshot approval.
## Rulings (binding)
### 1. History is event aggregation, not counter sampling
Outcomes are aggregated into their wall-clock UTC minute at the moment they are recorded, and the aggregates are flushed to storage. Nothing samples `total_successes`/`total_failures` and subtracts. Consequences a test must prove:
- a restart cannot invert or reset any persisted number (rows are additive facts; a restart within the same minute adds to the same row);
- a crash loses at most the currently unflushed aggregates — bounded undercount, never a negative delta;
- the failure that happened inside a period is reportable with its error name, because the minute row carries it.
### 2. Storage: two tables in `querylog.db`, identity is the URL
Appended to `querylog_schema.ddl` (which changes the DDL fingerprint; by the established policy — querylog_schema.zig:1-7, PLAN §3.7 — every existing `querylog.db` fails the fingerprint check on upgrade and is recreated, with the previous file renamed aside as `querylog.db.corrupt-<timestamp>` rather than deleted; the changelog must state that query history restarts empty and where the old file sits):
```sql
CREATE TABLE upstream_targets (
id INTEGER PRIMARY KEY,
url TEXT NOT NULL UNIQUE
);
CREATE TABLE upstream_minute (
upstream_id INTEGER NOT NULL REFERENCES upstream_targets(id),
minute_ts INTEGER NOT NULL,
successes INTEGER NOT NULL,
failures INTEGER NOT NULL,
last_failure_ts INTEGER,
last_error TEXT,
PRIMARY KEY (upstream_id, minute_ts),
CHECK (successes >= 0),
CHECK (failures >= 0)
) WITHOUT ROWID;
CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
```
- `minute_ts` is the UTC minute start in seconds (`@divFloor(ts, 60) * 60`), stamped from `std.Io.Clock.real` — history participates in wall-clock periods, so it uses the wall clock. Routing state stays on `.awake` untouched.
- Identity is the URL through a dimension table, not the `config.db` upstream id: ids cannot be foreign keys across database files and may be deleted or reused; the URL is the immutable historical identity the query log already uses. A URL edit deliberately starts a new history.
- Rows exist only for minutes that had at least one attempt. Bound: upstreams × 1440 rows/day.
- A deleted upstream's history stays until retention removes it and is simply not returned (ruling 6 returns current pool entries only).
### 3. Recording: accumulate in memory at the record points, never touch SQLite on the query path
New module `src/upstream/history.zig`:
```zig
pub const max_pending = 4096;
pub const Accumulator = struct {
pub const Cell = struct {
url: []const u8, // borrowed from the pool entry's endpoint; entries live for the process
minute_ts: i64,
successes: u32,
failures: u32,
last_failure_ts: ?i64,
last_error_buf: [48]u8, // the error-name capacity health.State.last_error_buf uses (health.zig:58); S1 should lift 48 into a shared pub const rather than duplicate the literal
last_error_len: u8,
};
pub const Stats = struct {
flushes: u64 = 0,
flush_failures: u64 = 0,
rows_dropped: u64 = 0,
pending: u32 = 0,
};
mutex: std.Io.Mutex = .init,
cells: [max_pending]Cell,
count: u32,
last_drop_minute: ?i64, // max minute_ts ever dropped; feeds `complete` (ruling 6)
last_flush_failed: bool, // set by a failed flush, cleared by the next successful one (ruling 7)
// counters atomic, same shape as retention.zig's Counters
/// The declared init: count 0, last_drop_minute null, last_flush_failed false,
/// every counter zero, cells undefined (a cell is written before it is read).
pub const init: Accumulator = ...;
pub fn recordSuccess(self: *Accumulator, io: std.Io, url: []const u8, wall_s: i64) void;
pub fn recordFailure(self: *Accumulator, io: std.Io, url: []const u8, wall_s: i64, error_name: []const u8) void;
/// The only read surface. Web, health and metrics consumers read through
/// this mutex-protected snapshot; the fields above are private to the module.
pub fn snapshotStats(self: *Accumulator, io: std.Io) Stats; // Stats gains last_drop_minute: ?i64, last_flush_failed: bool
};
```
- `Pool.recordSuccess` / `Pool.recordFailure` (pool.zig:309, :318) gain, after their existing health bookkeeping and **after releasing the pool mutex**, a call into an optional `history: ?*history_mod.Accumulator` field on `Pool`, passing `entry.endpoint.url` and `std.Io.Clock.real.now(io).toSeconds()`. Restructure both wrappers so the health update sits in an inner block whose close releases the pool mutex, and the history call follows the block; a comment on each states that ordering is the constraint. The accumulator has its own mutex (`lockUncancelable`, mirroring the health sections' reasoning at pool.zig:310-313); no lock is ever held while taking the other, so no ordering deadlock exists. Proven by: (1) an S1 unit test that wires an `Accumulator` into a `Pool` built from the existing `Fake` client machinery (pool.zig tests), drives one success and one failure, and asserts both outcomes landed in the accumulator's cells; (2) the S1 report citing the restructured wrapper bodies showing the call after the mutex scope closes — the earlier draft's "hook takes the pool mutex" deadlock test is withdrawn as not runnable without production callback seams.
- Lookup is linear over the live cells (url pointer equality first, then bytes; at household scale the live set is a handful). On a miss with `count == max_pending`, evict the cell with the oldest `minute_ts`, set `last_drop_minute = @max(last_drop_minute orelse evicted.minute_ts, evicted.minute_ts)` (never plain assignment — a merge-back after eviction must not move the watermark backwards), and bump `rows_dropped`. Dropping is the overflow behavior, retrying is the flush-failure behavior — the two must not be conflated.
- On failure, keep the newest `last_failure_ts` and its error name in the cell (same max-wins rule the SQL upsert applies).
### 4. Flushing: a dedicated task, one transaction per pass, additive upserts
New repo `src/storage/repositories/upstream_history_repo.zig` — S1 owns the whole repository contract including the ranged read; S2 consumes it and defines nothing of its own SQL:
```zig
pub const FlushRow = struct { url: []const u8, minute_ts: i64, successes: u32, failures: u32, last_failure_ts: ?i64, last_error: []const u8 };
pub fn flush(database: *db.Db, rows: []const FlushRow) db.Error!void; // one Tx: ensure targets, then upsert minutes
pub const WindowStats = struct {
attempts: u64,
successes: u64,
failures: u64,
last_failure_ts: ?i64,
/// The error name of the row holding the newest last_failure_ts in the
/// window; "" when the window holds no failure.
last_failure_error_buf: [48]u8,
last_failure_error_len: u8,
};
pub fn windowStats(database: *db.Db, url: []const u8, since: i64, until: i64) db.Error!WindowStats;
pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!i64;
```
`windowStats` is **one statement** — two statements would not share a SQLite snapshot, and the flush connection committing between them could pair `max(last_failure_ts)` from one state with an error text from another. The error lookup is a scalar subquery inside the same statement, with a deterministic tiebreak on its ORDER BY:
```sql
SELECT coalesce(sum(m.successes), 0), coalesce(sum(m.failures), 0), max(m.last_failure_ts),
(SELECT e.last_error FROM upstream_minute e
WHERE e.upstream_id = m.upstream_id AND e.minute_ts >= ?2 AND e.minute_ts < ?3
AND e.last_failure_ts IS NOT NULL
ORDER BY e.last_failure_ts DESC, e.minute_ts DESC LIMIT 1)
FROM upstream_minute m JOIN upstream_targets t ON t.id = m.upstream_id
WHERE t.url = ?1 AND m.minute_ts >= ?2 AND m.minute_ts < ?3;
```
(Adjust the correlation to the repo's statement idioms; the binding requirements are one atomic statement and the deterministic tiebreak. A bare `SELECT max(last_failure_ts), last_error` without the subquery would pair the max with an arbitrary row's error — SQLite's bare-column-with-aggregate behavior.) An unknown URL or an empty window returns zeros and the empty error. `pruneOlderThan` wraps the minute delete and the unreferenced-target cleanup in one `db.Tx`; the returned count is deleted minute rows only, and a test proves the target delete does not inflate it.
The upsert is exactly:
```sql
INSERT INTO upstream_minute (upstream_id, minute_ts, successes, failures, last_failure_ts, last_error)
VALUES (?1, ?2, ?3, ?4, ?5, ?6)
ON CONFLICT(upstream_id, minute_ts) DO UPDATE SET
successes = successes + excluded.successes,
failures = failures + excluded.failures,
last_failure_ts = coalesce(max(last_failure_ts, excluded.last_failure_ts), last_failure_ts, excluded.last_failure_ts),
last_error = CASE
WHEN excluded.last_failure_ts IS NOT NULL
AND (last_failure_ts IS NULL OR excluded.last_failure_ts >= last_failure_ts)
THEN excluded.last_error
ELSE last_error
END
```
(`max()` over a NULL is NULL in SQLite, hence the coalesce; a test proves a success-only upsert leaves an existing failure column intact.)
The flush loop lives on `Accumulator`:
- `run(self, io, database)` — every 60 s on the `.boot` clock (retention.zig:144-146's reasoning: a box that suspends must still see its interval elapse), `flushOnce` then sleep, `std.Io.Cancelable!void`.
- `flushOnce` (seam signature two bullets below) uses a **swap, never subtraction**: under the mutex, move the dirty cells into a flush-owned `[max_pending]Cell` buffer and clear the accumulator (`count = 0`); release the mutex; run `flush` on the buffer. On success the buffer is done. On failure, set `last_flush_failed`, bump `flush_failures`, log one `warn`, retake the mutex and **merge the buffer back additively** through the same cell-merge rules recording uses — cells recorded during the write keep their outcomes, a merged cell that collides sums, and a merge that overflows `max_pending` follows the normal drop policy (oldest evicted, `rows_dropped` and `last_drop_minute` updated). No subtraction exists anywhere: the earlier copy-and-subtract draft was unsound — while SQLite ran outside the mutex, a full accumulator could evict a copied cell and later recreate the same `(url, minute_ts)`, and the post-flush subtract then destroyed newly recorded outcomes. Tests must include the interleaving the swap makes possible: during a stubbed in-flight flush, recreate a swapped-out cell's `(url, minute_ts)` and fill the accumulator to overflow; fail the flush; assert the drop policy's accounting held — drops counted, `last_drop_minute` advanced via `@max`, and the flush-owned buffer unaffected by the recording that happened beside it. (Overflow loss is the specced policy, not a defect the merge-back must prevent.)
- Counter meanings, pinned by tests: `flushes` counts successful flush transactions only; `flush_failures` counts failed flush attempts; a pass with nothing pending counts neither. A successful flush clears `last_flush_failed`.
- The flush write goes through a seam so tests can fail it deterministically: `flushOnce(self, io, database, write: *const fn (*db.Db, []const upstream_history_repo.FlushRow) db.Error!void)` with the production caller passing `upstream_history_repo.flush`; `run` closes over the real function. Tests pass a failing or recording stub.
- Clean shutdown order, explicitly: drain the query logger, cancel and join the task group, then run one final `flushOnce`, then close the history connection. A crash loses at most the pending cells.
- The task owns a dedicated `querylog.db` connection from `cli.DataDir.reopenQuerylogDb` (retention.zig:143-157 explains why handles are never shared; app.zig:525 is the pattern).
- The disk-monitor gate does not block the flush (prune-sized writes, same category as the query logger's own writes, which are ungated).
### 5. Retention: the existing daily pass, a fixed window, no knob
`Retention.runOnce` gains an upstream-history prune step placed **with the prune and checkpoint steps, before the vacuum-cadence logic** — the vacuum block early-returns (`passes_since_vacuum < vacuum_every_passes`, and the gated path returns too; retention.zig:114-123), so a step after it would be skipped on most passes. The step is `upstream_history_repo.pruneOlderThan(database, now - retention_window_s)` with `pub const retention_window_s: i64 = 31 * 86_400` on the repo. On the arithmetic: `stats.window` gives `since = until - width * count` with `until > now`, so a 30-day window's `since` never precedes `now - 30d`; 31 days is a full day of slack, not a bound the windows require, chosen so a pass that runs late clips nothing (deviation from the Codex text, which suggested computing the cutoff through `stats.window(.@"30d")` — that would import `web/handlers/stats.zig` into `storage/`, and the constant dominates every aligned window anyway). `logging.retention_days` does not apply to upstream history; a test proves a 1-day query-log retention still keeps 30 days of upstream minutes. Pruned minute rows do **not** count into the existing `rows_pruned` counter (`nxdns_retention_rows_pruned_total` stays query-log-only); `Retention.Stats` gains a separate `upstream_rows_pruned` counter with its own metric line, and a test pins that a pass pruning both kinds moves each counter by its own amount.
### 6. API: `GET /api/upstream/health?period=` with a now/period split
`src/web/handlers/upstream_health.zig` is rewritten. Period handling reuses `stats.periodParam` and `stats.window` verbatim — absent defaults to `24h`, anything else unreadable is the same 400 text stats uses. The aggregation runs on the web task's own `state.querylog_db` connection over `[since, until)` by `minute_ts`.
```zig
pub const PeriodStats = struct {
attempts: u64,
successes: u64,
failures: u64,
success_rate: ?f32, // null when attempts == 0 — no observations is not perfect reliability
last_failure_at: ?i64, // newest last_failure_ts inside the window
last_failure_error: ?[]const u8,
};
pub const Upstream = struct {
url: []const u8,
enabled: bool,
available: bool,
period: PeriodStats,
};
pub const Body = struct {
period: []const u8,
since: i64,
until: i64,
available: u32,
total: u32,
complete: bool,
upstreams: []const Upstream,
};
```
- Rows come from the current pool snapshot (`metrics.poolSnapshot`), joined to history by URL through `upstream_history_repo.windowStats` — the handler owns no SQL of its own: current upstreams only, deleted URLs' history is not returned, an upstream with no rows in the window gets zeros and nulls. `WindowStats` returns `last_failure_error_buf` by value; the handler must copy the selected error text into the request arena before it builds the response entry — a slice into the loop-local `WindowStats` dangles into stack storage the next iteration reuses.
- `complete` is per-window and stateless: false iff the accumulator's `last_drop_minute` is non-null and `>= since`. The field's definition is deliberately narrow, and the openapi description carries it verbatim: "No capacity drops known in this process within the selected window; up to about a minute of the newest outcomes may not have flushed yet, and outcomes lost in an unclean shutdown are not detectable." A window that starts after the last drop is complete again, so one historical overflow does not mark every future response. The UI shows the muted incomplete note when false and asserts nothing affirmatively when true — absence of a warning, never a "complete" badge.
- Removed from the response entirely: `consecutive_failures`, `total_successes`, `total_failures`, the last-32 `success_rate`, process-lifetime `last_error`/`last_error_age_s`. They remain in `health.State` for routing and in `/metrics` unchanged. openapi.yaml: the route gains the shared `period` query-parameter reference the stats routes use and a 400 response for a bad period, and the schema is rewritten to this shape (required: all fields; nullables marked); samples regenerated and `admin/src/lib/types.ts` updated in the same session.
### 7. Failure visibility
- `metrics.zig` gains a group fed by `Accumulator.snapshotStats`: `nxdns_upstream_history_flushes_total`, `nxdns_upstream_history_flush_failures_total`, `nxdns_upstream_history_rows_dropped_total`, `nxdns_upstream_history_pending` (gauge).
- The `/api/health` rollup (handlers/health.zig) gains one boolean in its `Input`: `history_flush_failing`, fed from the accumulator's `last_flush_failed` — current state, set by a failed flush and cleared by the next successful one. The rollup degrades to `degraded` while it is true and recovers when it clears (`/api/health` has no `warn` status). Nothing in `collect`/`rollup` compares cumulative counters across samples (they are stateless, so "grows between collections" is unimplementable there), and `rows_dropped` does not feed the rollup at all — a historical overflow must not latch `/api/health` to `degraded` forever; drops surface through the metric and through ruling 6's per-window `complete`. Extend the degraded-matrix table test with the new row in both states.
### 8. UI: one ranged table, live state labeled locally
- `upstreamHealthQuery(period)` in `admin/src/lib/queries.ts` — the query key includes the period, so the picker refetches upstream health with totals and timeseries. `UpstreamHealth` types match ruling 6.
- The upstreams card rejoins the ranged content (the m26 layout deletes the "Right now" section the previous fix added; DiskCard becomes a card titled "Storage now"). Columns: Upstream · Status now · Attempts · Failures · Success rate · Last failure. A grouped header labels the last four "Selected period · {label}".
- "Status now" is one word from live state: `Disabled` when not enabled, else `Available`/`Backing off` from `available`. The yes/no Enabled and Available columns are gone.
- Zero attempts: `0`, `0`, `—`, `—`; when every upstream has zero attempts the card shows "No upstream attempts in this period." A `100.0%` must be unreachable from zero attempts (vitest proves it).
- Last failure renders `ErrorName · age` via the existing `formatAge` against `last_failure_at` and the response's `until`… no — against `Date.now()` at render, same as the m25.5 cell; `—` when null. `complete: false` renders a muted "history incomplete" note on the card.
- Numeric columns use `shared.tabularNums`.
## Sessions
Sequential: S1 → S2 → S3. No parallel sessions; each depends on the previous session's files.
### Session S1: schema, accumulator, flush, retention, metrics, health rollup
Owns: `src/storage/querylog_schema.zig`, `src/upstream/history.zig` (new), `src/storage/repositories/upstream_history_repo.zig` (new), `src/upstream/pool.zig`, `src/storage/retention.zig`, `src/app.zig`, `src/cli.zig` (its connection-count comment on the querylog reopen becomes false — the count is writer + retention + history = three background connections, plus the web connection when the web server is enabled; correct it), `src/web/metrics.zig`, `src/web/handlers/health.zig`, `src/tests.zig`.
Tests (each watched failing): additive upsert including the NULL/max interaction and success-only-preserves-error; restart-shaped double flush into one minute row sums; `windowStats` sums only the window and pairs the newest failure with its own error row, not an arbitrary one; eviction at `max_pending` drops oldest, counts, and moves `last_drop_minute` only forward; the ruling-4 eviction-and-reinsertion-during-flush interleaving (fail the flush; assert the correct drop count, the forward-only watermark, and the unaffected flush-owned buffer); flush failure sets `last_flush_failed` and the next success clears it, with the counter meanings of ruling 4 (a no-op pass counts neither); retention prunes a 32-day-old minute row and its orphaned target while keeping day-29 rows under `retention_days = 1`, inside one transaction, counting minute rows only, into `upstream_rows_pruned` and not `rows_pruned`; the pruning step runs on a pass where the vacuum logic early-returns; the pool-with-accumulator test of ruling 3 (Fake-client success and failure both land in cells); metrics render the four new series; rollup degrades on `history_flush_failing` and recovers, and does not degrade on `rows_dropped`.
Acceptance (S1):
- [ ] `zig build test` and `-Dintegration` green with explicit counts from the test binary.
- [ ] The DDL fingerprint changed and the schema test names both new tables and the index.
- [ ] No SQLite call reachable from `exchangeLoopLen` (grep-level review stated in the report).
### Session S2: the ranged endpoint and the API contract
Owns: `src/web/handlers/upstream_health.zig`, `src/web/openapi.yaml`, `admin/src/lib/types.ts`, `admin/src/lib/contractSamples.gen.ts` (regenerated).
Tests (watched failing): window aggregation sums only `[since, until)`; zero attempts → `success_rate: null` and null failure fields; last failure inside the window beats an older one outside; `complete` flips on a drop stamped inside the window and not on one before it; the 400 and default-period behavior match stats; serialization pins the exact field set (the removed fields must not appear — assert their absence); two upstreams with different last-failure errors serialize both error texts correctly (no dangling or cross-row reuse from the by-value `WindowStats` buffer).
Acceptance (S2):
- [ ] Both Zig suites green; regenerated samples show the new shape and none of the removed fields.
- [ ] `cd admin && npx tsc --noEmit` green against the new types.
### Session S3: dashboard UI
Owns: `admin/src/lib/queries.ts`, `admin/src/lib/api.ts` (`getUpstreamHealth` takes the period), `admin/src/features/dashboard/DashboardPage.tsx`, `UpstreamHealthTable.tsx`, `DiskCard.tsx`, their test files, `admin/src/ui/styles.ts` (only if a shared style is genuinely reused), `CHANGELOG.md` (the Unreleased notes of the milestone acceptance).
Tests (watched failing): period in the query key refetches on picker change; column set and grouped header; `Backing off` and `Disabled` states; zero-attempt row renders em-dashes and never `100.0%`; card-level no-attempts message; incomplete note; the m25.5 "Right now" section is gone and "Storage now" exists.
Acceptance (S3):
- [ ] `cd admin && npx tsc --noEmit && npx vitest run && npx prettier --check . && npm run lint` all green with counts.
## Module layout
New files: `src/upstream/history.zig`, `src/storage/repositories/upstream_history_repo.zig`. Deleted surface: the unranged fields of `/api/upstream/health` (breaking API change, pre-v0.1, changelog notes it), the dashboard's "Right now" section.
## File ownership
| File | Session |
| --- | --- |
| `src/storage/querylog_schema.zig`, `src/upstream/history.zig`, `src/storage/repositories/upstream_history_repo.zig` | S1 |
| `src/upstream/pool.zig`, `src/storage/retention.zig`, `src/app.zig`, `src/cli.zig`, `src/web/metrics.zig`, `src/web/handlers/health.zig`, `src/tests.zig` | S1 |
| `src/web/handlers/upstream_health.zig`, `src/web/openapi.yaml`, `admin/src/lib/types.ts`, `admin/src/lib/contractSamples.gen.ts` | S2 |
| `admin/src/lib/queries.ts`, `admin/src/lib/api.ts`, `admin/src/features/dashboard/*`, `CHANGELOG.md` | S3 |
Sessions are strictly sequential; the table exists so a fix agent knows whose file it is touching.
## Acceptance (milestone complete)
- [ ] All session boxes.
- [ ] Live smoke per the verify-against-the-real-network rule: a scratch server with one dead upstream (TEST-NET address) and one real one; real queries; confirm the 1h window shows the dead upstream's failures with `success_rate` 0% and the real one's successes; restart the server mid-window and confirm the counts continue rather than reset; wait past a flush and confirm rows in `upstream_minute` via sqlite3; switch the picker across all four periods and confirm the numbers change window, not meaning.
- [ ] UI screenshots (dashboard across at least two periods, plus the zero-attempts state) reviewed by the user in Firefox before anything is pushed — the user's standing rule; no push, no release without it.
- [ ] `src/web/openapi.yaml` reviewed by hand against the response struct; reviewer says so in `## Recorded`.
- [ ] Changelog notes: ranged upstream health, the removed API fields, and that the upgrade recreates `querylog.db` — query history restarts empty and the previous file is kept aside as `querylog.db.corrupt-<timestamp>`, per the established schema-change policy.
## Anti-requirements
- No config knobs: cadence, capacity, retention window are `pub const`s.
- No per-exchange rows anywhere; the finest grain is the minute aggregate.
- No third database file and no history in `config.db`.
- No counter-delta sampling and no code path that can produce a negative count; restarts add, never subtract.
- No `100.0%` (or `1.0`) from zero attempts — null, rendered as an em-dash.
- No SQLite work on the query path; the accumulator is memory-only under its own lock.
- No page-wide "the period does not apply" banner; live values are labeled locally (`Status now`, `Storage now`).
- No removal of `health.State` internals — the last-32 window, lifetime totals, consecutive failures and backoff stay for routing and `/metrics`.
## Recorded (implementation)
Built S1 → S2 → S3, reviewed (one Codex round, two findings, both fixed), live-smoked. Deviations and observations, per session:
S1:
- `WebState` gained a `history` field so the handler can reach the accumulator; `web/state.zig` was not in the ownership table.
- The accumulator is heap-allocated, following the `sse.Hub` pattern, so its address is stable across the task group and the web server.
- Shutdown calls `group.cancel` explicitly before the final flush; the spec's order (drain logger → cancel and join → final flush → close connection) holds, the explicitness is the deviation.
- The `/api/health` change touched only the rollup `Input`; `collect` stayed as it was.
- Honest negative from the S1 report: a bare-column mutation of `last_failure_ts` in the upsert survives the tests because SQLite's `max()` NULL rule absorbs it; the deterministic-tiebreak test covers the observable behavior instead.
Review round:
- app.zig error-path teardown now runs the ruling-4 shutdown sequence via a single `defer`; no test — the structural argument (one defer, one sequence, no early-return can skip it) is the recorded justification.
- The u32 saturation in the accumulator's cell merge got a constraint comment.
S2:
- `bad_period_message` is duplicated in `upstream_health.zig` because `stats.badPeriod` is private; follow-up noted to export it and delete the copy.
- The route-level 400 is not testable through `handle` for either period route; the behavior is covered at the `periodParam` level, shared with stats.
- A 500 response was added to the openapi entry beyond the spec, matching the other database-backed routes.
- A float-spelling correction in the contract samples (regenerated, not hand-edited).
S3:
- Changelog entries sit under Unreleased pending the release commit.
- The incomplete note reads "history incomplete for this period" — wording chosen in session, within ruling 8's intent.
Live smoke: all four periods returned window-scoped numbers; an invalid period returned 400; restart continuity held (counts continued, no reset); the schema-fingerprint aside-rename fired and produced `querylog.db.corrupt-1786956534`; `upstream_minute` rows confirmed via sqlite3 after a flush.
+285
View File
@@ -0,0 +1,285 @@
# Milestone 27: diagnostics vertical slice
Step 1 of the UI redesign (`specs/ui-redesign.md`). A curated log of operational failure episodes: one `operational_events` table in `config.db`, a serialized event store, fifteen typed event codes emitted at existing failure sites, `GET /api/diagnostics`, and a Diagnostics page in the admin SPA. The rest of the UI stays intact; the full navigation restructure is later steps.
Design authority: `specs/ui-redesign.md` §Diagnostics, as amended by the Fable review rulings recorded in that file. No `api.storage` code. No periodic probe. No new index beyond the two specced.
## Sessions
S1 (store + API) → S2 (emitters) and S3 (SPA) in parallel → orchestrator integration (contract samples regen, final wiring check).
---
## Session S1: event store, schema, API, health, metrics
### S1.1 Schema — `src/storage/config_schema.zig`
Append to `ddl_v1` (pre-v0.1: edit the baseline, no migration step; update PLAN §11.2 to match):
```sql
CREATE TABLE operational_events (
id INTEGER PRIMARY KEY,
code TEXT NOT NULL,
subject_key TEXT NOT NULL,
subject_label TEXT NOT NULL,
severity TEXT NOT NULL CHECK (severity IN ('warning', 'error')),
first_seen INTEGER NOT NULL,
last_seen INTEGER NOT NULL,
occurrences INTEGER NOT NULL CHECK (occurrences > 0),
resolved_at INTEGER,
detail TEXT NOT NULL DEFAULT '',
CHECK (resolved_at IS NULL OR resolved_at >= first_seen)
);
CREATE UNIQUE INDEX idx_operational_events_active
ON operational_events(code, subject_key) WHERE resolved_at IS NULL;
CREATE INDEX idx_operational_events_last_seen
ON operational_events(last_seen DESC);
```
`operational_events` is runtime state, NOT configuration. It stays out of `table_names` and `delete_order` (`config_schema.zig:112-129`). Tests that must be updated deliberately, not silently: `migrations.zig:365` asserts `count(tables) == delete_order.len + 1` — becomes `+ 2`, and its comment names `operational_events` as the second exclusion alongside `schema_version`. `src/config/reconcile.zig:268` (`comptime assert(delete_order.len == 10)`) is untouched — the list does not change. Add a test asserting export/import round-trips leave `operational_events` rows intact.
Editing `ddl_v1` changes nothing for existing installs pre-0.1 EXCEPT the silent-divergence hazard already on record: the Pi's `config.db` is stamped version 1 and will not get the new table. This milestone accepts that with a bridge placed at the END of `migrations.migrate`, after the version stamp: execute the three `operational_events` statements with `IF NOT EXISTS` added. It must NOT live in `cli.openConfigDb``openConfigDb` runs before migration everywhere (`app.zig:333`, `cli.zig:484`, `cli.zig:516`), and creating the table there would make a fresh database's non-`IF NOT EXISTS` `ddl_v1` `CREATE TABLE` fail. The bridge is explicit, commented as removable when the 0.1 adoption gate lands, and touches no other table. Test: a version-1 database created WITHOUT the table gains exactly it after `migrate`, and a fresh database migrates cleanly (no double-create).
### S1.2 Event codes — `src/storage/events.zig` (new)
```zig
pub const Code = enum {
disk_space, disk_probe,
blocklist_refresh, blocklist_snapshot, blocklist_storage,
certificate_reload,
query_log_write, query_log_maintenance, query_log_recreated,
upstream_history_write, upstream_exchange,
client_names_storage, clients_storage,
listener_start, configuration_load,
};
```
Fifteen enum members — that count is the single cardinality every exhaustive test, route fixture and frontend copy union uses. The enum is the truth; the wire form is the dotted string (`disk_space``"disk.space"`, `blocklist_refresh``"blocklist.refresh"`, etc.) via `pub fn wire(code: Code) []const u8` — an exhaustive switch, tested against every member. `component(code)` returns the prefix before the dot, also exhaustive.
Severity is fixed per emit call, not per code (a disk transition to `warn` is warning, to `critical` is error).
### S1.3 Event store — `src/storage/events.zig`
```zig
pub const Store = struct {
pub const max_detail_len = 512;
pub const max_subject_key_len = 256; // longer keys become sha256: digests — see the identity rule below
pub const max_subject_label_len = 128;
pub const resolved_retention_s: i64 = 90 * 86_400;
pub const max_resolved_rows: i64 = 5_000;
mutex: std.Io.Mutex,
database: *db.Db, // dedicated connection; ALL access goes through the Store's mutex
write_failed: std.atomic.Value(bool),
write_failures: std.atomic.Value(u64), // feeds the metrics counter
active: ActiveSet, // in-memory mirror of active (code, subject_key) rows
untracked_active_count: u32, // active rows NOT in the mirror (overflow); loaded at init
pub fn init(io, database, now_s) db.Error!Store // loads ActiveSet + untracked count, prunes; failure = no store
pub const max_kept_keys = 64; // resolveExcept bound; over it the call is refused, counted and latched — truncating the kept list would close episodes that are still true
pub fn report(self, io, now_s, code, subject_key, subject_label, severity, detail) void
pub fn resolve(self, io, now_s, code, subject_key) void
pub fn reportResolved(self, io, now_s, code, subject_key, subject_label, severity, detail) void
pub fn resolveExcept(self, io, now_s, code, kept_keys: []const []const u8) void
pub fn prune(self, io, now_s) void
pub fn writeFailed(self) bool
pub fn activeCounts(self, io) struct { warnings: u32, errors: u32 }
pub fn selectEvents(self, io, arena, filter) db.Error!EventsPage
pub fn selectOne(self, io, arena, id: i64) db.Error!?Event
};
```
Contract:
- **Time is a parameter, not a stored seam.** Every mutating method takes `now_s: i64` from the caller, matching how `history.zig:132` receives `wall_s` and `logger.zig:72` receives entry timestamps. Production callers compute it from `Clock.real`; tests pass literals. No function pointer, no clock inside the store.
- **`resolve` is hot-path safe.** It checks `active` under the mutex and, when the key is absent AND `untracked_active_count == 0`, returns without any SQLite statement. `pool.recordSuccess` calls it on every successful exchange; steady state must cost a mutex acquire and a lookup. The no-SQL guarantee is tested through a debug-only statement counter on the Store (incremented before every repo call) — NOT via `sqlite3_total_changes`, which a `SELECT` probe would not move.
- **Overflow is exact, not heuristic.** `ActiveSet` capacity is 256. The no-SQL fast path belongs to `resolve` ONLY: mirror miss + `untracked_active_count == 0` → return, no SQL (that is the steady-state success). `report` on a mirror miss always writes — when the count is zero it inserts directly (a new episode in normal state, no probe needed); when the count is nonzero it probes and upserts (touch the untracked active row if one exists — a second insert would collide with the partial unique index — else insert), incrementing `untracked_active_count` only when a new row was inserted AND the mirror is full. A successful slow-path `resolve` decrements the count. `init` loads the count as `active rows - mirrored rows`. `events_repo.resolveExcept` performs the bulk resolve and a count of TOTAL remaining active rows in ONE transaction, returning that total only after commit — the repo knows nothing of the mirror. The Store then removes resolved keys from its mirror and sets `untracked_active_count = total_active - active.len`; on any failure it changes neither.
- **Key canonicalization happens at every entry point.** `report`, `resolve`, `reportResolved` and each `kept_keys` element of `resolveExcept` all pass the caller's key through the same digest-if-over-length rule before any lookup or SQL. Tests cover a long-key report resolved with the same long key, and `resolveExcept` keeping a long kept key.
- **All reads go through the Store.** The handler calls `store.selectEvents` / `store.selectOne`, which lock the same mutex around `events_repo` — nothing touches `store.database` from outside. One connection, one owner.
- `report` upserts on the active row: present → `last_seen = now_s`, `occurrences += 1`, `detail` replaced, severity raised to the worse of the two, never lowered. Absent → insert new active row and add to `active`.
- `resolve` on an active row sets `resolved_at = now_s` and removes it from the mirror. A later failure inserts a NEW row (new episode) — the partial unique index enforces one active row per key.
- `reportResolved` inserts a row with `resolved_at = first_seen = last_seen = now_s`, `occurrences = 1`, and never touches `active` (one-shot events: `query_log_recreated`).
- `resolveExcept(code, kept_keys)`: resolves every active row of `code` whose `subject_key` is not in `kept_keys`, in one serialized operation. Exists for the boot-finalized codes (S2); nothing else may use it.
- Write failures: on `db.Error`, set `write_failed = true`, increment `write_failures`, drop the event. **Log only on the `false → true` transition** — a broken diagnostics database plus a busy pool must not produce warnings at query rate; the counter and the health surface carry the ongoing state. The next successful write clears the latch (and that recovery may log once).
- **No error propagates to a producer** — mutating methods return `void` by design; a diagnostics failure must never break the subsystem reporting it. `init` is the exception: it returns `db.Error`, and `app.zig` responds by running with no store (`null` everywhere) and logging once — a store built on an unverified mirror would produce false no-op resolves, which is worse than no store.
- `subject_key` identity is exact at any length: a key at or under `max_subject_key_len` is stored verbatim; a longer one (operator URLs are unbounded — `safe_url.zig:12` imposes no input limit, and `manager.zig:179`'s 255 cap bounds only a display copy) is replaced by `"sha256:" ++ hex(SHA-256(key))` — 71 bytes, deterministic, collision-free in practice, so distinct long URLs never merge and the same URL always maps to the same episode. Never truncate and never reject a key. `subject_label` truncates to `max_subject_label_len`, `detail` to `max_detail_len` — those are display fields. `subject_key` never leaves the process; `subject_label` is the redacted display identity (`safe_url.redactQuoted` where the subject is a URL).
- `prune`: delete resolved rows older than `resolved_retention_s`, then oldest resolved rows beyond `max_resolved_rows`. Active rows are never pruned. Called at store init and once per retention pass (S2).
### S1.4 Repository — `src/storage/repositories/events_repo.zig` (new)
Free functions on `*db.Db`, matching `upstream_history_repo.zig` conventions (file-scope SQL constants, by-value row structs with fixed buffers, prose test names, `:memory:` fixtures). Functions: `insertActive`, `touchActive`, `resolveActive`, `resolveActiveByKey`, `selectActiveId`, `countActive`, `resolveExcept`, `insertResolved`, `loadActive`, `selectEvents(filter)`, `selectOne(id)`, `pruneResolved`, `activeCounts` (the three beyond the original list serve the exact-overflow contract). Only the `Store` calls these in production. `selectEvents` filter: `state` (active/resolved/all), `severity`, `component` (matched on `code` prefix), `since`/`until` with the repo's `[since, until)` convention: an empty range (`since >= until`) returns nothing, and overlap is `first_seen < until AND (resolved_at IS NULL OR resolved_at > since)` — strict `>`, an episode resolved exactly at `since` does not overlap (matches `queries_repo.zig:211`). `limit`, `before` (keyset on id descending).
### S1.5 API — `src/web/handlers/diagnostics.zig` (new)
`GET /api/diagnostics` — params `state` (`active`|`resolved`|`all`, default `all`), `severity` (`warning`|`error`), `component`, `since`, `until`, `limit` (1..1000, default 100), `before` (positive id). 400 with a naming message on any bad param (copy `queries.zig` conventions: `parseFilter` + `message(err)`). 503 when the store is absent. Response:
```json
{ "events": [ { "id": 42, "code": "blocklist.refresh", "component": "blocklist",
"subject": "StevenBlack", "severity": "warning",
"first_seen": 1787118000, "last_seen": 1787118300, "occurrences": 3,
"resolved_at": null, "detail": "download failed: ConnectionTimedOut" } ],
"next_before": null,
"active": { "warnings": 1, "errors": 0 } }
```
`subject` serializes `subject_label`. `subject_key` has no wire form — assert that in a test. Pagination contract identical to `/api/queries` (full page carries cursor, short page nulls it).
`GET /api/diagnostics/{id}` — the same event object, 404 after retention or for an unknown id.
Routes: two entries in `routes.zig`, `.auth = .session`, `.policy = .read`. Update the pinned route-count test (`routes.zig:145`, 56 → 58) and every routing invariant test that enumerates. OpenAPI: paths + `DiagnosticsPage` / `DiagnosticEvent` schemas, following the `/api/queries` exemplar; the openapi drift tests must stay green.
### S1.6 Health — `src/web/handlers/health.zig`
`Input` gains `diagnostics_present: bool = true` (benign default, matching the all-defaulted convention), `diagnostics_write_failed: bool = false`, `diagnostics_active_warnings: u32 = 0`, `diagnostics_active_errors: u32 = 0`. Body gains `"diagnostics": { "state": "recording"|"unavailable", "active_warnings": N, "active_errors": N }``unavailable` when `!present` or `write_failed`. `degraded` adds `diagnostics_write_failed or !diagnostics_present`: in a serving process the store is absent only when `Store.init` failed, which is a real degradation, and the health endpoint never runs in subcommands. `collect` must assign `diagnostics_present = state.events != null` explicitly — the natural `if (state.events) |store|` shape would leave an absent store reported as recording under the benign default. **This milestone does NOT yet remove `history_flush_failing` or restructure health to the full redesign shape** — that lands with step 4 (Overview replacement); here health only gains the diagnostics block. Update the degraded-matrix test to cover both new inputs.
### S1.7 Metrics — `src/web/metrics.zig`
Gauges `nxdns_diagnostics_active_warnings`, `nxdns_diagnostics_active_errors`; counter `nxdns_diagnostics_write_failures_total` (increment in the store on each failed write). Follow the upstream-history block pattern.
### S1.8 WebState — `src/web/server.zig`
`events: ?*events.Store = null`.
### S1.9 Acceptance criteria (S1)
- [ ] `zig build test` green; new store tests cover: episode open/dedupe/severity-raise, resolve-then-new-episode, one-shot insert, no-op resolve executes no SQL, write-failure latch set and cleared (log only on transition), label/detail truncation, over-length subject_key digested to the same identity twice, prune retention + cap, overflow slow paths for report (upsert, no unique-index collision) and resolve (count decrements), resolveExcept keeping mirror and count exact.
- [ ] `events_repo` tests cover every function against `:memory:` including overlap-window selection and keyset pagination.
- [ ] Handler tests: each param rejected with 400 + naming message, 503 without store, pagination cursor behavior, active counts, 404 on `{id}`.
- [ ] Route-pin, openapi-drift, health-matrix, schema-count tests all updated intentionally and green.
- [ ] Export/import round-trip leaves `operational_events` intact (test).
---
## Session S2: emitters (after S1)
Thread the store as `?*events.Store` using the established `gate: ?*disk_monitor.Monitor` idiom (`app.zig:776`): per-call parameter for run-loops, optional post-init field for `Manager`/`Pool`/`CertStore`. Every subsystem must build and test with `null` (no store). app.zig opens the dedicated connection right after migration (`app.zig:335`), unconditionally — not gated on `cfg.web.enabled`; diagnostics record whether or not the UI is on:
```zig
var events_db = try data.openConfigDb(io);
defer events_db.close();
const boot_now = std.Io.Clock.real.now(io).toSeconds();
var event_store_storage: ?events.Store = events.Store.init(io, &events_db, boot_now) catch |err| blk: {
log.warn("diagnostics store unavailable: {s}", .{@errorName(err)});
break :blk null;
};
const event_store: ?*events.Store = if (event_store_storage) |*s| s else null;
```
`event_store` is what gets threaded; a failed init leaves it null and health reports `unavailable` + degraded.
Emit sites, from the verified survey. Each row: failure emit, recovery resolve, subject_key / subject_label, severity.
| Code | Failure (file:line today) | Recovery | subject_key → label | Severity |
| --- | --- | --- | --- | --- |
| `disk.space` | `disk_monitor.zig:137-147` transition to warn/critical | same `publish`, transition to ok | `"data"` singleton | warn→warning, critical→error |
| `disk.probe` | `disk_monitor.zig:94-97` (change `catch {}``catch \|err\|`), `:101-105`, `:109-113` | next successful branch of the same probe | operation name (`statvfs`/`data_dir`/`log_dir`) | warning |
| `blocklist.refresh` | `manager.zig:1102-1138` (fetch/compile/empty), `:1520-1522` (load) | `SourceStatus.succeed` paths `:840`, `:870`, `:1552` | source URL → source name | warning |
| `blocklist.snapshot` | `app.zig:572-577`, `manager.zig:1164-1167`, `:1184-1187` | `manager.zig:539-544` (post-swap) | singleton | error |
| `blocklist.storage` | `manager.zig:1200-1203`, `:1336`, `:1451`, `:1458`, `:1470` | **the success branch of each matching operation, NOT pass-end**`deleteQuietly` absorbs failures at `manager.zig:1466` and `pruneOrphans` still returns success at `:1350`, so a pass-end resolve would close the very event its own pass emitted | operation name | warning |
| `certificate.reload` | `cert_store.zig:256-262` (capture the discarded err), `:273-279` | `:271-273` success reload | endpoint kind — **`CertStore` gains a `kind: enum { doh, dot }` field set at `openCertStore` (`app.zig:611/:617`)** | warning — a stat failure can be a transient rename window and a failed reload keeps the loaded certificate serving (`cert_store.zig:250`); nothing is down |
| `query_log.write` | `logger.zig:252-260` (init, from the failing `BatchWriter.init` and its catch), `:400-404` (batch) | `:405` successful batch. Init failure has no recovery (writer exits) — the event stays active, which is the truth | `writer`/`batch` | error |
| `query_log.maintenance` | `retention.zig:113`, `:127`, `:133`, `:143`, `:151` | matching success branch same pass | operation name | warning |
| `query_log.recreated` | `querylog_schema.open` result. **Two changes: `OpenResult` gains the aside name (fixed `[path_buf_len]u8` + len — today it exists only in a stack buffer inside `open`, `querylog_schema.zig:137`), and `cli.openQuerylogDb` (`cli.zig:335`) stops discarding `recreated` and returns it to `app.zig`**, which calls `reportResolved` once the store exists. **Never emitted for `.missing`** — first creation has no aside file and is logged as "created" (`querylog_schema.zig:131`); a fresh install must not record a warning with an impossible aside path | one-shot | reason tag; detail carries the aside filename | warning |
| `upstream_history.write` | `history.zig:262-266` | `:257-261` | singleton | warning |
| `upstream.exchange` | `pool.zig:332-346` `recordFailure` — emit OUTSIDE the pool mutex, same placement discipline as `recordHistory` | `pool.zig:316-330` `recordSuccess``resolve` (no-op hot path) | upstream URL → `safe_url.redactQuoted` | warning |
| `client_names.storage` | `client_names.zig:125-131`, `:141-147` | clean-pass determination `client_names.zig:161-164` | `read`/`write` | warning |
| `clients.storage` | `clients.zig:196-202`, `:218-224` | success branches `:195-197`, `:214-217` | `materialise`/`prune` | warning |
| `listener.start` | `app.zig:896/:900` (DoH), `:915/:919` (DoT) | boot-finalized: after the bind phase, one `resolveExcept(.listener_start, failed_keys)` closes prior episodes for endpoints that started clean this boot — including endpoints now disabled | `doh`/`dot` | error |
| `configuration.load` | `app.zig:686`, `:1039-1045`, `:1056-1061` and managed-file `validate.Diagnostics` warnings rendered at `app.zig:204` | boot-finalized: after all boot findings are emitted, one `resolveExcept(.configuration_load, emitted_keys)` | setting/upstream identity (redacted) | warning |
Rules that bind every site:
- The existing `log.warn`/`log.err` lines STAY. Events are additive; journald keeps the raw stream.
- Failure text passed as `detail` is `@errorName(err)` plus the site's existing message fragment — no new prose invented, no allocation: format into a stack buffer of `Store.max_detail_len`.
- **Lock discipline is collect-then-flush, not "emit after the mutex closes".** Several manager sites cannot simply move: `publishRefresh` runs under `writer_lock` by contract (`manager.zig:792`) and `pruneOrphans` holds both `refresh_lock` and `writer_lock` through its filesystem work (`manager.zig:1299`). At such sites, record outcomes while the locks are held and flush to the store after the outer locked operation returns. The collection must not lose events: per-source outcomes ride the manager's existing per-source status allocations (one outcome per source, bounded by the source count), and same-operation failures within one pass aggregate into a single `report` per pass (the row's `occurrences` then counts failing passes, and `detail` carries the last error plus how many failures that pass held) — never an unbounded list, never a silent drop. Pool follows the existing `recordHistory` placement.
- Boot-finalized codes (`listener.start`, `configuration.load`) use `resolveExcept` exactly once each, after their boot phase completes; restart is the recovery, matching the design table.
- `retention.zig` `runOnce` gains `events: ?*events.Store` and calls `store.prune` once per pass.
### S2 acceptance criteria
Emitters fall into three classes, each with its own test obligation (the author watches each fail first, repo ruling F-f):
- **Episodic** (disk, blocklist ×3, certificate, query_log.write batch, query_log.maintenance, upstream_history.write, upstream.exchange, client_names, clients): force the failure (existing seams: injected db errors, missing files, dead fetch server), assert the `(code, subject_key, severity)` row; then force recovery and assert resolution.
- **Boot-finalized** (`listener.start`, `configuration.load`): assert the emit on a failing boot, and assert `resolveExcept` closes a pre-seeded stale episode on a clean boot.
- **One-shot / permanent** (`query_log.recreated`; `query_log.write` init): assert the one-shot row inserts already resolved; assert the init-failure episode exists and that no recovery path claims it.
- [ ] `pool` exchange tests: success with no active episode performs no store SQL; failure→success round-trip produces exactly one resolved episode with correct occurrences.
- [ ] All subsystems still pass with `events = null` (the existing suites running unchanged).
---
## Session S3: admin SPA (after S1, parallel with S2)
- `lib/types.ts`: `DiagnosticEvent`, `DiagnosticsPage`, plus the health body's new `diagnostics` block.
- `lib/api.ts`: `getDiagnostics(filter)`, `getDiagnostic(id)`.
- `lib/queries.ts`: `diagnosticsQuery` (infinite, keyset via `next_before`, `keepPreviousData`), `diagnosticQuery(id)`; refetch interval matching healthQuery's cadence for the active view.
- `routes.tsx`: `/diagnostics` route (loader + lazy component) and `/diagnostics/$id`. Nav: add `Diagnostics` to `NAV_ITEMS` between Lookup and Settings (full nav restructure is later milestones); update `AppShell.test.tsx`.
- `features/diagnostics/DiagnosticsPage.tsx`: active episodes first (severity, title from an exhaustive code→copy map, subject, "active for …" age, occurrences), then resolved history with the standard filters (state, severity, component) reflected in the URL search params. **Empty state is the healthy state**: one line, "No active operational issues.", quiet styling — it must read as good news, not as a broken page (Fable's caution).
- `features/diagnostics/DiagnosticDetailPage.tsx`: the ordered detail from the design — state, first/last seen, occurrences, resolution, impact and remediation text from the same exhaustive code map, last error detail, and links (the config surfaces linked are today's routes; they get re-pointed when later milestones move them).
- The code→copy map lives in `features/diagnostics/eventCopy.ts`: an exhaustive `Record<Code, {title, impact, remediation, link?}>` over a string-literal union of the fifteen wire codes; a test iterates the union and asserts every member has copy.
- Tests: page renders fixtures (active + resolved + empty), filters drive the URL, detail renders every code's copy, pagination fetches more.
### S3 acceptance criteria
- [ ] `npm run typecheck`, `npx vitest run`, `npx prettier --check`, lint, build, assert-bundled all green; byte budget respected.
---
## Orchestrator integration (after S2+S3)
- Regenerate `contractSamples.gen.ts` (`zig build test -Dintegration -Dcontract-samples-out=…`) — requires the integration capture in `web_integration_test.zig` to seed at least one active and one resolved event.
- Update `docs/reference/api.md` and any drift-guarded reference pages; changelog entry under Unreleased.
- Live smoke on the real binary before anything ships: force a blocklist failure (dead URL), watch the episode appear, fix the URL, watch it resolve; screenshot the page per standing rule.
## Module layout (new files)
| File | Purpose |
| --- | --- |
| `src/storage/events.zig` | `Code`, wire mapping, `Store` |
| `src/storage/repositories/events_repo.zig` | SQL |
| `src/web/handlers/diagnostics.zig` | list + detail handlers |
| `admin/src/features/diagnostics/…` | page, detail, copy map, tests |
## File ownership
S1: `config_schema.zig`, `events.zig`, `events_repo.zig`, `diagnostics.zig` (handler), `routes.zig`, `openapi.yaml`, `health.zig`, `metrics.zig`, `server.zig`, `migrations.zig` (bridge + test), PLAN §11.2. S2: every emitter file + `app.zig` + `retention.zig` + `cert_store.zig` + `cli.zig` (openQuerylogDb) + `querylog_schema.zig` (OpenResult aside name) and its tests — S2 starts after S1 lands, so nothing is shared in parallel. S3: `admin/` only. No parallel writers on any file (S2 and S3 run concurrently and are disjoint).
## Anti-requirements
- No `info` severity, no acknowledgement/dismissal state, no manual-resolve endpoint, no raw-log endpoint, no generic remediation action schema.
- No `api.storage` code. No periodic probe for `write_failed`. No configurable retention.
- No changes to the existing dashboard, query log, live, or lookup pages.
- No new dependency, front or back.
## Acceptance criteria (milestone complete)
- [ ] All session criteria; full `zig build test` + `-Dintegration`, admin suite, byte budgets.
- [ ] Live smoke: forced failure → visible episode → recovery → resolved row; `/api/health` shows the diagnostics block; `/metrics` shows the three series.
- [ ] Screenshots of the page (active, resolved, empty states) shown before push.
## Recorded (as built)
Deviations from the sections above, found during build, review and live smoke. The code is the authority; this section says where it moved.
- `src/storage/events_fixture.zig` exists (test-only): a shared migrated-config.db + Store fixture the storage and filter tests use.
- The threading fields through `app.zig`/`server.zig` are named `.diagnostics`; `WebState` carries `.events`. The live smoke caught that `.events` was never assigned in `app.zig` while every suite stayed green — the integration tests build their own `WebState`. One line fixed it; the lesson is the standing "verify against the real network" rule.
- `runScheduler`'s loop body is the pub function `scheduledPass`, so tests drive one pass deterministically. `runScheduler` flushes only in its catch branch (for the snapshot note its failure path adds).
- Occurrences count failing passes, not flushes: `SourceStatus` carries `pass_outcome`/`pass_failures`, set in `fail()`/`succeed()`, drained by `flushSourceDiagnostics`. Detail format: `"{state}: {error} ({N} this pass)"`.
- Pass accounting lives in exactly one table copy at a time: `mergeStatuses` zeroes the pass fields on carried entries, `installStatuses` folds the live table's unflushed accounting in by id under the exclusive lock, and the flush drains by claiming one outcome-bearing entry per lock round (immune to concurrent table replacement).
- A pass flushes before it releases the lock that serializes it: the flush defer registers after the unlock defer in `refreshSource`, `refreshAll`, `startupPass`; standalone `reload` takes `writer_lock` itself and flushes inside it. This amends the collect-then-flush rule — the store runs on its own connection and mutex, so no shared resource is held across the store call.
- Deleted sources: `flushSourceDiagnostics` ends in `resolveDeletedSources``resolveExcept(.blocklist_refresh, all current keys)`, gated on `generation != 0` (an empty pre-reload table means "not read yet", not "all deleted") and skipped above 64 keyed entries (`max_kept_keys`; the episode then lingers until the count drops). `Store.resolveExcept` is no longer boot-finalized-only; the rule is the kept list must be the whole current subject set.
- `events_repo.resolveExcept` is mark-then-unmark (sentinel `maxInt(i64)`, one transaction): the prior resolve-all-then-revive matched the revive on `resolved_at = now_s` and revived an episode resolved by the drain in the same second.
### Addendum: manual purge of resolved events
Resolution stays automatic; the operator decides when resolved history disappears. Two endpoints, both mutations under the usual auth, both allowed in file mode (diagnostics are runtime state, not configuration):
- `DELETE /api/diagnostics/{id}` — purges one resolved event. 409 `{ "error": ... }` when the event is active; 404 when no row has that id.
- `DELETE /api/diagnostics` — purges every resolved event, returns `{ "purged": N }`.
Store: purge functions under the store mutex; they touch only rows with `resolved_at NOT NULL`, so the ActiveSet mirror and untracked count never change. UI: a purge action on each resolved row and on the resolved detail page, plus a "Purge all resolved" control on the list when at least one resolved event shows; active events show no purge affordance. OpenAPI, routes count, contract goldens, api.md updated.
Accepted limitations (reviewed with Codex, ruled in proportion to household scale; each is bounded and self-correcting, counts stay visible in the detail text):
- A standalone web reload interleaving with a refresh pass can merge two passes' accounting into one occurrence.
- `pruneOrphans` flushes its storage aggregates with the writer locks released; two concurrent prunes can merge into one report.
- The deletion sweep's kept-key snapshot can go stale against a concurrently added source: its fresh episode can be resolved once and reopens on the next failing pass with `first_seen`/`occurrences` reset.
+25
View File
@@ -0,0 +1,25 @@
# Query-log batching: one transaction per interval, not per query
Measured on the deployed Pi: ~0.5 GiB/day of process writes to persist ~2.3 MB/day of query rows, because the writer's 100 ms batch window degenerates to one transaction per query at household rates (~0.25 qps). Write wear killed the previous SD card. Fix: widen the window, Pi-hole precedent (FTL DBinterval=60).
## Contract
- New config field `logging.query_log_flush_interval_s: u16`, default 60, valid 03600, dedicated validation classification (not folded into an existing one).
- `0` means: do not wait for more entries; immediately flush the entry plus whatever is already queued, up to `flush_batch` rows. It is NOT per-query power-loss durability — `synchronous=NORMAL` never promised that; say so in the reference.
- Writer loop (`src/storage/logger.zig`): block on `getOne`, then deadline = now + interval on the **`.boot` clock** (matching upstream history; `.awake` would stretch the window across suspend), fill until deadline or `flush_batch` (100) rows, one transaction per flush; if the queue holds more, keep flushing in `flush_batch` chunks.
- Shutdown (the Codex-found race, fix required): today `Logger.shutdown` closes the queue and the app then cancels the task group containing the writer, so cancellation can beat the close-observation and drop the held batch. New order: stop and join query producers first, close the queue, await the writer outside the cancelled group (give the writer a lifetime separate from the producers' group). If the disk gate forbids the final write, drain and count the entries as dropped — never hang shutdown, never lose them uncounted.
- Unchanged on purpose: queue cap `query_log_buffer_max` + drop-oldest backpressure, `flush_batch`, pragmas, wal_autocheckpoint, retention, domains interning. One commit per minute makes those second-order.
## Propagation (config field checklist)
Model key + round-trip drift guards, validation + validation reference, settings API view (`src/web/handlers/settings.zig`), openapi.yaml, admin types + settings control, contract samples regenerated, docs config reference, CHANGELOG.
## Documentation wording
- Crash-loss window: up to `interval` seconds of query history on process failure; power loss can additionally lose recent committed transactions (WAL + synchronous=NORMAL). Query history is the least valuable data on the box.
- Staleness: every query-log-backed read (query-log page, totals, timeseries) lags up to `interval` seconds. The live view is unaffected — it is fed from the hub before the queue.
## Acceptance
- [ ] Deterministic tests: interval batching (entries within the window land in one transaction), flush_batch early flush, 0-sentinel immediate flush, shutdown drains a held batch and the queue (the race sequence: producers stopped → queue closed → writer awaited), disk-gated final drain counts drops.
- [ ] Suites: `zig build test` and `-Dintegration` 0 failed; admin typecheck/vitest/prettier clean; goldens regenerated.
+304
View File
@@ -0,0 +1,304 @@
# UI redesign proposal
Author: Codex (gpt-5.6-sol, extra-high effort), 2026-08-19. **Not accepted yet.** Untracked on purpose until Mokhtar rules on the open questions at the end.
Answers that shaped it: the server and API may change; the surface-ownership split is right; file-mode configuration pages are read-only; diagnostics are curated structured events in the vein of Pi-hole's; time scoping is per workflow; a past query must be explainable exactly; Query Log and Live merge.
## Navigation
Five primary items. Configuration expands to three task-shaped subpages and holds no landing route of its own.
| Navigation | Route | Operator question | Replaces |
| --- | --- | --- | --- |
| Overview | `/overview` | Is DNS healthy and protecting the household now, and what happened in this period? | Dashboard |
| Activity | `/activity` | What requests are happening or happened, and why did nxdns handle them that way? | Query Log, Live, Lookup |
| Clients | `/clients` | Who is this address, which policy applies, and what has it been querying? | Clients |
| Diagnostics | `/diagnostics` | What is failing or has failed, what is affected, what should I do? | new |
| Protection | `/configuration/protection` | What policy governs each group, and which rules and lists produce it? | Groups, Blocklists, Rules |
| Resolution | `/configuration/resolution` | Where does nxdns answer or forward permitted names? | Local DNS, Upstreams |
| System | `/configuration/system` | What service, storage, logging, TLS and web settings is this process running with? | Settings |
Secondary routes, reached from those surfaces rather than the nav: `/activity/queries/:id`, `/activity/test`, `/clients/:id`, `/diagnostics/:id`.
No current page survives unchanged. Login, logout and pause survive functionally, restyled into the new shell.
### Activity
Two modes over the same columns and filters. History is persisted queries with keyset pagination and an absolute range. Live is follow-by-default with Freeze/Resume over the existing bounded 500-row buffer.
Columns: Time, Domain, Client, Type, Result, Route, Duration. Rule matches, source URLs and upstream errors live in the detail view, never on every row.
Domain testing stays as an Activity action labelled "Current policy simulation". It must never read as an explanation of a historical query.
### Protection
Group-centred: group list, selected group detail, effective safe-search setting, assigned blocklist sources, rules scoped to the group, client count linking to matching clients. A Sources tab holds the shared blocklist catalogue and the "Update now" runtime action.
### Resolution
Three tabs: upstream pool, local records, forward zones.
### Clients
Keeps primary navigation because identifying and naming unknown devices is an operational job, not configuration. The *learned* marker appears only here, beside the name. Prefix assignments live here as "Network assignments".
## Overview
Three sections, nothing else.
**1. Current status.** Five current facts, each conveyed by text and icon as well as colour. Healthy rows stay quiet; degraded rows link to the diagnostic or configuration surface that explains them.
| Status | Shows | Why it belongs |
| --- | --- | --- |
| Protection | Active, paused until a timestamp, or unavailable, with Pause/Resume | Confirms filtering is in force, and carries the valid runtime action |
| Upstreams | available / configured, now | Confirms DNS can leave the network |
| Query history | Recording, losing rows, or writer failed | Says whether Activity can be trusted |
| Diagnostics | Recording or unavailable | A failure reporter that cannot record failures must itself be visible |
| Storage | ok / low / critical, free bytes | Says whether writes are safe, and explains write gating |
The shell carries a small global "Protection active/paused" indicator linking back to Overview. The controls themselves stay on Overview and beside blocked-query details.
**2. Active issues.** Severity, short title, affected object, how long it has been active, link to the detail. When none exist, one restrained line: "No active operational issues." Resolved failures never appear here, and healthy subsystems never get permanent green cards.
**3. Activity over a period.** The existing 1h / 24h / 7d / 30d control. Every value uses exactly the returned `[since, until)` window: queries, blocked count and rate, distinct clients, average response time, one query-volume timeline split blocked/cached/other, and "Open activity for this period" carrying the exact bounds. The timeline stays the existing lightweight SVG; no charting dependency.
If the selected period predates available data, the section says "Query history is available from …" rather than charting the missing span as zero.
Removed from Overview: the historical upstream table, the "last failure" text, per-upstream period rates, the database and log byte breakdown, the standalone cache card.
## Diagnostics
Not a journald viewer, and it does not subscribe to `std.log`. Producers emit a finite set of typed events at the failure boundary.
### Event model
One row is one failure episode.
| Field | Meaning |
| --- | --- |
| `id` | Durable identifier |
| `code` | Fixed machine-readable kind |
| `subject_key` | Internal stable identity, may hold a full URL, never serialized |
| `subject_label` | Bounded, redacted, operator-facing identity |
| `severity` | `warning` or `error` |
| `first_seen` / `last_seen` | Episode bounds |
| `occurrences` | Deduplicated report count |
| `resolved_at` | Null while active |
| `detail` | Bounded last error or current condition |
No `info` severity. Normal starts, refreshes and reloads do not become entries; a success resolves its prior failure.
Active dedup key is `(code, subject_key)`. A repeat updates `last_seen`, `occurrences`, severity and detail. A success resolves the row. A later failure opens a new episode rather than reopening the old one. Severity records the worst state reached. There is no acknowledgement or manual dismissal: active means the component has not demonstrated recovery. One-shot material events, such as a query-log recreation, are inserted already resolved.
### Storage
Events live in `config.db` as runtime state, excluded from export, import and file reconciliation. Active rows are never pruned. Resolved rows keep 90 days, with a hard cap of the newest 5,000. `detail` caps at 512 bytes. Pruning runs at startup and from existing maintenance; no new scheduler.
One table, one repository, one fixed event-code enum. A small serialized event store owns a dedicated `config.db` connection; background producers report synchronously through its mutex. This is the existing pattern, not a new one: `app.zig:333` opens a connection for migration and reconciliation and `app.zig:544` opens a separate `web_config_db`, and the background producers (fetcher, disk monitor, logger writer) have no other safe path into `config.db`.
If the store itself cannot write, an atomic `event_store_failed` state appears in `/api/health` and journald. It clears on the next successful write — no periodic probe. The store receives a write whenever anything fails or recovers, and a flag left set while nothing needs writing costs nothing.
### Event sources
| Source | Event identity | Recovery |
| --- | --- | --- |
| Disk warn/critical transitions | `disk.space`, singleton | next `ok` sample |
| Failed `statvfs` or directory sizing | `disk.probe`, keyed by operation/path | next successful probe |
| Per-source download, HTTP, parse, compile or file-read failure | `blocklist.refresh`, keyed by source URL | that source refreshes |
| Initial snapshot or whole-pass failure | `blocklist.snapshot`, singleton | a snapshot publishes |
| Blocklist file cleanup failure | `blocklist.storage`, keyed by operation | that operation succeeds |
| Certificate stat or reload failure | `certificate.reload`, keyed by `doh`/`dot` | files readable and reload succeeds |
| Query writer init or batch failure | `query_log.write`, keyed by `writer`/`batch`/`queue` | writer starts, or a batch succeeds without drops |
| Query retention prune/checkpoint/vacuum failure | `query_log.maintenance`, keyed by operation | that operation succeeds |
| Upstream-history flush failure | `upstream_history.write`, singleton | next flush succeeds |
| Client-name selection or persistence failure | `client_names.storage`, keyed by operation | next pass succeeds |
| Client materialisation or pruning failure | `clients.storage`, keyed by operation | next pass succeeds |
| Upstream exchange failure | `upstream.exchange`, keyed by upstream URL | next successful exchange |
| Enabled DoH/DoT listener that cannot start | `listener.start`, keyed by endpoint | successful start after restart |
| Query-log recreation | `query_log.recreated`, one-shot | inserted resolved |
| Configuration warning leaving a capability skipped | `configuration.load`, keyed by setting | clean load after restart |
Sixteen codes. Codex proposed a seventeenth, `api.storage`, for a storage failure that produced an HTTP 500 — cut on review, because it breaks this section's own exclusion rule: a 500 already answered its caller. Do not merge the remaining codes to shrink the count either. A merged code forces `subject_key` to carry what the code no longer says.
An upstream event describes a consecutive failure episode, not one row per retry. One timeout followed by success is one resolved episode.
Outside Diagnostics: invalid requests, conflicts, rate limits and failed logins already answered to their caller; expected reverse-DNS outcomes; individual TLS handshake failures already counted; development asset-server warnings; fatal startup failures that stop the UI existing. Disk-gated skips do not duplicate — the active disk event explains the cause and counters keep the totals.
### From event to remediation
The detail shows current or resolved state; first seen, last seen, occurrences, resolution time; impact in operator language; the last bounded underlying error; the exact next action; how nxdns will verify recovery; links to the relevant configuration and activity window.
The frontend uses one exhaustive `switch` over the fixed codes for titles, impact, remediation and routes. No generic action schema, no plugin mechanism.
## Query provenance
Stays in the expendable `querylog.db`. Does not go into the operational-events table.
`block_reason` is replaced by the fuller `policy_reason`. New columns:
| Column | Type | Purpose |
| --- | --- | --- |
| `qclass` | INTEGER NOT NULL | explains filtering bypass for non-IN questions |
| `rcode` | INTEGER NOT NULL | the client-visible result, including SERVFAIL |
| `group_id` | INTEGER | group at query time, not a foreign key |
| `group_name` | TEXT | historical label, survives a rename |
| `policy_action` | TEXT NOT NULL | `not_evaluated`, `allow`, `block` |
| `policy_reason` | TEXT NOT NULL | pipeline or matcher reason |
| `matched` | TEXT | exact rule pattern or list entry |
| `source_id` | INTEGER | blocklist source at query time |
| `source_name` | TEXT | historical source label |
| `cname_target` | TEXT | target that caused an uncloaked block |
| `safe_search_target` | TEXT | name used for the rewrite |
| `route_kind` | TEXT NOT NULL | `blocked`, `local`, `forward_zone`, `upstream` |
| `forward_zone` | TEXT | exact matched zone |
`upstream` changes from the unhelpful `"pool"` marker to the actual configured upstream or forward resolver **on the exchange that actually happened**. On a cache hit `route_kind` is `cache` and `upstream` is NULL. Credentials are redacted at the serialization boundary.
Codex wanted the upstream recorded on cache hits too. Cut on review: it would widen every `src/cache/dns_cache.zig` entry to carry an upstream label, and it states a half-truth, because on a cache hit no upstream answered. The interface change it does need is real and worth doing — `handler.zig:79-83` records ruling 20, which deliberately keeps the pool's answering endpoint out of `transport.Client`'s reach. Exposing it touches `src/upstream/`, not the pure core. Do not fall back to `"pool"`: the exact upstream on a SERVFAIL row is the single most useful correlation this redesign adds.
`policy_reason` is a closed enum: `local_record`, `forward_zone`, `non_in_class`, `paused`, `snapshot_unavailable`, `no_match`, plus the existing rule allow/block and blocklist exception/domain/wildcard reasons.
For a CNAME-uncloaked block, the policy fields describe the target's decision and `cname_target` preserves the target responsible.
Every syntactically parsed request that receives a response is logged, including synthesized SERVFAIL. Requests too malformed to identify a question stay counters, not fabricated rows.
No response payloads, answer RR sets, EDNS data or packet bytes are stored. The record explains nxdns's own decision, not the external resolver's answer.
Privacy transforms apply to every new domain-bearing field, not only `domain`: with `hide_domains` on, matched names, CNAME targets and safe-search targets hide consistently.
A one-row `querylog_meta (created_at INTEGER NOT NULL)` table lets the stats and query APIs return a conservative `available_since`, which distinguishes "zero queries" from "history does not exist".
### Historical query detail
Ordered explanation: request (time, domain, client, type, class); group (historical id and name); policy (evaluated or not, allow/block, exact matched rule or list candidate, historical source, whether filtering was paused or unavailable); rewrites (safe-search target, CNAME target); route (local, forward zone, cache, or selected upstream); response (rcode, duration); related actions (test the domain against current policy, view the client, view activity for the same domain or client, view diagnostics in a five-minute window around the query).
Historical and current facts are visually separated. A rule, source or group that no longer exists stays visible as a historical value and is not linked to a different current object.
Live SSE events carry the same provenance shape without a persisted `id`. A frozen live row shows its in-memory detail; no correlation id is invented to link it to a row SQLite has not written.
## Time scoping
One contract: unix seconds UTC, `since` inclusive, `until` exclusive, point data qualifies on `since <= ts < until`, diagnostic episodes qualify when their active interval overlaps the range, current state is labelled "Now" and no historical selector touches it.
URLs: `/overview?period=24h` with the server returning the exact aligned bounds; `/activity?mode=history&since=…&until=…` with `domain`, `client`, `blocked` and the other filters in the URL; `/diagnostics?since=…&until=…&severity=…&component=…`; `/activity?mode=live` with Follow/Freeze as ephemeral UI state. Investigation links always carry absolute bounds, so a viewed incident does not drift as time passes. Timestamps display in the browser's timezone; URLs and APIs stay timezone-independent.
## File mode
One persistent authority line in the shell:
> File-managed · `/etc/nxdns/config.zon` · loaded 19 Aug 2026, 08:42
It sits in the Configuration sub-navigation and appears elsewhere as a compact lock indicator. No full-width banner on every route. The wording is "running configuration loaded from", not "file contents" — the server cannot prove a since-edited file still matches the running process.
A file-managed page uses definition lists for scalars and tables or cards for collections, with human labels and the exact ZON key shown secondarily (`logging.retention_days`). No text inputs, no checkboxes, no Add/Edit/Delete, no disabled form shells, no simulated Save. A short page note says where edits happen and that a restart may be needed.
Runtime actions stay ordinary enabled buttons: pause/resume, update blocklists now, reload certificates, delete an observed undeclared client, login/logout.
Database mode uses the same information architecture with real edit actions, plus a server-owned `restart_pending` boolean. The current client-only restart banner and its local store are removed, so a browser refresh cannot erase the warning.
## API changes
`GET /api/config/status``{authority, path, reconciled_at, restart_pending}`. `restart_pending` is process state: database-mode mutations that need a restart set it, a successful restart clears it.
`GET /api/health` becomes explicit about every condition that contributes to degradation — `protection`, `upstreams`, `query_history`, `upstream_history`, `diagnostics`, `disk`, each an object with its own state. The current hidden `history_flush_failing` contribution is eliminated: nothing may degrade the rollup without appearing in the response.
`GET /api/diagnostics?state=&severity=&component=&since=&until=&limit=&before=` returns `{events[], next_before, active:{warnings, errors}}`. `GET /api/diagnostics/{id}` returns one event or 404 after retention. No acknowledgement, dismissal, generic-action or raw-log endpoints.
`GET /api/queries` keeps keyset pagination and its filters; rows gain `rcode`, `route_kind`, `policy_action` and the short policy reason the table needs, and the body gains `coverage: {complete, available_since}`. `GET /api/queries/{id}` returns nested `request` / `policy` / `route` / `response` provenance. `GET /api/queries/live` sends the same object without `id`.
`GET /api/stats` and `/api/stats/timeseries` add `complete` and `available_since`.
Existing mutation endpoints stay specific. Diagnostics introduces no generic "perform remediation" endpoint; it invokes the existing blocklist-refresh and certificate-reload operations.
## Schema changes
`config.db`:
```sql
CREATE TABLE operational_events (
id INTEGER PRIMARY KEY,
code TEXT NOT NULL,
subject_key TEXT NOT NULL,
subject_label TEXT NOT NULL,
severity TEXT NOT NULL CHECK (severity IN ('warning', 'error')),
first_seen INTEGER NOT NULL,
last_seen INTEGER NOT NULL,
occurrences INTEGER NOT NULL CHECK (occurrences > 0),
resolved_at INTEGER,
detail TEXT NOT NULL DEFAULT '',
CHECK (resolved_at IS NULL OR resolved_at >= first_seen)
);
CREATE UNIQUE INDEX idx_operational_events_active
ON operational_events(code, subject_key)
WHERE resolved_at IS NULL;
CREATE INDEX idx_operational_events_last_seen
ON operational_events(last_seen DESC);
```
Absent from the configuration table lists and the reconciliation delete order.
`querylog.db`: `querylog_meta` and the provenance columns above. No new index — Codex proposed `idx_query_log_rcode`, cut on review, because every rcode question the UI asks is time-scoped and `idx_query_log_ts` already bounds the scan. 700k rows on a Pi 5 do not need a second index for a rare filter, and every index taxes the hot insert path in `logger.zig`.
No new provenance table, no key/value store — household retention makes nullable columns cheaper than a normalized graph of decision objects.
### Entry buffer widths
`logger.zig`'s `Entry` uses fixed buffers sized by `max_reason_len = 32` and `max_upstream_len = 64`, and travels through the `Io.Queue` by value. The new fields roughly triple it: `matched` holds a full pattern, and `cname_target`, `safe_search_target` and `forward_zone` each hold up to 253 bytes. That is fine at `flush_batch = 100`, but the widths are part of this contract and must be set explicitly, not left to whatever the first implementation picks.
`transformed()` must hide `matched`, `cname_target` and `safe_search_target` under `hide_domains`, not only `domain`.
On recreation: query rows, provenance and upstream-minute history reset together as today; the old file stays aside; `config.db` diagnostics survive; a resolved `query_log.recreated` event records the reason, the aside path and the new coverage start; Overview and Activity report the incomplete range instead of charting zero.
## Deletions and their cost
| Deleted | Lost |
| --- | --- |
| Separate Query Log and Live pages | separate bookmarks; both modes remain in Activity |
| Standalone Lookup page | a top-level bookmark; testing remains under Activity |
| Top-level Groups, Blocklists, Rules, Local DNS, Upstreams, Settings | direct resource navigation; all capabilities remain under task-shaped configuration |
| Historical upstream table on Overview | at-a-glance period rates; availability stays, failures move to Diagnostics |
| "last failure · 9h ago" text | nothing actionable; the episode becomes a diagnostic |
| Detailed DB/log byte gauges | exact component sizes stay in Prometheus; free space stays on Overview |
| Standalone cache card | one prominent number; cache stays in the timeline and metrics |
| Ephemeral blocklist `SourceStatusSection` | transient success detail after navigation; durable counters and diagnostics remain |
| Disabled configuration forms in file mode | the illusion that fields can be edited |
| Global read-only banner | repeated warning text; authority stays visible once in the shell |
| Client-only restart banner state | nothing reliable; server-owned `restart_pending` replaces it |
| Old route aliases and redirects | existing bookmarks break; no permanent duplicate routing layer |
## Build sequence
Each step leaves the app working and shippable, and updates its OpenAPI contract, generated sample, TypeScript types and deterministic tests before landing.
1. **Diagnostics vertical slice.** `operational_events` schema, repository, serialized store, retention, health state, every typed emitter, API, UI, navigation. Recovery paths and deterministic failure-injection tests per event code. The rest of the UI stays intact.
2. **Query provenance vertical slice.** Query-log fingerprint and schema, metadata table, provenance capture in the handler and logger, parsed-SERVFAIL logging, detail and coverage APIs, the historical detail route, and the `query_log.recreated` emission. Existing list summary fields stay so the current pages keep working.
**This step destroys the existing query history.** The provenance DDL edit changes the fingerprint, so `querylog_schema.open` recreates the file and keeps the old one aside as `querylog.db.schema-changed-<unix seconds>`. Acceptable pre-v0.1, and `available_since` carries the story in the UI, but it is a consequence of this step and must be stated in its spec and its changelog entry. Its acceptance tests cover the recreate, the aside name and the coverage sequence — which is also the natural first `query_log.recreated` emission.
Parsed-SERVFAIL logging reverses ruling 20: `handler.zig:507` counts today rather than logging. The `Context` exists at every `servFail` site that follows question parsing. Pre-parse failures correctly stay counters.
3. **Activity consolidation.** The unified History/Live surface, URL filters, freeze/follow, live detail, current-policy test, historical detail links. Query Log, Live and Lookup routes and code are removed in the same change. Route, SSE, accessibility, reconnect and bounded-buffer tests.
4. **Overview replacement.** Current status, active diagnostics, one coherent activity section. New health contract and completeness states. The upstream-history table, stale-failure text, detailed DiskCard and cache card go.
5. **Task-shaped configuration and file mode.** `/api/config/status` and server-owned `restart_pending`. Protection, Resolution and System in both read-only and editable forms. Clients and its detail route redesigned. Old configuration routes replaced atomically; global banner and disabled forms removed.
6. **Contract closure.** Remove obsolete queries, types, stores, CSS, tests and route fixtures. Regenerate contract samples, update OpenAPI and reference docs, add cross-surface acceptance tests for investigation links, file authority, query-log recreation, active-event recovery and time bounds. Zig, frontend, integration, accessibility and byte-budget checks; no new dependency.
## Codex's least-certain calls
- **Clients in primary navigation.** It earns the slot if identifying unknown devices and checking their group is routine. If Mokhtar almost always reaches a client from a query, Clients moves under Protection and leaves the nav.
- **Recording the exact selected upstream per query.** It materially improves correlating SERVFAIL queries with upstream events, but needs the pool exchange result to expose the selected target. Drop back to `"pool"` only if that interface change proves invasive and exact resolver identity never changes an action.
- **90-day / 5,000-event retention.** Conservative fixed bounds, not settings. Change only after measuring real row size and event rate on the Pi; do not add configurable retention pre-emptively.
## Rulings (Fable review, 2026-08-20)
Verdict: build it, with the four cuts folded in above and the rulings below.
**Diagnostics live in `config.db`. Accepted.** The `querylog.db` alternative is self-refuting: that file is recreated on any schema edit or corruption, so the recreation event dies with the thing it describes. A third database needs either its own migration discipline or a recreate policy that loses the events — the same problem with more files. The write-traffic objection is overstated twice: event volume is failure-rate volume, deduplicated, with no `info` severity, so it is near zero in steady state; and `config.db` already takes operational writes, because client rows are materialized from traffic (PLAN §3.5, `clients.last_seen`, `learned_name`). Pre-v0.1 the table is one edit to `ddl_v1` with no migration step. Keeping it out of `delete_order` and `table_names` is enforceable — `config_schema.zig` has tests pinning those lists.
**`group_name`, `source_name` and `matched` stay TEXT. Accepted, and my inconsistency objection was weaker than I put it.** The `domains` table exists because a domain appears on every row, runs to 253 bytes, and feeds `GROUP BY` stats. None of that holds here. The closer precedent is `client_ip`, which is TEXT with the comment "not a FK: log rows are immutable facts" (`querylog_schema.zig:37`). `group_name` is short and mostly `default`; `source_name` and `matched` are non-NULL only on blocked rows; `matched` has cardinality high enough that normalizing buys nothing. Keep both halves of each id/name pair: the id links to the same object across a rename, the name survives a delete, and the detail view needs both.
**Provenance capture does not violate the pure core.** Verified against the code. The policy decision is made in `src/server/handler.zig`, which is already impure and already holds `Io`, the clock and the log call. `matcher.Decision` (`matcher.zig:39-51`) already returns `reason`, `matched` and `source`; the handler throws `matched` and `source` away at `handler.zig:501`. Capture is mostly widening `logger.zig`'s `Entry` and `LogFields`, not threading state through `dns/`, `filter/`, `local/` or `cache/`.
**Six milestones under the harness, one per step.** Do not fold step 6 into step 5: the closure sweep regenerates the contract samples and adds the cross-surface acceptance tests, and it deserves its own verify gate. If anything needs splitting it is step 1, which touches about ten subsystems — store, API and UI first, then the emitters. An event store with three emitters is already shippable and honest.
**One caution, not a blocker.** Diagnostics will be an empty page most of the year. The "No active operational issues" line has to make empty read as healthy, not broken.
+395 -26
View File
@@ -44,10 +44,12 @@ const doh_client = @import("upstream/doh_client.zig");
const doh_server = @import("server/doh_server.zig");
const dot_client = @import("upstream/dot_client.zig");
const dot_server = @import("server/dot_server.zig");
const events = @import("storage/events.zig");
const faults = @import("config/faults.zig");
const fetcher = @import("filter/fetcher.zig");
const forward_zones = @import("local/forward_zones.zig");
const handler = @import("server/handler.zig");
const history_mod = @import("upstream/history.zig");
const http_util = @import("web/http_util.zig");
const loader = @import("config/loader.zig");
const local_records = @import("local/records.zig");
@@ -68,6 +70,7 @@ const shutdown = @import("server/shutdown.zig");
const sse = @import("web/sse.zig");
const static = @import("web/static.zig");
const tcp_server = @import("server/tcp_server.zig");
const upstream_history_repo = @import("storage/repositories/upstream_history_repo.zig");
const transport = @import("upstream/transport.zig");
const udp_server = @import("server/udp_server.zig");
const validate = @import("config/validate.zig");
@@ -152,6 +155,7 @@ fn reconcileFromFile(
config_db: *db.Db,
dir: std.Io.Dir,
config_path: []const u8,
config_load: ?*ConfigLoad,
) !i64 {
return reconcileFromFileAt(
r,
@@ -159,6 +163,7 @@ fn reconcileFromFile(
dir,
config_path,
std.Io.Clock.real.now(r.io).toSeconds(),
config_load,
);
}
@@ -182,6 +187,7 @@ fn reconcileFromFileAt(
dir: std.Io.Dir,
config_path: []const u8,
pass_now: i64,
config_load: ?*ConfigLoad,
) !i64 {
var arena_state: std.heap.ArenaAllocator = .init(r.gpa);
defer arena_state.deinit();
@@ -201,9 +207,32 @@ fn reconcileFromFileAt(
diags.writeAll(r.err) catch {};
r.err.flush() catch {};
// Warnings only. A `.fail` rejects the file and the process exits, so there
// is nobody left to read a diagnostics row about it; a warning is the case
// where the box serves on with a setting the operator did not mean.
if (config_load) |collector| {
for (diags.problems.items) |problem| {
if (problem.severity != .warn) continue;
collector.note(problem.path, problem.path, problem.message);
}
}
return result;
}
/// An upstream's identity is its url: the whole url is the key, and the
/// redaction is the label, because a url can carry an account token.
fn noteUpstream(config_load: *ConfigLoad, url: []const u8, message: []const u8) void {
var label_buf: [events.Store.max_subject_label_len]u8 = undefined;
const label = std.fmt.bufPrint(&label_buf, "{f}", .{safe_url.redact(url)}) catch &label_buf;
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&detail_buf, "upstream {f} {s}", .{
safe_url.redactQuoted(url),
message,
}) catch &detail_buf;
config_load.note(url, label, detail);
}
/// Returns the moment the transaction committed, which is what the settings
/// envelope reports as `reconciled_at`.
fn applyManagedFile(
@@ -316,6 +345,55 @@ fn printSummary(
try r.out.flush();
}
/// The `configuration.load` findings of one boot.
///
/// This code is boot-finalized: nothing during the run can fix a setting, so a
/// restart is its recovery. Every finding is reported as it is made and its key
/// kept, and one `resolveExcept` after the last of them closes the episodes of
/// settings that were wrong last boot and are not wrong now.
const ConfigLoad = struct {
store: ?*events.Store,
io: std.Io,
now_s: i64,
keys: [events.Store.max_kept_keys][events.Store.max_subject_key_len]u8 = undefined,
lens: [events.Store.max_kept_keys]u16 = @splat(0),
len: usize = 0,
/// Set when a boot produced more distinct findings than `resolveExcept`
/// carries. The bulk resolve is then refused rather than truncated: a stale
/// episode left open is honest, and one that is still true closed is not.
/// The refusal goes through the store, so it is counted and latched.
overflowed: bool = false,
fn note(self: *ConfigLoad, key: []const u8, label: []const u8, detail: []const u8) void {
const store = self.store orelse return;
store.report(self.io, self.now_s, .configuration_load, key, label, .warning, detail);
self.keep(key);
}
fn keep(self: *ConfigLoad, key: []const u8) void {
var canon_buf: [events.Store.max_subject_key_len]u8 = undefined;
const canon = events.canonicalKey(key, &canon_buf);
for (0..self.len) |i| {
if (std.mem.eql(u8, self.keys[i][0..self.lens[i]], canon)) return;
}
if (self.len == self.keys.len) {
self.overflowed = true;
return;
}
@memcpy(self.keys[self.len][0..canon.len], canon);
self.lens[self.len] = @intCast(canon.len);
self.len += 1;
}
fn finalize(self: *ConfigLoad) void {
const store = self.store orelse return;
if (self.overflowed) return store.refuseResolveExcept(self.io, self.now_s);
var kept: [events.Store.max_kept_keys][]const u8 = undefined;
for (0..self.len) |i| kept[i] = self.keys[i][0..self.lens[i]];
store.resolveExcept(self.io, self.now_s, .configuration_load, kept[0..self.len]);
}
};
fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
const io = r.io;
const gpa = r.gpa;
@@ -332,13 +410,29 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
defer config_db.close();
_ = try migrations.migrate(&config_db);
// Opened unconditionally, not gated on `cfg.web.enabled`: diagnostics record
// what went wrong whether or not anyone is running the UI, and the store
// owns this connection outright (`storage/events.zig`).
var events_db = try data.openConfigDb(io);
defer events_db.close();
const boot_now_s = std.Io.Clock.real.now(io).toSeconds();
var event_store_storage: ?events.Store = events.Store.init(io, &events_db, boot_now_s) catch |err| blk: {
// No store rather than a store on a mirror it could not verify: the
// latter answers `resolve` with confident no-ops. `/api/health` reports
// the absence as `unavailable` and degrades on it.
log.warn("diagnostics store unavailable: {s}", .{@errorName(err)});
break :blk null;
};
const event_store: ?*events.Store = if (event_store_storage) |*s| s else null;
var config_load: ConfigLoad = .{ .store = event_store, .io = io, .now_s = boot_now_s };
// Ruling 1: the presence of `--config` is the whole authority decision. With
// it, the file is the sole declarative source and the database is converged
// onto it here, before anything reads the database. Without it the database
// is authority and this step does not exist — a file on disk that no flag
// names changes nothing.
const reconciled_at: ?i64 = if (args.config) |config_path|
try reconcileFromFile(r, &config_db, std.Io.Dir.cwd(), config_path)
try reconcileFromFile(r, &config_db, std.Io.Dir.cwd(), config_path, &config_load)
else
null;
@@ -435,7 +529,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
defer bundle.deinit(gpa);
var bundle_lock: std.Io.RwLock = .init;
var upstreams = try Upstreams.build(gpa, cfg.upstreams, &dns_http, &bundle, &bundle_lock);
var upstreams = try Upstreams.build(gpa, cfg.upstreams, &dns_http, &bundle, &bundle_lock, &config_load);
defer upstreams.deinit(gpa);
var pool: pool_mod.Pool = .init(
@@ -448,6 +542,15 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
@truncate(@as(u96, @bitCast(std.Io.Clock.real.now(io).nanoseconds))),
);
// On the heap, not in this frame: the accumulator carries its pending cells
// and the flush task's buffer inline, which is about a megabyte.
const history = try gpa.create(history_mod.Accumulator);
defer gpa.destroy(history);
history.* = .init;
history.diagnostics = event_store;
pool.history = history;
pool.diagnostics = event_store;
// -----------------------------------------------------------------------
// per-query state
// -----------------------------------------------------------------------
@@ -463,15 +566,18 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
var paused: pause.Pause = .{};
var tracker: clients.Tracker = .init(cfg.logging.retention_days);
tracker.diagnostics = event_store;
// Naming rides the tracker's pass, on the tracker's task and connection
// (milestone-25 ruling 1), and reads the live forward zones.
var client_names_resolver: client_names.Resolver = .init(&tables);
client_names_resolver.diagnostics = event_store;
// The queue holds waiting tasks in intrusive lists, so neither the buffer
// nor the `Logger` may move once a task has touched either.
const queue_buf = try gpa.alloc(logger_mod.Entry, cfg.logging.query_log_buffer_max);
defer gpa.free(queue_buf);
var query_logger: logger_mod.Logger = .init(cfg.logging, queue_buf);
query_logger.diagnostics = event_store;
// Milestone 8 fans every logged query out to the SSE hub as well. The hub
// exists only when the web interface does (ruling 6) — without it the sink
@@ -501,6 +607,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// Ruling 17. The scheduler consults it before every scheduled pass; the
// startup `reload` below is an operator action and stays ungated.
manager.monitor = &monitor;
manager.diagnostics = event_store;
// One synchronous sample before anything can consult the gate. `Monitor`
// initializes to `.ok`, and `Monitor.run` takes its first sample inside the
@@ -516,14 +623,41 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// fails therefore leaves the state at `.ok` — an unreadable filesystem is
// not evidence that the disk is full, which is the monitor's documented
// policy and the right one here too.
monitor.sample(io);
monitor.sample(io, event_store, boot_now_s);
var retention: retention_mod.Retention = .init(cfg.logging);
var querylog_writer_db = try data.openQuerylogDb(io);
var querylog_opened = try data.openQuerylogDb(io);
var querylog_writer_db = querylog_opened.database;
defer querylog_writer_db.close();
// One-shot and already over: the file was recreated during this boot, and
// there is nothing to recover from. Never emitted for `.missing` — a first
// creation renames nothing aside, so the event would carry an aside path
// that does not exist and would greet every fresh install with a warning.
if (querylog_opened.recreated) |cause| {
if (cause != .missing) {
if (event_store) |store| {
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&detail_buf, "previous file kept as '{s}'", .{
querylog_opened.aside(),
}) catch detail_buf[0..];
store.reportResolved(
io,
boot_now_s,
.query_log_recreated,
@tagName(cause),
@tagName(cause),
.warning,
detail,
);
}
}
}
var querylog_retention_db = try data.reopenQuerylogDb(io);
defer querylog_retention_db.close();
var querylog_history_db = try data.reopenQuerylogDb(io);
defer querylog_history_db.close();
var tracker_db = try data.openConfigDb(io);
defer tracker_db.close();
@@ -563,6 +697,17 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
"loading the blocklist snapshot failed ({s}); serving unfiltered until the next refresh",
.{@errorName(err)},
);
// No manager lock is held here, so this reports directly rather than
// through the manager's collector.
if (event_store) |store| {
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(
&buf,
"loading the blocklist snapshot failed: {s}",
.{@errorName(err)},
) catch buf[0..];
store.report(io, boot_now_s, .blocklist_snapshot, "snapshot", "blocklist snapshot", .@"error", detail);
}
};
// -----------------------------------------------------------------------
@@ -598,12 +743,16 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
defer if (doh_certs) |*store| store.deinit(io);
if (cfg.doh_server.enabled) {
doh_certs = try openCertStore(r, gpa, io, cfg.doh_server, "doh_server", doh_server.alpn_protocols);
doh_certs.?.kind = .doh;
doh_certs.?.diagnostics = event_store;
}
var dot_certs: ?cert_store.CertStore = null;
defer if (dot_certs) |*store| store.deinit(io);
if (cfg.dot_server.enabled) {
dot_certs = try openCertStore(r, gpa, io, cfg.dot_server, "dot_server", dot_alpn);
dot_certs.?.kind = .dot;
dot_certs.?.diagnostics = event_store;
}
// Bound here, in this frame, rather than through doh_server's module-level
@@ -611,13 +760,35 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// only a listener that lives in this frame has an address to wire there.
// A failed bind warns and stays off (ruling 1, the web precedent): TLS DNS
// failing to come up must not stop the plain-DNS side this box exists for.
var failed_listeners: [2][]const u8 = undefined;
var failed_listener_count: usize = 0;
var doh: ?doh_server.DohServer = null;
defer if (doh) |*server| server.deinit(io);
if (doh_certs) |*store| doh = bindDoh(gpa, io, cfg.doh_server, &h, store);
if (doh_certs) |*store| {
doh = bindDoh(gpa, io, cfg.doh_server, &h, store, event_store, boot_now_s);
if (doh == null) {
failed_listeners[failed_listener_count] = "doh";
failed_listener_count += 1;
}
}
var dot: ?dot_server.DotServer = null;
defer if (dot) |*server| server.deinit(io);
if (dot_certs) |*store| dot = bindDot(gpa, io, cfg.dot_server, &h, store);
if (dot_certs) |*store| {
dot = bindDot(gpa, io, cfg.dot_server, &h, store, event_store, boot_now_s);
if (dot == null) {
failed_listeners[failed_listener_count] = "dot";
failed_listener_count += 1;
}
}
// Boot-finalized: one call closes whatever the last boot left open for an
// endpoint that started clean this time — including an endpoint now
// disabled, which contributes no key and so is not kept.
if (event_store) |store| {
store.resolveExcept(io, boot_now_s, .listener_start, failed_listeners[0..failed_listener_count]);
}
// -----------------------------------------------------------------------
// web interface (ruling 26)
@@ -646,6 +817,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
.local_tables = &tables,
.logger = &query_logger,
.retention = &retention,
.history = history,
.sessions = if (sessions) |*s| s else null,
.limiter = if (web_limiter) |*l| l else null,
.hub = hub,
@@ -656,6 +828,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
.dot_listener = if (dot) |*server| server else null,
.config_db = if (web_config_db) |*database| database else null,
.querylog_db = if (web_querylog_db) |*database| database else null,
.events = event_store,
.version = version.string,
.started_unix = std.Io.Clock.real.now(io).toSeconds(),
// Ruling 24: `--admin-dev` serves from disk with no cache headers;
@@ -672,6 +845,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
var udp6: ?udp_server.UdpServer = udp_server.UdpServer.bind(gpa, io, v6_bind, &h, .{}) catch |err| bound: {
if (!ipv6Unavailable(err)) return reportBind(r, "udp", v6_bind, err);
log.warn("this system has no IPv6; serving IPv4 only", .{});
config_load.note("dns.bind_ipv6", "dns.bind_ipv6", "this system has no IPv6; serving IPv4 only");
break :bound null;
};
defer if (udp6) |*s| s.deinit(gpa, io);
@@ -700,6 +874,12 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
};
defer if (tcp4) |*s| s.deinit(io);
// Every `configuration.load` finding of this boot is in by here: the managed
// file, the upstream table and the IPv6 bind above. Finalizing any earlier
// would resolve an episode this boot is about to reopen, so consecutive
// IPv6-less boots would read as a new episode each time.
config_load.finalize();
// Ruling 13: `/metrics` sums each transport's listeners into one family, so
// the web state carries pointers to whichever of the four came up. The
// arrays are declared here rather than beside `web_state` because a
@@ -735,12 +915,49 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
shutdown.install(io);
// Declared after everything it borrows, so its `cancel` — which both
// requests cancellation and joins — is the first thing that runs on the way
// out (ruling 22). Nothing below this line may be released while a task
// could still touch it.
// Declared after everything it borrows, so the teardown below — whose
// `cancel` both requests cancellation and joins — is the first thing that
// runs on the way out (ruling 22). Nothing below this line may be released
// while a task could still touch it.
var group: std.Io.Group = .init;
defer group.cancel(io);
// The gate every non-essential write consults. Reading it before the
// monitor's own task has sampled is safe: a fresh `Monitor` publishes `.ok`
// (disk_monitor.zig:63), so nothing is refused for want of a sample.
const gate: ?*disk_monitor.Monitor = &monitor;
// The query-log writer is deliberately *not* in `group`, and starts before
// every producer. Inside the group its life would end with the same
// `cancel` that stops the producers, and cancellation would race the
// queue's close: whichever landed first decided whether the batch the
// writer was holding reached the database or was counted as dropped. Given
// its own future, it outlives the producers by construction, and the
// teardown below can close the queue with nobody left to fill it and then
// wait for the writer to finish emptying it.
var writer_future = try io.concurrent(
logger_mod.Logger.runWriter,
.{ &query_logger, io, &querylog_writer_db, gate },
);
// Ruling 4's shutdown order, on the one path every exit from here takes:
// every producer stops and is joined, then the queue closes, then the
// writer is awaited — so the last batch is written rather than raced — and
// only then does the final history flush run, with no recording task left
// that could add a cell after it. A writer the disk gate will not let write
// counts its batch as dropped instead of holding the exit open
// (`logger.zig`), so this wait always ends.
//
// A `defer` and not straight-line code after `shutdown.wait`, because a
// `concurrent` spawn below can fail with the DNS listeners already
// serving; an orderly error teardown owes the operator the same drain a
// signal gets. The `querylog_history_db` this flush writes through is
// declared above, so its `close` runs after it.
defer {
group.cancel(io);
query_logger.shutdown(io);
writer_future.await(io) catch {};
history.flushOnce(io, &querylog_history_db, upstream_history_repo.flush);
}
if (udp6) |*s| try group.concurrent(io, udp_server.UdpServer.serve, .{ s, io });
if (udp4) |*s| try group.concurrent(io, udp_server.UdpServer.serve, .{ s, io });
@@ -751,10 +968,11 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
if (doh_certs) |*store| try group.concurrent(io, cert_store.CertStore.watch, .{ store, io });
if (dot_certs) |*store| try group.concurrent(io, cert_store.CertStore.watch, .{ store, io });
const gate: ?*disk_monitor.Monitor = &monitor;
try group.concurrent(io, logger_mod.Logger.runWriter, .{ &query_logger, io, &querylog_writer_db, gate });
try group.concurrent(io, retention_mod.Retention.run, .{ &retention, io, &querylog_retention_db, gate });
try group.concurrent(io, disk_monitor.Monitor.run, .{ &monitor, io });
try group.concurrent(io, retention_mod.Retention.run, .{ &retention, io, &querylog_retention_db, gate, event_store });
// Ungated: a flush writes at most one row per upstream per minute, the same
// category as the query logger's own writes, which are ungated too.
try group.concurrent(io, history_mod.Accumulator.run, .{ history, io, &querylog_history_db });
try group.concurrent(io, disk_monitor.Monitor.run, .{ &monitor, io, event_store });
try group.concurrent(io, manager_mod.Manager.runScheduler, .{ &manager, io });
try group.concurrent(io, clients.Tracker.run, .{ &tracker, io, &tracker_db, gate, &client_names_resolver });
try group.concurrent(io, runMaintenance, .{ &h, if (web_limiter) |*l| l else null, io });
@@ -774,14 +992,11 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
});
// A canceled wait is a shutdown request too: whoever canceled this task
// wants the process to stop, and the teardown below is how it stops.
// wants the process to stop, and returning into the teardown deferred above
// is how it stops.
shutdown.wait(io) catch {};
log.info("shutting down", .{});
// Before the group is canceled, so the writer sees a closed queue and
// drains what it holds rather than losing it to cancellation (ruling 22).
query_logger.shutdown(io);
return cli.exit_ok;
}
@@ -842,6 +1057,23 @@ fn openCertStore(
};
}
/// An error, not a warning: an endpoint the operator enabled is not serving.
fn reportListener(
store: ?*events.Store,
io: std.Io,
now_s: i64,
kind: []const u8,
comptime fmt: []const u8,
args: anytype,
) void {
const s = store orelse return;
var buf: [events.Store.max_detail_len]u8 = undefined;
var w: std.Io.Writer = .fixed(&buf);
w.print("{s} listener ", .{kind}) catch {};
w.print(fmt, args) catch {};
s.report(io, now_s, .listener_start, kind, kind, .@"error", w.buffered());
}
/// Ruling 1: a listener that cannot bind warns and stays off. The bind text
/// itself gets the same treatment — `validate` refuses it, but a hand-edited
/// database can still carry one, and it is not worth taking DNS down over.
@@ -851,13 +1083,17 @@ fn bindDoh(
endpoint: model.TlsEndpoint,
h: *handler.Handler,
store: *cert_store.CertStore,
diagnostics: ?*events.Store,
now_s: i64,
) ?doh_server.DohServer {
const bind_address = net.IpAddress.parse(endpoint.bind, endpoint.port) catch {
log.warn("doh_server.bind '{s}' is not an IP address; DoH is disabled", .{endpoint.bind});
reportListener(diagnostics, io, now_s, "doh", "bind '{s}' is not an IP address", .{endpoint.bind});
return null;
};
const server = doh_server.DohServer.listen(gpa, io, bind_address, h, store, .{}) catch |err| {
log.warn("doh listener cannot listen on {s}:{d}: {t}", .{ endpoint.bind, endpoint.port, err });
reportListener(diagnostics, io, now_s, "doh", "cannot listen on {s}:{d}: {t}", .{ endpoint.bind, endpoint.port, err });
return null;
};
log.info("doh listener on {f}", .{server.boundAddress()});
@@ -870,13 +1106,17 @@ fn bindDot(
endpoint: model.TlsEndpoint,
h: *handler.Handler,
store: *cert_store.CertStore,
diagnostics: ?*events.Store,
now_s: i64,
) ?dot_server.DotServer {
const bind_address = net.IpAddress.parse(endpoint.bind, endpoint.port) catch {
log.warn("dot_server.bind '{s}' is not an IP address; DoT is disabled", .{endpoint.bind});
reportListener(diagnostics, io, now_s, "dot", "bind '{s}' is not an IP address", .{endpoint.bind});
return null;
};
const server = dot_server.DotServer.listen(gpa, io, bind_address, h, store, .{}) catch |err| {
log.warn("dot listener cannot listen on {s}:{d}: {t}", .{ endpoint.bind, endpoint.port, err });
reportListener(diagnostics, io, now_s, "dot", "cannot listen on {s}:{d}: {t}", .{ endpoint.bind, endpoint.port, err });
return null;
};
log.info("dot listener on {f}", .{server.boundAddress()});
@@ -966,6 +1206,7 @@ const Upstreams = struct {
http: *std.http.Client,
bundle: *Certificate.Bundle,
bundle_lock: *std.Io.RwLock,
config_load: *ConfigLoad,
) (Allocator.Error || error{NoUsableUpstreams})!Upstreams {
var enabled: usize = 0;
for (servers) |server| {
@@ -1001,6 +1242,7 @@ const Upstreams = struct {
"upstream {f} is not an https:// or tls:// endpoint; skipped",
.{safe_url.redactQuoted(server.url)},
);
noteUpstream(config_load, server.url, "not an https:// or tls:// endpoint; skipped");
continue;
};
@@ -1018,6 +1260,7 @@ const Upstreams = struct {
"upstream {f} is not a usable DoH url; skipped",
.{safe_url.redactQuoted(server.url)},
);
noteUpstream(config_load, server.url, "not a usable DoH url; skipped");
continue;
};
doh_count += 1;
@@ -1114,6 +1357,9 @@ fn parseBind(
return addr;
}
const events_fixture = @import("storage/events_fixture.zig");
const testing = std.testing;
test "parseBind refuses a bind address of the wrong family" {
var out_buf: [8]u8 = undefined;
var err_buf: [256]u8 = undefined;
@@ -1285,7 +1531,7 @@ test "a start in file mode prints the warnings the file earned" {
// The file is valid, so the start succeeds and the database converges onto
// it. The returned stamp is what the settings envelope reports.
const reconciled_at = try reconcileFromFile(r, &database, tmp.dir, "config.zon");
const reconciled_at = try reconcileFromFile(r, &database, tmp.dir, "config.zon", null);
try std.testing.expect(reconciled_at > 0);
try std.testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM upstreams"));
@@ -1391,7 +1637,7 @@ test "reconciled_at is stamped after the commit, not from the clock the pass wro
const pass_now: i64 = 42;
const before = std.Io.Clock.real.now(io).toSeconds();
const reconciled_at = try reconcileFromFileAt(r, &database, tmp.dir, "config.zon", pass_now);
const reconciled_at = try reconcileFromFileAt(r, &database, tmp.dir, "config.zon", pass_now, null);
// The pinned clock reached the engine, so the two values really are separate
// inputs rather than the same read twice.
@@ -1434,7 +1680,7 @@ test "the startup summary reports what the reconcile changed, then that nothing
var err: Writer = .fixed(&err_buf);
const r: cli.Runner = .{ .io = io, .gpa = gpa, .out = &out_writer.interface, .err = &err };
_ = try reconcileFromFile(r, &database, tmp.dir, "config.zon");
_ = try reconcileFromFile(r, &database, tmp.dir, "config.zon", null);
{
// Read back through the file: `serve` does not return for as long as the
// service runs, so a summary still in the buffer is a summary nobody
@@ -1451,7 +1697,7 @@ test "the startup summary reports what the reconcile changed, then that nothing
}
try tmp.dir.writeFile(io, .{ .sub_path = "stdout.txt", .data = "" });
_ = try reconcileFromFile(r, &database, tmp.dir, "config.zon");
_ = try reconcileFromFile(r, &database, tmp.dir, "config.zon", null);
{
const printed = try tmp.dir.readFileAlloc(io, "stdout.txt", gpa, .limited(8192));
defer gpa.free(printed);
@@ -1508,7 +1754,7 @@ test "a broken error writer does not replace the reason a managed file was rejec
try std.testing.expectError(
error.MissingDefaultGroup,
reconcileFromFile(r, &database, tmp.dir, "config.zon"),
reconcileFromFile(r, &database, tmp.dir, "config.zon", null),
);
// Empty, so the writer did fail — without this the assertion above would
@@ -1554,7 +1800,7 @@ test "a broken error writer does not stop a start whose file applied" {
var err_writer = brokenErrWriter(io, err_file, &err_buf);
const r: cli.Runner = .{ .io = io, .gpa = gpa, .out = &out, .err = &err_writer.interface };
_ = try reconcileFromFile(r, &database, tmp.dir, "config.zon");
_ = try reconcileFromFile(r, &database, tmp.dir, "config.zon", null);
try std.testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM upstreams"));
// Nothing reached the file, so the flush really did fail.
@@ -1739,3 +1985,126 @@ test "one maintenance pass drops the api limiter's stale buckets" {
// A limiter the app did not build is not a reason for the pass to fail.
try maintenanceOnce(&h, null, io);
}
test "a failing listener bind opens an error episode a clean boot closes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
reportListener(&fx.store, io, 1000, "dot", "cannot listen on {s}:{d}: {t}", .{
"0.0.0.0",
@as(u16, 853),
error.AddressInUse,
});
try testing.expectEqualStrings("listener.start", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("dot", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("error", try fx.text("SELECT severity FROM operational_events"));
// The next boot: DoH failed, DoT started clean. One `resolveExcept` over
// the failed keys closes the stale DoT episode and leaves the DoH one open.
reportListener(&fx.store, io, 1100, "doh", "cannot listen on {s}:{d}: {t}", .{
"0.0.0.0",
@as(u16, 443),
error.AddressInUse,
});
fx.store.resolveExcept(io, 1100, .listener_start, &.{"doh"});
try testing.expectEqual(
@as(i64, 1),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
try testing.expectEqualStrings("doh", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
}
test "a boot with no listener finding closes every listener episode" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
fx.store.report(io, 900, .listener_start, "doh", "doh", .@"error", "stale");
fx.store.report(io, 900, .listener_start, "dot", "dot", .@"error", "stale");
fx.store.resolveExcept(io, 1000, .listener_start, &.{});
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a configuration finding is reported once and finalize closes the rest" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
// Two stale episodes from a previous boot.
fx.store.report(io, 900, .configuration_load, "dns.bind_ipv6", "dns.bind_ipv6", .warning, "stale");
fx.store.report(io, 900, .configuration_load, "upstreams[0].url", "upstreams[0].url", .warning, "stale");
var collector: ConfigLoad = .{ .store = &fx.store, .io = io, .now_s = 1000 };
collector.note("dns.bind_ipv6", "dns.bind_ipv6", "this system has no IPv6; serving IPv4 only");
// The same finding twice is one episode and one kept key.
collector.note("dns.bind_ipv6", "dns.bind_ipv6", "this system has no IPv6; serving IPv4 only");
try testing.expectEqual(@as(usize, 1), collector.len);
collector.finalize();
try testing.expectEqual(
@as(i64, 1),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
try testing.expectEqualStrings("dns.bind_ipv6", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
// The stale episode is still the same one: this boot bumped it twice.
try testing.expectEqual(@as(i64, 3), try fx.count(
"SELECT occurrences FROM operational_events WHERE resolved_at IS NULL",
));
}
test "an over-long boot finding list refuses to finalize rather than truncate" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
fx.store.report(io, 900, .configuration_load, "stale.setting", "stale.setting", .warning, "stale");
var collector: ConfigLoad = .{ .store = &fx.store, .io = io, .now_s = 1000 };
var key_buf: [32]u8 = undefined;
for (0..events.Store.max_kept_keys + 1) |i| {
const key = try std.fmt.bufPrint(&key_buf, "upstreams[{d}].url", .{i});
collector.note(key, key, "not a usable url");
}
try testing.expect(collector.overflowed);
collector.finalize();
// Closing more than the boot meant would resolve episodes that are still
// true, so the stale one stays open instead.
try testing.expectEqual(@as(i64, 1), try fx.count(
"SELECT count(*) FROM operational_events WHERE subject_key = 'stale.setting' AND resolved_at IS NULL",
));
// Refused, and refused out loud: the store counted and latched it, so
// `/api/health` reports the diagnostics log as not recording.
try testing.expect(fx.store.writeFailed());
try testing.expectEqual(@as(u64, 1), fx.store.writeFailures());
}
+11 -7
View File
@@ -332,17 +332,21 @@ pub const DataDir = struct {
/// `querylog_schema.open` resolves the path through SQLite's VFS as well as
/// through the directory handle, so it is given `cwd` and the joined path
/// rather than `self.dir` and a name (see its doc comment).
pub fn openQuerylogDb(self: *const DataDir, io: std.Io) !db.Db {
const opened = try querylog_schema.open(io, std.Io.Dir.cwd(), self.querylog_db_path);
var database = opened.database;
errdefer database.close();
/// Returns the whole `OpenResult`, recreate reason and aside name included:
/// the composition root records a recreate as a one-shot diagnostics event,
/// and the aside name is what tells an operator where the old file went.
pub fn openQuerylogDb(self: *const DataDir, io: std.Io) !querylog_schema.OpenResult {
var opened = try querylog_schema.open(io, std.Io.Dir.cwd(), self.querylog_db_path);
errdefer opened.database.close();
try self.restrictQuerylogPermissions(io);
return database;
return opened;
}
/// An additional connection to a `querylog.db` that `openQuerylogDb` has
/// already established. A running server needs two — the log writer and the
/// retention pass each own one (`retention.zig`'s contract).
/// already established. A running server needs three background ones — the
/// log writer, the retention pass and the upstream-history flush each own
/// one (`retention.zig`'s contract) — plus a fourth for the web task when
/// the web interface is enabled.
pub fn reopenQuerylogDb(self: *const DataDir, io: std.Io) !db.Db {
_ = io;
var database = try db.Db.open(self.querylog_db_path, .{ .mode = .read_write_existing });
+50
View File
@@ -151,6 +151,7 @@ fn reportDeletes(diags: *validate.Diagnostics, summary: reconcile.Summary) error
const testing = std.testing;
const config_schema = @import("../storage/config_schema.zig");
const export_mod = @import("export.zig");
const migrations = @import("../storage/migrations.zig");
fn openMigrated() !db.Db {
@@ -282,6 +283,55 @@ test "importSource converges a migrated database and group 'default' keeps id 1"
);
}
test "an export and re-import leaves the operational_events log untouched" {
// `operational_events` is runtime state, not configuration: it is out of
// `table_names` and out of `delete_order`, so an export must not emit it and
// an import's wipe must not reach it. A diagnostics log destroyed by a
// routine `nxdns import` would take the record of what the box has been
// doing with it.
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
const gpa = testing.allocator;
var database = try openMigrated();
defer database.close();
try importText(io, &database, full_source, .{});
try database.exec(
\\INSERT INTO operational_events
\\ (code, subject_key, subject_label, severity, first_seen, last_seen, occurrences, resolved_at)
\\VALUES ('disk.space', 'data', 'data', 'warning', 100, 200, 3, NULL),
\\ ('blocklist.refresh', 'https://a.example', 'A', 'warning', 100, 150, 1, 300);
);
var rendered: std.Io.Writer.Allocating = .init(gpa);
defer rendered.deinit();
try export_mod.writeToWriter(gpa, &database, &rendered.writer);
const source = try gpa.dupeZ(u8, rendered.written());
defer gpa.free(source);
// The export is the whole declared configuration and says nothing about
// the log.
try testing.expect(!std.mem.containsAtLeast(u8, source, 1, "operational_events"));
try testing.expect(!std.mem.containsAtLeast(u8, source, 1, "disk.space"));
// A round trip is by definition delete-free for the config tables, and the
// events survive it untouched, resolved and active alike.
try importText(io, &database, source, .{ .allow_delete = true });
var stmt = try database.prepare(
"SELECT code, occurrences, resolved_at FROM operational_events ORDER BY id",
);
defer stmt.deinit();
try testing.expect(try stmt.step());
try testing.expectEqualStrings("disk.space", stmt.columnText(0));
try testing.expectEqual(@as(i64, 3), stmt.columnInt(1));
try testing.expect(stmt.isNull(2));
try testing.expect(try stmt.step());
try testing.expectEqualStrings("blocklist.refresh", stmt.columnText(0));
try testing.expectEqual(@as(i64, 300), stmt.columnInt(2));
try testing.expect(!try stmt.step());
}
test "an import whose diff deletes rows is refused, names the tables, and changes nothing" {
// Ruling 6: the emptiness guard is gone, so this is what stops
// `nxdns import ./wrong.zon` from emptying a configured database.
+6
View File
@@ -231,6 +231,10 @@ pub const Logging = struct {
level: LogLevel = .info,
retention_days: u16 = 30,
query_log_buffer_max: u32 = 10000,
/// How long the query-log writer gathers entries before it commits them.
/// `0` does not wait at all: it flushes the entry that woke the writer plus
/// whatever is already queued.
query_log_flush_interval_s: u16 = 60,
hide_domains: bool = false,
hide_client_ips: bool = false,
output: LogOutput = .stderr,
@@ -565,6 +569,7 @@ const expected_keys = [_][]const u8{
"logging.max_size_mb",
"logging.output",
"logging.query_log_buffer_max",
"logging.query_log_flush_interval_s",
"logging.retention_days",
"upstream.attempt_timeout_ms",
"upstream.read_timeout_ms",
@@ -657,6 +662,7 @@ test "toSettings and fromSettings round-trip a non-default config" {
.level = .err,
.retention_days = 41,
.query_log_buffer_max = 43,
.query_log_flush_interval_s = 44,
.hide_domains = true,
.hide_client_ips = true,
.output = .file,
+32
View File
@@ -93,6 +93,7 @@ pub const ValidateError = error{
BadTtl,
BadCacheSize,
BadRetention,
BadFlushInterval,
BadLogRotation,
BadDiskThresholds,
BadRateLimit,
@@ -335,6 +336,11 @@ const max_rate_window_seconds = 3_600;
/// the box, and nxdns does not try to know that.
const max_boot_entries = 1_000_000;
/// The ceiling on the query-log flush window. An hour of queries is already
/// more history than a crash is allowed to cost; beyond that the setting stops
/// being a batching knob and becomes a way to lose a working day of rows.
const max_flush_interval_s = 3_600;
fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void {
const up = cfg.upstream;
try checkTimeout(diags, up.attempt_timeout_ms, "upstream.attempt_timeout_ms");
@@ -464,6 +470,16 @@ fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void {
.{ max_boot_entries, cfg.logging.query_log_buffer_max },
);
}
// No floor: 0 is the documented "do not wait" setting, not a mistake.
if (cfg.logging.query_log_flush_interval_s > max_flush_interval_s) {
try diags.add(
error.BadFlushInterval,
"logging.query_log_flush_interval_s",
.{},
"must be at most {d}, got {d}",
.{ max_flush_interval_s, cfg.logging.query_log_flush_interval_s },
);
}
if (cfg.logging.max_size_mb < 1) {
try diags.add(error.BadLogRotation, "logging.max_size_mb", .{}, "must be at least 1", .{});
}
@@ -1946,6 +1962,22 @@ test "error.BadRetention" {
try expectProblem(huge_buffer, error.BadRetention, "logging.query_log_buffer_max");
}
test "error.BadFlushInterval" {
var cfg = baseConfig();
cfg.logging.query_log_flush_interval_s = 3601;
try expectProblem(cfg, error.BadFlushInterval, "logging.query_log_flush_interval_s");
// 0 is the "do not wait" setting and 3600 is the ceiling itself: both are
// legal, and a floor check would reject the first.
var immediate = baseConfig();
immediate.logging.query_log_flush_interval_s = 0;
try expectClean(immediate);
var edge = baseConfig();
edge.logging.query_log_flush_interval_s = max_flush_interval_s;
try expectClean(edge);
}
test "error.BadLogRotation" {
var cfg = baseConfig();
cfg.logging.max_files = 0;
+160 -1
View File
@@ -24,6 +24,7 @@ const net = std.Io.net;
const model = @import("../config/model.zig");
const reconcile = @import("../config/reconcile.zig");
const db = @import("../storage/db.zig");
const events_fixture = @import("../storage/events_fixture.zig");
const migrations = @import("../storage/migrations.zig");
const context = @import("../storage/repositories/context.zig");
const groups_repo = @import("../storage/repositories/groups_repo.zig");
@@ -2328,7 +2329,6 @@ test "23: a name moving from the list body to the wild body forces a republish"
try testing.expectEqual(@as(i64, 0), row.wildcard_count);
@memcpy(&first_checksum, row.checksum orelse return error.TestNoChecksum);
}
}
test "22: a list that changed only its skipped lines still updates both skip counters" {
@@ -2405,3 +2405,162 @@ test "22: a list that changed only its skipped lines still updates both skip cou
try testing.expectEqual(@as(u32, 1), restored.counts.skipped_regex);
try testing.expectEqual(@as(u32, 2), restored.counts.skipped_unsupported);
}
// ---------------------------------------------------------------------------
// diagnostics: the events the manager records (milestone 27)
// ---------------------------------------------------------------------------
test "27: a failing refresh opens a blocklist.refresh episode a good one closes" {
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
const env = try Env.create(gpa);
defer env.destroy();
const io = env.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
env.mgr.diagnostics = &fx.store;
var fixture = try HttpFixture.init(io, http_body);
defer fixture.deinit(io);
fixture.setRoute(.oversize);
var group: std.Io.Group = .init;
defer group.cancel(io);
try group.concurrent(io, HttpFixture.serve, .{ &fixture, io });
var url_buf: [64]u8 = undefined;
const url = try fixture.url(&url_buf);
_ = try seedSource(&env.database, url);
try testing.expect(!try refreshOnce(env, url));
try testing.expectEqualStrings("blocklist.refresh", try fx.text(
"SELECT code FROM operational_events WHERE resolved_at IS NULL",
));
// The whole url, not the display copy: `subject_key` is the identity.
try testing.expectEqualStrings(url, try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings("warning", try fx.text(
"SELECT severity FROM operational_events WHERE resolved_at IS NULL",
));
fixture.setRoute(.body);
try testing.expect(try refreshOnce(env, url));
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE code = 'blocklist.refresh' AND resolved_at IS NULL"),
);
}
test "27: one refreshAll pass records one occurrence of a failing source" {
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
const env = try Env.create(gpa);
defer env.destroy();
const io = env.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
env.mgr.diagnostics = &fx.store;
var fixture = try HttpFixture.init(io, http_body);
defer fixture.deinit(io);
fixture.setRoute(.oversize);
var group: std.Io.Group = .init;
defer group.cancel(io);
try group.concurrent(io, HttpFixture.serve, .{ &fixture, io });
var url_buf: [64]u8 = undefined;
const url = try fixture.url(&url_buf);
_ = try seedSource(&env.database, url);
// `scheduledPass` is what an elapsed interval runs, and it ends in
// `refreshAll`, which ends in a reload. One pass is one flush: a reload
// that flushed on its own, or a scheduler that flushed after a pass that
// already had, would report this failure twice.
try env.mgr.scheduledPass(io);
try testing.expectEqual(@as(i64, 1), try fx.count(
"SELECT occurrences FROM operational_events WHERE code = 'blocklist.refresh'",
));
// A second pass on a still-failing source is one more occurrence: the
// failures one pass held ride the detail, and the flushes a pass makes are
// not what `occurrences` counts.
try env.mgr.scheduledPass(io);
try testing.expectEqual(@as(i64, 2), try fx.count(
"SELECT occurrences FROM operational_events WHERE code = 'blocklist.refresh'",
));
const detail = try fx.text("SELECT detail FROM operational_events WHERE code = 'blocklist.refresh'");
try testing.expect(std.mem.indexOf(u8, detail, "this pass") != null);
}
test "27: a clean reload resolves blocklist.snapshot and records no storage failure" {
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
const env = try Env.create(gpa);
defer env.destroy();
const io = env.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
// A stale episode from a previous boot, which a published snapshot closes.
fx.store.report(io, 900, .blocklist_snapshot, "snapshot", "blocklist snapshot", .@"error", "stale");
env.mgr.diagnostics = &fx.store;
try env.mgr.reload(io);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
try testing.expectEqual(@as(i64, 900), try fx.count("SELECT first_seen FROM operational_events"));
}
test "27: an unreadable blocklist directory opens a blocklist.storage episode" {
if (!build_options.integration) return error.SkipZigTest;
// Mode bits do not apply to root, so the denial the test needs cannot happen.
if (std.c.geteuid() == 0) return error.SkipZigTest;
const gpa = testing.allocator;
const env = try Env.create(gpa);
defer env.destroy();
const io = env.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
env.mgr.diagnostics = &fx.store;
var dir = try env.blocklistDir();
dir.close(io);
try env.tmp.dir.setPermissions(io, .fromMode(0o600));
const failed = env.mgr.pruneOrphans(io);
try env.tmp.dir.setPermissions(io, .fromMode(0o700));
try testing.expectError(error.FileSystem, failed);
try testing.expectEqualStrings("blocklist.storage", try fx.text(
"SELECT code FROM operational_events WHERE resolved_at IS NULL",
));
// `createDirPathStatus` is the first call to touch the unreadable parent,
// so it is the operation that fails.
try testing.expectEqualStrings("create_dir", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
// A pass that can read the directory again closes it.
try env.mgr.pruneOrphans(io);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
+627 -25
View File
@@ -59,6 +59,7 @@ const groups_repo = @import("../storage/repositories/groups_repo.zig");
const rules_repo = @import("../storage/repositories/rules_repo.zig");
const sources_repo = @import("../storage/repositories/sources_repo.zig");
const disk_monitor = @import("../storage/disk_monitor.zig");
const events = @import("../storage/events.zig");
const compiler = @import("compiler.zig");
const fetcher = @import("fetcher.zig");
const matcher = @import("matcher.zig");
@@ -162,6 +163,51 @@ pub const State = enum {
}
};
/// The `blocklist.storage` operations, each its own episode subject.
///
/// A fixed set on purpose: several of these fail once per file in a pass, and
/// one slot per operation is what turns that into one report per pass instead
/// of an unbounded list of them.
pub const StorageOp = enum { sweep, directory_read, create_dir, open_dir, delete };
/// One operation's outcome across one pass. `detail` keeps the last failure,
/// and `failures` says how many that pass held — the row's `occurrences` counts
/// failing passes, so the count belongs in the text.
const Aggregate = struct {
failures: u32 = 0,
succeeded: bool = false,
detail: [events.Store.max_detail_len]u8 = @splat(0),
detail_len: u16 = 0,
fn detailText(self: *const Aggregate) []const u8 {
return self.detail[0..self.detail_len];
}
};
/// What a locked body observed, held until `Manager.flushDiagnostics` can
/// report it with no manager lock held.
///
/// Two of this file's operations cannot report from where they stand:
/// `publishRefresh` runs under `writer_lock` by contract and `pruneOrphans`
/// holds both writer mutexes through its filesystem work. Collect-then-flush is
/// what keeps their outcomes without holding a lock across a store call, and
/// nothing here can grow: the storage slots are an enum array and a refresh
/// outcome rides the status entry the source already has.
///
/// The flush still happens inside the lock that serializes passes — a pass
/// drains its own outcomes before it releases `refresh_lock` (or, for a
/// standalone `reload`, `writer_lock`). What collect-then-flush avoids is
/// holding a *manager* lock across a store call, not deferring the report until
/// the next pass could merge into it.
const Pending = struct {
mutex: std.Io.Mutex = .init,
storage: std.EnumArray(StorageOp, Aggregate) = .initFill(.{}),
/// Null until a pass observes a snapshot outcome at all.
snapshot_failed: ?bool = null,
snapshot_detail: [events.Store.max_detail_len]u8 = @splat(0),
snapshot_detail_len: u16 = 0,
};
/// A status is a value with no borrowed memory, so a copy handed to the API
/// outlives every reload. The url is held inline for that reason.
pub const SourceStatus = struct {
@@ -182,6 +228,32 @@ pub const SourceStatus = struct {
url_len: u8 = 0,
last_error: [max_error_len]u8 = @splat(0),
last_error_len: u8 = 0,
/// The diagnostics identity of this source, canonicalized from the WHOLE
/// url by `setUrl`. `url` above is a display copy truncated at
/// `max_url_len`, and two urls sharing a 255-byte prefix would share one
/// episode if that copy were the key.
event_key: [events.Store.max_subject_key_len]u8 = @splat(0),
event_key_len: u16 = 0,
/// Diagnostics accounting for the pass in progress, cleared by every
/// `flushDiagnostics`. `pass_outcome` says this source recorded one at all;
/// `pass_failures` counts the failing ones, which a pass can hold more than
/// one of (a refresh that failed, then the reload that could not load the
/// files it did not write). One flush reports one `blocklist.refresh`
/// occurrence per source, so `occurrences` counts failing passes rather
/// than flushes, and the detail carries how many failures the pass held.
///
/// These two fields live in exactly one copy of the status table at a time,
/// which is what makes that count right while reloads replace the table
/// underneath: a candidate built by `mergeStatuses` carries none of them,
/// `installStatuses` folds the live table's in as it swaps, and the flush
/// claims an entry by copying it and zeroing both fields in one locked
/// step. Copy them anywhere else and the outcome gets reported twice.
pass_outcome: bool = false,
pass_failures: u16 = 0,
pub fn eventKey(self: *const SourceStatus) []const u8 {
return self.event_key[0..self.event_key_len];
}
pub fn errorText(self: *const SourceStatus) []const u8 {
return self.last_error[0..self.last_error_len];
@@ -196,9 +268,12 @@ pub const SourceStatus = struct {
@memcpy(self.url[0..kept], url[0..kept]);
@memset(self.url[kept..], 0);
self.url_len = @intCast(kept);
self.event_key_len = @intCast(events.canonicalKey(url, &self.event_key).len);
}
fn fail(self: *SourceStatus, state: State, text: []const u8) void {
self.pass_failures +|= 1;
self.pass_outcome = true;
self.state = state;
const kept = @min(text.len, max_error_len);
@memcpy(self.last_error[0..kept], text[0..kept]);
@@ -207,6 +282,7 @@ pub const SourceStatus = struct {
}
fn succeed(self: *SourceStatus, at: i64, counts: compiler.Counts) void {
self.pass_outcome = true;
self.state = .ok;
self.counts = counts;
self.last_success = at;
@@ -294,6 +370,14 @@ pub const Manager = struct {
/// what every test and `nxdns check` want. Only the scheduler consults it —
/// see `refreshGated`.
monitor: ?*disk_monitor.Monitor = null,
/// The diagnostics store, wired the same way as `monitor` and null
/// everywhere else. Never touched while a manager lock is held: see
/// `flushDiagnostics`.
diagnostics: ?*events.Store = null,
/// What the locked bodies observed and could not report from where they
/// stood. Bounded by construction — one slot per storage operation, one
/// snapshot outcome — and drained by `flushDiagnostics`.
pending: Pending = .{},
/// Scheduled refresh passes skipped by the disk gate. The `/api/health`
/// rollup reads it through `refreshesGated`.
refreshes_gated: std.atomic.Value(u64) = .init(0),
@@ -388,6 +472,228 @@ pub const Manager = struct {
return kept;
}
// -----------------------------------------------------------------------
// diagnostics
// -----------------------------------------------------------------------
/// Records one storage operation's failure. Callable from anywhere,
/// including under both writer mutexes: it touches `pending` only.
fn noteStorageFailure(
self: *Manager,
io: std.Io,
op: StorageOp,
comptime fmt: []const u8,
args: anytype,
) void {
if (self.diagnostics == null) return;
self.pending.mutex.lockUncancelable(io);
defer self.pending.mutex.unlock(io);
const slot = self.pending.storage.getPtr(op);
slot.failures +|= 1;
var w: std.Io.Writer = .fixed(&slot.detail);
w.print(fmt, args) catch {};
slot.detail_len = @intCast(w.end);
}
fn noteStorageSuccess(self: *Manager, io: std.Io, op: StorageOp) void {
if (self.diagnostics == null) return;
self.pending.mutex.lockUncancelable(io);
defer self.pending.mutex.unlock(io);
self.pending.storage.getPtr(op).succeeded = true;
}
/// Records whether a snapshot was published. `reason` null is the post-swap
/// success; anything else is the pass that could not publish one.
fn noteSnapshot(self: *Manager, io: std.Io, reason: ?[]const u8) void {
if (self.diagnostics == null) return;
self.pending.mutex.lockUncancelable(io);
defer self.pending.mutex.unlock(io);
self.pending.snapshot_failed = reason != null;
const text = reason orelse "";
const kept = @min(text.len, self.pending.snapshot_detail.len);
@memcpy(self.pending.snapshot_detail[0..kept], text[0..kept]);
self.pending.snapshot_detail_len = @intCast(kept);
}
/// Drains `pending` and the status table into the store, holding no manager
/// lock across a store call.
///
/// Called by every pass that can fill either one, and *before that pass
/// releases the lock serializing it* — `refresh_lock` for a refresh pass,
/// `writer_lock` for a standalone `reload`. Draining after the release
/// would let the next pass record its own outcomes on the same entries
/// first, and two failing passes would reach the store as one occurrence.
/// The `defer` that calls this is registered after the unlock `defer` for
/// that reason; defers run last-registered-first.
///
/// It is idempotent: a drained collector reports nothing.
pub fn flushDiagnostics(self: *Manager, io: std.Io) void {
const store = self.diagnostics orelse return;
const now_s = std.Io.Clock.real.now(io).toSeconds();
var storage: std.EnumArray(StorageOp, Aggregate) = undefined;
var snapshot_failed: ?bool = null;
var snapshot_detail: [events.Store.max_detail_len]u8 = undefined;
var snapshot_detail_len: u16 = 0;
{
self.pending.mutex.lockUncancelable(io);
defer self.pending.mutex.unlock(io);
storage = self.pending.storage;
snapshot_failed = self.pending.snapshot_failed;
snapshot_detail = self.pending.snapshot_detail;
snapshot_detail_len = self.pending.snapshot_detail_len;
self.pending.storage = .initFill(.{});
self.pending.snapshot_failed = null;
self.pending.snapshot_detail_len = 0;
}
var it = storage.iterator();
while (it.next()) |kv| {
const op = @tagName(kv.key);
if (kv.value.failures != 0) {
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "{s} ({d} this pass)", .{
kv.value.detailText(),
kv.value.failures,
}) catch buf[0..];
store.report(io, now_s, .blocklist_storage, op, op, .warning, detail);
} else if (kv.value.succeeded) {
store.resolve(io, now_s, .blocklist_storage, op);
}
}
if (snapshot_failed) |failed| {
if (failed) {
store.report(
io,
now_s,
.blocklist_snapshot,
snapshot_key,
"blocklist snapshot",
.@"error",
snapshot_detail[0..snapshot_detail_len],
);
} else {
store.resolve(io, now_s, .blocklist_snapshot, snapshot_key);
}
}
self.flushSourceDiagnostics(io, store, now_s);
}
/// One `blocklist.refresh` episode per source, from the status table.
///
/// The table IS the per-source collection the collect-then-flush rule asks
/// for: `prepareRefresh`, `publishRefresh` and the reload's load outcomes
/// all write their result into the entry, under locks this cannot take. So
/// one entry is copied out at a time under the exclusive lock and the store
/// is called with nothing held.
///
/// The walk is a drain, not an index scan: a reload can replace the whole
/// table between two iterations, and an index into the table it replaced
/// would skip or repeat entries. Each round takes the lock, claims the
/// first entry that still carries pass accounting by copying it out and
/// zeroing the two fields, and reports it with nothing held. Claiming and
/// clearing are one locked step, so an outcome is reported once: a table
/// swapped in mid-drain carries the entries this flush has not claimed yet,
/// and `installStatuses` folded them in for exactly that reason. The drain
/// ends when a scan finds nothing left to claim.
///
/// The drain reaches only the sources the table still holds, so the sweep
/// below is what closes the episode of one that is gone.
fn flushSourceDiagnostics(self: *Manager, io: std.Io, store: *events.Store, now_s: i64) void {
drain: while (true) {
var status: SourceStatus = undefined;
{
self.lock.lockUncancelable(io);
defer self.lock.unlock(io);
const claimed = for (self.statuses) |*entry| {
if (!entry.pass_outcome) continue;
status = entry.*;
entry.pass_outcome = false;
entry.pass_failures = 0;
// A source with no diagnostics identity has nothing to
// report under, but its accounting is cleared all the same:
// left set, it would make every later scan claim it and the
// drain would never end.
if (entry.event_key_len == 0) continue;
break true;
} else false;
if (!claimed) break :drain;
}
if (!status.state.isRefreshFailure() and status.state != .load_failed) {
store.resolve(io, now_s, .blocklist_refresh, status.eventKey());
continue;
}
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "{t}: {s} ({d} this pass)", .{
status.state,
status.errorText(),
status.pass_failures,
}) catch buf[0..];
var label_buf: [events.Store.max_subject_label_len]u8 = undefined;
const label = std.fmt.bufPrint(&label_buf, "{f}", .{
safe_url.redact(status.urlText()),
}) catch &label_buf;
store.report(io, now_s, .blocklist_refresh, status.eventKey(), label, .warning, detail);
}
self.resolveDeletedSources(io, store, now_s);
}
/// Closes the `blocklist.refresh` episode of a source that no longer exists.
///
/// Nothing else can. An episode of this code is closed by its source
/// succeeding, and a source deleted through the API or dropped by a config
/// import never succeeds again: the drain above walks the status table, the
/// deleted source has no entry in it, and the resolved-row pruning never
/// touches an active row. Without this the operator keeps a warning about a
/// list they removed on purpose, and no restart clears it.
///
/// The status table holds every source at every flush site — `refreshAll`
/// syncs it before it refreshes anything and a reload rebuilds it from the
/// rows — so its keys are exactly the episodes that may stay open.
fn resolveDeletedSources(self: *Manager, io: std.Io, store: *events.Store, now_s: i64) void {
var storage: [events.Store.max_kept_keys][events.Store.max_subject_key_len]u8 = undefined;
var lens: [events.Store.max_kept_keys]u16 = undefined;
var len: usize = 0;
{
// Shared: this reads the table and changes nothing in it. The keys
// are copied out because the arena they live in is freed by the
// next `installStatuses`, and the store is called below with
// nothing held.
self.lock.lockSharedUncancelable(io);
defer self.lock.unlockShared(io);
// An empty table before the first published snapshot means "no
// source set has been read yet", not "every source was deleted".
// Sweeping on it would close every episode the last run left open,
// and the pass that follows would reopen each one as a new episode
// with its history reset.
if (self.generation == 0) return;
for (self.statuses) |*entry| {
if (entry.event_key_len == 0) continue;
// `resolveExcept` refuses a kept list longer than
// `max_kept_keys`, because it canonicalizes onto the stack.
// Over that many keyed sources the sweep is skipped whole: the
// alternative is a truncated kept list, which would close
// episodes that are still true. A source deleted while the
// household is over the cap keeps its episode until the count
// falls back under it.
if (len == storage.len) return;
const key = entry.eventKey();
@memcpy(storage[len][0..key.len], key);
lens[len] = entry.event_key_len;
len += 1;
}
}
var kept: [events.Store.max_kept_keys][]const u8 = undefined;
for (0..len) |i| kept[i] = storage[i][0..lens[i]];
store.resolveExcept(io, now_s, .blocklist_refresh, kept[0..len]);
}
// -----------------------------------------------------------------------
// reload
// -----------------------------------------------------------------------
@@ -407,12 +713,35 @@ pub const Manager = struct {
/// table keeps describing that snapshot too — the table is rebuilt off to
/// the side and the load findings are written into it there, so a reload
/// that never publishes changes neither.
///
/// A standalone reload is its own pass, and `writer_lock` is what serializes
/// it against every other writer of the status table. So it flushes inside
/// that lock: the load outcomes it wrote at the swap are drained before any
/// other pass can add its own to the same entries, which is what keeps two
/// failing passes two occurrences instead of one.
pub fn reload(self: *Manager, io: std.Io) Error!void {
// Must not be entered with `writer_lock` held.
self.writer_lock.lockUncancelable(io);
defer self.writer_lock.unlock(io);
// Registered after the unlock so it runs before it, and `defer` and not
// straight-line code after the call: a reload that fails has already
// collected the outcomes that explain why, and leaving them pending
// would hold them until some later pass flushed them under the wrong
// timestamp.
defer self.flushDiagnostics(io);
return self.reloadLocked(io);
}
/// `reload` without the flush, for a caller that is inside a pass with a
/// flush of its own. One pass flushes once: flushing here as well would
/// split the pass's outcomes across two reports.
fn reloadCollecting(self: *Manager, io: std.Io) Error!void {
// Must not be entered with `writer_lock` held.
self.writer_lock.lockUncancelable(io);
defer self.writer_lock.unlock(io);
try self.reloadLocked(io);
}
fn reloadLocked(self: *Manager, io: std.Io) Error!void {
var rows = try sources_repo.listSourceRows(self.database, self.gpa);
defer rows.deinit(self.gpa);
@@ -543,6 +872,7 @@ pub const Manager = struct {
rows.items.len,
memory_bytes,
});
self.noteSnapshot(io, null);
}
/// What one enabled source contributes to the snapshot being built. Nothing
@@ -642,6 +972,11 @@ pub const Manager = struct {
pub fn refreshSource(self: *Manager, io: std.Io, row: sources_repo.SourceRow) Error!bool {
self.refresh_lock.lockUncancelable(io);
defer self.refresh_lock.unlock(io);
// Registered after the unlock, so it runs before it: a pass drains its
// own outcomes while it still holds `refresh_lock`. `defer` at all, so
// a refresh that fails outright still reports what it collected instead
// of leaving it for an unrelated later flush.
defer self.flushDiagnostics(io);
return self.refreshSourceLocked(io, row);
}
@@ -706,6 +1041,9 @@ pub const Manager = struct {
pub fn refreshAll(self: *Manager, io: std.Io) Error!void {
self.refresh_lock.lockUncancelable(io);
defer self.refresh_lock.unlock(io);
// Inside `refresh_lock`, by being registered after the unlock: see
// `refreshSource`.
defer self.flushDiagnostics(io);
var rows = try sources_repo.listSourceRows(self.database, self.gpa);
defer rows.deinit(self.gpa);
@@ -718,8 +1056,9 @@ pub const Manager = struct {
_ = try self.refreshSourceLocked(io, row);
}
// `reload` takes `writer_lock`, which the pass has been careful not to
// hold: the order is `refresh_lock` first, always.
return self.reload(io);
// hold: the order is `refresh_lock` first, always. The collecting
// variant, because the `defer` above is this pass's one flush.
return self.reloadCollecting(io);
}
/// The three temporary files one refresh compiles into, before the header
@@ -1161,9 +1500,16 @@ pub const Manager = struct {
// ask the same filesystem for.
try self.sweepOrphans(io);
// `startupPass` flushes its own outcomes before it releases
// `refresh_lock`, so the only flush left here is the one the failure
// note below needs.
self.startupPass(io) catch |err| switch (err) {
error.Canceled => return error.Canceled,
else => log.warn("blocklist startup pass failed: {s}", .{@errorName(err)}),
else => {
log.warn("blocklist startup pass failed: {s}", .{@errorName(err)});
self.noteSnapshot(io, @errorName(err));
self.flushDiagnostics(io);
},
};
if (!self.update.enabled) return;
@@ -1175,19 +1521,33 @@ pub const Manager = struct {
};
while (true) {
try interval.sleep(io);
// Ahead of the gate as well as ahead of the pass: the sweep only
// unlinks, so it is the one thing here that can give a critically
// full disk room back, and gating it would keep the residue that
// helped fill the disk in the first place.
try self.sweepOrphans(io);
if (self.refreshGated()) continue;
self.refreshAll(io) catch |err| switch (err) {
error.Canceled => return error.Canceled,
else => log.warn("blocklist refresh pass failed: {s}", .{@errorName(err)}),
};
try self.scheduledPass(io);
}
}
/// What one elapsed interval does. Split from the loop above so a test can
/// run the pass without waiting the interval out; nothing in production
/// calls it but `runScheduler`.
pub fn scheduledPass(self: *Manager, io: std.Io) std.Io.Cancelable!void {
// Ahead of the gate as well as ahead of the refresh: the sweep only
// unlinks, so it is the one thing here that can give a critically full
// disk room back, and gating it would keep the residue that helped fill
// the disk in the first place.
try self.sweepOrphans(io);
if (self.refreshGated()) return;
// `refreshAll` flushes the pass itself, so the only flush left here is
// the one the failure note below needs: flushing unconditionally would
// report every outcome of the pass a second time.
self.refreshAll(io) catch |err| switch (err) {
error.Canceled => return error.Canceled,
else => {
log.warn("blocklist refresh pass failed: {s}", .{@errorName(err)});
self.noteSnapshot(io, @errorName(err));
self.flushDiagnostics(io);
},
};
}
/// `pruneOrphans` with its failure absorbed. Leftover bytes under
/// `<data_dir>/blocklists/` are not an outage, and a sweep that could not
/// read the directory must not cost the household the refresh pass behind
@@ -1197,10 +1557,16 @@ pub const Manager = struct {
/// Taken from outside every `*Locked` body: `pruneOrphans` takes both
/// writer mutexes itself and neither is reentrant.
fn sweepOrphans(self: *Manager, io: std.Io) std.Io.Cancelable!void {
self.pruneOrphans(io) catch |err| switch (err) {
if (self.pruneOrphans(io)) {
self.noteStorageSuccess(io, .sweep);
} else |err| switch (err) {
error.Canceled => return error.Canceled,
else => log.warn("pruning orphaned blocklist files failed: {s}", .{@errorName(err)}),
};
else => {
log.warn("pruning orphaned blocklist files failed: {s}", .{@errorName(err)});
self.noteStorageFailure(io, .sweep, "pruning orphaned blocklist files failed: {s}", .{@errorName(err)});
},
}
self.flushDiagnostics(io);
}
/// The §11.6 gate, consulted by scheduled passes only (ruling 17). A
@@ -1232,11 +1598,14 @@ pub const Manager = struct {
fn startupPass(self: *Manager, io: std.Io) Error!void {
self.refresh_lock.lockUncancelable(io);
defer self.refresh_lock.unlock(io);
// Inside `refresh_lock`, by being registered after the unlock: see
// `refreshSource`.
defer self.flushDiagnostics(io);
// Ahead of the gate on purpose: loading the compiled files that already
// exist is a read. A full disk must not cost the household its
// filtering as well as its downloads.
try self.reload(io);
try self.reloadCollecting(io);
if (self.refreshGated()) return;
@@ -1251,7 +1620,7 @@ pub const Manager = struct {
if (!self.needsRefresh(io, row, now)) continue;
if (try self.refreshSourceLocked(io, row)) refreshed = true;
}
if (refreshed) try self.reload(io);
if (refreshed) try self.reloadCollecting(io);
}
fn needsRefresh(self: *Manager, io: std.Io, row: sources_repo.SourceRow, now: i64) bool {
@@ -1297,6 +1666,13 @@ pub const Manager = struct {
/// `<data_dir>/blocklists/` if nothing has yet, and an empty directory
/// sweeps to nothing.
pub fn pruneOrphans(self: *Manager, io: std.Io) Error!void {
defer self.flushDiagnostics(io);
return self.pruneOrphansLocked(io);
}
/// Assumes nothing and takes both writer mutexes itself. Split from
/// `pruneOrphans` so the diagnostics flush above happens with neither held.
fn pruneOrphansLocked(self: *Manager, io: std.Io) Error!void {
// `refresh_lock` first, and for the reason it exists: the download and
// the compile are the only writers of `.raw.tmp`, `.list.tmp`,
// `.wild.tmp` and `.allow.tmp`, and they hold it for as long as they
@@ -1334,6 +1710,7 @@ pub const Manager = struct {
error.Canceled => return error.Canceled,
else => {
log.warn("pruning blocklists: reading the directory failed: {s}", .{@errorName(err)});
self.noteStorageFailure(io, .directory_read, "reading the blocklist directory failed: {s}", .{@errorName(err)});
return error.FileSystem;
},
} orelse break;
@@ -1343,6 +1720,8 @@ pub const Manager = struct {
try doomed.append(self.gpa, try self.gpa.dupe(u8, entry.name));
}
self.noteStorageSuccess(io, .directory_read);
for (doomed.items) |name| {
self.deleteQuietly(io, dir, name);
log.info("pruned orphaned blocklist file {s}", .{name});
@@ -1381,8 +1760,21 @@ pub const Manager = struct {
}
/// Publishes a built table and frees the one it replaces. The caller holds
/// the exclusive lock, so no reader is inside the old table.
/// the exclusive lock, so no reader is inside the old table and no flush is
/// half way through draining it.
///
/// The pass accounting the live table still holds is folded into the
/// incoming entry of the same id first. `mergeStatuses` left the candidate
/// carrying none, so an outcome recorded after the candidate was built —
/// and any a flush has not drained yet — survives the swap exactly once. An
/// outcome a flush already reported is zero in the live table, so nothing
/// here resurrects it.
fn installStatuses(self: *Manager, table: StatusTable) void {
for (table.items) |*incoming| {
const live = entryFor(self.statuses, incoming.id) orelse continue;
incoming.pass_failures +|= live.pass_failures;
incoming.pass_outcome = incoming.pass_outcome or live.pass_outcome;
}
self.status_arena.deinit();
self.status_arena = table.arena;
self.statuses = table.items;
@@ -1449,29 +1841,46 @@ pub const Manager = struct {
error.Canceled => return error.Canceled,
else => {
log.warn("creating {s} failed: {s}", .{ self.paths.subdir, @errorName(err) });
self.noteStorageFailure(io, .create_dir, "creating {s} failed: {s}", .{ self.paths.subdir, @errorName(err) });
return error.FileSystem;
},
};
return self.paths.dir.openDir(io, self.paths.subdir, options) catch |err| switch (err) {
self.noteStorageSuccess(io, .create_dir);
const dir = self.paths.dir.openDir(io, self.paths.subdir, options) catch |err| switch (err) {
error.Canceled => return error.Canceled,
else => {
log.warn("opening {s} failed: {s}", .{ self.paths.subdir, @errorName(err) });
self.noteStorageFailure(io, .open_dir, "opening {s} failed: {s}", .{ self.paths.subdir, @errorName(err) });
return error.FileSystem;
},
};
self.noteStorageSuccess(io, .open_dir);
return dir;
}
/// A temporary that cannot be removed is not a failure of the operation
/// that made it, but it is not nothing either: it is left visible.
fn deleteQuietly(self: *Manager, io: std.Io, dir: std.Io.Dir, name: []const u8) void {
_ = self;
dir.deleteFile(io, name) catch |err| switch (err) {
error.FileNotFound => {},
else => log.warn("deleting {s} failed: {s}", .{ name, @errorName(err) }),
};
if (dir.deleteFile(io, name)) {
self.noteStorageSuccess(io, .delete);
} else |err| switch (err) {
// A name that was never created is the ordinary case: the temporary
// deletes are installed before the files exist.
error.FileNotFound => self.noteStorageSuccess(io, .delete),
else => {
log.warn("deleting {s} failed: {s}", .{ name, @errorName(err) });
self.noteStorageFailure(io, .delete, "deleting {s} failed: {s}", .{ name, @errorName(err) });
},
}
}
};
/// The one subject `blocklist.snapshot` ever has: a box publishes exactly one
/// snapshot, and every source that failed to load is its own
/// `blocklist.refresh` episode.
const snapshot_key = "snapshot";
/// A status table and the arena holding it. Until `installStatuses` takes it,
/// it is a candidate nobody can see, and `deinit` frees it whole.
const StatusTable = struct {
@@ -1488,6 +1897,14 @@ const StatusTable = struct {
/// over from `previous`. A source deleted since `previous` was built is gone; a
/// source added since starts blank. `previous` is only read, so the caller's
/// published table is untouched by this.
///
/// The pass accounting is *not* carried: it lives in exactly one table copy at
/// a time. `previous` is a snapshot of the published table taken outside the
/// swap, so copying its counters here would leave the same outcomes in two
/// tables — the live one for a flush to drain, and this candidate for the
/// reload's own flush to report a second time. A candidate holds only what
/// `applyLoadOutcomes` writes into it; what the live table holds is folded in
/// by `installStatuses` under the exclusive lock.
fn mergeStatuses(
table: []SourceStatus,
rows: []const sources_repo.SourceRow,
@@ -1500,6 +1917,8 @@ fn mergeStatuses(
status.* = prior;
break;
}
status.pass_outcome = false;
status.pass_failures = 0;
// After the carry-over: a url edited on the row wins over the one the
// prior entry recorded.
status.setUrl(row.url);
@@ -1716,6 +2135,7 @@ fn containsId(rows: []const sources_repo.SourceRow, id: i64) bool {
// real swaps under load are the integration suite's (S9).
const testing = std.testing;
const events_fixture = @import("../storage/events_fixture.zig");
const migrations = @import("../storage/migrations.zig");
fn openMigrated() !db.Db {
@@ -2275,6 +2695,13 @@ test "a candidate table carries prior entries over and leaves the published one
try testing.expectEqual(State.never_fetched, candidate[1].state);
try testing.expect(!candidate[1].loaded);
// The one thing a candidate does not carry. `published[0]` is holding a
// failure no flush has drained yet; copying its accounting here would leave
// the same outcome in two tables, and the flush of each would report it.
try testing.expect(published[0].pass_outcome);
try testing.expect(!candidate[0].pass_outcome);
try testing.expectEqual(@as(u16, 0), candidate[0].pass_failures);
// The published table is untouched, so a reload that fails before the swap
// leaves it describing the snapshot that is still serving — including the
// entry of the deleted source, which that snapshot still enforces.
@@ -2284,6 +2711,181 @@ test "a candidate table carries prior entries over and leaves the published one
try testing.expectEqualStrings("https://lists.example/one.txt", published[0].urlText());
}
test "installing a table folds the live pass accounting in by id" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var f: fetcher.Fetcher = undefined;
var mgr = try testManager(&database, &f);
defer mgr.deinit(io);
// Source 1 recorded a failure the flush has not drained; source 2 was
// drained already; source 3 recorded one this reload knows nothing about.
var live: std.heap.ArenaAllocator = .init(testing.allocator);
const live_items = try live.allocator().alloc(SourceStatus, 3);
live_items[0] = .{ .id = 1, .pass_outcome = true, .pass_failures = 2 };
live_items[1] = .{ .id = 2 };
live_items[2] = .{ .id = 3, .pass_outcome = true, .pass_failures = 1 };
{
mgr.lock.lockUncancelable(io);
defer mgr.lock.unlock(io);
mgr.installStatuses(.{ .arena = live, .items = live_items });
}
// What a reload built beside it, carrying only its own load outcomes.
var incoming: std.heap.ArenaAllocator = .init(testing.allocator);
const incoming_items = try incoming.allocator().alloc(SourceStatus, 3);
incoming_items[0] = .{ .id = 1, .pass_outcome = true, .pass_failures = 1 };
incoming_items[1] = .{ .id = 2, .pass_outcome = true, .pass_failures = 4 };
incoming_items[2] = .{ .id = 3 };
{
mgr.lock.lockUncancelable(io);
defer mgr.lock.unlock(io);
mgr.installStatuses(.{ .arena = incoming, .items = incoming_items });
}
try testing.expectEqual(@as(u16, 3), mgr.statuses[0].pass_failures);
try testing.expect(mgr.statuses[0].pass_outcome);
// A drained entry adds nothing: what the swap publishes is the reload's own
// accounting and no resurrection of what was already reported.
try testing.expectEqual(@as(u16, 4), mgr.statuses[1].pass_failures);
try testing.expect(mgr.statuses[1].pass_outcome);
// The half the swap used to lose: an outcome the live table held and the
// candidate never saw.
try testing.expectEqual(@as(u16, 1), mgr.statuses[2].pass_failures);
try testing.expect(mgr.statuses[2].pass_outcome);
}
test "the flush claims every entry that carries pass accounting, once" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1_700_000_000);
defer fx.deinit();
var f: fetcher.Fetcher = undefined;
var mgr = try testManager(&database, &f);
defer mgr.deinit(io);
mgr.diagnostics = &fx.store;
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
const items = try arena.allocator().alloc(SourceStatus, 3);
items[0] = .{ .id = 1 };
items[0].setUrl("https://lists.example/one.txt");
items[0].fail(.fetch_failed, "HttpStatus");
// No url, so no episode to report under. The drain has to claim it anyway:
// an entry left with `pass_outcome` set is the one every later scan finds
// first, and the entry behind it would never be reached.
items[1] = .{ .id = 2, .pass_outcome = true, .pass_failures = 1 };
items[2] = .{ .id = 3 };
items[2].setUrl("https://lists.example/three.txt");
items[2].succeed(1_700_000_000, .{ .domains = 3 });
{
mgr.lock.lockUncancelable(io);
defer mgr.lock.unlock(io);
mgr.installStatuses(.{ .arena = arena, .items = items });
}
mgr.flushDiagnostics(io);
try testing.expectEqual(@as(i64, 1), try fx.count(
"SELECT count(*) FROM operational_events WHERE code = 'blocklist.refresh'",
));
try testing.expectEqual(@as(i64, 1), try fx.count(
"SELECT occurrences FROM operational_events WHERE code = 'blocklist.refresh'",
));
for (mgr.statuses) |entry| {
try testing.expect(!entry.pass_outcome);
try testing.expectEqual(@as(u16, 0), entry.pass_failures);
}
// Drained: flushing the same table again reports nothing a second time.
mgr.flushDiagnostics(io);
try testing.expectEqual(@as(i64, 1), try fx.count(
"SELECT occurrences FROM operational_events WHERE code = 'blocklist.refresh'",
));
}
test "the flush closes the episode of a source that is no longer in the table" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1_700_000_000);
defer fx.deinit();
var f: fetcher.Fetcher = undefined;
var mgr = try testManager(&database, &f);
defer mgr.deinit(io);
mgr.diagnostics = &fx.store;
// What the operator deleted while it was failing. Nothing will ever record
// a success for it, so nothing but the sweep can close this.
fx.store.report(
io,
1_700_000_000,
.blocklist_refresh,
"https://lists.example/deleted.txt",
"lists.example/deleted.txt",
.warning,
"fetch_failed: HttpStatus (1 this pass)",
);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
const items = try arena.allocator().alloc(SourceStatus, 1);
items[0] = .{ .id = 1 };
items[0].setUrl("https://lists.example/one.txt");
items[0].fail(.fetch_failed, "HttpStatus");
{
mgr.lock.lockUncancelable(io);
defer mgr.lock.unlock(io);
mgr.installStatuses(.{ .arena = arena, .items = items });
}
// No snapshot published yet, so the table is not known to describe the
// source set and the sweep must not run: the drain reports the failing
// source and the deleted one's episode is left alone.
mgr.flushDiagnostics(io);
try testing.expectEqual(@as(i64, 2), try fx.count(
"SELECT count(*) FROM operational_events WHERE code = 'blocklist.refresh' AND resolved_at IS NULL",
));
mgr.generation = 1;
mgr.flushDiagnostics(io);
// One left active, and it is the source that still exists.
try testing.expectEqual(@as(i64, 1), try fx.count(
"SELECT count(*) FROM operational_events WHERE code = 'blocklist.refresh' AND resolved_at IS NULL",
));
try testing.expectEqualStrings(
"https://lists.example/one.txt",
try fx.text(
"SELECT subject_key FROM operational_events WHERE code = 'blocklist.refresh' AND resolved_at IS NULL",
),
);
try testing.expectEqualStrings(
"https://lists.example/deleted.txt",
try fx.text(
"SELECT subject_key FROM operational_events WHERE code = 'blocklist.refresh' AND resolved_at IS NOT NULL",
),
);
}
test "a disabled source stops being loaded" {
var statuses = [_]SourceStatus{.{ .id = 1 }};
statuses[0].succeed(1_700_000_000, .{ .domains = 9 });
+130 -10
View File
@@ -13,6 +13,8 @@
//! through the size.
const std = @import("std");
const events = @import("../storage/events.zig");
const tls_server = @import("../platform/tls_server.zig");
const log = std.log.scoped(.cert_store);
@@ -101,6 +103,11 @@ pub const ReloadHook = struct {
};
pub const CertStore = struct {
/// Which endpoint's certificate this store holds. It is the subject of
/// every `certificate.reload` event, and a box serving both DoH and DoT
/// runs two stores over two file pairs.
pub const Kind = enum { doh, dot };
gpa: std.mem.Allocator,
/// Borrowed from the config; must outlive the store.
cert_path: []const u8,
@@ -132,6 +139,12 @@ pub const CertStore = struct {
/// `reload` synchronously (that would self-deadlock on `reload_mutex`).
after_load_hook: ?ReloadHook,
/// Set by the composition root right after `init`, with `diagnostics`.
kind: Kind = .doh,
/// Wired the same way and for the same reason as every other subsystem's:
/// the store is fully usable without it, and `nxdns check` has none.
diagnostics: ?*events.Store = null,
reloads: std.atomic.Value(u64),
reload_failures: std.atomic.Value(u64),
/// Wall-clock second of the last successful load, including the one in
@@ -243,7 +256,7 @@ pub const CertStore = struct {
};
while (true) {
try interval.sleep(io);
self.pollOnce(io);
self.pollOnce(io, std.Io.Clock.real.now(io).toSeconds());
}
}
@@ -252,13 +265,15 @@ pub const CertStore = struct {
/// changed, and the old one keeps serving either way. A failed reload
/// warns and counts (`reload_failures`); the signature stays at the loaded
/// pair, so every subsequent poll retries until the files parse.
pub fn pollOnce(self: *CertStore, io: std.Io) void {
const cert_sig = statSig(io, self.cert_path) catch {
pub fn pollOnce(self: *CertStore, io: std.Io, now_s: i64) void {
const cert_sig = statSig(io, self.cert_path) catch |err| {
log.warn("stat {s} failed; keeping the loaded certificate", .{self.cert_path});
self.reportReload(io, now_s, "stat of the certificate failed", @errorName(err));
return;
};
const key_sig = statSig(io, self.key_path) catch {
const key_sig = statSig(io, self.key_path) catch |err| {
log.warn("stat {s} failed; keeping the loaded certificate", .{self.key_path});
self.reportReload(io, now_s, "stat of the private key failed", @errorName(err));
return;
};
const observed: Signature = .{ .cert = cert_sig, .key = key_sig };
@@ -266,18 +281,36 @@ pub const CertStore = struct {
self.mutex.lockUncancelable(io);
const loaded = self.loaded;
self.mutex.unlock(io);
if (!changed(loaded, observed)) return;
if (!changed(loaded, observed)) {
// A poll that stat'ed both files and found nothing to do is a
// fully healthy pass, so it closes any episode a transient stat
// failure opened. Without this, a file that never changes again
// would leave that episode open forever.
if (self.diagnostics) |store| store.resolve(io, now_s, .certificate_reload, @tagName(self.kind));
return;
}
if (self.reload(io)) {
log.info("certificate reloaded from {s}", .{self.cert_path});
if (self.diagnostics) |store| store.resolve(io, now_s, .certificate_reload, @tagName(self.kind));
} else |err| {
log.warn("certificate reload from {s} failed ({s}); the old certificate keeps serving", .{
self.cert_path,
humanMessage(err),
});
self.reportReload(io, now_s, "certificate reload failed", humanMessage(err));
}
}
/// A warning, never an error: a stat failure can be a rename window, and a
/// failed reload leaves the loaded certificate serving. Nothing is down.
fn reportReload(self: *CertStore, io: std.Io, now_s: i64, message: []const u8, reason: []const u8) void {
const store = self.diagnostics orelse return;
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "{s}: {s}", .{ message, reason }) catch buf[0..];
store.report(io, now_s, .certificate_reload, @tagName(self.kind), @tagName(self.kind), .warning, detail);
}
pub fn snapshotStats(self: *const CertStore) Stats {
return .{
.reloads = self.reloads.load(.monotonic),
@@ -406,6 +439,7 @@ fn statSig(io: std.Io, path: []const u8) !FileSig {
// tests
// ---------------------------------------------------------------------------
const events_fixture = @import("../storage/events_fixture.zig");
const fixtures = @import("test_fixtures");
const testing = std.testing;
@@ -699,7 +733,7 @@ test "pollOnce reloads on a changed stat pair and stays put on an unchanged one"
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
store.pollOnce(io);
store.pollOnce(io, 1000);
try testing.expectEqual(@as(u64, 0), store.snapshotStats().reloads);
// Same certificate plus a trailing newline: the PEM still parses and the
@@ -710,14 +744,14 @@ test "pollOnce reloads on a changed stat pair and stays put on an unchanged one"
const old = store.acquire(io);
store.release(io, old);
store.pollOnce(io);
store.pollOnce(io, 1000);
try testing.expectEqual(@as(u64, 1), store.snapshotStats().reloads);
const fresh = store.acquire(io);
try testing.expect(fresh != old);
store.release(io, fresh);
store.pollOnce(io);
store.pollOnce(io, 1000);
try testing.expectEqual(@as(u64, 1), store.snapshotStats().reloads);
}
@@ -734,7 +768,7 @@ test "pollOnce warns and keeps serving when a reload fails" {
store.release(io, before);
try env.tmp.dir.writeFile(io, .{ .sub_path = "cert.pem", .data = "still not a certificate" });
store.pollOnce(io);
store.pollOnce(io, 1000);
const stats = store.snapshotStats();
try testing.expectEqual(@as(u64, 0), stats.reloads);
@@ -755,7 +789,7 @@ test "pollOnce does nothing when a file cannot be stat'ed" {
defer store.deinit(io);
try env.tmp.dir.deleteFile(io, "cert.pem");
store.pollOnce(io);
store.pollOnce(io, 1000);
const stats = store.snapshotStats();
try testing.expectEqual(@as(u64, 0), stats.reloads);
@@ -873,3 +907,89 @@ test "a reload overlapping another reload's window publishes last" {
store.mutex.unlock(io);
try testing.expectEqual(@as(u64, grown.len), final.cert.size);
}
test "a failed reload opens an episode keyed by endpoint kind and a good one closes it" {
var env: TestEnv = undefined;
try env.init();
defer env.deinit();
const io = env.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
store.kind = .dot;
store.diagnostics = &fx.store;
try env.tmp.dir.writeFile(io, .{ .sub_path = "cert.pem", .data = "still not a certificate" });
store.pollOnce(io, 1000);
try testing.expectEqualStrings("certificate.reload", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("dot", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("warning", try fx.text("SELECT severity FROM operational_events"));
// The same certificate plus a newline: it parses, and the size differs even
// within one timestamp granule.
const grown = try std.mem.concat(testing.allocator, u8, &.{ fixtures.cert_pem, "\n" });
defer testing.allocator.free(grown);
try env.tmp.dir.writeFile(io, .{ .sub_path = "cert.pem", .data = grown });
store.pollOnce(io, 1100);
try testing.expectEqual(@as(u64, 1), store.snapshotStats().reloads);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a failed stat opens the same episode a failed reload would" {
var env: TestEnv = undefined;
try env.init();
defer env.deinit();
const io = env.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
store.diagnostics = &fx.store;
try env.tmp.dir.deleteFile(io, "cert.pem");
store.pollOnce(io, 1000);
try testing.expectEqualStrings("doh", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqual(@as(u64, 0), store.snapshotStats().reload_failures);
}
test "an unchanged poll closes the episode a transient stat failure opened" {
var env: TestEnv = undefined;
try env.init();
defer env.deinit();
const io = env.io();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var store = try CertStore.init(testing.allocator, io, env.cert_path, env.key_path, null);
defer store.deinit(io);
store.diagnostics = &fx.store;
// What a stat failure in a rename window left open. The file it named is
// back and unchanged, so no reload will ever close this episode.
fx.store.report(io, 1000, .certificate_reload, "doh", "doh", .warning, "stat of the certificate failed: FileNotFound");
store.pollOnce(io, 1100);
try testing.expectEqual(@as(u64, 0), store.snapshotStats().reloads);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
+105 -1
View File
@@ -30,6 +30,7 @@ const clients_repo = @import("../storage/repositories/clients_repo.zig");
const db = @import("../storage/db.zig");
const dns_header = @import("../dns/header.zig");
const edns = @import("../dns/edns.zig");
const events = @import("../storage/events.zig");
const forward_client = @import("../local/forward_client.zig");
const local_tables = @import("local_tables.zig");
const name_mod = @import("../dns/name.zig");
@@ -103,6 +104,9 @@ pub const Resolver = struct {
tables: *local_tables.LocalTables,
stats: Stats = .{},
exchange_fn: ExchangeFn = defaultExchange,
/// Wired by the composition root after `init`, following the
/// `gate: ?*disk_monitor.Monitor` idiom. Null in every unit test here.
diagnostics: ?*events.Store = null,
pub fn init(tables: *local_tables.LocalTables) Resolver {
return .{ .tables = tables };
@@ -127,10 +131,13 @@ pub const Resolver = struct {
self.mutex.lockUncancelable(io);
self.stats.read_failures += 1;
self.mutex.unlock(io);
self.reportStorage(io, now_s, "read", "selecting clients to name failed", @errorName(err), 1);
return;
};
if (self.diagnostics) |store| store.resolve(io, now_s, .client_names_storage, "read");
var write_failures: u64 = 0;
var last_write_error: []const u8 = "";
for (candidates[0..count]) |*candidate| {
var learned_buf: [types.max_name_len]u8 = undefined;
const result = self.attempt(io, candidate.ip(), &learned_buf);
@@ -144,6 +151,7 @@ pub const Resolver = struct {
log.warn("recording the name of {s} failed: {s}", .{ candidate.ip(), @errorName(err) });
}
write_failures += 1;
last_write_error = @errorName(err);
}
self.mutex.lockUncancelable(io);
@@ -158,10 +166,38 @@ pub const Resolver = struct {
self.mutex.unlock(io);
}
if (write_failures == 0) return;
// The clean-pass determination is what closes a `write` episode: one
// aggregated outcome per pass, so a row's `occurrences` counts failing
// passes rather than failing rows.
if (write_failures == 0) {
if (self.diagnostics) |store| store.resolve(io, now_s, .client_names_storage, "write");
return;
}
self.mutex.lockUncancelable(io);
self.stats.write_failures += write_failures;
self.mutex.unlock(io);
self.reportStorage(io, now_s, "write", "recording a client name failed", last_write_error, write_failures);
}
/// One aggregated report per pass. `count` is how many failures that pass
/// held; the detail carries it because the row itself counts passes.
fn reportStorage(
self: *Resolver,
io: std.Io,
now_s: i64,
operation: []const u8,
message: []const u8,
error_name: []const u8,
count: u64,
) void {
const store = self.diagnostics orelse return;
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "{s}: {s} ({d} this pass)", .{
message,
error_name,
count,
}) catch buf[0..];
store.report(io, now_s, .client_names_storage, operation, operation, .warning, detail);
}
const Attempt = struct {
@@ -355,6 +391,7 @@ fn exchangeOnce(
const forward_zones = @import("../local/forward_zones.zig");
const migrations = @import("../storage/migrations.zig");
const events_fixture = @import("../storage/events_fixture.zig");
const testing = std.testing;
/// A stub exchange whose reply and error are set by the test. `calls` is the
@@ -1157,3 +1194,70 @@ test "an extended rcode of zero still reads as the header's rcode" {
try testing.expectEqualStrings("nas.lan", (try f.learned("192.168.1.10", &buf)).?);
try testing.expectEqual(@as(u64, 1), f.resolver.snapshotStats(f.io()).answered);
}
test "a failing write opens one episode per pass and a clean pass closes it" {
var f: Fixture = undefined;
try fixture(&f, "168.192.in-addr.arpa");
defer f.deinit();
var fx: events_fixture.Fixture = .{};
try fx.init(f.io(), 1000);
defer fx.deinit();
f.resolver.diagnostics = &fx.store;
// Two addresses outside the declared zone: both reach the write step
// without any exchange, which is not what this test is about.
try clients_repo.upsertSeen(&f.database, "10.0.0.1", 1700000000);
try clients_repo.upsertSeen(&f.database, "10.0.0.2", 1700000000);
try f.database.exec(
\\CREATE TRIGGER refuse_update BEFORE UPDATE ON clients
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
f.resolver.runPass(f.io(), &f.database, 1700000000);
// Two failing rows, one aggregated report: the row counts failing passes.
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqualStrings("client_names.storage", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("write", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("warning", try fx.text("SELECT severity FROM operational_events"));
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT occurrences FROM operational_events"));
f.resolver.runPass(f.io(), &f.database, 1700000060);
try testing.expectEqual(@as(i64, 2), try fx.count("SELECT occurrences FROM operational_events"));
try f.database.exec("DROP TRIGGER refuse_update;");
f.resolver.runPass(f.io(), &f.database, 1700000120);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a failing candidate select opens a read episode the next clean pass closes" {
var f: Fixture = undefined;
try fixture(&f, "168.192.in-addr.arpa");
defer f.deinit();
var fx: events_fixture.Fixture = .{};
try fx.init(f.io(), 1000);
defer fx.deinit();
f.resolver.diagnostics = &fx.store;
try clients_repo.upsertSeen(&f.database, "10.0.0.1", 1700000000);
try f.database.exec("ALTER TABLE clients RENAME TO clients_aside;");
f.resolver.runPass(f.io(), &f.database, 1700000000);
try testing.expectEqualStrings("read", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
try f.database.exec("ALTER TABLE clients_aside RENAME TO clients;");
f.resolver.runPass(f.io(), &f.database, 1700000060);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
+118
View File
@@ -25,6 +25,7 @@ const client_names = @import("client_names.zig");
const clients_repo = @import("../storage/repositories/clients_repo.zig");
const db = @import("../storage/db.zig");
const disk_monitor = @import("../storage/disk_monitor.zig");
const events = @import("../storage/events.zig");
const logger = @import("../storage/logger.zig");
const log = std.log.scoped(.clients);
@@ -63,6 +64,9 @@ pub const Tracker = struct {
count: u32,
passes: u64,
stats: Stats,
/// Wired by the composition root after `init`, following the
/// `gate: ?*disk_monitor.Monitor` idiom. Null in every unit test here.
diagnostics: ?*events.Store = null,
/// `retention_days` is `logging.retention_days`, the same knob the query log
/// prunes by (milestone-7 ruling 16). A client silent for that long is as
@@ -185,6 +189,7 @@ pub const Tracker = struct {
var flushed: u64 = 0;
var failures: u64 = 0;
var last_failure: []const u8 = "";
for (batch) |entry| {
// `logger.max_client_len` is the RFC 5952 bound every address text
// in this program is sized by, so `format` cannot fail here.
@@ -199,9 +204,19 @@ pub const Tracker = struct {
log.warn("materialising client {s} failed: {s}", .{ w.buffered(), @errorName(err) });
}
failures += 1;
last_failure = @errorName(err);
}
}
// One aggregated outcome per pass, so the row's `occurrences` counts
// failing passes rather than failing addresses. A pass that wrote
// nothing resolves nothing: an empty batch is not evidence of success.
if (failures != 0) {
self.reportStorage(io, now_s, "materialise", "materialising a client failed", last_failure, failures);
} else if (flushed != 0) {
if (self.diagnostics) |store| store.resolve(io, now_s, .clients_storage, "materialise");
}
self.mutex.lockUncancelable(io);
self.passes += 1;
self.stats.flushed += flushed;
@@ -215,17 +230,39 @@ pub const Tracker = struct {
self.mutex.lockUncancelable(io);
self.stats.pruned += deleted;
self.mutex.unlock(io);
if (self.diagnostics) |store| store.resolve(io, now_s, .clients_storage, "prune");
} else |err| {
log.warn("pruning clients before {d} failed: {s}", .{ cutoff, @errorName(err) });
self.mutex.lockUncancelable(io);
self.stats.flush_failures += 1;
self.mutex.unlock(io);
self.reportStorage(io, now_s, "prune", "pruning stale clients failed", @errorName(err), 1);
}
}
if (names) |resolver| resolver.runPass(io, database, now_s);
}
/// One aggregated report per pass; `count` is how many failures it held.
fn reportStorage(
self: *Tracker,
io: std.Io,
now_s: i64,
operation: []const u8,
message: []const u8,
error_name: []const u8,
count: u64,
) void {
const store = self.diagnostics orelse return;
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "{s}: {s} ({d} this pass)", .{
message,
error_name,
count,
}) catch buf[0..];
store.report(io, now_s, .clients_storage, operation, operation, .warning, detail);
}
pub fn snapshotStats(self: *Tracker, io: std.Io) Stats {
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
@@ -258,6 +295,7 @@ pub const Tracker = struct {
// tests
// ---------------------------------------------------------------------------
const events_fixture = @import("../storage/events_fixture.zig");
const migrations = @import("../storage/migrations.zig");
const testing = std.testing;
@@ -649,3 +687,83 @@ test "a gated pass attempts no naming either" {
try testing.expectEqual(@as(usize, 0), CountingExchange.calls);
try testing.expectEqual(@as(u64, 0), names.snapshotStats(io).attempted);
}
test "a failing materialise opens one episode per pass and a clean pass closes it" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var tracker: Tracker = .init(30);
tracker.diagnostics = &fx.store;
try database.exec(
\\CREATE TRIGGER refuse_insert BEFORE INSERT ON clients
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000000);
_ = tracker.trackAt(io, parsed("192.168.1.11"), 1700000001);
tracker.flushOnce(io, &database, true, null);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqualStrings("clients.storage", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("materialise", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT occurrences FROM operational_events"));
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000060);
tracker.flushOnce(io, &database, true, null);
try testing.expectEqual(@as(i64, 2), try fx.count("SELECT occurrences FROM operational_events"));
try database.exec("DROP TRIGGER refuse_insert;");
_ = tracker.trackAt(io, parsed("192.168.1.10"), 1700000120);
tracker.flushOnce(io, &database, true, null);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a failing prune opens its own episode the next due pass closes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openMigrated();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var tracker: Tracker = .init(30);
tracker.diagnostics = &fx.store;
// One pass short of due, so the pass below is the pruning one.
tracker.passes = Tracker.prune_every_passes - 1;
try database.exec(
\\CREATE TRIGGER refuse_delete BEFORE DELETE ON clients
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
try clients_repo.upsertSeen(&database, "192.168.1.10", 1);
tracker.flushOnce(io, &database, true, null);
try testing.expectEqualStrings("prune", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
try database.exec("DROP TRIGGER refuse_delete;");
tracker.passes = Tracker.prune_every_passes - 1;
tracker.flushOnce(io, &database, true, null);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
+101 -1
View File
@@ -98,6 +98,57 @@ pub const ddl_v1: [:0]const u8 =
\\);
\\
\\CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT NOT NULL);
\\
\\CREATE TABLE operational_events (
\\ id INTEGER PRIMARY KEY,
\\ code TEXT NOT NULL,
\\ subject_key TEXT NOT NULL,
\\ subject_label TEXT NOT NULL,
\\ severity TEXT NOT NULL CHECK (severity IN ('warning', 'error')),
\\ first_seen INTEGER NOT NULL,
\\ last_seen INTEGER NOT NULL,
\\ occurrences INTEGER NOT NULL CHECK (occurrences > 0),
\\ resolved_at INTEGER,
\\ detail TEXT NOT NULL DEFAULT '',
\\ CHECK (resolved_at IS NULL OR resolved_at >= first_seen)
\\);
\\CREATE UNIQUE INDEX idx_operational_events_active
\\ ON operational_events(code, subject_key) WHERE resolved_at IS NULL;
\\CREATE INDEX idx_operational_events_last_seen
\\ ON operational_events(last_seen DESC);
;
/// The same three statements, each made conditional, for the bridge at the end
/// of `migrations.migrate`.
///
/// A database stamped version 1 before `operational_events` joined `ddl_v1`
/// never runs step 1 again, so nothing would ever create the table there. The
/// bridge closes that divergence for the pre-0.1 installs that exist; it is
/// **removable the moment the v0.1 adoption gate lands**, because from then on
/// a schema change is an append-only migration step and this hazard cannot
/// recur.
///
/// It must not be folded into `ddl_v1`: a fresh database would then create the
/// table twice, and the unconditional `CREATE TABLE` above is what proves the
/// baseline and this text stay in step.
pub const operational_events_bridge: [:0]const u8 =
\\CREATE TABLE IF NOT EXISTS operational_events (
\\ id INTEGER PRIMARY KEY,
\\ code TEXT NOT NULL,
\\ subject_key TEXT NOT NULL,
\\ subject_label TEXT NOT NULL,
\\ severity TEXT NOT NULL CHECK (severity IN ('warning', 'error')),
\\ first_seen INTEGER NOT NULL,
\\ last_seen INTEGER NOT NULL,
\\ occurrences INTEGER NOT NULL CHECK (occurrences > 0),
\\ resolved_at INTEGER,
\\ detail TEXT NOT NULL DEFAULT '',
\\ CHECK (resolved_at IS NULL OR resolved_at >= first_seen)
\\);
\\CREATE UNIQUE INDEX IF NOT EXISTS idx_operational_events_active
\\ ON operational_events(code, subject_key) WHERE resolved_at IS NULL;
\\CREATE INDEX IF NOT EXISTS idx_operational_events_last_seen
\\ ON operational_events(last_seen DESC);
;
/// Child-before-parent, and correct under `foreign_keys = ON`. The reconcile
@@ -108,7 +159,10 @@ pub const ddl_v1: [:0]const u8 =
/// `upstreams`, `local_records`, `forward_zones` and `settings` have no foreign
/// keys, so their position is free; `groups` and `blocklist_sources` must come
/// last, after every referrer. `schema_version` is deliberately absent — an
/// import must never erase the stamped migration version.
/// import must never erase the stamped migration version — and so is
/// `operational_events`, which is runtime state rather than configuration: an
/// import that wiped the diagnostics log would destroy the record of what the
/// box has been doing.
pub const delete_order = [_][]const u8{
"group_sources", "rules", "client_prefixes", "clients",
"upstreams", "local_records", "forward_zones", "settings",
@@ -120,12 +174,16 @@ pub const delete_order = [_][]const u8{
/// that renumbered a group.
///
/// `schema_version` is absent: it is the migration's, not the operator's.
/// `operational_events` is absent for the same class of reason: it is the
/// program's own record of its failures, and an export of it would be a log
/// dump, not a configuration.
pub const table_names = [_][]const u8{
"groups", "clients", "client_prefixes", "upstreams",
"blocklist_sources", "group_sources", "rules", "local_records",
"forward_zones", "settings",
};
const db = @import("db.zig");
const testing = std.testing;
test "table_names names exactly the tables delete_order does" {
@@ -135,6 +193,48 @@ test "table_names names exactly the tables delete_order does" {
}
try testing.expect(indexOf(&table_names, "groups") != null);
try testing.expect(indexOf(&table_names, "schema_version") == null);
// Runtime state, not configuration: neither list may reach it, or an
// import would wipe the diagnostics log and an export would emit it.
try testing.expect(indexOf(&table_names, "operational_events") == null);
try testing.expect(indexOf(&delete_order, "operational_events") == null);
}
test "the bridge creates exactly what the baseline does" {
// The two texts are separate on purpose (a fresh database must not create
// the table twice), which is exactly how they could drift. Applying each to
// its own database and comparing `sqlite_schema` is what keeps them equal.
const baseline = try schemaOf(ddl_v1);
defer testing.allocator.free(baseline);
const bridged = try schemaOf(operational_events_bridge);
defer testing.allocator.free(bridged);
// SQLite stores the `CREATE` text verbatim, so the conditional is the one
// difference the two are allowed to have.
const size = std.mem.replacementSize(u8, bridged, " IF NOT EXISTS", "");
const plain = try testing.allocator.alloc(u8, size);
defer testing.allocator.free(plain);
_ = std.mem.replace(u8, bridged, " IF NOT EXISTS", "", plain);
try testing.expectEqualStrings(baseline, plain);
}
/// Every `sqlite_schema` row of `operational_events`, after applying `sql`.
fn schemaOf(sql: [:0]const u8) ![]u8 {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try database.exec(sql);
var out: std.Io.Writer.Allocating = .init(testing.allocator);
errdefer out.deinit();
var stmt = try database.prepare(
\\SELECT type, name, sql FROM sqlite_schema
\\ WHERE tbl_name = 'operational_events' ORDER BY name
);
defer stmt.deinit();
while (try stmt.step()) {
try out.writer.print("{s} {s}\n{s}\n", .{ stmt.columnText(0), stmt.columnText(1), stmt.columnText(2) });
}
return out.toOwnedSlice();
}
test "delete_order lists every referrer before the table it references" {
+168 -13
View File
@@ -9,6 +9,7 @@
//! milestone-5 file; the gate is pulled, not pushed.
const std = @import("std");
const events = @import("events.zig");
const model = @import("../config/model.zig");
const statfs = @import("../platform/statfs.zig");
@@ -89,43 +90,49 @@ pub const Monitor = struct {
/// evidence that the disk filled — and a failed size scan leaves that one
/// gauge at its previous reading. Every failure increments
/// `sample_failures` and logs one line at `warn`.
pub fn sample(self: *Monitor, io: std.Io) void {
const free = statfs.freeBytes(self.data_path) catch {
pub fn sample(self: *Monitor, io: std.Io, store: ?*events.Store, now_s: i64) void {
const free = statfs.freeBytes(self.data_path) catch |err| {
self.countFailure();
log.warn("statvfs on {s} failed", .{self.data_path});
probeFailed(store, io, now_s, "statvfs", "statvfs on the data path failed", err);
return;
};
self.free_bytes.store(free, .monotonic);
if (store) |s| s.resolve(io, now_s, .disk_probe, "statvfs");
if (sumDir(io, self.data_dir, isDatabaseFile)) |bytes| {
self.db_bytes.store(bytes, .monotonic);
if (store) |s| s.resolve(io, now_s, .disk_probe, "data_dir");
} else |err| {
self.countFailure();
log.warn("sizing the data directory failed: {s}", .{@errorName(err)});
probeFailed(store, io, now_s, "data_dir", "sizing the data directory failed", err);
}
if (self.log_dir_path) |path| {
if (self.sumLogDir(io, path)) |bytes| {
self.log_bytes.store(bytes, .monotonic);
if (store) |s| s.resolve(io, now_s, .disk_probe, "log_dir");
} else |err| {
self.countFailure();
log.warn("sizing {s} failed: {s}", .{ path, @errorName(err) });
probeFailed(store, io, now_s, "log_dir", "sizing the log directory failed", err);
}
}
self.publish(classify(free, self.cfg), free);
self.publish(io, store, now_s, classify(free, self.cfg), free);
}
/// Sample first, then sleep: a process that starts on a full disk must not
/// serve a whole interval believing the state is `.ok`. `.boot` so a
/// suspended box still sees the interval elapse.
pub fn run(self: *Monitor, io: std.Io) std.Io.Cancelable!void {
pub fn run(self: *Monitor, io: std.Io, store: ?*events.Store) std.Io.Cancelable!void {
const interval: std.Io.Clock.Duration = .{
.raw = .fromSeconds(sample_interval_s),
.clock = .boot,
};
while (true) {
self.sample(io);
self.sample(io, store, std.Io.Clock.real.now(io).toSeconds());
try interval.sleep(io);
}
}
@@ -136,7 +143,14 @@ pub const Monitor = struct {
/// Logs on transitions only. A disk that sits at `.warn` for a week
/// produces one line, not ten thousand.
fn publish(self: *Monitor, next: State, free: u64) void {
fn publish(
self: *Monitor,
io: std.Io,
store: ?*events.Store,
now_s: i64,
next: State,
free: u64,
) void {
const previous: State = @enumFromInt(self.state_raw.swap(@intFromEnum(next), .monotonic));
if (previous == next) return;
log.warn("disk state {t} -> {t}: {d} bytes free on {s}", .{
@@ -145,6 +159,24 @@ pub const Monitor = struct {
free,
self.data_path,
});
const s = store orelse return;
if (next == .ok) {
s.resolve(io, now_s, .disk_space, disk_space_key);
return;
}
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "disk state {t} -> {t}: {d} bytes free on {s}", .{
previous,
next,
free,
self.data_path,
}) catch buf[0..];
s.report(io, now_s, .disk_space, disk_space_key, "data directory", switch (next) {
.warn => .warning,
.critical => .@"error",
.ok => unreachable,
}, detail);
}
fn sumLogDir(self: *Monitor, io: std.Io, path: [:0]const u8) !u64 {
@@ -155,6 +187,26 @@ pub const Monitor = struct {
}
};
/// The one subject `disk.space` ever has: this box has exactly one data
/// directory, and its filesystem is what the thresholds classify.
const disk_space_key = "data";
/// Every probe failure is a warning, not an error: an unreadable filesystem is
/// a gap in what the monitor can see, and the state it published last stands.
fn probeFailed(
store: ?*events.Store,
io: std.Io,
now_s: i64,
operation: []const u8,
message: []const u8,
err: anyerror,
) void {
const s = store orelse return;
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "{s}: {s}", .{ message, @errorName(err) }) catch buf[0..];
s.report(io, now_s, .disk_probe, operation, operation, .warning, detail);
}
fn everyFile(_: []const u8) bool {
return true;
}
@@ -190,6 +242,7 @@ fn sumDir(io: std.Io, dir: std.Io.Dir, accept: *const fn ([]const u8) bool) !u64
// tests
// ---------------------------------------------------------------------------
const events_fixture = @import("events_fixture.zig");
const testing = std.testing;
const mb = 1024 * 1024;
@@ -280,7 +333,7 @@ test "a sample sizes the databases and ignores every other file" {
try tmp.dir.writeFile(io, .{ .sub_path = "notes.txt", .data = &[_]u8{'d'} ** 4096 });
var monitor: Monitor = .init(.{ .min_free_mb = 0, .warn_free_mb = 0 }, tmp.dir, ".", null);
monitor.sample(io);
monitor.sample(io, null, 0);
const g = monitor.gauges();
try testing.expectEqual(@as(u64, 160), g.db_bytes);
@@ -308,7 +361,7 @@ test "a sample sizes every file in the log directory" {
const log_path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/logs", .{tmp.sub_path});
var monitor: Monitor = .init(.{ .min_free_mb = 0, .warn_free_mb = 0 }, tmp.dir, ".", log_path);
monitor.sample(io);
monitor.sample(io, null, 0);
try testing.expectEqual(@as(u64, 1000), monitor.gauges().log_bytes);
try testing.expectEqual(@as(u64, 0), monitor.sample_failures.load(.monotonic));
@@ -321,7 +374,7 @@ test "a failed statvfs counts and keeps the previous state" {
var monitor: Monitor = .init(.{}, std.Io.Dir.cwd(), "./nxdns-no-such-path-7c21", null);
monitor.state_raw.store(@intFromEnum(State.warn), .monotonic);
monitor.sample(io);
monitor.sample(io, null, 0);
try testing.expectEqual(State.warn, monitor.state());
try testing.expectEqual(@as(u64, 1), monitor.sample_failures.load(.monotonic));
@@ -342,7 +395,7 @@ test "an unreadable log directory counts a failure but still publishes a state"
".",
"./nxdns-no-such-dir-4f8a",
);
monitor.sample(io);
monitor.sample(io, null, 0);
try testing.expectEqual(@as(u64, 1), monitor.sample_failures.load(.monotonic));
try testing.expectEqual(State.ok, monitor.state());
@@ -386,7 +439,7 @@ test "an unreadable data directory fails the scan and keeps the previous gauge"
// The handle keeps its read permission from open time, so `iterate` still
// lists the file, but path resolution under the directory now fails.
try tmp.dir.setPermissions(io, .fromMode(0o600));
monitor.sample(io);
monitor.sample(io, null, 0);
try tmp.dir.setPermissions(io, .fromMode(0o700));
try testing.expectEqual(@as(u64, 4096), monitor.gauges().db_bytes);
@@ -409,13 +462,115 @@ test "a threshold above the real free space drives the state to critical" {
".",
null,
);
monitor.sample(io);
monitor.sample(io, null, 0);
try testing.expectEqual(State.critical, monitor.state());
try testing.expect(!monitor.writesAllowed());
monitor.cfg = .{ .min_free_mb = 0, .warn_free_mb = 0 };
monitor.sample(io);
monitor.sample(io, null, 0);
try testing.expectEqual(State.ok, monitor.state());
try testing.expect(monitor.writesAllowed());
try testing.expectEqual(@as(u64, 0), monitor.sample_failures.load(.monotonic));
}
test "a disk transition records an episode per severity and closes it on recovery" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
const unreachable_mb = std.math.maxInt(u32);
var monitor: Monitor = .init(
.{ .min_free_mb = unreachable_mb, .warn_free_mb = unreachable_mb },
tmp.dir,
".",
null,
);
monitor.sample(io, &fx.store, 1000);
try testing.expectEqual(State.critical, monitor.state());
try testing.expectEqualStrings("disk.space", try fx.text(
"SELECT code FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings("error", try fx.text(
"SELECT severity FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings("data", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
// A second critical sample is the same episode, not a second row: `publish`
// only reports on a transition.
monitor.sample(io, &fx.store, 1060);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
monitor.cfg = .{ .min_free_mb = 0, .warn_free_mb = 0 };
monitor.sample(io, &fx.store, 1120);
try testing.expectEqual(State.ok, monitor.state());
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
try testing.expectEqual(@as(i64, 1120), try fx.count("SELECT resolved_at FROM operational_events"));
}
test "a failed probe opens an episode the next clean pass closes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var monitor: Monitor = .init(
.{ .min_free_mb = 0, .warn_free_mb = 0 },
tmp.dir,
".",
"./nxdns-no-such-dir-4f8a",
);
monitor.sample(io, &fx.store, 1000);
try testing.expectEqualStrings("disk.probe", try fx.text(
"SELECT code FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings("log_dir", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings("warning", try fx.text(
"SELECT severity FROM operational_events WHERE resolved_at IS NULL",
));
try tmp.dir.createDirPath(io, "logs");
var path_buf: [256]u8 = undefined;
monitor.log_dir_path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/logs", .{tmp.sub_path});
monitor.sample(io, &fx.store, 1100);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "every emit site is inert when the store is absent" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var monitor: Monitor = .init(
.{ .min_free_mb = std.math.maxInt(u32), .warn_free_mb = std.math.maxInt(u32) },
std.Io.Dir.cwd(),
"./nxdns-no-such-path-7c21",
"./nxdns-no-such-dir-4f8a",
);
monitor.sample(io, null, 0);
try testing.expectEqual(@as(u64, 1), monitor.sample_failures.load(.monotonic));
}
File diff suppressed because it is too large Load Diff
+47
View File
@@ -0,0 +1,47 @@
//! Test support: a migrated in-memory `config.db` with an `events.Store` over
//! it, for the emitter tests that live beside their own subsystem.
//!
//! Every emitter takes `?*events.Store` and must work with `null`; the tests
//! that prove an emitter *does* record something need a real store, and nine
//! subsystems needing the same six lines is what this file removes.
//!
//! Built in place rather than returned by value: a `Store` holds a `*db.Db`, so
//! a fixture that moved after `Store.init` would leave that pointer behind.
const std = @import("std");
const db = @import("db.zig");
const events = @import("events.zig");
const migrations = @import("migrations.zig");
pub const Fixture = struct {
database: db.Db = undefined,
store: events.Store = undefined,
text_buf: [1024]u8 = undefined,
pub fn init(self: *Fixture, io: std.Io, now_s: i64) !void {
self.database = try db.Db.open(":memory:", .{ .mode = .memory });
errdefer self.database.close();
try db.applyPragmas(&self.database, .{});
_ = try migrations.migrate(&self.database);
self.store = try events.Store.init(io, &self.database, now_s);
}
pub fn deinit(self: *Fixture) void {
self.database.close();
}
pub fn count(self: *Fixture, sql: []const u8) !i64 {
return self.database.queryInt(sql);
}
/// The one column a test names most often, for the newest row of `code`.
pub fn text(self: *Fixture, sql: []const u8) ![]const u8 {
var stmt = try self.database.prepare(sql);
defer stmt.deinit();
if (!try stmt.step()) return error.NoRow;
const value = stmt.columnText(0);
@memcpy(self.text_buf[0..value.len], value);
return self.text_buf[0..value.len];
}
};
+396 -32
View File
@@ -13,9 +13,19 @@
//! order; `QuerySink` calls them separately so both of its consumers see the
//! one transformed entry.
//!
//! One transaction per `logging.query_log_flush_interval_s`, not one per query.
//! At household rates a per-query commit costs orders of magnitude more disk
//! writes than the rows are worth — a minute of batching is what keeps an SD
//! card alive. The window is also what a crash costs: while the writer is
//! healthy and the disk gate is open, a process that dies loses about one
//! interval of query history. It is not a ceiling — a batch the gate is
//! holding, or one waiting on a write lock, is older than that — and nothing
//! here promises one, because query history is the cheapest data on the box.
//!
//! Log rows are expendable. A full queue drops the oldest unflushed entry, a
//! failed batch is dropped whole, and a disk that crossed the critical
//! threshold holds batches back indefinitely. Each of the three has a counter.
//! threshold holds batches back until it recovers — or, if the process is
//! already stopping, drops what it holds. Each of the three has a counter.
//! A writer that cannot prepare its statements closes the queue and marks
//! `writer_failed`, so the loss is visible rather than silent.
@@ -23,6 +33,7 @@ const std = @import("std");
const db = @import("db.zig");
const disk_monitor = @import("disk_monitor.zig");
const events = @import("events.zig");
const model = @import("../config/model.zig");
const queries_repo = @import("repositories/queries_repo.zig");
@@ -30,10 +41,10 @@ const queries_repo = @import("repositories/queries_repo.zig");
/// and the two names collide inside the struct.
const scope = std.log.scoped(.query_logger);
/// Flush tuning is comptime: §12.1 defines no configuration keys for it and a
/// household deployment has no reason to tune it.
/// Rows per transaction. Comptime: the window, not the batch size, is what an
/// operator has a reason to move, and a batch this size already amortizes the
/// commit at any household rate.
pub const flush_batch = 100;
pub const flush_interval_ms = 100;
/// What `hide_domains` and `hide_client_ips` store instead of the real value.
pub const hidden_marker = "hidden";
@@ -160,6 +171,14 @@ fn emptyAsNull(value: []const u8) ?[]const u8 {
const EntryQueue = std.Io.Queue(Entry);
/// A database failure is not in here: a batch the database refuses is dropped,
/// counted and reported where it happens, and the writer carries on.
/// `GatedAtShutdown` is the one condition `flush` cannot settle by itself —
/// the disk gate is shut and the process is stopping, so the batch in hand is
/// lost and so is everything still queued behind it. Only `runWriter` can see
/// both, so it does the counting and files the single episode.
pub const FlushError = std.Io.Cancelable || error{GatedAtShutdown};
/// What the flush interval race can produce. `Select` demands that each field
/// type match its task's return type exactly.
const Outcome = union(enum) {
@@ -177,6 +196,13 @@ pub const Logger = struct {
/// closed and every entry counts as dropped from that point, so a caller
/// that sees this must not expect rows.
writer_failed: std.atomic.Value(bool),
/// Set by `shutdown`, read by the disk gate. The gate holds a batch for as
/// long as the disk stays critical, which is right while the process is
/// running and a hang once it is stopping — nothing will ever release it.
draining: std.atomic.Value(bool),
/// Wired by the composition root after `init`, following the
/// `gate: ?*disk_monitor.Monitor` idiom. Null in every unit test here.
diagnostics: ?*events.Store = null,
/// `queue_buf.len` is the backpressure cap — the composition root
/// (`app.zig:311`) allocates `cfg.logging.query_log_buffer_max` entries,
@@ -191,6 +217,7 @@ pub const Logger = struct {
.rows_written = .init(0),
.batches_gated = .init(0),
.writer_failed = .init(false),
.draining = .init(false),
};
}
@@ -242,6 +269,11 @@ pub const Logger = struct {
/// whole life. Returns when `shutdown` closes the queue and the last batch
/// is flushed, or when the task is canceled.
///
/// It must outlive the producers rather than share their lifetime: a
/// cancellation that races the close decides at random whether the batch in
/// hand is written or counted as dropped. `app.zig` spawns this outside the
/// group it cancels for exactly that reason.
///
/// `monitor` is the §11.6 gate. Null disables gating.
pub fn runWriter(
self: *Logger,
@@ -254,8 +286,11 @@ pub const Logger = struct {
// Without a writer there is no consumer, so leaving the queue open
// would silently swallow every later entry.
self.writer_failed.store(true, .release);
// No recovery path claims this episode: the writer is gone for the
// life of the process, so the row stays active, which is the truth.
self.reportWrite(io, "writer", "preparing the batch statements failed", @errorName(err), 0);
self.queue.close(io);
self.dropRemaining(io);
_ = self.dropRemaining(io);
return;
};
defer writer.deinit();
@@ -268,10 +303,7 @@ pub const Logger = struct {
error.Closed => return,
error.Canceled => |e| return e,
};
const deadline: std.Io.Clock.Timestamp = .fromNow(io, .{
.raw = .fromMilliseconds(flush_interval_ms),
.clock = .awake,
});
const deadline = self.flushDeadline(io);
// `n` is live across both calls: entries already taken off the
// queue are lost if either one is canceled, so they must count.
var n: usize = 1;
@@ -279,17 +311,55 @@ pub const Logger = struct {
self.countDropped(n);
return err;
};
self.flush(io, &writer, batch[0..n], monitor) catch |err| {
self.countDropped(n);
return err;
self.flush(io, &writer, batch[0..n], monitor) catch |err| switch (err) {
error.Canceled => |e| {
self.countDropped(n);
return e;
},
// Nothing will open the gate now. Everything still queued is
// lost with the batch in hand, so it is counted here and
// announced once — a per-chunk report would write to the very
// disk that is out of space, dozens of times, on the way out.
error.GatedAtShutdown => {
self.countDropped(n);
const lost = n + self.dropRemaining(io);
scope.warn(
"query log: {d} rows dropped at shutdown, the disk gate was closed",
.{lost},
);
self.reportWrite(
io,
"batch",
"query log rows were dropped at shutdown",
"DiskCritical",
lost,
);
return;
},
};
}
}
/// Counts every entry left in a closed queue as dropped. The drain is
/// uncancelable: a cancellation racing the writer's own failure would
/// otherwise abandon the buffered entries without counting them.
fn dropRemaining(self: *Logger, io: std.Io) void {
/// When the batch that starts now must be committed. `.boot` and not
/// `.awake`: a suspended box would otherwise stretch the window by however
/// long it slept, and the rows are already in memory waiting.
///
/// An interval of 0 yields a deadline that has already passed, which is
/// exactly the documented sentinel — `fill` then takes what is queued and
/// returns without waiting for anything.
fn flushDeadline(self: *const Logger, io: std.Io) std.Io.Clock.Timestamp {
return .fromNow(io, .{
.raw = .fromSeconds(self.cfg.query_log_flush_interval_s),
.clock = .boot,
});
}
/// Counts every entry left in a closed queue as dropped, and returns how
/// many. The drain is uncancelable: a cancellation racing the writer's own
/// failure would otherwise abandon the buffered entries without counting
/// them.
fn dropRemaining(self: *Logger, io: std.Io) usize {
var total: usize = 0;
var leftover: [flush_batch]Entry = undefined;
while (true) {
const n = self.queue.getUncancelable(io, &leftover, 0) catch |err| switch (err) {
@@ -297,18 +367,24 @@ pub const Logger = struct {
};
if (n == 0) break;
self.countDropped(n);
total += n;
}
return total;
}
/// Closes the queue. `log` drops from here on and `runWriter` returns once
/// it has flushed what was left.
/// Closes the queue and tells the writer to stop waiting on anything that
/// may never arrive. `log` drops from here on, and `runWriter` returns once
/// it has flushed what was left — a blocked `getOne` on a closed queue
/// returns immediately, so the interval is never waited out at shutdown.
///
/// A writer held by the disk gate keeps holding: it flushes when the disk
/// recovers, and the `group.cancel` that follows this call in `app.zig`
/// stops a writer that will not wait. A canceled
/// writer counts the batch it holds under `queries_dropped`.
/// The close comes first: a writer that sees `draining` set must be able to
/// drain the queue to the end, and only a closed queue reports its end.
///
/// Every producer must be stopped and joined before this is called
/// (`app.zig`): an entry enqueued after the close is a dropped entry.
pub fn shutdown(self: *Logger, io: std.Io) void {
self.queue.close(io);
self.draining.store(true, .release);
}
/// Fills `batch` behind the entry already in slot 0, until it is full or
@@ -380,7 +456,7 @@ pub const Logger = struct {
writer: *queries_repo.BatchWriter,
entries: []const Entry,
monitor: ?*disk_monitor.Monitor,
) std.Io.Cancelable!void {
) FlushError!void {
if (entries.len == 0) return;
if (monitor) |m| {
@@ -389,6 +465,13 @@ pub const Logger = struct {
.clock = .awake,
};
while (!m.writesAllowed()) {
// Waiting for the disk to recover is right while the process
// runs and wrong once it is stopping: nothing is going to free
// space during shutdown, so the batch is lost either way and
// the only choice left is between losing it counted and
// hanging the exit. The counting and the one report belong to
// `runWriter`, which knows the rest of the queue is lost too.
if (self.draining.load(.acquire)) return error.GatedAtShutdown;
_ = self.batches_gated.fetchAdd(1, .monotonic);
try pause.sleep(io);
}
@@ -400,9 +483,41 @@ pub const Logger = struct {
writer.writeBatch(rows[0..entries.len]) catch |err| {
scope.warn("query log batch of {d} rows dropped: {s}", .{ entries.len, @errorName(err) });
self.countDropped(entries.len);
self.reportWrite(io, "batch", "a query log batch was dropped", @errorName(err), entries.len);
return;
};
_ = self.rows_written.fetchAdd(entries.len, .monotonic);
if (self.diagnostics) |store| {
store.resolve(io, std.Io.Clock.real.now(io).toSeconds(), .query_log_write, "batch");
}
}
/// An error, not a warning: dropped query rows are gone, and a writer that
/// never started means every later row is gone too.
fn reportWrite(
self: *Logger,
io: std.Io,
operation: []const u8,
message: []const u8,
error_name: []const u8,
rows: usize,
) void {
const store = self.diagnostics orelse return;
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "{s}: {s} ({d} rows)", .{
message,
error_name,
rows,
}) catch buf[0..];
store.report(
io,
std.Io.Clock.real.now(io).toSeconds(),
.query_log_write,
operation,
operation,
.@"error",
detail,
);
}
fn countDropped(self: *Logger, n: usize) void {
@@ -441,6 +556,7 @@ fn outcomeEntry(outcome: Outcome) ?Entry {
// tests
// ---------------------------------------------------------------------------
const events_fixture = @import("events_fixture.zig");
const querylog_schema = @import("querylog_schema.zig");
const testing = std.testing;
@@ -651,7 +767,7 @@ test "log after shutdown drops instead of blocking" {
try testing.expectEqual(@as(u64, 1), logger.queries_dropped.load(.monotonic));
}
test "the writer drains every entry and shutdown ends it" {
test "shutdown writes the batch the writer holds and the rest of the queue" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
@@ -660,7 +776,9 @@ test "the writer drains every entry and shutdown ends it" {
defer database.close();
var buf: [512]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
// An hour: nothing here can be explained by the window expiring. Every row
// that lands does so because the close released it.
var logger: Logger = .init(.{ .query_log_flush_interval_s = 3600 }, &buf);
var future = try io.concurrent(Logger.runWriter, .{
&logger,
@@ -674,6 +792,10 @@ test "the writer drains every entry and shutdown ends it" {
const written = try std.fmt.bufPrint(name, "d{d}.example", .{i % 10});
logger.log(io, sampleEntry(@intCast(i), written));
}
// The app.zig order: the only producer is this task and it is done, so the
// close cannot lose an entry, and the writer is awaited rather than
// canceled.
logger.shutdown(io);
try future.await(io);
@@ -683,7 +805,57 @@ test "the writer drains every entry and shutdown ends it" {
try testing.expectEqual(@as(i64, 10), try queries_repo.countDomains(&database));
}
test "the writer flushes an entry once the interval passes" {
test "entries that arrive inside one window reach the database in one batch" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var writer = try queries_repo.BatchWriter.init(&database);
defer writer.deinit();
var buf: [16]Entry = undefined;
var logger: Logger = .init(.{ .query_log_flush_interval_s = 3600 }, &buf);
for (0..6) |i| logger.log(io, sampleEntry(@intCast(i), "batched.example"));
// No further producer exists, so the close is what ends the fill — the same
// thing that ends it at shutdown.
logger.shutdown(io);
var batch: [flush_batch]Entry = undefined;
batch[0] = try logger.queue.getOne(io);
var n: usize = 1;
try logger.fill(io, &batch, logger.flushDeadline(io), &n);
try testing.expectEqual(@as(usize, 6), n);
// One `writeBatch` call, which is one transaction (`queries_repo.zig`).
try logger.flush(io, &writer, batch[0..n], null);
try testing.expectEqual(@as(u64, 6), logger.rows_written.load(.monotonic));
try testing.expectEqual(@as(i64, 6), try queries_repo.countRows(&database));
}
test "a zero interval takes what is queued and waits for nothing" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var buf: [8]Entry = undefined;
var logger: Logger = .init(.{ .query_log_flush_interval_s = 0 }, &buf);
logger.log(io, sampleEntry(1, "now.example"));
logger.log(io, sampleEntry(2, "now.example"));
var batch: [flush_batch]Entry = undefined;
batch[0] = try logger.queue.getOne(io);
var n: usize = 1;
// The queue is open and the batch has room: any non-zero interval blocks
// here until it expires. Zero returns with what was already queued.
try logger.fill(io, &batch, logger.flushDeadline(io), &n);
try testing.expectEqual(@as(usize, 2), n);
}
test "the writer holds an entry for the length of the flush interval" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
@@ -692,7 +864,7 @@ test "the writer flushes an entry once the interval passes" {
defer database.close();
var buf: [8]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
var logger: Logger = .init(.{ .query_log_flush_interval_s = 1 }, &buf);
var future = try io.concurrent(Logger.runWriter, .{
&logger,
@@ -703,20 +875,71 @@ test "the writer flushes an entry once the interval passes" {
logger.log(io, sampleEntry(1, "only.example"));
// Sampled, not asserted, while the writer runs: an assertion that fails
// here would return before the writer is stopped, and `Threaded.deinit`
// then waits on a task nothing will ever end.
const quarter: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(250), .clock = .awake };
try quarter.sleep(io);
const written_at_a_quarter = logger.rows_written.load(.monotonic);
// Three times the interval. A flush that has not happened by then is a
// failure, not slowness.
const poll: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(5), .clock = .awake };
var waited: usize = 0;
while (logger.rows_written.load(.monotonic) == 0) : (waited += 1) {
// Ten times the interval; a flush that has not happened by then is a
// failure, not slowness.
try testing.expect(waited < 200);
while (logger.rows_written.load(.monotonic) == 0 and waited < 600) : (waited += 1) {
try poll.sleep(io);
}
logger.shutdown(io);
try future.await(io);
// A writer that commits per query has already written at a quarter of the
// window; this one has not.
try testing.expectEqual(@as(u64, 0), written_at_a_quarter);
try testing.expect(waited < 600);
try testing.expectEqual(@as(u64, 1), logger.rows_written.load(.monotonic));
try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(&database));
}
test "a full batch flushes without waiting for the interval" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var buf: [256]Entry = undefined;
var logger: Logger = .init(.{ .query_log_flush_interval_s = 3600 }, &buf);
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
&database,
@as(?*disk_monitor.Monitor, null),
});
for (0..150) |i| logger.log(io, sampleEntry(@intCast(i), "burst.example"));
// The window is an hour away, so only a full batch can release a flush.
const poll: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(5), .clock = .awake };
var waited: usize = 0;
while (logger.rows_written.load(.monotonic) < flush_batch and waited < 400) : (waited += 1) {
try poll.sleep(io);
}
// Sampled with the writer still running, asserted once it has stopped.
const written_before_shutdown = logger.rows_written.load(.monotonic);
logger.shutdown(io);
try future.await(io);
// Exactly one batch went out early, and the 50 behind it waited for the
// close rather than for the hour.
try testing.expect(waited < 400);
try testing.expectEqual(@as(u64, flush_batch), written_before_shutdown);
try testing.expectEqual(@as(i64, 150), try queries_repo.countRows(&database));
}
test "a gated flush holds the batch until the disk recovers" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -827,7 +1050,9 @@ test "a canceled writer counts the batch it was holding" {
defer database.close();
var buf: [8]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
// Zero interval: the writer reaches the gate with what is queued instead of
// waiting a minute for a producer that no longer exists.
var logger: Logger = .init(.{ .query_log_flush_interval_s = 0 }, &buf);
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
@@ -858,6 +1083,67 @@ test "a canceled writer counts the batch it was holding" {
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
}
test "a disk-gated writer drops what it holds at shutdown instead of hanging" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var buf: [512]Entry = undefined;
var logger: Logger = .init(.{ .query_log_flush_interval_s = 0 }, &buf);
logger.diagnostics = &fx.store;
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
logger.log(io, sampleEntry(1, "held.example"));
logger.log(io, sampleEntry(2, "held.example"));
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
&database,
@as(?*disk_monitor.Monitor, &monitor),
});
const poll: std.Io.Clock.Duration = .{ .raw = .fromMilliseconds(5), .clock = .awake };
var waited: usize = 0;
while (logger.batches_gated.load(.monotonic) == 0 and waited < 400) : (waited += 1) {
try poll.sleep(io);
}
// Two and a half chunks wait behind the batch the gate is holding, so a
// drain that reported per chunk would file three episodes' worth of writes
// to the disk that is out of space.
const queued = 250;
for (0..queued) |i| logger.log(io, sampleEntry(@intCast(i + 3), "queued.example"));
// The disk never recovers. The wait still ends, and every entry is counted.
logger.shutdown(io);
try future.await(io);
try testing.expect(waited < 400);
try testing.expectEqual(@as(u64, queued + 2), logger.queries_dropped.load(.monotonic));
try testing.expectEqual(@as(u64, 0), logger.rows_written.load(.monotonic));
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
// One episode, reported once, and its detail carries the whole loss rather
// than the size of whichever chunk was last.
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT occurrences FROM operational_events"));
try testing.expectEqualStrings("batch", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings(
"query log rows were dropped at shutdown: DiskCritical (252 rows)",
try fx.text("SELECT detail FROM operational_events"),
);
}
test "an empty batch touches neither the database nor the counters" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -881,3 +1167,81 @@ test "an empty batch touches neither the database nor the counters" {
try testing.expectEqual(@as(u64, 0), logger.rows_written.load(.monotonic));
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
}
test "a dropped batch opens an error episode the next good batch closes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
try database.exec(
\\CREATE TRIGGER refuse_boom BEFORE INSERT ON query_log
\\WHEN new.client_ip = 'boom'
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var writer = try queries_repo.BatchWriter.init(&database);
defer writer.deinit();
var buf: [4]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
logger.diagnostics = &fx.store;
var doomed = sampleEntry(10, "poison.example");
doomed.setClientIp("boom");
const bad = [_]Entry{doomed};
try logger.flush(io, &writer, &bad, null);
try testing.expectEqualStrings("query_log.write", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("batch", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("error", try fx.text("SELECT severity FROM operational_events"));
const good = [_]Entry{sampleEntry(11, "next.example")};
try logger.flush(io, &writer, &good, null);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a writer that cannot prepare leaves an episode no recovery path claims" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
// No schema: `BatchWriter.init` cannot prepare against a missing table.
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var buf: [8]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
logger.diagnostics = &fx.store;
try logger.runWriter(io, &database, null);
try testing.expectEqualStrings("writer", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
try testing.expectEqualStrings("error", try fx.text(
"SELECT severity FROM operational_events WHERE resolved_at IS NULL",
));
// The writer returned, so nothing can ever close this. A second run finds
// the queue closed and adds no second episode.
try logger.runWriter(io, &database, null);
try testing.expectEqual(
@as(i64, 1),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
+113 -4
View File
@@ -55,7 +55,24 @@ fn assertOrdered(list: []const Step) void {
/// foreign_keys` is a no-op inside a transaction, so applying it afterwards
/// would silently leave referential integrity off.
pub fn migrate(database: *db.Db) Error!u32 {
return migrateSteps(database, &steps);
const version = try migrateSteps(database, &steps);
try bridgeOperationalEvents(database);
return version;
}
/// **Removable when the v0.1 adoption gate lands.**
///
/// `operational_events` joined `config_schema.ddl_v1` after databases stamped
/// version 1 already existed, and a stamped database never runs step 1 again —
/// so on those installs nothing would ever create the table, silently, and the
/// diagnostics store would fail to open forever. This runs after the stamp,
/// with every statement conditional, and touches no other table.
///
/// It belongs here and not in `cli.openConfigDb`: that runs *before* migration
/// everywhere (`app.zig`, `cli.zig`), so creating the table there would make a
/// fresh database's unconditional `CREATE TABLE` in `ddl_v1` fail.
fn bridgeOperationalEvents(database: *db.Db) db.Error!void {
return database.exec(config_schema.operational_events_bridge);
}
/// Same logic against an injected step list. The seam exists for the rollback
@@ -159,7 +176,7 @@ test "migrate on a fresh database creates every table and seeds the default grou
const expected = [_][]const u8{
"schema_version", "groups", "clients", "client_prefixes",
"upstreams", "rules", "local_records", "forward_zones",
"blocklist_sources", "group_sources", "settings",
"blocklist_sources", "group_sources", "settings", "operational_events",
};
for (expected) |name| {
try testing.expect(try tableExists(&database, name));
@@ -373,9 +390,101 @@ test "delete_order and table_names name exactly the tables the schema creates" {
for (config_schema.table_names) |name| {
try testing.expect(try tableExists(&database, name));
}
// delete_order covers every table except `schema_version`.
// delete_order covers every table except two: `schema_version`, which is
// the migration's own, and `operational_events`, which is runtime state an
// import must never wipe.
try testing.expect(!try tableExists(&database, "no_such_table"));
try testing.expect(try tableExists(&database, "operational_events"));
try testing.expectEqual(
@as(i64, config_schema.delete_order.len + 1),
@as(i64, config_schema.delete_order.len + 2),
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
);
}
test "a version-1 database created without operational_events gains exactly it" {
// The silent divergence the bridge exists for: this is what the Pi's
// `config.db` looks like — stamped 1, so step 1 never runs again.
var database = try openMigrated();
defer database.close();
try database.exec(config_schema.ddl_v1);
try database.exec("DROP TABLE operational_events;");
try database.exec("INSERT INTO schema_version (version) VALUES (1);");
try testing.expect(!try tableExists(&database, "operational_events"));
const tables_before = try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'");
const groups_before = try database.queryInt("SELECT count(*) FROM groups");
try testing.expectEqual(@as(u32, 1), try migrate(&database));
try testing.expect(try tableExists(&database, "operational_events"));
try testing.expectEqual(
tables_before + 1,
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
);
// Both indexes came with it, and no other table moved.
try testing.expectEqual(
@as(i64, 2),
try database.queryInt(
"SELECT count(*) FROM sqlite_schema WHERE type='index' AND tbl_name='operational_events'",
),
);
try testing.expectEqual(groups_before, try database.queryInt("SELECT count(*) FROM groups"));
}
test "the bridge does not double-create on a fresh database or on a second run" {
var database = try openMigrated();
defer database.close();
// `ddl_v1` creates the table unconditionally, so a bridge that ran as part
// of the step list would fail here rather than be a no-op.
try testing.expectEqual(@as(u32, 1), try migrate(&database));
const schema_rows = try database.queryInt("SELECT count(*) FROM sqlite_schema");
try database.exec(
\\INSERT INTO operational_events
\\ (code, subject_key, subject_label, severity, first_seen, last_seen, occurrences)
\\VALUES ('disk.space', 'data', 'data', 'warning', 100, 100, 1);
);
try testing.expectEqual(@as(u32, 1), try migrate(&database));
try testing.expectEqual(schema_rows, try database.queryInt("SELECT count(*) FROM sqlite_schema"));
// A `CREATE TABLE IF NOT EXISTS` that had somehow replaced the table would
// show up as a lost row, not as a schema difference.
try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM operational_events"));
}
test "the partial unique index allows one active row per key and any number of resolved ones" {
var database = try openMigrated();
defer database.close();
_ = try migrate(&database);
const insert =
\\INSERT INTO operational_events
\\ (code, subject_key, subject_label, severity, first_seen, last_seen, occurrences, resolved_at)
\\VALUES ('blocklist.refresh', 'https://a.example', 'a', 'warning', 100, 100, 1, ?1)
;
{
var stmt = try database.prepare(insert);
defer stmt.deinit();
try stmt.bindNull(1);
try stmt.exec();
}
{
// A second active row for the same (code, subject_key) is what
// `report`'s overflow probe exists to avoid, and the index proves it.
// Its own statement: `Stmt.reset` re-reports the code of a failed step,
// so a reused one would answer `error.Constraint` a second time.
var stmt = try database.prepare(insert);
defer stmt.deinit();
try stmt.bindNull(1);
try testing.expectError(error.Constraint, stmt.step());
}
var resolved = try database.prepare(insert);
defer resolved.deinit();
for ([_]i64{ 200, 300 }) |resolved_at| {
try resolved.reset();
try resolved.bindInt(1, resolved_at);
try resolved.exec();
}
try testing.expectEqual(@as(i64, 3), try database.queryInt("SELECT count(*) FROM operational_events"));
}
+19 -10
View File
@@ -228,7 +228,11 @@ test "S8 case 2: a single entry reaches the file once the flush interval passes"
defer log_db.deinit();
var queue_buf: [8]logger.Entry = undefined;
var query_log: logger.Logger = .init(.{}, &queue_buf);
// A one-second window, not the 60-second default: this case is about the
// entry reaching the file on the interval alone, with nothing to close the
// queue for it.
const flush_interval_s = 1;
var query_log: logger.Logger = .init(.{ .query_log_flush_interval_s = flush_interval_s }, &queue_buf);
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
@@ -239,9 +243,9 @@ test "S8 case 2: a single entry reaches the file once the flush interval passes"
query_log.log(io, entryAt(1, "only.example"));
// Ten flush intervals of headroom: a row that has not landed by then is a
// Three flush intervals of headroom: a row that has not landed by then is a
// failure of the interval race, not a slow machine.
const limit = 10 * logger.flush_interval_ms / 5;
const limit = 3 * flush_interval_s * 1000 / 5;
try awaitCount(&query_log.rows_written, 1, limit);
try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(log_db.database()));
@@ -268,7 +272,9 @@ test "S8 case 3: a full queue drops the oldest entries and the newest survive" {
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
var queue_buf: [8]logger.Entry = undefined;
var query_log: logger.Logger = .init(.{}, &queue_buf);
// Zero interval: this case is about the queue cap and the gate, so the
// writer takes what is queued and goes straight to the gate.
var query_log: logger.Logger = .init(.{ .query_log_flush_interval_s = 0 }, &queue_buf);
// The whole burst is enqueued before the writer starts. A writer already
// draining the queue would take entries out of it mid-burst and make the
@@ -286,8 +292,9 @@ test "S8 case 3: a full queue drops the oldest entries and the newest survive" {
try awaitCount(&query_log.batches_gated, 1, 200);
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(log_db.database()));
// Un-gate before the shutdown: a writer held by the disk gate holds its
// batch, and `shutdown` alone would never release it.
// Un-gate before the shutdown: a writer still held by the gate counts its
// batch as dropped rather than writing it, which is a different case (the
// logger's own suite covers it).
monitor.state_raw.store(@intFromEnum(disk_monitor.State.ok), .monotonic);
try awaitCount(&query_log.rows_written, 8, 300);
query_log.shutdown(io);
@@ -360,7 +367,7 @@ test "S8 case 5: a retention pass prunes the old rows and truncates the write-ah
try testing.expect(try f.sizeOf("querylog.db-wal") > 0);
var pass: retention.Retention = .init(.{ .retention_days = 30 });
pass.runOnce(io, log_db.database(), null);
pass.runOnce(io, log_db.database(), null, null);
try testing.expectEqual(@as(u64, 1), pass.snapshotStats().passes);
try testing.expectEqual(@as(u64, 2), pass.snapshotStats().rows_pruned);
@@ -396,14 +403,16 @@ test "S8 case 6: a critical disk gates the flushes and recovery releases them" {
data_path,
null,
);
monitor.sample(io);
monitor.sample(io, null, 0);
try testing.expectEqual(disk_monitor.State.critical, monitor.state());
try testing.expect(!monitor.writesAllowed());
try testing.expect(monitor.gauges().free_bytes > 0);
try testing.expect(monitor.gauges().db_bytes > 0);
var queue_buf: [64]logger.Entry = undefined;
var query_log: logger.Logger = .init(.{}, &queue_buf);
// Zero interval: the case is the gate's hold and release, so the writer
// takes the five queued entries straight to it.
var query_log: logger.Logger = .init(.{ .query_log_flush_interval_s = 0 }, &queue_buf);
for (0..5) |i| query_log.log(io, entryAt(@intCast(i), "gated.example"));
var future = try io.concurrent(logger.Logger.runWriter, .{
@@ -418,7 +427,7 @@ test "S8 case 6: a critical disk gates the flushes and recovery releases them" {
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(log_db.database()));
monitor.cfg = .{ .min_free_mb = 0, .warn_free_mb = 0 };
monitor.sample(io);
monitor.sample(io, null, 0);
try testing.expectEqual(disk_monitor.State.ok, monitor.state());
try testing.expect(monitor.writesAllowed());
+121 -11
View File
@@ -45,6 +45,24 @@ pub const ddl: [:0]const u8 =
\\CREATE INDEX idx_query_log_ts ON query_log(timestamp);
\\CREATE INDEX idx_query_log_client ON query_log(client_ip);
\\CREATE INDEX idx_query_log_domain ON query_log(domain_id);
\\
\\CREATE TABLE upstream_targets (
\\ id INTEGER PRIMARY KEY,
\\ url TEXT NOT NULL UNIQUE -- the historical identity: config.db ids cannot cross database files
\\);
\\
\\CREATE TABLE upstream_minute (
\\ upstream_id INTEGER NOT NULL REFERENCES upstream_targets(id),
\\ minute_ts INTEGER NOT NULL,
\\ successes INTEGER NOT NULL,
\\ failures INTEGER NOT NULL,
\\ last_failure_ts INTEGER,
\\ last_error TEXT,
\\ PRIMARY KEY (upstream_id, minute_ts),
\\ CHECK (successes >= 0),
\\ CHECK (failures >= 0)
\\) WITHOUT ROWID;
\\CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
;
/// `PRAGMA user_version` is a signed 32-bit field. Deriving the fingerprint from
@@ -70,6 +88,18 @@ pub const OpenResult = struct {
database: db.Db,
/// Non-null feeds a counter and the `/api/health` rollup.
recreated: ?RecreateReason,
/// The path the previous file was kept as, by value. It existed only in a
/// stack buffer inside `open` before the diagnostics event needed it, and a
/// slice of that buffer would dangle the moment `open` returned.
///
/// Empty when nothing was renamed aside, which `.missing` and a clean open
/// both are.
aside_buf: [path_buf_len]u8 = undefined,
aside_len: u16 = 0,
pub fn aside(self: *const OpenResult) []const u8 {
return self.aside_buf[0..self.aside_len];
}
};
pub const Error = db.Error || error{AsideNameCollision} ||
@@ -120,7 +150,7 @@ pub fn open(io: std.Io, dir: std.Io.Dir, path: [:0]const u8) Error!OpenResult {
const aside: ?[]const u8 = if (cause == .missing)
null
else
try renameAside(io, dir, path, &aside_buf);
try renameAside(io, dir, path, cause, &aside_buf);
// Not optional: a stale WAL left beside the renamed database would be
// replayed into the freshly created file and corrupt it immediately. Any
@@ -138,7 +168,12 @@ pub fn open(io: std.Io, dir: std.Io.Dir, path: [:0]const u8) Error!OpenResult {
aside.?,
});
}
return .{ .database = fresh, .recreated = cause };
var result: OpenResult = .{ .database = fresh, .recreated = cause };
if (aside) |name| {
result.aside_len = @intCast(name.len);
@memcpy(result.aside_buf[0..name.len], name);
}
return result;
}
/// The whitelist. `null` means "propagate, do not touch the file".
@@ -160,20 +195,37 @@ fn quickCheck(database: *db.Db) db.Error!bool {
return std.ascii.eqlIgnoreCase(stmt.columnText(0), "ok");
}
/// What the aside file's name calls the reason it was set aside.
///
/// The name is the only account of the reason an operator gets: the log line
/// naming it scrolls away, the file stays for months. `fingerprint_mismatch` is
/// a database with nothing wrong with it — this build's DDL moved — so calling
/// its file "corrupt" invites the operator to delete evidence of a healthy file.
fn asideTag(reason: RecreateReason) []const u8 {
return switch (reason) {
.missing => unreachable, // there is no file to rename
.corrupt => "corrupt",
.not_a_database => "not-a-database",
.quick_check_failed => "quick-check-failed",
.fingerprint_mismatch => "schema-changed",
};
}
/// Renames the unusable file out of the way and returns the name it now has.
///
/// `renamePreserve` is `RENAME_NOREPLACE`: it returns `error.PathAlreadyExists`
/// instead of overwriting. A previously saved corrupt file must never be
/// destroyed by the next recreate, and two recreates in the same second are not
/// hypothetical on a boot loop — hence the uniquifying retries.
fn renameAside(io: std.Io, dir: std.Io.Dir, path: []const u8, buf: []u8) Error![]const u8 {
/// instead of overwriting. A previously saved file must never be destroyed by
/// the next recreate, and two recreates in the same second are not hypothetical
/// on a boot loop — hence the uniquifying retries.
fn renameAside(io: std.Io, dir: std.Io.Dir, path: []const u8, reason: RecreateReason, buf: []u8) Error![]const u8 {
const tag = asideTag(reason);
const seconds = std.Io.Clock.real.now(io).toSeconds();
var attempt: u32 = 0;
while (attempt < 100) : (attempt += 1) {
const aside = if (attempt == 0)
std.fmt.bufPrint(buf, "{s}.corrupt-{d}", .{ path, seconds }) catch return error.NameTooLong
std.fmt.bufPrint(buf, "{s}.{s}-{d}", .{ path, tag, seconds }) catch return error.NameTooLong
else
std.fmt.bufPrint(buf, "{s}.corrupt-{d}-{d}", .{ path, seconds, attempt }) catch return error.NameTooLong;
std.fmt.bufPrint(buf, "{s}.{s}-{d}-{d}", .{ path, tag, seconds, attempt }) catch return error.NameTooLong;
dir.renamePreserve(path, dir, aside, io) catch |e| switch (e) {
error.PathAlreadyExists => continue,
@@ -215,20 +267,21 @@ test "fingerprint matches a fresh hash of the DDL" {
try testing.expectEqual(fingerprint, @as(i32, @bitCast(std.hash.Crc32.hash(ddl))));
}
test "ddl creates domains, query_log and the three indexes" {
test "ddl creates the query-log tables, the upstream-history tables and every index" {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try db.applyPragmas(&database, .{});
try database.exec(ddl);
try testing.expectEqual(
@as(i64, 2),
@as(i64, 4),
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
);
const objects = [_][]const u8{
"domains", "query_log",
"idx_query_log_ts", "idx_query_log_client",
"idx_query_log_domain",
"idx_query_log_domain", "upstream_targets",
"upstream_minute", "idx_upstream_minute_ts",
};
for (objects) |name| {
var stmt = try database.prepare("SELECT count(*) FROM sqlite_schema WHERE name = ?1");
@@ -268,6 +321,13 @@ test "recreatable is a whitelist and never selects a resource error" {
}
}
test "the aside name says why, and a healthy file is never called corrupt" {
try testing.expectEqualStrings("corrupt", asideTag(.corrupt));
try testing.expectEqualStrings("not-a-database", asideTag(.not_a_database));
try testing.expectEqualStrings("quick-check-failed", asideTag(.quick_check_failed));
try testing.expectEqualStrings("schema-changed", asideTag(.fingerprint_mismatch));
}
test "recreatable selects exactly two of db.Error's members" {
// Exhaustive over the whole set, so a variant added to `db.Error` later
// defaults to propagate. The list above only proves the named errors are
@@ -278,3 +338,53 @@ test "recreatable selects exactly two of db.Error's members" {
}
try testing.expectEqual(@as(usize, 2), whitelisted);
}
test "a recreate returns the aside name by value and a fresh create returns none" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
var path_buf: [path_buf_len]u8 = undefined;
const path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/querylog.db", .{tmp.sub_path});
// First open: the file is missing, so nothing is renamed aside. The
// one-shot event is deliberately not emitted for this case.
var created = try open(io, std.Io.Dir.cwd(), path);
created.database.close();
try testing.expectEqual(RecreateReason.missing, created.recreated.?);
try testing.expectEqualStrings("", created.aside());
try tmp.dir.writeFile(io, .{ .sub_path = "querylog.db", .data = "not a database at all" });
var recreated = try open(io, std.Io.Dir.cwd(), path);
recreated.database.close();
try testing.expectEqual(RecreateReason.not_a_database, recreated.recreated.?);
try testing.expect(recreated.aside().len != 0);
// The name is a real file, which is the whole reason it travels out.
const kept = std.fs.path.basename(recreated.aside());
try tmp.dir.access(io, kept, .{});
}
test "a clean reopen reports no recreate and no aside" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
var path_buf: [path_buf_len]u8 = undefined;
const path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/querylog.db", .{tmp.sub_path});
var first = try open(io, std.Io.Dir.cwd(), path);
first.database.close();
var second = try open(io, std.Io.Dir.cwd(), path);
second.database.close();
try testing.expectEqual(@as(?RecreateReason, null), second.recreated);
try testing.expectEqualStrings("", second.aside());
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,493 @@
//! `upstream_minute` and its `upstream_targets` dimension table in
//! `querylog.db` (milestone-26 rulings 2, 4, 5).
//!
//! One row per upstream per wall-clock UTC minute that had at least one
//! attempt. Rows are additive facts: a flush adds to whatever is already there,
//! so a restart inside a minute continues that minute's row rather than
//! replacing it, and nothing here can lower a stored count.
//!
//! Identity is the url, not the `config.db` upstream id: ids cannot be foreign
//! keys across database files and may be deleted or reused. Editing an
//! upstream's url deliberately starts a new history.
//!
//! Nothing here retries. The accumulator owns what a failed flush means
//! (`upstream/history.zig`).
const std = @import("std");
const db = @import("../db.zig");
const health = @import("../../upstream/health.zig");
/// How far back `Retention` keeps minute rows. A fixed window, not a knob
/// (m26 anti-requirements), and deliberately wider than the widest dashboard
/// period: `stats.window` derives `since = until - width * count` with
/// `until > now`, so a 30-day window never asks for anything older than
/// `now - 30d`. The extra day is slack for a retention pass that runs late.
///
/// `logging.retention_days` does not apply here. It bounds the query log, whose
/// rows are per-query; these are per-minute aggregates whose whole purpose is
/// to outlive them.
pub const retention_window_s: i64 = 31 * 86_400;
/// One minute of one upstream's outcomes, as the accumulator hands it over.
/// Every string is borrowed for the duration of the call: `Stmt.bindText` binds
/// with `SQLITE_TRANSIENT`, so SQLite copies before `flush` returns.
pub const FlushRow = struct {
url: []const u8,
minute_ts: i64,
successes: u32,
failures: u32,
last_failure_ts: ?i64,
/// Empty when the minute held no failure.
last_error: []const u8,
};
const insert_target_sql = "INSERT OR IGNORE INTO upstream_targets (url) VALUES (?1)";
const select_target_sql = "SELECT id FROM upstream_targets WHERE url = ?1";
/// Additive, and the timestamp columns are max-wins, which is what makes a
/// flush safe to repeat against a row another process already wrote.
///
/// `max()` over a NULL is NULL in SQLite, so the coalesce is what keeps an
/// existing `last_failure_ts` when the incoming row carries none. The `CASE`
/// moves `last_error` with the timestamp it belongs to: a success-only upsert
/// leaves the stored failure and its name exactly as they were.
const upsert_minute_sql =
\\INSERT INTO upstream_minute (upstream_id, minute_ts, successes, failures, last_failure_ts, last_error)
\\VALUES (?1, ?2, ?3, ?4, ?5, ?6)
\\ON CONFLICT(upstream_id, minute_ts) DO UPDATE SET
\\ successes = successes + excluded.successes,
\\ failures = failures + excluded.failures,
\\ last_failure_ts = coalesce(max(last_failure_ts, excluded.last_failure_ts), last_failure_ts, excluded.last_failure_ts),
\\ last_error = CASE
\\ WHEN excluded.last_failure_ts IS NOT NULL
\\ AND (last_failure_ts IS NULL OR excluded.last_failure_ts >= last_failure_ts)
\\ THEN excluded.last_error
\\ ELSE last_error
\\ END
;
/// One transaction for the whole batch: either every minute of the pass lands
/// or none of it does, so a failed flush leaves nothing half-written for the
/// caller's merge-back to double-count.
pub fn flush(database: *db.Db, rows: []const FlushRow) db.Error!void {
if (rows.len == 0) return;
var insert_target = try database.prepare(insert_target_sql);
defer insert_target.deinit();
var select_target = try database.prepare(select_target_sql);
defer select_target.deinit();
var upsert = try database.prepare(upsert_minute_sql);
defer upsert.deinit();
var tx = try db.Tx.begin(database);
errdefer tx.rollback();
for (rows) |row| {
const upstream_id = try internTarget(&insert_target, &select_target, row.url);
try upsert.reset();
try upsert.bindInt(1, upstream_id);
try upsert.bindInt(2, row.minute_ts);
try upsert.bindInt(3, row.successes);
try upsert.bindInt(4, row.failures);
if (row.last_failure_ts) |at| try upsert.bindInt(5, at) else try upsert.bindNull(5);
try upsert.bindText(6, row.last_error);
try upsert.exec();
}
try tx.commit();
}
fn internTarget(insert: *db.Stmt, select: *db.Stmt, url: []const u8) db.Error!i64 {
try insert.reset();
try insert.bindText(1, url);
try insert.exec();
try select.reset();
try select.bindText(1, url);
// The insert above either created the row or found it already there, so a
// miss means the table changed under this connection.
if (!try select.step()) return error.NotFound;
const id = select.columnInt(0);
// A statement stopped on a row keeps its cursor open until it is reset;
// the transaction must not carry that to the next row.
try select.reset();
return id;
}
/// What `GET /api/upstream/health` reports for one upstream over one window.
pub const WindowStats = struct {
attempts: u64,
successes: u64,
failures: u64,
last_failure_ts: ?i64,
/// The error name of the row holding the newest `last_failure_ts` in the
/// window; empty when the window holds no failure.
///
/// By value rather than by slice: the caller loops over upstreams and
/// reuses one `WindowStats`, so a borrowed slice would dangle into the
/// storage the next iteration overwrites.
last_failure_error_buf: [health.error_name_capacity]u8,
last_failure_error_len: u8,
pub fn lastFailureError(self: *const WindowStats) []const u8 {
return self.last_failure_error_buf[0..self.last_failure_error_len];
}
};
/// **One statement, deliberately.** Two statements would not share a SQLite
/// snapshot: the flush connection can commit between them, and the read would
/// then pair a `max(last_failure_ts)` taken from one state with an error text
/// taken from another.
///
/// The error lookup is a scalar subquery for the same reason it is not a bare
/// column: `SELECT max(last_failure_ts), last_error` lets SQLite return the
/// `last_error` of an arbitrary row of the group. `ORDER BY ... DESC, minute_ts
/// DESC` makes the choice deterministic when two minutes share a timestamp.
const window_stats_sql =
\\SELECT coalesce(sum(m.successes), 0), coalesce(sum(m.failures), 0), max(m.last_failure_ts),
\\ (SELECT e.last_error FROM upstream_minute e
\\ WHERE e.upstream_id = m.upstream_id AND e.minute_ts >= ?2 AND e.minute_ts < ?3
\\ AND e.last_failure_ts IS NOT NULL
\\ ORDER BY e.last_failure_ts DESC, e.minute_ts DESC LIMIT 1)
\\ FROM upstream_minute m JOIN upstream_targets t ON t.id = m.upstream_id
\\ WHERE t.url = ?1 AND m.minute_ts >= ?2 AND m.minute_ts < ?3
;
/// Aggregates `[since, until)` by `minute_ts`. An unknown url or an empty
/// window is zeros, a null timestamp and the empty error — not an error.
pub fn windowStats(database: *db.Db, url: []const u8, since: i64, until: i64) db.Error!WindowStats {
var stmt = try database.prepare(window_stats_sql);
defer stmt.deinit();
try stmt.bindText(1, url);
try stmt.bindInt(2, since);
try stmt.bindInt(3, until);
// A bare aggregate always produces exactly one row; no row means the
// statement is not the one this function prepared.
if (!try stmt.step()) return error.Misuse;
const successes = try countOf(stmt.columnInt(0));
const failures = try countOf(stmt.columnInt(1));
var out: WindowStats = .{
.attempts = successes + failures,
.successes = successes,
.failures = failures,
.last_failure_ts = if (stmt.isNull(2)) null else stmt.columnInt(2),
.last_failure_error_buf = @splat(0),
.last_failure_error_len = 0,
};
const name = stmt.columnText(3);
const copied = @min(name.len, out.last_failure_error_buf.len);
@memcpy(out.last_failure_error_buf[0..copied], name[0..copied]);
out.last_failure_error_len = @intCast(copied);
return out;
}
/// `sum` over `CHECK (… >= 0)` columns cannot go negative; a negative value
/// means the row came from something other than this schema.
fn countOf(value: i64) db.Error!u64 {
if (value < 0) return error.Mismatch;
return @intCast(value);
}
/// Deletes every `upstream_minute` row strictly older than `cutoff_ts`, then
/// the targets no surviving row references, and returns how many **minute**
/// rows went.
///
/// One transaction: a target dropped without its rows, or rows dropped while
/// the target delete failed, would leave the foreign key pointing at nothing.
/// The count is minute rows only, so the metric an operator watches counts
/// aggregates rather than dimension-table housekeeping.
pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!i64 {
var tx = try db.Tx.begin(database);
errdefer tx.rollback();
var minutes = try database.prepare("DELETE FROM upstream_minute WHERE minute_ts < ?1");
defer minutes.deinit();
try minutes.bindInt(1, cutoff_ts);
try minutes.exec();
const deleted = database.changes();
try database.exec(
\\DELETE FROM upstream_targets
\\ WHERE id NOT IN (SELECT upstream_id FROM upstream_minute);
);
try tx.commit();
return deleted;
}
pub fn countMinutes(database: *db.Db) db.Error!i64 {
return database.queryInt("SELECT count(*) FROM upstream_minute");
}
pub fn countTargets(database: *db.Db) db.Error!i64 {
return database.queryInt("SELECT count(*) FROM upstream_targets");
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const querylog_schema = @import("../querylog_schema.zig");
const testing = std.testing;
fn openLog() !db.Db {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
errdefer database.close();
try db.applyPragmas(&database, .{});
try database.exec(querylog_schema.ddl);
return database;
}
/// `columnText` is borrowed until the statement is finalized, so the stored
/// error name is copied out rather than returned as a slice.
const StoredMinute = struct {
successes: u32,
failures: u32,
last_failure_ts: ?i64,
error_buf: [health.error_name_capacity]u8,
error_len: u8,
fn lastError(self: *const StoredMinute) []const u8 {
return self.error_buf[0..self.error_len];
}
};
fn readMinute(database: *db.Db, url: []const u8, minute_ts: i64) !StoredMinute {
var stmt = try database.prepare(
\\SELECT m.successes, m.failures, m.last_failure_ts, m.last_error
\\ FROM upstream_minute m JOIN upstream_targets t ON t.id = m.upstream_id
\\ WHERE t.url = ?1 AND m.minute_ts = ?2
);
defer stmt.deinit();
try stmt.bindText(1, url);
try stmt.bindInt(2, minute_ts);
try testing.expect(try stmt.step());
var out: StoredMinute = .{
.successes = @intCast(stmt.columnInt(0)),
.failures = @intCast(stmt.columnInt(1)),
.last_failure_ts = if (stmt.isNull(2)) null else stmt.columnInt(2),
.error_buf = @splat(0),
.error_len = 0,
};
const name = stmt.columnText(3);
@memcpy(out.error_buf[0..name.len], name);
out.error_len = @intCast(name.len);
return out;
}
test "the upsert adds to the row already there rather than replacing it" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 3, .failures = 1, .last_failure_ts = 70, .last_error = "Timeout" },
});
// The shape a restart inside one minute takes: a second process writes the
// same (url, minute) and the counts continue.
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 2, .failures = 4, .last_failure_ts = 90, .last_error = "ConnectFailed" },
});
const row = try readMinute(&database, "https://a.example", 60);
try testing.expectEqual(@as(u32, 5), row.successes);
try testing.expectEqual(@as(u32, 5), row.failures);
try testing.expectEqual(@as(?i64, 90), row.last_failure_ts);
try testing.expectEqualStrings("ConnectFailed", row.lastError());
// One row and one target, not two of either.
try testing.expectEqual(@as(i64, 1), try countMinutes(&database));
try testing.expectEqual(@as(i64, 1), try countTargets(&database));
}
test "a success-only upsert keeps the failure timestamp and its error" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 0, .failures = 1, .last_failure_ts = 70, .last_error = "Timeout" },
});
// `max()` over a NULL is NULL in SQLite, so without the coalesce this
// upsert would erase the timestamp it knows nothing about.
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 5, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
const row = try readMinute(&database, "https://a.example", 60);
try testing.expectEqual(@as(u32, 5), row.successes);
try testing.expectEqual(@as(u32, 1), row.failures);
try testing.expectEqual(@as(?i64, 70), row.last_failure_ts);
try testing.expectEqualStrings("Timeout", row.lastError());
}
test "an older failure does not overwrite the newer error already stored" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 0, .failures = 1, .last_failure_ts = 90, .last_error = "Timeout" },
});
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 0, .failures = 1, .last_failure_ts = 70, .last_error = "ConnectFailed" },
});
const row = try readMinute(&database, "https://a.example", 60);
try testing.expectEqual(@as(?i64, 90), row.last_failure_ts);
try testing.expectEqualStrings("Timeout", row.lastError());
}
test "flush interns each url once and writes every minute of the batch" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
.{ .url = "https://a.example", .minute_ts = 120, .successes = 2, .failures = 0, .last_failure_ts = null, .last_error = "" },
.{ .url = "https://b.example", .minute_ts = 60, .successes = 3, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
try testing.expectEqual(@as(i64, 3), try countMinutes(&database));
try testing.expectEqual(@as(i64, 2), try countTargets(&database));
// An empty batch opens no transaction: one is already open here, so a
// `BEGIN IMMEDIATE` would fail.
var tx = try db.Tx.begin(&database);
try flush(&database, &.{});
tx.rollback();
}
test "windowStats sums only the window and pairs the newest failure with its own error" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
// Before the window.
.{ .url = "https://a.example", .minute_ts = 0, .successes = 9, .failures = 9, .last_failure_ts = 30, .last_error = "Outside" },
.{ .url = "https://a.example", .minute_ts = 60, .successes = 2, .failures = 1, .last_failure_ts = 100, .last_error = "ConnectFailed" },
.{ .url = "https://a.example", .minute_ts = 120, .successes = 4, .failures = 2, .last_failure_ts = 170, .last_error = "Timeout" },
// A later minute with no failure at all: the error must still come from
// the minute holding the newest `last_failure_ts`, not from this one.
.{ .url = "https://a.example", .minute_ts = 180, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
// At the exclusive end of the window.
.{ .url = "https://a.example", .minute_ts = 240, .successes = 7, .failures = 7, .last_failure_ts = 250, .last_error = "After" },
// A different upstream in the same minutes.
.{ .url = "https://b.example", .minute_ts = 120, .successes = 5, .failures = 5, .last_failure_ts = 175, .last_error = "Other" },
});
const stats = try windowStats(&database, "https://a.example", 60, 240);
try testing.expectEqual(@as(u64, 7), stats.successes);
try testing.expectEqual(@as(u64, 3), stats.failures);
try testing.expectEqual(@as(u64, 10), stats.attempts);
try testing.expectEqual(@as(?i64, 170), stats.last_failure_ts);
try testing.expectEqualStrings("Timeout", stats.lastFailureError());
}
test "two minutes sharing the newest failure timestamp resolve to the later minute" {
// The tiebreak the subquery's ORDER BY owns. `max(last_failure_ts)` alone
// cannot choose between these two rows, so without a deterministic second
// key the answer is whichever row SQLite happened to visit.
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 0, .failures = 1, .last_failure_ts = 100, .last_error = "Earlier" },
.{ .url = "https://a.example", .minute_ts = 120, .successes = 0, .failures = 1, .last_failure_ts = 100, .last_error = "Later" },
});
const stats = try windowStats(&database, "https://a.example", 0, 1000);
try testing.expectEqual(@as(?i64, 100), stats.last_failure_ts);
try testing.expectEqualStrings("Later", stats.lastFailureError());
}
test "windowStats over an unknown url or an empty window is zeros and no error" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 1, .failures = 1, .last_failure_ts = 70, .last_error = "Timeout" },
});
for ([_][3]i64{ .{ 0, 60, 0 }, .{ 120, 180, 0 }, .{ 60, 60, 0 } }) |window| {
const stats = try windowStats(&database, "https://a.example", window[0], window[1]);
try testing.expectEqual(@as(u64, 0), stats.attempts);
try testing.expectEqual(@as(u64, 0), stats.successes);
try testing.expectEqual(@as(u64, 0), stats.failures);
try testing.expectEqual(@as(?i64, null), stats.last_failure_ts);
try testing.expectEqualStrings("", stats.lastFailureError());
}
const unknown = try windowStats(&database, "https://never.example", 0, 1000);
try testing.expectEqual(@as(u64, 0), unknown.attempts);
try testing.expectEqual(@as(?i64, null), unknown.last_failure_ts);
try testing.expectEqualStrings("", unknown.lastFailureError());
}
test "a window whose only failures are outside it reports no failure at all" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 0, .successes = 0, .failures = 1, .last_failure_ts = 30, .last_error = "Timeout" },
.{ .url = "https://a.example", .minute_ts = 60, .successes = 4, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
const stats = try windowStats(&database, "https://a.example", 60, 120);
try testing.expectEqual(@as(u64, 4), stats.attempts);
try testing.expectEqual(@as(?i64, null), stats.last_failure_ts);
try testing.expectEqualStrings("", stats.lastFailureError());
}
test "pruneOlderThan counts minute rows only and drops the orphaned target" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://old.example", .minute_ts = 60, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
.{ .url = "https://old.example", .minute_ts = 120, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
.{ .url = "https://kept.example", .minute_ts = 120, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
.{ .url = "https://kept.example", .minute_ts = 300, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
// Three minute rows go; the two target deletes must not join the count.
try testing.expectEqual(@as(i64, 3), try pruneOlderThan(&database, 300));
try testing.expectEqual(@as(i64, 1), try countMinutes(&database));
// "old.example" has nothing left, "kept.example" still does.
try testing.expectEqual(@as(i64, 1), try countTargets(&database));
// The row exactly at the cutoff stays, and a second pass finds nothing.
try testing.expectEqual(@as(i64, 0), try pruneOlderThan(&database, 300));
try testing.expectEqual(@as(i64, 1), try countMinutes(&database));
}
test "a failed prune leaves both tables as they were" {
var database = try openLog();
defer database.close();
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
try database.exec(
\\CREATE TRIGGER refuse_target_delete BEFORE DELETE ON upstream_targets
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
// The minute delete succeeds and the target delete does not; one
// transaction means neither survives.
try testing.expectError(error.Constraint, pruneOlderThan(&database, 300));
try testing.expectEqual(@as(i64, 1), try countMinutes(&database));
try testing.expectEqual(@as(i64, 1), try countTargets(&database));
}
test "an error name longer than the buffer is truncated, not overflowed" {
var database = try openLog();
defer database.close();
const long = "A" ** 200;
try flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = 60, .successes = 0, .failures = 1, .last_failure_ts = 70, .last_error = long },
});
const stats = try windowStats(&database, "https://a.example", 60, 120);
try testing.expectEqual(@as(usize, health.error_name_capacity), stats.lastFailureError().len);
try testing.expectEqualStrings(long[0..health.error_name_capacity], stats.lastFailureError());
}
+245 -16
View File
@@ -13,8 +13,10 @@ const std = @import("std");
const db = @import("db.zig");
const disk_monitor = @import("disk_monitor.zig");
const events = @import("events.zig");
const model = @import("../config/model.zig");
const queries_repo = @import("repositories/queries_repo.zig");
const upstream_history_repo = @import("repositories/upstream_history_repo.zig");
const log = std.log.scoped(.retention);
@@ -31,6 +33,10 @@ pub const pass_interval_s = 86_400;
pub const Stats = struct {
passes: u64 = 0,
rows_pruned: u64 = 0,
/// Upstream-history minute rows, counted apart from `rows_pruned`: that
/// counter is the query log's, and an operator watching it must not see it
/// move because a different table was tidied.
upstream_rows_pruned: u64 = 0,
checkpoints: u64 = 0,
vacuums: u64 = 0,
/// Vacuums the disk monitor refused. The pass still pruned and
@@ -44,6 +50,7 @@ pub const Stats = struct {
const Counters = struct {
passes: std.atomic.Value(u64) = .init(0),
rows_pruned: std.atomic.Value(u64) = .init(0),
upstream_rows_pruned: std.atomic.Value(u64) = .init(0),
checkpoints: std.atomic.Value(u64) = .init(0),
vacuums: std.atomic.Value(u64) = .init(0),
vacuums_gated: std.atomic.Value(u64) = .init(0),
@@ -67,6 +74,7 @@ pub const Retention = struct {
return .{
.passes = self.counters.passes.load(.monotonic),
.rows_pruned = self.counters.rows_pruned.load(.monotonic),
.upstream_rows_pruned = self.counters.upstream_rows_pruned.load(.monotonic),
.checkpoints = self.counters.checkpoints.load(.monotonic),
.vacuums = self.counters.vacuums.load(.monotonic),
.vacuums_gated = self.counters.vacuums_gated.load(.monotonic),
@@ -95,20 +103,47 @@ pub const Retention = struct {
io: std.Io,
database: *db.Db,
monitor: ?*disk_monitor.Monitor,
store: ?*events.Store,
) void {
add(&self.counters.passes, 1);
const cutoff = std.Io.Clock.real.now(io).toSeconds() - model.retentionSeconds(self.cfg);
const now = std.Io.Clock.real.now(io).toSeconds();
const cutoff = now - model.retentionSeconds(self.cfg);
// Diagnostics retention rides this pass rather than a schedule of its
// own: one daily housekeeping task, and a box restarted every night
// still prunes through `Store.init`.
if (store) |s| s.prune(io, now);
if (queries_repo.pruneOlderThan(database, cutoff)) |deleted| {
add(&self.counters.rows_pruned, @intCast(deleted));
maintenance(store, io, now, "prune", null);
} else |err| {
log.warn("retention prune before {d} failed: {s}", .{ cutoff, @errorName(err) });
maintenance(store, io, now, "prune", @errorName(err));
}
// Before the vacuum-cadence logic below, which returns early on six
// passes out of seven and again whenever the monitor refuses the
// vacuum. A step placed after it would almost never run.
//
// `logging.retention_days` is not the window here: these are per-minute
// aggregates whose whole purpose is to outlive the per-query rows, so
// the window is the repository's own constant.
const history_cutoff = now - upstream_history_repo.retention_window_s;
if (upstream_history_repo.pruneOlderThan(database, history_cutoff)) |deleted| {
add(&self.counters.upstream_rows_pruned, @intCast(deleted));
maintenance(store, io, now, "history_prune", null);
} else |err| {
log.warn("upstream history prune before {d} failed: {s}", .{ history_cutoff, @errorName(err) });
maintenance(store, io, now, "history_prune", @errorName(err));
}
if (queries_repo.checkpointTruncate(database)) {
add(&self.counters.checkpoints, 1);
maintenance(store, io, now, "checkpoint", null);
} else |err| {
log.warn("retention checkpoint failed: {s}", .{@errorName(err)});
maintenance(store, io, now, "checkpoint", @errorName(err));
}
self.passes_since_vacuum += 1;
@@ -119,17 +154,37 @@ pub const Retention = struct {
if (monitor) |m| if (!m.writesAllowed()) {
add(&self.counters.vacuums_gated, 1);
log.warn("retention vacuum skipped: the disk monitor refuses writes", .{});
maintenance(store, io, now, "vacuum", "the disk monitor refuses writes");
return;
};
if (queries_repo.vacuum(database)) {
add(&self.counters.vacuums, 1);
self.passes_since_vacuum = 0;
maintenance(store, io, now, "vacuum", null);
} else |err| {
log.warn("retention vacuum failed: {s}", .{@errorName(err)});
maintenance(store, io, now, "vacuum", @errorName(err));
}
}
/// One step's outcome. `reason` null is the success branch of that same
/// step in that same pass, which is what closes its episode; a gated vacuum
/// is a failure of the step, because the work it owes is still owed.
fn maintenance(
store: ?*events.Store,
io: std.Io,
now_s: i64,
operation: []const u8,
reason: ?[]const u8,
) void {
const s = store orelse return;
const text = reason orelse return s.resolve(io, now_s, .query_log_maintenance, operation);
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "retention {s} failed: {s}", .{ operation, text }) catch buf[0..];
s.report(io, now_s, .query_log_maintenance, operation, operation, .warning, detail);
}
fn add(counter: *std.atomic.Value(u64), delta: u64) void {
_ = counter.fetchAdd(delta, .monotonic);
}
@@ -160,13 +215,14 @@ pub const Retention = struct {
io: std.Io,
database: *db.Db,
monitor: ?*disk_monitor.Monitor,
store: ?*events.Store,
) std.Io.Cancelable!void {
const interval: std.Io.Clock.Duration = .{
.raw = .fromSeconds(pass_interval_s),
.clock = .boot,
};
while (true) {
self.runOnce(io, database, monitor);
self.runOnce(io, database, monitor, store);
try interval.sleep(io);
}
}
@@ -176,6 +232,7 @@ pub const Retention = struct {
// tests
// ---------------------------------------------------------------------------
const events_fixture = @import("events_fixture.zig");
const querylog_schema = @import("querylog_schema.zig");
const testing = std.testing;
@@ -221,7 +278,7 @@ test "a pass prunes the rows past the retention window and keeps the rest" {
try writeRows(&database, &.{ now - 40 * day, now - 31 * day, now - 29 * day, now - 60 });
var retention: Retention = .init(.{ .retention_days = 30 });
retention.runOnce(io, &database, null);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 2), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().passes);
@@ -245,12 +302,12 @@ test "the cutoff follows retention_days" {
try writeRows(&database, &.{now - 3 * day});
var keeps: Retention = .init(.{ .retention_days = 7 });
keeps.runOnce(io, &database, null);
keeps.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 0), keeps.snapshotStats().rows_pruned);
var prunes: Retention = .init(.{ .retention_days = 1 });
prunes.runOnce(io, &database, null);
prunes.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 1), prunes.snapshotStats().rows_pruned);
}
@@ -265,16 +322,16 @@ test "the seventh pass vacuums and the six before it do not" {
var retention: Retention = .init(.{});
for (0..6) |_| {
retention.runOnce(io, &database, null);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(u64, 0), retention.snapshotStats().vacuums);
}
retention.runOnce(io, &database, null);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(u64, 7), retention.snapshotStats().passes);
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().vacuums);
try testing.expectEqual(@as(u64, 7), retention.snapshotStats().checkpoints);
for (0..7) |_| retention.runOnce(io, &database, null);
for (0..7) |_| retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(u64, 14), retention.snapshotStats().passes);
try testing.expectEqual(@as(u64, 2), retention.snapshotStats().vacuums);
}
@@ -293,7 +350,7 @@ test "a gated pass skips the vacuum, counts it, and vacuums on the next pass" {
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
var gated: Retention = .init(.{});
for (0..vacuum_every_passes) |_| gated.runOnce(io, &database, &monitor);
for (0..vacuum_every_passes) |_| gated.runOnce(io, &database, &monitor, null);
// Prune and checkpoint ran on every pass; only the vacuum was refused.
try testing.expectEqual(@as(u64, vacuum_every_passes), gated.snapshotStats().passes);
@@ -303,12 +360,12 @@ test "a gated pass skips the vacuum, counts it, and vacuums on the next pass" {
// The vacuum is due again immediately, not seven passes later.
monitor.state_raw.store(@intFromEnum(disk_monitor.State.ok), .monotonic);
gated.runOnce(io, &database, &monitor);
gated.runOnce(io, &database, &monitor, null);
try testing.expectEqual(@as(u64, 1), gated.snapshotStats().vacuums);
try testing.expectEqual(@as(u64, 1), gated.snapshotStats().vacuums_gated);
// And the counter reset, so the next six passes vacuum nothing.
for (0..vacuum_every_passes - 1) |_| gated.runOnce(io, &database, &monitor);
for (0..vacuum_every_passes - 1) |_| gated.runOnce(io, &database, &monitor, null);
try testing.expectEqual(@as(u64, 1), gated.snapshotStats().vacuums);
}
@@ -324,7 +381,7 @@ test "a warn state still allows the vacuum" {
monitor.state_raw.store(@intFromEnum(disk_monitor.State.warn), .monotonic);
var retention: Retention = .init(.{});
for (0..vacuum_every_passes) |_| retention.runOnce(io, &database, &monitor);
for (0..vacuum_every_passes) |_| retention.runOnce(io, &database, &monitor, null);
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().vacuums);
try testing.expectEqual(@as(u64, 0), retention.snapshotStats().vacuums_gated);
@@ -339,7 +396,7 @@ test "a pass over an empty database still counts" {
defer database.close();
var retention: Retention = .init(.{});
retention.runOnce(io, &database, null);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().passes);
try testing.expectEqual(@as(u64, 0), retention.snapshotStats().rows_pruned);
@@ -363,7 +420,7 @@ test "a failing prune counts the pass and leaves the rows alone" {
);
var retention: Retention = .init(.{ .retention_days = 30 });
retention.runOnce(io, &database, null);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 1), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().passes);
@@ -372,6 +429,83 @@ test "a failing prune counts the pass and leaves the rows alone" {
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().checkpoints);
}
test "the upstream-history window is its own, and a one-day query log does not shrink it" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const now = std.Io.Clock.real.now(io).toSeconds();
const day = 86_400;
try writeRows(&database, &.{ now - 2 * day, now - 60 });
try upstream_history_repo.flush(&database, &.{
.{ .url = "https://gone.example", .minute_ts = now - 32 * day, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
.{ .url = "https://kept.example", .minute_ts = now - 29 * day, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
// A query log kept for one day, and 30 days of upstream minutes beside it.
var retention: Retention = .init(.{ .retention_days = 1 });
retention.runOnce(io, &database, null, null);
const stats = retention.snapshotStats();
// One query-log row is older than one day; one minute row is older than the
// fixed 31-day upstream window. Each counter moved by its own amount.
try testing.expectEqual(@as(u64, 1), stats.rows_pruned);
try testing.expectEqual(@as(u64, 1), stats.upstream_rows_pruned);
try testing.expectEqual(@as(i64, 1), try upstream_history_repo.countMinutes(&database));
// The day-29 row is exactly what a 30-day dashboard window asks for.
const kept = try upstream_history_repo.windowStats(
&database,
"https://kept.example",
now - 30 * day,
now,
);
try testing.expectEqual(@as(u64, 1), kept.attempts);
// And the emptied target went with its rows.
try testing.expectEqual(@as(i64, 1), try upstream_history_repo.countTargets(&database));
}
test "the history prune runs on the passes where the vacuum logic returns early" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const now = std.Io.Clock.real.now(io).toSeconds();
const old_minute = now - 40 * 86_400;
// The first pass: `passes_since_vacuum` is 1, so the vacuum block returns
// before it does anything. A prune placed after that block would never run
// on six passes out of seven.
try upstream_history_repo.flush(&database, &.{
.{ .url = "https://a.example", .minute_ts = old_minute, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
var early: Retention = .init(.{});
early.runOnce(io, &database, null, null);
try testing.expectEqual(@as(u64, 1), early.snapshotStats().upstream_rows_pruned);
try testing.expectEqual(@as(i64, 0), try upstream_history_repo.countMinutes(&database));
// The gated pass: the disk monitor refuses the vacuum and that branch
// returns too, and the prune still has to have happened before it.
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
var gated: Retention = .init(.{});
for (0..vacuum_every_passes - 1) |_| gated.runOnce(io, &database, &monitor, null);
try upstream_history_repo.flush(&database, &.{
.{ .url = "https://b.example", .minute_ts = old_minute, .successes = 1, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
gated.runOnce(io, &database, &monitor, null);
try testing.expectEqual(@as(u64, 1), gated.snapshotStats().vacuums_gated);
try testing.expectEqual(@as(u64, 1), gated.snapshotStats().upstream_rows_pruned);
try testing.expectEqual(@as(i64, 0), try upstream_history_repo.countMinutes(&database));
}
test "the next pass retries what the failed one could not do" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -388,13 +522,108 @@ test "the next pass retries what the failed one could not do" {
);
var retention: Retention = .init(.{ .retention_days = 30 });
retention.runOnce(io, &database, null);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 2), try queries_repo.countRows(&database));
try database.exec("DROP TRIGGER refuse_delete;");
retention.runOnce(io, &database, null);
retention.runOnce(io, &database, null, null);
try testing.expectEqual(@as(i64, 0), try queries_repo.countRows(&database));
try testing.expectEqual(@as(u64, 2), retention.snapshotStats().passes);
try testing.expectEqual(@as(u64, 2), retention.snapshotStats().rows_pruned);
}
test "a failing prune opens a maintenance episode the next clean pass closes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
const now = std.Io.Clock.real.now(io).toSeconds();
try writeRows(&database, &.{now - 40 * 86_400});
try database.exec(
\\CREATE TRIGGER refuse_delete BEFORE DELETE ON query_log
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
var retention: Retention = .init(.{});
retention.runOnce(io, &database, null, &fx.store);
// Only the prune failed; checkpoint and history prune succeeded, and a
// success writes no row of its own.
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqualStrings("query_log.maintenance", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("prune", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("warning", try fx.text("SELECT severity FROM operational_events"));
try database.exec("DROP TRIGGER refuse_delete;");
retention.runOnce(io, &database, null, &fx.store);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a gated vacuum is a maintenance failure the next ungated pass closes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
var monitor: disk_monitor.Monitor = .init(.{}, std.Io.Dir.cwd(), ".", null);
monitor.state_raw.store(@intFromEnum(disk_monitor.State.critical), .monotonic);
var retention: Retention = .init(.{});
for (0..vacuum_every_passes) |_| retention.runOnce(io, &database, &monitor, &fx.store);
try testing.expectEqualStrings("vacuum", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
));
monitor.state_raw.store(@intFromEnum(disk_monitor.State.ok), .monotonic);
retention.runOnce(io, &database, &monitor, &fx.store);
try testing.expectEqual(@as(u64, 1), retention.snapshotStats().vacuums);
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a pass prunes the diagnostics store once" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
// Resolved further back than the retention window, so the pass must drop it.
const now = std.Io.Clock.real.now(io).toSeconds();
const stale = now - events.Store.resolved_retention_s - 86_400;
fx.store.reportResolved(io, stale, .query_log_recreated, "one-shot", "one-shot", .warning, "aside kept");
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
var retention: Retention = .init(.{});
retention.runOnce(io, &database, null, &fx.store);
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
}
+11 -1
View File
@@ -190,7 +190,7 @@ fn countLines(text: []const u8) usize {
// querylog aside files
// ---------------------------------------------------------------------------
const aside_prefix = "querylog.db.corrupt-";
const aside_prefix = "querylog.db.";
const Names = struct {
items: std.ArrayList([]u8),
@@ -412,6 +412,10 @@ test "S7 case 3: a wrong user_version recreates and keeps the old file aside" {
defer asides.deinit();
try testing.expectEqual(@as(usize, 1), asides.items.items.len);
// The file was healthy: this build's schema moved, the database did not rot.
// An operator who reads "corrupt" here deletes a file that was never broken.
try testing.expect(std.mem.startsWith(u8, asides.items.items[0], "querylog.db.schema-changed-"));
const kept = try f.read(asides.items.items[0]);
defer testing.allocator.free(kept);
try testing.expectEqualSlices(u8, original, kept);
@@ -441,6 +445,12 @@ test "S7 case 4: a garbage file recreates and the garbage is preserved" {
defer asides.deinit();
try testing.expectEqual(@as(usize, 1), asides.items.items.len);
const expected: []const u8 = if (reason == .corrupt)
"querylog.db.corrupt-"
else
"querylog.db.not-a-database-";
try testing.expect(std.mem.startsWith(u8, asides.items.items[0], expected));
const kept = try f.read(asides.items.items[0]);
defer testing.allocator.free(kept);
try testing.expectEqualSlices(u8, &garbage, kept);
+6
View File
@@ -75,6 +75,11 @@ comptime {
_ = @import("cache/dns_cache.zig");
_ = @import("server/rate_limiter.zig");
_ = @import("storage/repositories/queries_repo.zig");
_ = @import("storage/repositories/upstream_history_repo.zig");
_ = @import("storage/repositories/events_repo.zig");
_ = @import("storage/events.zig");
_ = @import("storage/events_fixture.zig");
_ = @import("upstream/history.zig");
_ = @import("storage/logger.zig");
_ = @import("platform/statfs.zig");
_ = @import("storage/disk_monitor.zig");
@@ -98,6 +103,7 @@ comptime {
_ = @import("web/metrics.zig");
_ = @import("web/handlers/stats.zig");
_ = @import("web/handlers/queries.zig");
_ = @import("web/handlers/diagnostics.zig");
_ = @import("web/handlers/lookup.zig");
_ = @import("web/handlers/upstream_health.zig");
_ = @import("web/handlers/health.zig");
+7 -1
View File
@@ -45,6 +45,12 @@ pub const Config = struct {
/// `State.window`.
pub const window_len = 32;
/// Bytes kept of an `@errorName`, truncated to fit. Shared rather than repeated:
/// `history.Accumulator.Cell` and `upstream_history_repo.WindowStats` carry the
/// same name through the minute aggregates, and three buffers of three different
/// sizes would truncate one error name three ways.
pub const error_name_capacity = 48;
/// The shift is capped so `base_backoff_ms << shift` cannot run away; by then
/// `max_backoff_ms` has clamped the result many doublings ago.
const max_shift = 20;
@@ -55,7 +61,7 @@ pub const State = struct {
total_failures: u64,
last_success_at: ?std.Io.Timestamp,
last_error_at: ?std.Io.Timestamp,
last_error_buf: [48]u8,
last_error_buf: [error_name_capacity]u8,
/// Length of the `@errorName` held in `last_error_buf`, truncated to fit.
last_error_len: u8,
backoff_until: ?std.Io.Timestamp,
+716
View File
@@ -0,0 +1,716 @@
//! Per-minute upstream outcome history (milestone-26 rulings 1, 3, 4).
//!
//! Outcomes are aggregated into their wall-clock UTC minute at the moment the
//! pool records them, and the aggregates are flushed to `querylog.db` once a
//! minute. **Nothing samples a lifetime counter and subtracts.** That is what
//! makes the stored numbers additive facts: a restart inside a minute adds to
//! the same row, a crash loses at most the cells that had not flushed yet, and
//! no path anywhere can produce a negative delta.
//!
//! The query path may not touch SQLite, so recording is memory-only under this
//! module's own mutex and the writing happens on a task of its own.
//!
//! Two failure modes, deliberately kept apart:
//!
//! * **Overflow.** More live `(url, minute)` pairs than `max_pending`. The
//! oldest minute is dropped, `rows_dropped` counts it and
//! `last_drop_minute` remembers how new the newest lost minute was, so a
//! window that starts after it can still be reported as complete.
//! * **A failed flush.** Nothing is dropped: the rows go back into the
//! accumulator and the next pass writes them again.
//!
//! The wall clock, not `.awake`: history participates in wall-clock periods, so
//! a minute here is the same minute the dashboard's period picker means.
//! Routing state (`health.zig`) stays on `.awake` and is untouched by this
//! file.
const std = @import("std");
const db = @import("../storage/db.zig");
const events = @import("../storage/events.zig");
const health = @import("health.zig");
const upstream_history_repo = @import("../storage/repositories/upstream_history_repo.zig");
const log = std.log.scoped(.upstream_history);
/// Live `(url, minute)` cells. Not a knob (m26 anti-requirements). A household
/// pool is a handful of upstreams, so the bound is only reachable when flushing
/// has been failing for hours.
pub const max_pending = 4096;
/// One flush pass per minute, so an unflushed cell is at most about a minute
/// old. `.boot` rather than `.awake`, for `retention.zig`'s reason: a box that
/// suspends must still see its interval elapse.
pub const flush_interval_s = 60;
/// The UTC minute `wall_s` falls in, as its start in seconds. `@divFloor`, not
/// `@divTrunc`: a negative second belongs to the minute before it.
pub fn minuteOf(wall_s: i64) i64 {
return @divFloor(wall_s, 60) * 60;
}
/// The write seam. Production passes `upstream_history_repo.flush`; a test
/// passes a stub that fails, or one that records what it was handed.
pub const WriteFn = *const fn (*db.Db, []const upstream_history_repo.FlushRow) db.Error!void;
pub const Accumulator = struct {
pub const Cell = struct {
/// Borrowed from the pool entry's endpoint, which lives as long as the
/// process. Nothing here copies it, and nothing here may outlive it.
url: []const u8,
minute_ts: i64,
/// Both counters saturate instead of wrapping: every write site uses
/// `+|=` — `recordSuccess`, `recordFailure`, and `mergeBack`, which
/// sums a failed flush's copy back into the live cell. The saturation
/// is deliberate and unreachable: a cell counts one upstream's
/// outcomes inside a single wall-clock minute, so filling a `u32`
/// would take about 72 million exchanges per second with that one
/// upstream. Nothing reports it, by design — `last_drop_minute` and
/// the `complete` flag it feeds describe capacity drops, and a
/// saturated counter is not a drop.
successes: u32,
failures: u32,
last_failure_ts: ?i64,
last_error_buf: [health.error_name_capacity]u8,
last_error_len: u8,
fn lastError(self: *const Cell) []const u8 {
return self.last_error_buf[0..self.last_error_len];
}
};
/// A consistent copy for `/metrics`, `/api/health` and the API layer.
pub const Stats = struct {
flushes: u64 = 0,
flush_failures: u64 = 0,
rows_dropped: u64 = 0,
pending: u32 = 0,
/// The newest minute capacity has ever cost this process, or null when
/// nothing was ever dropped. A window that starts after it is complete
/// again, so one historical overflow does not mark every later answer.
last_drop_minute: ?i64 = null,
/// Current state, not a count: set by a failed flush and cleared by the
/// next successful one. Feeds the `/api/health` rollup.
last_flush_failed: bool = false,
};
/// Atomic for the reason `retention.zig`'s are: the flush task writes them
/// and the web task reads them, on different threads. They are bumped
/// outside the mutex, so the lock is not what orders them.
const Counters = struct {
flushes: std.atomic.Value(u64) = .init(0),
flush_failures: std.atomic.Value(u64) = .init(0),
rows_dropped: std.atomic.Value(u64) = .init(0),
};
mutex: std.Io.Mutex = .init,
cells: [max_pending]Cell,
count: u32,
last_drop_minute: ?i64,
last_flush_failed: bool,
counters: Counters,
/// Owned by whichever task runs `flushOnce`, which is one task. It is a
/// field rather than a local so that the megabyte it costs lives wherever
/// the accumulator was placed instead of on a task's stack.
flush_cells: [max_pending]Cell,
flush_rows: [max_pending]upstream_history_repo.FlushRow,
flush_count: u32,
/// Set once by the composition root after `init`, following the
/// `gate: ?*disk_monitor.Monitor` idiom. Null everywhere else, and every
/// emit site below is inert when it is.
diagnostics: ?*events.Store = null,
/// `cells` and `flush_cells` are `undefined`: a cell is always written
/// before it is read, and `count` is what says which ones exist.
pub const init: Accumulator = .{
.mutex = .init,
.cells = undefined,
.count = 0,
.last_drop_minute = null,
.last_flush_failed = false,
.counters = .{},
.flush_cells = undefined,
.flush_rows = undefined,
.flush_count = 0,
};
pub fn recordSuccess(self: *Accumulator, io: std.Io, url: []const u8, wall_s: i64) void {
// Uncancelable for the reason the pool's health sections are: this
// takes no Io and never blocks on a peer, and losing the record of a
// completed exchange to a cancellation would undercount for good.
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
const cell = self.cellFor(url, minuteOf(wall_s));
cell.successes +|= 1;
}
pub fn recordFailure(
self: *Accumulator,
io: std.Io,
url: []const u8,
wall_s: i64,
error_name: []const u8,
) void {
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
const cell = self.cellFor(url, minuteOf(wall_s));
cell.failures +|= 1;
noteFailure(cell, wall_s, error_name);
}
/// The only read surface. Every field above is private to this module, so
/// no consumer can read one of them without the mutex.
pub fn snapshotStats(self: *Accumulator, io: std.Io) Stats {
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
return .{
.flushes = self.counters.flushes.load(.monotonic),
.flush_failures = self.counters.flush_failures.load(.monotonic),
.rows_dropped = self.counters.rows_dropped.load(.monotonic),
.pending = self.count,
.last_drop_minute = self.last_drop_minute,
.last_flush_failed = self.last_flush_failed,
};
}
/// The cell for `(url, minute_ts)`, created if it is not there yet. The
/// caller holds the mutex.
///
/// Linear: the live set is one cell per upstream per unflushed minute,
/// which at household scale is a handful. A miss at capacity evicts the
/// oldest minute — see `evictOldest`.
fn cellFor(self: *Accumulator, url: []const u8, minute_ts: i64) *Cell {
for (self.cells[0..self.count]) |*cell| {
if (cell.minute_ts != minute_ts) continue;
// Pointer equality first: every call from the pool passes the same
// `Entry.endpoint.url`, so the byte compare is the cold path.
if (cell.url.ptr == url.ptr and cell.url.len == url.len) return cell;
if (std.mem.eql(u8, cell.url, url)) return cell;
}
const slot = if (self.count < max_pending) fresh: {
const index = self.count;
self.count += 1;
break :fresh &self.cells[index];
} else self.evictOldest();
slot.* = .{
.url = url,
.minute_ts = minute_ts,
.successes = 0,
.failures = 0,
.last_failure_ts = null,
.last_error_buf = @splat(0),
.last_error_len = 0,
};
return slot;
}
/// Frees the cell holding the oldest minute and accounts for what it cost.
///
/// `last_drop_minute` moves through `@max` and never through assignment: a
/// merge-back after a failed flush can evict a cell older than one already
/// dropped, and a watermark that moved backwards would report a window as
/// complete when outcomes inside it are gone.
fn evictOldest(self: *Accumulator) *Cell {
var oldest: usize = 0;
for (self.cells[1..self.count], 1..) |*cell, i| {
if (cell.minute_ts < self.cells[oldest].minute_ts) oldest = i;
}
const evicted = self.cells[oldest].minute_ts;
self.last_drop_minute = @max(self.last_drop_minute orelse evicted, evicted);
_ = self.counters.rows_dropped.fetchAdd(1, .monotonic);
return &self.cells[oldest];
}
/// One flush pass: swap the dirty cells out, write them, and on failure put
/// them back.
///
/// **A swap, never a subtraction.** SQLite runs outside the mutex, so while
/// it does, a full accumulator can evict a cell that was copied out and
/// then recreate the same `(url, minute_ts)`. A post-flush subtract would
/// then destroy outcomes recorded during the write. Moving the cells out
/// makes the flush own them: what is recorded beside it is new data, and a
/// failure merges the two additively.
///
/// Every failure is counted and warned about once; nothing here returns an
/// error, because there is no caller that could do anything the next pass
/// will not do anyway.
pub fn flushOnce(self: *Accumulator, io: std.Io, database: *db.Db, write: WriteFn) void {
{
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
if (self.count == 0) return;
@memcpy(self.flush_cells[0..self.count], self.cells[0..self.count]);
self.flush_count = self.count;
self.count = 0;
}
const rows = self.flush_rows[0..self.flush_count];
for (self.flush_cells[0..self.flush_count], rows) |*cell, *row| {
row.* = .{
.url = cell.url,
.minute_ts = cell.minute_ts,
.successes = cell.successes,
.failures = cell.failures,
.last_failure_ts = cell.last_failure_ts,
.last_error = cell.lastError(),
};
}
if (write(database, rows)) {
_ = self.counters.flushes.fetchAdd(1, .monotonic);
if (self.diagnostics) |store| {
store.resolve(io, std.Io.Clock.real.now(io).toSeconds(), .upstream_history_write, flush_key);
}
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
self.last_flush_failed = false;
return;
} else |err| {
_ = self.counters.flush_failures.fetchAdd(1, .monotonic);
log.warn("flushing {d} upstream history rows failed: {s}", .{ rows.len, @errorName(err) });
if (self.diagnostics) |store| {
var buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&buf, "flushing {d} upstream history rows failed: {s}", .{
rows.len,
@errorName(err),
}) catch buf[0..];
store.report(
io,
std.Io.Clock.real.now(io).toSeconds(),
.upstream_history_write,
flush_key,
"upstream history flush",
.warning,
detail,
);
}
self.mergeBack(io);
}
}
/// Puts a failed pass's cells back through the rules recording uses: a cell
/// recorded during the write keeps its outcomes and the merge sums into it,
/// and a merge that overflows follows the ordinary drop policy.
fn mergeBack(self: *Accumulator, io: std.Io) void {
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
self.last_flush_failed = true;
for (self.flush_cells[0..self.flush_count]) |*saved| {
const cell = self.cellFor(saved.url, saved.minute_ts);
cell.successes +|= saved.successes;
cell.failures +|= saved.failures;
if (saved.last_failure_ts) |at| noteFailure(cell, at, saved.lastError());
}
}
/// Daily-loop shape (`retention.zig`): flush first, then sleep, so a
/// process that is about to be canceled has already written once.
pub fn run(self: *Accumulator, io: std.Io, database: *db.Db) std.Io.Cancelable!void {
const interval: std.Io.Clock.Duration = .{
.raw = .fromSeconds(flush_interval_s),
.clock = .boot,
};
while (true) {
self.flushOnce(io, database, upstream_history_repo.flush);
try interval.sleep(io);
}
}
};
/// One accumulator, one flush task, one table: the subject of every
/// `upstream_history.write` episode is that single writer.
const flush_key = "flush";
/// Max-wins, matching the SQL upsert exactly: the newest failure in the minute
/// is the one whose name the cell keeps.
fn noteFailure(cell: *Accumulator.Cell, at: i64, error_name: []const u8) void {
if (cell.last_failure_ts) |existing| {
if (at < existing) return;
}
cell.last_failure_ts = at;
const copied = @min(error_name.len, cell.last_error_buf.len);
@memcpy(cell.last_error_buf[0..copied], error_name[0..copied]);
cell.last_error_len = @intCast(copied);
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const events_fixture = @import("../storage/events_fixture.zig");
const querylog_schema = @import("../storage/querylog_schema.zig");
const testing = std.testing;
fn openLog() !db.Db {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
errdefer database.close();
try db.applyPragmas(&database, .{});
try database.exec(querylog_schema.ddl);
return database;
}
/// The accumulator is about a megabyte, which is more than a test frame should
/// carry.
fn newAccumulator() !*Accumulator {
const acc = try testing.allocator.create(Accumulator);
acc.* = .init;
return acc;
}
fn failingWrite(_: *db.Db, _: []const upstream_history_repo.FlushRow) db.Error!void {
return error.Busy;
}
/// What the last stubbed flush was handed, copied out so an assertion can read
/// it after the pass returned.
var recorded: [8]upstream_history_repo.FlushRow = undefined;
var recorded_len: usize = 0;
fn recordingWrite(_: *db.Db, rows: []const upstream_history_repo.FlushRow) db.Error!void {
recorded_len = @min(rows.len, recorded.len);
@memcpy(recorded[0..recorded_len], rows[0..recorded_len]);
}
/// The interleaving of ruling 4: a flush is in flight, and the recording side
/// recreates a swapped-out cell and then fills the accumulator to overflow.
var interleaved: ?*Accumulator = null;
var interleave_io: ?std.Io = null;
var interleave_urls: [max_pending][8]u8 = undefined;
fn interleavingWrite(_: *db.Db, _: []const upstream_history_repo.FlushRow) db.Error!void {
const acc = interleaved.?;
const io = interleave_io.?;
// The very `(url, minute_ts)` the flush is holding, recorded again while
// the write runs.
acc.recordSuccess(io, "https://a.example", 60);
// And then enough distinct minutes to fill the accumulator and evict.
for (&interleave_urls, 0..) |*name, i| {
const url = std.fmt.bufPrint(name, "u{d:0>6}", .{i}) catch unreachable;
acc.recordSuccess(io, url, @as(i64, @intCast(i)) * 600 + 6000);
}
return error.Busy;
}
test "minuteOf floors to the minute, including before the epoch" {
try testing.expectEqual(@as(i64, 0), minuteOf(0));
try testing.expectEqual(@as(i64, 0), minuteOf(59));
try testing.expectEqual(@as(i64, 60), minuteOf(60));
try testing.expectEqual(@as(i64, 120), minuteOf(179));
try testing.expectEqual(@as(i64, -60), minuteOf(-1));
}
test "outcomes land in the cell of their own minute and upstream" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.recordSuccess(io, "https://a.example", 65);
acc.recordSuccess(io, "https://a.example", 119);
acc.recordFailure(io, "https://a.example", 100, "Timeout");
acc.recordSuccess(io, "https://a.example", 130);
acc.recordSuccess(io, "https://b.example", 70);
// Three cells: a/60, a/120 and b/60.
try testing.expectEqual(@as(u32, 3), acc.snapshotStats(io).pending);
const first = acc.cellFor("https://a.example", 60);
try testing.expectEqual(@as(u32, 2), first.successes);
try testing.expectEqual(@as(u32, 1), first.failures);
try testing.expectEqual(@as(?i64, 100), first.last_failure_ts);
try testing.expectEqualStrings("Timeout", first.lastError());
const second = acc.cellFor("https://a.example", 120);
try testing.expectEqual(@as(u32, 1), second.successes);
try testing.expectEqual(@as(u32, 0), second.failures);
try testing.expectEqual(@as(?i64, null), second.last_failure_ts);
}
test "a cell keeps the newest failure's error and ignores an older one" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.recordFailure(io, "https://a.example", 100, "Timeout");
acc.recordFailure(io, "https://a.example", 80, "ConnectFailed");
const cell = acc.cellFor("https://a.example", 60);
try testing.expectEqual(@as(u32, 2), cell.failures);
try testing.expectEqual(@as(?i64, 100), cell.last_failure_ts);
try testing.expectEqualStrings("Timeout", cell.lastError());
acc.recordFailure(io, "https://a.example", 110, "BadResponse");
try testing.expectEqual(@as(?i64, 110), cell.last_failure_ts);
try testing.expectEqualStrings("BadResponse", cell.lastError());
const long = "A" ** 200;
acc.recordFailure(io, "https://a.example", 115, long);
try testing.expectEqual(@as(usize, health.error_name_capacity), cell.lastError().len);
}
test "at capacity the oldest minute is dropped, counted, and the watermark only moves forward" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
var names: [max_pending][8]u8 = undefined;
for (&names, 0..) |*name, i| {
const url = std.fmt.bufPrint(name, "u{d:0>6}", .{i}) catch unreachable;
// Minute 600 is the oldest; every later cell is newer.
acc.recordSuccess(io, url, 600 + @as(i64, @intCast(i)) * 60);
}
try testing.expectEqual(@as(u32, max_pending), acc.snapshotStats(io).pending);
try testing.expectEqual(@as(u64, 0), acc.snapshotStats(io).rows_dropped);
try testing.expectEqual(@as(?i64, null), acc.snapshotStats(io).last_drop_minute);
// One more cell evicts the oldest minute and nothing else.
acc.recordSuccess(io, "https://new.example", 10_000_000);
const after = acc.snapshotStats(io);
try testing.expectEqual(@as(u32, max_pending), after.pending);
try testing.expectEqual(@as(u64, 1), after.rows_dropped);
try testing.expectEqual(@as(?i64, 600), after.last_drop_minute);
// A later eviction of an *older* minute must not move the watermark back.
acc.recordSuccess(io, "https://older.example", 120);
const back = acc.snapshotStats(io);
try testing.expectEqual(@as(u64, 2), back.rows_dropped);
try testing.expectEqual(@as(?i64, 660), back.last_drop_minute);
acc.recordSuccess(io, "https://newer.example", 20_000_000);
const forward = acc.snapshotStats(io);
try testing.expectEqual(@as(u64, 3), forward.rows_dropped);
// The minute just evicted is 120, older than the 660 already recorded.
try testing.expectEqual(@as(?i64, 660), forward.last_drop_minute);
}
test "a successful flush hands over every cell, empties the accumulator and counts once" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.recordSuccess(io, "https://a.example", 65);
acc.recordFailure(io, "https://a.example", 100, "Timeout");
acc.flushOnce(io, &database, recordingWrite);
try testing.expectEqual(@as(usize, 1), recorded_len);
try testing.expectEqualStrings("https://a.example", recorded[0].url);
try testing.expectEqual(@as(i64, 60), recorded[0].minute_ts);
try testing.expectEqual(@as(u32, 1), recorded[0].successes);
try testing.expectEqual(@as(u32, 1), recorded[0].failures);
try testing.expectEqual(@as(?i64, 100), recorded[0].last_failure_ts);
try testing.expectEqualStrings("Timeout", recorded[0].last_error);
const stats = acc.snapshotStats(io);
try testing.expectEqual(@as(u32, 0), stats.pending);
try testing.expectEqual(@as(u64, 1), stats.flushes);
try testing.expectEqual(@as(u64, 0), stats.flush_failures);
try testing.expect(!stats.last_flush_failed);
}
test "a pass with nothing pending counts neither a flush nor a failure" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.flushOnce(io, &database, failingWrite);
acc.flushOnce(io, &database, recordingWrite);
const stats = acc.snapshotStats(io);
try testing.expectEqual(@as(u64, 0), stats.flushes);
try testing.expectEqual(@as(u64, 0), stats.flush_failures);
try testing.expect(!stats.last_flush_failed);
}
test "a failed flush keeps the rows, sets the flag, and the next success clears it" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.recordSuccess(io, "https://a.example", 65);
acc.recordFailure(io, "https://a.example", 100, "Timeout");
acc.flushOnce(io, &database, failingWrite);
const failed = acc.snapshotStats(io);
try testing.expectEqual(@as(u64, 0), failed.flushes);
try testing.expectEqual(@as(u64, 1), failed.flush_failures);
try testing.expectEqual(@as(u64, 0), failed.rows_dropped);
try testing.expect(failed.last_flush_failed);
// Nothing was lost: the cell is back, whole.
try testing.expectEqual(@as(u32, 1), failed.pending);
const cell = acc.cellFor("https://a.example", 60);
try testing.expectEqual(@as(u32, 1), cell.successes);
try testing.expectEqual(@as(u32, 1), cell.failures);
try testing.expectEqualStrings("Timeout", cell.lastError());
// The retry writes what the failed pass could not.
acc.recordSuccess(io, "https://a.example", 70);
acc.flushOnce(io, &database, recordingWrite);
const cleared = acc.snapshotStats(io);
try testing.expectEqual(@as(u64, 1), cleared.flushes);
try testing.expectEqual(@as(u64, 1), cleared.flush_failures);
try testing.expect(!cleared.last_flush_failed);
try testing.expectEqual(@as(usize, 1), recorded_len);
try testing.expectEqual(@as(u32, 2), recorded[0].successes);
try testing.expectEqual(@as(u32, 1), recorded[0].failures);
}
test "a merge-back sums into what was recorded beside the flush" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
// The write recreates the swapped-out cell before it fails, which is the
// interleaving the swap makes possible.
interleaved = acc;
interleave_io = io;
defer {
interleaved = null;
interleave_io = null;
}
acc.recordSuccess(io, "https://a.example", 65);
acc.recordFailure(io, "https://a.example", 100, "Timeout");
acc.flushOnce(io, &database, interleavingWrite);
const stats = acc.snapshotStats(io);
try testing.expect(stats.last_flush_failed);
try testing.expectEqual(@as(u64, 1), stats.flush_failures);
// Two drops, and they are different drops: filling to capacity during the
// write evicted the recreated minute 60, and the merge-back then found no
// room either and evicted the oldest of what the write had left, 6000.
// Overflow loss is the specced policy; what matters is that it is counted.
try testing.expectEqual(@as(u32, max_pending), stats.pending);
try testing.expectEqual(@as(u64, 2), stats.rows_dropped);
// Forward only: the second eviction was the newer minute of the two.
try testing.expectEqual(@as(?i64, 6000), stats.last_drop_minute);
// The merged cell is back, carrying what the failed flush was holding.
const merged = acc.cellFor("https://a.example", 60);
try testing.expectEqual(@as(u32, 1), merged.successes);
try testing.expectEqual(@as(u32, 1), merged.failures);
try testing.expectEqualStrings("Timeout", merged.lastError());
// The flush-owned buffer was not touched by any of the recording that
// happened beside it: it still holds exactly what was swapped out.
try testing.expectEqual(@as(u32, 1), acc.flush_count);
try testing.expectEqual(@as(u32, 1), acc.flush_cells[0].successes);
try testing.expectEqual(@as(u32, 1), acc.flush_cells[0].failures);
try testing.expectEqual(@as(i64, 60), acc.flush_cells[0].minute_ts);
}
test "a flush against the real repository writes the minute rows" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.recordSuccess(io, "https://a.example", 65);
acc.recordFailure(io, "https://a.example", 100, "Timeout");
acc.recordSuccess(io, "https://b.example", 130);
acc.flushOnce(io, &database, upstream_history_repo.flush);
// The second pass is the restart shape: the same minute, added to.
acc.recordSuccess(io, "https://a.example", 90);
acc.flushOnce(io, &database, upstream_history_repo.flush);
const stats = try upstream_history_repo.windowStats(&database, "https://a.example", 0, 200);
try testing.expectEqual(@as(u64, 2), stats.successes);
try testing.expectEqual(@as(u64, 1), stats.failures);
try testing.expectEqual(@as(u64, 3), stats.attempts);
try testing.expectEqual(@as(?i64, 100), stats.last_failure_ts);
try testing.expectEqualStrings("Timeout", stats.lastFailureError());
try testing.expectEqual(@as(i64, 2), try upstream_history_repo.countMinutes(&database));
try testing.expectEqual(@as(u64, 2), acc.snapshotStats(io).flushes);
}
test "a failed flush opens one episode and the next successful flush closes it" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.diagnostics = &fx.store;
acc.recordFailure(io, "https://a.example", 1_700_000_000, "Timeout");
acc.flushOnce(io, &database, failingWrite);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqualStrings("upstream_history.write", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("flush", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("warning", try fx.text("SELECT severity FROM operational_events"));
try testing.expectEqual(
@as(i64, 1),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
acc.flushOnce(io, &database, upstream_history_repo.flush);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
}
test "a flush with no store attached records nothing and still flushes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const acc = try newAccumulator();
defer testing.allocator.destroy(acc);
acc.recordFailure(io, "https://a.example", 1_700_000_000, "Timeout");
acc.flushOnce(io, &database, failingWrite);
acc.flushOnce(io, &database, upstream_history_repo.flush);
try testing.expectEqual(@as(u64, 1), acc.counters.flushes.load(.monotonic));
}
+203 -9
View File
@@ -47,7 +47,9 @@
const std = @import("std");
const events = @import("../storage/events.zig");
const health = @import("health.zig");
const history_mod = @import("history.zig");
const safe_url = @import("../safe_url.zig");
const transport = @import("transport.zig");
@@ -129,6 +131,15 @@ pub const Pool = struct {
timeouts: Timeouts,
mutex: std.Io.Mutex,
rng: std.Random.DefaultPrng,
/// Where recorded outcomes also go, as per-minute aggregates for the
/// dashboard's ranged view (m26). Defaulted rather than an `init`
/// parameter: the composition root wires it after the pool exists, and the
/// pool is fully usable without it — `nxdns check` and every unit test here
/// run with no history at all.
history: ?*history_mod.Accumulator = null,
/// The diagnostics store, wired the same way and for the same reason as
/// `history`. Every emit here sits outside `mutex`; see `recordHistory`.
diagnostics: ?*events.Store = null,
pub fn init(
entries: []Entry,
@@ -307,12 +318,20 @@ pub const Pool = struct {
}
fn recordSuccess(self: *Pool, io: std.Io, entry: *Entry, at: std.Io.Timestamp) void {
// Uncancelable: this section takes no Io and never blocks on a peer.
// Losing the bookkeeping for a completed exchange to a cancellation
// that arrives one instruction later would corrupt health for good.
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
entry.health.recordSuccess(at);
{
// Uncancelable: this section takes no Io and never blocks on a
// peer. Losing the bookkeeping for a completed exchange to a
// cancellation that arrives one instruction later would corrupt
// health for good.
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
entry.health.recordSuccess(at);
}
// The block above closes before this line, and that ordering is the
// constraint: the accumulator takes a mutex of its own, and no task may
// hold one of the two while it takes the other.
self.recordHistory(io, entry, .success);
self.recordDiagnostics(io, entry, .success);
}
fn recordFailure(
@@ -322,12 +341,61 @@ pub const Pool = struct {
at: std.Io.Timestamp,
err: transport.ExchangeError,
) void {
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
entry.health.recordFailure(at, @errorName(err), self.cfg, self.rng.random().int(u32));
{
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
entry.health.recordFailure(at, @errorName(err), self.cfg, self.rng.random().int(u32));
}
// After the pool mutex is released, for the reason `recordSuccess`
// states.
self.recordHistory(io, entry, .{ .failure = @errorName(err) });
self.recordDiagnostics(io, entry, .{ .failure = @errorName(err) });
}
const Outcome = union(enum) { success, failure: []const u8 };
/// The wall clock, not the `.awake` timestamp the health state runs on:
/// history is aggregated into wall-clock minutes so a dashboard period
/// means the same thing here as everywhere else on the page.
fn recordHistory(self: *Pool, io: std.Io, entry: *Entry, outcome: Outcome) void {
const history = self.history orelse return;
const wall_s = std.Io.Clock.real.now(io).toSeconds();
switch (outcome) {
.success => history.recordSuccess(io, entry.endpoint.url, wall_s),
.failure => |name| history.recordFailure(io, entry.endpoint.url, wall_s, name),
}
}
/// The same placement discipline as `recordHistory`: the store takes a
/// mutex of its own, so this runs after the pool's is released.
///
/// A success is the steady state of the whole program, so `resolve` is
/// built to issue no SQL when nothing is open (`storage/events.zig`).
fn recordDiagnostics(self: *Pool, io: std.Io, entry: *Entry, outcome: Outcome) void {
const store = self.diagnostics orelse return;
const url = entry.endpoint.url;
const now_s = std.Io.Clock.real.now(io).toSeconds();
switch (outcome) {
.success => store.resolve(io, now_s, .upstream_exchange, url),
.failure => |name| {
var label_buf: [events.Store.max_subject_label_len]u8 = undefined;
const label = std.fmt.bufPrint(&label_buf, "{f}", .{safe_url.redact(url)}) catch &label_buf;
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&detail_buf, "upstream {f} failed: {s}", .{
safe_url.redactQuoted(url),
name,
}) catch &detail_buf;
store.report(io, now_s, .upstream_exchange, url, label, .warning, detail);
},
}
}
};
const db = @import("../storage/db.zig");
const events_fixture = @import("../storage/events_fixture.zig");
const querylog_schema = @import("../storage/querylog_schema.zig");
const upstream_history_repo = @import("../storage/repositories/upstream_history_repo.zig");
const testing = std.testing;
/// A query for example.com A: id 0x1234, RD set, one question.
@@ -710,6 +778,66 @@ test "every entry disabled yields ConnectFailed without waiting out the total bu
try testing.expectEqual(@as(usize, 0), two.calls);
}
test "a wired accumulator receives both outcomes the pool records" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var bad: Fake = .{ .behavior = .{ .fail = error.Timeout } };
var good: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{
testEntry("https://bad.example/dns-query", &bad, 10),
testEntry("https://good.example/dns-query", &good, 20),
};
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
const acc = try testing.allocator.create(history_mod.Accumulator);
defer testing.allocator.destroy(acc);
acc.* = .init;
pool.history = acc;
var buf: [512]u8 = undefined;
// One exchange: the first entry fails over into the second, so this drives
// one failure and one success.
_ = try pool.exchange(io, query_bytes, &buf);
// Two cells, one per url, in whatever minute the wall clock is in.
try testing.expectEqual(@as(u32, 2), acc.snapshotStats(io).pending);
// Read back through the flush path rather than through the accumulator's
// private cells: the whole point of the hook is that these outcomes reach
// storage under the right url.
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try db.applyPragmas(&database, .{});
try database.exec(querylog_schema.ddl);
acc.flushOnce(io, &database, upstream_history_repo.flush);
// A window wide enough that a minute boundary crossed mid-test changes
// nothing about what it contains.
const now = std.Io.Clock.real.now(io).toSeconds();
const failing = try upstream_history_repo.windowStats(
&database,
"https://bad.example/dns-query",
now - 3600,
now + 3600,
);
try testing.expectEqual(@as(u64, 1), failing.failures);
try testing.expectEqual(@as(u64, 0), failing.successes);
try testing.expect(failing.last_failure_ts != null);
try testing.expectEqualStrings("Timeout", failing.lastFailureError());
const succeeding = try upstream_history_repo.windowStats(
&database,
"https://good.example/dns-query",
now - 3600,
now + 3600,
);
try testing.expectEqual(@as(u64, 1), succeeding.successes);
try testing.expectEqual(@as(u64, 0), succeeding.failures);
try testing.expectEqual(@as(?i64, null), succeeding.last_failure_ts);
}
test "snapshot reports the counters in pool order" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -851,3 +979,69 @@ test "an entry that enters backoff while a task waits on it is not attempted" {
try testing.expectEqual(@as(u64, 1), entries[0].health.total_failures);
try testing.expect(entries[0].health.backoff_until != null);
}
test "a successful exchange with nothing open costs the store no statement" {
if (@FieldType(events.Store, "statements") != u64) return error.SkipZigTest;
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var good: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{testEntry("https://good.example/dns-query", &good, 10)};
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
pool.diagnostics = &fx.store;
var buf: [512]u8 = undefined;
const before = fx.store.statements;
for (0..20) |_| _ = try pool.exchange(io, query_bytes, &buf);
try testing.expectEqual(before, fx.store.statements);
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
}
test "a failing then recovering upstream leaves exactly one resolved episode" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var flaky: Fake = .{ .behavior = .{ .fail = error.Timeout } };
var standby: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{
testEntry("https://flaky.example/dns-query", &flaky, 10),
testEntry("https://standby.example/dns-query", &standby, 20),
};
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
pool.diagnostics = &fx.store;
var buf: [512]u8 = undefined;
_ = try pool.exchange(io, query_bytes, &buf);
// Backoff would park the failing entry, so the second failure is driven
// through `recordFailure` itself rather than through another exchange.
pool.recordFailure(io, &entries[0], std.Io.Clock.awake.now(io), error.ConnectFailed);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqual(@as(i64, 2), try fx.count("SELECT occurrences FROM operational_events"));
try testing.expectEqualStrings("upstream.exchange", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings(
"https://flaky.example/dns-query",
try fx.text("SELECT subject_key FROM operational_events"),
);
flaky.behavior = .{ .reply = response_bytes };
pool.recordSuccess(io, &entries[0], std.Io.Clock.awake.now(io));
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
try testing.expectEqual(@as(i64, 2), try fx.count("SELECT occurrences FROM operational_events"));
}
+448
View File
@@ -0,0 +1,448 @@
//! `GET /api/diagnostics` and `GET /api/diagnostics/{id}` — the operational
//! event log, newest first.
//!
//! Keyset pagination and the same page envelope as `/api/queries`, for the same
//! reason: the table is append-only at the head, so `id < before` is one index
//! seek however deep a client has scrolled, and rows arriving between two pages
//! cannot shift the window and duplicate one.
//!
//! Filter parsing is separated from fetching, because parsing is where PLAN
//! §19's input validation lives and it is worth testing on its own. Every value
//! is length-capped here and bound as a SQL parameter by the repository;
//! nothing this file reads is ever concatenated into a statement.
//!
//! No SQL and no connection of its own: `events.Store` owns the one diagnostics
//! connection and locks its mutex around every read, so this file cannot race
//! the emitters writing through it.
const std = @import("std");
const db = @import("../../storage/db.zig");
const events = @import("../../storage/events.zig");
const http_util = @import("../http_util.zig");
const server = @import("../server.zig");
const log = std.log.scoped(.web_diagnostics);
pub const default_limit: u32 = 100;
pub const max_limit: u32 = events.max_limit;
/// Wide enough for every component `events.component` can produce, and for a
/// mistyped one to still be reported as a bad filter rather than a truncated
/// match.
pub const max_component_len = 64;
/// Where the string filters are copied to. The parsed filter borrows them, so
/// it must not outlive the buffers — in the handler both live in the same stack
/// frame.
pub const Buffers = struct {
state: [16]u8 = undefined,
severity: [16]u8 = undefined,
component: [max_component_len]u8 = undefined,
};
pub const FilterError = error{
BadState,
BadSeverity,
BadComponent,
BadSince,
BadUntil,
BadLimit,
BadBefore,
};
/// An absent parameter drops the filter; a malformed one is a 400 rather than a
/// filter silently left off, which would answer a question the client did not
/// ask.
pub fn parseFilter(query: []const u8, buffers: *Buffers) FilterError!events.Filter {
var filter: events.Filter = .{ .limit = default_limit };
if (http_util.queryValue(query, "state", &buffers.state) catch return error.BadState) |text| {
filter.state = std.meta.stringToEnum(events.State, text) orelse return error.BadState;
}
if (http_util.queryValue(query, "severity", &buffers.severity) catch return error.BadSeverity) |text| {
// Bound as text by the repository, so it is normalised to one of the
// two stored spellings here rather than passed through.
const severity = std.meta.stringToEnum(events.Severity, text) orelse return error.BadSeverity;
filter.severity = severity.text();
}
if (http_util.queryValue(query, "component", &buffers.component) catch return error.BadComponent) |text| {
if (text.len != 0) filter.component = text;
}
filter.since = http_util.queryInt(i64, query, "since") catch return error.BadSince;
filter.until = http_util.queryInt(i64, query, "until") catch return error.BadUntil;
if (http_util.queryInt(u32, query, "limit") catch return error.BadLimit) |limit| {
if (limit == 0 or limit > max_limit) return error.BadLimit;
filter.limit = limit;
}
if (http_util.queryInt(i64, query, "before") catch return error.BadBefore) |before| {
// Row ids are positive, so a non-positive cursor is a client bug, not
// an empty page.
if (before <= 0) return error.BadBefore;
filter.before = before;
}
return filter;
}
pub fn message(err: FilterError) []const u8 {
return switch (err) {
error.BadState => "state must be active, resolved or all",
error.BadSeverity => "severity must be warning or error",
error.BadComponent => "component is not a valid filter",
error.BadSince => "since must be a unix timestamp in seconds",
error.BadUntil => "until must be a unix timestamp in seconds",
error.BadLimit => "limit must be between 1 and 1000",
error.BadBefore => "before must be a positive row id",
};
}
const unavailable_message = "diagnostics unavailable";
pub fn list(
state: *server.WebState,
io: std.Io,
request: *http_util.Request,
) http_util.HandlerError!void {
var buffers: Buffers = .{};
const filter = parseFilter(request.query, &buffers) catch |err| {
return http_util.respondError(request, .bad_request, message(err));
};
const store = state.events orelse
return http_util.respondError(request, .service_unavailable, unavailable_message);
const page = store.selectEvents(io, request.arena, filter) catch |err| {
// The one thing this handler logs: a database fault is a property of
// the box, not of the request, and the client is told nothing about it.
log.warn("diagnostics read failed: {s}", .{@errorName(err)});
return http_util.respondError(request, .internal_server_error, "internal error");
};
return http_util.respondJson(request, .ok, page, &.{});
}
pub fn get(
state: *server.WebState,
io: std.Io,
request: *http_util.Request,
) http_util.HandlerError!void {
const store = state.events orelse
return http_util.respondError(request, .service_unavailable, unavailable_message);
const row = store.selectOne(io, request.arena, request.id.?) catch |err| {
log.warn("diagnostics read failed: {s}", .{@errorName(err)});
return http_util.respondError(request, .internal_server_error, "internal error");
};
// An id retention has removed and one that never existed are the same
// answer, and the API does not pretend to tell them apart.
const event = row orelse return http_util.respondError(request, .not_found, "not found");
return http_util.respondJson(request, .ok, event, &.{});
}
/// An open episode is the current state of the box, so it is not history to
/// throw away — and the message says what would make it purgeable.
const still_active_message = "the event is still active; it can be purged once it resolves";
/// `DELETE /api/diagnostics` — how many resolved events went.
pub const PurgeResult = struct {
purged: i64,
};
/// `DELETE /api/diagnostics/{id}`. Resolution stays automatic; this is only
/// about when the history disappears, which is the operator's call.
///
/// Classified `runtime_action` in the route table, not `config_write`: the
/// event log is runtime state that no configuration file declares, so file
/// authority has nothing to say about it and the router lets this through in
/// both modes.
pub fn purge(
state: *server.WebState,
io: std.Io,
request: *http_util.Request,
) http_util.HandlerError!void {
const store = state.events orelse
return http_util.respondError(request, .service_unavailable, unavailable_message);
// No log here: the store already latches and logs the false→true
// transition, and a warning per retried request would spam.
const outcome = store.purge(io, request.id.?) catch
return http_util.respondError(request, .internal_server_error, "internal error");
return switch (outcome) {
.deleted => http_util.respondEmpty(request, .no_content),
.active => http_util.respondError(request, .conflict, still_active_message),
.absent => http_util.respondError(request, .not_found, "not found"),
};
}
/// `DELETE /api/diagnostics` — the whole resolved history at once. Active
/// episodes are never touched, so an operator clearing the page cannot lose the
/// events that are still true.
pub fn purgeAll(
state: *server.WebState,
io: std.Io,
request: *http_util.Request,
) http_util.HandlerError!void {
const store = state.events orelse
return http_util.respondError(request, .service_unavailable, unavailable_message);
const purged = store.purgeAll(io) catch
return http_util.respondError(request, .internal_server_error, "internal error");
return http_util.respondJson(request, .ok, PurgeResult{ .purged = purged }, &.{});
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const migrations = @import("../../storage/migrations.zig");
const testing = std.testing;
test "an empty query string is the default page over everything" {
var buffers: Buffers = .{};
const filter = try parseFilter("", &buffers);
try testing.expectEqual(default_limit, filter.limit);
try testing.expectEqual(events.State.all, filter.state);
try testing.expectEqual(@as(?[]const u8, null), filter.severity);
try testing.expectEqual(@as(?[]const u8, null), filter.component);
try testing.expectEqual(@as(?i64, null), filter.before);
try testing.expectEqual(@as(?i64, null), filter.since);
}
test "every filter reaches the store untouched" {
var buffers: Buffers = .{};
const filter = try parseFilter(
"state=resolved&severity=error&component=query_log&since=100&until=200&limit=250&before=900",
&buffers,
);
try testing.expectEqual(events.State.resolved, filter.state);
try testing.expectEqualStrings("error", filter.severity.?);
try testing.expectEqualStrings("query_log", filter.component.?);
try testing.expectEqual(@as(?i64, 100), filter.since);
try testing.expectEqual(@as(?i64, 200), filter.until);
try testing.expectEqual(@as(u32, 250), filter.limit);
try testing.expectEqual(@as(?i64, 900), filter.before);
}
test "an empty component is no filter at all" {
var buffers: Buffers = .{};
try testing.expectEqual(@as(?[]const u8, null), (try parseFilter("component=", &buffers)).component);
}
test "each malformed parameter names itself in a 400" {
var buffers: Buffers = .{};
try testing.expectError(error.BadState, parseFilter("state=open", &buffers));
try testing.expectError(error.BadState, parseFilter("state=", &buffers));
try testing.expectError(error.BadSeverity, parseFilter("severity=info", &buffers));
try testing.expectError(error.BadSeverity, parseFilter("severity=WARNING", &buffers));
try testing.expectError(error.BadSince, parseFilter("since=yesterday", &buffers));
try testing.expectError(error.BadUntil, parseFilter("until=", &buffers));
try testing.expectError(error.BadLimit, parseFilter("limit=0", &buffers));
try testing.expectError(error.BadLimit, parseFilter("limit=1001", &buffers));
try testing.expectError(error.BadLimit, parseFilter("limit=ten", &buffers));
try testing.expectError(error.BadBefore, parseFilter("before=0", &buffers));
try testing.expectError(error.BadBefore, parseFilter("before=-4", &buffers));
try testing.expectError(error.BadComponent, parseFilter("component=%zz", &buffers));
var long: [max_component_len + 8]u8 = @splat('a');
var text: std.ArrayList(u8) = .empty;
defer text.deinit(testing.allocator);
try text.appendSlice(testing.allocator, "component=");
try text.appendSlice(testing.allocator, &long);
try testing.expectError(error.BadComponent, parseFilter(text.items, &buffers));
// Every member of the error set has its own wording, and none of them is
// the empty string.
inline for (comptime std.meta.fieldNames(FilterError)) |name| {
try testing.expect(message(@field(FilterError, name)).len != 0);
}
}
test "the limit cap is the store's" {
var buffers: Buffers = .{};
try testing.expectEqual(max_limit, (try parseFilter("limit=1000", &buffers)).limit);
try testing.expectEqual(@as(u32, 1000), events.max_limit);
}
/// A store over a migrated in-memory `config.db`, built in place: a `Store`
/// holds a `*db.Db`, so a fixture that moved after `init` would leave that
/// pointer behind.
const Fixture = struct {
threaded: std.Io.Threaded = undefined,
io: std.Io = undefined,
database: db.Db = undefined,
store: events.Store = undefined,
state: server.WebState = undefined,
fn init(self: *Fixture) !void {
self.threaded = .init(testing.allocator, .{});
errdefer self.threaded.deinit();
self.io = self.threaded.io();
self.database = try db.Db.open(":memory:", .{ .mode = .memory });
errdefer self.database.close();
try db.applyPragmas(&self.database, .{});
_ = try migrations.migrate(&self.database);
self.store = try events.Store.init(self.io, &self.database, 1000);
self.state = .{ .gpa = testing.allocator, .events = &self.store };
}
fn deinit(self: *Fixture) void {
self.database.close();
self.threaded.deinit();
}
fn seed(self: *Fixture) void {
const store = &self.store;
store.report(self.io, 1000, .blocklist_refresh, "https://a.example", "StevenBlack", .warning, "ConnectionTimedOut");
store.report(self.io, 1100, .blocklist_refresh, "https://a.example", "StevenBlack", .warning, "ConnectionTimedOut");
store.report(self.io, 1200, .listener_start, "doh", "doh", .@"error", "AddressInUse");
store.report(self.io, 1300, .query_log_write, "batch", "batch", .@"error", "Busy");
store.resolve(self.io, 1400, .query_log_write, "batch");
}
};
test "a page carries the events, the cursor and the active counts" {
var fx: Fixture = .{};
try fx.init();
defer fx.deinit();
fx.seed();
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const page = try fx.store.selectEvents(fx.io, arena.allocator(), .{ .limit = 100 });
try testing.expectEqual(@as(usize, 3), page.events.len);
try testing.expectEqual(@as(?i64, null), page.next_before);
try testing.expectEqual(events.Counts{ .warnings = 1, .errors = 1 }, page.active);
// Newest first, and the episode that repeated counts rather than repeats.
try testing.expectEqualStrings("query_log.write", page.events[0].code);
try testing.expectEqualStrings("blocklist.refresh", page.events[2].code);
try testing.expectEqual(@as(i64, 2), page.events[2].occurrences);
try testing.expectEqualStrings("StevenBlack", page.events[2].subject);
}
test "a full page carries a cursor and the last page does not" {
var fx: Fixture = .{};
try fx.init();
defer fx.deinit();
fx.seed();
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const gpa = arena.allocator();
const first = try fx.store.selectEvents(fx.io, gpa, .{ .limit = 2 });
try testing.expectEqual(@as(usize, 2), first.events.len);
try testing.expectEqual(first.events[1].id, first.next_before.?);
const second = try fx.store.selectEvents(fx.io, gpa, .{ .limit = 2, .before = first.next_before });
try testing.expectEqual(@as(usize, 1), second.events.len);
try testing.expectEqual(@as(?i64, null), second.next_before);
}
test "the parsed filters narrow the page the store answers with" {
var fx: Fixture = .{};
try fx.init();
defer fx.deinit();
fx.seed();
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const gpa = arena.allocator();
var buffers: Buffers = .{};
const active = try fx.store.selectEvents(fx.io, gpa, try parseFilter("state=active", &buffers));
try testing.expectEqual(@as(usize, 2), active.events.len);
const resolved = try fx.store.selectEvents(fx.io, gpa, try parseFilter("state=resolved", &buffers));
try testing.expectEqual(@as(usize, 1), resolved.events.len);
try testing.expectEqual(@as(?i64, 1400), resolved.events[0].resolved_at);
const errors = try fx.store.selectEvents(fx.io, gpa, try parseFilter("severity=error", &buffers));
try testing.expectEqual(@as(usize, 2), errors.events.len);
const blocklist = try fx.store.selectEvents(fx.io, gpa, try parseFilter("component=blocklist", &buffers));
try testing.expectEqual(@as(usize, 1), blocklist.events.len);
try testing.expectEqualStrings("blocklist", blocklist.events[0].component);
}
test "the wire object carries the label and never the subject key" {
var fx: Fixture = .{};
try fx.init();
defer fx.deinit();
// A key that would be unmistakable on the wire if it ever leaked: the
// upstream urls this store keys on can carry a token.
fx.store.report(fx.io, 1000, .upstream_exchange, "https://dns.example/secret-token", "dns.example", .warning, "Timeout");
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const page = try fx.store.selectEvents(fx.io, arena.allocator(), .{});
var allocating: std.Io.Writer.Allocating = .init(testing.allocator);
defer allocating.deinit();
try std.json.Stringify.value(page, .{}, &allocating.writer);
const text = allocating.written();
try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "secret-token"));
try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "subject_key"));
try testing.expect(std.mem.startsWith(u8, text, "{\"events\":["));
for ([_][]const u8{
"\"id\":", "\"code\":\"upstream.exchange\"",
"\"component\":\"upstream\"", "\"subject\":\"dns.example\"",
"\"severity\":\"warning\"", "\"first_seen\":",
"\"last_seen\":", "\"occurrences\":",
"\"resolved_at\":null", "\"detail\":\"Timeout\"",
"\"next_before\":null", "\"active\":{\"warnings\":1,\"errors\":0}",
}) |field| {
errdefer std.debug.print("missing {s} in {s}\n", .{ field, text });
try testing.expect(std.mem.containsAtLeast(u8, text, 1, field));
}
}
test "a detail page answers by id and reports an unknown one as absent" {
var fx: Fixture = .{};
try fx.init();
defer fx.deinit();
fx.seed();
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const found = (try fx.store.selectOne(fx.io, arena.allocator(), 1)).?;
try testing.expectEqualStrings("blocklist.refresh", found.code);
// The 404 the handler answers with is this null.
try testing.expect((try fx.store.selectOne(fx.io, arena.allocator(), 9999)) == null);
}
test "the purge-all body is the count and nothing else" {
var allocating: std.Io.Writer.Allocating = .init(testing.allocator);
defer allocating.deinit();
try std.json.Stringify.value(PurgeResult{ .purged = 3 }, .{}, &allocating.writer);
try testing.expectEqualStrings("{\"purged\":3}", allocating.written());
// The 409 says what would make the event purgeable, so it is not the
// generic conflict wording.
try testing.expect(still_active_message.len != 0);
try testing.expect(std.mem.containsAtLeast(u8, still_active_message, 1, "resolves"));
}
test "a state with no store answers 503 rather than an empty page" {
// The handler's only branch that does not need a request: a `WebState`
// whose store failed to open reports the endpoint unavailable, and never
// an empty list that would read as "nothing is wrong".
const state: server.WebState = .{ .gpa = testing.allocator };
try testing.expectEqual(@as(?*events.Store, null), state.events);
try testing.expect(unavailable_message.len != 0);
}
+183 -4
View File
@@ -25,6 +25,15 @@ pub const Disk = struct {
sample_failures: u64,
};
/// The diagnostics store's own state, not a summary of what it holds: `state`
/// answers "is the operational log recording", and the two counts answer "what
/// is open right now".
pub const Diagnostics = struct {
state: []const u8,
active_warnings: u32,
active_errors: u32,
};
pub const Upstreams = struct {
available: u32,
total: u32,
@@ -34,6 +43,7 @@ pub const Body = struct {
status: []const u8,
disk: Disk,
upstreams: Upstreams,
diagnostics: Diagnostics,
queries_dropped: u64,
writer_failed: bool,
refreshes_gated: u64,
@@ -52,6 +62,25 @@ pub const Input = struct {
upstreams_total: u32 = 0,
queries_dropped: u64 = 0,
writer_failed: bool = false,
/// Current state, not a count: the upstream-history flush is failing right
/// now. Cleared by the next flush that succeeds (m26 ruling 7).
///
/// `rows_dropped` deliberately does not appear here. It is cumulative, and
/// a rollup that is computed statelessly cannot ask whether a counter grew
/// — so feeding it in would latch `/api/health` to degraded forever after
/// one overflow. Drops surface through the metric and through the API's
/// per-window `complete` instead.
history_flush_failing: bool = false,
/// The diagnostics store exists. The benign default matches every other
/// field here — a half-wired `Input` reports a box with nothing wrong — but
/// `collect` must assign it explicitly, because in a serving process an
/// absent store means `Store.init` failed.
diagnostics_present: bool = true,
/// The last diagnostics write failed. Current state, cleared by the next
/// write that succeeds, like `history_flush_failing`.
diagnostics_write_failed: bool = false,
diagnostics_active_warnings: u32 = 0,
diagnostics_active_errors: u32 = 0,
refreshes_gated: u64 = 0,
snapshot_generation: ?u64 = null,
};
@@ -59,11 +88,25 @@ pub const Input = struct {
pub const status_ok = "ok";
pub const status_degraded = "degraded";
/// Ruling 22's three conditions. Each one is something an operator must act on:
/// a disk that is filling stops the query log, a pool with nothing available
/// stops resolution, and a failed writer means rows are being lost right now.
pub const diagnostics_recording = "recording";
pub const diagnostics_unavailable = "unavailable";
/// The operational log is not recording — either the store never opened or its
/// writes are failing. Both mean the same thing to an operator: the record of
/// what went wrong is not being kept.
pub fn diagnosticsUnavailable(input: Input) bool {
return !input.diagnostics_present or input.diagnostics_write_failed;
}
/// Conditions an operator must act on, and every one of them is a fact about
/// now rather than a count of the past: a disk that is filling stops the query
/// log, a pool with nothing available stops resolution, a failed writer means
/// rows are being lost right now, and a failing history flush means the
/// dashboard's upstream numbers are not being recorded. Each clears itself when
/// the underlying condition does.
pub fn degraded(input: Input) bool {
return input.disk_state != .ok or input.upstreams_available == 0 or input.writer_failed;
return input.disk_state != .ok or input.upstreams_available == 0 or
input.writer_failed or input.history_flush_failing or diagnosticsUnavailable(input);
}
pub fn rollup(input: Input) Body {
@@ -77,6 +120,11 @@ pub fn rollup(input: Input) Body {
.sample_failures = input.disk_sample_failures,
},
.upstreams = .{ .available = input.upstreams_available, .total = input.upstreams_total },
.diagnostics = .{
.state = if (diagnosticsUnavailable(input)) diagnostics_unavailable else diagnostics_recording,
.active_warnings = input.diagnostics_active_warnings,
.active_errors = input.diagnostics_active_errors,
},
.queries_dropped = input.queries_dropped,
.writer_failed = input.writer_failed,
.refreshes_gated = input.refreshes_gated,
@@ -115,6 +163,21 @@ pub fn collect(state: *server.WebState, io: std.Io) Input {
input.writer_failed = logger.writer_failed.load(.monotonic);
}
// Assigned before the `if`, not inside it: the field's benign default is
// `true`, so the natural `if (state.events) |store|` shape would report an
// absent store as recording — the one case that must degrade.
input.diagnostics_present = state.events != null;
if (state.events) |store| {
input.diagnostics_write_failed = store.writeFailed();
const counts = store.activeCounts(io);
input.diagnostics_active_warnings = counts.warnings;
input.diagnostics_active_errors = counts.errors;
}
if (state.history) |history| {
input.history_flush_failing = history.snapshotStats(io).last_flush_failed;
}
if (state.manager) |manager| {
input.refreshes_gated = manager.refreshesGated();
if (manager.acquire(io)) |acquired| {
@@ -130,7 +193,12 @@ pub fn collect(state: *server.WebState, io: std.Io) Input {
// tests
// ---------------------------------------------------------------------------
const db = @import("../../storage/db.zig");
const events_mod = @import("../../storage/events.zig");
const migrations = @import("../../storage/migrations.zig");
const history_mod = @import("../../upstream/history.zig");
const logger_mod = @import("../../storage/logger.zig");
const upstream_history_repo = @import("../../storage/repositories/upstream_history_repo.zig");
const testing = std.testing;
/// A box with nothing wrong with it: one upstream up, disk ok, writer alive.
@@ -148,6 +216,15 @@ test "the degraded matrix covers disk state, availability and the writer" {
.{ .input = withDisk(healthy, .critical), .degraded = true },
.{ .input = withAvailable(healthy, 0), .degraded = true },
.{ .input = withWriterFailed(healthy), .degraded = true },
// A failing upstream-history flush is losing the dashboard's numbers
// right now, and it recovers on its own the moment a flush succeeds.
.{ .input = withHistoryFailing(healthy, true), .degraded = true },
.{ .input = withHistoryFailing(healthy, false), .degraded = false },
// The operational log not recording is itself a fault an operator must
// act on: whatever fails next will leave no record of having failed.
.{ .input = withDiagnostics(healthy, false, false), .degraded = true },
.{ .input = withDiagnostics(healthy, true, true), .degraded = true },
.{ .input = withDiagnostics(healthy, true, false), .degraded = false },
// Two faults at once still report one status.
.{ .input = withWriterFailed(withDisk(healthy, .critical)), .degraded = true },
// Some upstreams down is not degraded while one still answers.
@@ -182,6 +259,108 @@ fn withWriterFailed(input: Input) Input {
return out;
}
fn withHistoryFailing(input: Input, failing: bool) Input {
var out = input;
out.history_flush_failing = failing;
return out;
}
fn withDiagnostics(input: Input, present: bool, write_failed: bool) Input {
var out = input;
out.diagnostics_present = present;
out.diagnostics_write_failed = write_failed;
return out;
}
test "the diagnostics block reports the state and the open counts" {
const recording = rollup(.{
.upstreams_available = 1,
.diagnostics_active_warnings = 3,
.diagnostics_active_errors = 1,
});
try testing.expectEqualStrings(diagnostics_recording, recording.diagnostics.state);
try testing.expectEqual(@as(u32, 3), recording.diagnostics.active_warnings);
try testing.expectEqual(@as(u32, 1), recording.diagnostics.active_errors);
// Open episodes are what the box is doing, not a fault of the log: they do
// not degrade on their own.
try testing.expectEqualStrings(status_ok, recording.status);
// Failing writes: the counts are whatever was last read, and the state is
// the honest one.
const failing = rollup(.{ .upstreams_available = 1, .diagnostics_write_failed = true });
try testing.expectEqualStrings(diagnostics_unavailable, failing.diagnostics.state);
try testing.expectEqualStrings(status_degraded, failing.status);
const absent = rollup(.{ .upstreams_available = 1, .diagnostics_present = false });
try testing.expectEqualStrings(diagnostics_unavailable, absent.diagnostics.state);
try testing.expectEqualStrings(status_degraded, absent.status);
}
test "collect reports an absent store as unavailable rather than as recording" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
// `diagnostics_present` defaults to true like every other benign default,
// so an assignment `collect` forgot would read as a healthy log here.
var state: server.WebState = .{ .gpa = testing.allocator };
const absent = collect(&state, io);
try testing.expect(!absent.diagnostics_present);
try testing.expectEqualStrings(diagnostics_unavailable, rollup(absent).diagnostics.state);
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try db.applyPragmas(&database, .{});
_ = try migrations.migrate(&database);
var store = try events_mod.Store.init(io, &database, 1000);
store.report(io, 1000, .disk_space, "data", "data", .warning, "low");
store.report(io, 1000, .listener_start, "doh", "doh", .@"error", "AddressInUse");
state.events = &store;
const present = collect(&state, io);
try testing.expect(present.diagnostics_present);
try testing.expect(!present.diagnostics_write_failed);
try testing.expectEqual(@as(u32, 1), present.diagnostics_active_warnings);
try testing.expectEqual(@as(u32, 1), present.diagnostics_active_errors);
try testing.expectEqualStrings(diagnostics_recording, rollup(present).diagnostics.state);
}
test "a history overflow that already happened does not degrade the rollup" {
// `rows_dropped` is cumulative and the rollup is stateless, so the only
// thing it could do with a drop count is latch on it. The accumulator's
// drops reach an operator through `/metrics` and through the per-window
// `complete` flag, and never through this.
const dropped: Input = .{
.upstreams_available = 1,
.upstreams_total = 1,
.history_flush_failing = false,
};
try testing.expect(!degraded(dropped));
try testing.expectEqualStrings(status_ok, rollup(dropped).status);
}
test "collect reads the accumulator's current flush state" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
const acc = try testing.allocator.create(history_mod.Accumulator);
defer testing.allocator.destroy(acc);
acc.* = .init;
var state: server.WebState = .{ .gpa = testing.allocator, .history = acc };
try testing.expect(!collect(&state, io).history_flush_failing);
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
acc.recordSuccess(io, "https://a.example", 60);
// No schema in this database, so the real write fails and the flag is set
// by the production path rather than by a test poking a field.
acc.flushOnce(io, &database, upstream_history_repo.flush);
try testing.expect(collect(&state, io).history_flush_failing);
try testing.expectEqualStrings("degraded", rollup(collect(&state, io)).status);
}
test "the body reports every input verbatim" {
const body = rollup(.{
.disk_state = .warn,
+2
View File
@@ -203,6 +203,7 @@ const LoggingView = struct {
level: []const u8,
retention_days: u16,
query_log_buffer_max: u32,
query_log_flush_interval_s: u16,
hide_domains: bool,
hide_client_ips: bool,
output: []const u8,
@@ -263,6 +264,7 @@ pub fn view(cfg: model.Config) View {
.level = cfg.logging.level.toDb(),
.retention_days = cfg.logging.retention_days,
.query_log_buffer_max = cfg.logging.query_log_buffer_max,
.query_log_flush_interval_s = cfg.logging.query_log_flush_interval_s,
.hide_domains = cfg.logging.hide_domains,
.hide_client_ips = cfg.logging.hide_client_ips,
.output = cfg.logging.output.toDb(),
+479 -70
View File
@@ -1,52 +1,116 @@
//! `GET /api/upstream/health` — the pool's own view of its upstreams.
//! `GET /api/upstream/health?period=` — the pool's upstreams over the window
//! the dashboard's period picker selected (milestone-26 ruling 6).
//!
//! The rows are `Pool.Snapshot` with the borrowed strings copied. `last_error`
//! points into the entry that produced it and is rewritten by that entry's next
//! failure, so it is duplicated into the request arena before the pool's mutex
//! is out of sight.
//! Two kinds of fact, kept apart on the wire because they answer different
//! questions. `enabled`/`available` are live routing state, read from the pool
//! under its mutex: what the resolver would do with this upstream right now.
//! Everything under `period` is history, aggregated out of `upstream_minute`
//! over `[since, until)` — the same window `/api/stats` reports, so a page
//! cannot show a rate that disagrees with the chart beside it.
//!
//! No timestamps: the health fields are stamped on the `awake` clock, which
//! stops while the box is suspended and means nothing to a client reading wall
//! time. What an operator needs — is it up, how often does it fail, what did it
//! say last — is here without them.
//! Nothing here reads a process-lifetime counter. The lifetime totals, the
//! last-32-exchange window and the consecutive-failure count still live in
//! `health.State` for routing and in `/metrics`; they are not this response's
//! business, because a number that starts at process start cannot be scoped to
//! a period and a dashboard that shows one beside a picker lies about it.
//!
//! The aggregation runs on the web task's own query-log connection (m7 ruling
//! 21) and this file owns no SQL: `upstream_history_repo` does.
const std = @import("std");
const Allocator = std.mem.Allocator;
const db = @import("../../storage/db.zig");
const history_mod = @import("../../upstream/history.zig");
const http_util = @import("../http_util.zig");
const metrics = @import("../metrics.zig");
const pool_mod = @import("../../upstream/pool.zig");
const server = @import("../server.zig");
const stats = @import("stats.zig");
const upstream_history_repo = @import("../../storage/repositories/upstream_history_repo.zig");
const log = std.log.scoped(.web_upstream_health);
/// One upstream's outcomes inside the selected window.
pub const PeriodStats = struct {
attempts: u64,
successes: u64,
failures: u64,
/// Null when `attempts == 0`. No observations is not perfect reliability,
/// and a `100.0%` from an idle upstream is the exact misreading this
/// milestone exists to remove.
success_rate: ?f32,
/// The newest failure inside the window, on the wall clock the minute rows
/// are stamped with. Null when the window holds no failure, even if the
/// upstream failed before it.
last_failure_at: ?i64,
/// The error name belonging to `last_failure_at`; null exactly when it is.
last_failure_error: ?[]const u8,
};
pub const Upstream = struct {
url: []const u8,
/// Live: configuration, not history.
enabled: bool,
/// Live: false while the upstream is backing off.
available: bool,
consecutive_failures: u32,
total_successes: u64,
total_failures: u64,
success_rate: f32,
/// "" when the upstream has never failed.
last_error: []const u8,
period: PeriodStats,
};
pub const Body = struct {
upstreams: []const Upstream,
period: []const u8,
since: i64,
until: i64,
available: u32,
total: u32,
/// See `isComplete`.
complete: bool,
upstreams: []const Upstream,
};
/// The same text `/api/stats` sends (`stats.zig`'s `badPeriod`). One period
/// grammar serves the whole dashboard, so the two routes must not disagree
/// about what a typo means.
pub const bad_period_message = "period must be one of 1h, 24h, 7d, 30d";
pub fn handle(
state: *server.WebState,
io: std.Io,
request: *http_util.Request,
) http_util.HandlerError!void {
const period = stats.periodParam(request.query) catch
return http_util.respondError(request, .bad_request, bad_period_message);
const pool = state.pool orelse
return http_util.respondError(request, .service_unavailable, "no upstream pool");
return http_util.respondJson(request, .ok, try collect(pool, io, request.arena), &.{});
const database = state.querylog_db orelse
return http_util.respondError(request, .service_unavailable, "query log unavailable");
const now = std.Io.Clock.real.now(io).toSeconds();
const body = collect(request.arena, io, pool, state.history, database, period, now) catch |err| {
if (err == error.OutOfMemory) return error.OutOfMemory;
// A failed aggregate is a fault in the box, not a property of the
// request (ruling 8, PLAN §19).
log.warn("upstream health window failed: {s}", .{@errorName(err)});
return http_util.respondError(request, .internal_server_error, "internal error");
};
return http_util.respondJson(request, .ok, body, &.{});
}
pub fn collect(pool: *pool_mod.Pool, io: std.Io, arena: Allocator) Allocator.Error!Body {
/// `db.Error` already carries `OutOfMemory`, so the arena's failures and
/// SQLite's share one set.
pub const Error = Allocator.Error || db.Error;
pub fn collect(
arena: Allocator,
io: std.Io,
pool: *pool_mod.Pool,
history: ?*history_mod.Accumulator,
database: *db.Db,
period: stats.Period,
now_unix: i64,
) Error!Body {
const span = stats.window(period, now_unix);
var raw: [metrics.max_upstreams]pool_mod.Snapshot = undefined;
const count = metrics.poolSnapshot(pool, io, &raw);
@@ -54,25 +118,76 @@ pub fn collect(pool: *pool_mod.Pool, io: std.Io, arena: Allocator) Allocator.Err
var available: u32 = 0;
for (raw[0..count], out) |entry, *slot| {
if (entry.available) available += 1;
// Rows come from the current pool only: an upstream deleted from the
// configuration keeps its history in storage until retention takes it,
// and nothing joins it back into this response.
const window_stats = try upstream_history_repo.windowStats(
database,
entry.url,
span.since,
span.until,
);
slot.* = .{
.url = try arena.dupe(u8, entry.url),
.enabled = entry.enabled,
.available = entry.available,
.consecutive_failures = entry.consecutive_failures,
.total_successes = entry.total_successes,
.total_failures = entry.total_failures,
.success_rate = entry.success_rate,
.last_error = try arena.dupe(u8, entry.last_error),
.period = .{
.attempts = window_stats.attempts,
.successes = window_stats.successes,
.failures = window_stats.failures,
.success_rate = successRate(window_stats),
.last_failure_at = window_stats.last_failure_ts,
// `WindowStats` carries its error name by value, in storage this
// loop is done with as soon as the iteration ends. The copy into
// the arena is what keeps the response from pointing at bytes
// the next upstream's row overwrites.
.last_failure_error = if (window_stats.last_failure_ts == null)
null
else
try arena.dupe(u8, window_stats.lastFailureError()),
},
};
}
return .{ .upstreams = out, .available = available, .total = @intCast(count) };
return .{
.period = period.label(),
.since = span.since,
.until = span.until,
.available = available,
.total = @intCast(count),
.complete = isComplete(history, io, span.since),
.upstreams = out,
};
}
fn successRate(window_stats: upstream_history_repo.WindowStats) ?f32 {
if (window_stats.attempts == 0) return null;
const successes: f32 = @floatFromInt(window_stats.successes);
const attempts: f32 = @floatFromInt(window_stats.attempts);
return successes / attempts;
}
/// Per-window and stateless (ruling 6): false iff capacity has cost this
/// process a minute that falls inside the window. A window that starts after
/// the newest such minute is complete again, so one historical overflow does
/// not mark every later response.
///
/// It says nothing about the newest outcomes, which may not have flushed yet,
/// and nothing about an unclean shutdown, which is not detectable here — the
/// openapi description spells both out.
fn isComplete(history: ?*history_mod.Accumulator, io: std.Io, since: i64) bool {
const accumulator = history orelse return true;
const dropped = accumulator.snapshotStats(io).last_drop_minute orelse return true;
return dropped < since;
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const querylog_schema = @import("../../storage/querylog_schema.zig");
const transport = @import("../../upstream/transport.zig");
const testing = std.testing;
@@ -94,83 +209,377 @@ fn testPool(entries: []pool_mod.Entry) pool_mod.Pool {
}, 1);
}
test "every upstream is copied, counted and owned by the arena" {
fn openLog() !db.Db {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
errdefer database.close();
try db.applyPragmas(&database, .{});
try database.exec(querylog_schema.ddl);
return database;
}
const url_a = "https://a.test/dns-query";
const url_b = "https://b.test/dns-query";
/// A minute-aligned instant, so a window derived from it lands on round
/// numbers the assertions below can name.
const aligned_now: i64 = 1_699_999_980;
test "the window sums the minutes inside it and nothing outside" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var entries = [_]pool_mod.Entry{
testEntry("https://a.test/dns-query", true),
testEntry("https://b.test/dns-query", false),
};
var database = try openLog();
defer database.close();
const span = stats.window(.@"1h", aligned_now);
try upstream_history_repo.flush(&database, &.{
// One minute before the window.
.{ .url = url_a, .minute_ts = span.since - 60, .successes = 100, .failures = 100, .last_failure_ts = span.since - 30, .last_error = "Outside" },
.{ .url = url_a, .minute_ts = span.since, .successes = 3, .failures = 1, .last_failure_ts = span.since + 10, .last_error = "Timeout" },
.{ .url = url_a, .minute_ts = span.until - 60, .successes = 5, .failures = 0, .last_failure_ts = null, .last_error = "" },
// The window's exclusive end.
.{ .url = url_a, .minute_ts = span.until, .successes = 200, .failures = 200, .last_failure_ts = span.until + 5, .last_error = "After" },
});
var entries = [_]pool_mod.Entry{testEntry(url_a, true)};
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(&pool, io, arena.allocator());
try testing.expectEqual(@as(u32, 2), body.total);
const body = try collect(arena.allocator(), io, &pool, null, &database, .@"1h", aligned_now);
try testing.expectEqualStrings("1h", body.period);
try testing.expectEqual(span.since, body.since);
try testing.expectEqual(span.until, body.until);
try testing.expectEqual(@as(u32, 1), body.total);
try testing.expectEqual(@as(u32, 1), body.available);
const period = body.upstreams[0].period;
try testing.expectEqual(@as(u64, 8), period.successes);
try testing.expectEqual(@as(u64, 1), period.failures);
try testing.expectEqual(@as(u64, 9), period.attempts);
try testing.expectEqual(@as(?i64, span.since + 10), period.last_failure_at);
try testing.expectEqualStrings("Timeout", period.last_failure_error.?);
}
test "an upstream with no attempts in the window reports null, never a perfect rate" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const span = stats.window(.@"1h", aligned_now);
// Only `b` has history, and only outside the window.
try upstream_history_repo.flush(&database, &.{
.{ .url = url_b, .minute_ts = span.since - 600, .successes = 4, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
var entries = [_]pool_mod.Entry{ testEntry(url_a, true), testEntry(url_b, true) };
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(arena.allocator(), io, &pool, null, &database, .@"1h", aligned_now);
for (body.upstreams) |upstream| {
try testing.expectEqual(@as(u64, 0), upstream.period.attempts);
try testing.expectEqual(@as(u64, 0), upstream.period.successes);
try testing.expectEqual(@as(u64, 0), upstream.period.failures);
try testing.expectEqual(@as(?f32, null), upstream.period.success_rate);
try testing.expectEqual(@as(?i64, null), upstream.period.last_failure_at);
try testing.expectEqual(@as(?[]const u8, null), upstream.period.last_failure_error);
}
}
test "the success rate is the window's own, not a lifetime one" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const span = stats.window(.@"1h", aligned_now);
try upstream_history_repo.flush(&database, &.{
// A clean past that a lifetime rate would average into the window.
.{ .url = url_a, .minute_ts = span.since - 600, .successes = 1000, .failures = 0, .last_failure_ts = null, .last_error = "" },
.{ .url = url_a, .minute_ts = span.since, .successes = 1, .failures = 3, .last_failure_ts = span.since + 1, .last_error = "Timeout" },
});
var entries = [_]pool_mod.Entry{testEntry(url_a, true)};
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(arena.allocator(), io, &pool, null, &database, .@"1h", aligned_now);
try testing.expectEqual(@as(?f32, 0.25), body.upstreams[0].period.success_rate);
}
test "the newest failure inside the window wins over an older one outside it" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const span = stats.window(.@"1h", aligned_now);
try upstream_history_repo.flush(&database, &.{
.{ .url = url_a, .minute_ts = span.since - 120, .successes = 0, .failures = 1, .last_failure_ts = span.since - 100, .last_error = "Older" },
.{ .url = url_a, .minute_ts = span.since, .successes = 0, .failures = 1, .last_failure_ts = span.since + 5, .last_error = "Newer" },
// A later minute with no failure at all must not blank the error.
.{ .url = url_a, .minute_ts = span.since + 60, .successes = 2, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
var entries = [_]pool_mod.Entry{testEntry(url_a, true)};
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(arena.allocator(), io, &pool, null, &database, .@"1h", aligned_now);
try testing.expectEqual(@as(?i64, span.since + 5), body.upstreams[0].period.last_failure_at);
try testing.expectEqualStrings("Newer", body.upstreams[0].period.last_failure_error.?);
}
test "two upstreams keep their own last-failure errors" {
// The by-value `WindowStats` buffer is reused per iteration, so a response
// that borrowed it would show the second upstream's error on the first, or
// point at stack storage that is gone by the time it is serialized.
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const span = stats.window(.@"1h", aligned_now);
try upstream_history_repo.flush(&database, &.{
.{ .url = url_a, .minute_ts = span.since, .successes = 1, .failures = 1, .last_failure_ts = span.since + 1, .last_error = "ConnectFailed" },
.{ .url = url_b, .minute_ts = span.since, .successes = 0, .failures = 2, .last_failure_ts = span.since + 2, .last_error = "TlsHandshakeFailed" },
});
var entries = [_]pool_mod.Entry{ testEntry(url_a, true), testEntry(url_b, true) };
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(arena.allocator(), io, &pool, null, &database, .@"1h", aligned_now);
try testing.expectEqual(@as(usize, 2), body.upstreams.len);
try testing.expectEqualStrings("https://a.test/dns-query", body.upstreams[0].url);
try testing.expect(body.upstreams[0].enabled);
try testing.expect(body.upstreams[0].available);
try testing.expectEqualStrings(url_a, body.upstreams[0].url);
try testing.expectEqualStrings("ConnectFailed", body.upstreams[0].period.last_failure_error.?);
try testing.expectEqualStrings(url_b, body.upstreams[1].url);
try testing.expectEqualStrings("TlsHandshakeFailed", body.upstreams[1].period.last_failure_error.?);
// Serializing after every row is read is what a real response does; the
// texts must still be the ones their own rows carried.
var allocating: std.Io.Writer.Allocating = .init(testing.allocator);
defer allocating.deinit();
try std.json.Stringify.value(body, .{}, &allocating.writer);
const text = allocating.written();
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"last_failure_error\":\"ConnectFailed\""));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"last_failure_error\":\"TlsHandshakeFailed\""));
}
test "a disabled upstream is not counted available and still gets its window" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
const span = stats.window(.@"24h", aligned_now);
try upstream_history_repo.flush(&database, &.{
.{ .url = url_b, .minute_ts = span.since, .successes = 2, .failures = 0, .last_failure_ts = null, .last_error = "" },
});
var entries = [_]pool_mod.Entry{ testEntry(url_a, true), testEntry(url_b, false) };
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(arena.allocator(), io, &pool, null, &database, .@"24h", aligned_now);
try testing.expectEqual(@as(u32, 2), body.total);
try testing.expectEqual(@as(u32, 1), body.available);
try testing.expect(!body.upstreams[1].enabled);
try testing.expect(!body.upstreams[1].available);
// A disabled upstream is not available, so it is not counted.
try testing.expectEqual(@as(u32, 1), body.available);
try testing.expectEqualStrings("", body.upstreams[0].last_error);
try testing.expectEqual(@as(u64, 2), body.upstreams[1].period.attempts);
}
test "the copied strings survive the entry they came from" {
/// Fills the accumulator and then overflows it, so `last_drop_minute` is
/// `minute` — the only way to set it, because the accumulator's fields are
/// private to its module and `snapshotStats` is the read surface.
fn accumulatorDroppingAt(
io: std.Io,
minute: i64,
names: *[history_mod.max_pending][8]u8,
) !*history_mod.Accumulator {
const accumulator = try testing.allocator.create(history_mod.Accumulator);
accumulator.* = .init;
for (names, 0..) |*name, i| {
const url = std.fmt.bufPrint(name, "u{d:0>6}", .{i}) catch unreachable;
accumulator.recordSuccess(io, url, minute);
}
// One more cell than capacity: the oldest minute goes, and every cell above
// holds `minute`.
accumulator.recordSuccess(io, "https://overflow.test", minute + 60);
return accumulator;
}
test "complete is false only while a dropped minute falls inside the window" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var entries = [_]pool_mod.Entry{testEntry("https://a.test/dns-query", true)};
var pool = testPool(&entries);
var database = try openLog();
defer database.close();
const at = std.Io.Clock.awake.now(io);
entries[0].health.recordFailure(at, "ConnectFailed", .{}, 0);
var entries = [_]pool_mod.Entry{testEntry(url_a, true)};
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(&pool, io, arena.allocator());
try testing.expectEqualStrings("ConnectFailed", body.upstreams[0].last_error);
// The entry rewrites its buffer; the copy must not change with it.
entries[0].health.recordFailure(at, "Timeout", .{}, 0);
try testing.expectEqualStrings("ConnectFailed", body.upstreams[0].last_error);
// Two `now`s one minute apart, so the same drop is inside the first
// window and one minute before the second.
const inside_now = aligned_now;
const inside = stats.window(.@"1h", inside_now);
const after = stats.window(.@"1h", inside_now + 60);
try testing.expectEqual(inside.since + 60, after.since);
var names: [history_mod.max_pending][8]u8 = undefined;
const accumulator = try accumulatorDroppingAt(io, inside.since, &names);
defer testing.allocator.destroy(accumulator);
try testing.expectEqual(@as(?i64, inside.since), accumulator.snapshotStats(io).last_drop_minute);
const flagged = try collect(arena.allocator(), io, &pool, accumulator, &database, .@"1h", inside_now);
try testing.expect(!flagged.complete);
const recovered = try collect(arena.allocator(), io, &pool, accumulator, &database, .@"1h", inside_now + 60);
try testing.expect(recovered.complete);
// No accumulator at all is no known drop, not an incomplete window.
const unwired = try collect(arena.allocator(), io, &pool, null, &database, .@"1h", inside_now);
try testing.expect(unwired.complete);
}
test "the body serializes with snake_case field names" {
const upstreams = [_]Upstream{.{
.url = "https://a.test/dns-query",
test "a drop with no overflow leaves every window complete" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var entries = [_]pool_mod.Entry{testEntry(url_a, true)};
var pool = testPool(&entries);
const accumulator = try testing.allocator.create(history_mod.Accumulator);
defer testing.allocator.destroy(accumulator);
accumulator.* = .init;
accumulator.recordFailure(io, url_a, aligned_now, "Timeout");
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const body = try collect(arena.allocator(), io, &pool, accumulator, &database, .@"1h", aligned_now);
try testing.expect(body.complete);
}
test "the route's period grammar and its 400 text are the ones /api/stats serves" {
// The picker scopes the whole page, so one bad spelling must mean the same
// thing on every route it drives.
try testing.expectEqual(stats.default_period, try stats.periodParam(""));
try testing.expectEqual(stats.Period.@"24h", try stats.periodParam(""));
try testing.expectEqual(stats.Period.@"7d", try stats.periodParam("period=7d"));
try testing.expectError(error.BadPeriod, stats.periodParam("period=12h"));
try testing.expectError(error.BadPeriod, stats.periodParam("period=1hhhhhhhhhh"));
try testing.expectEqualStrings("period must be one of 1h, 24h, 7d, 30d", bad_period_message);
}
test "every period the grammar accepts produces the window that period names" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var entries = [_]pool_mod.Entry{testEntry(url_a, true)};
var pool = testPool(&entries);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
for (std.enums.values(stats.Period)) |period| {
const span = stats.window(period, aligned_now);
const body = try collect(arena.allocator(), io, &pool, null, &database, period, aligned_now);
try testing.expectEqualStrings(period.label(), body.period);
try testing.expectEqual(span.since, body.since);
try testing.expectEqual(span.until, body.until);
}
}
test "the body serializes exactly the ranged field set" {
const upstreams = [_]Upstream{ .{
.url = url_a,
.enabled = true,
.available = false,
.consecutive_failures = 3,
.total_successes = 10,
.total_failures = 4,
.success_rate = 0.5,
.last_error = "ConnectFailed",
}};
.period = .{
.attempts = 8,
.successes = 6,
.failures = 2,
.success_rate = 0.75,
.last_failure_at = 1_700_000_000,
.last_failure_error = "ConnectFailed",
},
}, .{
.url = url_b,
.enabled = false,
.available = false,
.period = .{
.attempts = 0,
.successes = 0,
.failures = 0,
.success_rate = null,
.last_failure_at = null,
.last_failure_error = null,
},
} };
var allocating: std.Io.Writer.Allocating = .init(testing.allocator);
defer allocating.deinit();
try std.json.Stringify.value(
Body{ .upstreams = &upstreams, .available = 0, .total = 1 },
.{},
&allocating.writer,
);
const text = allocating.written();
try std.json.Stringify.value(Body{
.period = "1h",
.since = 1_699_996_400,
.until = 1_700_000_000,
.available = 0,
.total = 2,
.complete = true,
.upstreams = &upstreams,
}, .{}, &allocating.writer);
try testing.expectEqualStrings(
\\{"period":"1h","since":1699996400,"until":1700000000,"available":0,"total":2,"complete":true,"upstreams":[{"url":"https://a.test/dns-query","enabled":true,"available":false,"period":{"attempts":8,"successes":6,"failures":2,"success_rate":0.75,"last_failure_at":1700000000,"last_failure_error":"ConnectFailed"}},{"url":"https://b.test/dns-query","enabled":false,"available":false,"period":{"attempts":0,"successes":0,"failures":0,"success_rate":null,"last_failure_at":null,"last_failure_error":null}}]}
, allocating.written());
// The lifetime fields m26 removed. They still exist in `health.State` and in
// `/metrics`; a client of this route must not find them here and start
// reading them as if they were scoped to the period.
for ([_][]const u8{
"\"consecutive_failures\":3",
"\"total_successes\":10",
"\"total_failures\":4",
"\"last_error\":\"ConnectFailed\"",
"\"available\":0",
"\"total\":1",
}) |fragment| {
try testing.expect(std.mem.containsAtLeast(u8, text, 1, fragment));
"consecutive_failures",
"total_successes",
"total_failures",
"last_error_age_s",
"\"last_error\"",
}) |gone| {
try testing.expect(!std.mem.containsAtLeast(u8, allocating.written(), 1, gone));
}
}
+157
View File
@@ -30,6 +30,8 @@ const dns_cache = @import("../cache/dns_cache.zig");
const dns_handler = @import("../server/handler.zig");
const disk_monitor = @import("../storage/disk_monitor.zig");
const dot_server = @import("../server/dot_server.zig");
const events_mod = @import("../storage/events.zig");
const history_mod = @import("../upstream/history.zig");
const http_util = @import("http_util.zig");
const logging = @import("../platform/logging.zig");
const pool_mod = @import("../upstream/pool.zig");
@@ -114,6 +116,15 @@ pub const UpstreamSample = struct {
success_rate: f32,
};
/// The diagnostics store, as one scrape sees it. Two gauges and a counter,
/// written out rather than reflected over a stats struct because they are not
/// all the same kind of number.
pub const DiagnosticsSample = struct {
active_warnings: u32,
active_errors: u32,
write_failures: u64,
};
/// Everything one scrape reports. A null section is a collaborator the state
/// does not have.
pub const Sample = struct {
@@ -125,6 +136,14 @@ pub const Sample = struct {
tracker: ?TrackerSample = null,
client_names: ?client_names.Resolver.Stats = null,
retention: ?retention_mod.Stats = null,
/// The upstream-history flush loop's counters (m26 ruling 7). Absent while
/// no accumulator is wired, like every other collaborator.
history: ?history_mod.Accumulator.Stats = null,
/// The diagnostics store's open episodes and its failed writes. Absent
/// while no store is wired, like every other collaborator — an operator
/// distinguishes "no series" from "zero episodes" through `/api/health`,
/// which says which of the two it is.
diagnostics: ?DiagnosticsSample = null,
blocklist: ?BlocklistSample = null,
disk: ?DiskSample = null,
/// One entry per enabled TLS endpoint (milestone-10 ruling 10). Rendered
@@ -199,6 +218,17 @@ pub fn collect(state: *server.WebState, io: std.Io, arena: Allocator) Allocator.
if (state.retention) |retention| sample.retention = retention.snapshotStats();
if (state.history) |history| sample.history = history.snapshotStats(io);
if (state.events) |store| {
const counts = store.activeCounts(io);
sample.diagnostics = .{
.active_warnings = counts.warnings,
.active_errors = counts.errors,
.write_failures = store.writeFailures(),
};
}
if (state.manager) |manager| {
const generation: ?u64 = if (manager.acquire(io)) |acquired| gen: {
defer acquired.release(io);
@@ -352,6 +382,57 @@ pub fn render(w: *std.Io.Writer, sample: Sample) std.Io.Writer.Error!void {
try counterGroup(w, "nxdns_retention_", "Query log retention counter", retention);
}
if (sample.history) |history| {
// Written out rather than reflected over `Accumulator.Stats`: three of
// its fields are counters, one is a gauge, and two — the drop watermark
// and the current flush state — are not exposition numbers at all.
try counter(
w,
"nxdns_upstream_history_flushes_total",
"Upstream history flush transactions that committed.",
history.flushes,
);
try counter(
w,
"nxdns_upstream_history_flush_failures_total",
"Upstream history flush attempts that failed; the rows are retried on the next pass.",
history.flush_failures,
);
try counter(
w,
"nxdns_upstream_history_rows_dropped_total",
"Upstream history minutes dropped because the accumulator was full.",
history.rows_dropped,
);
try gauge(
w,
"nxdns_upstream_history_pending",
"Upstream history minutes recorded but not yet flushed.",
history.pending,
);
}
if (sample.diagnostics) |diagnostics| {
try gauge(
w,
"nxdns_diagnostics_active_warnings",
"Operational event episodes open right now at warning severity.",
diagnostics.active_warnings,
);
try gauge(
w,
"nxdns_diagnostics_active_errors",
"Operational event episodes open right now at error severity.",
diagnostics.active_errors,
);
try counter(
w,
"nxdns_diagnostics_write_failures_total",
"Operational events dropped because the diagnostics database refused the write.",
diagnostics.write_failures,
);
}
if (sample.blocklist) |blocklist| {
try counter(
w,
@@ -602,8 +683,10 @@ fn writeLabelValue(w: *std.Io.Writer, value: []const u8) std.Io.Writer.Error!voi
// tests
// ---------------------------------------------------------------------------
const db = @import("../storage/db.zig");
const local_tables = @import("../server/local_tables.zig");
const logger_mod = @import("../storage/logger.zig");
const migrations = @import("../storage/migrations.zig");
const testing = std.testing;
/// A handler with no upstream reachable: every test here reads counters and
@@ -720,6 +803,80 @@ test "a full sample renders the whole exposition, byte for byte" {
));
}
test "the upstream-history family renders three counters and one gauge" {
const text = try renderToString(testing.allocator, .{
.history = .{
.flushes = 12,
.flush_failures = 2,
.rows_dropped = 5,
.pending = 3,
.last_drop_minute = 1_700_000_040,
.last_flush_failed = true,
},
});
defer testing.allocator.free(text);
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_upstream_history_flushes_total 12\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_upstream_history_flush_failures_total 2\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_upstream_history_rows_dropped_total 5\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "# TYPE nxdns_upstream_history_pending gauge\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_upstream_history_pending 3\n"));
// No accumulator is an absent family, not a family of zeros.
const bare = try renderToString(testing.allocator, .{});
defer testing.allocator.free(bare);
try testing.expect(!std.mem.containsAtLeast(u8, bare, 1, "nxdns_upstream_history_"));
}
test "the diagnostics family renders two gauges and one counter" {
const text = try renderToString(testing.allocator, .{
.diagnostics = .{ .active_warnings = 3, .active_errors = 1, .write_failures = 7 },
});
defer testing.allocator.free(text);
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "# TYPE nxdns_diagnostics_active_warnings gauge\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_diagnostics_active_warnings 3\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "# TYPE nxdns_diagnostics_active_errors gauge\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_diagnostics_active_errors 1\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "# TYPE nxdns_diagnostics_write_failures_total counter\n"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_diagnostics_write_failures_total 7\n"));
// No store is an absent family, not a family of zeros: "no episodes open"
// and "nothing is recording them" must not render the same.
const bare = try renderToString(testing.allocator, .{});
defer testing.allocator.free(bare);
try testing.expect(!std.mem.containsAtLeast(u8, bare, 1, "nxdns_diagnostics_"));
}
test "collect reads the diagnostics store's open episodes and failed writes" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try db.applyPragmas(&database, .{});
_ = try migrations.migrate(&database);
var store = try events_mod.Store.init(io, &database, 1000);
store.report(io, 1000, .disk_space, "data", "data", .warning, "low");
var state: server.WebState = .{ .gpa = testing.allocator, .events = &store };
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const sample = try collect(&state, io, arena.allocator());
try testing.expectEqual(@as(u32, 1), sample.diagnostics.?.active_warnings);
try testing.expectEqual(@as(u32, 0), sample.diagnostics.?.active_errors);
try testing.expectEqual(@as(u64, 0), sample.diagnostics.?.write_failures);
// A write that cannot land is counted here, through the production path
// rather than by poking the field.
try database.exec("DROP TABLE operational_events;");
store.report(io, 1100, .disk_space, "data", "data", .warning, "low");
const after = try collect(&state, io, arena.allocator());
try testing.expect(after.diagnostics.?.write_failures >= 1);
}
test "every HELP line has a TYPE line and a sample, and every sample a name" {
const text = try renderToString(testing.allocator, .{});
defer testing.allocator.free(text);
+294 -16
View File
@@ -256,6 +256,131 @@ paths:
"503":
$ref: "#/components/responses/Unavailable"
/api/diagnostics:
get:
summary: Operational event log
description: |
Failure episodes, newest first. One event is one subject failing
continuously: it opens on the first failure, counts repeats in
`occurrences`, and gets a `resolved_at` when the subject recovers. A
subject that fails again opens a new event rather than reopening the
old one. Keyset pagination — follow `next_before` until it is null.
parameters:
- name: state
in: query
schema: { type: string, enum: [active, resolved, all], default: all }
- name: severity
in: query
schema: { type: string, enum: [warning, error] }
- name: component
in: query
description: Matches the part of `code` before the dot, exactly.
schema: { type: string, maxLength: 64 }
- name: since
in: query
description: |
Unix seconds. With `until`, selects episodes overlapping the
window; an episode resolved exactly at `since` does not overlap.
schema: { type: integer }
- name: until
in: query
description: Unix seconds, exclusive.
schema: { type: integer }
- name: limit
in: query
schema: { type: integer, minimum: 1, maximum: 1000, default: 100 }
- name: before
in: query
description: Return events with id strictly below this cursor.
schema: { type: integer, minimum: 1 }
responses:
"200":
description: One page.
content:
application/json:
schema:
$ref: "#/components/schemas/DiagnosticsPage"
"400":
$ref: "#/components/responses/BadRequest"
"401":
$ref: "#/components/responses/Unauthorized"
"429":
$ref: "#/components/responses/RateLimited"
"500":
$ref: "#/components/responses/Internal"
"503":
$ref: "#/components/responses/Unavailable"
delete:
summary: Purge every resolved event
description: |
Deletes the resolved history and answers with how many rows went.
Active events are never touched, so clearing the page cannot lose an
episode that is still failing. Resolution stays automatic; this only
decides when the history disappears. A runtime action, served in file
mode too — the event log is not configuration.
responses:
"200":
description: How many resolved events were removed.
content:
application/json:
schema:
$ref: "#/components/schemas/DiagnosticsPurge"
"401":
$ref: "#/components/responses/Unauthorized"
"429":
$ref: "#/components/responses/RateLimited"
"500":
$ref: "#/components/responses/Internal"
"503":
$ref: "#/components/responses/Unavailable"
/api/diagnostics/{id}:
parameters:
- $ref: "#/components/parameters/RowId"
get:
summary: One operational event
description: |
404 for an id that never existed and for one retention has removed —
the API does not distinguish them.
responses:
"200":
description: The event.
content:
application/json:
schema:
$ref: "#/components/schemas/DiagnosticEvent"
"401":
$ref: "#/components/responses/Unauthorized"
"404":
$ref: "#/components/responses/NotFound"
"429":
$ref: "#/components/responses/RateLimited"
"500":
$ref: "#/components/responses/Internal"
"503":
$ref: "#/components/responses/Unavailable"
delete:
summary: Purge one resolved event
description: |
Deletes a resolved event. An event that is still active answers 409 —
an open episode is the current state of the box, not history — and an
id no row holds answers 404. A runtime action, served in file mode too.
responses:
"204":
description: Purged.
"401":
$ref: "#/components/responses/Unauthorized"
"404":
$ref: "#/components/responses/NotFound"
"409":
$ref: "#/components/responses/Conflict"
"429":
$ref: "#/components/responses/RateLimited"
"500":
$ref: "#/components/responses/Internal"
"503":
$ref: "#/components/responses/Unavailable"
/api/stats:
get:
summary: Totals for a period
@@ -342,7 +467,15 @@ paths:
/api/upstream/health:
get:
summary: Upstream pool health
summary: Upstream pool health for a period
description: |
Each upstream's live routing state beside its recorded outcomes over
the period's window, which is the same UTC-aligned window `/api/stats`
reports for that period. The outcome counts come from per-minute
history in the query log, not from process-lifetime counters, so they
scope to the period and survive a restart.
parameters:
- $ref: "#/components/parameters/Period"
responses:
"200":
description: Per-upstream state and the availability rollup.
@@ -350,10 +483,14 @@ paths:
application/json:
schema:
$ref: "#/components/schemas/UpstreamHealth"
"400":
$ref: "#/components/responses/BadRequest"
"401":
$ref: "#/components/responses/Unauthorized"
"429":
$ref: "#/components/responses/RateLimited"
"500":
$ref: "#/components/responses/Internal"
"503":
$ref: "#/components/responses/Unavailable"
@@ -1652,11 +1789,21 @@ components:
Health:
type: object
required: [status, disk, upstreams, queries_dropped, writer_failed, refreshes_gated, snapshot_generation]
required: [status, disk, upstreams, diagnostics, queries_dropped, writer_failed, refreshes_gated, snapshot_generation]
properties:
status:
type: string
enum: [ok, degraded]
diagnostics:
type: object
required: [state, active_warnings, active_errors]
properties:
state:
type: string
enum: [recording, unavailable]
description: unavailable when the event store failed to open or its writes are failing; either state degrades health.
active_warnings: { type: integer }
active_errors: { type: integer }
disk:
type: object
required: [state, free_bytes, db_bytes, log_bytes, sample_failures]
@@ -1748,6 +1895,92 @@ components:
nullable: true
description: Cursor for the next page; null on the last page.
DiagnosticEvent:
type: object
required: [id, code, component, subject, severity, first_seen, last_seen, occurrences, resolved_at, detail]
properties:
id: { type: integer }
code:
type: string
description: |
The failure kind, as `component.name`. One of a fixed set of
fifteen; new codes are added with new releases.
enum:
- disk.space
- disk.probe
- blocklist.refresh
- blocklist.snapshot
- blocklist.storage
- certificate.reload
- query_log.write
- query_log.maintenance
- query_log.recreated
- upstream_history.write
- upstream.exchange
- client_names.storage
- clients.storage
- listener.start
- configuration.load
component:
type: string
description: The part of `code` before the dot, repeated for filtering.
subject:
type: string
description: |
What failed, as a display name: a blocklist source name, an
endpoint, an operation. Redacted where it derives from a url; the
store's internal identity for the subject is never exposed.
severity:
type: string
enum: [warning, error]
first_seen:
type: integer
description: When this episode opened, unix seconds.
last_seen:
type: integer
description: The most recent failure of this episode, unix seconds.
occurrences:
type: integer
description: How many failures this episode has held; at least 1.
resolved_at:
type: integer
nullable: true
description: |
When the subject recovered, unix seconds. Null while the episode is
still open. A subject that fails again opens a new event rather than
reopening this one.
detail:
type: string
description: The last error of this episode, truncated to 512 bytes.
DiagnosticsPage:
type: object
required: [events, next_before, active]
properties:
events:
type: array
items:
$ref: "#/components/schemas/DiagnosticEvent"
next_before:
type: integer
nullable: true
description: Cursor for the next page; null on the last page.
active:
type: object
required: [warnings, errors]
description: Episodes open right now, whatever this page filtered to.
properties:
warnings: { type: integer }
errors: { type: integer }
DiagnosticsPurge:
type: object
required: [purged]
properties:
purged:
type: integer
description: How many resolved events the purge removed; zero when there were none.
StatsTotals:
type: object
required: [period, since, until, queries, blocked, cached, clients, avg_response_time_us]
@@ -1826,28 +2059,71 @@ components:
type: string
nullable: true
UpstreamPeriodStats:
type: object
required: [attempts, successes, failures, success_rate, last_failure_at, last_failure_error]
properties:
attempts:
type: integer
description: Exchanges recorded against this upstream inside the window.
successes: { type: integer }
failures: { type: integer }
success_rate:
type: number
nullable: true
description: >
`successes / attempts`, from 0 to 1. Null when `attempts` is 0: no
observations is not perfect reliability.
last_failure_at:
type: integer
nullable: true
description: >
The newest failure inside the window, unix seconds. Null when the
window holds no failure, even if the upstream failed before it.
last_failure_error:
type: string
nullable: true
description: The error name belonging to `last_failure_at`; null exactly when it is.
UpstreamHealth:
type: object
required: [upstreams, available, total]
required: [period, since, until, available, total, complete, upstreams]
properties:
period:
type: string
enum: [1h, 24h, 7d, 30d]
since:
type: integer
description: Window start, unix seconds, inclusive.
until:
type: integer
description: Window end, unix seconds, exclusive.
available:
type: integer
description: How many upstreams the pool would route to right now.
total: { type: integer }
complete:
type: boolean
description: >
No capacity drops known in this process within the selected window;
up to about a minute of the newest outcomes may not have flushed
yet, and outcomes lost in an unclean shutdown are not detectable.
upstreams:
type: array
description: The upstreams configured now; a deleted upstream's history is not returned.
items:
type: object
required: [url, enabled, available, consecutive_failures, total_successes, total_failures, success_rate, last_error]
required: [url, enabled, available, period]
properties:
url: { type: string }
enabled: { type: boolean }
available: { type: boolean }
consecutive_failures: { type: integer }
total_successes: { type: integer }
total_failures: { type: integer }
success_rate: { type: number }
last_error:
type: string
description: Empty when the upstream never failed.
available: { type: integer }
total: { type: integer }
enabled:
type: boolean
description: Live configuration, not history.
available:
type: boolean
description: Live state, not history; false while the upstream is backing off.
period:
$ref: "#/components/schemas/UpstreamPeriodStats"
Group:
type: object
@@ -2227,13 +2503,14 @@ components:
enum: [strip, forward]
logging:
type: object
required: [level, retention_days, query_log_buffer_max, hide_domains, hide_client_ips, output, file_path, max_size_mb, max_files]
required: [level, retention_days, query_log_buffer_max, query_log_flush_interval_s, hide_domains, hide_client_ips, output, file_path, max_size_mb, max_files]
properties:
level:
type: string
enum: [error, warn, info, debug]
retention_days: { type: integer }
query_log_buffer_max: { type: integer }
query_log_flush_interval_s: { type: integer }
hide_domains: { type: boolean }
hide_client_ips: { type: boolean }
output:
@@ -2375,6 +2652,7 @@ components:
level: { type: string }
retention_days: { type: integer }
query_log_buffer_max: { type: integer }
query_log_flush_interval_s: { type: integer }
hide_domains: { type: boolean }
hide_client_ips: { type: boolean }
output: { type: string }
+12 -1
View File
@@ -36,6 +36,7 @@ const auth = @import("handlers/auth.zig");
const blocklists = @import("handlers/blocklists.zig");
const certs = @import("handlers/certs.zig");
const clients = @import("handlers/clients.zig");
const diagnostics = @import("handlers/diagnostics.zig");
const groups = @import("handlers/groups.zig");
const health = @import("handlers/health.zig");
const live = @import("handlers/live.zig");
@@ -71,6 +72,14 @@ pub const table: []const router.RouteInfo = &.{
.{ .method = .GET, .pattern = "/api/lookup", .auth = .session, .policy = .read, .handler = lookup.handle },
.{ .method = .GET, .pattern = "/api/upstream/health", .auth = .session, .policy = .read, .handler = upstream_health.handle },
// Diagnostics: the operational event log (milestone 27). The two purges are
// `runtime_action` — the event log is runtime state no configuration file
// declares, so file authority has nothing to say about deleting from it.
.{ .method = .GET, .pattern = "/api/diagnostics", .auth = .session, .policy = .read, .handler = diagnostics.list },
.{ .method = .DELETE, .pattern = "/api/diagnostics", .auth = .session, .policy = .runtime_action, .handler = diagnostics.purgeAll },
.{ .method = .GET, .pattern = "/api/diagnostics/{id}", .auth = .session, .policy = .read, .handler = diagnostics.get },
.{ .method = .DELETE, .pattern = "/api/diagnostics/{id}", .auth = .session, .policy = .runtime_action, .handler = diagnostics.purge },
// Groups.
.{ .method = .GET, .pattern = "/api/groups", .auth = .session, .policy = .read, .handler = groups.list },
.{ .method = .POST, .pattern = "/api/groups", .auth = .session, .policy = .config_write, .handler = groups.create },
@@ -143,7 +152,7 @@ const std = @import("std");
const testing = std.testing;
test "the table carries every endpoint of the milestone" {
try testing.expectEqual(@as(usize, 56), table.len);
try testing.expectEqual(@as(usize, 60), table.len);
}
test "no two entries claim the same method and pattern" {
@@ -231,6 +240,8 @@ test "the runtime actions are exactly ruling 7's list" {
"POST /api/auth/login",
"POST /api/auth/logout",
"POST /api/blocklists/update",
"DELETE /api/diagnostics",
"DELETE /api/diagnostics/{id}",
"DELETE /api/clients/{id}",
"POST /api/pause",
"POST /api/certs/reload",
+11
View File
@@ -32,6 +32,7 @@ const disk_monitor = @import("../storage/disk_monitor.zig");
const dns_handler = @import("../server/handler.zig");
const doh_server = @import("../server/doh_server.zig");
const dot_server = @import("../server/dot_server.zig");
const events_mod = @import("../storage/events.zig");
const http_util = @import("http_util.zig");
const listener_core = @import("../server/listener.zig");
const local_tables_mod = @import("../server/local_tables.zig");
@@ -39,6 +40,7 @@ const logger_mod = @import("../storage/logger.zig");
const manager_mod = @import("../filter/manager.zig");
const model = @import("../config/model.zig");
const pause_mod = @import("../server/pause.zig");
const history_mod = @import("../upstream/history.zig");
const pool_mod = @import("../upstream/pool.zig");
const query_sink = @import("../server/query_sink.zig");
const retention_mod = @import("../storage/retention.zig");
@@ -135,6 +137,10 @@ pub const WebState = struct {
client_names: ?*client_names.Resolver = null,
manager: ?*manager_mod.Manager = null,
pool: ?*pool_mod.Pool = null,
/// The upstream-outcome accumulator, for `metrics.collect` and the
/// `/api/health` rollup (m26 ruling 7). The ranged endpoint reads the
/// flushed rows through `querylog_db`, not through this.
history: ?*history_mod.Accumulator = null,
monitor: ?*disk_monitor.Monitor = null,
/// The local records and forward zones the DNS path reads. The
/// local-records and forward-zones handlers rebuild and swap them
@@ -178,6 +184,11 @@ pub const WebState = struct {
/// concurrent writes would misread each other's row counts.
config_lock: std.Io.Mutex = .init,
querylog_db: ?*db.Db = null,
/// The diagnostics event store, which owns a third connection of its own
/// and serializes every access — read and write — through its mutex. Null
/// when `Store.init` failed, which `/api/health` reports as `unavailable`
/// and treats as degraded.
events: ?*events_mod.Store = null,
version: []const u8 = "",
/// The `--admin-dev` asset directory, read by the dev-mode fallback. Empty
+200
View File
@@ -31,6 +31,7 @@ const auth = @import("auth.zig");
const clients_repo = @import("../storage/repositories/clients_repo.zig");
const db = @import("../storage/db.zig");
const dns_handler = @import("../server/handler.zig");
const events_mod = @import("../storage/events.zig");
const fetcher = @import("../filter/fetcher.zig");
const groups_repo = @import("../storage/repositories/groups_repo.zig");
const header = @import("../dns/header.zig");
@@ -58,6 +59,7 @@ const upstreams_repo = @import("../storage/repositories/upstreams_repo.zig");
const handlers_blocklists = @import("handlers/blocklists.zig");
const handlers_certs = @import("handlers/certs.zig");
const handlers_diagnostics = @import("handlers/diagnostics.zig");
const handlers_health = @import("handlers/health.zig");
const handlers_live = @import("handlers/live.zig");
const handlers_lookup = @import("handlers/lookup.zig");
@@ -277,6 +279,10 @@ const Env = struct {
tmp: testing.TmpDir,
config_db: db.Db,
querylog_db: db.Db,
/// The diagnostics store's own connection, as in production: the store
/// serializes every access through its mutex and shares it with nobody.
events_db: db.Db,
events_store: events_mod.Store,
http_client: std.http.Client,
transfer_buf: [fetcher.min_transfer_buf]u8,
redirect_buf: [fetcher.redirect_buffer_len]u8,
@@ -317,6 +323,13 @@ const Env = struct {
try self.querylog_db.exec(querylog_schema.ddl);
try seedQueryLog(&self.querylog_db);
self.events_db = try db.Db.open(":memory:", .{ .mode = .memory });
errdefer self.events_db.close();
try db.applyPragmas(&self.events_db, .{});
_ = try migrations.migrate(&self.events_db);
self.events_store = try events_mod.Store.init(ioh, &self.events_db, seeded_now);
seedEvents(ioh, &self.events_store);
// Real fetcher wiring; nothing in this suite downloads (the one
// refreshAll in the contract walk runs with zero source rows).
self.http_client = .{ .allocator = gpa, .io = ioh };
@@ -388,6 +401,7 @@ const Env = struct {
.hub = self.hub,
.config_db = &self.config_db,
.querylog_db = &self.querylog_db,
.events = &self.events_store,
.version = "w10-test",
.started_unix = std.Io.Clock.real.now(ioh).toSeconds(),
.fallback = options.fallback,
@@ -419,6 +433,7 @@ const Env = struct {
self.limiter.deinit();
self.mgr.deinit(ioh);
self.http_client.deinit();
self.events_db.close();
self.querylog_db.close();
self.config_db.close();
self.tmp.cleanup();
@@ -480,6 +495,21 @@ fn seedQueryLog(database: *db.Db) !void {
}
}
/// A fixed instant, like every other seeded timestamp here: the contract
/// samples are byte-compared, so nothing the walk writes may come from a clock.
const seeded_now: i64 = 1_787_118_000;
/// One active episode and one resolved one, so `/api/diagnostics` answers with
/// both states and the committed contract sample describes a real page rather
/// than an empty one.
fn seedEvents(io: std.Io, store: *events_mod.Store) void {
store.report(io, seeded_now, .blocklist_refresh, "https://lists.example/ads.txt", "StevenBlack", .warning, "download failed: ConnectionTimedOut");
store.report(io, seeded_now + 300, .blocklist_refresh, "https://lists.example/ads.txt", "StevenBlack", .warning, "download failed: ConnectionTimedOut");
store.report(io, seeded_now + 60, .upstream_history_write, "history", "history", .warning, "Busy");
store.resolve(io, seeded_now + 120, .upstream_history_write, "history");
}
// ---------------------------------------------------------------------------
// the contract table (ruling 23)
// ---------------------------------------------------------------------------
@@ -568,6 +598,7 @@ const SettingsView = struct {
level: []const u8,
retention_days: u16,
query_log_buffer_max: u32,
query_log_flush_interval_s: u16,
hide_domains: bool,
hide_client_ips: bool,
output: []const u8,
@@ -627,6 +658,15 @@ const contract = [_]Contract{
.{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .target = "/api/stats/timeseries?period=1h", .status = 200, .check = jsonShape(handlers_stats.TimeseriesBody) },
.{ .method = .GET, .pattern = "/api/upstream/health", .auth = .session, .policy = .read, .target = "/api/upstream/health", .status = 200, .check = jsonShape(handlers_upstream_health.Body) },
// Diagnostics. The seeded store holds one active episode (id 1) and one
// resolved one, so both the page and the detail answer with real rows.
.{ .method = .GET, .pattern = "/api/diagnostics", .auth = .session, .policy = .read, .target = "/api/diagnostics?limit=10", .status = 200, .check = jsonShape(events_mod.EventsPage) },
.{ .method = .GET, .pattern = "/api/diagnostics/{id}", .auth = .session, .policy = .read, .target = "/api/diagnostics/1", .status = 200, .check = jsonShape(events_mod.Event) },
// The purges follow the reads: id 2 is the seeded resolved episode, and the
// sweep after it takes whatever resolved history is left (none).
.{ .method = .DELETE, .pattern = "/api/diagnostics/{id}", .auth = .session, .policy = .runtime_action, .target = "/api/diagnostics/2", .status = 204, .kind = .none },
.{ .method = .DELETE, .pattern = "/api/diagnostics", .auth = .session, .policy = .runtime_action, .target = "/api/diagnostics", .status = 200, .check = jsonShape(handlers_diagnostics.PurgeResult) },
// Groups. The migrated schema seeds `default` as id 1; the POST creates
// id 2, which the delete at the end of the walk removes.
.{ .method = .GET, .pattern = "/api/groups", .auth = .session, .policy = .read, .target = "/api/groups", .status = 200, .check = jsonShape(GroupsList) },
@@ -1000,6 +1040,157 @@ fn fileModeClasses(io: std.Io, env: *Env) anyerror!void {
try conn.request("POST", "/api/certs/reload", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), response.status);
// Diagnostics are runtime state, not configuration: purging resolved
// history is served under file authority like any other runtime action.
try conn.request("DELETE", "/api/diagnostics", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), response.status);
try testing.expectEqualStrings("{\"purged\":1}", response.body);
try conn.request("DELETE", "/api/diagnostics/1", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 409), response.status);
}
fn diagnosticsRejections(io: std.Io, env: *Env) anyerror!void {
var body_buf: [8192]u8 = undefined;
var conn: Conn = undefined;
try conn.connect(io, env.addr);
defer conn.close(io);
// Every bad parameter is a 400 whose message names the parameter, rather
// than a filter silently dropped — which would answer a question the client
// did not ask.
const bad = [_]struct { target: []const u8, needle: []const u8 }{
.{ .target = "/api/diagnostics?state=open", .needle = "state" },
.{ .target = "/api/diagnostics?severity=info", .needle = "severity" },
.{ .target = "/api/diagnostics?since=yesterday", .needle = "since" },
.{ .target = "/api/diagnostics?until=", .needle = "until" },
.{ .target = "/api/diagnostics?limit=0", .needle = "limit" },
.{ .target = "/api/diagnostics?limit=1001", .needle = "limit" },
.{ .target = "/api/diagnostics?before=0", .needle = "before" },
};
for (bad) |case| {
try conn.request("GET", case.target, null, null);
const response = try conn.receive(&body_buf);
errdefer std.debug.print("{s}: {d} {s}\n", .{ case.target, response.status, response.body });
try testing.expectEqual(@as(u16, 400), response.status);
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, case.needle));
}
// The resolved episode the seed left behind is reachable by id, and an id
// nothing holds is a 404 rather than an empty object.
try conn.request("GET", "/api/diagnostics?state=resolved", null, null);
var response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), response.status);
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "upstream_history.write"));
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "\"active\":{\"warnings\":1,\"errors\":0}"));
try conn.request("GET", "/api/diagnostics/999999", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 404), response.status);
}
fn diagnosticsPurge(io: std.Io, env: *Env) anyerror!void {
var body_buf: [8192]u8 = undefined;
var conn: Conn = undefined;
try conn.connect(io, env.addr);
defer conn.close(io);
// Row 1 is the seeded active episode: still the state of the box, so the
// purge is refused with a message that says what would change that.
try conn.request("DELETE", "/api/diagnostics/1", null, null);
var response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 409), response.status);
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "still active"));
try conn.request("DELETE", "/api/diagnostics/999999", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 404), response.status);
// Row 2 is the seeded resolved episode.
try conn.request("DELETE", "/api/diagnostics/2", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 204), response.status);
try testing.expectEqualStrings("", response.body);
// Gone is a different answer from still open, even for a row that existed a
// moment ago.
try conn.request("DELETE", "/api/diagnostics/2", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 404), response.status);
// Nothing resolved is left, and the sweep says so rather than failing.
try conn.request("DELETE", "/api/diagnostics", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), response.status);
try testing.expectEqualStrings("{\"purged\":0}", response.body);
// The active episode survived every one of those, counts included.
try conn.request("GET", "/api/diagnostics", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), response.status);
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "blocklist.refresh"));
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "\"active\":{\"warnings\":1,\"errors\":0}"));
try testing.expect(!std.mem.containsAtLeast(u8, response.body, 1, "upstream_history.write"));
}
test "W10 milestone 27: a purge takes resolved events only, and says which of the three answers it gave" {
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
var env = try Env.create(gpa, .{});
defer env.destroy();
try bounded(env.io(), default_budget, diagnosticsPurge, .{ env.io(), env });
}
fn diagnosticsPurgeAll(io: std.Io, env: *Env) anyerror!void {
var body_buf: [8192]u8 = undefined;
var conn: Conn = undefined;
try conn.connect(io, env.addr);
defer conn.close(io);
// A second resolved episode, so the count the sweep reports is a number it
// had to compute rather than the one row the seed leaves.
env.events_store.reportResolved(io, seeded_now, .query_log_recreated, "one-shot", "corrupt", .warning, "aside");
try conn.request("DELETE", "/api/diagnostics", null, null);
var response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), response.status);
try testing.expectEqualStrings("{\"purged\":2}", response.body);
try conn.request("GET", "/api/diagnostics?state=resolved", null, null);
response = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), response.status);
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "\"events\":[]"));
// And the episode that is still failing is untouched: the operator clearing
// the page cannot lose what is still true.
try conn.request("GET", "/api/diagnostics?state=active", null, null);
response = try conn.receive(&body_buf);
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "blocklist.refresh"));
}
test "W10 milestone 27: purging all resolved events counts them and leaves the active ones" {
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
var env = try Env.create(gpa, .{});
defer env.destroy();
try bounded(env.io(), default_budget, diagnosticsPurgeAll, .{ env.io(), env });
}
test "W10 milestone 27: every diagnostics filter names itself in a 400, and an unknown id is a 404" {
if (!build_options.integration) return error.SkipZigTest;
const gpa = testing.allocator;
var env = try Env.create(gpa, .{});
defer env.destroy();
try bounded(env.io(), default_budget, diagnosticsRejections, .{ env.io(), env });
}
test "W10 milestone 20: file authority rejects configuration writes and spares the rest" {
@@ -2197,6 +2388,15 @@ const contract_sample_walk = [_]ContractSample{
.{ .name = "login", .ts_type = "LoginResponse", .method = "POST", .target = "/api/auth/login", .body = "{\"password\":\"\"}", .status = 200 },
.{ .name = "logout", .ts_type = "LogoutResponse", .method = "POST", .target = "/api/auth/logout", .body = "{}", .status = 200 },
// Diagnostics, ahead of every write below: the seeded store holds one
// active episode (id 1) and one resolved one, and a later pass that
// reported an event of its own would move the page under the golden.
.{ .name = "get_diagnostics", .ts_type = "DiagnosticsPage", .method = "GET", .target = "/api/diagnostics?limit=10", .status = 200 },
.{ .name = "get_diagnostic", .ts_type = "DiagnosticEvent", .method = "GET", .target = "/api/diagnostics/1", .status = 200 },
// The sweep runs after both reads and takes the seeded resolved episode;
// the per-id purge answers 204, which has no body to sample.
.{ .name = "purge_diagnostics", .ts_type = "DiagnosticsPurge", .method = "DELETE", .target = "/api/diagnostics", .status = 200 },
// Blocklists. The row is created disabled so the refresh below has a status
// to report and still downloads nothing.
.{ .name = "create_blocklist", .ts_type = "BlocklistEcho", .method = "POST", .target = "/api/blocklists", .body = "{\"url\":\"https://lists.example/ads.txt\",\"name\":\"ads\",\"enabled\":false}", .status = 201 },