build: bump version to 0.0.23

AGENTS.md

@@ -49,6 +49,16 @@ That function returns `null` for every input. `fetchAdd` is the only trigger: a
 
 `Owner.published` in `src/upstream/owner.zig` is a plain `u64` under the owner's mutex for this reason. Do not "modernize" it to `std.atomic.Value(u64)`.
 
+## Testing the NixOS module
+
+`nix/module.nix` (`services.nxdns`) has two acceptance checks in `nix/tests.nix`, and no CI job runs them: the VM test needs KVM, and both download a nixpkgs closure that the shared Gitea runner must not fetch on every push. The cut's `nix flake check --no-build` evaluates them, which catches a module that no longer evaluates, but builds nothing. After any change to `nix/module.nix`, `nix/tests.nix`, the package in `flake.nix` or `deploy/systemd/nxdns.service`, build both on a host with KVM before the commit:
+
+```
+nix build --no-link -L .#checks.x86_64-linux.nixos-module .#checks.x86_64-linux.nixos-module-rejects-config
+```
+
+The checks run the package the flake pins, which is the last release, so a change to `deploy/systemd/nxdns.service` reaches the VM test only after the release that ships it.
+
 ## Regenerating the contract samples
 
 `admin/src/lib/contractSamples.gen.ts` is a committed golden of canonicalized API responses, byte-compared against the live server by a `-Dintegration` test and type-checked by `tsc`. After a deliberate API contract change, regenerate it with:

CHANGELOG.md

@@ -4,6 +4,42 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
 
 Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
 
+## [0.0.23] - 2026-10-20
+
+### Fixed
+
+- **A blackhole route to an upstream no longer stops failover.** On a route like that, `connect()` fails with an errno the runtime has no name for, and nxdns took it for a fault of its own: the query got SERVFAIL at once, the next upstream was never tried, and nothing was charged to the unreachable one. A connect failure of that kind now counts against that upstream as `ConnectFailed (cause Unexpected)`, and the query fails over.
+- **The Overview charts no longer draw the current bucket as a fall in traffic.** Every Overview window ends at the end of the bucket that holds the present moment, so its last bucket is still counting. Queries over time draws the curve into it dashed over a fainter fill, and Client activity draws its column faint inside a dashed outline. Both legends name "Still counting", and the tooltip and the chart's data table give that bucket's time as "<time> · still counting".
+- **The query detail page names the device.** The Client fact showed only the address, while the Activity list showed "iPhone (192.168.50.52)" for the same query. It now uses the list's label. The query log records only the address, so the record's intro says the device name is the one the address has today.
+
+### Added
+
+- **A NixOS module.** The flake exports `nixosModules.default`, which adds `services.nxdns` with four options: `enable`, `package`, `configFile` and `openFirewall`. It runs the release's own `nxdns.service` with a drop-in for file mode, checks `configFile` with the pinned binary's `nxdns import` at build time so a configuration that does not parse, validate or reconcile fails the build, and restarts the service when only the configuration changes. The flake's package now also installs `lib/systemd/system/nxdns.service`. See "Install nxdns with Nix".
+- **`nxdns_upstream_local_resource_total`** counts exchanges that ended on this process's own resources (memory, file descriptors, an unnamed system error) before any upstream answered. Until now, such a SERVFAIL left no record of its cause anywhere.
+
+### Changed
+
+- **The compose file names its own release.** `deploy/docker/compose.yaml` defaults its image to the release it is tagged with instead of `:latest`, so `docker compose up -d` with the file from a tag runs that tag's image, and the install and upgrade steps no longer set `NXDNS_VERSION`. The variable still picks another release, and `NXDNS_IMAGE` still names a local image. Compose files from `v0.0.22` and earlier keep the `:latest` default.
+- **A DoH upstream that answers with an error status says which.** The Diagnostics episode and `/api/health` read `HttpStatus (status 505 HTTP Version Not Supported)` instead of `HttpStatus (cause HttpStatus)`. A 505 is a server that requires HTTP/2, which nxdns does not speak.
+
+## [0.0.22] - 2026-10-19
+
+### Added
+
+- **`/metrics` reports the most concurrent work the server has seen.** Five gauges hold peaks since start: `nxdns_dns_in_flight_peak` (queries handled at once, every listener together), `nxdns_tcp_server_connections_peak`, `nxdns_dot_server_connections_peak` and `nxdns_doh_server_connections_peak` (connections one listener held at once), and `nxdns_querylog_queue_peak` (entries waiting for the query log writer at once). The memory budget work sizes its limits from them; "Measure performance" shows how to read them beside the process's resident memory, thread count and worker stacks.
+
+## [0.0.21] - 2026-09-12
+
+### Changed
+
+- **An upstream warning means the pool stopped trusting the endpoint.** A Diagnostics episode opens when an upstream reaches the health failure threshold and closes on the success that clears it, instead of one card per lost exchange. Failures during backoff raise the card's occurrence count. The episode detail and the `nxdns check` report name the concrete cause behind the classification, `SendFailed (cause BrokenPipe)`.
+
+## [0.0.20] - 2026-09-12
+
+### Fixed
+
+- **A blocklist source no longer fails on a stale pooled connection.** The fetcher reused a keep-alive connection from an earlier pass that the server had closed, and the standard client never retries one; every later download from that host failed with `HttpConnectionClosing` at 0 ms. Downloads now send `connection: close` and never enter the pool.
+
 ## [0.0.19] - 2026-09-12
 
 ### Changed

PLAN.md

@@ -37,9 +37,10 @@ Serves a household LAN (≈2–20 devices). Portfolio-grade public repo with ext
 - Partial-segment wildcards (`ads*.example.com`) as a rule kind. The regex kind covers the need without a second globbing dialect.
 - DHCP server.
 - DNSSEC validation (DO bit passthrough only).
-- DoQ (QUIC), HTTP/2 upstream transport.
+- DoQ (DNS over QUIC). Out of scope until the Zig stdlib, or an equivalent dependency, provides QUIC; neither the stdlib nor mbedTLS does at 0.16.0. This is the one conditional entry in this list.
+- HTTP/2 upstream transport.
+- DNSCrypt. It is older than DoH and DoT and not an IETF standard, and DoH and DoT already cover encrypted DNS.
 - Clustering / distributed state.
-- Project website. The repository, its README and `docs/` are the whole published surface.
 
 ---
 

README.md

@@ -28,13 +28,11 @@ One static Zig binary. SQLite holds the state and the query log, which the web U
 
 ## Install
 
-**No release exists yet.** This repository has no tags, nothing has been published to <https://git.mial.net/mokhtar/nxdns/releases>, and no container image has been pushed. Every release URL on this page and in the how-to guides is a 404 today, and `docker pull` finds nothing. Until the first tag ships, building from source is the only way to get nxdns.
+Releases are published at <https://git.mial.net/mokhtar/nxdns/releases>, and the container image at `git.mial.net/mokhtar/nxdns`, which pulls without an account. Each tag publishes five assets — two static musl tarballs (`nxdns-<version>-x86_64-linux-musl.tar.gz`, `nxdns-<version>-aarch64-linux-musl.tar.gz`), `IMAGE-DIGEST.txt` naming the multi-architecture container image by digest, `SHA256SUMS.txt` covering those three files, and `SHA256SUMS.txt.asc`, a detached OpenPGP signature over the checksum file. Verify what you downloaded before you run it: [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md), which also says what that signature does and does not prove.
-
-What a tag will publish, once one exists: five assets — two static musl tarballs (`nxdns-<version>-x86_64-linux-musl.tar.gz`, `nxdns-<version>-aarch64-linux-musl.tar.gz`), `IMAGE-DIGEST.txt` naming the multi-architecture container image by digest, `SHA256SUMS.txt` covering those three files, and `SHA256SUMS.txt.asc`, a detached OpenPGP signature over the checksum file. Verify what you downloaded before you run it: [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md), which also says what that signature does and does not prove.
 
 ## Quickstart (docker compose)
 
-Write a minimal configuration and start the published image. This is what the first release will make possible; it does not work today, because there is no image in the registry to pull:
+Write a minimal configuration and start the published image:
 
 ```sh
 cd deploy/docker
@@ -46,14 +44,14 @@ cat > etc-nxdns/config.zon <<'EOF'
     .web = .{ .password = "choose-a-real-password" },
 }
 EOF
-NXDNS_VERSION=<version> docker compose up -d
+docker compose up -d
 ```
 
-The compose file defaults to `:latest`; pin a version for anything you intend to keep running. To run it before a release exists, build the image yourself and name it — `NXDNS_IMAGE=nxdns docker compose up -d` — as [docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md) describes. DNS is on port 53, the web UI on <http://localhost:8080>.
+The compose file names the release it belongs to, so a checkout of a release tag runs that release's image. `NXDNS_VERSION=<version> docker compose up -d` runs another release. To run an image you built yourself, name it — `NXDNS_IMAGE=nxdns docker compose up -d` — as [docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md) describes. DNS is on port 53, the web UI on <http://localhost:8080>.
 
 The compose file runs `nxdns run --config=/etc/nxdns/config.zon`, which makes that file the configuration: every start reconciles the database onto it, and the UI refuses configuration edits. Edit the file and restart to change anything. Drop the `command:` line to run bare `nxdns run` instead, where the database is the configuration and changes go through the UI, the API, or `nxdns export` / `nxdns import` — the packaged systemd unit does that. Which mode is live is printed at every start (`authority: database` / `authority: file (<path>)`); see [docs/explanation/configuration-model.md](docs/explanation/configuration-model.md).
 
-Full install instructions, including the systemd path and the Pi 5 recipe, are in [docs/how-to/install-with-systemd.md](docs/how-to/install-with-systemd.md) and [docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md).
+Full install instructions, including the systemd path and the Pi 5 recipe, are in [docs/how-to/install-with-systemd.md](docs/how-to/install-with-systemd.md), [docs/how-to/install-with-docker.md](docs/how-to/install-with-docker.md) and, for NixOS, [docs/how-to/install-with-nix.md](docs/how-to/install-with-nix.md).
 
 ## Building from source
 

admin/src/features/activity/ActivityDetailPage.test.tsx

@@ -1,4 +1,5 @@
 import { render, screen, waitFor, within } from "@testing-library/react";
+import * as stylex from "@stylexjs/stylex";
 import { QueryClientProvider } from "@tanstack/react-query";
 import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
 import { AuthProvider } from "@/auth/store";
@@ -7,6 +8,7 @@ import { createAppRouter } from "@/routes";
 import type { QueryDetail } from "@/lib/types";
 import { provenance } from "@/features/provenance/provenanceFixture";
 import { health } from "@/lib/healthFixture";
+import { styles as shared } from "@/ui/styles";
 
 function detail(id: number, sections: Parameters<typeof provenance>[0] = {}): QueryDetail {
 	return { id, ...provenance(sections) };
@@ -282,24 +284,40 @@ async function relatedText(expected: string) {
 	await waitFor(() => expect(related.textContent?.replace(/\s+/g, " ")).toContain(expected));
 }
 
-test("the record keeps the address the query came from, and Related carries the name it has now", async () => {
+/**
+ * The Activity list labels a client "Kids iPad (192.0.2.12)"; the detail page
+ * labels the same query the same way. The address is the recorded fact, and the
+ * intro says once that the device name is today's.
+ */
+test("the client is the device name with its address, as in the Activity list", async () => {
 	responses["/api/queries/12"] = detail(12, { request: { domain: "shop.example", client: "192.0.2.12" } });
 	responses["/api/clients"] = clientList({ ip: "192.0.2.12", name: "Kids iPad", learned_name: "ipad.lan" });
 	renderDetail(12);
 
 	await screen.findByRole("heading", { name: "shop.example" });
-	await relatedText("The client list currently names 192.0.2.12 “Kids iPad”.");
+	await waitFor(() => expect(factValue("Client")).toBe("Kids iPad (192.0.2.12)"));
-	expect(factValue("Client")).toBe("192.0.2.12");
+	expect(screen.getByText(/The device name is the one its address has today\./)).toBeTruthy();
 });
 
-test("a learned name is told as the reverse-DNS lookup it is, never as a recorded fact", async () => {
+test("an address the client list does not know stays a bare address", async () => {
+	responses["/api/queries/14"] = detail(14, { request: { client: "192.0.2.14" } });
+	responses["/api/clients"] = clientList({ ip: "192.0.2.99", name: "Elsewhere", learned_name: "" });
+	renderDetail(14);
+
+	await screen.findByRole("heading", { name: "example.com" });
+	await relatedText("These read the current configuration");
+	expect(factValue("Client")).toBe("192.0.2.14");
+});
+
+test("a learned name is drawn as learned, as in the Activity list", async () => {
 	responses["/api/queries/13"] = detail(13, { request: { client: "192.0.2.13" } });
 	responses["/api/clients"] = clientList({ ip: "192.0.2.13", name: "", learned_name: "printer.lan" });
 	renderDetail(13);
 
 	await screen.findByRole("heading", { name: "example.com" });
-	await relatedText("Reverse DNS currently resolves 192.0.2.13 to printer.lan.");
+	await waitFor(() => expect(factValue("Client")).toBe("printer.lan (192.0.2.13)"));
-	expect(factValue("Client")).toBe("192.0.2.13");
+	const label = screen.getByText("Client").nextElementSibling?.firstElementChild;
+	expect(label?.className).toBe(stylex.props(shared.learnedName).className);
 });
 
 test("a row retention has pruned explains the 404 and keeps the way back to the log", async () => {

admin/src/features/activity/ProvenanceDetail.tsx

@@ -12,7 +12,7 @@ import type { ReactNode } from "react";
 import * as stylex from "@stylexjs/stylex";
 import { formatMicros, formatTime } from "@/lib/format";
 import type { PolicyReason, Provenance } from "@/lib/types";
-import { clientLabel, useClientNames } from "@/features/clients/clientNames";
+import { ClientName, useClientNames } from "@/features/clients/clientNames";
 import {
 	policyActionLabel,
 	policyReasonLabel,
@@ -178,7 +178,6 @@ interface Props {
 export default function ProvenanceDetail({ provenance, persistedId, relatedActions }: Props) {
 	const { request, group, policy, rewrites, route, response } = provenance;
 	const clientNames = useClientNames();
-	const currentClient = clientLabel(request.client, clientNames);
 
 	return (
 		<>
@@ -192,7 +191,8 @@ export default function ProvenanceDetail({ provenance, persistedId, relatedActio
 			<div {...stylex.props(styles.record)}>
 				<p {...stylex.props(styles.recordNote)}>
 					What was recorded when this query was answered. Group and blocklist names are the ones in force at
-					that moment; they may have been renamed or deleted since.
+					that moment; they may have been renamed or deleted since. The device name is the one its address has
+					today.
 					{persistedId === null &&
 						" This is the event as it was streamed; the query log may not have written it yet."}
 				</p>
@@ -203,7 +203,7 @@ export default function ProvenanceDetail({ provenance, persistedId, relatedActio
 						<Mono>{request.domain}</Mono>
 					</Fact>
 					<Fact label="Client">
-						<Mono>{request.client}</Mono>
+						<ClientName ip={request.client} names={clientNames} />
 					</Fact>
 					<Fact label="Type">{qtypeName(request.qtype)}</Fact>
 					<Fact label="Class">{qclassName(request.qclass)}</Fact>
@@ -284,20 +284,6 @@ export default function ProvenanceDetail({ provenance, persistedId, relatedActio
 				<p {...stylex.props(styles.relatedNote)}>
 					These read the current configuration, which may no longer be the one that decided this query.
 				</p>
-				{currentClient !== null && (
-					<p {...stylex.props(styles.relatedNote)}>
-						{currentClient.learned ? (
-							<>
-								Reverse DNS currently resolves <Mono>{request.client}</Mono> to{" "}
-								<Mono>{currentClient.text}</Mono>.
-							</>
-						) : (
-							<>
-								The client list currently names <Mono>{request.client}</Mono> “{currentClient.text}”.
-							</>
-						)}
-					</p>
-				)}
 				<div {...stylex.props(styles.relatedList)}>{relatedActions}</div>
 			</section>
 		</>

admin/src/features/overview/ClientChart.test.tsx

@@ -4,6 +4,7 @@ import { createQueryClient } from "@/lib/queryClient";
 import { formatTime } from "@/lib/format";
 import ClientChart, { type ClientChartData } from "./ClientChart";
 import { OTHER_KEY, clientSeriesColor, seriesColor } from "./seriesColors";
+import { STILL_COUNTING_OPACITY } from "./chartKit";
 
 const SINCE = 1_700_000_000;
 const BUCKET = 1800;
@@ -73,7 +74,10 @@ test("the columns are timestamped from this response's own window", () => {
 	render(TWO_BUCKETS);
 
 	const rows = within(screen.getByRole("table")).getAllByRole("rowheader");
-	expect(rows.map((row) => row.textContent)).toEqual([formatTime(SINCE), formatTime(SINCE + BUCKET)]);
+	expect(rows.map((row) => row.textContent)).toEqual([
+		formatTime(SINCE),
+		`${formatTime(SINCE + BUCKET)} · still counting`,
+	]);
 });
 
 /**
@@ -100,7 +104,7 @@ test("the series are coloured by rank, and Other in its own gray", () => {
 		),
 	);
 
-	const fills = Array.from(container.querySelectorAll("rect"))
+	const fills = Array.from(container.querySelectorAll("rect:not([data-still-counting])"))
 		.map((rect) => rect.getAttribute("fill"))
 		.filter((fill) => fill !== "transparent");
 	expect(fills).toEqual([clientSeriesColor(0), clientSeriesColor(1), seriesColor(OTHER_KEY)]);
@@ -209,8 +213,9 @@ test("a refresh that rolls the window takes the tooltip down instead of relabell
 	rerender({ ...TWO_BUCKETS, since: SINCE + BUCKET });
 
 	expect(container.querySelectorAll("dl")).toHaveLength(0);
+	// No dimming is left; the last column stays faint because it is still counting.
 	const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
-	expect(stacks.every((group) => group.getAttribute("opacity") === "1")).toBe(true);
+	expect(stacks.map((group) => group.getAttribute("opacity"))).toEqual(["1", String(STILL_COUNTING_OPACITY)]);
 });
 
 /** The placement the hand-rolled tooltip had, restored over visx's 10px defaults. */
@@ -261,7 +266,7 @@ test("pointing at a bucket names its busiest clients, then Other, then the total
 	expect(swatches[2]).toContain(seriesColor(OTHER_KEY));
 
 	const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
-	expect(stacks.map((group) => group.getAttribute("opacity"))).toEqual(["1", "0.55"]);
+	expect(stacks.map((group) => group.getAttribute("opacity"))).toEqual(["1", String(0.55 * STILL_COUNTING_OPACITY)]);
 });
 
 /**
@@ -294,8 +299,9 @@ test("leaving the chart takes the tooltip and the dimming with it", () => {
 
 	fireEvent.mouseOut(container.querySelector("svg") as SVGSVGElement);
 	expect(container.querySelectorAll("dl")).toHaveLength(0);
+	// No dimming is left; the last column stays faint because it is still counting.
 	const stacks = Array.from(container.querySelectorAll("svg > g.visx-group[opacity]"));
-	expect(stacks.every((group) => group.getAttribute("opacity") === "1")).toBe(true);
+	expect(stacks.map((group) => group.getAttribute("opacity"))).toEqual(["1", String(STILL_COUNTING_OPACITY)]);
 });
 
 /**
@@ -310,6 +316,7 @@ test("a window where Other counted nothing keeps it in the legend and the table,
 	expect(Array.from(container.querySelectorAll("ul li")).map((item) => item.textContent)).toEqual([
 		"192.0.2.30",
 		"Other",
+		"Still counting",
 	]);
 	expect(
 		within(screen.getByRole("table"))
@@ -326,3 +333,28 @@ test("the hidden table groups its counts like every other figure on the page", (
 	render(clients([{ client: "192.0.2.30", buckets: [12345] }], [0]));
 	expect(within(screen.getByRole("table")).getByRole("cell", { name: "12,345" })).toBeTruthy();
 });
+
+/**
+ * The API ends every window at the end of the bucket that holds "now", so the
+ * last column is still filling. Drawn like the others it reads as a collapse in
+ * traffic; it is drawn faint inside a dashed outline, and every text says so.
+ */
+test("the last column is still counting, and says so in the drawing, the tooltip and the table", () => {
+	const { container } = render(TWO_BUCKETS);
+
+	const outlines = container.querySelectorAll("rect[data-still-counting]");
+	expect(outlines).toHaveLength(1);
+	expect(outlines[0].getAttribute("stroke-dasharray")).toBeTruthy();
+	expect(screen.getByRole("img").getAttribute("aria-label")).toMatch(/the last still counting$/);
+
+	fireEvent.mouseOver(overlayRects(container)[1]);
+	const tooltip = container.querySelector("dl") as HTMLElement;
+	expect(tooltip.previousElementSibling?.textContent).toBe(`${formatTime(SINCE + BUCKET)} · still counting`);
+});
+
+test("a last column with nothing in it yet draws no outline", () => {
+	const { container } = render(clients([{ client: "192.0.2.30", buckets: [10, 0] }], [2, 0]));
+
+	expect(container.querySelectorAll("rect[data-still-counting]")).toHaveLength(0);
+	expect(Array.from(container.querySelectorAll("ul li")).at(-1)?.textContent).toBe("Still counting");
+});

admin/src/features/overview/ClientChart.tsx

@@ -12,9 +12,10 @@
 import * as stylex from "@stylexjs/stylex";
 import { Group } from "@visx/group";
 import { BarStack } from "@visx/shape";
-import { formatCount, formatTime } from "@/lib/format";
+import { formatCount } from "@/lib/format";
 import type { OverviewClientSeries } from "@/lib/types";
 import { styles as shared } from "@/ui/styles";
+import { colors } from "@/ui/tokens.stylex";
 import { clientLabel, useClientNames, type ClientNames } from "@/features/clients/clientNames";
 import {
 	CHART_HEIGHT,
@@ -25,8 +26,11 @@ import {
 	EmptyChart,
 	HitBands,
 	STACK_BLEED,
+	STILL_COUNTING_DASH,
+	STILL_COUNTING_OPACITY,
 	StackSegment,
 	bandScale,
+	bucketTitle,
 	labelTickValues,
 	plotArea,
 	slotCenter,
@@ -38,6 +42,15 @@ import {
 } from "./chartKit";
 import { OTHER_KEY, clientKey, clientSeriesColor, seriesColor } from "./seriesColors";
 
+const styles = stylex.create({
+	/** The column's outline, over its faint fill: the bucket that is still filling. */
+	stillCounting: {
+		fill: "none",
+		stroke: colors.textSecondary,
+		strokeWidth: 1,
+	},
+});
+
 /** How many named clients a bucket's tooltip lists before the aggregate and the total. */
 const TOOLTIP_CLIENTS = 4;
 
@@ -110,6 +123,7 @@ export default function ClientChart({ data }: { data: ClientChartData }) {
 	const yTicks = valueTicks(yScale);
 	const colorOf = new Map(series.map((one) => [one.key, one.color]));
 	const centers = timestamps.map((ts) => slotCenter(xScale, ts, plot));
+	const last = bucketCount - 1;
 
 	/**
 	 * The bucket's busiest clients, then Other, then the total — eight named
@@ -131,7 +145,7 @@ export default function ClientChart({ data }: { data: ClientChartData }) {
 			value: formatCount(column[one.key]),
 		}));
 		return {
-			title: formatTime(column.ts),
+			title: bucketTitle(column.ts, index === last),
 			rows: [...rows, { key: "total", label: "All clients", value: formatCount(totals[index]) }],
 		};
 	}
@@ -140,7 +154,7 @@ export default function ClientChart({ data }: { data: ClientChartData }) {
 		<ChartRoot containerRef={containerRef}>
 			<svg
 				role="img"
-				aria-label={`Client activity over time, ${formatCount(bucketCount)} buckets, ${formatCount(series.length)} series`}
+				aria-label={`Client activity over time, ${formatCount(bucketCount)} buckets, ${formatCount(series.length)} series, the last still counting`}
 				width="100%"
 				height={CHART_HEIGHT}
 				viewBox={`0 0 ${width} ${CHART_HEIGHT}`}
@@ -166,7 +180,10 @@ export default function ClientChart({ data }: { data: ClientChartData }) {
 						columns.map((column, index) => (
 							<Group
 								key={column.ts}
-								opacity={hovered.index === null || hovered.index === index ? 1 : 0.55}
+								opacity={
+									(hovered.index === null || hovered.index === index ? 1 : 0.55) *
+									(index === last ? STILL_COUNTING_OPACITY : 1)
+								}
 							>
 								{stacks.map((stack) => {
 									const bar = stack.bars[index];
@@ -187,6 +204,17 @@ export default function ClientChart({ data }: { data: ClientChartData }) {
 						))
 					}
 				</BarStack>
+				{totals[last] > 0 && (
+					<rect
+						data-still-counting=""
+						x={xScale(columns[last].ts) ?? plot.x}
+						y={yScale(totals[last])}
+						width={xScale.bandwidth()}
+						height={plot.bottom - yScale(totals[last])}
+						strokeDasharray={STILL_COUNTING_DASH}
+						{...stylex.props(styles.stillCounting)}
+					/>
+				)}
 				<HitBands plot={plot} centers={centers} onEnter={hovered.show} />
 			</svg>
 			{hovered.index !== null && (
@@ -207,9 +235,9 @@ export default function ClientChart({ data }: { data: ClientChartData }) {
 						</tr>
 					</thead>
 					<tbody>
-						{columns.map((column) => (
+						{columns.map((column, index) => (
 							<tr key={column.ts}>
-								<th scope="row">{formatTime(column.ts)}</th>
+								<th scope="row">{bucketTitle(column.ts, index === last)}</th>
 								{series.map((one) => (
 									<td key={one.key}>{formatCount(column[one.key])}</td>
 								))}

admin/src/features/overview/TimeseriesChart.test.tsx

@@ -450,7 +450,7 @@ test("the legend names the two curves, and the table adds the remainder", () =>
 	);
 
 	const legend = Array.from(container.querySelectorAll("ul li")).map((item) => item.textContent);
-	expect(legend).toEqual(["Total queries", "Blocked"]);
+	expect(legend).toEqual(["Total queries", "Blocked", "Still counting"]);
 	expect(
 		within(screen.getByRole("table"))
 			.getAllByRole("columnheader")
@@ -470,3 +470,42 @@ test("the hidden table groups its counts like every other figure on the page", (
 			.map((cell) => cell.textContent),
 	).toEqual(["12,345", "1,234", "11,111"]);
 });
+
+/**
+ * The API ends every window at the end of the bucket that holds "now", so the
+ * last bucket is still filling, and a curve falling into it read as a collapse
+ * in traffic. The stretch into the last point is drawn dashed over a fainter
+ * fill, the settled curve stops where it starts, and every text says so.
+ */
+test("the curve into the last bucket is drawn as still counting", () => {
+	const { container } = render(
+		<TimeseriesChart
+			data={timeseries([
+				{ ts: SINCE, queries: 900, blocked: 90, cached: 0 },
+				{ ts: SINCE + 1800, queries: 800, blocked: 80, cached: 0 },
+				{ ts: SINCE + 3600, queries: 130, blocked: 13, cached: 0 },
+			])}
+		/>,
+	);
+
+	const counting = container.querySelector('g[data-series="queries"] g[data-still-counting]') as SVGGElement;
+	const [area, line] = Array.from(counting.querySelectorAll("path"));
+	expect(line.getAttribute("stroke-dasharray")).toBeTruthy();
+	expect(Number(area.getAttribute("fill-opacity"))).toBeLessThan(0.13);
+	expect(seriesPaths(container, "queries").line.getAttribute("stroke-dasharray")).toBeNull();
+
+	// The two clips meet at the second-to-last point, where the settled curve ends.
+	const middleX = pathPoints(seriesPaths(container, "queries").line.getAttribute("d") ?? "")[3][0];
+	const clips = Array.from(container.querySelectorAll("defs rect"));
+	expect(clips).toHaveLength(2);
+	expect(clips[0].getAttribute("width")).toBe(clips[1].getAttribute("x"));
+	expect(Number(clips[0].getAttribute("width"))).toBeCloseTo(middleX, 0);
+
+	expect(screen.getByRole("img").getAttribute("aria-label")).toMatch(/the last still counting$/);
+	fireEvent.mouseOver(overlayRects(container)[2]);
+	const tooltip = container.querySelector("dl") as HTMLElement;
+	expect(tooltip.previousElementSibling?.textContent).toBe(`${formatTime(SINCE + 3600)} · still counting`);
+	const rows = within(screen.getByRole("table")).getAllByRole("rowheader");
+	expect(rows.at(-1)?.textContent).toBe(`${formatTime(SINCE + 3600)} · still counting`);
+	expect(rows[0].textContent).toBe(formatTime(SINCE));
+});

admin/src/features/overview/TimeseriesChart.tsx

@@ -6,8 +6,9 @@
  * the reader is looking at.
  */
 
+import { useId } from "react";
 import * as stylex from "@stylexjs/stylex";
-import { formatCount, formatTime } from "@/lib/format";
+import { formatCount } from "@/lib/format";
 import type { Bucket } from "@/lib/types";
 import { styles as shared } from "@/ui/styles";
 import { colors } from "@/ui/tokens.stylex";
@@ -19,11 +20,15 @@ import {
 	ChartTooltip,
 	EmptyChart,
 	HitBands,
+	STILL_COUNTING_DASH,
+	STILL_COUNTING_OPACITY,
 	areaPath,
+	bucketTitle,
 	labelTickValues,
 	plotArea,
 	pointScale,
 	smoothPath,
+	svgId,
 	useActiveIndex,
 	useMeasuredWidth,
 	valueScale,
@@ -75,9 +80,9 @@ function columnsOf(buckets: Bucket[]): Column[] {
 	}));
 }
 
-function tooltipOf(column: Column): TooltipContent {
+function tooltipOf(column: Column, stillCounting: boolean): TooltipContent {
 	return {
-		title: formatTime(column.ts),
+		title: bucketTitle(column.ts, stillCounting),
 		rows: [
 			...SERIES.map((series) => ({
 				key: series.key,
@@ -105,6 +110,7 @@ export default function TimeseriesChart({ data }: { data: TimeseriesData }) {
 	// A hover survives a re-render only while it still names the same bucket at
 	// the same place: a poll that rolls the window, or a resize, retires it.
 	const hovered = useActiveIndex(`${data.since}:${data.bucket_seconds}:${data.buckets.length}:${width}`);
+	const clipId = svgId(useId());
 
 	if (data.buckets.length === 0 || data.buckets.every((bucket) => bucket.queries === 0)) {
 		return <EmptyChart containerRef={containerRef} />;
@@ -120,12 +126,16 @@ export default function TimeseriesChart({ data }: { data: TimeseriesData }) {
 	const centers = timestamps.map((ts) => xScale(ts) ?? plot.x);
 	const pointsOf = (key: SeriesKey): [number, number][] =>
 		columns.map((column, i) => [centers[i], yScale(column[key])]);
+	const last = columns.length - 1;
+	// The curve into the last point belongs to the bucket that is still
+	// filling, so everything right of the point before it is drawn as such.
+	const settledEdge = last > 0 ? centers[last - 1] : plot.x;
 
 	return (
 		<ChartRoot containerRef={containerRef}>
 			<svg
 				role="img"
-				aria-label={`Queries over time, ${formatCount(data.buckets.length)} buckets: total and blocked queries per bucket`}
+				aria-label={`Queries over time, ${formatCount(data.buckets.length)} buckets: total and blocked queries per bucket, the last still counting`}
 				width="100%"
 				height={CHART_HEIGHT}
 				viewBox={`0 0 ${width} ${CHART_HEIGHT}`}
@@ -139,23 +149,45 @@ export default function TimeseriesChart({ data }: { data: TimeseriesData }) {
 					xTickValues={labelTickValues(timestamps, plot.width)}
 					bucketSeconds={data.bucket_seconds}
 				/>
+				<defs>
+					<clipPath id={`${clipId}-settled`}>
+						<rect x={0} y={0} width={settledEdge} height={CHART_HEIGHT} />
+					</clipPath>
+					<clipPath id={`${clipId}-counting`}>
+						<rect x={settledEdge} y={0} width={Math.max(0, width - settledEdge)} height={CHART_HEIGHT} />
+					</clipPath>
+				</defs>
 				{SERIES.map((series) => {
 					const points = pointsOf(series.key);
 					const line = smoothPath(points, plot.y, plot.bottom);
+					const area = areaPath(line, points, plot);
 					return (
 						<g key={series.key} data-series={series.key}>
-							<path
+							<g clipPath={`url(#${clipId}-settled)`}>
-								d={areaPath(line, points, plot)}
+								<path d={area} fill={series.color} fillOpacity={series.fillOpacity} />
-								fill={series.color}
+								<path
-								fillOpacity={series.fillOpacity}
+									d={line}
-							/>
+									fill="none"
-							<path
+									stroke={series.color}
-								d={line}
+									strokeWidth={LINE_WIDTH}
-								fill="none"
+									strokeLinejoin="round"
-								stroke={series.color}
+								/>
-								strokeWidth={LINE_WIDTH}
+							</g>
-								strokeLinejoin="round"
+							<g clipPath={`url(#${clipId}-counting)`} data-still-counting="">
-							/>
+								<path
+									d={area}
+									fill={series.color}
+									fillOpacity={series.fillOpacity * STILL_COUNTING_OPACITY}
+								/>
+								<path
+									d={line}
+									fill="none"
+									stroke={series.color}
+									strokeWidth={LINE_WIDTH}
+									strokeLinejoin="round"
+									strokeDasharray={STILL_COUNTING_DASH}
+								/>
+							</g>
 						</g>
 					);
 				})}
@@ -185,7 +217,7 @@ export default function TimeseriesChart({ data }: { data: TimeseriesData }) {
 			{hovered.index !== null && (
 				<ChartTooltip
 					index={hovered.index}
-					content={tooltipOf(columns[hovered.index])}
+					content={tooltipOf(columns[hovered.index], hovered.index === last)}
 					left={centers[hovered.index]}
 					top={yScale(columns[hovered.index].queries)}
 					beside
@@ -206,9 +238,9 @@ export default function TimeseriesChart({ data }: { data: TimeseriesData }) {
 						</tr>
 					</thead>
 					<tbody>
-						{columns.map((column) => (
+						{columns.map((column, index) => (
 							<tr key={column.ts}>
-								<th scope="row">{formatTime(column.ts)}</th>
+								<th scope="row">{bucketTitle(column.ts, index === last)}</th>
 								<td>{formatCount(column.queries)}</td>
 								<td>{formatCount(column.blocked)}</td>
 								<td>{formatCount(column.allowed)}</td>

admin/src/features/overview/chartKit.tsx

@@ -15,7 +15,7 @@ import { AxisBottom, AxisLeft, type TickRendererProps } from "@visx/axis";
 import { GridRows } from "@visx/grid";
 import { scaleBand, scaleLinear, scalePoint } from "@visx/scale";
 import { TooltipWithBounds } from "@visx/tooltip";
-import { formatBucketTime, formatCount } from "@/lib/format";
+import { formatBucketTime, formatCount, formatTime } from "@/lib/format";
 import { styles as shared } from "@/ui/styles";
 import { colors, layers, metrics } from "@/ui/tokens.stylex";
 
@@ -405,6 +405,30 @@ export function StackSegment({
 /** The amount one upper segment overlaps the one under it, in plot units. */
 export const STACK_BLEED = 0.5;
 
+/**
+ * The name for the last bucket of every window. The API ends a window at the
+ * end of the bucket that holds "now" (overview.zig `window`), so that bucket is
+ * still filling, and drawn like the others it reads as a collapse in traffic.
+ * Both charts draw it faint and dashed, and the legend says what that means, so
+ * the meaning does not depend on a hover.
+ */
+export const STILL_COUNTING = "Still counting";
+
+/** Faint enough to read as unfinished beside the settled buckets, not as absent. */
+export const STILL_COUNTING_OPACITY = 0.45;
+
+export const STILL_COUNTING_DASH = "4 3";
+
+/** A tooltip or table title for a bucket, naming the one that is still filling. */
+export function bucketTitle(ts: number, stillCounting: boolean): string {
+	return stillCounting ? `${formatTime(ts)} · ${STILL_COUNTING.toLowerCase()}` : formatTime(ts);
+}
+
+/** `useId` output made safe inside an SVG `url(#…)` reference. */
+export function svgId(reactId: string): string {
+	return reactId.replace(/[^A-Za-z0-9_-]/g, "");
+}
+
 /**
  * The transparent hit targets: one per bucket, spanning the whole plot height so
  * that pointing at the empty space above a short column or a low point still
@@ -607,6 +631,11 @@ const legendStyles = stylex.create({
 		borderRadius: "0.125rem",
 	},
 	swatchColor: (color: string) => ({ backgroundColor: color }),
+	swatchCounting: {
+		borderWidth: 1,
+		borderStyle: "dashed",
+		borderColor: colors.textSecondary,
+	},
 });
 
 /** The key under a chart: one swatch and one name per drawn series, in drawing order. */
@@ -622,6 +651,10 @@ export function ChartLegend({ entries }: { entries: { key: string; label: string
 					{entry.label}
 				</li>
 			))}
+			<li {...stylex.props(legendStyles.item)}>
+				<span aria-hidden="true" {...stylex.props(legendStyles.swatch, legendStyles.swatchCounting)} />
+				{STILL_COUNTING}
+			</li>
 		</ul>
 	);
 }

build.zig.zon

@@ -1,6 +1,6 @@
 .{
     .name = .nxdns,
-    .version = "0.0.19",
+    .version = "0.0.23",
     .minimum_zig_version = "0.16.0",
     .paths = .{""},
     .fingerprint = 0x3307b311dded1d91,

deploy/docker/compose.yaml

@@ -1,10 +1,11 @@
 services:
   nxdns:
-    # Published multi-architecture image. Pin an exact version in production:
+    # Published multi-architecture image. The default is the release this file
-    # NXDNS_VERSION=0.0.1 docker compose up -d. To run a locally built image
+    # is tagged with: `zig build cut` moves it in the bump commit, so the file
-    # instead, build it first (see deploy/docker/Dockerfile) and set
+    # from a tag runs that tag's image. NXDNS_VERSION picks another release. To
-    # NXDNS_IMAGE=nxdns.
+    # run a locally built image instead, build it first (see
-    image: ${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-latest}}
+    # deploy/docker/Dockerfile) and set NXDNS_IMAGE=nxdns.
+    image: ${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-0.0.23}}
     restart: unless-stopped
     # `run --config` makes the file the sole source of configuration: nxdns
     # reconciles the database onto ./etc-nxdns/config.zon at every start, and

docs/README.md

@@ -22,7 +22,7 @@ Steps for a goal you already have. They assume you know what nxdns is.
 - [how-to/verify-a-release.md](how-to/verify-a-release.md) — check the signature and the checksums before you run anything, and what they prove.
 - [how-to/install-with-systemd.md](how-to/install-with-systemd.md) — a real install as a system service, including the Raspberry Pi 5 aarch64 binary.
 - [how-to/install-with-docker.md](how-to/install-with-docker.md) — the published container image and the compose file.
-- [how-to/install-with-nix.md](how-to/install-with-nix.md) — the flake input pinned to a release tag, and Renovate for tag bumps.
+- [how-to/install-with-nix.md](how-to/install-with-nix.md) — the flake input pinned to a release tag, the `services.nxdns` NixOS module, and Renovate for tag bumps.
 - [how-to/upgrade.md](how-to/upgrade.md) — move to a new release without losing state.
 - [how-to/troubleshoot.md](how-to/troubleshoot.md) — what to do when it does not answer, does not block, or will not start.
 - [how-to/enable-doh-and-dot.md](how-to/enable-doh-and-dot.md) — serve encrypted DNS with certificates.

docs/explanation/performance-and-testing.md

@@ -79,8 +79,6 @@ Two honest gaps remain, stated so nobody has to rediscover them:
 
 The same distinction applies one level out, to the artifacts. A release is signed, and the signature is worth having: it says the artifact came from this project's pipeline and reached you unaltered. It does not say the binary was built from the source in this repository, because the machine that ran the build also held the signing key. An attacker with that machine produces something that verifies cleanly and contains whatever they put in it.
 
-The control that closes that gap is a reproducibility gate — an independent build, in a different directory on a different machine, landing on the same bytes. It does not exist. It is a recorded deferral (`specs/milestone-14.md` ruling 12), not something nobody thought of, and until it exists no document here describes the build as reproducible: nobody has measured whether it is. The cheap inputs to reproducibility are already in place — `gzip -n`, `--mtime=@0`, `LC_ALL=C`, `TZ=UTC`, exact Zig and Node pins — which makes the gate cheap to add later and proves nothing on its own.
+The control that closes that gap is a reproducibility gate — an independent build, in a different directory on a different machine, landing on the same bytes. Every release from 0.0.17 on passes one (milestone 40, which closed the deferral in `specs/milestone-14.md` ruling 12). The cut builds the tarballs on the maintainer's machine and writes their hashes into `flake.nix`; the release job rebuilds them on the CI runner from the tagged tree, and `zig build verify-pins` refuses to let anything be uploaded unless the two builds landed on the same bytes. What makes that hold: the binary embeds no commit sha, the archive is written by the project's own tool with fixed modes, zero timestamps and sorted entries, and Zig, Node and npm are pinned to exact versions and run in a normalized environment.
 
-What the release pipeline is required to hold to is narrower: two runs of `zig build dist` on the same commit **in the same directory** produce byte-identical tarballs. Same-directory determinism is a much weaker property than reproducibility, and conflating the two is exactly the kind of claim this page exists to refuse.
+What the gate does not show: both builds run on machines the maintainer controls, so it proves the pipeline is deterministic across two hosts, not that someone independent of the project has rebuilt it. That is what the rebuild recipe in [Verify a release](../how-to/verify-a-release.md) is for, and anyone can run it. It reproduces the release only on x86_64 Linux with glibc, because the admin bundle is built with host-native Rolldown and Lightning CSS bindings.
-
-[Verify a release](../how-to/verify-a-release.md) states the same limits where an operator will actually meet them, and gives the rebuild-and-compare recipe with the caveat that a differing hash is not evidence of tampering while this gap is open.

docs/how-to/install-with-docker.md

@@ -6,18 +6,9 @@ The image is multi-architecture — `linux/amd64` and `linux/arm64` — so the s
 
 For what each configuration field means, see [the configuration reference](../reference/configuration.md).
 
-> Verification: the failure modes, the run and the two checks in step 3 were run
+> Verification: on 2026-10-02, steps 1 to 3 were run against the published `v0.0.21` release with an image pulled anonymously from the registry, on the host ports printed below; the notes in those steps carry the detail. The failure modes in step 2 were run on the machine that wrote an earlier revision of this page, against an image built from this checkout. The `chown` to uid 65532 needs root and was not run.
-> on the machine that wrote an earlier revision of this page, against an image
-> built from this checkout rather than pulled from the registry — no release is
-> published yet, so nothing on this page could be run against a pulled image,
-> and step 1 could not be run at all. One command was run in altered form: host
-> port 8080 was occupied there, so the run and the verification commands in
-> step 3 were executed with the host side of the port mappings moved to 25353
-> and 28088 rather than the 53 and 8080 printed below. The container side was
-> unchanged. See the note in step 3. The `chown` to uid 65532 needs root and was
-> not run.
 >
-> **Not re-run for the file-mode revision.** The compose file now ships
+> **Earlier, for the file-mode revision.** The compose file now ships
 > `command: ["run", "--config=/etc/nxdns/config.zon"]`, and no container was
 > started against that command on this host: staging a release image needs
 > `zig build dist`, which refuses to run while `admin/dist` is stale, and the web
@@ -56,12 +47,7 @@ cut -d@ -f2 IMAGE-DIGEST.txt
 
 The last two have to print the same string — `IMAGE-DIGEST.txt` holds a whole pinned reference, `name:tag@sha256:…`, so the `cut` is what reduces it to the digest `imagetools` prints. [Verify a release](verify-a-release.md) covers the key, the fingerprint, every failure message, and what the signature does and does not prove.
 
-> Not verified on this host: no image and no release are published yet, so
+> Verified on 2026-10-02 against `v0.0.21`, with an empty `DOCKER_CONFIG`, so no registry credentials: the lookup printed `0.0.21`, `docker pull` succeeded without a login, `gpg --verify` printed `Good signature` with the primary fingerprint above, `sha256sum -c` printed `OK` for all three files, and `imagetools inspect` and the `cut` both printed `sha256:ce1fef32c7ee6c23656abd94e53e4d4d6b1d34fe88a2c842b82c7202d8b904c3`.
-> `docker pull` and every URL here fail today, and the `releases/latest` lookup
-> returns 404 and leaves `VERSION` empty. That lookup was run against
-> `gitea.com/gitea/tea` on Gitea `1.27.0+dev` and printed `0.15.1`. The
-> `docker buildx imagetools inspect --format` shape was run here against
-> `alpine:3.22` on Docker Hub and printed that image's index digest.
 
 ## 2. Get the compose file and write the configuration
 
@@ -72,9 +58,7 @@ mkdir -p ~/nxdns && cd ~/nxdns
 curl -fLO "$BASE/raw/tag/v$VERSION/deploy/docker/compose.yaml"
 ```
 
-> Not verified against nxdns: there is no tag yet. The Gitea raw-file URL shape
+> Verified on 2026-10-02: the `curl` returned `compose.yaml` from the `v0.0.21` tag with a 200.
-> `<repo>/raw/tag/<tag>/<path>` was run here against `gitea.com/gitea/tea` on
-> Gitea `1.27.0+dev` and returned the file with a 200.
 
 Compose bind-mounts `deploy/docker/etc-nxdns` read-only at `/etc/nxdns`. Create it and put the configuration in it:
 
@@ -148,17 +132,17 @@ chmod 0600 deploy/docker/etc-nxdns/config.zon
 ## 3. Run it
 
 ```sh
-NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml up -d
+docker compose -f deploy/docker/compose.yaml up -d
 docker compose -f deploy/docker/compose.yaml logs -f
 ```
 
-`compose.yaml` reads the image from two variables: `${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-latest}}`. Set `NXDNS_VERSION` to pin a release; set `NXDNS_IMAGE` to run something else entirely, which is what the build-from-source section at the bottom does.
+`compose.yaml` names the release it was tagged with: its image is `${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-<release>}}`, and `zig build cut` writes `<release>` in the same commit that bumps `build.zig.zon`, so the file from the `v$VERSION` tag runs `$VERSION` with no variable set. Set `NXDNS_VERSION` to run a different release with the same file; set `NXDNS_IMAGE` to run something else entirely, which is what the build-from-source section at the bottom does. Compose files from `v0.0.22` and earlier default to `:latest` instead, so with one of those, set `NXDNS_VERSION=$VERSION` on every command.
 
-> Verified on this host with `docker compose -f deploy/docker/compose.yaml
+> Verified on 2026-10-20 with `docker compose -f deploy/docker/compose.yaml
-> config`, which resolves the variables without contacting a registry: no
+> config` on a tree whose `build.zig.zon` declares `0.0.22`, which resolves
-> variables gives `git.mial.net/mokhtar/nxdns:latest`, `NXDNS_VERSION=0.0.1`
+> the variables without contacting a registry: no variables gives
-> gives `git.mial.net/mokhtar/nxdns:0.0.1`, and `NXDNS_IMAGE=nxdns` gives
+> `git.mial.net/mokhtar/nxdns:0.0.22`, `NXDNS_VERSION=0.0.21` gives
-> `nxdns`.
+> `git.mial.net/mokhtar/nxdns:0.0.21`, and `NXDNS_IMAGE=nxdns` gives `nxdns`.
 
 Every block on this page runs from the repository root, and none of them change directory, so they can be pasted in order. `-f` is what makes that work: Compose resolves the relative paths inside `compose.yaml` — the `etc-nxdns` bind mount — against the directory holding the file, not against your shell, and it takes the project name `docker` from that directory either way, which is why the container is `docker-nxdns-1`.
 
@@ -166,7 +150,7 @@ A healthy first start logs the reconcile, the authority and the bound sockets:
 
 ```
 info(migrations): config.db migrated from schema version 0 to 1
-reconciled '/etc/nxdns/config.zon': upstreams +1 ~0 -0; settings +45 ~0 -0;
+reconciled '/etc/nxdns/config.zon': upstreams +1 ~0 -0; settings +46 ~0 -0;
 settings keys changed: dns.bind_ipv4 dns.bind_ipv6 dns.port web.bind web.port …
 web authentication is now enabled
 info(nxdns): authority: file (/etc/nxdns/config.zon)
@@ -183,15 +167,7 @@ dig @127.0.0.1 example.com A +short
 curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8080/
 ```
 
-> On the machine that wrote this page, host port 8080 was already taken by an
+> Verified on 2026-10-02 with the pulled `v0.0.21` image, the configuration above and the unchanged `compose.yaml`, on host ports 53 and 8080: the first start logged the lines above, `dig` returned the A records for `example.com`, `curl` returned 200, and `/api/health` reported `"status":"ok"` with one of one upstreams available. `docker compose up -d` fails with `failed to bind host port 0.0.0.0:8080/tcp: address already in use` when a host port is occupied; free the port or edit the `ports:` list.
-> unrelated process, so the container was verified with the host side of the
-> port mappings moved to 25353 and 28088 — everything inside the container was
-> unchanged, and the log still reads `serving on udp [::]:53`. Against those
-> ports `dig` returned the A records for `example.com` and `curl` returned 200.
-> `docker compose up -d` fails with
-> `failed to bind host port 0.0.0.0:8080/tcp: address already in use` when a
-> host port is occupied; free the port or edit the `ports:` list. The image
-> under test was built locally, not pulled: there is nothing published to pull.
 
 The compose file publishes 53/udp, 53/tcp and 8080, keeps `/var/lib/nxdns` in the named volume `nxdns-data`, and sets the per-namespace sysctl `net.ipv4.ip_unprivileged_port_start=0` so uid 65532 can bind port 53 without any capability. Uncomment the 443 and 853 mappings when you enable the DoH or DoT listener; see [Enable DoH and DoT](enable-doh-and-dot.md).
 
@@ -209,9 +185,7 @@ To pull the arm64 image from an x86_64 machine — to inspect it, or to save and
 docker pull --platform linux/arm64 git.mial.net/mokhtar/nxdns:$VERSION
 ```
 
-> Not verified on this host: nothing is published to pull, and this host is
+> Verified on 2026-10-02 against `v0.0.21`: `imagetools inspect` lists `linux/amd64` and `linux/arm64`, and the pull above fetched an image whose `docker image inspect` reports `linux/arm64`. It was not started: this host is x86_64 with no emulation.
-> x86_64 with no emulation, so an arm64 image could not be started here even
-> if it were.
 
 ## Build the image from source instead
 

docs/how-to/install-with-nix.md

@@ -1,12 +1,12 @@
 # Install nxdns with Nix
 
-Adds nxdns to a NixOS machine as a flake input pinned to a release tag. At the end `pkgs`-style references to `inputs.nxdns.packages.${system}.default` resolve to the published release binary, and a Renovate custom manager opens a pull request when a new tag appears.
+Adds nxdns to a NixOS machine as a flake input pinned to a release tag and runs it with the flake's `services.nxdns` module. At the end the machine serves DNS on port 53 from a configuration file that the pinned binary checked at build time, and a Renovate custom manager opens a pull request when a new tag appears.
 
 nxdns publishes its own `flake.nix`. Its packages do not build nxdns from source: each one fetches the release tarball for the target and pins its SHA-256 hash, so a changed byte fails the build. The two supported systems are `aarch64-linux` and `x86_64-linux`, both static musl builds that need nothing on the host.
 
 For the signature and checksum checks a human does once, see [verify a release](verify-a-release.md). For what each configuration field means, see [the configuration reference](../reference/configuration.md).
 
-> Verification: the commands in step 1 and step 2 were run on the machine that wrote this page, against the flake at nxdns 0.0.16. `nix flake check --no-build` passed and `nix build` produced a binary that printed `nxdns 0.0.16`. The consumer snippets in step 3 and step 4 are copied from the shape `rpi.mial.net` already uses for another flake input of the same author; they were not evaluated from this checkout, which is not a NixOS configuration.
+> Verification: the commands in step 1 and step 2 were run on the machine that wrote this page, against the flake at nxdns 0.0.16. `nix flake check --no-build` passed and `nix build` produced a binary that printed `nxdns 0.0.16`. Step 3 was verified on 2026-10-20 by the flake's own checks, built with Nix 2.35.3 on an x86_64 host with KVM and the module running the 0.0.22 release: `checks.x86_64-linux.nixos-module` boots a NixOS VM with the module and `openFirewall`, gets `nas.lan` from `dig @127.0.0.1`, finds the packaged unit with its drop-in in `systemctl cat`, finds port 53 open over UDP and TCP, and switches to a configuration-only change that restarts the service and changes the answer; `checks.x86_64-linux.nixos-module-rejects-config` fails a system build on a bad regex with `FAIL rules[0].pattern: '(' is not a valid regex pattern`. The Renovate snippet in step 4 is copied from the shape `rpi.mial.net` already uses for another flake input of the same author.
 
 ## 1. Add the input, pinned to a tag
 
@@ -42,18 +42,56 @@ nix eval .#packages.x86_64-linux.default.outPath
 
 Evaluation does not fetch the tarball. The download happens at build time, and the hash in nxdns's flake is what the build asserts the bytes against.
 
-## 3. Use the package in a NixOS configuration
+## 3. Run the service
 
-Pass the flake inputs to the module system, then reference the package:
+`nixosModules.default` exists from the first release after `v0.0.22`; pin that tag or a later one. A tag up to `v0.0.22` has the package only.
 
-```nix
+Write the configuration next to the NixOS configuration that uses it. The smallest file that starts is one group named `default` and one enabled upstream:
-{ inputs, pkgs, ... }:
+
-{
+```zon
-  environment.systemPackages = [ inputs.nxdns.packages.${pkgs.stdenv.hostPlatform.system}.default ];
+.{
+    .groups = .{ .{ .name = "default" } },
+    .upstreams = .{ .{ .url = "https://cloudflare-dns.com/dns-query" } },
 }
 ```
 
-The derivation installs `bin/nxdns`, plus `LICENSE` and `THIRD-PARTY-NOTICES` under `share/doc/nxdns`. It sets `meta.mainProgram`, so `lib.getExe` resolves to the binary. Run it as a service with the unit from [install with systemd](install-with-systemd.md), or write your own module around it.
+Pass the flake inputs to the module system, import the module, and enable it:
+
+```nix
+{ inputs, ... }:
+{
+  imports = [ inputs.nxdns.nixosModules.default ];
+
+  services.nxdns = {
+    enable = true;
+    configFile = ./nxdns-config.zon;
+    openFirewall = true;
+  };
+}
+```
+
+Then `nixos-rebuild switch`. The module has four options:
+
+| Option | Default | What it does |
+| --- | --- | --- |
+| `enable` | `false` | Runs nxdns as `nxdns.service`, as the `nxdns` system user. |
+| `package` | the release of the pinned tag | The nxdns release to run. |
+| `configFile` | none, required | The configuration, in ZON. It is installed at `/etc/nxdns/config.zon`. |
+| `openFirewall` | `false` | Opens port 53 over UDP and TCP. The web interface on port 8080 and the DoH and DoT ports stay closed; open them yourself if you want them reachable from the network. |
+
+`configFile` is copied into the Nix store, which every local user can read. Do not put a plaintext `web.password` in it: put the `web.password_hash` from an export there instead, as [set up admin authentication](set-up-admin-authentication.md) describes.
+
+What the module does with them:
+
+- **The build checks the configuration.** `nxdns import` from the pinned `package` reads `configFile` into a throwaway database during the build. A file that `import` rejects, because it does not parse, breaks a field rule or does not reconcile, fails `nixos-rebuild` with the binary's own reason, as in `FAIL rules[0].pattern: ...`, and nothing is deployed. `nxdns check` is not used, because it queries every upstream and the build sandbox has no network. So the build does not load the DoH and DoT certificate and key files, and does not reach the upstreams: a missing certificate still stops the service at start. The build also says nothing about the database already on the machine; see the rollback note below.
+- **nxdns runs in file mode.** The service runs `nxdns run --config=/etc/nxdns/config.zon`: every start reconciles the database onto the file, and the web interface refuses configuration edits. Change the file and switch to change anything. A switch that changes only the configuration restarts the service.
+- **The unit is the packaged one.** The package installs `deploy/systemd/nxdns.service` from the release tarball, with its sandboxing and its `RestartPreventExitStatus=2 64`, and the module adds a drop-in that changes only the command line and leaves `/etc/nxdns` to NixOS. `systemctl cat nxdns.service` shows both.
+
+The module does not touch the machine's own resolver. If you point `networking.nameservers` at `127.0.0.1`, the machine loses name resolution whenever nxdns is down, including while you repair it; a resolver outside the machine avoids that.
+
+**Rolling back.** A `nixos-rebuild --rollback` or an older generation puts an older binary in front of a database that the newer one may have migrated. That binary refuses the database with `SchemaTooNew`, exit 2, and the unit stops instead of restarting. [Upgrade](upgrade.md#what-happens-to-the-database) describes the recovery.
+
+The package alone is also available, for a machine that runs nxdns some other way: `inputs.nxdns.packages.${pkgs.stdenv.hostPlatform.system}.default` installs `bin/nxdns`, `lib/systemd/system/nxdns.service`, and `LICENSE` and `THIRD-PARTY-NOTICES` under `share/doc/nxdns`. It sets `meta.mainProgram`, so `lib.getExe` resolves to the binary.
 
 ## 4. Let Renovate bump the tag
 

docs/how-to/install-with-systemd.md

@@ -14,12 +14,7 @@ For what each flag does, see [the CLI reference](../reference/cli.md); for what
 > target host — `install`, `systemd-sysusers`, `systemctl` — were not run; they
 > are marked where they appear.
 >
-> The download in step 1 could not be run at all: this repository has no tags
+> The download in step 1 was run on 2026-10-02 against the published `v0.0.21` release; its note carries the detail.
-> and no published release, so every release URL on this page is a 404 today.
-> Its commands are the ones [Verify a release](verify-a-release.md) covers in
-> full, and the URL shapes — including the `releases/latest` redirect the
-> version is read from — were probed there against `gitea.com`, a public
-> instance running the same Gitea series.
 >
 > The `zig build dist` and `zig build verify-dist` blocks in the last section
 > were run here, both to completion and both exiting 0; that section carries
@@ -62,10 +57,7 @@ The extracted directory `nxdns-$VERSION-x86_64-linux-musl/` holds everything thi
 | `THIRD-PARTY-NOTICES` | Licences of everything compiled or bundled in |
 | `INSTALL.md` | A short version of this page |
 
-> Not verified on this host: no release exists yet, so none of these commands
+> Verified on 2026-10-02 against `v0.0.21`, from a clean directory with a clean `GNUPGHOME` holding only the key from keys.openpgp.org: the lookup printed `0.0.21`, `gpg --verify` printed `Good signature` over primary fingerprint `A206 1F6A B24D F2C0 E923  46FD 1509 B549 46D0 8A95`, `sha256sum -c` printed `OK` for both tarballs and `IMAGE-DIGEST.txt`, and the extracted directory held the six files in the table below at the modes given, with `./nxdns version` printing `nxdns 0.0.21`.
-> could be run against one — the `releases/latest` lookup returns 404 for this
-> repository and leaves `VERSION` empty. The same lookup was run against
-> `gitea.com/gitea/tea` on Gitea `1.27.0+dev` and printed `0.15.1`.
 
 ## 2. Copy the files to the target
 
@@ -364,9 +356,7 @@ scp nxdns nxdns.service nxdns.conf pi:/tmp/
 
 Then follow steps 3 to 6 on the Pi.
 
-> Not verified on this host: no release exists to download, `pi` is a
+> Partly verified on 2026-10-02: the `v0.0.21` aarch64 tarball downloaded and `sha256sum -c` printed `OK` for it. The `scp` and steps 3 to 6 were not run: `pi` is a placeholder for your Pi's hostname, and this page was written on an x86_64 machine with no Pi attached.
-> placeholder for your Pi's hostname, and this page was written on an x86_64
-> machine with no Pi attached.
 
 ## Build from source instead
 

docs/how-to/measure-performance.md

@@ -114,6 +114,26 @@ Read the client's own rate and the server's `/metrics` together: a load generato
 
 **Not verified on this host:** `dnsperf` is not installed here and the target platform is a Raspberry Pi 5, not this development machine. The command above is the shape of the measurement, not a transcript.
 
+## Measure memory on a running server
+
+The bench measures one structure at a time. A running server's memory also depends on the most concurrent work it has seen since it started, so read it on the server itself, after it has served a normal day:
+
+```sh
+pid=$(pidof nxdns)
+grep -E 'VmRSS|VmHWM|Threads' /proc/$pid/status
+curl -s http://127.0.0.1:8080/metrics | grep '^nxdns_.*_peak'
+```
+
+- `VmRSS` is what the process holds resident now. `VmHWM` is the most it has ever held, which is normally the last blocklist refresh.
+- `Threads` is a high-water too: the I/O runtime adds a worker thread only when every existing worker is busy, and never retires one. The count is the largest the worker pool has been since start, plus the main thread.
+- `nxdns_dns_in_flight_peak` is the most queries handled at once, every listener together.
+- `nxdns_tcp_server_connections_peak`, `nxdns_dot_server_connections_peak` and `nxdns_doh_server_connections_peak` are the most connections one listener has held at once. TCP/53 has one listener per address family; the gauge is the larger of the two.
+- `nxdns_querylog_queue_peak` is the most entries that have waited for the query log writer at once. It can overstate the true peak by the entries in transit at that instant, never understate it.
+
+Each worker thread's stack is its own anonymous mapping in `/proc/$pid/smaps`, directly above a one-page `---p` guard, and every worker's mapping has the same size. Its `Rss` is the stack that thread has actually touched.
+
+**Verified on this host** against a static x86_64 musl build under a burst of 300 UDP queries: the gauges and the stack mappings behave as described. Not yet run on the Pi.
+
 ## Where to run it
 
 The target platform is a Raspberry Pi 5. Numbers from a development x86_64 box do not transfer — the Pi's Cortex-A76 is far slower — so a passing run here is evidence the harness works and a baseline for spotting regressions on the machine development happens on, and nothing more. Run `--assert` on the Pi, where the numbers mean something.

docs/how-to/troubleshoot.md

@@ -273,6 +273,28 @@ curl -s http://127.0.0.1:8080/metrics | grep nxdns_upstream_
 
 **Fix.** Raise `upstream.total_timeout_ms` if the queue drains but drains too slowly for the budget. Otherwise the queue is telling you the upstream is slow: an upstream answering in a few milliseconds does not fill eight concurrent slots at household query rates, so sustained queueing points at the resolver you configured, and a faster one is the fix. Adding a second upstream helps only with saturation, not with latency: failover is strict priority, not load spreading. A query does take the next upstream when the first one's slots are all busy — priority orders the candidates that can be admitted right now — but once every eligible upstream is full it blocks on the highest-priority one, and it still sends one query to one upstream at a time. The per-upstream slot count is compiled, not configured, so there is no knob to widen one upstream either.
 
+## SERVFAIL rows that name no upstream
+
+**Symptom.** The query log shows SERVFAIL rows whose `upstream` is empty, and `/api/health` shows every upstream healthy.
+
+**Diagnosis.** A failed exchange names no upstream only when no attributable attempt happened, and two pool-wide counters say why:
+
+```sh
+curl -s http://127.0.0.1:8080/metrics | grep -E 'nxdns_upstream_(budget_exhausted|local_resource)_total'
+```
+
+`nxdns_upstream_budget_exhausted_total` is a query whose own budget ran out first; the previous section covers it. `nxdns_upstream_local_resource_total` is an exchange this process ended on its own resources: out of memory, out of file descriptors, or a system error the runtime has no name for. Neither is charged to an upstream, because no upstream caused it, and a local failure does not fail over, because the next upstream would hit the same wall. Both count exchanges, not rows without an upstream: when an earlier upstream had already failed in the same exchange, the row names that one.
+
+**Fix.** For `local_resource_total`, look at the process's limits: `grep -E 'Max open files' /proc/$(pidof nxdns)/limits` against `ls /proc/$(pidof nxdns)/fd | wc -l`, and `VmRSS` in `/proc/$(pidof nxdns)/status` against the memory the host has. A connect that fails for a reason the runtime cannot name — a blackhole route to the upstream is one — is not counted here: it is charged to that upstream as `ConnectFailed (cause Unexpected)`, and the query fails over.
+
+## A DoH upstream fails with `HttpStatus (status 505 HTTP Version Not Supported)`
+
+**Symptom.** Diagnostics shows the upstream failing with that cause, and no query ever succeeds against it.
+
+**Diagnosis.** The server refuses HTTP/1.1. nxdns speaks DoH over HTTP/1.1 only, and some public resolvers (Quad9's `https://dns.quad9.net/dns-query`, for one) require HTTP/2. Other statuses name themselves the same way: a 403 or 404 is usually a wrong URL path.
+
+**Fix.** Use that provider's DoT endpoint instead (for Quad9, `tls://9.9.9.9:853` with `tls_name = "dns.quad9.net"`), or a DoH upstream that accepts HTTP/1.1.
+
 ## The disk is filling up
 
 **Symptom.** Writes stop but DNS keeps answering. The journal shows the transition:

docs/how-to/upgrade.md

@@ -13,10 +13,7 @@ The normal path is to download the new release, verify it, and swap the binary.
 > where the substitution matters, or which was not run at all, carries its own
 > note. Nothing here was verified except where a note says so.
 >
-> Step 2 could not be run at all: no nxdns release is published yet, so every
+> The download in step 2 was run on 2026-10-02 against the published `v0.0.21` release; its note carries the detail.
-> release URL and the `docker compose pull` on this page fail today. The URL
-> shapes and the verification commands are covered by
-> [Verify a release](verify-a-release.md), which says what was probed and how.
 
 ## Breaking change: `run --config` now means file authority
 
@@ -137,10 +134,7 @@ Take `aarch64-linux-musl` for a Raspberry Pi 5. Verify every time, not only on t
 
 Under Docker there is nothing to download: step 3 pulls the image, and the `IMAGE-DIGEST.txt` asset is what you verify instead.
 
-> Not verified on this host: no release exists yet, so the `releases/latest`
+> Verified on 2026-10-02 against `v0.0.21`: the lookup printed `0.0.21`, every `curl` succeeded, `gpg --verify` printed `Good signature`, `sha256sum -c` printed `OK`, and the tarball extracted. [Install with systemd](install-with-systemd.md) has the detail.
-> lookup returns 404 and leaves `VERSION` empty, and every `curl` below it is a
-> 404 too. The lookup form was run against `gitea.com/gitea/tea` on Gitea
-> `1.27.0+dev` and printed `0.15.1`.
 
 ## 3. Replace the binary
 
@@ -174,20 +168,36 @@ journalctl -u nxdns -f
 
 ### Docker
 
+The compose file names the release it belongs to, so upgrading the image is upgrading the file. A new release's compose file can change more than the image — the `command:` line or a port — so both ways below replace yours: if you edited it by hand, carry your edits over to the new one. To keep your own file as it is, skip the `curl` or the `git checkout` and set `NXDNS_VERSION=$VERSION` on the `pull` and `up` lines instead: it overrides the release the file names. A compose file from `v0.0.22` or earlier defaults to `:latest`, so with one of those the variable is needed on both lines, or `pull` and `up` can land on different images if a release happens between them.
+
+With the single downloaded file of [Install with Docker](install-with-docker.md), fetch the new release's copy over it in its directory, then pull and recreate:
+
 ```sh
-NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml pull
+cd ~/nxdns
-NXDNS_VERSION=$VERSION docker compose -f deploy/docker/compose.yaml up -d
+curl -fLO "$BASE/raw/tag/v$VERSION/deploy/docker/compose.yaml"
+docker compose pull
+docker compose up -d
+```
+
+With a checkout of the repository, fetch the tag and move to it, from the repository root:
+
+```sh
+git fetch origin tag "v$VERSION"
+git checkout "v$VERSION"
+docker compose -f deploy/docker/compose.yaml pull
+docker compose -f deploy/docker/compose.yaml up -d
 ```
 
 Compose recreates the container against the same `nxdns-data` volume. What happens to the file in `etc-nxdns` depends on the `command:` in your compose file: with the shipped `run --config=/etc/nxdns/config.zon` the file is the configuration and the restart reconciles onto it; without it, the database in the volume is the configuration and the file is read by nothing.
 
-Set `NXDNS_VERSION` on both lines, or export it. Without it the compose file falls back to `:latest`, and `pull` and `up` could then land on different images if a release happens between them.
+> Not run on this host: `pull` and `up` were not run against a new release.
-
+> What was run on 2026-10-20 is `docker compose -f deploy/docker/compose.yaml
-> Not run on this host: `pull` needs a published image, and there is none.
+> config` on a tree whose `build.zig.zon` declares `0.0.22`, which resolves
-> What was run is `docker compose -f deploy/docker/compose.yaml config`, which
+> the variables without contacting a registry: no variable gave
-> resolves the variables without contacting a registry: `NXDNS_VERSION=0.0.1`
+> `image: git.mial.net/mokhtar/nxdns:0.0.22`, and `NXDNS_VERSION=0.0.21` gave
-> gave `image: git.mial.net/mokhtar/nxdns:0.0.1`, and no variable at all gave
+> `image: git.mial.net/mokhtar/nxdns:0.0.21`.
-> `:latest`.
+> The fetch half was run the same day against `v0.0.22`: the `curl` URL
+> returned 200, and `git fetch --dry-run origin tag v0.0.22` exited 0.
 
 ## 4. Confirm the upgrade
 

docs/how-to/verify-a-release.md

@@ -97,7 +97,7 @@ curl -fsSL https://keys.openpgp.org/vks/v1/by-fingerprint/A2061F6AB24DF2C0E92346
 ```
 
 > Verified: the key is published, and that exact `curl | gpg --import` reported
-> `key 1509B54946D08A95: public key "Mokhtar Mial (pc) <mokhtar@mial.net>"
+> `key 1509B54946D08A95: public key "Mokhtar Mial (pc) <…>"
 > imported` into a clean `GNUPGHOME`.
 
 Then verify:
@@ -109,7 +109,7 @@ gpg --verify SHA256SUMS.txt.asc SHA256SUMS.txt
 ```
 gpg: Signature made Sun 09 Aug 2026 01:26:42 AM CEST
 gpg:                using EDDSA key 019D00DF8417EBFDA5471E5EF7319CC024FB5A96
-gpg: Good signature from "Mokhtar Mial (pc) <mokhtar@mial.net>" [unknown]
+gpg: Good signature from "Mokhtar Mial (pc) <…>" [unknown]
 gpg: WARNING: This key is not certified with a trusted signature!
 gpg:          There is no indication that the signature belongs to the owner.
 Primary key fingerprint: A206 1F6A B24D F2C0 E923  46FD 1509 B549 46D0 8A95
@@ -134,7 +134,7 @@ A tampered `SHA256SUMS.txt` looks like this, and exits 1:
 ```
 gpg: Signature made Sun 09 Aug 2026 01:26:42 AM CEST
 gpg:                using EDDSA key 019D00DF8417EBFDA5471E5EF7319CC024FB5A96
-gpg: BAD signature from "Mokhtar Mial (pc) <mokhtar@mial.net>" [unknown]
+gpg: BAD signature from "Mokhtar Mial (pc) <…>" [unknown]
 ```
 
 > Verified against `v0.0.1`, from a clean `GNUPGHOME` holding only the imported
@@ -309,7 +309,7 @@ Compare your two tarball hashes against two things: the published `SHA256SUMS.tx
 
 A hash that differs is a signal to check the toolchain and the environment first. A distribution's Zig package instead of the official tarball is the first thing to rule out; an unpinned Node or npm version is the next ordinary explanation, and a build run outside the normalized environment above is the next one. Rule both out before you conclude anything about the release itself.
 
-> Verified against `v0.0.1`, before the build was reproducible: the recipe ran from a fresh clone, `git verify-tag v0.0.1` printed `Good signature` under the release subkey, and the rebuilt tarball hashes did not match the published `SHA256SUMS.txt` (Node 24.14.1 against the pinned 24.19.0, a different build path, and an archive written by the host's `tar`). Releases from 0.0.17 on are built and checked by the pinned pipeline this page describes, and the cut records the local hashes in `flake.nix` before CI rebuilds them.
+> Verified on 2026-10-19 against `v0.0.21`, on x86_64 Linux with glibc, Node 24.19.0 and npm 11.17.0: from a fresh clone, `git verify-tag v0.0.21` printed `Good signature`, the freshly downloaded Zig tarball passed its `sha256sum -c`, and both rebuilt tarball hashes matched the published `SHA256SUMS.txt` and the `flake.nix` pins at the tag (`26615c4b…b3b4ce` for x86_64, `29746c9a…7f19af` for aarch64). The two `sh -c` wrappers were replaced by the same nine-variable `env -i` and a `022` umask set in the calling shell. The host was the maintainer's own machine, in a fresh directory with a private `--cache-dir`, so this run shows the recipe works; it is not a third-party reproduction. An earlier run against `v0.0.1`, before the build was reproducible, did not match.
 
 ## If a check fails
 

docs/reference/configuration.md

@@ -207,6 +207,8 @@ A `tls://` upstream's host must be an IP literal. nxdns does not resolve an upst
 
 A DoH upstream is not affected: `https://` goes through the HTTP client, which resolves normally, so a hostname there is the usual form.
 
+The DoH client speaks HTTP/1.1 only (HTTP/2 is out of scope), so a DoH server that requires HTTP/2 fails every exchange. Quad9 is one: `https://dns.quad9.net/dns-query` answers HTTP/1.1 with `505` (checked 2026-10-19). Reach Quad9 over DoT instead, as in the example above.
+
 ### clients
 
 Known clients with a fixed group assignment.
@@ -425,7 +427,7 @@ The smallest file that passes validation: a `default` group and one enabled upst
 ```zon
 .{
     .groups = .{ .{ .name = "default" } },
-    .upstreams = .{ .{ .url = "https://dns.quad9.net/dns-query" } },
+    .upstreams = .{ .{ .url = "https://cloudflare-dns.com/dns-query" } },
 }
 ```
 
@@ -483,7 +485,7 @@ The smallest file that passes validation: a `default` group and one enabled upst
     // tls_name is needed when a tls:// upstream is written as an IP
     // literal, so certificate verification has a DNS name to match.
     .upstreams = .{
-        .{ .url = "https://dns.quad9.net/dns-query", .priority = 10 },
+        .{ .url = "https://cloudflare-dns.com/dns-query", .priority = 10 },
         .{ .url = "tls://9.9.9.9:853", .priority = 20, .tls_name = "dns.quad9.net" },
     },
 

flake.nix

@@ -4,14 +4,14 @@
   inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
 
   outputs =
-    { nixpkgs, ... }:
+    { self, nixpkgs, ... }:
     let
       # `zig build cut` rewrites the lines between the delimiters by text replacement.
       # BEGIN GENERATED BY zig build cut
-      version = "0.0.19";
+      version = "0.0.23";
       hashes = {
-        "aarch64-linux" = "sha256-sNIjsnJ5TPRpSE/wDtwb0j7lYyyMXCPMwc7IAv5sy3s=";
+        "aarch64-linux" = "sha256-VjvjI8bvFgHDpUcjOqcNysw4RQchCqVNLH78+DMUgFU=";
-        "x86_64-linux" = "sha256-SvWV8lmyJWceVBhuAOtldMbwGA8hLx/puJVda2t0sCg=";
+        "x86_64-linux" = "sha256-JV/6gbCc2Ewl6SxpnDuL5Z6ivCh6zNymLtgfUhnYUq8=";
       };
       # END GENERATED BY zig build cut
 
@@ -46,6 +46,9 @@
             runHook preInstall
 
             install -Dm755 nxdns $out/bin/nxdns
+            install -Dm644 nxdns.service $out/lib/systemd/system/nxdns.service
+            substituteInPlace $out/lib/systemd/system/nxdns.service \
+              --replace-fail "ExecStart=/usr/local/bin/nxdns run" "ExecStart=$out/bin/nxdns run"
             install -Dm644 LICENSE $out/share/doc/nxdns/LICENSE
             install -Dm644 THIRD-PARTY-NOTICES $out/share/doc/nxdns/THIRD-PARTY-NOTICES
 
@@ -110,5 +113,12 @@
       packages = nixpkgs.lib.genAttrs systems (system: {
         default = package system;
       });
+
+      nixosModules.default = import ./nix/module.nix { inherit self; };
+
+      checks.x86_64-linux = import ./nix/tests.nix {
+        inherit self;
+        pkgs = nixpkgs.legacyPackages.x86_64-linux;
+      };
     };
 }

nix/module.nix

@@ -0,0 +1,105 @@
+# `services.nxdns`: the packaged unit, run in file mode against a configuration
+# that the pinned binary accepted at build time.
+#
+# The unit is `deploy/systemd/nxdns.service` from the release tarball, installed
+# by the package and picked up through `systemd.packages`. This module only adds
+# a drop-in, so the hardening lives in that one file and a change to it reaches
+# NixOS with the release that ships it.
+{ self }:
+{
+  config,
+  lib,
+  pkgs,
+  ...
+}:
+let
+  cfg = config.services.nxdns;
+
+  # Nix treats the ZON as opaque data, so without this a configuration the
+  # pinned binary rejects evaluates, builds and deploys, and fails only when
+  # the service starts. `import` into a throwaway data directory is the offline
+  # validator: it shares the parser, the validator and the reconcile engine
+  # with `run`, and its delete gate is inert on an empty database. `check` is
+  # not usable here: it probes every upstream, and the build sandbox has no
+  # network.
+  validatedConfig =
+    pkgs.runCommand "nxdns-config.zon"
+      {
+        nativeBuildInputs = [ cfg.package ];
+        src = cfg.configFile;
+      }
+      ''
+        mkdir -p "$TMPDIR/validate"
+        nxdns import --data-dir "$TMPDIR/validate" "$src"
+        cp "$src" "$out"
+      '';
+in
+{
+  options.services.nxdns = {
+    enable = lib.mkEnableOption "nxdns, a DNS sinkhole for a household network";
+
+    package = lib.mkOption {
+      type = lib.types.package;
+      default = self.packages.${pkgs.stdenv.hostPlatform.system}.default;
+      defaultText = lib.literalExpression "nxdns.packages.\${pkgs.stdenv.hostPlatform.system}.default";
+      description = "The nxdns release to run. The default is the release of the flake revision this module comes from.";
+    };
+
+    configFile = lib.mkOption {
+      type = lib.types.path;
+      example = lib.literalExpression "./nxdns-config.zon";
+      description = ''
+        The nxdns configuration, in ZON. nxdns runs in file mode: every start
+        reconciles the database onto this file, and the web interface refuses
+        configuration edits. The build fails when `nxdns import` from
+        `package` rejects the file.
+      '';
+    };
+
+    openFirewall = lib.mkOption {
+      type = lib.types.bool;
+      default = false;
+      description = "Open port 53 over UDP and TCP. The web interface port and the DoH and DoT ports stay closed.";
+    };
+  };
+
+  config = lib.mkIf cfg.enable {
+    environment.etc."nxdns/config.zon".source = validatedConfig;
+
+    users.users.nxdns = {
+      isSystemUser = true;
+      group = "nxdns";
+      description = "nxdns DNS sinkhole";
+    };
+    users.groups.nxdns = { };
+
+    systemd.packages = [ cfg.package ];
+
+    systemd.services.nxdns = {
+      wantedBy = [ "multi-user.target" ];
+
+      # The unit file does not change when only the configuration does, so
+      # without this a configuration-only switch restarts nothing and the old
+      # configuration keeps serving with no error anywhere.
+      restartTriggers = [ validatedConfig ];
+
+      serviceConfig = {
+        # The empty entry clears the packaged ExecStart: a drop-in cannot
+        # replace it otherwise.
+        ExecStart = [
+          ""
+          "${lib.getExe cfg.package} run --config=/etc/nxdns/config.zon"
+        ];
+        # `environment.etc` creates /etc/nxdns and owns what is in it. A
+        # ConfigurationDirectory would make systemd adjust the same directory
+        # at every start.
+        ConfigurationDirectory = "";
+      };
+    };
+
+    networking.firewall = lib.mkIf cfg.openFirewall {
+      allowedUDPPorts = [ 53 ];
+      allowedTCPPorts = [ 53 ];
+    };
+  };
+}

nix/tests.nix

@@ -0,0 +1,97 @@
+# The acceptance checks for `services.nxdns`. Neither runs in CI: the VM test
+# needs KVM, and both need a nixpkgs closure the shared runner should not
+# download on every push. AGENTS.md says when to run them.
+{ self, pkgs }:
+let
+  lib = pkgs.lib;
+
+  # A local record answers without any upstream, which the test VM cannot reach.
+  configWith =
+    address:
+    pkgs.writeText "nxdns-test.zon" ''
+      .{
+          .groups = .{ .{ .name = "default" } },
+          .upstreams = .{ .{ .url = "tls://192.0.2.1:853", .tls_name = "dns.example" } },
+          .local_records = .{ .{ .name = "nas.lan", .rtype = .a, .value = "${address}" } },
+      }
+    '';
+
+  rejected = pkgs.writeText "nxdns-rejected.zon" ''
+    .{
+        .groups = .{ .{ .name = "default" } },
+        .upstreams = .{ .{ .url = "tls://192.0.2.1:853", .tls_name = "dns.example" } },
+        .rules = .{ .{ .group = "default", .pattern = "(", .kind = .regex, .action = .block } },
+    }
+  '';
+
+  machineWith = configFile: {
+    imports = [ self.nixosModules.default ];
+    services.nxdns = {
+      enable = true;
+      inherit configFile;
+    };
+    boot.loader.grub.enable = false;
+    fileSystems."/" = {
+      device = "none";
+      fsType = "tmpfs";
+    };
+    system.stateVersion = lib.trivial.release;
+  };
+in
+{
+  nixos-module = pkgs.testers.runNixOSTest {
+    name = "nxdns-module";
+
+    nodes.machine =
+      { pkgs, ... }:
+      {
+        imports = [ self.nixosModules.default ];
+        services.nxdns = {
+          enable = true;
+          configFile = configWith "192.0.2.5";
+          openFirewall = true;
+        };
+        environment.systemPackages = [ pkgs.dnsutils ];
+        specialisation.changed.configuration.services.nxdns.configFile = lib.mkForce (
+          configWith "192.0.2.6"
+        );
+      };
+
+    testScript = ''
+      machine.wait_for_unit("nxdns.service")
+      machine.wait_until_succeeds("dig +short @127.0.0.1 nas.lan A | grep -x 192.0.2.5")
+
+      # The packaged unit runs, with the module's drop-in on top of it.
+      unit = machine.succeed("systemctl cat nxdns.service")
+      assert "/lib/systemd/system/nxdns.service" in unit, unit
+      assert "ProtectSystem=strict" in unit, unit
+      assert "run --config=/etc/nxdns/config.zon" in unit, unit
+      machine.succeed("systemctl show -p User --value nxdns.service | grep -x nxdns")
+      machine.succeed("test -z \"$(systemctl show -p ConfigurationDirectory --value nxdns.service)\"")
+      machine.succeed("systemctl show -p ReadOnlyPaths --value nxdns.service | tr ' ' '\\n' | grep -Fx /etc/nxdns")
+      machine.succeed("stat -c %U /etc/nxdns | grep -x root")
+      machine.succeed("iptables -S nixos-fw | grep -- '-p udp .*--dport 53 '")
+      machine.succeed("iptables -S nixos-fw | grep -- '-p tcp .*--dport 53 '")
+
+      # A switch that changes only the configuration restarts the service.
+      before = machine.succeed("systemctl show -p InvocationID --value nxdns.service").strip()
+      machine.succeed("/run/current-system/specialisation/changed/bin/switch-to-configuration test")
+      machine.wait_for_unit("nxdns.service")
+      after = machine.succeed("systemctl show -p InvocationID --value nxdns.service").strip()
+      assert before != after, "a configuration-only switch did not restart nxdns"
+      machine.wait_until_succeeds("dig +short @127.0.0.1 nas.lan A | grep -x 192.0.2.6")
+    '';
+  };
+
+  # A configuration the pinned binary rejects must fail the system build, and
+  # the build log must carry the binary's own reason.
+  nixos-module-rejects-config =
+    let
+      system = pkgs.nixos (machineWith rejected);
+      failure = pkgs.testers.testBuildFailure system.config.environment.etc."nxdns/config.zon".source;
+    in
+    pkgs.runCommand "nxdns-module-rejects-config" { } ''
+      grep -F 'FAIL rules[0].pattern' ${failure}/testBuildFailure.log
+      touch "$out"
+    '';
+}

specs/memory-budget.md

@@ -0,0 +1,97 @@
+# Memory budget: pay for what the household uses
+
+Measured on the deployed Pi (0.0.21, aarch64, 2.85M blocklist entries, 18 days up): VmRSS 238 MiB, VmHWM 358 MiB, 207 threads, almost all private anonymous memory. `docs/explanation/performance-and-testing.md` promises "memory with ~1M blocked domains < 100 MiB". Nothing leaks. Every MiB below has a measured owner, and each owner pays for a worst case the household never reaches, or pays twice.
+
+| Owner | Resident | How it was measured |
+|---|---|---|
+| Blocklist snapshot | 71 MiB, of which ~32 MiB is ABP domains stored twice | Pi smaps; `.list` and `.wild` bodies of the ABP sources are byte-identical (md5) |
+| Fixed buffers | ~52 MiB: query-log writer queue 17.5, two TCP listeners 2 × 8.7, two UDP listeners 2 × 4.8, SSE hub 3.6, cache slots 3.3 | Sizes from source; resident at once because safe builds fill new allocations with 0xaa |
+| Idle worker threads | ~41 MiB for ~200 parked workers | `std.Io.Threaded` never retires a worker; one burst of 64 in-flight UDP queries × 3 tasks each grows the pool for good (replica: 14 → 206 threads) |
+| Heap slabs kept after peaks | ~61 MiB free + ~8 MiB live | Decoded `SmpAllocator` free lists from two `/proc/<pid>/mem` dumps; owners from each slot's size header and a local tracing build |
+| Peak during refresh | +120 MiB over steady state | Old snapshot + new snapshot + every compiled body read at once; replica reproduced 350 MiB |
+
+Heap owners, from the dumps and the trace:
+
+- 8 KiB class, 25 MiB free: SQLite page-cache blocks (4,376 B requests in 8 KiB slots). Seven connections, no `cache_size` pragma, so each keeps the 2 MiB default, which costs ~3.7 MiB of slabs once filled.
+- 128 B + 256 B classes, ~21 MiB free: `std.Io.Threaded` `Group.Task` blocks — one per UDP query (`UdpServer.receiveLoop`), two per upstream exchange (`transport.raceUntilTagged`), two per query-log wait (`Logger.getWithin`).
+- 512 B class, 7.7 MiB free: `DnsCache.put` copies of cached responses (`src/cache/dns_cache.zig:356`).
+
+Why freed small blocks stay resident: with libc linked, `malloc` is zig's `lib/c/malloc.zig` on `std.heap.SmpAllocator`, which never unmaps a 64 KiB slab. `free` pushes onto the freeing thread's slot and `alloc` searches one other slot before it maps a new slab (`max_alloc_search = 1`), so producer/consumer pairs map more slabs than their live peak needs. Upstream: ziglang/zig#23592. Fewer tasks and smaller peaks shrink this retained memory; nothing in this spec frees it after the fact.
+
+## Decisions
+
+- Fix each owner at its root. Do not bypass the safe-build 0xaa fill: it only makes the oversizing visible.
+- Do not change the allocator in this work. Replacing `init.gpa` does not move SQLite, which calls C `malloc`. Revisit only if the measurement gate still names small-class slabs as a top owner.
+- No compiled-format change, no config-key migration, no new deadline mechanism. Each of those was considered and rejected in review: the format change makes the upgrade depend on the network, packed query-log rings cost more complexity than sizing the queue, and a socket-shutdown deadline cannot bound `connect` (`std.Io.Threaded` has no connect timeout) or the pool admission wait.
+- Not `SQLITE_CONFIG_PAGECACHE`, not a database page-size migration.
+
+## Contract
+
+### 0. Make the budget measurable first
+
+The admission budget, the listener caps, the stack size and the queue size are chosen from the Pi's real peaks, so those peaks become measurements before anything is resized:
+
+- high-water of in-flight DNS queries: `nxdns_dns_in_flight_peak`, counted around `Handler.handle`, every listener together;
+- high-water of active stream connections, per listener (today `connections` counts accepts over time, not concurrency): `nxdns_tcp_server_connections_peak` (the larger of the two address families), `nxdns_dot_server_connections_peak`, `nxdns_doh_server_connections_peak`;
+- high-water of the `std.Io.Threaded` worker pool: the process thread count. `Threaded` spawns a worker only when every worker is busy and never retires one (`Threaded.zig:2110`), so the count is that high-water already and needs no code. It counts workers, not tasks: `async` work can run inline or wait in the run queue;
+- high-water of the query-log writer queue's occupancy (not arrivals per interval: a full batch flushes early, and a disk gate or a slow write lets backlog build past one interval): `nxdns_querylog_queue_peak`. `std.Io.Queue` exposes no length, so a count beside it is raised before each insert and lowered after each removal; it can overstate the peak by the entries in transit, never understate it;
+- worker call-stack use, read from `/proc/<pid>/smaps`: each worker stack is an anonymous mapping directly above its own guard page, all of one size, and its `Rss` is what the thread has touched and still holds resident. That finds the expensive paths; it is not a depth bound. Before item 3 lowers `stack_size`, each expensive path gets a measured stack high-water, and the size keeps headroom above the deepest one. The signal stack (`std_options.signal_stack_size`) is a separate allocation for signal handling and stack traces and keeps its stdlib default unless its own use is measured.
+
+The five gauges are in `/metrics` beside the existing per-reason drop counters. One release carrying only these measurements runs on the Pi before the budget values are fixed.
+
+First reading, static x86_64 musl build, one burst of 300 UDP queries: the in-flight peak reached 64 (the UDP slot pool), threads went from 14 to 205, and of 201 worker stacks 175 touched 32 kB or less while 26 touched 2 MiB each — the same 2 MiB with only forward-zone traffic and no DoT exchange, so the TLS client is not the owner. Item 3 identifies that owner before it lowers `stack_size`: a stack smaller than that path needs is a crash, not a saving.
+
+### 1. Store each blocklist name once in memory
+
+Within one source, as compiled entries, an exact `x` in `.list` together with `x` in `.wild` means exactly what `||x^` compiles to. The compiled files, their checksum and their counts stay as they are.
+
+- The snapshot builder merges one source's sorted `.list` and `.wild` bodies into one name set with two flags per name: `exact` and `subdomains`. A name in both bodies is stored once. Sources are never merged with each other, so group eligibility and the reported source do not change.
+- `.allow` stays a separate set, and level 7 (exceptions) still runs before both flag checks.
+- Level 8 tests the full name for `exact`; level 9 tests proper parents for `subdomains`. Sources still walk in ascending index order. The reported result stays what it is today: when the `x` entry wins, the apex reports `blocklist_domain` with matched `x` and a proper subdomain reports `blocklist_wildcard` with matched parent `x`. An exception, an exact entry for the queried name or a nearer matching parent still wins first.
+
+### 2. Lower the refresh and boot peak
+
+- Boot keeps one synchronous initial load before the listeners start (`src/app.zig:692`). The startup pass uses that result for its refresh decisions and reloads only after a source actually refreshed, instead of rebuilding unconditionally (`src/filter/manager.zig:1823`).
+- `Snapshot` gets a pure `Builder`: add one source's bodies, then free them before reading the next. The old snapshot is still freed only after the swap, so a reload holds at most old snapshot + new snapshot + one source's bodies. That is the reload bound, not the process peak: refresh preparation (download, compile) can run beside a reload and holds its own allocations, and the Pi peak measurement includes it.
+
+### 3. Bound concurrent DNS work
+
+- `main` takes `std.process.Init.Minimal` and owns its `std.Io.Threaded`, with `stack_size` set from the item-0 stack measurement and a `concurrent_limit` derived from the chosen query and connection caps, the tasks each can nest, and the fixed services. The measured task high-water already counts every caller, so it is the cross-check, not a term in the sum.
+- `ConcurrencyUnavailable` is counted at every boundary where it can surface. The spec work enumerates each `io.concurrent` and `Group.concurrent` call; today the logger turns the error into an early flush without counting it (`src/storage/logger.zig:748`) and the listener races fold it into a generic failure (`src/server/listener.zig:102`). Each gets its own counter or a reason on an existing one.
+- Two kinds of bound: one cap on active DNS queries, shared by every listener (UDP and TCP, both families, DoT/DoH when enabled), and a slot cap per stream listener on accepted connections, each from the item-0 high-waters. A query takes a permit only while it does work; a keep-alive connection releases its permit between requests.
+- UDP keeps today's counted drop when no permit is free (`dropped_no_slot`). The kernel buffer does not absorb overload invisibly.
+- The upstream exchange keeps its whole-operation `Select` deadline and its distinction between peer result, budget expiry and cancellation. The thread high-water falls because the query budget is smaller, not because the race changes.
+- UDP slots and TCP connection buffers are sized from these bounds, not 64 per listener.
+
+### 4. Size the query-log buffers by use
+
+- The writer queue default (`logging.query_log_buffer_max`) comes from the item-0 occupancy high-water, with headroom stated in the reference. Drop-oldest backpressure and its counter stay.
+- The SSE hub allocates a subscriber's ring on subscribe and frees it on unsubscribe; `max_subscribers` stays a cap, not a reservation. `subscribe` becomes fallible: an allocation failure is distinct from "all slots occupied", is counted, answers with its own error rather than "live stream is full" (`src/web/handlers/live.zig:75`), and leaves the slot reusable. Overflow keeps today's contract: the hub ends that subscriber's stream so `EventSource` reconnects (`src/web/sse.zig:8`).
+
+### 5. SQLite caches chosen per connection
+
+- An explicit runtime-connection initialization step sets `cache_size` from a constant per role (query-log writer, retention, readers, config). `Db.open` stays free of pragmas, because migration callers control when pragmas apply (`src/storage/db.zig:302`). Every runtime handle reads its value back.
+
+### 6. DNS cache bounded in bytes
+
+- `DnsCache` keeps its entry count and adds a budget on live response bytes, tracked separately from `memoryBytes` (which also counts slots and index). `put` evicts by CLOCK until the copy fits. A response larger than the whole budget is not cached and is counted. A replacement keeps the old entry if allocating the new copy fails. A failed copy allocation is counted; today `handler.zig:918` discards it with `catch {}` and `DnsCache.Stats` has no field for it. The answer already obtained is still sent. The budget, the live bytes and both counters appear in cache diagnostics.
+- New config key `cache.max_bytes`, through the config field checklist. Default from the Pi's measured live response bytes.
+
+## Propagation
+
+- Config: `cache.max_bytes`; the changed default of `logging.query_log_buffer_max`. Config field checklist for the new key: model key + round-trip drift guards, validation + reference, settings API view, openapi.yaml, admin types + settings control, contract samples regenerated, docs config reference, CHANGELOG.
+- Diagnostics: the item-0 counters, `ConcurrencyUnavailable`, oversized cache responses, cache live bytes and budget.
+- Docs: `performance-and-testing.md` states the measured RSS, refresh peak and thread count at the Pi's blocklist size, replacing the unmeasured target.
+
+## Acceptance
+
+- [ ] Matcher equivalence: a deterministic corpus (hosts, domains, ABP with `||x^`, `*.x` alone, `x` alone, `x` with `*.x`, `@@`, an exact entry for a subdomain of a wildcard, nested wildcards `*.x` and `*.y.x`, overlapping sources) gives the same `Decision` for every probe name before and after item 1.
+- [ ] Builder: a reload holds at most one source's bodies (tested through an allocator that records peak bytes).
+- [ ] Boot: one snapshot build when no source refreshes; listeners never start without the initial load having run.
+- [ ] Measurements: each of the five item-0 gauges rises under a scripted load and is visible in `/metrics`; the thread count and the worker stack residency are recorded from `/proc` beside them.
+- [ ] Bounds: with the query budget full, a further UDP query is dropped and counted; a stream listener at its slot cap refuses and counts; a keep-alive connection between requests holds no query permit; `ConcurrencyUnavailable` is counted at each enumerated boundary, including the logger wait and the listener races.
+- [ ] SSE: no ring is allocated without a subscriber; an allocation failure is counted, reported as its own error and leaves the slot reusable; overflow still ends the stream.
+- [ ] SQLite: each runtime handle reports its configured `cache_size`; migrations open without it.
+- [ ] DNS cache: puts beyond the byte budget evict; live bytes never exceed it; an oversized response is skipped and counted; a failed replacement keeps the old entry; a failed copy allocation is counted and the answer is still sent.
+- [ ] Suites: `zig build test` and `-Dintegration` 0 failed; admin typecheck/vitest/prettier clean; contract samples regenerated.
+- [ ] Measurement record, on the Pi with the same blocklists: steady VmRSS after 24 h, VmHWM across one scheduled refresh (including its concurrent download and compile), thread count, and a decoded heap dump per size class, each beside the baseline in the table above. The owner chose a record, not a pass threshold: each owner must be reduced as specified, and the numbers go into `performance-and-testing.md`. If small-class slabs remain a top owner, the allocator question reopens with that data.

specs/milestone-3.md

@@ -810,3 +810,39 @@ The implementation matches the spec with these review-driven refinements (three
 - `dot_client.zig`: handshake `ReadFailed`/`WriteFailed` unwrap the stream's stored cause through `transport.mapLocal` first, and certificate-bundle `OutOfMemory` is a local resource, not a peer fault.
 - `transport.zig`: `Endpoint.parse` rejects userinfo/query/fragment delimiters (`@`, `?`, `#`) in the authority, and `?`/`#` in a DoH path.
 - `dot_client.zig` (follow-up, tls_name commit): `DotClient.init` takes a `tls_name`; it is the SNI and certificate-verification name, while the dial target stays `endpoint.host`. Empty keeps the endpoint host, which is the behavior described above. The same follow-up fixed a send bug this file had from the start, invisible until a DoT handshake first succeeded: `tls.Client.flush` only encrypts into the socket writer's buffer and never flushes it, so the query never left the process and the peer eventually closed the connection (`ReceiveFailed`/`EndOfStream`). `TlsStream.flush` now does both flushes and `exchange` calls it; a hermetic loopback test in `tls_client_integration_test.zig` covers it.
+
+## Addendum (2026-09-12): an upstream episode follows health, and a peer fault carries its cause
+
+Observed on the Pi over three days (nxdns 0.0.17 to 0.0.19): 20 `upstream.exchange` warning episodes, 17 with one occurrence, 15 resolved within ten seconds; since the 0.0.20 restart 1229 successes, 0 failures, 401 silent stale-session redials. `Pool.recordFailure` reports an episode on every failed exchange and `recordSuccess` resolves it on the next success, while `health.State` trips backoff only at `failure_threshold` consecutive failures. Two rules over one failure stream, and the Diagnostics page shows the looser one. Second gap: the episode detail is `upstream 'tls://1.1.1.1:853' failed: SendFailed`; the DoT client unwraps the concrete cause into `Transact.Failure.cause` and then returns a bare `PeerFault` error to the pool, so the cause is lost before anything records it. Codex reviewed the design (thread 01a09648) and its eight findings are folded in below.
+
+### 1. Health owns "failing"; the episode is a projection of its transitions
+
+- `health.State.recordFailure(at, fault, cfg, rand)` and `recordSuccess(at, cfg)` return an `Effect`: `{ revision: u64, state: union(enum) { clear, tripped: Fault } }`. `revision` is a per-state counter incremented by every mutation. The effect is the complete desired state of the episode after the mutation, never a transition: `tripped` when `consecutive_failures >= cfg.failure_threshold` (the predicate line 142 already uses, exposed as `tripped(cfg)`; not `available()`, which turns true when a backoff expires before recovery is proven), `clear` otherwise. The fault in `tripped` is the effective last fault the state holds after the mutation, not the incoming one: a stale failure (case 1 at `recordFailure`) never displaces a newer cause, so the episode detail cannot regress to an older one. A tagged union, not an action enum with a payload field, so a report without a fault is unrepresentable.
+- The pool projects every effect onto the store: `tripped` reports, `clear` resolves. A resolve when nothing is open issues no SQL (`storage/events.zig`), so the steady state costs nothing. Because every effect carries the whole desired state, last-writer-wins by revision is correct: the pool applies effects per entry in revision order under a separate `diagnostics_mutex` with a per-entry `applied_revision`, and drops an effect whose revision is below the applied one. Codex's reordering case (a failure trips health, a newer success clears it and its resolve reaches the store first, then the delayed failure arrives) is dropped instead of opening an episode nothing will resolve; and the converse cases, where a `none` transition would have overtaken a required report or resolve, cannot occur because there is no `none`. Health mutates under `Pool.mutex` as today and the effect is computed there; the store call runs after that mutex is released. Lock order is pool mutex, then nothing; diagnostics mutex, then store mutex. No task holds the pool mutex while it takes either of the other two.
+- Projection also reconciles a latched store failure for what an entry owns: a `clear` that failed to write is retried by the next `clear` the entry projects, and a `tripped` by the next failure. What no entry owns is handled by the reconciliation points below.
+- A success that leaves the state tripped (a stale success behind a newer failure, health case 1) projects nothing: `recordSuccess` returns `?Effect`, null in that case. The card is already open from the failure that tripped it, a report on a success would count a successful exchange as an occurrence, and a no-op that advanced the revision would reintroduce the dropped-report hole. Such a success does not touch `applied_revision`.
+- Reconciliation at the two points revision order cannot reach. (1) At boot, once the first generation is built: `store.resolveExcept(.upstream_exchange, kept = the enabled urls of the pool)` closes persisted episodes for upstreams that are gone, disabled or failed to build, and `Pool.reconcile(io)` projects every entry's current effect, which on a fresh pool resolves the rest. A re-asserted `tripped` must not count as an occurrence: the store gains `ensureOpen`, which opens the episode with the detail when none is active and otherwise leaves `occurrences` and `last_seen` untouched; `reconcile` uses it, a recorded failure keeps using `report`. (2) At every retirement of a displaced generation, both the pinned path (`Owner.release` at `refs == 0`, the point after which no exchange of it can still project) and the idle path (`Owner.replace` handing a zero-ref generation back to the caller), the same two calls run against the current generation. One owner function does the retire-and-reconcile and both paths call it, so a third retirement site cannot forget it. A removed or disabled upstream's episode is closed at the next reconciliation point; if the store had latched a failure at that moment, the next reconciliation point retries it. Before this addendum such an episode was never closed at all. `resolveExcept` is the right call for this code: the pool is the only reporter of `upstream_exchange`, unlike `configuration.load`, whose scoped rule in owner.zig stays. The previous "one exchange of lag" claim is withdrawn; the retirement point is exact.
+- Occurrences are the count of applied reports. Under concurrent reordering an older report behind a newer one is dropped, so the count can undercount; the health counters on `/metrics` are exact. Stated, not fixed: exact occurrence counts would need the store to accept out-of-order increments, and nobody reads the count as a metric.
+- The store folds a repeated report into the open episode (`occurrences`, `last_seen`, detail), so failures during backoff raise counts on one card. A warning card now means exactly what the pool means: it stopped trusting this endpoint.
+- Boundary, stated on purpose: an endpoint failing every other exchange never trips, because every newest success clears the count, so it opens no episode and `/api/health` stays `ok`. The rolling success rate on `/metrics` shows it. Diagnostics is for episodes, not chronic rates; a second predicate for that is not added.
+- No new configuration. The threshold that exists is the threshold, and it is at least 1: `health.Config` is validated where it is constructed (a comptime check on the default; if it is user-configurable, the config loader refuses 0), because a tripped state with no recorded failure has no fault to report.
+
+### 2. A peer fault is a value between the leaf client and the pool
+
+- `transport.zig` gains `Fault = struct { kind: PeerFault, cause: anyerror }`, `Outcome = union(enum) { reply: []u8, fault: Fault }`, and a leaf interface `Leaf` with `exchangeFn(ptr, io, query, response_buf) (LocalResource || Cancellation)!Outcome`. `kind` is the taxonomy as today; `cause` is the concrete unwrapped error (`BrokenPipe`, `ConnectionResetByPeer`, `TlsAlert`, `HttpConnectionClosing`, ...). No phase field: the taxonomy already names it for every kind that has one, and `TlsFailed` cannot say where it failed. No HTTP status number for `HttpStatus`; the claim is limited to concrete I/O causes. No optional metadata bag.
+- `dot_client.zig` and `doh_client.zig` implement `Leaf`; a `Transact.Failure` becomes a returned `Fault` instead of a thrown error. The DoT stale-session redial is unchanged and still counted through `reuse_recoveries`, never as a fault. The retry list is not widened; with the cause recorded, a stale-session cause outside the four lifecycle errors shows in the next episode and the list grows on evidence.
+- `Pool` consumes `Leaf` for its slots and keeps implementing `transport.Client` toward the handler, the forward client, and the fakes that sit on that side. The outer error set, `transport.group`, and the handler are untouched; the pool returns `last_fault.kind` as the error it returns today. The race harness stays generic: the pool races the leaf through a function whose error set adds `Timeout`; a leaf never returns `error.Timeout` itself (its own timeouts are faults), so `error.Timeout` out of the race is the harness. With `race == .expired and !truncated` the pool synthesizes `Fault{ .kind = error.Timeout, .cause = error.Timeout }` and records it: an untruncated attempt expiry is peer evidence. With `truncated` it stays `error.BudgetExhausted`, unattributed, exactly as today.
+- Health stores the effective fault (kind and cause names, `error_name_capacity` sized for both). Every surface that printed the taxonomy prints one text: `<Kind> (cause <Cause>)`, e.g. `SendFailed (cause BrokenPipe)`. Those surfaces are the episode detail, `upstream 'tls://1.1.1.1:853' failed: SendFailed (cause BrokenPipe)`, `Snapshot.last_error` and its one reader, the `nxdns check` FAIL line, and the pool's debug `AttemptFailure` line. `/api/health`, `/api/upstreams` and `/metrics` carry no fault text today and gain none here.
+- `transport.raceUntilTagged` becomes generic over the raced function's own error set: it returns `RacedError(f) || error{ Timeout, SystemResources, Canceled }`, so the pool's attempt wrapper returns `LeafError || error{Timeout}` with no `@errorCast` and no `unreachable`; `error.Timeout` out of the race is the harness by type, which sets `race = .expired` before returning it; the loop's catch asserts that and handles it in the expiry branch only, so a regression of the harness contract traps in ReleaseSafe instead of blaming an endpoint. The compiler enforces at the race boundary that a leaf cannot throw a `PeerFault` or `BudgetExhausted`. A leaf's own timeout is a returned `Fault{ .kind = error.Timeout, .cause = <leaf cause> }` with `race == .completed`, and a pool test proves it is recorded with the leaf's cause and never confused with the harness expiry.
+- The blocklist fetcher's `last_failure` side field stays as it is; same shape, other subsystem, its own addendum.
+
+### Tests
+
+- health: one failure returns `clear`; the second returns `tripped`; a success on a tripped state returns `clear`; a stale success behind a newer failure projects nothing; a stale failure returns the newer effective fault; revisions strictly increase.
+- pool: a `tripped` effect delivered after a newer `clear` effect is dropped and no episode is open afterwards; a `clear` delivered after a newer `tripped` is dropped and the episode stays open; an episode open in the store before the pool exists is resolved by boot reconciliation, one for a url the pool does not have and one for a url it has; a retired generation's late `tripped` is corrected by the reconciliation at its last release, and the idle-replace path reconciles too; a reconcile of a tripped entry with an open episode moves neither `occurrences` nor `last_seen`; a stale success while tripped projects nothing and the occurrence count does not move; one failed exchange opens no episode and the second opens one that the next success resolves; a truncated expiry records nothing and returns `BudgetExhausted`; an untruncated expiry records `Timeout (cause Timeout)`; a leaf fault of kind `Timeout` is recorded with the leaf's cause.
+- clients: the DoT integration tests assert the returned `Fault` with kind and cause, not the kind alone; a DoH integration test drives `DohClient.exchange` against a loopback port that is bound but never listens and stays bound for the test's duration, so the connect is refused deterministically with no port reuse race, and asserts the cause survives to the returned `Fault`; the stub-connection helper tests keep their causes.
+- surfaces: the `nxdns check` test asserts the `<Kind> (cause <Cause>)` text.
+
+### Out of scope
+
+An idle timer on the DoT session, a chronic-rate predicate, the fetcher migration, and any change to the DoT retry list.

specs/milestone-5.md

@@ -1390,3 +1390,13 @@ Change:
 - Tests: `mapError` tests unchanged; a test that a head failure with a stashed `ConnectionResetByPeer` yields `last_failure.cause == error.ConnectionResetByPeer` and phase `receive` (the doh tests' stub connection pattern); a test that a body failure records `bytes_read` and `receive_body`; a `reportDownloadFailure` format test with the fixed text `blocklist source 3 'ads' 'https://lists.example': download failed: ReceiveFailed (receive_body, cause HttpChunkTruncated, http 200, 1048576 bytes, 812 ms)`; the existing secrecy assertions on the label stay.
 
 Out of scope: a TLS handshake failure stays `TlsInitializationFailed`, because `std.http.Client` collapses every handshake fault at `Client.zig:1470` before this code sees it; distinguishing an alert from a certificate failure would mean driving `std.crypto.tls.Client` by hand. Retries and a per-source backoff are not added.
+
+## Addendum (2026-09-12): a blocklist download never reuses a pooled connection
+
+The line above found the cause the same day nxdns 0.0.19 reached the Pi. A manual refresh at 14:28 failed the same four sources with one record each: `ReceiveFailed (receive, cause HttpConnectionClosing, http none, 0 bytes, 0 ms)`. Under one millisecond of elapsed time, and no response head byte after the request was sent, means the peer had already closed the socket. `std.http.Client` returns `HttpConnectionClosing` when a reused connection hits EOF before the first head byte (`std/http.zig:400`), and `std/http/test.zig:1331` pins that a client never retries a stale pooled connection. The startup pass at 13:03 had fetched `big.oisd.nl` successfully and the client parked that keep-alive connection in `fetch_http`'s pool; the server closed it long before 14:28, and the next request to the same host picked it up. The same shape explains every earlier observation: the first fetch to a host in a process succeeds, a later fetch to that host fails, and the daily pass reuses connections a day old. The `TlsFailed` variant of the older lines is the same dead socket failing inside the TLS layer instead of at the head.
+
+Change: `fetcher.fetch` requests with `.keep_alive = false`, and `fetch_http` in app.zig is created with `connection_pool.free_size = 0`, so `release` destroys every connection regardless of the request flag (`Client.zig:133`), including the one a failed send leaves in the `.ready` state that `Request.deinit` would otherwise pool, and no connection is ever taken from that pool. The flag alone is not enough: `Client.request` takes a pooled connection before it stores the flag (`Client.zig:1725`, `:1742`). The request then carries `connection: close`, the client marks the connection closing at the head, and `release` destroys it instead of pooling it (`Client.zig:133`, `:1167`). A blocklist download is one bulk transfer per source per day; there is nothing to gain from a pooled connection and a stale one costs a source for a day. `fetch_http` stays a separate client from `dns_http`, so the DoH pool is untouched.
+
+Tests: an integration test against `HttpFixture` where the server closes the socket after each response: two consecutive fetches to the same URL both succeed. Without the change the second fails with `HttpConnectionClosing`, which is the exact record the Pi produced. A unit test that the request header the fetcher sends carries `connection: close` if the fixture route is the cheaper path.
+
+Out of scope: a retry on `HttpConnectionClosing` for the DoH client, which keeps pooling by design and has its own health logic.

specs/nixos-module.md

@@ -0,0 +1,30 @@
+# NixOS module: `services.nxdns`
+
+Owner ruling (2026-10-20): ship `nixosModules.default` from the flake, with the options `enable`, `package`, `configFile` and `openFirewall`, file mode only, no `settings` attrset, and the shipped `nxdns.service` reused through `systemd.packages` plus a drop-in. The owner left Nix in CI, the VM test and a distinct exit code for `SchemaTooNew` to the implementer, within one constraint: the self-hosted Gitea runner (`../mial.net/manifests/gitea/`) is shared and must not be monopolized.
+
+## Why
+
+The flake already shipped a package and had one consumer, `rpi.mial.net`, which wrote its own module. Four parts of that module are not specific to the Pi, and two encode incidents: the build-time `nxdns import` validation, and `restartTriggers`, without which a configuration-only deploy restarted nothing (2026-08-16). A package with no service is half an install on NixOS.
+
+## What ships
+
+- `nix/module.nix`. `configFile` goes through `nxdns import --data-dir $TMPDIR/validate` from `package` in a `runCommand`, and the result is `/etc/nxdns/config.zon`. The `nxdns` system user and group. `systemd.packages = [ package ]`, and a drop-in that clears `ExecStart` and sets `run --config=/etc/nxdns/config.zon`, clears `ConfigurationDirectory` because `environment.etc` owns `/etc/nxdns`, adds `wantedBy` and `restartTriggers`. `openFirewall` opens 53/udp and 53/tcp and nothing else.
+- The package installs the tarball's `nxdns.service` to `lib/systemd/system/` with `ExecStart` pointed at its own binary. `--replace-fail` makes a changed `ExecStart` line in the unit a package build failure, not a silent mismatch.
+- Not shipped: database mode, a Nix-to-ZON renderer, and the Pi's choices about its own resolver, which stay in `rpi.mial.net`.
+
+## Decisions left to the implementer
+
+- **No Nix in CI.** No job installs Nix today. The checks need a nixpkgs closure and, for the VM test, KVM; downloading that on every push costs the shared runner disk and network, and the runner's cache already caused a disk-pressure eviction on 2026-08-23. The cut's existing `nix flake check --no-build` evaluates both checks at every release, which catches a module that no longer evaluates.
+- **The VM test runs on the developer's host.** `checks.x86_64-linux.nixos-module` and `checks.x86_64-linux.nixos-module-rejects-config` are built locally, with KVM, after any change to the module, the checks, the package or the unit. AGENTS.md says so. Whether the runner node exposes `/dev/kvm` was not checked, and nothing here depends on it.
+- **No new exit code.** `SchemaTooNew` is already in `config/faults.zig`, so `run` exits 2 for it and the unit's `RestartPreventExitStatus=2 64` stops instead of restart-looping (`src/app.zig` tests the mapping). A NixOS rollback onto a migrated database therefore stops the unit, and `docs/how-to/upgrade.md` describes the recovery. The exit-1 behaviour in the Pi module's notes dates from 0.0.2.
+
+## Acceptance
+
+- `nix flake check --no-build` passes.
+- `nixos-module`: a VM with the module and `openFirewall` answers `nas.lan` from a local record; `systemctl cat` shows the packaged unit and the drop-in; the service runs as `nxdns`; the effective `ConfigurationDirectory` is empty, `ReadOnlyPaths` still holds `/etc/nxdns`, and `/etc/nxdns` belongs to root; port 53 is open over UDP and TCP; `switch-to-configuration test` into a specialisation that changes only `configFile` gives a new `InvocationID` and the new answer. Without `restartTriggers` the test fails on the restart assertion, and without the `ConfigurationDirectory` clear it fails on that one (each checked once by removing the line).
+- `nixos-module-rejects-config`: a system build with an invalid regex rule fails, and its log carries `FAIL rules[0].pattern`.
+- `docs/how-to/install-with-nix.md` documents the module, the options, the store-readability of `configFile`, and rollback.
+
+## After the release
+
+`rpi.mial.net` replaces its module with `services.nxdns` and keeps its Pi-only parts, through the rpi-nixos-iac peer.

specs/release-cut.md

@@ -102,3 +102,13 @@ On a resumed cut — the bump commit exists and its tag does not — the pin sta
 ### Tests
 
 The `gates.yml` env parser against a fixture and against the real file; the environment builder, asserting the nine keys and nothing else; and the `sh` wrapper, asserting it applies the umask, keeps the real command as the direct child, and reports its exit code. The stage's remaining steps are process spawns of `npm`, `zig` and `nix`, which are not mocked — the same rule the rest of this file follows.
+
+## Addendum: the compose file names its release (post-0.0.22)
+
+Owner question (2026-10-20), on the site's `NXDNS_VERSION=0.0.22 docker compose up -d`: "why do we need the NXDNS_VERSION to pull up a docker compose file? shouldnt the version be specified in the file already??" Until 0.0.22, `deploy/docker/compose.yaml` read `${NXDNS_VERSION:-latest}`, so the file fetched from a tag ran whatever `:latest` was that day, and every install and upgrade command had to carry the variable.
+
+The image default is now the release itself: `${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-<version>}}`. `NXDNS_VERSION` and `NXDNS_IMAGE` stay as overrides.
+
+- **The bump moves it.** This amends step 2 of the pin stage: the cut reads the compose file and computes its rewrite before anything is written, so a file without exactly one `${NXDNS_VERSION:-…}` default is a refusal on a clean tree. Then `build.zig.zon` and the compose file are both written atomically and read back. Step 5's commit takes `build.zig.zon`, `flake.nix` and `deploy/docker/compose.yaml`, and the working-tree diff is asserted to be a subset of those three paths.
+- **One guard, on every commit.** A cut-tool test reads the real `build.zig.zon` and `deploy/docker/compose.yaml` and asserts that the default equals the declared version. It runs in `zig build test`, so a hand edit of either file fails CI on that commit. The tag's tree is the bump commit's tree, which passed CI before the tag was made, so `verify-dist` gets no second copy of the check.
+- **The docs drop the variable from the plain flow.** README, install-with-docker and upgrade run `docker compose up -d`; the variables are documented as overrides, and the pages say that a compose file from `v0.0.22` or earlier still defaults to `:latest`.

src/app.zig

@@ -482,7 +482,19 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
     // Two HTTP clients on purpose. A blocklist download streams tens of
     // megabytes and holds its connection for the whole of it; DoH queries must
     // not queue behind that, and the two have nothing to share but a type.
-    var fetch_http: std.http.Client = .{ .allocator = gpa, .io = io };
+    // `free_size = 0` makes `release` destroy every connection and the pool
+    // hand none back (`Client.zig:133`). The fetcher's `.keep_alive = false` is
+    // what sends `connection: close`, but it cannot carry this alone: `request`
+    // takes a pooled connection before it stores the flag (`Client.zig:1725`,
+    // `:1742`), and a failed send leaves a `.ready` connection that
+    // `Request.deinit` would pool (`Client.zig:890`). A pass runs once a day,
+    // so any kept connection is closed at the peer by the next one, and the
+    // client never retries a stale one — it fails the source for the day.
+    var fetch_http: std.http.Client = .{
+        .allocator = gpa,
+        .io = io,
+        .connection_pool = .{ .free_size = 0 },
+    };
     defer fetch_http.deinit();
     var dns_http: std.http.Client = .{ .allocator = gpa, .io = io };
     defer dns_http.deinit();
@@ -545,6 +557,13 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
 
     var upstreams: upstream_owner.Owner = .init(upstream_generation);
     defer upstreams.deinit(io);
+    upstreams.diagnostics = event_store;
+    // The store outlived the process that wrote it, so every `upstream.exchange`
+    // episode in it describes a pool that no longer exists. This closes the ones
+    // no enabled upstream of this boot can justify, and projects the health of
+    // the ones that remain — a fresh pool is all clear, so a warning that
+    // survives this boot is one this process opened.
+    upstreams.reconcileDiagnostics(io, upstream_generation);
 
     // -----------------------------------------------------------------------
     // per-query state

src/cli.zig

@@ -978,14 +978,14 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize {
         // what keeps the close off `dot` while it is still undefined.
         var dot_wired = false;
         defer if (dot_wired) dot.close(r.io);
-        const client: transport.Client = switch (endpoint.scheme) {
+        const client: transport.Leaf = switch (endpoint.scheme) {
             .doh => doh: {
                 doh = doh_client.DohClient.init(&http, endpoint, &request_buf, &transfer_buf) catch {
                     try r.out.print("FAIL upstreams[{d}] {f}: not a usable DoH url\n", .{ i, safe_url.redactQuoted(server.url) });
                     failures += 1;
                     continue;
                 };
-                break :doh doh.client();
+                break :doh doh.leaf();
             },
             .dot => dot: {
                 dot = dot_client.DotClient.init(endpoint, server.tls_name, r.gpa, &bundle, &bundle_lock, null, .{
@@ -995,7 +995,7 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize {
                     .stream_write = tls_buffers[3 * chunk ..],
                 });
                 dot_wired = true;
-                break :dot dot.client();
+                break :dot dot.leaf();
             },
         };
 
@@ -1792,8 +1792,8 @@ test "a failed DoT probe reaches close through the per-iteration defer without a
 
     try testing.expectEqual(@as(usize, 2), try probeUpstreams(r, cfg));
     try testing.expectEqualStrings(
-        "FAIL upstreams[0] 'tls://dns.example:853': ConnectFailed\n" ++
+        "FAIL upstreams[0] 'tls://dns.example:853': ConnectFailed (cause ConnectFailed)\n" ++
-            "FAIL upstreams[1] 'tls://other.example:853': ConnectFailed\n",
+            "FAIL upstreams[1] 'tls://other.example:853': ConnectFailed (cause ConnectFailed)\n",
         captured.out.written(),
     );
 }

src/filter/fetcher.zig

@@ -1,9 +1,12 @@
 //! Blocklist download over HTTP/1.1.
 //!
-//! One `Fetcher` wraps a caller-owned `std.http.Client`, which owns the
+//! One `Fetcher` wraps a caller-owned `std.http.Client`, which owns the CA
-//! connection pool and the CA bundle, exactly as `upstream/doh_client.zig`
+//! bundle, as `upstream/doh_client.zig` does. Nothing here is pooled: every
-//! does. This file knows nothing about parsing, files or the database: it GETs
+//! request asks for `connection: close`, and the caller is expected to give
-//! a URL and streams the bytes into a writer the caller supplies.
+//! this file a client whose pool holds nothing (`free_size = 0`, set at the
+//! one construction site in `app.zig`). This file knows nothing about parsing,
+//! files or the database: it GETs a URL and streams the bytes into a writer
+//! the caller supplies.
 //!
 //! The body is never held whole. A blocklist can reach `max_body_bytes`, and
 //! the caller writes into a temporary file anyway, so nothing here allocates.
@@ -109,7 +112,16 @@ pub const Fetcher = struct {
         const uri = parseUrl(url) catch |err| return self.record(err, .connect, err, null, 0);
 
         var req = self.http.request(.GET, uri, .{
-            .keep_alive = true,
+            // Sends `connection: close`, so the peer ends the connection and
+            // `release` destroys it rather than pooling it (`Client.zig:133`,
+            // `:1167`). A pass runs once a day: a kept connection is one the
+            // peer has already closed by the next pass, and `std.http.Client`
+            // never retries a reused connection that reaches EOF before the
+            // first head byte — it returns `error.HttpConnectionClosing` and
+            // the source fails for the day. This flag cannot carry that alone,
+            // because `request` takes a pooled connection before it reads the
+            // flag; `app.zig` empties the pool itself.
+            .keep_alive = false,
             .headers = .{
                 // Identity only: a compressed transfer encoding would need
                 // `Response.readerDecompressing`, a decompression buffer and a

src/filter/filter_integration_test.zig

@@ -421,7 +421,7 @@ const Env = struct {
 // fixtures: the loopback http server
 // ---------------------------------------------------------------------------
 
-const Route = enum(u8) { body, redirect, not_found, oversize, chunked, stall, changed };
+const Route = enum(u8) { body, redirect, not_found, oversize, chunked, stall, changed, stale_keep_alive };
 
 /// How long the `stall` route holds a reply open when nothing releases it.
 ///
@@ -542,9 +542,10 @@ const HttpFixture = struct {
         }
     }
 
-    /// Every reply closes the connection. A keep-alive reply would leave the
+    /// Every reply but `stale_keep_alive` closes the connection. A keep-alive
-    /// fetcher holding the connection open while this server waits to accept a
+    /// reply would leave the fetcher holding the connection open while this
-    /// second one that never comes (milestone-5 spec, S9 note from S8).
+    /// server waits to accept a second one that never comes (milestone-5 spec,
+    /// S9 note from S8).
     fn respond(self: *HttpFixture, io: std.Io, request: *std.http.Server.Request) !void {
         switch (@as(Route, @enumFromInt(self.route.load(.acquire)))) {
             .body => try request.respond(self.body, .{ .keep_alive = false }),
@@ -566,6 +567,11 @@ const HttpFixture = struct {
                 .transfer_encoding = .none,
                 .extra_headers = &.{.{ .name = "content-length", .value = oversize_length }},
             }),
+            // Advertises keep-alive and then closes anyway, because `serve`
+            // takes one request per connection. That is the Pi's failure in
+            // miniature: a client that pools this connection reuses one the
+            // peer has already closed on its next request to the host.
+            .stale_keep_alive => try request.respond(self.body, .{ .keep_alive = true }),
             .chunked => try self.respondChunked(request),
             .stall => try self.respondStalled(io, request),
         }
@@ -1254,6 +1260,44 @@ fn publishFixtureFiles(env: *Env, id: i64, list_body: []const u8, wild_body: []c
     });
 }
 
+test "8b: a second download to the same host never reuses a dead pooled connection" {
+    if (!build_options.integration) return error.SkipZigTest;
+
+    const gpa = testing.allocator;
+    const env = try Env.create(gpa);
+    defer env.destroy();
+    const io = env.io();
+
+    var fixture = try HttpFixture.init(io, http_body);
+    defer fixture.deinit(io);
+    // The server says keep-alive and then drops the socket, which is what the
+    // Pi's upstreams do between one daily pass and the next.
+    fixture.setRoute(.stale_keep_alive);
+    var group: std.Io.Group = .init;
+    defer group.cancel(io);
+    try group.concurrent(io, HttpFixture.serve, .{ &fixture, io });
+
+    var url_buf: [64]u8 = undefined;
+    const url = try fixture.url(&url_buf);
+
+    var sink_buf: [0]u8 = .{};
+    var first: std.Io.Writer.Discarding = .init(&sink_buf);
+    const one = try env.f.fetch(io, url, &first.writer);
+    try testing.expectEqual(@as(u64, http_body.len), one.bytes_read);
+
+    // The whole test is this second call. With a pooling client the request
+    // goes out over a connection the peer has already closed, no response head
+    // byte arrives, and `error.HttpConnectionClosing` comes back with no retry
+    // — the exact record the Pi produced at 0 bytes and under a millisecond.
+    var second: std.Io.Writer.Discarding = .init(&sink_buf);
+    const two = try env.f.fetch(io, url, &second.writer);
+    try testing.expectEqual(@as(u64, http_body.len), two.bytes_read);
+    try testing.expectEqual(@as(?fetcher.Failure, null), env.f.last_failure);
+
+    // Two connections, not one reused: the proof the first was not pooled.
+    try testing.expectEqual(@as(u32, 2), fixture.accepted.load(.monotonic));
+}
+
 test "9: a reload swaps under a held handle and the new generation follows the release" {
     if (!build_options.integration) return error.SkipZigTest;
 

src/server/doh_server.zig

@@ -148,6 +148,10 @@ pub const DohServer = struct {
         return self.core.boundAddress();
     }
 
+    pub fn connectionsPeak(self: *const DohServer) u64 {
+        return self.core.connectionsPeak();
+    }
+
     pub fn snapshotStats(self: *const DohServer) Snapshot {
         const core = &self.core.stats;
         return .{

src/server/dot_server.zig

@@ -125,6 +125,10 @@ pub const DotServer = struct {
         self.* = undefined;
     }
 
+    pub fn connectionsPeak(self: *const DotServer) u64 {
+        return self.core.connectionsPeak();
+    }
+
     pub fn snapshotStats(self: *const DotServer) StatsSnapshot {
         const core = &self.core.stats;
         return .{

src/server/handler.zig

@@ -130,6 +130,11 @@ pub const Handler = struct {
     pause: ?*pause.Pause = null,
     tracker: ?*clients.Tracker = null,
     stats: Stats = .{},
+    /// Queries inside `handle` right now, across every listener, and the most
+    /// there have been at once (specs/memory-budget.md item 0). Gauges, so they
+    /// stay out of `Stats`, which `/metrics` exports field by field as counters.
+    in_flight: std.atomic.Value(u64) = .init(0),
+    in_flight_peak: std.atomic.Value(u64) = .init(0),
 
     pub const Stats = struct {
         /// Queries answered by the upstream pool.
@@ -229,6 +234,21 @@ pub const Handler = struct {
         query: []const u8,
         response_buf: []u8,
         scratch: *Scratch,
+    ) Outcome {
+        const now_in_flight = self.in_flight.fetchAdd(1, .monotonic) + 1;
+        _ = self.in_flight_peak.fetchMax(now_in_flight, .monotonic);
+        defer _ = self.in_flight.fetchSub(1, .monotonic);
+        return self.answer(io, which, from, query, response_buf, scratch);
+    }
+
+    fn answer(
+        self: *Handler,
+        io: std.Io,
+        which: Transport,
+        from: address.NetAddress,
+        query: []const u8,
+        response_buf: []u8,
+        scratch: *Scratch,
     ) Outcome {
         std.debug.assert(response_buf.len >= udp_limit_min);
         std.debug.assert(response_buf.len <= transport.max_message_len);
@@ -1459,6 +1479,24 @@ test "a response arriving on the listener port is dropped" {
     try testing.expectEqual(@as(usize, 0), fake.calls);
 }
 
+test "every query counts in flight for exactly its own handling" {
+    var t: TestIo = .init();
+    defer t.deinit();
+
+    var fake: FakeUpstream = .{ .reply = response_bytes };
+    var h_owner: upstream_owner.Borrowed = .{};
+    var h = bare(h_owner.client(fake.client()));
+    // Two queries other listeners are still handling.
+    h.in_flight.store(2, .monotonic);
+
+    var buf: [udp_limit_min]u8 = undefined;
+    try testing.expect(udp(&h, t.io(), query_bytes, &buf) == .reply);
+    try testing.expectEqual(Handler.Outcome.drop, udp(&h, t.io(), response_bytes, &buf));
+
+    try testing.expectEqual(@as(u64, 2), h.in_flight.load(.monotonic));
+    try testing.expectEqual(@as(u64, 3), h.in_flight_peak.load(.monotonic));
+}
+
 test "a query shorter than a header is dropped" {
     var t: TestIo = .init();
     defer t.deinit();

src/server/listener.zig

@@ -83,6 +83,10 @@ pub const CoreStats = struct {
     accept_errors: std.atomic.Value(u64) = .init(0),
     connection_errors: std.atomic.Value(u64) = .init(0),
     idle_timeouts: std.atomic.Value(u64) = .init(0),
+    /// The most slots that have been taken at once (specs/memory-budget.md
+    /// item 0). A gauge, so no listener's `Snapshot` carries it: those export
+    /// every field as a counter. Written only under `Core.mutex`.
+    connections_peak: std.atomic.Value(u64) = .init(0),
 };
 
 // ---------------------------------------------------------------------------
@@ -255,6 +259,9 @@ pub fn Core(comptime Cfg: type) type {
         /// that shuts the active connections down, so a stream that arrives
         /// after that scan can never claim a slot the scan will not visit again.
         shutdown_begun: bool,
+        /// Guarded by `mutex`. Slots that are not `.free`: a slot is taken from
+        /// `claim` until `finish` frees it, closing included.
+        taken: u64,
         stats: CoreStats,
         run_state: std.atomic.Value(State),
         stopped: std.Io.Event,
@@ -299,6 +306,7 @@ pub fn Core(comptime Cfg: type) type {
                 .conns = conns,
                 .mutex = .init,
                 .shutdown_begun = false,
+                .taken = 0,
                 .stats = .{},
                 .run_state = .init(.idle),
                 .stopped = .unset,
@@ -336,6 +344,10 @@ pub fn Core(comptime Cfg: type) type {
             return self.listener.socket.address;
         }
 
+        pub fn connectionsPeak(self: *const Self) u64 {
+            return self.stats.connections_peak.load(.monotonic);
+        }
+
         /// Accept loop. Returns when the task is canceled or `deinit` stops it.
         pub fn serve(self: *Self, io: std.Io) void {
             if (self.run_state.cmpxchgStrong(.idle, .serving, .acq_rel, .acquire) != null) return;
@@ -443,6 +455,10 @@ pub fn Core(comptime Cfg: type) type {
                     self.conns[index].stream = stream;
                     self.conns[index].peer = stream.socket.address;
                     self.conns[index].state = .active;
+                    self.taken += 1;
+                    if (self.taken > self.stats.connections_peak.load(.monotonic)) {
+                        self.stats.connections_peak.store(self.taken, .monotonic);
+                    }
                 },
                 .at_capacity, .shutting_down => {},
             }
@@ -464,6 +480,7 @@ pub fn Core(comptime Cfg: type) type {
 
             self.mutex.lockUncancelable(io);
             conn.state = .free;
+            self.taken -= 1;
             self.mutex.unlock(io);
         }
 

src/server/resolver_integration_test.zig

@@ -93,15 +93,21 @@ fn queryWithId(buf: *[query_bytes.len]u8, id: u16) []const u8 {
 /// Echoes the question and appends one A record. This is the smallest thing a
 /// real upstream could return that the handler forwards unchanged, so the
 /// assertions below check bytes that travelled the whole path.
-fn answerQuery(query: []const u8, response_buf: []u8) transport.ExchangeError![]u8 {
+fn answerQuery(query: []const u8, response_buf: []u8) transport.Outcome {
-    const request = packet.parse(query) catch return error.BadResponse;
+    const request = packet.parse(query) catch return peerFault(error.BadResponse);
-    const q = packet.firstQuestion(request) orelse return error.BadResponse;
+    const q = packet.firstQuestion(request) orelse return peerFault(error.BadResponse);
 
     var b = packet.ResponseBuilder.init(response_buf, request.header, q) catch
-        return error.ResponseTooLarge;
+        return peerFault(error.ResponseTooLarge);
     b.addAnswer(q.name, .a, .in, answer_ttl, &answer_rdata) catch
-        return error.ResponseTooLarge;
+        return peerFault(error.ResponseTooLarge);
-    return b.finish();
+    return .{ .reply = b.finish() };
+}
+
+/// A fault whose cause is its own classification: these fixtures fail on
+/// purpose and have no concrete cause behind the classification.
+fn peerFault(kind: transport.PeerFault) transport.Outcome {
+    return .{ .fault = .{ .kind = kind, .cause = kind } };
 }
 
 /// The healthy upstream. `calls` is atomic because the listener tasks run on
@@ -114,16 +120,14 @@ const GoodUpstream = struct {
         io: std.Io,
         query: []const u8,
         response_buf: []u8,
-        selected: *?[]const u8,
+    ) transport.LeafError!transport.Outcome {
-    ) transport.ExchangeError![]u8 {
         _ = io;
-        selected.* = "fake://good-upstream";
         const self: *GoodUpstream = @ptrCast(@alignCast(ptr));
         _ = self.calls.fetchAdd(1, .monotonic);
         return answerQuery(query, response_buf);
     }
 
-    fn client(self: *GoodUpstream) transport.Client {
+    fn leaf(self: *GoodUpstream) transport.Leaf {
         return .{ .ptr = self, .exchangeFn = exchangeFn };
     }
 };
@@ -140,17 +144,15 @@ const FaultyUpstream = struct {
         io: std.Io,
         query: []const u8,
         response_buf: []u8,
-        selected: *?[]const u8,
+    ) transport.LeafError!transport.Outcome {
-    ) transport.ExchangeError![]u8 {
         _ = io;
-        selected.* = "fake://faulty-upstream";
         const self: *FaultyUpstream = @ptrCast(@alignCast(ptr));
         const seen = self.calls.fetchAdd(1, .monotonic);
-        if (seen < self.fail_first) return self.fault;
+        if (seen < self.fail_first) return peerFault(self.fault);
         return answerQuery(query, response_buf);
     }
 
-    fn client(self: *FaultyUpstream) transport.Client {
+    fn leaf(self: *FaultyUpstream) transport.Leaf {
         return .{ .ptr = self, .exchangeFn = exchangeFn };
     }
 };
@@ -166,10 +168,10 @@ const EntryStorage = struct {
     fn entry(
         self: *EntryStorage,
         url: []const u8,
-        upstream_client: transport.Client,
+        upstream_leaf: transport.Leaf,
         priority: i32,
     ) pool.Entry {
-        self.slots[0] = .{ .client = upstream_client };
+        self.slots[0] = .{ .client = upstream_leaf };
         return .{
             .endpoint = transport.Endpoint.parse(url) catch unreachable,
             .slots = &self.slots,
@@ -270,8 +272,8 @@ test "the whole resolver answers over udp and tcp and fails over to a healthy up
     var bad_storage: EntryStorage = .{};
     var good_storage: EntryStorage = .{};
     var entries = [_]pool.Entry{
-        bad_storage.entry("https://bad.example/dns-query", bad.client(), 10),
+        bad_storage.entry("https://bad.example/dns-query", bad.leaf(), 10),
-        good_storage.entry("tls://good.example", good.client(), 20),
+        good_storage.entry("tls://good.example", good.leaf(), 20),
     };
     var upstreams: pool.Pool = .init(&entries, test_cfg, pool_timeouts, 1);
 

src/server/tcp_server.zig

@@ -99,6 +99,10 @@ pub const TcpServer = struct {
         return self.core.boundAddress();
     }
 
+    pub fn connectionsPeak(self: *const TcpServer) u64 {
+        return self.core.connectionsPeak();
+    }
+
     /// The counters, read one at a time. A scrape that lands mid-accept can see
     /// a connection counted before its outcome is; a lock would buy a
     /// consistency no consumer needs.

src/server/tcp_server_integration_test.zig

@@ -329,6 +329,52 @@ test "a canceled serve does not wait for a live connection" {
     try client_result;
 }
 
+test "the connections peak is the most slots held at once" {
+    if (!build_options.integration) return error.SkipZigTest;
+
+    const gpa = testing.allocator;
+    var threaded: std.Io.Threaded = .init(gpa, .{});
+    defer threaded.deinit();
+    const io = threaded.io();
+
+    var fake: FakeUpstream = .{ .reply = response_bytes };
+    var h_owner: upstream_owner.Borrowed = .{};
+    var h = bareHandler(h_owner.client(fake.client()));
+
+    const listen_address: net.IpAddress = try .parse("127.0.0.1", 0);
+    var server = try tcp_server.TcpServer.listen(gpa, io, listen_address, &h, .{
+        .max_connections = 3,
+        .idle_timeout = .{ .raw = .fromSeconds(600), .clock = .awake },
+    });
+    const server_address = server.boundAddress();
+
+    var serving: std.Io.Group = .init;
+    try serving.concurrent(io, tcp_server.TcpServer.serve, .{ &server, io });
+
+    var answered: [2]std.Io.Event = @splat(.unset);
+    var client_results: [2]anyerror!void = @splat({});
+    var clients_group: std.Io.Group = .init;
+    for (&answered, &client_results) |*event, *result| {
+        try clients_group.concurrent(io, holdConnection, .{ io, server_address, event, result });
+    }
+    for (&answered) |*event| event.waitUncancelable(io);
+
+    // Both replies prove both slots are claimed, and the listener has room for
+    // a third that never arrives.
+    const held_open = server.connectionsPeak();
+
+    server.deinit(io);
+    serving.await(io) catch |err| switch (err) {
+        error.Canceled => unreachable,
+    };
+    clients_group.await(io) catch |err| switch (err) {
+        error.Canceled => unreachable,
+    };
+
+    for (client_results) |result| try result;
+    try testing.expectEqual(@as(u64, 2), held_open);
+}
+
 /// The first slot the server would hand out, read after `serve` has returned so
 /// nothing can be writing it.
 fn firstFreeSlot(server: *const tcp_server.TcpServer) ?usize {

src/storage/events.zig

@@ -322,6 +322,74 @@ pub const Store = struct {
         if (!self.active.put(code, digest, id)) self.untracked_active_count += 1;
     }
 
+    /// Opens the episode of `subject_key` when none is open, and restates the
+    /// severity and the detail of the one that is.
+    ///
+    /// The projection counterpart of `report`. A reconciliation asserts the
+    /// state an endpoint is in now; it is not a new observation, so it must not
+    /// raise `occurrences` or move `last_seen`. It does own the text: a retired
+    /// generation's late report can leave a stale cause on a card the live
+    /// generation still holds open, and this is what restores the true one.
+    /// Only a recorded failure calls `report`.
+    ///
+    /// The open check is a statement rather than a mirror lookup: the mirror is
+    /// a hint that can point at a row that is gone or already closed, and
+    /// `report` recovers from that through the touch it was making anyway.
+    /// There is no write here to learn it from, so this asks. That costs one
+    /// SELECT per reconciled subject, on a path that runs at boot and at a
+    /// generation retirement.
+    pub fn ensureOpen(
+        self: *Store,
+        io: std.Io,
+        now_s: i64,
+        code: Code,
+        subject_key: []const u8,
+        subject_label: []const u8,
+        severity: Severity,
+        detail: []const u8,
+    ) void {
+        var key_buf: [max_subject_key_len]u8 = undefined;
+        const key = canonicalKey(subject_key, &key_buf);
+        const label = truncate(subject_label, max_subject_label_len);
+        const text = truncate(detail, max_detail_len);
+        const digest = digestOf(key);
+
+        self.mutex.lockUncancelable(io);
+        defer self.mutex.unlock(io);
+
+        self.count();
+        const existing = events_repo.selectActiveId(self.database, wire(code), key) catch |err|
+            return self.recordFailure(err);
+
+        // The read is not what clears the write latch: a `resolveExcept` that
+        // failed a moment ago is still the last word on whether this store can
+        // write, and only a write of our own can answer that.
+        if (existing) |id| {
+            self.count();
+            _ = events_repo.restateActive(self.database, id, severity.text(), text) catch |err|
+                return self.recordFailure(err);
+            return self.recordSuccess();
+        }
+
+        // Nothing is open, so a mirror entry claiming otherwise is stale and
+        // would make the insert below look like a collision.
+        if (self.active.find(code, digest)) |entry| self.active.remove(entry);
+
+        self.count();
+        const id = events_repo.insertActive(
+            self.database,
+            now_s,
+            wire(code),
+            key,
+            label,
+            severity.text(),
+            text,
+        ) catch |err| return self.recordFailure(err);
+        self.recordSuccess();
+
+        if (!self.active.put(code, digest, id)) self.untracked_active_count += 1;
+    }
+
     /// Records that `subject_key` is working again, closing its episode if one
     /// is open.
     ///
@@ -930,6 +998,38 @@ test "resolving a subject with nothing open executes no SQL at all" {
     try testing.expectEqual(after_resolve, store.statements);
 }
 
+test "ensureOpen opens a missing episode and restates an open one without counting it" {
+    var fx: Fixture = .{};
+    try fx.init(1000);
+    defer fx.deinit();
+    const store = &fx.store;
+
+    // The reconciliation path: it must be able to reassert a subject that is
+    // still failing without inventing an occurrence no exchange produced.
+    store.ensureOpen(fx.io, 1000, .upstream_exchange, "https://a.example", "a.example", .warning, "Timeout (cause Timeout)");
+    try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"));
+    try testing.expectEqual(@as(i64, 1), try fx.count("SELECT occurrences FROM operational_events"));
+    try testing.expectEqual(@as(i64, 1000), try fx.count("SELECT last_seen FROM operational_events"));
+
+    // The second call is a reconciliation of a card that is already open: the
+    // text and the severity are the reconciler's to state, and the counters
+    // belong to the exchanges that actually failed.
+    store.ensureOpen(fx.io, 1500, .upstream_exchange, "https://a.example", "a.example", .@"error", "SendFailed (cause BrokenPipe)");
+    try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
+    try testing.expectEqual(@as(i64, 1), try fx.count("SELECT occurrences FROM operational_events"));
+    try testing.expectEqual(@as(i64, 1000), try fx.count("SELECT first_seen FROM operational_events"));
+    try testing.expectEqual(@as(i64, 1000), try fx.count("SELECT last_seen FROM operational_events"));
+    try testing.expectEqualStrings("error", try fx.text("SELECT severity FROM operational_events"));
+    try testing.expectEqualStrings("SendFailed (cause BrokenPipe)", try fx.text("SELECT detail FROM operational_events"));
+
+    // And once the episode is resolved it opens a second one, like any other
+    // entry point.
+    store.resolve(fx.io, 1600, .upstream_exchange, "https://a.example");
+    store.ensureOpen(fx.io, 1700, .upstream_exchange, "https://a.example", "a.example", .warning, "Timeout (cause Timeout)");
+    try testing.expectEqual(@as(i64, 2), try fx.count("SELECT count(*) FROM operational_events"));
+    try testing.expectEqual(@as(i64, 1700), try fx.count("SELECT first_seen FROM operational_events WHERE id = 2"));
+}
+
 test "a one-shot event inserts already resolved and never enters the mirror" {
     var fx: Fixture = .{};
     try fx.init(1000);

src/storage/logger.zig

@@ -320,7 +320,68 @@ fn emptyAsNull(value: []const u8) ?[]const u8 {
     return if (value.len == 0) null else value;
 }
 
-const EntryQueue = std.Io.Queue(Entry);
+/// `std.Io.Queue` plus the occupancy high-water that sizes
+/// `logging.query_log_buffer_max` (specs/memory-budget.md item 0).
+///
+/// The queue keeps its length behind its own mutex with no accessor, so the
+/// count lives beside it. Raised before an insert and lowered after a removal,
+/// it never reads below the true occupancy; it overstates it by at most the
+/// producers inside `tryPut` plus the entries a getter has taken and not yet
+/// subtracted. An upper bound is the safe side for sizing a buffer.
+const EntryQueue = struct {
+    inner: std.Io.Queue(Entry),
+    held: std.atomic.Value(u64) = .init(0),
+    peak: std.atomic.Value(u64) = .init(0),
+
+    pub fn init(buffer: []Entry) EntryQueue {
+        return .{ .inner = .init(buffer) };
+    }
+
+    pub fn capacity(q: *const EntryQueue) usize {
+        return q.inner.capacity();
+    }
+
+    pub fn close(q: *EntryQueue, io: std.Io) void {
+        q.inner.close(io);
+    }
+
+    pub fn peakOccupancy(q: *const EntryQueue) u64 {
+        return q.peak.load(.monotonic);
+    }
+
+    /// Takes `entry` if there is room or a waiting getter, without blocking.
+    pub fn tryPut(q: *EntryQueue, io: std.Io, entry: Entry) (std.Io.QueueClosedError || std.Io.Cancelable)!bool {
+        const held = q.held.fetchAdd(1, .monotonic) + 1;
+        const put = q.inner.put(io, &.{entry}, 0) catch |err| {
+            _ = q.held.fetchSub(1, .monotonic);
+            return err;
+        };
+        if (put == 0) {
+            _ = q.held.fetchSub(1, .monotonic);
+            return false;
+        }
+        _ = q.peak.fetchMax(held, .monotonic);
+        return true;
+    }
+
+    pub fn get(q: *EntryQueue, io: std.Io, buffer: []Entry, min: usize) (std.Io.QueueClosedError || std.Io.Cancelable)!usize {
+        const n = try q.inner.get(io, buffer, min);
+        _ = q.held.fetchSub(n, .monotonic);
+        return n;
+    }
+
+    pub fn getUncancelable(q: *EntryQueue, io: std.Io, buffer: []Entry, min: usize) std.Io.QueueClosedError!usize {
+        const n = try q.inner.getUncancelable(io, buffer, min);
+        _ = q.held.fetchSub(n, .monotonic);
+        return n;
+    }
+
+    pub fn getOne(q: *EntryQueue, io: std.Io) (std.Io.QueueClosedError || std.Io.Cancelable)!Entry {
+        const entry = try q.inner.getOne(io);
+        _ = q.held.fetchSub(1, .monotonic);
+        return entry;
+    }
+};
 
 /// A database failure is not in here: a batch the database refuses is dropped,
 /// counted and reported where it happens, and the writer carries on.
@@ -525,8 +586,7 @@ pub const Logger = struct {
         while (true) {
             // A closed queue or a canceled task means shutdown is underway;
             // both leave this entry unwritten, which is what the counter says.
-            const put = self.queue.put(io, &.{entry}, 0) catch break;
+            if (self.queue.tryPut(io, entry) catch break) return;
-            if (put == 1) return;
 
             // A zero-capacity queue holds nothing to drop: the put above was
             // this entry's one chance at a waiting getter.
@@ -1442,6 +1502,34 @@ test "a zero-capacity queue drops every entry exactly once" {
     try testing.expectEqual(@as(u64, 5), logger.queries_dropped.load(.monotonic));
 }
 
+test "the queue peak is the most entries held at once, never the arrivals" {
+    var threaded: std.Io.Threaded = .init(testing.allocator, .{});
+    defer threaded.deinit();
+    const io = threaded.io();
+
+    var buf: [4]Entry = undefined;
+    var logger: Logger = .init(.{}, &buf);
+    try testing.expectEqual(@as(u64, 0), logger.queue.peakOccupancy());
+
+    logger.log(io, sampleEntry(1, "one.example"));
+    logger.log(io, sampleEntry(2, "two.example"));
+    logger.log(io, sampleEntry(3, "three.example"));
+    _ = try logger.queue.getOne(io);
+    _ = try logger.queue.getOne(io);
+    logger.log(io, sampleEntry(4, "four.example"));
+    try testing.expectEqual(@as(u64, 3), logger.queue.peakOccupancy());
+
+    // A full queue evicts to make room, so the peak stops at the capacity.
+    for (5..10) |i| logger.log(io, sampleEntry(@intCast(i), "more.example"));
+    try testing.expectEqual(@as(u64, 4), logger.queue.peakOccupancy());
+    try testing.expectEqual(@as(u64, 4), logger.queue.held.load(.monotonic));
+
+    var drained: [4]Entry = undefined;
+    try testing.expectEqual(@as(usize, 4), try logger.queue.get(io, &drained, 0));
+    try testing.expectEqual(@as(u64, 0), logger.queue.held.load(.monotonic));
+    try testing.expectEqual(@as(u64, 4), logger.queue.peakOccupancy());
+}
+
 test "log after shutdown drops instead of blocking" {
     var threaded: std.Io.Threaded = .init(testing.allocator, .{});
     defer threaded.deinit();

src/storage/logger_controller.zig

@@ -63,12 +63,14 @@ const Totals = struct {
     rows_written: u64 = 0,
     batches_gated: u64 = 0,
     last_drop_s: i64 = 0,
+    queue_peak: u64 = 0,
 
     fn fold(self: *Totals, lg: *const logger.Logger) void {
         self.queries_dropped += lg.queries_dropped.load(.monotonic);
         self.rows_written += lg.rows_written.load(.monotonic);
         self.batches_gated += lg.batches_gated.load(.monotonic);
         self.last_drop_s = @max(self.last_drop_s, lg.last_drop_s.load(.monotonic));
+        self.queue_peak = @max(self.queue_peak, lg.queue.peakOccupancy());
     }
 };
 
@@ -80,6 +82,8 @@ pub const Sample = struct {
     batches_gated: u64,
     /// Null while nothing has been dropped, exactly as `Logger.lastDropSeconds`.
     last_drop_s: ?i64,
+    /// The most entries any generation's queue has held at once.
+    queue_peak: u64,
     /// The LIVE writer's state. A retired writer's failures are in the counters.
     writer_failed: bool,
     /// The worst episode any outstanding generation is in. A retired writer
@@ -376,6 +380,7 @@ pub const Controller = struct {
             .rows_written = totals.rows_written,
             .batches_gated = totals.batches_gated,
             .last_drop_s = if (totals.last_drop_s == 0) null else totals.last_drop_s,
+            .queue_peak = totals.queue_peak,
             .writer_failed = self.live.logger.writer_failed.load(.acquire),
             .gate_episode = episode,
         };

src/storage/repositories/events_repo.zig

@@ -135,6 +135,35 @@ pub fn touchActive(
     return database.changes() != 0;
 }
 
+const restate_sql =
+    \\UPDATE operational_events
+    \\   SET detail = ?2,
+    \\       severity = ?3
+    \\ WHERE id = ?1 AND resolved_at IS NULL
+;
+
+/// Restates what an open episode says without claiming it happened again:
+/// `occurrences`, `first_seen` and `last_seen` keep the values the real
+/// failures wrote. The severity `CASE` of `touch_sql` is deliberately absent —
+/// a reconciliation asserts the current state of the subject, so it must be
+/// able to lower a severity a retired generation's late report raised.
+///
+/// False means the row was not there or was already resolved.
+pub fn restateActive(
+    database: *db.Db,
+    id: i64,
+    severity: []const u8,
+    detail: []const u8,
+) db.Error!bool {
+    var stmt = try database.prepare(restate_sql);
+    defer stmt.deinit();
+    try stmt.bindInt(1, id);
+    try stmt.bindText(2, detail);
+    try stmt.bindText(3, severity);
+    try stmt.exec();
+    return database.changes() != 0;
+}
+
 const resolve_by_id_sql =
     "UPDATE operational_events SET resolved_at = ?2 WHERE id = ?1 AND resolved_at IS NULL";
 

src/tests.zig

@@ -21,6 +21,7 @@ comptime {
     _ = @import("upstream/health.zig");
     _ = @import("upstream/doh_client.zig");
     _ = @import("upstream/doh_client_live_test.zig");
+    _ = @import("upstream/doh_client_integration_test.zig");
     _ = @import("upstream/pool.zig");
     _ = @import("upstream/owner.zig");
     _ = @import("upstream/dot_client.zig");

src/upstream/doh_client.zig

@@ -66,7 +66,7 @@ pub const DohClient = struct {
         };
     }
 
-    pub fn client(self: *DohClient) transport.Client {
+    pub fn leaf(self: *DohClient) transport.Leaf {
         return .{ .ptr = self, .exchangeFn = exchangeFn };
     }
 
@@ -75,12 +75,8 @@ pub const DohClient = struct {
         io: std.Io,
         query: []const u8,
         response_buf: []u8,
-        selected: *?[]const u8,
+    ) transport.LeafError!transport.Outcome {
-    ) transport.ExchangeError![]u8 {
         const self: *DohClient = @ptrCast(@alignCast(ptr));
-        // The endpoint outlives the client, so the borrow is safe for the whole
-        // query. Set before the attempt: a failure names this resolver too.
-        selected.* = self.endpoint.url;
         return self.exchange(io, query, response_buf);
     }
 
@@ -96,7 +92,7 @@ pub const DohClient = struct {
         io: std.Io,
         query: []const u8,
         response_buf: []u8,
-    ) transport.ExchangeError![]u8 {
+    ) transport.LeafError!transport.Outcome {
         // `std.http.Client` carries the `std.Io` it was constructed with and
         // takes none per request, so the interface's `io` is unused here. It
         // stays in the signature because DoT and the pool need it.
@@ -117,22 +113,26 @@ pub const DohClient = struct {
             // `Request.Headers` has no `accept` field, so this one goes in by
             // hand.
             .extra_headers = &.{.{ .name = "accept", .value = media_type }},
-        }) catch |err| return mapError(err, .connect);
+        }) catch |err| return fault(err, .connect);
         defer req.deinit();
 
         req.sendBodyComplete(self.request_buf[0..query.len]) catch |err|
-            return mapError(sendCause(&req, err), .send);
+            return fault(sendCause(&req, err), .send);
 
         // An empty redirect buffer is legal under `.not_allowed`: a redirect
         // is an error before the location is ever read.
-        var resp = req.receiveHead(&.{}) catch |err| return mapError(headCause(&req, err), .receive);
+        var resp = req.receiveHead(&.{}) catch |err| return fault(headCause(&req, err), .receive);
 
-        if (resp.head.status != .ok) return error.HttpStatus;
+        if (resp.head.status != .ok) return .{ .fault = .{
+            .kind = error.HttpStatus,
+            .cause = error.HttpStatus,
+            .http_status = resp.head.status,
+        } };
         // `head.content_type` points into memory that `resp.reader` invalidates,
         // so the check happens before the body stream starts.
-        if (!contentTypeOk(resp.head.content_type)) return error.HttpContentType;
+        if (!contentTypeOk(resp.head.content_type)) return peerFault(error.HttpContentType, error.HttpContentType);
         if (resp.head.content_length) |declared| {
-            if (declared > response_buf.len) return error.ResponseTooLarge;
+            if (declared > response_buf.len) return peerFault(error.ResponseTooLarge, error.ResponseTooLarge);
         }
 
         const body = resp.reader(self.transfer_buf);
@@ -140,7 +140,7 @@ pub const DohClient = struct {
         var ended = false;
         while (len < response_buf.len) {
             const n = body.readSliceShort(response_buf[len..]) catch |err|
-                return mapError(bodyCause(&resp, err), .receive);
+                return fault(bodyCause(&resp, err), .receive);
             len += n;
             if (n == 0) {
                 ended = true;
@@ -152,12 +152,13 @@ pub const DohClient = struct {
             // that fits from one that was cut off.
             var probe: [1]u8 = undefined;
             const n = body.readSliceShort(&probe) catch |err|
-                return mapError(bodyCause(&resp, err), .receive);
+                return fault(bodyCause(&resp, err), .receive);
-            if (n != 0) return error.ResponseTooLarge;
+            if (n != 0) return peerFault(error.ResponseTooLarge, error.ResponseTooLarge);
         }
 
-        try transport.validateResponse(query, response_buf[0..len]);
+        transport.validateResponse(query, response_buf[0..len]) catch |err|
-        return response_buf[0..len];
+            return peerFault(err, err);
+        return .{ .reply = response_buf[0..len] };
     }
 };
 
@@ -177,8 +178,30 @@ const Phase = enum { connect, send, receive };
 ///   `Connection.getReadError` (Client.zig:392), so its record-layer members
 ///   arrive here as themselves. Without them a decode error or a bad record MAC
 ///   would be reported as a plain receive failure.
-fn mapError(err: anyerror, phase: Phase) transport.ExchangeError {
+fn fault(err: anyerror, phase: Phase) transport.LeafError!transport.Outcome {
-    if (transport.mapLocal(err)) |local| return local;
+    // `.connect` is the whole of `http.request`: the name lookup, the TCP
+    // connect and the handshake. The handshake's only `error.Unexpected` is the
+    // errno the socket stream stored (std/http/Client.zig:1466; the TLS
+    // client's `InitError` has none), and a CA bundle scan fails as
+    // `CertificateBundleLoadFailure`. The lookup's resolver socket can fail
+    // locally, but that still leaves this upstream unreachable while a DoT
+    // upstream, always an IP literal, needs no lookup, so the fault belongs to
+    // this endpoint and the pool is right to move on.
+    const local = switch (phase) {
+        .connect => transport.mapLocalConnect(err),
+        .send, .receive => transport.mapLocal(err),
+    };
+    if (local) |claimed| return claimed;
+    return peerFault(kindOf(err, phase), err);
+}
+
+/// A peer fault as an `Outcome`. Written out rather than inlined at every call
+/// site so the classification and the cause cannot drift apart by a typo.
+fn peerFault(kind: transport.PeerFault, cause: anyerror) transport.Outcome {
+    return .{ .fault = .{ .kind = kind, .cause = cause } };
+}
+
+fn kindOf(err: anyerror, phase: Phase) transport.PeerFault {
     switch (err) {
         error.TlsInitializationFailed,
         error.CertificateBundleLoadFailure,
@@ -209,7 +232,7 @@ fn mapError(err: anyerror, phase: Phase) transport.ExchangeError {
 comptime {
     for (@typeInfo(std.crypto.tls.Client.ReadError).error_set.?) |member| {
         const value: anyerror = @field(std.crypto.tls.Client.ReadError, member.name);
-        if (mapError(value, .receive) != error.TlsFailed) {
+        if (kindOf(value, .receive) != error.TlsFailed) {
             @compileError("unclassified TLS read cause: " ++ member.name);
         }
     }
@@ -268,7 +291,7 @@ test "init builds a uri from the endpoint url" {
     try testing.expectEqualStrings("dns.example", doh.endpoint.host);
 }
 
-test "DohClient satisfies the transport.Client interface" {
+test "DohClient satisfies the transport.Leaf interface" {
     var http: std.http.Client = undefined;
     var request_buf: [min_request_buf]u8 = undefined;
     var transfer_buf: [min_transfer_buf]u8 = undefined;
@@ -278,7 +301,7 @@ test "DohClient satisfies the transport.Client interface" {
     // Instantiation is the check: the vtable is built from `exchangeFn`, so a
     // signature drift is a compile error here. The `std.http.Client` above is
     // never driven, and no exchange runs.
-    const c: transport.Client = doh.client();
+    const c: transport.Leaf = doh.leaf();
     try testing.expectEqual(@as(*anyopaque, @ptrCast(&doh)), c.ptr);
     try testing.expectEqual(
         @as(@TypeOf(c.exchangeFn), DohClient.exchangeFn),
@@ -318,34 +341,52 @@ test "contentTypeOk rejects anything else" {
     try testing.expect(!contentTypeOk("application/dns-message-extra"));
 }
 
-test "mapError maps local errors before phase errors" {
+test "a local cause stays an error instead of becoming a fault" {
-    try testing.expectEqual(error.OutOfMemory, mapError(error.OutOfMemory, .connect));
+    try testing.expectError(error.OutOfMemory, fault(error.OutOfMemory, .connect));
-    try testing.expectEqual(error.Canceled, mapError(error.Canceled, .receive));
+    try testing.expectError(error.Canceled, fault(error.Canceled, .receive));
-    try testing.expectEqual(error.Unexpected, mapError(error.Unexpected, .send));
+    try testing.expectError(error.Unexpected, fault(error.Unexpected, .send));
 }
 
-test "mapError maps the collapsed tls errors regardless of phase" {
+test "an unnamed connect errno is a connect fault, so the pool fails over" {
-    try testing.expectEqual(error.TlsFailed, mapError(error.TlsInitializationFailed, .connect));
+    const outcome = try fault(error.Unexpected, .connect);
-    try testing.expectEqual(error.TlsFailed, mapError(error.TlsInitializationFailed, .receive));
+    try testing.expectEqual(transport.PeerFault.ConnectFailed, outcome.fault.kind);
-    try testing.expectEqual(error.TlsFailed, mapError(error.CertificateBundleLoadFailure, .connect));
+    try testing.expectEqual(error.Unexpected, outcome.fault.cause);
 }
 
-test "mapError maps every unwrapped record-layer cause to TlsFailed" {
+test "kindOf maps the collapsed tls errors regardless of phase" {
+    try testing.expectEqual(error.TlsFailed, kindOf(error.TlsInitializationFailed, .connect));
+    try testing.expectEqual(error.TlsFailed, kindOf(error.TlsInitializationFailed, .receive));
+    try testing.expectEqual(error.TlsFailed, kindOf(error.CertificateBundleLoadFailure, .connect));
+}
+
+test "kindOf maps every unwrapped record-layer cause to TlsFailed" {
     // The set is the one `Connection.getReadError` can hand back, so the loop
     // fails the day std adds a member the switch does not name.
     inline for (@typeInfo(std.crypto.tls.Client.ReadError).error_set.?) |member| {
         try testing.expectEqual(
-            transport.ExchangeError.TlsFailed,
+            transport.PeerFault.TlsFailed,
-            mapError(@field(std.crypto.tls.Client.ReadError, member.name), .receive),
+            kindOf(@field(std.crypto.tls.Client.ReadError, member.name), .receive),
         );
     }
 }
 
-test "mapError maps remaining errors by phase" {
+test "kindOf maps remaining errors by phase" {
-    try testing.expectEqual(error.ConnectFailed, mapError(error.ConnectionRefused, .connect));
+    try testing.expectEqual(error.ConnectFailed, kindOf(error.ConnectionRefused, .connect));
-    try testing.expectEqual(error.SendFailed, mapError(error.WriteFailed, .send));
+    try testing.expectEqual(error.SendFailed, kindOf(error.WriteFailed, .send));
-    try testing.expectEqual(error.ReceiveFailed, mapError(error.ReadFailed, .receive));
+    try testing.expectEqual(error.ReceiveFailed, kindOf(error.ReadFailed, .receive));
-    try testing.expectEqual(error.ReceiveFailed, mapError(error.HttpHeadersInvalid, .receive));
+    try testing.expectEqual(error.ReceiveFailed, kindOf(error.HttpHeadersInvalid, .receive));
+}
+
+test "a fault carries the classification and the concrete cause" {
+    const failed = try faultOf(error.ConnectionRefused, .connect);
+    try testing.expectEqual(transport.PeerFault.ConnectFailed, failed.kind);
+    try testing.expectEqual(@as(anyerror, error.ConnectionRefused), failed.cause);
+
+    var buf: [64]u8 = undefined;
+    try testing.expectEqualStrings(
+        "ConnectFailed (cause ConnectionRefused)",
+        try std.fmt.bufPrint(&buf, "{f}", .{failed}),
+    );
 }
 
 /// Only the fields the unwrap helpers read are set. The rest of a `Connection`
@@ -367,6 +408,15 @@ fn stubConnection(
     return connection;
 }
 
+/// The fault half of `fault`, for the unwrap tests. A local cause leaves this
+/// as an error, which is what those tests assert instead.
+fn faultOf(err: anyerror, phase: Phase) !transport.Fault {
+    return switch (try fault(err, phase)) {
+        .reply => error.TestExpectedFault,
+        .fault => |f| f,
+    };
+}
+
 fn stubRequest(connection: *Connection, body_err: ?std.http.Reader.BodyError) Request {
     var req: Request = undefined;
     req.connection = connection;
@@ -377,59 +427,46 @@ fn stubRequest(connection: *Connection, body_err: ?std.http.Reader.BodyError) Re
 test "the send unwrap keeps a cancelled write out of the peer fault group" {
     var connection = stubConnection(null, error.Canceled);
     var req = stubRequest(&connection, null);
-    const mapped = mapError(sendCause(&req, error.WriteFailed), .send);
+    try testing.expectError(error.Canceled, fault(sendCause(&req, error.WriteFailed), .send));
-    try testing.expectEqual(transport.ExchangeError.Canceled, mapped);
-    try testing.expectEqual(transport.Group.cancellation, transport.group(mapped));
 }
 
 test "the send unwrap keeps a local resource write failure out of the peer fault group" {
     var connection = stubConnection(null, error.SystemResources);
     var req = stubRequest(&connection, null);
-    const mapped = mapError(sendCause(&req, error.WriteFailed), .send);
+    try testing.expectError(error.SystemResources, fault(sendCause(&req, error.WriteFailed), .send));
-    try testing.expectEqual(transport.ExchangeError.SystemResources, mapped);
-    try testing.expectEqual(transport.Group.local_resource, transport.group(mapped));
 }
 
 test "the send unwrap reports a peer side cause as a send fault" {
     var connection = stubConnection(null, error.ConnectionResetByPeer);
     var req = stubRequest(&connection, null);
-    try testing.expectEqual(
+    const failed = try faultOf(sendCause(&req, error.WriteFailed), .send);
-        transport.ExchangeError.SendFailed,
+    try testing.expectEqual(transport.PeerFault.SendFailed, failed.kind);
-        mapError(sendCause(&req, error.WriteFailed), .send),
+    try testing.expectEqual(@as(anyerror, error.ConnectionResetByPeer), failed.cause);
-    );
 }
 
 test "the head unwrap keeps a local resource read failure out of the peer fault group" {
     var connection = stubConnection(error.SystemResources, null);
     var req = stubRequest(&connection, null);
-    const mapped = mapError(headCause(&req, error.ReadFailed), .receive);
+    try testing.expectError(error.SystemResources, fault(headCause(&req, error.ReadFailed), .receive));
-    try testing.expectEqual(transport.ExchangeError.SystemResources, mapped);
-    try testing.expectEqual(transport.Group.local_resource, transport.group(mapped));
 
     var canceled = stubConnection(error.Canceled, null);
     var canceled_req = stubRequest(&canceled, null);
-    try testing.expectEqual(
+    try testing.expectError(error.Canceled, fault(headCause(&canceled_req, error.ReadFailed), .receive));
-        transport.ExchangeError.Canceled,
-        mapError(headCause(&canceled_req, error.ReadFailed), .receive),
-    );
 }
 
 test "the head unwrap reports a peer side cause as a receive fault" {
     var connection = stubConnection(error.ConnectionResetByPeer, null);
     var req = stubRequest(&connection, null);
-    try testing.expectEqual(
+    const failed = try faultOf(headCause(&req, error.ReadFailed), .receive);
-        transport.ExchangeError.ReceiveFailed,
+    try testing.expectEqual(transport.PeerFault.ReceiveFailed, failed.kind);
-        mapError(headCause(&req, error.ReadFailed), .receive),
+    try testing.expectEqual(@as(anyerror, error.ConnectionResetByPeer), failed.cause);
-    );
 }
 
 test "the body unwrap keeps a cancelled read out of the peer fault group" {
     var connection = stubConnection(error.Canceled, null);
     var req = stubRequest(&connection, null);
     const resp: Response = .{ .request = &req, .head = undefined };
-    const mapped = mapError(bodyCause(&resp, error.ReadFailed), .receive);
+    try testing.expectError(error.Canceled, fault(bodyCause(&resp, error.ReadFailed), .receive));
-    try testing.expectEqual(transport.ExchangeError.Canceled, mapped);
-    try testing.expectEqual(transport.Group.cancellation, transport.group(mapped));
 }
 
 test "the body unwrap prefers an http framing fault over the connection" {
@@ -440,10 +477,9 @@ test "the body unwrap prefers an http framing fault over the connection" {
     var req = stubRequest(&connection, error.HttpChunkTruncated);
     const resp: Response = .{ .request = &req, .head = undefined };
     try testing.expectEqual(error.HttpChunkTruncated, bodyCause(&resp, error.ReadFailed));
-    try testing.expectEqual(
+    const failed = try faultOf(bodyCause(&resp, error.ReadFailed), .receive);
-        transport.ExchangeError.ReceiveFailed,
+    try testing.expectEqual(transport.PeerFault.ReceiveFailed, failed.kind);
-        mapError(bodyCause(&resp, error.ReadFailed), .receive),
+    try testing.expectEqual(@as(anyerror, error.HttpChunkTruncated), failed.cause);
-    );
 }
 
 test "the unwraps report the collapsed error when no cause was stored" {
@@ -455,14 +491,13 @@ test "the unwraps report the collapsed error when no cause was stored" {
     try testing.expectEqual(error.ReadFailed, headCause(&req, error.ReadFailed));
     try testing.expectEqual(error.ReadFailed, bodyCause(&resp, error.ReadFailed));
 
-    try testing.expectEqual(
+    const sent = try faultOf(sendCause(&req, error.WriteFailed), .send);
-        transport.ExchangeError.SendFailed,
+    try testing.expectEqual(transport.PeerFault.SendFailed, sent.kind);
-        mapError(sendCause(&req, error.WriteFailed), .send),
+    try testing.expectEqual(@as(anyerror, error.WriteFailed), sent.cause);
-    );
+
-    try testing.expectEqual(
+    const received = try faultOf(bodyCause(&resp, error.ReadFailed), .receive);
-        transport.ExchangeError.ReceiveFailed,
+    try testing.expectEqual(transport.PeerFault.ReceiveFailed, received.kind);
-        mapError(bodyCause(&resp, error.ReadFailed), .receive),
+    try testing.expectEqual(@as(anyerror, error.ReadFailed), received.cause);
-    );
 }
 
 test "the unwraps pass a non-collapsed error through untouched" {
@@ -475,8 +510,7 @@ test "the unwraps pass a non-collapsed error through untouched" {
     try testing.expectEqual(error.EndOfStream, sendCause(&req, error.EndOfStream));
     try testing.expectEqual(error.HttpHeadersInvalid, headCause(&req, error.HttpHeadersInvalid));
     try testing.expectEqual(error.EndOfStream, bodyCause(&resp, error.EndOfStream));
-    try testing.expectEqual(
+    const failed = try faultOf(headCause(&req, error.HttpHeadersInvalid), .receive);
-        transport.ExchangeError.ReceiveFailed,
+    try testing.expectEqual(transport.PeerFault.ReceiveFailed, failed.kind);
-        mapError(headCause(&req, error.HttpHeadersInvalid), .receive),
+    try testing.expectEqual(@as(anyerror, error.HttpHeadersInvalid), failed.cause);
-    );
 }

src/upstream/doh_client_integration_test.zig

@@ -0,0 +1,78 @@
+//! Hermetic loopback test for `doh_client.zig`.
+//!
+//! It lives in its own file because it needs `@import("build_options")`, which
+//! only exists when the compilation is driven by build.zig. `-Dintegration`
+//! gates it; nothing here leaves the machine and nothing here resolves a name.
+//!
+//! What it covers that the stub-connection tests in `doh_client.zig` cannot:
+//! those call the unwrap and classification helpers directly, so they prove the
+//! helpers and not the path. This drives the whole of `DohClient.exchange`
+//! against a peer that is provably not listening, and asserts that the concrete
+//! cause survives the classification and reaches the returned `Fault`. Without
+//! it, a future `exchange` that dropped the cause on the floor would still pass
+//! every other test in this tree.
+
+const std = @import("std");
+const build_options = @import("build_options");
+const net = std.Io.net;
+
+const doh_client = @import("doh_client.zig");
+const transport = @import("transport.zig");
+
+const testing = std.testing;
+
+/// A query for example.com A: id 0x1234, RD set, one question.
+const query_bytes =
+    "\x12\x34\x01\x00\x00\x01\x00\x00\x00\x00\x00\x00" ++
+    "\x07example\x03com\x00\x00\x01\x00\x01";
+
+/// A loopback TCP port that is bound and never listening, held open for the
+/// whole test.
+///
+/// Bound, because an ephemeral port the kernel hands out is the only port a
+/// test can be sure of; a hardcoded one could belong to something. Held rather
+/// than closed, because a closed port is free for another process on this
+/// machine to take between the close and the connect, and then the refusal this
+/// test asserts would be a connection instead. Never listening, because a
+/// connect to a bound TCP port with no accept queue is refused, which is the
+/// deterministic peer failure the test needs.
+fn bindDeadPort(io: std.Io) !net.Socket {
+    return (net.IpAddress{ .ip4 = .loopback(0) }).bind(io, .{ .mode = .stream });
+}
+
+test "a refused connection reaches the caller as a ConnectFailed carrying its cause" {
+    if (!build_options.integration) return error.SkipZigTest;
+
+    const gpa = testing.allocator;
+    var threaded: std.Io.Threaded = .init(gpa, .{});
+    defer threaded.deinit();
+    const io = threaded.io();
+
+    var dead = try bindDeadPort(io);
+    defer dead.close(io);
+
+    var url_buf: [64]u8 = undefined;
+    const url = try std.fmt.bufPrint(&url_buf, "https://127.0.0.1:{d}/dns-query", .{dead.address.ip4.port});
+
+    var http: std.http.Client = .{ .allocator = gpa, .io = io };
+    defer http.deinit();
+
+    var request_buf: [doh_client.min_request_buf]u8 = undefined;
+    var transfer_buf: [doh_client.min_transfer_buf]u8 = undefined;
+    var doh = try doh_client.DohClient.init(&http, try .parse(url), &request_buf, &transfer_buf);
+
+    var response_buf: [512]u8 = undefined;
+    switch (try doh.exchange(io, query_bytes, &response_buf)) {
+        .reply => return error.TestExpectedFault,
+        .fault => |fault| {
+            try testing.expectEqual(transport.PeerFault.ConnectFailed, fault.kind);
+            try testing.expectEqual(@as(anyerror, error.ConnectionRefused), fault.cause);
+
+            var text: [64]u8 = undefined;
+            try testing.expectEqualStrings(
+                "ConnectFailed (cause ConnectionRefused)",
+                try std.fmt.bufPrint(&text, "{f}", .{fault}),
+            );
+        },
+    }
+}

src/upstream/doh_client_live_test.zig

@@ -44,10 +44,10 @@ fn runExchange(io: std.Io, params: Params) anyerror!usize {
     const endpoint = try transport.Endpoint.parse("https://cloudflare-dns.com/dns-query");
     var doh = try doh_client.DohClient.init(&http, endpoint, &request_buf, &transfer_buf);
 
-    var selected: ?[]const u8 = null;
+    return switch (try doh.leaf().exchange(io, query_bytes, params.response_buf)) {
-    const reply = try doh.client().exchange(io, query_bytes, params.response_buf, &selected);
+        .reply => |reply| reply.len,
-    std.debug.assert(std.mem.eql(u8, selected.?, endpoint.url));
+        .fault => |failed| failed.kind,
-    return reply.len;
+    };
 }
 
 fn expire(io: std.Io, duration: std.Io.Clock.Duration) std.Io.Cancelable!void {

src/upstream/dot_client.zig

@@ -186,7 +186,7 @@ pub const DotClient = struct {
         }
     }
 
-    pub fn client(self: *DotClient) transport.Client {
+    pub fn leaf(self: *DotClient) transport.Leaf {
         return .{ .ptr = self, .exchangeFn = exchangeFn };
     }
 
@@ -199,12 +199,8 @@ pub const DotClient = struct {
         io: std.Io,
         query: []const u8,
         response_buf: []u8,
-        selected: *?[]const u8,
+    ) transport.LeafError!transport.Outcome {
-    ) transport.ExchangeError![]u8 {
         const self: *DotClient = @ptrCast(@alignCast(ptr));
-        // The endpoint outlives the client, so the borrow is safe for the whole
-        // query. Set before the attempt: a failure names this resolver too.
-        selected.* = self.endpoint.url;
         return self.exchange(io, query, response_buf);
     }
 
@@ -222,24 +218,24 @@ pub const DotClient = struct {
         io: std.Io,
         query: []const u8,
         response_buf: []u8,
-    ) transport.ExchangeError![]u8 {
+    ) transport.LeafError!transport.Outcome {
         // The length prefix is 16-bit, so a longer query cannot be framed. No
         // listener in this process can produce one; a caller that does gets a
         // local error rather than a silently truncated frame.
         if (query.len > transport.max_message_len) return error.BufferTooSmall;
 
         const reused = self.session != null;
-        if (!reused) try self.dial(io);
+        if (!reused) if (try self.dial(io)) |dial_fault| return .{ .fault = dial_fault };
 
         const failure = switch (self.transact(query, response_buf)) {
-            .ok => |reply| return reply,
+            .ok => |reply| return .{ .reply = reply },
             .failed => |failure| failure,
         };
 
         switch (retryDecision(reused, failure.received_any, failure.cause)) {
             .final => {
                 self.close(io);
-                return transport.mapPhase(failure.cause, failure.phase);
+                return .{ .fault = try transport.faultOrLocal(failure.cause, failure.phase) };
             },
             .retry => {},
         }
@@ -249,38 +245,42 @@ pub const DotClient = struct {
         // `reuse_recoveries` is what makes the churn visible.
         log.debug("{f}", .{self.diagnose(.{ .stale_session = failure.cause })});
         self.close(io);
-        try self.dial(io);
+        if (try self.dial(io)) |dial_fault| return .{ .fault = dial_fault };
 
         switch (self.transact(query, response_buf)) {
             .ok => |reply| {
                 if (self.reuse_recoveries) |counter| _ = counter.fetchAdd(1, .monotonic);
-                return reply;
+                return .{ .reply = reply };
             },
             // The retry's outcome is the exchange's outcome: one redial, never
             // two.
             .failed => |retried| {
                 self.close(io);
-                return transport.mapPhase(retried.cause, retried.phase);
+                return .{ .fault = try transport.faultOrLocal(retried.cause, retried.phase) };
             },
         }
     }
 
     /// Opens a session and leaves it in `self.session`, or leaves `self.session`
-    /// null and returns the classified failure. No partially initialized
+    /// null and returns the classified fault. `null` means a session is open. No
-    /// session ever survives this call.
+    /// partially initialized session ever survives this call.
-    fn dial(self: *DotClient, io: std.Io) transport.ExchangeError!void {
+    fn dial(self: *DotClient, io: std.Io) transport.LeafError!?transport.Fault {
         std.debug.assert(self.session == null);
 
         const address = resolveAddress(self.endpoint) catch |err| {
+            // A host that is not an IP literal is a config error, and
+            // `resolveAddress` has already folded the parse failure into the
+            // classification, so the cause it carries is the classification.
             log.warn("{f}", .{self.diagnose(.not_an_ip_literal)});
-            return err;
+            return .{ .kind = err, .cause = err };
         };
 
-        try self.ensureBundle(io);
+        if (try self.ensureBundle(io)) |bundle_fault| return bundle_fault;
 
         const stream = address.connect(io, .{ .mode = .stream }) catch |err| {
             log.debug("{f}", .{self.diagnose(.{ .connect_failed = err })});
-            return transport.mapPhase(err, error.ConnectFailed);
+            if (transport.mapLocalConnect(err)) |local| return local;
+            return .{ .kind = error.ConnectFailed, .cause = err };
         };
 
         // Emplaced before the handshake, never built beside it and copied in:
@@ -314,8 +314,9 @@ pub const DotClient = struct {
                 .verify_name = self.verify_name,
                 .cause = cause,
             } })});
-            return transport.mapPhase(cause, error.TlsFailed);
+            return try transport.faultOrLocal(cause, error.TlsFailed);
         };
+        return null;
     }
 
     /// One query and one reply on the open session, with enough detail on
@@ -380,17 +381,17 @@ pub const DotClient = struct {
     /// cancellation into `error.CertificateBundleLoadFailure`. That name cannot
     /// tell an `error.OutOfMemory` from a corrupt PEM file, and the first is a
     /// local resource failure that must not count against the upstream's
-    /// health. Scanning here keeps the concrete error for `transport.mapPhase`.
+    /// health. Scanning here keeps the concrete error for the fault.
-    fn ensureBundle(self: *DotClient, io: std.Io) transport.ExchangeError!void {
+    fn ensureBundle(self: *DotClient, io: std.Io) transport.LeafError!?transport.Fault {
         {
             try self.bundle_lock.lockShared(io);
             defer self.bundle_lock.unlockShared(io);
-            if (self.bundle.map.count() != 0) return;
+            if (self.bundle.map.count() != 0) return null;
         }
 
         try self.bundle_lock.lock(io);
         defer self.bundle_lock.unlock(io);
-        if (self.bundle.map.count() != 0) return;
+        if (self.bundle.map.count() != 0) return null;
 
         // A partial scan leaves entries in `map`, which the check above would
         // read as "already loaded". Reset so the next exchange scans again.
@@ -398,8 +399,9 @@ pub const DotClient = struct {
             self.bundle.deinit(self.gpa);
             self.bundle.* = .empty;
             log.warn("{f}", .{self.diagnose(.{ .bundle_load_failed = err })});
-            return transport.mapPhase(err, error.TlsFailed);
+            return try transport.faultOrLocal(err, error.TlsFailed);
         };
+        return null;
     }
 };
 
@@ -642,40 +644,39 @@ fn stubStream(
 
 test "the handshake unwrap keeps a cancelled read out of the peer fault group" {
     var stream = stubStream(error.Canceled, null, null);
-    const mapped = transport.mapPhase(concreteHandshake(&stream, error.ReadFailed), error.TlsFailed);
+    try testing.expectError(
-    try testing.expectEqual(transport.ExchangeError.Canceled, mapped);
+        error.Canceled,
-    try testing.expectEqual(transport.Group.cancellation, transport.group(mapped));
+        transport.faultOrLocal(concreteHandshake(&stream, error.ReadFailed), error.TlsFailed),
+    );
 }
 
 test "the handshake unwrap keeps a local resource write failure out of the peer fault group" {
     var stream = stubStream(null, error.SystemResources, null);
-    const mapped = transport.mapPhase(concreteHandshake(&stream, error.WriteFailed), error.TlsFailed);
+    try testing.expectError(
-    try testing.expectEqual(transport.ExchangeError.SystemResources, mapped);
+        error.SystemResources,
-    try testing.expectEqual(transport.Group.local_resource, transport.group(mapped));
+        transport.faultOrLocal(concreteHandshake(&stream, error.WriteFailed), error.TlsFailed),
+    );
 }
 
 test "the handshake unwrap reports a peer side cause as a TLS fault" {
     var reset = stubStream(error.ConnectionResetByPeer, null, null);
-    try testing.expectEqual(
+    const read_failed = try transport.faultOrLocal(concreteHandshake(&reset, error.ReadFailed), error.TlsFailed);
-        transport.ExchangeError.TlsFailed,
+    try testing.expectEqual(transport.PeerFault.TlsFailed, read_failed.kind);
-        transport.mapPhase(concreteHandshake(&reset, error.ReadFailed), error.TlsFailed),
+    try testing.expectEqual(@as(anyerror, error.ConnectionResetByPeer), read_failed.cause);
-    );
 
     var refused = stubStream(null, error.ConnectionRefused, null);
-    try testing.expectEqual(
+    const write_failed = try transport.faultOrLocal(concreteHandshake(&refused, error.WriteFailed), error.TlsFailed);
-        transport.ExchangeError.TlsFailed,
+    try testing.expectEqual(transport.PeerFault.TlsFailed, write_failed.kind);
-        transport.mapPhase(concreteHandshake(&refused, error.WriteFailed), error.TlsFailed),
+    try testing.expectEqual(@as(anyerror, error.ConnectionRefused), write_failed.cause);
-    );
 }
 
 test "the handshake unwrap reports a TLS fault when no cause was stored" {
     var stream = stubStream(null, null, null);
     try testing.expectEqual(error.ReadFailed, concreteHandshake(&stream, error.ReadFailed));
     try testing.expectEqual(error.WriteFailed, concreteHandshake(&stream, error.WriteFailed));
-    try testing.expectEqual(
+    const failed = try transport.faultOrLocal(concreteHandshake(&stream, error.ReadFailed), error.TlsFailed);
-        transport.ExchangeError.TlsFailed,
+    try testing.expectEqual(transport.PeerFault.TlsFailed, failed.kind);
-        transport.mapPhase(concreteHandshake(&stream, error.ReadFailed), error.TlsFailed),
+    try testing.expectEqual(@as(anyerror, error.ReadFailed), failed.cause);
-    );
 }
 
 test "the handshake unwrap passes other errors through untouched" {
@@ -685,42 +686,52 @@ test "the handshake unwrap passes other errors through untouched" {
         concreteHandshake(&stream, error.CertificateExpired),
     );
     try testing.expectEqual(error.Canceled, concreteHandshake(&stream, error.Canceled));
-    try testing.expectEqual(
+    const expired = try transport.faultOrLocal(concreteHandshake(&stream, error.CertificateExpired), error.TlsFailed);
-        transport.ExchangeError.TlsFailed,
+    try testing.expectEqual(transport.PeerFault.TlsFailed, expired.kind);
-        transport.mapPhase(concreteHandshake(&stream, error.CertificateExpired), error.TlsFailed),
+    try testing.expectEqual(@as(anyerror, error.CertificateExpired), expired.cause);
-    );
+    try testing.expectError(
-    try testing.expectEqual(
+        error.Canceled,
-        transport.ExchangeError.Canceled,
+        transport.faultOrLocal(concreteHandshake(&stream, error.Canceled), error.TlsFailed),
-        transport.mapPhase(concreteHandshake(&stream, error.Canceled), error.TlsFailed),
     );
 }
 
-/// What `exchange` would return for a failure `transact` reported.
+/// The fault `exchange` returns for a failure `transact` reported. A local cause leaves this as an
-fn mappedFailure(outcome: Transact) transport.ExchangeError {
+/// error, which is what the tests of those causes assert instead.
-    return transport.mapPhase(outcome.failed.cause, outcome.failed.phase);
+fn mappedFailure(outcome: Transact) transport.LeafError!transport.Fault {
+    return transport.faultOrLocal(outcome.failed.cause, outcome.failed.phase);
 }
 
 test "the send and receive unwraps prefer the stored cause" {
     var send = stubStream(null, error.Canceled, null);
-    try testing.expectEqual(
+    try testing.expectError(error.Canceled, mappedFailure(sendFailed(&send, error.WriteFailed)));
-        transport.ExchangeError.Canceled,
-        mappedFailure(sendFailed(&send, error.WriteFailed)),
-    );
 
     // The TLS client's own error wins over the socket reader's.
     var receive = stubStream(error.ConnectionResetByPeer, null, error.TlsAlert);
-    try testing.expectEqual(
+    const received = try mappedFailure(receiveFailed(&receive, error.ReadFailed, false));
-        transport.ExchangeError.ReceiveFailed,
+    try testing.expectEqual(transport.PeerFault.ReceiveFailed, received.kind);
-        mappedFailure(receiveFailed(&receive, error.ReadFailed, false)),
+    try testing.expectEqual(@as(anyerror, error.TlsAlert), received.cause);
-    );
 
     var socket = stubStream(error.SystemResources, null, null);
-    try testing.expectEqual(
+    try testing.expectError(
-        transport.ExchangeError.SystemResources,
+        error.SystemResources,
         mappedFailure(receiveFailed(&socket, error.ReadFailed, true)),
     );
 }
 
+test "a transact failure becomes a fault carrying its concrete cause" {
+    var stream = stubStream(null, error.ConnectionResetByPeer, null);
+
+    const sent = try mappedFailure(sendFailed(&stream, error.WriteFailed));
+    try testing.expectEqual(transport.PeerFault.SendFailed, sent.kind);
+    try testing.expectEqual(@as(anyerror, error.ConnectionResetByPeer), sent.cause);
+
+    var text: [64]u8 = undefined;
+    try testing.expectEqualStrings(
+        "SendFailed (cause ConnectionResetByPeer)",
+        try std.fmt.bufPrint(&text, "{f}", .{sent}),
+    );
+}
+
 test "a send failure is always pre-first-byte, and a receive failure reports what it read" {
     // The retry rule reads `received_any`, so where it comes from is part of the
     // contract rather than an incidental field: nothing is read before the query
@@ -804,14 +815,13 @@ test "a validation failure is final and its phase survives the mapping" {
     };
     for (outcomes) |outcome| {
         try testing.expectEqual(RetryDecision.final, retryDecision(true, true, outcome.cause));
-        try testing.expectEqual(
+        const failed = try mappedFailure(.{ .failed = .{
-            @as(transport.ExchangeError, outcome.phase),
+            .cause = outcome.cause,
-            mappedFailure(.{ .failed = .{
+            .phase = outcome.phase,
-                .cause = outcome.cause,
+            .received_any = true,
-                .phase = outcome.phase,
+        } });
-                .received_any = true,
+        try testing.expectEqual(outcome.phase, failed.kind);
-            } }),
+        try testing.expectEqual(outcome.cause, failed.cause);
-        );
     }
 }
 
@@ -830,26 +840,21 @@ test "a CA bundle scan failure keeps local resource errors out of the peer fault
             transport.Group.local_resource,
             transport.group(transport.mapPhase(err, error.TlsFailed)),
         );
+        try testing.expectError(err, transport.faultOrLocal(err, error.TlsFailed));
     }
 
-    try testing.expectEqual(
+    try testing.expectError(error.Canceled, transport.faultOrLocal(error.Canceled, error.TlsFailed));
-        transport.ExchangeError.Canceled,
-        transport.mapPhase(error.Canceled, error.TlsFailed),
-    );
 
     // A missing or corrupt bundle is not this process running out of anything,
-    // so it stays a TLS fault.
+    // so it stays a TLS fault, and the cause names which of the two it was.
-    try testing.expectEqual(
+    for ([_]anyerror{ error.FileNotFound, error.MissingEndCertificateMarker }) |err| {
-        transport.ExchangeError.TlsFailed,
+        const failed = try transport.faultOrLocal(err, error.TlsFailed);
-        transport.mapPhase(error.FileNotFound, error.TlsFailed),
+        try testing.expectEqual(transport.PeerFault.TlsFailed, failed.kind);
-    );
+        try testing.expectEqual(err, failed.cause);
-    try testing.expectEqual(
+    }
-        transport.ExchangeError.TlsFailed,
-        transport.mapPhase(error.MissingEndCertificateMarker, error.TlsFailed),
-    );
 }
 
-test "DotClient satisfies the Client interface" {
+test "DotClient satisfies the Leaf interface" {
     const gpa = testing.allocator;
 
     const buffer = try gpa.alloc(u8, 4 * tls.Client.min_buffer_len);
@@ -878,7 +883,7 @@ test "DotClient satisfies the Client interface" {
     dot.close(undefined);
     try testing.expect(dot.session == null);
 
-    const iface: transport.Client = dot.client();
+    const iface: transport.Leaf = dot.leaf();
     try testing.expectEqual(@as(*anyopaque, @ptrCast(&dot)), iface.ptr);
 }
 

src/upstream/dot_client_integration_test.zig

@@ -391,12 +391,12 @@ fn twoExchangesOverOneSession(io: std.Io, fixture: *ClientFixture) anyerror!void
     var buf: [512]u8 = undefined;
 
     const first = try fixture.dot.exchange(io, query_bytes, &buf);
-    try testing.expectEqualSlices(u8, response_bytes, first);
+    try testing.expectEqualSlices(u8, response_bytes, first.reply);
     // The point of the milestone: the connection outlives the exchange.
     try testing.expect(fixture.dot.session != null);
 
     const second = try fixture.dot.exchange(io, query_bytes, &buf);
-    try testing.expectEqualSlices(u8, response_bytes, second);
+    try testing.expectEqualSlices(u8, response_bytes, second.reply);
     try testing.expect(fixture.dot.session != null);
 }
 
@@ -451,7 +451,7 @@ test "a session the upstream closed is recovered by one redial and counted, not
     // Through a real pool entry, because the claim is about what the pool does
     // *not* see: an upstream reaping an idle connection must not cost it health
     // or raise an operational event.
-    var slots = [_]pool_mod.Slot{.{ .client = fixture.dot.client() }};
+    var slots = [_]pool_mod.Slot{.{ .client = fixture.dot.leaf() }};
     var entries = [_]pool_mod.Entry{.{
         .endpoint = fixture.dot.endpoint,
         .slots = &slots,
@@ -496,7 +496,7 @@ fn exchangeThenReadTruncatedReply(io: std.Io, fixture: *ClientFixture) anyerror!
 
     // One prefix byte arrived, so the reply had started: re-sending the query on
     // a fresh connection would be a second question, not a recovery.
-    try testing.expectError(error.ReceiveFailed, fixture.dot.exchange(io, query_bytes, &buf));
+    try expectFault(error.ReceiveFailed, error.EndOfStream, try fixture.dot.exchange(io, query_bytes, &buf));
     try testing.expect(fixture.dot.session == null);
 }
 
@@ -532,7 +532,7 @@ fn exchangeThenReadWrongId(io: std.Io, fixture: *ClientFixture) anyerror!void {
     // The read succeeded; only the bytes are wrong. Nothing about that says the
     // connection is stale, so it is final — but the stream position after a
     // frame this client will not trust is unknowable, so the session goes.
-    try testing.expectError(error.ResponseMismatch, fixture.dot.exchange(io, query_bytes, &buf));
+    try expectFault(error.ResponseMismatch, error.ResponseMismatch, try fixture.dot.exchange(io, query_bytes, &buf));
     try testing.expect(fixture.dot.session == null);
 }
 
@@ -571,7 +571,7 @@ fn exchangeThenFailTheRetry(io: std.Io, fixture: *ClientFixture) anyerror!void {
     // taken, and connected. The exchange on that fresh session then fails its
     // read, and the retry's outcome is the exchange's outcome — no third dial,
     // because a session this call dialed itself is never retried.
-    try testing.expectError(error.ReceiveFailed, fixture.dot.exchange(io, query_bytes, &buf));
+    try expectFault(error.ReceiveFailed, error.EndOfStream, try fixture.dot.exchange(io, query_bytes, &buf));
     try testing.expect(fixture.dot.session == null);
 
     // What the `nxdns check` probe loop relies on: closing a client whose
@@ -609,7 +609,20 @@ test "a stale session whose retry fails is final and leaves no session behind" {
     try testing.expectEqual(@as(u64, 0), fixture.recoveries.load(.acquire));
 }
 
-fn oneExchange(io: std.Io, fixture: *ClientFixture) transport.ExchangeError!void {
+/// A returned fault, whole: the classification the pool counts and the concrete
+/// cause it records. Both are asserted, because the cause is what an operator
+/// reads and only a real exchange proves it survives the wire.
+fn expectFault(kind: transport.PeerFault, cause: anyerror, outcome: transport.Outcome) !void {
+    switch (outcome) {
+        .reply => return error.TestExpectedFault,
+        .fault => |f| {
+            try testing.expectEqual(kind, f.kind);
+            try testing.expectEqual(cause, f.cause);
+        },
+    }
+}
+
+fn oneExchange(io: std.Io, fixture: *ClientFixture) transport.LeafError!void {
     var buf: [512]u8 = undefined;
     _ = try fixture.dot.exchange(io, query_bytes, &buf);
 }

src/upstream/dot_client_live_test.zig

@@ -62,10 +62,10 @@ fn runExchange(io: std.Io, params: Params) anyerror!usize {
     // so the task that built the client is what has to close it. Without this
     // the socket and its TLS state outlive the test.
     defer client.close(io);
-    var selected: ?[]const u8 = null;
+    return switch (try client.leaf().exchange(io, query_bytes, params.response_buf)) {
-    const reply = try client.client().exchange(io, query_bytes, params.response_buf, &selected);
+        .reply => |reply| reply.len,
-    std.debug.assert(std.mem.eql(u8, selected.?, endpoint.url));
+        .fault => |failed| failed.kind,
-    return reply.len;
+    };
 }
 
 fn expire(io: std.Io, duration: std.Io.Clock.Duration) std.Io.Cancelable!void {

src/upstream/health.zig

@@ -20,7 +20,7 @@
 //!   * Data is data. `total_successes`, `total_failures`, the window and the
 //!     `@max` of `last_success_at` / `last_error_at` always update, however old
 //!     `at` is.
-//!   * `consecutive_failures`, `backoff_until` and `last_error_buf` describe
+//!   * `consecutive_failures`, `backoff_until` and the last fault describe
 //!     the present, so a newer recorded outcome overrules a stale call.
 //!
 //! `recordSuccess` clears `consecutive_failures` and `backoff_until` only when
@@ -33,25 +33,67 @@
 //! `State` carries no lock. The pool owns the mutex.
 
 const std = @import("std");
+const transport = @import("transport.zig");
 
 pub const Config = struct {
-    /// Consecutive peer faults before the endpoint is put in backoff.
+    /// Consecutive peer faults before the endpoint is put in backoff. At least
+    /// 1: a state tripped by zero failures has no fault to name, and the
+    /// episode the pool opens from it could not say what went wrong. Nothing
+    /// builds this from user input, so the default is checked below and
+    /// `Pool.init` asserts what a caller passes.
     failure_threshold: u8 = 2,
     base_backoff_ms: u32 = 500,
     max_backoff_ms: u32 = 60_000,
 };
 
+comptime {
+    const default: Config = .{};
+    if (default.failure_threshold < 1) @compileError("failure_threshold must be at least 1");
+}
+
 /// Rolling success-rate window, in samples. Equal to the bit width of
 /// `State.window`.
 pub const window_len = 32;
 
-/// Bytes kept of an `@errorName`, truncated to fit.
+/// Bytes kept of a rendered `transport.Fault`, truncated to fit. Sized so the
-pub const error_name_capacity = 48;
+/// widest classification and the widest concrete cause name fit whole, and so
+/// does the widest HTTP status with its phrase; the comptime-driven test below
+/// recomputes that worst case from std, so a wider name std adds fails the test
+/// run rather than silently truncating an operator's only diagnostic.
+pub const error_name_capacity = 64;
 
 /// The shift is capped so `base_backoff_ms << shift` cannot run away; by then
 /// `max_backoff_ms` has clamped the result many doublings ago.
 const max_shift = 20;
 
+/// The complete desired state of this endpoint's Diagnostics episode after one
+/// recorded outcome. This file decides it, because "failing" is this file's
+/// predicate and nobody else's; the pool projects it onto the store.
+///
+/// A state, never a transition. A transition has to be applied to whatever the
+/// store holds, so two effects that reach the pool out of order can leave the
+/// wrong one standing: a "nothing changed" overtaking a required report would
+/// drop the report for good. A desired state makes last-writer-wins by
+/// revision correct: a projection of the whole state is also what repairs a
+/// card left over from a latched store write, on the next outcome. What a
+/// per-pool revision cannot see — an episode that outlived a restart, or one a
+/// retired generation opened — is repaired by `Pool.reconcile` instead, at boot
+/// and at every retirement, not by waiting for an outcome.
+pub const Effect = struct {
+    /// Per-state counter, incremented by every mutation. The pool applies
+    /// effects in this order and drops one that arrives behind a newer sibling.
+    revision: u64,
+    state: Episode,
+
+    /// A tagged union rather than an enum beside a fault field, so an episode
+    /// without the fault that explains it is unrepresentable. Named `Episode`
+    /// rather than `State` because this file's `State` is the health record.
+    pub const Episode = union(enum) {
+        clear,
+        tripped: transport.Fault,
+    };
+};
+
 pub const State = struct {
     consecutive_failures: u32,
     total_successes: u64,
@@ -62,6 +104,12 @@ pub const State = struct {
     /// Length of the `@errorName` held in `last_error_buf`, truncated to fit.
     last_error_len: u8,
     backoff_until: ?std.Io.Timestamp,
+    /// The effective last fault, as a value. `last_error_buf` is its rendering;
+    /// both are kept because the text is what every surface prints and the
+    /// value is what an `Effect` carries out of the mutex.
+    last_fault: ?transport.Fault,
+    /// Incremented by every mutation; see `Effect.revision`.
+    revision: u64,
     /// Bitset, 1 = success, LSB = most recent.
     window: u32,
     window_filled: u8,
@@ -75,11 +123,22 @@ pub const State = struct {
         .last_error_buf = @splat(0),
         .last_error_len = 0,
         .backoff_until = null,
+        .last_fault = null,
+        .revision = 0,
         .window = 0,
         .window_filled = 0,
     };
 
-    pub fn recordSuccess(self: *State, at: std.Io.Timestamp) void {
+    /// A success that is the newest outcome clears the count, and the effect
+    /// then says `clear`.
+    ///
+    /// `null` when the state stays tripped, which is the stale success behind a
+    /// newer failure (case 1 at `recordFailure`). The card that failure opened
+    /// is already right, and projecting `tripped` again would count a
+    /// successful exchange as an occurrence of the episode. A `clear`-shaped
+    /// no-op is not an option either: it would advance `applied_revision` past
+    /// a report still in flight and drop it.
+    pub fn recordSuccess(self: *State, at: std.Io.Timestamp, cfg: Config) ?Effect {
         if (self.isNewestOutcome(at)) {
             self.consecutive_failures = 0;
             self.backoff_until = null;
@@ -87,6 +146,32 @@ pub const State = struct {
         self.total_successes += 1;
         self.push(1);
         self.last_success_at = later(self.last_success_at, at);
+        self.revision += 1;
+        if (self.tripped(cfg)) return null;
+        return self.effect(cfg);
+    }
+
+    /// The pool's definition of "failing this endpoint": the predicate that
+    /// puts it in backoff. Not `available`, which turns true again the moment a
+    /// backoff expires, before any recovery is proven.
+    pub fn tripped(self: *const State, cfg: Config) bool {
+        return self.consecutive_failures >= cfg.failure_threshold;
+    }
+
+    /// The desired episode state this endpoint is in right now, without a
+    /// mutation and without advancing the revision. The pool projects this at
+    /// the two points a recorded outcome cannot reach: the first generation's
+    /// boot, and the retirement of a generation whose late outcomes are now
+    /// impossible.
+    pub fn currentEffect(self: *const State, cfg: Config) Effect {
+        return self.effect(cfg);
+    }
+
+    /// A tripped state has always recorded a failure, because the threshold is
+    /// at least 1, so the fault is always there to name.
+    fn effect(self: *const State, cfg: Config) Effect {
+        if (!self.tripped(cfg)) return .{ .revision = self.revision, .state = .clear };
+        return .{ .revision = self.revision, .state = .{ .tripped = self.last_fault.? } };
     }
 
     /// True when no recorded outcome is newer than `at`. Both timestamp fields
@@ -96,9 +181,9 @@ pub const State = struct {
         return !newerThan(self.last_success_at, at) and !newerThan(self.last_error_at, at);
     }
 
-    /// `err_name` is `@errorName` of a PeerFault member. `rand` supplies
+    /// `fault` is the peer fault as a value: `transport.group` has already
-    /// jitter; the caller owns the RNG so this stays pure and the test is
+    /// ruled that this is the peer's doing. `rand` supplies jitter; the caller
-    /// deterministic.
+    /// owns the RNG so this stays pure and the test is deterministic.
     ///
     /// A stale failure, that is one whose `at` is older than an outcome already
     /// recorded, is held to the mirror image of the stale-success rule. Three
@@ -119,27 +204,29 @@ pub const State = struct {
     pub fn recordFailure(
         self: *State,
         at: std.Io.Timestamp,
-        err_name: []const u8,
+        fault: transport.Fault,
         cfg: Config,
         rand: u32,
-    ) void {
+    ) Effect {
         const newer_success = newerThan(self.last_success_at, at);
         const newer_failure = newerThan(self.last_error_at, at);
 
         if (!newer_failure) {
-            const copied = @min(err_name.len, self.last_error_buf.len);
+            self.last_fault = fault;
-            @memcpy(self.last_error_buf[0..copied], err_name[0..copied]);
+            var writer: std.Io.Writer = .fixed(&self.last_error_buf);
-            self.last_error_len = @intCast(copied);
+            writer.print("{f}", .{fault}) catch {};
+            self.last_error_len = @intCast(writer.end);
         }
 
         self.total_failures += 1;
         self.push(0);
         self.last_error_at = later(self.last_error_at, at);
+        self.revision += 1;
 
-        if (newer_success) return;
+        if (newer_success) return self.effect(cfg);
 
         self.consecutive_failures +|= 1;
-        if (self.consecutive_failures < cfg.failure_threshold) return;
+        if (self.consecutive_failures < cfg.failure_threshold) return self.effect(cfg);
 
         const delay_ms = backoffDelayMs(self.consecutive_failures, cfg);
         const half = delay_ms / 2;
@@ -148,6 +235,7 @@ pub const State = struct {
             .nanoseconds = at.nanoseconds + @as(i96, jittered) * std.time.ns_per_ms,
         };
         self.backoff_until = later(self.backoff_until, deadline);
+        return self.effect(cfg);
     }
 
     pub fn available(self: *const State, now: std.Io.Timestamp) bool {
@@ -212,10 +300,16 @@ fn ms(count: i96) i96 {
     return count * std.time.ns_per_ms;
 }
 
+/// A fault whose cause is its own classification: the shape an expiry has, and
+/// short enough to keep the timestamp tests about timestamps.
+fn peerFault(kind: transport.PeerFault) transport.Fault {
+    return .{ .kind = kind, .cause = kind };
+}
+
 test "a failure below the threshold leaves the endpoint available" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(0), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
 
     try testing.expectEqual(@as(u32, 1), state.consecutive_failures);
     try testing.expectEqual(@as(u64, 1), state.total_failures);
@@ -226,8 +320,8 @@ test "a failure below the threshold leaves the endpoint available" {
 test "reaching the threshold puts the endpoint in backoff until the deadline" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(0), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(0), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
 
     // Two failures, threshold 2, shift 0: delay 500 ms, jitter 0 => 250 ms.
     const until = state.backoff_until.?;
@@ -244,7 +338,7 @@ test "consecutive failures grow the delay and saturate at max_backoff_ms" {
     var previous: i96 = -1;
     var i: usize = 0;
     while (i < 40) : (i += 1) {
-        state.recordFailure(ts(0), "Timeout", cfg, 0);
+        _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
         if (state.backoff_until) |until| {
             try testing.expect(until.nanoseconds >= previous);
             previous = until.nanoseconds;
@@ -262,11 +356,11 @@ test "consecutive failures grow the delay and saturate at max_backoff_ms" {
 test "a success resets the consecutive count, the window and the backoff" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(0), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(0), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
     try testing.expect(state.backoff_until != null);
 
-    state.recordSuccess(ts(ms(1)));
+    _ = state.recordSuccess(ts(ms(1)), cfg);
     try testing.expectEqual(@as(u32, 0), state.consecutive_failures);
     try testing.expectEqual(@as(?std.Io.Timestamp, null), state.backoff_until);
     try testing.expectEqual(@as(u64, 1), state.total_successes);
@@ -280,8 +374,8 @@ test "jitter stays inside half the delay and the whole delay" {
 
     for ([_]u32{ 0, std.math.maxInt(u32), 1, 12345 }) |rand| {
         var state: State = .init;
-        state.recordFailure(ts(0), "Timeout", cfg, rand);
+        _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, rand);
-        state.recordFailure(ts(0), "Timeout", cfg, rand);
+        _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, rand);
         const offset = state.backoff_until.?.nanoseconds;
         try testing.expect(offset >= ms(delay / 2));
         try testing.expect(offset <= ms(delay));
@@ -289,20 +383,21 @@ test "jitter stays inside half the delay and the whole delay" {
 }
 
 test "an out-of-order success does not move last_success_at backwards" {
+    const cfg: Config = .{};
     var state: State = .init;
-    state.recordSuccess(ts(100));
+    _ = state.recordSuccess(ts(100), cfg);
-    state.recordSuccess(ts(50));
+    _ = state.recordSuccess(ts(50), cfg);
     try testing.expectEqual(@as(i96, 100), state.last_success_at.?.nanoseconds);
 }
 
 test "an out-of-order failure does not shorten the backoff" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
     const until = state.backoff_until.?.nanoseconds;
 
-    state.recordFailure(ts(ms(50)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(50)), peerFault(error.Timeout), cfg, 0);
     try testing.expect(state.backoff_until.?.nanoseconds >= until);
     try testing.expectEqual(@as(i96, ms(100)), state.last_error_at.?.nanoseconds);
 }
@@ -310,11 +405,11 @@ test "an out-of-order failure does not shorten the backoff" {
 test "an out-of-order success does not clear the backoff of a newer failure" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
     const until = state.backoff_until.?.nanoseconds;
 
-    state.recordSuccess(ts(ms(50)));
+    _ = state.recordSuccess(ts(ms(50)), cfg);
     try testing.expectEqual(@as(i96, until), state.backoff_until.?.nanoseconds);
     try testing.expectEqual(@as(u32, 2), state.consecutive_failures);
     try testing.expectEqual(@as(u64, 1), state.total_successes);
@@ -325,10 +420,10 @@ test "an out-of-order success does not clear the backoff of a newer failure" {
 test "a stale failure behind a newer success does not raise the consecutive count" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(10)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(10)), peerFault(error.Timeout), cfg, 0);
-    state.recordSuccess(ts(ms(100)));
+    _ = state.recordSuccess(ts(ms(100)), cfg);
-    state.recordFailure(ts(ms(20)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(20)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(30)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(30)), peerFault(error.Timeout), cfg, 0);
 
     try testing.expectEqual(@as(u32, 0), state.consecutive_failures);
     try testing.expectEqual(@as(?std.Io.Timestamp, null), state.backoff_until);
@@ -338,21 +433,21 @@ test "a stale failure behind a newer success does not raise the consecutive coun
 test "a stale failure behind a newer success still counts into the totals" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordSuccess(ts(ms(100)));
+    _ = state.recordSuccess(ts(ms(100)), cfg);
-    state.recordFailure(ts(ms(20)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(20)), peerFault(error.Timeout), cfg, 0);
 
     try testing.expectEqual(@as(u64, 1), state.total_failures);
     try testing.expectEqual(@as(u8, 2), state.window_filled);
     try testing.expectEqual(@as(u32, 0), state.window & 1);
     try testing.expectEqual(@as(i96, ms(20)), state.last_error_at.?.nanoseconds);
-    try testing.expectEqualStrings("Timeout", state.lastError());
+    try testing.expectEqualStrings("Timeout (cause Timeout)", state.lastError());
 }
 
 test "a stale failure with no newer success still counts as consecutive" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(50)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(50)), peerFault(error.Timeout), cfg, 0);
 
     try testing.expectEqual(@as(u32, 2), state.consecutive_failures);
     try testing.expect(state.backoff_until != null);
@@ -362,19 +457,19 @@ test "a stale failure with no newer success still counts as consecutive" {
 test "a stale failure does not overwrite the error of a newer failure" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(50)), "ConnectFailed", cfg, 0);
+    _ = state.recordFailure(ts(ms(50)), peerFault(error.ConnectFailed), cfg, 0);
 
-    try testing.expectEqualStrings("Timeout", state.lastError());
+    try testing.expectEqualStrings("Timeout (cause Timeout)", state.lastError());
 }
 
 test "the newest failure records its own error and extends the backoff" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(50)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(50)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(100)), "ConnectFailed", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.ConnectFailed), cfg, 0);
 
-    try testing.expectEqualStrings("ConnectFailed", state.lastError());
+    try testing.expectEqualStrings("ConnectFailed (cause ConnectFailed)", state.lastError());
     try testing.expectEqual(@as(u32, 2), state.consecutive_failures);
     try testing.expectEqual(ms(100) + ms(250), state.backoff_until.?.nanoseconds);
 }
@@ -382,11 +477,11 @@ test "the newest failure records its own error and extends the backoff" {
 test "the newest success clears the backoff of an older failure" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
     try testing.expect(state.backoff_until != null);
 
-    state.recordSuccess(ts(ms(101)));
+    _ = state.recordSuccess(ts(ms(101)), cfg);
     try testing.expectEqual(@as(?std.Io.Timestamp, null), state.backoff_until);
     try testing.expectEqual(@as(u32, 0), state.consecutive_failures);
 }
@@ -394,10 +489,10 @@ test "the newest success clears the backoff of an older failure" {
 test "a success at the timestamp of the newest failure clears the backoff" {
     const cfg: Config = .{};
     var state: State = .init;
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
-    state.recordFailure(ts(ms(100)), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(ms(100)), peerFault(error.Timeout), cfg, 0);
 
-    state.recordSuccess(ts(ms(100)));
+    _ = state.recordSuccess(ts(ms(100)), cfg);
     try testing.expectEqual(@as(?std.Io.Timestamp, null), state.backoff_until);
     try testing.expectEqual(@as(u32, 0), state.consecutive_failures);
 }
@@ -409,37 +504,188 @@ test "successRate over a half-success window is 0.5" {
 
     var i: usize = 0;
     while (i < 4) : (i += 1) {
-        state.recordSuccess(ts(0));
+        _ = state.recordSuccess(ts(0), cfg);
-        state.recordFailure(ts(0), "Timeout", cfg, 0);
+        _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
     }
     try testing.expectEqual(@as(u8, 8), state.window_filled);
     try testing.expectEqual(@as(f32, 0.5), state.successRate());
 }
 
 test "successRate counts only the filled part of the window" {
+    const cfg: Config = .{};
     var state: State = .init;
-    state.recordSuccess(ts(0));
+    _ = state.recordSuccess(ts(0), cfg);
     try testing.expectEqual(@as(f32, 1.0), state.successRate());
 
     var i: usize = 0;
-    while (i < window_len * 2) : (i += 1) state.recordSuccess(ts(0));
+    while (i < window_len * 2) : (i += 1) _ = state.recordSuccess(ts(0), cfg);
     try testing.expectEqual(@as(u8, window_len), state.window_filled);
     try testing.expectEqual(@as(f32, 1.0), state.successRate());
 }
 
-test "lastError returns the last recorded name, truncated not overflowed" {
+test "lastError renders the classification and the concrete cause" {
     const cfg: Config = .{};
     var state: State = .init;
     try testing.expectEqualStrings("", state.lastError());
 
-    state.recordFailure(ts(0), "ConnectFailed", cfg, 0);
+    _ = state.recordFailure(ts(0), .{ .kind = error.SendFailed, .cause = error.BrokenPipe }, cfg, 0);
-    try testing.expectEqualStrings("ConnectFailed", state.lastError());
+    try testing.expectEqualStrings("SendFailed (cause BrokenPipe)", state.lastError());
+    try testing.expectEqual(transport.PeerFault.SendFailed, state.last_fault.?.kind);
 
-    state.recordFailure(ts(0), "Timeout", cfg, 0);
+    _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
-    try testing.expectEqualStrings("Timeout", state.lastError());
+    try testing.expectEqualStrings("Timeout (cause Timeout)", state.lastError());
+}
 
-    const long = "A" ** 200;
+test "lastError truncates a cause too wide for the buffer instead of overflowing" {
-    state.recordFailure(ts(0), long, cfg, 0);
+    const cfg: Config = .{};
-    try testing.expectEqual(@as(usize, 48), state.lastError().len);
+    var state: State = .init;
-    try testing.expectEqualStrings(long[0..48], state.lastError());
+    const Wide = error{AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA};
+    _ = state.recordFailure(
+        ts(0),
+        .{ .kind = error.ReceiveFailed, .cause = Wide.AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA },
+        cfg,
+        0,
+    );
+
+    const whole = "ReceiveFailed (cause " ++ "A" ** 64 ++ ")";
+    try testing.expectEqual(@as(usize, error_name_capacity), state.lastError().len);
+    try testing.expectEqualStrings(whole[0..error_name_capacity], state.lastError());
+}
+
+test "error_name_capacity holds the widest classification, cause and status whole" {
+    // The widest concrete cause an upstream unwrap can produce. The three sets
+    // are the ones `transport.zig`'s unwraps return a member of; a wider name
+    // std adds fails here rather than truncating an operator's diagnostic.
+    const widest_cause = comptime blk: {
+        var widest: []const u8 = "";
+        for (@typeInfo(std.crypto.tls.Client.ReadError).error_set.?) |member| {
+            if (member.name.len > widest.len) widest = member.name;
+        }
+        for (@typeInfo(std.http.Client.RequestError).error_set.?) |member| {
+            if (member.name.len > widest.len) widest = member.name;
+        }
+        for (@typeInfo(std.http.Reader.BodyError).error_set.?) |member| {
+            if (member.name.len > widest.len) widest = member.name;
+        }
+        break :blk widest;
+    };
+    const widest_kind = comptime blk: {
+        var widest: []const u8 = "";
+        for (@typeInfo(transport.PeerFault).error_set.?) |member| {
+            if (member.name.len > widest.len) widest = member.name;
+        }
+        break :blk widest;
+    };
+
+    try testing.expectEqualStrings("DetectingNetworkConfigurationFailed", widest_cause);
+    try testing.expect(widest_kind.len + " (cause ".len + widest_cause.len + ")".len <= error_name_capacity);
+
+    const widest_phrase = comptime blk: {
+        var widest: []const u8 = "";
+        for (@typeInfo(std.http.Status).@"enum".fields) |field| {
+            const phrase = @as(std.http.Status, @enumFromInt(field.value)).phrase() orelse continue;
+            if (phrase.len > widest.len) widest = phrase;
+        }
+        break :blk widest;
+    };
+    // `std.http.Status` is a u10, so a code has at most four digits.
+    const widest_status = "HttpStatus (status ".len + 4 + " ".len + widest_phrase.len + ")".len;
+    try testing.expect(widest_status <= error_name_capacity);
+}
+
+/// The fault of a `tripped` effect, or an error when the effect says `clear`.
+fn trippedFault(effect: Effect) !transport.Fault {
+    return switch (effect.state) {
+        .clear => error.TestExpectedTripped,
+        .tripped => |fault| fault,
+    };
+}
+
+test "one failure stays clear and the second trips the episode" {
+    const cfg: Config = .{};
+    var state: State = .init;
+
+    const first = state.recordFailure(ts(0), peerFault(error.ConnectFailed), cfg, 0);
+    try testing.expectEqual(Effect.Episode.clear, first.state);
+
+    const second = state.recordFailure(ts(ms(1)), .{ .kind = error.SendFailed, .cause = error.BrokenPipe }, cfg, 0);
+    const fault = try trippedFault(second);
+    try testing.expectEqual(transport.PeerFault.SendFailed, fault.kind);
+    try testing.expectEqual(@as(anyerror, error.BrokenPipe), fault.cause);
+    try testing.expect(second.revision > first.revision);
+}
+
+test "a failure while tripped stays tripped" {
+    const cfg: Config = .{};
+    var state: State = .init;
+    _ = state.recordFailure(ts(0), peerFault(error.Timeout), cfg, 0);
+    _ = state.recordFailure(ts(ms(1)), peerFault(error.Timeout), cfg, 0);
+
+    const third = state.recordFailure(ts(ms(2)), peerFault(error.Timeout), cfg, 0);
+    _ = try trippedFault(third);
+}
+
+test "a success on a tripped state returns clear, and so does one below the threshold" {
+    const cfg: Config = .{};
+    var state: State = .init;
+
+    try testing.expectEqual(Effect.Episode.clear, state.recordSuccess(ts(0), cfg).?.state);
+
+    _ = state.recordFailure(ts(ms(1)), peerFault(error.Timeout), cfg, 0);
+    try testing.expectEqual(Effect.Episode.clear, state.recordSuccess(ts(ms(2)), cfg).?.state);
+
+    _ = state.recordFailure(ts(ms(3)), peerFault(error.Timeout), cfg, 0);
+    _ = state.recordFailure(ts(ms(4)), peerFault(error.Timeout), cfg, 0);
+    try testing.expectEqual(Effect.Episode.clear, state.recordSuccess(ts(ms(5)), cfg).?.state);
+    try testing.expectEqual(Effect.Episode.clear, state.recordSuccess(ts(ms(6)), cfg).?.state);
+}
+
+test "a stale success behind a newer failure projects nothing" {
+    const cfg: Config = .{};
+    var state: State = .init;
+    _ = state.recordFailure(ts(ms(10)), peerFault(error.Timeout), cfg, 0);
+    _ = state.recordFailure(ts(ms(20)), peerFault(error.Timeout), cfg, 0);
+
+    // The endpoint is still failing, so the card the failures opened is
+    // already right. Reporting it again on a success would count that success
+    // as an occurrence of the episode.
+    try testing.expectEqual(@as(?Effect, null), state.recordSuccess(ts(ms(5)), cfg));
+    try testing.expect(state.tripped(cfg));
+    try testing.expectEqual(@as(u64, 1), state.total_successes);
+}
+
+test "a stale failure carries the newer effective fault, not its own" {
+    const cfg: Config = .{};
+    var state: State = .init;
+    _ = state.recordFailure(ts(ms(10)), peerFault(error.Timeout), cfg, 0);
+    const newest = state.recordFailure(
+        ts(ms(20)),
+        .{ .kind = error.SendFailed, .cause = error.BrokenPipe },
+        cfg,
+        0,
+    );
+    _ = try trippedFault(newest);
+
+    const stale = state.recordFailure(ts(ms(15)), .{ .kind = error.ConnectFailed, .cause = error.ConnectionRefused }, cfg, 0);
+    const fault = try trippedFault(stale);
+    try testing.expectEqual(transport.PeerFault.SendFailed, fault.kind);
+    try testing.expectEqual(@as(anyerror, error.BrokenPipe), fault.cause);
+    try testing.expectEqualStrings("SendFailed (cause BrokenPipe)", state.lastError());
+}
+
+test "every mutation advances the revision" {
+    const cfg: Config = .{};
+    var state: State = .init;
+    var previous: u64 = 0;
+
+    // The successes here are all the newest outcome, so none of them is the
+    // one case that projects nothing.
+    for (0..8) |i| {
+        const effect = if (i % 3 == 0)
+            state.recordSuccess(ts(ms(@intCast(i))), cfg).?
+        else
+            state.recordFailure(ts(ms(@intCast(i))), peerFault(error.Timeout), cfg, 0);
+        try testing.expect(effect.revision > previous);
+        previous = effect.revision;
+    }
 }

src/upstream/owner.zig

@@ -23,6 +23,7 @@ const tls = std.crypto.tls;
 const doh_client = @import("doh_client.zig");
 const dot_client = @import("dot_client.zig");
 const events = @import("../storage/events.zig");
+const events_fixture = @import("../storage/events_fixture.zig");
 const health = @import("health.zig");
 const model = @import("../config/model.zig");
 const pool_mod = @import("pool.zig");
@@ -285,6 +286,11 @@ pub const Owner = struct {
     /// `old.refs == 0` comparison and the branch on its result, and `replace`
     /// then returns null for every input. See AGENTS.md.
     published: u64 = 0,
+    /// The Diagnostics store, for the `upstream.exchange` reconciliation at
+    /// boot and at every retirement. Defaulted rather than an `init` parameter,
+    /// for the reason `Pool.diagnostics` is: the composition root wires it
+    /// after the owner exists, and every unit test here runs without one.
+    diagnostics: ?*events.Store = null,
 
     pub fn init(live: *Generation) Owner {
         return .{ .live = live };
@@ -323,13 +329,84 @@ pub const Owner = struct {
         return copies;
     }
 
+    /// Drops one pin, and tears the generation down when it was retired and
+    /// this was its last reader.
     pub fn release(self: *Owner, io: std.Io, generation: *Generation) void {
+        if (self.drop(io, generation)) self.retireDisplaced(io, generation);
+    }
+
+    /// Drops one pin and says whether that made the generation this caller's to
+    /// tear down. The mutex is released before the caller acts on the answer,
+    /// because tearing a generation down closes sockets.
+    fn drop(self: *Owner, io: std.Io, generation: *Generation) bool {
         self.mutex.lockUncancelable(io);
+        defer self.mutex.unlock(io);
         std.debug.assert(generation.refs > 0);
         generation.refs -= 1;
-        const retire_it = generation.retired and generation.refs == 0;
+        return generation.retired and generation.refs == 0;
-        self.mutex.unlock(io);
+    }
-        if (retire_it) generation.retire(io);
+
+    /// Tears a displaced generation down and reconciles its episodes against
+    /// whatever is live now.
+    ///
+    /// The one retire-and-reconcile, and both paths a displaced generation can
+    /// leave by call it: `release` when the last reader of a retired generation
+    /// goes, and the publisher when `replace` found no reader at all and handed
+    /// the generation back. Two call sites and one rule, so the idle path
+    /// cannot skip the reconciliation the pinned path does — which is what it
+    /// did before this.
+    ///
+    /// This instant is the whole point: after it, no exchange of `generation`
+    /// can project anything onto the store, so whatever it left standing is
+    /// nobody's truth and the live generation's health is the answer.
+    ///
+    /// The loop is the pin the reconcile itself takes. Reconciling reads the
+    /// generation that is live now, which a concurrent `replace` can retire
+    /// under it, and dropping that pin lands back here. One iteration per
+    /// concurrent replace, and a replace is a configuration write.
+    pub fn retireDisplaced(self: *Owner, io: std.Io, generation: *Generation) void {
+        // Only the publisher's idle path and a reader's last release reach
+        // here, and both hold a generation the swap already displaced.
+        std.debug.assert(generation.retired);
+        std.debug.assert(generation.refs == 0);
+
+        var target = generation;
+        while (true) {
+            target.retire(io);
+            // No store means nothing to reconcile, and then no pin to take.
+            if (self.diagnostics == null) return;
+
+            const live = self.acquire(io);
+            self.reconcileDiagnostics(io, live);
+            if (!self.drop(io, live)) return;
+            target = live;
+        }
+    }
+
+    /// Closes every `upstream.exchange` episode `generation` cannot justify,
+    /// and projects the health of the endpoints it can.
+    ///
+    /// `resolveExcept` rather than the scoped walk `reconcileReport` does,
+    /// because the two codes have different owners. Every `upstream.exchange`
+    /// episode in the store was opened by a pool of this process, so an active
+    /// one outside the kept set names an upstream that was removed, disabled or
+    /// failed to build, and closing it is right. `configuration.load` is shared
+    /// with the boot collector, which is why its rule stays scoped.
+    ///
+    /// Called at the two points revision order cannot reach: the first
+    /// generation at boot, and the retirement of a displaced one.
+    pub fn reconcileDiagnostics(self: *Owner, io: std.Io, generation: *Generation) void {
+        const store = self.diagnostics orelse return;
+        const pool = generation.pool orelse return;
+
+        // The store refuses a kept list longer than it can canonicalize, and
+        // refusing is right there, so the two bounds must agree rather than one
+        // silently clipping the other.
+        comptime std.debug.assert(pool_mod.Pool.max_entries <= events.Store.max_kept_keys);
+        var kept: [pool_mod.Pool.max_entries][]const u8 = undefined;
+        const count = pool.enabledUrls(&kept);
+        store.resolveExcept(io, std.Io.Clock.real.now(io).toSeconds(), .upstream_exchange, kept[0..count]);
+        pool.reconcile(io);
     }
 
     /// Publishes `prepared` and retires the live generation. Infallible and
@@ -534,7 +611,7 @@ const Upstreams = struct {
                 self.doh_buf[base + doh_request_buf_len ..][0..doh_transfer_buf_len],
             ) catch return false;
             self.doh_used = index + 1;
-            slot.* = .{ .client = self.doh[index].client() };
+            slot.* = .{ .client = self.doh[index].leaf() };
         }
         return true;
     }
@@ -571,7 +648,7 @@ const Upstreams = struct {
                 },
             );
             self.dot_used = index + 1;
-            slot.* = .{ .client = self.dot[index].client() };
+            slot.* = .{ .client = self.dot[index].leaf() };
         }
     }
 
@@ -679,6 +756,21 @@ fn buildTestGeneration(
     bundle: *Certificate.Bundle,
     bundle_lock: *std.Io.RwLock,
     servers: []const model.UpstreamServer,
+) BuildError!*Generation {
+    return buildTestGenerationWith(io, gpa, http, bundle, bundle_lock, servers, null);
+}
+
+/// The generation a reconciliation test needs: its pool projects onto the same
+/// store the owner reconciles against, which is how the composition root wires
+/// the two.
+fn buildTestGenerationWith(
+    io: std.Io,
+    gpa: Allocator,
+    http: *std.http.Client,
+    bundle: *Certificate.Bundle,
+    bundle_lock: *std.Io.RwLock,
+    servers: []const model.UpstreamServer,
+    store: ?*events.Store,
 ) BuildError!*Generation {
     return build(.{
         .gpa = gpa,
@@ -689,6 +781,7 @@ fn buildTestGeneration(
         .bundle_lock = bundle_lock,
         .timeouts = test_timeouts,
         .seed = 1,
+        .diagnostics = store,
     });
 }
 
@@ -743,7 +836,7 @@ test "a replace with no reader holding the live generation retires it through th
     // Nobody holds G1: `replace` must hand it back, because no release will.
     const displaced = owner.replace(io, g2) orelse return error.ExpectedIdleGeneration;
     try testing.expectEqual(g1, displaced);
-    displaced.retire(io);
+    owner.retireDisplaced(io, displaced);
 
     owner.deinit(io);
 }
@@ -918,9 +1011,177 @@ test "a metrics scrape running against the owner survives a replace under it" {
     var future = try io.concurrent(Scrape.run, .{ &owner, io, &started, &seen });
 
     started.waitUncancelable(io);
-    if (owner.replace(io, g2)) |old| old.retire(io);
+    if (owner.replace(io, g2)) |old| owner.retireDisplaced(io, old);
 
     future.await(io);
     // Every one of the 256 scrapes read at least one intact URL.
     try testing.expect(seen >= 256);
 }
+
+test "boot reconciliation closes the episodes of upstreams this generation does not serve" {
+    var t: TestIo = .init(testing.allocator);
+    defer t.deinit();
+    const io = t.io();
+
+    var http: std.http.Client = .{ .allocator = testing.allocator, .io = io };
+    defer http.deinit();
+    var bundle: Certificate.Bundle = .empty;
+    defer bundle.deinit(testing.allocator);
+    var bundle_lock: std.Io.RwLock = .init;
+
+    // Two episodes outlived the process that opened them. One names the
+    // upstream this generation serves and one does not, and neither has a pool
+    // that could ever resolve it through an exchange: the first has recorded
+    // nothing yet, and the second no longer exists in the configuration at all.
+    var fx: events_fixture.Fixture = .{};
+    try fx.init(io, 1000);
+    defer fx.deinit();
+    fx.store.report(io, 1000, .upstream_exchange, "https://kept.example/dns-query", "https://kept.example", .warning, "before the restart");
+    fx.store.report(io, 1000, .upstream_exchange, "https://gone.example/dns-query", "https://gone.example", .warning, "before the restart");
+
+    const generation = try buildTestGenerationWith(io, testing.allocator, &http, &bundle, &bundle_lock, &.{
+        .{ .url = "https://kept.example/dns-query" },
+    }, &fx.store);
+    var owner: Owner = .init(generation);
+    defer owner.deinit(io);
+    owner.diagnostics = &fx.store;
+
+    owner.reconcileDiagnostics(io, generation);
+
+    try testing.expectEqual(
+        @as(i64, 0),
+        try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
+    );
+}
+
+test "an idle replace that drops an upstream closes its episode through retireDisplaced" {
+    var t: TestIo = .init(testing.allocator);
+    defer t.deinit();
+    const io = t.io();
+
+    var http: std.http.Client = .{ .allocator = testing.allocator, .io = io };
+    defer http.deinit();
+    var bundle: Certificate.Bundle = .empty;
+    defer bundle.deinit(testing.allocator);
+    var bundle_lock: std.Io.RwLock = .init;
+
+    var fx: events_fixture.Fixture = .{};
+    try fx.init(io, 1000);
+    defer fx.deinit();
+
+    const g1 = try buildTestGenerationWith(io, testing.allocator, &http, &bundle, &bundle_lock, &.{
+        .{ .url = "https://gone.example/dns-query" },
+    }, &fx.store);
+    const g2 = try buildTestGenerationWith(io, testing.allocator, &http, &bundle, &bundle_lock, &.{
+        .{ .url = "https://kept.example/dns-query" },
+    }, &fx.store);
+
+    var owner: Owner = .init(g1);
+    defer owner.deinit(io);
+    owner.diagnostics = &fx.store;
+
+    fx.store.report(io, 1100, .upstream_exchange, "https://gone.example/dns-query", "https://gone.example", .warning, "before the reload");
+
+    // No reader holds G1, so `replace` hands it back for the publisher to tear
+    // down. That teardown is the reconciliation point for the upstream the new
+    // configuration no longer serves.
+    const displaced = owner.replace(io, g2) orelse return error.TestUnexpectedResult;
+    try testing.expectEqual(
+        @as(i64, 1),
+        try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
+    );
+
+    owner.retireDisplaced(io, displaced);
+
+    try testing.expectEqual(
+        @as(i64, 0),
+        try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
+    );
+}
+
+test "an idle replace resolves an episode the live pool never opened" {
+    var t: TestIo = .init(testing.allocator);
+    defer t.deinit();
+    const io = t.io();
+
+    var http: std.http.Client = .{ .allocator = testing.allocator, .io = io };
+    defer http.deinit();
+    var bundle: Certificate.Bundle = .empty;
+    defer bundle.deinit(testing.allocator);
+    var bundle_lock: std.Io.RwLock = .init;
+
+    var fx: events_fixture.Fixture = .{};
+    try fx.init(io, 1000);
+    defer fx.deinit();
+
+    const g1 = try buildTestGenerationWith(io, testing.allocator, &http, &bundle, &bundle_lock, &.{
+        .{ .url = "https://one.example/dns-query" },
+    }, &fx.store);
+    const g2 = try buildTestGenerationWith(io, testing.allocator, &http, &bundle, &bundle_lock, &.{
+        .{ .url = "https://one.example/dns-query" },
+    }, &fx.store);
+
+    var owner: Owner = .init(g1);
+    defer owner.deinit(io);
+    owner.diagnostics = &fx.store;
+
+    // The same upstream survives the reload, so `resolveExcept` keeps the card
+    // and only the fresh pool's own projection can close it. That pool has
+    // recorded nothing, so it projects clear.
+    fx.store.report(io, 1100, .upstream_exchange, "https://one.example/dns-query", "https://one.example", .warning, "opened by the old generation");
+
+    const displaced = owner.replace(io, g2) orelse return error.TestUnexpectedResult;
+    owner.retireDisplaced(io, displaced);
+
+    try testing.expectEqual(
+        @as(i64, 0),
+        try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
+    );
+}
+
+test "a retired generation's late report is corrected when its last reader leaves" {
+    var t: TestIo = .init(testing.allocator);
+    defer t.deinit();
+    const io = t.io();
+
+    var http: std.http.Client = .{ .allocator = testing.allocator, .io = io };
+    defer http.deinit();
+    var bundle: Certificate.Bundle = .empty;
+    defer bundle.deinit(testing.allocator);
+    var bundle_lock: std.Io.RwLock = .init;
+
+    var fx: events_fixture.Fixture = .{};
+    try fx.init(io, 1000);
+    defer fx.deinit();
+
+    const g1 = try buildTestGenerationWith(io, testing.allocator, &http, &bundle, &bundle_lock, &.{
+        .{ .url = "https://one.example/dns-query" },
+    }, &fx.store);
+    const g2 = try buildTestGenerationWith(io, testing.allocator, &http, &bundle, &bundle_lock, &.{
+        .{ .url = "https://one.example/dns-query" },
+    }, &fx.store);
+
+    var owner: Owner = .init(g1);
+    defer owner.deinit(io);
+    owner.diagnostics = &fx.store;
+
+    // A reader pins G1 across the replace, which is what lets G1 project after
+    // G2 is live. Revisions are per pool, so G2 has no way to know that report
+    // happened; the release below is the point after which G1 can project no
+    // more, and reconciling there is what corrects it.
+    const held = owner.acquire(io);
+    try testing.expectEqual(@as(?*Generation, null), owner.replace(io, g2));
+
+    fx.store.report(io, 1100, .upstream_exchange, "https://one.example/dns-query", "https://one.example", .warning, "the retired generation's last word");
+    try testing.expectEqual(
+        @as(i64, 1),
+        try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
+    );
+
+    owner.release(io, held);
+
+    try testing.expectEqual(
+        @as(i64, 0),
+        try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
+    );
+}

src/upstream/transport.zig

@@ -190,6 +190,11 @@ pub const LocalResource = error{
 
 pub const Cancellation = error{Canceled};
 
+/// What a leaf client may fail with. A leaf never *errors* on a peer fault: it
+/// returns one as an `Outcome.fault` value, because the pool needs the concrete
+/// cause and an error cannot carry one.
+pub const LeafError = LocalResource || Cancellation;
+
 /// The caller's own time ran out before any peer could be given the observation
 /// interval it was configured to get. Evidence about this process's budget, not
 /// about any endpoint, so it is never recorded against health — that is the
@@ -238,7 +243,7 @@ pub fn group(err: ExchangeError) Group {
 /// This is the only place a foreign error set is folded in. Everywhere else
 /// the call site names the peer fault it means, because the call site is what
 /// knows whether it was connecting, sending or receiving.
-pub fn mapLocal(err: anyerror) ?ExchangeError {
+pub fn mapLocal(err: anyerror) ?LeafError {
     return switch (err) {
         error.OutOfMemory => error.OutOfMemory,
         error.SystemResources => error.SystemResources,
@@ -258,6 +263,20 @@ pub fn mapPhase(err: anyerror, phase: PeerFault) ExchangeError {
     return mapLocal(err) orelse phase;
 }
 
+/// `mapLocal` for the connect phase, where `error.Unexpected` belongs to the
+/// peer.
+///
+/// zig 0.16.0's `connect()` names no `EINVAL` (std/Io/Threaded.zig:11909), and
+/// `EINVAL` is what a blackhole route to the peer returns, so a route fault
+/// arrives as `error.Unexpected`. Claimed as local, it would end the exchange
+/// without failover and record nothing against the endpoint. In this phase an
+/// errno std does not name is about the path to this peer, so it stays a peer
+/// fault and the pool moves on to the next endpoint.
+pub fn mapLocalConnect(err: anyerror) ?LeafError {
+    if (err == error.Unexpected) return null;
+    return mapLocal(err);
+}
+
 /// Closes `target` with cancellation blocked.
 ///
 /// A transport's close runs from a `defer` chain that a lost timeout race is
@@ -279,21 +298,37 @@ pub fn closeBlocked(io: std.Io, target: anytype) void {
     }
 }
 
-/// The payload of `f`'s return type, which the race harness requires to be
+/// The error union `f` returns, which the race harness requires it to have.
-/// `ExchangeError!T`. A raced function with any other error set would let a
+fn racedUnion(comptime f: anytype) std.builtin.Type.ErrorUnion {
-/// failure reach the pool without passing through `group`.
-fn RacedPayload(comptime f: anytype) type {
     const info = @typeInfo(@TypeOf(f));
     if (info != .@"fn") @compileError("the race harness needs a function, found " ++ @typeName(@TypeOf(f)));
     const Return = info.@"fn".return_type orelse
         @compileError("the race harness needs a function with a concrete return type");
-    const union_info = switch (@typeInfo(Return)) {
+    return switch (@typeInfo(Return)) {
         .error_union => |u| u,
-        else => @compileError("the race harness needs `ExchangeError!T`, found " ++ @typeName(Return)),
+        else => @compileError("the race harness needs `E!T`, found " ++ @typeName(Return)),
     };
-    if (union_info.error_set != ExchangeError)
+}
-        @compileError("the race harness needs `ExchangeError!T`, found " ++ @typeName(Return));
+
-    return union_info.payload;
+/// The payload of `f`'s return type.
+fn RacedPayload(comptime f: anytype) type {
+    return racedUnion(f).payload;
+}
+
+/// `f`'s own error set.
+fn RacedError(comptime f: anytype) type {
+    return racedUnion(f).error_set;
+}
+
+/// What racing `f` can fail with: `f`'s own errors, plus the three the harness
+/// itself produces — the expiry, a backend that cannot start a second task, and
+/// the whole task being torn down.
+///
+/// Derived rather than fixed at `ExchangeError`, so a caller's narrow error set
+/// survives the race. That is what lets the pool prove at the type level that a
+/// leaf cannot hand it a `PeerFault`, instead of asserting it.
+pub fn RaceError(comptime f: anytype) type {
+    return RacedError(f) || error{ Timeout, SystemResources, Canceled };
 }
 
 /// Runs `f(args...)` raced against `budget`, and cancels the loser.
@@ -312,7 +347,7 @@ pub fn raceWithin(
     budget: std.Io.Clock.Duration,
     comptime f: anytype,
     args: anytype,
-) ExchangeError!RacedPayload(f) {
+) RaceError(f)!RacedPayload(f) {
     var outcome: RaceOutcome = .completed;
     return raceUntilTagged(io, .fromNow(io, budget), &outcome, f, args);
 }
@@ -344,9 +379,9 @@ pub fn raceUntilTagged(
     outcome: *RaceOutcome,
     comptime f: anytype,
     args: anytype,
-) ExchangeError!RacedPayload(f) {
+) RaceError(f)!RacedPayload(f) {
     const Slot = union(enum) {
-        raced: ExchangeError!RacedPayload(f),
+        raced: RacedError(f)!RacedPayload(f),
         expiry: std.Io.Cancelable!void,
     };
 
@@ -378,8 +413,12 @@ fn expire(io: std.Io, expiry_at: std.Io.Clock.Timestamp) std.Io.Cancelable!void
     return expiry_at.wait(io);
 }
 
-/// A thing that sends one DNS message and returns one validated DNS message.
+/// A thing that sends one DNS message and returns one validated DNS message,
-/// Implemented by DohClient, DotClient, Pool, and test fakes.
+/// with a peer fault already reduced to an error.
+///
+/// Implemented by `Pool`, the forward client, and the fakes that stand in for
+/// either. The leaf clients are on the other side of the pool and implement
+/// `Leaf` instead, which keeps the fault as a value.
 pub const Client = struct {
     ptr: *anyopaque,
     exchangeFn: *const fn (
@@ -394,7 +433,8 @@ pub const Client = struct {
     /// passed `validateResponse` against `query`.
     ///
     /// `selected` names the resolver the exchange used. A single-endpoint
-    /// implementation (DoH, DoT, the forward client, test fakes) may write it
+    /// implementation — the forward client and the test fakes; the DoH and DoT
+    /// clients are `Leaf`s and a `Pool` carries their answer here — may write it
     /// *before* each attempt: it has one resolver and records no health, so
     /// "the one I tried" is an honest answer even for a failure, and a SERVFAIL
     /// row without its resolver explains nothing.
@@ -419,6 +459,79 @@ pub const Client = struct {
     }
 };
 
+/// One peer fault as a value: the taxonomy `PeerFault` names, plus the concrete
+/// error that produced it.
+///
+/// The classification is what health and backoff count; the cause is what tells
+/// an operator which failure it was. `SendFailed` alone cannot separate a peer
+/// that reset the connection from one whose TLS record was rejected, and the
+/// leaf that unwrapped the cause is the only place that still holds it.
+///
+/// There is no phase field: the taxonomy already names the phase for every kind
+/// that has one, and `TlsFailed` cannot say where it failed.
+pub const Fault = struct {
+    kind: PeerFault,
+    cause: anyerror,
+    /// The status the peer answered with, when that is the fault. An error
+    /// name cannot carry it, and it is what says why the peer refused: a 505
+    /// is a DoH server that will not speak HTTP/1.1.
+    http_status: ?std.http.Status = null,
+
+    /// The one text every surface prints: `<Kind> (cause <Cause>)`, or
+    /// `<Kind> (status <code> <phrase>)` for a peer that answered with a status.
+    pub fn format(self: Fault, w: *std.Io.Writer) std.Io.Writer.Error!void {
+        const status = self.http_status orelse
+            return w.print("{t} (cause {s})", .{ self.kind, @errorName(self.cause) });
+        try w.print("{t} (status {d}", .{ self.kind, @intFromEnum(status) });
+        if (status.phrase()) |phrase| try w.print(" {s}", .{phrase});
+        try w.writeByte(')');
+    }
+};
+
+/// What one exchange against one endpoint produced.
+pub const Outcome = union(enum) {
+    /// A prefix of the caller's `response_buf`, already validated against the
+    /// query.
+    reply: []u8,
+    fault: Fault,
+};
+
+/// A client of exactly one endpoint: DoH, DoT, and the pool's test fakes.
+///
+/// Separate from `Client` because the two answer different questions. A `Leaf`
+/// reports what the peer did, faults included, and leaves every judgement to
+/// its caller. A `Client` is the resolver the handler asks for an answer, and a
+/// fault has already become an error by the time it is reached.
+///
+/// No `selected` out-parameter: the pool discards a leaf's own identity anyway,
+/// since the entry's endpoint is the pool's naming of the same resolver.
+pub const Leaf = struct {
+    ptr: *anyopaque,
+    exchangeFn: *const fn (
+        ptr: *anyopaque,
+        io: std.Io,
+        query: []const u8,
+        response_buf: []u8,
+    ) LeafError!Outcome,
+
+    pub fn exchange(
+        self: Leaf,
+        io: std.Io,
+        query: []const u8,
+        response_buf: []u8,
+    ) LeafError!Outcome {
+        return self.exchangeFn(self.ptr, io, query, response_buf);
+    }
+};
+
+/// The fault a call site's phase means, unless `err` is one this process owns.
+/// The leaf counterpart of `mapPhase`: same rule, but the peer case comes back
+/// as a value carrying the cause instead of as a bare error.
+pub fn faultOrLocal(err: anyerror, phase: PeerFault) LeafError!Fault {
+    if (mapLocal(err)) |local| return local;
+    return .{ .kind = phase, .cause = err };
+}
+
 /// The unwrap helpers below turn the single collapsed error `std.http.Client`
 /// reports into the concrete cause it stashed. Every HTTP caller in this tree
 /// uses them: the DoH client classifies by the unwrapped cause, the blocklist
@@ -686,6 +799,37 @@ test "mapPhase names the phase unless the error is this process's own" {
     );
 }
 
+test "mapLocalConnect leaves an unnamed connect errno to the peer" {
+    try testing.expectEqual(@as(?ExchangeError, null), mapLocalConnect(error.Unexpected));
+    try testing.expectEqual(@as(?ExchangeError, error.OutOfMemory), mapLocalConnect(error.OutOfMemory));
+    try testing.expectEqual(
+        @as(?ExchangeError, error.ProcessFdQuotaExceeded),
+        mapLocalConnect(error.ProcessFdQuotaExceeded),
+    );
+    try testing.expectEqual(@as(?ExchangeError, error.Canceled), mapLocalConnect(error.Canceled));
+    try testing.expectEqual(@as(?ExchangeError, null), mapLocalConnect(error.NetworkUnreachable));
+}
+
+test "a fault with a status prints the code and its phrase" {
+    var buf: [64]u8 = undefined;
+    const refused: Fault = .{
+        .kind = error.HttpStatus,
+        .cause = error.HttpStatus,
+        .http_status = .http_version_not_supported,
+    };
+    try testing.expectEqualStrings(
+        "HttpStatus (status 505 HTTP Version Not Supported)",
+        try std.fmt.bufPrint(&buf, "{f}", .{refused}),
+    );
+
+    // A code std has no name for still prints, without a phrase.
+    const unnamed: Fault = .{ .kind = error.HttpStatus, .cause = error.HttpStatus, .http_status = @enumFromInt(599) };
+    try testing.expectEqualStrings("HttpStatus (status 599)", try std.fmt.bufPrint(&buf, "{f}", .{unnamed}));
+
+    const plain: Fault = .{ .kind = error.ConnectFailed, .cause = error.Unexpected };
+    try testing.expectEqualStrings("ConnectFailed (cause Unexpected)", try std.fmt.bufPrint(&buf, "{f}", .{plain}));
+}
+
 /// Stands in for whatever the pool or the forward client races. The variants
 /// are the three ways such a task ends: in time, too late, or with a failure of
 /// its own.
@@ -743,14 +887,16 @@ test "raceWithin passes the raced task's own failure through" {
     );
 }
 
-test "raceUntilTagged tells a leaf Timeout apart from an expiry" {
+test "raceUntilTagged tells a raced Timeout apart from an expiry" {
     var threaded: std.Io.Threaded = .init(testing.allocator, .{});
     defer threaded.deinit();
     const io = threaded.io();
 
-    // The leaf's own timeout: it returned, so the peer really did time out and
+    // A raced function that returned `error.Timeout` of its own: it completed,
-    // the outcome is `completed` even though the error is the same one an
+    // so the tag says `completed` even though the error is the one an expiry
-    // expiry produces.
+    // produces. No leaf does this — a leaf's own timeout is a `Fault` — but the
+    // harness serves callers with any error set, and the tag is what separates
+    // the two for every one of them.
     var outcome: RaceOutcome = .expired;
     const far: std.Io.Clock.Timestamp = .fromNow(io, .{ .raw = .fromSeconds(30), .clock = .awake });
     try testing.expectError(

src/web/handlers/apply.zig

@@ -636,9 +636,16 @@ pub const Plan = struct {
                 );
             }
             // A generation no reader held at the swap has no release left to
-            // tear it down, so the publisher does — after the reconciliation,
+            // tear it down, so the publisher does — after the reconciliation
-            // which reads only the copies taken at prepare.
+            // above, which reads only the copies taken at prepare.
-            if (self.retired_upstream) |old| old.retire(io);
+            //
+            // `retireDisplaced`, not `retire`: the teardown of a displaced
+            // generation is also when its `upstream.exchange` episodes are
+            // reconciled, and this path and a reader's release are the two ways
+            // a generation reaches it. The `configuration.load` half above
+            // stays here, because that code is shared with the boot collector
+            // and its rule is scoped.
+            if (self.retired_upstream) |old| self.state.upstreams.?.retireDisplaced(io, old);
         }
 
         self.* = undefined;

src/web/metrics.zig

@@ -130,6 +130,20 @@ pub const DiagnosticsSample = struct {
     write_failures: u64,
 };
 
+/// The most of each kind of concurrent work there has been at once since start
+/// (specs/memory-budget.md item 0): what the admission budget, the listener
+/// caps and the query-log queue are sized from. Gauges, so none of them rides
+/// in a counter group. A null field is a collaborator the state does not have.
+pub const PeakSample = struct {
+    dns_in_flight: ?u64 = null,
+    querylog_queue: ?u64 = null,
+    /// The larger of the two address families' peaks. Each family has its own
+    /// slot pool, so a cap is sized per listener, never from their sum.
+    tcp_connections: ?u64 = null,
+    dot_connections: ?u64 = null,
+    doh_connections: ?u64 = null,
+};
+
 /// Everything one scrape reports. A null section is a collaborator the state
 /// does not have.
 pub const Sample = struct {
@@ -172,6 +186,10 @@ pub const Sample = struct {
     /// label and cannot live in `UpstreamSample`. Absent while no pool is
     /// wired, like every other collaborator.
     upstream_budget_exhausted_total: ?u64 = null,
+    /// Exchanges the pool ended on this process's own resources. Pool-wide for
+    /// the same reason as `upstream_budget_exhausted_total`.
+    upstream_local_resource_total: ?u64 = null,
+    peaks: PeakSample = .{},
 };
 
 pub fn handle(
@@ -193,6 +211,7 @@ pub fn collect(state: *server.WebState, io: std.Io, arena: Allocator) Allocator.
 
     if (state.handler) |handler| {
         sample.dns = dnsCounters(&handler.stats);
+        sample.peaks.dns_in_flight = handler.in_flight_peak.load(.monotonic);
 
         // The pointer is loaded under the same mutex a query loads it under,
         // never before it: a live `cache.size` or `dns.rate_limit` change
@@ -223,6 +242,7 @@ pub fn collect(state: *server.WebState, io: std.Io, arena: Allocator) Allocator.
             .rows_written = reading.rows_written,
             .batches_gated = reading.batches_gated,
         };
+        sample.peaks.querylog_queue = reading.queue_peak;
     }
 
     if (state.tracker) |tracker| sample.tracker = .{
@@ -270,9 +290,12 @@ pub fn collect(state: *server.WebState, io: std.Io, arena: Allocator) Allocator.
         };
     }
     if (state.dot_listener) |listener| sample.dot_listener = listener.snapshotStats();
+    if (state.doh_listener) |listener| sample.peaks.doh_connections = listener.connectionsPeak();
+    if (state.dot_listener) |listener| sample.peaks.dot_connections = listener.connectionsPeak();
 
     sample.udp_listener = sumListeners(udp_server.Snapshot, udp_server.UdpServer, state.udp_listeners);
     sample.tcp_listener = sumListeners(tcp_server.Snapshot, tcp_server.TcpServer, state.tcp_listeners);
+    sample.peaks.tcp_connections = largestConnectionsPeak(state.tcp_listeners);
 
     if (state.upstreams) |owner| {
         const generation = owner.acquire(io);
@@ -280,6 +303,7 @@ pub fn collect(state: *server.WebState, io: std.Io, arena: Allocator) Allocator.
         if (generation.pool) |pool| {
             sample.upstreams = try upstreams(pool, io, arena);
             sample.upstream_budget_exhausted_total = pool.budgetExhaustedTotal();
+            sample.upstream_local_resource_total = pool.localResourceTotal();
         }
     }
 
@@ -308,6 +332,13 @@ fn sumListeners(comptime Snapshot: type, comptime Server: type, listeners: []con
     return total;
 }
 
+/// Null for an empty slice, like `sumListeners`.
+fn largestConnectionsPeak(listeners: []const *tcp_server.TcpServer) ?u64 {
+    var largest: ?u64 = null;
+    for (listeners) |listener| largest = @max(largest orelse 0, listener.connectionsPeak());
+    return largest;
+}
+
 fn dnsCounters(stats: *const dns_handler.Handler.Stats) DnsCounters {
     var out: DnsCounters = undefined;
     inline for (dns_stat_fields, 0..) |field, i| {
@@ -471,6 +502,8 @@ pub fn render(w: *std.Io.Writer, sample: Sample) std.Io.Writer.Error!void {
         try counterGroup(w, "nxdns_dot_server_", "DoT listener counter", listener);
     }
 
+    try renderPeaks(w, sample.peaks);
+
     if (sample.doh_certs != null or sample.dot_certs != null) try renderCerts(w, sample);
 
     if (sample.upstreams.len != 0) try renderUpstreams(w, sample.upstreams);
@@ -483,6 +516,48 @@ pub fn render(w: *std.Io.Writer, sample: Sample) std.Io.Writer.Error!void {
         );
         try w.print("nxdns_upstream_budget_exhausted_total {d}\n", .{total});
     }
+    if (sample.upstream_local_resource_total) |total| {
+        try labeledHead(
+            w,
+            "nxdns_upstream_local_resource_total",
+            "Exchanges this process ended on its own resources (memory, descriptors, an unnamed system error) before any upstream answered.",
+            "counter",
+        );
+        try w.print("nxdns_upstream_local_resource_total {d}\n", .{total});
+    }
+}
+
+fn renderPeaks(w: *std.Io.Writer, peaks: PeakSample) std.Io.Writer.Error!void {
+    if (peaks.dns_in_flight) |value| try gauge(
+        w,
+        "nxdns_dns_in_flight_peak",
+        "Most DNS queries handled at once since start, across every listener.",
+        value,
+    );
+    if (peaks.querylog_queue) |value| try gauge(
+        w,
+        "nxdns_querylog_queue_peak",
+        "Upper bound on the most entries the query log queue has held at once since start.",
+        value,
+    );
+    if (peaks.tcp_connections) |value| try gauge(
+        w,
+        "nxdns_tcp_server_connections_peak",
+        "Most connections one TCP/53 listener has held at once since start.",
+        value,
+    );
+    if (peaks.dot_connections) |value| try gauge(
+        w,
+        "nxdns_dot_server_connections_peak",
+        "Most connections the DoT listener has held at once since start.",
+        value,
+    );
+    if (peaks.doh_connections) |value| try gauge(
+        w,
+        "nxdns_doh_server_connections_peak",
+        "Most connections the DoH listener has held at once since start.",
+        value,
+    );
 }
 
 fn renderCerts(w: *std.Io.Writer, sample: Sample) std.Io.Writer.Error!void {
@@ -979,6 +1054,35 @@ test "an unwired collaborator omits its family rather than reporting zeros" {
     try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "nxdns_dot_server_"));
     try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "nxdns_udp_server_"));
     try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "nxdns_tcp_server_"));
+    try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "_peak"));
+}
+
+test "each peak is a gauge, and an unwired one is absent" {
+    const text = try renderToString(testing.allocator, .{ .peaks = .{
+        .dns_in_flight = 7,
+        .querylog_queue = 120,
+        .tcp_connections = 3,
+    } });
+    defer testing.allocator.free(text);
+
+    try testing.expect(std.mem.containsAtLeast(u8, text, 1, "# TYPE nxdns_dns_in_flight_peak gauge\nnxdns_dns_in_flight_peak 7\n"));
+    try testing.expect(std.mem.containsAtLeast(u8, text, 1, "# TYPE nxdns_querylog_queue_peak gauge\nnxdns_querylog_queue_peak 120\n"));
+    try testing.expect(std.mem.containsAtLeast(u8, text, 1, "# TYPE nxdns_tcp_server_connections_peak gauge\nnxdns_tcp_server_connections_peak 3\n"));
+    try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "nxdns_dot_server_connections_peak"));
+    try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "nxdns_doh_server_connections_peak"));
+}
+
+test "the TCP peak is the larger listener's, never the sum" {
+    var tcp6: tcp_server.TcpServer = undefined;
+    tcp6.core.stats = .{};
+    tcp6.core.stats.connections_peak.store(4, .monotonic);
+
+    var tcp4: tcp_server.TcpServer = undefined;
+    tcp4.core.stats = .{};
+    tcp4.core.stats.connections_peak.store(6, .monotonic);
+
+    try testing.expectEqual(@as(?u64, 6), largestConnectionsPeak(&.{ &tcp6, &tcp4 }));
+    try testing.expectEqual(@as(?u64, null), largestConnectionsPeak(&.{}));
 }
 
 test "the plain-DNS listener families carry every counter of both listeners" {
@@ -1520,18 +1624,16 @@ const AnsweringClient = struct {
         io: std.Io,
         query: []const u8,
         response_buf: []u8,
-        selected: *?[]const u8,
+    ) transport.LeafError!transport.Outcome {
-    ) transport.ExchangeError![]u8 {
         _ = io;
         _ = query;
         const self: *AnsweringClient = @ptrCast(@alignCast(ptr));
         _ = self.calls.fetchAdd(1, .acq_rel);
-        selected.* = "fake://leaf";
         @memcpy(response_buf[0..reply.len], reply);
-        return response_buf[0..reply.len];
+        return .{ .reply = response_buf[0..reply.len] };
     }
 
-    fn client(self: *AnsweringClient) transport.Client {
+    fn leaf(self: *AnsweringClient) transport.Leaf {
         return .{ .ptr = self, .exchangeFn = exchangeFn };
     }
 };
@@ -1546,10 +1648,10 @@ test "the queue families carry what a real pool recorded, through the real snaps
     defer threaded.deinit();
     const io = threaded.io();
 
-    var leaf: AnsweringClient = .{};
+    var answering: AnsweringClient = .{};
     var slots = [_]pool_mod.Slot{
-        .{ .client = leaf.client() },
+        .{ .client = answering.leaf() },
-        .{ .client = leaf.client() },
+        .{ .client = answering.leaf() },
     };
     var recoveries: std.atomic.Value(u64) = .init(0);
     var entries = [_]pool_mod.Entry{.{
@@ -1569,7 +1671,7 @@ test "the queue families carry what a real pool recorded, through the real snaps
     var buf: [512]u8 = undefined;
     var selected: ?[]const u8 = null;
     _ = try pool.exchange(io, "\x12\x34\x01\x00", &buf, &selected);
-    try testing.expectEqual(@as(u32, 1), leaf.calls.load(.acquire));
+    try testing.expectEqual(@as(u32, 1), answering.calls.load(.acquire));
 
     // The counters a burst would move, written straight into the entry: this
     // test is about what the snapshot path carries, not about reproducing a
@@ -1699,3 +1801,20 @@ test "the budget-exhausted counter renders pool-wide, without a url label" {
     defer testing.allocator.free(absent);
     try testing.expect(!std.mem.containsAtLeast(u8, absent, 1, "nxdns_upstream_budget_exhausted_total"));
 }
+
+test "the local-resource counter renders pool-wide, without a url label" {
+    const text = try renderToString(testing.allocator, .{ .upstream_local_resource_total = 4 });
+    defer testing.allocator.free(text);
+
+    try testing.expect(std.mem.containsAtLeast(
+        u8,
+        text,
+        1,
+        "# TYPE nxdns_upstream_local_resource_total counter\n",
+    ));
+    try testing.expect(std.mem.containsAtLeast(u8, text, 1, "nxdns_upstream_local_resource_total 4\n"));
+
+    const absent = try renderToString(testing.allocator, .{});
+    defer testing.allocator.free(absent);
+    try testing.expect(!std.mem.containsAtLeast(u8, absent, 1, "nxdns_upstream_local_resource_total"));
+}

tools/cut.zig

@@ -447,6 +447,45 @@ fn rewriteZonVersion(arena: Allocator, source: []const u8, version: []const u8)
     }, 0);
 }
 
+/// The compose file a user fetches from a release tag. Its image default names
+/// that release, so `docker compose up -d` from the tag runs the tag's image and
+/// `NXDNS_VERSION` stays an override, never a requirement.
+const compose_path = "deploy/docker/compose.yaml";
+
+const compose_image_prefix = "image: ${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-";
+
+/// The byte range of `<version>` in the image line
+/// `image: ${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-<version>}}`.
+/// Only a line that starts with that text, after its indentation, is the image:
+/// the same text in a comment or after another key pins nothing. The file must
+/// hold exactly one such line, or the rewrite would have to guess.
+fn findComposeVersion(source: []const u8) error{ NoComposeVersion, ManyComposeVersions }!VersionSpan {
+    var found: ?VersionSpan = null;
+    var offset: usize = 0;
+    var lines = std.mem.splitScalar(u8, source, '\n');
+    while (lines.next()) |line| {
+        const line_start = offset;
+        offset += line.len + 1;
+
+        const indent = line.len - std.mem.trimStart(u8, line, " ").len;
+        if (!std.mem.startsWith(u8, line[indent..], compose_image_prefix)) continue;
+        if (found != null) return error.ManyComposeVersions;
+
+        const start = indent + compose_image_prefix.len;
+        const rest = std.mem.trimEnd(u8, line[start..], " \r");
+        if (rest.len <= "}}".len or !std.mem.endsWith(u8, rest, "}}")) return error.NoComposeVersion;
+        const version = rest[0 .. rest.len - "}}".len];
+        if (std.mem.indexOfAny(u8, version, "{}$ ") != null) return error.NoComposeVersion;
+        found = .{ .start = line_start + start, .end = line_start + start + version.len };
+    }
+    return found orelse error.NoComposeVersion;
+}
+
+fn rewriteComposeVersion(arena: Allocator, source: []const u8, version: []const u8) ![]u8 {
+    const at = try findComposeVersion(source);
+    return std.mem.concat(arena, u8, &.{ source[0..at.start], version, source[at.end..] });
+}
+
 /// The primary certificate fingerprint of a `git verify-tag --raw` VALIDSIG
 /// line: its LAST field. The line is
 ///
@@ -2243,9 +2282,9 @@ fn cut(ctx: *Ctx, kind_text: []const u8) !void {
     const token = checked.authorization orelse return CheckFailed;
 
     // The one commit this cut makes is assembled in three moves, in this order
-    // and no other: the manifest declares the version, the pin stage builds that
+    // and no other: the manifest and the compose file declare the version, the
-    // version and writes its hashes into `flake.nix`, and the commit takes both
+    // pin stage builds that version and writes its hashes into `flake.nix`, and
-    // files. `verify-dist` refuses a build whose `-Dversion-string` disagrees
+    // the commit takes all three files. `verify-dist` refuses a build whose `-Dversion-string` disagrees
     // with the manifest, so the pins cannot be computed before the bump; and the
     // pins belong to the commit, so the commit cannot be made before the pins.
     // Both of these run before the manifest is rewritten: they can refuse, and
@@ -2258,7 +2297,11 @@ fn cut(ctx: *Ctx, kind_text: []const u8) !void {
     };
     defer environ.deinit();
 
-    if (bump_needed) try writeZonVersion(ctx, version, zon_source);
+    if (bump_needed) {
+        const compose = try composeWithVersion(ctx, version);
+        try writeZonVersion(ctx, version, zon_source);
+        try writeComposeVersion(ctx, version, compose);
+    }
     try pinStage(ctx, version, &environ, zig_exe);
     try commitBump(ctx, version, bump_needed);
 
@@ -2942,11 +2985,50 @@ fn verifyOriginTag(ctx: *Ctx, tag: []const u8, tag_ref: []const u8, commit: []co
     ctx.pass("origin-tag", "{s} ({s}) on origin is this tool's own signed tag at {s}", .{ tag, object, commit });
 }
 
-/// Rewrites `build.zig.zon` and commits it, and nothing else.
+/// The files the bump commit may touch, and the only ones. `flake.nix` joins
-/// The two files the bump commit may touch, and the only two. `flake.nix` joins
 /// `build.zig.zon` because the pin stage writes the release hashes into it
-/// between the manifest write and the commit.
+/// between the manifest write and the commit; the compose file joins because
-const bump_paths = [_][]const u8{ "build.zig.zon", "flake.nix" };
+/// its image default is the release a user of the tag gets.
+const bump_paths = [_][]const u8{ "build.zig.zon", "flake.nix", compose_path };
+
+/// Replaces `path` with `bytes` atomically: a cut interrupted mid-write must not
+/// leave a half-written file that neither zig, compose nor a re-run can parse.
+fn replaceFile(ctx: *Ctx, path: []const u8, bytes: []const u8) !void {
+    var atomic = Io.Dir.cwd().createFileAtomic(ctx.io, path, .{ .replace = true }) catch |err| {
+        ctx.soft("bump", "cannot open {s} for an atomic replace: {t}", .{ path, err });
+        return CheckFailed;
+    };
+    defer atomic.deinit(ctx.io);
+    atomic.file.writeStreamingAll(ctx.io, bytes) catch |err| {
+        ctx.soft("bump", "cannot write {s}: {t}", .{ path, err });
+        return CheckFailed;
+    };
+    atomic.file.sync(ctx.io) catch |err| {
+        ctx.soft("bump", "cannot flush {s} to disk: {t}", .{ path, err });
+        return CheckFailed;
+    };
+    atomic.replace(ctx.io) catch |err| {
+        ctx.soft("bump", "cannot replace {s}: {t}", .{ path, err });
+        return CheckFailed;
+    };
+}
+
+/// Read back from disk rather than trusting the buffer: this is the file the
+/// rest of the release reads, and a rewrite that produced something its grammar
+/// rejects must be found now.
+fn readBack(ctx: *Ctx, path: []const u8) ![:0]u8 {
+    return Io.Dir.cwd().readFileAllocOptions(
+        ctx.io,
+        path,
+        ctx.arena,
+        .limited(max_input_bytes),
+        .of(u8),
+        0,
+    ) catch |err| {
+        ctx.soft("bump", "cannot read {s} back: {t}", .{ path, err });
+        return CheckFailed;
+    };
+}
 
 /// Writes the new version into `build.zig.zon` and proves the file still parses.
 ///
@@ -2959,41 +3041,9 @@ fn writeZonVersion(ctx: *Ctx, version: []const u8, source: [:0]const u8) !void {
         ctx.soft("bump", "cannot rewrite the .version field of build.zig.zon: {t}", .{err});
         return CheckFailed;
     };
+    try replaceFile(ctx, "build.zig.zon", rewritten);
 
-    // Atomic: a cut interrupted mid-write must not leave a half-written
+    const written = try readBack(ctx, "build.zig.zon");
-    // manifest that neither zig nor a re-run can parse.
-    var atomic = Io.Dir.cwd().createFileAtomic(ctx.io, "build.zig.zon", .{ .replace = true }) catch |err| {
-        ctx.soft("bump", "cannot open build.zig.zon for an atomic replace: {t}", .{err});
-        return CheckFailed;
-    };
-    defer atomic.deinit(ctx.io);
-    atomic.file.writeStreamingAll(ctx.io, rewritten) catch |err| {
-        ctx.soft("bump", "cannot write build.zig.zon: {t}", .{err});
-        return CheckFailed;
-    };
-    atomic.file.sync(ctx.io) catch |err| {
-        ctx.soft("bump", "cannot flush build.zig.zon to disk: {t}", .{err});
-        return CheckFailed;
-    };
-    atomic.replace(ctx.io) catch |err| {
-        ctx.soft("bump", "cannot replace build.zig.zon: {t}", .{err});
-        return CheckFailed;
-    };
-
-    // Read back from disk rather than trusting the buffer: this is the file the
-    // rest of the release reads, and a rewrite that produced something the zon
-    // grammar rejects must be found now.
-    const written = Io.Dir.cwd().readFileAllocOptions(
-        ctx.io,
-        "build.zig.zon",
-        ctx.arena,
-        .limited(max_input_bytes),
-        .of(u8),
-        0,
-    ) catch |err| {
-        ctx.soft("bump", "cannot read build.zig.zon back: {t}", .{err});
-        return CheckFailed;
-    };
     const reparsed = parseZonVersion(ctx.arena, written) catch |err| {
         ctx.soft("bump", "the rewritten build.zig.zon does not parse: {t}", .{err});
         return CheckFailed;
@@ -3005,8 +3055,39 @@ fn writeZonVersion(ctx: *Ctx, version: []const u8, source: [:0]const u8) !void {
     ctx.pass("bump", "build.zig.zon declares {s}", .{version});
 }
 
-/// Commits the manifest bump and the pins as one commit, after asserting the
+/// The compose file with its image default moved to `version`, computed before
-/// working tree holds those two files and nothing else.
+/// anything is written: a compose file the rewrite cannot read is a refusal
+/// while the tree is still clean.
+fn composeWithVersion(ctx: *Ctx, version: []const u8) ![]u8 {
+    const source = Io.Dir.cwd().readFileAlloc(ctx.io, compose_path, ctx.arena, .limited(max_input_bytes)) catch |err| {
+        ctx.soft("bump", "cannot read {s}: {t}", .{ compose_path, err });
+        return CheckFailed;
+    };
+    return rewriteComposeVersion(ctx.arena, source, version) catch |err| {
+        ctx.soft("bump", "cannot rewrite the image default of {s}: {t}", .{ compose_path, err });
+        return CheckFailed;
+    };
+}
+
+fn writeComposeVersion(ctx: *Ctx, version: []const u8, rewritten: []const u8) !void {
+    try replaceFile(ctx, compose_path, rewritten);
+
+    const written = try readBack(ctx, compose_path);
+    const at = findComposeVersion(written) catch |err| {
+        ctx.soft("bump", "the rewritten {s} has no single image default: {t}", .{ compose_path, err });
+        return CheckFailed;
+    };
+    if (!std.mem.eql(u8, written[at.start..at.end], version)) {
+        ctx.soft("bump", "the rewritten {s} defaults to '{s}', expected '{s}'", .{
+            compose_path, written[at.start..at.end], version,
+        });
+        return CheckFailed;
+    }
+    ctx.pass("bump", "{s} defaults to {s}", .{ compose_path, version });
+}
+
+/// Commits the manifest bump, the compose default and the pins as one commit,
+/// after asserting the working tree holds those files and nothing else.
 ///
 /// `bump_needed` is false on a resumed cut: the commit already exists, so the
 /// pin stage has just recomputed pins that are already in it. An empty diff is
@@ -3036,8 +3117,8 @@ fn commitBump(ctx: *Ctx, version: []const u8, bump_needed: bool) !void {
         for (bump_paths) |allowed| {
             if (std.mem.eql(u8, path, allowed)) break;
         } else {
-            ctx.soft("bump", "the cut changed '{s}'; the bump commit is {s} and {s} and nothing else", .{
+            ctx.soft("bump", "the cut changed '{s}'; the bump commit is {s}, {s} and {s} and nothing else", .{
-                path, bump_paths[0], bump_paths[1],
+                path, bump_paths[0], bump_paths[1], bump_paths[2],
             });
             return CheckFailed;
         }
@@ -3067,9 +3148,10 @@ fn commitBump(ctx: *Ctx, version: []const u8, bump_needed: bool) !void {
         "-S",                                              "-m",
         ctx.fmt("build: bump version to {s}", .{version}), "--",
         bump_paths[0],                                     bump_paths[1],
+        bump_paths[2],
     });
-    ctx.pass("bump", "one commit: {s} declares {s} and {s} pins its release hashes", .{
+    ctx.pass("bump", "one commit: {s} declares {s}, {s} defaults to it and {s} pins its release hashes", .{
-        bump_paths[0], version, bump_paths[1],
+        bump_paths[0], version, bump_paths[2], bump_paths[1],
     });
 }
 
@@ -3698,6 +3780,56 @@ test "the zon version survives a rewrite round trip" {
     try testing.expectError(error.NoVersionField, findVersionValue("// .version = \"9.9.9\""));
 }
 
+test "the compose image default survives a rewrite round trip" {
+    var arena_state = std.heap.ArenaAllocator.init(testing.allocator);
+    defer arena_state.deinit();
+    const arena = arena_state.allocator();
+
+    const source =
+        \\services:
+        \\  nxdns:
+        \\    # NXDNS_VERSION=0.0.1 docker compose up -d runs another release.
+        \\    image: ${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-0.0.7}}
+        \\    restart: unless-stopped
+    ;
+    const at = try findComposeVersion(source);
+    try testing.expectEqualStrings("0.0.7", source[at.start..at.end]);
+
+    const rewritten = try rewriteComposeVersion(arena, source, "0.0.8");
+    const moved = try findComposeVersion(rewritten);
+    try testing.expectEqualStrings("0.0.8", rewritten[moved.start..moved.end]);
+    // Only the default moved: the comment naming another version and the
+    // outer `NXDNS_IMAGE` default are untouched.
+    try testing.expect(std.mem.indexOf(u8, rewritten, "# NXDNS_VERSION=0.0.1 docker") != null);
+    try testing.expect(std.mem.indexOf(u8, rewritten, "${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:") != null);
+    try testing.expectEqual(source.len, rewritten.len);
+
+    const image = "    image: ${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:${NXDNS_VERSION:-";
+    try testing.expectError(error.NoComposeVersion, findComposeVersion("    image: nxdns:latest"));
+    try testing.expectError(error.NoComposeVersion, findComposeVersion(image ++ "}}"));
+    try testing.expectError(error.NoComposeVersion, findComposeVersion(image ++ "0.0.7}"));
+    try testing.expectError(error.NoComposeVersion, findComposeVersion(image ++ "${OTHER}}}"));
+    // The text in a comment, or after any other key, is not the image.
+    try testing.expectError(error.NoComposeVersion, findComposeVersion("    # " ++ image[4..] ++ "0.0.7}}"));
+    try testing.expectError(error.NoComposeVersion, findComposeVersion("    x-image: ${NXDNS_VERSION:-0.0.7}"));
+    try testing.expectError(error.ManyComposeVersions, findComposeVersion(image ++ "0.0.7}}\n" ++ image ++ "0.0.7}}"));
+}
+
+test "this repository's compose file defaults to the version build.zig.zon declares" {
+    // The bump commit moves both together; this catches a hand edit of either
+    // between two cuts, on every CI run rather than at the next release.
+    var threaded: std.Io.Threaded = .init(testing.allocator, .{});
+    defer threaded.deinit();
+
+    var arena_state = std.heap.ArenaAllocator.init(testing.allocator);
+    defer arena_state.deinit();
+    const arena = arena_state.allocator();
+    const zon = try Io.Dir.cwd().readFileAllocOptions(threaded.io(), "build.zig.zon", arena, .limited(max_input_bytes), .of(u8), 0);
+    const compose = try Io.Dir.cwd().readFileAlloc(threaded.io(), compose_path, arena, .limited(max_input_bytes));
+    const at = try findComposeVersion(compose);
+    try testing.expectEqualStrings(try parseZonVersion(arena, zon), compose[at.start..at.end]);
+}
+
 test "the changelog section must exist, be dated and say something" {
     const good =
         \\# Changelog

website/PLAN.md

@@ -1,6 +1,6 @@
 # nxdns.sh — marketing and docs website plan
 
-Agreed in an interview on 2026-08-15. This directory will hold the site's source.
+Agreed in an interview on 2026-08-15, revised on 2026-10-02, 2026-10-19 (screenshots in, section order set, the personal story moved to the owner's blog) and 2026-10-20 (install tabs, phone and light/dark screenshots). This directory will hold the site's source.
 
 ## Purpose
 
@@ -19,13 +19,14 @@ Convince, then teach. A marketing front page sells the project; the docs sit one
 
 ## Landing page
 
-Three sections:
+Four sections, in this order:
 
-1. Hero: one-line pitch and a copyable install command. Releases 0.0.1–0.0.3 exist, so the command is real.
+1. Hero: the README pitch — "A DNS sinkhole for your LAN. It blocks the names you do not want, and forwards the rest over an encrypted connection." — with the README's flow diagram animated. The install commands sit in their own section near the end, in three tabs: docker, tarball, nix. Each tab is a complete path or names the guide that finishes it; the docker tab writes the smallest config that starts, so nobody faces an empty file. Commands use no shell variables and plain `curl -O`. The registry allows anonymous pulls (verified 2026-10-02 against v0.0.21), so the Docker tab works without an account. The site goes live only with a release that contains aa8c9e2 (the compose file pins its own version, so the tab ends in a plain `docker compose up -d`) and f351469 (the `services.nxdns` NixOS module the nix tab uses): 0.0.23 or later.
-2. Feature grid: filtering, config-as-code, DoH/DoT, per-client groups, metrics, query log.
+2. Screenshots of the web UI, taken from a running instance with realistic data: Overview, Activity, Query detail, Clients, Diagnostics. Each comes in the admin UI's light and dark theme, behind a light/dark switch with light first. Phones get their own set, captured at a 390px viewport where the admin UI switches to its drawer layout, served through `<picture>`, so a phone shows readable screens instead of a shrunken desktop. All twenty images come from one session on the release the site launches with; the current set is 0.0.21 desktop-only.
-3. Comparison table vs Pi-hole, AdGuard Home, Technitium — short, only to help people choose.
+3. Feature grid: filtering, config-as-code, DoH/DoT, per-client groups, metrics, query log.
+4. Comparison table vs Pi-hole, AdGuard Home, Technitium — only to help people choose. Seven rows: how it ships; config as a file in git that the tool converges onto; encrypted upstreams built in; serves DoH/DoT to clients; signed, reproducible releases; DHCP server; DNSSEC validation. The last two are rows nxdns loses, kept because a table that only lists wins does not help anyone choose. Competitor cells were checked against Pi-hole 6.5 (FTL 6.8), AdGuard Home 0.107.79 and Technitium 15.6.2 on 2026-10-19.
 
-The pitch leads with what nxdns does, not with attacks on the incumbents. No screenshots.
+The pitch leads with what nxdns does, not with attacks on the incumbents. The page is about the tool: why the owner built it is a blog article, not a landing-page section.
 
 ## Docs section
 
@@ -35,10 +36,16 @@ The pitch leads with what nxdns does, not with attacks on the incumbents. No scr
 
 ## Look
 
-- Clean technical. Light default with a dark toggle.
+Approved on 2026-10-20 from a local mock (concept 2), after one rejected direction that copied Wispr Flow's landing page too closely. The mock is the reference for the build.
-- Orange/amber accent.
+
-- Brand is the lowercase `nxdns` wordmark in a distinctive typeface. No drawn logo.
+- Light only. No dark mode, no toggle. Warm paper, ink and amber: paper `#f5f2ea`, card `#fbf9f4`, ink `#1c1b18`, muted `#5f5b52`, amber `#f2a93b`, amber text `#a14d0a`. Every text pair passes 4.5:1. Two sections (screenshots, install) sit on rounded ink slabs.
-- Copy tone: personal / indie, first person — why I built it and what it fixes — in the honest voice the docs already use.
+- Type: Fraunces for headings and the wordmark, Figtree for body text, JetBrains Mono for labels and code. All three self-hosted and subset; no request leaves for Google Fonts.
+- Brand is the lowercase `nxdns` wordmark in Fraunces with an amber `x`. No drawn logo.
+- Section labels are dig-style comments: `;; ANSWER SECTION`, `;; VERIFY`, `;; FEATURES`. The footer ends with a zone-file line: `nxdns.sh. 3600 IN TXT "…"`.
+- The hero is the README's flow diagram, animated: one query at a time travels from the devices to nxdns, a tag shows the decision, and the packet goes on to the upstream or the sinkhole and back. Slow enough to read. A still frame for `prefers-reduced-motion`.
+- No illustrations, unless an image model makes them.
+- Design fundamentals from the emil-design-engineering rules, not from any one site's look: no layout shift (tab panels share one grid cell), 44px tap targets, hover only under `(hover: hover)`, copy buttons on code, grey/black/white focus rings.
+- Copy tone: the honest, direct voice the docs already use, about the tool. The personal story is the blog's. No personal email address on the site.
 
 ## Links
 
@@ -46,6 +53,4 @@ Source links point to `git.mial.net/mokhtar/nxdns` only. No GitHub mirror.
 
 ## Open questions
 
-1. Wordmark typeface — resolve in the design pass.
+1. Re-check the comparison cells against the then-current competitor versions before publication. AdGuard Home had a 1.0 beta on 2026-09-30, so its cells are the most likely to move.
-2. Does the Gitea container registry allow anonymous pulls? That decides whether the hero shows `docker run` or the binary install from INSTALL.md.
-3. Fact-check the comparison table against current competitor versions before publication.

website/demo/.gitignore

@@ -0,0 +1,5 @@
+node_modules/
+__pycache__/
+shots/
+screenshots.html
+.cookies

website/demo/README.md

@@ -0,0 +1,81 @@
+# Screenshot demo
+
+A synthetic household that produces the admin screenshots in `../src/assets/screenshots/`. Six devices on a docker LAN send real DNS traffic through the published nxdns image, real public blocklists decide what is blocked, and the bulk of the traffic is then spread over the previous 24 hours so the Overview shows a believable day. Nothing here is real household data, and no device carries a person's name.
+
+| Device | Address | Group |
+| --- | --- | --- |
+| nxdns | 192.168.50.2 | — |
+| Living room TV | 192.168.50.41 | default |
+| iPhone | 192.168.50.52 | default |
+| Pixel | 192.168.50.53 | default |
+| Kids' phone | 192.168.50.54 | kids (safe search, gambling and NSFW lists) |
+| Desktop PC | 192.168.50.20 | default |
+| MacBook Air | 192.168.50.21 | default |
+
+The LAN is 192.168.50.0/24 because the machine that first ran this uses 192.168.1.0/24 for its real network. The admin UI is on <http://127.0.0.1:18080>, password `nxdns-demo`. Upstreams are Cloudflare DoH with Quad9 DoT as failover; Quad9's DoH endpoint requires HTTP/2, which nxdns does not speak.
+
+Run every command from this directory. It needs docker, Node.js and Python 3.
+
+## Generate the household
+
+`NXDNS_IMAGE` picks the image; it defaults to the version the committed screenshots show. Set it to the release the new screenshots are for.
+
+```sh
+NXDNS_IMAGE=git.mial.net/mokhtar/nxdns:<version> ./start.sh
+```
+
+Wait until `/api/blocklists` reports all four lists loaded. Then shrink the cache for the compressed bulk run, so the cache-hit share stays near what a real day produces, and send the bulk traffic:
+
+```sh
+curl -s -c .cookies -H 'content-type: application/json' -d '{"password":"nxdns-demo"}' http://127.0.0.1:18080/api/auth/login
+curl -s -b .cookies -X PUT -H 'content-type: application/json' -d '{"cache":{"size":48}}' http://127.0.0.1:18080/api/settings
+./devices.sh bulk
+```
+
+The bulk run sends about 18,000 queries. Wait until all six device containers have exited, then restore the cache and rewrite the history while nxdns is stopped. `check_coherence.py` runs the repository's projection oracle and must print no `FAIL`:
+
+```sh
+curl -s -b .cookies -X PUT -H 'content-type: application/json' -d '{"cache":{"size":10000}}' http://127.0.0.1:18080/api/settings
+docker stop nxdemo-nxdns
+./db.sh /gen/rewrite_history.py /data
+./db.sh /gen/check_coherence.py /data
+docker start nxdemo-nxdns
+./devices.sh live
+```
+
+`gen/check_profiles.py` prints each device profile's expected blocked share through `/api/lookup`. Use it when you change `gen/profiles.py`.
+
+## Produce a Diagnostics episode
+
+Cut the primary DoH upstream off inside nxdns's network namespace, wait about two minutes, and restore it. nxdns opens an "Upstream failing" episode and closes it on its own after the routes come back. Check that the addresses are still the ones `cloudflare-dns.com` resolves to.
+
+```sh
+docker run --rm --net container:nxdemo-nxdns --cap-add NET_ADMIN python:3-alpine ip route add blackhole 104.16.248.249/32
+docker run --rm --net container:nxdemo-nxdns --cap-add NET_ADMIN python:3-alpine ip route add blackhole 104.16.249.249/32
+docker run --rm --net container:nxdemo-nxdns --cap-add NET_ADMIN python:3-alpine ip route del blackhole 104.16.248.249/32
+docker run --rm --net container:nxdemo-nxdns --cap-add NET_ADMIN python:3-alpine ip route del blackhole 104.16.249.249/32
+```
+
+## Take the screenshots
+
+The first time, install the pinned Playwright and its browser:
+
+```sh
+npm ci
+npx playwright install chromium
+```
+
+Then shoot all five pages in light and dark at 1440×900, 2× scale, and build a contact sheet to review them:
+
+```sh
+node gen/shoot.mjs shots "$(./db.sh /gen/blocked_query_id.py /data)"
+python3 -I gen/contact_sheet.py shots screenshots.html
+```
+
+Review the sheet before anything replaces the files in `../src/assets/screenshots/`.
+
+## Tear down
+
+```sh
+./teardown.sh
+```

website/demo/db.sh

@@ -0,0 +1,8 @@
+#!/bin/sh
+# usage: db.sh /gen/<script>.py /data [args]
+# Runs a gen/ script against the nxdemo-data volume as nxdns's uid. TZ is Europe/Paris written as a
+# POSIX rule, because alpine ships no zoneinfo; rewrite_history.py shapes the day in local time.
+set -eu
+DEMO=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+exec docker run --rm -i --user 65532:65532 -e TZ=CET-1CEST,M3.5.0,M10.5.0/3 \
+  -v nxdemo-data:/data -v "$DEMO/gen:/gen:ro" python:3-alpine python -I "$@"

website/demo/devices.sh

@@ -0,0 +1,17 @@
+#!/bin/sh
+# usage: devices.sh bulk          one container per device sends its lookup budget and exits
+#        devices.sh live          one container per device trickles traffic forever (restart unless-stopped)
+set -eu
+DEMO=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+MODE=$1
+for spec in tv:192.168.50.41:20 iphone:192.168.50.52:25 pixel:192.168.50.53:30 kids:192.168.50.54:35 desktop:192.168.50.20:45 mac:192.168.50.21:40; do
+  name=${spec%%:*}; rest=${spec#*:}; ip=${rest%%:*}; mean=${rest#*:}
+  docker rm -f "nxdemo-$name" >/dev/null 2>&1 || true
+  if [ "$MODE" = bulk ]; then
+    docker run -d --name "nxdemo-$name" --network nxdemo-lan --ip "$ip" -v "$DEMO/gen:/gen:ro" \
+      python:3-alpine python -I /gen/traffic.py "$name" bulk
+  else
+    docker run -d --name "nxdemo-$name" --restart unless-stopped --network nxdemo-lan --ip "$ip" -v "$DEMO/gen:/gen:ro" \
+      python:3-alpine python -I /gen/traffic.py "$name" live "$mean"
+  fi
+done

website/demo/etc/config.zon

@@ -0,0 +1,52 @@
+.{
+    .dns = .{ .rate_limit = 100000, .rate_window_seconds = 60 },
+    .web = .{ .port = 8080, .password = "nxdns-demo", .api_rate_limit_per_min = 5000 },
+    .logging = .{ .query_log_flush_interval_s = 5 },
+
+    .groups = .{
+        .{ .name = "default" },
+        .{ .name = "kids", .safe_search = true },
+    },
+
+    .upstreams = .{
+        .{ .url = "https://cloudflare-dns.com/dns-query", .priority = 10 },
+        .{ .url = "tls://9.9.9.9:853", .priority = 20, .tls_name = "dns.quad9.net" },
+    },
+
+    .clients = .{
+        .{ .ip = "192.168.50.41", .name = "Living room TV" },
+        .{ .ip = "192.168.50.52", .name = "iPhone" },
+        .{ .ip = "192.168.50.53", .name = "Pixel" },
+        .{ .ip = "192.168.50.54", .name = "Kids' phone", .group = "kids" },
+        .{ .ip = "192.168.50.20", .name = "Desktop PC" },
+        .{ .ip = "192.168.50.21", .name = "MacBook Air" },
+    },
+
+    .blocklist_sources = .{
+        .{ .url = "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts", .name = "StevenBlack unified hosts" },
+        .{ .url = "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt", .name = "HaGeZi Multi PRO" },
+        .{ .url = "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/gambling.txt", .name = "HaGeZi Gambling" },
+        .{ .url = "https://nsfw-small.oisd.nl/domainswild2", .name = "OISD NSFW small" },
+    },
+    .group_sources = .{
+        .{ .group = "default", .source_url = "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" },
+        .{ .group = "default", .source_url = "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt" },
+        .{ .group = "kids", .source_url = "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" },
+        .{ .group = "kids", .source_url = "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/pro.txt" },
+        .{ .group = "kids", .source_url = "https://raw.githubusercontent.com/hagezi/dns-blocklists/main/adblock/gambling.txt" },
+        .{ .group = "kids", .source_url = "https://nsfw-small.oisd.nl/domainswild2" },
+    },
+
+    .rules = .{
+        .{ .group = "default", .pattern = "s.youtube.com", .kind = .exact, .action = .allow },
+        .{ .group = "default", .pattern = "*.samsungacr.com", .kind = .wildcard, .action = .block },
+        .{ .group = "kids", .pattern = "s.youtube.com", .kind = .exact, .action = .allow },
+        .{ .group = "kids", .pattern = "*.tiktokv.com", .kind = .wildcard, .action = .block },
+    },
+
+    .local_records = .{
+        .{ .name = "nas.lan", .rtype = .a, .value = "192.168.50.5" },
+        .{ .name = "printer.lan", .rtype = .a, .value = "192.168.50.8" },
+        .{ .name = "files.lan", .rtype = .cname, .value = "nas.lan" },
+    },
+}

website/demo/gen/blocked_query_id.py

@@ -0,0 +1,13 @@
+"""Print the id of the newest blocked app-measurement.com lookup from the iPhone: the query the
+detail screenshot shows. usage: blocked_query_id.py DATA_DIR"""
+import sqlite3
+import sys
+
+row = sqlite3.connect(f"file:{sys.argv[1]}/querylog.db?mode=ro", uri=True).execute(
+    "SELECT q.id FROM query_log q JOIN domains d ON d.id = q.domain_id"
+    " WHERE q.client_ip = '192.168.50.52' AND q.policy_reason = 'blocklist_domain'"
+    " AND d.domain = 'app-measurement.com' ORDER BY q.id DESC LIMIT 1"
+).fetchone()
+if row is None:
+    sys.exit("no blocked app-measurement.com lookup from 192.168.50.52 in the log yet")
+print(row[0])

website/demo/gen/check_coherence.py

@@ -0,0 +1,62 @@
+"""The repository's projection-coherence oracle (`recompute_checks` and
+`expectProjectionsMatchRecompute` in src/storage/repositories/queries_repo.zig),
+plus file-level invariants. Exits 1 on any violation. usage: check_coherence.py DATA_DIR"""
+import sqlite3
+import sys
+
+DATA = sys.argv[1]
+BUCKET = "(CASE WHEN timestamp >= 0 THEN timestamp / 1800 * 1800 ELSE (timestamp - 1799) / 1800 * 1800 END)"
+CHECKS = [
+    ("SELECT bucket, queries, blocked, cached, rt_sum, rt_count FROM bucket_totals",
+     f"SELECT {BUCKET}, count(*), coalesce(sum(blocked <> 0), 0), coalesce(sum(cache_hit = 1), 0),"
+     " coalesce(sum(response_time_us), 0), count(response_time_us) FROM query_log GROUP BY 1"),
+    ("SELECT bucket, client_ip, queries FROM bucket_clients",
+     f"SELECT {BUCKET}, client_ip, count(*) FROM query_log GROUP BY 1, 2"),
+    ("SELECT bucket, qtype, count FROM bucket_types",
+     f"SELECT {BUCKET}, coalesce(qtype, -1), count(*) FROM query_log GROUP BY 1, 2"),
+    ("SELECT bucket, route_kind, source_present, source_text, count FROM bucket_routes",
+     f"SELECT {BUCKET}, route_kind, source IS NOT NULL, coalesce(source, ''), count(*)"
+     " FROM (SELECT timestamp, route_kind, CASE route_kind WHEN 'upstream' THEN upstream"
+     " WHEN 'forward_zone' THEN forward_zone END AS source FROM query_log) GROUP BY 1, 2, 3, 4"),
+]
+
+ql = sqlite3.connect(f"file:{DATA}/querylog.db?mode=ro", uri=True)
+failures = 0
+
+
+def expect(label, ok, detail=""):
+    global failures
+    print(("ok   " if ok else "FAIL ") + label + (f" ({detail})" if detail else ""))
+    failures += not ok
+
+
+for projection, recompute in CHECKS:
+    diff = ql.execute(
+        f"SELECT (SELECT count(*) FROM ({projection} EXCEPT {recompute}))"
+        f" + (SELECT count(*) FROM ({recompute} EXCEPT {projection}))"
+    ).fetchone()[0]
+    expect(f"projection == recompute: {projection.split('FROM ')[1]}", diff == 0, f"{diff} differing rows")
+
+integrity = ql.execute("PRAGMA integrity_check").fetchone()[0]
+expect("querylog integrity_check", integrity == "ok", integrity)
+expect("querylog user_version == 1", ql.execute("PRAGMA user_version").fetchone()[0] == 1)
+orphans = ql.execute("SELECT count(*) FROM query_log WHERE domain_id NOT IN (SELECT id FROM domains)").fetchone()[0]
+expect("every query_log.domain_id names a domain", orphans == 0, f"{orphans} orphans")
+created, since, first = ql.execute(
+    "SELECT created_at, available_since, (SELECT min(timestamp) FROM query_log) FROM querylog_meta").fetchone()
+expect("available_since = created_at + 1 <= first row", since == created + 1 and (first is None or since <= first),
+       f"created_at={created} available_since={since} first_row={first}")
+disorder = ql.execute(
+    "SELECT count(*) FROM query_log a JOIN query_log b ON b.id = a.id + 1 WHERE b.timestamp < a.timestamp").fetchone()[0]
+expect("query_log ids ascend with timestamps", disorder == 0, f"{disorder} inversions")
+
+cfg = sqlite3.connect(f"file:{DATA}/config.db?mode=ro", uri=True)
+cfg.execute(f"ATTACH 'file:{DATA}/querylog.db?mode=ro' AS ql")
+bad = cfg.execute("""SELECT count(*) FROM clients c WHERE EXISTS (SELECT 1 FROM ql.query_log q WHERE q.client_ip = c.ip)
+    AND (c.first_seen > (SELECT min(timestamp) FROM ql.query_log WHERE client_ip = c.ip)
+      OR c.last_seen < (SELECT max(timestamp) FROM ql.query_log WHERE client_ip = c.ip))""").fetchone()[0]
+expect("clients first_seen/last_seen bracket their logged rows", bad == 0, f"{bad} clients out of range")
+unknown = cfg.execute("SELECT count(DISTINCT client_ip) FROM ql.query_log WHERE client_ip NOT IN (SELECT ip FROM clients)").fetchone()[0]
+expect("every logged client is a named client row", unknown == 0, f"{unknown} unknown")
+expect("config integrity_check", cfg.execute("PRAGMA integrity_check").fetchone()[0] == "ok")
+sys.exit(1 if failures else 0)

website/demo/gen/check_profiles.py

@@ -0,0 +1,41 @@
+"""Print each profile's expected blocked share (by query weight) via /api/lookup,
+and every domain that does not resolve on the host."""
+import json
+import socket
+import sys
+import urllib.request
+
+sys.path.insert(0, sys.argv[1])
+from profiles import PROFILES  # noqa: E402
+
+COOKIE = open(sys.argv[2]).read().split("nxdns_session")[1].split()[0]
+BASE = "http://127.0.0.1:18080"
+GROUP = {"kids": 2}
+
+
+def lookup(domain, group_id):
+    req = urllib.request.Request(f"{BASE}/api/lookup?domain={domain}&group_id={group_id}")
+    req.add_header("Cookie", f"nxdns_session={COOKIE}")
+    return json.load(urllib.request.urlopen(req))
+
+
+cache = {}
+for name, prof in PROFILES.items():
+    gid = GROUP.get(name, 1)
+    total = blocked = 0
+    for domain, weight in prof["domains"]:
+        res = lookup(domain, gid)
+        total += weight
+        if sys.argv[3:] == ["-v"]:
+            print(name, domain, json.dumps(res)[:200])
+        is_blocked = res.get("blocked") or (res.get("decision") or {}).get("action") == "block"
+        if is_blocked:
+            blocked += weight
+        elif domain not in cache and not domain.endswith(".lan"):
+            try:
+                socket.getaddrinfo(domain, None)
+                cache[domain] = True
+            except OSError:
+                cache[domain] = False
+                print("UNRESOLVED", name, domain)
+    print(f"{name}: blocked weight share {blocked / total:.1%}")

website/demo/gen/contact_sheet.py

@@ -0,0 +1,47 @@
+"""usage: contact_sheet.py SHOTS_DIR OUT_HTML — light and dark side by side, images inlined."""
+import base64
+import html
+import sys
+
+SHOTS, OUT = sys.argv[1], sys.argv[2]
+PAGES = [
+    ("overview", "Overview — 24 hours of a six-device household: total and blocked queries, and which device made them."),
+    ("activity", "Activity — recent queries, each with its client, result and route."),
+    ("query-detail", "Query detail — a blocked app-analytics lookup and the provenance that blocked it."),
+    ("clients", "Clients — each device by name and group; the kids' phone runs with safe search and extra lists."),
+    ("diagnostics", "Diagnostics — a resolved upstream episode: the primary DoH resolver was unreachable for about two minutes."),
+]
+
+
+def img(name, scheme):
+    with open(f"{SHOTS}/{name}-{scheme}.png", "rb") as f:
+        data = base64.b64encode(f.read()).decode()
+    return f'<figure><img alt="{name} {scheme}" src="data:image/png;base64,{data}"><figcaption>{scheme}</figcaption></figure>'
+
+
+rows = "\n".join(
+    f'<section><h2>{html.escape(caption)}</h2><div class="pair">{img(name, "light")}{img(name, "dark")}</div></section>'
+    for name, caption in PAGES
+)
+page = f"""<!doctype html>
+<html lang="en"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
+<title>nxdns screenshots</title>
+<style>
+:root {{ --bg: #f6f6f7; --fg: #1d1d22; --muted: #66666e; --line: #d9d9de; }}
+@media (prefers-color-scheme: dark) {{ :root {{ --bg: #141417; --fg: #ececf0; --muted: #9a9aa3; --line: #2c2c33; }} }}
+body {{ margin: 0; padding: 24px 16px; background: var(--bg); color: var(--fg); font: 15px/1.5 system-ui, sans-serif; }}
+h1 {{ font-size: 20px; margin: 0 0 24px; }}
+h2 {{ font-size: 15px; font-weight: 500; margin: 32px 0 12px; }}
+.pair {{ display: grid; grid-template-columns: 1fr 1fr; gap: 16px; }}
+@media (max-width: 800px) {{ .pair {{ grid-template-columns: 1fr; }} }}
+figure {{ margin: 0; }}
+img {{ width: 100%; height: auto; border: 1px solid var(--line); border-radius: 6px; display: block; }}
+figcaption {{ color: var(--muted); font-size: 13px; margin-top: 4px; }}
+</style></head>
+<body><h1>nxdns admin — marketing screenshots (1440×900 @2x)</h1>
+{rows}
+</body></html>
+"""
+with open(OUT, "w") as f:
+    f.write(page)
+print(OUT)

website/demo/gen/profiles.py

@@ -0,0 +1,157 @@
+"""Per-device domain profiles: (domain, weight). Weights are relative lookup
+frequencies within a device; each lookup emits the device's query-type set."""
+
+# Query-type sets per lookup, with their probabilities.
+A, AAAA, HTTPS = 1, 28, 65
+APPLE_TYPES = [((A, AAAA, HTTPS), 0.75), ((A, AAAA), 0.15), ((A,), 0.10)]
+ANDROID_TYPES = [((A, AAAA), 0.55), ((A, AAAA, HTTPS), 0.30), ((A,), 0.15)]
+WINDOWS_TYPES = [((A, AAAA), 0.60), ((A, AAAA, HTTPS), 0.20), ((A,), 0.20)]
+TV_TYPES = [((A,), 0.70), ((A, AAAA), 0.30)]
+
+COMMON_WEB = [
+    ("www.google.com", 30), ("www.gstatic.com", 18), ("fonts.googleapis.com", 8),
+    ("fonts.gstatic.com", 8), ("www.wikipedia.org", 3), ("en.wikipedia.org", 4),
+    ("upload.wikimedia.org", 3), ("www.bbc.co.uk", 3), ("www.theguardian.com", 3),
+    ("www.reddit.com", 6), ("www.redditstatic.com", 3), ("i.redd.it", 3),
+    ("www.amazon.com", 4), ("m.media-amazon.com", 4), ("www.ebay.com", 2),
+    ("github.com", 2), ("cdn.jsdelivr.net", 3), ("cdnjs.cloudflare.com", 3),
+    ("www.youtube.com", 10), ("i.ytimg.com", 8), ("yt3.ggpht.com", 4),
+    ("www.nytimes.com", 2), ("static01.nyt.com", 2), ("www.weather.com", 2),
+    ("duckduckgo.com", 2), ("www.bing.com", 2), ("stackoverflow.com", 2),
+    ("www.imdb.com", 1), ("www.booking.com", 1), ("www.paypal.com", 1),
+    ("ajax.googleapis.com", 3), ("maps.googleapis.com", 3), ("www.google-analytics.com", 6),
+    ("www.googletagmanager.com", 6), ("googleads.g.doubleclick.net", 4),
+    ("securepubads.g.doubleclick.net", 3), ("pagead2.googlesyndication.com", 3),
+    ("connect.facebook.net", 2), ("bat.bing.com", 2), ("static.ads-twitter.com", 1),
+    ("c.amazon-adsystem.com", 2), ("cdn.taboola.com", 1), ("widgets.outbrain.com", 1),
+    ("sb.scorecardresearch.com", 2), ("js-agent.newrelic.com", 1), ("browser.sentry-cdn.com", 1),
+]
+
+PROFILES = {
+    "tv": {
+        "types": TV_TYPES,
+        "domains": [
+            ("api-global.netflix.com", 40), ("nrdp.prod.ftl.netflix.com", 18), ("assets.nflxext.com", 14),
+            ("ichnaea.netflix.com", 10), ("www.youtube.com", 22), ("youtubei.googleapis.com", 28),
+            ("i.ytimg.com", 22), ("redirector.googlevideo.com", 14), ("www.googleapis.com", 8),
+            ("disney.api.edge.bamgrid.com", 12), ("bam-sdk-configs.bamgrid.com", 4),
+            ("appconfigs.disney-plus.net", 5), ("apresolve.spotify.com", 4), ("spclient.wg.spotify.com", 5),
+            ("time.samsungcloudsolution.com", 6), ("otn.samsungcloudsolution.com", 4),
+            ("www.samsungotn.net", 2), ("osb-apps.samsungqbe.com", 4), ("gpm.samsungqbe.com", 3),
+            ("cdn.samsungcloudsolution.com", 3), 
+            ("connectivitycheck.gstatic.com", 4),
+            # Trackers and ads a smart TV really emits.
+            ("log-config.samsungacr.com", 9), ("log-ingestion-eu.samsungacr.com", 8),
+            ("config.samsungads.com", 7), ("samsung-com.112.2o7.net", 5), ("ad.samsungadhub.com", 4),
+            ("tracking.samsungads.com", 4), ("googleads.g.doubleclick.net", 6),
+            ("pubads.g.doubleclick.net", 5), ("s0.2mdn.net", 3), ("imasdk.googleapis.com", 4),
+            ("logs.netflix.com", 4), ("events.samsungads.com", 3),
+        ],
+        "lookups": 3900,
+    },
+    "iphone": {
+        "types": APPLE_TYPES,
+        "domains": COMMON_WEB + [
+            ("gateway.icloud.com", 14), ("p53-caldav.icloud.com", 4), ("api.apple-cloudkit.com", 8),
+            ("gspe1-ssl.ls.apple.com", 6), ("mesu.apple.com", 2), ("init.itunes.apple.com", 4),
+            ("bag.itunes.apple.com", 3), ("play.itunes.apple.com", 2), ("captive.apple.com", 6),
+            ("weather-data.apple.com", 4), ("configuration.ls.apple.com", 3),
+            ("graph.instagram.com", 8), ("i.instagram.com", 12), ("scontent.cdninstagram.com", 10),
+            ("static.cdninstagram.com", 4), ("graph.facebook.com", 5), ("g.whatsapp.net", 12),
+            ("mmg.whatsapp.net", 6), ("api.spotify.com", 5),
+            ("spclient.wg.spotify.com", 6), ("i.scdn.co", 4), ("www.google.com", 6),
+            ("maps.apple.com", 2), ("mask.icloud.com", 3),
+            # App analytics and ad SDKs.
+            ("app-measurement.com", 10), ("firebaselogging-pa.googleapis.com", 6),
+            ("app-analytics-services.com", 4), ("api.mixpanel.com", 3), ("api2.amplitude.com", 3),
+            ("graph.facebook.com", 2), ("ads.tiktok.com", 2), ("cdn.branch.io", 2), ("api2.branch.io", 2),
+            ("app.adjust.com", 3), ("in.appcenter.ms", 2), ("events.appsflyer.com", 3),
+            ("sessions.bugsnag.com", 2), ("iadsdk.apple.com", 3), ("metrics.icloud.com", 3),
+            ("notify.bugsnag.com", 1), ("ads.mopub.com", 1),
+        ],
+        "lookups": 1300,
+    },
+    "pixel": {
+        "types": ANDROID_TYPES,
+        "domains": COMMON_WEB + [
+            ("android.googleapis.com", 10), ("play.googleapis.com", 8), ("www.googleapis.com", 10),
+            ("android.clients.google.com", 6), ("mtalk.google.com", 10), ("connectivitycheck.gstatic.com", 8),
+            ("clients4.google.com", 4), ("play-lh.googleusercontent.com", 6), ("lh3.googleusercontent.com", 5),
+            ("photos.googleapis.com", 3), ("gmail.googleapis.com", 4), ("inbox.google.com", 2),
+            ("youtubei.googleapis.com", 10), ("i.ytimg.com", 6), ("rr1---sn-4g5e6nzz.googlevideo.com", 3),
+            ("g.whatsapp.net", 10), ("mmg.whatsapp.net", 5), ("web.whatsapp.com", 1),
+            ("i.instagram.com", 8), ("scontent.cdninstagram.com", 7), ("api.spotify.com", 4),
+            ("spclient.wg.spotify.com", 5), ("discord.com", 3), ("gateway.discord.gg", 3),
+            ("cdn.discordapp.com", 3), ("time.android.com", 3),
+            # App analytics and ad SDKs.
+            ("app-measurement.com", 10), ("firebaselogging-pa.googleapis.com", 8),
+            ("crashlyticsreports-pa.googleapis.com", 4), ("settings.crashlytics.com", 3),
+            ("googleads.g.doubleclick.net", 5), ("adservice.google.com", 3),
+            ("graph.facebook.com", 3), ("an.facebook.com", 2), ("api.mixpanel.com", 2),
+            ("app.adjust.com", 2), ("t.appsflyer.com", 2), ("ads.mopub.com", 1),
+        ],
+        "lookups": 1450,
+    },
+    "kids": {
+        "types": ANDROID_TYPES,
+        "domains": [
+            ("www.google.com", 12), ("www.youtube.com", 20), ("youtubei.googleapis.com", 18),
+            ("i.ytimg.com", 14), ("yt3.ggpht.com", 6), ("android.googleapis.com", 6),
+            ("mtalk.google.com", 6), ("connectivitycheck.gstatic.com", 5), ("www.gstatic.com", 5),
+            ("play.googleapis.com", 5), ("www.roblox.com", 10), ("apis.roblox.com", 10),
+            ("tr.rbxcdn.com", 6), ("www.minecraft.net", 3),
+            ("api.minecraftservices.com", 4), ("pocket.realms.minecraft.net", 2),
+            ("api.spotify.com", 3), ("www.duolingo.com", 3), ("android-api.duolingo.com", 4),
+            ("en.wikipedia.org", 2), ("www.bing.com", 2), ("g.whatsapp.net", 4),
+            ("www.tiktok.com", 4), ("api16-normal-c-useast1a.tiktokv.com", 4),
+            ("v16-webapp.tiktok.com", 3), ("discord.com", 2),
+            # Game ad networks and analytics.
+            ("unityads.unity3d.com", 7), ("config.unityads.unity3d.com", 4),
+            ("auction.unityads.unity3d.com", 4), ("ms.applovin.com", 5), ("a.applovin.com", 4),
+            ("rt.applovin.com", 3), ("googleads.g.doubleclick.net", 6), ("app-measurement.com", 7),
+            ("firebaselogging-pa.googleapis.com", 4), ("supersonicads.com", 2),
+            ("outcome-ssp.supersonicads.com", 2), ("init.supersonicads.com", 2),
+            ("events.appsflyer.com", 2), ("ads.vungle.com", 2), ("api.vungle.com", 2),
+            # Casino ads in free games: only the kids group blocks these.
+            ("www.pokerstars.com", 1), ("www.bet365.com", 1), ("www.888casino.com", 1)],
+        "lookups": 1150,
+    },
+    "desktop": {
+        "types": WINDOWS_TYPES,
+        "domains": COMMON_WEB + [
+            ("www.msftconnecttest.com", 6), ("login.live.com", 6), ("login.microsoftonline.com", 4),
+            ("settings-win.data.microsoft.com", 6), ("v10.events.data.microsoft.com", 10),
+            ("watson.telemetry.microsoft.com", 4), ("self.events.data.microsoft.com", 5),
+            ("browser.events.data.msn.com", 4), ("arc.msn.com", 3), ("ris.api.iris.microsoft.com", 3),
+            ("www.bing.com", 6), ("assets.msn.com", 4), ("ntp.msn.com", 4), ("img-s-msn-com.akamaized.net", 3),
+            ("fe3cr.delivery.mp.microsoft.com", 3), ("tsfe.trafficshaping.dsp.mp.microsoft.com", 2),
+            ("dl.delivery.mp.microsoft.com", 2), ("ctldl.windowsupdate.com", 3), ("time.windows.com", 2),
+            ("outlook.office365.com", 6), ("teams.microsoft.com", 4), ("graph.microsoft.com", 4),
+            ("onedrive.live.com", 2), ("store.steampowered.com", 3), ("api.steampowered.com", 4),
+            ("steamcdn-a.akamaihd.net", 2), ("discord.com", 3), ("gateway.discord.gg", 3),
+            ("clients2.google.com", 3), ("update.googleapis.com", 2), ("nas.lan", 3), ("printer.lan", 1),
+            ("vortex.data.microsoft.com", 4), ("mobile.events.data.microsoft.com", 3),
+            ("telemetry.nvidia.com", 2), ("events.gfe.nvidia.com", 2),
+        ],
+        "lookups": 900,
+    },
+    "mac": {
+        "types": APPLE_TYPES,
+        "domains": COMMON_WEB + [
+            ("gateway.icloud.com", 10), ("api.apple-cloudkit.com", 6), ("gs-loc.apple.com", 3),
+            ("swscan.apple.com", 2), ("swdist.apple.com", 1), ("xp.apple.com", 4),
+            ("ocsp2.apple.com", 6), ("valid.apple.com", 2), ("captive.apple.com", 4),
+            ("configuration.ls.apple.com", 2), ("time.apple.com", 2), ("api.github.com", 4),
+            ("avatars.githubusercontent.com", 3), ("objects.githubusercontent.com", 2),
+            ("registry.npmjs.org", 3), ("slack.com", 6), ("wss-primary.slack.com", 4),
+            ("edgeapi.slack.com", 3), ("a.slack-edge.com", 3), ("zoom.us", 3), ("us05web.zoom.us", 2),
+            ("www.notion.so", 4), ("api.spotify.com", 3), ("spclient.wg.spotify.com", 4),
+            ("docs.google.com", 5), ("drive.google.com", 3), ("mail.google.com", 6),
+            ("nas.lan", 4), ("files.lan", 2), ("printer.lan", 2),
+            ("metrics.icloud.com", 3), ("api-adservices.apple.com", 2), ("notes-analytics-events.apple.com", 2),
+            ("stats.g.doubleclick.net", 3), ("o0.ingest.sentry.io", 2), ("api.segment.io", 2),
+            ("cdn.segment.com", 2), ("api.mixpanel.com", 1), ("in.hotjar.com", 1), ("static.hotjar.com", 1),
+        ],
+        "lookups": 800,
+    },
+}

website/demo/gen/rewrite_history.py

@@ -0,0 +1,133 @@
+"""Spread the bulk-generated query log over the previous day with a household
+rhythm, then rebuild every projection from the rewritten rows.
+
+Run only while nxdns is stopped. usage: rewrite_history.py DATA_DIR
+"""
+import datetime
+import random
+import sqlite3
+import sys
+import time
+
+DATA = sys.argv[1]
+NOW = int(time.time())
+START = NOW - 24 * 3600 + 120
+END = NOW - 300
+CREATED_AT = START - 1200
+
+# Relative activity per local hour of day, per device.
+CURVES = {
+    "192.168.50.41": [2, 1, 1, 1, 1, 1, 3, 20, 25, 8, 6, 6, 10, 8, 6, 8, 15, 45, 80, 100, 100, 90, 60, 20],  # TV
+    "192.168.50.52": [6, 3, 3, 3, 3, 5, 25, 70, 60, 35, 35, 40, 60, 45, 35, 40, 45, 60, 70, 80, 85, 80, 60, 25],  # iPhone
+    "192.168.50.53": [5, 3, 3, 3, 3, 4, 15, 60, 70, 40, 40, 45, 65, 50, 40, 45, 50, 65, 75, 80, 75, 70, 50, 20],  # Pixel
+    "192.168.50.54": [1, 1, 1, 1, 1, 1, 4, 35, 20, 2, 2, 2, 4, 2, 2, 40, 90, 100, 85, 80, 60, 25, 4, 2],  # Kids' phone
+    "192.168.50.20": [2, 1, 1, 1, 1, 1, 1, 4, 10, 15, 15, 15, 20, 15, 15, 20, 30, 50, 80, 100, 100, 85, 50, 15],  # Desktop PC
+    "192.168.50.21": [2, 1, 1, 1, 1, 1, 2, 15, 60, 100, 100, 95, 60, 90, 100, 95, 80, 40, 25, 25, 20, 12, 5, 3],  # MacBook Air
+}
+
+
+def weight(ip, ts):
+    return CURVES[ip][datetime.datetime.fromtimestamp(ts).hour]
+
+
+def sample_times(ip, n):
+    minutes = list(range(START, END, 60))
+    weights = [weight(ip, m) for m in minutes]
+    return sorted(m + random.randrange(60) for m in random.choices(minutes, weights, k=n))
+
+
+def lookups(rows):
+    """Group a client's rows into lookups: the A/AAAA/HTTPS burst of one name."""
+    groups, last = [], None
+    for row_id, ts, domain_id in rows:
+        if last and last[0] == domain_id and ts - last[1] <= 1:
+            groups[-1].append(row_id)
+        else:
+            groups.append([row_id])
+        last = (domain_id, ts)
+    return groups
+
+
+# The repository's own bucket recompute (src/storage/repositories/queries_repo.zig,
+# `recompute_bucket_sql`), verbatim; ?1 is the bucket start, ?2 its exclusive end.
+RECOMPUTE_BUCKET_SQL = [
+    """INSERT INTO bucket_totals (bucket, queries, blocked, cached, rt_sum, rt_count)
+SELECT ?1, count(*), coalesce(sum(blocked <> 0), 0), coalesce(sum(cache_hit = 1), 0),
+       coalesce(sum(response_time_us), 0), count(response_time_us)
+  FROM query_log
+ WHERE timestamp >= ?1 AND timestamp < ?2
+HAVING count(*) > 0""",
+    """INSERT INTO bucket_clients (bucket, client_ip, queries)
+SELECT ?1, client_ip, count(*)
+  FROM query_log
+ WHERE timestamp >= ?1 AND timestamp < ?2
+ GROUP BY client_ip""",
+    """INSERT INTO bucket_types (bucket, qtype, count)
+SELECT ?1, coalesce(qtype, -1), count(*)
+  FROM query_log
+ WHERE timestamp >= ?1 AND timestamp < ?2
+ GROUP BY coalesce(qtype, -1)""",
+    """INSERT INTO bucket_routes (bucket, route_kind, source_present, source_text, count)
+SELECT ?1, route_kind, source IS NOT NULL, coalesce(source, ''), count(*)
+  FROM (SELECT route_kind,
+               CASE route_kind
+                    WHEN 'upstream' THEN upstream
+                    WHEN 'forward_zone' THEN forward_zone
+               END AS source
+          FROM query_log
+         WHERE timestamp >= ?1 AND timestamp < ?2)
+ GROUP BY route_kind, source""",
+]
+GRAIN = 1800
+
+COLUMNS = (
+    "timestamp, domain_id, client_ip, qtype, blocked, response_time_us, cache_hit, upstream, qclass, rcode,"
+    " group_id, group_name, policy_action, policy_reason, matched, source_id, source_name, cname_target,"
+    " safe_search_target, route_kind, forward_zone"
+)
+
+ql = sqlite3.connect(f"{DATA}/querylog.db", isolation_level=None)
+ql.execute("BEGIN IMMEDIATE")
+new_ts = {}
+for (ip,) in ql.execute("SELECT DISTINCT client_ip FROM query_log").fetchall():
+    rows = ql.execute(
+        "SELECT id, timestamp, domain_id FROM query_log WHERE client_ip = ? ORDER BY timestamp, id", (ip,)
+    ).fetchall()
+    groups = lookups(rows)
+    for group, ts in zip(groups, sample_times(ip, len(groups))):
+        for row_id in group:
+            new_ts[row_id] = ts
+ql.execute("CREATE TEMP TABLE remap (old_id INTEGER PRIMARY KEY, ts INTEGER NOT NULL)")
+ql.executemany("INSERT INTO remap VALUES (?, ?)", new_ts.items())
+rest = COLUMNS.split(", ", 1)[1]
+ql.execute(f"CREATE TEMP TABLE rewritten AS SELECT r.ts AS timestamp, {rest}, q.id AS old_id"
+           " FROM query_log q JOIN remap r ON r.old_id = q.id")
+before = ql.execute("SELECT count(*) FROM query_log").fetchone()[0]
+ql.execute("DELETE FROM query_log")
+ql.execute(f"INSERT INTO query_log ({COLUMNS}) SELECT {COLUMNS} FROM rewritten ORDER BY timestamp, old_id")
+after = ql.execute("SELECT count(*) FROM query_log").fetchone()[0]
+assert before == after == len(new_ts), (before, after, len(new_ts))
+
+for table in ("bucket_totals", "bucket_clients", "bucket_types", "bucket_routes"):
+    ql.execute(f"DELETE FROM {table}")
+buckets = [b for (b,) in ql.execute(f"SELECT DISTINCT timestamp / {GRAIN} * {GRAIN} FROM query_log")]
+for bucket in buckets:
+    for sql in RECOMPUTE_BUCKET_SQL:
+        ql.execute(sql, (bucket, bucket + GRAIN))
+ql.execute("UPDATE querylog_meta SET created_at = ?, available_since = ? WHERE id = 1", (CREATED_AT, CREATED_AT + 1))
+ql.execute("COMMIT")
+ql.execute("PRAGMA wal_checkpoint(TRUNCATE)")
+span = ql.execute("SELECT min(timestamp), max(timestamp), count(*) FROM query_log").fetchone()
+print(f"querylog: {span[2]} rows from {span[0]} to {span[1]}, {len(buckets)} buckets rebuilt")
+
+cfg = sqlite3.connect(f"{DATA}/config.db", isolation_level=None)
+cfg.execute("ATTACH ? AS ql", (f"{DATA}/querylog.db",))
+cfg.execute("BEGIN IMMEDIATE")
+cfg.execute("""UPDATE clients SET
+    first_seen = coalesce((SELECT min(timestamp) FROM ql.query_log WHERE client_ip = clients.ip), first_seen),
+    last_seen  = coalesce((SELECT max(timestamp) FROM ql.query_log WHERE client_ip = clients.ip), last_seen)""")
+cfg.execute("UPDATE rules SET created_at = ?", (CREATED_AT,))
+cfg.execute("COMMIT")
+cfg.execute("PRAGMA wal_checkpoint(TRUNCATE)")
+for row in cfg.execute("SELECT ip, name, first_seen, last_seen FROM clients ORDER BY ip"):
+    print("client", row)

website/demo/gen/shoot.mjs

@@ -0,0 +1,50 @@
+// usage: node shoot.mjs OUT_DIR QUERY_ID
+import { chromium } from "playwright";
+
+const [outDir, queryId] = process.argv.slice(2);
+const BASE = "http://127.0.0.1:18080";
+const PAGES = [
+	["overview", "/overview?period=24h"],
+	["activity", "/activity"],
+	["query-detail", `/activity/queries/${queryId}`],
+	["clients", "/clients"],
+	["diagnostics", "/diagnostics"],
+];
+
+async function settle(page) {
+	await page.waitForLoadState("networkidle").catch(() => {});
+	await page.waitForFunction(
+		() => !document.querySelector('[aria-busy="true"], [role="progressbar"]'),
+		null,
+		{ timeout: 15000 },
+	).catch(() => console.log("  still busy after 15s"));
+	await page.waitForTimeout(2500);
+}
+
+const browser = await chromium.launch();
+for (const scheme of ["light", "dark"]) {
+	const context = await browser.newContext({
+		viewport: { width: 1440, height: 900 },
+		deviceScaleFactor: 2,
+		colorScheme: scheme,
+		timezoneId: "Europe/Paris",
+		locale: "en-GB",
+	});
+	const page = await context.newPage();
+	page.on("console", (m) => m.type() === "error" && console.log(`  console error: ${m.text()}`));
+	await page.goto(`${BASE}/login`);
+	await page.locator("#password").fill("nxdns-demo");
+	await page.locator('button[type="submit"]').click();
+	await page.waitForURL((u) => !u.pathname.startsWith("/login"));
+	for (const [name, path] of PAGES) {
+		await page.goto(BASE + path);
+		await page.mouse.move(1430, 890);
+		await settle(page);
+		if (name === "activity") await page.waitForTimeout(8000);
+		const file = `${outDir}/${name}-${scheme}.png`;
+		await page.screenshot({ path: file });
+		console.log(file);
+	}
+	await context.close();
+}
+await browser.close();

website/demo/gen/traffic.py

@@ -0,0 +1,67 @@
+"""One household device's DNS traffic, stdlib only.
+
+usage: traffic.py PROFILE bulk          send the profile's lookup budget fast, then exit
+       traffic.py PROFILE live MEAN_S   one lookup every ~MEAN_S seconds, forever
+"""
+import os
+import random
+import socket
+import struct
+import sys
+import time
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+from profiles import PROFILES  # noqa: E402
+
+SERVER = (os.environ.get("NXDNS", "192.168.50.2"), 53)
+
+
+def build_query(qid, name, qtype):
+    header = struct.pack("!HHHHHH", qid, 0x0100, 1, 0, 0, 1)
+    qname = b"".join(bytes([len(label)]) + label.encode() for label in name.split(".")) + b"\0"
+    opt = b"\0" + struct.pack("!HHIH", 41, 1232, 0, 0)
+    return header + qname + struct.pack("!HH", qtype, 1) + opt
+
+
+def pick_types(type_sets):
+    sets, weights = zip(*type_sets)
+    return random.choices(sets, weights)[0]
+
+
+def main():
+    name, mode = sys.argv[1], sys.argv[2]
+    profile = PROFILES[name]
+    domains, weights = zip(*profile["domains"])
+    sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
+    sock.settimeout(6)
+    sent = answered = failed = 0
+    lookups = profile["lookups"] if mode == "bulk" else None
+    mean = float(sys.argv[3]) if mode == "live" else 0.0
+    done = 0
+    while lookups is None or done < lookups:
+        domain = random.choices(domains, weights)[0]
+        for qtype in pick_types(profile["types"]):
+            qid = random.randrange(65536)
+            sock.sendto(build_query(qid, domain, qtype), SERVER)
+            sent += 1
+            try:
+                while True:
+                    data, _ = sock.recvfrom(4096)
+                    if struct.unpack("!H", data[:2])[0] == qid:
+                        answered += 1
+                        break
+            except socket.timeout:
+                failed += 1
+                print(f"{name}: no answer for {domain} type {qtype}", flush=True)
+        done += 1
+        if mode == "bulk":
+            time.sleep(random.uniform(0.05, 0.25))
+        else:
+            time.sleep(random.expovariate(1.0 / mean))
+        if done % 500 == 0:
+            print(f"{name}: {done} lookups, {sent} sent, {answered} answered, {failed} unanswered", flush=True)
+    print(f"{name}: finished {done} lookups, {sent} sent, {answered} answered, {failed} unanswered", flush=True)
+
+
+if __name__ == "__main__":
+    main()

website/demo/package-lock.json

@@ -0,0 +1,42 @@
+{
+  "name": "nxdns-screenshot-demo",
+  "lockfileVersion": 3,
+  "requires": true,
+  "packages": {
+    "": {
+      "name": "nxdns-screenshot-demo",
+      "devDependencies": {
+        "playwright": "1.64.0"
+      }
+    },
+    "node_modules/playwright": {
+      "version": "1.64.0",
+      "resolved": "https://registry.npmjs.org/playwright/-/playwright-1.64.0.tgz",
+      "integrity": "sha512-kVzTnYFYEyQ9RGzBjNCd2+yEqWmVbxl3AHPynv3qQ6JTjv4a+kwRcnqQwIpuP3NnrnIrE/Q6wWAIAvMfIE2/Yg==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "dependencies": {
+        "playwright-core": "1.64.0"
+      },
+      "bin": {
+        "playwright": "cli.js"
+      },
+      "engines": {
+        "node": ">=20"
+      }
+    },
+    "node_modules/playwright-core": {
+      "version": "1.64.0",
+      "resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.64.0.tgz",
+      "integrity": "sha512-T9r+MZkTECl2+oUcZ26YgZzTrYidTsIhYYkJix+6iDOymOb1FB2RLLSOxRGnZGv5DStt+aduIVbt+E8WygXYpg==",
+      "dev": true,
+      "license": "Apache-2.0",
+      "bin": {
+        "playwright-core": "cli.js"
+      },
+      "engines": {
+        "node": ">=20"
+      }
+    }
+  }
+}

website/demo/package.json

@@ -0,0 +1,7 @@
+{
+  "name": "nxdns-screenshot-demo",
+  "private": true,
+  "devDependencies": {
+    "playwright": "1.64.0"
+  }
+}

website/demo/start.sh

@@ -0,0 +1,16 @@
+#!/bin/sh
+# Starts nxdns on the demo LAN, importing etc/config.zon into a fresh nxdemo-data volume on the
+# first run. nxdns keeps the image's own uid 65532: under a host uid it was signalled from outside
+# docker twice. The data lives in a named volume so no file in this tree is owned by that uid.
+set -eu
+DEMO=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
+IMAGE=${NXDNS_IMAGE:-git.mial.net/mokhtar/nxdns:0.0.21}
+docker network inspect nxdemo-lan >/dev/null 2>&1 ||
+  docker network create --driver bridge --subnet 192.168.50.0/24 --gateway 192.168.50.1 nxdemo-lan
+if ! docker volume inspect nxdemo-data >/dev/null 2>&1; then
+  docker volume create nxdemo-data >/dev/null
+  docker run --rm -v "$DEMO/etc:/etc/nxdns:ro" -v nxdemo-data:/var/lib/nxdns "$IMAGE" import /etc/nxdns/config.zon
+fi
+docker run -d --name nxdemo-nxdns --restart unless-stopped \
+  --network nxdemo-lan --ip 192.168.50.2 --sysctl net.ipv4.ip_unprivileged_port_start=0 \
+  -p 127.0.0.1:18080:8080 -v nxdemo-data:/var/lib/nxdns "$IMAGE" run

website/demo/teardown.sh

@@ -0,0 +1,7 @@
+#!/bin/sh
+# Removes every container, the volume and the network the demo created. Each step runs even if an
+# earlier one finds nothing to remove.
+set -u
+docker rm -f nxdemo-tv nxdemo-iphone nxdemo-pixel nxdemo-kids nxdemo-desktop nxdemo-mac nxdemo-nxdns
+docker volume rm nxdemo-data
+docker network rm nxdemo-lan