diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3bffaff..a82b6eb 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -4,7 +4,15 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
-## [0.0.11] - 2026-08-24
+## [0.0.12] - 2026-08-27
+
+Overview stops re-reading the whole query log. One endpoint, one snapshot, pre-aggregated buckets — a 30-day view now costs the same on a month of history as on a day of it. Read the upgrade note first: it resets your query history.
+
+### Changed
+
+- **Upgrading resets your query history.** The query-log schema gains the aggregate tables described below, and `querylog.db` is never migrated: the first start after the upgrade sets the old file aside (kept on disk next to the new one, named with the reason) and begins a fresh log. Settings, groups, blocklists and every other configuration are untouched.
+- **The Overview is served by one endpoint, `GET /api/overview`.** It replaces `GET /api/stats`, `/api/stats/timeseries`, `/api/stats/types`, `/api/stats/routes` and `/api/stats/clients`, which are gone. The five panels now come from a single database snapshot, so they can no longer disagree with each other, and the page shows one loading and one error state instead of five.
+- **Query statistics are pre-aggregated as they are written.** The query log now maintains 30-minute aggregate tables in the same transaction that stores the rows, and the 24-hour, 7-day and 30-day views read those instead of scanning every logged query. The cost of opening the Overview no longer grows with the size of the log: measured at three million rows, the 30-day view went from roughly eight-tenths of a second of scanning to under fifty milliseconds, at the price of about ten percent on each background write batch and ~1.5 MB of disk. The server also keeps the most recent response per period in memory and serves repeat polls from it while nothing has changed — until new queries land, retention prunes, or the period's time window rolls forward — so on a quiet network most of the steady 30-second refreshes do no database work at all.
The Overview charts move to visx and grow up: one hover treatment across all four, honest labels, and maintained d3 math under the app's own rendering.
diff --git a/PLAN.md b/PLAN.md
index 91a346a..cfadb37 100644
--- a/PLAN.md
+++ b/PLAN.md
@@ -238,7 +238,7 @@ src/
web/
server.zig router.zig auth.zig sse.zig static.zig metrics.zig openapi.zig
handlers/
- auth.zig stats.zig queries.zig clients.zig groups.zig blocklists.zig
+ auth.zig overview.zig queries.zig clients.zig groups.zig blocklists.zig
rules.zig local.zig lookup.zig pause.zig settings.zig
upstream_health.zig certs.zig health.zig version.zig
@@ -484,6 +484,8 @@ CREATE INDEX idx_query_log_client ON query_log(client_ip);
CREATE INDEX idx_query_log_domain ON query_log(domain_id);
```
+The sketch above is the original shape; `src/storage/querylog_schema.zig` is the authority, and the provenance columns milestone 28 added are not repeated here. Beside the raw rows the file carries four projection tables — `bucket_totals`, `bucket_clients`, `bucket_types`, `bucket_routes` — on a 30-minute grain, which is what `GET /api/overview` reads for the 24h, 7d and 30d windows instead of scanning every row. They are maintained by the batch writer and by retention inside the same transaction as the raw rows, so SQLite's transaction is the whole coherence story: no second file, no backfill, no rebuild command. The 1h window is narrower than the grain and takes one raw scan.
+
### 11.4 Query Logger
- In-memory buffer, mutex guarded, hard cap `query_log_buffer_max` (default 10000).
@@ -493,7 +495,7 @@ CREATE INDEX idx_query_log_domain ON query_log(domain_id);
### 11.5 Retention
-Periodic delete of rows older than `retention_days`; scheduled checkpoint/VACUUM on `querylog.db` only.
+Periodic delete of rows older than `retention_days`, dropping the projection buckets behind the cutoff and recomputing the straddling one in the same transaction; scheduled checkpoint/VACUUM on `querylog.db` only.
### 11.6 Disk Discipline (cloudflared lesson)
@@ -536,7 +538,7 @@ Scalars in `settings(key, value)`; ordered/structured items in dedicated tables.
### 13.1 Endpoints
- `POST /api/auth/login`, `POST /api/auth/logout`
-- `GET /api/stats?period=…`, `GET /api/stats/timeseries?period=…`, `GET /api/stats/types?period=…`, `GET /api/stats/routes?period=…`, `GET /api/stats/clients?period=…`
+- `GET /api/overview?period=…` — every Overview panel in one response over one read transaction
- `GET /api/queries` (filter + paginate), `GET /api/queries/live` (SSE, per-IP cap)
- `GET/PUT /api/clients/{id}`
- `GET/POST/PUT/DELETE /api/groups…`, `/api/blocklists…`, `/api/rules…`, `/api/local-records…`, `/api/forward-zones…`
@@ -664,7 +666,7 @@ The project publishes released binaries and container images from its own Gitea
6. Disk-fill degrades gracefully; no silent log-flood failure mode.
7. Web UI + API provide full admin functionality; OpenAPI contract tests green.
8. `nxdns export` round-trips via `nxdns import`.
-9. Query logging, stats, SSE live stream work; querylog.db corruption self-heals.
+9. Query logging, the overview, SSE live stream work; querylog.db corruption self-heals.
10. Local DoH + DoT endpoints serve LAN clients.
11. Schema upgrade = install + restart (migration test proves it).
12. All suites green in Gitea CI for both targets.
diff --git a/admin/src/features/configuration/authority.test.tsx b/admin/src/features/configuration/authority.test.tsx
index 72112db..0306397 100644
--- a/admin/src/features/configuration/authority.test.tsx
+++ b/admin/src/features/configuration/authority.test.tsx
@@ -99,14 +99,17 @@ test("a failed status is announced by the shell on a page that is not configurat
stubApi(DATABASE, {
responses: {
"GET /api/config/status": new Response(JSON.stringify({ error: "gone" }), { status: 404 }),
- "GET /api/stats?period=24h": {
+ "GET /api/overview?period=24h": {
period: "24h",
since: 0,
until: 86400,
- queries: 0,
- blocked: 0,
- clients: 0,
- avg_response_time_us: null,
+ bucket_seconds: 1800,
+ totals: { queries: 0, blocked: 0, clients: 0, avg_response_time_us: null },
+ buckets: [],
+ clients: [],
+ other: [],
+ types: [],
+ routes: [],
coverage: { complete: true, available_since: 0 },
},
},
diff --git a/admin/src/features/overview/ClientChart.test.tsx b/admin/src/features/overview/ClientChart.test.tsx
index 967c4cc..0c9f2b6 100644
--- a/admin/src/features/overview/ClientChart.test.tsx
+++ b/admin/src/features/overview/ClientChart.test.tsx
@@ -2,23 +2,14 @@ import { fireEvent, render as renderBare, screen, within } from "@testing-librar
import { QueryClientProvider } from "@tanstack/react-query";
import { createQueryClient } from "@/lib/queryClient";
import { formatTime } from "@/lib/format";
-import type { StatsClients } from "@/lib/types";
-import ClientChart from "./ClientChart";
+import ClientChart, { type ClientChartData } from "./ClientChart";
import { OTHER_KEY, clientKey, seriesColor } from "./seriesColors";
const SINCE = 1_700_000_000;
const BUCKET = 1800;
-function clients(named: { client: string; buckets: number[] }[], other: number[]): StatsClients {
- return {
- period: "24h",
- since: SINCE,
- until: SINCE + other.length * BUCKET,
- bucket_seconds: BUCKET,
- coverage: { complete: true, available_since: SINCE },
- clients: named,
- other,
- };
+function clients(named: { client: string; buckets: number[] }[], other: number[]): ClientChartData {
+ return { since: SINCE, bucket_seconds: BUCKET, clients: named, other };
}
const TWO_BUCKETS = clients(
@@ -34,15 +25,15 @@ const TWO_BUCKETS = clients(
* client is registered in these fixtures, which is what leaves the addresses on
* screen as the labels.
*/
-function render(data: StatsClients) {
+function render(data: ClientChartData) {
const client = createQueryClient();
- const tree = (next: StatsClients) => (
+ const tree = (next: ClientChartData) => (
);
const result = renderBare(tree(data));
- return { ...result, rerender: (next: StatsClients) => result.rerender(tree(next)) };
+ return { ...result, rerender: (next: ClientChartData) => result.rerender(tree(next)) };
}
beforeEach(() => {
diff --git a/admin/src/features/overview/ClientChart.tsx b/admin/src/features/overview/ClientChart.tsx
index b68a656..412c3dd 100644
--- a/admin/src/features/overview/ClientChart.tsx
+++ b/admin/src/features/overview/ClientChart.tsx
@@ -13,7 +13,7 @@ import * as stylex from "@stylexjs/stylex";
import { Group } from "@visx/group";
import { BarStack } from "@visx/shape";
import { formatTime } from "@/lib/format";
-import type { StatsClients } from "@/lib/types";
+import type { OverviewClientSeries } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { clientLabel, useClientNames, type ClientNames } from "@/features/clients/clientNames";
@@ -82,7 +82,7 @@ interface Series {
* and a table column all saying zero. The named clients stay at zero, because a
* client that went quiet is something the reader wants to see.
*/
-function seriesOf(data: StatsClients, names: ClientNames): Series[] {
+function seriesOf(data: ClientChartData, names: ClientNames): Series[] {
const named = data.clients.map((client) => ({
key: clientKey(client.client),
// The name if the client is registered under one, the address otherwise —
@@ -100,10 +100,21 @@ function seriesOf(data: StatsClients, names: ClientNames): Series[] {
];
}
+/**
+ * The slice of the Overview body this chart draws. Declared here rather than
+ * taken whole, so what the chart reads is stated where it is read.
+ */
+export interface ClientChartData {
+ since: number;
+ bucket_seconds: number;
+ clients: OverviewClientSeries[];
+ other: number[];
+}
+
/** One column: the timestamp plus one entry per series, keyed by the series key. */
type Column = { ts: number } & Record;
-export default function ClientChart({ data }: { data: StatsClients }) {
+export default function ClientChart({ data }: { data: ClientChartData }) {
const [containerRef, width] = useMeasuredWidth();
// A hover survives a re-render only while it still names the same bucket at
// the same place: a poll that rolls the window, or a resize, retires it.
diff --git a/admin/src/features/overview/OverviewFrame.tsx b/admin/src/features/overview/OverviewFrame.tsx
new file mode 100644
index 0000000..2edf33e
--- /dev/null
+++ b/admin/src/features/overview/OverviewFrame.tsx
@@ -0,0 +1,133 @@
+/**
+ * The part of Overview that does not wait for anything: the heading, the period
+ * picker, and the pulsing body the page shows while the window is in flight.
+ *
+ * It lives apart from `OverviewPage` so the route's pending component can render
+ * the identical surface while the page chunk loads. Importing the page itself
+ * would pull the charts into the main bundle, and a second hand-written copy of
+ * the frame would drift. Nothing here imports a chart.
+ */
+
+import * as stylex from "@stylexjs/stylex";
+import { useNavigate, useSearch } from "@tanstack/react-router";
+import type { Period } from "@/lib/types";
+import { styles as shared } from "@/ui/styles";
+import { colors } from "@/ui/tokens.stylex";
+import { DEFAULT_PERIOD, PERIODS } from "./period";
+
+const styles = stylex.create({
+ page: {
+ display: "flex",
+ flexDirection: "column",
+ gap: "1rem",
+ },
+ headingRow: {
+ display: "flex",
+ flexWrap: "wrap",
+ alignItems: "center",
+ justifyContent: "space-between",
+ gap: "0.75rem",
+ },
+ heading: {
+ fontSize: "1.5rem",
+ lineHeight: "2rem",
+ fontWeight: 600,
+ },
+ periodGroup: {
+ display: "flex",
+ gap: "0.25rem",
+ },
+ period: {
+ borderStyle: "none",
+ borderRadius: "0.25rem",
+ paddingInline: "0.625rem",
+ paddingBlock: "0.25rem",
+ fontSize: "0.875rem",
+ lineHeight: "1.25rem",
+ },
+ /** The pressed fill is heavier than `surfaceHover`, so a hover cannot mimic it. */
+ periodSelected: {
+ backgroundColor: {
+ default: "oklch(92% 0.004 286.32)",
+ "@media (prefers-color-scheme: dark)": "oklch(37% 0.013 285.805)",
+ },
+ color: colors.text,
+ fontWeight: 500,
+ },
+ periodIdle: {
+ backgroundColor: { default: "transparent", ":hover": colors.surfaceHover },
+ color: colors.textSecondary,
+ },
+ loading: {
+ fontSize: "0.875rem",
+ lineHeight: "1.25rem",
+ color: colors.textMuted,
+ },
+});
+
+export function PeriodPicker({ period, onChange }: { period: Period; onChange: (period: Period) => void }) {
+ return (
+
+ );
+}
+
+/**
+ * The route's pending surface. The picker stays live because it only writes the
+ * search parameter, which the route already re-reads on its own.
+ */
+export function OverviewPending() {
+ const period = useSearch({ from: "/shell/overview" }).period ?? DEFAULT_PERIOD;
+ const navigate = useNavigate({ from: "/overview" });
+ return (
+ void navigate({ search: (prev) => ({ ...prev, period: next }) })}
+ >
+
+
+ );
+}
diff --git a/admin/src/features/overview/OverviewPage.test.tsx b/admin/src/features/overview/OverviewPage.test.tsx
index b551803..7ac2d65 100644
--- a/admin/src/features/overview/OverviewPage.test.tsx
+++ b/admin/src/features/overview/OverviewPage.test.tsx
@@ -16,64 +16,32 @@ import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import { clientKey, qtypeKey, seriesColor } from "./seriesColors";
import { health } from "@/lib/healthFixture";
-import type { Health, StatsClients, StatsRoutes, StatsTimeseries, StatsTotals, StatsTypes } from "@/lib/types";
+import type { Health, Overview } from "@/lib/types";
const SINCE = Date.UTC(2026, 0, 1, 0, 0) / 1000;
const UNTIL = Date.UTC(2026, 0, 2, 0, 0) / 1000;
const COVERAGE = { complete: true, available_since: SINCE };
-const TOTALS: StatsTotals = {
- period: "24h",
- since: SINCE,
- until: UNTIL,
- queries: 1000,
- blocked: 250,
- clients: 7,
- avg_response_time_us: 2345,
- coverage: COVERAGE,
-};
-
-const SERIES: StatsTimeseries = {
+const OVERVIEW: Overview = {
period: "24h",
since: SINCE,
until: UNTIL,
bucket_seconds: 1800,
+ totals: { queries: 1000, blocked: 250, clients: 7, avg_response_time_us: 2345 },
buckets: [
{ ts: SINCE, queries: 60, blocked: 20, cached: 10 },
{ ts: SINCE + 1800, queries: 40, blocked: 0, cached: 0 },
],
- coverage: COVERAGE,
-};
-
-const CLIENTS: StatsClients = {
- period: "24h",
- since: SINCE,
- until: UNTIL,
- bucket_seconds: 1800,
clients: [
{ client: "192.0.2.30", buckets: [40, 20] },
{ client: "192.0.2.31", buckets: [20, 20] },
],
other: [0, 0],
- coverage: COVERAGE,
-};
-
-const TYPES: StatsTypes = {
- period: "24h",
- since: SINCE,
- until: UNTIL,
types: [
{ qtype: 1, count: 600 },
{ qtype: 28, count: 300 },
{ qtype: null, count: 100 },
],
- coverage: COVERAGE,
-};
-
-const ROUTES: StatsRoutes = {
- period: "24h",
- since: SINCE,
- until: UNTIL,
routes: [
{ route: "upstream", source: "https://dns.example/dns-query", count: 500 },
{ route: "blocked", source: null, count: 250 },
@@ -83,22 +51,27 @@ const ROUTES: StatsRoutes = {
coverage: COVERAGE,
};
-/** The same shapes an hour wide, so a period change is observable in every panel. */
-const HOUR = {
- totals: { ...TOTALS, period: "1h", since: UNTIL - 3600, queries: 12, blocked: 3, clients: 2 } as StatsTotals,
- timeseries: { ...SERIES, period: "1h", since: UNTIL - 3600, bucket_seconds: 60, buckets: [] } as StatsTimeseries,
- clients: { ...CLIENTS, period: "1h", since: UNTIL - 3600, clients: [], other: [] } as StatsClients,
- types: { ...TYPES, period: "1h", since: UNTIL - 3600, types: [] } as StatsTypes,
- routes: { ...ROUTES, period: "1h", since: UNTIL - 3600, routes: [] } as StatsRoutes,
+/** The same shape an hour wide and empty, so a period change is observable. */
+const HOUR: Overview = {
+ ...OVERVIEW,
+ period: "1h",
+ since: UNTIL - 3600,
+ bucket_seconds: 60,
+ totals: { queries: 12, blocked: 3, clients: 2, avg_response_time_us: 2345 },
+ buckets: [],
+ clients: [],
+ other: [],
+ types: [],
+ routes: [],
};
let healthBody: Health;
-let failing: Set;
+let failing: boolean;
/** The registered clients, as `/api/clients` answers them. */
let registered: { ip: string; name: string; learned_name: string }[];
let coverageComplete: boolean;
-/** Paths held in flight, so a test can look at the page while one is pending. */
-let delayed: Map>;
+/** Held in flight, so a test can look at the page while the request is pending. */
+let delayed: Promise | null;
function json(payload: unknown, status = 200): Response {
return new Response(JSON.stringify(payload), { status, headers: { "content-type": "application/json" } });
@@ -110,27 +83,18 @@ function withCoverage(body: T): T {
beforeEach(() => {
healthBody = health();
- failing = new Set();
+ failing = false;
registered = [];
coverageComplete = true;
- delayed = new Map();
+ delayed = null;
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
- const hour = url.includes("period=1h");
- for (const [path, body] of [
- ["/api/stats/timeseries", hour ? HOUR.timeseries : SERIES],
- ["/api/stats/clients", hour ? HOUR.clients : CLIENTS],
- ["/api/stats/types", hour ? HOUR.types : TYPES],
- ["/api/stats/routes", hour ? HOUR.routes : ROUTES],
- ["/api/stats", hour ? HOUR.totals : TOTALS],
- ] as const) {
- if (!url.startsWith(path)) continue;
- if (failing.has(path)) return json({ error: "endpoint unavailable" }, 400);
- const held = delayed.get(path);
- if (held !== undefined) await held;
- return json(withCoverage(body));
+ if (url.startsWith("/api/overview")) {
+ if (failing) return json({ error: "endpoint unavailable" }, 400);
+ if (delayed !== null) await delayed;
+ return json(withCoverage(url.includes("period=1h") ? HOUR : OVERVIEW));
}
if (url === "/api/clients") {
return json({
@@ -215,25 +179,27 @@ test("the page builds a donut slice's colour from the entry's identity", async (
expect(swatch.getAttribute("style")).toContain(seriesColor(qtypeKey(1)));
});
-test("a slow endpoint does not hold the page back: the panels that answered render beside it", async () => {
- // Through the real route, which is the point: the loader starts the five
- // requests and awaits none of them. If it awaited, the router would hold the
- // whole page until the slowest answered and this would time out on the tiles.
+test("a request in flight leaves the heading and the picker usable behind one loading surface", async () => {
+ // Through the real route, which is the point: the loader starts the request
+ // and awaits it nowhere. If it awaited, the router would hold the whole page —
+ // heading and period picker included — until the response landed.
let release = () => {};
- delayed.set("/api/stats/routes", new Promise((resolve) => (release = resolve)));
+ delayed = new Promise((resolve) => (release = resolve));
renderApp();
- // The tiles and both charts are readable while the routes request is still
- // in flight, and the panel waiting on it says so for itself.
- await screen.findByText("1,000");
- expect(within(panel("Queries over time")).getAllByText("Blocked").length).toBeGreaterThan(0);
- expect(within(panel("Client activity over time")).getAllByText("192.0.2.30")).toHaveLength(2);
- expect(within(panel("Query types")).getAllByText("A")).toHaveLength(2);
- expect(within(panel("Upstream servers")).getByRole("status").textContent).toBe("Loading…");
+ await screen.findByRole("heading", { name: "Overview", level: 1 });
+ expect(screen.getByRole("button", { name: "1h" })).toBeTruthy();
+ // One loading state for the whole page, not one per panel.
+ const loading = await screen.findByText("Loading…");
+ expect(loading.getAttribute("role")).toBe("status");
+ expect(screen.getAllByText("Loading…")).toHaveLength(1);
+ expect(screen.queryByRole("heading", { name: "Query types" })).toBeNull();
release();
- await waitFor(() => expect(within(panel("Upstream servers")).queryByRole("status")).toBeNull());
+ delayed = null;
+ await screen.findByText("1,000");
+ expect(screen.queryByText("Loading…")).toBeNull();
});
test("a registered client is named in the chart, an unregistered one keeps its address", async () => {
@@ -392,17 +358,24 @@ test("the picker rescopes every panel and writes the period into the url", async
expect(screen.queryByText("1,000")).toBeNull();
});
-test("one failing panel keeps its own error and leaves the rest of the page standing", async () => {
- failing.add("/api/stats/routes");
+test("a failed request is one error for the whole page, stated once and retryable", async () => {
+ failing = true;
renderApp();
- await screen.findByText("1,000");
- await waitFor(() => expect(within(panel("Upstream servers")).getByText("endpoint unavailable")).toBeTruthy());
- expect(within(panel("Upstream servers")).getByRole("button", { name: "Retry" })).toBeTruthy();
- // A failed donut never blanks the charts.
- expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
- expect(screen.getByRole("img", { name: /client activity over time/i })).toBeTruthy();
+ await screen.findByText("endpoint unavailable");
+ // One statement of the failure, not one per panel: there is a single request
+ // behind every panel, so a second copy would only repeat this sentence.
+ expect(screen.getAllByText("endpoint unavailable")).toHaveLength(1);
+ expect(screen.getAllByRole("button", { name: "Retry" })).toHaveLength(1);
+ // The heading and the picker survive it, so the reader can rescope or retry.
+ expect(screen.getByRole("heading", { name: "Overview", level: 1 })).toBeTruthy();
+ expect(screen.getByRole("button", { name: "1h" })).toBeTruthy();
expect(screen.queryByText("Something went wrong")).toBeNull();
+
+ failing = false;
+ fireEvent.click(screen.getByRole("button", { name: "Retry" }));
+ await screen.findByText("1,000");
+ expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
});
test("an incomplete window states its watermark once for the whole page", async () => {
diff --git a/admin/src/features/overview/OverviewPage.tsx b/admin/src/features/overview/OverviewPage.tsx
index c13f744..4afab44 100644
--- a/admin/src/features/overview/OverviewPage.tsx
+++ b/admin/src/features/overview/OverviewPage.tsx
@@ -8,9 +8,9 @@
* The period is URL state, so a view is a link: `/overview?period=1h` opens
* exactly what the sender was reading.
*
- * Every panel reads the same window (`overviewWindow.ts`) and renders on its
- * own. A donut whose request failed shows its own error while the charts keep
- * their data, and no two panels ever describe different spans.
+ * One request feeds every panel (`overviewWindow.ts`), so the page has one
+ * loading state and one error state rather than six: there is no longer a
+ * partial answer to render, and nothing left for a panel to disagree about.
*/
import * as stylex from "@stylexjs/stylex";
@@ -18,16 +18,16 @@ import { useNavigate, useSearch } from "@tanstack/react-router";
import CoverageNotice from "@/lib/CoverageNotice";
import InlineError from "@/lib/InlineError";
import { qtypeName } from "@/features/provenance/qtype";
-import type { Period, StatsRoutes, StatsTypes } from "@/lib/types";
-import { styles as shared } from "@/ui/styles";
+import type { Overview, OverviewRouteRow, OverviewTypeRow } from "@/lib/types";
import { colors } from "@/ui/tokens.stylex";
import ClientChart from "./ClientChart";
import Donut from "./Donut";
import StatTiles from "./StatTiles";
import TimeseriesChart from "./TimeseriesChart";
import type { DonutSlice } from "./Donut";
+import { OverviewFrame, OverviewLoading } from "./OverviewFrame";
import { useOverviewWindow, type Panel } from "./overviewWindow";
-import { DEFAULT_PERIOD, PERIODS } from "./period";
+import { DEFAULT_PERIOD } from "./period";
import { qtypeKey, routeKey, seriesColor } from "./seriesColors";
/**
@@ -48,48 +48,6 @@ const ROUTE_LABELS = {
} as const;
const styles = stylex.create({
- page: {
- display: "flex",
- flexDirection: "column",
- gap: "1rem",
- },
- headingRow: {
- display: "flex",
- flexWrap: "wrap",
- alignItems: "center",
- justifyContent: "space-between",
- gap: "0.75rem",
- },
- heading: {
- fontSize: "1.5rem",
- lineHeight: "2rem",
- fontWeight: 600,
- },
- periodGroup: {
- display: "flex",
- gap: "0.25rem",
- },
- period: {
- borderStyle: "none",
- borderRadius: "0.25rem",
- paddingInline: "0.625rem",
- paddingBlock: "0.25rem",
- fontSize: "0.875rem",
- lineHeight: "1.25rem",
- },
- /** The pressed fill is heavier than `surfaceHover`, so a hover cannot mimic it. */
- periodSelected: {
- backgroundColor: {
- default: "oklch(92% 0.004 286.32)",
- "@media (prefers-color-scheme: dark)": "oklch(37% 0.013 285.805)",
- },
- color: colors.text,
- fontWeight: 500,
- },
- periodIdle: {
- backgroundColor: { default: "transparent", ":hover": colors.surfaceHover },
- color: colors.textSecondary,
- },
panel: {
borderRadius: "0.25rem",
borderWidth: 1,
@@ -110,54 +68,20 @@ const styles = stylex.create({
gap: "1rem",
gridTemplateColumns: { default: "minmax(0, 1fr)", [TWO_COLUMN]: "repeat(2, minmax(0, 1fr))" },
},
- loading: {
- fontSize: "0.875rem",
- lineHeight: "1.25rem",
- color: colors.textMuted,
- },
});
-function PeriodPicker({ period, onChange }: { period: Period; onChange: (period: Period) => void }) {
- return (
-
- {PERIODS.map((option) => (
-
- ))}
-
- );
-}
-
/**
- * One panel's three states. Loading and error are the panel's own: a failure
- * here never reaches past this box, which is what keeps a failed donut from
- * blanking the charts beside it.
+ * The page's three states. The heading and the period picker stay put through
+ * all three, so the reader can rescope or retry without waiting for anything.
*/
-function PanelBody({ panel, children }: { panel: Panel; children: (data: T) => React.ReactNode }) {
+function PageBody({ panel, children }: { panel: Panel; children: (data: Overview) => React.ReactNode }) {
if (panel.status === "error") return ;
- if (panel.status === "loading") {
- return (
-
- Loading…
-
- );
- }
+ if (panel.status === "loading") return ;
return <>{children(panel.data)}>;
}
-function typeSlices(data: StatsTypes): DonutSlice[] {
- return data.types.map((row) => ({
+function typeSlices(types: OverviewTypeRow[]): DonutSlice[] {
+ return types.map((row) => ({
key: qtypeKey(row.qtype),
label: row.qtype === null ? "Unknown" : qtypeName(row.qtype),
value: row.count,
@@ -171,8 +95,8 @@ function typeSlices(data: StatsTypes): DonutSlice[] {
* appear under two kinds and two rows can both be "Unknown". The four
* source-less kinds are their own label and need no qualifier.
*/
-function routeSlices(data: StatsRoutes): DonutSlice[] {
- return data.routes.map((row) => {
+function routeSlices(routes: OverviewRouteRow[]): DonutSlice[] {
+ return routes.map((row) => {
const named = row.route === "upstream" || row.route === "forward_zone";
return {
key: routeKey(row.route, row.source),
@@ -190,59 +114,54 @@ export default function OverviewPage() {
const overview = useOverviewWindow(period);
return (
-
+ void navigate({ search: (prev) => ({ ...prev, period: next }) })}
+ >
+
+ {(data) => (
+ <>
+
- {(totals) => }
+ {/* One notice for the page: every panel came out of this one
+ response, so a second copy would only repeat this sentence. */}
+
- {/* One notice for the page: every panel is judged against the same window,
- so a second copy would only repeat this sentence. */}
- {overview.coverage !== null && }
+
+
+ Queries over time
+
+
+
-
-
- Queries over time
-
- {(data) => }
-
+
+
+ Client activity over time
+
+
+
-
-
- Client activity over time
-
- {(data) => }
-
-
-
-
-
- Query types
-
-
- {(data) => }
-
-
-
-
- Upstream servers
-
-
- {(data) => (
-
- )}
-
-
-
-
+
+
+
+ Query types
+
+
+
+
+
+ Upstream servers
+
+
+
+
+ >
+ )}
+
+
);
}
diff --git a/admin/src/features/overview/StatTiles.tsx b/admin/src/features/overview/StatTiles.tsx
index 4853477..424b470 100644
--- a/admin/src/features/overview/StatTiles.tsx
+++ b/admin/src/features/overview/StatTiles.tsx
@@ -5,7 +5,7 @@
* typographic, so the eye ranks the figures rather than the panels, and a tile
* never implies a state it is not reporting.
*
- * The Activity links carry the bounds the **stats response** returned, not
+ * The Activity links carry the bounds the **overview response** returned, not
* bounds computed here — a client-computed window would send the reader to a
* slightly different span than the one they were just reading.
*/
@@ -13,7 +13,7 @@
import * as stylex from "@stylexjs/stylex";
import { Link } from "@tanstack/react-router";
import { formatMicros } from "@/lib/format";
-import type { StatsTotals } from "@/lib/types";
+import type { OverviewTotals } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
@@ -101,7 +101,13 @@ function Tile({
);
}
-export default function StatTiles({ stats }: { stats: StatsTotals }) {
+/** The window's totals with the bounds they were measured over. */
+export interface StatTilesData extends OverviewTotals {
+ since: number;
+ until: number;
+}
+
+export default function StatTiles({ stats }: { stats: StatTilesData }) {
const window = {
mode: "history" as const,
since: stats.since,
diff --git a/admin/src/features/overview/TimeseriesChart.test.tsx b/admin/src/features/overview/TimeseriesChart.test.tsx
index 1ed22c4..5cb21bb 100644
--- a/admin/src/features/overview/TimeseriesChart.test.tsx
+++ b/admin/src/features/overview/TimeseriesChart.test.tsx
@@ -1,24 +1,17 @@
import { fireEvent, render, screen, within } from "@testing-library/react";
import * as stylex from "@stylexjs/stylex";
import { formatTime } from "@/lib/format";
-import type { Bucket, StatsTimeseries } from "@/lib/types";
+import type { Bucket } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
-import TimeseriesChart from "./TimeseriesChart";
+import TimeseriesChart, { type TimeseriesData } from "./TimeseriesChart";
const SINCE = 1_700_000_000;
-function timeseries(buckets: Bucket[]): StatsTimeseries {
- return {
- period: "24h",
- since: SINCE,
- until: SINCE + buckets.length * 1800,
- bucket_seconds: 1800,
- coverage: { complete: true, available_since: SINCE },
- buckets,
- };
+function timeseries(buckets: Bucket[]): TimeseriesData {
+ return { since: SINCE, bucket_seconds: 1800, buckets };
}
-function counting(bucketCount: number): StatsTimeseries {
+function counting(bucketCount: number): TimeseriesData {
return timeseries(
Array.from({ length: bucketCount }, (_, i) => ({
ts: SINCE + i * 1800,
diff --git a/admin/src/features/overview/TimeseriesChart.tsx b/admin/src/features/overview/TimeseriesChart.tsx
index a6e7197..94b987b 100644
--- a/admin/src/features/overview/TimeseriesChart.tsx
+++ b/admin/src/features/overview/TimeseriesChart.tsx
@@ -2,7 +2,7 @@ import * as stylex from "@stylexjs/stylex";
import { Group } from "@visx/group";
import { BarStack } from "@visx/shape";
import { formatTime } from "@/lib/format";
-import type { Bucket, StatsTimeseries } from "@/lib/types";
+import type { Bucket } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import {
@@ -103,7 +103,17 @@ function tooltipOf(column: Column): TooltipContent {
};
}
-export default function TimeseriesChart({ data }: { data: StatsTimeseries }) {
+/**
+ * The slice of the Overview body this chart draws. Declared here rather than
+ * taken whole, so what the chart reads is stated where it is read.
+ */
+export interface TimeseriesData {
+ since: number;
+ bucket_seconds: number;
+ buckets: Bucket[];
+}
+
+export default function TimeseriesChart({ data }: { data: TimeseriesData }) {
const [containerRef, width] = useMeasuredWidth();
// A hover survives a re-render only while it still names the same bucket at
// the same place: a poll that rolls the window, or a resize, retires it.
diff --git a/admin/src/features/overview/overviewWindow.test.tsx b/admin/src/features/overview/overviewWindow.test.tsx
index f8d8f55..9c353d0 100644
--- a/admin/src/features/overview/overviewWindow.test.tsx
+++ b/admin/src/features/overview/overviewWindow.test.tsx
@@ -1,74 +1,41 @@
/**
- * Window coherence across the five Overview requests, migrated from the
- * two-request `activityWindow` this replaces. Every behaviour that hook pinned
- * is pinned here — the identity, the one retry per mismatch episode, the
- * terminal error, the discarded previous-period pair and the stale completion
- * that must not speak — now over five endpoints and with the watermark in the
- * identity, plus the per-panel isolation the layout added.
+ * The hook over the single `/api/overview` request.
+ *
+ * The five-endpoint build reconciled five window identities here — the retry per
+ * mismatch episode, the terminal "different window" error, the orphaned stale
+ * completion. One request cannot disagree with itself, so those behaviours have
+ * no subject left and are gone rather than ported. What survived the collapse is
+ * pinned below: the three states, and the one rule a single request still does
+ * not settle — that a `keepPreviousData` body from the period the reader left
+ * must never render under the new period's label.
*/
import { render, screen, waitFor } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { createQueryClient } from "@/lib/queryClient";
-import type { Coverage, Period } from "@/lib/types";
-import {
- newerWindow,
- sameWindow,
- useOverviewWindow,
- windowIdOf,
- OVERVIEW_ENDPOINTS,
- type OverviewEndpoint,
-} from "./overviewWindow";
+import type { Period } from "@/lib/types";
+import { useOverviewWindow } from "./overviewWindow";
const SINCE = Date.UTC(2026, 0, 1, 0, 0) / 1000;
const UNTIL = Date.UTC(2026, 0, 2, 0, 0) / 1000;
-const COVERAGE: Coverage = { complete: true, available_since: SINCE };
-/** Where each endpoint's body currently ends, and what watermark it admits. */
-interface Bounds {
- until: number;
- availableSince: number;
-}
+let failing: boolean;
+let calls: number;
-const PATHS: Record = {
- totals: "/api/stats?period=",
- timeseries: "/api/stats/timeseries?period=",
- clients: "/api/stats/clients?period=",
- types: "/api/stats/types?period=",
- routes: "/api/stats/routes?period=",
-};
-
-let bounds: Record;
-let failing: Set;
-let calls: Record;
-/** Endpoints that answer for the page's window from their second call onward. */
-let catchUp: Set;
-/** Held to keep one answer in flight while the test moves the page on. */
-let hold: { promise: Promise; release: () => void } | null;
-
-function endpointOf(url: string): OverviewEndpoint | null {
- // Longest prefix first: `/api/stats?` and `/api/stats/…` share a stem.
- for (const endpoint of ["timeseries", "clients", "types", "routes", "totals"] as const) {
- if (url.startsWith(PATHS[endpoint])) return endpoint;
- }
- return null;
-}
-
-function body(endpoint: OverviewEndpoint, period: Period): unknown {
- const { until, availableSince } = bounds[endpoint];
- const shared = { period, since: SINCE, until, coverage: { ...COVERAGE, available_since: availableSince } };
- switch (endpoint) {
- case "totals":
- return { ...shared, queries: 10, blocked: 2, clients: 1, avg_response_time_us: 1000 };
- case "timeseries":
- return { ...shared, bucket_seconds: 3600, buckets: [] };
- case "clients":
- return { ...shared, bucket_seconds: 3600, clients: [], other: [] };
- case "types":
- return { ...shared, types: [] };
- case "routes":
- return { ...shared, routes: [] };
- }
+function body(period: Period): unknown {
+ return {
+ period,
+ since: SINCE,
+ until: UNTIL,
+ bucket_seconds: 1800,
+ totals: { queries: 10, blocked: 2, clients: 1, avg_response_time_us: 1000 },
+ buckets: [],
+ clients: [],
+ other: [],
+ types: [],
+ routes: [],
+ coverage: { complete: true, available_since: SINCE },
+ };
}
function json(payload: unknown, status = 200): Response {
@@ -76,55 +43,36 @@ function json(payload: unknown, status = 200): Response {
}
beforeEach(() => {
- bounds = {
- totals: { until: UNTIL, availableSince: SINCE },
- timeseries: { until: UNTIL, availableSince: SINCE },
- clients: { until: UNTIL, availableSince: SINCE },
- types: { until: UNTIL, availableSince: SINCE },
- routes: { until: UNTIL, availableSince: SINCE },
- };
- failing = new Set();
- catchUp = new Set();
- hold = null;
- calls = { totals: 0, timeseries: 0, clients: 0, types: 0, routes: 0 };
+ failing = false;
+ calls = 0;
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
- const endpoint = endpointOf(url);
- if (endpoint === null) return json({ error: "not stubbed" }, 404);
- calls[endpoint] += 1;
- if (failing.has(endpoint)) return json({ error: "endpoint unavailable" }, 400);
- if (catchUp.has(endpoint) && calls[endpoint] >= 2)
- bounds[endpoint] = { until: UNTIL, availableSince: SINCE };
+ if (!url.startsWith("/api/overview")) return json({ error: "not stubbed" }, 404);
+ calls += 1;
+ if (failing) return json({ error: "endpoint unavailable" }, 400);
const period = (new URLSearchParams(url.split("?")[1]).get("period") ?? "24h") as Period;
- // Built before the wait, so a held answer carries what its own request
- // would have returned rather than what the page has moved on to.
- const payload = json(body(endpoint, period));
- if (hold !== null && endpoint === "routes" && calls.routes === 2) await hold.promise;
- return payload;
+ return json(body(period));
}),
);
});
afterEach(() => vi.unstubAllGlobals());
+/** The last retry the hook handed out, so a test can spend it. */
+let lastRetry: () => void;
+
function Probe({ period }: { period: Period }) {
- const overview = useOverviewWindow(period);
- return (
-
;
}
function renderProbe(period: Period = "24h") {
@@ -144,149 +92,36 @@ function renderProbe(period: Period = "24h") {
};
}
-function line(endpoint: OverviewEndpoint): string {
- const item = screen.getAllByRole("listitem").find((element) => element.textContent?.startsWith(`${endpoint}:`));
- if (item === undefined) throw new Error(`no probe line for ${endpoint}`);
- return item.textContent ?? "";
+function line(): string {
+ return screen.getByRole("paragraph").textContent ?? "";
}
-test("the window identity is the period, both bounds and the watermark together", () => {
- const base = { period: "24h" as const, since: SINCE, until: UNTIL, availableSince: SINCE };
- expect(sameWindow(base, { ...base })).toBe(true);
- expect(sameWindow(base, { ...base, period: "1h" })).toBe(false);
- expect(sameWindow(base, { ...base, since: SINCE - 1 })).toBe(false);
- expect(sameWindow(base, { ...base, until: UNTIL + 1 })).toBe(false);
- // The bounds agree and the answers still describe different windows: a prune
- // between the two requests moved what the same span can be answered for.
- expect(sameWindow(base, { ...base, availableSince: SINCE + 60 })).toBe(false);
-});
-
-test("the newer until wins, and for equal bounds the later watermark does", () => {
- const base = { period: "24h" as const, since: SINCE, until: UNTIL, availableSince: SINCE };
- expect(newerWindow(base, { ...base, until: UNTIL + 60 }).until).toBe(UNTIL + 60);
- expect(newerWindow({ ...base, until: UNTIL + 60 }, base).until).toBe(UNTIL + 60);
- expect(newerWindow(base, { ...base, availableSince: SINCE + 60 }).availableSince).toBe(SINCE + 60);
- // A newer watermark does not outrank an older window's later bound.
- expect(newerWindow({ ...base, until: UNTIL + 60 }, { ...base, availableSince: SINCE + 60 }).until).toBe(UNTIL + 60);
-});
-
-test("windowIdOf reads the four fields off any of the five bodies", () => {
- expect(windowIdOf({ period: "7d", since: 1, until: 2, coverage: { complete: false, available_since: 3 } })).toEqual(
- {
- period: "7d",
- since: 1,
- until: 2,
- availableSince: 3,
- },
- );
-});
-
-test("five responses for one window render as five ready panels", async () => {
+test("the page is loading until the body for the selected period arrives", async () => {
renderProbe();
- await waitFor(() => expect(line("totals")).toContain("ready"));
- for (const endpoint of OVERVIEW_ENDPOINTS) {
- expect(line(endpoint)).toBe(`${endpoint}:ready:24h@${UNTIL}/${SINCE}`);
- }
+ expect(line()).toBe("loading:");
+ await waitFor(() => expect(line()).toBe(`ready:24h@${UNTIL}`));
});
-test("one endpoint behind a bucket boundary is refetched once and then agrees", async () => {
- // Behind on its first answer, caught up by the time the hook asks again.
- bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
- catchUp.add("routes");
+test("a failed request is one error for the whole page, with a retry that refetches", async () => {
+ failing = true;
renderProbe();
- await waitFor(() => expect(line("routes")).toContain("ready"));
- expect(calls.routes).toBe(2);
- expect(calls.totals).toBe(1);
-});
+ await waitFor(() => expect(line()).toBe("error:endpoint unavailable"));
+ const spent = calls;
-test("a laggard that stays behind fails its own panel and leaves the rest rendering", async () => {
- bounds.types = { until: UNTIL - 3600, availableSince: SINCE };
- renderProbe();
-
- await waitFor(() => expect(line("types")).toContain("error"));
- expect(line("types")).toContain("different window");
- // One retry, not a loop.
- expect(calls.types).toBe(2);
- for (const endpoint of ["totals", "timeseries", "clients", "routes"] as const) {
- expect(line(endpoint)).toContain("ready");
- }
-});
-
-test("a failed request degrades its own panel; the charts keep the window", async () => {
- failing.add("routes");
- renderProbe();
-
- await waitFor(() => expect(line("routes")).toContain("error"));
- expect(line("routes")).toContain("endpoint unavailable");
- expect(line("timeseries")).toContain("ready");
- expect(line("totals")).toContain("ready");
-});
-
-test("a watermark that advanced mid-page is a mismatch, not a mixed window", async () => {
- // Same bounds, later watermark: retention pruned between the two responses.
- bounds.clients = { until: UNTIL, availableSince: SINCE + 600 };
- renderProbe();
-
- await waitFor(() => expect(line("clients")).toContain(`/${SINCE + 600}`));
- // The page adopts the later watermark, so the four older answers are the
- // laggards and each gets its one retry rather than rendering beside it.
- await waitFor(() => expect(calls.totals).toBe(2));
- expect(line("clients")).toContain("ready");
+ failing = false;
+ lastRetry();
+ await waitFor(() => expect(line()).toBe(`ready:24h@${UNTIL}`));
+ expect(calls).toBeGreaterThan(spent);
});
test("a retained previous-period body never renders under the new period's label", async () => {
const { rerenderWith } = renderProbe("24h");
- await waitFor(() => expect(line("totals")).toBe(`totals:ready:24h@${UNTIL}/${SINCE}`));
+ await waitFor(() => expect(line()).toBe(`ready:24h@${UNTIL}`));
rerenderWith("1h");
- // Whatever `keepPreviousData` is holding, no panel may claim it answers 1h.
- await waitFor(() => expect(line("totals")).toBe(`totals:ready:1h@${UNTIL}/${SINCE}`));
- for (const endpoint of OVERVIEW_ENDPOINTS) expect(line(endpoint)).toContain("1h@");
-});
-
-test("a period change buys the new window its own retry", async () => {
- bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
- const { rerenderWith } = renderProbe("24h");
- await waitFor(() => expect(line("routes")).toContain("error"));
- const spent = calls.routes;
-
- rerenderWith("1h");
- // The mismatch persists under the new period, and the episode key changed
- // with it: the retry the abandoned period spent is not the new one's.
- await waitFor(() => expect(calls.routes).toBeGreaterThan(spent));
- await waitFor(() => expect(line("routes")).toContain("error"));
-});
-
-test("a retry in flight when the period changes cannot spend the window's retry later", async () => {
- // The stale completion the tokens exist to orphan: routes lags under 24h, the
- // hook issues its one retry, and the reader picks 1h before that retry lands.
- bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
- let release = () => {};
- hold = { promise: new Promise((resolve) => (release = resolve)), release: () => release() };
-
- const { rerenderWith } = renderProbe("24h");
- await waitFor(() => expect(calls.routes).toBe(2));
-
- bounds.routes = { until: UNTIL, availableSince: SINCE };
- rerenderWith("1h");
- await waitFor(() => expect(line("routes")).toContain("1h@"));
-
- // The abandoned retry lands now, under a period it was never asked for.
- hold.release();
- hold = null;
- await waitFor(() => expect(line("routes")).toContain("ready"));
-
- // Back to the window it was issued for, still lagging. The stale completion
- // must not have marked this episode spent: the panel gets a real retry before
- // it is allowed to reach the terminal error.
- bounds.routes = { until: UNTIL - 3600, availableSince: SINCE };
- rerenderWith("24h");
-
- // The cached lagging body is there to render immediately, and the panel must
- // not state the terminal error off it: that error means "retried and still
- // behind", and this visit has not retried anything yet. An abandoned
- // completion recording the episode as spent is what would produce it here.
- expect(line("routes")).toContain("loading");
- await waitFor(() => expect(line("routes")).toContain("different window"));
+ // `keepPreviousData` is holding the 24h body. It is a complete answer and
+ // still the wrong one to draw under "1h", so the page waits.
+ expect(line()).toBe("loading:");
+ await waitFor(() => expect(line()).toBe(`ready:1h@${UNTIL}`));
});
diff --git a/admin/src/features/overview/overviewWindow.ts b/admin/src/features/overview/overviewWindow.ts
index be16424..af89900 100644
--- a/admin/src/features/overview/overviewWindow.ts
+++ b/admin/src/features/overview/overviewWindow.ts
@@ -1,249 +1,34 @@
/**
- * One period, five requests, one window.
+ * One period, one request, one window.
*
- * Totals, the timeline, the per-client series and the two breakdowns are
- * separate calls, so a refresh that straddles a bucket boundary — or a retention
- * pass that advances the watermark mid-page — can answer them for different
- * windows. Rendering them side by side anyway would put a headline count above
- * charts of a different span, a mixed page that looks exactly like a real one.
+ * The five per-panel endpoints this replaces could each answer for a different
+ * span, so the page had to reconcile five window identities, retry the laggards
+ * and fail the ones that stayed behind. `GET /api/overview` answers every panel
+ * out of a single read transaction: the totals, both timelines and both
+ * breakdowns describe the same span and the same database state by construction,
+ * and none of that reconciliation has anything left to reconcile.
*
- * This is **window** coherence, not data-snapshot coherence: matching bounds
- * cannot prove a common database state, and live inserts between requests may
- * still shift counts slightly between panels. What it does guarantee is that no
- * two panels ever describe different spans.
- *
- * Rendering is per panel. A panel whose request is still in flight shows its own
- * loading state and a panel whose request failed shows its own error, while the
- * panels that match the window keep rendering — a failed donut never blanks the
- * charts.
+ * What remains is the one rule a single request does not settle by itself.
+ * `keepPreviousData` holds the body of the period the reader just left — a
+ * complete, self-consistent answer, and still the wrong one to draw under the
+ * new label — so a body is a member of this window only while its own `period`
+ * is the selected one. Until then the page is loading.
*/
-import { useCallback, useEffect, useRef, useState } from "react";
-import { keepPreviousData, useQuery, type UseQueryResult } from "@tanstack/react-query";
-import { statsClientsQuery, statsQuery, statsRoutesQuery, statsTypesQuery, timeseriesQuery } from "@/lib/queries";
-import type {
- Coverage,
- Period,
- StatsClients,
- StatsRoutes,
- StatsTimeseries,
- StatsTotals,
- StatsTypes,
-} from "@/lib/types";
-
-export const OVERVIEW_ENDPOINTS = ["totals", "timeseries", "clients", "types", "routes"] as const;
-export type OverviewEndpoint = (typeof OVERVIEW_ENDPOINTS)[number];
-
-interface EndpointBodies {
- totals: StatsTotals;
- timeseries: StatsTimeseries;
- clients: StatsClients;
- types: StatsTypes;
- routes: StatsRoutes;
-}
-
-/**
- * What makes two responses the same window. `available_since` joins the bounds
- * because retention advancing between requests changes what the same `[since,
- * until)` can answer for, and mixing a pre-prune answer with a post-prune one is
- * the failure the bounds alone would not catch.
- */
-export interface WindowId {
- period: Period;
- since: number;
- until: number;
- availableSince: number;
-}
-
-/** The four fields every window-bounded stats body carries. */
-interface Bounded {
- period: Period;
- since: number;
- until: number;
- coverage: Coverage;
-}
-
-export function windowIdOf(body: Bounded): WindowId {
- return {
- period: body.period,
- since: body.since,
- until: body.until,
- availableSince: body.coverage.available_since,
- };
-}
-
-export function sameWindow(a: WindowId, b: WindowId): boolean {
- return a.period === b.period && a.since === b.since && a.until === b.until && a.availableSince === b.availableSince;
-}
-
-/**
- * Which of two candidate windows the page adopts: the one that reaches further
- * forward in time, and for identical bounds the one that admits the later
- * watermark. Both rules pick the answer a laggard has to catch up to.
- */
-export function newerWindow(a: WindowId, b: WindowId): WindowId {
- if (b.until !== a.until) return b.until > a.until ? b : a;
- return b.availableSince > a.availableSince ? b : a;
-}
-
-function keyOf(id: WindowId): string {
- return `${id.period}|${id.since}|${id.until}|${id.availableSince}`;
-}
+import { useCallback } from "react";
+import { keepPreviousData, useQuery } from "@tanstack/react-query";
+import { overviewQuery } from "@/lib/queries";
+import type { Overview, Period } from "@/lib/types";
export type Panel =
{ status: "loading" } | { status: "error"; error: unknown; retry: () => void } | { status: "ready"; data: T };
-export interface OverviewWindow {
- /** Null until one response for the selected period has arrived. */
- window: WindowId | null;
- /** The adopted window's watermark, for the page's single coverage notice. */
- coverage: Coverage | null;
- totals: Panel;
- timeseries: Panel;
- clients: Panel;
- types: Panel;
- routes: Panel;
-}
-
-/**
- * A laggard that stayed behind after its one retry. Not an `ApiError`: nothing
- * failed, the endpoint simply never caught up, and `InlineError` renders the
- * message verbatim.
- */
-export const MISMATCH = new Error("This panel is for a different window than the rest of the page. Try again.");
-
-export function useOverviewWindow(period: Period): OverviewWindow {
- const queries: { [K in OverviewEndpoint]: UseQueryResult } = {
- totals: useQuery({ ...statsQuery(period), placeholderData: keepPreviousData }),
- timeseries: useQuery({ ...timeseriesQuery(period), placeholderData: keepPreviousData }),
- clients: useQuery({ ...statsClientsQuery(period), placeholderData: keepPreviousData }),
- types: useQuery({ ...statsTypesQuery(period), placeholderData: keepPreviousData }),
- routes: useQuery({ ...statsRoutesQuery(period), placeholderData: keepPreviousData }),
- };
-
- // A `keepPreviousData` placeholder for the period just left is a complete,
- // self-consistent body — and still the wrong one to show under the new label,
- // so it is neither a candidate for the window nor a member of it.
- const answers = new Map();
- for (const endpoint of OVERVIEW_ENDPOINTS) {
- const data = queries[endpoint].data;
- if (data !== undefined && data.period === period) answers.set(endpoint, windowIdOf(data));
- }
-
- let window: WindowId | null = null;
- for (const id of answers.values()) window = window === null ? id : newerWindow(window, id);
-
- // The effect below runs on what the responses say, not on how many times they
- // arrived: a poll that returns byte-identical data must not restart the retry
- // bookkeeping. The refetchers ride a ref for the same reason — TanStack hands
- // back a fresh function identity on some renders, and depending on it would
- // re-enter the effect with nothing changed.
- const answersKey = OVERVIEW_ENDPOINTS.map((endpoint) => {
- const id = answers.get(endpoint);
- return id === undefined ? "" : keyOf(id);
- }).join("~");
- const latest = useRef({ answers, refetch: queries });
- latest.current = { answers, refetch: queries };
-
- // Which mismatch episode each endpoint has already spent its retry on, keyed
- // by endpoint and window identity so a new window buys a new attempt.
- const retriedFor = useRef(new Map());
- // Which retry each endpoint is waiting on. Per endpoint, because one shared
- // counter would let a second endpoint's retry silence the first's completion;
- // bumped on every retry issued, so a completion from a window or a period the
- // page has left can neither clear an error the current one reached nor spend
- // the current window's one retry.
- const tokens = useRef(new Map());
- // State, not a ref: a retry that returns byte-identical data changes nothing
- // else a render could see, and the panel still has to reach its error.
- const [landedFor, setLandedFor] = useState(new Map());
-
- // Leaving a period ends every episode it opened. A retry issued for the old
- // period can still be in flight, and without this its completion would land
- // under the new one holding a token the map still honours: it would record an
- // episode as spent, so a return to that window would reach the terminal error
- // without the retry that error is supposed to follow. Bumping the tokens
- // orphans those answers, and the cleared maps let the new window start clean.
- const [lastPeriod, setLastPeriod] = useState(period);
- if (lastPeriod !== period) {
- setLastPeriod(period);
- for (const endpoint of OVERVIEW_ENDPOINTS) {
- tokens.current.set(endpoint, (tokens.current.get(endpoint) ?? 0) + 1);
- }
- retriedFor.current.clear();
- setLandedFor(new Map());
- }
-
- const windowKey = window === null ? null : keyOf(window);
-
- useEffect(() => {
- if (windowKey === null) return;
- for (const [endpoint, identity] of latest.current.answers) {
- if (keyOf(identity) === windowKey) {
- retriedFor.current.delete(endpoint);
- continue;
- }
- const episode = `${endpoint}|${windowKey}`;
- if (retriedFor.current.get(endpoint) === episode) continue;
- retriedFor.current.set(endpoint, episode);
- const token = (tokens.current.get(endpoint) ?? 0) + 1;
- tokens.current.set(endpoint, token);
- const landed = () => {
- if (tokens.current.get(endpoint) !== token) return;
- setLandedFor((previous) => new Map(previous).set(endpoint, episode));
- };
- void latest.current.refetch[endpoint].refetch().then(landed, landed);
- }
- }, [answersKey, windowKey]);
-
- const retry = useCallback((endpoint: OverviewEndpoint) => {
- retriedFor.current.delete(endpoint);
- tokens.current.set(endpoint, (tokens.current.get(endpoint) ?? 0) + 1);
- setLandedFor((previous) => {
- const next = new Map(previous);
- next.delete(endpoint);
- return next;
- });
- void latest.current.refetch[endpoint].refetch();
- }, []);
-
- function panelOf(endpoint: K): Panel {
- const query = queries[endpoint];
- const onRetry = () => retry(endpoint);
- if (query.isError) return { status: "error", error: query.error, retry: onRetry };
- const data = query.data;
- if (
- data !== undefined &&
- windowKey !== null &&
- data.period === period &&
- keyOf(windowIdOf(data)) === windowKey
- ) {
- return { status: "ready", data };
- }
- if (windowKey !== null && landedFor.get(endpoint) === `${endpoint}|${windowKey}`) {
- return { status: "error", error: MISMATCH, retry: onRetry };
- }
- return { status: "loading" };
- }
-
- const panels = {
- totals: panelOf("totals"),
- timeseries: panelOf("timeseries"),
- clients: panelOf("clients"),
- types: panelOf("types"),
- routes: panelOf("routes"),
- };
-
- // The notice describes the window, so any member of it can supply the
- // watermark: whichever panel arrived says the same thing about coverage.
- let coverage: Coverage | null = null;
- for (const endpoint of OVERVIEW_ENDPOINTS) {
- const panel = panels[endpoint];
- if (panel.status === "ready") {
- coverage = panel.data.coverage;
- break;
- }
- }
-
- return { window, coverage, ...panels };
+export function useOverviewWindow(period: Period): Panel {
+ const query = useQuery({ ...overviewQuery(period), placeholderData: keepPreviousData });
+ const { refetch } = query;
+ const retry = useCallback(() => void refetch(), [refetch]);
+
+ if (query.isError) return { status: "error", error: query.error, retry };
+ if (query.data !== undefined && query.data.period === period) return { status: "ready", data: query.data };
+ return { status: "loading" };
}
diff --git a/admin/src/lib/api.test.ts b/admin/src/lib/api.test.ts
index 8ed6f49..6e241b3 100644
--- a/admin/src/lib/api.test.ts
+++ b/admin/src/lib/api.test.ts
@@ -1,4 +1,4 @@
-import { ApiError, deleteGroup, getQueries, getStats, listGroups, login, putGroupSources } from "@/lib/api";
+import { ApiError, deleteGroup, getOverview, getQueries, listGroups, login, putGroupSources } from "@/lib/api";
function jsonResponse(payload: unknown, status = 200, headers: Record = {}): Response {
return new Response(JSON.stringify(payload), {
@@ -50,7 +50,7 @@ test("falls back to a status message on a non-JSON error body", async () => {
test("parses Retry-After on 429", async () => {
fetchMock.mockResolvedValue(jsonResponse({ error: "rate limited" }, 429, { "Retry-After": "17" }));
- const failure = await getStats("1h").catch((e: unknown) => e);
+ const failure = await getOverview("1h").catch((e: unknown) => e);
expect(failure).toBeInstanceOf(ApiError);
expect((failure as ApiError).status).toBe(429);
expect((failure as ApiError).retryAfter).toBe(17);
@@ -58,7 +58,7 @@ test("parses Retry-After on 429", async () => {
test("ignores a malformed Retry-After header", async () => {
fetchMock.mockResolvedValue(jsonResponse({ error: "rate limited" }, 429, { "Retry-After": "soon" }));
- const failure = await getStats().catch((e: unknown) => e);
+ const failure = await getOverview().catch((e: unknown) => e);
expect((failure as ApiError).retryAfter).toBeUndefined();
});
diff --git a/admin/src/lib/api.ts b/admin/src/lib/api.ts
index d1d9fb2..164fea7 100644
--- a/admin/src/lib/api.ts
+++ b/admin/src/lib/api.ts
@@ -23,6 +23,7 @@ import type {
LoginResponse,
LogoutResponse,
LookupResult,
+ Overview,
PausePost,
PauseState,
Period,
@@ -35,11 +36,6 @@ import type {
SettingsEnvelope,
SettingsPatch,
SourceStatus,
- StatsClients,
- StatsRoutes,
- StatsTimeseries,
- StatsTotals,
- StatsTypes,
Upstream,
UpstreamEcho,
UpstreamInput,
@@ -112,7 +108,7 @@ export const login = (body: LoginRequest): Promise =>
request("/api/auth/login", { method: "POST", body });
export const logout = (): Promise => request("/api/auth/logout", { method: "POST", body: {} });
-// Query log + stats
+// Query log + overview
export const getQueries = (filter: QueriesFilter = {}): Promise =>
request(`/api/queries${qs({ ...filter })}`);
@@ -123,13 +119,8 @@ export const getQueryDetail = (id: number): Promise => request(`/ap
/** `EventSource` URL for the live stream; not a fetch route. */
export const liveQueriesUrl = "/api/queries/live";
-export const getStats = (period?: Period): Promise => request(`/api/stats${qs({ period })}`);
-export const getStatsTimeseries = (period?: Period): Promise =>
- request(`/api/stats/timeseries${qs({ period })}`);
-export const getStatsTypes = (period?: Period): Promise => request(`/api/stats/types${qs({ period })}`);
-export const getStatsRoutes = (period?: Period): Promise => request(`/api/stats/routes${qs({ period })}`);
-export const getStatsClients = (period?: Period): Promise =>
- request(`/api/stats/clients${qs({ period })}`);
+/** Every Overview panel for one window, from one read transaction. */
+export const getOverview = (period?: Period): Promise => request(`/api/overview${qs({ period })}`);
export const getLookup = (domain: string, groupId?: number): Promise =>
request(`/api/lookup${qs({ domain, group_id: groupId })}`);
diff --git a/admin/src/lib/contractSamples.gen.ts b/admin/src/lib/contractSamples.gen.ts
index ebeca98..d16cc8b 100644
--- a/admin/src/lib/contractSamples.gen.ts
+++ b/admin/src/lib/contractSamples.gen.ts
@@ -29,6 +29,7 @@ import type {
LoginResponse,
LogoutResponse,
LookupResult,
+ Overview,
PauseState,
QueriesPage,
QueryDetail,
@@ -36,11 +37,6 @@ import type {
RuleEcho,
SettingsEnvelope,
SourceStatus,
- StatsClients,
- StatsRoutes,
- StatsTimeseries,
- StatsTotals,
- StatsTypes,
Upstream,
UpstreamEcho,
Version,
@@ -588,39 +584,6 @@ export const sample_get_query_detail: QueryDetail = {
},
};
-export const sample_get_stats: StatsTotals = {
- avg_response_time_us: null,
- blocked: 0,
- clients: 0,
- coverage: {
- available_since: 0,
- complete: true,
- },
- period: "1h",
- queries: 0,
- since: 0,
- until: 0,
-};
-
-export const sample_get_stats_timeseries: StatsTimeseries = {
- bucket_seconds: 0,
- buckets: [
- {
- blocked: 0,
- cached: 0,
- queries: 0,
- ts: 0,
- },
- ],
- coverage: {
- available_since: 0,
- complete: true,
- },
- period: "1h",
- since: 0,
- until: 0,
-};
-
export const sample_get_pause: PauseState = {
paused: false,
until: null,
@@ -837,31 +800,35 @@ export const sample_error_not_found: ErrorEnvelope = {
error: "not found",
};
-export const sample_get_stats_types: StatsTypes = {
- coverage: {
- available_since: 0,
- complete: true,
- },
- period: "1h",
- since: 0,
- types: [
+export const sample_get_overview: Overview = {
+ bucket_seconds: 0,
+ buckets: [
{
- count: 0,
- qtype: 0,
+ blocked: 0,
+ cached: 0,
+ queries: 0,
+ ts: 0,
},
+ ],
+ clients: [
{
- count: 0,
- qtype: null,
+ buckets: [0],
+ client: "192.0.2.30",
+ },
+ {
+ buckets: [0],
+ client: "192.0.2.31",
+ },
+ {
+ buckets: [0],
+ client: "192.0.2.32",
},
],
- until: 0,
-};
-
-export const sample_get_stats_routes: StatsRoutes = {
coverage: {
available_since: 0,
complete: true,
},
+ other: [0],
period: "1h",
routes: [
{
@@ -906,32 +873,22 @@ export const sample_get_stats_routes: StatsRoutes = {
},
],
since: 0,
- until: 0,
-};
-
-export const sample_get_stats_clients: StatsClients = {
- bucket_seconds: 0,
- clients: [
+ totals: {
+ avg_response_time_us: 0,
+ blocked: 0,
+ clients: 0,
+ queries: 0,
+ },
+ types: [
{
- buckets: [0],
- client: "192.0.2.30",
+ count: 0,
+ qtype: 0,
},
{
- buckets: [0],
- client: "192.0.2.31",
- },
- {
- buckets: [0],
- client: "192.0.2.32",
+ count: 0,
+ qtype: null,
},
],
- coverage: {
- available_since: 0,
- complete: true,
- },
- other: [0],
- period: "1h",
- since: 0,
until: 0,
};
diff --git a/admin/src/lib/queries.ts b/admin/src/lib/queries.ts
index af1c74f..ddf507f 100644
--- a/admin/src/lib/queries.ts
+++ b/admin/src/lib/queries.ts
@@ -21,11 +21,7 @@ import type {
export const queryKeys = {
health: ["health"] as const,
version: ["version"] as const,
- stats: (period: Period) => ["stats", period] as const,
- timeseries: (period: Period) => ["stats", "timeseries", period] as const,
- statsTypes: (period: Period) => ["stats", "types", period] as const,
- statsRoutes: (period: Period) => ["stats", "routes", period] as const,
- statsClients: (period: Period) => ["stats", "clients", period] as const,
+ overview: (period: Period) => ["overview", period] as const,
queriesInfinite: (filter: QueriesFilter) => ["queries", "infinite", filter] as const,
queryDetail: (id: number) => ["queries", "detail", id] as const,
diagnosticsInfinite: (filter: DiagnosticsFilter) => ["diagnostics", "infinite", filter] as const,
@@ -54,34 +50,10 @@ export const healthQuery = () =>
export const versionQuery = () =>
queryOptions({ queryKey: queryKeys.version, queryFn: api.getVersion, staleTime: Infinity });
-export const statsQuery = (period: Period = "24h") =>
- queryOptions({ queryKey: queryKeys.stats(period), queryFn: () => api.getStats(period), refetchInterval: 30_000 });
-
-export const timeseriesQuery = (period: Period = "24h") =>
+export const overviewQuery = (period: Period = "24h") =>
queryOptions({
- queryKey: queryKeys.timeseries(period),
- queryFn: () => api.getStatsTimeseries(period),
- refetchInterval: 30_000,
- });
-
-export const statsTypesQuery = (period: Period = "24h") =>
- queryOptions({
- queryKey: queryKeys.statsTypes(period),
- queryFn: () => api.getStatsTypes(period),
- refetchInterval: 30_000,
- });
-
-export const statsRoutesQuery = (period: Period = "24h") =>
- queryOptions({
- queryKey: queryKeys.statsRoutes(period),
- queryFn: () => api.getStatsRoutes(period),
- refetchInterval: 30_000,
- });
-
-export const statsClientsQuery = (period: Period = "24h") =>
- queryOptions({
- queryKey: queryKeys.statsClients(period),
- queryFn: () => api.getStatsClients(period),
+ queryKey: queryKeys.overview(period),
+ queryFn: () => api.getOverview(period),
refetchInterval: 30_000,
});
diff --git a/admin/src/lib/types.ts b/admin/src/lib/types.ts
index 7a552d5..0fe1f26 100644
--- a/admin/src/lib/types.ts
+++ b/admin/src/lib/types.ts
@@ -291,15 +291,13 @@ export interface DiagnosticsFilter {
before?: number;
}
-export interface StatsTotals {
- period: Period;
- since: number;
- until: number;
+/** The window's four headline numbers. */
+export interface OverviewTotals {
queries: number;
blocked: number;
+ /** Distinct clients seen in the window, not a sum of per-bucket counts. */
clients: number;
avg_response_time_us: number | null;
- coverage: Coverage;
}
export interface Bucket {
@@ -309,67 +307,53 @@ export interface Bucket {
cached: number;
}
-export interface StatsTimeseries {
- period: Period;
- since: number;
- until: number;
- bucket_seconds: number;
- buckets: Bucket[];
- coverage: Coverage;
-}
-
/**
* One DNS type's share of the window. `qtype` is the numeric code as logged:
* naming it is the admin's job (`features/provenance/qtype.ts`), and a row whose
* type was never recorded keeps its own `null` group rather than disappearing.
*/
-export interface StatsTypeRow {
+export interface OverviewTypeRow {
qtype: number | null;
count: number;
}
-export interface StatsTypes {
- period: Period;
- since: number;
- until: number;
- types: StatsTypeRow[];
- coverage: Coverage;
-}
-
/**
* How the window's queries were answered. `source` names the answering upstream
* on `upstream` rows and the zone on `forward_zone` rows; every other route kind
* carries null, as does a row whose identity was not recorded.
*/
-export interface StatsRouteRow {
+export interface OverviewRouteRow {
route: RouteKind;
source: string | null;
count: number;
}
-export interface StatsRoutes {
- period: Period;
- since: number;
- until: number;
- routes: StatsRouteRow[];
- coverage: Coverage;
-}
-
-/** One client's per-bucket counts, aligned to `StatsTimeseries`'s buckets. */
-export interface StatsClientSeries {
+/** One client's per-bucket counts, aligned to `Overview.buckets`. */
+export interface OverviewClientSeries {
client: string;
buckets: number[];
}
-export interface StatsClients {
+/**
+ * Everything the Overview page draws, for one window, from one request. The
+ * server answers all six panels out of a single read transaction, so the
+ * headline totals, the two timelines and the two breakdowns are guaranteed to
+ * describe the same span *and* the same database state — a coherence the five
+ * endpoints this replaces could not offer.
+ */
+export interface Overview {
period: Period;
since: number;
until: number;
bucket_seconds: number;
+ totals: OverviewTotals;
+ buckets: Bucket[];
/** The eight busiest clients in the window, ranked by total count. */
- clients: StatsClientSeries[];
+ clients: OverviewClientSeries[];
/** Everything outside the top eight. Always present and always bucket-count-sized. */
other: number[];
+ types: OverviewTypeRow[];
+ routes: OverviewRouteRow[];
coverage: Coverage;
}
diff --git a/admin/src/routes.tsx b/admin/src/routes.tsx
index b9b8e94..b7a3250 100644
--- a/admin/src/routes.tsx
+++ b/admin/src/routes.tsx
@@ -32,18 +32,15 @@ import {
groupsQuery,
healthQuery,
localRecordsQuery,
+ overviewQuery,
queriesInfiniteQuery,
queryDetailQuery,
rulesQuery,
settingsQuery,
- statsClientsQuery,
- statsQuery,
- statsRoutesQuery,
- statsTypesQuery,
- timeseriesQuery,
upstreamsQuery,
} from "@/lib/queries";
import { DEFAULT_PERIOD, parsePeriod } from "@/features/overview/period";
+import { OverviewPending } from "@/features/overview/OverviewFrame";
import {
validateGroupId,
validateProtectionSearch,
@@ -168,26 +165,28 @@ const overviewRoute = createRoute({
}),
loaderDeps: ({ search }): { period: Period } => ({ period: search.period ?? DEFAULT_PERIOD }),
/**
- * Started here, awaited nowhere. Every panel reads these with `useQuery` and
- * owns its own loading and error surface, so awaiting would trade that whole
- * contract for one blocking navigation: the page would sit on the slowest of
- * five requests and then appear complete, instead of the four that answered
- * rendering beside the one still in flight. The rejections are caught only to
- * keep them from going unhandled; the panels state them.
+ * Started here, awaited nowhere. The page reads these with `useQuery` and owns
+ * its own loading and error surface, so awaiting would trade that contract for
+ * one blocking navigation: nothing at all until the request answered, rather
+ * than the heading and the period picker while it is in flight. The rejections
+ * are caught only to keep them from going unhandled; the page states them.
*/
loader: ({ context, deps }) => {
const start = (promise: Promise) => void promise.catch(() => {});
start(context.queryClient.ensureQueryData(healthQuery()));
- start(context.queryClient.ensureQueryData(statsQuery(deps.period)));
- start(context.queryClient.ensureQueryData(timeseriesQuery(deps.period)));
- start(context.queryClient.ensureQueryData(statsClientsQuery(deps.period)));
+ start(context.queryClient.ensureQueryData(overviewQuery(deps.period)));
// The registered names the client chart labels its series with. Started here
// so the lookup is not a second round trip after the page chunk lands.
start(context.queryClient.ensureQueryData(clientsQuery()));
- start(context.queryClient.ensureQueryData(statsTypesQuery(deps.period)));
- start(context.queryClient.ensureQueryData(statsRoutesQuery(deps.period)));
},
component: lazyRouteComponent(() => import("@/features/overview/OverviewPage")),
+ /**
+ * The page's own loading surface, rendered while its chunk is still in flight.
+ * The default pending component would put a second, differently-placed
+ * "Loading…" before it, which reads as a stutter rather than one wait.
+ * `OverviewFrame` is a separate module so this import leaves the charts lazy.
+ */
+ pendingComponent: OverviewPending,
});
/**
diff --git a/admin/src/shell/AppShell.test.tsx b/admin/src/shell/AppShell.test.tsx
index 3e68f44..76058e3 100644
--- a/admin/src/shell/AppShell.test.tsx
+++ b/admin/src/shell/AppShell.test.tsx
@@ -19,44 +19,16 @@ const RECONCILED_AT = 1754899200;
const DATABASE: ConfigStatus = { authority: "database", path: null, reconciled_at: null, restart_pending: false };
const RESPONSES: Record = {
- "/api/stats?period=24h": {
- period: "24h",
- since: 0,
- until: 86400,
- queries: 0,
- blocked: 0,
- clients: 0,
- avg_response_time_us: null,
- coverage: { complete: true, available_since: 0 },
- },
- "/api/stats/timeseries?period=24h": {
+ "/api/overview?period=24h": {
period: "24h",
since: 0,
until: 86400,
bucket_seconds: 1800,
+ totals: { queries: 0, blocked: 0, clients: 0, avg_response_time_us: null },
buckets: [],
- coverage: { complete: true, available_since: 0 },
- },
- "/api/stats/clients?period=24h": {
- period: "24h",
- since: 0,
- until: 86400,
- bucket_seconds: 1800,
clients: [],
other: [],
- coverage: { complete: true, available_since: 0 },
- },
- "/api/stats/types?period=24h": {
- period: "24h",
- since: 0,
- until: 86400,
types: [],
- coverage: { complete: true, available_since: 0 },
- },
- "/api/stats/routes?period=24h": {
- period: "24h",
- since: 0,
- until: 86400,
routes: [],
coverage: { complete: true, available_since: 0 },
},
diff --git a/docs/how-to/set-up-admin-authentication.md b/docs/how-to/set-up-admin-authentication.md
index c536c4e..34da68c 100644
--- a/docs/how-to/set-up-admin-authentication.md
+++ b/docs/how-to/set-up-admin-authentication.md
@@ -62,7 +62,7 @@ Which of steps 4 and 5 applies to your server depends on its authority. Under `n
Login is `POST /api/auth/login` with a JSON body. Without a session, the API answers 401:
```sh
-curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8451/api/stats
+curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8451/api/overview
```
```
@@ -92,7 +92,7 @@ The cookie is named `nxdns_session` and carries `HttpOnly; SameSite=Lax; Path=/`
```sh
curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w '%{http_code}\n' \
- http://127.0.0.1:8451/api/stats
+ http://127.0.0.1:8451/api/overview
```
```
@@ -125,13 +125,13 @@ Sessions live in memory only. A restart logs everyone out. Thirty-two concurrent
```sh
curl -sS -b /tmp/nxdns-lab/cookies.txt -c /tmp/nxdns-lab/cookies.txt \
-X POST http://127.0.0.1:8451/api/auth/logout
-curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w 'stats: %{http_code}\n' \
- http://127.0.0.1:8451/api/stats
+curl -sS -b /tmp/nxdns-lab/cookies.txt -o /dev/null -w 'overview: %{http_code}\n' \
+ http://127.0.0.1:8451/api/overview
```
```
{"authenticated":false}
-stats: 401
+overview: 401
```
Logging out with a stale cookie, or with none, answers the same way. The point of logging out is to end up logged out, and that is where such a request already is.
@@ -154,7 +154,7 @@ Changing the password ends every session, including the one that made the change
```sh
curl -sS -b /tmp/nxdns-lab/c2.txt -o /dev/null -w 'old session: %{http_code}\n' \
- http://127.0.0.1:8451/api/stats
+ http://127.0.0.1:8451/api/overview
curl -sS -X POST http://127.0.0.1:8451/api/auth/login \
-H 'content-type: application/json' -d '{"password":"lab-password"}' \
-w ' (old password)\n'
@@ -217,14 +217,14 @@ curl -sS -X POST http://127.0.0.1:8451/api/auth/login \
curl -sS -c /tmp/nxdns-lab/c5.txt -X POST http://127.0.0.1:8451/api/auth/login \
-H 'content-type: application/json' -d '{"password":"offline-password"}' \
-w ' (new password, http %{http_code})\n'
-curl -sS -b /tmp/nxdns-lab/c5.txt -o /dev/null -w 'stats: %{http_code}\n' \
- http://127.0.0.1:8451/api/stats
+curl -sS -b /tmp/nxdns-lab/c5.txt -o /dev/null -w 'overview: %{http_code}\n' \
+ http://127.0.0.1:8451/api/overview
```
```
{"error":"invalid password"} (old password, http 401)
{"authenticated":true,"auth_required":true} (new password, http 200)
-stats: 200
+overview: 200
```
The next export shows the new hash and a null `password` again:
@@ -245,7 +245,7 @@ See [back up and restore](back-up-and-restore.md) for when `import` does need `-
Authentication is off. Every route is open, and a login attempt succeeds without minting anything — there is nothing to log in to, and a session that authorises nothing would be a lie for the browser to store:
```sh
-curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8453/api/stats
+curl -sS -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8453/api/overview
curl -sS -X POST http://127.0.0.1:8453/api/auth/login \
-H 'content-type: application/json' -d '{"password":"anything"}'
```
diff --git a/docs/reference/api.md b/docs/reference/api.md
index f372f02..a64a6e6 100644
--- a/docs/reference/api.md
+++ b/docs/reference/api.md
@@ -109,11 +109,7 @@ Auth `open` means no session is required; `session` means a valid session cookie
| GET | `/api/queries` | session | counted | read | Query log rows for the Activity page |
| GET | `/api/queries/{id}` | session | counted | read | One query, fully explained |
| GET | `/api/queries/live` | session | exempt | read | Live query stream (server-sent events) |
-| GET | `/api/stats` | session | counted | read | Totals for a period |
-| GET | `/api/stats/timeseries` | session | counted | read | Bucketed counts for a period |
-| GET | `/api/stats/types` | session | counted | read | Query-type breakdown for a period |
-| GET | `/api/stats/routes` | session | counted | read | How the period's queries were answered |
-| GET | `/api/stats/clients` | session | counted | read | Per-client bucketed counts for a period |
+| GET | `/api/overview` | session | counted | read | Everything the Overview page draws, for one period |
| GET | `/api/lookup` | session | counted | read | Explain a domain |
| GET | `/api/diagnostics` | session | counted | read | Operational event log |
| DELETE | `/api/diagnostics` | session | counted | runtime action | Purge every resolved event |
@@ -221,6 +217,6 @@ A non-empty `rewrites.cname_target` on a query detail means the decision landed
### Coverage
-Every window-bounded read — `GET /api/queries` and the five `GET /api/stats*` endpoints — answers with a `coverage` object: `available_since` is the oldest instant the query log is still complete for, and `complete` is true only when the window the request asked about starts at or after it. Retention deletes rows and advances the watermark in one transaction, so a client can tell an empty window from a pruned one instead of charting the gap as zero. A request with no lower bound at all asks about the whole of history, and is never complete.
+Every window-bounded read — `GET /api/queries` and `GET /api/overview` — answers with a `coverage` object: `available_since` is the oldest instant the query log is still complete for, and `complete` is true only when the window the request asked about starts at or after it. Retention deletes rows and advances the watermark in one transaction, so a client can tell an empty window from a pruned one instead of charting the gap as zero. A request with no lower bound at all asks about the whole of history, and is never complete.
-Each of these responses reads its rows and its watermark inside one SQLite read transaction, so retention cannot prune between the two and hand back pre-prune rows tagged with a post-prune `available_since`. Coherence stops there: two separate requests are two separate reads, and queries logged between them can move the counts.
+Each of these responses reads its rows and its watermark inside one SQLite read transaction, so retention cannot prune between the two and hand back pre-prune rows tagged with a post-prune `available_since`. `GET /api/overview` puts every Overview panel inside that one transaction, so its totals and its four breakdowns describe one database state. Coherence stops there: two separate requests are two separate reads, and queries logged between them can move the counts.
diff --git a/specs/milestone-36.md b/specs/milestone-36.md
new file mode 100644
index 0000000..7685220
--- /dev/null
+++ b/specs/milestone-36.md
@@ -0,0 +1,417 @@
+# Milestone 36: Overview performance — combined endpoint, projections, cache
+
+Replace the five per-panel stats endpoints with one `GET /api/overview` served
+from materialized projections in `querylog.db` plus an in-memory response
+cache, so Overview cost stops growing with query-log size.
+
+## Motivation (measured)
+
+Today each Overview load runs five separate scans of every raw row in the
+window, serialized on `WebState.querylog_lock`, re-polled every 30 s. Measured
+x86 ReleaseSafe (bench at scratchpad `statsbench2/`, production-like skew;
+Pi ≈ 3–3.5× slower):
+
+| Rows | 30d, five scans (today) | 30d, one combined scan | 30d, projections |
+|-----:|------------------------:|-----------------------:|-----------------:|
+| 1M | ~2.2 s | 264 ms | 41 ms |
+| 3M | ~6.6 s | 793 ms | 38 ms |
+| 5M | ~11.8 s | 1,346 ms | 40 ms |
+
+Projection maintenance costs +10% per 100-row insert batch, and ~1.5 MB of
+disk in the bench — a size bounded by retained buckets × distinct
+client/type/route keys, independent of raw query volume. Production is on a ~100k rows/day growth
+curve (≈3M rows at 30-day retention), so the projection path is the design
+target, not a contingency. Both computations were cross-checked for identical
+output in the bench.
+
+Design ruling (owner + Codex consultation, 2026-08-27): stay on SQLite;
+projections live in the same file as the raw rows and are updated in the same
+transaction, so SQLite's transaction is the coherence mechanism — no second
+file, no epoch protocol. The DDL change re-fingerprints `querylog.db`; the
+existing rename-aside path handles old files (one-time history reset,
+disclosed in the changelog). No backfill migration.
+
+## Sessions
+
+Four sessions. A first. B and C after A, in parallel (disjoint files). D after B.
+
+- A: storage — projection schema, writer maintenance, retention, new read path.
+ A does NOT delete the five existing aggregate functions — `stats.zig` still
+ calls them until B lands, and A must leave `zig build test` green.
+- B: web — `/api/overview` handler, response cache, removal of the five old
+ endpoints, OpenAPI/contract regeneration.
+- C: admin — one overview query, types, component/data plumbing, tests.
+- D: storage cleanup — delete the five now-unreferenced aggregate functions.
+
+---
+
+## Session A: storage
+
+### A.1 Schema (src/storage/querylog_schema.zig)
+
+Append four projection tables to `ddl`. Grain: 30-minute buckets, `bucket` =
+floor-to-grid of the row timestamp: `@divFloor(timestamp, 1800) * 1800` in Zig
+and the equivalent floor semantics in any SQL (SQLite integer `/` truncates
+toward zero, which differs on negative timestamps — use floor everywhere, as
+`window()` does). 1800 divides every serving
+width ≥ 30 min (1800, 3600, 21600), which is what makes one grain serve the
+24h, 7d and 30d windows exactly. The 1h window (60 s buckets) is NOT served
+from projections (A.4).
+
+```sql
+CREATE TABLE bucket_totals (
+ bucket INTEGER PRIMARY KEY,
+ queries INTEGER NOT NULL,
+ blocked INTEGER NOT NULL,
+ cached INTEGER NOT NULL,
+ rt_sum INTEGER NOT NULL, -- sum(response_time_us) over timed rows
+ rt_count INTEGER NOT NULL -- count(response_time_us)
+) WITHOUT ROWID;
+
+CREATE TABLE bucket_clients (
+ bucket INTEGER NOT NULL,
+ client_ip TEXT NOT NULL,
+ queries INTEGER NOT NULL,
+ PRIMARY KEY (bucket, client_ip)
+) WITHOUT ROWID;
+
+CREATE TABLE bucket_types (
+ bucket INTEGER NOT NULL,
+ qtype INTEGER NOT NULL, -- -1 encodes a NULL qtype, losslessly
+ count INTEGER NOT NULL,
+ PRIMARY KEY (bucket, qtype)
+) WITHOUT ROWID;
+
+CREATE TABLE bucket_routes (
+ bucket INTEGER NOT NULL,
+ route_kind TEXT NOT NULL,
+ source_present INTEGER NOT NULL, -- 0: source NULL; 1: source = source_text
+ source_text TEXT NOT NULL, -- '' when source_present = 0
+ count INTEGER NOT NULL,
+ PRIMARY KEY (bucket, route_kind, source_present, source_text),
+ CHECK (source_present IN (0, 1)),
+ CHECK (source_present = 1 OR source_text = '')
+) WITHOUT ROWID;
+```
+
+Column semantics match the existing aggregates exactly: `blocked` counts
+`blocked <> 0`; `cached` counts `cache_hit = 1`; routes' `source` is the
+existing CASE (`upstream` rows → `upstream`, `forward_zone` rows →
+`forward_zone`, else NULL). The fingerprint moves automatically; do not touch
+the fingerprint machinery.
+
+### A.2 Writer maintenance (src/storage/repositories/queries_repo.zig)
+
+`BatchWriter.writeBatch` updates all four projections inside the same
+transaction that inserts the raw rows:
+
+- Aggregate the batch in Zig first, producing per-key deltas; then one UPSERT
+ per touched key:
+ `INSERT ... ON CONFLICT(...) DO UPDATE SET queries = queries + excluded.queries, ...`.
+ The aggregation must accept any slice length — `writeBatch`'s API does not
+ enforce the logger's 100-row batching, so no fixed-size arrays sized to it.
+ `BatchWriter` currently owns no allocator: `init` gains one, owned for the
+ writer's life, used only for the per-batch delta maps; scratch is freed (or
+ a retained map cleared) at the end of every `writeBatch`, and
+ `error.OutOfMemory` fails the batch before the transaction opens — no
+ hidden global allocator, no implicit size cap, no quadratic rescanning.
+- No per-row SQL, no triggers.
+- Failure contract: any failed projection statement rolls the whole
+ transaction back — raw rows and projections together — resets every
+ projection statement, and leaves the writer usable for the next batch
+ (`resetAll` discipline as for the raw statements today). Fault-injection
+ acceptance: a batch whose projection update fails leaves the database
+ unchanged, and the next batch succeeds.
+- The bench measured this at 0.39 ms vs 0.34 ms per batch — acceptance is
+ correctness, not speed.
+
+### A.3 Retention (src/storage/repositories/queries_repo.zig, prune path)
+
+In the same transaction as `pruneOlderThan(cutoff)`'s raw delete:
+
+1. Delete projection rows with `bucket < floor(cutoff / 1800) * 1800` from all
+ four tables.
+2. If `cutoff` is not on a bucket boundary, recompute the straddling bucket
+ (`floor(cutoff/1800)*1800`) from the remaining raw rows and replace its
+ projection rows in all four tables. Never approximate.
+
+Failure atomicity: a failure during the projection delete or the
+straddling-bucket replacement rolls back the raw delete, the watermark
+advance and every projection change together — one transaction, tested by
+fault injection.
+
+Implementation note (Session A, recorded post-build): the bucket_totals
+recompute carries `HAVING count(*) > 0` — a bare SQL aggregate always yields
+one row, and an emptied straddling bucket must disappear, not persist as
+zeros.
+
+### A.4 Read path (src/storage/repositories/queries_repo.zig)
+
+One function producing the whole Overview payload for a window, from one
+already-open read transaction (the caller owns transaction + lock, as today):
+
+```zig
+pub const Overview = struct {
+ totals: StatsTotals,
+ buckets: []const Bucket, // bucket_count entries, zero-filled
+ clients: ClientsBreakdown, // top-8 + other, as today
+ types: []const TypeCount, // sorted as stats_types_sql sorts
+ routes: []const RouteCount, // sorted as stats_routes_sql sorts
+};
+
+pub fn overview(
+ database: *db.Db,
+ arena: Allocator,
+ since: i64,
+ bucket_seconds: u32,
+ bucket_count: u32,
+) db.Error!Overview
+```
+
+Storage owns these scalars — no import of any web module. `until` is derived
+as `since + bucket_seconds * bucket_count` with the same overflow checks as
+`timeseries`. Preconditions, checked before path selection and tested:
+`bucket_seconds != 0` and `bucket_count != 0` (else `error.Misuse`, matching
+the existing clients contract); on the projection path
+(`bucket_seconds >= 1800`) additionally `since` a multiple of 1800 and
+`bucket_seconds % 1800 == 0`, else `error.Misuse`. The handler's `window()`
+guarantees all of them.
+
+Two implementations behind one entry point, chosen by `bucket_seconds`:
+
+- `bucket_seconds >= 1800` (24h, 7d, 30d): read the four projection tables
+ over `[since, until)`, aggregating 30-min rows up to the serving width in
+ Zig. `distinct_clients` comes from grouping `bucket_clients` by `client_ip`
+ over the window — never from summing per-bucket counts.
+ `avg_response_time_us` = `sum(rt_sum) / sum(rt_count)`, null when
+ `rt_count` sums to 0. Top-8 clients ranked by window total desc, ties by
+ `client_ip` asc (BINARY), residual summed into `other` — identical cut
+ semantics to `statsClients`.
+- `bucket_seconds < 1800` (1h): one single pass over the raw rows in the
+ window (one SELECT of the needed columns, stepped once), aggregating
+ everything in Zig. Memory bound: O(distinct clients + distinct qtypes +
+ distinct routes) in the window — explicitly permitted; this is a household
+ LAN and the same bound the arena-returning aggregates already carry. This
+ replaces today's five scans and the clients rank+bucket double scan. Same
+ output contracts.
+
+Sort orders and tie-breaks must reproduce the existing SQL orderings exactly
+(types: count desc, null last within tie, qtype asc; routes: count desc,
+route_kind asc, null source last, source asc) — the goldens' byte-stability
+argument carries over. Note: `RouteKind`'s enum declaration order is not
+alphabetical; "route_kind asc" means the stored text's byte order, so any Zig
+comparator orders by `@tagName` bytes, never by enum ordinal (Session A's
+accumulator already does; mutation-tested).
+
+### A.5 Acceptance criteria
+
+- [ ] `zig build test` green.
+- [ ] Property test: after an arbitrary interleaving of batches and prunes
+ (including a prune cutoff off the bucket grid), every projection table
+ equals a from-scratch recomputation from `query_log`.
+- [ ] Equivalence test: `overview()` output (both paths) equals a test-only
+ oracle over the same window on the same data — including empty windows,
+ NULL qtype, NULL source on an `upstream` row, ties in ranking, and a
+ window whose last bucket is in progress. The oracle is a copy of the
+ five existing SQL aggregates living in the test file, so it survives
+ Session D's deletion of the production functions.
+- [ ] Fingerprint test updated (table/index count assertions in
+ querylog_schema tests).
+
+---
+
+## Session B: web
+
+### B.1 Endpoint (src/web/handlers/overview.zig, replacing stats.zig's five)
+
+`GET /api/overview?period=1h|24h|30d|7d` (same grammar, default 24h, same 400
+text). One read transaction under `WebState.querylog_lock` covering the
+aggregate and `coverage.read` — one snapshot, no cross-panel skew. Response:
+
+```json
+{
+ "period": "24h", "since": ..., "until": ..., "bucket_seconds": 1800,
+ "totals": { "queries": n, "blocked": n, "clients": n, "avg_response_time_us": n|null },
+ "buckets": [ { "ts": ..., "queries": n, "blocked": n, "cached": n }, ... ],
+ "clients": [ { "client": "ip", "buckets": [n, ...] }, ... ],
+ "other": [n, ...],
+ "types": [ { "qtype": n|null, "count": n }, ... ],
+ "routes": [ { "route": "...", "source": "..."|null, "count": n }, ... ],
+ "coverage": { ... }
+}
+```
+
+Field shapes and semantics are exactly today's five bodies merged; `Period`,
+`window()`, `max_buckets` move to (or stay importable from) the new handler.
+503 when the query log is unavailable; 500 logging unchanged. Route metadata
+identical to the removed endpoints: same authentication (`.session`), same
+rate-limit class (`.counted`), same authority policy (`.read`).
+
+Remove `GET /api/stats`, `/api/stats/timeseries`, `/api/stats/types`,
+`/api/stats/routes`, `/api/stats/clients` and their routes.
+
+Contract surface (B owns all of it): add the new path and schema to the
+OpenAPI document, remove the five old operations and their schemas, update
+every drift guard that lists them, add a contract sample for
+`/api/overview`, and regenerate `admin/src/lib/contractSamples.gen.ts`
+(reserved for B — Session C must not touch it). Update the API listings in
+`docs/` and `PLAN.md` that name the five endpoints or the querylog layout.
+
+### B.2 Response cache (src/web/server.zig WebState + overview.zig)
+
+Per-period cached response body, invalidated by data change or window roll.
+Key: `(period, window.until, data_version)` where `data_version` is `PRAGMA
+data_version` on the web task's connection (it changes when any other
+connection — logger, retention — commits).
+
+The entire cache decision happens under `querylog_lock`; nothing touches the
+shared connection or the slots outside it. Exact sequence per request:
+
+1. Acquire `querylog_lock` — ONCE. `server.QuerylogRead.open` acquires this
+ lock itself, so the overview handler must not call it after step 1: B
+ refactors the scope into a lock-owning wrapper plus a
+ locked-caller variant (for example `QuerylogRead.openLocked`, documented
+ as requiring the lock), and the overview path uses the locked-caller
+ variant for step 4. A literal "lock, then QuerylogRead.open" deadlocks.
+2. Sample `PRAGMA data_version` (inside the lock — the shared connection may
+ otherwise have a foreign transaction open, and the slots need the mutual
+ exclusion anyway).
+3. Hit (`slot.period == period and slot.until == window.until and
+ slot.data_version == sampled`): copy the stored bytes into the request
+ arena, release the lock, respond. The copy is what makes a concurrent
+ rebuild's free-and-replace safe.
+4. Miss: open the read transaction, build the body, commit. Publish to the
+ slot ONLY after a successful commit, keyed by the version sampled in
+ step 2 (a commit landing during the build bumps `data_version`, so the
+ next request rebuilds — stale-under-new-key is impossible). A failed
+ commit or build publishes nothing and responds 500 as today.
+5. Copy to the request arena, release the lock, write the socket. The lock
+ never spans a socket write (existing discipline).
+
+Because the check happens only under the lock, `querylog_lock` is the
+single-flight: a second request for the same key waits and then hits.
+
+Storage: one slot per period (4 slots) in `WebState`; body bytes allocated
+from `WebState.gpa`, replaced on rebuild (free old, install new), freed in
+`deinit`. No capacity limit beyond the allocator — a body is bounded by the
+fixed bucket counts plus the household client/type/route cardinality.
+
+No adaptive polling and no combined-endpoint staging: with projections + this
+cache a rebuild is ~40 ms x86 / ~0.13 s Pi, so the admin's existing 30 s
+cadence is fine.
+
+### B.3 Acceptance criteria
+
+- [ ] `zig build test` green; handler tests ported from stats.zig (period
+ grammar, window math, one-snapshot behavior) plus: cache hit returns
+ byte-identical body; a logger commit (data_version bump) invalidates;
+ a window roll invalidates; a retention prune committed through another
+ connection invalidates (both the aggregates and the cached
+ `coverage.available_since` are replaced); a failed read-transaction
+ commit neither installs nor replaces a cache entry.
+- [ ] `curl /api/overview?period=30d` on a seeded scratch instance returns all
+ panels consistent (breakdowns sum to totals on a quiet database).
+- [ ] The five old routes return 404.
+
+---
+
+## Session C: admin
+
+### C.1 Data layer
+
+- `admin/src/lib/types.ts`: one `Overview` type mirroring B.1; remove the five
+ per-panel response types.
+- `admin/src/lib/api.ts`: `getOverview(period)`; remove the five getters.
+- `admin/src/lib/queries.ts`: `overviewQuery(period)` with
+ `refetchInterval: 30_000` and key `["overview", period]`; remove the five
+ stats query factories and their keys.
+
+### C.2 Overview page
+
+`admin/src/features/overview/overviewWindow.ts` and the chart components
+consume the single query: one `useQuery` where five ran in parallel. Loading,
+error and coverage handling collapse to one page-level surface (one spinner
+state, one error state for the whole Overview); the per-panel shells,
+layout, copy, chart dimensions and accessibility attributes stay exactly as
+they are. Chart components (TimeseriesChart, ClientChart, Donut) keep their
+props — adapt the mapping layer, not the charts. C must not touch
+`contractSamples.gen.ts` (B owns its regeneration).
+
+### C.3 Acceptance criteria
+
+- [ ] `npm test` green in admin/ (mock the one endpoint; port the five-query
+ tests).
+- [ ] `npm run typecheck` green — the build alone does not run tsc. B and C
+ are file-disjoint but type-coupled through `contractSamples.gen.ts`:
+ C's typecheck/test/build gates run (or re-run) AFTER B has regenerated
+ that file. Implementation may proceed in parallel; the green gate is
+ sequenced.
+- [ ] `npm run build` green; bundle-size assertion still passes.
+- [ ] Manual: scratch instance renders all Overview panels from the new
+ endpoint on all four periods.
+
+---
+
+## Session D: storage cleanup
+
+After B is merged and green: delete `statsTotals`, `timeseries`, `statsTypes`,
+`statsRoutes`, `statsClients` and their SQL constants from
+`queries_repo.zig` — nothing references them once `stats.zig` is gone. The
+test-only oracle from A.5 stays. Acceptance: `zig build test` green, no dead
+stats SQL remains in production code.
+
+---
+
+## Module layout
+
+- `src/web/handlers/overview.zig` — new; replaces `src/web/handlers/stats.zig`
+ (deleted by B).
+- `src/storage/querylog_schema.zig` — projection DDL appended.
+- `src/storage/repositories/queries_repo.zig` — writer maintenance, retention
+ integration, `overview()` read path (A); five old aggregates deleted (D).
+- Admin files per C.1/C.2.
+
+## File ownership
+
+- A: `src/storage/*` (old aggregates left in place), plus the mechanical
+ allocator-plumbing at every `BatchWriter.init` call site outside storage
+ (`src/web/handlers/*`, `src/web/web_integration_test.zig`, any test using
+ the writer) — A runs before B, so this is sequential, not shared,
+ ownership; A's gate is the full `zig build test`.
+- B: `src/web/*`, plus: `src/app.zig` (cache cleanup at the composition
+ root), `src/tests.zig` (handler import swap), the OpenAPI document and its
+ drift guards, contract samples including
+ `admin/src/lib/contractSamples.gen.ts`, `docs/**` API listings, `PLAN.md`
+ stale sections, and the Overview/API sections of `specs/ui-redesign.md`
+ (which still mandates the five endpoints and must be amended, not obeyed).
+- C: `admin/*` EXCEPT `admin/src/lib/contractSamples.gen.ts`.
+- D: `src/storage/repositories/queries_repo.zig` (sequential, after B).
+- Orchestrator: `CHANGELOG.md` (hand-written, per release process).
+B and C run in parallel; the one shared-tree exception above is reserved to B.
+
+## Acceptance criteria (milestone complete)
+
+- [ ] `zig build test` and `zig build test -Dintegration` green (the
+ integration suite carries the live route walk, contract-sample
+ comparison, concurrent querylog reads and OpenAPI guards); admin
+ `npm test`/`typecheck`/`build` green.
+- [ ] Scratch-instance smoke: seeded data + live digs; Overview correct on all
+ periods; old endpoints gone.
+- [ ] Changelog discloses: schema change resets query history (rename-aside),
+ five endpoints replaced by `/api/overview`.
+- [ ] Release gate (`zig build cut` fingerprint check) satisfied.
+
+## Anti-requirements
+
+- No second database file, no epoch/validity protocol, no ATTACH.
+- No backfill migration, no rebuild command, no catch-up cursor — projections
+ are born with the file and maintained transactionally; that is the whole
+ coherence story.
+- No connection pool, no adaptive polling, no DuckDB.
+- No new indexes on `query_log`, no triggers, no per-row projection SQL.
+- Do not change the 1h/24h/7d/30d period grammar, bucket widths or counts.
+- No HTTP-level caching of any kind: no ETag, no `Cache-Control`, no
+ stale-while-revalidate, no background refresh, and never cache a 500/503
+ body. The cache is exactly the in-process design of B.2.
+- No visual redesign, no chart-prop changes, no cache configuration knobs,
+ no cache metrics. This milestone changes data acquisition and storage only.
diff --git a/specs/ui-redesign.md b/specs/ui-redesign.md
index 671636d..6d5be18 100644
--- a/specs/ui-redesign.md
+++ b/specs/ui-redesign.md
@@ -54,14 +54,14 @@ One question, answered over a period the reader chooses: what did the resolver d
Top to bottom, edge to edge:
-1. **Four stat tiles**, neutral chrome throughout — no coloured accents; emphasis is typographic. Queries, Blocked (count and rate), Clients, Average response. Each tile carries the way into the rows behind its number: Queries and Blocked open Activity for exactly the bounds the stats response returned, Clients opens the clients page, and Average response has nothing to open.
+1. **Four stat tiles**, neutral chrome throughout — no coloured accents; emphasis is typographic. Queries, Blocked (count and rate), Clients, Average response. Each tile carries the way into the rows behind its number: Queries and Blocked open Activity for exactly the bounds the overview response returned, Clients opens the clients page, and Average response has nothing to open.
2. **Queries over time** — the existing query-volume timeline, split blocked/cached/other, full width.
3. **Client activity over time** — one stacked series per named client plus "other", on the same bucket alignment as the timeline so the two charts share an x-axis. A client registered under a name is labelled by it, with the same precedence the query tables apply and the address kept as the title; colour keys on the address, so naming a client never repaints its series.
4. **Query types** and **Upstream servers** — two donuts, side by side above 1280px and stacked below, with the ring and its legend centred in the panel while stacked and left-anchored once they are a pair. Types are labelled by the admin's own `qtypeName()`; routes by route-kind labels and by the answering resolver or zone. Each donut's SVG is decoration (`aria-hidden`, `focusable="false"`); a visible legend and a visually hidden table are the accessible surface. An empty window says "No queries in this period." rather than drawing nothing.
Colours key on semantic identity — the qtype value, the client string, the `(route, source)` pair — so a rank change between two polls never repaints an entry. Charts stay lightweight SVG; no charting dependency.
-**Window coherence, five requests.** Totals, timeseries, clients, types and routes are separate calls, and the page holds one window identified by `(period, since, until, coverage.available_since)` — the watermark joins the identity because retention advancing mid-page changes what the same span can answer for. A response is a member only if all four fields match. Rendering is per panel: a member renders, a panel still in flight shows its own loading state, a panel whose request failed shows its own error and Retry, and the members keep rendering throughout — a failed donut never blanks the charts. A response behind the window is refetched once per endpoint-keyed episode and, if it stays behind, that panel alone shows an error. This is window coherence, not data-snapshot coherence: live inserts between requests may shift counts slightly between panels, and that is accepted. One coverage notice for the page, from the window's watermark.
+**One request, one snapshot (superseded 2026-08-27 by milestone 36; the paragraph below replaces the original five-request window-coherence design).** The page makes one call, `GET /api/overview?period=…`, whose body carries totals, timeseries, clients, types, routes and coverage from a single read transaction — data-snapshot coherence, so the panels cannot disagree and no reconciliation layer exists. Loading and error are page-level: one loading surface, one error with Retry for the whole Overview. The per-panel shells, layout, copy and accessibility surfaces are unchanged. One coverage notice for the page, from the response's watermark. A `keepPreviousData` body whose own `period` is not the selected one keeps the page in loading.
**The shell.** The header carries no protection display at all. The Pause/Resume control sits at the foot of the sidebar, above the version label, in both the desktop rail and the mobile drawer; it is the only global runtime action, and it belongs to the resolver rather than to any page. It still appears beside the detail of a query that was blocked. The control states a pause with itself — "Paused until 14:05", or "Paused" when the pause has no end — because "Resume" names an action without naming the state it would end, and with the indicator and the status rows both gone the sidebar is the only place a page other than Diagnostics can carry that fact. An active resolver gets no line; the button says Pause, which is the whole message. The line and the health strip read one `protection` condition through one clock format, so they cannot disagree. The Diagnostics navigation item carries a badge: the open-episode count, or a neutral "!" when the rollup is degraded with nothing open and when the latest health poll failed — an unknown must never read as healthy. It is hidden only when health data exists, the latest poll succeeded, and the rollup is ok with nothing open.
@@ -168,7 +168,7 @@ No response payloads, answer RR sets, EDNS data or packet bytes are stored. The
Privacy transforms apply to every new domain-bearing field, not only `domain`: with `hide_domains` on, matched names, CNAME targets and safe-search targets hide consistently.
-A one-row `querylog_meta (created_at INTEGER NOT NULL)` table lets the stats and query APIs return a conservative `available_since`, which distinguishes "zero queries" from "history does not exist".
+A one-row `querylog_meta (created_at INTEGER NOT NULL)` table lets the overview and query APIs return a conservative `available_since`, which distinguishes "zero queries" from "history does not exist".
### Historical query detail
@@ -208,9 +208,9 @@ Database mode uses the same information architecture with real edit actions, plu
`GET /api/queries` keeps keyset pagination and its filters; rows gain `rcode`, `route_kind`, `policy_action` and the short policy reason the table needs, and the body gains `coverage: {complete, available_since}`. `GET /api/queries/{id}` returns nested `request` / `policy` / `route` / `response` provenance. `GET /api/queries/live` sends the same object without `id`.
-`GET /api/stats` and `/api/stats/timeseries` add `complete` and `available_since`.
+**Superseded by milestone 36 (2026-08-27).** This section originally specified five per-panel endpoints — `GET /api/stats`, `/api/stats/timeseries`, `/api/stats/types`, `/api/stats/routes` and `/api/stats/clients`. Five requests could promise a shared window but never a shared snapshot, and each one scanned every raw row in it. They are replaced by a single `GET /api/overview?period=1h|24h|7d|30d`, which returns `{period, since, until, bucket_seconds, totals:{queries, blocked, clients, avg_response_time_us}, buckets:[{ts, queries, blocked, cached}], clients:[{client, buckets}], other, types:[{qtype, count}], routes:[{route, source, count}], coverage:{complete, available_since}}` — every field with the semantics the five bodies gave it, over one deferred SQLite read transaction, so the breakdowns and the coverage watermark describe one database state. The 24h, 7d and 30d windows are served from 30-minute projection tables maintained transactionally beside the raw rows; the 1h window takes one raw scan. Per-period response caching keyed on `(window.until, PRAGMA data_version)` lives in the web layer.
-**Three period aggregations (added 2026-08-22)** to feed the new Overview panels, all taking the same `period` parameter and reporting over the same aligned window, and all reading their rows and their coverage watermark inside one deferred SQLite read transaction. `GET /api/stats/types` → `{period, since, until, coverage, types:[{qtype, count}]}`, the numeric type only — naming types stays the admin's job, and a second table in the server would drift out of agreement with it — with the rows that recorded no type kept as their own `null` group. `GET /api/stats/routes` → `{period, since, until, coverage, routes:[{route, source, count}]}`, grouping `upstream` rows by the answering resolver and `forward_zone` rows by the zone, with blocked, cache, local and rejected carrying no source. `GET /api/stats/clients` → `{period, since, until, bucket_seconds, coverage, clients:[{client, buckets}], other}`, bucketed exactly as `/api/stats/timeseries`, the eight busiest clients named and everything else summed into `other`, which is always present and always bucket-count-sized. No new writers and no new state: all three are pure reads over the query log's provenance columns.
+The panel semantics the five endpoints defined all carry over unchanged: types are the numeric type only — naming types stays the admin's job, and a second table in the server would drift out of agreement with it — with the rows that recorded no type kept as their own `null` group; routes group `upstream` rows by the answering resolver and `forward_zone` rows by the zone, with blocked, cache, local and rejected carrying no source; clients name the eight busiest and sum everything else into `other`, which is always present and always bucket-count-sized.
Existing mutation endpoints stay specific. Diagnostics introduces no generic "perform remediation" endpoint; it invokes the existing blocklist-refresh and certificate-reload operations.
diff --git a/src/app.zig b/src/app.zig
index 64e74f7..0f892c8 100644
--- a/src/app.zig
+++ b/src/app.zig
@@ -817,6 +817,9 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
// Same argument as the live hash: a settings PUT may have installed an
// owned generation, and this runs after `group.cancel`.
defer web_state.proxies.deinit(gpa);
+ // The Overview response cache owns its bodies from `gpa`. Same argument
+ // again: no web task can still be reading a slot once the group is cancelled.
+ defer web_state.overview_cache.deinit(gpa);
if (cfg.web.enabled) web_state = .{
.gpa = gpa,
.web = cfg.web,
diff --git a/src/storage/logger.zig b/src/storage/logger.zig
index 65e852b..84c5037 100644
--- a/src/storage/logger.zig
+++ b/src/storage/logger.zig
@@ -30,6 +30,7 @@
//! `writer_failed`, so the loss is visible rather than silent.
const std = @import("std");
+const Allocator = std.mem.Allocator;
const builtin = @import("builtin");
const db = @import("db.zig");
@@ -555,13 +556,17 @@ pub const Logger = struct {
/// group it cancels for exactly that reason.
///
/// `monitor` is the §11.6 gate. Null disables gating.
+ ///
+ /// `gpa` belongs to the `BatchWriter` for that writer's whole life; it
+ /// allocates the projection deltas of one batch and nothing else.
pub fn runWriter(
self: *Logger,
io: std.Io,
+ gpa: Allocator,
database: *db.Db,
monitor: ?*disk_monitor.Monitor,
) std.Io.Cancelable!void {
- var writer = queries_repo.BatchWriter.init(database) catch |err| {
+ var writer = queries_repo.BatchWriter.init(gpa, database) catch |err| {
scope.warn("query logger: preparing the batch statements failed: {s}", .{@errorName(err)});
// Without a writer there is no consumer, so leaving the queue open
// would silently swallow every later entry.
@@ -1134,7 +1139,7 @@ test "an entry with every provenance field set survives the queue, toRow, insert
var database = try openLog();
defer database.close();
- var writer = try queries_repo.BatchWriter.init(&database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -1466,6 +1471,7 @@ test "shutdown writes the batch the writer holds and the rest of the queue" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
+ testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
@@ -1502,7 +1508,7 @@ test "entries that arrive inside one window reach the database in one batch" {
var database = try openLog();
defer database.close();
- var writer = try queries_repo.BatchWriter.init(&database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [16]Entry = undefined;
@@ -1559,6 +1565,7 @@ test "the writer holds an entry for the length of the flush interval" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
+ testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
@@ -1611,6 +1618,7 @@ test "a full batch flushes without waiting for the interval" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
+ testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
@@ -1658,6 +1666,7 @@ test "the writer's next cycle uses the interval set since its last one" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
+ testing.allocator,
&database,
@as(?*disk_monitor.Monitor, null),
});
@@ -1700,7 +1709,7 @@ test "a gated flush holds the batch until the disk recovers" {
var database = try openLog();
defer database.close();
- var writer = try queries_repo.BatchWriter.init(&database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -1754,7 +1763,7 @@ test "a failing batch is dropped whole and the writer stays usable" {
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
- var writer = try queries_repo.BatchWriter.init(&database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -1789,7 +1798,7 @@ test "a writer that cannot prepare closes the queue and counts every entry" {
for (0..3) |i| logger.log(io, sampleEntry(@intCast(i), "early.example"));
- try logger.runWriter(io, &database, null);
+ try logger.runWriter(io, testing.allocator, &database, null);
try testing.expect(logger.writer_failed.load(.acquire));
try testing.expectEqual(@as(u64, 3), logger.queries_dropped.load(.monotonic));
@@ -1842,6 +1851,7 @@ test "the gating episode opens on the gate, turns losing on a drop, and clears o
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
+ testing.allocator,
&database,
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -2023,6 +2033,7 @@ test "a canceled writer counts the batch it was holding" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
+ testing.allocator,
&database,
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -2072,6 +2083,7 @@ test "a disk-gated writer drops what it holds at shutdown instead of hanging" {
var future = try io.concurrent(Logger.runWriter, .{
&logger,
io,
+ testing.allocator,
&database,
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -2121,7 +2133,7 @@ test "an empty batch touches neither the database nor the counters" {
var database = try openLog();
defer database.close();
- var writer = try queries_repo.BatchWriter.init(&database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -2155,7 +2167,7 @@ test "a dropped batch opens an error episode the next good batch closes" {
try fx.init(io, 1000);
defer fx.deinit();
- var writer = try queries_repo.BatchWriter.init(&database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var buf: [4]Entry = undefined;
@@ -2198,7 +2210,7 @@ test "a writer that cannot prepare leaves an episode no recovery path claims" {
var logger: Logger = .init(.{}, &buf);
logger.diagnostics = &fx.store;
- try logger.runWriter(io, &database, null);
+ try logger.runWriter(io, testing.allocator, &database, null);
try testing.expectEqualStrings("writer", try fx.text(
"SELECT subject_key FROM operational_events WHERE resolved_at IS NULL",
@@ -2209,7 +2221,7 @@ test "a writer that cannot prepare leaves an episode no recovery path claims" {
// The writer returned, so nothing can ever close this. A second run finds
// the queue closed and adds no second episode.
- try logger.runWriter(io, &database, null);
+ try logger.runWriter(io, testing.allocator, &database, null);
try testing.expectEqual(
@as(i64, 1),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
diff --git a/src/storage/logger_controller.zig b/src/storage/logger_controller.zig
index ead8c1e..646d23b 100644
--- a/src/storage/logger_controller.zig
+++ b/src/storage/logger_controller.zig
@@ -248,6 +248,7 @@ pub const Controller = struct {
owned.writer = try io.concurrent(logger.Logger.runWriter, .{
generation.logger,
io,
+ opts.gpa,
&owned.database,
opts.monitor,
});
@@ -434,7 +435,7 @@ pub const Controller = struct {
errdefer generation.deinit(self.gpa);
const owned = &generation.owned.?;
- owned.writer = try io.concurrent(runParkedWriter, .{ generation, io, self.monitor });
+ owned.writer = try io.concurrent(runParkedWriter, .{ generation, io, self.gpa, self.monitor });
// The statements are prepared before anything is published, so a
// failure here is a refused settings change rather than a writer that
@@ -565,11 +566,12 @@ fn drain(generation: *Generation, io: std.Io) void {
fn runParkedWriter(
generation: *Generation,
io: std.Io,
+ gpa: Allocator,
monitor: ?*disk_monitor.Monitor,
) std.Io.Cancelable!void {
const owned = &generation.owned.?;
- var writer = queries_repo.BatchWriter.init(&owned.database) catch |err| {
+ var writer = queries_repo.BatchWriter.init(gpa, &owned.database) catch |err| {
owned.prepare_error = err;
owned.ready.set(io);
return;
diff --git a/src/storage/phase6_integration_test.zig b/src/storage/phase6_integration_test.zig
index 80d033a..651950b 100644
--- a/src/storage/phase6_integration_test.zig
+++ b/src/storage/phase6_integration_test.zig
@@ -144,7 +144,7 @@ fn awaitCount(counter: *const std.atomic.Value(u64), target: u64, limit: usize)
}
fn writeRows(database: *db.Db, timestamps: []const i64, domain: []const u8) !void {
- var writer = try queries_repo.BatchWriter.init(database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, database);
defer writer.deinit();
var rows: [16]queries_repo.Row = undefined;
@@ -208,6 +208,7 @@ test "S8 case 1: the logger writes a real querylog.db end to end" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
+ testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, null),
});
@@ -256,6 +257,7 @@ test "S8 case 2: a single entry reaches the file once the flush interval passes"
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
+ testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, null),
});
@@ -311,6 +313,7 @@ test "S8 case 3: a full queue drops the oldest entries and the newest survive" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
+ testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, &monitor),
});
@@ -359,6 +362,7 @@ test "S8 case 4: the privacy transforms reach the stored rows" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
+ testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, null),
});
@@ -459,6 +463,7 @@ test "S8 case 6: a critical disk gates the flushes and recovery releases them" {
var future = try io.concurrent(logger.Logger.runWriter, .{
&query_log,
io,
+ testing.allocator,
log_db.database(),
@as(?*disk_monitor.Monitor, &monitor),
});
diff --git a/src/storage/querylog_schema.zig b/src/storage/querylog_schema.zig
index 2ef1c29..0bddcf7 100644
--- a/src/storage/querylog_schema.zig
+++ b/src/storage/querylog_schema.zig
@@ -25,7 +25,7 @@ const log = std.log.scoped(.querylog_schema);
/// PLAN §11.3, plus the coverage watermark of milestone 28. Multi-statement
/// text — it goes through `db.Db.exec`, never through `prepare`.
///
-/// The trailing INSERT seeds `querylog_meta`, which is part of the schema
+/// The INSERT seeds `querylog_meta`, which is part of the schema
/// rather than a later step: a `query_log` with no watermark beside it cannot
/// answer whether an empty result means "no queries" or "no history", and every
/// database this program reads from is created by executing this string.
@@ -36,6 +36,13 @@ const log = std.log.scoped(.querylog_schema);
/// logged in the same second the file was created is not evidence that the
/// second is completely covered, and the watermark's whole job is to be
/// conservative. From there it only ever advances, in `queries_repo.pruneOlderThan`.
+///
+/// The four `bucket_*` tables are the Overview projections (milestone 36), on a
+/// 30-minute grain that divides every serving width the API offers. They carry
+/// no history of their own: they are born with the file and maintained in the
+/// same transaction as every insert and every prune, so SQLite's transaction is
+/// the only coherence mechanism there is. There is no backfill path — a file
+/// whose projections could disagree with its rows cannot exist.
pub const ddl: [:0]const u8 =
\\CREATE TABLE domains (
\\ id INTEGER PRIMARY KEY,
@@ -78,6 +85,40 @@ pub const ddl: [:0]const u8 =
\\);
\\INSERT INTO querylog_meta (id, created_at, available_since)
\\VALUES (1, unixepoch(), unixepoch() + 1);
+ \\
+ \\CREATE TABLE bucket_totals (
+ \\ bucket INTEGER PRIMARY KEY,
+ \\ queries INTEGER NOT NULL,
+ \\ blocked INTEGER NOT NULL,
+ \\ cached INTEGER NOT NULL,
+ \\ rt_sum INTEGER NOT NULL, -- sum(response_time_us) over timed rows
+ \\ rt_count INTEGER NOT NULL -- count(response_time_us)
+ \\) WITHOUT ROWID;
+ \\
+ \\CREATE TABLE bucket_clients (
+ \\ bucket INTEGER NOT NULL,
+ \\ client_ip TEXT NOT NULL,
+ \\ queries INTEGER NOT NULL,
+ \\ PRIMARY KEY (bucket, client_ip)
+ \\) WITHOUT ROWID;
+ \\
+ \\CREATE TABLE bucket_types (
+ \\ bucket INTEGER NOT NULL,
+ \\ qtype INTEGER NOT NULL, -- -1 encodes a NULL qtype, losslessly
+ \\ count INTEGER NOT NULL,
+ \\ PRIMARY KEY (bucket, qtype)
+ \\) WITHOUT ROWID;
+ \\
+ \\CREATE TABLE bucket_routes (
+ \\ bucket INTEGER NOT NULL,
+ \\ route_kind TEXT NOT NULL,
+ \\ source_present INTEGER NOT NULL, -- 0: source NULL; 1: source = source_text
+ \\ source_text TEXT NOT NULL, -- '' when source_present = 0
+ \\ count INTEGER NOT NULL,
+ \\ PRIMARY KEY (bucket, route_kind, source_present, source_text),
+ \\ CHECK (source_present IN (0, 1)),
+ \\ CHECK (source_present = 1 OR source_text = '')
+ \\) WITHOUT ROWID;
;
/// The fingerprint of an arbitrary DDL text. `tools/cut.zig` calls this at
@@ -324,13 +365,23 @@ test "ddl creates the query-log tables and every index" {
try database.exec(ddl);
try testing.expectEqual(
- @as(i64, 3),
+ @as(i64, 7),
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
);
+ // The three explicit indexes plus `domains.domain`'s autoindex, and
+ // nothing else: the four projection tables are WITHOUT ROWID, so each
+ // one's PRIMARY KEY *is* its storage rather than a second b-tree to keep
+ // in step on every insert.
+ try testing.expectEqual(
+ @as(i64, 4),
+ try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='index'"),
+ );
const objects = [_][]const u8{
"domains", "query_log",
"idx_query_log_ts", "idx_query_log_client",
"idx_query_log_domain", "querylog_meta",
+ "bucket_totals", "bucket_clients",
+ "bucket_types", "bucket_routes",
};
for (objects) |name| {
var stmt = try database.prepare("SELECT count(*) FROM sqlite_schema WHERE name = ?1");
diff --git a/src/storage/repositories/queries_repo.zig b/src/storage/repositories/queries_repo.zig
index 0cccd6e..8e3fe8c 100644
--- a/src/storage/repositories/queries_repo.zig
+++ b/src/storage/repositories/queries_repo.zig
@@ -57,6 +57,40 @@ pub const Row = struct {
forward_zone: ?[]const u8,
};
+/// The projection grain of the four `bucket_*` tables, in seconds.
+///
+/// 1800 divides every serving width the Overview offers at or above half an
+/// hour (1800, 3600, 21600), which is what lets one grain answer the 24h, 7d
+/// and 30d windows exactly. The 1h window is 60-second buckets and is served
+/// from the raw rows instead.
+pub const grain: i64 = 1800;
+
+/// Floors a timestamp onto the projection grid.
+///
+/// `@divFloor`, never `@divTrunc` or SQLite's `/`: both truncate toward zero,
+/// which for a negative timestamp names the bucket *after* the one the row
+/// belongs to. Every producer and every consumer of a `bucket` value in this
+/// file goes through this function or an equivalent floor expression.
+pub fn bucketOf(timestamp: i64) i64 {
+ return @divFloor(timestamp, grain) * grain;
+}
+
+/// `bucket_types.qtype` is NOT NULL, so the rows that carry no query type need
+/// a value of their own. No qtype is a `u16`, so -1 cannot collide with one.
+pub const null_qtype: i64 = -1;
+
+/// The answering resolver's identity, as the routes breakdown defines it: the
+/// upstream url for `upstream` rows, the zone for `forward_zone` rows, null
+/// everywhere else. The Zig twin of `overview_raw_sql`'s CASE, and the only
+/// place the writer decides what a route's source is.
+fn routeSourceOf(row: Row) ?[]const u8 {
+ return switch (row.route_kind) {
+ .upstream => row.upstream,
+ .forward_zone => row.forward_zone,
+ else => null,
+ };
+}
+
const insert_domain_sql = "INSERT OR IGNORE INTO domains (domain) VALUES (?1)";
const select_domain_sql = "SELECT id FROM domains WHERE domain = ?1";
@@ -72,38 +106,148 @@ const insert_row_sql =
\\ ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19, ?20, ?21)
;
+/// One `bucket_totals` row's worth of change from a single batch.
+const TotalsDelta = struct {
+ queries: i64 = 0,
+ blocked: i64 = 0,
+ cached: i64 = 0,
+ rt_sum: i64 = 0,
+ rt_count: i64 = 0,
+};
+
+/// The delta-map keys borrow the batch's strings. They live only from the
+/// aggregation pass to the end of the `writeBatch` that produced them, which is
+/// inside the borrow `Row` already documents.
+const ClientKey = struct { bucket: i64, client_ip: []const u8 };
+const TypeKey = struct { bucket: i64, qtype: i64 };
+const RouteKey = struct { bucket: i64, route: provenance.RouteKind, source: ?[]const u8 };
+
+const ClientKeyContext = struct {
+ pub fn hash(_: ClientKeyContext, key: ClientKey) u64 {
+ var hasher: std.hash.Wyhash = .init(@bitCast(key.bucket));
+ hasher.update(key.client_ip);
+ return hasher.final();
+ }
+ pub fn eql(_: ClientKeyContext, a: ClientKey, b: ClientKey) bool {
+ return a.bucket == b.bucket and std.mem.eql(u8, a.client_ip, b.client_ip);
+ }
+};
+
+const RouteKeyContext = struct {
+ pub fn hash(_: RouteKeyContext, key: RouteKey) u64 {
+ var hasher: std.hash.Wyhash = .init(@bitCast(key.bucket));
+ hasher.update(@tagName(key.route));
+ hasher.update(&[_]u8{@intFromBool(key.source != null)});
+ hasher.update(key.source orelse "");
+ return hasher.final();
+ }
+ pub fn eql(_: RouteKeyContext, a: RouteKey, b: RouteKey) bool {
+ return a.bucket == b.bucket and a.route == b.route and sameSource(a.source, b.source);
+ }
+};
+
+const load_percentage = std.hash_map.default_max_load_percentage;
+
+const upsert_totals_sql =
+ \\INSERT INTO bucket_totals (bucket, queries, blocked, cached, rt_sum, rt_count)
+ \\VALUES (?1, ?2, ?3, ?4, ?5, ?6)
+ \\ON CONFLICT(bucket) DO UPDATE SET
+ \\ queries = queries + excluded.queries,
+ \\ blocked = blocked + excluded.blocked,
+ \\ cached = cached + excluded.cached,
+ \\ rt_sum = rt_sum + excluded.rt_sum,
+ \\ rt_count = rt_count + excluded.rt_count
+;
+
+const upsert_clients_sql =
+ \\INSERT INTO bucket_clients (bucket, client_ip, queries)
+ \\VALUES (?1, ?2, ?3)
+ \\ON CONFLICT(bucket, client_ip) DO UPDATE SET queries = queries + excluded.queries
+;
+
+const upsert_types_sql =
+ \\INSERT INTO bucket_types (bucket, qtype, count)
+ \\VALUES (?1, ?2, ?3)
+ \\ON CONFLICT(bucket, qtype) DO UPDATE SET count = count + excluded.count
+;
+
+const upsert_routes_sql =
+ \\INSERT INTO bucket_routes (bucket, route_kind, source_present, source_text, count)
+ \\VALUES (?1, ?2, ?3, ?4, ?5)
+ \\ON CONFLICT(bucket, route_kind, source_present, source_text)
+ \\DO UPDATE SET count = count + excluded.count
+;
+
/// Owns the prepared statements of the flush loop. Init once, reuse per batch.
///
/// `database` must outlive the writer and must not move: every `Stmt` holds a
/// `*Db`. Neither `Db` nor `Stmt` is thread-safe, so one writer belongs to one
/// task.
+///
+/// The writer also maintains the four `bucket_*` projections, in the same
+/// transaction as the rows they summarise. `gpa` is owned for the writer's life
+/// and is used for nothing but the per-batch delta maps, whose capacity is
+/// retained across batches and whose contents are dropped at the end of every
+/// `writeBatch`.
pub const BatchWriter = struct {
+ gpa: Allocator,
database: *db.Db,
insert_domain: db.Stmt,
select_domain: db.Stmt,
insert_row: db.Stmt,
+ upsert_totals: db.Stmt,
+ upsert_clients: db.Stmt,
+ upsert_types: db.Stmt,
+ upsert_routes: db.Stmt,
- pub fn init(database: *db.Db) db.Error!BatchWriter {
+ totals_deltas: std.AutoHashMapUnmanaged(i64, TotalsDelta) = .empty,
+ client_deltas: std.HashMapUnmanaged(ClientKey, i64, ClientKeyContext, load_percentage) = .empty,
+ type_deltas: std.AutoHashMapUnmanaged(TypeKey, i64) = .empty,
+ route_deltas: std.HashMapUnmanaged(RouteKey, i64, RouteKeyContext, load_percentage) = .empty,
+
+ pub fn init(gpa: Allocator, database: *db.Db) db.Error!BatchWriter {
var insert_domain = try database.prepare(insert_domain_sql);
errdefer insert_domain.deinit();
var select_domain = try database.prepare(select_domain_sql);
errdefer select_domain.deinit();
- const insert_row = try database.prepare(insert_row_sql);
+ var insert_row = try database.prepare(insert_row_sql);
+ errdefer insert_row.deinit();
+ var upsert_totals = try database.prepare(upsert_totals_sql);
+ errdefer upsert_totals.deinit();
+ var upsert_clients = try database.prepare(upsert_clients_sql);
+ errdefer upsert_clients.deinit();
+ var upsert_types = try database.prepare(upsert_types_sql);
+ errdefer upsert_types.deinit();
+ const upsert_routes = try database.prepare(upsert_routes_sql);
return .{
+ .gpa = gpa,
.database = database,
.insert_domain = insert_domain,
.select_domain = select_domain,
.insert_row = insert_row,
+ .upsert_totals = upsert_totals,
+ .upsert_clients = upsert_clients,
+ .upsert_types = upsert_types,
+ .upsert_routes = upsert_routes,
};
}
pub fn deinit(self: *BatchWriter) void {
+ self.route_deltas.deinit(self.gpa);
+ self.type_deltas.deinit(self.gpa);
+ self.client_deltas.deinit(self.gpa);
+ self.totals_deltas.deinit(self.gpa);
+ self.upsert_routes.deinit();
+ self.upsert_types.deinit();
+ self.upsert_clients.deinit();
+ self.upsert_totals.deinit();
self.insert_row.deinit();
self.select_domain.deinit();
self.insert_domain.deinit();
}
- /// One transaction for the whole batch. Domains are interned through
+ /// One transaction for the whole batch: the raw rows and every projection
+ /// they touch commit together or not at all. Domains are interned through
/// `INSERT OR IGNORE` followed by `SELECT id`.
///
/// On any failure the transaction rolls back, so a batch is all or
@@ -111,6 +255,12 @@ pub const BatchWriter = struct {
pub fn writeBatch(self: *BatchWriter, rows: []const Row) db.Error!void {
if (rows.len == 0) return;
+ // Declared before the aggregation so it runs after every `errdefer`
+ // below, and placed before `BEGIN` so an `error.OutOfMemory` fails the
+ // batch with no transaction outstanding.
+ defer self.clearDeltas();
+ try self.aggregate(rows);
+
var tx = try db.Tx.begin(self.database);
// `errdefer`s run in reverse: the statements are released before the
// ROLLBACK, so no read cursor is still open when it runs.
@@ -121,9 +271,107 @@ pub const BatchWriter = struct {
const domain_id = try self.internDomain(row.domain);
try self.write(row, domain_id);
}
+ try self.applyProjections();
try tx.commit();
}
+ /// Folds the batch into per-key deltas, so the transaction below runs one
+ /// UPSERT per touched key rather than per row. Any slice length: the
+ /// logger's 100-row batching is its own policy, not this API's.
+ fn aggregate(self: *BatchWriter, rows: []const Row) Allocator.Error!void {
+ for (rows) |row| {
+ const bucket = bucketOf(row.timestamp);
+
+ const totals = try self.totals_deltas.getOrPut(self.gpa, bucket);
+ if (!totals.found_existing) totals.value_ptr.* = .{};
+ totals.value_ptr.queries += 1;
+ if (row.blocked) totals.value_ptr.blocked += 1;
+ if (row.cache_hit orelse false) totals.value_ptr.cached += 1;
+ if (row.response_time_us) |us| {
+ totals.value_ptr.rt_sum += us;
+ totals.value_ptr.rt_count += 1;
+ }
+
+ try bump(&self.client_deltas, self.gpa, ClientKey{
+ .bucket = bucket,
+ .client_ip = row.client_ip,
+ });
+ try bump(&self.type_deltas, self.gpa, TypeKey{
+ .bucket = bucket,
+ .qtype = if (row.qtype) |v| @as(i64, v) else null_qtype,
+ });
+ try bump(&self.route_deltas, self.gpa, RouteKey{
+ .bucket = bucket,
+ .route = row.route_kind,
+ .source = routeSourceOf(row),
+ });
+ }
+ }
+
+ /// One more row on `key`. `map` is any of the three count maps — they
+ /// differ only in their key type and its hashing.
+ fn bump(map: anytype, gpa: Allocator, key: anytype) Allocator.Error!void {
+ const entry = try map.getOrPut(gpa, key);
+ if (!entry.found_existing) entry.value_ptr.* = 0;
+ entry.value_ptr.* += 1;
+ }
+
+ fn applyProjections(self: *BatchWriter) db.Error!void {
+ var totals = self.totals_deltas.iterator();
+ while (totals.next()) |entry| {
+ const stmt = &self.upsert_totals;
+ try stmt.reset();
+ try stmt.bindInt(1, entry.key_ptr.*);
+ try stmt.bindInt(2, entry.value_ptr.queries);
+ try stmt.bindInt(3, entry.value_ptr.blocked);
+ try stmt.bindInt(4, entry.value_ptr.cached);
+ try stmt.bindInt(5, entry.value_ptr.rt_sum);
+ try stmt.bindInt(6, entry.value_ptr.rt_count);
+ try stmt.exec();
+ }
+
+ var clients = self.client_deltas.iterator();
+ while (clients.next()) |entry| {
+ const stmt = &self.upsert_clients;
+ try stmt.reset();
+ try stmt.bindInt(1, entry.key_ptr.bucket);
+ try stmt.bindText(2, entry.key_ptr.client_ip);
+ try stmt.bindInt(3, entry.value_ptr.*);
+ try stmt.exec();
+ }
+
+ var types = self.type_deltas.iterator();
+ while (types.next()) |entry| {
+ const stmt = &self.upsert_types;
+ try stmt.reset();
+ try stmt.bindInt(1, entry.key_ptr.bucket);
+ try stmt.bindInt(2, entry.key_ptr.qtype);
+ try stmt.bindInt(3, entry.value_ptr.*);
+ try stmt.exec();
+ }
+
+ var routes = self.route_deltas.iterator();
+ while (routes.next()) |entry| {
+ const stmt = &self.upsert_routes;
+ try stmt.reset();
+ try stmt.bindInt(1, entry.key_ptr.bucket);
+ try stmt.bindText(2, @tagName(entry.key_ptr.route));
+ try stmt.bindInt(3, @intFromBool(entry.key_ptr.source != null));
+ try stmt.bindText(4, entry.key_ptr.source orelse "");
+ try stmt.bindInt(5, entry.value_ptr.*);
+ try stmt.exec();
+ }
+ }
+
+ /// Capacity is retained; only the entries go. The keys borrow the batch's
+ /// strings, so nothing may outlive the `writeBatch` that filled them.
+ fn clearDeltas(self: *BatchWriter) void {
+ self.totals_deltas.clearRetainingCapacity();
+ self.client_deltas.clearRetainingCapacity();
+ self.type_deltas.clearRetainingCapacity();
+ self.route_deltas.clearRetainingCapacity();
+ }
+
fn internDomain(self: *BatchWriter, domain: []const u8) db.Error!i64 {
try self.insert_domain.reset();
try self.insert_domain.bindText(1, domain);
@@ -171,6 +419,10 @@ pub const BatchWriter = struct {
/// Best effort: this runs on the failure path, where the error that
/// matters is the one already on its way to the caller.
fn resetAll(self: *BatchWriter) void {
+ self.upsert_routes.reset() catch {};
+ self.upsert_types.reset() catch {};
+ self.upsert_clients.reset() catch {};
+ self.upsert_totals.reset() catch {};
self.insert_row.reset() catch {};
self.select_domain.reset() catch {};
self.insert_domain.reset() catch {};
@@ -204,6 +456,12 @@ pub const PruneResult = struct {
/// regression. Orphaned `domains` rows stay — it is a dimension table,
/// re-interning a name costs one indexed insert, and §11.3 asks for no
/// collection.
+///
+/// The four projections are pruned in the same transaction. Buckets entirely
+/// before the cutoff's own bucket are deleted outright; a cutoff that does not
+/// land on the grid leaves one straddling bucket, which is recomputed from the
+/// rows that survived rather than reduced by an estimate. Any failure in any of
+/// that rolls back the raw delete and the watermark advance with it.
pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!PruneResult {
var tx = try db.Tx.begin(database);
errdefer tx.rollback();
@@ -214,6 +472,17 @@ pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!PruneResult {
try deleting.exec();
const deleted = database.changes();
+ const grid = bucketOf(cutoff_ts);
+ for (delete_before_bucket_sql) |sql| {
+ var stmt = try database.prepare(sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, grid);
+ try stmt.exec();
+ }
+ // On the grid there is nothing partial to fix: the delete above took whole
+ // buckets and the loop took their projection rows.
+ if (cutoff_ts != grid) try recomputeBucket(database, grid);
+
var advancing = try database.prepare(
"UPDATE querylog_meta SET available_since = max(available_since, ?1) WHERE id = 1",
);
@@ -226,6 +495,78 @@ pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!PruneResult {
return .{ .deleted = deleted, .available_since = watermark };
}
+const delete_before_bucket_sql = [_][]const u8{
+ "DELETE FROM bucket_totals WHERE bucket < ?1",
+ "DELETE FROM bucket_clients WHERE bucket < ?1",
+ "DELETE FROM bucket_types WHERE bucket < ?1",
+ "DELETE FROM bucket_routes WHERE bucket < ?1",
+};
+
+const delete_one_bucket_sql = [_][]const u8{
+ "DELETE FROM bucket_totals WHERE bucket = ?1",
+ "DELETE FROM bucket_clients WHERE bucket = ?1",
+ "DELETE FROM bucket_types WHERE bucket = ?1",
+ "DELETE FROM bucket_routes WHERE bucket = ?1",
+};
+
+/// `?1` is the bucket start and `?2` its exclusive end. Each statement rebuilds
+/// one projection table's rows for that bucket from the raw rows still in it.
+///
+/// `HAVING count(*) > 0` on the totals statement is what keeps an emptied
+/// bucket from being written back as a row of zeros: a bare aggregate always
+/// produces one row, and a from-scratch recomputation produces none. The other
+/// three group by a key, so an empty range already yields nothing.
+const recompute_bucket_sql = [_][]const u8{
+ \\INSERT INTO bucket_totals (bucket, queries, blocked, cached, rt_sum, rt_count)
+ \\SELECT ?1, count(*), coalesce(sum(blocked <> 0), 0), coalesce(sum(cache_hit = 1), 0),
+ \\ coalesce(sum(response_time_us), 0), count(response_time_us)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2
+ \\HAVING count(*) > 0
+ ,
+ \\INSERT INTO bucket_clients (bucket, client_ip, queries)
+ \\SELECT ?1, client_ip, count(*)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2
+ \\ GROUP BY client_ip
+ ,
+ \\INSERT INTO bucket_types (bucket, qtype, count)
+ \\SELECT ?1, coalesce(qtype, -1), count(*)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2
+ \\ GROUP BY coalesce(qtype, -1)
+ ,
+ \\INSERT INTO bucket_routes (bucket, route_kind, source_present, source_text, count)
+ \\SELECT ?1, route_kind, source IS NOT NULL, coalesce(source, ''), count(*)
+ \\ FROM (SELECT route_kind,
+ \\ CASE route_kind
+ \\ WHEN 'upstream' THEN upstream
+ \\ WHEN 'forward_zone' THEN forward_zone
+ \\ END AS source
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2)
+ \\ GROUP BY route_kind, source
+ ,
+};
+
+/// Replaces one bucket's projection rows with a recomputation from the raw rows
+/// that are still in it. The caller owns the transaction.
+fn recomputeBucket(database: *db.Db, bucket: i64) db.Error!void {
+ for (delete_one_bucket_sql) |sql| {
+ var stmt = try database.prepare(sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, bucket);
+ try stmt.exec();
+ }
+ for (recompute_bucket_sql) |sql| {
+ var stmt = try database.prepare(sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, bucket);
+ try stmt.bindInt(2, bucket + grain);
+ try stmt.exec();
+ }
+}
+
/// The oldest timestamp this file can still answer for. A query window that
/// starts before it is incomplete, and the API says so rather than charting the
/// gap as zero.
@@ -260,7 +601,7 @@ pub fn countDomains(database: *db.Db) db.Error!i64 {
}
// ---------------------------------------------------------------------------
-// the API read layer (`GET /api/queries`, `/api/stats`, `/api/stats/timeseries`)
+// the API read layer (`GET /api/queries`, `GET /api/overview`)
// ---------------------------------------------------------------------------
/// One row of `GET /api/queries`, joined back through the `domains` dimension.
@@ -542,8 +883,10 @@ fn likePattern(arena: Allocator, needle: []const u8) Allocator.Error![]const u8
return out.items;
}
-/// The `/api/stats` rollup for one period. `avg_response_time_us` is `null` when
-/// no row in the window recorded a response time.
+/// The Overview rollup for one window. `avg_response_time_us` is `null` when no
+/// row in the window recorded a response time. The mean is derived from a sum
+/// and a count rather than SQL's `avg`, which returns REAL: `Stmt` reads
+/// integers, and integer microseconds are exact.
pub const StatsTotals = struct {
queries: u64,
blocked: u64,
@@ -551,39 +894,6 @@ pub const StatsTotals = struct {
avg_response_time_us: ?i64,
};
-/// The mean is derived from a sum and a count rather than SQL's `avg`, which
-/// returns REAL: `Stmt` reads integers, and integer microseconds are exact.
-const stats_totals_sql =
- \\SELECT count(*),
- \\ coalesce(sum(blocked <> 0), 0),
- \\ count(DISTINCT client_ip),
- \\ coalesce(sum(response_time_us), 0),
- \\ count(response_time_us)
- \\ FROM query_log
- \\ WHERE timestamp >= ?1 AND timestamp < ?2
-;
-
-/// Aggregates `[since, until)`. An empty window is all zeros with a null mean,
-/// not an error.
-pub fn statsTotals(database: *db.Db, since: i64, until: i64) db.Error!StatsTotals {
- var stmt = try database.prepare(stats_totals_sql);
- defer stmt.deinit();
- try stmt.bindInt(1, since);
- try stmt.bindInt(2, until);
-
- // A bare aggregate always produces exactly one row; no row means the
- // statement is not the one this function prepared.
- if (!try stmt.step()) return error.Misuse;
-
- const timed = stmt.columnInt(4);
- return .{
- .queries = try countOf(stmt.columnInt(0)),
- .blocked = try countOf(stmt.columnInt(1)),
- .distinct_clients = try countOf(stmt.columnInt(2)),
- .avg_response_time_us = if (timed == 0) null else @divTrunc(stmt.columnInt(3), timed),
- };
-}
-
/// `count` and `sum` over non-negative columns cannot go negative; a negative
/// value means the row came from something other than this schema.
fn countOf(value: i64) db.Error!u64 {
@@ -591,7 +901,7 @@ fn countOf(value: i64) db.Error!u64 {
return @intCast(value);
}
-/// One bucket of `/api/stats/timeseries`. `ts` is the bucket's inclusive start.
+/// One bucket of the Overview timeseries. `ts` is the bucket's inclusive start.
pub const Bucket = struct {
ts: i64,
queries: u64,
@@ -599,169 +909,52 @@ pub const Bucket = struct {
cached: u64,
};
-const timeseries_sql =
- \\SELECT (timestamp - ?1) / ?2,
- \\ count(*),
- \\ coalesce(sum(blocked <> 0), 0),
- \\ coalesce(sum(cache_hit = 1), 0)
- \\ FROM query_log
- \\ WHERE timestamp >= ?1 AND timestamp < ?3
- \\ GROUP BY 1
-;
-
-/// Fills `out` with `out.len` buckets of `bucket_seconds` each, covering
-/// `[since, since + bucket_seconds * out.len)`, and returns how many it wrote.
-///
-/// Every bucket is present: a window with no rows in it is written with zeros
-/// rather than skipped, so the caller charts a contiguous axis without
-/// reconstructing the gaps. Buckets are aligned to `since`, so the caller —
-/// which knows the period grammar of ruling 13 — owns UTC alignment by choosing
-/// `since`.
-pub fn timeseries(database: *db.Db, since: i64, bucket_seconds: u32, out: []Bucket) db.Error!usize {
- if (out.len == 0) return 0;
- // Both are caller bugs, not runtime conditions: a zero width would make the
- // SQL divide by zero (SQLite yields NULL, silently emptying the chart), and
- // a window that does not fit i64 cannot be asked about.
- if (bucket_seconds == 0) return error.Misuse;
- const width: i64 = bucket_seconds;
- const span = std.math.mul(i64, width, std.math.cast(i64, out.len) orelse
- return error.Misuse) catch return error.Misuse;
- const until = std.math.add(i64, since, span) catch return error.Misuse;
-
- for (out, 0..) |*bucket, i| {
- bucket.* = .{ .ts = since + width * @as(i64, @intCast(i)), .queries = 0, .blocked = 0, .cached = 0 };
- }
-
- var stmt = try database.prepare(timeseries_sql);
- defer stmt.deinit();
- try stmt.bindInt(1, since);
- try stmt.bindInt(2, width);
- try stmt.bindInt(3, until);
-
- while (try stmt.step()) {
- // The WHERE clause already bounds the index to `out`; the check is
- // cheap and keeps a schema surprise from writing past the slice.
- const index = std.math.cast(usize, stmt.columnInt(0)) orelse return error.Mismatch;
- if (index >= out.len) return error.Mismatch;
- out[index].queries = try countOf(stmt.columnInt(1));
- out[index].blocked = try countOf(stmt.columnInt(2));
- out[index].cached = try countOf(stmt.columnInt(3));
- }
- return out.len;
-}
-
-/// One row of `/api/stats/types`. `qtype` is nullable in the schema, so the
-/// rows that carry no type group into a row of their own rather than
+/// One row of the Overview type breakdown. `qtype` is nullable in the schema, so
+/// the rows that carry no type group into a row of their own rather than
/// disappearing from a breakdown that claims to add up.
///
/// No name field: the only qtype-name table lives in the admin, and a second
/// copy here would drift out of agreement with it.
+///
+/// The list carries no zero rows: a type absent from the window is absent from
+/// it. Its order is an order, not an identity — a caller keys on the `qtype`
+/// value, never on a row's position, because a rank change between refreshes
+/// moves rows and must not move what they mean.
pub const TypeCount = struct {
qtype: ?u16,
count: u64,
};
-/// `qtype IS NULL` sorts 0 before 1, which puts the null row last within a tie.
-/// The ordering is total, so two reads of one window return the same list in
-/// the same order — which is what makes the goldens byte-stable. It is an
-/// order, not an identity: a caller keys on the `qtype` value, never on a row's
-/// position, because a rank change between refreshes moves rows and must not
-/// move what they mean.
-const stats_types_sql =
- \\SELECT qtype, count(*)
- \\ FROM query_log
- \\ WHERE timestamp >= ?1 AND timestamp < ?2
- \\ GROUP BY qtype
- \\ ORDER BY count(*) DESC, qtype IS NULL, qtype ASC
-;
-
-/// The query-type breakdown of `[since, until)`. No zero rows: a type absent
-/// from the window is absent from the list.
-pub fn statsTypes(
- database: *db.Db,
- arena: Allocator,
- since: i64,
- until: i64,
-) db.Error!std.ArrayList(TypeCount) {
- var out: std.ArrayList(TypeCount) = .empty;
-
- var stmt = try database.prepare(stats_types_sql);
- defer stmt.deinit();
- try stmt.bindInt(1, since);
- try stmt.bindInt(2, until);
-
- while (try stmt.step()) {
- try out.append(arena, .{
- .qtype = if (stmt.isNull(0)) null else try columnU16(&stmt, 0),
- .count = try countOf(stmt.columnInt(1)),
- });
- }
- return out;
-}
-
-/// One row of `/api/stats/routes`: how a slice of the window was answered.
+/// One row of the Overview route breakdown: how a slice of the window was
+/// answered.
///
/// `source` is the answering resolver's identity and nothing else — the
/// upstream url for `upstream` rows, the zone for `forward_zone` rows, null
/// everywhere else. It is deliberately not `source_name`, which names the
/// blocklist a block came from and would read as an upstream here.
+///
+/// A null source on an `upstream` row is its own group, not a dropped row: it
+/// is a real state of the log and the caller labels it.
pub const RouteCount = struct {
route: provenance.RouteKind,
source: ?[]const u8,
count: u64,
};
-/// A null upstream on an `upstream` row is its own group, not a dropped row: it
-/// is a real state of the log and the caller labels it.
-const stats_routes_sql =
- \\SELECT route_kind,
- \\ CASE route_kind
- \\ WHEN 'upstream' THEN upstream
- \\ WHEN 'forward_zone' THEN forward_zone
- \\ END AS source,
- \\ count(*)
- \\ FROM query_log
- \\ WHERE timestamp >= ?1 AND timestamp < ?2
- \\ GROUP BY route_kind, source
- \\ ORDER BY count(*) DESC, route_kind ASC, source IS NULL, source ASC
-;
-
-/// The answering-route breakdown of `[since, until)`. Strings are copied into
-/// `arena`, which outlives the statement.
-pub fn statsRoutes(
- database: *db.Db,
- arena: Allocator,
- since: i64,
- until: i64,
-) db.Error!std.ArrayList(RouteCount) {
- var out: std.ArrayList(RouteCount) = .empty;
-
- var stmt = try database.prepare(stats_routes_sql);
- defer stmt.deinit();
- try stmt.bindInt(1, since);
- try stmt.bindInt(2, until);
-
- while (try stmt.step()) {
- try out.append(arena, .{
- .route = try provenance.parse(provenance.RouteKind, stmt.columnText(0)),
- .source = try stmt.columnTextAllocOrNull(arena, 1),
- .count = try countOf(stmt.columnInt(2)),
- });
- }
- return out;
-}
-
-/// How many clients `/api/stats/clients` names before the rest become `other`.
-/// Eight is what one legend can carry without becoming a second table.
+/// How many clients the Overview names before the rest become `other`. Eight is
+/// what one legend can carry without becoming a second table.
pub const max_client_series = 8;
/// One named client's series. `buckets` is always the caller's bucket count
-/// long, zero-filled, and aligned exactly like `timeseries`.
+/// long, zero-filled, and aligned exactly like `Overview.buckets`.
pub const ClientSeries = struct {
client: []const u8,
buckets: []const u64,
};
+/// Named clients are ranked by in-window total, ties broken by address, so the
+/// cut at `max_client_series` is the same cut on every read over the same data.
+///
/// `other` is always present and always bucket-count sized — including for an
/// empty window and for a window with eight clients or fewer. A caller charting
/// a stack must not have to invent the residual series.
@@ -770,90 +963,365 @@ pub const ClientsBreakdown = struct {
other: []const u64,
};
-/// Ranked by in-window total, ties broken by address, so the cut at eight is
-/// the same cut on every request over the same data.
-const stats_clients_rank_sql =
- \\SELECT client_ip
+fn zeroedBuckets(arena: Allocator, bucket_count: u32) Allocator.Error![]u64 {
+ const buckets = try arena.alloc(u64, bucket_count);
+ @memset(buckets, 0);
+ return buckets;
+}
+
+// ---------------------------------------------------------------------------
+// the Overview read path (milestone 36)
+// ---------------------------------------------------------------------------
+
+/// Everything one Overview response reports about one window, from one read of
+/// one already-open transaction.
+///
+/// Storage owns these scalars: `since`, `bucket_seconds` and `bucket_count` are
+/// numbers, not a web module's period type, so nothing here has to know the
+/// period grammar the handler enforces.
+pub const Overview = struct {
+ totals: StatsTotals,
+ /// `bucket_count` entries, one per bucket of the window, zero-filled and
+ /// aligned to `since`.
+ buckets: []const Bucket,
+ clients: ClientsBreakdown,
+ types: []const TypeCount,
+ routes: []const RouteCount,
+};
+
+/// One client's running window total beside its per-bucket series, so the cut
+/// at `max_client_series` can be made after the whole window is folded rather
+/// than by a second pass over the data.
+const ClientAccum = struct {
+ client: []const u8,
+ total: u64,
+ buckets: []u64,
+};
+
+/// The shared shape both paths fold into. Every allocation is from `arena`, so
+/// there is nothing to unwind on failure.
+const Accum = struct {
+ arena: Allocator,
+ since: i64,
+ width: i64,
+ bucket_count: u32,
+ buckets: []Bucket,
+ rt_sum: i64 = 0,
+ rt_count: i64 = 0,
+ clients: std.ArrayList(ClientAccum) = .empty,
+ /// `client_ip` to its index in `clients`. The names are arena copies, so
+ /// they outlive the statement that produced them.
+ client_index: std.StringHashMapUnmanaged(u32) = .empty,
+ types: std.ArrayList(TypeCount) = .empty,
+ type_index: std.AutoHashMapUnmanaged(i64, u32) = .empty,
+ /// Household cardinality: a handful of upstreams plus six route kinds. A
+ /// linear scan is the whole index this needs.
+ routes: std.ArrayList(RouteCount) = .empty,
+
+ fn init(arena: Allocator, since: i64, width: i64, bucket_count: u32) Allocator.Error!Accum {
+ const buckets = try arena.alloc(Bucket, bucket_count);
+ for (buckets, 0..) |*bucket, i| bucket.* = .{
+ .ts = since + width * @as(i64, @intCast(i)),
+ .queries = 0,
+ .blocked = 0,
+ .cached = 0,
+ };
+ return .{ .arena = arena, .since = since, .width = width, .bucket_count = bucket_count, .buckets = buckets };
+ }
+
+ /// The serving-bucket index a grid bucket or a raw timestamp falls in.
+ /// Both paths bound their reads by the window already; the check is what
+ /// keeps a schema surprise from writing past the slice.
+ fn indexOf(self: *const Accum, timestamp: i64) db.Error!usize {
+ const offset = @divFloor(timestamp - self.since, self.width);
+ const index = std.math.cast(usize, offset) orelse return error.Mismatch;
+ if (index >= self.buckets.len) return error.Mismatch;
+ return index;
+ }
+
+ fn addTotals(self: *Accum, index: usize, queries: u64, blocked: u64, cached: u64) void {
+ self.buckets[index].queries += queries;
+ self.buckets[index].blocked += blocked;
+ self.buckets[index].cached += cached;
+ }
+
+ fn addClient(self: *Accum, client: []const u8, index: usize, count: u64) Allocator.Error!void {
+ const slot = try self.client_index.getOrPut(self.arena, client);
+ if (!slot.found_existing) {
+ const owned = try self.arena.dupe(u8, client);
+ slot.key_ptr.* = owned;
+ slot.value_ptr.* = @intCast(self.clients.items.len);
+ try self.clients.append(self.arena, .{
+ .client = owned,
+ .total = 0,
+ .buckets = try zeroedBuckets(self.arena, self.bucket_count),
+ });
+ }
+ const accum = &self.clients.items[slot.value_ptr.*];
+ accum.total += count;
+ accum.buckets[index] += count;
+ }
+
+ /// `qtype` is the stored encoding: `null_qtype` for the rows that carry no
+ /// query type.
+ fn addType(self: *Accum, qtype: i64, count: u64) db.Error!void {
+ const slot = try self.type_index.getOrPut(self.arena, qtype);
+ if (!slot.found_existing) {
+ slot.value_ptr.* = @intCast(self.types.items.len);
+ try self.types.append(self.arena, .{
+ .qtype = if (qtype == null_qtype) null else std.math.cast(u16, qtype) orelse
+ return error.Mismatch,
+ .count = 0,
+ });
+ }
+ self.types.items[slot.value_ptr.*].count += count;
+ }
+
+ fn addRoute(self: *Accum, route: provenance.RouteKind, source: ?[]const u8, count: u64) Allocator.Error!void {
+ for (self.routes.items) |*existing| {
+ if (existing.route == route and sameSource(existing.source, source)) {
+ existing.count += count;
+ return;
+ }
+ }
+ try self.routes.append(self.arena, .{
+ .route = route,
+ .source = if (source) |s| try self.arena.dupe(u8, s) else null,
+ .count = count,
+ });
+ }
+
+ fn finish(self: *Accum) Allocator.Error!Overview {
+ var queries: u64 = 0;
+ var blocked: u64 = 0;
+ for (self.buckets) |bucket| {
+ queries += bucket.queries;
+ blocked += bucket.blocked;
+ }
+
+ // Every comparator below is a total order on a unique key, so an
+ // unstable sort still produces one list for one window.
+ std.sort.pdq(TypeCount, self.types.items, {}, typeBefore);
+ std.sort.pdq(RouteCount, self.routes.items, {}, routeBefore);
+ std.sort.pdq(ClientAccum, self.clients.items, {}, clientBefore);
+
+ const named = @min(self.clients.items.len, max_client_series);
+ const other = try zeroedBuckets(self.arena, self.bucket_count);
+ for (self.clients.items[named..]) |folded| {
+ for (other, folded.buckets) |*slot, count| slot.* += count;
+ }
+ const series = try self.arena.alloc(ClientSeries, named);
+ for (series, self.clients.items[0..named]) |*entry, accum| {
+ entry.* = .{ .client = accum.client, .buckets = accum.buckets };
+ }
+
+ return .{
+ .totals = .{
+ .queries = queries,
+ .blocked = blocked,
+ .distinct_clients = self.clients.items.len,
+ .avg_response_time_us = if (self.rt_count == 0)
+ null
+ else
+ @divTrunc(self.rt_sum, self.rt_count),
+ },
+ .buckets = self.buckets,
+ .clients = .{ .clients = series, .other = other },
+ .types = self.types.items,
+ .routes = self.routes.items,
+ };
+ }
+};
+
+/// Absent and present are different sources, and so are two different present
+/// ones: an `upstream` row the log recorded no resolver for is its own group,
+/// exactly as `GROUP BY ... source` makes it.
+fn sameSource(a: ?[]const u8, b: ?[]const u8) bool {
+ const left = a orelse return b == null;
+ const right = b orelse return false;
+ return std.mem.eql(u8, left, right);
+}
+
+/// The oracle's `ORDER BY count(*) DESC, qtype IS NULL, qtype ASC`.
+fn typeBefore(_: void, a: TypeCount, b: TypeCount) bool {
+ if (a.count != b.count) return a.count > b.count;
+ const a_type = a.qtype orelse return false;
+ const b_type = b.qtype orelse return true;
+ return a_type < b_type;
+}
+
+/// The oracle's `ORDER BY count(*) DESC, route_kind ASC, source IS NULL,
+/// source ASC`. `route_kind` is compared as the stored text, not as the
+/// enum's declaration order — the two disagree, and the stored text is what the
+/// SQL sorted.
+fn routeBefore(_: void, a: RouteCount, b: RouteCount) bool {
+ if (a.count != b.count) return a.count > b.count;
+ switch (std.mem.order(u8, @tagName(a.route), @tagName(b.route))) {
+ .lt => return true,
+ .gt => return false,
+ .eq => {},
+ }
+ const a_source = a.source orelse return false;
+ const b_source = b.source orelse return true;
+ return std.mem.order(u8, a_source, b_source) == .lt;
+}
+
+/// The oracle's `ORDER BY count(*) DESC, client_ip ASC`, which is what makes
+/// the cut at `max_client_series` the same cut on every read.
+fn clientBefore(_: void, a: ClientAccum, b: ClientAccum) bool {
+ if (a.total != b.total) return a.total > b.total;
+ return std.mem.order(u8, a.client, b.client) == .lt;
+}
+
+const overview_projection_totals_sql =
+ \\SELECT bucket, queries, blocked, cached, rt_sum, rt_count
+ \\ FROM bucket_totals
+ \\ WHERE bucket >= ?1 AND bucket < ?2
+;
+
+const overview_projection_clients_sql =
+ \\SELECT bucket, client_ip, queries
+ \\ FROM bucket_clients
+ \\ WHERE bucket >= ?1 AND bucket < ?2
+;
+
+const overview_projection_types_sql =
+ \\SELECT qtype, sum(count)
+ \\ FROM bucket_types
+ \\ WHERE bucket >= ?1 AND bucket < ?2
+ \\ GROUP BY qtype
+;
+
+const overview_projection_routes_sql =
+ \\SELECT route_kind, source_present, source_text, sum(count)
+ \\ FROM bucket_routes
+ \\ WHERE bucket >= ?1 AND bucket < ?2
+ \\ GROUP BY route_kind, source_present, source_text
+;
+
+const overview_raw_sql =
+ \\SELECT timestamp, client_ip, qtype, blocked, cache_hit, response_time_us,
+ \\ route_kind,
+ \\ CASE route_kind
+ \\ WHEN 'upstream' THEN upstream
+ \\ WHEN 'forward_zone' THEN forward_zone
+ \\ END AS source
\\ FROM query_log
\\ WHERE timestamp >= ?1 AND timestamp < ?2
- \\ GROUP BY client_ip
- \\ ORDER BY count(*) DESC, client_ip ASC
- \\ LIMIT ?3
;
-const stats_clients_buckets_sql =
- \\SELECT client_ip, (timestamp - ?1) / ?2, count(*)
- \\ FROM query_log
- \\ WHERE timestamp >= ?1 AND timestamp < ?3
- \\ GROUP BY 1, 2
-;
-
-/// Per-client counts over `bucket_count` buckets of `bucket_seconds` starting
-/// at `since`. Everything outside the top `max_client_series` sums into
-/// `other`, so the series still add up to the window's total.
+/// The whole Overview payload for `[since, since + bucket_seconds *
+/// bucket_count)`, from one already-open read transaction — the caller owns the
+/// transaction and the lock, as it does for every other read here.
///
-/// Two statements, one ranking and one bucketing: the caller runs them inside
-/// one read transaction, so the rank and the buckets describe one state.
-pub fn statsClients(
+/// Two implementations behind one entry point. At or above the projection grain
+/// the four `bucket_*` tables answer the window, and the cost stops depending
+/// on how many raw rows it holds; below it (the 1h window, on 60-second
+/// buckets) one pass over the raw rows in the window does the same work in Zig.
+/// Both produce the same contracts, which is what the equivalence test asserts.
+///
+/// Preconditions are caller bugs, not runtime conditions: a zero width or count
+/// is `error.Misuse`, and so is a projection-path window that is not on the
+/// grid, because the projections cannot express it. The handler's `window()`
+/// guarantees all of them.
+pub fn overview(
database: *db.Db,
arena: Allocator,
since: i64,
bucket_seconds: u32,
bucket_count: u32,
-) db.Error!ClientsBreakdown {
+) db.Error!Overview {
if (bucket_seconds == 0 or bucket_count == 0) return error.Misuse;
const width: i64 = bucket_seconds;
const span = std.math.mul(i64, width, bucket_count) catch return error.Misuse;
const until = std.math.add(i64, since, span) catch return error.Misuse;
+ const from_projections = width >= grain;
+ if (from_projections and (@mod(since, grain) != 0 or @rem(width, grain) != 0)) return error.Misuse;
- var names: std.ArrayList([]const u8) = .empty;
- var series: std.ArrayList([]u64) = .empty;
+ var accum = try Accum.init(arena, since, width, bucket_count);
+ if (from_projections) {
+ try readProjections(database, &accum, since, until);
+ } else {
+ try readRaw(database, &accum, since, until);
+ }
+ return try accum.finish();
+}
+
+fn readProjections(database: *db.Db, accum: *Accum, since: i64, until: i64) db.Error!void {
{
- var stmt = try database.prepare(stats_clients_rank_sql);
+ var stmt = try database.prepare(overview_projection_totals_sql);
defer stmt.deinit();
try stmt.bindInt(1, since);
try stmt.bindInt(2, until);
- try stmt.bindInt(3, max_client_series);
-
while (try stmt.step()) {
- try names.append(arena, try stmt.columnTextAlloc(arena, 0));
- try series.append(arena, try zeroedBuckets(arena, bucket_count));
+ const index = try accum.indexOf(stmt.columnInt(0));
+ accum.addTotals(
+ index,
+ try countOf(stmt.columnInt(1)),
+ try countOf(stmt.columnInt(2)),
+ try countOf(stmt.columnInt(3)),
+ );
+ accum.rt_sum += stmt.columnInt(4);
+ accum.rt_count += stmt.columnInt(5);
}
}
-
- const other = try zeroedBuckets(arena, bucket_count);
-
- var stmt = try database.prepare(stats_clients_buckets_sql);
- defer stmt.deinit();
- try stmt.bindInt(1, since);
- try stmt.bindInt(2, width);
- try stmt.bindInt(3, until);
-
- while (try stmt.step()) {
- // The WHERE clause bounds the index already; the check keeps a schema
- // surprise from writing past the slice.
- const index = std.math.cast(usize, stmt.columnInt(1)) orelse return error.Mismatch;
- if (index >= bucket_count) return error.Mismatch;
- const count = try countOf(stmt.columnInt(2));
-
- const client = stmt.columnText(0);
- const target = for (names.items, series.items) |name_, buckets| {
- if (std.mem.eql(u8, name_, client)) break buckets;
- } else other;
- target[index] += count;
+ {
+ var stmt = try database.prepare(overview_projection_clients_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+ while (try stmt.step()) {
+ const index = try accum.indexOf(stmt.columnInt(0));
+ try accum.addClient(stmt.columnText(1), index, try countOf(stmt.columnInt(2)));
+ }
}
-
- const clients = try arena.alloc(ClientSeries, names.items.len);
- for (clients, names.items, series.items) |*entry, name_, buckets| {
- entry.* = .{ .client = name_, .buckets = buckets };
+ {
+ var stmt = try database.prepare(overview_projection_types_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+ while (try stmt.step()) {
+ try accum.addType(stmt.columnInt(0), try countOf(stmt.columnInt(1)));
+ }
+ }
+ {
+ var stmt = try database.prepare(overview_projection_routes_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+ while (try stmt.step()) {
+ const route = try provenance.parse(provenance.RouteKind, stmt.columnText(0));
+ const source: ?[]const u8 = if (stmt.columnInt(1) == 0) null else stmt.columnText(2);
+ try accum.addRoute(route, source, try countOf(stmt.columnInt(3)));
+ }
}
- return .{ .clients = clients, .other = other };
}
-fn zeroedBuckets(arena: Allocator, bucket_count: u32) Allocator.Error![]u64 {
- const buckets = try arena.alloc(u64, bucket_count);
- @memset(buckets, 0);
- return buckets;
+fn readRaw(database: *db.Db, accum: *Accum, since: i64, until: i64) db.Error!void {
+ var stmt = try database.prepare(overview_raw_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+
+ while (try stmt.step()) {
+ const index = try accum.indexOf(stmt.columnInt(0));
+ const blocked: u64 = @intFromBool(stmt.columnBool(3));
+ // `cache_hit = 1`, not "non-zero": the SQL aggregates this replaces
+ // compare against the literal, and a NULL compares to neither.
+ const cached: u64 = @intFromBool(!stmt.isNull(4) and stmt.columnInt(4) == 1);
+ accum.addTotals(index, 1, blocked, cached);
+ if (!stmt.isNull(5)) {
+ accum.rt_sum += stmt.columnInt(5);
+ accum.rt_count += 1;
+ }
+ try accum.addClient(stmt.columnText(1), index, 1);
+ try accum.addType(if (stmt.isNull(2)) null_qtype else stmt.columnInt(2), 1);
+ try accum.addRoute(
+ try provenance.parse(provenance.RouteKind, stmt.columnText(6)),
+ stmt.columnTextOrNull(7),
+ 1,
+ );
+ }
}
// ---------------------------------------------------------------------------
@@ -947,7 +1415,7 @@ test "a foreign row with an rcode wider than twelve bits is refused, not truncat
test "writeBatch inserts every row and interns each domain once" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
try writer.writeBatch(&.{
@@ -970,7 +1438,7 @@ test "writeBatch inserts every row and interns each domain once" {
test "a second batch reuses the interned domain id" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
try writer.writeBatch(&.{plainRow(100, "example.com")});
@@ -991,7 +1459,7 @@ test "a second batch reuses the interned domain id" {
test "every column round-trips a value and a null" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
try writer.writeBatch(&.{
@@ -1124,7 +1592,7 @@ test "a stored enum value the schema does not define is a data error, not a pass
test "an empty batch writes nothing and opens no transaction" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
// A transaction is already open, so a `BEGIN IMMEDIATE` from `writeBatch`
@@ -1140,7 +1608,7 @@ test "an empty batch writes nothing and opens no transaction" {
test "pruneOlderThan deletes strictly older rows and returns the count" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
try writer.writeBatch(&.{
@@ -1164,7 +1632,7 @@ test "pruneOlderThan deletes strictly older rows and returns the count" {
test "a prune advances the coverage watermark to its own cutoff" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
// The seeded watermark is `created_at + 1`, which is now-ish; the cutoffs
@@ -1204,7 +1672,7 @@ test "the watermark never moves backward" {
test "a failed delete leaves both the rows and the watermark untouched" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
const start = try availableSince(&database);
@@ -1223,7 +1691,7 @@ test "a failed delete leaves both the rows and the watermark untouched" {
test "a failed watermark update leaves the rows it had already deleted" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
const start = try availableSince(&database);
@@ -1245,7 +1713,7 @@ test "a failed watermark update leaves the rows it had already deleted" {
test "a failed commit rolls back the delete and the watermark together" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
const start = try availableSince(&database);
@@ -1280,7 +1748,7 @@ test "a failed commit rolls back the delete and the watermark together" {
test "pruneOlderThan leaves the domains dimension table intact" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(11, "b.example") });
@@ -1299,7 +1767,7 @@ test "a failing row rolls the whole batch back and the writer survives it" {
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
var doomed = plainRow(20, "second.example");
@@ -1321,7 +1789,7 @@ test "a failing row rolls the whole batch back and the writer survives it" {
test "countRows and countDomains agree with what the batches wrote" {
var database = try openLog();
defer database.close();
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
try testing.expectEqual(@as(i64, 0), try countRows(&database));
@@ -1366,7 +1834,7 @@ test "checkpointTruncate and vacuum run against a WAL file database" {
}
try database.exec(querylog_schema.ddl);
- var writer = try BatchWriter.init(&database);
+ var writer = try BatchWriter.init(testing.allocator, &database);
defer writer.deinit();
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(20, "b.example") });
@@ -1385,7 +1853,7 @@ test "checkpointTruncate and vacuum run against a WAL file database" {
/// every test below knows the id of each seeded row: the nth row of the nth
/// batch has id n.
fn seed(database: *db.Db, rows: []const Row) !void {
- var writer = try BatchWriter.init(database);
+ var writer = try BatchWriter.init(testing.allocator, database);
defer writer.deinit();
try writer.writeBatch(rows);
}
@@ -1652,114 +2120,6 @@ test "a domain substring matches % and _ as literal characters" {
try testing.expectEqual(@as(usize, 5), all.items.len);
}
-test "statsTotals aggregates the window and averages only the timed rows" {
- var database = try openLog();
- defer database.close();
-
- var timed = plainRow(100, "a.example");
- timed.response_time_us = 100;
- var blocked_row = plainRow(150, "ads.example");
- blocked_row.blocked = true;
- blocked_row.policy_action = .block;
- blocked_row.policy_reason = .blocklist_domain;
- blocked_row.route_kind = .blocked;
- blocked_row.response_time_us = 200;
- var cached = plainRow(199, "b.example");
- cached.client_ip = "192.0.2.99";
- cached.cache_hit = true;
- cached.response_time_us = null;
- try seed(&database, &.{ timed, blocked_row, cached, plainRow(200, "outside.example") });
-
- const totals = try statsTotals(&database, 100, 200);
- try testing.expectEqual(@as(u64, 3), totals.queries);
- try testing.expectEqual(@as(u64, 1), totals.blocked);
- try testing.expectEqual(@as(u64, 2), totals.distinct_clients);
- // (100 + 200) / 2 — the untimed row is not in the divisor.
- try testing.expectEqual(@as(?i64, 150), totals.avg_response_time_us);
-}
-
-test "statsTotals over an empty window is zeros with a null average" {
- var database = try openLog();
- defer database.close();
- try seed(&database, &.{plainRow(100, "a.example")});
-
- for ([_][2]i64{ .{ 500, 600 }, .{ 100, 100 } }) |window| {
- const totals = try statsTotals(&database, window[0], window[1]);
- try testing.expectEqual(@as(u64, 0), totals.queries);
- try testing.expectEqual(@as(u64, 0), totals.blocked);
- try testing.expectEqual(@as(u64, 0), totals.distinct_clients);
- try testing.expectEqual(@as(?i64, null), totals.avg_response_time_us);
- }
-}
-
-test "timeseries writes every bucket, including the ones with no rows" {
- var database = try openLog();
- defer database.close();
-
- var blocked_row = plainRow(1020, "ads.example");
- blocked_row.blocked = true;
- blocked_row.policy_action = .block;
- blocked_row.policy_reason = .blocklist_domain;
- blocked_row.route_kind = .blocked;
- var cached = plainRow(1035, "b.example");
- cached.cache_hit = true;
- try seed(&database, &.{
- plainRow(995, "before.example"),
- plainRow(1000, "a.example"),
- plainRow(1009, "a.example"),
- blocked_row,
- cached,
- plainRow(1040, "after.example"),
- });
-
- var buckets: [4]Bucket = undefined;
- try testing.expectEqual(@as(usize, 4), try timeseries(&database, 1000, 10, &buckets));
-
- // The row at 995 is before the window and the row at 1040 is past its end;
- // neither lands in a bucket.
- try testing.expectEqualSlices(Bucket, &.{
- .{ .ts = 1000, .queries = 2, .blocked = 0, .cached = 0 },
- .{ .ts = 1010, .queries = 0, .blocked = 0, .cached = 0 },
- .{ .ts = 1020, .queries = 1, .blocked = 1, .cached = 0 },
- .{ .ts = 1030, .queries = 1, .blocked = 0, .cached = 1 },
- }, &buckets);
-}
-
-test "timeseries over an empty table still writes the whole axis" {
- var database = try openLog();
- defer database.close();
-
- var buckets: [3]Bucket = undefined;
- try testing.expectEqual(@as(usize, 3), try timeseries(&database, 0, 60, &buckets));
- try testing.expectEqualSlices(Bucket, &.{
- .{ .ts = 0, .queries = 0, .blocked = 0, .cached = 0 },
- .{ .ts = 60, .queries = 0, .blocked = 0, .cached = 0 },
- .{ .ts = 120, .queries = 0, .blocked = 0, .cached = 0 },
- }, &buckets);
-}
-
-test "timeseries rejects a zero-width bucket and accepts an empty slice" {
- var database = try openLog();
- defer database.close();
-
- var buckets: [2]Bucket = undefined;
- try testing.expectError(error.Misuse, timeseries(&database, 0, 0, &buckets));
-
- var none: [0]Bucket = undefined;
- try testing.expectEqual(@as(usize, 0), try timeseries(&database, 0, 0, &none));
-}
-
-test "timeseries reports a window that does not fit an i64 rather than wrapping" {
- var database = try openLog();
- defer database.close();
-
- var buckets: [4]Bucket = undefined;
- try testing.expectError(
- error.Misuse,
- timeseries(&database, std.math.maxInt(i64) - 1, 3600, &buckets),
- );
-}
-
test "the built SQL never carries a filter value and fits its buffer" {
var sql: Sql = .{};
sql.put(select_head);
@@ -1798,7 +2158,6 @@ test "likePattern wraps the needle and neutralises every metacharacter" {
const agg_since: i64 = 1_700_000_000;
const agg_width: u32 = 60;
const agg_buckets: u32 = 10;
-const agg_until: i64 = agg_since + agg_width * agg_buckets;
/// One row of the aggregation fixtures. Everything the three breakdowns read
/// is a parameter; everything else is the same on every row, so a test that
@@ -1854,7 +2213,7 @@ test "the type breakdown groups by qtype, keeps the null row and orders it last"
aggRow(-1, "192.0.2.10", 255, .upstream, "9.9.9.9"),
});
- const rows = (try statsTypes(&database, arena, agg_since, agg_until)).items;
+ const rows = (try overview(&database, arena, agg_since, agg_width, agg_buckets)).types;
try testing.expectEqual(@as(usize, 4), rows.len);
try testing.expectEqual(@as(?u16, 1), rows[0].qtype);
try testing.expectEqual(@as(u64, 3), rows[0].count);
@@ -1870,7 +2229,7 @@ test "an empty window has no type rows at all" {
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
- const rows = (try statsTypes(&database, arena_state.allocator(), agg_since, agg_until)).items;
+ const rows = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).types;
try testing.expectEqual(@as(usize, 0), rows.len);
}
@@ -1894,7 +2253,7 @@ test "the route breakdown keys on the answering resolver, not on blocklist prove
aggRow(8, "192.0.2.10", 1, .rejected, null),
});
- const rows = (try statsRoutes(&database, arena, agg_since, agg_until)).items;
+ const rows = (try overview(&database, arena, agg_since, agg_width, agg_buckets)).routes;
// Two upstreams, one null-source upstream, one forward zone and four
// source-less kinds. Every row carries the same `source_name`, so a
// breakdown that grouped by it would collapse to one row.
@@ -1926,7 +2285,7 @@ test "an empty window has no route rows at all" {
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
- const rows = (try statsRoutes(&database, arena_state.allocator(), agg_since, agg_until)).items;
+ const rows = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).routes;
try testing.expectEqual(@as(usize, 0), rows.len);
}
@@ -1936,7 +2295,7 @@ test "an empty window still has a zero-filled other series and no named clients"
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
- const result = try statsClients(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets);
+ const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).clients;
try testing.expectEqual(@as(usize, 0), result.clients.len);
try testing.expectEqual(@as(usize, agg_buckets), result.other.len);
for (result.other) |count| try testing.expectEqual(@as(u64, 0), count);
@@ -1958,7 +2317,7 @@ test "client series are bucket-aligned, zero-filled and ranked by in-window tota
aggRow(agg_width * agg_buckets, "192.0.2.10", 1, .upstream, "9.9.9.9"),
});
- const result = try statsClients(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets);
+ const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).clients;
try testing.expectEqual(@as(usize, 2), result.clients.len);
try testing.expectEqualStrings("192.0.2.20", result.clients[0].client);
try testing.expectEqualStrings("192.0.2.10", result.clients[1].client);
@@ -1990,7 +2349,7 @@ test "the ninth client folds into other and the cut is the same on every read" {
}
}
- const result = try statsClients(&database, arena, agg_since, agg_width, agg_buckets);
+ const result = (try overview(&database, arena, agg_since, agg_width, agg_buckets)).clients;
try testing.expectEqual(@as(usize, max_client_series), result.clients.len);
// The busiest is 192.0.2.108 with nine rows; the lone folded client is
// 192.0.2.100 with one.
@@ -2029,25 +2388,22 @@ test "the three breakdowns conserve the window's total" {
)});
}
- const totals = try statsTotals(&database, agg_since, agg_until);
- try testing.expect(totals.queries > 0);
+ const result = try overview(&database, arena, agg_since, agg_width, agg_buckets);
+ try testing.expect(result.totals.queries > 0);
var typed: u64 = 0;
- for ((try statsTypes(&database, arena, agg_since, agg_until)).items) |row| typed += row.count;
- try testing.expectEqual(totals.queries, typed);
+ for (result.types) |row| typed += row.count;
+ try testing.expectEqual(result.totals.queries, typed);
var routed: u64 = 0;
- for ((try statsRoutes(&database, arena, agg_since, agg_until)).items) |row| routed += row.count;
- try testing.expectEqual(totals.queries, routed);
+ for (result.routes) |row| routed += row.count;
+ try testing.expectEqual(result.totals.queries, routed);
- var buckets: [agg_buckets]Bucket = undefined;
- _ = try timeseries(&database, agg_since, agg_width, &buckets);
- const clients = try statsClients(&database, arena, agg_since, agg_width, agg_buckets);
// Per bucket, not just in total: a series misaligned by one bucket would
// still sum correctly over the window.
- for (buckets, 0..) |bucket, at| {
- var summed: u64 = clients.other[at];
- for (clients.clients) |series| summed += series.buckets[at];
+ for (result.buckets, 0..) |bucket, at| {
+ var summed: u64 = result.clients.other[at];
+ for (result.clients.clients) |series| summed += series.buckets[at];
try testing.expectEqual(bucket.queries, summed);
}
}
@@ -2066,7 +2422,7 @@ test "the aggregations pass a redacted client through as the log stored it" {
aggRow(1, logger.hidden_marker, 1, .upstream, "9.9.9.9"),
});
- const result = try statsClients(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets);
+ const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).clients;
try testing.expectEqual(@as(usize, 1), result.clients.len);
try testing.expectEqualStrings(logger.hidden_marker, result.clients[0].client);
try testing.expectEqual(@as(u64, 2), result.clients[0].buckets[0]);
@@ -2099,8 +2455,12 @@ test "a prune committed mid-read is invisible to the reader's transaction" {
defer pruner.close();
try db.applyPragmas(&pruner, .{});
+ var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
+ defer arena_state.deinit();
+ const arena = arena_state.allocator();
+
var tx = try db.ReadTx.begin(&reader);
- const before = try statsTotals(&reader, agg_since, agg_since + 1000);
+ const before = (try overview(&reader, arena, agg_since, 1000, 1)).totals;
const watermark_before = try availableSince(&reader);
try testing.expectEqual(@as(u64, 3), before.queries);
@@ -2111,13 +2471,685 @@ test "a prune committed mid-read is invisible to the reader's transaction" {
// Neither half of the answer moved: the rows the reader would report and
// the watermark it would tag them with still describe one state.
- const during = try statsTotals(&reader, agg_since, agg_since + 1000);
+ const during = (try overview(&reader, arena, agg_since, 1000, 1)).totals;
try testing.expectEqual(before.queries, during.queries);
try testing.expectEqual(watermark_before, try availableSince(&reader));
try tx.commit();
// The next response sees the prune — both halves of it.
- const after = try statsTotals(&reader, agg_since, agg_since + 1000);
+ const after = (try overview(&reader, arena, agg_since, 1000, 1)).totals;
try testing.expectEqual(@as(u64, 1), after.queries);
try testing.expectEqual(pruned.available_since, try availableSince(&reader));
}
+
+// ---------------------------------------------------------------------------
+// the projections and the Overview read path (milestone 36)
+// ---------------------------------------------------------------------------
+
+/// Floors a `query_log.timestamp` onto the projection grid in SQL. SQLite's `/`
+/// truncates toward zero, so a negative timestamp needs the bias — the same
+/// reason `bucketOf` is `@divFloor`.
+const bucket_expr = "(CASE WHEN timestamp >= 0 THEN timestamp / 1800 * 1800" ++
+ " ELSE (timestamp - 1799) / 1800 * 1800 END)";
+
+/// Each projection table beside a from-scratch recomputation of it. Both sides
+/// have unique keys, so `EXCEPT` in both directions is an exact equality test
+/// rather than a containment one.
+const recompute_checks = [_]struct { projection: []const u8, recompute: []const u8 }{
+ .{
+ .projection = "SELECT bucket, queries, blocked, cached, rt_sum, rt_count FROM bucket_totals",
+ .recompute = "SELECT " ++ bucket_expr ++ ", count(*), coalesce(sum(blocked <> 0), 0)," ++
+ " coalesce(sum(cache_hit = 1), 0), coalesce(sum(response_time_us), 0)," ++
+ " count(response_time_us) FROM query_log GROUP BY 1",
+ },
+ .{
+ .projection = "SELECT bucket, client_ip, queries FROM bucket_clients",
+ .recompute = "SELECT " ++ bucket_expr ++ ", client_ip, count(*) FROM query_log GROUP BY 1, 2",
+ },
+ .{
+ .projection = "SELECT bucket, qtype, count FROM bucket_types",
+ .recompute = "SELECT " ++ bucket_expr ++ ", coalesce(qtype, -1), count(*) FROM query_log GROUP BY 1, 2",
+ },
+ .{
+ .projection = "SELECT bucket, route_kind, source_present, source_text, count FROM bucket_routes",
+ .recompute = "SELECT " ++ bucket_expr ++ ", route_kind, source IS NOT NULL, coalesce(source, ''), count(*)" ++
+ " FROM (SELECT timestamp, route_kind, CASE route_kind WHEN 'upstream' THEN upstream" ++
+ " WHEN 'forward_zone' THEN forward_zone END AS source FROM query_log) GROUP BY 1, 2, 3, 4",
+ },
+};
+
+fn expectProjectionsMatchRecompute(database: *db.Db) !void {
+ for (recompute_checks) |check| {
+ var buf: [4096]u8 = undefined;
+ const sql = try std.fmt.bufPrint(
+ &buf,
+ "SELECT (SELECT count(*) FROM ({s} EXCEPT {s})) + (SELECT count(*) FROM ({s} EXCEPT {s}))",
+ .{ check.projection, check.recompute, check.recompute, check.projection },
+ );
+ try testing.expectEqual(@as(i64, 0), try database.queryInt(sql));
+ }
+}
+
+test "bucketOf floors onto the grid on both sides of the epoch" {
+ try testing.expectEqual(@as(i64, 0), bucketOf(0));
+ try testing.expectEqual(@as(i64, 0), bucketOf(1799));
+ try testing.expectEqual(@as(i64, grain), bucketOf(grain));
+ try testing.expectEqual(@as(i64, grain), bucketOf(grain + 1));
+ // Truncation toward zero would name bucket 0 for all three of these, which
+ // is the bucket *after* the one the row belongs to.
+ try testing.expectEqual(@as(i64, -grain), bucketOf(-1));
+ try testing.expectEqual(@as(i64, -grain), bucketOf(-grain));
+ try testing.expectEqual(@as(i64, -2 * grain), bucketOf(-grain - 1));
+}
+
+test "a batch maintains all four projections in the transaction that writes the rows" {
+ var database = try openLog();
+ defer database.close();
+ var writer = try BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+
+ var blocked_row = aggRow(0, "192.0.2.20", null, .blocked, null);
+ blocked_row.response_time_us = null;
+ try writer.writeBatch(&.{
+ aggRow(0, "192.0.2.10", 1, .upstream, "9.9.9.9"),
+ aggRow(5, "192.0.2.10", 1, .cache, null),
+ blocked_row,
+ // The next grid bucket, so the floor is what separates them.
+ aggRow(grain, "192.0.2.10", 28, .upstream, null),
+ });
+
+ // Two grid buckets: three rows floored into the first, one into the second.
+ try testing.expectEqual(@as(i64, 2), try database.queryInt("SELECT count(*) FROM bucket_totals"));
+ try testing.expectEqual(bucketOf(agg_since), try database.queryInt("SELECT min(bucket) FROM bucket_totals"));
+ try testing.expectEqual(@as(i64, 3), try database.queryInt("SELECT queries FROM bucket_totals ORDER BY bucket LIMIT 1"));
+ try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT blocked FROM bucket_totals ORDER BY bucket LIMIT 1"));
+ try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT cached FROM bucket_totals ORDER BY bucket LIMIT 1"));
+ // The blocked row carries no response time, so two of the three are timed.
+ try testing.expectEqual(@as(i64, 2), try database.queryInt("SELECT rt_count FROM bucket_totals ORDER BY bucket LIMIT 1"));
+ try expectProjectionsMatchRecompute(&database);
+
+ // A second batch accumulates onto the same keys rather than replacing them.
+ try writer.writeBatch(&.{aggRow(7, "192.0.2.10", 1, .upstream, "9.9.9.9")});
+ try testing.expectEqual(@as(i64, 4), try database.queryInt("SELECT queries FROM bucket_totals ORDER BY bucket LIMIT 1"));
+ try testing.expectEqual(
+ @as(i64, 3),
+ try database.queryInt("SELECT queries FROM bucket_clients ORDER BY bucket, client_ip LIMIT 1"),
+ );
+ try expectProjectionsMatchRecompute(&database);
+}
+
+test "a NULL qtype and a NULL route source are stored losslessly" {
+ var database = try openLog();
+ defer database.close();
+ var writer = try BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+
+ try writer.writeBatch(&.{
+ aggRow(0, "192.0.2.10", null, .upstream, null),
+ aggRow(1, "192.0.2.10", 1, .upstream, ""),
+ });
+
+ try testing.expectEqual(@as(i64, null_qtype), try database.queryInt("SELECT min(qtype) FROM bucket_types"));
+ // A row whose resolver the log did not record, and a row whose resolver is
+ // the empty string, are different keys — which is what `source_present`
+ // buys over "empty means absent".
+ try testing.expectEqual(@as(i64, 2), try database.queryInt("SELECT count(*) FROM bucket_routes"));
+ try testing.expectEqual(
+ @as(i64, 1),
+ try database.queryInt("SELECT count(*) FROM bucket_routes WHERE source_present = 0 AND source_text = ''"),
+ );
+ try testing.expectEqual(
+ @as(i64, 1),
+ try database.queryInt("SELECT count(*) FROM bucket_routes WHERE source_present = 1 AND source_text = ''"),
+ );
+ try expectProjectionsMatchRecompute(&database);
+}
+
+test "a failed projection update rolls the whole batch back and the writer survives it" {
+ var database = try openLog();
+ defer database.close();
+ var writer = try BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+
+ try writer.writeBatch(&.{aggRow(0, "192.0.2.10", 1, .upstream, "9.9.9.9")});
+ try database.exec(
+ \\CREATE TRIGGER refuse_projection BEFORE INSERT ON bucket_clients
+ \\WHEN new.client_ip = 'boom'
+ \\BEGIN SELECT RAISE(ABORT, 'refused'); END;
+ );
+
+ try testing.expectError(error.Constraint, writer.writeBatch(&.{
+ aggRow(1, "boom", 1, .upstream, "9.9.9.9"),
+ }));
+
+ // The raw row went in before the projection statement failed; neither
+ // survived, because they were one transaction.
+ try testing.expectEqual(@as(i64, 1), try countRows(&database));
+ try expectProjectionsMatchRecompute(&database);
+
+ try database.exec("DROP TRIGGER refuse_projection;");
+ try writer.writeBatch(&.{aggRow(2, "192.0.2.11", 1, .upstream, "9.9.9.9")});
+ try testing.expectEqual(@as(i64, 2), try countRows(&database));
+ try expectProjectionsMatchRecompute(&database);
+}
+
+test "a prune on the grid drops whole projection buckets and leaves the rest" {
+ var database = try openLog();
+ defer database.close();
+ var writer = try BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+
+ const base = bucketOf(agg_since);
+ try seedAt(&database, &.{ base, base + grain, base + 2 * grain });
+
+ _ = try pruneOlderThan(&database, base + 2 * grain);
+ try testing.expectEqual(@as(i64, 1), try countRows(&database));
+ try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM bucket_totals"));
+ try testing.expectEqual(base + 2 * grain, try database.queryInt("SELECT bucket FROM bucket_totals"));
+ try expectProjectionsMatchRecompute(&database);
+}
+
+test "a prune off the grid recomputes the straddling bucket instead of estimating it" {
+ var database = try openLog();
+ defer database.close();
+
+ const base = bucketOf(agg_since);
+ // Four rows in one bucket; the cutoff falls between the second and third.
+ try seedAt(&database, &.{ base, base + 100, base + 500, base + 900 });
+
+ _ = try pruneOlderThan(&database, base + 400);
+ try testing.expectEqual(@as(i64, 2), try countRows(&database));
+ try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM bucket_totals"));
+ try testing.expectEqual(@as(i64, 2), try database.queryInt("SELECT queries FROM bucket_totals"));
+ try expectProjectionsMatchRecompute(&database);
+
+ // Emptying the straddling bucket removes its row rather than writing zeros.
+ _ = try pruneOlderThan(&database, base + grain - 1);
+ try testing.expectEqual(@as(i64, 0), try countRows(&database));
+ try testing.expectEqual(@as(i64, 0), try database.queryInt("SELECT count(*) FROM bucket_totals"));
+ try expectProjectionsMatchRecompute(&database);
+}
+
+test "a failed projection prune rolls back the raw delete and the watermark with it" {
+ var database = try openLog();
+ defer database.close();
+
+ const base = bucketOf(agg_since);
+ try seedAt(&database, &.{ base, base + grain });
+ const watermark = try availableSince(&database);
+
+ try database.exec(
+ \\CREATE TRIGGER refuse_bucket_prune BEFORE DELETE ON bucket_clients
+ \\BEGIN SELECT RAISE(ABORT, 'refused'); END;
+ );
+
+ try testing.expectError(error.Constraint, pruneOlderThan(&database, base + grain));
+
+ try testing.expectEqual(@as(i64, 2), try countRows(&database));
+ try testing.expectEqual(watermark, try availableSince(&database));
+ try expectProjectionsMatchRecompute(&database);
+
+ try database.exec("DROP TRIGGER refuse_bucket_prune;");
+ _ = try pruneOlderThan(&database, base + grain);
+ try testing.expectEqual(@as(i64, 1), try countRows(&database));
+ try expectProjectionsMatchRecompute(&database);
+}
+
+test "a failed straddling-bucket replacement rolls back the raw delete and the watermark with it" {
+ var database = try openLog();
+ defer database.close();
+
+ const base = bucketOf(agg_since);
+ // One whole bucket to delete outright, and one the cutoff cuts in half.
+ try seedAt(&database, &.{ base, base + grain, base + grain + 100, base + grain + 900 });
+ const watermark = try availableSince(&database);
+ const straddling_totals = "SELECT queries FROM bucket_totals ORDER BY bucket DESC LIMIT 1";
+ const straddling_queries = try database.queryInt(straddling_totals);
+
+ // Only the recompute inserts into a projection table; the bulk prune above
+ // it deletes. So this trigger fires during the straddling-bucket
+ // replacement and nowhere else in the pass.
+ try database.exec(
+ \\CREATE TRIGGER refuse_recompute BEFORE INSERT ON bucket_totals
+ \\BEGIN SELECT RAISE(ABORT, 'refused'); END;
+ );
+
+ try testing.expectError(error.Constraint, pruneOlderThan(&database, base + grain + 500));
+
+ // The raw delete, the watermark advance, the whole-bucket projection
+ // deletes and the straddling bucket's own delete all went back together.
+ try testing.expectEqual(@as(i64, 4), try countRows(&database));
+ try testing.expectEqual(watermark, try availableSince(&database));
+ try testing.expectEqual(@as(i64, 2), try database.queryInt("SELECT count(*) FROM bucket_totals"));
+ try testing.expectEqual(straddling_queries, try database.queryInt(straddling_totals));
+ try expectProjectionsMatchRecompute(&database);
+
+ try database.exec("DROP TRIGGER refuse_recompute;");
+ _ = try pruneOlderThan(&database, base + grain + 500);
+ try testing.expectEqual(@as(i64, 1), try countRows(&database));
+ try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT queries FROM bucket_totals"));
+ try expectProjectionsMatchRecompute(&database);
+}
+
+/// One row per timestamp, all in the same batch, with the fixture defaults.
+fn seedAt(database: *db.Db, timestamps: []const i64) !void {
+ var writer = try BatchWriter.init(testing.allocator, database);
+ defer writer.deinit();
+
+ var rows: [16]Row = undefined;
+ for (rows[0..timestamps.len], timestamps) |*row, ts| {
+ row.* = plainRow(ts, "example.com");
+ }
+ try writer.writeBatch(rows[0..timestamps.len]);
+}
+
+test "an arbitrary interleaving of batches and prunes leaves every projection exact" {
+ var database = try openLog();
+ defer database.close();
+ var writer = try BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+
+ var prng: std.Random.DefaultPrng = .init(0x36a5_0000_36a5);
+ const random = prng.random();
+
+ const kinds = [_]provenance.RouteKind{ .blocked, .local, .forward_zone, .upstream, .cache, .rejected };
+ const sources = [_]?[]const u8{ "9.9.9.9", "https://a.example/dns-query", null, "" };
+ const base = bucketOf(agg_since);
+
+ var rows: [24]Row = undefined;
+ var addresses: [24][20]u8 = undefined;
+
+ var step: usize = 0;
+ while (step < 60) : (step += 1) {
+ if (step % 5 == 4) {
+ // Half the cutoffs land off the grid on purpose: the straddling
+ // bucket is the only case a delete alone cannot get right.
+ const offset = random.intRangeAtMost(i64, 0, 6 * grain);
+ const skew: i64 = if (random.boolean()) 0 else random.intRangeLessThan(i64, 1, grain);
+ _ = try pruneOlderThan(&database, base + offset + skew);
+ } else {
+ const count = random.intRangeAtMost(usize, 1, rows.len);
+ for (rows[0..count], addresses[0..count]) |*row, *address| {
+ const client = try std.fmt.bufPrint(address, "192.0.2.{d}", .{random.intRangeAtMost(u8, 1, 5)});
+ const kind = kinds[random.uintLessThan(usize, kinds.len)];
+ row.* = plainRow(base + random.intRangeAtMost(i64, 0, 8 * grain), "example.com");
+ row.client_ip = client;
+ row.qtype = if (random.boolean()) null else random.intRangeAtMost(u16, 1, 3);
+ row.blocked = kind == .blocked;
+ row.cache_hit = if (random.boolean()) null else kind == .cache;
+ row.response_time_us = if (random.boolean()) null else random.intRangeAtMost(i64, 0, 5000);
+ row.route_kind = kind;
+ const source = sources[random.uintLessThan(usize, sources.len)];
+ row.upstream = if (kind == .upstream) source else null;
+ row.forward_zone = if (kind == .forward_zone) source else null;
+ row.policy_action = if (kind == .blocked) .block else .allow;
+ row.policy_reason = if (kind == .blocked) .blocklist_domain else .no_match;
+ }
+ try writer.writeBatch(rows[0..count]);
+ }
+ try expectProjectionsMatchRecompute(&database);
+ }
+
+ // The walk has to have exercised both halves of it.
+ try testing.expect((try countRows(&database)) > 0);
+}
+
+// --- the Overview equivalence oracle --------------------------------------
+
+/// A test-only copy of the five SQL aggregates `overview` replaces, kept here
+/// so it survives their deletion. It is the independent statement of the
+/// contract: if `overview` and this disagree on any window, one of them is
+/// wrong, and this one is the one the goldens were written against.
+const oracle = struct {
+ const totals_sql =
+ \\SELECT count(*),
+ \\ coalesce(sum(blocked <> 0), 0),
+ \\ count(DISTINCT client_ip),
+ \\ coalesce(sum(response_time_us), 0),
+ \\ count(response_time_us)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2
+ ;
+
+ const buckets_sql =
+ \\SELECT (timestamp - ?1) / ?2,
+ \\ count(*),
+ \\ coalesce(sum(blocked <> 0), 0),
+ \\ coalesce(sum(cache_hit = 1), 0)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?3
+ \\ GROUP BY 1
+ ;
+
+ const types_sql =
+ \\SELECT qtype, count(*)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2
+ \\ GROUP BY qtype
+ \\ ORDER BY count(*) DESC, qtype IS NULL, qtype ASC
+ ;
+
+ const routes_sql =
+ \\SELECT route_kind,
+ \\ CASE route_kind
+ \\ WHEN 'upstream' THEN upstream
+ \\ WHEN 'forward_zone' THEN forward_zone
+ \\ END AS source,
+ \\ count(*)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2
+ \\ GROUP BY route_kind, source
+ \\ ORDER BY count(*) DESC, route_kind ASC, source IS NULL, source ASC
+ ;
+
+ const clients_rank_sql =
+ \\SELECT client_ip
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?2
+ \\ GROUP BY client_ip
+ \\ ORDER BY count(*) DESC, client_ip ASC
+ \\ LIMIT ?3
+ ;
+
+ const clients_buckets_sql =
+ \\SELECT client_ip, (timestamp - ?1) / ?2, count(*)
+ \\ FROM query_log
+ \\ WHERE timestamp >= ?1 AND timestamp < ?3
+ \\ GROUP BY 1, 2
+ ;
+
+ fn build(
+ database: *db.Db,
+ arena: Allocator,
+ since: i64,
+ bucket_seconds: u32,
+ bucket_count: u32,
+ ) !Overview {
+ const width: i64 = bucket_seconds;
+ const until = since + width * bucket_count;
+
+ var totals: StatsTotals = undefined;
+ {
+ var stmt = try database.prepare(totals_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+ try testing.expect(try stmt.step());
+ const timed = stmt.columnInt(4);
+ totals = .{
+ .queries = try countOf(stmt.columnInt(0)),
+ .blocked = try countOf(stmt.columnInt(1)),
+ .distinct_clients = try countOf(stmt.columnInt(2)),
+ .avg_response_time_us = if (timed == 0) null else @divTrunc(stmt.columnInt(3), timed),
+ };
+ }
+
+ const buckets = try arena.alloc(Bucket, bucket_count);
+ for (buckets, 0..) |*bucket, i| bucket.* = .{
+ .ts = since + width * @as(i64, @intCast(i)),
+ .queries = 0,
+ .blocked = 0,
+ .cached = 0,
+ };
+ {
+ var stmt = try database.prepare(buckets_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, width);
+ try stmt.bindInt(3, until);
+ while (try stmt.step()) {
+ const index: usize = @intCast(stmt.columnInt(0));
+ buckets[index].queries = try countOf(stmt.columnInt(1));
+ buckets[index].blocked = try countOf(stmt.columnInt(2));
+ buckets[index].cached = try countOf(stmt.columnInt(3));
+ }
+ }
+
+ var types: std.ArrayList(TypeCount) = .empty;
+ {
+ var stmt = try database.prepare(types_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+ while (try stmt.step()) {
+ try types.append(arena, .{
+ .qtype = if (stmt.isNull(0)) null else try columnU16(&stmt, 0),
+ .count = try countOf(stmt.columnInt(1)),
+ });
+ }
+ }
+
+ var routes: std.ArrayList(RouteCount) = .empty;
+ {
+ var stmt = try database.prepare(routes_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+ while (try stmt.step()) {
+ try routes.append(arena, .{
+ .route = try provenance.parse(provenance.RouteKind, stmt.columnText(0)),
+ .source = try stmt.columnTextAllocOrNull(arena, 1),
+ .count = try countOf(stmt.columnInt(2)),
+ });
+ }
+ }
+
+ var names: std.ArrayList([]const u8) = .empty;
+ var series: std.ArrayList([]u64) = .empty;
+ {
+ var stmt = try database.prepare(clients_rank_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, until);
+ try stmt.bindInt(3, max_client_series);
+ while (try stmt.step()) {
+ try names.append(arena, try stmt.columnTextAlloc(arena, 0));
+ try series.append(arena, try zeroedBuckets(arena, bucket_count));
+ }
+ }
+ const other = try zeroedBuckets(arena, bucket_count);
+ {
+ var stmt = try database.prepare(clients_buckets_sql);
+ defer stmt.deinit();
+ try stmt.bindInt(1, since);
+ try stmt.bindInt(2, width);
+ try stmt.bindInt(3, until);
+ while (try stmt.step()) {
+ const index: usize = @intCast(stmt.columnInt(1));
+ const count = try countOf(stmt.columnInt(2));
+ const client = stmt.columnText(0);
+ const target = for (names.items, series.items) |name, target_buckets| {
+ if (std.mem.eql(u8, name, client)) break target_buckets;
+ } else other;
+ target[index] += count;
+ }
+ }
+ const clients = try arena.alloc(ClientSeries, names.items.len);
+ for (clients, names.items, series.items) |*entry, name, entry_buckets| {
+ entry.* = .{ .client = name, .buckets = entry_buckets };
+ }
+
+ return .{
+ .totals = totals,
+ .buckets = buckets,
+ .clients = .{ .clients = clients, .other = other },
+ .types = types.items,
+ .routes = routes.items,
+ };
+ }
+};
+
+fn expectOverviewEqual(expected: Overview, actual: Overview) !void {
+ try testing.expectEqual(expected.totals.queries, actual.totals.queries);
+ try testing.expectEqual(expected.totals.blocked, actual.totals.blocked);
+ try testing.expectEqual(expected.totals.distinct_clients, actual.totals.distinct_clients);
+ try testing.expectEqual(expected.totals.avg_response_time_us, actual.totals.avg_response_time_us);
+ try testing.expectEqualSlices(Bucket, expected.buckets, actual.buckets);
+
+ try testing.expectEqual(expected.types.len, actual.types.len);
+ for (expected.types, actual.types) |want, got| {
+ try testing.expectEqual(want.qtype, got.qtype);
+ try testing.expectEqual(want.count, got.count);
+ }
+
+ try testing.expectEqual(expected.routes.len, actual.routes.len);
+ for (expected.routes, actual.routes) |want, got| {
+ try testing.expectEqual(want.route, got.route);
+ try testing.expectEqual(want.count, got.count);
+ if (want.source) |source| {
+ try testing.expectEqualStrings(source, got.source orelse return error.TestExpectedEqual);
+ } else {
+ try testing.expectEqual(@as(?[]const u8, null), got.source);
+ }
+ }
+
+ try testing.expectEqual(expected.clients.clients.len, actual.clients.clients.len);
+ for (expected.clients.clients, actual.clients.clients) |want, got| {
+ try testing.expectEqualStrings(want.client, got.client);
+ try testing.expectEqualSlices(u64, want.buckets, got.buckets);
+ }
+ try testing.expectEqualSlices(u64, expected.clients.other, actual.clients.other);
+}
+
+/// Every window the equivalence test walks. The first is the raw path (60 s
+/// buckets, the 1h period); the rest are the projection path at each serving
+/// width the period grammar offers.
+const overview_windows = [_]struct { bucket_seconds: u32, bucket_count: u32 }{
+ .{ .bucket_seconds = 60, .bucket_count = 60 },
+ .{ .bucket_seconds = 1800, .bucket_count = 48 },
+ .{ .bucket_seconds = 3600, .bucket_count = 168 },
+ .{ .bucket_seconds = 21600, .bucket_count = 120 },
+};
+
+fn expectOverviewMatchesOracle(database: *db.Db, since: i64) !void {
+ var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
+ defer arena_state.deinit();
+ const arena = arena_state.allocator();
+
+ for (overview_windows) |window| {
+ const want = try oracle.build(database, arena, since, window.bucket_seconds, window.bucket_count);
+ const got = try overview(database, arena, since, window.bucket_seconds, window.bucket_count);
+ expectOverviewEqual(want, got) catch |err| {
+ std.debug.print("overview mismatch at bucket_seconds={d}\n", .{window.bucket_seconds});
+ return err;
+ };
+ }
+}
+
+test "overview equals the oracle on an empty database" {
+ var database = try openLog();
+ defer database.close();
+ try expectOverviewMatchesOracle(&database, bucketOf(agg_since));
+}
+
+test "overview equals the oracle over a window with no rows in it" {
+ var database = try openLog();
+ defer database.close();
+ try seedAt(&database, &.{ agg_since, agg_since + 10 });
+
+ // Two full 30-day spans later: every panel is empty and the axis is still
+ // whole.
+ try expectOverviewMatchesOracle(&database, bucketOf(agg_since) + 240 * grain);
+}
+
+test "overview equals the oracle across nulls, ties and a bucket still in progress" {
+ var database = try openLog();
+ defer database.close();
+ var writer = try BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+
+ const since = bucketOf(agg_since);
+ const kinds = [_]provenance.RouteKind{ .blocked, .local, .forward_zone, .upstream, .cache, .rejected };
+
+ // Ten clients so the top-eight cut has a residual, and the counts are
+ // deliberately equal in pairs so the `client_ip` tie-break decides.
+ var rows: [120]Row = undefined;
+ var addresses: [120][20]u8 = undefined;
+ for (&rows, &addresses, 0..) |*row, *address, i| {
+ const client = try std.fmt.bufPrint(address, "192.0.2.{d}", .{100 + (i / 2) % 10});
+ const kind = kinds[i % kinds.len];
+ row.* = plainRow(since + @as(i64, @intCast(i)) * 137, "example.com");
+ row.client_ip = client;
+ // Every seventh row carries no query type, and the types collide in
+ // count so the null-last tie-break is exercised.
+ row.qtype = if (i % 7 == 0) null else @intCast(1 + i % 3);
+ row.blocked = kind == .blocked;
+ row.cache_hit = if (i % 5 == 0) null else kind == .cache;
+ row.response_time_us = if (i % 4 == 0) null else @intCast(100 + i);
+ row.route_kind = kind;
+ // Every eleventh upstream row records no resolver: the NULL-source
+ // group the routes breakdown must keep rather than drop.
+ const source: ?[]const u8 = if (i % 11 == 0) null else if (i % 2 == 0) "9.9.9.9" else "https://a.example/dns-query";
+ row.upstream = if (kind == .upstream) source else null;
+ row.forward_zone = if (kind == .forward_zone) source else null;
+ row.policy_action = if (kind == .blocked) .block else .allow;
+ row.policy_reason = if (kind == .blocked) .blocklist_domain else .no_match;
+ }
+ try writer.writeBatch(&rows);
+
+ try expectOverviewMatchesOracle(&database, since);
+
+ // A window whose last bucket is only partly filled: `since` moved so the
+ // newest rows land inside the final bucket rather than closing it.
+ try expectOverviewMatchesOracle(&database, since - 40 * grain);
+
+ // And after a prune off the grid, so the projections the read path uses are
+ // the recomputed ones.
+ _ = try pruneOlderThan(&database, since + 900);
+ try expectProjectionsMatchRecompute(&database);
+ try expectOverviewMatchesOracle(&database, since);
+}
+
+test "overview equals the oracle after an interleaving of batches and prunes" {
+ var database = try openLog();
+ defer database.close();
+ var writer = try BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+
+ const since = bucketOf(agg_since);
+ var prng: std.Random.DefaultPrng = .init(0x0e17_3a99);
+ const random = prng.random();
+
+ var rows: [16]Row = undefined;
+ var addresses: [16][20]u8 = undefined;
+ var step: usize = 0;
+ while (step < 12) : (step += 1) {
+ const count = random.intRangeAtMost(usize, 1, rows.len);
+ for (rows[0..count], addresses[0..count]) |*row, *address| {
+ const client = try std.fmt.bufPrint(address, "10.0.0.{d}", .{random.intRangeAtMost(u8, 1, 12)});
+ row.* = plainRow(since + random.intRangeAtMost(i64, 0, 20 * grain), "example.com");
+ row.client_ip = client;
+ row.qtype = if (random.boolean()) null else random.intRangeAtMost(u16, 1, 4);
+ row.response_time_us = if (random.boolean()) null else random.intRangeAtMost(i64, 0, 9000);
+ row.cache_hit = if (random.boolean()) null else random.boolean();
+ }
+ try writer.writeBatch(rows[0..count]);
+ if (step % 4 == 3) {
+ _ = try pruneOlderThan(&database, since + random.intRangeAtMost(i64, 0, 10 * grain));
+ }
+ try expectOverviewMatchesOracle(&database, since);
+ }
+}
+
+test "overview refuses a window the projections cannot express" {
+ var database = try openLog();
+ defer database.close();
+ var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
+ defer arena_state.deinit();
+ const arena = arena_state.allocator();
+
+ const since = bucketOf(agg_since);
+
+ try testing.expectError(error.Misuse, overview(&database, arena, since, 0, 48));
+ try testing.expectError(error.Misuse, overview(&database, arena, since, 1800, 0));
+ try testing.expectError(error.Misuse, overview(&database, arena, std.math.maxInt(i64) - 1, 3600, 48));
+
+ // On the projection path the grid is part of the contract: an unaligned
+ // start or a width that is not a whole number of grains cannot be answered
+ // from 30-minute rows, and guessing would be worse than refusing.
+ try testing.expectError(error.Misuse, overview(&database, arena, since + 1, 1800, 48));
+ try testing.expectError(error.Misuse, overview(&database, arena, since, 2700, 48));
+
+ // Below the grain none of that applies: the raw rows carry every second.
+ _ = try overview(&database, arena, since + 1, 60, 60);
+}
diff --git a/src/storage/retention.zig b/src/storage/retention.zig
index 1ba507f..b1c2893 100644
--- a/src/storage/retention.zig
+++ b/src/storage/retention.zig
@@ -251,7 +251,7 @@ fn openLog() !db.Db {
}
fn writeRows(database: *db.Db, timestamps: []const i64) !void {
- var writer = try queries_repo.BatchWriter.init(database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, database);
defer writer.deinit();
var rows: [8]queries_repo.Row = undefined;
for (timestamps, rows[0..timestamps.len]) |timestamp, *row| {
diff --git a/src/tests.zig b/src/tests.zig
index fe34d50..cfac243 100644
--- a/src/tests.zig
+++ b/src/tests.zig
@@ -106,7 +106,7 @@ comptime {
_ = @import("web/server_integration_test.zig");
_ = @import("server/local_tables.zig");
_ = @import("web/metrics.zig");
- _ = @import("web/handlers/stats.zig");
+ _ = @import("web/handlers/overview.zig");
_ = @import("web/handlers/queries.zig");
_ = @import("web/handlers/diagnostics.zig");
_ = @import("web/handlers/lookup.zig");
diff --git a/src/web/coverage.zig b/src/web/coverage.zig
index 381ec15..3f2300f 100644
--- a/src/web/coverage.zig
+++ b/src/web/coverage.zig
@@ -6,10 +6,9 @@
//! complete for. Without that fact on the wire a chart draws a pruned week as a
//! week of silence, which is the one reading that is certainly wrong.
//!
-//! Three endpoints carry it — `/api/queries`, `/api/stats` and
-//! `/api/stats/timeseries` — and they judge it against their own effective
-//! lower bound: the client's `since` for the query log, the period's aligned
-//! window start for the two stats endpoints.
+//! Two endpoints carry it — `/api/queries` and `/api/overview` — and they judge
+//! it against their own effective lower bound: the client's `since` for the
+//! query log, the period's aligned window start for the overview.
const std = @import("std");
diff --git a/src/web/handlers/overview.zig b/src/web/handlers/overview.zig
new file mode 100644
index 0000000..736cabd
--- /dev/null
+++ b/src/web/handlers/overview.zig
@@ -0,0 +1,647 @@
+//! `GET /api/overview`: everything the Overview page draws, for one period, in
+//! one response.
+//!
+//! It replaces the five per-panel endpoints milestone 30 shipped. Those cost
+//! five scans of every raw row in the window and, being five requests, could
+//! only promise a shared *window* — queries logged between two of them moved
+//! one panel and not the other. One request over one read transaction promises
+//! a shared *snapshot*: the totals, the four breakdowns and the coverage
+//! watermark beside them all describe one database state, so the breakdowns sum
+//! to the totals for a reason and not by luck.
+//!
+//! Buckets are aligned to the UTC grid, not to the moment of the request. Every
+//! width divides a day, so flooring the current time to a multiple of the width
+//! puts each bucket on the same boundary a human reads off a clock, and two
+//! requests a second apart return the same bucket starts. The last bucket is
+//! the one in progress; it fills as the period runs.
+//!
+//! The aggregate runs on the web task's own query-log connection (m7 ruling
+//! 21), which every connection task shares. SQLite's serialized mode makes one
+//! call safe; it does not make a transaction safe, so `WebState.querylog_lock`
+//! covers the whole read and a second BEGIN can never land inside the first.
+//! The transaction is deferred, not `db.Tx`'s BEGIN IMMEDIATE, which would
+//! stall the logger and retention behind an HTTP response.
+//!
+//! The lock is released before the response is written: the body is already
+//! built in the request arena, and holding a database lock across a socket
+//! write would let one slow client serialize every other reader.
+
+const std = @import("std");
+
+const coverage = @import("../coverage.zig");
+const db = @import("../../storage/db.zig");
+const http_util = @import("../http_util.zig");
+const queries_repo = @import("../../storage/repositories/queries_repo.zig");
+const server = @import("../server.zig");
+
+const log = std.log.scoped(.web_overview);
+
+/// The four periods ruling 13 defines. The tag names are the wire spellings.
+pub const Period = enum {
+ @"1h",
+ @"24h",
+ @"7d",
+ @"30d",
+
+ pub fn parse(text: []const u8) ?Period {
+ return std.meta.stringToEnum(Period, text);
+ }
+
+ /// Ruling 13: 1h→60×1m, 24h→48×30m, 7d→168×1h, 30d→120×6h.
+ pub fn bucketSeconds(self: Period) u32 {
+ return switch (self) {
+ .@"1h" => 60,
+ .@"24h" => 30 * 60,
+ .@"7d" => 60 * 60,
+ .@"30d" => 6 * 60 * 60,
+ };
+ }
+
+ pub fn bucketCount(self: Period) u32 {
+ return switch (self) {
+ .@"1h" => 60,
+ .@"24h" => 48,
+ .@"7d" => 168,
+ .@"30d" => 120,
+ };
+ }
+
+ pub fn label(self: Period) []const u8 {
+ return @tagName(self);
+ }
+};
+
+pub const default_period: Period = .@"24h";
+
+/// The widest period's bucket count. Nothing here allocates by it any more —
+/// the repository returns arena slices — but it is the bound the response size
+/// argument rests on, and the assertion below is what keeps it true.
+pub const max_buckets = 168;
+
+comptime {
+ std.debug.assert(std.enums.values(Period).len == server.OverviewCache.slot_count);
+ for (std.enums.values(Period)) |period| {
+ std.debug.assert(period.bucketCount() <= max_buckets);
+ // The UTC alignment argument holds only while every width divides a day.
+ std.debug.assert(86_400 % period.bucketSeconds() == 0);
+ }
+}
+
+pub const Window = struct {
+ /// Inclusive, on the bucket grid.
+ since: i64,
+ /// Exclusive: the end of the bucket that `now` falls in.
+ until: i64,
+ bucket_seconds: u32,
+ bucket_count: u32,
+};
+
+pub fn window(period: Period, now_unix: i64) Window {
+ const width: i64 = period.bucketSeconds();
+ const count: i64 = period.bucketCount();
+ const until = @divFloor(now_unix, width) * width + width;
+ return .{
+ .since = until - width * count,
+ .until = until,
+ .bucket_seconds = period.bucketSeconds(),
+ .bucket_count = period.bucketCount(),
+ };
+}
+
+pub const Totals = struct {
+ queries: u64,
+ blocked: u64,
+ /// Distinct client addresses in the window.
+ clients: u64,
+ avg_response_time_us: ?i64,
+};
+
+pub const Body = struct {
+ period: []const u8,
+ since: i64,
+ until: i64,
+ bucket_seconds: u32,
+ totals: Totals,
+ buckets: []const queries_repo.Bucket,
+ clients: []const queries_repo.ClientSeries,
+ other: []const u64,
+ types: []const queries_repo.TypeCount,
+ routes: []const queries_repo.RouteCount,
+ /// Judged against `since`, which is the window this body reports on — so a
+ /// dashboard can say "history starts here" instead of charting a pruned
+ /// stretch as a quiet one.
+ coverage: coverage.Coverage,
+};
+
+pub fn handle(
+ state: *server.WebState,
+ io: std.Io,
+ request: *http_util.Request,
+) http_util.HandlerError!void {
+ const period = periodParam(request.query) catch return badPeriod(request);
+ const database = state.querylog_db orelse return unavailable(request);
+ const span = window(period, std.Io.Clock.real.now(io).toSeconds());
+
+ const body = cachedBody(state, io, database, request.arena, period, span) catch |err| {
+ return internal(request, err);
+ };
+ return http_util.respondBytes(request, .ok, body, http_util.content_type_json, &.{});
+}
+
+/// The whole cache decision, start to finish, under one hold of
+/// `querylog_lock`. Returns bytes owned by `arena`, so the caller writes the
+/// socket with the lock already released.
+///
+/// `data_version` is sampled inside the lock and the rebuild is published under
+/// that same sample: a commit landing on another connection while this task
+/// builds moves the pragma, so the entry it installs is keyed to a version the
+/// next request will not ask for and that request rebuilds. Stale bytes under a
+/// current key are therefore not reachable. A failed build or a failed commit
+/// publishes nothing and leaves whatever the slot already held.
+fn cachedBody(
+ state: *server.WebState,
+ io: std.Io,
+ database: *db.Db,
+ arena: std.mem.Allocator,
+ period: Period,
+ span: Window,
+) db.Error![]const u8 {
+ state.querylog_lock.lockUncancelable(io);
+ defer state.querylog_lock.unlock(io);
+
+ const index = @intFromEnum(period);
+ const data_version = try database.queryInt("PRAGMA data_version");
+
+ if (state.overview_cache.get(index, span.until, data_version)) |cached| {
+ // The copy is what makes a later rebuild's free-and-replace safe: the
+ // response is written after the lock is gone, and by then these bytes
+ // may belong to nobody.
+ return arena.dupe(u8, cached);
+ }
+
+ const body = try buildBody(state, io, database, arena, period, span);
+ const owned = try state.gpa.dupe(u8, body);
+ state.overview_cache.put(state.gpa, index, span.until, data_version, owned);
+ return body;
+}
+
+/// One read transaction, one snapshot, one serialized body in `arena`. The
+/// caller holds `querylog_lock` and keeps holding it.
+fn buildBody(
+ state: *server.WebState,
+ io: std.Io,
+ database: *db.Db,
+ arena: std.mem.Allocator,
+ period: Period,
+ span: Window,
+) db.Error![]const u8 {
+ var scope = try server.QuerylogRead.openLocked(state, io, database);
+ errdefer scope.abort();
+ const data = try queries_repo.overview(
+ database,
+ arena,
+ span.since,
+ span.bucket_seconds,
+ span.bucket_count,
+ );
+ const window_coverage = try coverage.read(database, span.since);
+ try scope.commit();
+
+ var allocating: std.Io.Writer.Allocating = .init(arena);
+ errdefer allocating.deinit();
+ std.json.Stringify.value(Body{
+ .period = period.label(),
+ .since = span.since,
+ .until = span.until,
+ .bucket_seconds = span.bucket_seconds,
+ .totals = .{
+ .queries = data.totals.queries,
+ .blocked = data.totals.blocked,
+ .clients = data.totals.distinct_clients,
+ .avg_response_time_us = data.totals.avg_response_time_us,
+ },
+ .buckets = data.buckets,
+ .clients = data.clients.clients,
+ .other = data.clients.other,
+ .types = data.types,
+ .routes = data.routes,
+ .coverage = window_coverage,
+ }, .{}, &allocating.writer) catch return error.OutOfMemory;
+ return allocating.written();
+}
+
+pub const PeriodError = error{BadPeriod};
+
+/// An absent `period` is the default; anything else it cannot read is a 400,
+/// never a silent fallback — a typo must not return a window nobody asked for.
+fn periodParam(query: []const u8) PeriodError!Period {
+ var buf: [8]u8 = undefined;
+ const found = http_util.queryValue(query, "period", &buf) catch return error.BadPeriod;
+ const text = found orelse return default_period;
+ return Period.parse(text) orelse error.BadPeriod;
+}
+
+fn badPeriod(request: *http_util.Request) http_util.HandlerError!void {
+ return http_util.respondError(request, .bad_request, "period must be one of 1h, 24h, 7d, 30d");
+}
+
+fn unavailable(request: *http_util.Request) http_util.HandlerError!void {
+ return http_util.respondError(request, .service_unavailable, "query log unavailable");
+}
+
+/// The one thing this file logs. A failed aggregate is a fault in the box, not
+/// a property of the request, and the client is told nothing beyond "internal
+/// error" (ruling 8, PLAN §19).
+fn internal(request: *http_util.Request, err: db.Error) http_util.HandlerError!void {
+ log.warn("overview failed: {s}", .{@errorName(err)});
+ return http_util.respondError(request, .internal_server_error, "internal error");
+}
+
+// ---------------------------------------------------------------------------
+// tests
+// ---------------------------------------------------------------------------
+
+const querylog_schema = @import("../../storage/querylog_schema.zig");
+const testing = std.testing;
+
+test "the period grammar accepts exactly the four spellings" {
+ try testing.expectEqual(Period.@"1h", Period.parse("1h").?);
+ try testing.expectEqual(Period.@"24h", Period.parse("24h").?);
+ try testing.expectEqual(Period.@"7d", Period.parse("7d").?);
+ try testing.expectEqual(Period.@"30d", Period.parse("30d").?);
+ try testing.expectEqual(@as(?Period, null), Period.parse("12h"));
+ try testing.expectEqual(@as(?Period, null), Period.parse("1H"));
+ try testing.expectEqual(@as(?Period, null), Period.parse(""));
+}
+
+test "an absent period defaults and a bad one is rejected" {
+ try testing.expectEqual(default_period, try periodParam(""));
+ try testing.expectEqual(default_period, try periodParam("limit=5"));
+ try testing.expectEqual(Period.@"7d", try periodParam("period=7d"));
+ try testing.expectError(error.BadPeriod, periodParam("period=12h"));
+ try testing.expectError(error.BadPeriod, periodParam("period=%2"));
+ // Longer than any spelling: rejected rather than truncated to "1h".
+ try testing.expectError(error.BadPeriod, periodParam("period=1hhhhhhhhhh"));
+}
+
+test "each period spans its own bucket width times its count" {
+ for (std.enums.values(Period)) |period| {
+ const span = window(period, 1_700_000_000);
+ const width: i64 = period.bucketSeconds();
+ try testing.expectEqual(width * @as(i64, period.bucketCount()), span.until - span.since);
+ }
+}
+
+test "the window sits on the UTC grid and ends with the bucket in progress" {
+ // 2023-11-14T22:13:20Z, which is not on any bucket boundary.
+ const now: i64 = 1_700_000_000;
+ const span = window(.@"24h", now);
+
+ try testing.expectEqual(@as(i64, 0), @rem(span.since, 1800));
+ try testing.expectEqual(@as(i64, 0), @rem(span.until, 1800));
+ try testing.expect(span.until > now);
+ try testing.expect(span.until - now <= 1800);
+ try testing.expectEqual(@as(u32, 48), span.bucket_count);
+}
+
+test "two requests inside one bucket see the same window" {
+ // A bucket boundary, so the offsets below stay inside one minute.
+ const boundary: i64 = 1_700_000_000 - @rem(1_700_000_000, 60);
+ const first = window(.@"1h", boundary);
+ const second = window(.@"1h", boundary + 59);
+ try testing.expectEqual(first.since, second.since);
+ try testing.expectEqual(first.until, second.until);
+
+ const next = window(.@"1h", boundary + 60);
+ try testing.expectEqual(first.until + 60, next.until);
+}
+
+test "a timestamp exactly on a boundary starts a new bucket" {
+ const span = window(.@"7d", 1_700_000_000 - 1_700_000_000 % 3600);
+ try testing.expectEqual(@as(i64, 0), @rem(span.since, 3600));
+ try testing.expectEqual(@as(u32, 168), span.bucket_count);
+}
+
+test "every period's window is a window the repository will serve" {
+ // The projection path needs the window start on the 30-minute grid and the
+ // width a whole number of grains; `window` is the only producer of either.
+ for (std.enums.values(Period)) |period| {
+ const span = window(period, 1_700_000_123);
+ if (span.bucket_seconds < 1800) continue;
+ try testing.expectEqual(@as(i64, 0), @mod(span.since, 1800));
+ try testing.expectEqual(@as(u32, 0), span.bucket_seconds % 1800);
+ }
+}
+
+fn openLog() !db.Db {
+ var database = try db.Db.open(":memory:", .{ .mode = .memory });
+ errdefer database.close();
+ try db.applyPragmas(&database, .{});
+ try database.exec(querylog_schema.ddl);
+ return database;
+}
+
+fn writeRow(writer: *queries_repo.BatchWriter, timestamp: i64, blocked: bool, cached: ?bool) !void {
+ const rows = [_]queries_repo.Row{.{
+ .timestamp = timestamp,
+ .domain = "example.com",
+ .client_ip = "192.0.2.10",
+ .qtype = 1,
+ .qclass = 1,
+ .rcode = 0,
+ .blocked = blocked,
+ .response_time_us = 1000,
+ .cache_hit = cached,
+ .upstream = null,
+ .group_id = 1,
+ .group_name = "default",
+ .policy_action = if (blocked) .block else .allow,
+ .policy_reason = if (blocked) .blocklist_domain else .no_match,
+ .matched = null,
+ .source_id = null,
+ .source_name = null,
+ .cname_target = null,
+ .safe_search_target = null,
+ .route_kind = if (blocked) .blocked else .upstream,
+ .forward_zone = null,
+ }};
+ try writer.writeBatch(&rows);
+}
+
+test "one overview answers totals and buckets that agree over the same window" {
+ var database = try openLog();
+ defer database.close();
+
+ var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
+ defer arena_state.deinit();
+
+ const span = window(.@"1h", 1_700_000_000);
+
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, &database);
+ defer writer.deinit();
+ // One row in the first bucket, two in the last, one just outside.
+ try writeRow(&writer, span.since, false, false);
+ try writeRow(&writer, span.until - 1, true, false);
+ try writeRow(&writer, span.until - 2, false, true);
+ try writeRow(&writer, span.since - 1, false, false);
+
+ const data = try queries_repo.overview(
+ &database,
+ arena_state.allocator(),
+ span.since,
+ span.bucket_seconds,
+ span.bucket_count,
+ );
+
+ try testing.expectEqual(@as(u64, 3), data.totals.queries);
+ try testing.expectEqual(@as(u64, 1), data.totals.blocked);
+ try testing.expectEqual(@as(u64, 1), data.totals.distinct_clients);
+ try testing.expectEqual(@as(?i64, 1000), data.totals.avg_response_time_us);
+
+ try testing.expectEqual(@as(usize, 60), data.buckets.len);
+ var summed: u64 = 0;
+ var blocked: u64 = 0;
+ for (data.buckets) |bucket| {
+ summed += bucket.queries;
+ blocked += bucket.blocked;
+ }
+ try testing.expectEqual(data.totals.queries, summed);
+ try testing.expectEqual(data.totals.blocked, blocked);
+
+ try testing.expectEqual(span.since, data.buckets[0].ts);
+ try testing.expectEqual(@as(u64, 1), data.buckets[0].queries);
+ try testing.expectEqual(@as(u64, 2), data.buckets[59].queries);
+ try testing.expectEqual(span.until - span.bucket_seconds, data.buckets[59].ts);
+}
+
+test "an empty window reports zeros with a null mean" {
+ var database = try openLog();
+ defer database.close();
+
+ var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
+ defer arena_state.deinit();
+
+ const span = window(.@"30d", 1_700_000_000);
+ const data = try queries_repo.overview(
+ &database,
+ arena_state.allocator(),
+ span.since,
+ span.bucket_seconds,
+ span.bucket_count,
+ );
+
+ try testing.expectEqual(@as(u64, 0), data.totals.queries);
+ try testing.expectEqual(@as(?i64, null), data.totals.avg_response_time_us);
+ try testing.expectEqual(@as(usize, 120), data.buckets.len);
+ for (data.buckets) |bucket| try testing.expectEqual(@as(u64, 0), bucket.queries);
+ // `other` is bucket-count sized even here: a chart must never have to
+ // invent the residual series.
+ try testing.expectEqual(@as(usize, 120), data.clients.other.len);
+}
+
+test "the cache serves one period's bytes and rebuilds when the key moves" {
+ const gpa = testing.allocator;
+ var cache: server.OverviewCache = .{};
+ defer cache.deinit(gpa);
+
+ try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 7));
+
+ cache.put(gpa, 0, 100, 7, try gpa.dupe(u8, "first"));
+ try testing.expectEqualStrings("first", cache.get(0, 100, 7).?);
+ // A different period, a rolled window and a bumped data version are three
+ // different keys, and none of them hits.
+ try testing.expectEqual(@as(?[]const u8, null), cache.get(1, 100, 7));
+ try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 101, 7));
+ try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 8));
+
+ // A rebuild replaces the entry and frees the old body; the leak checker in
+ // `testing.allocator` is the assertion.
+ cache.put(gpa, 0, 100, 8, try gpa.dupe(u8, "second"));
+ try testing.expectEqualStrings("second", cache.get(0, 100, 8).?);
+}
+
+/// Two connections onto one file, which is the only arrangement in which
+/// `PRAGMA data_version` moves at all: it reports commits by *other*
+/// connections, so an in-memory database — where there is no other connection —
+/// could never witness the invalidation these tests are about.
+const CacheFixture = struct {
+ threaded: std.Io.Threaded,
+ tmp: std.testing.TmpDir,
+ /// The web task's connection, the one the cache is keyed on.
+ reader: db.Db,
+ /// Stands in for the logger and for retention.
+ writer: db.Db,
+ state: server.WebState,
+ arena_state: std.heap.ArenaAllocator,
+
+ fn init(self: *CacheFixture, gpa: std.mem.Allocator) !void {
+ self.threaded = .init(gpa, .{});
+ errdefer self.threaded.deinit();
+ self.tmp = std.testing.tmpDir(.{});
+ errdefer self.tmp.cleanup();
+
+ var path_buf: [256]u8 = undefined;
+ const path = try std.fmt.bufPrintZ(
+ &path_buf,
+ ".zig-cache/tmp/{s}/querylog.db",
+ .{self.tmp.sub_path},
+ );
+
+ self.writer = try db.Db.open(path, .{ .mode = .read_write_create });
+ errdefer self.writer.close();
+ try db.applyPragmas(&self.writer, .{});
+ try self.writer.exec(querylog_schema.ddl);
+ // The DDL stamps `created_at` from the wall clock, and the watermark
+ // with it. These tests work over a fixed 2023 window, so a 2026
+ // watermark would report every one of them as uncovered and the prune
+ // below would not move it.
+ try self.writer.exec(
+ "UPDATE querylog_meta SET created_at = 1600000000, available_since = 1600000000 WHERE id = 1",
+ );
+
+ self.reader = try db.Db.open(path, .{ .mode = .read_write_existing });
+ errdefer self.reader.close();
+ try db.applyPragmas(&self.reader, .{});
+
+ self.state = .{ .gpa = gpa, .querylog_db = &self.reader };
+ self.arena_state = .init(gpa);
+ }
+
+ fn deinit(self: *CacheFixture) void {
+ self.arena_state.deinit();
+ self.state.overview_cache.deinit(self.state.gpa);
+ self.reader.close();
+ self.writer.close();
+ self.tmp.cleanup();
+ self.threaded.deinit();
+ }
+
+ fn io(self: *CacheFixture) std.Io {
+ return self.threaded.io();
+ }
+
+ fn body(self: *CacheFixture, period: Period, span: Window) db.Error![]const u8 {
+ return cachedBody(
+ &self.state,
+ self.io(),
+ &self.reader,
+ self.arena_state.allocator(),
+ period,
+ span,
+ );
+ }
+
+ /// One row through the writer connection, which commits and so moves the
+ /// reader's `PRAGMA data_version`.
+ fn log(self: *CacheFixture, timestamp: i64) !void {
+ var batch = try queries_repo.BatchWriter.init(self.state.gpa, &self.writer);
+ defer batch.deinit();
+ try writeRow(&batch, timestamp, false, false);
+ }
+};
+
+test "a cache hit answers without opening a read transaction" {
+ var fx: CacheFixture = undefined;
+ try fx.init(testing.allocator);
+ defer fx.deinit();
+
+ const span = window(.@"1h", 1_700_000_000);
+ try fx.log(span.since + 10);
+
+ const first = try fx.body(.@"1h", span);
+ try testing.expect(first.len > 0);
+
+ // A hit never reaches the database, so a fault armed on the next commit is
+ // never spent — and the bytes are the stored ones, not a rebuild's.
+ db.read_tx_faults.failNextCommit();
+ const second = try fx.body(.@"1h", span);
+ try testing.expectEqualStrings(first, second);
+ try testing.expect(first.ptr != second.ptr);
+
+ // Spend the armed fault so it cannot leak into a later test. A miss does
+ // reach the database, so this one trips.
+ db.read_tx_faults.beginCapture();
+ defer _ = db.read_tx_faults.endCapture();
+ try testing.expectError(error.Internal, fx.body(.@"24h", window(.@"24h", 1_700_000_000)));
+}
+
+test "a commit on another connection invalidates the cached body" {
+ var fx: CacheFixture = undefined;
+ try fx.init(testing.allocator);
+ defer fx.deinit();
+
+ const span = window(.@"1h", 1_700_000_000);
+ try fx.log(span.since + 10);
+ const before = try fx.body(.@"1h", span);
+
+ try fx.log(span.since + 20);
+ const after = try fx.body(.@"1h", span);
+ try testing.expect(!std.mem.eql(u8, before, after));
+ try testing.expect(std.mem.containsAtLeast(u8, after, 1, "\"queries\":2"));
+}
+
+test "a retention prune through another connection replaces the body and the watermark" {
+ var fx: CacheFixture = undefined;
+ try fx.init(testing.allocator);
+ defer fx.deinit();
+
+ const span = window(.@"1h", 1_700_000_000);
+ try fx.log(span.since + 10);
+ const before = try fx.body(.@"1h", span);
+ try testing.expect(std.mem.containsAtLeast(u8, before, 1, "\"queries\":1"));
+
+ // Past the whole window: the row goes and the watermark advances, and both
+ // halves of the response must move together.
+ _ = try queries_repo.pruneOlderThan(&fx.writer, span.until);
+ const after = try fx.body(.@"1h", span);
+ try testing.expect(std.mem.containsAtLeast(u8, after, 1, "\"queries\":0"));
+
+ var watermark_buf: [64]u8 = undefined;
+ const watermark = try std.fmt.bufPrint(
+ &watermark_buf,
+ "\"available_since\":{d}",
+ .{span.until},
+ );
+ try testing.expect(std.mem.containsAtLeast(u8, after, 1, watermark));
+}
+
+test "a window roll rebuilds even with the data unchanged" {
+ var fx: CacheFixture = undefined;
+ try fx.init(testing.allocator);
+ defer fx.deinit();
+
+ const now: i64 = 1_700_000_000;
+ const first = try fx.body(.@"1h", window(.@"1h", now));
+ // One bucket later: same data, a different window, and so a different body.
+ const rolled = try fx.body(.@"1h", window(.@"1h", now + 60));
+ try testing.expect(!std.mem.eql(u8, first, rolled));
+
+ // The slot now holds the rolled window; asking for the earlier one again
+ // rebuilds rather than answering from a key that no longer matches.
+ const again = try fx.body(.@"1h", window(.@"1h", now));
+ try testing.expectEqualStrings(first, again);
+}
+
+test "a failed commit installs nothing and leaves the stored entry alone" {
+ var fx: CacheFixture = undefined;
+ try fx.init(testing.allocator);
+ defer fx.deinit();
+
+ const span = window(.@"1h", 1_700_000_000);
+ try fx.log(span.since + 10);
+ const stored = try fx.body(.@"1h", span);
+
+ // A commit that fails on a rebuild: the key has moved, so this is a miss.
+ try fx.log(span.since + 20);
+ db.read_tx_faults.failNextCommit();
+ db.read_tx_faults.beginCapture();
+ try testing.expectError(error.Internal, fx.body(.@"1h", span));
+ try testing.expectEqual(@as(usize, 1), db.read_tx_faults.endCapture());
+
+ // Nothing was published under the new key: the next request rebuilds and
+ // sees the second row, rather than being served the failed read's work or
+ // the first row's body under a key that now describes two.
+ const rebuilt = try fx.body(.@"1h", span);
+ try testing.expect(!std.mem.eql(u8, stored, rebuilt));
+ try testing.expect(std.mem.containsAtLeast(u8, rebuilt, 1, "\"queries\":2"));
+}
diff --git a/src/web/handlers/queries.zig b/src/web/handlers/queries.zig
index 0d05996..4388347 100644
--- a/src/web/handlers/queries.zig
+++ b/src/web/handlers/queries.zig
@@ -281,7 +281,7 @@ fn openLog() !db.Db {
/// upstream answer carries one. A fixture that broke those ties would let a
/// serializer regression pass here and fail on real rows.
fn seed(database: *db.Db, count: usize) !void {
- var writer = try queries_repo.BatchWriter.init(database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, database);
defer writer.deinit();
var rows: [16]queries_repo.Row = undefined;
for (rows[0..count], 0..) |*row, i| {
diff --git a/src/web/handlers/stats.zig b/src/web/handlers/stats.zig
deleted file mode 100644
index 4bd6e38..0000000
--- a/src/web/handlers/stats.zig
+++ /dev/null
@@ -1,574 +0,0 @@
-//! The five period endpoints: `GET /api/stats` and `/api/stats/timeseries`
-//! (ruling 13), and `/api/stats/types`, `/api/stats/routes` and
-//! `/api/stats/clients` (milestone 30).
-//!
-//! One period grammar, four widths, and one window shared by all five: a
-//! request for the same period gets the same `since`/`until` from every
-//! endpoint, so the totals describe exactly the span the charts draw rather
-//! than a neighbouring one.
-//!
-//! That is window coherence, not identical counts. Each endpoint is its own
-//! request against its own snapshot, so queries logged between two of them move
-//! one panel and not the other. Only a box with nothing writing to it — a test
-//! — can expect the breakdowns to sum to the totals exactly.
-//!
-//! Buckets are aligned to the UTC grid, not to the moment of the request. Every
-//! width divides a day, so flooring the current time to a multiple of the width
-//! puts each bucket on the same boundary a human reads off a clock, and two
-//! requests a second apart return the same bucket starts. The last bucket is
-//! the one in progress; it fills as the period runs.
-//!
-//! The aggregates run on the web task's own query-log connection (m7 ruling 21),
-//! which every connection task shares. SQLite's serialized mode makes one call
-//! safe; it does not make a transaction safe, so `WebState.querylog_lock` covers
-//! the whole read and a second BEGIN can never land inside the first. Each
-//! response takes one deferred read transaction, so its aggregate and the
-//! `coverage` beside it describe one database state: retention cannot prune
-//! between them and hand a client pre-prune rows tagged with a post-prune
-//! watermark. Deferred, not `db.Tx`'s BEGIN IMMEDIATE, which would stall the
-//! logger and retention behind an HTTP response.
-//!
-//! The lock is released before the response is written: the body is already
-//! built in the request arena, and holding a database lock across a socket
-//! write would let one slow client serialize every other reader.
-
-const std = @import("std");
-
-const coverage = @import("../coverage.zig");
-const db = @import("../../storage/db.zig");
-const http_util = @import("../http_util.zig");
-const queries_repo = @import("../../storage/repositories/queries_repo.zig");
-const server = @import("../server.zig");
-
-const log = std.log.scoped(.web_stats);
-
-/// The four periods ruling 13 defines. The tag names are the wire spellings.
-pub const Period = enum {
- @"1h",
- @"24h",
- @"7d",
- @"30d",
-
- pub fn parse(text: []const u8) ?Period {
- return std.meta.stringToEnum(Period, text);
- }
-
- /// Ruling 13: 1h→60×1m, 24h→48×30m, 7d→168×1h, 30d→120×6h.
- pub fn bucketSeconds(self: Period) u32 {
- return switch (self) {
- .@"1h" => 60,
- .@"24h" => 30 * 60,
- .@"7d" => 60 * 60,
- .@"30d" => 6 * 60 * 60,
- };
- }
-
- pub fn bucketCount(self: Period) u32 {
- return switch (self) {
- .@"1h" => 60,
- .@"24h" => 48,
- .@"7d" => 168,
- .@"30d" => 120,
- };
- }
-
- pub fn label(self: Period) []const u8 {
- return @tagName(self);
- }
-};
-
-pub const default_period: Period = .@"24h";
-
-/// The widest period's bucket count, so one stack array serves every request.
-pub const max_buckets = 168;
-
-comptime {
- for (std.enums.values(Period)) |period| {
- std.debug.assert(period.bucketCount() <= max_buckets);
- // The UTC alignment argument holds only while every width divides a day.
- std.debug.assert(86_400 % period.bucketSeconds() == 0);
- }
-}
-
-pub const Window = struct {
- /// Inclusive, on the bucket grid.
- since: i64,
- /// Exclusive: the end of the bucket that `now` falls in.
- until: i64,
- bucket_seconds: u32,
- bucket_count: u32,
-};
-
-pub fn window(period: Period, now_unix: i64) Window {
- const width: i64 = period.bucketSeconds();
- const count: i64 = period.bucketCount();
- const until = @divFloor(now_unix, width) * width + width;
- return .{
- .since = until - width * count,
- .until = until,
- .bucket_seconds = period.bucketSeconds(),
- .bucket_count = period.bucketCount(),
- };
-}
-
-pub const TotalsBody = struct {
- period: []const u8,
- since: i64,
- until: i64,
- queries: u64,
- blocked: u64,
- clients: u64,
- avg_response_time_us: ?i64,
- /// Judged against `since`, which is the window this body reports on — so a
- /// dashboard can say "history starts here" instead of charting a pruned
- /// stretch as a quiet one.
- coverage: coverage.Coverage,
-};
-
-pub const TimeseriesBody = struct {
- period: []const u8,
- since: i64,
- until: i64,
- bucket_seconds: u32,
- buckets: []const queries_repo.Bucket,
- coverage: coverage.Coverage,
-};
-
-pub const TypesBody = struct {
- period: []const u8,
- since: i64,
- until: i64,
- types: []const queries_repo.TypeCount,
- coverage: coverage.Coverage,
-};
-
-pub const RoutesBody = struct {
- period: []const u8,
- since: i64,
- until: i64,
- routes: []const queries_repo.RouteCount,
- coverage: coverage.Coverage,
-};
-
-pub const ClientsBody = struct {
- period: []const u8,
- since: i64,
- until: i64,
- bucket_seconds: u32,
- clients: []const queries_repo.ClientSeries,
- other: []const u64,
- coverage: coverage.Coverage,
-};
-
-/// Everything one response reads from the query log, so the caller can end the
-/// transaction and drop the lock before it serializes anything.
-fn Read(comptime T: type) type {
- return struct {
- data: T,
- coverage: coverage.Coverage,
- };
-}
-
-const ReadScope = server.QuerylogRead;
-
-fn readTotals(
- state: *server.WebState,
- io: std.Io,
- database: *db.Db,
- span: Window,
-) db.Error!Read(queries_repo.StatsTotals) {
- var scope = try ReadScope.open(state, io, database);
- errdefer scope.abort();
- const read: Read(queries_repo.StatsTotals) = .{
- .data = try queries_repo.statsTotals(database, span.since, span.until),
- .coverage = try coverage.read(database, span.since),
- };
- try scope.commit();
- return read;
-}
-
-fn readTimeseries(
- state: *server.WebState,
- io: std.Io,
- database: *db.Db,
- span: Window,
- out: []queries_repo.Bucket,
-) db.Error!Read(usize) {
- var scope = try ReadScope.open(state, io, database);
- errdefer scope.abort();
- const read: Read(usize) = .{
- .data = try queries_repo.timeseries(database, span.since, span.bucket_seconds, out),
- .coverage = try coverage.read(database, span.since),
- };
- try scope.commit();
- return read;
-}
-
-fn readTypes(
- state: *server.WebState,
- io: std.Io,
- database: *db.Db,
- arena: std.mem.Allocator,
- span: Window,
-) db.Error!Read([]const queries_repo.TypeCount) {
- var scope = try ReadScope.open(state, io, database);
- errdefer scope.abort();
- const list = try queries_repo.statsTypes(database, arena, span.since, span.until);
- const read: Read([]const queries_repo.TypeCount) = .{
- .data = list.items,
- .coverage = try coverage.read(database, span.since),
- };
- try scope.commit();
- return read;
-}
-
-fn readRoutes(
- state: *server.WebState,
- io: std.Io,
- database: *db.Db,
- arena: std.mem.Allocator,
- span: Window,
-) db.Error!Read([]const queries_repo.RouteCount) {
- var scope = try ReadScope.open(state, io, database);
- errdefer scope.abort();
- const list = try queries_repo.statsRoutes(database, arena, span.since, span.until);
- const read: Read([]const queries_repo.RouteCount) = .{
- .data = list.items,
- .coverage = try coverage.read(database, span.since),
- };
- try scope.commit();
- return read;
-}
-
-fn readClients(
- state: *server.WebState,
- io: std.Io,
- database: *db.Db,
- arena: std.mem.Allocator,
- span: Window,
-) db.Error!Read(queries_repo.ClientsBreakdown) {
- var scope = try ReadScope.open(state, io, database);
- errdefer scope.abort();
- const read: Read(queries_repo.ClientsBreakdown) = .{
- .data = try queries_repo.statsClients(
- database,
- arena,
- span.since,
- span.bucket_seconds,
- span.bucket_count,
- ),
- .coverage = try coverage.read(database, span.since),
- };
- try scope.commit();
- return read;
-}
-
-pub fn totals(
- state: *server.WebState,
- io: std.Io,
- request: *http_util.Request,
-) http_util.HandlerError!void {
- const period = periodParam(request.query) catch return badPeriod(request);
- const database = state.querylog_db orelse return unavailable(request);
- const span = window(period, std.Io.Clock.real.now(io).toSeconds());
-
- const read = readTotals(state, io, database, span) catch |err| {
- return internal(request, "stats totals", err);
- };
-
- return http_util.respondJson(request, .ok, TotalsBody{
- .period = period.label(),
- .since = span.since,
- .until = span.until,
- .queries = read.data.queries,
- .blocked = read.data.blocked,
- .clients = read.data.distinct_clients,
- .avg_response_time_us = read.data.avg_response_time_us,
- .coverage = read.coverage,
- }, &.{});
-}
-
-pub fn timeseries(
- state: *server.WebState,
- io: std.Io,
- request: *http_util.Request,
-) http_util.HandlerError!void {
- const period = periodParam(request.query) catch return badPeriod(request);
- const database = state.querylog_db orelse return unavailable(request);
- const span = window(period, std.Io.Clock.real.now(io).toSeconds());
-
- var buckets: [max_buckets]queries_repo.Bucket = undefined;
- const out = buckets[0..span.bucket_count];
- const read = readTimeseries(state, io, database, span, out) catch |err| {
- return internal(request, "stats timeseries", err);
- };
-
- return http_util.respondJson(request, .ok, TimeseriesBody{
- .period = period.label(),
- .since = span.since,
- .until = span.until,
- .bucket_seconds = span.bucket_seconds,
- .buckets = out[0..read.data],
- .coverage = read.coverage,
- }, &.{});
-}
-
-pub fn types(
- state: *server.WebState,
- io: std.Io,
- request: *http_util.Request,
-) http_util.HandlerError!void {
- const period = periodParam(request.query) catch return badPeriod(request);
- const database = state.querylog_db orelse return unavailable(request);
- const span = window(period, std.Io.Clock.real.now(io).toSeconds());
-
- const read = readTypes(state, io, database, request.arena, span) catch |err| {
- return internal(request, "stats types", err);
- };
-
- return http_util.respondJson(request, .ok, TypesBody{
- .period = period.label(),
- .since = span.since,
- .until = span.until,
- .types = read.data,
- .coverage = read.coverage,
- }, &.{});
-}
-
-pub fn routes(
- state: *server.WebState,
- io: std.Io,
- request: *http_util.Request,
-) http_util.HandlerError!void {
- const period = periodParam(request.query) catch return badPeriod(request);
- const database = state.querylog_db orelse return unavailable(request);
- const span = window(period, std.Io.Clock.real.now(io).toSeconds());
-
- const read = readRoutes(state, io, database, request.arena, span) catch |err| {
- return internal(request, "stats routes", err);
- };
-
- return http_util.respondJson(request, .ok, RoutesBody{
- .period = period.label(),
- .since = span.since,
- .until = span.until,
- .routes = read.data,
- .coverage = read.coverage,
- }, &.{});
-}
-
-pub fn clients(
- state: *server.WebState,
- io: std.Io,
- request: *http_util.Request,
-) http_util.HandlerError!void {
- const period = periodParam(request.query) catch return badPeriod(request);
- const database = state.querylog_db orelse return unavailable(request);
- const span = window(period, std.Io.Clock.real.now(io).toSeconds());
-
- const read = readClients(state, io, database, request.arena, span) catch |err| {
- return internal(request, "stats clients", err);
- };
-
- return http_util.respondJson(request, .ok, ClientsBody{
- .period = period.label(),
- .since = span.since,
- .until = span.until,
- .bucket_seconds = span.bucket_seconds,
- .clients = read.data.clients,
- .other = read.data.other,
- .coverage = read.coverage,
- }, &.{});
-}
-
-pub const PeriodError = error{BadPeriod};
-
-/// An absent `period` is the default; anything else it cannot read is a 400,
-/// never a silent fallback — a typo must not return a window nobody asked for.
-fn periodParam(query: []const u8) PeriodError!Period {
- var buf: [8]u8 = undefined;
- const found = http_util.queryValue(query, "period", &buf) catch return error.BadPeriod;
- const text = found orelse return default_period;
- return Period.parse(text) orelse error.BadPeriod;
-}
-
-fn badPeriod(request: *http_util.Request) http_util.HandlerError!void {
- return http_util.respondError(request, .bad_request, "period must be one of 1h, 24h, 7d, 30d");
-}
-
-fn unavailable(request: *http_util.Request) http_util.HandlerError!void {
- return http_util.respondError(request, .service_unavailable, "query log unavailable");
-}
-
-/// The one thing this file logs. A failed aggregate is a fault in the box, not
-/// a property of the request, and the client is told nothing beyond "internal
-/// error" (ruling 8, PLAN §19).
-fn internal(
- request: *http_util.Request,
- what: []const u8,
- err: db.Error,
-) http_util.HandlerError!void {
- log.warn("{s} failed: {s}", .{ what, @errorName(err) });
- return http_util.respondError(request, .internal_server_error, "internal error");
-}
-
-// ---------------------------------------------------------------------------
-// tests
-// ---------------------------------------------------------------------------
-
-const querylog_schema = @import("../../storage/querylog_schema.zig");
-const testing = std.testing;
-
-test "the period grammar accepts exactly the four spellings" {
- try testing.expectEqual(Period.@"1h", Period.parse("1h").?);
- try testing.expectEqual(Period.@"24h", Period.parse("24h").?);
- try testing.expectEqual(Period.@"7d", Period.parse("7d").?);
- try testing.expectEqual(Period.@"30d", Period.parse("30d").?);
- try testing.expectEqual(@as(?Period, null), Period.parse("12h"));
- try testing.expectEqual(@as(?Period, null), Period.parse("1H"));
- try testing.expectEqual(@as(?Period, null), Period.parse(""));
-}
-
-test "an absent period defaults and a bad one is rejected" {
- try testing.expectEqual(default_period, try periodParam(""));
- try testing.expectEqual(default_period, try periodParam("limit=5"));
- try testing.expectEqual(Period.@"7d", try periodParam("period=7d"));
- try testing.expectError(error.BadPeriod, periodParam("period=12h"));
- try testing.expectError(error.BadPeriod, periodParam("period=%2"));
- // Longer than any spelling: rejected rather than truncated to "1h".
- try testing.expectError(error.BadPeriod, periodParam("period=1hhhhhhhhhh"));
-}
-
-test "each period spans its own bucket width times its count" {
- for (std.enums.values(Period)) |period| {
- const span = window(period, 1_700_000_000);
- const width: i64 = period.bucketSeconds();
- try testing.expectEqual(width * @as(i64, period.bucketCount()), span.until - span.since);
- }
-}
-
-test "the window sits on the UTC grid and ends with the bucket in progress" {
- // 2023-11-14T22:13:20Z, which is not on any bucket boundary.
- const now: i64 = 1_700_000_000;
- const span = window(.@"24h", now);
-
- try testing.expectEqual(@as(i64, 0), @rem(span.since, 1800));
- try testing.expectEqual(@as(i64, 0), @rem(span.until, 1800));
- try testing.expect(span.until > now);
- try testing.expect(span.until - now <= 1800);
- try testing.expectEqual(@as(u32, 48), span.bucket_count);
-}
-
-test "two requests inside one bucket see the same window" {
- // A bucket boundary, so the offsets below stay inside one minute.
- const boundary: i64 = 1_700_000_000 - @rem(1_700_000_000, 60);
- const first = window(.@"1h", boundary);
- const second = window(.@"1h", boundary + 59);
- try testing.expectEqual(first.since, second.since);
- try testing.expectEqual(first.until, second.until);
-
- const next = window(.@"1h", boundary + 60);
- try testing.expectEqual(first.until + 60, next.until);
-}
-
-test "a timestamp exactly on a boundary starts a new bucket" {
- const span = window(.@"7d", 1_700_000_000 - 1_700_000_000 % 3600);
- try testing.expectEqual(@as(i64, 0), @rem(span.since, 3600));
- try testing.expectEqual(@as(u32, 168), span.bucket_count);
-}
-
-fn openLog() !db.Db {
- var database = try db.Db.open(":memory:", .{ .mode = .memory });
- errdefer database.close();
- try db.applyPragmas(&database, .{});
- try database.exec(querylog_schema.ddl);
- return database;
-}
-
-fn writeRow(writer: *queries_repo.BatchWriter, timestamp: i64, blocked: bool, cached: ?bool) !void {
- const rows = [_]queries_repo.Row{.{
- .timestamp = timestamp,
- .domain = "example.com",
- .client_ip = "192.0.2.10",
- .qtype = 1,
- .qclass = 1,
- .rcode = 0,
- .blocked = blocked,
- .response_time_us = 1000,
- .cache_hit = cached,
- .upstream = null,
- .group_id = 1,
- .group_name = "default",
- .policy_action = if (blocked) .block else .allow,
- .policy_reason = if (blocked) .blocklist_domain else .no_match,
- .matched = null,
- .source_id = null,
- .source_name = null,
- .cname_target = null,
- .safe_search_target = null,
- .route_kind = if (blocked) .blocked else .upstream,
- .forward_zone = null,
- }};
- try writer.writeBatch(&rows);
-}
-
-test "the totals and the buckets agree over the same window" {
- var database = try openLog();
- defer database.close();
-
- const now: i64 = 1_700_000_000;
- const span = window(.@"1h", now);
-
- var writer = try queries_repo.BatchWriter.init(&database);
- defer writer.deinit();
- // One row in the first bucket, two in the last, one just outside.
- try writeRow(&writer, span.since, false, false);
- try writeRow(&writer, span.until - 1, true, false);
- try writeRow(&writer, span.until - 2, false, true);
- try writeRow(&writer, span.since - 1, false, false);
-
- const result = try queries_repo.statsTotals(&database, span.since, span.until);
- try testing.expectEqual(@as(u64, 3), result.queries);
- try testing.expectEqual(@as(u64, 1), result.blocked);
- try testing.expectEqual(@as(u64, 1), result.distinct_clients);
- try testing.expectEqual(@as(?i64, 1000), result.avg_response_time_us);
-
- var buckets: [max_buckets]queries_repo.Bucket = undefined;
- const out = buckets[0..span.bucket_count];
- const written = try queries_repo.timeseries(&database, span.since, span.bucket_seconds, out);
- try testing.expectEqual(@as(usize, 60), written);
-
- var summed: u64 = 0;
- var blocked: u64 = 0;
- for (out) |bucket| {
- summed += bucket.queries;
- blocked += bucket.blocked;
- }
- try testing.expectEqual(result.queries, summed);
- try testing.expectEqual(result.blocked, blocked);
-
- try testing.expectEqual(span.since, out[0].ts);
- try testing.expectEqual(@as(u64, 1), out[0].queries);
- try testing.expectEqual(@as(u64, 2), out[59].queries);
- try testing.expectEqual(span.until - span.bucket_seconds, out[59].ts);
-}
-
-test "an empty window reports zeros with a null mean" {
- var database = try openLog();
- defer database.close();
-
- const span = window(.@"30d", 1_700_000_000);
- const result = try queries_repo.statsTotals(&database, span.since, span.until);
- try testing.expectEqual(@as(u64, 0), result.queries);
- try testing.expectEqual(@as(?i64, null), result.avg_response_time_us);
-
- var buckets: [max_buckets]queries_repo.Bucket = undefined;
- const out = buckets[0..span.bucket_count];
- try testing.expectEqual(@as(usize, 120), try queries_repo.timeseries(
- &database,
- span.since,
- span.bucket_seconds,
- out,
- ));
- for (out) |bucket| try testing.expectEqual(@as(u64, 0), bucket.queries);
-}
diff --git a/src/web/openapi.yaml b/src/web/openapi.yaml
index 7222e7a..b96cd7f 100644
--- a/src/web/openapi.yaml
+++ b/src/web/openapi.yaml
@@ -422,139 +422,29 @@ paths:
"503":
$ref: "#/components/responses/Unavailable"
- /api/stats:
+ /api/overview:
get:
- summary: Totals for a period
- parameters:
- - $ref: "#/components/parameters/Period"
- responses:
- "200":
- description: Totals over the period's UTC-aligned window.
- content:
- application/json:
- schema:
- $ref: "#/components/schemas/StatsTotals"
- "400":
- $ref: "#/components/responses/BadRequest"
- "401":
- $ref: "#/components/responses/Unauthorized"
- "429":
- $ref: "#/components/responses/RateLimited"
- "500":
- $ref: "#/components/responses/Internal"
- "503":
- $ref: "#/components/responses/Unavailable"
-
- /api/stats/timeseries:
- get:
- summary: Bucketed counts for a period
+ summary: Everything the Overview page draws, for one period
description: |
- Fixed-width UTC buckets covering the same window `/api/stats`
- reports for the period: 1h into 60 one-minute buckets, 24h into 48
- half-hour buckets, 7d into 168 one-hour buckets, 30d into 120
- six-hour buckets. Empty buckets are zero-filled.
- parameters:
- - $ref: "#/components/parameters/Period"
- responses:
- "200":
- description: The bucket series.
- content:
- application/json:
- schema:
- $ref: "#/components/schemas/StatsTimeseries"
- "400":
- $ref: "#/components/responses/BadRequest"
- "401":
- $ref: "#/components/responses/Unauthorized"
- "429":
- $ref: "#/components/responses/RateLimited"
- "500":
- $ref: "#/components/responses/Internal"
- "503":
- $ref: "#/components/responses/Unavailable"
+ One response over one read transaction: the period's totals, its
+ fixed-width UTC buckets, the per-client series, the query-type
+ breakdown and the answering-route breakdown, plus the coverage
+ watermark judged against the same window. The panels therefore describe
+ one database state rather than five, so the breakdowns sum to the
+ totals on a quiet box.
- /api/stats/types:
- get:
- summary: Query-type breakdown for a period
- description: |
- How many queries of each DNS type the period's window holds, over the
- same UTC-aligned window `/api/stats` reports for. Rows carry the numeric
- type only: the type-name table lives in the admin, and a second copy
- here would drift out of agreement with it. `qtype` is nullable in the
- query log, so the rows that carry no type group into a row of their own
- rather than vanishing from a breakdown that claims to add up. Ordered by
- count descending, then type ascending with the null row last. Types
- absent from the window are absent from the list.
+ Buckets are UTC-aligned and zero-filled: 1h into 60 one-minute buckets,
+ 24h into 48 half-hour buckets, 7d into 168 one-hour buckets, 30d into
+ 120 six-hour buckets. The last bucket is the one in progress.
parameters:
- $ref: "#/components/parameters/Period"
responses:
"200":
- description: The type breakdown.
+ description: The period's overview.
content:
application/json:
schema:
- $ref: "#/components/schemas/StatsTypes"
- "400":
- $ref: "#/components/responses/BadRequest"
- "401":
- $ref: "#/components/responses/Unauthorized"
- "429":
- $ref: "#/components/responses/RateLimited"
- "500":
- $ref: "#/components/responses/Internal"
- "503":
- $ref: "#/components/responses/Unavailable"
-
- /api/stats/routes:
- get:
- summary: How the period's queries were answered
- description: |
- A breakdown by answering route over the same window `/api/stats`
- reports for. `source` is the answering resolver's identity — the
- upstream url on `upstream` rows, the zone on `forward_zone` rows, null
- on every other kind and on rows whose identity the log did not record.
- It is not the blocklist a block came from. Ordered by count descending,
- then route ascending, then source ascending with nulls last.
- parameters:
- - $ref: "#/components/parameters/Period"
- responses:
- "200":
- description: The route breakdown.
- content:
- application/json:
- schema:
- $ref: "#/components/schemas/StatsRoutes"
- "400":
- $ref: "#/components/responses/BadRequest"
- "401":
- $ref: "#/components/responses/Unauthorized"
- "429":
- $ref: "#/components/responses/RateLimited"
- "500":
- $ref: "#/components/responses/Internal"
- "503":
- $ref: "#/components/responses/Unavailable"
-
- /api/stats/clients:
- get:
- summary: Per-client bucketed counts for a period
- description: |
- One zero-filled series per client, bucketed exactly like
- `/api/stats/timeseries` so the two charts share an x-axis. The eight
- clients with the most queries in the window are named, ranked by count
- descending then address ascending; every other client sums into
- `other`, which is always present and always holds one entry per bucket
- in the window — including when `clients` is empty, when no client fell
- outside the named eight, and when the window holds no queries at all.
- parameters:
- - $ref: "#/components/parameters/Period"
- responses:
- "200":
- description: The per-client series.
- content:
- application/json:
- schema:
- $ref: "#/components/schemas/StatsClients"
+ $ref: "#/components/schemas/Overview"
"400":
$ref: "#/components/responses/BadRequest"
"401":
@@ -2316,31 +2206,6 @@ components:
type: integer
description: How many resolved events the purge removed; zero when there were none.
- StatsTotals:
- type: object
- required: [period, since, until, queries, blocked, clients, avg_response_time_us, coverage]
- properties:
- period:
- type: string
- enum: [1h, 24h, 7d, 30d]
- since:
- type: integer
- description: Window start, unix seconds, inclusive.
- until:
- type: integer
- description: Window end, unix seconds, exclusive.
- queries: { type: integer }
- blocked: { type: integer }
- clients:
- type: integer
- description: Distinct client addresses in the window.
- avg_response_time_us:
- type: integer
- nullable: true
- description: Null when no query in the window recorded a time.
- coverage:
- $ref: "#/components/schemas/Coverage"
-
Bucket:
type: object
required: [ts, queries, blocked, cached]
@@ -2352,23 +2217,6 @@ components:
blocked: { type: integer }
cached: { type: integer }
- StatsTimeseries:
- type: object
- required: [period, since, until, bucket_seconds, buckets, coverage]
- properties:
- period:
- type: string
- enum: [1h, 24h, 7d, 30d]
- since: { type: integer }
- until: { type: integer }
- bucket_seconds: { type: integer }
- buckets:
- type: array
- items:
- $ref: "#/components/schemas/Bucket"
- coverage:
- $ref: "#/components/schemas/Coverage"
-
TypeCount:
type: object
required: [qtype, count]
@@ -2381,22 +2229,6 @@ components:
recorded no type, not an absent row.
count: { type: integer }
- StatsTypes:
- type: object
- required: [period, since, until, types, coverage]
- properties:
- period:
- type: string
- enum: [1h, 24h, 7d, 30d]
- since: { type: integer }
- until: { type: integer }
- types:
- type: array
- items:
- $ref: "#/components/schemas/TypeCount"
- coverage:
- $ref: "#/components/schemas/Coverage"
-
RouteCount:
type: object
required: [route, source, count]
@@ -2412,22 +2244,6 @@ components:
row recorded no identity.
count: { type: integer }
- StatsRoutes:
- type: object
- required: [period, since, until, routes, coverage]
- properties:
- period:
- type: string
- enum: [1h, 24h, 7d, 30d]
- since: { type: integer }
- until: { type: integer }
- routes:
- type: array
- items:
- $ref: "#/components/schemas/RouteCount"
- coverage:
- $ref: "#/components/schemas/Coverage"
-
ClientSeries:
type: object
required: [client, buckets]
@@ -2443,18 +2259,47 @@ components:
items:
type: integer
- StatsClients:
+ OverviewTotals:
type: object
- required: [period, since, until, bucket_seconds, clients, other, coverage]
+ required: [queries, blocked, clients, avg_response_time_us]
+ properties:
+ queries: { type: integer }
+ blocked: { type: integer }
+ clients:
+ type: integer
+ description: Distinct client addresses in the window.
+ avg_response_time_us:
+ type: integer
+ nullable: true
+ description: Null when no query in the window recorded a time.
+
+ Overview:
+ type: object
+ required: [period, since, until, bucket_seconds, totals, buckets, clients, other, types, routes, coverage]
properties:
period:
type: string
enum: [1h, 24h, 7d, 30d]
- since: { type: integer }
- until: { type: integer }
+ since:
+ type: integer
+ description: Window start, unix seconds, inclusive.
+ until:
+ type: integer
+ description: Window end, unix seconds, exclusive.
bucket_seconds: { type: integer }
+ totals:
+ $ref: "#/components/schemas/OverviewTotals"
+ buckets:
+ type: array
+ description: One entry per bucket in the window, zero-filled.
+ items:
+ $ref: "#/components/schemas/Bucket"
clients:
type: array
+ description: |
+ The eight clients with the most queries in the window, ranked by
+ count descending then address ascending. Every other client sums
+ into `other`.
items:
$ref: "#/components/schemas/ClientSeries"
other:
@@ -2466,6 +2311,30 @@ components:
eight, and when the window holds no queries at all.
items:
type: integer
+ types:
+ type: array
+ description: |
+ How many queries of each DNS type the window holds. Rows carry the
+ numeric type only: the type-name table lives in the admin, and a
+ second copy here would drift out of agreement with it. `qtype` is
+ nullable in the query log, so the rows that carry no type group
+ into a row of their own rather than vanishing from a breakdown that
+ claims to add up. Ordered by count descending, then type ascending
+ with the null row last. Types absent from the window are absent
+ from the list.
+ items:
+ $ref: "#/components/schemas/TypeCount"
+ routes:
+ type: array
+ description: |
+ A breakdown by answering route. `source` is the answering
+ resolver's identity — the upstream url on `upstream` rows, the zone
+ on `forward_zone` rows, null on every other kind and on rows whose
+ identity the log did not record. It is not the blocklist a block
+ came from. Ordered by count descending, then route ascending, then
+ source ascending with nulls last.
+ items:
+ $ref: "#/components/schemas/RouteCount"
coverage:
$ref: "#/components/schemas/Coverage"
diff --git a/src/web/routes.zig b/src/web/routes.zig
index 642ce37..a2b2f61 100644
--- a/src/web/routes.zig
+++ b/src/web/routes.zig
@@ -45,11 +45,11 @@ const local = @import("handlers/local.zig");
const lookup = @import("handlers/lookup.zig");
const metrics = @import("metrics.zig");
const openapi = @import("openapi.zig");
+const overview = @import("handlers/overview.zig");
const pause = @import("handlers/pause.zig");
const queries = @import("handlers/queries.zig");
const rules = @import("handlers/rules.zig");
const settings = @import("handlers/settings.zig");
-const stats = @import("handlers/stats.zig");
const upstreams = @import("handlers/upstreams.zig");
const version = @import("handlers/version.zig");
@@ -64,18 +64,14 @@ pub const table: []const router.RouteInfo = &.{
.{ .method = .POST, .pattern = "/api/auth/login", .auth = .open, .policy = .runtime_action, .handler = auth.login },
.{ .method = .POST, .pattern = "/api/auth/logout", .auth = .session, .policy = .runtime_action, .handler = auth.logout },
- // Query log, stats, live stream, lookup.
+ // Query log, overview, live stream, lookup.
.{ .method = .GET, .pattern = "/api/queries", .auth = .session, .policy = .read, .handler = queries.list },
.{ .method = .GET, .pattern = "/api/queries/live", .auth = .session, .policy = .read, .handler = live.stream, .rate_limit = .exempt },
// Listed after the literal `live`, which a linear first-match scan reaches
// first — though `{id}` would refuse it anyway, since it captures a
// positive integer and nothing else.
.{ .method = .GET, .pattern = "/api/queries/{id}", .auth = .session, .policy = .read, .handler = queries.detail },
- .{ .method = .GET, .pattern = "/api/stats", .auth = .session, .policy = .read, .handler = stats.totals },
- .{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .handler = stats.timeseries },
- .{ .method = .GET, .pattern = "/api/stats/types", .auth = .session, .policy = .read, .handler = stats.types },
- .{ .method = .GET, .pattern = "/api/stats/routes", .auth = .session, .policy = .read, .handler = stats.routes },
- .{ .method = .GET, .pattern = "/api/stats/clients", .auth = .session, .policy = .read, .handler = stats.clients },
+ .{ .method = .GET, .pattern = "/api/overview", .auth = .session, .policy = .read, .handler = overview.handle },
.{ .method = .GET, .pattern = "/api/lookup", .auth = .session, .policy = .read, .handler = lookup.handle },
// Diagnostics: the operational event log (milestone 27). The two purges are
@@ -161,7 +157,7 @@ const std = @import("std");
const testing = std.testing;
test "the table carries every endpoint of the milestone" {
- try testing.expectEqual(@as(usize, 64), table.len);
+ try testing.expectEqual(@as(usize, 60), table.len);
}
test "no two entries claim the same method and pattern" {
diff --git a/src/web/server.zig b/src/web/server.zig
index 1007483..bae1e19 100644
--- a/src/web/server.zig
+++ b/src/web/server.zig
@@ -175,6 +175,67 @@ pub const UpstreamBuild = struct {
bundle_lock: *std.Io.RwLock,
};
+/// The Overview response cache: one already-serialized body per period.
+///
+/// A slot is valid for exactly one `(window.until, data_version)` pair, so it
+/// expires both ways a stale Overview can arise — the window rolls onto the
+/// next bucket, or another connection (the logger, retention) commits and moves
+/// `PRAGMA data_version`. There is no time-to-live and no background refresh:
+/// nothing here can serve bytes that describe a database state the reader could
+/// not have seen.
+///
+/// Every field is read and written under `WebState.querylog_lock`, which is
+/// also what makes the cache single-flight: a second request for the same key
+/// waits for the first rebuild and then hits. The type carries no lock of its
+/// own precisely so that nobody can touch it without the one that matters.
+pub const OverviewCache = struct {
+ /// One per `overview.Period`, indexed by `@intFromEnum`. The handler asserts
+ /// the two counts agree.
+ pub const slot_count = 4;
+
+ const Slot = struct {
+ /// Empty until the first successful build; never a valid empty body,
+ /// since every response carries at least the period and the window.
+ body: []u8 = &.{},
+ until: i64 = 0,
+ data_version: i64 = 0,
+ };
+
+ slots: [slot_count]Slot = @splat(.{}),
+
+ /// The stored bytes for this key, or null. The caller copies them into its
+ /// request arena before releasing the lock: a later rebuild frees this
+ /// allocation.
+ pub fn get(self: *const OverviewCache, period_index: usize, until: i64, data_version: i64) ?[]const u8 {
+ const slot = &self.slots[period_index];
+ if (slot.body.len == 0) return null;
+ if (slot.until != until or slot.data_version != data_version) return null;
+ return slot.body;
+ }
+
+ /// Takes ownership of `body`, which must be a `gpa` allocation, and frees
+ /// whatever the slot held.
+ pub fn put(
+ self: *OverviewCache,
+ gpa: Allocator,
+ period_index: usize,
+ until: i64,
+ data_version: i64,
+ body: []u8,
+ ) void {
+ const slot = &self.slots[period_index];
+ gpa.free(slot.body);
+ slot.* = .{ .body = body, .until = until, .data_version = data_version };
+ }
+
+ pub fn deinit(self: *OverviewCache, gpa: Allocator) void {
+ for (&self.slots) |*slot| {
+ gpa.free(slot.body);
+ slot.* = .{};
+ }
+ }
+};
+
pub const WebState = struct {
gpa: Allocator,
web: model.Web = .{},
@@ -272,6 +333,9 @@ pub const WebState = struct {
/// the shared connection would fail and a third task's reads would land
/// inside someone else's snapshot.
querylog_lock: std.Io.Mutex = .init,
+ /// The Overview response cache, guarded by `querylog_lock` above. Whoever
+ /// owns the `WebState` calls `overview_cache.deinit`.
+ overview_cache: OverviewCache = .{},
/// The diagnostics event store, which owns a third connection of its own
/// and serializes every access — read and write — through its mutex. Null
/// when `Store.init` failed, which `/api/health` reports as `unavailable`
@@ -350,11 +414,24 @@ pub const QuerylogRead = struct {
pub fn open(state: *WebState, io: std.Io, database: *db.Db) db.Error!QuerylogRead {
state.querylog_lock.lockUncancelable(io);
errdefer state.querylog_lock.unlock(io);
+ var scope = try openLocked(state, io, database);
+ scope.held = true;
+ return scope;
+ }
+
+ /// The transaction alone, for a caller that already holds `querylog_lock`
+ /// and keeps holding it past `commit` — the overview handler, which decides
+ /// its response cache under the same one hold. Calling `open` there would
+ /// deadlock on a mutex the task already owns.
+ ///
+ /// The returned scope releases nothing: `commit` and `abort` end the
+ /// transaction and leave the lock to whoever took it.
+ pub fn openLocked(state: *WebState, io: std.Io, database: *db.Db) db.Error!QuerylogRead {
return .{
.state = state,
.io = io,
.tx = try db.ReadTx.begin(database),
- .held = true,
+ .held = false,
};
}
diff --git a/src/web/web_integration_test.zig b/src/web/web_integration_test.zig
index f07fc06..1818a31 100644
--- a/src/web/web_integration_test.zig
+++ b/src/web/web_integration_test.zig
@@ -83,7 +83,7 @@ const handlers_lookup = @import("handlers/lookup.zig");
const handlers_pause = @import("handlers/pause.zig");
const handlers_queries = @import("handlers/queries.zig");
const handlers_settings = @import("handlers/settings.zig");
-const handlers_stats = @import("handlers/stats.zig");
+const handlers_overview = @import("handlers/overview.zig");
const handlers_version = @import("handlers/version.zig");
const Certificate = std.crypto.Certificate;
@@ -680,6 +680,7 @@ const Env = struct {
self.state.live_hash.deinit(gpa);
self.state.proxies.deinit(gpa);
+ self.state.overview_cache.deinit(gpa);
self.tables.deinit(gpa);
gpa.destroy(self.hub);
self.limiter.deinit();
@@ -738,7 +739,7 @@ fn seedQueryLog(database: *db.Db) !void {
\\UPDATE querylog_meta SET created_at = 1700000000, available_since = 1700000000 WHERE id = 1
);
- var writer = try queries_repo.BatchWriter.init(database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, database);
defer writer.deinit();
var domain_buf: [32]u8 = undefined;
@@ -838,7 +839,7 @@ fn seedQueryLog(database: *db.Db) !void {
const recent_clients = 3;
fn seedRecentTraffic(database: *db.Db, now: i64) !void {
- var writer = try queries_repo.BatchWriter.init(database);
+ var writer = try queries_repo.BatchWriter.init(testing.allocator, database);
defer writer.deinit();
const Shape = struct {
@@ -1048,15 +1049,11 @@ const contract = [_]Contract{
// Refresh-all before any source row exists: nothing to fetch, 202 anyway.
.{ .method = .POST, .pattern = "/api/blocklists/update", .auth = .session, .policy = .runtime_action, .target = "/api/blocklists/update", .status = 202, .check = jsonShape(StatusList) },
- // Query log, stats, live stream, upstream health.
+ // Query log, overview, live stream, upstream health.
.{ .method = .GET, .pattern = "/api/queries", .auth = .session, .policy = .read, .target = "/api/queries?limit=10", .status = 200, .check = jsonShape(handlers_queries.Page) },
.{ .method = .GET, .pattern = "/api/queries/{id}", .auth = .session, .policy = .read, .target = "/api/queries/27", .status = 200, .check = jsonShape(provenance_view.QueryDetail) },
.{ .method = .GET, .pattern = "/api/queries/live", .auth = .session, .policy = .read, .rate_limit = .exempt, .target = "/api/queries/live", .status = 200, .kind = .sse },
- .{ .method = .GET, .pattern = "/api/stats", .auth = .session, .policy = .read, .target = "/api/stats?period=1h", .status = 200, .check = jsonShape(handlers_stats.TotalsBody) },
- .{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .target = "/api/stats/timeseries?period=1h", .status = 200, .check = jsonShape(handlers_stats.TimeseriesBody) },
- .{ .method = .GET, .pattern = "/api/stats/types", .auth = .session, .policy = .read, .target = "/api/stats/types?period=1h", .status = 200, .check = jsonShape(handlers_stats.TypesBody) },
- .{ .method = .GET, .pattern = "/api/stats/routes", .auth = .session, .policy = .read, .target = "/api/stats/routes?period=1h", .status = 200, .check = jsonShape(handlers_stats.RoutesBody) },
- .{ .method = .GET, .pattern = "/api/stats/clients", .auth = .session, .policy = .read, .target = "/api/stats/clients?period=1h", .status = 200, .check = jsonShape(handlers_stats.ClientsBody) },
+ .{ .method = .GET, .pattern = "/api/overview", .auth = .session, .policy = .read, .target = "/api/overview?period=1h", .status = 200, .check = jsonShape(handlers_overview.Body) },
// Diagnostics. The seeded store holds one active episode (id 1) and one
// resolved one, so both the page and the detail answer with real rows.
@@ -2587,11 +2584,7 @@ fn detailUnavailable(io: std.Io, env: *Env) anyerror!void {
const targets = [_][]const u8{
"/api/queries/1",
"/api/queries?limit=1",
- "/api/stats",
- "/api/stats/timeseries",
- "/api/stats/types",
- "/api/stats/routes",
- "/api/stats/clients",
+ "/api/overview",
};
for (targets) |target| {
try conn.request("GET", target, null, null);
@@ -2657,27 +2650,20 @@ fn coverageWalk(io: std.Io, env: *Env) anyerror!void {
);
try testing.expect(!partial.coverage.complete);
- // The stats endpoints judge the same watermark against their own aligned
- // window, which for any live period starts well after the seeded rows.
- try conn.request("GET", "/api/stats?period=1h", null, null);
- const totals = try std.json.parseFromSliceLeaky(
- handlers_stats.TotalsBody,
+ // The overview judges the same watermark against its own aligned window,
+ // which for any live period starts well after the seeded rows.
+ try conn.request("GET", "/api/overview?period=1h", null, null);
+ const overview_body = try std.json.parseFromSliceLeaky(
+ handlers_overview.Body,
arena,
(try conn.receive(&body_buf)).body,
.{ .ignore_unknown_fields = false },
);
- try testing.expectEqual(seeded_available_since, totals.coverage.available_since);
- try testing.expectEqual(totals.since >= seeded_available_since, totals.coverage.complete);
-
- try conn.request("GET", "/api/stats/timeseries?period=1h", null, null);
- const series = try std.json.parseFromSliceLeaky(
- handlers_stats.TimeseriesBody,
- arena,
- (try conn.receive(&body_buf)).body,
- .{ .ignore_unknown_fields = false },
+ try testing.expectEqual(seeded_available_since, overview_body.coverage.available_since);
+ try testing.expectEqual(
+ overview_body.since >= seeded_available_since,
+ overview_body.coverage.complete,
);
- try testing.expectEqual(totals.since, series.since);
- try testing.expectEqual(totals.coverage.complete, series.coverage.complete);
}
fn getJson(
@@ -2708,33 +2694,43 @@ fn emptyAggregations(io: std.Io, env: *Env) anyerror!void {
var body_buf: [256 * 1024]u8 = undefined;
// This environment's only rows are the fixed 2023 seed, so every live
- // window is empty. The empty bodies are exact, not merely parseable.
- const types_body = try getJson(handlers_stats.TypesBody, arena, &conn, "/api/stats/types?period=1h", &body_buf);
- try testing.expectEqualStrings("1h", types_body.period);
- try testing.expectEqual(@as(usize, 0), types_body.types.len);
-
- const routes_body = try getJson(handlers_stats.RoutesBody, arena, &conn, "/api/stats/routes?period=1h", &body_buf);
- try testing.expectEqual(@as(usize, 0), routes_body.routes.len);
+ // window is empty. The empty body is exact, not merely parseable.
+ const body = try getJson(handlers_overview.Body, arena, &conn, "/api/overview?period=1h", &body_buf);
+ try testing.expectEqualStrings("1h", body.period);
+ try testing.expectEqual(@as(u64, 0), body.totals.queries);
+ try testing.expectEqual(@as(?i64, null), body.totals.avg_response_time_us);
+ try testing.expectEqual(@as(usize, 0), body.types.len);
+ try testing.expectEqual(@as(usize, 0), body.routes.len);
// `other` is present and bucket-count sized even here: a chart must never
// have to invent the residual series.
- const clients = try getJson(handlers_stats.ClientsBody, arena, &conn, "/api/stats/clients?period=1h", &body_buf);
- try testing.expectEqual(@as(usize, 0), clients.clients.len);
- try testing.expectEqual(@as(u32, 60), clients.bucket_seconds);
- try testing.expectEqual(@as(usize, 60), clients.other.len);
- for (clients.other) |count| try testing.expectEqual(@as(u64, 0), count);
+ try testing.expectEqual(@as(usize, 0), body.clients.len);
+ try testing.expectEqual(@as(u32, 60), body.bucket_seconds);
+ try testing.expectEqual(@as(usize, 60), body.buckets.len);
+ try testing.expectEqual(@as(usize, 60), body.other.len);
+ for (body.other) |count| try testing.expectEqual(@as(u64, 0), count);
// A window nobody covers is still reported as such, not as a quiet hour.
- try testing.expectEqual(seeded_available_since, types_body.coverage.available_since);
- try testing.expect(types_body.coverage.complete);
+ try testing.expectEqual(seeded_available_since, body.coverage.available_since);
+ try testing.expect(body.coverage.complete);
- for ([_][]const u8{ "/api/stats/types", "/api/stats/routes", "/api/stats/clients" }) |path| {
- var target_buf: [64]u8 = undefined;
- const target = try std.fmt.bufPrint(&target_buf, "{s}?period=12h", .{path});
- try conn.request("GET", target, null, null);
- const bad = try conn.receive(&body_buf);
- try testing.expectEqual(@as(u16, 400), bad.status);
- try testing.expect(std.mem.containsAtLeast(u8, bad.body, 1, "period must be one of"));
+ try conn.request("GET", "/api/overview?period=12h", null, null);
+ const bad = try conn.receive(&body_buf);
+ try testing.expectEqual(@as(u16, 400), bad.status);
+ try testing.expect(std.mem.containsAtLeast(u8, bad.body, 1, "period must be one of"));
+
+ // Milestone 36 removed the five per-panel endpoints. They are gone from the
+ // table, not merely unreferenced by the admin, so the server refuses them.
+ for ([_][]const u8{
+ "/api/stats",
+ "/api/stats/timeseries",
+ "/api/stats/types",
+ "/api/stats/routes",
+ "/api/stats/clients",
+ }) |gone| {
+ try conn.request("GET", gone, null, null);
+ const missing = try conn.receive(&body_buf);
+ try testing.expectEqual(@as(u16, 404), missing.status);
}
}
@@ -2759,23 +2755,16 @@ fn populatedAggregations(io: std.Io, env: *Env) anyerror!void {
var body_buf: [256 * 1024]u8 = undefined;
- const totals = try getJson(handlers_stats.TotalsBody, arena, &conn, "/api/stats?period=1h", &body_buf);
- const series = try getJson(handlers_stats.TimeseriesBody, arena, &conn, "/api/stats/timeseries?period=1h", &body_buf);
- const types_body = try getJson(handlers_stats.TypesBody, arena, &conn, "/api/stats/types?period=1h", &body_buf);
- const routes_body = try getJson(handlers_stats.RoutesBody, arena, &conn, "/api/stats/routes?period=1h", &body_buf);
- const clients = try getJson(handlers_stats.ClientsBody, arena, &conn, "/api/stats/clients?period=1h", &body_buf);
+ const body = try getJson(handlers_overview.Body, arena, &conn, "/api/overview?period=1h", &body_buf);
- // Nothing writes to this box between the five requests, so the window is
- // one state and conservation is a real assertion rather than a race.
- try testing.expectEqual(totals.since, series.since);
- try testing.expectEqual(totals.since, types_body.since);
- try testing.expectEqual(totals.since, routes_body.since);
- try testing.expectEqual(totals.since, clients.since);
- try testing.expect(totals.queries > 0);
+ // One response over one snapshot, so conservation is a property of the
+ // payload rather than of a quiet box between five requests.
+ try testing.expect(body.totals.queries > 0);
+ const totals = body.totals;
var typed: u64 = 0;
var null_qtype_rows: usize = 0;
- for (types_body.types) |row| {
+ for (body.types) |row| {
typed += row.count;
if (row.qtype == null) null_qtype_rows += 1;
}
@@ -2786,7 +2775,7 @@ fn populatedAggregations(io: std.Io, env: *Env) anyerror!void {
var routed: u64 = 0;
var null_source_upstreams: usize = 0;
var named_upstreams: usize = 0;
- for (routes_body.routes) |row| {
+ for (body.routes) |row| {
routed += row.count;
if (row.route != .upstream) continue;
if (row.source == null) null_source_upstreams += 1 else named_upstreams += 1;
@@ -2795,17 +2784,20 @@ fn populatedAggregations(io: std.Io, env: *Env) anyerror!void {
try testing.expectEqual(@as(usize, 1), null_source_upstreams);
try testing.expectEqual(@as(usize, 2), named_upstreams);
- try testing.expectEqual(@as(usize, recent_clients), clients.clients.len);
- try testing.expectEqual(series.buckets.len, clients.other.len);
- for (clients.clients) |entry| try testing.expectEqual(series.buckets.len, entry.buckets.len);
+ try testing.expectEqual(@as(usize, recent_clients), body.clients.len);
+ try testing.expectEqual(body.buckets.len, body.other.len);
+ for (body.clients) |entry| try testing.expectEqual(body.buckets.len, entry.buckets.len);
// Per bucket, not just over the window: a series off by one bucket would
// still sum correctly in total.
- for (series.buckets, 0..) |bucket, at| {
- var summed: u64 = clients.other[at];
- for (clients.clients) |entry| summed += entry.buckets[at];
+ var bucketed: u64 = 0;
+ for (body.buckets, 0..) |bucket, at| {
+ bucketed += bucket.queries;
+ var summed: u64 = body.other[at];
+ for (body.clients) |entry| summed += entry.buckets[at];
try testing.expectEqual(bucket.queries, summed);
}
+ try testing.expectEqual(totals.queries, bucketed);
}
test "W10 milestone 30: the three breakdowns conserve the totals over one window" {
@@ -2826,11 +2818,8 @@ fn hammerQuerylog(io: std.Io, env: *Env) anyerror!void {
var body_buf: [256 * 1024]u8 = undefined;
const targets = [_][]const u8{
- "/api/stats?period=1h",
- "/api/stats/timeseries?period=1h",
- "/api/stats/types?period=1h",
- "/api/stats/routes?period=1h",
- "/api/stats/clients?period=1h",
+ "/api/overview?period=1h",
+ "/api/overview?period=24h",
"/api/queries?limit=5",
"/api/queries/27",
};
@@ -2875,7 +2864,7 @@ fn failedCommitIsBounded(io: std.Io, env: *Env) anyerror!void {
// the connection recovers (the rollback attempt worked, so the next
// `BEGIN` is not refused).
db.read_tx_faults.failNextCommit();
- try conn.request("GET", "/api/stats/types?period=1h", null, null);
+ try conn.request("GET", "/api/overview?period=1h", null, null);
const failed = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 500), failed.status);
try testing.expect(std.mem.containsAtLeast(u8, failed.body, 1, "internal error"));
@@ -2883,16 +2872,13 @@ fn failedCommitIsBounded(io: std.Io, env: *Env) anyerror!void {
// Same connection, same shared query-log handle: a request after the fault
// is an ordinary 200. This is the assertion the double-unlock bug failed —
// it panicked here instead of answering.
- try conn.request("GET", "/api/stats/types?period=1h", null, null);
+ try conn.request("GET", "/api/overview?period=1h", null, null);
const recovered = try conn.receive(&body_buf);
try testing.expectEqual(@as(u16, 200), recovered.status);
// And every other query-log route still works on that connection.
for ([_][]const u8{
- "/api/stats?period=1h",
- "/api/stats/timeseries?period=1h",
- "/api/stats/routes?period=1h",
- "/api/stats/clients?period=1h",
+ "/api/overview?period=24h",
"/api/queries?limit=5",
"/api/queries/27",
}) |target| {
@@ -3012,7 +2998,7 @@ fn credentialSweep(
// a closed queue is empty, and a zero flush interval makes it commit the
// batch it holds rather than wait for company.
query_logger.shutdown(io);
- try query_logger.runWriter(io, &env.querylog_db, null);
+ try query_logger.runWriter(io, testing.allocator, &env.querylog_db, null);
try testing.expectEqual(@as(u64, 1), query_logger.rows_written.load(.monotonic));
var stmt = try env.querylog_db.prepare(
@@ -3905,15 +3891,13 @@ test "drift guard c: the health rollup matches the five objects it documents" {
try expectSchemaMatches(gpa, handlers_health.Body, "Health");
}
-test "drift guard c: the stats schemas match the structs that serialize them" {
+test "drift guard c: the overview schema matches the struct that serializes it" {
// Guard b counts operations and guard a matches paths, so neither noticed
// that `cached` outlived the field it documented. This one would have.
+ // It recurses, so `Bucket`, `ClientSeries`, `TypeCount` and `RouteCount`
+ // are held to their schemas here too.
const gpa = testing.allocator;
- try expectSchemaMatches(gpa, handlers_stats.TotalsBody, "StatsTotals");
- try expectSchemaMatches(gpa, handlers_stats.TimeseriesBody, "StatsTimeseries");
- try expectSchemaMatches(gpa, handlers_stats.TypesBody, "StatsTypes");
- try expectSchemaMatches(gpa, handlers_stats.RoutesBody, "StatsRoutes");
- try expectSchemaMatches(gpa, handlers_stats.ClientsBody, "StatsClients");
+ try expectSchemaMatches(gpa, handlers_overview.Body, "Overview");
}
test "drift guard c: the query-log schemas match the structs that serialize them" {
@@ -4094,8 +4078,6 @@ const contract_sample_walk = [_]ContractSample{
// a matched pattern, so the golden exercises every nested object rather
// than a row of nulls.
.{ .name = "get_query_detail", .ts_type = "QueryDetail", .method = "GET", .target = "/api/queries/27", .status = 200 },
- .{ .name = "get_stats", .ts_type = "StatsTotals", .method = "GET", .target = "/api/stats?period=1h", .status = 200 },
- .{ .name = "get_stats_timeseries", .ts_type = "StatsTimeseries", .method = "GET", .target = "/api/stats/timeseries?period=1h", .status = 200 },
// Pause: the GET before the POST, so one sample carries `until: null` and
// the other the deadline.
@@ -4122,13 +4104,11 @@ const contract_sample_walk = [_]ContractSample{
.{ .name = "error_not_found", .ts_type = "ErrorEnvelope", .method = "GET", .target = "/api/nope", .status = 404 },
};
-/// The three period aggregations, captured against an environment with live
-/// traffic in it: over the fixed 2023 seed every one of them would answer with
-/// an empty array, which describes no field at all.
+/// The overview, captured against an environment with live traffic in it: over
+/// the fixed 2023 seed its four breakdowns would every one answer with an empty
+/// array, which describes no field at all.
const stats_sample_walk = [_]ContractSample{
- .{ .name = "get_stats_types", .ts_type = "StatsTypes", .method = "GET", .target = "/api/stats/types?period=1h", .status = 200 },
- .{ .name = "get_stats_routes", .ts_type = "StatsRoutes", .method = "GET", .target = "/api/stats/routes?period=1h", .status = 200 },
- .{ .name = "get_stats_clients", .ts_type = "StatsClients", .method = "GET", .target = "/api/stats/clients?period=1h", .status = 200 },
+ .{ .name = "get_overview", .ts_type = "Overview", .method = "GET", .target = "/api/overview?period=1h", .status = 200 },
};
/// A session-authenticated environment answers this without a cookie.