milestone 28: query provenance — every logged query is exactly explainable
Gates / frontend (push) Successful in 1m36s
Gates / test (push) Successful in 1m56s
Gates / test-aarch64 (push) Successful in 7m37s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 13s
CI / gates (push) Successful in 19m4s
Gates / frontend (push) Successful in 1m36s
Gates / test (push) Successful in 1m56s
Gates / test-aarch64 (push) Successful in 7m37s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 13s
CI / gates (push) Successful in 19m4s
query rows gain qclass, rcode, group, policy action and reason, the matched rule or list entry with its source, cname and safe-search targets, route kind, forward zone, and the resolver that actually answered — the pool and local markers die. servfails are logged and name the resolver that lost; post-parse protocol refusals become rows. a detail page at /queries/:id renders the ordered explanation, and coverage watermarks distinguish an empty history from a missing one. the schema fingerprint changes: existing query history is recreated with the old file kept aside and the reset filed as a resolved diagnostic. fixes an oversized udp reply being rebuilt as noerror, which handed clients a truncated nxdomain as success.
This commit is contained in:
@@ -4,6 +4,26 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
|
|||||||
|
|
||||||
Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
|
Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
|
||||||
|
|
||||||
|
## [Unreleased]
|
||||||
|
|
||||||
|
Query provenance: every logged query becomes exactly explainable — what the policy decided, what matched, where the answer came from and what the client saw. The handler records all of it as the reply goes out, `query_log` stores it, and a detail page reads one query back in the order the pipeline decided it. Read the upgrade note below first: it resets your query history.
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Every logged query has a detail page.** A row in the query log now links to `/queries/{id}`, which explains that one query in the order it was decided: the request, the group it was matched under, the policy verdict with the rule that produced it and the blocklist source that rule came from, any CNAME uncloaking or safe-search rewrite, the route the answer took — blocked, local, forward zone, upstream or cache — and what the client got back, RCODE and duration included. `GET /api/queries/{id}` serves the same object; an id that retention has already deleted is a 404. The live view carries the same provenance for the queries it streams, so a query is explainable as it happens as well as afterwards.
|
||||||
|
- **The query log and the dashboard say how far back the history goes.** `GET /api/queries`, `/api/stats` and `/api/stats/timeseries` each carry a `coverage` object: `available_since`, the first second the file can answer for, and `complete`, whether the window you asked for begins inside it. A period that starts before the query log does now says so instead of charting the missing part as zero — which is what a recreate, a retention pass or a fresh install would otherwise look like.
|
||||||
|
|
||||||
|
### Changed
|
||||||
|
|
||||||
|
- **`GET /api/queries` rows changed shape.** Each row gains `qclass`, `rcode`, `policy_action`, `policy_reason` and `route_kind`, and `block_reason` is gone: the reason a query was blocked is now one of a closed set of values rather than a formatted string, and the status column reads it from `policy_reason`. `blocked`, `cache_hit`, `upstream` and every other existing field are unchanged.
|
||||||
|
- **Upgrading resets your query history.** The `query_log` table gains the provenance columns below, and `querylog.db` is never migrated (it holds expendable log rows, so a schema change replaces the file instead of upgrading it). On the first start after the upgrade the old file is set aside as `querylog.db.schema-changed-<unix seconds>` and a fresh one is created. Nothing else is touched: `config.db` keeps your configuration and your diagnostics history. The recreate files a resolved `query_log.recreated` diagnostics entry naming the file that was kept and the timestamp the new history begins at, and a new `querylog_meta` table records that coverage start, so the dashboard can say "history is available from ..." instead of charting an empty range as zero. The set-aside file is a working SQLite database and can be deleted once you have decided you do not want it.
|
||||||
|
- **`logging.query_log_buffer_max` now accepts 1 to 37449, down from 1 to 1000000.** The queued entry carries every new provenance field by value and is about four times as wide as before — 1792 bytes against 432 — so the meaningful bound is bytes rather than entries. The ceiling is computed at compile time from the width of the entry so that the queue's worst case stays within 64 MiB, and it moves whenever that width does. The default of 10000 is unchanged and costs about 17 MiB. A configuration above the new ceiling is rejected at startup with the ceiling in the message.
|
||||||
|
- **Group and blocklist source names are now capped at 64 bytes.** Both are copied into every query-log row that mentions them, so an unbounded name was an unbounded cost per row. A longer name is rejected as `GroupNameTooLong` or `SourceNameTooLong`.
|
||||||
|
|
||||||
|
### Fixed
|
||||||
|
|
||||||
|
- **A UDP reply that has to be truncated keeps the answer's RCODE.** When an answer does not fit the client's UDP buffer, nxdns replaces it with an empty reply carrying the TC bit, which tells the client to retry over TCP. That replacement was always built as NOERROR, whatever the answer said — so an oversized NXDOMAIN reached the client as a success, and an EDNS extended RCODE above 15 lost the eight upper bits it needs an OPT record to carry. The truncated reply now carries the full twelve-bit code the answer had, split across the header and the reply's OPT record where the code needs it, and the query-log row records the code the client actually saw. The retry over TCP always returned the right RCODE; this was the UDP answer that preceded it.
|
||||||
|
|
||||||
## [0.0.8] - 2026-08-21
|
## [0.0.8] - 2026-08-21
|
||||||
|
|
||||||
One constant, chosen from the 0.0.7 field numbers: the checkpoint cadence was the last first-order write cost on the Pi's SD card.
|
One constant, chosen from the 0.0.7 field numbers: the checkpoint cadence was the last first-order write cost on the Pi's SD card.
|
||||||
|
|||||||
@@ -37,17 +37,27 @@ export function useClientNames(): ClientNames {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
export function ClientName({ ip, names }: { ip: string; names: ClientNames }) {
|
/**
|
||||||
|
* The name the loaded list gives this address right now, or null when it gives
|
||||||
|
* none. Callers that must distinguish "named" from "bare address" — rather than
|
||||||
|
* just render whichever applies — read this instead of re-deriving precedence.
|
||||||
|
*/
|
||||||
|
export function clientLabel(ip: string, names: ClientNames): { text: string; learned: boolean } | null {
|
||||||
const client = names.get(ip);
|
const client = names.get(ip);
|
||||||
if (client === undefined || (client.name === "" && client.learned_name === "")) {
|
if (client === undefined) return null;
|
||||||
return <span {...stylex.props(shared.mono)}>{ip}</span>;
|
if (client.name !== "") return { text: client.name, learned: false };
|
||||||
}
|
if (client.learned_name !== "") return { text: client.learned_name, learned: true };
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function ClientName({ ip, names }: { ip: string; names: ClientNames }) {
|
||||||
|
const label = clientLabel(ip, names);
|
||||||
|
if (label === null) return <span {...stylex.props(shared.mono)}>{ip}</span>;
|
||||||
// The name replaces the address on screen, so the address stays reachable
|
// The name replaces the address on screen, so the address stays reachable
|
||||||
// as the tooltip rather than disappearing from the row entirely.
|
// as the tooltip rather than disappearing from the row entirely.
|
||||||
if (client.name !== "") return <span title={ip}>{client.name}</span>;
|
|
||||||
return (
|
return (
|
||||||
<span title={ip} {...stylex.props(shared.learnedName)}>
|
<span title={ip} {...stylex.props(label.learned && shared.learnedName)}>
|
||||||
{client.learned_name}
|
{label.text}
|
||||||
</span>
|
</span>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -18,6 +18,7 @@ const RESPONSES: Record<string, unknown> = {
|
|||||||
cached: 100,
|
cached: 100,
|
||||||
clients: 7,
|
clients: 7,
|
||||||
avg_response_time_us: 2345,
|
avg_response_time_us: 2345,
|
||||||
|
coverage: { complete: true, available_since: 0 },
|
||||||
},
|
},
|
||||||
"/api/stats/timeseries?period=24h": {
|
"/api/stats/timeseries?period=24h": {
|
||||||
period: "24h",
|
period: "24h",
|
||||||
@@ -29,6 +30,7 @@ const RESPONSES: Record<string, unknown> = {
|
|||||||
{ ts: 1800, queries: 40, blocked: 0, cached: 0 },
|
{ ts: 1800, queries: 40, blocked: 0, cached: 0 },
|
||||||
{ ts: 3600, queries: 0, blocked: 0, cached: 0 },
|
{ ts: 3600, queries: 0, blocked: 0, cached: 0 },
|
||||||
],
|
],
|
||||||
|
coverage: { complete: true, available_since: 0 },
|
||||||
},
|
},
|
||||||
"/api/stats?period=1h": {
|
"/api/stats?period=1h": {
|
||||||
period: "1h",
|
period: "1h",
|
||||||
@@ -39,6 +41,7 @@ const RESPONSES: Record<string, unknown> = {
|
|||||||
cached: 0,
|
cached: 0,
|
||||||
clients: 2,
|
clients: 2,
|
||||||
avg_response_time_us: null,
|
avg_response_time_us: null,
|
||||||
|
coverage: { complete: true, available_since: 0 },
|
||||||
},
|
},
|
||||||
"/api/stats/timeseries?period=1h": {
|
"/api/stats/timeseries?period=1h": {
|
||||||
period: "1h",
|
period: "1h",
|
||||||
@@ -46,6 +49,7 @@ const RESPONSES: Record<string, unknown> = {
|
|||||||
until: 3600,
|
until: 3600,
|
||||||
bucket_seconds: 60,
|
bucket_seconds: 60,
|
||||||
buckets: [],
|
buckets: [],
|
||||||
|
coverage: { complete: true, available_since: 0 },
|
||||||
},
|
},
|
||||||
"/api/health": {
|
"/api/health": {
|
||||||
status: "degraded",
|
status: "degraded",
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ import { keepPreviousData, useQuery } from "@tanstack/react-query";
|
|||||||
import * as stylex from "@stylexjs/stylex";
|
import * as stylex from "@stylexjs/stylex";
|
||||||
import { healthQuery, statsQuery, timeseriesQuery, upstreamHealthQuery } from "@/lib/queries";
|
import { healthQuery, statsQuery, timeseriesQuery, upstreamHealthQuery } from "@/lib/queries";
|
||||||
import type { Period } from "@/lib/types";
|
import type { Period } from "@/lib/types";
|
||||||
|
import CoverageNotice from "@/lib/CoverageNotice";
|
||||||
import InlineError from "@/lib/InlineError";
|
import InlineError from "@/lib/InlineError";
|
||||||
import { styles as shared } from "@/ui/styles";
|
import { styles as shared } from "@/ui/styles";
|
||||||
import { colors } from "@/ui/tokens.stylex";
|
import { colors } from "@/ui/tokens.stylex";
|
||||||
@@ -142,6 +143,10 @@ export default function DashboardPage() {
|
|||||||
<StatCards stats={stats.data} />
|
<StatCards stats={stats.data} />
|
||||||
)}
|
)}
|
||||||
|
|
||||||
|
{/* One notice for the period: the chart is judged against the same
|
||||||
|
aligned lower bound as the totals, so it would say the same thing. */}
|
||||||
|
{stats.data !== undefined && <CoverageNotice coverage={stats.data.coverage} />}
|
||||||
|
|
||||||
<div {...stylex.props(styles.panelGrid)}>
|
<div {...stylex.props(styles.panelGrid)}>
|
||||||
<section {...stylex.props(styles.panel)}>
|
<section {...stylex.props(styles.panel)}>
|
||||||
<h2 {...stylex.props(styles.panelHeading)}>Queries over time</h2>
|
<h2 {...stylex.props(styles.panelHeading)}>Queries over time</h2>
|
||||||
|
|||||||
@@ -12,6 +12,7 @@ function timeseries(bucketCount: number): StatsTimeseries {
|
|||||||
since: SINCE,
|
since: SINCE,
|
||||||
until: SINCE + bucketCount * 1800,
|
until: SINCE + bucketCount * 1800,
|
||||||
bucket_seconds: 1800,
|
bucket_seconds: 1800,
|
||||||
|
coverage: { complete: true, available_since: SINCE },
|
||||||
buckets: Array.from({ length: bucketCount }, (_, i) => ({
|
buckets: Array.from({ length: bucketCount }, (_, i) => ({
|
||||||
ts: SINCE + i * 1800,
|
ts: SINCE + i * 1800,
|
||||||
queries: i + 1,
|
queries: i + 1,
|
||||||
|
|||||||
@@ -1,7 +1,11 @@
|
|||||||
import { act, fireEvent, render, screen, within } from "@testing-library/react";
|
import { act, fireEvent, render, screen, within } from "@testing-library/react";
|
||||||
import { QueryClientProvider } from "@tanstack/react-query";
|
import { QueryClientProvider } from "@tanstack/react-query";
|
||||||
|
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
|
||||||
|
import { AuthProvider } from "@/auth/store";
|
||||||
import { createQueryClient } from "@/lib/queryClient";
|
import { createQueryClient } from "@/lib/queryClient";
|
||||||
import type { Client, LiveQueryEvent } from "@/lib/types";
|
import { createAppRouter } from "@/routes";
|
||||||
|
import type { Client } from "@/lib/types";
|
||||||
|
import { provenance, queryRow } from "@/features/queries/provenanceFixture";
|
||||||
import { FakeEventSource } from "./fakeEventSource";
|
import { FakeEventSource } from "./fakeEventSource";
|
||||||
import LiveLogPage from "./LiveLogPage";
|
import LiveLogPage from "./LiveLogPage";
|
||||||
|
|
||||||
@@ -44,19 +48,16 @@ afterEach(() => {
|
|||||||
vi.unstubAllGlobals();
|
vi.unstubAllGlobals();
|
||||||
});
|
});
|
||||||
|
|
||||||
function frame(ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): { data: string } {
|
function json(payload: unknown): Response {
|
||||||
const payload: LiveQueryEvent = {
|
return new Response(JSON.stringify(payload), { status: 200, headers: { "content-type": "application/json" } });
|
||||||
ts,
|
}
|
||||||
domain,
|
|
||||||
client_ip: "192.0.2.10",
|
function frame(ts: number, domain: string, sections: Parameters<typeof provenance>[0] = {}): { data: string } {
|
||||||
qtype: 1,
|
const payload = provenance({
|
||||||
blocked: false,
|
...sections,
|
||||||
block_reason: "",
|
request: { time: ts, domain, ...sections.request },
|
||||||
response_time_us: 500,
|
route: { kind: "cache", upstream: "", ...sections.route },
|
||||||
cache_hit: true,
|
});
|
||||||
upstream: "",
|
|
||||||
...overrides,
|
|
||||||
};
|
|
||||||
return { data: JSON.stringify(payload) };
|
return { data: JSON.stringify(payload) };
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -87,13 +88,17 @@ test("streams rows, flags blocked ones, and freezes the display", () => {
|
|||||||
sources[0]!.emit("query", frame(1000, "ok.example"));
|
sources[0]!.emit("query", frame(1000, "ok.example"));
|
||||||
sources[0]!.emit(
|
sources[0]!.emit(
|
||||||
"query",
|
"query",
|
||||||
frame(1001, "ads.example", { blocked: true, block_reason: "blocklist:stevenblack", qtype: 28 }),
|
frame(1001, "ads.example", {
|
||||||
|
request: { qtype: 28 },
|
||||||
|
policy: { action: "block", reason: "blocklist_wildcard" },
|
||||||
|
route: { kind: "blocked" },
|
||||||
|
}),
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
|
|
||||||
expect(screen.getByText("ok.example")).toBeTruthy();
|
expect(screen.getByText("ok.example")).toBeTruthy();
|
||||||
expect(screen.getByText("Blocked")).toBeTruthy();
|
expect(screen.getByText("Blocked")).toBeTruthy();
|
||||||
expect(screen.getByText("blocklist:stevenblack")).toBeTruthy();
|
expect(screen.getByText("Blocklist (wildcard)")).toBeTruthy();
|
||||||
expect(screen.getByText("AAAA")).toBeTruthy();
|
expect(screen.getByText("AAAA")).toBeTruthy();
|
||||||
// StyleX compiles to opaque class names, so the check is structural: a blocked
|
// StyleX compiles to opaque class names, so the check is structural: a blocked
|
||||||
// row carries every class a plain row does, plus the ones the flag adds.
|
// row carries every class a plain row does, plus the ones the flag adds.
|
||||||
@@ -120,10 +125,10 @@ test("resolves each row's client to its display name, keeping the IP as the tool
|
|||||||
const sources = renderPage();
|
const sources = renderPage();
|
||||||
act(() => sources[0]!.emit("open"));
|
act(() => sources[0]!.emit("open"));
|
||||||
act(() => {
|
act(() => {
|
||||||
sources[0]!.emit("query", frame(1000, "named.example", { client_ip: "192.0.2.10" }));
|
sources[0]!.emit("query", frame(1000, "named.example", { request: { client: "192.0.2.10" } }));
|
||||||
sources[0]!.emit("query", frame(1001, "learned.example", { client_ip: "192.0.2.11" }));
|
sources[0]!.emit("query", frame(1001, "learned.example", { request: { client: "192.0.2.11" } }));
|
||||||
sources[0]!.emit("query", frame(1002, "nameless.example", { client_ip: "192.0.2.12" }));
|
sources[0]!.emit("query", frame(1002, "nameless.example", { request: { client: "192.0.2.12" } }));
|
||||||
sources[0]!.emit("query", frame(1003, "stranger.example", { client_ip: "192.0.2.99" }));
|
sources[0]!.emit("query", frame(1003, "stranger.example", { request: { client: "192.0.2.99" } }));
|
||||||
});
|
});
|
||||||
|
|
||||||
// A hand-typed name wins outright; the learned name never surfaces for it.
|
// A hand-typed name wins outright; the learned name never surfaces for it.
|
||||||
@@ -170,7 +175,7 @@ test("rows stream in as bare IPs while the client list is still loading", async
|
|||||||
|
|
||||||
const sources = renderPage();
|
const sources = renderPage();
|
||||||
act(() => sources[0]!.emit("open"));
|
act(() => sources[0]!.emit("open"));
|
||||||
act(() => sources[0]!.emit("query", frame(1000, "named.example", { client_ip: "192.0.2.10" })));
|
act(() => sources[0]!.emit("query", frame(1000, "named.example", { request: { client: "192.0.2.10" } })));
|
||||||
|
|
||||||
expect(screen.getByText("192.0.2.10")).toBeTruthy();
|
expect(screen.getByText("192.0.2.10")).toBeTruthy();
|
||||||
expect(screen.queryByText("Kitchen Pi")).toBeNull();
|
expect(screen.queryByText("Kitchen Pi")).toBeNull();
|
||||||
@@ -192,3 +197,57 @@ test("repeated connection failures show the viewer-cap state with a retry button
|
|||||||
expect(sources).toHaveLength(2);
|
expect(sources).toHaveLength(2);
|
||||||
expect(screen.getByText("Connecting…")).toBeTruthy();
|
expect(screen.getByText("Connecting…")).toBeTruthy();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("a row recovered by the reconnect fetch links to its stored detail; a streamed one cannot", async () => {
|
||||||
|
const sources: FakeEventSource[] = [];
|
||||||
|
vi.stubGlobal(
|
||||||
|
"EventSource",
|
||||||
|
class {
|
||||||
|
constructor(url: string) {
|
||||||
|
const es = new FakeEventSource(url);
|
||||||
|
sources.push(es);
|
||||||
|
return es as unknown as EventSource;
|
||||||
|
}
|
||||||
|
},
|
||||||
|
);
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn(async (input: RequestInfo | URL) => {
|
||||||
|
const url = String(input);
|
||||||
|
if (url === "/api/clients") return json({ clients: CLIENTS });
|
||||||
|
if (url.startsWith("/api/queries?")) {
|
||||||
|
return json({
|
||||||
|
queries: [queryRow(88, { ts: 1001, domain: "recovered.example" })],
|
||||||
|
next_before: null,
|
||||||
|
coverage: { complete: true, available_since: 0 },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return json({ version: "0.0.0-test", git_commit: "0", zig_version: "0.16.0", uptime_seconds: 1 });
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
// The route, not the bare page: only a recovered row renders a link, so the
|
||||||
|
// test needs the router the link resolves against.
|
||||||
|
const queryClient = createQueryClient();
|
||||||
|
const router = createAppRouter(createMemoryHistory({ initialEntries: ["/live"] }), queryClient);
|
||||||
|
render(
|
||||||
|
<AuthProvider>
|
||||||
|
<QueryClientProvider client={queryClient}>
|
||||||
|
<RouterProvider router={router} />
|
||||||
|
</QueryClientProvider>
|
||||||
|
</AuthProvider>,
|
||||||
|
);
|
||||||
|
|
||||||
|
await screen.findByRole("button", { name: "Freeze" });
|
||||||
|
const source = sources[0]!;
|
||||||
|
act(() => source.emit("open"));
|
||||||
|
act(() => source.emit("query", frame(1000, "streamed.example")));
|
||||||
|
act(() => source.emit("error"));
|
||||||
|
act(() => source.emit("open"));
|
||||||
|
|
||||||
|
const recovered = await screen.findByRole("link", { name: "recovered.example" });
|
||||||
|
expect(recovered.getAttribute("href")).toBe("/queries/88");
|
||||||
|
// The streamed frame precedes its own insert, so it has no row to link to.
|
||||||
|
expect(screen.queryByRole("link", { name: "streamed.example" })).toBeNull();
|
||||||
|
expect(screen.getByText("streamed.example")).toBeTruthy();
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,7 +1,7 @@
|
|||||||
import * as stylex from "@stylexjs/stylex";
|
import * as stylex from "@stylexjs/stylex";
|
||||||
import { useClientNames } from "@/features/clients/clientNames";
|
import { useClientNames } from "@/features/clients/clientNames";
|
||||||
import { QueryCells, QueryTableHead } from "@/features/queries/QueryLogPage";
|
import { QueryCells, QueryTableHead } from "@/features/queries/QueryLogPage";
|
||||||
import { RING_CAPACITY } from "./ringBuffer";
|
import { RING_CAPACITY, summaryOf } from "./ringBuffer";
|
||||||
import { useLiveQueries, type EventSourceFactory, type StreamStatus } from "./useLiveQueries";
|
import { useLiveQueries, type EventSourceFactory, type StreamStatus } from "./useLiveQueries";
|
||||||
import { styles as shared } from "@/ui/styles";
|
import { styles as shared } from "@/ui/styles";
|
||||||
import { colors } from "@/ui/tokens.stylex";
|
import { colors } from "@/ui/tokens.stylex";
|
||||||
@@ -240,11 +240,17 @@ export default function LiveLogPage({ createEventSource }: { createEventSource?:
|
|||||||
<table {...stylex.props(styles.table)}>
|
<table {...stylex.props(styles.table)}>
|
||||||
<QueryTableHead />
|
<QueryTableHead />
|
||||||
<tbody>
|
<tbody>
|
||||||
{live.rows.map((row) => (
|
{live.rows.map((row) => {
|
||||||
<tr key={row.key} {...stylex.props(styles.row, row.blocked && styles.rowBlocked)}>
|
const summary = summaryOf(row);
|
||||||
<QueryCells row={row} clientNames={clientNames} />
|
return (
|
||||||
</tr>
|
<tr
|
||||||
))}
|
key={row.key}
|
||||||
|
{...stylex.props(styles.row, summary.blocked && styles.rowBlocked)}
|
||||||
|
>
|
||||||
|
<QueryCells row={summary} clientNames={clientNames} />
|
||||||
|
</tr>
|
||||||
|
);
|
||||||
|
})}
|
||||||
</tbody>
|
</tbody>
|
||||||
</table>
|
</table>
|
||||||
</div>
|
</div>
|
||||||
|
|||||||
@@ -1,27 +1,21 @@
|
|||||||
import type { LiveQueryEvent, QueryRow } from "@/lib/types";
|
import type { Provenance, QueryRow } from "@/lib/types";
|
||||||
import { RING_CAPACITY, mergeGap, pushRow, type LiveRow } from "./ringBuffer";
|
import { provenance, queryRow } from "@/features/queries/provenanceFixture";
|
||||||
|
import { RING_CAPACITY, mergeGap, pushRow, summaryOf, type LiveRow } from "./ringBuffer";
|
||||||
|
|
||||||
function event(ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): LiveQueryEvent {
|
function streamed(key: number, ts: number, domain: string, sections: Parameters<typeof provenance>[0] = {}): LiveRow {
|
||||||
return {
|
return {
|
||||||
ts,
|
kind: "streamed",
|
||||||
domain,
|
key,
|
||||||
client_ip: "192.0.2.10",
|
event: provenance({
|
||||||
qtype: 1,
|
...sections,
|
||||||
blocked: false,
|
request: { time: ts, domain, ...sections.request },
|
||||||
block_reason: "",
|
route: { upstream: "udp://9.9.9.9:53", ...sections.route },
|
||||||
response_time_us: 500,
|
}),
|
||||||
cache_hit: false,
|
|
||||||
upstream: "udp://9.9.9.9:53",
|
|
||||||
...overrides,
|
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
function liveRow(key: number, ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): LiveRow {
|
function fetchedRow(id: number, ts: number, domain: string, overrides: Partial<QueryRow> = {}): QueryRow {
|
||||||
return { ...event(ts, domain, overrides), key };
|
return queryRow(id, { ts, domain, upstream: "udp://9.9.9.9:53", ...overrides });
|
||||||
}
|
|
||||||
|
|
||||||
function fetchedRow(id: number, ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): QueryRow {
|
|
||||||
return { id, ...event(ts, domain, overrides) };
|
|
||||||
}
|
}
|
||||||
|
|
||||||
function counter(start = 100): () => number {
|
function counter(start = 100): () => number {
|
||||||
@@ -29,31 +23,103 @@ function counter(start = 100): () => number {
|
|||||||
return () => ++n;
|
return () => ++n;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function domains(rows: LiveRow[]): string[] {
|
||||||
|
return rows.map((row) => summaryOf(row).domain);
|
||||||
|
}
|
||||||
|
|
||||||
|
describe("summaryOf", () => {
|
||||||
|
test("a streamed frame projects every summary field from the provenance it carries", () => {
|
||||||
|
const event: Provenance = provenance({
|
||||||
|
request: { time: 1700, domain: "ads.example", client: "192.0.2.11", qtype: 28 },
|
||||||
|
policy: { action: "block", reason: "blocklist_wildcard" },
|
||||||
|
route: { kind: "blocked", upstream: "" },
|
||||||
|
response: { duration_us: 42 },
|
||||||
|
});
|
||||||
|
expect(summaryOf({ kind: "streamed", key: 1, event })).toEqual({
|
||||||
|
id: null,
|
||||||
|
ts: 1700,
|
||||||
|
domain: "ads.example",
|
||||||
|
client_ip: "192.0.2.11",
|
||||||
|
qtype: 28,
|
||||||
|
blocked: true,
|
||||||
|
policy_reason: "blocklist_wildcard",
|
||||||
|
response_time_us: 42,
|
||||||
|
cache_hit: null,
|
||||||
|
upstream: "",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a recovered row projects its stored fields and keeps its id", () => {
|
||||||
|
const row = queryRow(77, { domain: "news.example", cache_hit: true, policy_reason: "rule_allow_exact" });
|
||||||
|
expect(summaryOf({ kind: "recovered", key: 2, row })).toMatchObject({
|
||||||
|
id: 77,
|
||||||
|
domain: "news.example",
|
||||||
|
cache_hit: true,
|
||||||
|
policy_reason: "rule_allow_exact",
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The guard the discriminated union exists for: a field added to the wire
|
||||||
|
* DTO must be either projected into the summary or consciously left to the
|
||||||
|
* detail page. A silent addition fails here rather than going unrendered.
|
||||||
|
*/
|
||||||
|
test("every provenance field is either projected or knowingly detail-only", () => {
|
||||||
|
const projected = [
|
||||||
|
"request.time",
|
||||||
|
"request.domain",
|
||||||
|
"request.client",
|
||||||
|
"request.qtype",
|
||||||
|
"policy.action",
|
||||||
|
"policy.reason",
|
||||||
|
"route.kind",
|
||||||
|
"route.upstream",
|
||||||
|
"response.duration_us",
|
||||||
|
];
|
||||||
|
const detailOnly = [
|
||||||
|
"request.qclass",
|
||||||
|
"group.id",
|
||||||
|
"group.name",
|
||||||
|
"policy.matched",
|
||||||
|
"policy.source_id",
|
||||||
|
"policy.source_name",
|
||||||
|
"rewrites.cname_target",
|
||||||
|
"rewrites.safe_search_target",
|
||||||
|
"route.forward_zone",
|
||||||
|
"response.rcode",
|
||||||
|
];
|
||||||
|
const leaves = Object.entries(provenance()).flatMap(([section, fields]) =>
|
||||||
|
Object.keys(fields as Record<string, unknown>).map((field) => `${section}.${field}`),
|
||||||
|
);
|
||||||
|
expect(leaves.sort()).toEqual([...projected, ...detailOnly].sort());
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
describe("pushRow", () => {
|
describe("pushRow", () => {
|
||||||
test("prepends newest-first", () => {
|
test("prepends newest-first", () => {
|
||||||
let rows: LiveRow[] = [];
|
let rows: LiveRow[] = [];
|
||||||
rows = pushRow(rows, liveRow(1, 10, "a.example"));
|
rows = pushRow(rows, streamed(1, 10, "a.example"));
|
||||||
rows = pushRow(rows, liveRow(2, 11, "b.example"));
|
rows = pushRow(rows, streamed(2, 11, "b.example"));
|
||||||
expect(rows.map((r) => r.domain)).toEqual(["b.example", "a.example"]);
|
expect(domains(rows)).toEqual(["b.example", "a.example"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("drops the oldest beyond capacity", () => {
|
test("drops the oldest beyond capacity", () => {
|
||||||
let rows: LiveRow[] = [];
|
let rows: LiveRow[] = [];
|
||||||
for (let i = 0; i < 5; i++) rows = pushRow(rows, liveRow(i, i, `d${i}.example`), 3);
|
for (let i = 0; i < 5; i++) rows = pushRow(rows, streamed(i, i, `d${i}.example`), 3);
|
||||||
expect(rows).toHaveLength(3);
|
expect(rows).toHaveLength(3);
|
||||||
expect(rows.map((r) => r.key)).toEqual([4, 3, 2]);
|
expect(rows.map((r) => r.key)).toEqual([4, 3, 2]);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("default capacity is 500", () => {
|
test("default capacity is 500", () => {
|
||||||
let rows: LiveRow[] = [];
|
let rows: LiveRow[] = [];
|
||||||
for (let i = 0; i < RING_CAPACITY + 10; i++) rows = pushRow(rows, liveRow(i, i, "x.example"));
|
for (let i = 0; i < RING_CAPACITY + 10; i++) rows = pushRow(rows, streamed(i, i, "x.example"));
|
||||||
expect(rows).toHaveLength(RING_CAPACITY);
|
expect(rows).toHaveLength(RING_CAPACITY);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
describe("mergeGap", () => {
|
describe("mergeGap", () => {
|
||||||
test("skips rows already in the buffer and counts only new ones", () => {
|
test("skips rows already in the buffer and counts only new ones", () => {
|
||||||
const buffer = [liveRow(2, 100, "seen.example"), liveRow(1, 99, "old.example")];
|
const buffer = [streamed(2, 100, "seen.example"), streamed(1, 99, "old.example")];
|
||||||
const fetched = [
|
const fetched = [
|
||||||
fetchedRow(30, 102, "gap2.example"),
|
fetchedRow(30, 102, "gap2.example"),
|
||||||
fetchedRow(29, 101, "gap1.example"),
|
fetchedRow(29, 101, "gap1.example"),
|
||||||
@@ -61,41 +127,120 @@ describe("mergeGap", () => {
|
|||||||
];
|
];
|
||||||
const { rows, missed } = mergeGap(buffer, fetched, counter());
|
const { rows, missed } = mergeGap(buffer, fetched, counter());
|
||||||
expect(missed).toBe(2);
|
expect(missed).toBe(2);
|
||||||
expect(rows.map((r) => r.domain)).toEqual(["gap2.example", "gap1.example", "seen.example", "old.example"]);
|
expect(domains(rows)).toEqual(["gap2.example", "gap1.example", "seen.example", "old.example"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("no additions returns the buffer unchanged with missed 0", () => {
|
test("no additions returns the buffer unchanged with missed 0", () => {
|
||||||
const buffer = [liveRow(1, 100, "seen.example")];
|
const buffer = [streamed(1, 100, "seen.example")];
|
||||||
const { rows, missed } = mergeGap(buffer, [fetchedRow(5, 100, "seen.example")], counter());
|
const { rows, missed } = mergeGap(buffer, [fetchedRow(5, 100, "seen.example")], counter());
|
||||||
expect(missed).toBe(0);
|
expect(missed).toBe(0);
|
||||||
expect(rows).toBe(buffer);
|
expect(rows).toBe(buffer);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("rows differing only in qtype are not deduplicated", () => {
|
/**
|
||||||
const buffer = [liveRow(1, 100, "dual.example", { qtype: 1 })];
|
* A household repeats itself: one client, one name, three lookups inside the
|
||||||
const fetched = [fetchedRow(5, 100, "dual.example", { qtype: 28 })];
|
* same second. The stream delivered one of them before the connection broke,
|
||||||
const { missed } = mergeGap(buffer, fetched, counter());
|
* so the gap fetch must recover the other two rather than let the one row in
|
||||||
expect(missed).toBe(1);
|
* the buffer stand for all three.
|
||||||
|
*/
|
||||||
|
test("repeated identical queries drop only as many rows as the buffer already holds", () => {
|
||||||
|
const buffer = [streamed(1, 100, "dup.example")];
|
||||||
|
const fetched = [
|
||||||
|
fetchedRow(12, 100, "dup.example"),
|
||||||
|
fetchedRow(11, 100, "dup.example"),
|
||||||
|
fetchedRow(10, 100, "dup.example"),
|
||||||
|
];
|
||||||
|
const { rows, missed } = mergeGap(buffer, fetched, counter());
|
||||||
|
expect(missed).toBe(2);
|
||||||
|
expect(domains(rows)).toEqual(["dup.example", "dup.example", "dup.example"]);
|
||||||
|
const recoveredIds = rows.flatMap((row) => (row.kind === "recovered" ? [row.row.id] : []));
|
||||||
|
expect(new Set(recoveredIds).size).toBe(2);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("assigns fresh keys from the counter and drops the id", () => {
|
test("a gap fetch that repeats the whole buffer adds nothing", () => {
|
||||||
|
const buffer = [streamed(2, 100, "dup.example"), streamed(1, 100, "dup.example")];
|
||||||
|
const fetched = [fetchedRow(12, 100, "dup.example"), fetchedRow(11, 100, "dup.example")];
|
||||||
|
const { rows, missed } = mergeGap(buffer, fetched, counter());
|
||||||
|
expect(missed).toBe(0);
|
||||||
|
expect(rows).toBe(buffer);
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Two queries of the same name from the same client in the same second are
|
||||||
|
* still separate facts when any stored column differs — the record type or
|
||||||
|
* class, the response code, the policy that decided them, how long they took,
|
||||||
|
* the route taken. The gap fetch here returns the differing row *first* and
|
||||||
|
* the one the buffer already holds second, so an identity blind to the column
|
||||||
|
* would let the differing row consume the buffered occurrence: the buffered
|
||||||
|
* query would come back duplicated and the other would vanish, at an
|
||||||
|
* unchanged `missed`. Order is what exposes that — the count alone is 1
|
||||||
|
* either way.
|
||||||
|
*
|
||||||
|
* `blocked` and `cache_hit` have no case of their own: the server derives
|
||||||
|
* them from `policy_action` and `route_kind`, so they cannot differ while
|
||||||
|
* everything else holds, and the two columns they follow are covered here.
|
||||||
|
*/
|
||||||
|
test.each([
|
||||||
|
{ column: "qtype", sections: { request: { qtype: 1 } }, held: { qtype: 1 }, differing: { qtype: 28 } },
|
||||||
|
{ column: "qclass", sections: { request: { qclass: 1 } }, held: { qclass: 1 }, differing: { qclass: 3 } },
|
||||||
|
{ column: "rcode", sections: { response: { rcode: 0 } }, held: { rcode: 0 }, differing: { rcode: 2 } },
|
||||||
|
{
|
||||||
|
column: "response_time_us",
|
||||||
|
sections: { response: { duration_us: 1234 } },
|
||||||
|
held: { response_time_us: 1234 },
|
||||||
|
differing: { response_time_us: 9999 },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
column: "route_kind",
|
||||||
|
sections: { route: { kind: "upstream" } },
|
||||||
|
held: { route_kind: "upstream" },
|
||||||
|
differing: { route_kind: "forward_zone" },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
column: "policy_action",
|
||||||
|
sections: { policy: { action: "allow" } },
|
||||||
|
held: { policy_action: "allow" },
|
||||||
|
differing: { policy_action: "not_evaluated" },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
column: "policy_reason",
|
||||||
|
sections: { policy: { reason: "no_match" } },
|
||||||
|
held: { policy_reason: "no_match" },
|
||||||
|
differing: { policy_reason: "rule_allow_exact" },
|
||||||
|
},
|
||||||
|
] satisfies readonly {
|
||||||
|
column: string;
|
||||||
|
sections: Parameters<typeof provenance>[0];
|
||||||
|
held: Partial<QueryRow>;
|
||||||
|
differing: Partial<QueryRow>;
|
||||||
|
}[])("rows differing only in $column survive the gap merge", ({ sections, held, differing }) => {
|
||||||
|
const buffer = [streamed(1, 100, "dual.example", sections)];
|
||||||
|
const fetched = [fetchedRow(6, 100, "dual.example", differing), fetchedRow(5, 100, "dual.example", held)];
|
||||||
|
const { rows, missed } = mergeGap(buffer, fetched, counter());
|
||||||
|
expect(missed).toBe(1);
|
||||||
|
expect(rows).toEqual([{ kind: "recovered", key: expect.any(Number), row: fetched[0] }, buffer[0]]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("recovered rows keep their id and take a fresh key", () => {
|
||||||
const { rows } = mergeGap([], [fetchedRow(77, 100, "gap.example")], counter(200));
|
const { rows } = mergeGap([], [fetchedRow(77, 100, "gap.example")], counter(200));
|
||||||
expect(rows[0]?.key).toBe(201);
|
const recovered = rows[0];
|
||||||
expect("id" in (rows[0] ?? {})).toBe(false);
|
expect(recovered?.key).toBe(201);
|
||||||
|
expect(recovered?.kind).toBe("recovered");
|
||||||
|
expect(recovered !== undefined && recovered.kind === "recovered" ? recovered.row.id : null).toBe(77);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("result is capped at capacity, keeping the newest", () => {
|
test("result is capped at capacity, keeping the newest", () => {
|
||||||
const buffer = [liveRow(3, 300, "live.example")];
|
const buffer = [streamed(3, 300, "live.example")];
|
||||||
const fetched = [fetchedRow(2, 302, "g2.example"), fetchedRow(1, 301, "g1.example")];
|
const fetched = [fetchedRow(2, 302, "g2.example"), fetchedRow(1, 301, "g1.example")];
|
||||||
const { rows, missed } = mergeGap(buffer, fetched, counter(), 2);
|
const { rows, missed } = mergeGap(buffer, fetched, counter(), 2);
|
||||||
expect(missed).toBe(2);
|
expect(missed).toBe(2);
|
||||||
expect(rows.map((r) => r.domain)).toEqual(["g2.example", "g1.example"]);
|
expect(domains(rows)).toEqual(["g2.example", "g1.example"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("merged rows stay sorted newest-first by ts", () => {
|
test("merged rows stay sorted newest-first by ts", () => {
|
||||||
const buffer = [liveRow(4, 105, "after-reopen.example"), liveRow(3, 100, "before.example")];
|
const buffer = [streamed(4, 105, "after-reopen.example"), streamed(3, 100, "before.example")];
|
||||||
const fetched = [fetchedRow(9, 103, "gap.example")];
|
const fetched = [fetchedRow(9, 103, "gap.example")];
|
||||||
const { rows } = mergeGap(buffer, fetched, counter());
|
const { rows } = mergeGap(buffer, fetched, counter());
|
||||||
expect(rows.map((r) => r.ts)).toEqual([105, 103, 100]);
|
expect(rows.map((row) => summaryOf(row).ts)).toEqual([105, 103, 100]);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,8 +1,23 @@
|
|||||||
import type { LiveQueryEvent, QueryRow } from "@/lib/types";
|
import type { LiveQueryEvent, QueryRow } from "@/lib/types";
|
||||||
|
import { summarizeEvent, summarizeRow, type QuerySummary } from "@/features/queries/querySummary";
|
||||||
|
|
||||||
/** A live stream row; `key` is a client-side monotonic counter (SSE frames carry no id). */
|
/**
|
||||||
export interface LiveRow extends LiveQueryEvent {
|
* A row in the live buffer. `key` is a client-side monotonic counter, because
|
||||||
key: number;
|
* neither arm has a stable identity of its own on arrival.
|
||||||
|
*
|
||||||
|
* The two arms are genuinely different facts, not two encodings of one. A
|
||||||
|
* streamed frame carries the full provenance of a query the server has not
|
||||||
|
* written yet; a row recovered by the reconnect gap-fetch is the stored summary
|
||||||
|
* of a query that *was* written, and cannot fabricate the provenance it never
|
||||||
|
* received. Only the recovered arm has a row id to link to.
|
||||||
|
*/
|
||||||
|
export type LiveRow = { key: number } & (
|
||||||
|
{ kind: "streamed"; event: LiveQueryEvent } | { kind: "recovered"; row: QueryRow }
|
||||||
|
);
|
||||||
|
|
||||||
|
/** The flat cells both arms render, and the shared identity for gap dedupe. */
|
||||||
|
export function summaryOf(row: LiveRow): QuerySummary {
|
||||||
|
return row.kind === "streamed" ? summarizeEvent(row.event) : summarizeRow(row.row);
|
||||||
}
|
}
|
||||||
|
|
||||||
export const RING_CAPACITY = 500;
|
export const RING_CAPACITY = 500;
|
||||||
@@ -13,16 +28,98 @@ export function pushRow(rows: LiveRow[], row: LiveRow, capacity: number = RING_C
|
|||||||
return next.length > capacity ? next.slice(0, capacity) : next;
|
return next.length > capacity ? next.slice(0, capacity) : next;
|
||||||
}
|
}
|
||||||
|
|
||||||
// `since` on GET /api/queries is inclusive, so the re-sync fetch returns the
|
/**
|
||||||
// last-seen row(s) again; live rows have no id, so identity is this tuple.
|
* What the gap merge compares two queries by: the whole stored row bar its id.
|
||||||
function signature(row: LiveQueryEvent): string {
|
*
|
||||||
return `${row.ts}|${row.domain}|${row.client_ip}|${row.qtype ?? -1}|${row.blocked}|${row.upstream}`;
|
* `since` on GET /api/queries is inclusive, so the re-sync fetch returns the
|
||||||
|
* last-seen row(s) again and the merge has to recognise them. The id cannot
|
||||||
|
* serve as the identity — a streamed frame precedes its own insert and has none
|
||||||
|
* — so the comparison is by value, and every stored column has to take part.
|
||||||
|
* Two queries alike in name, client and second but differing in class, rcode,
|
||||||
|
* the policy that decided them or the route taken are separate facts; if they
|
||||||
|
* hashed alike, the fetched row that does *not* match the buffered one would
|
||||||
|
* consume its occurrence, duplicating one query and losing the other.
|
||||||
|
*
|
||||||
|
* `QuerySummary` is the wrong basis for that: it is what the table renders, and
|
||||||
|
* it drops qclass, rcode, policy_action and route_kind. `Omit<QueryRow, "id">`
|
||||||
|
* instead makes the compiler demand a derivation for every stored column, so a
|
||||||
|
* column added to the row cannot quietly fall out of the identity.
|
||||||
|
*/
|
||||||
|
type GapIdentity = Omit<QueryRow, "id">;
|
||||||
|
|
||||||
|
function identityOfRow(row: QueryRow): GapIdentity {
|
||||||
|
return {
|
||||||
|
ts: row.ts,
|
||||||
|
domain: row.domain,
|
||||||
|
client_ip: row.client_ip,
|
||||||
|
qtype: row.qtype,
|
||||||
|
qclass: row.qclass,
|
||||||
|
rcode: row.rcode,
|
||||||
|
blocked: row.blocked,
|
||||||
|
response_time_us: row.response_time_us,
|
||||||
|
cache_hit: row.cache_hit,
|
||||||
|
upstream: row.upstream,
|
||||||
|
policy_action: row.policy_action,
|
||||||
|
policy_reason: row.policy_reason,
|
||||||
|
route_kind: row.route_kind,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The same identity out of a live frame, which carries every stored column in
|
||||||
|
* its provenance. The columns the server derives rather than sends — `blocked`
|
||||||
|
* and `cache_hit` — come through `summarizeEvent` so that derivation keeps
|
||||||
|
* living in exactly one place.
|
||||||
|
*/
|
||||||
|
function identityOfEvent(event: LiveQueryEvent): GapIdentity {
|
||||||
|
const summary = summarizeEvent(event);
|
||||||
|
return {
|
||||||
|
ts: summary.ts,
|
||||||
|
domain: summary.domain,
|
||||||
|
client_ip: summary.client_ip,
|
||||||
|
qtype: summary.qtype,
|
||||||
|
qclass: event.request.qclass,
|
||||||
|
rcode: event.response.rcode,
|
||||||
|
blocked: summary.blocked,
|
||||||
|
response_time_us: summary.response_time_us,
|
||||||
|
cache_hit: summary.cache_hit,
|
||||||
|
upstream: summary.upstream,
|
||||||
|
policy_action: event.policy.action,
|
||||||
|
policy_reason: summary.policy_reason,
|
||||||
|
route_kind: event.route.kind,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
function identityOf(row: LiveRow): GapIdentity {
|
||||||
|
return row.kind === "streamed" ? identityOfEvent(row.event) : identityOfRow(row.row);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Sorted keys so the hash cannot depend on the order the two arms happen to build their literals in. */
|
||||||
|
function signature(identity: GapIdentity): string {
|
||||||
|
return JSON.stringify(identity, Object.keys(identity).sort());
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How many times each signature is already in the buffer. A signature is not
|
||||||
|
* unique: one client asking for one name twice within the same second is an
|
||||||
|
* ordinary household pattern, and the two queries are separate facts. Counting
|
||||||
|
* the occurrences lets the merge drop exactly as many fetched rows as the
|
||||||
|
* buffer already holds, instead of letting one buffered row hide all of them.
|
||||||
|
*/
|
||||||
|
function occurrences(rows: LiveRow[]): Map<string, number> {
|
||||||
|
const counts = new Map<string, number>();
|
||||||
|
for (const row of rows) {
|
||||||
|
const key = signature(identityOf(row));
|
||||||
|
counts.set(key, (counts.get(key) ?? 0) + 1);
|
||||||
|
}
|
||||||
|
return counts;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Merge rows fetched for a reconnect gap (newest-first, from GET /api/queries)
|
* Merge rows fetched for a reconnect gap (newest-first, from GET /api/queries)
|
||||||
* into the buffer. Rows already present are skipped; `missed` counts what was
|
* into the buffer. Each fetched row consumes one buffered occurrence of its
|
||||||
* actually added. The result stays newest-first (stable sort by ts) and capped.
|
* signature and is skipped; the rest are genuinely missed and `missed` counts
|
||||||
|
* them. The result stays newest-first (stable sort by ts) and capped.
|
||||||
*/
|
*/
|
||||||
export function mergeGap(
|
export function mergeGap(
|
||||||
rows: LiveRow[],
|
rows: LiveRow[],
|
||||||
@@ -30,24 +127,18 @@ export function mergeGap(
|
|||||||
nextKey: () => number,
|
nextKey: () => number,
|
||||||
capacity: number = RING_CAPACITY,
|
capacity: number = RING_CAPACITY,
|
||||||
): { rows: LiveRow[]; missed: number } {
|
): { rows: LiveRow[]; missed: number } {
|
||||||
const seen = new Set(rows.map(signature));
|
const buffered = occurrences(rows);
|
||||||
const added: LiveRow[] = [];
|
const added: LiveRow[] = [];
|
||||||
for (const row of fetched) {
|
for (const row of fetched) {
|
||||||
const event: LiveQueryEvent = {
|
const key = signature(identityOfRow(row));
|
||||||
ts: row.ts,
|
const count = buffered.get(key) ?? 0;
|
||||||
domain: row.domain,
|
if (count > 0) {
|
||||||
client_ip: row.client_ip,
|
buffered.set(key, count - 1);
|
||||||
qtype: row.qtype,
|
continue;
|
||||||
blocked: row.blocked,
|
}
|
||||||
block_reason: row.block_reason,
|
added.push({ kind: "recovered", row, key: nextKey() });
|
||||||
response_time_us: row.response_time_us,
|
|
||||||
cache_hit: row.cache_hit,
|
|
||||||
upstream: row.upstream,
|
|
||||||
};
|
|
||||||
if (seen.has(signature(event))) continue;
|
|
||||||
added.push({ ...event, key: nextKey() });
|
|
||||||
}
|
}
|
||||||
if (added.length === 0) return { rows, missed: 0 };
|
if (added.length === 0) return { rows, missed: 0 };
|
||||||
const merged = [...added, ...rows].sort((a, b) => b.ts - a.ts).slice(0, capacity);
|
const merged = [...added, ...rows].sort((a, b) => summaryOf(b).ts - summaryOf(a).ts).slice(0, capacity);
|
||||||
return { rows: merged, missed: added.length };
|
return { rows: merged, missed: added.length };
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,8 @@
|
|||||||
import { act, renderHook, waitFor } from "@testing-library/react";
|
import { act, renderHook, waitFor } from "@testing-library/react";
|
||||||
import { ApiError } from "@/lib/api";
|
import { ApiError } from "@/lib/api";
|
||||||
import type { LiveQueryEvent, QueriesPage, QueryRow } from "@/lib/types";
|
import type { QueriesPage, QueryRow } from "@/lib/types";
|
||||||
|
import { provenance, queryRow } from "@/features/queries/provenanceFixture";
|
||||||
|
import { summaryOf, type LiveRow } from "./ringBuffer";
|
||||||
import { FakeEventSource } from "./fakeEventSource";
|
import { FakeEventSource } from "./fakeEventSource";
|
||||||
import { CAP_ERROR_THRESHOLD, useLiveQueries } from "./useLiveQueries";
|
import { CAP_ERROR_THRESHOLD, useLiveQueries } from "./useLiveQueries";
|
||||||
|
|
||||||
@@ -12,37 +14,24 @@ function stubLocationAssign() {
|
|||||||
return assign;
|
return assign;
|
||||||
}
|
}
|
||||||
|
|
||||||
function frame(ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): { data: string } {
|
function frame(ts: number, domain: string): { data: string } {
|
||||||
const payload: LiveQueryEvent = {
|
const payload = provenance({
|
||||||
ts,
|
request: { time: ts, domain },
|
||||||
domain,
|
route: { upstream: "udp://9.9.9.9:53" },
|
||||||
client_ip: "192.0.2.10",
|
});
|
||||||
qtype: 1,
|
|
||||||
blocked: false,
|
|
||||||
block_reason: "",
|
|
||||||
response_time_us: 500,
|
|
||||||
cache_hit: false,
|
|
||||||
upstream: "udp://9.9.9.9:53",
|
|
||||||
...overrides,
|
|
||||||
};
|
|
||||||
return { data: JSON.stringify(payload) };
|
return { data: JSON.stringify(payload) };
|
||||||
}
|
}
|
||||||
|
|
||||||
function fetchedRow(id: number, ts: number, domain: string): QueryRow {
|
function fetchedRow(id: number, ts: number, domain: string): QueryRow {
|
||||||
return {
|
return queryRow(id, { ts, domain, upstream: "udp://9.9.9.9:53" });
|
||||||
id,
|
|
||||||
ts,
|
|
||||||
domain,
|
|
||||||
client_ip: "192.0.2.10",
|
|
||||||
qtype: 1,
|
|
||||||
blocked: false,
|
|
||||||
block_reason: "",
|
|
||||||
response_time_us: 500,
|
|
||||||
cache_hit: false,
|
|
||||||
upstream: "udp://9.9.9.9:53",
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function domains(rows: LiveRow[]): string[] {
|
||||||
|
return rows.map((row) => summaryOf(row).domain);
|
||||||
|
}
|
||||||
|
|
||||||
|
const FULL_COVERAGE = { complete: true, available_since: 0 };
|
||||||
|
|
||||||
function setup(fetchSince?: (since: number) => Promise<QueriesPage>, probeSession?: () => Promise<unknown>) {
|
function setup(fetchSince?: (since: number) => Promise<QueriesPage>, probeSession?: () => Promise<unknown>) {
|
||||||
const sources: FakeEventSource[] = [];
|
const sources: FakeEventSource[] = [];
|
||||||
const createEventSource = (url: string) => {
|
const createEventSource = (url: string) => {
|
||||||
@@ -68,7 +57,7 @@ test("open then frames: rows newest-first with increasing keys", () => {
|
|||||||
sources[0]!.emit("query", frame(1001, "b.example"));
|
sources[0]!.emit("query", frame(1001, "b.example"));
|
||||||
});
|
});
|
||||||
const rows = hook.result.current.rows;
|
const rows = hook.result.current.rows;
|
||||||
expect(rows.map((r) => r.domain)).toEqual(["b.example", "a.example"]);
|
expect(domains(rows)).toEqual(["b.example", "a.example"]);
|
||||||
expect(rows[0]!.key).toBeGreaterThan(rows[1]!.key);
|
expect(rows[0]!.key).toBeGreaterThan(rows[1]!.key);
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -87,6 +76,7 @@ test("error then reopen re-syncs the gap since the last seen ts", async () => {
|
|||||||
return Promise.resolve({
|
return Promise.resolve({
|
||||||
queries: [fetchedRow(9, 1002, "gap.example"), fetchedRow(8, since, "a.example")],
|
queries: [fetchedRow(9, 1002, "gap.example"), fetchedRow(8, since, "a.example")],
|
||||||
next_before: null,
|
next_before: null,
|
||||||
|
coverage: FULL_COVERAGE,
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
const { sources, hook } = setup(fetchSince);
|
const { sources, hook } = setup(fetchSince);
|
||||||
@@ -103,7 +93,7 @@ test("error then reopen re-syncs the gap since the last seen ts", async () => {
|
|||||||
expect(fetchSince).toHaveBeenCalledWith(1000);
|
expect(fetchSince).toHaveBeenCalledWith(1000);
|
||||||
|
|
||||||
await waitFor(() => expect(hook.result.current.missed).toBe(1));
|
await waitFor(() => expect(hook.result.current.missed).toBe(1));
|
||||||
expect(hook.result.current.rows.map((r) => r.domain)).toEqual(["gap.example", "a.example"]);
|
expect(domains(hook.result.current.rows)).toEqual(["gap.example", "a.example"]);
|
||||||
|
|
||||||
act(() => hook.result.current.dismissMissed());
|
act(() => hook.result.current.dismissMissed());
|
||||||
expect(hook.result.current.missed).toBeNull();
|
expect(hook.result.current.missed).toBeNull();
|
||||||
@@ -235,12 +225,12 @@ test("freeze keeps the display fixed while the buffer keeps filling", () => {
|
|||||||
sources[0]!.emit("query", frame(1001, "b.example"));
|
sources[0]!.emit("query", frame(1001, "b.example"));
|
||||||
sources[0]!.emit("query", frame(1002, "c.example"));
|
sources[0]!.emit("query", frame(1002, "c.example"));
|
||||||
});
|
});
|
||||||
expect(hook.result.current.rows.map((r) => r.domain)).toEqual(["a.example"]);
|
expect(domains(hook.result.current.rows)).toEqual(["a.example"]);
|
||||||
expect(hook.result.current.liveCount).toBe(3);
|
expect(hook.result.current.liveCount).toBe(3);
|
||||||
|
|
||||||
act(() => hook.result.current.toggleFreeze());
|
act(() => hook.result.current.toggleFreeze());
|
||||||
expect(hook.result.current.frozen).toBe(false);
|
expect(hook.result.current.frozen).toBe(false);
|
||||||
expect(hook.result.current.rows.map((r) => r.domain)).toEqual(["c.example", "b.example", "a.example"]);
|
expect(domains(hook.result.current.rows)).toEqual(["c.example", "b.example", "a.example"]);
|
||||||
});
|
});
|
||||||
|
|
||||||
test("stale sources are ignored after retry and closed on unmount", () => {
|
test("stale sources are ignored after retry and closed on unmount", () => {
|
||||||
|
|||||||
@@ -121,8 +121,12 @@ export function useLiveQueries(options?: LiveQueriesOptions): LiveQueries {
|
|||||||
} catch {
|
} catch {
|
||||||
return;
|
return;
|
||||||
}
|
}
|
||||||
lastSeenTsRef.current = payload.ts;
|
lastSeenTsRef.current = payload.request.time;
|
||||||
bufferRef.current = pushRow(bufferRef.current, { ...payload, key: ++keyRef.current });
|
bufferRef.current = pushRow(bufferRef.current, {
|
||||||
|
kind: "streamed",
|
||||||
|
event: payload,
|
||||||
|
key: ++keyRef.current,
|
||||||
|
});
|
||||||
setRows(bufferRef.current);
|
setRows(bufferRef.current);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
|||||||
@@ -1,5 +1,6 @@
|
|||||||
import { useState, type FormEvent, type ReactNode } from "react";
|
import { useState, type FormEvent, type ReactNode } from "react";
|
||||||
import { useQuery, useSuspenseQuery } from "@tanstack/react-query";
|
import { useQuery, useSuspenseQuery } from "@tanstack/react-query";
|
||||||
|
import { useSearch } from "@tanstack/react-router";
|
||||||
import * as stylex from "@stylexjs/stylex";
|
import * as stylex from "@stylexjs/stylex";
|
||||||
import { ApiError } from "@/lib/api";
|
import { ApiError } from "@/lib/api";
|
||||||
import { groupsQuery, lookupQuery } from "@/lib/queries";
|
import { groupsQuery, lookupQuery } from "@/lib/queries";
|
||||||
@@ -253,9 +254,14 @@ export default function LookupPage() {
|
|||||||
const groups = useSuspenseQuery(groupsQuery()).data;
|
const groups = useSuspenseQuery(groupsQuery()).data;
|
||||||
const preselectedGroupId = defaultGroupId(groups);
|
const preselectedGroupId = defaultGroupId(groups);
|
||||||
|
|
||||||
const [domain, setDomain] = useState("");
|
// A `?domain=` link (from a query's detail page) arrives already asking the
|
||||||
|
// question, so it runs the lookup rather than leaving a filled-in form.
|
||||||
|
const search = useSearch({ from: "/shell/lookup" });
|
||||||
|
const [domain, setDomain] = useState(search.domain ?? "");
|
||||||
const [groupId, setGroupId] = useState(preselectedGroupId);
|
const [groupId, setGroupId] = useState(preselectedGroupId);
|
||||||
const [submitted, setSubmitted] = useState<Submitted | null>(null);
|
const [submitted, setSubmitted] = useState<Submitted | null>(
|
||||||
|
search.domain === undefined ? null : { domain: search.domain, groupId: preselectedGroupId },
|
||||||
|
);
|
||||||
|
|
||||||
const lookup = useQuery({
|
const lookup = useQuery({
|
||||||
...lookupQuery(submitted?.domain ?? "", submitted?.groupId),
|
...lookupQuery(submitted?.domain ?? "", submitted?.groupId),
|
||||||
|
|||||||
@@ -0,0 +1,247 @@
|
|||||||
|
import { render, screen, waitFor, within } from "@testing-library/react";
|
||||||
|
import { QueryClientProvider } from "@tanstack/react-query";
|
||||||
|
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
|
||||||
|
import { AuthProvider } from "@/auth/store";
|
||||||
|
import { createQueryClient } from "@/lib/queryClient";
|
||||||
|
import { createAppRouter } from "@/routes";
|
||||||
|
import type { QueryDetail } from "@/lib/types";
|
||||||
|
import { provenance } from "./provenanceFixture";
|
||||||
|
|
||||||
|
function detail(id: number, sections: Parameters<typeof provenance>[0] = {}): QueryDetail {
|
||||||
|
return { id, ...provenance(sections) };
|
||||||
|
}
|
||||||
|
|
||||||
|
let responses: Record<string, unknown>;
|
||||||
|
|
||||||
|
beforeEach(() => {
|
||||||
|
responses = {
|
||||||
|
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
|
||||||
|
};
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn(async (input: RequestInfo | URL) => {
|
||||||
|
const payload = responses[String(input)];
|
||||||
|
if (payload === undefined) {
|
||||||
|
return new Response(JSON.stringify({ error: "no such query" }), {
|
||||||
|
status: 404,
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return new Response(JSON.stringify(payload), {
|
||||||
|
status: 200,
|
||||||
|
headers: { "content-type": "application/json" },
|
||||||
|
});
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
afterEach(() => vi.unstubAllGlobals());
|
||||||
|
|
||||||
|
function renderDetail(id: number) {
|
||||||
|
const queryClient = createQueryClient();
|
||||||
|
const router = createAppRouter(createMemoryHistory({ initialEntries: [`/queries/${id}`] }), queryClient);
|
||||||
|
render(
|
||||||
|
<AuthProvider>
|
||||||
|
<QueryClientProvider client={queryClient}>
|
||||||
|
<RouterProvider router={router} />
|
||||||
|
</QueryClientProvider>
|
||||||
|
</AuthProvider>,
|
||||||
|
);
|
||||||
|
return router;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The value beside a term, so a section's facts are read as pairs. */
|
||||||
|
function factValue(label: string): string {
|
||||||
|
const term = screen.getByText(label);
|
||||||
|
const value = term.nextElementSibling;
|
||||||
|
return value?.textContent ?? "";
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The line under the domain, which is what a page is read as at a glance. */
|
||||||
|
function subtitle(domain: string): string {
|
||||||
|
const heading = screen.getByRole("heading", { name: domain });
|
||||||
|
return heading.nextElementSibling?.textContent ?? "";
|
||||||
|
}
|
||||||
|
|
||||||
|
test("a blocked query explains itself in the six sections, in pipeline order", async () => {
|
||||||
|
responses["/api/queries/42"] = detail(42, {
|
||||||
|
request: { time: 1_700_000_000, domain: "ads.example", client: "192.0.2.11", qtype: 28, qclass: 1 },
|
||||||
|
group: { id: 3, name: "kids" },
|
||||||
|
policy: {
|
||||||
|
action: "block",
|
||||||
|
reason: "blocklist_wildcard",
|
||||||
|
matched: "||tracker.example^",
|
||||||
|
source_id: 5,
|
||||||
|
source_name: "StevenBlack",
|
||||||
|
},
|
||||||
|
rewrites: { cname_target: "cdn.tracker.example" },
|
||||||
|
route: { kind: "blocked", upstream: "" },
|
||||||
|
response: { rcode: 0, duration_us: 1234 },
|
||||||
|
});
|
||||||
|
renderDetail(42);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "ads.example" });
|
||||||
|
|
||||||
|
const headings = screen.getAllByRole("heading", { level: 2 }).map((node) => node.textContent);
|
||||||
|
expect(headings).toEqual(["Request", "Group", "Policy", "Rewrites", "Route", "Response", "Related"]);
|
||||||
|
|
||||||
|
expect(factValue("Client")).toBe("192.0.2.11");
|
||||||
|
expect(factValue("Type")).toBe("AAAA");
|
||||||
|
expect(factValue("Class")).toBe("IN (1)");
|
||||||
|
expect(factValue("Name")).toBe("kids");
|
||||||
|
expect(factValue("Id")).toBe("3");
|
||||||
|
expect(factValue("Decision")).toBe("Blocked");
|
||||||
|
expect(factValue("Reason")).toBe("Blocklist (wildcard)");
|
||||||
|
expect(factValue("Matched")).toBe("||tracker.example^");
|
||||||
|
expect(factValue("Blocklist")).toBe("StevenBlack (#5)");
|
||||||
|
expect(factValue("CNAME target")).toBe("cdn.tracker.example");
|
||||||
|
expect(factValue("Answered by")).toBe("Blocked locally");
|
||||||
|
expect(factValue("Upstream")).toBe("No upstream exchange");
|
||||||
|
expect(factValue("Result")).toBe("NOERROR (0)");
|
||||||
|
expect(factValue("Took")).toBe("1.2 ms");
|
||||||
|
// NOERROR is the ordinary case and adds nothing to the verdict.
|
||||||
|
expect(subtitle("ads.example")).toMatch(/ — Blocked$/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an upstream SERVFAIL names the resolver that failed and the code the client saw", async () => {
|
||||||
|
responses["/api/queries/7"] = detail(7, {
|
||||||
|
request: { domain: "news.example" },
|
||||||
|
route: { kind: "upstream", upstream: "https://dns.example/dns-query" },
|
||||||
|
response: { rcode: 2, duration_us: null },
|
||||||
|
});
|
||||||
|
renderDetail(7);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "news.example" });
|
||||||
|
expect(factValue("Answered by")).toBe("Upstream resolver");
|
||||||
|
expect(factValue("Upstream")).toBe("https://dns.example/dns-query");
|
||||||
|
expect(factValue("Result")).toBe("SERVFAIL (2)");
|
||||||
|
expect(factValue("Took")).toBe("Not measured");
|
||||||
|
// The policy allowed the query; the client still got nothing, and the
|
||||||
|
// headline has to say so rather than reading as a success.
|
||||||
|
expect(subtitle("news.example")).toMatch(/ — Allowed — SERVFAIL \(2\)$/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a forward-zone answer says the matcher never ran, not that nothing matched", async () => {
|
||||||
|
responses["/api/queries/14"] = detail(14, {
|
||||||
|
request: { domain: "nas.lan.home" },
|
||||||
|
policy: { action: "allow", reason: "forward_zone", matched: "" },
|
||||||
|
route: { kind: "forward_zone", forward_zone: "lan.home", upstream: "udp://192.168.1.1:53" },
|
||||||
|
});
|
||||||
|
renderDetail(14);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "nas.lan.home" });
|
||||||
|
expect(factValue("Reason")).toBe("Forward zone");
|
||||||
|
expect(factValue("Matched")).toBe("The matcher never ran");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a query the matcher did evaluate keeps the honest empty verdict", async () => {
|
||||||
|
responses["/api/queries/15"] = detail(15, {
|
||||||
|
policy: { action: "allow", reason: "no_match", matched: "" },
|
||||||
|
});
|
||||||
|
renderDetail(15);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "example.com" });
|
||||||
|
expect(factValue("Reason")).toBe("No match");
|
||||||
|
expect(factValue("Matched")).toBe("Nothing matched");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("empty text fields read as absent facts, never as blank values", async () => {
|
||||||
|
responses["/api/queries/8"] = detail(8, {
|
||||||
|
group: { id: null, name: "" },
|
||||||
|
policy: { action: "not_evaluated", reason: "paused", matched: "", source_id: null, source_name: "" },
|
||||||
|
route: { kind: "upstream", forward_zone: "", upstream: "udp://9.9.9.9:53" },
|
||||||
|
});
|
||||||
|
renderDetail(8);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "example.com" });
|
||||||
|
expect(factValue("Name")).toBe("No group recorded");
|
||||||
|
expect(factValue("Matched")).toBe("The matcher never ran");
|
||||||
|
expect(factValue("Blocklist")).toBe("Not a blocklist decision");
|
||||||
|
expect(factValue("Safe search")).toBe("No rewrite");
|
||||||
|
expect(factValue("Reason")).toBe("Filtering paused");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a log with hidden domains renders the server's marker, with nothing invented around it", async () => {
|
||||||
|
responses["/api/queries/9"] = detail(9, {
|
||||||
|
request: { domain: "hidden" },
|
||||||
|
policy: { action: "block", reason: "blocklist_domain", matched: "hidden", source_name: "StevenBlack" },
|
||||||
|
rewrites: { cname_target: "hidden", safe_search_target: "hidden" },
|
||||||
|
route: { kind: "blocked", upstream: "" },
|
||||||
|
});
|
||||||
|
renderDetail(9);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "hidden" });
|
||||||
|
expect(factValue("Domain")).toBe("hidden");
|
||||||
|
expect(factValue("Matched")).toBe("hidden");
|
||||||
|
expect(factValue("CNAME target")).toBe("hidden");
|
||||||
|
expect(factValue("Safe search")).toBe("hidden");
|
||||||
|
// The client is governed by its own flag and stays visible here.
|
||||||
|
expect(factValue("Client")).toBe("192.0.2.10");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("the related actions carry the query's own domain and client into the live pages", async () => {
|
||||||
|
responses["/api/queries/11"] = detail(11, { request: { domain: "shop.example", client: "192.0.2.12" } });
|
||||||
|
renderDetail(11);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "shop.example" });
|
||||||
|
const related = screen.getByRole("heading", { name: "Related" }).parentElement!;
|
||||||
|
expect(within(related).getByRole("link", { name: "Look up this domain now" }).getAttribute("href")).toBe(
|
||||||
|
"/lookup?domain=shop.example",
|
||||||
|
);
|
||||||
|
expect(within(related).getByRole("link", { name: "All queries for this domain" }).getAttribute("href")).toBe(
|
||||||
|
"/queries?domain=shop.example",
|
||||||
|
);
|
||||||
|
expect(within(related).getByRole("link", { name: "All queries from this client" }).getAttribute("href")).toBe(
|
||||||
|
"/queries?client=192.0.2.12",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
function clientList(client: { ip: string; name: string; learned_name: string }) {
|
||||||
|
return {
|
||||||
|
clients: [
|
||||||
|
{
|
||||||
|
id: 1,
|
||||||
|
group_id: 1,
|
||||||
|
group: "default",
|
||||||
|
hand_edited: client.name !== "",
|
||||||
|
first_seen: 1_700_000_000,
|
||||||
|
last_seen: 1_700_000_100,
|
||||||
|
...client,
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The related section, whose text is read whole because it is prose, not facts. */
|
||||||
|
async function relatedText(expected: string) {
|
||||||
|
const related = screen.getByRole("heading", { name: "Related" }).parentElement!;
|
||||||
|
await waitFor(() => expect(related.textContent?.replace(/\s+/g, " ")).toContain(expected));
|
||||||
|
}
|
||||||
|
|
||||||
|
test("the record keeps the address the query came from, and Related carries the name it has now", async () => {
|
||||||
|
responses["/api/queries/12"] = detail(12, { request: { domain: "shop.example", client: "192.0.2.12" } });
|
||||||
|
responses["/api/clients"] = clientList({ ip: "192.0.2.12", name: "Kids iPad", learned_name: "ipad.lan" });
|
||||||
|
renderDetail(12);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "shop.example" });
|
||||||
|
await relatedText("The client list currently names 192.0.2.12 “Kids iPad”.");
|
||||||
|
expect(factValue("Client")).toBe("192.0.2.12");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a learned name is told as the reverse-DNS lookup it is, never as a recorded fact", async () => {
|
||||||
|
responses["/api/queries/13"] = detail(13, { request: { client: "192.0.2.13" } });
|
||||||
|
responses["/api/clients"] = clientList({ ip: "192.0.2.13", name: "", learned_name: "printer.lan" });
|
||||||
|
renderDetail(13);
|
||||||
|
|
||||||
|
await screen.findByRole("heading", { name: "example.com" });
|
||||||
|
await relatedText("Reverse DNS currently resolves 192.0.2.13 to printer.lan.");
|
||||||
|
expect(factValue("Client")).toBe("192.0.2.13");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a row retention has pruned explains the 404 and keeps the way back to the log", async () => {
|
||||||
|
renderDetail(404);
|
||||||
|
|
||||||
|
await screen.findByRole("alert");
|
||||||
|
expect(screen.getByText(/no such query/)).toBeTruthy();
|
||||||
|
expect(screen.getByRole("link", { name: "← Query log" }).getAttribute("href")).toBe("/queries");
|
||||||
|
});
|
||||||
@@ -0,0 +1,329 @@
|
|||||||
|
import type { ReactNode } from "react";
|
||||||
|
import { useQuery } from "@tanstack/react-query";
|
||||||
|
import { Link, useParams } from "@tanstack/react-router";
|
||||||
|
import * as stylex from "@stylexjs/stylex";
|
||||||
|
import InlineError from "@/lib/InlineError";
|
||||||
|
import { formatMicros, formatTime } from "@/lib/format";
|
||||||
|
import { queryDetailQuery } from "@/lib/queries";
|
||||||
|
import type { PolicyReason, QueryDetail } from "@/lib/types";
|
||||||
|
import { clientLabel, useClientNames } from "@/features/clients/clientNames";
|
||||||
|
import { policyActionLabel, policyReasonLabel, qclassName, rcodeName, routeKindLabel } from "./provenanceCopy";
|
||||||
|
import { qtypeName } from "./qtype";
|
||||||
|
import { styles as shared } from "@/ui/styles";
|
||||||
|
import { colors } from "@/ui/tokens.stylex";
|
||||||
|
|
||||||
|
const styles = stylex.create({
|
||||||
|
back: {
|
||||||
|
fontSize: "0.875rem",
|
||||||
|
lineHeight: "1.25rem",
|
||||||
|
color: colors.primaryOnSurface,
|
||||||
|
textDecorationLine: "none",
|
||||||
|
},
|
||||||
|
heading: {
|
||||||
|
marginTop: "0.5rem",
|
||||||
|
fontSize: "1.5rem",
|
||||||
|
lineHeight: "2rem",
|
||||||
|
fontWeight: 600,
|
||||||
|
wordBreak: "break-all",
|
||||||
|
},
|
||||||
|
subtitle: {
|
||||||
|
marginTop: "0.25rem",
|
||||||
|
color: colors.textSecondary,
|
||||||
|
},
|
||||||
|
/** The recorded facts, fenced off from the live links below them. */
|
||||||
|
record: {
|
||||||
|
marginTop: "1rem",
|
||||||
|
maxWidth: "48rem",
|
||||||
|
borderRadius: "0.25rem",
|
||||||
|
borderWidth: 1,
|
||||||
|
borderStyle: "solid",
|
||||||
|
borderColor: colors.border,
|
||||||
|
backgroundColor: colors.surfaceRaised,
|
||||||
|
padding: "1rem",
|
||||||
|
},
|
||||||
|
recordNote: {
|
||||||
|
fontSize: "0.8125rem",
|
||||||
|
lineHeight: "1.25rem",
|
||||||
|
color: colors.textMuted,
|
||||||
|
},
|
||||||
|
section: {
|
||||||
|
marginTop: "1rem",
|
||||||
|
borderTopWidth: { default: 1, ":first-of-type": 0 },
|
||||||
|
borderTopStyle: "solid",
|
||||||
|
borderTopColor: colors.border,
|
||||||
|
paddingTop: { default: "1rem", ":first-of-type": 0 },
|
||||||
|
},
|
||||||
|
sectionHeading: {
|
||||||
|
fontSize: "0.75rem",
|
||||||
|
lineHeight: "1rem",
|
||||||
|
fontWeight: 600,
|
||||||
|
letterSpacing: "0.05em",
|
||||||
|
textTransform: "uppercase",
|
||||||
|
color: colors.textMuted,
|
||||||
|
},
|
||||||
|
facts: {
|
||||||
|
marginTop: "0.5rem",
|
||||||
|
marginBottom: 0,
|
||||||
|
display: "grid",
|
||||||
|
gap: "0.375rem 1rem",
|
||||||
|
gridTemplateColumns: {
|
||||||
|
default: "auto",
|
||||||
|
"@media (min-width: 640px)": "max-content 1fr",
|
||||||
|
},
|
||||||
|
fontSize: "0.875rem",
|
||||||
|
lineHeight: "1.25rem",
|
||||||
|
},
|
||||||
|
term: {
|
||||||
|
color: colors.textMuted,
|
||||||
|
},
|
||||||
|
value: {
|
||||||
|
margin: 0,
|
||||||
|
wordBreak: "break-all",
|
||||||
|
},
|
||||||
|
muted: {
|
||||||
|
color: colors.textMuted,
|
||||||
|
},
|
||||||
|
related: {
|
||||||
|
marginTop: "1.5rem",
|
||||||
|
maxWidth: "48rem",
|
||||||
|
},
|
||||||
|
relatedHeading: {
|
||||||
|
fontSize: "1.125rem",
|
||||||
|
lineHeight: "1.75rem",
|
||||||
|
fontWeight: 600,
|
||||||
|
},
|
||||||
|
relatedNote: {
|
||||||
|
marginTop: "0.25rem",
|
||||||
|
fontSize: "0.875rem",
|
||||||
|
lineHeight: "1.25rem",
|
||||||
|
color: colors.textMuted,
|
||||||
|
},
|
||||||
|
relatedList: {
|
||||||
|
marginTop: "0.5rem",
|
||||||
|
display: "flex",
|
||||||
|
flexWrap: "wrap",
|
||||||
|
gap: "0.75rem",
|
||||||
|
fontSize: "0.875rem",
|
||||||
|
lineHeight: "1.25rem",
|
||||||
|
},
|
||||||
|
link: {
|
||||||
|
color: colors.primaryOnSurface,
|
||||||
|
},
|
||||||
|
loading: {
|
||||||
|
marginTop: "1rem",
|
||||||
|
color: colors.textMuted,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
function Section({ title, children }: { title: string; children: ReactNode }) {
|
||||||
|
return (
|
||||||
|
<div {...stylex.props(styles.section)}>
|
||||||
|
<h2 {...stylex.props(styles.sectionHeading)}>{title}</h2>
|
||||||
|
<dl {...stylex.props(styles.facts)}>{children}</dl>
|
||||||
|
</div>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
function Fact({ label, children }: { label: string; children: ReactNode }) {
|
||||||
|
return (
|
||||||
|
<>
|
||||||
|
<dt {...stylex.props(styles.term)}>{label}</dt>
|
||||||
|
<dd {...stylex.props(styles.value)}>{children}</dd>
|
||||||
|
</>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* A recorded name, in the face the rest of the interface gives to names. It
|
||||||
|
* wraps the value rather than the row, so the prose that stands in for a
|
||||||
|
* missing one is not set in the same typewriter face.
|
||||||
|
*/
|
||||||
|
function Mono({ children }: { children: string }) {
|
||||||
|
return <span {...stylex.props(shared.mono)}>{children}</span>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** An empty text field means the server recorded nothing there, never an empty value. */
|
||||||
|
function Absent({ children }: { children: string }) {
|
||||||
|
return <span {...stylex.props(styles.muted)}>{children}</span>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What an empty `matched` means. `no_match` is the only reason the matcher
|
||||||
|
* itself records with nothing to show; every other empty one names a pipeline
|
||||||
|
* step that answered before filtering — a local record, a forward zone, a pause
|
||||||
|
* (see `PolicyReason` in src/storage/provenance.zig) — where "nothing matched"
|
||||||
|
* would claim an evaluation that never happened.
|
||||||
|
*/
|
||||||
|
function unmatchedLabel(reason: PolicyReason): string {
|
||||||
|
return reason === "no_match" ? "Nothing matched" : "The matcher never ran";
|
||||||
|
}
|
||||||
|
|
||||||
|
export default function QueryDetailPage() {
|
||||||
|
const { id } = useParams({ from: "/shell/queries/$id" });
|
||||||
|
const rowId = Number(id);
|
||||||
|
const { data, error, isPending, refetch } = useQuery(queryDetailQuery(rowId));
|
||||||
|
const clientNames = useClientNames();
|
||||||
|
|
||||||
|
if (isPending) {
|
||||||
|
return (
|
||||||
|
<p {...stylex.props(styles.loading, shared.pulse)} role="status">
|
||||||
|
Loading query…
|
||||||
|
</p>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (data === undefined) {
|
||||||
|
return (
|
||||||
|
<section>
|
||||||
|
<Link to="/queries" {...stylex.props(styles.back, shared.focusRing)}>
|
||||||
|
← Query log
|
||||||
|
</Link>
|
||||||
|
<InlineError error={error} onRetry={() => void refetch()} />
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
const detail: QueryDetail = data;
|
||||||
|
const { request, group, policy, rewrites, route, response } = detail;
|
||||||
|
const currentClient = clientLabel(request.client, clientNames);
|
||||||
|
|
||||||
|
return (
|
||||||
|
<section>
|
||||||
|
<Link to="/queries" {...stylex.props(styles.back, shared.focusRing)}>
|
||||||
|
← Query log
|
||||||
|
</Link>
|
||||||
|
<h1 {...stylex.props(styles.heading, shared.mono)}>{request.domain}</h1>
|
||||||
|
<p {...stylex.props(styles.subtitle)}>
|
||||||
|
{formatTime(request.time)} — {policyActionLabel(policy.action)}
|
||||||
|
{/* The verdict alone reads as a success; a non-NOERROR answer says otherwise. */}
|
||||||
|
{response.rcode !== 0 && ` — ${rcodeName(response.rcode)}`}
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<div {...stylex.props(styles.record)}>
|
||||||
|
<p {...stylex.props(styles.recordNote)}>
|
||||||
|
What was recorded when this query was answered. Group and blocklist names are the ones in force at
|
||||||
|
that moment; they may have been renamed or deleted since.
|
||||||
|
</p>
|
||||||
|
|
||||||
|
<Section title="Request">
|
||||||
|
<Fact label="Time">{formatTime(request.time)}</Fact>
|
||||||
|
<Fact label="Domain">
|
||||||
|
<Mono>{request.domain}</Mono>
|
||||||
|
</Fact>
|
||||||
|
<Fact label="Client">
|
||||||
|
<Mono>{request.client}</Mono>
|
||||||
|
</Fact>
|
||||||
|
<Fact label="Type">{qtypeName(request.qtype)}</Fact>
|
||||||
|
<Fact label="Class">{qclassName(request.qclass)}</Fact>
|
||||||
|
</Section>
|
||||||
|
|
||||||
|
<Section title="Group">
|
||||||
|
<Fact label="Name">{group.name === "" ? <Absent>No group recorded</Absent> : group.name}</Fact>
|
||||||
|
<Fact label="Id">{group.id === null ? <Absent>—</Absent> : group.id}</Fact>
|
||||||
|
</Section>
|
||||||
|
|
||||||
|
<Section title="Policy">
|
||||||
|
<Fact label="Decision">{policyActionLabel(policy.action)}</Fact>
|
||||||
|
<Fact label="Reason">{policyReasonLabel(policy.reason)}</Fact>
|
||||||
|
<Fact label="Matched">
|
||||||
|
{policy.matched === "" ? (
|
||||||
|
<Absent>{unmatchedLabel(policy.reason)}</Absent>
|
||||||
|
) : (
|
||||||
|
<Mono>{policy.matched}</Mono>
|
||||||
|
)}
|
||||||
|
</Fact>
|
||||||
|
<Fact label="Blocklist">
|
||||||
|
{policy.source_name === "" ? (
|
||||||
|
<Absent>Not a blocklist decision</Absent>
|
||||||
|
) : policy.source_id === null ? (
|
||||||
|
policy.source_name
|
||||||
|
) : (
|
||||||
|
`${policy.source_name} (#${policy.source_id})`
|
||||||
|
)}
|
||||||
|
</Fact>
|
||||||
|
</Section>
|
||||||
|
|
||||||
|
<Section title="Rewrites">
|
||||||
|
<Fact label="CNAME target">
|
||||||
|
{rewrites.cname_target === "" ? (
|
||||||
|
<Absent>The queried name was decided directly</Absent>
|
||||||
|
) : (
|
||||||
|
<Mono>{rewrites.cname_target}</Mono>
|
||||||
|
)}
|
||||||
|
</Fact>
|
||||||
|
<Fact label="Safe search">
|
||||||
|
{rewrites.safe_search_target === "" ? (
|
||||||
|
<Absent>No rewrite</Absent>
|
||||||
|
) : (
|
||||||
|
<Mono>{rewrites.safe_search_target}</Mono>
|
||||||
|
)}
|
||||||
|
</Fact>
|
||||||
|
</Section>
|
||||||
|
|
||||||
|
<Section title="Route">
|
||||||
|
<Fact label="Answered by">{routeKindLabel(route.kind)}</Fact>
|
||||||
|
<Fact label="Forward zone">
|
||||||
|
{route.forward_zone === "" ? <Absent>—</Absent> : <Mono>{route.forward_zone}</Mono>}
|
||||||
|
</Fact>
|
||||||
|
<Fact label="Upstream">
|
||||||
|
{route.upstream === "" ? <Absent>No upstream exchange</Absent> : <Mono>{route.upstream}</Mono>}
|
||||||
|
</Fact>
|
||||||
|
</Section>
|
||||||
|
|
||||||
|
<Section title="Response">
|
||||||
|
<Fact label="Result">{rcodeName(response.rcode)}</Fact>
|
||||||
|
<Fact label="Took">
|
||||||
|
{response.duration_us === null ? (
|
||||||
|
<Absent>Not measured</Absent>
|
||||||
|
) : (
|
||||||
|
formatMicros(response.duration_us)
|
||||||
|
)}
|
||||||
|
</Fact>
|
||||||
|
</Section>
|
||||||
|
</div>
|
||||||
|
|
||||||
|
<div {...stylex.props(styles.related)}>
|
||||||
|
<h2 {...stylex.props(styles.relatedHeading)}>Related</h2>
|
||||||
|
<p {...stylex.props(styles.relatedNote)}>
|
||||||
|
These read the current configuration, which may no longer be the one that decided this query.
|
||||||
|
</p>
|
||||||
|
{currentClient !== null && (
|
||||||
|
<p {...stylex.props(styles.relatedNote)}>
|
||||||
|
{currentClient.learned ? (
|
||||||
|
<>
|
||||||
|
Reverse DNS currently resolves <Mono>{request.client}</Mono> to{" "}
|
||||||
|
<Mono>{currentClient.text}</Mono>.
|
||||||
|
</>
|
||||||
|
) : (
|
||||||
|
<>
|
||||||
|
The client list currently names <Mono>{request.client}</Mono> “{currentClient.text}”.
|
||||||
|
</>
|
||||||
|
)}
|
||||||
|
</p>
|
||||||
|
)}
|
||||||
|
<div {...stylex.props(styles.relatedList)}>
|
||||||
|
<Link
|
||||||
|
to="/lookup"
|
||||||
|
search={{ domain: request.domain }}
|
||||||
|
{...stylex.props(styles.link, shared.focusRing)}
|
||||||
|
>
|
||||||
|
Look up this domain now
|
||||||
|
</Link>
|
||||||
|
<Link
|
||||||
|
to="/queries"
|
||||||
|
search={{ domain: request.domain }}
|
||||||
|
{...stylex.props(styles.link, shared.focusRing)}
|
||||||
|
>
|
||||||
|
All queries for this domain
|
||||||
|
</Link>
|
||||||
|
<Link
|
||||||
|
to="/queries"
|
||||||
|
search={{ client: request.client }}
|
||||||
|
{...stylex.props(styles.link, shared.focusRing)}
|
||||||
|
>
|
||||||
|
All queries from this client
|
||||||
|
</Link>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</section>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -1,8 +1,11 @@
|
|||||||
import { act, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
|
import { act, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
|
||||||
import { QueryClientProvider } from "@tanstack/react-query";
|
import { QueryClientProvider } from "@tanstack/react-query";
|
||||||
|
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
|
||||||
|
import { AuthProvider } from "@/auth/store";
|
||||||
import { createQueryClient } from "@/lib/queryClient";
|
import { createQueryClient } from "@/lib/queryClient";
|
||||||
import type { Client, QueriesPage, QueryRow } from "@/lib/types";
|
import { createAppRouter } from "@/routes";
|
||||||
import QueryLogPage from "./QueryLogPage";
|
import type { Client, Coverage, QueriesPage, QueryRow } from "@/lib/types";
|
||||||
|
import { queryRow } from "./provenanceFixture";
|
||||||
|
|
||||||
function client(id: number, ip: string, name: string, learnedName: string): Client {
|
function client(id: number, ip: string, name: string, learnedName: string): Client {
|
||||||
return {
|
return {
|
||||||
@@ -25,41 +28,38 @@ const CLIENTS: Client[] = [
|
|||||||
];
|
];
|
||||||
|
|
||||||
function row(id: number, domain: string, overrides: Partial<QueryRow> = {}): QueryRow {
|
function row(id: number, domain: string, overrides: Partial<QueryRow> = {}): QueryRow {
|
||||||
return {
|
return queryRow(id, { ts: 1_700_000_000 + id, domain, upstream: "udp://9.9.9.9:53", ...overrides });
|
||||||
id,
|
|
||||||
ts: 1_700_000_000 + id,
|
|
||||||
domain,
|
|
||||||
client_ip: "192.0.2.10",
|
|
||||||
qtype: 1,
|
|
||||||
blocked: false,
|
|
||||||
block_reason: "",
|
|
||||||
response_time_us: 1234,
|
|
||||||
cache_hit: false,
|
|
||||||
upstream: "udp://9.9.9.9:53",
|
|
||||||
...overrides,
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const COMPLETE: Coverage = { complete: true, available_since: 1_600_000_000 };
|
||||||
|
|
||||||
|
/** The blocked row every page fixture reuses. */
|
||||||
|
const BLOCKED = {
|
||||||
|
blocked: true,
|
||||||
|
policy_action: "block",
|
||||||
|
policy_reason: "blocklist_wildcard",
|
||||||
|
route_kind: "blocked",
|
||||||
|
upstream: "",
|
||||||
|
} as const satisfies Partial<QueryRow>;
|
||||||
|
|
||||||
const PAGES: Record<string, QueriesPage> = {
|
const PAGES: Record<string, QueriesPage> = {
|
||||||
"/api/queries": {
|
"/api/queries": {
|
||||||
queries: [
|
queries: [
|
||||||
row(20, "first.example", { qtype: 65, cache_hit: true, upstream: "" }),
|
row(20, "first.example", { qtype: 65, cache_hit: true }),
|
||||||
row(19, "ads.example", {
|
row(19, "ads.example", { ...BLOCKED, response_time_us: null, cache_hit: null }),
|
||||||
blocked: true,
|
|
||||||
block_reason: "blocklist:stevenblack",
|
|
||||||
response_time_us: null,
|
|
||||||
cache_hit: null,
|
|
||||||
}),
|
|
||||||
],
|
],
|
||||||
next_before: 19,
|
next_before: 19,
|
||||||
|
coverage: COMPLETE,
|
||||||
},
|
},
|
||||||
"/api/queries?before=19": {
|
"/api/queries?before=19": {
|
||||||
queries: [row(5, "older.example")],
|
queries: [row(5, "older.example")],
|
||||||
next_before: null,
|
next_before: null,
|
||||||
|
coverage: COMPLETE,
|
||||||
},
|
},
|
||||||
"/api/queries?domain=ads": {
|
"/api/queries?domain=ads": {
|
||||||
queries: [row(19, "ads.example", { blocked: true, block_reason: "blocklist:stevenblack" })],
|
queries: [row(19, "ads.example", BLOCKED)],
|
||||||
next_before: null,
|
next_before: null,
|
||||||
|
coverage: COMPLETE,
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -83,12 +83,19 @@ afterEach(() => {
|
|||||||
vi.unstubAllGlobals();
|
vi.unstubAllGlobals();
|
||||||
});
|
});
|
||||||
|
|
||||||
function renderPage() {
|
/**
|
||||||
|
* The whole router, not the bare component: the rows link into `/queries/$id`
|
||||||
|
* and the filter form seeds itself from the url, so both need real routing.
|
||||||
|
*/
|
||||||
|
function renderPage(path = "/queries") {
|
||||||
const client = createQueryClient();
|
const client = createQueryClient();
|
||||||
|
const router = createAppRouter(createMemoryHistory({ initialEntries: [path] }), client);
|
||||||
render(
|
render(
|
||||||
<QueryClientProvider client={client}>
|
<AuthProvider>
|
||||||
<QueryLogPage />
|
<QueryClientProvider client={client}>
|
||||||
</QueryClientProvider>,
|
<RouterProvider router={router} />
|
||||||
|
</QueryClientProvider>
|
||||||
|
</AuthProvider>,
|
||||||
);
|
);
|
||||||
return client;
|
return client;
|
||||||
}
|
}
|
||||||
@@ -104,7 +111,7 @@ test("renders the first page with type names, blocked badge, and formatted cells
|
|||||||
expect(screen.getByText("HTTPS")).toBeTruthy();
|
expect(screen.getByText("HTTPS")).toBeTruthy();
|
||||||
expect(screen.getByText("A")).toBeTruthy();
|
expect(screen.getByText("A")).toBeTruthy();
|
||||||
expect(screen.getByText("Blocked")).toBeTruthy();
|
expect(screen.getByText("Blocked")).toBeTruthy();
|
||||||
expect(screen.getByText("blocklist:stevenblack")).toBeTruthy();
|
expect(screen.getByText("Blocklist (wildcard)")).toBeTruthy();
|
||||||
expect(screen.getByText("1.2 ms")).toBeTruthy();
|
expect(screen.getByText("1.2 ms")).toBeTruthy();
|
||||||
expect(screen.getByText("hit")).toBeTruthy();
|
expect(screen.getByText("hit")).toBeTruthy();
|
||||||
expect(screen.getByText("udp://9.9.9.9:53")).toBeTruthy();
|
expect(screen.getByText("udp://9.9.9.9:53")).toBeTruthy();
|
||||||
@@ -126,6 +133,7 @@ test("resolves each row's client to its display name, keeping the IP as the tool
|
|||||||
row(17, "stranger.example", { client_ip: "192.0.2.99" }),
|
row(17, "stranger.example", { client_ip: "192.0.2.99" }),
|
||||||
],
|
],
|
||||||
next_before: null,
|
next_before: null,
|
||||||
|
coverage: COMPLETE,
|
||||||
} satisfies QueriesPage);
|
} satisfies QueriesPage);
|
||||||
}),
|
}),
|
||||||
);
|
);
|
||||||
@@ -229,12 +237,14 @@ test("a load-more that resolves after a filter change is discarded", async () =>
|
|||||||
test("load more is disabled while a filter change shows placeholder data, then uses the fresh cursor", async () => {
|
test("load more is disabled while a filter change shows placeholder data, then uses the fresh cursor", async () => {
|
||||||
let releaseFiltered: () => void = () => {};
|
let releaseFiltered: () => void = () => {};
|
||||||
const filteredPage: QueriesPage = {
|
const filteredPage: QueriesPage = {
|
||||||
queries: [row(19, "ads.example", { blocked: true, block_reason: "blocklist:stevenblack" })],
|
queries: [row(19, "ads.example", BLOCKED)],
|
||||||
next_before: 7,
|
next_before: 7,
|
||||||
|
coverage: COMPLETE,
|
||||||
};
|
};
|
||||||
const filteredOlderPage: QueriesPage = {
|
const filteredOlderPage: QueriesPage = {
|
||||||
queries: [row(3, "ads.older.example")],
|
queries: [row(3, "ads.older.example")],
|
||||||
next_before: null,
|
next_before: null,
|
||||||
|
coverage: COMPLETE,
|
||||||
};
|
};
|
||||||
const fetchMock = vi.fn((input: RequestInfo | URL) => {
|
const fetchMock = vi.fn((input: RequestInfo | URL) => {
|
||||||
const url = String(input);
|
const url = String(input);
|
||||||
@@ -292,14 +302,27 @@ test("a background refetch after new rows arrive leaves no gap between the loade
|
|||||||
// Refetching only the first page would drop n20 and n19 out of the middle
|
// Refetching only the first page would drop n20 and n19 out of the middle
|
||||||
// of the table; the second page must be replayed from the fresh cursor.
|
// of the table; the second page must be replayed from the fresh cursor.
|
||||||
const before: Record<string, QueriesPage> = {
|
const before: Record<string, QueriesPage> = {
|
||||||
"/api/queries": { queries: [row(20, "n20.example"), row(19, "n19.example")], next_before: 19 },
|
"/api/queries": {
|
||||||
"/api/queries?before=19": { queries: [row(18, "n18.example"), row(17, "n17.example")], next_before: null },
|
queries: [row(20, "n20.example"), row(19, "n19.example")],
|
||||||
|
next_before: 19,
|
||||||
|
coverage: COMPLETE,
|
||||||
|
},
|
||||||
|
"/api/queries?before=19": {
|
||||||
|
queries: [row(18, "n18.example"), row(17, "n17.example")],
|
||||||
|
next_before: null,
|
||||||
|
coverage: COMPLETE,
|
||||||
|
},
|
||||||
};
|
};
|
||||||
const after: Record<string, QueriesPage> = {
|
const after: Record<string, QueriesPage> = {
|
||||||
"/api/queries": { queries: [row(22, "n22.example"), row(21, "n21.example")], next_before: 21 },
|
"/api/queries": {
|
||||||
|
queries: [row(22, "n22.example"), row(21, "n21.example")],
|
||||||
|
next_before: 21,
|
||||||
|
coverage: COMPLETE,
|
||||||
|
},
|
||||||
"/api/queries?before=21": {
|
"/api/queries?before=21": {
|
||||||
queries: [row(20, "n20.example"), row(19, "n19.example"), row(18, "n18.example"), row(17, "n17.example")],
|
queries: [row(20, "n20.example"), row(19, "n19.example"), row(18, "n18.example"), row(17, "n17.example")],
|
||||||
next_before: null,
|
next_before: null,
|
||||||
|
coverage: COMPLETE,
|
||||||
},
|
},
|
||||||
};
|
};
|
||||||
let live = before;
|
let live = before;
|
||||||
@@ -361,3 +384,69 @@ test("a 401 on load more routes through handleUnauthorized instead of the inline
|
|||||||
expect(screen.queryByRole("alert")).toBeNull();
|
expect(screen.queryByRole("alert")).toBeNull();
|
||||||
expect(screen.queryByText(/Failed to load more/)).toBeNull();
|
expect(screen.queryByText(/Failed to load more/)).toBeNull();
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test("each row links into its own detail page by domain, reachable from the keyboard", async () => {
|
||||||
|
renderPage();
|
||||||
|
await screen.findByText("first.example");
|
||||||
|
|
||||||
|
const link = screen.getByRole("link", { name: "first.example" });
|
||||||
|
expect(link.getAttribute("href")).toBe("/queries/20");
|
||||||
|
// An <a href> is in the tab order by default; nothing here may opt it out.
|
||||||
|
expect(link.getAttribute("tabindex")).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("an allowed query names the rule that allowed it; an unremarkable one stays blank", async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn(async (input: RequestInfo | URL) => {
|
||||||
|
const url = String(input);
|
||||||
|
if (url === "/api/clients") return json({ clients: CLIENTS });
|
||||||
|
if (url !== "/api/queries") return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
|
||||||
|
return json({
|
||||||
|
queries: [row(20, "allowed.example", { policy_reason: "rule_allow_exact" }), row(19, "plain.example")],
|
||||||
|
next_before: null,
|
||||||
|
coverage: COMPLETE,
|
||||||
|
} satisfies QueriesPage);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage();
|
||||||
|
|
||||||
|
await screen.findByText("allowed.example");
|
||||||
|
expect(screen.getByText("Allow rule (exact)")).toBeTruthy();
|
||||||
|
expect(screen.queryByText("Blocked")).toBeNull();
|
||||||
|
expect(within(screen.getByText("plain.example").closest("tr")!).getByText("—")).toBeTruthy();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a pruned window tells the reader when history starts", async () => {
|
||||||
|
vi.stubGlobal(
|
||||||
|
"fetch",
|
||||||
|
vi.fn(async (input: RequestInfo | URL) => {
|
||||||
|
const url = String(input);
|
||||||
|
if (url === "/api/clients") return json({ clients: CLIENTS });
|
||||||
|
if (url !== "/api/queries") return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
|
||||||
|
return json({
|
||||||
|
queries: [row(20, "kept.example")],
|
||||||
|
next_before: null,
|
||||||
|
coverage: { complete: false, available_since: 1_700_000_000 },
|
||||||
|
} satisfies QueriesPage);
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
renderPage();
|
||||||
|
|
||||||
|
await screen.findByText("kept.example");
|
||||||
|
expect(screen.getByText(/Query history is available from/)).toBeTruthy();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a complete window shows no coverage notice", async () => {
|
||||||
|
renderPage();
|
||||||
|
await screen.findByText("first.example");
|
||||||
|
expect(screen.queryByText(/Query history is available from/)).toBeNull();
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a ?domain= link seeds the filter and fetches that domain on arrival", async () => {
|
||||||
|
renderPage("/queries?domain=ads");
|
||||||
|
|
||||||
|
await screen.findByText("ads.example");
|
||||||
|
expect(screen.getByLabelText("Domain contains")).toHaveProperty("value", "ads");
|
||||||
|
expect(screen.queryByText("first.example")).toBeNull();
|
||||||
|
});
|
||||||
|
|||||||
@@ -1,11 +1,15 @@
|
|||||||
import { useState, type FormEvent } from "react";
|
import { useState, type FormEvent } from "react";
|
||||||
import { useInfiniteQuery } from "@tanstack/react-query";
|
import { useInfiniteQuery } from "@tanstack/react-query";
|
||||||
|
import { Link, useSearch } from "@tanstack/react-router";
|
||||||
import * as stylex from "@stylexjs/stylex";
|
import * as stylex from "@stylexjs/stylex";
|
||||||
import * as api from "@/lib/api";
|
import * as api from "@/lib/api";
|
||||||
|
import CoverageNotice from "@/lib/CoverageNotice";
|
||||||
import { formatMicros, formatTime } from "@/lib/format";
|
import { formatMicros, formatTime } from "@/lib/format";
|
||||||
import { queriesInfiniteQuery } from "@/lib/queries";
|
import { queriesInfiniteQuery } from "@/lib/queries";
|
||||||
import type { QueriesFilter, QueryRow } from "@/lib/types";
|
import type { QueriesFilter, QueryRow } from "@/lib/types";
|
||||||
import { ClientName, useClientNames, type ClientNames } from "@/features/clients/clientNames";
|
import { ClientName, useClientNames, type ClientNames } from "@/features/clients/clientNames";
|
||||||
|
import { isUninformativeReason, policyReasonLabel } from "./provenanceCopy";
|
||||||
|
import { summarizeRow, type QuerySummary } from "./querySummary";
|
||||||
import { qtypeName } from "./qtype";
|
import { qtypeName } from "./qtype";
|
||||||
import Select from "@/ui/Select";
|
import Select from "@/ui/Select";
|
||||||
import { styles as shared } from "@/ui/styles";
|
import { styles as shared } from "@/ui/styles";
|
||||||
@@ -107,6 +111,11 @@ const styles = stylex.create({
|
|||||||
breakAll: {
|
breakAll: {
|
||||||
wordBreak: "break-all",
|
wordBreak: "break-all",
|
||||||
},
|
},
|
||||||
|
/** The row's way into the detail page; a real link, so tab and enter reach it. */
|
||||||
|
domainLink: {
|
||||||
|
color: colors.primaryOnSurface,
|
||||||
|
textDecorationLine: "none",
|
||||||
|
},
|
||||||
small: {
|
small: {
|
||||||
fontSize: "0.75rem",
|
fontSize: "0.75rem",
|
||||||
lineHeight: "1rem",
|
lineHeight: "1rem",
|
||||||
@@ -153,27 +162,54 @@ function datetimeLocalToUnix(value: string): number | undefined {
|
|||||||
return Number.isFinite(ms) ? Math.floor(ms / 1000) : undefined;
|
return Number.isFinite(ms) ? Math.floor(ms / 1000) : undefined;
|
||||||
}
|
}
|
||||||
|
|
||||||
export function BlockedCell({ row }: { row: Pick<QueryRow, "blocked" | "block_reason"> }) {
|
/**
|
||||||
if (!row.blocked) return <span {...stylex.props(styles.muted)}>—</span>;
|
* What the policy decided, and why. The reason is the stored enum rather than
|
||||||
|
* the old free-text block reason, so an allowed query that a rule or a blocklist
|
||||||
|
* exception explains says so too — only `no_match`, the answer for most allowed
|
||||||
|
* queries, stays blank.
|
||||||
|
*/
|
||||||
|
export function StatusCell({ row }: { row: Pick<QuerySummary, "blocked" | "policy_reason"> }) {
|
||||||
|
const reason = policyReasonLabel(row.policy_reason);
|
||||||
|
if (!row.blocked) {
|
||||||
|
if (isUninformativeReason(row.policy_reason)) return <span {...stylex.props(styles.muted)}>—</span>;
|
||||||
|
return <span {...stylex.props(styles.small, styles.muted)}>{reason}</span>;
|
||||||
|
}
|
||||||
return (
|
return (
|
||||||
<span {...stylex.props(styles.blockedWrap)}>
|
<span {...stylex.props(styles.blockedWrap)}>
|
||||||
<span {...stylex.props(styles.blockedBadge)}>Blocked</span>
|
<span {...stylex.props(styles.blockedBadge)}>Blocked</span>
|
||||||
{row.block_reason !== "" && <span {...stylex.props(styles.small, styles.muted)}>{row.block_reason}</span>}
|
<span {...stylex.props(styles.small, styles.muted)}>{reason}</span>
|
||||||
</span>
|
</span>
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
export function QueryCells({ row, clientNames }: { row: Omit<QueryRow, "id">; clientNames: ClientNames }) {
|
/**
|
||||||
|
* The eight shared cells. `row.id` is null for a live frame the server has not
|
||||||
|
* written yet, which is the one case with no detail page to link to.
|
||||||
|
*/
|
||||||
|
export function QueryCells({ row, clientNames }: { row: QuerySummary; clientNames: ClientNames }) {
|
||||||
|
const id = row.id;
|
||||||
return (
|
return (
|
||||||
<>
|
<>
|
||||||
<td {...stylex.props(styles.cell, styles.nowrap, styles.muted)}>{formatTime(row.ts)}</td>
|
<td {...stylex.props(styles.cell, styles.nowrap, styles.muted)}>{formatTime(row.ts)}</td>
|
||||||
<td {...stylex.props(styles.cell, styles.small, styles.breakAll, shared.mono)}>{row.domain}</td>
|
<td {...stylex.props(styles.cell, styles.small, styles.breakAll, shared.mono)}>
|
||||||
|
{id === null ? (
|
||||||
|
row.domain
|
||||||
|
) : (
|
||||||
|
<Link
|
||||||
|
to="/queries/$id"
|
||||||
|
params={{ id: String(id) }}
|
||||||
|
{...stylex.props(styles.domainLink, shared.focusRing)}
|
||||||
|
>
|
||||||
|
{row.domain}
|
||||||
|
</Link>
|
||||||
|
)}
|
||||||
|
</td>
|
||||||
<td {...stylex.props(styles.cell, styles.small, styles.nowrap)}>
|
<td {...stylex.props(styles.cell, styles.small, styles.nowrap)}>
|
||||||
<ClientName ip={row.client_ip} names={clientNames} />
|
<ClientName ip={row.client_ip} names={clientNames} />
|
||||||
</td>
|
</td>
|
||||||
<td {...stylex.props(styles.cell, styles.nowrap)}>{qtypeName(row.qtype)}</td>
|
<td {...stylex.props(styles.cell, styles.nowrap)}>{qtypeName(row.qtype)}</td>
|
||||||
<td {...stylex.props(styles.cell)}>
|
<td {...stylex.props(styles.cell)}>
|
||||||
<BlockedCell row={row} />
|
<StatusCell row={row} />
|
||||||
</td>
|
</td>
|
||||||
<td {...stylex.props(styles.cell, styles.nowrap, shared.tabularNums)}>
|
<td {...stylex.props(styles.cell, styles.nowrap, shared.tabularNums)}>
|
||||||
{row.response_time_us === null ? "—" : formatMicros(row.response_time_us)}
|
{row.response_time_us === null ? "—" : formatMicros(row.response_time_us)}
|
||||||
@@ -206,19 +242,29 @@ export function QueryTableHead() {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export default function QueryLogPage() {
|
export default function QueryLogPage() {
|
||||||
const [domain, setDomain] = useState("");
|
// The two url filters exist so a detail page can link back to "every query
|
||||||
const [client, setClient] = useState("");
|
// for this domain". They seed the form once; typing from here on is local
|
||||||
|
// state, as the other three filters always were.
|
||||||
|
const search = useSearch({ from: "/shell/queries" });
|
||||||
|
const [domain, setDomain] = useState(search.domain ?? "");
|
||||||
|
const [client, setClient] = useState(search.client ?? "");
|
||||||
const [blocked, setBlocked] = useState("any");
|
const [blocked, setBlocked] = useState("any");
|
||||||
const [since, setSince] = useState("");
|
const [since, setSince] = useState("");
|
||||||
const [until, setUntil] = useState("");
|
const [until, setUntil] = useState("");
|
||||||
|
|
||||||
const [applied, setApplied] = useState<QueriesFilter>({});
|
const [applied, setApplied] = useState<QueriesFilter>(() => {
|
||||||
|
const initial: QueriesFilter = {};
|
||||||
|
if (search.domain !== undefined) initial.domain = search.domain;
|
||||||
|
if (search.client !== undefined) initial.client = search.client;
|
||||||
|
return initial;
|
||||||
|
});
|
||||||
|
|
||||||
const base = useInfiniteQuery(queriesInfiniteQuery(applied));
|
const base = useInfiniteQuery(queriesInfiniteQuery(applied));
|
||||||
const clientNames = useClientNames();
|
const clientNames = useClientNames();
|
||||||
|
|
||||||
const pages = base.data?.pages ?? [];
|
const pages = base.data?.pages ?? [];
|
||||||
const rows: QueryRow[] = pages.flatMap((page) => page.queries);
|
const rows: QueryRow[] = pages.flatMap((page) => page.queries);
|
||||||
|
const coverage = pages[0]?.coverage;
|
||||||
const filterActive = Object.keys(applied).length > 0;
|
const filterActive = Object.keys(applied).length > 0;
|
||||||
// `base.hasNextPage` reads the query state, which is empty while placeholder
|
// `base.hasNextPage` reads the query state, which is empty while placeholder
|
||||||
// data stands in for a filter change; derive the cursor from what is on
|
// data stands in for a filter change; derive the cursor from what is on
|
||||||
@@ -324,6 +370,8 @@ export default function QueryLogPage() {
|
|||||||
</div>
|
</div>
|
||||||
</form>
|
</form>
|
||||||
|
|
||||||
|
{coverage !== undefined && <CoverageNotice coverage={coverage} />}
|
||||||
|
|
||||||
{base.data === undefined ? (
|
{base.data === undefined ? (
|
||||||
<p {...stylex.props(styles.empty, shared.pulse)} role="status">
|
<p {...stylex.props(styles.empty, shared.pulse)} role="status">
|
||||||
Loading query log…
|
Loading query log…
|
||||||
@@ -340,7 +388,7 @@ export default function QueryLogPage() {
|
|||||||
<tbody>
|
<tbody>
|
||||||
{rows.map((row) => (
|
{rows.map((row) => (
|
||||||
<tr key={row.id} {...stylex.props(styles.row)}>
|
<tr key={row.id} {...stylex.props(styles.row)}>
|
||||||
<QueryCells row={row} clientNames={clientNames} />
|
<QueryCells row={summarizeRow(row)} clientNames={clientNames} />
|
||||||
</tr>
|
</tr>
|
||||||
))}
|
))}
|
||||||
</tbody>
|
</tbody>
|
||||||
|
|||||||
@@ -0,0 +1,39 @@
|
|||||||
|
import {
|
||||||
|
ENUM_VALUES,
|
||||||
|
policyActionLabel,
|
||||||
|
policyReasonLabel,
|
||||||
|
qclassName,
|
||||||
|
rcodeName,
|
||||||
|
routeKindLabel,
|
||||||
|
} from "./provenanceCopy";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `tsc` proves the maps total over the union; this proves the union is the set
|
||||||
|
* the server actually stores, and that no entry was left as its raw tag name.
|
||||||
|
*/
|
||||||
|
test("every stored enum value has a label of its own", () => {
|
||||||
|
const labels = [
|
||||||
|
...ENUM_VALUES.policyAction.map(policyActionLabel),
|
||||||
|
...ENUM_VALUES.policyReason.map(policyReasonLabel),
|
||||||
|
...ENUM_VALUES.routeKind.map(routeKindLabel),
|
||||||
|
];
|
||||||
|
for (const label of labels) {
|
||||||
|
expect(label).not.toBe("");
|
||||||
|
expect(label).not.toMatch(/_/);
|
||||||
|
}
|
||||||
|
expect(new Set(ENUM_VALUES.policyReason.map(policyReasonLabel)).size).toBe(ENUM_VALUES.policyReason.length);
|
||||||
|
});
|
||||||
|
|
||||||
|
test("response codes read by name where one exists, by number where none does", () => {
|
||||||
|
expect(rcodeName(0)).toBe("NOERROR (0)");
|
||||||
|
expect(rcodeName(3)).toBe("NXDOMAIN (3)");
|
||||||
|
expect(rcodeName(16)).toBe("BADVERS (16)");
|
||||||
|
// The column holds the twelve-bit extended code, most of which is unassigned.
|
||||||
|
expect(rcodeName(3841)).toBe("RCODE 3841");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("query classes read the same way", () => {
|
||||||
|
expect(qclassName(1)).toBe("IN (1)");
|
||||||
|
expect(qclassName(255)).toBe("ANY (255)");
|
||||||
|
expect(qclassName(42)).toBe("CLASS 42");
|
||||||
|
});
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
import { POLICY_ACTIONS, POLICY_REASONS, ROUTE_KINDS } from "@/lib/types";
|
||||||
|
import type { PolicyAction, PolicyReason, RouteKind } from "@/lib/types";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Display names for the three stored enums. `Record` over the union, so a value
|
||||||
|
* added to `src/storage/provenance.zig` and mirrored into `lib/types.ts` fails
|
||||||
|
* `tsc` here instead of reaching a cell as a raw tag name.
|
||||||
|
*/
|
||||||
|
const POLICY_ACTION_LABELS: Record<PolicyAction, string> = {
|
||||||
|
not_evaluated: "Not evaluated",
|
||||||
|
allow: "Allowed",
|
||||||
|
block: "Blocked",
|
||||||
|
};
|
||||||
|
|
||||||
|
const POLICY_REASON_LABELS: Record<PolicyReason, string> = {
|
||||||
|
rule_allow_exact: "Allow rule (exact)",
|
||||||
|
rule_block_exact: "Block rule (exact)",
|
||||||
|
rule_allow_wildcard: "Allow rule (wildcard)",
|
||||||
|
rule_block_wildcard: "Block rule (wildcard)",
|
||||||
|
rule_allow_regex: "Allow rule (regex)",
|
||||||
|
rule_block_regex: "Block rule (regex)",
|
||||||
|
blocklist_exception: "Blocklist exception",
|
||||||
|
blocklist_domain: "Blocklist (domain)",
|
||||||
|
blocklist_wildcard: "Blocklist (wildcard)",
|
||||||
|
local_record: "Local record",
|
||||||
|
forward_zone: "Forward zone",
|
||||||
|
non_in_class: "Not class IN",
|
||||||
|
paused: "Filtering paused",
|
||||||
|
snapshot_unavailable: "No filter snapshot",
|
||||||
|
no_match: "No match",
|
||||||
|
protocol_error: "Protocol refusal",
|
||||||
|
};
|
||||||
|
|
||||||
|
const ROUTE_KIND_LABELS: Record<RouteKind, string> = {
|
||||||
|
blocked: "Blocked locally",
|
||||||
|
local: "Local record",
|
||||||
|
forward_zone: "Forward zone",
|
||||||
|
upstream: "Upstream resolver",
|
||||||
|
cache: "Cache",
|
||||||
|
rejected: "Rejected",
|
||||||
|
};
|
||||||
|
|
||||||
|
export function policyActionLabel(action: PolicyAction): string {
|
||||||
|
return POLICY_ACTION_LABELS[action];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function policyReasonLabel(reason: PolicyReason): string {
|
||||||
|
return POLICY_REASON_LABELS[reason];
|
||||||
|
}
|
||||||
|
|
||||||
|
export function routeKindLabel(kind: RouteKind): string {
|
||||||
|
return ROUTE_KIND_LABELS[kind];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `no_match` is the answer for the overwhelming majority of allowed queries and
|
||||||
|
* says nothing an operator scanning a table wants to read, so the status column
|
||||||
|
* leaves it blank. Every other reason names a decision worth seeing.
|
||||||
|
*/
|
||||||
|
export function isUninformativeReason(reason: PolicyReason): boolean {
|
||||||
|
return reason === "no_match";
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The enum value sets, for tests that prove the maps exhaustive at runtime too. */
|
||||||
|
export const ENUM_VALUES = {
|
||||||
|
policyAction: POLICY_ACTIONS,
|
||||||
|
policyReason: POLICY_REASONS,
|
||||||
|
routeKind: ROUTE_KINDS,
|
||||||
|
} as const;
|
||||||
|
|
||||||
|
const RCODE_NAMES: Record<number, string> = {
|
||||||
|
0: "NOERROR",
|
||||||
|
1: "FORMERR",
|
||||||
|
2: "SERVFAIL",
|
||||||
|
3: "NXDOMAIN",
|
||||||
|
4: "NOTIMP",
|
||||||
|
5: "REFUSED",
|
||||||
|
6: "YXDOMAIN",
|
||||||
|
7: "YXRRSET",
|
||||||
|
8: "NXRRSET",
|
||||||
|
9: "NOTAUTH",
|
||||||
|
10: "NOTZONE",
|
||||||
|
16: "BADVERS",
|
||||||
|
};
|
||||||
|
|
||||||
|
/** The twelve-bit extended code as `NXDOMAIN (3)`; an unassigned code keeps its number. */
|
||||||
|
export function rcodeName(rcode: number): string {
|
||||||
|
const name = RCODE_NAMES[rcode];
|
||||||
|
return name === undefined ? `RCODE ${rcode}` : `${name} (${rcode})`;
|
||||||
|
}
|
||||||
|
|
||||||
|
const QCLASS_NAMES: Record<number, string> = {
|
||||||
|
1: "IN",
|
||||||
|
3: "CH",
|
||||||
|
4: "HS",
|
||||||
|
254: "NONE",
|
||||||
|
255: "ANY",
|
||||||
|
};
|
||||||
|
|
||||||
|
export function qclassName(qclass: number): string {
|
||||||
|
const name = QCLASS_NAMES[qclass];
|
||||||
|
return name === undefined ? `CLASS ${qclass}` : `${name} (${qclass})`;
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
import type { Provenance, QueryRow } from "@/lib/types";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Fixture builders for the provenance shapes, shared by the query-log, detail
|
||||||
|
* and live-stream tests the way `features/live/fakeEventSource.ts` is shared.
|
||||||
|
*
|
||||||
|
* The defaults describe the dullest possible query — an allowed name nothing
|
||||||
|
* matched, answered upstream — so each test states only the fields it is about.
|
||||||
|
*/
|
||||||
|
type Sections = {
|
||||||
|
[K in keyof Provenance]?: Partial<Provenance[K]>;
|
||||||
|
};
|
||||||
|
|
||||||
|
export function provenance(sections: Sections = {}): Provenance {
|
||||||
|
return {
|
||||||
|
request: {
|
||||||
|
time: 1_700_000_000,
|
||||||
|
domain: "example.com",
|
||||||
|
client: "192.0.2.10",
|
||||||
|
qtype: 1,
|
||||||
|
qclass: 1,
|
||||||
|
...sections.request,
|
||||||
|
},
|
||||||
|
group: { id: 1, name: "default", ...sections.group },
|
||||||
|
policy: {
|
||||||
|
action: "allow",
|
||||||
|
reason: "no_match",
|
||||||
|
matched: "",
|
||||||
|
source_id: null,
|
||||||
|
source_name: "",
|
||||||
|
...sections.policy,
|
||||||
|
},
|
||||||
|
rewrites: { cname_target: "", safe_search_target: "", ...sections.rewrites },
|
||||||
|
route: { kind: "upstream", forward_zone: "", upstream: "https://dns.example/dns-query", ...sections.route },
|
||||||
|
response: { rcode: 0, duration_us: 1234, ...sections.response },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The flat stored row of the same dull query. */
|
||||||
|
export function queryRow(id: number, overrides: Partial<QueryRow> = {}): QueryRow {
|
||||||
|
return {
|
||||||
|
id,
|
||||||
|
ts: 1_700_000_000,
|
||||||
|
domain: "example.com",
|
||||||
|
client_ip: "192.0.2.10",
|
||||||
|
qtype: 1,
|
||||||
|
qclass: 1,
|
||||||
|
rcode: 0,
|
||||||
|
blocked: false,
|
||||||
|
response_time_us: 1234,
|
||||||
|
cache_hit: false,
|
||||||
|
upstream: "https://dns.example/dns-query",
|
||||||
|
policy_action: "allow",
|
||||||
|
policy_reason: "no_match",
|
||||||
|
route_kind: "upstream",
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
import type { LiveQueryEvent, PolicyReason, QueryRow, RouteKind } from "@/lib/types";
|
||||||
|
|
||||||
|
/**
|
||||||
|
* What the query-log table renders for one row, whichever surface it came from.
|
||||||
|
*
|
||||||
|
* The stored list row and the live stream's provenance event describe the same
|
||||||
|
* query in two different shapes — flat summary against nested full detail — and
|
||||||
|
* both pages share one set of cells, so both project into this.
|
||||||
|
*
|
||||||
|
* `id` is null for a streamed event: the frame precedes its own insert, so no
|
||||||
|
* row exists to link to yet.
|
||||||
|
*/
|
||||||
|
export interface QuerySummary {
|
||||||
|
id: number | null;
|
||||||
|
ts: number;
|
||||||
|
domain: string;
|
||||||
|
client_ip: string;
|
||||||
|
qtype: number | null;
|
||||||
|
blocked: boolean;
|
||||||
|
policy_reason: PolicyReason;
|
||||||
|
response_time_us: number | null;
|
||||||
|
cache_hit: boolean | null;
|
||||||
|
upstream: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether the cache answered, or null where it never applied. Mirrors
|
||||||
|
* `Context.cacheHit` in src/server/handler.zig, which derives the stored
|
||||||
|
* `cache_hit` column from the same route: a local record, a blocked answer and
|
||||||
|
* a protocol refusal all bypass the cache, and "miss" would claim a lookup that
|
||||||
|
* never happened.
|
||||||
|
*/
|
||||||
|
export function cacheHitFor(kind: RouteKind): boolean | null {
|
||||||
|
switch (kind) {
|
||||||
|
case "cache":
|
||||||
|
return true;
|
||||||
|
case "upstream":
|
||||||
|
case "forward_zone":
|
||||||
|
return false;
|
||||||
|
case "local":
|
||||||
|
case "blocked":
|
||||||
|
case "rejected":
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
export function summarizeRow(row: QueryRow): QuerySummary {
|
||||||
|
return {
|
||||||
|
id: row.id,
|
||||||
|
ts: row.ts,
|
||||||
|
domain: row.domain,
|
||||||
|
client_ip: row.client_ip,
|
||||||
|
qtype: row.qtype,
|
||||||
|
blocked: row.blocked,
|
||||||
|
policy_reason: row.policy_reason,
|
||||||
|
response_time_us: row.response_time_us,
|
||||||
|
cache_hit: row.cache_hit,
|
||||||
|
upstream: row.upstream,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The same summary out of a live frame. `blocked` and `cache_hit` are derived
|
||||||
|
* rather than sent: the server derives the stored columns from exactly these
|
||||||
|
* two fields (handler.zig's `Entry.init` call), so the projection reproduces
|
||||||
|
* them instead of the DTO carrying the same fact twice.
|
||||||
|
*/
|
||||||
|
export function summarizeEvent(event: LiveQueryEvent): QuerySummary {
|
||||||
|
return {
|
||||||
|
id: null,
|
||||||
|
ts: event.request.time,
|
||||||
|
domain: event.request.domain,
|
||||||
|
client_ip: event.request.client,
|
||||||
|
qtype: event.request.qtype,
|
||||||
|
blocked: event.policy.action === "block",
|
||||||
|
policy_reason: event.policy.reason,
|
||||||
|
response_time_us: event.response.duration_us,
|
||||||
|
cache_hit: cacheHitFor(event.route.kind),
|
||||||
|
upstream: event.route.upstream,
|
||||||
|
};
|
||||||
|
}
|
||||||
@@ -0,0 +1,31 @@
|
|||||||
|
import { render, screen } from "@testing-library/react";
|
||||||
|
import CoverageNotice from "./CoverageNotice";
|
||||||
|
import { formatTime } from "./format";
|
||||||
|
|
||||||
|
const WATERMARK = 1_700_000_000;
|
||||||
|
|
||||||
|
test("a window the log covers in full says nothing", () => {
|
||||||
|
const { container } = render(<CoverageNotice coverage={{ complete: true, available_since: WATERMARK }} />);
|
||||||
|
expect(container.textContent).toBe("");
|
||||||
|
});
|
||||||
|
|
||||||
|
test("a window reaching past the watermark names the instant history starts", () => {
|
||||||
|
render(<CoverageNotice coverage={{ complete: false, available_since: WATERMARK }} />);
|
||||||
|
const notice = screen.getByRole("status");
|
||||||
|
expect(notice.textContent).toContain("Query history is available from");
|
||||||
|
expect(notice.textContent).toContain(formatTime(WATERMARK));
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The server judges completeness, not the page: an unbounded request is
|
||||||
|
* incomplete whatever the watermark reads, and the notice must follow that
|
||||||
|
* verdict rather than compare timestamps itself.
|
||||||
|
*/
|
||||||
|
test("the server's verdict decides, not the numbers beside it", () => {
|
||||||
|
const { rerender, container } = render(
|
||||||
|
<CoverageNotice coverage={{ complete: true, available_since: WATERMARK }} />,
|
||||||
|
);
|
||||||
|
expect(container.textContent).toBe("");
|
||||||
|
rerender(<CoverageNotice coverage={{ complete: false, available_since: 0 }} />);
|
||||||
|
expect(screen.getByRole("status").textContent).toContain(formatTime(0));
|
||||||
|
});
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
import * as stylex from "@stylexjs/stylex";
|
||||||
|
import { formatTime } from "@/lib/format";
|
||||||
|
import type { Coverage } from "@/lib/types";
|
||||||
|
import { colors } from "@/ui/tokens.stylex";
|
||||||
|
|
||||||
|
const styles = stylex.create({
|
||||||
|
notice: {
|
||||||
|
marginTop: "0.75rem",
|
||||||
|
borderRadius: "0.25rem",
|
||||||
|
borderWidth: 1,
|
||||||
|
borderStyle: "solid",
|
||||||
|
borderColor: colors.border,
|
||||||
|
backgroundColor: colors.surfaceHover,
|
||||||
|
paddingInline: "0.75rem",
|
||||||
|
paddingBlock: "0.5rem",
|
||||||
|
fontSize: "0.875rem",
|
||||||
|
lineHeight: "1.25rem",
|
||||||
|
color: colors.textSecondary,
|
||||||
|
},
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How far back the numbers on this page can reach.
|
||||||
|
*
|
||||||
|
* Rendered whenever a response says its window is incomplete, which includes
|
||||||
|
* the common case of a request with no lower bound at all. The line states the
|
||||||
|
* watermark and nothing more: the same incompleteness covers a log retention
|
||||||
|
* has pruned and one that simply has not been running long enough, and the
|
||||||
|
* response does not say which.
|
||||||
|
*/
|
||||||
|
export default function CoverageNotice({ coverage }: { coverage: Coverage }) {
|
||||||
|
if (coverage.complete) return null;
|
||||||
|
return (
|
||||||
|
<p role="status" {...stylex.props(styles.notice)}>
|
||||||
|
Query history is available from {formatTime(coverage.available_since)}.
|
||||||
|
</p>
|
||||||
|
);
|
||||||
|
}
|
||||||
@@ -26,6 +26,7 @@ import type {
|
|||||||
Period,
|
Period,
|
||||||
QueriesFilter,
|
QueriesFilter,
|
||||||
QueriesPage,
|
QueriesPage,
|
||||||
|
QueryDetail,
|
||||||
Rule,
|
Rule,
|
||||||
RuleEcho,
|
RuleEcho,
|
||||||
RuleInput,
|
RuleInput,
|
||||||
@@ -112,6 +113,9 @@ export const logout = (): Promise<LogoutResponse> => request("/api/auth/logout",
|
|||||||
export const getQueries = (filter: QueriesFilter = {}): Promise<QueriesPage> =>
|
export const getQueries = (filter: QueriesFilter = {}): Promise<QueriesPage> =>
|
||||||
request(`/api/queries${qs({ ...filter })}`);
|
request(`/api/queries${qs({ ...filter })}`);
|
||||||
|
|
||||||
|
/** Full provenance of one logged row. 404 when retention has removed it, 503 with no query log. */
|
||||||
|
export const getQueryDetail = (id: number): Promise<QueryDetail> => request(`/api/queries/${id}`);
|
||||||
|
|
||||||
/** `EventSource` URL for the live stream; not a fetch route. */
|
/** `EventSource` URL for the live stream; not a fetch route. */
|
||||||
export const liveQueriesUrl = "/api/queries/live";
|
export const liveQueriesUrl = "/api/queries/live";
|
||||||
|
|
||||||
|
|||||||
@@ -29,6 +29,7 @@ import type {
|
|||||||
LookupResult,
|
LookupResult,
|
||||||
PauseState,
|
PauseState,
|
||||||
QueriesPage,
|
QueriesPage,
|
||||||
|
QueryDetail,
|
||||||
Rule,
|
Rule,
|
||||||
RuleEcho,
|
RuleEcho,
|
||||||
SettingsEnvelope,
|
SettingsEnvelope,
|
||||||
@@ -415,76 +416,139 @@ export const sample_get_upstream_health: UpstreamHealth = {
|
|||||||
};
|
};
|
||||||
|
|
||||||
export const sample_get_queries: QueriesPage = {
|
export const sample_get_queries: QueriesPage = {
|
||||||
|
coverage: {
|
||||||
|
available_since: 0,
|
||||||
|
complete: false,
|
||||||
|
},
|
||||||
next_before: 0,
|
next_before: 0,
|
||||||
queries: [
|
queries: [
|
||||||
{
|
{
|
||||||
block_reason: "",
|
blocked: true,
|
||||||
|
cache_hit: null,
|
||||||
|
client_ip: "192.0.2.11",
|
||||||
|
domain: "shop.example",
|
||||||
|
id: 0,
|
||||||
|
policy_action: "block",
|
||||||
|
policy_reason: "blocklist_wildcard",
|
||||||
|
qclass: 0,
|
||||||
|
qtype: 0,
|
||||||
|
rcode: 0,
|
||||||
|
response_time_us: 0,
|
||||||
|
route_kind: "blocked",
|
||||||
|
ts: 0,
|
||||||
|
upstream: "",
|
||||||
|
},
|
||||||
|
{
|
||||||
|
blocked: false,
|
||||||
|
cache_hit: false,
|
||||||
|
client_ip: "192.0.2.10",
|
||||||
|
domain: "news.example",
|
||||||
|
id: 0,
|
||||||
|
policy_action: "allow",
|
||||||
|
policy_reason: "no_match",
|
||||||
|
qclass: 0,
|
||||||
|
qtype: 0,
|
||||||
|
rcode: 0,
|
||||||
|
response_time_us: 0,
|
||||||
|
route_kind: "upstream",
|
||||||
|
ts: 0,
|
||||||
|
upstream: "https://dns.example/dns-query",
|
||||||
|
},
|
||||||
|
{
|
||||||
blocked: false,
|
blocked: false,
|
||||||
cache_hit: true,
|
cache_hit: true,
|
||||||
client_ip: "192.0.2.10",
|
client_ip: "192.0.2.10",
|
||||||
domain: "d24.example",
|
domain: "d24.example",
|
||||||
id: 0,
|
id: 0,
|
||||||
|
policy_action: "allow",
|
||||||
|
policy_reason: "no_match",
|
||||||
|
qclass: 0,
|
||||||
qtype: 0,
|
qtype: 0,
|
||||||
|
rcode: 0,
|
||||||
response_time_us: 0,
|
response_time_us: 0,
|
||||||
|
route_kind: "cache",
|
||||||
ts: 0,
|
ts: 0,
|
||||||
upstream: "https://dns.example/dns-query",
|
upstream: "",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
block_reason: "",
|
|
||||||
blocked: false,
|
blocked: false,
|
||||||
cache_hit: false,
|
cache_hit: false,
|
||||||
client_ip: "192.0.2.10",
|
client_ip: "192.0.2.10",
|
||||||
domain: "d23.example",
|
domain: "d23.example",
|
||||||
id: 0,
|
id: 0,
|
||||||
|
policy_action: "allow",
|
||||||
|
policy_reason: "no_match",
|
||||||
|
qclass: 0,
|
||||||
qtype: 0,
|
qtype: 0,
|
||||||
|
rcode: 0,
|
||||||
response_time_us: 0,
|
response_time_us: 0,
|
||||||
|
route_kind: "upstream",
|
||||||
ts: 0,
|
ts: 0,
|
||||||
upstream: "https://dns.example/dns-query",
|
upstream: "https://dns.example/dns-query",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
block_reason: "",
|
|
||||||
blocked: false,
|
blocked: false,
|
||||||
cache_hit: true,
|
cache_hit: true,
|
||||||
client_ip: "192.0.2.10",
|
client_ip: "192.0.2.10",
|
||||||
domain: "d22.example",
|
domain: "d22.example",
|
||||||
id: 0,
|
id: 0,
|
||||||
|
policy_action: "allow",
|
||||||
|
policy_reason: "no_match",
|
||||||
|
qclass: 0,
|
||||||
qtype: 0,
|
qtype: 0,
|
||||||
|
rcode: 0,
|
||||||
response_time_us: 0,
|
response_time_us: 0,
|
||||||
ts: 0,
|
route_kind: "cache",
|
||||||
upstream: "https://dns.example/dns-query",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
block_reason: "",
|
|
||||||
blocked: false,
|
|
||||||
cache_hit: false,
|
|
||||||
client_ip: "192.0.2.10",
|
|
||||||
domain: "d21.example",
|
|
||||||
id: 0,
|
|
||||||
qtype: 0,
|
|
||||||
response_time_us: 0,
|
|
||||||
ts: 0,
|
|
||||||
upstream: "https://dns.example/dns-query",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
block_reason: "blocklist_domain",
|
|
||||||
blocked: true,
|
|
||||||
cache_hit: null,
|
|
||||||
client_ip: "192.0.2.10",
|
|
||||||
domain: "d20.example",
|
|
||||||
id: 0,
|
|
||||||
qtype: 0,
|
|
||||||
response_time_us: 0,
|
|
||||||
ts: 0,
|
ts: 0,
|
||||||
upstream: "",
|
upstream: "",
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
};
|
};
|
||||||
|
|
||||||
|
export const sample_get_query_detail: QueryDetail = {
|
||||||
|
group: {
|
||||||
|
id: 0,
|
||||||
|
name: "kids",
|
||||||
|
},
|
||||||
|
id: 0,
|
||||||
|
policy: {
|
||||||
|
action: "block",
|
||||||
|
matched: "||tracker.example^",
|
||||||
|
reason: "blocklist_wildcard",
|
||||||
|
source_id: 0,
|
||||||
|
source_name: "StevenBlack",
|
||||||
|
},
|
||||||
|
request: {
|
||||||
|
client: "192.0.2.11",
|
||||||
|
domain: "shop.example",
|
||||||
|
qclass: 0,
|
||||||
|
qtype: 0,
|
||||||
|
time: 0,
|
||||||
|
},
|
||||||
|
response: {
|
||||||
|
duration_us: 0,
|
||||||
|
rcode: 0,
|
||||||
|
},
|
||||||
|
rewrites: {
|
||||||
|
cname_target: "cdn.tracker.example",
|
||||||
|
safe_search_target: "",
|
||||||
|
},
|
||||||
|
route: {
|
||||||
|
forward_zone: "",
|
||||||
|
kind: "blocked",
|
||||||
|
upstream: "",
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
export const sample_get_stats: StatsTotals = {
|
export const sample_get_stats: StatsTotals = {
|
||||||
avg_response_time_us: null,
|
avg_response_time_us: null,
|
||||||
blocked: 0,
|
blocked: 0,
|
||||||
cached: 0,
|
cached: 0,
|
||||||
clients: 0,
|
clients: 0,
|
||||||
|
coverage: {
|
||||||
|
available_since: 0,
|
||||||
|
complete: true,
|
||||||
|
},
|
||||||
period: "1h",
|
period: "1h",
|
||||||
queries: 0,
|
queries: 0,
|
||||||
since: 0,
|
since: 0,
|
||||||
@@ -501,6 +565,10 @@ export const sample_get_stats_timeseries: StatsTimeseries = {
|
|||||||
ts: 0,
|
ts: 0,
|
||||||
},
|
},
|
||||||
],
|
],
|
||||||
|
coverage: {
|
||||||
|
available_since: 0,
|
||||||
|
complete: true,
|
||||||
|
},
|
||||||
period: "1h",
|
period: "1h",
|
||||||
since: 0,
|
since: 0,
|
||||||
until: 0,
|
until: 0,
|
||||||
|
|||||||
@@ -25,6 +25,7 @@ export const queryKeys = {
|
|||||||
stats: (period: Period) => ["stats", period] as const,
|
stats: (period: Period) => ["stats", period] as const,
|
||||||
timeseries: (period: Period) => ["stats", "timeseries", period] as const,
|
timeseries: (period: Period) => ["stats", "timeseries", period] as const,
|
||||||
queriesInfinite: (filter: QueriesFilter) => ["queries", "infinite", filter] as const,
|
queriesInfinite: (filter: QueriesFilter) => ["queries", "infinite", filter] as const,
|
||||||
|
queryDetail: (id: number) => ["queries", "detail", id] as const,
|
||||||
diagnosticsInfinite: (filter: DiagnosticsFilter) => ["diagnostics", "infinite", filter] as const,
|
diagnosticsInfinite: (filter: DiagnosticsFilter) => ["diagnostics", "infinite", filter] as const,
|
||||||
diagnostic: (id: number) => ["diagnostics", "event", id] as const,
|
diagnostic: (id: number) => ["diagnostics", "event", id] as const,
|
||||||
/** Prefix of every diagnostics entry, page and detail alike; the purge target. */
|
/** Prefix of every diagnostics entry, page and detail alike; the purge target. */
|
||||||
@@ -75,6 +76,9 @@ export const queriesInfiniteQuery = (filter: QueriesFilter = {}) =>
|
|||||||
placeholderData: keepPreviousData,
|
placeholderData: keepPreviousData,
|
||||||
});
|
});
|
||||||
|
|
||||||
|
export const queryDetailQuery = (id: number) =>
|
||||||
|
queryOptions({ queryKey: queryKeys.queryDetail(id), queryFn: () => api.getQueryDetail(id) });
|
||||||
|
|
||||||
// Keyset pagination on `next_before`, exactly as the query log pages
|
// Keyset pagination on `next_before`, exactly as the query log pages
|
||||||
// (handlers/diagnostics.zig copies the /api/queries contract). The active view
|
// (handlers/diagnostics.zig copies the /api/queries contract). The active view
|
||||||
// polls on healthQuery's cadence because an episode opening is the same news a
|
// polls on healthQuery's cadence because an episode opening is the same news a
|
||||||
|
|||||||
+113
-3
@@ -60,25 +60,133 @@ export interface LogoutResponse {
|
|||||||
authenticated: false;
|
authenticated: false;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* The three closed enums `src/storage/provenance.zig` stores, as values rather
|
||||||
|
* than bare types: the copy maps in `features/queries/provenanceCopy.ts` have to
|
||||||
|
* be proven exhaustive at runtime as well as by `tsc`, exactly as
|
||||||
|
* `DIAGNOSTIC_CODES` below.
|
||||||
|
*/
|
||||||
|
export const POLICY_ACTIONS = ["not_evaluated", "allow", "block"] as const;
|
||||||
|
export type PolicyAction = (typeof POLICY_ACTIONS)[number];
|
||||||
|
|
||||||
|
export const POLICY_REASONS = [
|
||||||
|
"rule_allow_exact",
|
||||||
|
"rule_block_exact",
|
||||||
|
"rule_allow_wildcard",
|
||||||
|
"rule_block_wildcard",
|
||||||
|
"rule_allow_regex",
|
||||||
|
"rule_block_regex",
|
||||||
|
"blocklist_exception",
|
||||||
|
"blocklist_domain",
|
||||||
|
"blocklist_wildcard",
|
||||||
|
"local_record",
|
||||||
|
"forward_zone",
|
||||||
|
"non_in_class",
|
||||||
|
"paused",
|
||||||
|
"snapshot_unavailable",
|
||||||
|
"no_match",
|
||||||
|
"protocol_error",
|
||||||
|
] as const;
|
||||||
|
export type PolicyReason = (typeof POLICY_REASONS)[number];
|
||||||
|
|
||||||
|
export const ROUTE_KINDS = ["blocked", "local", "forward_zone", "upstream", "cache", "rejected"] as const;
|
||||||
|
export type RouteKind = (typeof ROUTE_KINDS)[number];
|
||||||
|
|
||||||
|
/** The summary projection the query-log table scans; full provenance is at `/api/queries/{id}`. */
|
||||||
export interface QueryRow {
|
export interface QueryRow {
|
||||||
id: number;
|
id: number;
|
||||||
ts: number;
|
ts: number;
|
||||||
domain: string;
|
domain: string;
|
||||||
client_ip: string;
|
client_ip: string;
|
||||||
qtype: number | null;
|
qtype: number | null;
|
||||||
|
qclass: number;
|
||||||
|
rcode: number;
|
||||||
blocked: boolean;
|
blocked: boolean;
|
||||||
block_reason: string;
|
|
||||||
response_time_us: number | null;
|
response_time_us: number | null;
|
||||||
cache_hit: boolean | null;
|
cache_hit: boolean | null;
|
||||||
upstream: string;
|
upstream: string;
|
||||||
|
policy_action: PolicyAction;
|
||||||
|
policy_reason: PolicyReason;
|
||||||
|
route_kind: RouteKind;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** SSE `event: query` payload: a QueryRow minus `id` (precedes persistence). */
|
export interface ProvenanceRequest {
|
||||||
export type LiveQueryEvent = Omit<QueryRow, "id">;
|
time: number;
|
||||||
|
domain: string;
|
||||||
|
client: string;
|
||||||
|
qtype: number | null;
|
||||||
|
qclass: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** The group as a historical fact: the id may name a group since renamed or deleted. */
|
||||||
|
export interface ProvenanceGroup {
|
||||||
|
id: number | null;
|
||||||
|
name: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProvenancePolicy {
|
||||||
|
action: PolicyAction;
|
||||||
|
reason: PolicyReason;
|
||||||
|
matched: string;
|
||||||
|
source_id: number | null;
|
||||||
|
source_name: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProvenanceRewrites {
|
||||||
|
cname_target: string;
|
||||||
|
safe_search_target: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProvenanceRoute {
|
||||||
|
kind: RouteKind;
|
||||||
|
forward_zone: string;
|
||||||
|
/** Non-empty only for an attempted upstream or forward-zone exchange; already redacted. */
|
||||||
|
upstream: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ProvenanceResponse {
|
||||||
|
/** The twelve-bit EDNS extended code, not the four header bits alone. */
|
||||||
|
rcode: number;
|
||||||
|
duration_us: number | null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* One query, fully explained, in the order a query meets the pipeline
|
||||||
|
* (`src/web/provenance_view.zig`). Every text field follows the repository
|
||||||
|
* convention: `""` means absent.
|
||||||
|
*/
|
||||||
|
export interface Provenance {
|
||||||
|
request: ProvenanceRequest;
|
||||||
|
group: ProvenanceGroup;
|
||||||
|
policy: ProvenancePolicy;
|
||||||
|
rewrites: ProvenanceRewrites;
|
||||||
|
route: ProvenanceRoute;
|
||||||
|
response: ProvenanceResponse;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `GET /api/queries/{id}`: the same six groups plus the row id. */
|
||||||
|
export interface QueryDetail extends Provenance {
|
||||||
|
id: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** SSE `event: query` payload: the full provenance, minus an id it cannot have yet. */
|
||||||
|
export type LiveQueryEvent = Provenance;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* How much of the requested window the query log can answer for. Retention
|
||||||
|
* deletes rows and advances the watermark in one transaction, so an empty
|
||||||
|
* window is distinguishable from a pruned one.
|
||||||
|
*/
|
||||||
|
export interface Coverage {
|
||||||
|
complete: boolean;
|
||||||
|
/** The oldest instant the log is complete for, unix seconds. */
|
||||||
|
available_since: number;
|
||||||
|
}
|
||||||
|
|
||||||
export interface QueriesPage {
|
export interface QueriesPage {
|
||||||
queries: QueryRow[];
|
queries: QueryRow[];
|
||||||
next_before: number | null;
|
next_before: number | null;
|
||||||
|
coverage: Coverage;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface QueriesFilter {
|
export interface QueriesFilter {
|
||||||
@@ -171,6 +279,7 @@ export interface StatsTotals {
|
|||||||
cached: number;
|
cached: number;
|
||||||
clients: number;
|
clients: number;
|
||||||
avg_response_time_us: number | null;
|
avg_response_time_us: number | null;
|
||||||
|
coverage: Coverage;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface Bucket {
|
export interface Bucket {
|
||||||
@@ -186,6 +295,7 @@ export interface StatsTimeseries {
|
|||||||
until: number;
|
until: number;
|
||||||
bucket_seconds: number;
|
bucket_seconds: number;
|
||||||
buckets: Bucket[];
|
buckets: Bucket[];
|
||||||
|
coverage: Coverage;
|
||||||
}
|
}
|
||||||
|
|
||||||
export interface LookupResult {
|
export interface LookupResult {
|
||||||
|
|||||||
+39
-2
@@ -12,7 +12,7 @@ import {
|
|||||||
import AppShell from "@/shell/AppShell";
|
import AppShell from "@/shell/AppShell";
|
||||||
import { ApiError } from "@/lib/api";
|
import { ApiError } from "@/lib/api";
|
||||||
import { createQueryClient } from "@/lib/queryClient";
|
import { createQueryClient } from "@/lib/queryClient";
|
||||||
import type { DiagnosticSeverity, DiagnosticState, DiagnosticsFilter } from "@/lib/types";
|
import type { DiagnosticSeverity, DiagnosticState, DiagnosticsFilter, QueriesFilter } from "@/lib/types";
|
||||||
import {
|
import {
|
||||||
blocklistsQuery,
|
blocklistsQuery,
|
||||||
clientPrefixesQuery,
|
clientPrefixesQuery,
|
||||||
@@ -24,6 +24,7 @@ import {
|
|||||||
healthQuery,
|
healthQuery,
|
||||||
localRecordsQuery,
|
localRecordsQuery,
|
||||||
queriesInfiniteQuery,
|
queriesInfiniteQuery,
|
||||||
|
queryDetailQuery,
|
||||||
rulesQuery,
|
rulesQuery,
|
||||||
settingsQuery,
|
settingsQuery,
|
||||||
statsQuery,
|
statsQuery,
|
||||||
@@ -138,13 +139,43 @@ const dashboardRoute = createRoute({
|
|||||||
component: lazyRouteComponent(() => import("@/features/dashboard/DashboardPage")),
|
component: lazyRouteComponent(() => import("@/features/dashboard/DashboardPage")),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `domain` and `client` seed the filter form, so a detail page can link to
|
||||||
|
* "every query for this domain". Anything else in the search object is dropped:
|
||||||
|
* an unknown value would reach the api as a parameter the handler 400s.
|
||||||
|
*/
|
||||||
const queriesRoute = createRoute({
|
const queriesRoute = createRoute({
|
||||||
getParentRoute: () => shellRoute,
|
getParentRoute: () => shellRoute,
|
||||||
path: "/queries",
|
path: "/queries",
|
||||||
loader: ({ context }) => context.queryClient.ensureInfiniteQueryData(queriesInfiniteQuery({})),
|
validateSearch: (search: Record<string, unknown>): { domain?: string; client?: string } => {
|
||||||
|
const domain = search["domain"];
|
||||||
|
const client = search["client"];
|
||||||
|
return {
|
||||||
|
domain: typeof domain === "string" && domain !== "" ? domain : undefined,
|
||||||
|
client: typeof client === "string" && client !== "" ? client : undefined,
|
||||||
|
};
|
||||||
|
},
|
||||||
|
loaderDeps: ({ search }) => search,
|
||||||
|
loader: ({ context, deps }) => {
|
||||||
|
const filter: QueriesFilter = {};
|
||||||
|
if (deps.domain !== undefined) filter.domain = deps.domain;
|
||||||
|
if (deps.client !== undefined) filter.client = deps.client;
|
||||||
|
return context.queryClient.ensureInfiniteQueryData(queriesInfiniteQuery(filter));
|
||||||
|
},
|
||||||
component: lazyRouteComponent(() => import("@/features/queries/QueryLogPage")),
|
component: lazyRouteComponent(() => import("@/features/queries/QueryLogPage")),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const queryDetailRoute = createRoute({
|
||||||
|
getParentRoute: () => shellRoute,
|
||||||
|
path: "/queries/$id",
|
||||||
|
// Swallowed on purpose, as the diagnostics detail route does: a row
|
||||||
|
// retention has pruned is a 404 the page explains, with the way back to the
|
||||||
|
// log. The whole-page error component would call it a request failure.
|
||||||
|
loader: ({ context, params }) =>
|
||||||
|
context.queryClient.ensureQueryData(queryDetailQuery(Number(params.id))).catch(() => undefined),
|
||||||
|
component: lazyRouteComponent(() => import("@/features/queries/QueryDetailPage")),
|
||||||
|
});
|
||||||
|
|
||||||
const liveRoute = createRoute({
|
const liveRoute = createRoute({
|
||||||
getParentRoute: () => shellRoute,
|
getParentRoute: () => shellRoute,
|
||||||
path: "/live",
|
path: "/live",
|
||||||
@@ -210,9 +241,14 @@ const upstreamsRoute = createRoute({
|
|||||||
component: lazyRouteComponent(() => import("@/features/upstreams/UpstreamsPage")),
|
component: lazyRouteComponent(() => import("@/features/upstreams/UpstreamsPage")),
|
||||||
});
|
});
|
||||||
|
|
||||||
|
/** `domain` prefills and runs the lookup, so a query detail page can link into it. */
|
||||||
const lookupRoute = createRoute({
|
const lookupRoute = createRoute({
|
||||||
getParentRoute: () => shellRoute,
|
getParentRoute: () => shellRoute,
|
||||||
path: "/lookup",
|
path: "/lookup",
|
||||||
|
validateSearch: (search: Record<string, unknown>): { domain?: string } => {
|
||||||
|
const domain = search["domain"];
|
||||||
|
return { domain: typeof domain === "string" && domain !== "" ? domain : undefined };
|
||||||
|
},
|
||||||
loader: ({ context }) => context.queryClient.ensureQueryData(groupsQuery()),
|
loader: ({ context }) => context.queryClient.ensureQueryData(groupsQuery()),
|
||||||
component: lazyRouteComponent(() => import("@/features/lookup/LookupPage")),
|
component: lazyRouteComponent(() => import("@/features/lookup/LookupPage")),
|
||||||
});
|
});
|
||||||
@@ -275,6 +311,7 @@ const routeTree = rootRoute.addChildren([
|
|||||||
shellRoute.addChildren([
|
shellRoute.addChildren([
|
||||||
dashboardRoute,
|
dashboardRoute,
|
||||||
queriesRoute,
|
queriesRoute,
|
||||||
|
queryDetailRoute,
|
||||||
liveRoute,
|
liveRoute,
|
||||||
clientsRoute,
|
clientsRoute,
|
||||||
groupsRoute,
|
groupsRoute,
|
||||||
|
|||||||
@@ -30,8 +30,16 @@ const RESPONSES: Record<string, unknown> = {
|
|||||||
cached: 0,
|
cached: 0,
|
||||||
clients: 0,
|
clients: 0,
|
||||||
avg_response_time_us: null,
|
avg_response_time_us: null,
|
||||||
|
coverage: { complete: true, available_since: 0 },
|
||||||
|
},
|
||||||
|
"/api/stats/timeseries?period=24h": {
|
||||||
|
period: "24h",
|
||||||
|
since: 0,
|
||||||
|
until: 86400,
|
||||||
|
bucket_seconds: 1800,
|
||||||
|
buckets: [],
|
||||||
|
coverage: { complete: true, available_since: 0 },
|
||||||
},
|
},
|
||||||
"/api/stats/timeseries?period=24h": { period: "24h", since: 0, until: 86400, bucket_seconds: 1800, buckets: [] },
|
|
||||||
"/api/health": {
|
"/api/health": {
|
||||||
status: "ok",
|
status: "ok",
|
||||||
disk: { state: "ok", free_bytes: 0, db_bytes: 0, log_bytes: 0, sample_failures: 0 },
|
disk: { state: "ok", free_bytes: 0, db_bytes: 0, log_bytes: 0, sample_failures: 0 },
|
||||||
|
|||||||
+27
-6
@@ -4,7 +4,7 @@ nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080), a
|
|||||||
|
|
||||||
The machine-readable contract is `src/web/openapi.yaml`, which the running server hands out unauthenticated at `GET /api/openapi.yaml`. Request and response schemas for every operation live there. When this page and the YAML disagree, the YAML wins.
|
The machine-readable contract is `src/web/openapi.yaml`, which the running server hands out unauthenticated at `GET /api/openapi.yaml`. Request and response schemas for every operation live there. When this page and the YAML disagree, the YAML wins.
|
||||||
|
|
||||||
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 60 of its entries.
|
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 61 of its entries.
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
@@ -45,7 +45,7 @@ A token bucket per client address: capacity and refill are both `web.api_rate_li
|
|||||||
`GET /api/queries/live` is server-sent events over chunked transfer, `Content-Type: text/event-stream`, `Cache-Control: no-store`.
|
`GET /api/queries/live` is server-sent events over chunked transfer, `Content-Type: text/event-stream`, `Cache-Control: no-store`.
|
||||||
|
|
||||||
- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on its own after a drop.
|
- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on its own after a drop.
|
||||||
- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload carries the `GET /api/queries` row fields minus `id` (a live entry precedes persistence): `ts`, `domain`, `client_ip`, `qtype`, `blocked`, `block_reason`, `response_time_us`, `cache_hit`, `upstream`.
|
- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload is the `Provenance` object — the body of `GET /api/queries/{id}` without its `id`, which does not exist yet because a live entry precedes its own insert. Its six groups are `request`, `group`, `policy`, `rewrites`, `route` and `response`.
|
||||||
- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping middleboxes from reaping the idle connection.
|
- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping middleboxes from reaping the idle connection.
|
||||||
- Each subscriber buffers up to 64 entries. A client too slow for the query rate overflows its buffer and the server ends the stream cleanly after delivering what the buffer held — queries are never held back for a slow reader. There is no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the missed rows.
|
- Each subscriber buffers up to 64 entries. A client too slow for the query rate overflows its buffer and the server ends the stream cleanly after delivering what the buffer held — queries are never held back for a slow reader. There is no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the missed rows.
|
||||||
- Connections per client address are capped at `web.sse_max_connections_per_ip` (default 3); over the cap is a 429. The cap binds loopback too. The server holds at most 32 concurrent streams in total; when all slots are taken, the answer is a 503.
|
- Connections per client address are capped at `web.sse_max_connections_per_ip` (default 3); over the cap is a 429. The cap binds loopback too. The server holds at most 32 concurrent streams in total; when all slots are taken, the answer is a 503.
|
||||||
@@ -104,6 +104,7 @@ Auth `open` means no session is required; `session` means a valid session cookie
|
|||||||
| POST | `/api/auth/login` | open | counted | runtime action | Log in |
|
| POST | `/api/auth/login` | open | counted | runtime action | Log in |
|
||||||
| POST | `/api/auth/logout` | session | counted | runtime action | Log out |
|
| POST | `/api/auth/logout` | session | counted | runtime action | Log out |
|
||||||
| GET | `/api/queries` | session | counted | read | Query log page |
|
| GET | `/api/queries` | session | counted | read | Query log page |
|
||||||
|
| GET | `/api/queries/{id}` | session | counted | read | One query, fully explained |
|
||||||
| GET | `/api/queries/live` | session | exempt | read | Live query stream (server-sent events) |
|
| GET | `/api/queries/live` | session | exempt | read | Live query stream (server-sent events) |
|
||||||
| GET | `/api/stats` | session | counted | read | Totals for a period |
|
| GET | `/api/stats` | session | counted | read | Totals for a period |
|
||||||
| GET | `/api/stats/timeseries` | session | counted | read | Bucketed counts for a period |
|
| GET | `/api/stats/timeseries` | session | counted | read | Bucketed counts for a period |
|
||||||
@@ -174,9 +175,11 @@ In file mode `PUT /api/settings` is refused with the 403 above, password changes
|
|||||||
|
|
||||||
Request and response schemas for every operation live in the OpenAPI document: `src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a running server.
|
Request and response schemas for every operation live in the OpenAPI document: `src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a running server.
|
||||||
|
|
||||||
### Block reasons
|
### Policy reasons
|
||||||
|
|
||||||
Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup` answer. The tag names the level that decided the query, and the levels are listed here in the order they are consulted — the first one that matches wins, so a rule always outranks a list.
|
Two places carry the same closed set of tags: `policy_reason` on a `GET /api/queries` row and on a `GET /api/queries/{id}` body (where it is `policy.reason`, and where the live stream sends the same field), and `reason` on a `GET /api/lookup` answer. The tag names what decided the query.
|
||||||
|
|
||||||
|
The first nine are the matcher's own verdicts, listed in the order they are consulted — the first that matches wins, so a rule always outranks a list.
|
||||||
|
|
||||||
| Tag | Decided by |
|
| Tag | Decided by |
|
||||||
| --- | --- |
|
| --- | --- |
|
||||||
@@ -190,6 +193,24 @@ Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `bl
|
|||||||
| `blocklist_domain` | A plain name in a downloaded list |
|
| `blocklist_domain` | A plain name in a downloaded list |
|
||||||
| `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list |
|
| `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list |
|
||||||
|
|
||||||
`/api/lookup` also answers `none` when nothing matched. A query row never carries `none`: `block_reason` is null unless the query was blocked.
|
The rest name a pipeline step that answered the query without consulting the matcher, and appear on a query row only.
|
||||||
|
|
||||||
A `cname:` prefix means the decision landed on a CNAME target rather than on the name the client asked for, so `cname:blocklist_domain` reads as "the list blocks a name this answer redirects to". Only `/api/queries` and the live stream show the prefix; `/api/lookup` does not follow CNAMEs.
|
| Tag | Decided by |
|
||||||
|
| --- | --- |
|
||||||
|
| `local_record` | A configured local record, answered before filtering |
|
||||||
|
| `forward_zone` | A configured forward zone, answered before filtering |
|
||||||
|
| `non_in_class` | The question was not class IN, so no rule could apply |
|
||||||
|
| `paused` | Filtering was paused |
|
||||||
|
| `snapshot_unavailable` | No filter snapshot was published yet, so the query went unfiltered |
|
||||||
|
| `no_match` | The matcher evaluated the name and nothing matched |
|
||||||
|
| `protocol_error` | A parsed request refused on protocol grounds — BADVERS, NOTIMP, a malformed EDNS OPT |
|
||||||
|
|
||||||
|
`policy_action` says which way the verdict went: `block`, `allow`, or `not_evaluated` for a query answered before any policy could apply. `/api/lookup` answers `none` when nothing matched, where a query row says `no_match`.
|
||||||
|
|
||||||
|
`route_kind` says where the answer came from: `blocked`, `local`, `forward_zone`, `upstream`, `cache` or `rejected`.
|
||||||
|
|
||||||
|
A non-empty `rewrites.cname_target` on a query detail means the decision landed on a CNAME target rather than on the name the client asked for; `policy.reason` is then the target's own reason. `/api/lookup` does not follow CNAMEs.
|
||||||
|
|
||||||
|
### Coverage
|
||||||
|
|
||||||
|
`GET /api/queries`, `GET /api/stats` and `GET /api/stats/timeseries` each answer with a `coverage` object: `available_since` is the oldest instant the query log is still complete for, and `complete` is true only when the window the request asked about starts at or after it. Retention deletes rows and advances the watermark in one transaction, so a client can tell an empty window from a pruned one instead of charting the gap as zero. A request with no lower bound at all asks about the whole of history, and is never complete.
|
||||||
|
|||||||
@@ -131,7 +131,7 @@ Process log and query log behavior.
|
|||||||
|---|---|---|---|---|---|
|
|---|---|---|---|---|---|
|
||||||
| `logging.level` | enum `.err` \| `.warn` \| `.info` \| `.debug` | `.info` | — | one of the four tags; stored as `"error"` / `"warn"` / `"info"` / `"debug"` | log threshold (`src/platform/logging.zig`) |
|
| `logging.level` | enum `.err` \| `.warn` \| `.info` \| `.debug` | `.info` | — | one of the four tags; stored as `"error"` / `"warn"` / `"info"` / `"debug"` | log threshold (`src/platform/logging.zig`) |
|
||||||
| `logging.retention_days` | u16 | 30 | days | at least 1 | query-log pruning cutoff (`src/storage/retention.zig`) and the client tracker's last-seen cutoff (`src/server/clients.zig`) |
|
| `logging.retention_days` | u16 | 30 | days | at least 1 | query-log pruning cutoff (`src/storage/retention.zig`) and the client tracker's last-seen cutoff (`src/server/clients.zig`) |
|
||||||
| `logging.query_log_buffer_max` | u32 | 10000 | entries | 1–1000000 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`) |
|
| `logging.query_log_buffer_max` | u32 | 10000 | entries | 1–37449 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`); the ceiling is derived at compile time from `@sizeOf(logger.Entry)` so the queue's worst case stays within 64 MiB, and it moves whenever the entry's width does |
|
||||||
| `logging.query_log_flush_interval_s` | u16 | 60 | seconds | 0–3600 | how long the query-log writer gathers entries before committing them in one transaction (`src/storage/logger.zig`); see the note below |
|
| `logging.query_log_flush_interval_s` | u16 | 60 | seconds | 0–3600 | how long the query-log writer gathers entries before committing them in one transaction (`src/storage/logger.zig`); see the note below |
|
||||||
| `logging.hide_domains` | bool | false | — | — | the query log stores a hidden marker instead of the domain |
|
| `logging.hide_domains` | bool | false | — | — | the query log stores a hidden marker instead of the domain |
|
||||||
| `logging.hide_client_ips` | bool | false | — | — | the query log stores a hidden marker instead of the client address |
|
| `logging.hide_client_ips` | bool | false | — | — | the query log stores a hidden marker instead of the client address |
|
||||||
@@ -385,10 +385,12 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
|
|||||||
| `MissingDefaultGroup` | no group is named `default` |
|
| `MissingDefaultGroup` | no group is named `default` |
|
||||||
| `DuplicateGroupName` | two groups share a `name` |
|
| `DuplicateGroupName` | two groups share a `name` |
|
||||||
| `EmptyGroupName` | a group `name` is empty |
|
| `EmptyGroupName` | a group `name` is empty |
|
||||||
|
| `GroupNameTooLong` | a group `name` is longer than 64 bytes; it is copied into every logged query |
|
||||||
| `UnknownGroup` | a client, prefix, group source or rule names a group that is not declared |
|
| `UnknownGroup` | a client, prefix, group source or rule names a group that is not declared |
|
||||||
| `BadClientIp` / `DuplicateClientIp` | a client `ip` is unparseable, or collides after canonicalization |
|
| `BadClientIp` / `DuplicateClientIp` | a client `ip` is unparseable, or collides after canonicalization |
|
||||||
| `BadClientPrefix` / `DuplicateClientPrefix` | the same for a `client_prefixes.prefix` |
|
| `BadClientPrefix` / `DuplicateClientPrefix` | the same for a `client_prefixes.prefix` |
|
||||||
| `BadSourceUrl` / `DuplicateSourceUrl` / `EmptySourceName` | blocklist source fields |
|
| `BadSourceUrl` / `DuplicateSourceUrl` / `EmptySourceName` | blocklist source fields |
|
||||||
|
| `SourceNameTooLong` | a blocklist source `name` is longer than 64 bytes; it is copied into every logged query |
|
||||||
| `UnknownSource` / `DuplicateGroupSource` | `group_sources` links |
|
| `UnknownSource` / `DuplicateGroupSource` | `group_sources` links |
|
||||||
| `BadRulePattern` | a rule `pattern` does not match its `kind` |
|
| `BadRulePattern` | a rule `pattern` does not match its `kind` |
|
||||||
| `BadLocalRecordName` / `BadLocalRecordValue` / `DuplicateLocalRecord` | local record fields |
|
| `BadLocalRecordName` / `BadLocalRecordValue` / `DuplicateLocalRecord` | local record fields |
|
||||||
@@ -397,7 +399,7 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
|
|||||||
| `BadTimeout` | a timeout is outside 100–120000 ms, or `attempt` is above `total` |
|
| `BadTimeout` | a timeout is outside 100–120000 ms, or `attempt` is above `total` |
|
||||||
| `BadTtl` | `blocking.ttl`, `cache.negative_ttl_max`, a record `ttl`, `web.session_ttl_hours` or `blocklist_update.interval_hours` outside its range |
|
| `BadTtl` | `blocking.ttl`, `cache.negative_ttl_max`, a record `ttl`, `web.session_ttl_hours` or `blocklist_update.interval_hours` outside its range |
|
||||||
| `BadCacheSize` | `cache.size` outside 1–1000000 |
|
| `BadCacheSize` | `cache.size` outside 1–1000000 |
|
||||||
| `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 1–1000000 |
|
| `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 1–37449 |
|
||||||
| `BadFlushInterval` | `logging.query_log_flush_interval_s` above 3600 |
|
| `BadFlushInterval` | `logging.query_log_flush_interval_s` above 3600 |
|
||||||
| `BadLogRotation` | `logging.max_size_mb` or `logging.max_files` below 1 |
|
| `BadLogRotation` | `logging.max_size_mb` or `logging.max_files` below 1 |
|
||||||
| `BadDiskThresholds` | a threshold below 1, or `min_free_mb` above `warn_free_mb` |
|
| `BadDiskThresholds` | a threshold below 1, or `min_free_mb` above `warn_free_mb` |
|
||||||
|
|||||||
@@ -0,0 +1,210 @@
|
|||||||
|
# Milestone 28: query provenance
|
||||||
|
|
||||||
|
Redesign step 2 of specs/ui-redesign.md ("Query provenance", "Schema changes", "Entry buffer widths", "API changes", rulings). Every logged query becomes exactly explainable: what policy decided, what matched, where the answer came from, and what the client saw. The existing Query Log/Live/Lookup pages keep working; their replacement is step 3 (milestone 29). Codex spec review folded in (thread 01a02643); its corrections are marked where they changed a ruling.
|
||||||
|
|
||||||
|
**This milestone destroys existing query history.** The DDL edit changes the CRC fingerprint (querylog_schema.zig:71-78), so `open` recreates the file and sets the old one aside as `querylog.db.schema-changed-<unix seconds>`. Acceptable pre-v0.1. The changelog entry must say so, and the recreate is the natural first `query_log.recreated` diagnostics emission.
|
||||||
|
|
||||||
|
## Sessions
|
||||||
|
|
||||||
|
S1 (storage) and S2 (upstream identity) run in parallel — disjoint files. S3 (handler capture) needs both. S4 (web API + contracts) needs S3. S5 (admin) needs S4.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session S1: querylog schema, repo, logger, coverage watermark
|
||||||
|
|
||||||
|
### S1.1 DDL (src/storage/querylog_schema.zig)
|
||||||
|
|
||||||
|
`query_log` drops `block_reason` and gains, after the existing columns:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
qclass INTEGER NOT NULL,
|
||||||
|
rcode INTEGER NOT NULL,
|
||||||
|
group_id INTEGER,
|
||||||
|
group_name TEXT,
|
||||||
|
policy_action TEXT NOT NULL,
|
||||||
|
policy_reason TEXT NOT NULL,
|
||||||
|
matched TEXT,
|
||||||
|
source_id INTEGER,
|
||||||
|
source_name TEXT,
|
||||||
|
cname_target TEXT,
|
||||||
|
safe_search_target TEXT,
|
||||||
|
route_kind TEXT NOT NULL,
|
||||||
|
forward_zone TEXT
|
||||||
|
```
|
||||||
|
|
||||||
|
Existing columns (`blocked`, `cache_hit`, `upstream`, …) stay — stats and the step-3 filters still use them. No new index (ruling: `idx_query_log_ts` bounds every time-scoped question; the insert path pays for indexes).
|
||||||
|
|
||||||
|
New singleton table, in the same DDL string (Codex: enforce one row):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE querylog_meta (
|
||||||
|
id INTEGER PRIMARY KEY CHECK (id = 1),
|
||||||
|
created_at INTEGER NOT NULL,
|
||||||
|
available_since INTEGER NOT NULL
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
`createFresh` inserts the row with `created_at = now` and `available_since = now + 1` (conservative: an old row logged in the same second as recreation must not let `since = now` claim completeness). **`available_since` is a monotonic coverage watermark, not a constant** (Codex): pruning advances it to the retention cutoff **in the same transaction as the delete** — the repo exposes one transactional `pruneOlderThan(cutoff)` that deletes and advances together; a failure of either rolls back both. It never moves backward. `queries_repo` gains `availableSince() i64`.
|
||||||
|
|
||||||
|
The object-count test (querylog_schema.zig:295-318) updates to 5 tables. TEXT id/name pairs are deliberate (ruling: `client_ip` precedent, "log rows are immutable facts"); ids are NOT foreign keys.
|
||||||
|
|
||||||
|
### S1.2 Closed enums — neutral module `src/storage/provenance.zig` (Codex: logger already imports queries_repo, so enums owned by logger would cycle)
|
||||||
|
|
||||||
|
Imported by logger, queries_repo, handler and web code. Stored as `@tagName` TEXT; the read path parses text back to the enum and treats an unknown value as a data error, not a passthrough string.
|
||||||
|
|
||||||
|
- `PolicyAction`: `not_evaluated`, `allow`, `block`.
|
||||||
|
- `PolicyReason`: the nine serializable `matcher.Reason` tags (matcher.zig:24-37 minus `none`) plus `local_record`, `forward_zone`, `non_in_class`, `paused`, `snapshot_unavailable`, `no_match`, `protocol_error`. Matcher→PolicyReason conversion is an exhaustive switch. The CNAME case is NOT a `cname:` string prefix any more: `cname_target` non-NULL carries that fact, and `policy_reason` holds the target's own reason.
|
||||||
|
- `RouteKind`: `blocked`, `local`, `forward_zone`, `upstream`, `cache`, `rejected`.
|
||||||
|
|
||||||
|
`protocol_error` + `rejected` cover post-parse protocol refusals (BADVERS, NOTIMP, EDNS FORMERR) — see S3.4; this extends the redesign's enum list, recorded here as an amendment required by its own "every syntactically parsed request that receives a response is logged" rule.
|
||||||
|
|
||||||
|
`blockReason()`, `cname_reason_prefix` and the comptime width proof (handler.zig:66-74, 816-823) die in S3.
|
||||||
|
|
||||||
|
### S1.3 Entry widening (src/storage/logger.zig)
|
||||||
|
|
||||||
|
`Entry` stays a by-value fixed-buffer struct through `Io.Queue`. New fields with explicit widths:
|
||||||
|
|
||||||
|
| field | width / type |
|
||||||
|
| --- | --- |
|
||||||
|
| `qclass` | `u16` |
|
||||||
|
| `rcode` | `u16`, validated ≤ 0xFFF (12-bit EDNS extended RCODE, edns.zig:219; Codex: u8 cannot hold it) |
|
||||||
|
| `group_id` | `?i64` |
|
||||||
|
| `group_name` | buffer sized by the new `max_group_name_len` (S1.5) |
|
||||||
|
| `policy_action` | `provenance.PolicyAction` |
|
||||||
|
| `policy_reason` | `provenance.PolicyReason` |
|
||||||
|
| `matched` | buffer sized by the rule maximum — the regex engine accepts 256-byte patterns (regex.zig:48), so 256 bytes with a **`u16` length** (the generic `copyInto` returns `u8`; widen it or add a u16 variant — 256 does not fit u8) |
|
||||||
|
| `source_id` | `?i64` |
|
||||||
|
| `source_name` | buffer sized by the new `max_source_name_len` (S1.5) |
|
||||||
|
| `cname_target` | 253-byte buffer |
|
||||||
|
| `safe_search_target` | 253-byte buffer |
|
||||||
|
| `forward_zone` | 253-byte buffer |
|
||||||
|
| `upstream` | widened: sized for the maximum redacted `scheme://host:port` form (host up to 253 bytes), `u16` length — the current 64-byte buffer silently truncates a long valid DoH hostname (Codex); the endpoint host bound gets validated where endpoints are parsed |
|
||||||
|
|
||||||
|
`Entry.Fields` gains the borrowed equivalents with `""`/null defaults. `reason_buf`/`max_reason_len` are removed with `block_reason`. The truncation test (logger.zig:610-625) updates.
|
||||||
|
|
||||||
|
`transformed()` (logger.zig:234-239) additionally rewrites `matched`, `cname_target`, `safe_search_target` to `hidden_marker` under `hide_domains`. `forward_zone`, `group_name`, `source_name` are configuration labels, not query-derived, and stay visible.
|
||||||
|
|
||||||
|
### S1.4 Entry memory budget (Codex: validation permits 1,000,000 queued entries, validate.zig:464, and sse.zig:54 embeds 2,048 entries)
|
||||||
|
|
||||||
|
The widened `@sizeOf(Entry)` gets a documented byte budget: `query_log_buffer_max`'s validation upper bound is recomputed so the queue's worst case stays ≤ 64 MiB (`max = 64 MiB / @sizeOf(Entry)`, computed at comptime, stated in the validation reference and CHANGELOG since the accepted range shrinks). Tests assert the default and the new maximum fit the budget, and the SSE hub comment states its embedded-entry cost.
|
||||||
|
|
||||||
|
### S1.5 Name caps — neutral module `src/config/limits.zig` (Codex: logger→validate for caps plus validate→logger for `@sizeOf(Entry)` is a cycle)
|
||||||
|
|
||||||
|
No length cap exists today for group or source names (validate.zig:721, :853 reject only empty). New `config/limits.zig` owns `max_group_name_len = 64` and `max_source_name_len = 64`; validate.zig enforces them (boundary tests at 64 and 65 bytes, new classification entries per the existing pattern); logger.zig sizes its buffers from them and **exports the computed `query_log_buffer_max` ceiling** (S1.4), which validate.zig imports.
|
||||||
|
|
||||||
|
### S1.6 Repo (src/storage/repositories/queries_repo.zig)
|
||||||
|
|
||||||
|
- `Row`/`insert_row_sql`/`BatchWriter` bind the new columns; `Sql.capacity` (:262-265) updated.
|
||||||
|
- `QueryRow` (read) gains `qclass: u16`, `rcode: u16`, `route_kind`, `policy_action`, `policy_reason` (parsed enums, serialized as strings); loses `block_reason`. NULL→`""` convention unchanged for text.
|
||||||
|
- New `detailById(id) ?QueryDetail`: full provenance row joined with domains, for `GET /api/queries/{id}`.
|
||||||
|
- `availableSince()` and the transactional prune-plus-advance per S1.1; retention.zig calls the combined operation.
|
||||||
|
- `stats_totals_sql`/`timeseries_sql` unchanged.
|
||||||
|
|
||||||
|
### S1.7 `query_log.recreated` emission
|
||||||
|
|
||||||
|
The existing emission site (app.zig:638) already fires on recreate; it gains the **initial coverage start** (the fresh `available_since`) in its detail alongside reason and aside filename (ui-redesign.md:252 requires the new coverage start). Verify open order (events store vs querylog open) and carry the `OpenResult` rather than reordering database opens if needed.
|
||||||
|
|
||||||
|
### S1.8 Acceptance (S1)
|
||||||
|
|
||||||
|
- [ ] Fingerprint tests updated; recreate test proves aside name, `querylog_meta` singleton row, and the recreated-event detail carrying the coverage start (end-to-end recreate→coverage test).
|
||||||
|
- [ ] Watermark tests: prune advances `available_since` to the cutoff atomically; a failed delete, a failed watermark update, and a failed commit each leave both untouched (rollback proven); it never regresses.
|
||||||
|
- [ ] Round-trip test: an `Entry` with every provenance field set survives queue → `toRow` → insert → `detailById` intact (modulo NULL mapping); includes a 256-byte `matched` boundary case.
|
||||||
|
- [ ] `transformed()` tests split by flag (Codex: the client is governed by `hide_client_ips`, not `hide_domains`): `hide_domains` hides domain, matched, cname_target, safe_search_target and preserves client_ip; `hide_client_ips` hides client_ip and preserves the rest; both flags leave group/source/zone names.
|
||||||
|
- [ ] Buffer-budget tests per S1.4; name-cap boundary tests per S1.5.
|
||||||
|
- [ ] `zig build test` 0 failed.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Session S2: selected-resolver identity (every `transport.Client` implementor)
|
||||||
|
|
||||||
|
Reverses ruling 20 for the exchange that actually happened. `transport.Client` (transport.zig:325-343) gains a per-call out-parameter: `exchangeFn(ptr, io, query, response_buf, selected: *?[]const u8)`.
|
||||||
|
|
||||||
|
**Contract (Codex critical): the identity survives failure.** Each implementation sets `selected.*` to the resolver it is about to attempt, before the attempt; after an all-failed exchange it names the last attempted resolver. The handler consumes it on success AND on the error path — a SERVFAIL row carrying its resolver is the single most useful correlation this redesign adds (ui-redesign.md:157). The slice must stay valid for the query's duration: `Pool` uses `entry.endpoint.url` (Endpoint-owned, stable); the pointer is per-call, threaded into `exchangeLoopLen` (pool.zig:203-266) — never a `*Pool` field (concurrent queries race).
|
||||||
|
|
||||||
|
`Pool` reports the **raw** URL; redaction happens in the handler when formatting into the Entry (S3) — `safe_url.redact` is a formatter, so the pool test asserts the raw selected identity and the credential end-to-end test lives in S3/S4 (Codex).
|
||||||
|
|
||||||
|
Callers initialize the output to null before the call. `ForwardClient` (forward_client.zig) holds only a parsed `Resolver`, so it gains **owned identity storage** — the formatted resolver text lives in the client and the out-parameter borrows it (Codex: this is real behavior, not a mechanical discard; the "discards only" framing was wrong). Fakes set explicit stable identities.
|
||||||
|
|
||||||
|
S2 owns every implementor, fake and call site: pool.zig, forward_client.zig, transport.zig fakes (:635-660), doh_client_live_test.zig:47 and dot_client_live_test.zig:60 (imported by tests.zig — they break compilation if missed, Codex), handler.zig's inline fake (:1074) and its call site (:457, pass-and-ignore — S3 consumes it), doh_server.zig, dot_server.zig, udp/tcp/resolver/phase7 integration tests, web_integration_test.zig. This overlaps S1 on zero files; the handler/web edits are signature-level and complete before S3/S4 start (sequential).
|
||||||
|
|
||||||
|
- [ ] Pool tests: winning endpoint reported; failover reports the answerer, not the first attempt; all-failed reports the last attempted; a timeout mid-flight reports the in-flight resolver.
|
||||||
|
- [ ] `zig build test` 0 failed.
|
||||||
|
|
||||||
|
## Session S3: handler capture (src/server/handler.zig + server tests)
|
||||||
|
|
||||||
|
After S1+S2. All provenance assembled in `Context` and passed through `LogFields` → `Entry.Fields`:
|
||||||
|
|
||||||
|
- `qclass` from `ctx.q.qclass`; `group_id`/`group_name` from `snapshot.groups[ctx.group]` when snapshot non-null, else null/"" with `policy_reason = snapshot_unavailable` on the unfiltered path.
|
||||||
|
- Path mapping: qclass≠IN → `not_evaluated`/`non_in_class`, route `upstream`; pause → `not_evaluated`/`paused`; local → `allow`/`local_record`, route `local`; forward zone → `allow`/`forward_zone`, route `forward_zone` + zone name; cache hits → route `cache`, upstream NULL; upstream answers → route `upstream`, upstream = S2's selected identity redacted via `safe_url` into a Context-local buffer before `Entry.init` — `"pool"`/`pool_upstream` die; allowed matcher decisions (`rule_allow_*`, `blocklist_exception`) → `allow` with exact reason, matched and source captured (today discarded at :442, Codex); no match → `allow`/`no_match`; blocked → `block`/matcher reason, route `blocked`.
|
||||||
|
- **`upstream` is non-null only for attempted upstream or forward-zone exchanges, including their failures** (Codex). It is NULL for local, blocked, cache and rejected routes — the `"local"` marker (handler.zig:392) dies with `"pool"`. Asserted per route in the table tests.
|
||||||
|
- `matched` + `source_id`/`source_name` from `matcher.Decision` and `snapshot.sources[decision.source.?]`. **Copy `matched` and the uncloak target into Context-local buffers before the uncloak loop continues** — the scratch buffers are reused per chain step (matcher.zig:733-736).
|
||||||
|
- CNAME-uncloaked block: policy fields describe the target's decision; `cname_target` holds the target name. `Context.uncloak` (:727-742) widens its return to the target's full decision + name.
|
||||||
|
- Safe search: `safe_search_target` = the rewrite target; policy stays `allow`.
|
||||||
|
|
||||||
|
### S3.1 rcode capture and the truncation bug (Codex)
|
||||||
|
|
||||||
|
`reply`'s UDP truncation rebuild (:522-529) currently rewrites every oversized response to NOERROR — an existing defect: an oversized NXDOMAIN reaches the client as success. Fix here: parse the source rcode before rebuilding and preserve it **via `splitRcode` into both the header and the response OPT** (edns.zig:219-225; header-only preservation loses the upper 8 bits, Codex), then parse the final bytes once for logging. No per-path "known rcode" plumbing: the logged rcode is always derived from the final packet + OPT. Tests: oversized NXDOMAIN keeps NXDOMAIN+TC; an oversized extended-RCODE response keeps the full 12-bit value on the wire and in the log.
|
||||||
|
|
||||||
|
### S3.2 servFail logging
|
||||||
|
|
||||||
|
Every `servFail` site (8, all inside Context) now replies AND logs: `rcode = servfail`, upstream = the last attempted resolver when the failure came from an exchange, policy/route fields as far as the pipeline got.
|
||||||
|
|
||||||
|
### S3.3 Post-parse protocol refusals
|
||||||
|
|
||||||
|
BADVERS (:245), NOTIMP (:253) and bad-EDNS FORMERR (:231) answer an identifiable question but precede `Context`. Construct the logging context as soon as one question is parsed and log these as `not_evaluated`/`protocol_error`, route `rejected`, with the actual rcode. Pre-question failures (rate-limit REFUSED, unparseable, qdcount≠1) stay counters — unchanged.
|
||||||
|
|
||||||
|
### S3.4 Acceptance (S3)
|
||||||
|
|
||||||
|
- [ ] Table-driven provenance tests, one asserted row per path: non-IN, paused, no-snapshot, local, forward-zone, forward-zone cache hit, upstream cache hit, upstream answer (exact redacted URL asserted), rule allow, blocklist exception (source id+name), no-match, rule block, blocklist block with source id+name, CNAME-uncloaked block (cname_target + target's reason), safe-search rewrite, each servFail flavor (exchange-failure case asserts the last-attempted resolver), BADVERS, NOTIMP, bad-EDNS.
|
||||||
|
- [ ] Credential tests (Codex: `Endpoint.parse` rejects `@`, so userinfo cannot come through production config): a handler test with an injected userinfo-bearing identity proves redaction before `Entry.init`; the production-config cross-surface sweep (row, SSE, detail all secret-free, using an accepted credential-bearing DoH path shape) lives in S4.
|
||||||
|
- [ ] Truncation-rcode regression tests per S3.1.
|
||||||
|
- [ ] `pool_upstream` and the `"pool"` marker are gone from provenance producers and serializers (scoped grep — logging fixtures elsewhere are out of scope, Codex).
|
||||||
|
- [ ] `zig build test` 0 failed.
|
||||||
|
|
||||||
|
## Session S4: web API + contracts (src/web/, openapi.yaml, contract samples)
|
||||||
|
|
||||||
|
- List `QueryRow` serialization gains `qclass`, `rcode`, `route_kind`, `policy_action`, `policy_reason`; `block_reason` is gone (S5 updates the admin in the same milestone; `blocked`/`cache_hit`/`upstream` survive, satisfying "existing summary fields stay").
|
||||||
|
- New `GET /api/queries/{id}` → 200 nested `{request:{time,domain,client,qtype,qclass}, group:{id,name}, policy:{action,reason,matched,source_id,source_name}, rewrites:{cname_target,safe_search_target}, route:{kind,forward_zone,upstream}, response:{rcode,duration_us}}`; 404 unknown/pruned; 503 no querylog. Route added; the openapi path-count guard (web_integration_test.zig:~2775) and the route-table cardinality assertion (routes.zig:154) update together.
|
||||||
|
- **One shared full-provenance DTO** (Codex): define `Provenance` (the nested body above) once; `QueryDetail = {id} + Provenance` and the SSE event = `Provenance` exactly — not "QueryRow minus id" (ui-redesign.md:177 requires full live detail). The list row stays a separate summary projection. The live.zig lockstep test compares field names AND types against the shared DTO.
|
||||||
|
- `GET /api/queries` body gains `coverage: {complete: bool, available_since: i64}`; `complete = (filter.since != null and filter.since >= available_since)` with `available_since` the S1 watermark. `/api/stats` and `/api/stats/timeseries` gain the same pair, judged against the period's aligned `since`. Documented in openapi.
|
||||||
|
- openapi.yaml: all schemas updated; `/api/queries/{id}` documented; the three enums enumerated. New focused drift guards for QueryRow, QueryDetail (nested objects included), the coverage object and the three enum value sets against `provenance.zig` — comparing field names, types, nullability and requiredness, not names alone (the path-count guard protects none of that, Codex).
|
||||||
|
- Contract samples regenerated (justfile goldens recipe); add `get_query_detail` sample; drift test green.
|
||||||
|
- [ ] Web integration: detail 200/404/503; coverage fields present and correct across a since-bounded and an unbounded request; keyset walk green. `zig build test` + `-Dintegration` 0 failed.
|
||||||
|
|
||||||
|
## Session S5: admin (admin/src)
|
||||||
|
|
||||||
|
Scope deliberately thin — Activity is milestone 29:
|
||||||
|
|
||||||
|
- `lib/types.ts`: `QueryRow` updated; new `Provenance` + `QueryDetail = {id} & Provenance`; `QueriesPage` gains `coverage`; `LiveQueryEvent = Provenance`. **`LiveRow` becomes a discriminated union** (Codex: a reconnect gap-fetch returns summary `QueryRow`s, which cannot fabricate full provenance): `{kind:"streamed", event: LiveQueryEvent}` | `{kind:"recovered", row: QueryRow}` — recovered rows keep their `id` and link to `/queries/$id`; a shared summary projection feeds the flat `QueryCells` from either arm. ringBuffer.ts and useLiveQueries.ts updated; tests cover both arms and assert the streamed projection drops no field silently.
|
||||||
|
- QueryLogPage/Live status cell switches from `block_reason` to `policy_reason`; no other column changes.
|
||||||
|
- New route `/queries/$id` + detail page modeled on `diagnosticDetailRoute`: the ordered explanation (request, group, policy, rewrites, route, response), historical facts visually separated from current-state links, related actions (lookup the domain, filtered query-log links). Navigation is a real keyboard-focusable link in the row; row-wide pointer click is an enhancement only (Codex).
|
||||||
|
- Coverage: whenever a response says `complete == false`, show "Query history is available from …" against the effective lower bound (not only when the watermark postdates the window, Codex). This touches the current dashboard's stats consumers — permitted: the anti-requirement below bans an Overview redesign, not this notice.
|
||||||
|
- [ ] vitest: detail page renders each section from a fixture; hidden-domain fixture renders the marker; coverage-line tests (watermark inside the window and after it); ring-buffer projection test. Typecheck/prettier/oxlint clean.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## File ownership
|
||||||
|
|
||||||
|
| Files | Session |
|
||||||
|
| --- | --- |
|
||||||
|
| storage/querylog_schema.zig, storage/provenance.zig (new), config/limits.zig (new), storage/logger.zig, storage/repositories/queries_repo.zig, storage/retention.zig, config/validate.zig, storage tests, app.zig (recreated-event detail) | S1 |
|
||||||
|
| upstream/* (incl. both live tests), local/forward_client.zig, server transport fakes + signature fixes in handler.zig/doh/dot/integration/web tests | S2 |
|
||||||
|
| server/handler.zig + server tests (behavioral) | S3 |
|
||||||
|
| web/*, openapi.yaml, web_integration_test.zig, contract samples | S4 |
|
||||||
|
| admin/src/* | S5 |
|
||||||
|
|
||||||
|
S1/S2 are disjoint. S2's mechanical signature edits in S3/S4 territory land before those sessions start.
|
||||||
|
|
||||||
|
## Anti-requirements
|
||||||
|
|
||||||
|
- No response payloads, RR sets, EDNS payloads or packet bytes stored.
|
||||||
|
- No new querylog index; no normalized provenance tables.
|
||||||
|
- No upstream label on cache hits.
|
||||||
|
- No route aliases; no Activity consolidation (step 3); no Overview redesign (step 4) — the coverage notice on existing pages is in scope.
|
||||||
|
- No config knobs for any of this.
|
||||||
|
|
||||||
|
## Acceptance (milestone complete)
|
||||||
|
|
||||||
|
- [ ] `zig build test` and `-Dintegration` 0 failed; `zig fmt --check` clean; admin typecheck/vitest/oxlint/prettier clean; goldens drift test green.
|
||||||
|
- [ ] Live smoke on the real binary: a blocked, an allowed-with-match, a cached, an upstream, a forward-zone and a SERVFAIL query each produce a correct detail page; screenshots taken.
|
||||||
|
- [ ] CHANGELOG Unreleased entry states the history reset, the aside filename pattern, the truncation-rcode fix, and the shrunk `query_log_buffer_max` range.
|
||||||
+245
-23
@@ -61,7 +61,9 @@ const migrations = @import("storage/migrations.zig");
|
|||||||
const model = @import("config/model.zig");
|
const model = @import("config/model.zig");
|
||||||
const pause = @import("server/pause.zig");
|
const pause = @import("server/pause.zig");
|
||||||
const pool_mod = @import("upstream/pool.zig");
|
const pool_mod = @import("upstream/pool.zig");
|
||||||
|
const queries_repo = @import("storage/repositories/queries_repo.zig");
|
||||||
const query_sink = @import("server/query_sink.zig");
|
const query_sink = @import("server/query_sink.zig");
|
||||||
|
const querylog_schema = @import("storage/querylog_schema.zig");
|
||||||
const rate_limiter = @import("server/rate_limiter.zig");
|
const rate_limiter = @import("server/rate_limiter.zig");
|
||||||
const reconcile = @import("config/reconcile.zig");
|
const reconcile = @import("config/reconcile.zig");
|
||||||
const retention_mod = @import("storage/retention.zig");
|
const retention_mod = @import("storage/retention.zig");
|
||||||
@@ -631,29 +633,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
|
|||||||
var querylog_writer_db = querylog_opened.database;
|
var querylog_writer_db = querylog_opened.database;
|
||||||
defer querylog_writer_db.close();
|
defer querylog_writer_db.close();
|
||||||
|
|
||||||
// One-shot and already over: the file was recreated during this boot, and
|
reportQuerylogRecreated(event_store, io, boot_now_s, &querylog_opened, &querylog_writer_db);
|
||||||
// there is nothing to recover from. Never emitted for `.missing` — a first
|
|
||||||
// creation renames nothing aside, so the event would carry an aside path
|
|
||||||
// that does not exist and would greet every fresh install with a warning.
|
|
||||||
if (querylog_opened.recreated) |cause| {
|
|
||||||
if (cause != .missing) {
|
|
||||||
if (event_store) |store| {
|
|
||||||
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
|
|
||||||
const detail = std.fmt.bufPrint(&detail_buf, "previous file kept as '{s}'", .{
|
|
||||||
querylog_opened.aside(),
|
|
||||||
}) catch detail_buf[0..];
|
|
||||||
store.reportResolved(
|
|
||||||
io,
|
|
||||||
boot_now_s,
|
|
||||||
.query_log_recreated,
|
|
||||||
@tagName(cause),
|
|
||||||
@tagName(cause),
|
|
||||||
.warning,
|
|
||||||
detail,
|
|
||||||
);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
var querylog_retention_db = try data.reopenQuerylogDb(io);
|
var querylog_retention_db = try data.reopenQuerylogDb(io);
|
||||||
defer querylog_retention_db.close();
|
defer querylog_retention_db.close();
|
||||||
var querylog_history_db = try data.reopenQuerylogDb(io);
|
var querylog_history_db = try data.reopenQuerylogDb(io);
|
||||||
@@ -1357,9 +1337,251 @@ fn parseBind(
|
|||||||
return addr;
|
return addr;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Files the one-shot `query_log.recreated` event for a boot that replaced the
|
||||||
|
/// query log.
|
||||||
|
///
|
||||||
|
/// One-shot and already over: the file was recreated during this boot, and
|
||||||
|
/// there is nothing to recover from. Never emitted for `.missing` — a first
|
||||||
|
/// creation renames nothing aside, so the event would carry an aside path that
|
||||||
|
/// does not exist and would greet every fresh install with a warning.
|
||||||
|
///
|
||||||
|
/// `database` is the connection to the file that was just created; the coverage
|
||||||
|
/// start is read from it rather than recomputed, so the event states the value
|
||||||
|
/// the API will.
|
||||||
|
fn reportQuerylogRecreated(
|
||||||
|
store: ?*events.Store,
|
||||||
|
io: std.Io,
|
||||||
|
now_s: i64,
|
||||||
|
opened: *const querylog_schema.OpenResult,
|
||||||
|
database: *db.Db,
|
||||||
|
) void {
|
||||||
|
const cause = opened.recreated orelse return;
|
||||||
|
if (cause == .missing) return;
|
||||||
|
const s = store orelse return;
|
||||||
|
|
||||||
|
// The coverage start belongs in this detail: the recreate is exactly the
|
||||||
|
// moment the history the operator had stops existing, and the watermark is
|
||||||
|
// the answer to "from when can I still ask?".
|
||||||
|
const coverage_start: ?i64 = queries_repo.availableSince(database) catch null;
|
||||||
|
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
|
||||||
|
const detail = recreatedDetail(&detail_buf, opened.aside(), coverage_start);
|
||||||
|
s.reportResolved(io, now_s, .query_log_recreated, @tagName(cause), @tagName(cause), .warning, detail);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The `query_log.recreated` detail line: what was kept, and from when the new
|
||||||
|
/// file can answer.
|
||||||
|
///
|
||||||
|
/// The coverage start is the operator's actual remedy information — the event
|
||||||
|
/// says "this history is gone" and this says "and here is where the new history
|
||||||
|
/// begins". Null only when the fresh file would not answer, which is already a
|
||||||
|
/// separate failure; the line still names the aside rather than saying nothing.
|
||||||
|
///
|
||||||
|
/// The aside is a full path under the data directory, which can be longer than
|
||||||
|
/// the whole detail column, so the two facts compete for the buffer. The
|
||||||
|
/// watermark always wins and the name degrades in whole steps: full path, then
|
||||||
|
/// basename — which the event's own database directory disambiguates — then no
|
||||||
|
/// name at all. Never a path cut mid-string, which names no file on disk and
|
||||||
|
/// reads as if it did.
|
||||||
|
fn recreatedDetail(
|
||||||
|
buf: *[events.Store.max_detail_len]u8,
|
||||||
|
aside: []const u8,
|
||||||
|
coverage_start: ?i64,
|
||||||
|
) []const u8 {
|
||||||
|
const names = [_][]const u8{ aside, std.fs.path.basename(aside) };
|
||||||
|
const since = coverage_start orelse {
|
||||||
|
for (names) |name| {
|
||||||
|
return std.fmt.bufPrint(buf, "previous file kept as '{s}'", .{name}) catch continue;
|
||||||
|
}
|
||||||
|
return "previous file kept aside";
|
||||||
|
};
|
||||||
|
for (names) |name| {
|
||||||
|
return std.fmt.bufPrint(
|
||||||
|
buf,
|
||||||
|
"previous file kept as '{s}'; query history is available from {d}",
|
||||||
|
.{ name, since },
|
||||||
|
) catch continue;
|
||||||
|
}
|
||||||
|
// The buffer is `max_detail_len`, which no i64 can overrun on its own.
|
||||||
|
return std.fmt.bufPrint(buf, "query history is available from {d}", .{since}) catch unreachable;
|
||||||
|
}
|
||||||
|
|
||||||
const events_fixture = @import("storage/events_fixture.zig");
|
const events_fixture = @import("storage/events_fixture.zig");
|
||||||
const testing = std.testing;
|
const testing = std.testing;
|
||||||
|
|
||||||
|
test "the recreated detail names the aside and the new coverage start" {
|
||||||
|
var buf: [events.Store.max_detail_len]u8 = undefined;
|
||||||
|
|
||||||
|
try std.testing.expectEqualStrings(
|
||||||
|
"previous file kept as 'querylog.db.schema-changed-1700000000'; " ++
|
||||||
|
"query history is available from 1700000001",
|
||||||
|
recreatedDetail(&buf, "querylog.db.schema-changed-1700000000", 1700000001),
|
||||||
|
);
|
||||||
|
|
||||||
|
// A fresh file that will not answer is a separate failure; the line still
|
||||||
|
// says what was kept rather than reporting nothing.
|
||||||
|
try std.testing.expectEqualStrings(
|
||||||
|
"previous file kept as 'querylog.db.corrupt-1700000000'",
|
||||||
|
recreatedDetail(&buf, "querylog.db.corrupt-1700000000", null),
|
||||||
|
);
|
||||||
|
|
||||||
|
// A data directory deep enough that its path alone would fill the column:
|
||||||
|
// the watermark is complete and the name degrades to the basename, which
|
||||||
|
// still names a real file.
|
||||||
|
const deep = "/srv/" ++ ("d" ** 60 ++ "/") ** 8 ++ "querylog.db.corrupt-1700000000";
|
||||||
|
try std.testing.expectEqualStrings(
|
||||||
|
"previous file kept as 'querylog.db.corrupt-1700000000'; " ++
|
||||||
|
"query history is available from 1700000001",
|
||||||
|
recreatedDetail(&buf, deep, 1700000001),
|
||||||
|
);
|
||||||
|
try std.testing.expectEqualStrings(
|
||||||
|
"previous file kept as 'querylog.db.corrupt-1700000000'",
|
||||||
|
recreatedDetail(&buf, deep, null),
|
||||||
|
);
|
||||||
|
|
||||||
|
// No filesystem produces a name this long, but a truncated one would name
|
||||||
|
// nothing: the watermark survives alone rather than half-named.
|
||||||
|
const unnameable = "/srv/" ++ "n" ** 500;
|
||||||
|
try std.testing.expectEqualStrings(
|
||||||
|
"query history is available from 1700000001",
|
||||||
|
recreatedDetail(&buf, unnameable, 1700000001),
|
||||||
|
);
|
||||||
|
try std.testing.expectEqualStrings(
|
||||||
|
"previous file kept aside",
|
||||||
|
recreatedDetail(&buf, unnameable, null),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a fingerprint recreate files a resolved event naming the real aside and watermark" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var tmp = testing.tmpDir(.{ .iterate = true });
|
||||||
|
defer tmp.cleanup();
|
||||||
|
|
||||||
|
var path_buf: [256]u8 = undefined;
|
||||||
|
const path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/querylog.db", .{tmp.sub_path});
|
||||||
|
|
||||||
|
var fx: events_fixture.Fixture = .{};
|
||||||
|
try fx.init(io, 1000);
|
||||||
|
defer fx.deinit();
|
||||||
|
|
||||||
|
// A fresh install: the file was missing, nothing was set aside, and the
|
||||||
|
// event would name a path that does not exist.
|
||||||
|
var created = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
|
||||||
|
reportQuerylogRecreated(&fx.store, io, 1000, &created, &created.database);
|
||||||
|
created.database.close();
|
||||||
|
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
|
||||||
|
|
||||||
|
// A healthy file this build's DDL no longer matches, which is what an
|
||||||
|
// upgrade that edits the schema produces.
|
||||||
|
{
|
||||||
|
var stamped = try db.Db.open(path, .{ .mode = .read_write_existing });
|
||||||
|
defer stamped.close();
|
||||||
|
var sql_buf: [64]u8 = undefined;
|
||||||
|
try stamped.exec(try std.fmt.bufPrintZ(
|
||||||
|
&sql_buf,
|
||||||
|
"PRAGMA user_version = {d};",
|
||||||
|
.{querylog_schema.fingerprint +% 1},
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
var recreated = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
|
||||||
|
defer recreated.database.close();
|
||||||
|
try testing.expectEqual(querylog_schema.RecreateReason.fingerprint_mismatch, recreated.recreated.?);
|
||||||
|
|
||||||
|
reportQuerylogRecreated(&fx.store, io, 2000, &recreated, &recreated.database);
|
||||||
|
|
||||||
|
try testing.expectEqualStrings("query_log.recreated", try fx.text("SELECT code FROM operational_events"));
|
||||||
|
try testing.expectEqualStrings("fingerprint_mismatch", try fx.text("SELECT subject_key FROM operational_events"));
|
||||||
|
try testing.expectEqualStrings("warning", try fx.text("SELECT severity FROM operational_events"));
|
||||||
|
// One-shot: already over when it is filed, so it never becomes an open
|
||||||
|
// episode `/api/health` counts.
|
||||||
|
try testing.expectEqual(
|
||||||
|
@as(i64, 0),
|
||||||
|
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
|
||||||
|
);
|
||||||
|
|
||||||
|
// The detail carries the path that is actually on disk and the watermark
|
||||||
|
// the API will serve, both read back from the recreate rather than from
|
||||||
|
// the arguments the event was built with.
|
||||||
|
try tmp.dir.access(io, std.fs.path.basename(recreated.aside()), .{});
|
||||||
|
const coverage = try queries_repo.availableSince(&recreated.database);
|
||||||
|
var expected_buf: [events.Store.max_detail_len]u8 = undefined;
|
||||||
|
const expected = try std.fmt.bufPrint(
|
||||||
|
&expected_buf,
|
||||||
|
"previous file kept as '{s}'; query history is available from {d}",
|
||||||
|
.{ recreated.aside(), coverage },
|
||||||
|
);
|
||||||
|
try testing.expectEqualStrings(expected, try fx.text("SELECT detail FROM operational_events"));
|
||||||
|
|
||||||
|
// A boot with no diagnostics store configured is not a failure path.
|
||||||
|
reportQuerylogRecreated(null, io, 2000, &recreated, &recreated.database);
|
||||||
|
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a recreate under a long data directory keeps the watermark and a usable name" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var tmp = testing.tmpDir(.{ .iterate = true });
|
||||||
|
defer tmp.cleanup();
|
||||||
|
|
||||||
|
// Deep enough that the aside outgrows the detail column, shallow enough
|
||||||
|
// that SQLite's unix VFS still opens the file: it caps a path at 512 bytes.
|
||||||
|
const nested = ("d" ** 60 ++ "/") ** 6 ++ "d" ** 60;
|
||||||
|
try tmp.dir.createDirPath(io, nested);
|
||||||
|
|
||||||
|
var path_buf: [1024]u8 = undefined;
|
||||||
|
const path = try std.fmt.bufPrintZ(
|
||||||
|
&path_buf,
|
||||||
|
".zig-cache/tmp/{s}/{s}/querylog.db",
|
||||||
|
.{ tmp.sub_path, nested },
|
||||||
|
);
|
||||||
|
|
||||||
|
var fx: events_fixture.Fixture = .{};
|
||||||
|
try fx.init(io, 1000);
|
||||||
|
defer fx.deinit();
|
||||||
|
|
||||||
|
{
|
||||||
|
var created = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
|
||||||
|
defer created.database.close();
|
||||||
|
var sql_buf: [64]u8 = undefined;
|
||||||
|
try created.database.exec(try std.fmt.bufPrintZ(
|
||||||
|
&sql_buf,
|
||||||
|
"PRAGMA user_version = {d};",
|
||||||
|
.{querylog_schema.fingerprint +% 1},
|
||||||
|
));
|
||||||
|
}
|
||||||
|
|
||||||
|
var recreated = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
|
||||||
|
defer recreated.database.close();
|
||||||
|
try testing.expectEqual(querylog_schema.RecreateReason.fingerprint_mismatch, recreated.recreated.?);
|
||||||
|
const line_overhead = "previous file kept as ''; query history is available from ".len;
|
||||||
|
try testing.expect(recreated.aside().len + line_overhead > events.Store.max_detail_len);
|
||||||
|
|
||||||
|
reportQuerylogRecreated(&fx.store, io, 2000, &recreated, &recreated.database);
|
||||||
|
|
||||||
|
const detail = try fx.text("SELECT detail FROM operational_events");
|
||||||
|
const coverage = try queries_repo.availableSince(&recreated.database);
|
||||||
|
const name = std.fs.path.basename(recreated.aside());
|
||||||
|
var expected_buf: [events.Store.max_detail_len]u8 = undefined;
|
||||||
|
const expected = try std.fmt.bufPrint(
|
||||||
|
&expected_buf,
|
||||||
|
"previous file kept as '{s}'; query history is available from {d}",
|
||||||
|
.{ name, coverage },
|
||||||
|
);
|
||||||
|
// The watermark is whole — the fact that would be lost to a mid-string cut
|
||||||
|
// — and the name it kept is the file's, not a prefix of its path.
|
||||||
|
try testing.expectEqualStrings(expected, detail);
|
||||||
|
try testing.expect(detail.len <= events.Store.max_detail_len);
|
||||||
|
|
||||||
|
var deep = try tmp.dir.openDir(io, nested, .{});
|
||||||
|
defer deep.close(io);
|
||||||
|
try deep.access(io, name, .{});
|
||||||
|
}
|
||||||
|
|
||||||
test "parseBind refuses a bind address of the wrong family" {
|
test "parseBind refuses a bind address of the wrong family" {
|
||||||
var out_buf: [8]u8 = undefined;
|
var out_buf: [8]u8 = undefined;
|
||||||
var err_buf: [256]u8 = undefined;
|
var err_buf: [256]u8 = undefined;
|
||||||
|
|||||||
+2
-1
@@ -1007,7 +1007,8 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize {
|
|||||||
}};
|
}};
|
||||||
var single: pool.Pool = .init(&entries, .{}, timeouts, seed);
|
var single: pool.Pool = .init(&entries, .{}, timeouts, seed);
|
||||||
|
|
||||||
if (single.exchange(r.io, probe_query, response_buf)) |_| {
|
var selected: ?[]const u8 = null;
|
||||||
|
if (single.exchange(r.io, probe_query, response_buf, &selected)) |_| {
|
||||||
try r.out.print("OK upstreams[{d}] {f}\n", .{ i, safe_url.redact(server.url) });
|
try r.out.print("OK upstreams[{d}] {f}\n", .{ i, safe_url.redact(server.url) });
|
||||||
} else |_| {
|
} else |_| {
|
||||||
// The concrete cause lives in the entry's health, which is where the
|
// The concrete cause lives in the entry's health, which is where the
|
||||||
|
|||||||
@@ -0,0 +1,19 @@
|
|||||||
|
//! Length caps on the configuration labels that the query log copies into every
|
||||||
|
//! row it writes.
|
||||||
|
//!
|
||||||
|
//! They live in a module of their own because two files need them and neither
|
||||||
|
//! may import the other: `config/validate.zig` rejects a name that exceeds a cap
|
||||||
|
//! and `storage/logger.zig` sizes an `Entry` buffer from it, while `validate`
|
||||||
|
//! already imports `logger` for the entry-size budget it derives. A cap owned by
|
||||||
|
//! either file would close that loop.
|
||||||
|
//!
|
||||||
|
//! The values are deliberately short. A group or a source name is a label an
|
||||||
|
//! operator reads in a table cell, and every byte of it is copied into every
|
||||||
|
//! logged row — the cap is what keeps a pasted paragraph out of an `Entry` that
|
||||||
|
//! travels by value through the queue.
|
||||||
|
|
||||||
|
/// `groups[].name`.
|
||||||
|
pub const max_group_name_len = 64;
|
||||||
|
|
||||||
|
/// `blocklist_sources[].name`.
|
||||||
|
pub const max_source_name_len = 64;
|
||||||
+104
-5
@@ -6,9 +6,13 @@
|
|||||||
//! instead of a line number, so every `UNIQUE` and every foreign key in
|
//! instead of a line number, so every `UNIQUE` and every foreign key in
|
||||||
//! PLAN §11.2 has a check here.
|
//! PLAN §11.2 has a check here.
|
||||||
//!
|
//!
|
||||||
//! Pure: no `std.Io` value is a parameter anywhere, no SQLite, no clock. The
|
//! Pure: no `std.Io` value is a parameter anywhere, no SQLite call, no clock.
|
||||||
//! only `std.Io` type used is `std.Io.Writer`, for rendering diagnostics. The
|
//! The only `std.Io` type used is `std.Io.Writer`, for rendering diagnostics.
|
||||||
//! allocator exists for diagnostic text and scratch bookkeeping alone.
|
//! The allocator exists for diagnostic text and scratch bookkeeping alone. The
|
||||||
|
//! `storage/logger.zig` import is a comptime one — `query_log_buffer_max` is
|
||||||
|
//! derived from `@sizeOf(logger.Entry)`, because the bound this file enforces
|
||||||
|
//! on the queue is a bound on bytes and only the logger knows how wide a queued
|
||||||
|
//! entry is. Nothing in this file calls into storage.
|
||||||
//!
|
//!
|
||||||
//! Parsers are reused, never reimplemented: `transport.Endpoint.parse` for
|
//! Parsers are reused, never reimplemented: `transport.Endpoint.parse` for
|
||||||
//! upstream URLs, `NetAddress.parse` / `Prefix.parse` for addresses, and
|
//! upstream URLs, `NetAddress.parse` / `Prefix.parse` for addresses, and
|
||||||
@@ -44,6 +48,8 @@ const Writer = std.Io.Writer;
|
|||||||
const model = @import("model.zig");
|
const model = @import("model.zig");
|
||||||
const address = @import("../platform/address.zig");
|
const address = @import("../platform/address.zig");
|
||||||
const dns_name = @import("../dns/name.zig");
|
const dns_name = @import("../dns/name.zig");
|
||||||
|
const limits = @import("limits.zig");
|
||||||
|
const logger = @import("../storage/logger.zig");
|
||||||
const regex = @import("../filter/regex.zig");
|
const regex = @import("../filter/regex.zig");
|
||||||
const safe_url = @import("../safe_url.zig");
|
const safe_url = @import("../safe_url.zig");
|
||||||
const transport = @import("../upstream/transport.zig");
|
const transport = @import("../upstream/transport.zig");
|
||||||
@@ -72,6 +78,7 @@ pub const ValidateError = error{
|
|||||||
DuplicateGroupName,
|
DuplicateGroupName,
|
||||||
UnknownGroup,
|
UnknownGroup,
|
||||||
EmptyGroupName,
|
EmptyGroupName,
|
||||||
|
GroupNameTooLong,
|
||||||
BadClientIp,
|
BadClientIp,
|
||||||
DuplicateClientIp,
|
DuplicateClientIp,
|
||||||
BadClientPrefix,
|
BadClientPrefix,
|
||||||
@@ -79,6 +86,7 @@ pub const ValidateError = error{
|
|||||||
BadSourceUrl,
|
BadSourceUrl,
|
||||||
DuplicateSourceUrl,
|
DuplicateSourceUrl,
|
||||||
EmptySourceName,
|
EmptySourceName,
|
||||||
|
SourceNameTooLong,
|
||||||
UnknownSource,
|
UnknownSource,
|
||||||
DuplicateGroupSource,
|
DuplicateGroupSource,
|
||||||
BadRulePattern,
|
BadRulePattern,
|
||||||
@@ -461,13 +469,16 @@ fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void {
|
|||||||
if (cfg.logging.query_log_buffer_max < 1) {
|
if (cfg.logging.query_log_buffer_max < 1) {
|
||||||
try diags.add(error.BadRetention, "logging.query_log_buffer_max", .{}, "must be at least 1", .{});
|
try diags.add(error.BadRetention, "logging.query_log_buffer_max", .{}, "must be at least 1", .{});
|
||||||
}
|
}
|
||||||
if (cfg.logging.query_log_buffer_max > max_boot_entries) {
|
// Its own ceiling, not `max_boot_entries`: a queued `logger.Entry` carries
|
||||||
|
// every provenance field by value, so the queue's cost is bytes rather than
|
||||||
|
// entries and the bound follows the width of the entry.
|
||||||
|
if (cfg.logging.query_log_buffer_max > logger.query_log_buffer_max) {
|
||||||
try diags.add(
|
try diags.add(
|
||||||
error.BadRetention,
|
error.BadRetention,
|
||||||
"logging.query_log_buffer_max",
|
"logging.query_log_buffer_max",
|
||||||
.{},
|
.{},
|
||||||
"must be at most {d}, got {d}",
|
"must be at most {d}, got {d}",
|
||||||
.{ max_boot_entries, cfg.logging.query_log_buffer_max },
|
.{ logger.query_log_buffer_max, cfg.logging.query_log_buffer_max },
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
// No floor: 0 is the documented "do not wait" setting, not a mistake.
|
// No floor: 0 is the documented "do not wait" setting, not a mistake.
|
||||||
@@ -714,6 +725,29 @@ fn checkDotHost(
|
|||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The shared shape of the two label caps.
|
||||||
|
///
|
||||||
|
/// Both names are copied by value into every `query_log` row that mentions
|
||||||
|
/// them, so the cap is what keeps a pasted paragraph out of the fixed buffers
|
||||||
|
/// of `storage/logger.zig`. Bytes, not codepoints: the buffer counts bytes.
|
||||||
|
fn checkNameLength(
|
||||||
|
diags: *Diagnostics,
|
||||||
|
comptime fault: ValidateError,
|
||||||
|
comptime path: []const u8,
|
||||||
|
path_args: anytype,
|
||||||
|
value: []const u8,
|
||||||
|
cap: usize,
|
||||||
|
) error{OutOfMemory}!void {
|
||||||
|
if (value.len <= cap) return;
|
||||||
|
try diags.add(
|
||||||
|
fault,
|
||||||
|
path,
|
||||||
|
path_args,
|
||||||
|
"must be at most {d} bytes, got {d}; the name is copied into every logged query",
|
||||||
|
.{ cap, value.len },
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{OutOfMemory}!void {
|
fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{OutOfMemory}!void {
|
||||||
var group_names: IndexSet = .empty;
|
var group_names: IndexSet = .empty;
|
||||||
var has_default = false;
|
var has_default = false;
|
||||||
@@ -729,6 +763,16 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{
|
|||||||
.{safe_url.quoteText(group.name)},
|
.{safe_url.quoteText(group.name)},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
// Independent of the chain above: an over-long name is still a name,
|
||||||
|
// and a duplicate of one is still a duplicate.
|
||||||
|
try checkNameLength(
|
||||||
|
diags,
|
||||||
|
error.GroupNameTooLong,
|
||||||
|
"groups[{d}].name",
|
||||||
|
.{i},
|
||||||
|
group.name,
|
||||||
|
limits.max_group_name_len,
|
||||||
|
);
|
||||||
if (std.mem.eql(u8, group.name, "default")) has_default = true;
|
if (std.mem.eql(u8, group.name, "default")) has_default = true;
|
||||||
}
|
}
|
||||||
if (!has_default) {
|
if (!has_default) {
|
||||||
@@ -859,6 +903,14 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{
|
|||||||
.{},
|
.{},
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
try checkNameLength(
|
||||||
|
diags,
|
||||||
|
error.SourceNameTooLong,
|
||||||
|
"blocklist_sources[{d}].name",
|
||||||
|
.{i},
|
||||||
|
source.name,
|
||||||
|
limits.max_source_name_len,
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
var group_source_pairs: IndexSet = .empty;
|
var group_source_pairs: IndexSet = .empty;
|
||||||
@@ -1308,6 +1360,24 @@ test "an https:// upstream may name a host" {
|
|||||||
try expectClean(cfg);
|
try expectClean(cfg);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The longest host `transport.Endpoint.parse` accepts: four labels, 253 bytes.
|
||||||
|
const host_at_bound = ("a" ** 63 ++ ".") ** 3 ++ "a" ** 61;
|
||||||
|
|
||||||
|
test "an upstream host at the length bound validates cleanly" {
|
||||||
|
var cfg = baseConfig();
|
||||||
|
cfg.upstreams = &.{.{ .url = "https://" ++ host_at_bound ++ "/dns-query" }};
|
||||||
|
try expectClean(cfg);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "error.BadUpstreamUrl on an upstream host one byte past the length bound" {
|
||||||
|
// The bound is the query log's `upstream` width and every other identity
|
||||||
|
// built from the endpoint, so an over-long host has to fail here rather
|
||||||
|
// than be shortened downstream.
|
||||||
|
var cfg = baseConfig();
|
||||||
|
cfg.upstreams = &.{.{ .url = "https://" ++ host_at_bound ++ "a/dns-query" }};
|
||||||
|
try expectProblem(cfg, error.BadUpstreamUrl, "upstreams[0].url");
|
||||||
|
}
|
||||||
|
|
||||||
test "an IPv6 literal tls:// upstream validates cleanly" {
|
test "an IPv6 literal tls:// upstream validates cleanly" {
|
||||||
// `Endpoint.parse` strips the brackets, so the host reaching the check is
|
// `Endpoint.parse` strips the brackets, so the host reaching the check is
|
||||||
// exactly what the client hands to the address parser.
|
// exactly what the client hands to the address parser.
|
||||||
@@ -1371,6 +1441,35 @@ test "error.EmptyGroupName" {
|
|||||||
try expectProblem(cfg, error.EmptyGroupName, "groups[1].name");
|
try expectProblem(cfg, error.EmptyGroupName, "groups[1].name");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "error.GroupNameTooLong" {
|
||||||
|
const cap = limits.max_group_name_len;
|
||||||
|
// Exactly at the cap is accepted; one byte past it is not. The cap is what
|
||||||
|
// `storage/logger.zig` sizes its `Entry` buffer from, so a name that passes
|
||||||
|
// here is a name a logged row stores whole.
|
||||||
|
var at_cap = baseConfig();
|
||||||
|
at_cap.groups = &.{ .{ .name = "default" }, .{ .name = "g" ** cap } };
|
||||||
|
try expectClean(at_cap);
|
||||||
|
|
||||||
|
var over = baseConfig();
|
||||||
|
over.groups = &.{ .{ .name = "default" }, .{ .name = "g" ** (cap + 1) } };
|
||||||
|
try expectProblem(over, error.GroupNameTooLong, "groups[1].name");
|
||||||
|
}
|
||||||
|
|
||||||
|
test "error.SourceNameTooLong" {
|
||||||
|
const cap = limits.max_source_name_len;
|
||||||
|
const url = "https://lists.example/hosts.txt";
|
||||||
|
|
||||||
|
var at_cap = baseConfig();
|
||||||
|
at_cap.blocklist_sources = &.{.{ .url = url, .name = "s" ** cap }};
|
||||||
|
at_cap.group_sources = &.{.{ .group = "default", .source_url = url }};
|
||||||
|
try expectClean(at_cap);
|
||||||
|
|
||||||
|
var over = baseConfig();
|
||||||
|
over.blocklist_sources = &.{.{ .url = url, .name = "s" ** (cap + 1) }};
|
||||||
|
over.group_sources = &.{.{ .group = "default", .source_url = url }};
|
||||||
|
try expectProblem(over, error.SourceNameTooLong, "blocklist_sources[0].name");
|
||||||
|
}
|
||||||
|
|
||||||
test "error.BadClientIp" {
|
test "error.BadClientIp" {
|
||||||
var cfg = baseConfig();
|
var cfg = baseConfig();
|
||||||
cfg.clients = &.{.{ .ip = "nonsense" }};
|
cfg.clients = &.{.{ .ip = "nonsense" }};
|
||||||
|
|||||||
@@ -33,8 +33,17 @@ const log = std.log.scoped(.forward_client);
|
|||||||
/// each half large enough to frame a query in one write, not a capacity.
|
/// each half large enough to frame a query in one write, not a capacity.
|
||||||
pub const min_frame_buf: usize = 1024;
|
pub const min_frame_buf: usize = 1024;
|
||||||
|
|
||||||
|
/// `tcp://[` + the longest IPv6 text form + `]:65535`, the widest spelling
|
||||||
|
/// `identityText` can produce.
|
||||||
|
pub const max_identity_len: usize = "tcp://[".len + 45 + "]:65535".len;
|
||||||
|
|
||||||
pub const ForwardClient = struct {
|
pub const ForwardClient = struct {
|
||||||
resolver: validate.Resolver,
|
resolver: validate.Resolver,
|
||||||
|
/// The resolver as text, owned here so the `transport.Client` out-parameter
|
||||||
|
/// has something stable to borrow: `validate.Resolver` is a parsed address,
|
||||||
|
/// and a caller logging the exchange needs its spelling.
|
||||||
|
identity_buf: [max_identity_len]u8 = undefined,
|
||||||
|
identity_len: usize = 0,
|
||||||
/// Caller-owned scratch for the TCP length-prefixed path.
|
/// Caller-owned scratch for the TCP length-prefixed path.
|
||||||
frame_buf: []u8,
|
frame_buf: []u8,
|
||||||
/// On the `.awake` clock at the caller's choosing, so a suspended host does
|
/// On the `.awake` clock at the caller's choosing, so a suspended host does
|
||||||
@@ -64,11 +73,20 @@ pub const ForwardClient = struct {
|
|||||||
read_timeout: std.Io.Clock.Duration,
|
read_timeout: std.Io.Clock.Duration,
|
||||||
) ForwardClient {
|
) ForwardClient {
|
||||||
std.debug.assert(frame_buf.len >= min_frame_buf);
|
std.debug.assert(frame_buf.len >= min_frame_buf);
|
||||||
return .{
|
var self: ForwardClient = .{
|
||||||
.resolver = resolver,
|
.resolver = resolver,
|
||||||
.frame_buf = frame_buf,
|
.frame_buf = frame_buf,
|
||||||
.read_timeout = read_timeout,
|
.read_timeout = read_timeout,
|
||||||
};
|
};
|
||||||
|
self.identity_len = identityText(resolver, &self.identity_buf).len;
|
||||||
|
return self;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// `udp://192.168.1.1:53`, `tcp://[fd00::1]:53` — the same spelling
|
||||||
|
/// `validate.parseResolver` accepts, so a log row names the configured
|
||||||
|
/// value. Valid for as long as this client is.
|
||||||
|
pub fn identity(self: *const ForwardClient) []const u8 {
|
||||||
|
return self.identity_buf[0..self.identity_len];
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn client(self: *ForwardClient) transport.Client {
|
pub fn client(self: *ForwardClient) transport.Client {
|
||||||
@@ -80,8 +98,12 @@ pub const ForwardClient = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
const self: *ForwardClient = @ptrCast(@alignCast(ptr));
|
const self: *ForwardClient = @ptrCast(@alignCast(ptr));
|
||||||
|
// Set before the attempt: a failed forward-zone exchange still names
|
||||||
|
// the resolver it was sent to.
|
||||||
|
selected.* = self.identity();
|
||||||
return self.exchange(io, query, response_buf);
|
return self.exchange(io, query, response_buf);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -238,6 +260,24 @@ pub const ForwardClient = struct {
|
|||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
|
fn identityText(resolver: validate.Resolver, buf: *[max_identity_len]u8) []const u8 {
|
||||||
|
var w: std.Io.Writer = .fixed(buf);
|
||||||
|
w.writeAll(switch (resolver.scheme) {
|
||||||
|
.udp => "udp://",
|
||||||
|
.tcp => "tcp://",
|
||||||
|
}) catch unreachable;
|
||||||
|
|
||||||
|
const bracketed = switch (resolver.addr) {
|
||||||
|
.ip4 => false,
|
||||||
|
.ip6 => true,
|
||||||
|
};
|
||||||
|
if (bracketed) w.writeByte('[') catch unreachable;
|
||||||
|
resolver.addr.format(&w) catch unreachable;
|
||||||
|
if (bracketed) w.writeByte(']') catch unreachable;
|
||||||
|
w.print(":{d}", .{resolver.port}) catch unreachable;
|
||||||
|
return w.buffered();
|
||||||
|
}
|
||||||
|
|
||||||
/// The local address a datagram to `dest` is sent from: same family, port
|
/// The local address a datagram to `dest` is sent from: same family, port
|
||||||
/// chosen by the kernel.
|
/// chosen by the kernel.
|
||||||
fn wildcardFor(dest: net.IpAddress) net.IpAddress {
|
fn wildcardFor(dest: net.IpAddress) net.IpAddress {
|
||||||
@@ -286,6 +326,27 @@ test "ForwardClient satisfies the Client interface" {
|
|||||||
try testing.expectEqual(@as(u16, 53), fc.resolver.port);
|
try testing.expectEqual(@as(u16, 53), fc.resolver.port);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "the client owns its resolver identity in both address families" {
|
||||||
|
var buf = testBuf();
|
||||||
|
const v4: ForwardClient = .init(
|
||||||
|
try validate.parseResolver("udp://192.168.1.1:5300"),
|
||||||
|
&buf,
|
||||||
|
.{ .raw = .fromSeconds(1), .clock = .awake },
|
||||||
|
);
|
||||||
|
try testing.expectEqualStrings("udp://192.168.1.1:5300", v4.identity());
|
||||||
|
|
||||||
|
var buf6 = testBuf();
|
||||||
|
const v6: ForwardClient = .init(
|
||||||
|
try validate.parseResolver("tcp://[fd00::1]:5353"),
|
||||||
|
&buf6,
|
||||||
|
.{ .raw = .fromSeconds(1), .clock = .awake },
|
||||||
|
);
|
||||||
|
try testing.expectEqualStrings("tcp://[fd00::1]:5353", v6.identity());
|
||||||
|
|
||||||
|
// The borrow points into the client, not into `init`'s frame.
|
||||||
|
try testing.expect(@intFromPtr(v6.identity().ptr) >= @intFromPtr(&v6));
|
||||||
|
}
|
||||||
|
|
||||||
test "the stats struct starts at zero" {
|
test "the stats struct starts at zero" {
|
||||||
const stats: ForwardClient.Stats = .{};
|
const stats: ForwardClient.Stats = .{};
|
||||||
try testing.expectEqual(@as(u64, 0), stats.queries);
|
try testing.expectEqual(@as(u64, 0), stats.queries);
|
||||||
|
|||||||
@@ -592,11 +592,13 @@ const FailingUpstream = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = ptr;
|
_ = ptr;
|
||||||
_ = io;
|
_ = io;
|
||||||
_ = query;
|
_ = query;
|
||||||
_ = response_buf;
|
_ = response_buf;
|
||||||
|
selected.* = "fake://failing-upstream";
|
||||||
return error.ConnectFailed;
|
return error.ConnectFailed;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -351,8 +351,10 @@ const FakeUpstream = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
|
selected.* = "fake://dot-server-upstream";
|
||||||
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
||||||
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
|
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
|
||||||
@memcpy(response_buf[0..self.reply.len], self.reply);
|
@memcpy(response_buf[0..self.reply.len], self.reply);
|
||||||
|
|||||||
+1390
-183
File diff suppressed because it is too large
Load Diff
@@ -33,6 +33,7 @@ const handler = @import("handler.zig");
|
|||||||
const header = @import("../dns/header.zig");
|
const header = @import("../dns/header.zig");
|
||||||
const local_tables = @import("local_tables.zig");
|
const local_tables = @import("local_tables.zig");
|
||||||
const logger_mod = @import("../storage/logger.zig");
|
const logger_mod = @import("../storage/logger.zig");
|
||||||
|
const provenance = @import("../storage/provenance.zig");
|
||||||
const manager = @import("../filter/manager.zig");
|
const manager = @import("../filter/manager.zig");
|
||||||
const matcher = @import("../filter/matcher.zig");
|
const matcher = @import("../filter/matcher.zig");
|
||||||
const migrations = @import("../storage/migrations.zig");
|
const migrations = @import("../storage/migrations.zig");
|
||||||
@@ -183,8 +184,10 @@ const FakeUpstream = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
|
selected.* = "fake://phase7-upstream";
|
||||||
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
||||||
_ = self.calls.fetchAdd(1, .monotonic);
|
_ = self.calls.fetchAdd(1, .monotonic);
|
||||||
|
|
||||||
@@ -346,7 +349,8 @@ test "S7 case 1: a blocked domain is answered with the zero address and logged"
|
|||||||
try testing.expectEqual(@as(usize, 1), logged.len);
|
try testing.expectEqual(@as(usize, 1), logged.len);
|
||||||
try testing.expectEqual(true, logged[0].blocked);
|
try testing.expectEqual(true, logged[0].blocked);
|
||||||
try testing.expectEqualStrings("ads.example.com", logged[0].domain());
|
try testing.expectEqualStrings("ads.example.com", logged[0].domain());
|
||||||
try testing.expectEqualStrings("rule_block_exact", logged[0].blockReason());
|
try testing.expectEqual(provenance.PolicyAction.block, logged[0].policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.rule_block_exact, logged[0].policy_reason);
|
||||||
try testing.expectEqualStrings("127.0.0.1", logged[0].clientIp());
|
try testing.expectEqualStrings("127.0.0.1", logged[0].clientIp());
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -593,7 +597,7 @@ test "S7 case 5: a cached answer comes back with a fresh id, an aged ttl and a l
|
|||||||
const logged = drainLog(&lg, io, &entries);
|
const logged = drainLog(&lg, io, &entries);
|
||||||
try testing.expectEqual(@as(usize, 3), logged.len);
|
try testing.expectEqual(@as(usize, 3), logged.len);
|
||||||
try testing.expectEqual(@as(?bool, false), logged[0].cache_hit);
|
try testing.expectEqual(@as(?bool, false), logged[0].cache_hit);
|
||||||
try testing.expectEqualStrings("pool", logged[0].upstream());
|
try testing.expectEqualStrings("fake://phase7-upstream", logged[0].upstream());
|
||||||
try testing.expectEqual(@as(?bool, true), logged[1].cache_hit);
|
try testing.expectEqual(@as(?bool, true), logged[1].cache_hit);
|
||||||
try testing.expectEqualStrings("", logged[1].upstream());
|
try testing.expectEqualStrings("", logged[1].upstream());
|
||||||
try testing.expectEqual(@as(?bool, true), logged[2].cache_hit);
|
try testing.expectEqual(@as(?bool, true), logged[2].cache_hit);
|
||||||
@@ -651,7 +655,9 @@ test "S7 case 6: a cname into a blocked target blocks the original question" {
|
|||||||
const logged = drainLog(&lg, io, &entries);
|
const logged = drainLog(&lg, io, &entries);
|
||||||
try testing.expectEqual(@as(usize, 1), logged.len);
|
try testing.expectEqual(@as(usize, 1), logged.len);
|
||||||
try testing.expectEqual(true, logged[0].blocked);
|
try testing.expectEqual(true, logged[0].blocked);
|
||||||
try testing.expectEqualStrings("cname:rule_block_exact", logged[0].blockReason());
|
// The reason describes the target's own decision; milestone 28 S3 adds the
|
||||||
|
// target name that says a CNAME chain was followed.
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.rule_block_exact, logged[0].policy_reason);
|
||||||
try testing.expectEqualStrings("cdn.example.com", logged[0].domain());
|
try testing.expectEqualStrings("cdn.example.com", logged[0].domain());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -114,8 +114,10 @@ const GoodUpstream = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
|
selected.* = "fake://good-upstream";
|
||||||
const self: *GoodUpstream = @ptrCast(@alignCast(ptr));
|
const self: *GoodUpstream = @ptrCast(@alignCast(ptr));
|
||||||
_ = self.calls.fetchAdd(1, .monotonic);
|
_ = self.calls.fetchAdd(1, .monotonic);
|
||||||
return answerQuery(query, response_buf);
|
return answerQuery(query, response_buf);
|
||||||
@@ -138,8 +140,10 @@ const FaultyUpstream = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
|
selected.* = "fake://faulty-upstream";
|
||||||
const self: *FaultyUpstream = @ptrCast(@alignCast(ptr));
|
const self: *FaultyUpstream = @ptrCast(@alignCast(ptr));
|
||||||
const seen = self.calls.fetchAdd(1, .monotonic);
|
const seen = self.calls.fetchAdd(1, .monotonic);
|
||||||
if (seen < self.fail_first) return self.fault;
|
if (seen < self.fail_first) return self.fault;
|
||||||
|
|||||||
@@ -72,8 +72,10 @@ const FakeUpstream = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
|
selected.* = "fake://tcp-server-upstream";
|
||||||
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
||||||
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
|
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
|
||||||
@memcpy(response_buf[0..self.reply.len], self.reply);
|
@memcpy(response_buf[0..self.reply.len], self.reply);
|
||||||
|
|||||||
@@ -73,8 +73,10 @@ const FakeUpstream = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
|
selected.* = "fake://udp-server-upstream";
|
||||||
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
||||||
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
|
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
|
||||||
@memcpy(response_buf[0..self.reply.len], self.reply);
|
@memcpy(response_buf[0..self.reply.len], self.reply);
|
||||||
|
|||||||
+416
-53
@@ -34,8 +34,12 @@ const std = @import("std");
|
|||||||
const db = @import("db.zig");
|
const db = @import("db.zig");
|
||||||
const disk_monitor = @import("disk_monitor.zig");
|
const disk_monitor = @import("disk_monitor.zig");
|
||||||
const events = @import("events.zig");
|
const events = @import("events.zig");
|
||||||
|
const limits = @import("../config/limits.zig");
|
||||||
const model = @import("../config/model.zig");
|
const model = @import("../config/model.zig");
|
||||||
|
const provenance = @import("provenance.zig");
|
||||||
const queries_repo = @import("repositories/queries_repo.zig");
|
const queries_repo = @import("repositories/queries_repo.zig");
|
||||||
|
const regex = @import("../filter/regex.zig");
|
||||||
|
const safe_url = @import("../safe_url.zig");
|
||||||
|
|
||||||
/// Named `scope` rather than `log`: `Logger.log` is the enqueue entry point,
|
/// Named `scope` rather than `log`: `Logger.log` is the enqueue entry point,
|
||||||
/// and the two names collide inside the struct.
|
/// and the two names collide inside the struct.
|
||||||
@@ -58,10 +62,32 @@ pub const gate_retry_s = 1;
|
|||||||
pub const max_domain_len = 253;
|
pub const max_domain_len = 253;
|
||||||
/// RFC 5952 text of any IPv6 address, zone identifier included.
|
/// RFC 5952 text of any IPv6 address, zone identifier included.
|
||||||
pub const max_client_len = 45;
|
pub const max_client_len = 45;
|
||||||
pub const max_reason_len = 32;
|
|
||||||
pub const max_upstream_len = 64;
|
/// `matched` holds the rule that decided the query, and the widest rule the
|
||||||
|
/// configuration accepts is a regex pattern at `regex.max_pattern_len`. It does
|
||||||
|
/// not fit a `u8` length, which is why this one field carries a `u16`.
|
||||||
|
pub const max_matched_len = regex.max_pattern_len;
|
||||||
|
|
||||||
|
/// The redacted resolver identity of the exchange that actually happened. Two
|
||||||
|
/// bounds apply and the buffer takes the larger, so neither producer truncates:
|
||||||
|
/// the longest well-formed `scheme://host:port` with a maximal host, and
|
||||||
|
/// `safe_url.redact`'s own output bound (`max_len` plus the `...` it appends
|
||||||
|
/// when it truncates).
|
||||||
|
pub const max_upstream_len = @max(
|
||||||
|
"https://".len + max_domain_len + ":65535".len,
|
||||||
|
safe_url.max_len + 3,
|
||||||
|
);
|
||||||
|
|
||||||
|
/// `cname_target`, `safe_search_target` and `forward_zone` each hold a domain
|
||||||
|
/// name, so they are all the same width as `domain`.
|
||||||
|
const max_name_len = max_domain_len;
|
||||||
|
|
||||||
/// One row on its way to `query_log`, carrying its own bytes.
|
/// One row on its way to `query_log`, carrying its own bytes.
|
||||||
|
///
|
||||||
|
/// Every field is by value: `Io.Queue` copies elements as raw bytes, so nothing
|
||||||
|
/// here may borrow from the query that produced it. The buffer widths above are
|
||||||
|
/// therefore the row's real storage cost, multiplied by the queue capacity —
|
||||||
|
/// see `query_log_buffer_max`.
|
||||||
pub const Entry = struct {
|
pub const Entry = struct {
|
||||||
timestamp: i64,
|
timestamp: i64,
|
||||||
domain_buf: [max_domain_len]u8,
|
domain_buf: [max_domain_len]u8,
|
||||||
@@ -69,28 +95,68 @@ pub const Entry = struct {
|
|||||||
client_buf: [max_client_len]u8,
|
client_buf: [max_client_len]u8,
|
||||||
client_len: u8,
|
client_len: u8,
|
||||||
qtype: ?u16,
|
qtype: ?u16,
|
||||||
|
qclass: u16,
|
||||||
|
/// The client-visible RCODE. Twelve bits, not four: an EDNS extended code
|
||||||
|
/// carries eight more bits in the OPT record than the header's four. The
|
||||||
|
/// type is the enforcement — the column's `CHECK` in `querylog_schema.ddl`
|
||||||
|
/// bounds the same value against every other writer of the file.
|
||||||
|
rcode: u12,
|
||||||
blocked: bool,
|
blocked: bool,
|
||||||
reason_buf: [max_reason_len]u8,
|
|
||||||
reason_len: u8,
|
|
||||||
response_time_us: ?i64,
|
response_time_us: ?i64,
|
||||||
cache_hit: ?bool,
|
cache_hit: ?bool,
|
||||||
upstream_buf: [max_upstream_len]u8,
|
upstream_buf: [max_upstream_len]u8,
|
||||||
upstream_len: u8,
|
upstream_len: u16,
|
||||||
|
|
||||||
|
group_id: ?i64,
|
||||||
|
group_buf: [limits.max_group_name_len]u8,
|
||||||
|
group_len: u8,
|
||||||
|
policy_action: provenance.PolicyAction,
|
||||||
|
policy_reason: provenance.PolicyReason,
|
||||||
|
matched_buf: [max_matched_len]u8,
|
||||||
|
matched_len: u16,
|
||||||
|
source_id: ?i64,
|
||||||
|
source_buf: [limits.max_source_name_len]u8,
|
||||||
|
source_len: u8,
|
||||||
|
cname_buf: [max_name_len]u8,
|
||||||
|
cname_len: u8,
|
||||||
|
safe_search_buf: [max_name_len]u8,
|
||||||
|
safe_search_len: u8,
|
||||||
|
route_kind: provenance.RouteKind,
|
||||||
|
forward_zone_buf: [max_name_len]u8,
|
||||||
|
forward_zone_len: u8,
|
||||||
|
|
||||||
/// The borrowed shape of an entry. `init` copies out of it, so a caller can
|
/// The borrowed shape of an entry. `init` copies out of it, so a caller can
|
||||||
/// build one from slices that die with the query.
|
/// build one from slices that die with the query.
|
||||||
|
///
|
||||||
|
/// Every text field defaults to `""`, which reaches a nullable column as
|
||||||
|
/// NULL. The three fields with no sensible empty value — the two enums and
|
||||||
|
/// the route — default to what a query the pipeline has not yet explained
|
||||||
|
/// would honestly say about itself.
|
||||||
pub const Fields = struct {
|
pub const Fields = struct {
|
||||||
timestamp: i64,
|
timestamp: i64,
|
||||||
domain: []const u8,
|
domain: []const u8,
|
||||||
client_ip: []const u8,
|
client_ip: []const u8,
|
||||||
qtype: ?u16 = null,
|
qtype: ?u16 = null,
|
||||||
|
qclass: u16 = 0,
|
||||||
|
rcode: u12 = 0,
|
||||||
blocked: bool = false,
|
blocked: bool = false,
|
||||||
/// Empty means "no reason", which reaches the database as NULL.
|
|
||||||
block_reason: []const u8 = "",
|
|
||||||
response_time_us: ?i64 = null,
|
response_time_us: ?i64 = null,
|
||||||
cache_hit: ?bool = null,
|
cache_hit: ?bool = null,
|
||||||
/// Empty means "no upstream", which reaches the database as NULL.
|
/// Empty means "no upstream was attempted", which reaches the database
|
||||||
|
/// as NULL. Already redacted by the caller.
|
||||||
upstream: []const u8 = "",
|
upstream: []const u8 = "",
|
||||||
|
|
||||||
|
group_id: ?i64 = null,
|
||||||
|
group_name: []const u8 = "",
|
||||||
|
policy_action: provenance.PolicyAction = .not_evaluated,
|
||||||
|
policy_reason: provenance.PolicyReason = .no_match,
|
||||||
|
matched: []const u8 = "",
|
||||||
|
source_id: ?i64 = null,
|
||||||
|
source_name: []const u8 = "",
|
||||||
|
cname_target: []const u8 = "",
|
||||||
|
safe_search_target: []const u8 = "",
|
||||||
|
route_kind: provenance.RouteKind = .upstream,
|
||||||
|
forward_zone: []const u8 = "",
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Copies each string in, truncated to what its buffer holds. A name longer
|
/// Copies each string in, truncated to what its buffer holds. A name longer
|
||||||
@@ -104,27 +170,61 @@ pub const Entry = struct {
|
|||||||
.client_buf = undefined,
|
.client_buf = undefined,
|
||||||
.client_len = 0,
|
.client_len = 0,
|
||||||
.qtype = f.qtype,
|
.qtype = f.qtype,
|
||||||
|
.qclass = f.qclass,
|
||||||
|
.rcode = f.rcode,
|
||||||
.blocked = f.blocked,
|
.blocked = f.blocked,
|
||||||
.reason_buf = undefined,
|
|
||||||
.reason_len = 0,
|
|
||||||
.response_time_us = f.response_time_us,
|
.response_time_us = f.response_time_us,
|
||||||
.cache_hit = f.cache_hit,
|
.cache_hit = f.cache_hit,
|
||||||
.upstream_buf = undefined,
|
.upstream_buf = undefined,
|
||||||
.upstream_len = 0,
|
.upstream_len = 0,
|
||||||
|
.group_id = f.group_id,
|
||||||
|
.group_buf = undefined,
|
||||||
|
.group_len = 0,
|
||||||
|
.policy_action = f.policy_action,
|
||||||
|
.policy_reason = f.policy_reason,
|
||||||
|
.matched_buf = undefined,
|
||||||
|
.matched_len = 0,
|
||||||
|
.source_id = f.source_id,
|
||||||
|
.source_buf = undefined,
|
||||||
|
.source_len = 0,
|
||||||
|
.cname_buf = undefined,
|
||||||
|
.cname_len = 0,
|
||||||
|
.safe_search_buf = undefined,
|
||||||
|
.safe_search_len = 0,
|
||||||
|
.route_kind = f.route_kind,
|
||||||
|
.forward_zone_buf = undefined,
|
||||||
|
.forward_zone_len = 0,
|
||||||
};
|
};
|
||||||
entry.setDomain(f.domain);
|
entry.setDomain(f.domain);
|
||||||
entry.setClientIp(f.client_ip);
|
entry.setClientIp(f.client_ip);
|
||||||
entry.reason_len = copyInto(&entry.reason_buf, f.block_reason);
|
copyInto(&entry.upstream_buf, &entry.upstream_len, f.upstream);
|
||||||
entry.upstream_len = copyInto(&entry.upstream_buf, f.upstream);
|
copyInto(&entry.group_buf, &entry.group_len, f.group_name);
|
||||||
|
entry.setMatched(f.matched);
|
||||||
|
copyInto(&entry.source_buf, &entry.source_len, f.source_name);
|
||||||
|
entry.setCnameTarget(f.cname_target);
|
||||||
|
entry.setSafeSearchTarget(f.safe_search_target);
|
||||||
|
copyInto(&entry.forward_zone_buf, &entry.forward_zone_len, f.forward_zone);
|
||||||
return entry;
|
return entry;
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn setDomain(self: *Entry, value: []const u8) void {
|
pub fn setDomain(self: *Entry, value: []const u8) void {
|
||||||
self.domain_len = copyInto(&self.domain_buf, value);
|
copyInto(&self.domain_buf, &self.domain_len, value);
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn setClientIp(self: *Entry, value: []const u8) void {
|
pub fn setClientIp(self: *Entry, value: []const u8) void {
|
||||||
self.client_len = copyInto(&self.client_buf, value);
|
copyInto(&self.client_buf, &self.client_len, value);
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn setMatched(self: *Entry, value: []const u8) void {
|
||||||
|
copyInto(&self.matched_buf, &self.matched_len, value);
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn setCnameTarget(self: *Entry, value: []const u8) void {
|
||||||
|
copyInto(&self.cname_buf, &self.cname_len, value);
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn setSafeSearchTarget(self: *Entry, value: []const u8) void {
|
||||||
|
copyInto(&self.safe_search_buf, &self.safe_search_len, value);
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn domain(self: *const Entry) []const u8 {
|
pub fn domain(self: *const Entry) []const u8 {
|
||||||
@@ -135,19 +235,56 @@ pub const Entry = struct {
|
|||||||
return self.client_buf[0..self.client_len];
|
return self.client_buf[0..self.client_len];
|
||||||
}
|
}
|
||||||
|
|
||||||
pub fn blockReason(self: *const Entry) []const u8 {
|
|
||||||
return self.reason_buf[0..self.reason_len];
|
|
||||||
}
|
|
||||||
|
|
||||||
pub fn upstream(self: *const Entry) []const u8 {
|
pub fn upstream(self: *const Entry) []const u8 {
|
||||||
return self.upstream_buf[0..self.upstream_len];
|
return self.upstream_buf[0..self.upstream_len];
|
||||||
}
|
}
|
||||||
|
|
||||||
|
pub fn groupName(self: *const Entry) []const u8 {
|
||||||
|
return self.group_buf[0..self.group_len];
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn matched(self: *const Entry) []const u8 {
|
||||||
|
return self.matched_buf[0..self.matched_len];
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn sourceName(self: *const Entry) []const u8 {
|
||||||
|
return self.source_buf[0..self.source_len];
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn cnameTarget(self: *const Entry) []const u8 {
|
||||||
|
return self.cname_buf[0..self.cname_len];
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn safeSearchTarget(self: *const Entry) []const u8 {
|
||||||
|
return self.safe_search_buf[0..self.safe_search_len];
|
||||||
|
}
|
||||||
|
|
||||||
|
pub fn forwardZone(self: *const Entry) []const u8 {
|
||||||
|
return self.forward_zone_buf[0..self.forward_zone_len];
|
||||||
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
fn copyInto(buf: []u8, value: []const u8) u8 {
|
/// The memory budget the queue is allowed to occupy. `Entry` travels by value,
|
||||||
|
/// so the composition root allocates `query_log_buffer_max` of them in full at
|
||||||
|
/// boot (`app.zig`) and the SSE hub embeds a ring of them per subscriber.
|
||||||
|
const queue_budget_bytes = 64 * 1024 * 1024;
|
||||||
|
|
||||||
|
/// The ceiling `config/validate.zig` enforces on `logging.query_log_buffer_max`,
|
||||||
|
/// derived from the width of `Entry` rather than picked.
|
||||||
|
///
|
||||||
|
/// The provenance columns of milestone 28 roughly tripled `Entry`, so the bound
|
||||||
|
/// that matters is bytes, not entries: an operator who asks for a million
|
||||||
|
/// entries is asking for well over a gigabyte of queue. This is a sanity bound,
|
||||||
|
/// not a memory-fit guarantee — what actually fits depends on the box.
|
||||||
|
pub const query_log_buffer_max: u32 = @intCast(queue_budget_bytes / @sizeOf(Entry));
|
||||||
|
|
||||||
|
/// Copies as much of `value` as `buf` holds, and stores the length through
|
||||||
|
/// `len`. `len`'s type never bounds anything — `buf.len` does — so the same
|
||||||
|
/// helper serves the `u8` fields and the `u16` ones.
|
||||||
|
fn copyInto(buf: []u8, len: anytype, value: []const u8) void {
|
||||||
const n = @min(buf.len, value.len);
|
const n = @min(buf.len, value.len);
|
||||||
@memcpy(buf[0..n], value[0..n]);
|
@memcpy(buf[0..n], value[0..n]);
|
||||||
return @intCast(n);
|
len.* = @intCast(n);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The row borrows from `entry`, which must outlive the `writeBatch` call.
|
/// The row borrows from `entry`, which must outlive the `writeBatch` call.
|
||||||
@@ -157,11 +294,23 @@ fn toRow(entry: *const Entry) queries_repo.Row {
|
|||||||
.domain = entry.domain(),
|
.domain = entry.domain(),
|
||||||
.client_ip = entry.clientIp(),
|
.client_ip = entry.clientIp(),
|
||||||
.qtype = entry.qtype,
|
.qtype = entry.qtype,
|
||||||
|
.qclass = entry.qclass,
|
||||||
|
.rcode = entry.rcode,
|
||||||
.blocked = entry.blocked,
|
.blocked = entry.blocked,
|
||||||
.block_reason = emptyAsNull(entry.blockReason()),
|
|
||||||
.response_time_us = entry.response_time_us,
|
.response_time_us = entry.response_time_us,
|
||||||
.cache_hit = entry.cache_hit,
|
.cache_hit = entry.cache_hit,
|
||||||
.upstream = emptyAsNull(entry.upstream()),
|
.upstream = emptyAsNull(entry.upstream()),
|
||||||
|
.group_id = entry.group_id,
|
||||||
|
.group_name = emptyAsNull(entry.groupName()),
|
||||||
|
.policy_action = entry.policy_action,
|
||||||
|
.policy_reason = entry.policy_reason,
|
||||||
|
.matched = emptyAsNull(entry.matched()),
|
||||||
|
.source_id = entry.source_id,
|
||||||
|
.source_name = emptyAsNull(entry.sourceName()),
|
||||||
|
.cname_target = emptyAsNull(entry.cnameTarget()),
|
||||||
|
.safe_search_target = emptyAsNull(entry.safeSearchTarget()),
|
||||||
|
.route_kind = entry.route_kind,
|
||||||
|
.forward_zone = emptyAsNull(entry.forwardZone()),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -231,9 +380,23 @@ pub const Logger = struct {
|
|||||||
/// and hands the result to every consumer, so nothing downstream — the
|
/// and hands the result to every consumer, so nothing downstream — the
|
||||||
/// database or the event stream — can observe a value the operator asked
|
/// database or the event stream — can observe a value the operator asked
|
||||||
/// to hide.
|
/// to hide.
|
||||||
|
/// `hide_domains` covers every field derived from the query name, not just
|
||||||
|
/// `domain`: a matched wildcard, a CNAME target and a safe-search target
|
||||||
|
/// each name the very thing the operator asked to keep out of the log.
|
||||||
|
///
|
||||||
|
/// `forward_zone`, `group_name` and `source_name` stay visible. They are
|
||||||
|
/// configuration labels the operator wrote, identical on every row that
|
||||||
|
/// hits them, and they say nothing about which name a client looked up.
|
||||||
pub fn transformed(self: *const Logger, entry: Entry) Entry {
|
pub fn transformed(self: *const Logger, entry: Entry) Entry {
|
||||||
var out = entry;
|
var out = entry;
|
||||||
if (self.cfg.hide_domains) out.setDomain(hidden_marker);
|
if (self.cfg.hide_domains) {
|
||||||
|
out.setDomain(hidden_marker);
|
||||||
|
// Only where there is something to hide: an empty field means the
|
||||||
|
// query had no such value, and writing a marker would claim it did.
|
||||||
|
if (out.matched_len != 0) out.setMatched(hidden_marker);
|
||||||
|
if (out.cname_len != 0) out.setCnameTarget(hidden_marker);
|
||||||
|
if (out.safe_search_len != 0) out.setSafeSearchTarget(hidden_marker);
|
||||||
|
}
|
||||||
if (self.cfg.hide_client_ips) out.setClientIp(hidden_marker);
|
if (self.cfg.hide_client_ips) out.setClientIp(hidden_marker);
|
||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
@@ -574,6 +737,34 @@ fn sampleEntry(timestamp: i64, domain: []const u8) Entry {
|
|||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Every provenance field set to a distinct recognisable value, so a test that
|
||||||
|
/// loses one loses it visibly.
|
||||||
|
fn fullFields(timestamp: i64) Entry.Fields {
|
||||||
|
return .{
|
||||||
|
.timestamp = timestamp,
|
||||||
|
.domain = "ads.example.com",
|
||||||
|
.client_ip = "2001:db8::1",
|
||||||
|
.qtype = 28,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 3,
|
||||||
|
.blocked = true,
|
||||||
|
.response_time_us = 42,
|
||||||
|
.cache_hit = true,
|
||||||
|
.upstream = "https://dns.example/dns-query",
|
||||||
|
.group_id = 7,
|
||||||
|
.group_name = "kids",
|
||||||
|
.policy_action = .block,
|
||||||
|
.policy_reason = .blocklist_wildcard,
|
||||||
|
.matched = "*.ads.example",
|
||||||
|
.source_id = 3,
|
||||||
|
.source_name = "steven black",
|
||||||
|
.cname_target = "tracker.cdn.example",
|
||||||
|
.safe_search_target = "forcesafesearch.google.com",
|
||||||
|
.route_kind = .blocked,
|
||||||
|
.forward_zone = "home.arpa",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
fn openLog() !db.Db {
|
fn openLog() !db.Db {
|
||||||
var database = try db.Db.open(":memory:", .{ .mode = .memory });
|
var database = try db.Db.open(":memory:", .{ .mode = .memory });
|
||||||
errdefer database.close();
|
errdefer database.close();
|
||||||
@@ -583,44 +774,72 @@ fn openLog() !db.Db {
|
|||||||
}
|
}
|
||||||
|
|
||||||
test "an entry carries its own bytes and reads them back" {
|
test "an entry carries its own bytes and reads them back" {
|
||||||
const entry: Entry = .init(.{
|
const entry: Entry = .init(fullFields(1700000000));
|
||||||
.timestamp = 1700000000,
|
|
||||||
.domain = "ads.example.com",
|
|
||||||
.client_ip = "2001:db8::1",
|
|
||||||
.qtype = 28,
|
|
||||||
.blocked = true,
|
|
||||||
.block_reason = "blocklist",
|
|
||||||
.response_time_us = 42,
|
|
||||||
.cache_hit = true,
|
|
||||||
.upstream = "dns.example",
|
|
||||||
});
|
|
||||||
|
|
||||||
try testing.expectEqualStrings("ads.example.com", entry.domain());
|
try testing.expectEqualStrings("ads.example.com", entry.domain());
|
||||||
try testing.expectEqualStrings("2001:db8::1", entry.clientIp());
|
try testing.expectEqualStrings("2001:db8::1", entry.clientIp());
|
||||||
try testing.expectEqualStrings("blocklist", entry.blockReason());
|
try testing.expectEqualStrings("https://dns.example/dns-query", entry.upstream());
|
||||||
try testing.expectEqualStrings("dns.example", entry.upstream());
|
|
||||||
try testing.expectEqual(@as(?u16, 28), entry.qtype);
|
try testing.expectEqual(@as(?u16, 28), entry.qtype);
|
||||||
|
try testing.expectEqual(@as(u16, 1), entry.qclass);
|
||||||
|
try testing.expectEqual(@as(u12, 3), entry.rcode);
|
||||||
try testing.expect(entry.blocked);
|
try testing.expect(entry.blocked);
|
||||||
try testing.expectEqual(@as(?i64, 42), entry.response_time_us);
|
try testing.expectEqual(@as(?i64, 42), entry.response_time_us);
|
||||||
try testing.expectEqual(@as(?bool, true), entry.cache_hit);
|
try testing.expectEqual(@as(?bool, true), entry.cache_hit);
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(?i64, 7), entry.group_id);
|
||||||
|
try testing.expectEqualStrings("kids", entry.groupName());
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.block, entry.policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, entry.policy_reason);
|
||||||
|
try testing.expectEqualStrings("*.ads.example", entry.matched());
|
||||||
|
try testing.expectEqual(@as(?i64, 3), entry.source_id);
|
||||||
|
try testing.expectEqualStrings("steven black", entry.sourceName());
|
||||||
|
try testing.expectEqualStrings("tracker.cdn.example", entry.cnameTarget());
|
||||||
|
try testing.expectEqualStrings("forcesafesearch.google.com", entry.safeSearchTarget());
|
||||||
|
try testing.expectEqual(provenance.RouteKind.blocked, entry.route_kind);
|
||||||
|
try testing.expectEqualStrings("home.arpa", entry.forwardZone());
|
||||||
}
|
}
|
||||||
|
|
||||||
test "an oversize string is truncated to what its buffer holds" {
|
test "an oversize string is truncated to what its buffer holds" {
|
||||||
const long_domain = "a" ** 400;
|
|
||||||
const entry: Entry = .init(.{
|
const entry: Entry = .init(.{
|
||||||
.timestamp = 1,
|
.timestamp = 1,
|
||||||
.domain = long_domain,
|
.domain = "a" ** 400,
|
||||||
.client_ip = "192.0.2.1",
|
.client_ip = "c" ** 80,
|
||||||
.block_reason = "r" ** 64,
|
.upstream = "u" ** 600,
|
||||||
.upstream = "u" ** 128,
|
.group_name = "g" ** 200,
|
||||||
|
.matched = "m" ** 600,
|
||||||
|
.source_name = "s" ** 200,
|
||||||
|
.cname_target = "n" ** 400,
|
||||||
|
.safe_search_target = "f" ** 400,
|
||||||
|
.forward_zone = "z" ** 400,
|
||||||
});
|
});
|
||||||
|
|
||||||
try testing.expectEqual(@as(usize, max_domain_len), entry.domain().len);
|
try testing.expectEqual(@as(usize, max_domain_len), entry.domain().len);
|
||||||
try testing.expectEqual(@as(usize, max_reason_len), entry.blockReason().len);
|
try testing.expectEqual(@as(usize, max_client_len), entry.clientIp().len);
|
||||||
try testing.expectEqual(@as(usize, max_upstream_len), entry.upstream().len);
|
try testing.expectEqual(@as(usize, max_upstream_len), entry.upstream().len);
|
||||||
|
try testing.expectEqual(@as(usize, limits.max_group_name_len), entry.groupName().len);
|
||||||
|
try testing.expectEqual(@as(usize, max_matched_len), entry.matched().len);
|
||||||
|
try testing.expectEqual(@as(usize, limits.max_source_name_len), entry.sourceName().len);
|
||||||
|
try testing.expectEqual(@as(usize, max_name_len), entry.cnameTarget().len);
|
||||||
|
try testing.expectEqual(@as(usize, max_name_len), entry.safeSearchTarget().len);
|
||||||
|
try testing.expectEqual(@as(usize, max_name_len), entry.forwardZone().len);
|
||||||
try testing.expectEqualStrings("a" ** max_domain_len, entry.domain());
|
try testing.expectEqualStrings("a" ** max_domain_len, entry.domain());
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "a 256-byte matched pattern is stored whole" {
|
||||||
|
// The widest rule the configuration accepts is a regex at
|
||||||
|
// `regex.max_pattern_len`, and it does not fit a `u8` length — which is the
|
||||||
|
// whole reason `matched_len` is a `u16`.
|
||||||
|
const widest = "p" ** regex.max_pattern_len;
|
||||||
|
const entry: Entry = .init(.{
|
||||||
|
.timestamp = 1,
|
||||||
|
.domain = "example.com",
|
||||||
|
.client_ip = "192.0.2.1",
|
||||||
|
.matched = widest,
|
||||||
|
});
|
||||||
|
try testing.expectEqualStrings(widest, entry.matched());
|
||||||
|
try testing.expectEqual(@as(u16, regex.max_pattern_len), entry.matched_len);
|
||||||
|
}
|
||||||
|
|
||||||
test "toRow maps the empty strings to null and passes the rest through" {
|
test "toRow maps the empty strings to null and passes the rest through" {
|
||||||
const bare: Entry = .init(.{
|
const bare: Entry = .init(.{
|
||||||
.timestamp = 7,
|
.timestamp = 7,
|
||||||
@@ -631,23 +850,85 @@ test "toRow maps the empty strings to null and passes the rest through" {
|
|||||||
try testing.expectEqual(@as(i64, 7), bare_row.timestamp);
|
try testing.expectEqual(@as(i64, 7), bare_row.timestamp);
|
||||||
try testing.expectEqualStrings("example.com", bare_row.domain);
|
try testing.expectEqualStrings("example.com", bare_row.domain);
|
||||||
try testing.expectEqualStrings("192.0.2.5", bare_row.client_ip);
|
try testing.expectEqualStrings("192.0.2.5", bare_row.client_ip);
|
||||||
try testing.expectEqual(@as(?[]const u8, null), bare_row.block_reason);
|
|
||||||
try testing.expectEqual(@as(?[]const u8, null), bare_row.upstream);
|
|
||||||
try testing.expectEqual(@as(?u16, null), bare_row.qtype);
|
try testing.expectEqual(@as(?u16, null), bare_row.qtype);
|
||||||
try testing.expectEqual(@as(?bool, null), bare_row.cache_hit);
|
try testing.expectEqual(@as(?bool, null), bare_row.cache_hit);
|
||||||
|
// Every optional text field of an entry nothing filled in reaches its
|
||||||
|
// column as NULL rather than as an empty string.
|
||||||
|
try testing.expectEqual(@as(?[]const u8, null), bare_row.upstream);
|
||||||
|
try testing.expectEqual(@as(?[]const u8, null), bare_row.group_name);
|
||||||
|
try testing.expectEqual(@as(?[]const u8, null), bare_row.matched);
|
||||||
|
try testing.expectEqual(@as(?[]const u8, null), bare_row.source_name);
|
||||||
|
try testing.expectEqual(@as(?[]const u8, null), bare_row.cname_target);
|
||||||
|
try testing.expectEqual(@as(?[]const u8, null), bare_row.safe_search_target);
|
||||||
|
try testing.expectEqual(@as(?[]const u8, null), bare_row.forward_zone);
|
||||||
|
|
||||||
const full: Entry = .init(.{
|
const full: Entry = .init(fullFields(8));
|
||||||
.timestamp = 8,
|
|
||||||
.domain = "blocked.example",
|
|
||||||
.client_ip = "192.0.2.6",
|
|
||||||
.blocked = true,
|
|
||||||
.block_reason = "blocklist",
|
|
||||||
.upstream = "9.9.9.9",
|
|
||||||
});
|
|
||||||
const full_row = toRow(&full);
|
const full_row = toRow(&full);
|
||||||
try testing.expect(full_row.blocked);
|
try testing.expect(full_row.blocked);
|
||||||
try testing.expectEqualStrings("blocklist", full_row.block_reason.?);
|
try testing.expectEqual(@as(u16, 1), full_row.qclass);
|
||||||
try testing.expectEqualStrings("9.9.9.9", full_row.upstream.?);
|
try testing.expectEqual(@as(u12, 3), full_row.rcode);
|
||||||
|
try testing.expectEqualStrings("https://dns.example/dns-query", full_row.upstream.?);
|
||||||
|
try testing.expectEqual(@as(?i64, 7), full_row.group_id);
|
||||||
|
try testing.expectEqualStrings("kids", full_row.group_name.?);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.block, full_row.policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, full_row.policy_reason);
|
||||||
|
try testing.expectEqualStrings("*.ads.example", full_row.matched.?);
|
||||||
|
try testing.expectEqual(@as(?i64, 3), full_row.source_id);
|
||||||
|
try testing.expectEqualStrings("steven black", full_row.source_name.?);
|
||||||
|
try testing.expectEqualStrings("tracker.cdn.example", full_row.cname_target.?);
|
||||||
|
try testing.expectEqualStrings("forcesafesearch.google.com", full_row.safe_search_target.?);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.blocked, full_row.route_kind);
|
||||||
|
try testing.expectEqualStrings("home.arpa", full_row.forward_zone.?);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "an entry with every provenance field set survives the queue, toRow, insert and detailById" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
var writer = try queries_repo.BatchWriter.init(&database);
|
||||||
|
defer writer.deinit();
|
||||||
|
|
||||||
|
var buf: [4]Entry = undefined;
|
||||||
|
var logger: Logger = .init(.{}, &buf);
|
||||||
|
|
||||||
|
// The widest `matched` the configuration accepts, carried the whole way:
|
||||||
|
// 256 bytes does not fit the `u8` length every other text field uses.
|
||||||
|
const widest_matched = "p" ** max_matched_len;
|
||||||
|
var fields = fullFields(1234);
|
||||||
|
fields.matched = widest_matched;
|
||||||
|
logger.log(io, .init(fields));
|
||||||
|
|
||||||
|
const queued = try logger.queue.getOne(io);
|
||||||
|
try logger.flush(io, &writer, &.{queued}, null);
|
||||||
|
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
const stored = (try queries_repo.detailById(&database, arena_state.allocator(), 1)).?;
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, 1234), stored.ts);
|
||||||
|
try testing.expectEqualStrings("ads.example.com", stored.domain);
|
||||||
|
try testing.expectEqualStrings("2001:db8::1", stored.client_ip);
|
||||||
|
try testing.expectEqual(@as(?u16, 28), stored.qtype);
|
||||||
|
try testing.expectEqual(@as(u16, 1), stored.qclass);
|
||||||
|
try testing.expectEqual(@as(u12, 3), stored.rcode);
|
||||||
|
try testing.expect(stored.blocked);
|
||||||
|
try testing.expectEqual(@as(?i64, 42), stored.response_time_us);
|
||||||
|
try testing.expectEqual(@as(?bool, true), stored.cache_hit);
|
||||||
|
try testing.expectEqualStrings("https://dns.example/dns-query", stored.upstream);
|
||||||
|
try testing.expectEqual(@as(?i64, 7), stored.group_id);
|
||||||
|
try testing.expectEqualStrings("kids", stored.group_name);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.block, stored.policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, stored.policy_reason);
|
||||||
|
try testing.expectEqualStrings(widest_matched, stored.matched);
|
||||||
|
try testing.expectEqual(@as(?i64, 3), stored.source_id);
|
||||||
|
try testing.expectEqualStrings("steven black", stored.source_name);
|
||||||
|
try testing.expectEqualStrings("tracker.cdn.example", stored.cname_target);
|
||||||
|
try testing.expectEqualStrings("forcesafesearch.google.com", stored.safe_search_target);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.blocked, stored.route_kind);
|
||||||
|
try testing.expectEqualStrings("home.arpa", stored.forward_zone);
|
||||||
}
|
}
|
||||||
|
|
||||||
test "log applies both privacy transforms before the entry reaches the queue" {
|
test "log applies both privacy transforms before the entry reaches the queue" {
|
||||||
@@ -667,6 +948,88 @@ test "log applies both privacy transforms before the entry reaches the queue" {
|
|||||||
try testing.expectEqual(@as(u64, 0), logger.queries_dropped.load(.monotonic));
|
try testing.expectEqual(@as(u64, 0), logger.queries_dropped.load(.monotonic));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "hide_domains hides every query-derived name and leaves the labels alone" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var buf: [4]Entry = undefined;
|
||||||
|
var logger: Logger = .init(.{ .hide_domains = true }, &buf);
|
||||||
|
logger.log(io, .init(fullFields(1)));
|
||||||
|
const hidden = try logger.queue.getOne(io);
|
||||||
|
|
||||||
|
// Every field derived from the name the client asked for.
|
||||||
|
try testing.expectEqualStrings(hidden_marker, hidden.domain());
|
||||||
|
try testing.expectEqualStrings(hidden_marker, hidden.matched());
|
||||||
|
try testing.expectEqualStrings(hidden_marker, hidden.cnameTarget());
|
||||||
|
try testing.expectEqualStrings(hidden_marker, hidden.safeSearchTarget());
|
||||||
|
|
||||||
|
// The client is governed by `hide_client_ips`, not by this flag.
|
||||||
|
try testing.expectEqualStrings("2001:db8::1", hidden.clientIp());
|
||||||
|
|
||||||
|
// Configuration labels the operator wrote. They are identical on every row
|
||||||
|
// that hits them and say nothing about which name a client looked up.
|
||||||
|
try testing.expectEqualStrings("kids", hidden.groupName());
|
||||||
|
try testing.expectEqualStrings("steven black", hidden.sourceName());
|
||||||
|
try testing.expectEqualStrings("home.arpa", hidden.forwardZone());
|
||||||
|
try testing.expectEqualStrings("https://dns.example/dns-query", hidden.upstream());
|
||||||
|
}
|
||||||
|
|
||||||
|
test "hide_client_ips hides the client and nothing else" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var buf: [4]Entry = undefined;
|
||||||
|
var logger: Logger = .init(.{ .hide_client_ips = true }, &buf);
|
||||||
|
logger.log(io, .init(fullFields(1)));
|
||||||
|
const hidden = try logger.queue.getOne(io);
|
||||||
|
|
||||||
|
try testing.expectEqualStrings(hidden_marker, hidden.clientIp());
|
||||||
|
try testing.expectEqualStrings("ads.example.com", hidden.domain());
|
||||||
|
try testing.expectEqualStrings("*.ads.example", hidden.matched());
|
||||||
|
try testing.expectEqualStrings("tracker.cdn.example", hidden.cnameTarget());
|
||||||
|
try testing.expectEqualStrings("forcesafesearch.google.com", hidden.safeSearchTarget());
|
||||||
|
try testing.expectEqualStrings("kids", hidden.groupName());
|
||||||
|
try testing.expectEqualStrings("steven black", hidden.sourceName());
|
||||||
|
try testing.expectEqualStrings("home.arpa", hidden.forwardZone());
|
||||||
|
}
|
||||||
|
|
||||||
|
test "hide_domains writes no marker into a field the query never had" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var buf: [4]Entry = undefined;
|
||||||
|
var logger: Logger = .init(.{ .hide_domains = true }, &buf);
|
||||||
|
// An ordinary allowed query: no rule matched, no CNAME was uncloaked, no
|
||||||
|
// safe-search rewrite happened. Marking those "hidden" would claim the
|
||||||
|
// query had values it did not.
|
||||||
|
logger.log(io, sampleEntry(1, "plain.example"));
|
||||||
|
const hidden = try logger.queue.getOne(io);
|
||||||
|
|
||||||
|
try testing.expectEqualStrings(hidden_marker, hidden.domain());
|
||||||
|
try testing.expectEqualStrings("", hidden.matched());
|
||||||
|
try testing.expectEqualStrings("", hidden.cnameTarget());
|
||||||
|
try testing.expectEqualStrings("", hidden.safeSearchTarget());
|
||||||
|
}
|
||||||
|
|
||||||
|
test "the entry queue's worst case stays inside its byte budget" {
|
||||||
|
// The bound `config/validate.zig` enforces is derived from this, so the
|
||||||
|
// budget is what a maximal configuration can actually cost.
|
||||||
|
try testing.expect(@as(usize, query_log_buffer_max) * @sizeOf(Entry) <= queue_budget_bytes);
|
||||||
|
// One more entry than the ceiling would exceed it, so the ceiling is the
|
||||||
|
// largest value that fits rather than a round number under it.
|
||||||
|
try testing.expect((@as(usize, query_log_buffer_max) + 1) * @sizeOf(Entry) > queue_budget_bytes);
|
||||||
|
|
||||||
|
// The shipped default has to be comfortably inside the budget, or the
|
||||||
|
// out-of-the-box configuration is the one that spends it. At the widths
|
||||||
|
// above it costs about 17 MiB, roughly a quarter of the ceiling.
|
||||||
|
const default_max: usize = (model.Logging{}).query_log_buffer_max;
|
||||||
|
try testing.expect(default_max <= query_log_buffer_max);
|
||||||
|
try testing.expect(default_max * @sizeOf(Entry) <= queue_budget_bytes / 2);
|
||||||
|
}
|
||||||
|
|
||||||
test "log hides only the field its switch names" {
|
test "log hides only the field its switch names" {
|
||||||
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
defer threaded.deinit();
|
defer threaded.deinit();
|
||||||
|
|||||||
@@ -154,11 +154,23 @@ fn writeRows(database: *db.Db, timestamps: []const i64, domain: []const u8) !voi
|
|||||||
.domain = domain,
|
.domain = domain,
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = false,
|
.blocked = false,
|
||||||
.block_reason = null,
|
|
||||||
.response_time_us = null,
|
.response_time_us = null,
|
||||||
.cache_hit = null,
|
.cache_hit = null,
|
||||||
.upstream = null,
|
.upstream = null,
|
||||||
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = .allow,
|
||||||
|
.policy_reason = .no_match,
|
||||||
|
.matched = null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
try writer.writeBatch(rows[0..timestamps.len]);
|
try writer.writeBatch(rows[0..timestamps.len]);
|
||||||
|
|||||||
@@ -0,0 +1,137 @@
|
|||||||
|
//! The closed enums a `query_log` row stores to explain one query: what the
|
||||||
|
//! policy decided, why, and where the answer came from.
|
||||||
|
//!
|
||||||
|
//! They live in a module of their own because everything that touches a logged
|
||||||
|
//! row needs them — `storage/logger.zig`, `storage/repositories/queries_repo.zig`,
|
||||||
|
//! `server/handler.zig` and the web serializers — and `logger` already imports
|
||||||
|
//! `queries_repo`, so enums owned by either would close a loop.
|
||||||
|
//!
|
||||||
|
//! Each value is stored as its `@tagName` and read back through `parse`. The
|
||||||
|
//! read path treats an unrecognised value as a data error rather than passing
|
||||||
|
//! the text through: the column is a closed set, and a row that disagrees came
|
||||||
|
//! from something other than this schema.
|
||||||
|
|
||||||
|
const std = @import("std");
|
||||||
|
|
||||||
|
const matcher = @import("../filter/matcher.zig");
|
||||||
|
|
||||||
|
/// Whether the filtering policy reached a verdict on this query, and which one.
|
||||||
|
///
|
||||||
|
/// `not_evaluated` is the honest answer for a query the pipeline answered before
|
||||||
|
/// filtering could apply — a non-IN question, a paused resolver, a protocol
|
||||||
|
/// refusal — and is not the same as "allowed".
|
||||||
|
pub const PolicyAction = enum {
|
||||||
|
not_evaluated,
|
||||||
|
allow,
|
||||||
|
block,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Why the policy landed where it did.
|
||||||
|
///
|
||||||
|
/// The first nine are `filter/matcher.zig`'s serializable reasons, one for one.
|
||||||
|
/// The rest name the pipeline steps that decide a query without consulting the
|
||||||
|
/// matcher at all.
|
||||||
|
pub const PolicyReason = enum {
|
||||||
|
rule_allow_exact,
|
||||||
|
rule_block_exact,
|
||||||
|
rule_allow_wildcard,
|
||||||
|
rule_block_wildcard,
|
||||||
|
rule_allow_regex,
|
||||||
|
rule_block_regex,
|
||||||
|
blocklist_exception,
|
||||||
|
blocklist_domain,
|
||||||
|
blocklist_wildcard,
|
||||||
|
|
||||||
|
/// Answered from `local_records`, before filtering.
|
||||||
|
local_record,
|
||||||
|
/// Answered by a configured forward zone, before filtering.
|
||||||
|
forward_zone,
|
||||||
|
/// The question was not class IN, so no rule could apply to it.
|
||||||
|
non_in_class,
|
||||||
|
/// Filtering was paused.
|
||||||
|
paused,
|
||||||
|
/// No filter snapshot was published yet, so the query went unfiltered.
|
||||||
|
snapshot_unavailable,
|
||||||
|
/// The matcher evaluated the name and nothing matched.
|
||||||
|
no_match,
|
||||||
|
/// A syntactically parsed request refused on protocol grounds — BADVERS,
|
||||||
|
/// NOTIMP, a malformed EDNS OPT. It names a question, so it is logged, but
|
||||||
|
/// no policy ever saw it.
|
||||||
|
protocol_error,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Where the answer the client received came from.
|
||||||
|
pub const RouteKind = enum {
|
||||||
|
blocked,
|
||||||
|
local,
|
||||||
|
forward_zone,
|
||||||
|
upstream,
|
||||||
|
cache,
|
||||||
|
rejected,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// The matcher's verdict in the query log's vocabulary.
|
||||||
|
///
|
||||||
|
/// Exhaustive on purpose: a reason added to the matcher must be given a stored
|
||||||
|
/// name here rather than silently reaching a row as something else. `.none` is
|
||||||
|
/// the matcher's "nothing matched", which is exactly `no_match`.
|
||||||
|
pub fn fromMatcherReason(reason: matcher.Reason) PolicyReason {
|
||||||
|
return switch (reason) {
|
||||||
|
.none => .no_match,
|
||||||
|
.rule_allow_exact => .rule_allow_exact,
|
||||||
|
.rule_block_exact => .rule_block_exact,
|
||||||
|
.rule_allow_wildcard => .rule_allow_wildcard,
|
||||||
|
.rule_block_wildcard => .rule_block_wildcard,
|
||||||
|
.rule_allow_regex => .rule_allow_regex,
|
||||||
|
.rule_block_regex => .rule_block_regex,
|
||||||
|
.blocklist_exception => .blocklist_exception,
|
||||||
|
.blocklist_domain => .blocklist_domain,
|
||||||
|
.blocklist_wildcard => .blocklist_wildcard,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reads a stored `@tagName` back. `error.Mismatch` is the same error the
|
||||||
|
/// repositories return for a column that does not hold what the schema says it
|
||||||
|
/// holds, which is what an unknown value here is.
|
||||||
|
pub fn parse(comptime Enum: type, text: []const u8) error{Mismatch}!Enum {
|
||||||
|
return std.meta.stringToEnum(Enum, text) orelse error.Mismatch;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
const testing = std.testing;
|
||||||
|
|
||||||
|
test "every matcher reason has a stored name" {
|
||||||
|
// The mapping is checked here rather than trusted: a reason added to the
|
||||||
|
// matcher fails the exhaustive switch at compile time, and a reason
|
||||||
|
// *renamed* would still compile while changing what a row says.
|
||||||
|
inline for (@typeInfo(matcher.Reason).@"enum".fields) |field| {
|
||||||
|
const reason: matcher.Reason = @enumFromInt(field.value);
|
||||||
|
const mapped = fromMatcherReason(reason);
|
||||||
|
if (reason == .none) {
|
||||||
|
try testing.expectEqual(PolicyReason.no_match, mapped);
|
||||||
|
} else {
|
||||||
|
try testing.expectEqualStrings(field.name, @tagName(mapped));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test "parse round-trips every value of every enum" {
|
||||||
|
inline for ([_]type{ PolicyAction, PolicyReason, RouteKind }) |Enum| {
|
||||||
|
inline for (@typeInfo(Enum).@"enum".fields) |field| {
|
||||||
|
const value: Enum = @enumFromInt(field.value);
|
||||||
|
try testing.expectEqual(value, try parse(Enum, @tagName(value)));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test "parse rejects a value the schema does not define" {
|
||||||
|
try testing.expectError(error.Mismatch, parse(PolicyAction, "allowed"));
|
||||||
|
try testing.expectError(error.Mismatch, parse(PolicyAction, ""));
|
||||||
|
// A value that belongs to a different one of the three enums is no more
|
||||||
|
// acceptable than a typo.
|
||||||
|
try testing.expectError(error.Mismatch, parse(RouteKind, "allow"));
|
||||||
|
try testing.expectError(error.Mismatch, parse(PolicyReason, "cache"));
|
||||||
|
}
|
||||||
@@ -22,8 +22,20 @@ const db = @import("db.zig");
|
|||||||
|
|
||||||
const log = std.log.scoped(.querylog_schema);
|
const log = std.log.scoped(.querylog_schema);
|
||||||
|
|
||||||
/// Verbatim from PLAN §11.3. Multi-statement text — it goes through
|
/// PLAN §11.3, plus the coverage watermark of milestone 28. Multi-statement
|
||||||
/// `db.Db.exec`, never through `prepare`.
|
/// text — it goes through `db.Db.exec`, never through `prepare`.
|
||||||
|
///
|
||||||
|
/// The trailing INSERT seeds `querylog_meta`, which is part of the schema
|
||||||
|
/// rather than a later step: a `query_log` with no watermark beside it cannot
|
||||||
|
/// answer whether an empty result means "no queries" or "no history", and every
|
||||||
|
/// database this program reads from is created by executing this string.
|
||||||
|
/// `unixepoch()` is SQLite's own UTC clock, which is the clock every
|
||||||
|
/// `timestamp` in the file is measured against.
|
||||||
|
///
|
||||||
|
/// `available_since` starts one second *after* `created_at` on purpose. A row
|
||||||
|
/// logged in the same second the file was created is not evidence that the
|
||||||
|
/// second is completely covered, and the watermark's whole job is to be
|
||||||
|
/// conservative. From there it only ever advances, in `queries_repo.pruneOlderThan`.
|
||||||
pub const ddl: [:0]const u8 =
|
pub const ddl: [:0]const u8 =
|
||||||
\\CREATE TABLE domains (
|
\\CREATE TABLE domains (
|
||||||
\\ id INTEGER PRIMARY KEY,
|
\\ id INTEGER PRIMARY KEY,
|
||||||
@@ -37,10 +49,23 @@ pub const ddl: [:0]const u8 =
|
|||||||
\\ client_ip TEXT NOT NULL, -- text, not a FK: log rows are immutable facts
|
\\ client_ip TEXT NOT NULL, -- text, not a FK: log rows are immutable facts
|
||||||
\\ qtype INTEGER,
|
\\ qtype INTEGER,
|
||||||
\\ blocked INTEGER NOT NULL,
|
\\ blocked INTEGER NOT NULL,
|
||||||
\\ block_reason TEXT,
|
|
||||||
\\ response_time_us INTEGER,
|
\\ response_time_us INTEGER,
|
||||||
\\ cache_hit INTEGER,
|
\\ cache_hit INTEGER,
|
||||||
\\ upstream TEXT
|
\\ upstream TEXT,
|
||||||
|
\\ qclass INTEGER NOT NULL,
|
||||||
|
\\ rcode INTEGER NOT NULL,
|
||||||
|
\\ group_id INTEGER, -- text/id pairs, not FKs: a renamed
|
||||||
|
\\ group_name TEXT, -- group must not rewrite history
|
||||||
|
\\ policy_action TEXT NOT NULL,
|
||||||
|
\\ policy_reason TEXT NOT NULL,
|
||||||
|
\\ matched TEXT,
|
||||||
|
\\ source_id INTEGER,
|
||||||
|
\\ source_name TEXT,
|
||||||
|
\\ cname_target TEXT,
|
||||||
|
\\ safe_search_target TEXT,
|
||||||
|
\\ route_kind TEXT NOT NULL,
|
||||||
|
\\ forward_zone TEXT,
|
||||||
|
\\ CHECK (rcode BETWEEN 0 AND 4095) -- twelve bits (RFC 6891 6.1.3)
|
||||||
\\);
|
\\);
|
||||||
\\CREATE INDEX idx_query_log_ts ON query_log(timestamp);
|
\\CREATE INDEX idx_query_log_ts ON query_log(timestamp);
|
||||||
\\CREATE INDEX idx_query_log_client ON query_log(client_ip);
|
\\CREATE INDEX idx_query_log_client ON query_log(client_ip);
|
||||||
@@ -63,6 +88,14 @@ pub const ddl: [:0]const u8 =
|
|||||||
\\ CHECK (failures >= 0)
|
\\ CHECK (failures >= 0)
|
||||||
\\) WITHOUT ROWID;
|
\\) WITHOUT ROWID;
|
||||||
\\CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
|
\\CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
|
||||||
|
\\
|
||||||
|
\\CREATE TABLE querylog_meta (
|
||||||
|
\\ id INTEGER PRIMARY KEY CHECK (id = 1), -- one row, enforced by the schema
|
||||||
|
\\ created_at INTEGER NOT NULL,
|
||||||
|
\\ available_since INTEGER NOT NULL
|
||||||
|
\\);
|
||||||
|
\\INSERT INTO querylog_meta (id, created_at, available_since)
|
||||||
|
\\VALUES (1, unixepoch(), unixepoch() + 1);
|
||||||
;
|
;
|
||||||
|
|
||||||
/// `PRAGMA user_version` is a signed 32-bit field. Deriving the fingerprint from
|
/// `PRAGMA user_version` is a signed 32-bit field. Deriving the fingerprint from
|
||||||
@@ -299,7 +332,7 @@ test "ddl creates the query-log tables, the upstream-history tables and every in
|
|||||||
try database.exec(ddl);
|
try database.exec(ddl);
|
||||||
|
|
||||||
try testing.expectEqual(
|
try testing.expectEqual(
|
||||||
@as(i64, 4),
|
@as(i64, 5),
|
||||||
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
|
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
|
||||||
);
|
);
|
||||||
const objects = [_][]const u8{
|
const objects = [_][]const u8{
|
||||||
@@ -307,6 +340,7 @@ test "ddl creates the query-log tables, the upstream-history tables and every in
|
|||||||
"idx_query_log_ts", "idx_query_log_client",
|
"idx_query_log_ts", "idx_query_log_client",
|
||||||
"idx_query_log_domain", "upstream_targets",
|
"idx_query_log_domain", "upstream_targets",
|
||||||
"upstream_minute", "idx_upstream_minute_ts",
|
"upstream_minute", "idx_upstream_minute_ts",
|
||||||
|
"querylog_meta",
|
||||||
};
|
};
|
||||||
for (objects) |name| {
|
for (objects) |name| {
|
||||||
var stmt = try database.prepare("SELECT count(*) FROM sqlite_schema WHERE name = ?1");
|
var stmt = try database.prepare("SELECT count(*) FROM sqlite_schema WHERE name = ?1");
|
||||||
@@ -317,6 +351,69 @@ test "ddl creates the query-log tables, the upstream-history tables and every in
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "the schema refuses an rcode outside twelve bits" {
|
||||||
|
var database = try db.Db.open(":memory:", .{ .mode = .memory });
|
||||||
|
defer database.close();
|
||||||
|
try db.applyPragmas(&database, .{});
|
||||||
|
try database.exec(ddl);
|
||||||
|
try database.exec("INSERT INTO domains (id, domain) VALUES (1, 'a.example');");
|
||||||
|
|
||||||
|
var stmt = try database.prepare(
|
||||||
|
\\INSERT INTO query_log
|
||||||
|
\\ (timestamp, domain_id, client_ip, blocked, qclass, rcode,
|
||||||
|
\\ policy_action, policy_reason, route_kind)
|
||||||
|
\\VALUES (1, 1, '10.0.0.1', 0, 1, ?1, 'not_evaluated', 'no_match', 'upstream')
|
||||||
|
);
|
||||||
|
defer stmt.deinit();
|
||||||
|
|
||||||
|
// The whole range an EDNS extended RCODE can express, and nothing wider:
|
||||||
|
// the producers are `u12`, and this is what stops any other writer — a
|
||||||
|
// hand-run UPDATE included — from putting a value in the column that the
|
||||||
|
// read path would have to reject.
|
||||||
|
for ([_]i64{ 0, 4095 }) |accepted| {
|
||||||
|
try stmt.reset();
|
||||||
|
try stmt.bindInt(1, accepted);
|
||||||
|
try stmt.exec();
|
||||||
|
}
|
||||||
|
for ([_]i64{ -1, 4096, 65535 }) |refused| {
|
||||||
|
// `sqlite3_reset` repeats the error of the statement it is resetting,
|
||||||
|
// which for every iteration after the first is the constraint failure
|
||||||
|
// this loop just asserted — the same reason `BatchWriter.resetAll`
|
||||||
|
// discards it.
|
||||||
|
stmt.reset() catch {};
|
||||||
|
try stmt.bindInt(1, refused);
|
||||||
|
try testing.expectError(error.Constraint, stmt.exec());
|
||||||
|
}
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, 2), try database.queryInt("SELECT count(*) FROM query_log"));
|
||||||
|
}
|
||||||
|
|
||||||
|
test "querylog_meta is seeded with one row the schema will not let a second join" {
|
||||||
|
var database = try db.Db.open(":memory:", .{ .mode = .memory });
|
||||||
|
defer database.close();
|
||||||
|
try db.applyPragmas(&database, .{});
|
||||||
|
try database.exec(ddl);
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM querylog_meta"));
|
||||||
|
|
||||||
|
const created = try database.queryInt("SELECT created_at FROM querylog_meta");
|
||||||
|
const since = try database.queryInt("SELECT available_since FROM querylog_meta");
|
||||||
|
// Conservative by exactly one second: a row logged in the creating second
|
||||||
|
// must not let a query claim that second is completely covered.
|
||||||
|
try testing.expectEqual(created + 1, since);
|
||||||
|
try testing.expect(created > 1_700_000_000);
|
||||||
|
|
||||||
|
// `CHECK (id = 1)` is what makes "the singleton row" a schema fact rather
|
||||||
|
// than a convention the read path has to defend against.
|
||||||
|
try testing.expectError(error.Constraint, database.exec(
|
||||||
|
"INSERT INTO querylog_meta (id, created_at, available_since) VALUES (2, 1, 1);",
|
||||||
|
));
|
||||||
|
try testing.expectError(error.Constraint, database.exec(
|
||||||
|
"INSERT INTO querylog_meta (id, created_at, available_since) VALUES (1, 1, 1);",
|
||||||
|
));
|
||||||
|
try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM querylog_meta"));
|
||||||
|
}
|
||||||
|
|
||||||
test "the user_version statement stamps the fingerprint" {
|
test "the user_version statement stamps the fingerprint" {
|
||||||
var database = try db.Db.open(":memory:", .{ .mode = .memory });
|
var database = try db.Db.open(":memory:", .{ .mode = .memory });
|
||||||
defer database.close();
|
defer database.close();
|
||||||
@@ -394,6 +491,57 @@ test "a recreate returns the aside name by value and a fresh create returns none
|
|||||||
try tmp.dir.access(io, kept, .{});
|
try tmp.dir.access(io, kept, .{});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "a recreate resets coverage to the new file and keeps the old one aside" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var tmp = testing.tmpDir(.{ .iterate = true });
|
||||||
|
defer tmp.cleanup();
|
||||||
|
|
||||||
|
var path_buf: [path_buf_len]u8 = undefined;
|
||||||
|
const path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/querylog.db", .{tmp.sub_path});
|
||||||
|
|
||||||
|
var created = try open(io, std.Io.Dir.cwd(), path);
|
||||||
|
const first_coverage = try created.database.queryInt("SELECT available_since FROM querylog_meta");
|
||||||
|
// A row in the file the operator is about to lose.
|
||||||
|
try created.database.exec("INSERT INTO domains (domain) VALUES ('old.example');");
|
||||||
|
created.database.close();
|
||||||
|
|
||||||
|
// A healthy file this build's DDL no longer matches — the case milestone
|
||||||
|
// 28's own schema edit produces on every upgrade.
|
||||||
|
{
|
||||||
|
var stamped = try db.Db.open(path, .{ .mode = .read_write_existing });
|
||||||
|
defer stamped.close();
|
||||||
|
var sql_buf: [64]u8 = undefined;
|
||||||
|
try stamped.exec(try std.fmt.bufPrintZ(&sql_buf, "PRAGMA user_version = {d};", .{fingerprint +% 1}));
|
||||||
|
}
|
||||||
|
|
||||||
|
var recreated = try open(io, std.Io.Dir.cwd(), path);
|
||||||
|
defer recreated.database.close();
|
||||||
|
|
||||||
|
try testing.expectEqual(RecreateReason.fingerprint_mismatch, recreated.recreated.?);
|
||||||
|
// The name says the file was healthy and this build moved, not that it rotted.
|
||||||
|
try testing.expect(std.mem.indexOf(u8, recreated.aside(), ".schema-changed-") != null);
|
||||||
|
try tmp.dir.access(io, std.fs.path.basename(recreated.aside()), .{});
|
||||||
|
|
||||||
|
// Exactly one meta row, and coverage starts at the recreate rather than
|
||||||
|
// carrying the replaced file's promise forward.
|
||||||
|
try testing.expectEqual(
|
||||||
|
@as(i64, 1),
|
||||||
|
try recreated.database.queryInt("SELECT count(*) FROM querylog_meta"),
|
||||||
|
);
|
||||||
|
const new_coverage = try recreated.database.queryInt("SELECT available_since FROM querylog_meta");
|
||||||
|
try testing.expect(new_coverage >= first_coverage);
|
||||||
|
|
||||||
|
// Nothing of the old file came across: the history is genuinely gone, which
|
||||||
|
// is what the coverage start has to tell the operator.
|
||||||
|
try testing.expectEqual(
|
||||||
|
@as(i64, 0),
|
||||||
|
try recreated.database.queryInt("SELECT count(*) FROM domains"),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
test "a clean reopen reports no recreate and no aside" {
|
test "a clean reopen reports no recreate and no aside" {
|
||||||
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
defer threaded.deinit();
|
defer threaded.deinit();
|
||||||
|
|||||||
@@ -19,20 +19,42 @@ const std = @import("std");
|
|||||||
const Allocator = std.mem.Allocator;
|
const Allocator = std.mem.Allocator;
|
||||||
|
|
||||||
const db = @import("../db.zig");
|
const db = @import("../db.zig");
|
||||||
|
const provenance = @import("../provenance.zig");
|
||||||
|
|
||||||
/// One `query_log` row. The logger applies the privacy transforms of PLAN
|
/// One `query_log` row. The logger applies the privacy transforms of PLAN
|
||||||
/// §11.4 before it builds this, so `domain` and `client_ip` are already
|
/// §11.4 before it builds this, so every domain-bearing field is already
|
||||||
/// whatever the operator agreed to store.
|
/// whatever the operator agreed to store.
|
||||||
|
///
|
||||||
|
/// A `null` text field is a fact the query did not have — no upstream was
|
||||||
|
/// attempted, no rule matched, no CNAME was uncloaked — and reaches the column
|
||||||
|
/// as NULL. The three closed enums have no such state: every logged query has a
|
||||||
|
/// policy verdict, a reason for it and a route, even when the verdict is
|
||||||
|
/// "not evaluated".
|
||||||
pub const Row = struct {
|
pub const Row = struct {
|
||||||
timestamp: i64,
|
timestamp: i64,
|
||||||
domain: []const u8,
|
domain: []const u8,
|
||||||
client_ip: []const u8,
|
client_ip: []const u8,
|
||||||
qtype: ?u16,
|
qtype: ?u16,
|
||||||
|
qclass: u16,
|
||||||
|
/// Twelve bits: the EDNS extended RCODE the client saw. The column's
|
||||||
|
/// `CHECK` bounds it to the same range, so a value this type cannot hold
|
||||||
|
/// is one the schema would have refused anyway.
|
||||||
|
rcode: u12,
|
||||||
blocked: bool,
|
blocked: bool,
|
||||||
block_reason: ?[]const u8,
|
|
||||||
response_time_us: ?i64,
|
response_time_us: ?i64,
|
||||||
cache_hit: ?bool,
|
cache_hit: ?bool,
|
||||||
upstream: ?[]const u8,
|
upstream: ?[]const u8,
|
||||||
|
group_id: ?i64,
|
||||||
|
group_name: ?[]const u8,
|
||||||
|
policy_action: provenance.PolicyAction,
|
||||||
|
policy_reason: provenance.PolicyReason,
|
||||||
|
matched: ?[]const u8,
|
||||||
|
source_id: ?i64,
|
||||||
|
source_name: ?[]const u8,
|
||||||
|
cname_target: ?[]const u8,
|
||||||
|
safe_search_target: ?[]const u8,
|
||||||
|
route_kind: provenance.RouteKind,
|
||||||
|
forward_zone: ?[]const u8,
|
||||||
};
|
};
|
||||||
|
|
||||||
const insert_domain_sql = "INSERT OR IGNORE INTO domains (domain) VALUES (?1)";
|
const insert_domain_sql = "INSERT OR IGNORE INTO domains (domain) VALUES (?1)";
|
||||||
@@ -41,9 +63,13 @@ const select_domain_sql = "SELECT id FROM domains WHERE domain = ?1";
|
|||||||
|
|
||||||
const insert_row_sql =
|
const insert_row_sql =
|
||||||
\\INSERT INTO query_log
|
\\INSERT INTO query_log
|
||||||
\\ (timestamp, domain_id, client_ip, qtype, blocked, block_reason,
|
\\ (timestamp, domain_id, client_ip, qtype, blocked,
|
||||||
\\ response_time_us, cache_hit, upstream)
|
\\ response_time_us, cache_hit, upstream, qclass, rcode,
|
||||||
\\VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
|
\\ group_id, group_name, policy_action, policy_reason, matched,
|
||||||
|
\\ source_id, source_name, cname_target, safe_search_target,
|
||||||
|
\\ route_kind, forward_zone)
|
||||||
|
\\VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10,
|
||||||
|
\\ ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19, ?20, ?21)
|
||||||
;
|
;
|
||||||
|
|
||||||
/// Owns the prepared statements of the flush loop. Init once, reuse per batch.
|
/// Owns the prepared statements of the flush loop. Init once, reuse per batch.
|
||||||
@@ -123,10 +149,22 @@ pub const BatchWriter = struct {
|
|||||||
try stmt.bindText(3, row.client_ip);
|
try stmt.bindText(3, row.client_ip);
|
||||||
try bindIntOrNull(stmt, 4, if (row.qtype) |v| @as(i64, v) else null);
|
try bindIntOrNull(stmt, 4, if (row.qtype) |v| @as(i64, v) else null);
|
||||||
try stmt.bindBool(5, row.blocked);
|
try stmt.bindBool(5, row.blocked);
|
||||||
try stmt.bindTextOrNull(6, row.block_reason);
|
try bindIntOrNull(stmt, 6, row.response_time_us);
|
||||||
try bindIntOrNull(stmt, 7, row.response_time_us);
|
try bindIntOrNull(stmt, 7, if (row.cache_hit) |v| @as(i64, @intFromBool(v)) else null);
|
||||||
try bindIntOrNull(stmt, 8, if (row.cache_hit) |v| @as(i64, @intFromBool(v)) else null);
|
try stmt.bindTextOrNull(8, row.upstream);
|
||||||
try stmt.bindTextOrNull(9, row.upstream);
|
try stmt.bindInt(9, row.qclass);
|
||||||
|
try stmt.bindInt(10, row.rcode);
|
||||||
|
try bindIntOrNull(stmt, 11, row.group_id);
|
||||||
|
try stmt.bindTextOrNull(12, row.group_name);
|
||||||
|
try stmt.bindText(13, @tagName(row.policy_action));
|
||||||
|
try stmt.bindText(14, @tagName(row.policy_reason));
|
||||||
|
try stmt.bindTextOrNull(15, row.matched);
|
||||||
|
try bindIntOrNull(stmt, 16, row.source_id);
|
||||||
|
try stmt.bindTextOrNull(17, row.source_name);
|
||||||
|
try stmt.bindTextOrNull(18, row.cname_target);
|
||||||
|
try stmt.bindTextOrNull(19, row.safe_search_target);
|
||||||
|
try stmt.bindText(20, @tagName(row.route_kind));
|
||||||
|
try stmt.bindTextOrNull(21, row.forward_zone);
|
||||||
try stmt.exec();
|
try stmt.exec();
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -144,17 +182,55 @@ fn bindIntOrNull(stmt: *db.Stmt, idx: c_int, value: ?i64) db.Error!void {
|
|||||||
return stmt.bindNull(idx);
|
return stmt.bindNull(idx);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Deletes every `query_log` row strictly older than `cutoff_ts` and returns
|
/// What one prune did, and where coverage now begins.
|
||||||
/// how many went.
|
pub const PruneResult = struct {
|
||||||
|
deleted: i64,
|
||||||
|
/// The watermark after the prune, which is what a later `availableSince`
|
||||||
|
/// will return. Handed back so the caller need not re-read it.
|
||||||
|
available_since: i64,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Deletes every `query_log` row strictly older than `cutoff_ts` and advances
|
||||||
|
/// the coverage watermark to the same cutoff, in one transaction.
|
||||||
///
|
///
|
||||||
/// Orphaned `domains` rows stay: it is a dimension table, re-interning a name
|
/// **The two are one operation, not two.** The watermark is the promise that
|
||||||
/// costs one indexed insert, and §11.3 asks for no collection.
|
/// every query since it is still in the file; a delete that commits without the
|
||||||
pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!i64 {
|
/// advance breaks that promise, and an advance that commits without the delete
|
||||||
var stmt = try database.prepare("DELETE FROM query_log WHERE timestamp < ?1");
|
/// hides rows the file still holds. Either failure rolls both back, and the
|
||||||
defer stmt.deinit();
|
/// caller retries the whole thing on its next pass.
|
||||||
try stmt.bindInt(1, cutoff_ts);
|
///
|
||||||
try stmt.exec();
|
/// The watermark never moves backward: `max` is what makes a prune with a
|
||||||
return database.changes();
|
/// cutoff older than the file's own creation a no-op on it rather than a
|
||||||
|
/// regression. Orphaned `domains` rows stay — it is a dimension table,
|
||||||
|
/// re-interning a name costs one indexed insert, and §11.3 asks for no
|
||||||
|
/// collection.
|
||||||
|
pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!PruneResult {
|
||||||
|
var tx = try db.Tx.begin(database);
|
||||||
|
errdefer tx.rollback();
|
||||||
|
|
||||||
|
var deleting = try database.prepare("DELETE FROM query_log WHERE timestamp < ?1");
|
||||||
|
defer deleting.deinit();
|
||||||
|
try deleting.bindInt(1, cutoff_ts);
|
||||||
|
try deleting.exec();
|
||||||
|
const deleted = database.changes();
|
||||||
|
|
||||||
|
var advancing = try database.prepare(
|
||||||
|
"UPDATE querylog_meta SET available_since = max(available_since, ?1) WHERE id = 1",
|
||||||
|
);
|
||||||
|
defer advancing.deinit();
|
||||||
|
try advancing.bindInt(1, cutoff_ts);
|
||||||
|
try advancing.exec();
|
||||||
|
|
||||||
|
const watermark = try database.queryInt("SELECT available_since FROM querylog_meta WHERE id = 1");
|
||||||
|
try tx.commit();
|
||||||
|
return .{ .deleted = deleted, .available_since = watermark };
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The oldest timestamp this file can still answer for. A query window that
|
||||||
|
/// starts before it is incomplete, and the API says so rather than charting the
|
||||||
|
/// gap as zero.
|
||||||
|
pub fn availableSince(database: *db.Db) db.Error!i64 {
|
||||||
|
return database.queryInt("SELECT available_since FROM querylog_meta WHERE id = 1");
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `PRAGMA wal_checkpoint(TRUNCATE)`: moves the WAL into the database and
|
/// `PRAGMA wal_checkpoint(TRUNCATE)`: moves the WAL into the database and
|
||||||
@@ -189,21 +265,60 @@ pub fn countDomains(database: *db.Db) db.Error!i64 {
|
|||||||
|
|
||||||
/// One row of `GET /api/queries`, joined back through the `domains` dimension.
|
/// One row of `GET /api/queries`, joined back through the `domains` dimension.
|
||||||
///
|
///
|
||||||
/// `block_reason` and `upstream` are nullable columns, and a NULL reads as `""`
|
/// A summary projection, deliberately narrower than `QueryDetail`: the list is
|
||||||
/// — the same convention `Stmt.columnText` already uses. Neither column is ever
|
/// a table the operator scans, and the full provenance of a row is one request
|
||||||
/// written as an empty string (a reason is a word, an upstream is a URL), so the
|
/// away at `GET /api/queries/{id}`.
|
||||||
/// mapping loses nothing and the API layer can treat `""` as "absent".
|
///
|
||||||
|
/// The nullable text columns read a NULL as `""` — the same convention
|
||||||
|
/// `Stmt.columnText` already uses. None of them is ever written as an empty
|
||||||
|
/// string, so the mapping loses nothing and the API layer can treat `""` as
|
||||||
|
/// "absent".
|
||||||
pub const QueryRow = struct {
|
pub const QueryRow = struct {
|
||||||
id: i64,
|
id: i64,
|
||||||
ts: i64,
|
ts: i64,
|
||||||
domain: []const u8,
|
domain: []const u8,
|
||||||
client_ip: []const u8,
|
client_ip: []const u8,
|
||||||
qtype: ?u16,
|
qtype: ?u16,
|
||||||
|
qclass: u16,
|
||||||
|
rcode: u12,
|
||||||
blocked: bool,
|
blocked: bool,
|
||||||
block_reason: []const u8,
|
|
||||||
response_time_us: ?i64,
|
response_time_us: ?i64,
|
||||||
cache_hit: ?bool,
|
cache_hit: ?bool,
|
||||||
upstream: []const u8,
|
upstream: []const u8,
|
||||||
|
policy_action: provenance.PolicyAction,
|
||||||
|
policy_reason: provenance.PolicyReason,
|
||||||
|
route_kind: provenance.RouteKind,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Everything one `query_log` row records about one query, for
|
||||||
|
/// `GET /api/queries/{id}`.
|
||||||
|
///
|
||||||
|
/// Same NULL-reads-as-`""` convention as `QueryRow`, and the same closed enums:
|
||||||
|
/// a stored value the schema does not define is `error.Mismatch`, never passed
|
||||||
|
/// through as text.
|
||||||
|
pub const QueryDetail = struct {
|
||||||
|
id: i64,
|
||||||
|
ts: i64,
|
||||||
|
domain: []const u8,
|
||||||
|
client_ip: []const u8,
|
||||||
|
qtype: ?u16,
|
||||||
|
qclass: u16,
|
||||||
|
rcode: u12,
|
||||||
|
blocked: bool,
|
||||||
|
response_time_us: ?i64,
|
||||||
|
cache_hit: ?bool,
|
||||||
|
upstream: []const u8,
|
||||||
|
group_id: ?i64,
|
||||||
|
group_name: []const u8,
|
||||||
|
policy_action: provenance.PolicyAction,
|
||||||
|
policy_reason: provenance.PolicyReason,
|
||||||
|
matched: []const u8,
|
||||||
|
source_id: ?i64,
|
||||||
|
source_name: []const u8,
|
||||||
|
cname_target: []const u8,
|
||||||
|
safe_search_target: []const u8,
|
||||||
|
route_kind: provenance.RouteKind,
|
||||||
|
forward_zone: []const u8,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Every field is an independent narrowing; `null` means "do not filter on it".
|
/// Every field is an independent narrowing; `null` means "do not filter on it".
|
||||||
@@ -230,7 +345,8 @@ pub const max_limit: u32 = 1000;
|
|||||||
|
|
||||||
const select_head =
|
const select_head =
|
||||||
\\SELECT q.id, q.timestamp, d.domain, q.client_ip, q.qtype, q.blocked,
|
\\SELECT q.id, q.timestamp, d.domain, q.client_ip, q.qtype, q.blocked,
|
||||||
\\ q.block_reason, q.response_time_us, q.cache_hit, q.upstream
|
\\ q.response_time_us, q.cache_hit, q.upstream, q.qclass, q.rcode,
|
||||||
|
\\ q.policy_action, q.policy_reason, q.route_kind
|
||||||
\\ FROM query_log q JOIN domains d ON d.id = q.domain_id
|
\\ FROM query_log q JOIN domains d ON d.id = q.domain_id
|
||||||
;
|
;
|
||||||
|
|
||||||
@@ -337,15 +453,81 @@ pub fn selectQueries(database: *db.Db, arena: Allocator, filter: QueryFilter) db
|
|||||||
.qtype = if (stmt.isNull(4)) null else std.math.cast(u16, stmt.columnInt(4)) orelse
|
.qtype = if (stmt.isNull(4)) null else std.math.cast(u16, stmt.columnInt(4)) orelse
|
||||||
return error.Mismatch,
|
return error.Mismatch,
|
||||||
.blocked = stmt.columnBool(5),
|
.blocked = stmt.columnBool(5),
|
||||||
.block_reason = try stmt.columnTextAlloc(arena, 6),
|
.response_time_us = if (stmt.isNull(6)) null else stmt.columnInt(6),
|
||||||
.response_time_us = if (stmt.isNull(7)) null else stmt.columnInt(7),
|
.cache_hit = if (stmt.isNull(7)) null else stmt.columnBool(7),
|
||||||
.cache_hit = if (stmt.isNull(8)) null else stmt.columnBool(8),
|
.upstream = try stmt.columnTextAlloc(arena, 8),
|
||||||
.upstream = try stmt.columnTextAlloc(arena, 9),
|
.qclass = try columnU16(&stmt, 9),
|
||||||
|
.rcode = try columnU12(&stmt, 10),
|
||||||
|
.policy_action = try provenance.parse(provenance.PolicyAction, stmt.columnText(11)),
|
||||||
|
.policy_reason = try provenance.parse(provenance.PolicyReason, stmt.columnText(12)),
|
||||||
|
.route_kind = try provenance.parse(provenance.RouteKind, stmt.columnText(13)),
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
return out;
|
return out;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// A `NOT NULL` integer column that the schema bounds to 16 bits. A value
|
||||||
|
/// outside that range means the row came from something other than this schema.
|
||||||
|
fn columnU16(stmt: *db.Stmt, col: c_int) db.Error!u16 {
|
||||||
|
return std.math.cast(u16, stmt.columnInt(col)) orelse error.Mismatch;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The `rcode` column, which the schema's `CHECK` bounds to twelve bits. This
|
||||||
|
/// build cannot write a wider value — the field is a `u12` all the way from the
|
||||||
|
/// handler — so a row that carries one was written by something else, and is
|
||||||
|
/// `error.Mismatch` rather than a value truncated into shape.
|
||||||
|
fn columnU12(stmt: *db.Stmt, col: c_int) db.Error!u12 {
|
||||||
|
return std.math.cast(u12, stmt.columnInt(col)) orelse error.Mismatch;
|
||||||
|
}
|
||||||
|
|
||||||
|
const select_detail_sql =
|
||||||
|
\\SELECT q.id, q.timestamp, d.domain, q.client_ip, q.qtype, q.blocked,
|
||||||
|
\\ q.response_time_us, q.cache_hit, q.upstream, q.qclass, q.rcode,
|
||||||
|
\\ q.group_id, q.group_name, q.policy_action, q.policy_reason,
|
||||||
|
\\ q.matched, q.source_id, q.source_name, q.cname_target,
|
||||||
|
\\ q.safe_search_target, q.route_kind, q.forward_zone
|
||||||
|
\\ FROM query_log q JOIN domains d ON d.id = q.domain_id
|
||||||
|
\\ WHERE q.id = ?1
|
||||||
|
;
|
||||||
|
|
||||||
|
/// One row's full provenance, or `null` when no row has that id — which is what
|
||||||
|
/// an id the operator kept from before a retention pass looks like, and is a
|
||||||
|
/// 404 rather than an error.
|
||||||
|
///
|
||||||
|
/// Every string is allocated from `arena`, on the same terms as
|
||||||
|
/// `selectQueries`.
|
||||||
|
pub fn detailById(database: *db.Db, arena: Allocator, id: i64) db.Error!?QueryDetail {
|
||||||
|
var stmt = try database.prepare(select_detail_sql);
|
||||||
|
defer stmt.deinit();
|
||||||
|
try stmt.bindInt(1, id);
|
||||||
|
if (!try stmt.step()) return null;
|
||||||
|
|
||||||
|
return .{
|
||||||
|
.id = stmt.columnInt(0),
|
||||||
|
.ts = stmt.columnInt(1),
|
||||||
|
.domain = try stmt.columnTextAlloc(arena, 2),
|
||||||
|
.client_ip = try stmt.columnTextAlloc(arena, 3),
|
||||||
|
.qtype = if (stmt.isNull(4)) null else try columnU16(&stmt, 4),
|
||||||
|
.blocked = stmt.columnBool(5),
|
||||||
|
.response_time_us = if (stmt.isNull(6)) null else stmt.columnInt(6),
|
||||||
|
.cache_hit = if (stmt.isNull(7)) null else stmt.columnBool(7),
|
||||||
|
.upstream = try stmt.columnTextAlloc(arena, 8),
|
||||||
|
.qclass = try columnU16(&stmt, 9),
|
||||||
|
.rcode = try columnU12(&stmt, 10),
|
||||||
|
.group_id = if (stmt.isNull(11)) null else stmt.columnInt(11),
|
||||||
|
.group_name = try stmt.columnTextAlloc(arena, 12),
|
||||||
|
.policy_action = try provenance.parse(provenance.PolicyAction, stmt.columnText(13)),
|
||||||
|
.policy_reason = try provenance.parse(provenance.PolicyReason, stmt.columnText(14)),
|
||||||
|
.matched = try stmt.columnTextAlloc(arena, 15),
|
||||||
|
.source_id = if (stmt.isNull(16)) null else stmt.columnInt(16),
|
||||||
|
.source_name = try stmt.columnTextAlloc(arena, 17),
|
||||||
|
.cname_target = try stmt.columnTextAlloc(arena, 18),
|
||||||
|
.safe_search_target = try stmt.columnTextAlloc(arena, 19),
|
||||||
|
.route_kind = try provenance.parse(provenance.RouteKind, stmt.columnText(20)),
|
||||||
|
.forward_zone = try stmt.columnTextAlloc(arena, 21),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/// Wraps `needle` in `%` and neutralises the two `LIKE` metacharacters, so a
|
/// Wraps `needle` in `%` and neutralises the two `LIKE` metacharacters, so a
|
||||||
/// user searching for `a_b` gets domains containing `a_b` and not domains
|
/// user searching for `a_b` gets domains containing `a_b` and not domains
|
||||||
/// containing `axb`. The escape character escapes itself.
|
/// containing `axb`. The escape character escapes itself.
|
||||||
@@ -493,11 +675,23 @@ fn plainRow(timestamp: i64, domain: []const u8) Row {
|
|||||||
.domain = domain,
|
.domain = domain,
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = false,
|
.blocked = false,
|
||||||
.block_reason = null,
|
|
||||||
.response_time_us = 1200,
|
.response_time_us = 1200,
|
||||||
.cache_hit = false,
|
.cache_hit = false,
|
||||||
.upstream = "9.9.9.9",
|
.upstream = "9.9.9.9",
|
||||||
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = .allow,
|
||||||
|
.policy_reason = .no_match,
|
||||||
|
.matched = null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -509,6 +703,43 @@ fn domainIdOf(database: *db.Db, domain: []const u8) !i64 {
|
|||||||
return stmt.columnInt(0);
|
return stmt.columnInt(0);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "a foreign row with an rcode wider than twelve bits is refused, not truncated" {
|
||||||
|
var database = try db.Db.open(":memory:", .{ .mode = .memory });
|
||||||
|
defer database.close();
|
||||||
|
try db.applyPragmas(&database, .{});
|
||||||
|
|
||||||
|
// The shipped schema's `CHECK` makes this row impossible in a file this
|
||||||
|
// build created, so the table is built without it. The read path's job is
|
||||||
|
// to refuse a `querylog.db` that came from somewhere else rather than to
|
||||||
|
// narrow a value it cannot represent.
|
||||||
|
try database.exec(
|
||||||
|
\\CREATE TABLE domains (id INTEGER PRIMARY KEY, domain TEXT NOT NULL UNIQUE);
|
||||||
|
\\CREATE TABLE query_log (
|
||||||
|
\\ id INTEGER PRIMARY KEY, timestamp INTEGER NOT NULL,
|
||||||
|
\\ domain_id INTEGER NOT NULL, client_ip TEXT NOT NULL,
|
||||||
|
\\ qtype INTEGER, blocked INTEGER NOT NULL, response_time_us INTEGER,
|
||||||
|
\\ cache_hit INTEGER, upstream TEXT, qclass INTEGER NOT NULL,
|
||||||
|
\\ rcode INTEGER NOT NULL, group_id INTEGER, group_name TEXT,
|
||||||
|
\\ policy_action TEXT NOT NULL, policy_reason TEXT NOT NULL,
|
||||||
|
\\ matched TEXT, source_id INTEGER, source_name TEXT,
|
||||||
|
\\ cname_target TEXT, safe_search_target TEXT,
|
||||||
|
\\ route_kind TEXT NOT NULL, forward_zone TEXT
|
||||||
|
\\);
|
||||||
|
\\INSERT INTO domains (id, domain) VALUES (1, 'a.example');
|
||||||
|
\\INSERT INTO query_log
|
||||||
|
\\ (id, timestamp, domain_id, client_ip, blocked, qclass, rcode,
|
||||||
|
\\ policy_action, policy_reason, route_kind)
|
||||||
|
\\VALUES (1, 10, 1, '192.0.2.10', 0, 1, 4096, 'allow', 'no_match', 'upstream');
|
||||||
|
);
|
||||||
|
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
const arena = arena_state.allocator();
|
||||||
|
|
||||||
|
try testing.expectError(error.Mismatch, selectQueries(&database, arena, .{}));
|
||||||
|
try testing.expectError(error.Mismatch, detailById(&database, arena, 1));
|
||||||
|
}
|
||||||
|
|
||||||
test "writeBatch inserts every row and interns each domain once" {
|
test "writeBatch inserts every row and interns each domain once" {
|
||||||
var database = try openLog();
|
var database = try openLog();
|
||||||
defer database.close();
|
defer database.close();
|
||||||
@@ -553,7 +784,7 @@ test "a second batch reuses the interned domain id" {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
test "nullable columns round-trip a value and a null" {
|
test "every column round-trips a value and a null" {
|
||||||
var database = try openLog();
|
var database = try openLog();
|
||||||
defer database.close();
|
defer database.close();
|
||||||
var writer = try BatchWriter.init(&database);
|
var writer = try BatchWriter.init(&database);
|
||||||
@@ -565,54 +796,125 @@ test "nullable columns round-trip a value and a null" {
|
|||||||
.domain = "blocked.example",
|
.domain = "blocked.example",
|
||||||
.client_ip = "2001:db8::1",
|
.client_ip = "2001:db8::1",
|
||||||
.qtype = 28,
|
.qtype = 28,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 3,
|
||||||
.blocked = true,
|
.blocked = true,
|
||||||
.block_reason = "blocklist",
|
|
||||||
.response_time_us = 42,
|
.response_time_us = 42,
|
||||||
.cache_hit = true,
|
.cache_hit = true,
|
||||||
.upstream = "dns.example",
|
.upstream = "https://dns.example/dns-query",
|
||||||
|
.group_id = 7,
|
||||||
|
.group_name = "kids",
|
||||||
|
.policy_action = .block,
|
||||||
|
.policy_reason = .blocklist_wildcard,
|
||||||
|
.matched = "*.ads.example",
|
||||||
|
.source_id = 3,
|
||||||
|
.source_name = "steven black",
|
||||||
|
.cname_target = "tracker.cdn.example",
|
||||||
|
.safe_search_target = "forcesafesearch.google.com",
|
||||||
|
.route_kind = .blocked,
|
||||||
|
.forward_zone = "home.arpa",
|
||||||
},
|
},
|
||||||
|
// Every nullable column absent at once, which is the shape of a query
|
||||||
|
// the pipeline answered before any of them applied.
|
||||||
.{
|
.{
|
||||||
.timestamp = 11,
|
.timestamp = 11,
|
||||||
.domain = "quiet.example",
|
.domain = "quiet.example",
|
||||||
.client_ip = "hidden",
|
.client_ip = "hidden",
|
||||||
.qtype = null,
|
.qtype = null,
|
||||||
|
.qclass = 3,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = false,
|
.blocked = false,
|
||||||
.block_reason = null,
|
|
||||||
.response_time_us = null,
|
.response_time_us = null,
|
||||||
.cache_hit = null,
|
.cache_hit = null,
|
||||||
.upstream = null,
|
.upstream = null,
|
||||||
|
.group_id = null,
|
||||||
|
.group_name = null,
|
||||||
|
.policy_action = .not_evaluated,
|
||||||
|
.policy_reason = .non_in_class,
|
||||||
|
.matched = null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
var stmt = try database.prepare(
|
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
\\SELECT d.domain, q.client_ip, q.qtype, q.blocked, q.block_reason,
|
defer arena_state.deinit();
|
||||||
\\ q.response_time_us, q.cache_hit, q.upstream
|
const arena = arena_state.allocator();
|
||||||
\\ FROM query_log q JOIN domains d ON d.id = q.domain_id
|
|
||||||
\\ ORDER BY q.timestamp
|
|
||||||
);
|
|
||||||
defer stmt.deinit();
|
|
||||||
|
|
||||||
try testing.expect(try stmt.step());
|
const full = (try detailById(&database, arena, 1)).?;
|
||||||
try testing.expectEqualStrings("blocked.example", stmt.columnText(0));
|
try testing.expectEqualStrings("blocked.example", full.domain);
|
||||||
try testing.expectEqualStrings("2001:db8::1", stmt.columnText(1));
|
try testing.expectEqualStrings("2001:db8::1", full.client_ip);
|
||||||
try testing.expectEqual(@as(i64, 28), stmt.columnInt(2));
|
try testing.expectEqual(@as(?u16, 28), full.qtype);
|
||||||
try testing.expect(stmt.columnBool(3));
|
try testing.expectEqual(@as(u16, 1), full.qclass);
|
||||||
try testing.expectEqualStrings("blocklist", stmt.columnText(4));
|
try testing.expectEqual(@as(u12, 3), full.rcode);
|
||||||
try testing.expectEqual(@as(i64, 42), stmt.columnInt(5));
|
try testing.expect(full.blocked);
|
||||||
try testing.expect(stmt.columnBool(6));
|
try testing.expectEqual(@as(?i64, 42), full.response_time_us);
|
||||||
try testing.expectEqualStrings("dns.example", stmt.columnText(7));
|
try testing.expectEqual(@as(?bool, true), full.cache_hit);
|
||||||
|
try testing.expectEqualStrings("https://dns.example/dns-query", full.upstream);
|
||||||
|
try testing.expectEqual(@as(?i64, 7), full.group_id);
|
||||||
|
try testing.expectEqualStrings("kids", full.group_name);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.block, full.policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, full.policy_reason);
|
||||||
|
try testing.expectEqualStrings("*.ads.example", full.matched);
|
||||||
|
try testing.expectEqual(@as(?i64, 3), full.source_id);
|
||||||
|
try testing.expectEqualStrings("steven black", full.source_name);
|
||||||
|
try testing.expectEqualStrings("tracker.cdn.example", full.cname_target);
|
||||||
|
try testing.expectEqualStrings("forcesafesearch.google.com", full.safe_search_target);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.blocked, full.route_kind);
|
||||||
|
try testing.expectEqualStrings("home.arpa", full.forward_zone);
|
||||||
|
|
||||||
try testing.expect(try stmt.step());
|
// A NULL text column reads as the empty string, by the documented
|
||||||
try testing.expectEqualStrings("quiet.example", stmt.columnText(0));
|
// convention; a NULL integer stays null, because 0 is a real id.
|
||||||
try testing.expectEqualStrings("hidden", stmt.columnText(1));
|
const bare = (try detailById(&database, arena, 2)).?;
|
||||||
try testing.expect(stmt.isNull(2));
|
try testing.expectEqual(@as(?u16, null), bare.qtype);
|
||||||
try testing.expect(!stmt.columnBool(3));
|
try testing.expectEqual(@as(u16, 3), bare.qclass);
|
||||||
try testing.expect(stmt.isNull(4));
|
try testing.expectEqual(@as(?i64, null), bare.response_time_us);
|
||||||
try testing.expect(stmt.isNull(5));
|
try testing.expectEqual(@as(?bool, null), bare.cache_hit);
|
||||||
try testing.expect(stmt.isNull(6));
|
try testing.expectEqualStrings("", bare.upstream);
|
||||||
try testing.expect(stmt.isNull(7));
|
try testing.expectEqual(@as(?i64, null), bare.group_id);
|
||||||
|
try testing.expectEqualStrings("", bare.group_name);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.not_evaluated, bare.policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.non_in_class, bare.policy_reason);
|
||||||
|
try testing.expectEqualStrings("", bare.matched);
|
||||||
|
try testing.expectEqual(@as(?i64, null), bare.source_id);
|
||||||
|
try testing.expectEqualStrings("", bare.source_name);
|
||||||
|
try testing.expectEqualStrings("", bare.cname_target);
|
||||||
|
try testing.expectEqualStrings("", bare.safe_search_target);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.upstream, bare.route_kind);
|
||||||
|
try testing.expectEqualStrings("", bare.forward_zone);
|
||||||
|
}
|
||||||
|
|
||||||
try testing.expect(!try stmt.step());
|
test "detailById returns null for an id no row has" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
|
||||||
|
try seed(&database, &.{plainRow(10, "a.example")});
|
||||||
|
|
||||||
|
// An id from before a retention pass looks exactly like this, and is a 404
|
||||||
|
// rather than an error.
|
||||||
|
try testing.expectEqual(@as(?QueryDetail, null), try detailById(&database, arena_state.allocator(), 2));
|
||||||
|
try testing.expectEqual(@as(?QueryDetail, null), try detailById(&database, arena_state.allocator(), 0));
|
||||||
|
try testing.expect((try detailById(&database, arena_state.allocator(), 1)) != null);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a stored enum value the schema does not define is a data error, not a passthrough" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
const arena = arena_state.allocator();
|
||||||
|
|
||||||
|
try seed(&database, &.{plainRow(10, "a.example")});
|
||||||
|
try database.exec("UPDATE query_log SET policy_reason = 'whatever' WHERE id = 1;");
|
||||||
|
|
||||||
|
try testing.expectError(error.Mismatch, detailById(&database, arena, 1));
|
||||||
|
try testing.expectError(error.Mismatch, selectQueries(&database, arena, .{}));
|
||||||
}
|
}
|
||||||
|
|
||||||
test "an empty batch writes nothing and opens no transaction" {
|
test "an empty batch writes nothing and opens no transaction" {
|
||||||
@@ -644,7 +946,7 @@ test "pruneOlderThan deletes strictly older rows and returns the count" {
|
|||||||
plainRow(300, "fresh.example"),
|
plainRow(300, "fresh.example"),
|
||||||
});
|
});
|
||||||
|
|
||||||
try testing.expectEqual(@as(i64, 2), try pruneOlderThan(&database, 200));
|
try testing.expectEqual(@as(i64, 2), (try pruneOlderThan(&database, 200)).deleted);
|
||||||
try testing.expectEqual(@as(i64, 2), try countRows(&database));
|
try testing.expectEqual(@as(i64, 2), try countRows(&database));
|
||||||
// The row exactly at the cutoff stays.
|
// The row exactly at the cutoff stays.
|
||||||
try testing.expectEqual(
|
try testing.expectEqual(
|
||||||
@@ -652,7 +954,123 @@ test "pruneOlderThan deletes strictly older rows and returns the count" {
|
|||||||
try database.queryInt("SELECT count(*) FROM query_log WHERE timestamp = 200"),
|
try database.queryInt("SELECT count(*) FROM query_log WHERE timestamp = 200"),
|
||||||
);
|
);
|
||||||
// A second pass over the same cutoff finds nothing left to do.
|
// A second pass over the same cutoff finds nothing left to do.
|
||||||
try testing.expectEqual(@as(i64, 0), try pruneOlderThan(&database, 200));
|
try testing.expectEqual(@as(i64, 0), (try pruneOlderThan(&database, 200)).deleted);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a prune advances the coverage watermark to its own cutoff" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
var writer = try BatchWriter.init(&database);
|
||||||
|
defer writer.deinit();
|
||||||
|
|
||||||
|
// The seeded watermark is `created_at + 1`, which is now-ish; the cutoffs
|
||||||
|
// below are all in the past, so they start out behind it.
|
||||||
|
const start = try availableSince(&database);
|
||||||
|
try writer.writeBatch(&.{ plainRow(start + 100, "a.example"), plainRow(start + 300, "b.example") });
|
||||||
|
|
||||||
|
const first = try pruneOlderThan(&database, start + 200);
|
||||||
|
try testing.expectEqual(@as(i64, 1), first.deleted);
|
||||||
|
try testing.expectEqual(start + 200, first.available_since);
|
||||||
|
try testing.expectEqual(start + 200, try availableSince(&database));
|
||||||
|
|
||||||
|
// A prune that deletes nothing still advances: the window it swept is
|
||||||
|
// covered whether or not it held rows.
|
||||||
|
const second = try pruneOlderThan(&database, start + 250);
|
||||||
|
try testing.expectEqual(@as(i64, 0), second.deleted);
|
||||||
|
try testing.expectEqual(start + 250, try availableSince(&database));
|
||||||
|
}
|
||||||
|
|
||||||
|
test "the watermark never moves backward" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
|
||||||
|
const start = try availableSince(&database);
|
||||||
|
const advanced = try pruneOlderThan(&database, start + 1000);
|
||||||
|
try testing.expectEqual(start + 1000, advanced.available_since);
|
||||||
|
|
||||||
|
// A shortened `retention_days`, a clock that stepped back, a pass with a
|
||||||
|
// stale cutoff: none of them may widen the promise the file makes.
|
||||||
|
for ([_]i64{ start + 999, start, start - 100_000, 0 }) |older| {
|
||||||
|
const result = try pruneOlderThan(&database, older);
|
||||||
|
try testing.expectEqual(start + 1000, result.available_since);
|
||||||
|
try testing.expectEqual(start + 1000, try availableSince(&database));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a failed delete leaves both the rows and the watermark untouched" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
var writer = try BatchWriter.init(&database);
|
||||||
|
defer writer.deinit();
|
||||||
|
|
||||||
|
const start = try availableSince(&database);
|
||||||
|
try writer.writeBatch(&.{plainRow(start - 100, "old.example")});
|
||||||
|
try database.exec(
|
||||||
|
\\CREATE TRIGGER refuse_delete BEFORE DELETE ON query_log
|
||||||
|
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
|
||||||
|
);
|
||||||
|
|
||||||
|
try testing.expectError(error.Constraint, pruneOlderThan(&database, start + 1000));
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, 1), try countRows(&database));
|
||||||
|
try testing.expectEqual(start, try availableSince(&database));
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a failed watermark update leaves the rows it had already deleted" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
var writer = try BatchWriter.init(&database);
|
||||||
|
defer writer.deinit();
|
||||||
|
|
||||||
|
const start = try availableSince(&database);
|
||||||
|
try writer.writeBatch(&.{plainRow(start - 100, "old.example")});
|
||||||
|
// The delete succeeds and the advance does not. Without one transaction
|
||||||
|
// around the pair, this is the case that loses rows the watermark still
|
||||||
|
// promises.
|
||||||
|
try database.exec(
|
||||||
|
\\CREATE TRIGGER refuse_advance BEFORE UPDATE ON querylog_meta
|
||||||
|
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
|
||||||
|
);
|
||||||
|
|
||||||
|
try testing.expectError(error.Constraint, pruneOlderThan(&database, start + 1000));
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, 1), try countRows(&database));
|
||||||
|
try testing.expectEqual(start, try availableSince(&database));
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a failed commit rolls back the delete and the watermark together" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
var writer = try BatchWriter.init(&database);
|
||||||
|
defer writer.deinit();
|
||||||
|
|
||||||
|
const start = try availableSince(&database);
|
||||||
|
try writer.writeBatch(&.{plainRow(start - 100, "old.example")});
|
||||||
|
|
||||||
|
// Both statements succeed and COMMIT is what fails: the advance inserts a
|
||||||
|
// `query_log` row whose `domain_id` references nothing, and
|
||||||
|
// `defer_foreign_keys` holds that violation back until the commit checks
|
||||||
|
// it (SQLite's documented semantics for the pragma; the assertions below
|
||||||
|
// observe the rollback, not the moment the check ran).
|
||||||
|
try database.exec(
|
||||||
|
\\CREATE TRIGGER break_at_commit AFTER UPDATE ON querylog_meta
|
||||||
|
\\BEGIN INSERT INTO query_log
|
||||||
|
\\ (timestamp, domain_id, client_ip, blocked, qclass, rcode,
|
||||||
|
\\ policy_action, policy_reason, route_kind)
|
||||||
|
\\VALUES (1, 999999, 'x', 0, 1, 0, 'allow', 'no_match', 'upstream'); END;
|
||||||
|
);
|
||||||
|
try database.exec("PRAGMA defer_foreign_keys = ON;");
|
||||||
|
|
||||||
|
try testing.expectError(error.Constraint, pruneOlderThan(&database, start + 1000));
|
||||||
|
|
||||||
|
// Nothing survived: not the delete, not the advance, not the row the
|
||||||
|
// trigger inserted.
|
||||||
|
try testing.expectEqual(@as(i64, 1), try countRows(&database));
|
||||||
|
try testing.expectEqual(start, try availableSince(&database));
|
||||||
|
try testing.expectEqual(
|
||||||
|
@as(i64, 0),
|
||||||
|
try database.queryInt("SELECT count(*) FROM query_log WHERE client_ip = 'x'"),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
test "pruneOlderThan leaves the domains dimension table intact" {
|
test "pruneOlderThan leaves the domains dimension table intact" {
|
||||||
@@ -662,7 +1080,7 @@ test "pruneOlderThan leaves the domains dimension table intact" {
|
|||||||
defer writer.deinit();
|
defer writer.deinit();
|
||||||
|
|
||||||
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(11, "b.example") });
|
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(11, "b.example") });
|
||||||
try testing.expectEqual(@as(i64, 2), try pruneOlderThan(&database, 1000));
|
try testing.expectEqual(@as(i64, 2), (try pruneOlderThan(&database, 1000)).deleted);
|
||||||
|
|
||||||
try testing.expectEqual(@as(i64, 0), try countRows(&database));
|
try testing.expectEqual(@as(i64, 0), try countRows(&database));
|
||||||
try testing.expectEqual(@as(i64, 2), try countDomains(&database));
|
try testing.expectEqual(@as(i64, 2), try countDomains(&database));
|
||||||
@@ -749,7 +1167,7 @@ test "checkpointTruncate and vacuum run against a WAL file database" {
|
|||||||
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(20, "b.example") });
|
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(20, "b.example") });
|
||||||
|
|
||||||
try checkpointTruncate(&database);
|
try checkpointTruncate(&database);
|
||||||
try testing.expectEqual(@as(i64, 1), try pruneOlderThan(&database, 20));
|
try testing.expectEqual(@as(i64, 1), (try pruneOlderThan(&database, 20)).deleted);
|
||||||
try checkpointTruncate(&database);
|
try checkpointTruncate(&database);
|
||||||
try vacuum(&database);
|
try vacuum(&database);
|
||||||
|
|
||||||
@@ -785,22 +1203,46 @@ test "selectQueries returns the newest row first and reads every column" {
|
|||||||
.domain = "ads.example.net",
|
.domain = "ads.example.net",
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 28,
|
.qtype = 28,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = true,
|
.blocked = true,
|
||||||
.block_reason = "blocklist",
|
|
||||||
.response_time_us = 4200,
|
.response_time_us = 4200,
|
||||||
.cache_hit = true,
|
.cache_hit = true,
|
||||||
.upstream = "https://dns.example/dns-query",
|
.upstream = "https://dns.example/dns-query",
|
||||||
|
.group_id = 2,
|
||||||
|
.group_name = "kids",
|
||||||
|
.policy_action = .block,
|
||||||
|
.policy_reason = .blocklist_domain,
|
||||||
|
.matched = "ads.example.net",
|
||||||
|
.source_id = 5,
|
||||||
|
.source_name = "steven black",
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .blocked,
|
||||||
|
.forward_zone = null,
|
||||||
},
|
},
|
||||||
.{
|
.{
|
||||||
.timestamp = 20,
|
.timestamp = 20,
|
||||||
.domain = "quiet.example",
|
.domain = "quiet.example",
|
||||||
.client_ip = "hidden",
|
.client_ip = "hidden",
|
||||||
.qtype = null,
|
.qtype = null,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 2,
|
||||||
.blocked = false,
|
.blocked = false,
|
||||||
.block_reason = null,
|
|
||||||
.response_time_us = null,
|
.response_time_us = null,
|
||||||
.cache_hit = null,
|
.cache_hit = null,
|
||||||
.upstream = null,
|
.upstream = null,
|
||||||
|
.group_id = null,
|
||||||
|
.group_name = null,
|
||||||
|
.policy_action = .not_evaluated,
|
||||||
|
.policy_reason = .snapshot_unavailable,
|
||||||
|
.matched = null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .rejected,
|
||||||
|
.forward_zone = null,
|
||||||
},
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
@@ -813,12 +1255,16 @@ test "selectQueries returns the newest row first and reads every column" {
|
|||||||
try testing.expectEqualStrings("quiet.example", newest.domain);
|
try testing.expectEqualStrings("quiet.example", newest.domain);
|
||||||
try testing.expectEqualStrings("hidden", newest.client_ip);
|
try testing.expectEqualStrings("hidden", newest.client_ip);
|
||||||
try testing.expectEqual(@as(?u16, null), newest.qtype);
|
try testing.expectEqual(@as(?u16, null), newest.qtype);
|
||||||
|
try testing.expectEqual(@as(u16, 1), newest.qclass);
|
||||||
|
try testing.expectEqual(@as(u12, 2), newest.rcode);
|
||||||
try testing.expect(!newest.blocked);
|
try testing.expect(!newest.blocked);
|
||||||
// A NULL text column reads as the empty string, by documented convention.
|
|
||||||
try testing.expectEqualStrings("", newest.block_reason);
|
|
||||||
try testing.expectEqual(@as(?i64, null), newest.response_time_us);
|
try testing.expectEqual(@as(?i64, null), newest.response_time_us);
|
||||||
try testing.expectEqual(@as(?bool, null), newest.cache_hit);
|
try testing.expectEqual(@as(?bool, null), newest.cache_hit);
|
||||||
|
// A NULL text column reads as the empty string, by documented convention.
|
||||||
try testing.expectEqualStrings("", newest.upstream);
|
try testing.expectEqualStrings("", newest.upstream);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.not_evaluated, newest.policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.snapshot_unavailable, newest.policy_reason);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.rejected, newest.route_kind);
|
||||||
|
|
||||||
const oldest = rows.items[1];
|
const oldest = rows.items[1];
|
||||||
try testing.expectEqual(@as(i64, 1), oldest.id);
|
try testing.expectEqual(@as(i64, 1), oldest.id);
|
||||||
@@ -826,11 +1272,15 @@ test "selectQueries returns the newest row first and reads every column" {
|
|||||||
try testing.expectEqualStrings("ads.example.net", oldest.domain);
|
try testing.expectEqualStrings("ads.example.net", oldest.domain);
|
||||||
try testing.expectEqualStrings("192.0.2.10", oldest.client_ip);
|
try testing.expectEqualStrings("192.0.2.10", oldest.client_ip);
|
||||||
try testing.expectEqual(@as(?u16, 28), oldest.qtype);
|
try testing.expectEqual(@as(?u16, 28), oldest.qtype);
|
||||||
|
try testing.expectEqual(@as(u16, 1), oldest.qclass);
|
||||||
|
try testing.expectEqual(@as(u12, 0), oldest.rcode);
|
||||||
try testing.expect(oldest.blocked);
|
try testing.expect(oldest.blocked);
|
||||||
try testing.expectEqualStrings("blocklist", oldest.block_reason);
|
|
||||||
try testing.expectEqual(@as(?i64, 4200), oldest.response_time_us);
|
try testing.expectEqual(@as(?i64, 4200), oldest.response_time_us);
|
||||||
try testing.expectEqual(@as(?bool, true), oldest.cache_hit);
|
try testing.expectEqual(@as(?bool, true), oldest.cache_hit);
|
||||||
try testing.expectEqualStrings("https://dns.example/dns-query", oldest.upstream);
|
try testing.expectEqualStrings("https://dns.example/dns-query", oldest.upstream);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.block, oldest.policy_action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.blocklist_domain, oldest.policy_reason);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.blocked, oldest.route_kind);
|
||||||
}
|
}
|
||||||
|
|
||||||
test "selectQueries honours the limit and caps it at max_limit" {
|
test "selectQueries honours the limit and caps it at max_limit" {
|
||||||
@@ -895,7 +1345,9 @@ test "each filter narrows the result on its own" {
|
|||||||
var blocked_row = plainRow(200, "ads.example.net");
|
var blocked_row = plainRow(200, "ads.example.net");
|
||||||
blocked_row.client_ip = "192.0.2.20";
|
blocked_row.client_ip = "192.0.2.20";
|
||||||
blocked_row.blocked = true;
|
blocked_row.blocked = true;
|
||||||
blocked_row.block_reason = "blocklist";
|
blocked_row.policy_action = .block;
|
||||||
|
blocked_row.policy_reason = .blocklist_domain;
|
||||||
|
blocked_row.route_kind = .blocked;
|
||||||
try seed(&database, &.{
|
try seed(&database, &.{
|
||||||
plainRow(100, "one.example.com"),
|
plainRow(100, "one.example.com"),
|
||||||
blocked_row,
|
blocked_row,
|
||||||
@@ -1004,7 +1456,9 @@ test "statsTotals aggregates the window and averages only the timed rows" {
|
|||||||
timed.response_time_us = 100;
|
timed.response_time_us = 100;
|
||||||
var blocked_row = plainRow(150, "ads.example");
|
var blocked_row = plainRow(150, "ads.example");
|
||||||
blocked_row.blocked = true;
|
blocked_row.blocked = true;
|
||||||
blocked_row.block_reason = "blocklist";
|
blocked_row.policy_action = .block;
|
||||||
|
blocked_row.policy_reason = .blocklist_domain;
|
||||||
|
blocked_row.route_kind = .blocked;
|
||||||
blocked_row.response_time_us = 200;
|
blocked_row.response_time_us = 200;
|
||||||
var cached = plainRow(199, "b.example");
|
var cached = plainRow(199, "b.example");
|
||||||
cached.client_ip = "192.0.2.99";
|
cached.client_ip = "192.0.2.99";
|
||||||
@@ -1042,7 +1496,9 @@ test "timeseries writes every bucket, including the ones with no rows" {
|
|||||||
|
|
||||||
var blocked_row = plainRow(1020, "ads.example");
|
var blocked_row = plainRow(1020, "ads.example");
|
||||||
blocked_row.blocked = true;
|
blocked_row.blocked = true;
|
||||||
blocked_row.block_reason = "blocklist";
|
blocked_row.policy_action = .block;
|
||||||
|
blocked_row.policy_reason = .blocklist_domain;
|
||||||
|
blocked_row.route_kind = .blocked;
|
||||||
var cached = plainRow(1035, "b.example");
|
var cached = plainRow(1035, "b.example");
|
||||||
cached.cache_hit = true;
|
cached.cache_hit = true;
|
||||||
try seed(&database, &.{
|
try seed(&database, &.{
|
||||||
|
|||||||
@@ -114,8 +114,10 @@ pub const Retention = struct {
|
|||||||
// still prunes through `Store.init`.
|
// still prunes through `Store.init`.
|
||||||
if (store) |s| s.prune(io, now);
|
if (store) |s| s.prune(io, now);
|
||||||
|
|
||||||
if (queries_repo.pruneOlderThan(database, cutoff)) |deleted| {
|
// One operation, not two: the delete and the coverage watermark it
|
||||||
add(&self.counters.rows_pruned, @intCast(deleted));
|
// advances commit together or not at all (`queries_repo`).
|
||||||
|
if (queries_repo.pruneOlderThan(database, cutoff)) |pruned| {
|
||||||
|
add(&self.counters.rows_pruned, @intCast(pruned.deleted));
|
||||||
maintenance(store, io, now, "prune", null);
|
maintenance(store, io, now, "prune", null);
|
||||||
} else |err| {
|
} else |err| {
|
||||||
log.warn("retention prune before {d} failed: {s}", .{ cutoff, @errorName(err) });
|
log.warn("retention prune before {d} failed: {s}", .{ cutoff, @errorName(err) });
|
||||||
@@ -255,11 +257,23 @@ fn writeRows(database: *db.Db, timestamps: []const i64) !void {
|
|||||||
.domain = "example.com",
|
.domain = "example.com",
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = false,
|
.blocked = false,
|
||||||
.block_reason = null,
|
|
||||||
.response_time_us = null,
|
.response_time_us = null,
|
||||||
.cache_hit = null,
|
.cache_hit = null,
|
||||||
.upstream = null,
|
.upstream = null,
|
||||||
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = .allow,
|
||||||
|
.policy_reason = .no_match,
|
||||||
|
.matched = null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
try writer.writeBatch(rows[0..timestamps.len]);
|
try writer.writeBatch(rows[0..timestamps.len]);
|
||||||
|
|||||||
@@ -33,11 +33,13 @@ comptime {
|
|||||||
_ = @import("server/resolver_integration_test.zig");
|
_ = @import("server/resolver_integration_test.zig");
|
||||||
_ = @import("storage/db.zig");
|
_ = @import("storage/db.zig");
|
||||||
_ = @import("config/model.zig");
|
_ = @import("config/model.zig");
|
||||||
|
_ = @import("config/limits.zig");
|
||||||
_ = @import("config/validate.zig");
|
_ = @import("config/validate.zig");
|
||||||
_ = @import("config/faults.zig");
|
_ = @import("config/faults.zig");
|
||||||
_ = @import("storage/config_schema.zig");
|
_ = @import("storage/config_schema.zig");
|
||||||
_ = @import("storage/migrations.zig");
|
_ = @import("storage/migrations.zig");
|
||||||
_ = @import("storage/querylog_schema.zig");
|
_ = @import("storage/querylog_schema.zig");
|
||||||
|
_ = @import("storage/provenance.zig");
|
||||||
_ = @import("storage/repositories/context.zig");
|
_ = @import("storage/repositories/context.zig");
|
||||||
_ = @import("storage/repositories/crud.zig");
|
_ = @import("storage/repositories/crud.zig");
|
||||||
_ = @import("storage/repositories/groups_repo.zig");
|
_ = @import("storage/repositories/groups_repo.zig");
|
||||||
@@ -92,6 +94,8 @@ comptime {
|
|||||||
_ = @import("server/shutdown.zig");
|
_ = @import("server/shutdown.zig");
|
||||||
_ = @import("server/phase7_integration_test.zig");
|
_ = @import("server/phase7_integration_test.zig");
|
||||||
_ = @import("web/sse.zig");
|
_ = @import("web/sse.zig");
|
||||||
|
_ = @import("web/coverage.zig");
|
||||||
|
_ = @import("web/provenance_view.zig");
|
||||||
_ = @import("server/query_sink.zig");
|
_ = @import("server/query_sink.zig");
|
||||||
_ = @import("web/auth.zig");
|
_ = @import("web/auth.zig");
|
||||||
_ = @import("web/api_limiter.zig");
|
_ = @import("web/api_limiter.zig");
|
||||||
|
|||||||
@@ -75,8 +75,12 @@ pub const DohClient = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
const self: *DohClient = @ptrCast(@alignCast(ptr));
|
const self: *DohClient = @ptrCast(@alignCast(ptr));
|
||||||
|
// The endpoint outlives the client, so the borrow is safe for the whole
|
||||||
|
// query. Set before the attempt: a failure names this resolver too.
|
||||||
|
selected.* = self.endpoint.url;
|
||||||
return self.exchange(io, query, response_buf);
|
return self.exchange(io, query, response_buf);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -44,7 +44,9 @@ fn runExchange(io: std.Io, params: Params) anyerror!usize {
|
|||||||
const endpoint = try transport.Endpoint.parse("https://cloudflare-dns.com/dns-query");
|
const endpoint = try transport.Endpoint.parse("https://cloudflare-dns.com/dns-query");
|
||||||
var doh = try doh_client.DohClient.init(&http, endpoint, &request_buf, &transfer_buf);
|
var doh = try doh_client.DohClient.init(&http, endpoint, &request_buf, &transfer_buf);
|
||||||
|
|
||||||
const reply = try doh.client().exchange(io, query_bytes, params.response_buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
const reply = try doh.client().exchange(io, query_bytes, params.response_buf, &selected);
|
||||||
|
std.debug.assert(std.mem.eql(u8, selected.?, endpoint.url));
|
||||||
return reply.len;
|
return reply.len;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -146,8 +146,12 @@ pub const DotClient = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
const self: *DotClient = @ptrCast(@alignCast(ptr));
|
const self: *DotClient = @ptrCast(@alignCast(ptr));
|
||||||
|
// The endpoint outlives the client, so the borrow is safe for the whole
|
||||||
|
// query. Set before the attempt: a failure names this resolver too.
|
||||||
|
selected.* = self.endpoint.url;
|
||||||
return self.exchange(io, query, response_buf);
|
return self.exchange(io, query, response_buf);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -57,7 +57,9 @@ fn runExchange(io: std.Io, params: Params) anyerror!usize {
|
|||||||
params.bundle_lock,
|
params.bundle_lock,
|
||||||
params.buffers,
|
params.buffers,
|
||||||
);
|
);
|
||||||
const reply = try client.client().exchange(io, query_bytes, params.response_buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
const reply = try client.client().exchange(io, query_bytes, params.response_buf, &selected);
|
||||||
|
std.debug.assert(std.mem.eql(u8, selected.?, endpoint.url));
|
||||||
return reply.len;
|
return reply.len;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
+291
-46
@@ -172,9 +172,10 @@ pub const Pool = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
const self: *Pool = @ptrCast(@alignCast(ptr));
|
const self: *Pool = @ptrCast(@alignCast(ptr));
|
||||||
return self.exchange(io, query, response_buf);
|
return self.exchange(io, query, response_buf, selected);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// `response_buf` is handed to each attempt in turn, so a failed attempt
|
/// `response_buf` is handed to each attempt in turn, so a failed attempt
|
||||||
@@ -191,15 +192,16 @@ pub const Pool = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
const len = try transport.raceWithin(io, self.timeouts.total, exchangeLoopLen, .{
|
const len = try transport.raceWithin(io, self.timeouts.total, exchangeLoopLen, .{
|
||||||
self, io, query, response_buf,
|
self, io, query, response_buf, selected,
|
||||||
});
|
});
|
||||||
return response_buf[0..len];
|
return response_buf[0..len];
|
||||||
}
|
}
|
||||||
|
|
||||||
/// The two-pass failover loop, as a raceable task. It returns the reply's
|
/// The two-pass failover loop, as a raceable task. It returns the reply's
|
||||||
/// length rather than its slice for the reason `exchangeLen` in the tests
|
/// length rather than its slice for the reason `Attributed` in the tests
|
||||||
/// below does: `Io.concurrent` stores the future's return value, so the
|
/// below does: `Io.concurrent` stores the future's return value, so the
|
||||||
/// bytes are read back out of the caller's `response_buf` by `exchange`.
|
/// bytes are read back out of the caller's `response_buf` by `exchange`.
|
||||||
fn exchangeLoopLen(
|
fn exchangeLoopLen(
|
||||||
@@ -207,6 +209,7 @@ pub const Pool = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError!usize {
|
) transport.ExchangeError!usize {
|
||||||
const now = std.Io.Clock.awake.now(io);
|
const now = std.Io.Clock.awake.now(io);
|
||||||
var last_fault: ?transport.ExchangeError = null;
|
var last_fault: ?transport.ExchangeError = null;
|
||||||
@@ -240,6 +243,13 @@ pub const Pool = struct {
|
|||||||
}
|
}
|
||||||
attempted = true;
|
attempted = true;
|
||||||
|
|
||||||
|
// Before the attempt, not after it: a failover reports whoever
|
||||||
|
// answered, an all-failed exchange reports the last endpoint
|
||||||
|
// tried, and a cancellation mid-flight reports the endpoint the
|
||||||
|
// query was in. The url is owned by the Endpoint, which outlives
|
||||||
|
// the pool, so the borrow stays valid past this loop.
|
||||||
|
selected.* = entry.endpoint.url;
|
||||||
|
|
||||||
const result = self.attempt(io, entry.client, query, response_buf);
|
const result = self.attempt(io, entry.client, query, response_buf);
|
||||||
const completed_at = std.Io.Clock.awake.now(io);
|
const completed_at = std.Io.Clock.awake.now(io);
|
||||||
|
|
||||||
@@ -294,6 +304,12 @@ pub const Pool = struct {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/// One exchange raced against the per-attempt budget.
|
/// One exchange raced against the per-attempt budget.
|
||||||
|
///
|
||||||
|
/// The leaf client reports an identity of its own, which the pool discards:
|
||||||
|
/// the entry's endpoint is the pool's own naming of the same resolver, and
|
||||||
|
/// it is what the caller was handed. A leaf writing its identity into the
|
||||||
|
/// caller's slot would let a test fake overwrite the endpoint that actually
|
||||||
|
/// answered.
|
||||||
fn attempt(
|
fn attempt(
|
||||||
self: *Pool,
|
self: *Pool,
|
||||||
io: std.Io,
|
io: std.Io,
|
||||||
@@ -301,8 +317,9 @@ pub const Pool = struct {
|
|||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
|
var leaf_selected: ?[]const u8 = null;
|
||||||
return transport.raceWithin(io, self.timeouts.attempt, transport.Client.exchange, .{
|
return transport.raceWithin(io, self.timeouts.attempt, transport.Client.exchange, .{
|
||||||
entry_client, io, query, response_buf,
|
entry_client, io, query, response_buf, &leaf_selected,
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -409,11 +426,23 @@ const response_bytes =
|
|||||||
"\x07example\x03com\x00\x00\x01\x00\x01" ++
|
"\x07example\x03com\x00\x00\x01\x00\x01" ++
|
||||||
"\xc0\x0c\x00\x01\x00\x01\x00\x00\x01\x2c\x00\x04\x5d\xb8\xd8\x22";
|
"\xc0\x0c\x00\x01\x00\x01\x00\x00\x01\x2c\x00\x04\x5d\xb8\xd8\x22";
|
||||||
|
|
||||||
|
/// The same question answered differently, so a reply identifies the entry that
|
||||||
|
/// produced it: one A record, a different address.
|
||||||
|
const alt_response_bytes =
|
||||||
|
"\x12\x34\x81\x80\x00\x01\x00\x01\x00\x00\x00\x00" ++
|
||||||
|
"\x07example\x03com\x00\x00\x01\x00\x01" ++
|
||||||
|
"\xc0\x0c\x00\x01\x00\x01\x00\x00\x01\x2c\x00\x04\x0a\x00\x00\x01";
|
||||||
|
|
||||||
/// Stands in for a DoH or DoT client. Every behaviour the pool has to react to
|
/// Stands in for a DoH or DoT client. Every behaviour the pool has to react to
|
||||||
/// is one variant, and every call is counted so a test can assert that an entry
|
/// is one variant, and every call is counted so a test can assert that an entry
|
||||||
/// in backoff was not touched.
|
/// in backoff was not touched.
|
||||||
const Fake = struct {
|
const Fake = struct {
|
||||||
behavior: Behavior,
|
behavior: Behavior,
|
||||||
|
/// Replaces `behavior` after the first call. One entry that fails the task
|
||||||
|
/// which reaches it first and answers the next is what makes two concurrent
|
||||||
|
/// exchanges end on different entries; a single behaviour cannot say that.
|
||||||
|
/// Mutated under the entry's `busy` lock, like `calls`.
|
||||||
|
then: ?Behavior = null,
|
||||||
calls: usize = 0,
|
calls: usize = 0,
|
||||||
in_flight: std.atomic.Value(u32) = .init(0),
|
in_flight: std.atomic.Value(u32) = .init(0),
|
||||||
/// The most tasks ever inside `exchangeFn` at once. The per-entry lock is
|
/// The most tasks ever inside `exchangeFn` at once. The per-entry lock is
|
||||||
@@ -437,15 +466,24 @@ const Fake = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = query;
|
_ = query;
|
||||||
|
// Deliberately not the endpoint url: the pool must report its own
|
||||||
|
// entry, so a test can tell the two apart.
|
||||||
|
selected.* = "fake://leaf";
|
||||||
const self: *Fake = @ptrCast(@alignCast(ptr));
|
const self: *Fake = @ptrCast(@alignCast(ptr));
|
||||||
const entrants = self.in_flight.fetchAdd(1, .acq_rel) + 1;
|
const entrants = self.in_flight.fetchAdd(1, .acq_rel) + 1;
|
||||||
defer _ = self.in_flight.fetchSub(1, .acq_rel);
|
defer _ = self.in_flight.fetchSub(1, .acq_rel);
|
||||||
_ = self.peak_in_flight.fetchMax(entrants, .acq_rel);
|
_ = self.peak_in_flight.fetchMax(entrants, .acq_rel);
|
||||||
|
|
||||||
self.calls += 1;
|
self.calls += 1;
|
||||||
switch (self.behavior) {
|
const behavior = self.behavior;
|
||||||
|
if (self.then) |next| {
|
||||||
|
self.behavior = next;
|
||||||
|
self.then = null;
|
||||||
|
}
|
||||||
|
switch (behavior) {
|
||||||
.reply => |bytes| return copy(bytes, response_buf),
|
.reply => |bytes| return copy(bytes, response_buf),
|
||||||
.fail => |err| return err,
|
.fail => |err| return err,
|
||||||
.slow => |slow| {
|
.slow => |slow| {
|
||||||
@@ -539,10 +577,106 @@ test "Pool satisfies the Client interface" {
|
|||||||
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
const reply = try pool.client().exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
const reply = try pool.client().exchange(io, query_bytes, &buf, &selected);
|
||||||
try testing.expectEqualSlices(u8, response_bytes, reply);
|
try testing.expectEqualSlices(u8, response_bytes, reply);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "the pool reports the endpoint that answered, not the leaf client's own name" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var fake: Fake = .{ .behavior = .{ .reply = response_bytes } };
|
||||||
|
var entries = [_]Entry{testEntry("https://a.example/dns-query", &fake, 10)};
|
||||||
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
|
var buf: [512]u8 = undefined;
|
||||||
|
var selected: ?[]const u8 = null;
|
||||||
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
|
try testing.expectEqualStrings("https://a.example/dns-query", selected.?);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a failover reports the endpoint that answered, not the first one tried" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var bad: Fake = .{ .behavior = .{ .fail = error.Timeout } };
|
||||||
|
var good: Fake = .{ .behavior = .{ .reply = response_bytes } };
|
||||||
|
var entries = [_]Entry{
|
||||||
|
testEntry("https://bad.example/dns-query", &bad, 10),
|
||||||
|
testEntry("https://good.example/dns-query", &good, 20),
|
||||||
|
};
|
||||||
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
|
var buf: [512]u8 = undefined;
|
||||||
|
var selected: ?[]const u8 = null;
|
||||||
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
|
try testing.expectEqualStrings("https://good.example/dns-query", selected.?);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "an all-failed exchange reports the last endpoint attempted" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var first: Fake = .{ .behavior = .{ .fail = error.ConnectFailed } };
|
||||||
|
var last: Fake = .{ .behavior = .{ .fail = error.Timeout } };
|
||||||
|
var entries = [_]Entry{
|
||||||
|
testEntry("https://first.example/dns-query", &first, 10),
|
||||||
|
testEntry("https://last.example/dns-query", &last, 20),
|
||||||
|
};
|
||||||
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
|
var buf: [512]u8 = undefined;
|
||||||
|
var selected: ?[]const u8 = null;
|
||||||
|
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
|
try testing.expectEqualStrings("https://last.example/dns-query", selected.?);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a timeout mid-flight reports the endpoint the query was in" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var stalling: Fake = .{ .behavior = .{ .slow = .{
|
||||||
|
.duration = .{ .raw = .fromSeconds(30), .clock = .awake },
|
||||||
|
.reply = response_bytes,
|
||||||
|
} } };
|
||||||
|
var untouched: Fake = .{ .behavior = .{ .reply = response_bytes } };
|
||||||
|
var entries = [_]Entry{
|
||||||
|
testEntry("https://stalling.example/dns-query", &stalling, 10),
|
||||||
|
testEntry("https://untouched.example/dns-query", &untouched, 20),
|
||||||
|
};
|
||||||
|
var pool: Pool = .init(&entries, test_cfg, .{
|
||||||
|
.attempt = .{ .raw = .fromMilliseconds(200), .clock = .awake },
|
||||||
|
.total = .{ .raw = .fromMilliseconds(60), .clock = .awake },
|
||||||
|
}, 1);
|
||||||
|
|
||||||
|
var buf: [512]u8 = undefined;
|
||||||
|
var selected: ?[]const u8 = null;
|
||||||
|
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
|
try testing.expectEqualStrings("https://stalling.example/dns-query", selected.?);
|
||||||
|
try testing.expectEqual(@as(usize, 0), untouched.calls);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "an exchange that attempted nothing reports no endpoint" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
var fake: Fake = .{ .behavior = .{ .reply = response_bytes } };
|
||||||
|
var entries = [_]Entry{testEntry("https://a.example/dns-query", &fake, 10)};
|
||||||
|
entries[0].enabled = false;
|
||||||
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
|
var buf: [512]u8 = undefined;
|
||||||
|
var selected: ?[]const u8 = null;
|
||||||
|
try testing.expectError(error.ConnectFailed, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
|
try testing.expect(selected == null);
|
||||||
|
}
|
||||||
|
|
||||||
test "entries are tried in ascending priority order" {
|
test "entries are tried in ascending priority order" {
|
||||||
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
defer threaded.deinit();
|
defer threaded.deinit();
|
||||||
@@ -560,7 +694,8 @@ test "entries are tried in ascending priority order" {
|
|||||||
try testing.expectEqual(@as(i32, 10), entries[0].priority);
|
try testing.expectEqual(@as(i32, 10), entries[0].priority);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
try testing.expectEqual(@as(usize, 1), low.calls);
|
try testing.expectEqual(@as(usize, 1), low.calls);
|
||||||
try testing.expectEqual(@as(usize, 0), high.calls);
|
try testing.expectEqual(@as(usize, 0), high.calls);
|
||||||
}
|
}
|
||||||
@@ -579,7 +714,8 @@ test "a peer fault fails over to the next entry and is recorded" {
|
|||||||
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
const reply = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
const reply = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
try testing.expectEqualSlices(u8, response_bytes, reply);
|
try testing.expectEqualSlices(u8, response_bytes, reply);
|
||||||
|
|
||||||
try testing.expectEqual(@as(u32, 1), entries[0].health.consecutive_failures);
|
try testing.expectEqual(@as(u32, 1), entries[0].health.consecutive_failures);
|
||||||
@@ -603,12 +739,13 @@ test "an entry in backoff is skipped while another is available" {
|
|||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
// Two failures reach `failure_threshold` and open a backoff window.
|
// Two failures reach `failure_threshold` and open a backoff window.
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
try testing.expectEqual(@as(usize, 2), bad.calls);
|
try testing.expectEqual(@as(usize, 2), bad.calls);
|
||||||
try testing.expect(entries[0].health.backoff_until != null);
|
try testing.expect(entries[0].health.backoff_until != null);
|
||||||
|
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
try testing.expectEqual(@as(usize, 2), bad.calls);
|
try testing.expectEqual(@as(usize, 2), bad.calls);
|
||||||
try testing.expectEqual(@as(usize, 3), good.calls);
|
try testing.expectEqual(@as(usize, 3), good.calls);
|
||||||
}
|
}
|
||||||
@@ -627,13 +764,14 @@ test "every entry in backoff is still probed" {
|
|||||||
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf));
|
var selected: ?[]const u8 = null;
|
||||||
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf));
|
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
|
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
try testing.expect(entries[0].health.backoff_until != null);
|
try testing.expect(entries[0].health.backoff_until != null);
|
||||||
try testing.expect(entries[1].health.backoff_until != null);
|
try testing.expect(entries[1].health.backoff_until != null);
|
||||||
|
|
||||||
// Pass one now has no candidate at all. Pass two probes both anyway.
|
// Pass one now has no candidate at all. Pass two probes both anyway.
|
||||||
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf));
|
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
try testing.expectEqual(@as(usize, 3), first.calls);
|
try testing.expectEqual(@as(usize, 3), first.calls);
|
||||||
try testing.expectEqual(@as(usize, 3), second.calls);
|
try testing.expectEqual(@as(usize, 3), second.calls);
|
||||||
}
|
}
|
||||||
@@ -652,7 +790,8 @@ test "a local resource error short-circuits and records nothing" {
|
|||||||
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
try testing.expectError(error.OutOfMemory, pool.exchange(io, query_bytes, &buf));
|
var selected: ?[]const u8 = null;
|
||||||
|
try testing.expectError(error.OutOfMemory, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
try testing.expectEqual(@as(usize, 0), good.calls);
|
try testing.expectEqual(@as(usize, 0), good.calls);
|
||||||
try testing.expectEqual(@as(u64, 0), entries[0].health.total_failures);
|
try testing.expectEqual(@as(u64, 0), entries[0].health.total_failures);
|
||||||
try testing.expectEqual(@as(u32, 0), entries[0].health.consecutive_failures);
|
try testing.expectEqual(@as(u32, 0), entries[0].health.consecutive_failures);
|
||||||
@@ -672,7 +811,8 @@ test "a cancellation short-circuits and records nothing" {
|
|||||||
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
try testing.expectError(error.Canceled, pool.exchange(io, query_bytes, &buf));
|
var selected: ?[]const u8 = null;
|
||||||
|
try testing.expectError(error.Canceled, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
try testing.expectEqual(@as(usize, 0), good.calls);
|
try testing.expectEqual(@as(usize, 0), good.calls);
|
||||||
try testing.expectEqual(@as(u64, 0), entries[0].health.total_failures);
|
try testing.expectEqual(@as(u64, 0), entries[0].health.total_failures);
|
||||||
}
|
}
|
||||||
@@ -699,7 +839,8 @@ test "an attempt that outruns the budget is a recorded Timeout" {
|
|||||||
}, 1);
|
}, 1);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
const reply = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
const reply = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
try testing.expectEqualSlices(u8, response_bytes, reply);
|
try testing.expectEqualSlices(u8, response_bytes, reply);
|
||||||
|
|
||||||
try testing.expectEqual(@as(usize, 1), slow.calls);
|
try testing.expectEqual(@as(usize, 1), slow.calls);
|
||||||
@@ -736,7 +877,8 @@ test "two stalling upstreams cost the total budget, not one budget each" {
|
|||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
const started = std.Io.Clock.awake.now(io);
|
const started = std.Io.Clock.awake.now(io);
|
||||||
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf));
|
var selected: ?[]const u8 = null;
|
||||||
|
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
const elapsed_ns = std.Io.Clock.awake.now(io).nanoseconds - started.nanoseconds;
|
const elapsed_ns = std.Io.Clock.awake.now(io).nanoseconds - started.nanoseconds;
|
||||||
|
|
||||||
// Under one attempt budget, so the outer deadline is provably what fired.
|
// Under one attempt budget, so the outer deadline is provably what fired.
|
||||||
@@ -770,7 +912,8 @@ test "every entry disabled yields ConnectFailed without waiting out the total bu
|
|||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
const started = std.Io.Clock.awake.now(io);
|
const started = std.Io.Clock.awake.now(io);
|
||||||
try testing.expectError(error.ConnectFailed, pool.exchange(io, query_bytes, &buf));
|
var selected: ?[]const u8 = null;
|
||||||
|
try testing.expectError(error.ConnectFailed, pool.exchange(io, query_bytes, &buf, &selected));
|
||||||
const elapsed_ns = std.Io.Clock.awake.now(io).nanoseconds - started.nanoseconds;
|
const elapsed_ns = std.Io.Clock.awake.now(io).nanoseconds - started.nanoseconds;
|
||||||
|
|
||||||
try testing.expect(elapsed_ns < @as(i96, 5) * std.time.ns_per_s);
|
try testing.expect(elapsed_ns < @as(i96, 5) * std.time.ns_per_s);
|
||||||
@@ -799,7 +942,8 @@ test "a wired accumulator receives both outcomes the pool records" {
|
|||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
// One exchange: the first entry fails over into the second, so this drives
|
// One exchange: the first entry fails over into the second, so this drives
|
||||||
// one failure and one success.
|
// one failure and one success.
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
|
|
||||||
// Two cells, one per url, in whatever minute the wall clock is in.
|
// Two cells, one per url, in whatever minute the wall clock is in.
|
||||||
try testing.expectEqual(@as(u32, 2), acc.snapshotStats(io).pending);
|
try testing.expectEqual(@as(u32, 2), acc.snapshotStats(io).pending);
|
||||||
@@ -852,8 +996,9 @@ test "snapshot reports the counters in pool order" {
|
|||||||
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
|
|
||||||
var out: [4]Snapshot = undefined;
|
var out: [4]Snapshot = undefined;
|
||||||
const written = try pool.snapshot(io, &out);
|
const written = try pool.snapshot(io, &out);
|
||||||
@@ -882,13 +1027,30 @@ test "snapshot reports the counters in pool order" {
|
|||||||
try testing.expectEqual(@as(usize, 1), try pool.snapshot(io, &one));
|
try testing.expectEqual(@as(usize, 1), try pool.snapshot(io, &one));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// One concurrent call's whole result: how much of its buffer the reply filled,
|
||||||
|
/// and which resolver the pool said answered it.
|
||||||
|
///
|
||||||
/// `Io.concurrent` stores the return value in the future, so the reply slice is
|
/// `Io.concurrent` stores the return value in the future, so the reply slice is
|
||||||
/// reduced to its length here and the bytes are read back out of the caller's
|
/// reduced to its length here and the bytes are read back out of the caller's
|
||||||
/// buffer. Returning a `usize` also lets the test discard a result with
|
/// buffer. `selected` survives the trip because it borrows an `Endpoint.url`,
|
||||||
/// `catch 0`.
|
/// which outlives the pool.
|
||||||
fn exchangeLen(pool: *Pool, io: std.Io, buf: []u8) transport.ExchangeError!usize {
|
const Attributed = struct {
|
||||||
const reply = try pool.exchange(io, query_bytes, buf);
|
reply_len: usize,
|
||||||
return reply.len;
|
selected: ?[]const u8,
|
||||||
|
|
||||||
|
/// What a teardown `await` discards into once the assertions above it have
|
||||||
|
/// already taken the value.
|
||||||
|
const discarded: Attributed = .{ .reply_len = 0, .selected = null };
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Each call keeps its own `selected` out-value rather than dropping it: the
|
||||||
|
/// pointer is written per call, on the stack of the task that made it, and a
|
||||||
|
/// pool that hung it off `*Pool` instead would hand one call's identity to
|
||||||
|
/// another. Nothing but a per-call capture can see that.
|
||||||
|
fn exchangeAttributed(pool: *Pool, io: std.Io, buf: []u8) transport.ExchangeError!Attributed {
|
||||||
|
var selected: ?[]const u8 = null;
|
||||||
|
const reply = try pool.exchange(io, query_bytes, buf, &selected);
|
||||||
|
return .{ .reply_len = reply.len, .selected = selected };
|
||||||
}
|
}
|
||||||
|
|
||||||
test "concurrent exchanges through one entry do not overlap" {
|
test "concurrent exchanges through one entry do not overlap" {
|
||||||
@@ -909,25 +1071,102 @@ test "concurrent exchanges through one entry do not overlap" {
|
|||||||
var buf_a: [512]u8 = undefined;
|
var buf_a: [512]u8 = undefined;
|
||||||
var buf_b: [512]u8 = undefined;
|
var buf_b: [512]u8 = undefined;
|
||||||
|
|
||||||
var first = io.concurrent(exchangeLen, .{ &pool, io, &buf_a }) catch |err| switch (err) {
|
var first = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_a }) catch |err| switch (err) {
|
||||||
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
||||||
};
|
};
|
||||||
defer _ = first.await(io) catch 0;
|
defer _ = first.await(io) catch Attributed.discarded;
|
||||||
var second = io.concurrent(exchangeLen, .{ &pool, io, &buf_b }) catch |err| switch (err) {
|
var second = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_b }) catch |err| switch (err) {
|
||||||
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
||||||
};
|
};
|
||||||
defer _ = second.await(io) catch 0;
|
defer _ = second.await(io) catch Attributed.discarded;
|
||||||
|
|
||||||
const len_a = try first.await(io);
|
const result_a = try first.await(io);
|
||||||
const len_b = try second.await(io);
|
const result_b = try second.await(io);
|
||||||
|
|
||||||
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..len_a]);
|
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..result_a.reply_len]);
|
||||||
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..len_b]);
|
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..result_b.reply_len]);
|
||||||
|
try testing.expectEqualStrings("https://only.example/dns-query", result_a.selected.?);
|
||||||
|
try testing.expectEqualStrings("https://only.example/dns-query", result_b.selected.?);
|
||||||
try testing.expectEqual(@as(usize, 2), fake.calls);
|
try testing.expectEqual(@as(usize, 2), fake.calls);
|
||||||
try testing.expectEqual(@as(u32, 1), fake.peak_in_flight.load(.acquire));
|
try testing.expectEqual(@as(u32, 1), fake.peak_in_flight.load(.acquire));
|
||||||
try testing.expectEqual(@as(u64, 2), entries[0].health.total_successes);
|
try testing.expectEqual(@as(u64, 2), entries[0].health.total_successes);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The two entries of the test below, each answering with bytes only it
|
||||||
|
/// produces so a call's reply proves which entry served it independently of
|
||||||
|
/// what the pool reported.
|
||||||
|
const divergent_first_url = "https://first.example/dns-query";
|
||||||
|
const divergent_second_url = "https://second.example/dns-query";
|
||||||
|
|
||||||
|
fn expectAnsweredByReporter(result: Attributed, buf: []const u8) !void {
|
||||||
|
const url = result.selected orelse return error.TestExpectedSelectedResolver;
|
||||||
|
const reply = if (std.mem.eql(u8, url, divergent_first_url))
|
||||||
|
alt_response_bytes
|
||||||
|
else if (std.mem.eql(u8, url, divergent_second_url))
|
||||||
|
response_bytes
|
||||||
|
else
|
||||||
|
return error.TestUnexpectedResolver;
|
||||||
|
try testing.expectEqualSlices(u8, reply, buf[0..result.reply_len]);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "overlapping exchanges each report the entry that answered that call" {
|
||||||
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
|
defer threaded.deinit();
|
||||||
|
const io = threaded.io();
|
||||||
|
|
||||||
|
// The first entry fails the task that reaches it first, slowly enough that
|
||||||
|
// the second task is queued behind it, then answers that second task. One
|
||||||
|
// failure is under `test_cfg`'s threshold of two, so no backoff steers the
|
||||||
|
// waiting task away and the two calls end on different entries.
|
||||||
|
var first_entry: Fake = .{
|
||||||
|
.behavior = .{ .slow_fail = .{
|
||||||
|
.duration = .{ .raw = .fromMilliseconds(50), .clock = .awake },
|
||||||
|
.err = error.ConnectFailed,
|
||||||
|
} },
|
||||||
|
.then = .{ .reply = alt_response_bytes },
|
||||||
|
};
|
||||||
|
// Slow too, so the failed-over call is still in flight while the other call
|
||||||
|
// is being answered — a shared identity would be overwritten under it.
|
||||||
|
var second_entry: Fake = .{ .behavior = .{ .slow = .{
|
||||||
|
.duration = .{ .raw = .fromMilliseconds(50), .clock = .awake },
|
||||||
|
.reply = response_bytes,
|
||||||
|
} } };
|
||||||
|
var entries = [_]Entry{
|
||||||
|
testEntry(divergent_first_url, &first_entry, 10),
|
||||||
|
testEntry(divergent_second_url, &second_entry, 20),
|
||||||
|
};
|
||||||
|
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
|
||||||
|
|
||||||
|
var buf_a: [512]u8 = undefined;
|
||||||
|
var buf_b: [512]u8 = undefined;
|
||||||
|
|
||||||
|
var first = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_a }) catch |err| switch (err) {
|
||||||
|
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
||||||
|
};
|
||||||
|
defer _ = first.await(io) catch Attributed.discarded;
|
||||||
|
var second = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_b }) catch |err| switch (err) {
|
||||||
|
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
||||||
|
};
|
||||||
|
defer _ = second.await(io) catch Attributed.discarded;
|
||||||
|
|
||||||
|
const result_a = try first.await(io);
|
||||||
|
const result_b = try second.await(io);
|
||||||
|
|
||||||
|
// Which task lands where depends on the order they take the first entry's
|
||||||
|
// lock, so the claim is over the pair: one identity each, and each one
|
||||||
|
// matching the bytes that call received.
|
||||||
|
try testing.expect(!std.mem.eql(u8, result_a.selected.?, result_b.selected.?));
|
||||||
|
try expectAnsweredByReporter(result_a, &buf_a);
|
||||||
|
try expectAnsweredByReporter(result_b, &buf_b);
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(usize, 2), first_entry.calls);
|
||||||
|
try testing.expectEqual(@as(usize, 1), second_entry.calls);
|
||||||
|
try testing.expectEqual(@as(u32, 1), first_entry.peak_in_flight.load(.acquire));
|
||||||
|
try testing.expectEqual(@as(u64, 1), entries[0].health.total_failures);
|
||||||
|
try testing.expectEqual(@as(u64, 1), entries[0].health.total_successes);
|
||||||
|
try testing.expectEqual(@as(u64, 1), entries[1].health.total_successes);
|
||||||
|
}
|
||||||
|
|
||||||
test "an entry that enters backoff while a task waits on it is not attempted" {
|
test "an entry that enters backoff while a task waits on it is not attempted" {
|
||||||
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
|
||||||
defer threaded.deinit();
|
defer threaded.deinit();
|
||||||
@@ -956,21 +1195,25 @@ test "an entry that enters backoff while a task waits on it is not attempted" {
|
|||||||
var buf_a: [512]u8 = undefined;
|
var buf_a: [512]u8 = undefined;
|
||||||
var buf_b: [512]u8 = undefined;
|
var buf_b: [512]u8 = undefined;
|
||||||
|
|
||||||
var first = io.concurrent(exchangeLen, .{ &pool, io, &buf_a }) catch |err| switch (err) {
|
var first = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_a }) catch |err| switch (err) {
|
||||||
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
||||||
};
|
};
|
||||||
defer _ = first.await(io) catch 0;
|
defer _ = first.await(io) catch Attributed.discarded;
|
||||||
var second = io.concurrent(exchangeLen, .{ &pool, io, &buf_b }) catch |err| switch (err) {
|
var second = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_b }) catch |err| switch (err) {
|
||||||
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
error.ConcurrencyUnavailable => return error.SkipZigTest,
|
||||||
};
|
};
|
||||||
defer _ = second.await(io) catch 0;
|
defer _ = second.await(io) catch Attributed.discarded;
|
||||||
|
|
||||||
const len_a = try first.await(io);
|
const result_a = try first.await(io);
|
||||||
const len_b = try second.await(io);
|
const result_b = try second.await(io);
|
||||||
|
|
||||||
// Both tasks fail over to the healthy entry and get an answer.
|
// Both tasks fail over to the healthy entry and get an answer, and both
|
||||||
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..len_a]);
|
// report it: the identity is the entry that answered, not the one that
|
||||||
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..len_b]);
|
// failed on the way there.
|
||||||
|
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..result_a.reply_len]);
|
||||||
|
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..result_b.reply_len]);
|
||||||
|
try testing.expectEqualStrings("https://good.example/dns-query", result_a.selected.?);
|
||||||
|
try testing.expectEqualStrings("https://good.example/dns-query", result_b.selected.?);
|
||||||
try testing.expectEqual(@as(usize, 2), good.calls);
|
try testing.expectEqual(@as(usize, 2), good.calls);
|
||||||
|
|
||||||
// The point of the test: the entry was attempted once, not twice. Without
|
// The point of the test: the entry was attempted once, not twice. Without
|
||||||
@@ -998,7 +1241,8 @@ test "a successful exchange with nothing open costs the store no statement" {
|
|||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
const before = fx.store.statements;
|
const before = fx.store.statements;
|
||||||
for (0..20) |_| _ = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
for (0..20) |_| _ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
try testing.expectEqual(before, fx.store.statements);
|
try testing.expectEqual(before, fx.store.statements);
|
||||||
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
|
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
|
||||||
}
|
}
|
||||||
@@ -1022,7 +1266,8 @@ test "a failing then recovering upstream leaves exactly one resolved episode" {
|
|||||||
pool.diagnostics = &fx.store;
|
pool.diagnostics = &fx.store;
|
||||||
|
|
||||||
var buf: [512]u8 = undefined;
|
var buf: [512]u8 = undefined;
|
||||||
_ = try pool.exchange(io, query_bytes, &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
_ = try pool.exchange(io, query_bytes, &buf, &selected);
|
||||||
// Backoff would park the failing entry, so the second failure is driven
|
// Backoff would park the failing entry, so the second failure is driven
|
||||||
// through `recordFailure` itself rather than through another exchange.
|
// through `recordFailure` itself rather than through another exchange.
|
||||||
pool.recordFailure(io, &entries[0], std.Io.Clock.awake.now(io), error.ConnectFailed);
|
pool.recordFailure(io, &entries[0], std.Io.Clock.awake.now(io), error.ConnectFailed);
|
||||||
|
|||||||
@@ -35,6 +35,13 @@ pub fn parsePrefix(bytes: [prefix_len]u8) u16 {
|
|||||||
return std.mem.readInt(u16, &bytes, .big);
|
return std.mem.readInt(u16, &bytes, .big);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The longest DNS name in text form, and so the longest host an endpoint url
|
||||||
|
/// can name. Every consumer of `Endpoint.host` sizes itself from this bound
|
||||||
|
/// rather than re-deriving it: the query log's `upstream` column is built for
|
||||||
|
/// the widest `scheme://host:port` this permits, so a longer host would reach
|
||||||
|
/// storage only as a silently shortened identity.
|
||||||
|
pub const max_host_len = 253;
|
||||||
|
|
||||||
pub const doh_default_port = 443;
|
pub const doh_default_port = 443;
|
||||||
pub const dot_default_port = 853; // RFC 7858 §3.1
|
pub const dot_default_port = 853; // RFC 7858 §3.1
|
||||||
pub const doh_default_path = "/dns-query"; // RFC 8484 §4.1 well-known template
|
pub const doh_default_path = "/dns-query"; // RFC 8484 §4.1 well-known template
|
||||||
@@ -46,14 +53,15 @@ pub const Endpoint = struct {
|
|||||||
scheme: Scheme,
|
scheme: Scheme,
|
||||||
/// The original text, for logs and the health API.
|
/// The original text, for logs and the health API.
|
||||||
url: []const u8,
|
url: []const u8,
|
||||||
/// No brackets, no port. Used for SNI and certificate verification.
|
/// No brackets, no port, never empty, at most `max_host_len` bytes. Used
|
||||||
|
/// for SNI and certificate verification.
|
||||||
host: []const u8,
|
host: []const u8,
|
||||||
port: u16,
|
port: u16,
|
||||||
/// DoH only; always starts with '/'; `doh_default_path` when absent. A DoT
|
/// DoH only; always starts with '/'; `doh_default_path` when absent. A DoT
|
||||||
/// endpoint has no request path, so it carries "/" and nothing reads it.
|
/// endpoint has no request path, so it carries "/" and nothing reads it.
|
||||||
path: []const u8,
|
path: []const u8,
|
||||||
|
|
||||||
pub const ParseError = error{ UnsupportedScheme, MissingHost, BadPort, BadUrl };
|
pub const ParseError = error{ UnsupportedScheme, MissingHost, HostTooLong, BadPort, BadUrl };
|
||||||
|
|
||||||
const doh_prefix = "https://";
|
const doh_prefix = "https://";
|
||||||
const dot_prefix = "tls://";
|
const dot_prefix = "tls://";
|
||||||
@@ -82,6 +90,10 @@ pub const Endpoint = struct {
|
|||||||
|
|
||||||
const host, const port_text = try splitAuthority(authority);
|
const host, const port_text = try splitAuthority(authority);
|
||||||
if (host.len == 0) return error.MissingHost;
|
if (host.len == 0) return error.MissingHost;
|
||||||
|
// Rejected here rather than tolerated: every identity built from this
|
||||||
|
// endpoint is bounded by `max_host_len`, so a longer host would parse
|
||||||
|
// clean and then be shortened where it is stored or logged.
|
||||||
|
if (host.len > max_host_len) return error.HostTooLong;
|
||||||
|
|
||||||
const port: u16 = if (port_text) |text| blk: {
|
const port: u16 = if (port_text) |text| blk: {
|
||||||
if (text.len == 0) return error.BadPort;
|
if (text.len == 0) return error.BadPort;
|
||||||
@@ -329,17 +341,27 @@ pub const Client = struct {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) ExchangeError![]u8,
|
) ExchangeError![]u8,
|
||||||
|
|
||||||
/// Returns a prefix of `response_buf`. The returned message has already
|
/// Returns a prefix of `response_buf`. The returned message has already
|
||||||
/// passed `validateResponse` against `query`.
|
/// passed `validateResponse` against `query`.
|
||||||
|
///
|
||||||
|
/// `selected` names the resolver the exchange used. An implementation
|
||||||
|
/// writes it *before* each attempt, never after, so a failed exchange still
|
||||||
|
/// names the last resolver it tried — a SERVFAIL row without its resolver
|
||||||
|
/// explains nothing. The slice must outlive the call; every implementation
|
||||||
|
/// borrows storage it owns for at least the query's duration. Callers
|
||||||
|
/// initialize it to null: a `null` after the call means no resolver was
|
||||||
|
/// reached at all.
|
||||||
pub fn exchange(
|
pub fn exchange(
|
||||||
self: Client,
|
self: Client,
|
||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) ExchangeError![]u8 {
|
) ExchangeError![]u8 {
|
||||||
return self.exchangeFn(self.ptr, io, query, response_buf);
|
return self.exchangeFn(self.ptr, io, query, response_buf, selected);
|
||||||
}
|
}
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -449,6 +471,31 @@ test "parse rejects an empty host" {
|
|||||||
try testing.expectError(error.MissingHost, Endpoint.parse("tls://"));
|
try testing.expectError(error.MissingHost, Endpoint.parse("tls://"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "parse takes a host at the length bound and rejects one past it" {
|
||||||
|
// Four labels, the widest a 253-byte name allows: 3 * (63 + 1) + 61.
|
||||||
|
const at_bound = ("a" ** 63 ++ ".") ** 3 ++ "a" ** 61;
|
||||||
|
comptime std.debug.assert(at_bound.len == max_host_len);
|
||||||
|
|
||||||
|
const accepted = try Endpoint.parse("https://" ++ at_bound ++ "/dns-query");
|
||||||
|
try testing.expectEqualStrings(at_bound, accepted.host);
|
||||||
|
|
||||||
|
// One byte more is one byte no consumer of `host` has room for.
|
||||||
|
try testing.expectError(
|
||||||
|
error.HostTooLong,
|
||||||
|
Endpoint.parse("https://" ++ at_bound ++ "a/dns-query"),
|
||||||
|
);
|
||||||
|
// The bound is on the host alone, so a port and a path do not spend it,
|
||||||
|
// and the bracketed form is measured with the brackets removed.
|
||||||
|
try testing.expectError(
|
||||||
|
error.HostTooLong,
|
||||||
|
Endpoint.parse("tls://" ++ at_bound ++ "a:853"),
|
||||||
|
);
|
||||||
|
try testing.expectError(
|
||||||
|
error.HostTooLong,
|
||||||
|
Endpoint.parse("https://[" ++ at_bound ++ "a]:8443/dns-query"),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
test "parse rejects a bad port" {
|
test "parse rejects a bad port" {
|
||||||
try testing.expectError(error.BadPort, Endpoint.parse("https://h:99999/"));
|
try testing.expectError(error.BadPort, Endpoint.parse("https://h:99999/"));
|
||||||
try testing.expectError(error.BadPort, Endpoint.parse("https://h:/"));
|
try testing.expectError(error.BadPort, Endpoint.parse("https://h:/"));
|
||||||
@@ -641,8 +688,10 @@ test "a fake client satisfies the Client interface" {
|
|||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) ExchangeError![]u8 {
|
) ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
|
selected.* = "fake://echo";
|
||||||
const self: *@This() = @ptrCast(@alignCast(ptr));
|
const self: *@This() = @ptrCast(@alignCast(ptr));
|
||||||
self.calls += 1;
|
self.calls += 1;
|
||||||
if (query.len > response_buf.len) return error.ResponseTooLarge;
|
if (query.len > response_buf.len) return error.ResponseTooLarge;
|
||||||
@@ -657,9 +706,11 @@ test "a fake client satisfies the Client interface" {
|
|||||||
|
|
||||||
var fake: Fake = .{};
|
var fake: Fake = .{};
|
||||||
var buf: [16]u8 = undefined;
|
var buf: [16]u8 = undefined;
|
||||||
const echoed = try fake.client().exchange(undefined, "hello", &buf);
|
var selected: ?[]const u8 = null;
|
||||||
|
const echoed = try fake.client().exchange(undefined, "hello", &buf, &selected);
|
||||||
try testing.expectEqualStrings("hello", echoed);
|
try testing.expectEqualStrings("hello", echoed);
|
||||||
try testing.expectEqual(@as(usize, 1), fake.calls);
|
try testing.expectEqual(@as(usize, 1), fake.calls);
|
||||||
|
try testing.expectEqualStrings("fake://echo", selected.?);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A query for example.com A: id 0x1234, RD set, one question.
|
/// A query for example.com A: id 0x1234, RD set, one question.
|
||||||
|
|||||||
@@ -0,0 +1,56 @@
|
|||||||
|
//! How much of the window a client asked about the query log can still answer
|
||||||
|
//! for.
|
||||||
|
//!
|
||||||
|
//! Retention deletes old rows and advances a watermark in the same transaction
|
||||||
|
//! (`queries_repo.pruneOlderThan`), so the file knows the oldest instant it is
|
||||||
|
//! complete for. Without that fact on the wire a chart draws a pruned week as a
|
||||||
|
//! week of silence, which is the one reading that is certainly wrong.
|
||||||
|
//!
|
||||||
|
//! Three endpoints carry it — `/api/queries`, `/api/stats` and
|
||||||
|
//! `/api/stats/timeseries` — and they judge it against their own effective
|
||||||
|
//! lower bound: the client's `since` for the query log, the period's aligned
|
||||||
|
//! window start for the two stats endpoints.
|
||||||
|
|
||||||
|
const std = @import("std");
|
||||||
|
|
||||||
|
const db = @import("../storage/db.zig");
|
||||||
|
const queries_repo = @import("../storage/repositories/queries_repo.zig");
|
||||||
|
|
||||||
|
pub const Coverage = struct {
|
||||||
|
/// True only when the whole requested window is inside what the file still
|
||||||
|
/// holds. A request with no lower bound at all asks about all of history,
|
||||||
|
/// which no file that has ever pruned can promise.
|
||||||
|
complete: bool,
|
||||||
|
/// The oldest instant the file is complete for, unix seconds.
|
||||||
|
available_since: i64,
|
||||||
|
};
|
||||||
|
|
||||||
|
pub fn of(available_since: i64, since: ?i64) Coverage {
|
||||||
|
return .{
|
||||||
|
.complete = if (since) |lower_bound| lower_bound >= available_since else false,
|
||||||
|
.available_since = available_since,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Reads the watermark for a request that is about to answer.
|
||||||
|
pub fn read(database: *db.Db, since: ?i64) db.Error!Coverage {
|
||||||
|
return of(try queries_repo.availableSince(database), since);
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
const testing = std.testing;
|
||||||
|
|
||||||
|
test "a window that starts at or after the watermark is complete" {
|
||||||
|
try testing.expect(of(1000, 1000).complete);
|
||||||
|
try testing.expect(of(1000, 1001).complete);
|
||||||
|
try testing.expect(!of(1000, 999).complete);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "an unbounded window is never complete" {
|
||||||
|
const unbounded = of(1000, null);
|
||||||
|
try testing.expect(!unbounded.complete);
|
||||||
|
try testing.expectEqual(@as(i64, 1000), unbounded.available_since);
|
||||||
|
}
|
||||||
+33
-37
@@ -23,7 +23,7 @@ const std = @import("std");
|
|||||||
|
|
||||||
const address = @import("../../platform/address.zig");
|
const address = @import("../../platform/address.zig");
|
||||||
const http_util = @import("../http_util.zig");
|
const http_util = @import("../http_util.zig");
|
||||||
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
|
const provenance_view = @import("../provenance_view.zig");
|
||||||
const server = @import("../server.zig");
|
const server = @import("../server.zig");
|
||||||
const sse = @import("../sse.zig");
|
const sse = @import("../sse.zig");
|
||||||
|
|
||||||
@@ -36,32 +36,13 @@ pub const heartbeat_interval: std.Io.Clock.Duration = .{
|
|||||||
.clock = .awake,
|
.clock = .awake,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// One event's `data:` payload — the `/api/queries` row fields (ruling 20),
|
/// One event's `data:` payload: the shared full-provenance DTO, exactly. A live
|
||||||
/// minus `id`: a live entry precedes persistence, so no row id exists yet.
|
/// event says everything `GET /api/queries/{id}` would say about the same query
|
||||||
pub const EventView = struct {
|
/// except its id, which does not exist yet — the entry precedes its own insert.
|
||||||
ts: i64,
|
pub const EventView = provenance_view.Provenance;
|
||||||
domain: []const u8,
|
|
||||||
client_ip: []const u8,
|
|
||||||
qtype: ?u16,
|
|
||||||
blocked: bool,
|
|
||||||
block_reason: []const u8,
|
|
||||||
response_time_us: ?i64,
|
|
||||||
cache_hit: ?bool,
|
|
||||||
upstream: []const u8,
|
|
||||||
};
|
|
||||||
|
|
||||||
pub fn view(entry: *const sse.Entry) EventView {
|
pub fn view(entry: *const sse.Entry) EventView {
|
||||||
return .{
|
return provenance_view.fromEntry(entry);
|
||||||
.ts = entry.timestamp,
|
|
||||||
.domain = entry.domain(),
|
|
||||||
.client_ip = entry.clientIp(),
|
|
||||||
.qtype = entry.qtype,
|
|
||||||
.blocked = entry.blocked,
|
|
||||||
.block_reason = entry.blockReason(),
|
|
||||||
.response_time_us = entry.response_time_us,
|
|
||||||
.cache_hit = entry.cache_hit,
|
|
||||||
.upstream = entry.upstream(),
|
|
||||||
};
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// One `event: query` frame. JSON never contains a raw newline, so the whole
|
/// One `event: query` frame. JSON never contains a raw newline, so the whole
|
||||||
@@ -139,14 +120,18 @@ pub fn stream(
|
|||||||
|
|
||||||
const testing = std.testing;
|
const testing = std.testing;
|
||||||
|
|
||||||
test "the event payload carries the /api/queries row fields, minus id" {
|
test "the event payload is the detail body minus its id, name and type for name" {
|
||||||
const row_fields = @typeInfo(queries_repo.QueryRow).@"struct".fields;
|
const detail_fields = @typeInfo(provenance_view.QueryDetail).@"struct".fields;
|
||||||
const view_fields = @typeInfo(EventView).@"struct".fields;
|
const view_fields = @typeInfo(EventView).@"struct".fields;
|
||||||
comptime {
|
comptime {
|
||||||
std.debug.assert(view_fields.len == row_fields.len - 1);
|
std.debug.assert(view_fields.len == detail_fields.len - 1);
|
||||||
std.debug.assert(std.mem.eql(u8, row_fields[0].name, "id"));
|
std.debug.assert(std.mem.eql(u8, detail_fields[0].name, "id"));
|
||||||
for (row_fields[1..], view_fields) |row_field, view_field| {
|
for (detail_fields[1..], view_fields) |detail_field, view_field| {
|
||||||
std.debug.assert(std.mem.eql(u8, row_field.name, view_field.name));
|
std.debug.assert(std.mem.eql(u8, detail_field.name, view_field.name));
|
||||||
|
// Names alone would let a group keep its key while changing what it
|
||||||
|
// holds, which is the drift a live viewer would see and a detail
|
||||||
|
// page would not.
|
||||||
|
std.debug.assert(detail_field.type == view_field.type);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -157,11 +142,19 @@ test "a frame is one event line and one data line of JSON" {
|
|||||||
.domain = "ads.example",
|
.domain = "ads.example",
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = true,
|
.blocked = true,
|
||||||
.block_reason = "blocklist_domain",
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = .block,
|
||||||
|
.policy_reason = .blocklist_domain,
|
||||||
|
.matched = "ads.example",
|
||||||
|
.source_id = 3,
|
||||||
|
.source_name = "StevenBlack",
|
||||||
|
.route_kind = .blocked,
|
||||||
.response_time_us = 42,
|
.response_time_us = 42,
|
||||||
.cache_hit = false,
|
.cache_hit = false,
|
||||||
.upstream = "https://dns.example/dns-query",
|
|
||||||
});
|
});
|
||||||
|
|
||||||
var buf: [1024]u8 = undefined;
|
var buf: [1024]u8 = undefined;
|
||||||
@@ -172,10 +165,12 @@ test "a frame is one event line and one data line of JSON" {
|
|||||||
try testing.expect(std.mem.startsWith(u8, frame, "event: query\ndata: {"));
|
try testing.expect(std.mem.startsWith(u8, frame, "event: query\ndata: {"));
|
||||||
try testing.expect(std.mem.endsWith(u8, frame, "}\n\n"));
|
try testing.expect(std.mem.endsWith(u8, frame, "}\n\n"));
|
||||||
try testing.expectEqual(@as(usize, 3), std.mem.count(u8, frame, "\n"));
|
try testing.expectEqual(@as(usize, 3), std.mem.count(u8, frame, "\n"));
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"ts\":1700000000"));
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"time\":1700000000"));
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"domain\":\"ads.example\""));
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"domain\":\"ads.example\""));
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"blocked\":true"));
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"group\":{\"id\":1,\"name\":\"default\"}"));
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"block_reason\":\"blocklist_domain\""));
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"reason\":\"blocklist_domain\""));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"source_name\":\"StevenBlack\""));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"kind\":\"blocked\""));
|
||||||
}
|
}
|
||||||
|
|
||||||
test "an unlogged field stays null and an empty string stays a string" {
|
test "an unlogged field stays null and an empty string stays a string" {
|
||||||
@@ -191,6 +186,7 @@ test "an unlogged field stays null and an empty string stays a string" {
|
|||||||
const frame = writer.buffered();
|
const frame = writer.buffered();
|
||||||
|
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"qtype\":null"));
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"qtype\":null"));
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"cache_hit\":null"));
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"duration_us\":null"));
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"upstream\":\"\""));
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"upstream\":\"\""));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"id\":null"));
|
||||||
}
|
}
|
||||||
|
|||||||
+148
-10
@@ -1,4 +1,5 @@
|
|||||||
//! `GET /api/queries` — the query log, newest first (ruling 11).
|
//! `GET /api/queries` — the query log, newest first (ruling 11) — and
|
||||||
|
//! `GET /api/queries/{id}`, one row of it fully explained.
|
||||||
//!
|
//!
|
||||||
//! Keyset pagination rather than an offset: the table is append-only and the
|
//! Keyset pagination rather than an offset: the table is append-only and the
|
||||||
//! UI reads the head of it, so `id < before` is one index seek no matter how
|
//! UI reads the head of it, so `id < before` is one index seek no matter how
|
||||||
@@ -13,9 +14,11 @@
|
|||||||
const std = @import("std");
|
const std = @import("std");
|
||||||
const Allocator = std.mem.Allocator;
|
const Allocator = std.mem.Allocator;
|
||||||
|
|
||||||
|
const coverage = @import("../coverage.zig");
|
||||||
const db = @import("../../storage/db.zig");
|
const db = @import("../../storage/db.zig");
|
||||||
const http_util = @import("../http_util.zig");
|
const http_util = @import("../http_util.zig");
|
||||||
const logger = @import("../../storage/logger.zig");
|
const logger = @import("../../storage/logger.zig");
|
||||||
|
const provenance_view = @import("../provenance_view.zig");
|
||||||
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
|
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
|
||||||
const server = @import("../server.zig");
|
const server = @import("../server.zig");
|
||||||
|
|
||||||
@@ -41,6 +44,10 @@ pub const Page = struct {
|
|||||||
queries: []const queries_repo.QueryRow,
|
queries: []const queries_repo.QueryRow,
|
||||||
/// The cursor for the next page, or null when this page is the last one.
|
/// The cursor for the next page, or null when this page is the last one.
|
||||||
next_before: ?i64,
|
next_before: ?i64,
|
||||||
|
/// Whether the log still covers the window the filter asked for. A client
|
||||||
|
/// that reads rows without reading this cannot tell an empty window from a
|
||||||
|
/// pruned one.
|
||||||
|
coverage: coverage.Coverage,
|
||||||
};
|
};
|
||||||
|
|
||||||
pub const FilterError = error{
|
pub const FilterError = error{
|
||||||
@@ -110,6 +117,7 @@ pub fn page(
|
|||||||
return .{
|
return .{
|
||||||
.queries = rows.items,
|
.queries = rows.items,
|
||||||
.next_before = if (full and rows.items.len != 0) rows.items[rows.items.len - 1].id else null,
|
.next_before = if (full and rows.items.len != 0) rows.items[rows.items.len - 1].id else null,
|
||||||
|
.coverage = try coverage.read(database, filter.since),
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -138,10 +146,37 @@ pub fn list(
|
|||||||
return http_util.respondJson(request, .ok, result, &.{});
|
return http_util.respondJson(request, .ok, result, &.{});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// `GET /api/queries/{id}` — one query, fully explained.
|
||||||
|
///
|
||||||
|
/// The row is the whole answer: every field is a fact recorded when the query
|
||||||
|
/// was answered, so nothing here is joined against current configuration. A
|
||||||
|
/// group or blocklist renamed since keeps the name it had.
|
||||||
|
pub fn detail(
|
||||||
|
state: *server.WebState,
|
||||||
|
io: std.Io,
|
||||||
|
request: *http_util.Request,
|
||||||
|
) http_util.HandlerError!void {
|
||||||
|
_ = io;
|
||||||
|
|
||||||
|
const database = state.querylog_db orelse
|
||||||
|
return http_util.respondError(request, .service_unavailable, "query log unavailable");
|
||||||
|
|
||||||
|
const row = queries_repo.detailById(database, request.arena, request.id.?) catch |err| {
|
||||||
|
log.warn("query log read failed: {s}", .{@errorName(err)});
|
||||||
|
return http_util.respondError(request, .internal_server_error, "internal error");
|
||||||
|
};
|
||||||
|
|
||||||
|
// An id retention has pruned and one that never existed are the same
|
||||||
|
// answer, and the API does not pretend to tell them apart.
|
||||||
|
const found = row orelse return http_util.respondError(request, .not_found, "not found");
|
||||||
|
return http_util.respondJson(request, .ok, provenance_view.fromDetail(found), &.{});
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// tests
|
// tests
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
const provenance = @import("../../storage/provenance.zig");
|
||||||
const querylog_schema = @import("../../storage/querylog_schema.zig");
|
const querylog_schema = @import("../../storage/querylog_schema.zig");
|
||||||
const testing = std.testing;
|
const testing = std.testing;
|
||||||
|
|
||||||
@@ -210,21 +245,40 @@ fn openLog() !db.Db {
|
|||||||
return database;
|
return database;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// Rows the resolver could actually have written. Ruling 20 ties the three
|
||||||
|
/// route facts together: a block never consulted the cache, so its `cache_hit`
|
||||||
|
/// is NULL rather than false; a cache hit has no upstream to name; and only an
|
||||||
|
/// upstream answer carries one. A fixture that broke those ties would let a
|
||||||
|
/// serializer regression pass here and fail on real rows.
|
||||||
fn seed(database: *db.Db, count: usize) !void {
|
fn seed(database: *db.Db, count: usize) !void {
|
||||||
var writer = try queries_repo.BatchWriter.init(database);
|
var writer = try queries_repo.BatchWriter.init(database);
|
||||||
defer writer.deinit();
|
defer writer.deinit();
|
||||||
var rows: [16]queries_repo.Row = undefined;
|
var rows: [16]queries_repo.Row = undefined;
|
||||||
for (rows[0..count], 0..) |*row, i| {
|
for (rows[0..count], 0..) |*row, i| {
|
||||||
|
const blocked = i % 2 == 0;
|
||||||
|
const from_cache = i % 4 == 1;
|
||||||
row.* = .{
|
row.* = .{
|
||||||
.timestamp = 1_700_000_000 + @as(i64, @intCast(i)),
|
.timestamp = 1_700_000_000 + @as(i64, @intCast(i)),
|
||||||
.domain = if (i % 2 == 0) "ads.example" else "safe.example",
|
.domain = if (blocked) "ads.example" else "safe.example",
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
.blocked = i % 2 == 0,
|
.qclass = 1,
|
||||||
.block_reason = if (i % 2 == 0) "blocklist_domain" else null,
|
.rcode = 0,
|
||||||
|
.blocked = blocked,
|
||||||
.response_time_us = 500,
|
.response_time_us = 500,
|
||||||
.cache_hit = false,
|
.cache_hit = if (blocked) null else from_cache,
|
||||||
.upstream = null,
|
.upstream = if (blocked or from_cache) null else "9.9.9.9",
|
||||||
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = if (blocked) .block else .allow,
|
||||||
|
.policy_reason = if (blocked) .blocklist_domain else .no_match,
|
||||||
|
.matched = if (blocked) "ads.example" else null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = if (blocked) .blocked else if (from_cache) .cache else .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
try writer.writeBatch(rows[0..count]);
|
try writer.writeBatch(rows[0..count]);
|
||||||
@@ -297,6 +351,37 @@ test "the parsed filters narrow the rows the page returns" {
|
|||||||
try testing.expectEqual(@as(usize, 0), nobody.queries.len);
|
try testing.expectEqual(@as(usize, 0), nobody.queries.len);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
test "the seeded rows carry only the route shapes ruling 20 allows" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
try seed(&database, 4);
|
||||||
|
|
||||||
|
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
|
defer arena.deinit();
|
||||||
|
|
||||||
|
var seen: std.EnumSet(provenance.RouteKind) = .initEmpty();
|
||||||
|
for ((try page(&database, arena.allocator(), .{})).queries) |row| {
|
||||||
|
seen.insert(row.route_kind);
|
||||||
|
switch (row.route_kind) {
|
||||||
|
.blocked => {
|
||||||
|
try testing.expectEqual(@as(?bool, null), row.cache_hit);
|
||||||
|
try testing.expectEqualStrings("", row.upstream);
|
||||||
|
},
|
||||||
|
.cache => {
|
||||||
|
try testing.expectEqual(@as(?bool, true), row.cache_hit);
|
||||||
|
try testing.expectEqualStrings("", row.upstream);
|
||||||
|
},
|
||||||
|
.upstream => {
|
||||||
|
try testing.expectEqual(@as(?bool, false), row.cache_hit);
|
||||||
|
try testing.expectEqualStrings("9.9.9.9", row.upstream);
|
||||||
|
},
|
||||||
|
.local, .forward_zone, .rejected => return error.UnseededRouteKind,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
// All three, so the serializer tests below read every shape the fixture claims.
|
||||||
|
try testing.expectEqual(@as(usize, 3), seen.count());
|
||||||
|
}
|
||||||
|
|
||||||
test "the page serializes as the envelope ruling 11 defines" {
|
test "the page serializes as the envelope ruling 11 defines" {
|
||||||
var database = try openLog();
|
var database = try openLog();
|
||||||
defer database.close();
|
defer database.close();
|
||||||
@@ -312,14 +397,67 @@ test "the page serializes as the envelope ruling 11 defines" {
|
|||||||
const text = allocating.written();
|
const text = allocating.written();
|
||||||
|
|
||||||
try testing.expect(std.mem.startsWith(u8, text, "{\"queries\":["));
|
try testing.expect(std.mem.startsWith(u8, text, "{\"queries\":["));
|
||||||
try testing.expect(std.mem.endsWith(u8, text, "\"next_before\":null}"));
|
|
||||||
for ([_][]const u8{
|
for ([_][]const u8{
|
||||||
"\"id\":", "\"ts\":", "\"domain\":", "\"client_ip\":",
|
"\"id\":", "\"ts\":", "\"domain\":", "\"client_ip\":",
|
||||||
"\"qtype\":", "\"blocked\":", "\"cache_hit\":", "\"upstream\":",
|
"\"qtype\":", "\"qclass\":", "\"rcode\":", "\"blocked\":",
|
||||||
"\"upstream\":", "\"response_time_us\":", "\"block_reason\":",
|
"\"cache_hit\":", "\"upstream\":", "\"response_time_us\":", "\"policy_action\":",
|
||||||
|
"\"policy_reason\":", "\"route_kind\":",
|
||||||
}) |field| {
|
}) |field| {
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, text, 1, field));
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, field));
|
||||||
}
|
}
|
||||||
// W1's ruling: a NULL column reads as "", and "" stays "" on the wire.
|
// W1's ruling: a NULL column reads as "", and "" stays "" on the wire.
|
||||||
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"upstream\":\"\""));
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"upstream\":\"\""));
|
||||||
|
// The column the provenance columns replaced.
|
||||||
|
try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "block_reason"));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"coverage\":{\"complete\":"));
|
||||||
|
}
|
||||||
|
|
||||||
|
test "the coverage of a page answers the window the filter asked for" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
try seed(&database, 1);
|
||||||
|
|
||||||
|
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
|
defer arena.deinit();
|
||||||
|
|
||||||
|
const watermark = try queries_repo.availableSince(&database);
|
||||||
|
|
||||||
|
const unbounded = try page(&database, arena.allocator(), .{});
|
||||||
|
try testing.expectEqual(watermark, unbounded.coverage.available_since);
|
||||||
|
try testing.expect(!unbounded.coverage.complete);
|
||||||
|
|
||||||
|
const covered = try page(&database, arena.allocator(), .{ .since = watermark });
|
||||||
|
try testing.expect(covered.coverage.complete);
|
||||||
|
|
||||||
|
const older = try page(&database, arena.allocator(), .{ .since = watermark - 1 });
|
||||||
|
try testing.expect(!older.coverage.complete);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "a detail row carries every provenance field the row stored" {
|
||||||
|
var database = try openLog();
|
||||||
|
defer database.close();
|
||||||
|
try seed(&database, 1);
|
||||||
|
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
const arena = arena_state.allocator();
|
||||||
|
|
||||||
|
const row = (try queries_repo.detailById(&database, arena, 1)).?;
|
||||||
|
const view = provenance_view.fromDetail(row);
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, 1), view.id);
|
||||||
|
try testing.expectEqualStrings("ads.example", view.request.domain);
|
||||||
|
try testing.expectEqualStrings("192.0.2.10", view.request.client);
|
||||||
|
try testing.expectEqual(@as(u16, 1), view.request.qclass);
|
||||||
|
try testing.expectEqualStrings("default", view.group.name);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.block, view.policy.action);
|
||||||
|
try testing.expectEqualStrings("ads.example", view.policy.matched);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.blocked, view.route.kind);
|
||||||
|
try testing.expectEqualStrings("", view.route.upstream);
|
||||||
|
try testing.expectEqual(@as(?i64, 500), view.response.duration_us);
|
||||||
|
|
||||||
|
try testing.expectEqual(
|
||||||
|
@as(?queries_repo.QueryDetail, null),
|
||||||
|
try queries_repo.detailById(&database, arena, 99),
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -14,6 +14,7 @@
|
|||||||
|
|
||||||
const std = @import("std");
|
const std = @import("std");
|
||||||
|
|
||||||
|
const coverage = @import("../coverage.zig");
|
||||||
const db = @import("../../storage/db.zig");
|
const db = @import("../../storage/db.zig");
|
||||||
const http_util = @import("../http_util.zig");
|
const http_util = @import("../http_util.zig");
|
||||||
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
|
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
|
||||||
@@ -99,6 +100,10 @@ pub const TotalsBody = struct {
|
|||||||
cached: u64,
|
cached: u64,
|
||||||
clients: u64,
|
clients: u64,
|
||||||
avg_response_time_us: ?i64,
|
avg_response_time_us: ?i64,
|
||||||
|
/// Judged against `since`, which is the window this body reports on — so a
|
||||||
|
/// dashboard can say "history starts here" instead of charting a pruned
|
||||||
|
/// stretch as a quiet one.
|
||||||
|
coverage: coverage.Coverage,
|
||||||
};
|
};
|
||||||
|
|
||||||
pub const TimeseriesBody = struct {
|
pub const TimeseriesBody = struct {
|
||||||
@@ -107,6 +112,7 @@ pub const TimeseriesBody = struct {
|
|||||||
until: i64,
|
until: i64,
|
||||||
bucket_seconds: u32,
|
bucket_seconds: u32,
|
||||||
buckets: []const queries_repo.Bucket,
|
buckets: []const queries_repo.Bucket,
|
||||||
|
coverage: coverage.Coverage,
|
||||||
};
|
};
|
||||||
|
|
||||||
pub fn totals(
|
pub fn totals(
|
||||||
@@ -121,6 +127,9 @@ pub fn totals(
|
|||||||
const result = queries_repo.statsTotals(database, span.since, span.until) catch |err| {
|
const result = queries_repo.statsTotals(database, span.since, span.until) catch |err| {
|
||||||
return internal(request, "stats totals", err);
|
return internal(request, "stats totals", err);
|
||||||
};
|
};
|
||||||
|
const covered = coverage.read(database, span.since) catch |err| {
|
||||||
|
return internal(request, "stats coverage", err);
|
||||||
|
};
|
||||||
|
|
||||||
return http_util.respondJson(request, .ok, TotalsBody{
|
return http_util.respondJson(request, .ok, TotalsBody{
|
||||||
.period = period.label(),
|
.period = period.label(),
|
||||||
@@ -131,6 +140,7 @@ pub fn totals(
|
|||||||
.cached = result.cached,
|
.cached = result.cached,
|
||||||
.clients = result.distinct_clients,
|
.clients = result.distinct_clients,
|
||||||
.avg_response_time_us = result.avg_response_time_us,
|
.avg_response_time_us = result.avg_response_time_us,
|
||||||
|
.coverage = covered,
|
||||||
}, &.{});
|
}, &.{});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -148,6 +158,9 @@ pub fn timeseries(
|
|||||||
const written = queries_repo.timeseries(database, span.since, span.bucket_seconds, out) catch |err| {
|
const written = queries_repo.timeseries(database, span.since, span.bucket_seconds, out) catch |err| {
|
||||||
return internal(request, "stats timeseries", err);
|
return internal(request, "stats timeseries", err);
|
||||||
};
|
};
|
||||||
|
const covered = coverage.read(database, span.since) catch |err| {
|
||||||
|
return internal(request, "stats coverage", err);
|
||||||
|
};
|
||||||
|
|
||||||
return http_util.respondJson(request, .ok, TimeseriesBody{
|
return http_util.respondJson(request, .ok, TimeseriesBody{
|
||||||
.period = period.label(),
|
.period = period.label(),
|
||||||
@@ -155,6 +168,7 @@ pub fn timeseries(
|
|||||||
.until = span.until,
|
.until = span.until,
|
||||||
.bucket_seconds = span.bucket_seconds,
|
.bucket_seconds = span.bucket_seconds,
|
||||||
.buckets = out[0..written],
|
.buckets = out[0..written],
|
||||||
|
.coverage = covered,
|
||||||
}, &.{});
|
}, &.{});
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -268,11 +282,23 @@ fn writeRow(writer: *queries_repo.BatchWriter, timestamp: i64, blocked: bool, ca
|
|||||||
.domain = "example.com",
|
.domain = "example.com",
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = blocked,
|
.blocked = blocked,
|
||||||
.block_reason = if (blocked) "blocklist_domain" else null,
|
|
||||||
.response_time_us = 1000,
|
.response_time_us = 1000,
|
||||||
.cache_hit = cached,
|
.cache_hit = cached,
|
||||||
.upstream = null,
|
.upstream = null,
|
||||||
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = if (blocked) .block else .allow,
|
||||||
|
.policy_reason = if (blocked) .blocklist_domain else .no_match,
|
||||||
|
.matched = null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = if (blocked) .blocked else .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
}};
|
}};
|
||||||
try writer.writeBatch(&rows);
|
try writer.writeBatch(&rows);
|
||||||
}
|
}
|
||||||
|
|||||||
+218
-10
@@ -226,14 +226,47 @@ paths:
|
|||||||
"503":
|
"503":
|
||||||
$ref: "#/components/responses/Unavailable"
|
$ref: "#/components/responses/Unavailable"
|
||||||
|
|
||||||
|
/api/queries/{id}:
|
||||||
|
get:
|
||||||
|
summary: One query, fully explained
|
||||||
|
description: |
|
||||||
|
The full provenance of one logged query: what was asked, which group's
|
||||||
|
policy applied, what that policy decided and matched on, what was
|
||||||
|
rewritten, where the answer came from, and what the client received.
|
||||||
|
Every field is a fact recorded when the query was answered, so a group
|
||||||
|
or blocklist renamed since keeps the name it had.
|
||||||
|
parameters:
|
||||||
|
- name: id
|
||||||
|
in: path
|
||||||
|
required: true
|
||||||
|
schema: { type: integer, minimum: 1 }
|
||||||
|
responses:
|
||||||
|
"200":
|
||||||
|
description: The query.
|
||||||
|
content:
|
||||||
|
application/json:
|
||||||
|
schema:
|
||||||
|
$ref: "#/components/schemas/QueryDetail"
|
||||||
|
"401":
|
||||||
|
$ref: "#/components/responses/Unauthorized"
|
||||||
|
"404":
|
||||||
|
$ref: "#/components/responses/NotFound"
|
||||||
|
"429":
|
||||||
|
$ref: "#/components/responses/RateLimited"
|
||||||
|
"500":
|
||||||
|
$ref: "#/components/responses/Internal"
|
||||||
|
"503":
|
||||||
|
$ref: "#/components/responses/Unavailable"
|
||||||
|
|
||||||
/api/queries/live:
|
/api/queries/live:
|
||||||
get:
|
get:
|
||||||
summary: Live query stream (server-sent events)
|
summary: Live query stream (server-sent events)
|
||||||
description: |
|
description: |
|
||||||
`text/event-stream`. The stream opens with `retry: 3000`, then sends
|
`text/event-stream`. The stream opens with `retry: 3000`, then sends
|
||||||
one `event: query` frame per resolved query whose `data:` line is a
|
one `event: query` frame per resolved query whose `data:` line is a
|
||||||
JSON object with the `/api/queries` row fields minus `id` (the entry
|
`Provenance` object — the body of `/api/queries/{id}` without its `id`,
|
||||||
precedes persistence). A `: ping` comment goes out every 15 seconds.
|
which does not exist yet because the entry precedes its own insert.
|
||||||
|
A `: ping` comment goes out every 15 seconds.
|
||||||
A client that falls more than 64 events behind is disconnected and
|
A client that falls more than 64 events behind is disconnected and
|
||||||
should re-sync via `/api/queries` after reconnecting. Connections
|
should re-sync via `/api/queries` after reconnecting. Connections
|
||||||
per address are capped by `web.sse_max_connections_per_ip`; the
|
per address are capped by `web.sse_max_connections_per_ip`; the
|
||||||
@@ -1855,9 +1888,33 @@ components:
|
|||||||
type: boolean
|
type: boolean
|
||||||
description: False when no password is configured; no cookie is set.
|
description: False when no password is configured; no cookie is set.
|
||||||
|
|
||||||
|
PolicyAction:
|
||||||
|
type: string
|
||||||
|
description: |
|
||||||
|
Whether the filtering policy reached a verdict. `not_evaluated` is the
|
||||||
|
honest answer for a query answered before filtering could apply, and is
|
||||||
|
not the same as `allow`.
|
||||||
|
enum: [not_evaluated, allow, block]
|
||||||
|
|
||||||
|
PolicyReason:
|
||||||
|
type: string
|
||||||
|
description: |
|
||||||
|
Why the policy landed where it did. The first nine are the matcher's own
|
||||||
|
verdicts; the rest name a pipeline step that decided without consulting
|
||||||
|
the matcher.
|
||||||
|
enum: [rule_allow_exact, rule_block_exact, rule_allow_wildcard, rule_block_wildcard, rule_allow_regex, rule_block_regex, blocklist_exception, blocklist_domain, blocklist_wildcard, local_record, forward_zone, non_in_class, paused, snapshot_unavailable, no_match, protocol_error]
|
||||||
|
|
||||||
|
RouteKind:
|
||||||
|
type: string
|
||||||
|
description: Where the answer the client received came from.
|
||||||
|
enum: [blocked, local, forward_zone, upstream, cache, rejected]
|
||||||
|
|
||||||
QueryRow:
|
QueryRow:
|
||||||
type: object
|
type: object
|
||||||
required: [id, ts, domain, client_ip, qtype, blocked, block_reason, response_time_us, cache_hit, upstream]
|
description: |
|
||||||
|
The summary projection the query-log table scans. The full provenance of
|
||||||
|
a row is one request away at `/api/queries/{id}`.
|
||||||
|
required: [id, ts, domain, client_ip, qtype, qclass, rcode, blocked, response_time_us, cache_hit, upstream, policy_action, policy_reason, route_kind]
|
||||||
properties:
|
properties:
|
||||||
id: { type: integer }
|
id: { type: integer }
|
||||||
ts:
|
ts:
|
||||||
@@ -1868,10 +1925,11 @@ components:
|
|||||||
qtype:
|
qtype:
|
||||||
type: integer
|
type: integer
|
||||||
nullable: true
|
nullable: true
|
||||||
|
qclass: { type: integer }
|
||||||
|
rcode:
|
||||||
|
type: integer
|
||||||
|
description: The twelve-bit EDNS extended code, not the four header bits alone.
|
||||||
blocked: { type: boolean }
|
blocked: { type: boolean }
|
||||||
block_reason:
|
|
||||||
type: string
|
|
||||||
description: Empty when the query was not blocked.
|
|
||||||
response_time_us:
|
response_time_us:
|
||||||
type: integer
|
type: integer
|
||||||
nullable: true
|
nullable: true
|
||||||
@@ -1880,11 +1938,155 @@ components:
|
|||||||
nullable: true
|
nullable: true
|
||||||
upstream:
|
upstream:
|
||||||
type: string
|
type: string
|
||||||
description: Empty for cache hits and local answers.
|
description: Empty for cache hits, local answers and blocked queries.
|
||||||
|
policy_action:
|
||||||
|
$ref: "#/components/schemas/PolicyAction"
|
||||||
|
policy_reason:
|
||||||
|
$ref: "#/components/schemas/PolicyReason"
|
||||||
|
route_kind:
|
||||||
|
$ref: "#/components/schemas/RouteKind"
|
||||||
|
|
||||||
|
ProvenanceRequest:
|
||||||
|
type: object
|
||||||
|
required: [time, domain, client, qtype, qclass]
|
||||||
|
properties:
|
||||||
|
time:
|
||||||
|
type: integer
|
||||||
|
description: Unix seconds.
|
||||||
|
domain: { type: string }
|
||||||
|
client: { type: string }
|
||||||
|
qtype:
|
||||||
|
type: integer
|
||||||
|
nullable: true
|
||||||
|
qclass: { type: integer }
|
||||||
|
|
||||||
|
ProvenanceGroup:
|
||||||
|
type: object
|
||||||
|
description: |
|
||||||
|
The client's filtering group at the time of the query, as a historical
|
||||||
|
fact: the id may name a group since renamed or deleted.
|
||||||
|
required: [id, name]
|
||||||
|
properties:
|
||||||
|
id:
|
||||||
|
type: integer
|
||||||
|
nullable: true
|
||||||
|
name: { type: string }
|
||||||
|
|
||||||
|
ProvenancePolicy:
|
||||||
|
type: object
|
||||||
|
required: [action, reason, matched, source_id, source_name]
|
||||||
|
properties:
|
||||||
|
action:
|
||||||
|
$ref: "#/components/schemas/PolicyAction"
|
||||||
|
reason:
|
||||||
|
$ref: "#/components/schemas/PolicyReason"
|
||||||
|
matched:
|
||||||
|
type: string
|
||||||
|
description: The rule pattern or list entry that decided; empty when nothing matched.
|
||||||
|
source_id:
|
||||||
|
type: integer
|
||||||
|
nullable: true
|
||||||
|
source_name:
|
||||||
|
type: string
|
||||||
|
description: The blocklist the match came from; empty for a rule.
|
||||||
|
|
||||||
|
ProvenanceRewrites:
|
||||||
|
type: object
|
||||||
|
required: [cname_target, safe_search_target]
|
||||||
|
properties:
|
||||||
|
cname_target:
|
||||||
|
type: string
|
||||||
|
description: Set when the decision was made about a CNAME target rather than the queried name.
|
||||||
|
safe_search_target: { type: string }
|
||||||
|
|
||||||
|
ProvenanceRoute:
|
||||||
|
type: object
|
||||||
|
required: [kind, forward_zone, upstream]
|
||||||
|
properties:
|
||||||
|
kind:
|
||||||
|
$ref: "#/components/schemas/RouteKind"
|
||||||
|
forward_zone: { type: string }
|
||||||
|
upstream:
|
||||||
|
type: string
|
||||||
|
description: |
|
||||||
|
Non-empty only for an attempted upstream or forward-zone exchange,
|
||||||
|
including one that failed. Already redacted: the userinfo, path,
|
||||||
|
query and fragment of a resolver url never reach here.
|
||||||
|
|
||||||
|
ProvenanceResponse:
|
||||||
|
type: object
|
||||||
|
required: [rcode, duration_us]
|
||||||
|
properties:
|
||||||
|
rcode:
|
||||||
|
type: integer
|
||||||
|
description: The twelve-bit EDNS extended code, not the four header bits alone.
|
||||||
|
duration_us:
|
||||||
|
type: integer
|
||||||
|
nullable: true
|
||||||
|
|
||||||
|
Provenance:
|
||||||
|
type: object
|
||||||
|
description: |
|
||||||
|
One query, fully explained, in the order a query meets the pipeline. The
|
||||||
|
`data:` payload of a live-stream `event: query` frame is exactly this.
|
||||||
|
required: [request, group, policy, rewrites, route, response]
|
||||||
|
properties:
|
||||||
|
request:
|
||||||
|
$ref: "#/components/schemas/ProvenanceRequest"
|
||||||
|
group:
|
||||||
|
$ref: "#/components/schemas/ProvenanceGroup"
|
||||||
|
policy:
|
||||||
|
$ref: "#/components/schemas/ProvenancePolicy"
|
||||||
|
rewrites:
|
||||||
|
$ref: "#/components/schemas/ProvenanceRewrites"
|
||||||
|
route:
|
||||||
|
$ref: "#/components/schemas/ProvenanceRoute"
|
||||||
|
response:
|
||||||
|
$ref: "#/components/schemas/ProvenanceResponse"
|
||||||
|
|
||||||
|
QueryDetail:
|
||||||
|
type: object
|
||||||
|
description: |
|
||||||
|
`Provenance` plus the row id. Written out rather than composed with
|
||||||
|
`allOf` so the drift guard reads one property list per schema.
|
||||||
|
required: [id, request, group, policy, rewrites, route, response]
|
||||||
|
properties:
|
||||||
|
id: { type: integer }
|
||||||
|
request:
|
||||||
|
$ref: "#/components/schemas/ProvenanceRequest"
|
||||||
|
group:
|
||||||
|
$ref: "#/components/schemas/ProvenanceGroup"
|
||||||
|
policy:
|
||||||
|
$ref: "#/components/schemas/ProvenancePolicy"
|
||||||
|
rewrites:
|
||||||
|
$ref: "#/components/schemas/ProvenanceRewrites"
|
||||||
|
route:
|
||||||
|
$ref: "#/components/schemas/ProvenanceRoute"
|
||||||
|
response:
|
||||||
|
$ref: "#/components/schemas/ProvenanceResponse"
|
||||||
|
|
||||||
|
Coverage:
|
||||||
|
type: object
|
||||||
|
description: |
|
||||||
|
How much of the requested window the query log can still answer for.
|
||||||
|
Retention deletes rows and advances the watermark in one transaction, so
|
||||||
|
a client can tell an empty window from a pruned one instead of charting
|
||||||
|
the gap as zero.
|
||||||
|
required: [complete, available_since]
|
||||||
|
properties:
|
||||||
|
complete:
|
||||||
|
type: boolean
|
||||||
|
description: |
|
||||||
|
True only when the window's lower bound is at or after
|
||||||
|
`available_since`. A request with no lower bound asks about all of
|
||||||
|
history, which no file that has ever pruned can promise.
|
||||||
|
available_since:
|
||||||
|
type: integer
|
||||||
|
description: The oldest instant the file is complete for, unix seconds.
|
||||||
|
|
||||||
QueriesPage:
|
QueriesPage:
|
||||||
type: object
|
type: object
|
||||||
required: [queries, next_before]
|
required: [queries, next_before, coverage]
|
||||||
properties:
|
properties:
|
||||||
queries:
|
queries:
|
||||||
type: array
|
type: array
|
||||||
@@ -1894,6 +2096,8 @@ components:
|
|||||||
type: integer
|
type: integer
|
||||||
nullable: true
|
nullable: true
|
||||||
description: Cursor for the next page; null on the last page.
|
description: Cursor for the next page; null on the last page.
|
||||||
|
coverage:
|
||||||
|
$ref: "#/components/schemas/Coverage"
|
||||||
|
|
||||||
DiagnosticEvent:
|
DiagnosticEvent:
|
||||||
type: object
|
type: object
|
||||||
@@ -1983,7 +2187,7 @@ components:
|
|||||||
|
|
||||||
StatsTotals:
|
StatsTotals:
|
||||||
type: object
|
type: object
|
||||||
required: [period, since, until, queries, blocked, cached, clients, avg_response_time_us]
|
required: [period, since, until, queries, blocked, cached, clients, avg_response_time_us, coverage]
|
||||||
properties:
|
properties:
|
||||||
period:
|
period:
|
||||||
type: string
|
type: string
|
||||||
@@ -2004,6 +2208,8 @@ components:
|
|||||||
type: integer
|
type: integer
|
||||||
nullable: true
|
nullable: true
|
||||||
description: Null when no query in the window recorded a time.
|
description: Null when no query in the window recorded a time.
|
||||||
|
coverage:
|
||||||
|
$ref: "#/components/schemas/Coverage"
|
||||||
|
|
||||||
Bucket:
|
Bucket:
|
||||||
type: object
|
type: object
|
||||||
@@ -2018,7 +2224,7 @@ components:
|
|||||||
|
|
||||||
StatsTimeseries:
|
StatsTimeseries:
|
||||||
type: object
|
type: object
|
||||||
required: [period, since, until, bucket_seconds, buckets]
|
required: [period, since, until, bucket_seconds, buckets, coverage]
|
||||||
properties:
|
properties:
|
||||||
period:
|
period:
|
||||||
type: string
|
type: string
|
||||||
@@ -2030,6 +2236,8 @@ components:
|
|||||||
type: array
|
type: array
|
||||||
items:
|
items:
|
||||||
$ref: "#/components/schemas/Bucket"
|
$ref: "#/components/schemas/Bucket"
|
||||||
|
coverage:
|
||||||
|
$ref: "#/components/schemas/Coverage"
|
||||||
|
|
||||||
Lookup:
|
Lookup:
|
||||||
type: object
|
type: object
|
||||||
|
|||||||
@@ -0,0 +1,278 @@
|
|||||||
|
//! The wire shape of one query's provenance, defined once.
|
||||||
|
//!
|
||||||
|
//! Two surfaces answer with it: `GET /api/queries/{id}`, which reads a stored
|
||||||
|
//! row, and the `event: query` frames of `GET /api/queries/live`, which read a
|
||||||
|
//! queued entry that has not been written yet. They must describe a query the
|
||||||
|
//! same way — an operator watching the stream and an operator opening the row
|
||||||
|
//! afterwards are looking at the same facts — so the live event *is* this DTO
|
||||||
|
//! and the detail body is this DTO plus the row id.
|
||||||
|
//!
|
||||||
|
//! It is nested rather than flat because the six groups answer six different
|
||||||
|
//! questions, in the order a query meets them: what was asked, which group's
|
||||||
|
//! policy applied, what that policy decided, what was rewritten on the way,
|
||||||
|
//! where the answer came from, and what the client got back.
|
||||||
|
//!
|
||||||
|
//! The list row (`queries_repo.QueryRow`) stays a separate, flatter summary.
|
||||||
|
//! A table the operator scans wants columns, not a tree, and the full story is
|
||||||
|
//! one request away.
|
||||||
|
//!
|
||||||
|
//! Every text field follows the repository's convention: a NULL column reads as
|
||||||
|
//! `""`, and `""` on the wire means "absent". No field is ever written as an
|
||||||
|
//! empty string that means something else.
|
||||||
|
|
||||||
|
const std = @import("std");
|
||||||
|
|
||||||
|
const logger = @import("../storage/logger.zig");
|
||||||
|
const provenance = @import("../storage/provenance.zig");
|
||||||
|
const queries_repo = @import("../storage/repositories/queries_repo.zig");
|
||||||
|
|
||||||
|
/// What the client asked. `time` is unix seconds; `qtype` is null for a
|
||||||
|
/// question whose type the log never recorded.
|
||||||
|
pub const Request = struct {
|
||||||
|
time: i64,
|
||||||
|
domain: []const u8,
|
||||||
|
client: []const u8,
|
||||||
|
qtype: ?u16,
|
||||||
|
qclass: u16,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// The client's filtering group at the time of the query, as a historical fact:
|
||||||
|
/// the id may name a group that has since been renamed or deleted, which is why
|
||||||
|
/// the name is stored beside it rather than joined at read time.
|
||||||
|
pub const Group = struct {
|
||||||
|
id: ?i64,
|
||||||
|
name: []const u8,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// What the policy decided and what it matched on. `matched` is the rule
|
||||||
|
/// pattern or list entry that decided; `source_id`/`source_name` name the
|
||||||
|
/// blocklist it came from, and are absent for a rule.
|
||||||
|
pub const Policy = struct {
|
||||||
|
action: provenance.PolicyAction,
|
||||||
|
reason: provenance.PolicyReason,
|
||||||
|
matched: []const u8,
|
||||||
|
source_id: ?i64,
|
||||||
|
source_name: []const u8,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// The two rewrites that can happen between the question and the answer.
|
||||||
|
/// `cname_target` is set when the decision was made about a CNAME target rather
|
||||||
|
/// than the queried name.
|
||||||
|
pub const Rewrites = struct {
|
||||||
|
cname_target: []const u8,
|
||||||
|
safe_search_target: []const u8,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// Where the answer came from. `upstream` is non-empty only for an attempted
|
||||||
|
/// upstream or forward-zone exchange, including one that failed, and is already
|
||||||
|
/// redacted — the path, query and userinfo of a resolver url never reach here.
|
||||||
|
pub const Route = struct {
|
||||||
|
kind: provenance.RouteKind,
|
||||||
|
forward_zone: []const u8,
|
||||||
|
upstream: []const u8,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// What the client saw. `rcode` is the twelve-bit EDNS extended code, not the
|
||||||
|
/// four header bits alone.
|
||||||
|
pub const Response = struct {
|
||||||
|
rcode: u16,
|
||||||
|
duration_us: ?i64,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// One query, fully explained. The live stream's `data:` payload is exactly
|
||||||
|
/// this.
|
||||||
|
pub const Provenance = struct {
|
||||||
|
request: Request,
|
||||||
|
group: Group,
|
||||||
|
policy: Policy,
|
||||||
|
rewrites: Rewrites,
|
||||||
|
route: Route,
|
||||||
|
response: Response,
|
||||||
|
};
|
||||||
|
|
||||||
|
/// `GET /api/queries/{id}`: the same six groups, plus the id the caller asked
|
||||||
|
/// for. Spelled out rather than composed, because a JSON object is flat at its
|
||||||
|
/// top level and Zig has no field-splicing; the `comptime` block below is what
|
||||||
|
/// keeps the two from drifting.
|
||||||
|
pub const QueryDetail = struct {
|
||||||
|
id: i64,
|
||||||
|
request: Request,
|
||||||
|
group: Group,
|
||||||
|
policy: Policy,
|
||||||
|
rewrites: Rewrites,
|
||||||
|
route: Route,
|
||||||
|
response: Response,
|
||||||
|
};
|
||||||
|
|
||||||
|
comptime {
|
||||||
|
const detail = @typeInfo(QueryDetail).@"struct".fields;
|
||||||
|
const shared = @typeInfo(Provenance).@"struct".fields;
|
||||||
|
std.debug.assert(detail.len == shared.len + 1);
|
||||||
|
std.debug.assert(std.mem.eql(u8, detail[0].name, "id"));
|
||||||
|
for (detail[1..], shared) |a, b| {
|
||||||
|
std.debug.assert(std.mem.eql(u8, a.name, b.name));
|
||||||
|
std.debug.assert(a.type == b.type);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A stored row, as the detail endpoint answers it. Borrows `row`'s strings,
|
||||||
|
/// which the caller's arena owns.
|
||||||
|
pub fn fromDetail(row: queries_repo.QueryDetail) QueryDetail {
|
||||||
|
return .{
|
||||||
|
.id = row.id,
|
||||||
|
.request = .{
|
||||||
|
.time = row.ts,
|
||||||
|
.domain = row.domain,
|
||||||
|
.client = row.client_ip,
|
||||||
|
.qtype = row.qtype,
|
||||||
|
.qclass = row.qclass,
|
||||||
|
},
|
||||||
|
.group = .{ .id = row.group_id, .name = row.group_name },
|
||||||
|
.policy = .{
|
||||||
|
.action = row.policy_action,
|
||||||
|
.reason = row.policy_reason,
|
||||||
|
.matched = row.matched,
|
||||||
|
.source_id = row.source_id,
|
||||||
|
.source_name = row.source_name,
|
||||||
|
},
|
||||||
|
.rewrites = .{
|
||||||
|
.cname_target = row.cname_target,
|
||||||
|
.safe_search_target = row.safe_search_target,
|
||||||
|
},
|
||||||
|
.route = .{
|
||||||
|
.kind = row.route_kind,
|
||||||
|
.forward_zone = row.forward_zone,
|
||||||
|
.upstream = row.upstream,
|
||||||
|
},
|
||||||
|
.response = .{ .rcode = row.rcode, .duration_us = row.response_time_us },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
/// A queued entry, as the live stream sends it. Borrows the entry's buffers, so
|
||||||
|
/// the result must not outlive the entry it was taken from — in the stream both
|
||||||
|
/// live in one loop iteration.
|
||||||
|
///
|
||||||
|
/// There is no id: the entry precedes its own insert, so no row id exists yet.
|
||||||
|
pub fn fromEntry(entry: *const logger.Entry) Provenance {
|
||||||
|
return .{
|
||||||
|
.request = .{
|
||||||
|
.time = entry.timestamp,
|
||||||
|
.domain = entry.domain(),
|
||||||
|
.client = entry.clientIp(),
|
||||||
|
.qtype = entry.qtype,
|
||||||
|
.qclass = entry.qclass,
|
||||||
|
},
|
||||||
|
.group = .{ .id = entry.group_id, .name = entry.groupName() },
|
||||||
|
.policy = .{
|
||||||
|
.action = entry.policy_action,
|
||||||
|
.reason = entry.policy_reason,
|
||||||
|
.matched = entry.matched(),
|
||||||
|
.source_id = entry.source_id,
|
||||||
|
.source_name = entry.sourceName(),
|
||||||
|
},
|
||||||
|
.rewrites = .{
|
||||||
|
.cname_target = entry.cnameTarget(),
|
||||||
|
.safe_search_target = entry.safeSearchTarget(),
|
||||||
|
},
|
||||||
|
.route = .{
|
||||||
|
.kind = entry.route_kind,
|
||||||
|
.forward_zone = entry.forwardZone(),
|
||||||
|
.upstream = entry.upstream(),
|
||||||
|
},
|
||||||
|
.response = .{ .rcode = entry.rcode, .duration_us = entry.response_time_us },
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// tests
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
const testing = std.testing;
|
||||||
|
|
||||||
|
test "a stored row and a queued entry describe the same query identically" {
|
||||||
|
const row: queries_repo.QueryDetail = .{
|
||||||
|
.id = 7,
|
||||||
|
.ts = 1_700_000_000,
|
||||||
|
.domain = "ads.example",
|
||||||
|
.client_ip = "192.0.2.10",
|
||||||
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 3,
|
||||||
|
.blocked = true,
|
||||||
|
.response_time_us = 1234,
|
||||||
|
.cache_hit = false,
|
||||||
|
.upstream = "https://dns.example",
|
||||||
|
.group_id = 2,
|
||||||
|
.group_name = "kids",
|
||||||
|
.policy_action = .block,
|
||||||
|
.policy_reason = .blocklist_domain,
|
||||||
|
.matched = "tracker.example",
|
||||||
|
.source_id = 5,
|
||||||
|
.source_name = "StevenBlack",
|
||||||
|
.cname_target = "tracker.example",
|
||||||
|
.safe_search_target = "forcesafesearch.example",
|
||||||
|
.route_kind = .blocked,
|
||||||
|
.forward_zone = "lan",
|
||||||
|
};
|
||||||
|
|
||||||
|
const entry: logger.Entry = .init(.{
|
||||||
|
.timestamp = row.ts,
|
||||||
|
.domain = row.domain,
|
||||||
|
.client_ip = row.client_ip,
|
||||||
|
.qtype = row.qtype,
|
||||||
|
.qclass = row.qclass,
|
||||||
|
.rcode = row.rcode,
|
||||||
|
.blocked = row.blocked,
|
||||||
|
.response_time_us = row.response_time_us,
|
||||||
|
.cache_hit = row.cache_hit,
|
||||||
|
.upstream = row.upstream,
|
||||||
|
.group_id = row.group_id,
|
||||||
|
.group_name = row.group_name,
|
||||||
|
.policy_action = row.policy_action,
|
||||||
|
.policy_reason = row.policy_reason,
|
||||||
|
.matched = row.matched,
|
||||||
|
.source_id = row.source_id,
|
||||||
|
.source_name = row.source_name,
|
||||||
|
.cname_target = row.cname_target,
|
||||||
|
.safe_search_target = row.safe_search_target,
|
||||||
|
.route_kind = row.route_kind,
|
||||||
|
.forward_zone = row.forward_zone,
|
||||||
|
});
|
||||||
|
|
||||||
|
const from_row = fromDetail(row);
|
||||||
|
const from_entry = fromEntry(&entry);
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, 7), from_row.id);
|
||||||
|
inline for (@typeInfo(Provenance).@"struct".fields) |field| {
|
||||||
|
const a = @field(from_row, field.name);
|
||||||
|
const b = @field(from_entry, field.name);
|
||||||
|
inline for (@typeInfo(field.type).@"struct".fields) |inner| {
|
||||||
|
const left = @field(a, inner.name);
|
||||||
|
const right = @field(b, inner.name);
|
||||||
|
if (@TypeOf(left) == []const u8) {
|
||||||
|
try testing.expectEqualStrings(left, right);
|
||||||
|
} else {
|
||||||
|
try testing.expectEqual(left, right);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test "an unexplained query serializes as nulls and empty strings, not as absent keys" {
|
||||||
|
const entry: logger.Entry = .init(.{
|
||||||
|
.timestamp = 1,
|
||||||
|
.domain = "safe.example",
|
||||||
|
.client_ip = "192.0.2.11",
|
||||||
|
});
|
||||||
|
|
||||||
|
var allocating: std.Io.Writer.Allocating = .init(testing.allocator);
|
||||||
|
defer allocating.deinit();
|
||||||
|
try std.json.Stringify.value(fromEntry(&entry), .{}, &allocating.writer);
|
||||||
|
const text = allocating.written();
|
||||||
|
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"qtype\":null"));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"duration_us\":null"));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"id\":null"));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"upstream\":\"\""));
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"action\":\"not_evaluated\""));
|
||||||
|
}
|
||||||
+5
-1
@@ -67,6 +67,10 @@ pub const table: []const router.RouteInfo = &.{
|
|||||||
// Query log, stats, live stream, lookup.
|
// Query log, stats, live stream, lookup.
|
||||||
.{ .method = .GET, .pattern = "/api/queries", .auth = .session, .policy = .read, .handler = queries.list },
|
.{ .method = .GET, .pattern = "/api/queries", .auth = .session, .policy = .read, .handler = queries.list },
|
||||||
.{ .method = .GET, .pattern = "/api/queries/live", .auth = .session, .policy = .read, .handler = live.stream, .rate_limit = .exempt },
|
.{ .method = .GET, .pattern = "/api/queries/live", .auth = .session, .policy = .read, .handler = live.stream, .rate_limit = .exempt },
|
||||||
|
// Listed after the literal `live`, which a linear first-match scan reaches
|
||||||
|
// first — though `{id}` would refuse it anyway, since it captures a
|
||||||
|
// positive integer and nothing else.
|
||||||
|
.{ .method = .GET, .pattern = "/api/queries/{id}", .auth = .session, .policy = .read, .handler = queries.detail },
|
||||||
.{ .method = .GET, .pattern = "/api/stats", .auth = .session, .policy = .read, .handler = stats.totals },
|
.{ .method = .GET, .pattern = "/api/stats", .auth = .session, .policy = .read, .handler = stats.totals },
|
||||||
.{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .handler = stats.timeseries },
|
.{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .handler = stats.timeseries },
|
||||||
.{ .method = .GET, .pattern = "/api/lookup", .auth = .session, .policy = .read, .handler = lookup.handle },
|
.{ .method = .GET, .pattern = "/api/lookup", .auth = .session, .policy = .read, .handler = lookup.handle },
|
||||||
@@ -152,7 +156,7 @@ const std = @import("std");
|
|||||||
const testing = std.testing;
|
const testing = std.testing;
|
||||||
|
|
||||||
test "the table carries every endpoint of the milestone" {
|
test "the table carries every endpoint of the milestone" {
|
||||||
try testing.expectEqual(@as(usize, 60), table.len);
|
try testing.expectEqual(@as(usize, 61), table.len);
|
||||||
}
|
}
|
||||||
|
|
||||||
test "no two entries claim the same method and pattern" {
|
test "no two entries claim the same method and pattern" {
|
||||||
|
|||||||
@@ -25,6 +25,14 @@ pub const max_subscribers = 32;
|
|||||||
|
|
||||||
/// Entries one subscriber may fall behind by. At household query rates this is
|
/// Entries one subscriber may fall behind by. At household query rates this is
|
||||||
/// several seconds of slack on a stalled TCP connection.
|
/// several seconds of slack on a stalled TCP connection.
|
||||||
|
///
|
||||||
|
/// The ring is embedded in the slot, so this multiplies `@sizeOf(logger.Entry)`
|
||||||
|
/// — about 1.8 KiB once milestone 28 widened it for provenance. One subscriber
|
||||||
|
/// therefore costs roughly 115 KiB of ring and the whole hub roughly 3.6 MiB,
|
||||||
|
/// allocated once for the life of the process. That is the reason to keep both
|
||||||
|
/// this and `max_subscribers` small: they are paid whether or not anyone is
|
||||||
|
/// watching. `logger.query_log_buffer_max` bounds the other consumer of the
|
||||||
|
/// same width, the writer queue.
|
||||||
pub const ring_capacity = 64;
|
pub const ring_capacity = 64;
|
||||||
|
|
||||||
pub const SubscriberId = enum(u8) { _ };
|
pub const SubscriberId = enum(u8) { _ };
|
||||||
|
|||||||
@@ -38,15 +38,20 @@ const header = @import("../dns/header.zig");
|
|||||||
const http_util = @import("http_util.zig");
|
const http_util = @import("http_util.zig");
|
||||||
const local_repo = @import("../storage/repositories/local_repo.zig");
|
const local_repo = @import("../storage/repositories/local_repo.zig");
|
||||||
const local_tables_mod = @import("../server/local_tables.zig");
|
const local_tables_mod = @import("../server/local_tables.zig");
|
||||||
|
const logger_mod = @import("../storage/logger.zig");
|
||||||
const manager_mod = @import("../filter/manager.zig");
|
const manager_mod = @import("../filter/manager.zig");
|
||||||
const migrations = @import("../storage/migrations.zig");
|
const migrations = @import("../storage/migrations.zig");
|
||||||
const name = @import("../dns/name.zig");
|
const name = @import("../dns/name.zig");
|
||||||
const openapi = @import("openapi.zig");
|
const openapi = @import("openapi.zig");
|
||||||
const packet = @import("../dns/packet.zig");
|
const packet = @import("../dns/packet.zig");
|
||||||
const pause_mod = @import("../server/pause.zig");
|
const pause_mod = @import("../server/pause.zig");
|
||||||
|
const coverage_mod = @import("coverage.zig");
|
||||||
const pool_mod = @import("../upstream/pool.zig");
|
const pool_mod = @import("../upstream/pool.zig");
|
||||||
|
const provenance = @import("../storage/provenance.zig");
|
||||||
|
const provenance_view = @import("provenance_view.zig");
|
||||||
const queries_repo = @import("../storage/repositories/queries_repo.zig");
|
const queries_repo = @import("../storage/repositories/queries_repo.zig");
|
||||||
const querylog_schema = @import("../storage/querylog_schema.zig");
|
const querylog_schema = @import("../storage/querylog_schema.zig");
|
||||||
|
const query_sink = @import("../server/query_sink.zig");
|
||||||
const question = @import("../dns/question.zig");
|
const question = @import("../dns/question.zig");
|
||||||
const router = @import("router.zig");
|
const router = @import("router.zig");
|
||||||
const server = @import("server.zig");
|
const server = @import("server.zig");
|
||||||
@@ -257,7 +262,10 @@ fn contentLength(head: []const u8) ?usize {
|
|||||||
// the environment: the real web stack over in-memory databases
|
// the environment: the real web stack over in-memory databases
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
const seeded_query_rows = 25;
|
/// The rows the uniform loop writes, before the two provenance-rich ones the
|
||||||
|
/// detail endpoint and the credential sweep read.
|
||||||
|
const seeded_plain_query_rows = 25;
|
||||||
|
const seeded_query_rows = seeded_plain_query_rows + 2;
|
||||||
|
|
||||||
const EnvOptions = struct {
|
const EnvOptions = struct {
|
||||||
password_hash: []const u8 = "",
|
password_hash: []const u8 = "",
|
||||||
@@ -270,6 +278,10 @@ const EnvOptions = struct {
|
|||||||
/// wants; the file-authority tests below name a path.
|
/// wants; the file-authority tests below name a path.
|
||||||
authority: server.Authority = .database,
|
authority: server.Authority = .database,
|
||||||
reconciled_at: ?i64 = null,
|
reconciled_at: ?i64 = null,
|
||||||
|
/// False detaches the query log from the web state, which is the box a
|
||||||
|
/// `logging.query_log = false` operator runs. Every query-log route then
|
||||||
|
/// answers 503 rather than an empty page, which would be a lie.
|
||||||
|
querylog: bool = true,
|
||||||
};
|
};
|
||||||
|
|
||||||
/// Heap-allocated because `state` and the listener hold pointers into it.
|
/// Heap-allocated because `state` and the listener hold pointers into it.
|
||||||
@@ -400,7 +412,7 @@ const Env = struct {
|
|||||||
.limiter = &self.limiter,
|
.limiter = &self.limiter,
|
||||||
.hub = self.hub,
|
.hub = self.hub,
|
||||||
.config_db = &self.config_db,
|
.config_db = &self.config_db,
|
||||||
.querylog_db = &self.querylog_db,
|
.querylog_db = if (options.querylog) &self.querylog_db else null,
|
||||||
.events = &self.events_store,
|
.events = &self.events_store,
|
||||||
.version = "w10-test",
|
.version = "w10-test",
|
||||||
.started_unix = std.Io.Clock.real.now(ioh).toSeconds(),
|
.started_unix = std.Io.Clock.real.now(ioh).toSeconds(),
|
||||||
@@ -472,27 +484,108 @@ fn seedConfig(database: *db.Db) !void {
|
|||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// The oldest instant the seeded log is complete for. Pinned rather than taken
|
||||||
|
/// from `unixepoch()`, which the schema's own seed uses: the contract samples
|
||||||
|
/// are byte-compared, so a clock in `coverage.available_since` would make the
|
||||||
|
/// golden a property of the machine that generated it.
|
||||||
|
///
|
||||||
|
/// It equals the oldest seeded row's timestamp, so a request bounded at exactly
|
||||||
|
/// this instant is complete and one bounded a second earlier is not.
|
||||||
|
const seeded_available_since: i64 = 1_700_000_000;
|
||||||
|
|
||||||
fn seedQueryLog(database: *db.Db) !void {
|
fn seedQueryLog(database: *db.Db) !void {
|
||||||
|
try database.exec(
|
||||||
|
\\UPDATE querylog_meta SET created_at = 1700000000, available_since = 1700000000 WHERE id = 1
|
||||||
|
);
|
||||||
|
|
||||||
var writer = try queries_repo.BatchWriter.init(database);
|
var writer = try queries_repo.BatchWriter.init(database);
|
||||||
defer writer.deinit();
|
defer writer.deinit();
|
||||||
|
|
||||||
var domain_buf: [32]u8 = undefined;
|
var domain_buf: [32]u8 = undefined;
|
||||||
var index: usize = 0;
|
var index: usize = 0;
|
||||||
while (index < seeded_query_rows) : (index += 1) {
|
while (index < seeded_plain_query_rows) : (index += 1) {
|
||||||
const domain = std.fmt.bufPrint(&domain_buf, "d{d}.example", .{index}) catch unreachable;
|
const domain = std.fmt.bufPrint(&domain_buf, "d{d}.example", .{index}) catch unreachable;
|
||||||
const blocked = index % 5 == 0;
|
const blocked = index % 5 == 0;
|
||||||
|
// The three states the handler can actually produce (`Context.cacheHit`
|
||||||
|
// and `route_kind` are set together): a blocked answer consulted no
|
||||||
|
// cache and named no resolver, a cache hit named no resolver, and only
|
||||||
|
// an upstream exchange did both.
|
||||||
|
const from_cache = !blocked and index % 2 == 0;
|
||||||
try writer.writeBatch(&.{.{
|
try writer.writeBatch(&.{.{
|
||||||
.timestamp = 1_700_000_000 + @as(i64, @intCast(index)),
|
.timestamp = 1_700_000_000 + @as(i64, @intCast(index)),
|
||||||
.domain = domain,
|
.domain = domain,
|
||||||
.client_ip = "192.0.2.10",
|
.client_ip = "192.0.2.10",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
.blocked = blocked,
|
.blocked = blocked,
|
||||||
.block_reason = if (blocked) "blocklist_domain" else null,
|
|
||||||
.response_time_us = 250,
|
.response_time_us = 250,
|
||||||
.cache_hit = if (blocked) null else (index % 2 == 0),
|
.cache_hit = if (blocked) null else from_cache,
|
||||||
.upstream = if (blocked) null else "https://dns.example/dns-query",
|
.upstream = if (blocked or from_cache) null else "https://dns.example/dns-query",
|
||||||
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = if (blocked) .block else .allow,
|
||||||
|
.policy_reason = if (blocked) .blocklist_domain else .no_match,
|
||||||
|
.matched = if (blocked) domain else null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = if (blocked) .blocked else if (from_cache) .cache else .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
}});
|
}});
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Two rows with provenance the loop above never produces, so the detail
|
||||||
|
// endpoint and its contract sample have a real row to read. They are the
|
||||||
|
// newest rows, so a first page shows them.
|
||||||
|
try writer.writeBatch(&.{.{
|
||||||
|
.timestamp = 1_700_000_000 + seeded_plain_query_rows,
|
||||||
|
.domain = "news.example",
|
||||||
|
.client_ip = "192.0.2.10",
|
||||||
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 0,
|
||||||
|
.blocked = false,
|
||||||
|
.response_time_us = 18_400,
|
||||||
|
.cache_hit = false,
|
||||||
|
.upstream = "https://dns.example/dns-query",
|
||||||
|
.group_id = 1,
|
||||||
|
.group_name = "default",
|
||||||
|
.policy_action = .allow,
|
||||||
|
.policy_reason = .no_match,
|
||||||
|
.matched = null,
|
||||||
|
.source_id = null,
|
||||||
|
.source_name = null,
|
||||||
|
.cname_target = null,
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .upstream,
|
||||||
|
.forward_zone = null,
|
||||||
|
}});
|
||||||
|
|
||||||
|
try writer.writeBatch(&.{.{
|
||||||
|
.timestamp = 1_700_000_000 + seeded_plain_query_rows + 1,
|
||||||
|
.domain = "shop.example",
|
||||||
|
.client_ip = "192.0.2.11",
|
||||||
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
|
.rcode = 3,
|
||||||
|
.blocked = true,
|
||||||
|
.response_time_us = 900,
|
||||||
|
.cache_hit = null,
|
||||||
|
.upstream = null,
|
||||||
|
.group_id = 2,
|
||||||
|
.group_name = "kids",
|
||||||
|
.policy_action = .block,
|
||||||
|
.policy_reason = .blocklist_wildcard,
|
||||||
|
.matched = "||tracker.example^",
|
||||||
|
.source_id = 4,
|
||||||
|
.source_name = "StevenBlack",
|
||||||
|
.cname_target = "cdn.tracker.example",
|
||||||
|
.safe_search_target = null,
|
||||||
|
.route_kind = .blocked,
|
||||||
|
.forward_zone = null,
|
||||||
|
}});
|
||||||
}
|
}
|
||||||
|
|
||||||
/// A fixed instant, like every other seeded timestamp here: the contract
|
/// A fixed instant, like every other seeded timestamp here: the contract
|
||||||
@@ -653,6 +746,7 @@ const contract = [_]Contract{
|
|||||||
|
|
||||||
// Query log, stats, live stream, upstream health.
|
// Query log, stats, live stream, upstream health.
|
||||||
.{ .method = .GET, .pattern = "/api/queries", .auth = .session, .policy = .read, .target = "/api/queries?limit=10", .status = 200, .check = jsonShape(handlers_queries.Page) },
|
.{ .method = .GET, .pattern = "/api/queries", .auth = .session, .policy = .read, .target = "/api/queries?limit=10", .status = 200, .check = jsonShape(handlers_queries.Page) },
|
||||||
|
.{ .method = .GET, .pattern = "/api/queries/{id}", .auth = .session, .policy = .read, .target = "/api/queries/27", .status = 200, .check = jsonShape(provenance_view.QueryDetail) },
|
||||||
.{ .method = .GET, .pattern = "/api/queries/live", .auth = .session, .policy = .read, .rate_limit = .exempt, .target = "/api/queries/live", .status = 200, .kind = .sse },
|
.{ .method = .GET, .pattern = "/api/queries/live", .auth = .session, .policy = .read, .rate_limit = .exempt, .target = "/api/queries/live", .status = 200, .kind = .sse },
|
||||||
.{ .method = .GET, .pattern = "/api/stats", .auth = .session, .policy = .read, .target = "/api/stats?period=1h", .status = 200, .check = jsonShape(handlers_stats.TotalsBody) },
|
.{ .method = .GET, .pattern = "/api/stats", .auth = .session, .policy = .read, .target = "/api/stats?period=1h", .status = 200, .check = jsonShape(handlers_stats.TotalsBody) },
|
||||||
.{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .target = "/api/stats/timeseries?period=1h", .status = 200, .check = jsonShape(handlers_stats.TimeseriesBody) },
|
.{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .target = "/api/stats/timeseries?period=1h", .status = 200, .check = jsonShape(handlers_stats.TimeseriesBody) },
|
||||||
@@ -1653,12 +1747,15 @@ fn sseStream(io: std.Io, env: *Env) anyerror!void {
|
|||||||
.domain = "live.example",
|
.domain = "live.example",
|
||||||
.client_ip = "192.0.2.99",
|
.client_ip = "192.0.2.99",
|
||||||
.qtype = 1,
|
.qtype = 1,
|
||||||
|
.qclass = 1,
|
||||||
.blocked = true,
|
.blocked = true,
|
||||||
.block_reason = "blocklist_domain",
|
.policy_action = .block,
|
||||||
|
.policy_reason = .blocklist_domain,
|
||||||
|
.route_kind = .blocked,
|
||||||
}));
|
}));
|
||||||
try conn.readChunkedUntil(&seen, env.gpa, "event: query");
|
try conn.readChunkedUntil(&seen, env.gpa, "event: query");
|
||||||
try conn.readChunkedUntil(&seen, env.gpa, "\"domain\":\"live.example\"");
|
try conn.readChunkedUntil(&seen, env.gpa, "\"domain\":\"live.example\"");
|
||||||
try conn.readChunkedUntil(&seen, env.gpa, "\"blocked\":true");
|
try conn.readChunkedUntil(&seen, env.gpa, "\"reason\":\"blocklist_domain\"");
|
||||||
|
|
||||||
// The cap is per address and the environment allows one stream: a second
|
// The cap is per address and the environment allows one stream: a second
|
||||||
// subscriber from the same address is refused while the first is open.
|
// subscriber from the same address is refused while the first is open.
|
||||||
@@ -1784,7 +1881,7 @@ fn paginationWalk(io: std.Io, env: *Env) anyerror!void {
|
|||||||
try testing.expect(pages < 10);
|
try testing.expect(pages < 10);
|
||||||
}
|
}
|
||||||
|
|
||||||
// 25 seeded rows walk as 10, 10 and 5, with the cursor ending exactly
|
// 27 seeded rows walk as 10, 10 and 7, with the cursor ending exactly
|
||||||
// after the third page.
|
// after the third page.
|
||||||
try testing.expectEqual(@as(usize, seeded_query_rows), total);
|
try testing.expectEqual(@as(usize, seeded_query_rows), total);
|
||||||
try testing.expectEqual(@as(usize, 3), pages);
|
try testing.expectEqual(@as(usize, 3), pages);
|
||||||
@@ -1800,6 +1897,333 @@ test "W10 keyset pagination walks the seeded log exactly once, newest first" {
|
|||||||
try bounded(env.io(), default_budget, paginationWalk, .{ env.io(), env });
|
try bounded(env.io(), default_budget, paginationWalk, .{ env.io(), env });
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// query provenance: the detail endpoint, coverage, and the credential sweep
|
||||||
|
// (milestone 28)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
/// The id of the seeded CNAME-uncloaked block, which is the last row written.
|
||||||
|
const seeded_detail_id = seeded_query_rows;
|
||||||
|
|
||||||
|
fn detailWalk(io: std.Io, env: *Env) anyerror!void {
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(env.gpa);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
|
||||||
|
var conn: Conn = undefined;
|
||||||
|
try conn.connect(io, env.addr);
|
||||||
|
defer conn.close(io);
|
||||||
|
|
||||||
|
var body_buf: [64 * 1024]u8 = undefined;
|
||||||
|
var target_buf: [64]u8 = undefined;
|
||||||
|
|
||||||
|
const target = try std.fmt.bufPrint(&target_buf, "/api/queries/{d}", .{seeded_detail_id});
|
||||||
|
try conn.request("GET", target, null, null);
|
||||||
|
const response = try conn.receive(&body_buf);
|
||||||
|
try testing.expectEqual(@as(u16, 200), response.status);
|
||||||
|
|
||||||
|
const detail = try std.json.parseFromSliceLeaky(
|
||||||
|
provenance_view.QueryDetail,
|
||||||
|
arena_state.allocator(),
|
||||||
|
response.body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
|
||||||
|
try testing.expectEqual(@as(i64, seeded_detail_id), detail.id);
|
||||||
|
try testing.expectEqualStrings("shop.example", detail.request.domain);
|
||||||
|
try testing.expectEqualStrings("192.0.2.11", detail.request.client);
|
||||||
|
try testing.expectEqual(@as(u16, 1), detail.request.qclass);
|
||||||
|
try testing.expectEqual(@as(?i64, 2), detail.group.id);
|
||||||
|
try testing.expectEqualStrings("kids", detail.group.name);
|
||||||
|
try testing.expectEqual(provenance.PolicyAction.block, detail.policy.action);
|
||||||
|
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, detail.policy.reason);
|
||||||
|
try testing.expectEqualStrings("||tracker.example^", detail.policy.matched);
|
||||||
|
try testing.expectEqual(@as(?i64, 4), detail.policy.source_id);
|
||||||
|
try testing.expectEqualStrings("StevenBlack", detail.policy.source_name);
|
||||||
|
try testing.expectEqualStrings("cdn.tracker.example", detail.rewrites.cname_target);
|
||||||
|
try testing.expectEqualStrings("", detail.rewrites.safe_search_target);
|
||||||
|
try testing.expectEqual(provenance.RouteKind.blocked, detail.route.kind);
|
||||||
|
// A blocked query attempted no exchange, so it names no resolver.
|
||||||
|
try testing.expectEqualStrings("", detail.route.upstream);
|
||||||
|
try testing.expectEqual(@as(u16, 3), detail.response.rcode);
|
||||||
|
try testing.expectEqual(@as(?i64, 900), detail.response.duration_us);
|
||||||
|
|
||||||
|
// An id past the end of the log and an id retention would have pruned are
|
||||||
|
// the same answer.
|
||||||
|
const missing = try std.fmt.bufPrint(&target_buf, "/api/queries/{d}", .{seeded_query_rows + 1000});
|
||||||
|
try conn.request("GET", missing, null, null);
|
||||||
|
const not_found = try conn.receive(&body_buf);
|
||||||
|
try testing.expectEqual(@as(u16, 404), not_found.status);
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, not_found.body, 1, "\"error\""));
|
||||||
|
|
||||||
|
// A non-positive id never reaches SQL: the pattern captures a positive
|
||||||
|
// integer or does not match, so this is a routing 404.
|
||||||
|
try conn.request("GET", "/api/queries/0", null, null);
|
||||||
|
try testing.expectEqual(@as(u16, 404), (try conn.receive(&body_buf)).status);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "W10 milestone 28: the detail endpoint answers one row and 404s the rest" {
|
||||||
|
if (!build_options.integration) return error.SkipZigTest;
|
||||||
|
|
||||||
|
const gpa = testing.allocator;
|
||||||
|
var env = try Env.create(gpa, .{});
|
||||||
|
defer env.destroy();
|
||||||
|
|
||||||
|
try bounded(env.io(), default_budget, detailWalk, .{ env.io(), env });
|
||||||
|
}
|
||||||
|
|
||||||
|
fn detailUnavailable(io: std.Io, env: *Env) anyerror!void {
|
||||||
|
var conn: Conn = undefined;
|
||||||
|
try conn.connect(io, env.addr);
|
||||||
|
defer conn.close(io);
|
||||||
|
|
||||||
|
var body_buf: [8 * 1024]u8 = undefined;
|
||||||
|
for ([_][]const u8{ "/api/queries/1", "/api/queries?limit=1", "/api/stats", "/api/stats/timeseries" }) |target| {
|
||||||
|
try conn.request("GET", target, null, null);
|
||||||
|
const response = try conn.receive(&body_buf);
|
||||||
|
try testing.expectEqual(@as(u16, 503), response.status);
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, response.body, 1, "query log unavailable"));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test "W10 milestone 28: a box with no query log answers 503, not an empty page" {
|
||||||
|
if (!build_options.integration) return error.SkipZigTest;
|
||||||
|
|
||||||
|
const gpa = testing.allocator;
|
||||||
|
var env = try Env.create(gpa, .{ .querylog = false });
|
||||||
|
defer env.destroy();
|
||||||
|
|
||||||
|
try bounded(env.io(), default_budget, detailUnavailable, .{ env.io(), env });
|
||||||
|
}
|
||||||
|
|
||||||
|
fn coverageWalk(io: std.Io, env: *Env) anyerror!void {
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(env.gpa);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
const arena = arena_state.allocator();
|
||||||
|
|
||||||
|
var conn: Conn = undefined;
|
||||||
|
try conn.connect(io, env.addr);
|
||||||
|
defer conn.close(io);
|
||||||
|
|
||||||
|
var body_buf: [64 * 1024]u8 = undefined;
|
||||||
|
var target_buf: [64]u8 = undefined;
|
||||||
|
|
||||||
|
// No lower bound: the request asks about all of history, which a file that
|
||||||
|
// may have pruned cannot promise.
|
||||||
|
try conn.request("GET", "/api/queries?limit=1", null, null);
|
||||||
|
const unbounded = try std.json.parseFromSliceLeaky(
|
||||||
|
handlers_queries.Page,
|
||||||
|
arena,
|
||||||
|
(try conn.receive(&body_buf)).body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
try testing.expectEqual(seeded_available_since, unbounded.coverage.available_since);
|
||||||
|
try testing.expect(!unbounded.coverage.complete);
|
||||||
|
|
||||||
|
// Bounded exactly at the watermark.
|
||||||
|
const at = try std.fmt.bufPrint(&target_buf, "/api/queries?limit=1&since={d}", .{seeded_available_since});
|
||||||
|
try conn.request("GET", at, null, null);
|
||||||
|
const covered = try std.json.parseFromSliceLeaky(
|
||||||
|
handlers_queries.Page,
|
||||||
|
arena,
|
||||||
|
(try conn.receive(&body_buf)).body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
try testing.expect(covered.coverage.complete);
|
||||||
|
|
||||||
|
// One second earlier, and the window reaches past what the file holds.
|
||||||
|
const before = try std.fmt.bufPrint(&target_buf, "/api/queries?limit=1&since={d}", .{seeded_available_since - 1});
|
||||||
|
try conn.request("GET", before, null, null);
|
||||||
|
const partial = try std.json.parseFromSliceLeaky(
|
||||||
|
handlers_queries.Page,
|
||||||
|
arena,
|
||||||
|
(try conn.receive(&body_buf)).body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
try testing.expect(!partial.coverage.complete);
|
||||||
|
|
||||||
|
// The stats endpoints judge the same watermark against their own aligned
|
||||||
|
// window, which for any live period starts well after the seeded rows.
|
||||||
|
try conn.request("GET", "/api/stats?period=1h", null, null);
|
||||||
|
const totals = try std.json.parseFromSliceLeaky(
|
||||||
|
handlers_stats.TotalsBody,
|
||||||
|
arena,
|
||||||
|
(try conn.receive(&body_buf)).body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
try testing.expectEqual(seeded_available_since, totals.coverage.available_since);
|
||||||
|
try testing.expectEqual(totals.since >= seeded_available_since, totals.coverage.complete);
|
||||||
|
|
||||||
|
try conn.request("GET", "/api/stats/timeseries?period=1h", null, null);
|
||||||
|
const series = try std.json.parseFromSliceLeaky(
|
||||||
|
handlers_stats.TimeseriesBody,
|
||||||
|
arena,
|
||||||
|
(try conn.receive(&body_buf)).body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
try testing.expectEqual(totals.since, series.since);
|
||||||
|
try testing.expectEqual(totals.coverage.complete, series.coverage.complete);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "W10 milestone 28: every window-bounded endpoint reports its own coverage" {
|
||||||
|
if (!build_options.integration) return error.SkipZigTest;
|
||||||
|
|
||||||
|
const gpa = testing.allocator;
|
||||||
|
var env = try Env.create(gpa, .{});
|
||||||
|
defer env.destroy();
|
||||||
|
|
||||||
|
try bounded(env.io(), default_budget, coverageWalk, .{ env.io(), env });
|
||||||
|
}
|
||||||
|
|
||||||
|
/// NextDNS's shape: the account id rides in the path, which is exactly where a
|
||||||
|
/// credential lives in a url an operator may legitimately configure.
|
||||||
|
/// `Endpoint.parse` refuses userinfo, so the path is the shape a real
|
||||||
|
/// configuration can carry a secret in — and the path is what
|
||||||
|
/// `safe_url.redact` drops.
|
||||||
|
const sweep_token = "b1c2d3";
|
||||||
|
const sweep_upstream_url = "https://dns.nextdns.io/" ++ sweep_token;
|
||||||
|
/// What every surface must show instead. The origin survives redaction — an
|
||||||
|
/// operator reading a failure has to know where the query went — so each
|
||||||
|
/// surface is checked for it too: one that showed nothing at all would pass a
|
||||||
|
/// secret check by saying nothing.
|
||||||
|
const sweep_redacted_upstream = "https://dns.nextdns.io";
|
||||||
|
/// The name the swept query asks for, so each surface can be pinned to the row
|
||||||
|
/// this test produced rather than to a seeded one.
|
||||||
|
const sweep_domain = "creds.example";
|
||||||
|
|
||||||
|
/// Names the resolver and never its token.
|
||||||
|
fn expectRedacted(text: []const u8) !void {
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, text, 1, sweep_redacted_upstream));
|
||||||
|
try testing.expect(!std.mem.containsAtLeast(u8, text, 1, sweep_token));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The cross-surface credential sweep, driven end to end: a real `Handler`
|
||||||
|
/// answers a real query through a resolver whose url carries a token, and the
|
||||||
|
/// entry travels the production path — `QuerySink`, then the hub and the
|
||||||
|
/// logger, then the query log the API reads. Nothing here redacts anything, so
|
||||||
|
/// a handler that stopped redacting fails this test.
|
||||||
|
///
|
||||||
|
/// Four surfaces read the same query back: the stored row, straight out of
|
||||||
|
/// SQLite, and the three the operator's browser sees — the live frame, the list
|
||||||
|
/// page and the detail body. A leak on any one of them is a secret in a browser
|
||||||
|
/// history, and the four are separate code paths to the same text.
|
||||||
|
fn credentialSweep(
|
||||||
|
io: std.Io,
|
||||||
|
env: *Env,
|
||||||
|
query_logger: *logger_mod.Logger,
|
||||||
|
handler: *dns_handler.Handler,
|
||||||
|
) anyerror!void {
|
||||||
|
var arena_state: std.heap.ArenaAllocator = .init(env.gpa);
|
||||||
|
defer arena_state.deinit();
|
||||||
|
const arena = arena_state.allocator();
|
||||||
|
|
||||||
|
// Subscribed before the query runs: the hub publishes to whoever is
|
||||||
|
// listening at that moment and keeps nothing for a later reader.
|
||||||
|
var seen: std.ArrayList(u8) = .empty;
|
||||||
|
defer seen.deinit(env.gpa);
|
||||||
|
var stream: Conn = undefined;
|
||||||
|
try openLiveStream(io, env, &stream, &seen);
|
||||||
|
defer stream.close(io);
|
||||||
|
|
||||||
|
var query_buf: [512]u8 = undefined;
|
||||||
|
var response_buf: [512]u8 = undefined;
|
||||||
|
var scratch: dns_handler.Scratch = undefined;
|
||||||
|
const from = address.NetAddress.fromIp(.{ .ip4 = .loopback(53100) });
|
||||||
|
const query = queryFor(&query_buf, 0x4444, sweep_domain, .a);
|
||||||
|
try testing.expect(handler.handle(io, .udp, from, query, &response_buf, &scratch) == .reply);
|
||||||
|
|
||||||
|
// The live frame. Waiting on the redacted origin rather than on the whole
|
||||||
|
// frame is safe in both directions: a leaked url starts with it.
|
||||||
|
try stream.readChunkedUntil(&seen, env.gpa, sweep_redacted_upstream);
|
||||||
|
try testing.expect(std.mem.containsAtLeast(u8, seen.items, 1, sweep_domain));
|
||||||
|
try expectRedacted(seen.items);
|
||||||
|
|
||||||
|
// The stored row. The producer has already run, so closing the queue and
|
||||||
|
// running the writer inline drains it in one call: `runWriter` returns when
|
||||||
|
// a closed queue is empty, and a zero flush interval makes it commit the
|
||||||
|
// batch it holds rather than wait for company.
|
||||||
|
query_logger.shutdown(io);
|
||||||
|
try query_logger.runWriter(io, &env.querylog_db, null);
|
||||||
|
try testing.expectEqual(@as(u64, 1), query_logger.rows_written.load(.monotonic));
|
||||||
|
|
||||||
|
var stmt = try env.querylog_db.prepare(
|
||||||
|
\\SELECT query_log.id, query_log.upstream
|
||||||
|
\\FROM query_log JOIN domains ON domains.id = query_log.domain_id
|
||||||
|
\\WHERE domains.domain = ?
|
||||||
|
);
|
||||||
|
defer stmt.deinit();
|
||||||
|
try stmt.bindText(1, sweep_domain);
|
||||||
|
try testing.expect(try stmt.step());
|
||||||
|
const row_id = stmt.columnInt(0);
|
||||||
|
try testing.expectEqualStrings(sweep_redacted_upstream, stmt.columnText(1));
|
||||||
|
try testing.expect(!try stmt.step());
|
||||||
|
|
||||||
|
var conn: Conn = undefined;
|
||||||
|
try conn.connect(io, env.addr);
|
||||||
|
defer conn.close(io);
|
||||||
|
|
||||||
|
var body_buf: [64 * 1024]u8 = undefined;
|
||||||
|
var target_buf: [64]u8 = undefined;
|
||||||
|
|
||||||
|
// The list row. The swept query is the newest in the log, so a page of one
|
||||||
|
// is it.
|
||||||
|
try conn.request("GET", "/api/queries?limit=1", null, null);
|
||||||
|
const rows = try conn.receive(&body_buf);
|
||||||
|
try testing.expectEqual(@as(u16, 200), rows.status);
|
||||||
|
const page = try std.json.parseFromSliceLeaky(
|
||||||
|
handlers_queries.Page,
|
||||||
|
arena,
|
||||||
|
rows.body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
try testing.expectEqual(@as(usize, 1), page.queries.len);
|
||||||
|
try testing.expectEqualStrings(sweep_domain, page.queries[0].domain);
|
||||||
|
try testing.expectEqualStrings(sweep_redacted_upstream, page.queries[0].upstream);
|
||||||
|
try expectRedacted(rows.body);
|
||||||
|
|
||||||
|
// The detail body, read by the row id the database just handed over.
|
||||||
|
const one = try std.fmt.bufPrint(&target_buf, "/api/queries/{d}", .{row_id});
|
||||||
|
try conn.request("GET", one, null, null);
|
||||||
|
const detail_response = try conn.receive(&body_buf);
|
||||||
|
try testing.expectEqual(@as(u16, 200), detail_response.status);
|
||||||
|
const detail = try std.json.parseFromSliceLeaky(
|
||||||
|
provenance_view.QueryDetail,
|
||||||
|
arena,
|
||||||
|
detail_response.body,
|
||||||
|
.{ .ignore_unknown_fields = false },
|
||||||
|
);
|
||||||
|
try testing.expectEqualStrings(sweep_domain, detail.request.domain);
|
||||||
|
try testing.expectEqualStrings(sweep_redacted_upstream, detail.route.upstream);
|
||||||
|
try expectRedacted(detail_response.body);
|
||||||
|
}
|
||||||
|
|
||||||
|
test "W10 milestone 28: no query surface echoes a resolver credential" {
|
||||||
|
if (!build_options.integration) return error.SkipZigTest;
|
||||||
|
|
||||||
|
const gpa = testing.allocator;
|
||||||
|
var env = try Env.create(gpa, .{});
|
||||||
|
defer env.destroy();
|
||||||
|
const io = env.io();
|
||||||
|
|
||||||
|
var queue_buf: [4]logger_mod.Entry = undefined;
|
||||||
|
// Zero flush interval: the drain below is synchronous, and nothing else
|
||||||
|
// will ever put an entry on this queue for the writer to wait for.
|
||||||
|
var query_logger: logger_mod.Logger = .init(.{ .query_log_flush_interval_s = 0 }, &queue_buf);
|
||||||
|
var sink: query_sink.QuerySink = .init(&query_logger, env.hub);
|
||||||
|
|
||||||
|
var fake: FakeUpstream = .{ .identity = sweep_upstream_url };
|
||||||
|
var handler: dns_handler.Handler = .{
|
||||||
|
.upstream = fake.client(),
|
||||||
|
.blocking = .{ .mode = .zero, .ttl = 5 },
|
||||||
|
.forward_read_timeout = .{ .raw = .fromSeconds(2), .clock = .awake },
|
||||||
|
.manager = &env.mgr,
|
||||||
|
.pause = &env.pauser,
|
||||||
|
.sink = &sink,
|
||||||
|
};
|
||||||
|
|
||||||
|
try bounded(io, default_budget, credentialSweep, .{ io, env, &query_logger, &handler });
|
||||||
|
try testing.expectEqual(@as(u64, 1), fake.calls.load(.monotonic));
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// mutation → reload observed (ruling 12)
|
// mutation → reload observed (ruling 12)
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
@@ -1980,15 +2404,21 @@ fn queryFor(buf: []u8, id: u16, domain: []const u8, qtype: types.Type) []const u
|
|||||||
/// can tell whether the filter let the query through.
|
/// can tell whether the filter let the query through.
|
||||||
const FakeUpstream = struct {
|
const FakeUpstream = struct {
|
||||||
calls: std.atomic.Value(u64) = .init(0),
|
calls: std.atomic.Value(u64) = .init(0),
|
||||||
|
/// The resolver the handler reports as having answered. Operator-supplied
|
||||||
|
/// text in production, so the credential sweep points it at a url with a
|
||||||
|
/// token in its path.
|
||||||
|
identity: []const u8 = "fake://web-upstream",
|
||||||
|
|
||||||
fn exchangeFn(
|
fn exchangeFn(
|
||||||
ptr: *anyopaque,
|
ptr: *anyopaque,
|
||||||
io: std.Io,
|
io: std.Io,
|
||||||
query: []const u8,
|
query: []const u8,
|
||||||
response_buf: []u8,
|
response_buf: []u8,
|
||||||
|
selected: *?[]const u8,
|
||||||
) transport.ExchangeError![]u8 {
|
) transport.ExchangeError![]u8 {
|
||||||
_ = io;
|
_ = io;
|
||||||
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
|
||||||
|
selected.* = self.identity;
|
||||||
_ = self.calls.fetchAdd(1, .monotonic);
|
_ = self.calls.fetchAdd(1, .monotonic);
|
||||||
|
|
||||||
const request = packet.parse(query) catch return error.BadResponse;
|
const request = packet.parse(query) catch return error.BadResponse;
|
||||||
@@ -2348,6 +2778,272 @@ test "drift guard a bites: methods swapped between two documented paths fail the
|
|||||||
try testing.expectEqual(@as(usize, 2), swapped_routes);
|
try testing.expectEqual(@as(usize, 2), swapped_routes);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
// focused schema drift guards (milestone 28)
|
||||||
|
// ---------------------------------------------------------------------------
|
||||||
|
//
|
||||||
|
// Guard a proves every served route is documented and guard b counts the
|
||||||
|
// operations, and neither looks inside a schema. A field renamed, retyped, made
|
||||||
|
// nullable or dropped from `required` passes both while breaking every client
|
||||||
|
// that reads the document — and the query-log provenance shapes are exactly
|
||||||
|
// where a rename is easy and a wrong `nullable` is silent.
|
||||||
|
//
|
||||||
|
// So these read the schema back and hold it to the Zig struct that produces it:
|
||||||
|
// the same property names, the same types, the same nullability, the same
|
||||||
|
// requiredness, and no extra property on either side. A `$ref` recurses, so
|
||||||
|
// checking `QueryDetail` checks all six of its nested objects.
|
||||||
|
//
|
||||||
|
// The YAML reader below understands only the shape this document is written in
|
||||||
|
// — two-space indentation, schemas at four, properties at eight, inline `{ ... }`
|
||||||
|
// or an indented block, and single-line flow sequences. It is not a YAML parser
|
||||||
|
// and must not become one; a document it cannot read is a document that stopped
|
||||||
|
// matching the house style.
|
||||||
|
|
||||||
|
/// One schema's body: everything from its key line to the next schema key.
|
||||||
|
fn yamlSchema(schema_name: []const u8) ?[]const u8 {
|
||||||
|
var key_buf: [64]u8 = undefined;
|
||||||
|
const key = std.fmt.bufPrint(&key_buf, "\n {s}:\n", .{schema_name}) catch return null;
|
||||||
|
const at = std.mem.indexOf(u8, openapi.yaml, key) orelse return null;
|
||||||
|
const body = openapi.yaml[at + key.len ..];
|
||||||
|
|
||||||
|
var end: usize = 0;
|
||||||
|
var lines = std.mem.splitScalar(u8, body, '\n');
|
||||||
|
while (lines.next()) |line| {
|
||||||
|
if (line.len != 0 and !std.mem.startsWith(u8, line, " ")) break;
|
||||||
|
end += line.len + 1;
|
||||||
|
}
|
||||||
|
return body[0..@min(end, body.len)];
|
||||||
|
}
|
||||||
|
|
||||||
|
/// One property's definition: the rest of its line for the inline form, or the
|
||||||
|
/// indented block that follows it.
|
||||||
|
fn yamlProperty(schema: []const u8, property_name: []const u8) ?[]const u8 {
|
||||||
|
const properties_at = std.mem.indexOf(u8, schema, "\n properties:\n") orelse return null;
|
||||||
|
const properties = schema[properties_at..];
|
||||||
|
|
||||||
|
var key_buf: [64]u8 = undefined;
|
||||||
|
const key = std.fmt.bufPrint(&key_buf, "\n {s}:", .{property_name}) catch return null;
|
||||||
|
const at = std.mem.indexOf(u8, properties, key) orelse return null;
|
||||||
|
const rest = properties[at + key.len ..];
|
||||||
|
|
||||||
|
const line_end = std.mem.indexOfScalar(u8, rest, '\n') orelse rest.len;
|
||||||
|
if (std.mem.trim(u8, rest[0..line_end], " ").len != 0) return rest[0..line_end];
|
||||||
|
|
||||||
|
var end: usize = line_end + 1;
|
||||||
|
var lines = std.mem.splitScalar(u8, rest[line_end + 1 ..], '\n');
|
||||||
|
while (lines.next()) |line| {
|
||||||
|
if (line.len != 0 and !std.mem.startsWith(u8, line, " ")) break;
|
||||||
|
end += line.len + 1;
|
||||||
|
}
|
||||||
|
return rest[0..@min(end, rest.len)];
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The comma-separated items of a single-line flow sequence, `key: [a, b, c]`.
|
||||||
|
fn yamlFlowSeq(schema: []const u8, key: []const u8, out: *std.ArrayList([]const u8), gpa: Allocator) !void {
|
||||||
|
var key_buf: [32]u8 = undefined;
|
||||||
|
const needle = try std.fmt.bufPrint(&key_buf, "\n {s}: [", .{key});
|
||||||
|
const at = std.mem.indexOf(u8, schema, needle) orelse return error.TestUnexpectedResult;
|
||||||
|
const rest = schema[at + needle.len ..];
|
||||||
|
const close = std.mem.indexOfScalar(u8, rest, ']') orelse return error.TestUnexpectedResult;
|
||||||
|
|
||||||
|
var items = std.mem.splitScalar(u8, rest[0..close], ',');
|
||||||
|
while (items.next()) |item| try out.append(gpa, std.mem.trim(u8, item, " "));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The property names the schema declares, in document order.
|
||||||
|
fn yamlPropertyNames(schema: []const u8, out: *std.ArrayList([]const u8), gpa: Allocator) !void {
|
||||||
|
const properties_at = std.mem.indexOf(u8, schema, "\n properties:\n") orelse
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
var lines = std.mem.splitScalar(u8, schema[properties_at + 1 ..], '\n');
|
||||||
|
_ = lines.next();
|
||||||
|
while (lines.next()) |line| {
|
||||||
|
if (line.len != 0 and !std.mem.startsWith(u8, line, " ")) break;
|
||||||
|
if (!std.mem.startsWith(u8, line, " ") or std.mem.startsWith(u8, line, " ")) continue;
|
||||||
|
const colon = std.mem.indexOfScalar(u8, line, ':') orelse continue;
|
||||||
|
try out.append(gpa, line[8..colon]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The OpenAPI `type` a Zig field must be documented as, or `null` when the
|
||||||
|
/// field is a nested object and must be a `$ref` instead.
|
||||||
|
fn documentedType(comptime T: type) ?[]const u8 {
|
||||||
|
const Payload = switch (@typeInfo(T)) {
|
||||||
|
.optional => |o| o.child,
|
||||||
|
else => T,
|
||||||
|
};
|
||||||
|
return switch (@typeInfo(Payload)) {
|
||||||
|
.int => "integer",
|
||||||
|
.bool => "boolean",
|
||||||
|
// A closed enum is a string on the wire, documented as its own schema.
|
||||||
|
.@"enum" => null,
|
||||||
|
.pointer => "string",
|
||||||
|
.@"struct" => null,
|
||||||
|
else => @compileError("no documented type for " ++ @typeName(Payload)),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
fn isOptional(comptime T: type) bool {
|
||||||
|
return @typeInfo(T) == .optional;
|
||||||
|
}
|
||||||
|
|
||||||
|
/// The schema name a `$ref` property points at.
|
||||||
|
fn refTarget(property: []const u8) ?[]const u8 {
|
||||||
|
const marker = "$ref: \"#/components/schemas/";
|
||||||
|
const at = std.mem.indexOf(u8, property, marker) orelse return null;
|
||||||
|
const rest = property[at + marker.len ..];
|
||||||
|
const close = std.mem.indexOfScalar(u8, rest, '"') orelse return null;
|
||||||
|
return rest[0..close];
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Holds `schema_name` to `T`: same properties, same types, same nullability,
|
||||||
|
/// same requiredness, nothing extra on either side. Recurses through `$ref`.
|
||||||
|
fn expectSchemaMatches(gpa: Allocator, comptime T: type, schema_name: []const u8) !void {
|
||||||
|
const schema = yamlSchema(schema_name) orelse {
|
||||||
|
std.debug.print("openapi.yaml has no schema {s}\n", .{schema_name});
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
};
|
||||||
|
|
||||||
|
var required: std.ArrayList([]const u8) = .empty;
|
||||||
|
defer required.deinit(gpa);
|
||||||
|
try yamlFlowSeq(schema, "required", &required, gpa);
|
||||||
|
|
||||||
|
const fields = @typeInfo(T).@"struct".fields;
|
||||||
|
inline for (fields) |field| {
|
||||||
|
const property = yamlProperty(schema, field.name) orelse {
|
||||||
|
std.debug.print("{s}: no property {s}\n", .{ schema_name, field.name });
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
};
|
||||||
|
|
||||||
|
// Ahead of both branches: a `$ref` property is as free to go null as a
|
||||||
|
// scalar one, and a nested object or enum the server may omit is
|
||||||
|
// exactly the drift a client reading the document cannot see coming.
|
||||||
|
const documented_nullable = std.mem.containsAtLeast(u8, property, 1, "nullable: true");
|
||||||
|
if (documented_nullable != isOptional(field.type)) {
|
||||||
|
std.debug.print(
|
||||||
|
"{s}.{s}: nullable is {} in the document and {} in Zig\n",
|
||||||
|
.{ schema_name, field.name, documented_nullable, isOptional(field.type) },
|
||||||
|
);
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
}
|
||||||
|
|
||||||
|
if (comptime documentedType(field.type)) |wanted| {
|
||||||
|
var type_buf: [32]u8 = undefined;
|
||||||
|
const needle = try std.fmt.bufPrint(&type_buf, "type: {s}", .{wanted});
|
||||||
|
if (!std.mem.containsAtLeast(u8, property, 1, needle)) {
|
||||||
|
std.debug.print("{s}.{s}: not documented as {s}\n", .{ schema_name, field.name, wanted });
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
const target = refTarget(property) orelse {
|
||||||
|
std.debug.print("{s}.{s}: not a $ref\n", .{ schema_name, field.name });
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
};
|
||||||
|
const Payload = switch (@typeInfo(field.type)) {
|
||||||
|
.optional => |o| o.child,
|
||||||
|
else => field.type,
|
||||||
|
};
|
||||||
|
switch (@typeInfo(Payload)) {
|
||||||
|
.@"enum" => try expectEnumMatches(gpa, Payload, target),
|
||||||
|
else => try expectSchemaMatches(gpa, Payload, target),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var listed = false;
|
||||||
|
for (required.items) |listed_name| listed = listed or std.mem.eql(u8, listed_name, field.name);
|
||||||
|
if (!listed) {
|
||||||
|
std.debug.print("{s}.{s}: not in required\n", .{ schema_name, field.name });
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
var documented: std.ArrayList([]const u8) = .empty;
|
||||||
|
defer documented.deinit(gpa);
|
||||||
|
try yamlPropertyNames(schema, &documented, gpa);
|
||||||
|
try testing.expectEqual(fields.len, documented.items.len);
|
||||||
|
try testing.expectEqual(fields.len, required.items.len);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// Holds an enum schema to its Zig enum: the same values, in the same order.
|
||||||
|
fn expectEnumMatches(gpa: Allocator, comptime T: type, schema_name: []const u8) !void {
|
||||||
|
const schema = yamlSchema(schema_name) orelse {
|
||||||
|
std.debug.print("openapi.yaml has no schema {s}\n", .{schema_name});
|
||||||
|
return error.TestUnexpectedResult;
|
||||||
|
};
|
||||||
|
|
||||||
|
var values: std.ArrayList([]const u8) = .empty;
|
||||||
|
defer values.deinit(gpa);
|
||||||
|
try yamlFlowSeq(schema, "enum", &values, gpa);
|
||||||
|
|
||||||
|
const tags = @typeInfo(T).@"enum".fields;
|
||||||
|
try testing.expectEqual(tags.len, values.items.len);
|
||||||
|
inline for (tags, 0..) |tag, index| {
|
||||||
|
try testing.expectEqualStrings(tag.name, values.items[index]);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test "drift guard c: the query-log schemas match the structs that serialize them" {
|
||||||
|
const gpa = testing.allocator;
|
||||||
|
try expectSchemaMatches(gpa, queries_repo.QueryRow, "QueryRow");
|
||||||
|
try expectSchemaMatches(gpa, provenance_view.QueryDetail, "QueryDetail");
|
||||||
|
try expectSchemaMatches(gpa, provenance_view.Provenance, "Provenance");
|
||||||
|
try expectSchemaMatches(gpa, coverage_mod.Coverage, "Coverage");
|
||||||
|
}
|
||||||
|
|
||||||
|
test "drift guard c: the three closed enums are documented value for value" {
|
||||||
|
const gpa = testing.allocator;
|
||||||
|
try expectEnumMatches(gpa, provenance.PolicyAction, "PolicyAction");
|
||||||
|
try expectEnumMatches(gpa, provenance.PolicyReason, "PolicyReason");
|
||||||
|
try expectEnumMatches(gpa, provenance.RouteKind, "RouteKind");
|
||||||
|
}
|
||||||
|
|
||||||
|
test "drift guard c bites: a renamed, retyped or newly optional field fails it" {
|
||||||
|
const gpa = testing.allocator;
|
||||||
|
|
||||||
|
// A field the document does not name at all.
|
||||||
|
const Renamed = struct { complete: bool, available_from: i64 };
|
||||||
|
try testing.expectError(error.TestUnexpectedResult, expectSchemaMatches(gpa, Renamed, "Coverage"));
|
||||||
|
|
||||||
|
// A field the document names, with the wrong type.
|
||||||
|
const Retyped = struct { complete: bool, available_since: []const u8 };
|
||||||
|
try testing.expectError(error.TestUnexpectedResult, expectSchemaMatches(gpa, Retyped, "Coverage"));
|
||||||
|
|
||||||
|
// A field the document names and types correctly, but which Zig may now
|
||||||
|
// send as null while `nullable` is absent from the document.
|
||||||
|
const Nullable = struct { complete: bool, available_since: ?i64 };
|
||||||
|
try testing.expectError(error.TestUnexpectedResult, expectSchemaMatches(gpa, Nullable, "Coverage"));
|
||||||
|
|
||||||
|
// The same drift behind a `$ref`, where the property carries no `type:` of
|
||||||
|
// its own: a nested object the server may now omit.
|
||||||
|
const NullableObject = struct {
|
||||||
|
request: provenance_view.Request,
|
||||||
|
group: ?provenance_view.Group,
|
||||||
|
policy: provenance_view.Policy,
|
||||||
|
rewrites: provenance_view.Rewrites,
|
||||||
|
route: provenance_view.Route,
|
||||||
|
response: provenance_view.Response,
|
||||||
|
};
|
||||||
|
try testing.expectError(error.TestUnexpectedResult, expectSchemaMatches(gpa, NullableObject, "Provenance"));
|
||||||
|
|
||||||
|
// And behind a `$ref` to an enum, whose values would still line up.
|
||||||
|
const NullableEnum = struct {
|
||||||
|
action: ?provenance.PolicyAction,
|
||||||
|
reason: provenance.PolicyReason,
|
||||||
|
matched: []const u8,
|
||||||
|
source_id: ?i64,
|
||||||
|
source_name: []const u8,
|
||||||
|
};
|
||||||
|
try testing.expectError(error.TestUnexpectedResult, expectSchemaMatches(gpa, NullableEnum, "ProvenancePolicy"));
|
||||||
|
|
||||||
|
// A struct short one documented property, which excess-property checking on
|
||||||
|
// the client side would never catch.
|
||||||
|
const Narrowed = struct { complete: bool };
|
||||||
|
try testing.expectError(error.TestExpectedEqual, expectSchemaMatches(gpa, Narrowed, "Coverage"));
|
||||||
|
|
||||||
|
// An enum missing one of the document's values.
|
||||||
|
const Short = enum { not_evaluated, allow };
|
||||||
|
try testing.expectError(error.TestExpectedEqual, expectEnumMatches(gpa, Short, "PolicyAction"));
|
||||||
|
}
|
||||||
|
|
||||||
// ---------------------------------------------------------------------------
|
// ---------------------------------------------------------------------------
|
||||||
// contract samples: the frontend's consumed shapes against real responses
|
// contract samples: the frontend's consumed shapes against real responses
|
||||||
// (milestone-17 ruling 5)
|
// (milestone-17 ruling 5)
|
||||||
@@ -2443,6 +3139,10 @@ const contract_sample_walk = [_]ContractSample{
|
|||||||
// Query log and stats. `limit=5` reaches seeded row 21, the blocked one, so
|
// Query log and stats. `limit=5` reaches seeded row 21, the blocked one, so
|
||||||
// the page carries both the null-bearing and the populated row shape.
|
// the page carries both the null-bearing and the populated row shape.
|
||||||
.{ .name = "get_queries", .ts_type = "QueriesPage", .method = "GET", .target = "/api/queries?limit=5", .status = 200 },
|
.{ .name = "get_queries", .ts_type = "QueriesPage", .method = "GET", .target = "/api/queries?limit=5", .status = 200 },
|
||||||
|
// The newest seeded row: a CNAME-uncloaked block with a group, a source and
|
||||||
|
// a matched pattern, so the golden exercises every nested object rather
|
||||||
|
// than a row of nulls.
|
||||||
|
.{ .name = "get_query_detail", .ts_type = "QueryDetail", .method = "GET", .target = "/api/queries/27", .status = 200 },
|
||||||
.{ .name = "get_stats", .ts_type = "StatsTotals", .method = "GET", .target = "/api/stats?period=1h", .status = 200 },
|
.{ .name = "get_stats", .ts_type = "StatsTotals", .method = "GET", .target = "/api/stats?period=1h", .status = 200 },
|
||||||
.{ .name = "get_stats_timeseries", .ts_type = "StatsTimeseries", .method = "GET", .target = "/api/stats/timeseries?period=1h", .status = 200 },
|
.{ .name = "get_stats_timeseries", .ts_type = "StatsTimeseries", .method = "GET", .target = "/api/stats/timeseries?period=1h", .status = 200 },
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user