5 Commits
Author SHA1 Message Date
mokhtar fa323c7ed4 milestone 29: activity — history, live and policy simulation on one surface
Gates / test (push) Successful in 1m40s
Gates / package (push) Successful in 3m58s
Gates / container (push) Successful in 14s
CI / gates (push) Successful in 12m51s
Gates / frontend (push) Successful in 1m18s
Gates / test-aarch64 (push) Successful in 6m57s
query log, live and lookup merge into /activity. history filters live
in the url, so a pasted link or back/forward reproduces the exact
view; the result column separates servfail and nxdomain from success
in the list. live is follow-by-default with freeze, and a streamed
row opens its in-memory provenance detail — no correlation invented
for rows sqlite has not written. lookup survives as the current
policy simulation under /activity/test. investigation links carry
absolute bounds, and the diagnostics page now honors since/until
instead of ignoring them. the old routes are gone without aliases.
2026-08-22 10:52:56 +02:00
mokhtar 0fd6bbd312 milestone 28: query provenance — every logged query is exactly explainable
Gates / frontend (push) Successful in 1m36s
Gates / test (push) Successful in 1m56s
Gates / test-aarch64 (push) Successful in 7m37s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 13s
CI / gates (push) Successful in 19m4s
query rows gain qclass, rcode, group, policy action and reason, the
matched rule or list entry with its source, cname and safe-search
targets, route kind, forward zone, and the resolver that actually
answered — the pool and local markers die. servfails are logged and
name the resolver that lost; post-parse protocol refusals become rows.
a detail page at /queries/:id renders the ordered explanation, and
coverage watermarks distinguish an empty history from a missing one.

the schema fingerprint changes: existing query history is recreated
with the old file kept aside and the reset filed as a resolved
diagnostic. fixes an oversized udp reply being rebuilt as noerror,
which handed clients a truncated nxdomain as success.
2026-08-22 09:16:40 +02:00
mokhtar 7e6cb507d2 release cut: bump-kind justfile recipe and a compiled, tested cut tool
Gates / frontend (push) Successful in 1m11s
Gates / test (push) Successful in 1m38s
Gates / test-aarch64 (push) Successful in 6m31s
Gates / container (push) Successful in 9s
CI / gates (push) Successful in 26m52s
Gates / package (push) Successful in 5m27s
2026-08-21 23:34:13 +02:00
mokhtar 8a17e9ed21 admin: fix dashboard phantom scroll, drop last-failure column from upstream table
Gates / frontend (push) Successful in 1m11s
Gates / test (push) Successful in 1m40s
Gates / test-aarch64 (push) Successful in 6m38s
Gates / package (push) Successful in 5m42s
Gates / container (push) Successful in 17s
CI / gates (push) Successful in 27m54s
the chart's screen-reader table wore srOnly directly; overflow and
height do not apply to a table box, so it laid out 1200px tall below
the page while clip-path hid the paint. wrap it in a hidden div, which
clips properly and keeps the table role. failure detail is the
diagnostics page's job since milestone 27; the column and the now
dead formatAge go.
2026-08-21 23:33:32 +02:00
mokhtar 0107df5f99 spec: measurement window is the deployment side's call
Gates / frontend (push) Successful in 1m13s
Gates / test (push) Successful in 1m37s
Gates / test-aarch64 (push) Successful in 6m35s
Gates / package (push) Successful in 5m40s
Gates / container (push) Successful in 9s
CI / gates (push) Successful in 22m27s
2026-08-21 18:49:05 +02:00
103 changed files with 12499 additions and 1912 deletions
+27
View File
@@ -4,6 +4,33 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
## [Unreleased]
Query provenance: every logged query becomes exactly explainable — what the policy decided, what matched, where the answer came from and what the client saw. The handler records all of it as the reply goes out, `query_log` stores it, and a detail page reads one query back in the order the pipeline decided it. Read the upgrade note below first: it resets your query history.
### Added
- **Every logged query has a detail page.** A row in Activity now links to `/activity/queries/{id}`, which explains that one query in the order it was decided: the request, the group it was matched under, the policy verdict with the rule that produced it and the blocklist source that rule came from, any CNAME uncloaking or safe-search rewrite, the route the answer took — blocked, local, forward zone, upstream or cache — and what the client got back, RCODE and duration included. `GET /api/queries/{id}` serves the same object; an id that retention has already deleted is a 404. The live view carries the same provenance for the queries it streams, so a query is explainable as it happens as well as afterwards.
- **Query Log, Live and Lookup are one Activity page.** `/activity` is the single surface for what nxdns answered: History reads the stored log, Live reads the stream, and both show the same seven columns — Time, Domain, Client, Type, Result, Route, Duration. The mode and every filter live in the URL, so an investigation is one link that shows the recipient exactly what you were looking at, and an absolute time range stays that range instead of drifting as the day goes on. A new **Result** column says what the client actually got — `Blocked`, `NOERROR`, `SERVFAIL` and the rest — with the **Route** column beside it saying how the answer was produced, which is the pair the old Status column could not show: a blocked name is answered with NOERROR, and reading only the code made a block look like a success. Both unhappy cases are marked by weight and shape as well as colour. Switching between History and Live keeps your filters, and leaving Live closes the stream instead of holding a viewer slot open. A live row that the log has not written yet opens its own provenance in place — no invented row id — and the open detail stays put while the 500-row buffer scrolls past underneath it. Domain testing moves to `/activity/test` as **Current policy simulation**, worded so it can never be misread as an account of a query that already happened.
- **Diagnostics can be scoped to an absolute window.** `/diagnostics?since=…&until=…` now validates and applies both bounds to the active and resolved lists, and the page states the window it is showing with a way to clear it. A query's detail page links here with the five minutes either side of that query, which is where the underlying failure text for a SERVFAIL lives.
- **The query log and the dashboard say how far back the history goes.** `GET /api/queries`, `/api/stats` and `/api/stats/timeseries` each carry a `coverage` object: `available_since`, the first second the file can answer for, and `complete`, whether the window you asked for begins inside it. A period that starts before the query log does now says so instead of charting the missing part as zero — which is what a recreate, a retention pass or a fresh install would otherwise look like.
### Removed
- **`/queries`, `/queries/{id}`, `/live` and `/lookup` are gone, and bookmarks to them break.** There is no redirect and no alias: the paths simply stop resolving, and the app shows its not-found page. Everything those pages did is on `/activity`, `/activity/queries/{id}` and `/activity/test`. Three navigation entries collapse into one, "Activity". The API is untouched — `/api/queries`, `/api/queries/{id}`, `/api/queries/live` and `/api/lookup` all answer exactly as before.
- **The Status column, and the block reason on every row.** The reason a query was blocked was repeated on each of a hundred rows and pushed the answer the client saw off the table. Result and Route replace it; the exact rule, the blocklist source and the historical group stay one click away on the query's detail page, which is the only place they were ever readable.
### Changed
- **`GET /api/queries` rows changed shape.** Each row gains `qclass`, `rcode`, `policy_action`, `policy_reason` and `route_kind`, and `block_reason` is gone: the reason a query was blocked is now one of a closed set of values rather than a formatted string. No table column shows it — the reason is read on the query's detail page, and by an API client from `policy_reason` on the row. `blocked`, `cache_hit`, `upstream` and every other existing field are unchanged.
- **Upgrading resets your query history.** The `query_log` table gains the provenance columns below, and `querylog.db` is never migrated (it holds expendable log rows, so a schema change replaces the file instead of upgrading it). On the first start after the upgrade the old file is set aside as `querylog.db.schema-changed-<unix seconds>` and a fresh one is created. Nothing else is touched: `config.db` keeps your configuration and your diagnostics history. The recreate files a resolved `query_log.recreated` diagnostics entry naming the file that was kept and the timestamp the new history begins at, and a new `querylog_meta` table records that coverage start, so the dashboard can say "history is available from ..." instead of charting an empty range as zero. The set-aside file is a working SQLite database and can be deleted once you have decided you do not want it.
- **`logging.query_log_buffer_max` now accepts 1 to 37449, down from 1 to 1000000.** The queued entry carries every new provenance field by value and is about four times as wide as before — 1792 bytes against 432 — so the meaningful bound is bytes rather than entries. The ceiling is computed at compile time from the width of the entry so that the queue's worst case stays within 64 MiB, and it moves whenever that width does. The default of 10000 is unchanged and costs about 17 MiB. A configuration above the new ceiling is rejected at startup with the ceiling in the message.
- **Group and blocklist source names are now capped at 64 bytes.** Both are copied into every query-log row that mentions them, so an unbounded name was an unbounded cost per row. A longer name is rejected as `GroupNameTooLong` or `SourceNameTooLong`.
### Fixed
- **A UDP reply that has to be truncated keeps the answer's RCODE.** When an answer does not fit the client's UDP buffer, nxdns replaces it with an empty reply carrying the TC bit, which tells the client to retry over TCP. That replacement was always built as NOERROR, whatever the answer said — so an oversized NXDOMAIN reached the client as a success, and an EDNS extended RCODE above 15 lost the eight upper bits it needs an OPT record to carry. The truncated reply now carries the full twelve-bit code the answer had, split across the header and the reply's OPT record where the code needs it, and the query-log row records the code the client actually saw. The retry over TCP always returned the right RCODE; this was the UDP answer that preceded it.
## [0.0.8] - 2026-08-21
One constant, chosen from the 0.0.7 field numbers: the checkpoint cadence was the last first-order write cost on the Pi's SD card.
+2 -2
View File
@@ -87,8 +87,8 @@ test("429 login shows a ticking countdown and keeps submit disabled until it end
test("safeRedirect only allows same-origin absolute paths", () => {
expect(safeRedirect(undefined)).toBe("/");
expect(safeRedirect("/queries")).toBe("/queries");
expect(safeRedirect("/queries?x=1")).toBe("/queries?x=1");
expect(safeRedirect("/activity")).toBe("/activity");
expect(safeRedirect("/activity?x=1")).toBe("/activity?x=1");
expect(safeRedirect("//evil.example")).toBe("/");
expect(safeRedirect("https://evil.example")).toBe("/");
expect(safeRedirect("/\\evil.example")).toBe("/");
@@ -0,0 +1,312 @@
import { render, screen, waitFor, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import type { QueryDetail } from "@/lib/types";
import { provenance } from "@/features/queries/provenanceFixture";
function detail(id: number, sections: Parameters<typeof provenance>[0] = {}): QueryDetail {
return { id, ...provenance(sections) };
}
let responses: Record<string, unknown>;
beforeEach(() => {
responses = {
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
};
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const payload = responses[String(input)];
if (payload === undefined) {
return new Response(JSON.stringify({ error: "no such query" }), {
status: 404,
headers: { "content-type": "application/json" },
});
}
return new Response(JSON.stringify(payload), {
status: 200,
headers: { "content-type": "application/json" },
});
}),
);
});
afterEach(() => vi.unstubAllGlobals());
function renderDetail(id: number, search = "") {
const queryClient = createQueryClient();
const router = createAppRouter(
createMemoryHistory({ initialEntries: [`/activity/queries/${id}${search}`] }),
queryClient,
);
render(
<AuthProvider>
<QueryClientProvider client={queryClient}>
<RouterProvider router={router} />
</QueryClientProvider>
</AuthProvider>,
);
return router;
}
/** The search parameters a link carries, so an assertion states them by name. */
function hrefSearch(link: HTMLElement): Record<string, string> {
const query = link.getAttribute("href")?.split("?")[1] ?? "";
return Object.fromEntries(new URLSearchParams(query));
}
/** The value beside a term, so a section's facts are read as pairs. */
function factValue(label: string): string {
const term = screen.getByText(label);
const value = term.nextElementSibling;
return value?.textContent ?? "";
}
/** The line under the domain, which is what a page is read as at a glance. */
function subtitle(domain: string): string {
const heading = screen.getByRole("heading", { name: domain });
return heading.nextElementSibling?.textContent ?? "";
}
test("a blocked query explains itself in the six sections, in pipeline order", async () => {
responses["/api/queries/42"] = detail(42, {
request: { time: 1_700_000_000, domain: "ads.example", client: "192.0.2.11", qtype: 28, qclass: 1 },
group: { id: 3, name: "kids" },
policy: {
action: "block",
reason: "blocklist_wildcard",
matched: "||tracker.example^",
source_id: 5,
source_name: "StevenBlack",
},
rewrites: { cname_target: "cdn.tracker.example" },
route: { kind: "blocked", upstream: "" },
response: { rcode: 0, duration_us: 1234 },
});
renderDetail(42);
await screen.findByRole("heading", { name: "ads.example" });
const headings = screen.getAllByRole("heading", { level: 2 }).map((node) => node.textContent);
expect(headings).toEqual(["Request", "Group", "Policy", "Rewrites", "Route", "Response", "Related"]);
expect(factValue("Client")).toBe("192.0.2.11");
expect(factValue("Type")).toBe("AAAA");
expect(factValue("Class")).toBe("IN (1)");
expect(factValue("Name")).toBe("kids");
expect(factValue("Id")).toBe("3");
expect(factValue("Decision")).toBe("Blocked");
expect(factValue("Reason")).toBe("Blocklist (wildcard)");
expect(factValue("Matched")).toBe("||tracker.example^");
expect(factValue("Blocklist")).toBe("StevenBlack (#5)");
expect(factValue("CNAME target")).toBe("cdn.tracker.example");
expect(factValue("Answered by")).toBe("Blocked locally");
expect(factValue("Upstream")).toBe("No upstream exchange");
expect(factValue("Result")).toBe("NOERROR (0)");
expect(factValue("Took")).toBe("1.2 ms");
// NOERROR is the ordinary case and adds nothing to the verdict.
expect(subtitle("ads.example")).toMatch(/ — Blocked$/);
});
test("an upstream SERVFAIL names the resolver that failed and the code the client saw", async () => {
responses["/api/queries/7"] = detail(7, {
request: { domain: "news.example" },
route: { kind: "upstream", upstream: "https://dns.example/dns-query" },
response: { rcode: 2, duration_us: null },
});
renderDetail(7);
await screen.findByRole("heading", { name: "news.example" });
expect(factValue("Answered by")).toBe("Upstream resolver");
expect(factValue("Upstream")).toBe("https://dns.example/dns-query");
expect(factValue("Result")).toBe("SERVFAIL (2)");
expect(factValue("Took")).toBe("Not measured");
// The policy allowed the query; the client still got nothing, and the
// headline has to say so rather than reading as a success.
expect(subtitle("news.example")).toMatch(/ — Allowed — SERVFAIL \(2\)$/);
});
test("a forward-zone answer says the matcher never ran, not that nothing matched", async () => {
responses["/api/queries/14"] = detail(14, {
request: { domain: "nas.lan.home" },
policy: { action: "allow", reason: "forward_zone", matched: "" },
route: { kind: "forward_zone", forward_zone: "lan.home", upstream: "udp://192.168.1.1:53" },
});
renderDetail(14);
await screen.findByRole("heading", { name: "nas.lan.home" });
expect(factValue("Reason")).toBe("Forward zone");
expect(factValue("Matched")).toBe("The matcher never ran");
});
test("a query the matcher did evaluate keeps the honest empty verdict", async () => {
responses["/api/queries/15"] = detail(15, {
policy: { action: "allow", reason: "no_match", matched: "" },
});
renderDetail(15);
await screen.findByRole("heading", { name: "example.com" });
expect(factValue("Reason")).toBe("No match");
expect(factValue("Matched")).toBe("Nothing matched");
});
test("empty text fields read as absent facts, never as blank values", async () => {
responses["/api/queries/8"] = detail(8, {
group: { id: null, name: "" },
policy: { action: "not_evaluated", reason: "paused", matched: "", source_id: null, source_name: "" },
route: { kind: "upstream", forward_zone: "", upstream: "udp://9.9.9.9:53" },
});
renderDetail(8);
await screen.findByRole("heading", { name: "example.com" });
expect(factValue("Name")).toBe("No group recorded");
expect(factValue("Matched")).toBe("The matcher never ran");
expect(factValue("Blocklist")).toBe("Not a blocklist decision");
expect(factValue("Safe search")).toBe("No rewrite");
expect(factValue("Reason")).toBe("Filtering paused");
});
test("a log with hidden domains renders the server's marker, with nothing invented around it", async () => {
responses["/api/queries/9"] = detail(9, {
request: { domain: "hidden" },
policy: { action: "block", reason: "blocklist_domain", matched: "hidden", source_name: "StevenBlack" },
rewrites: { cname_target: "hidden", safe_search_target: "hidden" },
route: { kind: "blocked", upstream: "" },
});
renderDetail(9);
await screen.findByRole("heading", { name: "hidden" });
expect(factValue("Domain")).toBe("hidden");
expect(factValue("Matched")).toBe("hidden");
expect(factValue("CNAME target")).toBe("hidden");
expect(factValue("Safe search")).toBe("hidden");
// The client is governed by its own flag and stays visible here.
expect(factValue("Client")).toBe("192.0.2.10");
});
test("the related actions carry absolute bounds around the query, and the domain into the simulation", async () => {
responses["/api/queries/11"] = detail(11, {
request: { time: 1_700_000_000, domain: "shop.example", client: "192.0.2.12" },
});
renderDetail(11);
await screen.findByRole("heading", { name: "shop.example" });
const related = screen.getByRole("heading", { name: "Related" }).parentElement!;
expect(
within(related)
.getByRole("link", { name: /Test this domain/ })
.getAttribute("href"),
).toBe("/activity/test?domain=shop.example");
// No origin bound at all: five minutes either side of the query itself.
expect(hrefSearch(within(related).getByRole("link", { name: "All activity for this domain" }))).toEqual({
mode: "history",
domain: "shop.example",
since: "1699999700",
until: "1700000300",
});
expect(hrefSearch(within(related).getByRole("link", { name: "All activity from this client" }))).toEqual({
mode: "history",
client: "192.0.2.12",
since: "1699999700",
until: "1700000300",
});
// The diagnostics window is the query's own moment, never the origin's.
expect(hrefSearch(within(related).getByRole("link", { name: /Diagnostics around/ }))).toEqual({
since: "1699999700",
until: "1700000300",
});
});
test("an origin bound wins over the default window, one bound at a time", async () => {
responses["/api/queries/17"] = detail(17, {
request: { time: 1_700_000_000, domain: "shop.example", client: "192.0.2.12" },
});
renderDetail(17, "?mode=history&since=1600000000");
await screen.findByRole("heading", { name: "shop.example" });
const related = screen.getByRole("heading", { name: "Related" }).parentElement!;
const link = hrefSearch(within(related).getByRole("link", { name: "All activity for this domain" }));
expect(link["since"]).toBe("1600000000");
expect(link["until"]).toBe("1700000300");
});
test("both origin bounds carry through untouched", async () => {
responses["/api/queries/18"] = detail(18, { request: { time: 1_700_000_000, domain: "shop.example" } });
renderDetail(18, "?mode=history&since=1600000000&until=1600000060");
await screen.findByRole("heading", { name: "shop.example" });
const related = screen.getByRole("heading", { name: "Related" }).parentElement!;
const link = hrefSearch(within(related).getByRole("link", { name: "All activity for this domain" }));
expect(link["since"]).toBe("1600000000");
expect(link["until"]).toBe("1600000060");
});
test("the back link restores the investigation the reader came from", async () => {
responses["/api/queries/19"] = detail(19, { request: { domain: "shop.example" } });
renderDetail(19, "?mode=history&domain=shop&since=1600000000&blocked=true");
await screen.findByRole("heading", { name: "shop.example" });
expect(hrefSearch(screen.getByRole("link", { name: "← Activity" }))).toEqual({
mode: "history",
domain: "shop",
since: "1600000000",
blocked: "true",
});
});
function clientList(client: { ip: string; name: string; learned_name: string }) {
return {
clients: [
{
id: 1,
group_id: 1,
group: "default",
hand_edited: client.name !== "",
first_seen: 1_700_000_000,
last_seen: 1_700_000_100,
...client,
},
],
};
}
/** The related section, whose text is read whole because it is prose, not facts. */
async function relatedText(expected: string) {
const related = screen.getByRole("heading", { name: "Related" }).parentElement!;
await waitFor(() => expect(related.textContent?.replace(/\s+/g, " ")).toContain(expected));
}
test("the record keeps the address the query came from, and Related carries the name it has now", async () => {
responses["/api/queries/12"] = detail(12, { request: { domain: "shop.example", client: "192.0.2.12" } });
responses["/api/clients"] = clientList({ ip: "192.0.2.12", name: "Kids iPad", learned_name: "ipad.lan" });
renderDetail(12);
await screen.findByRole("heading", { name: "shop.example" });
await relatedText("The client list currently names 192.0.2.12 “Kids iPad”.");
expect(factValue("Client")).toBe("192.0.2.12");
});
test("a learned name is told as the reverse-DNS lookup it is, never as a recorded fact", async () => {
responses["/api/queries/13"] = detail(13, { request: { client: "192.0.2.13" } });
responses["/api/clients"] = clientList({ ip: "192.0.2.13", name: "", learned_name: "printer.lan" });
renderDetail(13);
await screen.findByRole("heading", { name: "example.com" });
await relatedText("Reverse DNS currently resolves 192.0.2.13 to printer.lan.");
expect(factValue("Client")).toBe("192.0.2.13");
});
test("a row retention has pruned explains the 404 and keeps the way back to the log", async () => {
renderDetail(404, "?mode=history&domain=gone");
await screen.findByRole("alert");
expect(screen.getByText(/no such query/)).toBeTruthy();
expect(hrefSearch(screen.getByRole("link", { name: "← Activity" }))).toEqual({
mode: "history",
domain: "gone",
});
});
@@ -0,0 +1,75 @@
import { useQuery } from "@tanstack/react-query";
import { Link, useParams, useSearch } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import InlineError from "@/lib/InlineError";
import { queryDetailQuery } from "@/lib/queries";
import type { QueryDetail } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import ProvenanceDetail from "./ProvenanceDetail";
import RelatedActions from "./RelatedActions";
import type { ActivitySearch } from "./search";
const styles = stylex.create({
back: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.primaryOnSurface,
textDecorationLine: "none",
},
loading: {
marginTop: "1rem",
color: colors.textMuted,
},
});
/**
* The way back to the investigation, not to a bare list. The originating
* Activity search rides in this route's own search, so the reader returns to
* the mode, the filters and the absolute window they left — a plain `/activity`
* would silently widen the range they had chosen.
*/
function BackLink({ origin }: { origin: ActivitySearch }) {
return (
<Link to="/activity" search={origin} {...stylex.props(styles.back, shared.focusRing)}>
Activity
</Link>
);
}
export default function ActivityDetailPage() {
const { id } = useParams({ from: "/shell/activity/queries/$id" });
const origin = useSearch({ from: "/shell/activity/queries/$id" });
const rowId = Number(id);
const { data, error, isPending, refetch } = useQuery(queryDetailQuery(rowId));
if (isPending) {
return (
<p {...stylex.props(styles.loading, shared.pulse)} role="status">
Loading query
</p>
);
}
if (data === undefined) {
return (
<section>
<BackLink origin={origin} />
<InlineError error={error} onRetry={() => void refetch()} />
</section>
);
}
const detail: QueryDetail = data;
const { domain, client, time } = detail.request;
return (
<section>
<BackLink origin={origin} />
<ProvenanceDetail
provenance={detail}
persistedId={detail.id}
relatedActions={<RelatedActions domain={domain} client={client} ts={time} origin={origin} />}
/>
</section>
);
}
@@ -0,0 +1,230 @@
/**
* The filter row over the Activity table.
*
* The applied state is the URL, never this form: what the reader sees is what
* the link they can paste to a housemate will show. So this holds a draft only,
* and the page remounts it whenever the applied search changes — a back button
* or a pasted URL has to move the form with it, and a form that seeded itself
* once would keep showing the previous investigation's filters.
*
* In live mode the row stays visible and disabled rather than disappearing: the
* filters are retained in the URL and apply again the moment history comes
* back, and hiding them would read as having lost them. The stream itself is
* unfiltered — the server sends every query — so a row that looked usable here
* would promise filtering that is not happening.
*/
import { useState, type FormEvent } from "react";
import * as stylex from "@stylexjs/stylex";
import Select from "@/ui/Select";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { datetimeField, editDatetimeField, resolveDatetimeField, type DatetimeField } from "./datetime";
import type { ActivitySearch } from "./search";
const STATUS_OPTIONS = [
{ value: "any", label: "All" },
{ value: "blocked", label: "Blocked only" },
{ value: "allowed", label: "Allowed only" },
];
const styles = stylex.create({
/** One column on a phone, two from `sm`, five from `lg`. */
grid: {
marginTop: "1rem",
display: "grid",
gap: "0.75rem",
gridTemplateColumns: {
default: "repeat(1, minmax(0, 1fr))",
"@media (min-width: 640px)": "repeat(2, minmax(0, 1fr))",
"@media (min-width: 1024px)": "repeat(5, minmax(0, 1fr))",
},
},
label: {
display: "block",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
input: {
marginTop: "0.25rem",
width: "100%",
// A disabled native input keeps its value legible but reads as inert,
// matching what RAC does to the Select trigger beside it.
cursor: { default: null, ":disabled": "not-allowed" },
opacity: { default: null, ":disabled": 0.55 },
},
buttonRow: {
display: "flex",
alignItems: "flex-end",
gap: "0.5rem",
gridColumn: {
default: null,
"@media (min-width: 640px)": "span 2 / span 2",
"@media (min-width: 1024px)": "span 5 / span 5",
},
},
toolbarButton: {
fontWeight: 500,
},
error: {
marginTop: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.dangerText,
},
});
/** The applied filters, with `mode` left to the page that owns the switch. */
export type AppliedFilters = Omit<ActivitySearch, "mode">;
/** Every filter off — what Clear applies, and the loader's empty-filter case. */
export const NO_FILTERS: AppliedFilters = {
domain: undefined,
client: undefined,
blocked: undefined,
since: undefined,
until: undefined,
};
function blockedOption(blocked: boolean | undefined): string {
if (blocked === undefined) return "any";
return blocked ? "blocked" : "allowed";
}
function optionBlocked(value: string): boolean | undefined {
if (value === "blocked") return true;
return value === "allowed" ? false : undefined;
}
function boundError(label: string, reason: "unparseable" | "nonexistent"): string {
return reason === "unparseable"
? `${label} is not a complete date and time.`
: `${label} names a local time that does not exist — the clock jumps over it for daylight saving.`;
}
interface Props {
applied: AppliedFilters;
isDisabled: boolean;
onApply: (filters: AppliedFilters) => void;
onClear: () => void;
}
export default function ActivityFilters({ applied, isDisabled, onApply, onClear }: Props) {
const [domain, setDomain] = useState(applied.domain ?? "");
const [client, setClient] = useState(applied.client ?? "");
const [blocked, setBlocked] = useState(blockedOption(applied.blocked));
const [since, setSince] = useState<DatetimeField>(() => datetimeField(applied.since));
const [until, setUntil] = useState<DatetimeField>(() => datetimeField(applied.until));
const [error, setError] = useState<string | null>(null);
function submit(event: FormEvent) {
event.preventDefault();
const sinceValue = resolveDatetimeField(since);
if (!sinceValue.ok) {
setError(boundError("Since", sinceValue.reason));
return;
}
const untilValue = resolveDatetimeField(until);
if (!untilValue.ok) {
setError(boundError("Until", untilValue.reason));
return;
}
setError(null);
onApply({
domain: domain.trim() === "" ? undefined : domain.trim(),
client: client.trim() === "" ? undefined : client.trim(),
blocked: optionBlocked(blocked),
since: sinceValue.value,
until: untilValue.value,
});
}
function clear() {
setDomain("");
setClient("");
setBlocked("any");
setSince(datetimeField(undefined));
setUntil(datetimeField(undefined));
setError(null);
onClear();
}
return (
<>
<form onSubmit={submit} {...stylex.props(styles.grid)}>
<label {...stylex.props(styles.label)}>
Domain contains
<input
type="text"
value={domain}
disabled={isDisabled}
onChange={(event) => setDomain(event.target.value)}
{...stylex.props(shared.smallInput, styles.input, shared.focusRing)}
/>
</label>
<label {...stylex.props(styles.label)}>
Client (exact)
<input
type="text"
value={client}
disabled={isDisabled}
onChange={(event) => setClient(event.target.value)}
{...stylex.props(shared.smallInput, styles.input, shared.focusRing)}
/>
</label>
<Select
variant="compactField"
label="Result"
value={blocked}
isDisabled={isDisabled}
onChange={setBlocked}
options={STATUS_OPTIONS}
/>
<label {...stylex.props(styles.label)}>
Since
<input
type="datetime-local"
step={1}
value={since.text}
disabled={isDisabled}
onChange={(event) => setSince(editDatetimeField(since, event.target.value))}
{...stylex.props(shared.smallInput, styles.input, shared.focusRing)}
/>
</label>
<label {...stylex.props(styles.label)}>
Until
<input
type="datetime-local"
step={1}
value={until.text}
disabled={isDisabled}
onChange={(event) => setUntil(editDatetimeField(until, event.target.value))}
{...stylex.props(shared.smallInput, styles.input, shared.focusRing)}
/>
</label>
<div {...stylex.props(styles.buttonRow)}>
<button
type="submit"
disabled={isDisabled}
{...stylex.props(shared.button, styles.toolbarButton, shared.focusRing)}
>
Apply filters
</button>
<button
type="button"
onClick={clear}
disabled={isDisabled}
{...stylex.props(shared.button, styles.toolbarButton, shared.focusRing)}
>
Clear
</button>
</div>
</form>
{error !== null && (
<p role="alert" {...stylex.props(styles.error)}>
{error}
</p>
)}
</>
);
}
@@ -0,0 +1,554 @@
/**
* Activity in history mode, through the real router: the URL is the applied
* state, so nothing here can be checked by rendering the page on its own.
*/
import { act, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import type { Client, Coverage, QueriesPage, QueryRow } from "@/lib/types";
import { queryRow } from "@/features/queries/provenanceFixture";
function client(id: number, ip: string, name: string, learnedName: string): Client {
return {
id,
ip,
name,
learned_name: learnedName,
group_id: 1,
group: "default",
hand_edited: name !== "",
first_seen: 1_700_000_000,
last_seen: 1_700_000_100,
};
}
const CLIENTS: Client[] = [
client(1, "192.0.2.10", "Kitchen Pi", "pi.lan"),
client(2, "192.0.2.11", "", "laptop.lan"),
client(3, "192.0.2.12", "", ""),
];
function row(id: number, domain: string, overrides: Partial<QueryRow> = {}): QueryRow {
return queryRow(id, { ts: 1_700_000_000 + id, domain, upstream: "udp://9.9.9.9:53", ...overrides });
}
const COMPLETE: Coverage = { complete: true, available_since: 1_600_000_000 };
/** The blocked row every page fixture reuses. */
const BLOCKED = {
blocked: true,
policy_action: "block",
policy_reason: "blocklist_wildcard",
route_kind: "blocked",
upstream: "",
} as const satisfies Partial<QueryRow>;
const PAGES: Record<string, QueriesPage> = {
"/api/queries": {
queries: [
row(20, "first.example", { qtype: 65, cache_hit: true, route_kind: "cache" }),
row(19, "ads.example", { ...BLOCKED, response_time_us: null, cache_hit: null }),
],
next_before: 19,
coverage: COMPLETE,
},
"/api/queries?before=19": {
queries: [row(5, "older.example")],
next_before: null,
coverage: COMPLETE,
},
"/api/queries?domain=ads": {
queries: [row(19, "ads.example", BLOCKED)],
next_before: null,
coverage: COMPLETE,
},
"/api/queries?domain=ads&blocked=true": {
queries: [row(19, "ads.example", BLOCKED)],
next_before: null,
coverage: COMPLETE,
},
"/api/queries?since=1700000000": {
queries: [row(20, "first.example")],
next_before: null,
coverage: COMPLETE,
},
};
let fetchMock: ReturnType<typeof vi.fn>;
function json(payload: unknown): Response {
return new Response(JSON.stringify(payload), { status: 200, headers: { "content-type": "application/json" } });
}
const VERSION = { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 };
/** The shell's own requests, which every test serves the same way. */
function stubFetch(handler: (url: string) => Response | Promise<Response>) {
fetchMock = vi.fn((input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/version") return Promise.resolve(json(VERSION));
return Promise.resolve(handler(url));
});
vi.stubGlobal("fetch", fetchMock);
}
function fromPages(url: string): Response {
if (url === "/api/clients") return json({ clients: CLIENTS });
const payload = PAGES[url];
if (payload === undefined) return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return json(payload);
}
beforeEach(() => {
stubFetch(fromPages);
});
afterEach(() => {
vi.unstubAllGlobals();
});
function renderPage(path = "/activity") {
const queryClient = createQueryClient();
const history = createMemoryHistory({ initialEntries: [path] });
const router = createAppRouter(history, queryClient);
render(
<AuthProvider>
<QueryClientProvider client={queryClient}>
<RouterProvider router={router} />
</QueryClientProvider>
</AuthProvider>,
);
return { queryClient, history };
}
/** Every `/api/queries` URL the run asked for, list pages only. */
function queryCalls(): string[] {
return fetchMock.mock.calls
.map((call) => String(call[0]))
.filter((url) => url === "/api/queries" || url.startsWith("/api/queries?"));
}
test("renders the first page with the seven columns filled in", async () => {
renderPage();
await screen.findByText("first.example");
expect(screen.getAllByRole("columnheader").map((header) => header.textContent)).toEqual([
"Time",
"Domain",
"Client",
"Type",
"Result",
"Route",
"Duration",
]);
const first = screen.getByText("first.example").closest("tr")!;
expect(within(first).getByText("HTTPS")).toBeTruthy();
expect(within(first).getByText("NOERROR")).toBeTruthy();
expect(within(first).getByText("Cache")).toBeTruthy();
expect(within(first).getByText("1.2 ms")).toBeTruthy();
const blocked = screen.getByText("ads.example").closest("tr")!;
// The Result cell says Blocked even though the client saw NOERROR, and the
// Route cell says how: this is the pair the old Status column could not show.
expect(within(blocked).getAllByText("Blocked")).toHaveLength(2);
expect(within(blocked).getByText("—")).toBeTruthy();
expect(screen.getByText(/Showing 2 queries/)).toBeTruthy();
});
test("resolves each row's client to its display name, keeping the IP as the tooltip", async () => {
stubFetch((url) => {
if (url === "/api/clients") return json({ clients: CLIENTS });
if (url !== "/api/queries") return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return json({
queries: [
row(20, "named.example", { client_ip: "192.0.2.10" }),
row(19, "learned.example", { client_ip: "192.0.2.11" }),
row(18, "nameless.example", { client_ip: "192.0.2.12" }),
row(17, "stranger.example", { client_ip: "192.0.2.99" }),
],
next_before: null,
coverage: COMPLETE,
} satisfies QueriesPage);
});
renderPage();
// A hand-typed name wins outright; the learned name never surfaces for it.
const named = await screen.findByText("Kitchen Pi");
expect(named.getAttribute("title")).toBe("192.0.2.10");
expect(screen.queryByText("pi.lan")).toBeNull();
// A learned name reads muted and nothing more here: the "learned" tag would
// repeat on every row of the table, so the Clients page carries it instead.
const learned = screen.getByText("laptop.lan");
expect(learned.getAttribute("title")).toBe("192.0.2.11");
expect(within(learned.closest("tr")!).queryByText("learned")).toBeNull();
// A known client with neither name, and a client the loaded list has never
// seen, both fall back to the bare address with no tooltip standing in.
expect(screen.getByText("192.0.2.12").getAttribute("title")).toBeNull();
expect(screen.getByText("192.0.2.99").getAttribute("title")).toBeNull();
});
test("load more appends the next page and stops at the end of the log", async () => {
renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await screen.findByText("older.example");
expect(screen.getByText("first.example")).toBeTruthy();
expect(screen.getByText(/Showing 3 queries — end of log/)).toBeTruthy();
expect(screen.queryByRole("button", { name: "Load more" })).toBeNull();
});
test("applying a filter puts it in the url, refetches, and resets the accumulated list", async () => {
const { history } = renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await screen.findByText("older.example");
fireEvent.change(screen.getByLabelText("Domain contains"), { target: { value: "ads" } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
await screen.findByText(/Showing 1 query /);
expect(history.location.search).toContain("domain=ads");
expect(screen.getByText("ads.example")).toBeTruthy();
expect(screen.queryByText("first.example")).toBeNull();
expect(screen.queryByText("older.example")).toBeNull();
});
test("a load-more that resolves after a filter change is discarded", async () => {
let releaseLoadMore: () => void = () => {};
stubFetch((url) => {
if (url === "/api/queries?before=19") {
return new Promise<Response>((resolve) => {
releaseLoadMore = () => resolve(json(PAGES["/api/queries?before=19"]));
});
}
return fromPages(url);
});
renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
fireEvent.change(screen.getByLabelText("Domain contains"), { target: { value: "ads" } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
await screen.findByText(/Showing 1 query /);
releaseLoadMore();
await act(async () => {
await new Promise((resolve) => setTimeout(resolve, 0));
});
expect(screen.queryByText("older.example")).toBeNull();
expect(screen.getByText(/Showing 1 query /)).toBeTruthy();
expect(screen.queryByRole("alert")).toBeNull();
});
test("load more is disabled while a filter change shows placeholder data, then uses the fresh cursor", async () => {
let releaseFiltered: () => void = () => {};
const filteredPage: QueriesPage = {
queries: [row(19, "ads.example", BLOCKED)],
next_before: 7,
coverage: COMPLETE,
};
const filteredOlderPage: QueriesPage = {
queries: [row(3, "ads.older.example")],
next_before: null,
coverage: COMPLETE,
};
stubFetch((url) => {
if (url === "/api/queries?domain=ads") {
return new Promise<Response>((resolve) => {
releaseFiltered = () => resolve(json(filteredPage));
});
}
if (url === "/api/queries?domain=ads&before=7") return json(filteredOlderPage);
return fromPages(url);
});
renderPage();
await screen.findByText("first.example");
fireEvent.change(screen.getByLabelText("Domain contains"), { target: { value: "ads" } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
const staleButton = await screen.findByRole("button", { name: "Load more" });
expect(staleButton).toHaveProperty("disabled", true);
fireEvent.click(staleButton);
expect(queryCalls()).not.toContain("/api/queries?domain=ads&before=19");
releaseFiltered();
await waitFor(() => {
expect(screen.queryByText("first.example")).toBeNull();
});
const freshButton = screen.getByRole("button", { name: "Load more" });
expect(freshButton).toHaveProperty("disabled", false);
fireEvent.click(freshButton);
await screen.findByText("ads.older.example");
expect(queryCalls()).toContain("/api/queries?domain=ads&before=7");
expect(screen.getByText(/Showing 2 queries — end of log/)).toBeTruthy();
});
test("a background refetch after new rows arrive leaves no gap between the loaded pages", async () => {
// The newest-100 window moves up while the reader has a second page open.
// Refetching only the first page would drop n20 and n19 out of the middle
// of the table; the second page must be replayed from the fresh cursor.
const before: Record<string, QueriesPage> = {
"/api/queries": {
queries: [row(20, "n20.example"), row(19, "n19.example")],
next_before: 19,
coverage: COMPLETE,
},
"/api/queries?before=19": {
queries: [row(18, "n18.example"), row(17, "n17.example")],
next_before: null,
coverage: COMPLETE,
},
};
const after: Record<string, QueriesPage> = {
"/api/queries": {
queries: [row(22, "n22.example"), row(21, "n21.example")],
next_before: 21,
coverage: COMPLETE,
},
"/api/queries?before=21": {
queries: [row(20, "n20.example"), row(19, "n19.example"), row(18, "n18.example"), row(17, "n17.example")],
next_before: null,
coverage: COMPLETE,
},
};
let live = before;
stubFetch((url) => {
const payload = live[url];
if (payload === undefined) return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return json(payload);
});
const { queryClient } = renderPage();
await screen.findByText("n20.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await screen.findByText("n17.example");
live = after;
await act(async () => {
await queryClient.invalidateQueries({ queryKey: ["queries"] });
});
await screen.findByText("n22.example");
const shown = screen.getAllByText(/^n\d+\.example$/).map((cell) => cell.textContent);
expect(shown).toEqual(["n22.example", "n21.example", "n20.example", "n19.example", "n18.example", "n17.example"]);
expect(screen.getByText(/Showing 6 queries — end of log/)).toBeTruthy();
});
test("a 401 on load more routes through handleUnauthorized instead of the inline error", async () => {
const assign = vi.fn();
vi.stubGlobal("location", { pathname: "/activity", search: "", assign });
stubFetch((url) => {
if (url === "/api/queries?before=19") {
return new Response(JSON.stringify({ error: "unauthorized" }), {
status: 401,
headers: { "content-type": "application/json" },
});
}
return fromPages(url);
});
renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await waitFor(() => {
expect(assign).toHaveBeenCalledWith(`/login?redirect=${encodeURIComponent("/activity")}`);
});
expect(screen.queryByRole("alert")).toBeNull();
expect(screen.queryByText(/Failed to load more/)).toBeNull();
});
test("each row links into its own detail page, carrying the investigation with it", async () => {
renderPage("/activity?mode=history&since=1700000000");
await screen.findByText("first.example");
const link = screen.getByRole("link", { name: "first.example" });
expect(link.getAttribute("href")).toContain("/activity/queries/20");
expect(link.getAttribute("href")).toContain("since=1700000000");
// An <a href> is in the tab order by default; nothing here may opt it out.
expect(link.getAttribute("tabindex")).toBeNull();
});
test("a pruned window tells the reader when history starts", async () => {
stubFetch((url) => {
if (url === "/api/clients") return json({ clients: CLIENTS });
if (url !== "/api/queries") return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return json({
queries: [row(20, "kept.example")],
next_before: null,
coverage: { complete: false, available_since: 1_700_000_000 },
} satisfies QueriesPage);
});
renderPage();
await screen.findByText("kept.example");
expect(screen.getByText(/Query history is available from/)).toBeTruthy();
});
test("a complete window shows no coverage notice", async () => {
renderPage();
await screen.findByText("first.example");
expect(screen.queryByText(/Query history is available from/)).toBeNull();
});
test("a ?domain= link seeds the filter form and fetches that domain on arrival", async () => {
renderPage("/activity?domain=ads");
await screen.findByText("ads.example");
expect(screen.getByLabelText("Domain contains")).toHaveProperty("value", "ads");
expect(screen.queryByText("first.example")).toBeNull();
});
test("history forwards exactly the six normalized filter fields and nothing else", async () => {
stubFetch((url) => {
if (url === "/api/clients") return json({ clients: CLIENTS });
return json({ queries: [], next_before: null, coverage: COMPLETE } satisfies QueriesPage);
});
renderPage(
"/activity?mode=history&domain=%20ads%20&client=192.0.2.5&blocked=true&since=1700000000&until=1700000600&bogus=1&limit=9999",
);
await screen.findByText("No queries match the current filters.");
expect(queryCalls()).toEqual([
"/api/queries?domain=ads&client=192.0.2.5&blocked=true&since=1700000000&until=1700000600",
]);
});
test("a rejected search parameter is dropped rather than guessed at", async () => {
renderPage("/activity?since=1.5&blocked=%22true%22&domain=%20%20");
await screen.findByText("first.example");
// Nothing survived validation, so the request is the unfiltered one.
expect(queryCalls()).toEqual(["/api/queries"]);
expect(screen.getByLabelText("Domain contains")).toHaveProperty("value", "");
});
test("the form draft follows the url back and forward, seconds included", async () => {
stubFetch((url) => {
if (url === "/api/clients") return json({ clients: CLIENTS });
return json({ queries: [], next_before: null, coverage: COMPLETE } satisfies QueriesPage);
});
// A bound with non-zero seconds: the round trip has to keep them, and the
// untouched field has to carry the original number rather than re-parse.
const seeded = 1_700_000_017;
const { history } = renderPage(`/activity?mode=history&domain=first&since=${seeded}`);
const domainInput = await screen.findByLabelText("Domain contains");
expect(domainInput).toHaveProperty("value", "first");
const sinceInput = screen.getByLabelText("Since") as HTMLInputElement;
expect(sinceInput.value).toContain(":37");
fireEvent.change(domainInput, { target: { value: "second" } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
await waitFor(() => expect(history.location.search).toContain("domain=second"));
// The untouched Since bound applied as the exact second it was seeded with.
expect(queryCalls()).toContain(`/api/queries?domain=second&since=${seeded}`);
act(() => history.back());
await waitFor(() => {
expect(screen.getByLabelText("Domain contains")).toHaveProperty("value", "first");
});
act(() => history.forward());
await waitFor(() => {
expect(screen.getByLabelText("Domain contains")).toHaveProperty("value", "second");
});
});
/** 2 a.m. on the EU spring-forward date: an hour that exists in some zones and not others. */
const DST_WALL_TIME = "2026-03-29T02:30:00";
/**
* Whether that wall time names an instant in the timezone the suite runs in.
* `new Date` slides a spring-forward gap silently forward, so an hour or minute
* that comes back different from the one written *is* the gap.
*/
function wallTimeExists(text: string): boolean {
const written = /T(\d{2}):(\d{2})/.exec(text)!;
const parsed = new Date(text);
return parsed.getHours() === Number(written[1]) && parsed.getMinutes() === Number(written[2]);
}
test("a wall-clock time the daylight-saving jump skips is refused, not silently moved", async () => {
// One expected outcome per timezone, decided here rather than accepted from
// the page: in a zone with the jump the bound must be refused outright, and
// in a zone without it the same text is an ordinary instant that applies.
const inGap = !wallTimeExists(DST_WALL_TIME);
const unix = Math.floor(new Date(DST_WALL_TIME).getTime() / 1000);
stubFetch((url) => {
if (url === `/api/queries?since=${unix}`) {
return json({ queries: [], next_before: null, coverage: COMPLETE } satisfies QueriesPage);
}
return fromPages(url);
});
const { history } = renderPage();
await screen.findByText("first.example");
const callsBefore = queryCalls().length;
const searchBefore = history.location.search;
fireEvent.change(screen.getByLabelText("Since"), { target: { value: DST_WALL_TIME } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
if (inGap) {
expect(screen.getByRole("alert").textContent).toContain("daylight saving");
// Refused means refused: no navigation, and no request for the hour the
// operator did not ask for.
expect(history.location.search).toBe(searchBefore);
expect(queryCalls()).toHaveLength(callsBefore);
return;
}
await waitFor(() => expect(history.location.search).toContain(`since=${unix}`));
expect(queryCalls()).toContain(`/api/queries?since=${unix}`);
expect(screen.queryByRole("alert")).toBeNull();
});
test("the policy simulation is reachable from the header, with no rows to click through", async () => {
stubFetch((url) => {
if (url === "/api/clients") return json({ clients: CLIENTS });
if (url === "/api/groups") return json({ groups: [{ id: 1, name: "default", safe_search: false }] });
return json({ queries: [], next_before: null, coverage: COMPLETE } satisfies QueriesPage);
});
const { history } = renderPage();
// An empty log is exactly the case a row-borne link cannot serve.
await screen.findByText("No queries logged yet.");
const link = screen.getByRole("link", { name: "Current policy simulation" });
expect(link.getAttribute("href")).toBe("/activity/test");
fireEvent.click(link);
await screen.findByRole("heading", { level: 1, name: "Current policy simulation" });
expect(history.location.pathname).toBe("/activity/test");
});
test("Clear empties the url as well as the form", async () => {
const { history } = renderPage("/activity?mode=history&domain=ads&blocked=true");
await screen.findByText("ads.example");
fireEvent.click(screen.getByRole("button", { name: "Clear" }));
await waitFor(() => {
expect(history.location.search).not.toContain("domain");
});
expect(history.location.search).not.toContain("blocked");
expect(screen.getByLabelText("Domain contains")).toHaveProperty("value", "");
});
@@ -0,0 +1,152 @@
/**
* Activity: one surface over the queries nxdns answered, in two modes.
*
* History reads the persisted log and Live reads the stream, but they are the
* same seven columns over the same filters, and the reader moves between them
* without losing the question they were asking. The mode lives in the URL with
* the filters, so an investigation is one link — including which half of it the
* recipient should be looking at.
*/
import { Link, useNavigate, useSearch } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import ActivityFilters, { NO_FILTERS, type AppliedFilters } from "./ActivityFilters";
import HistoryActivity from "./HistoryActivity";
import LiveActivity from "./LiveActivity";
import type { ActivityMode } from "./search";
const MODES: ReadonlyArray<{ mode: ActivityMode; label: string }> = [
{ mode: "history", label: "History" },
{ mode: "live", label: "Live" },
];
const styles = stylex.create({
header: {
display: "flex",
flexWrap: "wrap",
alignItems: "center",
gap: "0.75rem",
},
heading: {
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
},
switch: {
display: "flex",
gap: "0.25rem",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
padding: "0.125rem",
},
modeButton: {
borderStyle: "none",
borderRadius: "0.1875rem",
paddingInline: "0.75rem",
paddingBlock: "0.25rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
fontWeight: 500,
cursor: "pointer",
},
modeIdle: {
backgroundColor: { default: "transparent", ":hover": colors.surfaceHover },
color: { default: colors.textSecondary, ":hover": colors.text },
},
/** The selected mode reads as a filled chip, the same weight the nav uses. */
modeSelected: {
backgroundColor: colors.primary,
color: colors.primaryText,
},
/** The one way into the simulation from here, so it cannot sit behind a row. */
simulationLink: {
marginInlineStart: "auto",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.primaryOnSurface,
textDecorationLine: "none",
},
liveNote: {
marginTop: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
});
export default function ActivityPage() {
const search = useSearch({ from: "/shell/activity" });
const navigate = useNavigate({ from: "/activity" });
const live = search.mode === "live";
// The functional form, not a replacement object: the filters are retained
// across a mode switch on purpose, and spelling out a new search here would
// drop every one of them on the way to Live and back.
function selectMode(mode: ActivityMode) {
if (mode === search.mode) return;
void navigate({ search: (prev) => ({ ...prev, mode }) });
}
function apply(filters: AppliedFilters) {
void navigate({ search: { mode: search.mode, ...filters } });
}
return (
<section>
<div {...stylex.props(styles.header)}>
<h1 {...stylex.props(styles.heading)}>Activity</h1>
<div role="group" aria-label="Activity mode" {...stylex.props(styles.switch)}>
{MODES.map((option) => {
const selected = option.mode === search.mode;
return (
<button
key={option.mode}
type="button"
aria-pressed={selected}
onClick={() => selectMode(option.mode)}
{...stylex.props(
styles.modeButton,
selected ? styles.modeSelected : styles.modeIdle,
shared.focusRing,
)}
>
{option.label}
</button>
);
})}
</div>
<Link to="/activity/test" {...stylex.props(styles.simulationLink, shared.focusRing)}>
Current policy simulation
</Link>
</div>
{/*
* Remounted whenever the applied search changes, which is what makes
* the back button work: the draft is derived state, and the browser
* moving the URL under it has to move the form with it.
*/}
<ActivityFilters
key={`${search.domain ?? ""}|${search.client ?? ""}|${String(search.blocked)}|${String(search.since)}|${String(search.until)}`}
applied={search}
isDisabled={live}
onApply={apply}
onClear={() => apply(NO_FILTERS)}
/>
{live ? (
<>
<p {...stylex.props(styles.liveNote)}>
The stream carries every query the server answers; these filters apply to history only.
</p>
<LiveActivity origin={search} />
</>
) : (
<HistoryActivity search={search} />
)}
</section>
);
}
@@ -0,0 +1,181 @@
/**
* Activity in history mode: the persisted queries the URL's filters select,
* paged by keyset cursor.
*
* The filters arrive already applied — the URL is the applied state — so this
* only reads them. Everything about how the reader got here lives one level up.
*/
import { useInfiniteQuery } from "@tanstack/react-query";
import { Link } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import * as api from "@/lib/api";
import CoverageNotice from "@/lib/CoverageNotice";
import InlineError from "@/lib/InlineError";
import { queriesInfiniteQuery } from "@/lib/queries";
import type { QueryRow } from "@/lib/types";
import { useClientNames } from "@/features/clients/clientNames";
import { summarizeRow } from "@/features/queries/querySummary";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { ActivityCells, ActivityTableHead, activityDomainLink } from "./cells";
import { queriesFilterOf, type ActivitySearch } from "./search";
const styles = stylex.create({
empty: {
marginTop: "1.5rem",
color: colors.textMuted,
},
tableWrap: {
marginTop: "1rem",
overflowX: "auto",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
},
table: {
width: "100%",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
/** `divide-y`: a hairline between rows, so the first row carries none. */
row: {
borderTopWidth: { default: 1, ":first-child": 0 },
borderTopStyle: "solid",
borderTopColor: colors.border,
},
footer: {
marginTop: "0.75rem",
display: "flex",
alignItems: "center",
gap: "0.75rem",
},
note: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
refetching: {
marginTop: "0.75rem",
},
moreButton: {
fontWeight: 500,
},
moreError: {
marginTop: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.dangerText,
},
});
function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
export default function HistoryActivity({ search }: { search: ActivitySearch }) {
const filter = queriesFilterOf(search);
const base = useInfiniteQuery(queriesInfiniteQuery(filter));
const clientNames = useClientNames();
const pages = base.data?.pages ?? [];
const rows: QueryRow[] = pages.flatMap((page) => page.queries);
const coverage = pages[0]?.coverage;
const filterActive = Object.keys(filter).length > 0;
// `base.hasNextPage` reads the query state, which is empty while placeholder
// data stands in for a filter change; derive the cursor from what is on
// screen so the button keeps its place instead of flashing "end of log".
const lastPage = pages[pages.length - 1];
const hasMore = lastPage !== undefined && lastPage.next_before !== null;
// A 401 is already redirecting via the cache-level handleUnauthorized.
const isUnauthorized = base.error instanceof api.ApiError && base.error.status === 401;
const moreError = base.isFetchNextPageError && !isUnauthorized ? errorMessage(base.error) : null;
function loadMore() {
if (!hasMore || base.isFetchingNextPage || base.isPlaceholderData) return;
void base.fetchNextPage();
}
// The loader starts this fetch but does not wait for it, so both the first
// paint and a failed first page are this component's to render.
if (base.status === "error" && base.data === undefined) {
return <InlineError error={base.error} onRetry={() => void base.refetch()} />;
}
if (base.data === undefined) {
return (
<p {...stylex.props(styles.empty, shared.pulse)} role="status">
Loading activity
</p>
);
}
return (
<>
{base.isFetching && (
<p {...stylex.props(styles.note, styles.refetching)} role="status">
Loading
</p>
)}
{coverage !== undefined && <CoverageNotice coverage={coverage} />}
{rows.length === 0 ? (
<p {...stylex.props(styles.empty)}>
{filterActive ? "No queries match the current filters." : "No queries logged yet."}
</p>
) : (
<>
<div {...stylex.props(styles.tableWrap)}>
<table {...stylex.props(styles.table)}>
<ActivityTableHead />
<tbody>
{rows.map((row) => (
<tr key={row.id} {...stylex.props(styles.row)}>
<ActivityCells
row={summarizeRow(row)}
clientNames={clientNames}
renderDomain={(id, children) =>
id === null ? (
children
) : (
<Link
to="/activity/queries/$id"
params={{ id: String(id) }}
search={search}
{...stylex.props(activityDomainLink, shared.focusRing)}
>
{children}
</Link>
)
}
/>
</tr>
))}
</tbody>
</table>
</div>
<div {...stylex.props(styles.footer)}>
<p {...stylex.props(styles.note)}>
Showing {rows.length} {rows.length === 1 ? "query" : "queries"}
{hasMore ? "" : " — end of log"}
</p>
{hasMore && (
<button
type="button"
onClick={loadMore}
disabled={base.isFetchingNextPage || base.isPlaceholderData}
{...stylex.props(shared.button, styles.moreButton, shared.focusRing)}
>
{base.isFetchingNextPage ? "Loading…" : "Load more"}
</button>
)}
</div>
{moreError !== null && (
<p role="alert" {...stylex.props(styles.moreError)}>
Failed to load more: {moreError}
</p>
)}
</>
)}
</>
);
}
@@ -0,0 +1,399 @@
/**
* Activity in live mode, through the real router.
*
* The EventSource is a global here rather than an injected factory: whether the
* connection exists at all is the thing under test, and that is decided by
* which subtree the URL mounts, not by a prop a caller could pass.
*/
import { act, fireEvent, render, screen, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import type { Client } from "@/lib/types";
import { provenance, queryRow } from "@/features/queries/provenanceFixture";
import { FakeEventSource } from "./fakeEventSource";
function client(ip: string, name: string, learnedName: string): Client {
return {
id: Number(ip.split(".").pop()),
ip,
name,
learned_name: learnedName,
group_id: 1,
group: "default",
hand_edited: name !== "",
first_seen: 1_700_000_000,
last_seen: 1_700_000_100,
};
}
const CLIENTS: Client[] = [
client("192.0.2.10", "Kitchen Pi", "pi.lan"),
client("192.0.2.11", "", "laptop.lan"),
client("192.0.2.12", "", ""),
];
const VERSION = { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 };
let sources: FakeEventSource[];
let fetchMock: ReturnType<typeof vi.fn>;
function json(payload: unknown): Response {
return new Response(JSON.stringify(payload), { status: 200, headers: { "content-type": "application/json" } });
}
function stubFetch(handler: (url: string) => Response | Promise<Response> = () => json({})) {
fetchMock = vi.fn((input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/version") return Promise.resolve(json(VERSION));
if (url === "/api/clients") return Promise.resolve(json({ clients: CLIENTS }));
return Promise.resolve(handler(url));
});
vi.stubGlobal("fetch", fetchMock);
}
beforeEach(() => {
sources = [];
vi.stubGlobal(
"EventSource",
class {
constructor(url: string) {
const source = new FakeEventSource(url);
sources.push(source);
return source as unknown as EventSource;
}
},
);
stubFetch();
});
afterEach(() => {
vi.unstubAllGlobals();
});
function frame(ts: number, domain: string, sections: Parameters<typeof provenance>[0] = {}): { data: string } {
return {
data: JSON.stringify(
provenance({
...sections,
request: { time: ts, domain, ...sections.request },
route: { kind: "cache", upstream: "", ...sections.route },
}),
),
};
}
function renderPage(path = "/activity?mode=live") {
const queryClient = createQueryClient();
const history = createMemoryHistory({ initialEntries: [path] });
const router = createAppRouter(history, queryClient);
render(
<AuthProvider>
<QueryClientProvider client={queryClient}>
<RouterProvider router={router} />
</QueryClientProvider>
</AuthProvider>,
);
return { history };
}
async function openLive(path?: string) {
const rendered = renderPage(path);
await screen.findByRole("button", { name: "Freeze" });
act(() => sources[0]!.emit("open"));
return rendered;
}
function queryCalls(): string[] {
return fetchMock.mock.calls
.map((call) => String(call[0]))
.filter((url) => url === "/api/queries" || url.startsWith("/api/queries?"));
}
test("streams rows, flags blocked ones, and freezes the display", async () => {
await openLive();
expect(screen.getByRole("status", { name: "Live" })).toBeTruthy();
expect(screen.getByText("Waiting for queries…")).toBeTruthy();
act(() => {
sources[0]!.emit("query", frame(1000, "ok.example"));
sources[0]!.emit(
"query",
frame(1001, "ads.example", {
request: { qtype: 28 },
policy: { action: "block", reason: "blocklist_wildcard" },
route: { kind: "blocked" },
}),
);
});
expect(screen.getByText("ok.example")).toBeTruthy();
expect(screen.getByText("AAAA")).toBeTruthy();
const blockedRow = screen.getByText("ads.example").closest("tr")!;
expect(within(blockedRow).getAllByText("Blocked")).toHaveLength(2);
// StyleX compiles to opaque class names, so the check is structural: a blocked
// row carries every class a plain row does, plus the ones the flag adds.
const plainRow = screen.getByText("ok.example").closest("tr")!;
const blockedClasses = new Set(blockedRow.className.split(" "));
const plainClasses = plainRow.className.split(" ");
expect(plainClasses.every((name) => blockedClasses.has(name))).toBe(true);
expect(blockedClasses.size).toBeGreaterThan(plainClasses.length);
const freeze = screen.getByRole("button", { name: "Freeze" });
fireEvent.click(freeze);
expect(freeze.getAttribute("aria-pressed")).toBe("true");
act(() => sources[0]!.emit("query", frame(1002, "later.example")));
expect(screen.queryByText("later.example")).toBeNull();
expect(screen.getByText(/3 in buffer/)).toBeTruthy();
fireEvent.click(screen.getByRole("button", { name: "Resume" }));
expect(screen.getByText("later.example")).toBeTruthy();
});
test("resolves each row's client to its display name, keeping the IP as the tooltip", async () => {
await openLive();
act(() => {
sources[0]!.emit("query", frame(1000, "named.example", { request: { client: "192.0.2.10" } }));
sources[0]!.emit("query", frame(1001, "learned.example", { request: { client: "192.0.2.11" } }));
sources[0]!.emit("query", frame(1002, "nameless.example", { request: { client: "192.0.2.12" } }));
sources[0]!.emit("query", frame(1003, "stranger.example", { request: { client: "192.0.2.99" } }));
});
const named = await screen.findByText("Kitchen Pi");
expect(named.getAttribute("title")).toBe("192.0.2.10");
expect(screen.queryByText("pi.lan")).toBeNull();
const learned = screen.getByText("laptop.lan");
expect(learned.getAttribute("title")).toBe("192.0.2.11");
expect(within(learned.closest("tr")!).queryByText("learned")).toBeNull();
expect(screen.getByText("192.0.2.12").getAttribute("title")).toBeNull();
expect(screen.getByText("192.0.2.99").getAttribute("title")).toBeNull();
});
test("rows stream in as bare IPs while the client list is still loading", async () => {
let releaseClients: () => void = () => {};
fetchMock = vi.fn((input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/version") return Promise.resolve(json(VERSION));
return new Promise<Response>((resolve) => {
if (url !== "/api/clients") {
resolve(json({}));
return;
}
releaseClients = () => resolve(json({ clients: CLIENTS }));
});
});
vi.stubGlobal("fetch", fetchMock);
await openLive();
act(() => sources[0]!.emit("query", frame(1000, "named.example", { request: { client: "192.0.2.10" } })));
expect(screen.getByText("192.0.2.10")).toBeTruthy();
expect(screen.queryByText("Kitchen Pi")).toBeNull();
releaseClients();
expect(await screen.findByText("Kitchen Pi")).toBeTruthy();
});
test("repeated connection failures show the viewer-cap state with a retry button", async () => {
renderPage();
await screen.findByRole("button", { name: "Freeze" });
act(() => {
sources[0]!.emit("error");
sources[0]!.emit("error");
sources[0]!.emit("error");
});
expect(screen.getByRole("alert").textContent).toContain("too many live viewers");
fireEvent.click(screen.getByRole("button", { name: "Retry" }));
expect(sources).toHaveLength(2);
expect(screen.getByText("Connecting…")).toBeTruthy();
});
test("a recovered row links to its stored detail; a streamed one opens in place instead", async () => {
stubFetch((url) => {
if (url.startsWith("/api/queries?")) {
return json({
queries: [queryRow(88, { ts: 1001, domain: "recovered.example" })],
next_before: null,
coverage: { complete: true, available_since: 0 },
});
}
return json({});
});
await openLive();
act(() => sources[0]!.emit("query", frame(1000, "streamed.example")));
act(() => sources[0]!.emit("error"));
act(() => sources[0]!.emit("open"));
const recovered = await screen.findByRole("link", { name: "recovered.example" });
expect(recovered.getAttribute("href")).toContain("/activity/queries/88");
// The streamed frame precedes its own insert, so it has no row to link to —
// but it does carry its own provenance, so it still has a detail.
expect(screen.queryByRole("link", { name: "streamed.example" })).toBeNull();
expect(screen.getByRole("button", { name: "streamed.example" })).toBeTruthy();
});
test("a streamed row opens its own provenance, from the keyboard as well as the pointer", async () => {
await openLive();
act(() =>
sources[0]!.emit(
"query",
frame(1000, "streamed.example", {
policy: { action: "block", reason: "blocklist_domain", matched: "streamed.example" },
route: { kind: "blocked", upstream: "" },
}),
),
);
const trigger = screen.getByRole("button", { name: "streamed.example" });
// A real <button> is in the tab order and activates on Enter and Space; the
// only way to lose that is to opt out of it, which nothing here may do.
expect(trigger.tagName).toBe("BUTTON");
expect(trigger.getAttribute("tabindex")).toBeNull();
act(() => trigger.focus());
expect(document.activeElement).toBe(trigger);
fireEvent.click(trigger);
const heading = screen.getByRole("heading", { level: 1, name: "streamed.example" });
expect(heading).toBeTruthy();
const panel = heading.closest("div")!.parentElement!;
expect(within(panel).getByText("Blocked locally")).toBeTruthy();
expect(panel.textContent).toContain("the query log may not have written it yet");
fireEvent.click(screen.getByRole("button", { name: "Close" }));
expect(screen.queryByRole("heading", { level: 1, name: "streamed.example" })).toBeNull();
});
test("the detail takes focus when a row opens it and hands it back when it closes", async () => {
await openLive();
act(() => sources[0]!.emit("query", frame(1000, "streamed.example")));
const trigger = screen.getByRole("button", { name: "streamed.example" });
expect(trigger.getAttribute("aria-expanded")).toBe("false");
expect(trigger.getAttribute("aria-controls")).toBeNull();
// A native button activates on Enter and Space; jsdom does not synthesize
// the click those keys fire, so the click is the activation.
act(() => trigger.focus());
fireEvent.click(trigger);
const panel = screen.getByRole("group", { name: "Streamed query" });
expect(trigger.getAttribute("aria-expanded")).toBe("true");
expect(trigger.getAttribute("aria-controls")).toBe(panel.id);
// The panel is inserted above the table, behind the trigger in tab order, so
// the only thing that keeps a forward tab inside it is focus moving in.
expect(panel.compareDocumentPosition(trigger) & Node.DOCUMENT_POSITION_FOLLOWING).toBeTruthy();
expect(document.activeElement).toBe(panel);
expect(panel.contains(screen.getByRole("button", { name: "Close" }))).toBe(true);
fireEvent.click(screen.getByRole("button", { name: "Close" }));
expect(screen.queryByRole("group", { name: "Streamed query" })).toBeNull();
expect(document.activeElement).toBe(trigger);
expect(trigger.getAttribute("aria-expanded")).toBe("false");
expect(trigger.getAttribute("aria-controls")).toBeNull();
});
test("opening a second row moves the expanded state and the focus with it", async () => {
await openLive();
act(() => {
sources[0]!.emit("query", frame(1000, "first.example"));
sources[0]!.emit("query", frame(1001, "second.example"));
});
const first = screen.getByRole("button", { name: "first.example" });
const second = screen.getByRole("button", { name: "second.example" });
fireEvent.click(first);
fireEvent.click(second);
const panel = screen.getByRole("group", { name: "Streamed query" });
expect(within(panel).getByRole("heading", { level: 1, name: "second.example" })).toBeTruthy();
expect(document.activeElement).toBe(panel);
expect(first.getAttribute("aria-expanded")).toBe("false");
expect(second.getAttribute("aria-expanded")).toBe("true");
fireEvent.click(screen.getByRole("button", { name: "Close" }));
expect(document.activeElement).toBe(second);
});
test("an open streamed detail survives the row being evicted from the ring buffer", async () => {
await openLive();
act(() => sources[0]!.emit("query", frame(1000, "evicted.example")));
fireEvent.click(screen.getByRole("button", { name: "evicted.example" }));
expect(screen.getByRole("heading", { level: 1, name: "evicted.example" })).toBeTruthy();
// 500 more queries: the ring keeps the newest 500, so the selected row is
// gone from the table. The detail is a snapshot, not a lookup into the ring.
act(() => {
for (let index = 0; index < 500; index += 1) {
sources[0]!.emit("query", frame(2000 + index, `filler${index}.example`));
}
});
expect(screen.queryByRole("button", { name: "evicted.example" })).toBeNull();
expect(screen.getByRole("heading", { level: 1, name: "evicted.example" })).toBeTruthy();
});
test("an open streamed detail survives Freeze and Resume", async () => {
await openLive();
act(() => sources[0]!.emit("query", frame(1000, "held.example")));
fireEvent.click(screen.getByRole("button", { name: "held.example" }));
fireEvent.click(screen.getByRole("button", { name: "Freeze" }));
expect(screen.getByRole("heading", { level: 1, name: "held.example" })).toBeTruthy();
fireEvent.click(screen.getByRole("button", { name: "Resume" }));
expect(screen.getByRole("heading", { level: 1, name: "held.example" })).toBeTruthy();
});
test("the filter row stays visible, keeps its values, and is out of the tab order", async () => {
await openLive("/activity?mode=live&domain=ads&client=192.0.2.10&blocked=true");
const domain = screen.getByLabelText("Domain contains") as HTMLInputElement;
expect(domain.value).toBe("ads");
expect(domain.disabled).toBe(true);
expect((screen.getByLabelText("Client (exact)") as HTMLInputElement).disabled).toBe(true);
expect((screen.getByLabelText("Since") as HTMLInputElement).disabled).toBe(true);
expect((screen.getByLabelText("Until") as HTMLInputElement).disabled).toBe(true);
const form = domain.closest("form")!;
const controls = [...form.querySelectorAll("input, button, select, textarea, a[href], [tabindex]")];
expect(controls.length).toBeGreaterThan(0);
for (const control of controls) {
// A disabled form control is skipped by the browser's tab order, and RAC
// pins its own trigger out of it as well. Nothing in the row may
// reintroduce itself with a reachable tabindex.
expect(control.hasAttribute("disabled")).toBe(true);
const tabindex = control.getAttribute("tabindex");
expect(tabindex === null || tabindex === "-1").toBe(true);
}
});
test("live mode asks for no query pages, whatever filters the url retained", async () => {
await openLive("/activity?mode=live&domain=ads&client=192.0.2.10&blocked=true&since=1700000000&bogus=1");
act(() => sources[0]!.emit("query", frame(1000, "streamed.example")));
expect(queryCalls()).toEqual([]);
});
test("leaving live closes the stream, and coming back opens exactly one fresh one", async () => {
await openLive("/activity?mode=live&domain=ads");
fireEvent.click(screen.getByRole("button", { name: "History" }));
await screen.findByRole("button", { name: "Apply filters" });
expect(sources).toHaveLength(1);
expect(sources[0]!.closed).toBe(true);
// The filters came along, which is the point of switching rather than
// navigating: the reader keeps the question they were asking.
expect((screen.getByLabelText("Domain contains") as HTMLInputElement).value).toBe("ads");
expect((screen.getByLabelText("Domain contains") as HTMLInputElement).disabled).toBe(false);
fireEvent.click(screen.getByRole("button", { name: "Live" }));
await screen.findByRole("button", { name: "Freeze" });
expect(sources).toHaveLength(2);
expect(sources[1]!.closed).toBe(false);
});
@@ -1,25 +1,41 @@
/**
* Activity in live mode: the SSE stream, its bounded ring buffer, and the
* in-place detail a streamed row opens.
*
* This subtree is mounted only while the URL says `mode=live`, which is what
* closes the EventSource on the way back to history: the connection is a
* server-side resource capped per address, so a page that kept it open while
* showing something else would spend a viewer slot on nothing.
*
* Freeze and Follow are display state and stay out of the URL. They describe
* what the screen is doing right now, not what it is showing, so a shared link
* would carry a frozen moment the recipient never saw fill.
*/
import { useEffect, useRef, useState } from "react";
import { Link } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import { useClientNames } from "@/features/clients/clientNames";
import { QueryCells, QueryTableHead } from "@/features/queries/QueryLogPage";
import { RING_CAPACITY } from "./ringBuffer";
import { useLiveQueries, type EventSourceFactory, type StreamStatus } from "./useLiveQueries";
import { summarizeEvent } from "@/features/queries/querySummary";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import { ActivityCells, ActivityTableHead, activityDomainLink } from "./cells";
import ProvenanceDetail from "./ProvenanceDetail";
import RelatedActions from "./RelatedActions";
import { RING_CAPACITY, summaryOf, type StreamedRow } from "./ringBuffer";
import type { ActivitySearch } from "./search";
import { useLiveQueries, type StreamStatus } from "./useLiveQueries";
const DARK = "@media (prefers-color-scheme: dark)";
const styles = stylex.create({
toolbar: {
marginTop: "1rem",
display: "flex",
flexWrap: "wrap",
alignItems: "center",
gap: "0.75rem",
},
heading: {
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
},
toolbarButton: {
fontWeight: 500,
},
@@ -134,6 +150,40 @@ const styles = stylex.create({
[DARK]: "oklch(25.8% 0.092 26.042 / 0.4)",
},
},
/** A streamed row opens its detail here rather than at a route, so it is a button that looks like the link beside it. */
domainButton: {
borderStyle: "none",
backgroundColor: "transparent",
padding: 0,
font: "inherit",
textAlign: "left",
cursor: "pointer",
textDecorationLine: "underline",
textDecorationStyle: "dotted",
},
detailPanel: {
marginTop: "1rem",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.borderStrong,
backgroundColor: colors.surface,
padding: "1rem",
},
detailBar: {
display: "flex",
alignItems: "baseline",
justifyContent: "space-between",
gap: "0.75rem",
},
detailLabel: {
fontSize: "0.75rem",
lineHeight: "1rem",
fontWeight: 600,
letterSpacing: "0.05em",
textTransform: "uppercase",
color: colors.textMuted,
},
footnote: {
marginTop: "0.75rem",
fontSize: "0.875rem",
@@ -142,6 +192,10 @@ const styles = stylex.create({
},
});
/** One panel at a time, so the trigger that opened it can name it in `aria-controls`. */
const DETAIL_PANEL_ID = "live-query-detail";
const DETAIL_LABEL_ID = "live-query-detail-label";
const PILL_LABELS: Record<StreamStatus, string> = {
connecting: "Connecting…",
open: "Live",
@@ -165,16 +219,85 @@ function StatusPill({ status }: { status: StreamStatus }) {
);
}
/** Freeze is display-only: the stream stays open and the 500-row ring buffer keeps
* filling; Resume shows the current buffer (anything pushed out meanwhile is gone). */
export default function LiveLogPage({ createEventSource }: { createEventSource?: EventSourceFactory } = {}) {
const live = useLiveQueries({ createEventSource });
const clientNames = useClientNames();
/**
* The open detail, held as the selected row itself rather than as a key into
* the buffer.
*
* The buffer is a 500-row ring that a gap merge also rewrites: a reference by
* key would go stale under the reader while they were still reading it, and the
* panel would blank out for no reason they could see. The snapshot is the whole
* fact a streamed frame carries its own provenance so it survives eviction,
* a merge and a Freeze/Resume, and closes only when the reader closes it or
* leaves live mode.
*/
function LiveDetail({ row, origin, onClose }: { row: StreamedRow; origin: ActivitySearch; onClose: () => void }) {
const summary = summarizeEvent(row.event);
const panel = useRef<HTMLDivElement>(null);
// The panel opens above the table, behind the trigger in tab order, so a
// forward tab from the row would walk past it. Focus moves in on open —
// keyed on the row, so choosing a second row moves it again — and the
// closer puts it back on the trigger.
useEffect(() => {
panel.current?.focus();
}, [row.key]);
return (
<section>
<div
ref={panel}
id={DETAIL_PANEL_ID}
tabIndex={-1}
role="group"
aria-labelledby={DETAIL_LABEL_ID}
{...stylex.props(styles.detailPanel)}
>
<div {...stylex.props(styles.detailBar)}>
<span id={DETAIL_LABEL_ID} {...stylex.props(styles.detailLabel)}>
Streamed query
</span>
<button type="button" onClick={onClose} {...stylex.props(shared.button, shared.focusRing)}>
Close
</button>
</div>
<ProvenanceDetail
provenance={row.event}
persistedId={null}
relatedActions={
<RelatedActions
domain={summary.domain}
client={summary.client_ip}
ts={summary.ts}
origin={origin}
/>
}
/>
</div>
);
}
export default function LiveActivity({ origin }: { origin: ActivitySearch }) {
const live = useLiveQueries();
const clientNames = useClientNames();
const [selected, setSelected] = useState<StreamedRow | null>(null);
const trigger = useRef<HTMLButtonElement | null>(null);
function open(row: StreamedRow, from: HTMLButtonElement) {
trigger.current = from;
setSelected(row);
}
// The row that opened the panel takes focus back, unless the ring has
// already evicted it: a detached button cannot be focused, and the browser
// falls back to the document, which is the best available answer.
function close() {
setSelected(null);
trigger.current?.focus();
trigger.current = null;
}
return (
<>
<div {...stylex.props(styles.toolbar)}>
<h1 {...stylex.props(styles.heading)}>Live</h1>
<StatusPill status={live.status} />
<button
type="button"
@@ -228,6 +351,8 @@ export default function LiveLogPage({ createEventSource }: { createEventSource?:
</div>
)}
{selected !== null && <LiveDetail row={selected} origin={origin} onClose={close} />}
{live.rows.length === 0 ? (
live.status !== "capped" && (
<p {...stylex.props(styles.empty)}>
@@ -238,13 +363,50 @@ export default function LiveLogPage({ createEventSource }: { createEventSource?:
<>
<div {...stylex.props(styles.tableWrap)}>
<table {...stylex.props(styles.table)}>
<QueryTableHead />
<ActivityTableHead />
<tbody>
{live.rows.map((row) => (
<tr key={row.key} {...stylex.props(styles.row, row.blocked && styles.rowBlocked)}>
<QueryCells row={row} clientNames={clientNames} />
</tr>
))}
{live.rows.map((row) => {
const summary = summaryOf(row);
return (
<tr
key={row.key}
{...stylex.props(styles.row, summary.blocked && styles.rowBlocked)}
>
<ActivityCells
row={summary}
clientNames={clientNames}
renderDomain={(_id, children) =>
row.kind === "streamed" ? (
<button
type="button"
aria-expanded={selected?.key === row.key}
aria-controls={
selected?.key === row.key ? DETAIL_PANEL_ID : undefined
}
onClick={(event) => open(row, event.currentTarget)}
{...stylex.props(
styles.domainButton,
activityDomainLink,
shared.focusRing,
)}
>
{children}
</button>
) : (
<Link
to="/activity/queries/$id"
params={{ id: String(row.row.id) }}
search={origin}
{...stylex.props(activityDomainLink, shared.focusRing)}
>
{children}
</Link>
)
}
/>
</tr>
);
})}
</tbody>
</table>
</div>
@@ -254,6 +416,6 @@ export default function LiveLogPage({ createEventSource }: { createEventSource?:
</p>
</>
)}
</section>
</>
);
}
@@ -0,0 +1,178 @@
import { fireEvent, render, screen } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import type { LookupResult } from "@/lib/types";
const BLOCKED: LookupResult = {
domain: "ads.example",
group_id: 1,
local_records: false,
forward_zone: null,
blocked: true,
reason: "blocklist_domain",
matched: "ads.example",
source_url: "https://lists.test/a",
safe_search_rewrite: null,
};
let fetchMock: ReturnType<typeof createFetchMock>;
/** What `/api/lookup` answers, so a test can make it fail without rebuilding the mock. */
let lookup: (url: string) => Response;
function json(payload: unknown, status = 200, headers: Record<string, string> = {}): Response {
return new Response(JSON.stringify(payload), {
status,
headers: { "content-type": "application/json", ...headers },
});
}
function createFetchMock() {
return vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/groups") {
return json({
groups: [
{ id: 1, name: "default", safe_search: false },
{ id: 2, name: "kids", safe_search: true },
],
});
}
if (url.startsWith("/api/lookup")) return lookup(url);
return json({ error: "not stubbed" }, 404);
});
}
beforeEach(() => {
lookup = (url) =>
url === "/api/lookup?domain=ads.example&group_id=1" ? json(BLOCKED) : json({ error: "not stubbed" }, 404);
fetchMock = createFetchMock();
vi.stubGlobal("fetch", fetchMock);
});
afterEach(() => {
vi.unstubAllGlobals();
});
/**
* `retry: false` for the failure tests: the shared client retries a 5xx twice
* and a 429 after its Retry-After, so the surfaced error is what the page does
* once the client has given up, not something a test should sit out in real
* time.
*/
function renderPage(path = "/activity/test", { retry = true } = {}) {
const queryClient = createQueryClient();
if (!retry) {
const defaults = queryClient.getDefaultOptions();
queryClient.setDefaultOptions({ ...defaults, queries: { ...defaults.queries, retry: false } });
}
const router = createAppRouter(createMemoryHistory({ initialEntries: [path] }), queryClient);
render(
<AuthProvider>
<QueryClientProvider client={queryClient}>
<RouterProvider router={router} />
</QueryClientProvider>
</AuthProvider>,
);
}
function lookupCalls(): string[] {
return fetchMock.mock.calls.map(([input]) => String(input)).filter((url) => url.startsWith("/api/lookup"));
}
test("fetches nothing until submit, then renders the blocked verdict", async () => {
renderPage();
await screen.findByRole("heading", { name: "Current policy simulation" });
await screen.findByLabelText("Group");
expect(lookupCalls()).toEqual([]);
fireEvent.change(screen.getByLabelText("Domain"), { target: { value: "ads.example" } });
expect(lookupCalls()).toEqual([]);
fireEvent.click(screen.getByRole("button", { name: "Simulate" }));
await screen.findByRole("heading", { name: "Blocked" });
expect(lookupCalls()).toEqual(["/api/lookup?domain=ads.example&group_id=1"]);
expect(screen.getByText("blocklist_domain")).toBeTruthy();
const link = screen.getByRole("link", { name: "https://lists.test/a" }) as HTMLAnchorElement;
expect(link.href).toBe("https://lists.test/a");
expect(screen.getByText("Queries for this name get a blocked response.")).toBeTruthy();
});
test("a ?domain= link asks the question on arrival instead of leaving a filled-in form", async () => {
renderPage("/activity/test?domain=ads.example");
// No submit here: the link is the question, so the verdict is what arrives.
await screen.findByRole("heading", { name: "Blocked" });
expect(lookupCalls()).toEqual(["/api/lookup?domain=ads.example&group_id=1"]);
expect(screen.getByLabelText("Domain")).toHaveProperty("value", "ads.example");
});
test("no filter snapshot reads as a server that is starting, not as a verdict", async () => {
lookup = () => json({ error: "no snapshot" }, 503);
renderPage("/activity/test", { retry: false });
fireEvent.change(await screen.findByLabelText("Domain"), { target: { value: "ads.example" } });
fireEvent.click(screen.getByRole("button", { name: "Simulate" }));
const alert = await screen.findByRole("alert");
expect(alert.textContent).toContain("No filter snapshot is loaded yet");
expect(lookupCalls()).toEqual(["/api/lookup?domain=ads.example&group_id=1"]);
// Nothing may read as an answer while the lookup has none.
expect(screen.queryByRole("heading", { name: "Blocked" })).toBeNull();
expect(screen.queryByText("Simulating…")).toBeNull();
});
test("a rate limit says how long to wait, from the server's own Retry-After", async () => {
lookup = () => json({ error: "rate limited" }, 429, { "retry-after": "12" });
renderPage("/activity/test", { retry: false });
fireEvent.change(await screen.findByLabelText("Domain"), { target: { value: "ads.example" } });
fireEvent.click(screen.getByRole("button", { name: "Simulate" }));
const alert = await screen.findByRole("alert");
expect(alert.textContent).toBe("Rate limited. Try again in 12s.");
});
test("resubmitting the same domain and group refetches rather than showing a stale verdict", async () => {
renderPage();
fireEvent.change(await screen.findByLabelText("Domain"), { target: { value: "ads.example" } });
fireEvent.click(screen.getByRole("button", { name: "Simulate" }));
await screen.findByRole("heading", { name: "Blocked" });
expect(lookupCalls()).toHaveLength(1);
// The policy can change between two identical questions, so the second one
// has to reach the server even though the query key has not moved.
lookup = () => json({ ...BLOCKED, blocked: false, reason: "no_match", matched: "", source_url: null });
fireEvent.click(screen.getByRole("button", { name: "Simulate" }));
await screen.findByRole("heading", { name: "Allowed" });
expect(lookupCalls()).toEqual([
"/api/lookup?domain=ads.example&group_id=1",
"/api/lookup?domain=ads.example&group_id=1",
]);
});
test("defaults the group select to the default group (id 1)", async () => {
renderPage();
// A RAC Select names its trigger with the current value and then the label, so
// the selected group's name is the only thing the trigger shows.
const trigger = await screen.findByRole("button", { name: /Group$/ });
expect(trigger.textContent).toContain("default");
});
test("the framing is forward-tense, so it cannot be read as an account of a past query", async () => {
renderPage();
await screen.findByRole("heading", { name: "Current policy simulation" });
const intro = screen.getByRole("heading", { name: "Current policy simulation" }).nextElementSibling;
expect(intro?.textContent).toContain("would");
expect(intro?.textContent).toContain("right now");
// Nothing on the page may claim to explain a query that already happened.
expect(document.body.textContent).not.toContain("Look up");
});
@@ -1,5 +1,6 @@
import { useState, type FormEvent, type ReactNode } from "react";
import { useQuery, useSuspenseQuery } from "@tanstack/react-query";
import { useSearch } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import { ApiError } from "@/lib/api";
import { groupsQuery, lookupQuery } from "@/lib/queries";
@@ -249,13 +250,18 @@ function VerdictCard({ result, groups }: { result: LookupResult; groups: Group[]
);
}
export default function LookupPage() {
export default function PolicyTestPage() {
const groups = useSuspenseQuery(groupsQuery()).data;
const preselectedGroupId = defaultGroupId(groups);
const [domain, setDomain] = useState("");
// A `?domain=` link (from a query's detail page) arrives already asking the
// question, so it runs the simulation rather than leaving a filled-in form.
const search = useSearch({ from: "/shell/activity/test" });
const [domain, setDomain] = useState(search.domain ?? "");
const [groupId, setGroupId] = useState(preselectedGroupId);
const [submitted, setSubmitted] = useState<Submitted | null>(null);
const [submitted, setSubmitted] = useState<Submitted | null>(
search.domain === undefined ? null : { domain: search.domain, groupId: preselectedGroupId },
);
const lookup = useQuery({
...lookupQuery(submitted?.domain ?? "", submitted?.groupId),
@@ -275,17 +281,19 @@ export default function LookupPage() {
return (
<section>
<h1 {...stylex.props(styles.heading)}>Lookup</h1>
<h1 {...stylex.props(styles.heading)}>Current policy simulation</h1>
<p {...stylex.props(styles.intro)}>
What the pipeline would do with a domain: local records, forward zones, block decision, safe search.
What the pipeline <em>would</em> do with a domain right now: local records, forward zones, block
decision, safe search. This reads the configuration in force at this moment, so it explains nothing
about a query already answered a detail page does that.
</p>
<form onSubmit={onSubmit} {...stylex.props(styles.form)}>
<div {...stylex.props(styles.domainField)}>
<label htmlFor="lookup-domain" {...stylex.props(styles.fieldLabel)}>
<label htmlFor="policy-test-domain" {...stylex.props(styles.fieldLabel)}>
Domain
</label>
<input
id="lookup-domain"
id="policy-test-domain"
required
value={domain}
onChange={(event) => setDomain(event.target.value)}
@@ -306,10 +314,10 @@ export default function LookupPage() {
disabled={lookup.isFetching}
{...stylex.props(shared.largePrimaryButton, shared.focusRing)}
>
Look up
Simulate
</button>
</form>
{lookup.isFetching && <p {...stylex.props(styles.note)}>Looking up</p>}
{lookup.isFetching && <p {...stylex.props(styles.note)}>Simulating</p>}
{!lookup.isFetching && lookup.isError && (
<p role="alert" {...stylex.props(styles.error)}>
{errorMessage(lookup.error)}
@@ -0,0 +1,300 @@
/**
* One query, explained in the order it met the pipeline.
*
* This is the body of the detail surface, with no route in it, because two
* surfaces show it: the persisted detail page, which fetched the row by id, and
* a live row, whose provenance arrived in the stream frame and which SQLite may
* not have written yet. The related actions are links into routes, so the
* caller passes them in already built.
*/
import type { ReactNode } from "react";
import * as stylex from "@stylexjs/stylex";
import { formatMicros, formatTime } from "@/lib/format";
import type { PolicyReason, Provenance } from "@/lib/types";
import { clientLabel, useClientNames } from "@/features/clients/clientNames";
import {
policyActionLabel,
policyReasonLabel,
qclassName,
rcodeName,
routeKindLabel,
} from "@/features/queries/provenanceCopy";
import { qtypeName } from "@/features/queries/qtype";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
const styles = stylex.create({
heading: {
marginTop: "0.5rem",
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
wordBreak: "break-all",
},
subtitle: {
marginTop: "0.25rem",
color: colors.textSecondary,
},
/** The recorded facts, fenced off from the live links below them. */
record: {
marginTop: "1rem",
maxWidth: "48rem",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
backgroundColor: colors.surfaceRaised,
padding: "1rem",
},
recordNote: {
fontSize: "0.8125rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
section: {
marginTop: "1rem",
borderTopWidth: { default: 1, ":first-of-type": 0 },
borderTopStyle: "solid",
borderTopColor: colors.border,
paddingTop: { default: "1rem", ":first-of-type": 0 },
},
sectionHeading: {
fontSize: "0.75rem",
lineHeight: "1rem",
fontWeight: 600,
letterSpacing: "0.05em",
textTransform: "uppercase",
color: colors.textMuted,
},
facts: {
marginTop: "0.5rem",
marginBottom: 0,
display: "grid",
gap: "0.375rem 1rem",
gridTemplateColumns: {
default: "auto",
"@media (min-width: 640px)": "max-content 1fr",
},
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
term: {
color: colors.textMuted,
},
value: {
margin: 0,
wordBreak: "break-all",
},
muted: {
color: colors.textMuted,
},
related: {
marginTop: "1.5rem",
maxWidth: "48rem",
},
relatedHeading: {
fontSize: "1.125rem",
lineHeight: "1.75rem",
fontWeight: 600,
},
relatedNote: {
marginTop: "0.25rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
relatedList: {
marginTop: "0.5rem",
display: "flex",
flexWrap: "wrap",
gap: "0.75rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
link: {
color: colors.primaryOnSurface,
},
});
/** The look of one related action, so every caller's links match. */
export const provenanceRelatedLink = styles.link;
function Section({ title, children }: { title: string; children: ReactNode }) {
return (
<div {...stylex.props(styles.section)}>
<h2 {...stylex.props(styles.sectionHeading)}>{title}</h2>
<dl {...stylex.props(styles.facts)}>{children}</dl>
</div>
);
}
function Fact({ label, children }: { label: string; children: ReactNode }) {
return (
<>
<dt {...stylex.props(styles.term)}>{label}</dt>
<dd {...stylex.props(styles.value)}>{children}</dd>
</>
);
}
/**
* A recorded name, in the face the rest of the interface gives to names. It
* wraps the value rather than the row, so the prose that stands in for a
* missing one is not set in the same typewriter face.
*/
function Mono({ children }: { children: string }) {
return <span {...stylex.props(shared.mono)}>{children}</span>;
}
/** An empty text field means the server recorded nothing there, never an empty value. */
function Absent({ children }: { children: string }) {
return <span {...stylex.props(styles.muted)}>{children}</span>;
}
/**
* What an empty `matched` means. `no_match` is the only reason the matcher
* itself records with nothing to show; every other empty one names a pipeline
* step that answered before filtering — a local record, a forward zone, a pause
* (see `PolicyReason` in src/storage/provenance.zig) — where "nothing matched"
* would claim an evaluation that never happened.
*/
function unmatchedLabel(reason: PolicyReason): string {
return reason === "no_match" ? "Nothing matched" : "The matcher never ran";
}
interface Props {
provenance: Provenance;
/**
* The log row this explains, or null for a frame read straight off the
* stream. Null is the same fact `QuerySummary.id` carries: the event
* precedes its own insert, so there is no row to name and none is invented.
*/
persistedId: number | null;
/** The related-action links, built by whichever surface owns the routes. */
relatedActions: ReactNode;
}
export default function ProvenanceDetail({ provenance, persistedId, relatedActions }: Props) {
const { request, group, policy, rewrites, route, response } = provenance;
const clientNames = useClientNames();
const currentClient = clientLabel(request.client, clientNames);
return (
<>
<h1 {...stylex.props(styles.heading, shared.mono)}>{request.domain}</h1>
<p {...stylex.props(styles.subtitle)}>
{formatTime(request.time)} {policyActionLabel(policy.action)}
{/* The verdict alone reads as a success; a non-NOERROR answer says otherwise. */}
{response.rcode !== 0 && `${rcodeName(response.rcode)}`}
</p>
<div {...stylex.props(styles.record)}>
<p {...stylex.props(styles.recordNote)}>
What was recorded when this query was answered. Group and blocklist names are the ones in force at
that moment; they may have been renamed or deleted since.
{persistedId === null &&
" This is the event as it was streamed; the query log may not have written it yet."}
</p>
<Section title="Request">
<Fact label="Time">{formatTime(request.time)}</Fact>
<Fact label="Domain">
<Mono>{request.domain}</Mono>
</Fact>
<Fact label="Client">
<Mono>{request.client}</Mono>
</Fact>
<Fact label="Type">{qtypeName(request.qtype)}</Fact>
<Fact label="Class">{qclassName(request.qclass)}</Fact>
</Section>
<Section title="Group">
<Fact label="Name">{group.name === "" ? <Absent>No group recorded</Absent> : group.name}</Fact>
<Fact label="Id">{group.id === null ? <Absent></Absent> : group.id}</Fact>
</Section>
<Section title="Policy">
<Fact label="Decision">{policyActionLabel(policy.action)}</Fact>
<Fact label="Reason">{policyReasonLabel(policy.reason)}</Fact>
<Fact label="Matched">
{policy.matched === "" ? (
<Absent>{unmatchedLabel(policy.reason)}</Absent>
) : (
<Mono>{policy.matched}</Mono>
)}
</Fact>
<Fact label="Blocklist">
{policy.source_name === "" ? (
<Absent>Not a blocklist decision</Absent>
) : policy.source_id === null ? (
policy.source_name
) : (
`${policy.source_name} (#${policy.source_id})`
)}
</Fact>
</Section>
<Section title="Rewrites">
<Fact label="CNAME target">
{rewrites.cname_target === "" ? (
<Absent>The queried name was decided directly</Absent>
) : (
<Mono>{rewrites.cname_target}</Mono>
)}
</Fact>
<Fact label="Safe search">
{rewrites.safe_search_target === "" ? (
<Absent>No rewrite</Absent>
) : (
<Mono>{rewrites.safe_search_target}</Mono>
)}
</Fact>
</Section>
<Section title="Route">
<Fact label="Answered by">{routeKindLabel(route.kind)}</Fact>
<Fact label="Forward zone">
{route.forward_zone === "" ? <Absent></Absent> : <Mono>{route.forward_zone}</Mono>}
</Fact>
<Fact label="Upstream">
{route.upstream === "" ? <Absent>No upstream exchange</Absent> : <Mono>{route.upstream}</Mono>}
</Fact>
</Section>
<Section title="Response">
<Fact label="Result">{rcodeName(response.rcode)}</Fact>
<Fact label="Took">
{response.duration_us === null ? (
<Absent>Not measured</Absent>
) : (
formatMicros(response.duration_us)
)}
</Fact>
</Section>
</div>
<div {...stylex.props(styles.related)}>
<h2 {...stylex.props(styles.relatedHeading)}>Related</h2>
<p {...stylex.props(styles.relatedNote)}>
These read the current configuration, which may no longer be the one that decided this query.
</p>
{currentClient !== null && (
<p {...stylex.props(styles.relatedNote)}>
{currentClient.learned ? (
<>
Reverse DNS currently resolves <Mono>{request.client}</Mono> to{" "}
<Mono>{currentClient.text}</Mono>.
</>
) : (
<>
The client list currently names <Mono>{request.client}</Mono> {currentClient.text}.
</>
)}
</p>
)}
<div {...stylex.props(styles.relatedList)}>{relatedActions}</div>
</div>
</>
);
}
@@ -0,0 +1,59 @@
/**
* The links out of one query's detail, shared by the persisted detail page and
* the in-place detail a streamed row opens.
*
* Both surfaces answer the same four follow-up questions, and both are read
* from an investigation that has a time range. Every link therefore carries
* absolute bounds: a link that said "recently" would show a different set of
* queries every time it was opened, which is the opposite of what linking to an
* incident is for.
*/
import { Link } from "@tanstack/react-router";
import * as stylex from "@stylexjs/stylex";
import { styles as shared } from "@/ui/styles";
import { provenanceRelatedLink } from "./ProvenanceDetail";
import { diagnosticsBounds, relatedBounds } from "./relatedLinks";
import type { ActivitySearch } from "./search";
interface Props {
domain: string;
client: string;
/** The second this query was answered, which every window is centred on. */
ts: number;
/** The Activity search the reader came from; its bounds win over the defaults. */
origin: Pick<ActivitySearch, "since" | "until">;
}
export default function RelatedActions({ domain, client, ts, origin }: Props) {
const bounds = relatedBounds(ts, origin);
const window = diagnosticsBounds(ts);
return (
<>
<Link to="/activity/test" search={{ domain }} {...stylex.props(provenanceRelatedLink, shared.focusRing)}>
Test this domain against current policy
</Link>
<Link
to="/activity"
search={{ mode: "history", domain, client: undefined, blocked: undefined, ...bounds }}
{...stylex.props(provenanceRelatedLink, shared.focusRing)}
>
All activity for this domain
</Link>
<Link
to="/activity"
search={{ mode: "history", client, domain: undefined, blocked: undefined, ...bounds }}
{...stylex.props(provenanceRelatedLink, shared.focusRing)}
>
All activity from this client
</Link>
<Link
to="/diagnostics"
search={{ since: window.since, until: window.until }}
{...stylex.props(provenanceRelatedLink, shared.focusRing)}
>
Diagnostics around this query
</Link>
</>
);
}
+114
View File
@@ -0,0 +1,114 @@
import { render, screen } from "@testing-library/react";
import type { QueryRow } from "@/lib/types";
import type { ClientNames } from "@/features/clients/clientNames";
import { queryRow } from "@/features/queries/provenanceFixture";
import { summarizeRow, type QuerySummary } from "@/features/queries/querySummary";
import { ACTIVITY_COLUMNS, ActivityCells, ActivityTableHead, resultLabel, routeLabel } from "./cells";
const noNames: ClientNames = new Map();
function renderRow(overrides: Partial<QueryRow> = {}): HTMLTableRowElement {
const row: QuerySummary = summarizeRow(queryRow(1, overrides));
render(
<table>
<ActivityTableHead />
<tbody>
<tr data-testid="row">
<ActivityCells
row={row}
clientNames={noNames}
renderDomain={(id, children) => <a href={`/activity/queries/${id}`}>{children}</a>}
/>
</tr>
</tbody>
</table>,
);
return screen.getByTestId("row") as HTMLTableRowElement;
}
/** The cell under a header, read by its column name rather than its index. */
function cell(row: HTMLTableRowElement, column: (typeof ACTIVITY_COLUMNS)[number]): string {
const index = ACTIVITY_COLUMNS.indexOf(column);
return row.cells[index]?.textContent ?? "";
}
test("the head names the seven columns in order", () => {
render(
<table>
<ActivityTableHead />
</table>,
);
const headers = screen.getAllByRole("columnheader").map((header) => header.textContent);
expect(headers).toEqual(["Time", "Domain", "Client", "Type", "Result", "Route", "Duration"]);
});
test("an allowed NOERROR row reads as the answer it got, with no badge", () => {
const row = renderRow({ blocked: false, rcode: 0, route_kind: "upstream", response_time_us: 1234 });
expect(cell(row, "Result")).toBe("NOERROR");
expect(cell(row, "Route")).toBe("Upstream");
expect(cell(row, "Duration")).toBe("1.2 ms");
expect(cell(row, "Type")).toBe("A");
});
test("a blocked row reads Blocked even though the client got NOERROR", () => {
const row = renderRow({ blocked: true, rcode: 0, route_kind: "blocked", policy_reason: "blocklist_domain" });
expect(cell(row, "Result")).toBe("Blocked");
expect(cell(row, "Route")).toBe("Blocked");
});
test("a SERVFAIL row names the code", () => {
const row = renderRow({ blocked: false, rcode: 2, route_kind: "upstream", response_time_us: null });
expect(cell(row, "Result")).toBe("SERVFAIL");
expect(cell(row, "Duration")).toBe("—");
});
test("a cache hit names the cache as the route", () => {
const row = renderRow({ blocked: false, rcode: 0, route_kind: "cache", cache_hit: true, upstream: "" });
expect(cell(row, "Route")).toBe("Cache");
expect(cell(row, "Result")).toBe("NOERROR");
});
test("an unassigned extended rcode keeps the numeric fallback, without the long form's parentheses", () => {
const row = renderRow({ blocked: false, rcode: 3841 });
expect(cell(row, "Result")).toBe("RCODE 3841");
});
test("a persisted row links its domain to the detail the caller chose", () => {
renderRow({ domain: "ads.example" });
expect(screen.getByRole("link", { name: "ads.example" }).getAttribute("href")).toBe("/activity/queries/1");
});
test("a streamed row reaches the renderer with a null id, and can render as plain text", () => {
render(
<table>
<tbody>
<tr data-testid="row">
<ActivityCells
row={{ ...summarizeRow(queryRow(1)), id: null }}
clientNames={noNames}
renderDomain={(id, children) => {
expect(id).toBeNull();
return children;
}}
/>
</tr>
</tbody>
</table>,
);
expect(screen.queryByRole("link")).toBeNull();
expect(screen.getByTestId("row").textContent).toContain("example.com");
});
test("every route kind has a compact label", () => {
expect(routeLabel("blocked")).toBe("Blocked");
expect(routeLabel("local")).toBe("Local");
expect(routeLabel("forward_zone")).toBe("Forward zone");
expect(routeLabel("upstream")).toBe("Upstream");
expect(routeLabel("cache")).toBe("Cache");
expect(routeLabel("rejected")).toBe("Rejected");
});
test("resultLabel is the pure form of the Result cell", () => {
expect(resultLabel({ blocked: true, rcode: 2 })).toBe("Blocked");
expect(resultLabel({ blocked: false, rcode: 5 })).toBe("REFUSED");
});
+171
View File
@@ -0,0 +1,171 @@
/**
* The seven columns of the Activity table: Time, Domain, Client, Type, Result,
* Route and Duration.
*
* The labels here are the compact forms a scanned table needs. The detail page
* keeps `provenanceCopy`'s long forms, which spell out the same facts with room
* for the rcode number and the "answered by" phrasing.
*
* The cells render both a stored row and a streamed event, so they know nothing
* about routes: the caller renders the Domain cell's contents and decides what,
* if anything, a row opens. A streamed row has no id — the frame precedes its
* own insert — and only the caller knows whether it has a surface for one.
*/
import type { ReactNode } from "react";
import * as stylex from "@stylexjs/stylex";
import { formatMicros, formatTime } from "@/lib/format";
import type { RouteKind } from "@/lib/types";
import { ClientName, type ClientNames } from "@/features/clients/clientNames";
import { rcodeShortName } from "@/features/queries/provenanceCopy";
import { qtypeName } from "@/features/queries/qtype";
import type { QuerySummary } from "@/features/queries/querySummary";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
const DARK = "@media (prefers-color-scheme: dark)";
const styles = stylex.create({
head: {
backgroundColor: { default: "oklch(98.5% 0 none)", [DARK]: "oklch(21% 0.006 285.885)" },
textAlign: "left",
},
th: {
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
fontWeight: 500,
color: colors.textSecondary,
},
cell: {
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
},
nowrap: {
whiteSpace: "nowrap",
},
breakAll: {
wordBreak: "break-all",
},
small: {
fontSize: "0.75rem",
lineHeight: "1rem",
},
muted: {
color: colors.textMuted,
},
domainLink: {
color: colors.primaryOnSurface,
textDecorationLine: "none",
},
/**
* The badge shape and its weight are the signal; the tint only says which
* kind of unhappy answer this was. A monochrome or colour-blind reading of
* the table still separates a blocked or failed row from a plain NOERROR
* one, which a hue alone would not.
*/
badge: {
display: "inline-block",
borderRadius: "0.25rem",
paddingInline: "0.375rem",
paddingBlock: "0.125rem",
fontSize: "0.75rem",
lineHeight: "1rem",
fontWeight: 600,
},
blockedBadge: {
backgroundColor: { default: "oklch(93.6% 0.032 17.717)", [DARK]: "oklch(39.6% 0.141 25.723)" },
color: { default: "oklch(44.4% 0.177 26.899)", [DARK]: "oklch(88.5% 0.062 18.334)" },
},
faultBadge: {
backgroundColor: { default: "oklch(96.2% 0.059 95.617)", [DARK]: "oklch(41.4% 0.112 45.904)" },
color: { default: "oklch(47.3% 0.137 46.201)", [DARK]: "oklch(90.1% 0.076 70.697)" },
},
});
const ROUTE_LABELS: Record<RouteKind, string> = {
blocked: "Blocked",
local: "Local",
forward_zone: "Forward zone",
upstream: "Upstream",
cache: "Cache",
rejected: "Rejected",
};
/** The compact Route label. `Record` over the union, so a new kind fails `tsc`. */
export function routeLabel(kind: RouteKind): string {
return ROUTE_LABELS[kind];
}
/**
* The compact Result label. A block is the answer the operator asked nxdns for,
* so it wins over the rcode it was delivered as — a blocked name answered with
* NOERROR and a zero address is still "Blocked". Everything else reads as the
* code the client saw.
*/
export function resultLabel(row: Pick<QuerySummary, "blocked" | "rcode">): string {
return row.blocked ? "Blocked" : rcodeShortName(row.rcode);
}
/**
* What a row's domain is wrapped in. The caller owns the routes, so it builds
* the element; the look stays here, as `activityDomainLink`, which the caller
* spreads onto the control itself — a link's own colour beats one inherited
* from a wrapper. `id` is null for a streamed row, which history has no surface
* for and live opens from memory.
*/
export type DomainRenderer = (id: number | null, children: ReactNode) => ReactNode;
export const activityDomainLink = styles.domainLink;
export function ResultCellContent({ row }: { row: Pick<QuerySummary, "blocked" | "rcode"> }) {
const label = resultLabel(row);
if (row.blocked) return <span {...stylex.props(styles.badge, styles.blockedBadge)}>{label}</span>;
if (row.rcode !== 0) return <span {...stylex.props(styles.badge, styles.faultBadge)}>{label}</span>;
return <span {...stylex.props(styles.small, styles.muted)}>{label}</span>;
}
export function ActivityCells({
row,
clientNames,
renderDomain,
}: {
row: QuerySummary;
clientNames: ClientNames;
renderDomain: DomainRenderer;
}) {
return (
<>
<td {...stylex.props(styles.cell, styles.nowrap, styles.muted)}>{formatTime(row.ts)}</td>
<td {...stylex.props(styles.cell, styles.small, styles.breakAll, shared.mono)}>
{renderDomain(row.id, row.domain)}
</td>
<td {...stylex.props(styles.cell, styles.small, styles.nowrap)}>
<ClientName ip={row.client_ip} names={clientNames} />
</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>{qtypeName(row.qtype)}</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>
<ResultCellContent row={row} />
</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>{routeLabel(row.route_kind)}</td>
<td {...stylex.props(styles.cell, styles.nowrap, shared.tabularNums)}>
{row.response_time_us === null ? "—" : formatMicros(row.response_time_us)}
</td>
</>
);
}
export const ACTIVITY_COLUMNS = ["Time", "Domain", "Client", "Type", "Result", "Route", "Duration"] as const;
export function ActivityTableHead() {
return (
<thead {...stylex.props(styles.head)}>
<tr>
{ACTIVITY_COLUMNS.map((column) => (
<th key={column} {...stylex.props(styles.th)}>
{column}
</th>
))}
</tr>
</thead>
);
}
@@ -0,0 +1,100 @@
import {
datetimeField,
datetimeLocalToUnix,
editDatetimeField,
resolveDatetimeField,
unixToDatetimeLocal,
} from "./datetime";
/**
* Every assertion here is about local time, so the zone has to be pinned. New
* York is the zone the DST cases are written for: the fold is 2024-11-03 01:30
* and the gap is 2024-03-10 02:30.
*/
// The app never reads `process`, so `src` is typed without node's globals; the
// test host is node, where assigning `TZ` re-reads the zone for `Date`.
declare const process: { env: Record<string, string | undefined> };
const originalTz = process.env["TZ"];
beforeAll(() => {
process.env["TZ"] = "America/New_York";
});
afterAll(() => {
process.env["TZ"] = originalTz;
});
/** 2024-06-01T12:34:56 EDT. */
const SUMMER = 1_717_259_696;
test("a non-zero-second instant round trips", () => {
expect(unixToDatetimeLocal(SUMMER)).toBe("2024-06-01T12:34:56");
expect(datetimeLocalToUnix("2024-06-01T12:34:56")).toBe(SUMMER);
});
test("text without seconds parses as :00", () => {
expect(datetimeLocalToUnix("2024-06-01T12:34")).toBe(datetimeLocalToUnix("2024-06-01T12:34:00"));
});
test("text that is not a datetime-local value names no instant", () => {
expect(datetimeLocalToUnix("")).toBeUndefined();
expect(datetimeLocalToUnix("yesterday")).toBeUndefined();
expect(datetimeLocalToUnix("2024-06-01")).toBeUndefined();
expect(datetimeLocalToUnix("2024-13-01T00:00:00")).toBeUndefined();
});
/** 2024-11-03 01:30 EDT and 01:30 EST: two instants, one wall clock. */
const FOLD_FIRST = 1_730_611_800;
const FOLD_SECOND = 1_730_615_400;
test("the fall-back fold gives two instants the same text", () => {
expect(unixToDatetimeLocal(FOLD_FIRST)).toBe("2024-11-03T01:30:00");
expect(unixToDatetimeLocal(FOLD_SECOND)).toBe("2024-11-03T01:30:00");
expect(datetimeLocalToUnix("2024-11-03T01:30:00")).toBe(FOLD_FIRST);
});
test("an untouched fold bound applies the instant it was seeded with, not a re-parse of its text", () => {
const field = datetimeField(FOLD_SECOND);
expect(field.text).toBe("2024-11-03T01:30:00");
expect(resolveDatetimeField(field)).toEqual({ ok: true, value: FOLD_SECOND });
});
test("an edited fold bound resolves to the first of the two instants, which is what its text says", () => {
const field = editDatetimeField(datetimeField(FOLD_SECOND), "2024-11-03T01:30:00");
expect(resolveDatetimeField(field)).toEqual({ ok: true, value: FOLD_FIRST });
});
test("a spring-forward time that exists on no clock is rejected rather than slid forward an hour", () => {
const field = editDatetimeField(datetimeField(undefined), "2024-03-10T02:30:00");
expect(datetimeLocalToUnix("2024-03-10T02:30:00")).toBe(1_710_055_800);
expect(unixToDatetimeLocal(1_710_055_800)).toBe("2024-03-10T03:30:00");
expect(resolveDatetimeField(field)).toEqual({ ok: false, reason: "nonexistent" });
});
test("an edited bound round trips with non-zero seconds", () => {
const field = editDatetimeField(datetimeField(undefined), "2024-06-01T12:34:56");
expect(resolveDatetimeField(field)).toEqual({ ok: true, value: SUMMER });
});
test("clearing an edited bound drops the filter", () => {
const field = editDatetimeField(datetimeField(SUMMER), "");
expect(resolveDatetimeField(field)).toEqual({ ok: true, value: undefined });
});
test("an unparseable edit is reported, never silently dropped", () => {
const field = editDatetimeField(datetimeField(undefined), "2024-06-32T99:99");
expect(resolveDatetimeField(field)).toEqual({ ok: false, reason: "unparseable" });
});
test("an unset bound seeds an empty field that stays unset", () => {
const field = datetimeField(undefined);
expect(field.text).toBe("");
expect(resolveDatetimeField(field)).toEqual({ ok: true, value: undefined });
});
test("a fractional part on the seconds is parsed and dropped, not rejected", () => {
// jsdom, and any engine that sanitizes to the full grammar, hands the input
// back with milliseconds attached; a bound is a whole second either way.
const field = editDatetimeField(datetimeField(undefined), "2023-11-14T23:13:37.000");
const resolved = resolveDatetimeField(field);
expect(resolved).toEqual({ ok: true, value: datetimeLocalToUnix("2023-11-14T23:13:37") });
});
+95
View File
@@ -0,0 +1,95 @@
/**
* The bridge between a `datetime-local` input and the unix seconds the URL and
* the API speak.
*
* Local wall-clock text is lossy in a way unix seconds are not. Twice a year a
* fall-back fold gives two instants the same text, and a spring-forward gap
* gives an hour of text no instant at all. So the text is never the authority:
* a bound the operator did not touch is carried through as the number it
* already was, and a bound they did edit is accepted only when it survives a
* round trip unchanged.
*/
/** Zero-padded to the width the `datetime-local` grammar requires. */
function pad(value: number, width: number): string {
return String(value).padStart(width, "0");
}
/**
* Unix seconds → the local wall-clock text a `datetime-local` input holds,
* always with seconds, because the inputs run at `step={1}`.
*/
export function unixToDatetimeLocal(unix: number): string {
const date = new Date(unix * 1000);
const day = `${pad(date.getFullYear(), 4)}-${pad(date.getMonth() + 1, 2)}-${pad(date.getDate(), 2)}`;
const time = `${pad(date.getHours(), 2)}:${pad(date.getMinutes(), 2)}:${pad(date.getSeconds(), 2)}`;
return `${day}T${time}`;
}
const DATETIME_LOCAL = /^(\d{4})-(\d{2})-(\d{2})T(\d{2}):(\d{2})(?::(\d{2})(?:\.\d{1,3})?)?$/;
/**
* The text with its seconds spelled out, or undefined when it is not a
* `datetime-local` value at all. A browser omits `:00` seconds even at
* `step={1}`, so the canonical form is what a round trip compares against.
*
* The grammar allows a fractional part after the seconds and some engines emit
* one; a bound is a whole second here and on the wire, so it is parsed and then
* dropped rather than treated as text we do not recognise.
*/
function canonicalize(value: string): string | undefined {
const match = DATETIME_LOCAL.exec(value);
if (match === null) return undefined;
return `${match[1]}-${match[2]}-${match[3]}T${match[4]}:${match[5]}:${match[6] ?? "00"}`;
}
/** Local wall-clock text → unix seconds, or undefined when it names no instant. */
export function datetimeLocalToUnix(value: string): number | undefined {
const canonical = canonicalize(value);
if (canonical === undefined) return undefined;
const ms = new Date(canonical).getTime();
return Number.isFinite(ms) ? Math.floor(ms / 1000) : undefined;
}
/**
* One bound of the filter form: what the input shows, what the applied search
* carried, and whether the operator has touched it since.
*/
export interface DatetimeField {
text: string;
/** The applied value this field was seeded from, reused while `dirty` is false. */
original: number | undefined;
dirty: boolean;
}
export type DatetimeResolution =
{ ok: true; value: number | undefined } | { ok: false; reason: "unparseable" | "nonexistent" };
export function datetimeField(original: number | undefined): DatetimeField {
return { text: original === undefined ? "" : unixToDatetimeLocal(original), original, dirty: false };
}
export function editDatetimeField(field: DatetimeField, text: string): DatetimeField {
return { ...field, text, dirty: true };
}
/**
* The unix value this bound applies.
*
* An untouched field resolves to the number it was seeded with, never to a
* re-parse of its own text: the text of a fall-back instant names two of them,
* and re-parsing would silently move a bound the operator never edited.
*
* An edited field is parsed, then formatted back. A wall-clock time inside the
* spring-forward gap exists on no clock, and `Date` quietly slides it forward
* an hour; the round trip catches that and the caller reports it instead of
* filtering on an hour nobody asked for.
*/
export function resolveDatetimeField(field: DatetimeField): DatetimeResolution {
if (!field.dirty) return { ok: true, value: field.original };
if (field.text.trim() === "") return { ok: true, value: undefined };
const unix = datetimeLocalToUnix(field.text);
if (unix === undefined) return { ok: false, reason: "unparseable" };
if (unixToDatetimeLocal(unix) !== canonicalize(field.text)) return { ok: false, reason: "nonexistent" };
return { ok: true, value: unix };
}
@@ -0,0 +1,42 @@
/**
* The absolute bounds an investigation link carries.
*
* Every link out of a query detail is time-scoped on purpose: a relative window
* would answer a different question tomorrow than it does today, and the whole
* point of linking to an episode is that the link keeps showing that episode.
*/
import type { ActivitySearch } from "./search";
/** The five-minute window the redesign puts around one query. */
export const RELATED_WINDOW_SECONDS = 300;
export interface Bounds {
since: number;
until: number;
}
/**
* The bounds for "all activity for this domain/client", decided per bound.
*
* When the reader arrived from a bounded investigation, that bound is the one
* they are working in and it carries over. A bound they never set falls back to
* the five-minute window around this query — never to no bound at all, which
* would answer with the whole retained history and lose the episode in it. The
* two bounds are decided separately, so a half-bounded origin keeps its half.
*/
export function relatedBounds(ts: number, origin: Pick<ActivitySearch, "since" | "until">): Bounds {
return {
since: origin.since ?? ts - RELATED_WINDOW_SECONDS,
until: origin.until ?? ts + RELATED_WINDOW_SECONDS,
};
}
/**
* The Diagnostics window around one query. Fixed at five minutes either side of
* the query, not inherited: the reader is asking what else was failing while
* this query was answered, which is a question about the query's own moment.
*/
export function diagnosticsBounds(ts: number): Bounds {
return { since: ts - RELATED_WINDOW_SECONDS, until: ts + RELATED_WINDOW_SECONDS };
}
@@ -0,0 +1,248 @@
import type { Provenance, QueryRow } from "@/lib/types";
import { provenance, queryRow } from "@/features/queries/provenanceFixture";
import { RING_CAPACITY, mergeGap, pushRow, summaryOf, type LiveRow } from "./ringBuffer";
function streamed(key: number, ts: number, domain: string, sections: Parameters<typeof provenance>[0] = {}): LiveRow {
return {
kind: "streamed",
key,
event: provenance({
...sections,
request: { time: ts, domain, ...sections.request },
route: { upstream: "udp://9.9.9.9:53", ...sections.route },
}),
};
}
function fetchedRow(id: number, ts: number, domain: string, overrides: Partial<QueryRow> = {}): QueryRow {
return queryRow(id, { ts, domain, upstream: "udp://9.9.9.9:53", ...overrides });
}
function counter(start = 100): () => number {
let n = start;
return () => ++n;
}
function domains(rows: LiveRow[]): string[] {
return rows.map((row) => summaryOf(row).domain);
}
describe("summaryOf", () => {
test("a streamed frame projects every summary field from the provenance it carries", () => {
const event: Provenance = provenance({
request: { time: 1700, domain: "ads.example", client: "192.0.2.11", qtype: 28 },
policy: { action: "block", reason: "blocklist_wildcard" },
route: { kind: "blocked", upstream: "" },
response: { duration_us: 42 },
});
expect(summaryOf({ kind: "streamed", key: 1, event })).toEqual({
id: null,
ts: 1700,
domain: "ads.example",
client_ip: "192.0.2.11",
qtype: 28,
blocked: true,
policy_reason: "blocklist_wildcard",
rcode: 0,
route_kind: "blocked",
response_time_us: 42,
cache_hit: null,
upstream: "",
});
});
test("a recovered row projects its stored fields and keeps its id", () => {
const row = queryRow(77, { domain: "news.example", cache_hit: true, policy_reason: "rule_allow_exact" });
expect(summaryOf({ kind: "recovered", key: 2, row })).toMatchObject({
id: 77,
domain: "news.example",
cache_hit: true,
policy_reason: "rule_allow_exact",
});
});
/**
* The guard the discriminated union exists for: a field added to the wire
* DTO must be either projected into the summary or consciously left to the
* detail page. A silent addition fails here rather than going unrendered.
*/
test("every provenance field is either projected or knowingly detail-only", () => {
const projected = [
"request.time",
"request.domain",
"request.client",
"request.qtype",
"policy.action",
"policy.reason",
"route.kind",
"route.upstream",
"response.duration_us",
];
const detailOnly = [
"request.qclass",
"group.id",
"group.name",
"policy.matched",
"policy.source_id",
"policy.source_name",
"rewrites.cname_target",
"rewrites.safe_search_target",
"route.forward_zone",
"response.rcode",
];
const leaves = Object.entries(provenance()).flatMap(([section, fields]) =>
Object.keys(fields as Record<string, unknown>).map((field) => `${section}.${field}`),
);
expect(leaves.sort()).toEqual([...projected, ...detailOnly].sort());
});
});
describe("pushRow", () => {
test("prepends newest-first", () => {
let rows: LiveRow[] = [];
rows = pushRow(rows, streamed(1, 10, "a.example"));
rows = pushRow(rows, streamed(2, 11, "b.example"));
expect(domains(rows)).toEqual(["b.example", "a.example"]);
});
test("drops the oldest beyond capacity", () => {
let rows: LiveRow[] = [];
for (let i = 0; i < 5; i++) rows = pushRow(rows, streamed(i, i, `d${i}.example`), 3);
expect(rows).toHaveLength(3);
expect(rows.map((r) => r.key)).toEqual([4, 3, 2]);
});
test("default capacity is 500", () => {
let rows: LiveRow[] = [];
for (let i = 0; i < RING_CAPACITY + 10; i++) rows = pushRow(rows, streamed(i, i, "x.example"));
expect(rows).toHaveLength(RING_CAPACITY);
});
});
describe("mergeGap", () => {
test("skips rows already in the buffer and counts only new ones", () => {
const buffer = [streamed(2, 100, "seen.example"), streamed(1, 99, "old.example")];
const fetched = [
fetchedRow(30, 102, "gap2.example"),
fetchedRow(29, 101, "gap1.example"),
fetchedRow(28, 100, "seen.example"),
];
const { rows, missed } = mergeGap(buffer, fetched, counter());
expect(missed).toBe(2);
expect(domains(rows)).toEqual(["gap2.example", "gap1.example", "seen.example", "old.example"]);
});
test("no additions returns the buffer unchanged with missed 0", () => {
const buffer = [streamed(1, 100, "seen.example")];
const { rows, missed } = mergeGap(buffer, [fetchedRow(5, 100, "seen.example")], counter());
expect(missed).toBe(0);
expect(rows).toBe(buffer);
});
/**
* A household repeats itself: one client, one name, three lookups inside the
* same second. The stream delivered one of them before the connection broke,
* so the gap fetch must recover the other two rather than let the one row in
* the buffer stand for all three.
*/
test("repeated identical queries drop only as many rows as the buffer already holds", () => {
const buffer = [streamed(1, 100, "dup.example")];
const fetched = [
fetchedRow(12, 100, "dup.example"),
fetchedRow(11, 100, "dup.example"),
fetchedRow(10, 100, "dup.example"),
];
const { rows, missed } = mergeGap(buffer, fetched, counter());
expect(missed).toBe(2);
expect(domains(rows)).toEqual(["dup.example", "dup.example", "dup.example"]);
const recoveredIds = rows.flatMap((row) => (row.kind === "recovered" ? [row.row.id] : []));
expect(new Set(recoveredIds).size).toBe(2);
});
test("a gap fetch that repeats the whole buffer adds nothing", () => {
const buffer = [streamed(2, 100, "dup.example"), streamed(1, 100, "dup.example")];
const fetched = [fetchedRow(12, 100, "dup.example"), fetchedRow(11, 100, "dup.example")];
const { rows, missed } = mergeGap(buffer, fetched, counter());
expect(missed).toBe(0);
expect(rows).toBe(buffer);
});
/**
* Two queries of the same name from the same client in the same second are
* still separate facts when any stored column differs — the record type or
* class, the response code, the policy that decided them, how long they took,
* the route taken. The gap fetch here returns the differing row *first* and
* the one the buffer already holds second, so an identity blind to the column
* would let the differing row consume the buffered occurrence: the buffered
* query would come back duplicated and the other would vanish, at an
* unchanged `missed`. Order is what exposes that — the count alone is 1
* either way.
*
* `blocked` and `cache_hit` have no case of their own: the server derives
* them from `policy_action` and `route_kind`, so they cannot differ while
* everything else holds, and the two columns they follow are covered here.
*/
test.each([
{ column: "qtype", sections: { request: { qtype: 1 } }, held: { qtype: 1 }, differing: { qtype: 28 } },
{ column: "qclass", sections: { request: { qclass: 1 } }, held: { qclass: 1 }, differing: { qclass: 3 } },
{ column: "rcode", sections: { response: { rcode: 0 } }, held: { rcode: 0 }, differing: { rcode: 2 } },
{
column: "response_time_us",
sections: { response: { duration_us: 1234 } },
held: { response_time_us: 1234 },
differing: { response_time_us: 9999 },
},
{
column: "route_kind",
sections: { route: { kind: "upstream" } },
held: { route_kind: "upstream" },
differing: { route_kind: "forward_zone" },
},
{
column: "policy_action",
sections: { policy: { action: "allow" } },
held: { policy_action: "allow" },
differing: { policy_action: "not_evaluated" },
},
{
column: "policy_reason",
sections: { policy: { reason: "no_match" } },
held: { policy_reason: "no_match" },
differing: { policy_reason: "rule_allow_exact" },
},
] satisfies readonly {
column: string;
sections: Parameters<typeof provenance>[0];
held: Partial<QueryRow>;
differing: Partial<QueryRow>;
}[])("rows differing only in $column survive the gap merge", ({ sections, held, differing }) => {
const buffer = [streamed(1, 100, "dual.example", sections)];
const fetched = [fetchedRow(6, 100, "dual.example", differing), fetchedRow(5, 100, "dual.example", held)];
const { rows, missed } = mergeGap(buffer, fetched, counter());
expect(missed).toBe(1);
expect(rows).toEqual([{ kind: "recovered", key: expect.any(Number), row: fetched[0] }, buffer[0]]);
});
test("recovered rows keep their id and take a fresh key", () => {
const { rows } = mergeGap([], [fetchedRow(77, 100, "gap.example")], counter(200));
const recovered = rows[0];
expect(recovered?.key).toBe(201);
expect(recovered?.kind).toBe("recovered");
expect(recovered !== undefined && recovered.kind === "recovered" ? recovered.row.id : null).toBe(77);
});
test("result is capped at capacity, keeping the newest", () => {
const buffer = [streamed(3, 300, "live.example")];
const fetched = [fetchedRow(2, 302, "g2.example"), fetchedRow(1, 301, "g1.example")];
const { rows, missed } = mergeGap(buffer, fetched, counter(), 2);
expect(missed).toBe(2);
expect(domains(rows)).toEqual(["g2.example", "g1.example"]);
});
test("merged rows stay sorted newest-first by ts", () => {
const buffer = [streamed(4, 105, "after-reopen.example"), streamed(3, 100, "before.example")];
const fetched = [fetchedRow(9, 103, "gap.example")];
const { rows } = mergeGap(buffer, fetched, counter());
expect(rows.map((row) => summaryOf(row).ts)).toEqual([105, 103, 100]);
});
});
+147
View File
@@ -0,0 +1,147 @@
import type { LiveQueryEvent, QueryRow } from "@/lib/types";
import { summarizeEvent, summarizeRow, type QuerySummary } from "@/features/queries/querySummary";
/**
* A row in the live buffer. `key` is a client-side monotonic counter, because
* neither arm has a stable identity of its own on arrival.
*
* The two arms are genuinely different facts, not two encodings of one. A
* streamed frame carries the full provenance of a query the server has not
* written yet; a row recovered by the reconnect gap-fetch is the stored summary
* of a query that *was* written, and cannot fabricate the provenance it never
* received. Only the recovered arm has a row id to link to.
*/
export type LiveRow = { key: number } & (
{ kind: "streamed"; event: LiveQueryEvent } | { kind: "recovered"; row: QueryRow }
);
/** The arm that carries its own provenance, and so its own detail surface. */
export type StreamedRow = Extract<LiveRow, { kind: "streamed" }>;
/** The flat cells both arms render, and the shared identity for gap dedupe. */
export function summaryOf(row: LiveRow): QuerySummary {
return row.kind === "streamed" ? summarizeEvent(row.event) : summarizeRow(row.row);
}
export const RING_CAPACITY = 500;
/** Prepend `row` (rows are newest-first) and drop the oldest beyond `capacity`. */
export function pushRow(rows: LiveRow[], row: LiveRow, capacity: number = RING_CAPACITY): LiveRow[] {
const next = [row, ...rows];
return next.length > capacity ? next.slice(0, capacity) : next;
}
/**
* What the gap merge compares two queries by: the whole stored row bar its id.
*
* `since` on GET /api/queries is inclusive, so the re-sync fetch returns the
* last-seen row(s) again and the merge has to recognise them. The id cannot
* serve as the identity — a streamed frame precedes its own insert and has none
* — so the comparison is by value, and every stored column has to take part.
* Two queries alike in name, client and second but differing in class, rcode,
* the policy that decided them or the route taken are separate facts; if they
* hashed alike, the fetched row that does *not* match the buffered one would
* consume its occurrence, duplicating one query and losing the other.
*
* `QuerySummary` is the wrong basis for that: it is what the table renders, and
* it drops qclass and policy_action. `Omit<QueryRow, "id">`
* instead makes the compiler demand a derivation for every stored column, so a
* column added to the row cannot quietly fall out of the identity.
*/
type GapIdentity = Omit<QueryRow, "id">;
function identityOfRow(row: QueryRow): GapIdentity {
return {
ts: row.ts,
domain: row.domain,
client_ip: row.client_ip,
qtype: row.qtype,
qclass: row.qclass,
rcode: row.rcode,
blocked: row.blocked,
response_time_us: row.response_time_us,
cache_hit: row.cache_hit,
upstream: row.upstream,
policy_action: row.policy_action,
policy_reason: row.policy_reason,
route_kind: row.route_kind,
};
}
/**
* The same identity out of a live frame, which carries every stored column in
* its provenance. The columns the server derives rather than sends — `blocked`
* and `cache_hit` — come through `summarizeEvent` so that derivation keeps
* living in exactly one place.
*/
function identityOfEvent(event: LiveQueryEvent): GapIdentity {
const summary = summarizeEvent(event);
return {
ts: summary.ts,
domain: summary.domain,
client_ip: summary.client_ip,
qtype: summary.qtype,
qclass: event.request.qclass,
rcode: event.response.rcode,
blocked: summary.blocked,
response_time_us: summary.response_time_us,
cache_hit: summary.cache_hit,
upstream: summary.upstream,
policy_action: event.policy.action,
policy_reason: summary.policy_reason,
route_kind: event.route.kind,
};
}
function identityOf(row: LiveRow): GapIdentity {
return row.kind === "streamed" ? identityOfEvent(row.event) : identityOfRow(row.row);
}
/** Sorted keys so the hash cannot depend on the order the two arms happen to build their literals in. */
function signature(identity: GapIdentity): string {
return JSON.stringify(identity, Object.keys(identity).sort());
}
/**
* How many times each signature is already in the buffer. A signature is not
* unique: one client asking for one name twice within the same second is an
* ordinary household pattern, and the two queries are separate facts. Counting
* the occurrences lets the merge drop exactly as many fetched rows as the
* buffer already holds, instead of letting one buffered row hide all of them.
*/
function occurrences(rows: LiveRow[]): Map<string, number> {
const counts = new Map<string, number>();
for (const row of rows) {
const key = signature(identityOf(row));
counts.set(key, (counts.get(key) ?? 0) + 1);
}
return counts;
}
/**
* Merge rows fetched for a reconnect gap (newest-first, from GET /api/queries)
* into the buffer. Each fetched row consumes one buffered occurrence of its
* signature and is skipped; the rest are genuinely missed and `missed` counts
* them. The result stays newest-first (stable sort by ts) and capped.
*/
export function mergeGap(
rows: LiveRow[],
fetched: QueryRow[],
nextKey: () => number,
capacity: number = RING_CAPACITY,
): { rows: LiveRow[]; missed: number } {
const buffered = occurrences(rows);
const added: LiveRow[] = [];
for (const row of fetched) {
const key = signature(identityOfRow(row));
const count = buffered.get(key) ?? 0;
if (count > 0) {
buffered.set(key, count - 1);
continue;
}
added.push({ kind: "recovered", row, key: nextKey() });
}
if (added.length === 0) return { rows, missed: 0 };
const merged = [...added, ...rows].sort((a, b) => summaryOf(b).ts - summaryOf(a).ts).slice(0, capacity);
return { rows: merged, missed: added.length };
}
+113
View File
@@ -0,0 +1,113 @@
import { validateActivitySearch, validateBlocked, validateMode, validateText, validateTimestamp } from "./search";
test("mode is the two-value union, defaulting to history", () => {
expect(validateMode("live")).toBe("live");
expect(validateMode("history")).toBe("history");
expect(validateMode(undefined)).toBe("history");
expect(validateMode("Live")).toBe("history");
expect(validateMode("")).toBe("history");
expect(validateMode(0)).toBe("history");
expect(validateMode(["live"])).toBe("history");
});
test("a bound is a safe integer or nothing at all", () => {
expect(validateTimestamp(1_700_000_000)).toBe(1_700_000_000);
expect(validateTimestamp(0)).toBe(0);
expect(validateTimestamp(-1)).toBe(-1);
});
test.each([
["a fraction", 1_700_000_000.5],
["Infinity", Number.POSITIVE_INFINITY],
["-Infinity", Number.NEGATIVE_INFINITY],
["NaN", Number.NaN],
["past the safe range", Number.MAX_SAFE_INTEGER + 1],
["1e21", 1e21],
["a numeric string", "1700000000"],
["an empty string", ""],
["null", null],
["undefined", undefined],
["a boolean", true],
["an array", [1_700_000_000]],
["a bigint", 1_700_000_000n],
])("a bound rejects %s", (_name, value) => {
expect(validateTimestamp(value)).toBeUndefined();
});
test("blocked keeps false, which is the allowed-only filter", () => {
expect(validateBlocked(true)).toBe(true);
expect(validateBlocked(false)).toBe(false);
});
test.each([
["the string true", "true"],
["the string false", "false"],
["1", 1],
["0", 0],
["null", null],
["undefined", undefined],
])("blocked rejects %s", (_name, value) => {
expect(validateBlocked(value)).toBeUndefined();
});
test("a text filter is trimmed, and an empty one is no filter", () => {
expect(validateText("ads.example")).toBe("ads.example");
expect(validateText(" ads.example ")).toBe("ads.example");
expect(validateText("")).toBeUndefined();
expect(validateText(" ")).toBeUndefined();
expect(validateText("\t\n")).toBeUndefined();
expect(validateText(42)).toBeUndefined();
expect(validateText(undefined)).toBeUndefined();
});
test("a whole search normalizes every field and drops nothing else in", () => {
expect(
validateActivitySearch({
mode: "live",
since: 1_700_000_000,
until: 1_700_000_600,
domain: " ads.example ",
client: "192.0.2.10",
blocked: false,
unknown: "kept out",
}),
).toEqual({
mode: "live",
since: 1_700_000_000,
until: 1_700_000_600,
domain: "ads.example",
client: "192.0.2.10",
blocked: false,
});
});
test("an empty search is history with no filters", () => {
expect(validateActivitySearch({})).toEqual({
mode: "history",
since: undefined,
until: undefined,
domain: undefined,
client: undefined,
blocked: undefined,
});
});
test("a search of junk applies nothing", () => {
expect(
validateActivitySearch({
mode: "HISTORY ",
since: "1700000000",
until: Number.POSITIVE_INFINITY,
domain: " ",
client: null,
blocked: "true",
}),
).toEqual({
mode: "history",
since: undefined,
until: undefined,
domain: undefined,
client: undefined,
blocked: undefined,
});
});
+88
View File
@@ -0,0 +1,88 @@
/**
* The Activity search parameters, validated as pure functions so the route's
* `validateSearch` stays a one-liner and every rejection is testable without a
* router.
*
* A search value arrives from a URL, from history state, or from a hand-typed
* link, so nothing about its type is given. Anything that is not exactly the
* value the API can filter on becomes `undefined`: an unbounded page is honest,
* a page filtered on a coerced guess is not.
*/
import type { QueriesFilter } from "@/lib/types";
export const ACTIVITY_MODES = ["history", "live"] as const;
export type ActivityMode = (typeof ACTIVITY_MODES)[number];
export interface ActivitySearch {
mode: ActivityMode;
since: number | undefined;
until: number | undefined;
domain: string | undefined;
client: string | undefined;
blocked: boolean | undefined;
}
/** History is the surface a bare `/activity` should open on: it answers questions. */
export function validateMode(value: unknown): ActivityMode {
return value === "live" ? "live" : "history";
}
/**
* A unix-second bound. `Number.isSafeInteger` is the whole test: it rejects a
* fraction, an infinity, a NaN and a magnitude past 2^53 in one step, and a
* string never passes, so `?since=now` cannot reach the API as garbage.
*/
export function validateTimestamp(value: unknown): number | undefined {
return Number.isSafeInteger(value) ? (value as number) : undefined;
}
/**
* The blocked filter. `false` is a real filter — "allowed only" — so it must
* survive; only a genuine boolean does, because `"false"` out of a URL parser
* that did not decode JSON would otherwise read as true.
*/
export function validateBlocked(value: unknown): boolean | undefined {
return typeof value === "boolean" ? value : undefined;
}
/**
* A text filter, trimmed. An empty result becomes `undefined` rather than `""`:
* the server treats an empty filter as no filter, and a URL that showed
* `domain=` as applied state would claim a filter that is not filtering.
*/
export function validateText(value: unknown): string | undefined {
if (typeof value !== "string") return undefined;
const trimmed = value.trim();
return trimmed === "" ? undefined : trimmed;
}
/**
* The API filter for a validated search, built field by field.
*
* Only the fields that are actually set are written, so an unfiltered request
* carries no keys at all: `GET /api/queries` rejects a parameter it does not
* know, and the infinite query's cache key is the filter object, so a key
* present-but-undefined and a key absent must not be two different windows onto
* the same rows. `mode` never appears — it selects the surface, not the rows.
*/
export function queriesFilterOf(search: Omit<ActivitySearch, "mode">): QueriesFilter {
const filter: QueriesFilter = {};
if (search.domain !== undefined) filter.domain = search.domain;
if (search.client !== undefined) filter.client = search.client;
if (search.blocked !== undefined) filter.blocked = search.blocked;
if (search.since !== undefined) filter.since = search.since;
if (search.until !== undefined) filter.until = search.until;
return filter;
}
export function validateActivitySearch(search: Record<string, unknown>): ActivitySearch {
return {
mode: validateMode(search["mode"]),
since: validateTimestamp(search["since"]),
until: validateTimestamp(search["until"]),
domain: validateText(search["domain"]),
client: validateText(search["client"]),
blocked: validateBlocked(search["blocked"]),
};
}
@@ -1,6 +1,8 @@
import { act, renderHook, waitFor } from "@testing-library/react";
import { ApiError } from "@/lib/api";
import type { LiveQueryEvent, QueriesPage, QueryRow } from "@/lib/types";
import type { QueriesPage, QueryRow } from "@/lib/types";
import { provenance, queryRow } from "@/features/queries/provenanceFixture";
import { summaryOf, type LiveRow } from "./ringBuffer";
import { FakeEventSource } from "./fakeEventSource";
import { CAP_ERROR_THRESHOLD, useLiveQueries } from "./useLiveQueries";
@@ -8,41 +10,28 @@ afterEach(() => vi.unstubAllGlobals());
function stubLocationAssign() {
const assign = vi.fn();
vi.stubGlobal("location", { pathname: "/live", search: "", assign });
vi.stubGlobal("location", { pathname: "/activity", search: "?mode=live", assign });
return assign;
}
function frame(ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): { data: string } {
const payload: LiveQueryEvent = {
ts,
domain,
client_ip: "192.0.2.10",
qtype: 1,
blocked: false,
block_reason: "",
response_time_us: 500,
cache_hit: false,
upstream: "udp://9.9.9.9:53",
...overrides,
};
function frame(ts: number, domain: string): { data: string } {
const payload = provenance({
request: { time: ts, domain },
route: { upstream: "udp://9.9.9.9:53" },
});
return { data: JSON.stringify(payload) };
}
function fetchedRow(id: number, ts: number, domain: string): QueryRow {
return {
id,
ts,
domain,
client_ip: "192.0.2.10",
qtype: 1,
blocked: false,
block_reason: "",
response_time_us: 500,
cache_hit: false,
upstream: "udp://9.9.9.9:53",
};
return queryRow(id, { ts, domain, upstream: "udp://9.9.9.9:53" });
}
function domains(rows: LiveRow[]): string[] {
return rows.map((row) => summaryOf(row).domain);
}
const FULL_COVERAGE = { complete: true, available_since: 0 };
function setup(fetchSince?: (since: number) => Promise<QueriesPage>, probeSession?: () => Promise<unknown>) {
const sources: FakeEventSource[] = [];
const createEventSource = (url: string) => {
@@ -68,7 +57,7 @@ test("open then frames: rows newest-first with increasing keys", () => {
sources[0]!.emit("query", frame(1001, "b.example"));
});
const rows = hook.result.current.rows;
expect(rows.map((r) => r.domain)).toEqual(["b.example", "a.example"]);
expect(domains(rows)).toEqual(["b.example", "a.example"]);
expect(rows[0]!.key).toBeGreaterThan(rows[1]!.key);
});
@@ -87,6 +76,7 @@ test("error then reopen re-syncs the gap since the last seen ts", async () => {
return Promise.resolve({
queries: [fetchedRow(9, 1002, "gap.example"), fetchedRow(8, since, "a.example")],
next_before: null,
coverage: FULL_COVERAGE,
});
});
const { sources, hook } = setup(fetchSince);
@@ -103,7 +93,7 @@ test("error then reopen re-syncs the gap since the last seen ts", async () => {
expect(fetchSince).toHaveBeenCalledWith(1000);
await waitFor(() => expect(hook.result.current.missed).toBe(1));
expect(hook.result.current.rows.map((r) => r.domain)).toEqual(["gap.example", "a.example"]);
expect(domains(hook.result.current.rows)).toEqual(["gap.example", "a.example"]);
act(() => hook.result.current.dismissMissed());
expect(hook.result.current.missed).toBeNull();
@@ -127,7 +117,7 @@ test("a 401 gap re-sync redirects to login instead of setting resyncFailed", asy
act(() => sources[0]!.emit("query", frame(1000, "a.example")));
act(() => sources[0]!.emit("error"));
act(() => sources[0]!.emit("open"));
await waitFor(() => expect(assign).toHaveBeenCalledWith("/login?redirect=%2Flive"));
await waitFor(() => expect(assign).toHaveBeenCalledWith("/login?redirect=%2Factivity%3Fmode%3Dlive"));
expect(hook.result.current.resyncFailed).toBe(false);
});
@@ -152,7 +142,7 @@ test("cap trip with an expired session redirects to login", async () => {
act(() => {
for (let i = 0; i < CAP_ERROR_THRESHOLD; i++) sources[0]!.emit("error");
});
await waitFor(() => expect(assign).toHaveBeenCalledWith("/login?redirect=%2Flive"));
await waitFor(() => expect(assign).toHaveBeenCalledWith("/login?redirect=%2Factivity%3Fmode%3Dlive"));
expect(probeSession).toHaveBeenCalledTimes(1);
});
@@ -193,7 +183,7 @@ test("a fatal rejection with an expired session redirects to login", async () =>
act(() => sources[0]!.failFatal());
await waitFor(() => expect(assign).toHaveBeenCalledWith("/login?redirect=%2Flive"));
await waitFor(() => expect(assign).toHaveBeenCalledWith("/login?redirect=%2Factivity%3Fmode%3Dlive"));
expect(probeSession).toHaveBeenCalledTimes(1);
});
@@ -235,12 +225,12 @@ test("freeze keeps the display fixed while the buffer keeps filling", () => {
sources[0]!.emit("query", frame(1001, "b.example"));
sources[0]!.emit("query", frame(1002, "c.example"));
});
expect(hook.result.current.rows.map((r) => r.domain)).toEqual(["a.example"]);
expect(domains(hook.result.current.rows)).toEqual(["a.example"]);
expect(hook.result.current.liveCount).toBe(3);
act(() => hook.result.current.toggleFreeze());
expect(hook.result.current.frozen).toBe(false);
expect(hook.result.current.rows.map((r) => r.domain)).toEqual(["c.example", "b.example", "a.example"]);
expect(domains(hook.result.current.rows)).toEqual(["c.example", "b.example", "a.example"]);
});
test("stale sources are ignored after retry and closed on unmount", () => {
@@ -121,8 +121,12 @@ export function useLiveQueries(options?: LiveQueriesOptions): LiveQueries {
} catch {
return;
}
lastSeenTsRef.current = payload.ts;
bufferRef.current = pushRow(bufferRef.current, { ...payload, key: ++keyRef.current });
lastSeenTsRef.current = payload.request.time;
bufferRef.current = pushRow(bufferRef.current, {
kind: "streamed",
event: payload,
key: ++keyRef.current,
});
setRows(bufferRef.current);
});
+17 -7
View File
@@ -37,17 +37,27 @@ export function useClientNames(): ClientNames {
);
}
export function ClientName({ ip, names }: { ip: string; names: ClientNames }) {
/**
* The name the loaded list gives this address right now, or null when it gives
* none. Callers that must distinguish "named" from "bare address" — rather than
* just render whichever applies — read this instead of re-deriving precedence.
*/
export function clientLabel(ip: string, names: ClientNames): { text: string; learned: boolean } | null {
const client = names.get(ip);
if (client === undefined || (client.name === "" && client.learned_name === "")) {
return <span {...stylex.props(shared.mono)}>{ip}</span>;
}
if (client === undefined) return null;
if (client.name !== "") return { text: client.name, learned: false };
if (client.learned_name !== "") return { text: client.learned_name, learned: true };
return null;
}
export function ClientName({ ip, names }: { ip: string; names: ClientNames }) {
const label = clientLabel(ip, names);
if (label === null) return <span {...stylex.props(shared.mono)}>{ip}</span>;
// The name replaces the address on screen, so the address stays reachable
// as the tooltip rather than disappearing from the row entirely.
if (client.name !== "") return <span title={ip}>{client.name}</span>;
return (
<span title={ip} {...stylex.props(shared.learnedName)}>
{client.learned_name}
<span title={ip} {...stylex.props(label.learned && shared.learnedName)}>
{label.text}
</span>
);
}
@@ -18,6 +18,7 @@ const RESPONSES: Record<string, unknown> = {
cached: 100,
clients: 7,
avg_response_time_us: 2345,
coverage: { complete: true, available_since: 0 },
},
"/api/stats/timeseries?period=24h": {
period: "24h",
@@ -29,6 +30,7 @@ const RESPONSES: Record<string, unknown> = {
{ ts: 1800, queries: 40, blocked: 0, cached: 0 },
{ ts: 3600, queries: 0, blocked: 0, cached: 0 },
],
coverage: { complete: true, available_since: 0 },
},
"/api/stats?period=1h": {
period: "1h",
@@ -39,6 +41,7 @@ const RESPONSES: Record<string, unknown> = {
cached: 0,
clients: 2,
avg_response_time_us: null,
coverage: { complete: true, available_since: 0 },
},
"/api/stats/timeseries?period=1h": {
period: "1h",
@@ -46,6 +49,7 @@ const RESPONSES: Record<string, unknown> = {
until: 3600,
bucket_seconds: 60,
buckets: [],
coverage: { complete: true, available_since: 0 },
},
"/api/health": {
status: "degraded",
@@ -193,7 +197,6 @@ test("dashboard renders stats, chart, disk card, upstream table and health banne
expect(screen.getByText("https://dns.example/dns-query")).toBeTruthy();
expect(screen.getByText("90.0%")).toBeTruthy();
expect(screen.getByText("100.0%")).toBeTruthy();
expect(screen.getByText("timeout · 3h ago")).toBeTruthy();
expect(screen.getByText("1/2 available")).toBeTruthy();
});
@@ -3,6 +3,7 @@ import { keepPreviousData, useQuery } from "@tanstack/react-query";
import * as stylex from "@stylexjs/stylex";
import { healthQuery, statsQuery, timeseriesQuery, upstreamHealthQuery } from "@/lib/queries";
import type { Period } from "@/lib/types";
import CoverageNotice from "@/lib/CoverageNotice";
import InlineError from "@/lib/InlineError";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
@@ -142,6 +143,10 @@ export default function DashboardPage() {
<StatCards stats={stats.data} />
)}
{/* One notice for the period: the chart is judged against the same
aligned lower bound as the totals, so it would say the same thing. */}
{stats.data !== undefined && <CoverageNotice coverage={stats.data.coverage} />}
<div {...stylex.props(styles.panelGrid)}>
<section {...stylex.props(styles.panel)}>
<h2 {...stylex.props(styles.panelHeading)}>Queries over time</h2>
@@ -0,0 +1,52 @@
import { render, screen, within } from "@testing-library/react";
import * as stylex from "@stylexjs/stylex";
import type { StatsTimeseries } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import TimeseriesChart from "./TimeseriesChart";
const SINCE = 1_700_000_000;
function timeseries(bucketCount: number): StatsTimeseries {
return {
period: "24h",
since: SINCE,
until: SINCE + bucketCount * 1800,
bucket_seconds: 1800,
coverage: { complete: true, available_since: SINCE },
buckets: Array.from({ length: bucketCount }, (_, i) => ({
ts: SINCE + i * 1800,
queries: i + 1,
blocked: 1,
cached: 1,
})),
};
}
/** The element wearing the shared hidden style, found by its compiled classes. */
function hiddenElement(container: HTMLElement): Element | null {
const classes = stylex.props(shared.srOnly).className?.split(" ").filter(Boolean) ?? [];
expect(classes.length).toBeGreaterThan(0);
return container.querySelector(classes.map((name) => `.${name}`).join(""));
}
test("the data table is the SVG's accessible equivalent", () => {
render(<TimeseriesChart data={timeseries(3)} />);
expect(screen.getByRole("img", { name: /queries over time/i })).toBeTruthy();
const table = screen.getByRole("table", { name: "Queries per time bucket" });
expect(within(table).getAllByRole("row").length).toBe(4);
});
/**
* `overflow` does not apply to a table box and `height` on one is a minimum, so
* the hidden style has to sit on a block container wrapping the table. Worn by
* the table itself it clips the paint but not the layout, and 48 invisible rows
* push the document's scroll height a screen past the app shell.
*/
test("the hidden data table is clipped by a block wrapper, not by the table itself", () => {
const { container } = render(<TimeseriesChart data={timeseries(48)} />);
const hidden = hiddenElement(container);
expect(hidden?.tagName).toBe("DIV");
expect(hidden?.querySelector("table")).not.toBeNull();
});
@@ -292,29 +292,31 @@ export default function TimeseriesChart({ data }: { data: StatsTimeseries }) {
</li>
))}
</ul>
<table {...stylex.props(shared.srOnly)}>
<caption>Queries per time bucket</caption>
<thead>
<tr>
<th scope="col">Time</th>
<th scope="col">Queries</th>
<th scope="col">Blocked</th>
<th scope="col">Cached</th>
<th scope="col">Other</th>
</tr>
</thead>
<tbody>
{layout.bars.map((bar) => (
<tr key={bar.bucket.ts}>
<th scope="row">{formatTime(bar.bucket.ts)}</th>
<td>{bar.bucket.queries}</td>
<td>{bar.bucket.blocked}</td>
<td>{bar.bucket.cached}</td>
<td>{bar.other}</td>
<div {...stylex.props(shared.srOnly)}>
<table>
<caption>Queries per time bucket</caption>
<thead>
<tr>
<th scope="col">Time</th>
<th scope="col">Queries</th>
<th scope="col">Blocked</th>
<th scope="col">Cached</th>
<th scope="col">Other</th>
</tr>
))}
</tbody>
</table>
</thead>
<tbody>
{layout.bars.map((bar) => (
<tr key={bar.bucket.ts}>
<th scope="row">{formatTime(bar.bucket.ts)}</th>
<td>{bar.bucket.queries}</td>
<td>{bar.bucket.blocked}</td>
<td>{bar.bucket.cached}</td>
<td>{bar.other}</td>
</tr>
))}
</tbody>
</table>
</div>
</div>
);
}
@@ -4,14 +4,6 @@ import UpstreamHealthTable from "./UpstreamHealthTable";
const NOW_S = 1_700_000_000;
beforeEach(() => {
vi.spyOn(Date, "now").mockReturnValue(NOW_S * 1000);
});
afterEach(() => {
vi.restoreAllMocks();
});
const ZERO: UpstreamPeriodStats = {
attempts: 0,
successes: 0,
@@ -27,7 +19,6 @@ function period(overrides: Partial<UpstreamPeriodStats> = {}): UpstreamPeriodSta
successes: 90,
failures: 10,
success_rate: 0.9,
// 3h30m ago, far from a unit boundary.
last_failure_at: NOW_S - 12_600,
last_failure_error: "Timeout",
...overrides,
@@ -69,7 +60,7 @@ test("the ranged columns sit under a header naming the selected period", () => {
renderTable([entry()]);
expect(screen.getByRole("columnheader", { name: "Selected period · 24h" })).toBeTruthy();
for (const name of ["Upstream", "Status now", "Attempts", "Failures", "Success rate", "Last failure"]) {
for (const name of ["Upstream", "Status now", "Attempts", "Failures", "Success rate"]) {
expect(screen.getByRole("columnheader", { name })).toBeTruthy();
}
@@ -78,6 +69,14 @@ test("the ranged columns sit under a header naming the selected period", () => {
expect(screen.queryByRole("columnheader", { name: "Available" })).toBeNull();
});
test("failure detail is the Diagnostics page's job; the card never shows it", () => {
renderTable([entry()]);
expect(screen.queryByRole("columnheader", { name: "Last failure" })).toBeNull();
expect(screen.queryByText(/Timeout/)).toBeNull();
expect(screen.queryByText(/ago$/)).toBeNull();
});
test("status now is one word from live state, not from the window", () => {
renderTable([
entry({ url: "https://a.example/dns-query" }),
@@ -90,20 +89,7 @@ test("status now is one word from live state, not from the window", () => {
expect(within(rowOf("https://c.example/dns-query")).getByText("Disabled")).toBeTruthy();
});
test("last failure pairs the error name with its age, em-dash when the window holds none", () => {
renderTable([
entry({ url: "https://a.example/dns-query" }),
entry({
url: "https://b.example/dns-query",
period: period({ last_failure_at: null, last_failure_error: null }),
}),
]);
expect(within(rowOf("https://a.example/dns-query")).getByText("Timeout · 3h ago")).toBeTruthy();
expect(within(rowOf("https://b.example/dns-query")).getByText("—")).toBeTruthy();
});
test("a window with no attempts renders em-dashes and never a perfect rate", () => {
test("a window with no attempts renders an em-dash and never a perfect rate", () => {
renderTable([entry({ period: ZERO })]);
const cells = within(rowOf("https://dns.example/dns-query")).getAllByRole("cell");
@@ -113,7 +99,6 @@ test("a window with no attempts renders em-dashes and never a perfect rate", ()
"0",
"0",
"—",
"—",
]);
expect(screen.queryByText("100.0%")).toBeNull();
expect(screen.queryByText("0.0%")).toBeNull();
@@ -1,5 +1,4 @@
import * as stylex from "@stylexjs/stylex";
import { formatAge } from "@/lib/format";
import type { UpstreamHealth, UpstreamHealthEntry, UpstreamPeriodStats } from "@/lib/types";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
@@ -137,19 +136,11 @@ function successRate(period: UpstreamPeriodStats): string {
}
/**
* The age is formatted once, when the row renders; nothing here ticks. It is
* measured against the browser's clock rather than the response's `until`, so a
* cached response ages visibly instead of freezing at the moment it was served.
* The dashboard answers availability only. Failure detail what failed, when,
* and how often is the Diagnostics page's job, so `last_failure_at` and
* `last_failure_error` are read there rather than repeated in this row.
*/
function lastFailure(period: UpstreamPeriodStats, nowSeconds: number): string {
if (period.last_failure_at === null) return "—";
const age = formatAge(Math.max(0, nowSeconds - period.last_failure_at));
const error = period.last_failure_error;
return error === null || error === "" ? age : `${error} · ${age}`;
}
export default function UpstreamHealthTable({ health }: { health: UpstreamHealth }) {
const nowSeconds = Math.floor(Date.now() / 1000);
const idle = health.upstreams.length > 0 && health.upstreams.every(({ period }) => period.attempts === 0);
return (
@@ -168,7 +159,7 @@ export default function UpstreamHealthTable({ health }: { health: UpstreamHealth
<thead>
<tr {...stylex.props(styles.groupRow)}>
<td colSpan={2} />
<th scope="colgroup" colSpan={4} {...stylex.props(styles.groupHead)}>
<th scope="colgroup" colSpan={3} {...stylex.props(styles.groupHead)}>
Selected period · {health.period}
</th>
</tr>
@@ -188,9 +179,6 @@ export default function UpstreamHealthTable({ health }: { health: UpstreamHealth
<th scope="col" {...stylex.props(styles.th, styles.thRight)}>
Success rate
</th>
<th scope="col" {...stylex.props(styles.th)}>
Last failure
</th>
</tr>
</thead>
<tbody>
@@ -220,9 +208,6 @@ export default function UpstreamHealthTable({ health }: { health: UpstreamHealth
<td {...stylex.props(styles.cell, styles.cellRight, shared.tabularNums)}>
{successRate(upstream.period)}
</td>
<td {...stylex.props(styles.cell, styles.small, styles.muted)}>
{lastFailure(upstream.period, nowSeconds)}
</td>
</tr>
);
})}
@@ -337,3 +337,38 @@ test("an episode links to its own detail page", async () => {
const link = await screen.findByRole("link", { name: "Blocklist source failed to update" });
expect(link.getAttribute("href")).toBe("/diagnostics/42");
});
test("an absolute window reaches both requests and is stated on the page", async () => {
const bounded = "since=1699999700&until=1700000300";
responses[`/api/diagnostics?${bounded}&state=active`] = ACTIVE;
responses[`/api/diagnostics?${bounded}&state=resolved`] = RESOLVED;
renderRoute(`/diagnostics?${bounded}`);
await screen.findByRole("heading", { name: "Active" });
expect(requested).toContain(`/api/diagnostics?${bounded}&state=active`);
expect(requested).toContain(`/api/diagnostics?${bounded}&state=resolved`);
// An empty section inside a five-minute window means something different
// from an empty section over the whole history, so the page has to say so.
expect(screen.getByText(/Showing events that overlap/)).toBeTruthy();
});
test("clearing the range drops both bounds from the url", async () => {
const bounded = "since=1699999700&until=1700000300";
responses[`/api/diagnostics?${bounded}&state=active`] = ACTIVE;
responses[`/api/diagnostics?${bounded}&state=resolved`] = RESOLVED;
const router = renderRoute(`/diagnostics?${bounded}`);
fireEvent.click(await screen.findByRole("button", { name: "Clear the time range" }));
await waitFor(() => {
expect(router.state.location.search).not.toContain("since");
});
expect(router.state.location.search).not.toContain("until");
});
test("a bound that is not a whole second is dropped, leaving the page unbounded", async () => {
renderRoute("/diagnostics?since=1.5&until=Infinity");
await screen.findByRole("heading", { name: "Active" });
expect(requested).toContain("/api/diagnostics?state=active");
expect(screen.queryByText(/Showing events that overlap/)).toBeNull();
});
@@ -12,18 +12,13 @@ import * as api from "@/lib/api";
import InlineError from "@/lib/InlineError";
import { formatDuration, formatTime } from "@/lib/format";
import { diagnosticPurgeMutation, diagnosticsInfiniteQuery, diagnosticsPurgeResolvedMutation } from "@/lib/queries";
import type {
DiagnosticEvent,
DiagnosticSeverity,
DiagnosticState,
DiagnosticsFilter,
DiagnosticsPage as Page,
} from "@/lib/types";
import type { DiagnosticEvent, DiagnosticSeverity, DiagnosticState, DiagnosticsPage as Page } from "@/lib/types";
import ConfirmDialog from "@/ui/ConfirmDialog";
import Select from "@/ui/Select";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
import SeverityBadge from "./SeverityBadge";
import { diagnosticsFilterOf } from "./filter";
import { DIAGNOSTIC_COMPONENTS, componentLabel, copyFor } from "./eventCopy";
const DARK = "@media (prefers-color-scheme: dark)";
@@ -138,6 +133,19 @@ const styles = stylex.create({
lineHeight: "1rem",
color: colors.textMuted,
},
rangeNotice: {
marginTop: "0.75rem",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
backgroundColor: colors.surfaceHover,
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textSecondary,
},
tableWrap: {
marginTop: "0.75rem",
overflowX: "auto",
@@ -263,6 +271,33 @@ function MoreButton({ section }: { section: Section }) {
);
}
/**
* The window the page is bounded to, whenever it is bounded.
*
* A link from a query detail arrives with an absolute five-minute window, and
* an empty Active section inside it means something very different from an
* empty Active section over the whole history. The page has to say which it is
* showing, and offer the way out of it.
*/
function RangeNotice({ since, until }: { since?: number; until?: number }) {
const navigate = useNavigate({ from: "/diagnostics" });
if (since === undefined && until === undefined) return null;
const from = since === undefined ? "the start of the history" : formatTime(since);
const to = until === undefined ? "now" : formatTime(until);
return (
<p role="status" {...stylex.props(styles.rangeNotice)}>
Showing events that overlap {from} to {to}.{" "}
<button
type="button"
onClick={() => void navigate({ search: (prev) => ({ ...prev, since: undefined, until: undefined }) })}
{...stylex.props(shared.linkButton, shared.focusRing)}
>
Clear the time range
</button>
</p>
);
}
function ActiveCard({ event, now }: { event: DiagnosticEvent; now: number }) {
const copy = copyFor(event.code);
return (
@@ -327,9 +362,7 @@ export default function DiagnosticsPage() {
const navigate = useNavigate({ from: "/diagnostics" });
const state = search.state ?? "all";
const base: DiagnosticsFilter = {};
if (search.severity !== undefined) base.severity = search.severity;
if (search.component !== undefined) base.component = search.component;
const base = diagnosticsFilterOf(search);
const active = useInfiniteQuery(diagnosticsInfiniteQuery({ ...base, state: "active" }, state !== "resolved"));
const history = useInfiniteQuery(diagnosticsInfiniteQuery({ ...base, state: "resolved" }, state !== "active"));
@@ -368,6 +401,8 @@ export default function DiagnosticsPage() {
repeats, and closes when the subject recovers.
</p>
<RangeNotice since={search.since} until={search.until} />
<div {...stylex.props(styles.filterGrid)}>
<Select
variant="compactField"
+28
View File
@@ -0,0 +1,28 @@
/**
* The Diagnostics search parameters and the API filter they build.
*
* The route, the page and the two infinite queries all have to agree on what
* the URL asked for the page renders the same window the loader prefetched
* so the projection lives in one place rather than being spelled out at each.
*/
import type { DiagnosticSeverity, DiagnosticState, DiagnosticsFilter } from "@/lib/types";
export interface DiagnosticsSearch {
state?: DiagnosticState;
severity?: DiagnosticSeverity;
component?: string;
/** Unix seconds, inclusive. An episode qualifies when its interval overlaps. */
since?: number;
until?: number;
}
/** Field by field, so an unset filter is an absent key rather than `undefined`. */
export function diagnosticsFilterOf(search: DiagnosticsSearch): DiagnosticsFilter {
const filter: DiagnosticsFilter = {};
if (search.severity !== undefined) filter.severity = search.severity;
if (search.component !== undefined) filter.component = search.component;
if (search.since !== undefined) filter.since = search.since;
if (search.until !== undefined) filter.until = search.until;
return filter;
}
@@ -1,194 +0,0 @@
import { act, fireEvent, render, screen, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { createQueryClient } from "@/lib/queryClient";
import type { Client, LiveQueryEvent } from "@/lib/types";
import { FakeEventSource } from "./fakeEventSource";
import LiveLogPage from "./LiveLogPage";
function client(ip: string, name: string, learnedName: string): Client {
return {
id: Number(ip.split(".").pop()),
ip,
name,
learned_name: learnedName,
group_id: 1,
group: "default",
hand_edited: name !== "",
first_seen: 1_700_000_000,
last_seen: 1_700_000_100,
};
}
const CLIENTS: Client[] = [
client("192.0.2.10", "Kitchen Pi", "pi.lan"),
client("192.0.2.11", "", "laptop.lan"),
client("192.0.2.12", "", ""),
];
beforeEach(() => {
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
if (String(input) !== "/api/clients") {
return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
}
return new Response(JSON.stringify({ clients: CLIENTS }), {
status: 200,
headers: { "content-type": "application/json" },
});
}),
);
});
afterEach(() => {
vi.unstubAllGlobals();
});
function frame(ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): { data: string } {
const payload: LiveQueryEvent = {
ts,
domain,
client_ip: "192.0.2.10",
qtype: 1,
blocked: false,
block_reason: "",
response_time_us: 500,
cache_hit: true,
upstream: "",
...overrides,
};
return { data: JSON.stringify(payload) };
}
function renderPage() {
const sources: FakeEventSource[] = [];
const createEventSource = (url: string) => {
const es = new FakeEventSource(url);
sources.push(es);
return es;
};
render(
<QueryClientProvider client={createQueryClient()}>
<LiveLogPage createEventSource={createEventSource} />
</QueryClientProvider>,
);
return sources;
}
test("streams rows, flags blocked ones, and freezes the display", () => {
const sources = renderPage();
expect(screen.getByText("Connecting…")).toBeTruthy();
act(() => sources[0]!.emit("open"));
expect(screen.getByRole("status", { name: "Live" })).toBeTruthy();
expect(screen.getByText("Waiting for queries…")).toBeTruthy();
act(() => {
sources[0]!.emit("query", frame(1000, "ok.example"));
sources[0]!.emit(
"query",
frame(1001, "ads.example", { blocked: true, block_reason: "blocklist:stevenblack", qtype: 28 }),
);
});
expect(screen.getByText("ok.example")).toBeTruthy();
expect(screen.getByText("Blocked")).toBeTruthy();
expect(screen.getByText("blocklist:stevenblack")).toBeTruthy();
expect(screen.getByText("AAAA")).toBeTruthy();
// StyleX compiles to opaque class names, so the check is structural: a blocked
// row carries every class a plain row does, plus the ones the flag adds.
const blockedRow = screen.getByText("ads.example").closest("tr");
const plainRow = screen.getByText("ok.example").closest("tr");
const blockedClasses = new Set(blockedRow?.className.split(" "));
const plainClasses = plainRow?.className.split(" ") ?? [];
expect(plainClasses.every((name) => blockedClasses.has(name))).toBe(true);
expect(blockedClasses.size).toBeGreaterThan(plainClasses.length);
const freeze = screen.getByRole("button", { name: "Freeze" });
fireEvent.click(freeze);
expect(freeze.getAttribute("aria-pressed")).toBe("true");
act(() => sources[0]!.emit("query", frame(1002, "later.example")));
expect(screen.queryByText("later.example")).toBeNull();
expect(screen.getByText(/3 in buffer/)).toBeTruthy();
fireEvent.click(screen.getByRole("button", { name: "Resume" }));
expect(screen.getByText("later.example")).toBeTruthy();
});
test("resolves each row's client to its display name, keeping the IP as the tooltip", async () => {
const sources = renderPage();
act(() => sources[0]!.emit("open"));
act(() => {
sources[0]!.emit("query", frame(1000, "named.example", { client_ip: "192.0.2.10" }));
sources[0]!.emit("query", frame(1001, "learned.example", { client_ip: "192.0.2.11" }));
sources[0]!.emit("query", frame(1002, "nameless.example", { client_ip: "192.0.2.12" }));
sources[0]!.emit("query", frame(1003, "stranger.example", { client_ip: "192.0.2.99" }));
});
// A hand-typed name wins outright; the learned name never surfaces for it.
const named = await screen.findByText("Kitchen Pi");
expect(named.getAttribute("title")).toBe("192.0.2.10");
expect(screen.queryByText("pi.lan")).toBeNull();
// A learned name reads muted and nothing more here: the "learned" tag would
// repeat on every row of the table, so the Clients page carries it instead.
const learned = screen.getByText("laptop.lan");
expect(learned.getAttribute("title")).toBe("192.0.2.11");
expect(within(learned.closest("tr")!).queryByText("learned")).toBeNull();
// A known client with neither name, and a client the loaded list has never
// seen, both fall back to the bare address with no tooltip standing in.
const nameless = screen.getByText("192.0.2.12");
expect(nameless.getAttribute("title")).toBeNull();
const stranger = screen.getByText("192.0.2.99");
expect(stranger.getAttribute("title")).toBeNull();
expect(screen.getByText("stranger.example").closest("tr")?.textContent).toContain("192.0.2.99");
});
test("rows stream in as bare IPs while the client list is still loading", async () => {
let releaseClients: () => void = () => {};
vi.stubGlobal(
"fetch",
vi.fn(
(input: RequestInfo | URL) =>
new Promise<Response>((resolve) => {
if (String(input) !== "/api/clients") {
resolve(new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 }));
return;
}
releaseClients = () =>
resolve(
new Response(JSON.stringify({ clients: CLIENTS }), {
status: 200,
headers: { "content-type": "application/json" },
}),
);
}),
),
);
const sources = renderPage();
act(() => sources[0]!.emit("open"));
act(() => sources[0]!.emit("query", frame(1000, "named.example", { client_ip: "192.0.2.10" })));
expect(screen.getByText("192.0.2.10")).toBeTruthy();
expect(screen.queryByText("Kitchen Pi")).toBeNull();
releaseClients();
expect(await screen.findByText("Kitchen Pi")).toBeTruthy();
});
test("repeated connection failures show the viewer-cap state with a retry button", () => {
const sources = renderPage();
act(() => {
sources[0]!.emit("error");
sources[0]!.emit("error");
sources[0]!.emit("error");
});
expect(screen.getByRole("alert").textContent).toContain("too many live viewers");
fireEvent.click(screen.getByRole("button", { name: "Retry" }));
expect(sources).toHaveLength(2);
expect(screen.getByText("Connecting…")).toBeTruthy();
});
-101
View File
@@ -1,101 +0,0 @@
import type { LiveQueryEvent, QueryRow } from "@/lib/types";
import { RING_CAPACITY, mergeGap, pushRow, type LiveRow } from "./ringBuffer";
function event(ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): LiveQueryEvent {
return {
ts,
domain,
client_ip: "192.0.2.10",
qtype: 1,
blocked: false,
block_reason: "",
response_time_us: 500,
cache_hit: false,
upstream: "udp://9.9.9.9:53",
...overrides,
};
}
function liveRow(key: number, ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): LiveRow {
return { ...event(ts, domain, overrides), key };
}
function fetchedRow(id: number, ts: number, domain: string, overrides: Partial<LiveQueryEvent> = {}): QueryRow {
return { id, ...event(ts, domain, overrides) };
}
function counter(start = 100): () => number {
let n = start;
return () => ++n;
}
describe("pushRow", () => {
test("prepends newest-first", () => {
let rows: LiveRow[] = [];
rows = pushRow(rows, liveRow(1, 10, "a.example"));
rows = pushRow(rows, liveRow(2, 11, "b.example"));
expect(rows.map((r) => r.domain)).toEqual(["b.example", "a.example"]);
});
test("drops the oldest beyond capacity", () => {
let rows: LiveRow[] = [];
for (let i = 0; i < 5; i++) rows = pushRow(rows, liveRow(i, i, `d${i}.example`), 3);
expect(rows).toHaveLength(3);
expect(rows.map((r) => r.key)).toEqual([4, 3, 2]);
});
test("default capacity is 500", () => {
let rows: LiveRow[] = [];
for (let i = 0; i < RING_CAPACITY + 10; i++) rows = pushRow(rows, liveRow(i, i, "x.example"));
expect(rows).toHaveLength(RING_CAPACITY);
});
});
describe("mergeGap", () => {
test("skips rows already in the buffer and counts only new ones", () => {
const buffer = [liveRow(2, 100, "seen.example"), liveRow(1, 99, "old.example")];
const fetched = [
fetchedRow(30, 102, "gap2.example"),
fetchedRow(29, 101, "gap1.example"),
fetchedRow(28, 100, "seen.example"),
];
const { rows, missed } = mergeGap(buffer, fetched, counter());
expect(missed).toBe(2);
expect(rows.map((r) => r.domain)).toEqual(["gap2.example", "gap1.example", "seen.example", "old.example"]);
});
test("no additions returns the buffer unchanged with missed 0", () => {
const buffer = [liveRow(1, 100, "seen.example")];
const { rows, missed } = mergeGap(buffer, [fetchedRow(5, 100, "seen.example")], counter());
expect(missed).toBe(0);
expect(rows).toBe(buffer);
});
test("rows differing only in qtype are not deduplicated", () => {
const buffer = [liveRow(1, 100, "dual.example", { qtype: 1 })];
const fetched = [fetchedRow(5, 100, "dual.example", { qtype: 28 })];
const { missed } = mergeGap(buffer, fetched, counter());
expect(missed).toBe(1);
});
test("assigns fresh keys from the counter and drops the id", () => {
const { rows } = mergeGap([], [fetchedRow(77, 100, "gap.example")], counter(200));
expect(rows[0]?.key).toBe(201);
expect("id" in (rows[0] ?? {})).toBe(false);
});
test("result is capped at capacity, keeping the newest", () => {
const buffer = [liveRow(3, 300, "live.example")];
const fetched = [fetchedRow(2, 302, "g2.example"), fetchedRow(1, 301, "g1.example")];
const { rows, missed } = mergeGap(buffer, fetched, counter(), 2);
expect(missed).toBe(2);
expect(rows.map((r) => r.domain)).toEqual(["g2.example", "g1.example"]);
});
test("merged rows stay sorted newest-first by ts", () => {
const buffer = [liveRow(4, 105, "after-reopen.example"), liveRow(3, 100, "before.example")];
const fetched = [fetchedRow(9, 103, "gap.example")];
const { rows } = mergeGap(buffer, fetched, counter());
expect(rows.map((r) => r.ts)).toEqual([105, 103, 100]);
});
});
-53
View File
@@ -1,53 +0,0 @@
import type { LiveQueryEvent, QueryRow } from "@/lib/types";
/** A live stream row; `key` is a client-side monotonic counter (SSE frames carry no id). */
export interface LiveRow extends LiveQueryEvent {
key: number;
}
export const RING_CAPACITY = 500;
/** Prepend `row` (rows are newest-first) and drop the oldest beyond `capacity`. */
export function pushRow(rows: LiveRow[], row: LiveRow, capacity: number = RING_CAPACITY): LiveRow[] {
const next = [row, ...rows];
return next.length > capacity ? next.slice(0, capacity) : next;
}
// `since` on GET /api/queries is inclusive, so the re-sync fetch returns the
// last-seen row(s) again; live rows have no id, so identity is this tuple.
function signature(row: LiveQueryEvent): string {
return `${row.ts}|${row.domain}|${row.client_ip}|${row.qtype ?? -1}|${row.blocked}|${row.upstream}`;
}
/**
* Merge rows fetched for a reconnect gap (newest-first, from GET /api/queries)
* into the buffer. Rows already present are skipped; `missed` counts what was
* actually added. The result stays newest-first (stable sort by ts) and capped.
*/
export function mergeGap(
rows: LiveRow[],
fetched: QueryRow[],
nextKey: () => number,
capacity: number = RING_CAPACITY,
): { rows: LiveRow[]; missed: number } {
const seen = new Set(rows.map(signature));
const added: LiveRow[] = [];
for (const row of fetched) {
const event: LiveQueryEvent = {
ts: row.ts,
domain: row.domain,
client_ip: row.client_ip,
qtype: row.qtype,
blocked: row.blocked,
block_reason: row.block_reason,
response_time_us: row.response_time_us,
cache_hit: row.cache_hit,
upstream: row.upstream,
};
if (seen.has(signature(event))) continue;
added.push({ ...event, key: nextKey() });
}
if (added.length === 0) return { rows, missed: 0 };
const merged = [...added, ...rows].sort((a, b) => b.ts - a.ts).slice(0, capacity);
return { rows: merged, missed: added.length };
}
@@ -1,95 +0,0 @@
import { fireEvent, render, screen } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { RouterProvider, createMemoryHistory } from "@tanstack/react-router";
import { AuthProvider } from "@/auth/store";
import { createQueryClient } from "@/lib/queryClient";
import { createAppRouter } from "@/routes";
import type { LookupResult } from "@/lib/types";
const BLOCKED: LookupResult = {
domain: "ads.example",
group_id: 1,
local_records: false,
forward_zone: null,
blocked: true,
reason: "blocklist_domain",
matched: "ads.example",
source_url: "https://lists.test/a",
safe_search_rewrite: null,
};
let fetchMock: ReturnType<typeof createFetchMock>;
function json(payload: unknown, status = 200): Response {
return new Response(JSON.stringify(payload), { status, headers: { "content-type": "application/json" } });
}
function createFetchMock() {
return vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/groups") {
return json({
groups: [
{ id: 1, name: "default", safe_search: false },
{ id: 2, name: "kids", safe_search: true },
],
});
}
if (url === "/api/lookup?domain=ads.example&group_id=1") return json(BLOCKED);
return json({ error: "not stubbed" }, 404);
});
}
beforeEach(() => {
fetchMock = createFetchMock();
vi.stubGlobal("fetch", fetchMock);
});
afterEach(() => {
vi.unstubAllGlobals();
});
function renderPage() {
const queryClient = createQueryClient();
const router = createAppRouter(createMemoryHistory({ initialEntries: ["/lookup"] }), queryClient);
render(
<AuthProvider>
<QueryClientProvider client={queryClient}>
<RouterProvider router={router} />
</QueryClientProvider>
</AuthProvider>,
);
}
function lookupCalls(): string[] {
return fetchMock.mock.calls.map(([input]) => String(input)).filter((url) => url.startsWith("/api/lookup"));
}
test("fetches nothing until submit, then renders the blocked verdict", async () => {
renderPage();
await screen.findByRole("heading", { name: "Lookup" });
await screen.findByLabelText("Group");
expect(lookupCalls()).toEqual([]);
fireEvent.change(screen.getByLabelText("Domain"), { target: { value: "ads.example" } });
expect(lookupCalls()).toEqual([]);
fireEvent.click(screen.getByRole("button", { name: "Look up" }));
await screen.findByRole("heading", { name: "Blocked" });
expect(lookupCalls()).toEqual(["/api/lookup?domain=ads.example&group_id=1"]);
expect(screen.getByText("blocklist_domain")).toBeTruthy();
const link = screen.getByRole("link", { name: "https://lists.test/a" }) as HTMLAnchorElement;
expect(link.href).toBe("https://lists.test/a");
expect(screen.getByText("Queries for this name get a blocked response.")).toBeTruthy();
});
test("defaults the group select to the default group (id 1)", async () => {
renderPage();
// A RAC Select names its trigger with the current value and then the label, so
// the selected group's name is the only thing the trigger shows.
const trigger = await screen.findByRole("button", { name: /Group$/ });
expect(trigger.textContent).toContain("default");
});
@@ -1,363 +0,0 @@
import { act, fireEvent, render, screen, waitFor, within } from "@testing-library/react";
import { QueryClientProvider } from "@tanstack/react-query";
import { createQueryClient } from "@/lib/queryClient";
import type { Client, QueriesPage, QueryRow } from "@/lib/types";
import QueryLogPage from "./QueryLogPage";
function client(id: number, ip: string, name: string, learnedName: string): Client {
return {
id,
ip,
name,
learned_name: learnedName,
group_id: 1,
group: "default",
hand_edited: name !== "",
first_seen: 1_700_000_000,
last_seen: 1_700_000_100,
};
}
const CLIENTS: Client[] = [
client(1, "192.0.2.10", "Kitchen Pi", "pi.lan"),
client(2, "192.0.2.11", "", "laptop.lan"),
client(3, "192.0.2.12", "", ""),
];
function row(id: number, domain: string, overrides: Partial<QueryRow> = {}): QueryRow {
return {
id,
ts: 1_700_000_000 + id,
domain,
client_ip: "192.0.2.10",
qtype: 1,
blocked: false,
block_reason: "",
response_time_us: 1234,
cache_hit: false,
upstream: "udp://9.9.9.9:53",
...overrides,
};
}
const PAGES: Record<string, QueriesPage> = {
"/api/queries": {
queries: [
row(20, "first.example", { qtype: 65, cache_hit: true, upstream: "" }),
row(19, "ads.example", {
blocked: true,
block_reason: "blocklist:stevenblack",
response_time_us: null,
cache_hit: null,
}),
],
next_before: 19,
},
"/api/queries?before=19": {
queries: [row(5, "older.example")],
next_before: null,
},
"/api/queries?domain=ads": {
queries: [row(19, "ads.example", { blocked: true, block_reason: "blocklist:stevenblack" })],
next_before: null,
},
};
beforeEach(() => {
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/clients") return json({ clients: CLIENTS });
const payload = PAGES[url];
if (payload === undefined) return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return new Response(JSON.stringify(payload), {
status: 200,
headers: { "content-type": "application/json" },
});
}),
);
});
afterEach(() => {
vi.unstubAllGlobals();
});
function renderPage() {
const client = createQueryClient();
render(
<QueryClientProvider client={client}>
<QueryLogPage />
</QueryClientProvider>,
);
return client;
}
function json(payload: unknown): Response {
return new Response(JSON.stringify(payload), { status: 200, headers: { "content-type": "application/json" } });
}
test("renders the first page with type names, blocked badge, and formatted cells", async () => {
renderPage();
await screen.findByText("first.example");
expect(screen.getByText("HTTPS")).toBeTruthy();
expect(screen.getByText("A")).toBeTruthy();
expect(screen.getByText("Blocked")).toBeTruthy();
expect(screen.getByText("blocklist:stevenblack")).toBeTruthy();
expect(screen.getByText("1.2 ms")).toBeTruthy();
expect(screen.getByText("hit")).toBeTruthy();
expect(screen.getByText("udp://9.9.9.9:53")).toBeTruthy();
expect(screen.getByText(/Showing 2 queries/)).toBeTruthy();
});
test("resolves each row's client to its display name, keeping the IP as the tooltip", async () => {
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/clients") return json({ clients: CLIENTS });
if (url !== "/api/queries") return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return json({
queries: [
row(20, "named.example", { client_ip: "192.0.2.10" }),
row(19, "learned.example", { client_ip: "192.0.2.11" }),
row(18, "nameless.example", { client_ip: "192.0.2.12" }),
row(17, "stranger.example", { client_ip: "192.0.2.99" }),
],
next_before: null,
} satisfies QueriesPage);
}),
);
renderPage();
// A hand-typed name wins outright; the learned name never surfaces for it.
const named = await screen.findByText("Kitchen Pi");
expect(named.getAttribute("title")).toBe("192.0.2.10");
expect(screen.queryByText("pi.lan")).toBeNull();
// A learned name reads muted and nothing more here: the "learned" tag would
// repeat on every row of the table, so the Clients page carries it instead.
const learned = screen.getByText("laptop.lan");
expect(learned.getAttribute("title")).toBe("192.0.2.11");
expect(within(learned.closest("tr")!).queryByText("learned")).toBeNull();
// A known client with neither name, and a client the loaded list has never
// seen, both fall back to the bare address with no tooltip standing in.
expect(screen.getByText("192.0.2.12").getAttribute("title")).toBeNull();
expect(screen.getByText("192.0.2.99").getAttribute("title")).toBeNull();
});
test("load more appends the next page and stops at the end of the log", async () => {
renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await screen.findByText("older.example");
expect(screen.getByText("first.example")).toBeTruthy();
expect(screen.getByText(/Showing 3 queries — end of log/)).toBeTruthy();
expect(screen.queryByRole("button", { name: "Load more" })).toBeNull();
});
test("applying a filter refetches and resets the accumulated list", async () => {
renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await screen.findByText("older.example");
fireEvent.change(screen.getByLabelText("Domain contains"), { target: { value: "ads" } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
await screen.findByText(/Showing 1 query /);
expect(screen.getByText("ads.example")).toBeTruthy();
expect(screen.queryByText("first.example")).toBeNull();
expect(screen.queryByText("older.example")).toBeNull();
});
test("a load-more that resolves after a filter change is discarded", async () => {
let releaseLoadMore: () => void = () => {};
vi.stubGlobal(
"fetch",
vi.fn((input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/queries?before=19") {
return new Promise<Response>((resolve) => {
releaseLoadMore = () => {
resolve(
new Response(JSON.stringify(PAGES["/api/queries?before=19"]), {
status: 200,
headers: { "content-type": "application/json" },
}),
);
};
});
}
const payload = PAGES[url];
if (payload === undefined)
return Promise.resolve(new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 }));
return Promise.resolve(
new Response(JSON.stringify(payload), {
status: 200,
headers: { "content-type": "application/json" },
}),
);
}),
);
renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
fireEvent.change(screen.getByLabelText("Domain contains"), { target: { value: "ads" } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
await screen.findByText(/Showing 1 query /);
releaseLoadMore();
await act(async () => {
await new Promise((resolve) => setTimeout(resolve, 0));
});
expect(screen.queryByText("older.example")).toBeNull();
expect(screen.getByText(/Showing 1 query /)).toBeTruthy();
expect(screen.queryByRole("alert")).toBeNull();
});
test("load more is disabled while a filter change shows placeholder data, then uses the fresh cursor", async () => {
let releaseFiltered: () => void = () => {};
const filteredPage: QueriesPage = {
queries: [row(19, "ads.example", { blocked: true, block_reason: "blocklist:stevenblack" })],
next_before: 7,
};
const filteredOlderPage: QueriesPage = {
queries: [row(3, "ads.older.example")],
next_before: null,
};
const fetchMock = vi.fn((input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/queries?domain=ads") {
return new Promise<Response>((resolve) => {
releaseFiltered = () => {
resolve(
new Response(JSON.stringify(filteredPage), {
status: 200,
headers: { "content-type": "application/json" },
}),
);
};
});
}
const payload = url === "/api/queries?domain=ads&before=7" ? filteredOlderPage : PAGES[url];
if (payload === undefined)
return Promise.resolve(new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 }));
return Promise.resolve(
new Response(JSON.stringify(payload), {
status: 200,
headers: { "content-type": "application/json" },
}),
);
});
vi.stubGlobal("fetch", fetchMock);
renderPage();
await screen.findByText("first.example");
fireEvent.change(screen.getByLabelText("Domain contains"), { target: { value: "ads" } });
fireEvent.click(screen.getByRole("button", { name: "Apply filters" }));
const staleButton = screen.getByRole("button", { name: "Load more" });
expect(staleButton).toHaveProperty("disabled", true);
fireEvent.click(staleButton);
expect(fetchMock.mock.calls.map((call) => String(call[0]))).not.toContain("/api/queries?domain=ads&before=19");
releaseFiltered();
await waitFor(() => {
expect(screen.queryByText("first.example")).toBeNull();
});
const freshButton = screen.getByRole("button", { name: "Load more" });
expect(freshButton).toHaveProperty("disabled", false);
fireEvent.click(freshButton);
await screen.findByText("ads.older.example");
expect(fetchMock.mock.calls.map((call) => String(call[0]))).toContain("/api/queries?domain=ads&before=7");
expect(screen.getByText(/Showing 2 queries — end of log/)).toBeTruthy();
});
test("a background refetch after new rows arrive leaves no gap between the loaded pages", async () => {
// The newest-100 window moves up while the reader has a second page open.
// Refetching only the first page would drop n20 and n19 out of the middle
// of the table; the second page must be replayed from the fresh cursor.
const before: Record<string, QueriesPage> = {
"/api/queries": { queries: [row(20, "n20.example"), row(19, "n19.example")], next_before: 19 },
"/api/queries?before=19": { queries: [row(18, "n18.example"), row(17, "n17.example")], next_before: null },
};
const after: Record<string, QueriesPage> = {
"/api/queries": { queries: [row(22, "n22.example"), row(21, "n21.example")], next_before: 21 },
"/api/queries?before=21": {
queries: [row(20, "n20.example"), row(19, "n19.example"), row(18, "n18.example"), row(17, "n17.example")],
next_before: null,
},
};
let live = before;
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const payload = live[String(input)];
if (payload === undefined) return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return json(payload);
}),
);
const client = renderPage();
await screen.findByText("n20.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await screen.findByText("n17.example");
live = after;
await act(async () => {
await client.invalidateQueries({ queryKey: ["queries"] });
});
await screen.findByText("n22.example");
const shown = screen.getAllByText(/^n\d+\.example$/).map((cell) => cell.textContent);
expect(shown).toEqual(["n22.example", "n21.example", "n20.example", "n19.example", "n18.example", "n17.example"]);
expect(screen.getByText(/Showing 6 queries — end of log/)).toBeTruthy();
});
test("a 401 on load more routes through handleUnauthorized instead of the inline error", async () => {
const assign = vi.fn();
vi.stubGlobal("location", { pathname: "/queries", search: "", assign });
vi.stubGlobal(
"fetch",
vi.fn(async (input: RequestInfo | URL) => {
const url = String(input);
if (url === "/api/queries?before=19") {
return new Response(JSON.stringify({ error: "unauthorized" }), {
status: 401,
headers: { "content-type": "application/json" },
});
}
const payload = PAGES[url];
if (payload === undefined) return new Response(JSON.stringify({ error: "not stubbed" }), { status: 404 });
return new Response(JSON.stringify(payload), {
status: 200,
headers: { "content-type": "application/json" },
});
}),
);
renderPage();
await screen.findByText("first.example");
fireEvent.click(screen.getByRole("button", { name: "Load more" }));
await waitFor(() => {
expect(assign).toHaveBeenCalledWith(`/login?redirect=${encodeURIComponent("/queries")}`);
});
expect(screen.queryByRole("alert")).toBeNull();
expect(screen.queryByText(/Failed to load more/)).toBeNull();
});
-374
View File
@@ -1,374 +0,0 @@
import { useState, type FormEvent } from "react";
import { useInfiniteQuery } from "@tanstack/react-query";
import * as stylex from "@stylexjs/stylex";
import * as api from "@/lib/api";
import { formatMicros, formatTime } from "@/lib/format";
import { queriesInfiniteQuery } from "@/lib/queries";
import type { QueriesFilter, QueryRow } from "@/lib/types";
import { ClientName, useClientNames, type ClientNames } from "@/features/clients/clientNames";
import { qtypeName } from "./qtype";
import Select from "@/ui/Select";
import { styles as shared } from "@/ui/styles";
import { colors } from "@/ui/tokens.stylex";
const DARK = "@media (prefers-color-scheme: dark)";
const STATUS_OPTIONS = [
{ value: "any", label: "All" },
{ value: "blocked", label: "Blocked only" },
{ value: "allowed", label: "Allowed only" },
];
const styles = stylex.create({
heading: {
fontSize: "1.5rem",
lineHeight: "2rem",
fontWeight: 600,
},
/** One column on a phone, two from `sm`, five from `lg`, as before. */
filterGrid: {
marginTop: "1rem",
display: "grid",
gap: "0.75rem",
gridTemplateColumns: {
default: "repeat(1, minmax(0, 1fr))",
"@media (min-width: 640px)": "repeat(2, minmax(0, 1fr))",
"@media (min-width: 1024px)": "repeat(5, minmax(0, 1fr))",
},
},
filterLabel: {
display: "block",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
filterInput: {
marginTop: "0.25rem",
width: "100%",
},
buttonRow: {
display: "flex",
alignItems: "flex-end",
gap: "0.5rem",
gridColumn: {
default: null,
"@media (min-width: 640px)": "span 2 / span 2",
"@media (min-width: 1024px)": "span 5 / span 5",
},
},
toolbarButton: {
fontWeight: 500,
},
note: {
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textMuted,
},
empty: {
marginTop: "1.5rem",
color: colors.textMuted,
},
tableWrap: {
marginTop: "1rem",
overflowX: "auto",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
},
table: {
width: "100%",
fontSize: "0.875rem",
lineHeight: "1.25rem",
},
/** The header tint is a shade off the ground in each scheme, not a token role. */
head: {
backgroundColor: { default: "oklch(98.5% 0 none)", [DARK]: "oklch(21% 0.006 285.885)" },
textAlign: "left",
},
th: {
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
fontWeight: 500,
color: colors.textSecondary,
},
/** `divide-y`: a hairline between rows, so the first row carries none. */
row: {
borderTopWidth: { default: 1, ":first-child": 0 },
borderTopStyle: "solid",
borderTopColor: colors.border,
},
cell: {
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
},
nowrap: {
whiteSpace: "nowrap",
},
breakAll: {
wordBreak: "break-all",
},
small: {
fontSize: "0.75rem",
lineHeight: "1rem",
},
muted: {
color: colors.textMuted,
},
blockedWrap: {
display: "inline-flex",
alignItems: "center",
gap: "0.375rem",
},
blockedBadge: {
borderRadius: "0.25rem",
paddingInline: "0.375rem",
paddingBlock: "0.125rem",
fontSize: "0.75rem",
lineHeight: "1rem",
fontWeight: 500,
backgroundColor: { default: "oklch(93.6% 0.032 17.717)", [DARK]: "oklch(39.6% 0.141 25.723)" },
color: { default: "oklch(44.4% 0.177 26.899)", [DARK]: "oklch(88.5% 0.062 18.334)" },
},
footer: {
marginTop: "0.75rem",
display: "flex",
alignItems: "center",
gap: "0.75rem",
},
moreError: {
marginTop: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.dangerText,
},
});
function errorMessage(error: unknown): string {
return error instanceof Error ? error.message : String(error);
}
function datetimeLocalToUnix(value: string): number | undefined {
if (value === "") return undefined;
const ms = new Date(value).getTime();
return Number.isFinite(ms) ? Math.floor(ms / 1000) : undefined;
}
export function BlockedCell({ row }: { row: Pick<QueryRow, "blocked" | "block_reason"> }) {
if (!row.blocked) return <span {...stylex.props(styles.muted)}></span>;
return (
<span {...stylex.props(styles.blockedWrap)}>
<span {...stylex.props(styles.blockedBadge)}>Blocked</span>
{row.block_reason !== "" && <span {...stylex.props(styles.small, styles.muted)}>{row.block_reason}</span>}
</span>
);
}
export function QueryCells({ row, clientNames }: { row: Omit<QueryRow, "id">; clientNames: ClientNames }) {
return (
<>
<td {...stylex.props(styles.cell, styles.nowrap, styles.muted)}>{formatTime(row.ts)}</td>
<td {...stylex.props(styles.cell, styles.small, styles.breakAll, shared.mono)}>{row.domain}</td>
<td {...stylex.props(styles.cell, styles.small, styles.nowrap)}>
<ClientName ip={row.client_ip} names={clientNames} />
</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>{qtypeName(row.qtype)}</td>
<td {...stylex.props(styles.cell)}>
<BlockedCell row={row} />
</td>
<td {...stylex.props(styles.cell, styles.nowrap, shared.tabularNums)}>
{row.response_time_us === null ? "—" : formatMicros(row.response_time_us)}
</td>
<td {...stylex.props(styles.cell, styles.nowrap)}>
{row.cache_hit === null ? "—" : row.cache_hit ? "hit" : "miss"}
</td>
<td {...stylex.props(styles.cell, styles.small, styles.breakAll, shared.mono)}>
{row.upstream === "" ? "—" : row.upstream}
</td>
</>
);
}
export function QueryTableHead() {
return (
<thead {...stylex.props(styles.head)}>
<tr>
<th {...stylex.props(styles.th)}>Time</th>
<th {...stylex.props(styles.th)}>Domain</th>
<th {...stylex.props(styles.th)}>Client</th>
<th {...stylex.props(styles.th)}>Type</th>
<th {...stylex.props(styles.th)}>Status</th>
<th {...stylex.props(styles.th)}>Response</th>
<th {...stylex.props(styles.th)}>Cache</th>
<th {...stylex.props(styles.th)}>Upstream</th>
</tr>
</thead>
);
}
export default function QueryLogPage() {
const [domain, setDomain] = useState("");
const [client, setClient] = useState("");
const [blocked, setBlocked] = useState("any");
const [since, setSince] = useState("");
const [until, setUntil] = useState("");
const [applied, setApplied] = useState<QueriesFilter>({});
const base = useInfiniteQuery(queriesInfiniteQuery(applied));
const clientNames = useClientNames();
const pages = base.data?.pages ?? [];
const rows: QueryRow[] = pages.flatMap((page) => page.queries);
const filterActive = Object.keys(applied).length > 0;
// `base.hasNextPage` reads the query state, which is empty while placeholder
// data stands in for a filter change; derive the cursor from what is on
// screen so the button keeps its place instead of flashing "end of log".
const lastPage = pages[pages.length - 1];
const hasMore = lastPage !== undefined && lastPage.next_before !== null;
// A 401 is already redirecting via the cache-level handleUnauthorized.
const isUnauthorized = base.error instanceof api.ApiError && base.error.status === 401;
const moreError = base.isFetchNextPageError && !isUnauthorized ? errorMessage(base.error) : null;
function applyFilters(event: FormEvent) {
event.preventDefault();
const filter: QueriesFilter = {};
if (domain.trim() !== "") filter.domain = domain.trim();
if (client.trim() !== "") filter.client = client.trim();
if (blocked === "blocked") filter.blocked = true;
if (blocked === "allowed") filter.blocked = false;
const sinceTs = datetimeLocalToUnix(since);
if (sinceTs !== undefined) filter.since = sinceTs;
const untilTs = datetimeLocalToUnix(until);
if (untilTs !== undefined) filter.until = untilTs;
setApplied(filter);
}
function clearFilters() {
setDomain("");
setClient("");
setBlocked("any");
setSince("");
setUntil("");
setApplied({});
}
function loadMore() {
if (!hasMore || base.isFetchingNextPage || base.isPlaceholderData) return;
void base.fetchNextPage();
}
return (
<section>
<h1 {...stylex.props(styles.heading)}>Query Log</h1>
<form onSubmit={applyFilters} {...stylex.props(styles.filterGrid)}>
<label {...stylex.props(styles.filterLabel)}>
Domain contains
<input
type="text"
value={domain}
onChange={(event) => setDomain(event.target.value)}
{...stylex.props(shared.smallInput, styles.filterInput, shared.focusRing)}
/>
</label>
<label {...stylex.props(styles.filterLabel)}>
Client (exact)
<input
type="text"
value={client}
onChange={(event) => setClient(event.target.value)}
{...stylex.props(shared.smallInput, styles.filterInput, shared.focusRing)}
/>
</label>
<Select
variant="compactField"
label="Status"
value={blocked}
onChange={setBlocked}
options={STATUS_OPTIONS}
/>
<label {...stylex.props(styles.filterLabel)}>
Since
<input
type="datetime-local"
value={since}
onChange={(event) => setSince(event.target.value)}
{...stylex.props(shared.smallInput, styles.filterInput, shared.focusRing)}
/>
</label>
<label {...stylex.props(styles.filterLabel)}>
Until
<input
type="datetime-local"
value={until}
onChange={(event) => setUntil(event.target.value)}
{...stylex.props(shared.smallInput, styles.filterInput, shared.focusRing)}
/>
</label>
<div {...stylex.props(styles.buttonRow)}>
<button type="submit" {...stylex.props(shared.button, styles.toolbarButton, shared.focusRing)}>
Apply filters
</button>
<button
type="button"
onClick={clearFilters}
{...stylex.props(shared.button, styles.toolbarButton, shared.focusRing)}
>
Clear
</button>
{base.isFetching && (
<span {...stylex.props(styles.note)} role="status">
Loading
</span>
)}
</div>
</form>
{base.data === undefined ? (
<p {...stylex.props(styles.empty, shared.pulse)} role="status">
Loading query log
</p>
) : rows.length === 0 ? (
<p {...stylex.props(styles.empty)}>
{filterActive ? "No queries match the current filters." : "No queries logged yet."}
</p>
) : (
<>
<div {...stylex.props(styles.tableWrap)}>
<table {...stylex.props(styles.table)}>
<QueryTableHead />
<tbody>
{rows.map((row) => (
<tr key={row.id} {...stylex.props(styles.row)}>
<QueryCells row={row} clientNames={clientNames} />
</tr>
))}
</tbody>
</table>
</div>
<div {...stylex.props(styles.footer)}>
<p {...stylex.props(styles.note)}>
Showing {rows.length} {rows.length === 1 ? "query" : "queries"}
{hasMore ? "" : " — end of log"}
</p>
{hasMore && (
<button
type="button"
onClick={loadMore}
disabled={base.isFetchingNextPage || base.isPlaceholderData}
{...stylex.props(shared.button, styles.toolbarButton, shared.focusRing)}
>
{base.isFetchingNextPage ? "Loading…" : "Load more"}
</button>
)}
</div>
{moreError !== null && (
<p role="alert" {...stylex.props(styles.moreError)}>
Failed to load more: {moreError}
</p>
)}
</>
)}
</section>
);
}
@@ -0,0 +1,39 @@
import {
ENUM_VALUES,
policyActionLabel,
policyReasonLabel,
qclassName,
rcodeName,
routeKindLabel,
} from "./provenanceCopy";
/**
* `tsc` proves the maps total over the union; this proves the union is the set
* the server actually stores, and that no entry was left as its raw tag name.
*/
test("every stored enum value has a label of its own", () => {
const labels = [
...ENUM_VALUES.policyAction.map(policyActionLabel),
...ENUM_VALUES.policyReason.map(policyReasonLabel),
...ENUM_VALUES.routeKind.map(routeKindLabel),
];
for (const label of labels) {
expect(label).not.toBe("");
expect(label).not.toMatch(/_/);
}
expect(new Set(ENUM_VALUES.policyReason.map(policyReasonLabel)).size).toBe(ENUM_VALUES.policyReason.length);
});
test("response codes read by name where one exists, by number where none does", () => {
expect(rcodeName(0)).toBe("NOERROR (0)");
expect(rcodeName(3)).toBe("NXDOMAIN (3)");
expect(rcodeName(16)).toBe("BADVERS (16)");
// The column holds the twelve-bit extended code, most of which is unassigned.
expect(rcodeName(3841)).toBe("RCODE 3841");
});
test("query classes read the same way", () => {
expect(qclassName(1)).toBe("IN (1)");
expect(qclassName(255)).toBe("ANY (255)");
expect(qclassName(42)).toBe("CLASS 42");
});
@@ -0,0 +1,103 @@
import { POLICY_ACTIONS, POLICY_REASONS, ROUTE_KINDS } from "@/lib/types";
import type { PolicyAction, PolicyReason, RouteKind } from "@/lib/types";
/**
* Display names for the three stored enums. `Record` over the union, so a value
* added to `src/storage/provenance.zig` and mirrored into `lib/types.ts` fails
* `tsc` here instead of reaching a cell as a raw tag name.
*/
const POLICY_ACTION_LABELS: Record<PolicyAction, string> = {
not_evaluated: "Not evaluated",
allow: "Allowed",
block: "Blocked",
};
const POLICY_REASON_LABELS: Record<PolicyReason, string> = {
rule_allow_exact: "Allow rule (exact)",
rule_block_exact: "Block rule (exact)",
rule_allow_wildcard: "Allow rule (wildcard)",
rule_block_wildcard: "Block rule (wildcard)",
rule_allow_regex: "Allow rule (regex)",
rule_block_regex: "Block rule (regex)",
blocklist_exception: "Blocklist exception",
blocklist_domain: "Blocklist (domain)",
blocklist_wildcard: "Blocklist (wildcard)",
local_record: "Local record",
forward_zone: "Forward zone",
non_in_class: "Not class IN",
paused: "Filtering paused",
snapshot_unavailable: "No filter snapshot",
no_match: "No match",
protocol_error: "Protocol refusal",
};
const ROUTE_KIND_LABELS: Record<RouteKind, string> = {
blocked: "Blocked locally",
local: "Local record",
forward_zone: "Forward zone",
upstream: "Upstream resolver",
cache: "Cache",
rejected: "Rejected",
};
export function policyActionLabel(action: PolicyAction): string {
return POLICY_ACTION_LABELS[action];
}
export function policyReasonLabel(reason: PolicyReason): string {
return POLICY_REASON_LABELS[reason];
}
export function routeKindLabel(kind: RouteKind): string {
return ROUTE_KIND_LABELS[kind];
}
/** The enum value sets, for tests that prove the maps exhaustive at runtime too. */
export const ENUM_VALUES = {
policyAction: POLICY_ACTIONS,
policyReason: POLICY_REASONS,
routeKind: ROUTE_KINDS,
} as const;
const RCODE_NAMES: Record<number, string> = {
0: "NOERROR",
1: "FORMERR",
2: "SERVFAIL",
3: "NXDOMAIN",
4: "NOTIMP",
5: "REFUSED",
6: "YXDOMAIN",
7: "YXRRSET",
8: "NXRRSET",
9: "NOTAUTH",
10: "NOTZONE",
16: "BADVERS",
};
/**
* The bare mnemonic, for a table cell with no room for the number. An
* unassigned code has no mnemonic to shorten, so it keeps the same `RCODE <n>`
* shape the long form falls back to.
*/
export function rcodeShortName(rcode: number): string {
return RCODE_NAMES[rcode] ?? `RCODE ${rcode}`;
}
/** The twelve-bit extended code as `NXDOMAIN (3)`; an unassigned code keeps its number. */
export function rcodeName(rcode: number): string {
const name = RCODE_NAMES[rcode];
return name === undefined ? `RCODE ${rcode}` : `${name} (${rcode})`;
}
const QCLASS_NAMES: Record<number, string> = {
1: "IN",
3: "CH",
4: "HS",
254: "NONE",
255: "ANY",
};
export function qclassName(qclass: number): string {
const name = QCLASS_NAMES[qclass];
return name === undefined ? `CLASS ${qclass}` : `${name} (${qclass})`;
}
@@ -0,0 +1,58 @@
import type { Provenance, QueryRow } from "@/lib/types";
/**
* Fixture builders for the provenance shapes, shared by the query-log, detail
* and live-stream tests the way `features/activity/fakeEventSource.ts` is shared.
*
* The defaults describe the dullest possible query an allowed name nothing
* matched, answered upstream so each test states only the fields it is about.
*/
type Sections = {
[K in keyof Provenance]?: Partial<Provenance[K]>;
};
export function provenance(sections: Sections = {}): Provenance {
return {
request: {
time: 1_700_000_000,
domain: "example.com",
client: "192.0.2.10",
qtype: 1,
qclass: 1,
...sections.request,
},
group: { id: 1, name: "default", ...sections.group },
policy: {
action: "allow",
reason: "no_match",
matched: "",
source_id: null,
source_name: "",
...sections.policy,
},
rewrites: { cname_target: "", safe_search_target: "", ...sections.rewrites },
route: { kind: "upstream", forward_zone: "", upstream: "https://dns.example/dns-query", ...sections.route },
response: { rcode: 0, duration_us: 1234, ...sections.response },
};
}
/** The flat stored row of the same dull query. */
export function queryRow(id: number, overrides: Partial<QueryRow> = {}): QueryRow {
return {
id,
ts: 1_700_000_000,
domain: "example.com",
client_ip: "192.0.2.10",
qtype: 1,
qclass: 1,
rcode: 0,
blocked: false,
response_time_us: 1234,
cache_hit: false,
upstream: "https://dns.example/dns-query",
policy_action: "allow",
policy_reason: "no_match",
route_kind: "upstream",
...overrides,
};
}
@@ -0,0 +1,88 @@
import type { LiveQueryEvent, PolicyReason, QueryRow, RouteKind } from "@/lib/types";
/**
* What the query-log table renders for one row, whichever surface it came from.
*
* The stored list row and the live stream's provenance event describe the same
* query in two different shapes flat summary against nested full detail and
* both pages share one set of cells, so both project into this.
*
* `id` is null for a streamed event: the frame precedes its own insert, so no
* row exists to link to yet.
*/
export interface QuerySummary {
id: number | null;
ts: number;
domain: string;
client_ip: string;
qtype: number | null;
blocked: boolean;
policy_reason: PolicyReason;
/** The twelve-bit extended code the client saw, including a synthesized SERVFAIL. */
rcode: number;
route_kind: RouteKind;
response_time_us: number | null;
cache_hit: boolean | null;
upstream: string;
}
/**
* Whether the cache answered, or null where it never applied. Mirrors
* `Context.cacheHit` in src/server/handler.zig, which derives the stored
* `cache_hit` column from the same route: a local record, a blocked answer and
* a protocol refusal all bypass the cache, and "miss" would claim a lookup that
* never happened.
*/
export function cacheHitFor(kind: RouteKind): boolean | null {
switch (kind) {
case "cache":
return true;
case "upstream":
case "forward_zone":
return false;
case "local":
case "blocked":
case "rejected":
return null;
}
}
export function summarizeRow(row: QueryRow): QuerySummary {
return {
id: row.id,
ts: row.ts,
domain: row.domain,
client_ip: row.client_ip,
qtype: row.qtype,
blocked: row.blocked,
policy_reason: row.policy_reason,
rcode: row.rcode,
route_kind: row.route_kind,
response_time_us: row.response_time_us,
cache_hit: row.cache_hit,
upstream: row.upstream,
};
}
/**
* The same summary out of a live frame. `blocked` and `cache_hit` are derived
* rather than sent: the server derives the stored columns from exactly these
* two fields (handler.zig's `Entry.init` call), so the projection reproduces
* them instead of the DTO carrying the same fact twice.
*/
export function summarizeEvent(event: LiveQueryEvent): QuerySummary {
return {
id: null,
ts: event.request.time,
domain: event.request.domain,
client_ip: event.request.client,
qtype: event.request.qtype,
blocked: event.policy.action === "block",
policy_reason: event.policy.reason,
rcode: event.response.rcode,
route_kind: event.route.kind,
response_time_us: event.response.duration_us,
cache_hit: cacheHitFor(event.route.kind),
upstream: event.route.upstream,
};
}
+31
View File
@@ -0,0 +1,31 @@
import { render, screen } from "@testing-library/react";
import CoverageNotice from "./CoverageNotice";
import { formatTime } from "./format";
const WATERMARK = 1_700_000_000;
test("a window the log covers in full says nothing", () => {
const { container } = render(<CoverageNotice coverage={{ complete: true, available_since: WATERMARK }} />);
expect(container.textContent).toBe("");
});
test("a window reaching past the watermark names the instant history starts", () => {
render(<CoverageNotice coverage={{ complete: false, available_since: WATERMARK }} />);
const notice = screen.getByRole("status");
expect(notice.textContent).toContain("Query history is available from");
expect(notice.textContent).toContain(formatTime(WATERMARK));
});
/**
* The server judges completeness, not the page: an unbounded request is
* incomplete whatever the watermark reads, and the notice must follow that
* verdict rather than compare timestamps itself.
*/
test("the server's verdict decides, not the numbers beside it", () => {
const { rerender, container } = render(
<CoverageNotice coverage={{ complete: true, available_since: WATERMARK }} />,
);
expect(container.textContent).toBe("");
rerender(<CoverageNotice coverage={{ complete: false, available_since: 0 }} />);
expect(screen.getByRole("status").textContent).toContain(formatTime(0));
});
+38
View File
@@ -0,0 +1,38 @@
import * as stylex from "@stylexjs/stylex";
import { formatTime } from "@/lib/format";
import type { Coverage } from "@/lib/types";
import { colors } from "@/ui/tokens.stylex";
const styles = stylex.create({
notice: {
marginTop: "0.75rem",
borderRadius: "0.25rem",
borderWidth: 1,
borderStyle: "solid",
borderColor: colors.border,
backgroundColor: colors.surfaceHover,
paddingInline: "0.75rem",
paddingBlock: "0.5rem",
fontSize: "0.875rem",
lineHeight: "1.25rem",
color: colors.textSecondary,
},
});
/**
* How far back the numbers on this page can reach.
*
* Rendered whenever a response says its window is incomplete, which includes
* the common case of a request with no lower bound at all. The line states the
* watermark and nothing more: the same incompleteness covers a log retention
* has pruned and one that simply has not been running long enough, and the
* response does not say which.
*/
export default function CoverageNotice({ coverage }: { coverage: Coverage }) {
if (coverage.complete) return null;
return (
<p role="status" {...stylex.props(styles.notice)}>
Query history is available from {formatTime(coverage.available_since)}.
</p>
);
}
+4
View File
@@ -26,6 +26,7 @@ import type {
Period,
QueriesFilter,
QueriesPage,
QueryDetail,
Rule,
RuleEcho,
RuleInput,
@@ -112,6 +113,9 @@ export const logout = (): Promise<LogoutResponse> => request("/api/auth/logout",
export const getQueries = (filter: QueriesFilter = {}): Promise<QueriesPage> =>
request(`/api/queries${qs({ ...filter })}`);
/** Full provenance of one logged row. 404 when retention has removed it, 503 with no query log. */
export const getQueryDetail = (id: number): Promise<QueryDetail> => request(`/api/queries/${id}`);
/** `EventSource` URL for the live stream; not a fetch route. */
export const liveQueriesUrl = "/api/queries/live";
+96 -28
View File
@@ -29,6 +29,7 @@ import type {
LookupResult,
PauseState,
QueriesPage,
QueryDetail,
Rule,
RuleEcho,
SettingsEnvelope,
@@ -415,76 +416,139 @@ export const sample_get_upstream_health: UpstreamHealth = {
};
export const sample_get_queries: QueriesPage = {
coverage: {
available_since: 0,
complete: false,
},
next_before: 0,
queries: [
{
block_reason: "",
blocked: true,
cache_hit: null,
client_ip: "192.0.2.11",
domain: "shop.example",
id: 0,
policy_action: "block",
policy_reason: "blocklist_wildcard",
qclass: 0,
qtype: 0,
rcode: 0,
response_time_us: 0,
route_kind: "blocked",
ts: 0,
upstream: "",
},
{
blocked: false,
cache_hit: false,
client_ip: "192.0.2.10",
domain: "news.example",
id: 0,
policy_action: "allow",
policy_reason: "no_match",
qclass: 0,
qtype: 0,
rcode: 0,
response_time_us: 0,
route_kind: "upstream",
ts: 0,
upstream: "https://dns.example/dns-query",
},
{
blocked: false,
cache_hit: true,
client_ip: "192.0.2.10",
domain: "d24.example",
id: 0,
policy_action: "allow",
policy_reason: "no_match",
qclass: 0,
qtype: 0,
rcode: 0,
response_time_us: 0,
route_kind: "cache",
ts: 0,
upstream: "https://dns.example/dns-query",
upstream: "",
},
{
block_reason: "",
blocked: false,
cache_hit: false,
client_ip: "192.0.2.10",
domain: "d23.example",
id: 0,
policy_action: "allow",
policy_reason: "no_match",
qclass: 0,
qtype: 0,
rcode: 0,
response_time_us: 0,
route_kind: "upstream",
ts: 0,
upstream: "https://dns.example/dns-query",
},
{
block_reason: "",
blocked: false,
cache_hit: true,
client_ip: "192.0.2.10",
domain: "d22.example",
id: 0,
policy_action: "allow",
policy_reason: "no_match",
qclass: 0,
qtype: 0,
rcode: 0,
response_time_us: 0,
ts: 0,
upstream: "https://dns.example/dns-query",
},
{
block_reason: "",
blocked: false,
cache_hit: false,
client_ip: "192.0.2.10",
domain: "d21.example",
id: 0,
qtype: 0,
response_time_us: 0,
ts: 0,
upstream: "https://dns.example/dns-query",
},
{
block_reason: "blocklist_domain",
blocked: true,
cache_hit: null,
client_ip: "192.0.2.10",
domain: "d20.example",
id: 0,
qtype: 0,
response_time_us: 0,
route_kind: "cache",
ts: 0,
upstream: "",
},
],
};
export const sample_get_query_detail: QueryDetail = {
group: {
id: 0,
name: "kids",
},
id: 0,
policy: {
action: "block",
matched: "||tracker.example^",
reason: "blocklist_wildcard",
source_id: 0,
source_name: "StevenBlack",
},
request: {
client: "192.0.2.11",
domain: "shop.example",
qclass: 0,
qtype: 0,
time: 0,
},
response: {
duration_us: 0,
rcode: 0,
},
rewrites: {
cname_target: "cdn.tracker.example",
safe_search_target: "",
},
route: {
forward_zone: "",
kind: "blocked",
upstream: "",
},
};
export const sample_get_stats: StatsTotals = {
avg_response_time_us: null,
blocked: 0,
cached: 0,
clients: 0,
coverage: {
available_since: 0,
complete: true,
},
period: "1h",
queries: 0,
since: 0,
@@ -501,6 +565,10 @@ export const sample_get_stats_timeseries: StatsTimeseries = {
ts: 0,
},
],
coverage: {
available_since: 0,
complete: true,
},
period: "1h",
since: 0,
until: 0,
+10 -14
View File
@@ -1,4 +1,4 @@
import { formatAge, formatBytes, formatDuration, formatMicros, formatTime } from "@/lib/format";
import { formatBytes, formatDuration, formatMicros, formatTime } from "@/lib/format";
test("formatTime renders unix seconds in the given locale and zone", () => {
// 2024-01-01T00:00:00Z; ICU emits U+202F before AM/PM in recent Node.
@@ -15,23 +15,19 @@ test("formatBytes humanizes with binary units", () => {
expect(formatBytes(2 * 1024 ** 4)).toBe("2.0 TiB");
});
test("formatAge steps up a unit at each boundary and truncates", () => {
expect(formatAge(0)).toBe("0s ago");
expect(formatAge(59)).toBe("59s ago");
expect(formatAge(60)).toBe("1m ago");
expect(formatAge(3599)).toBe("59m ago");
expect(formatAge(3600)).toBe("1h ago");
expect(formatAge(10800)).toBe("3h ago");
expect(formatAge(86399)).toBe("23h ago");
expect(formatAge(86400)).toBe("1d ago");
expect(formatAge(400000)).toBe("4d ago");
});
test("formatDuration is the same span without the 'ago', and never negative", () => {
test("formatDuration steps up a unit at each boundary and truncates", () => {
expect(formatDuration(0)).toBe("0s");
expect(formatDuration(59)).toBe("59s");
expect(formatDuration(60)).toBe("1m");
expect(formatDuration(3599)).toBe("59m");
expect(formatDuration(3600)).toBe("1h");
expect(formatDuration(10800)).toBe("3h");
expect(formatDuration(86399)).toBe("23h");
expect(formatDuration(86400)).toBe("1d");
expect(formatDuration(400000)).toBe("4d");
});
test("formatDuration is never negative", () => {
// Clock skew between the server's timestamps and the browser's clock.
expect(formatDuration(-5)).toBe("0s");
});
+3 -15
View File
@@ -28,21 +28,9 @@ const AGE_UNITS = [
] as const;
/**
* Seconds of elapsed time a coarse "3h ago". Truncating and single-unit on
* purpose: this labels a snapshot the caller renders once, so a reader must not
* take it for a live count. Nothing re-renders it as it ages.
*/
export function formatAge(seconds: number): string {
for (const unit of AGE_UNITS) {
if (seconds >= unit.seconds) return `${Math.floor(seconds / unit.seconds)}${unit.suffix} ago`;
}
return `${Math.floor(seconds)}s ago`;
}
/**
* Seconds of elapsed time a coarse "3h", the same single truncated unit as
* `formatAge` without the "ago". For a span the caller labels itself, as in
* "active for 3h". A negative span reads "0s": clock skew is not a duration.
* Seconds of elapsed time a coarse "3h". Truncating and single-unit on
* purpose, for a span the caller labels itself, as in "active for 3h". A
* negative span reads "0s": clock skew is not a duration.
*/
export function formatDuration(seconds: number): string {
for (const unit of AGE_UNITS) {
+4
View File
@@ -25,6 +25,7 @@ export const queryKeys = {
stats: (period: Period) => ["stats", period] as const,
timeseries: (period: Period) => ["stats", "timeseries", period] as const,
queriesInfinite: (filter: QueriesFilter) => ["queries", "infinite", filter] as const,
queryDetail: (id: number) => ["queries", "detail", id] as const,
diagnosticsInfinite: (filter: DiagnosticsFilter) => ["diagnostics", "infinite", filter] as const,
diagnostic: (id: number) => ["diagnostics", "event", id] as const,
/** Prefix of every diagnostics entry, page and detail alike; the purge target. */
@@ -75,6 +76,9 @@ export const queriesInfiniteQuery = (filter: QueriesFilter = {}) =>
placeholderData: keepPreviousData,
});
export const queryDetailQuery = (id: number) =>
queryOptions({ queryKey: queryKeys.queryDetail(id), queryFn: () => api.getQueryDetail(id) });
// Keyset pagination on `next_before`, exactly as the query log pages
// (handlers/diagnostics.zig copies the /api/queries contract). The active view
// polls on healthQuery's cadence because an episode opening is the same news a
+113 -3
View File
@@ -60,25 +60,133 @@ export interface LogoutResponse {
authenticated: false;
}
/**
* The three closed enums `src/storage/provenance.zig` stores, as values rather
* than bare types: the copy maps in `features/queries/provenanceCopy.ts` have to
* be proven exhaustive at runtime as well as by `tsc`, exactly as
* `DIAGNOSTIC_CODES` below.
*/
export const POLICY_ACTIONS = ["not_evaluated", "allow", "block"] as const;
export type PolicyAction = (typeof POLICY_ACTIONS)[number];
export const POLICY_REASONS = [
"rule_allow_exact",
"rule_block_exact",
"rule_allow_wildcard",
"rule_block_wildcard",
"rule_allow_regex",
"rule_block_regex",
"blocklist_exception",
"blocklist_domain",
"blocklist_wildcard",
"local_record",
"forward_zone",
"non_in_class",
"paused",
"snapshot_unavailable",
"no_match",
"protocol_error",
] as const;
export type PolicyReason = (typeof POLICY_REASONS)[number];
export const ROUTE_KINDS = ["blocked", "local", "forward_zone", "upstream", "cache", "rejected"] as const;
export type RouteKind = (typeof ROUTE_KINDS)[number];
/** The summary projection the query-log table scans; full provenance is at `/api/queries/{id}`. */
export interface QueryRow {
id: number;
ts: number;
domain: string;
client_ip: string;
qtype: number | null;
qclass: number;
rcode: number;
blocked: boolean;
block_reason: string;
response_time_us: number | null;
cache_hit: boolean | null;
upstream: string;
policy_action: PolicyAction;
policy_reason: PolicyReason;
route_kind: RouteKind;
}
/** SSE `event: query` payload: a QueryRow minus `id` (precedes persistence). */
export type LiveQueryEvent = Omit<QueryRow, "id">;
export interface ProvenanceRequest {
time: number;
domain: string;
client: string;
qtype: number | null;
qclass: number;
}
/** The group as a historical fact: the id may name a group since renamed or deleted. */
export interface ProvenanceGroup {
id: number | null;
name: string;
}
export interface ProvenancePolicy {
action: PolicyAction;
reason: PolicyReason;
matched: string;
source_id: number | null;
source_name: string;
}
export interface ProvenanceRewrites {
cname_target: string;
safe_search_target: string;
}
export interface ProvenanceRoute {
kind: RouteKind;
forward_zone: string;
/** Non-empty only for an attempted upstream or forward-zone exchange; already redacted. */
upstream: string;
}
export interface ProvenanceResponse {
/** The twelve-bit EDNS extended code, not the four header bits alone. */
rcode: number;
duration_us: number | null;
}
/**
* One query, fully explained, in the order a query meets the pipeline
* (`src/web/provenance_view.zig`). Every text field follows the repository
* convention: `""` means absent.
*/
export interface Provenance {
request: ProvenanceRequest;
group: ProvenanceGroup;
policy: ProvenancePolicy;
rewrites: ProvenanceRewrites;
route: ProvenanceRoute;
response: ProvenanceResponse;
}
/** `GET /api/queries/{id}`: the same six groups plus the row id. */
export interface QueryDetail extends Provenance {
id: number;
}
/** SSE `event: query` payload: the full provenance, minus an id it cannot have yet. */
export type LiveQueryEvent = Provenance;
/**
* How much of the requested window the query log can answer for. Retention
* deletes rows and advances the watermark in one transaction, so an empty
* window is distinguishable from a pruned one.
*/
export interface Coverage {
complete: boolean;
/** The oldest instant the log is complete for, unix seconds. */
available_since: number;
}
export interface QueriesPage {
queries: QueryRow[];
next_before: number | null;
coverage: Coverage;
}
export interface QueriesFilter {
@@ -171,6 +279,7 @@ export interface StatsTotals {
cached: number;
clients: number;
avg_response_time_us: number | null;
coverage: Coverage;
}
export interface Bucket {
@@ -186,6 +295,7 @@ export interface StatsTimeseries {
until: number;
bucket_seconds: number;
buckets: Bucket[];
coverage: Coverage;
}
export interface LookupResult {
+91 -30
View File
@@ -12,7 +12,14 @@ import {
import AppShell from "@/shell/AppShell";
import { ApiError } from "@/lib/api";
import { createQueryClient } from "@/lib/queryClient";
import type { DiagnosticSeverity, DiagnosticState, DiagnosticsFilter } from "@/lib/types";
import { diagnosticsFilterOf, type DiagnosticsSearch } from "@/features/diagnostics/filter";
import {
queriesFilterOf,
validateActivitySearch,
validateText,
validateTimestamp,
type ActivitySearch,
} from "@/features/activity/search";
import {
blocklistsQuery,
clientPrefixesQuery,
@@ -24,6 +31,7 @@ import {
healthQuery,
localRecordsQuery,
queriesInfiniteQuery,
queryDetailQuery,
rulesQuery,
settingsQuery,
statsQuery,
@@ -138,17 +146,72 @@ const dashboardRoute = createRoute({
component: lazyRouteComponent(() => import("@/features/dashboard/DashboardPage")),
});
const queriesRoute = createRoute({
/**
* Activity. The URL is the applied state: mode, the five filters, and nothing
* else. Everything is validated by `activity/search.ts`, so a hand-typed or
* stale parameter becomes `undefined` here rather than reaching the API as a
* value it answers 400 to.
*/
const activityRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/queries",
loader: ({ context }) => context.queryClient.ensureInfiniteQueryData(queriesInfiniteQuery({})),
component: lazyRouteComponent(() => import("@/features/queries/QueryLogPage")),
path: "/activity",
validateSearch: validateActivitySearch,
// An explicit projection, not `search` itself: the router hands the loader
// whatever else the URL carried, and an unknown key would make two
// otherwise-identical loads look like different deps.
loaderDeps: ({ search }): ActivitySearch => ({
mode: search.mode,
since: search.since,
until: search.until,
domain: search.domain,
client: search.client,
blocked: search.blocked,
}),
/**
* Starts the first page in parallel with the component chunk, and does not
* wait for it. Awaiting would make every Apply a blocking navigation, which
* throws away the `keepPreviousData` placeholder the list is built on: the
* reader would lose the rows they were reading to a pending page instead of
* watching them be replaced. The page owns the loading and error surfaces,
* so the rejection is caught here only to keep it from going unhandled.
*
* Live mode reads the SSE stream and nothing else. Prefetching the log for
* it would spend a request per navigation on rows the page never renders,
* with the retained filters attached to make it look deliberate.
*/
loader: ({ context, deps }) => {
if (deps.mode !== "history") return;
void context.queryClient.ensureInfiniteQueryData(queriesInfiniteQuery(queriesFilterOf(deps))).catch(() => {});
},
component: lazyRouteComponent(() => import("@/features/activity/ActivityPage")),
});
const liveRoute = createRoute({
/**
* One logged query. Its search is the Activity search the reader arrived from,
* validated by the same functions, so the back link and every related action
* restore the exact investigation instead of a default view of it.
*/
const activityDetailRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/live",
component: lazyRouteComponent(() => import("@/features/live/LiveLogPage")),
path: "/activity/queries/$id",
validateSearch: validateActivitySearch,
// Swallowed on purpose, as the diagnostics detail route does: a row
// retention has pruned is a 404 the page explains, with the way back to the
// log. The whole-page error component would call it a request failure.
loader: ({ context, params }) =>
context.queryClient.ensureQueryData(queryDetailQuery(Number(params.id))).catch(() => undefined),
component: lazyRouteComponent(() => import("@/features/activity/ActivityDetailPage")),
});
/** `domain` prefills and runs the simulation, so a query detail can link into it. */
const activityTestRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/activity/test",
validateSearch: (search: Record<string, unknown>): { domain?: string } => ({
domain: validateText(search["domain"]),
}),
loader: ({ context }) => context.queryClient.ensureQueryData(groupsQuery()),
component: lazyRouteComponent(() => import("@/features/activity/PolicyTestPage")),
});
const clientsRoute = createRoute({
@@ -210,40 +273,38 @@ const upstreamsRoute = createRoute({
component: lazyRouteComponent(() => import("@/features/upstreams/UpstreamsPage")),
});
const lookupRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/lookup",
loader: ({ context }) => context.queryClient.ensureQueryData(groupsQuery()),
component: lazyRouteComponent(() => import("@/features/lookup/LookupPage")),
});
/**
* The three filters live in the url so an episode can be linked to as it was
* read. Anything else in the search object is dropped: an unknown value would
* reach the api as a query parameter the handler rejects with a 400.
* The filters and the window live in the url so an episode can be linked to as
* it was read a query detail links here with an absolute five-minute window
* around one query, which only means anything if the page applies it. Anything
* else in the search object is dropped: an unknown value would reach the api as
* a query parameter the handler rejects with a 400.
*/
const diagnosticsRoute = createRoute({
getParentRoute: () => shellRoute,
path: "/diagnostics",
validateSearch: (
search: Record<string, unknown>,
): { state?: DiagnosticState; severity?: DiagnosticSeverity; component?: string } => {
validateSearch: (search: Record<string, unknown>): DiagnosticsSearch => {
const state = search["state"];
const severity = search["severity"];
const component = search["component"];
return {
state: state === "active" || state === "resolved" ? state : undefined,
severity: severity === "warning" || severity === "error" ? severity : undefined,
component: typeof component === "string" && component !== "" ? component : undefined,
component: validateText(search["component"]),
since: validateTimestamp(search["since"]),
until: validateTimestamp(search["until"]),
};
},
loaderDeps: ({ search }) => search,
loaderDeps: ({ search }): DiagnosticsSearch => ({
state: search.state,
severity: search.severity,
component: search.component,
since: search.since,
until: search.until,
}),
// allSettled: the two sections render their own state, and the resolved
// history failing must not replace the active list with the error page.
loader: ({ context, deps }) => {
const base: DiagnosticsFilter = {};
if (deps.severity !== undefined) base.severity = deps.severity;
if (deps.component !== undefined) base.component = deps.component;
const base = diagnosticsFilterOf(deps);
return Promise.allSettled([
context.queryClient.ensureInfiniteQueryData(diagnosticsInfiniteQuery({ ...base, state: "active" })),
context.queryClient.ensureInfiniteQueryData(diagnosticsInfiniteQuery({ ...base, state: "resolved" })),
@@ -274,15 +335,15 @@ const routeTree = rootRoute.addChildren([
loginRoute,
shellRoute.addChildren([
dashboardRoute,
queriesRoute,
liveRoute,
activityRoute,
activityDetailRoute,
activityTestRoute,
clientsRoute,
groupsRoute,
blocklistsRoute,
rulesRoute,
localDnsRoute,
upstreamsRoute,
lookupRoute,
diagnosticsRoute,
diagnosticDetailRoute,
settingsRoute,
+10 -4
View File
@@ -7,15 +7,13 @@ import { createAppRouter } from "@/routes";
const NAV_LABELS = [
"Dashboard",
"Query Log",
"Live",
"Activity",
"Clients",
"Groups",
"Blocklists",
"Rules",
"Local DNS",
"Upstreams",
"Lookup",
"Diagnostics",
"Settings",
];
@@ -30,8 +28,16 @@ const RESPONSES: Record<string, unknown> = {
cached: 0,
clients: 0,
avg_response_time_us: null,
coverage: { complete: true, available_since: 0 },
},
"/api/stats/timeseries?period=24h": {
period: "24h",
since: 0,
until: 86400,
bucket_seconds: 1800,
buckets: [],
coverage: { complete: true, available_since: 0 },
},
"/api/stats/timeseries?period=24h": { period: "24h", since: 0, until: 86400, bucket_seconds: 1800, buckets: [] },
"/api/health": {
status: "ok",
disk: { state: "ok", free_bytes: 0, db_bytes: 0, log_bytes: 0, sample_failures: 0 },
+1 -3
View File
@@ -17,15 +17,13 @@ const DARK = "@media (prefers-color-scheme: dark)";
const NAV_ITEMS = [
{ to: "/", label: "Dashboard" },
{ to: "/queries", label: "Query Log" },
{ to: "/live", label: "Live" },
{ to: "/activity", label: "Activity" },
{ to: "/clients", label: "Clients" },
{ to: "/groups", label: "Groups" },
{ to: "/blocklists", label: "Blocklists" },
{ to: "/rules", label: "Rules" },
{ to: "/local-dns", label: "Local DNS" },
{ to: "/upstreams", label: "Upstreams" },
{ to: "/lookup", label: "Lookup" },
{ to: "/diagnostics", label: "Diagnostics" },
{ to: "/settings", label: "Settings" },
] as const;
+34
View File
@@ -0,0 +1,34 @@
import { fireEvent, render, screen } from "@testing-library/react";
import Select from "./Select";
const OPTIONS = [
{ value: "any", label: "All" },
{ value: "blocked", label: "Blocked only" },
];
/** RAC opens a Select from the keyboard as readily as from a pointer. */
function open(trigger: HTMLElement) {
fireEvent.keyDown(trigger, { key: "Enter" });
fireEvent.keyUp(trigger, { key: "Enter" });
}
test("a disabled select keeps its value on screen but takes no input", () => {
const onChange = vi.fn();
render(<Select label="Status" options={OPTIONS} value="blocked" onChange={onChange} isDisabled />);
const trigger = screen.getByRole("button");
expect(trigger.textContent).toContain("Blocked only");
// A disabled button is out of the tab order by definition, so the filter row
// cannot be reached by keyboard while live mode owns it.
expect(trigger).toHaveProperty("disabled", true);
open(trigger);
expect(screen.queryByRole("listbox")).toBeNull();
expect(onChange).not.toHaveBeenCalled();
});
test("an enabled select still opens", () => {
render(<Select label="Status" options={OPTIONS} value="any" onChange={vi.fn()} />);
open(screen.getByRole("button"));
expect(screen.getByRole("listbox")).toBeTruthy();
});
+16 -2
View File
@@ -32,6 +32,8 @@ interface Props {
* dialog uses, `inline` a control sitting in a row of other controls.
*/
variant?: "field" | "compactField" | "inline";
/** Visible but inert, keeping its value on screen; RAC also drops it from the tab order. */
isDisabled?: boolean;
}
const styles = stylex.create({
@@ -50,7 +52,10 @@ const styles = stylex.create({
justifyContent: "space-between",
gap: "0.5rem",
textAlign: "left",
cursor: "pointer",
// RAC renders a real `<button disabled>`, so the state is reachable as a
// pseudo-class rather than needing a second style object.
cursor: { default: "pointer", ":disabled": "not-allowed" },
opacity: { default: null, ":disabled": 0.55 },
},
compact: {
marginTop: "0.25rem",
@@ -105,12 +110,21 @@ const styles = stylex.create({
},
});
export default function Select({ options, value, onChange, label, "aria-label": ariaLabel, variant = "field" }: Props) {
export default function Select({
options,
value,
onChange,
label,
"aria-label": ariaLabel,
variant = "field",
isDisabled = false,
}: Props) {
const base = variant === "field" ? shared.input : shared.smallInput;
const block = variant === "compactField" ? styles.compact : null;
return (
<AriaSelect
aria-label={ariaLabel}
isDisabled={isDisabled}
value={value}
onChange={(key) => onChange(String(key ?? ""))}
{...stylex.props(styles.root)}
+9 -1
View File
@@ -193,7 +193,15 @@ export const styles = stylex.create({
mono: {
fontFamily: "ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, monospace",
},
/** Visible to a screen reader only; the element keeps its place in the a11y tree. */
/**
* Visible to a screen reader only; the element keeps its place in the a11y tree.
*
* Apply it to a block container. `overflow` has no effect on a table box, and
* `height` on one is a minimum, so a `<table>` wearing this still lays out at
* its full content height and pushes the page's scrollable overflow past the
* app shell invisible, because `clip-path` still hides the paint. Wrap the
* table in a hidden `<div>` instead of hiding the table itself.
*/
srOnly: {
position: "absolute",
width: 1,
+31
View File
@@ -269,6 +269,37 @@ pub fn build(b: *std.Build) void {
});
test_step.dependOn(&b.addRunArtifact(container_check_tests).step);
// The release cut (specs/release-cut.md). A host tool like the two above,
// but run from the build graph rather than installed: it takes a bump kind
// on the command line (`zig build cut -- patch`), reads its own token, and
// needs the operator's terminal so `git commit -S` can reach pinentry —
// none of which a workflow supplies and all of which a Run step passes
// through.
const cut_tool = hostTool(b, "cut");
const cut_run = b.addRunArtifact(cut_tool);
// It pushes commits and tags, so it must never be answered from the run
// cache, and it must run at the build root whatever directory `zig build`
// was invoked from.
cut_run.has_side_effects = true;
cut_run.stdio = .inherit;
cut_run.setCwd(b.path("."));
if (b.args) |args| cut_run.addArgs(args);
b.step("cut", "Cut a release: preflight, bump, push, wait for CI, signed tag, watch the run")
.dependOn(&cut_run.step);
// Its pure decisions — semver strictness, the zon rewrite, the changelog
// section check, the runs-payload read and the tea-config token lookup —
// are the reason it is a program rather than a shell script.
const cut_tests = b.addTest(.{
.name = "cut-tool",
.root_module = b.createModule(.{
.root_source_file = b.path("tools/cut.zig"),
.target = b.graph.host,
.optimize = optimize,
}),
});
test_step.dependOn(&b.addRunArtifact(cut_tests).step);
addDist(b, options, admin_assets, .{
.version = version_option,
.version_string = version_string,
+27 -6
View File
@@ -4,7 +4,7 @@ nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080), a
The machine-readable contract is `src/web/openapi.yaml`, which the running server hands out unauthenticated at `GET /api/openapi.yaml`. Request and response schemas for every operation live there. When this page and the YAML disagree, the YAML wins.
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 60 of its entries.
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 61 of its entries.
## Conventions
@@ -45,7 +45,7 @@ A token bucket per client address: capacity and refill are both `web.api_rate_li
`GET /api/queries/live` is server-sent events over chunked transfer, `Content-Type: text/event-stream`, `Cache-Control: no-store`.
- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on its own after a drop.
- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload carries the `GET /api/queries` row fields minus `id` (a live entry precedes persistence): `ts`, `domain`, `client_ip`, `qtype`, `blocked`, `block_reason`, `response_time_us`, `cache_hit`, `upstream`.
- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload is the `Provenance` object — the body of `GET /api/queries/{id}` without its `id`, which does not exist yet because a live entry precedes its own insert. Its six groups are `request`, `group`, `policy`, `rewrites`, `route` and `response`.
- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping middleboxes from reaping the idle connection.
- Each subscriber buffers up to 64 entries. A client too slow for the query rate overflows its buffer and the server ends the stream cleanly after delivering what the buffer held — queries are never held back for a slow reader. There is no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the missed rows.
- Connections per client address are capped at `web.sse_max_connections_per_ip` (default 3); over the cap is a 429. The cap binds loopback too. The server holds at most 32 concurrent streams in total; when all slots are taken, the answer is a 503.
@@ -104,6 +104,7 @@ Auth `open` means no session is required; `session` means a valid session cookie
| POST | `/api/auth/login` | open | counted | runtime action | Log in |
| POST | `/api/auth/logout` | session | counted | runtime action | Log out |
| GET | `/api/queries` | session | counted | read | Query log page |
| GET | `/api/queries/{id}` | session | counted | read | One query, fully explained |
| GET | `/api/queries/live` | session | exempt | read | Live query stream (server-sent events) |
| GET | `/api/stats` | session | counted | read | Totals for a period |
| GET | `/api/stats/timeseries` | session | counted | read | Bucketed counts for a period |
@@ -174,9 +175,11 @@ In file mode `PUT /api/settings` is refused with the 403 above, password changes
Request and response schemas for every operation live in the OpenAPI document: `src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a running server.
### Block reasons
### Policy reasons
Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup` answer. The tag names the level that decided the query, and the levels are listed here in the order they are consulted — the first one that matches wins, so a rule always outranks a list.
Two places carry the same closed set of tags: `policy_reason` on a `GET /api/queries` row and on a `GET /api/queries/{id}` body (where it is `policy.reason`, and where the live stream sends the same field), and `reason` on a `GET /api/lookup` answer. The tag names what decided the query.
The first nine are the matcher's own verdicts, listed in the order they are consulted — the first that matches wins, so a rule always outranks a list.
| Tag | Decided by |
| --- | --- |
@@ -190,6 +193,24 @@ Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `bl
| `blocklist_domain` | A plain name in a downloaded list |
| `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list |
`/api/lookup` also answers `none` when nothing matched. A query row never carries `none`: `block_reason` is null unless the query was blocked.
The rest name a pipeline step that answered the query without consulting the matcher, and appear on a query row only.
A `cname:` prefix means the decision landed on a CNAME target rather than on the name the client asked for, so `cname:blocklist_domain` reads as "the list blocks a name this answer redirects to". Only `/api/queries` and the live stream show the prefix; `/api/lookup` does not follow CNAMEs.
| Tag | Decided by |
| --- | --- |
| `local_record` | A configured local record, answered before filtering |
| `forward_zone` | A configured forward zone, answered before filtering |
| `non_in_class` | The question was not class IN, so no rule could apply |
| `paused` | Filtering was paused |
| `snapshot_unavailable` | No filter snapshot was published yet, so the query went unfiltered |
| `no_match` | The matcher evaluated the name and nothing matched |
| `protocol_error` | A parsed request refused on protocol grounds — BADVERS, NOTIMP, a malformed EDNS OPT |
`policy_action` says which way the verdict went: `block`, `allow`, or `not_evaluated` for a query answered before any policy could apply. `/api/lookup` answers `none` when nothing matched, where a query row says `no_match`.
`route_kind` says where the answer came from: `blocked`, `local`, `forward_zone`, `upstream`, `cache` or `rejected`.
A non-empty `rewrites.cname_target` on a query detail means the decision landed on a CNAME target rather than on the name the client asked for; `policy.reason` is then the target's own reason. `/api/lookup` does not follow CNAMEs.
### Coverage
`GET /api/queries`, `GET /api/stats` and `GET /api/stats/timeseries` each answer with a `coverage` object: `available_since` is the oldest instant the query log is still complete for, and `complete` is true only when the window the request asked about starts at or after it. Retention deletes rows and advances the watermark in one transaction, so a client can tell an empty window from a pruned one instead of charting the gap as zero. A request with no lower bound at all asks about the whole of history, and is never complete.
+4 -2
View File
@@ -131,7 +131,7 @@ Process log and query log behavior.
|---|---|---|---|---|---|
| `logging.level` | enum `.err` \| `.warn` \| `.info` \| `.debug` | `.info` | — | one of the four tags; stored as `"error"` / `"warn"` / `"info"` / `"debug"` | log threshold (`src/platform/logging.zig`) |
| `logging.retention_days` | u16 | 30 | days | at least 1 | query-log pruning cutoff (`src/storage/retention.zig`) and the client tracker's last-seen cutoff (`src/server/clients.zig`) |
| `logging.query_log_buffer_max` | u32 | 10000 | entries | 11000000 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`) |
| `logging.query_log_buffer_max` | u32 | 10000 | entries | 137449 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`); the ceiling is derived at compile time from `@sizeOf(logger.Entry)` so the queue's worst case stays within 64 MiB, and it moves whenever the entry's width does |
| `logging.query_log_flush_interval_s` | u16 | 60 | seconds | 03600 | how long the query-log writer gathers entries before committing them in one transaction (`src/storage/logger.zig`); see the note below |
| `logging.hide_domains` | bool | false | — | — | the query log stores a hidden marker instead of the domain |
| `logging.hide_client_ips` | bool | false | — | — | the query log stores a hidden marker instead of the client address |
@@ -385,10 +385,12 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
| `MissingDefaultGroup` | no group is named `default` |
| `DuplicateGroupName` | two groups share a `name` |
| `EmptyGroupName` | a group `name` is empty |
| `GroupNameTooLong` | a group `name` is longer than 64 bytes; it is copied into every logged query |
| `UnknownGroup` | a client, prefix, group source or rule names a group that is not declared |
| `BadClientIp` / `DuplicateClientIp` | a client `ip` is unparseable, or collides after canonicalization |
| `BadClientPrefix` / `DuplicateClientPrefix` | the same for a `client_prefixes.prefix` |
| `BadSourceUrl` / `DuplicateSourceUrl` / `EmptySourceName` | blocklist source fields |
| `SourceNameTooLong` | a blocklist source `name` is longer than 64 bytes; it is copied into every logged query |
| `UnknownSource` / `DuplicateGroupSource` | `group_sources` links |
| `BadRulePattern` | a rule `pattern` does not match its `kind` |
| `BadLocalRecordName` / `BadLocalRecordValue` / `DuplicateLocalRecord` | local record fields |
@@ -397,7 +399,7 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
| `BadTimeout` | a timeout is outside 100120000 ms, or `attempt` is above `total` |
| `BadTtl` | `blocking.ttl`, `cache.negative_ttl_max`, a record `ttl`, `web.session_ttl_hours` or `blocklist_update.interval_hours` outside its range |
| `BadCacheSize` | `cache.size` outside 11000000 |
| `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 11000000 |
| `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 137449 |
| `BadFlushInterval` | `logging.query_log_flush_interval_s` above 3600 |
| `BadLogRotation` | `logging.max_size_mb` or `logging.max_files` below 1 |
| `BadDiskThresholds` | a threshold below 1, or `min_free_mb` above `warn_free_mb` |
+52
View File
@@ -0,0 +1,52 @@
# Repetitive workflows. Anything a release or a review runs twice belongs here,
# so a step cannot be forgotten by hand.
#
# Recipes only. Every decision the release makes lives in tools/cut.zig, which
# `zig build test` type-checks and covers; this file exists so nobody has to
# remember the invocation.
# every recipe, described
default:
@just --list
# unit suite
test:
zig build test
# unit + integration suite, which already runs the whole ordinary suite
itest:
zig build test -Dintegration
# The committed npm scripts, never npx: npx can fetch an unpinned package.
# admin: typecheck, tests, lint, formatting
admin-check:
cd admin && npm run typecheck && npm run test && npm run lint && npm run format:check
# admin/src/lib/contractSamples.gen.ts is a committed golden of live API bodies
# the admin tests assert against; this is the invocation AGENTS.md documents.
# regenerate the admin contract goldens from live responses
goldens:
zig build test -Dintegration -Dcontract-samples-out="$PWD/admin/src/lib/contractSamples.gen.ts"
# the binary with the real admin UI embedded (a plain `zig build` embeds a placeholder page)
build:
cd admin && npm run build
zig build -Dadmin-dist=admin/dist
# Not the CI gate: this skips the admin `npm run build` and `assert-bundled`,
# the cross-target builds and verify-dist.
# the fast local checks
verify: itest admin-check
zig fmt --check build.zig src tools
# kind is major, minor or patch: the version itself is derived from
# build.zig.zon, never typed, because a published tag cannot be corrected.
# Requires a clean tree, master, and a dated CHANGELOG.md section for the
# derived version. Preflight, bump, push, wait for CI, signed tag, watch the run.
# cut a release
release kind:
zig build cut -- {{kind}}
+210
View File
@@ -0,0 +1,210 @@
# Milestone 28: query provenance
Redesign step 2 of specs/ui-redesign.md ("Query provenance", "Schema changes", "Entry buffer widths", "API changes", rulings). Every logged query becomes exactly explainable: what policy decided, what matched, where the answer came from, and what the client saw. The existing Query Log/Live/Lookup pages keep working; their replacement is step 3 (milestone 29). Codex spec review folded in (thread 01a02643); its corrections are marked where they changed a ruling.
**This milestone destroys existing query history.** The DDL edit changes the CRC fingerprint (querylog_schema.zig:71-78), so `open` recreates the file and sets the old one aside as `querylog.db.schema-changed-<unix seconds>`. Acceptable pre-v0.1. The changelog entry must say so, and the recreate is the natural first `query_log.recreated` diagnostics emission.
## Sessions
S1 (storage) and S2 (upstream identity) run in parallel — disjoint files. S3 (handler capture) needs both. S4 (web API + contracts) needs S3. S5 (admin) needs S4.
---
## Session S1: querylog schema, repo, logger, coverage watermark
### S1.1 DDL (src/storage/querylog_schema.zig)
`query_log` drops `block_reason` and gains, after the existing columns:
```sql
qclass INTEGER NOT NULL,
rcode INTEGER NOT NULL,
group_id INTEGER,
group_name TEXT,
policy_action TEXT NOT NULL,
policy_reason TEXT NOT NULL,
matched TEXT,
source_id INTEGER,
source_name TEXT,
cname_target TEXT,
safe_search_target TEXT,
route_kind TEXT NOT NULL,
forward_zone TEXT
```
Existing columns (`blocked`, `cache_hit`, `upstream`, …) stay — stats and the step-3 filters still use them. No new index (ruling: `idx_query_log_ts` bounds every time-scoped question; the insert path pays for indexes).
New singleton table, in the same DDL string (Codex: enforce one row):
```sql
CREATE TABLE querylog_meta (
id INTEGER PRIMARY KEY CHECK (id = 1),
created_at INTEGER NOT NULL,
available_since INTEGER NOT NULL
);
```
`createFresh` inserts the row with `created_at = now` and `available_since = now + 1` (conservative: an old row logged in the same second as recreation must not let `since = now` claim completeness). **`available_since` is a monotonic coverage watermark, not a constant** (Codex): pruning advances it to the retention cutoff **in the same transaction as the delete** — the repo exposes one transactional `pruneOlderThan(cutoff)` that deletes and advances together; a failure of either rolls back both. It never moves backward. `queries_repo` gains `availableSince() i64`.
The object-count test (querylog_schema.zig:295-318) updates to 5 tables. TEXT id/name pairs are deliberate (ruling: `client_ip` precedent, "log rows are immutable facts"); ids are NOT foreign keys.
### S1.2 Closed enums — neutral module `src/storage/provenance.zig` (Codex: logger already imports queries_repo, so enums owned by logger would cycle)
Imported by logger, queries_repo, handler and web code. Stored as `@tagName` TEXT; the read path parses text back to the enum and treats an unknown value as a data error, not a passthrough string.
- `PolicyAction`: `not_evaluated`, `allow`, `block`.
- `PolicyReason`: the nine serializable `matcher.Reason` tags (matcher.zig:24-37 minus `none`) plus `local_record`, `forward_zone`, `non_in_class`, `paused`, `snapshot_unavailable`, `no_match`, `protocol_error`. Matcher→PolicyReason conversion is an exhaustive switch. The CNAME case is NOT a `cname:` string prefix any more: `cname_target` non-NULL carries that fact, and `policy_reason` holds the target's own reason.
- `RouteKind`: `blocked`, `local`, `forward_zone`, `upstream`, `cache`, `rejected`.
`protocol_error` + `rejected` cover post-parse protocol refusals (BADVERS, NOTIMP, EDNS FORMERR) — see S3.4; this extends the redesign's enum list, recorded here as an amendment required by its own "every syntactically parsed request that receives a response is logged" rule.
`blockReason()`, `cname_reason_prefix` and the comptime width proof (handler.zig:66-74, 816-823) die in S3.
### S1.3 Entry widening (src/storage/logger.zig)
`Entry` stays a by-value fixed-buffer struct through `Io.Queue`. New fields with explicit widths:
| field | width / type |
| --- | --- |
| `qclass` | `u16` |
| `rcode` | `u16`, validated ≤ 0xFFF (12-bit EDNS extended RCODE, edns.zig:219; Codex: u8 cannot hold it) |
| `group_id` | `?i64` |
| `group_name` | buffer sized by the new `max_group_name_len` (S1.5) |
| `policy_action` | `provenance.PolicyAction` |
| `policy_reason` | `provenance.PolicyReason` |
| `matched` | buffer sized by the rule maximum — the regex engine accepts 256-byte patterns (regex.zig:48), so 256 bytes with a **`u16` length** (the generic `copyInto` returns `u8`; widen it or add a u16 variant — 256 does not fit u8) |
| `source_id` | `?i64` |
| `source_name` | buffer sized by the new `max_source_name_len` (S1.5) |
| `cname_target` | 253-byte buffer |
| `safe_search_target` | 253-byte buffer |
| `forward_zone` | 253-byte buffer |
| `upstream` | widened: sized for the maximum redacted `scheme://host:port` form (host up to 253 bytes), `u16` length — the current 64-byte buffer silently truncates a long valid DoH hostname (Codex); the endpoint host bound gets validated where endpoints are parsed |
`Entry.Fields` gains the borrowed equivalents with `""`/null defaults. `reason_buf`/`max_reason_len` are removed with `block_reason`. The truncation test (logger.zig:610-625) updates.
`transformed()` (logger.zig:234-239) additionally rewrites `matched`, `cname_target`, `safe_search_target` to `hidden_marker` under `hide_domains`. `forward_zone`, `group_name`, `source_name` are configuration labels, not query-derived, and stay visible.
### S1.4 Entry memory budget (Codex: validation permits 1,000,000 queued entries, validate.zig:464, and sse.zig:54 embeds 2,048 entries)
The widened `@sizeOf(Entry)` gets a documented byte budget: `query_log_buffer_max`'s validation upper bound is recomputed so the queue's worst case stays ≤ 64 MiB (`max = 64 MiB / @sizeOf(Entry)`, computed at comptime, stated in the validation reference and CHANGELOG since the accepted range shrinks). Tests assert the default and the new maximum fit the budget, and the SSE hub comment states its embedded-entry cost.
### S1.5 Name caps — neutral module `src/config/limits.zig` (Codex: logger→validate for caps plus validate→logger for `@sizeOf(Entry)` is a cycle)
No length cap exists today for group or source names (validate.zig:721, :853 reject only empty). New `config/limits.zig` owns `max_group_name_len = 64` and `max_source_name_len = 64`; validate.zig enforces them (boundary tests at 64 and 65 bytes, new classification entries per the existing pattern); logger.zig sizes its buffers from them and **exports the computed `query_log_buffer_max` ceiling** (S1.4), which validate.zig imports.
### S1.6 Repo (src/storage/repositories/queries_repo.zig)
- `Row`/`insert_row_sql`/`BatchWriter` bind the new columns; `Sql.capacity` (:262-265) updated.
- `QueryRow` (read) gains `qclass: u16`, `rcode: u16`, `route_kind`, `policy_action`, `policy_reason` (parsed enums, serialized as strings); loses `block_reason`. NULL→`""` convention unchanged for text.
- New `detailById(id) ?QueryDetail`: full provenance row joined with domains, for `GET /api/queries/{id}`.
- `availableSince()` and the transactional prune-plus-advance per S1.1; retention.zig calls the combined operation.
- `stats_totals_sql`/`timeseries_sql` unchanged.
### S1.7 `query_log.recreated` emission
The existing emission site (app.zig:638) already fires on recreate; it gains the **initial coverage start** (the fresh `available_since`) in its detail alongside reason and aside filename (ui-redesign.md:252 requires the new coverage start). Verify open order (events store vs querylog open) and carry the `OpenResult` rather than reordering database opens if needed.
### S1.8 Acceptance (S1)
- [ ] Fingerprint tests updated; recreate test proves aside name, `querylog_meta` singleton row, and the recreated-event detail carrying the coverage start (end-to-end recreate→coverage test).
- [ ] Watermark tests: prune advances `available_since` to the cutoff atomically; a failed delete, a failed watermark update, and a failed commit each leave both untouched (rollback proven); it never regresses.
- [ ] Round-trip test: an `Entry` with every provenance field set survives queue → `toRow` → insert → `detailById` intact (modulo NULL mapping); includes a 256-byte `matched` boundary case.
- [ ] `transformed()` tests split by flag (Codex: the client is governed by `hide_client_ips`, not `hide_domains`): `hide_domains` hides domain, matched, cname_target, safe_search_target and preserves client_ip; `hide_client_ips` hides client_ip and preserves the rest; both flags leave group/source/zone names.
- [ ] Buffer-budget tests per S1.4; name-cap boundary tests per S1.5.
- [ ] `zig build test` 0 failed.
---
## Session S2: selected-resolver identity (every `transport.Client` implementor)
Reverses ruling 20 for the exchange that actually happened. `transport.Client` (transport.zig:325-343) gains a per-call out-parameter: `exchangeFn(ptr, io, query, response_buf, selected: *?[]const u8)`.
**Contract (Codex critical): the identity survives failure.** Each implementation sets `selected.*` to the resolver it is about to attempt, before the attempt; after an all-failed exchange it names the last attempted resolver. The handler consumes it on success AND on the error path — a SERVFAIL row carrying its resolver is the single most useful correlation this redesign adds (ui-redesign.md:157). The slice must stay valid for the query's duration: `Pool` uses `entry.endpoint.url` (Endpoint-owned, stable); the pointer is per-call, threaded into `exchangeLoopLen` (pool.zig:203-266) — never a `*Pool` field (concurrent queries race).
`Pool` reports the **raw** URL; redaction happens in the handler when formatting into the Entry (S3) — `safe_url.redact` is a formatter, so the pool test asserts the raw selected identity and the credential end-to-end test lives in S3/S4 (Codex).
Callers initialize the output to null before the call. `ForwardClient` (forward_client.zig) holds only a parsed `Resolver`, so it gains **owned identity storage** — the formatted resolver text lives in the client and the out-parameter borrows it (Codex: this is real behavior, not a mechanical discard; the "discards only" framing was wrong). Fakes set explicit stable identities.
S2 owns every implementor, fake and call site: pool.zig, forward_client.zig, transport.zig fakes (:635-660), doh_client_live_test.zig:47 and dot_client_live_test.zig:60 (imported by tests.zig — they break compilation if missed, Codex), handler.zig's inline fake (:1074) and its call site (:457, pass-and-ignore — S3 consumes it), doh_server.zig, dot_server.zig, udp/tcp/resolver/phase7 integration tests, web_integration_test.zig. This overlaps S1 on zero files; the handler/web edits are signature-level and complete before S3/S4 start (sequential).
- [ ] Pool tests: winning endpoint reported; failover reports the answerer, not the first attempt; all-failed reports the last attempted; a timeout mid-flight reports the in-flight resolver.
- [ ] `zig build test` 0 failed.
## Session S3: handler capture (src/server/handler.zig + server tests)
After S1+S2. All provenance assembled in `Context` and passed through `LogFields``Entry.Fields`:
- `qclass` from `ctx.q.qclass`; `group_id`/`group_name` from `snapshot.groups[ctx.group]` when snapshot non-null, else null/"" with `policy_reason = snapshot_unavailable` on the unfiltered path.
- Path mapping: qclass≠IN → `not_evaluated`/`non_in_class`, route `upstream`; pause → `not_evaluated`/`paused`; local → `allow`/`local_record`, route `local`; forward zone → `allow`/`forward_zone`, route `forward_zone` + zone name; cache hits → route `cache`, upstream NULL; upstream answers → route `upstream`, upstream = S2's selected identity redacted via `safe_url` into a Context-local buffer before `Entry.init``"pool"`/`pool_upstream` die; allowed matcher decisions (`rule_allow_*`, `blocklist_exception`) → `allow` with exact reason, matched and source captured (today discarded at :442, Codex); no match → `allow`/`no_match`; blocked → `block`/matcher reason, route `blocked`.
- **`upstream` is non-null only for attempted upstream or forward-zone exchanges, including their failures** (Codex). It is NULL for local, blocked, cache and rejected routes — the `"local"` marker (handler.zig:392) dies with `"pool"`. Asserted per route in the table tests.
- `matched` + `source_id`/`source_name` from `matcher.Decision` and `snapshot.sources[decision.source.?]`. **Copy `matched` and the uncloak target into Context-local buffers before the uncloak loop continues** — the scratch buffers are reused per chain step (matcher.zig:733-736).
- CNAME-uncloaked block: policy fields describe the target's decision; `cname_target` holds the target name. `Context.uncloak` (:727-742) widens its return to the target's full decision + name.
- Safe search: `safe_search_target` = the rewrite target; policy stays `allow`.
### S3.1 rcode capture and the truncation bug (Codex)
`reply`'s UDP truncation rebuild (:522-529) currently rewrites every oversized response to NOERROR — an existing defect: an oversized NXDOMAIN reaches the client as success. Fix here: parse the source rcode before rebuilding and preserve it **via `splitRcode` into both the header and the response OPT** (edns.zig:219-225; header-only preservation loses the upper 8 bits, Codex), then parse the final bytes once for logging. No per-path "known rcode" plumbing: the logged rcode is always derived from the final packet + OPT. Tests: oversized NXDOMAIN keeps NXDOMAIN+TC; an oversized extended-RCODE response keeps the full 12-bit value on the wire and in the log.
### S3.2 servFail logging
Every `servFail` site (8, all inside Context) now replies AND logs: `rcode = servfail`, upstream = the last attempted resolver when the failure came from an exchange, policy/route fields as far as the pipeline got.
### S3.3 Post-parse protocol refusals
BADVERS (:245), NOTIMP (:253) and bad-EDNS FORMERR (:231) answer an identifiable question but precede `Context`. Construct the logging context as soon as one question is parsed and log these as `not_evaluated`/`protocol_error`, route `rejected`, with the actual rcode. Pre-question failures (rate-limit REFUSED, unparseable, qdcount≠1) stay counters — unchanged.
### S3.4 Acceptance (S3)
- [ ] Table-driven provenance tests, one asserted row per path: non-IN, paused, no-snapshot, local, forward-zone, forward-zone cache hit, upstream cache hit, upstream answer (exact redacted URL asserted), rule allow, blocklist exception (source id+name), no-match, rule block, blocklist block with source id+name, CNAME-uncloaked block (cname_target + target's reason), safe-search rewrite, each servFail flavor (exchange-failure case asserts the last-attempted resolver), BADVERS, NOTIMP, bad-EDNS.
- [ ] Credential tests (Codex: `Endpoint.parse` rejects `@`, so userinfo cannot come through production config): a handler test with an injected userinfo-bearing identity proves redaction before `Entry.init`; the production-config cross-surface sweep (row, SSE, detail all secret-free, using an accepted credential-bearing DoH path shape) lives in S4.
- [ ] Truncation-rcode regression tests per S3.1.
- [ ] `pool_upstream` and the `"pool"` marker are gone from provenance producers and serializers (scoped grep — logging fixtures elsewhere are out of scope, Codex).
- [ ] `zig build test` 0 failed.
## Session S4: web API + contracts (src/web/, openapi.yaml, contract samples)
- List `QueryRow` serialization gains `qclass`, `rcode`, `route_kind`, `policy_action`, `policy_reason`; `block_reason` is gone (S5 updates the admin in the same milestone; `blocked`/`cache_hit`/`upstream` survive, satisfying "existing summary fields stay").
- New `GET /api/queries/{id}` → 200 nested `{request:{time,domain,client,qtype,qclass}, group:{id,name}, policy:{action,reason,matched,source_id,source_name}, rewrites:{cname_target,safe_search_target}, route:{kind,forward_zone,upstream}, response:{rcode,duration_us}}`; 404 unknown/pruned; 503 no querylog. Route added; the openapi path-count guard (web_integration_test.zig:~2775) and the route-table cardinality assertion (routes.zig:154) update together.
- **One shared full-provenance DTO** (Codex): define `Provenance` (the nested body above) once; `QueryDetail = {id} + Provenance` and the SSE event = `Provenance` exactly — not "QueryRow minus id" (ui-redesign.md:177 requires full live detail). The list row stays a separate summary projection. The live.zig lockstep test compares field names AND types against the shared DTO.
- `GET /api/queries` body gains `coverage: {complete: bool, available_since: i64}`; `complete = (filter.since != null and filter.since >= available_since)` with `available_since` the S1 watermark. `/api/stats` and `/api/stats/timeseries` gain the same pair, judged against the period's aligned `since`. Documented in openapi.
- openapi.yaml: all schemas updated; `/api/queries/{id}` documented; the three enums enumerated. New focused drift guards for QueryRow, QueryDetail (nested objects included), the coverage object and the three enum value sets against `provenance.zig` — comparing field names, types, nullability and requiredness, not names alone (the path-count guard protects none of that, Codex).
- Contract samples regenerated (justfile goldens recipe); add `get_query_detail` sample; drift test green.
- [ ] Web integration: detail 200/404/503; coverage fields present and correct across a since-bounded and an unbounded request; keyset walk green. `zig build test` + `-Dintegration` 0 failed.
## Session S5: admin (admin/src)
Scope deliberately thin — Activity is milestone 29:
- `lib/types.ts`: `QueryRow` updated; new `Provenance` + `QueryDetail = {id} & Provenance`; `QueriesPage` gains `coverage`; `LiveQueryEvent = Provenance`. **`LiveRow` becomes a discriminated union** (Codex: a reconnect gap-fetch returns summary `QueryRow`s, which cannot fabricate full provenance): `{kind:"streamed", event: LiveQueryEvent}` | `{kind:"recovered", row: QueryRow}` — recovered rows keep their `id` and link to `/queries/$id`; a shared summary projection feeds the flat `QueryCells` from either arm. ringBuffer.ts and useLiveQueries.ts updated; tests cover both arms and assert the streamed projection drops no field silently.
- QueryLogPage/Live status cell switches from `block_reason` to `policy_reason`; no other column changes.
- New route `/queries/$id` + detail page modeled on `diagnosticDetailRoute`: the ordered explanation (request, group, policy, rewrites, route, response), historical facts visually separated from current-state links, related actions (lookup the domain, filtered query-log links). Navigation is a real keyboard-focusable link in the row; row-wide pointer click is an enhancement only (Codex).
- Coverage: whenever a response says `complete == false`, show "Query history is available from …" against the effective lower bound (not only when the watermark postdates the window, Codex). This touches the current dashboard's stats consumers — permitted: the anti-requirement below bans an Overview redesign, not this notice.
- [ ] vitest: detail page renders each section from a fixture; hidden-domain fixture renders the marker; coverage-line tests (watermark inside the window and after it); ring-buffer projection test. Typecheck/prettier/oxlint clean.
---
## File ownership
| Files | Session |
| --- | --- |
| storage/querylog_schema.zig, storage/provenance.zig (new), config/limits.zig (new), storage/logger.zig, storage/repositories/queries_repo.zig, storage/retention.zig, config/validate.zig, storage tests, app.zig (recreated-event detail) | S1 |
| upstream/* (incl. both live tests), local/forward_client.zig, server transport fakes + signature fixes in handler.zig/doh/dot/integration/web tests | S2 |
| server/handler.zig + server tests (behavioral) | S3 |
| web/*, openapi.yaml, web_integration_test.zig, contract samples | S4 |
| admin/src/* | S5 |
S1/S2 are disjoint. S2's mechanical signature edits in S3/S4 territory land before those sessions start.
## Anti-requirements
- No response payloads, RR sets, EDNS payloads or packet bytes stored.
- No new querylog index; no normalized provenance tables.
- No upstream label on cache hits.
- No route aliases; no Activity consolidation (step 3); no Overview redesign (step 4) — the coverage notice on existing pages is in scope.
- No config knobs for any of this.
## Acceptance (milestone complete)
- [ ] `zig build test` and `-Dintegration` 0 failed; `zig fmt --check` clean; admin typecheck/vitest/oxlint/prettier clean; goldens drift test green.
- [ ] Live smoke on the real binary: a blocked, an allowed-with-match, a cached, an upstream, a forward-zone and a SERVFAIL query each produce a correct detail page; screenshots taken.
- [ ] CHANGELOG Unreleased entry states the history reset, the aside filename pattern, the truncation-rcode fix, and the shrunk `query_log_buffer_max` range.
+91
View File
@@ -0,0 +1,91 @@
# Milestone 29: Activity consolidation
Redesign step 3 of specs/ui-redesign.md ("Activity", "Time scoping", "Deletions and their cost", build-sequence step 3). Query Log, Live and Lookup merge into one Activity surface; the three old routes and their code are removed in the same change. Admin-only — no Zig, wire or openapi changes; if a session believes it needs one, that is a spec bug to report, not code to write. Codex design review folded in (thread 01a02857); its corrections are marked where they changed the shape.
## Sessions
S1: route-independent primitives only — it must not touch routes.tsx, AppShell.tsx, or move any route-bound page (Codex: the original move-then-delete split could not keep the tree shippable between sessions). S2: one atomic landable session — routes, page assembly, moves, deletion, retargeting, test migration, smoke. Sequential.
---
## Session S1: primitives (no route changes)
### S1.1 Summary projection and cells
- `QuerySummary` (querySummary.ts:13) gains `rcode` and `route_kind` only. **Result derives from the existing `blocked` projection** (which already folds live `policy.action` — Codex: the summary has no `policy_action` and does not need one), else the rcode.
- New `admin/src/features/activity/cells.tsx`: the seven-column set. The Domain cell takes a **route-neutral link-renderer prop** (Codex: S1 cannot reference the not-yet-existing typed route without failing typecheck); S2 supplies the typed `Link` to `/activity/queries/$id`. — Time, Domain, Client, Type, Result, Route, Duration. Table-only compact labels, defined and tested exactly here (the detail page keeps `provenanceCopy`'s long forms): Result → `Blocked` | `NOERROR` | `SERVFAIL` | … (bare rcode name, no numeric suffix); Route → `Blocked` | `Local` | `Forward zone` | `Upstream` | `Cache` | `Rejected`. Non-NOERROR result carries a non-color signal (weight/icon), same treatment as Blocked. Duration formats `response_time_us`, em dash when null.
- Tests: fixtures for blocked, allowed-NOERROR (**explicitly `blocked=false`** — Codex: the allowed case must be pinned, not implied), SERVFAIL, cache hit; each cell's exact text, including an unassigned extended rcode preserving provenanceCopy's `RCODE <n>` fallback shape (compact form without the parenthesized number).
### S1.2 Shared provenance-detail renderer (Codex critical: streamed live rows must keep a detail surface)
Extract the ordered-explanation body of QueryDetailPage (request / group / policy / rewrites / route / response sections, historical-vs-current separation, the honesty wordings) into `admin/src/features/activity/ProvenanceDetail.tsx` taking a `Provenance` plus optional persisted id. QueryDetailPage becomes a thin route wrapper around it (page stays at its current route in S1 — only the body moves to a route-independent component). The renderer takes related-action links as props so S1 stays route-agnostic.
### S1.3 Form/URL plumbing primitives
- `Select` (ui/Select.tsx:23) gains `isDisabled`, passed through to React Aria (Codex: the filter row cannot otherwise be disabled).
- Datetime conversion becomes bidirectional and second-exact: `datetimeLocalToUnix` gets its inverse (`unixToDatetimeLocal`, local time), inputs use `step={1}`. **DST makes local text lossy** (Codex: a fall-back fold maps two instants to one string; a spring-forward gap silently normalizes): the form keeps the original unix value plus a per-field dirty flag, reuses the original unless the operator edited that field, and rejects an edited value that does not format back identically after parsing. Tests: non-zero-second round trips, a DST-fold instant, a nonexistent spring-forward time.
- Search-param validators as pure functions in `activity/search.ts`: `since`/`until` accepted only via `Number.isSafeInteger`; `blocked` only when `typeof === "boolean"`; `mode` in the two-value union defaulting `"history"`; `domain`/`client` trimmed, with the empty string normalized to `undefined` (Codex: `domain=` must not persist as applied state the server treats as no filter). Tests: fractions, Infinity, overflow, quoted booleans, `false`, empty and whitespace strings.
### S1.4 Acceptance (S1)
- [ ] Typecheck, vitest, oxlint, format clean; existing pages still work untouched (only additive files plus the QueryDetailPage body extraction and Select prop).
## Session S2: the Activity surface, atomically
### S2.1 Routes (routes.tsx) — created and deleted in one change
- `/activity` → ActivityPage, `validateSearch` from S1's validators. `loaderDeps` returns an **explicit object** of `{mode, since, until, domain, client, blocked}` — never the whole search (Codex: unknown keys create spurious matches without `search.strict`); the history loader builds `QueriesFilter` **field-by-field** and only calls `ensureInfiniteQueryData` in history mode.
- `/activity/queries/$id` → the detail wrapper (loader unchanged, incl. the deliberate 404 swallow). The route's search carries the originating Activity search (validated by the same functions), so the back link restores the exact investigation view (Codex: a bare `/activity` back link discards context and violates absolute time scoping).
- `/activity/test` → policy simulation, search `domain`, `?domain=` auto-run preserved.
- `/queries`, `/queries/$id`, `/live`, `/lookup` deleted from the route tree in the same edit. No aliases, no redirects; unknown paths fall to the router's existing not-found handling (verify sane, build nothing).
### S2.2 ActivityPage
- **Mode switch is URL-controlled** (the uncontrolled Tabs at ui/Tabs.tsx:76 does not fit — a controlled switch or controlled-Tabs variant), updating via the functional form `search: prev => ({...prev, mode})` so filters survive (Codex: object-replacement navigate loses them). The Live subtree (`LiveActivity`) mounts **only** in live mode so no SSE connection lingers in history. Tests: switching to history closes the EventSource; switching back creates exactly one fresh source.
- History mode: URL is the applied state. The form draft resets whenever the applied search changes (back/forward, pasted URL — Codex: the current seed-once pattern fails back/forward; test both, with non-zero seconds in the bounds). Apply = navigate with the explicit search object; keyset/infinite mechanics (`next_before`, `isPlaceholderData` guard, `hasMore` derivation) carry over; CoverageNotice stays.
- Live mode: follow-by-default, Freeze/Resume ephemeral (never URL). The live stack moves as-is: useLiveQueries (freeze snapshot, gap re-sync, 401 probe, cap threshold, stale-source guards), ringBuffer (LiveRow union, occurrence-counting mergeGap, rationale comment verbatim), fakeEventSource injection, status pill, `aria-pressed` Freeze, missed-recovered note, resync-failed alert, capped alert + Retry, blocked-row tint, footnote count.
- **Live detail**: a streamed row (id null) opens `ProvenanceDetail` in-place from its in-memory `event` — no invented correlation id, no persisted fetch (ui-redesign.md:177). Recovered rows link to `/activity/queries/$id`. **Retention contract** (Codex): selection stores the selected `LiveRow` snapshot itself; the open detail survives ring eviction, gap merges, and Freeze/Resume, closing only on explicit close or mode unmount. Tests pin: opening a streamed row's detail, keyboard access, survival across capacity eviction (501+ events) and across Freeze/Resume.
- Filter row in live mode: disabled via S1's `isDisabled` (visible, value-preserving), including native inputs and Apply/Clear; keyboard/tab-order test proves nothing in the row is tabbable while disabled.
- Detail-only facts, stated per what the wire actually carries (Codex: no source URL or upstream error text exists on `QueryDetail`): matched pattern, historical source label, selected upstream, rcode. Underlying failure text lives in Diagnostics — the detail page's related actions gain the accepted absolute-window Diagnostics link: `/diagnostics?since=<ts-300>&until=<ts+300>` (the redesign's five-minute window around the query, ui-redesign.md:173).
- Related domain/client links carry absolute bounds: `/activity?mode=history&domain=…&since=…&until=…` with **`since = origin.since ?? ts - 300` and `until = origin.until ?? ts + 300`** per bound independently (Codex: the fallback is product contract, not implementation choice). Tests cover zero-, one-, and two-bound origins.
- **Diagnostics bounds must actually apply** (Codex: `/diagnostics` neither validates nor uses `since`/`until` at HEAD, so the ±300s link would render an unbounded page): the diagnostics route gains safe-integer `since`/`until` search validation and explicit loaderDeps, DiagnosticsPage applies them to both the active and resolved queries, and the page shows a visible range indication when bounded. Tests assert the outgoing `/api/diagnostics` requests carry the bounds.
### S2.3 Policy simulation
LookupPage moves to `activity/` with the "Current policy simulation" framing (forward-tense property preserved); verdict rendering, 503/429 handling, group select, same-pair refetch carry over. Reached from an Activity action, not primary nav.
### S2.4 Navigation, deletion, migration
- `NAV_ITEMS`: Query Log/Live/Lookup out, `{to: "/activity", label: "Activity"}` in their position.
- Old page files, their cells/stylex exports, and dead helpers deleted. One home per module under `activity/`; no re-export shims.
- Test migration with the no-shrink rule: every behavior pinned by the old tests is re-pinned at the new home or listed in the report as intentionally dead with its page. Files: QueryLogPage.test, LiveLogPage.test, useLiveQueries.test, ringBuffer.test, LookupPage.test, QueryDetailPage.test, AppShell.test, LoginPage safeRedirect samples (cosmetic).
- Sweep: no import, string or test references `/queries`, `/live`, `/lookup` as routes (API paths `/api/queries*` stay).
- **CHANGELOG.md is S2-owned** (Codex): retarget the still-Unreleased m28 entry's `/queries/{id}` mention to the new route, add the removal notice (bookmarks break) and the Activity surface + Result column.
### S2.5 Smoke + screenshots
Rebuild bundle (`npm run build`, `zig build -Dadmin-dist=admin/dist`), run the real binary with the smoke28 scratchpad config: URL-driven history filters (paste a full URL, screenshot the reproduced view), live streaming with freeze and an open streamed-row detail, the SERVFAIL row distinguishable in the list, detail via recovered/history row with the context-preserving back link, policy simulation, five-minute Diagnostics link landing on the bounded window, mobile-width nav drawer.
### S2.6 Acceptance (S2)
- [ ] Typecheck, vitest, oxlint, format clean; `zig build -Dadmin-dist=admin/dist` succeeds; `zig build test` untouched-green.
- [ ] The S2.2 test list green, including EventSource lifecycle, back/forward, disabled-row tab order, streamed-row detail.
- [ ] Router-level loader regression (Codex): `/activity?mode=live` — including with retained filters and unknown search keys — makes no `/api/queries` request; history mode forwards exactly the six normalized filter fields and nothing else.
- [ ] Screenshots per S2.5.
## File ownership
Sequential; S2 starts only after S1's gates are green. S1: additive files under activity/, querySummary.ts, ui/Select.tsx, the QueryDetailPage body extraction, their tests. S2: everything else listed, admin/src/* (including the diagnostics route/page bounds work) plus CHANGELOG.md.
## Anti-requirements
- No server/API/openapi changes; no golden regeneration.
- No route aliases, redirects, or bookmark shims; no custom 404.
- No filter push-down into the SSE stream.
- No Overview changes (step 4), no configuration-page changes (step 5).
- No new dependencies; no virtualization.
## Acceptance (milestone complete)
- [ ] All admin gates clean; zig suites untouched-green; bundled build succeeds; screenshots per S2.5.
- [ ] CHANGELOG updated per S2.4.
+1 -1
View File
@@ -40,4 +40,4 @@ One constant. The v0.0.7 batching cut process writes from ~0.5 to 0.281 GiB/day
## Gates
1. `zig build test` and `-Dintegration` 0 failed; fmt clean.
2. Field verification on the released build (the Pi deploys releases, not branches, so this necessarily follows the cut — owner-ordered 2026-08-21): one 24-hour run spanning several autocheckpoints and a retention pass — WAL resets normally, writeback falls materially, no dropped batches; measured as process write_bytes AND device sectors (/sys/block/mmcblk0/stat). The rpi-nixos-iac session runs it; a bad result reverts the constant in a follow-up patch release.
2. Field verification on the released build (the Pi deploys releases, not branches, so this necessarily follows the cut — owner-ordered 2026-08-21): confirm the WAL resets normally, writeback falls materially, and no batches drop. What to measure and over what window is the deployment side's call; a bad result reverts the constant in a follow-up patch release.
+56
View File
@@ -0,0 +1,56 @@
# Local release cut: justfile + tools/cut.zig
Cutting v0.0.7 by hand missed the build.zig.zon bump; verify-dist caught it one CI round late. The cut becomes a compiled, tested tool (the same ruling that moved publication out of workflow shell into tools/release.zig), invoked from a thin justfile. Design reviewed and accepted by Codex (thread 01a020f9); its findings are folded in below.
## justfile (repo root)
Recipes only — no variables, no embedded logic:
- `test``zig build test`
- `itest``zig build test -Dintegration`
- `admin-check``cd admin && npm run typecheck && npm run test && npm run lint && npm run format:check` (the committed npm scripts, never npx: the repo records that npx can fetch an unpinned package)
- `goldens` → the documented `-Dcontract-samples-out` invocation; comment says what `admin/src/lib/contractSamples.gen.ts` is: captured live API bodies the admin tests assert against
- `build``cd admin && npm run build`, then `zig build -Dadmin-dist=admin/dist` (the default build embeds a placeholder page)
- `verify``itest` + `admin-check` + `zig fmt --check build.zig src tools` — described honestly as the fast local checks, not the CI gate (it skips admin `npm run build`/`assert-bundled`, cross-targets, verify-dist)
- `release kind``zig build cut -- {{kind}}` (kind ∈ major, minor, patch)
`verify` must NOT also depend on `test`: `-Dintegration` already runs the whole ordinary suite.
## tools/cut.zig
Wired like the other host tools (`hostTool` + `addRunArtifact`, see build.zig ~230): `zig build cut -- {major|minor|patch}`. NOT installed to zig-out/bin. Its tests join `zig build test`.
Constants: one repo API base `https://git.mial.net/api/v1/repos/mokhtar/nxdns` (the tool can only ever target this repo — no configurability). The runs API needs a token (verified: anonymous GET is 401); read it from `~/.config/tea/config.yml` (logins entry for git.mial.net); a missing token is a clear error naming the file.
### Sequence
1. **Derive the version**: the argument is a bump kind — `major`, `minor` or `patch` — never a free-form number (a number validated as "greater semver" still admits every typo, and a published tag is immutable). Parse `.version` from build.zig.zon (precedent: tools/verify_dist.zig:220, container_check.zig:317). If `v<zon-version>` exists on origin, derive the next version from the bump kind (patch 0.0.7→0.0.8, minor→0.1.0, major→1.0.0; checked u32 arithmetic). If it is absent, the zon version is an in-progress cut: resume it instead of incrementing again, reported explicitly — this preserves the bump-committed-but-untagged rerun and local-tag adoption. The CHANGELOG check in preflight applies to the derived (or resumed) version, so a changelog written for the wrong bump kind fails as a mismatch.
2. **Preflight**: working tree clean; branch master; CHANGELOG.md has a `## [<v>] - YYYY-MM-DD` heading (dated, the repo's observed form; do NOT require the date be today) with a non-empty section body (the CI tool refuses a blank section — failing later just burns the tag); the tag-free check follows the plan: a derived version does its own `git ls-remote` and refuses if `v<v>` exists; a resumed version reuses the absence that selected it (asking twice invites two answers). Transport/auth failure is always distinguished from "no match" (exit code + stderr, never "nonzero means absent") and refuses — on the resume path an error read as absent would resume a released version. The local tag namespace is checked too — see resumability.
3. **Bump if needed**: rewrite build.zig.zon atomically, reparse it, assert the git diff contains exactly that one file, then `git commit -S -m "build: bump version to <v>"`. git runs with inherited stdio so pinentry can prompt; check the child's actual termination state.
4. **Push master**, capture the exact HEAD sha; all later status lookups and the tag use that sha explicitly.
5. **Wait for CI**: poll the runs API for the ci.yml run matching that sha and event push. A run-id floor captured before the push scopes the match — applied only when the push actually moved the ref (`git push --porcelain` destination flag), so a no-op push (resume case, or a concurrent identical push) adopts the existing run for that sha. Require the run to appear within a startup deadline; wait for its terminal conclusion. Refuse to tag on anything but success, naming the failing job from the commit-status contexts. Every HTTP attempt individually bounded by a monotonic deadline; transport/JSON errors are reported, never silently treated as pending.
6. **Reassert** HEAD and tree unchanged, then `git tag -s v<v> -m v<v> <sha>` and push the tag.
7. **Wait for the release run** (matched by workflow path `release.yml@refs/tags/v<v>` — Gitea reports tag pushes as event "push"): startup deadline for the run to appear; the completion clock starts at first sighting, with a ceiling derived from the workflow's sequential jobs — guard 15 + gates 60 (the tool's own CI bound) + publish 120 (release.yml:220) = 195 minutes. Before tagging, reassert HEAD, the tree, and (for an adopted tag) the tag object id; every tag pushed — created or adopted — must carry a signature whose VALIDSIG primary fingerprint equals the one release.yml pins as TAG_SIGNING_FPR. Report the terminal conclusion; on failure name the failing context.
8. **On success**: GET the release object, require it published (not draft), print the tag and asset names.
### Resumability
A failed run must not strand the operator:
- Bump pushed, then failure: rerun continues (preflight sees the version already bumped).
- Local tag exists but never reached origin: verify it is an annotated tag by this tool's convention pointing at the current HEAD — adopt it; otherwise refuse with the exact `git tag -d` to run. Never delete a tag that exists on origin.
### Tests (in-file, join `zig build test`)
Pure functions unit-tested: semver validation (accept/reject table incl. leading zeroes, `v` prefix), bump-kind parse, derivation table with the minor/major resets and overflow refusals, derive-vs-resume decision for all three kinds, zon `.version` parse + rewrite round-trip, changelog heading + non-empty body check, runs-JSON → decision (running / success / failure / no-run), tea-config token extraction. Process spawning and HTTP live behind thin call sites and are not mocked.
## Anti-requirements
- No general release framework; no shared process plumbing extracted unless a third caller appears (container_check.zig:24 rule).
- No confirmation prompts — invoking `just release <kind>` is the authorization.
- No secrets in argv, no token printed.
## Acceptance
- [ ] `just --list` shows the recipes; `just verify` passes locally.
- [ ] `zig build cut -- patch` derives the next version and refuses in preflight on a dirty tree or a missing changelog section, mutating nothing; `zig build cut -- 0.0.9` and `-- banana` refuse naming the three kinds.
- [ ] `zig build test` and `-Dintegration` 0 failed; `zig fmt --check` clean.
+245 -23
View File
@@ -61,7 +61,9 @@ const migrations = @import("storage/migrations.zig");
const model = @import("config/model.zig");
const pause = @import("server/pause.zig");
const pool_mod = @import("upstream/pool.zig");
const queries_repo = @import("storage/repositories/queries_repo.zig");
const query_sink = @import("server/query_sink.zig");
const querylog_schema = @import("storage/querylog_schema.zig");
const rate_limiter = @import("server/rate_limiter.zig");
const reconcile = @import("config/reconcile.zig");
const retention_mod = @import("storage/retention.zig");
@@ -631,29 +633,7 @@ fn serve(r: cli.Runner, args: cli.RunArgs) !u8 {
var querylog_writer_db = querylog_opened.database;
defer querylog_writer_db.close();
// One-shot and already over: the file was recreated during this boot, and
// there is nothing to recover from. Never emitted for `.missing` — a first
// creation renames nothing aside, so the event would carry an aside path
// that does not exist and would greet every fresh install with a warning.
if (querylog_opened.recreated) |cause| {
if (cause != .missing) {
if (event_store) |store| {
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
const detail = std.fmt.bufPrint(&detail_buf, "previous file kept as '{s}'", .{
querylog_opened.aside(),
}) catch detail_buf[0..];
store.reportResolved(
io,
boot_now_s,
.query_log_recreated,
@tagName(cause),
@tagName(cause),
.warning,
detail,
);
}
}
}
reportQuerylogRecreated(event_store, io, boot_now_s, &querylog_opened, &querylog_writer_db);
var querylog_retention_db = try data.reopenQuerylogDb(io);
defer querylog_retention_db.close();
var querylog_history_db = try data.reopenQuerylogDb(io);
@@ -1357,9 +1337,251 @@ fn parseBind(
return addr;
}
/// Files the one-shot `query_log.recreated` event for a boot that replaced the
/// query log.
///
/// One-shot and already over: the file was recreated during this boot, and
/// there is nothing to recover from. Never emitted for `.missing` — a first
/// creation renames nothing aside, so the event would carry an aside path that
/// does not exist and would greet every fresh install with a warning.
///
/// `database` is the connection to the file that was just created; the coverage
/// start is read from it rather than recomputed, so the event states the value
/// the API will.
fn reportQuerylogRecreated(
store: ?*events.Store,
io: std.Io,
now_s: i64,
opened: *const querylog_schema.OpenResult,
database: *db.Db,
) void {
const cause = opened.recreated orelse return;
if (cause == .missing) return;
const s = store orelse return;
// The coverage start belongs in this detail: the recreate is exactly the
// moment the history the operator had stops existing, and the watermark is
// the answer to "from when can I still ask?".
const coverage_start: ?i64 = queries_repo.availableSince(database) catch null;
var detail_buf: [events.Store.max_detail_len]u8 = undefined;
const detail = recreatedDetail(&detail_buf, opened.aside(), coverage_start);
s.reportResolved(io, now_s, .query_log_recreated, @tagName(cause), @tagName(cause), .warning, detail);
}
/// The `query_log.recreated` detail line: what was kept, and from when the new
/// file can answer.
///
/// The coverage start is the operator's actual remedy information — the event
/// says "this history is gone" and this says "and here is where the new history
/// begins". Null only when the fresh file would not answer, which is already a
/// separate failure; the line still names the aside rather than saying nothing.
///
/// The aside is a full path under the data directory, which can be longer than
/// the whole detail column, so the two facts compete for the buffer. The
/// watermark always wins and the name degrades in whole steps: full path, then
/// basename — which the event's own database directory disambiguates — then no
/// name at all. Never a path cut mid-string, which names no file on disk and
/// reads as if it did.
fn recreatedDetail(
buf: *[events.Store.max_detail_len]u8,
aside: []const u8,
coverage_start: ?i64,
) []const u8 {
const names = [_][]const u8{ aside, std.fs.path.basename(aside) };
const since = coverage_start orelse {
for (names) |name| {
return std.fmt.bufPrint(buf, "previous file kept as '{s}'", .{name}) catch continue;
}
return "previous file kept aside";
};
for (names) |name| {
return std.fmt.bufPrint(
buf,
"previous file kept as '{s}'; query history is available from {d}",
.{ name, since },
) catch continue;
}
// The buffer is `max_detail_len`, which no i64 can overrun on its own.
return std.fmt.bufPrint(buf, "query history is available from {d}", .{since}) catch unreachable;
}
const events_fixture = @import("storage/events_fixture.zig");
const testing = std.testing;
test "the recreated detail names the aside and the new coverage start" {
var buf: [events.Store.max_detail_len]u8 = undefined;
try std.testing.expectEqualStrings(
"previous file kept as 'querylog.db.schema-changed-1700000000'; " ++
"query history is available from 1700000001",
recreatedDetail(&buf, "querylog.db.schema-changed-1700000000", 1700000001),
);
// A fresh file that will not answer is a separate failure; the line still
// says what was kept rather than reporting nothing.
try std.testing.expectEqualStrings(
"previous file kept as 'querylog.db.corrupt-1700000000'",
recreatedDetail(&buf, "querylog.db.corrupt-1700000000", null),
);
// A data directory deep enough that its path alone would fill the column:
// the watermark is complete and the name degrades to the basename, which
// still names a real file.
const deep = "/srv/" ++ ("d" ** 60 ++ "/") ** 8 ++ "querylog.db.corrupt-1700000000";
try std.testing.expectEqualStrings(
"previous file kept as 'querylog.db.corrupt-1700000000'; " ++
"query history is available from 1700000001",
recreatedDetail(&buf, deep, 1700000001),
);
try std.testing.expectEqualStrings(
"previous file kept as 'querylog.db.corrupt-1700000000'",
recreatedDetail(&buf, deep, null),
);
// No filesystem produces a name this long, but a truncated one would name
// nothing: the watermark survives alone rather than half-named.
const unnameable = "/srv/" ++ "n" ** 500;
try std.testing.expectEqualStrings(
"query history is available from 1700000001",
recreatedDetail(&buf, unnameable, 1700000001),
);
try std.testing.expectEqualStrings(
"previous file kept aside",
recreatedDetail(&buf, unnameable, null),
);
}
test "a fingerprint recreate files a resolved event naming the real aside and watermark" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
var path_buf: [256]u8 = undefined;
const path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/querylog.db", .{tmp.sub_path});
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
// A fresh install: the file was missing, nothing was set aside, and the
// event would name a path that does not exist.
var created = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
reportQuerylogRecreated(&fx.store, io, 1000, &created, &created.database);
created.database.close();
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
// A healthy file this build's DDL no longer matches, which is what an
// upgrade that edits the schema produces.
{
var stamped = try db.Db.open(path, .{ .mode = .read_write_existing });
defer stamped.close();
var sql_buf: [64]u8 = undefined;
try stamped.exec(try std.fmt.bufPrintZ(
&sql_buf,
"PRAGMA user_version = {d};",
.{querylog_schema.fingerprint +% 1},
));
}
var recreated = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer recreated.database.close();
try testing.expectEqual(querylog_schema.RecreateReason.fingerprint_mismatch, recreated.recreated.?);
reportQuerylogRecreated(&fx.store, io, 2000, &recreated, &recreated.database);
try testing.expectEqualStrings("query_log.recreated", try fx.text("SELECT code FROM operational_events"));
try testing.expectEqualStrings("fingerprint_mismatch", try fx.text("SELECT subject_key FROM operational_events"));
try testing.expectEqualStrings("warning", try fx.text("SELECT severity FROM operational_events"));
// One-shot: already over when it is filed, so it never becomes an open
// episode `/api/health` counts.
try testing.expectEqual(
@as(i64, 0),
try fx.count("SELECT count(*) FROM operational_events WHERE resolved_at IS NULL"),
);
// The detail carries the path that is actually on disk and the watermark
// the API will serve, both read back from the recreate rather than from
// the arguments the event was built with.
try tmp.dir.access(io, std.fs.path.basename(recreated.aside()), .{});
const coverage = try queries_repo.availableSince(&recreated.database);
var expected_buf: [events.Store.max_detail_len]u8 = undefined;
const expected = try std.fmt.bufPrint(
&expected_buf,
"previous file kept as '{s}'; query history is available from {d}",
.{ recreated.aside(), coverage },
);
try testing.expectEqualStrings(expected, try fx.text("SELECT detail FROM operational_events"));
// A boot with no diagnostics store configured is not a failure path.
reportQuerylogRecreated(null, io, 2000, &recreated, &recreated.database);
try testing.expectEqual(@as(i64, 1), try fx.count("SELECT count(*) FROM operational_events"));
}
test "a recreate under a long data directory keeps the watermark and a usable name" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
// Deep enough that the aside outgrows the detail column, shallow enough
// that SQLite's unix VFS still opens the file: it caps a path at 512 bytes.
const nested = ("d" ** 60 ++ "/") ** 6 ++ "d" ** 60;
try tmp.dir.createDirPath(io, nested);
var path_buf: [1024]u8 = undefined;
const path = try std.fmt.bufPrintZ(
&path_buf,
".zig-cache/tmp/{s}/{s}/querylog.db",
.{ tmp.sub_path, nested },
);
var fx: events_fixture.Fixture = .{};
try fx.init(io, 1000);
defer fx.deinit();
{
var created = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer created.database.close();
var sql_buf: [64]u8 = undefined;
try created.database.exec(try std.fmt.bufPrintZ(
&sql_buf,
"PRAGMA user_version = {d};",
.{querylog_schema.fingerprint +% 1},
));
}
var recreated = try querylog_schema.open(io, std.Io.Dir.cwd(), path);
defer recreated.database.close();
try testing.expectEqual(querylog_schema.RecreateReason.fingerprint_mismatch, recreated.recreated.?);
const line_overhead = "previous file kept as ''; query history is available from ".len;
try testing.expect(recreated.aside().len + line_overhead > events.Store.max_detail_len);
reportQuerylogRecreated(&fx.store, io, 2000, &recreated, &recreated.database);
const detail = try fx.text("SELECT detail FROM operational_events");
const coverage = try queries_repo.availableSince(&recreated.database);
const name = std.fs.path.basename(recreated.aside());
var expected_buf: [events.Store.max_detail_len]u8 = undefined;
const expected = try std.fmt.bufPrint(
&expected_buf,
"previous file kept as '{s}'; query history is available from {d}",
.{ name, coverage },
);
// The watermark is whole — the fact that would be lost to a mid-string cut
// — and the name it kept is the file's, not a prefix of its path.
try testing.expectEqualStrings(expected, detail);
try testing.expect(detail.len <= events.Store.max_detail_len);
var deep = try tmp.dir.openDir(io, nested, .{});
defer deep.close(io);
try deep.access(io, name, .{});
}
test "parseBind refuses a bind address of the wrong family" {
var out_buf: [8]u8 = undefined;
var err_buf: [256]u8 = undefined;
+2 -1
View File
@@ -1007,7 +1007,8 @@ fn probeUpstreams(r: Runner, cfg: model.Config) !usize {
}};
var single: pool.Pool = .init(&entries, .{}, timeouts, seed);
if (single.exchange(r.io, probe_query, response_buf)) |_| {
var selected: ?[]const u8 = null;
if (single.exchange(r.io, probe_query, response_buf, &selected)) |_| {
try r.out.print("OK upstreams[{d}] {f}\n", .{ i, safe_url.redact(server.url) });
} else |_| {
// The concrete cause lives in the entry's health, which is where the
+19
View File
@@ -0,0 +1,19 @@
//! Length caps on the configuration labels that the query log copies into every
//! row it writes.
//!
//! They live in a module of their own because two files need them and neither
//! may import the other: `config/validate.zig` rejects a name that exceeds a cap
//! and `storage/logger.zig` sizes an `Entry` buffer from it, while `validate`
//! already imports `logger` for the entry-size budget it derives. A cap owned by
//! either file would close that loop.
//!
//! The values are deliberately short. A group or a source name is a label an
//! operator reads in a table cell, and every byte of it is copied into every
//! logged row — the cap is what keeps a pasted paragraph out of an `Entry` that
//! travels by value through the queue.
/// `groups[].name`.
pub const max_group_name_len = 64;
/// `blocklist_sources[].name`.
pub const max_source_name_len = 64;
+104 -5
View File
@@ -6,9 +6,13 @@
//! instead of a line number, so every `UNIQUE` and every foreign key in
//! PLAN §11.2 has a check here.
//!
//! Pure: no `std.Io` value is a parameter anywhere, no SQLite, no clock. The
//! only `std.Io` type used is `std.Io.Writer`, for rendering diagnostics. The
//! allocator exists for diagnostic text and scratch bookkeeping alone.
//! Pure: no `std.Io` value is a parameter anywhere, no SQLite call, no clock.
//! The only `std.Io` type used is `std.Io.Writer`, for rendering diagnostics.
//! The allocator exists for diagnostic text and scratch bookkeeping alone. The
//! `storage/logger.zig` import is a comptime one — `query_log_buffer_max` is
//! derived from `@sizeOf(logger.Entry)`, because the bound this file enforces
//! on the queue is a bound on bytes and only the logger knows how wide a queued
//! entry is. Nothing in this file calls into storage.
//!
//! Parsers are reused, never reimplemented: `transport.Endpoint.parse` for
//! upstream URLs, `NetAddress.parse` / `Prefix.parse` for addresses, and
@@ -44,6 +48,8 @@ const Writer = std.Io.Writer;
const model = @import("model.zig");
const address = @import("../platform/address.zig");
const dns_name = @import("../dns/name.zig");
const limits = @import("limits.zig");
const logger = @import("../storage/logger.zig");
const regex = @import("../filter/regex.zig");
const safe_url = @import("../safe_url.zig");
const transport = @import("../upstream/transport.zig");
@@ -72,6 +78,7 @@ pub const ValidateError = error{
DuplicateGroupName,
UnknownGroup,
EmptyGroupName,
GroupNameTooLong,
BadClientIp,
DuplicateClientIp,
BadClientPrefix,
@@ -79,6 +86,7 @@ pub const ValidateError = error{
BadSourceUrl,
DuplicateSourceUrl,
EmptySourceName,
SourceNameTooLong,
UnknownSource,
DuplicateGroupSource,
BadRulePattern,
@@ -461,13 +469,16 @@ fn checkScalars(cfg: Config, diags: *Diagnostics) error{OutOfMemory}!void {
if (cfg.logging.query_log_buffer_max < 1) {
try diags.add(error.BadRetention, "logging.query_log_buffer_max", .{}, "must be at least 1", .{});
}
if (cfg.logging.query_log_buffer_max > max_boot_entries) {
// Its own ceiling, not `max_boot_entries`: a queued `logger.Entry` carries
// every provenance field by value, so the queue's cost is bytes rather than
// entries and the bound follows the width of the entry.
if (cfg.logging.query_log_buffer_max > logger.query_log_buffer_max) {
try diags.add(
error.BadRetention,
"logging.query_log_buffer_max",
.{},
"must be at most {d}, got {d}",
.{ max_boot_entries, cfg.logging.query_log_buffer_max },
.{ logger.query_log_buffer_max, cfg.logging.query_log_buffer_max },
);
}
// No floor: 0 is the documented "do not wait" setting, not a mistake.
@@ -714,6 +725,29 @@ fn checkDotHost(
};
}
/// The shared shape of the two label caps.
///
/// Both names are copied by value into every `query_log` row that mentions
/// them, so the cap is what keeps a pasted paragraph out of the fixed buffers
/// of `storage/logger.zig`. Bytes, not codepoints: the buffer counts bytes.
fn checkNameLength(
diags: *Diagnostics,
comptime fault: ValidateError,
comptime path: []const u8,
path_args: anytype,
value: []const u8,
cap: usize,
) error{OutOfMemory}!void {
if (value.len <= cap) return;
try diags.add(
fault,
path,
path_args,
"must be at most {d} bytes, got {d}; the name is copied into every logged query",
.{ cap, value.len },
);
}
fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{OutOfMemory}!void {
var group_names: IndexSet = .empty;
var has_default = false;
@@ -729,6 +763,16 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{
.{safe_url.quoteText(group.name)},
);
}
// Independent of the chain above: an over-long name is still a name,
// and a duplicate of one is still a duplicate.
try checkNameLength(
diags,
error.GroupNameTooLong,
"groups[{d}].name",
.{i},
group.name,
limits.max_group_name_len,
);
if (std.mem.eql(u8, group.name, "default")) has_default = true;
}
if (!has_default) {
@@ -859,6 +903,14 @@ fn checkCollections(cfg: Config, diags: *Diagnostics, scratch: Allocator) error{
.{},
);
}
try checkNameLength(
diags,
error.SourceNameTooLong,
"blocklist_sources[{d}].name",
.{i},
source.name,
limits.max_source_name_len,
);
}
var group_source_pairs: IndexSet = .empty;
@@ -1308,6 +1360,24 @@ test "an https:// upstream may name a host" {
try expectClean(cfg);
}
/// The longest host `transport.Endpoint.parse` accepts: four labels, 253 bytes.
const host_at_bound = ("a" ** 63 ++ ".") ** 3 ++ "a" ** 61;
test "an upstream host at the length bound validates cleanly" {
var cfg = baseConfig();
cfg.upstreams = &.{.{ .url = "https://" ++ host_at_bound ++ "/dns-query" }};
try expectClean(cfg);
}
test "error.BadUpstreamUrl on an upstream host one byte past the length bound" {
// The bound is the query log's `upstream` width and every other identity
// built from the endpoint, so an over-long host has to fail here rather
// than be shortened downstream.
var cfg = baseConfig();
cfg.upstreams = &.{.{ .url = "https://" ++ host_at_bound ++ "a/dns-query" }};
try expectProblem(cfg, error.BadUpstreamUrl, "upstreams[0].url");
}
test "an IPv6 literal tls:// upstream validates cleanly" {
// `Endpoint.parse` strips the brackets, so the host reaching the check is
// exactly what the client hands to the address parser.
@@ -1371,6 +1441,35 @@ test "error.EmptyGroupName" {
try expectProblem(cfg, error.EmptyGroupName, "groups[1].name");
}
test "error.GroupNameTooLong" {
const cap = limits.max_group_name_len;
// Exactly at the cap is accepted; one byte past it is not. The cap is what
// `storage/logger.zig` sizes its `Entry` buffer from, so a name that passes
// here is a name a logged row stores whole.
var at_cap = baseConfig();
at_cap.groups = &.{ .{ .name = "default" }, .{ .name = "g" ** cap } };
try expectClean(at_cap);
var over = baseConfig();
over.groups = &.{ .{ .name = "default" }, .{ .name = "g" ** (cap + 1) } };
try expectProblem(over, error.GroupNameTooLong, "groups[1].name");
}
test "error.SourceNameTooLong" {
const cap = limits.max_source_name_len;
const url = "https://lists.example/hosts.txt";
var at_cap = baseConfig();
at_cap.blocklist_sources = &.{.{ .url = url, .name = "s" ** cap }};
at_cap.group_sources = &.{.{ .group = "default", .source_url = url }};
try expectClean(at_cap);
var over = baseConfig();
over.blocklist_sources = &.{.{ .url = url, .name = "s" ** (cap + 1) }};
over.group_sources = &.{.{ .group = "default", .source_url = url }};
try expectProblem(over, error.SourceNameTooLong, "blocklist_sources[0].name");
}
test "error.BadClientIp" {
var cfg = baseConfig();
cfg.clients = &.{.{ .ip = "nonsense" }};
+62 -1
View File
@@ -33,8 +33,17 @@ const log = std.log.scoped(.forward_client);
/// each half large enough to frame a query in one write, not a capacity.
pub const min_frame_buf: usize = 1024;
/// `tcp://[` + the longest IPv6 text form + `]:65535`, the widest spelling
/// `identityText` can produce.
pub const max_identity_len: usize = "tcp://[".len + 45 + "]:65535".len;
pub const ForwardClient = struct {
resolver: validate.Resolver,
/// The resolver as text, owned here so the `transport.Client` out-parameter
/// has something stable to borrow: `validate.Resolver` is a parsed address,
/// and a caller logging the exchange needs its spelling.
identity_buf: [max_identity_len]u8 = undefined,
identity_len: usize = 0,
/// Caller-owned scratch for the TCP length-prefixed path.
frame_buf: []u8,
/// On the `.awake` clock at the caller's choosing, so a suspended host does
@@ -64,11 +73,20 @@ pub const ForwardClient = struct {
read_timeout: std.Io.Clock.Duration,
) ForwardClient {
std.debug.assert(frame_buf.len >= min_frame_buf);
return .{
var self: ForwardClient = .{
.resolver = resolver,
.frame_buf = frame_buf,
.read_timeout = read_timeout,
};
self.identity_len = identityText(resolver, &self.identity_buf).len;
return self;
}
/// `udp://192.168.1.1:53`, `tcp://[fd00::1]:53` — the same spelling
/// `validate.parseResolver` accepts, so a log row names the configured
/// value. Valid for as long as this client is.
pub fn identity(self: *const ForwardClient) []const u8 {
return self.identity_buf[0..self.identity_len];
}
pub fn client(self: *ForwardClient) transport.Client {
@@ -80,8 +98,12 @@ pub const ForwardClient = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
const self: *ForwardClient = @ptrCast(@alignCast(ptr));
// Set before the attempt: a failed forward-zone exchange still names
// the resolver it was sent to.
selected.* = self.identity();
return self.exchange(io, query, response_buf);
}
@@ -238,6 +260,24 @@ pub const ForwardClient = struct {
}
};
fn identityText(resolver: validate.Resolver, buf: *[max_identity_len]u8) []const u8 {
var w: std.Io.Writer = .fixed(buf);
w.writeAll(switch (resolver.scheme) {
.udp => "udp://",
.tcp => "tcp://",
}) catch unreachable;
const bracketed = switch (resolver.addr) {
.ip4 => false,
.ip6 => true,
};
if (bracketed) w.writeByte('[') catch unreachable;
resolver.addr.format(&w) catch unreachable;
if (bracketed) w.writeByte(']') catch unreachable;
w.print(":{d}", .{resolver.port}) catch unreachable;
return w.buffered();
}
/// The local address a datagram to `dest` is sent from: same family, port
/// chosen by the kernel.
fn wildcardFor(dest: net.IpAddress) net.IpAddress {
@@ -286,6 +326,27 @@ test "ForwardClient satisfies the Client interface" {
try testing.expectEqual(@as(u16, 53), fc.resolver.port);
}
test "the client owns its resolver identity in both address families" {
var buf = testBuf();
const v4: ForwardClient = .init(
try validate.parseResolver("udp://192.168.1.1:5300"),
&buf,
.{ .raw = .fromSeconds(1), .clock = .awake },
);
try testing.expectEqualStrings("udp://192.168.1.1:5300", v4.identity());
var buf6 = testBuf();
const v6: ForwardClient = .init(
try validate.parseResolver("tcp://[fd00::1]:5353"),
&buf6,
.{ .raw = .fromSeconds(1), .clock = .awake },
);
try testing.expectEqualStrings("tcp://[fd00::1]:5353", v6.identity());
// The borrow points into the client, not into `init`'s frame.
try testing.expect(@intFromPtr(v6.identity().ptr) >= @intFromPtr(&v6));
}
test "the stats struct starts at zero" {
const stats: ForwardClient.Stats = .{};
try testing.expectEqual(@as(u64, 0), stats.queries);
+2
View File
@@ -592,11 +592,13 @@ const FailingUpstream = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = ptr;
_ = io;
_ = query;
_ = response_buf;
selected.* = "fake://failing-upstream";
return error.ConnectFailed;
}
+2
View File
@@ -351,8 +351,10 @@ const FakeUpstream = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = io;
selected.* = "fake://dot-server-upstream";
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
@memcpy(response_buf[0..self.reply.len], self.reply);
+1390 -183
View File
File diff suppressed because it is too large Load Diff
+9 -3
View File
@@ -33,6 +33,7 @@ const handler = @import("handler.zig");
const header = @import("../dns/header.zig");
const local_tables = @import("local_tables.zig");
const logger_mod = @import("../storage/logger.zig");
const provenance = @import("../storage/provenance.zig");
const manager = @import("../filter/manager.zig");
const matcher = @import("../filter/matcher.zig");
const migrations = @import("../storage/migrations.zig");
@@ -183,8 +184,10 @@ const FakeUpstream = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = io;
selected.* = "fake://phase7-upstream";
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
_ = self.calls.fetchAdd(1, .monotonic);
@@ -346,7 +349,8 @@ test "S7 case 1: a blocked domain is answered with the zero address and logged"
try testing.expectEqual(@as(usize, 1), logged.len);
try testing.expectEqual(true, logged[0].blocked);
try testing.expectEqualStrings("ads.example.com", logged[0].domain());
try testing.expectEqualStrings("rule_block_exact", logged[0].blockReason());
try testing.expectEqual(provenance.PolicyAction.block, logged[0].policy_action);
try testing.expectEqual(provenance.PolicyReason.rule_block_exact, logged[0].policy_reason);
try testing.expectEqualStrings("127.0.0.1", logged[0].clientIp());
}
@@ -593,7 +597,7 @@ test "S7 case 5: a cached answer comes back with a fresh id, an aged ttl and a l
const logged = drainLog(&lg, io, &entries);
try testing.expectEqual(@as(usize, 3), logged.len);
try testing.expectEqual(@as(?bool, false), logged[0].cache_hit);
try testing.expectEqualStrings("pool", logged[0].upstream());
try testing.expectEqualStrings("fake://phase7-upstream", logged[0].upstream());
try testing.expectEqual(@as(?bool, true), logged[1].cache_hit);
try testing.expectEqualStrings("", logged[1].upstream());
try testing.expectEqual(@as(?bool, true), logged[2].cache_hit);
@@ -651,7 +655,9 @@ test "S7 case 6: a cname into a blocked target blocks the original question" {
const logged = drainLog(&lg, io, &entries);
try testing.expectEqual(@as(usize, 1), logged.len);
try testing.expectEqual(true, logged[0].blocked);
try testing.expectEqualStrings("cname:rule_block_exact", logged[0].blockReason());
// The reason describes the target's own decision; milestone 28 S3 adds the
// target name that says a CNAME chain was followed.
try testing.expectEqual(provenance.PolicyReason.rule_block_exact, logged[0].policy_reason);
try testing.expectEqualStrings("cdn.example.com", logged[0].domain());
}
+4
View File
@@ -114,8 +114,10 @@ const GoodUpstream = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = io;
selected.* = "fake://good-upstream";
const self: *GoodUpstream = @ptrCast(@alignCast(ptr));
_ = self.calls.fetchAdd(1, .monotonic);
return answerQuery(query, response_buf);
@@ -138,8 +140,10 @@ const FaultyUpstream = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = io;
selected.* = "fake://faulty-upstream";
const self: *FaultyUpstream = @ptrCast(@alignCast(ptr));
const seen = self.calls.fetchAdd(1, .monotonic);
if (seen < self.fail_first) return self.fault;
@@ -72,8 +72,10 @@ const FakeUpstream = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = io;
selected.* = "fake://tcp-server-upstream";
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
@memcpy(response_buf[0..self.reply.len], self.reply);
@@ -73,8 +73,10 @@ const FakeUpstream = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = io;
selected.* = "fake://udp-server-upstream";
const self: *FakeUpstream = @ptrCast(@alignCast(ptr));
if (self.reply.len > response_buf.len) return error.ResponseTooLarge;
@memcpy(response_buf[0..self.reply.len], self.reply);
+416 -53
View File
@@ -34,8 +34,12 @@ const std = @import("std");
const db = @import("db.zig");
const disk_monitor = @import("disk_monitor.zig");
const events = @import("events.zig");
const limits = @import("../config/limits.zig");
const model = @import("../config/model.zig");
const provenance = @import("provenance.zig");
const queries_repo = @import("repositories/queries_repo.zig");
const regex = @import("../filter/regex.zig");
const safe_url = @import("../safe_url.zig");
/// Named `scope` rather than `log`: `Logger.log` is the enqueue entry point,
/// and the two names collide inside the struct.
@@ -58,10 +62,32 @@ pub const gate_retry_s = 1;
pub const max_domain_len = 253;
/// RFC 5952 text of any IPv6 address, zone identifier included.
pub const max_client_len = 45;
pub const max_reason_len = 32;
pub const max_upstream_len = 64;
/// `matched` holds the rule that decided the query, and the widest rule the
/// configuration accepts is a regex pattern at `regex.max_pattern_len`. It does
/// not fit a `u8` length, which is why this one field carries a `u16`.
pub const max_matched_len = regex.max_pattern_len;
/// The redacted resolver identity of the exchange that actually happened. Two
/// bounds apply and the buffer takes the larger, so neither producer truncates:
/// the longest well-formed `scheme://host:port` with a maximal host, and
/// `safe_url.redact`'s own output bound (`max_len` plus the `...` it appends
/// when it truncates).
pub const max_upstream_len = @max(
"https://".len + max_domain_len + ":65535".len,
safe_url.max_len + 3,
);
/// `cname_target`, `safe_search_target` and `forward_zone` each hold a domain
/// name, so they are all the same width as `domain`.
const max_name_len = max_domain_len;
/// One row on its way to `query_log`, carrying its own bytes.
///
/// Every field is by value: `Io.Queue` copies elements as raw bytes, so nothing
/// here may borrow from the query that produced it. The buffer widths above are
/// therefore the row's real storage cost, multiplied by the queue capacity —
/// see `query_log_buffer_max`.
pub const Entry = struct {
timestamp: i64,
domain_buf: [max_domain_len]u8,
@@ -69,28 +95,68 @@ pub const Entry = struct {
client_buf: [max_client_len]u8,
client_len: u8,
qtype: ?u16,
qclass: u16,
/// The client-visible RCODE. Twelve bits, not four: an EDNS extended code
/// carries eight more bits in the OPT record than the header's four. The
/// type is the enforcement — the column's `CHECK` in `querylog_schema.ddl`
/// bounds the same value against every other writer of the file.
rcode: u12,
blocked: bool,
reason_buf: [max_reason_len]u8,
reason_len: u8,
response_time_us: ?i64,
cache_hit: ?bool,
upstream_buf: [max_upstream_len]u8,
upstream_len: u8,
upstream_len: u16,
group_id: ?i64,
group_buf: [limits.max_group_name_len]u8,
group_len: u8,
policy_action: provenance.PolicyAction,
policy_reason: provenance.PolicyReason,
matched_buf: [max_matched_len]u8,
matched_len: u16,
source_id: ?i64,
source_buf: [limits.max_source_name_len]u8,
source_len: u8,
cname_buf: [max_name_len]u8,
cname_len: u8,
safe_search_buf: [max_name_len]u8,
safe_search_len: u8,
route_kind: provenance.RouteKind,
forward_zone_buf: [max_name_len]u8,
forward_zone_len: u8,
/// The borrowed shape of an entry. `init` copies out of it, so a caller can
/// build one from slices that die with the query.
///
/// Every text field defaults to `""`, which reaches a nullable column as
/// NULL. The three fields with no sensible empty value — the two enums and
/// the route — default to what a query the pipeline has not yet explained
/// would honestly say about itself.
pub const Fields = struct {
timestamp: i64,
domain: []const u8,
client_ip: []const u8,
qtype: ?u16 = null,
qclass: u16 = 0,
rcode: u12 = 0,
blocked: bool = false,
/// Empty means "no reason", which reaches the database as NULL.
block_reason: []const u8 = "",
response_time_us: ?i64 = null,
cache_hit: ?bool = null,
/// Empty means "no upstream", which reaches the database as NULL.
/// Empty means "no upstream was attempted", which reaches the database
/// as NULL. Already redacted by the caller.
upstream: []const u8 = "",
group_id: ?i64 = null,
group_name: []const u8 = "",
policy_action: provenance.PolicyAction = .not_evaluated,
policy_reason: provenance.PolicyReason = .no_match,
matched: []const u8 = "",
source_id: ?i64 = null,
source_name: []const u8 = "",
cname_target: []const u8 = "",
safe_search_target: []const u8 = "",
route_kind: provenance.RouteKind = .upstream,
forward_zone: []const u8 = "",
};
/// Copies each string in, truncated to what its buffer holds. A name longer
@@ -104,27 +170,61 @@ pub const Entry = struct {
.client_buf = undefined,
.client_len = 0,
.qtype = f.qtype,
.qclass = f.qclass,
.rcode = f.rcode,
.blocked = f.blocked,
.reason_buf = undefined,
.reason_len = 0,
.response_time_us = f.response_time_us,
.cache_hit = f.cache_hit,
.upstream_buf = undefined,
.upstream_len = 0,
.group_id = f.group_id,
.group_buf = undefined,
.group_len = 0,
.policy_action = f.policy_action,
.policy_reason = f.policy_reason,
.matched_buf = undefined,
.matched_len = 0,
.source_id = f.source_id,
.source_buf = undefined,
.source_len = 0,
.cname_buf = undefined,
.cname_len = 0,
.safe_search_buf = undefined,
.safe_search_len = 0,
.route_kind = f.route_kind,
.forward_zone_buf = undefined,
.forward_zone_len = 0,
};
entry.setDomain(f.domain);
entry.setClientIp(f.client_ip);
entry.reason_len = copyInto(&entry.reason_buf, f.block_reason);
entry.upstream_len = copyInto(&entry.upstream_buf, f.upstream);
copyInto(&entry.upstream_buf, &entry.upstream_len, f.upstream);
copyInto(&entry.group_buf, &entry.group_len, f.group_name);
entry.setMatched(f.matched);
copyInto(&entry.source_buf, &entry.source_len, f.source_name);
entry.setCnameTarget(f.cname_target);
entry.setSafeSearchTarget(f.safe_search_target);
copyInto(&entry.forward_zone_buf, &entry.forward_zone_len, f.forward_zone);
return entry;
}
pub fn setDomain(self: *Entry, value: []const u8) void {
self.domain_len = copyInto(&self.domain_buf, value);
copyInto(&self.domain_buf, &self.domain_len, value);
}
pub fn setClientIp(self: *Entry, value: []const u8) void {
self.client_len = copyInto(&self.client_buf, value);
copyInto(&self.client_buf, &self.client_len, value);
}
pub fn setMatched(self: *Entry, value: []const u8) void {
copyInto(&self.matched_buf, &self.matched_len, value);
}
pub fn setCnameTarget(self: *Entry, value: []const u8) void {
copyInto(&self.cname_buf, &self.cname_len, value);
}
pub fn setSafeSearchTarget(self: *Entry, value: []const u8) void {
copyInto(&self.safe_search_buf, &self.safe_search_len, value);
}
pub fn domain(self: *const Entry) []const u8 {
@@ -135,19 +235,56 @@ pub const Entry = struct {
return self.client_buf[0..self.client_len];
}
pub fn blockReason(self: *const Entry) []const u8 {
return self.reason_buf[0..self.reason_len];
}
pub fn upstream(self: *const Entry) []const u8 {
return self.upstream_buf[0..self.upstream_len];
}
pub fn groupName(self: *const Entry) []const u8 {
return self.group_buf[0..self.group_len];
}
pub fn matched(self: *const Entry) []const u8 {
return self.matched_buf[0..self.matched_len];
}
pub fn sourceName(self: *const Entry) []const u8 {
return self.source_buf[0..self.source_len];
}
pub fn cnameTarget(self: *const Entry) []const u8 {
return self.cname_buf[0..self.cname_len];
}
pub fn safeSearchTarget(self: *const Entry) []const u8 {
return self.safe_search_buf[0..self.safe_search_len];
}
pub fn forwardZone(self: *const Entry) []const u8 {
return self.forward_zone_buf[0..self.forward_zone_len];
}
};
fn copyInto(buf: []u8, value: []const u8) u8 {
/// The memory budget the queue is allowed to occupy. `Entry` travels by value,
/// so the composition root allocates `query_log_buffer_max` of them in full at
/// boot (`app.zig`) and the SSE hub embeds a ring of them per subscriber.
const queue_budget_bytes = 64 * 1024 * 1024;
/// The ceiling `config/validate.zig` enforces on `logging.query_log_buffer_max`,
/// derived from the width of `Entry` rather than picked.
///
/// The provenance columns of milestone 28 roughly tripled `Entry`, so the bound
/// that matters is bytes, not entries: an operator who asks for a million
/// entries is asking for well over a gigabyte of queue. This is a sanity bound,
/// not a memory-fit guarantee — what actually fits depends on the box.
pub const query_log_buffer_max: u32 = @intCast(queue_budget_bytes / @sizeOf(Entry));
/// Copies as much of `value` as `buf` holds, and stores the length through
/// `len`. `len`'s type never bounds anything — `buf.len` does — so the same
/// helper serves the `u8` fields and the `u16` ones.
fn copyInto(buf: []u8, len: anytype, value: []const u8) void {
const n = @min(buf.len, value.len);
@memcpy(buf[0..n], value[0..n]);
return @intCast(n);
len.* = @intCast(n);
}
/// The row borrows from `entry`, which must outlive the `writeBatch` call.
@@ -157,11 +294,23 @@ fn toRow(entry: *const Entry) queries_repo.Row {
.domain = entry.domain(),
.client_ip = entry.clientIp(),
.qtype = entry.qtype,
.qclass = entry.qclass,
.rcode = entry.rcode,
.blocked = entry.blocked,
.block_reason = emptyAsNull(entry.blockReason()),
.response_time_us = entry.response_time_us,
.cache_hit = entry.cache_hit,
.upstream = emptyAsNull(entry.upstream()),
.group_id = entry.group_id,
.group_name = emptyAsNull(entry.groupName()),
.policy_action = entry.policy_action,
.policy_reason = entry.policy_reason,
.matched = emptyAsNull(entry.matched()),
.source_id = entry.source_id,
.source_name = emptyAsNull(entry.sourceName()),
.cname_target = emptyAsNull(entry.cnameTarget()),
.safe_search_target = emptyAsNull(entry.safeSearchTarget()),
.route_kind = entry.route_kind,
.forward_zone = emptyAsNull(entry.forwardZone()),
};
}
@@ -231,9 +380,23 @@ pub const Logger = struct {
/// and hands the result to every consumer, so nothing downstream — the
/// database or the event stream — can observe a value the operator asked
/// to hide.
/// `hide_domains` covers every field derived from the query name, not just
/// `domain`: a matched wildcard, a CNAME target and a safe-search target
/// each name the very thing the operator asked to keep out of the log.
///
/// `forward_zone`, `group_name` and `source_name` stay visible. They are
/// configuration labels the operator wrote, identical on every row that
/// hits them, and they say nothing about which name a client looked up.
pub fn transformed(self: *const Logger, entry: Entry) Entry {
var out = entry;
if (self.cfg.hide_domains) out.setDomain(hidden_marker);
if (self.cfg.hide_domains) {
out.setDomain(hidden_marker);
// Only where there is something to hide: an empty field means the
// query had no such value, and writing a marker would claim it did.
if (out.matched_len != 0) out.setMatched(hidden_marker);
if (out.cname_len != 0) out.setCnameTarget(hidden_marker);
if (out.safe_search_len != 0) out.setSafeSearchTarget(hidden_marker);
}
if (self.cfg.hide_client_ips) out.setClientIp(hidden_marker);
return out;
}
@@ -574,6 +737,34 @@ fn sampleEntry(timestamp: i64, domain: []const u8) Entry {
});
}
/// Every provenance field set to a distinct recognisable value, so a test that
/// loses one loses it visibly.
fn fullFields(timestamp: i64) Entry.Fields {
return .{
.timestamp = timestamp,
.domain = "ads.example.com",
.client_ip = "2001:db8::1",
.qtype = 28,
.qclass = 1,
.rcode = 3,
.blocked = true,
.response_time_us = 42,
.cache_hit = true,
.upstream = "https://dns.example/dns-query",
.group_id = 7,
.group_name = "kids",
.policy_action = .block,
.policy_reason = .blocklist_wildcard,
.matched = "*.ads.example",
.source_id = 3,
.source_name = "steven black",
.cname_target = "tracker.cdn.example",
.safe_search_target = "forcesafesearch.google.com",
.route_kind = .blocked,
.forward_zone = "home.arpa",
};
}
fn openLog() !db.Db {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
errdefer database.close();
@@ -583,44 +774,72 @@ fn openLog() !db.Db {
}
test "an entry carries its own bytes and reads them back" {
const entry: Entry = .init(.{
.timestamp = 1700000000,
.domain = "ads.example.com",
.client_ip = "2001:db8::1",
.qtype = 28,
.blocked = true,
.block_reason = "blocklist",
.response_time_us = 42,
.cache_hit = true,
.upstream = "dns.example",
});
const entry: Entry = .init(fullFields(1700000000));
try testing.expectEqualStrings("ads.example.com", entry.domain());
try testing.expectEqualStrings("2001:db8::1", entry.clientIp());
try testing.expectEqualStrings("blocklist", entry.blockReason());
try testing.expectEqualStrings("dns.example", entry.upstream());
try testing.expectEqualStrings("https://dns.example/dns-query", entry.upstream());
try testing.expectEqual(@as(?u16, 28), entry.qtype);
try testing.expectEqual(@as(u16, 1), entry.qclass);
try testing.expectEqual(@as(u12, 3), entry.rcode);
try testing.expect(entry.blocked);
try testing.expectEqual(@as(?i64, 42), entry.response_time_us);
try testing.expectEqual(@as(?bool, true), entry.cache_hit);
try testing.expectEqual(@as(?i64, 7), entry.group_id);
try testing.expectEqualStrings("kids", entry.groupName());
try testing.expectEqual(provenance.PolicyAction.block, entry.policy_action);
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, entry.policy_reason);
try testing.expectEqualStrings("*.ads.example", entry.matched());
try testing.expectEqual(@as(?i64, 3), entry.source_id);
try testing.expectEqualStrings("steven black", entry.sourceName());
try testing.expectEqualStrings("tracker.cdn.example", entry.cnameTarget());
try testing.expectEqualStrings("forcesafesearch.google.com", entry.safeSearchTarget());
try testing.expectEqual(provenance.RouteKind.blocked, entry.route_kind);
try testing.expectEqualStrings("home.arpa", entry.forwardZone());
}
test "an oversize string is truncated to what its buffer holds" {
const long_domain = "a" ** 400;
const entry: Entry = .init(.{
.timestamp = 1,
.domain = long_domain,
.client_ip = "192.0.2.1",
.block_reason = "r" ** 64,
.upstream = "u" ** 128,
.domain = "a" ** 400,
.client_ip = "c" ** 80,
.upstream = "u" ** 600,
.group_name = "g" ** 200,
.matched = "m" ** 600,
.source_name = "s" ** 200,
.cname_target = "n" ** 400,
.safe_search_target = "f" ** 400,
.forward_zone = "z" ** 400,
});
try testing.expectEqual(@as(usize, max_domain_len), entry.domain().len);
try testing.expectEqual(@as(usize, max_reason_len), entry.blockReason().len);
try testing.expectEqual(@as(usize, max_client_len), entry.clientIp().len);
try testing.expectEqual(@as(usize, max_upstream_len), entry.upstream().len);
try testing.expectEqual(@as(usize, limits.max_group_name_len), entry.groupName().len);
try testing.expectEqual(@as(usize, max_matched_len), entry.matched().len);
try testing.expectEqual(@as(usize, limits.max_source_name_len), entry.sourceName().len);
try testing.expectEqual(@as(usize, max_name_len), entry.cnameTarget().len);
try testing.expectEqual(@as(usize, max_name_len), entry.safeSearchTarget().len);
try testing.expectEqual(@as(usize, max_name_len), entry.forwardZone().len);
try testing.expectEqualStrings("a" ** max_domain_len, entry.domain());
}
test "a 256-byte matched pattern is stored whole" {
// The widest rule the configuration accepts is a regex at
// `regex.max_pattern_len`, and it does not fit a `u8` length — which is the
// whole reason `matched_len` is a `u16`.
const widest = "p" ** regex.max_pattern_len;
const entry: Entry = .init(.{
.timestamp = 1,
.domain = "example.com",
.client_ip = "192.0.2.1",
.matched = widest,
});
try testing.expectEqualStrings(widest, entry.matched());
try testing.expectEqual(@as(u16, regex.max_pattern_len), entry.matched_len);
}
test "toRow maps the empty strings to null and passes the rest through" {
const bare: Entry = .init(.{
.timestamp = 7,
@@ -631,23 +850,85 @@ test "toRow maps the empty strings to null and passes the rest through" {
try testing.expectEqual(@as(i64, 7), bare_row.timestamp);
try testing.expectEqualStrings("example.com", bare_row.domain);
try testing.expectEqualStrings("192.0.2.5", bare_row.client_ip);
try testing.expectEqual(@as(?[]const u8, null), bare_row.block_reason);
try testing.expectEqual(@as(?[]const u8, null), bare_row.upstream);
try testing.expectEqual(@as(?u16, null), bare_row.qtype);
try testing.expectEqual(@as(?bool, null), bare_row.cache_hit);
// Every optional text field of an entry nothing filled in reaches its
// column as NULL rather than as an empty string.
try testing.expectEqual(@as(?[]const u8, null), bare_row.upstream);
try testing.expectEqual(@as(?[]const u8, null), bare_row.group_name);
try testing.expectEqual(@as(?[]const u8, null), bare_row.matched);
try testing.expectEqual(@as(?[]const u8, null), bare_row.source_name);
try testing.expectEqual(@as(?[]const u8, null), bare_row.cname_target);
try testing.expectEqual(@as(?[]const u8, null), bare_row.safe_search_target);
try testing.expectEqual(@as(?[]const u8, null), bare_row.forward_zone);
const full: Entry = .init(.{
.timestamp = 8,
.domain = "blocked.example",
.client_ip = "192.0.2.6",
.blocked = true,
.block_reason = "blocklist",
.upstream = "9.9.9.9",
});
const full: Entry = .init(fullFields(8));
const full_row = toRow(&full);
try testing.expect(full_row.blocked);
try testing.expectEqualStrings("blocklist", full_row.block_reason.?);
try testing.expectEqualStrings("9.9.9.9", full_row.upstream.?);
try testing.expectEqual(@as(u16, 1), full_row.qclass);
try testing.expectEqual(@as(u12, 3), full_row.rcode);
try testing.expectEqualStrings("https://dns.example/dns-query", full_row.upstream.?);
try testing.expectEqual(@as(?i64, 7), full_row.group_id);
try testing.expectEqualStrings("kids", full_row.group_name.?);
try testing.expectEqual(provenance.PolicyAction.block, full_row.policy_action);
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, full_row.policy_reason);
try testing.expectEqualStrings("*.ads.example", full_row.matched.?);
try testing.expectEqual(@as(?i64, 3), full_row.source_id);
try testing.expectEqualStrings("steven black", full_row.source_name.?);
try testing.expectEqualStrings("tracker.cdn.example", full_row.cname_target.?);
try testing.expectEqualStrings("forcesafesearch.google.com", full_row.safe_search_target.?);
try testing.expectEqual(provenance.RouteKind.blocked, full_row.route_kind);
try testing.expectEqualStrings("home.arpa", full_row.forward_zone.?);
}
test "an entry with every provenance field set survives the queue, toRow, insert and detailById" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var database = try openLog();
defer database.close();
var writer = try queries_repo.BatchWriter.init(&database);
defer writer.deinit();
var buf: [4]Entry = undefined;
var logger: Logger = .init(.{}, &buf);
// The widest `matched` the configuration accepts, carried the whole way:
// 256 bytes does not fit the `u8` length every other text field uses.
const widest_matched = "p" ** max_matched_len;
var fields = fullFields(1234);
fields.matched = widest_matched;
logger.log(io, .init(fields));
const queued = try logger.queue.getOne(io);
try logger.flush(io, &writer, &.{queued}, null);
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const stored = (try queries_repo.detailById(&database, arena_state.allocator(), 1)).?;
try testing.expectEqual(@as(i64, 1234), stored.ts);
try testing.expectEqualStrings("ads.example.com", stored.domain);
try testing.expectEqualStrings("2001:db8::1", stored.client_ip);
try testing.expectEqual(@as(?u16, 28), stored.qtype);
try testing.expectEqual(@as(u16, 1), stored.qclass);
try testing.expectEqual(@as(u12, 3), stored.rcode);
try testing.expect(stored.blocked);
try testing.expectEqual(@as(?i64, 42), stored.response_time_us);
try testing.expectEqual(@as(?bool, true), stored.cache_hit);
try testing.expectEqualStrings("https://dns.example/dns-query", stored.upstream);
try testing.expectEqual(@as(?i64, 7), stored.group_id);
try testing.expectEqualStrings("kids", stored.group_name);
try testing.expectEqual(provenance.PolicyAction.block, stored.policy_action);
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, stored.policy_reason);
try testing.expectEqualStrings(widest_matched, stored.matched);
try testing.expectEqual(@as(?i64, 3), stored.source_id);
try testing.expectEqualStrings("steven black", stored.source_name);
try testing.expectEqualStrings("tracker.cdn.example", stored.cname_target);
try testing.expectEqualStrings("forcesafesearch.google.com", stored.safe_search_target);
try testing.expectEqual(provenance.RouteKind.blocked, stored.route_kind);
try testing.expectEqualStrings("home.arpa", stored.forward_zone);
}
test "log applies both privacy transforms before the entry reaches the queue" {
@@ -667,6 +948,88 @@ test "log applies both privacy transforms before the entry reaches the queue" {
try testing.expectEqual(@as(u64, 0), logger.queries_dropped.load(.monotonic));
}
test "hide_domains hides every query-derived name and leaves the labels alone" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var buf: [4]Entry = undefined;
var logger: Logger = .init(.{ .hide_domains = true }, &buf);
logger.log(io, .init(fullFields(1)));
const hidden = try logger.queue.getOne(io);
// Every field derived from the name the client asked for.
try testing.expectEqualStrings(hidden_marker, hidden.domain());
try testing.expectEqualStrings(hidden_marker, hidden.matched());
try testing.expectEqualStrings(hidden_marker, hidden.cnameTarget());
try testing.expectEqualStrings(hidden_marker, hidden.safeSearchTarget());
// The client is governed by `hide_client_ips`, not by this flag.
try testing.expectEqualStrings("2001:db8::1", hidden.clientIp());
// Configuration labels the operator wrote. They are identical on every row
// that hits them and say nothing about which name a client looked up.
try testing.expectEqualStrings("kids", hidden.groupName());
try testing.expectEqualStrings("steven black", hidden.sourceName());
try testing.expectEqualStrings("home.arpa", hidden.forwardZone());
try testing.expectEqualStrings("https://dns.example/dns-query", hidden.upstream());
}
test "hide_client_ips hides the client and nothing else" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var buf: [4]Entry = undefined;
var logger: Logger = .init(.{ .hide_client_ips = true }, &buf);
logger.log(io, .init(fullFields(1)));
const hidden = try logger.queue.getOne(io);
try testing.expectEqualStrings(hidden_marker, hidden.clientIp());
try testing.expectEqualStrings("ads.example.com", hidden.domain());
try testing.expectEqualStrings("*.ads.example", hidden.matched());
try testing.expectEqualStrings("tracker.cdn.example", hidden.cnameTarget());
try testing.expectEqualStrings("forcesafesearch.google.com", hidden.safeSearchTarget());
try testing.expectEqualStrings("kids", hidden.groupName());
try testing.expectEqualStrings("steven black", hidden.sourceName());
try testing.expectEqualStrings("home.arpa", hidden.forwardZone());
}
test "hide_domains writes no marker into a field the query never had" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var buf: [4]Entry = undefined;
var logger: Logger = .init(.{ .hide_domains = true }, &buf);
// An ordinary allowed query: no rule matched, no CNAME was uncloaked, no
// safe-search rewrite happened. Marking those "hidden" would claim the
// query had values it did not.
logger.log(io, sampleEntry(1, "plain.example"));
const hidden = try logger.queue.getOne(io);
try testing.expectEqualStrings(hidden_marker, hidden.domain());
try testing.expectEqualStrings("", hidden.matched());
try testing.expectEqualStrings("", hidden.cnameTarget());
try testing.expectEqualStrings("", hidden.safeSearchTarget());
}
test "the entry queue's worst case stays inside its byte budget" {
// The bound `config/validate.zig` enforces is derived from this, so the
// budget is what a maximal configuration can actually cost.
try testing.expect(@as(usize, query_log_buffer_max) * @sizeOf(Entry) <= queue_budget_bytes);
// One more entry than the ceiling would exceed it, so the ceiling is the
// largest value that fits rather than a round number under it.
try testing.expect((@as(usize, query_log_buffer_max) + 1) * @sizeOf(Entry) > queue_budget_bytes);
// The shipped default has to be comfortably inside the budget, or the
// out-of-the-box configuration is the one that spends it. At the widths
// above it costs about 17 MiB, roughly a quarter of the ceiling.
const default_max: usize = (model.Logging{}).query_log_buffer_max;
try testing.expect(default_max <= query_log_buffer_max);
try testing.expect(default_max * @sizeOf(Entry) <= queue_budget_bytes / 2);
}
test "log hides only the field its switch names" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
+13 -1
View File
@@ -154,11 +154,23 @@ fn writeRows(database: *db.Db, timestamps: []const i64, domain: []const u8) !voi
.domain = domain,
.client_ip = "192.0.2.10",
.qtype = 1,
.qclass = 1,
.rcode = 0,
.blocked = false,
.block_reason = null,
.response_time_us = null,
.cache_hit = null,
.upstream = null,
.group_id = 1,
.group_name = "default",
.policy_action = .allow,
.policy_reason = .no_match,
.matched = null,
.source_id = null,
.source_name = null,
.cname_target = null,
.safe_search_target = null,
.route_kind = .upstream,
.forward_zone = null,
};
}
try writer.writeBatch(rows[0..timestamps.len]);
+137
View File
@@ -0,0 +1,137 @@
//! The closed enums a `query_log` row stores to explain one query: what the
//! policy decided, why, and where the answer came from.
//!
//! They live in a module of their own because everything that touches a logged
//! row needs them — `storage/logger.zig`, `storage/repositories/queries_repo.zig`,
//! `server/handler.zig` and the web serializers — and `logger` already imports
//! `queries_repo`, so enums owned by either would close a loop.
//!
//! Each value is stored as its `@tagName` and read back through `parse`. The
//! read path treats an unrecognised value as a data error rather than passing
//! the text through: the column is a closed set, and a row that disagrees came
//! from something other than this schema.
const std = @import("std");
const matcher = @import("../filter/matcher.zig");
/// Whether the filtering policy reached a verdict on this query, and which one.
///
/// `not_evaluated` is the honest answer for a query the pipeline answered before
/// filtering could apply — a non-IN question, a paused resolver, a protocol
/// refusal — and is not the same as "allowed".
pub const PolicyAction = enum {
not_evaluated,
allow,
block,
};
/// Why the policy landed where it did.
///
/// The first nine are `filter/matcher.zig`'s serializable reasons, one for one.
/// The rest name the pipeline steps that decide a query without consulting the
/// matcher at all.
pub const PolicyReason = enum {
rule_allow_exact,
rule_block_exact,
rule_allow_wildcard,
rule_block_wildcard,
rule_allow_regex,
rule_block_regex,
blocklist_exception,
blocklist_domain,
blocklist_wildcard,
/// Answered from `local_records`, before filtering.
local_record,
/// Answered by a configured forward zone, before filtering.
forward_zone,
/// The question was not class IN, so no rule could apply to it.
non_in_class,
/// Filtering was paused.
paused,
/// No filter snapshot was published yet, so the query went unfiltered.
snapshot_unavailable,
/// The matcher evaluated the name and nothing matched.
no_match,
/// A syntactically parsed request refused on protocol grounds — BADVERS,
/// NOTIMP, a malformed EDNS OPT. It names a question, so it is logged, but
/// no policy ever saw it.
protocol_error,
};
/// Where the answer the client received came from.
pub const RouteKind = enum {
blocked,
local,
forward_zone,
upstream,
cache,
rejected,
};
/// The matcher's verdict in the query log's vocabulary.
///
/// Exhaustive on purpose: a reason added to the matcher must be given a stored
/// name here rather than silently reaching a row as something else. `.none` is
/// the matcher's "nothing matched", which is exactly `no_match`.
pub fn fromMatcherReason(reason: matcher.Reason) PolicyReason {
return switch (reason) {
.none => .no_match,
.rule_allow_exact => .rule_allow_exact,
.rule_block_exact => .rule_block_exact,
.rule_allow_wildcard => .rule_allow_wildcard,
.rule_block_wildcard => .rule_block_wildcard,
.rule_allow_regex => .rule_allow_regex,
.rule_block_regex => .rule_block_regex,
.blocklist_exception => .blocklist_exception,
.blocklist_domain => .blocklist_domain,
.blocklist_wildcard => .blocklist_wildcard,
};
}
/// Reads a stored `@tagName` back. `error.Mismatch` is the same error the
/// repositories return for a column that does not hold what the schema says it
/// holds, which is what an unknown value here is.
pub fn parse(comptime Enum: type, text: []const u8) error{Mismatch}!Enum {
return std.meta.stringToEnum(Enum, text) orelse error.Mismatch;
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const testing = std.testing;
test "every matcher reason has a stored name" {
// The mapping is checked here rather than trusted: a reason added to the
// matcher fails the exhaustive switch at compile time, and a reason
// *renamed* would still compile while changing what a row says.
inline for (@typeInfo(matcher.Reason).@"enum".fields) |field| {
const reason: matcher.Reason = @enumFromInt(field.value);
const mapped = fromMatcherReason(reason);
if (reason == .none) {
try testing.expectEqual(PolicyReason.no_match, mapped);
} else {
try testing.expectEqualStrings(field.name, @tagName(mapped));
}
}
}
test "parse round-trips every value of every enum" {
inline for ([_]type{ PolicyAction, PolicyReason, RouteKind }) |Enum| {
inline for (@typeInfo(Enum).@"enum".fields) |field| {
const value: Enum = @enumFromInt(field.value);
try testing.expectEqual(value, try parse(Enum, @tagName(value)));
}
}
}
test "parse rejects a value the schema does not define" {
try testing.expectError(error.Mismatch, parse(PolicyAction, "allowed"));
try testing.expectError(error.Mismatch, parse(PolicyAction, ""));
// A value that belongs to a different one of the three enums is no more
// acceptable than a typo.
try testing.expectError(error.Mismatch, parse(RouteKind, "allow"));
try testing.expectError(error.Mismatch, parse(PolicyReason, "cache"));
}
+153 -5
View File
@@ -22,8 +22,20 @@ const db = @import("db.zig");
const log = std.log.scoped(.querylog_schema);
/// Verbatim from PLAN §11.3. Multi-statement text — it goes through
/// `db.Db.exec`, never through `prepare`.
/// PLAN §11.3, plus the coverage watermark of milestone 28. Multi-statement
/// text — it goes through `db.Db.exec`, never through `prepare`.
///
/// The trailing INSERT seeds `querylog_meta`, which is part of the schema
/// rather than a later step: a `query_log` with no watermark beside it cannot
/// answer whether an empty result means "no queries" or "no history", and every
/// database this program reads from is created by executing this string.
/// `unixepoch()` is SQLite's own UTC clock, which is the clock every
/// `timestamp` in the file is measured against.
///
/// `available_since` starts one second *after* `created_at` on purpose. A row
/// logged in the same second the file was created is not evidence that the
/// second is completely covered, and the watermark's whole job is to be
/// conservative. From there it only ever advances, in `queries_repo.pruneOlderThan`.
pub const ddl: [:0]const u8 =
\\CREATE TABLE domains (
\\ id INTEGER PRIMARY KEY,
@@ -37,10 +49,23 @@ pub const ddl: [:0]const u8 =
\\ client_ip TEXT NOT NULL, -- text, not a FK: log rows are immutable facts
\\ qtype INTEGER,
\\ blocked INTEGER NOT NULL,
\\ block_reason TEXT,
\\ response_time_us INTEGER,
\\ cache_hit INTEGER,
\\ upstream TEXT
\\ upstream TEXT,
\\ qclass INTEGER NOT NULL,
\\ rcode INTEGER NOT NULL,
\\ group_id INTEGER, -- text/id pairs, not FKs: a renamed
\\ group_name TEXT, -- group must not rewrite history
\\ policy_action TEXT NOT NULL,
\\ policy_reason TEXT NOT NULL,
\\ matched TEXT,
\\ source_id INTEGER,
\\ source_name TEXT,
\\ cname_target TEXT,
\\ safe_search_target TEXT,
\\ route_kind TEXT NOT NULL,
\\ forward_zone TEXT,
\\ CHECK (rcode BETWEEN 0 AND 4095) -- twelve bits (RFC 6891 6.1.3)
\\);
\\CREATE INDEX idx_query_log_ts ON query_log(timestamp);
\\CREATE INDEX idx_query_log_client ON query_log(client_ip);
@@ -63,6 +88,14 @@ pub const ddl: [:0]const u8 =
\\ CHECK (failures >= 0)
\\) WITHOUT ROWID;
\\CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
\\
\\CREATE TABLE querylog_meta (
\\ id INTEGER PRIMARY KEY CHECK (id = 1), -- one row, enforced by the schema
\\ created_at INTEGER NOT NULL,
\\ available_since INTEGER NOT NULL
\\);
\\INSERT INTO querylog_meta (id, created_at, available_since)
\\VALUES (1, unixepoch(), unixepoch() + 1);
;
/// `PRAGMA user_version` is a signed 32-bit field. Deriving the fingerprint from
@@ -299,7 +332,7 @@ test "ddl creates the query-log tables, the upstream-history tables and every in
try database.exec(ddl);
try testing.expectEqual(
@as(i64, 4),
@as(i64, 5),
try database.queryInt("SELECT count(*) FROM sqlite_schema WHERE type='table'"),
);
const objects = [_][]const u8{
@@ -307,6 +340,7 @@ test "ddl creates the query-log tables, the upstream-history tables and every in
"idx_query_log_ts", "idx_query_log_client",
"idx_query_log_domain", "upstream_targets",
"upstream_minute", "idx_upstream_minute_ts",
"querylog_meta",
};
for (objects) |name| {
var stmt = try database.prepare("SELECT count(*) FROM sqlite_schema WHERE name = ?1");
@@ -317,6 +351,69 @@ test "ddl creates the query-log tables, the upstream-history tables and every in
}
}
test "the schema refuses an rcode outside twelve bits" {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try db.applyPragmas(&database, .{});
try database.exec(ddl);
try database.exec("INSERT INTO domains (id, domain) VALUES (1, 'a.example');");
var stmt = try database.prepare(
\\INSERT INTO query_log
\\ (timestamp, domain_id, client_ip, blocked, qclass, rcode,
\\ policy_action, policy_reason, route_kind)
\\VALUES (1, 1, '10.0.0.1', 0, 1, ?1, 'not_evaluated', 'no_match', 'upstream')
);
defer stmt.deinit();
// The whole range an EDNS extended RCODE can express, and nothing wider:
// the producers are `u12`, and this is what stops any other writer — a
// hand-run UPDATE included — from putting a value in the column that the
// read path would have to reject.
for ([_]i64{ 0, 4095 }) |accepted| {
try stmt.reset();
try stmt.bindInt(1, accepted);
try stmt.exec();
}
for ([_]i64{ -1, 4096, 65535 }) |refused| {
// `sqlite3_reset` repeats the error of the statement it is resetting,
// which for every iteration after the first is the constraint failure
// this loop just asserted — the same reason `BatchWriter.resetAll`
// discards it.
stmt.reset() catch {};
try stmt.bindInt(1, refused);
try testing.expectError(error.Constraint, stmt.exec());
}
try testing.expectEqual(@as(i64, 2), try database.queryInt("SELECT count(*) FROM query_log"));
}
test "querylog_meta is seeded with one row the schema will not let a second join" {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try db.applyPragmas(&database, .{});
try database.exec(ddl);
try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM querylog_meta"));
const created = try database.queryInt("SELECT created_at FROM querylog_meta");
const since = try database.queryInt("SELECT available_since FROM querylog_meta");
// Conservative by exactly one second: a row logged in the creating second
// must not let a query claim that second is completely covered.
try testing.expectEqual(created + 1, since);
try testing.expect(created > 1_700_000_000);
// `CHECK (id = 1)` is what makes "the singleton row" a schema fact rather
// than a convention the read path has to defend against.
try testing.expectError(error.Constraint, database.exec(
"INSERT INTO querylog_meta (id, created_at, available_since) VALUES (2, 1, 1);",
));
try testing.expectError(error.Constraint, database.exec(
"INSERT INTO querylog_meta (id, created_at, available_since) VALUES (1, 1, 1);",
));
try testing.expectEqual(@as(i64, 1), try database.queryInt("SELECT count(*) FROM querylog_meta"));
}
test "the user_version statement stamps the fingerprint" {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
@@ -394,6 +491,57 @@ test "a recreate returns the aside name by value and a fresh create returns none
try tmp.dir.access(io, kept, .{});
}
test "a recreate resets coverage to the new file and keeps the old one aside" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
var path_buf: [path_buf_len]u8 = undefined;
const path = try std.fmt.bufPrintZ(&path_buf, ".zig-cache/tmp/{s}/querylog.db", .{tmp.sub_path});
var created = try open(io, std.Io.Dir.cwd(), path);
const first_coverage = try created.database.queryInt("SELECT available_since FROM querylog_meta");
// A row in the file the operator is about to lose.
try created.database.exec("INSERT INTO domains (domain) VALUES ('old.example');");
created.database.close();
// A healthy file this build's DDL no longer matches — the case milestone
// 28's own schema edit produces on every upgrade.
{
var stamped = try db.Db.open(path, .{ .mode = .read_write_existing });
defer stamped.close();
var sql_buf: [64]u8 = undefined;
try stamped.exec(try std.fmt.bufPrintZ(&sql_buf, "PRAGMA user_version = {d};", .{fingerprint +% 1}));
}
var recreated = try open(io, std.Io.Dir.cwd(), path);
defer recreated.database.close();
try testing.expectEqual(RecreateReason.fingerprint_mismatch, recreated.recreated.?);
// The name says the file was healthy and this build moved, not that it rotted.
try testing.expect(std.mem.indexOf(u8, recreated.aside(), ".schema-changed-") != null);
try tmp.dir.access(io, std.fs.path.basename(recreated.aside()), .{});
// Exactly one meta row, and coverage starts at the recreate rather than
// carrying the replaced file's promise forward.
try testing.expectEqual(
@as(i64, 1),
try recreated.database.queryInt("SELECT count(*) FROM querylog_meta"),
);
const new_coverage = try recreated.database.queryInt("SELECT available_since FROM querylog_meta");
try testing.expect(new_coverage >= first_coverage);
// Nothing of the old file came across: the history is genuinely gone, which
// is what the coverage start has to tell the operator.
try testing.expectEqual(
@as(i64, 0),
try recreated.database.queryInt("SELECT count(*) FROM domains"),
);
}
test "a clean reopen reports no recreate and no aside" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
+528 -72
View File
@@ -19,20 +19,42 @@ const std = @import("std");
const Allocator = std.mem.Allocator;
const db = @import("../db.zig");
const provenance = @import("../provenance.zig");
/// One `query_log` row. The logger applies the privacy transforms of PLAN
/// §11.4 before it builds this, so `domain` and `client_ip` are already
/// §11.4 before it builds this, so every domain-bearing field is already
/// whatever the operator agreed to store.
///
/// A `null` text field is a fact the query did not have — no upstream was
/// attempted, no rule matched, no CNAME was uncloaked — and reaches the column
/// as NULL. The three closed enums have no such state: every logged query has a
/// policy verdict, a reason for it and a route, even when the verdict is
/// "not evaluated".
pub const Row = struct {
timestamp: i64,
domain: []const u8,
client_ip: []const u8,
qtype: ?u16,
qclass: u16,
/// Twelve bits: the EDNS extended RCODE the client saw. The column's
/// `CHECK` bounds it to the same range, so a value this type cannot hold
/// is one the schema would have refused anyway.
rcode: u12,
blocked: bool,
block_reason: ?[]const u8,
response_time_us: ?i64,
cache_hit: ?bool,
upstream: ?[]const u8,
group_id: ?i64,
group_name: ?[]const u8,
policy_action: provenance.PolicyAction,
policy_reason: provenance.PolicyReason,
matched: ?[]const u8,
source_id: ?i64,
source_name: ?[]const u8,
cname_target: ?[]const u8,
safe_search_target: ?[]const u8,
route_kind: provenance.RouteKind,
forward_zone: ?[]const u8,
};
const insert_domain_sql = "INSERT OR IGNORE INTO domains (domain) VALUES (?1)";
@@ -41,9 +63,13 @@ const select_domain_sql = "SELECT id FROM domains WHERE domain = ?1";
const insert_row_sql =
\\INSERT INTO query_log
\\ (timestamp, domain_id, client_ip, qtype, blocked, block_reason,
\\ response_time_us, cache_hit, upstream)
\\VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9)
\\ (timestamp, domain_id, client_ip, qtype, blocked,
\\ response_time_us, cache_hit, upstream, qclass, rcode,
\\ group_id, group_name, policy_action, policy_reason, matched,
\\ source_id, source_name, cname_target, safe_search_target,
\\ route_kind, forward_zone)
\\VALUES (?1, ?2, ?3, ?4, ?5, ?6, ?7, ?8, ?9, ?10,
\\ ?11, ?12, ?13, ?14, ?15, ?16, ?17, ?18, ?19, ?20, ?21)
;
/// Owns the prepared statements of the flush loop. Init once, reuse per batch.
@@ -123,10 +149,22 @@ pub const BatchWriter = struct {
try stmt.bindText(3, row.client_ip);
try bindIntOrNull(stmt, 4, if (row.qtype) |v| @as(i64, v) else null);
try stmt.bindBool(5, row.blocked);
try stmt.bindTextOrNull(6, row.block_reason);
try bindIntOrNull(stmt, 7, row.response_time_us);
try bindIntOrNull(stmt, 8, if (row.cache_hit) |v| @as(i64, @intFromBool(v)) else null);
try stmt.bindTextOrNull(9, row.upstream);
try bindIntOrNull(stmt, 6, row.response_time_us);
try bindIntOrNull(stmt, 7, if (row.cache_hit) |v| @as(i64, @intFromBool(v)) else null);
try stmt.bindTextOrNull(8, row.upstream);
try stmt.bindInt(9, row.qclass);
try stmt.bindInt(10, row.rcode);
try bindIntOrNull(stmt, 11, row.group_id);
try stmt.bindTextOrNull(12, row.group_name);
try stmt.bindText(13, @tagName(row.policy_action));
try stmt.bindText(14, @tagName(row.policy_reason));
try stmt.bindTextOrNull(15, row.matched);
try bindIntOrNull(stmt, 16, row.source_id);
try stmt.bindTextOrNull(17, row.source_name);
try stmt.bindTextOrNull(18, row.cname_target);
try stmt.bindTextOrNull(19, row.safe_search_target);
try stmt.bindText(20, @tagName(row.route_kind));
try stmt.bindTextOrNull(21, row.forward_zone);
try stmt.exec();
}
@@ -144,17 +182,55 @@ fn bindIntOrNull(stmt: *db.Stmt, idx: c_int, value: ?i64) db.Error!void {
return stmt.bindNull(idx);
}
/// Deletes every `query_log` row strictly older than `cutoff_ts` and returns
/// how many went.
/// What one prune did, and where coverage now begins.
pub const PruneResult = struct {
deleted: i64,
/// The watermark after the prune, which is what a later `availableSince`
/// will return. Handed back so the caller need not re-read it.
available_since: i64,
};
/// Deletes every `query_log` row strictly older than `cutoff_ts` and advances
/// the coverage watermark to the same cutoff, in one transaction.
///
/// Orphaned `domains` rows stay: it is a dimension table, re-interning a name
/// costs one indexed insert, and §11.3 asks for no collection.
pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!i64 {
var stmt = try database.prepare("DELETE FROM query_log WHERE timestamp < ?1");
defer stmt.deinit();
try stmt.bindInt(1, cutoff_ts);
try stmt.exec();
return database.changes();
/// **The two are one operation, not two.** The watermark is the promise that
/// every query since it is still in the file; a delete that commits without the
/// advance breaks that promise, and an advance that commits without the delete
/// hides rows the file still holds. Either failure rolls both back, and the
/// caller retries the whole thing on its next pass.
///
/// The watermark never moves backward: `max` is what makes a prune with a
/// cutoff older than the file's own creation a no-op on it rather than a
/// regression. Orphaned `domains` rows stay — it is a dimension table,
/// re-interning a name costs one indexed insert, and §11.3 asks for no
/// collection.
pub fn pruneOlderThan(database: *db.Db, cutoff_ts: i64) db.Error!PruneResult {
var tx = try db.Tx.begin(database);
errdefer tx.rollback();
var deleting = try database.prepare("DELETE FROM query_log WHERE timestamp < ?1");
defer deleting.deinit();
try deleting.bindInt(1, cutoff_ts);
try deleting.exec();
const deleted = database.changes();
var advancing = try database.prepare(
"UPDATE querylog_meta SET available_since = max(available_since, ?1) WHERE id = 1",
);
defer advancing.deinit();
try advancing.bindInt(1, cutoff_ts);
try advancing.exec();
const watermark = try database.queryInt("SELECT available_since FROM querylog_meta WHERE id = 1");
try tx.commit();
return .{ .deleted = deleted, .available_since = watermark };
}
/// The oldest timestamp this file can still answer for. A query window that
/// starts before it is incomplete, and the API says so rather than charting the
/// gap as zero.
pub fn availableSince(database: *db.Db) db.Error!i64 {
return database.queryInt("SELECT available_since FROM querylog_meta WHERE id = 1");
}
/// `PRAGMA wal_checkpoint(TRUNCATE)`: moves the WAL into the database and
@@ -189,21 +265,60 @@ pub fn countDomains(database: *db.Db) db.Error!i64 {
/// One row of `GET /api/queries`, joined back through the `domains` dimension.
///
/// `block_reason` and `upstream` are nullable columns, and a NULL reads as `""`
/// the same convention `Stmt.columnText` already uses. Neither column is ever
/// written as an empty string (a reason is a word, an upstream is a URL), so the
/// mapping loses nothing and the API layer can treat `""` as "absent".
/// A summary projection, deliberately narrower than `QueryDetail`: the list is
/// a table the operator scans, and the full provenance of a row is one request
/// away at `GET /api/queries/{id}`.
///
/// The nullable text columns read a NULL as `""` — the same convention
/// `Stmt.columnText` already uses. None of them is ever written as an empty
/// string, so the mapping loses nothing and the API layer can treat `""` as
/// "absent".
pub const QueryRow = struct {
id: i64,
ts: i64,
domain: []const u8,
client_ip: []const u8,
qtype: ?u16,
qclass: u16,
rcode: u12,
blocked: bool,
block_reason: []const u8,
response_time_us: ?i64,
cache_hit: ?bool,
upstream: []const u8,
policy_action: provenance.PolicyAction,
policy_reason: provenance.PolicyReason,
route_kind: provenance.RouteKind,
};
/// Everything one `query_log` row records about one query, for
/// `GET /api/queries/{id}`.
///
/// Same NULL-reads-as-`""` convention as `QueryRow`, and the same closed enums:
/// a stored value the schema does not define is `error.Mismatch`, never passed
/// through as text.
pub const QueryDetail = struct {
id: i64,
ts: i64,
domain: []const u8,
client_ip: []const u8,
qtype: ?u16,
qclass: u16,
rcode: u12,
blocked: bool,
response_time_us: ?i64,
cache_hit: ?bool,
upstream: []const u8,
group_id: ?i64,
group_name: []const u8,
policy_action: provenance.PolicyAction,
policy_reason: provenance.PolicyReason,
matched: []const u8,
source_id: ?i64,
source_name: []const u8,
cname_target: []const u8,
safe_search_target: []const u8,
route_kind: provenance.RouteKind,
forward_zone: []const u8,
};
/// Every field is an independent narrowing; `null` means "do not filter on it".
@@ -230,7 +345,8 @@ pub const max_limit: u32 = 1000;
const select_head =
\\SELECT q.id, q.timestamp, d.domain, q.client_ip, q.qtype, q.blocked,
\\ q.block_reason, q.response_time_us, q.cache_hit, q.upstream
\\ q.response_time_us, q.cache_hit, q.upstream, q.qclass, q.rcode,
\\ q.policy_action, q.policy_reason, q.route_kind
\\ FROM query_log q JOIN domains d ON d.id = q.domain_id
;
@@ -337,15 +453,81 @@ pub fn selectQueries(database: *db.Db, arena: Allocator, filter: QueryFilter) db
.qtype = if (stmt.isNull(4)) null else std.math.cast(u16, stmt.columnInt(4)) orelse
return error.Mismatch,
.blocked = stmt.columnBool(5),
.block_reason = try stmt.columnTextAlloc(arena, 6),
.response_time_us = if (stmt.isNull(7)) null else stmt.columnInt(7),
.cache_hit = if (stmt.isNull(8)) null else stmt.columnBool(8),
.upstream = try stmt.columnTextAlloc(arena, 9),
.response_time_us = if (stmt.isNull(6)) null else stmt.columnInt(6),
.cache_hit = if (stmt.isNull(7)) null else stmt.columnBool(7),
.upstream = try stmt.columnTextAlloc(arena, 8),
.qclass = try columnU16(&stmt, 9),
.rcode = try columnU12(&stmt, 10),
.policy_action = try provenance.parse(provenance.PolicyAction, stmt.columnText(11)),
.policy_reason = try provenance.parse(provenance.PolicyReason, stmt.columnText(12)),
.route_kind = try provenance.parse(provenance.RouteKind, stmt.columnText(13)),
});
}
return out;
}
/// A `NOT NULL` integer column that the schema bounds to 16 bits. A value
/// outside that range means the row came from something other than this schema.
fn columnU16(stmt: *db.Stmt, col: c_int) db.Error!u16 {
return std.math.cast(u16, stmt.columnInt(col)) orelse error.Mismatch;
}
/// The `rcode` column, which the schema's `CHECK` bounds to twelve bits. This
/// build cannot write a wider value — the field is a `u12` all the way from the
/// handler — so a row that carries one was written by something else, and is
/// `error.Mismatch` rather than a value truncated into shape.
fn columnU12(stmt: *db.Stmt, col: c_int) db.Error!u12 {
return std.math.cast(u12, stmt.columnInt(col)) orelse error.Mismatch;
}
const select_detail_sql =
\\SELECT q.id, q.timestamp, d.domain, q.client_ip, q.qtype, q.blocked,
\\ q.response_time_us, q.cache_hit, q.upstream, q.qclass, q.rcode,
\\ q.group_id, q.group_name, q.policy_action, q.policy_reason,
\\ q.matched, q.source_id, q.source_name, q.cname_target,
\\ q.safe_search_target, q.route_kind, q.forward_zone
\\ FROM query_log q JOIN domains d ON d.id = q.domain_id
\\ WHERE q.id = ?1
;
/// One row's full provenance, or `null` when no row has that id — which is what
/// an id the operator kept from before a retention pass looks like, and is a
/// 404 rather than an error.
///
/// Every string is allocated from `arena`, on the same terms as
/// `selectQueries`.
pub fn detailById(database: *db.Db, arena: Allocator, id: i64) db.Error!?QueryDetail {
var stmt = try database.prepare(select_detail_sql);
defer stmt.deinit();
try stmt.bindInt(1, id);
if (!try stmt.step()) return null;
return .{
.id = stmt.columnInt(0),
.ts = stmt.columnInt(1),
.domain = try stmt.columnTextAlloc(arena, 2),
.client_ip = try stmt.columnTextAlloc(arena, 3),
.qtype = if (stmt.isNull(4)) null else try columnU16(&stmt, 4),
.blocked = stmt.columnBool(5),
.response_time_us = if (stmt.isNull(6)) null else stmt.columnInt(6),
.cache_hit = if (stmt.isNull(7)) null else stmt.columnBool(7),
.upstream = try stmt.columnTextAlloc(arena, 8),
.qclass = try columnU16(&stmt, 9),
.rcode = try columnU12(&stmt, 10),
.group_id = if (stmt.isNull(11)) null else stmt.columnInt(11),
.group_name = try stmt.columnTextAlloc(arena, 12),
.policy_action = try provenance.parse(provenance.PolicyAction, stmt.columnText(13)),
.policy_reason = try provenance.parse(provenance.PolicyReason, stmt.columnText(14)),
.matched = try stmt.columnTextAlloc(arena, 15),
.source_id = if (stmt.isNull(16)) null else stmt.columnInt(16),
.source_name = try stmt.columnTextAlloc(arena, 17),
.cname_target = try stmt.columnTextAlloc(arena, 18),
.safe_search_target = try stmt.columnTextAlloc(arena, 19),
.route_kind = try provenance.parse(provenance.RouteKind, stmt.columnText(20)),
.forward_zone = try stmt.columnTextAlloc(arena, 21),
};
}
/// Wraps `needle` in `%` and neutralises the two `LIKE` metacharacters, so a
/// user searching for `a_b` gets domains containing `a_b` and not domains
/// containing `axb`. The escape character escapes itself.
@@ -493,11 +675,23 @@ fn plainRow(timestamp: i64, domain: []const u8) Row {
.domain = domain,
.client_ip = "192.0.2.10",
.qtype = 1,
.qclass = 1,
.rcode = 0,
.blocked = false,
.block_reason = null,
.response_time_us = 1200,
.cache_hit = false,
.upstream = "9.9.9.9",
.group_id = 1,
.group_name = "default",
.policy_action = .allow,
.policy_reason = .no_match,
.matched = null,
.source_id = null,
.source_name = null,
.cname_target = null,
.safe_search_target = null,
.route_kind = .upstream,
.forward_zone = null,
};
}
@@ -509,6 +703,43 @@ fn domainIdOf(database: *db.Db, domain: []const u8) !i64 {
return stmt.columnInt(0);
}
test "a foreign row with an rcode wider than twelve bits is refused, not truncated" {
var database = try db.Db.open(":memory:", .{ .mode = .memory });
defer database.close();
try db.applyPragmas(&database, .{});
// The shipped schema's `CHECK` makes this row impossible in a file this
// build created, so the table is built without it. The read path's job is
// to refuse a `querylog.db` that came from somewhere else rather than to
// narrow a value it cannot represent.
try database.exec(
\\CREATE TABLE domains (id INTEGER PRIMARY KEY, domain TEXT NOT NULL UNIQUE);
\\CREATE TABLE query_log (
\\ id INTEGER PRIMARY KEY, timestamp INTEGER NOT NULL,
\\ domain_id INTEGER NOT NULL, client_ip TEXT NOT NULL,
\\ qtype INTEGER, blocked INTEGER NOT NULL, response_time_us INTEGER,
\\ cache_hit INTEGER, upstream TEXT, qclass INTEGER NOT NULL,
\\ rcode INTEGER NOT NULL, group_id INTEGER, group_name TEXT,
\\ policy_action TEXT NOT NULL, policy_reason TEXT NOT NULL,
\\ matched TEXT, source_id INTEGER, source_name TEXT,
\\ cname_target TEXT, safe_search_target TEXT,
\\ route_kind TEXT NOT NULL, forward_zone TEXT
\\);
\\INSERT INTO domains (id, domain) VALUES (1, 'a.example');
\\INSERT INTO query_log
\\ (id, timestamp, domain_id, client_ip, blocked, qclass, rcode,
\\ policy_action, policy_reason, route_kind)
\\VALUES (1, 10, 1, '192.0.2.10', 0, 1, 4096, 'allow', 'no_match', 'upstream');
);
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
try testing.expectError(error.Mismatch, selectQueries(&database, arena, .{}));
try testing.expectError(error.Mismatch, detailById(&database, arena, 1));
}
test "writeBatch inserts every row and interns each domain once" {
var database = try openLog();
defer database.close();
@@ -553,7 +784,7 @@ test "a second batch reuses the interned domain id" {
);
}
test "nullable columns round-trip a value and a null" {
test "every column round-trips a value and a null" {
var database = try openLog();
defer database.close();
var writer = try BatchWriter.init(&database);
@@ -565,54 +796,125 @@ test "nullable columns round-trip a value and a null" {
.domain = "blocked.example",
.client_ip = "2001:db8::1",
.qtype = 28,
.qclass = 1,
.rcode = 3,
.blocked = true,
.block_reason = "blocklist",
.response_time_us = 42,
.cache_hit = true,
.upstream = "dns.example",
.upstream = "https://dns.example/dns-query",
.group_id = 7,
.group_name = "kids",
.policy_action = .block,
.policy_reason = .blocklist_wildcard,
.matched = "*.ads.example",
.source_id = 3,
.source_name = "steven black",
.cname_target = "tracker.cdn.example",
.safe_search_target = "forcesafesearch.google.com",
.route_kind = .blocked,
.forward_zone = "home.arpa",
},
// Every nullable column absent at once, which is the shape of a query
// the pipeline answered before any of them applied.
.{
.timestamp = 11,
.domain = "quiet.example",
.client_ip = "hidden",
.qtype = null,
.qclass = 3,
.rcode = 0,
.blocked = false,
.block_reason = null,
.response_time_us = null,
.cache_hit = null,
.upstream = null,
.group_id = null,
.group_name = null,
.policy_action = .not_evaluated,
.policy_reason = .non_in_class,
.matched = null,
.source_id = null,
.source_name = null,
.cname_target = null,
.safe_search_target = null,
.route_kind = .upstream,
.forward_zone = null,
},
});
var stmt = try database.prepare(
\\SELECT d.domain, q.client_ip, q.qtype, q.blocked, q.block_reason,
\\ q.response_time_us, q.cache_hit, q.upstream
\\ FROM query_log q JOIN domains d ON d.id = q.domain_id
\\ ORDER BY q.timestamp
);
defer stmt.deinit();
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
try testing.expect(try stmt.step());
try testing.expectEqualStrings("blocked.example", stmt.columnText(0));
try testing.expectEqualStrings("2001:db8::1", stmt.columnText(1));
try testing.expectEqual(@as(i64, 28), stmt.columnInt(2));
try testing.expect(stmt.columnBool(3));
try testing.expectEqualStrings("blocklist", stmt.columnText(4));
try testing.expectEqual(@as(i64, 42), stmt.columnInt(5));
try testing.expect(stmt.columnBool(6));
try testing.expectEqualStrings("dns.example", stmt.columnText(7));
const full = (try detailById(&database, arena, 1)).?;
try testing.expectEqualStrings("blocked.example", full.domain);
try testing.expectEqualStrings("2001:db8::1", full.client_ip);
try testing.expectEqual(@as(?u16, 28), full.qtype);
try testing.expectEqual(@as(u16, 1), full.qclass);
try testing.expectEqual(@as(u12, 3), full.rcode);
try testing.expect(full.blocked);
try testing.expectEqual(@as(?i64, 42), full.response_time_us);
try testing.expectEqual(@as(?bool, true), full.cache_hit);
try testing.expectEqualStrings("https://dns.example/dns-query", full.upstream);
try testing.expectEqual(@as(?i64, 7), full.group_id);
try testing.expectEqualStrings("kids", full.group_name);
try testing.expectEqual(provenance.PolicyAction.block, full.policy_action);
try testing.expectEqual(provenance.PolicyReason.blocklist_wildcard, full.policy_reason);
try testing.expectEqualStrings("*.ads.example", full.matched);
try testing.expectEqual(@as(?i64, 3), full.source_id);
try testing.expectEqualStrings("steven black", full.source_name);
try testing.expectEqualStrings("tracker.cdn.example", full.cname_target);
try testing.expectEqualStrings("forcesafesearch.google.com", full.safe_search_target);
try testing.expectEqual(provenance.RouteKind.blocked, full.route_kind);
try testing.expectEqualStrings("home.arpa", full.forward_zone);
try testing.expect(try stmt.step());
try testing.expectEqualStrings("quiet.example", stmt.columnText(0));
try testing.expectEqualStrings("hidden", stmt.columnText(1));
try testing.expect(stmt.isNull(2));
try testing.expect(!stmt.columnBool(3));
try testing.expect(stmt.isNull(4));
try testing.expect(stmt.isNull(5));
try testing.expect(stmt.isNull(6));
try testing.expect(stmt.isNull(7));
// A NULL text column reads as the empty string, by the documented
// convention; a NULL integer stays null, because 0 is a real id.
const bare = (try detailById(&database, arena, 2)).?;
try testing.expectEqual(@as(?u16, null), bare.qtype);
try testing.expectEqual(@as(u16, 3), bare.qclass);
try testing.expectEqual(@as(?i64, null), bare.response_time_us);
try testing.expectEqual(@as(?bool, null), bare.cache_hit);
try testing.expectEqualStrings("", bare.upstream);
try testing.expectEqual(@as(?i64, null), bare.group_id);
try testing.expectEqualStrings("", bare.group_name);
try testing.expectEqual(provenance.PolicyAction.not_evaluated, bare.policy_action);
try testing.expectEqual(provenance.PolicyReason.non_in_class, bare.policy_reason);
try testing.expectEqualStrings("", bare.matched);
try testing.expectEqual(@as(?i64, null), bare.source_id);
try testing.expectEqualStrings("", bare.source_name);
try testing.expectEqualStrings("", bare.cname_target);
try testing.expectEqualStrings("", bare.safe_search_target);
try testing.expectEqual(provenance.RouteKind.upstream, bare.route_kind);
try testing.expectEqualStrings("", bare.forward_zone);
}
try testing.expect(!try stmt.step());
test "detailById returns null for an id no row has" {
var database = try openLog();
defer database.close();
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
try seed(&database, &.{plainRow(10, "a.example")});
// An id from before a retention pass looks exactly like this, and is a 404
// rather than an error.
try testing.expectEqual(@as(?QueryDetail, null), try detailById(&database, arena_state.allocator(), 2));
try testing.expectEqual(@as(?QueryDetail, null), try detailById(&database, arena_state.allocator(), 0));
try testing.expect((try detailById(&database, arena_state.allocator(), 1)) != null);
}
test "a stored enum value the schema does not define is a data error, not a passthrough" {
var database = try openLog();
defer database.close();
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
try seed(&database, &.{plainRow(10, "a.example")});
try database.exec("UPDATE query_log SET policy_reason = 'whatever' WHERE id = 1;");
try testing.expectError(error.Mismatch, detailById(&database, arena, 1));
try testing.expectError(error.Mismatch, selectQueries(&database, arena, .{}));
}
test "an empty batch writes nothing and opens no transaction" {
@@ -644,7 +946,7 @@ test "pruneOlderThan deletes strictly older rows and returns the count" {
plainRow(300, "fresh.example"),
});
try testing.expectEqual(@as(i64, 2), try pruneOlderThan(&database, 200));
try testing.expectEqual(@as(i64, 2), (try pruneOlderThan(&database, 200)).deleted);
try testing.expectEqual(@as(i64, 2), try countRows(&database));
// The row exactly at the cutoff stays.
try testing.expectEqual(
@@ -652,7 +954,123 @@ test "pruneOlderThan deletes strictly older rows and returns the count" {
try database.queryInt("SELECT count(*) FROM query_log WHERE timestamp = 200"),
);
// A second pass over the same cutoff finds nothing left to do.
try testing.expectEqual(@as(i64, 0), try pruneOlderThan(&database, 200));
try testing.expectEqual(@as(i64, 0), (try pruneOlderThan(&database, 200)).deleted);
}
test "a prune advances the coverage watermark to its own cutoff" {
var database = try openLog();
defer database.close();
var writer = try BatchWriter.init(&database);
defer writer.deinit();
// The seeded watermark is `created_at + 1`, which is now-ish; the cutoffs
// below are all in the past, so they start out behind it.
const start = try availableSince(&database);
try writer.writeBatch(&.{ plainRow(start + 100, "a.example"), plainRow(start + 300, "b.example") });
const first = try pruneOlderThan(&database, start + 200);
try testing.expectEqual(@as(i64, 1), first.deleted);
try testing.expectEqual(start + 200, first.available_since);
try testing.expectEqual(start + 200, try availableSince(&database));
// A prune that deletes nothing still advances: the window it swept is
// covered whether or not it held rows.
const second = try pruneOlderThan(&database, start + 250);
try testing.expectEqual(@as(i64, 0), second.deleted);
try testing.expectEqual(start + 250, try availableSince(&database));
}
test "the watermark never moves backward" {
var database = try openLog();
defer database.close();
const start = try availableSince(&database);
const advanced = try pruneOlderThan(&database, start + 1000);
try testing.expectEqual(start + 1000, advanced.available_since);
// A shortened `retention_days`, a clock that stepped back, a pass with a
// stale cutoff: none of them may widen the promise the file makes.
for ([_]i64{ start + 999, start, start - 100_000, 0 }) |older| {
const result = try pruneOlderThan(&database, older);
try testing.expectEqual(start + 1000, result.available_since);
try testing.expectEqual(start + 1000, try availableSince(&database));
}
}
test "a failed delete leaves both the rows and the watermark untouched" {
var database = try openLog();
defer database.close();
var writer = try BatchWriter.init(&database);
defer writer.deinit();
const start = try availableSince(&database);
try writer.writeBatch(&.{plainRow(start - 100, "old.example")});
try database.exec(
\\CREATE TRIGGER refuse_delete BEFORE DELETE ON query_log
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
try testing.expectError(error.Constraint, pruneOlderThan(&database, start + 1000));
try testing.expectEqual(@as(i64, 1), try countRows(&database));
try testing.expectEqual(start, try availableSince(&database));
}
test "a failed watermark update leaves the rows it had already deleted" {
var database = try openLog();
defer database.close();
var writer = try BatchWriter.init(&database);
defer writer.deinit();
const start = try availableSince(&database);
try writer.writeBatch(&.{plainRow(start - 100, "old.example")});
// The delete succeeds and the advance does not. Without one transaction
// around the pair, this is the case that loses rows the watermark still
// promises.
try database.exec(
\\CREATE TRIGGER refuse_advance BEFORE UPDATE ON querylog_meta
\\BEGIN SELECT RAISE(ABORT, 'refused'); END;
);
try testing.expectError(error.Constraint, pruneOlderThan(&database, start + 1000));
try testing.expectEqual(@as(i64, 1), try countRows(&database));
try testing.expectEqual(start, try availableSince(&database));
}
test "a failed commit rolls back the delete and the watermark together" {
var database = try openLog();
defer database.close();
var writer = try BatchWriter.init(&database);
defer writer.deinit();
const start = try availableSince(&database);
try writer.writeBatch(&.{plainRow(start - 100, "old.example")});
// Both statements succeed and COMMIT is what fails: the advance inserts a
// `query_log` row whose `domain_id` references nothing, and
// `defer_foreign_keys` holds that violation back until the commit checks
// it (SQLite's documented semantics for the pragma; the assertions below
// observe the rollback, not the moment the check ran).
try database.exec(
\\CREATE TRIGGER break_at_commit AFTER UPDATE ON querylog_meta
\\BEGIN INSERT INTO query_log
\\ (timestamp, domain_id, client_ip, blocked, qclass, rcode,
\\ policy_action, policy_reason, route_kind)
\\VALUES (1, 999999, 'x', 0, 1, 0, 'allow', 'no_match', 'upstream'); END;
);
try database.exec("PRAGMA defer_foreign_keys = ON;");
try testing.expectError(error.Constraint, pruneOlderThan(&database, start + 1000));
// Nothing survived: not the delete, not the advance, not the row the
// trigger inserted.
try testing.expectEqual(@as(i64, 1), try countRows(&database));
try testing.expectEqual(start, try availableSince(&database));
try testing.expectEqual(
@as(i64, 0),
try database.queryInt("SELECT count(*) FROM query_log WHERE client_ip = 'x'"),
);
}
test "pruneOlderThan leaves the domains dimension table intact" {
@@ -662,7 +1080,7 @@ test "pruneOlderThan leaves the domains dimension table intact" {
defer writer.deinit();
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(11, "b.example") });
try testing.expectEqual(@as(i64, 2), try pruneOlderThan(&database, 1000));
try testing.expectEqual(@as(i64, 2), (try pruneOlderThan(&database, 1000)).deleted);
try testing.expectEqual(@as(i64, 0), try countRows(&database));
try testing.expectEqual(@as(i64, 2), try countDomains(&database));
@@ -749,7 +1167,7 @@ test "checkpointTruncate and vacuum run against a WAL file database" {
try writer.writeBatch(&.{ plainRow(10, "a.example"), plainRow(20, "b.example") });
try checkpointTruncate(&database);
try testing.expectEqual(@as(i64, 1), try pruneOlderThan(&database, 20));
try testing.expectEqual(@as(i64, 1), (try pruneOlderThan(&database, 20)).deleted);
try checkpointTruncate(&database);
try vacuum(&database);
@@ -785,22 +1203,46 @@ test "selectQueries returns the newest row first and reads every column" {
.domain = "ads.example.net",
.client_ip = "192.0.2.10",
.qtype = 28,
.qclass = 1,
.rcode = 0,
.blocked = true,
.block_reason = "blocklist",
.response_time_us = 4200,
.cache_hit = true,
.upstream = "https://dns.example/dns-query",
.group_id = 2,
.group_name = "kids",
.policy_action = .block,
.policy_reason = .blocklist_domain,
.matched = "ads.example.net",
.source_id = 5,
.source_name = "steven black",
.cname_target = null,
.safe_search_target = null,
.route_kind = .blocked,
.forward_zone = null,
},
.{
.timestamp = 20,
.domain = "quiet.example",
.client_ip = "hidden",
.qtype = null,
.qclass = 1,
.rcode = 2,
.blocked = false,
.block_reason = null,
.response_time_us = null,
.cache_hit = null,
.upstream = null,
.group_id = null,
.group_name = null,
.policy_action = .not_evaluated,
.policy_reason = .snapshot_unavailable,
.matched = null,
.source_id = null,
.source_name = null,
.cname_target = null,
.safe_search_target = null,
.route_kind = .rejected,
.forward_zone = null,
},
});
@@ -813,12 +1255,16 @@ test "selectQueries returns the newest row first and reads every column" {
try testing.expectEqualStrings("quiet.example", newest.domain);
try testing.expectEqualStrings("hidden", newest.client_ip);
try testing.expectEqual(@as(?u16, null), newest.qtype);
try testing.expectEqual(@as(u16, 1), newest.qclass);
try testing.expectEqual(@as(u12, 2), newest.rcode);
try testing.expect(!newest.blocked);
// A NULL text column reads as the empty string, by documented convention.
try testing.expectEqualStrings("", newest.block_reason);
try testing.expectEqual(@as(?i64, null), newest.response_time_us);
try testing.expectEqual(@as(?bool, null), newest.cache_hit);
// A NULL text column reads as the empty string, by documented convention.
try testing.expectEqualStrings("", newest.upstream);
try testing.expectEqual(provenance.PolicyAction.not_evaluated, newest.policy_action);
try testing.expectEqual(provenance.PolicyReason.snapshot_unavailable, newest.policy_reason);
try testing.expectEqual(provenance.RouteKind.rejected, newest.route_kind);
const oldest = rows.items[1];
try testing.expectEqual(@as(i64, 1), oldest.id);
@@ -826,11 +1272,15 @@ test "selectQueries returns the newest row first and reads every column" {
try testing.expectEqualStrings("ads.example.net", oldest.domain);
try testing.expectEqualStrings("192.0.2.10", oldest.client_ip);
try testing.expectEqual(@as(?u16, 28), oldest.qtype);
try testing.expectEqual(@as(u16, 1), oldest.qclass);
try testing.expectEqual(@as(u12, 0), oldest.rcode);
try testing.expect(oldest.blocked);
try testing.expectEqualStrings("blocklist", oldest.block_reason);
try testing.expectEqual(@as(?i64, 4200), oldest.response_time_us);
try testing.expectEqual(@as(?bool, true), oldest.cache_hit);
try testing.expectEqualStrings("https://dns.example/dns-query", oldest.upstream);
try testing.expectEqual(provenance.PolicyAction.block, oldest.policy_action);
try testing.expectEqual(provenance.PolicyReason.blocklist_domain, oldest.policy_reason);
try testing.expectEqual(provenance.RouteKind.blocked, oldest.route_kind);
}
test "selectQueries honours the limit and caps it at max_limit" {
@@ -895,7 +1345,9 @@ test "each filter narrows the result on its own" {
var blocked_row = plainRow(200, "ads.example.net");
blocked_row.client_ip = "192.0.2.20";
blocked_row.blocked = true;
blocked_row.block_reason = "blocklist";
blocked_row.policy_action = .block;
blocked_row.policy_reason = .blocklist_domain;
blocked_row.route_kind = .blocked;
try seed(&database, &.{
plainRow(100, "one.example.com"),
blocked_row,
@@ -1004,7 +1456,9 @@ test "statsTotals aggregates the window and averages only the timed rows" {
timed.response_time_us = 100;
var blocked_row = plainRow(150, "ads.example");
blocked_row.blocked = true;
blocked_row.block_reason = "blocklist";
blocked_row.policy_action = .block;
blocked_row.policy_reason = .blocklist_domain;
blocked_row.route_kind = .blocked;
blocked_row.response_time_us = 200;
var cached = plainRow(199, "b.example");
cached.client_ip = "192.0.2.99";
@@ -1042,7 +1496,9 @@ test "timeseries writes every bucket, including the ones with no rows" {
var blocked_row = plainRow(1020, "ads.example");
blocked_row.blocked = true;
blocked_row.block_reason = "blocklist";
blocked_row.policy_action = .block;
blocked_row.policy_reason = .blocklist_domain;
blocked_row.route_kind = .blocked;
var cached = plainRow(1035, "b.example");
cached.cache_hit = true;
try seed(&database, &.{
+17 -3
View File
@@ -114,8 +114,10 @@ pub const Retention = struct {
// still prunes through `Store.init`.
if (store) |s| s.prune(io, now);
if (queries_repo.pruneOlderThan(database, cutoff)) |deleted| {
add(&self.counters.rows_pruned, @intCast(deleted));
// One operation, not two: the delete and the coverage watermark it
// advances commit together or not at all (`queries_repo`).
if (queries_repo.pruneOlderThan(database, cutoff)) |pruned| {
add(&self.counters.rows_pruned, @intCast(pruned.deleted));
maintenance(store, io, now, "prune", null);
} else |err| {
log.warn("retention prune before {d} failed: {s}", .{ cutoff, @errorName(err) });
@@ -255,11 +257,23 @@ fn writeRows(database: *db.Db, timestamps: []const i64) !void {
.domain = "example.com",
.client_ip = "192.0.2.10",
.qtype = 1,
.qclass = 1,
.rcode = 0,
.blocked = false,
.block_reason = null,
.response_time_us = null,
.cache_hit = null,
.upstream = null,
.group_id = 1,
.group_name = "default",
.policy_action = .allow,
.policy_reason = .no_match,
.matched = null,
.source_id = null,
.source_name = null,
.cname_target = null,
.safe_search_target = null,
.route_kind = .upstream,
.forward_zone = null,
};
}
try writer.writeBatch(rows[0..timestamps.len]);
+4
View File
@@ -33,11 +33,13 @@ comptime {
_ = @import("server/resolver_integration_test.zig");
_ = @import("storage/db.zig");
_ = @import("config/model.zig");
_ = @import("config/limits.zig");
_ = @import("config/validate.zig");
_ = @import("config/faults.zig");
_ = @import("storage/config_schema.zig");
_ = @import("storage/migrations.zig");
_ = @import("storage/querylog_schema.zig");
_ = @import("storage/provenance.zig");
_ = @import("storage/repositories/context.zig");
_ = @import("storage/repositories/crud.zig");
_ = @import("storage/repositories/groups_repo.zig");
@@ -92,6 +94,8 @@ comptime {
_ = @import("server/shutdown.zig");
_ = @import("server/phase7_integration_test.zig");
_ = @import("web/sse.zig");
_ = @import("web/coverage.zig");
_ = @import("web/provenance_view.zig");
_ = @import("server/query_sink.zig");
_ = @import("web/auth.zig");
_ = @import("web/api_limiter.zig");
+4
View File
@@ -75,8 +75,12 @@ pub const DohClient = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
const self: *DohClient = @ptrCast(@alignCast(ptr));
// The endpoint outlives the client, so the borrow is safe for the whole
// query. Set before the attempt: a failure names this resolver too.
selected.* = self.endpoint.url;
return self.exchange(io, query, response_buf);
}
+3 -1
View File
@@ -44,7 +44,9 @@ fn runExchange(io: std.Io, params: Params) anyerror!usize {
const endpoint = try transport.Endpoint.parse("https://cloudflare-dns.com/dns-query");
var doh = try doh_client.DohClient.init(&http, endpoint, &request_buf, &transfer_buf);
const reply = try doh.client().exchange(io, query_bytes, params.response_buf);
var selected: ?[]const u8 = null;
const reply = try doh.client().exchange(io, query_bytes, params.response_buf, &selected);
std.debug.assert(std.mem.eql(u8, selected.?, endpoint.url));
return reply.len;
}
+4
View File
@@ -146,8 +146,12 @@ pub const DotClient = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
const self: *DotClient = @ptrCast(@alignCast(ptr));
// The endpoint outlives the client, so the borrow is safe for the whole
// query. Set before the attempt: a failure names this resolver too.
selected.* = self.endpoint.url;
return self.exchange(io, query, response_buf);
}
+3 -1
View File
@@ -57,7 +57,9 @@ fn runExchange(io: std.Io, params: Params) anyerror!usize {
params.bundle_lock,
params.buffers,
);
const reply = try client.client().exchange(io, query_bytes, params.response_buf);
var selected: ?[]const u8 = null;
const reply = try client.client().exchange(io, query_bytes, params.response_buf, &selected);
std.debug.assert(std.mem.eql(u8, selected.?, endpoint.url));
return reply.len;
}
+291 -46
View File
@@ -172,9 +172,10 @@ pub const Pool = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
const self: *Pool = @ptrCast(@alignCast(ptr));
return self.exchange(io, query, response_buf);
return self.exchange(io, query, response_buf, selected);
}
/// `response_buf` is handed to each attempt in turn, so a failed attempt
@@ -191,15 +192,16 @@ pub const Pool = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
const len = try transport.raceWithin(io, self.timeouts.total, exchangeLoopLen, .{
self, io, query, response_buf,
self, io, query, response_buf, selected,
});
return response_buf[0..len];
}
/// The two-pass failover loop, as a raceable task. It returns the reply's
/// length rather than its slice for the reason `exchangeLen` in the tests
/// length rather than its slice for the reason `Attributed` in the tests
/// below does: `Io.concurrent` stores the future's return value, so the
/// bytes are read back out of the caller's `response_buf` by `exchange`.
fn exchangeLoopLen(
@@ -207,6 +209,7 @@ pub const Pool = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError!usize {
const now = std.Io.Clock.awake.now(io);
var last_fault: ?transport.ExchangeError = null;
@@ -240,6 +243,13 @@ pub const Pool = struct {
}
attempted = true;
// Before the attempt, not after it: a failover reports whoever
// answered, an all-failed exchange reports the last endpoint
// tried, and a cancellation mid-flight reports the endpoint the
// query was in. The url is owned by the Endpoint, which outlives
// the pool, so the borrow stays valid past this loop.
selected.* = entry.endpoint.url;
const result = self.attempt(io, entry.client, query, response_buf);
const completed_at = std.Io.Clock.awake.now(io);
@@ -294,6 +304,12 @@ pub const Pool = struct {
}
/// One exchange raced against the per-attempt budget.
///
/// The leaf client reports an identity of its own, which the pool discards:
/// the entry's endpoint is the pool's own naming of the same resolver, and
/// it is what the caller was handed. A leaf writing its identity into the
/// caller's slot would let a test fake overwrite the endpoint that actually
/// answered.
fn attempt(
self: *Pool,
io: std.Io,
@@ -301,8 +317,9 @@ pub const Pool = struct {
query: []const u8,
response_buf: []u8,
) transport.ExchangeError![]u8 {
var leaf_selected: ?[]const u8 = null;
return transport.raceWithin(io, self.timeouts.attempt, transport.Client.exchange, .{
entry_client, io, query, response_buf,
entry_client, io, query, response_buf, &leaf_selected,
});
}
@@ -409,11 +426,23 @@ const response_bytes =
"\x07example\x03com\x00\x00\x01\x00\x01" ++
"\xc0\x0c\x00\x01\x00\x01\x00\x00\x01\x2c\x00\x04\x5d\xb8\xd8\x22";
/// The same question answered differently, so a reply identifies the entry that
/// produced it: one A record, a different address.
const alt_response_bytes =
"\x12\x34\x81\x80\x00\x01\x00\x01\x00\x00\x00\x00" ++
"\x07example\x03com\x00\x00\x01\x00\x01" ++
"\xc0\x0c\x00\x01\x00\x01\x00\x00\x01\x2c\x00\x04\x0a\x00\x00\x01";
/// Stands in for a DoH or DoT client. Every behaviour the pool has to react to
/// is one variant, and every call is counted so a test can assert that an entry
/// in backoff was not touched.
const Fake = struct {
behavior: Behavior,
/// Replaces `behavior` after the first call. One entry that fails the task
/// which reaches it first and answers the next is what makes two concurrent
/// exchanges end on different entries; a single behaviour cannot say that.
/// Mutated under the entry's `busy` lock, like `calls`.
then: ?Behavior = null,
calls: usize = 0,
in_flight: std.atomic.Value(u32) = .init(0),
/// The most tasks ever inside `exchangeFn` at once. The per-entry lock is
@@ -437,15 +466,24 @@ const Fake = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) transport.ExchangeError![]u8 {
_ = query;
// Deliberately not the endpoint url: the pool must report its own
// entry, so a test can tell the two apart.
selected.* = "fake://leaf";
const self: *Fake = @ptrCast(@alignCast(ptr));
const entrants = self.in_flight.fetchAdd(1, .acq_rel) + 1;
defer _ = self.in_flight.fetchSub(1, .acq_rel);
_ = self.peak_in_flight.fetchMax(entrants, .acq_rel);
self.calls += 1;
switch (self.behavior) {
const behavior = self.behavior;
if (self.then) |next| {
self.behavior = next;
self.then = null;
}
switch (behavior) {
.reply => |bytes| return copy(bytes, response_buf),
.fail => |err| return err,
.slow => |slow| {
@@ -539,10 +577,106 @@ test "Pool satisfies the Client interface" {
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
const reply = try pool.client().exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
const reply = try pool.client().exchange(io, query_bytes, &buf, &selected);
try testing.expectEqualSlices(u8, response_bytes, reply);
}
test "the pool reports the endpoint that answered, not the leaf client's own name" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var fake: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{testEntry("https://a.example/dns-query", &fake, 10)};
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
var selected: ?[]const u8 = null;
_ = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqualStrings("https://a.example/dns-query", selected.?);
}
test "a failover reports the endpoint that answered, not the first one tried" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var bad: Fake = .{ .behavior = .{ .fail = error.Timeout } };
var good: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{
testEntry("https://bad.example/dns-query", &bad, 10),
testEntry("https://good.example/dns-query", &good, 20),
};
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
var selected: ?[]const u8 = null;
_ = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqualStrings("https://good.example/dns-query", selected.?);
}
test "an all-failed exchange reports the last endpoint attempted" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var first: Fake = .{ .behavior = .{ .fail = error.ConnectFailed } };
var last: Fake = .{ .behavior = .{ .fail = error.Timeout } };
var entries = [_]Entry{
testEntry("https://first.example/dns-query", &first, 10),
testEntry("https://last.example/dns-query", &last, 20),
};
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
var selected: ?[]const u8 = null;
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expectEqualStrings("https://last.example/dns-query", selected.?);
}
test "a timeout mid-flight reports the endpoint the query was in" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var stalling: Fake = .{ .behavior = .{ .slow = .{
.duration = .{ .raw = .fromSeconds(30), .clock = .awake },
.reply = response_bytes,
} } };
var untouched: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{
testEntry("https://stalling.example/dns-query", &stalling, 10),
testEntry("https://untouched.example/dns-query", &untouched, 20),
};
var pool: Pool = .init(&entries, test_cfg, .{
.attempt = .{ .raw = .fromMilliseconds(200), .clock = .awake },
.total = .{ .raw = .fromMilliseconds(60), .clock = .awake },
}, 1);
var buf: [512]u8 = undefined;
var selected: ?[]const u8 = null;
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expectEqualStrings("https://stalling.example/dns-query", selected.?);
try testing.expectEqual(@as(usize, 0), untouched.calls);
}
test "an exchange that attempted nothing reports no endpoint" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
var fake: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{testEntry("https://a.example/dns-query", &fake, 10)};
entries[0].enabled = false;
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
var selected: ?[]const u8 = null;
try testing.expectError(error.ConnectFailed, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expect(selected == null);
}
test "entries are tried in ascending priority order" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -560,7 +694,8 @@ test "entries are tried in ascending priority order" {
try testing.expectEqual(@as(i32, 10), entries[0].priority);
var buf: [512]u8 = undefined;
_ = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
_ = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqual(@as(usize, 1), low.calls);
try testing.expectEqual(@as(usize, 0), high.calls);
}
@@ -579,7 +714,8 @@ test "a peer fault fails over to the next entry and is recorded" {
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
const reply = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
const reply = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqualSlices(u8, response_bytes, reply);
try testing.expectEqual(@as(u32, 1), entries[0].health.consecutive_failures);
@@ -603,12 +739,13 @@ test "an entry in backoff is skipped while another is available" {
var buf: [512]u8 = undefined;
// Two failures reach `failure_threshold` and open a backoff window.
_ = try pool.exchange(io, query_bytes, &buf);
_ = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
_ = try pool.exchange(io, query_bytes, &buf, &selected);
_ = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqual(@as(usize, 2), bad.calls);
try testing.expect(entries[0].health.backoff_until != null);
_ = try pool.exchange(io, query_bytes, &buf);
_ = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqual(@as(usize, 2), bad.calls);
try testing.expectEqual(@as(usize, 3), good.calls);
}
@@ -627,13 +764,14 @@ test "every entry in backoff is still probed" {
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf));
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf));
var selected: ?[]const u8 = null;
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expect(entries[0].health.backoff_until != null);
try testing.expect(entries[1].health.backoff_until != null);
// Pass one now has no candidate at all. Pass two probes both anyway.
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf));
try testing.expectError(error.BadResponse, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expectEqual(@as(usize, 3), first.calls);
try testing.expectEqual(@as(usize, 3), second.calls);
}
@@ -652,7 +790,8 @@ test "a local resource error short-circuits and records nothing" {
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
try testing.expectError(error.OutOfMemory, pool.exchange(io, query_bytes, &buf));
var selected: ?[]const u8 = null;
try testing.expectError(error.OutOfMemory, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expectEqual(@as(usize, 0), good.calls);
try testing.expectEqual(@as(u64, 0), entries[0].health.total_failures);
try testing.expectEqual(@as(u32, 0), entries[0].health.consecutive_failures);
@@ -672,7 +811,8 @@ test "a cancellation short-circuits and records nothing" {
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
try testing.expectError(error.Canceled, pool.exchange(io, query_bytes, &buf));
var selected: ?[]const u8 = null;
try testing.expectError(error.Canceled, pool.exchange(io, query_bytes, &buf, &selected));
try testing.expectEqual(@as(usize, 0), good.calls);
try testing.expectEqual(@as(u64, 0), entries[0].health.total_failures);
}
@@ -699,7 +839,8 @@ test "an attempt that outruns the budget is a recorded Timeout" {
}, 1);
var buf: [512]u8 = undefined;
const reply = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
const reply = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqualSlices(u8, response_bytes, reply);
try testing.expectEqual(@as(usize, 1), slow.calls);
@@ -736,7 +877,8 @@ test "two stalling upstreams cost the total budget, not one budget each" {
var buf: [512]u8 = undefined;
const started = std.Io.Clock.awake.now(io);
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf));
var selected: ?[]const u8 = null;
try testing.expectError(error.Timeout, pool.exchange(io, query_bytes, &buf, &selected));
const elapsed_ns = std.Io.Clock.awake.now(io).nanoseconds - started.nanoseconds;
// Under one attempt budget, so the outer deadline is provably what fired.
@@ -770,7 +912,8 @@ test "every entry disabled yields ConnectFailed without waiting out the total bu
var buf: [512]u8 = undefined;
const started = std.Io.Clock.awake.now(io);
try testing.expectError(error.ConnectFailed, pool.exchange(io, query_bytes, &buf));
var selected: ?[]const u8 = null;
try testing.expectError(error.ConnectFailed, pool.exchange(io, query_bytes, &buf, &selected));
const elapsed_ns = std.Io.Clock.awake.now(io).nanoseconds - started.nanoseconds;
try testing.expect(elapsed_ns < @as(i96, 5) * std.time.ns_per_s);
@@ -799,7 +942,8 @@ test "a wired accumulator receives both outcomes the pool records" {
var buf: [512]u8 = undefined;
// One exchange: the first entry fails over into the second, so this drives
// one failure and one success.
_ = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
_ = try pool.exchange(io, query_bytes, &buf, &selected);
// Two cells, one per url, in whatever minute the wall clock is in.
try testing.expectEqual(@as(u32, 2), acc.snapshotStats(io).pending);
@@ -852,8 +996,9 @@ test "snapshot reports the counters in pool order" {
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf: [512]u8 = undefined;
_ = try pool.exchange(io, query_bytes, &buf);
_ = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
_ = try pool.exchange(io, query_bytes, &buf, &selected);
_ = try pool.exchange(io, query_bytes, &buf, &selected);
var out: [4]Snapshot = undefined;
const written = try pool.snapshot(io, &out);
@@ -882,13 +1027,30 @@ test "snapshot reports the counters in pool order" {
try testing.expectEqual(@as(usize, 1), try pool.snapshot(io, &one));
}
/// One concurrent call's whole result: how much of its buffer the reply filled,
/// and which resolver the pool said answered it.
///
/// `Io.concurrent` stores the return value in the future, so the reply slice is
/// reduced to its length here and the bytes are read back out of the caller's
/// buffer. Returning a `usize` also lets the test discard a result with
/// `catch 0`.
fn exchangeLen(pool: *Pool, io: std.Io, buf: []u8) transport.ExchangeError!usize {
const reply = try pool.exchange(io, query_bytes, buf);
return reply.len;
/// buffer. `selected` survives the trip because it borrows an `Endpoint.url`,
/// which outlives the pool.
const Attributed = struct {
reply_len: usize,
selected: ?[]const u8,
/// What a teardown `await` discards into once the assertions above it have
/// already taken the value.
const discarded: Attributed = .{ .reply_len = 0, .selected = null };
};
/// Each call keeps its own `selected` out-value rather than dropping it: the
/// pointer is written per call, on the stack of the task that made it, and a
/// pool that hung it off `*Pool` instead would hand one call's identity to
/// another. Nothing but a per-call capture can see that.
fn exchangeAttributed(pool: *Pool, io: std.Io, buf: []u8) transport.ExchangeError!Attributed {
var selected: ?[]const u8 = null;
const reply = try pool.exchange(io, query_bytes, buf, &selected);
return .{ .reply_len = reply.len, .selected = selected };
}
test "concurrent exchanges through one entry do not overlap" {
@@ -909,25 +1071,102 @@ test "concurrent exchanges through one entry do not overlap" {
var buf_a: [512]u8 = undefined;
var buf_b: [512]u8 = undefined;
var first = io.concurrent(exchangeLen, .{ &pool, io, &buf_a }) catch |err| switch (err) {
var first = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_a }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = first.await(io) catch 0;
var second = io.concurrent(exchangeLen, .{ &pool, io, &buf_b }) catch |err| switch (err) {
defer _ = first.await(io) catch Attributed.discarded;
var second = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_b }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = second.await(io) catch 0;
defer _ = second.await(io) catch Attributed.discarded;
const len_a = try first.await(io);
const len_b = try second.await(io);
const result_a = try first.await(io);
const result_b = try second.await(io);
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..len_a]);
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..len_b]);
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..result_a.reply_len]);
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..result_b.reply_len]);
try testing.expectEqualStrings("https://only.example/dns-query", result_a.selected.?);
try testing.expectEqualStrings("https://only.example/dns-query", result_b.selected.?);
try testing.expectEqual(@as(usize, 2), fake.calls);
try testing.expectEqual(@as(u32, 1), fake.peak_in_flight.load(.acquire));
try testing.expectEqual(@as(u64, 2), entries[0].health.total_successes);
}
/// The two entries of the test below, each answering with bytes only it
/// produces so a call's reply proves which entry served it independently of
/// what the pool reported.
const divergent_first_url = "https://first.example/dns-query";
const divergent_second_url = "https://second.example/dns-query";
fn expectAnsweredByReporter(result: Attributed, buf: []const u8) !void {
const url = result.selected orelse return error.TestExpectedSelectedResolver;
const reply = if (std.mem.eql(u8, url, divergent_first_url))
alt_response_bytes
else if (std.mem.eql(u8, url, divergent_second_url))
response_bytes
else
return error.TestUnexpectedResolver;
try testing.expectEqualSlices(u8, reply, buf[0..result.reply_len]);
}
test "overlapping exchanges each report the entry that answered that call" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
const io = threaded.io();
// The first entry fails the task that reaches it first, slowly enough that
// the second task is queued behind it, then answers that second task. One
// failure is under `test_cfg`'s threshold of two, so no backoff steers the
// waiting task away and the two calls end on different entries.
var first_entry: Fake = .{
.behavior = .{ .slow_fail = .{
.duration = .{ .raw = .fromMilliseconds(50), .clock = .awake },
.err = error.ConnectFailed,
} },
.then = .{ .reply = alt_response_bytes },
};
// Slow too, so the failed-over call is still in flight while the other call
// is being answered — a shared identity would be overwritten under it.
var second_entry: Fake = .{ .behavior = .{ .slow = .{
.duration = .{ .raw = .fromMilliseconds(50), .clock = .awake },
.reply = response_bytes,
} } };
var entries = [_]Entry{
testEntry(divergent_first_url, &first_entry, 10),
testEntry(divergent_second_url, &second_entry, 20),
};
var pool: Pool = .init(&entries, test_cfg, test_timeouts, 1);
var buf_a: [512]u8 = undefined;
var buf_b: [512]u8 = undefined;
var first = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_a }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = first.await(io) catch Attributed.discarded;
var second = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_b }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = second.await(io) catch Attributed.discarded;
const result_a = try first.await(io);
const result_b = try second.await(io);
// Which task lands where depends on the order they take the first entry's
// lock, so the claim is over the pair: one identity each, and each one
// matching the bytes that call received.
try testing.expect(!std.mem.eql(u8, result_a.selected.?, result_b.selected.?));
try expectAnsweredByReporter(result_a, &buf_a);
try expectAnsweredByReporter(result_b, &buf_b);
try testing.expectEqual(@as(usize, 2), first_entry.calls);
try testing.expectEqual(@as(usize, 1), second_entry.calls);
try testing.expectEqual(@as(u32, 1), first_entry.peak_in_flight.load(.acquire));
try testing.expectEqual(@as(u64, 1), entries[0].health.total_failures);
try testing.expectEqual(@as(u64, 1), entries[0].health.total_successes);
try testing.expectEqual(@as(u64, 1), entries[1].health.total_successes);
}
test "an entry that enters backoff while a task waits on it is not attempted" {
var threaded: std.Io.Threaded = .init(testing.allocator, .{});
defer threaded.deinit();
@@ -956,21 +1195,25 @@ test "an entry that enters backoff while a task waits on it is not attempted" {
var buf_a: [512]u8 = undefined;
var buf_b: [512]u8 = undefined;
var first = io.concurrent(exchangeLen, .{ &pool, io, &buf_a }) catch |err| switch (err) {
var first = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_a }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = first.await(io) catch 0;
var second = io.concurrent(exchangeLen, .{ &pool, io, &buf_b }) catch |err| switch (err) {
defer _ = first.await(io) catch Attributed.discarded;
var second = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_b }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = second.await(io) catch 0;
defer _ = second.await(io) catch Attributed.discarded;
const len_a = try first.await(io);
const len_b = try second.await(io);
const result_a = try first.await(io);
const result_b = try second.await(io);
// Both tasks fail over to the healthy entry and get an answer.
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..len_a]);
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..len_b]);
// Both tasks fail over to the healthy entry and get an answer, and both
// report it: the identity is the entry that answered, not the one that
// failed on the way there.
try testing.expectEqualSlices(u8, response_bytes, buf_a[0..result_a.reply_len]);
try testing.expectEqualSlices(u8, response_bytes, buf_b[0..result_b.reply_len]);
try testing.expectEqualStrings("https://good.example/dns-query", result_a.selected.?);
try testing.expectEqualStrings("https://good.example/dns-query", result_b.selected.?);
try testing.expectEqual(@as(usize, 2), good.calls);
// The point of the test: the entry was attempted once, not twice. Without
@@ -998,7 +1241,8 @@ test "a successful exchange with nothing open costs the store no statement" {
var buf: [512]u8 = undefined;
const before = fx.store.statements;
for (0..20) |_| _ = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
for (0..20) |_| _ = try pool.exchange(io, query_bytes, &buf, &selected);
try testing.expectEqual(before, fx.store.statements);
try testing.expectEqual(@as(i64, 0), try fx.count("SELECT count(*) FROM operational_events"));
}
@@ -1022,7 +1266,8 @@ test "a failing then recovering upstream leaves exactly one resolved episode" {
pool.diagnostics = &fx.store;
var buf: [512]u8 = undefined;
_ = try pool.exchange(io, query_bytes, &buf);
var selected: ?[]const u8 = null;
_ = try pool.exchange(io, query_bytes, &buf, &selected);
// Backoff would park the failing entry, so the second failure is driven
// through `recordFailure` itself rather than through another exchange.
pool.recordFailure(io, &entries[0], std.Io.Clock.awake.now(io), error.ConnectFailed);
+55 -4
View File
@@ -35,6 +35,13 @@ pub fn parsePrefix(bytes: [prefix_len]u8) u16 {
return std.mem.readInt(u16, &bytes, .big);
}
/// The longest DNS name in text form, and so the longest host an endpoint url
/// can name. Every consumer of `Endpoint.host` sizes itself from this bound
/// rather than re-deriving it: the query log's `upstream` column is built for
/// the widest `scheme://host:port` this permits, so a longer host would reach
/// storage only as a silently shortened identity.
pub const max_host_len = 253;
pub const doh_default_port = 443;
pub const dot_default_port = 853; // RFC 7858 §3.1
pub const doh_default_path = "/dns-query"; // RFC 8484 §4.1 well-known template
@@ -46,14 +53,15 @@ pub const Endpoint = struct {
scheme: Scheme,
/// The original text, for logs and the health API.
url: []const u8,
/// No brackets, no port. Used for SNI and certificate verification.
/// No brackets, no port, never empty, at most `max_host_len` bytes. Used
/// for SNI and certificate verification.
host: []const u8,
port: u16,
/// DoH only; always starts with '/'; `doh_default_path` when absent. A DoT
/// endpoint has no request path, so it carries "/" and nothing reads it.
path: []const u8,
pub const ParseError = error{ UnsupportedScheme, MissingHost, BadPort, BadUrl };
pub const ParseError = error{ UnsupportedScheme, MissingHost, HostTooLong, BadPort, BadUrl };
const doh_prefix = "https://";
const dot_prefix = "tls://";
@@ -82,6 +90,10 @@ pub const Endpoint = struct {
const host, const port_text = try splitAuthority(authority);
if (host.len == 0) return error.MissingHost;
// Rejected here rather than tolerated: every identity built from this
// endpoint is bounded by `max_host_len`, so a longer host would parse
// clean and then be shortened where it is stored or logged.
if (host.len > max_host_len) return error.HostTooLong;
const port: u16 = if (port_text) |text| blk: {
if (text.len == 0) return error.BadPort;
@@ -329,17 +341,27 @@ pub const Client = struct {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) ExchangeError![]u8,
/// Returns a prefix of `response_buf`. The returned message has already
/// passed `validateResponse` against `query`.
///
/// `selected` names the resolver the exchange used. An implementation
/// writes it *before* each attempt, never after, so a failed exchange still
/// names the last resolver it tried — a SERVFAIL row without its resolver
/// explains nothing. The slice must outlive the call; every implementation
/// borrows storage it owns for at least the query's duration. Callers
/// initialize it to null: a `null` after the call means no resolver was
/// reached at all.
pub fn exchange(
self: Client,
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) ExchangeError![]u8 {
return self.exchangeFn(self.ptr, io, query, response_buf);
return self.exchangeFn(self.ptr, io, query, response_buf, selected);
}
};
@@ -449,6 +471,31 @@ test "parse rejects an empty host" {
try testing.expectError(error.MissingHost, Endpoint.parse("tls://"));
}
test "parse takes a host at the length bound and rejects one past it" {
// Four labels, the widest a 253-byte name allows: 3 * (63 + 1) + 61.
const at_bound = ("a" ** 63 ++ ".") ** 3 ++ "a" ** 61;
comptime std.debug.assert(at_bound.len == max_host_len);
const accepted = try Endpoint.parse("https://" ++ at_bound ++ "/dns-query");
try testing.expectEqualStrings(at_bound, accepted.host);
// One byte more is one byte no consumer of `host` has room for.
try testing.expectError(
error.HostTooLong,
Endpoint.parse("https://" ++ at_bound ++ "a/dns-query"),
);
// The bound is on the host alone, so a port and a path do not spend it,
// and the bracketed form is measured with the brackets removed.
try testing.expectError(
error.HostTooLong,
Endpoint.parse("tls://" ++ at_bound ++ "a:853"),
);
try testing.expectError(
error.HostTooLong,
Endpoint.parse("https://[" ++ at_bound ++ "a]:8443/dns-query"),
);
}
test "parse rejects a bad port" {
try testing.expectError(error.BadPort, Endpoint.parse("https://h:99999/"));
try testing.expectError(error.BadPort, Endpoint.parse("https://h:/"));
@@ -641,8 +688,10 @@ test "a fake client satisfies the Client interface" {
io: std.Io,
query: []const u8,
response_buf: []u8,
selected: *?[]const u8,
) ExchangeError![]u8 {
_ = io;
selected.* = "fake://echo";
const self: *@This() = @ptrCast(@alignCast(ptr));
self.calls += 1;
if (query.len > response_buf.len) return error.ResponseTooLarge;
@@ -657,9 +706,11 @@ test "a fake client satisfies the Client interface" {
var fake: Fake = .{};
var buf: [16]u8 = undefined;
const echoed = try fake.client().exchange(undefined, "hello", &buf);
var selected: ?[]const u8 = null;
const echoed = try fake.client().exchange(undefined, "hello", &buf, &selected);
try testing.expectEqualStrings("hello", echoed);
try testing.expectEqual(@as(usize, 1), fake.calls);
try testing.expectEqualStrings("fake://echo", selected.?);
}
/// A query for example.com A: id 0x1234, RD set, one question.
+56
View File
@@ -0,0 +1,56 @@
//! How much of the window a client asked about the query log can still answer
//! for.
//!
//! Retention deletes old rows and advances a watermark in the same transaction
//! (`queries_repo.pruneOlderThan`), so the file knows the oldest instant it is
//! complete for. Without that fact on the wire a chart draws a pruned week as a
//! week of silence, which is the one reading that is certainly wrong.
//!
//! Three endpoints carry it — `/api/queries`, `/api/stats` and
//! `/api/stats/timeseries` — and they judge it against their own effective
//! lower bound: the client's `since` for the query log, the period's aligned
//! window start for the two stats endpoints.
const std = @import("std");
const db = @import("../storage/db.zig");
const queries_repo = @import("../storage/repositories/queries_repo.zig");
pub const Coverage = struct {
/// True only when the whole requested window is inside what the file still
/// holds. A request with no lower bound at all asks about all of history,
/// which no file that has ever pruned can promise.
complete: bool,
/// The oldest instant the file is complete for, unix seconds.
available_since: i64,
};
pub fn of(available_since: i64, since: ?i64) Coverage {
return .{
.complete = if (since) |lower_bound| lower_bound >= available_since else false,
.available_since = available_since,
};
}
/// Reads the watermark for a request that is about to answer.
pub fn read(database: *db.Db, since: ?i64) db.Error!Coverage {
return of(try queries_repo.availableSince(database), since);
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const testing = std.testing;
test "a window that starts at or after the watermark is complete" {
try testing.expect(of(1000, 1000).complete);
try testing.expect(of(1000, 1001).complete);
try testing.expect(!of(1000, 999).complete);
}
test "an unbounded window is never complete" {
const unbounded = of(1000, null);
try testing.expect(!unbounded.complete);
try testing.expectEqual(@as(i64, 1000), unbounded.available_since);
}
+33 -37
View File
@@ -23,7 +23,7 @@ const std = @import("std");
const address = @import("../../platform/address.zig");
const http_util = @import("../http_util.zig");
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
const provenance_view = @import("../provenance_view.zig");
const server = @import("../server.zig");
const sse = @import("../sse.zig");
@@ -36,32 +36,13 @@ pub const heartbeat_interval: std.Io.Clock.Duration = .{
.clock = .awake,
};
/// One event's `data:` payload the `/api/queries` row fields (ruling 20),
/// minus `id`: a live entry precedes persistence, so no row id exists yet.
pub const EventView = struct {
ts: i64,
domain: []const u8,
client_ip: []const u8,
qtype: ?u16,
blocked: bool,
block_reason: []const u8,
response_time_us: ?i64,
cache_hit: ?bool,
upstream: []const u8,
};
/// One event's `data:` payload: the shared full-provenance DTO, exactly. A live
/// event says everything `GET /api/queries/{id}` would say about the same query
/// except its id, which does not exist yet — the entry precedes its own insert.
pub const EventView = provenance_view.Provenance;
pub fn view(entry: *const sse.Entry) EventView {
return .{
.ts = entry.timestamp,
.domain = entry.domain(),
.client_ip = entry.clientIp(),
.qtype = entry.qtype,
.blocked = entry.blocked,
.block_reason = entry.blockReason(),
.response_time_us = entry.response_time_us,
.cache_hit = entry.cache_hit,
.upstream = entry.upstream(),
};
return provenance_view.fromEntry(entry);
}
/// One `event: query` frame. JSON never contains a raw newline, so the whole
@@ -139,14 +120,18 @@ pub fn stream(
const testing = std.testing;
test "the event payload carries the /api/queries row fields, minus id" {
const row_fields = @typeInfo(queries_repo.QueryRow).@"struct".fields;
test "the event payload is the detail body minus its id, name and type for name" {
const detail_fields = @typeInfo(provenance_view.QueryDetail).@"struct".fields;
const view_fields = @typeInfo(EventView).@"struct".fields;
comptime {
std.debug.assert(view_fields.len == row_fields.len - 1);
std.debug.assert(std.mem.eql(u8, row_fields[0].name, "id"));
for (row_fields[1..], view_fields) |row_field, view_field| {
std.debug.assert(std.mem.eql(u8, row_field.name, view_field.name));
std.debug.assert(view_fields.len == detail_fields.len - 1);
std.debug.assert(std.mem.eql(u8, detail_fields[0].name, "id"));
for (detail_fields[1..], view_fields) |detail_field, view_field| {
std.debug.assert(std.mem.eql(u8, detail_field.name, view_field.name));
// Names alone would let a group keep its key while changing what it
// holds, which is the drift a live viewer would see and a detail
// page would not.
std.debug.assert(detail_field.type == view_field.type);
}
}
}
@@ -157,11 +142,19 @@ test "a frame is one event line and one data line of JSON" {
.domain = "ads.example",
.client_ip = "192.0.2.10",
.qtype = 1,
.qclass = 1,
.rcode = 0,
.blocked = true,
.block_reason = "blocklist_domain",
.group_id = 1,
.group_name = "default",
.policy_action = .block,
.policy_reason = .blocklist_domain,
.matched = "ads.example",
.source_id = 3,
.source_name = "StevenBlack",
.route_kind = .blocked,
.response_time_us = 42,
.cache_hit = false,
.upstream = "https://dns.example/dns-query",
});
var buf: [1024]u8 = undefined;
@@ -172,10 +165,12 @@ test "a frame is one event line and one data line of JSON" {
try testing.expect(std.mem.startsWith(u8, frame, "event: query\ndata: {"));
try testing.expect(std.mem.endsWith(u8, frame, "}\n\n"));
try testing.expectEqual(@as(usize, 3), std.mem.count(u8, frame, "\n"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"ts\":1700000000"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"time\":1700000000"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"domain\":\"ads.example\""));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"blocked\":true"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"block_reason\":\"blocklist_domain\""));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"group\":{\"id\":1,\"name\":\"default\"}"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"reason\":\"blocklist_domain\""));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"source_name\":\"StevenBlack\""));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"kind\":\"blocked\""));
}
test "an unlogged field stays null and an empty string stays a string" {
@@ -191,6 +186,7 @@ test "an unlogged field stays null and an empty string stays a string" {
const frame = writer.buffered();
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"qtype\":null"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"cache_hit\":null"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"duration_us\":null"));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"upstream\":\"\""));
try testing.expect(std.mem.containsAtLeast(u8, frame, 1, "\"id\":null"));
}
+148 -10
View File
@@ -1,4 +1,5 @@
//! `GET /api/queries` — the query log, newest first (ruling 11).
//! `GET /api/queries` — the query log, newest first (ruling 11) — and
//! `GET /api/queries/{id}`, one row of it fully explained.
//!
//! Keyset pagination rather than an offset: the table is append-only and the
//! UI reads the head of it, so `id < before` is one index seek no matter how
@@ -13,9 +14,11 @@
const std = @import("std");
const Allocator = std.mem.Allocator;
const coverage = @import("../coverage.zig");
const db = @import("../../storage/db.zig");
const http_util = @import("../http_util.zig");
const logger = @import("../../storage/logger.zig");
const provenance_view = @import("../provenance_view.zig");
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
const server = @import("../server.zig");
@@ -41,6 +44,10 @@ pub const Page = struct {
queries: []const queries_repo.QueryRow,
/// The cursor for the next page, or null when this page is the last one.
next_before: ?i64,
/// Whether the log still covers the window the filter asked for. A client
/// that reads rows without reading this cannot tell an empty window from a
/// pruned one.
coverage: coverage.Coverage,
};
pub const FilterError = error{
@@ -110,6 +117,7 @@ pub fn page(
return .{
.queries = rows.items,
.next_before = if (full and rows.items.len != 0) rows.items[rows.items.len - 1].id else null,
.coverage = try coverage.read(database, filter.since),
};
}
@@ -138,10 +146,37 @@ pub fn list(
return http_util.respondJson(request, .ok, result, &.{});
}
/// `GET /api/queries/{id}` — one query, fully explained.
///
/// The row is the whole answer: every field is a fact recorded when the query
/// was answered, so nothing here is joined against current configuration. A
/// group or blocklist renamed since keeps the name it had.
pub fn detail(
state: *server.WebState,
io: std.Io,
request: *http_util.Request,
) http_util.HandlerError!void {
_ = io;
const database = state.querylog_db orelse
return http_util.respondError(request, .service_unavailable, "query log unavailable");
const row = queries_repo.detailById(database, request.arena, request.id.?) catch |err| {
log.warn("query log read failed: {s}", .{@errorName(err)});
return http_util.respondError(request, .internal_server_error, "internal error");
};
// An id retention has pruned and one that never existed are the same
// answer, and the API does not pretend to tell them apart.
const found = row orelse return http_util.respondError(request, .not_found, "not found");
return http_util.respondJson(request, .ok, provenance_view.fromDetail(found), &.{});
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const provenance = @import("../../storage/provenance.zig");
const querylog_schema = @import("../../storage/querylog_schema.zig");
const testing = std.testing;
@@ -210,21 +245,40 @@ fn openLog() !db.Db {
return database;
}
/// Rows the resolver could actually have written. Ruling 20 ties the three
/// route facts together: a block never consulted the cache, so its `cache_hit`
/// is NULL rather than false; a cache hit has no upstream to name; and only an
/// upstream answer carries one. A fixture that broke those ties would let a
/// serializer regression pass here and fail on real rows.
fn seed(database: *db.Db, count: usize) !void {
var writer = try queries_repo.BatchWriter.init(database);
defer writer.deinit();
var rows: [16]queries_repo.Row = undefined;
for (rows[0..count], 0..) |*row, i| {
const blocked = i % 2 == 0;
const from_cache = i % 4 == 1;
row.* = .{
.timestamp = 1_700_000_000 + @as(i64, @intCast(i)),
.domain = if (i % 2 == 0) "ads.example" else "safe.example",
.domain = if (blocked) "ads.example" else "safe.example",
.client_ip = "192.0.2.10",
.qtype = 1,
.blocked = i % 2 == 0,
.block_reason = if (i % 2 == 0) "blocklist_domain" else null,
.qclass = 1,
.rcode = 0,
.blocked = blocked,
.response_time_us = 500,
.cache_hit = false,
.upstream = null,
.cache_hit = if (blocked) null else from_cache,
.upstream = if (blocked or from_cache) null else "9.9.9.9",
.group_id = 1,
.group_name = "default",
.policy_action = if (blocked) .block else .allow,
.policy_reason = if (blocked) .blocklist_domain else .no_match,
.matched = if (blocked) "ads.example" else null,
.source_id = null,
.source_name = null,
.cname_target = null,
.safe_search_target = null,
.route_kind = if (blocked) .blocked else if (from_cache) .cache else .upstream,
.forward_zone = null,
};
}
try writer.writeBatch(rows[0..count]);
@@ -297,6 +351,37 @@ test "the parsed filters narrow the rows the page returns" {
try testing.expectEqual(@as(usize, 0), nobody.queries.len);
}
test "the seeded rows carry only the route shapes ruling 20 allows" {
var database = try openLog();
defer database.close();
try seed(&database, 4);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
var seen: std.EnumSet(provenance.RouteKind) = .initEmpty();
for ((try page(&database, arena.allocator(), .{})).queries) |row| {
seen.insert(row.route_kind);
switch (row.route_kind) {
.blocked => {
try testing.expectEqual(@as(?bool, null), row.cache_hit);
try testing.expectEqualStrings("", row.upstream);
},
.cache => {
try testing.expectEqual(@as(?bool, true), row.cache_hit);
try testing.expectEqualStrings("", row.upstream);
},
.upstream => {
try testing.expectEqual(@as(?bool, false), row.cache_hit);
try testing.expectEqualStrings("9.9.9.9", row.upstream);
},
.local, .forward_zone, .rejected => return error.UnseededRouteKind,
}
}
// All three, so the serializer tests below read every shape the fixture claims.
try testing.expectEqual(@as(usize, 3), seen.count());
}
test "the page serializes as the envelope ruling 11 defines" {
var database = try openLog();
defer database.close();
@@ -312,14 +397,67 @@ test "the page serializes as the envelope ruling 11 defines" {
const text = allocating.written();
try testing.expect(std.mem.startsWith(u8, text, "{\"queries\":["));
try testing.expect(std.mem.endsWith(u8, text, "\"next_before\":null}"));
for ([_][]const u8{
"\"id\":", "\"ts\":", "\"domain\":", "\"client_ip\":",
"\"qtype\":", "\"blocked\":", "\"cache_hit\":", "\"upstream\":",
"\"upstream\":", "\"response_time_us\":", "\"block_reason\":",
"\"id\":", "\"ts\":", "\"domain\":", "\"client_ip\":",
"\"qtype\":", "\"qclass\":", "\"rcode\":", "\"blocked\":",
"\"cache_hit\":", "\"upstream\":", "\"response_time_us\":", "\"policy_action\":",
"\"policy_reason\":", "\"route_kind\":",
}) |field| {
try testing.expect(std.mem.containsAtLeast(u8, text, 1, field));
}
// W1's ruling: a NULL column reads as "", and "" stays "" on the wire.
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"upstream\":\"\""));
// The column the provenance columns replaced.
try testing.expect(!std.mem.containsAtLeast(u8, text, 1, "block_reason"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"coverage\":{\"complete\":"));
}
test "the coverage of a page answers the window the filter asked for" {
var database = try openLog();
defer database.close();
try seed(&database, 1);
var arena: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena.deinit();
const watermark = try queries_repo.availableSince(&database);
const unbounded = try page(&database, arena.allocator(), .{});
try testing.expectEqual(watermark, unbounded.coverage.available_since);
try testing.expect(!unbounded.coverage.complete);
const covered = try page(&database, arena.allocator(), .{ .since = watermark });
try testing.expect(covered.coverage.complete);
const older = try page(&database, arena.allocator(), .{ .since = watermark - 1 });
try testing.expect(!older.coverage.complete);
}
test "a detail row carries every provenance field the row stored" {
var database = try openLog();
defer database.close();
try seed(&database, 1);
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
const row = (try queries_repo.detailById(&database, arena, 1)).?;
const view = provenance_view.fromDetail(row);
try testing.expectEqual(@as(i64, 1), view.id);
try testing.expectEqualStrings("ads.example", view.request.domain);
try testing.expectEqualStrings("192.0.2.10", view.request.client);
try testing.expectEqual(@as(u16, 1), view.request.qclass);
try testing.expectEqualStrings("default", view.group.name);
try testing.expectEqual(provenance.PolicyAction.block, view.policy.action);
try testing.expectEqualStrings("ads.example", view.policy.matched);
try testing.expectEqual(provenance.RouteKind.blocked, view.route.kind);
try testing.expectEqualStrings("", view.route.upstream);
try testing.expectEqual(@as(?i64, 500), view.response.duration_us);
try testing.expectEqual(
@as(?queries_repo.QueryDetail, null),
try queries_repo.detailById(&database, arena, 99),
);
}
+27 -1
View File
@@ -14,6 +14,7 @@
const std = @import("std");
const coverage = @import("../coverage.zig");
const db = @import("../../storage/db.zig");
const http_util = @import("../http_util.zig");
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
@@ -99,6 +100,10 @@ pub const TotalsBody = struct {
cached: u64,
clients: u64,
avg_response_time_us: ?i64,
/// Judged against `since`, which is the window this body reports on — so a
/// dashboard can say "history starts here" instead of charting a pruned
/// stretch as a quiet one.
coverage: coverage.Coverage,
};
pub const TimeseriesBody = struct {
@@ -107,6 +112,7 @@ pub const TimeseriesBody = struct {
until: i64,
bucket_seconds: u32,
buckets: []const queries_repo.Bucket,
coverage: coverage.Coverage,
};
pub fn totals(
@@ -121,6 +127,9 @@ pub fn totals(
const result = queries_repo.statsTotals(database, span.since, span.until) catch |err| {
return internal(request, "stats totals", err);
};
const covered = coverage.read(database, span.since) catch |err| {
return internal(request, "stats coverage", err);
};
return http_util.respondJson(request, .ok, TotalsBody{
.period = period.label(),
@@ -131,6 +140,7 @@ pub fn totals(
.cached = result.cached,
.clients = result.distinct_clients,
.avg_response_time_us = result.avg_response_time_us,
.coverage = covered,
}, &.{});
}
@@ -148,6 +158,9 @@ pub fn timeseries(
const written = queries_repo.timeseries(database, span.since, span.bucket_seconds, out) catch |err| {
return internal(request, "stats timeseries", err);
};
const covered = coverage.read(database, span.since) catch |err| {
return internal(request, "stats coverage", err);
};
return http_util.respondJson(request, .ok, TimeseriesBody{
.period = period.label(),
@@ -155,6 +168,7 @@ pub fn timeseries(
.until = span.until,
.bucket_seconds = span.bucket_seconds,
.buckets = out[0..written],
.coverage = covered,
}, &.{});
}
@@ -268,11 +282,23 @@ fn writeRow(writer: *queries_repo.BatchWriter, timestamp: i64, blocked: bool, ca
.domain = "example.com",
.client_ip = "192.0.2.10",
.qtype = 1,
.qclass = 1,
.rcode = 0,
.blocked = blocked,
.block_reason = if (blocked) "blocklist_domain" else null,
.response_time_us = 1000,
.cache_hit = cached,
.upstream = null,
.group_id = 1,
.group_name = "default",
.policy_action = if (blocked) .block else .allow,
.policy_reason = if (blocked) .blocklist_domain else .no_match,
.matched = null,
.source_id = null,
.source_name = null,
.cname_target = null,
.safe_search_target = null,
.route_kind = if (blocked) .blocked else .upstream,
.forward_zone = null,
}};
try writer.writeBatch(&rows);
}
+218 -10
View File
@@ -226,14 +226,47 @@ paths:
"503":
$ref: "#/components/responses/Unavailable"
/api/queries/{id}:
get:
summary: One query, fully explained
description: |
The full provenance of one logged query: what was asked, which group's
policy applied, what that policy decided and matched on, what was
rewritten, where the answer came from, and what the client received.
Every field is a fact recorded when the query was answered, so a group
or blocklist renamed since keeps the name it had.
parameters:
- name: id
in: path
required: true
schema: { type: integer, minimum: 1 }
responses:
"200":
description: The query.
content:
application/json:
schema:
$ref: "#/components/schemas/QueryDetail"
"401":
$ref: "#/components/responses/Unauthorized"
"404":
$ref: "#/components/responses/NotFound"
"429":
$ref: "#/components/responses/RateLimited"
"500":
$ref: "#/components/responses/Internal"
"503":
$ref: "#/components/responses/Unavailable"
/api/queries/live:
get:
summary: Live query stream (server-sent events)
description: |
`text/event-stream`. The stream opens with `retry: 3000`, then sends
one `event: query` frame per resolved query whose `data:` line is a
JSON object with the `/api/queries` row fields minus `id` (the entry
precedes persistence). A `: ping` comment goes out every 15 seconds.
`Provenance` object the body of `/api/queries/{id}` without its `id`,
which does not exist yet because the entry precedes its own insert.
A `: ping` comment goes out every 15 seconds.
A client that falls more than 64 events behind is disconnected and
should re-sync via `/api/queries` after reconnecting. Connections
per address are capped by `web.sse_max_connections_per_ip`; the
@@ -1855,9 +1888,33 @@ components:
type: boolean
description: False when no password is configured; no cookie is set.
PolicyAction:
type: string
description: |
Whether the filtering policy reached a verdict. `not_evaluated` is the
honest answer for a query answered before filtering could apply, and is
not the same as `allow`.
enum: [not_evaluated, allow, block]
PolicyReason:
type: string
description: |
Why the policy landed where it did. The first nine are the matcher's own
verdicts; the rest name a pipeline step that decided without consulting
the matcher.
enum: [rule_allow_exact, rule_block_exact, rule_allow_wildcard, rule_block_wildcard, rule_allow_regex, rule_block_regex, blocklist_exception, blocklist_domain, blocklist_wildcard, local_record, forward_zone, non_in_class, paused, snapshot_unavailable, no_match, protocol_error]
RouteKind:
type: string
description: Where the answer the client received came from.
enum: [blocked, local, forward_zone, upstream, cache, rejected]
QueryRow:
type: object
required: [id, ts, domain, client_ip, qtype, blocked, block_reason, response_time_us, cache_hit, upstream]
description: |
The summary projection the query-log table scans. The full provenance of
a row is one request away at `/api/queries/{id}`.
required: [id, ts, domain, client_ip, qtype, qclass, rcode, blocked, response_time_us, cache_hit, upstream, policy_action, policy_reason, route_kind]
properties:
id: { type: integer }
ts:
@@ -1868,10 +1925,11 @@ components:
qtype:
type: integer
nullable: true
qclass: { type: integer }
rcode:
type: integer
description: The twelve-bit EDNS extended code, not the four header bits alone.
blocked: { type: boolean }
block_reason:
type: string
description: Empty when the query was not blocked.
response_time_us:
type: integer
nullable: true
@@ -1880,11 +1938,155 @@ components:
nullable: true
upstream:
type: string
description: Empty for cache hits and local answers.
description: Empty for cache hits, local answers and blocked queries.
policy_action:
$ref: "#/components/schemas/PolicyAction"
policy_reason:
$ref: "#/components/schemas/PolicyReason"
route_kind:
$ref: "#/components/schemas/RouteKind"
ProvenanceRequest:
type: object
required: [time, domain, client, qtype, qclass]
properties:
time:
type: integer
description: Unix seconds.
domain: { type: string }
client: { type: string }
qtype:
type: integer
nullable: true
qclass: { type: integer }
ProvenanceGroup:
type: object
description: |
The client's filtering group at the time of the query, as a historical
fact: the id may name a group since renamed or deleted.
required: [id, name]
properties:
id:
type: integer
nullable: true
name: { type: string }
ProvenancePolicy:
type: object
required: [action, reason, matched, source_id, source_name]
properties:
action:
$ref: "#/components/schemas/PolicyAction"
reason:
$ref: "#/components/schemas/PolicyReason"
matched:
type: string
description: The rule pattern or list entry that decided; empty when nothing matched.
source_id:
type: integer
nullable: true
source_name:
type: string
description: The blocklist the match came from; empty for a rule.
ProvenanceRewrites:
type: object
required: [cname_target, safe_search_target]
properties:
cname_target:
type: string
description: Set when the decision was made about a CNAME target rather than the queried name.
safe_search_target: { type: string }
ProvenanceRoute:
type: object
required: [kind, forward_zone, upstream]
properties:
kind:
$ref: "#/components/schemas/RouteKind"
forward_zone: { type: string }
upstream:
type: string
description: |
Non-empty only for an attempted upstream or forward-zone exchange,
including one that failed. Already redacted: the userinfo, path,
query and fragment of a resolver url never reach here.
ProvenanceResponse:
type: object
required: [rcode, duration_us]
properties:
rcode:
type: integer
description: The twelve-bit EDNS extended code, not the four header bits alone.
duration_us:
type: integer
nullable: true
Provenance:
type: object
description: |
One query, fully explained, in the order a query meets the pipeline. The
`data:` payload of a live-stream `event: query` frame is exactly this.
required: [request, group, policy, rewrites, route, response]
properties:
request:
$ref: "#/components/schemas/ProvenanceRequest"
group:
$ref: "#/components/schemas/ProvenanceGroup"
policy:
$ref: "#/components/schemas/ProvenancePolicy"
rewrites:
$ref: "#/components/schemas/ProvenanceRewrites"
route:
$ref: "#/components/schemas/ProvenanceRoute"
response:
$ref: "#/components/schemas/ProvenanceResponse"
QueryDetail:
type: object
description: |
`Provenance` plus the row id. Written out rather than composed with
`allOf` so the drift guard reads one property list per schema.
required: [id, request, group, policy, rewrites, route, response]
properties:
id: { type: integer }
request:
$ref: "#/components/schemas/ProvenanceRequest"
group:
$ref: "#/components/schemas/ProvenanceGroup"
policy:
$ref: "#/components/schemas/ProvenancePolicy"
rewrites:
$ref: "#/components/schemas/ProvenanceRewrites"
route:
$ref: "#/components/schemas/ProvenanceRoute"
response:
$ref: "#/components/schemas/ProvenanceResponse"
Coverage:
type: object
description: |
How much of the requested window the query log can still answer for.
Retention deletes rows and advances the watermark in one transaction, so
a client can tell an empty window from a pruned one instead of charting
the gap as zero.
required: [complete, available_since]
properties:
complete:
type: boolean
description: |
True only when the window's lower bound is at or after
`available_since`. A request with no lower bound asks about all of
history, which no file that has ever pruned can promise.
available_since:
type: integer
description: The oldest instant the file is complete for, unix seconds.
QueriesPage:
type: object
required: [queries, next_before]
required: [queries, next_before, coverage]
properties:
queries:
type: array
@@ -1894,6 +2096,8 @@ components:
type: integer
nullable: true
description: Cursor for the next page; null on the last page.
coverage:
$ref: "#/components/schemas/Coverage"
DiagnosticEvent:
type: object
@@ -1983,7 +2187,7 @@ components:
StatsTotals:
type: object
required: [period, since, until, queries, blocked, cached, clients, avg_response_time_us]
required: [period, since, until, queries, blocked, cached, clients, avg_response_time_us, coverage]
properties:
period:
type: string
@@ -2004,6 +2208,8 @@ components:
type: integer
nullable: true
description: Null when no query in the window recorded a time.
coverage:
$ref: "#/components/schemas/Coverage"
Bucket:
type: object
@@ -2018,7 +2224,7 @@ components:
StatsTimeseries:
type: object
required: [period, since, until, bucket_seconds, buckets]
required: [period, since, until, bucket_seconds, buckets, coverage]
properties:
period:
type: string
@@ -2030,6 +2236,8 @@ components:
type: array
items:
$ref: "#/components/schemas/Bucket"
coverage:
$ref: "#/components/schemas/Coverage"
Lookup:
type: object
+278
View File
@@ -0,0 +1,278 @@
//! The wire shape of one query's provenance, defined once.
//!
//! Two surfaces answer with it: `GET /api/queries/{id}`, which reads a stored
//! row, and the `event: query` frames of `GET /api/queries/live`, which read a
//! queued entry that has not been written yet. They must describe a query the
//! same way — an operator watching the stream and an operator opening the row
//! afterwards are looking at the same facts — so the live event *is* this DTO
//! and the detail body is this DTO plus the row id.
//!
//! It is nested rather than flat because the six groups answer six different
//! questions, in the order a query meets them: what was asked, which group's
//! policy applied, what that policy decided, what was rewritten on the way,
//! where the answer came from, and what the client got back.
//!
//! The list row (`queries_repo.QueryRow`) stays a separate, flatter summary.
//! A table the operator scans wants columns, not a tree, and the full story is
//! one request away.
//!
//! Every text field follows the repository's convention: a NULL column reads as
//! `""`, and `""` on the wire means "absent". No field is ever written as an
//! empty string that means something else.
const std = @import("std");
const logger = @import("../storage/logger.zig");
const provenance = @import("../storage/provenance.zig");
const queries_repo = @import("../storage/repositories/queries_repo.zig");
/// What the client asked. `time` is unix seconds; `qtype` is null for a
/// question whose type the log never recorded.
pub const Request = struct {
time: i64,
domain: []const u8,
client: []const u8,
qtype: ?u16,
qclass: u16,
};
/// The client's filtering group at the time of the query, as a historical fact:
/// the id may name a group that has since been renamed or deleted, which is why
/// the name is stored beside it rather than joined at read time.
pub const Group = struct {
id: ?i64,
name: []const u8,
};
/// What the policy decided and what it matched on. `matched` is the rule
/// pattern or list entry that decided; `source_id`/`source_name` name the
/// blocklist it came from, and are absent for a rule.
pub const Policy = struct {
action: provenance.PolicyAction,
reason: provenance.PolicyReason,
matched: []const u8,
source_id: ?i64,
source_name: []const u8,
};
/// The two rewrites that can happen between the question and the answer.
/// `cname_target` is set when the decision was made about a CNAME target rather
/// than the queried name.
pub const Rewrites = struct {
cname_target: []const u8,
safe_search_target: []const u8,
};
/// Where the answer came from. `upstream` is non-empty only for an attempted
/// upstream or forward-zone exchange, including one that failed, and is already
/// redacted — the path, query and userinfo of a resolver url never reach here.
pub const Route = struct {
kind: provenance.RouteKind,
forward_zone: []const u8,
upstream: []const u8,
};
/// What the client saw. `rcode` is the twelve-bit EDNS extended code, not the
/// four header bits alone.
pub const Response = struct {
rcode: u16,
duration_us: ?i64,
};
/// One query, fully explained. The live stream's `data:` payload is exactly
/// this.
pub const Provenance = struct {
request: Request,
group: Group,
policy: Policy,
rewrites: Rewrites,
route: Route,
response: Response,
};
/// `GET /api/queries/{id}`: the same six groups, plus the id the caller asked
/// for. Spelled out rather than composed, because a JSON object is flat at its
/// top level and Zig has no field-splicing; the `comptime` block below is what
/// keeps the two from drifting.
pub const QueryDetail = struct {
id: i64,
request: Request,
group: Group,
policy: Policy,
rewrites: Rewrites,
route: Route,
response: Response,
};
comptime {
const detail = @typeInfo(QueryDetail).@"struct".fields;
const shared = @typeInfo(Provenance).@"struct".fields;
std.debug.assert(detail.len == shared.len + 1);
std.debug.assert(std.mem.eql(u8, detail[0].name, "id"));
for (detail[1..], shared) |a, b| {
std.debug.assert(std.mem.eql(u8, a.name, b.name));
std.debug.assert(a.type == b.type);
}
}
/// A stored row, as the detail endpoint answers it. Borrows `row`'s strings,
/// which the caller's arena owns.
pub fn fromDetail(row: queries_repo.QueryDetail) QueryDetail {
return .{
.id = row.id,
.request = .{
.time = row.ts,
.domain = row.domain,
.client = row.client_ip,
.qtype = row.qtype,
.qclass = row.qclass,
},
.group = .{ .id = row.group_id, .name = row.group_name },
.policy = .{
.action = row.policy_action,
.reason = row.policy_reason,
.matched = row.matched,
.source_id = row.source_id,
.source_name = row.source_name,
},
.rewrites = .{
.cname_target = row.cname_target,
.safe_search_target = row.safe_search_target,
},
.route = .{
.kind = row.route_kind,
.forward_zone = row.forward_zone,
.upstream = row.upstream,
},
.response = .{ .rcode = row.rcode, .duration_us = row.response_time_us },
};
}
/// A queued entry, as the live stream sends it. Borrows the entry's buffers, so
/// the result must not outlive the entry it was taken from — in the stream both
/// live in one loop iteration.
///
/// There is no id: the entry precedes its own insert, so no row id exists yet.
pub fn fromEntry(entry: *const logger.Entry) Provenance {
return .{
.request = .{
.time = entry.timestamp,
.domain = entry.domain(),
.client = entry.clientIp(),
.qtype = entry.qtype,
.qclass = entry.qclass,
},
.group = .{ .id = entry.group_id, .name = entry.groupName() },
.policy = .{
.action = entry.policy_action,
.reason = entry.policy_reason,
.matched = entry.matched(),
.source_id = entry.source_id,
.source_name = entry.sourceName(),
},
.rewrites = .{
.cname_target = entry.cnameTarget(),
.safe_search_target = entry.safeSearchTarget(),
},
.route = .{
.kind = entry.route_kind,
.forward_zone = entry.forwardZone(),
.upstream = entry.upstream(),
},
.response = .{ .rcode = entry.rcode, .duration_us = entry.response_time_us },
};
}
// ---------------------------------------------------------------------------
// tests
// ---------------------------------------------------------------------------
const testing = std.testing;
test "a stored row and a queued entry describe the same query identically" {
const row: queries_repo.QueryDetail = .{
.id = 7,
.ts = 1_700_000_000,
.domain = "ads.example",
.client_ip = "192.0.2.10",
.qtype = 1,
.qclass = 1,
.rcode = 3,
.blocked = true,
.response_time_us = 1234,
.cache_hit = false,
.upstream = "https://dns.example",
.group_id = 2,
.group_name = "kids",
.policy_action = .block,
.policy_reason = .blocklist_domain,
.matched = "tracker.example",
.source_id = 5,
.source_name = "StevenBlack",
.cname_target = "tracker.example",
.safe_search_target = "forcesafesearch.example",
.route_kind = .blocked,
.forward_zone = "lan",
};
const entry: logger.Entry = .init(.{
.timestamp = row.ts,
.domain = row.domain,
.client_ip = row.client_ip,
.qtype = row.qtype,
.qclass = row.qclass,
.rcode = row.rcode,
.blocked = row.blocked,
.response_time_us = row.response_time_us,
.cache_hit = row.cache_hit,
.upstream = row.upstream,
.group_id = row.group_id,
.group_name = row.group_name,
.policy_action = row.policy_action,
.policy_reason = row.policy_reason,
.matched = row.matched,
.source_id = row.source_id,
.source_name = row.source_name,
.cname_target = row.cname_target,
.safe_search_target = row.safe_search_target,
.route_kind = row.route_kind,
.forward_zone = row.forward_zone,
});
const from_row = fromDetail(row);
const from_entry = fromEntry(&entry);
try testing.expectEqual(@as(i64, 7), from_row.id);
inline for (@typeInfo(Provenance).@"struct".fields) |field| {
const a = @field(from_row, field.name);
const b = @field(from_entry, field.name);
inline for (@typeInfo(field.type).@"struct".fields) |inner| {
const left = @field(a, inner.name);
const right = @field(b, inner.name);
if (@TypeOf(left) == []const u8) {
try testing.expectEqualStrings(left, right);
} else {
try testing.expectEqual(left, right);
}
}
}
}
test "an unexplained query serializes as nulls and empty strings, not as absent keys" {
const entry: logger.Entry = .init(.{
.timestamp = 1,
.domain = "safe.example",
.client_ip = "192.0.2.11",
});
var allocating: std.Io.Writer.Allocating = .init(testing.allocator);
defer allocating.deinit();
try std.json.Stringify.value(fromEntry(&entry), .{}, &allocating.writer);
const text = allocating.written();
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"qtype\":null"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"duration_us\":null"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"id\":null"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"upstream\":\"\""));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"action\":\"not_evaluated\""));
}
+5 -1
View File
@@ -67,6 +67,10 @@ pub const table: []const router.RouteInfo = &.{
// Query log, stats, live stream, lookup.
.{ .method = .GET, .pattern = "/api/queries", .auth = .session, .policy = .read, .handler = queries.list },
.{ .method = .GET, .pattern = "/api/queries/live", .auth = .session, .policy = .read, .handler = live.stream, .rate_limit = .exempt },
// Listed after the literal `live`, which a linear first-match scan reaches
// first — though `{id}` would refuse it anyway, since it captures a
// positive integer and nothing else.
.{ .method = .GET, .pattern = "/api/queries/{id}", .auth = .session, .policy = .read, .handler = queries.detail },
.{ .method = .GET, .pattern = "/api/stats", .auth = .session, .policy = .read, .handler = stats.totals },
.{ .method = .GET, .pattern = "/api/stats/timeseries", .auth = .session, .policy = .read, .handler = stats.timeseries },
.{ .method = .GET, .pattern = "/api/lookup", .auth = .session, .policy = .read, .handler = lookup.handle },
@@ -152,7 +156,7 @@ const std = @import("std");
const testing = std.testing;
test "the table carries every endpoint of the milestone" {
try testing.expectEqual(@as(usize, 60), table.len);
try testing.expectEqual(@as(usize, 61), table.len);
}
test "no two entries claim the same method and pattern" {

Some files were not shown because too many files have changed in this diff Show More