admin: overview redesign, device scope, one formatting contract (milestone 39)
Gates / frontend (push) Successful in 1m57s
Gates / test (push) Successful in 2m34s
Gates / test-aarch64 (push) Successful in 8m9s
Gates / package (push) Successful in 7m14s
Gates / container (push) Successful in 17s
CI / gates (push) Successful in 18m17s
Release / guard (push) Successful in 33s
Gates / test-aarch64 (push) Successful in 7m22s
Gates / container (push) Successful in 11s
Release / gates (push) Successful in 10m35s
Gates / frontend (push) Successful in 2m8s
Gates / test (push) Successful in 2m16s
Gates / package (push) Successful in 44s
Release / publish (push) Successful in 10m4s
Gates / frontend (push) Successful in 1m57s
Gates / test (push) Successful in 2m34s
Gates / test-aarch64 (push) Successful in 8m9s
Gates / package (push) Successful in 7m14s
Gates / container (push) Successful in 17s
CI / gates (push) Successful in 18m17s
Release / guard (push) Successful in 33s
Gates / test-aarch64 (push) Successful in 7m22s
Gates / container (push) Successful in 11s
Release / gates (push) Successful in 10m35s
Gates / frontend (push) Successful in 2m8s
Gates / test (push) Successful in 2m16s
Gates / package (push) Successful in 44s
Release / publish (push) Successful in 10m4s
The Overview page takes the decided visual language (specs/ui-visual-redesign.md): four centred totals with their Activity links, a smoothed area chart of total and blocked queries with point hover and a tooltip centred beside the point, a stacked client chart in eight distinct hues plus one Other band that is always a series, and a card row with the cache hit rate, the query types as a single-hue ramp ring, and the upstream breakdown. The count axis grows its margin with the widest grouped tick and draws whole-number ticks only. GET /api/overview takes a client parameter; the scoped read uses idx_query_log_ts and the cache keeps scoped slots. The device selector beside the period selector is URL state, so a scoped view is a link, and the tile links carry the scope into Activity. The route reduces a pasted IPv6 scope to the RFC 5952 spelling the logger stores, mapped addresses included, and drops anything that is not an address. A failed device list says so under the selector with a retry. All measured quantities go through admin/src/lib/format.ts: grouped counts, two-decimal percentages, one-decimal rates, durations as the two largest nonzero units. Identifiers, configured values and preset labels render as written; the module header states that scope. A sweep test refuses toFixed, toLocaleString, Intl.NumberFormat and padStart anywhere else. Chrome: one 4px radius from the metrics constants, shared Card with a prominent title and a one-line description on every panel, the settings form sections on the same card with a floated legend, the sidebar grouped into Monitoring and System with a status block (protection, queries per minute on Overview, uptime), keyboard-focusable table scroll wrappers, and the accent darkened to 5.43:1 on its wash. Not built: the spec's ranked-list primitive, which has no consumer and no API rows. Codex reviewed sessions B to D over five rounds (thirty-three findings fixed, thirteen rejected as non-quantities); the owner skipped a sixth round. Claude-Session: https://claude.ai/code/session_01VTgx3a1zz1R78o4K55kkwR
This commit is contained in:
@@ -1240,6 +1240,21 @@ const overview_raw_sql =
|
||||
\\ WHERE timestamp >= ?1 AND timestamp < ?2
|
||||
;
|
||||
|
||||
/// `INDEXED BY` because the planner, offered both indexes, walks
|
||||
/// `idx_query_log_client` — the device's whole retained history — and filters
|
||||
/// the window out of it afterwards. The window is the small side: a device is
|
||||
/// a slice of every row ever kept, and the window is a slice of hours.
|
||||
const overview_raw_client_sql =
|
||||
\\SELECT timestamp, client_ip, qtype, blocked, cache_hit, response_time_us,
|
||||
\\ route_kind,
|
||||
\\ CASE route_kind
|
||||
\\ WHEN 'upstream' THEN upstream
|
||||
\\ WHEN 'forward_zone' THEN forward_zone
|
||||
\\ END AS source
|
||||
\\ FROM query_log INDEXED BY idx_query_log_ts
|
||||
\\ WHERE timestamp >= ?1 AND timestamp < ?2 AND client_ip = ?3
|
||||
;
|
||||
|
||||
/// The whole Overview payload for `[since, since + bucket_seconds *
|
||||
/// bucket_count)`, from one already-open read transaction — the caller owns the
|
||||
/// transaction and the lock, as it does for every other read here.
|
||||
@@ -1250,6 +1265,13 @@ const overview_raw_sql =
|
||||
/// buckets) one pass over the raw rows in the window does the same work in Zig.
|
||||
/// Both produce the same contracts, which is what the equivalence test asserts.
|
||||
///
|
||||
/// A `client` scopes the whole payload to that address's rows (milestone 39).
|
||||
/// The projections aggregate across clients for everything but the per-client
|
||||
/// series, so a scoped read always takes the raw path, whatever the width: one
|
||||
/// scan of the window's rows over `idx_query_log_ts`, which household volume
|
||||
/// wears. Per-client projections would be a second copy of every table for one
|
||||
/// filter.
|
||||
///
|
||||
/// Preconditions are caller bugs, not runtime conditions: a zero width or count
|
||||
/// is `error.Misuse`, and so is a projection-path window that is not on the
|
||||
/// grid, because the projections cannot express it. The handler's `window()`
|
||||
@@ -1260,19 +1282,20 @@ pub fn overview(
|
||||
since: i64,
|
||||
bucket_seconds: u32,
|
||||
bucket_count: u32,
|
||||
client: ?[]const u8,
|
||||
) db.Error!Overview {
|
||||
if (bucket_seconds == 0 or bucket_count == 0) return error.Misuse;
|
||||
const width: i64 = bucket_seconds;
|
||||
const span = std.math.mul(i64, width, bucket_count) catch return error.Misuse;
|
||||
const until = std.math.add(i64, since, span) catch return error.Misuse;
|
||||
const from_projections = width >= grain;
|
||||
const from_projections = client == null and width >= grain;
|
||||
if (from_projections and (@mod(since, grain) != 0 or @rem(width, grain) != 0)) return error.Misuse;
|
||||
|
||||
var accum = try Accum.init(arena, since, width, bucket_count);
|
||||
if (from_projections) {
|
||||
try readProjections(database, &accum, since, until);
|
||||
} else {
|
||||
try readRaw(database, &accum, since, until);
|
||||
try readRaw(database, &accum, since, until, client);
|
||||
}
|
||||
return try accum.finish();
|
||||
}
|
||||
@@ -1327,11 +1350,12 @@ fn readProjections(database: *db.Db, accum: *Accum, since: i64, until: i64) db.E
|
||||
}
|
||||
}
|
||||
|
||||
fn readRaw(database: *db.Db, accum: *Accum, since: i64, until: i64) db.Error!void {
|
||||
var stmt = try database.prepare(overview_raw_sql);
|
||||
fn readRaw(database: *db.Db, accum: *Accum, since: i64, until: i64, client: ?[]const u8) db.Error!void {
|
||||
var stmt = try database.prepare(if (client == null) overview_raw_sql else overview_raw_client_sql);
|
||||
defer stmt.deinit();
|
||||
try stmt.bindInt(1, since);
|
||||
try stmt.bindInt(2, until);
|
||||
if (client) |address| try stmt.bindText(3, address);
|
||||
|
||||
while (try stmt.step()) {
|
||||
const index = try accum.indexOf(stmt.columnInt(0));
|
||||
@@ -1371,6 +1395,21 @@ fn openLog() !db.Db {
|
||||
return database;
|
||||
}
|
||||
|
||||
test "a scoped overview read walks the timestamp index, not the client's history" {
|
||||
var database = try openLog();
|
||||
defer database.close();
|
||||
|
||||
var stmt = try database.prepare("EXPLAIN QUERY PLAN " ++ overview_raw_client_sql);
|
||||
defer stmt.deinit();
|
||||
try stmt.bindInt(1, 0);
|
||||
try stmt.bindInt(2, 1);
|
||||
try stmt.bindText(3, "192.0.2.10");
|
||||
try testing.expect(try stmt.step());
|
||||
const detail = stmt.columnText(3);
|
||||
try testing.expect(std.mem.indexOf(u8, detail, "idx_query_log_ts") != null);
|
||||
try testing.expect(std.mem.indexOf(u8, detail, "idx_query_log_client") == null);
|
||||
}
|
||||
|
||||
fn plainRow(timestamp: i64, domain: []const u8) Row {
|
||||
return .{
|
||||
.timestamp = timestamp,
|
||||
@@ -2309,7 +2348,7 @@ test "the type breakdown groups by qtype, keeps the null row and orders it last"
|
||||
aggRow(-1, "192.0.2.10", 255, .upstream, "9.9.9.9"),
|
||||
});
|
||||
|
||||
const rows = (try overview(&database, arena, agg_since, agg_width, agg_buckets)).types;
|
||||
const rows = (try overview(&database, arena, agg_since, agg_width, agg_buckets, null)).types;
|
||||
try testing.expectEqual(@as(usize, 4), rows.len);
|
||||
try testing.expectEqual(@as(?u16, 1), rows[0].qtype);
|
||||
try testing.expectEqual(@as(u64, 3), rows[0].count);
|
||||
@@ -2325,7 +2364,7 @@ test "an empty window has no type rows at all" {
|
||||
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||
defer arena_state.deinit();
|
||||
|
||||
const rows = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).types;
|
||||
const rows = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets, null)).types;
|
||||
try testing.expectEqual(@as(usize, 0), rows.len);
|
||||
}
|
||||
|
||||
@@ -2349,7 +2388,7 @@ test "the route breakdown keys on the answering resolver, not on blocklist prove
|
||||
aggRow(8, "192.0.2.10", 1, .rejected, null),
|
||||
});
|
||||
|
||||
const rows = (try overview(&database, arena, agg_since, agg_width, agg_buckets)).routes;
|
||||
const rows = (try overview(&database, arena, agg_since, agg_width, agg_buckets, null)).routes;
|
||||
// Two upstreams, one null-source upstream, one forward zone and four
|
||||
// source-less kinds. Every row carries the same `source_name`, so a
|
||||
// breakdown that grouped by it would collapse to one row.
|
||||
@@ -2381,7 +2420,7 @@ test "an empty window has no route rows at all" {
|
||||
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||
defer arena_state.deinit();
|
||||
|
||||
const rows = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).routes;
|
||||
const rows = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets, null)).routes;
|
||||
try testing.expectEqual(@as(usize, 0), rows.len);
|
||||
}
|
||||
|
||||
@@ -2391,7 +2430,7 @@ test "an empty window still has a zero-filled other series and no named clients"
|
||||
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
|
||||
defer arena_state.deinit();
|
||||
|
||||
const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).clients;
|
||||
const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets, null)).clients;
|
||||
try testing.expectEqual(@as(usize, 0), result.clients.len);
|
||||
try testing.expectEqual(@as(usize, agg_buckets), result.other.len);
|
||||
for (result.other) |count| try testing.expectEqual(@as(u64, 0), count);
|
||||
@@ -2413,7 +2452,7 @@ test "client series are bucket-aligned, zero-filled and ranked by in-window tota
|
||||
aggRow(agg_width * agg_buckets, "192.0.2.10", 1, .upstream, "9.9.9.9"),
|
||||
});
|
||||
|
||||
const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).clients;
|
||||
const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets, null)).clients;
|
||||
try testing.expectEqual(@as(usize, 2), result.clients.len);
|
||||
try testing.expectEqualStrings("192.0.2.20", result.clients[0].client);
|
||||
try testing.expectEqualStrings("192.0.2.10", result.clients[1].client);
|
||||
@@ -2445,7 +2484,7 @@ test "the ninth client folds into other and the cut is the same on every read" {
|
||||
}
|
||||
}
|
||||
|
||||
const result = (try overview(&database, arena, agg_since, agg_width, agg_buckets)).clients;
|
||||
const result = (try overview(&database, arena, agg_since, agg_width, agg_buckets, null)).clients;
|
||||
try testing.expectEqual(@as(usize, max_client_series), result.clients.len);
|
||||
// The busiest is 192.0.2.108 with nine rows; the lone folded client is
|
||||
// 192.0.2.100 with one.
|
||||
@@ -2484,7 +2523,7 @@ test "the three breakdowns conserve the window's total" {
|
||||
)});
|
||||
}
|
||||
|
||||
const result = try overview(&database, arena, agg_since, agg_width, agg_buckets);
|
||||
const result = try overview(&database, arena, agg_since, agg_width, agg_buckets, null);
|
||||
try testing.expect(result.totals.queries > 0);
|
||||
|
||||
var typed: u64 = 0;
|
||||
@@ -2518,7 +2557,7 @@ test "the aggregations pass a redacted client through as the log stored it" {
|
||||
aggRow(1, logger.hidden_marker, 1, .upstream, "9.9.9.9"),
|
||||
});
|
||||
|
||||
const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets)).clients;
|
||||
const result = (try overview(&database, arena_state.allocator(), agg_since, agg_width, agg_buckets, null)).clients;
|
||||
try testing.expectEqual(@as(usize, 1), result.clients.len);
|
||||
try testing.expectEqualStrings(logger.hidden_marker, result.clients[0].client);
|
||||
try testing.expectEqual(@as(u64, 2), result.clients[0].buckets[0]);
|
||||
@@ -2556,7 +2595,7 @@ test "a prune committed mid-read is invisible to the reader's transaction" {
|
||||
const arena = arena_state.allocator();
|
||||
|
||||
var tx = try db.ReadTx.begin(&reader);
|
||||
const before = (try overview(&reader, arena, agg_since, 1000, 1)).totals;
|
||||
const before = (try overview(&reader, arena, agg_since, 1000, 1, null)).totals;
|
||||
const watermark_before = try availableSince(&reader);
|
||||
try testing.expectEqual(@as(u64, 3), before.queries);
|
||||
|
||||
@@ -2567,13 +2606,13 @@ test "a prune committed mid-read is invisible to the reader's transaction" {
|
||||
|
||||
// Neither half of the answer moved: the rows the reader would report and
|
||||
// the watermark it would tag them with still describe one state.
|
||||
const during = (try overview(&reader, arena, agg_since, 1000, 1)).totals;
|
||||
const during = (try overview(&reader, arena, agg_since, 1000, 1, null)).totals;
|
||||
try testing.expectEqual(before.queries, during.queries);
|
||||
try testing.expectEqual(watermark_before, try availableSince(&reader));
|
||||
try tx.commit();
|
||||
|
||||
// The next response sees the prune — both halves of it.
|
||||
const after = (try overview(&reader, arena, agg_since, 1000, 1)).totals;
|
||||
const after = (try overview(&reader, arena, agg_since, 1000, 1, null)).totals;
|
||||
try testing.expectEqual(@as(u64, 1), after.queries);
|
||||
try testing.expectEqual(pruned.available_since, try availableSince(&reader));
|
||||
}
|
||||
@@ -3127,7 +3166,7 @@ fn expectOverviewMatchesOracle(database: *db.Db, since: i64) !void {
|
||||
|
||||
for (overview_windows) |window| {
|
||||
const want = try oracle.build(database, arena, since, window.bucket_seconds, window.bucket_count);
|
||||
const got = try overview(database, arena, since, window.bucket_seconds, window.bucket_count);
|
||||
const got = try overview(database, arena, since, window.bucket_seconds, window.bucket_count, null);
|
||||
expectOverviewEqual(want, got) catch |err| {
|
||||
std.debug.print("overview mismatch at bucket_seconds={d}\n", .{window.bucket_seconds});
|
||||
return err;
|
||||
@@ -3239,16 +3278,16 @@ test "overview refuses a window the projections cannot express" {
|
||||
|
||||
const since = bucketOf(agg_since);
|
||||
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since, 0, 48));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since, 1800, 0));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, std.math.maxInt(i64) - 1, 3600, 48));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since, 0, 48, null));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since, 1800, 0, null));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, std.math.maxInt(i64) - 1, 3600, 48, null));
|
||||
|
||||
// On the projection path the grid is part of the contract: an unaligned
|
||||
// start or a width that is not a whole number of grains cannot be answered
|
||||
// from 30-minute rows, and guessing would be worse than refusing.
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since + 1, 1800, 48));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since, 2700, 48));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since + 1, 1800, 48, null));
|
||||
try testing.expectError(error.Misuse, overview(&database, arena, since, 2700, 48, null));
|
||||
|
||||
// Below the grain none of that applies: the raw rows carry every second.
|
||||
_ = try overview(&database, arena, since + 1, 60, 60);
|
||||
_ = try overview(&database, arena, since + 1, 60, 60, null);
|
||||
}
|
||||
|
||||
@@ -31,6 +31,7 @@ const std = @import("std");
|
||||
const coverage = @import("../coverage.zig");
|
||||
const db = @import("../../storage/db.zig");
|
||||
const http_util = @import("../http_util.zig");
|
||||
const logger = @import("../../storage/logger.zig");
|
||||
const queries_repo = @import("../../storage/repositories/queries_repo.zig");
|
||||
const server = @import("../server.zig");
|
||||
|
||||
@@ -139,10 +140,12 @@ pub fn handle(
|
||||
request: *http_util.Request,
|
||||
) http_util.HandlerError!void {
|
||||
const period = periodParam(request.query) catch return badPeriod(request);
|
||||
var client_buf: [max_client_value_len]u8 = undefined;
|
||||
const client = clientParam(request.query, &client_buf) catch return badClient(request);
|
||||
const database = state.querylog_db orelse return unavailable(request);
|
||||
const span = window(period, std.Io.Clock.real.now(io).toSeconds());
|
||||
|
||||
const body = cachedBody(state, io, database, request.arena, period, span) catch |err| {
|
||||
const body = cachedBody(state, io, database, request.arena, period, span, client) catch |err| {
|
||||
return internal(request, err);
|
||||
};
|
||||
return http_util.respondBytes(request, .ok, body, http_util.content_type_json, &.{});
|
||||
@@ -165,6 +168,7 @@ fn cachedBody(
|
||||
arena: std.mem.Allocator,
|
||||
period: Period,
|
||||
span: Window,
|
||||
client: ?[]const u8,
|
||||
) db.Error![]const u8 {
|
||||
state.querylog_lock.lockUncancelable(io);
|
||||
defer state.querylog_lock.unlock(io);
|
||||
@@ -172,16 +176,16 @@ fn cachedBody(
|
||||
const index = @intFromEnum(period);
|
||||
const data_version = try database.queryInt("PRAGMA data_version");
|
||||
|
||||
if (state.overview_cache.get(index, span.until, data_version)) |cached| {
|
||||
if (state.overview_cache.get(index, span.until, data_version, client)) |cached| {
|
||||
// The copy is what makes a later rebuild's free-and-replace safe: the
|
||||
// response is written after the lock is gone, and by then these bytes
|
||||
// may belong to nobody.
|
||||
return arena.dupe(u8, cached);
|
||||
}
|
||||
|
||||
const body = try buildBody(state, io, database, arena, period, span);
|
||||
const body = try buildBody(state, io, database, arena, period, span, client);
|
||||
const owned = try state.gpa.dupe(u8, body);
|
||||
state.overview_cache.put(state.gpa, index, span.until, data_version, owned);
|
||||
try state.overview_cache.put(state.gpa, index, span.until, data_version, client, owned);
|
||||
return body;
|
||||
}
|
||||
|
||||
@@ -194,6 +198,7 @@ fn buildBody(
|
||||
arena: std.mem.Allocator,
|
||||
period: Period,
|
||||
span: Window,
|
||||
client: ?[]const u8,
|
||||
) db.Error![]const u8 {
|
||||
var scope = try server.QuerylogRead.openLocked(state, io, database);
|
||||
errdefer scope.abort();
|
||||
@@ -203,6 +208,7 @@ fn buildBody(
|
||||
span.since,
|
||||
span.bucket_seconds,
|
||||
span.bucket_count,
|
||||
client,
|
||||
);
|
||||
const window_coverage = try coverage.read(database, span.since);
|
||||
try scope.commit();
|
||||
@@ -245,6 +251,29 @@ fn badPeriod(request: *http_util.Request) http_util.HandlerError!void {
|
||||
return http_util.respondError(request, .bad_request, "period must be one of 1h, 24h, 7d, 30d");
|
||||
}
|
||||
|
||||
pub const ClientError = error{BadClient};
|
||||
|
||||
/// `queryValue` measures the value before it decodes it, and a browser
|
||||
/// percent-encodes every colon of an IPv6 address: three bytes on the wire
|
||||
/// per byte of address.
|
||||
const max_client_value_len = 3 * logger.max_client_len;
|
||||
|
||||
/// One exact client address, or nothing. An absent or empty value is the
|
||||
/// whole household, as the query-log filter reads it; the length bound is that
|
||||
/// filter's too. A comma is a list, and the overview scopes to one device.
|
||||
fn clientParam(query: []const u8, buf: *[max_client_value_len]u8) ClientError!?[]const u8 {
|
||||
const found = http_util.queryValue(query, "client", buf) catch return error.BadClient;
|
||||
const text = found orelse return null;
|
||||
if (text.len == 0) return null;
|
||||
if (text.len > logger.max_client_len) return error.BadClient;
|
||||
if (std.mem.indexOfScalar(u8, text, ',') != null) return error.BadClient;
|
||||
return text;
|
||||
}
|
||||
|
||||
fn badClient(request: *http_util.Request) http_util.HandlerError!void {
|
||||
return http_util.respondError(request, .bad_request, "client must be a single client address");
|
||||
}
|
||||
|
||||
fn unavailable(request: *http_util.Request) http_util.HandlerError!void {
|
||||
return http_util.respondError(request, .service_unavailable, "query log unavailable");
|
||||
}
|
||||
@@ -284,6 +313,23 @@ test "an absent period defaults and a bad one is rejected" {
|
||||
try testing.expectError(error.BadPeriod, periodParam("period=1hhhhhhhhhh"));
|
||||
}
|
||||
|
||||
test "an absent client is the whole household and a bad one is rejected" {
|
||||
var buf: [max_client_value_len]u8 = undefined;
|
||||
try testing.expectEqual(@as(?[]const u8, null), try clientParam("period=1h", &buf));
|
||||
try testing.expectEqualStrings("192.0.2.10", (try clientParam("client=192.0.2.10", &buf)).?);
|
||||
try testing.expectEqual(@as(?[]const u8, null), try clientParam("client=", &buf));
|
||||
try testing.expectError(error.BadClient, clientParam("client=192.0.2.10,192.0.2.11", &buf));
|
||||
try testing.expectError(error.BadClient, clientParam("client=%2", &buf));
|
||||
// A browser encodes the colons; the decoded address is what the bound
|
||||
// applies to.
|
||||
try testing.expectEqualStrings(
|
||||
"2001:db8:85a3:8d3:1319:8a2e:370:7348",
|
||||
(try clientParam("client=2001%3Adb8%3A85a3%3A8d3%3A1319%3A8a2e%3A370%3A7348", &buf)).?,
|
||||
);
|
||||
// Longer than any address: rejected rather than truncated.
|
||||
try testing.expectError(error.BadClient, clientParam("client=" ++ "1" ** 46, &buf));
|
||||
}
|
||||
|
||||
test "each period spans its own bucket width times its count" {
|
||||
for (std.enums.values(Period)) |period| {
|
||||
const span = window(period, 1_700_000_000);
|
||||
@@ -391,6 +437,7 @@ test "one overview answers totals and buckets that agree over the same window" {
|
||||
span.since,
|
||||
span.bucket_seconds,
|
||||
span.bucket_count,
|
||||
null,
|
||||
);
|
||||
|
||||
try testing.expectEqual(@as(u64, 3), data.totals.queries);
|
||||
@@ -428,6 +475,7 @@ test "an empty window reports zeros with a null mean" {
|
||||
span.since,
|
||||
span.bucket_seconds,
|
||||
span.bucket_count,
|
||||
null,
|
||||
);
|
||||
|
||||
try testing.expectEqual(@as(u64, 0), data.totals.queries);
|
||||
@@ -444,20 +492,37 @@ test "the cache serves one period's bytes and rebuilds when the key moves" {
|
||||
var cache: server.OverviewCache = .{};
|
||||
defer cache.deinit(gpa);
|
||||
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 7));
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 7, null));
|
||||
|
||||
cache.put(gpa, 0, 100, 7, try gpa.dupe(u8, "first"));
|
||||
try testing.expectEqualStrings("first", cache.get(0, 100, 7).?);
|
||||
try cache.put(gpa, 0, 100, 7, null, try gpa.dupe(u8, "first"));
|
||||
try testing.expectEqualStrings("first", cache.get(0, 100, 7, null).?);
|
||||
// A different period, a rolled window and a bumped data version are three
|
||||
// different keys, and none of them hits.
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(1, 100, 7));
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 101, 7));
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 8));
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(1, 100, 7, null));
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 101, 7, null));
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 8, null));
|
||||
|
||||
// A rebuild replaces the entry and frees the old body; the leak checker in
|
||||
// `testing.allocator` is the assertion.
|
||||
cache.put(gpa, 0, 100, 8, try gpa.dupe(u8, "second"));
|
||||
try testing.expectEqualStrings("second", cache.get(0, 100, 8).?);
|
||||
try cache.put(gpa, 0, 100, 8, null, try gpa.dupe(u8, "second"));
|
||||
try testing.expectEqualStrings("second", cache.get(0, 100, 8, null).?);
|
||||
}
|
||||
|
||||
test "a scoped body has its own slot, keyed on the client too" {
|
||||
const gpa = testing.allocator;
|
||||
var cache: server.OverviewCache = .{};
|
||||
defer cache.deinit(gpa);
|
||||
|
||||
try cache.put(gpa, 0, 100, 7, null, try gpa.dupe(u8, "household"));
|
||||
try cache.put(gpa, 0, 100, 7, "192.0.2.10", try gpa.dupe(u8, "one device"));
|
||||
// Neither evicts the other.
|
||||
try testing.expectEqualStrings("household", cache.get(0, 100, 7, null).?);
|
||||
try testing.expectEqualStrings("one device", cache.get(0, 100, 7, "192.0.2.10").?);
|
||||
// Another client is a miss, and its rebuild takes the scoped slot over.
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 7, "192.0.2.11"));
|
||||
try cache.put(gpa, 0, 100, 7, "192.0.2.11", try gpa.dupe(u8, "another"));
|
||||
try testing.expectEqual(@as(?[]const u8, null), cache.get(0, 100, 7, "192.0.2.10"));
|
||||
try testing.expectEqualStrings("household", cache.get(0, 100, 7, null).?);
|
||||
}
|
||||
|
||||
/// Two connections onto one file, which is the only arrangement in which
|
||||
@@ -528,6 +593,7 @@ const CacheFixture = struct {
|
||||
self.arena_state.allocator(),
|
||||
period,
|
||||
span,
|
||||
null,
|
||||
);
|
||||
}
|
||||
|
||||
|
||||
@@ -438,8 +438,18 @@ paths:
|
||||
Buckets are UTC-aligned and zero-filled: 1h into 60 one-minute buckets,
|
||||
24h into 48 half-hour buckets, 7d into 168 one-hour buckets, 30d into
|
||||
120 six-hour buckets. The last bucket is the one in progress.
|
||||
|
||||
With `client`, every field describes that one device's rows: `clients`
|
||||
carries at most that device and `other` is present and all zero. A
|
||||
device the log never saw answers zeros, not 404.
|
||||
parameters:
|
||||
- $ref: "#/components/parameters/Period"
|
||||
- name: client
|
||||
in: query
|
||||
description: Scope every figure to one exact client address. Absent or empty is the whole household.
|
||||
schema:
|
||||
type: string
|
||||
maxLength: 45
|
||||
responses:
|
||||
"200":
|
||||
description: The period's overview.
|
||||
|
||||
+33
-6
@@ -199,38 +199,65 @@ pub const OverviewCache = struct {
|
||||
body: []u8 = &.{},
|
||||
until: i64 = 0,
|
||||
data_version: i64 = 0,
|
||||
/// The client the body is scoped to; empty for the whole household.
|
||||
client: []u8 = &.{},
|
||||
};
|
||||
|
||||
slots: [slot_count]Slot = @splat(.{}),
|
||||
/// One scoped body per period, beside the household one: a device pick
|
||||
/// must not evict the household body the next unscoped request wants.
|
||||
/// Only the latest client per period is kept — the picker moves one
|
||||
/// device at a time.
|
||||
scoped: [slot_count]Slot = @splat(.{}),
|
||||
|
||||
/// The stored bytes for this key, or null. The caller copies them into its
|
||||
/// request arena before releasing the lock: a later rebuild frees this
|
||||
/// allocation.
|
||||
pub fn get(self: *const OverviewCache, period_index: usize, until: i64, data_version: i64) ?[]const u8 {
|
||||
const slot = &self.slots[period_index];
|
||||
pub fn get(
|
||||
self: *const OverviewCache,
|
||||
period_index: usize,
|
||||
until: i64,
|
||||
data_version: i64,
|
||||
client: ?[]const u8,
|
||||
) ?[]const u8 {
|
||||
const slot = if (client == null) &self.slots[period_index] else &self.scoped[period_index];
|
||||
if (slot.body.len == 0) return null;
|
||||
if (slot.until != until or slot.data_version != data_version) return null;
|
||||
if (!std.mem.eql(u8, slot.client, client orelse "")) return null;
|
||||
return slot.body;
|
||||
}
|
||||
|
||||
/// Takes ownership of `body`, which must be a `gpa` allocation, and frees
|
||||
/// whatever the slot held.
|
||||
/// whatever the slot held. On allocation failure the body is freed and the
|
||||
/// slot keeps its previous entry.
|
||||
pub fn put(
|
||||
self: *OverviewCache,
|
||||
gpa: Allocator,
|
||||
period_index: usize,
|
||||
until: i64,
|
||||
data_version: i64,
|
||||
client: ?[]const u8,
|
||||
body: []u8,
|
||||
) void {
|
||||
const slot = &self.slots[period_index];
|
||||
) Allocator.Error!void {
|
||||
const owned_client = gpa.dupe(u8, client orelse "") catch |err| {
|
||||
gpa.free(body);
|
||||
return err;
|
||||
};
|
||||
const slot = if (client == null) &self.slots[period_index] else &self.scoped[period_index];
|
||||
gpa.free(slot.body);
|
||||
slot.* = .{ .body = body, .until = until, .data_version = data_version };
|
||||
gpa.free(slot.client);
|
||||
slot.* = .{ .body = body, .until = until, .data_version = data_version, .client = owned_client };
|
||||
}
|
||||
|
||||
pub fn deinit(self: *OverviewCache, gpa: Allocator) void {
|
||||
for (&self.slots) |*slot| {
|
||||
gpa.free(slot.body);
|
||||
gpa.free(slot.client);
|
||||
slot.* = .{};
|
||||
}
|
||||
for (&self.scoped) |*slot| {
|
||||
gpa.free(slot.body);
|
||||
gpa.free(slot.client);
|
||||
slot.* = .{};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -2810,6 +2810,94 @@ test "W10 milestone 30: the three breakdowns conserve the totals over one window
|
||||
try bounded(env.io(), default_budget, populatedAggregations, .{ env.io(), env });
|
||||
}
|
||||
|
||||
/// Milestone 39: `client` narrows every figure to one device, over the same
|
||||
/// seeded matrix `populatedAggregations` reads unscoped.
|
||||
fn scopedAggregations(io: std.Io, env: *Env) anyerror!void {
|
||||
var arena_state: std.heap.ArenaAllocator = .init(env.gpa);
|
||||
defer arena_state.deinit();
|
||||
const arena = arena_state.allocator();
|
||||
|
||||
var conn: Conn = undefined;
|
||||
try conn.connect(io, env.addr);
|
||||
defer conn.close(io);
|
||||
|
||||
var body_buf: [256 * 1024]u8 = undefined;
|
||||
|
||||
const household = try getJson(handlers_overview.Body, arena, &conn, "/api/overview?period=1h", &body_buf);
|
||||
|
||||
// The three device slices sum to the household on every figure a tile or
|
||||
// a breakdown draws, so scoped and unscoped describe the same rows.
|
||||
var sliced_queries: u64 = 0;
|
||||
var sliced_blocked: u64 = 0;
|
||||
var sliced_typed: u64 = 0;
|
||||
var sliced_routed: u64 = 0;
|
||||
for ([_][]const u8{ "192.0.2.30", "192.0.2.31", "192.0.2.32" }) |device| {
|
||||
const path = try std.fmt.allocPrint(arena, "/api/overview?period=1h&client={s}", .{device});
|
||||
const body = try getJson(handlers_overview.Body, arena, &conn, path, &body_buf);
|
||||
sliced_queries += body.totals.queries;
|
||||
sliced_blocked += body.totals.blocked;
|
||||
try testing.expectEqual(@as(u64, 1), body.totals.clients);
|
||||
for (body.types) |row| sliced_typed += row.count;
|
||||
for (body.routes) |row| sliced_routed += row.count;
|
||||
|
||||
try testing.expectEqual(@as(usize, 1), body.clients.len);
|
||||
try testing.expectEqualStrings(device, body.clients[0].client);
|
||||
try testing.expectEqual(body.buckets.len, body.other.len);
|
||||
var bucketed: u64 = 0;
|
||||
for (body.buckets, 0..) |bucket, at| {
|
||||
bucketed += bucket.queries;
|
||||
try testing.expectEqual(@as(u64, 0), body.other[at]);
|
||||
try testing.expectEqual(bucket.queries, body.clients[0].buckets[at]);
|
||||
}
|
||||
try testing.expectEqual(body.totals.queries, bucketed);
|
||||
try testing.expectEqualStrings("1h", body.period);
|
||||
try testing.expectEqual(household.coverage.available_since, body.coverage.available_since);
|
||||
}
|
||||
try testing.expectEqual(household.totals.queries, sliced_queries);
|
||||
try testing.expectEqual(household.totals.blocked, sliced_blocked);
|
||||
try testing.expectEqual(household.totals.queries, sliced_typed);
|
||||
try testing.expectEqual(household.totals.queries, sliced_routed);
|
||||
|
||||
// The exact slices, pinned: the seed writes four rows for .30 and three
|
||||
// for .31 with one of them blocked. The wide period takes the raw path
|
||||
// too, and reads the same rows.
|
||||
const thirty = try getJson(handlers_overview.Body, arena, &conn, "/api/overview?period=1h&client=192.0.2.30", &body_buf);
|
||||
try testing.expectEqual(@as(u64, 4), thirty.totals.queries);
|
||||
try testing.expectEqual(@as(u64, 0), thirty.totals.blocked);
|
||||
const thirty_one = try getJson(handlers_overview.Body, arena, &conn, "/api/overview?period=30d&client=192.0.2.31", &body_buf);
|
||||
try testing.expectEqual(@as(u64, 3), thirty_one.totals.queries);
|
||||
try testing.expectEqual(@as(u64, 1), thirty_one.totals.blocked);
|
||||
try testing.expectEqual(@as(usize, 120), thirty_one.buckets.len);
|
||||
|
||||
// A device the log never saw is zeros with the full shape, not a 404: the
|
||||
// picker may still name a client whose rows retention has since pruned.
|
||||
const unknown = try getJson(handlers_overview.Body, arena, &conn, "/api/overview?period=1h&client=192.0.2.99", &body_buf);
|
||||
try testing.expectEqual(@as(u64, 0), unknown.totals.queries);
|
||||
try testing.expectEqual(@as(u64, 0), unknown.totals.clients);
|
||||
try testing.expectEqual(@as(usize, 0), unknown.clients.len);
|
||||
try testing.expectEqual(@as(usize, 60), unknown.other.len);
|
||||
for (unknown.other) |count| try testing.expectEqual(@as(u64, 0), count);
|
||||
|
||||
// An empty value is the household, like the query-log filter; a list is not
|
||||
// a device.
|
||||
const empty = try getJson(handlers_overview.Body, arena, &conn, "/api/overview?period=1h&client=", &body_buf);
|
||||
try testing.expectEqual(household.totals.queries, empty.totals.queries);
|
||||
try conn.request("GET", "/api/overview?period=1h&client=192.0.2.30,192.0.2.31", null, null);
|
||||
const bad = try conn.receive(&body_buf);
|
||||
try testing.expectEqual(@as(u16, 400), bad.status);
|
||||
try testing.expect(std.mem.containsAtLeast(u8, bad.body, 1, "client must be a single client address"));
|
||||
}
|
||||
|
||||
test "W10 milestone 39: a client scope narrows every figure and its slices sum to the household" {
|
||||
if (!build_options.integration) return error.SkipZigTest;
|
||||
|
||||
const gpa = testing.allocator;
|
||||
var env = try Env.create(gpa, .{ .recent_traffic = true });
|
||||
defer env.destroy();
|
||||
|
||||
try bounded(env.io(), default_budget, scopedAggregations, .{ env.io(), env });
|
||||
}
|
||||
|
||||
/// One connection walking every query-log endpoint several times over.
|
||||
fn hammerQuerylog(io: std.Io, env: *Env) anyerror!void {
|
||||
var conn: Conn = undefined;
|
||||
|
||||
Reference in New Issue
Block a user