milestone 8: web server, rest api, sse, auth, metrics and static assets
This commit is contained in:
@@ -0,0 +1,856 @@
|
||||
# Milestone 8: web server, REST API, SSE, auth, metrics (PLAN Phase 8, Zig side)
|
||||
|
||||
Goal: the complete Zig web layer — HTTP server + router, every REST handler, SSE live query
|
||||
stream, optional argon2id auth with sessions, API token-bucket rate limiting, `/metrics`,
|
||||
`/api/health`, OpenAPI served + contract tests, asset embedding + dev-mode disk serving —
|
||||
running as one more task in app.zig's group.
|
||||
|
||||
Ground truth: PLAN.md:610-612 (phase text), :537-550 (endpoints), §3.11/§12.1/§19 (auth),
|
||||
§10:333 (API limiting), §11.4:455 (SSE precedes persistence), §13.2 (OpenAPI), scratchpad
|
||||
notes m8-explore-{plan,repo,zig}.md. Load-bearing Zig facts are restated inline; verify
|
||||
anything else against /home/mokhtar/app/zig tag 0.16.0.
|
||||
|
||||
## Rulings (binding)
|
||||
|
||||
1. **Sequencing**: this milestone is the Zig web layer end to end. The React SPA (PLAN §3.14,
|
||||
§14 — ten pages) is milestone 9, built against this milestone's finished, contract-tested
|
||||
API. Not a scope cut: the asset embedding, dev-mode disk serving, ETag and gzip machinery
|
||||
all ship NOW and are complete; milestone 9 only swaps the dist content. The embedded dist
|
||||
in this milestone is a minimal real page (see ruling 24) — machinery is not stubbed.
|
||||
2. **`POST /api/certs/reload` is Phase 9's**, whole: endpoint, openapi.yaml entry and docs
|
||||
land together with the DoH/DoT server it reloads. No 501 stub, no dangling contract entry.
|
||||
3. **`docs/api/` rendering is Phase 10** (the docs phase). Phase 8 serves the yaml.
|
||||
4. **Server model**: per-connection `std.http.Server` (Server.zig:25) over our own accept
|
||||
loop, copied from lib/std/Build/WebServer.zig:152-185: listener task in the app group; an
|
||||
inner `Io.Group` of connection tasks; keep-alive loop on `receiveHead` with
|
||||
`error.HttpConnectionClosing => return`. Cancel semantics mirror tcp_server's S3 As-built:
|
||||
on cancellation the inner group is CANCELED, not awaited (a keep-alive client must not
|
||||
hold shutdown open); on listener close (`SocketNotListening`) it drains.
|
||||
5. **Bind**: one listener socket on `web.bind:web.port` exactly as configured (default
|
||||
0.0.0.0:8080). No dual-stack ceremony — milestone 7's parity ruling was about DNS client
|
||||
identity, not the admin UI. An operator who wants v6 sets `web.bind = "::"` (dual-stack by
|
||||
Linux default).
|
||||
6. **`web.enabled = false`** skips the entire subsystem, `/metrics` included. No web task, no
|
||||
web DB connections.
|
||||
7. **Limits**: recv buffer 8 KiB (this caps the request head — Server.zig:32); send buffer
|
||||
4 KiB; request body cap 1 MiB (`Io.Limit`, read via `readerExpectNone`/`allocRemaining`);
|
||||
64 concurrent connections (accept beyond that: respond 503 and close — never silently
|
||||
drop). No per-request timeout this milestone (LAN-facing; the cancel path bounds
|
||||
shutdown); documented in server.zig.
|
||||
8. **JSON shape**: snake_case field names everywhere (matches settings keys and SQL). Error
|
||||
envelope `{"error":"<message>"}`. Status codes: 400 validation, 401 unauthenticated,
|
||||
404 missing, 405 wrong method (with Allow), 409 conflict (duplicate key), 413 body too
|
||||
large, 429 rate limited (with Retry-After), 500 internal (generic message, detail to log
|
||||
as warn), 503 over connection cap.
|
||||
9. **Resource paths** (freezing PLAN:541's ellipses): collections `GET`+`POST` and items
|
||||
`GET`+`PUT`+`DELETE` by numeric row id for: `/api/groups[/{id}]`,
|
||||
`/api/blocklists[/{id}]` (the sources table), `/api/rules[/{id}]`,
|
||||
`/api/local-records[/{id}]`, `/api/forward-zones[/{id}]`, `/api/upstreams[/{id}]`
|
||||
(PLAN's list omitted upstreams; the Settings page must edit them; new resource, same
|
||||
pattern). Group-source assignment: `PUT /api/groups/{id}/sources` with `{"source_ids":
|
||||
[..]}` — idempotent full-set replace. Clients: `GET /api/clients` (ALL rows, materialized
|
||||
included, each with `hand_edited`), `GET/PUT/DELETE /api/clients/{id}`; PUT sets
|
||||
`hand_edited=1` and may change name/group; DELETE removes the row (a live client
|
||||
re-materializes). Client prefixes: `GET/PUT /api/client-prefixes` as a whole-list
|
||||
resource (tiny table, atomic replace). No POST for clients — creation is by DNS activity
|
||||
or import (PLAN:540 gives clients no POST deliberately).
|
||||
10. **Repo layer**: every list row the API serves carries its row id; each mutated resource
|
||||
gains `getX(db, id)`, `updateX(db, id, item)`, `deleteX(db, id)` (strict: 0 rows touched
|
||||
→ error.NotFound), written in the house repo idiom with prepared statements. Existing
|
||||
import-path functions stay frozen.
|
||||
11. **`GET /api/queries`**: keyset pagination `?limit` (default 100, max 1000) +
|
||||
`?before=<row id>`, ordered id DESC; filters `domain=` (substring), `client=` (exact),
|
||||
`blocked=` (bool), `since=`/`until=` (unix seconds). Response `{"queries":[...],
|
||||
"next_before": <id>|null}`. Row fields: id, ts, domain, client_ip, qtype, blocked,
|
||||
block_reason, response_time_us, cache_hit, upstream.
|
||||
12. **Live vs restart**: mutations to rules, blocklists, groups, group-sources, clients and
|
||||
client-prefixes call `Manager.reload(io)` and take effect live (the handler's next
|
||||
`acquire` sees the new snapshot). Local records and forward zones ALSO apply live: a new
|
||||
`src/server/local_tables.zig` RCU holder (acquire/release + swap, mirroring the manager's
|
||||
pattern at small scale) owns `{records, zones}`; `Handler` reads through it instead of
|
||||
`*const` fields; the local-records/forward-zones handlers rebuild and swap. Upstream
|
||||
mutations and everything in `/api/settings` are restart-required. `POST
|
||||
/api/blocklists/update` = `Manager.refreshAll` then reload, 202 with the status snapshot.
|
||||
13. **Stats grammar**: `period=1h|24h|7d|30d` (else 400). `/api/stats` returns totals
|
||||
{queries, blocked, cached, clients (distinct), avg_response_time_us} for the period.
|
||||
`/api/stats/timeseries` buckets: 1h→60×1m, 24h→48×30m, 7d→168×1h, 30d→120×6h; UTC; each
|
||||
bucket {ts, queries, blocked, cached}. SQL aggregates over query_log on the web task's
|
||||
own read connection.
|
||||
14. **`GET /api/lookup?domain=&group_id=`** (group optional, default group when absent):
|
||||
normalizes, then reports the full pipeline view: `{domain, group_id, local_records:
|
||||
bool, forward_zone: <zone>|null, blocked, reason, matched, source_url|null,
|
||||
safe_search_rewrite: <target>|null}` via `snapshot.evaluate` + records/zones lookups.
|
||||
Null snapshot → 503 `{"error":"no snapshot loaded"}`.
|
||||
15. **Pause API**: `GET /api/pause` → `{"paused": bool, "until": <unix s>|null}` (null while
|
||||
unpaused OR indefinite — disambiguated by `paused`). `POST /api/pause` body
|
||||
`{"paused": true, "duration_seconds": <u32, optional>}` or `{"paused": false}`.
|
||||
16. **`GET/PUT /api/settings`**: the typed config sections that live in settings rows (dns,
|
||||
blocking, cache, edns, upstream, logging, disk, blocklist_update, safe_search flag home
|
||||
— read model.toSettings for the exact key list), minus `web.password*` (never
|
||||
serialized; a PUT carrying `web.password` re-hashes via the import path's argon2id
|
||||
params and stores only the hash). GET marks every key `restart_required: true` except
|
||||
none — ALL settings are restart-required this milestone (live behavior comes from the
|
||||
resource endpoints and pause; a static table in the handler carries the flag so the UI
|
||||
banner has transport). PUT validates the merged config through `validate.validate`
|
||||
before writing any row; partial updates allowed.
|
||||
17. **Auth**: enabled iff `web.password_hash != ""`. Login verifies with
|
||||
`std.crypto.pwhash.argon2.strVerify(hash, password, .{ .allocator = gpa }, io)`
|
||||
(argon2.zig:619 — PHC string carries its params). Sessions: in-memory fixed table of 32
|
||||
(LRU evict), storing SHA-256 of a 32-byte `io.randomSecure` token; lookup compares
|
||||
digests with `std.crypto.timing_safe.eql([32]u8, ...)` (slices not accepted —
|
||||
timing_safe.zig:12). Cookie `nxdns_session=<url_safe_no_pad base64>`; `HttpOnly;
|
||||
SameSite=Lax; Path=/`; `Secure` NOT set (nxdns serves plain HTTP; TLS termination is the
|
||||
operator's proxy — documented). Expiry `web.session_ttl_hours`. Logout deletes the
|
||||
session. A `PUT /api/settings` that changes `password_hash` clears every session.
|
||||
18. **Auth exemptions**: `/api/health`, `/api/version`, `/metrics`, `/api/openapi.yaml`, and
|
||||
the static assets are always unauthenticated (monitoring endpoints; the SPA shell must
|
||||
load to show a login form). Everything else 401s without a valid session when auth is
|
||||
enabled. `/api/auth/login` is necessarily exempt; failed logins count and are rate
|
||||
limited like any request.
|
||||
19. **API rate limiting**: token bucket per client IP (NOT the DNS fixed-window limiter —
|
||||
m6 ruling 8 reserved the bucket shape for the API): capacity and refill
|
||||
`web.api_rate_limit_per_min` per 60 s, 4096 tracked IPs (same fixed-table idiom as
|
||||
rate_limiter/tracker). Localhost exemption: NEW config field `web.api_localhost_exempt:
|
||||
bool = true` (PLAN §10 says "configurable" and §12.1 forgot the field; model + validate
|
||||
+ settings key + export round trip — settings are kv rows, no migration needed). 429 +
|
||||
`Retry-After: <s>`. SSE: connect consumes a token AND respects
|
||||
`web.sse_max_connections_per_ip`; `/metrics` and `/api/health` are limiter-exempt
|
||||
(Prometheus must never see 429).
|
||||
20. **SSE**: `GET /api/queries/live`, `text/event-stream` via `respondStreaming(&.{}, ...)`
|
||||
(EMPTY buffer — `BodyWriter.flush` does not flush the body writer's own buffer,
|
||||
http.zig:780; empty buffer makes every write go straight to the chunked drain), flush
|
||||
headers before the first event (test.zig:498 pattern). Fanout: milestone-6 ruling 4
|
||||
concretized as `src/server/query_sink.zig`: `QuerySink { logger: *Logger, hub: ?*sse.Hub
|
||||
}` with `log(io, entry)` = transform once, publish to the hub, enqueue to the logger.
|
||||
Logger gains additive `transformed(entry) Entry` (pure) and `logTransformed(io, entry)`;
|
||||
existing `log` = both, frozen behavior. `Handler.logger: ?*Logger` becomes `sink:
|
||||
?*QuerySink` (mechanical rename at the two call sites + app wiring). Hub: fixed 32
|
||||
subscriber slots, each a 64-entry ring + `std.Io.Event`; publish never blocks (full ring
|
||||
→ disconnect that subscriber; SSE clients auto-reconnect). Wire format: `retry: 3000`
|
||||
once, then `event: query` + `data: <JSON, same fields as /api/queries rows>` per entry;
|
||||
heartbeat comment `: ping` every 15 s from the subscriber task (Event.waitTimeout).
|
||||
Fanout precedes persistence (PLAN:455) — the sink publishes before enqueue.
|
||||
21. **`/metrics`**: Prometheus text format 0.0.4, `nxdns_` prefix, `# HELP`/`# TYPE` lines.
|
||||
Counters: every `Handler.Stats` field (17, individually, `nxdns_dns_<field>_total`),
|
||||
logger (queries_dropped, rows_written, batches_gated), cache stats (6, under
|
||||
`Handler.cache_mutex`), DNS limiter stats (under `limiter_mutex`), tracker stats,
|
||||
retention stats, log-sink stats, `nxdns_blocklist_refreshes_gated_total`. Gauges:
|
||||
cache len + memory bytes, disk free/db/log bytes, tracked DNS clients, pending tracker
|
||||
clients, blocklist generation, `nxdns_up 1`. Per-upstream, labeled `{url="..."}`:
|
||||
up (available), success_rate, consecutive_failures, total_successes/_failures (copy
|
||||
`Pool.Snapshot` fields while holding the returned count — `last_error` is borrowed,
|
||||
copy immediately). No timestamps. Auth-exempt, limiter-exempt.
|
||||
22. **`GET /api/health`**: `{"status":"ok"|"degraded","disk":{"state":...,
|
||||
"free_bytes":...,"db_bytes":...,"log_bytes":...,"sample_failures":...},
|
||||
"upstreams":{"available":N,"total":M},"queries_dropped":N,"writer_failed":bool,
|
||||
"refreshes_gated":N,"snapshot_generation":N|null}`. Degraded iff disk state != ok, or
|
||||
zero upstreams available, or writer_failed. 200 either way (degraded is data, not an
|
||||
HTTP failure).
|
||||
23. **OpenAPI + contract tests**: hand-written `src/web/openapi.yaml`, `@embedFile`d and
|
||||
served verbatim. No external validator (dependencies are liabilities): the contract IS a
|
||||
Zig table in the integration test — every route with method, auth requirement, a seeded
|
||||
request, expected status, and a response-shape struct that `std.json.parseFromSlice`
|
||||
must accept with `.ignore_unknown_fields = false`. Two drift guards: (a) the router
|
||||
exposes `pub const routes: []const RouteInfo`; a test asserts every entry's path+method
|
||||
appears textually in openapi.yaml; (b) the table covers every `routes` entry (count
|
||||
equality). CI gets one new job step running the existing `-Dintegration` suite (contract
|
||||
tests live inside it) — no new tooling.
|
||||
24. **Static assets**: build.zig gains `-Dweb-dist=<path>` (default `web/dist-placeholder/`,
|
||||
committed, containing a real minimal `index.html` — current status via /api/health
|
||||
fetch, links to /metrics; plus favicon). A build step copies the dist dir via
|
||||
`b.addWriteFiles()` + generated `assets.zig` index (the fixtures anonymous-import
|
||||
pattern, per research notes — @embedFile paths must live inside the module root),
|
||||
exposing `pub const files: []const File{path, bytes, content_type, etag}`; etag =
|
||||
comptime hash. Serving: exact path match, `/` → index.html, SPA fallback (unknown
|
||||
non-/api path → index.html, 200 — TanStack Router needs it in M9), `ETag`/
|
||||
`If-None-Match` → 304, `content-encoding: gzip` when a sibling `<name>.gz` exists in
|
||||
the dist AND the request accepts it. No Date/Last-Modified headers (std has no RFC 1123
|
||||
formatter; ETag is strictly better for immutable embedded content). Dev mode: CLI flag
|
||||
`nxdns run --web-dev <dir>` serves from disk (no cache headers), bypassing the embed.
|
||||
25. **Head-string trap**: `request.head.target` is invalidated by body reads
|
||||
(Server.zig:594/230). The router copies target into a stack buffer before any body
|
||||
read. Query strings: split on '?', `std.mem.splitScalar` for pairs,
|
||||
`Uri.percentDecodeInPlace` per value (std has no query iterator; std.Uri does not parse
|
||||
origin-form targets).
|
||||
26. **Wiring**: one `web_server.serve` task in the app group, started last, canceled by the
|
||||
same group.cancel. `WebState` struct (in web/server.zig) of borrowed pointers assembled
|
||||
by app.serve: handler (stats + mutexes + cache/limiter), pause, tracker, manager, pool,
|
||||
monitor, logger, retention stats, local_tables, sink/hub, config arena view (web cfg),
|
||||
its OWN `config.db` + `querylog.db` connections (m7 ruling 21), version string, started
|
||||
timestamp. Web DB connections open only when `web.enabled`.
|
||||
27. **No new schema migration**: query_log already carries what /api/queries needs; the
|
||||
read session verifies existing indexes and reports if a needed index is missing (then
|
||||
the orchestrator rules on adding migration v3 — do not add one silently).
|
||||
28. **Client disconnect**: stream writes surface `error.WriteFailed` with the real cause in
|
||||
`stream_writer.err` (ConnectionResetByPeer / SocketUnconnected=EPIPE — MSG.NOSIGNAL is
|
||||
set, Threaded.zig:13071, no SIGPIPE masking needed). A failed write ends the connection
|
||||
task quietly (debug log at most): clients vanishing is normal.
|
||||
29. **Input sanitation** (PLAN §19): every path/query/body input is length-capped and
|
||||
validated before SQL (prepared statements everywhere, already house rule); domain
|
||||
inputs run through matcher.normalize; no secrets ever logged (password, tokens, cookie
|
||||
values — assert by review, and the login handler logs only the client IP and outcome).
|
||||
|
||||
## Sessions
|
||||
|
||||
Wave 1 (parallel): W1 query-log reads, W2 repo CRUD, W3 web core, W4 auth+limiter,
|
||||
W5 sink+SSE hub+logger split.
|
||||
Wave 2 (after W3, parallel where files allow): W6 read handlers + metrics, W7 mutation
|
||||
handlers + local_tables + handler.zig sink/local_tables changes.
|
||||
Wave 3: W8 static assets + build.zig + openapi.yaml + router registration of everything.
|
||||
Wave 4: W9 app/cli wiring. Wave 5: W10 integration + contract tests + CI.
|
||||
The orchestrator wires src/tests.zig per wave and rules on anything a session flags.
|
||||
|
||||
---
|
||||
|
||||
## Session W1: query-log read layer
|
||||
|
||||
Owns: `src/storage/repositories/queries_repo.zig` (additive; BatchWriter and friends frozen).
|
||||
|
||||
```zig
|
||||
pub const QueryRow = struct { id: i64, ts: i64, domain: []const u8, client_ip: []const u8,
|
||||
qtype: ?u16, blocked: bool, block_reason: []const u8, response_time_us: ?i64,
|
||||
cache_hit: ?bool, upstream: []const u8 };
|
||||
pub const QueryFilter = struct { limit: u32 = 100, before: ?i64 = null,
|
||||
domain_substring: ?[]const u8 = null, client: ?[]const u8 = null,
|
||||
blocked: ?bool = null, since: ?i64 = null, until: ?i64 = null };
|
||||
/// Rows come back newest-first. Strings are arena-allocated.
|
||||
pub fn selectQueries(database: *db.Db, arena: Allocator, filter: QueryFilter)
|
||||
db.Error!std.ArrayList(QueryRow)
|
||||
pub const StatsTotals = struct { queries: u64, blocked: u64, cached: u64,
|
||||
distinct_clients: u64, avg_response_time_us: ?i64 };
|
||||
pub fn statsTotals(database: *db.Db, since: i64, until: i64) db.Error!StatsTotals
|
||||
pub const Bucket = struct { ts: i64, queries: u64, blocked: u64, cached: u64 };
|
||||
/// Fills `out` with fixed-width buckets covering [since, until); returns the count.
|
||||
pub fn timeseries(database: *db.Db, since: i64, bucket_seconds: u32, out: []Bucket)
|
||||
db.Error!usize
|
||||
```
|
||||
|
||||
Read the real query_log schema first (querylog_schema.zig) — domain interning means a JOIN;
|
||||
verify what indexes exist and REPORT (ruling 27) if `ts` or the join needs one that is
|
||||
missing rather than adding a migration. Dynamic WHERE assembly must still use bound
|
||||
parameters only (build the SQL from fixed fragments, never interpolate values). Cap
|
||||
`limit` at 1000 in the repo too. Tests: in-memory db seeded via BatchWriter; filters,
|
||||
keyset paging across a boundary, bucket alignment, empty ranges, substring escaping
|
||||
(`%`/`_` in domain must not act as wildcards — use ESCAPE).
|
||||
|
||||
Acceptance: fmt/ast clean; temp-root (-lc -lsqlite3) green; frozen surface untouched.
|
||||
|
||||
### W1 As built
|
||||
|
||||
`pub const max_limit: u32 = 1000` exported. WHERE assembly: six comptime fragments copied
|
||||
into a comptime-sized stack buffer (overflow impossible by construction), one bare `?` per
|
||||
predicate, bind order = append order; `likePattern` escapes `%`, `_` and `\` with
|
||||
`ESCAPE '\'`. Ruling 27 discharged: EXPLAIN QUERY PLAN over all six query shapes rides
|
||||
idx_query_log_client / idx_query_log_ts / rowid — NO migration v3; the time-windowed page's
|
||||
temp B-tree re-sort and the DISTINCT count's temp B-tree are inherent and cheap at household
|
||||
volume. `statsTotals` computes the mean as sum/count integer division (db.Stmt has no float
|
||||
column reader; avg() returns REAL) — exact microseconds, null when no row recorded a time.
|
||||
NULL `block_reason`/`upstream` arrive as `""` (columnText convention; neither is ever
|
||||
written as an empty string). **W6 convention, ruled now: `""` serializes as `""`, not JSON
|
||||
null** — `upstream=""` already means "cache hit" per milestone-7 ruling 20 and clients
|
||||
branch on `blocked`/`cache_hit`, not on reason presence. `timeseries` returns
|
||||
`error.Misuse` for bucket_seconds == 0 or an i64 window overflow, returns 0 for an empty
|
||||
`out` without touching the database, and aligns buckets to `since` — the HANDLER passes a
|
||||
grid-aligned since per ruling 13. 14 new tests (43 in file).
|
||||
|
||||
---
|
||||
|
||||
## Session W2: repo CRUD by id
|
||||
|
||||
Owns every file in `src/storage/repositories/` EXCEPT queries_repo.zig (W1's), plus nothing
|
||||
else. Additive only; import-path functions frozen.
|
||||
|
||||
For groups, clients (+prefixes), upstreams, sources, rules, local records, forward zones:
|
||||
id-carrying list variants (or extend existing rows where the shape already has no
|
||||
consumers outside export — check callers first; export/import must keep compiling
|
||||
unchanged), `getX(db, id) db.Error!?XRow`, `insertXRow(db, item) db.Error!i64` (returns
|
||||
id; distinct from frozen import inserts where semantics differ — e.g. clients insert with
|
||||
hand_edited=1), `updateX(db, id, item) db.Error!void` (0 rows → error.NotFound),
|
||||
`deleteX(db, id) db.Error!void` (same), `setGroupSources(db, group_id, source_ids)
|
||||
db.Error!void` (transactional replace), `replaceClientPrefixes(db, items) db.Error!void`.
|
||||
Respect FK constraints: deleting a group with clients → error.Constraint surfaces as 409 at
|
||||
the handler; document each. `settings_repo`: `putSetting(db, key, value)` single-key upsert
|
||||
if absent. Tests per resource: round trip, NotFound on both update and delete, constraint
|
||||
surfacing, group-sources replace idempotence.
|
||||
|
||||
Acceptance: fmt/ast clean; temp-root green; `zig build test` green (export/import tests
|
||||
prove the frozen surface).
|
||||
|
||||
### W2 As built
|
||||
|
||||
New shared `crud.zig`: `execStrict` turns zero-rows-touched into `error.NotFound`. Accepted
|
||||
deviations: `getX(database, gpa, id) db.Error!?XRow` (rows hold heap strings);
|
||||
`insertClientRow`/`insertRuleRow` take `now_s` (NOT NULL columns, repos take no Io — the
|
||||
InsertContext.now mirror); client writes split `ClientInput{ip,name,group_id}` (create) vs
|
||||
`ClientEdit{name,group_id}` (edit) — **ip is not editable**: it is the identity upsertSeen
|
||||
matches, and ruling 9 promises only name/group on PUT; client_prefixes get list + replace
|
||||
only (whole-list resource, per-id would be unused generality); write shapes carry group_id
|
||||
(a missing group surfaces as the FK violation → 409, not an id-map miss); added
|
||||
`listGroupSourceIds` (the PUT needs a read counterpart; frozen listGroupSources speaks
|
||||
names). Constraint map for W7: deleting a group with clients → Constraint (clients.group_id
|
||||
has no ON DELETE; rules/prefixes/group-sources cascade); bad group_id on client/rule writes
|
||||
→ Constraint; url/ip/zone/name uniques → Constraint; deletes of clients/upstreams/rules/
|
||||
local-records/forward-zones can only be NotFound. `setGroupSources`: BEGIN IMMEDIATE,
|
||||
existence check first (empty set must not report success for a missing group), delete +
|
||||
distinct insert (set semantics dedupe); `replaceClientPrefixes` transactional whole-table,
|
||||
duplicate prefix REJECTED as Constraint (two rows for one prefix is a contradiction, not a
|
||||
set). Schema facts: `upstreams.tls_name` comes from migration v2 (config_schema.zig is v1
|
||||
only!) — the openapi /api/upstreams entry needs the fifth field; `SourceRow` gained
|
||||
`is_suggested: bool = false` appended last (manager.zig literals compile unchanged, no
|
||||
column index moved). `putSetting` is a primary-key upsert — partial settings PUT needs no
|
||||
read-first and fires no constraint. 45 new tests + 3 crud.
|
||||
|
||||
---
|
||||
|
||||
## Session W3: web core — server, router, http plumbing
|
||||
|
||||
Owns: `src/web/server.zig`, `src/web/router.zig`, `src/web/http_util.zig`.
|
||||
|
||||
- server.zig: `WebState` (ruling 26 — declare the struct; fields it cannot yet point at get
|
||||
wired by W9), `pub fn serve(state: *WebState, io: std.Io) std.Io.Cancelable!void` — bind
|
||||
per ruling 5 (bind failure: warn and return — the DNS side must keep serving; app treats
|
||||
web bind failure as non-fatal, W9 documents), accept loop per ruling 4 with the Stop
|
||||
split, connection budget per ruling 7 (503 over cap), per-connection buffers (8 KiB recv
|
||||
/ 4 KiB send), keep-alive loop, dispatch into router.
|
||||
- router.zig: `RouteInfo { method: http.Method, pattern: []const u8, auth: enum {open,
|
||||
session}, handler: *const fn(...) }`; `pub const routes: []const RouteInfo` (ruling 23
|
||||
depends on it); match = exact segments + one `{id}` numeric capture; 405 with Allow when
|
||||
the path matches another method; target copied before body reads (ruling 25); query-pair
|
||||
iterator + percent-decode helpers (in http_util, pure, tested hard: '+', '%2F', truncated
|
||||
%, overlong values → 400).
|
||||
- http_util.zig: JSON respond helpers (`std.json.Stringify` streaming into a
|
||||
`Writer.Allocating` for content-length, or direct respond for small bodies), the error
|
||||
envelope, request-body reader with the 1 MiB cap (413), cookie parse/format, bearer of
|
||||
nothing else.
|
||||
- The connection handler calls: limiter (W4, via a comptime-checkable interface field on
|
||||
WebState so W3 compiles before W4 lands — define `ApiLimiter` and `Sessions` as W3-owned
|
||||
INTERFACE structs? NO: simpler, W3 declares the WebState fields as `*auth.Sessions` /
|
||||
`*auth.ApiLimiter` types and W3 is built AFTER W4's file exists on disk in the same wave
|
||||
— to keep the wave parallel, W3 instead keeps auth/limiter checks behind two function
|
||||
pointers on WebState (`check_auth`, `check_limit`) that W9 wires; W3 tests them with test
|
||||
doubles. This is the one seam where indirection is warranted; document it.)
|
||||
- Tests: router matching table, 405/404, query decoding, body cap, cookie round trip,
|
||||
keep-alive across two requests (loopback socket), 503 over cap, cancel-during-idle
|
||||
keep-alive returns promptly.
|
||||
|
||||
Acceptance: fmt/ast clean; temp-root green (may import std only + own files); no
|
||||
listener/handler file touched.
|
||||
|
||||
### W3 As built
|
||||
|
||||
Handler signature: `fn(state: *server.WebState, io: std.Io, request: *http_util.Request)
|
||||
http_util.HandlerError!void`, `HandlerError = {WriteFailed, HttpExpectationFailed,
|
||||
OutOfMemory}` — domain outcomes are status codes, only those three escape. `Request` folds
|
||||
the id capture in (`request.id: ?i64`; no separate params type) and carries a per-request
|
||||
arena. `RouteInfo` gained `rate_limit: enum{counted,exempt} = .counted` (ruling 19's
|
||||
exemptions as data, not path matching); WebState gained `fallback: ?HandlerFn` (SPA
|
||||
fallback is not a route entry; unmatched non-/api → fallback, unmatched /api → JSON 404)
|
||||
and `reload_fn` (W7's ruling-12 seam). routes.zig: W8 fills `pub const table`; router
|
||||
re-exports as `routes`; WebState.routes defaults to it, so the drift guards read the array
|
||||
the server matches. W4/W5 landed mid-session, so sessions/limiter/hub/sink are REAL typed
|
||||
pointers and check_auth/check_limit default to real implementations (`sessionAuth`,
|
||||
`bucketLimit`; `allowAll`/`neverLimit` exported as test doubles) — a half-wired server
|
||||
fails closed. serve returns void (cancellation is classified into the Stop enum;
|
||||
tcp_server precedent). Own validating percent-decoder — std.Uri's copies malformed escapes
|
||||
through as literal text (Uri.zig:172-192), ruling 29 wants 400; segments split before
|
||||
decode so %2F cannot forge a boundary. Loopback tests live in web/server_integration_test
|
||||
.zig (house gating pattern). Accepted tradeoff: over-capacity 503 is written and the
|
||||
socket closed without draining, so the client may see RST instead — draining would be an
|
||||
unbounded blocking read on an at-capacity accept loop (documented). RULINGS RECORDED:
|
||||
W7 may add the `local_tables` field + import to server.zig (two-line exception, W3
|
||||
consents); Retention.stats is a cross-task data race — W6 owns storage/retention.zig
|
||||
additively this wave: atomic counters + `snapshotStats` (logger-counters pattern), and
|
||||
/metrics reads only through it. 39 tests.
|
||||
|
||||
**Amendment (post-W9 critical fix)**: `serveConn` normalizes the request head immediately
|
||||
after `receiveHead`: a body-carrying method (POST/PUT/PATCH) with neither Content-Length
|
||||
nor Transfer-Encoding gets `head.content_length = 0` before any dispatch or respond.
|
||||
RFC 9110 §8.6 defines such a request as having an empty body, but std leaves the head
|
||||
saying "unknown" and `respond` → `discardBody` (http/Server.zig:631) asserts one of the
|
||||
two is set — so a bare `curl -X POST` panicked the whole process, DNS included. Zero
|
||||
content-length satisfies every downstream path: `bodyReader` (http.zig:445) goes straight
|
||||
to `.ready`, `discardRemaining` sees EndOfStream, keep-alive survives, and
|
||||
`http_util.readBody` yields an empty slice. Regression test in
|
||||
server_integration_test.zig: raw POST without either header → well-formed response, empty
|
||||
body observed by the handler, fresh connection answered after, connection_errors 0.
|
||||
|
||||
---
|
||||
|
||||
## Session W4: auth + API limiter
|
||||
|
||||
Owns: `src/web/auth.zig`, `src/web/api_limiter.zig`, plus the `model.zig`/`validate.zig`
|
||||
additions for `web.api_localhost_exempt` (ruling 19) — model field, validate rule (none
|
||||
needed beyond type), settings key list, and the export/import round-trip expectations
|
||||
(check model.toSettings reflection picks it up automatically; add the settings-count test
|
||||
adjustments).
|
||||
|
||||
- auth.zig: `Sessions` fixed 32-slot store per ruling 17 (create → cookie value out;
|
||||
validate cookie → bool; logout; clearAll; expiry sweep on access; LRU evict), login
|
||||
verify via argon2 strVerify (io + allocator required — argon2.zig:600), token =
|
||||
32 bytes randomSecure → url_safe_no_pad; storage = SHA-256 digest; compare
|
||||
timing_safe.eql([32]u8,...). `authEnabled(cfg) bool`. Pure unit tests with fixed tokens
|
||||
(seed io.random? no — inject the token bytes via a testable `createWithToken`).
|
||||
- api_limiter.zig: token bucket per ruling 19, fixed 4096-key table (house idiom), awake
|
||||
clock, `check(now, key, is_localhost) Result{allowed, retry_after_s}`; localhost exempt
|
||||
when configured; `sse_connections` per-IP counter with `tryAcquireSse/releaseSse` against
|
||||
`sse_max_connections_per_ip`. Tests: refill math at boundaries, burst=capacity, exempt
|
||||
localhost, sse cap acquire/release, eviction.
|
||||
|
||||
Acceptance: fmt/ast clean; temp-root green; `zig build test` green (model/validate/export
|
||||
tests still pass with the new field).
|
||||
|
||||
### W4 As built
|
||||
|
||||
Sessions: digests only (SHA-256), full-slot scan on validate so work does not depend on
|
||||
match position; `verifyPassword` returns `Outcome{ok,denied,unavailable}` — `.unavailable`
|
||||
(broken PHC string, OOM) maps to 500, not 401; nothing secret is ever formatted (warn
|
||||
prints `@errorName` only). `createWithToken`/`validateAt` are the test seams. Cookie value
|
||||
is 43 chars url_safe_no_pad; `cookie_attributes = "HttpOnly; SameSite=Lax; Path=/"`.
|
||||
Limiter deviations, accepted: (1) `check(io, now, addr)` — the limiter holds its OWN
|
||||
std.Io.Mutex (W3 dispatches connections concurrently; one lock inside beats N outside) and
|
||||
derives loopback itself via exported `isLoopback`, so callers cannot disagree; (2) refill
|
||||
counts millionths of a token and advances the clock only by the span converted, so the
|
||||
truncated remainder survives — at 1 req/min the 60 s boundary is exact (tests at 999/1000
|
||||
ms); (3) the SSE per-IP cap binds loopback clients too — `api_localhost_exempt` exempts
|
||||
the RATE, not the fixed hub-slot resource. Full table → unknown addresses allowed +
|
||||
`untracked` (DNS-limiter precedent); `sweep` drops only full, SSE-free, window-idle
|
||||
buckets; `retry_after_s` never 0 on refusal. Model ripple: `web.api_localhost_exempt` in
|
||||
Web + expected_keys + fixture; validate.zig needed no edit (bool; decodeValue already
|
||||
rejects non-bool text); no count assertion outside W4 files existed. 12 auth + 14 limiter
|
||||
tests.
|
||||
|
||||
---
|
||||
|
||||
## Session W5: query sink, SSE hub, logger split
|
||||
|
||||
Owns: `src/server/query_sink.zig` (new), `src/web/sse.zig` (new), `src/storage/logger.zig`
|
||||
(additive split ONLY: `transformed(entry) Entry` pure + `logTransformed(io, entry)`;
|
||||
existing `log` becomes transform+logTransformed with byte-identical behavior — the
|
||||
existing tests must pass unchanged), and `src/server/handler.zig` (mechanical: field
|
||||
`logger: ?*logger_mod.Logger` → `sink: ?*query_sink.QuerySink`, the log call site, tests
|
||||
updated to construct a sink around their logger).
|
||||
|
||||
- query_sink.zig per ruling 20: publish BEFORE enqueue (PLAN:455).
|
||||
- sse.zig `Hub`: 32 slots × 64-entry Entry rings + Event per subscriber;
|
||||
`subscribe() ?SubscriberId` / `unsubscribe(id)`; `publish(io, entry)` copies the entry
|
||||
into every live ring (Entry is self-contained, ~400 B; a full ring marks the subscriber
|
||||
`overflowed` and sets its event — the subscriber task sees the flag and ends the
|
||||
response); `wait(id, timeout)` for the heartbeat loop. All under one mutex; publish is
|
||||
called on the DNS hot path — it must stay allocation-free and short (copy + flag set).
|
||||
Tests: publish/consume order, overflow disconnect flag, unsubscribe under load,
|
||||
heartbeat timeout returns empty.
|
||||
- Handler tests: one added test proving the sink publishes and logs (tiny hub + logger).
|
||||
|
||||
Acceptance: fmt/ast clean; temp-root green including ALL existing handler and logger
|
||||
tests; `zig build test` + `-Dintegration` green (phase7 tests construct Handlers — they
|
||||
compile against the renamed field; update them, they are in your ownership for THIS
|
||||
mechanical change only — list every touched line in the report).
|
||||
|
||||
### W5 As built
|
||||
|
||||
Every Hub method takes `io` (the mutex needs it); all locking is `lockUncancelable`
|
||||
(milestone-7 tracker precedent: `handle` has no error union). `Hub.init(self) void`
|
||||
initializes IN PLACE — 32×64 rings ≈ 900 KiB, so **W9 must `gpa.create(Hub)`, never a
|
||||
stack local**. Read surface ruling 20 left open, as built: `next(io, id) ?Entry` (pop
|
||||
oldest), `overflowed(io, id) bool` (sticky; pre-overflow entries stay readable — the
|
||||
subscriber drains then disconnects), `wait(io, id, timeout) Cancelable!Wake` with
|
||||
`Wake = {ready, timeout}`; `wait` resets the event under the mutex only while the ring is
|
||||
empty, so a racing publish is never lost, and a spurious futex wake reports `.timeout`
|
||||
(costs one heartbeat). Publish-before-persist is pinned structurally (enqueue never
|
||||
blocks, so a clock cannot observe order): the test closes the logger queue and the entry
|
||||
still reaches the hub while counting `queries_dropped`. W5 also made the minimal app.zig
|
||||
compile fix (sink around the logger, hub = null, `.sink = &sink`) — **W9: the sink already
|
||||
exists in app.zig; only the hub and the rest of the wiring are missing.** 11 sse + 4 sink
|
||||
+ 1 logger-split + 1 handler test; all pre-existing logger/handler tests unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Session W6: read handlers + metrics + health + version + openapi route
|
||||
|
||||
Owns: `src/web/handlers/stats.zig`, `queries.zig`, `lookup.zig`, `upstream_health.zig`,
|
||||
`health.zig`, `version.zig`, `metrics.zig` (in web/, not handlers/ — PLAN layout :227).
|
||||
|
||||
Implement rulings 11, 13, 14, 21, 22; `GET /api/version` = version.zig string + git commit
|
||||
build option. Every handler is `fn(state, io, request-ish, params) !void` matching W3's
|
||||
handler signature; JSON via http_util. Metrics: mind the mutex discipline (cache/limiter
|
||||
stats under the handler's mutexes; pool snapshot copies borrowed strings immediately;
|
||||
atomics via .monotonic loads). Handlers read the querylog via state's own connection.
|
||||
Tests: pure formatting tests per handler where the data is injectable (metrics text
|
||||
golden test, health degraded matrix, stats bucket math via W1 fixtures on an in-memory
|
||||
db).
|
||||
|
||||
Acceptance: fmt/ast clean; temp-root green.
|
||||
|
||||
### W6 As built
|
||||
|
||||
Every handler is split into a pure core plus a thin `handle`, so the decisions are tested
|
||||
without an `http.Server.Request`: `metrics.collect`/`metrics.render`, `health.collect`/
|
||||
`rollup`, `stats.window`/`periodParam`, `queries.parseFilter`/`page`, `lookup.evaluate`/
|
||||
`body`, `upstream_health.collect`, `version.body`. Entry points for W8's table:
|
||||
`metrics.handle`, `stats.totals`, `stats.timeseries`, `queries.list`, `lookup.handle`,
|
||||
`upstream_health.handle`, `health.handle`, `version.handle` — all match `router.HandlerFn`
|
||||
(proven by a temp-root assignment). `/metrics` and `/api/health` are `auth = .open,
|
||||
rate_limit = .exempt`; `/api/version` is `.open, .counted`; the other four are `.session,
|
||||
.counted`. **W8 must add `_ = @import("web/metrics.zig")` and the six handler files to
|
||||
src/tests.zig** — this session did not wire the orchestrator's file.
|
||||
Retention (ruled in W3's As built): `Stats` is now the plain snapshot type and the live
|
||||
counters moved to a private `Counters` of `std.atomic.Value(u64)` in `Retention.counters`;
|
||||
`snapshotStats()` reads them `.monotonic`; `runOnce` derives its pass number from the
|
||||
`fetchAdd` result rather than re-reading. Test assertions changed mechanically at
|
||||
retention.zig lines 192-195, 215, 220, 234, 238-240, 243-244, 258-260, 283-286, 312-313 and
|
||||
at phase6_integration_test.zig lines 365-368 (`x.stats.f` → `x.snapshotStats().f`) — that
|
||||
second file is the only consumer outside retention.zig, and it compiles solely under
|
||||
`-Dintegration`, so the plain suite does not catch a break there. No behavior changed.
|
||||
Decisions and deviations, all accepted:
|
||||
- `/metrics` carries the DNS counters as an ARRAY keyed by `Handler.Stats`'s comptime field
|
||||
list, so a new counter in the handler appears in the exposition with no edit here; the
|
||||
same `counterGroup` helper walks every plain stats struct. Missing collaborator = the
|
||||
whole family is omitted (a zero would read as health, an absent series reads as a gap).
|
||||
- Added beyond ruling 21's list: `nxdns_disk_sample_failures_total` (a failure mode that
|
||||
ruling 22 already surfaces) and `nxdns_upstream_enabled` (an upstream that is down and
|
||||
one that is switched off are different operator problems). Logger `writer_failed` is
|
||||
deliberately NOT a metric: it is in `/api/health` and would need a gauge family of one.
|
||||
- `metrics.poolSnapshot` is the one place that copies pool health, shared with the health
|
||||
rollup: cancel protection around `Pool.snapshot` (server.zig's precedent), then every
|
||||
borrowed string duped into the request arena immediately. Cache/limiter stats are read
|
||||
under `Handler.cache_mutex`/`limiter_mutex` with `lockUncancelable` — `HandlerError` has
|
||||
no `Canceled`, so the W5 precedent applies.
|
||||
- Ruling 13's window: `until` = end of the bucket `now` falls in, `since = until - width ×
|
||||
count`, both on the absolute epoch grid (every width divides 86 400, asserted at
|
||||
comptime), so `/api/stats` and `/api/stats/timeseries` cover the identical span — tested
|
||||
by summing the buckets against the totals. Both bodies also carry `period`, `since` and
|
||||
`until` (a chart cannot label an axis without them); the timeseries body adds
|
||||
`bucket_seconds`. **openapi.yaml must document those fields.**
|
||||
- `/api/queries`: `limit` outside 1..1000 is a 400 rather than a silent clamp, `before` must
|
||||
be positive, and each bad parameter has its own message. `next_before` is the last row's
|
||||
id only on a full page. `domain=`/`client=` empty read as absent.
|
||||
- `/api/lookup` resolves `group_id` through `groupIndexById`; an id no snapshot group has is
|
||||
a 400 (`unknown group_id`), a missing manager or snapshot is 503 `no snapshot loaded`, and
|
||||
the domain runs through `name.fromText` + `matcher.normalize` (ruling 29). `source_url`
|
||||
comes from `sources_repo.getSource` on the config connection keyed by the snapshot's
|
||||
source row id — `SourceSets` carries the display name, not the URL; an unreadable row
|
||||
leaves the field null rather than failing the lookup. Records and zones are read through
|
||||
`state.handler.?.local_tables` (acquire/release bracketed by one defer), NOT a WebState
|
||||
field, because that pointer already exists and is the same table W7 swaps.
|
||||
- `/api/upstream/health` reports url, enabled, available, consecutive_failures,
|
||||
total_successes, total_failures, success_rate, last_error, plus `available`/`total`
|
||||
rollup counts. No timestamps: they are `awake`-clock stamps and mean nothing to a client.
|
||||
- `/api/version` adds `zig_version` and `uptime_seconds` (from `WebState.started_unix`) to
|
||||
the version and commit strings.
|
||||
- The only logging in this session is one `warn` per failed database read or pool/source
|
||||
fault, on the 500 path; nothing is logged for a normal request, and no `std.log.err`.
|
||||
40 new tests (metrics 5, stats 8, queries 9, lookup 7, health 4, upstream_health 3,
|
||||
version 4); retention's 6 tests unchanged in number and green.
|
||||
|
||||
---
|
||||
|
||||
## Session W7: mutation handlers + local_tables + pause + settings + auth handlers
|
||||
|
||||
Owns: `src/web/handlers/` `groups.zig`, `blocklists.zig`, `rules.zig`, `local.zig`,
|
||||
`clients.zig`, `settings.zig`, `pause.zig`, `auth.zig` (login/logout), plus
|
||||
`src/server/local_tables.zig` (new, ruling 12) and the `src/server/handler.zig` edit to
|
||||
read records/zones through it (second mechanical handler change; coordinate: W5 already
|
||||
edits handler.zig — W7 runs AFTER W5 lands; sequence inside wave 2).
|
||||
|
||||
- local_tables.zig: `LocalTables { lock: std.Io.RwLock, records, zones }` with
|
||||
`acquire/release` (shared) and `swap(io, new_records, new_zones)` (exclusive; frees the
|
||||
old under the lock after swap — no reader can hold across queries since acquire/release
|
||||
brackets each query). Handler: acquire in handle, release on every exit (same single
|
||||
defer discipline as the snapshot).
|
||||
- Handlers: rulings 9, 12, 15, 16, 17-18 behaviors; every mutation validates (shared
|
||||
validators — reuse validate.zig section rules by constructing the candidate row set;
|
||||
where validate.zig only does whole-config, extract the per-section check it already has
|
||||
— validate.zig is NOT in your ownership: if extraction is needed, report it and use a
|
||||
local check this milestone), writes via W2 repos on state's config connection, then
|
||||
reload/rebuild per ruling 12. blocklists/update → refreshAll + reload, 202 + status.
|
||||
settings PUT: merge, validate whole config, write keys, clear sessions on hash change.
|
||||
- Tests: per handler with in-memory dbs and a real Manager where cheap, else assert repo
|
||||
effects + reload-called flag via a seam (a `reload_fn` pointer on WebState set by W9;
|
||||
tests inject a counter).
|
||||
|
||||
Acceptance: fmt/ast clean; temp-root green; phase7 integration still green.
|
||||
|
||||
### W7 As built
|
||||
|
||||
Eleven files, 4199 lines, 89 tests (local_tables 5, mutations 6, groups 13, blocklists 8,
|
||||
rules 7, local 12, clients 9, upstreams 6, pause 7, settings 10, auth 6).
|
||||
- **The apply/respond split (house pattern for later waves).** `http_util`'s response
|
||||
helpers need a live `*http.Server.Request`, which a unit test cannot construct. Every
|
||||
route splits into an `apply*` decision function driven against an in-memory database and
|
||||
a thin HTTP wrapper.
|
||||
- **`mutations.zig` is a new shared file, owned by W7.** It holds the `Failure` union, the
|
||||
constraint map (NotFound → 404, Constraint → 409, validation → 400), the `reload` and
|
||||
`swapLocalTables` seams, and the `Bench` test fixture.
|
||||
- **`config_lock` lives on WebState** (ruled post-report): connection tasks share one
|
||||
config database connection, and `changes()`/`lastInsertRowid()` are connection state that
|
||||
`execStrict` reads, so writes serialize through `state.config_lock`.
|
||||
- **Per-row validation**: a minimal valid skeleton config plus the one candidate row runs
|
||||
the real `validate.validate`; no validator duplicated. Duplicate keys and foreign-key
|
||||
violations surface from the database as 409, not 400.
|
||||
- **`upstreams.zig` is W7's** (ruling 9 listed the resource; neither ownership list named
|
||||
it). It never calls the reload seam, refuses to delete the last enabled upstream (409),
|
||||
and reports `restart_required: true`.
|
||||
- **Ruled additions**: `GET /api/settings` returns a derived `web.auth_enabled` flag; the
|
||||
default group cannot be renamed or deleted (409); a group that still holds clients cannot
|
||||
be deleted (409).
|
||||
- **handler.zig**: `records`/`zones` replaced by `local_tables: ?*LocalTables = null`;
|
||||
`handle` acquires once per query, releases on defer — one query reads one generation of
|
||||
records, zones and the filter snapshot together. `Context` gained `records`/`zones`.
|
||||
- **Ruled mechanical ripples**: web/server.zig two-line exception (import + WebState
|
||||
field, plus the config_lock field above); app.zig minimal fix (`var tables: LocalTables
|
||||
= .empty` + build calls — **W9 must review**); udp/tcp/resolver integration tests lost
|
||||
their `empty_records`/`empty_zones` consts and two `bareHandler` initialisers; phase7
|
||||
test converted to real tables where needed.
|
||||
- **Routes for W8** (all match W3's signature; id routes read `request.id.?`): groups
|
||||
`list get create update remove getSources putSources`; blocklists `list get create
|
||||
update remove refresh` (refresh = POST /api/blocklists/update, 202 + status snapshot);
|
||||
rules `list get create update remove`; local `listRecords getRecord createRecord
|
||||
updateRecord removeRecord listZones getZone createZone updateZone removeZone`; clients
|
||||
`list get update remove listPrefixes putPrefixes` (no create); upstreams `list get
|
||||
create update remove`; pause `get post`; settings `get put`; auth `login logout`.
|
||||
|
||||
---
|
||||
|
||||
## Session W8: static assets, build pipeline, openapi.yaml, route table completion
|
||||
|
||||
Owns: `src/web/static.zig`, `src/web/openapi.zig`, `src/web/openapi.yaml`,
|
||||
`web/dist-placeholder/` (index.html + favicon.svg), `build.zig` (the -Dweb-dist option +
|
||||
asset module generation), and the final `router.zig` route-table registration IF W3 left
|
||||
registration data-driven (coordinate: W3 owns router.zig — W8 only ADDS the route array
|
||||
entries file `src/web/routes.zig` if W3 designed it that way; otherwise W8 hands W3's
|
||||
owner a patch — resolve by making W3 expose `routes` as a separate file owned by W8 from
|
||||
the start; W3 must read this paragraph).
|
||||
|
||||
Ruling 24 in full: asset module via addWriteFiles + generated index; etag comptime hash;
|
||||
gzip sibling serving; SPA fallback; --web-dev disk serving (cli flag lands in W9 — W8
|
||||
exposes `serveFromDisk(dir, ...)`). openapi.yaml documents EVERY route in the table with
|
||||
schemas matching the handlers' JSON (snake_case, error envelope, auth markers, 429/401
|
||||
responses). Tests: static matching, 304 flow, gzip negotiation, SPA fallback vs /api 404,
|
||||
placeholder embeds and serves.
|
||||
|
||||
Acceptance: fmt/ast clean; `zig build` with default dist green; temp-root green.
|
||||
|
||||
### W8 As built
|
||||
|
||||
Seven files: static.zig 362, openapi.zig 66, routes.zig 194 (55 routes), openapi.yaml
|
||||
2255, handlers/live.zig 184 (ruled mid-session — no session owned the SSE HTTP handler),
|
||||
tools/gen_web_assets.zig 201 (new build tool, W8's), web/dist-placeholder (real page:
|
||||
fetches /api/health + /api/version, renders status/upstreams/disk). 21 new tests
|
||||
(static 9, openapi 3, routes 6, live 3).
|
||||
- **Asset pipeline**: gzip cannot run in the build system itself, so
|
||||
`tools/gen_web_assets.zig` runs as a build step: dist staged through WriteFiles (Run
|
||||
hashes only a directory arg's path string; staging bakes the content hash into the
|
||||
path, so edits invalidate — proven), tool output merged with generated `.gz` siblings
|
||||
and `assets.zig` into the `web_assets` anonymous import (exe, cross exes, tests).
|
||||
- **`web_assets` module surface**: `File{path, bytes, content_type, etag}`,
|
||||
`files: []const File`; root is the generated assets.zig.
|
||||
- **ETag** computed in the tool, not comptime (same build-time constant; avoids the
|
||||
comptime interpreter hashing large M9 bundles). SHA-256/128-bit, quoted, strong.
|
||||
`.gz` siblings are their own `files` entries with their own ETag; direct `*.gz` paths
|
||||
are not addressable. The tool skips gz when it does not shrink or below 128 bytes.
|
||||
- **static.zig surface for W9**: `state.fallback = static.fallback` (embedded) or wrap
|
||||
`static.serveFromDisk(dir, io, request)` in a HandlerFn for `--web-dev` (app.zig owns
|
||||
the dir storage; no cache headers in dev mode). Traversal guard in `diskRelativePath`.
|
||||
- **live.zig** (SSE, ruling 20): respondStreaming with empty buffer, `retry: 3000`,
|
||||
`event: query` frames, 15 s heartbeat (awake clock) via `Hub.wait`, per-IP cap via
|
||||
`tryAcquireSse`/`releaseSse`, ring overflow ends the stream with a clean chunked
|
||||
terminator so EventSource reconnects. RULED: route is `.session, .exempt` (overrides
|
||||
ruling 19's connect-token line; the per-IP SSE cap is the guard, loopback included).
|
||||
RULED: the SSE payload omits `id` (a live entry precedes persistence); a comptime test
|
||||
pins the remaining nine field names to `QueryRow`.
|
||||
- **Route policy**: open = {/metrics, /api/health, /api/version, /api/openapi.yaml,
|
||||
/api/auth/login}; limiter-exempt = {/metrics, /api/health, /api/queries/live}; logout
|
||||
is `.session` (ruling 18).
|
||||
- **openapi.yaml** documents all 55 routes, cookie security scheme, 401 on every session
|
||||
route, 429 + Retry-After on every counted route, the W6 stats fields (period, since,
|
||||
until, bucket_seconds), SSE as text/event-stream; no certs/reload. openapi.zig's test
|
||||
enforces route ⊆ yaml at unit level; W10 still owns the count-equality drift guard (b).
|
||||
|
||||
---
|
||||
|
||||
## Session W9: app + cli wiring
|
||||
|
||||
Owns: `src/app.zig`, `src/cli.zig`, `src/main.zig` (if forced).
|
||||
|
||||
WebState assembly (ruling 26) gated on `web.enabled`; two extra DB connections; hub +
|
||||
sink construction (sink wraps logger always — hub only when web enabled; DNS path cost
|
||||
without web = one null check); local_tables construction replacing the direct
|
||||
records/zones locals; web serve task in the group (started last); reload_fn/check seams
|
||||
wired; `--web-dev <dir>` CLI flag (parseArgs + help text + runCheck untouched); web bind
|
||||
failure = warn + continue serving DNS (ruling in W3 — confirm and document in app).
|
||||
Shutdown order unchanged (web task dies with group.cancel; its connection group cancels
|
||||
per ruling 4). Smoke test yours: boot with web enabled, curl /api/health, /api/version,
|
||||
/metrics, /, login flow with a password set (import a config with web.password, verify
|
||||
cookie + 401 without), SIGTERM exit 0. Report the transcript.
|
||||
|
||||
Acceptance: fmt/ast clean; zig build; both test suites green; smoke transcript.
|
||||
|
||||
### W9 As built
|
||||
|
||||
- `cli.Command.run` payload is now `RunArgs { paths: Paths, web_dev: ?[]const u8 }`;
|
||||
`parseRunArgs` handles `--web-dev DIR` (both spellings); `check` rejects it;
|
||||
`runCheck` untouched. Anything spawning `app.run` passes `RunArgs`.
|
||||
- app.zig serve order: zero-guard (`web.api_rate_limit_per_min`/`session_ttl_hours` zero
|
||||
on an enabled web → `error.BadRateLimit`, exit 2 — collaborators assert nonzero and a
|
||||
hand-edited database bypasses validate); heap `sse.Hub` via `gpa.create` + in-place
|
||||
init (W5 rule), gated on `web.enabled`; sink wraps `(&logger, hub)` ALWAYS (hub null
|
||||
without web = one branch); two web DB connections (openConfigDb + reopenQuerylogDb,
|
||||
after openQuerylogDb establishes the file), gated; Sessions + ApiLimiter, gated;
|
||||
WebState assembled after the DNS handler (`reload_fn = app.reloadManager`,
|
||||
`fallback = static.fallback` or the dev wrapper, version.string, started_unix); web
|
||||
task added to the group LAST; bind failure = warn + return, DNS keeps serving.
|
||||
Shutdown order unchanged. `web.enabled = false` → no hub, no web DBs, no
|
||||
sessions/limiter, no web task.
|
||||
- `--web-dev` dir lives in a file-scope var in app.zig (`WebState.fallback` is a bare fn
|
||||
pointer, no closure; one composition root; written once before the task starts).
|
||||
- W7's `var tables: LocalTables` holder confirmed correct: `WebState.local_tables`
|
||||
points at the same holder the DNS handler reads, so `swapLocalTables` swaps both.
|
||||
- Mechanical ripple: phase7 test line 967 `cli.Paths{...}` → `RunArgs{ .paths = ... }`.
|
||||
- Smoke test passed end to end: import with web.password → boot → 200 on /api/health,
|
||||
/api/version, /metrics, /, /api/openapi.yaml → 401 without cookie → login sets
|
||||
HttpOnly/SameSite=Lax cookie → authed read → logout kills the cookie → wrong password
|
||||
401 → SIGTERM exit 0. No secrets in logs. `--web-dev` smoked: disk edits appear
|
||||
without restart.
|
||||
- **Critical finding (fixed post-session in W3's files)**: a bodyless POST/PUT — no
|
||||
Content-Length, no Transfer-Encoding, exactly `curl -X POST` — hit the
|
||||
`discardBody` assert in std (Server.zig:631) and panicked the process, DNS included.
|
||||
RFC 9110 gives such a request an empty body, so the fix normalizes in the request
|
||||
path before any respond call (see W3 As-built amendment); W10 pins it with a contract
|
||||
test.
|
||||
|
||||
---
|
||||
|
||||
## Session W10: integration + contract tests + CI
|
||||
|
||||
Owns: `src/web/web_integration_test.zig` (new), `.gitea/workflows/ci.yml` (additive job
|
||||
steps only).
|
||||
|
||||
Contract table per ruling 23 covering every route; auth on/off matrix (seeded
|
||||
password_hash vs empty); SSE test (connect, cause one query via a real Handler or direct
|
||||
sink.publish, assert `event: query` frame + heartbeat + cap enforcement via
|
||||
sse_max_connections_per_ip=1); rate-limit 429 + Retry-After; pagination walk; mutation →
|
||||
reload observed (snapshot generation bump); pause via API affects a real handler decision;
|
||||
settings PUT round trip; static + ETag 304; drift guards (a) and (b). CI: ensure the
|
||||
integration job still covers everything (contract tests ride -Dintegration; add a
|
||||
`zig build test -Dintegration` step only if missing — read the current yml first).
|
||||
|
||||
Acceptance: `zig build test -Dintegration` green with zero err lines; plain suite green.
|
||||
|
||||
### W10 As built
|
||||
|
||||
web_integration_test.zig, 1406 lines, 14 tests: 11 integration-gated (contract walk over
|
||||
all 55 routes with strict `ignore_unknown_fields = false` response shapes, auth on, auth
|
||||
off, 429 + Retry-After, SSE, pagination walk, mutation → reload, pause, settings round
|
||||
trip, static/ETag 304, bodyless POST) + 3 ungated pure guards (contract-covers-routes,
|
||||
drift guard a route ⊆ yaml, drift guard b: 55 yaml operations == router.routes.len).
|
||||
ci.yml UNTOUCHED — its test job already runs `zig build test -Dintegration` (line 25).
|
||||
- Environment fully real except two seams: real Manager/reload over an in-memory config
|
||||
DB, real Sessions/ApiLimiter/Hub (heap Hub per W5 rule), seeded querylog via
|
||||
BatchWriter. The pool transport and fetcher are never invoked —
|
||||
POST /api/blocklists/update runs in the walk BEFORE any source row exists (hermetic).
|
||||
- SSE reader de-frames chunked transfer coding (the unbuffered SSE writer emits one frame
|
||||
as many chunks; std's chunked writer emits each chunk's closing CRLF lazily — reading
|
||||
separators eagerly deadlocks until the next heartbeat).
|
||||
- Heartbeat asserted against the real 15 s interval (`live.heartbeat_interval` is not
|
||||
injectable): the SSE test runs ~15-20 s in a 40 s budget; the integration suite gained
|
||||
roughly 45 s of wall clock. Accepted rather than weakening the assertion.
|
||||
- Pause test drives Handler.handle directly with real DNS packets (phase7 queryFor
|
||||
pattern): blocked → API pause → forwarded (paused_queries bumps) → unpause → blocked.
|
||||
- Mutation-reload observed as generation 1 → 2 plus the rule live via GET /api/lookup.
|
||||
- Contract markers asserted by lookup into router.routes (method+pattern → auth and
|
||||
rate_limit equality, exact one-to-one coverage); response shapes reference the
|
||||
handlers' pub types where they exist; the strict settings parse also proves
|
||||
`web.password*` never serializes.
|
||||
- One expected `warn` line ("web login refused") from the wrong-password case; zero err
|
||||
lines suite-wide; no secrets logged.
|
||||
|
||||
---
|
||||
|
||||
## Module layout (new)
|
||||
|
||||
src/web/{server,router,http_util,auth,api_limiter,sse,static,openapi,metrics}.zig,
|
||||
src/web/routes.zig (W8), src/web/handlers/{auth,stats,queries,clients,groups,blocklists,
|
||||
rules,local,lookup,pause,settings,upstream_health,health,version}.zig,
|
||||
src/server/{query_sink,local_tables}.zig, src/web/openapi.yaml, web/dist-placeholder/,
|
||||
src/web/web_integration_test.zig.
|
||||
|
||||
## File ownership
|
||||
|
||||
W1 queries_repo; W2 other repositories/*; W3 web/{server,router,http_util}; W4
|
||||
web/{auth,api_limiter} + model/validate additions; W5 server/query_sink, web/sse,
|
||||
storage/logger (additive), server/handler (sink rename) + phase7 test compile fixes; W6
|
||||
web/handlers read set + web/metrics; W7 web/handlers mutation set (incl. mutations.zig,
|
||||
upstreams.zig) + server/local_tables + server/handler (local_tables read path; AFTER W5); W8 web/{static,openapi,routes} +
|
||||
openapi.yaml + dist-placeholder + build.zig; W9 app/cli/main; W10 its test file + ci.yml.
|
||||
Orchestrator: src/tests.zig, spec. Within-wave parallel sessions never share a file;
|
||||
handler.zig is touched by W5 then W7, strictly sequenced.
|
||||
|
||||
## Acceptance (milestone complete)
|
||||
|
||||
- [ ] Both suites + cross green; fmt clean; zero err-level lines in integration output.
|
||||
- [ ] Every PLAN:537-550 endpoint except certs/reload (ruling 2) implemented, documented in
|
||||
openapi.yaml, and contract-tested; auth on/off matrix green.
|
||||
- [ ] SSE live stream works end to end with per-IP cap; fanout precedes persistence.
|
||||
- [ ] /metrics scrapes clean (golden test); /api/health rollup correct.
|
||||
- [ ] W9 smoke transcript: boot, API answers, login round trip, SIGTERM 0.
|
||||
- [ ] Placeholder UI loads at /; assets ETag/304/gzip machinery proven.
|
||||
- [ ] Spec As-built synced per session.
|
||||
|
||||
## Review (Codex, As built)
|
||||
|
||||
Round 1: 6 important + 3 minor, all fixed.
|
||||
- local.zig: all six mutation paths hold `config_lock` across write AND rebuild+swap, so
|
||||
LocalTables swaps publish in database-write order (deadlock-checked: no callee takes
|
||||
the lock).
|
||||
- settings/auth/server/app: GET /api/settings reads under `config_lock`. New
|
||||
`auth.LiveHash` (mutex holder on `WebState.live_hash`, generation-checked — see round
|
||||
2): login copies the hash out under a short lock and verifies OUTSIDE it; settings PUT
|
||||
dupes the new hash with gpa BEFORE the transaction, installs + clears sessions after
|
||||
commit; `sessionAuth` gates on `live_hash.enabled()`, so a first password set via PUT
|
||||
locks routes without restart. Boot hash borrows the config arena (owned=false);
|
||||
replacements are gpa-owned, freed on the next install or deinit (app.zig defer /
|
||||
Bench.deinit / Env.destroy).
|
||||
- mutations.zig `checkUpstream`: candidate validated beside a companion enabled upstream
|
||||
(URL collision-avoided), so disabled upstreams are creatable. upstreams.zig
|
||||
`applyUpdate` gained the missing last-enabled guard: disabling the last enabled
|
||||
upstream is 409 (delete path already had it; create needs none).
|
||||
- clients.zig: prefixes canonicalized via platform/address.zig (parse zeroes host bits;
|
||||
RFC 5952 text) before a set-level duplicate check (409, constraint-map ruling) and
|
||||
whole-list validation through the real validate.validate; canonical text is stored.
|
||||
- api_limiter.zig: full table fails closed — `bucketLocked` first evicts a bucket that
|
||||
refills to capacity with sse==0 (behaviorally identical to fresh; sweep's idle window
|
||||
is churn hysteresis, not correctness), else `check` refuses with retry_after and
|
||||
`tryAcquireSse` returns false. `untracked` is now a subset of `refused`.
|
||||
- static.zig: `acceptsGzip` parses every entry per the header grammar (specific gzip
|
||||
beats wildcard; malformed q = entry unusable). Dev mode realpath-checks containment of
|
||||
target AND index fallback (no-follow-on-open would miss symlinked intermediate
|
||||
directories).
|
||||
- web_integration_test.zig: drift guard (a) requires the method key inside the specific
|
||||
path's yaml block; a negative test doctors the yaml (method swap between two paths)
|
||||
and proves the guard bites.
|
||||
|
||||
Round 2: 1 important — login could copy the old hash, race a password-change PUT
|
||||
(install + clearAll), finish argon2 verification, and mint a session with the revoked
|
||||
password. Fixed via a generation counter on LiveHash: applyLogin verifies the copy
|
||||
outside any lock, then `confirmSession` checks the generation and inserts the session
|
||||
digest under LiveHash.mutex (lock order: live_hash → sessions, single direction at every
|
||||
site); a changed generation denies the login.
|
||||
|
||||
Round 3: 2 minors, both fixed. (a) install-then-clearAll had a gap where a legitimate
|
||||
new-password cookie could be minted and then wiped — `install` is replaced by
|
||||
`installAndRevoke(io, gpa, sessions, new_hash)`: swap, generation bump, and nested
|
||||
clearAll in one LiveHash-ordered operation; a confirm either precedes it (wiped) or
|
||||
follows it (stale generation, denied). (b) confirmSession no longer holds LiveHash.mutex
|
||||
across entropy/clock work: token (randomSecure, zeroed on exit) and timestamp are
|
||||
generated before the lock; only the generation check + `createWithToken` digest insert
|
||||
run under the ordered locks. `Sessions.create` removed (createWithToken is the seam).
|
||||
|
||||
Round 4: no findings.
|
||||
|
||||
## Anti-requirements
|
||||
|
||||
- No React/SPA content (milestone 9). No DoH/DoT server or certs/reload (Phase 9). No
|
||||
docs rendering (Phase 10). No WebSocket. No HTTP/2, no TLS on the web port. No external
|
||||
schema validator or yaml parser. No new DB schema migration without an explicit
|
||||
orchestrator ruling (27). No pause persistence. No per-request timeouts. No changes to
|
||||
dns/, filter/matcher, cache, or the DNS pipeline order.
|
||||
Reference in New Issue
Block a user