milestone 28: query provenance — every logged query is exactly explainable
Gates / frontend (push) Successful in 1m36s
Gates / test (push) Successful in 1m56s
Gates / test-aarch64 (push) Successful in 7m37s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 13s
CI / gates (push) Successful in 19m4s
Gates / frontend (push) Successful in 1m36s
Gates / test (push) Successful in 1m56s
Gates / test-aarch64 (push) Successful in 7m37s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 13s
CI / gates (push) Successful in 19m4s
query rows gain qclass, rcode, group, policy action and reason, the matched rule or list entry with its source, cname and safe-search targets, route kind, forward zone, and the resolver that actually answered — the pool and local markers die. servfails are logged and name the resolver that lost; post-parse protocol refusals become rows. a detail page at /queries/:id renders the ordered explanation, and coverage watermarks distinguish an empty history from a missing one. the schema fingerprint changes: existing query history is recreated with the old file kept aside and the reset filed as a resolved diagnostic. fixes an oversized udp reply being rebuilt as noerror, which handed clients a truncated nxdomain as success.
This commit is contained in:
+27
-6
@@ -4,7 +4,7 @@ nxdns serves its admin API itself, on `web.bind:web.port` (default port 8080), a
|
||||
|
||||
The machine-readable contract is `src/web/openapi.yaml`, which the running server hands out unauthenticated at `GET /api/openapi.yaml`. Request and response schemas for every operation live there. When this page and the YAML disagree, the YAML wins.
|
||||
|
||||
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 60 of its entries.
|
||||
The route table is `src/web/routes.zig`; the [Operations](#operations) table below carries all 61 of its entries.
|
||||
|
||||
## Conventions
|
||||
|
||||
@@ -45,7 +45,7 @@ A token bucket per client address: capacity and refill are both `web.api_rate_li
|
||||
`GET /api/queries/live` is server-sent events over chunked transfer, `Content-Type: text/event-stream`, `Cache-Control: no-store`.
|
||||
|
||||
- The stream opens with `retry: 3000`, so a browser `EventSource` reconnects on its own after a drop.
|
||||
- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload carries the `GET /api/queries` row fields minus `id` (a live entry precedes persistence): `ts`, `domain`, `client_ip`, `qtype`, `blocked`, `block_reason`, `response_time_us`, `cache_hit`, `upstream`.
|
||||
- Each query is one frame: `event: query` and a single `data:` line of JSON. The payload is the `Provenance` object — the body of `GET /api/queries/{id}` without its `id`, which does not exist yet because a live entry precedes its own insert. Its six groups are `request`, `group`, `policy`, `rewrites`, `route` and `response`.
|
||||
- A `: ping` comment heartbeat goes out after 15 s of quiet, keeping middleboxes from reaping the idle connection.
|
||||
- Each subscriber buffers up to 64 entries. A client too slow for the query rate overflows its buffer and the server ends the stream cleanly after delivering what the buffer held — queries are never held back for a slow reader. There is no gap marker: on reconnect, re-sync through `GET /api/queries`, which has the missed rows.
|
||||
- Connections per client address are capped at `web.sse_max_connections_per_ip` (default 3); over the cap is a 429. The cap binds loopback too. The server holds at most 32 concurrent streams in total; when all slots are taken, the answer is a 503.
|
||||
@@ -104,6 +104,7 @@ Auth `open` means no session is required; `session` means a valid session cookie
|
||||
| POST | `/api/auth/login` | open | counted | runtime action | Log in |
|
||||
| POST | `/api/auth/logout` | session | counted | runtime action | Log out |
|
||||
| GET | `/api/queries` | session | counted | read | Query log page |
|
||||
| GET | `/api/queries/{id}` | session | counted | read | One query, fully explained |
|
||||
| GET | `/api/queries/live` | session | exempt | read | Live query stream (server-sent events) |
|
||||
| GET | `/api/stats` | session | counted | read | Totals for a period |
|
||||
| GET | `/api/stats/timeseries` | session | counted | read | Bucketed counts for a period |
|
||||
@@ -174,9 +175,11 @@ In file mode `PUT /api/settings` is refused with the 403 above, password changes
|
||||
|
||||
Request and response schemas for every operation live in the OpenAPI document: `src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a running server.
|
||||
|
||||
### Block reasons
|
||||
### Policy reasons
|
||||
|
||||
Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup` answer. The tag names the level that decided the query, and the levels are listed here in the order they are consulted — the first one that matches wins, so a rule always outranks a list.
|
||||
Two places carry the same closed set of tags: `policy_reason` on a `GET /api/queries` row and on a `GET /api/queries/{id}` body (where it is `policy.reason`, and where the live stream sends the same field), and `reason` on a `GET /api/lookup` answer. The tag names what decided the query.
|
||||
|
||||
The first nine are the matcher's own verdicts, listed in the order they are consulted — the first that matches wins, so a rule always outranks a list.
|
||||
|
||||
| Tag | Decided by |
|
||||
| --- | --- |
|
||||
@@ -190,6 +193,24 @@ Three places carry the same tag: `block_reason` on a `GET /api/queries` row, `bl
|
||||
| `blocklist_domain` | A plain name in a downloaded list |
|
||||
| `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list |
|
||||
|
||||
`/api/lookup` also answers `none` when nothing matched. A query row never carries `none`: `block_reason` is null unless the query was blocked.
|
||||
The rest name a pipeline step that answered the query without consulting the matcher, and appear on a query row only.
|
||||
|
||||
A `cname:` prefix means the decision landed on a CNAME target rather than on the name the client asked for, so `cname:blocklist_domain` reads as "the list blocks a name this answer redirects to". Only `/api/queries` and the live stream show the prefix; `/api/lookup` does not follow CNAMEs.
|
||||
| Tag | Decided by |
|
||||
| --- | --- |
|
||||
| `local_record` | A configured local record, answered before filtering |
|
||||
| `forward_zone` | A configured forward zone, answered before filtering |
|
||||
| `non_in_class` | The question was not class IN, so no rule could apply |
|
||||
| `paused` | Filtering was paused |
|
||||
| `snapshot_unavailable` | No filter snapshot was published yet, so the query went unfiltered |
|
||||
| `no_match` | The matcher evaluated the name and nothing matched |
|
||||
| `protocol_error` | A parsed request refused on protocol grounds — BADVERS, NOTIMP, a malformed EDNS OPT |
|
||||
|
||||
`policy_action` says which way the verdict went: `block`, `allow`, or `not_evaluated` for a query answered before any policy could apply. `/api/lookup` answers `none` when nothing matched, where a query row says `no_match`.
|
||||
|
||||
`route_kind` says where the answer came from: `blocked`, `local`, `forward_zone`, `upstream`, `cache` or `rejected`.
|
||||
|
||||
A non-empty `rewrites.cname_target` on a query detail means the decision landed on a CNAME target rather than on the name the client asked for; `policy.reason` is then the target's own reason. `/api/lookup` does not follow CNAMEs.
|
||||
|
||||
### Coverage
|
||||
|
||||
`GET /api/queries`, `GET /api/stats` and `GET /api/stats/timeseries` each answer with a `coverage` object: `available_since` is the oldest instant the query log is still complete for, and `complete` is true only when the window the request asked about starts at or after it. Retention deletes rows and advances the watermark in one transaction, so a client can tell an empty window from a pruned one instead of charting the gap as zero. A request with no lower bound at all asks about the whole of history, and is never complete.
|
||||
|
||||
@@ -131,7 +131,7 @@ Process log and query log behavior.
|
||||
|---|---|---|---|---|---|
|
||||
| `logging.level` | enum `.err` \| `.warn` \| `.info` \| `.debug` | `.info` | — | one of the four tags; stored as `"error"` / `"warn"` / `"info"` / `"debug"` | log threshold (`src/platform/logging.zig`) |
|
||||
| `logging.retention_days` | u16 | 30 | days | at least 1 | query-log pruning cutoff (`src/storage/retention.zig`) and the client tracker's last-seen cutoff (`src/server/clients.zig`) |
|
||||
| `logging.query_log_buffer_max` | u32 | 10000 | entries | 1–1000000 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`) |
|
||||
| `logging.query_log_buffer_max` | u32 | 10000 | entries | 1–37449 | in-memory query-log ring size and backpressure cap (`src/storage/logger.zig`); the ceiling is derived at compile time from `@sizeOf(logger.Entry)` so the queue's worst case stays within 64 MiB, and it moves whenever the entry's width does |
|
||||
| `logging.query_log_flush_interval_s` | u16 | 60 | seconds | 0–3600 | how long the query-log writer gathers entries before committing them in one transaction (`src/storage/logger.zig`); see the note below |
|
||||
| `logging.hide_domains` | bool | false | — | — | the query log stores a hidden marker instead of the domain |
|
||||
| `logging.hide_client_ips` | bool | false | — | — | the query log stores a hidden marker instead of the client address |
|
||||
@@ -385,10 +385,12 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
|
||||
| `MissingDefaultGroup` | no group is named `default` |
|
||||
| `DuplicateGroupName` | two groups share a `name` |
|
||||
| `EmptyGroupName` | a group `name` is empty |
|
||||
| `GroupNameTooLong` | a group `name` is longer than 64 bytes; it is copied into every logged query |
|
||||
| `UnknownGroup` | a client, prefix, group source or rule names a group that is not declared |
|
||||
| `BadClientIp` / `DuplicateClientIp` | a client `ip` is unparseable, or collides after canonicalization |
|
||||
| `BadClientPrefix` / `DuplicateClientPrefix` | the same for a `client_prefixes.prefix` |
|
||||
| `BadSourceUrl` / `DuplicateSourceUrl` / `EmptySourceName` | blocklist source fields |
|
||||
| `SourceNameTooLong` | a blocklist source `name` is longer than 64 bytes; it is copied into every logged query |
|
||||
| `UnknownSource` / `DuplicateGroupSource` | `group_sources` links |
|
||||
| `BadRulePattern` | a rule `pattern` does not match its `kind` |
|
||||
| `BadLocalRecordName` / `BadLocalRecordValue` / `DuplicateLocalRecord` | local record fields |
|
||||
@@ -397,7 +399,7 @@ The error set is `validate.ValidateError` in `src/config/validate.zig`:
|
||||
| `BadTimeout` | a timeout is outside 100–120000 ms, or `attempt` is above `total` |
|
||||
| `BadTtl` | `blocking.ttl`, `cache.negative_ttl_max`, a record `ttl`, `web.session_ttl_hours` or `blocklist_update.interval_hours` outside its range |
|
||||
| `BadCacheSize` | `cache.size` outside 1–1000000 |
|
||||
| `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 1–1000000 |
|
||||
| `BadRetention` | `logging.retention_days` below 1, or `logging.query_log_buffer_max` outside 1–37449 |
|
||||
| `BadFlushInterval` | `logging.query_log_flush_interval_s` above 3600 |
|
||||
| `BadLogRotation` | `logging.max_size_mb` or `logging.max_files` below 1 |
|
||||
| `BadDiskThresholds` | a threshold below 1, or `min_free_mb` above `warn_free_mb` |
|
||||
|
||||
Reference in New Issue
Block a user