milestone 28: query provenance — every logged query is exactly explainable
Gates / frontend (push) Successful in 1m36s
Gates / test (push) Successful in 1m56s
Gates / test-aarch64 (push) Successful in 7m37s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 13s
CI / gates (push) Successful in 19m4s
Gates / frontend (push) Successful in 1m36s
Gates / test (push) Successful in 1m56s
Gates / test-aarch64 (push) Successful in 7m37s
Gates / package (push) Successful in 9m12s
Gates / container (push) Successful in 13s
CI / gates (push) Successful in 19m4s
query rows gain qclass, rcode, group, policy action and reason, the matched rule or list entry with its source, cname and safe-search targets, route kind, forward zone, and the resolver that actually answered — the pool and local markers die. servfails are logged and name the resolver that lost; post-parse protocol refusals become rows. a detail page at /queries/:id renders the ordered explanation, and coverage watermarks distinguish an empty history from a missing one. the schema fingerprint changes: existing query history is recreated with the old file kept aside and the reset filed as a resolved diagnostic. fixes an oversized udp reply being rebuilt as noerror, which handed clients a truncated nxdomain as success.
This commit is contained in:
+218
-10
@@ -226,14 +226,47 @@ paths:
|
||||
"503":
|
||||
$ref: "#/components/responses/Unavailable"
|
||||
|
||||
/api/queries/{id}:
|
||||
get:
|
||||
summary: One query, fully explained
|
||||
description: |
|
||||
The full provenance of one logged query: what was asked, which group's
|
||||
policy applied, what that policy decided and matched on, what was
|
||||
rewritten, where the answer came from, and what the client received.
|
||||
Every field is a fact recorded when the query was answered, so a group
|
||||
or blocklist renamed since keeps the name it had.
|
||||
parameters:
|
||||
- name: id
|
||||
in: path
|
||||
required: true
|
||||
schema: { type: integer, minimum: 1 }
|
||||
responses:
|
||||
"200":
|
||||
description: The query.
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
$ref: "#/components/schemas/QueryDetail"
|
||||
"401":
|
||||
$ref: "#/components/responses/Unauthorized"
|
||||
"404":
|
||||
$ref: "#/components/responses/NotFound"
|
||||
"429":
|
||||
$ref: "#/components/responses/RateLimited"
|
||||
"500":
|
||||
$ref: "#/components/responses/Internal"
|
||||
"503":
|
||||
$ref: "#/components/responses/Unavailable"
|
||||
|
||||
/api/queries/live:
|
||||
get:
|
||||
summary: Live query stream (server-sent events)
|
||||
description: |
|
||||
`text/event-stream`. The stream opens with `retry: 3000`, then sends
|
||||
one `event: query` frame per resolved query whose `data:` line is a
|
||||
JSON object with the `/api/queries` row fields minus `id` (the entry
|
||||
precedes persistence). A `: ping` comment goes out every 15 seconds.
|
||||
`Provenance` object — the body of `/api/queries/{id}` without its `id`,
|
||||
which does not exist yet because the entry precedes its own insert.
|
||||
A `: ping` comment goes out every 15 seconds.
|
||||
A client that falls more than 64 events behind is disconnected and
|
||||
should re-sync via `/api/queries` after reconnecting. Connections
|
||||
per address are capped by `web.sse_max_connections_per_ip`; the
|
||||
@@ -1855,9 +1888,33 @@ components:
|
||||
type: boolean
|
||||
description: False when no password is configured; no cookie is set.
|
||||
|
||||
PolicyAction:
|
||||
type: string
|
||||
description: |
|
||||
Whether the filtering policy reached a verdict. `not_evaluated` is the
|
||||
honest answer for a query answered before filtering could apply, and is
|
||||
not the same as `allow`.
|
||||
enum: [not_evaluated, allow, block]
|
||||
|
||||
PolicyReason:
|
||||
type: string
|
||||
description: |
|
||||
Why the policy landed where it did. The first nine are the matcher's own
|
||||
verdicts; the rest name a pipeline step that decided without consulting
|
||||
the matcher.
|
||||
enum: [rule_allow_exact, rule_block_exact, rule_allow_wildcard, rule_block_wildcard, rule_allow_regex, rule_block_regex, blocklist_exception, blocklist_domain, blocklist_wildcard, local_record, forward_zone, non_in_class, paused, snapshot_unavailable, no_match, protocol_error]
|
||||
|
||||
RouteKind:
|
||||
type: string
|
||||
description: Where the answer the client received came from.
|
||||
enum: [blocked, local, forward_zone, upstream, cache, rejected]
|
||||
|
||||
QueryRow:
|
||||
type: object
|
||||
required: [id, ts, domain, client_ip, qtype, blocked, block_reason, response_time_us, cache_hit, upstream]
|
||||
description: |
|
||||
The summary projection the query-log table scans. The full provenance of
|
||||
a row is one request away at `/api/queries/{id}`.
|
||||
required: [id, ts, domain, client_ip, qtype, qclass, rcode, blocked, response_time_us, cache_hit, upstream, policy_action, policy_reason, route_kind]
|
||||
properties:
|
||||
id: { type: integer }
|
||||
ts:
|
||||
@@ -1868,10 +1925,11 @@ components:
|
||||
qtype:
|
||||
type: integer
|
||||
nullable: true
|
||||
qclass: { type: integer }
|
||||
rcode:
|
||||
type: integer
|
||||
description: The twelve-bit EDNS extended code, not the four header bits alone.
|
||||
blocked: { type: boolean }
|
||||
block_reason:
|
||||
type: string
|
||||
description: Empty when the query was not blocked.
|
||||
response_time_us:
|
||||
type: integer
|
||||
nullable: true
|
||||
@@ -1880,11 +1938,155 @@ components:
|
||||
nullable: true
|
||||
upstream:
|
||||
type: string
|
||||
description: Empty for cache hits and local answers.
|
||||
description: Empty for cache hits, local answers and blocked queries.
|
||||
policy_action:
|
||||
$ref: "#/components/schemas/PolicyAction"
|
||||
policy_reason:
|
||||
$ref: "#/components/schemas/PolicyReason"
|
||||
route_kind:
|
||||
$ref: "#/components/schemas/RouteKind"
|
||||
|
||||
ProvenanceRequest:
|
||||
type: object
|
||||
required: [time, domain, client, qtype, qclass]
|
||||
properties:
|
||||
time:
|
||||
type: integer
|
||||
description: Unix seconds.
|
||||
domain: { type: string }
|
||||
client: { type: string }
|
||||
qtype:
|
||||
type: integer
|
||||
nullable: true
|
||||
qclass: { type: integer }
|
||||
|
||||
ProvenanceGroup:
|
||||
type: object
|
||||
description: |
|
||||
The client's filtering group at the time of the query, as a historical
|
||||
fact: the id may name a group since renamed or deleted.
|
||||
required: [id, name]
|
||||
properties:
|
||||
id:
|
||||
type: integer
|
||||
nullable: true
|
||||
name: { type: string }
|
||||
|
||||
ProvenancePolicy:
|
||||
type: object
|
||||
required: [action, reason, matched, source_id, source_name]
|
||||
properties:
|
||||
action:
|
||||
$ref: "#/components/schemas/PolicyAction"
|
||||
reason:
|
||||
$ref: "#/components/schemas/PolicyReason"
|
||||
matched:
|
||||
type: string
|
||||
description: The rule pattern or list entry that decided; empty when nothing matched.
|
||||
source_id:
|
||||
type: integer
|
||||
nullable: true
|
||||
source_name:
|
||||
type: string
|
||||
description: The blocklist the match came from; empty for a rule.
|
||||
|
||||
ProvenanceRewrites:
|
||||
type: object
|
||||
required: [cname_target, safe_search_target]
|
||||
properties:
|
||||
cname_target:
|
||||
type: string
|
||||
description: Set when the decision was made about a CNAME target rather than the queried name.
|
||||
safe_search_target: { type: string }
|
||||
|
||||
ProvenanceRoute:
|
||||
type: object
|
||||
required: [kind, forward_zone, upstream]
|
||||
properties:
|
||||
kind:
|
||||
$ref: "#/components/schemas/RouteKind"
|
||||
forward_zone: { type: string }
|
||||
upstream:
|
||||
type: string
|
||||
description: |
|
||||
Non-empty only for an attempted upstream or forward-zone exchange,
|
||||
including one that failed. Already redacted: the userinfo, path,
|
||||
query and fragment of a resolver url never reach here.
|
||||
|
||||
ProvenanceResponse:
|
||||
type: object
|
||||
required: [rcode, duration_us]
|
||||
properties:
|
||||
rcode:
|
||||
type: integer
|
||||
description: The twelve-bit EDNS extended code, not the four header bits alone.
|
||||
duration_us:
|
||||
type: integer
|
||||
nullable: true
|
||||
|
||||
Provenance:
|
||||
type: object
|
||||
description: |
|
||||
One query, fully explained, in the order a query meets the pipeline. The
|
||||
`data:` payload of a live-stream `event: query` frame is exactly this.
|
||||
required: [request, group, policy, rewrites, route, response]
|
||||
properties:
|
||||
request:
|
||||
$ref: "#/components/schemas/ProvenanceRequest"
|
||||
group:
|
||||
$ref: "#/components/schemas/ProvenanceGroup"
|
||||
policy:
|
||||
$ref: "#/components/schemas/ProvenancePolicy"
|
||||
rewrites:
|
||||
$ref: "#/components/schemas/ProvenanceRewrites"
|
||||
route:
|
||||
$ref: "#/components/schemas/ProvenanceRoute"
|
||||
response:
|
||||
$ref: "#/components/schemas/ProvenanceResponse"
|
||||
|
||||
QueryDetail:
|
||||
type: object
|
||||
description: |
|
||||
`Provenance` plus the row id. Written out rather than composed with
|
||||
`allOf` so the drift guard reads one property list per schema.
|
||||
required: [id, request, group, policy, rewrites, route, response]
|
||||
properties:
|
||||
id: { type: integer }
|
||||
request:
|
||||
$ref: "#/components/schemas/ProvenanceRequest"
|
||||
group:
|
||||
$ref: "#/components/schemas/ProvenanceGroup"
|
||||
policy:
|
||||
$ref: "#/components/schemas/ProvenancePolicy"
|
||||
rewrites:
|
||||
$ref: "#/components/schemas/ProvenanceRewrites"
|
||||
route:
|
||||
$ref: "#/components/schemas/ProvenanceRoute"
|
||||
response:
|
||||
$ref: "#/components/schemas/ProvenanceResponse"
|
||||
|
||||
Coverage:
|
||||
type: object
|
||||
description: |
|
||||
How much of the requested window the query log can still answer for.
|
||||
Retention deletes rows and advances the watermark in one transaction, so
|
||||
a client can tell an empty window from a pruned one instead of charting
|
||||
the gap as zero.
|
||||
required: [complete, available_since]
|
||||
properties:
|
||||
complete:
|
||||
type: boolean
|
||||
description: |
|
||||
True only when the window's lower bound is at or after
|
||||
`available_since`. A request with no lower bound asks about all of
|
||||
history, which no file that has ever pruned can promise.
|
||||
available_since:
|
||||
type: integer
|
||||
description: The oldest instant the file is complete for, unix seconds.
|
||||
|
||||
QueriesPage:
|
||||
type: object
|
||||
required: [queries, next_before]
|
||||
required: [queries, next_before, coverage]
|
||||
properties:
|
||||
queries:
|
||||
type: array
|
||||
@@ -1894,6 +2096,8 @@ components:
|
||||
type: integer
|
||||
nullable: true
|
||||
description: Cursor for the next page; null on the last page.
|
||||
coverage:
|
||||
$ref: "#/components/schemas/Coverage"
|
||||
|
||||
DiagnosticEvent:
|
||||
type: object
|
||||
@@ -1983,7 +2187,7 @@ components:
|
||||
|
||||
StatsTotals:
|
||||
type: object
|
||||
required: [period, since, until, queries, blocked, cached, clients, avg_response_time_us]
|
||||
required: [period, since, until, queries, blocked, cached, clients, avg_response_time_us, coverage]
|
||||
properties:
|
||||
period:
|
||||
type: string
|
||||
@@ -2004,6 +2208,8 @@ components:
|
||||
type: integer
|
||||
nullable: true
|
||||
description: Null when no query in the window recorded a time.
|
||||
coverage:
|
||||
$ref: "#/components/schemas/Coverage"
|
||||
|
||||
Bucket:
|
||||
type: object
|
||||
@@ -2018,7 +2224,7 @@ components:
|
||||
|
||||
StatsTimeseries:
|
||||
type: object
|
||||
required: [period, since, until, bucket_seconds, buckets]
|
||||
required: [period, since, until, bucket_seconds, buckets, coverage]
|
||||
properties:
|
||||
period:
|
||||
type: string
|
||||
@@ -2030,6 +2236,8 @@ components:
|
||||
type: array
|
||||
items:
|
||||
$ref: "#/components/schemas/Bucket"
|
||||
coverage:
|
||||
$ref: "#/components/schemas/Coverage"
|
||||
|
||||
Lookup:
|
||||
type: object
|
||||
|
||||
Reference in New Issue
Block a user