milestone 21: abp list exceptions and a regex rule kind

PLAN.md

@@ -23,17 +23,19 @@ Serves a household LAN (≈2–20 devices). Portfolio-grade public repo with ext
 - DNS server for LAN clients: UDP/53, TCP/53, DoH server, DoT server.
 - Upstream resolution: DoH (HTTP/1.1), DoT.
 - Local DNS records (A/AAAA/CNAME) + conditional forwarding (zone → designated resolver, plain UDP/TCP allowed).
-- Domain filtering: blocklists (hosts/domains/ABP), custom rules (allow/block; exact, parent-walk, wildcard), CNAME uncloaking (depth 8), per-group safe-search rewrite.
+- Domain filtering: blocklists (hosts/domains/ABP, including `@@||name^` exception lines), custom rules (allow/block; exact, parent-walk, wildcard, regex), CNAME uncloaking (depth 8), per-group safe-search rewrite.
 - DNS caching: positive + negative, in-memory only.
 - Client/group model: IPv4 + IPv6 parity, per-client group assignment, per-group source assignments.
 - Query logging + analytics: async batched writes to SQLite (WAL), retention cleanup, dashboard + time buckets, live SSE stream.
 - Web app + REST API: LAN/Tailscale admin UI, optional password auth, OpenAPI schema + CI contract tests.
 - Observability: upstream health API + UI, disk monitor with UI banner, bounded log rotation, Prometheus `/metrics`.
-- Ops: DB-as-truth config, `nxdns export`/`import` (ZON), scheduled + manual blocklist updates, TLS cert watcher + reload, auto-migration on upgrade, systemd service + Dockerfile + compose.
+- Ops: config authority chosen by the invocation (database, or a file named by `--config`), `nxdns export`/`import` (ZON), scheduled + manual blocklist updates, TLS cert watcher + reload, auto-migration on upgrade, systemd service + Dockerfile + compose.
 
 ### 2.2 Out of Scope (permanent scope decisions, not deferrals)
 
-- Regex rules. Wildcards + parent-walk cover the real use cases; regex on the DNS hot path means ReDoS exposure plus an immature dependency or a homegrown engine. Regex lines in blocklists are counted, skipped, and the skip count is surfaced in the UI.
+- Regex from downloaded blocklists. A list is other people's code running on the household's DNS, and the engine exists for rules the operator wrote. Regex lines in blocklists stay counted and skipped, and the skip count stays surfaced in the UI. (Operator regex rules themselves are **in** scope as of milestone 21: the ReDoS objection that once ruled them out is answered by `filter/regex.zig`, a homegrown Pike VM whose running time is bounded by program length × name length by construction, with no dependency and no backtracking. It is reached only after every hash and wildcard level has missed, and only on a cache miss.)
+- ABP syntax beyond domain anchors and `@@||name^` exceptions. `$dnstype`, `$dnsrewrite`, `$client`, `$denyallow` and every browser modifier stay unsupported and counted; the one tolerated modifier is a `$important` suffix on an exception line, which changes nothing about where that exception lands.
+- Partial-segment wildcards (`ads*.example.com`) as a rule kind. The regex kind covers the need without a second globbing dialect.
 - DHCP server.
 - DNSSEC validation (DO bit passthrough only).
 - DoQ (QUIC), HTTP/2 upstream transport.
@@ -74,8 +76,8 @@ Verified: 0.16.0 ships `std.crypto.tls.Client` only. There is no server-side TLS
 
 ### 3.5 Config Format + Truth Model (Decision F)
 
-- **DB is truth. Config file format is ZON** (`std.zon` parse + stringify — typed parsing into config structs, exact round-trip, stdlib-maintained, comments supported). No TOML: a third-party parser plus a hand-written serializer is two failure surfaces in the correctness-critical bootstrap/round-trip path, bought for syntax familiarity.
-- First start: if DB empty and `/etc/nxdns/config.zon` exists, validate → seed DB. Subsequent starts ignore the file.
+- **The invocation picks truth (see the next point). Config file format is ZON** (`std.zon` parse + stringify — typed parsing into config structs, exact round-trip, stdlib-maintained, comments supported). No TOML: a third-party parser plus a hand-written serializer is two failure surfaces in the correctness-critical round-trip path, bought for syntax familiarity.
+- The invocation picks the authority, and nothing else does (m20): `nxdns run` makes the database the configuration, `nxdns run --config FILE` makes the file the sole source and refuses the API routes that would edit configuration. There is no seeding and no first-start special case.
 - `nxdns export [--out file.zon]` dumps DB state as canonical ZON. `nxdns import <file.zon>` validates + replaces DB contents (`--force` if the DB holds configuration; client rows materialised from traffic do not count, and survive the replacement. First-seen/last-seen are runtime state, not configuration: they follow the address, so an import never restamps a device the DB already knew). Export/import = backup + host migration, **not** upgrades (§3.7).
 - No file watcher, no auto-regeneration.
 
@@ -96,14 +98,14 @@ Two SQLite files with opposite write profiles, isolated from each other:
 
 Blocklist domains are **not** stored in SQLite — they are a cache of re-downloadable remote artifacts, not config or state:
 
-- Each source compiles to `/var/lib/nxdns/blocklists/<source_id>.list`: normalized, one domain per line, small header (source URL, fetch time, count, checksum). Wildcard/regex-flavored lines: wildcards go to `<source_id>.wild`; regex lines are counted + skipped (count in metadata → UI).
+- Each source compiles to `/var/lib/nxdns/blocklists/<source_id>.list`: normalized, one domain per line, small header (source URL, fetch time, counts, checksum). Wildcard/regex/exception-flavored lines: wildcards go to `<source_id>.wild`, ABP exceptions (`@@||name^`) to `<source_id>.allow`; regex lines are counted + skipped (counts in metadata → UI). The checksum covers the three bodies in that order, so a source with no exceptions keeps the digest it had when only two existed and no upgrade forces a refetch.
 - `config.db` keeps source **metadata only** (`blocklist_sources`).
 - Startup + post-update: parse files into the immutable in-memory matcher (RCU swap, §9.5).
 - Corruption recovery is per-file: checksum mismatch → re-download one list.
 
 ### 3.9 Rule Model (Decision B)
 
-Rule kinds: `exact`, parent-walk (implicit via candidate chain), `wildcard` (`*` segment patterns, e.g. `*.doubleclick.net`, `ads.*.example.com`). Actions: `allow` | `block`. Kind is an explicit column — the model is extensible without breakage, but regex stays out of scope (§2.2).
+Rule kinds: `exact`, parent-walk (implicit via candidate chain), `wildcard` (`*` segment patterns, e.g. `*.doubleclick.net`, `ads.*.example.com`), `regex` (the linear-time engine of `filter/regex.zig`, matched unanchored against the whole normalized name). Actions: `allow` | `block`. Kind is an explicit column, which is what let the third kind arrive as one migration step rather than a schema break. A regex pattern is stored exactly as written — it is not a name, so it is never lowercased or dot-stripped — and is compiled at both edges: `config/validate.zig` refuses a bad one with the limit it hit, and `filter/rules.zig` compiles it once per snapshot.
 
 ### 3.10 Filtering Precedence
 
@@ -111,11 +113,25 @@ Rule kinds: `exact`, parent-walk (implicit via candidate chain), `wildcard` (`*`
 2. Exact/parent **block** rules
 3. Wildcard allow rules
 4. Wildcard block rules
-5. Blocklist domains
-6. Blocklist wildcards
+5. Regex allow rules
+6. Regex block rules
+7. Blocklist exceptions (`@@||name^`)
+8. Blocklist domains
+9. Blocklist wildcards
 
 Tie-break at same specificity: **allow wins**.
 
+Each level is checked against the whole candidate chain before the next level is
+checked against any, which is what makes an allow rule on a parent beat a block
+rule on the child.
+
+Two positions carry an argument rather than a preference. The regex levels come
+last among the operator rules because they are the only ones that are not a set
+lookup or a label walk: a regex runs only once every cheaper level has missed.
+Blocklist exceptions come below **every** operator level because a downloaded
+list may cancel what another list blocked and must never cancel what the
+operator decided — no list can open an allow hole the operator did not open.
+
 ### 3.11 Network Posture
 
 - Default web bind: LAN/Tailscale-friendly (non-loopback allowed).
@@ -128,9 +144,9 @@ IPv4 + IPv6 full parity for: client identity, rate limiting, logging, group assi
 
 ### 3.13 Filesystem Layout (FHS)
 
-- `/etc/nxdns/config.zon` — bootstrap (first start only).
+- `/etc/nxdns/config.zon` — the declarative source, read only when `run --config` names it. A file no flag names changes nothing.
 - `/var/lib/nxdns/config.db`, `/var/lib/nxdns/querylog.db`
-- `/var/lib/nxdns/blocklists/*.list|*.wild` (plus `*.raw.tmp|*.list.tmp|*.wild.tmp` during a refresh)
+- `/var/lib/nxdns/blocklists/*.list|*.wild|*.allow` (plus `*.raw.tmp|*.list.tmp|*.wild.tmp|*.allow.tmp` during a refresh)
 - `/var/log/nxdns/nxdns.log` — only in file output mode; default is stderr → journald.
 
 ### 3.14 Frontend Stack (Decision I)
@@ -173,7 +189,7 @@ Client DNS Query
 Response to client
 ```
 
-Cross-cutting: `ConfigManager` (bootstrap/import/export/settings), `BlocklistManager` (fetch/compile/swap), `Storage` (two SQLite DBs), `Cache`, `UpstreamHealth`, `DiskMonitor`, `Auth`, `Web API` (REST + SSE + metrics).
+Cross-cutting: `ConfigManager` (load/reconcile/import/export/settings), `BlocklistManager` (fetch/compile/swap), `Storage` (two SQLite DBs), `Cache`, `UpstreamHealth`, `DiskMonitor`, `Auth`, `Web API` (REST + SSE + metrics).
 
 ---
 
@@ -203,7 +219,8 @@ src/
   filter/                # pure
     matcher.zig  rules.zig  wildcard.zig
     parser_hosts.zig  parser_domains.zig  parser_abp.zig
-    fetcher.zig  compiler.zig            # list download -> compiled .list/.wild files
+    fetcher.zig  compiler.zig            # list download -> compiled .list/.wild/.allow files
+    regex.zig                            # linear-time Pike VM for operator regex rules
     safesearch.zig
 
   local/                 # pure
@@ -223,7 +240,8 @@ src/
     logger.zig  retention.zig  disk_monitor.zig
 
   config/
-    model.zig  bootstrap.zig  import.zig  export.zig  validate.zig   # all ZON via std.zon
+    model.zig  loader.zig  reconcile.zig  import.zig  export.zig    # all ZON via std.zon
+    validate.zig  faults.zig
 
   web/
     server.zig  router.zig  auth.zig  sse.zig  static.zig  metrics.zig  openapi.zig
@@ -276,13 +294,16 @@ For `{domain, group_id}` (the qtype travels with the query for logging and respo
 for matching):
 1. Normalize: lowercase, trim trailing dot.
 2. Build candidate chain (full, parent1, parent2, …).
-3. Explicit rules per §3.10 precedence, evaluated against every candidate in the chain.
-4. Group's blocklist domains (hash set over compiled lists), matched against the query name only.
-5. Group's blocklist wildcards, matched against every proper parent of the query name.
-6. No match → allow.
+3. Explicit rules per §3.10 precedence, allow before block at each level: exact rules against every candidate in the chain, then wildcard patterns and then regex patterns against the whole name (both kinds express their own reach, so neither walks the chain).
+4. Group's blocklist exceptions (`@@` entries), against every candidate in the chain. They cancel a block a list made and never one a rule made.
+5. Group's blocklist domains (hash set over compiled lists), matched against the query name only.
+6. Group's blocklist wildcards, matched against every proper parent of the query name.
+7. No match → allow.
 
-Blocklist entries do not parent-walk; only rules do (§3.9). ABP `||x.y^` emits both a domain entry
-`x.y` and a wildcard entry `x.y`, which together give domain-and-subdomains semantics.
+Blocklist *domain* entries do not parent-walk: they are matched against the query name alone. Wildcard
+entries match every proper parent, and exception entries walk the candidate chain the way rules do
+(§3.9), so `@@||good.ads.example^` also lifts `y.good.ads.example`. ABP `||x.y^` emits both a domain
+entry `x.y` and a wildcard entry `x.y`, which together give domain-and-subdomains semantics.
 
 ### 7.2 Group Assignment
 
@@ -341,7 +362,15 @@ Per-group boolean. Rewrites known engine domains to their safe-search CNAME targ
 
 `journal_mode=WAL`, `synchronous=NORMAL`, `foreign_keys=ON`, `busy_timeout` set.
 
-### 11.2 config.db Schema
+### 11.2 config.db Schema (v1 baseline)
+
+The DDL below is the **version 1** schema this plan froze, kept for the table
+shapes it argues for. It is not the live schema and must not be implemented
+against: that is `src/storage/config_schema.zig` plus the steps in
+`src/storage/migrations.zig`, currently at **version 4**. The steps since v1 add
+`upstreams.tls_name` (2), `blocklist_sources.exception_count` (3), and the
+`regex` rule kind (4) — so the `kind` CHECK below admits two of the three kinds
+the database now accepts, lacking `regex`.
 
 ```sql
 CREATE TABLE schema_version (version INTEGER NOT NULL);
@@ -471,52 +500,24 @@ Periodic delete of rows older than `retention_days`; scheduled checkpoint/VACUUM
 
 ## 12. Configuration
 
-### 12.1 Bootstrap ZON Shape
+### 12.1 Config ZON Shape
+
+The canonical shape is not duplicated here. It lives in
+[docs/reference/configuration.md](docs/reference/configuration.md), which is
+handwritten against `config/model.zig` and only partly guarded (the drift test
+covers settings-key rows, not the whole shape, so a new collection can go
+undocumented while the guard stays green), and `nxdns export` emits it. A copy in
+this document is how §12.1 came to describe an `.upstream.servers` field that
+never existed and to omit the required `.groups` and `.upstreams` — a sample
+nobody could load. The skeleton, for orientation only:
 
 ```zon
 .{
-    .upstream = .{
-        .servers = .{ "https://cloudflare-dns.com/dns-query", "tls://dns.google:853" },
-        .read_timeout_ms = 3000,
+    .groups = .{ .{ .name = "default" } },
+    .upstreams = .{ .{ .url = "https://dns.example/dns-query" } },
+    .rules = .{
+        .{ .group = "default", .pattern = "^ad[0-9]+-", .kind = .regex, .action = .block },
     },
-    .dns = .{
-        .bind_ipv4 = "0.0.0.0",
-        .bind_ipv6 = "::",
-        .port = 53,
-        .rate_limit = 1000,
-        .rate_window_seconds = 60,
-    },
-    .blocking = .{ .response = .zero, .ttl = 5 },       // .zero | .nxdomain
-    .cache = .{ .size = 10000, .negative_ttl_max = 3600 },
-    .web = .{
-        .enabled = true,
-        .bind = "0.0.0.0",
-        .port = 8080,
-        .password = "",                                  // empty => auth disabled
-        .session_ttl_hours = 24,
-        .api_rate_limit_per_min = 300,
-        .sse_max_connections_per_ip = 3,
-    },
-    .doh_server = .{ .enabled = false, .bind = "0.0.0.0", .port = 443,
-                     .cert_path = "/etc/nxdns/cert.pem", .key_path = "/etc/nxdns/key.pem" },
-    .dot_server = .{ .enabled = false, .bind = "0.0.0.0", .port = 853,
-                     .cert_path = "/etc/nxdns/cert.pem", .key_path = "/etc/nxdns/key.pem" },
-    .edns = .{ .ecs_mode = .strip },                     // .strip | .forward
-    .local_records = .{ .{ .name = "nas.lan", .rtype = .A, .value = "192.168.1.10" } },
-    .forward_zones = .{ .{ .zone = "lan.home", .resolver = "udp://192.168.1.1:53" } },
-    .logging = .{
-        .level = .info,
-        .retention_days = 30,
-        .query_log_buffer_max = 10000,
-        .hide_domains = false,
-        .hide_client_ips = false,
-        .output = .stderr,                               // .stderr | .syslog | .file
-        .file_path = "/var/log/nxdns/nxdns.log",
-        .max_size_mb = 50,
-        .max_files = 5,
-    },
-    .disk = .{ .min_free_mb = 200, .warn_free_mb = 500 },
-    .blocklist_update = .{ .enabled = true, .interval_hours = 24 },
 }
 ```
 
@@ -592,11 +593,11 @@ UDP server, TCP server, DoH + DoT upstream clients, pool + failover/backoff + he
 Exit: A/AAAA forwarding over UDP + TCP; health populated.
 
 ### Phase 4 — Storage + Config
-SQLite wrapper; config.db schema + migration runner; querylog.db schema + recreate-on-mismatch; repositories; ZON bootstrap + import/export; `nxdns check`.
-Exit: first start seeds DB from ZON; export → import round-trips byte-stable.
+SQLite wrapper; config.db schema + migration runner; querylog.db schema + recreate-on-mismatch; repositories; ZON loading + import/export; `nxdns check`.
+Exit: export → import round-trips byte-stable. (The ZON bootstrap this phase shipped was replaced in m20 by the two authority modes above.)
 
 ### Phase 5 — Filtering + Local DNS
-Rule matcher (exact/parent/wildcard); blocklist parsers → compiled file format; fetcher + scheduled update; RCU swap; per-group safe-search; local records; forward zones.
+Rule matcher (exact/parent/wildcard; `regex` added in m21); blocklist parsers → compiled file format; fetcher + scheduled update; RCU swap; per-group safe-search; local records; forward zones.
 Exit: precedence table validated by tests; local zone answers + conditional forwards work.
 
 ### Phase 6 — Cache + Rate Limit + Logging + Disk Monitor
@@ -623,7 +624,7 @@ Exit: documented deployment works end-to-end on the Pi 5.
 
 ## 17. Testing Strategy
 
-- **Unit**: DNS encode/decode; rule precedence + wildcard matcher; cache put/get/TTL rewrite; ZON bootstrap + export round-trip; rate limiter; migration runner (fresh + stepwise upgrade).
+- **Unit**: DNS encode/decode; rule precedence + wildcard matcher; cache put/get/TTL rewrite; ZON loading + export round-trip; rate limiter; migration runner (fresh + stepwise upgrade).
 - **Fuzz**: DNS parser malformed-packet fuzzing; blocklist parser fuzzing.
 - **Integration**: UDP/TCP query path; blocked path; allow-over-block; wildcard precedence; CNAME uncloaking block; local records + forward zones; upstream failover/backoff/health; disk-full degradation; querylog.db corruption recovery; API CRUD; auth on/off; SSE; contract tests.
 - **Manual**: `dig @pi example.com` / blocked domain / local record; DoH/DoT client checks; dashboard + live log.
@@ -697,7 +698,7 @@ longer the only path.
 | # | Decision |
 |---|----------|
 | A | Blocklists compile to flat files under `/var/lib/nxdns/blocklists/`; DB stores source metadata only |
-| B | Rule kinds: exact, parent-walk, wildcard. Regex permanently out of scope |
+| B | Rule kinds: exact, parent-walk, wildcard, regex (m21, own linear-time engine). Regex *from downloaded lists* stays out of scope |
 | C | In scope: local DoH/DoT server, local records, conditional forwarding. Out: HTTP/2, DoQ, DHCP, DNSSEC, clustering |
 | D | mbedTLS (vendored) terminates server TLS; stdlib TLS for upstream client |
 | E | `std.Io` injected everywhere; `Threaded` backend (io_uring flag dropped in m11 — Evented networking is stubbed at 0.16.0); no custom thread pool |