# Milestone 21: ABP exceptions from lists, regex rules for operators

Goal: honor `@@||domain^` exception lines in downloaded blocklists as allow
entries scoped below operator rules and above list blocks (tier 1), and add a
`regex` rule kind for operator rules backed by a homegrown linear-time engine
(tier 2). Nothing else from the ABP syntax enters scope.

Design written 2026-08-09 against HEAD `ffc3ca6`; every anchor re-verified
2026-08-11 against `a8e0fe4` after milestone 20 landed. The milestone amends
PLAN §2.2, which currently rules regex out permanently; the amendment is part
of session S3, not a side effect.

## Implementation contract (read first)

- Read `AGENTS.md`, then this spec whole, before session work starts.
- Pure core stays pure: `src/filter/` files that are fuzz-module roots import
  only `std` (`src/filter/parsers.zig:1-14`). The new `src/filter/regex.zig`
  obeys the same constraint.
- Every new `src/**.zig` file must be listed in `src/tests.zig`
  (`build.zig:494-539` fatals otherwise).
- Frozen DDL is frozen: schema changes are new migration steps
  (`src/storage/config_schema.zig:1-6`, `src/storage/migrations.zig:21-22`).
- After any API shape change, regenerate the contract samples
  (`web/src/lib/contractSamples.gen.ts`; procedure in AGENTS.md).

## Rulings (binding)

### 1. Exception lines are `@@||name^` and nothing else, plus one modifier

`src/filter/parser_abp.zig:22` currently maps every `@@` line to
`.unsupported`. After this milestone, a line is an exception when it is
`@@||name^` or `@@||name` (same trailing-`^` and `rule_tokens` treatment as the
block anchor at parser_abp.zig:26-34), optionally suffixed with the literal
`$important` — that suffix is the common form in AdGuard-authored lists and
changes nothing about the meaning here, because list exceptions already sit
below every operator rule. Any other `@@` form (`@@name` without the anchor,
any other `$` modifier, a path, a scheme) stays `.unsupported`. The
parser-header policy paragraph (parser_abp.zig:5-6) is rewritten to state the
new rule and its precedence justification: a list exception can cancel only
list blocks, never an operator decision, so no downloaded list can open an
allow hole the operator did not open.

### 2. Precedence: list exceptions sit between operator rules and list blocks

`matcher.Snapshot.evaluate` (`src/filter/matcher.zig:286-338`) gains one level
between the operator wildcard-block walk (level 4) and the blocklist domain
probe (level 5): for each attached source, an exception match — full name or
parent walk, apex covered — returns `blocked = false`, reason
`.blocklist_exception`, `matched` = the matching entry, `source` = the source
index. The doc comment at matcher.zig:269-285 and PLAN §3.10 (PLAN.md:108-117)
are both updated with the new level. Regex rules (ruling 6) slot in as levels
after the wildcard rules and before list exceptions, allow before block, so
the full order is: exact allow, exact block, wildcard allow, wildcard block,
regex allow, regex block, list exception, list domain, list wildcard.
"Tie-break at same specificity: allow wins" is preserved.

### 3. The compiled-source format grows a third body, checksum-compatibly

`compiler.compile` (`src/filter/compiler.zig:50-56`) takes a third writer
(`allow_w`) and `Counts` (compiler.zig:23-35) gains `exceptions: u32 = 0`.
Exception candidates go through the existing `addCandidate` path into a third
`Entries` and emit as a sorted, deduplicated `.allow` body. The shared SHA-256
covers the bodies in order list, wild, allow — because SHA-256 of
`list ++ wild ++ ""` equals the current SHA-256 of `list ++ wild`, every
already-published checksum stays valid, and `Manager.loadSource`
(`src/filter/manager.zig:543-598`) treats a missing `<id>.allow` file as an
empty body. No refetch is forced by upgrading. `bodyChecksum`
(manager.zig:1572) follows the same order; `source_file_suffixes`
(manager.zig:1589) gains `.allow.tmp` and `.allow` with longest-suffix-first
order preserved; the on-disk header (manager.zig:221-241) gains
`# exceptions {d}` after the `# wildcards` line and the pinning test at
manager.zig:1914-1946 is extended, not weakened.

### 4. Exception counts persist and surface

Migration step 3 (`ddl_v3`, appended at `src/storage/migrations.zig:23-26`):
`ALTER TABLE blocklist_sources ADD COLUMN exception_count INTEGER NOT NULL
DEFAULT 0;`. `sources_repo.SourceRow` and `updateSourceStats`
(`src/storage/repositories/sources_repo.zig:80-93,159-169`) carry it, and the
checksum doc line at sources_repo.zig:100 is updated alongside
`bodyChecksum`'s per ruling 3;
`SourceStatus` rehydration (`manager.zig:1458-1502`) restores it alongside the
existing three counts; `StatusView` (`src/web/handlers/blocklists.zig:52-78`)
gains `exceptions: u32`; the blocklists UI shows it where `skipped_regex`
already shows (`web/src/features/blocklists/SourceStatusSection.tsx`,
`web/src/lib/types.ts:163,192`).

### 5. The regex engine is a Pike VM, linear-time by construction, `std` only

New file `src/filter/regex.zig`. Syntax: literal bytes, `.`, character
classes `[...]` with ranges and leading-`^` negation, escapes
`\. \\ \- \d \w`, repetition `* + ? {n} {n,m}`, alternation `|`,
non-capturing grouping `(...)`, anchors `^` and `$`. No backreferences, no
lookaround, no captures. Matching is unanchored unless anchors are written
(POSIX-grep convention, matching Pi-hole user expectations). Input is the
normalized lowercase name, ≤ `types.max_name_len` bytes. Hard limits, each a
distinct error: pattern ≤ 256 bytes (`PatternTooLong`), compiled program
≤ 1024 instructions (`PatternTooComplex`). Public API:

```zig
pub const Error = error{ OutOfMemory, BadPattern, PatternTooLong, PatternTooComplex };
pub const Program = struct { ... , pub fn deinit(self: *Program, gpa: Allocator) void };
pub fn compile(gpa: Allocator, pattern: []const u8) Error!Program;
pub fn matches(prog: *const Program, input: []const u8) bool;
```

`matches` is a Pike VM: two thread lists, each program counter admitted at
most once per input position, worst case O(program × input) with zero
allocation at match time (thread lists sized from the program at compile
time). The engine is a fuzz-module root like parsers.zig and imports only
`std`.

### 6. `regex` is a third rule kind, validated at the edge, memoized by the cache

Migration step 4 (`ddl_v4`): the 12-step rebuild of `rules` with
`CHECK(kind IN ('exact','wildcard','regex'))` — the frozen v1 DDL
(`src/storage/config_schema.zig:65-72`) cannot be edited. The rebuilt table
keeps the name `rules`: `config_schema.table_names`
(config_schema.zig:112-119) and the invariant tests at
config_schema.zig:120-127 and migrations.zig:356 assert the schema's table
set, and a rename would fail both. `model.RuleKind`
(`src/config/model.zig:259-275`) gains `.regex`; the exhaustive switches in
`config/validate.zig:1099-1130` (compile the pattern, report
`"... is not a valid regex pattern"` through the existing error path at
validate.zig:891-902) and `src/filter/rules.zig:62-68` extend. `RuleSet`
(`rules.zig:28-36`) grows `regex_allow` and `regex_block` slices holding
compiled `Program`s plus their pattern texts (for `Decision.matched`);
`bucketOf` (rules.zig:133-143) becomes a six-bucket layout;
`max_regex_per_group: usize = 256` with `TooManyRegexRules` mirroring
`max_wildcards_per_group` (rules.zig:26). A pattern that fails to compile is
`error.BadPattern` at snapshot build, never skipped (rules.zig:41-49 doc
holds). Reason tags `rule_allow_regex` and `rule_block_regex` join
`matcher.Reason` (matcher.zig:24-32); `/api/lookup` and the query log pick
them up automatically via `@tagName` (`src/web/handlers/lookup.zig:89`;
`max_reason_len = 32` in `src/storage/logger.zig` fits both at 16 chars).
Regex evaluation runs only after every hash and wildcard level missed, and
answers are memoized by the existing DNS cache like every other decision, so
the per-query cost lands on cache misses only.

### 7. The web contract names the third kind everywhere it names the first two

`src/web/handlers/rules.zig`: `toInput` accepts `"regex"`; the 400 string at
rules.zig:42 becomes `"kind must be 'exact', 'wildcard' or 'regex'"`.
`src/web/openapi.yaml:1948,1962,1976`: all three `enum: [exact, wildcard]` become
`[exact, wildcard, regex]`. `web/src/lib/types.ts:195`:
`RuleKind = "exact" | "wildcard" | "regex"`; the rules page kind selector
gains the option. Contract samples regenerated. `nxdns export` / `import`
round-trip the new kind with no extra work once `RuleKind.toDb/fromDb` extend
— the enum round-trip test at model.zig:782 is extended to prove it.

### 8. PLAN amendments land with the code, in S3

PLAN.md:36 (§2.2) is rewritten: operator regex rules are in scope, backed by
the linear-time engine of ruling 5; regex lines in downloaded lists stay
counted and skipped; `$` modifiers (except the `$important` suffix of ruling
1), partial-segment wildcards, and browser-syntax honoring stay permanently
out. PLAN.md:26 (§2.1 filtering sentence), PLAN.md:106 (§3.9), PLAN.md:108-117
(§3.10 precedence) and PLAN.md:700 (decision B) are updated to match rulings
2 and 6. In-code echoes of the old §2.2 move with it:
`src/filter/wildcard.zig:6-7,20-23`, `src/filter/parsers.zig:25`,
`src/filter/parser_abp.zig:5-6`, `src/filter/compiler.zig:129-131`.

### 9. Milestone 20's authority modes bound where rules are written

m20 classifies POST/PUT/DELETE `/api/rules` as `.config_write`
(`src/web/routes.zig:94-97`), and under `.managed_file` authority the router
answers 403 (`src/web/router.zig:173-178`). Everything in this milestone
works in both modes, but the write path differs: in `.database` mode regex
rules arrive through the API; in `.managed_file` mode they arrive through the
config file and `src/config/reconcile.zig` (`reconcileRules`,
reconcile.zig:566-598), whose enum comparison carries `.regex` with no code
change. S3 owns a reconcile test proving a file-declared regex rule converges
into the table. Every S3 API acceptance check runs a server in `.database`
authority (no `--config`). The `SELECT *` dump helpers
(`src/config/import.zig:167-186`, `src/config/reconcile.zig:981`) will emit
the new `exception_count` column after `ddl_v3`; golden dump assertions in
those suites are updated in S1, which owns that migration. Reconcile's
runtime-column protection (reconcile.zig:11-15,435) covers `exception_count`
with no change — the column survives reconciles untouched.

### 10. Fuzz invariants move, never lapse

`tests/fuzz/blocklist_fuzz.zig` header invariant "covers_apex only on
`.wildcard`" (its stated form at :4-28) becomes "only on `.wildcard` or
`.exception`". `tests/fuzz/compiler_fuzz.zig:43` reads `Format` from
`compile`'s parameter list by index — the new `allow_w` parameter appends
after `wild_w`, leaving index 2 valid; the session touching `compile` runs the
fuzz suite and fixes that line if the assumption fails. A new
`tests/fuzz/regex_fuzz.zig` target asserts: `compile` on arbitrary bytes
never crashes and either errors or produces a program within the ruling-5
limits; `matches` terminates and its VM step count never exceeds
program length × (input length + 1); compile-then-match is deterministic.

## Sessions

Three sessions. S1 and S2 run in parallel — they share no files. S3 starts
after both land.

### Session S1: list exceptions end to end (tier 1)

Owns: `src/filter/parsers.zig`, `src/filter/parser_abp.zig`,
`src/filter/compiler.zig`, `src/filter/manager.zig`, `src/filter/matcher.zig`,
`src/filter/domain_set.zig` (only if a helper is needed; expected untouched),
`src/filter/filter_integration_test.zig`, `src/storage/migrations.zig`
(step 3 only), `src/storage/repositories/sources_repo.zig`,
`src/web/handlers/blocklists.zig`, `src/web/handlers/lookup.zig` (doc
sentence only), `src/web/openapi.yaml` (StatusView shape only),
`web/src/features/blocklists/*`, `web/src/lib/types.ts` (source-stat fields
only), `web/src/lib/contractSamples.gen.ts`,
`tests/fuzz/blocklist_fuzz.zig`, `tests/fuzz/compiler_fuzz.zig`, and the
dump-golden assertions in `src/config/import.zig` and
`src/config/reconcile.zig` test suites only (ruling 9's `exception_count`
fallout from `ddl_v3`; S1 touches no reconcile logic).

- S1.1 `Kind.exception` in parsers.zig; parser_abp emits it per ruling 1;
  parser_hosts and parser_domains never emit it (no change beyond the enum).
- S1.2 compiler third body per ruling 3; `Counts.exceptions`.
- S1.3 manager: suffixes, header line, `bodyChecksum`, `loadSource`
  missing-file-is-empty, `Snapshot.Compiled.allow_body`,
  `prepareRefresh`/`publishRefresh`/`applyLoadOutcomes` carry the count.
  `rejectedWithoutEntries` (manager.zig:1563-1570) treats a compile with only
  exceptions as loadable, not rejected.
- S1.4 matcher: `SourceSets.exceptions`, `Reason.blocklist_exception`,
  the new evaluate level per ruling 2, `memoryBytes` includes the new sets.
- S1.5 storage + API + UI per ruling 4.
- S1.6 tests: parser cases (`@@||x^`, `@@||x`, `@@||x^$important`,
  `@@||x^$third-party` → unsupported, `@@x` → unsupported); compiler
  three-body + checksum-compat cases (empty allow body reproduces the old
  digest byte for byte); manager header pin extended; matcher precedence
  cases: operator block beats list exception, list exception beats list
  domain and list wildcard, exception parent walk covers apex and
  subdomains; an integration case through
  `filter_integration_test.zig` with a real ABP fixture carrying `@@` lines.

Acceptance (S1):
- [ ] `zig build test` passes; fuzz targets build and run.
- [ ] A fixture list with `||ads.example^` and `@@||good.ads.example^`
      compiled and loaded blocks `ads.example` and `x.ads.example`, does not
      block `good.ads.example` or `y.good.ads.example`, and
      `/api/lookup` reports `blocklist_exception` with the source id for the
      latter two.
- [ ] A pre-milestone data directory (no `.allow` files, old checksums)
      loads with zero checksum mismatches.
- [ ] `POST /api/blocklists/update` response rows carry `exceptions`.

### Session S2: the regex engine (tier 2, engine only)

Owns: `src/filter/regex.zig` (new), `tests/fuzz/regex_fuzz.zig` (new),
`src/tests.zig` (one added line), `build.zig` (fuzz-suite wiring for the new
target only).

- S2.1 the engine per ruling 5: parser → AST → NFA program → Pike VM.
- S2.2 unit tests in-file: every syntax form; anchored and unanchored
  matching; negated classes; `{n,m}` bounds; each error case; the
  pathological backtracker-killers (`(a+)+b` against `aaaaaaaaaaaaaaaaaaaaX`,
  nested alternation) complete within the step bound.
- S2.3 the fuzz target per ruling 10.

Acceptance (S2):
- [ ] `zig build test` passes with the new file in `src/tests.zig`.
- [ ] The step-bound property holds under the fuzz corpus.
- [ ] `regex.zig` imports nothing but `std`.

### Session S3: the regex rule kind, wired through (needs S1 + S2)

Owns: `src/storage/migrations.zig` (step 4), `src/storage/config_schema.zig`
(comment only if needed), `src/config/model.zig`, `src/config/validate.zig`,
`src/filter/rules.zig`, `src/filter/matcher.zig`,
`src/storage/repositories/rules_repo.zig`, `src/web/handlers/rules.zig`,
`src/web/handlers/mutations.zig` (only if `checkRule` needs the kind),
`src/web/openapi.yaml`, `web/src/features/rules/*`, `web/src/lib/types.ts`,
`web/src/lib/contractSamples.gen.ts`, `src/config/reconcile.zig` (test per
ruling 9), `PLAN.md`, `src/filter/wildcard.zig` (comments),
`src/filter/parsers.zig` (comment), `src/filter/parser_abp.zig` (comment),
`src/filter/compiler.zig` (comment), `tools/bench.zig`.

- S3.1 migration step 4 per ruling 6; repo and model layers.
- S3.2 validate at both edges (config import, API) per rulings 6 and 7.
- S3.3 `RuleSet` six buckets; matcher levels per ruling 2; reasons.
- S3.4 web + UI + openapi + samples per ruling 7.
- S3.5 PLAN and comment amendments per ruling 8.
- S3.6 bench: the filter suite in `tools/bench.zig` gains a variant with 32
  regex rules loaded; the existing p95 < 1 ms assertion covers it.
- S3.7 tests: rule CRUD with kind `regex` through the API including the 400
  for a bad pattern at insert time; precedence cases regex-allow over
  regex-block, wildcard over regex, regex over list entries; export/import
  round trip; a migration test upgrading a v3 database.

Acceptance (S3):
- [ ] `zig build test` and `cd web && npm test` pass.
- [ ] `POST /api/rules` with `{"kind":"regex","pattern":"^ad[0-9]+-"}`
      returns 201; with `"pattern":"("` returns 400 naming the pattern.
- [ ] A regex block rule blocks a matching name; `/api/lookup` reports
      `rule_block_regex` and `matched` carries the pattern text.
- [ ] `zig build bench -Doptimize=ReleaseFast -- filter` passes its targets
      with the regex variant present.
- [ ] PLAN §2.2 no longer forbids operator regex; all listed echoes updated.

### Orchestrator

Verify S1 and S2 acceptance before starting S3. After S3: run the full gate
set (`zig build test`, `test-aarch64` if qemu present, `npm test`,
`npm run assert-bundled`), then a live smoke against a scratch server: load
one real ABP list with `@@` lines, add one regex rule, verify both over dig
and `/api/lookup`. Record deviations in `## Recorded (implementation)`.

## Module layout

New files:
- `src/filter/regex.zig` — the linear-time engine (ruling 5).
- `tests/fuzz/regex_fuzz.zig` — its fuzz target (ruling 10).

Deleted surface: none.

## Acceptance (milestone complete)

- [ ] All session acceptance boxes above.
- [ ] Schema at version 4; a v2 database migrates cleanly with data intact.
- [ ] A pre-milestone blocklist data directory loads without refetch.
- [ ] The six-level operator precedence plus three list levels behave per
      ruling 2, proven by matcher tests that enumerate adjacent-level pairs.
- [ ] No `src/filter/` fuzz-root file imports anything but `std`.
- [ ] Contract samples, openapi.yaml and `web/src/lib/types.ts` agree with
      the server (the drift guards pass).

## Anti-requirements

- No `$` modifier support beyond tolerating `$important` on exception lines.
  `$dnstype`, `$dnsrewrite`, `$client`, `$denyallow` and every browser
  modifier stay unsupported and counted.
- No regex from downloaded lists. `.regex` lines stay counted and skipped;
  `skipped_regex` keeps its meaning. The engine exists for operator rules
  only.
- No partial-segment wildcards (`ads*.example.com`) — writing one as a rule
  stays rejected; the regex kind covers the need.
- No backreferences, lookaround, captures, named groups or Unicode classes in
  the engine, ever. A pattern needing them is rejected, not approximated.
- No PCRE2, RE2 or any external regex dependency.
- No exception-rule UI editor: exceptions come from lists; operators write
  allow rules.
- No re-download forced by the upgrade; checksum compatibility (ruling 3) is
  a requirement, not an optimization.
