docs: unwrap hand-wrapped prose repo-wide
Gates / frontend (push) Successful in 1m2s
Gates / test (push) Successful in 1m38s
Gates / package (push) Successful in 5m5s
Gates / test-aarch64 (push) Successful in 6m30s
Gates / container (push) Successful in 15s
CI / gates (push) Successful in 13m30s

This commit is contained in:
2026-08-15 16:27:36 +02:00
parent 50b8fd5c61
commit 5b3d1cd65c
48 changed files with 2691 additions and 11699 deletions
+78 -362
View File
@@ -1,247 +1,93 @@
# Milestone 24: `skipped_unsupported` is persisted, surfaced and explained
Goal: close the compile-pipeline silent drop that misleads the operator. `Counts.
skipped_unsupported` (`src/filter/compiler.zig:34`) is counted on every compile
(compiler.zig:92) and then discarded on the happy path — it reaches the
compiled-file header and the `NoValidEntries` error text, but no database
column, no API field and no UI cell. Its sibling `skipped_regex` reaches all
three. A blocklist made almost entirely of cosmetic browser filters therefore
compiles to almost nothing and looks, in the UI, exactly like a clean list.
AGENTS.md forbids exactly this ("no silent drops"). The milestone persists the
count, surfaces it everywhere `skipped_regex` is surfaced, and explains to the
operator why the two skip counters are different facts.
Goal: close the compile-pipeline silent drop that misleads the operator. `Counts. skipped_unsupported` (`src/filter/compiler.zig:34`) is counted on every compile (compiler.zig:92) and then discarded on the happy path — it reaches the compiled-file header and the `NoValidEntries` error text, but no database column, no API field and no UI cell. Its sibling `skipped_regex` reaches all three. A blocklist made almost entirely of cosmetic browser filters therefore compiles to almost nothing and looks, in the UI, exactly like a clean list. AGENTS.md forbids exactly this ("no silent drops"). The milestone persists the count, surfaces it everywhere `skipped_regex` is surfaced, and explains to the operator why the two skip counters are different facts.
Design written 2026-08-13 against HEAD `21571e4`, revised after a Codex review
of the first draft.
Design written 2026-08-13 against HEAD `21571e4`, revised after a Codex review of the first draft.
## Implementation contract (read first)
- Read `AGENTS.md`, then this spec whole, before session work starts.
- **The v1 baseline is editable and there is no migration step.** As of commit
`21571e4`, `src/storage/migrations.zig` holds exactly one step and
`target_version` is 1 (migrations.zig:29-36). nxdns has zero installs; the
baseline freezes at v0.1 (`config_schema.zig:6-12`, PLAN §3.7, §11.2). The
new column is one line edited into `config_schema.ddl_v1` plus the identical
line in PLAN §11.2, which is kept byte-identical to it. A diff that adds a
`ddl_v2` or a second `Step` is wrong and must be reverted, not merged.
- After the API shape change, regenerate the contract samples
(`web/src/lib/contractSamples.gen.ts`; procedure in AGENTS.md — the
`zig build test -Dintegration -Dcontract-samples-out=...` command, never a
hand edit) and update `web/src/lib/types.ts` to match.
- **The v1 baseline is editable and there is no migration step.** As of commit `21571e4`, `src/storage/migrations.zig` holds exactly one step and `target_version` is 1 (migrations.zig:29-36). nxdns has zero installs; the baseline freezes at v0.1 (`config_schema.zig:6-12`, PLAN §3.7, §11.2). The new column is one line edited into `config_schema.ddl_v1` plus the identical line in PLAN §11.2, which is kept byte-identical to it. A diff that adds a `ddl_v2` or a second `Step` is wrong and must be reverted, not merged.
- After the API shape change, regenerate the contract samples (`web/src/lib/contractSamples.gen.ts`; procedure in AGENTS.md — the `zig build test -Dintegration -Dcontract-samples-out=...` command, never a hand edit) and update `web/src/lib/types.ts` to match.
- No new `src/**.zig` files, so `src/tests.zig` and `build.zig` are untouched.
## Rulings (binding)
### 1. The column is `skipped_unsupported_count`, one edited line, no step
`skipped_regex_count` (config_schema.zig:63) sets the convention; the new
column follows it. In `config_schema.ddl_v1`, directly after the
`skipped_regex_count` line inside `CREATE TABLE blocklist_sources`:
`skipped_regex_count` (config_schema.zig:63) sets the convention; the new column follows it. In `config_schema.ddl_v1`, directly after the `skipped_regex_count` line inside `CREATE TABLE blocklist_sources`:
```sql
skipped_unsupported_count INTEGER NOT NULL DEFAULT 0,
```
The identical line goes into PLAN §11.2 after PLAN.md:421 — that section is
kept byte-identical to `ddl_v1` and currently is (verified: the fenced SQL
matches the Zig string literal line for line).
The identical line goes into PLAN §11.2 after PLAN.md:421 — that section is kept byte-identical to `ddl_v1` and currently is (verified: the fenced SQL matches the Zig string literal line for line).
Consequence to accept, not to fix: a development database stamped version 1
before this edit will fail the first `SELECT` naming the column with "no such
column", because the stamped version equals `target_version` and the step never
reruns. That is the documented pre-v0.1 contract (`config_schema.zig:6-12`) —
delete the scratch database. Do not add fallback SQL, `PRAGMA table_info`
probing, or a migration step to paper over it.
Consequence to accept, not to fix: a development database stamped version 1 before this edit will fail the first `SELECT` naming the column with "no such column", because the stamped version equals `target_version` and the step never reruns. That is the documented pre-v0.1 contract (`config_schema.zig:6-12`) — delete the scratch database. Do not add fallback SQL, `PRAGMA table_info` probing, or a migration step to paper over it.
The baseline test "a fresh database reaches the baseline with every v1 column
and rule kind" (migrations.zig:266-283) gains
`try testing.expect(try columnExists(&database, "blocklist_sources", "skipped_unsupported_count"));`
beside the existing `exception_count` probe.
The baseline test "a fresh database reaches the baseline with every v1 column and rule kind" (migrations.zig:266-283) gains `try testing.expect(try columnExists(&database, "blocklist_sources", "skipped_unsupported_count"));` beside the existing `exception_count` probe.
### 2. The repo persists it beside `skipped_regex_count`
`src/storage/repositories/sources_repo.zig`, mirroring `skipped_regex_count`
exactly (no default on either struct field, so every construction site fails
to compile until it names the count — that is the visibility this milestone is
about):
`src/storage/repositories/sources_repo.zig`, mirroring `skipped_regex_count` exactly (no default on either struct field, so every construction site fails to compile until it names the count — that is the visibility this milestone is about):
- `SourceRow` (sources_repo.zig:80-97) gains `skipped_unsupported_count: i64`
after `skipped_regex_count`.
- `SourceRow` (sources_repo.zig:80-97) gains `skipped_unsupported_count: i64` after `skipped_regex_count`.
- `SourceStats` (sources_repo.zig:99-110) gains the same field.
- `row_columns_sql` (sources_repo.zig:112-117) appends
`skipped_unsupported_count` to the SELECT list (index 11);
`readSourceRow` (sources_repo.zig:128-148) reads it with
`stmt.columnInt(11)`.
- `update_stats_sql` (sources_repo.zig:159-164) adds
`skipped_unsupported_count = ?8`; `updateSourceStats`
(sources_repo.zig:168-179) binds it.
- The module doc comment (sources_repo.zig:3-5) adds the column to its list of
server-produced facts, and so does the `src/config/model.zig:14` doc comment
that repeats that list.
- `row_columns_sql` (sources_repo.zig:112-117) appends `skipped_unsupported_count` to the SELECT list (index 11); `readSourceRow` (sources_repo.zig:128-148) reads it with `stmt.columnInt(11)`.
- `update_stats_sql` (sources_repo.zig:159-164) adds `skipped_unsupported_count = ?8`; `updateSourceStats` (sources_repo.zig:168-179) binds it.
- The module doc comment (sources_repo.zig:3-5) adds the column to its list of server-produced facts, and so does the `src/config/model.zig:14` doc comment that repeats that list.
**The sum at sources_repo.zig:318 includes the new column.** That sum lives in
the test "insertBlocklistSource leaves the runtime columns at their defaults";
it exists to prove an insert leaves every runtime counter at 0, and the new
column is a runtime counter. This is a deliberate decision, not an oversight:
no production query sums these columns into a "total entries" figure, and none
may start to — `skipped_regex_count` and `skipped_unsupported_count` count
lines *not* written, unlike `domain_count`, `wildcard_count` and
`exception_count`, so any future entries total must exclude both. The test sum
asserts defaults, which is the one context where adding them is correct.
**The sum at sources_repo.zig:318 includes the new column.** That sum lives in the test "insertBlocklistSource leaves the runtime columns at their defaults"; it exists to prove an insert leaves every runtime counter at 0, and the new column is a runtime counter. This is a deliberate decision, not an oversight: no production query sums these columns into a "total entries" figure, and none may start to — `skipped_regex_count` and `skipped_unsupported_count` count lines *not* written, unlike `domain_count`, `wildcard_count` and `exception_count`, so any future entries total must exclude both. The test sum asserts defaults, which is the one context where adding them is correct.
Existing `SourceStats` / assertion sites in this file (the literals at
sources_repo.zig:386-393, 420-427, 483-493 and the zero-default loop at
sources_repo.zig:365-374) carry the new field with distinct non-zero values
where their siblings have them, and the round-trip assertions extend to it.
Existing `SourceStats` / assertion sites in this file (the literals at sources_repo.zig:386-393, 420-427, 483-493 and the zero-default loop at sources_repo.zig:365-374) carry the new field with distinct non-zero values where their siblings have them, and the round-trip assertions extend to it.
### 3. The manager writes it, and rehydration restores it
`src/filter/manager.zig`. The header already prints
`# skipped_unsupported {d}` (manager.zig:243) and `SourceStatus.counts` is a
full `compiler.Counts` (manager.zig:178), so within one process the count
already reaches the status table. What is missing is the database, which is
the only thing a restart reads — rehydration never reparses headers.
`src/filter/manager.zig`. The header already prints `# skipped_unsupported {d}` (manager.zig:243) and `SourceStatus.counts` is a full `compiler.Counts` (manager.zig:178), so within one process the count already reaches the status table. What is missing is the database, which is the only thing a restart reads — rehydration never reparses headers.
- The fresh-publish path (manager.zig:854-861) passes
`.skipped_unsupported_count = compiled.result.counts.skipped_unsupported` to
`updateSourceStats`.
- **The checksum-unchanged path (manager.zig:821-836) carries a bug this
milestone fixes.** It writes the *stored* `row.skipped_regex_count` back to
the database (manager.zig:830) while handing the *fresh*
`compiled.result.counts` to the in-memory status (manager.zig:833). The
checksum covers the `.list`, `.wild` and `.allow` bodies only, and a skipped
line lands in none of them — so a list that changes only its regex or
browser-syntax lines keeps its checksum, the running server shows the new
number, the database keeps the old one, and the next restart silently reverts
what the operator saw. That is milestone 21's column, wrong today.
- The fresh-publish path (manager.zig:854-861) passes `.skipped_unsupported_count = compiled.result.counts.skipped_unsupported` to `updateSourceStats`.
- **The checksum-unchanged path (manager.zig:821-836) carries a bug this milestone fixes.** It writes the *stored* `row.skipped_regex_count` back to the database (manager.zig:830) while handing the *fresh* `compiled.result.counts` to the in-memory status (manager.zig:833). The checksum covers the `.list`, `.wild` and `.allow` bodies only, and a skipped line lands in none of them — so a list that changes only its regex or browser-syntax lines keeps its checksum, the running server shows the new number, the database keeps the old one, and the next restart silently reverts what the operator saw. That is milestone 21's column, wrong today.
Both skip counters take `compiled.result.counts` on this path:
`.skipped_regex_count = compiled.result.counts.skipped_regex` and
`.skipped_unsupported_count = compiled.result.counts.skipped_unsupported`.
~~`domain_count`, `wildcard_count` and `exception_count` keep reading from
`row` — they count written entries, so an unchanged checksum does mean an
unchanged value for them.~~ **Corrected post-commit:** that claim was false.
The unframed digest could not distinguish an entry in `.list` from the same
entry in `.wild`, so an unchanged checksum did not vouch for the entry
counts either. The addendum below frames the hash and makes all five stat
fields read fresh on this path.
Both skip counters take `compiled.result.counts` on this path: `.skipped_regex_count = compiled.result.counts.skipped_regex` and `.skipped_unsupported_count = compiled.result.counts.skipped_unsupported`. ~~`domain_count`, `wildcard_count` and `exception_count` keep reading from `row` — they count written entries, so an unchanged checksum does mean an unchanged value for them.~~ **Corrected post-commit:** that claim was false. The unframed digest could not distinguish an entry in `.list` from the same entry in `.wild`, so an unchanged checksum did not vouch for the entry counts either. The addendum below frames the hash and makes all five stat fields read fresh on this path.
This needs a test the author has watched fail with the fix reverted: two
compiles of bodies whose written entries are identical but whose skipped
lines differ, asserting the same checksum, then asserting the database holds
the second compile's counts, then rehydrating through `applyLoadOutcomes` to
prove the restored status carries them. Report the observed failure output.
- `applyLoadOutcomes` rehydration (manager.zig:1545-1550) adds
`.skipped_unsupported = countOf(row.skipped_unsupported_count)` to the
`Counts` it rebuilds, so a restarted server reports what the last compile
skipped instead of 0.
- Test literals name the field: one `SourceStats` (manager.zig:1928) and two
`SourceRow` (manager.zig:2051, 2225). The rehydration test that feeds a row
through
`applyLoadOutcomes` asserts a non-zero `skipped_unsupported` lands in
`status.counts`. The header pin test (manager.zig:2030-2031) already covers
the `# skipped_unsupported` line and is extended only if its fixture counts
change.
- `rejectedWithoutEntries` (manager.zig:1640-1643) and `failNoValidEntries`
(manager.zig:1127-1128) already read the count and are unchanged.
This needs a test the author has watched fail with the fix reverted: two compiles of bodies whose written entries are identical but whose skipped lines differ, asserting the same checksum, then asserting the database holds the second compile's counts, then rehydrating through `applyLoadOutcomes` to prove the restored status carries them. Report the observed failure output.
- `applyLoadOutcomes` rehydration (manager.zig:1545-1550) adds `.skipped_unsupported = countOf(row.skipped_unsupported_count)` to the `Counts` it rebuilds, so a restarted server reports what the last compile skipped instead of 0.
- Test literals name the field: one `SourceStats` (manager.zig:1928) and two `SourceRow` (manager.zig:2051, 2225). The rehydration test that feeds a row through `applyLoadOutcomes` asserts a non-zero `skipped_unsupported` lands in `status.counts`. The header pin test (manager.zig:2030-2031) already covers the `# skipped_unsupported` line and is extended only if its fixture counts change.
- `rejectedWithoutEntries` (manager.zig:1640-1643) and `failNoValidEntries` (manager.zig:1127-1128) already read the count and are unchanged.
`src/filter/filter_integration_test.zig` names `.skipped_regex_count` in six
`SourceStats` literals (:826, :858, :1075, :1145, :1257, :2158) — each gains
the sibling field. The stats round-trip assertion at :915 gains a sibling
assertion for the new column.
`src/filter/filter_integration_test.zig` names `.skipped_regex_count` in six `SourceStats` literals (:826, :858, :1075, :1145, :1257, :2158) — each gains the sibling field. The stats round-trip assertion at :915 gains a sibling assertion for the new column.
**Do not add an unsupported line to the existing fixture.** The fixture at
filter_integration_test.zig:112 is hosts-shaped, and `detectFormat`
(`src/filter/parsers.zig:118`) assigns one format to a whole source. A single
`##.ad-banner` or `$`-modifier line flips it to ABP, which re-parses every
existing line: the `*.wild` entry becomes unsupported and address fields
tokenize as domains (`src/filter/parser_abp.zig:68`). Every expected count in
that test would move, for a reason unrelated to this milestone.
**Do not add an unsupported line to the existing fixture.** The fixture at filter_integration_test.zig:112 is hosts-shaped, and `detectFormat` (`src/filter/parsers.zig:118`) assigns one format to a whole source. A single `##.ad-banner` or `$`-modifier line flips it to ABP, which re-parses every existing line: the `*.wild` entry becomes unsupported and address fields tokenize as domains (`src/filter/parser_abp.zig:68`). Every expected count in that test would move, for a reason unrelated to this milestone.
Add a **separate ABP-format fixture and test** carrying `##.ad-banner` and
`||ads.example^$third-party` beside two blockable names, and assert
`skipped_unsupported_count = 2` through the compile-persist-read round trip
there.
Add a **separate ABP-format fixture and test** carrying `##.ad-banner` and `||ads.example^$third-party` beside two blockable names, and assert `skipped_unsupported_count = 2` through the compile-persist-read round trip there.
`src/config/reconcile.zig`: the runtime-column preservation test seeds stats
at reconcile.zig:1096-1110 and asserts survival at reconcile.zig:1173. The
seed gains `.skipped_unsupported_count` with a distinct value and the
assertion block gains its expectation. No reconcile logic changes: the engine
updates declarative columns by name and never touches runtime columns, so the
new column survives with zero code change — the test is the proof.
`src/config/reconcile.zig`: the runtime-column preservation test seeds stats at reconcile.zig:1096-1110 and asserts survival at reconcile.zig:1173. The seed gains `.skipped_unsupported_count` with a distinct value and the assertion block gains its expectation. No reconcile logic changes: the engine updates declarative columns by name and never touches runtime columns, so the new column survives with zero code change — the test is the proof.
### 4. The API speaks it in both shapes
Two shapes carry blocklist counters, and the new count joins both under the
names its siblings set:
Two shapes carry blocklist counters, and the new count joins both under the names its siblings set:
- **`Blocklist`** (rows of `GET /api/blocklists`, serialized straight from
`SourceRow`): the field arrives automatically once `SourceRow` has it, as
`skipped_unsupported_count`. `src/web/openapi.yaml:1877-1895` adds it to
`required` and `properties`.
- **`SourceStatus`** (rows of `POST /api/blocklists/update`): `StatusView`
(`src/web/handlers/blocklists.zig:52-80`) gains
`skipped_unsupported: u32` after `skipped_regex`, mapped from
`status.counts.skipped_unsupported` in `from`.
`src/web/openapi.yaml:1920-1938` adds it to `required` and `properties`.
The handler test literals at blocklists.zig:323 (`SourceStats`) and :386
(`counts`) carry the field with non-zero values and the response assertions
extend to it.
- **`Blocklist`** (rows of `GET /api/blocklists`, serialized straight from `SourceRow`): the field arrives automatically once `SourceRow` has it, as `skipped_unsupported_count`. `src/web/openapi.yaml:1877-1895` adds it to `required` and `properties`.
- **`SourceStatus`** (rows of `POST /api/blocklists/update`): `StatusView` (`src/web/handlers/blocklists.zig:52-80`) gains `skipped_unsupported: u32` after `skipped_regex`, mapped from `status.counts.skipped_unsupported` in `from`. `src/web/openapi.yaml:1920-1938` adds it to `required` and `properties`. The handler test literals at blocklists.zig:323 (`SourceStats`) and :386 (`counts`) carry the field with non-zero values and the response assertions extend to it.
Contract fallout, all in the same session (ruling 6 explains why):
- Regenerate `web/src/lib/contractSamples.gen.ts` with the AGENTS.md command.
- `web/src/lib/types.ts`: `Blocklist` gains
`skipped_unsupported_count: number` (types.ts:154-166); `SourceStatus`
gains `skipped_unsupported: number` (types.ts:183-195).
- The web test mocks are **not** typed against these interfaces, so `tsc` will
not force them. `BLOCKLISTS` in
`web/src/features/blocklists/BlocklistsPage.test.tsx` is an inferred object
literal handed to a `Record<string, unknown>` (BlocklistsPage.test.tsx:40),
and the same untyped-fetch pattern holds in
`web/src/features/groups/GroupsPage.test.tsx:16` and
`web/src/features/settings/authority.test.tsx:64` — both of which already
omit `exception_count` without failing. Updating a mock here is a semantic
fixture change, not a typecheck fix, and the implementer must not expect a
compiler error to point at them.
- `web/src/lib/types.ts`: `Blocklist` gains `skipped_unsupported_count: number` (types.ts:154-166); `SourceStatus` gains `skipped_unsupported: number` (types.ts:183-195).
- The web test mocks are **not** typed against these interfaces, so `tsc` will not force them. `BLOCKLISTS` in `web/src/features/blocklists/BlocklistsPage.test.tsx` is an inferred object literal handed to a `Record<string, unknown>` (BlocklistsPage.test.tsx:40), and the same untyped-fetch pattern holds in `web/src/features/groups/GroupsPage.test.tsx:16` and `web/src/features/settings/authority.test.tsx:64` — both of which already omit `exception_count` without failing. Updating a mock here is a semantic fixture change, not a typecheck fix, and the implementer must not expect a compiler error to point at them.
So: the BlocklistsPage mocks (:21, :34 blocklist rows; :104, :155, :168
status rows) gain the field because S2's rendering assertions read it. The
groups and settings mocks are left alone — they render no counter column,
and widening them buys nothing. S1 picks values no other cell in the same
table already shows (`3` and `7` are taken), e.g.
`skipped_unsupported_count: 21` and `skipped_unsupported: 17`.
So: the BlocklistsPage mocks (:21, :34 blocklist rows; :104, :155, :168 status rows) gain the field because S2's rendering assertions read it. The groups and settings mocks are left alone — they render no counter column, and widening them buys nothing. S1 picks values no other cell in the same table already shows (`3` and `7` are taken), e.g. `skipped_unsupported_count: 21` and `skipped_unsupported: 17`.
### 5. The UI shows it always, in both tables, and says what it means
Both tables gain a `Skipped unsupported` column directly after
`Skipped regex`, rendered unconditionally:
Both tables gain a `Skipped unsupported` column directly after `Skipped regex`, rendered unconditionally:
- `web/src/features/blocklists/BlocklistsPage.tsx`: header after :157, cell
`{b.skipped_unsupported_count}` after :190, same
`shared.td, shared.tabularNums` props as its neighbours.
- `web/src/features/blocklists/SourceStatusSection.tsx`: header after :91,
cell `{source.skipped_unsupported}` after :114.
- `web/src/features/blocklists/BlocklistsPage.tsx`: header after :157, cell `{b.skipped_unsupported_count}` after :190, same `shared.td, shared.tabularNums` props as its neighbours.
- `web/src/features/blocklists/SourceStatusSection.tsx`: header after :91, cell `{source.skipped_unsupported}` after :114.
Always-shown is a decision, not a default: every other counter column here is
unconditional, including `Skipped regex` and `Exceptions`, which are 0 for
every plain hosts list, and the tutorial already explains those zeros. A
column that appears only when non-zero would make two lists' tables disagree
in shape, would hide the header that gives the number its meaning, and would
special-case exactly the counter this milestone exists to make visible. The
zero cell is not noise; it states that nothing in this list was classified as
unsupported — which is narrower than "clean", since `invalid` and `long_lines`
stay unsurfaced (ruling 7).
Always-shown is a decision, not a default: every other counter column here is unconditional, including `Skipped regex` and `Exceptions`, which are 0 for every plain hosts list, and the tutorial already explains those zeros. A column that appears only when non-zero would make two lists' tables disagree in shape, would hide the header that gives the number its meaning, and would special-case exactly the counter this milestone exists to make visible. The zero cell is not noise; it states that nothing in this list was classified as unsupported — which is narrower than "clean", since `invalid` and `long_lines` stay unsurfaced (ruling 7).
The two counters mean different things and the page must say so once. A muted
paragraph (the existing `styles.empty`-style muted text, matching
`SourceStatusSection`'s `note` treatment) rendered under the sources table in
`BlocklistsPage.tsx`, inside the same `else` branch as the table — the note
describes the two skip columns, so it appears exactly when they do. "Always
visible" above means unconditional on values, never hidden at 0; it does not
mean the empty state (`blocklists.length === 0`) carries a paragraph about
columns that are not on screen. The empty state stays one instruction. Exact
copy:
The two counters mean different things and the page must say so once. A muted paragraph (the existing `styles.empty`-style muted text, matching `SourceStatusSection`'s `note` treatment) rendered under the sources table in `BlocklistsPage.tsx`, inside the same `else` branch as the table — the note describes the two skip columns, so it appears exactly when they do. "Always visible" above means unconditional on values, never hidden at 0; it does not mean the empty state (`blocklists.length === 0`) carries a paragraph about columns that are not on screen. The empty state stays one instruction. Exact copy:
> Both “Skipped” columns count lines nxdns read and did not take. Skipped
> regex lines are patterns nxdns accepts only from you — adopt one you trust
@@ -251,15 +97,11 @@ copy:
> list is written for a browser extension, and its DNS or hosts variant will
> block more here.
`BlocklistsPage.test.tsx` asserts: both new headers render, the mock values
(`21`, and `17` after an update snapshot) render, and the note text is
present. No threshold logic, no badge, no coloring by magnitude (see
anti-requirements).
`BlocklistsPage.test.tsx` asserts: both new headers render, the mock values (`21`, and `17` after an update snapshot) render, and the note text is present. No threshold logic, no badge, no coloring by magnitude (see anti-requirements).
### 6. The docs explain both counters without contradicting what stands
- `docs/reference/configuration.md`, section `### blocklist_sources`, after
the exceptions paragraph (configuration.md:282-288), a new paragraph:
- `docs/reference/configuration.md`, section `### blocklist_sources`, after the exceptions paragraph (configuration.md:282-288), a new paragraph:
> Two counters report what a compile skipped, and they are different facts.
> `skipped_regex` counts regex lines: nxdns has a regex engine, but it takes
@@ -277,107 +119,43 @@ anti-requirements).
> hosts variant will block more here. Both appear per source in the
> blocklists UI and on `/api/blocklists`.
- `docs/tutorial/first-run.md`: the recorded JSON at first-run.md:223 gains
`"skipped_unsupported":0` after `"skipped_regex":0` (the endpoint now
returns it; the recorded values themselves stand). The paragraph at
first-run.md:226-231 becomes four zeros, keeps its `skipped_regex` sentence
as written, and **drops the "parts of this list that a hosts file cannot
have" framing** — a deliberate small widening ruled during implementation
review. The framing was false before this milestone touched it:
`parser_hosts.zig` recognizes regex lines (:16), reports a bare sink
address as unsupported (:21), and a `*.`-prefixed name compiles to a
wildcard in any format. Only `exceptions` is Adblock-Plus-only. The
paragraph now presents the zeros as facts about this particular download
and says so.
- The `$` modifier is named as skipped **with its one exception**: `$important`
on an anchored `@@` exception line is accepted and lands in
`exception_count` (`parser_abp.zig`, PLAN §2.2). The configuration.md
paragraph above, and the `SourceRow.skipped_unsupported_count` doc comment
in `sources_repo.zig` that mirrors it, both carry the qualifier — an
unqualified "rules carrying a `$` modifier" contradicts the parser.
- PLAN.md:101 (§3.8) currently reads "regex lines are counted + skipped
(counts in metadata → UI)". It becomes: regex lines *and* browser-syntax
lines are counted and skipped, both counts in metadata → UI. One sentence;
§2.2 (PLAN.md:36-37) already says unsupported forms "stay unsupported and
counted" and needs no edit.
- `docs/reference/files-and-directories.md` is untouched: the compiled-file
header already carried `# skipped_unsupported` before this milestone and
the file table there does not enumerate header lines.
- `docs/tutorial/first-run.md`: the recorded JSON at first-run.md:223 gains `"skipped_unsupported":0` after `"skipped_regex":0` (the endpoint now returns it; the recorded values themselves stand). The paragraph at first-run.md:226-231 becomes four zeros, keeps its `skipped_regex` sentence as written, and **drops the "parts of this list that a hosts file cannot have" framing** — a deliberate small widening ruled during implementation review. The framing was false before this milestone touched it: `parser_hosts.zig` recognizes regex lines (:16), reports a bare sink address as unsupported (:21), and a `*.`-prefixed name compiles to a wildcard in any format. Only `exceptions` is Adblock-Plus-only. The paragraph now presents the zeros as facts about this particular download and says so.
- The `$` modifier is named as skipped **with its one exception**: `$important` on an anchored `@@` exception line is accepted and lands in `exception_count` (`parser_abp.zig`, PLAN §2.2). The configuration.md paragraph above, and the `SourceRow.skipped_unsupported_count` doc comment in `sources_repo.zig` that mirrors it, both carry the qualifier — an unqualified "rules carrying a `$` modifier" contradicts the parser.
- PLAN.md:101 (§3.8) currently reads "regex lines are counted + skipped (counts in metadata → UI)". It becomes: regex lines *and* browser-syntax lines are counted and skipped, both counts in metadata → UI. One sentence; §2.2 (PLAN.md:36-37) already says unsupported forms "stay unsupported and counted" and needs no edit.
- `docs/reference/files-and-directories.md` is untouched: the compiled-file header already carried `# skipped_unsupported` before this milestone and the file table there does not enumerate header lines.
### 7. The other three counters stay out, and the reasons differ per counter
The first draft claimed `invalid`, `long_lines` and `duplicates` were already
"header-and-error-path only". That is false:
The first draft claimed `invalid`, `long_lines` and `duplicates` were already "header-and-error-path only". That is false:
- `invalid` reaches the compiled-file header (manager.zig:244) and the
`NoValidEntries` text (manager.zig:1127).
- `invalid` reaches the compiled-file header (manager.zig:244) and the `NoValidEntries` text (manager.zig:1127).
- `long_lines` reaches the `NoValidEntries` text only — not the header.
- `duplicates` reaches **nothing**: counted at compile, discarded on every
path.
- `duplicates` reaches **nothing**: counted at compile, discarded on every path.
They stay out of scope, and not for one shared reason — which is why the goal
above says "the silent drop that misleads the operator" rather than "the one
silent drop left":
They stay out of scope, and not for one shared reason — which is why the goal above says "the silent drop that misleads the operator" rather than "the one silent drop left":
- `duplicates` hides no failure. A deduplicated name still blocks; the count
measures input redundancy, not lost coverage. Discarding it discards a
curiosity, so "no silent drops" does not reach it — there is no drop.
- `invalid` and `long_lines` do measure dropped lines, and they remain only
partially surfaced. The catastrophic form — a download that is all rejects —
already fails loudly (`rejectedWithoutEntries`, state `no_valid_entries`);
the residual is a list that loses some lines and still loads, visible in
the header file and nowhere the UI reaches. That residual is real, it is
recorded here deliberately, and it is not this milestone: the two skip
counters name an action the operator can take (adopt the patterns; fetch
the DNS variant), while these two say only "the list is malformed", which
no column makes more actionable.
- `duplicates` hides no failure. A deduplicated name still blocks; the count measures input redundancy, not lost coverage. Discarding it discards a curiosity, so "no silent drops" does not reach it — there is no drop.
- `invalid` and `long_lines` do measure dropped lines, and they remain only partially surfaced. The catastrophic form — a download that is all rejects — already fails loudly (`rejectedWithoutEntries`, state `no_valid_entries`); the residual is a list that loses some lines and still loads, visible in the header file and nowhere the UI reaches. That residual is real, it is recorded here deliberately, and it is not this milestone: the two skip counters name an action the operator can take (adopt the patterns; fetch the DNS variant), while these two say only "the list is malformed", which no column makes more actionable.
The decision is reversible; a later milestone can widen the table. What this
spec may not do is claim the three are visible when `duplicates` is not.
The decision is reversible; a later milestone can widen the table. What this spec may not do is claim the three are visible when `duplicates` is not.
### 8. Export, import and reconcile change nothing
`nxdns export` / `import` carry configuration; the compile counters are facts
a running server produces. `model.BlocklistSource` holds only the four
configuration columns, the insert leaves runtime columns at their defaults
(sources_repo.zig:1-9, :48-61), and the dump helpers used by the import and
reconcile suites are `SELECT *` compared dump-to-dump
(`src/config/import.zig:167-186`, `src/config/reconcile.zig:988`), so no
golden text names columns and the new column appears in both sides of every
comparison. Ruling 3's reconcile-test extension is the only touch in
`src/config/`, plus the model.zig:14 comment from ruling 2.
`nxdns export` / `import` carry configuration; the compile counters are facts a running server produces. `model.BlocklistSource` holds only the four configuration columns, the insert leaves runtime columns at their defaults (sources_repo.zig:1-9, :48-61), and the dump helpers used by the import and reconcile suites are `SELECT *` compared dump-to-dump (`src/config/import.zig:167-186`, `src/config/reconcile.zig:988`), so no golden text names columns and the new column appears in both sides of every comparison. Ruling 3's reconcile-test extension is the only touch in `src/config/`, plus the model.zig:14 comment from ruling 2.
## Sessions
Two sessions, strictly sequential: S1 then S2. No parallelism — deliberately.
The regenerated `contractSamples.gen.ts` typechecks only against a `types.ts`
that already carries the new fields, and the samples are produced by a Zig
integration run, so the generator and the interface it must satisfy sit on
opposite sides of the language boundary and cannot land independently. S2's
rendering assertions then read fields that only exist once S1 has landed both.
Two sessions, strictly sequential: S1 then S2. No parallelism — deliberately. The regenerated `contractSamples.gen.ts` typechecks only against a `types.ts` that already carries the new fields, and the samples are produced by a Zig integration run, so the generator and the interface it must satisfy sit on opposite sides of the language boundary and cannot land independently. S2's rendering assertions then read fields that only exist once S1 has landed both.
(The mocks are *not* part of this argument: they are untyped, so widening
`types.ts` does not break them. The first draft claimed otherwise.)
(The mocks are *not* part of this argument: they are untyped, so widening `types.ts` does not break them. The first draft claimed otherwise.)
### Session S1: column, persistence, API, contract
Owns: `src/storage/config_schema.zig`, `src/storage/migrations.zig` (test
only), `src/storage/repositories/sources_repo.zig`, `src/filter/manager.zig`,
`src/filter/filter_integration_test.zig`, `src/config/reconcile.zig` (test
only), `src/config/model.zig` (comment only),
`src/web/handlers/blocklists.zig`, `src/web/openapi.yaml`,
`web/src/lib/types.ts`, `web/src/lib/contractSamples.gen.ts` (regenerated,
never hand-edited), `web/src/features/blocklists/BlocklistsPage.test.tsx`
(mock fields only — no rendering assertions),
`PLAN.md` (the §11.2 line and
the §3.8 sentence).
Owns: `src/storage/config_schema.zig`, `src/storage/migrations.zig` (test only), `src/storage/repositories/sources_repo.zig`, `src/filter/manager.zig`, `src/filter/filter_integration_test.zig`, `src/config/reconcile.zig` (test only), `src/config/model.zig` (comment only), `src/web/handlers/blocklists.zig`, `src/web/openapi.yaml`, `web/src/lib/types.ts`, `web/src/lib/contractSamples.gen.ts` (regenerated, never hand-edited), `web/src/features/blocklists/BlocklistsPage.test.tsx` (mock fields only — no rendering assertions), `PLAN.md` (the §11.2 line and the §3.8 sentence).
- S1.1 rulings 1 and 2: the DDL line in both copies, the repo columns, the
migrations and repo tests.
- S1.2 ruling 3: both `updateSourceStats` call sites, rehydration, test
literals, the reconcile preservation assertion.
- S1.3 ruling 4: `StatusView`, openapi.yaml, sample regeneration, `types.ts`,
mock fields.
- S1.1 rulings 1 and 2: the DDL line in both copies, the repo columns, the migrations and repo tests.
- S1.2 ruling 3: both `updateSourceStats` call sites, rehydration, test literals, the reconcile preservation assertion.
- S1.3 ruling 4: `StatusView`, openapi.yaml, sample regeneration, `types.ts`, mock fields.
Acceptance (S1):
- [ ] `zig build test` passes; the migrations baseline test proves
@@ -400,10 +178,7 @@ Acceptance (S1):
### Session S2: UI and docs (needs S1)
Owns: `web/src/features/blocklists/BlocklistsPage.tsx`,
`web/src/features/blocklists/SourceStatusSection.tsx`,
`web/src/features/blocklists/BlocklistsPage.test.tsx` (rendering assertions),
`docs/reference/configuration.md`, `docs/tutorial/first-run.md`.
Owns: `web/src/features/blocklists/BlocklistsPage.tsx`, `web/src/features/blocklists/SourceStatusSection.tsx`, `web/src/features/blocklists/BlocklistsPage.test.tsx` (rendering assertions), `docs/reference/configuration.md`, `docs/tutorial/first-run.md`.
- S2.1 ruling 5: both columns, the note paragraph, the rendering assertions.
- S2.2 ruling 6: the two doc edits.
@@ -419,22 +194,12 @@ Acceptance (S2):
### Orchestrator
Verify S1 acceptance before starting S2. After S2, run the full gate set
(`zig build test`, `zig build test -Dintegration`, `test-aarch64` if qemu is
present, `cd web && npm test`, `npm run assert-bundled`), then a live smoke
against a scratch server with two real sources:
Verify S1 acceptance before starting S2. After S2, run the full gate set (`zig build test`, `zig build test -Dintegration`, `test-aarch64` if qemu is present, `cd web && npm test`, `npm run assert-bundled`), then a live smoke against a scratch server with two real sources:
- `https://easylist.to/easylist/easylist.txt` — a browser-targeted list;
expect a `skipped_unsupported` several times its `domains` (do not assert an
exact number; assert the ratio and non-zero).
- `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` — expect
`skipped_unsupported` 0.
- `https://easylist.to/easylist/easylist.txt` — a browser-targeted list; expect a `skipped_unsupported` several times its `domains` (do not assert an exact number; assert the ratio and non-zero).
- `https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts` expect `skipped_unsupported` 0.
Verify the numbers appear in the `POST /api/blocklists/update` response, in
`GET /api/blocklists`, and in both UI tables. Then restart the server and
verify the counts survive into the status table without a refresh — that is
ruling 3's rehydration working against the real database. Record deviations
in `## Recorded (implementation)`.
Verify the numbers appear in the `POST /api/blocklists/update` response, in `GET /api/blocklists`, and in both UI tables. Then restart the server and verify the counts survive into the status table without a refresh — that is ruling 3's rehydration working against the real database. Record deviations in `## Recorded (implementation)`.
## Module layout
@@ -480,89 +245,40 @@ New files: none. Deleted surface: none.
## Anti-requirements
- No migration step, no `ddl_v2`, no runtime schema probing. The baseline is
edited in place per §3.7; a pre-edit scratch database is deleted, not
reconciled.
- No merging of the two skip counters into one number, anywhere — not in the
API, not in a UI total, not in prose. They are different facts.
- No "this list targets browsers" heuristic: no threshold, badge, warning
color or ratio computation in the UI. The number plus the note paragraph is
the surface; a cutoff would be an invented policy.
- No conditional rendering of the new column. It shows at 0 like every other
counter column.
- No change to `nxdns export` / `import` ZON: compile statistics are not
configuration.
- No per-line diagnostics, no sample of skipped lines in logs, API or UI —
counters and health surfaces, not log spam (AGENTS.md).
- No migration step, no `ddl_v2`, no runtime schema probing. The baseline is edited in place per §3.7; a pre-edit scratch database is deleted, not reconciled.
- No merging of the two skip counters into one number, anywhere — not in the API, not in a UI total, not in prose. They are different facts.
- No "this list targets browsers" heuristic: no threshold, badge, warning color or ratio computation in the UI. The number plus the note paragraph is the surface; a cutoff would be an invented policy.
- No conditional rendering of the new column. It shows at 0 like every other counter column.
- No change to `nxdns export` / `import` ZON: compile statistics are not configuration.
- No per-line diagnostics, no sample of skipped lines in logs, API or UI — counters and health surfaces, not log spam (AGENTS.md).
- No change to `rejectedWithoutEntries` or `failNoValidEntries` semantics.
- No hand edits to `contractSamples.gen.ts`.
- No new columns beyond the one. See ruling 7 for what that leaves open and
why — the reason is a scope decision, not a claim that the other counters
are already visible.
- No new columns beyond the one. See ruling 7 for what that leaves open and why — the reason is a scope decision, not a claim that the other counters are already visible.
## Addendum (post-`1bce81e`): the body checksum is framed
A filtering correctness bug found by the implementation review, predating this
milestone. Fixed as a follow-up commit; this addendum is its design record —
the user ruled it a follow-up, not a milestone 25.
A filtering correctness bug found by the implementation review, predating this milestone. Fixed as a follow-up commit; this addendum is its design record — the user ruled it a follow-up, not a milestone 25.
### The defect
`bodyChecksum` (manager.zig:1660) and the compiler's incremental hashing
(compiler.zig:105-108) both digest the unframed concatenation
`list ++ wild ++ allow`. The compiler strips `*.` from a wildcard candidate
(compiler.zig:130-133), so upstream `a.example` (list `a.example\n`, wild
empty) and upstream `*.a.example` (list empty, wild `a.example\n`) hash the
same bytes. `diskBodiesMatch` (manager.zig:1085) recomputes with the same
function, so the refresh takes the unchanged-checksum branch and a list that
switches an exact block to a wildcard block never takes effect. Ruling 3's
entry-count argument rested on the digest distinguishing bodies; it does not,
and the strikethrough above records that.
`bodyChecksum` (manager.zig:1660) and the compiler's incremental hashing (compiler.zig:105-108) both digest the unframed concatenation `list ++ wild ++ allow`. The compiler strips `*.` from a wildcard candidate (compiler.zig:130-133), so upstream `a.example` (list `a.example\n`, wild empty) and upstream `*.a.example` (list empty, wild `a.example\n`) hash the same bytes. `diskBodiesMatch` (manager.zig:1085) recomputes with the same function, so the refresh takes the unchanged-checksum branch and a list that switches an exact block to a wildcard block never takes effect. Ruling 3's entry-count argument rested on the digest distinguishing bodies; it does not, and the strikethrough above records that.
### The fix: a `0x00` separator after each body
The digest becomes `SHA-256(list ‖ 00 ‖ wild ‖ 00 ‖ allow ‖ 00)` — one zero
byte fed to the hasher **after each of the three bodies**, same order as
today. Soundness: a compiled body holds only validated name bytes and `\n`;
`addCandidate` rejects any byte ≥ `0x80` or control byte
(compiler.zig:151-155), so `0x00` cannot occur in a body and the three
boundaries are unambiguous. Two distinct `(list, wild, allow)` triples cannot
produce one digest short of SHA-256 itself.
The digest becomes `SHA-256(list ‖ 00 ‖ wild ‖ 00 ‖ allow ‖ 00)` — one zero byte fed to the hasher **after each of the three bodies**, same order as today. Soundness: a compiled body holds only validated name bytes and `\n`; `addCandidate` rejects any byte ≥ `0x80` or control byte (compiler.zig:151-155), so `0x00` cannot occur in a body and the three boundaries are unambiguous. Two distinct `(list, wild, allow)` triples cannot produce one digest short of SHA-256 itself.
A separator, not a length prefix, because the compiler hashes while it emits
and does not know a body's length up front; a trailing byte needs no pre-pass.
A separator, not a length prefix, because the compiler hashes while it emits and does not know a body's length up front; a trailing byte needs no pre-pass.
Both producers move together or every refresh republishes forever:
`compiler.compile` feeds the byte after each `emit` call, and
`manager.bodyChecksum` feeds it after each body slice. Export the separator as
a `pub const` from `compiler.zig` and have `bodyChecksum` use it — two literal
`0`s in two files is how the next drift starts.
Both producers move together or every refresh republishes forever: `compiler.compile` feeds the byte after each `emit` call, and `manager.bodyChecksum` feeds it after each body slice. Export the separator as a `pub const` from `compiler.zig` and have `bodyChecksum` use it — two literal `0`s in two files is how the next drift starts.
### Consequences, accepted
- Every stored checksum changes once. Zero installs; a development data
directory fails its startup checksum verification and the startup refresh
pass re-downloads and repairs it (or delete the data directory — the ruling
1 stance).
- Checksum compatibility with pre-exception digests — milestone 21 ruling 3's
"empty allow body reproduces the old digest" property — is dead, and its
rationale comments go with it: `Result.checksum` (compiler.zig:44-53), the
`Header` doc and `bodyChecksum` doc (manager.zig:218-225, 1645-1648),
`SourceStats.checksum` (sources_repo.zig), and the PLAN §3.8 sentence
"keeps the digest it had when only two existed" (PLAN.md:101). All are
rewritten to state the framed digest. The body order stays list, wild,
allow.
- Every stored checksum changes once. Zero installs; a development data directory fails its startup checksum verification and the startup refresh pass re-downloads and repairs it (or delete the data directory — the ruling 1 stance).
- Checksum compatibility with pre-exception digests — milestone 21 ruling 3's "empty allow body reproduces the old digest" property — is dead, and its rationale comments go with it: `Result.checksum` (compiler.zig:44-53), the `Header` doc and `bodyChecksum` doc (manager.zig:218-225, 1645-1648), `SourceStats.checksum` (sources_repo.zig), and the PLAN §3.8 sentence "keeps the digest it had when only two existed" (PLAN.md:101). All are rewritten to state the framed digest. The body order stays list, wild, allow.
### The branch reads all five fresh
On the checksum-unchanged path, all five stat fields —
`domain_count`, `wildcard_count`, `exception_count`, `skipped_regex_count`,
`skipped_unsupported_count` — now read from `compiled.result.counts`;
`checksum` keeps passing `stored`. With framing, fresh and stored entry counts
are provably equal, so this is not a correctness requirement — it removes the
per-field vouching argument from the code entirely, and any future digest
weakness then degrades to consistent stats rather than a split between
database and status table.
On the checksum-unchanged path, all five stat fields — `domain_count`, `wildcard_count`, `exception_count`, `skipped_regex_count`, `skipped_unsupported_count` — now read from `compiled.result.counts`; `checksum` keeps passing `stored`. With framing, fresh and stored entry counts are provably equal, so this is not a correctness requirement — it removes the per-field vouching argument from the code entirely, and any future digest weakness then degrades to consistent stats rather than a split between database and status table.
### Regression tests (each watched failing with the framing reverted)