filter: separate the compiled bodies in the checksum, a name moving between them was invisible

This commit is contained in:
2026-08-14 02:09:25 +02:00
parent 1bce81eea0
commit 21c5ce1f36
6 changed files with 274 additions and 87 deletions
+86 -2
View File
@@ -118,9 +118,13 @@ the only thing a restart reads — rehydration never reparses headers.
Both skip counters take `compiled.result.counts` on this path:
`.skipped_regex_count = compiled.result.counts.skipped_regex` and
`.skipped_unsupported_count = compiled.result.counts.skipped_unsupported`.
`domain_count`, `wildcard_count` and `exception_count` keep reading from
~~`domain_count`, `wildcard_count` and `exception_count` keep reading from
`row` — they count written entries, so an unchanged checksum does mean an
unchanged value for them.
unchanged value for them.~~ **Corrected post-commit:** that claim was false.
The unframed digest could not distinguish an entry in `.list` from the same
entry in `.wild`, so an unchanged checksum did not vouch for the entry
counts either. The addendum below frames the hash and makes all five stat
fields read fresh on this path.
This needs a test the author has watched fail with the fix reverted: two
compiles of bodies whose written entries are identical but whose skipped
@@ -495,3 +499,83 @@ New files: none. Deleted surface: none.
- No new columns beyond the one. See ruling 7 for what that leaves open and
why — the reason is a scope decision, not a claim that the other counters
are already visible.
## Addendum (post-`1bce81e`): the body checksum is framed
A filtering correctness bug found by the implementation review, predating this
milestone. Fixed as a follow-up commit; this addendum is its design record —
the user ruled it a follow-up, not a milestone 25.
### The defect
`bodyChecksum` (manager.zig:1660) and the compiler's incremental hashing
(compiler.zig:105-108) both digest the unframed concatenation
`list ++ wild ++ allow`. The compiler strips `*.` from a wildcard candidate
(compiler.zig:130-133), so upstream `a.example` (list `a.example\n`, wild
empty) and upstream `*.a.example` (list empty, wild `a.example\n`) hash the
same bytes. `diskBodiesMatch` (manager.zig:1085) recomputes with the same
function, so the refresh takes the unchanged-checksum branch and a list that
switches an exact block to a wildcard block never takes effect. Ruling 3's
entry-count argument rested on the digest distinguishing bodies; it does not,
and the strikethrough above records that.
### The fix: a `0x00` separator after each body
The digest becomes `SHA-256(list ‖ 00 ‖ wild ‖ 00 ‖ allow ‖ 00)` — one zero
byte fed to the hasher **after each of the three bodies**, same order as
today. Soundness: a compiled body holds only validated name bytes and `\n`;
`addCandidate` rejects any byte ≥ `0x80` or control byte
(compiler.zig:151-155), so `0x00` cannot occur in a body and the three
boundaries are unambiguous. Two distinct `(list, wild, allow)` triples cannot
produce one digest short of SHA-256 itself.
A separator, not a length prefix, because the compiler hashes while it emits
and does not know a body's length up front; a trailing byte needs no pre-pass.
Both producers move together or every refresh republishes forever:
`compiler.compile` feeds the byte after each `emit` call, and
`manager.bodyChecksum` feeds it after each body slice. Export the separator as
a `pub const` from `compiler.zig` and have `bodyChecksum` use it — two literal
`0`s in two files is how the next drift starts.
### Consequences, accepted
- Every stored checksum changes once. Zero installs; a development data
directory fails its startup checksum verification and the startup refresh
pass re-downloads and repairs it (or delete the data directory — the ruling
1 stance).
- Checksum compatibility with pre-exception digests — milestone 21 ruling 3's
"empty allow body reproduces the old digest" property — is dead, and its
rationale comments go with it: `Result.checksum` (compiler.zig:44-53), the
`Header` doc and `bodyChecksum` doc (manager.zig:218-225, 1645-1648),
`SourceStats.checksum` (sources_repo.zig), and the PLAN §3.8 sentence
"keeps the digest it had when only two existed" (PLAN.md:101). All are
rewritten to state the framed digest. The body order stays list, wild,
allow.
### The branch reads all five fresh
On the checksum-unchanged path, all five stat fields —
`domain_count`, `wildcard_count`, `exception_count`, `skipped_regex_count`,
`skipped_unsupported_count` — now read from `compiled.result.counts`;
`checksum` keeps passing `stored`. With framing, fresh and stored entry counts
are provably equal, so this is not a correctness requirement — it removes the
per-field vouching argument from the code entirely, and any future digest
weakness then degrades to consistent stats rather than a split between
database and status table.
### Regression tests (each watched failing with the framing reverted)
- [ ] Compiler: compiling `a.example` and compiling `*.a.example` produce
**different** checksums. This is the collision itself.
- [ ] Manager: publish a source whose body is `a.example`; refresh it with
upstream bytes `*.a.example`. Assert the unchanged-checksum branch is
**not** taken: the on-disk `.wild` stripped body is `a.example\n`, the
`.list` body is empty, and the database row reads `domain_count = 0`,
`wildcard_count = 1`.
- [ ] Agreement: `bodyChecksum` over the three stripped on-disk bodies equals
`compile`'s reported checksum for a fixture where **all three bodies are
non-empty** (extend the existing agreement coverage if it exists; the
empty-allow case no longer exercises the third frame).
Report the observed failure output for each, per the ruling 3 convention.