Performance targets and what the tests prove

Two related questions: why the performance numbers are the numbers, and why CI does not enforce them — and then, less comfortably, what a green test suite here does and does not tell you.

For the targets and the measured results as data, see reference/performance.md; to run the bench yourself, how-to/measure-performance.md.

Where the targets come from

PLAN §18 sets five:

sustained ≥ 100 qps on a Raspberry Pi 5;
blocklist lookup p95 < 1 ms;
cached response p95 < 5 ms;
memory with ~1M blocked domains < 100 MiB;
stripped static binary ≤ 10,485,760 bytes per arch, ≤ 15,728,640 bytes with the embedded frontend.

They are household-scale numbers, and they are deliberately unambitious. 100 qps is far more than a house generates; the point of the target is not speed but that a Pi 5 with an SD card never becomes the reason the internet feels broken. The latency targets exist for the same reason: DNS sits in front of every connection anyone makes, so the failure people notice is not throughput but a stall. The memory target is what keeps a 1M-entry blocklist from competing with everything else on a 4 GB board. The binary-size target is about what a static single-binary deployment is for — if it does not fit on a constrained box and copy over a slow link in one step, the packaging decision has not paid for itself.

tools/bench.zig (zig build bench) measures the three that are measurable in-process: filter (normalize plus snapshot evaluate against a ~1M-entry snapshot), cache (key build plus cache get plus id patch), and compile (the blocklist compiler over a 1M-line body, informational — there is no §18 target for it because no prior datapoint exists). Memory comes from /proc/self/status VmRSS. The qps target is not in the harness at all: it is end-to-end against the real binary with a DNS load generator, because a harness number for "queries per second" would measure the harness.

The bench is tools/, not src/, on purpose: src/ is the shipped product, and src/tests.zig aggregates everything shippable.

Why CI does not gate on performance

Required CI stays deterministic (AGENTS.md). Latency assertions on shared runners measure the runner's noisy neighbours; the same commit passes and fails depending on what else the host is doing. A gate that flakes does not protect anything — it trains people to re-run the job, and once re-running is routine, a real regression gets re-run too. The flaky gate is worse than no gate, because it also consumes the attention a real gate would need.

So the bench defaults to informational, and --assert — which exits non-zero on a missed target — exists for hardware you control. Run it on the Pi, where the numbers describe the machine the software actually has to run on. The x86_64 development-host numbers in reference/performance.md are a regression baseline for the machine development happens on, not a claim about the target platform; a Cortex-A76 is far slower and those numbers do not transfer.

CI does gate on the one performance property that is deterministic: binary size. The package job builds the release artifacts and zig build verify-dist asserts both §18 budgets against them. Size is a function of the input, not of the runner's mood, so it is exactly the kind of thing a shared runner can measure honestly.

The budgets are asserted as exact byte counts, and the asset-free budget gets its own build against a generated empty assets directory rather than against web/dist-placeholder. The placeholder is not buildable by dist at all — that is the guard against a release shipping a stub admin page — and letting it back in through a size check would have defeated the guard for the sake of one number.

What the test suite is

Every blocking check lives in .gitea/workflows/gates.yml, which is a workflow_call workflow with nothing in it but jobs. ci.yml calls it on push and pull request for master, and release.yml calls it before it builds anything publishable. That shape exists for one reason: a check that lived in ci.yml alone would be a check a release could skip.

Five jobs, all required:

test — the Zig suite with -Dintegration.
test-aarch64 — the same suite cross-built for aarch64 and executed under qemu-user, plain tier only.
frontend — format, lint, typecheck, the vitest cases, build.
package — zig build dist and zig build verify-dist, which is where the size budgets, the ELF static-linkage assert and the archive layout checks are.
container — builds the image, asserts the binary inside it is byte-identical to the one in the matching tarball, and smoke-tests it by booting the container and polling /api/health.

The Zig suite has three tiers, gated by build flags:

plain zig build test — pure logic. No sockets, no threads, no clock budgets. This is the tier the purity rule (architecture.md) exists to make possible.
-Dintegration — hermetic integration: loopback sockets, :memory: databases, temp directories. Nothing leaves the host.
-Dlive — the only tests that reach the public internet (DoH and DoT handshakes against real resolvers). Four tests, and they run in a manual-dispatch workflow, never on push or pull request.

The aarch64 job runs the plain tier only. The integration tests are multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown turns those budgets into a flake source — the same reasoning that keeps the bench out of CI. What aarch64 needs to prove is portable correctness of the DNS, filter and cache logic, and the plain tier is exactly that.

At the time of writing, plain zig build test is 1175 of 1288 passing with 113 skipped and 0 failed, the skips being the integration-gated tests. Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with -Dintegration (the 4 skips are the live-network tests) and 1159 passing under qemu, 0 failed in each.

What it does not prove

The suite is hermetic by design. That is the right default: it is fast, it is deterministic, it can gate merges. But hermetic and correct are different properties, and the gap has already cost this project twice.

The blocklist download aborted the process on first real use. The fetcher constructed the HTTP response reader over transfer_buf and then read into that same buffer. Reader.readSliceShort starts by @memcpy-ing the reader's already-buffered bytes into the caller's destination — so source and destination were the same allocation, and Zig's @memcpy requires them not to overlap. It aborts.

The reason no test caught it is precise and instructive. The copy length is zero whenever the reader has nothing buffered, and a zero-length @memcpy is fine. Bytes only accumulate in the reader's own buffer when a read comes back short of filling the destination and the loop goes round again — that is, when the body arrives in more than one stream call. The loopback fixture answers every request with one small in-memory body that lands in a single read, and the one over-size test never streams a byte, because the fetcher refuses an oversized content-length on the response head. Every test in the suite was on the zero-length-memcpy side of the branch. The first real download — a multi-megabyte list over TLS across the WAN, arriving in many TCP segments — was on the other side, and took the process down. The fix (commit 35f2324) streams the body straight into the caller's writer, so the reader's buffer is never a destination slice, and it came with four regression tests that put a fully-buffered reader into exactly the state the old code could not survive.

A stale embedded SPA bundle shipped a settings page that crashed on load, while 121 web tests passed. web/dist/ is gitignored and -Dweb-dist=web/dist embeds whatever bytes are sitting in that directory. The frontend tests ran against the sources, in jsdom, and were green; the binary carried an older build. The tests were testing something the artifact did not contain.

Note what these two have in common. Neither was a logic bug that a better unit test would have caught. One lived in the seam between the pure core and its one I/O edge; the other lived in the seam between two build systems. Hermetic tests are constructed to exclude exactly those seams — that is what makes them hermetic.

The lesson, and where it now lives

Green hermetic tests are a floor, not a ceiling. They prove the logic is consistent with itself. They cannot prove the program works, because the things they deliberately exclude — real network reads, real TLS, real file sizes, real build artifacts — are where a program meets reality.

The response is not to make CI non-deterministic. It is to require that the real paths get exercised by a human before work is called done. That is now ruling 3 of specs/milestone-13.md: every command block in the tutorial and the how-to pages is executed verbatim, on the host, by the session that writes it, and a command that cannot run there is marked in the page as unverified with the reason. Documentation written from source-reading alone is how both of these shipped; documentation that has been run is a second, independent test suite that exercises precisely the paths the hermetic one skips.

Two honest gaps remain, stated so nobody has to rediscover them:

Nothing in the suite drives a multi-read HTTP body through the fetcher end to end. The regression tests cover pumpBody directly over a pre-buffered reader; the loopback fixture still sends one small body per connection.
There is no freshness check on web/dist. CI cannot embed a stale bundle, because the jobs that pass -Dweb-dist rebuild the frontend immediately beforehand. A local build can, and will do it without a warning.

What a signed release does not prove either

The same distinction applies one level out, to the artifacts. A release is signed, and the signature is worth having: it says the artifact came from this project's pipeline and reached you unaltered. It does not say the binary was built from the source in this repository, because the machine that ran the build also held the signing key. An attacker with that machine produces something that verifies cleanly and contains whatever they put in it.

The control that closes that gap is a reproducibility gate — an independent build, in a different directory on a different machine, landing on the same bytes. It does not exist. It is a recorded deferral (specs/milestone-14.md ruling 12), not something nobody thought of, and until it exists no document here describes the build as reproducible: nobody has measured whether it is. The cheap inputs to reproducibility are already in place — gzip -n, --mtime=@0, LC_ALL=C, TZ=UTC, exact Zig and Node pins — which makes the gate cheap to add later and proves nothing on its own.

What the release pipeline is required to hold to is narrower: two runs of zig build dist on the same commit in the same directory produce byte-identical tarballs. Same-directory determinism is a much weaker property than reproducibility, and conflating the two is exactly the kind of claim this page exists to refuse.

Verify a release states the same limits where an operator will actually meet them, and gives the rebuild-and-compare recipe with the caveat that a differing hash is not evidence of tampering while this gap is open.

12 KiB Raw Blame History