milestone 12: performance bench harness, measured docs, aarch64 tests under qemu and no-dist size assert

2026-08-02 16:08:23 +02:00
parent bdb6ffab7a
commit d522b1f947
5 changed files with 760 additions and 0 deletions
@@ -0,0 +1,81 @@
+# Performance
+
+PLAN §18 sets the targets; `tools/bench.zig` measures the three that are
+measurable in-process. Run it with:
+
+```
+zig build bench -Doptimize=ReleaseFast
+```
+
+Subcommands `filter|cache|compile|all` (default `all`) select a suite; flags
+`--domains=N` (default 1,000,000), `--iters=N` (default 200,000) and `--seed=N`
+(default 0x5eed) shape the load. The default run is informational; `--assert`
+exits non-zero when a target below is exceeded.
+
+## Targets (PLAN §18)
+
+| Target | Where it is checked |
+| --- | --- |
+| Sustained ≥ 100 qps on Raspberry Pi 5 | End-to-end against the real binary on the Pi (see below); not a harness number |
+| Blocklist lookup p95 < 1 ms | `bench filter`: `matcher.normalize` + `Snapshot.evaluate` per op |
+| Cached response p95 < 5 ms | `bench cache`: `buildKey` + `DnsCache.get` + `packet.setId` per op |
+| Memory with ~1M blocked domains < 100 MB | `bench filter`: VmRSS with the 1M-domain snapshot loaded |
+| Stripped static binary < 10 MB per arch (< 15 MB with embedded frontend) | CI size assert on the `cross` artifacts |
+
+## Measured: x86_64 development host (2026-08-02)
+
+Intel Core i7-14700K, Linux 6.18, Zig 0.16.0, `-Doptimize=ReleaseFast`,
+defaults (1,000,000 domains, 200,000 iterations per suite, seed 0x5eed).
+**This is not the target platform** — the Pi 5's Cortex-A76 is far slower and
+these numbers do not transfer; they establish the harness works and set a
+baseline for regressions on the machine development happens on.
+
+```
+suite           ops     p50(us)     p95(us)     p99(us)     max(us)
+filter       200000        0.11        0.18        0.27       16.41
+  blocked 66699/200000, Snapshot.memoryBytes 28.0 MiB, VmRSS 31.8 MiB
+  target p95 < 1ms: PASS
+  target VmRSS < 100 MiB: PASS
+cache        200000        0.10        0.14        0.17        3.53
+  hits 100000/200000, DnsCache.memoryBytes 4.3 MiB, VmRSS 7.6 MiB
+  target p95 < 5ms: PASS
+compile     1000000   wall 96.025ms, 10413949 lines/s, 1000000 domains kept (informational)
+```
+
+Every in-process §18 target passes on this host: the two latency targets by
+three-to-four orders of magnitude, the memory target by about 3x.
+
+Two memory figures appear on purpose. `Snapshot.memoryBytes` /
+`DnsCache.memoryBytes` are the in-repo accounting of the structures themselves
+(the regression guard); VmRSS is what the kernel actually holds resident for
+the whole process, allocator slack and code included. The truth sits between
+them, and the §18 memory target is judged on VmRSS. The filter suite frees the
+generated list source before reading VmRSS, so the number reflects the loaded
+snapshot rather than the generator. The cache line's VmRSS is lower because the
+filter suite's snapshot has been freed by then.
+
+## Raspberry Pi 5 (target platform)
+
+To be measured on hardware. One command, run on the Pi:
+
+```
+zig build bench -Doptimize=ReleaseFast -- --assert
+```
+
+| Target | Result |
+| --- | --- |
+| Blocklist lookup p95 < 1 ms | to be measured on hardware |
+| Cached response p95 < 5 ms | to be measured on hardware |
+| Memory with ~1M blocked domains < 100 MB | to be measured on hardware |
+| Sustained ≥ 100 qps | to be measured on hardware, end-to-end |
+
+The qps target is end-to-end and belongs to the real binary, not the harness:
+run `nxdns run` on the Pi and drive it over the LAN with a DNS load generator
+(for example `dnsperf`) against real blocklists.
+
+## Why CI does not gate on performance
+
+Required CI stays deterministic (AGENTS.md); latency assertions on shared
+runners measure the runner's noisy neighbours, not nxdns, and a perf gate that
+flakes trains people to re-run it. The bench exists for hardware you control:
+run `--assert` on the Pi, where the numbers mean something.