mokhtar/nxdns
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

milestone 13: restructure docs to diataxis, tutorial, every command executed

Browse Source
This commit is contained in:
mokhtar
2026-08-02 18:02:58 +02:00
parent 35f23240e7
commit 16c9de2414
26 changed files with 3627 additions and 901 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/explanation/performance-and-testing.md
+161
View File
@@ -0,0 +1,161 @@
# Performance targets and what the tests prove
Two related questions: why the performance numbers are the numbers, and why CI
does not enforce them — and then, less comfortably, what a green test suite
here does and does not tell you.
For the targets and the measured results as data, see
[reference/performance.md](../reference/performance.md); to run the bench
yourself, [how-to/measure-performance.md](../how-to/measure-performance.md).
## Where the targets come from
PLAN §18 sets five:
- sustained ≥ 100 qps on a Raspberry Pi 5;
- blocklist lookup p95 < 1 ms;
- cached response p95 < 5 ms;
- memory with ~1M blocked domains < 100 MB;
- stripped static binary < 10 MB per arch, < 15 MB with the embedded frontend.
They are household-scale numbers, and they are deliberately unambitious. 100
qps is far more than a house generates; the point of the target is not speed
but that a Pi 5 with an SD card never becomes the reason the internet feels
broken. The latency targets exist for the same reason: DNS sits in front of
every connection anyone makes, so the failure people notice is not throughput
but a stall. The memory target is what keeps a 1M-entry blocklist from
competing with everything else on a 4 GB board. The binary-size target is
about what a static single-binary deployment is for — if it does not fit on a
constrained box and copy over a slow link in one step, the packaging decision
has not paid for itself.
`tools/bench.zig` (`zig build bench`) measures the three that are measurable
in-process: `filter` (normalize plus snapshot evaluate against a ~1M-entry
snapshot), `cache` (key build plus cache get plus id patch), and `compile`
(the blocklist compiler over a 1M-line body, informational — there is no §18
target for it because no prior datapoint exists). Memory comes from
`/proc/self/status` VmRSS. The qps target is not in the harness at all: it is
end-to-end against the real binary with a DNS load generator, because a
harness number for "queries per second" would measure the harness.
The bench is `tools/`, not `src/`, on purpose: `src/` is the shipped product,
and `src/tests.zig` aggregates everything shippable.
## Why CI does not gate on performance
Required CI stays deterministic (AGENTS.md). Latency assertions on shared
runners measure the runner's noisy neighbours; the same commit passes and
fails depending on what else the host is doing. A gate that flakes does not
protect anything — it trains people to re-run the job, and once re-running is
routine, a real regression gets re-run too. The flaky gate is worse than no
gate, because it also consumes the attention a real gate would need.
So the bench defaults to informational, and `--assert` — which exits non-zero
on a missed target — exists for hardware you control. Run it on the Pi, where
the numbers describe the machine the software actually has to run on. The
x86_64 development-host numbers in
[reference/performance.md](../reference/performance.md) are a regression
baseline for the machine development happens on, not a claim about the target
platform; a Cortex-A76 is far slower and those numbers do not transfer.
CI does gate on the one performance property that *is* deterministic: binary
size. The `cross` job strips the release binaries and asserts them under the
§18 budgets. Size is a function of the input, not of the runner's mood, so it
is exactly the kind of thing a shared runner can measure honestly.
## What the test suite is
The blocking CI (Gitea Actions, `.gitea/workflows/ci.yml`) runs five jobs, all
required: the Zig suite with `-Dintegration`; the same suite cross-built for
aarch64 and executed under qemu-user; the frontend (format, lint, typecheck,
121 vitest cases, build); the cross-build with the two stripped-size asserts;
and a Docker smoke run that boots the image and polls `/api/health`.
The Zig suite has three tiers, gated by build flags:
- **plain `zig build test`** — pure logic. No sockets, no threads, no clock
budgets. This is the tier the purity rule
([architecture.md](architecture.md)) exists to make possible.
- **`-Dintegration`** — hermetic integration: loopback sockets, `:memory:`
databases, temp directories. Nothing leaves the host.
- **`-Dlive`** — the only tests that reach the public internet (DoH and DoT
handshakes against real resolvers). Four tests, and they run in a
manual-dispatch workflow, never on push or pull request.
The aarch64 job runs the plain tier only. The integration tests are
multithreaded loopback TLS with wall-clock budgets, and qemu-user's slowdown
turns those budgets into a flake source — the same reasoning that keeps the
bench out of CI. What aarch64 needs to prove is portable correctness of the
DNS, filter and cache logic, and the plain tier is exactly that.
At the time of writing, plain `zig build test` is 1175 of 1288 passing with
113 skipped and 0 failed, the skips being the integration-gated tests.
Milestone 12 recorded the other two tiers on the same tree: 1280 of 1284 with
`-Dintegration` (the 4 skips are the live-network tests) and 1159 passing
under qemu, 0 failed in each.
## What it does not prove
The suite is hermetic by design. That is the right default: it is fast, it is
deterministic, it can gate merges. But hermetic and correct are different
properties, and the gap has already cost this project twice.
**The blocklist download aborted the process on first real use.** The fetcher
constructed the HTTP response reader over `transfer_buf` and then read *into*
that same buffer. `Reader.readSliceShort` starts by `@memcpy`-ing the reader's
already-buffered bytes into the caller's destination — so source and
destination were the same allocation, and Zig's `@memcpy` requires them not to
overlap. It aborts.
The reason no test caught it is precise and instructive. The copy length is
zero whenever the reader has nothing buffered, and a zero-length `@memcpy` is
fine. Bytes only accumulate in the reader's own buffer when a read comes back
short of filling the destination and the loop goes round again — that is, when
the body arrives in more than one stream call. The loopback fixture answers
every request with one small in-memory body that lands in a single read, and
the one over-size test never streams a byte, because the fetcher refuses an
oversized `content-length` on the response head. Every test in the suite was
on the zero-length-memcpy side of the branch. The first real download — a
multi-megabyte list over TLS across the WAN, arriving in many TCP segments —
was on the other side, and took the process down. The fix (commit 35f2324)
streams the body straight into the caller's writer, so the reader's buffer is
never a destination slice, and it came with four regression tests that put a
fully-buffered reader into exactly the state the old code could not survive.
**A stale embedded SPA bundle shipped a settings page that crashed on load,
while 121 web tests passed.** `web/dist/` is gitignored and
`-Dweb-dist=web/dist` embeds whatever bytes are sitting in that directory. The
frontend tests ran against the sources, in jsdom, and were green; the binary
carried an older build. The tests were testing something the artifact did not
contain.
Note what these two have in common. Neither was a logic bug that a better unit
test would have caught. One lived in the seam between the pure core and its
one I/O edge; the other lived in the seam between two build systems. Hermetic
tests are constructed to exclude exactly those seams — that is what makes them
hermetic.
## The lesson, and where it now lives
Green hermetic tests are a floor, not a ceiling. They prove the logic is
consistent with itself. They cannot prove the program works, because the
things they deliberately exclude — real network reads, real TLS, real file
sizes, real build artifacts — are where a program meets reality.
The response is not to make CI non-deterministic. It is to require that the
real paths get exercised by a human before work is called done. That is now
ruling 3 of `specs/milestone-13.md`: every command block in the tutorial and
the how-to pages is executed verbatim, on the host, by the session that writes
it, and a command that cannot run there is marked in the page as unverified
with the reason. Documentation written from source-reading alone is how both
of these shipped; documentation that has been run is a second, independent
test suite that exercises precisely the paths the hermetic one skips.
Two honest gaps remain, stated so nobody has to rediscover them:
- Nothing in the suite drives a multi-read HTTP body through the fetcher end
to end. The regression tests cover `pumpBody` directly over a pre-buffered
reader; the loopback fixture still sends one small body per connection.
- There is no freshness check on `web/dist`. CI cannot embed a stale bundle,
because the jobs that pass `-Dweb-dist` rebuild the frontend immediately
beforehand. A local build can, and will do it without a warning.