164 lines
6.2 KiB
Markdown
164 lines
6.2 KiB
Markdown
# Measure performance
|
|
|
|
`tools/bench.zig` measures the three things nxdns can measure in-process:
|
|
blocklist lookup latency, cache-hit latency, and blocklist compile throughput.
|
|
Sustained query rate is not one of them — that one is end-to-end and needs a
|
|
load generator pointed at a running server.
|
|
|
|
The numbers this project treats as targets, and the numbers measured so far,
|
|
are in the [performance reference](../reference/performance.md). Why those
|
|
targets exist and why CI does not gate on them is in
|
|
[performance and testing](../explanation/performance-and-testing.md).
|
|
|
|
## Run the whole bench
|
|
|
|
```sh
|
|
zig build bench -Doptimize=ReleaseFast
|
|
```
|
|
|
|
That runs all three suites with the defaults: 1,000,000 domains, 200,000
|
|
iterations per suite, seed `0x5eed`. It takes minutes, most of it generating and
|
|
loading the million-domain list.
|
|
|
|
`-Doptimize=ReleaseFast` is not optional if you want the numbers to mean
|
|
anything. A Debug build says so before it prints:
|
|
|
|
```
|
|
warning: Debug build; run with -Doptimize=ReleaseFast for meaningful numbers
|
|
```
|
|
|
|
## Run one suite, smaller
|
|
|
|
Everything after `--` goes to the harness. A suite name selects one of
|
|
`filter`, `cache`, `compile` (the default is `all`), and `--domains` /
|
|
`--iters` shrink the load:
|
|
|
|
```sh
|
|
zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000
|
|
```
|
|
|
|
```
|
|
nxdns bench suite=filter domains=100000 iters=20000 seed=0x5eed optimize=ReleaseFast
|
|
|
|
suite ops p50(us) p95(us) p99(us) max(us)
|
|
filter 20000 0.14 0.25 0.27 0.51
|
|
blocked 6670/20000, Snapshot.memoryBytes 3.0 MiB, VmRSS 5.4 MiB
|
|
target p95 < 1ms: PASS
|
|
target VmRSS < 100 MiB: PASS
|
|
```
|
|
|
|
A reduced run is good for checking the harness works and for a rough
|
|
regression signal. It is not a result: the memory figure scales with
|
|
`--domains`, so 100,000 domains says nothing about the 1,000,000-domain memory
|
|
target.
|
|
|
|
The other two suites:
|
|
|
|
```sh
|
|
zig build bench -Doptimize=ReleaseFast -- cache --iters=20000
|
|
```
|
|
|
|
```
|
|
suite ops p50(us) p95(us) p99(us) max(us)
|
|
cache 20000 0.12 0.21 0.23 0.54
|
|
hits 10000/20000, DnsCache.memoryBytes 4.3 MiB, VmRSS 6.2 MiB
|
|
target p95 < 5ms: PASS
|
|
```
|
|
|
|
```sh
|
|
zig build bench -Doptimize=ReleaseFast -- compile --domains=100000
|
|
```
|
|
|
|
```
|
|
suite ops p50(us) p95(us) p99(us) max(us)
|
|
compile 100000 wall 11.512ms, 8686215 lines/s, 100000 domains kept (informational)
|
|
```
|
|
|
|
`--seed=N` changes the generated domains and the query order; the default is
|
|
`0x5eed`, so two runs on the same machine are comparable. `--domains` caps at
|
|
4,000,000, and the `compile` suite additionally refuses more than 2,000,000 —
|
|
the compiler's own limit.
|
|
|
|
An argument the harness does not recognise stops it before any measuring:
|
|
|
|
```
|
|
error: unknown argument 'nosuch'
|
|
usage: zig build bench -Doptimize=ReleaseFast -- [filter|cache|compile|all] [--domains=N] [--iters=N] [--seed=N] [--assert]
|
|
```
|
|
|
|
## Read the output
|
|
|
|
- `p50`/`p95`/`p99`/`max` are per-operation microseconds, nearest-rank over
|
|
every iteration. What one operation means differs per suite: for `filter` it
|
|
is normalising a name plus evaluating it against the snapshot; for `cache` it
|
|
is building the key, getting the entry and stamping the response id.
|
|
- `blocked N/M` and `hits N/M` are sanity counters. The harness aborts if either
|
|
is zero — a suite that never hits its own path measures nothing.
|
|
- Two memory figures appear on purpose. `Snapshot.memoryBytes` and
|
|
`DnsCache.memoryBytes` are the in-repo accounting of those structures; `VmRSS`
|
|
is what the kernel holds resident for the whole process, allocator slack and
|
|
code included. The truth is between them, and the memory target is judged on
|
|
`VmRSS`.
|
|
- `target ...: PASS` / `FAIL` lines appear for the targets a suite covers. On a
|
|
plain run they are informational and the exit code stays 0.
|
|
|
|
## Fail the run when a target is missed
|
|
|
|
`--assert` turns those lines into an exit code — 1 when any target was
|
|
exceeded, 0 otherwise. This is meant for an acceptance run on hardware you
|
|
control, not for CI:
|
|
|
|
```sh
|
|
zig build bench -Doptimize=ReleaseFast -- --assert
|
|
```
|
|
|
|
The full-scale form is the one worth asserting on, because the memory target
|
|
only means something at a million domains. On this development host the reduced
|
|
form was used to check the flag itself:
|
|
|
|
```sh
|
|
zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000 --assert
|
|
```
|
|
|
|
```
|
|
filter 20000 0.14 0.26 0.27 2.42
|
|
blocked 6670/20000, Snapshot.memoryBytes 3.0 MiB, VmRSS 5.4 MiB
|
|
target p95 < 1ms: PASS
|
|
target VmRSS < 100 MiB: PASS
|
|
```
|
|
|
|
**Not verified on this host at full scale:** the plain
|
|
`zig build bench -Doptimize=ReleaseFast -- --assert` above was not run during
|
|
the writing of this page — the default run takes minutes. The reduced runs
|
|
shown were all executed as written. The full-scale numbers already recorded for
|
|
this host are in the [performance reference](../reference/performance.md).
|
|
|
|
## Measure sustained query rate
|
|
|
|
The bench harness cannot do this. Query rate is a property of the whole server
|
|
— sockets, upstreams, the query log writer — so it has to be driven from
|
|
outside, against the real binary, on the machine you care about.
|
|
|
|
Start nxdns with real blocklists configured, then drive it from another host on
|
|
the LAN with a DNS load generator such as `dnsperf`:
|
|
|
|
```sh
|
|
dnsperf -s 192.168.1.10 -p 53 -d queries.txt -c 20 -Q 200 -l 60
|
|
```
|
|
|
|
Read the client's own rate and the server's `/metrics` together: a load
|
|
generator that reports 200 qps while the server counts fewer has lost queries
|
|
somewhere, and that is the interesting number.
|
|
|
|
**Not verified on this host:** `dnsperf` is not installed here and the target
|
|
platform is a Raspberry Pi 5, not this development machine. The command above
|
|
is the shape of the measurement, not a transcript.
|
|
|
|
## Where to run it
|
|
|
|
The target platform is a Raspberry Pi 5. Numbers from a development x86_64 box
|
|
do not transfer — the Pi's Cortex-A76 is far slower — so a passing run here is
|
|
evidence the harness works and a baseline for spotting regressions on the
|
|
machine development happens on, and nothing more. Run `--assert` on the Pi,
|
|
where the numbers mean something.
|