# Memory budget: pay for what the household uses

Measured on the deployed Pi (0.0.21, aarch64, 2.85M blocklist entries, 18 days up): VmRSS 238 MiB, VmHWM 358 MiB, 207 threads, almost all private anonymous memory. The docs promised "memory with ~1M blocked domains < 100 MiB" (`docs/explanation/performance-and-testing.md`, removed in the user-docs rewrite); the Pi never met it. Nothing leaks. Every MiB below has a measured owner, and each owner pays for a worst case the household never reaches, or pays twice.

| Owner | Resident | How it was measured |
|---|---|---|
| Blocklist snapshot | 71 MiB, of which ~32 MiB is ABP domains stored twice | Pi smaps; `.list` and `.wild` bodies of the ABP sources are byte-identical (md5) |
| Fixed buffers | ~52 MiB: query-log writer queue 17.5, two TCP listeners 2 × 8.7, two UDP listeners 2 × 4.8, SSE hub 3.6, cache slots 3.3 | Sizes from source; resident at once because safe builds fill new allocations with 0xaa |
| Idle worker threads | ~41 MiB for ~200 parked workers | `std.Io.Threaded` never retires a worker; one burst of 64 in-flight UDP queries × 3 tasks each grows the pool for good (replica: 14 → 206 threads) |
| Heap slabs kept after peaks | ~61 MiB free + ~8 MiB live | Decoded `SmpAllocator` free lists from two `/proc/<pid>/mem` dumps; owners from each slot's size header and a local tracing build |
| Peak during refresh | +120 MiB over steady state | Old snapshot + new snapshot + every compiled body read at once; replica reproduced 350 MiB |

Heap owners, from the dumps and the trace:

- 8 KiB class, 25 MiB free: SQLite page-cache blocks (4,376 B requests in 8 KiB slots). Seven connections, no `cache_size` pragma, so each keeps the 2 MiB default, which costs ~3.7 MiB of slabs once filled.
- 128 B + 256 B classes, ~21 MiB free: `std.Io.Threaded` `Group.Task` blocks — one per UDP query (`UdpServer.receiveLoop`), two per upstream exchange (`transport.raceUntilTagged`), two per query-log wait (`Logger.getWithin`).
- 512 B class, 7.7 MiB free: `DnsCache.put` copies of cached responses (`src/cache/dns_cache.zig:356`).

Why freed small blocks stay resident: with libc linked, `malloc` is zig's `lib/c/malloc.zig` on `std.heap.SmpAllocator`, which never unmaps a 64 KiB slab. `free` pushes onto the freeing thread's slot and `alloc` searches one other slot before it maps a new slab (`max_alloc_search = 1`), so producer/consumer pairs map more slabs than their live peak needs. Upstream: ziglang/zig#23592. Fewer tasks and smaller peaks shrink this retained memory; nothing in this spec frees it after the fact.

## Decisions

- Fix each owner at its root. Do not bypass the safe-build 0xaa fill: it only makes the oversizing visible.
- Do not change the allocator in this work. Replacing `init.gpa` does not move SQLite, which calls C `malloc`. Revisit only if the measurement gate still names small-class slabs as a top owner.
- No compiled-format change, no config-key migration, no new deadline mechanism. Each of those was considered and rejected in review: the format change makes the upgrade depend on the network, packed query-log rings cost more complexity than sizing the queue, and a socket-shutdown deadline cannot bound `connect` (`std.Io.Threaded` has no connect timeout) or the pool admission wait.
- Not `SQLITE_CONFIG_PAGECACHE`, not a database page-size migration.

## Contract

### 0. Make the budget measurable first

The admission budget, the listener caps and the queue size are chosen from the Pi's real peaks, so those peaks become measurements before anything is resized:

- high-water of in-flight DNS queries: `nxdns_dns_in_flight_peak`, counted around `Handler.handle`, every listener together;
- high-water of active stream connections, per listener (today `connections` counts accepts over time, not concurrency): `nxdns_tcp_server_connections_peak` (the larger of the two address families), `nxdns_dot_server_connections_peak`, `nxdns_doh_server_connections_peak`;
- high-water of the `std.Io.Threaded` worker pool: the process thread count. `Threaded` spawns a worker only when every worker is busy and never retires one (`Threaded.zig:2110`), so the count is that high-water already and needs no code. It counts workers, not tasks: `async` work can run inline or wait in the run queue;
- high-water of the query-log writer queue's occupancy (not arrivals per interval: a full batch flushes early, and a disk gate or a slow write lets backlog build past one interval): `nxdns_querylog_queue_peak`. `std.Io.Queue` exposes no length, so a count beside it is raised before each insert and lowered after each removal; it can overstate the peak by the entries in transit, never understate it;
- worker call-stack use, read from `/proc/<pid>/smaps`: each worker stack is an anonymous mapping directly above its own guard page, all of one size, and its `Rss` is what the thread has touched and still holds resident. It is a record, not an input: a worker stack costs only the pages it touches, so `stack_size` is address space and lowering it saves no resident memory. `stack_size` keeps its default. The signal stack (`std_options.signal_stack_size`) is a separate allocation for signal handling and stack traces and keeps its stdlib default unless its own use is measured.

The five gauges are in `/metrics` beside the existing per-reason drop counters. One release carrying only these measurements runs on the Pi before the budget values are fixed.

First reading, static x86_64 musl build, one burst of 300 UDP queries: the in-flight peak reached 64 (the UDP slot pool), threads went from 14 to 205, and of 201 worker stacks 175 touched 32 kB or less while 26 touched 2 MiB each — the same 2 MiB with only forward-zone traffic and no DoT exchange, so the TLS client is not the owner. That 2 MiB was transparent huge pages, not a code path: the build host runs THP in `always` mode and backs a touched 2 MiB-aligned stack range with one huge page. The Pi runs `madvise`. Pi reading on 0.0.22 after 31 h (16 KiB pages): of 91 worker stacks, 6 hold 32 kB, 2 hold 80–96 kB, 73 hold 192 kB and 10 hold 208–256 kB, none with a huge page. The resident cost of the pool is therefore about 192 kB of stack per worker, which item 3 reduces by bounding the worker count.

### 1. Store each blocklist name once in memory

Within one source, as compiled entries, an exact `x` in `.list` together with `x` in `.wild` means exactly what `||x^` compiles to. The compiled files, their checksum and their counts stay as they are.

- The snapshot builder merges one source's sorted `.list` and `.wild` bodies into one name set with two flags per name: `exact` and `subdomains`. A name in both bodies is stored once. Sources are never merged with each other, so group eligibility and the reported source do not change.
- `.allow` stays a separate set, and level 7 (exceptions) still runs before both flag checks.
- Level 8 tests the full name for `exact`; level 9 tests proper parents for `subdomains`. Sources still walk in ascending index order. The reported result stays what it is today: when the `x` entry wins, the apex reports `blocklist_domain` with matched `x` and a proper subdomain reports `blocklist_wildcard` with matched parent `x`. An exception, an exact entry for the queried name or a nearer matching parent still wins first.

### 2. Lower the refresh and boot peak

- Boot keeps one synchronous initial load before the listeners start (`src/app.zig:692`). The startup pass uses that result for its refresh decisions and reloads only after a source actually refreshed, instead of rebuilding unconditionally (`src/filter/manager.zig:1823`).
- `Snapshot` gets a pure `Builder`: add one source's bodies, then free them before reading the next. The old snapshot is still freed only after the swap, so a reload holds at most old snapshot + new snapshot + one source's bodies. That is the reload bound, not the process peak: refresh preparation (download, compile) can run beside a reload and holds its own allocations, and the Pi peak measurement includes it.

### 3. Bound concurrent DNS work

- `main` takes `std.process.Init.Minimal` and owns its `std.Io.Threaded`, with a `concurrent_limit` derived from the chosen query and connection caps, the tasks each can nest, and the fixed services. The measured task high-water already counts every caller, so it is the cross-check, not a term in the sum.
- `ConcurrencyUnavailable` is counted at every boundary where it can surface. The spec work enumerates each `io.concurrent` and `Group.concurrent` call; today the logger turns the error into an early flush without counting it (`src/storage/logger.zig:748`) and the listener races fold it into a generic failure (`src/server/listener.zig:102`). Each gets its own counter or a reason on an existing one.
- Two kinds of bound: one cap on active DNS queries, shared by every listener (UDP and TCP, both families, DoT/DoH when enabled), and a slot cap per stream listener on accepted connections, each from the item-0 high-waters. A query takes a permit only while it does work; a keep-alive connection releases its permit between requests.
- UDP keeps today's counted drop when no permit is free (`dropped_no_slot`). The kernel buffer does not absorb overload invisibly.
- The upstream exchange keeps its whole-operation `Select` deadline and its distinction between peer result, budget expiry and cancellation. The thread high-water falls because the query budget is smaller, not because the race changes.
- UDP slots and TCP connection buffers are sized from these bounds, not 64 per listener.
- As built: 32 query permits for the whole process (`query_budget.permits`; the Pi's `nxdns_dns_in_flight_peak` was 27 over 37 h), each holding the datagram, the reply buffer and the handler scratch, about 2.4 MiB in all. 16 connections per DNS stream listener (`listener.dns_max_connections`), a policy rather than a measurement: the Pi's TCP peak was 0 and it runs neither DoT nor DoH. A stream query that finds no permit closes its connection and is counted as `dropped_no_permit`, on DoH too, where a 503 would need a reply buffer the query does not have. `async_limit` is `.nothing`, so every pool task is a counted `concurrent` one, and `app.concurrent_limit` enumerates them. `ConcurrencyUnavailable` is `nxdns_concurrency_unavailable_total{site}`, one label per boundary (`platform/concurrency.zig`).

### 4. Size the query-log buffers by use

- The writer queue default (`logging.query_log_buffer_max`) comes from the item-0 occupancy high-water, with headroom stated in the reference. Drop-oldest backpressure and its counter stay.
- The SSE hub allocates a subscriber's ring on subscribe and frees it on unsubscribe; `max_subscribers` stays a cap, not a reservation. `subscribe` becomes fallible: an allocation failure is distinct from "all slots occupied", is counted, answers with its own error rather than "live stream is full" (`src/web/handlers/live.zig:75`), and leaves the slot reusable. Overflow keeps today's contract: the hub ends that subscriber's stream so `EventSource` reconnects (`src/web/sse.zig:8`).

### 5. SQLite caches chosen per connection

- An explicit runtime-connection initialization step sets `cache_size` from a constant per role (query-log writer, retention, readers, config). `Db.open` stays free of pragmas, because migration callers control when pragmas apply (`src/storage/db.zig:302`). Every runtime handle reads its value back.

### 6. DNS cache bounded in bytes

- `DnsCache` keeps its entry count and adds a budget on live response bytes, tracked separately from `memoryBytes` (which also counts slots and index). `put` evicts by CLOCK until the copy fits. A response larger than the whole budget is not cached and is counted. A replacement keeps the old entry if allocating the new copy fails. A failed copy allocation is counted; today `handler.zig:918` discards it with `catch {}` and `DnsCache.Stats` has no field for it. The answer already obtained is still sent. The budget, the live bytes and both counters appear in cache diagnostics.
- The budget is not a config key (owner, 2026-10-21: an operator has no basis for choosing it). It follows `cache.size`: 256 bytes per entry (about 2.4 MiB at the default 10000), about 2.5 times the Pi's measured average of 100 bytes per cached response, and never less than one 65535-byte message, so a small cache still caches any answer a large one would. The live bytes and the budget are reported as `nxdns_cache_response_bytes` and `nxdns_cache_max_bytes`.

## Propagation

- Config: the changed default of `logging.query_log_buffer_max`. No new key (item 6).
- Diagnostics: the item-0 counters, `ConcurrencyUnavailable`, oversized cache responses, cache live bytes and budget.
- Docs: the user docs make no memory claim. The measured RSS, refresh peak and thread count at the Pi's blocklist size go into the measurement record below and the release's CHANGELOG entry.

## Acceptance

- [ ] Matcher equivalence: a deterministic corpus (hosts, domains, ABP with `||x^`, `*.x` alone, `x` alone, `x` with `*.x`, `@@`, an exact entry for a subdomain of a wildcard, nested wildcards `*.x` and `*.y.x`, overlapping sources) gives the same `Decision` for every probe name before and after item 1.
- [ ] Builder: a reload holds at most one source's bodies (tested through an allocator that records peak bytes).
- [ ] Boot: one snapshot build when no source refreshes; listeners never start without the initial load having run.
- [ ] Measurements: each of the five item-0 gauges rises under a scripted load and is visible in `/metrics`; the thread count and the worker stack residency are recorded from `/proc` beside them.
- [ ] Bounds: with the query budget full, a further UDP query is dropped and counted; a stream listener at its slot cap refuses and counts; a keep-alive connection between requests holds no query permit; `ConcurrencyUnavailable` is counted at each enumerated boundary, including the logger wait and the listener races.
- [ ] SSE: no ring is allocated without a subscriber; an allocation failure is counted, reported as its own error and leaves the slot reusable; overflow still ends the stream.
- [ ] SQLite: each runtime handle reports its configured `cache_size`; migrations open without it.
- [ ] DNS cache: puts beyond the byte budget evict; live bytes never exceed it; an oversized response is skipped and counted; a failed replacement keeps the old entry; a failed copy allocation is counted and the answer is still sent.
- [ ] Suites: `zig build test` and `-Dintegration` 0 failed; admin typecheck/vitest/prettier clean; contract samples regenerated.
- [ ] Measurement record, on the Pi with the same blocklists: steady VmRSS after 24 h, VmHWM across one scheduled refresh (including its concurrent download and compile), thread count, and a decoded heap dump per size class, each beside the baseline in the table above. The owner chose a record, not a pass threshold: each owner must be reduced as specified, and the numbers go into this spec beside the baseline and into the release's CHANGELOG entry. If small-class slabs remain a top owner, the allocator question reopens with that data.