# Memory budget: pay for what the household uses

Measured on the deployed Pi (0.0.21, aarch64, 2.85M blocklist entries, 18 days up): VmRSS 238 MiB, VmHWM 358 MiB, 207 threads, almost all private anonymous memory. The docs promised "memory with ~1M blocked domains < 100 MiB" (`docs/explanation/performance-and-testing.md`, removed in the user-docs rewrite); the Pi never met it. Nothing leaks. Every MiB below has a measured owner, and each owner pays for a worst case the household never reaches, or pays twice.

| Owner | Resident | How it was measured |
|---|---|---|
| Blocklist snapshot | 71 MiB, of which ~32 MiB is ABP domains stored twice | Pi smaps; `.list` and `.wild` bodies of the ABP sources are byte-identical (md5) |
| Fixed buffers | ~52 MiB: query-log writer queue 17.5, two TCP listeners 2 × 8.7, two UDP listeners 2 × 4.8, SSE hub 3.6, cache slots 3.3 | Sizes from source; resident at once because safe builds fill new allocations with 0xaa |
| Idle worker threads | ~41 MiB for ~200 parked workers | `std.Io.Threaded` never retires a worker; one burst of 64 in-flight UDP queries × 3 tasks each grows the pool for good (replica: 14 → 206 threads) |
| Heap slabs kept after peaks | ~61 MiB free + ~8 MiB live | Decoded `SmpAllocator` free lists from two `/proc/<pid>/mem` dumps; owners from each slot's size header and a local tracing build |
| Peak during refresh | +120 MiB over steady state | Old snapshot + new snapshot + every compiled body read at once; replica reproduced 350 MiB |

Heap owners, from the dumps and the trace:

- 8 KiB class, 25 MiB free: SQLite page-cache blocks (4,376 B requests in 8 KiB slots). Seven connections, no `cache_size` pragma, so each keeps the 2 MiB default, which costs ~3.7 MiB of slabs once filled.
- 128 B + 256 B classes, ~21 MiB free: `std.Io.Threaded` `Group.Task` blocks — one per UDP query (`UdpServer.receiveLoop`), two per upstream exchange (`transport.raceUntilTagged`), two per query-log wait (`Logger.getWithin`).
- 512 B class, 7.7 MiB free: `DnsCache.put` copies of cached responses (`src/cache/dns_cache.zig:356`).

Why freed small blocks stay resident: with libc linked, `malloc` is zig's `lib/c/malloc.zig` on `std.heap.SmpAllocator`, which never unmaps a 64 KiB slab. `free` pushes onto the freeing thread's slot and `alloc` searches one other slot before it maps a new slab (`max_alloc_search = 1`), so producer/consumer pairs map more slabs than their live peak needs. Upstream: ziglang/zig#23592. Fewer tasks and smaller peaks shrink this retained memory; nothing in this spec frees it after the fact.

## Decisions

- Fix each owner at its root. Do not bypass the safe-build 0xaa fill: it only makes the oversizing visible.
- Do not change the allocator in this work. Replacing `init.gpa` does not move SQLite, which calls C `malloc`. Revisit only if the measurement gate still names small-class slabs as a top owner.
- No compiled-format change, no config-key migration, no socket-shutdown deadline. Each of those was considered and rejected in review: the format change makes the upgrade depend on the network, packed query-log rings cost more complexity than sizing the queue, and a socket-shutdown deadline cannot bound `connect` (`std.Io.Threaded` has no connect timeout) or a wait for a permit. Permit waits take their deadline on a futex instead (item 3).
- Not `SQLITE_CONFIG_PAGECACHE`, not a database page-size migration.

## Contract

### 0. Make the budget measurable first

The admission budget, the listener caps and the queue size are chosen from the Pi's real peaks, so those peaks become measurements before anything is resized:

- high-water of in-flight DNS queries: `nxdns_dns_in_flight_peak`, counted around `Handler.handle`, every listener together;
- high-water of active stream connections, per listener (today `connections` counts accepts over time, not concurrency): `nxdns_tcp_server_connections_peak` (the larger of the two address families), `nxdns_dot_server_connections_peak`, `nxdns_doh_server_connections_peak`;
- high-water of the `std.Io.Threaded` worker pool: the process thread count. `Threaded` spawns a worker only when every worker is busy and never retires one (`Threaded.zig:2110`), so the count is that high-water already and needs no code. It counts workers, not tasks: `async` work can run inline or wait in the run queue;
- high-water of the query-log writer queue's occupancy (not arrivals per interval: a full batch flushes early, and a disk gate or a slow write lets backlog build past one interval): `nxdns_querylog_queue_peak`. `std.Io.Queue` exposes no length, so a count beside it is raised before each insert and lowered after each removal; it can overstate the peak by the entries in transit, never understate it;
- worker call-stack use, read from `/proc/<pid>/smaps`: each worker stack is an anonymous mapping directly above its own guard page, all of one size, and its `Rss` is what the thread has touched and still holds resident. It is a record, not an input: a worker stack costs only the pages it touches, so `stack_size` is address space and lowering it saves no resident memory. `stack_size` keeps its default. The signal stack (`std_options.signal_stack_size`) is a separate allocation for signal handling and stack traces and keeps its stdlib default unless its own use is measured.

The five gauges are in `/metrics` beside the existing per-reason drop counters. One release carrying only these measurements runs on the Pi before the budget values are fixed.

First reading, static x86_64 musl build, one burst of 300 UDP queries: the in-flight peak reached 64 (the UDP slot pool), threads went from 14 to 205, and of 201 worker stacks 175 touched 32 kB or less while 26 touched 2 MiB each — the same 2 MiB with only forward-zone traffic and no DoT exchange, so the TLS client is not the owner. That 2 MiB was transparent huge pages, not a code path: the build host runs THP in `always` mode and backs a touched 2 MiB-aligned stack range with one huge page. The Pi runs `madvise`. Pi reading on 0.0.22 after 31 h (16 KiB pages): of 91 worker stacks, 6 hold 32 kB, 2 hold 80–96 kB, 73 hold 192 kB and 10 hold 208–256 kB, none with a huge page. The resident cost of the pool is therefore about 192 kB of stack per worker, which item 3 reduces by bounding the worker count.

### 1. Store each blocklist name once in memory

Within one source, as compiled entries, an exact `x` in `.list` together with `x` in `.wild` means exactly what `||x^` compiles to. The compiled files, their checksum and their counts stay as they are.

- The snapshot builder merges one source's sorted `.list` and `.wild` bodies into one name set with two flags per name: `exact` and `subdomains`. A name in both bodies is stored once. Sources are never merged with each other, so group eligibility and the reported source do not change.
- `.allow` stays a separate set, and level 7 (exceptions) still runs before both flag checks.
- Level 8 tests the full name for `exact`; level 9 tests proper parents for `subdomains`. Sources still walk in ascending index order. The reported result stays what it is today: when the `x` entry wins, the apex reports `blocklist_domain` with matched `x` and a proper subdomain reports `blocklist_wildcard` with matched parent `x`. An exception, an exact entry for the queried name or a nearer matching parent still wins first.

### 2. Lower the refresh and boot peak

- Boot keeps one synchronous initial load before the listeners start (`src/app.zig:692`). The startup pass uses that result for its refresh decisions and reloads only after a source actually refreshed, instead of rebuilding unconditionally (`src/filter/manager.zig:1823`).
- `Snapshot` gets a pure `Builder`: add one source's bodies, then free them before reading the next. The old snapshot is still freed only after the swap, so a reload holds at most old snapshot + new snapshot + one source's bodies. That is the reload bound, not the process peak: refresh preparation (download, compile) can run beside a reload and holds its own allocations, and the Pi peak measurement includes it.

### 3. Bound concurrent DNS work

- Two kinds of bound: one budget of query permits, shared by every listener (UDP and TCP, both families, DoT/DoH when enabled), and a slot cap per stream listener on accepted connections. A query takes a permit only while it does work; a keep-alive connection releases its permit between requests.
- The work is bounded where it enters, not in the thread pool. `main` keeps `std.process.Init` and its `std.Io.Threaded` sets no `concurrent_limit`: a limit below the tasks the admitted work can nest turns a burst into `ConcurrencyUnavailable` failures inside admitted queries, and a limit sized to cover them bounds nothing the budget does not already bound. `ConcurrencyUnavailable` therefore means the system refused a thread or a task could not be allocated, and it is counted at every boundary where it can surface as `nxdns_concurrency_unavailable_total{site}` (`platform/concurrency.zig`).
- A query that finds every permit taken waits for one; it is neither dropped nor answered SERVFAIL. glibc's stub retries a SERVFAIL once at once and then fails the lookup, and a dropped UDP query costs the client its whole retry timeout, while a wait behind a permit that frees within the upstream deadline is invisible to it. Every held permit is bounded (the upstream total deadline, the stream write budget), so a wait always ends.
  - UDP: the receive loop receives into its own staging buffer, then waits for a permit holding that datagram, and receives nothing meanwhile. The backlog stays in the socket's kernel buffer, bounded by `SO_RCVBUF`; what overflows it is dropped by the kernel and exported from `/proc/net/udp{,6}` by socket inode as `nxdns_udp_server_kernel_drops_total`, with the queued bytes as `nxdns_udp_server_receive_queue_bytes` and a failed read as `nxdns_udp_server_kernel_read_failures_total`.
  - Streams: a whole query read off the connection waits at most the connection's idle budget. A wait that outlasts it ends the connection unanswered, because DNS-over-TCP has no framing for "no answer"; DoH closes too.
  - Every wait is counted: `nxdns_dns_query_permit_waits_total`, `nxdns_dns_query_permit_wait_seconds_total`, `nxdns_dns_query_permit_wait_timeouts_total` and the gauge `nxdns_dns_query_permits_waiting`.
  - A listener that is closing wakes the budget's waiters, so its `deinit` does not wait for a permit to free.
- A query waits on its own thread. The budget and the upstream pool's per-entry admission sleep on a futex word with the caller's deadline (`platform/wake_word.zig`) instead of racing a condition wait against a timer task, which cost two extra threads per waiter. A waiter that leaves without a permit hands its wake on if a permit is free.
- The upstream exchange keeps its whole-operation deadline race and its distinction between peer result, budget expiry and cancellation.
- Stream listeners keep `listener.race` for their idle reads, writes and handshakes: three threads per open connection, at most 16 per listener. Removing those two threads needs a non-blocking, socket-aware TLS and HTTP transport with its own deadline, reset and partial-I/O handling; std's timed file reads do not provide one (a blocking syscall follows the poll, and `ECONNRESET` on a timed write is an `errnoBug`). That transport costs more than the threads it saves at household scale.
- The process opts out of transparent huge pages at startup (`PR_SET_THP_DISABLE`, `platform/huge_pages.zig`), with one warning if the kernel refuses. With THP set to `always`, a worker stack that touches a few KiB holds a 2 MiB page and keeps it, because workers are never retired: a 64-query burst on such a host left about 40 of 200 workers at 2 MiB each.
- What this costs, stated plainly: a burst beyond the budget is answered later, not faster; a UDP burst that also outruns the socket buffer loses datagrams to the kernel, counted; a stream query can wait out its idle budget and get no answer; an upstream in backoff is still probed, so a dead upstream costs each query its deadline and ends in SERVFAIL from the handler, as before.
- As built: 64 query permits for the whole process (`query_budget.permits`), each holding the datagram, the reply buffer and the handler scratch, about 4.6 MiB in all. The Pi's `nxdns_dns_in_flight_peak` was 27 over a day of household traffic; 64 simultaneous uncached queries on two DoT upstreams (x86_64 musl ReleaseSafe) were all answered, the slowest in 0.2 s, and grew the pool from 14 to 205 threads, which is the all-active envelope of 64 × 3 plus the fixed services, about 40 MiB of stack at the measured 200 KiB high-water per worker. 16 connections per DNS stream listener (`listener.dns_max_connections`), a policy rather than a measurement: the Pi's TCP peak was 0 and it runs neither DoT nor DoH.

### 4. Size the query-log buffers by use

- The writer queue default (`logging.query_log_buffer_max`) comes from the item-0 occupancy high-water, with headroom stated in the reference. Drop-oldest backpressure and its counter stay.
- The SSE hub allocates a subscriber's ring on subscribe and frees it on unsubscribe; `max_subscribers` stays a cap, not a reservation. `subscribe` becomes fallible: an allocation failure is distinct from "all slots occupied", is counted, answers with its own error rather than "live stream is full" (`src/web/handlers/live.zig:75`), and leaves the slot reusable. Overflow keeps today's contract: the hub ends that subscriber's stream so `EventSource` reconnects (`src/web/sse.zig:8`).

### 5. SQLite caches chosen per connection

- An explicit runtime-connection initialization step sets `cache_size` from a constant per role (query-log writer, retention, readers, config). `Db.open` stays free of pragmas, because migration callers control when pragmas apply (`src/storage/db.zig:302`). Every runtime handle reads its value back.

### 6. DNS cache bounded in bytes

- `DnsCache` keeps its entry count and adds a budget on live response bytes, tracked separately from `memoryBytes` (which also counts slots and index). `put` evicts by CLOCK until the copy fits. A response larger than the whole budget is not cached and is counted. A replacement keeps the old entry if allocating the new copy fails. A failed copy allocation is counted; today `handler.zig:918` discards it with `catch {}` and `DnsCache.Stats` has no field for it. The answer already obtained is still sent. The budget, the live bytes and both counters appear in cache diagnostics.
- The budget is not a config key (owner, 2026-10-21: an operator has no basis for choosing it). It follows `cache.size`: 256 bytes per entry (about 2.4 MiB at the default 10000), about 2.5 times the Pi's measured average of 100 bytes per cached response, and never less than one 65535-byte message, so a small cache still caches any answer a large one would. The live bytes and the budget are reported as `nxdns_cache_response_bytes` and `nxdns_cache_max_bytes`.

## Propagation

- Config: the changed default of `logging.query_log_buffer_max`. No new key (item 6).
- Diagnostics: the item-0 counters, `ConcurrencyUnavailable`, oversized cache responses, cache live bytes and budget.
- Docs: the user docs make no memory claim. The measured RSS, refresh peak and thread count at the Pi's blocklist size go into the measurement record below and the release's CHANGELOG entry.

## Acceptance

- [ ] Matcher equivalence: a deterministic corpus (hosts, domains, ABP with `||x^`, `*.x` alone, `x` alone, `x` with `*.x`, `@@`, an exact entry for a subdomain of a wildcard, nested wildcards `*.x` and `*.y.x`, overlapping sources) gives the same `Decision` for every probe name before and after item 1.
- [ ] Builder: a reload holds at most one source's bodies (tested through an allocator that records peak bytes).
- [ ] Boot: one snapshot build when no source refreshes; listeners never start without the initial load having run.
- [ ] Measurements: each of the five item-0 gauges rises under a scripted load and is visible in `/metrics`; the thread count and the worker stack residency are recorded from `/proc` beside them.
- [ ] Bounds: with the query budget full, a further UDP query waits and is answered once a permit frees, and the wait is counted; a stream query waits at most its idle budget, and a timeout ends the connection and is counted; closing a listener ends its waits without a permit freeing; a waiter canceled or timed out hands its wake on; a stream listener at its slot cap refuses and counts; a keep-alive connection between requests holds no query permit; the UDP sockets' kernel queue and drops are exported; `ConcurrencyUnavailable` is counted at each enumerated boundary, including the logger wait and the listener races.
- [ ] SSE: no ring is allocated without a subscriber; an allocation failure is counted, reported as its own error and leaves the slot reusable; overflow still ends the stream.
- [ ] SQLite: each runtime handle reports its configured `cache_size`; migrations open without it.
- [ ] DNS cache: puts beyond the byte budget evict; live bytes never exceed it; an oversized response is skipped and counted; a failed replacement keeps the old entry; a failed copy allocation is counted and the answer is still sent.
- [ ] Suites: `zig build test` and `-Dintegration` 0 failed; admin typecheck/vitest/prettier clean; contract samples regenerated.
- [ ] Measurement record, on the Pi with the same blocklists: steady VmRSS after 24 h, VmHWM across one scheduled refresh (including its concurrent download and compile), thread count, and a decoded heap dump per size class, each beside the baseline in the table above. The owner chose a record, not a pass threshold: each owner must be reduced as specified, and the numbers go into this spec beside the baseline and into the release's CHANGELOG entry. If small-class slabs remain a top owner, the allocator question reopens with that data.