nxdns/specs/memory-budget.md

Memory budget: pay for what the household uses

Measured on the deployed Pi (0.0.21, aarch64, 2.85M blocklist entries, 18 days up): VmRSS 238 MiB, VmHWM 358 MiB, 207 threads, almost all private anonymous memory. The docs promised "memory with ~1M blocked domains < 100 MiB" (docs/explanation/performance-and-testing.md, removed in the user-docs rewrite); the Pi never met it. Nothing leaks. Every MiB below has a measured owner, and each owner pays for a worst case the household never reaches, or pays twice.

Owner	Resident	How it was measured
Blocklist snapshot	71 MiB, of which ~32 MiB is ABP domains stored twice	Pi smaps; .list and .wild bodies of the ABP sources are byte-identical (md5)
Fixed buffers	~52 MiB: query-log writer queue 17.5, two TCP listeners 2 × 8.7, two UDP listeners 2 × 4.8, SSE hub 3.6, cache slots 3.3	Sizes from source; resident at once because safe builds fill new allocations with 0xaa
Idle worker threads	~41 MiB for ~200 parked workers	std.Io.Threaded never retires a worker; one burst of 64 in-flight UDP queries × 3 tasks each grows the pool for good (replica: 14 → 206 threads)
Heap slabs kept after peaks	~61 MiB free + ~8 MiB live	Decoded SmpAllocator free lists from two /proc/<pid>/mem dumps; owners from each slot's size header and a local tracing build
Peak during refresh	+120 MiB over steady state	Old snapshot + new snapshot + every compiled body read at once; replica reproduced 350 MiB

Heap owners, from the dumps and the trace:

• 8 KiB class, 25 MiB free: SQLite page-cache blocks (4,376 B requests in 8 KiB slots). Seven connections, no cache_size pragma, so each keeps the 2 MiB default, which costs ~3.7 MiB of slabs once filled.
• 128 B + 256 B classes, ~21 MiB free: std.Io.Threaded Group.Task blocks — one per UDP query (UdpServer.receiveLoop), two per upstream exchange (transport.raceUntilTagged), two per query-log wait (Logger.getWithin).
• 512 B class, 7.7 MiB free: DnsCache.put copies of cached responses (src/cache/dns_cache.zig:356).

Why freed small blocks stay resident: with libc linked, malloc is zig's lib/c/malloc.zig on std.heap.SmpAllocator, which never unmaps a 64 KiB slab. free pushes onto the freeing thread's slot and alloc searches one other slot before it maps a new slab (max_alloc_search = 1), so producer/consumer pairs map more slabs than their live peak needs. Upstream: ziglang/zig#23592. Fewer tasks and smaller peaks shrink this retained memory; nothing in this spec frees it after the fact.

Decisions

• Fix each owner at its root. Do not bypass the safe-build 0xaa fill: it only makes the oversizing visible.
• Do not change the allocator in this work. Replacing init.gpa does not move SQLite, which calls C malloc. Revisit only if the measurement gate still names small-class slabs as a top owner.
• No compiled-format change, no config-key migration, no new deadline mechanism. Each of those was considered and rejected in review: the format change makes the upgrade depend on the network, packed query-log rings cost more complexity than sizing the queue, and a socket-shutdown deadline cannot bound connect (std.Io.Threaded has no connect timeout) or the pool admission wait.
• Not SQLITE_CONFIG_PAGECACHE, not a database page-size migration.

Contract

0. Make the budget measurable first

The admission budget, the listener caps and the queue size are chosen from the Pi's real peaks, so those peaks become measurements before anything is resized:

• high-water of in-flight DNS queries: nxdns_dns_in_flight_peak, counted around Handler.handle, every listener together;
• high-water of active stream connections, per listener (today connections counts accepts over time, not concurrency): nxdns_tcp_server_connections_peak (the larger of the two address families), nxdns_dot_server_connections_peak, nxdns_doh_server_connections_peak;
• high-water of the std.Io.Threaded worker pool: the process thread count. Threaded spawns a worker only when every worker is busy and never retires one (Threaded.zig:2110), so the count is that high-water already and needs no code. It counts workers, not tasks: async work can run inline or wait in the run queue;
• high-water of the query-log writer queue's occupancy (not arrivals per interval: a full batch flushes early, and a disk gate or a slow write lets backlog build past one interval): nxdns_querylog_queue_peak. std.Io.Queue exposes no length, so a count beside it is raised before each insert and lowered after each removal; it can overstate the peak by the entries in transit, never understate it;
• worker call-stack use, read from /proc/<pid>/smaps: each worker stack is an anonymous mapping directly above its own guard page, all of one size, and its Rss is what the thread has touched and still holds resident. It is a record, not an input: a worker stack costs only the pages it touches, so stack_size is address space and lowering it saves no resident memory. stack_size keeps its default. The signal stack (std_options.signal_stack_size) is a separate allocation for signal handling and stack traces and keeps its stdlib default unless its own use is measured.

The five gauges are in /metrics beside the existing per-reason drop counters. One release carrying only these measurements runs on the Pi before the budget values are fixed.

First reading, static x86_64 musl build, one burst of 300 UDP queries: the in-flight peak reached 64 (the UDP slot pool), threads went from 14 to 205, and of 201 worker stacks 175 touched 32 kB or less while 26 touched 2 MiB each — the same 2 MiB with only forward-zone traffic and no DoT exchange, so the TLS client is not the owner. That 2 MiB was transparent huge pages, not a code path: the build host runs THP in always mode and backs a touched 2 MiB-aligned stack range with one huge page. The Pi runs madvise. Pi reading on 0.0.22 after 31 h (16 KiB pages): of 91 worker stacks, 6 hold 32 kB, 2 hold 80–96 kB, 73 hold 192 kB and 10 hold 208–256 kB, none with a huge page. The resident cost of the pool is therefore about 192 kB of stack per worker, which item 3 reduces by bounding the worker count.

1. Store each blocklist name once in memory

Within one source, as compiled entries, an exact x in .list together with x in .wild means exactly what ||x^ compiles to. The compiled files, their checksum and their counts stay as they are.

• The snapshot builder merges one source's sorted .list and .wild bodies into one name set with two flags per name: exact and subdomains. A name in both bodies is stored once. Sources are never merged with each other, so group eligibility and the reported source do not change.
• .allow stays a separate set, and level 7 (exceptions) still runs before both flag checks.
• Level 8 tests the full name for exact; level 9 tests proper parents for subdomains. Sources still walk in ascending index order. The reported result stays what it is today: when the x entry wins, the apex reports blocklist_domain with matched x and a proper subdomain reports blocklist_wildcard with matched parent x. An exception, an exact entry for the queried name or a nearer matching parent still wins first.

2. Lower the refresh and boot peak

• Boot keeps one synchronous initial load before the listeners start (src/app.zig:692). The startup pass uses that result for its refresh decisions and reloads only after a source actually refreshed, instead of rebuilding unconditionally (src/filter/manager.zig:1823).
• Snapshot gets a pure Builder: add one source's bodies, then free them before reading the next. The old snapshot is still freed only after the swap, so a reload holds at most old snapshot + new snapshot + one source's bodies. That is the reload bound, not the process peak: refresh preparation (download, compile) can run beside a reload and holds its own allocations, and the Pi peak measurement includes it.

3. Bound concurrent DNS work

• main takes std.process.Init.Minimal and owns its std.Io.Threaded, with a concurrent_limit derived from the chosen query and connection caps, the tasks each can nest, and the fixed services. The measured task high-water already counts every caller, so it is the cross-check, not a term in the sum.
• ConcurrencyUnavailable is counted at every boundary where it can surface. The spec work enumerates each io.concurrent and Group.concurrent call; today the logger turns the error into an early flush without counting it (src/storage/logger.zig:748) and the listener races fold it into a generic failure (src/server/listener.zig:102). Each gets its own counter or a reason on an existing one.
• Two kinds of bound: one cap on active DNS queries, shared by every listener (UDP and TCP, both families, DoT/DoH when enabled), and a slot cap per stream listener on accepted connections, each from the item-0 high-waters. A query takes a permit only while it does work; a keep-alive connection releases its permit between requests.
• UDP keeps today's counted drop when no permit is free (dropped_no_slot). The kernel buffer does not absorb overload invisibly.
• The upstream exchange keeps its whole-operation Select deadline and its distinction between peer result, budget expiry and cancellation. The thread high-water falls because the query budget is smaller, not because the race changes.
• UDP slots and TCP connection buffers are sized from these bounds, not 64 per listener.
• As built: 32 query permits for the whole process (query_budget.permits; the Pi's nxdns_dns_in_flight_peak was 27 over 37 h), each holding the datagram, the reply buffer and the handler scratch, about 2.4 MiB in all. 16 connections per DNS stream listener (listener.dns_max_connections), a policy rather than a measurement: the Pi's TCP peak was 0 and it runs neither DoT nor DoH. A stream query that finds no permit closes its connection and is counted as dropped_no_permit, on DoH too, where a 503 would need a reply buffer the query does not have. async_limit is .nothing, so every pool task is a counted concurrent one, and app.concurrent_limit enumerates them. ConcurrencyUnavailable is nxdns_concurrency_unavailable_total{site}, one label per boundary (platform/concurrency.zig).

4. Size the query-log buffers by use

• The writer queue default (logging.query_log_buffer_max) comes from the item-0 occupancy high-water, with headroom stated in the reference. Drop-oldest backpressure and its counter stay.
• The SSE hub allocates a subscriber's ring on subscribe and frees it on unsubscribe; max_subscribers stays a cap, not a reservation. subscribe becomes fallible: an allocation failure is distinct from "all slots occupied", is counted, answers with its own error rather than "live stream is full" (src/web/handlers/live.zig:75), and leaves the slot reusable. Overflow keeps today's contract: the hub ends that subscriber's stream so EventSource reconnects (src/web/sse.zig:8).

5. SQLite caches chosen per connection

• An explicit runtime-connection initialization step sets cache_size from a constant per role (query-log writer, retention, readers, config). Db.open stays free of pragmas, because migration callers control when pragmas apply (src/storage/db.zig:302). Every runtime handle reads its value back.

6. DNS cache bounded in bytes

• DnsCache keeps its entry count and adds a budget on live response bytes, tracked separately from memoryBytes (which also counts slots and index). put evicts by CLOCK until the copy fits. A response larger than the whole budget is not cached and is counted. A replacement keeps the old entry if allocating the new copy fails. A failed copy allocation is counted; today handler.zig:918 discards it with catch {} and DnsCache.Stats has no field for it. The answer already obtained is still sent. The budget, the live bytes and both counters appear in cache diagnostics.
• The budget is not a config key (owner, 2026-10-21: an operator has no basis for choosing it). It follows cache.size: 256 bytes per entry (about 2.4 MiB at the default 10000), about 2.5 times the Pi's measured average of 100 bytes per cached response, and never less than one 65535-byte message, so a small cache still caches any answer a large one would. The live bytes and the budget are reported as nxdns_cache_response_bytes and nxdns_cache_max_bytes.

Propagation

• Config: the changed default of logging.query_log_buffer_max. No new key (item 6).
• Diagnostics: the item-0 counters, ConcurrencyUnavailable, oversized cache responses, cache live bytes and budget.
• Docs: the user docs make no memory claim. The measured RSS, refresh peak and thread count at the Pi's blocklist size go into the measurement record below and the release's CHANGELOG entry.

Acceptance

• Matcher equivalence: a deterministic corpus (hosts, domains, ABP with ||x^, *.x alone, x alone, x with *.x, @@, an exact entry for a subdomain of a wildcard, nested wildcards *.x and *.y.x, overlapping sources) gives the same Decision for every probe name before and after item 1.
• Builder: a reload holds at most one source's bodies (tested through an allocator that records peak bytes).
• Boot: one snapshot build when no source refreshes; listeners never start without the initial load having run.
• Measurements: each of the five item-0 gauges rises under a scripted load and is visible in /metrics; the thread count and the worker stack residency are recorded from /proc beside them.
• Bounds: with the query budget full, a further UDP query is dropped and counted; a stream listener at its slot cap refuses and counts; a keep-alive connection between requests holds no query permit; ConcurrencyUnavailable is counted at each enumerated boundary, including the logger wait and the listener races.
• SSE: no ring is allocated without a subscriber; an allocation failure is counted, reported as its own error and leaves the slot reusable; overflow still ends the stream.
• SQLite: each runtime handle reports its configured cache_size; migrations open without it.
• DNS cache: puts beyond the byte budget evict; live bytes never exceed it; an oversized response is skipped and counted; a failed replacement keeps the old entry; a failed copy allocation is counted and the answer is still sent.
• Suites: zig build test and -Dintegration 0 failed; admin typecheck/vitest/prettier clean; contract samples regenerated.
• Measurement record, on the Pi with the same blocklists: steady VmRSS after 24 h, VmHWM across one scheduled refresh (including its concurrent download and compile), thread count, and a decoded heap dump per size class, each beside the baseline in the table above. The owner chose a record, not a pass threshold: each owner must be reduced as specified, and the numbers go into this spec beside the baseline and into the release's CHANGELOG entry. If small-class slabs remain a top owner, the allocator question reopens with that data.