Files
nxdns/specs/milestone-34.md
T
mokhtar ce143d1d87
Gates / frontend (push) Successful in 1m43s
Gates / test (push) Successful in 2m14s
Gates / test-aarch64 (push) Successful in 8m3s
Gates / package (push) Successful in 5m42s
Gates / container (push) Successful in 54s
CI / gates (push) Successful in 50m24s
db-mode config changes apply live in-process
settings and upstream writes now follow a prepare, commit, publish, retire
contract: candidates are built and validated before the database transaction,
published as infallible pointer swaps, and old generations retire after their
readers drain. per-query policy values snapshot once per query; upstream pool,
cache, rate limiter, sessions, api limiter, log sink, blocklist scheduler and
the query-log queue each gained one named live operation. restart_required
shrinks from every scalar key to the bind keys and web.enabled; the admin ui
drops its restart notices for everything else. file mode is unchanged.
2026-08-24 00:04:28 +02:00

39 KiB
Raw Permalink Blame History

Milestone 34: hot-apply — tiers A and B

DB-mode config changes apply live, in-process, for every key that does not create a socket. The comptime "every key is restart-required" blanket is replaced by an explicit per-key apply table. Design: rev 3 of the hot-reload analysis; spec hardened through two Codex review rounds (thread 01a02efc). Listener rebind and web.enabled lifecycle are milestone 35.

The write contract (every session honors it)

Prepare → commit → publish → retire:

  1. Prepare: under the serialized config-mutation path, build and validate everything the change needs, from the FINAL MERGED config — ONE candidate per affected owner, never one per key. Prepared resources are heap-stable and owned (a candidate outlives the request arena). Nothing is published. Any failure: clean up every prepared resource, error to the client, NO db write.
  2. Commit: the DB transaction, only after every prepare succeeded. Commit failure: clean up all prepared resources.
  3. Publish: infallible, I/O-free operations — handle swaps, pointer swaps, stores under a lock. Closing files, joining tasks, freeing memory belong to retire, not publish.
  4. Retire: old generations/handles are drained, closed, and freed after their readers release.

Signatures: every operation that touches a std.Io primitive (Mutex, RwLock, Condition, concurrent, sleep) takes io: std.Io. The spec's named signatures include it; a builder adds it wherever else the primitive demands it.

Sessions

Six sessions, strictly sequential (S1 → … → S6); each session owns the tree while it runs and ends with zig build test green (S6 adds the admin suite).

Honesty rule: restart_pending/restart_required keep firing exactly as today until S5 swaps the mechanism atomically. No earlier session removes a restart signal.


Session S1: per-query policy snapshots (tier A)

S1.1 DNS policy snapshot

src/server/handler.zig: the per-query reads — blocking response + ttl, ecs_mode, forward_read_timeout, negative_ttl_max — move into one Policy struct behind std.Io.RwLock (the filter discipline, manager.zig:438/856). Each query copies the Policy ONCE at query start (shared lock only for the copy); ecs_mode at :782/:829/:835 reads the copy. Publish: pub fn setPolicy(h: *Handler, io: std.Io, p: Policy).

S1.2 Web trusted proxies

src/web/server.zig ~:531: trusted_proxies becomes an owned immutable generation in WebState, pointer-replaced under an RwLock (io-threaded), old generation retired after readers release. Prepare copies out of the request arena.

S1.3 Logger config split

src/storage/logger.zig: hide_domains + hide_client_ips (applied on the PRODUCER path :449) are ONE privacy policy: both pack into a single atomic (one u8), stored together by setPrivacy and loaded ONCE per entry — a producer can never observe a mixed policy that redacts the domain but exposes the client, or vice versa. query_log_flush_interval_s (:609) is an independent .monotonic atomic with setFlushInterval.

S1.4 Disk thresholds

src/storage/disk_monitor.zig: min_free_mb/warn_free_mb are one invariant pair (warn ≥ min): both u32s packed into ONE atomic u64; readers unpack a single load.

S1 implementation notes (post-build)

  • Context's provenance method policy(...) renamed notePolicy (collision with the new per-query field).
  • Pure-atomic setters (setPrivacy, setFlushInterval, setThresholds) take no io — they touch no Io primitive.
  • LiveProxies.install RETURNS the retired generation; the caller frees it in retire.
  • The trusted-proxies read hold ends at the client-address verdict, BEFORE router dispatch — a hold spanning dispatch would let a settings PUT deadlock on the shared lock its own request holds.
  • The flush-interval test covers short→long only; long→short is unobservable without waking a writer already parked on its old deadline (S5's PUT test inherits this bound).
  • S1 setters have no production caller until S5 wires the PUT flow — test-only until then, by design.

S1.5 Acceptance criteria

  • One query is internally consistent across a concurrent setPolicy (both ecs/blocking reads agree).
  • trusted_proxies replaced under concurrent request-path reads; testing allocator clean.
  • Privacy flip with FORCED producer interleaving: pre-flip entries unredacted, post-flip redacted.
  • Threshold reader vs concurrent pair-stores: every observed pair satisfies warn ≥ min.
  • Flush-interval change observed on the writer's next cycle via the existing timing seam.
  • zig build test green.

Session S2: upstream extraction + generation owner + lock hygiene

S2.1 Extract the upstream composition out of app.zig

Upstreams, build, deinit, and what they need from ConfigLoad (all private in app.zig ~:1158) move to src/upstream/owner.zig; app.zig imports it, never the reverse. build returns the generation plus a BuildReport. ConfigLoad.note is NOT just log output — it writes operational-event rows and retains canonical keys for finalize (app.zig:367). Contract: at BOOT, app.zig replays the successful BuildReport through the exact ConfigLoad.note path before finalize, so diagnostics are unchanged; at RUNTIME, candidate preparation is side-effect-free (no event rows) until commit, and the report is RECONCILED in RETIRE (after the pointer publish — event rows are SQLite I/O, banned from publish): retire reconciles SCOPED, not via ConfigLoad.finalize (finalize calls Store.resolveExcept, events.zig:402, which resolves EVERY active configuration.load event outside its kept set — at runtime that would falsely resolve unrelated boot warnings). Each generation OWNS its report keys (stored in the generation), and the retire payload is STABLE: prepare copies the LIVE generation's report keys (under the owner mutex) into the PUT-owned apply state, so reconciliation never depends on the old generation's lifetime and NEVER runs from a query's release path — it runs on the PUT task in retire. Retire: emit the new report's events, then individually resolve exactly copied_previous_keys new_report_keys. S5 tests through the real PUT path: introducing a malformed upstream raises the warning; fixing it resolves it; an UNRELATED active configuration warning survives the whole introduce/fix cycle.

S2.2 UpstreamOwner

Discipline: CertStore's (cert_store.zig:199/:237) — a mutex held briefly for refcounted borrows.

  • pub fn acquire(o: *Owner, io: std.Io) *Generation — lock, ++refs, return.
  • pub fn release(o: *Owner, io: std.Io, g: *Generation) — lock, --refs; a release that drops a RETIRED generation to zero deinits it (Upstreams.deinit needs io).
  • pub fn replace(o: *Owner, io: std.Io, prepared: *Generation) ?*Generation — lock, swap the live pointer, mark old retired; if the old generation's refs are ALREADY zero, return it for the caller to retire immediately (the CertStore refs==0 pattern) — otherwise null and the last release retires it.
  • pub fn deinit(o: *Owner, io: std.Io) — shutdown teardown of the live generation, called by serve() after listeners and metrics readers have stopped.
  • Prepared generations are heap-allocated and OWN every configuration string (URL text — transport.Endpoint borrows it, transport.zig:51 — plus tls_name, host, path): each generation carries its own arena covering every borrowed row string until retirement. Candidates built from request-arena rows copy into that arena at prepare.
  • Timeouts are baked into the generation; a timeout change is a replace.
  • Query path: Handler stores *Owner (replacing the boot-time type-erased client, handler.zig:91). Per exchange: acquire → generation's transport.Client → exchange → copy the resolver identity from selected (borrowed from the endpoint, transport.zig:350) into the PER-QUERY Context.upstream_buf (handler.zig:453 — never a Handler-owned buffer; Handler is shared across concurrent queries) → release.
  • Metrics/health: WebState stores *Owner (replacing *Pool, web/server.zig:144); a scrape acquires for its duration.

S2.3 Cache and rate-limiter lock hygiene

  • handler.zig:799 and app.zig:1124: cache pointer loads move inside cache_mutex; then replaceCache swaps under it (entries lost — accepted).
  • handler.zig:189: same for the rate limiter; replaceRateLimiter under its lock (windows reset — accepted).

S2 implementation notes (post-build)

  • The retire-time scoped reconciler is DEFERRED TO S5 (S2 has no runtime replace caller; uncalled code fails the values). S2 delivers its precondition: generations own copyable report keys. S5 must implement the reconciler + its tests (including unrelated-warning-survives).
  • Generation gains a test-only borrowing/borrowingPool payload so ~90 existing test sites keep their fake clients without heap generations; production always takes the built path.
  • One skipped-upstream log string reworded to unify with the event detail; the event detail (what the criterion asserts) is byte-identical.
  • Integration-gated call sites were mechanically touched (comptime-dead without -Dintegration; would have broken that build).

S2.4 Acceptance criteria

  • Concurrent exchanges vs replace: in-flight completes on G1; G1 deinits only after last release (or immediately when refs==0 at replace); new exchanges on G2; allocator clean.
  • A replace while NO reader holds G1 retires G1 via the replace return path (refs==0 branch covered).
  • Generation string ownership: build a candidate from a transient arena, free the arena, exchange still reads valid url/tls_name (allocator-poisoning test).
  • Metrics scrape concurrent with replace: clean.
  • Cache and rate-limiter swaps under concurrent use: clean.
  • Boot diagnostics unchanged: the events ConfigLoad.note wrote before the extraction are written identically (assert on the events store, not stdout).
  • Restart signals untouched this session.
  • zig build test green.

Session S3: in-place reconfigure, retention, certs, sink, scheduler

S3.1 Sessions TTL

Slots gain issued_at beside expires_at (auth.zig:311). setTtl(io, ttl) recomputes each live slot: expires_at = issued_at + ttl under the existing mutex. Semantics are SERVER-SIDE: shortening takes effect for every session; lengthening is bounded by the browser cookie's original Max-Age (the cookie is not refreshed — sessions do not become sliding; state this in a comment). Login cookie Max-Age (handlers/auth.zig:127) reads the LIVE ttl.

S3.2 ApiLimiter

ALL config reads move under the existing mutex (check reads localhost_exempt pre-lock, api_limiter.zig:135). setLimits(io, now, limits): refill each bucket THROUGH now at the old rate, clamp tokens to the new capacity, then install the new rate — no retroactive refill at the new rate, and never move a bucket's clock backward (a bucket already newer than now is clamped only). SSE per-IP counts untouched.

S3.3 Retention days

Two consumers (retention.zig:103, clients.zig:228): one shared .monotonic atomic u32 owned by app-level state, read per pass by both; setRetentionDays stores.

S3.4 Certificate paths

doh_server.cert_path/key_path, dot_server.cert_path/key_path on an ENABLED endpoint: CertStore's paths (borrowed boot slices, reread by reload at cert_store.zig:111, read outside mutex by reload/pollOnce at :226/:268) become owned, replaceable strings. The whole apply is serialized under the store's existing reload_mutex — the SAME mutex reload holds across load+publication: prepare (under reload_mutex) loads the candidate cert+key from the new paths; publish (still under reload_mutex, taking the generation mutex only for the swap — lock order: reload_mutex outer, mutex inner, matching reload today) swaps paths and generation together. pollOnce currently reads the paths BEFORE taking reload_mutex (cert_store.zig:268): it now takes reload_mutex around its path reads and calls a non-locking reloadLocked helper (reload becomes reload_mutex-lock + reloadLocked) so acquisition is never recursive. With that, no concurrent path borrow can outlive an apply. Connections pinning the old cert finish on it. POST /api/certs/reload rereads the owned paths. On a DISABLED endpoint no CertStore exists (handlers/certs.zig:31 — doh_certs/dot_certs are null): the change is DB-ONLY, and the key's table entry says so; the paths are validated when milestone 35 implements enable. This is stated in the table note and the API docs.

S3.5 Log sink

Three disjoint cases, decided from the merged config — no reuse marker, no lock spanning prepare and publish:

  • File target changed (the merged config has output=file AND the path differs from the current file target, or output switches TO file): prepare opens the NEW target (fallible, closing the TOCTOU in logging.zig:228's close-then-reopen) into an owned PreparedSink that carries the COMPLETE target state: the open file, its MEASURED length as the new file_pos, and rotate_pending = false (the handle couples to both fields, logging.zig:196 — inheriting the old position corrupts writes; inheriting a pending rotation rotates the new target spuriously). Publish installs {file, file_pos, rotate_pending} + config atomically under the sink's existing lock; the DETACHED old handle closes in retire. Races can only touch the OLD state, replaced wholesale.
  • File target removed (output switches FROM file to stderr/syslog): nothing to prepare; publish installs {file = null, file_pos = 0, rotate_pending = false} + config atomically; the detached file closes in retire.
  • File target unchanged (every remaining case: output stays stderr/syslog, or output=file with the same path — level, limits, or a file_path change while output is non-file): publish updates ONLY the config fields under the sink lock and NEVER touches the handle or its position/rotation state, which stay owned by the existing rotation/recovery machinery. A same-path config apply does not repair a broken handle (the existing per-write recovery does). No prepare-time handle inspection exists, so there is nothing to race. (A file_path change while output is stderr still updates the config so a later output=file switch — its own apply — opens the right target.) The disk monitor's measured directory derives from the final merged output + file_path on EVERY log_sink apply: output file → the file's directory (installed even if only output changed); output stderr/syslog → cleared (monitor stops measuring a log dir). Monitor.setLogDir(io, owned_path_or_null); sample borrows log_dir_path across directory I/O (disk_monitor.zig:112), so the path is a pinned generation — sample acquires (refcount or lock held for the borrow), setLogDir swaps, old path freed after the borrow releases. Owned path prepared pre-commit.

S3.6 Wakeable blocklist scheduler

manager.zig:1493 restructured into a wakeable loop parked on a condition:

  • The startup refresh pass runs exactly as today, including when disabled.
  • setSchedule(io, enabled, interval_hours) stores under the scheduler's mutex with a version counter (no lost wakes between check and park) and signals.
  • Anchor rule: the anchor is the completion time of the last refresh pass that RAN — success or failure both advance it; a disk-gate skip ALSO advances it (today's semantics: the scheduled slot is skipped, not retried early). Next refresh = anchor + interval; if that is in the past at set/enable time, refresh immediately.
  • Disabled parks; the task exits only on shutdown, exactly as today.
  • Production wake primitive: std.Io.Event.waitTimeout (Condition has no timed wait in 0.16). The Event is STICKY after set — the reset sequence prevents both spinning and lost wakes: under the scheduler's mutex the loop reads the version, RESETS the event, recomputes its deadline, releases the mutex, then waits; setSchedule (under the same mutex) bumps the version, then sets the event. A set that lands between the loop's reset and its wait completes the wait immediately; the version recheck decides whether anything changed.
  • Test seam: an injectable clock/step seam — validated intervals are ≥ 1 h; acceptance tests must not sleep real time.

S3 implementation notes (post-build)

  • The disabled-endpoint cert criterion (DB-only, no store call, no-restart response) is PURELY a PUT concern — deferred to S5's obligations beside the S2 reconciler.
  • model.retentionSeconds deleted (dead once both consumers read the shared cell); RetentionDays.seconds() is the single conversion point.
  • A disabled scheduler PARKS; the seam models shutdown (shutdown_at_first_park → error.Canceled) — the loop's only exit is shutdown.
  • publishPathChange reads no clock (loaded_at captured at prepare).
  • Monitor.deinit(io) added to free an installed log-dir generation (no-op at boot — arena-borrowed).
  • setLimits takes the full limiter Config (the three fields ARE the config; a twin struct would be invented generality).
  • Sink race criteria proven as the two reachable interleavings (publish shares the stderr lock with rotation/closure — no third ordering exists); the monitor criterion is a real two-task race.

S3.7 Acceptance criteria

  • TTL: shortening expires an over-age live session immediately; lengthening extends server-side validity; a fresh login's cookie Max-Age reflects the live ttl.
  • Limiter: no pre-lock config read remains; refill-through-now at old rate proven with a controlled clock; capacity cut clamps; SSE counts survive.
  • Retention: both consumers observe a change on their next pass.
  • Certs: bad candidate refused at prepare, store untouched; good change serves the new cert on the next handshake (-Dintegration loopback); a concurrent reload during an apply is serialized (test drives both under the seam).
  • Disabled-endpoint cert change: DB row changes, no store call, response marks no restart.
  • Sink: publish does no open/close (close observed in retire); bad path refused at prepare; new file receives lines; monitor samples the new dir; no use-after-free under concurrent sample; a same-path config-only apply concurrent with a forced rotation AND with a write-failure closure changes only config fields, never the handle; a target-change apply racing rotation swaps cleanly (old handle closed in retire); switching to a PRE-EXISTING nonempty file starts at its measured length; switching away from a target with rotate_pending set does not rotate the new target; a file → stderr apply detaches the handle (closed in retire) and clears position/rotation state.
  • Scheduler via the seam: shortened interval → next at new cadence; disable parks; re-enable anchors per rule; startup pass runs when disabled; failed pass advances the anchor; gate-skip advances the anchor.
  • zig build test green.

Session S4: logger queue controller

S4.1 Controller scope

logging.query_log_buffer_max. A stable controller owns the logger generation: buffer, Logger, writer future, and the writer's DB/gate dependencies (writer holds prepared statements for its run, logger.zig:514; future owned by serve today, app.zig:897). serve() creates the controller and delegates; shutdown ordering through the controller is EXACTLY today's safe order — quiesce producers → close queue / set draining → the writer drains the CLOSED queue → await the future (logger.zig:632, app.zig:921; never "drain then close": an empty writer blocks in getOne until close). Facade: QuerySink (query_sink.zig:18), metrics (web/metrics.zig:207) and health (web/handlers/health.zig:241, via web/server.zig:150) hold the CONTROLLER, not *Logger. The controller owns what must be continuous across swaps: counters, gate-episode state, last_drop_s, diagnostics references, AND the S1 privacy/flush atomics (setters address the controller and survive resize). writer_failed reflects the LIVE writer: a successful resize publish clears it (the new writer prepared cleanly); a retired writer's failures still land in the drop/diagnostic counters. Producer read-side protocol: a producer's log() acquires the live generation through the facade with a refcount (or a lock held through transform + enqueue) — a producer can never enqueue into a queue that retire has closed; retire waits for old-generation producer borrows to release before closing the old queue. This is what makes "no resize drop window" true, not an aspiration.

S4.2 Resize — all fallibility before commit

  1. Prepare (fallible): validate against the comptime ceiling (37449, startup's message); allocate the new buffer and Logger state; OPEN A NEW querylog DB connection for the new generation — one connection per writer for its whole life (logger.zig:514); two writers must never share querylog_writer_db. The connection factory: the canonical writer-connection opener (today cli.DataDir.reopenQuerylogDb + db.applyPragmas, cli.zig:350) MOVES into src/storage (a pub fn beside querylog_schema.open taking the data-dir handle + path); cli.zig delegates to it, and the controller receives the factory inputs at construction. The controller owns each generation's connection and closes it after that generation's writer is joined — and SPAWN the new writer PARKED: the spawn (io.concurrent can fail, Io.zig:2352) and statement preparation (runWriter can fail, logger.zig:524) happen NOW; the parked writer signals ready and waits on an activation gate. Any failure: tear down, close the new connection; nothing changed. If a PREVIOUS resize's retired generation has not finished retiring, prepare REFUSES with a distinct error ("previous resize still draining") — at most one retired generation exists, which bounds writers, connections, and buffers.
  2. Commit the DB row.
  3. Publish (infallible): swap the facade's live-generation pointer and open the activation gate. No waiting.
  4. Retire (controller-owned reaper): wait for old-producer borrows to release → close the old queue WITHOUT setting the process-shutdown draining flag (a distinct retirement mode: with draining set, a gate-held writer takes the GatedAtShutdown path and DROPS its batch, logger.zig:642/:726 — retirement instead lets it flush when the gate reopens) → the old writer drains the closed queue → await its future → close its DB connection → free logger + buffer. Reaper ownership: the reaper task is spawned ONCE at controller construction (boot — fallibility there is fine) and lives for the controller's life, processing retirements signaled by publish; publish never spawns. Future.await is not thread-safe (Io.zig:1198), so the reaper is the SOLE joiner of retired writers. Process shutdown, in order: quiesce producers → CLOSE the live queue (a writer blocked in getOne wakes only on close, logger.zig:642) → set draining on EVERY outstanding generation (live and retired — turning a gate-parked retired writer onto the counted GatedAtShutdown path) → join the LIVE writer → signal and join the REAPER (which finishes joining any retired writer). The f1a85d3 ordering holds within each join. Accounting: every entry accepted into the old queue is written when the gate allows, or counted by the existing gate/shutdown machinery; entries after publish land in the new queue. No resize-specific drop path exists.

S4 implementation notes (post-build)

  • The controller is a new file, src/storage/logger_controller.zig, not a second half of logger.zig (2190 lines before this session, and generations/retirement/the reaper are a separate concern from Entry and the writer loop). Add it to the module layout.
  • Logger keeps its exact public surface, so every f1a85d3 test passes unchanged. Continuity is achieved by SUMMING rather than by sharing cells: Controller.sample folds base (finals of joined generations) + the live generation + the outstanding retired one, under the controller mutex, and folds a retired generation's finals into base before freeing it. last_drop_s is a max, the gate episode is the worst of the outstanding generations, writer_failed is the live one's.
  • Logger.runWriter is split: it prepares its statements and calls the new pub fn runPrepared(io, writer, monitor). A resize generation prepares separately so a statement failure is a refused settings change. Logger.retire(io) is the close-without-draining retirement mode.
  • The BOOT generation deliberately takes the plain runWriter path, not the parked one: a boot that cannot prepare must still serve DNS with writer_failed set, which is today's behaviour. Only a resize can afford to refuse.
  • The connection factory is querylog_schema.reopen(io, dir, path); cli.DataDir.reopenQuerylogDb delegates. It takes the dir handle to make open's resolution pairing explicit at every call site, and does not dereference it.
  • The setters are Controller.setPrivacy(io, p) / setFlushInterval(io, s) and take io (they lock), unlike S1's pure-atomic pair. They store to every outstanding generation; each generation still keeps ONE packed privacy word, so a producer's single load stays the mixed-policy guarantee.
  • Controller.prepare returns typed errors; sizeMessage(err, requested, buf) renders config/validate.zig's exact wording for S5's client error.
  • Controller.retirementPending(io) is the "previous resize still draining" predicate, exposed for S5 and the tests.
  • Test sites keep their stack Loggers through logger_controller.Borrowed — S2's upstream_owner.Borrowed pattern.
  • S5 obligations unchanged: the S2 retire-time scoped reconciler and the S3 disabled-endpoint cert criterion. S4 adds none.

S4.3 Acceptance criteria

  • Resize under forced concurrent producers: no deadlock; exact accounting — old-queue entries all written (or gate-counted), new-queue entries written, produced == written + counted; no resize-specific drops.
  • Resize with the disk gate CLOSED: publish completes immediately; old writer retires after the gate reopens; nothing lost beyond what the gate itself counts.
  • Prepare failure (spawn or statement prep) leaves the running logger untouched and writes no DB row.
  • Invalid size refused with startup's message.
  • Counters/gate state/last_drop_s continuous across a swap (metrics read before and after agree modulo new writes).
  • Privacy interleaving test pauses a producer between its two field decisions across a concurrent setPrivacy: a mixed policy (domain redacted, client exposed, or the reverse) is impossible.
  • Shutdown while a retirement is gate-blocked: clean join, the retired writer's held batch is counted by GatedAtShutdown, no deadlock, no double-await.
  • f1a85d3 deadlock regression tests pass unchanged.
  • zig build test green.

Session S5: the apply table and the settings write path

S5.1 The key table

src/web/handlers/settings.zig: delete the comptime restart_required_keys generation (:68-95) and touchesRestartRequiredKey (:187-200). The table maps EVERY settings key to a CONCRETE operation enum — dns_policy, trusted_proxies, logger_privacy, logger_flush, disk_thresholds, upstream_generation, cache, rate_limiter, sessions_ttl, api_limiter, retention, certs_doh, certs_dot, log_sink, scheduler, logger_queue, bind, web_lifecycle — one entry per key, no second unguarded switch; the broad class (live/subsystem/bind/web_lifecycle) is DERIVED from the operation. bind (dns/web/doh/dot bind + port + doh/dot enabled) and web_lifecycle (web.enabled) set restart_pending exactly as today; milestone 35 executes them. Comptime test: walk @typeInfo(model.Config) as the old generator did; every scalar key appears exactly once (replaces :549).

S5.2 The PUT flow

validate → group changed keys by OPERATION → prepare one candidate per affected owner from the final merged config → any failure cleans up all prepared candidates, errors, no DB write → commit (failure: clean up) → publish each prepared candidate → retire. web.password keeps its existing path. Upstream create/update/delete (handlers/upstreams.zig): build the candidate generation from the HYPOTHETICAL post-mutation row set before the repository write; commit; publish. The direct restart_pending sets (:58/:90/:114) are deleted HERE. restart_required in upstream responses (upstreams.zig:156) becomes false; update the OpenAPI UpstreamEcho pin (openapi.yaml:2749), the /api/settings description that says every scalar setting requires restart (openapi.yaml:1699), the API reference claim that all upstream/settings mutations do (docs/reference/api.md:18), generated types, and contract samples.

S5.3 Config status

GET /api/config/status: restart_pending remains, fed only by bind/web_lifecycle operations. The settings envelope's restart_required list (:538) shrinks to those keys.

S5 implementation notes (post-build)

  • The table and the four phases live in a new file, src/web/handlers/apply.zig, not in settings.zig: the upstream RESOURCE handlers need the same prepare/publish/retire, so putting it in the settings handler would have made one handler the other's library. settings.zig re-exports restart_required_keys and nothing else of it. Add it to the module layout.
  • Operations are derived from the VALUES, not from the keys the patch named (changedOperations(before, after)). The admin form submits every field it read, and treating that as eighteen applies would resize the query log on every save — and refuse the second save with PreviousResizeDraining. A key rewritten to what it already was changes nothing, applies nothing, and owes no restart.
  • app.zig now heap-allocates the DNS cache and rate limiter and frees them THROUGH the handler (replaceCache(io, null) at teardown): after a cache.size apply, what the handler holds is not what boot built. Both are built inside one block whose errdefers end with it, so a boot that fails between the two creations frees them and a boot that fails later leaves them to the handler's teardown defers.
  • The OpenAPI overview and the three upstream mutation descriptions said restart-required. They now say the change is live, which is what the runtime does and what the restart_required: false pin already promised. docs/reference/api.md was already correct. contractSamples.gen.ts is generated from live responses rather than from the document, so no regeneration followed.
  • WebState gains upstream_build (the shared HTTP client, certificate bundle and its lock) and retention_days. Without the first, an upstream mutation is a database row and nothing else, which is what a handler test's borrowed owner wants.
  • Owner.published is a plain u64 under the owner mutex, with publishedCount(io), not an atomic: it is the observable the "one candidate per owner" tests read. An atomic counter in the same struct reproducibly perturbed the S2 test "a replace with no reader holding the live generation retires it through the return path" into failing (~1 run in 1); a mutex-guarded field is both the discipline the rest of Owner follows and stable. That S2 test's sensitivity to unrelated timing is worth a look on its own.
  • mutations.Resource now accepts a remove that returns error{OutOfMemory}!?Failure, because a delete that builds a candidate allocates. Both shapes are still checked exactly.
  • The upstream note rendering moved to upstream_owner.Rendered, shared by app.zig's boot replay and the runtime reconciler, so a warning raised at boot and the same warning raised by a write are byte-identical.
  • Reconciliation reads the new report by RE-ACQUIRING the owner rather than keeping the published pointer: a concurrent second write could retire and free that generation the moment its refs hit zero.
  • The S2 reconciler test drives its rebuilds through /api/upstreams, not /api/settings: a settings PUT validates the whole stored configuration and refuses a row malformed enough to produce a build finding, so the finding can never be reached from there.
  • logging.file_path must be absolute, so the integration environment resolves its tmp dir with Dir.realPath.
  • Plan.prepare abandons the partly built plan itself on a PROPAGATED error (errdefer self.abandon(io)), and only returns a Failure for the caller to abandon. An OutOfMemory out of the log-directory step used to bypass the caller's abandon, leaking whatever earlier owners had built and — because CertStore holds reload_mutex from its prepare to its publish — wedging every later certificate change and reload.
  • Controller.prepare takes the merged model.Logging, not an entry count. Seeding the candidate from the LIVE privacy and flush values was wrong for the one PUT that changes the buffer size and a privacy flag together: publish applies the privacy to the generation being retired, so the replacement went live writing what the operator had just asked to hide.
  • The api_limiter reading is taken in prepare (Plan.limiter_now) and used in publish. A clock is I/O, and publish is I/O-free. That reading is STALE by publish time whenever a request served in between refilled a bucket past it, so setLimits is monotonic per bucket: a bucket already newer than now is clamped to the new capacity and keeps its clock. Rewinding it would let the next request buy the same interval a second time, at the new rate.
  • metrics.collect loads handler.cache and handler.limiter INSIDE the mutexes that guard them, the same way the request path does: Plan.retire frees the displaced object as soon as the swap returns, so a pointer read before the lock can be freed under the reader.
  • CertStore.publishPathChange frees the displaced entry and the old paths BEFORE bumping reloads. A bool live across an atomic read-modify-write is the Debug-backend miscompile AGENTS.md documents.
  • Two S5.4 criteria are proven one step short of the wording. certs_doh/certs_dot assert the store reloaded and the owned paths moved, not a TLS handshake against the new certificate; dns_policy asserts policySnapshot, which is the read a query performs, not a query. Both are the collaborator's own observable, and neither gap is a claim about a path that is untested.

S5.4 Acceptance criteria — one real-PUT test per OPERATION

Every operation proven through the real route dispatch → prepare → commit → publish path:

  • dns_policy (blocking.ttl live on next query — integration), trusted_proxies, logger_privacy, logger_flush, disk_thresholds, cache (size change swaps, next lookup misses), rate_limiter, sessions_ttl, api_limiter, retention, certs_doh AND certs_dot separately (enabled: handshake serves new cert — integration; disabled: DB-only), log_sink (+ monitor re-point on path change AND on output change both directions), scheduler (via seam), logger_queue (accounting per S4), upstream_generation (row mutation: next exchange uses it, response restart_required: false; timeout change: generation counter +1).
  • web_lifecycle: a web.enabled PUT commits the DB row, sets restart_pending, and executes NOTHING (milestone 35 executes it).
  • A multi-key PUT touching one owner twice builds ONE candidate (generation counter delta == 1).
  • An upstream PUT while G1 is held by an in-flight exchange: publish + diagnostics reconciliation complete on the PUT task; G1 retires later via release; events correct throughout.
  • A PUT mixing a live key with a failing prepare writes NOTHING and changes nothing.
  • A bind key PUT still sets restart_pending; nothing else can.
  • Contract samples for /api/config/status, settings envelope, upstream responses regenerate; route-count pin passes.
  • zig build test and zig build test -Dintegration green.

Session S6: admin SPA

  • Per-field "needs restart" tags (SettingsForm.tsx:119-123/:209-223/:267) render only for keys the API lists (bind + web_lifecycle). Banner logic untouched. Upstream flows lose restart messaging; regenerated types carry restart_required: false. Live fields say nothing — silence is success. Regenerate goldens (just goldens).
  • From admin/: npm run typecheck && npm test && npm run lint && npm run format:check && npm run build && npm run assert-bundled green; bundle under ceiling.
  • A test: live-key edit renders no restart affordance; a port edit still does.

S6 implementation notes (post-build)

  • SettingsForm needed no change at all. It already built its mark set from envelope.restart_required and nothing else, so the twelve-key list arrived and the other fields went quiet on their own. The session's real work was the copy and the tests that had pinned the old answer.
  • The tests now take the key list from sample_get_settings.restart_required in the committed contract sample, re-exported as RESTART_REQUIRED_KEYS from the configuration fixtures, rather than from a hand-written array. A server-side change to the set now fails the tests that pin it instead of passing against a stale copy.
  • One test keeps a hand-written list, ["logging.level"]: no shipped restart-required key is enum-backed, so the Select rendering — where the mark reaches a screen reader through aria-describedby, not through the label — has no key left to exercise it. The list is the server's to change and the form must render any key it names, so the branch and its test stay, with the override stated in the test.
  • invalidateUpstreams dropped its /api/config/status invalidation. An upstream write rebuilds the pool in the running process, so the status answer after the write is the answer before it, and re-reading it only asserted a claim that is no longer true.
  • just goldens was not run: S5 regenerated the samples and this session changed no server response.

Module layout (new/changed)

src/upstream/owner.zig (new); src/storage/logger_controller.zig (new); src/storage/querylog_schema.zig; src/cli.zig; src/server/handler.zig; src/server/query_sink.zig; src/web/server.zig; src/storage/logger.zig; src/storage/disk_monitor.zig; src/storage/retention.zig; src/server/clients.zig; src/platform/logging.zig; src/filter/manager.zig; src/server/cert_store.zig; src/web/auth.zig + src/web/handlers/auth.zig; src/web/api_limiter.zig; src/web/handlers/apply.zig (new); src/web/handlers/settings.zig; src/web/handlers/upstreams.zig; src/web/handlers/certs.zig; src/web/metrics.zig; src/web/handlers/health.zig; src/web/openapi.yaml; docs/reference/api.md; src/app.zig; admin/src/features/configuration/*.

File ownership

Strictly sequential; each session owns the tree while it runs.

Acceptance criteria (milestone complete)

  • Every settings key except bind/web_lifecycle applies live through its table-declared operation, each proven by a real-PUT test (S5.4).
  • restart_pending only from bind/web_lifecycle; no API response or doc claims a restart for anything else.
  • No fallible or I/O work in any publish; every prepare/commit failure leaves DB and runtime unchanged.
  • Full gates: zig build test, zig build test -Dintegration, admin suite from admin/ incl. assert-bundled, zig fmt --check build.zig src tools, bundle ceiling.
  • Querylog schema fingerprint unchanged (cut's schema-gate takes the equal branch).

Anti-requirements

  • No generic hot-swap framework; one named operation per owner.
  • No listener rebind, no web.enabled execution, no applied: false surface, no restart_pending removal — milestone 35.
  • No config file watcher; file mode unchanged. No new config keys; no DB schema changes. Sessions do not become sliding.