Goal (PLAN §16 Phase 3): UDP server, TCP server, DoH + DoT upstream clients, upstream pool with priority failover, backoff and health. Exit: A/AAAA forwarding over UDP and TCP; health populated.

Read first: AGENTS.md (values), specs/research/zig-0.16-api-notes.md (verified stdlib facts — pre-0.16 knowledge is stale and MUST NOT be used), specs/milestone-1.md and specs/milestone-2.md (module conventions and the "As built" notes). The Zig source of truth is /home/mokhtar/app/zig at tag 0.16.0. RFCs: 1035 §4.2.2 (TCP length prefix), 7766 (DNS over TCP), 7858 (DoT), 8484 (DoH), 6891 (EDNS(0)), 4343 (name case-insensitivity), 9619 (QDCOUNT must be 1).

What already exists (do not respecify, import it)

src/dns/* — pure wire format. Used here: packet.parse, packet.Packet, packet.firstQuestion, packet.findOptRecord, packet.ResponseBuilder, packet.setId, header.parse, header.Flags, question.Question, name.eqlIgnoreCase, edns.parseOpt, types.Rcode, types.Type.
src/platform/address.zig — NetAddress, Prefix, matchLongest.
src/platform/tls_client.zig — TlsStream (pinned struct; init, reader, writer, close), classify, ErrorClass. DoT builds on this; do not open std.crypto.tls.Client directly.
src/platform/tls_server.zig — server-side TLS. Not used in this milestone (local DoH/DoT endpoints are Phase 9).
build.zig — -Dintegration (hermetic, loopback only, PR-blocking) and -Dlive (leaves the machine, manual workflow only) reach test files through @import("build_options").

Sessions

S1 (upstream/transport.zig + upstream/health.zig)
      |
      +--> S2 (doh_client)  S3 (dot_client)  S4 (pool)  S5 (handler)      [parallel]
                                   |
                                   +--> S6 (udp_server + tcp_server)
                                              |
                                              +--> S7 (e2e integration test)

S1 defines every type the other sessions share, so S2–S5 can be written against the spec alone. S6 depends on S5's Handler type; S7 depends on everything.

The orchestrator — not any session — wires src/tests.zig imports and any build.zig change. A session that needs a build change reports the exact change in its completion report.

Design invariants (all sessions)

The pure core stays pure. src/dns/ gains nothing. All std.Io use lives in src/server/ and src/upstream/. upstream/health.zig and the validation half of upstream/transport.zig are themselves pure (timestamps and randomness arrive as parameters), so they are unit-testable without a backend.
Failures are classified into three disjoint groups: peer fault, local resource, cancellation. Health and backoff count peer faults ONLY. A local OutOfMemory must never mark an upstream sick, and error.Canceled must never be recorded at all.
QDCOUNT must be exactly 1 (RFC 9619) — for inbound client queries (else FORMERR) and for upstream responses (else the response is a peer fault).
Every response from an upstream is validated against the request it answers before it reaches a client: ID equal, QDCOUNT 1 on both sides, QR set, question name equal case-insensitively (RFC 4343), qtype and qclass equal.
TCP and DoT messages carry a 2-byte big-endian length prefix (RFC 1035 §4.2.2). DoH does not.
No stream read or write in 0.16.0 accepts a timeout (verified: Io.Operation has no net-stream variants). Bound them by running the work under io.concurrent / std.Io.Select and cancelling the loser, exactly as src/platform/tls_client_integration_test.zig does. UDP receive is the one exception — Socket.receiveTimeout is implemented on the POSIX Threaded backend (verified: Io/Threaded.zig:2779 handles net_receive in the pollable batch path).
Never set net.IpAddress.ConnectOptions.timeout — the Threaded backend panics (Io/Threaded.zig:12077).
Every dropped datagram, refused connection and failed send increments a named counter. No silent drops (AGENTS.md).
Unit tests live in-file. Tests that touch loopback sockets live in a separate *_integration_test.zig file guarded by if (!build_options.integration) return error.SkipZigTest;. Tests that leave the machine are guarded by build_options.live. This mirrors milestone 1 — the guard is a runtime return so the body is always compiled and cannot rot.

Session S1: `src/upstream/transport.zig`, `src/upstream/health.zig`

Foundation. No sockets are opened in this session; transport.zig names the Io-taking interface that S2–S4 implement, and everything else in both files is pure.

S1.1 transport.zig — endpoint parsing

pub const max_message_len = 65535; // RFC 1035 §4.2.2 length prefix is 16-bit
pub const doh_default_port = 443;
pub const dot_default_port = 853; // RFC 7858 §3.1
pub const doh_default_path = "/dns-query"; // RFC 8484 §4.1 well-known template

pub const Scheme = enum { doh, dot };

/// Borrowed view over the configured URL text; the caller owns the string.
pub const Endpoint = struct {
    scheme: Scheme,
    url: []const u8,   // the original text, for logs and the health API
    host: []const u8,  // no brackets, no port; SNI and certificate verification name
    port: u16,
    path: []const u8,  // DoH only; always starts with '/'; `doh_default_path` when absent

    pub const ParseError = error{ UnsupportedScheme, MissingHost, BadPort, BadUrl };

    /// `https://…` => .doh, `tls://…` => .dot (PLAN §9). Accepts `[v6]:port` bracket form.
    pub fn parse(url: []const u8) ParseError!Endpoint;
};

Hand-written parse (a std.Uri round trip would hand back percent-encoded components that need re-decoding for one household-scale config value). Rules: reject any other scheme, reject an empty host, reject a port that is empty, non-numeric or > 65535, reject a tls:// URL that carries a path other than / or nothing.

Tests: https://cloudflare-dns.com/dns-query, https://dns.example/x (path preserved), https://dns.example (path defaults), tls://dns.google:853, tls://dns.google (port defaults to 853), https://[2606:4700:4700::1111]:8443/dns-query (host without brackets, port 8443), udp://1.1.1.1:53 → UnsupportedScheme, https:// → MissingHost, https://h:99999/ → BadPort.

S1.2 transport.zig — error groups

/// The upstream misbehaved, timed out, or was unreachable. Only these count against health.
pub const PeerFault = error{
    ConnectFailed,
    TlsFailed,
    SendFailed,
    ReceiveFailed,
    Timeout,
    BadResponse,      // unparseable, not a response, or QDCOUNT != 1
    ResponseMismatch, // ID or question does not match the query
    ResponseTooLarge, // does not fit the caller's buffer
    HttpStatus,       // DoH: status other than 200
    HttpContentType,  // DoH: content-type other than application/dns-message
};

/// This process ran out of something. Never the upstream's fault.
pub const LocalResource = error{
    OutOfMemory,
    SystemResources,
    ProcessFdQuotaExceeded,
    SystemFdQuotaExceeded,
    BufferTooSmall,   // caller-supplied buffer cannot hold even a query
    Unexpected,
};

pub const Cancellation = error{Canceled};

pub const ExchangeError = PeerFault || LocalResource || Cancellation;

pub const Group = enum { peer_fault, local_resource, cancellation };

/// Exhaustive switch over `ExchangeError` — no `else` arm. A new error member must break
/// the build here, so no failure can silently land in the wrong group.
pub fn group(err: ExchangeError) Group;

The three sets are disjoint by construction; a test asserts it by walking @typeInfo(PeerFault).error_set.? and friends and checking no name appears twice.

pub fn mapLocal(err: anyerror) ?ExchangeError — helper for S2/S3: returns the LocalResource or Cancellation member matching a foreign stdlib error by name, null when the caller should treat the error as a peer fault. Implemented with an explicit switch over the named errors listed above plus error.Canceled; this is the ONLY place a foreign error set is folded in.

S1.3 transport.zig — the client interface

/// A thing that sends one DNS message and returns one validated DNS message.
/// Implemented by DohClient, DotClient, Pool, and test fakes.
pub const Client = struct {
    ptr: *anyopaque,
    exchangeFn: *const fn (
        ptr: *anyopaque,
        io: std.Io,
        query: []const u8,
        response_buf: []u8,
    ) ExchangeError![]u8,

    /// Returns a prefix of `response_buf`. The returned message has already passed
    /// `validateResponse` against `query`.
    pub fn exchange(
        self: Client,
        io: std.Io,
        query: []const u8,
        response_buf: []u8,
    ) ExchangeError![]u8 {
        return self.exchangeFn(self.ptr, io, query, response_buf);
    }
};

S1.4 transport.zig — response validation (pure)

pub const ValidateError = error{ BadResponse, ResponseMismatch };

/// RFC 9619: exactly one question on both sides. RFC 4343: names compare case-insensitively,
/// so a case-mangling (0x20) upstream still matches. Does not inspect the answer section —
/// content policy is not this layer's business.
pub fn validateResponse(query: []const u8, response: []const u8) ValidateError!void;

Order of checks: response parses (packet.parse failure → BadResponse); response header flags.qr is true (else BadResponse); response qdcount == 1 (else BadResponse); query parses and has qdcount == 1 (else BadResponse — a caller bug, but never a crash); IDs equal (else ResponseMismatch); question qtype and qclass equal and name.eqlIgnoreCase (else ResponseMismatch).

Tests (hand-built byte fixtures, reuse the shape of src/dns/packet.zig's fixtures): matching pair passes; mixed-case question name passes; wrong ID → ResponseMismatch; different qtype → ResponseMismatch; different name → ResponseMismatch; QR clear → BadResponse; response with QDCOUNT 0 → BadResponse; response with QDCOUNT 2 → BadResponse; truncated garbage → BadResponse.

S1.5 health.zig — pure health and backoff state

pub const Config = struct {
    /// Consecutive peer faults before the endpoint is put in backoff.
    failure_threshold: u8 = 2,
    base_backoff_ms: u32 = 500,
    max_backoff_ms: u32 = 60_000,
};

pub const window_len = 32; // rolling success-rate window

pub const State = struct {
    consecutive_failures: u32,
    total_successes: u64,
    total_failures: u64,
    last_success_at: ?std.Io.Timestamp,
    last_error_at: ?std.Io.Timestamp,
    last_error_buf: [48]u8,
    last_error_len: u8,          // @errorName of the last peer fault, truncated
    backoff_until: ?std.Io.Timestamp,
    window: u32,                 // bitset, 1 = success, LSB = most recent
    window_filled: u8,

    pub const init: State = ...;

    pub fn recordSuccess(self: *State, at: std.Io.Timestamp) void;

    /// `err_name` is `@errorName` of a PeerFault member. `rand` supplies jitter; the caller
    /// owns the RNG so this stays pure and the test is deterministic.
    pub fn recordFailure(
        self: *State,
        at: std.Io.Timestamp,
        err_name: []const u8,
        cfg: Config,
        rand: u32,
    ) void;

    pub fn available(self: *const State, now: std.Io.Timestamp) bool;
    pub fn successRate(self: *const State) f32; // over the filled part of the window; 1.0 when empty
    pub fn lastError(self: *const State) []const u8;
};

Rules:

recordSuccess clears consecutive_failures and backoff_until, pushes a 1 into the window, and sets last_success_at to @max(existing, at) by nanoseconds.
recordFailure increments consecutive_failures and total_failures, pushes a 0, copies err_name (truncated to the buffer), sets last_error_at to @max(existing, at). When consecutive_failures >= cfg.failure_threshold, it computes delay_ms = min(max_backoff_ms, base_backoff_ms << shift) where shift = min(consecutive_failures - failure_threshold, 20), applies jitter jittered = delay_ms/2 + rand % (delay_ms/2 + 1), and sets backoff_until = @max(existing, at + jittered).
Out-of-order completions are the normal case, not an edge case: two concurrent exchanges against the same endpoint complete in either order, so at can move backwards between calls. Every timestamp field therefore updates through @max on .nanoseconds, and backoff_until is only ever extended, never shortened. available must never compute a negative duration.
State carries no lock. The pool owns the mutex.

Tests: threshold not reached → available(now) stays true; threshold reached → unavailable until backoff_until, available one nanosecond after; consecutive failures grow the delay and it saturates at max_backoff_ms (walk 40 failures with rand = 0); success resets consecutive count, window and backoff; jitter with rand = 0 and rand = maxInt(u32) both land inside [delay/2, delay]; an out-of-order pair — recordSuccess(t=100) then recordSuccess(t=50) — leaves last_success_at at 100; recordFailure(t=50) after recordFailure(t=100) does not shorten backoff_until; successRate over a half-success window is 0.5; lastError returns the last recorded name and is truncated, not overflowed, by a name longer than the buffer.

S1.6 Acceptance criteria

zig test src/upstream/transport.zig and zig test src/upstream/health.zig pass.
group compiles with no else arm over ExchangeError.
The disjointness test over the three error sets passes.
Every listed validation and health case has a test.
zig fmt --check clean on both files.

Session S2: `src/upstream/doh_client.zig` (+ live test)

DoH per RFC 8484 over std.http.Client (HTTP/1.1; HTTP/2 is permanently out of scope, PLAN §2.2).

S2.1 API

pub const DohClient = struct {
    http: *std.http.Client,        // caller-owned; shared across endpoints, pools connections
    endpoint: transport.Endpoint,
    uri: std.Uri,                  // built once in init from `endpoint`
    request_buf: []u8,             // caller-owned; sendBodyComplete needs a mutable body
    transfer_buf: []u8,            // caller-owned; HTTP body transfer buffer

    pub const InitError = error{BadUrl};

    pub fn init(
        http: *std.http.Client,
        endpoint: transport.Endpoint,
        request_buf: []u8,   // asserted >= 512
        transfer_buf: []u8,  // asserted >= 1024
    ) InitError!DohClient;

    pub fn client(self: *DohClient) transport.Client;
};

self.uri is derived from endpoint with std.Uri.parse(endpoint.url); a parse failure is error.BadUrl. endpoint remains the source of truth for host/port/scheme in logs.

S2.2 Exchange

if (query.len > self.request_buf.len) return error.BufferTooSmall; then copy query into request_buf — Request.sendBodyComplete takes []u8, not []const u8 (verified, std/http/Client.zig:935).

    .keep_alive = true,
    .redirect_behavior = .not_allowed,
    .headers = .{
        .content_type = .{ .override = "application/dns-message" },
        // Compressed bodies would need the decompressing reader; identity keeps the
        // response body byte-exact for `validateResponse`.
        .accept_encoding = .{ .override = "identity" },
    },
    .extra_headers = &.{.{ .name = "accept", .value = "application/dns-message" }},
}); defer req.deinit();

Request.Headers has no accept field (verified, std/http/Client.zig:845), hence extra_headers.

try req.sendBodyComplete(self.request_buf[0..query.len]);
var resp = try req.receiveHead(&.{}); — an empty redirect buffer is legal with .not_allowed.
if (resp.head.status != .ok) return error.HttpStatus;
Content-type: take resp.head.content_type orelse return error.HttpContentType, cut at the first ;, trim ASCII whitespace, std.ascii.eqlIgnoreCase against "application/dns-message", else error.HttpContentType.
const body = resp.reader(self.transfer_buf); then fill response_buf with repeated readSliceShort until it returns 0. If the body has not ended when response_buf is full → error.ResponseTooLarge. A content_length larger than response_buf.len short-circuits to the same error.
try transport.validateResponse(query, response_buf[0..len]); mapping ValidateError members straight through (both are PeerFault members).
Return response_buf[0..len].

The query ID is sent unchanged. RFC 8484 §4.1 suggests ID 0 for HTTP cache friendliness; nxdns puts no HTTP cache in this path, and keeping the ID preserves the request/response binding that validateResponse enforces. State that reason in a comment.

Error mapping: one fn mapError(err: anyerror) transport.ExchangeError that first tries transport.mapLocal(err), and otherwise maps by phase — request/connect errors → error.ConnectFailed, TLS-named errors (@errorName starting with "Tls" or "Certificate") → error.TlsFailed, sendBodyComplete errors → error.SendFailed, receiveHead and body-read errors → error.ReceiveFailed. Do the phase distinction at the call site (each catch knows its phase), not by guessing from the error name.

S2.3 Tests

In-file unit: init with a tls:// endpoint is rejected before any I/O (assert or BadUrl); init asserts on an undersized request_buf; the content-type matcher accepts "application/dns-message", "Application/DNS-Message" and "application/dns-message; charset=utf-8", and rejects "text/html" and a missing header. Factor the matcher into fn contentTypeOk(value: ?[]const u8) bool so it is testable without a server.
src/upstream/doh_client_live_test.zig, guarded by build_options.live: POST an A query for example.com to https://cloudflare-dns.com/dns-query, assert the reply validates and carries at least one answer record. Whole exchange raced against a 10 s budget with std.Io.Select, the pattern in src/platform/tls_client_integration_test.zig. This host's IPv6 egress is broken (API notes) and DoH needs name resolution — if the test stalls on AAAA, report it as an environment finding rather than working around it; the test is non-blocking by construction.

S2.4 Acceptance criteria

zig test src/upstream/doh_client.zig passes (unit tests, no network).
DohClient satisfies transport.Client (a compile-time test takes client() and calls exchange through the interface against a std.http.Client that is never driven — verify by instantiation, not by running an exchange).
The live test compiles in every build and skips without -Dlive.
zig fmt --check clean.

Session S3: `src/upstream/dot_client.zig` (+ live test)

DoT per RFC 7858: TLS on port 853, DNS messages framed exactly as over TCP (RFC 1035 §4.2.2).

S3.1 API

pub const DotClient = struct {
    endpoint: transport.Endpoint,
    gpa: std.mem.Allocator,
    bundle: *std.crypto.Certificate.Bundle, // caller-owned, shared
    bundle_lock: *std.Io.RwLock,            // caller-owned, shared
    buffers: Buffers,                       // caller-owned; one DotClient is used by one task

    pub const Buffers = struct {
        tls_read: []u8,     // >= std.crypto.tls.Client.min_buffer_len
        tls_write: []u8,    // >= std.crypto.tls.Client.min_buffer_len
        stream_read: []u8,  // >= std.crypto.tls.Client.min_buffer_len
        stream_write: []u8, // >= std.crypto.tls.Client.min_buffer_len
    };

    pub fn init(
        endpoint: transport.Endpoint,
        gpa: std.mem.Allocator,
        bundle: *std.crypto.Certificate.Bundle,
        bundle_lock: *std.Io.RwLock,
        buffers: Buffers,
    ) DotClient; // asserts endpoint.scheme == .dot and every buffer length

    pub fn client(self: *DotClient) transport.Client;
};

S3.2 Exchange

One TCP connection and one TLS handshake per exchange, closed before returning. Connection reuse is not built: at household query rates the pool's per-attempt budget and the failover path stay simple and every failure is attributable to one exchange, which is worth more here than the saved round trips. Say that in a comment — it is a decision, not a stub.

Resolve the endpoint to an std.Io.net.IpAddress with net.IpAddress.parse(endpoint.host, endpoint.port). A non-literal host is error.ConnectFailed with a log line naming the host: name resolution for upstreams is not in this milestone's scope, and a silent fallback would hide it. (tls://dns.google:853 from PLAN §12.1 therefore needs an IP literal in config; note this in the completion report so the orchestrator can carry it into the Phase 4 config validator.)
var stream = try address.connect(io, .{ .mode = .stream }); — no timeout field, ever. defer a close that runs under cancel protection (below).
var tls_stream: tls_client.TlsStream = undefined; then tls_stream.init(io, &stream, bundle, bundle_lock, gpa, .{ .host = endpoint.host, .ca = .system, … }). TlsStream is pinned — do not copy or move it after init.
Write [2]u8 big-endian length then query to tls_stream.writer(), then flush().
Read exactly 2 bytes, then that many bytes, from tls_stream.reader(). len == 0 → error.BadResponse. len > response_buf.len → error.ResponseTooLarge. A short read or EOF → error.ReceiveFailed.
try transport.validateResponse(query, response_buf[0..len]);
tls_stream.close(); then stream.close(io);.

Cleanup under cancellation: the pool cancels this task on timeout, and the next cancelable Io call in the defer chain would return error.Canceled and skip the socket close. Wrap the close path in const prev = io.swapCancelProtection(.blocked); defer _ = io.swapCancelProtection(prev); (verified: std/Io.zig:1342) so the socket is always released.

Error mapping: connect → error.ConnectFailed; TlsStream.init → error.TlsFailed (log tls_client.classify(err) alongside — that is what milestone 1 built it for); write/flush → error.SendFailed; read → error.ReceiveFailed; transport.mapLocal first at every site.

S3.3 Tests

In-file unit: framePrefix(len: u16) [2]u8 and parsePrefix(bytes: [2]u8) u16 round-trip, including 0 and 65535, big-endian byte order asserted explicitly; init asserts on a .doh endpoint; a non-literal host maps to error.ConnectFailed without touching the network (call the address-resolution helper directly).
src/upstream/dot_client_live_test.zig, guarded by build_options.live: exchange an A query for example.com against tls://1.1.1.1:853 with endpoint.host overridden to "1.1.1.1" — this machine's IPv6 egress is dead, so the documented anycast literal is used, and Cloudflare's certificate carries 1.1.1.1 as an IP SAN. If the stdlib verifier rejects an IP-literal SAN, do NOT weaken verification: report the finding. Raced against a 10 s budget with std.Io.Select.

S3.4 Acceptance criteria

zig test src/upstream/dot_client.zig passes (unit tests, no network).
DotClient satisfies transport.Client.
The live test compiles in every build and skips without -Dlive.
No direct std.crypto.tls.Client use — everything goes through platform/tls_client.zig.
zig fmt --check clean.

Session S4: `src/upstream/pool.zig`

Priority-ordered sequential failover with per-attempt deadline, health tracking and backoff (PLAN §9). The pool is itself a transport.Client, so the handler sees one interface.

S4.1 API

pub const Entry = struct {
    endpoint: transport.Endpoint,
    client: transport.Client,
    priority: i32,     // lower first (PLAN §11.2 `upstreams.priority`)
    enabled: bool,
    health: health.State,
};

pub const Snapshot = struct {
    url: []const u8,
    enabled: bool,
    available: bool,
    consecutive_failures: u32,
    total_successes: u64,
    total_failures: u64,
    success_rate: f32,
    last_success_at: ?std.Io.Timestamp,
    last_error_at: ?std.Io.Timestamp,
    last_error: []const u8, // borrowed from the entry; valid until the next failure
    backoff_until: ?std.Io.Timestamp,
};

pub const Pool = struct {
    entries: []Entry,   // caller-owned, sorted ascending by priority in `init`
    cfg: health.Config,
    attempt_timeout: std.Io.Clock.Duration,
    mutex: std.Io.Mutex,
    rng: std.Random.DefaultPrng,

    pub fn init(
        entries: []Entry,
        cfg: health.Config,
        attempt_timeout: std.Io.Clock.Duration,
        seed: u64,
    ) Pool; // asserts entries.len > 0

    pub fn client(self: *Pool) transport.Client;

    pub fn exchange(
        self: *Pool,
        io: std.Io,
        query: []const u8,
        response_buf: []u8,
    ) transport.ExchangeError![]u8;

    /// Copies health into `out` in pool order; returns the number written.
    /// Feeds `GET /api/upstream/health` in Phase 8.
    pub fn snapshot(self: *Pool, io: std.Io, out: []Snapshot) std.Io.Cancelable!usize;
};

attempt_timeout uses the .awake clock (.{ .raw = .fromMilliseconds(2000), .clock = .awake }) so a suspended Pi does not burn the budget.

S4.2 Failover algorithm

Take now = std.Io.Clock.awake.now(io).
Pass one: walk entries in order; skip !enabled; skip entries whose health.available(now) is false.
Pass two, only if pass one attempted nothing: walk again, skipping only !enabled. Every endpoint being in backoff must not turn into SERVFAIL for every client — a probe is better than a guaranteed failure, and it is how backoff recovers. Test this explicitly.
For each candidate, run attempt:
- Race entry.client.exchange(io, query, response_buf) against attempt_timeout.sleep(io) using std.Io.Select with a two-arm union, then race.cancelDiscard() on the way out (the milestone-1 pattern). The sleep arm winning yields error.Timeout.
- On success: lock the mutex, entry.health.recordSuccess(completed_at), unlock, return the slice.
- On error, switch on transport.group(err):
  - .peer_fault → lock, recordFailure(completed_at, @errorName(err), self.cfg, self.rng.random().int(u32)), unlock, remember it as last_fault, continue to the next candidate.
  - .local_resource → return the error immediately. Do not record. Do not try another upstream: the next one will hit the same wall.
  - .cancellation → return error.Canceled immediately. Do not record.
- completed_at is std.Io.Clock.awake.now(io) taken after the attempt returns.
All candidates exhausted → return last_fault (guaranteed non-null: entries.len > 0 and pass two attempts every enabled entry). If every entry is disabled, return error.ConnectFailed.

response_buf is handed to each attempt in turn; a failed attempt may have written into it, so the returned slice is only meaningful on success. Note this in a comment.

Concurrency: several handler tasks share one Pool. The mutex guards health mutation and snapshot only — never an in-flight exchange, so a slow upstream cannot block bookkeeping. Entry client values must therefore be safe for concurrent exchange calls; DohClient and DotClient own per-instance buffers, so one Entry per concurrent task, or one shared pool with per-entry serialization, is a wiring decision for the orchestrator. Record in the completion report which one this session assumed (the specified Entry layout assumes each entry's client is used by one task at a time; the e2e test in S7 exercises exactly one in-flight query at a time).

S4.3 Tests (in-file, no sockets — fake clients implement `transport.Client`)

Priority order: entry with priority 10 is tried before priority 100.
Failover: first fake returns error.Timeout, second returns a valid response → the pool returns the second one's bytes; first entry's consecutive_failures == 1, second's total_successes == 1.
Backoff skip: drive a fake past failure_threshold, then assert the next exchange does not call that fake (a call counter in the fake) while another entry is available.
All-backed-off probe: every entry in backoff → pass two still calls them.
.local_resource short-circuit: a fake returning error.OutOfMemory makes exchange return error.OutOfMemory, the second fake is never called, and the first entry's total_failures stays 0.
.cancellation short-circuit: a fake returning error.Canceled returns error.Canceled and records nothing.
Attempt timeout: a fake that sleeps longer than attempt_timeout yields a recorded error.Timeout peer fault and moves on. (Needs a live Io — use std.Io.Threaded inside the test; no sockets are involved, so this stays in the default zig build test.)
All entries disabled → error.ConnectFailed.
snapshot reports the counters set by the preceding cases.

S4.4 Acceptance criteria

zig test src/upstream/pool.zig passes, including the timeout test on a Threaded backend.
Every bullet in S4.3 exists as a named test.
Pool satisfies transport.Client.
zig fmt --check clean.

Session S5: `src/server/handler.zig`

The serving handler: bytes in from a listener, validated bytes out. It owns no socket and no clock beyond what it passes to the upstream. Phase 7 extends this file with filtering, cache, local records and logging — nothing of that appears here.

S5.1 API

pub const Transport = enum { udp, tcp };

pub const Handler = struct {
    upstream: transport.Client, // in production `pool.client()`
    stats: Stats = .{},

    pub const Stats = struct {
        queries: std.atomic.Value(u64) = .init(0),
        dropped_malformed: std.atomic.Value(u64) = .init(0),
        formerr: std.atomic.Value(u64) = .init(0),
        notimp: std.atomic.Value(u64) = .init(0),
        servfail: std.atomic.Value(u64) = .init(0),
        truncated: std.atomic.Value(u64) = .init(0),
    };

    pub const Outcome = union(enum) {
        reply: []u8, // a prefix of `response_buf`
        drop,        // no response is possible or appropriate
    };

    /// `response_buf.len` must be >= 512 and <= 65535 (ResponseBuilder asserts the upper bound).
    pub fn handle(
        self: *Handler,
        io: std.Io,
        which: Transport,
        query: []const u8,
        response_buf: []u8,
    ) Outcome;
};

handle never returns an error: every failure is either a DNS response or a counted drop. That is the "every failure mode visible" rule applied to the hot path.

S5.2 Behavior

header.parse(query) fails (< 12 bytes) → dropped_malformed, .drop (PLAN §6.1: severely truncated is a silent drop).
hdr.flags.qr == true → a response arriving on a listener port; dropped_malformed, .drop.
packet.parse(query):
- error.Truncated → dropped_malformed, .drop.
- any other WalkError → FORMERR reply built from hdr with no echoed question (the question is what failed to parse); formerr.
hdr.flags.opcode != .query → NOTIMP reply echoing nothing; notimp.
hdr.qdcount != 1 → FORMERR (RFC 9619 makes any other value invalid, in both directions); formerr.
Forward: self.upstream.exchange(io, query, response_buf).
- success → queries incremented; then the UDP size check in §5.3; .reply.
- transport.group(err) == .cancellation → .drop (the process is shutting down; nothing to say).
- .peer_fault or .local_resource → SERVFAIL reply echoing the question; servfail.
Every synthesized reply preserves the request ID, opcode and RD bit, sets QR and RA, and echoes the question when one parsed — all of that is already packet.ResponseBuilder.init's contract. When the query carried an OPT record (packet.findOptRecord + edns.parseOpt), the reply ends with addOptEcho(opt, opt.do_bit) so the DO bit passes through (PLAN §6.1).

S5.3 UDP size limit and truncation

The upstream answer can exceed what the client will accept over UDP.

pub fn udpLimit(query_packet: packet.Packet) u16 — the query's OPT udp_payload_size clamped to [512, 4096], or 512 when there is no OPT record (RFC 1035 §4.2.1 / RFC 6891 §6.2.3).
which == .udp and the upstream reply is longer than udpLimit → build a reply with TC = 1, RCODE NOERROR, the question echoed, no answer records, OPT echoed when present; increment truncated. The client then retries over TCP, which is exactly what RFC 1035 §4.2.1 prescribes.
which == .tcp → no size limit beyond response_buf.

The upstream reply lands in response_buf; building a truncated reply over it needs a second buffer. Give handle no extra parameter: build the truncated reply in a stack [512]u8 inside handle and copy it into response_buf before returning. 512 is enough — the message is a header, one question and at most one OPT record, and a question longer than ~270 bytes cannot exist (name ≤ 255).

S5.4 Tests (in-file, fake upstream implementing `transport.Client`)

Build queries with hand-encoded bytes or packet.ResponseBuilder; the fake upstream answers by copying a fixture and calling packet.setId.

A query for example.com A over UDP returns the fake's bytes unchanged.
A response (QR set) sent to the handler → .drop, dropped_malformed == 1.
An 8-byte query → .drop.
A query with QDCOUNT 0 → FORMERR reply, question absent, ID preserved.
A query with QDCOUNT 2 → FORMERR.
A structurally broken question (name pointer loop) → FORMERR, formerr == 1.
Opcode update → NOTIMP.
Fake returns error.Timeout → SERVFAIL with the question echoed and RD preserved.
Fake returns error.OutOfMemory → SERVFAIL (not a drop).
Fake returns error.Canceled → .drop.
udpLimit: no OPT → 512; OPT 1232 → 1232; OPT 200 → 512; OPT 9000 → 4096.
Over UDP, a fake reply of 900 bytes with no OPT in the query → TC=1 reply, ancount 0, question echoed, truncated == 1; the same reply over TCP passes through untouched.
DO bit set in the query's OPT → set in the SERVFAIL reply's OPT.
Every synthesized reply re-parses with packet.parse.

S5.5 Acceptance criteria

zig test src/server/handler.zig passes.
Every bullet in S5.4 exists as a named test.
handle has no error union in its return type.
zig fmt --check clean.

Session S6: `src/server/udp_server.zig`, `src/server/tcp_server.zig`

Listeners. Both take a *handler.Handler, both use a bounded pool of in-flight tasks so one slow upstream cannot stall the listener, and both count every failure.

S6.1 udp_server.zig

pub const max_datagram = 4096; // EDNS ceiling this server advertises

pub const Options = struct {
    max_in_flight: u16 = 64,
};

pub const Stats = struct {
    received: std.atomic.Value(u64) = .init(0),
    dropped_oversize: std.atomic.Value(u64) = .init(0),  // datagram arrived with flags.trunc
    dropped_no_slot: std.atomic.Value(u64) = .init(0),
    receive_errors: std.atomic.Value(u64) = .init(0),
    send_errors: std.atomic.Value(u64) = .init(0),
};

pub const UdpServer = struct {
    socket: std.Io.net.Socket,
    handler: *handler.Handler,
    slots: []Slot,       // allocated in `bind`, freed in `deinit`
    free: std.DynamicBitSetUnmanaged or a u64 free-list — implementer's call, guarded by `mutex`
    mutex: std.Io.Mutex,
    stats: Stats,

    pub const Slot = struct {
        query: [max_datagram]u8,
        reply: [max_datagram]u8,
        from: std.Io.net.IpAddress,
        len: usize,
    };

    pub const BindError = std.Io.net.IpAddress.BindError || error{OutOfMemory};

    pub fn bind(
        gpa: std.mem.Allocator,
        io: std.Io,
        address: std.Io.net.IpAddress,
        h: *handler.Handler,
        options: Options,
    ) BindError!UdpServer;

    /// The kernel-assigned address; port 0 in `bind` resolves here (needed by tests).
    pub fn boundAddress(self: *const UdpServer) std.Io.net.IpAddress;

    /// Receive loop. Returns when the task is canceled or the socket is closed.
    pub fn serve(self: *UdpServer, io: std.Io) void;

    pub fn deinit(self: *UdpServer, gpa: std.mem.Allocator, io: std.Io) void;
};

Loop:

const msg = self.socket.receive(io, slot_query_buf) into a claimed slot. error.Canceled → return. Any other error → receive_errors, log, continue (a per-datagram error must not kill the listener).
msg.flags.trunc → dropped_oversize, release the slot, continue. A truncated datagram cannot be parsed reliably and answering it would be a guess.
No free slot → dropped_no_slot, continue. Do not queue: an unbounded queue is the cloudflared failure mode (PLAN §1) in a different costume.
group.concurrent(io, respondOne, .{ self, io, slot_index }) — an std.Io.Group owned by serve. respondOne calls self.handler.handle(io, .udp, query, reply), sends .reply with self.socket.send(io, &slot.from, bytes) (a send failure increments send_errors and is logged at debug, deduplicated by the caller later), then releases the slot.
serve awaits the group before returning, under cancel protection so in-flight replies are not abandoned mid-send.

deinit closes the socket and frees the slots. Closing the socket makes a blocked receive fail; the loop treats that as a stop signal.

S6.2 tcp_server.zig

pub const Options = struct {
    max_connections: u16 = 64,
    /// RFC 7766 §6.2.3 recommends a few seconds of idle tolerance.
    idle_timeout: std.Io.Clock.Duration = .{ .raw = .fromSeconds(10), .clock = .awake },
};

pub const Stats = struct {
    accepted: std.atomic.Value(u64) = .init(0),
    rejected_at_capacity: std.atomic.Value(u64) = .init(0),
    accept_errors: std.atomic.Value(u64) = .init(0),
    connection_errors: std.atomic.Value(u64) = .init(0),
    idle_timeouts: std.atomic.Value(u64) = .init(0),
};

pub const TcpServer = struct {
    server: std.Io.net.Server,
    handler: *handler.Handler,
    conns: []Conn,          // fixed slots, same bounded-capacity rule as UDP
    mutex: std.Io.Mutex,
    stats: Stats,

    pub const ListenError = std.Io.net.IpAddress.ListenError || error{OutOfMemory};

    pub fn listen(gpa, io, address, h: *handler.Handler, options: Options) ListenError!TcpServer;
    pub fn boundAddress(self: *const TcpServer) std.Io.net.IpAddress;
    pub fn serve(self: *TcpServer, io: std.Io) void;
    pub fn deinit(self: *TcpServer, gpa: std.mem.Allocator, io: std.Io) void;
};

Accept loop: self.server.accept(io); error.Canceled or error.SocketNotListening → return (deinit shuts the listening socket down, which is the documented way to unblock a pending accept — API notes). At capacity → close the stream immediately, rejected_at_capacity. Otherwise a group task per connection.

Per connection (RFC 7766 §6.2.1.1 — a connection may carry several queries, handled serially):

Read the 2-byte length prefix. The read is raced against options.idle_timeout with std.Io.Select; the sleep arm winning means idle_timeouts and a clean close.
len == 0 → close (connection_errors). len > transport.max_message_len is impossible (u16).
Read exactly len bytes; a short read → close, connection_errors.
handler.handle(io, .tcp, query, reply); .drop closes the connection (there is nothing to frame); .reply writes the 2-byte prefix plus the bytes and flushes.
Loop back to step 1.
Close under cancel protection and release the slot.

Buffers: each Conn owns query: [transport.max_message_len]u8 and reply: [transport.max_message_len]u8 plus the stream read/write buffers. That is 128 KiB+ per connection slot; with the default 64 slots that is ~8 MiB, inside the PLAN §18 budget. If a session finds a materially smaller layout that keeps the 65535-byte ceiling, take it and say so.

S6.3 Tests

Unit tests that need no socket go in-file (length-prefix framing helper, capacity accounting). The loopback tests live in src/server/udp_server_integration_test.zig and src/server/tcp_server_integration_test.zig, both guarded by if (!build_options.integration) return error.SkipZigTest;:

UDP: bind 127.0.0.1:0, run serve in an Io.Group task, send a query from a second bound socket with a fake-upstream handler, assert the reply's ID and question match; then send a 5-byte datagram and assert no reply arrives within a short budget and dropped_malformed moved.
TCP: listen on 127.0.0.1:0, connect, send two length-prefixed queries on one connection, read two length-prefixed replies, assert both match, then close; assert accepted == 1.
TCP idle timeout: connect, send nothing, assert the server closes the connection and idle_timeouts == 1 (use a short idle_timeout in the test options).
Both: deinit while serve is blocked ends serve without a hang — the group awaits cleanly.

S6.4 Acceptance criteria

zig test src/server/udp_server.zig and .../tcp_server.zig pass (unit tests).
Both integration files compile in every build, skip without -Dintegration, and pass with it.
No unbounded queue, no unbounded allocation per datagram or per connection.
zig fmt --check clean.

Session S7: end-to-end integration test

src/server/resolver_integration_test.zig, guarded by if (!build_options.integration) return error.SkipZigTest;. Hermetic: loopback sockets and an in-process fake upstream, no external network.

S7.1 Fake upstream

In this file: a struct implementing transport.Client that parses the incoming query, echoes the question and appends one A record (93.184.216.34, TTL 300) with packet.ResponseBuilder, and counts calls. A second variant returns a configured transport.PeerFault on the first N calls.

S7.2 The test

std.Io.Threaded backend, io.
Two Pool entries: entry 0 = always-fails fake (priority 10), entry 1 = good fake (priority 20).
Handler over pool.client().
UdpServer bound to 127.0.0.1:0 and TcpServer on 127.0.0.1:0, both served in one Io.Group.
Client side: send an A query for example.com over UDP; assert the reply parses, ID matches, RCODE is NOERROR, ancount is 1, and the A record's rdata is 93.184.216.34.
Send the same query length-prefixed over TCP; assert the same.
Assert failover happened: entry 0's consecutive_failures >= 1 and backoff_until != null; entry 1's total_successes >= 2. Read them through pool.snapshot.
Send a third query and assert entry 0's call counter did not grow — it is in backoff.
deinit both servers, cancel and await the group, threaded.deinit() — the test must not hang.

S7.3 Acceptance criteria

zig build test -Dintegration runs this test and it passes.
It skips (not fails) under plain zig build test.
It opens no socket outside 127.0.0.1 and reaches no external host.
It completes in bounded time — every wait carries a budget; no unbounded blocking read.
zig fmt --check clean.

Module Layout

src/upstream/transport.zig                    S1  Endpoint, error groups, Client, validateResponse
src/upstream/health.zig                       S1  pure health/backoff state
src/upstream/doh_client.zig                   S2  RFC 8484 over std.http.Client
src/upstream/doh_client_live_test.zig         S2  -Dlive
src/upstream/dot_client.zig                   S3  RFC 7858 over platform/tls_client.zig
src/upstream/dot_client_live_test.zig         S3  -Dlive
src/upstream/pool.zig                         S4  priority failover, per-attempt deadline, health
src/server/handler.zig                        S5  query -> upstream -> validated reply
src/server/udp_server.zig                     S6  UDP/53 listener
src/server/tcp_server.zig                     S6  TCP/53 listener, length-prefixed
src/server/udp_server_integration_test.zig    S6  -Dintegration
src/server/tcp_server_integration_test.zig    S6  -Dintegration
src/server/resolver_integration_test.zig      S7  -Dintegration, end to end

File Ownership

Files	Owner	Notes
`src/upstream/transport.zig`, `src/upstream/health.zig`	S1	frozen after S1 verifies
`src/upstream/doh_client.zig`, `src/upstream/doh_client_live_test.zig`	S2
`src/upstream/dot_client.zig`, `src/upstream/dot_client_live_test.zig`	S3
`src/upstream/pool.zig`	S4
`src/server/handler.zig`	S5
`src/server/udp_server.zig`, `src/server/tcp_server.zig`, both `*_server_integration_test.zig`	S6
`src/server/resolver_integration_test.zig`	S7
`build.zig`, `src/tests.zig`	orchestrator	no session edits these

No session touches milestone 1 or milestone 2 files. A needed change there is reported, not made.

Acceptance Criteria (Milestone 3 Complete)

zig build test exits 0 with every new file wired into src/tests.zig.
zig build test -Dintegration exits 0: the milestone-1 loopback TLS echo, both listener integration tests, and the end-to-end resolver test all pass.
zig build test -Dintegration -Dlive exits 0 locally, or the DoH/DoT live failures are reported as environment findings with the exact error (a live-network failure is not a gate).
zig build cross still produces two statically linked executables.
transport.group is exhaustive over ExchangeError with no else arm.
pool.zig's failover, backoff-skip, all-backed-off probe, local-resource short-circuit and cancellation short-circuit each have a passing named test.
handler.zig's FORMERR (QDCOUNT 0 and 2), NOTIMP, SERVFAIL, drop and UDP-truncation paths each have a passing named test.
zig fmt --check clean repo-wide; GPG-signed lowercase commits.

Anti-Requirements

No caching of any kind (Phase 6). The handler forwards every query.
No filtering, blocklists, rules, or blocked-response synthesis (Phase 5).
No query logging, no SQLite, no storage, no config file or ZON parsing (Phase 4). Timeouts, ports and pool entries arrive as parameters or named constants.
No rate limiting (Phase 6) — src/server/rate_limiter.zig is not created here.
No local DoH or DoT server endpoints (Phase 9). Only upstream DoH/DoT clients are in scope; platform/tls_server.zig is untouched.
No local records, no conditional forwarding, no CNAME uncloaking (Phase 5/7).
No web UI, no REST API, no /metrics. Pool.snapshot exists so Phase 8 has something to read; it is not exposed anywhere yet.
No signal handling, no src/server/shutdown.zig, no main.zig wiring. Servers stop through deinit plus group cancellation; process lifetime is Phase 7's problem.
No DNS name resolution for upstream hosts — DoT endpoints take IP literals, DoH resolution is std.http.Client's business.
No HTTP/2, no DoQ, no connection reuse for DoT, no DNSSEC validation (DO bit passes through).
No 0x20 query-name randomization. Case-insensitive comparison per RFC 4343 is required; generating mixed-case queries is not.
No changes to src/dns/. If a helper is missing there, report it — do not add transport-aware code to the pure core.

As built

The implementation matches the spec with these review-driven refinements (three review rounds; findings went 9 → 3 → 0):

pool.zig: each Entry carries a busy: std.Io.Mutex held for the whole attempt against that entry, so one entry's client (and its buffers/TLS state) is never used by two tasks at once. Pass one re-checks available with a fresh now after acquiring busy (the wait can outlive the health it was admitted under); pass two probes without a re-check by design. Lock order is always busy then Pool.mutex; the pool mutex still guards health and snapshot only.
health.zig: staleness rules are symmetric. The newest outcome is the @max of last_success_at/last_error_at. A stale success counts into totals/window and @max-updates last_success_at, but does not clear consecutive_failures/backoff_until. A stale failure counts into totals/window and @max-updates last_error_at, but only increments consecutive_failures/extends backoff when no newer success exists, and never overwrites a newer failure's error text.
udp_server.zig: each slot's reply buffer is [transport.max_message_len]u8 (not max_datagram) so an oversized upstream reply reaches the handler and becomes a TC=1 reply instead of SERVFAIL (64 slots ≈ 4.4 MiB, inside the PLAN §18 budget). A dropped_handler stat counts .drop outcomes from the handler.
tcp_server.zig: a shutdown flag, set under the slot mutex before the active-connection scan, closes the accept-vs-deinit race — a stream accepted during shutdown is closed immediately instead of claimed.
handler.zig: OPT validation walks every record section. OPT in answer or authority, more than one OPT, a non-root OPT owner, or an unparseable OPT → FORMERR (RFC 6891); only a genuinely absent OPT takes the no-OPT path.
dot_client.zig: handshake ReadFailed/WriteFailed unwrap the stream's stored cause through transport.mapLocal first, and certificate-bundle OutOfMemory is a local resource, not a peer fault.
transport.zig: Endpoint.parse rejects userinfo/query/fragment delimiters (@, ?, #) in the authority, and ?/# in a DoH path.
dot_client.zig (follow-up, tls_name commit): DotClient.init takes a tls_name; it is the SNI and certificate-verification name, while the dial target stays endpoint.host. Empty keeps the endpoint host, which is the behavior described above. The same follow-up fixed a send bug this file had from the start, invisible until a DoT handshake first succeeded: tls.Client.flush only encrypts into the socket writer's buffer and never flushes it, so the query never left the process and the peer eventually closed the connection (ReceiveFailed/EndOfStream). TlsStream.flush now does both flushes and exchange calls it; a hermetic loopback test in tls_client_integration_test.zig covers it.

Addendum (2026-09-12): an upstream episode follows health, and a peer fault carries its cause

Observed on the Pi over three days (nxdns 0.0.17 to 0.0.19): 20 upstream.exchange warning episodes, 17 with one occurrence, 15 resolved within ten seconds; since the 0.0.20 restart 1229 successes, 0 failures, 401 silent stale-session redials. Pool.recordFailure reports an episode on every failed exchange and recordSuccess resolves it on the next success, while health.State trips backoff only at failure_threshold consecutive failures. Two rules over one failure stream, and the Diagnostics page shows the looser one. Second gap: the episode detail is upstream 'tls://1.1.1.1:853' failed: SendFailed; the DoT client unwraps the concrete cause into Transact.Failure.cause and then returns a bare PeerFault error to the pool, so the cause is lost before anything records it. Codex reviewed the design (thread 01a09648) and its eight findings are folded in below.

1. Health owns "failing"; the episode is a projection of its transitions

health.State.recordFailure(at, fault, cfg, rand) and recordSuccess(at, cfg) return an Effect: { revision: u64, state: union(enum) { clear, tripped: Fault } }. revision is a per-state counter incremented by every mutation. The effect is the complete desired state of the episode after the mutation, never a transition: tripped when consecutive_failures >= cfg.failure_threshold (the predicate line 142 already uses, exposed as tripped(cfg); not available(), which turns true when a backoff expires before recovery is proven), clear otherwise. The fault in tripped is the effective last fault the state holds after the mutation, not the incoming one: a stale failure (case 1 at recordFailure) never displaces a newer cause, so the episode detail cannot regress to an older one. A tagged union, not an action enum with a payload field, so a report without a fault is unrepresentable.
The pool projects every effect onto the store: tripped reports, clear resolves. A resolve when nothing is open issues no SQL (storage/events.zig), so the steady state costs nothing. Because every effect carries the whole desired state, last-writer-wins by revision is correct: the pool applies effects per entry in revision order under a separate diagnostics_mutex with a per-entry applied_revision, and drops an effect whose revision is below the applied one. Codex's reordering case (a failure trips health, a newer success clears it and its resolve reaches the store first, then the delayed failure arrives) is dropped instead of opening an episode nothing will resolve; and the converse cases, where a none transition would have overtaken a required report or resolve, cannot occur because there is no none. Health mutates under Pool.mutex as today and the effect is computed there; the store call runs after that mutex is released. Lock order is pool mutex, then nothing; diagnostics mutex, then store mutex. No task holds the pool mutex while it takes either of the other two.
Projection also reconciles a latched store failure for what an entry owns: a clear that failed to write is retried by the next clear the entry projects, and a tripped by the next failure. What no entry owns is handled by the reconciliation points below.
A success that leaves the state tripped (a stale success behind a newer failure, health case 1) projects nothing: recordSuccess returns ?Effect, null in that case. The card is already open from the failure that tripped it, a report on a success would count a successful exchange as an occurrence, and a no-op that advanced the revision would reintroduce the dropped-report hole. Such a success does not touch applied_revision.
Reconciliation at the two points revision order cannot reach. (1) At boot, once the first generation is built: store.resolveExcept(.upstream_exchange, kept = the enabled urls of the pool) closes persisted episodes for upstreams that are gone, disabled or failed to build, and Pool.reconcile(io) projects every entry's current effect, which on a fresh pool resolves the rest. A re-asserted tripped must not count as an occurrence: the store gains ensureOpen, which opens the episode with the detail when none is active and otherwise leaves occurrences and last_seen untouched; reconcile uses it, a recorded failure keeps using report. (2) At every retirement of a displaced generation, both the pinned path (Owner.release at refs == 0, the point after which no exchange of it can still project) and the idle path (Owner.replace handing a zero-ref generation back to the caller), the same two calls run against the current generation. One owner function does the retire-and-reconcile and both paths call it, so a third retirement site cannot forget it. A removed or disabled upstream's episode is closed at the next reconciliation point; if the store had latched a failure at that moment, the next reconciliation point retries it. Before this addendum such an episode was never closed at all. resolveExcept is the right call for this code: the pool is the only reporter of upstream_exchange, unlike configuration.load, whose scoped rule in owner.zig stays. The previous "one exchange of lag" claim is withdrawn; the retirement point is exact.
Occurrences are the count of applied reports. Under concurrent reordering an older report behind a newer one is dropped, so the count can undercount; the health counters on /metrics are exact. Stated, not fixed: exact occurrence counts would need the store to accept out-of-order increments, and nobody reads the count as a metric.
The store folds a repeated report into the open episode (occurrences, last_seen, detail), so failures during backoff raise counts on one card. A warning card now means exactly what the pool means: it stopped trusting this endpoint.
Boundary, stated on purpose: an endpoint failing every other exchange never trips, because every newest success clears the count, so it opens no episode and /api/health stays ok. The rolling success rate on /metrics shows it. Diagnostics is for episodes, not chronic rates; a second predicate for that is not added.
No new configuration. The threshold that exists is the threshold, and it is at least 1: health.Config is validated where it is constructed (a comptime check on the default; if it is user-configurable, the config loader refuses 0), because a tripped state with no recorded failure has no fault to report.

2. A peer fault is a value between the leaf client and the pool

transport.zig gains Fault = struct { kind: PeerFault, cause: anyerror }, Outcome = union(enum) { reply: []u8, fault: Fault }, and a leaf interface Leaf with exchangeFn(ptr, io, query, response_buf) (LocalResource || Cancellation)!Outcome. kind is the taxonomy as today; cause is the concrete unwrapped error (BrokenPipe, ConnectionResetByPeer, TlsAlert, HttpConnectionClosing, ...). No phase field: the taxonomy already names it for every kind that has one, and TlsFailed cannot say where it failed. No HTTP status number for HttpStatus; the claim is limited to concrete I/O causes. No optional metadata bag.
dot_client.zig and doh_client.zig implement Leaf; a Transact.Failure becomes a returned Fault instead of a thrown error. The DoT stale-session redial is unchanged and still counted through reuse_recoveries, never as a fault. The retry list is not widened; with the cause recorded, a stale-session cause outside the four lifecycle errors shows in the next episode and the list grows on evidence.
Pool consumes Leaf for its slots and keeps implementing transport.Client toward the handler, the forward client, and the fakes that sit on that side. The outer error set, transport.group, and the handler are untouched; the pool returns last_fault.kind as the error it returns today. The race harness stays generic: the pool races the leaf through a function whose error set adds Timeout; a leaf never returns error.Timeout itself (its own timeouts are faults), so error.Timeout out of the race is the harness. With race == .expired and !truncated the pool synthesizes Fault{ .kind = error.Timeout, .cause = error.Timeout } and records it: an untruncated attempt expiry is peer evidence. With truncated it stays error.BudgetExhausted, unattributed, exactly as today.
Health stores the effective fault (kind and cause names, error_name_capacity sized for both). Every surface that printed the taxonomy prints one text: <Kind> (cause <Cause>), e.g. SendFailed (cause BrokenPipe). Those surfaces are the episode detail, upstream 'tls://1.1.1.1:853' failed: SendFailed (cause BrokenPipe), Snapshot.last_error and its one reader, the nxdns check FAIL line, and the pool's debug AttemptFailure line. /api/health, /api/upstreams and /metrics carry no fault text today and gain none here.
transport.raceUntilTagged becomes generic over the raced function's own error set: it returns RacedError(f) || error{ Timeout, SystemResources, Canceled }, so the pool's attempt wrapper returns LeafError || error{Timeout} with no @errorCast and no unreachable; error.Timeout out of the race is the harness by type, which sets race = .expired before returning it; the loop's catch asserts that and handles it in the expiry branch only, so a regression of the harness contract traps in ReleaseSafe instead of blaming an endpoint. The compiler enforces at the race boundary that a leaf cannot throw a PeerFault or BudgetExhausted. A leaf's own timeout is a returned Fault{ .kind = error.Timeout, .cause = <leaf cause> } with race == .completed, and a pool test proves it is recorded with the leaf's cause and never confused with the harness expiry.
The blocklist fetcher's last_failure side field stays as it is; same shape, other subsystem, its own addendum.

Tests

health: one failure returns clear; the second returns tripped; a success on a tripped state returns clear; a stale success behind a newer failure projects nothing; a stale failure returns the newer effective fault; revisions strictly increase.
pool: a tripped effect delivered after a newer clear effect is dropped and no episode is open afterwards; a clear delivered after a newer tripped is dropped and the episode stays open; an episode open in the store before the pool exists is resolved by boot reconciliation, one for a url the pool does not have and one for a url it has; a retired generation's late tripped is corrected by the reconciliation at its last release, and the idle-replace path reconciles too; a reconcile of a tripped entry with an open episode moves neither occurrences nor last_seen; a stale success while tripped projects nothing and the occurrence count does not move; one failed exchange opens no episode and the second opens one that the next success resolves; a truncated expiry records nothing and returns BudgetExhausted; an untruncated expiry records Timeout (cause Timeout); a leaf fault of kind Timeout is recorded with the leaf's cause.
clients: the DoT integration tests assert the returned Fault with kind and cause, not the kind alone; a DoH integration test drives DohClient.exchange against a loopback port that is bound but never listens and stays bound for the test's duration, so the connect is refused deterministically with no port reuse race, and asserts the cause survives to the returned Fault; the stub-connection helper tests keep their causes.
surfaces: the nxdns check test asserts the <Kind> (cause <Cause>) text.

Out of scope

An idle timer on the DoT session, a chronic-rate predicate, the fetcher migration, and any change to the DoT retry list.

61 KiB Raw Blame History Unescape Escape

Milestone 3: Resolver Transport