nxdns/PLAN.md

# nxdns — Implementation Plan v3.0 (Zig 0.16.0 Stable)

Source of truth for **nxdns**, a self-hosted DNS sinkhole written in Zig 0.16.0 stable. All stdlib claims in this document are verified against the `0.16.0` tag of the Zig repo (`../zig`).

There is no v1/v2 versioning. Scope is binary: a feature is in scope (and gets built) or out of scope (and does not). "Done" = everything in scope implemented, tested, documented.

## 1. Purpose

Fix three pain points in Pi-hole/AdGuard/Technitium:
1. Clunky, dated admin UI.
2. No config-as-code story (no way to export current state as a text artifact).
3. Silent operational failures (cloudflared proxy-dns incident: broken DoH setup + unbounded error logs filled SD card).

Serves a household LAN (≈2–20 devices). Portfolio-grade public repo with extensive docs, self-hosted on Gitea.

---

## 2. Scope

### 2.1 In Scope

- DNS server for LAN clients: UDP/53, TCP/53, DoH server, DoT server.
- Upstream resolution: DoH (HTTP/1.1), DoT.
- Local DNS records (A/AAAA/CNAME) + conditional forwarding (zone → designated resolver, plain UDP/TCP allowed).
- Domain filtering: blocklists (hosts/domains/ABP, including `@@||name^` exception lines), custom rules (allow/block; exact, parent-walk, wildcard, regex), CNAME uncloaking (depth 8), per-group safe-search rewrite.
- DNS caching: positive + negative, in-memory only.
- Client/group model: IPv4 + IPv6 parity, per-client group assignment, per-group source assignments.
- Query logging + analytics: async batched writes to SQLite (WAL), retention cleanup, dashboard + time buckets, live SSE stream.
- Web app + REST API: LAN/Tailscale admin UI, optional password auth, OpenAPI schema + CI contract tests.
- Observability: upstream health API + UI, disk monitor with UI banner, bounded log rotation, Prometheus `/metrics`.
- Ops: config authority chosen by the invocation (database, or a file named by `--config`), `nxdns export`/`import` (ZON), scheduled + manual blocklist updates, TLS cert watcher + reload, auto-migration on upgrade, systemd service + Dockerfile + compose.

### 2.2 Out of Scope (permanent scope decisions, not deferrals)

- Regex from downloaded blocklists. A list is other people's code running on the household's DNS, and the engine exists for rules the operator wrote. Regex lines in blocklists stay counted and skipped, and the skip count stays surfaced in the UI. (Operator regex rules themselves are **in** scope as of milestone 21: the ReDoS objection that once ruled them out is answered by `filter/regex.zig`, a homegrown Pike VM whose running time is bounded by program length × name length by construction, with no dependency and no backtracking. It is reached only after every hash and wildcard level has missed, and only on a cache miss.)
- ABP syntax beyond domain anchors and `@@||name^` exceptions. `$dnstype`, `$dnsrewrite`, `$client`, `$denyallow` and every browser modifier stay unsupported and counted; the one tolerated modifier is a `$important` suffix on an exception line, which changes nothing about where that exception lands.
- Partial-segment wildcards (`ads*.example.com`) as a rule kind. The regex kind covers the need without a second globbing dialect.
- DHCP server.
- DNSSEC validation (DO bit passthrough only).
- DoQ (QUIC), HTTP/2 upstream transport.
- Clustering / distributed state.
- Project website. The repository, its README and `docs/` are the whole published surface.

---

## 3. Locked Decisions

### 3.1 Toolchain

- Compiler: **Zig 0.16.0 stable**. `build.zig` asserts major/minor.
- Targets: **x86_64-linux-musl** and **aarch64-linux-musl**, both first-class in CI. Release binaries are **static musl** — fully self-contained, no glibc version coupling on the Pi.

### 3.2 Concurrency: `std.Io` (Decision E)

Zig 0.16 stdlib is built on the `std.Io` interface: `std.Io.net` owns networking (`std.net` in its old form is gone) and `std.http.Client` requires an `io: Io`. Therefore:

- **`Io` is the injected platform abstraction.** Every component that does I/O takes `io: Io`. No project-owned wrapper interfaces around it — a second abstraction over an abstraction with one consumer is waste.
- **Backend: `std.Io.Threaded`** — the mature, debuggable path; at ≤20 devices throughput is a non-issue. The originally planned io_uring config flag was dropped in milestone 11: `std.Io.Evented` at Zig 0.16.0 stubs the networking a server needs — listen, accept, connect, lookup, and stream reads/writes return `error.NetworkDown` (`Uring.zig` netListenIp/netAccept/netConnectIp/netRead/netWrite) — so a selectable backend would boot a dead server. The code stays backend-agnostic by construction; revisit when std ships working evented networking.
- No hand-written thread pool. `Io` async/concurrent/Group covers task management.
- Core domain modules (`dns`, `filter`, `cache`) stay pure: no `Io`, no sockets — bytes in, bytes out. Only servers, upstream clients, and storage touch `Io`.

### 3.3 TLS (Decision D)

Verified: 0.16.0 ships `std.crypto.tls.Client` only. There is no server-side TLS in the stdlib.

- **Upstream client TLS (DoH/DoT): stdlib** `std.crypto.tls` via `std.http.Client` / direct. Maintained upstream, exercised by the stdlib itself.
- **Local server TLS (DoH/DoT termination): vendored mbedTLS.** Audited, embedded-focused, compiles cleanly with the Zig build system for both targets. We do not hand-roll a TLS server in a security-sensitive service.
- TLS glue lives in `platform/tls_client.zig` (stdlib wrapper: error classification, deadlines, retry policy) and `platform/tls_server.zig` (mbedTLS wrapper exposing `std.Io.Reader`/`Writer` so `std.http.Server` composes on top unchanged).

### 3.4 C Dependency Policy

- **`sqlite3` + `mbedtls`, both vendored** and compiled from source in `build.zig`. Pinned versions, hermetic cross-compilation, no system libs.
- SQLite: amalgamation + our own thin wrapper (`storage/db.zig`: open/close, prepared-statement cache, typed row mapping, error translation). No third-party binding — the needed surface is small enough to own (Decision G).
- Zig package deps: minimized; anything adopted is pinned in `build.zig.zon`. No TOML/regex/HTTP packages needed under current decisions.

### 3.5 Config Format + Truth Model (Decision F)

- **The invocation picks truth (see the next point). Config file format is ZON** (`std.zon` parse + stringify — typed parsing into config structs, exact round-trip, stdlib-maintained, comments supported). No TOML: a third-party parser plus a hand-written serializer is two failure surfaces in the correctness-critical round-trip path, bought for syntax familiarity.
- The invocation picks the authority, and nothing else does (m20): `nxdns run` makes the database the configuration, `nxdns run --config FILE` makes the file the sole source and refuses the API routes that would edit configuration. There is no seeding and no first-start special case.
- `nxdns export [--out file.zon]` dumps DB state as canonical ZON. `nxdns import <file.zon>` validates + replaces DB contents (`--force` if the DB holds configuration; client rows materialised from traffic do not count, and survive the replacement. First-seen/last-seen are runtime state, not configuration: they follow the address, so an import never restamps a device the DB already knew). Export/import = backup + host migration, **not** upgrades (§3.7).
- No file watcher, no auto-regeneration.

### 3.6 Storage Layout (Decision H)

Two SQLite files with opposite write profiles, isolated from each other:

- **`config.db`** — small, precious, rarely written: groups, clients, prefixes, upstreams, blocklist source metadata, rules, local records, forward zones, settings, schema version.
- **`querylog.db`** — high-churn, large, expendable: query log + its own private `domains` dimension table. Client identity stored as **IP text**, not a FK into config — log rows are immutable facts and must not point at mutable config rows. If `querylog.db` is missing or corrupt at startup, rename aside, recreate, keep serving. Log loss is not an outage.
- No cross-DB references. Retention/VACUUM churn never touches `config.db`; config backup is a copy of a tiny file.

### 3.7 Upgrades: Auto-Migration (Decision J)

- `config.db`: numbered, sequential SQL migration steps compiled into the binary. At startup: read schema version row, apply newer steps inside a transaction, continue. Operator upgrade = install binary, restart. Before v0.1 the list holds one step — the baseline of §11.2, edited in place — because nxdns has no installs and a step exists only to reconcile a database somebody already has.
- `querylog.db`: **no migrations.** On schema mismatch: rename aside, recreate fresh.

### 3.8 Blocklist Storage (Decision A)

Blocklist domains are **not** stored in SQLite — they are a cache of re-downloadable remote artifacts, not config or state:

- Each source compiles to `/var/lib/nxdns/blocklists/<source_id>.list`: normalized, one domain per line, small header (source URL, fetch time, counts, checksum). Wildcard/regex/exception-flavored lines: wildcards go to `<source_id>.wild`, ABP exceptions (`@@||name^`) to `<source_id>.allow`; regex lines and browser-syntax lines nxdns cannot translate into a DNS decision are counted + skipped, both counts in metadata → UI. The checksum covers the three bodies in that order, each followed by a separator byte so that moving a name between bodies — an upstream switching `a.example` to `*.a.example` — changes the digest and forces a republish.
- `config.db` keeps source **metadata only** (`blocklist_sources`).
- Startup + post-update: parse files into the immutable in-memory matcher (RCU swap, §9.5).
- Corruption recovery is per-file: checksum mismatch → re-download one list.

### 3.9 Rule Model (Decision B)

Rule kinds: `exact`, parent-walk (implicit via candidate chain), `wildcard` (`*` segment patterns, e.g. `*.doubleclick.net`, `ads.*.example.com`), `regex` (the linear-time engine of `filter/regex.zig`, matched unanchored against the whole normalized name). Actions: `allow` | `block`. Kind is an explicit column, so a fourth kind widens one `CHECK` and touches no other table. A regex pattern is stored exactly as written — it is not a name, so it is never lowercased or dot-stripped — and is compiled at both edges: `config/validate.zig` refuses a bad one with the limit it hit, and `filter/rules.zig` compiles it once per snapshot.

### 3.10 Filtering Precedence

1. Exact/parent **allow** rules
2. Exact/parent **block** rules
3. Wildcard allow rules
4. Wildcard block rules
5. Regex allow rules
6. Regex block rules
7. Blocklist exceptions (`@@||name^`)
8. Blocklist domains
9. Blocklist wildcards

Tie-break at same specificity: **allow wins**.

Each level is checked against the whole candidate chain before the next level is checked against any, which is what makes an allow rule on a parent beat a block rule on the child.

Two positions carry an argument rather than a preference. The regex levels come last among the operator rules because they are the only ones that are not a set lookup or a label walk: a regex runs only once every cheaper level has missed. Blocklist exceptions come below **every** operator level because a downloaded list may cancel what another list blocked and must never cancel what the operator decided — no list can open an allow hole the operator did not open.

### 3.11 Network Posture

- Default web bind: LAN/Tailscale-friendly (non-loopback allowed).
- Auth optional (empty `web.password` disables it). Set → argon2id-hashed password, session cookie (`HttpOnly`, `SameSite=Lax`, `Secure` on HTTPS).
- API per-IP rate limit on by default.

### 3.12 Addressing

IPv4 + IPv6 full parity for: client identity, rate limiting, logging, group assignment, prefix-based group matching. Dual-stack assumed.

### 3.13 Filesystem Layout (FHS)

- `/etc/nxdns/config.zon` — the declarative source, read only when `run --config` names it. A file no flag names changes nothing.
- `/var/lib/nxdns/config.db`, `/var/lib/nxdns/querylog.db`
- `/var/lib/nxdns/blocklists/*.list|*.wild|*.allow` (plus `*.raw.tmp|*.list.tmp|*.wild.tmp|*.allow.tmp` during a refresh)
- `/var/log/nxdns/nxdns.log` — only in file output mode; default is stderr → journald.

### 3.14 Frontend Stack (Decision I)

- Vite + React + TypeScript + StyleX + React Aria; TanStack Router + TanStack Query. SPA, no SSR.
- **Built assets embedded in the binary** at compile time: single self-contained artifact, no asset-path config, no binary/UI skew. `zig build` accepts the dist path; the release pipeline runs `npm run build` first (CI always does).
- Dev-mode flag serves assets from disk so UI iteration needs no Zig rebuild.

### 3.15 CI

- Self-hosted **Gitea + Gitea Actions runner**.
- Every blocking check lives in one reusable workflow, `.gitea/workflows/gates.yml`. `ci.yml` calls it on `master` pushes and pull requests; `release.yml` calls the same file before it publishes, so a release cannot skip a check that CI runs.
- Jobs: `zig build test`, fuzz smoke, integration tests, OpenAPI contract tests (live server validated against `openapi.yaml`), frontend build and tests, `dist` + `verify-dist` for both targets, and a container job that builds the image and asserts its binaries are byte-identical to the packaged ones. aarch64 test execution via qemu-user if the runner is x86_64.

---

## 4. Architecture Overview

```
Client DNS Query
   |
   v
[UDP / TCP / DoH / DoT servers]        (std.Io; mbedTLS terminates DoH/DoT)
   |
   v
[Request Handler]
   |- rate limit check
   |- parse DNS packet
   |- group resolution (v4/v6 parity)
   |- local records check          -> answer locally
   |- forward-zone check           -> forward to designated resolver (bypasses filtering)
   |- allow/block evaluation (rules + blocklists, per group)
   |- safe-search rewrite (per group)
   |- cache lookup
   |- upstream query (DoH/DoT pool) + health tracking
   |- CNAME uncloaking check
   |- cache store
   |- async query log + SSE fanout
   v
Response to client
```

Cross-cutting: `ConfigManager` (load/reconcile/import/export/settings), `BlocklistManager` (fetch/compile/swap), `Storage` (two SQLite DBs), `Cache`, `UpstreamHealth`, `DiskMonitor`, `Auth`, `Web API` (REST + SSE + metrics).

---

## 5. Source Layout

```
src/
  main.zig               # Io backend instantiation, wiring
  app.zig
  version.zig

  platform/
    address.zig          # NetAddress: v4/v6 parity type + canonical key (was net_adapter)
    tls_client.zig       # stdlib TLS wrapper: deadlines, error classes, retry policy
    tls_server.zig       # mbedTLS wrapper exposing std.Io.Reader/Writer

  dns/                   # pure: bytes in, bytes out; no Io
    types.zig  header.zig  name.zig  question.zig  record.zig  edns.zig  packet.zig

  server/
    udp_server.zig  tcp_server.zig  doh_server.zig  dot_server.zig
    handler.zig  rate_limiter.zig  shutdown.zig

  upstream/
    doh_client.zig  dot_client.zig  pool.zig  health.zig

  filter/                # pure
    matcher.zig  rules.zig  wildcard.zig
    parser_hosts.zig  parser_domains.zig  parser_abp.zig
    fetcher.zig  compiler.zig            # list download -> compiled .list/.wild/.allow files
    regex.zig                            # linear-time Pike VM for operator regex rules
    safesearch.zig

  local/                 # pure
    records.zig          # local A/AAAA/CNAME answers
    forward_zones.zig    # zone -> designated resolver matching

  cache/
    dns_cache.zig

  storage/
    db.zig               # sqlite3 thin wrapper
    config_schema.zig  migrations.zig
    querylog_schema.zig
    repositories/
      clients_repo.zig  groups_repo.zig  rules_repo.zig  sources_repo.zig
      queries_repo.zig  settings_repo.zig  upstreams_repo.zig  local_repo.zig
    logger.zig  retention.zig  disk_monitor.zig

  config/
    model.zig  loader.zig  reconcile.zig  import.zig  export.zig    # all ZON via std.zon
    validate.zig  faults.zig

  web/
    server.zig  router.zig  auth.zig  sse.zig  static.zig  metrics.zig  openapi.zig
    handlers/
      auth.zig  stats.zig  queries.zig  clients.zig  groups.zig  blocklists.zig
      rules.zig  local.zig  lookup.zig  pause.zig  settings.zig
      upstream_health.zig  certs.zig  health.zig  version.zig

admin/                   # Vite + React + TS + StyleX + React Aria + TanStack
vendor/                  # sqlite3 amalgamation, mbedtls (pinned)
docs/                    # tutorial/ how-to/ reference/ explanation/ (Diátaxis)
tests/                   # dns/ integration/ fuzz/
```

---

## 6. DNS Behavior

### 6.1 Protocol

- RFC 1035 parse/encode. EDNS OPT passthrough. DO bit passthrough (no validation). ECS: `strip` (default) or `forward`.
- Process first question only. Preserve query ID and RD flag.
- Malformed: `FORMERR` when a response is possible; silent drop for severely truncated.

### 6.2 Blocked Response Modes

`blocking.response`: `zero` (A=`0.0.0.0`, AAAA=`::`) or `nxdomain`. TTL = `blocking.ttl`.

### 6.3 CNAME Uncloaking

Walk chain to depth 8; any target hitting block logic → synthesize blocked response.

### 6.4 Local Records

- Table-backed A/AAAA/CNAME records, group-independent. Matched before filtering.
- CNAME targets resolve through the normal pipeline (uncloaking rules apply).

### 6.5 Conditional Forwarding

- `forward_zones`: suffix match (`lan.home`, `10.in-addr.arpa`, …) → designated resolver (`udp://192.168.1.1:53` style; plain UDP/TCP permitted — these are local infra resolvers).
- Matching queries bypass filtering and blocklists; responses cached normally.

---

## 7. Filtering Engine

### 7.1 Evaluation

For `{domain, group_id}` (the qtype travels with the query for logging and response synthesis, not for matching):
1. Normalize: lowercase, trim trailing dot.
2. Build candidate chain (full, parent1, parent2, …).
3. Explicit rules per §3.10 precedence, allow before block at each level: exact rules against every candidate in the chain, then wildcard patterns and then regex patterns against the whole name (both kinds express their own reach, so neither walks the chain).
4. Group's blocklist exceptions (`@@` entries), against every candidate in the chain. They cancel a block a list made and never one a rule made.
5. Group's blocklist domains (hash set over compiled lists), matched against the query name only.
6. Group's blocklist wildcards, matched against every proper parent of the query name.
7. No match → allow.

Blocklist *domain* entries do not parent-walk: they are matched against the query name alone. Wildcard entries match every proper parent, and exception entries walk the candidate chain the way rules do (§3.9), so `@@||good.ads.example^` also lifts `y.good.ads.example`. ABP `||x.y^` emits both a domain entry `x.y` and a wildcard entry `x.y`, which together give domain-and-subdomains semantics.

### 7.2 Group Assignment

- Per source IP: (1) exact match in `clients`, (2) longest-prefix match in `client_prefixes` (ties: longer prefix, then priority), (3) `default` group.
- Auto-materialization: first query from an unseen IP inserts a `clients` row (`hand_edited=0`, `first_seen=now`) for UI visibility and stable group assignment. The query log does **not** FK to it (§3.6).
- Materialized clients name themselves: the tracker's flush pass sends one PTR query per unnamed row through the declared forward zones (§6.5), and the answer is runtime state in `learned_name`, never configuration.
- Retention drops `hand_edited=0` clients with no queries in `retention_days`.

### 7.3 Reload

New immutable matcher built from DB + compiled list files → swap under an `std.Io.RwLock` with a generation counter. Readers take the shared lock for the microseconds of one evaluate; the writer takes the exclusive lock only for the swap, and the source status table is installed in the same critical section, so a failed reload publishes neither. (Deliberate deviation from "readers lock-free": freeing the old snapshot without a lock needs epoch-based reclamation, unjustifiable at household scale — see specs/milestone-5.md S8.3.)

### 7.4 Safe-Search

Per-group boolean. Rewrites known engine domains to their safe-search CNAME targets.

---

## 8. Cache

- Key: normalized qname + qtype + qclass + DO bit + forwarded ECS subnet (only when `ecs_mode=forward`).
- Value: full response bytes, `stored_at`, `expires_at`, positive/negative flag.
- Positive TTL: `min(answer TTLs)` bounded by sanity ceiling. Negative: SOA/minimum capped by `cache.negative_ttl_max` (0 disables).
- Hit path: clone bytes, overwrite transaction ID, decrement visible TTLs; TTL ≤ 0 → miss.
- Bounded by `cache.size`; CLOCK eviction; opportunistic + periodic expiry sweep.
- **In-memory only, no persistence.** Cold start refills within a minute; SD-corruption risk isn't worth pre-warmed entries.

---

## 9. Upstream Resolution

- Schemes: `https://…` → DoH, `tls://host:853` → DoT.
- Ordered by priority; sequential attempt; per-upstream failure counters; exponential backoff with jitter; success resets.
- `UpstreamHealth` per upstream: last_success_at, last_error_at, last_error_message, rolling success rate, consecutive failures, backoff-until. This is routing state: it drives failover and backoff, and is exposed through `/metrics` and `nxdns check`. `GET /api/upstream/health?period=…` exposes none of it except the live `enabled`/`available` pair; its counts, success rate and last failure are ranged aggregates read from the per-minute upstream history in `querylog.db`, so the dashboard's period scopes them like every other number on the page.
- DoH client: `std.http.Client` with `content-type/accept: application/dns-message`; strict status + payload checks.
- `platform/tls_client.zig` enforces per-connection read/write deadlines, classifies TLS errors explicitly, retries with backoff. Integration tests cover timeout/hang scenarios so compiler upgrades can't silently regress them.
- Connect, read, and total-budget timeouts each configurable.

---

## 10. Rate Limiting

- DNS: default 1000 req / 60s per client IP (v4/v6 keyed alike); exceeded → `REFUSED`.
- API: per-IP token bucket, on by default; separate SSE connection cap per IP; localhost relaxed (configurable).
- Limiter maps bounded; stale-key windowed sweep.

---

## 11. Storage

### 11.1 Pragmas (both DBs)

`journal_mode=WAL`, `synchronous=NORMAL`, `foreign_keys=ON`, `busy_timeout` set.

### 11.2 config.db Schema (v1 baseline)

The DDL below is the live schema, kept byte-identical to `src/storage/config_schema.zig`. `src/storage/migrations.zig` carries it as its one and only step, so a database is at **version 1** or it does not exist.

Until nxdns reaches v0.1 this baseline is **editable**: a schema change edits this section and `config_schema.zig` together and adds no migration step. nxdns has no installs, so there is no database for a step to reconcile. At v0.1 the baseline freezes and every later change becomes an append-only step.

```sql
CREATE TABLE schema_version (version INTEGER NOT NULL);

CREATE TABLE groups (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL UNIQUE,
    safe_search INTEGER NOT NULL DEFAULT 0
);
INSERT OR IGNORE INTO groups (id, name) VALUES (1, 'default');

CREATE TABLE clients (
    id INTEGER PRIMARY KEY,
    ip TEXT NOT NULL UNIQUE,            -- canonical text form (v4 dotted / v6 RFC 5952)
    name TEXT,
    learned_name TEXT,
    name_attempt_after INTEGER NOT NULL DEFAULT 0,
    group_id INTEGER NOT NULL REFERENCES groups(id),
    hand_edited INTEGER NOT NULL DEFAULT 0,
    first_seen INTEGER NOT NULL,
    last_seen INTEGER NOT NULL
);

CREATE TABLE client_prefixes (
    id INTEGER PRIMARY KEY,
    prefix TEXT NOT NULL UNIQUE,        -- "192.168.1.0/24", "fd00:abcd::/48"
    group_id INTEGER NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
    priority INTEGER NOT NULL DEFAULT 100
);

CREATE TABLE upstreams (
    id INTEGER PRIMARY KEY,
    url TEXT NOT NULL UNIQUE,
    priority INTEGER NOT NULL DEFAULT 100,
    enabled INTEGER NOT NULL DEFAULT 1,
    tls_name TEXT NOT NULL DEFAULT ''  -- DoT verification name; empty verifies against the url host
);

CREATE TABLE blocklist_sources (
    id INTEGER PRIMARY KEY,
    url TEXT NOT NULL UNIQUE,
    name TEXT NOT NULL,
    enabled INTEGER NOT NULL DEFAULT 1,
    is_suggested INTEGER NOT NULL DEFAULT 0,
    last_updated INTEGER,
    domain_count INTEGER NOT NULL DEFAULT 0,
    wildcard_count INTEGER NOT NULL DEFAULT 0,
    exception_count INTEGER NOT NULL DEFAULT 0,
    skipped_regex_count INTEGER NOT NULL DEFAULT 0,
    skipped_unsupported_count INTEGER NOT NULL DEFAULT 0,
    checksum TEXT
);

CREATE TABLE group_sources (
    group_id INTEGER NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
    source_id INTEGER NOT NULL REFERENCES blocklist_sources(id) ON DELETE CASCADE,
    PRIMARY KEY (group_id, source_id)
);

CREATE TABLE rules (
    id INTEGER PRIMARY KEY,
    group_id INTEGER NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
    pattern TEXT NOT NULL,
    kind TEXT NOT NULL CHECK(kind IN ('exact','wildcard','regex')),
    action TEXT NOT NULL CHECK(action IN ('allow','block')),
    created_at INTEGER NOT NULL
);

CREATE TABLE local_records (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    rtype TEXT NOT NULL CHECK(rtype IN ('A','AAAA','CNAME')),
    value TEXT NOT NULL,
    ttl INTEGER NOT NULL DEFAULT 300,
    UNIQUE(name, rtype, value)
);

CREATE TABLE forward_zones (
    id INTEGER PRIMARY KEY,
    zone TEXT NOT NULL UNIQUE,
    resolver TEXT NOT NULL              -- "udp://192.168.1.1:53"
);

CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT NOT NULL);

CREATE TABLE operational_events (
    id INTEGER PRIMARY KEY,
    code TEXT NOT NULL,
    subject_key TEXT NOT NULL,
    subject_label TEXT NOT NULL,
    severity TEXT NOT NULL CHECK (severity IN ('warning', 'error')),
    first_seen INTEGER NOT NULL,
    last_seen INTEGER NOT NULL,
    occurrences INTEGER NOT NULL CHECK (occurrences > 0),
    resolved_at INTEGER,
    detail TEXT NOT NULL DEFAULT '',
    CHECK (resolved_at IS NULL OR resolved_at >= first_seen)
);
CREATE UNIQUE INDEX idx_operational_events_active
    ON operational_events(code, subject_key) WHERE resolved_at IS NULL;
CREATE INDEX idx_operational_events_last_seen
    ON operational_events(last_seen DESC);
```

`operational_events` is the one table here that is **not** configuration. It is the diagnostics log of `src/storage/events.zig`: one row per failure episode, opened on the first failure and resolved when the same subject succeeds again. It is deliberately absent from `config_schema.table_names` and `config_schema.delete_order`, so `nxdns export` never emits it and `nxdns import` never wipes it.

### 11.3 querylog.db Schema

```sql
CREATE TABLE domains (
    id INTEGER PRIMARY KEY,
    domain TEXT NOT NULL UNIQUE
);

CREATE TABLE query_log (
    id INTEGER PRIMARY KEY,
    timestamp INTEGER NOT NULL,
    domain_id INTEGER NOT NULL REFERENCES domains(id),
    client_ip TEXT NOT NULL,            -- text, not a FK: log rows are immutable facts
    qtype INTEGER,
    blocked INTEGER NOT NULL,
    block_reason TEXT,
    response_time_us INTEGER,
    cache_hit INTEGER,
    upstream TEXT
);
CREATE INDEX idx_query_log_ts ON query_log(timestamp);
CREATE INDEX idx_query_log_client ON query_log(client_ip);
CREATE INDEX idx_query_log_domain ON query_log(domain_id);

CREATE TABLE upstream_targets (
    id INTEGER PRIMARY KEY,
    url TEXT NOT NULL UNIQUE   -- the historical identity: config.db ids cannot cross database files
);

CREATE TABLE upstream_minute (
    upstream_id INTEGER NOT NULL REFERENCES upstream_targets(id),
    minute_ts INTEGER NOT NULL,
    successes INTEGER NOT NULL,
    failures INTEGER NOT NULL,
    last_failure_ts INTEGER,
    last_error TEXT,
    PRIMARY KEY (upstream_id, minute_ts),
    CHECK (successes >= 0),
    CHECK (failures >= 0)
) WITHOUT ROWID;
CREATE INDEX idx_upstream_minute_ts ON upstream_minute(minute_ts);
```

### 11.4 Query Logger

- In-memory buffer, mutex guarded, hard cap `query_log_buffer_max` (default 10000).
- Flush: batch size (100, comptime) or max interval `query_log_flush_interval_s` (default 60s, 0–3600, 0 = do not wait). One transaction per interval: at household query rates a per-query commit costs orders of magnitude more disk writes than the rows are worth. The interval is also roughly what a crash costs, while the writer is healthy and the disk gate is open — a gated or lock-delayed batch is older, so it is a normal case, not a bound.
- Privacy transforms (hide_domains / hide_client_ips) applied before persist + SSE fanout.
- Backpressure: buffer full → drop oldest unflushed entry, increment monotonic `queries_dropped` (exposed in `/api/health` + `/metrics`). SSE fanout precedes buffer insert, so live viewers still see dropped-from-persistence entries.

### 11.5 Retention

Periodic delete of rows older than `retention_days`; scheduled checkpoint/VACUUM on `querylog.db` only.

### 11.6 Disk Discipline (cloudflared lesson)

- Log output default: **stderr** (journald rotates). File mode: rotate at `log.max_size_mb` (default 50), keep `log.max_files` (default 5).
- `DiskMonitor` samples DB + log-dir sizes every 60s:
  - free < `disk.warn_free_mb` → UI banner + `/api/health` degraded.
  - free < `disk.min_free_mb` → stop non-essential writes (blocklist updates, log flushes); buffer continues under backpressure policy.
- Upstream error log lines deduplicated per-upstream at 1/minute.

---

## 12. Configuration

### 12.1 Config ZON Shape

The canonical shape is not duplicated here. It lives in [docs/reference/configuration.md](docs/reference/configuration.md), which is handwritten against `config/model.zig` and only partly guarded (the drift test covers settings-key rows, not the whole shape, so a new collection can go undocumented while the guard stays green), and `nxdns export` emits it. A copy in this document is how §12.1 came to describe an `.upstream.servers` field that never existed and to omit the required `.groups` and `.upstreams` — a sample nobody could load. The skeleton, for orientation only:

```zon
.{
    .groups = .{ .{ .name = "default" } },
    .upstreams = .{ .{ .url = "https://dns.example/dns-query" } },
    .rules = .{
        .{ .group = "default", .pattern = "^ad[0-9]+-", .kind = .regex, .action = .block },
    },
}
```

### 12.2 Validation

At least one upstream; ports in range; resolver URLs parseable. `nxdns check` runs the validator, probes upstreams, and loads the cert+key pair of each enabled DoH/DoT server through the same `CertStore.init` the listeners use.

### 12.3 Settings Semantics

Scalars in `settings(key, value)`; ordered/structured items in dedicated tables. Restart-required settings flagged; UI shows "restart to apply" banner.

---

## 13. Web / API

### 13.1 Endpoints

- `POST /api/auth/login`, `POST /api/auth/logout`
- `GET /api/stats?period=…`, `GET /api/stats/timeseries?period=…`
- `GET /api/queries` (filter + paginate), `GET /api/queries/live` (SSE, per-IP cap)
- `GET/PUT /api/clients/{id}`
- `GET/POST/PUT/DELETE /api/groups…`, `/api/blocklists…`, `/api/rules…`, `/api/local-records…`, `/api/forward-zones…`
- `POST /api/blocklists/update`
- `GET /api/lookup?domain=…&group_id=…`
- `GET/POST /api/pause`
- `GET/PUT /api/settings`
- `GET /api/upstream/health?period=…`
- `POST /api/certs/reload`
- `GET /api/health` — overall + disk + upstream + queries_dropped rollup
- `GET /metrics` — Prometheus text exposition: query counters (total/blocked/cached), per-upstream health, cache stats, queries_dropped, disk gauges
- `GET /api/version`, `GET /api/openapi.yaml`

### 13.2 OpenAPI

Hand-maintained `openapi.yaml`, served at `GET /api/openapi.yaml` and mirrored by hand in `docs/reference/api.md`. No renderer is vendored. Two drift guards fail the build instead: `src/web/openapi.zig` asserts every served route appears in the spec, and `src/docs_drift_test.zig` asserts every served route has its own table row in the reference page.

---

## 14. Frontend

Pages: Dashboard (stats + upstream health + disk), Query log, Live log, Clients, Groups, Blocklists, Rules, Local DNS (records + forward zones), Domain lookup, Settings.

Requirements: responsive desktop/mobile; route loaders for initial fetch; TanStack Query for cache/retries; error/loading states on every data view; works with auth enabled or disabled; restart-required banner.

---

## 15. CLI

- `nxdns run` — start the server.
- `nxdns check` — validate config, probe upstreams, load each enabled listener's certificate and verify its key pairs with it; exit 2 on failure, 0 with warnings.
- `nxdns export [--out file.zon]`
- `nxdns import <file.zon> [--force]`
- `nxdns version` — app version, Zig version string, git commit. No build date: the version and the commit identify a build exactly, and a date is one more input a reproducible build would have to pin.

---

## 16. Implementation Order

### Phase 0 — Build Baseline
Scaffold tree; `build.zig` with 0.16 assertion, musl targets, vendored sqlite3 + mbedtls compiling; version plumbing; Gitea Actions workflows (test, integration, fuzz smoke, OpenAPI lint, frontend build). Exit: cross-compiled hello-world linking both C deps on both targets; CI green.

### Phase 1 — Platform Layer
`platform/address.zig` (v4/v6 parity + canonical keys); `platform/tls_client.zig` (deadlines, error classes); `platform/tls_server.zig` (mbedTLS handshake → `std.Io.Reader`/`Writer`). Exit: UDP echo over `std.Io`; TLS client handshake against a real host; mbedTLS server terminating a loopback TLS connection.

### Phase 2 — DNS Core
Types/header/name/question/record/packet; parser + encoder tests + fuzz target; EDNS + DO passthrough. Exit: unit + fuzz smoke pass.

### Phase 3 — Resolver Transport
UDP server, TCP server, DoH + DoT upstream clients, pool + failover/backoff + health. Exit: A/AAAA forwarding over UDP + TCP; health populated.

### Phase 4 — Storage + Config
SQLite wrapper; config.db schema + migration runner; querylog.db schema + recreate-on-mismatch; repositories; ZON loading + import/export; `nxdns check`. Exit: export → import round-trips byte-stable. (The ZON bootstrap this phase shipped was replaced in m20 by the two authority modes above.)

### Phase 5 — Filtering + Local DNS
Rule matcher (exact/parent/wildcard; `regex` added in m21); blocklist parsers → compiled file format; fetcher + scheduled update; RCU swap; per-group safe-search; local records; forward zones. Exit: precedence table validated by tests; local zone answers + conditional forwards work.

### Phase 6 — Cache + Rate Limit + Logging + Disk Monitor
TTL cache + reconstruction; v4/v6 rate limiter; async query logger + backpressure + retention; DiskMonitor + rotation + error-log dedup. Exit: disk thresholds trigger degradation + drop counters in integration test.

### Phase 7 — Handler Integration
Full pipeline composition; CNAME uncloaking; pause/resume. Exit: end-to-end DNS flow with blocking, local records, cache, failover.

### Phase 8 — Web / API / SSE / Auth / Metrics
HTTP server + router; handlers; SSE; optional auth; API rate limiting; `/metrics`; OpenAPI served + contract tests; embedded frontend + dev-mode disk serving. Exit: frontend fully drives config and operations; contract tests green.

### Phase 9 — Local DoH/DoT Endpoints
DoH server + DoT server on `platform/tls_server.zig`; cert watcher + reload. Exit: LAN client resolves via DoH and DoT against local certs.

### Phase 10 — Packaging + Ops + Docs
systemd unit (`AmbientCapabilities=CAP_NET_BIND_SERVICE`, hardened, writable `/var/lib/nxdns` + optional `/var/log/nxdns`); Dockerfile + compose (53/udp+tcp, 8080; mounts `/etc/nxdns`, `/var/lib/nxdns`); operator/architecture/config-reference/API docs. Exit: documented deployment works end-to-end on the Pi 5.

---

## 17. Testing Strategy

- **Unit**: DNS encode/decode; rule precedence + wildcard matcher; cache put/get/TTL rewrite; ZON loading + export round-trip; rate limiter; migration runner (fresh + stepwise upgrade).
- **Fuzz**: DNS parser malformed-packet fuzzing; blocklist parser fuzzing.
- **Integration**: UDP/TCP query path; blocked path; allow-over-block; wildcard precedence; CNAME uncloaking block; local records + forward zones; upstream failover/backoff/health; disk-full degradation; querylog.db corruption recovery; API CRUD; auth on/off; SSE; contract tests.
- **Manual**: `dig @pi example.com` / blocked domain / local record; DoH/DoT client checks; dashboard + live log.

---

## 18. Performance Targets

- Sustained ≥ 100 qps on Raspberry Pi 5.
- Blocklist lookup p95 < 1 ms.
- Cached response p95 < 5 ms.
- Memory with ~1M blocked domains < 100 MiB.
- Stripped static binary per arch: < 15,728,640 bytes with the embedded frontend assets, < 10,485,760 bytes without them. `zig build verify-dist` asserts both.

---

## 19. Security

- Sanitize all DNS and API inputs; prepared statements everywhere.
- argon2id password hashing; random session tokens with expiry.
- No secrets in logs. TLS keys readable by service user only.
- systemd hardening + least privilege.
- mbedTLS + sqlite3 versions pinned; upgrades are deliberate, reviewed bumps.

---

## 20. Publication

The project publishes released binaries and container images from its own Gitea instance. Building from source stays fully supported and documented; it is no longer the only path.

- **Trigger.** Pushing an annotated, GPG-signed tag `vX.Y.Z` to `git.mial.net/mokhtar/nxdns`. Nothing else publishes. Pre-release tags are rejected.
- **Version.** The tag is authoritative. `build.zig.zon`'s `.version` must equal the tag, and the packaging gate asserts it. Nowhere else stores a version.
- **Artifacts.** Per architecture (`x86_64-linux-musl`, `aarch64-linux-musl`) a `.tar.gz` holding the stripped ReleaseSafe binary, `LICENSE`, `THIRD-PARTY-NOTICES` and the README. Plus one `SHA256SUMS` covering both tarballs and the image digest, and one detached `SHA256SUMS.asc`.
- **Images.** One multi-architecture image at `git.mial.net/mokhtar/nxdns`, tagged with the exact version and `latest`. No `:edge`. `latest` moves only forward. The builder stage runs on `$BUILDPLATFORM`, so no release build needs qemu.
- **Signing.** A GPG signing subkey held only by Gitea signs `SHA256SUMS`. The tag itself is signed by the primary key, and the release job pins that primary fingerprint before it does anything else. Verification is documented in `docs/how-to/verify-a-release.md`.
- **Gates.** `release.yml` runs `gates.yml` — the same file CI runs — and publishes nothing if any gate fails.
- **Licensing.** EUPL-1.2. `THIRD-PARTY-NOTICES` is generated from `licenses/inventory.zon` and a drift test fails when a dependency changes without a matching notice entry.
- **Deferred.** Bit-for-bit reproducibility across machines, and SBOM generation, are deliberate deferrals. See `specs/milestone-14.md` ruling 12.

---

## 21. Success Criteria

1. `nxdns run` starts cleanly on Zig 0.16.0 stable, static musl, both arches.
2. UDP + TCP resolution works; blocked domains return the configured response; precedence per §3.10.
3. Local records answer; conditional forwarding works.
4. Cache hit path returns valid ID-adjusted responses.
5. Upstream failover + backoff works; health in UI, API, `/metrics`.
6. Disk-fill degrades gracefully; no silent log-flood failure mode.
7. Web UI + API provide full admin functionality; OpenAPI contract tests green.
8. `nxdns export` round-trips via `nxdns import`.
9. Query logging, stats, SSE live stream work; querylog.db corruption self-heals.
10. Local DoH + DoT endpoints serve LAN clients.
11. Schema upgrade = install + restart (migration test proves it).
12. All suites green in Gitea CI for both targets.
13. Operator, architecture, config-reference, and API docs complete.

---

## 22. Working Notes

- `dns/`, `filter/`, `local/`, `cache/` stay pure (no `Io`, no sockets). Servers, upstream clients, and storage take `io: Io`.
- Verify stdlib behavior against `../zig` (tag 0.16.0) instead of memory — the std.Io migration invalidated older knowledge once already.
- Keep OpenAPI in sync with handlers (CI enforced).
- When a plan decision and reality diverge during build: surface 2–3 alternatives with tradeoffs, get a call, then execute.

## Decision Log

| # | Decision |
|---|----------|
| A | Blocklists compile to flat files under `/var/lib/nxdns/blocklists/`; DB stores source metadata only |
| B | Rule kinds: exact, parent-walk, wildcard, regex (m21, own linear-time engine). Regex *from downloaded lists* stays out of scope |
| C | In scope: local DoH/DoT server, local records, conditional forwarding. Out: HTTP/2, DoQ, DHCP, DNSSEC, clustering |
| D | mbedTLS (vendored) terminates server TLS; stdlib TLS for upstream client |
| E | `std.Io` injected everywhere; `Threaded` backend (io_uring flag dropped in m11 — Evented networking is stubbed at 0.16.0); no custom thread pool |
| F | Config format: ZON via `std.zon`; DB is truth; export/import for backup + host moves |
| G | SQLite vendored amalgamation + own thin wrapper |
| H | Two DBs: `config.db` (precious) + `querylog.db` (expendable, self-contained, client IP as text) |
| I | Frontend embedded in binary; dev flag serves from disk; static musl release builds |
| J | Auto-migration for `config.db` at startup; `querylog.db` recreated on mismatch |
| — | Safe-search per-group; Prometheus `/metrics` in scope; CI on self-hosted Gitea Actions |