project plan, values, milestone 1 spec, zig 0.16 api research
This commit is contained in:
@@ -0,0 +1,683 @@
|
||||
# nxdns — Implementation Plan v3.0 (Zig 0.16.0 Stable)
|
||||
|
||||
Source of truth for **nxdns**, a self-hosted DNS sinkhole written in Zig 0.16.0 stable.
|
||||
All stdlib claims in this document are verified against the `0.16.0` tag of the Zig repo (`../zig`).
|
||||
|
||||
There is no v1/v2 versioning. Scope is binary: a feature is in scope (and gets built) or out of scope (and does not). "Done" = everything in scope implemented, tested, documented.
|
||||
|
||||
## 1. Purpose
|
||||
|
||||
Fix three pain points in Pi-hole/AdGuard/Technitium:
|
||||
1. Clunky, dated admin UI.
|
||||
2. No config-as-code story (no way to export current state as a text artifact).
|
||||
3. Silent operational failures (cloudflared proxy-dns incident: broken DoH setup + unbounded error logs filled SD card).
|
||||
|
||||
Serves a household LAN (≈2–20 devices). Portfolio-grade public repo with extensive docs, self-hosted on Gitea.
|
||||
|
||||
---
|
||||
|
||||
## 2. Scope
|
||||
|
||||
### 2.1 In Scope
|
||||
|
||||
- DNS server for LAN clients: UDP/53, TCP/53, DoH server, DoT server.
|
||||
- Upstream resolution: DoH (HTTP/1.1), DoT.
|
||||
- Local DNS records (A/AAAA/CNAME) + conditional forwarding (zone → designated resolver, plain UDP/TCP allowed).
|
||||
- Domain filtering: blocklists (hosts/domains/ABP), custom rules (allow/block; exact, parent-walk, wildcard), CNAME uncloaking (depth 8), per-group safe-search rewrite.
|
||||
- DNS caching: positive + negative, in-memory only.
|
||||
- Client/group model: IPv4 + IPv6 parity, per-client group assignment, per-group source assignments.
|
||||
- Query logging + analytics: async batched writes to SQLite (WAL), retention cleanup, dashboard + time buckets, live SSE stream.
|
||||
- Web app + REST API: LAN/Tailscale admin UI, optional password auth, OpenAPI schema + CI contract tests.
|
||||
- Observability: upstream health API + UI, disk monitor with UI banner, bounded log rotation, Prometheus `/metrics`.
|
||||
- Ops: DB-as-truth config, `nxdns export`/`import` (ZON), scheduled + manual blocklist updates, TLS cert watcher + reload, auto-migration on upgrade, systemd service + Dockerfile + compose.
|
||||
|
||||
### 2.2 Out of Scope (permanent scope decisions, not deferrals)
|
||||
|
||||
- Regex rules. Wildcards + parent-walk cover the real use cases; regex on the DNS hot path means ReDoS exposure plus an immature dependency or a homegrown engine. Regex lines in blocklists are counted, skipped, and the skip count is surfaced in the UI.
|
||||
- DHCP server.
|
||||
- DNSSEC validation (DO bit passthrough only).
|
||||
- DoQ (QUIC), HTTP/2 upstream transport.
|
||||
- Clustering / distributed state.
|
||||
- Prebuilt binaries / published Docker images / project website. Repo + documented build-it-yourself path only.
|
||||
|
||||
---
|
||||
|
||||
## 3. Locked Decisions
|
||||
|
||||
### 3.1 Toolchain
|
||||
|
||||
- Compiler: **Zig 0.16.0 stable**. `build.zig` asserts major/minor.
|
||||
- Targets: **x86_64-linux-musl** and **aarch64-linux-musl**, both first-class in CI. Release binaries are **static musl** — fully self-contained, no glibc version coupling on the Pi.
|
||||
|
||||
### 3.2 Concurrency: `std.Io` (Decision E)
|
||||
|
||||
Zig 0.16 stdlib is built on the `std.Io` interface: `std.Io.net` owns networking (`std.net` in its old form is gone) and `std.http.Client` requires an `io: Io`. Therefore:
|
||||
|
||||
- **`Io` is the injected platform abstraction.** Every component that does I/O takes `io: Io`. No project-owned wrapper interfaces around it — a second abstraction over an abstraction with one consumer is waste.
|
||||
- **Backend: `std.Io.Threaded` by default** — the mature, debuggable path; at ≤20 devices throughput is a non-issue. `std.Io.Evented` (io_uring) is selectable via config flag; the code is backend-agnostic by construction, so this is a switch, not a refactor.
|
||||
- No hand-written thread pool. `Io` async/concurrent/Group covers task management.
|
||||
- Core domain modules (`dns`, `filter`, `cache`) stay pure: no `Io`, no sockets — bytes in, bytes out. Only servers, upstream clients, and storage touch `Io`.
|
||||
|
||||
### 3.3 TLS (Decision D)
|
||||
|
||||
Verified: 0.16.0 ships `std.crypto.tls.Client` only. There is no server-side TLS in the stdlib.
|
||||
|
||||
- **Upstream client TLS (DoH/DoT): stdlib** `std.crypto.tls` via `std.http.Client` / direct. Maintained upstream, exercised by the stdlib itself.
|
||||
- **Local server TLS (DoH/DoT termination): vendored mbedTLS.** Audited, embedded-focused, compiles cleanly with the Zig build system for both targets. We do not hand-roll a TLS server in a security-sensitive service.
|
||||
- TLS glue lives in `platform/tls_client.zig` (stdlib wrapper: error classification, deadlines, retry policy) and `platform/tls_server.zig` (mbedTLS wrapper exposing `std.Io.Reader`/`Writer` so `std.http.Server` composes on top unchanged).
|
||||
|
||||
### 3.4 C Dependency Policy
|
||||
|
||||
- **`sqlite3` + `mbedtls`, both vendored** and compiled from source in `build.zig`. Pinned versions, hermetic cross-compilation, no system libs.
|
||||
- SQLite: amalgamation + our own thin wrapper (`storage/db.zig`: open/close, prepared-statement cache, typed row mapping, error translation). No third-party binding — the needed surface is small enough to own (Decision G).
|
||||
- Zig package deps: minimized; anything adopted is pinned in `build.zig.zon`. No TOML/regex/HTTP packages needed under current decisions.
|
||||
|
||||
### 3.5 Config Format + Truth Model (Decision F)
|
||||
|
||||
- **DB is truth. Config file format is ZON** (`std.zon` parse + stringify — typed parsing into config structs, exact round-trip, stdlib-maintained, comments supported). No TOML: a third-party parser plus a hand-written serializer is two failure surfaces in the correctness-critical bootstrap/round-trip path, bought for syntax familiarity.
|
||||
- First start: if DB empty and `/etc/nxdns/config.zon` exists, validate → seed DB. Subsequent starts ignore the file.
|
||||
- `nxdns export [--out file.zon]` dumps DB state as canonical ZON. `nxdns import <file.zon>` validates + replaces DB contents (`--force` if DB non-empty). Export/import = backup + host migration, **not** upgrades (§3.7).
|
||||
- No file watcher, no auto-regeneration.
|
||||
|
||||
### 3.6 Storage Layout (Decision H)
|
||||
|
||||
Two SQLite files with opposite write profiles, isolated from each other:
|
||||
|
||||
- **`config.db`** — small, precious, rarely written: groups, clients, prefixes, upstreams, blocklist source metadata, rules, local records, forward zones, settings, schema version.
|
||||
- **`querylog.db`** — high-churn, large, expendable: query log + its own private `domains` dimension table. Client identity stored as **IP text**, not a FK into config — log rows are immutable facts and must not point at mutable config rows. If `querylog.db` is missing or corrupt at startup, rename aside, recreate, keep serving. Log loss is not an outage.
|
||||
- No cross-DB references. Retention/VACUUM churn never touches `config.db`; config backup is a copy of a tiny file.
|
||||
|
||||
### 3.7 Upgrades: Auto-Migration (Decision J)
|
||||
|
||||
- `config.db`: numbered, sequential SQL migration steps compiled into the binary. At startup: read schema version row, apply newer steps inside a transaction, continue. Operator upgrade = install binary, restart.
|
||||
- `querylog.db`: **no migrations.** On schema mismatch: rename aside, recreate fresh.
|
||||
|
||||
### 3.8 Blocklist Storage (Decision A)
|
||||
|
||||
Blocklist domains are **not** stored in SQLite — they are a cache of re-downloadable remote artifacts, not config or state:
|
||||
|
||||
- Each source compiles to `/var/lib/nxdns/blocklists/<source_id>.list`: normalized, one domain per line, small header (source URL, fetch time, count, checksum). Wildcard/regex-flavored lines: wildcards go to `<source_id>.wild`; regex lines are counted + skipped (count in metadata → UI).
|
||||
- `config.db` keeps source **metadata only** (`blocklist_sources`).
|
||||
- Startup + post-update: parse files into the immutable in-memory matcher (RCU swap, §9.5).
|
||||
- Corruption recovery is per-file: checksum mismatch → re-download one list.
|
||||
|
||||
### 3.9 Rule Model (Decision B)
|
||||
|
||||
Rule kinds: `exact`, parent-walk (implicit via candidate chain), `wildcard` (`*` segment patterns, e.g. `*.doubleclick.net`, `ads.*.example.com`). Actions: `allow` | `block`. Kind is an explicit column — the model is extensible without breakage, but regex stays out of scope (§2.2).
|
||||
|
||||
### 3.10 Filtering Precedence
|
||||
|
||||
1. Exact/parent **allow** rules
|
||||
2. Exact/parent **block** rules
|
||||
3. Wildcard allow rules
|
||||
4. Wildcard block rules
|
||||
5. Blocklist domains
|
||||
6. Blocklist wildcards
|
||||
|
||||
Tie-break at same specificity: **allow wins**.
|
||||
|
||||
### 3.11 Network Posture
|
||||
|
||||
- Default web bind: LAN/Tailscale-friendly (non-loopback allowed).
|
||||
- Auth optional (empty `web.password` disables it). Set → argon2id-hashed password, session cookie (`HttpOnly`, `SameSite=Lax`, `Secure` on HTTPS).
|
||||
- API per-IP rate limit on by default.
|
||||
|
||||
### 3.12 Addressing
|
||||
|
||||
IPv4 + IPv6 full parity for: client identity, rate limiting, logging, group assignment, prefix-based group matching. Dual-stack assumed.
|
||||
|
||||
### 3.13 Filesystem Layout (FHS)
|
||||
|
||||
- `/etc/nxdns/config.zon` — bootstrap (first start only).
|
||||
- `/var/lib/nxdns/config.db`, `/var/lib/nxdns/querylog.db`
|
||||
- `/var/lib/nxdns/blocklists/*.list|*.wild`
|
||||
- `/var/log/nxdns/nxdns.log` — only in file output mode; default is stderr → journald.
|
||||
|
||||
### 3.14 Frontend Stack (Decision I)
|
||||
|
||||
- Vite + React + TypeScript + Tailwind; TanStack Router + TanStack Query. SPA, no SSR.
|
||||
- **Built assets embedded in the binary** at compile time: single self-contained artifact, no asset-path config, no binary/UI skew. `zig build` accepts the dist path; the release pipeline runs `npm run build` first (CI always does).
|
||||
- Dev-mode flag serves assets from disk so UI iteration needs no Zig rebuild.
|
||||
|
||||
### 3.15 CI
|
||||
|
||||
- Self-hosted **Gitea + Gitea Actions runner**.
|
||||
- Jobs: `zig build test`, fuzz smoke, integration tests, OpenAPI contract tests (live server validated against `openapi.yaml`), frontend build, cross-compile both targets. aarch64 test execution via qemu-user if the runner is x86_64.
|
||||
|
||||
---
|
||||
|
||||
## 4. Architecture Overview
|
||||
|
||||
```
|
||||
Client DNS Query
|
||||
|
|
||||
v
|
||||
[UDP / TCP / DoH / DoT servers] (std.Io; mbedTLS terminates DoH/DoT)
|
||||
|
|
||||
v
|
||||
[Request Handler]
|
||||
|- rate limit check
|
||||
|- parse DNS packet
|
||||
|- group resolution (v4/v6 parity)
|
||||
|- local records check -> answer locally
|
||||
|- forward-zone check -> forward to designated resolver (bypasses filtering)
|
||||
|- allow/block evaluation (rules + blocklists, per group)
|
||||
|- safe-search rewrite (per group)
|
||||
|- cache lookup
|
||||
|- upstream query (DoH/DoT pool) + health tracking
|
||||
|- CNAME uncloaking check
|
||||
|- cache store
|
||||
|- async query log + SSE fanout
|
||||
v
|
||||
Response to client
|
||||
```
|
||||
|
||||
Cross-cutting: `ConfigManager` (bootstrap/import/export/settings), `BlocklistManager` (fetch/compile/swap), `Storage` (two SQLite DBs), `Cache`, `UpstreamHealth`, `DiskMonitor`, `Auth`, `Web API` (REST + SSE + metrics).
|
||||
|
||||
---
|
||||
|
||||
## 5. Source Layout
|
||||
|
||||
```
|
||||
src/
|
||||
main.zig # Io backend instantiation, wiring
|
||||
app.zig
|
||||
version.zig
|
||||
|
||||
platform/
|
||||
address.zig # NetAddress: v4/v6 parity type + canonical key (was net_adapter)
|
||||
tls_client.zig # stdlib TLS wrapper: deadlines, error classes, retry policy
|
||||
tls_server.zig # mbedTLS wrapper exposing std.Io.Reader/Writer
|
||||
|
||||
dns/ # pure: bytes in, bytes out; no Io
|
||||
types.zig header.zig name.zig question.zig record.zig edns.zig packet.zig
|
||||
|
||||
server/
|
||||
udp_server.zig tcp_server.zig doh_server.zig dot_server.zig
|
||||
handler.zig rate_limiter.zig shutdown.zig
|
||||
|
||||
upstream/
|
||||
doh_client.zig dot_client.zig pool.zig health.zig
|
||||
|
||||
filter/ # pure
|
||||
matcher.zig rules.zig wildcard.zig
|
||||
parser_hosts.zig parser_domains.zig parser_abp.zig
|
||||
fetcher.zig compiler.zig # list download -> compiled .list/.wild files
|
||||
safesearch.zig
|
||||
|
||||
local/ # pure
|
||||
records.zig # local A/AAAA/CNAME answers
|
||||
forward_zones.zig # zone -> designated resolver matching
|
||||
|
||||
cache/
|
||||
dns_cache.zig
|
||||
|
||||
storage/
|
||||
db.zig # sqlite3 thin wrapper
|
||||
config_schema.zig migrations.zig
|
||||
querylog_schema.zig
|
||||
repositories/
|
||||
clients_repo.zig groups_repo.zig rules_repo.zig sources_repo.zig
|
||||
queries_repo.zig settings_repo.zig upstreams_repo.zig local_repo.zig
|
||||
logger.zig retention.zig disk_monitor.zig
|
||||
|
||||
config/
|
||||
model.zig bootstrap.zig import.zig export.zig validate.zig # all ZON via std.zon
|
||||
|
||||
web/
|
||||
server.zig router.zig auth.zig sse.zig static.zig metrics.zig openapi.zig
|
||||
handlers/
|
||||
auth.zig stats.zig queries.zig clients.zig groups.zig blocklists.zig
|
||||
rules.zig local.zig lookup.zig pause.zig settings.zig
|
||||
upstream_health.zig certs.zig health.zig version.zig
|
||||
|
||||
web/ # Vite + React + TS + Tailwind + TanStack
|
||||
vendor/ # sqlite3 amalgamation, mbedtls (pinned)
|
||||
docs/ # operator/ architecture/ config-reference/ api/
|
||||
tests/ # dns/ integration/ fuzz/
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. DNS Behavior
|
||||
|
||||
### 6.1 Protocol
|
||||
|
||||
- RFC 1035 parse/encode. EDNS OPT passthrough. DO bit passthrough (no validation). ECS: `strip` (default) or `forward`.
|
||||
- Process first question only. Preserve query ID and RD flag.
|
||||
- Malformed: `FORMERR` when a response is possible; silent drop for severely truncated.
|
||||
|
||||
### 6.2 Blocked Response Modes
|
||||
|
||||
`blocking.response`: `zero` (A=`0.0.0.0`, AAAA=`::`) or `nxdomain`. TTL = `blocking.ttl`.
|
||||
|
||||
### 6.3 CNAME Uncloaking
|
||||
|
||||
Walk chain to depth 8; any target hitting block logic → synthesize blocked response.
|
||||
|
||||
### 6.4 Local Records
|
||||
|
||||
- Table-backed A/AAAA/CNAME records, group-independent. Matched before filtering.
|
||||
- CNAME targets resolve through the normal pipeline (uncloaking rules apply).
|
||||
|
||||
### 6.5 Conditional Forwarding
|
||||
|
||||
- `forward_zones`: suffix match (`lan.home`, `10.in-addr.arpa`, …) → designated resolver (`udp://192.168.1.1:53` style; plain UDP/TCP permitted — these are local infra resolvers).
|
||||
- Matching queries bypass filtering and blocklists; responses cached normally.
|
||||
|
||||
---
|
||||
|
||||
## 7. Filtering Engine
|
||||
|
||||
### 7.1 Evaluation
|
||||
|
||||
For `{domain, qtype, group_id}`:
|
||||
1. Normalize: lowercase, trim trailing dot.
|
||||
2. Build candidate chain (full, parent1, parent2, …).
|
||||
3. Explicit rules per §3.10 precedence.
|
||||
4. Group's blocklist domains (hash set over compiled lists).
|
||||
5. Group's blocklist wildcards.
|
||||
6. No match → allow.
|
||||
|
||||
### 7.2 Group Assignment
|
||||
|
||||
- Per source IP: (1) exact match in `clients`, (2) longest-prefix match in `client_prefixes` (ties: longer prefix, then priority), (3) `default` group.
|
||||
- Auto-materialization: first query from an unseen IP inserts a `clients` row (`hand_edited=0`, `first_seen=now`) for UI visibility and stable group assignment. The query log does **not** FK to it (§3.6).
|
||||
- Retention drops `hand_edited=0` clients with no queries in `retention_days`.
|
||||
|
||||
### 7.3 Reload
|
||||
|
||||
New immutable matcher built from DB + compiled list files → atomic pointer swap (RCU, generation counter). Readers lock-free.
|
||||
|
||||
### 7.4 Safe-Search
|
||||
|
||||
Per-group boolean. Rewrites known engine domains to their safe-search CNAME targets.
|
||||
|
||||
---
|
||||
|
||||
## 8. Cache
|
||||
|
||||
- Key: normalized qname + qtype + qclass + DO bit + forwarded ECS subnet (only when `ecs_mode=forward`).
|
||||
- Value: full response bytes, `stored_at`, `expires_at`, positive/negative flag.
|
||||
- Positive TTL: `min(answer TTLs)` bounded by sanity ceiling. Negative: SOA/minimum capped by `cache.negative_ttl_max` (0 disables).
|
||||
- Hit path: clone bytes, overwrite transaction ID, decrement visible TTLs; TTL ≤ 0 → miss.
|
||||
- Bounded by `cache.size`; CLOCK eviction; opportunistic + periodic expiry sweep.
|
||||
- **In-memory only, no persistence.** Cold start refills within a minute; SD-corruption risk isn't worth pre-warmed entries.
|
||||
|
||||
---
|
||||
|
||||
## 9. Upstream Resolution
|
||||
|
||||
- Schemes: `https://…` → DoH, `tls://host:853` → DoT.
|
||||
- Ordered by priority; sequential attempt; per-upstream failure counters; exponential backoff with jitter; success resets.
|
||||
- `UpstreamHealth` per upstream: last_success_at, last_error_at, last_error_message, rolling success rate, consecutive failures, backoff-until. Exposed via `GET /api/upstream/health`, dashboard, `/metrics`, and `nxdns check`.
|
||||
- DoH client: `std.http.Client` with `content-type/accept: application/dns-message`; strict status + payload checks.
|
||||
- `platform/tls_client.zig` enforces per-connection read/write deadlines, classifies TLS errors explicitly, retries with backoff. Integration tests cover timeout/hang scenarios so compiler upgrades can't silently regress them.
|
||||
- Connect, read, and total-budget timeouts each configurable.
|
||||
|
||||
---
|
||||
|
||||
## 10. Rate Limiting
|
||||
|
||||
- DNS: default 1000 req / 60s per client IP (v4/v6 keyed alike); exceeded → `REFUSED`.
|
||||
- API: per-IP token bucket, on by default; separate SSE connection cap per IP; localhost relaxed (configurable).
|
||||
- Limiter maps bounded; stale-key windowed sweep.
|
||||
|
||||
---
|
||||
|
||||
## 11. Storage
|
||||
|
||||
### 11.1 Pragmas (both DBs)
|
||||
|
||||
`journal_mode=WAL`, `synchronous=NORMAL`, `foreign_keys=ON`, `busy_timeout` set.
|
||||
|
||||
### 11.2 config.db Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE schema_version (version INTEGER NOT NULL);
|
||||
|
||||
CREATE TABLE groups (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL UNIQUE,
|
||||
safe_search INTEGER NOT NULL DEFAULT 0
|
||||
);
|
||||
INSERT OR IGNORE INTO groups (id, name) VALUES (1, 'default');
|
||||
|
||||
CREATE TABLE clients (
|
||||
id INTEGER PRIMARY KEY,
|
||||
ip TEXT NOT NULL UNIQUE, -- canonical text form (v4 dotted / v6 RFC 5952)
|
||||
name TEXT,
|
||||
group_id INTEGER NOT NULL REFERENCES groups(id),
|
||||
hand_edited INTEGER NOT NULL DEFAULT 0,
|
||||
first_seen INTEGER NOT NULL,
|
||||
last_seen INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE client_prefixes (
|
||||
id INTEGER PRIMARY KEY,
|
||||
prefix TEXT NOT NULL UNIQUE, -- "192.168.1.0/24", "fd00:abcd::/48"
|
||||
group_id INTEGER NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
|
||||
priority INTEGER NOT NULL DEFAULT 100
|
||||
);
|
||||
|
||||
CREATE TABLE upstreams (
|
||||
id INTEGER PRIMARY KEY,
|
||||
url TEXT NOT NULL UNIQUE,
|
||||
priority INTEGER NOT NULL DEFAULT 100,
|
||||
enabled INTEGER NOT NULL DEFAULT 1
|
||||
);
|
||||
|
||||
CREATE TABLE blocklist_sources (
|
||||
id INTEGER PRIMARY KEY,
|
||||
url TEXT NOT NULL UNIQUE,
|
||||
name TEXT NOT NULL,
|
||||
enabled INTEGER NOT NULL DEFAULT 1,
|
||||
is_suggested INTEGER NOT NULL DEFAULT 0,
|
||||
last_updated INTEGER,
|
||||
domain_count INTEGER NOT NULL DEFAULT 0,
|
||||
wildcard_count INTEGER NOT NULL DEFAULT 0,
|
||||
skipped_regex_count INTEGER NOT NULL DEFAULT 0,
|
||||
checksum TEXT
|
||||
);
|
||||
|
||||
CREATE TABLE group_sources (
|
||||
group_id INTEGER NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
|
||||
source_id INTEGER NOT NULL REFERENCES blocklist_sources(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (group_id, source_id)
|
||||
);
|
||||
|
||||
CREATE TABLE rules (
|
||||
id INTEGER PRIMARY KEY,
|
||||
group_id INTEGER NOT NULL REFERENCES groups(id) ON DELETE CASCADE,
|
||||
pattern TEXT NOT NULL,
|
||||
kind TEXT NOT NULL CHECK(kind IN ('exact','wildcard')),
|
||||
action TEXT NOT NULL CHECK(action IN ('allow','block')),
|
||||
created_at INTEGER NOT NULL
|
||||
);
|
||||
|
||||
CREATE TABLE local_records (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
rtype TEXT NOT NULL CHECK(rtype IN ('A','AAAA','CNAME')),
|
||||
value TEXT NOT NULL,
|
||||
ttl INTEGER NOT NULL DEFAULT 300,
|
||||
UNIQUE(name, rtype, value)
|
||||
);
|
||||
|
||||
CREATE TABLE forward_zones (
|
||||
id INTEGER PRIMARY KEY,
|
||||
zone TEXT NOT NULL UNIQUE,
|
||||
resolver TEXT NOT NULL -- "udp://192.168.1.1:53"
|
||||
);
|
||||
|
||||
CREATE TABLE settings (key TEXT PRIMARY KEY, value TEXT NOT NULL);
|
||||
```
|
||||
|
||||
### 11.3 querylog.db Schema
|
||||
|
||||
```sql
|
||||
CREATE TABLE domains (
|
||||
id INTEGER PRIMARY KEY,
|
||||
domain TEXT NOT NULL UNIQUE
|
||||
);
|
||||
|
||||
CREATE TABLE query_log (
|
||||
id INTEGER PRIMARY KEY,
|
||||
timestamp INTEGER NOT NULL,
|
||||
domain_id INTEGER NOT NULL REFERENCES domains(id),
|
||||
client_ip TEXT NOT NULL, -- text, not a FK: log rows are immutable facts
|
||||
qtype INTEGER,
|
||||
blocked INTEGER NOT NULL,
|
||||
block_reason TEXT,
|
||||
response_time_us INTEGER,
|
||||
cache_hit INTEGER,
|
||||
upstream TEXT
|
||||
);
|
||||
CREATE INDEX idx_query_log_ts ON query_log(timestamp);
|
||||
CREATE INDEX idx_query_log_client ON query_log(client_ip);
|
||||
CREATE INDEX idx_query_log_domain ON query_log(domain_id);
|
||||
```
|
||||
|
||||
### 11.4 Query Logger
|
||||
|
||||
- In-memory buffer, mutex guarded, hard cap `query_log_buffer_max` (default 10000).
|
||||
- Flush: batch size (default 100) or max interval (default 100ms).
|
||||
- Privacy transforms (hide_domains / hide_client_ips) applied before persist + SSE fanout.
|
||||
- Backpressure: buffer full → drop oldest unflushed entry, increment monotonic `queries_dropped` (exposed in `/api/health` + `/metrics`). SSE fanout precedes buffer insert, so live viewers still see dropped-from-persistence entries.
|
||||
|
||||
### 11.5 Retention
|
||||
|
||||
Periodic delete of rows older than `retention_days`; scheduled checkpoint/VACUUM on `querylog.db` only.
|
||||
|
||||
### 11.6 Disk Discipline (cloudflared lesson)
|
||||
|
||||
- Log output default: **stderr** (journald rotates). File mode: rotate at `log.max_size_mb` (default 50), keep `log.max_files` (default 5).
|
||||
- `DiskMonitor` samples DB + log-dir sizes every 60s:
|
||||
- free < `disk.warn_free_mb` → UI banner + `/api/health` degraded.
|
||||
- free < `disk.min_free_mb` → stop non-essential writes (blocklist updates, log flushes); buffer continues under backpressure policy.
|
||||
- Upstream error log lines deduplicated per-upstream at 1/minute.
|
||||
|
||||
---
|
||||
|
||||
## 12. Configuration
|
||||
|
||||
### 12.1 Bootstrap ZON Shape
|
||||
|
||||
```zon
|
||||
.{
|
||||
.upstream = .{
|
||||
.servers = .{ "https://cloudflare-dns.com/dns-query", "tls://dns.google:853" },
|
||||
.connect_timeout_ms = 2000,
|
||||
.read_timeout_ms = 3000,
|
||||
},
|
||||
.dns = .{
|
||||
.bind_ipv4 = "0.0.0.0",
|
||||
.bind_ipv6 = "::",
|
||||
.port = 53,
|
||||
.rate_limit = 1000,
|
||||
.rate_window_seconds = 60,
|
||||
},
|
||||
.blocking = .{ .response = .zero, .ttl = 5 }, // .zero | .nxdomain
|
||||
.cache = .{ .size = 10000, .negative_ttl_max = 3600 },
|
||||
.web = .{
|
||||
.enabled = true,
|
||||
.bind = "0.0.0.0",
|
||||
.port = 8080,
|
||||
.password = "", // empty => auth disabled
|
||||
.session_ttl_hours = 24,
|
||||
.api_rate_limit_per_min = 300,
|
||||
.sse_max_connections_per_ip = 3,
|
||||
},
|
||||
.doh_server = .{ .enabled = false, .bind = "0.0.0.0", .port = 443,
|
||||
.cert_path = "/etc/nxdns/cert.pem", .key_path = "/etc/nxdns/key.pem" },
|
||||
.dot_server = .{ .enabled = false, .bind = "0.0.0.0", .port = 853,
|
||||
.cert_path = "/etc/nxdns/cert.pem", .key_path = "/etc/nxdns/key.pem" },
|
||||
.edns = .{ .ecs_mode = .strip }, // .strip | .forward
|
||||
.local_records = .{ .{ .name = "nas.lan", .rtype = .A, .value = "192.168.1.10" } },
|
||||
.forward_zones = .{ .{ .zone = "lan.home", .resolver = "udp://192.168.1.1:53" } },
|
||||
.logging = .{
|
||||
.level = .info,
|
||||
.retention_days = 30,
|
||||
.query_log_buffer_max = 10000,
|
||||
.hide_domains = false,
|
||||
.hide_client_ips = false,
|
||||
.output = .stderr, // .stderr | .syslog | .file
|
||||
.file_path = "/var/log/nxdns/nxdns.log",
|
||||
.max_size_mb = 50,
|
||||
.max_files = 5,
|
||||
},
|
||||
.disk = .{ .min_free_mb = 200, .warn_free_mb = 500 },
|
||||
.blocklist_update = .{ .enabled = true, .interval_hours = 24 },
|
||||
}
|
||||
```
|
||||
|
||||
### 12.2 Validation
|
||||
|
||||
At least one upstream; ports in range; cert+key readable if DoH/DoT server enabled; resolver URLs parseable. `nxdns check` runs the validator + probes upstreams.
|
||||
|
||||
### 12.3 Settings Semantics
|
||||
|
||||
Scalars in `settings(key, value)`; ordered/structured items in dedicated tables. Restart-required settings flagged; UI shows "restart to apply" banner.
|
||||
|
||||
---
|
||||
|
||||
## 13. Web / API
|
||||
|
||||
### 13.1 Endpoints
|
||||
|
||||
- `POST /api/auth/login`, `POST /api/auth/logout`
|
||||
- `GET /api/stats?period=…`, `GET /api/stats/timeseries?period=…`
|
||||
- `GET /api/queries` (filter + paginate), `GET /api/queries/live` (SSE, per-IP cap)
|
||||
- `GET/PUT /api/clients/{id}`
|
||||
- `GET/POST/PUT/DELETE /api/groups…`, `/api/blocklists…`, `/api/rules…`, `/api/local-records…`, `/api/forward-zones…`
|
||||
- `POST /api/blocklists/update`
|
||||
- `GET /api/lookup?domain=…&group_id=…`
|
||||
- `GET/POST /api/pause`
|
||||
- `GET/PUT /api/settings`
|
||||
- `GET /api/upstream/health`
|
||||
- `POST /api/certs/reload`
|
||||
- `GET /api/health` — overall + disk + upstream + queries_dropped rollup
|
||||
- `GET /metrics` — Prometheus text exposition: query counters (total/blocked/cached), per-upstream health, cache stats, queries_dropped, disk gauges
|
||||
- `GET /api/version`, `GET /api/openapi.yaml`
|
||||
|
||||
### 13.2 OpenAPI
|
||||
|
||||
Hand-maintained `openapi.yaml`, served + rendered into `docs/api/`. CI contract tests spin up a seeded server, hit every documented endpoint, and validate bodies/status/auth against the schema. Drift fails CI.
|
||||
|
||||
---
|
||||
|
||||
## 14. Frontend
|
||||
|
||||
Pages: Dashboard (stats + upstream health + disk), Query log, Live log, Clients, Groups, Blocklists, Rules, Local DNS (records + forward zones), Domain lookup, Settings.
|
||||
|
||||
Requirements: responsive desktop/mobile; route loaders for initial fetch; TanStack Query for cache/retries; error/loading states on every data view; works with auth enabled or disabled; restart-required banner.
|
||||
|
||||
---
|
||||
|
||||
## 15. CLI
|
||||
|
||||
- `nxdns run` — start the server.
|
||||
- `nxdns check` — validate config, probe upstreams, test cert readability; nonzero exit on failure.
|
||||
- `nxdns export [--out file.zon]`
|
||||
- `nxdns import <file.zon> [--force]`
|
||||
- `nxdns version` — app version, Zig version string, build date, git commit.
|
||||
|
||||
---
|
||||
|
||||
## 16. Implementation Order
|
||||
|
||||
### Phase 0 — Build Baseline
|
||||
Scaffold tree; `build.zig` with 0.16 assertion, musl targets, vendored sqlite3 + mbedtls compiling; version plumbing; Gitea Actions workflows (test, integration, fuzz smoke, OpenAPI lint, frontend build).
|
||||
Exit: cross-compiled hello-world linking both C deps on both targets; CI green.
|
||||
|
||||
### Phase 1 — Platform Layer
|
||||
`platform/address.zig` (v4/v6 parity + canonical keys); `platform/tls_client.zig` (deadlines, error classes); `platform/tls_server.zig` (mbedTLS handshake → `std.Io.Reader`/`Writer`).
|
||||
Exit: UDP echo over `std.Io`; TLS client handshake against a real host; mbedTLS server terminating a loopback TLS connection.
|
||||
|
||||
### Phase 2 — DNS Core
|
||||
Types/header/name/question/record/packet; parser + encoder tests + fuzz target; EDNS + DO passthrough.
|
||||
Exit: unit + fuzz smoke pass.
|
||||
|
||||
### Phase 3 — Resolver Transport
|
||||
UDP server, TCP server, DoH + DoT upstream clients, pool + failover/backoff + health.
|
||||
Exit: A/AAAA forwarding over UDP + TCP; health populated.
|
||||
|
||||
### Phase 4 — Storage + Config
|
||||
SQLite wrapper; config.db schema + migration runner; querylog.db schema + recreate-on-mismatch; repositories; ZON bootstrap + import/export; `nxdns check`.
|
||||
Exit: first start seeds DB from ZON; export → import round-trips byte-stable.
|
||||
|
||||
### Phase 5 — Filtering + Local DNS
|
||||
Rule matcher (exact/parent/wildcard); blocklist parsers → compiled file format; fetcher + scheduled update; RCU swap; per-group safe-search; local records; forward zones.
|
||||
Exit: precedence table validated by tests; local zone answers + conditional forwards work.
|
||||
|
||||
### Phase 6 — Cache + Rate Limit + Logging + Disk Monitor
|
||||
TTL cache + reconstruction; v4/v6 rate limiter; async query logger + backpressure + retention; DiskMonitor + rotation + error-log dedup.
|
||||
Exit: disk thresholds trigger degradation + drop counters in integration test.
|
||||
|
||||
### Phase 7 — Handler Integration
|
||||
Full pipeline composition; CNAME uncloaking; pause/resume.
|
||||
Exit: end-to-end DNS flow with blocking, local records, cache, failover.
|
||||
|
||||
### Phase 8 — Web / API / SSE / Auth / Metrics
|
||||
HTTP server + router; handlers; SSE; optional auth; API rate limiting; `/metrics`; OpenAPI served + contract tests; embedded frontend + dev-mode disk serving.
|
||||
Exit: frontend fully drives config and operations; contract tests green.
|
||||
|
||||
### Phase 9 — Local DoH/DoT Endpoints
|
||||
DoH server + DoT server on `platform/tls_server.zig`; cert watcher + reload.
|
||||
Exit: LAN client resolves via DoH and DoT against local certs.
|
||||
|
||||
### Phase 10 — Packaging + Ops + Docs
|
||||
systemd unit (`AmbientCapabilities=CAP_NET_BIND_SERVICE`, hardened, writable `/var/lib/nxdns` + optional `/var/log/nxdns`); Dockerfile + compose (53/udp+tcp, 8080; mounts `/etc/nxdns`, `/var/lib/nxdns`); operator/architecture/config-reference/API docs.
|
||||
Exit: documented deployment works end-to-end on the Pi 5.
|
||||
|
||||
---
|
||||
|
||||
## 17. Testing Strategy
|
||||
|
||||
- **Unit**: DNS encode/decode; rule precedence + wildcard matcher; cache put/get/TTL rewrite; ZON bootstrap + export round-trip; rate limiter; migration runner (fresh + stepwise upgrade).
|
||||
- **Fuzz**: DNS parser malformed-packet fuzzing; blocklist parser fuzzing.
|
||||
- **Integration**: UDP/TCP query path; blocked path; allow-over-block; wildcard precedence; CNAME uncloaking block; local records + forward zones; upstream failover/backoff/health; disk-full degradation; querylog.db corruption recovery; API CRUD; auth on/off; SSE; contract tests.
|
||||
- **Manual**: `dig @pi example.com` / blocked domain / local record; DoH/DoT client checks; dashboard + live log.
|
||||
|
||||
---
|
||||
|
||||
## 18. Performance Targets
|
||||
|
||||
- Sustained ≥ 100 qps on Raspberry Pi 5.
|
||||
- Blocklist lookup p95 < 1 ms.
|
||||
- Cached response p95 < 5 ms.
|
||||
- Memory with ~1M blocked domains < 100 MB.
|
||||
- Stripped static binary < 10 MB per arch (excluding embedded frontend assets; < 15 MB with them).
|
||||
|
||||
---
|
||||
|
||||
## 19. Security
|
||||
|
||||
- Sanitize all DNS and API inputs; prepared statements everywhere.
|
||||
- argon2id password hashing; random session tokens with expiry.
|
||||
- No secrets in logs. TLS keys readable by service user only.
|
||||
- systemd hardening + least privilege.
|
||||
- mbedTLS + sqlite3 versions pinned; upgrades are deliberate, reviewed bumps.
|
||||
|
||||
---
|
||||
|
||||
## 20. Success Criteria
|
||||
|
||||
1. `nxdns run` starts cleanly on Zig 0.16.0 stable, static musl, both arches.
|
||||
2. UDP + TCP resolution works; blocked domains return the configured response; precedence per §3.10.
|
||||
3. Local records answer; conditional forwarding works.
|
||||
4. Cache hit path returns valid ID-adjusted responses.
|
||||
5. Upstream failover + backoff works; health in UI, API, `/metrics`.
|
||||
6. Disk-fill degrades gracefully; no silent log-flood failure mode.
|
||||
7. Web UI + API provide full admin functionality; OpenAPI contract tests green.
|
||||
8. `nxdns export` round-trips via `nxdns import`.
|
||||
9. Query logging, stats, SSE live stream work; querylog.db corruption self-heals.
|
||||
10. Local DoH + DoT endpoints serve LAN clients.
|
||||
11. Schema upgrade = install + restart (migration test proves it).
|
||||
12. All suites green in Gitea CI for both targets.
|
||||
13. Operator, architecture, config-reference, and API docs complete.
|
||||
|
||||
---
|
||||
|
||||
## 21. Working Notes
|
||||
|
||||
- `dns/`, `filter/`, `local/`, `cache/` stay pure (no `Io`, no sockets). Servers, upstream clients, and storage take `io: Io`.
|
||||
- Verify stdlib behavior against `../zig` (tag 0.16.0) instead of memory — the std.Io migration invalidated older knowledge once already.
|
||||
- Keep OpenAPI in sync with handlers (CI enforced).
|
||||
- When a plan decision and reality diverge during build: surface 2–3 alternatives with tradeoffs, get a call, then execute.
|
||||
|
||||
## Decision Log
|
||||
|
||||
| # | Decision |
|
||||
|---|----------|
|
||||
| A | Blocklists compile to flat files under `/var/lib/nxdns/blocklists/`; DB stores source metadata only |
|
||||
| B | Rule kinds: exact, parent-walk, wildcard. Regex permanently out of scope |
|
||||
| C | In scope: local DoH/DoT server, local records, conditional forwarding. Out: HTTP/2, DoQ, DHCP, DNSSEC, clustering |
|
||||
| D | mbedTLS (vendored) terminates server TLS; stdlib TLS for upstream client |
|
||||
| E | `std.Io` injected everywhere; `Threaded` default backend, io_uring via flag; no custom thread pool |
|
||||
| F | Config format: ZON via `std.zon`; DB is truth; export/import for backup + host moves |
|
||||
| G | SQLite vendored amalgamation + own thin wrapper |
|
||||
| H | Two DBs: `config.db` (precious) + `querylog.db` (expendable, self-contained, client IP as text) |
|
||||
| I | Frontend embedded in binary; dev flag serves from disk; static musl release builds |
|
||||
| J | Auto-migration for `config.db` at startup; `querylog.db` recreated on mismatch |
|
||||
| — | Safe-search per-group; Prometheus `/metrics` in scope; CI on self-hosted Gitea Actions |
|
||||
Reference in New Issue
Block a user