mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 13 Activity

milestone 21: abp list exceptions and a regex rule kind

Browse Source
This commit is contained in:
mokhtar
2026-08-13 19:14:47 +02:00
parent b340521716
commit 2ab7c1f1de
51 changed files with 4016 additions and 465 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/explanation/architecture.md
+1 -1
View File
@@ -29,7 +29,7 @@ Directories:
| Directory | Role |
|---|---|
| `src/dns/` | Pure DNS wire format: header, names, questions, records, whole packets, EDNS(0)/ECS (`edns.zig`), enums and limits (`types.zig`). No allocation, no `std.Io` beyond writing to a caller's writer. |
| `src/filter/` | Blocklist pipeline: line parsers (hosts, domains, ABP), the compiler that turns a downloaded list into `.list`/`.wild` bodies, `domain_set.zig` (exact-match set, no Bloom filter), `matcher.zig` (the immutable snapshot every query evaluates against), per-group `rules.zig`, `wildcard.zig`, `safesearch.zig`, blocked-response synthesis (`response.zig`). Two I/O edges live here too: `fetcher.zig` (HTTP download) and `manager.zig` (files + DB + snapshot swap). |
| `src/filter/` | Blocklist pipeline: line parsers (hosts, domains, ABP), the compiler that turns a downloaded list into `.list`/`.wild`/`.allow` bodies, `domain_set.zig` (exact-match set, no Bloom filter), `matcher.zig` (the immutable snapshot every query evaluates against), per-group `rules.zig`, `wildcard.zig`, `regex.zig` (a Pike VM for the operator's regex rules, linear-time by construction), `safesearch.zig`, blocked-response synthesis (`response.zig`). Two I/O edges live here too: `fetcher.zig` (HTTP download) and `manager.zig` (files + DB + snapshot swap). |
| `src/local/` | Local DNS records and conditional forward zones: immutable lookup tables built once from DB rows (`records.zig`, `forward_zones.zig`), plus the plain UDP/TCP client for LAN resolvers (`forward_client.zig`). |
| `src/cache/` | `dns_cache.zig`: bounded in-memory TTL cache of whole response messages, keyed by the question. The clock arrives as a parameter. |
| `src/upstream/` | Upstream resolution: shared vocabulary and the `Client` interface (`transport.zig`), DoH client (RFC 8484), DoT client (RFC 7858), per-endpoint health and backoff (`health.zig`), and `pool.zig` — priority-ordered failover that is itself a `transport.Client`, so the handler sees one interface. |
docs/how-to/back-up-and-restore.md
+1 -1
View File
@@ -99,7 +99,7 @@ nxdns import /tmp/nxdns-lab/backup.zon --data-dir /tmp/nxdns-lab/data-restored
```
```
info(migrations): config.db migrated from schema version 0 to 2
info(migrations): config.db migrated from schema version 0 to 4
imported /tmp/nxdns-lab/backup.zon
```
docs/how-to/install-with-docker.md
+1 -1
View File
@@ -211,7 +211,7 @@ why the container is `docker-nxdns-1`.
A healthy first start logs the reconcile, the authority and the bound sockets:
```
info(migrations): config.db migrated from schema version 0 to 2
info(migrations): config.db migrated from schema version 0 to 4
reconciled '/etc/nxdns/config.zon': upstreams +1 ~0 -0; settings +45 ~0 -0;
settings keys changed: dns.bind_ipv4 dns.bind_ipv6 dns.port web.bind web.port …
web authentication is now enabled
docs/how-to/install-with-systemd.md
+8 -3
View File
@@ -184,8 +184,13 @@ was written before the first start.
0640 with group `nxdns` rather than 0600: `/etc/nxdns` is a
`ConfigurationDirectory`, which systemd leaves owned by root, and the service
runs as `nxdns` and has to read this file on the first start. A root-owned 0600
file would be unreadable to it.
runs as `nxdns`. A root-owned 0600 file would be unreadable to it.
Keep that group read bit for good, not just for the first boot. Under
`run --config` the service reads this file on **every** start, so tightening
the mode later breaks the next restart. Under database authority it is
`nxdns import` that reads the file, as whoever runs that command, and a bare
`nxdns run` never reads it at all.
Do not expect `nxdns check` to catch a permissive mode here. Its only
permission warning is for a TLS private key
@@ -220,7 +225,7 @@ nxdns import /etc/nxdns/config.zon
```
```
info(migrations): config.db migrated from schema version 0 to 2
info(migrations): config.db migrated from schema version 0 to 4
imported /etc/nxdns/config.zon
```
docs/how-to/measure-performance.md
+10 -7
View File
@@ -41,8 +41,8 @@ zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000
nxdns bench suite=filter domains=100000 iters=20000 seed=0x5eed optimize=ReleaseFast
suite ops p50(us) p95(us) p99(us) max(us)
filter 20000 0.14 0.25 0.27 0.51
blocked 6670/20000, Snapshot.memoryBytes 3.0 MiB, VmRSS 5.4 MiB
filter 20000 2.38 2.76 2.88 20.32
blocked 6670/20000, 32 regex rules, Snapshot.memoryBytes 3.0 MiB, VmRSS 5.6 MiB
target p95 < 1ms: PASS
target VmRSS < 100 MiB: PASS
```
@@ -60,8 +60,8 @@ zig build bench -Doptimize=ReleaseFast -- cache --iters=20000
```
suite ops p50(us) p95(us) p99(us) max(us)
cache 20000 0.12 0.21 0.23 0.54
hits 10000/20000, DnsCache.memoryBytes 4.3 MiB, VmRSS 6.2 MiB
cache 20000 0.14 0.18 0.20 5.14
hits 10000/20000, DnsCache.memoryBytes 4.3 MiB, VmRSS 6.3 MiB
target p95 < 5ms: PASS
```
@@ -71,7 +71,7 @@ zig build bench -Doptimize=ReleaseFast -- compile --domains=100000
```
suite ops p50(us) p95(us) p99(us) max(us)
compile 100000 wall 11.512ms, 8686215 lines/s, 100000 domains kept (informational)
compile 100000 wall 15.623ms, 6400464 lines/s, 100000 domains kept (informational)
```
`--seed=N` changes the generated domains and the query order; the default is
@@ -94,6 +94,9 @@ usage: zig build bench -Doptimize=ReleaseFast -- [filter|cache|compile|all] [--d
is building the key, getting the entry and stamping the response id.
- `blocked N/M` and `hits N/M` are sanity counters. The harness aborts if either
is zero — a suite that never hits its own path measures nothing.
- `32 regex rules` on the `filter` line is the rule set the suite loads. No
generated query matches any of them, so every operation runs all 32 programs
to their end, which is the costly case and the one worth measuring.
- Two memory figures appear on purpose. `Snapshot.memoryBytes` and
`DnsCache.memoryBytes` are the in-repo accounting of those structures; `VmRSS`
is what the kernel holds resident for the whole process, allocator slack and
@@ -121,8 +124,8 @@ zig build bench -Doptimize=ReleaseFast -- filter --domains=100000 --iters=20000
```
```
filter 20000 0.14 0.26 0.27 2.42
blocked 6670/20000, Snapshot.memoryBytes 3.0 MiB, VmRSS 5.4 MiB
filter 20000 2.34 2.71 2.85 15.06
blocked 6670/20000, 32 regex rules, Snapshot.memoryBytes 3.0 MiB, VmRSS 5.6 MiB
target p95 < 1ms: PASS
target VmRSS < 100 MiB: PASS
```
docs/how-to/troubleshoot.md
+19 -1
View File
@@ -401,12 +401,30 @@ snapshot loaded but has no sources in it. The line
a source in the admin interface, or a `blocklist_sources` entry to the
configuration file with a `group_sources` link naming a group.
One name resolving while its neighbours are blocked is a third case, and
`/api/lookup` answers it directly: it reports which level of the filtering
ladder decided, and against what.
```sh
curl -s 'http://127.0.0.1:8080/api/lookup?domain=api.ads.tvb.com'
```
```json
{"domain":"api.ads.tvb.com","group_id":1,"local_records":false,"forward_zone":null,"blocked":false,"reason":"blocklist_exception","matched":"api.ads.tvb.com","source_url":"https://adguardteam.github.io/HostlistsRegistry/assets/filter_1.txt","safe_search_rewrite":null}
```
`blocklist_exception` means a downloaded list lifted that name with an `@@`
line, and `source_url` names the list that did it. Nothing is broken, and the
list is not overruling you: an exception cancels only what another list blocks.
Your own rule wins over it. Adding an exact block rule for the same name and
asking again reports `rule_block_exact`, `blocked` true and a null `source_url`.
## A database stamped by a newer binary
**Symptom.** After putting an older binary back, it will not start:
```
warning(migrations): config.db is at schema version 99; this nxdns binary supports 2
warning(migrations): config.db is at schema version 99; this nxdns binary supports 4
nxdns run failed: SchemaTooNew
```
docs/how-to/upgrade.md
+8 -8
View File
@@ -307,7 +307,7 @@ opens the database immutable and never migrates, so on a database still one
version behind it reports the mismatch and exits 2 rather than fixing it:
```
FAIL /var/lib/nxdns/config.db: schema version 0, this nxdns expects 2; `nxdns run` migrates it, `check` will not
FAIL /var/lib/nxdns/config.db: schema version 0, this nxdns expects 4; `nxdns run` migrates it, `check` will not
```
That line was reproduced here against a database stamped at version 0; the path
@@ -317,13 +317,13 @@ A fresh database is created at the current schema version; an older one is
stepped up to it. The log line names both versions:
```
info(migrations): config.db migrated from schema version 0 to 2
info(migrations): config.db migrated from schema version 0 to 4
```
> Verified on this host: that exact line is what `nxdns import` printed when it
> created the scratch database used throughout this page. An empty data
> directory is schema version 0, which is why a first run reports a migration
> rather than nothing. The step from a populated older schema to 2 was not
> rather than nothing. The step from a populated older schema to 4 was not
> reproduced here — it needs a database written by an older binary, which this
> host does not have.
@@ -332,14 +332,14 @@ binary refuses to open, so an older binary against an upgraded data directory
fails to start:
```
warning(migrations): config.db is at schema version 99; this nxdns binary supports 2
warning(migrations): config.db is at schema version 99; this nxdns binary supports 4
nxdns run failed: SchemaTooNew
```
> Not reproduced on this host: the same missing ingredient as above, a
> database at a schema version this binary does not support. The two lines
> are the messages `src/storage/migrations.zig` emits, not a run captured
> here.
> Reproduced on this host, with one substitution: no binary from the future was
> available, so the scratch database's `schema_version` row was set to 99 by
> hand and `nxdns run` was pointed at it. The two lines above are that run's
> output.
That run exits 1. Recovering means importing the export you took in step 1 into
a fresh data directory with the older binary.
docs/reference/api.md
+28
View File
@@ -277,3 +277,31 @@ included. The password then lives where the rest of the configuration lives: set
Request and response schemas for every operation live in the OpenAPI document:
`src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a
running server.
### Block reasons
Three places carry the same tag: `block_reason` on a `GET /api/queries` row,
`block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup`
answer. The tag names the level that decided the query, and the levels are
listed here in the order they are consulted — the first one that matches wins,
so a rule always outranks a list.
| Tag | Decided by |
| --- | --- |
| `rule_allow_exact` | An `exact` rule with action `allow` |
| `rule_block_exact` | An `exact` rule with action `block` |
| `rule_allow_wildcard` | A `wildcard` rule with action `allow` |
| `rule_block_wildcard` | A `wildcard` rule with action `block` |
| `rule_allow_regex` | A `regex` rule with action `allow` |
| `rule_block_regex` | A `regex` rule with action `block` |
| `blocklist_exception` | An `@@` exception line in a downloaded list |
| `blocklist_domain` | A plain name in a downloaded list |
| `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list |
`/api/lookup` also answers `none` when nothing matched. A query row never
carries `none`: `block_reason` is null unless the query was blocked.
A `cname:` prefix means the decision landed on a CNAME target rather than on
the name the client asked for, so `cname:blocklist_domain` reads as "the list
blocks a name this answer redirects to". Only `/api/queries` and the live
stream show the prefix; `/api/lookup` does not follow CNAMEs.
docs/reference/cli.md
+2 -2
View File
@@ -62,8 +62,8 @@ reconciled '/etc/nxdns/config.zon': no changes
Blocklist state is not declarative and survives every reconcile: a source whose
URL the file still names keeps its row id, its checksum, its counters and its
compiled `<id>.list` and `<id>.wild`, so a restart in file mode downloads
nothing. Editing a source's URL is a new identity — a new row, a new id, and a
compiled `<id>.list`, `<id>.wild` and `<id>.allow`, so a restart in file mode
downloads nothing. Editing a source's URL is a new identity — a new row, a new id, and a
fresh download.
### Failing to start in file mode
docs/reference/configuration.md
+47 -4
View File
@@ -279,6 +279,14 @@ Consumed by the blocklist manager (`src/filter/manager.zig`): downloaded by the
fetcher and compiled into domain sets. A disabled source is neither downloaded
nor loaded.
A list in Adblock Plus syntax may also carry exception lines, `@@||name^` and
`@@||name`, either of which may end in `$important`. Those become allow entries
that cancel what any attached list blocks, for the name and its subdomains. They
cancel nothing an operator decided: every rule of the table above is checked
first, so a downloaded list can reopen only a hole another downloaded list dug.
Each source reports how many it carried as `exceptions`; there is no way to write
one by hand, and no reason to want one — write an allow rule instead.
### group_sources
Which groups consult which blocklist sources.
@@ -299,15 +307,48 @@ Per-group allow and block overrides, checked before the blocklists.
|---|---|---|---|
| `group` | string | required | must name a declared group |
| `pattern` | string | required | see below |
| `kind` | enum `.exact` \| `.wildcard` | required | — |
| `kind` | enum `.exact` \| `.wildcard` \| `.regex` | required | — |
| `action` | enum `.allow` \| `.block` | required | — |
Pattern rules: an `.exact` pattern is a plain domain name and may not contain
`*`. A `.wildcard` pattern must contain at least one label that is exactly `*`
(`*.tracker.example`, or `*` alone), and every other label must be a legal DNS
label. `ads*.example` is not a valid wildcard.
label. `ads*.example` is not a valid wildcard; a partial label is what the
`.regex` kind is for.
Consumed by the filter engine's rule sets (`src/filter/rules.zig`).
A `.regex` pattern is a regular expression matched against the whole normalized
lowercase name, unanchored unless you write `^` or `$` — the POSIX-grep
convention. It is stored exactly as you typed it, which the other two kinds are
not: lowercasing would turn `\D` into `\d`, and trimming a trailing `.` would
delete an any-byte atom. The engine (`src/filter/regex.zig`) accepts literal
bytes, `.` for any byte, character classes `[a-z0-9]` with a leading `^` for
negation, the escapes `\d` and `\w` plus `\` before any other ASCII punctuation
to make it a literal, the repetitions `*` `+` `?` `{n}` `{n,m}` `{n,}`,
alternation `|`, grouping `(...)`, and the anchors `^` and `$`.
Everything else is refused at the edge rather than approximated, so a pattern
written for another engine fails where you can read the diagnostic instead of
silently matching names you did not mean:
- backreferences, lookaround, captures, named groups, Unicode classes and the
`(?…)` prefix they share;
- any alphanumeric escape the list above omits — `\s`, `\b`, `\1`, `\D`;
- a `]` inside a class, unless written `\]`;
- an empty pattern, and an empty branch: `ads|` is refused rather than read as a
pattern that matches every name;
- a quantifier applied straight to another quantifier: `a+?` is refused rather
than read as `(a+)?`, which matches every name. Write `(a+)?` to mean that.
A pattern is at most 256 bytes and compiles to at most 1024 instructions, each
limit with its own diagnostic, and one group holds at most 256 regex rules.
Groups do not capture, and the engine simulates every alternative in lockstep,
so a pattern costs at most its compiled length times the length of the name —
`(a+)+b` is as cheap here as it is expensive in a backtracking engine.
Consumed by the filter engine's rule sets (`src/filter/rules.zig`), which checks
the three kinds in the order they are listed above, allow before block within
each. Regex is checked last of the three because it is the only kind that costs
more than a hash lookup or a label walk.
### local_records
@@ -548,10 +589,12 @@ upstream. Everything else keeps its default.
.{ .group = "kids", .source_url = "https://lists.example/ads.txt" },
},
// Overrides beat blocklists. Wildcards need a label that is exactly "*".
// Overrides beat blocklists. Wildcards need a label that is exactly "*";
// a partial label takes a regex, which is unanchored unless you say "^".
.rules = .{
.{ .group = "default", .pattern = "allowed.example", .kind = .exact, .action = .allow },
.{ .group = "kids", .pattern = "*.tracker.example", .kind = .wildcard, .action = .block },
.{ .group = "kids", .pattern = "^ad[0-9]+-", .kind = .regex, .action = .block },
},
// Local names, answered without any upstream.
docs/reference/files-and-directories.md
+10 -8
View File
@@ -46,19 +46,21 @@ older ones from the main file. That is the "uncheckpointed changes" failure in
| `blocklists/` | Compiled blocklist snapshots, one subdirectory of the data directory. | 0700 |
| `blocklists/<id>.list` | Exact domains for blocklist source `<id>`, one per line, behind a header. | 0600 |
| `blocklists/<id>.wild` | Wildcard entries for the same source. | 0600 |
| `blocklists/<id>.raw.tmp`, `<id>.list.tmp`, `<id>.wild.tmp` | Transient refresh state: the downloaded body and the two compile outputs before they are published by rename. | 0600 |
| `blocklists/<id>.allow` | Exception entries for the same source: the names its `@@` lines lift. Absent on a source compiled before exceptions were honoured, which reads as empty. | 0600 |
| `blocklists/<id>.raw.tmp`, `<id>.list.tmp`, `<id>.wild.tmp`, `<id>.allow.tmp` | Transient refresh state: the downloaded body and the three compile outputs before they are published by rename. | 0600 |
`<id>` is the `blocklist_sources` row id.
### The orphan sweep
The sweep decides by id, not by suffix. It matches all five names above and
The sweep decides by id, not by suffix. It matches all seven names above and
deletes those whose `<id>` is no longer a `blocklist_sources` row, so the
compiled `.list` and `.wild` of a removed source go, and so do a `.raw.tmp`,
`.list.tmp` or `.wild.tmp` left behind by a refresh that was killed before it
could clean up. Files belonging to a source that still has a row are never
touched, whatever state they are in: the sweep holds the same lock every refresh
takes, so it never reads the directory while a refresh is part-way through.
compiled `.list`, `.wild` and `.allow` of a removed source go, and so do a
`.raw.tmp`, `.list.tmp`, `.wild.tmp` or `.allow.tmp` left behind by a refresh
that was killed before it could clean up. Files belonging to a source that still
has a row are never touched, whatever state they are in: the sweep holds the same
lock every refresh takes, so it never reads the directory while a refresh is
part-way through.
It runs at three moments:
@@ -86,7 +88,7 @@ losing the refresh pass behind them, let alone the server.
The temporaries of a source that still exists are cleaned by the refresh that
owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`,
`.list.tmp` and `.wild.tmp` as it finishes, successfully or not.
`.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not.
A `querylog.db` is moved aside when it is missing nothing but usability:
SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not
docs/reference/performance.md
+16 -8
View File
@@ -27,7 +27,7 @@ the asset-free figure is a real measurement rather than an estimate.
## Measured: x86_64 development host
Date: 2026-08-02. Hardware and build: Intel Core i7-14700K, Linux 6.18,
Date: 2026-08-13. Hardware and build: Intel Core i7-14700K, Linux 6.18,
Zig 0.16.0, `-Doptimize=ReleaseFast`, harness defaults (1,000,000 domains,
200,000 iterations per suite, seed 0x5eed).
@@ -37,18 +37,26 @@ baseline for regressions on the machine development happens on.
```
suite ops p50(us) p95(us) p99(us) max(us)
filter 200000 0.11 0.18 0.27 16.41
blocked 66699/200000, Snapshot.memoryBytes 28.0 MiB, VmRSS 31.8 MiB
filter 200000 2.41 2.80 2.97 22.16
blocked 66699/200000, 32 regex rules, Snapshot.memoryBytes 28.0 MiB, VmRSS 32.0 MiB
target p95 < 1ms: PASS
target VmRSS < 100 MiB: PASS
cache 200000 0.10 0.14 0.17 3.53
hits 100000/200000, DnsCache.memoryBytes 4.3 MiB, VmRSS 7.6 MiB
cache 200000 0.11 0.17 0.22 5.40
hits 100000/200000, DnsCache.memoryBytes 4.3 MiB, VmRSS 7.7 MiB
target p95 < 5ms: PASS
compile 1000000 wall 96.025ms, 10413949 lines/s, 1000000 domains kept (informational)
compile 1000000 wall 98.597ms, 10142224 lines/s, 1000000 domains kept (informational)
```
Every in-process §18 target passes on this host: the two latency targets by
three to four orders of magnitude, the memory target by about 3x.
Every in-process §18 target passes on this host: the filter target by about
360x, the cache target by about four orders of magnitude, the memory target by
about 3x.
The filter suite loads 32 regex rules that no query in the mix matches, which is
the expensive case rather than the cheap one: the regex levels sit below every
hash and wildcard level, so a name no pattern matches is the name that runs all
32 programs to their end. Every op pays that, which is what moved the filter p95
from 0.18 µs before regex rules existed to the 2.80 µs above. The margin against
the 1 ms target is what makes paying it on every miss an acceptable price.
### The two memory figures
docs/tutorial/first-run.md
+42 -25
View File
@@ -9,13 +9,13 @@ delete the directory.
Follow the steps in order. Each one says what it did.
Every command below was executed on x86_64 Linux with Zig 0.16.0, Node.js
24.14.1, dig 9.20.26 and curl 8.21.0. Steps 4 to 11, 13 and 14 were re-run end
to end for this revision, and the transcripts are that run's output with the
24.14.1, dig 9.20.26 and curl 8.21.0. Steps 2, 4 to 11, 13 and 14 were re-run
end to end for this revision, and the transcripts are that run's output with the
tutorial directory substituted. Two things were not re-run: the browser page in
step 12 — its endpoints were exercised, the page itself was not opened — and the
two build commands in steps 1 and 2, which had already produced the binary under
test. The ZON block at the end of step 14 was checked with `nxdns check
--config` rather than started.
`npm` build in step 1, whose `web/dist` was already on disk and is the one the
binary under test embeds. The ZON block at the end of step 14 was checked with
`nxdns check --config` rather than started.
## What you need
@@ -109,7 +109,7 @@ zig-out/bin/nxdns import ~/nxdns-tutorial/config.zon --data-dir ~/nxdns-tutorial
```
```
info(migrations): config.db migrated from schema version 0 to 2
info(migrations): config.db migrated from schema version 0 to 4
imported /home/you/nxdns-tutorial/config.zon
```
@@ -127,7 +127,7 @@ zig-out/bin/nxdns run --data-dir ~/nxdns-tutorial/data
```
info(querylog_schema): created querylog database '/home/you/nxdns-tutorial/data/querylog.db'
info(blocklist_manager): blocklist snapshot generation 1: 0 of 0 sources loaded, 199 bytes
info(blocklist_manager): blocklist snapshot generation 1: 0 of 0 sources loaded, 231 bytes
info(nxdns): authority: database
info(nxdns): nxdns <version> serving on udp [::1]:15353 udp 127.0.0.1:15353 tcp [::1]:15353 tcp 127.0.0.1:15353; 1 upstream(s); blocklist generation 1
info(web_server): web interface listening on 127.0.0.1:8080
@@ -146,8 +146,8 @@ dig @127.0.0.1 -p 15353 example.com A +noall +answer
```
```
example.com. 90 IN A 172.66.147.243
example.com. 90 IN A 104.20.23.154
example.com. 229 IN A 172.66.147.243
example.com. 229 IN A 104.20.23.154
```
nxdns had no answer cached, so it forwarded the query to
@@ -220,20 +220,27 @@ curl -s -X POST http://127.0.0.1:8080/api/blocklists/update
```
```json
{"sources":[{"id":1,"state":"ok","loaded":true,"last_attempt":1786473715,"last_success":1786473715,"url":"https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts","last_error":"","domains":99559,"wildcards":0,"skipped_regex":0}]}
{"sources":[{"id":1,"state":"ok","loaded":true,"last_attempt":1786629237,"last_success":1786629238,"url":"https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts","last_error":"","domains":97648,"wildcards":0,"exceptions":0,"skipped_regex":0}]}
```
The three zeros are the parts of this list that a hosts file cannot have.
`wildcards` counts entries covering a name and its subdomains, `exceptions`
counts the `@@` lines an Adblock Plus list uses to lift a name another list
blocks, and `skipped_regex` counts the regex lines nxdns declines to take from a
downloaded list. Only the Adblock Plus syntax writes any of them; a hosts file is
one name per line.
The download is about 3 MB and takes a few seconds. Watch the first terminal
until this appears:
```
info(blocklist_manager): blocklist snapshot generation 6: 1 of 1 sources loaded, 3096006 bytes
info(blocklist_manager): blocklist snapshot generation 6: 1 of 1 sources loaded, 2523973 bytes
```
`1 of 1 sources loaded` is the line to wait for. nxdns builds each blocklist
snapshot in full and swaps it in atomically, so queries keep being answered from
the previous snapshot the whole time the new one is being built. After this, the
domain count in the JSON above — 99559 on the day this was run — is live.
domain count in the JSON above — 97648 on the day this was run — is live.
From here on, the list is on disk under `~/nxdns-tutorial/data/blocklists`.
Restarting nxdns does not re-download it.
@@ -259,16 +266,18 @@ dig @127.0.0.1 -p 15353 wikipedia.org A +noall +answer
```
```
wikipedia.org. 17 IN A 185.15.58.224
wikipedia.org. 130 IN A 185.15.58.224
```
One thing to know before you try other names: a blocklist entry blocks exactly
the name it names. `doubleclick.net` on the list does not block
`ads.doubleclick.net`; that name is blocked because the list happens to contain
it too. Blocklist entries do not walk up the parent chain — only rules you write
yourself can, with a wildcard pattern such as `*.doubleclick.net`. So when you
pick a domain to test, pick one that is literally in the file.
`www.google-analytics.com` is another that is.
One thing to know before you try other names: an entry in a hosts list blocks
exactly the name it names. `doubleclick.net` is on this list, and so are
`ad.doubleclick.net` and `www.google-analytics.com`. `ads.doubleclick.net` is
not on it, and nxdns does not block it — the entry for the parent says nothing
about the child. Two things do walk up the parent chain, and neither is in play
here: a rule you write yourself, with a wildcard pattern such as
`*.doubleclick.net`, and an Adblock Plus list's `||doubleclick.net^`, which
covers the name and everything under it. This list is a hosts file, so when you
pick a domain to test against it, pick one that is literally in the file.
## 12. Open the web interface
@@ -291,7 +300,7 @@ nxdns catches SIGINT and SIGTERM, stops serving and exits 0.
Start it again with the same command as in step 6 and read the first log lines:
```
info(blocklist_manager): blocklist snapshot generation 1: 1 of 1 sources loaded, 3096006 bytes
info(blocklist_manager): blocklist snapshot generation 1: 1 of 1 sources loaded, 2523973 bytes
info(nxdns): authority: database
```
@@ -314,8 +323,12 @@ zig-out/bin/nxdns run --data-dir ~/nxdns-tutorial/data --config ~/nxdns-tutorial
```
reconciled '/home/you/nxdns-tutorial/config.zon': sources +0 ~0 -1; group_sources +0 ~0 -1;
info(blocklist_manager): blocklist snapshot generation 1: 0 of 0 sources loaded, 199 bytes
info(blocklist_manager): blocklist snapshot generation 1: 0 of 0 sources loaded, 231 bytes
info(nxdns): authority: file (/home/you/nxdns-tutorial/config.zon)
info(nxdns): nxdns <version> serving on udp [::1]:15353 udp 127.0.0.1:15353 tcp [::1]:15353 tcp 127.0.0.1:15353; 1 upstream(s); blocklist generation 1
info(blocklist_manager): pruned orphaned blocklist file 1.allow
info(blocklist_manager): pruned orphaned blocklist file 1.wild
info(blocklist_manager): pruned orphaned blocklist file 1.list
```
**Read that first line.** The blocklist source is gone. That is not a bug — it is
@@ -324,8 +337,12 @@ source, and in file mode the file is the complete statement of what the
configuration is, so anything the database holds that the file does not name is
removed at every start. The reconcile said so in one line before doing it.
The compiled list is still on disk and the query log is untouched; what changed
is the configuration, and it now matches the file exactly.
The three `pruned` lines are the rest of that removal: with the row gone, the
compiled files it owned belong to nobody, so the sweep that runs at every start
deletes them. The three names are the three bodies one source compiles into —
exact domains, wildcards, and the exceptions an Adblock Plus list can lift. The
query log is untouched; what changed is the configuration, and it now matches
the file exactly.
Neither mode is the "advanced" one. Database mode suits a box someone
administers through the web interface. File mode suits a file kept in git and
@@ -345,7 +362,7 @@ Ctrl-C to stop it.
## What you have now
A resolver that answers real queries, a real blocklist of about 99000 domains
A resolver that answers real queries, a real blocklist of about 98000 domains
attached to the default group, a query log, and a web interface — all inside one
directory you can delete: