mokhtar/nxdns
1
0
Fork 0
Code Pull Requests Actions Packages Releases 13 Activity

milestone 21: abp list exceptions and a regex rule kind

Browse Source
This commit is contained in:
mokhtar
2026-08-13 19:14:47 +02:00
parent b340521716
commit 2ab7c1f1de
51 changed files with 4016 additions and 465 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/reference/api.md
+28
View File
@@ -277,3 +277,31 @@ included. The password then lives where the rest of the configuration lives: set
Request and response schemas for every operation live in the OpenAPI document:
`src/web/openapi.yaml` in the repository, or `GET /api/openapi.yaml` from a
running server.
### Block reasons
Three places carry the same tag: `block_reason` on a `GET /api/queries` row,
`block_reason` on a live-stream frame, and `reason` on a `GET /api/lookup`
answer. The tag names the level that decided the query, and the levels are
listed here in the order they are consulted — the first one that matches wins,
so a rule always outranks a list.
| Tag | Decided by |
| --- | --- |
| `rule_allow_exact` | An `exact` rule with action `allow` |
| `rule_block_exact` | An `exact` rule with action `block` |
| `rule_allow_wildcard` | A `wildcard` rule with action `allow` |
| `rule_block_wildcard` | A `wildcard` rule with action `block` |
| `rule_allow_regex` | A `regex` rule with action `allow` |
| `rule_block_regex` | A `regex` rule with action `block` |
| `blocklist_exception` | An `@@` exception line in a downloaded list |
| `blocklist_domain` | A plain name in a downloaded list |
| `blocklist_wildcard` | A domain anchor (`||name^`) in a downloaded list |
`/api/lookup` also answers `none` when nothing matched. A query row never
carries `none`: `block_reason` is null unless the query was blocked.
A `cname:` prefix means the decision landed on a CNAME target rather than on
the name the client asked for, so `cname:blocklist_domain` reads as "the list
blocks a name this answer redirects to". Only `/api/queries` and the live
stream show the prefix; `/api/lookup` does not follow CNAMEs.
docs/reference/cli.md
+2 -2
View File
@@ -62,8 +62,8 @@ reconciled '/etc/nxdns/config.zon': no changes
Blocklist state is not declarative and survives every reconcile: a source whose
URL the file still names keeps its row id, its checksum, its counters and its
compiled `<id>.list` and `<id>.wild`, so a restart in file mode downloads
nothing. Editing a source's URL is a new identity — a new row, a new id, and a
compiled `<id>.list`, `<id>.wild` and `<id>.allow`, so a restart in file mode
downloads nothing. Editing a source's URL is a new identity — a new row, a new id, and a
fresh download.
### Failing to start in file mode
docs/reference/configuration.md
+47 -4
View File
@@ -279,6 +279,14 @@ Consumed by the blocklist manager (`src/filter/manager.zig`): downloaded by the
fetcher and compiled into domain sets. A disabled source is neither downloaded
nor loaded.
A list in Adblock Plus syntax may also carry exception lines, `@@||name^` and
`@@||name`, either of which may end in `$important`. Those become allow entries
that cancel what any attached list blocks, for the name and its subdomains. They
cancel nothing an operator decided: every rule of the table above is checked
first, so a downloaded list can reopen only a hole another downloaded list dug.
Each source reports how many it carried as `exceptions`; there is no way to write
one by hand, and no reason to want one — write an allow rule instead.
### group_sources
Which groups consult which blocklist sources.
@@ -299,15 +307,48 @@ Per-group allow and block overrides, checked before the blocklists.
|---|---|---|---|
| `group` | string | required | must name a declared group |
| `pattern` | string | required | see below |
| `kind` | enum `.exact` \| `.wildcard` | required | — |
| `kind` | enum `.exact` \| `.wildcard` \| `.regex` | required | — |
| `action` | enum `.allow` \| `.block` | required | — |
Pattern rules: an `.exact` pattern is a plain domain name and may not contain
`*`. A `.wildcard` pattern must contain at least one label that is exactly `*`
(`*.tracker.example`, or `*` alone), and every other label must be a legal DNS
label. `ads*.example` is not a valid wildcard.
label. `ads*.example` is not a valid wildcard; a partial label is what the
`.regex` kind is for.
Consumed by the filter engine's rule sets (`src/filter/rules.zig`).
A `.regex` pattern is a regular expression matched against the whole normalized
lowercase name, unanchored unless you write `^` or `$` — the POSIX-grep
convention. It is stored exactly as you typed it, which the other two kinds are
not: lowercasing would turn `\D` into `\d`, and trimming a trailing `.` would
delete an any-byte atom. The engine (`src/filter/regex.zig`) accepts literal
bytes, `.` for any byte, character classes `[a-z0-9]` with a leading `^` for
negation, the escapes `\d` and `\w` plus `\` before any other ASCII punctuation
to make it a literal, the repetitions `*` `+` `?` `{n}` `{n,m}` `{n,}`,
alternation `|`, grouping `(...)`, and the anchors `^` and `$`.
Everything else is refused at the edge rather than approximated, so a pattern
written for another engine fails where you can read the diagnostic instead of
silently matching names you did not mean:
- backreferences, lookaround, captures, named groups, Unicode classes and the
`(?…)` prefix they share;
- any alphanumeric escape the list above omits — `\s`, `\b`, `\1`, `\D`;
- a `]` inside a class, unless written `\]`;
- an empty pattern, and an empty branch: `ads|` is refused rather than read as a
pattern that matches every name;
- a quantifier applied straight to another quantifier: `a+?` is refused rather
than read as `(a+)?`, which matches every name. Write `(a+)?` to mean that.
A pattern is at most 256 bytes and compiles to at most 1024 instructions, each
limit with its own diagnostic, and one group holds at most 256 regex rules.
Groups do not capture, and the engine simulates every alternative in lockstep,
so a pattern costs at most its compiled length times the length of the name —
`(a+)+b` is as cheap here as it is expensive in a backtracking engine.
Consumed by the filter engine's rule sets (`src/filter/rules.zig`), which checks
the three kinds in the order they are listed above, allow before block within
each. Regex is checked last of the three because it is the only kind that costs
more than a hash lookup or a label walk.
### local_records
@@ -548,10 +589,12 @@ upstream. Everything else keeps its default.
.{ .group = "kids", .source_url = "https://lists.example/ads.txt" },
},
// Overrides beat blocklists. Wildcards need a label that is exactly "*".
// Overrides beat blocklists. Wildcards need a label that is exactly "*";
// a partial label takes a regex, which is unanchored unless you say "^".
.rules = .{
.{ .group = "default", .pattern = "allowed.example", .kind = .exact, .action = .allow },
.{ .group = "kids", .pattern = "*.tracker.example", .kind = .wildcard, .action = .block },
.{ .group = "kids", .pattern = "^ad[0-9]+-", .kind = .regex, .action = .block },
},
// Local names, answered without any upstream.
docs/reference/files-and-directories.md
+10 -8
View File
@@ -46,19 +46,21 @@ older ones from the main file. That is the "uncheckpointed changes" failure in
| `blocklists/` | Compiled blocklist snapshots, one subdirectory of the data directory. | 0700 |
| `blocklists/<id>.list` | Exact domains for blocklist source `<id>`, one per line, behind a header. | 0600 |
| `blocklists/<id>.wild` | Wildcard entries for the same source. | 0600 |
| `blocklists/<id>.raw.tmp`, `<id>.list.tmp`, `<id>.wild.tmp` | Transient refresh state: the downloaded body and the two compile outputs before they are published by rename. | 0600 |
| `blocklists/<id>.allow` | Exception entries for the same source: the names its `@@` lines lift. Absent on a source compiled before exceptions were honoured, which reads as empty. | 0600 |
| `blocklists/<id>.raw.tmp`, `<id>.list.tmp`, `<id>.wild.tmp`, `<id>.allow.tmp` | Transient refresh state: the downloaded body and the three compile outputs before they are published by rename. | 0600 |
`<id>` is the `blocklist_sources` row id.
### The orphan sweep
The sweep decides by id, not by suffix. It matches all five names above and
The sweep decides by id, not by suffix. It matches all seven names above and
deletes those whose `<id>` is no longer a `blocklist_sources` row, so the
compiled `.list` and `.wild` of a removed source go, and so do a `.raw.tmp`,
`.list.tmp` or `.wild.tmp` left behind by a refresh that was killed before it
could clean up. Files belonging to a source that still has a row are never
touched, whatever state they are in: the sweep holds the same lock every refresh
takes, so it never reads the directory while a refresh is part-way through.
compiled `.list`, `.wild` and `.allow` of a removed source go, and so do a
`.raw.tmp`, `.list.tmp`, `.wild.tmp` or `.allow.tmp` left behind by a refresh
that was killed before it could clean up. Files belonging to a source that still
has a row are never touched, whatever state they are in: the sweep holds the same
lock every refresh takes, so it never reads the directory while a refresh is
part-way through.
It runs at three moments:
@@ -86,7 +88,7 @@ losing the refresh pass behind them, let alone the server.
The temporaries of a source that still exists are cleaned by the refresh that
owns them rather than by the sweep: each refresh deletes its own `.raw.tmp`,
`.list.tmp` and `.wild.tmp` as it finishes, successfully or not.
`.list.tmp`, `.wild.tmp` and `.allow.tmp` as it finishes, successfully or not.
A `querylog.db` is moved aside when it is missing nothing but usability:
SQLite reports it corrupt or not a database, `PRAGMA quick_check` does not
docs/reference/performance.md
+16 -8
View File
@@ -27,7 +27,7 @@ the asset-free figure is a real measurement rather than an estimate.
## Measured: x86_64 development host
Date: 2026-08-02. Hardware and build: Intel Core i7-14700K, Linux 6.18,
Date: 2026-08-13. Hardware and build: Intel Core i7-14700K, Linux 6.18,
Zig 0.16.0, `-Doptimize=ReleaseFast`, harness defaults (1,000,000 domains,
200,000 iterations per suite, seed 0x5eed).
@@ -37,18 +37,26 @@ baseline for regressions on the machine development happens on.
```
suite ops p50(us) p95(us) p99(us) max(us)
filter 200000 0.11 0.18 0.27 16.41
blocked 66699/200000, Snapshot.memoryBytes 28.0 MiB, VmRSS 31.8 MiB
filter 200000 2.41 2.80 2.97 22.16
blocked 66699/200000, 32 regex rules, Snapshot.memoryBytes 28.0 MiB, VmRSS 32.0 MiB
target p95 < 1ms: PASS
target VmRSS < 100 MiB: PASS
cache 200000 0.10 0.14 0.17 3.53
hits 100000/200000, DnsCache.memoryBytes 4.3 MiB, VmRSS 7.6 MiB
cache 200000 0.11 0.17 0.22 5.40
hits 100000/200000, DnsCache.memoryBytes 4.3 MiB, VmRSS 7.7 MiB
target p95 < 5ms: PASS
compile 1000000 wall 96.025ms, 10413949 lines/s, 1000000 domains kept (informational)
compile 1000000 wall 98.597ms, 10142224 lines/s, 1000000 domains kept (informational)
```
Every in-process §18 target passes on this host: the two latency targets by
three to four orders of magnitude, the memory target by about 3x.
Every in-process §18 target passes on this host: the filter target by about
360x, the cache target by about four orders of magnitude, the memory target by
about 3x.
The filter suite loads 32 regex rules that no query in the mix matches, which is
the expensive case rather than the cheap one: the regex levels sit below every
hash and wildcard level, so a name no pattern matches is the name that runs all
32 programs to their end. Every op pays that, which is what moved the filter p95
from 0.18 µs before regex rules existed to the 2.80 µs above. The margin against
the 1 ms target is what makes paying it on every miss an acceptable price.
### The two memory figures