15 Commits
Author SHA1 Message Date
mokhtar f4562cac26 filter: skip the read-only directory test as root, ci's runner ignores mode bits
Gates / frontend (push) Successful in 2m3s
Gates / test (push) Successful in 2m33s
Gates / test-aarch64 (push) Successful in 8m14s
Gates / package (push) Successful in 52s
Gates / container (push) Successful in 14s
CI / gates (push) Successful in 12m0s
Release / guard (push) Successful in 33s
Gates / frontend (push) Successful in 2m36s
Gates / test (push) Successful in 3m47s
Gates / test-aarch64 (push) Successful in 7m24s
Gates / package (push) Successful in 50s
Gates / container (push) Successful in 15s
Release / gates (push) Successful in 12m19s
Release / publish (push) Successful in 6m37s
2026-09-12 10:43:02 +02:00
mokhtar 250bdca7e7 build: bump version to 0.0.19
Gates / frontend (push) Successful in 1m58s
Gates / test (push) Failing after 2m31s
Gates / test-aarch64 (push) Successful in 8m21s
Gates / package (push) Successful in 4m20s
Gates / container (push) Successful in 10s
CI / gates (push) Failing after 37m36s
2026-09-12 10:01:19 +02:00
mokhtar 63f4f96a48 changelog: date the 0.0.19 section
Gates / frontend (push) Successful in 1m56s
Gates / test (push) Failing after 2m22s
Gates / test-aarch64 (push) Successful in 7m30s
Gates / package (push) Successful in 47s
Gates / container (push) Successful in 9s
CI / gates (push) Failing after 39m22s
2026-09-12 09:59:13 +02:00
mokhtar 52158198cf filter: a failed blocklist download names its phase, cause, status, bytes and elapsed time
Gates / frontend (push) Successful in 2m20s
Gates / test (push) Failing after 2m41s
Gates / test-aarch64 (push) Successful in 8m19s
Gates / package (push) Successful in 4m24s
Gates / container (push) Successful in 16s
CI / gates (push) Failing after 41m33s
2026-09-12 09:58:59 +02:00
mokhtar bf79a22584 docs: renovate's nix manager does not bump a tag ref; the regex manager recipe that does 2026-09-12 00:38:45 +02:00
mokhtar bcd9bce16c build: bump version to 0.0.18
Gates / frontend (push) Successful in 2m34s
Gates / test (push) Successful in 2m39s
Gates / test-aarch64 (push) Successful in 8m16s
Gates / package (push) Successful in 4m21s
Gates / container (push) Successful in 16s
CI / gates (push) Successful in 15m39s
Release / guard (push) Successful in 35s
Gates / frontend (push) Successful in 2m1s
Gates / test (push) Successful in 2m28s
Gates / test-aarch64 (push) Successful in 7m24s
Gates / package (push) Successful in 52s
Gates / container (push) Successful in 11s
Release / gates (push) Successful in 10m59s
Release / publish (push) Successful in 5m11s
2026-09-09 19:25:08 +02:00
mokhtar 2b790c4c3e changelog: 0.0.18 section 2026-09-09 19:23:07 +02:00
mokhtar 1860ff59f5 admin: the overview pickers fill their wrappers and share a narrow row 2026-09-09 19:04:44 +02:00
mokhtar f168247b33 cut: build the release with the official zig tarball, skip hidden bundle files; re-pin 0.0.17
Gates / frontend (push) Successful in 2m23s
Gates / test (push) Successful in 3m15s
Gates / test-aarch64 (push) Successful in 8m38s
Gates / package (push) Successful in 5m0s
Gates / container (push) Successful in 19s
CI / gates (push) Successful in 17m17s
Release / guard (push) Successful in 37s
Gates / frontend (push) Successful in 2m15s
Gates / test (push) Successful in 2m34s
Gates / test-aarch64 (push) Successful in 7m33s
Gates / package (push) Successful in 51s
Gates / container (push) Successful in 10s
Release / gates (push) Successful in 11m14s
Release / publish (push) Successful in 8m35s
The first 0.0.17 cut (run 687) failed verify-pins in CI for two reasons. The asset generator embedded admin/dist/.src-hash, a freshness stamp that CI's artifact copy does not carry; it now skips dotfiles. And the Arch zig package emits different code than the ziglang.org tarball that CI installs, so the cut downloads the pinned tarball (ZIG_TARBALL_SHA256 in gates.yml, the full digest keys the cache) and builds the release with it. flake.nix is re-pinned to the bytes both now produce.

The saturated-primary pool test gates its holders on a semaphore instead of sleeps and releases every spawned holder on the way out, so a loaded runner cannot flake it. The package job uploads the payload before the pin check and runs the check when the version or flake.nix changed against the parent. The verify-a-release recipe clones the tag first and builds with the official zig.
2026-09-08 23:57:07 +02:00
mokhtar d596fd788a build: bump version to 0.0.17
Gates / frontend (push) Successful in 3m10s
Gates / test (push) Failing after 3m36s
Gates / test-aarch64 (push) Successful in 8m46s
Gates / package (push) Failing after 6m15s
Gates / container (push) Skipped
CI / gates (push) Failing after 18m40s
2026-09-08 23:06:52 +02:00
mokhtar f067742adf cut: the pin stage compiles, and zig build test builds the cut binary
Two errors the test build never saw, because the test binary analyses only what the tests reference: `Io.Dir.makePath` does not exist in 0.16.0 (`createDirPath` does), and the tag push passed the optional authorization where the checked token was meant. The test step now depends on the cut compile as well.
2026-09-08 23:04:51 +02:00
mokhtar a4eb749fa7 changelog: 0.0.17 2026-09-08 23:02:33 +02:00
mokhtar 22abcd9b7b release: nix flake with tag-pinned hashes, reproducible tarballs (milestone 40)
flake.nix fetches the release tarballs and carries their SRI hashes in a generated block. The cut tool builds the release locally with the toolchain gates.yml pins, in a normalized nine-variable environment, writes the hashes into flake.nix, and commits it with build.zig.zon as the single bump commit. The package job verifies the pins on the bump commit and the publish job verifies them again on the tag, before anything is uploaded.

The tarballs are written by dist_stage (std.tar.Writer, flate gzip) instead of the runner's tar and gzip, and -ffile-prefix-map keeps checkout paths out of the C objects; two checkouts at different absolute paths produce byte-identical archives. nxdns version, /api/version and the admin footer report the version only: the bump commit cannot know its own sha.
2026-09-08 21:45:22 +02:00
mokhtar 3e57f43e08 cut: rerun a flaky release run once, resume after the tag
The v0.0.16 cut pushed the tag and then reported a failure because the release.yml run lost one gate to a timing-flaky test while its guard had passed and its publish job never ran. Rerunning the run by hand published the release; the tool had exited 1 and could not be run again, since a tag on origin meant "derive the next version".

The release stage now classifies a failed run from forge state: retryable when the conclusion is failure, the run is on its first attempt, every commit-status context in a failure state is a gate, and no release is published for the tag. A retryable run is rerun once through the actions API and the same run id is watched again, restarted only once the forge reports a higher run_attempt, so a concluded previous attempt is never read as the result. The rerun decision parses statuses strictly and treats anything it cannot read as terminal. A tag already on origin that peels to HEAD with no published release resumes at the release stage after the origin tag is verified (annotated, tool convention, pinned signing fingerprint). Both decisions are pure, table-tested functions. Spec bullets in specs/release-cut.md.
2026-09-08 01:47:09 +02:00
mokhtar 299d7af99c upstream: overlap attribution test builds its schedule with gates
The test "overlapping exchanges each report the entry that answered that call" failed once on a loaded release runner and passed on rerun. It started two exchanges back to back and relied on a 50 ms failing stall to keep the first in flight until the second arrived; nothing checked that the two calls were ever in flight together, and the failing schedule was never observed.

Two Fake behaviours, hold and hold_fail, wait on a test-owned std.Io.Semaphore and then reply or fail. The test holds both calls inside the first entry, waits on the entry's in_flight count for one and then two, releases the failing call so it fails over while the other is still held, then releases the other. The fake draws its behaviour before it raises in_flight, so a count of one also fixes which call holds which gate. Each gate has a deferred post registered after its task's deferred await, so any early return releases the held call instead of deadlocking. No sleep remains in the test; every assertion is kept. No pool bug was found. Addendum in specs/milestone-31.md.
2026-09-08 01:14:26 +02:00
53 changed files with 3628 additions and 355 deletions
+90 -4
View File
@@ -41,9 +41,20 @@ on:
env:
ZIG_VERSION: "0.16.0"
# The bytes of that zig, not just its version string: a distribution package
# of the same version is built against a different LLVM and emits different
# machine code, so its tarballs hash to something no CI run reproduces.
# setup-zig verifies these same bytes by minisign; `zig build cut` checks this
# digest before it builds a release.
ZIG_TARBALL_SHA256: "70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00"
# Exact patch, not a floating "24" (milestone-14 ruling 12): the bundled npm
# and the emitted bundle change under a floating major.
NODE_VERSION: "24.19.0"
# The npm the cut runs and the npm CI runs must be one version: npm writes the
# admin bundle whose bytes the release hashes are pinned to before CI ever
# rebuilds them. setup-node installs the npm that ships with the node above,
# so this pin is asserted, not installed.
NPM_VERSION: "11.17.0"
# There is deliberately no CI_VERSION literal here. Ruling 2 allows the
# version to exist in the tag and in build.zig.zon and nowhere else, and
# ruling 5 makes verify-dist fail when the version under build disagrees with
@@ -126,9 +137,25 @@ jobs:
cache: npm
cache-dependency-path: admin/package-lock.json
# The bundle is a release input whose hash is pinned before this run
# exists, so a runner on a different node or npm must fail here rather
# than emit different bytes further down.
- name: Assert the pinned Node and npm
run: |
test "$(node --version)" = "v${NODE_VERSION:?}"
test "$(npm --version)" = "${NPM_VERSION:?}"
test ! -e /nonexistent/npmrc-user
test ! -e /nonexistent/npmrc-global
# `npm ci` and `npm run build` run under the same normalized environment
# the cut builds the bundle in: exactly nine variables, a file mode from
# the umask, a C locale, UTC, and a zero build timestamp. Each of those
# can move the bytes the release hashes cover. The format, lint,
# typecheck and test steps below stay ambient on purpose: only the
# bundle's bytes are pinned, and those checks emit nothing that ships.
- name: Install dependencies
working-directory: admin
run: npm ci
run: env -i PATH="$PATH" HOME="$HOME" LC_ALL=C LANG=C TZ=UTC SOURCE_DATE_EPOCH=0 CI=true npm_config_userconfig=/nonexistent/npmrc-user npm_config_globalconfig=/nonexistent/npmrc-global sh -c 'umask 022 && npm ci'
- name: Check formatting
working-directory: admin
@@ -148,7 +175,7 @@ jobs:
- name: Build
working-directory: admin
run: npm run build
run: env -i PATH="$PATH" HOME="$HOME" LC_ALL=C LANG=C TZ=UTC SOURCE_DATE_EPOCH=0 CI=true npm_config_userconfig=/nonexistent/npmrc-user npm_config_globalconfig=/nonexistent/npmrc-global sh -c 'umask 022 && npm run build'
# The licence inventory has to cover every package whose bytes ship, and
# the lockfile does not answer that question: it lists what could be
@@ -191,7 +218,12 @@ jobs:
version: ${{ steps.zon-version.outputs.version }}
steps:
# Depth 2, not the default 1: the pin check at the foot of this job runs
# only on the commit that changed build.zig.zon, and answering that
# question needs HEAD's first parent. No other job here reads a parent.
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
with:
fetch-depth: 2
- name: Set up Zig
uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2.2.1
@@ -243,7 +275,6 @@ jobs:
set -euo pipefail
zig build dist \
-Dversion-string="$CI_VERSION" \
-Dgit-commit="$GITHUB_SHA" \
-Dadmin-dist=admin-dist-ci \
-Doptimize=ReleaseSafe
@@ -259,10 +290,14 @@ jobs:
set -euo pipefail
zig build verify-dist \
-Dversion-string="$CI_VERSION" \
-Dgit-commit="$GITHUB_SHA" \
-Dadmin-dist=admin-dist-ci \
-Doptimize=ReleaseSafe
# This upload runs before the pin check below: a pin mismatch is exactly
# the failure whose diagnosis needs the built bytes, so they must already
# be downloadable when that step fails. The container job's dependency on
# the artifact is unchanged.
#
# deploy/docker/Dockerfile copies both of these trees and nothing else
# out of zig-out/dist: the binary comes from dist/bin/<triple>/, and
# /LICENSE and /THIRD-PARTY-NOTICES come from the matching dist/stage/
@@ -290,6 +325,57 @@ jobs:
zig-out/bin/container_check
if-no-files-found: error
# The cut writes the release hashes into flake.nix before it makes the
# bump commit, so the bump commit is the one commit whose pins nothing has
# verified yet — the tag's own run (release.yml) is the next chance, and by
# then the tag is public. This step is that first chance.
#
# It is deliberately NOT part of verify-dist. Every other commit on master
# builds the same build.zig.zon version from a different tree, so its bytes
# legitimately differ from the pins and a check there would fail the whole
# branch. The commit is identified by the version it declares, not by its
# message: a message is a string anyone can write, and the pins follow
# the manifest.
#
# A root commit has no first parent. That is an error rather than a skip:
# this repository has history, so `HEAD^` failing means the checkout is
# shallower than the depth 2 declared above and the question went
# unanswered, which must never read as "nothing to check".
#
# The predicate is the declared VERSION, not the file: build.zig.zon also
# carries the dependency pins, and updating a sqlite or mbedTLS hash
# changes the file without cutting a release. Such a commit builds the
# same version from a different tree, so its bytes are not the pinned
# ones and this check would fail it.
#
# A commit that rewrites flake.nix runs the check too: a re-pin during a
# cut changes the pins without changing the version, and those pins must
# be verified before the tag as well.
#
# HEAD's version is CI_VERSION, parsed out of the working tree by the
# container gate tool. The parent's is read with `sed`, because that tool
# reads `build.zig.zon` at a fixed path and has no mode for a blob out of
# history. An empty parse is a failure, not a bump: it means the manifest
# moved and the question went unanswered.
- name: Verify the flake pins on a version bump
run: |
set -euo pipefail
parent="$(git rev-parse --verify HEAD^)"
parent_version="$(git show "$parent":build.zig.zon | sed -n 's/^[[:space:]]*\.version = "\([^"]*\)".*/\1/p')"
if [ -z "$parent_version" ]; then
echo "cannot read .version out of $parent:build.zig.zon" >&2
exit 1
fi
if [ "$parent_version" != "$CI_VERSION" ] || ! git diff --quiet "$parent" HEAD -- flake.nix; then
zig build verify-pins \
-Dversion-string="$CI_VERSION" \
-Dadmin-dist=admin-dist-ci \
-Doptimize=ReleaseSafe
else
echo "skipped: $GITHUB_SHA declares version $CI_VERSION, $parent already declared $parent_version and flake.nix is unchanged, so it is neither a version bump nor a re-pin and its bytes are not the ones flake.nix pins"
fi
container:
needs: [package]
runs-on: ubuntu-24.04
+35 -5
View File
@@ -60,8 +60,15 @@ env:
REGISTRY_HOST: "git.mial.net"
ZIG_VERSION: "0.16.0"
# The bytes of that zig, not just its version string: a distribution package
# of the same version is built against a different LLVM and emits different
# machine code, so its tarballs hash to something no CI run reproduces.
# setup-zig verifies these same bytes by minisign; `zig build cut` checks this
# digest before it builds a release.
ZIG_TARBALL_SHA256: "70e49664a74374b48b51e6f3fdfbf437f6395d42509050588bd49abe52ba3d00"
# Exact patch, not a floating "24" (ruling 12).
NODE_VERSION: "24.19.0"
NPM_VERSION: "11.17.0"
# The author's commit- and tag-signing key. `git verify-tag` alone proves
# only that *some* key in the keyring signed the tag, so the signature's
@@ -271,18 +278,29 @@ jobs:
cache: npm
cache-dependency-path: admin/package-lock.json
# The same pinned toolchain and normalized environment as the frontend
# job in gates.yml: the cut pinned the bundle's bytes into flake.nix
# before this run existed, and the pin check below compares against them.
- name: Assert the pinned Node and npm
run: |
test "$(node --version)" = "v${NODE_VERSION:?}"
test "$(npm --version)" = "${NPM_VERSION:?}"
test ! -e /nonexistent/npmrc-user
test ! -e /nonexistent/npmrc-global
# The same normalized environment the cut builds the bundle in: exactly
# nine variables, a file mode from the umask, a C locale, UTC and a zero
# build timestamp. Each of those can move the bytes the release hashes
# cover, and this job's bundle has to reproduce the one the cut pinned.
- name: Build the web UI
working-directory: admin
run: |
npm ci
npm run build
run: env -i PATH="$PATH" HOME="$HOME" LC_ALL=C LANG=C TZ=UTC SOURCE_DATE_EPOCH=0 CI=true npm_config_userconfig=/nonexistent/npmrc-user npm_config_globalconfig=/nonexistent/npmrc-global sh -c 'umask 022 && npm ci && npm run build'
# Step 8.
- name: Build the release artifacts
run: >
zig build dist
-Dversion-string="$VERSION"
-Dgit-commit="$TAG_COMMIT"
-Dadmin-dist=admin/dist
-Doptimize=ReleaseSafe
@@ -290,7 +308,19 @@ jobs:
run: >
zig build verify-dist
-Dversion-string="$VERSION"
-Dgit-commit="$TAG_COMMIT"
-Dadmin-dist=admin/dist
-Doptimize=ReleaseSafe
# Unconditional, and before the image push, the draft and every upload:
# the tag's tree IS the bump commit's tree, so the flake block must pin
# these exact bytes and name this exact version. A consumer who resolves
# the tag through the flake gets hashes that were written before this run
# existed; this is where the claim is proved, while nothing has yet left
# the runner.
- name: Verify the flake pins
run: >
zig build verify-pins
-Dversion-string="$VERSION"
-Dadmin-dist=admin/dist
-Doptimize=ReleaseSafe
+24
View File
@@ -4,6 +4,30 @@ All notable changes to nxdns are recorded here. The format follows [Keep a Chang
Sections are written by hand. Nothing here is generated from commit messages: the point of the file is to say what changed for an operator, which a commit subject rarely does.
## [0.0.19] - 2026-09-12
### Changed
- **A failed blocklist download says why.** The warning line and the Diagnostics event detail now carry the phase that failed, the concrete cause behind the classification (a reset connection, a truncated chunk, a TLS handshake fault), the HTTP status if a head arrived, the bytes received, and the elapsed time. The separate `http status` warning is folded into that line. Same one line per failed source per pass as before.
## [0.0.18] - 2026-09-09
### Fixed
- **The Overview scope pickers no longer sit apart, and a long device name no longer pushes the period picker off a phone screen.** The Device and Period dropdowns fill their wrappers and stand side by side; a long name truncates with an ellipsis in the trigger and stays fully readable in the list.
## [0.0.17] - 2026-09-08
### Added
- **A Nix flake with tag-pinned hashes.** `flake.nix` at the repository root builds `nxdns` for `aarch64-linux` and `x86_64-linux` from the release tarballs, and carries their hashes in a generated block. A consumer pins the flake to a release tag and gets the exact bytes that tag published; `docs/how-to/install-with-nix.md` covers the input, the `nixpkgs` follows line, and Renovate.
### Changed
- **The release archive is written by the project's own tool.** `zig build dist` no longer shells out to the runner's `tar` and `gzip`; it writes the tar stream and the gzip container itself, with sorted entries, fixed modes, and zero timestamps. The tarball bytes now depend on the source tree, the compiler, and the admin bundle, and on nothing the host supplies — which is what lets a hash be pinned before CI rebuilds it.
- **The cut pins the release hashes before it commits.** `zig build cut` asserts the local Node, npm, and Zig match the versions CI pins, builds the release from the bumped manifest in a normalized environment, writes the resulting hashes into `flake.nix`, and commits that file alongside `build.zig.zon` as one commit. CI reverifies the pins on the bump commit and again on the tag, before anything is uploaded.
- **`nxdns version` no longer reports a git commit.** The commit is gone from the command's output, from `GET /api/version`, and from the admin footer, which now shows `nxdns v<version>`. A release identifies itself by version, and the bytes are reproducible from the tag, so a commit embedded in the binary told a reader nothing the tag did not.
## [0.0.16] - 2026-09-07
The Overview page is redesigned around the two dashboards people already know — Pi-hole's layout, NextDNS's charts — and every number in the admin is spelled one way.
+1 -1
View File
@@ -571,7 +571,7 @@ Requirements: responsive desktop/mobile; route loaders for initial fetch; TanSta
- `nxdns check` — validate config, probe upstreams, load each enabled listener's certificate and verify its key pairs with it; exit 2 on failure, 0 with warnings.
- `nxdns export [--out file.zon]`
- `nxdns import <file.zon> [--force]`
- `nxdns version` — app version, Zig version string, git commit. No build date: the version and the commit identify a build exactly, and a date is one more input a reproducible build would have to pin.
- `nxdns version` — app version and Zig version string. No commit and no build date: the version names the release, the release tarballs are reproducible from the tag with the pinned toolchain, and `flake.nix` pins their hashes. A commit or a date would be one more build input that reproducibility has to pin.
---
+3 -3
View File
@@ -70,15 +70,15 @@ The release artifacts come out of the same build graph, so the whole release bui
```sh
(cd admin && npm ci && npm run build) # required: dist refuses the placeholder
VERSION=$(sed -n 's/^[[:space:]]*\.version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' build.zig.zon)
zig build dist -Dversion-string="$VERSION" -Dgit-commit=$(git rev-parse HEAD) \
zig build dist -Dversion-string="$VERSION" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe # tarballs -> zig-out/dist/
zig build verify-dist -Dversion-string="$VERSION" -Dgit-commit=$(git rev-parse HEAD) \
zig build verify-dist -Dversion-string="$VERSION" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe # the release checks
```
The version comes from `build.zig.zon` because `verify-dist` asserts the two agree; a tag sets both.
That is not a claim that your tarball will hash the same as a published one. Nothing in this project measures whether two builds of the same commit on two different machines land on the same bytes, so no document here describes the build as reproducible. The gate that would settle it is a recorded deferral — `specs/milestone-14.md` ruling 12 — and [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md) explains what a matching or differing hash is worth in the meantime.
On the toolchain versions pinned at the top of `.gitea/workflows/gates.yml`, your tarballs hash the same as the published ones of that version. [docs/how-to/verify-a-release.md](docs/how-to/verify-a-release.md) has the rebuild recipe and what to check when a hash differs.
## Documentation
+1 -1
View File
@@ -52,7 +52,7 @@
});
fetch("/api/version")
.then((r) => r.json())
.then((v) => { el("version").textContent = v.version + " (" + v.git_commit + ")"; })
.then((v) => { el("version").textContent = v.version; })
.catch(() => {});
</script>
</body>
@@ -16,7 +16,7 @@ let responses: Record<string, unknown>;
beforeEach(() => {
responses = {
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
"/api/version": { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 },
"/api/health": health(),
};
vi.stubGlobal(
@@ -86,7 +86,7 @@ function json(payload: unknown): Response {
return new Response(JSON.stringify(payload), { status: 200, headers: { "content-type": "application/json" } });
}
const VERSION = { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 };
const VERSION = { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 };
/** The shell's own requests, which every test serves the same way. */
function stubFetch(handler: (url: string) => Response | Promise<Response>) {
@@ -48,7 +48,7 @@ const CLIENTS: Client[] = [
client("192.0.2.12", "", ""),
];
const VERSION = { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 };
const VERSION = { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 };
let sources: FakeEventSource[];
let fetchMock: ReturnType<typeof vi.fn>;
+1 -1
View File
@@ -50,7 +50,7 @@ export const PREFIXES = {
client_prefixes: [{ id: 1, prefix: "192.168.1.0/24", group_id: 2, group: "kids", priority: 100 }],
};
const VERSION = { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 };
const VERSION = { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 };
export const DATABASE = { authority: "database", path: null, reconciled_at: null, restart_pending: false };
export const MANAGED_FILE = {
@@ -194,7 +194,7 @@ function defaultResponses(status: ConfigStatus): Record<string, unknown> {
return {
"GET /api/config/status": status,
"GET /api/health": health(),
"GET /api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
"GET /api/version": { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 },
"GET /api/groups": { groups: GROUPS },
"GET /api/groups/1/sources": { source_ids: [1] },
"GET /api/groups/2/sources": { source_ids: [] },
@@ -35,7 +35,7 @@ let requested: string[];
beforeEach(() => {
requested = [];
responses = {
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
"/api/version": { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 },
// The shell reads health for the Diagnostics nav badge on every route.
"/api/health": health(),
};
@@ -77,7 +77,7 @@ let requested: string[];
beforeEach(() => {
requested = [];
responses = {
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
"/api/version": { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 },
// The health strip at the top of the page; quiet on a healthy box, which is
// what every test below wants it to be.
"/api/health": health(),
@@ -42,7 +42,7 @@ beforeEach(() => {
return healthFails ? json({ error: "health unavailable" }, 400) : json(healthBody);
}
if (url === "/api/version")
return json({ version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 });
return json({ version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 });
if (url.startsWith("/api/diagnostics"))
return json({ events: [], next_before: null, active: { warnings: 0, errors: 0 } });
return json({ error: "not stubbed" }, 404);
@@ -60,10 +60,14 @@ const styles = stylex.create({
toolbar: {
display: "flex",
gap: "0.5rem",
// Without this the row's minimum is both labels at full length, and a
// long device name pushes the period picker past the viewport edge.
minWidth: 0,
flexBasis: { default: null, [NARROW]: "100%" },
},
control: {
minWidth: { default: "11rem", [NARROW]: 0 },
maxWidth: { default: "20rem", [NARROW]: "none" },
flex: { default: null, [NARROW]: 1 },
},
/** Under the Device control: the list behind it did not load, so the control offers the household only. */
@@ -113,7 +113,7 @@ beforeEach(() => {
}
if (url === "/api/health") return json(healthBody);
if (url === "/api/version")
return json({ version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 });
return json({ version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 });
if (url.startsWith("/api/diagnostics")) {
return json({ events: [], next_before: null, active: { warnings: 0, errors: 0 } });
}
-1
View File
@@ -70,7 +70,6 @@ export const sample_get_health: Health = {
};
export const sample_get_version: Version = {
git_commit: "<build>",
uptime_seconds: 0,
version: "w10-test",
zig_version: "<build>",
-1
View File
@@ -63,7 +63,6 @@ export interface Health {
export interface Version {
version: string;
git_commit: string;
zig_version: string;
uptime_seconds: number;
}
+1 -1
View File
@@ -33,7 +33,7 @@ const RESPONSES: Record<string, unknown> = {
coverage: { complete: true, available_since: 0 },
},
"/api/diagnostics?state=active": { events: [], next_before: null, active: { warnings: 0, errors: 0 } },
"/api/version": { version: "0.0.0-test", git_commit: "0000000", zig_version: "0.16.0", uptime_seconds: 1 },
"/api/version": { version: "0.0.0-test", zig_version: "0.16.0", uptime_seconds: 1 },
};
/** Null makes the health poll fail, which the nav badge has to treat as unknown. */
+1 -1
View File
@@ -318,7 +318,7 @@ function VersionFooter() {
const { data } = useQuery(versionQuery());
return (
<footer {...stylex.props(styles.versionFooter)}>
{data === undefined ? "nxdns" : `nxdns v${data.version} (${data.git_commit.slice(0, 7)})`}
{data === undefined ? "nxdns" : `nxdns v${data.version}`}
</footer>
);
}
+3
View File
@@ -80,6 +80,9 @@ const styles = stylex.create({
},
/** Rectangular, hairline, full hit height: the decision record's toolbar selector. */
toolbar: {
// A button keeps its content width even as a flex container, so without
// this the scope picker's 11rem wrapper shows as dead space beside it.
width: "100%",
minHeight: metrics.hitTarget,
borderRadius: metrics.radius,
borderWidth: 1,
+107 -46
View File
@@ -43,7 +43,6 @@ pub fn build(b: *std.Build) void {
// the default, so `zig build` and `zig build test` need no flag.
const version_option = b.option([]const u8, "version-string", "Version reported by `nxdns version` (required by `dist`)");
const version_string = version_option orelse "0.1.0-dev";
const git_commit = b.option([]const u8, "git-commit", "Git commit reported by `nxdns version`") orelse "unknown";
const admin_dist = b.option(
[]const u8,
"admin-dist",
@@ -77,7 +76,6 @@ pub fn build(b: *std.Build) void {
options.addOption(bool, "integration", integration);
options.addOption(bool, "live", live);
options.addOption([]const u8, "version_string", version_string);
options.addOption([]const u8, "git_commit", git_commit);
options.addOption([]const u8, "zig_version_string", builtin.zig_version_string);
options.addOption([]const u8, "contract_samples_out", contract_samples_out);
@@ -318,11 +316,39 @@ pub fn build(b: *std.Build) void {
// disk, so the test binary has to run at the build root.
cut_tests_run.setCwd(b.path("."));
test_step.dependOn(&cut_tests_run.step);
// The test binary analyses only what the tests reference; `main` and the
// stages behind it are compiled here so a type error in the cut itself
// fails `zig build test` and not the release.
test_step.dependOn(&cut_tool.step);
// `dist_stage` owns the release archive bytes, and its reproducibility is
// the property the flake pins depend on, so it is tested like any other
// decision this build makes.
const dist_stage_tests = b.addTest(.{
.name = "dist-stage-tool",
.root_module = b.createModule(.{
.root_source_file = b.path("tools/dist_stage.zig"),
.target = b.graph.host,
.optimize = optimize,
}),
});
test_step.dependOn(&b.addRunArtifact(dist_stage_tests).step);
// `gen_admin_assets` decides which files reach the binary, and that set must
// not depend on the bundle's provenance, so it is tested here too.
const gen_admin_assets_tests = b.addTest(.{
.name = "gen-admin-assets-tool",
.root_module = b.createModule(.{
.root_source_file = b.path("tools/gen_admin_assets.zig"),
.target = b.graph.host,
.optimize = optimize,
}),
});
test_step.dependOn(&b.addRunArtifact(gen_admin_assets_tests).step);
addDist(b, options, admin_assets, .{
.version = version_option,
.version_string = version_string,
.git_commit = git_commit,
.admin_dist = admin_dist,
});
}
@@ -355,14 +381,18 @@ const DistOptions = struct {
/// from one that happens to equal the default.
version: ?[]const u8,
version_string: []const u8,
git_commit: []const u8,
admin_dist: []const u8,
};
/// `dist` builds everything releasable; `verify-dist` asserts the result.
/// Both run on a laptop exactly as they run on the runner, which is the point:
/// release checks that only exist in CI shell are the brittleness milestone 14
/// set out to remove.
/// `dist` builds everything releasable; `verify-dist` asserts the result;
/// `pin-flake` writes the resulting hashes into `flake.nix` and `verify-pins`
/// asserts that they still describe the bytes under `zig-out/dist`.
///
/// All four run on a laptop exactly as they run on the runner, which is the
/// point: release checks that only exist in CI shell are the brittleness
/// milestone 14 set out to remove. The pins depend on it twice over — the cut
/// writes them here and CI recomputes them there, and the two only agree
/// because it is one build graph rather than two scripts.
fn addDist(
b: *std.Build,
options: *std.Build.Step.Options,
@@ -371,11 +401,15 @@ fn addDist(
) void {
const dist_step = b.step("dist", "Build the release tarballs, checksums and staged payloads");
const verify_step = b.step("verify-dist", "Verify the release artifacts under zig-out/dist");
const pin_step = b.step("pin-flake", "Write the release hashes under zig-out/dist into flake.nix");
const verify_pins_step = b.step("verify-pins", "Check flake.nix pins the hashes of the release under zig-out/dist");
if (distPreflight(b, dist_options)) |problem| {
const fail = b.addFail(problem);
dist_step.dependOn(&fail.step);
verify_step.dependOn(&fail.step);
pin_step.dependOn(&fail.step);
verify_pins_step.dependOn(&fail.step);
return;
}
@@ -401,7 +435,6 @@ fn addDist(
verify_run.addArgs(&.{ "--dist-dir", b.getInstallPath(.prefix, "dist") });
verify_run.addArgs(&.{ "--work-dir", b.getInstallPath(.prefix, "dist-verify") });
verify_run.addArgs(&.{ "--version", dist_options.version_string });
verify_run.addArgs(&.{ "--git-commit", dist_options.git_commit });
verify_run.addArg("--zon");
verify_run.addFileArg(b.path("build.zig.zon"));
verify_run.addArgs(&.{ "--max-bytes", b.fmt("{d}", .{max_binary_bytes}) });
@@ -447,36 +480,19 @@ fn addDist(
stage_run.addArg("--licenses");
stage_run.addDirectoryArg(staged_licenses);
// Two commands, never one: `addSystemCommand` executes argv directly
// and does not interpret `|`, and a shell pipeline without `pipefail`
// would report only gzip's status while a failed tar passed silently.
const tar_run = b.addSystemCommand(&.{
"tar",
"--format=gnu",
"--sort=name",
"--mtime=@0",
"--owner=0",
"--group=0",
"--numeric-owner",
"-c",
"-f",
});
setReproducibleEnv(tar_run);
const tar_file = tar_run.addOutputFileArg(b.fmt("{s}.tar", .{name}));
tar_run.addArg("-C");
// The tarball is written by our own tool rather than by the runner's
// `tar` and `gzip`: the release hashes are pinned in `flake.nix` before
// CI rebuilds them, so the bytes may depend on the staged tree and on
// nothing else the host supplies.
const archive_run = b.addRunArtifact(stage_tool);
archive_run.addArg("archive");
archive_run.addArg("--root");
// The staged payload is the sole entry of its cache directory, so its
// parent is what `-C` needs and declaring it declares the payload.
tar_run.addDirectoryArg(staged.dirname());
tar_run.addArg(name);
// `-n` is required because `--mtime=@0` normalises the tar member times
// but not the timestamp gzip writes into its own header. `-c` is
// required because plain `gzip <file>` rewrites its input in place, and
// the input here is a content-addressed cache entry.
const gzip_run = b.addSystemCommand(&.{ "gzip", "-n", "-9", "-c" });
setReproducibleEnv(gzip_run);
gzip_run.addFileArg(tar_file);
const tarball = gzip_run.captureStdOut(.{ .basename = b.fmt("{s}.tar.gz", .{name}) });
// parent is what `--root` needs and declaring it declares the payload.
archive_run.addDirectoryArg(staged.dirname());
archive_run.addArgs(&.{ "--payload", name });
archive_run.addArg("--out");
const tarball = archive_run.addOutputFileArg(b.fmt("{s}.tar.gz", .{name}));
const install_binary = b.addInstallFile(
staged.path(b, "nxdns"),
@@ -513,6 +529,38 @@ fn addDist(
verify_run.step.dependOn(dist_step);
verify_step.dependOn(&verify_run.step);
// The pins in `flake.nix` are written before CI ever builds the release, so
// the run that rebuilds it has to prove they describe its own bytes.
//
// It is a step of its own and NOT part of `verify-dist`. An ordinary commit
// between two cuts builds the `build.zig.zon` version from a tree that
// differs from the released one, so its bytes never match the pins and
// checking them there would fail every such build. The two CI jobs that may
// not skip it call it by name: the package job on the bump commit, and the
// publish job unconditionally before any upload.
const pin_check_run = b.addRunArtifact(stage_tool);
pin_check_run.has_side_effects = true;
pin_check_run.addArg("pin-check");
pin_check_run.addArgs(&.{ "--sums", b.getInstallPath(.prefix, "dist/SHA256SUMS") });
pin_check_run.addArg("--flake");
pin_check_run.addFileArg(b.path("flake.nix"));
pin_check_run.addArgs(&.{ "--version", dist_options.version_string });
pin_check_run.step.dependOn(dist_step);
verify_pins_step.dependOn(&pin_check_run.step);
// The write half, run by the cut and by nothing else. `flake.nix` is named
// as a plain path rather than a `LazyPath`: this run edits the source file
// in place, and a file argument would declare it an input of a step that is
// in fact its author.
const pin_run = b.addRunArtifact(stage_tool);
pin_run.has_side_effects = true;
pin_run.addArg("pin");
pin_run.addArgs(&.{ "--sums", b.getInstallPath(.prefix, "dist/SHA256SUMS") });
pin_run.addArgs(&.{ "--flake", b.pathFromRoot("flake.nix") });
pin_run.addArgs(&.{ "--version", dist_options.version_string });
pin_run.step.dependOn(dist_step);
pin_step.dependOn(&pin_run.step);
}
/// The one message `dist` and `verify-dist` fail with when the release inputs
@@ -564,13 +612,6 @@ fn hostTool(b: *std.Build, name: []const u8) *std.Build.Step.Compile {
});
}
/// Locale and time zone leak into archive metadata and into tool output.
/// Pinning both is the cheap half of reproducibility (milestone-14 ruling 12).
fn setReproducibleEnv(run: *std.Build.Step.Run) void {
run.setEnvironmentVariable("LC_ALL", "C");
run.setEnvironmentVariable("TZ", "UTC");
}
/// Milestone-15 ruling 4: every `*.zig` under `src/` must appear in
/// `src/tests.zig` as a line that trims to exactly `_ = @import("<path>");`,
/// where `<path>` is relative to `src/`. Whole-line equality, not a substring
@@ -744,7 +785,10 @@ fn addExecutable(
exe.root_module.addAnonymousImport("admin_assets", .{ .root_source_file = admin_assets });
exe.root_module.linkLibrary(sqliteLibrary(b, target, optimize));
exe.root_module.linkLibrary(mbedtlsLibrary(b, target, optimize));
exe.root_module.addCSourceFile(.{ .file = b.path("src/platform/mbedtls_shim.c") });
exe.root_module.addCSourceFile(.{
.file = b.path("src/platform/mbedtls_shim.c"),
.flags = &.{filePrefixMap(b, .build_root)},
});
addMbedtlsThreadingMacros(exe.root_module);
return exe;
}
@@ -810,6 +854,8 @@ fn sqliteLibrary(
"-DSQLITE_THREADSAFE=1",
"-DSQLITE_DEFAULT_WAL_SYNCHRONOUS=1",
"-DSQLITE_OMIT_LOAD_EXTENSION",
filePrefixMap(b, .build_root),
filePrefixMap(b, .global_cache),
},
});
return lib;
@@ -846,19 +892,34 @@ fn mbedtlsLibrary(
lib.root_module.addIncludePath(dep.path(include_dir));
}
const c_flags = [_][]const u8{ filePrefixMap(b, .build_root), filePrefixMap(b, .global_cache) };
lib.root_module.addCSourceFiles(.{
.root = dep.path("library"),
.files = &mbedtls_library_sources,
.flags = &c_flags,
});
lib.root_module.addCSourceFiles(.{
.root = dep.path("3rdparty"),
.files = &mbedtls_3rdparty_sources,
.flags = &c_flags,
});
lib.installHeadersDirectory(dep.path("include/mbedtls"), "mbedtls", .{});
lib.installHeadersDirectory(dep.path("include/psa"), "psa", .{});
return lib;
}
/// `__FILE__` in the C sources (mbedTLS debug and assertion macros) would
/// otherwise embed the absolute checkout path into the release binary, and the
/// flake pins require the bytes to be the same on every machine that builds
/// the tag. Both roots a dependency can be stored under are mapped.
fn filePrefixMap(b: *std.Build, root: enum { build_root, global_cache }) []const u8 {
const path = switch (root) {
.build_root => b.build_root.path orelse ".",
.global_cache => b.graph.global_cache_root.path orelse ".",
};
return b.fmt("-ffile-prefix-map={s}=.", .{path});
}
/// Context sizes change with threading enabled, so every compilation unit that
/// includes mbedTLS headers (the library itself and `mbedtls_shim.c`) must see
/// the same macros. Concurrent handshakes share `ssl_config`, the CTR-DRBG, and
+1 -1
View File
@@ -1,6 +1,6 @@
.{
.name = .nxdns,
.version = "0.0.16",
.version = "0.0.19",
.minimum_zig_version = "0.16.0",
.paths = .{""},
.fingerprint = 0x3307b311dded1d91,
+1 -1
View File
@@ -1,7 +1,7 @@
# The binary is NOT compiled here. Build it first, from the repository root:
#
# (cd admin && npm ci && npm run build)
# zig build dist -Dversion-string=<V> -Dgit-commit=<SHA> \
# zig build dist -Dversion-string=<V> \
# -Dadmin-dist=admin/dist -Doptimize=ReleaseSafe
#
# then build the image with the repository root as context:
+1
View File
@@ -22,6 +22,7 @@ Steps for a goal you already have. They assume you know what nxdns is.
- [how-to/verify-a-release.md](how-to/verify-a-release.md) — check the signature and the checksums before you run anything, and what they prove.
- [how-to/install-with-systemd.md](how-to/install-with-systemd.md) — a real install as a system service, including the Raspberry Pi 5 aarch64 binary.
- [how-to/install-with-docker.md](how-to/install-with-docker.md) — the published container image and the compose file.
- [how-to/install-with-nix.md](how-to/install-with-nix.md) — the flake input pinned to a release tag, and Renovate for tag bumps.
- [how-to/upgrade.md](how-to/upgrade.md) — move to a new release without losing state.
- [how-to/troubleshoot.md](how-to/troubleshoot.md) — what to do when it does not answer, does not block, or will not start.
- [how-to/enable-doh-and-dot.md](how-to/enable-doh-and-dot.md) — serve encrypted DNS with certificates.
+2
View File
@@ -11,6 +11,7 @@ pub const tutorial_first_run_md = @embedFile("tutorial/first-run.md");
pub const howto_back_up_and_restore_md = @embedFile("how-to/back-up-and-restore.md");
pub const howto_enable_doh_and_dot_md = @embedFile("how-to/enable-doh-and-dot.md");
pub const howto_install_with_docker_md = @embedFile("how-to/install-with-docker.md");
pub const howto_install_with_nix_md = @embedFile("how-to/install-with-nix.md");
pub const howto_install_with_systemd_md = @embedFile("how-to/install-with-systemd.md");
pub const howto_measure_performance_md = @embedFile("how-to/measure-performance.md");
pub const howto_set_up_admin_authentication_md = @embedFile("how-to/set-up-admin-authentication.md");
@@ -34,6 +35,7 @@ pub const pages: []const Page = &.{
.{ .path = "docs/how-to/back-up-and-restore.md", .text = howto_back_up_and_restore_md },
.{ .path = "docs/how-to/enable-doh-and-dot.md", .text = howto_enable_doh_and_dot_md },
.{ .path = "docs/how-to/install-with-docker.md", .text = howto_install_with_docker_md },
.{ .path = "docs/how-to/install-with-nix.md", .text = howto_install_with_nix_md },
.{ .path = "docs/how-to/install-with-systemd.md", .text = howto_install_with_systemd_md },
.{ .path = "docs/how-to/measure-performance.md", .text = howto_measure_performance_md },
.{ .path = "docs/how-to/set-up-admin-authentication.md", .text = howto_set_up_admin_authentication_md },
+1 -1
View File
@@ -220,7 +220,7 @@ The Dockerfile does not compile anything. It assembles a filesystem around binar
```sh
(cd admin && npm ci && npm run build)
VERSION=$(sed -n 's/^[[:space:]]*\.version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' build.zig.zon)
zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" \
zig build dist -Dversion-string="$VERSION" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe
DOCKER_BUILDKIT=1 docker build -t nxdns -f deploy/docker/Dockerfile .
```
+88
View File
@@ -0,0 +1,88 @@
# Install nxdns with Nix
Adds nxdns to a NixOS machine as a flake input pinned to a release tag. At the end `pkgs`-style references to `inputs.nxdns.packages.${system}.default` resolve to the published release binary, and a Renovate custom manager opens a pull request when a new tag appears.
nxdns publishes its own `flake.nix`. Its packages do not build nxdns from source: each one fetches the release tarball for the target and pins its SHA-256 hash, so a changed byte fails the build. The two supported systems are `aarch64-linux` and `x86_64-linux`, both static musl builds that need nothing on the host.
For the signature and checksum checks a human does once, see [verify a release](verify-a-release.md). For what each configuration field means, see [the configuration reference](../reference/configuration.md).
> Verification: the commands in step 1 and step 2 were run on the machine that wrote this page, against the flake at nxdns 0.0.16. `nix flake check --no-build` passed and `nix build` produced a binary that printed `nxdns 0.0.16`. The consumer snippets in step 3 and step 4 are copied from the shape `rpi.mial.net` already uses for another flake input of the same author; they were not evaluated from this checkout, which is not a NixOS configuration.
## 1. Add the input, pinned to a tag
In the consuming flake:
```nix
inputs = {
nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
nxdns = {
url = "git+https://git.mial.net/mokhtar/nxdns.git?ref=refs/tags/v0.0.16";
inputs.nixpkgs.follows = "nixpkgs";
};
};
```
Pin a tag, not a branch. A branch pin moves the version under you at the next `nix flake update`, and the hashes in the flake belong to whatever release that branch last cut.
The `follows` line is not cosmetic. Without it Nix fetches and evaluates nxdns's own nixpkgs as a second nixpkgs, which costs a download and an evaluation for a package that only needs `stdenv`, `fetchurl`, and `lib`.
Write the lock entry:
```sh
nix flake lock
```
## 2. Check what you pinned
```sh
nix flake check --no-build
nix eval .#packages.x86_64-linux.default.outPath
```
Evaluation does not fetch the tarball. The download happens at build time, and the hash in nxdns's flake is what the build asserts the bytes against.
## 3. Use the package in a NixOS configuration
Pass the flake inputs to the module system, then reference the package:
```nix
{ inputs, pkgs, ... }:
{
environment.systemPackages = [ inputs.nxdns.packages.${pkgs.stdenv.hostPlatform.system}.default ];
}
```
The derivation installs `bin/nxdns`, plus `LICENSE` and `THIRD-PARTY-NOTICES` under `share/doc/nxdns`. It sets `meta.mainProgram`, so `lib.getExe` resolves to the binary. Run it as a service with the unit from [install with systemd](install-with-systemd.md), or write your own module around it.
## 4. Let Renovate bump the tag
Renovate's built-in `nix` manager does not do this. It only advances `flake.lock` along the ref an input already tracks, and a `refs/tags/vX.Y.Z` ref never moves, so it reports no releases for a tag-pinned input. Verified on 2026-09-09 with Renovate 42.99.0 against a Gitea host: all flake inputs, GitHub-hosted ones included, came back with an empty release list.
What works is a regex custom manager that treats the tag in the input URL as a version string, with the `gitea-tags` datasource:
```json
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json",
"extends": ["config:recommended"],
"customManagers": [
{
"customType": "regex",
"managerFilePatterns": ["/^flake\\.nix$/"],
"matchStrings": ["git\\+https://git\\.mial\\.net/mokhtar/nxdns\\.git\\?ref=refs/tags/(?<currentValue>v\\d+\\.\\d+\\.\\d+)"],
"depNameTemplate": "mokhtar/nxdns",
"datasourceTemplate": "gitea-tags",
"registryUrlTemplate": "https://git.mial.net",
"versioningTemplate": "semver"
}
]
}
```
Verified on 2026-09-09: with this manager, Renovate opened the pull request for v0.0.18 on the day of the tag. That pull request rewrites the tag in `flake.nix` only. `flake.lock` still records the old revision, so the same pull request must also refresh the lock, either with a `postUpgradeTasks` command (`nix flake update nxdns`, which needs the self-hosted `allowedCommands` setting) or with a CI job on the Renovate branch that commits the lock. The datasource proposes only tags that already exist, so it never pins a release that has not been cut.
Between the tag push and the asset upload there is a window in which the tag exists and the release tarballs do not. A pin written by hand during that window evaluates, then fails at build time with a 404 from the release download URL. Wait for the release to be published, or re-run the build once it is. Renovate's own pull requests are not affected by the window in practice, because it runs on a schedule rather than on the tag push.
## Upgrading
Change the `?ref=refs/tags/vX.Y.Z` in the input, run `nix flake lock --update-input nxdns`, and rebuild. Read the release notes first: nxdns is pre-0.1 and breaks on purpose, and [upgrade](upgrade.md) lists what state a version change touches.
+3 -3
View File
@@ -377,13 +377,13 @@ Requires Zig 0.16.0 and Node.js. From the repository root:
```sh
(cd admin && npm ci && npm run build)
VERSION=$(sed -n 's/^[[:space:]]*\.version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' build.zig.zon)
zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" \
zig build dist -Dversion-string="$VERSION" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe
```
The first command builds the admin interface into `admin/dist`; the last one embeds that directory in the binary. Build the interface every time, before the binary: a stale `admin/dist` ships an admin UI that does not match the API it talks to. `dist` refuses to run against the `admin/dist-placeholder` default for exactly that reason, so there is no way to skip it by accident.
`-Dversion-string` is required and has no default. It is what `nxdns version` prints. Take it from `build.zig.zon` rather than inventing one: `verify-dist` asserts that the version under build equals `.version` there, so a made-up string like `0.0.0-local` builds but then fails verification. `-Dgit-commit` is what distinguishes your build from the published one of the same version.
`-Dversion-string` is required and has no default. It is what `nxdns version` prints. Take it from `build.zig.zon` rather than inventing one: `verify-dist` asserts that the version under build equals `.version` there, so a made-up string like `0.0.0-local` builds but then fails verification.
What comes out under `zig-out/dist/` is the same set a release publishes, minus the signature and the image digest:
@@ -397,7 +397,7 @@ The two targets are `x86_64-linux-musl` and `aarch64-linux-musl`. Both binaries
Check the result the same way the release pipeline does:
```sh
zig build verify-dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" \
zig build verify-dist -Dversion-string="$VERSION" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe
```
+3 -3
View File
@@ -218,7 +218,7 @@ OK: no problems found
>
> ```
> $ nxdns version
> nxdns <version> (unknown)
> nxdns <version>
> zig 0.16.0
> ```
>
@@ -338,13 +338,13 @@ If you are running something you built rather than a release, step 2 is a build
```sh
(cd admin && npm ci && npm run build)
VERSION=$(sed -n 's/^[[:space:]]*\.version[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p' build.zig.zon)
zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" \
zig build dist -Dversion-string="$VERSION" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe
```
Rebuild `admin/dist` before the binary on every upgrade. The admin interface is embedded at build time, and an old bundle against a new API is a broken System page. `dist` refuses the `admin/dist-placeholder` default outright, so the only way to ship a stale bundle is to leave an old `admin/dist` in place.
The staged payload for each target is under `zig-out/dist/stage/nxdns-<version>-<triple>/`, and step 3 continues from there with that path in place of the extracted one. The version string has to equal `.version` in `build.zig.zon``verify-dist` asserts it, so a made-up one builds and then fails verification. What tells your build apart from the published release of the same version is `-Dgit-commit`, which `nxdns version` prints beside the version.
The staged payload for each target is under `zig-out/dist/stage/nxdns-<version>-<triple>/`, and step 3 continues from there with that path in place of the extracted one. The version string has to equal `.version` in `build.zig.zon``verify-dist` asserts it, so a made-up one builds and then fails verification. Your build reports the same two lines as the published release of that version, so compare the tarball hashes rather than the `nxdns version` output when you need to tell them apart.
Under Docker, build the image and name it instead of pulling:
+32 -21
View File
@@ -197,13 +197,11 @@ tar -xzf nxdns-$VERSION-x86_64-linux-musl.tar.gz
./nxdns-$VERSION-x86_64-linux-musl/nxdns version
```
`version` prints the version and the git commit it was built from, then the Zig version. The version has to match the tag you downloaded, and the commit has to match the commit the tag points at.
`version` prints the version, then the Zig version the binary was built with. The version has to match the tag you downloaded. The binary carries no commit sha, so the check that ties a release to its source is the rebuild below rather than a string in this output.
> Verified against `v0.0.1`: both tarballs listed exactly the one directory and
> six files with the stated modes, no symlinks and no absolute or `..` paths,
> and the extracted binary printed `nxdns 0.0.1
> (3c2d0d41f04570038e805b759da4541e198eae17)` — the commit `v0.0.1` points at —
> then `zig 0.16.0`.
> and the extracted binary printed `nxdns 0.0.1` then `zig 0.16.0`.
## 6. Verify the container image
@@ -262,22 +260,42 @@ It proves two things:
It does not prove that the binary in the tarball was built from the source in this repository. The machine that ran the build also held the signing key, so a compromise of that machine produces an artifact that is signed, verifies cleanly, and contains whatever the attacker put in it. The signature is a statement about origin and integrity in transit. It is not a statement about provenance from source.
Closing that gap needs a reproducibility gate — an independent build, run somewhere else, that lands on the same bytes — and this project does not have one. It is a recorded deferral, not an oversight: see `specs/milestone-14.md` ruling 12. Until it exists, nothing here claims the build is reproducible, because nobody has measured whether it is.
Closing that gap needs a reproducible build — an independent build, run somewhere else, that lands on the same bytes. The release tarballs are reproducible: the binary carries no commit sha, the archive is written by the project's own tool with fixed modes, zero timestamps and sorted entries, and CI rebuilds the same bytes and compares them against the pinned hashes before it uploads anything. The rebuild below is how you check that for yourself.
The signing key is a subkey rather than the primary key, which limits the damage of the case above: a leaked release subkey is revoked on its own and the identity, the commit signatures and everyone's existing trust in the key survive.
## Rebuild it yourself
You can still build the same version from source and compare. That gets you a binary whose provenance you know, and the comparison is worth making — read the paragraph after the recipe before you draw a conclusion from it.
Build the same version from source and compare the hashes. Match the toolchain first: `ZIG_VERSION`, `NODE_VERSION` and `NPM_VERSION` at the top of `.gitea/workflows/gates.yml` are the exact versions the release used, and a different patch release of any of them changes the bytes.
Zig has to be the official ziglang.org tarball for that version, not your distribution's package of it. A distro package of `0.16.0` is built against the system LLVM and emits different machine code than the official build, and both print `0.16.0`, so the version string tells you nothing. This is not hypothetical: the first cut of `flake.nix` pinned hashes CI could not reproduce for exactly that reason, and the release cut now downloads the official tarball itself rather than trusting `PATH`. Fetch and verify it the same way, against the `ZIG_TARBALL_SHA256` pin that sits next to `ZIG_VERSION` in `gates.yml` (CI installs the same bytes through setup-zig, which checks them by minisign):
```sh
git clone https://git.mial.net/mokhtar/nxdns
cd nxdns
git checkout "v$VERSION"
git verify-tag "v$VERSION"
(cd admin && npm ci && npm run build)
zig build dist -Dversion-string="$VERSION" -Dgit-commit="$(git rev-parse HEAD)" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe
ZIG_VERSION=$(sed -n 's/^ ZIG_VERSION: "\(.*\)"$/\1/p' .gitea/workflows/gates.yml)
ZIG_TARBALL_SHA256=$(sed -n 's/^ ZIG_TARBALL_SHA256: "\(.*\)"$/\1/p' .gitea/workflows/gates.yml)
curl -fsSL -o ../zig.tar.xz "https://ziglang.org/download/$ZIG_VERSION/zig-x86_64-linux-$ZIG_VERSION.tar.xz"
(cd .. && echo "$ZIG_TARBALL_SHA256 zig.tar.xz" | sha256sum -c - && tar -xJf zig.tar.xz)
ZIG="$(cd .. && pwd)/zig-x86_64-linux-$ZIG_VERSION/zig"
"$ZIG" version
```
The host matters too: the admin bundle is built with host-native Rolldown and Lightning CSS bindings, which the lockfile ships per platform and libc, so the recipe reproduces the release only on x86_64 Linux with glibc, the runner CI uses. The two `npm_config_*` paths must not exist on your machine; the recipe relies on npm finding no config file there. The environment matters as much as the toolchain. The release cut and CI both build the bundle under exactly nine variables and a `022` umask, so the recipe below does the same: a locale, a time zone, a build timestamp or a file mode picked up from your shell each move the bytes. The Zig build gets a private `--cache-dir` so a stale local cache cannot leak into them; the global Zig cache stays shared, because it is content-addressed and a fresh one refetches every dependency (see `specs/release-cut.md`).
```sh
env -i PATH="$PATH" HOME="$HOME" LC_ALL=C LANG=C TZ=UTC SOURCE_DATE_EPOCH=0 CI=true npm_config_userconfig=/nonexistent/npmrc-user npm_config_globalconfig=/nonexistent/npmrc-global \
sh -c 'cd admin && umask 022 && npm ci && npm run build'
env -i PATH="$PATH" HOME="$HOME" LC_ALL=C LANG=C TZ=UTC SOURCE_DATE_EPOCH=0 CI=true npm_config_userconfig=/nonexistent/npmrc-user npm_config_globalconfig=/nonexistent/npmrc-global \
sh -c 'umask 022 && exec "'"$ZIG"'" build dist \
-Dversion-string="'"$VERSION"'" \
-Dadmin-dist=admin/dist -Doptimize=ReleaseSafe \
--cache-dir "$(mktemp -d)"'
sha256sum zig-out/dist/nxdns-"$VERSION"-*.tar.gz
```
@@ -285,20 +303,13 @@ sha256sum zig-out/dist/nxdns-"$VERSION"-*.tar.gz
`zig build dist` writes `zig-out/dist/`: the two tarballs, a staging directory per target under `stage/`, the stripped binaries under `bin/<triple>/`, and a `SHA256SUMS` covering the two tarballs. The published `SHA256SUMS.txt` is that file with a third line for `IMAGE-DIGEST.txt` appended by the release job, so the two tarball lines should match and the local file has no third line to compare.
Now the caveat, and it is the whole reason this section is last. **A hash that differs does not mean the release was tampered with.** Nothing in this project measures whether two builds of the same commit on two different machines produce the same bytes, and there are several ordinary reasons they would not: a different Zig patch release, a different Node version, a different path to the build directory, a different npm lockfile resolution. A hash that matches is real evidence. A hash that does not match tells you only that something about the two builds differed, and finding out what is on you.
The tarball is written by `zig build dist` itself rather than by the host's `tar` and `gzip`, so its layout is fixed and a rebuild on a matching toolchain reproduces it byte for byte. Entries are sorted by their full path as bytes, with the payload directory first. The payload directory and `nxdns` carry mode `0755`; every other file carries `0644`. Every entry has a zero modification time, uid 0, gid 0, and no user or group name. The gzip wrapper carries no original filename and a zero header timestamp, which is what makes two archives of the same tree compare equal.
If you want the comparison to mean as much as it can, match the toolchain the release used. The Zig version is the second line of `nxdns version`, and both it and the Node version are pinned to exact patch releases at the top of `.gitea/workflows/gates.yml`, which is the workflow the release runs.
Compare your two tarball hashes against two things: the published `SHA256SUMS.txt`, and the `hashes` block of `flake.nix` at the tag, which carries the same digests in SRI form. All three agree on a matching toolchain, and the release pipeline fails before it uploads anything if they do not.
> Verified against `v0.0.1`, and the result is the caveat above in action. The
> whole recipe ran from a fresh clone: `git verify-tag v0.0.1` printed
> `Good signature` under the same signing subkey as the release, and
> `zig build dist` produced both tarballs. The hashes did **not** match the
> published `SHA256SUMS.txt` — the binaries themselves already differ. The Zig
> version matched the pin exactly; the Node version did not (24.14.1 against
> the pinned 24.19.0) and the build path differed, two of the ordinary causes
> listed above. That is a measurement of what an unpinned rebuild gives you,
> not evidence of tampering: the signature, checksum and image checks earlier
> on this page all passed against the same release.
A hash that differs is a signal to check the toolchain and the environment first. A distribution's Zig package instead of the official tarball is the first thing to rule out; an unpinned Node or npm version is the next ordinary explanation, and a build run outside the normalized environment above is the next one. Rule both out before you conclude anything about the release itself.
> Verified against `v0.0.1`, before the build was reproducible: the recipe ran from a fresh clone, `git verify-tag v0.0.1` printed `Good signature` under the release subkey, and the rebuilt tarball hashes did not match the published `SHA256SUMS.txt` (Node 24.14.1 against the pinned 24.19.0, a different build path, and an archive written by the host's `tar`). Releases from 0.0.17 on are built and checked by the pinned pipeline this page describes, and the cut records the local hashes in `flake.nix` before CI rebuilds them.
## If a check fails
+6 -1
View File
@@ -202,7 +202,12 @@ If a restart and an import race for the write lock, one of them simply wins: bot
## `version`
Prints two lines: the nxdns version with the git commit, then the Zig version the binary was built with. Takes no flags and no arguments.
Prints two lines: the nxdns version, then the Zig version the binary was built with. Takes no flags and no arguments.
```
nxdns <version>
zig 0.16.0
```
## `help`
Generated
+27
View File
@@ -0,0 +1,27 @@
{
"nodes": {
"nixpkgs": {
"locked": {
"lastModified": 1788752844,
"narHash": "sha256-VaWGJ6+cIYN2erfSecbRV+4ljI185Ty2wUrXyvQbgOw=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "dc5d91f840324650bac8c379428c7037a416959a",
"type": "github"
},
"original": {
"owner": "NixOS",
"ref": "nixos-unstable",
"repo": "nixpkgs",
"type": "github"
}
},
"root": {
"inputs": {
"nixpkgs": "nixpkgs"
}
}
},
"root": "root",
"version": 7
}
+114
View File
@@ -0,0 +1,114 @@
{
description = "nxdns: DNS sinkhole with per-client policy groups";
inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
outputs =
{ nixpkgs, ... }:
let
# `zig build cut` rewrites the lines between the delimiters by text replacement.
# BEGIN GENERATED BY zig build cut
version = "0.0.19";
hashes = {
"aarch64-linux" = "sha256-sNIjsnJ5TPRpSE/wDtwb0j7lYyyMXCPMwc7IAv5sy3s=";
"x86_64-linux" = "sha256-SvWV8lmyJWceVBhuAOtldMbwGA8hLx/puJVda2t0sCg=";
};
# END GENERATED BY zig build cut
triples = {
"aarch64-linux" = "aarch64-linux-musl";
"x86_64-linux" = "x86_64-linux-musl";
};
systems = builtins.attrNames hashes;
package =
system:
let
pkgs = nixpkgs.legacyPackages.${system};
lib = pkgs.lib;
triple = triples.${system};
in
pkgs.stdenv.mkDerivation {
pname = "nxdns";
inherit version;
src = pkgs.fetchurl {
url = "https://git.mial.net/mokhtar/nxdns/releases/download/v${version}/nxdns-${version}-${triple}.tar.gz";
hash = hashes.${system};
};
# Static musl binary: the install check asserts that no interpreter was patched in.
dontPatchELF = true;
dontStrip = true;
installPhase = ''
runHook preInstall
install -Dm755 nxdns $out/bin/nxdns
install -Dm644 LICENSE $out/share/doc/nxdns/LICENSE
install -Dm644 THIRD-PARTY-NOTICES $out/share/doc/nxdns/THIRD-PARTY-NOTICES
runHook postInstall
'';
doInstallCheck = true;
# GNU readelf prints `INTERP`, not `PT_INTERP`; `${READELF:?}` keeps an unset variable from turning the grep into a pass.
installCheckPhase = ''
runHook preInstallCheck
# Captured first: piping readelf straight into grep hides its exit
# status, so a readelf that failed to read the file at all would
# print nothing, match nothing, and pass as "static".
if ! segments="$(''${READELF:?} -l "$out/bin/nxdns")"; then
echo "readelf -l failed on $out/bin/nxdns" >&2
exit 1
fi
if printf '%s' "$segments" | grep -q INTERP; then
echo "nxdns has an INTERP segment: it is dynamically linked, not static" >&2
exit 1
fi
if ! dynamic="$(''${READELF:?} -d "$out/bin/nxdns")"; then
echo "readelf -d failed on $out/bin/nxdns" >&2
exit 1
fi
if printf '%s' "$dynamic" | grep -q NEEDED; then
echo "nxdns has DT_NEEDED entries: it links against shared libraries" >&2
exit 1
fi
if ! reported="$($out/bin/nxdns version)"; then
echo "nxdns version exited non-zero: the binary does not run here" >&2
exit 1
fi
# Prefix match: releases before 0.0.17 print a commit sha after the version.
case "$reported" in
"nxdns ${version}"*) echo "nxdns version reports: $reported" ;;
*)
echo "nxdns version reports '$reported'; expected it to start with 'nxdns ${version}'" >&2
exit 1
;;
esac
runHook postInstallCheck
'';
meta = {
description = "DNS sinkhole with per-client policy groups";
homepage = "https://git.mial.net/mokhtar/nxdns";
license = lib.licenses.eupl12;
mainProgram = "nxdns";
platforms = systems;
sourceProvenance = [ lib.sourceTypes.binaryNativeCode ];
};
};
in
{
packages = nixpkgs.lib.genAttrs systems (system: {
default = package system;
});
};
}
+8
View File
@@ -199,3 +199,11 @@ Deviations the build kept, judged defensible in review:
- `Certificate.Bundle` has no in-memory PEM entry point in 0.16.0; the loopback test mirrors `addCertsFromFile`'s decode+parse calls to preload the fixture cert.
- The S2.4 probe-close criterion is pinned as far as the repo can observe it: the failed-probe path runs the per-iteration deferred `close` (cli test, honestly named), and close-after-failure/double-close safety is pinned in the DoT integration tests. Close after a *successful* probe is unreachable in-repo (the probe verifies against the system trust store, and the only in-repo DoT peer is self-signed); no production surface was added to force it.
- The metrics S3.3 "fixture pool" test drives a real `Pool` through the real `snapshot()``upstreams()` → render path with distinct per-field values.
## Addendum (2026-09-08): overlap test synchronizes on entry
The test `"overlapping exchanges each report the entry that answered that call"` failed once on a loaded CI runner and passed on rerun. It started two `io.concurrent` exchanges back to back and relied on the first entry's failing 50 ms stall to keep the first task in flight until the second task arrived. Nothing checked that the two calls were ever in flight together, and the schedule that produced two equal identities was not observed: every ordering traced by hand still ends the calls on different entries. The one thing known is that the test asserted an overlap it never proved.
The test now builds the schedule instead of timing it, and contains no sleep at all. Two new `Fake.Behavior` variants, `hold` and `hold_fail`, wait on a `std.Io.Semaphore` the test owns and then reply or fail; the wait propagates cancellation exactly as the existing `slow` sleep does. The first entry holds its first call on one gate and its second call on the other, and the second entry replies immediately. The test starts the first task, waits with `awaitInFlight(io, &first_entry, 1)`, starts the second task, waits with `awaitInFlight(io, &first_entry, 2)`, and only then posts the first gate. Reaching a count of two means both calls are inside the first entry at that moment, so `peak_in_flight == 2` holds by construction, and the first task's failover to the second entry provably runs while the second call is still held inside the first entry. Posting the second gate afterwards releases it. Every earlier assertion is kept, including the per-entry call counts and health counts. Each gate's deferred post is registered right after its task's deferred await, so it runs before that await and every early return (a failed wait, a skipped second task, a failed assertion) releases the held call instead of deadlocking. The fake draws its behaviour before it raises `in_flight`, so a count of one also fixes which call holds which gate; without that order the second call could draw the failing behaviour and the first await would wait on a gate posted only after it.
No pool bug was found, and the failing schedule was never observed. The original failure did not reproduce in 15 runs of the test binary pinned to two cores under six CPU hogs, with a diagnostic print on the assertion; that diagnostic was removed. `failover`'s attribution was not changed. After the change, 10 loaded two-core runs pass with the test reported `OK`, and each of those runs reports `1860 passed; 177 skipped; 0 failed.`. `zig build test` and `zig build test -Dintegration` both exit 0; the build runner prints no per-test summary of its own, only the known `failed command:` label described in AGENTS.md.
+93
View File
@@ -0,0 +1,93 @@
# Milestone 40: Nix flake with tag-pinned hashes, one commit per cut
nxdns publishes a `flake.nix` whose package derivations fetch the release tarballs and carry their SRI hashes. A consumer pins `git+https://git.mial.net/mokhtar/nxdns.git?ref=refs/tags/vX.Y.Z` and Renovate bumps the tag. The hashes for a release are known before the release exists because the cut tool builds the same bytes locally that CI builds later, and CI verifies that equality before it uploads anything.
Owner rulings (2026-09-08): the consumer pins a tag, not a branch. A cut stays at two commits and two CI runs (the bump commit runs ci.yml; the tag runs release.yml). No follow-up commit, no extra run, no post-release hash edit. Design agreed with Codex (thread 2026-09-08) and its findings are folded in below.
## Why the hashes can be known in advance
The tarball bytes depend on: the Zig compiler version, the source tree at the bump commit, the admin bundle, the archive format, and nothing else. Today two things break that: the binary embeds the git commit sha (which the bump commit cannot know about itself) and the archive is produced by the runner's `tar` and `gzip`. This milestone removes both. Reproducibility is then a property the repo enforces on every CI run, not a hope.
## Session A: version identity without a commit sha
The release binary reports its version only. The git commit is gone from every surface; it is not replaced by a tag name or by a sentinel.
- `build.zig`: delete the `git-commit` option and the `git_commit` build option. Nothing else in `build.zig` changes in this session (Session B owns the rest of the file).
- `src/version.zig` and `src/cli.zig` `runVersion`: print `nxdns {version}\nzig {zig_version}\n`.
- `src/web/handlers/version.zig`: the `Body` loses `git_commit`. `admin/src/lib/types.ts`, `admin/src/lib/contractSamples.gen.ts` (regenerate with `zig build test -Dintegration -Dcontract-samples-out=<path>`, then copy), `admin/src/shell/AppShell.tsx` footer shows `nxdns v{version}`, and `AppShell.test.tsx` follow.
- `tools/verify_dist.zig`: drop the `--git-commit` argument and the commit check; the `version` check stays.
- `.gitea/workflows/gates.yml` package job and `.gitea/workflows/release.yml` publish job: drop `-Dgit-commit=...` from every `zig build dist` and `zig build verify-dist` line.
- Docs: `docs/reference/cli.md`, `docs/how-to/verify-a-release.md`, `docs/how-to/upgrade.md`, `docs/how-to/install-with-docker.md` show the new `nxdns version` output. `verify-a-release.md` replaces the commit-matching step with the reproduction check from Session D (rebuild at the tag with the same toolchain, compare `SHA256SUMS` and `flake.nix`).
- Tests: every test that asserted the commit in the CLI output, the API body, or the footer asserts the new shape.
## Session B: Zig-owned archive and pin check
`tools/dist_stage.zig` gains two modes and `build.zig` stops calling system `tar` and `gzip`.
- `archive --root <stage dir> --payload <dir name> --out <file.tar.gz>`: walks the payload directory, sorts entries by full path (bytes), writes a GNU tar stream with `std.tar.Writer` (`writeDir` for the payload directory, `writeFile` for regular files, mode 0755 for the directory and the `nxdns` binary, 0644 for every other file, mtime 0, uid 0, gid 0, empty user and group names), finishes with `finishPedantically`, and compresses with `std.compress.flate.Compress` in the `.gzip` container at a fixed level. The output bytes must not depend on the absolute path of the stage directory, the umask, the clock, the locale, or the host's tar/gzip.
- `pin-check --sums <SHA256SUMS> --flake <flake.nix> --version <x.y.z>`: parses the generated block in `flake.nix` (see Session C), converts each `sha256` SRI value to hex, and fails with a per-target message when the tarball entry in `SHA256SUMS` differs or the block's version differs. Only tarball entries are compared.
- `pin --sums <SHA256SUMS> --flake <flake.nix> --version <x.y.z>`: rewrites the generated block from the sums file. Used by the cut tool (Session D).
- `build.zig`: the cross-targets loop replaces `tar_run` and `gzip_run` with one `dist_stage archive` run. A separate `verify-pins` step runs `dist_stage pin-check` against `flake.nix` and `zig-out/dist/SHA256SUMS`. It is not part of `verify-dist`: an ordinary commit between two cuts builds the `build.zig.zon` version from a tree that differs from the released one, so its bytes never match the pins and must not be checked against them. Correction (2026-09-08): Session B first put the check inside `verify-dist`; Session D moves it out.
- Reproducibility test (`zig build test`): stage the same fixture tree under two different absolute directories with different mtimes and file order on disk, archive both, assert byte equality, then read the archive back with `std.tar` and assert names, modes, and sizes.
- `.gitea/workflows/gates.yml`: add `NPM_VERSION` to the top-level `env` next to `NODE_VERSION`; the frontend job asserts `node --version` and `npm --version` equal the pinned values before `npm ci`. The frontend job runs `npm run build` with `umask 022`, `LC_ALL=C`, `LANG=C`, `TZ=UTC`, `SOURCE_DATE_EPOCH=0`. `admin/vite.config.ts` reads no other environment (verified: only `VITEST`).
- `docs/how-to/verify-a-release.md` describes the archive layout (fixed modes, zero timestamps, gzip without a name or mtime) so a reader can reproduce it.
Deviation (Session B): the reproducibility test stages its two trees under two `std.testing.tmpDir` directories rather than two absolute paths. Zig 0.16.0's std exposes no way to read the current working directory, so an absolute path would need a raw `getcwd` syscall. The two roots still differ in path text, file creation order and mtimes, which is what the assertion is about.
## Session C: flake.nix
- `flake.nix` at the repo root with one input, `nixpkgs`, and `packages.{aarch64-linux,x86_64-linux}.default` built from the derivation that `rpi.mial.net` carries in `nixos/pkgs/nxdns.nix` today: `fetchurl` of `https://git.mial.net/mokhtar/nxdns/releases/download/v${version}/nxdns-${version}-${triple}.tar.gz`, `dontPatchELF`, `dontStrip`, install `nxdns`, `LICENSE`, `THIRD-PARTY-NOTICES`, and an `installCheckPhase` that asserts no `INTERP` and no `NEEDED` with `readelf` and that `nxdns version` output starts with `nxdns ${version}`. `meta.license = eupl12`, `meta.mainProgram = "nxdns"`.
- The version and the two hashes live in one delimited block that Session B's `pin` mode rewrites and that nothing else edits by hand:
```nix
# BEGIN GENERATED BY zig build cut
version = "0.0.16";
hashes = {
"aarch64-linux" = "sha256-...";
"x86_64-linux" = "sha256-...";
};
# END GENERATED BY zig build cut
```
The tool finds the two delimiter lines by their text after leading whitespace, keeps the begin line's indentation, and writes the six inner lines at that indentation (the hash lines two spaces deeper). A file with zero or two begin lines is an error.
- `flake.lock` is committed. The consumer sets `inputs.nxdns.inputs.nixpkgs.follows = "nixpkgs"`.
- `nix flake check --no-build` passes locally against the committed file. Evaluation does not fetch the tarball.
- `docs/how-to/install-with-nix.md` documents the consumer side: the flake input with a tag ref, the `follows` line, and the Renovate `nix` manager. It states the window between the tag push and the asset upload during which a fresh pin fetches a 404, and that Renovate only proposes tags that already exist.
- `flake.nix` is not a build input of the tarballs; the pins do not feed back into the bytes they describe.
## Session D: the cut tool pins before it commits
`tools/cut.zig` gains a `pin` stage between the changelog gates and the bump commit. The bump commit's exact diff becomes `build.zig.zon` plus `flake.nix`.
1. Toolchain parity: read `NODE_VERSION`, `NPM_VERSION`, and `ZIG_VERSION` from `.gitea/workflows/gates.yml` and fail unless `node --version`, `npm --version`, and `zig version` match exactly, and unless the host is x86_64 Linux with glibc (host-native bundler bindings, shipped per platform and libc). The check runs before the manifest write so a refusal leaves a clean tree.
2. Admin bundle: in `admin/`, run `npm ci` (npm replaces `node_modules` itself) then `npm run build` with the same normalized environment as the CI frontend job (`umask 022`, `LC_ALL=C`, `LANG=C`, `TZ=UTC`, `SOURCE_DATE_EPOCH=0`, `CI=true`, `npm_config_userconfig=/nonexistent/npmrc-user`, `npm_config_globalconfig=/nonexistent/npmrc-global`, `PATH` and `HOME` passed through, nothing else). Fail on a dirty `admin/dist` that the build did not produce.
3. Dist: run `zig build dist -Dversion-string=<x.y.z> -Dadmin-dist=admin/dist -Doptimize=ReleaseSafe` with the same normalized environment and a private `--cache-dir` under the scratch directory, so a stale local cache cannot leak into the bytes. The global cache stays shared: it is content-addressed, and a fresh one refetches every dependency and trips on zig 0.16.0's unzip, which expects `<global>/tmp` to exist.
4. Pin: run `dist_stage pin` on `zig-out/dist/SHA256SUMS` and `flake.nix`, then `zig build verify-dist` and `zig build verify-pins` with the same flags, then `nix flake check --no-build`.
5. Commit: the existing bump commit now includes `flake.nix`; the clean-tree gate accepts `admin/dist` and `zig-out` as ignored paths only.
Where CI runs the pin check:
- `.gitea/workflows/gates.yml` package job: after `verify-dist` and after the release payload is uploaded (so a mismatch still leaves the bytes downloadable), run `zig build verify-pins` when the pushed commit changed the declared `.version` against its first parent or changed `flake.nix` against it (the checkout needs depth 2). That covers the bump commit and a re-pin during a cut, the commits whose pins are otherwise unverified before the tag. Every other commit skips the step, and the skip is printed, not silent.
- `.gitea/workflows/release.yml` publish job: after `verify-dist` and before the image push, the draft, and every upload, run `zig build verify-pins` unconditionally. The tag's tree is the bump commit's tree, and the flake block's version must equal the tag.
The remaining stages (push, ci.yml wait, tag, release.yml wait) are unchanged. CI rebuilds the same bytes from the same inputs; `verify-pins` in the package job fails the run before the tag if the bytes differ, and the same step in the publish job fails before any image push, draft, or upload. `specs/release-cut.md` gains the pin stage and the parity requirement.
Cut tool tests: the `.gitea` version parser, the environment builder (asserts the allowed keys and nothing else), and the pin stage on a fixture repo.
Deviation (Session D): the pin stage is tested in pieces rather than on a fixture repository. There is no fixture-repo helper in `tools/cut.zig` to extend, and the stage's remaining content is four process spawns whose fixture would have to run `npm ci` and two cross-compiled release builds inside `zig build test`. What is tested instead is the parser (against a fixture and against the real `gates.yml`), the environment builder, and the `sh` wrapper — asserting it applies the umask, keeps the real command as the direct child, and reports its exit code. The spawns themselves follow the file's existing rule that process plumbing lives behind thin call sites and is not mocked. Two facts the spec left open: `umask(2)` is not in zig 0.16.0's `std.posix` (only `std.c.umask`, which these tools do not link), so the wrapper is the mechanism; and `build.zig` had no `pin-flake` step, so Session D added one beside `verify-pins`.
## Out of scope
Signing the flake outputs, a Hydra or cachix binary cache, Darwin packages, and a NixOS module. Renovate tag discovery on Gitea is a consumer-side acceptance test run once on `rpi.mial.net` after the first tagged release with a flake.
Result of that test (2026-09-09, v0.0.18, Renovate 42.99.0): no pull request. Renovate's `nix` manager advances `flake.lock` along the ref an input already tracks and never moves a `refs/tags/vX.Y.Z` ref; it returned an empty release list for every flake input, GitHub-hosted ones included. The flake itself is correct and v0.0.17 deployed through it. Tag bumps need a regex custom manager with the `gitea-tags` datasource plus a lock refresh in the same pull request, which the consumer owns; `docs/how-to/install-with-nix.md` section 4 now says so instead of claiming the `nix` manager does it. With that manager in place, Renovate opened rpi.mial.net PR 5 the same day, bumping the input to v0.0.18 and refreshing `flake.lock`: the acceptance test passes.
## Acceptance
- `zig build test` green, including the two-path reproducibility test.
- `nxdns version` prints two lines and no commit; `/api/version` has no `git_commit`; the admin footer shows `nxdns v<version>`.
- `zig build dist` produces tarballs byte-identical across two clean checkouts on the same toolchain; `verify-pins` fails when `flake.nix` disagrees with `SHA256SUMS`; `verify-dist` passes on any tree.
- `nix flake check --no-build` passes.
- `zig build cut -- patch` on a fixture reaches the bump commit with `build.zig.zon` and `flake.nix` as its only diff.
- CHANGELOG `## [Unreleased]` records the flake, the archive change, and the removal of the commit sha from the version surfaces.
+13
View File
@@ -1377,3 +1377,16 @@ Deviations from the text above, recorded after the first six sessions verified.
**S9 integration + fuzz.** Case 3 splits into the two real failure modes: a checksum-mismatched file marks the source `.load_failed` and the reload succeeds without it (generation advances); only a checksum-clean but malformed body fails `reload` and leaves the previous snapshot serving. Case 7 proves the cap from the response head: an explicit `content-length` of 100 MiB returns `error.BodyTooLarge` before any body streams. Case 8 back-dates `last_updated` through `updateSourceStats` and asserts inode and mtime of the compiled file are unchanged. Case 15 asserts `budget/2 <= elapsed < 2*budget` (the POSIX backend wakes ~0.8 ms early on a 200 ms deadline). The fuzz corpus is inline — `tests/fuzz/corpus.zig` imports the `dns` module, which the blocklist-fuzz module does not have. Fuzz targets assert properties, not only absence of crashes: `Line.text` windows the input (pointer containment), `covers_apex` only on `.wildcard`, `detectFormat`'s answer survives `parseLine` over the same bytes, and `matches` is exercised on rejected patterns.
**Final wiring.** `build.zig` gained the `blocklist-fuzz` artifact (module import `parsers``src/filter/parsers.zig`, LLVM backend under `-Dfuzz`), hung off `test_step` beside the dns fuzz artifact. Evaluation: `zig build test`, `zig build test -Dintegration` and `zig build cross` all exit 0; both cross executables are statically linked.
## Addendum (2026-09-12): a download failure names its cause
Observed on the Pi (nxdns 0.0.17): four of eight sources fail on every refresh pass since 2026-09-11, the same four each time, and the log line for each is `download failed: TlsFailed` one pass and `download failed: ReceiveFailed` the next. The same binary loads all eight from another machine. The line cannot tell a TLS alert from a reset connection from a truncated chunk, because `fetcher.mapError` collapses the cause into the six-member taxonomy and `reportDownloadFailure` logs only the taxonomy name. The Diagnostics event carries the same text. A reader of either has nothing to act on. `upstream/doh_client.zig` already unwraps the concrete cause that `std.http.Client` stashes on the connection (`sendCause`, `headCause`, `bodyCause`, `readCause`); the fetcher does not.
Change:
- `fetcher.zig`: `Fetcher` gains `last_failure: ?Failure`, cleared at the top of `fetch` beside `last_status`, set on every error return. `Failure` is `{ phase: Phase, cause: anyerror, status: ?std.http.Status, bytes_read: u64 }`. `Phase` gains `receive_body`, distinct from the head. `cause` is the unwrapped concrete error: the three unwrap helpers move from `doh_client.zig` into `upstream/transport.zig` (or a sibling file both import; the coder picks the smaller diff), keep their tests, and both clients call them. The taxonomy `Error` and `mapError` are unchanged; a caller that ignores `last_failure` sees exactly what it saw before. `bytes_read` counts body bytes delivered to the writer before the failure, so a body that dies at byte 0 and one that dies at 5 MB read differently.
- `manager.zig`: `fetchWithin` measures elapsed milliseconds on the awake clock. `reportDownloadFailure` takes the failure and elapsed time and logs one line: `blocklist <label>: download failed: <taxonomy> (<phase>, cause <CauseName>, http <status or none>, <bytes> bytes, <ms> ms)`. On the `Timeout` outcome of the expiry race the losing fetch is cancelled before the line is built, and it records itself on the way out, so the line names the phase it was in, `cause Canceled`, the status if a head arrived, and the bytes delivered so far: `Timeout (receive_body, cause Canceled, http 200, 5242880 bytes, 300000 ms)`. `download` clears `last_failure` and `last_status` before anything else runs, so a null record means no fetch reached the network for this source (a local file failure, or no concurrency to spawn the race); that prints as `<taxonomy> (no fetch, http none, - bytes, <ms> ms)`. Elapsed is captured when the race resolves, before the loser is cancelled, so a timeout's milliseconds are the time the transfer was given. `SourceStatus.fail` receives the same compact text, so the Diagnostics event detail shows it; `max_error_len` rises from 128 to 192 so the longest cause name and the numbers fit without truncation (check the longest `std.crypto.tls.Client.ReadError` and `std.http.Client` error names; `events.Store.max_detail_len` is 512 and is not touched).
- Bound: this changes the content of an existing warning line, not its frequency. One line per failed source per pass, at most `sources × passes` per day; a pass runs once per `blocklist_update.interval_hours` or on a manual update. No per-chunk or per-retry logging is added. Nothing new is written at info or debug level.
- Tests: `mapError` tests unchanged; a test that a head failure with a stashed `ConnectionResetByPeer` yields `last_failure.cause == error.ConnectionResetByPeer` and phase `receive` (the doh tests' stub connection pattern); a test that a body failure records `bytes_read` and `receive_body`; a `reportDownloadFailure` format test with the fixed text `blocklist source 3 'ads' 'https://lists.example': download failed: ReceiveFailed (receive_body, cause HttpChunkTruncated, http 200, 1048576 bytes, 812 ms)`; the existing secrecy assertions on the label stay.
Out of scope: a TLS handshake failure stays `TlsInitializationFailed`, because `std.http.Client` collapses every handshake fault at `Client.zig:1470` before this code sees it; distinguishing an alert from a certificate failure would mean driving `std.crypto.tls.Client` by hand. Retries and a per-source backoff are not added.
+33 -2
View File
@@ -20,7 +20,7 @@ Recipes only — no variables, no embedded logic:
Wired like the other host tools (`hostTool` + `addRunArtifact`, see build.zig ~230): `zig build cut -- {major|minor|patch}`. NOT installed to zig-out/bin. Its tests join `zig build test`.
Constants: one repo API base `https://git.mial.net/api/v1/repos/mokhtar/nxdns` (the tool can only ever target this repo — no configurability). The runs API needs a token (verified: anonymous GET is 401); read it from `~/.config/tea/config.yml` (logins entry for git.mial.net); a missing token is a clear error naming the file.
Constants: one repo API base `https://git.mial.net/api/v1/repos/mokhtar/nxdns` (the tool can only ever target this repo — no configurability); the endpoints under it are `/actions/runs`, `/commits/{sha}/status`, `/releases/tags/{tag}` and `/actions/runs/{id}/rerun`, the last being the only non-GET this program makes. The runs API needs a token (verified: anonymous GET is 401); read it from `~/.config/tea/config.yml` (logins entry for git.mial.net); a missing token is a clear error naming the file.
### Sequence
@@ -38,10 +38,12 @@ Constants: one repo API base `https://git.mial.net/api/v1/repos/mokhtar/nxdns` (
A failed run must not strand the operator:
- Bump pushed, then failure: rerun continues (preflight sees the version already bumped).
- Local tag exists but never reached origin: verify it is an annotated tag by this tool's convention pointing at the current HEAD — adopt it; otherwise refuse with the exact `git tag -d` to run. Never delete a tag that exists on origin.
- Release run failed, retryably: one automatic rerun (`POST {api_base}/actions/runs/{id}/rerun`, 201, then the wait targets that same run id — a rerun keeps it — and treats the run as restarted only once the forge reports a HIGHER `run_attempt`, so the concluded previous attempt is never read as the result and a rerun that finishes between two polls still is). Retryable only while the run is on its first attempt (`run_attempt` is 1; an absent attempt number is not read as 1), the conclusion is `failure` (never `cancelled`), at least one commit-status context is in a failure state, every context in a failure state is a gate (`Gates / …` or `Release / gates`, event suffix dropped), and the release object for the tag is absent or a draft; a published release is terminal, as `release.yml`'s guard already states. The bound is the forge's attempt count and not a counter in this process, so a run already rerun by hand or by an earlier invocation is never rerun again. The statuses are parsed STRICTLY for this decision — an entry missing a string `context`, `status` or `target_url` refuses the whole payload — because an unreadable entry is not counted as a failure and would leave a list of nothing but gates; the failing-job report keeps the tolerant parse. Anything the tool cannot read — the statuses, the release object, the attempt number — is terminal rather than retried.
- Tag pushed, then failure: the tag on origin is no longer a reason to derive the next version. When origin's peeled tag object is the current HEAD and no release is published for it, the cut resumes at the release stage — no bump, no push, no CI wait, no tag, and no preflight, because everything the preflight guards has already happened — and reports it (`cut: resuming v0.0.16 at the release stage: the tag is on origin at HEAD and no release is published`). Peeling to HEAD is not on its own enough to resume on a tag: before the release wait, origin's tag object is verified the way the adopt path verifies a local one — annotated (a tag whose ref line and peeled line are one object is lightweight and refused), fetched when it is not here and compared against origin's object id when it is, carrying this tool's `v<version>` message, pointing at that commit, and signed under the fingerprint `release.yml` pins — so a lightweight or unsigned tag somebody pushed at HEAD is a refusal naming what is wrong, never a deletion or a move. A published release, or a tag pointing anywhere else, derives the next version as before. `planVersion` takes the manifest, the bump kind and that one three-state fact, and stays a pure table-tested function.
### Tests (in-file, join `zig build test`)
Pure functions unit-tested: semver validation (accept/reject table incl. leading zeroes, `v` prefix), bump-kind parse, derivation table with the minor/major resets and overflow refusals, derive-vs-resume decision for all three kinds, zon `.version` parse + rewrite round-trip, changelog heading + non-empty body check, runs-JSON → decision (running / success / failure / no-run), tea-config token extraction. Process spawning and HTTP live behind thin call sites and are not mocked.
Pure functions unit-tested: semver validation (accept/reject table incl. leading zeroes, `v` prefix), bump-kind parse, derivation table with the minor/major resets and overflow refusals, derive-vs-resume decision for all three kinds, zon `.version` parse + rewrite round-trip, changelog heading + non-empty body check, runs-JSON → decision (running / success / failure / no-run, the run-id narrowing a rerun needs, and whether an observed `run_attempt` has advanced past the one a rerun was asked for), failing-contexts JSON → retryable or terminal, release-object status and JSON → absent / draft / published, `ls-remote` peeling an annotated tag to its commit, tea-config token extraction. Process spawning and HTTP live behind thin call sites and are not mocked.
## Anti-requirements
@@ -71,3 +73,32 @@ Pure functions unit-tested: semver validation (accept/reject table incl. leading
Every step that cannot answer — the `ls-remote`, the `git show`, the extraction, an unreadable CHANGELOG.md — is a soft FAIL naming the step. A gate that does not know whether the schema moved must never report that it did not.
Fixing a FAIL is a sentence in the changelog, not a flag: there is no override, because the only thing the gate asks for is that the release notes be true.
## Addendum: the pin stage (milestone 40)
`flake.nix` carries the SHA256 of each release tarball, and a consumer pins the flake to a tag. The hashes therefore have to be written into the file that the tag points at, which means before the bump commit — the only commit the cut makes. The cut builds the release locally, pins what it built, and CI proves the pins describe the bytes it rebuilds.
This amends step 3 of the sequence above. What was one write and one commit is now three moves, in this order and no other:
1. **Toolchain parity.** Read `ZIG_VERSION`, `ZIG_TARBALL_SHA256`, `NODE_VERSION` and `NPM_VERSION` out of the top-level `env:` block of `.gitea/workflows/gates.yml` — a line parser, not a YAML library — and refuse unless `node --version` (less its leading `v`) and `npm --version` report those exact strings, and unless the host is x86_64 Linux with glibc (the admin bundle uses host-native Rolldown and Lightning CSS bindings, shipped per platform and libc, and CI is an x86_64 Ubuntu runner). Zig is not taken from `PATH` at all: the cut obtains the official ziglang.org tarball `https://ziglang.org/download/<ZIG_VERSION>/zig-x86_64-linux-<ZIG_VERSION>.tar.xz`, caches it under `$HOME/.cache/nxdns-cut/zig-<ZIG_VERSION>-<first 12 hex of ZIG_TARBALL_SHA256>/` (keyed by the digest too, so a changed pin never reuses an old extraction), and builds every release byte with that binary by absolute path. The first real cut pinned hashes CI could not reproduce because the Arch Linux 0.16.0 package is built against the system LLVM and emits different machine code than the tarball CI installs, and both print `0.16.0` — a version string cannot see the difference, so the bytes are pinned instead. When the cached binary is absent the tarball is downloaded with `curl -fsSL`, checked against `ZIG_TARBALL_SHA256` (setup-zig verifies the same bytes by minisign on the CI side), extracted with `tar -xJf`, and the extracted `zig version` is asserted; when it is present that last assertion still runs. There is no fall back to `PATH`. The `zig` on `PATH` still compiles the cut tool itself — that is `zig build cut`, whose bytes nobody hashes. The cut and CI must build with one toolchain, because the cut writes the hashes and CI recomputes them. A mismatch found here costs nothing; found in the release run it costs a public tag. The pins live in `gates.yml` and are read from there rather than copied, so the two can never drift; `release.yml` mirrors them and a test asserts the two files agree. Both halves run before the manifest is rewritten, so a refusal leaves the working tree clean.
2. **Write `build.zig.zon`.** The manifest declares the new version, atomically, and is reparsed off disk. Nothing is committed yet.
3. **Build the release.** In `admin/`, `npm ci` then `npm run build`; then `dist` through the official zig: `<cached zig> build dist -Dversion-string=<next> -Dadmin-dist=admin/dist -Doptimize=ReleaseSafe`, with `--cache-dir` under a scratch directory that is deleted and recreated first (the global cache stays shared: it is content-addressed, and a fresh one refetches every dependency and trips on zig 0.16.0's unzip, which expects `<global>/tmp` to exist). Every one of these runs under a constructed environment holding exactly nine variables — `PATH` and `HOME` from the parent, and `LC_ALL=C`, `LANG=C`, `TZ=UTC`, `SOURCE_DATE_EPOCH=0`, `CI=true`, `npm_config_userconfig=/nonexistent/npmrc-user`, `npm_config_globalconfig=/nonexistent/npmrc-global` (so neither `~/.npmrc` under the passed-through `HOME` nor the node install's `etc/npmrc` is read; the parity check refuses to run if either path exists, and CI asserts the same) — with `umask 022`. The umask arrives through an `sh -c 'umask 022 && exec "$@"'` wrapper because zig 0.16.0 exposes `umask(2)` only as a libc extern that these tools do not link, and `SpawnOptions` has no field for it. The wrapper's `exec` resolves the real command through the child's `PATH`, which is why `PATH` is a passthrough and not a pinned value.
4. **Pin and verify.** Through the same official zig, `build pin-flake` rewrites the generated block of `flake.nix` from `zig-out/dist/SHA256SUMS`; then `build verify-dist` and `build verify-pins` with the same flags and cache directories, then `nix flake check --no-build` — evaluation only, because the tarballs the block now names do not exist until the release run uploads them.
5. **Commit.** One `git commit -S`, taking `build.zig.zon` and `flake.nix`. The working-tree diff is asserted to be a subset of those two paths and to contain the manifest.
The bump must precede the build because `verify-dist` refuses a build whose `-Dversion-string` disagrees with the manifest; the build must precede the commit because the pins belong to the commit. There is no ordering that satisfies both differently.
On a resumed cut — the bump commit exists and its tag does not — the pin stage still runs in full, and the diff it leaves must be EMPTY. That emptiness is the check: the committed hashes reproduce on this machine today. A non-empty diff is a refusal, because the release CI is about to rebuild would not match the committed pins either.
`admin/dist` and `zig-out` are ignored paths, so the preflight's clean-tree gate is unaffected by the trees this stage writes; the build caches live in the scratch directory and never touch the repository.
### Where CI checks the pins
`verify-pins` is a step of its own and deliberately not part of `verify-dist`. An ordinary commit between two cuts builds the same `build.zig.zon` version from a different tree, so its bytes legitimately differ from the pins, and checking them inside `verify-dist` would fail every such build.
- `gates.yml`, package job: after `verify-dist`, run `verify-pins` only when `HEAD` changes `build.zig.zon` against its first parent. That commit is the bump commit, the one commit whose pins nothing has checked yet. The job's checkout takes `fetch-depth: 2` for the comparison. Every other commit prints why it skipped. A missing first parent is an error, not a skip: it means the checkout is shallower than declared and the question went unanswered.
- `release.yml`, publish job: after `verify-dist` and before the image push, the draft and every upload, run `verify-pins` unconditionally. The tag's tree is the bump commit's tree, so the block must pin these bytes and name this version.
### Tests
The `gates.yml` env parser against a fixture and against the real file; the environment builder, asserting the nine keys and nothing else; and the `sh` wrapper, asserting it applies the umask, keeps the real command as the direct child, and reports its exit code. The stage's remaining steps are process spawns of `npm`, `zig` and `nix`, which are not mocked — the same rule the rest of this file follows.
+1 -2
View File
@@ -453,9 +453,8 @@ pub fn runHelp(r: Runner) u8 {
}
pub fn runVersion(r: Runner) u8 {
r.out.print("nxdns {s} ({s})\nzig {s}\n", .{
r.out.print("nxdns {s}\nzig {s}\n", .{
version.string,
version.git_commit,
version.zig_version_string,
}) catch return finish(r, exit_runtime);
return finish(r, exit_ok);
+295 -28
View File
@@ -10,7 +10,9 @@
//!
//! There is no timeout parameter and no sleep. `std.http.Client` has no
//! per-request deadline, so the caller runs `fetch` under `io.concurrent` and
//! cancels the future; this file only propagates `error.Canceled`.
//! cancels the future. A cancel during the body copy surfaces as
//! `error.ReceiveFailed` with `Canceled` in `last_failure`; elsewhere it
//! propagates as `error.Canceled`.
const std = @import("std");
const transport = @import("../upstream/transport.zig");
@@ -43,6 +45,30 @@ pub const Result = struct {
status: std.http.Status,
};
/// The concrete fault behind an `Error`. `cause` is unwrapped from whatever
/// `std.http.Client` stashed on the connection or the response, so it names a
/// record-layer TLS fault or an HTTP framing fault rather than the collapsed
/// `error.ReadFailed`.
pub const Failure = struct {
phase: Phase,
cause: anyerror,
/// The response status if a head arrived before the failure.
status: ?std.http.Status,
/// Body bytes delivered to the caller's writer before the failure, and a
/// lower bound rather than an exact count: at least this many reached `w`.
/// A single `Reader.stream` call can hand bytes to `w` and then fail, and
/// that partial delivery is not counted. Zero on every phase before
/// `receive_body`.
bytes_read: u64,
};
/// Which call failed. The phase is what decides the classification, and only
/// the call site knows it — guessing it from an error name would be wrong the
/// first time two phases shared an error. `receive_body` is distinct from
/// `receive`: the head arrived, so the fault is in the transfer, not the
/// response.
pub const Phase = enum { connect, send, receive, receive_body };
pub const Fetcher = struct {
/// Caller-owned; shared across sources, pools connections.
http: *std.http.Client,
@@ -55,6 +81,12 @@ pub const Fetcher = struct {
/// one. `error.HttpStatus` carries no `Result`, and the operator's message
/// needs the number, so it is readable here after a failed `fetch`.
last_status: ?std.http.Status = null,
/// What the most recent `fetch` failed on, or null if it succeeded or has
/// not run. The taxonomy `Error` a caller receives names six outcomes; this
/// names the one concrete fault behind the outcome, so an operator reading
/// the log can tell a TLS alert from a reset connection from a truncated
/// chunk.
last_failure: ?Failure = null,
/// GETs `url` and streams the body into `w`.
pub fn fetch(
@@ -71,9 +103,10 @@ pub const Fetcher = struct {
std.debug.assert(self.transfer_buf.len >= min_transfer_buf);
std.debug.assert(self.redirect_buf.len >= redirect_buffer_len);
const uri = try parseUrl(url);
self.last_status = null;
self.last_failure = null;
const uri = parseUrl(url) catch |err| return self.record(err, .connect, err, null, 0);
var req = self.http.request(.GET, uri, .{
.keep_alive = true,
@@ -83,25 +116,88 @@ pub const Fetcher = struct {
// second failure surface, for a download that runs once a day.
.accept_encoding = .{ .override = "identity" },
},
}) catch |err| return mapError(err, .connect);
}) catch |err| return self.record(mapError(err, .connect), .connect, err, null, 0);
defer req.deinit();
req.sendBodiless() catch |err| return mapError(err, .send);
req.sendBodiless() catch |err|
return self.record(mapError(err, .send), .send, transport.sendCause(&req, err), null, 0);
var resp = req.receiveHead(self.redirect_buf) catch |err| return mapError(err, .receive);
var resp = req.receiveHead(self.redirect_buf) catch |err|
return self.record(mapError(err, .receive), .receive, transport.headCause(&req, err), null, 0);
self.last_status = resp.head.status;
if (resp.head.status != .ok) return error.HttpStatus;
if (resp.head.status != .ok) {
return self.record(error.HttpStatus, .receive, error.HttpStatus, resp.head.status, 0);
}
// `content-type` is deliberately not checked: blocklists are served as
// text/plain, application/octet-stream and text/html alike, and the
// compiler's invalid-line counters are the honest signal about content.
if (resp.head.content_length) |declared| {
if (declared > max_body_bytes) return error.BodyTooLarge;
if (declared > max_body_bytes) {
return self.record(
error.BodyTooLarge,
.receive,
error.BodyTooLarge,
resp.head.status,
0,
);
}
}
const body = resp.reader(self.transfer_buf);
return .{ .bytes_read = try pumpBody(body, w, max_body_bytes), .status = resp.head.status };
var pump: Pump = .{};
const bytes_read = pumpBody(body, w, max_body_bytes, &pump) catch |err| return self.record(
err,
.receive_body,
transport.bodyCause(&resp, pump.cause),
resp.head.status,
pump.delivered,
);
return .{ .bytes_read = bytes_read, .status = resp.head.status };
}
/// Stores the failure and hands back `mapped` unchanged.
///
/// `mapped` is the taxonomy error the call site already decided, never one
/// this function derives. `mapError` answers for a raw std error, and the
/// call sites that raise a taxonomy member directly — a non-200 head, a
/// declared length over the cap — pass that member: feeding either back
/// through `mapError` would find no case for it and fall to the phase
/// default, turning `error.HttpStatus` into `error.ReceiveFailed`.
///
/// `cause` never reaches the returned error. The unwrap is for the
/// operator's line, and a caller that ignores `last_failure` sees exactly
/// what it saw before the unwrap existed.
fn record(
self: *Fetcher,
mapped: Error,
phase: Phase,
cause: anyerror,
status: ?std.http.Status,
bytes_read: u64,
) Error {
self.last_failure = .{
.phase = phase,
.cause = cause,
.status = status,
.bytes_read = bytes_read,
};
return mapped;
}
};
/// What `pumpBody` reports alongside its error: the reader error before
/// `mapError` collapses it, and the bytes already handed to `w`. `pumpBody`
/// knows nothing of `std.http`, so the unwrap of `cause` happens in `fetch`,
/// which holds the response.
const Pump = struct {
cause: anyerror = error.Unexpected,
delivered: u64 = 0,
fn fail(self: *Pump, cause: anyerror, mapped: Error) Error {
self.cause = cause;
return mapped;
}
};
@@ -116,17 +212,19 @@ pub const Fetcher = struct {
/// `self.transfer_buf` — makes `@memcpy` copy the buffer onto itself
/// (Reader.zig:677) and panics the process on the first read that finds bytes
/// already buffered.
fn pumpBody(body: *std.Io.Reader, w: *std.Io.Writer, cap: usize) Error!u64 {
fn pumpBody(body: *std.Io.Reader, w: *std.Io.Writer, cap: usize, out: *Pump) Error!u64 {
var total: usize = 0;
while (total < cap) {
out.delivered = total;
total += body.stream(w, .limited(cap - total)) catch |err| switch (err) {
error.EndOfStream => return total,
// The caller owns `w` and can read the concrete failure from its
// own writer; this taxonomy has no member for a failing sink.
error.WriteFailed => return error.Unexpected,
else => |e| return mapError(e, .receive),
error.WriteFailed => return out.fail(err, error.Unexpected),
else => |e| return out.fail(e, mapError(e, .receive_body)),
};
}
out.delivered = total;
// `cap` written exactly. One more byte separates a body that fits from one
// that was cut off, and it must not reach `w`.
@@ -134,10 +232,10 @@ fn pumpBody(body: *std.Io.Reader, w: *std.Io.Writer, cap: usize) Error!u64 {
var probe: std.Io.Writer.Discarding = .init(&probe_buf);
const extra = body.stream(&probe.writer, .limited(1)) catch |err| switch (err) {
error.EndOfStream => return total,
error.WriteFailed => return error.Unexpected,
else => |e| return mapError(e, .receive),
error.WriteFailed => return out.fail(err, error.Unexpected),
else => |e| return out.fail(e, mapError(e, .receive_body)),
};
return if (extra == 0) total else error.BodyTooLarge;
return if (extra == 0) total else out.fail(error.BodyTooLarge, error.BodyTooLarge);
}
/// A scheme other than `http`/`https`, an unparseable URL and a URL with no
@@ -151,11 +249,6 @@ fn parseUrl(url: []const u8) Error!std.Uri {
return uri;
}
/// Which call failed. The phase is what decides the classification, and only
/// the call site knows it — guessing it from an error name would be wrong the
/// first time two phases shared an error.
const Phase = enum { connect, send, receive };
// `error.X` in an expression names a member into existence rather than
// referring to one, so the switch in `mapError` would keep compiling — and
// silently stop matching — if std renamed either of these. This is what fails
@@ -179,9 +272,10 @@ fn errorSetHas(comptime Set: type, comptime name: []const u8) bool {
/// `std.http.Client` collapses every handshake fault into
/// `error.TlsInitializationFailed` (Client.zig:1470) and every bundle fault into
/// `error.CertificateBundleLoadFailure`, and this file never unwraps a
/// connection's stashed read cause — it maps the collapsed `error.ReadFailed` by
/// phase — so the record-layer members of `std.crypto.tls.Client.ReadError`
/// cannot arrive here. `doh_client.zig` does unwrap, and names them.
/// connection's stashed read cause *for classification* — it maps the collapsed
/// `error.ReadFailed` by phase — so the record-layer members of
/// `std.crypto.tls.Client.ReadError` cannot arrive here. `fetch` does unwrap
/// them, into `Failure.cause`, which no classification reads.
fn mapError(err: anyerror, phase: Phase) Error {
if (transport.mapLocal(err)) |local| return narrowLocal(local);
switch (err) {
@@ -198,7 +292,7 @@ fn mapError(err: anyerror, phase: Phase) Error {
return switch (phase) {
.connect => error.ConnectFailed,
.send => error.SendFailed,
.receive => error.ReceiveFailed,
.receive, .receive_body => error.ReceiveFailed,
};
}
@@ -230,7 +324,8 @@ test "pumpBody streams a fully buffered body without aliasing its source" {
var out: [payload.len]u8 = undefined;
var sink: std.Io.Writer = .fixed(&out);
const n = try pumpBody(&body, &sink, max_body_bytes);
var pump: Pump = .{};
const n = try pumpBody(&body, &sink, max_body_bytes, &pump);
try testing.expectEqual(@as(u64, payload.len), n);
try testing.expectEqualStrings(payload, out[0..payload.len]);
}
@@ -241,7 +336,8 @@ test "pumpBody accepts a body of exactly the cap" {
var out: [payload.len]u8 = undefined;
var sink: std.Io.Writer = .fixed(&out);
const n = try pumpBody(&body, &sink, payload.len);
var pump: Pump = .{};
const n = try pumpBody(&body, &sink, payload.len, &pump);
try testing.expectEqual(@as(u64, payload.len), n);
try testing.expectEqualStrings(payload, out[0..payload.len]);
}
@@ -252,10 +348,12 @@ test "pumpBody refuses a body one byte over the cap" {
var out: [payload.len]u8 = undefined;
var sink: std.Io.Writer = .fixed(&out);
var pump: Pump = .{};
try testing.expectError(
error.BodyTooLarge,
pumpBody(&body, &sink, payload.len - 1),
pumpBody(&body, &sink, payload.len - 1, &pump),
);
try testing.expectEqual(error.BodyTooLarge, pump.cause);
}
test "pumpBody reports an empty body as zero bytes" {
@@ -263,7 +361,8 @@ test "pumpBody reports an empty body as zero bytes" {
var sink_buf: [0]u8 = .{};
var discarding: std.Io.Writer.Discarding = .init(&sink_buf);
try testing.expectEqual(@as(u64, 0), try pumpBody(&body, &discarding.writer, max_body_bytes));
var pump: Pump = .{};
try testing.expectEqual(@as(u64, 0), try pumpBody(&body, &discarding.writer, max_body_bytes, &pump));
}
fn undefinedFetcher(transfer_buf: []u8, redirect_buf: []u8) Fetcher {
@@ -359,3 +458,171 @@ test "caps are the values the memory budget was sized against" {
try testing.expectEqual(@as(usize, 64 * 1024 * 1024), max_body_bytes);
try testing.expectEqual(@as(usize, 8192), redirect_buffer_len);
}
/// Only the fields the unwrap helpers read are set; the rest of a `Connection`
/// is two buffered streams, a host name and a pool node. `.plain` for the
/// reason `doh_client.zig` gives: `getReadError` reaches a TLS connection's
/// cause through `@fieldParentPtr`, which on a stub would read memory that was
/// never a `Tls`.
fn stubConnection(read_err: ?std.Io.net.Stream.Reader.Error) std.http.Client.Connection {
var connection: std.http.Client.Connection = undefined;
connection.protocol = .plain;
connection.stream_reader.err = read_err;
connection.stream_writer.err = null;
return connection;
}
test "a head failure records the stashed cause, not the collapsed read error" {
var connection = stubConnection(error.ConnectionResetByPeer);
var req: std.http.Client.Request = undefined;
req.connection = &connection;
var transfer_buf: [min_transfer_buf]u8 = undefined;
var redirect_buf: [redirect_buffer_len]u8 = undefined;
var f = undefinedFetcher(&transfer_buf, &redirect_buf);
const err = error.ReadFailed;
const mapped = f.record(mapError(err, .receive), .receive, transport.headCause(&req, err), null, 0);
// The taxonomy the caller sees is what it was before the unwrap existed.
try testing.expectEqual(error.ReceiveFailed, mapped);
const failure = f.last_failure.?;
try testing.expectEqual(Phase.receive, failure.phase);
try testing.expectEqual(error.ConnectionResetByPeer, failure.cause);
try testing.expectEqual(@as(?std.http.Status, null), failure.status);
try testing.expectEqual(@as(u64, 0), failure.bytes_read);
}
test "a body failure records the bytes delivered before it and the body phase" {
const payload = "0.0.0.0 ads.example.com\n";
var buffer: [payload.len]u8 = (payload ++ "").*;
// Buffered bytes reach `w` first; the vtable is only asked for more once
// the buffer is drained, which is the shape of a transfer cut off mid-body.
var body: std.Io.Reader = .{
.vtable = std.Io.Reader.failing.vtable,
.buffer = &buffer,
.seek = 0,
.end = buffer.len,
};
var sink_buf: [0]u8 = .{};
var discarding: std.Io.Writer.Discarding = .init(&sink_buf);
var pump: Pump = .{};
try testing.expectError(
error.ReceiveFailed,
pumpBody(&body, &discarding.writer, max_body_bytes, &pump),
);
try testing.expectEqual(@as(u64, payload.len), pump.delivered);
var connection = stubConnection(null);
var req: std.http.Client.Request = undefined;
req.connection = &connection;
req.reader.body_err = error.HttpChunkTruncated;
const resp: std.http.Client.Response = .{ .request = &req, .head = undefined };
var transfer_buf: [min_transfer_buf]u8 = undefined;
var redirect_buf: [redirect_buffer_len]u8 = undefined;
var f = undefinedFetcher(&transfer_buf, &redirect_buf);
const mapped = f.record(
mapError(error.ReadFailed, .receive_body),
.receive_body,
transport.bodyCause(&resp, pump.cause),
.ok,
pump.delivered,
);
try testing.expectEqual(error.ReceiveFailed, mapped);
const failure = f.last_failure.?;
try testing.expectEqual(Phase.receive_body, failure.phase);
try testing.expectEqual(error.HttpChunkTruncated, failure.cause);
try testing.expectEqual(@as(?std.http.Status, .ok), failure.status);
try testing.expectEqual(@as(u64, payload.len), failure.bytes_read);
}
test "a rejected url records the phase without claiming a connect failure" {
var transfer_buf: [min_transfer_buf]u8 = undefined;
var redirect_buf: [redirect_buffer_len]u8 = undefined;
var f = undefinedFetcher(&transfer_buf, &redirect_buf);
var sink_buf: [0]u8 = .{};
var discarding: std.Io.Writer.Discarding = .init(&sink_buf);
try testing.expectError(
error.BadUrl,
f.fetch(undefined, "ftp://example.com/list.txt", &discarding.writer),
);
try testing.expectEqual(error.BadUrl, f.last_failure.?.cause);
}
test "record returns the taxonomy member the call site raised, not a remapped one" {
// `error.HttpStatus` and `error.BodyTooLarge` are raised by `fetch` itself
// rather than by std, so `mapError` has no case for either and would fall
// to the phase default. `record` must not run them through it.
var transfer_buf: [min_transfer_buf]u8 = undefined;
var redirect_buf: [redirect_buffer_len]u8 = undefined;
var f = undefinedFetcher(&transfer_buf, &redirect_buf);
try testing.expectEqual(
error.HttpStatus,
f.record(error.HttpStatus, .receive, error.HttpStatus, .not_found, 0),
);
try testing.expectEqual(@as(?std.http.Status, .not_found), f.last_failure.?.status);
try testing.expectEqual(error.HttpStatus, f.last_failure.?.cause);
try testing.expectEqual(
error.BodyTooLarge,
f.record(error.BodyTooLarge, .receive, error.BodyTooLarge, .ok, 0),
);
try testing.expectEqual(error.BodyTooLarge, f.last_failure.?.cause);
// This is why: `mapError` is unchanged and does collapse both to the phase
// default. `fetch` must never route these two through it.
try testing.expectEqual(error.ReceiveFailed, mapError(error.HttpStatus, .receive));
try testing.expectEqual(error.ReceiveFailed, mapError(error.BodyTooLarge, .receive));
}
test "a cancelled body read records the cancellation and the progress so far" {
// This is the shape the expiry race leaves behind. `Select.cancelDiscard`
// cancels the fetch, the cancellation surfaces from the socket as a
// stashed `error.Canceled` under a collapsed `error.ReadFailed`, and the
// fetch records it on its way out. The manager then reports its own
// `error.Timeout` beside this record.
const payload = "0.0.0.0 ads.example.com\n";
var buffer: [payload.len]u8 = (payload ++ "").*;
var body: std.Io.Reader = .{
.vtable = std.Io.Reader.failing.vtable,
.buffer = &buffer,
.seek = 0,
.end = buffer.len,
};
var sink_buf: [0]u8 = .{};
var discarding: std.Io.Writer.Discarding = .init(&sink_buf);
var pump: Pump = .{};
try testing.expectError(
error.ReceiveFailed,
pumpBody(&body, &discarding.writer, max_body_bytes, &pump),
);
var connection = stubConnection(error.Canceled);
var req: std.http.Client.Request = undefined;
req.connection = &connection;
req.reader.body_err = null;
const resp: std.http.Client.Response = .{ .request = &req, .head = undefined };
var transfer_buf: [min_transfer_buf]u8 = undefined;
var redirect_buf: [redirect_buffer_len]u8 = undefined;
var f = undefinedFetcher(&transfer_buf, &redirect_buf);
// `pumpBody` maps the collapsed `error.ReadFailed` it was handed, not the
// unwrapped cause, so the taxonomy stays `ReceiveFailed` here. That is what
// this path returned before `Failure` existed, and the addendum keeps it:
// only the recorded cause is new. The manager reports `error.Timeout` from
// the expiry race anyway, so nothing reads this return value on this path.
const cause = transport.bodyCause(&resp, error.ReadFailed);
const mapped = f.record(error.ReceiveFailed, .receive_body, cause, .ok, pump.delivered);
try testing.expectEqual(error.ReceiveFailed, mapped);
try testing.expectEqual(Phase.receive_body, f.last_failure.?.phase);
try testing.expectEqual(error.Canceled, f.last_failure.?.cause);
try testing.expectEqual(@as(?std.http.Status, .ok), f.last_failure.?.status);
try testing.expectEqual(@as(u64, payload.len), f.last_failure.?.bytes_read);
}
+69 -2
View File
@@ -1003,6 +1003,19 @@ test "5: a redirect is followed to the same result" {
try testing.expect(decision.blocked);
}
/// The download failure line ends in numbers no test can predict — an elapsed
/// time always, and a byte count where the failure is a cancelled transfer. The
/// fixed part is compared whole; the tail only has to be digits, spaces and the
/// two words those numbers carry.
fn expectFailureText(expected_prefix: []const u8, actual: []const u8) !void {
try testing.expectEqualStrings(expected_prefix, actual[0..@min(expected_prefix.len, actual.len)]);
const tail = actual[expected_prefix.len..];
try testing.expect(std.mem.endsWith(u8, tail, " ms)"));
for (tail[0 .. tail.len - " ms)".len]) |c| {
try testing.expect(std.ascii.isDigit(c) or c == ' ' or c == ',' or std.ascii.isAlphabetic(c));
}
}
test "6: a 404 leaves the compiled files and the snapshot untouched" {
if (!build_options.integration) return error.SkipZigTest;
@@ -1034,7 +1047,13 @@ test "6: a 404 leaves the compiled files and the snapshot untouched" {
const failed = try env.status(id);
try testing.expectEqual(manager.State.fetch_failed, failed.state);
try testing.expectEqualStrings("HttpStatus", failed.errorText());
// The whole line, not just the taxonomy: a 404 must stay `HttpStatus` and
// name the status it saw. Routing it back through `fetcher.mapError` would
// read `ReceiveFailed (receive, cause ReceiveFailed, ...)` here.
try expectFailureText(
"HttpStatus (receive, cause HttpStatus, http 404, 0 bytes, ",
failed.errorText(),
);
var after = try Bodies.read(gpa, io, dir, "1");
defer after.deinit(gpa);
@@ -1046,6 +1065,51 @@ test "6: a 404 leaves the compiled files and the snapshot untouched" {
try testing.expect(decision.blocked);
}
test "6b: a local failure reports itself, not the previous source's network fault" {
if (!build_options.integration) return error.SkipZigTest;
// Mode bits do not apply to root, so the denial the test needs cannot happen.
if (std.c.geteuid() == 0) return error.SkipZigTest;
const gpa = testing.allocator;
const env = try Env.create(gpa);
defer env.destroy();
const io = env.io();
var fixture = try HttpFixture.init(io, http_body);
defer fixture.deinit(io);
fixture.setRoute(.not_found);
var group: std.Io.Group = .init;
defer group.cancel(io);
try group.concurrent(io, HttpFixture.serve, .{ &fixture, io });
var url_buf: [64]u8 = undefined;
const url = try fixture.url(&url_buf);
const id = try seedSource(&env.database, url);
// First a real network failure, so the fetcher is holding a record.
try testing.expect(!try refreshOnce(env, url));
try expectFailureText(
"HttpStatus (receive, cause HttpStatus, http 404, 0 bytes, ",
(try env.status(id)).errorText(),
);
// Then a failure that never reaches the network: the raw file cannot be
// created. Without the clear at the top of `download` this would report the
// 404 above as this refresh's cause.
var dir = try env.blocklistDir();
defer dir.close(io);
try dir.setPermissions(io, .fromMode(0o500));
defer dir.setPermissions(io, .fromMode(0o700)) catch {};
try testing.expect(!try refreshOnce(env, url));
const failed = try env.status(id);
try testing.expectEqual(manager.State.fetch_failed, failed.state);
try expectFailureText("AccessDenied (no fetch, http none, - bytes, ", failed.errorText());
try testing.expect(!std.mem.containsAtLeast(u8, failed.errorText(), 1, "HttpStatus"));
try testing.expect(!std.mem.containsAtLeast(u8, failed.errorText(), 1, "404"));
}
test "7: a body over the cap fails the refresh and leaves no temporary file" {
if (!build_options.integration) return error.SkipZigTest;
@@ -1069,7 +1133,10 @@ test "7: a body over the cap fails the refresh and leaves no temporary file" {
const failed = try env.status(id);
try testing.expectEqual(manager.State.fetch_failed, failed.state);
try testing.expectEqualStrings("BodyTooLarge", failed.errorText());
try expectFailureText(
"BodyTooLarge (receive, cause BodyTooLarge, http 200, 0 bytes, ",
failed.errorText(),
);
var dir = try env.blocklistDir();
defer dir.close(io);
+196 -15
View File
@@ -69,8 +69,13 @@ const log = std.log.scoped(.blocklist_manager);
const Sha256 = std.crypto.hash.sha2.Sha256;
/// `SourceStatus.last_error` is fixed-size so the failure path allocates
/// nothing.
pub const max_error_len: usize = 128;
/// nothing. Sized so the longest `downloadFailureText` fits whole: the widest
/// cause name the unwraps can produce is `DetectingNetworkConfigurationFailed`
/// at 35 bytes, which with a 64 MiB byte count and an hour of milliseconds
/// makes a 111-byte line. The test below recomputes that worst case from the
/// std error sets, so a wider name std adds fails the build's test run rather
/// than silently truncating an operator's only diagnostic.
pub const max_error_len: usize = 192;
/// `SourceStatus.url` is fixed-size so a copied status borrows nothing. A
/// blocklist url longer than this is truncated in the status only; the row
@@ -131,6 +136,55 @@ const SourceLabel = struct {
}
};
/// The one place the download failure text is formatted. `phase`, `cause` and
/// the byte count come from the fetcher's record of the concrete fault.
///
/// A timeout carries a record like any other failure: `fetchWithin` cancels the
/// fetch on its way out of the race, and the cancelled fetch records the phase
/// it was in, the status if a head had arrived and the bytes delivered so far,
/// all before this function reads it. So an expiry reads
/// `Timeout (receive_body, cause Canceled, http 200, 5242880 bytes, 300000 ms)`.
/// `no fetch` and `-` are for a failure that never reached the network at all:
/// `download` clears the record before anything can fail, so a local fault —
/// the temporary file, an allocation, a refused task — reads as its own
/// taxonomy name with nothing borrowed from the source fetched before it.
fn downloadFailureText(
buf: []u8,
err: anyerror,
failure: ?fetcher.Failure,
last_status: ?std.http.Status,
elapsed_ms: u64,
) []const u8 {
var status_buf: [8]u8 = undefined;
const status_text = if (if (failure) |f| f.status else last_status) |code|
std.fmt.bufPrint(&status_buf, "{d}", .{@intFromEnum(code)}) catch "none"
else
"none";
var bytes_buf: [24]u8 = undefined;
const bytes_text = if (failure) |f|
std.fmt.bufPrint(&bytes_buf, "{d}", .{f.bytes_read}) catch "-"
else
"-";
if (failure) |f| {
return std.fmt.bufPrint(buf, "{s} ({s}, cause {s}, http {s}, {s} bytes, {d} ms)", .{
@errorName(err),
@tagName(f.phase),
@errorName(f.cause),
status_text,
bytes_text,
elapsed_ms,
}) catch @errorName(err);
}
return std.fmt.bufPrint(buf, "{s} (no fetch, http {s}, {s} bytes, {d} ms)", .{
@errorName(err),
status_text,
bytes_text,
elapsed_ms,
}) catch @errorName(err);
}
pub const Paths = struct {
/// `<data_dir>`, owned by the caller and left open for the manager's life.
dir: std.Io.Dir,
@@ -1169,11 +1223,12 @@ pub const Manager = struct {
raw_name: []const u8,
tmp: TempNames,
) Error!Prepared {
self.download(io, dir, raw_name, row) catch |err| switch (err) {
var elapsed_ms: u64 = 0;
self.download(io, dir, raw_name, row, &elapsed_ms) catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
error.Canceled => return error.Canceled,
else => {
self.reportFetchFailure(row, status, err);
self.reportDownloadFailure(row, status, err, elapsed_ms);
return .failed;
},
};
@@ -1297,7 +1352,18 @@ pub const Manager = struct {
dir: std.Io.Dir,
raw_name: []const u8,
row: sources_repo.SourceRow,
/// Milliseconds the download itself took, set whether it succeeded or
/// failed, so the failure line can say how long the peer had.
elapsed_ms: *u64,
) !void {
// Ahead of everything that can fail. The record is the fetcher's, not
// this source's, and a local failure here — the create, the allocation,
// a refused task — would otherwise leave the PREVIOUS source's network
// cause in place for `reportDownloadFailure` to print as this one's.
// Cleared here, a null record means no fetch reached the network.
self.fetcher.last_failure = null;
self.fetcher.last_status = null;
const file = try dir.createFile(io, raw_name, .{ .permissions = .fromMode(0o600) });
defer file.close(io);
@@ -1305,16 +1371,14 @@ pub const Manager = struct {
defer self.gpa.free(buffer);
var fw = file.writer(io, buffer);
const result = self.fetchWithin(io, row.url, &fw.interface) catch |err| {
const result = self.fetchWithin(io, row.url, &fw.interface, elapsed_ms) catch |err| {
// `fetcher.Error.Unexpected` is what a failing sink surfaces as;
// the concrete cause is on this writer, which the fetcher does not
// own.
if (fw.err) |cause| return cause;
if (err == error.HttpStatus) {
if (self.fetcher.last_status) |status| {
log.warn("blocklist {f}: http status {d}", .{ SourceLabel.of(row), @intFromEnum(status) });
}
}
// The status used to get a second warning of its own. It is a field
// of the one download-failure line now, so a second line would only
// repeat it.
return err;
};
try fw.interface.flush();
@@ -1332,7 +1396,12 @@ pub const Manager = struct {
io: std.Io,
url: []const u8,
w: *std.Io.Writer,
elapsed_ms: *u64,
) fetcher.Error!fetcher.Result {
// `awake` and not `real`: an operator reading "812 ms" wants the time
// the transfer was given, which a stepped wall clock would misreport.
const started = std.Io.Clock.awake.now(io);
var outcomes: [2]Outcome = undefined;
var race: std.Io.Select(Outcome) = .init(io, &outcomes);
defer race.cancelDiscard();
@@ -1344,7 +1413,12 @@ pub const Manager = struct {
error.ConcurrencyUnavailable => return error.SystemResources,
};
switch (try race.await()) {
// Read before the deferred `cancelDiscard`, which tears the loser down:
// the number is the time the peer had, not that plus the teardown.
const outcome = race.await();
elapsed_ms.* = @intCast(@max(0, started.durationTo(std.Io.Clock.awake.now(io)).toMilliseconds()));
switch (try outcome) {
.fetch => |result| return result,
.expiry => |result| {
// A canceled sleep means this task is being torn down, not that
@@ -1514,15 +1588,21 @@ pub const Manager = struct {
return compiledBodiesMatch(list_bytes, wild_bytes, allow_bytes, expected);
}
fn reportFetchFailure(
/// One line per failed source per pass, and the only place the download
/// failure text is built. The Diagnostics event detail gets the same text,
/// so an operator reading either can tell a TLS alert from a reset
/// connection from a truncated chunk.
fn reportDownloadFailure(
self: *Manager,
row: sources_repo.SourceRow,
status: *SourceStatus,
err: anyerror,
elapsed_ms: u64,
) void {
_ = self;
log.warn("blocklist {f}: download failed: {s}", .{ SourceLabel.of(row), @errorName(err) });
status.fail(.fetch_failed, @errorName(err));
var buf: [max_error_len]u8 = undefined;
const text = downloadFailureText(&buf, err, self.fetcher.last_failure, self.fetcher.last_status, elapsed_ms);
log.warn("blocklist {f}: download failed: {s}", .{ SourceLabel.of(row), text });
status.fail(.fetch_failed, text);
}
fn reportCompileFailure(
@@ -3580,3 +3660,104 @@ test "setSchedule is what the live schedule readers see" {
try testing.expectEqual(@as(u16, 6), live.interval_hours);
try testing.expect(manager.schedule_event.isSet());
}
test "a download failure line names the concrete cause behind the taxonomy" {
var buf: [1024]u8 = undefined;
const row: sources_repo.SourceRow = .{
.id = 3,
.url = "https://lists.example/download/token/hunter2/hosts.txt?apikey=s3cr3t",
.name = "ads",
.enabled = true,
.last_updated = null,
.domain_count = 0,
.wildcard_count = 0,
.skipped_regex_count = 0,
.skipped_unsupported_count = 0,
.checksum = null,
};
var text_buf: [max_error_len]u8 = undefined;
const text = downloadFailureText(&text_buf, error.ReceiveFailed, .{
.phase = .receive_body,
.cause = error.HttpChunkTruncated,
.status = .ok,
.bytes_read = 1024 * 1024,
}, .ok, 812);
const printed = try std.fmt.bufPrint(&buf, "blocklist {f}: download failed: {s}", .{
SourceLabel.of(row),
text,
});
try testing.expectEqualStrings(
"blocklist source 3 'ads' 'https://lists.example': download failed:" ++
" ReceiveFailed (receive_body, cause HttpChunkTruncated, http 200, 1048576 bytes, 812 ms)",
printed,
);
try testing.expect(!std.mem.containsAtLeast(u8, printed, 1, "hunter2"));
try testing.expect(!std.mem.containsAtLeast(u8, printed, 1, "s3cr3t"));
}
test "an expiry names the phase and the progress the cancelled fetch had made" {
// `fetchWithin` cancels the fetch before it reports, so a timeout normally
// does have a record: the cancellation the fetch saw, under the manager's
// own `error.Timeout`.
var text_buf: [max_error_len]u8 = undefined;
try testing.expectEqualStrings(
"Timeout (receive_body, cause Canceled, http 200, 5242880 bytes, 300000 ms)",
downloadFailureText(&text_buf, error.Timeout, .{
.phase = .receive_body,
.cause = error.Canceled,
.status = .ok,
.bytes_read = 5 * 1024 * 1024,
}, .ok, 300000),
);
}
test "a failure that never reached the network says so and borrows nothing" {
// `download` clears the fetcher's record before anything can fail, so a
// local fault reads as itself. Printing a phase and a cause here would be
// printing the previous source's network fault against this source.
var text_buf: [max_error_len]u8 = undefined;
try testing.expectEqualStrings(
"SystemResources (no fetch, http none, - bytes, 0 ms)",
downloadFailureText(&text_buf, error.SystemResources, null, null, 0),
);
try testing.expectEqualStrings(
"AccessDenied (no fetch, http none, - bytes, 3 ms)",
downloadFailureText(&text_buf, error.AccessDenied, null, null, 3),
);
}
test "max_error_len holds the widest download failure line whole" {
// `@errorName` of the unwrapped cause is the only unbounded-looking part.
// The two sets below are every set the unwraps in `upstream/transport.zig`
// can return a member of.
const widest_cause = comptime blk: {
var widest: []const u8 = "";
for (@typeInfo(std.crypto.tls.Client.ReadError).error_set.?) |member| {
if (member.name.len > widest.len) widest = member.name;
}
for (@typeInfo(std.http.Client.RequestError).error_set.?) |member| {
if (member.name.len > widest.len) widest = member.name;
}
for (@typeInfo(std.http.Reader.BodyError).error_set.?) |member| {
if (member.name.len > widest.len) widest = member.name;
}
break :blk widest;
};
var buf: [max_error_len]u8 = undefined;
const text = std.fmt.bufPrint(
&buf,
"{s} ({s}, cause {s}, http {d}, {d} bytes, {d} ms)",
.{
"SystemResources",
@tagName(fetcher.Phase.receive_body),
widest_cause,
@as(u16, 599),
@as(u64, fetcher.max_body_bytes),
@as(u64, std.time.ms_per_hour),
},
) catch unreachable;
try testing.expect(text.len <= max_error_len);
try testing.expectEqualStrings("DetectingNetworkConfigurationFailed", widest_cause);
}
+6 -37
View File
@@ -226,43 +226,12 @@ const Connection = std.http.Client.Connection;
const Request = std.http.Client.Request;
const Response = std.http.Client.Response;
/// `std.Io.Writer` collapses every send failure to `error.WriteFailed` and
/// stashes the cause on the connection's socket writer. Unwrapping it is what
/// keeps `error.Canceled` and the local resource errors out of the peer fault
/// group, exactly as `concreteWrite` does for DoT.
fn sendCause(req: *const Request, err: anyerror) anyerror {
if (err != error.WriteFailed) return err;
const connection = req.connection orelse return err;
return connection.stream_writer.err orelse err;
}
/// `receiveHead` collapses a transport failure to `error.ReadFailed` and names
/// `Connection.getReadError` as the accessor for the concrete cause.
fn headCause(req: *const Request, err: anyerror) anyerror {
if (err != error.ReadFailed) return err;
const connection = req.connection orelse return err;
return readCause(connection, err);
}
/// A body read reports two different kinds of failure through the same
/// `error.ReadFailed`. An HTTP framing fault lands on the response, and only a
/// read that never reached the framing leaves the connection's cause, so the
/// response is consulted first.
fn bodyCause(resp: *const Response, err: anyerror) anyerror {
if (err != error.ReadFailed) return err;
if (resp.bodyErr()) |cause| return cause;
const connection = resp.request.connection orelse return err;
return readCause(connection, err);
}
/// `Connection.getReadError` reads the socket reader's stashed cause with `.?`.
/// On a plain connection that is its only source, so calling it with nothing
/// stashed would panic rather than return null; the guard keeps this unwrap
/// total on the path this client can reach without TLS.
fn readCause(connection: *const Connection, err: anyerror) anyerror {
if (connection.protocol == .plain and connection.stream_reader.err == null) return err;
return connection.getReadError() orelse err;
}
// The unwraps live in `transport.zig` because the blocklist fetcher needs the
// same concrete causes. They are aliased rather than qualified so the call
// sites and the tests below read as they did when they were local.
const sendCause = transport.sendCause;
const headCause = transport.headCause;
const bodyCause = transport.bodyCause;
/// RFC 8484 §6: the response media type is `application/dns-message`. The
/// header may carry parameters (`; charset=…`) and the type is case-insensitive
+79 -38
View File
@@ -773,6 +773,16 @@ const Fake = struct {
/// Sleep, then fail. Holds the entry's lock long enough for a second
/// task to queue on it before the failure opens a backoff window.
slow_fail: struct { duration: std.Io.Clock.Duration, err: transport.ExchangeError },
/// Wait for the gate, then reply. A test that needs two calls provably
/// in flight together cannot get that from a duration: a loaded
/// scheduler can start the second call after the first one's sleep
/// expired, and no poll can recover an overlap that already passed.
/// The gate makes the test, not the clock, decide when a call returns.
hold: struct { gate: *std.Io.Semaphore, reply: []const u8 },
/// Wait for the gate, then fail. The failing half of `hold`, with the
/// same constraint: the test releases the call once it has observed
/// whatever had to be true while the call was still in flight.
hold_fail: struct { gate: *std.Io.Semaphore, err: transport.ExchangeError },
};
fn exchangeFn(
@@ -787,11 +797,11 @@ const Fake = struct {
// entry, so a test can tell the two apart.
selected.* = "fake://leaf";
const self: *Fake = @ptrCast(@alignCast(ptr));
const entrants = self.in_flight.fetchAdd(1, .acq_rel) + 1;
defer _ = self.in_flight.fetchSub(1, .acq_rel);
_ = self.peak_in_flight.fetchMax(entrants, .acq_rel);
_ = self.calls.fetchAdd(1, .acq_rel);
// The behaviour is taken before `in_flight` rises: a test that waits
// on the count to know a call is inside this fake also needs to know
// which behaviour that call drew, or a second call could still take
// the first behaviour from under it.
const behavior = behavior: {
self.mutex.lockUncancelable(io);
defer self.mutex.unlock(io);
@@ -802,6 +812,9 @@ const Fake = struct {
}
break :behavior current;
};
const entrants = self.in_flight.fetchAdd(1, .acq_rel) + 1;
defer _ = self.in_flight.fetchSub(1, .acq_rel);
_ = self.peak_in_flight.fetchMax(entrants, .acq_rel);
switch (behavior) {
.reply => |bytes| return copy(bytes, response_buf),
.fail => |err| return err,
@@ -813,6 +826,14 @@ const Fake = struct {
try slow.duration.sleep(io);
return slow.err;
},
.hold => |held| {
try held.gate.wait(io);
return copy(held.reply, response_buf);
},
.hold_fail => |held| {
try held.gate.wait(io);
return held.err;
},
}
}
@@ -1930,11 +1951,9 @@ test "a waiter that spends its whole budget queueing blames nobody but the budge
try testing.expectEqual(@as(usize, 2), fake.calls.load(.acquire));
}
/// The saturated-primary test's numbers. The holders stall longer than the
/// probe's whole budget, so the probe provably cannot be waiting for one of
/// their slots; the stall still ends inside the test so the holders' futures
/// can be awaited rather than abandoned.
const saturating_stall_ms = 800;
/// The saturated-primary test's budgets. The holders do not stall on a clock at
/// all: they are held on a gate the test releases, so the probe's own budget is
/// the only duration the test depends on.
const probe_attempt_ms = 150;
const probe_total_ms = 300;
@@ -1947,10 +1966,11 @@ test "a saturated primary defers to a standby that has capacity" {
// slow exchange, and the standby is idle. Priority orders the entries that
// can be admitted now; it does not entitle a saturated entry to hold a
// query until the whole budget is gone.
var primary: Fake = .{ .behavior = .{ .slow = .{
.duration = .{ .raw = .fromMilliseconds(saturating_stall_ms), .clock = .awake },
.reply = response_bytes,
} } };
// The holders are released by the test, not by a clock: a fixed stall can
// expire before a loaded scheduler lets the poller observe both calls, and
// an overlap that already passed cannot be recovered by polling.
var gate: std.Io.Semaphore = .{};
var primary: Fake = .{ .behavior = .{ .hold = .{ .gate = &gate, .reply = response_bytes } } };
var standby: Fake = .{ .behavior = .{ .reply = alt_response_bytes } };
var entries = [_]Entry{
testEntrySlots("https://primary.example/dns-query", &primary, 10, 2),
@@ -1968,14 +1988,20 @@ test "a saturated primary defers to a standby that has capacity" {
var bufs: [2][512]u8 = undefined;
var holders: [2]std.Io.Future(transport.ExchangeError!Attributed) = undefined;
var spawned: usize = 0;
// Every spawned holder is released and awaited on any exit, including the
// skip when the second spawn is refused: a holder left blocked on the gate
// would outlive the stack it reads.
defer for (holders[0..spawned]) |*holder| {
_ = holder.await(io) catch Attributed.discarded;
};
defer for (0..spawned) |_| gate.post(io);
for (&holders, &bufs) |*holder, *buf| {
holder.* = io.concurrent(exchangeAttributed, .{ &holder_pool, io, buf }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
spawned += 1;
}
defer for (&holders) |*holder| {
_ = holder.await(io) catch Attributed.discarded;
};
// Both permits provably taken before the probe asks for one.
try awaitInFlight(io, &primary, 2);
@@ -2018,27 +2044,21 @@ test "overlapping exchanges each report the entry that answered that call" {
defer threaded.deinit();
const io = threaded.io();
// Two gates, one per call the first entry takes. Nothing in this test
// sleeps: the test releases each call itself, so the schedule it asserts
// on is the schedule it built.
var gate_a: std.Io.Semaphore = .{};
var gate_b: std.Io.Semaphore = .{};
// The first entry has room for both tasks and treats them differently: it
// fails whichever reaches it first and answers the other, slowly enough
// that the two calls are in flight together. One failure is under
// `test_cfg`'s threshold of two, so no backoff steers the failed-over task
// away and the two calls end on different entries.
// fails whichever reaches it first and answers the other. One failure is
// under `test_cfg`'s threshold of two, so no backoff steers the failed-over
// task away and the two calls end on different entries.
var first_entry: Fake = .{
.behavior = .{ .slow_fail = .{
.duration = .{ .raw = .fromMilliseconds(50), .clock = .awake },
.err = error.ConnectFailed,
} },
.then = .{ .slow = .{
.duration = .{ .raw = .fromMilliseconds(100), .clock = .awake },
.reply = alt_response_bytes,
} },
.behavior = .{ .hold_fail = .{ .gate = &gate_a, .err = error.ConnectFailed } },
.then = .{ .hold = .{ .gate = &gate_b, .reply = alt_response_bytes } },
};
// Slow too, so the failed-over call is still in flight while the other call
// is being answered — a shared identity would be overwritten under it.
var second_entry: Fake = .{ .behavior = .{ .slow = .{
.duration = .{ .raw = .fromMilliseconds(50), .clock = .awake },
.reply = response_bytes,
} } };
var second_entry: Fake = .{ .behavior = .{ .reply = response_bytes } };
var entries = [_]Entry{
testEntrySlots(divergent_first_url, &first_entry, 10, 2),
testEntry(divergent_second_url, &second_entry, 20),
@@ -2052,17 +2072,37 @@ test "overlapping exchanges each report the entry that answered that call" {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = first.await(io) catch Attributed.discarded;
// Registered right after the task and after its await, so it runs before
// the await: any early return below (a failed wait, a skipped second
// task, a failed assertion) releases the held call first and the await
// then returns, instead of the test deadlocking on a gate nobody posted.
// Extra permits on an already-released gate are harmless.
defer gate_a.post(io);
// A count of one means the first call is inside the entry with its
// behaviour already drawn, so it holds gate A and the second call will draw
// gate B.
try awaitInFlight(io, &first_entry, 1);
var second = io.concurrent(exchangeAttributed, .{ &pool, io, &buf_b }) catch |err| switch (err) {
error.ConcurrencyUnavailable => return error.SkipZigTest,
};
defer _ = second.await(io) catch Attributed.discarded;
defer gate_b.post(io);
// Both tasks are now inside the first entry, the first on `hold_fail` and
// the second on `hold`. That is the overlap every assertion below rests
// on, and it is established here rather than assumed.
try awaitInFlight(io, &first_entry, 2);
// Release the first call only. It fails, fails over to the second entry
// and is answered there while the second call is still held inside the
// first entry, so a shared identity would be overwritten under it.
gate_a.post(io);
const result_a = try first.await(io);
gate_b.post(io);
const result_b = try second.await(io);
// Which task lands where depends on the order they take the first entry's
// lock, so the claim is over the pair: one identity each, and each one
// matching the bytes that call received.
// The first task reached the first entry first, but the claim stays over
// the pair: one identity each, and each one matching the bytes that call
// received.
try testing.expect(!std.mem.eql(u8, result_a.selected.?, result_b.selected.?));
try expectAnsweredByReporter(result_a, &buf_a);
try expectAnsweredByReporter(result_b, &buf_b);
@@ -2070,7 +2110,8 @@ test "overlapping exchanges each report the entry that answered that call" {
try testing.expectEqual(@as(usize, 2), first_entry.calls.load(.acquire));
try testing.expectEqual(@as(usize, 1), second_entry.calls.load(.acquire));
// Both tasks were admitted to the first entry at once, which is what makes
// them concurrent rather than serialized behind one permit.
// them concurrent rather than serialized behind one permit. The wait above
// already required this; the counter records that it held to the end.
try testing.expectEqual(@as(u32, 2), first_entry.peak_in_flight.load(.acquire));
try testing.expectEqual(@as(u64, 1), entries[0].health.total_failures);
try testing.expectEqual(@as(u64, 1), entries[0].health.total_successes);
+46
View File
@@ -419,6 +419,52 @@ pub const Client = struct {
}
};
/// The unwrap helpers below turn the single collapsed error `std.http.Client`
/// reports into the concrete cause it stashed. Every HTTP caller in this tree
/// uses them: the DoH client classifies by the unwrapped cause, the blocklist
/// fetcher keeps its own classification and records the cause for the operator.
const Connection = std.http.Client.Connection;
const Request = std.http.Client.Request;
const Response = std.http.Client.Response;
/// `std.Io.Writer` collapses every send failure to `error.WriteFailed` and
/// stashes the cause on the connection's socket writer. Unwrapping it is what
/// keeps `error.Canceled` and the local resource errors out of the peer fault
/// group.
pub fn sendCause(req: *const Request, err: anyerror) anyerror {
if (err != error.WriteFailed) return err;
const connection = req.connection orelse return err;
return connection.stream_writer.err orelse err;
}
/// `receiveHead` collapses a transport failure to `error.ReadFailed` and names
/// `Connection.getReadError` as the accessor for the concrete cause.
pub fn headCause(req: *const Request, err: anyerror) anyerror {
if (err != error.ReadFailed) return err;
const connection = req.connection orelse return err;
return readCause(connection, err);
}
/// A body read reports two different kinds of failure through the same
/// `error.ReadFailed`. An HTTP framing fault lands on the response, and only a
/// read that never reached the framing leaves the connection's cause, so the
/// response is consulted first.
pub fn bodyCause(resp: *const Response, err: anyerror) anyerror {
if (err != error.ReadFailed) return err;
if (resp.bodyErr()) |cause| return cause;
const connection = resp.request.connection orelse return err;
return readCause(connection, err);
}
/// `Connection.getReadError` reads the socket reader's stashed cause with `.?`.
/// On a plain connection that is its only source, so calling it with nothing
/// stashed would panic rather than return null; the guard keeps this unwrap
/// total on the path a plain connection can reach without TLS.
pub fn readCause(connection: *const Connection, err: anyerror) anyerror {
if (connection.protocol == .plain and connection.stream_reader.err == null) return err;
return connection.getReadError() orelse err;
}
pub const ValidateError = error{ BadResponse, ResponseMismatch };
/// RFC 9619: exactly one question on both sides. RFC 4343: names compare
-1
View File
@@ -1,5 +1,4 @@
const build_options = @import("build_options");
pub const string: []const u8 = build_options.version_string;
pub const git_commit: []const u8 = build_options.git_commit;
pub const zig_version_string: []const u8 = build_options.zig_version_string;
-4
View File
@@ -11,7 +11,6 @@ const version = @import("../../version.zig");
pub const Body = struct {
version: []const u8,
git_commit: []const u8,
zig_version: []const u8,
/// Seconds since the process started. Zero until `started_unix` is wired,
/// and never negative: a clock stepped backwards must not report a
@@ -31,7 +30,6 @@ pub fn handle(
pub fn body(version_string: []const u8, started_unix: i64, now_unix: i64) Body {
return .{
.version = if (version_string.len == 0) version.string else version_string,
.git_commit = version.git_commit,
.zig_version = version.zig_version_string,
.uptime_seconds = uptime(started_unix, now_unix),
};
@@ -47,7 +45,6 @@ const testing = std.testing;
test "the body carries the build strings and the elapsed time" {
const out = body("", 1_000, 1_060);
try testing.expectEqualStrings(version.string, out.version);
try testing.expectEqualStrings(version.git_commit, out.git_commit);
try testing.expectEqualStrings(version.zig_version_string, out.zig_version);
try testing.expectEqual(@as(u64, 60), out.uptime_seconds);
}
@@ -66,7 +63,6 @@ test "the body serializes with snake_case field names" {
var writer: std.Io.Writer = .fixed(&buffer);
try std.json.Stringify.value(body("1.2.3", 10, 20), .{}, &writer);
const text = writer.buffered();
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"git_commit\":"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"zig_version\":"));
try testing.expect(std.mem.containsAtLeast(u8, text, 1, "\"uptime_seconds\":10"));
}
+1 -2
View File
@@ -1913,10 +1913,9 @@ components:
Version:
type: object
required: [version, git_commit, zig_version, uptime_seconds]
required: [version, zig_version, uptime_seconds]
properties:
version: { type: string }
git_commit: { type: string }
zig_version: { type: string }
uptime_seconds: { type: integer }
+4 -5
View File
@@ -4223,11 +4223,10 @@ const rate_limited_sample: ContractSample = .{
const ts_print_width = 120;
const ts_tab_width = 4;
/// Build identity, not contract data: `git_commit` comes from `-Dgit-commit`
/// and `zig_version` from the compiler that built the test, so keeping either
/// verbatim would pin the golden to one machine. Neither name occurs anywhere
/// else in the contract.
const volatile_string_keys = [_][]const u8{ "git_commit", "zig_version" };
/// Build identity, not contract data: `zig_version` comes from the compiler
/// that built the test, so keeping it verbatim would pin the golden to one
/// machine. The name occurs nowhere else in the contract.
const volatile_string_keys = [_][]const u8{"zig_version"};
fn writeTabs(w: *std.Io.Writer, depth: usize) !void {
for (0..depth) |_| try w.writeByte('\t');
+1639 -104
View File
File diff suppressed because it is too large Load Diff
+559 -3
View File
@@ -1,6 +1,6 @@
//! Release payload staging for `zig build dist` (milestone-14 rulings 3 and 4).
//!
//! Two modes, both writing only into paths the build system handed them:
//! Five modes, all writing only into paths the build system handed them:
//!
//! dist_stage stage --out <dir> --binary <path> --service <path>
//! --sysusers <path> --license <path> --install-md <path>
@@ -12,6 +12,27 @@
//! scraped from the dependency tree, because a generated notices file that
//! nobody reads rots silently into a false statement.
//!
//! dist_stage archive --root <dir> --payload <name> --out <file.tar.gz>
//!
//! Writes the release tarball itself, so that the bytes depend on the
//! staged tree and nothing else. The host's `tar` and `gzip` cannot supply
//! that: their member order, their padding and their gzip header all vary
//! with the implementation, the locale and the clock.
//!
//! dist_stage pin --sums <SHA256SUMS> --flake <flake.nix> --version <x.y.z>
//! dist_stage pin-check --sums <SHA256SUMS> --flake <flake.nix> --version <x.y.z>
//!
//! Rewrite, or assert, the generated block of `flake.nix` from the sums
//! file. `pin` backs `zig build pin-flake`, which the cut runs before its
//! bump commit; `pin-check` backs `zig build verify-pins`, which CI runs on
//! that commit and again on the tag, so a release whose pinned hashes do not
//! describe its own bytes is refused before anything is uploaded.
//!
//! `verify-pins` is a step of its own and NOT part of `verify-dist`: an
//! ordinary commit between two cuts builds the same `build.zig.zon` version
//! from a different tree, so its bytes never match the pins and checking
//! them there would fail every such build.
//!
//! dist_stage sums --out <file> [--entry <name> <path>]...
//!
//! Writes `sha256sum`-format lines, one per `--entry`, in argument order.
@@ -55,11 +76,17 @@ pub fn main(init: std.process.Init) !void {
const arena = init.arena.allocator();
const io = init.io;
const args = try init.minimal.args.toSlice(arena);
if (args.len < 2) std.process.fatal("usage: dist_stage <stage|sums> ...", .{});
if (args.len < 2) std.process.fatal("usage: dist_stage <stage|sums|archive|pin|pin-check> ...", .{});
if (std.mem.eql(u8, args[1], "stage")) return stage(arena, io, args[2..]);
if (std.mem.eql(u8, args[1], "sums")) return sums(arena, io, args[2..]);
std.process.fatal("unknown mode '{s}': expected `stage` or `sums`", .{args[1]});
if (std.mem.eql(u8, args[1], "archive")) return archive(arena, io, args[2..]);
if (std.mem.eql(u8, args[1], "pin")) return pin(arena, io, args[2..], .rewrite);
if (std.mem.eql(u8, args[1], "pin-check")) return pin(arena, io, args[2..], .check);
std.process.fatal(
"unknown mode '{s}': expected `stage`, `sums`, `archive`, `pin` or `pin-check`",
.{args[1]},
);
}
fn stage(arena: Allocator, io: Io, args: []const []const u8) !void {
@@ -255,3 +282,532 @@ fn parseInventory(arena: Allocator, source: [:0]const u8) ![]const Component {
error.ParseZon => std.process.fatal("licenses/inventory.zon:\n{f}", .{&diagnostics}),
};
}
// --- archive -----------------------------------------------------------------
const binary_name = "nxdns";
/// gzip level 9, the level the release archives were produced at before this
/// tool owned them. The level is part of the output bytes, so it is fixed here
/// rather than read from anywhere.
const compression: std.compress.flate.Compress.Options = .level_9;
const ArchiveEntry = struct {
/// Path inside the tarball, payload directory included.
path: []const u8,
kind: std.Io.File.Kind,
fn lessThan(_: void, a: ArchiveEntry, b: ArchiveEntry) bool {
return std.mem.order(u8, a.path, b.path) == .lt;
}
};
fn archive(arena: Allocator, io: Io, args: []const []const u8) !void {
var root_path: ?[]const u8 = null;
var payload_name: ?[]const u8 = null;
var out_path: ?[]const u8 = null;
var i: usize = 0;
while (i < args.len) : (i += 2) {
if (i + 1 >= args.len) std.process.fatal("'{s}' needs a value", .{args[i]});
const value = args[i + 1];
if (std.mem.eql(u8, args[i], "--root")) {
root_path = value;
} else if (std.mem.eql(u8, args[i], "--payload")) {
payload_name = value;
} else if (std.mem.eql(u8, args[i], "--out")) {
out_path = value;
} else {
std.process.fatal("unknown flag '{s}'", .{args[i]});
}
}
const root = required(root_path, "--root");
const name = required(payload_name, "--payload");
const out = required(out_path, "--out");
const bytes = buildArchive(arena, io, root, name) catch |err| {
std.process.fatal("cannot archive '{s}/{s}': {t}", .{ root, name, err });
};
try writeFileWithMode(io, Io.Dir.cwd(), out, bytes, 0o644);
}
/// The tarball bytes for `<root>/<payload>`. Every field a tar member carries
/// is fixed here — order, mode, mtime, uid, gid, user and group names — so two
/// runs over the same file contents produce the same bytes whatever the
/// absolute path, the umask, the clock or the locale is.
fn buildArchive(arena: Allocator, io: Io, root: []const u8, name: []const u8) ![]const u8 {
const payload_path = try std.fs.path.join(arena, &.{ root, name });
var payload_dir = Io.Dir.cwd().openDir(io, payload_path, .{ .iterate = true }) catch |err| {
std.process.fatal("cannot open the payload directory '{s}': {t}", .{ payload_path, err });
};
defer payload_dir.close(io);
var entries: std.ArrayList(ArchiveEntry) = .empty;
var walker = try payload_dir.walk(arena);
defer walker.deinit();
while (walker.next(io) catch |err| {
std.process.fatal("cannot walk '{s}': {t}", .{ payload_path, err });
}) |entry| {
switch (entry.kind) {
.file, .directory => {},
// A payload that grew a symlink or a device node would ship
// something the verifier rejects; say so here instead.
else => std.process.fatal(
"'{s}/{s}' is a {t}: the release payload holds only files and directories",
.{ payload_path, entry.path, entry.kind },
),
}
try entries.append(arena, .{
.path = try std.fs.path.join(arena, &.{ name, entry.path }),
.kind = entry.kind,
});
}
std.mem.sort(ArchiveEntry, entries.items, {}, ArchiveEntry.lessThan);
var tar_bytes: Io.Writer.Allocating = try .initCapacity(arena, 1 << 20);
var tar_writer: std.tar.Writer = .{ .underlying_writer = &tar_bytes.writer };
try tar_writer.writeDir(name, .{ .mode = payload_dir_mode, .mtime = 0 });
for (entries.items) |entry| {
switch (entry.kind) {
.directory => try tar_writer.writeDir(entry.path, .{ .mode = payload_dir_mode, .mtime = 0 }),
.file => {
const relative = entry.path[name.len + 1 ..];
const bytes = payload_dir.readFileAlloc(io, relative, arena, .limited(max_input_bytes)) catch |err| {
std.process.fatal("cannot read '{s}/{s}': {t}", .{ payload_path, relative, err });
};
const mode: std.posix.mode_t = if (std.mem.eql(u8, relative, binary_name)) 0o755 else 0o644;
try tar_writer.writeFileBytes(entry.path, bytes, .{ .mode = mode, .mtime = 0 });
},
else => unreachable,
}
}
try tar_writer.finishPedantically();
var gz_bytes: Io.Writer.Allocating = try .initCapacity(arena, 1 << 20);
const window = try arena.alloc(u8, std.compress.flate.max_window_len);
var compress: std.compress.flate.Compress = try .init(&gz_bytes.writer, window, .gzip, compression);
try compress.writer.writeAll(tar_bytes.written());
try compress.finish();
return gz_bytes.written();
}
// --- pin / pin-check ---------------------------------------------------------
/// The two release systems, and the target triple each one's tarball carries.
/// Nix names the system, the release names the triple; the block is written in
/// this order.
const pin_targets = [_]struct { system: []const u8, triple: []const u8 }{
.{ .system = "aarch64-linux", .triple = "aarch64-linux-musl" },
.{ .system = "x86_64-linux", .triple = "x86_64-linux-musl" },
};
const begin_marker = "# BEGIN GENERATED BY zig build cut";
const end_marker = "# END GENERATED BY zig build cut";
const PinMode = enum { rewrite, check };
const digest_length = std.crypto.hash.sha2.Sha256.digest_length;
fn pin(arena: Allocator, io: Io, args: []const []const u8, mode: PinMode) !void {
var sums_path: ?[]const u8 = null;
var flake_path: ?[]const u8 = null;
var version: ?[]const u8 = null;
var i: usize = 0;
while (i < args.len) : (i += 2) {
if (i + 1 >= args.len) std.process.fatal("'{s}' needs a value", .{args[i]});
const value = args[i + 1];
if (std.mem.eql(u8, args[i], "--sums")) {
sums_path = value;
} else if (std.mem.eql(u8, args[i], "--flake")) {
flake_path = value;
} else if (std.mem.eql(u8, args[i], "--version")) {
version = value;
} else {
std.process.fatal("unknown flag '{s}'", .{args[i]});
}
}
const sums_file = required(sums_path, "--sums");
const flake_file = required(flake_path, "--flake");
const version_string = required(version, "--version");
const sums_text = Io.Dir.cwd().readFileAlloc(io, sums_file, arena, .limited(max_input_bytes)) catch |err| {
std.process.fatal("cannot read '{s}': {t}", .{ sums_file, err });
};
const flake_text = Io.Dir.cwd().readFileAlloc(io, flake_file, arena, .limited(max_input_bytes)) catch |err| {
std.process.fatal("cannot read '{s}': {t}", .{ flake_file, err });
};
var expected: [pin_targets.len][]const u8 = undefined;
for (pin_targets, &expected) |target, *slot| {
const asset = try std.fmt.allocPrint(arena, "nxdns-{s}-{s}.tar.gz", .{ version_string, target.triple });
slot.* = hashFromSums(sums_text, asset) orelse
std.process.fatal("'{s}' has no line for '{s}'", .{ sums_file, asset });
}
switch (mode) {
.rewrite => {
const rewritten = try rewriteBlock(arena, flake_text, flake_file, version_string, expected);
try writeFileWithMode(io, Io.Dir.cwd(), flake_file, rewritten, 0o644);
},
.check => {
const block = try readBlock(arena, flake_text, flake_file);
var failed = false;
if (!std.mem.eql(u8, block.version, version_string)) {
std.debug.print(
"{s}: the generated block pins version {s}, the release is {s}\n",
.{ flake_file, block.version, version_string },
);
failed = true;
}
for (pin_targets, block.hashes, expected) |target, found, want| {
const want_sri = try sriFromHex(arena, want);
if (!std.mem.eql(u8, found, want_sri)) {
std.debug.print(
"{s}: {s} is pinned to {s}, the release tarball hashes to {s}\n",
.{ flake_file, target.system, found, want_sri },
);
failed = true;
}
}
if (failed) std.process.fatal(
"flake.nix does not describe this release; run `zig build cut`, which pins before it commits",
.{},
);
},
}
}
/// The hex digest `sha256sum` prints for `asset`, or null when the file has no
/// such line. Lines are `<hex> <filename>`.
fn hashFromSums(text: []const u8, asset: []const u8) ?[]const u8 {
var lines = std.mem.splitScalar(u8, text, '\n');
while (lines.next()) |line| {
const separator = std.mem.indexOf(u8, line, " ") orelse continue;
if (!std.mem.eql(u8, std.mem.trimEnd(u8, line[separator + 2 ..], "\r"), asset)) continue;
return line[0..separator];
}
return null;
}
fn sriFromHex(arena: Allocator, hex: []const u8) ![]const u8 {
if (hex.len != digest_length * 2) std.process.fatal("'{s}' is not a sha256 hex digest", .{hex});
var raw: [digest_length]u8 = undefined;
_ = std.fmt.hexToBytes(&raw, hex) catch std.process.fatal("'{s}' is not a sha256 hex digest", .{hex});
const encoder = std.base64.standard.Encoder;
const out = try arena.alloc(u8, "sha256-".len + encoder.calcSize(raw.len));
@memcpy(out[0.."sha256-".len], "sha256-");
_ = encoder.encode(out["sha256-".len..], &raw);
return out;
}
const Block = struct {
version: []const u8,
hashes: [pin_targets.len][]const u8,
};
/// Byte offsets of the generated block in `text`, marker lines included.
const BlockSpan = struct {
start: usize,
end: usize,
indent: []const u8,
};
/// Why `locateBlock` refused. The generated block is rewritten in place, so a
/// file that does not delimit exactly one block has no unambiguous region to
/// rewrite and every reading of it is a guess.
const BlockError = error{
NotOneBegin,
NotOneEnd,
EndBeforeBegin,
};
fn locateBlock(text: []const u8) BlockError!BlockSpan {
var begins: usize = 0;
var ends: usize = 0;
var offset: usize = 0;
var begin_at: usize = 0;
var end_at: usize = 0;
var indent: []const u8 = "";
while (offset < text.len) {
const line_end = std.mem.indexOfScalarPos(u8, text, offset, '\n') orelse text.len;
const line = text[offset..line_end];
const trimmed = std.mem.trimStart(u8, line, " \t");
if (std.mem.eql(u8, trimmed, begin_marker)) {
begins += 1;
begin_at = offset;
indent = line[0 .. line.len - trimmed.len];
} else if (std.mem.eql(u8, trimmed, end_marker)) {
ends += 1;
end_at = @min(line_end + 1, text.len);
}
offset = line_end + 1;
}
if (begins != 1) return error.NotOneBegin;
if (ends != 1) return error.NotOneEnd;
if (end_at <= begin_at) return error.EndBeforeBegin;
return .{ .start = begin_at, .end = end_at, .indent = indent };
}
fn findBlock(text: []const u8, path: []const u8) BlockSpan {
return locateBlock(text) catch |err| switch (err) {
error.NotOneBegin => std.process.fatal(
"{s}: expected exactly one '{s}' line",
.{ path, begin_marker },
),
error.NotOneEnd => std.process.fatal(
"{s}: expected exactly one '{s}' line",
.{ path, end_marker },
),
error.EndBeforeBegin => std.process.fatal(
"{s}: the '{s}' line comes before the '{s}' line",
.{ path, end_marker, begin_marker },
),
};
}
fn readBlock(arena: Allocator, text: []const u8, path: []const u8) !Block {
const span = findBlock(text, path);
var block: Block = .{ .version = "", .hashes = @splat("") };
var lines = std.mem.splitScalar(u8, text[span.start..span.end], '\n');
while (lines.next()) |line| {
const trimmed = std.mem.trim(u8, line, " \t");
if (std.mem.startsWith(u8, trimmed, "version = ")) {
block.version = try quoted(arena, trimmed, path);
}
for (pin_targets, &block.hashes) |target, *slot| {
const prefix = try std.fmt.allocPrint(arena, "\"{s}\" = ", .{target.system});
if (std.mem.startsWith(u8, trimmed, prefix)) {
slot.* = try quoted(arena, trimmed[prefix.len..], path);
}
}
}
if (block.version.len == 0) std.process.fatal("{s}: the generated block has no `version` line", .{path});
for (pin_targets, block.hashes) |target, hash| {
if (hash.len == 0) std.process.fatal(
"{s}: the generated block has no hash for {s}",
.{ path, target.system },
);
}
return block;
}
/// The contents of the last double-quoted string on `line`.
fn quoted(arena: Allocator, line: []const u8, path: []const u8) ![]const u8 {
const open = std.mem.indexOfScalar(u8, line, '"') orelse
std.process.fatal("{s}: '{s}' has no quoted value", .{ path, line });
const close = std.mem.indexOfScalarPos(u8, line, open + 1, '"') orelse
std.process.fatal("{s}: '{s}' has no closing quote", .{ path, line });
return arena.dupe(u8, line[open + 1 .. close]);
}
fn rewriteBlock(
arena: Allocator,
text: []const u8,
path: []const u8,
version: []const u8,
hex: [pin_targets.len][]const u8,
) ![]const u8 {
const span = findBlock(text, path);
var out: Io.Writer.Allocating = try .initCapacity(arena, text.len + 512);
const w = &out.writer;
try w.writeAll(text[0..span.start]);
try w.print("{s}{s}\n", .{ span.indent, begin_marker });
try w.print("{s}version = \"{s}\";\n", .{ span.indent, version });
try w.print("{s}hashes = {{\n", .{span.indent});
for (pin_targets, hex) |target, digest| {
try w.print("{s} \"{s}\" = \"{s}\";\n", .{ span.indent, target.system, try sriFromHex(arena, digest) });
}
try w.print("{s}}};\n", .{span.indent});
try w.print("{s}{s}\n", .{ span.indent, end_marker });
try w.writeAll(text[span.end..]);
return out.written();
}
// --- tests -------------------------------------------------------------------
const testing = std.testing;
/// The release payload, in the byte order the archive must list it in.
const fixture = [_]struct { name: []const u8, contents: []const u8, mode: std.posix.mode_t }{
.{ .name = "INSTALL.md", .contents = "# install\n", .mode = 0o644 },
.{ .name = "LICENSE", .contents = "EUPL\n", .mode = 0o644 },
.{ .name = "THIRD-PARTY-NOTICES", .contents = "notices\n", .mode = 0o644 },
.{ .name = binary_name, .contents = "\x7fELF not really", .mode = 0o755 },
.{ .name = "nxdns.conf", .contents = "u nxdns\n", .mode = 0o644 },
.{ .name = "nxdns.service", .contents = "[Unit]\n", .mode = 0o644 },
};
/// Stages the fixture under `<root>/<name>`, writing the files in `order` and
/// stamping each one with `mtime` so the two staged trees differ in everything
/// but their contents.
fn stageFixture(
io: Io,
root: Io.Dir,
name: []const u8,
order: []const usize,
mtime: i96,
) !void {
try root.createDir(io, name, .fromMode(payload_dir_mode));
var dir = try root.openDir(io, name, .{ .iterate = true });
defer dir.close(io);
for (order) |index| {
const file = fixture[index];
var handle = try dir.createFile(io, file.name, .{});
defer handle.close(io);
try handle.writeStreamingAll(io, file.contents);
try handle.setPermissions(io, .fromMode(file.mode));
try handle.setTimestamps(io, .{
.access_timestamp = .{ .new = .{ .nanoseconds = mtime } },
.modify_timestamp = .{ .new = .{ .nanoseconds = mtime } },
});
}
}
test "archive bytes do not depend on the stage path, the file order or the mtimes" {
const io = testing.io;
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
var first_tmp = testing.tmpDir(.{ .iterate = true });
defer first_tmp.cleanup();
var second_tmp = testing.tmpDir(.{ .iterate = true });
defer second_tmp.cleanup();
const name = "nxdns-0.0.16-x86_64-linux-musl";
try stageFixture(io, first_tmp.dir, name, &.{ 0, 1, 2, 3, 4, 5 }, 0);
try stageFixture(io, second_tmp.dir, name, &.{ 5, 3, 1, 4, 2, 0 }, 1_700_000_000 * std.time.ns_per_s);
const first_root = try std.fmt.allocPrint(arena, ".zig-cache/tmp/{s}", .{first_tmp.sub_path});
const second_root = try std.fmt.allocPrint(arena, ".zig-cache/tmp/{s}", .{second_tmp.sub_path});
const first = try buildArchive(arena, io, first_root, name);
const second = try buildArchive(arena, io, second_root, name);
try testing.expectEqualSlices(u8, first, second);
// The gzip header carries no name and no mtime: bytes 4..8 are the mtime
// field of RFC 1952, and bit 3 of the flag byte would announce a name.
try testing.expectEqual(@as(u8, 0x1f), first[0]);
try testing.expectEqual(@as(u8, 0x8b), first[1]);
try testing.expectEqual(@as(u8, 0), first[3] & 0b0000_1000);
try testing.expectEqualSlices(u8, &.{ 0, 0, 0, 0 }, first[4..8]);
var gz_reader: Io.Reader = .fixed(first);
const window = try arena.alloc(u8, std.compress.flate.max_window_len);
var decompress: std.compress.flate.Decompress = .init(&gz_reader, .gzip, window);
var it: std.tar.Iterator = .init(&decompress.reader, .{
.file_name_buffer = try arena.alloc(u8, std.fs.max_path_bytes),
.link_name_buffer = try arena.alloc(u8, std.fs.max_path_bytes),
});
const root_entry = (try it.next()).?;
try testing.expectEqual(std.tar.FileKind.directory, root_entry.kind);
try testing.expectEqualStrings(name, std.mem.trimEnd(u8, root_entry.name, "/"));
try testing.expectEqual(@as(u32, payload_dir_mode), root_entry.mode & 0o7777);
for (fixture) |file| {
const entry = (try it.next()).?;
try testing.expectEqual(std.tar.FileKind.file, entry.kind);
const expected_name = try std.fmt.allocPrint(arena, "{s}/{s}", .{ name, file.name });
try testing.expectEqualStrings(expected_name, entry.name);
try testing.expectEqual(@as(u32, @intCast(file.mode)), entry.mode & 0o7777);
try testing.expectEqual(@as(u64, file.contents.len), entry.size);
var sink: Io.Writer.Allocating = .init(arena);
try it.streamRemaining(entry, &sink.writer);
try testing.expectEqualStrings(file.contents, sink.written());
}
try testing.expect((try it.next()) == null);
}
test "sriFromHex encodes the raw digest, not its hex text" {
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
const zeros = "00" ** digest_length;
try testing.expectEqualStrings(
"sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=",
try sriFromHex(arena, zeros),
);
}
test "hashFromSums picks the line for the named asset" {
const text =
"1111111111111111111111111111111111111111111111111111111111111111 nxdns-0.0.16-aarch64-linux-musl.tar.gz\n" ++
"2222222222222222222222222222222222222222222222222222222222222222 nxdns-0.0.16-x86_64-linux-musl.tar.gz\n";
try testing.expectEqualStrings(
"2222222222222222222222222222222222222222222222222222222222222222",
hashFromSums(text, "nxdns-0.0.16-x86_64-linux-musl.tar.gz").?,
);
try testing.expectEqual(@as(?[]const u8, null), hashFromSums(text, "nxdns-0.0.16-riscv64-linux-musl.tar.gz"));
}
test "rewriteBlock keeps the begin line's indentation and the rest of the file" {
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
const before =
"let\n" ++
" " ++ begin_marker ++ "\n" ++
" version = \"0.0.1\";\n" ++
" hashes = {\n" ++
" \"aarch64-linux\" = \"sha256-old\";\n" ++
" \"x86_64-linux\" = \"sha256-old\";\n" ++
" };\n" ++
" " ++ end_marker ++ "\n" ++
"in\n";
const rewritten = try rewriteBlock(arena, before, "flake.nix", "0.0.16", .{
"00" ** digest_length,
"ff" ** digest_length,
});
const expected =
"let\n" ++
" " ++ begin_marker ++ "\n" ++
" version = \"0.0.16\";\n" ++
" hashes = {\n" ++
" \"aarch64-linux\" = \"sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=\";\n" ++
" \"x86_64-linux\" = \"sha256-//////////////////////////////////////////8=\";\n" ++
" };\n" ++
" " ++ end_marker ++ "\n" ++
"in\n";
try testing.expectEqualStrings(expected, rewritten);
const block = try readBlock(arena, rewritten, "flake.nix");
try testing.expectEqualStrings("0.0.16", block.version);
try testing.expectEqualStrings("sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=", block.hashes[0]);
}
test "locateBlock refuses a file that does not delimit exactly one block" {
const body =
" version = \"0.0.1\";\n" ++
" hashes = {\n" ++
" };\n";
const one = "let\n" ++ " " ++ begin_marker ++ "\n" ++ body ++ " " ++ end_marker ++ "\nin\n";
const span = try locateBlock(one);
try testing.expectEqualStrings(" ", span.indent);
try testing.expectError(error.NotOneEnd, locateBlock(
"let\n" ++ " " ++ begin_marker ++ "\n" ++ body ++ "in\n",
));
try testing.expectError(error.NotOneEnd, locateBlock(
"let\n" ++ " " ++ begin_marker ++ "\n" ++ body ++
" " ++ end_marker ++ "\n" ++ " " ++ end_marker ++ "\nin\n",
));
try testing.expectError(error.EndBeforeBegin, locateBlock(
"let\n" ++ " " ++ end_marker ++ "\n" ++ body ++ " " ++ begin_marker ++ "\nin\n",
));
try testing.expectError(error.NotOneBegin, locateBlock(
"let\n" ++ body ++ " " ++ end_marker ++ "\nin\n",
));
}
+25
View File
@@ -102,6 +102,11 @@ fn collectSorted(arena: Allocator, io: std.Io, dist: std.Io.Dir) ![]const []cons
defer walker.deinit();
while (try walker.next(io)) |entry| {
if (entry.kind != .file) continue;
// A hidden file is build metadata, not a web asset: the freshness stamp
// `admin/dist/.src-hash` exists in a local tree and not in a bundle that
// went through a CI artifact round trip, and embedding it would make the
// two binaries differ.
if (entry.basename.len != 0 and entry.basename[0] == '.') continue;
for (entry.path) |c| {
if (!std.ascii.isAlphanumeric(c) and std.mem.findScalar(u8, "._-/", c) == null) {
std.process.fatal("asset name '{s}' has a character the index cannot carry", .{entry.path});
@@ -231,3 +236,23 @@ fn renderIndex(arena: Allocator, assets: []const Asset) ![]const u8 {
try w.writeAll("};\n");
return sink.written();
}
const testing = std.testing;
test "a hidden file is not indexed" {
const io = testing.io;
var arena_state: std.heap.ArenaAllocator = .init(testing.allocator);
defer arena_state.deinit();
const arena = arena_state.allocator();
var tmp = testing.tmpDir(.{ .iterate = true });
defer tmp.cleanup();
try tmp.dir.writeFile(io, .{ .sub_path = "index.html", .data = "<!doctype html>" });
try tmp.dir.writeFile(io, .{ .sub_path = ".src-hash", .data = "deadbeef" });
const names = try collectSorted(arena, io, tmp.dir);
try testing.expectEqual(@as(usize, 1), names.len);
try testing.expectEqualStrings("index.html", names[0]);
}
+2 -7
View File
@@ -7,7 +7,7 @@
//! Usage (the build system supplies all of it):
//!
//! verify_dist --dist-dir <dir> --work-dir <dir> --version <v>
//! --git-commit <c> --zon <build.zig.zon> --max-bytes <n>
//! --zon <build.zig.zon> --max-bytes <n>
//! --asset-free-max-bytes <n> --host-arch <arch> [--qemu]
//! [--archive <triple> <basename>]...
//! [--asset-free <triple> <path>]...
@@ -92,7 +92,6 @@ const Args = struct {
dist_dir: []const u8 = "",
work_dir: []const u8 = "",
version: []const u8 = "",
git_commit: []const u8 = "",
zon: []const u8 = "",
max_bytes: u64 = 0,
asset_free_max_bytes: u64 = 0,
@@ -186,8 +185,6 @@ fn parseArgs(arena: Allocator, argv: []const []const u8) !Args {
args.work_dir = value;
} else if (std.mem.eql(u8, flag, "--version")) {
args.version = value;
} else if (std.mem.eql(u8, flag, "--git-commit")) {
args.git_commit = value;
} else if (std.mem.eql(u8, flag, "--zon")) {
args.zon = value;
} else if (std.mem.eql(u8, flag, "--host-arch")) {
@@ -621,9 +618,7 @@ fn checkVersionOutput(
},
}
const expected = std.fmt.allocPrint(arena, "nxdns {s} ({s})", .{
args.version, args.git_commit,
}) catch @panic("OOM");
const expected = std.fmt.allocPrint(arena, "nxdns {s}", .{args.version}) catch @panic("OOM");
var lines = std.mem.splitScalar(u8, result.stdout, '\n');
const first = lines.next() orelse "";
if (!std.mem.eql(u8, first, expected)) {