Files
nxdns/specs/milestone-14.md
T
mokhtar 31a6f0c5e5
Gates / test (push) Successful in 1m18s
Gates / test-aarch64 (push) Successful in 5m4s
Gates / frontend (push) Successful in 49s
Gates / package (push) Successful in 5m24s
Gates / container (push) Successful in 2m18s
CI / gates (push) Successful in 14m54s
changelog and spec: dry run complete, restore 0.0.1, record deviations 25 and 26
2026-08-08 21:43:36 +02:00

41 KiB
Raw Permalink Blame History

Milestone 14: build, package and publish releases

Goal: turn a signed git tag into a published, verifiable release — two static musl tarballs and one multi-architecture container image on the self-hosted Gitea at git.mial.net, with a licence, third-party notices, a changelog, a detached signature, and documentation that leads with download-and-verify.

First tag: v0.0.1.

Rulings (binding)

1. PLAN.md is amended first

PLAN.md:41 lists "Prebuilt binaries / published Docker images / project website" under §2.2 Out of Scope (permanent scope decisions, not deferrals). AGENTS.md makes PLAN the source of truth, so this milestone is invalid until that line is amended. The orchestrator edits PLAN before any session starts:

  • §2.2: remove prebuilt binaries and published container images. The project website stays out of scope.
  • Add a §on publication: tag-triggered releases, the artifact set, the signing model, and the two deferrals in ruling 12.
  • PLAN.md:571: drop "build date" from the nxdns version line. The version string and the git commit identify a build exactly, and a date is one more input that a reproducible build would have to pin. src/version.zig is already correct; PLAN was wrong.
  • PLAN.md:638: state exact byte limits — 15,728,640 with embedded assets, 10,485,760 without — resolving milestone-13 discrepancy 9 in favour of what CI already asserts.

2. The version lives in the tag, and in exactly one other place

The tag is authoritative. v0.0.1 means -Dversion-string=0.0.1; -Dgit-commit is the tag's peeled commit.

build.zig.zon:3 holds .version, which Zig requires and nothing reads. It is bumped in the commit before each tag, and verify-dist fails when it disagrees with the version under build. Two strings, one assert, no third copy anywhere.

Tags are never moved. A tag that produced a bad release is abandoned; the fix ships as the next patch version. Only vMAJOR.MINOR.PATCH is accepted — the release workflow rejects any tag carrying a pre-release suffix.

3. Licence and third-party notices

LICENSE holds the English text of EUPL-1.2 verbatim, with Copyright (c) 2026 Mokhtar Mial. README.md gains a short notice naming the licence and the SPDX identifier EUPL-1.2. No per-file SPDX headers.

THIRD-PARTY-NOTICES is assembled by zig build dist from a committed, reviewed inventory under licenses/. It is not scraped from the dependency tree at build time: a generated notices file that nobody reads rots silently into a false statement.

The inventory must cover everything the shipped artifacts actually contain, not the direct dependency list:

  • musl libc (MIT) — statically linked into every binary.
  • Zig standard library and compiler-rt (MIT) — likewise.
  • SQLite 3.53.4 — public domain, no obligation, listed for completeness.
  • Mbed TLS 3.6.7 — with an explicit line recording that it is taken under the Apache-2.0 option of its dual Apache-2.0 OR GPL-2.0-or-later licence, followed by the full Apache-2.0 text. Naming the choice is good practice; shipping the text is the actual obligation.
  • Project Everest and p256-m — compiled in at build.zig:394 even though the stock config leaves both drivers disabled.
  • The web bundle's runtime closure — the transitive set, not the four direct entries in web/package.json. Tailwind is a devDependency whose generated CSS ships, so "production dependencies" understates it.

A CI guard records the identity of the dependency sets (build.zig.zon dependencies, and the npm closure) and fails when either changes without a matching change under licenses/. It detects drift; it does not derive the inventory.

The container image carries /LICENSE and /THIRD-PARTY-NOTICES too. Distributing the image is distribution, and the obligations do not live in the tarball.

4. zig build dist replaces zig build cross

The cross step is deleted. dist is the single command that produces everything releasable, and it runs on a laptop exactly as it runs on the runner.

Inputs: -Dversion-string (required — no default), -Dgit-commit, -Dweb-dist, -Doptimize=ReleaseSafe.

dist fails when -Dweb-dist resolves to web/dist-placeholder. The default at build.zig:24 is the placeholder, so a release built without the flag would silently ship a placeholder admin page. There is no override flag; an escape hatch here is a foot-gun with a safety label on it.

Per triple (x86_64-linux-musl, aarch64-linux-musl), dist builds ReleaseSafe with .linkage = .static and .strip = true. Stripping uses std.Build.Module.strip (-fstrip, Module.zig:545), which removes the objcopy and binutils-aarch64-linux-gnu dependency from the runner.

It stages one directory per triple, nxdns-<version>-<triple>/, holding:

File Mode
nxdns 0755
nxdns.service 0644
nxdns.conf 0644
LICENSE 0644
THIRD-PARTY-NOTICES 0644
INSTALL.md 0644

deploy/systemd/sysusers.conf is renamed to nxdns.conf — the name it is installed under at /usr/lib/sysusers.d/nxdns.conf. A file that changes name during install is a step an operator can get wrong.

Archiving runs as two b.addSystemCommand steps, never one:

tar --format=gnu --sort=name --mtime=@0 --owner=0 --group=0 \
    --numeric-owner -cf <name>.tar <dir>
gzip -n -9 <name>.tar

b.addSystemCommand executes argv directly and does not interpret |, and a shell wrapper without pipefail would report only gzip's exit status while a failed tar passed silently. gzip -n is required because --mtime=@0 normalises the tar member times but not the timestamp gzip writes into its own header. The environment sets LC_ALL=C and TZ=UTC.

Both steps declare the staging directory as a build input and the archive as an output, so Zig's cache cannot serve a stale artifact.

dist also emits a checksum file covering only the two tarballs. It cannot cover the image: the digest does not exist until buildx has pushed, which happens later and elsewhere. The release job appends that line (ruling 7).

5. zig build verify-dist asserts what CI shell asserts today

The 50 lines of shell at ci.yml:129 and :168 move into a build step, so a developer can run the release checks on a laptop. Logic that only runs in CI is the brittleness this milestone exists to remove.

It operates on the extracted archive, not the staging directory:

  • ELF header: correct e_machine per triple, no PT_INTERP, no DT_NEEDED. Matching the string statically linked from file(1) is not a static-linkage test.
  • Stripped binary ≤ 15,728,640 bytes.
  • Archive layout: exactly one top-level directory, the exact file allowlist from ruling 4, expected modes, no symlinks, no path traversal.
  • nxdns version prints the version under build and the git commit. Native architecture only — the aarch64 binary needs qemu and is skipped without -fqemu.
  • build.zig.zon .version equals the version under build.

The asset-free budget (10,485,760 bytes) gets its own build against a generated empty assets directory, not against the placeholder. Ruling 4 makes the placeholder unbuildable, and re-admitting it through a back door for one size check would defeat the point.

A second zig build invocation does not inherit the first one's -D options. zig build dist -Dversion-string=X followed by a bare zig build verify-dist verifies a differently configured build. Every caller — the workflows, the documentation, and this spec's own acceptance list — passes the same -Dversion-string, -Dgit-commit, -Dweb-dist and -Doptimize to both.

6. Container image

deploy/docker/Dockerfile:

  • Pin the base by digest: alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce.
  • Add --platform=$BUILDPLATFORM to the builder stage. It only copies files, so pinning it to the build host means the arm64 image needs no qemu.
  • Delete apk add --no-cache ca-certificates (Dockerfile:15). Verified: the Alpine base already ships /etc/ssl/certs/ca-certificates.crt (179,359 bytes) from ca-certificates-bundle. Removing it removes the only network fetch in the image build.
  • Keep the mkdir and chown 65532/var/lib/nxdns must exist with that ownership so Docker copies it onto a fresh named volume.
  • Add /LICENSE and /THIRD-PARTY-NOTICES (ruling 3).
  • OCI labels: source, revision, version, licenses=EUPL-1.2, created, title, description.

deploy/docker/.dockerignore is renamed to deploy/docker/Dockerfile.dockerignore. The build context is the repository root, so Docker never reads the current path, and the whole worktree — .git, node_modules, zig-cache — is being sent to the daemon today.

buildx runs with --provenance=false --sbom=false. Recent buildx adds provenance attestations by default, which create unknown/unknown platform entries and change the index digest; Gitea's OCI 1.1 support is unverified (go-gitea#25846) and this is not the milestone to find out. SOURCE_DATE_EPOCH comes from the tag date.

deploy/docker/compose.yaml references the published image rather than building one.

Verify before publishing: the binary inside each image is byte-identical to the binary in the matching tarball.

7. Workflows

.gitea/workflows/gates.yml, on: workflow_call, holds every blocking check: test (with -Dintegration), test-aarch64 (-fqemu), frontend, package (dist + verify-dist + the asset-free size check) and container (build the image and run the existing smoke test from ci.yml:219). Moving only the three test jobs would drop the packaging and container checks precisely when they matter most.

ci.yml calls it on push and pull_request for master. master is canonical: origin/main is deleted, and Gitea's default branch is changed to match. Today ci.yml:5 watches main while work happens on master, so milestone 13 has never run through CI.

.gitea/workflows/release.yml, on: push: tags: ['v*'], in this order:

  1. Checkout with fetch-depth: 0 and tags. The default shallow clone breaks ancestry checks, previous-tag lookup and changelog generation.
  2. Reject any tag that is not exactly vMAJOR.MINOR.PATCH.
  3. git verify-tag, requiring an annotated tag and requiring the signature's fingerprint to equal a fingerprint pinned in the workflow. A bare verify-tag proves only that some imported key signed it.
  4. Assert the tag's commit is an ancestor of origin/master.
  5. Assert no published release exists for this tag. Delete any leftover draft.
  6. Assert the version is greater than the highest published release version, so a late-finishing older tag cannot move latest backwards.
  7. Gates, blocking, via needs:.
  8. Build the web UI; zig build dist; zig build verify-dist.
  9. Extract the CHANGELOG.md section matching the version. Fail when absent.
  10. buildx build and push :<version> only. Capture the index digest from --metadata-file, validate it as sha256:<64 hex>, and confirm the pushed tag resolves to that digest with exactly the two intended platforms.
  11. Write the digest file, append its checksum line, verify the assembled checksum file.
  12. Sign the checksum file (ruling 8). Verify the signature locally before uploading it.
  13. Create the release as a draft; upload the assets.
  14. Move :latest and verify it resolves to the built digest. Re-check the monotonic-version invariant here: step 6 ran before the gates, and proves nothing about which of two in-flight tags finishes last.
  15. Publish the draft last. Publication is the one irreversible act, so it goes after everything that can still fail.

Amended after review. The original order published at 14 and moved :latest at 15, which deadlocks: a failure while moving :latest leaves a published release, and ruling 9 makes a re-run refuse a published release. Nothing could repair it. The cost of the corrected order is a short window where :latest serves the new image before the release page is public. That is recoverable by a re-run; the deadlock was not.

Every action in release.yml is pinned to a full commit SHA. actions/checkout@v4 and mlugg/setup-zig@v2 are mutable tags on another party's server, and a compromise upstream would run on the runner holding the signing subkey and the registry token. ci.yml may keep moving tags; it holds no secrets.

8. Signing

Two different keys are involved, and the original ruling conflated them.

  • The tag-signing key is the human's. git tag -s uses it, and step 3 checks it. What gets pinned in release.yml is the primary certificate fingerprint, which git verify-tag --raw emits as the last field of the VALIDSIG line. Field 3 is whichever key actually made the signature — the signing subkey once one exists. Pinning field 3 would mean that creating the release subkey below silently blocks every future release, with an error message that reads like a forged tag. Reproduced against a real keyring during review.
  • The artifact-signing subkey is the runner's, and it signs the checksum file. It is a dedicated GPG signing subkey of the author's existing key. A leaked subkey is revoked on its own; the identity, the commit signature history and everyone's existing trust survive.

Pinning the primary fingerprint means adding or rotating a signing subkey is a non-event for verification.

Every required secret is validated in the guard job, before the gates and before any registry push. Validating a fingerprint inside the signing step means a placeholder value burns an immutable version tag before it fails.

Implementation requirements: a temporary GNUPGHOME; assert the imported material contains no primary secret key; --local-user <subkey-fingerprint>! so GPG cannot fall back to another key; batch and loopback pinentry; verify the produced signature before upload; scrub GNUPGHOME and DOCKER_CONFIG and kill the agent on every exit path. The registry token is passed by --password-stdin into a temporary DOCKER_CONFIG, and persist-credentials is off.

Secrets: RELEASE_GPG_SUBKEY, RELEASE_GPG_PASSPHRASE, REGISTRY_TOKEN. The built-in GITEA_TOKEN cannot publish to the package registry, which is why the third exists; it can still create the release and upload assets.

Stated honestly, in docs/how-to/verify-a-release.md: the signature proves the artifact came from this pipeline and reached the operator unaltered. It does not prove the binary matches the source, because the machine that built it also held the key. Ruling 12 defers the control that closes that gap. Do not write a stronger claim than that.

9. Failure and recovery are specified, not improvised

  • The draft is the unit of work for the release record. The registry is not covered by it: the version tag becomes publicly pullable at step 10. Only the release page and its assets stay hidden until step 15.
  • The existence check happens before the push, not after. Gitea's container tags are mutable — immutability is a workflow invariant, not a registry guarantee — so a push-then-compare would already have overwritten the tag it claims to refuse. Read the existing digest with HEAD /v2/mokhtar/nxdns/manifests/<version>, sending an Accept header for the index media types and following the 401 bearer challenge with the PAT. Absent is 404; present returns Docker-Content-Digest.
  • A re-run deletes an existing draft and repeats. It refuses to touch a published release.
  • The registry version tag is immutable, and the workflow never writes it twice. A re-run that finds the tag present pushes nothing: it adopts the pushed digest and asserts the contents of that image against the artifacts it just built, which is the check that matters and the only one available. Comparing a rebuilt index digest was the first design and is not implementable — buildx cannot report an index digest without pushing, and cross-machine reproducibility is deferred (ruling 12), so the rebuilt digest is expected to differ even when nothing changed. See recorded deviation 10.
  • :latest moves last, so a failure between the image push and publication leaves the version tag pushed and latest untouched. That is recoverable by re-running.
  • If a tag is burned — the pipeline itself is broken and the fix is on master — the release is abandoned and reissued as the next patch version. This is the terminal path, and it must be written down before v0.0.1, not discovered during it.

10. Release notes

CHANGELOG.md at the repository root, Keep a Changelog format, with an ## [Unreleased] section maintained as work happens. The release body is that section, plus a generated appendix: git log --oneline since the previous tag inside a collapsed <details>, a compare link, the tarball hashes and the image digest.

The nullable value is "the previous reachable published release", never "the previous git tag". An abandoned tag from ruling 9 must not become the comparison base, and git describe would pick exactly that.

With no published base — the first release, and equally the case where v0.0.1 was abandoned and v0.0.2 becomes the first published one — the log is git log --oneline <tag> and the compare link is omitted. It is not a range with an empty left side: ..v0.0.1 resolves against HEAD and produces a wrong or empty appendix rather than "all history".

The v0.0.1 section is hand-written.

11. Documentation

install-with-systemd.md, install-with-docker.md and upgrade.md lead with download-and-verify; the existing build-from-source steps move to a later section of the same page. New docs/how-to/verify-a-release.md gives the verification commands and the rebuild recipe.

Milestone 13 ruling 3 applies unchanged: every command block in tutorial/ and how-to/ is executed on this host by the session that writes it, or marked in-page as unverified with the reason.

Two rot hazards the drift test cannot see. src/docs_drift_test.zig guards only reference/{api,configuration,cli}.md; the how-to pages are unguarded:

  • The transcripts hardcode nxdns 0.1.0-dev in four pages. After v0.0.1 they are wrong and nothing fails. Use a version-neutral placeholder, and add a guard that rejects a stale literal release version in the docs.
  • A download URL with the version in the path goes stale at v0.0.2. Use a latest download form if Gitea provides one, or a placeholder the reader substitutes. Probe first (ruling 13); do not guess.

Sweep the whole repository for surfaces the rewrite would otherwise miss: README.md, Dockerfile comments, compose.yaml, explanation/performance-and-testing.md, and any remaining zig build cross or source-only-distribution text.

12. Deferred, deliberately

Recorded in PLAN so they are decisions rather than oversights:

  • The reproducibility gate — build twice in two directory paths and assert identical hashes. Deferred until v0.0.1 proves the pipeline. Until it exists, no document may describe the build as reproducible. Do the cheap parts now regardless: pin Node to an exact patch in ci.yml and web/package.json (both float at 24 today), gzip -n, LC_ALL=C, TZ=UTC.
  • cosign signatures on the image — Gitea Actions has no OIDC identity token, so keyless signing is impossible and only a key-based signature is available. Whether Gitea's registry accepts a cosign signature manifest is unverified. Spike it before committing to it.

13. Manual prerequisites, done before the first tag

None of these are code, and all of them block v0.0.1:

The order matters, and the original list got parts of it wrong. Corrected:

  1. Create the artifact-signing subkey, export it with --export-secret-subkeys, and publish the public key to keys.openpgp.org. Separately, identify which key actually signs your tags — after step 1 that is normally the new signing subkey, whose primary fingerprint is what gets pinned.
  2. Pin the primary fingerprint in release.yml, replacing the placeholder. The guard fails closed on the placeholder, so no tag can succeed before this.
  3. Store the secrets: RELEASE_GPG_SUBKEY, RELEASE_GPG_PASSPHRASE, and a registry personal access token with package write scope as REGISTRY_TOKEN. Creating the token is not storing it.
  4. app.ini: add .asc and .sig to [attachment] ALLOWED_TYPES, and raise MAX_FILES from 5. Restart Gitea and confirm the effective settings through /api/v1/settings/attachment before trusting them. Verified against the live instance on 2026-08-05: max_size is 100 MB (ample), max_files is 5 (this release has exactly five assets, no headroom), and .asc is absent, so the signature is rejected today. The implementation names the assets SHA256SUMS.txt, SHA256SUMS.txt.asc and IMAGE-DIGEST.txt, so .txt and .asc are the two extensions that must be allowed.
  5. Probe: does the runner support docker buildx with the docker-container driver, and can it extract a foreign-platform image? The old docker job only proved plain docker build, and the arm64 image-versus-tarball check needs more than that.
  6. Land the workflows on master while main still exists, and let the reusable-gates workflow run green once, so Gitea emits its actual status-check context names.
  7. Configure branch protection on master using those observed names. The reusable-workflow refactor changes them; keeping the old required checks can make merges impossible or leave the intended gates non-required.
  8. Change Gitea's default branch to master.
  9. Delete origin/main last — Gitea refuses to delete the default branch.
  10. Merge the licence, changelog and build.zig.zon version commit, and let master go green.
  11. Dry-run the release path. A tag-triggered workflow cannot be exercised without pushing some tag, so use a disposable tag or a scratch repository. Never use v0.0.1 as the dry run and then expect to reuse it: tags are never moved (ruling 2), so a burned dry-run tag is spent.

Sessions

S1S5 run in parallel. S1 and S2 share one interface, fixed here so neither blocks: S2 owns everything under licenses/ and the licence texts; S1 owns the build step that assembles them into THIRD-PARTY-NOTICES. The contract is zig build dist -Dversion-string=X -Dgit-commit=Y -Dweb-dist=web/dist writing to zig-out/dist/.

Session S1: build system

Owns build.zig. Rulings 2, 4, 5. Deletes cross, adds dist and verify-dist, moves the CI shell asserts into the build graph.

Session S2: licence, notices, changelog

Owns LICENSE, licenses/, CHANGELOG.md, README.md (notice and links). Ruling 3, ruling 10. Compiles the inventory by auditing what the artifacts actually contain, and writes the CI drift guard for the dependency sets.

Session S3: container

Owns deploy/docker/*, deploy/systemd/* (the sysusers.conf rename). Ruling 6.

Session S4: workflows

Owns .gitea/workflows/*. Rulings 7, 8, 9. Must not edit build.zig — it consumes the step contract above.

Session S5: documentation

Owns docs/**. Ruling 11, and milestone 13 ruling 3.

Orchestrator

PLAN.md (ruling 1), this spec, deletion of origin/main, the manual prerequisites in ruling 13, and the v0.0.1 tag.

Recorded (implementation)

Accepted deviations and corrections from integration. The five rulings amended above (7, 8, 9, 10, 13, plus the option-inheritance note in 5 and the scoping of the last acceptance line) were all wrong as first written; each amendment says what it replaced and why.

  1. build.zig.zon said 0.1.0. It traced to the first build-baseline commit — a scaffold default never bumped. Ruling 2 makes verify-dist assert it against the version under build, so the CI packaging gate could never have passed. Set to 0.0.1, matching the first tag.
  2. Assets carry a .txt extension: SHA256SUMS.txt, SHA256SUMS.txt.asc, IMAGE-DIGEST.txt. This resolves ruling 13's extensionless-asset probe by construction — .txt is already in the live ALLOWED_TYPES, so only .asc still has to be added.
  3. INSTALL.md was added at the repository root. dist hard-requires it in the staged payload and verify-dist asserts its mode.
  4. The notices preamble moved to licenses/preamble.txt. The tarball and the image ship different sets, and the hardcoded preamble scoped itself to "a single static executable" — which the image is not. The image additionally redistributes Alpine's Mozilla CA bundle (MPL-2.0 AND MIT, 179,359 bytes); the tarball does not.
  5. Vite joined Tailwind in the inventory. Both are devDependencies whose generated output ships inside the binary. The original inventory applied that rule to one of them and stopped.
  6. Node is pinned to 24.19.0 in gates.yml, release.yml and web/package.json — the exact-patch pin ruling 12 asks for.
  7. gates.yml is SHA-pinned too. Ruling 7 pinned only release.yml, but release.yml calls gates.yml, and those jobs share the runner host and docker daemon with the job holding the signing subkey. live-tls.yml keeps moving tags; it references no secret.
  8. Determinism measured better than claimed. Two dist runs with separate cache directories and separate prefixes produced byte-identical tarballs — a genuine recompile, not a cache replay. The documentation still claims only same-directory determinism, because ruling 12's gate does not exist yet and an unguarded property decays. The stronger result is recorded here, not promised to operators.
  9. Test count moved from 1461 to 1481: twelve licence-drift tests, then eight more from the review pass below. Skip counts are unchanged.

The rest came out of an adversarial review of the finished implementation. Each was reproduced before it was fixed.

  1. The version tag was pushed before the immutability check ran. The first implementation pushed :$VERSION and then compared the resulting digest with the pre-push one — a check that reports a violation it just caused. On a re-run that produced different bytes the tag was already overwritten and the original image lost. Replaced by the probe-then-adopt design ruling 9 now describes: a real HEAD /v2/…/manifests/<version>, and an existing tag is adopted rather than rebuilt. Exercised against a fake registry covering 404, 200, the 401 bearer challenge, 500, and a 200 with no Docker-Content-Digest.

  2. The guard proved nothing about the signing key. It checked that RELEASE_GPG_SUBKEY and RELEASE_GPG_PASSPHRASE were non-empty. A public-only export, an export missing the pinned subkey, and a placeholder passphrase all passed it and failed for the first time in the signing step — after the image push. The guard now imports the material, asserts the pinned subkey is present as a secret key, and signs a throwaway file with the passphrase.

  3. jq … | grep … || true swallowed a malformed API response. The || true exists so grep's no-match is not fatal; it covers the whole pipeline, so a 200 carrying a JSON object instead of an array read as "no published releases" — the one wrong answer that moves :latest backwards. The payload is now type-checked before it is read.

  4. The monotonic re-check was not the concurrency backstop it claimed to be. Two concurrent releases are both drafts while they run, so neither appears in the other's published list and both pass. The workflow concurrency: group is the only thing that serialises them. A second check was added that does close it: :latest's own version label is read from the registry immediately before the tag moves, so the invariant is checked against the state being mutated.

  5. Publication could deadlock the tag. A PATCH that Gitea committed but whose response was lost left the release public and the step failed, after which the guard refused every re-run. The step now re-reads the release and treats an already-published one as success.

  6. The secret-scrub backstop killed the wrong gpg-agent. gpgconf --kill acts on the agent of the GNUPGHOME it is pointed at, and the bare call killed the runner's default agent while every leaked temporary home's agent kept running with the key cached. It now kills each home in its own home, and the guard job has the same if: always() backstop the publish job had.

  7. The Zig dependency guard could be silenced by pasting. Unlike the npm half, nothing tied build.zig.zon back to the inventory, so a Mbed TLS bump plus the suggested identity paste left the notices claiming the old version. Each dependency now has to appear in the inventory at the version its URL names. The Zig toolchain version and the entries vendored inside Mbed TLS are checked the same way.

  8. The container base image was not an input to any guard. It is the source of the CA bundle the image redistributes. deploy/docker/Dockerfile is now embedded in the licenses_files module, its digest-pinned FROM is a recorded identity section, and the CA bundle entry must name the Alpine release that FROM pins.

  9. A tree-shaken package that started shipping would have gone unnoticed. cookie-es, isbot, seroval and seroval-plugins are in the lockfile closure and in no shipped byte. If application code imported one, no lockfile, version or dependency set would change — only the bundle. The frontend gate now recomputes the set of packages in web/dist from a --sourcemap build and diffs it against a recorded section, and the drift test requires every name in that section to be inventoried and refuses one that is still listed as not shipped.

  10. The recorded npm licence token was parsed and discarded, so a package that relicensed passed as long as its version had not moved. Shipped packages must now all carry the licence the inventory's texts assume; npm_not_shipped is exempt, and isbot is Unlicense.

  11. The full-text licence checks were marker probes. They prove the right document is present but survive most of it being deleted. The Apache-2.0 and MPL-2.0 texts are now pinned by SHA-256.

  12. Zig's compiler-rt contains code ported from LLVM's. Zig's LICENSE is bare MIT naming only "Zig contributors". Reviewed: the ports carry Apache-2.0 WITH LLVM-exception, and that exception waives Apache §4(a), §4(b) and §4(d) for portions embedded in object form — the only form nxdns ships — so no further notice is owed. Recorded in the Zig inventory note rather than left as an unexamined gap.

  13. npx was replaced by the installed binary in the new frontend gate. npx downloads a package it cannot find locally, so a wrong working directory would have turned a licence check into an unpinned fetch. Reproduced: it fetched vite@8.2.0 over the pinned 8.1.5.

  14. actions/checkout destroys the annotated tag object. Found by the first live dry run, not by review: on a tag ref, checkout fetches the commit SHA into refs/tags/<tag>, so the signed tag reads as lightweight and the guard refuses it as unannotated. Both jobs that read the tag object — signature verification in the guard, the tagger date in the publish job — now force- refetch refs/tags/$TAG from origin first. The same run also proved the fail-closed secret guard for real: the first dry-run attempt ran with no secrets configured (they were on the wrong repository) and stopped in the guard with nothing built or pushed.

  15. Publication orchestration moved out of workflow shell into tools/release.zig. Ruling 5 already moved the packaging asserts out of CI shell for one reason — "checks that only exist inside a workflow file are the brittleness this exists to remove" — and the release job was the larger half of the same problem, left in place. Three live failures came out of it, and each was found by executing the workflow, which is the most expensive place to find anything: actions/checkout replacing the annotated tag object (deviation 23), the refetch that fixed it having no credentials because persist-credentials is off, and the multiline armored subkey escaping the runner's log masker, which masks per line.

    Twelve subcommands, one per step group: guard-tag, guard-ancestry, guard-releases, resolve, changelog, image, verify-image-binaries, sign, draft, latest, publish, scrub. Every behaviour recorded in deviations 10 to 15 and 23 is carried over unchanged — probe-adopt, the VALIDSIG last field, the subkey-only import and signing probe, the array-shape guard on the releases payload, the :latest label read, the publish re-read, the per-home gpgconf --kill, the tag refetch. What is new is that the semver ordering, VALIDSIG field selection, challenge parsing, changelog extraction, checksum-line parsing, colon-format parsing and payload-shape guard are 25 unit tests in zig build test rather than shell that only ever runs on a tag push. release.yml keeps the triggers, the concurrency group, the job graph, the SHA pins, the two pinned fingerprints and the fail-closed secret presence check — which stays as shell, deliberately, so that it runs before the tool is even compiled.

    The same reasoning applies to the jq pipeline of the bundled-package gate, which moved to web/scripts/bundledPackages.mjs with its own vitest coverage and an npm run assert-bundled entry point.

    Secret contract change: RELEASE_GPG_SUBKEY keeps its name but now holds base64 -w0 of the armored --export-secret-subkeys output rather than the armored text. Manual prerequisite 1 and 3 change accordingly. The tool decodes it in memory and writes it to a mode-600 file inside the temporary GNUPGHOME. A single-line secret is one the masker can actually mask.

  16. The first signing subkey was leaked into a job log and rotated. Dry-run attempt 3 failed inside the credential-less refetch, and the runner printed the failing step's env block; the multiline armored RELEASE_GPG_SUBKEY escaped the per-line masker while the single-line passphrase was masked. Exposure: the passphrase-protected secret subkey only — the passphrase and the primary key were never on the runner. Response: both runs that ever saw the secret were deleted (verified 404 via the API and absent from actions_log on disk), subkey B281CECC… was revoked with the primary, and its replacement 019D00DF… is the pinned RELEASE_SIGNING_FPR. The base64 contract in deviation 24 is the preventive half of this record.

  17. The image is named by the public registry host, never the server URL. Attempt 4 reached the registry and failed at docker login gitea:3000: inside the cluster GITHUB_SERVER_URL is http://gitea:3000, docker refuses plain-http registries, and an image named gitea:3000/… would be unpullable from anywhere that matters — a wrong name that would have been written into the released IMAGE-DIGEST.txt. release.yml now pins REGISTRY_HOST: git.mial.net; the tool uses it for docker and image naming, and keeps the internal URL for the manifest probe (same registry, no TLS dependency in the tool). The old shell had the identical latent bug; no run ever reached it.

Not verified, and why

  • The workflows' validation history. Before any live run, release.yml and gates.yml were validated by YAML parse and bash -n, plus two steps lifted out and run directly: the registry probe against a fake registry (five response shapes) and the bundled-package check against the real web/ build, proven able to fail. The live dry run then superseded this: attempt 5 published v0.0.0 end to end — guard, gates, image push to both platforms, binary-identity assertion, signing, draft, :latest, publication — and the assets verified from a clean directory (checksums OK, signature good under the rotated subkey). The throwaway release, tag and registry versions were deleted afterwards. Everything else that talks to the registry or the Gitea API — buildx build --push, imagetools, draft creation, asset upload, publication, the adopt-an-existing-tag path — is unexercised.
  • RELEASE_SIGNING_FPR is still the placeholder. By design: the guard fails closed on it. It also means the release workflow cannot succeed as committed until manual prerequisite 2 is done.
  • The aarch64 binary was never executed. No qemu-aarch64 on the build host, so verify-dist's aarch64 version check legitimately skips and test-aarch64 could not run. The binary is checked statically: ELF class, no PT_INTERP, no DT_NEEDED, size.
  • The multi-architecture image build is unverified. Only the native amd64 image was built and run.
  • shellcheck was not run — not installed on the build host.
  • The local Node is 24.14.1, not the pinned 24.19.0. There is no .npmrc, so engines does not hard-fail, and the frontend gates ran under the older patch.

Acceptance (milestone complete)

  • PLAN.md §2.2 amended; the build-date line and the byte limits corrected. §3.15 now describes the single gate set, and a new §20 records the publication model.
  • zig build dist -Dversion-string=0.0.1 -Dweb-dist=web/dist produces two tarballs and a checksum file, and fails without -Dweb-dist.
  • zig build verify-dist, given the same options as dist, passes and was proven able to fail: an oversized binary, a dynamically linked binary, a mismatched build.zig.zon version and a wrong archive mode each produce a named failure. Proven by repacking tarballs with each defect: binary-size, elf, zon-version and archive-mode each fired and the run exited 1. The oversize proof used a genuinely oversized binary for binary-size and a lowered budget for asset-free-size.
  • Two runs of zig build dist on the same commit produce byte-identical tarballs in the same directory. (Cross-directory reproducibility is ruling 12 and is not claimed here.)
  • The image builds for both platforms with no qemu, carries /LICENSE and /THIRD-PARTY-NOTICES and the OCI labels, and its binaries are byte-identical to the tarball binaries. Verified for the native amd64 image only; the arm64 half needs a runner with buildx.
  • gates.yml runs from both ci.yml and release.yml; ci.yml triggers on master; origin/main is gone. The first two are in the files; no workflow has run and origin/main still exists (manual prerequisite).
  • THIRD-PARTY-NOTICES covers musl, the Zig runtime, SQLite, Mbed TLS with its Apache-2.0 selection line and full text, Everest, p256-m and the web runtime closure. The dependency drift guard was proven able to fail: removing an inventory entry, staling a dependency version, staling the Zig version, changing the base image digest, editing a pinned licence text and dropping a package from the recorded bundle each produce a named failure.
  • A dry run of release.yml completes with publication disabled. Done with a disposable published tag instead: publication cannot be disabled without forking the flow it is supposed to prove, so v0.0.0 ran the real path end to end — five assets, verifying checksums and signature, a multi-architecture image — and was then deleted (release, git tag, both registry versions). Five attempts; the failures and their fixes are deviations 23-26.
  • v0.0.1 is published: five assets, a verifying signature, and an image at git.mial.net/mokhtar/nxdns:0.0.1 and :latest.
  • docs/how-to/verify-a-release.md was followed end to end against the published release, from a clean directory, on this host.
  • No zig build cross or source-only-distribution text remains on any active surface: build.zig, the workflows, deploy/, README.md and docs/. Historical milestone specs and TECH_DEBT.md keep their text — this spec contains the string itself, so "anywhere" was never satisfiable.

Anti-requirements

  • No GoReleaser, no nfpm, no .deb or .rpm, no Homebrew, no AUR.
  • No cosign, no SBOM, no SLSA provenance, no in-toto attestations.
  • No :edge image, no rolling dev build, no pre-release tags, no floating 0.0 or 0 image tags.
  • No third architecture, no non-Linux target, no glibc build.
  • No project website, no documentation site generator.
  • No changelog generation from commit messages, and no Conventional Commits.
  • No release automation that cannot be run from a laptop.