# Milestone 14: build, package and publish releases

Goal: turn a signed git tag into a published, verifiable release — two static musl tarballs and one multi-architecture container image on the self-hosted Gitea at `git.mial.net`, with a licence, third-party notices, a changelog, a detached signature, and documentation that leads with download-and-verify.

First tag: `v0.0.1`.

## Rulings (binding)

### 1. PLAN.md is amended first

`PLAN.md:41` lists "Prebuilt binaries / published Docker images / project website" under §2.2 *Out of Scope (permanent scope decisions, not deferrals)*. `AGENTS.md` makes PLAN the source of truth, so this milestone is invalid until that line is amended. The orchestrator edits PLAN before any session starts:

- §2.2: remove prebuilt binaries and published container images. **The project website stays out of scope.**
- Add a §on publication: tag-triggered releases, the artifact set, the signing model, and the two deferrals in ruling 12.
- `PLAN.md:571`: drop "build date" from the `nxdns version` line. The version string and the git commit identify a build exactly, and a date is one more input that a reproducible build would have to pin. `src/version.zig` is already correct; PLAN was wrong.
- `PLAN.md:638`: state exact byte limits — 15,728,640 with embedded assets, 10,485,760 without — resolving milestone-13 discrepancy 9 in favour of what CI already asserts.

### 2. The version lives in the tag, and in exactly one other place

The tag is authoritative. `v0.0.1` means `-Dversion-string=0.0.1`; `-Dgit-commit` is the tag's peeled commit.

`build.zig.zon:3` holds `.version`, which Zig requires and nothing reads. It is bumped in the commit before each tag, and `verify-dist` fails when it disagrees with the version under build. Two strings, one assert, no third copy anywhere.

Tags are **never moved**. A tag that produced a bad release is abandoned; the fix ships as the next patch version. Only `vMAJOR.MINOR.PATCH` is accepted — the release workflow rejects any tag carrying a pre-release suffix.

### 3. Licence and third-party notices

`LICENSE` holds the English text of EUPL-1.2 verbatim, with `Copyright (c) 2026 Mokhtar Mial`. `README.md` gains a short notice naming the licence and the SPDX identifier `EUPL-1.2`. **No per-file SPDX headers.**

`THIRD-PARTY-NOTICES` is assembled by `zig build dist` from a committed, reviewed inventory under `licenses/`. It is not scraped from the dependency tree at build time: a generated notices file that nobody reads rots silently into a false statement.

The inventory must cover everything the shipped artifacts actually contain, not the direct dependency list:

- **musl libc** (MIT) — statically linked into every binary.
- **Zig standard library and compiler-rt** (MIT) — likewise.
- **SQLite 3.53.4** — public domain, no obligation, listed for completeness.
- **Mbed TLS 3.6.7** — with an explicit line recording that it is taken under the Apache-2.0 option of its dual `Apache-2.0 OR GPL-2.0-or-later` licence, followed by the **full Apache-2.0 text**. Naming the choice is good practice; shipping the text is the actual obligation.
- **Project Everest and p256-m** — compiled in at `build.zig:394` even though the stock config leaves both drivers disabled.
- **The web bundle's runtime closure** — the transitive set, not the four direct entries in `web/package.json`. Tailwind is a devDependency whose generated CSS ships, so "production dependencies" understates it.

A CI guard records the identity of the dependency sets (`build.zig.zon` dependencies, and the npm closure) and fails when either changes without a matching change under `licenses/`. It detects drift; it does not derive the inventory.

The **container image carries `/LICENSE` and `/THIRD-PARTY-NOTICES` too**. Distributing the image is distribution, and the obligations do not live in the tarball.

### 4. `zig build dist` replaces `zig build cross`

The `cross` step is deleted. `dist` is the single command that produces everything releasable, and it runs on a laptop exactly as it runs on the runner.

Inputs: `-Dversion-string` (required — no default), `-Dgit-commit`, `-Dweb-dist`, `-Doptimize=ReleaseSafe`.

**`dist` fails when `-Dweb-dist` resolves to `web/dist-placeholder`.** The default at `build.zig:24` is the placeholder, so a release built without the flag would silently ship a placeholder admin page. There is no override flag; an escape hatch here is a foot-gun with a safety label on it.

Per triple (`x86_64-linux-musl`, `aarch64-linux-musl`), `dist` builds ReleaseSafe with `.linkage = .static` and `.strip = true`. Stripping uses `std.Build.Module.strip` (`-fstrip`, `Module.zig:545`), which removes the `objcopy` and `binutils-aarch64-linux-gnu` dependency from the runner.

It stages one directory per triple, `nxdns-<version>-<triple>/`, holding:

| File | Mode |
|---|---|
| `nxdns` | 0755 |
| `nxdns.service` | 0644 |
| `nxdns.conf` | 0644 |
| `LICENSE` | 0644 |
| `THIRD-PARTY-NOTICES` | 0644 |
| `INSTALL.md` | 0644 |

`deploy/systemd/sysusers.conf` is renamed to `nxdns.conf` — the name it is installed under at `/usr/lib/sysusers.d/nxdns.conf`. A file that changes name during install is a step an operator can get wrong.

Archiving runs as two `b.addSystemCommand` steps, never one:

```
tar --format=gnu --sort=name --mtime=@0 --owner=0 --group=0 \
    --numeric-owner -cf <name>.tar <dir>
gzip -n -9 <name>.tar
```

`b.addSystemCommand` executes argv directly and does not interpret `|`, and a shell wrapper without `pipefail` would report only `gzip`'s exit status while a failed `tar` passed silently. `gzip -n` is required because `--mtime=@0` normalises the tar member times but not the timestamp gzip writes into its own header. The environment sets `LC_ALL=C` and `TZ=UTC`.

Both steps declare the staging directory as a build input and the archive as an output, so Zig's cache cannot serve a stale artifact.

`dist` also emits a checksum file covering **only the two tarballs**. It cannot cover the image: the digest does not exist until buildx has pushed, which happens later and elsewhere. The release job appends that line (ruling 7).

### 5. `zig build verify-dist` asserts what CI shell asserts today

The 50 lines of shell at `ci.yml:129` and `:168` move into a build step, so a developer can run the release checks on a laptop. Logic that only runs in CI is the brittleness this milestone exists to remove.

It operates on the **extracted** archive, not the staging directory:

- ELF header: correct `e_machine` per triple, **no `PT_INTERP`, no `DT_NEEDED`**. Matching the string `statically linked` from `file(1)` is not a static-linkage test.
- Stripped binary ≤ 15,728,640 bytes.
- Archive layout: exactly one top-level directory, the exact file allowlist from ruling 4, expected modes, no symlinks, no path traversal.
- `nxdns version` prints the version under build and the git commit. Native architecture only — the aarch64 binary needs qemu and is skipped without `-fqemu`.
- `build.zig.zon` `.version` equals the version under build.

The asset-free budget (10,485,760 bytes) gets **its own build against a generated empty assets directory**, not against the placeholder. Ruling 4 makes the placeholder unbuildable, and re-admitting it through a back door for one size check would defeat the point.

**A second `zig build` invocation does not inherit the first one's `-D` options.** `zig build dist -Dversion-string=X` followed by a bare `zig build verify-dist` verifies a *differently configured* build. Every caller — the workflows, the documentation, and this spec's own acceptance list — passes the same `-Dversion-string`, `-Dgit-commit`, `-Dweb-dist` and `-Doptimize` to both.

### 6. Container image

`deploy/docker/Dockerfile`:

- Pin the base by digest: `alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce`.
- Add `--platform=$BUILDPLATFORM` to the builder stage. It only copies files, so pinning it to the build host means the arm64 image needs no qemu.
- **Delete `apk add --no-cache ca-certificates` (`Dockerfile:15`).** Verified: the Alpine base already ships `/etc/ssl/certs/ca-certificates.crt` (179,359 bytes) from `ca-certificates-bundle`. Removing it removes the only network fetch in the image build.
- Keep the `mkdir` and `chown 65532` — `/var/lib/nxdns` must exist with that ownership so Docker copies it onto a fresh named volume.
- Add `/LICENSE` and `/THIRD-PARTY-NOTICES` (ruling 3).
- OCI labels: `source`, `revision`, `version`, `licenses=EUPL-1.2`, `created`, `title`, `description`.

`deploy/docker/.dockerignore` is renamed to `deploy/docker/Dockerfile.dockerignore`. The build context is the repository root, so Docker never reads the current path, and the whole worktree — `.git`, `node_modules`, `zig-cache` — is being sent to the daemon today.

buildx runs with `--provenance=false --sbom=false`. Recent buildx adds provenance attestations by default, which create `unknown/unknown` platform entries and change the index digest; Gitea's OCI 1.1 support is unverified (go-gitea#25846) and this is not the milestone to find out. `SOURCE_DATE_EPOCH` comes from the tag date.

`deploy/docker/compose.yaml` references the published image rather than building one.

**Verify before publishing:** the binary inside each image is byte-identical to the binary in the matching tarball.

### 7. Workflows

`.gitea/workflows/gates.yml`, `on: workflow_call`, holds every blocking check: `test` (with `-Dintegration`), `test-aarch64` (`-fqemu`), `frontend`, `package` (`dist` + `verify-dist` + the asset-free size check) and `container` (build the image and run the existing smoke test from `ci.yml:219`). Moving only the three test jobs would drop the packaging and container checks precisely when they matter most.

`ci.yml` calls it on `push` and `pull_request` for **`master`**. `master` is canonical: `origin/main` is deleted, and Gitea's default branch is changed to match. Today `ci.yml:5` watches `main` while work happens on `master`, so milestone 13 has never run through CI.

`.gitea/workflows/release.yml`, `on: push: tags: ['v*']`, in this order:

1. Checkout with `fetch-depth: 0` and tags. The default shallow clone breaks ancestry checks, previous-tag lookup and changelog generation.
2. Reject any tag that is not exactly `vMAJOR.MINOR.PATCH`.
3. `git verify-tag`, requiring an annotated tag and requiring the signature's fingerprint to equal a fingerprint pinned in the workflow. A bare `verify-tag` proves only that *some* imported key signed it.
4. Assert the tag's commit is an ancestor of `origin/master`.
5. Assert no published release exists for this tag. Delete any leftover draft.
6. Assert the version is greater than the highest published release version, so a late-finishing older tag cannot move `latest` backwards.
7. Gates, blocking, via `needs:`.
8. Build the web UI; `zig build dist`; `zig build verify-dist`.
9. Extract the `CHANGELOG.md` section matching the version. **Fail when absent.**
10. buildx build and push **`:<version>` only**. Capture the index digest from `--metadata-file`, validate it as `sha256:<64 hex>`, and confirm the pushed tag resolves to that digest with exactly the two intended platforms.
11. Write the digest file, append its checksum line, verify the assembled checksum file.
12. Sign the checksum file (ruling 8). Verify the signature locally before uploading it.
13. Create the release as a **draft**; upload the assets.
14. Move `:latest` and verify it resolves to the built digest. Re-check the monotonic-version invariant here: step 6 ran before the gates, and proves nothing about which of two in-flight tags finishes last.
15. **Publish the draft last.** Publication is the one irreversible act, so it goes after everything that can still fail.

Amended after review. The original order published at 14 and moved `:latest` at 15, which deadlocks: a failure while moving `:latest` leaves a published release, and ruling 9 makes a re-run refuse a published release. Nothing could repair it. The cost of the corrected order is a short window where `:latest` serves the new image before the release page is public. That is recoverable by a re-run; the deadlock was not.

Every action in `release.yml` is pinned to a full commit SHA. `actions/checkout@v4` and `mlugg/setup-zig@v2` are mutable tags on another party's server, and a compromise upstream would run on the runner holding the signing subkey and the registry token. `ci.yml` may keep moving tags; it holds no secrets.

### 8. Signing

**Two different keys are involved, and the original ruling conflated them.**

- The **tag-signing key** is the human's. `git tag -s` uses it, and step 3 checks it. What gets pinned in `release.yml` is the **primary certificate fingerprint**, which `git verify-tag --raw` emits as the **last** field of the `VALIDSIG` line. Field 3 is whichever key actually made the signature — the signing subkey once one exists. Pinning field 3 would mean that creating the release subkey below silently blocks every future release, with an error message that reads like a forged tag. Reproduced against a real keyring during review.
- The **artifact-signing subkey** is the runner's, and it signs the checksum file. It is a dedicated GPG signing subkey of the author's existing key. A leaked subkey is revoked on its own; the identity, the commit signature history and everyone's existing trust survive.

Pinning the primary fingerprint means adding or rotating a signing subkey is a non-event for verification.

Every required secret is validated in the **guard job**, before the gates and before any registry push. Validating a fingerprint inside the signing step means a placeholder value burns an immutable version tag before it fails.

Implementation requirements: a temporary `GNUPGHOME`; assert the imported material contains no primary secret key; `--local-user <subkey-fingerprint>!` so GPG cannot fall back to another key; batch and loopback pinentry; verify the produced signature before upload; scrub `GNUPGHOME` and `DOCKER_CONFIG` and kill the agent on every exit path. The registry token is passed by `--password-stdin` into a temporary `DOCKER_CONFIG`, and `persist-credentials` is off.

Secrets: `RELEASE_GPG_SUBKEY`, `RELEASE_GPG_PASSPHRASE`, `REGISTRY_TOKEN`. The built-in `GITEA_TOKEN` cannot publish to the package registry, which is why the third exists; it can still create the release and upload assets.

**Stated honestly, in `docs/how-to/verify-a-release.md`:** the signature proves the artifact came from this pipeline and reached the operator unaltered. It does not prove the binary matches the source, because the machine that built it also held the key. Ruling 12 defers the control that closes that gap. Do not write a stronger claim than that.

### 9. Failure and recovery are specified, not improvised

- The draft is the unit of work for the *release record*. The registry is not covered by it: the version tag becomes publicly pullable at step 10. Only the release page and its assets stay hidden until step 15.
- The existence check happens **before** the push, not after. Gitea's container tags are mutable — immutability is a workflow invariant, not a registry guarantee — so a push-then-compare would already have overwritten the tag it claims to refuse. Read the existing digest with `HEAD /v2/mokhtar/nxdns/manifests/<version>`, sending an `Accept` header for the index media types and following the `401` bearer challenge with the PAT. Absent is `404`; present returns `Docker-Content-Digest`.
- A re-run deletes an existing **draft** and repeats. It refuses to touch a **published** release.
- The registry version tag is immutable, and the workflow never writes it twice. A re-run that finds the tag present pushes nothing: it adopts the pushed digest and asserts the *contents* of that image against the artifacts it just built, which is the check that matters and the only one available. Comparing a rebuilt index digest was the first design and is not implementable — buildx cannot report an index digest without pushing, and cross-machine reproducibility is deferred (ruling 12), so the rebuilt digest is expected to differ even when nothing changed. See recorded deviation 10.
- `:latest` moves last, so a failure between the image push and publication leaves the version tag pushed and `latest` untouched. That is recoverable by re-running.
- If a tag is burned — the pipeline itself is broken and the fix is on `master` — the release is abandoned and reissued as the next patch version. This is the terminal path, and it must be written down before `v0.0.1`, not discovered during it.

### 10. Release notes

`CHANGELOG.md` at the repository root, Keep a Changelog format, with an `## [Unreleased]` section maintained as work happens. The release body is that section, plus a generated appendix: `git log --oneline` since the previous tag inside a collapsed `<details>`, a compare link, the tarball hashes and the image digest.

The nullable value is "the previous reachable **published release**", never "the previous git tag". An abandoned tag from ruling 9 must not become the comparison base, and `git describe` would pick exactly that.

With no published base — the first release, and equally the case where `v0.0.1` was abandoned and `v0.0.2` becomes the first published one — the log is `git log --oneline <tag>` and the compare link is omitted. It is **not** a range with an empty left side: `..v0.0.1` resolves against `HEAD` and produces a wrong or empty appendix rather than "all history".

The `v0.0.1` section is hand-written.

### 11. Documentation

`install-with-systemd.md`, `install-with-docker.md` and `upgrade.md` lead with download-and-verify; the existing build-from-source steps move to a later section of the same page. New `docs/how-to/verify-a-release.md` gives the verification commands and the rebuild recipe.

**Milestone 13 ruling 3 applies unchanged**: every command block in `tutorial/` and `how-to/` is executed on this host by the session that writes it, or marked in-page as unverified with the reason.

Two rot hazards the drift test cannot see. `src/docs_drift_test.zig` guards only `reference/{api,configuration,cli}.md`; the how-to pages are unguarded:

- The transcripts hardcode `nxdns 0.1.0-dev` in four pages. After `v0.0.1` they are wrong and nothing fails. Use a version-neutral placeholder, and add a guard that rejects a stale literal release version in the docs.
- A download URL with the version in the path goes stale at `v0.0.2`. Use a `latest` download form if Gitea provides one, or a placeholder the reader substitutes. Probe first (ruling 13); do not guess.

Sweep the whole repository for surfaces the rewrite would otherwise miss: `README.md`, `Dockerfile` comments, `compose.yaml`, `explanation/performance-and-testing.md`, and any remaining `zig build cross` or source-only-distribution text.

### 12. Deferred, deliberately

Recorded in PLAN so they are decisions rather than oversights:

- **The reproducibility gate** — build twice in two directory paths and assert identical hashes. Deferred until `v0.0.1` proves the pipeline. Until it exists, no document may describe the build as reproducible. Do the cheap parts now regardless: pin Node to an exact patch in `ci.yml` and `web/package.json` (both float at `24` today), `gzip -n`, `LC_ALL=C`, `TZ=UTC`.
- **cosign signatures on the image** — Gitea Actions has no OIDC identity token, so keyless signing is impossible and only a key-based signature is available. Whether Gitea's registry accepts a cosign signature manifest is unverified. Spike it before committing to it.

### 13. Manual prerequisites, done before the first tag

None of these are code, and all of them block `v0.0.1`:

The order matters, and the original list got parts of it wrong. Corrected:

1. Create the artifact-signing subkey, export it with `--export-secret-subkeys`, and publish the public key to `keys.openpgp.org`. Separately, identify which key actually signs your tags — after step 1 that is normally the new signing subkey, whose **primary** fingerprint is what gets pinned.
2. Pin the primary fingerprint in `release.yml`, replacing the placeholder. The guard fails closed on the placeholder, so no tag can succeed before this.
3. **Store** the secrets: `RELEASE_GPG_SUBKEY`, `RELEASE_GPG_PASSPHRASE`, and a registry personal access token with package write scope as `REGISTRY_TOKEN`. Creating the token is not storing it.
4. `app.ini`: add `.asc` and `.sig` to `[attachment] ALLOWED_TYPES`, and raise `MAX_FILES` from 5. Restart Gitea and confirm the *effective* settings through `/api/v1/settings/attachment` before trusting them. Verified against the live instance on 2026-08-05: `max_size` is 100 MB (ample), `max_files` is 5 (this release has exactly five assets, no headroom), and `.asc` is absent, so the signature is rejected today. The implementation names the assets `SHA256SUMS.txt`, `SHA256SUMS.txt.asc` and `IMAGE-DIGEST.txt`, so `.txt` and `.asc` are the two extensions that must be allowed.
5. **Probe: does the runner support `docker buildx` with the `docker-container` driver, and can it extract a foreign-platform image?** The old `docker` job only proved plain `docker build`, and the arm64 image-versus-tarball check needs more than that.
6. Land the workflows on `master` **while `main` still exists**, and let the reusable-gates workflow run green once, so Gitea emits its actual status-check context names.
7. Configure branch protection on `master` using **those observed names**. The reusable-workflow refactor changes them; keeping the old required checks can make merges impossible or leave the intended gates non-required.
8. Change Gitea's default branch to `master`.
9. Delete `origin/main` **last** — Gitea refuses to delete the default branch.
10. Merge the licence, changelog and `build.zig.zon` version commit, and let `master` go green.
11. Dry-run the release path. A tag-triggered workflow cannot be exercised without pushing *some* tag, so use a disposable tag or a scratch repository. **Never use `v0.0.1` as the dry run** and then expect to reuse it: tags are never moved (ruling 2), so a burned dry-run tag is spent.

## Sessions

S1–S5 run in parallel. S1 and S2 share one interface, fixed here so neither blocks: S2 owns everything under `licenses/` and the licence texts; S1 owns the build step that assembles them into `THIRD-PARTY-NOTICES`. The contract is `zig build dist -Dversion-string=X -Dgit-commit=Y -Dweb-dist=web/dist` writing to `zig-out/dist/`.

### Session S1: build system

Owns `build.zig`. Rulings 2, 4, 5. Deletes `cross`, adds `dist` and `verify-dist`, moves the CI shell asserts into the build graph.

### Session S2: licence, notices, changelog

Owns `LICENSE`, `licenses/`, `CHANGELOG.md`, `README.md` (notice and links). Ruling 3, ruling 10. Compiles the inventory by auditing what the artifacts actually contain, and writes the CI drift guard for the dependency sets.

### Session S3: container

Owns `deploy/docker/*`, `deploy/systemd/*` (the `sysusers.conf` rename). Ruling 6.

### Session S4: workflows

Owns `.gitea/workflows/*`. Rulings 7, 8, 9. Must not edit `build.zig` — it consumes the step contract above.

### Session S5: documentation

Owns `docs/**`. Ruling 11, and milestone 13 ruling 3.

### Orchestrator

`PLAN.md` (ruling 1), this spec, deletion of `origin/main`, the manual prerequisites in ruling 13, and the `v0.0.1` tag.

## Recorded (implementation)

Accepted deviations and corrections from integration. The five rulings amended above (7, 8, 9, 10, 13, plus the option-inheritance note in 5 and the scoping of the last acceptance line) were all wrong as first written; each amendment says what it replaced and why.

1. **`build.zig.zon` said `0.1.0`.** It traced to the first build-baseline commit — a scaffold default never bumped. Ruling 2 makes `verify-dist` assert it against the version under build, so the CI packaging gate could never have passed. Set to `0.0.1`, matching the first tag.
2. **Assets carry a `.txt` extension**: `SHA256SUMS.txt`, `SHA256SUMS.txt.asc`, `IMAGE-DIGEST.txt`. This resolves ruling 13's extensionless-asset probe by construction — `.txt` is already in the live `ALLOWED_TYPES`, so only `.asc` still has to be added.
3. **`INSTALL.md` was added** at the repository root. `dist` hard-requires it in the staged payload and `verify-dist` asserts its mode.
4. **The notices preamble moved to `licenses/preamble.txt`.** The tarball and the image ship different sets, and the hardcoded preamble scoped itself to "a single static executable" — which the image is not. The image additionally redistributes Alpine's Mozilla CA bundle (`MPL-2.0 AND MIT`, 179,359 bytes); the tarball does not.
5. **Vite joined Tailwind in the inventory.** Both are devDependencies whose generated output ships inside the binary. The original inventory applied that rule to one of them and stopped.
6. **Node is pinned to `24.19.0`** in `gates.yml`, `release.yml` and `web/package.json` — the exact-patch pin ruling 12 asks for.
7. **`gates.yml` is SHA-pinned too.** Ruling 7 pinned only `release.yml`, but `release.yml` calls `gates.yml`, and those jobs share the runner host and docker daemon with the job holding the signing subkey. `live-tls.yml` keeps moving tags; it references no secret.
8. **Determinism measured better than claimed.** Two `dist` runs with separate cache directories and separate prefixes produced byte-identical tarballs — a genuine recompile, not a cache replay. The documentation still claims only same-directory determinism, because ruling 12's gate does not exist yet and an unguarded property decays. The stronger result is recorded here, not promised to operators.
9. **Test count moved from 1461 to 1481**: twelve licence-drift tests, then eight more from the review pass below. Skip counts are unchanged.

The rest came out of an adversarial review of the finished implementation. Each was reproduced before it was fixed.

10. **The version tag was pushed before the immutability check ran.** The first implementation pushed `:$VERSION` and then compared the resulting digest with the pre-push one — a check that reports a violation it just caused. On a re-run that produced different bytes the tag was already overwritten and the original image lost. Replaced by the probe-then-adopt design ruling 9 now describes: a real `HEAD /v2/…/manifests/<version>`, and an existing tag is adopted rather than rebuilt. Exercised against a fake registry covering `404`, `200`, the `401` bearer challenge, `500`, and a `200` with no `Docker-Content-Digest`.
11. **The guard proved nothing about the signing key.** It checked that `RELEASE_GPG_SUBKEY` and `RELEASE_GPG_PASSPHRASE` were non-empty. A public-only export, an export missing the pinned subkey, and a placeholder passphrase all passed it and failed for the first time in the signing step — after the image push. The guard now imports the material, asserts the pinned subkey is present as a secret key, and signs a throwaway file with the passphrase.
12. **`jq … | grep … || true` swallowed a malformed API response.** The `|| true` exists so grep's no-match is not fatal; it covers the whole pipeline, so a `200` carrying a JSON object instead of an array read as "no published releases" — the one wrong answer that moves `:latest` backwards. The payload is now type-checked before it is read.
13. **The monotonic re-check was not the concurrency backstop it claimed to be.** Two concurrent releases are both drafts while they run, so neither appears in the other's published list and both pass. The workflow `concurrency:` group is the only thing that serialises them. A second check was added that does close it: `:latest`'s own version label is read from the registry immediately before the tag moves, so the invariant is checked against the state being mutated.
14. **Publication could deadlock the tag.** A `PATCH` that Gitea committed but whose response was lost left the release public and the step failed, after which the guard refused every re-run. The step now re-reads the release and treats an already-published one as success.
15. **The secret-scrub backstop killed the wrong gpg-agent.** `gpgconf --kill` acts on the agent of the `GNUPGHOME` it is pointed at, and the bare call killed the runner's default agent while every leaked temporary home's agent kept running with the key cached. It now kills each home in its own home, and the guard job has the same `if: always()` backstop the publish job had.
16. **The Zig dependency guard could be silenced by pasting.** Unlike the npm half, nothing tied `build.zig.zon` back to the inventory, so a Mbed TLS bump plus the suggested identity paste left the notices claiming the old version. Each dependency now has to appear in the inventory at the version its URL names. The Zig toolchain version and the entries vendored inside Mbed TLS are checked the same way.
17. **The container base image was not an input to any guard.** It is the source of the CA bundle the image redistributes. `deploy/docker/Dockerfile` is now embedded in the `licenses_files` module, its digest-pinned `FROM` is a recorded identity section, and the CA bundle entry must name the Alpine release that `FROM` pins.
18. **A tree-shaken package that started shipping would have gone unnoticed.** `cookie-es`, `isbot`, `seroval` and `seroval-plugins` are in the lockfile closure and in no shipped byte. If application code imported one, no lockfile, version or dependency set would change — only the bundle. The frontend gate now recomputes the set of packages in `web/dist` from a `--sourcemap` build and diffs it against a recorded section, and the drift test requires every name in that section to be inventoried and refuses one that is still listed as not shipped.
19. **The recorded npm licence token was parsed and discarded**, so a package that relicensed passed as long as its version had not moved. Shipped packages must now all carry the licence the inventory's texts assume; `npm_not_shipped` is exempt, and `isbot` is Unlicense.
20. **The full-text licence checks were marker probes.** They prove the right document is present but survive most of it being deleted. The Apache-2.0 and MPL-2.0 texts are now pinned by SHA-256.
21. **Zig's compiler-rt contains code ported from LLVM's.** Zig's `LICENSE` is bare MIT naming only "Zig contributors". Reviewed: the ports carry Apache-2.0 WITH LLVM-exception, and that exception waives Apache §4(a), §4(b) and §4(d) for portions embedded in object form — the only form nxdns ships — so no further notice is owed. Recorded in the Zig inventory note rather than left as an unexamined gap.
22. **`npx` was replaced by the installed binary** in the new frontend gate. `npx` downloads a package it cannot find locally, so a wrong working directory would have turned a licence check into an unpinned fetch. Reproduced: it fetched `vite@8.2.0` over the pinned `8.1.5`.

23. **`actions/checkout` destroys the annotated tag object.** Found by the first live dry run, not by review: on a tag ref, checkout fetches the *commit* SHA into `refs/tags/<tag>`, so the signed tag reads as lightweight and the guard refuses it as unannotated. Both jobs that read the tag object — signature verification in the guard, the tagger date in the publish job — now force- refetch `refs/tags/$TAG` from origin first. The same run also proved the fail-closed secret guard for real: the first dry-run attempt ran with no secrets configured (they were on the wrong repository) and stopped in the guard with nothing built or pushed.

24. **Publication orchestration moved out of workflow shell into `tools/release.zig`.** Ruling 5 already moved the packaging asserts out of CI shell for one reason — "checks that only exist inside a workflow file are the brittleness this exists to remove" — and the release job was the larger half of the same problem, left in place. Three live failures came out of it, and each was found by executing the workflow, which is the most expensive place to find anything: `actions/checkout` replacing the annotated tag object (deviation 23), the refetch that fixed it having no credentials because `persist-credentials` is off, and the multiline armored subkey escaping the runner's log masker, which masks per line.

    Twelve subcommands, one per step group: `guard-tag`, `guard-ancestry`,
    `guard-releases`, `resolve`, `changelog`, `image`,
    `verify-image-binaries`, `sign`, `draft`, `latest`, `publish`, `scrub`. Every
    behaviour recorded in deviations 10 to 15 and 23 is carried over unchanged —
    probe-adopt, the VALIDSIG last field, the subkey-only import and signing
    probe, the array-shape guard on the releases payload, the `:latest` label
    read, the publish re-read, the per-home `gpgconf --kill`, the tag refetch.
    What is new is that the semver ordering, VALIDSIG field selection, challenge
    parsing, changelog extraction, checksum-line parsing, colon-format parsing
    and payload-shape guard are 25 unit tests in `zig build test` rather than
    shell that only ever runs on a tag push. `release.yml` keeps the triggers,
    the concurrency group, the job graph, the SHA pins, the two pinned
    fingerprints and the fail-closed secret presence check — which stays as
    shell, deliberately, so that it runs before the tool is even compiled.

    The same reasoning applies to the `jq` pipeline of the bundled-package gate,
    which moved to `web/scripts/bundledPackages.mjs` with its own vitest
    coverage and an `npm run assert-bundled` entry point.

    **Secret contract change:** `RELEASE_GPG_SUBKEY` keeps its name but now
    holds `base64 -w0` of the armored
    `--export-secret-subkeys` output rather than the armored text. Manual
    prerequisite 1 and 3 change accordingly. The tool decodes it in memory and
    writes it to a mode-600 file inside the temporary `GNUPGHOME`. A single-line
    secret is one the masker can actually mask.

25. **The first signing subkey was leaked into a job log and rotated.** Dry-run attempt 3 failed inside the credential-less refetch, and the runner printed the failing step's env block; the multiline armored `RELEASE_GPG_SUBKEY` escaped the per-line masker while the single-line passphrase was masked. Exposure: the passphrase-protected secret subkey only — the passphrase and the primary key were never on the runner. Response: both runs that ever saw the secret were deleted (verified 404 via the API and absent from `actions_log` on disk), subkey `B281CECC…` was revoked with the primary, and its replacement `019D00DF…` is the pinned `RELEASE_SIGNING_FPR`. The base64 contract in deviation 24 is the preventive half of this record.

26. **The image is named by the public registry host, never the server URL.** Attempt 4 reached the registry and failed at `docker login gitea:3000`: inside the cluster `GITHUB_SERVER_URL` is `http://gitea:3000`, docker refuses plain-http registries, and an image named `gitea:3000/…` would be unpullable from anywhere that matters — a wrong name that would have been written into the released `IMAGE-DIGEST.txt`. `release.yml` now pins `REGISTRY_HOST: git.mial.net`; the tool uses it for docker and image naming, and keeps the internal URL for the manifest probe (same registry, no TLS dependency in the tool). The old shell had the identical latent bug; no run ever reached it.

### Not verified, and why

- **The workflows' validation history.** Before any live run, `release.yml` and `gates.yml` were validated by YAML parse and `bash -n`, plus two steps lifted out and run directly: the registry probe against a fake registry (five response shapes) and the bundled-package check against the real `web/` build, proven able to fail. The live dry run then superseded this: attempt 5 published `v0.0.0` end to end — guard, gates, image push to both platforms, binary-identity assertion, signing, draft, `:latest`, publication — and the assets verified from a clean directory (checksums OK, signature good under the rotated subkey). The throwaway release, tag and registry versions were deleted afterwards. Everything else that talks to the registry or the Gitea API — `buildx build --push`, `imagetools`, draft creation, asset upload, publication, the adopt-an-existing-tag path — is unexercised.
- **`RELEASE_SIGNING_FPR` is still the placeholder.** By design: the guard fails closed on it. It also means the release workflow cannot succeed as committed until manual prerequisite 2 is done.
- **The aarch64 binary was never executed.** No `qemu-aarch64` on the build host, so `verify-dist`'s aarch64 version check legitimately skips and `test-aarch64` could not run. The binary is checked statically: ELF class, no `PT_INTERP`, no `DT_NEEDED`, size.
- **The multi-architecture image build is unverified.** Only the native amd64 image was built and run.
- **`shellcheck` was not run** — not installed on the build host.
- **The local Node is 24.14.1, not the pinned 24.19.0.** There is no `.npmrc`, so `engines` does not hard-fail, and the frontend gates ran under the older patch.

## Acceptance (milestone complete)

- [x] `PLAN.md` §2.2 amended; the build-date line and the byte limits corrected.
      §3.15 now describes the single gate set, and a new §20 records the
      publication model.
- [x] `zig build dist -Dversion-string=0.0.1 -Dweb-dist=web/dist` produces two
      tarballs and a checksum file, and fails without `-Dweb-dist`.
- [x] `zig build verify-dist`, **given the same options as `dist`**, passes and
      was proven able to fail: an oversized
      binary, a dynamically linked binary, a mismatched `build.zig.zon` version
      and a wrong archive mode each produce a named failure. Proven by repacking
      tarballs with each defect: `binary-size`, `elf`, `zon-version` and
      `archive-mode` each fired and the run exited 1. The oversize proof used a
      genuinely oversized binary for `binary-size` and a lowered budget for
      `asset-free-size`.
- [x] Two runs of `zig build dist` on the same commit produce byte-identical
      tarballs **in the same directory**. (Cross-directory reproducibility is
      ruling 12 and is not claimed here.)
- [x] The image builds for both platforms with no qemu, carries `/LICENSE` and
      `/THIRD-PARTY-NOTICES` and the OCI labels, and its binaries are
      byte-identical to the tarball binaries. The v0.0.1 run built and pushed
      both platforms on the runner; the published index lists exactly
      `linux/amd64 linux/arm64`, and `release verify-image-binaries` compared
      both binaries against the tarballs before publication.
- [x] `gates.yml` runs from both `ci.yml` and `release.yml`; `ci.yml` triggers
      on `master`; `origin/main` is gone. Proven live: pushes to `master` run
      the gates through `ci.yml`, and release runs 484-493 ran them through
      `release.yml`.
- [x] `THIRD-PARTY-NOTICES` covers musl, the Zig runtime, SQLite, Mbed TLS with
      its Apache-2.0 selection line and full text, Everest, p256-m and the web
      runtime closure. The dependency drift guard was proven able to fail:
      removing an inventory entry, staling a dependency version, staling the Zig
      version, changing the base image digest, editing a pinned licence text and
      dropping a package from the recorded bundle each produce a named failure.
- [x] A dry run of `release.yml` completes with publication disabled. Done with
      a disposable published tag instead: publication cannot be disabled without
      forking the flow it is supposed to prove, so `v0.0.0` ran the real path
      end to end — five assets, verifying checksums and signature, a
      multi-architecture image — and was then deleted (release, git tag, both
      registry versions). Five attempts; the failures and their fixes are
      deviations 23-26.
- [x] `v0.0.1` is published: five assets, a verifying signature, and an image at
      `git.mial.net/mokhtar/nxdns:0.0.1` and `:latest`. Run 493, all jobs green
      on the first attempt after the dry-run fixes.
- [x] `docs/how-to/verify-a-release.md` was followed end to end against the
      published release, from a clean directory, on this host, with a clean
      `GNUPGHOME` holding only the key fetched from keys.openpgp.org. Every
      command on the page passed: the `releases/latest` redirect printed
      `0.0.1`, both tarball downloads and the `latest` alias worked (and
      GitHub's spelling answered 404 as documented), the signature verified
      with matching primary and subkey fingerprints, `sha256sum -c` said OK for
      all three files, the tarball layout and modes matched, `nxdns version`
      printed the tag's commit, the tag digest equalled `IMAGE-DIGEST.txt`, the
      platform list was exactly `linux/amd64 linux/arm64`, and the binary
      copied out of the pulled-by-digest image hashed identical to the tarball
      binary.
- [x] No `zig build cross` or source-only-distribution text remains on any
      **active** surface: `build.zig`, the workflows, `deploy/`, `README.md` and
      `docs/`. Historical milestone specs and `TECH_DEBT.md` keep their text —
      this spec contains the string itself, so "anywhere" was never satisfiable.

## Anti-requirements

- No GoReleaser, no nfpm, no `.deb` or `.rpm`, no Homebrew, no AUR.
- No cosign, no SBOM, no SLSA provenance, no in-toto attestations.
- No `:edge` image, no rolling dev build, no pre-release tags, no floating `0.0` or `0` image tags.
- No third architecture, no non-Linux target, no glibc build.
- No project website, no documentation site generator.
- No changelog generation from commit messages, and no Conventional Commits.
- No release automation that cannot be run from a laptop.
