milestone 14 spec: build, package and publish releases

specs/milestone-14.md

@@ -0,0 +1,430 @@
+# Milestone 14: build, package and publish releases
+
+Goal: turn a signed git tag into a published, verifiable release — two static
+musl tarballs and one multi-architecture container image on the self-hosted
+Gitea at `git.mial.net`, with a licence, third-party notices, a changelog, a
+detached signature, and documentation that leads with download-and-verify.
+
+First tag: `v0.0.1`.
+
+## Rulings (binding)
+
+### 1. PLAN.md is amended first
+
+`PLAN.md:41` lists "Prebuilt binaries / published Docker images / project
+website" under §2.2 *Out of Scope (permanent scope decisions, not deferrals)*.
+`AGENTS.md` makes PLAN the source of truth, so this milestone is invalid until
+that line is amended. The orchestrator edits PLAN before any session starts:
+
+- §2.2: remove prebuilt binaries and published container images. **The project
+  website stays out of scope.**
+- Add a §on publication: tag-triggered releases, the artifact set, the signing
+  model, and the two deferrals in ruling 12.
+- `PLAN.md:571`: drop "build date" from the `nxdns version` line. The version
+  string and the git commit identify a build exactly, and a date is one more
+  input that a reproducible build would have to pin. `src/version.zig` is
+  already correct; PLAN was wrong.
+- `PLAN.md:638`: state exact byte limits — 15,728,640 with embedded assets,
+  10,485,760 without — resolving milestone-13 discrepancy 9 in favour of what
+  CI already asserts.
+
+### 2. The version lives in the tag, and in exactly one other place
+
+The tag is authoritative. `v0.0.1` means `-Dversion-string=0.0.1`;
+`-Dgit-commit` is the tag's peeled commit.
+
+`build.zig.zon:3` holds `.version`, which Zig requires and nothing reads. It is
+bumped in the commit before each tag, and `verify-dist` fails when it disagrees
+with the version under build. Two strings, one assert, no third copy anywhere.
+
+Tags are **never moved**. A tag that produced a bad release is abandoned; the
+fix ships as the next patch version. Only `vMAJOR.MINOR.PATCH` is accepted —
+the release workflow rejects any tag carrying a pre-release suffix.
+
+### 3. Licence and third-party notices
+
+`LICENSE` holds the English text of EUPL-1.2 verbatim, with
+`Copyright (c) 2026 Mokhtar Mial`. `README.md` gains a short notice naming the
+licence and the SPDX identifier `EUPL-1.2`. **No per-file SPDX headers.**
+
+`THIRD-PARTY-NOTICES` is assembled by `zig build dist` from a committed,
+reviewed inventory under `licenses/`. It is not scraped from the dependency
+tree at build time: a generated notices file that nobody reads rots silently
+into a false statement.
+
+The inventory must cover everything the shipped artifacts actually contain, not
+the direct dependency list:
+
+- **musl libc** (MIT) — statically linked into every binary.
+- **Zig standard library and compiler-rt** (MIT) — likewise.
+- **SQLite 3.53.4** — public domain, no obligation, listed for completeness.
+- **Mbed TLS 3.6.7** — with an explicit line recording that it is taken under
+  the Apache-2.0 option of its dual `Apache-2.0 OR GPL-2.0-or-later` licence,
+  followed by the **full Apache-2.0 text**. Naming the choice is good practice;
+  shipping the text is the actual obligation.
+- **Project Everest and p256-m** — compiled in at `build.zig:394` even though
+  the stock config leaves both drivers disabled.
+- **The web bundle's runtime closure** — the transitive set, not the four direct
+  entries in `web/package.json`. Tailwind is a devDependency whose generated CSS
+  ships, so "production dependencies" understates it.
+
+A CI guard records the identity of the dependency sets (`build.zig.zon`
+dependencies, and the npm closure) and fails when either changes without a
+matching change under `licenses/`. It detects drift; it does not derive the
+inventory.
+
+The **container image carries `/LICENSE` and `/THIRD-PARTY-NOTICES` too**.
+Distributing the image is distribution, and the obligations do not live in the
+tarball.
+
+### 4. `zig build dist` replaces `zig build cross`
+
+The `cross` step is deleted. `dist` is the single command that produces
+everything releasable, and it runs on a laptop exactly as it runs on the runner.
+
+Inputs: `-Dversion-string` (required — no default), `-Dgit-commit`,
+`-Dweb-dist`, `-Doptimize=ReleaseSafe`.
+
+**`dist` fails when `-Dweb-dist` resolves to `web/dist-placeholder`.** The
+default at `build.zig:24` is the placeholder, so a release built without the
+flag would silently ship a placeholder admin page. There is no override flag;
+an escape hatch here is a foot-gun with a safety label on it.
+
+Per triple (`x86_64-linux-musl`, `aarch64-linux-musl`), `dist` builds
+ReleaseSafe with `.linkage = .static` and `.strip = true`. Stripping uses
+`std.Build.Module.strip` (`-fstrip`, `Module.zig:545`), which removes the
+`objcopy` and `binutils-aarch64-linux-gnu` dependency from the runner.
+
+It stages one directory per triple, `nxdns-<version>-<triple>/`, holding:
+
+| File | Mode |
+|---|---|
+| `nxdns` | 0755 |
+| `nxdns.service` | 0644 |
+| `nxdns.conf` | 0644 |
+| `LICENSE` | 0644 |
+| `THIRD-PARTY-NOTICES` | 0644 |
+| `INSTALL.md` | 0644 |
+
+`deploy/systemd/sysusers.conf` is renamed to `nxdns.conf` — the name it is
+installed under at `/usr/lib/sysusers.d/nxdns.conf`. A file that changes name
+during install is a step an operator can get wrong.
+
+Archiving runs as two `b.addSystemCommand` steps, never one:
+
+```
+tar --format=gnu --sort=name --mtime=@0 --owner=0 --group=0 \
+    --numeric-owner -cf <name>.tar <dir>
+gzip -n -9 <name>.tar
+```
+
+`b.addSystemCommand` executes argv directly and does not interpret `|`, and a
+shell wrapper without `pipefail` would report only `gzip`'s exit status while a
+failed `tar` passed silently. `gzip -n` is required because `--mtime=@0`
+normalises the tar member times but not the timestamp gzip writes into its own
+header. The environment sets `LC_ALL=C` and `TZ=UTC`.
+
+Both steps declare the staging directory as a build input and the archive as an
+output, so Zig's cache cannot serve a stale artifact.
+
+`dist` also emits a checksum file covering **only the two tarballs**. It cannot
+cover the image: the digest does not exist until buildx has pushed, which
+happens later and elsewhere. The release job appends that line (ruling 7).
+
+### 5. `zig build verify-dist` asserts what CI shell asserts today
+
+The 50 lines of shell at `ci.yml:129` and `:168` move into a build step, so a
+developer can run the release checks on a laptop. Logic that only runs in CI is
+the brittleness this milestone exists to remove.
+
+It operates on the **extracted** archive, not the staging directory:
+
+- ELF header: correct `e_machine` per triple, **no `PT_INTERP`, no `DT_NEEDED`**.
+  Matching the string `statically linked` from `file(1)` is not a static-linkage
+  test.
+- Stripped binary ≤ 15,728,640 bytes.
+- Archive layout: exactly one top-level directory, the exact file allowlist from
+  ruling 4, expected modes, no symlinks, no path traversal.
+- `nxdns version` prints the version under build and the git commit. Native
+  architecture only — the aarch64 binary needs qemu and is skipped without
+  `-fqemu`.
+- `build.zig.zon` `.version` equals the version under build.
+
+The asset-free budget (10,485,760 bytes) gets **its own build against a
+generated empty assets directory**, not against the placeholder. Ruling 4 makes
+the placeholder unbuildable, and re-admitting it through a back door for one
+size check would defeat the point.
+
+### 6. Container image
+
+`deploy/docker/Dockerfile`:
+
+- Pin the base by digest:
+  `alpine:3.22@sha256:14358309a308569c32bdc37e2e0e9694be33a9d99e68afb0f5ff33cc1f695dce`.
+- Add `--platform=$BUILDPLATFORM` to the builder stage. It only copies files, so
+  pinning it to the build host means the arm64 image needs no qemu.
+- **Delete `apk add --no-cache ca-certificates` (`Dockerfile:15`).** Verified:
+  the Alpine base already ships `/etc/ssl/certs/ca-certificates.crt` (179,359
+  bytes) from `ca-certificates-bundle`. Removing it removes the only network
+  fetch in the image build.
+- Keep the `mkdir` and `chown 65532` — `/var/lib/nxdns` must exist with that
+  ownership so Docker copies it onto a fresh named volume.
+- Add `/LICENSE` and `/THIRD-PARTY-NOTICES` (ruling 3).
+- OCI labels: `source`, `revision`, `version`, `licenses=EUPL-1.2`, `created`,
+  `title`, `description`.
+
+`deploy/docker/.dockerignore` is renamed to `deploy/docker/Dockerfile.dockerignore`.
+The build context is the repository root, so Docker never reads the current
+path, and the whole worktree — `.git`, `node_modules`, `zig-cache` — is being
+sent to the daemon today.
+
+buildx runs with `--provenance=false --sbom=false`. Recent buildx adds
+provenance attestations by default, which create `unknown/unknown` platform
+entries and change the index digest; Gitea's OCI 1.1 support is unverified
+(go-gitea#25846) and this is not the milestone to find out. `SOURCE_DATE_EPOCH`
+comes from the tag date.
+
+`deploy/docker/compose.yaml` references the published image rather than
+building one.
+
+**Verify before publishing:** the binary inside each image is byte-identical to
+the binary in the matching tarball.
+
+### 7. Workflows
+
+`.gitea/workflows/gates.yml`, `on: workflow_call`, holds every blocking check:
+`test` (with `-Dintegration`), `test-aarch64` (`-fqemu`), `frontend`, `package`
+(`dist` + `verify-dist` + the asset-free size check) and `container` (build the
+image and run the existing smoke test from `ci.yml:219`). Moving only the three
+test jobs would drop the packaging and container checks precisely when they
+matter most.
+
+`ci.yml` calls it on `push` and `pull_request` for **`master`**. `master` is
+canonical: `origin/main` is deleted, and Gitea's default branch is changed to
+match. Today `ci.yml:5` watches `main` while work happens on `master`, so
+milestone 13 has never run through CI.
+
+`.gitea/workflows/release.yml`, `on: push: tags: ['v*']`, in this order:
+
+1. Checkout with `fetch-depth: 0` and tags. The default shallow clone breaks
+   ancestry checks, previous-tag lookup and changelog generation.
+2. Reject any tag that is not exactly `vMAJOR.MINOR.PATCH`.
+3. `git verify-tag`, requiring an annotated tag and requiring the signature's
+   fingerprint to equal a fingerprint pinned in the workflow. A bare
+   `verify-tag` proves only that *some* imported key signed it.
+4. Assert the tag's commit is an ancestor of `origin/master`.
+5. Assert no published release exists for this tag. Delete any leftover draft.
+6. Assert the version is greater than the highest published release version, so
+   a late-finishing older tag cannot move `latest` backwards.
+7. Gates, blocking, via `needs:`.
+8. Build the web UI; `zig build dist`; `zig build verify-dist`.
+9. Extract the `CHANGELOG.md` section matching the version. **Fail when absent.**
+10. buildx build and push **`:<version>` only**. Capture the index digest from
+    `--metadata-file`, validate it as `sha256:<64 hex>`, and confirm the pushed
+    tag resolves to that digest with exactly the two intended platforms.
+11. Write the digest file, append its checksum line, verify the assembled
+    checksum file.
+12. Sign the checksum file (ruling 8). Verify the signature locally before
+    uploading it.
+13. Create the release as a **draft**; upload the assets.
+14. Publish the draft.
+15. **Move `:latest` last.**
+
+Every action in `release.yml` is pinned to a full commit SHA.
+`actions/checkout@v4` and `mlugg/setup-zig@v2` are mutable tags on another
+party's server, and a compromise upstream would run on the runner holding the
+signing subkey and the registry token. `ci.yml` may keep moving tags; it holds
+no secrets.
+
+### 8. Signing
+
+Automated on the runner, with a dedicated GPG **signing subkey** of the author's
+existing commit-signing key. A leaked subkey is revoked on its own; the
+identity, the commit signature history and everyone's existing trust survive.
+
+Implementation requirements: a temporary `GNUPGHOME`; assert the imported
+material contains no primary secret key; `--local-user <subkey-fingerprint>!`
+so GPG cannot fall back to another key; batch and loopback pinentry; verify the
+produced signature before upload; scrub `GNUPGHOME` and `DOCKER_CONFIG` and kill
+the agent on every exit path. The registry token is passed by
+`--password-stdin` into a temporary `DOCKER_CONFIG`, and `persist-credentials`
+is off.
+
+Secrets: `RELEASE_GPG_SUBKEY`, `RELEASE_GPG_PASSPHRASE`, `REGISTRY_TOKEN`. The
+built-in `GITEA_TOKEN` cannot publish to the package registry, which is why the
+third exists; it can still create the release and upload assets.
+
+**Stated honestly, in `docs/how-to/verify-a-release.md`:** the signature proves
+the artifact came from this pipeline and reached the operator unaltered. It does
+not prove the binary matches the source, because the machine that built it also
+held the key. Ruling 12 defers the control that closes that gap. Do not write a
+stronger claim than that.
+
+### 9. Failure and recovery are specified, not improvised
+
+- The draft is the unit of work. Nothing is visible until step 14.
+- A re-run deletes an existing **draft** and repeats. It refuses to touch a
+  **published** release.
+- The registry version tag is immutable. A re-run accepts an existing tag whose
+  digest matches exactly what it just built, and refuses a differing one.
+- `:latest` moves last, so a failure between the image push and publication
+  leaves the version tag pushed and `latest` untouched. That is recoverable by
+  re-running.
+- If a tag is burned — the pipeline itself is broken and the fix is on `master` —
+  the release is abandoned and reissued as the next patch version. This is the
+  terminal path, and it must be written down before `v0.0.1`, not discovered
+  during it.
+
+### 10. Release notes
+
+`CHANGELOG.md` at the repository root, Keep a Changelog format, with an
+`## [Unreleased]` section maintained as work happens. The release body is that
+section, plus a generated appendix: `git log --oneline` since the previous tag
+inside a collapsed `<details>`, a compare link, the tarball hashes and the image
+digest.
+
+First release: no previous tag exists, `git describe` fails, and the compare
+link is omitted. Later releases select the highest reachable **published** plain
+release, not merely the nearest git tag — an abandoned tag from ruling 9 must
+not become the comparison base.
+
+The `v0.0.1` section is hand-written.
+
+### 11. Documentation
+
+`install-with-systemd.md`, `install-with-docker.md` and `upgrade.md` lead with
+download-and-verify; the existing build-from-source steps move to a later
+section of the same page. New `docs/how-to/verify-a-release.md` gives the
+verification commands and the rebuild recipe.
+
+**Milestone 13 ruling 3 applies unchanged**: every command block in `tutorial/`
+and `how-to/` is executed on this host by the session that writes it, or marked
+in-page as unverified with the reason.
+
+Two rot hazards the drift test cannot see. `src/docs_drift_test.zig` guards only
+`reference/{api,configuration,cli}.md`; the how-to pages are unguarded:
+
+- The transcripts hardcode `nxdns 0.1.0-dev` in four pages. After `v0.0.1` they
+  are wrong and nothing fails. Use a version-neutral placeholder, and add a
+  guard that rejects a stale literal release version in the docs.
+- A download URL with the version in the path goes stale at `v0.0.2`. Use a
+  `latest` download form if Gitea provides one, or a placeholder the reader
+  substitutes. Probe first (ruling 13); do not guess.
+
+Sweep the whole repository for surfaces the rewrite would otherwise miss:
+`README.md`, `Dockerfile` comments, `compose.yaml`, `explanation/performance-and-testing.md`,
+and any remaining `zig build cross` or source-only-distribution text.
+
+### 12. Deferred, deliberately
+
+Recorded in PLAN so they are decisions rather than oversights:
+
+- **The reproducibility gate** — build twice in two directory paths and assert
+  identical hashes. Deferred until `v0.0.1` proves the pipeline. Until it exists,
+  no document may describe the build as reproducible. Do the cheap parts now
+  regardless: pin Node to an exact patch in `ci.yml` and `web/package.json`
+  (both float at `24` today), `gzip -n`, `LC_ALL=C`, `TZ=UTC`.
+- **cosign signatures on the image** — Gitea Actions has no OIDC identity token,
+  so keyless signing is impossible and only a key-based signature is available.
+  Whether Gitea's registry accepts a cosign signature manifest is unverified.
+  Spike it before committing to it.
+
+### 13. Manual prerequisites, done before the first tag
+
+None of these are code, and all of them block `v0.0.1`:
+
+1. Create the GPG signing subkey, export it with `--export-secret-subkeys`,
+   publish the public key to `keys.openpgp.org`, and store the armored subkey
+   and its passphrase as secrets.
+2. Create the registry personal access token with package write scope.
+3. `app.ini`: add `.asc` and `.sig` to `[attachment] ALLOWED_TYPES`, and raise
+   `MAX_FILES` from 5. Verified against the live instance: `max_size` is 100 MB
+   (ample), `max_files` is 5 (this release has exactly five assets, no
+   headroom), and `.asc` is absent, so the signature is rejected today.
+4. **Probe: can an extensionless asset be uploaded?** `ALLOWED_TYPES` is
+   extension-based, and `SHA256SUMS` and `IMAGE-DIGEST` have none. If the API
+   rejects them, they become `SHA256SUMS.txt`, `SHA256SUMS.txt.asc` and
+   `IMAGE-DIGEST.txt`. Resolve by trying it against a scratch repository, not by
+   assuming.
+5. **Probe: does the runner support `docker buildx` with the `docker-container`
+   driver?** The current `docker` job only proves plain `docker build`.
+6. Push milestone 13 to `master`, set Gitea's default branch to `master`, then
+   delete `origin/main` last.
+7. Merge the licence, changelog and `build.zig.zon` version commit, and let it
+   pass on `master`, before creating the tag.
+8. Dry-run the whole release workflow with publication disabled.
+
+## Sessions
+
+S1–S5 run in parallel. S1 and S2 share one interface, fixed here so neither
+blocks: S2 owns everything under `licenses/` and the licence texts; S1 owns the
+build step that assembles them into `THIRD-PARTY-NOTICES`. The contract is
+`zig build dist -Dversion-string=X -Dgit-commit=Y -Dweb-dist=web/dist` writing
+to `zig-out/dist/`.
+
+### Session S1: build system
+
+Owns `build.zig`. Rulings 2, 4, 5. Deletes `cross`, adds `dist` and
+`verify-dist`, moves the CI shell asserts into the build graph.
+
+### Session S2: licence, notices, changelog
+
+Owns `LICENSE`, `licenses/`, `CHANGELOG.md`, `README.md` (notice and links).
+Ruling 3, ruling 10. Compiles the inventory by auditing what the artifacts
+actually contain, and writes the CI drift guard for the dependency sets.
+
+### Session S3: container
+
+Owns `deploy/docker/*`, `deploy/systemd/*` (the `sysusers.conf` rename).
+Ruling 6.
+
+### Session S4: workflows
+
+Owns `.gitea/workflows/*`. Rulings 7, 8, 9. Must not edit `build.zig` — it
+consumes the step contract above.
+
+### Session S5: documentation
+
+Owns `docs/**`. Ruling 11, and milestone 13 ruling 3.
+
+### Orchestrator
+
+`PLAN.md` (ruling 1), this spec, deletion of `origin/main`, the manual
+prerequisites in ruling 13, and the `v0.0.1` tag.
+
+## Acceptance (milestone complete)
+
+- [ ] `PLAN.md` §2.2 amended; the build-date line and the byte limits corrected.
+- [ ] `zig build dist -Dversion-string=0.0.1 -Dweb-dist=web/dist` produces two
+      tarballs and a checksum file, and fails without `-Dweb-dist`.
+- [ ] `zig build verify-dist` passes, and was proven able to fail: an oversized
+      binary, a dynamically linked binary, a mismatched `build.zig.zon` version
+      and a wrong archive mode each produce a named failure.
+- [ ] Two runs of `zig build dist` on the same commit produce byte-identical
+      tarballs **in the same directory**. (Cross-directory reproducibility is
+      ruling 12 and is not claimed here.)
+- [ ] The image builds for both platforms with no qemu, carries `/LICENSE` and
+      `/THIRD-PARTY-NOTICES` and the OCI labels, and its binaries are
+      byte-identical to the tarball binaries.
+- [ ] `gates.yml` runs from both `ci.yml` and `release.yml`; `ci.yml` triggers
+      on `master`; `origin/main` is gone.
+- [ ] `THIRD-PARTY-NOTICES` covers musl, the Zig runtime, SQLite, Mbed TLS with
+      its Apache-2.0 selection line and full text, Everest, p256-m and the web
+      runtime closure. The dependency drift guard was proven able to fail.
+- [ ] A dry run of `release.yml` completes with publication disabled.
+- [ ] `v0.0.1` is published: five assets, a verifying signature, and an image at
+      `git.mial.net/mokhtar/nxdns:0.0.1` and `:latest`.
+- [ ] `docs/how-to/verify-a-release.md` was followed end to end against the
+      published release, from a clean directory, on this host.
+- [ ] No `zig build cross` or source-only-distribution text remains anywhere.
+
+## Anti-requirements
+
+- No GoReleaser, no nfpm, no `.deb` or `.rpm`, no Homebrew, no AUR.
+- No cosign, no SBOM, no SLSA provenance, no in-toto attestations.
+- No `:edge` image, no rolling dev build, no pre-release tags, no floating
+  `0.0` or `0` image tags.
+- No third architecture, no non-Linux target, no glibc build.
+- No project website, no documentation site generator.
+- No changelog generation from commit messages, and no Conventional Commits.
+- No release automation that cannot be run from a laptop.