From e4c34784c8d894dd9e4628d6980c2808b79bd143 Mon Sep 17 00:00:00 2001 From: m5r Date: Sun, 9 Aug 2026 02:01:19 +0200 Subject: [PATCH] rewrite as a daemon with shutdown on low battery the oneshot+timer produced 8367 journal lines a day of systemd start/stop noise. now Type=notify with an internal ticker, logging transitions only. hardware sits behind sensor.Sensor; internal/monitor is pure, so the shutdown state machine is tested without a pi. four guards, each tested, because a false poweroff of a box with no physical access is worse than a missed one: - a failed read resets the arming window rather than pausing it - voltage must actually fall across the window, else it is a stuck sensor. this is the only defence against the ac line reading 0 while mains is connected, so it is load-bearing - a settle period stops a restart loop acting on early readings - flapping ac cannot accumulate verified on hardware 2026-08-09: after poweroff on battery the x1208 starts the pi again when mains returns, with no button press. that was the risk that could have made this feature unsafe. a failed tick rewrites the previous sample with sensor_healthy 0 and the failure counter, so last_update still freezes for the staleness alert while the failure stays visible. module path moved to git.mial.net. debian units and the compose overlay deleted; deployment lives in the infra repo. --- .gitignore | 3 + PLAN.md | 91 +++++++ README.md | 152 ++++++++--- flake.lock | 61 +++++ flake.nix | 63 +++++ go.mod | 4 +- go.sum | 2 + internal/max17040/max17040.go | 146 +++++++++++ internal/monitor/monitor.go | 257 ++++++++++++++++++ internal/monitor/monitor_test.go | 288 ++++++++++++++++++++ internal/pld/pld.go | 86 ++++++ internal/sensor/fake.go | 88 +++++++ internal/sensor/hardware.go | 54 ++++ internal/sensor/sensor.go | 26 ++ internal/textfile/textfile.go | 85 ++++++ internal/textfile/textfile_test.go | 125 +++++++++ justfile | 49 ++-- main.go | 405 ++++++++++++++++++----------- monitoring-overlay/README.md | 26 -- monitoring-overlay/compose.yaml | 42 --- monitoring-overlay/prometheus.yaml | 16 -- systemd/x1208-exporter.service | 15 -- systemd/x1208-exporter.timer | 11 - 23 files changed, 1774 insertions(+), 321 deletions(-) create mode 100644 PLAN.md create mode 100644 flake.lock create mode 100644 flake.nix create mode 100644 internal/max17040/max17040.go create mode 100644 internal/monitor/monitor.go create mode 100644 internal/monitor/monitor_test.go create mode 100644 internal/pld/pld.go create mode 100644 internal/sensor/fake.go create mode 100644 internal/sensor/hardware.go create mode 100644 internal/sensor/sensor.go create mode 100644 internal/textfile/textfile.go create mode 100644 internal/textfile/textfile_test.go delete mode 100644 monitoring-overlay/README.md delete mode 100644 monitoring-overlay/compose.yaml delete mode 100644 monitoring-overlay/prometheus.yaml delete mode 100644 systemd/x1208-exporter.service delete mode 100644 systemd/x1208-exporter.timer diff --git a/.gitignore b/.gitignore index f83240a..2674c37 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1,5 @@ /build/ +/result +/result-* +*.prom /monitoring-overlay/*.original diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..86ba0c7 --- /dev/null +++ b/PLAN.md @@ -0,0 +1,91 @@ +# x1208-exporter — daemon rewrite plan + +Oneshot+timer → long-running daemon. Why: the 30 s oneshot generates 8,367 journal lines/day (systemd Starting/Finished noise), ~33 KB SD writeback per sparse journal entry on the old box. Target: `Type=notify`, `Restart=always`, internal ticker, near-zero steady-state logging. Greenfield rules apply: breaking changes allowed, no debt carried. + +Consumer: `rpi.mial.net` (NixOS, `buildGoModule`, aarch64). Metric names are LOAD-BEARING (Grafana alert `RpiUpsExporterStale` + dashboard): `rpi_ups_last_update_seconds`, `rpi_ups_ac_power`, `rpi_ups_voltage_volts`, `rpi_ups_battery_percent` — never rename. + +## Decisions + +| # | Question | Call | Why | +|---|---|---|---| +| 1 | Textfile vs HTTP `/metrics` | **Keep the textfile.** | Zero blast radius: alert, dashboard, Alloy config, node-exporter scrape all unchanged (`no_data_state=Alerting` + `time()-last_update` verified correct as-is). HTTP would add `prometheus/client_golang` (heavy dep for 4 gauges + go runtime noise), a listener + port, a new Alloy scrape job in the other repo, and change alert semantics (frozen-file staleness → absent-series) for no functional gain. The atomic write already exists and is correct. Daemon rewrites the file per tick; textfile-as-IPC ambiguity is resolved by decision 4 (no second reader anymore). | +| 2 | Resource lifecycle | **Held open, reopen-on-error.** i2c fd + GPIO line acquired at startup, held across ticks. Any read error after the existing retry (3×20 ms): close BOTH handles, next tick reacquires from scratch. | Held-open = cheap ticks, no per-tick open/close syscall churn. Reopen-on-error gives per-read robustness where it matters: a wedged i2c bus or vanished gpiocdev line heals on the next tick instead of poisoning the fd forever. Failure to reacquire = failed tick, covered by decision 3; daemon never exits over hardware errors (`Restart=always` is the backstop for panics, not for a flaky bus). | +| 3 | Failure + logging policy | **Log transitions only. Failed tick = no write.** | Logs exactly: one startup line, AC state change (info), enter error state (warn, first failed tick only), exit error state (info, with failure count). NOTHING on a successful tick. Metric policy: the file is written only from a COMPLETE successful sample (all three readings); partial or failed tick leaves the previous file untouched, so `last_update` freezes and the existing staleness alert fires after its designed 3 m + 3 m — no new alert needed, no partial-file states to reason about. Additive: `rpi_ups_read_failures_total` counter (resets on restart; `rate()` handles that). | +| 4 | Shutdown responsibility | **In this daemon, flag-gated, default OFF.** Supersedes the separate `ups-shutdown.nix` file-reader in rpi.mial.net (that module shrinks to flags + polkit rule; flagged to the other repo, not edited here). | The daemon owns the readings; a second process re-parsing a textfile the first just wrote re-implements sampling, staleness guarding and hysteresis against a lossy serialization of state it could have had directly — that is the brittle architecture. Failure coupling is identical either way (dead exporter = stale file = separate reader must no-op too). Privilege solved without root: daemon execs `systemctl poweroff` (no D-Bus dep), authorized by a polkit rule matching a static group the unit joins. Trigger: `ac==0 AND (voltage < 3.5 OR soc < 15)` sustained N consecutive ticks (default N=4 → 2 min), thresholds ABOVE the 3.3 V `RpiUpsBatteryCritical` alert so the page fires before the poweroff. Staleness guard inherent: only fresh successful samples feed the state machine. One warn log when armed, one when disarmed, one at poweroff. | +| 5 | Testability | Interface seam + pure cores, below. | Threshold/transition/render logic unit-testable on any machine; hardware isolated to two leaf packages. | +| 6 | Packaging | **Flake in this repo** exporting the package; rpi.mial.net consumes it as a flake input (nixpkgs `follows`). Tagged releases `vX.Y.Z`; `ldflags "-X main.version="`. | vendorHash + version live NEXT to the code — one commit bumps both, Renovate bumps the consumer's flake.lock (nix manager already enabled there). `fetchFromGitea` + rev/hash pairs in the consumer would smear this repo's build metadata across two repos. Repo goes public (decided), so the build-time-fetch-needs-no-auth constraint is satisfied either way; flake is the cleaner shape. NO nixosModule in this repo: the systemd unit is infra policy and lives in `rpi.mial.net/nixos/modules/x1208-exporter.nix` — two sources of unit truth is exactly the split-brain to avoid. | +| 7 | Repo hygiene | Below. | — | +| 8 | Unit shape | Below (reference for the infra module). | — | + +## Architecture + +``` +main.go flags, wiring, signal handling — no logic +internal/max17040/ fuel gauge: open/ioctl/read-word/retry/scale (from current readBattery) +internal/pld/ GPIO 6 AC-present line via gpiocdev (from current readACPower) +internal/sensor/ type Reading { AC bool; Volts, SOC float64; At time.Time } + type Sensor interface { Read() (Reading, error); Close() error } + real: composes max17040+pld, owns held-open handles + reopen-on-error + fake: scripted readings/errors for tests +internal/monitor/ the daemon core, PURE: ticker loop driven by injected clock, + transition detection (AC change, error enter/exit), + shutdown state machine (N-sample hysteresis) → Action enum, + emits Reading → sink; no I/O, no syscalls +internal/textfile/ Render(Reading, now) string (golden-tested, byte-identical + metric names/HELP to today) + WriteAtomic (moved, unchanged) +internal/sdnotify/ READY=1 / WATCHDOG=1 datagram to $NOTIFY_SOCKET (~30 lines, no dep) +``` + +- `monitor` takes `Sensor`, clock, and callbacks (write file, log, act); unit tests cover: transition-only logging, failed-tick-no-write, N-sample shutdown hysteresis incl. flapping AC, staleness never triggers shutdown, counter increments. +- Keep: gpiochip0 by name (no Debian symlink dependency), validation ranges (2.0–5.0 V, ≤110 %), 3×20 ms retry, big-endian word read, atomic rename. +- Interval: `-interval 30s` flag, same cadence as today. `-out` flag kept. Shutdown: `-shutdown`, `-shutdown-voltage 3.5`, `-shutdown-soc 15`, `-shutdown-samples 4`, `-shutdown-cmd "systemctl poweroff"` (injectable for tests). + +## Systemd unit (reference — authoritative copy goes in rpi.mial.net) + +``` +Type=notify WatchdogSec=90s # 3 missed ticks → systemd restarts a hung daemon +Restart=always RestartSec=5s +DynamicUser=yes +SupplementaryGroups=i2c ups # i2c: /dev/i2c-1; ups: static group — owns the + # textfile dir (g+w) AND matches the polkit rule +DeviceAllow=/dev/i2c-1 rw +DeviceAllow=/dev/gpiochip0 rw +ReadWritePaths=/var/lib/node-exporter/textfiles +Nice=10 MemoryMax=32M +NoNewPrivileges / ProtectSystem=strict / ProtectHome / PrivateTmp / +ProtectKernel{Tunables,Modules,Logs} / ProtectControlGroups / ProtectClock / +RestrictAddressFamilies=AF_UNIX / RestrictNamespaces / RestrictRealtime / +LockPersonality / MemoryDenyWriteExecute / SystemCallFilter=@system-service / +SystemCallArchitectures=native / UMask=0022 +``` + +Timer: deleted — no timer exists anymore. +Root dropped entirely: device access via groups, poweroff via polkit (`org.freedesktop.login1.power-off` allowed for `unix-group:ups`). `MemoryDenyWriteExecute` is safe for pure Go. `/dev/gpiochip0` group access needs a udev rule on NixOS (infra repo's job — flagged, not assumed). + +## Repo hygiene + +| Item | Action | +|---|---| +| `build/x1208-exporter` committed binary | Remove. Repo has ONE commit and is about to go public: rewrite history (fresh init or amend) so the binary is not in it forever. Cheap now, impossible later. | +| `.gitignore` | `/build/`, `/result`, `/result-*`, `*.prom` | +| Module path `github.com/mokhtar/x1208-exporter` | Wrong host — repo lives at git.mial.net. Rename to `git.mial.net/mokhtar/x1208-exporter`. Breaking, allowed, correct. | +| `go.mod` Go 1.23 | Bump to current stable at rewrite time. Deps: keep `go-gpiocdev`; DROP `golang.org/x/sys` from direct use only if max17040 no longer needs it (it does — ioctl; keep). | +| `systemd/` (Debian units) + justfile scp-deploy recipes | Delete. NixOS module in rpi.mial.net is the only deployment; dead Debian paths are debt. justfile keeps: `build`, `test`, `lint`, `run-fake` (daemon against fake sensor, prints file to stdout). | +| README | Rewrite for daemon + flake usage; drop compose/Debian instructions. | + +## Order of work + +1. History rewrite + hygiene (binary out, gitignore, module path). Push before publicizing. +2. Package split with tests, behavior-preserving (still produces identical .prom bytes — golden test locks this). +3. Daemon loop: ticker, held-open sensor, transition logging, sdnotify + watchdog. +4. Shutdown state machine, flag-gated off. +5. Flake (package + devShell), tag `v1.0.0`. +6. Consumption + unit + polkit + udev + `ups` group land in rpi.mial.net (other repo — out of scope here; includes retiring `ups-shutdown.nix` as a separate reader). + +## Unresolved questions + +1. Shutdown thresholds sign-off: `3.5 V / 15 % / N=4` (2 min sustained) — above the 3.3 V page, below nuisance range. Confirm numbers. +2. Static group name for textfile-dir write + polkit match (`ups`?) — infra repo decides, this plan assumes one group serves both. +3. Polkit rule exact shape on NixOS (`security.polkit.extraConfig` matching `org.freedesktop.login1.power-off` for the group) — verify action id covers `systemctl poweroff` non-interactively before relying on it; fallback is a root-owned wrapper with a narrow sudoers line (worse; avoid if polkit works). +4. Does anything else read the textfile once ups-shutdown-as-reader dies? (Believed: only node-exporter. Confirm before changing write policy to no-partial-updates.) +5. `rpi_ups_read_failures_total`: worth a Grafana panel, or leave unalerted as diagnostic? (No alert planned — staleness alert remains the pager.) diff --git a/README.md b/README.md index 62b3c65..c4212b1 100644 --- a/README.md +++ b/README.md @@ -1,49 +1,129 @@ # x1208-exporter -One-shot Prometheus textfile exporter for the Geekworm X1208 UPS HAT on Raspberry Pi 5. +A Prometheus exporter for the [Geekworm X1208](https://wiki.geekworm.com/X1208) +UPS HAT on a Raspberry Pi 5. -Reads: - -- GPIO 6 (PLD) — `1` = AC plugged in, `0` = on battery. -- I2C `0x36` (MAX17040 fuel gauge) — battery voltage and state of charge. - -Writes `/var/lib/node-exporter/textfiles/x1208.prom` atomically. node-exporter's -textfile collector serves it at scrape time. +It reads cell voltage and state of charge from the MAX17040 fuel gauge over +i2c, reads the power-loss-detect line over GPIO, and writes a Prometheus +textfile that node-exporter's textfile collector scrapes. ## Metrics -| Name | Type | Description | -| ------------------------------- | ----- | -------------------------------------------- | -| `rpi_ups_ac_power` | gauge | `1` = AC present, `0` = on battery | -| `rpi_ups_voltage_volts` | gauge | Battery cell voltage | -| `rpi_ups_battery_percent` | gauge | State of charge (0–100) | -| `rpi_ups_last_update_seconds` | gauge | Unix timestamp of last exporter run | +| Metric | Type | Meaning | +| --- | --- | --- | +| `rpi_ups_last_update_seconds` | gauge | Unix time of the last COMPLETE sample | +| `rpi_ups_ac_power` | gauge | 1 on mains, 0 on cells | +| `rpi_ups_voltage_volts` | gauge | Cell voltage | +| `rpi_ups_battery_percent` | gauge | State of charge | +| `rpi_ups_sensor_healthy` | gauge | 1 if the most recent read succeeded | +| `rpi_ups_read_failures_total` | counter | Failed reads since start | -## Deploy +The first four names are load-bearing. Grafana alerts and the Node Exporter +Full dashboard reference them. A golden test locks their exact rendering, +including the rounding mode. Do not rename them and do not change their +formatting. -```bash -just deploy +## Failure behaviour + +A read either produces a complete sample or fails. There is no partial sample. + +When a read fails the exporter rewrites the file with the **previous** complete +sample, sets `rpi_ups_sensor_healthy` to 0, and increments the failure counter. +`rpi_ups_last_update_seconds` therefore stops advancing, which is what the +`RpiUpsExporterStale` alert measures. The failure stays visible in the +meantime rather than hiding behind a frozen file. + +The exporter logs transitions only: startup, AC changes, entering and leaving +an error state, and shutdown decisions. A healthy sample logs nothing. This +matters because the machine keeps its journal in RAM and archives it to an SD +card. + +## Shutdown on low battery + +The exporter can power the machine off when the cells run down. It is **off by +default** and must be enabled with `-shutdown`. + +It acts only when all of these hold: + +- The AC line reports no mains power. +- Cell voltage is below `-shutdown-voltage`, or state of charge is below + `-shutdown-soc`. +- That has held for `-shutdown-samples` consecutive ticks. +- More than `-shutdown-settle-ticks` ticks have passed since start. +- Cell voltage actually fell across the window, unless + `-shutdown-require-discharge=false`. + +Each guard exists because a false poweroff is worse than a missed one. The +machine it runs on has no physical access, so a wrong decision that halts it +cannot be undone remotely. + +- A **failed read resets** the window. It does not pause it. Otherwise "low, + low, a long outage, low, low" would reach the sample count without the + battery ever being low for that period. +- The **discharge requirement** is load-bearing. Do not turn it off. + + It separates a real discharge from a stuck sensor: a wedged fuel gauge + repeats one plausible low value forever, and four samples of one persistent + fault are not four independent confirmations. + + It is also the only defence against the worst remaining failure, an AC line + that reads 0 while mains is actually connected. A floating GPIO line reports + "on battery" and returns no error. On mains the charger holds cell voltage + flat or rising, so a false reading cannot satisfy this check. + `TestFalseACLossWhileChargingNeverTriggersShutdown` covers it, and + `TestWithoutDischargeGuardAFalseACReadingIsEnough` records what happens if + someone disables the guard. +- The **settle period** stops a restart loop from acting on readings taken + before the hardware settled. + +### The HAT does restore power after a software shutdown + +This was the one question that could have made the whole feature unsafe. The +X1208 cuts power to the Pi after halt. If mains returned while the board still +held charge, and the board did not then start the Pi, a clean shutdown would +strand a machine nobody can reach. + +Tested on 2026-08-09 against the real hardware: + +1. On mains, `rpi_ups_ac_power` read 1 at 4.149 V. +2. Mains disconnected at the wall. `rpi_ups_ac_power` read 0 at 4.109 V. +3. `sudo systemctl poweroff`. +4. Mains reconnected. **No button was pressed.** +5. The Pi booted by itself. + +The same run measured the discharge rate under normal load: 40 mV per minute. +Across a four-tick window at 30 s that is about 80 mV, against a gauge +resolution of 1.25 mV. The discharge guard therefore has a wide margin and +cannot block a real discharge. + +**Still untested:** a poweroff while mains stayed connected the whole time. +The test above disconnected mains first. If the board treats "input power +never dropped" differently, that path is unproven. It only matters if a false +AC reading gets past the discharge guard, which is why that guard is +load-bearing. Test it during the soak: run `systemctl poweroff` with mains +connected and record whether the Pi returns. + +## Development + +```sh +just test # unit tests; no hardware needed +just lint +just run-fake # daemon against a scripted discharging battery +just show +just nix-build # build the package as the Pi will ``` -Builds the arm64 binary, scps to `rpi`, installs systemd units, starts the timer. +The hardware sits behind `sensor.Sensor`. `internal/monitor` holds every +decision and is pure, so the shutdown state machine is tested without a Pi. +`internal/max17040` and `internal/pld` are the only packages that touch +devices. -## Verify +## Deployment -```bash -just run-once # forces one read, prints the prom file -just logs # journalctl tail -``` +This repository ships the **package only**. The systemd unit, device access, +groups, polkit rules and the decision to enable shutdown are host policy and +live in the infrastructure repository. Keeping a unit here as well would create +two sources of truth. -## How node-exporter picks it up - -The `monitoring` compose stack on the Pi must mount the textfile dir and pass -the flag to node-exporter: - -```yaml -node-exporter: - volumes: - - /var/lib/node-exporter/textfiles:/textfiles:ro - command: - - '--collector.textfile.directory=/textfiles' - # ...existing flags -``` +Consume it as a flake input and let the infrastructure repository define the +service. diff --git a/flake.lock b/flake.lock new file mode 100644 index 0000000..5825cc2 --- /dev/null +++ b/flake.lock @@ -0,0 +1,61 @@ +{ + "nodes": { + "flake-utils": { + "inputs": { + "systems": "systems" + }, + "locked": { + "lastModified": 1731533236, + "narHash": "sha256-l0KFg5HjrsfsO/JpG+r7fRrqm12kzFHyUHqHCVpMMbI=", + "owner": "numtide", + "repo": "flake-utils", + "rev": "11707dc2f618dd54ca8739b309ec4fc024de578b", + "type": "github" + }, + "original": { + "owner": "numtide", + "repo": "flake-utils", + "type": "github" + } + }, + "nixpkgs": { + "locked": { + "lastModified": 1785967620, + "narHash": "sha256-IItrdb7Puk05RqOBWZYFC5X6Wl1sJmCfh5MWVHw5iMM=", + "owner": "NixOS", + "repo": "nixpkgs", + "rev": "b7c2ada94fe99c15b0dbcf4d11fd7850b957a436", + "type": "github" + }, + "original": { + "owner": "NixOS", + "ref": "nixos-unstable", + "repo": "nixpkgs", + "type": "github" + } + }, + "root": { + "inputs": { + "flake-utils": "flake-utils", + "nixpkgs": "nixpkgs" + } + }, + "systems": { + "locked": { + "lastModified": 1681028828, + "narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=", + "owner": "nix-systems", + "repo": "default", + "rev": "da67096a3b9bf56a91d16901293e51ba5b49a27e", + "type": "github" + }, + "original": { + "owner": "nix-systems", + "repo": "default", + "type": "github" + } + } + }, + "root": "root", + "version": 7 +} diff --git a/flake.nix b/flake.nix new file mode 100644 index 0000000..0738ce2 --- /dev/null +++ b/flake.nix @@ -0,0 +1,63 @@ +{ + description = "Prometheus exporter for the Geekworm X1208 UPS HAT"; + + inputs = { + nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable"; + flake-utils.url = "github:numtide/flake-utils"; + }; + + outputs = + { + self, + nixpkgs, + flake-utils, + }: + # This flake exports the PACKAGE only. There is deliberately no + # nixosModule: the systemd unit carries host policy (device access, + # groups, polkit, hardening, whether shutdown is enabled at all) and lives + # in the infrastructure repo. Two sources of unit truth is the split-brain + # this avoids. + flake-utils.lib.eachDefaultSystem ( + system: + let + pkgs = nixpkgs.legacyPackages.${system}; + version = if self ? rev then "0.0.0+${builtins.substring 0 8 self.rev}" else "dev"; + in + { + packages.default = self.packages.${system}.x1208-exporter; + + packages.x1208-exporter = pkgs.buildGoModule { + pname = "x1208-exporter"; + inherit version; + src = ./.; + vendorHash = "sha256-vPzEdZJ3kZ79zl17r2ssFtk3anE21ARy/RRbYqBoiEI="; + + ldflags = [ + "-s" + "-w" + "-X main.version=${version}" + ]; + + # The hardware packages need a Pi; the pure ones do not. Only the + # pure packages carry tests, so the whole suite runs in the sandbox. + doCheck = true; + + meta = { + description = "Prometheus textfile exporter for the Geekworm X1208 UPS HAT"; + mainProgram = "x1208-exporter"; + platforms = pkgs.lib.platforms.linux; + }; + }; + + devShells.default = pkgs.mkShell { + packages = with pkgs; [ + go + gopls + golangci-lint + ]; + }; + + formatter = pkgs.nixfmt-rfc-style; + } + ); +} diff --git a/go.mod b/go.mod index 3d08c17..7707b8b 100644 --- a/go.mod +++ b/go.mod @@ -1,4 +1,4 @@ -module github.com/mokhtar/x1208-exporter +module git.mial.net/mokhtar/x1208-exporter go 1.23 @@ -6,3 +6,5 @@ require ( github.com/warthog618/go-gpiocdev v0.9.1 golang.org/x/sys v0.27.0 ) + +require github.com/coreos/go-systemd/v22 v22.7.0 diff --git a/go.sum b/go.sum index 431bb30..912722a 100644 --- a/go.sum +++ b/go.sum @@ -1,3 +1,5 @@ +github.com/coreos/go-systemd/v22 v22.7.0 h1:LAEzFkke61DFROc7zNLX/WA2i5J8gYqe0rSj9KI28KA= +github.com/coreos/go-systemd/v22 v22.7.0/go.mod h1:xNUYtjHu2EDXbsxz1i41wouACIwT7Ybq9o0BQhMwD0w= github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= diff --git a/internal/max17040/max17040.go b/internal/max17040/max17040.go new file mode 100644 index 0000000..bddcee8 --- /dev/null +++ b/internal/max17040/max17040.go @@ -0,0 +1,146 @@ +// Package max17040 reads the fuel gauge on the X1208 UPS HAT over i2c. +package max17040 + +import ( + "encoding/binary" + "fmt" + "time" + + "golang.org/x/sys/unix" +) + +const ( + // DefaultDevice is the i2c bus the HAT sits on. dtparam=i2c_arm=on. + DefaultDevice = "/dev/i2c-1" + + addr = 0x36 + regVoltage = 0x02 + regCapacity = 0x04 + i2cSlave = 0x0703 // I2C_SLAVE + + maxRetries = 3 + retryDelay = 20 * time.Millisecond + + voltageScale = 1.25 / 1000 / 16 + socScale = 1.0 / 256 + + minPlausibleVolts = 2.0 + maxPlausibleVolts = 5.0 + maxPlausibleSOC = 110 +) + +// Gauge holds the i2c file descriptor open across reads. Opening per read cost +// two extra syscalls every tick for no benefit. +type Gauge struct { + device string + fd int + open bool +} + +// New returns a Gauge that has not yet opened the bus. The first Read opens it. +func New(device string) *Gauge { + if device == "" { + device = DefaultDevice + } + return &Gauge{device: device, fd: -1} +} + +func (g *Gauge) ensureOpen() error { + if g.open { + return nil + } + fd, err := unix.Open(g.device, unix.O_RDWR, 0) + if err != nil { + return fmt.Errorf("open %s: %w", g.device, err) + } + if err := unix.IoctlSetInt(fd, i2cSlave, addr); err != nil { + unix.Close(fd) + return fmt.Errorf("ioctl I2C_SLAVE 0x%02x: %w", addr, err) + } + g.fd = fd + g.open = true + return nil +} + +// Close releases the bus. It is safe to call on an already-closed Gauge, and +// safe to call more than once. +func (g *Gauge) Close() error { + if !g.open { + return nil + } + g.open = false + fd := g.fd + g.fd = -1 + return unix.Close(fd) +} + +// Read returns cell voltage in volts and state of charge in percent. +// +// On any error the bus is closed, so the next Read reopens from scratch. This +// heals a wedged descriptor without a process restart. +func (g *Gauge) Read() (volts float64, soc float64, err error) { + if err := g.ensureOpen(); err != nil { + return 0, 0, err + } + defer func() { + if err != nil { + _ = g.Close() + } + }() + + rawV, err := g.readWordRetry(regVoltage, func(v uint16) bool { + f := float64(v) * voltageScale + return f > minPlausibleVolts && f < maxPlausibleVolts + }) + if err != nil { + return 0, 0, fmt.Errorf("voltage: %w", err) + } + + rawC, err := g.readWordRetry(regCapacity, func(v uint16) bool { + return float64(v)*socScale <= maxPlausibleSOC + }) + if err != nil { + return 0, 0, fmt.Errorf("capacity: %w", err) + } + + return float64(rawV) * voltageScale, float64(rawC) * socScale, nil +} + +// readWord writes the register pointer, then reads the word back. +// +// This is deliberately NOT an I2C_RDWR combined transaction. The bcm2835 i2c +// controller has known repeated-start limitations, and this two-step sequence +// is what has been running against this HAT. The sequence is only safe because +// nothing else on the system opens /dev/i2c-1 — if that ever changes, another +// process can move the register pointer between the write and the read. +func (g *Gauge) readWord(reg byte) (uint16, error) { + if _, err := unix.Write(g.fd, []byte{reg}); err != nil { + return 0, fmt.Errorf("write reg 0x%02x: %w", reg, err) + } + buf := make([]byte, 2) + n, err := unix.Read(g.fd, buf) + if err != nil { + return 0, fmt.Errorf("read: %w", err) + } + if n != 2 { + return 0, fmt.Errorf("short read: %d bytes", n) + } + return binary.BigEndian.Uint16(buf), nil +} + +func (g *Gauge) readWordRetry(reg byte, plausible func(uint16) bool) (uint16, error) { + var lastErr error + for i := 0; i < maxRetries; i++ { + v, err := g.readWord(reg) + switch { + case err != nil: + lastErr = err + case !plausible(v): + lastErr = fmt.Errorf("value out of range: 0x%04x", v) + default: + return v, nil + } + time.Sleep(retryDelay) + } + return 0, lastErr +} diff --git a/internal/monitor/monitor.go b/internal/monitor/monitor.go new file mode 100644 index 0000000..7e9f560 --- /dev/null +++ b/internal/monitor/monitor.go @@ -0,0 +1,257 @@ +// Package monitor holds the daemon's decision logic. +// +// Everything here is pure: it takes a reading and an error, and returns what +// to log, what to write, and whether to act. It performs no I/O and reads no +// clock. That is what makes the shutdown path testable without a Pi. +package monitor + +import ( + "fmt" + + "git.mial.net/mokhtar/x1208-exporter/internal/sensor" +) + +// Event names a state transition. The daemon logs transitions only: a healthy +// tick produces no output at all. +type Event string + +const ( + EventStartup Event = "startup" + EventACChanged Event = "ac_changed" + EventErrorEnter Event = "error_enter" + EventErrorExit Event = "error_exit" + EventShutdownArmed Event = "shutdown_armed" + EventShutdownDisarmed Event = "shutdown_disarmed" + EventShutdownInhibited Event = "shutdown_inhibited" + EventShutdownTriggered Event = "shutdown_triggered" +) + +// Log is one line the daemon must emit. +type Log struct { + Event Event + Message string +} + +// Config controls the shutdown policy. The zero value disables shutdown. +type Config struct { + // Shutdown enables the poweroff decision. Default off: the daemon is an + // exporter first, and actuation is opt-in. + Shutdown bool + + // Volts and SOC are the low-battery thresholds. Either one can arm. + Volts float64 + SOC float64 + + // Samples is how many consecutive qualifying ticks must pass before the + // daemon acts. + Samples int + + // MinUptimeTicks suppresses shutdown for the first N ticks after start. + // Without this a restart loop could power the machine off using readings + // taken before the hardware settled. + MinUptimeTicks int + + // RequireDischarge demands that cell voltage actually fall across the + // arming window before the daemon acts. + // + // This is the only check that distinguishes a real discharge from a stuck + // sensor. A wedged fuel gauge repeats one plausible low value forever, and + // consecutive samples of a persistent fault are not independent evidence. + // + // It is LOAD-BEARING, not a refinement. It is the sole defence against the + // worst remaining failure: the AC line falsely reading 0 while mains is + // actually connected. On mains the charger holds cell voltage flat or + // rising, so a false "on battery" cannot satisfy this check. Turn it off + // and a single stuck GPIO line can power the machine off during normal + // operation. + RequireDischarge bool +} + +// Result is the outcome of one tick. +type Result struct { + Logs []Log + + // Sample is the most recent COMPLETE reading, which is not necessarily + // this tick's. HaveSample is false until the first successful read. + Sample sensor.Reading + HaveSample bool + + // Healthy reports whether this tick's read succeeded. + Healthy bool + Failures uint64 + + // Shutdown is true on the single tick where the daemon decides to power + // the machine off. + Shutdown bool +} + +// Monitor tracks state across ticks. +type Monitor struct { + cfg Config + + started bool + ticks int + + lastAC bool + haveAC bool + + inError bool + errStreak int + failures uint64 + last sensor.Reading + haveLast bool + armed bool + armCount int + armVolts float64 + inhibited bool + firedShut bool +} + +// New returns a Monitor with the given policy. +func New(cfg Config) *Monitor { return &Monitor{cfg: cfg} } + +// Tick advances the state machine by one reading. +// +// Pass the error from the sensor as err; reading is ignored when err is +// non-nil. Tick never blocks and never panics. +func (m *Monitor) Tick(reading sensor.Reading, err error) Result { + m.ticks++ + res := Result{Failures: m.failures} + + if !m.started { + m.started = true + res.Logs = append(res.Logs, Log{EventStartup, m.startupMessage()}) + } + + if err != nil { + return m.tickFailed(res, err) + } + return m.tickSucceeded(res, reading) +} + +func (m *Monitor) startupMessage() string { + if !m.cfg.Shutdown { + return "x1208-exporter started; shutdown-on-battery disabled" + } + return fmt.Sprintf( + "x1208-exporter started; shutdown-on-battery ENABLED below %.2f V or %.0f%% for %d ticks (require-discharge=%t)", + m.cfg.Volts, m.cfg.SOC, m.cfg.Samples, m.cfg.RequireDischarge) +} + +func (m *Monitor) tickFailed(res Result, err error) Result { + m.failures++ + m.errStreak++ + res.Failures = m.failures + res.Healthy = false + res.Sample = m.last + res.HaveSample = m.haveLast + + if !m.inError { + m.inError = true + res.Logs = append(res.Logs, Log{EventErrorEnter, "UPS read failed: " + err.Error()}) + } + + // A failed tick RESETS the arming window rather than pausing it. + // + // Pausing would let "low, low, thirty minutes of errors, low, low" reach + // the sample count. That is not a sustained low battery and must not power + // the machine off. + res.Logs = append(res.Logs, m.disarm("sensor read failed")...) + return res +} + +func (m *Monitor) tickSucceeded(res Result, r sensor.Reading) Result { + if m.inError { + res.Logs = append(res.Logs, Log{EventErrorExit, + fmt.Sprintf("UPS read recovered after %d failed ticks", m.errStreak)}) + m.inError = false + m.errStreak = 0 + } + + if m.haveAC && m.lastAC != r.ACPresent { + res.Logs = append(res.Logs, Log{EventACChanged, acMessage(r.ACPresent)}) + } + m.lastAC = r.ACPresent + m.haveAC = true + + m.last = r + m.haveLast = true + + res.Healthy = true + res.Sample = r + res.HaveSample = true + + logs, shutdown := m.evaluateShutdown(r) + res.Logs = append(res.Logs, logs...) + res.Shutdown = shutdown + return res +} + +func acMessage(present bool) string { + if present { + return "AC power restored" + } + return "AC power lost; running on battery" +} + +func (m *Monitor) evaluateShutdown(r sensor.Reading) ([]Log, bool) { + if !m.cfg.Shutdown || m.firedShut { + return nil, false + } + + qualifies := !r.ACPresent && (r.Volts < m.cfg.Volts || r.SOC < m.cfg.SOC) + if !qualifies { + return m.disarm("battery recovered or AC returned"), false + } + + if m.ticks <= m.cfg.MinUptimeTicks { + if m.inhibited { + return nil, false + } + m.inhibited = true + return []Log{{EventShutdownInhibited, fmt.Sprintf( + "low battery within the first %d ticks after start; shutdown suppressed", + m.cfg.MinUptimeTicks)}}, false + } + + if !m.armed { + m.armed = true + m.armCount = 1 + m.armVolts = r.Volts + return []Log{{EventShutdownArmed, fmt.Sprintf( + "low battery on cells (%.3f V, %.1f%%); shutdown in %d more qualifying ticks", + r.Volts, r.SOC, m.cfg.Samples-1)}}, false + } + + m.armCount++ + if m.armCount < m.cfg.Samples { + return nil, false + } + + if m.cfg.RequireDischarge && r.Volts >= m.armVolts { + if m.inhibited { + return nil, false + } + m.inhibited = true + return []Log{{EventShutdownInhibited, fmt.Sprintf( + "battery low but voltage did not fall over %d ticks (%.3f V then %.3f V); "+ + "treating as a stuck sensor, not a discharge", + m.armCount, m.armVolts, r.Volts)}}, false + } + + m.firedShut = true + return []Log{{EventShutdownTriggered, fmt.Sprintf( + "powering off: on battery at %.3f V / %.1f%% for %d consecutive ticks", + r.Volts, r.SOC, m.armCount)}}, true +} + +func (m *Monitor) disarm(reason string) []Log { + m.inhibited = false + if !m.armed { + m.armCount = 0 + return nil + } + m.armed = false + m.armCount = 0 + return []Log{{EventShutdownDisarmed, "shutdown disarmed: " + reason}} +} diff --git a/internal/monitor/monitor_test.go b/internal/monitor/monitor_test.go new file mode 100644 index 0000000..2d6cb05 --- /dev/null +++ b/internal/monitor/monitor_test.go @@ -0,0 +1,288 @@ +package monitor + +import ( + "errors" + "testing" + "time" + + "git.mial.net/mokhtar/x1208-exporter/internal/sensor" +) + +func shutdownConfig() Config { + return Config{ + Shutdown: true, + Volts: 3.5, + SOC: 15, + Samples: 4, + MinUptimeTicks: 2, + RequireDischarge: true, + } +} + +// onBattery returns a low reading that falls by 10 mV per step, which is what +// a real discharge looks like. +func onBattery(step int) sensor.Reading { + return sensor.Reading{ + ACPresent: false, + Volts: 3.40 - float64(step)*0.01, + SOC: 10, + At: time.Unix(1754661600+int64(step)*30, 0), + } +} + +func onMains() sensor.Reading { + return sensor.Reading{ACPresent: true, Volts: 4.1, SOC: 100, At: time.Unix(1754661600, 0)} +} + +func events(r Result) []Event { + out := make([]Event, 0, len(r.Logs)) + for _, l := range r.Logs { + out = append(out, l.Event) + } + return out +} + +func contains(list []Event, want Event) bool { + for _, e := range list { + if e == want { + return true + } + } + return false +} + +// settle runs enough healthy mains ticks to clear the startup inhibition. +func settle(t *testing.T, m *Monitor) { + t.Helper() + for i := 0; i <= m.cfg.MinUptimeTicks; i++ { + if res := m.Tick(onMains(), nil); res.Shutdown { + t.Fatal("shutdown fired while on mains") + } + } +} + +func TestHealthyTicksLogNothingAfterStartup(t *testing.T) { + m := New(Config{}) + + first := m.Tick(onMains(), nil) + if !contains(events(first), EventStartup) { + t.Error("first tick must log startup") + } + + for i := 0; i < 5; i++ { + if res := m.Tick(onMains(), nil); len(res.Logs) != 0 { + t.Errorf("healthy tick %d logged %v, want nothing", i, events(res)) + } + } +} + +func TestACTransitionLogsOncePerChange(t *testing.T) { + m := New(Config{}) + m.Tick(onMains(), nil) + + res := m.Tick(onBattery(0), nil) + if !contains(events(res), EventACChanged) { + t.Fatal("losing AC must log a transition") + } + if res := m.Tick(onBattery(1), nil); len(res.Logs) != 0 { + t.Errorf("staying on battery logged %v, want nothing", events(res)) + } + if res := m.Tick(onMains(), nil); !contains(events(res), EventACChanged) { + t.Error("regaining AC must log a transition") + } +} + +func TestFailedTickKeepsPreviousSampleAndCounts(t *testing.T) { + m := New(Config{}) + m.Tick(onMains(), nil) + + res := m.Tick(sensor.Reading{}, errors.New("i2c wedged")) + if res.Healthy { + t.Error("failed tick must report unhealthy") + } + if res.Failures != 1 { + t.Errorf("failures = %d, want 1", res.Failures) + } + if !res.HaveSample || res.Sample.Volts != 4.1 { + t.Error("failed tick must carry the previous complete sample forward") + } + if !contains(events(res), EventErrorEnter) { + t.Error("first failure must log error_enter") + } + + if res := m.Tick(sensor.Reading{}, errors.New("still wedged")); contains(events(res), EventErrorEnter) { + t.Error("a continuing error must not log error_enter again") + } + if res := m.Tick(onMains(), nil); !contains(events(res), EventErrorExit) { + t.Error("recovery must log error_exit") + } +} + +func TestShutdownDisabledByDefault(t *testing.T) { + m := New(Config{}) + for i := 0; i < 20; i++ { + if m.Tick(onBattery(i), nil).Shutdown { + t.Fatal("shutdown fired with an empty Config; it must be opt-in") + } + } +} + +func TestShutdownFiresOnSustainedDischarge(t *testing.T) { + m := New(shutdownConfig()) + settle(t, m) + + for i := 0; i < 3; i++ { + if m.Tick(onBattery(i), nil).Shutdown { + t.Fatalf("shutdown fired on qualifying tick %d, before the sample count", i+1) + } + } + res := m.Tick(onBattery(3), nil) + if !res.Shutdown { + t.Fatal("shutdown must fire on the fourth consecutive qualifying tick") + } + if !contains(events(res), EventShutdownTriggered) { + t.Error("firing must log shutdown_triggered") + } + if m.Tick(onBattery(4), nil).Shutdown { + t.Error("shutdown must fire only once") + } +} + +// A wedged sensor repeats one plausible low value. Consecutive identical +// samples are one fault observed four times, not four confirmations. +func TestStuckSensorNeverTriggersShutdown(t *testing.T) { + m := New(shutdownConfig()) + settle(t, m) + + stuck := sensor.Reading{ACPresent: false, Volts: 3.40, SOC: 10, At: time.Unix(1754661600, 0)} + for i := 0; i < 50; i++ { + if m.Tick(stuck, nil).Shutdown { + t.Fatalf("shutdown fired at tick %d on a constant voltage", i) + } + } +} + +// "low, low, long outage, low, low" is not a sustained low battery. +func TestErrorResetsArmingWindow(t *testing.T) { + m := New(shutdownConfig()) + settle(t, m) + + m.Tick(onBattery(0), nil) + m.Tick(onBattery(1), nil) + + res := m.Tick(sensor.Reading{}, errors.New("bus error")) + if !contains(events(res), EventShutdownDisarmed) { + t.Error("a failed tick must disarm, not pause, the shutdown window") + } + + for i := 2; i < 5; i++ { + if m.Tick(onBattery(i), nil).Shutdown { + t.Fatalf("shutdown fired at tick %d; the window must restart after an error", i) + } + } +} + +func TestACReturnDisarms(t *testing.T) { + m := New(shutdownConfig()) + settle(t, m) + + m.Tick(onBattery(0), nil) + m.Tick(onBattery(1), nil) + + res := m.Tick(onMains(), nil) + if !contains(events(res), EventShutdownDisarmed) { + t.Error("AC returning must disarm") + } + for i := 2; i < 5; i++ { + if m.Tick(onBattery(i), nil).Shutdown { + t.Fatalf("shutdown fired at tick %d; the window must restart after AC returned", i) + } + } +} + +// A flapping AC line must not accumulate toward a poweroff. +func TestFlappingACNeverAccumulates(t *testing.T) { + m := New(shutdownConfig()) + settle(t, m) + + for i := 0; i < 40; i++ { + var res Result + if i%2 == 0 { + res = m.Tick(onBattery(i), nil) + } else { + res = m.Tick(onMains(), nil) + } + if res.Shutdown { + t.Fatalf("shutdown fired at tick %d on a flapping AC line", i) + } + } +} + +// A restart loop must not power the machine off using readings taken before +// the hardware settled. +func TestStartupInhibitionBlocksImmediateShutdown(t *testing.T) { + m := New(shutdownConfig()) + for i := 0; i < 2; i++ { + if m.Tick(onBattery(i), nil).Shutdown { + t.Fatalf("shutdown fired at tick %d, inside the startup inhibition", i+1) + } + } +} + +// The worst remaining failure: the AC line reads 0 while mains is actually +// connected. The charger then holds cell voltage flat or rising, which is what +// the discharge guard exists to catch. Without it a single stuck GPIO line +// powers the machine off during normal operation. +func TestFalseACLossWhileChargingNeverTriggersShutdown(t *testing.T) { + m := New(shutdownConfig()) + settle(t, m) + + for i := 0; i < 60; i++ { + r := sensor.Reading{ + ACPresent: false, // the line is lying + Volts: 3.40 + float64(i)*0.005, // but the charger is working + SOC: 10, + } + if m.Tick(r, nil).Shutdown { + t.Fatalf("shutdown fired at tick %d while voltage was RISING; "+ + "a false AC reading must not power the machine off", i) + } + } +} + +// Guard against a future change that quietly drops the discharge requirement: +// with it disabled, the same false reading DOES power the machine off. This +// test documents the consequence rather than endorsing the setting. +func TestWithoutDischargeGuardAFalseACReadingIsEnough(t *testing.T) { + cfg := shutdownConfig() + cfg.RequireDischarge = false + m := New(cfg) + settle(t, m) + + fired := false + for i := 0; i < 10 && !fired; i++ { + r := sensor.Reading{ACPresent: false, Volts: 3.40, SOC: 10} + fired = m.Tick(r, nil).Shutdown + } + if !fired { + t.Error("expected the unguarded config to act on a constant low reading; " + + "if this changed, update the RequireDischarge documentation") + } +} + +func TestLowSOCAloneCanArm(t *testing.T) { + m := New(shutdownConfig()) + settle(t, m) + + // Voltage stays above the threshold; only state of charge is low. + for i := 0; i < 3; i++ { + r := sensor.Reading{ACPresent: false, Volts: 3.9 - float64(i)*0.01, SOC: 5} + if m.Tick(r, nil).Shutdown { + t.Fatalf("fired early at tick %d", i) + } + } + r := sensor.Reading{ACPresent: false, Volts: 3.87, SOC: 5} + if !m.Tick(r, nil).Shutdown { + t.Error("a sustained low state of charge must trigger shutdown") + } +} diff --git a/internal/pld/pld.go b/internal/pld/pld.go new file mode 100644 index 0000000..042d445 --- /dev/null +++ b/internal/pld/pld.go @@ -0,0 +1,86 @@ +// Package pld reads the X1208 power-loss-detect line. +// +// The HAT drives GPIO 6 high while mains power is present and low while the +// Pi runs on cells. +package pld + +import ( + "fmt" + + "github.com/warthog618/go-gpiocdev" +) + +const ( + // DefaultChip is addressed by name, not by /dev path, so this does not + // depend on any distribution-specific symlink. + DefaultChip = "gpiochip0" + // DefaultLine is the power-loss-detect line on the X1208. + DefaultLine = 6 + + consumer = "x1208-exporter" +) + +// Line holds the GPIO request open across reads. The kernel grants the request +// exclusively, so reacquiring it per read risks EBUSY against our own +// not-yet-released handle. +type Line struct { + chip string + offset int + line *gpiocdev.Line +} + +// New returns a Line that has not yet been requested. The first Read requests it. +func New(chip string, offset int) *Line { + if chip == "" { + chip = DefaultChip + } + return &Line{chip: chip, offset: offset} +} + +func (l *Line) ensureOpen() error { + if l.line != nil { + return nil + } + line, err := gpiocdev.RequestLine(l.chip, l.offset, + gpiocdev.AsInput, + gpiocdev.WithConsumer(consumer)) + if err != nil { + return fmt.Errorf("request %s line %d: %w", l.chip, l.offset, err) + } + l.line = line + return nil +} + +// Close releases the line. It is safe to call more than once. +func (l *Line) Close() error { + if l.line == nil { + return nil + } + line := l.line + l.line = nil + return line.Close() +} + +// Read reports whether mains power is present. +// +// On error the line is released so the next Read requests a fresh one. +// +// A read that succeeds is not proof that the value is meaningful: a floating +// or misconfigured line returns 0 with no error. Callers that act on "no AC" +// must corroborate it, not trust this alone. +func (l *Line) Read() (acPresent bool, err error) { + if err := l.ensureOpen(); err != nil { + return false, err + } + defer func() { + if err != nil { + _ = l.Close() + } + }() + + v, err := l.line.Value() + if err != nil { + return false, fmt.Errorf("read %s line %d: %w", l.chip, l.offset, err) + } + return v != 0, nil +} diff --git a/internal/sensor/fake.go b/internal/sensor/fake.go new file mode 100644 index 0000000..0756f42 --- /dev/null +++ b/internal/sensor/fake.go @@ -0,0 +1,88 @@ +package sensor + +import ( + "fmt" + "sync" + "time" +) + +// Step is one scripted outcome for a Fake. +type Step struct { + Reading Reading + Err error + // Block holds the Read for this long before returning, so callers can + // exercise their read deadline. + Block time.Duration +} + +// Fake is a scripted Sensor for tests and for `just run-fake`. It replays +// Steps in order and repeats the last one once the script runs out. +type Fake struct { + mu sync.Mutex + steps []Step + index int + closed bool + // Now supplies the timestamp when a Step leaves Reading.At zero. + Now func() time.Time +} + +// NewFake returns a Fake that replays steps. +func NewFake(steps ...Step) *Fake { + return &Fake{steps: steps, Now: time.Now} +} + +// Read returns the next scripted outcome. +func (f *Fake) Read() (Reading, error) { + f.mu.Lock() + if f.closed { + f.mu.Unlock() + return Reading{}, fmt.Errorf("read after close") + } + if len(f.steps) == 0 { + f.mu.Unlock() + return Reading{}, fmt.Errorf("fake sensor has no steps") + } + step := f.steps[f.index] + if f.index < len(f.steps)-1 { + f.index++ + } + now := f.Now + f.mu.Unlock() + + if step.Block > 0 { + time.Sleep(step.Block) + } + if step.Err != nil { + return Reading{}, step.Err + } + r := step.Reading + if r.At.IsZero() { + r.At = now() + } + return r, nil +} + +// Close marks the Fake closed; later reads fail. +func (f *Fake) Close() error { + f.mu.Lock() + defer f.mu.Unlock() + f.closed = true + return nil +} + +// Discharging returns a script that starts on mains and then drains, which is +// the sequence the shutdown state machine must accept. +func Discharging(mainsTicks, batteryTicks int) *Fake { + steps := make([]Step, 0, mainsTicks+batteryTicks) + for i := 0; i < mainsTicks; i++ { + steps = append(steps, Step{Reading: Reading{ACPresent: true, Volts: 4.10, SOC: 100}}) + } + for i := 0; i < batteryTicks; i++ { + steps = append(steps, Step{Reading: Reading{ + ACPresent: false, + Volts: 3.60 - float64(i)*0.02, + SOC: float64(30 - i), + }}) + } + return NewFake(steps...) +} diff --git a/internal/sensor/hardware.go b/internal/sensor/hardware.go new file mode 100644 index 0000000..1ccf8a0 --- /dev/null +++ b/internal/sensor/hardware.go @@ -0,0 +1,54 @@ +package sensor + +import ( + "errors" + "time" + + "git.mial.net/mokhtar/x1208-exporter/internal/max17040" + "git.mial.net/mokhtar/x1208-exporter/internal/pld" +) + +// Hardware composes the fuel gauge and the AC-present line into one Sensor. +// +// Each device owns its own reacquire-on-error policy, so a fault on one bus +// does not discard a healthy handle on the other. +type Hardware struct { + gauge *max17040.Gauge + line *pld.Line + now func() time.Time +} + +// NewHardware returns a Sensor backed by the real HAT. Neither device is opened +// until the first Read. +func NewHardware(i2cDevice, gpioChip string, gpioLine int) *Hardware { + return &Hardware{ + gauge: max17040.New(i2cDevice), + line: pld.New(gpioChip, gpioLine), + now: time.Now, + } +} + +// Read returns a complete sample or an error. It never returns a partially +// populated Reading: a sample that is missing a channel cannot be rendered or +// acted on, so there is nothing useful to pass upwards. +// +// Both channels are attempted even when the first fails, so one tick reports +// every fault rather than hiding the second behind the first. +func (h *Hardware) Read() (Reading, error) { + acPresent, acErr := h.line.Read() + volts, soc, batErr := h.gauge.Read() + if err := errors.Join(acErr, batErr); err != nil { + return Reading{}, err + } + return Reading{ + ACPresent: acPresent, + Volts: volts, + SOC: soc, + At: h.now(), + }, nil +} + +// Close releases both devices, reporting every failure. +func (h *Hardware) Close() error { + return errors.Join(h.line.Close(), h.gauge.Close()) +} diff --git a/internal/sensor/sensor.go b/internal/sensor/sensor.go new file mode 100644 index 0000000..f4f7ac3 --- /dev/null +++ b/internal/sensor/sensor.go @@ -0,0 +1,26 @@ +// Package sensor defines the hardware seam. Everything above this interface is +// pure and testable on any machine; everything below it touches the Pi. +package sensor + +import "time" + +// Reading is one complete sample. A Reading only exists if every channel was +// read successfully — there is no partial Reading, because a partial sample +// cannot be reasoned about downstream. +type Reading struct { + ACPresent bool + Volts float64 + SOC float64 + At time.Time +} + +// Sensor reads the UPS. Read returns an error if any channel fails; callers +// must treat the Reading as invalid in that case. +// +// Implementations may hold OS handles open across calls. A Read that returns +// an error must leave the Sensor usable: the next Read reacquires whatever it +// needs. +type Sensor interface { + Read() (Reading, error) + Close() error +} diff --git a/internal/textfile/textfile.go b/internal/textfile/textfile.go new file mode 100644 index 0000000..7340ade --- /dev/null +++ b/internal/textfile/textfile.go @@ -0,0 +1,85 @@ +// Package textfile renders and writes the Prometheus textfile that +// node-exporter's textfile collector scrapes. +package textfile + +import ( + "fmt" + "os" + "path/filepath" + "strings" + + "git.mial.net/mokhtar/x1208-exporter/internal/sensor" +) + +// Render returns the exposition text for the last complete sample. +// +// The first four metrics are byte-identical to the pre-daemon exporter. Their +// names and value formatting are referenced by Grafana alerts and by the +// dashboard, so they must not change. The golden test locks this. +// +// healthy reports the outcome of the most recent read attempt; failures counts +// failed attempts since process start. Both are additive and safe to scrape. +// +// When a read fails the caller re-renders the PREVIOUS sample with healthy=0. +// rpi_ups_last_update_seconds therefore stops advancing, which is what the +// RpiUpsExporterStale alert measures, while the failure stays visible instead +// of being hidden behind a frozen file. +func Render(s sensor.Reading, healthy bool, failures uint64) string { + ac := 0 + if s.ACPresent { + ac = 1 + } + health := 0 + if healthy { + health = 1 + } + + var b strings.Builder + b.WriteString("# HELP rpi_ups_last_update_seconds Unix timestamp of last successful exporter run.\n") + b.WriteString("# TYPE rpi_ups_last_update_seconds gauge\n") + fmt.Fprintf(&b, "rpi_ups_last_update_seconds %d\n", s.At.Unix()) + b.WriteString("# HELP rpi_ups_ac_power AC adapter present (1=plugged in, 0=on battery).\n") + b.WriteString("# TYPE rpi_ups_ac_power gauge\n") + fmt.Fprintf(&b, "rpi_ups_ac_power %d\n", ac) + b.WriteString("# HELP rpi_ups_voltage_volts Battery cell voltage from MAX17040 fuel gauge.\n") + b.WriteString("# TYPE rpi_ups_voltage_volts gauge\n") + fmt.Fprintf(&b, "rpi_ups_voltage_volts %.3f\n", s.Volts) + b.WriteString("# HELP rpi_ups_battery_percent Battery state of charge from MAX17040 fuel gauge.\n") + b.WriteString("# TYPE rpi_ups_battery_percent gauge\n") + fmt.Fprintf(&b, "rpi_ups_battery_percent %.2f\n", s.SOC) + b.WriteString("# HELP rpi_ups_sensor_healthy Whether the most recent UPS read succeeded (1=yes, 0=no).\n") + b.WriteString("# TYPE rpi_ups_sensor_healthy gauge\n") + fmt.Fprintf(&b, "rpi_ups_sensor_healthy %d\n", health) + b.WriteString("# HELP rpi_ups_read_failures_total Cumulative failed UPS reads since exporter start.\n") + b.WriteString("# TYPE rpi_ups_read_failures_total counter\n") + fmt.Fprintf(&b, "rpi_ups_read_failures_total %d\n", failures) + return b.String() +} + +// WriteAtomic writes content to path via a temporary file and a rename, so a +// scrape never observes a half-written file. +func WriteAtomic(path, content string) error { + dir := filepath.Dir(path) + if err := os.MkdirAll(dir, 0o755); err != nil { + return err + } + tmp, err := os.CreateTemp(dir, ".x1208.*.tmp") + if err != nil { + return err + } + tmpPath := tmp.Name() + defer os.Remove(tmpPath) + + if _, err := tmp.WriteString(content); err != nil { + tmp.Close() + return err + } + if err := tmp.Chmod(0o644); err != nil { + tmp.Close() + return err + } + if err := tmp.Close(); err != nil { + return err + } + return os.Rename(tmpPath, path) +} diff --git a/internal/textfile/textfile_test.go b/internal/textfile/textfile_test.go new file mode 100644 index 0000000..016ff1e --- /dev/null +++ b/internal/textfile/textfile_test.go @@ -0,0 +1,125 @@ +package textfile + +import ( + "os" + "path/filepath" + "strings" + "testing" + "time" + + "git.mial.net/mokhtar/x1208-exporter/internal/sensor" +) + +func sample() sensor.Reading { + return sensor.Reading{ + ACPresent: true, + Volts: 4.0625, + SOC: 97.65625, + At: time.Unix(1754661600, 0), + } +} + +// The four original metrics must render exactly as the oneshot exporter did. +// Grafana alerts and the dashboard key on these names and this formatting. +// +// The sample values are exact binary fractions that land on a rounding tie, so +// this also locks the rounding mode: Go rounds half to even, which turns +// 4.0625 into 4.062 and 97.65625 into 97.66. +const goldenHealthy = `# HELP rpi_ups_last_update_seconds Unix timestamp of last successful exporter run. +# TYPE rpi_ups_last_update_seconds gauge +rpi_ups_last_update_seconds 1754661600 +# HELP rpi_ups_ac_power AC adapter present (1=plugged in, 0=on battery). +# TYPE rpi_ups_ac_power gauge +rpi_ups_ac_power 1 +# HELP rpi_ups_voltage_volts Battery cell voltage from MAX17040 fuel gauge. +# TYPE rpi_ups_voltage_volts gauge +rpi_ups_voltage_volts 4.062 +# HELP rpi_ups_battery_percent Battery state of charge from MAX17040 fuel gauge. +# TYPE rpi_ups_battery_percent gauge +rpi_ups_battery_percent 97.66 +# HELP rpi_ups_sensor_healthy Whether the most recent UPS read succeeded (1=yes, 0=no). +# TYPE rpi_ups_sensor_healthy gauge +rpi_ups_sensor_healthy 1 +# HELP rpi_ups_read_failures_total Cumulative failed UPS reads since exporter start. +# TYPE rpi_ups_read_failures_total counter +rpi_ups_read_failures_total 0 +` + +func TestRenderGolden(t *testing.T) { + got := Render(sample(), true, 0) + if got != goldenHealthy { + t.Errorf("render mismatch:\n--- got ---\n%s\n--- want ---\n%s", got, goldenHealthy) + } +} + +// A failed read must not advance the timestamp. That frozen value is exactly +// what RpiUpsExporterStale measures. +func TestRenderUnhealthyFreezesTimestamp(t *testing.T) { + last := sample() + got := Render(last, false, 3) + + if !strings.Contains(got, "rpi_ups_last_update_seconds 1754661600\n") { + t.Error("timestamp must stay at the last complete sample") + } + if !strings.Contains(got, "rpi_ups_sensor_healthy 0\n") { + t.Error("unhealthy sample must report rpi_ups_sensor_healthy 0") + } + if !strings.Contains(got, "rpi_ups_read_failures_total 3\n") { + t.Error("failure count must be visible while the sensor is failing") + } +} + +func TestRenderACAbsent(t *testing.T) { + s := sample() + s.ACPresent = false + if !strings.Contains(Render(s, true, 0), "rpi_ups_ac_power 0\n") { + t.Error("AC absent must render rpi_ups_ac_power 0") + } +} + +func TestWriteAtomicReplacesContent(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "sub", "x1208.prom") + + if err := WriteAtomic(path, "first\n"); err != nil { + t.Fatalf("first write: %v", err) + } + if err := WriteAtomic(path, "second\n"); err != nil { + t.Fatalf("second write: %v", err) + } + + got, err := os.ReadFile(path) + if err != nil { + t.Fatalf("read back: %v", err) + } + if string(got) != "second\n" { + t.Errorf("content = %q, want %q", got, "second\n") + } + + info, err := os.Stat(path) + if err != nil { + t.Fatalf("stat: %v", err) + } + if info.Mode().Perm() != 0o644 { + t.Errorf("mode = %v, want 0644 (node-exporter must be able to read it)", info.Mode().Perm()) + } +} + +// A rename-based write must never leave temporary files behind for the +// collector to trip over. +func TestWriteAtomicLeavesNoTempFiles(t *testing.T) { + dir := t.TempDir() + path := filepath.Join(dir, "x1208.prom") + for i := 0; i < 5; i++ { + if err := WriteAtomic(path, "x\n"); err != nil { + t.Fatalf("write %d: %v", i, err) + } + } + entries, err := os.ReadDir(dir) + if err != nil { + t.Fatal(err) + } + if len(entries) != 1 { + t.Errorf("directory holds %d entries, want 1", len(entries)) + } +} diff --git a/justfile b/justfile index edb700b..196bd7f 100644 --- a/justfile +++ b/justfile @@ -1,23 +1,40 @@ +# x1208-exporter — Geekworm X1208 UPS HAT exporter +# +# Deployment lives in the rpi.mial.net repo. There are no deploy recipes here. + default: @just --list +# Compile everything build: - GOOS=linux GOARCH=arm64 go build -ldflags="-s -w" -o build/x1208-exporter ./... + go build ./... -deploy: build - scp build/x1208-exporter rpi:/tmp/x1208-exporter - scp systemd/x1208-exporter.service systemd/x1208-exporter.timer rpi:/tmp/ - ssh rpi 'sudo install -m 0755 /tmp/x1208-exporter /usr/local/bin/x1208-exporter && \ - sudo install -m 0644 /tmp/x1208-exporter.service /etc/systemd/system/ && \ - sudo install -m 0644 /tmp/x1208-exporter.timer /etc/systemd/system/ && \ - sudo install -d -m 0755 -o root /var/lib/node-exporter/textfiles && \ - sudo systemctl daemon-reload && \ - sudo systemctl enable --now x1208-exporter.timer && \ - sudo systemctl start x1208-exporter.service && \ - cat /var/lib/node-exporter/textfiles/x1208.prom' +# Run the test suite (all pure packages; the hardware packages need a Pi) +test: + go test ./... -logs: - ssh rpi 'journalctl -u x1208-exporter.service -n 30 --no-pager' +test-verbose: + go test -v ./... -run-once: - ssh rpi 'sudo systemctl start x1208-exporter.service && cat /var/lib/node-exporter/textfiles/x1208.prom' +# Static checks +lint: + go vet ./... + gofmt -l . + +fmt: + gofmt -w . + +# Build the Nix package exactly as the Pi will +nix-build: + nix build .#x1208-exporter + +# Run the daemon against a scripted sensor. Touches no hardware and powers +# nothing off — the shutdown command is replaced with an echo. +run-fake out="/tmp/x1208.prom": + go run . -fake -interval 1s -out {{ out }} \ + -shutdown -shutdown-settle-ticks 2 -shutdown-samples 3 \ + -shutdown-cmd "/usr/bin/env echo POWEROFF-WOULD-RUN" + +# Print the metrics the fake run produced +show out="/tmp/x1208.prom": + @cat {{ out }} diff --git a/main.go b/main.go index 93693a8..b9e23d8 100644 --- a/main.go +++ b/main.go @@ -1,181 +1,270 @@ +// Command x1208-exporter reads the Geekworm X1208 UPS HAT and writes a +// Prometheus textfile for node-exporter's textfile collector. +// +// It runs as a long-running daemon under systemd (Type=notify). It replaces an +// earlier oneshot-plus-timer, which produced 8367 journal lines a day of +// systemd start/stop noise and a matching amount of SD card writeback. package main import ( - "encoding/binary" + "context" "flag" "fmt" + "log/slog" "os" - "path/filepath" + "os/exec" + "os/signal" + "strings" + "sync/atomic" + "syscall" "time" - "github.com/warthog618/go-gpiocdev" - "golang.org/x/sys/unix" + "github.com/coreos/go-systemd/v22/daemon" + + "git.mial.net/mokhtar/x1208-exporter/internal/max17040" + "git.mial.net/mokhtar/x1208-exporter/internal/monitor" + "git.mial.net/mokhtar/x1208-exporter/internal/pld" + "git.mial.net/mokhtar/x1208-exporter/internal/sensor" + "git.mial.net/mokhtar/x1208-exporter/internal/textfile" ) -const ( - i2cDev = "/dev/i2c-1" - gpioChip = "gpiochip0" - gpioPLDPin = 6 - maxAddr = 0x36 - regVoltage = 0x02 - regCapacity = 0x04 - i2cSlave = 0x0703 // I2C_SLAVE - maxRetries = 3 -) +// version is set at build time with -ldflags "-X main.version=vX.Y.Z". +var version = "dev" -func readI2CWord(fd int, reg byte) (uint16, error) { - if _, err := unix.Write(fd, []byte{reg}); err != nil { - return 0, fmt.Errorf("write reg: %w", err) - } - buf := make([]byte, 2) - n, err := unix.Read(fd, buf) - if err != nil { - return 0, fmt.Errorf("read: %w", err) - } - if n != 2 { - return 0, fmt.Errorf("short read: %d bytes", n) - } - return binary.BigEndian.Uint16(buf), nil +type options struct { + out string + interval time.Duration + readTimeout time.Duration + i2cDevice string + gpioChip string + gpioLine int + fake bool + + shutdown bool + shutdownVolts float64 + shutdownSOC float64 + shutdownSamples int + shutdownSettle int + shutdownDischarge bool + shutdownCmd string + + showVersion bool } -func readI2CWordRetry(fd int, reg byte, validate func(uint16) bool) (uint16, error) { - var lastErr error - for i := 0; i < maxRetries; i++ { - v, err := readI2CWord(fd, reg) - if err == nil && validate(v) { - return v, nil - } - if err != nil { - lastErr = err - } else { - lastErr = fmt.Errorf("read out of range: 0x%04x", v) - } - time.Sleep(20 * time.Millisecond) - } - return 0, lastErr -} +func parseFlags() *options { + o := &options{} + flag.StringVar(&o.out, "out", "/var/lib/node-exporter/textfiles/x1208.prom", "textfile path") + flag.DurationVar(&o.interval, "interval", 30*time.Second, "sample interval") + flag.DurationVar(&o.readTimeout, "read-timeout", 5*time.Second, "give up on a sensor read after this long") + flag.StringVar(&o.i2cDevice, "i2c-device", max17040.DefaultDevice, "i2c device holding the fuel gauge") + flag.StringVar(&o.gpioChip, "gpio-chip", pld.DefaultChip, "gpio chip carrying the AC-present line") + flag.IntVar(&o.gpioLine, "gpio-line", pld.DefaultLine, "gpio line number for AC-present") + flag.BoolVar(&o.fake, "fake", false, "use a scripted sensor instead of the HAT (development only)") -func readBattery() (voltage float64, percent float64, err error) { - fd, err := unix.Open(i2cDev, unix.O_RDWR, 0) - if err != nil { - return 0, 0, fmt.Errorf("open %s: %w", i2cDev, err) - } - defer unix.Close(fd) + flag.BoolVar(&o.shutdown, "shutdown", false, "power the machine off on a sustained low battery") + flag.Float64Var(&o.shutdownVolts, "shutdown-voltage", 3.5, "cell voltage that qualifies as low") + flag.Float64Var(&o.shutdownSOC, "shutdown-soc", 15, "state of charge percent that qualifies as low") + flag.IntVar(&o.shutdownSamples, "shutdown-samples", 4, "consecutive qualifying samples before acting") + flag.IntVar(&o.shutdownSettle, "shutdown-settle-ticks", 4, "suppress shutdown for this many ticks after start") + flag.BoolVar(&o.shutdownDischarge, "shutdown-require-discharge", true, + "require cell voltage to fall across the window; blocks a stuck sensor from powering the machine off") + flag.StringVar(&o.shutdownCmd, "shutdown-cmd", "/run/current-system/sw/bin/systemctl poweroff", + "command run to power the machine off") - if err := unix.IoctlSetInt(fd, i2cSlave, maxAddr); err != nil { - return 0, 0, fmt.Errorf("ioctl I2C_SLAVE: %w", err) - } - - rawV, err := readI2CWordRetry(fd, regVoltage, func(v uint16) bool { - f := float64(v) * 1.25 / 1000 / 16 - return f > 2.0 && f < 5.0 - }) - if err != nil { - return 0, 0, fmt.Errorf("voltage: %w", err) - } - voltage = float64(rawV) * 1.25 / 1000 / 16 - - rawC, err := readI2CWordRetry(fd, regCapacity, func(v uint16) bool { - return float64(v)/256 <= 110 - }) - if err != nil { - return 0, 0, fmt.Errorf("capacity: %w", err) - } - percent = float64(rawC) / 256 - return voltage, percent, nil -} - -func readACPower() (int, error) { - line, err := gpiocdev.RequestLine(gpioChip, gpioPLDPin, - gpiocdev.AsInput, - gpiocdev.WithConsumer("x1208-exporter")) - if err != nil { - return 0, fmt.Errorf("request gpio: %w", err) - } - defer line.Close() - - v, err := line.Value() - if err != nil { - return 0, fmt.Errorf("read gpio: %w", err) - } - return v, nil -} - -func writeAtomic(path string, content string) error { - dir := filepath.Dir(path) - if err := os.MkdirAll(dir, 0o755); err != nil { - return err - } - tmp, err := os.CreateTemp(dir, ".x1208.*.tmp") - if err != nil { - return err - } - tmpPath := tmp.Name() - defer os.Remove(tmpPath) - if _, err := tmp.WriteString(content); err != nil { - tmp.Close() - return err - } - if err := tmp.Chmod(0o644); err != nil { - tmp.Close() - return err - } - if err := tmp.Close(); err != nil { - return err - } - return os.Rename(tmpPath, path) + flag.BoolVar(&o.showVersion, "version", false, "print the version and exit") + flag.Parse() + return o } func main() { - out := flag.String("out", "/var/lib/node-exporter/textfiles/x1208.prom", "output prom file") - flag.Parse() - - var ( - ac int - voltage float64 - percent float64 - acErr, batErr error - ) - - ac, acErr = readACPower() - voltage, percent, batErr = readBattery() - - if acErr != nil && batErr != nil { - fmt.Fprintf(os.Stderr, "all reads failed: ac=%v bat=%v\n", acErr, batErr) - os.Exit(1) + opts := parseFlags() + if opts.showVersion { + fmt.Println(version) + return } - now := time.Now().Unix() - var content string - content += "# HELP rpi_ups_last_update_seconds Unix timestamp of last successful exporter run.\n" - content += "# TYPE rpi_ups_last_update_seconds gauge\n" - content += fmt.Sprintf("rpi_ups_last_update_seconds %d\n", now) - - if acErr == nil { - content += "# HELP rpi_ups_ac_power AC adapter present (1=plugged in, 0=on battery).\n" - content += "# TYPE rpi_ups_ac_power gauge\n" - content += fmt.Sprintf("rpi_ups_ac_power %d\n", ac) - } else { - fmt.Fprintf(os.Stderr, "ac read failed: %v\n", acErr) - } - - if batErr == nil { - content += "# HELP rpi_ups_voltage_volts Battery cell voltage from MAX17040 fuel gauge.\n" - content += "# TYPE rpi_ups_voltage_volts gauge\n" - content += fmt.Sprintf("rpi_ups_voltage_volts %.3f\n", voltage) - content += "# HELP rpi_ups_battery_percent Battery state of charge from MAX17040 fuel gauge.\n" - content += "# TYPE rpi_ups_battery_percent gauge\n" - content += fmt.Sprintf("rpi_ups_battery_percent %.2f\n", percent) - } else { - fmt.Fprintf(os.Stderr, "battery read failed: %v\n", batErr) - } - - if err := writeAtomic(*out, content); err != nil { - fmt.Fprintf(os.Stderr, "write %s: %v\n", *out, err) - os.Exit(1) - } - - if acErr != nil || batErr != nil { + log := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo})) + if err := run(opts, log); err != nil { + log.Error("exporter failed", "err", err) os.Exit(1) } } + +func run(opts *options, log *slog.Logger) error { + var src sensor.Sensor + if opts.fake { + src = sensor.Discharging(3, 40) + log.Warn("using the fake sensor; no hardware is being read") + } else { + src = sensor.NewHardware(opts.i2cDevice, opts.gpioChip, opts.gpioLine) + } + defer src.Close() + + mon := monitor.New(monitor.Config{ + Shutdown: opts.shutdown, + Volts: opts.shutdownVolts, + SOC: opts.shutdownSOC, + Samples: opts.shutdownSamples, + MinUptimeTicks: opts.shutdownSettle, + RequireDischarge: opts.shutdownDischarge, + }) + + ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM) + defer stop() + + // lastTick records when the sample loop last completed a full pass. The + // watchdog pings from this, not from its own liveness — a watchdog + // goroutine that reports health while the sample loop is wedged is worse + // than no watchdog at all. + var lastTick atomic.Int64 + lastTick.Store(time.Now().UnixNano()) + + stopWatchdog := startWatchdog(ctx, &lastTick, opts.interval, log) + defer stopWatchdog() + + if _, err := daemon.SdNotify(false, daemon.SdNotifyReady); err != nil { + log.Warn("could not notify systemd of readiness", "err", err) + } + + ticker := time.NewTicker(opts.interval) + defer ticker.Stop() + + for { + tick(ctx, src, mon, opts, log) + lastTick.Store(time.Now().UnixNano()) + + select { + case <-ctx.Done(): + log.Info("shutting down on signal") + return nil + case <-ticker.C: + } + } +} + +// tick performs one sample, writes the textfile, emits transition logs, and +// acts on a shutdown decision. +func tick(ctx context.Context, src sensor.Sensor, mon *monitor.Monitor, opts *options, log *slog.Logger) { + reading, err := readWithTimeout(ctx, src, opts.readTimeout) + res := mon.Tick(reading, err) + + for _, entry := range res.Logs { + switch entry.Event { + case monitor.EventErrorEnter, monitor.EventShutdownArmed, + monitor.EventShutdownInhibited, monitor.EventShutdownTriggered: + log.Warn(entry.Message, "event", string(entry.Event)) + default: + log.Info(entry.Message, "event", string(entry.Event)) + } + } + + // Nothing to publish until the first complete sample. Writing a zeroed + // file would report 1970 as the last update and a flat battery. + if res.HaveSample { + content := textfile.Render(res.Sample, res.Healthy, res.Failures) + if err := textfile.WriteAtomic(opts.out, content); err != nil { + log.Error("could not write textfile", "path", opts.out, "err", err) + } + } + + if res.Shutdown { + powerOff(opts.shutdownCmd, log) + } +} + +// readWithTimeout bounds a sensor read. +// +// A blocked i2c or GPIO operation would otherwise stall the loop until the +// systemd watchdog killed the whole process. The read goroutine is abandoned +// rather than cancelled: the kernel owns the call, and a leaked goroutine per +// wedged read is a smaller problem than a stalled sample loop. +func readWithTimeout(ctx context.Context, src sensor.Sensor, timeout time.Duration) (sensor.Reading, error) { + type outcome struct { + reading sensor.Reading + err error + } + done := make(chan outcome, 1) + go func() { + r, err := src.Read() + done <- outcome{r, err} + }() + + timer := time.NewTimer(timeout) + defer timer.Stop() + + select { + case o := <-done: + return o.reading, o.err + case <-timer.C: + return sensor.Reading{}, fmt.Errorf("sensor read exceeded %s", timeout) + case <-ctx.Done(): + return sensor.Reading{}, ctx.Err() + } +} + +// startWatchdog pings systemd only while the sample loop is making progress. +// +// The cadence comes from WATCHDOG_USEC, halved, as systemd documents. If the +// unit sets no watchdog, this does nothing. +func startWatchdog(ctx context.Context, lastTick *atomic.Int64, interval time.Duration, log *slog.Logger) func() { + timeout, err := daemon.SdWatchdogEnabled(false) + if err != nil || timeout == 0 { + return func() {} + } + + // A tick is considered fresh for two intervals, so one slow sample does + // not trip the watchdog while a genuinely wedged loop still does. + stale := 2 * interval + if stale >= timeout { + log.Warn("sample interval is too long for the configured watchdog; "+ + "systemd will restart the daemon during normal operation", + "interval", interval, "watchdog", timeout) + } + + ticker := time.NewTicker(timeout / 2) + done := make(chan struct{}) + go func() { + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return + case <-done: + return + case <-ticker.C: + last := time.Unix(0, lastTick.Load()) + if time.Since(last) > stale { + // Deliberately silent: withholding the ping IS the + // report. systemd restarts the daemon. + continue + } + if _, err := daemon.SdNotify(false, daemon.SdNotifyWatchdog); err != nil { + log.Warn("watchdog ping failed", "err", err) + } + } + } + }() + return func() { close(done) } +} + +func powerOff(command string, log *slog.Logger) { + fields := strings.Fields(command) + if len(fields) == 0 { + log.Error("shutdown requested but -shutdown-cmd is empty") + return + } + + if _, err := daemon.SdNotify(false, daemon.SdNotifyStopping); err != nil { + log.Warn("could not notify systemd that we are stopping", "err", err) + } + + out, err := exec.Command(fields[0], fields[1:]...).CombinedOutput() + if err != nil { + // Do not retry. A poweroff that fails needs a human, and a retry loop + // would fork repeatedly while the battery drains. + log.Error("poweroff command failed", + "cmd", command, "err", err, "output", strings.TrimSpace(string(out))) + return + } + log.Warn("poweroff requested", "cmd", command) +} diff --git a/monitoring-overlay/README.md b/monitoring-overlay/README.md deleted file mode 100644 index 91aff57..0000000 --- a/monitoring-overlay/README.md +++ /dev/null @@ -1,26 +0,0 @@ -# Monitoring overlay for the Pi - -These two files are the desired state of `/home/mokhtar/app/monitoring/` on the Pi -after the X1208 exporter is in place. The `.original` siblings are snapshots of -what's there right now (pulled with `ssh rpi cat ...`). - -## Diff summary - -**`compose.yaml`** — node-exporter service: - -- Add volume `/var/lib/node-exporter/textfiles:/textfiles:ro` -- Add command flag `--collector.textfile.directory=/textfiles` - -**`prometheus.yaml`** — global block: - -- Add `external_labels: { host: rpi }` so metrics are tagged in Grafana Cloud - (instead of relying on the Docker service name as `instance`). - -## Apply - -```bash -scp compose.yaml prometheus.yaml rpi:/home/mokhtar/app/monitoring/ -ssh rpi 'cd /home/mokhtar/app/monitoring && docker compose up -d' -``` - -This recreates the node-exporter container (~2s blip) and reloads Prometheus. diff --git a/monitoring-overlay/compose.yaml b/monitoring-overlay/compose.yaml deleted file mode 100644 index 656393d..0000000 --- a/monitoring-overlay/compose.yaml +++ /dev/null @@ -1,42 +0,0 @@ -services: - node-exporter: - image: prom/node-exporter:latest - hostname: node-exporter - restart: unless-stopped - volumes: - - /proc:/host/proc:ro - - /sys:/host/sys:ro - - /:/rootfs:ro - - /var/lib/node-exporter/textfiles:/textfiles:ro - command: - - '--path.procfs=/host/proc' - - '--path.rootfs=/rootfs' - - '--path.sysfs=/host/sys' - - '--collector.filesystem.mount-points-exclude=^/(sys|proc|dev|host|etc)($$|/)' - - '--collector.textfile.directory=/textfiles' - - '--web.telemetry-path=/_prom/metrics' - expose: - - 9100 - networks: - - monitoring-network - logging: - driver: journald - options: - tag: "container-{{.Name}}" - - prometheus: - image: prom/prometheus:latest - restart: unless-stopped - depends_on: - - node-exporter - volumes: - - "./prometheus.yaml:/etc/prometheus/prometheus.yml" - networks: - - monitoring-network - logging: - driver: journald - options: - tag: "container-{{.Name}}" - -networks: - monitoring-network: diff --git a/monitoring-overlay/prometheus.yaml b/monitoring-overlay/prometheus.yaml deleted file mode 100644 index 3d16e78..0000000 --- a/monitoring-overlay/prometheus.yaml +++ /dev/null @@ -1,16 +0,0 @@ -global: - scrape_interval: 60s - external_labels: - host: rpi - -scrape_configs: - - job_name: scrape_mokhtar-rpi - metrics_path: /_prom/metrics - static_configs: - - targets: ['node-exporter:9100'] - -remote_write: - - url: 'https://prometheus-prod-39-prod-eu-north-0.grafana.net/api/prom/push' - basic_auth: - username: '1830601' - password: 'glc_eyJvIjoiNTM3NDYzIiwibiI6InN0YWNrLTEwNTY4MDYtaG0td3JpdGUtcHJvbWV0aGV1cy1ycGkiLCJrIjoidDlXblM2ME1EMTNNMEpKM2IycDBTMWt1IiwibSI6eyJyIjoicHJvZC1ldS1ub3J0aC0wIn19' diff --git a/systemd/x1208-exporter.service b/systemd/x1208-exporter.service deleted file mode 100644 index 45b0b6b..0000000 --- a/systemd/x1208-exporter.service +++ /dev/null @@ -1,15 +0,0 @@ -[Unit] -Description=Geekworm X1208 UPS metrics exporter (one-shot) -After=local-fs.target - -[Service] -Type=oneshot -ExecStart=/usr/local/bin/x1208-exporter -Nice=10 -ProtectSystem=strict -ReadWritePaths=/var/lib/node-exporter/textfiles -ProtectHome=true -PrivateTmp=true -NoNewPrivileges=true -DeviceAllow=/dev/i2c-1 rw -DeviceAllow=/dev/gpiochip0 rw diff --git a/systemd/x1208-exporter.timer b/systemd/x1208-exporter.timer deleted file mode 100644 index d40a168..0000000 --- a/systemd/x1208-exporter.timer +++ /dev/null @@ -1,11 +0,0 @@ -[Unit] -Description=Run X1208 UPS metrics exporter every 30s - -[Timer] -OnBootSec=30s -OnUnitActiveSec=30s -AccuracySec=1s -Unit=x1208-exporter.service - -[Install] -WantedBy=timers.target