mokhtar
e4c34784c8
the oneshot+timer produced 8367 journal lines a day of systemd start/stop noise. now Type=notify with an internal ticker, logging transitions only. hardware sits behind sensor.Sensor; internal/monitor is pure, so the shutdown state machine is tested without a pi. four guards, each tested, because a false poweroff of a box with no physical access is worse than a missed one: - a failed read resets the arming window rather than pausing it - voltage must actually fall across the window, else it is a stuck sensor. this is the only defence against the ac line reading 0 while mains is connected, so it is load-bearing - a settle period stops a restart loop acting on early readings - flapping ac cannot accumulate verified on hardware 2026-08-09: after poweroff on battery the x1208 starts the pi again when mains returns, with no button press. that was the risk that could have made this feature unsafe. a failed tick rewrites the previous sample with sensor_healthy 0 and the failure counter, so last_update still freezes for the staleness alert while the failure stays visible. module path moved to git.mial.net. debian units and the compose overlay deleted; deployment lives in the infra repo.
130 lines
5.4 KiB
Markdown
130 lines
5.4 KiB
Markdown
# x1208-exporter
|
|
|
|
A Prometheus exporter for the [Geekworm X1208](https://wiki.geekworm.com/X1208)
|
|
UPS HAT on a Raspberry Pi 5.
|
|
|
|
It reads cell voltage and state of charge from the MAX17040 fuel gauge over
|
|
i2c, reads the power-loss-detect line over GPIO, and writes a Prometheus
|
|
textfile that node-exporter's textfile collector scrapes.
|
|
|
|
## Metrics
|
|
|
|
| Metric | Type | Meaning |
|
|
| --- | --- | --- |
|
|
| `rpi_ups_last_update_seconds` | gauge | Unix time of the last COMPLETE sample |
|
|
| `rpi_ups_ac_power` | gauge | 1 on mains, 0 on cells |
|
|
| `rpi_ups_voltage_volts` | gauge | Cell voltage |
|
|
| `rpi_ups_battery_percent` | gauge | State of charge |
|
|
| `rpi_ups_sensor_healthy` | gauge | 1 if the most recent read succeeded |
|
|
| `rpi_ups_read_failures_total` | counter | Failed reads since start |
|
|
|
|
The first four names are load-bearing. Grafana alerts and the Node Exporter
|
|
Full dashboard reference them. A golden test locks their exact rendering,
|
|
including the rounding mode. Do not rename them and do not change their
|
|
formatting.
|
|
|
|
## Failure behaviour
|
|
|
|
A read either produces a complete sample or fails. There is no partial sample.
|
|
|
|
When a read fails the exporter rewrites the file with the **previous** complete
|
|
sample, sets `rpi_ups_sensor_healthy` to 0, and increments the failure counter.
|
|
`rpi_ups_last_update_seconds` therefore stops advancing, which is what the
|
|
`RpiUpsExporterStale` alert measures. The failure stays visible in the
|
|
meantime rather than hiding behind a frozen file.
|
|
|
|
The exporter logs transitions only: startup, AC changes, entering and leaving
|
|
an error state, and shutdown decisions. A healthy sample logs nothing. This
|
|
matters because the machine keeps its journal in RAM and archives it to an SD
|
|
card.
|
|
|
|
## Shutdown on low battery
|
|
|
|
The exporter can power the machine off when the cells run down. It is **off by
|
|
default** and must be enabled with `-shutdown`.
|
|
|
|
It acts only when all of these hold:
|
|
|
|
- The AC line reports no mains power.
|
|
- Cell voltage is below `-shutdown-voltage`, or state of charge is below
|
|
`-shutdown-soc`.
|
|
- That has held for `-shutdown-samples` consecutive ticks.
|
|
- More than `-shutdown-settle-ticks` ticks have passed since start.
|
|
- Cell voltage actually fell across the window, unless
|
|
`-shutdown-require-discharge=false`.
|
|
|
|
Each guard exists because a false poweroff is worse than a missed one. The
|
|
machine it runs on has no physical access, so a wrong decision that halts it
|
|
cannot be undone remotely.
|
|
|
|
- A **failed read resets** the window. It does not pause it. Otherwise "low,
|
|
low, a long outage, low, low" would reach the sample count without the
|
|
battery ever being low for that period.
|
|
- The **discharge requirement** is load-bearing. Do not turn it off.
|
|
|
|
It separates a real discharge from a stuck sensor: a wedged fuel gauge
|
|
repeats one plausible low value forever, and four samples of one persistent
|
|
fault are not four independent confirmations.
|
|
|
|
It is also the only defence against the worst remaining failure, an AC line
|
|
that reads 0 while mains is actually connected. A floating GPIO line reports
|
|
"on battery" and returns no error. On mains the charger holds cell voltage
|
|
flat or rising, so a false reading cannot satisfy this check.
|
|
`TestFalseACLossWhileChargingNeverTriggersShutdown` covers it, and
|
|
`TestWithoutDischargeGuardAFalseACReadingIsEnough` records what happens if
|
|
someone disables the guard.
|
|
- The **settle period** stops a restart loop from acting on readings taken
|
|
before the hardware settled.
|
|
|
|
### The HAT does restore power after a software shutdown
|
|
|
|
This was the one question that could have made the whole feature unsafe. The
|
|
X1208 cuts power to the Pi after halt. If mains returned while the board still
|
|
held charge, and the board did not then start the Pi, a clean shutdown would
|
|
strand a machine nobody can reach.
|
|
|
|
Tested on 2026-08-09 against the real hardware:
|
|
|
|
1. On mains, `rpi_ups_ac_power` read 1 at 4.149 V.
|
|
2. Mains disconnected at the wall. `rpi_ups_ac_power` read 0 at 4.109 V.
|
|
3. `sudo systemctl poweroff`.
|
|
4. Mains reconnected. **No button was pressed.**
|
|
5. The Pi booted by itself.
|
|
|
|
The same run measured the discharge rate under normal load: 40 mV per minute.
|
|
Across a four-tick window at 30 s that is about 80 mV, against a gauge
|
|
resolution of 1.25 mV. The discharge guard therefore has a wide margin and
|
|
cannot block a real discharge.
|
|
|
|
**Still untested:** a poweroff while mains stayed connected the whole time.
|
|
The test above disconnected mains first. If the board treats "input power
|
|
never dropped" differently, that path is unproven. It only matters if a false
|
|
AC reading gets past the discharge guard, which is why that guard is
|
|
load-bearing. Test it during the soak: run `systemctl poweroff` with mains
|
|
connected and record whether the Pi returns.
|
|
|
|
## Development
|
|
|
|
```sh
|
|
just test # unit tests; no hardware needed
|
|
just lint
|
|
just run-fake # daemon against a scripted discharging battery
|
|
just show
|
|
just nix-build # build the package as the Pi will
|
|
```
|
|
|
|
The hardware sits behind `sensor.Sensor`. `internal/monitor` holds every
|
|
decision and is pure, so the shutdown state machine is tested without a Pi.
|
|
`internal/max17040` and `internal/pld` are the only packages that touch
|
|
devices.
|
|
|
|
## Deployment
|
|
|
|
This repository ships the **package only**. The systemd unit, device access,
|
|
groups, polkit rules and the decision to enable shutdown are host policy and
|
|
live in the infrastructure repository. Keeping a unit here as well would create
|
|
two sources of truth.
|
|
|
|
Consume it as a flake input and let the infrastructure repository define the
|
|
service.
|