rewrite as a daemon with shutdown on low battery
the oneshot+timer produced 8367 journal lines a day of systemd start/stop noise. now Type=notify with an internal ticker, logging transitions only. hardware sits behind sensor.Sensor; internal/monitor is pure, so the shutdown state machine is tested without a pi. four guards, each tested, because a false poweroff of a box with no physical access is worse than a missed one: - a failed read resets the arming window rather than pausing it - voltage must actually fall across the window, else it is a stuck sensor. this is the only defence against the ac line reading 0 while mains is connected, so it is load-bearing - a settle period stops a restart loop acting on early readings - flapping ac cannot accumulate verified on hardware 2026-08-09: after poweroff on battery the x1208 starts the pi again when mains returns, with no button press. that was the risk that could have made this feature unsafe. a failed tick rewrites the previous sample with sensor_healthy 0 and the failure counter, so last_update still freezes for the staleness alert while the failure stays visible. module path moved to git.mial.net. debian units and the compose overlay deleted; deployment lives in the infra repo.
This commit is contained in:
@@ -1,49 +1,129 @@
|
||||
# x1208-exporter
|
||||
|
||||
One-shot Prometheus textfile exporter for the Geekworm X1208 UPS HAT on Raspberry Pi 5.
|
||||
A Prometheus exporter for the [Geekworm X1208](https://wiki.geekworm.com/X1208)
|
||||
UPS HAT on a Raspberry Pi 5.
|
||||
|
||||
Reads:
|
||||
|
||||
- GPIO 6 (PLD) — `1` = AC plugged in, `0` = on battery.
|
||||
- I2C `0x36` (MAX17040 fuel gauge) — battery voltage and state of charge.
|
||||
|
||||
Writes `/var/lib/node-exporter/textfiles/x1208.prom` atomically. node-exporter's
|
||||
textfile collector serves it at scrape time.
|
||||
It reads cell voltage and state of charge from the MAX17040 fuel gauge over
|
||||
i2c, reads the power-loss-detect line over GPIO, and writes a Prometheus
|
||||
textfile that node-exporter's textfile collector scrapes.
|
||||
|
||||
## Metrics
|
||||
|
||||
| Name | Type | Description |
|
||||
| ------------------------------- | ----- | -------------------------------------------- |
|
||||
| `rpi_ups_ac_power` | gauge | `1` = AC present, `0` = on battery |
|
||||
| `rpi_ups_voltage_volts` | gauge | Battery cell voltage |
|
||||
| `rpi_ups_battery_percent` | gauge | State of charge (0–100) |
|
||||
| `rpi_ups_last_update_seconds` | gauge | Unix timestamp of last exporter run |
|
||||
| Metric | Type | Meaning |
|
||||
| --- | --- | --- |
|
||||
| `rpi_ups_last_update_seconds` | gauge | Unix time of the last COMPLETE sample |
|
||||
| `rpi_ups_ac_power` | gauge | 1 on mains, 0 on cells |
|
||||
| `rpi_ups_voltage_volts` | gauge | Cell voltage |
|
||||
| `rpi_ups_battery_percent` | gauge | State of charge |
|
||||
| `rpi_ups_sensor_healthy` | gauge | 1 if the most recent read succeeded |
|
||||
| `rpi_ups_read_failures_total` | counter | Failed reads since start |
|
||||
|
||||
## Deploy
|
||||
The first four names are load-bearing. Grafana alerts and the Node Exporter
|
||||
Full dashboard reference them. A golden test locks their exact rendering,
|
||||
including the rounding mode. Do not rename them and do not change their
|
||||
formatting.
|
||||
|
||||
```bash
|
||||
just deploy
|
||||
## Failure behaviour
|
||||
|
||||
A read either produces a complete sample or fails. There is no partial sample.
|
||||
|
||||
When a read fails the exporter rewrites the file with the **previous** complete
|
||||
sample, sets `rpi_ups_sensor_healthy` to 0, and increments the failure counter.
|
||||
`rpi_ups_last_update_seconds` therefore stops advancing, which is what the
|
||||
`RpiUpsExporterStale` alert measures. The failure stays visible in the
|
||||
meantime rather than hiding behind a frozen file.
|
||||
|
||||
The exporter logs transitions only: startup, AC changes, entering and leaving
|
||||
an error state, and shutdown decisions. A healthy sample logs nothing. This
|
||||
matters because the machine keeps its journal in RAM and archives it to an SD
|
||||
card.
|
||||
|
||||
## Shutdown on low battery
|
||||
|
||||
The exporter can power the machine off when the cells run down. It is **off by
|
||||
default** and must be enabled with `-shutdown`.
|
||||
|
||||
It acts only when all of these hold:
|
||||
|
||||
- The AC line reports no mains power.
|
||||
- Cell voltage is below `-shutdown-voltage`, or state of charge is below
|
||||
`-shutdown-soc`.
|
||||
- That has held for `-shutdown-samples` consecutive ticks.
|
||||
- More than `-shutdown-settle-ticks` ticks have passed since start.
|
||||
- Cell voltage actually fell across the window, unless
|
||||
`-shutdown-require-discharge=false`.
|
||||
|
||||
Each guard exists because a false poweroff is worse than a missed one. The
|
||||
machine it runs on has no physical access, so a wrong decision that halts it
|
||||
cannot be undone remotely.
|
||||
|
||||
- A **failed read resets** the window. It does not pause it. Otherwise "low,
|
||||
low, a long outage, low, low" would reach the sample count without the
|
||||
battery ever being low for that period.
|
||||
- The **discharge requirement** is load-bearing. Do not turn it off.
|
||||
|
||||
It separates a real discharge from a stuck sensor: a wedged fuel gauge
|
||||
repeats one plausible low value forever, and four samples of one persistent
|
||||
fault are not four independent confirmations.
|
||||
|
||||
It is also the only defence against the worst remaining failure, an AC line
|
||||
that reads 0 while mains is actually connected. A floating GPIO line reports
|
||||
"on battery" and returns no error. On mains the charger holds cell voltage
|
||||
flat or rising, so a false reading cannot satisfy this check.
|
||||
`TestFalseACLossWhileChargingNeverTriggersShutdown` covers it, and
|
||||
`TestWithoutDischargeGuardAFalseACReadingIsEnough` records what happens if
|
||||
someone disables the guard.
|
||||
- The **settle period** stops a restart loop from acting on readings taken
|
||||
before the hardware settled.
|
||||
|
||||
### The HAT does restore power after a software shutdown
|
||||
|
||||
This was the one question that could have made the whole feature unsafe. The
|
||||
X1208 cuts power to the Pi after halt. If mains returned while the board still
|
||||
held charge, and the board did not then start the Pi, a clean shutdown would
|
||||
strand a machine nobody can reach.
|
||||
|
||||
Tested on 2026-08-09 against the real hardware:
|
||||
|
||||
1. On mains, `rpi_ups_ac_power` read 1 at 4.149 V.
|
||||
2. Mains disconnected at the wall. `rpi_ups_ac_power` read 0 at 4.109 V.
|
||||
3. `sudo systemctl poweroff`.
|
||||
4. Mains reconnected. **No button was pressed.**
|
||||
5. The Pi booted by itself.
|
||||
|
||||
The same run measured the discharge rate under normal load: 40 mV per minute.
|
||||
Across a four-tick window at 30 s that is about 80 mV, against a gauge
|
||||
resolution of 1.25 mV. The discharge guard therefore has a wide margin and
|
||||
cannot block a real discharge.
|
||||
|
||||
**Still untested:** a poweroff while mains stayed connected the whole time.
|
||||
The test above disconnected mains first. If the board treats "input power
|
||||
never dropped" differently, that path is unproven. It only matters if a false
|
||||
AC reading gets past the discharge guard, which is why that guard is
|
||||
load-bearing. Test it during the soak: run `systemctl poweroff` with mains
|
||||
connected and record whether the Pi returns.
|
||||
|
||||
## Development
|
||||
|
||||
```sh
|
||||
just test # unit tests; no hardware needed
|
||||
just lint
|
||||
just run-fake # daemon against a scripted discharging battery
|
||||
just show
|
||||
just nix-build # build the package as the Pi will
|
||||
```
|
||||
|
||||
Builds the arm64 binary, scps to `rpi`, installs systemd units, starts the timer.
|
||||
The hardware sits behind `sensor.Sensor`. `internal/monitor` holds every
|
||||
decision and is pure, so the shutdown state machine is tested without a Pi.
|
||||
`internal/max17040` and `internal/pld` are the only packages that touch
|
||||
devices.
|
||||
|
||||
## Verify
|
||||
## Deployment
|
||||
|
||||
```bash
|
||||
just run-once # forces one read, prints the prom file
|
||||
just logs # journalctl tail
|
||||
```
|
||||
This repository ships the **package only**. The systemd unit, device access,
|
||||
groups, polkit rules and the decision to enable shutdown are host policy and
|
||||
live in the infrastructure repository. Keeping a unit here as well would create
|
||||
two sources of truth.
|
||||
|
||||
## How node-exporter picks it up
|
||||
|
||||
The `monitoring` compose stack on the Pi must mount the textfile dir and pass
|
||||
the flag to node-exporter:
|
||||
|
||||
```yaml
|
||||
node-exporter:
|
||||
volumes:
|
||||
- /var/lib/node-exporter/textfiles:/textfiles:ro
|
||||
command:
|
||||
- '--collector.textfile.directory=/textfiles'
|
||||
# ...existing flags
|
||||
```
|
||||
Consume it as a flake input and let the infrastructure repository define the
|
||||
service.
|
||||
|
||||
Reference in New Issue
Block a user