project
Field gateway that stops needing a site visit
A Raspberry Pi gateway for remote pump stations that recovers itself from the failures that used to mean a four-hour drive.
Twenty-odd pump stations, each with a gateway pushing telemetry over LTE. The hardware was fine. The problem was that any one of a dozen small faults — a wedged modem, a full disk, a broker that stopped acknowledging — looked identical from the office: silence. Someone had to drive out and power-cycle it.
What it does now
Every failure mode that used to need a human now has a defined recovery path, and the gateway walks down that path on its own before anyone is told anything.
The part that mattered
A ping is not liveness. The gateway can answer ICMP while its publish queue has been stalled for six hours. The health check had to be defined in terms of the thing the system is actually for — a message accepted by the broker in the last N minutes — and everything else followed from that.
Uptime across the fleet went from roughly 91% to a shade over 99.4% in the first quarter, and site visits dropped to the ones involving actual hardware damage.