Linux SBC Watchdog Design for Unattended Gateway and Control Products

A watchdog is easy to describe and surprisingly easy to implement badly. Software periodically proves the system is alive; if that proof stops, a timer expires and recovery begins. On an unattended product, however, the difficult part is deciding what “alive” means. A process can exist while its worker is deadlocked. Linux can be healthy while the cloud is unavailable. Resetting the board for every fault creates a different reliability problem.

For a production Linux SBC, watchdog design should therefore be treated as a recovery hierarchy, not a single checkbox in the BSP. The application, service manager, kernel, hardware timer, power circuit, storage, and boot process all have a role. The goal is not to produce more reboots. It is to recover the smallest failed layer, preserve enough evidence to diagnose the fault, and return the product to a safe operating state.

Start with the failures you actually expect

Before choosing a timeout, list the faults the product must survive. A remote data gateway may lose DNS, Ethernet, LTE registration, an MQTT session, or communication with a Modbus device. A controller may see a blocked application thread, a driver fault, an exhausted file system, or a kernel stall. Power can dip without staying off long enough for a clean shutdown.

These events should not all receive the same response.

This table forces a useful distinction: connectivity is not health. If an installation loses its upstream network for six hours, a well-designed gateway should continue local acquisition, buffer data within limits, and reconnect later. Rebooting every minute will add flash writes, delay local control, and make the event harder to investigate.

Use more than one recovery layer

The first layer is normally inside the application. A worker can time out a transaction, discard a damaged message, reopen a port, or recreate a client connection. This is fast and does not disturb unrelated functions.

The next layer is the service manager. On a systemd-based image, a crashed process can be restarted through Restart=, while WatchdogSec= can detect a process that remains present but stops sending watchdog notifications. The application must send a real health notification from a code path that proves its critical work is progressing. A timer thread that sends WATCHDOG=1 regardless of application state defeats the design.

The hardware watchdog is the final on-board layer. Linux normally exposes it through /dev/watchdog or /dev/watchdog0. Once enabled, the timer must be serviced before its timeout; otherwise the hardware resets the processor or board. The official Linux watchdog driver API documentation explains the keepalive, timeout, boot-status, pretimeout, and nowayout behavior available through the kernel interface. Support still varies by SoC, driver, and board design, so the exact BSP must be tested rather than assumed.

Some products also need an external supervisor. A processor reset may leave a modem, USB peripheral, Ethernet switch, or secondary MCU in a bad state. If a warm reset cannot clear the failure, provide controlled reset of the affected rail or complete assembly.

Decide who is allowed to feed the hardware watchdog

One process should own the hardware watchdog. Multiple independent feeders make fault coverage unclear: one healthy daemon may continue feeding while the critical control application is dead. On many systemd systems, PID 1 can manage the hardware watchdog through RuntimeWatchdogSec=. Individual services can then use systemd’s software watchdog and notification interface.

A service unit may begin with settings like these:

[Unit]
StartLimitIntervalSec=300s
StartLimitBurst=4

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/bin/gateway-service
Restart=on-failure
RestartSec=5s
WatchdogSec=20s

This is a starting pattern, not a universal configuration. The service must call sd_notify() with READY=1 after genuine initialization and send WATCHDOG=1 only while its critical loop is healthy. If restart limits are reached, the product should raise an alarm, enter a defined degraded mode, or escalate to broader recovery. Leaving the gateway half-alive is rarely acceptable.

During bring-up, use tools such as wdctl to confirm the device, driver, returned timeout, and available features. Hardware may round the requested timeout, and pretimeout or boot-status support varies. Record the observed behavior for the exact PCB revision and BSP release.

Set timeouts from measurements, not habit

A ten-second timeout sounds safer until a valid flash erase, database checkpoint, CPU load, slow peripheral startup, or update step delays feeding for twelve seconds. False trips can place units in a reboot loop.

Measure worst-case scheduling and initialization under realistic load, then define separate budgets for application health, service restart, and hardware reset. For example, a serial transaction may fail after two seconds, systemd may act after twenty seconds without notification, and hardware may use a longer system-level timeout. Product safety and availability requirements decide the actual values.

Do not simply disable protection during an update. Define whether the updater owns the feed, whether the timeout is extended, and how the bootloader returns to a known-good image. Watchdog and A/B or recovery-image design belong in the same review.

Preserve the reason for every reset

A unit that recovers but leaves no trace is difficult to improve. At minimum, store a boot counter, reset cause where the SoC or PMIC exposes one, software version, hardware revision, previous service state, and a small bounded set of logs. Upload the record after connectivity returns.

Do not rewrite flash on every health check. Keep routine health data in memory and commit bounded diagnostic records only when necessary. Log rotation and power-loss handling should follow the plan described in storage media planning for embedded Linux.

Reset-cause registers may be cleared during early boot, so the BSP should capture them promptly. Distinguish watchdog reset from brownout, manual reboot, update reboot, and factory test; otherwise field statistics will overstate software failures.

Validate by injecting failures

Bench uptime is not a watchdog test. Stop the feed and create the faults the hierarchy is meant to contain, using the release image, final board revision, and intended power supply:

1. Kill the main application and confirm a service restart without a board reboot.
2. Freeze the application so the PID remains but health notification stops.
3. Stop hardware watchdog feeding and measure the actual reset interval.
4. Disconnect WAN, Ethernet, RS485, and attached USB devices separately.
5. Fill the writable partition and verify that essential control continues or fails safely.
6. Repeat sudden power cuts while logs and local data are being written.
7. Force high CPU, memory pressure, and elevated temperature within product limits.
8. Interrupt an update at each critical stage and verify recovery or rollback.
9. After every reset, confirm interfaces, time, configuration, buffered data, and reset records.

Long-duration testing matters because a restart every four days will not appear in a one-hour test. Record the image, fault method, recovery time, reset reason, and pass criteria. Production should run the same core check with a shorter controlled timeout. The notes on pogo-pin fixtures and automated board testing provide related fixture context.

Review the board, BSP, and field role together

Watchdog capability is partly a software requirement and partly a hardware one. When reviewing an IoT gateway or industrial SBC, confirm the SoC watchdog, kernel driver, bootloader behavior, PMIC reset path, peripheral reset lines, reset-cause reporting, and BSP configuration. If a standard board cannot reset a problematic modem or meet the required recovery path, a custom SBC may need a supervisor, load switch, or additional reset control.

Changing the PMIC, storage, wireless module, or modem can alter recovery even when the CPU stays the same. Include watchdog and power-cycle tests in component-change validation.

The best unattended product is not one that never reports trouble. It is one that contains ordinary faults, escalates only when needed, comes back in a known state, and leaves enough evidence for an engineer to understand what happened. That standard turns a watchdog from a datasheet feature into a dependable part of the product architecture.