The Watchdog Pattern - From ESP32 to Python

The sensor node had been running for eleven days. The status LED blinked on schedule. ping returned in under a millisecond. But the last reading in the database was from Tuesday. The MQTT client had stopped publishing, and the main loop was stuck waiting on a mutex that would never be released. The device had wedged.

The fix: pull power, plug it back in. If you have spent time with embedded hardware on a bench or in the field, you have done this.

Name the problem before reaching for the solution. The device did not crash. It entered a state where it could not make progress, and nothing outside the firmware had authority to change that. Silent failure. No recovery path.

This is not an ESP32 problem. A Python worker that blocks on a dead database connection and never times out shows the same shape: the process exists, the supervisor thinks everything is fine, and the work stops. The MCU case is useful because the failure is physical, you can see the LED lie to you.

Something external must detect that failure and force recovery. On a microcontroller, that mechanism has a name you already know. What is the equivalent when your code runs on a server instead of a chip?

Inside the ESP32 Watchdog

The ESP32 watchdog is a hardware timer with a simple contract. You configure a timeout of say, five seconds. The timer counts down. Your firmware must reset it before the countdown reaches zero. If it does not, the hardware assumes the software has failed and resets the chip. No debugger required. No operator with a screwdriver.

The ESP32 has two watchdogs, and they watch different things. The Task Watchdog Timer (TWDT) monitors FreeRTOS tasks. If a task stops running blocked on a mutex, stuck in an infinite loop, or starved by a higher-priority task, the TWDT fires. The Interrupt Watchdog Timer (IWDT) monitors interrupt service routines. If an ISR runs too long or never returns, the IWDT fires. In MicroPython, machine.WDT wraps the task-level watchdog, you configure a timeout and call feed() to reset the timer. ISR hangs are still caught by the IWDT underneath.

Calling wdt.feed() is not a fix. It is proof that your code reached a known-good point and is still making progress. On a wedged device, the main loop never gets there, it spins on a mutex. The watchdog never gets fed. The countdown reaches zero. The chip resets.

That is the contract: something outside the running code must detect failure and force recovery. The watchdog is hardware enforcement of a rule your firmware cannot enforce on itself. A wedged task cannot un-wedge itself.

This contract is not unique to firmware.

Hope Is Not a Strategy

Hope is assuming your code keeps making progress because it has not crashed yet. The firmware is still running. The process still exists. The status light still blinks. Nothing has reported a failure, so you assume everything is fine.

That assumption fails the same way on a chip and on a server. Firmware can block on I/O, lose a task, or spin in a loop while looking healthy from the outside. A Python service can hang on a dead dependency while the supervisor still reports up. In both cases the system appears alive, and nothing external has authority to disagree. You are back to the power-cycle strategy, waiting for a human to notice.

Hope is not supervision. Supervision assumes failure will happen and plans for something outside the code to detect it and respond. Hope assumes failure will not happen, or that you will notice in time to fix it yourself.

Firmware cannot un-wedge itself. A hung worker cannot restart itself. The next section shows what external supervision looks like in code.

Build It: ESP32 and Python

The pattern is the same in both cases: monitor progress, enforce a timeout, recover when the timeout fires. Here are minimal pieces: one on the chip, the rest on the server.

On the ESP32:

from machine import WDT

wdt = WDT(timeout=5000)

while True:
    read_sensor()
    publish_reading()
    wdt.feed()

    # Uncomment to simulate a hang — the chip resets in ~5 seconds:
    # while True: pass

Start the watchdog. Do work. Feed it at the end of each loop iteration. If read_sensor() blocks forever or the hang line runs, wdt.feed() never runs. The hardware timer expires and the chip resets.

On a server:

import subprocess

TIMEOUT = 30

while True:
    proc = subprocess.Popen(["python", "worker.py"])
    try:
        proc.wait(timeout=TIMEOUT)
    except subprocess.TimeoutExpired:
        proc.kill()
        proc.wait()
        continue  # hung — kill and restart
    if proc.returncode != 0:
        continue  # bad exit — restart
    break  # clean exit — stop supervising

The outer loop is the supervisor. It starts the worker, waits up to 30 seconds, and kills it if the worker hangs. A worker that exits with a non-zero return code gets restarted too; a clean exit ends supervision.

The loop above shows the shape; in production you would probably use systemd:

In production (systemd):

[Unit]
Description=Sensor worker

[Service]
ExecStart=/usr/bin/python /opt/app/worker.py
Restart=on-failure
WatchdogSec=30
Type=notify

Restart=on-failure handles exits. WatchdogSec= handles hangs, but only if the worker sends WATCHDOG=1 to systemd via sd_notify on a schedule. No heartbeat, no hang detection.

Install systemd-python via pip, or use your distro's python3-systemd package:

import systemd.daemon

while True:
    do_work()
    systemd.daemon.notify("WATCHDOG=1")

For HTTP services (FastAPI):

@app.get("/health")
async def health():
    await db.execute("SELECT 1")  # prove the dependency is alive, not just the process
    return {"status": "ok"}

For Celery workers:

app.conf.task_time_limit = 30  # seconds — kill hung tasks

Inside the worker (asyncio):

import asyncio

async def main():
    while True:
        reading = await asyncio.wait_for(sensor.read(), timeout=5.0)
        await publish(reading)

asyncio.run(main())

asyncio.wait_for is an in-process timeout, it refuses to wait forever on a blocked call. It does not restart the process; it raises TimeoutError so your code can log, retry, or exit and let the supervisor restart.

External supervision and internal timeouts stack: the worker handles blocked calls, the supervisor handles a wedged worker.

This is deliberately minimal. No logging, exponential backoff, or alerting. Those matter, but they are not the contract. The contract is that something outside the worker enforces recovery.

Working recovery is not the same as good recovery.

When Recovery Becomes the Problem

A watchdog that resets on schedule is not a healthy device. It is a device that fails on schedule.

The wedged mutex from the opening would have triggered a reset and then wedged again on the next boot, assuming the bug survived the restart. The ESP32 comes back, runs the same initialization, hits the same mutex, and stops feeding the watchdog again. Reset. Boot. Wedge. Reset. You have traded a silent failure for a noisy one, and the field log shows a device that reboots every five seconds without ever publishing a reading.

The same loop appears in Python. A systemd unit with Restart=always and a worker that crashes on startup will enter a crash loop, systemctl reports active (restarting), but no work completes. Celery eventually replaces the worker process, but if the database is still down, the new worker blocks in the same place. The supervisor did its job. The root cause did not move.

Supervision without observability makes this worse. If you do not log why recovery fired, the reset cause, a backtrace, or a metric, then you only know the device came back. You do not know why it reset.

A watchdog that fires too often becomes noise, operators tune it out. One that fires too late means damage is already done: a motor left running, a valve stuck open, a buffer overwritten before the timeout expired.

Recovery is necessary. It is not sufficient. You still need to fix the mutex, fix the connection string, add the timeout that prevents the hang in the first place. The watchdog buys you uptime. It does not buy you correctness.

One Pattern, Two Domains

Every supervision design answers three questions. What are you monitoring? What triggers recovery? What happens after the reset?

Apply it to the wedged sensor node. A five-second WDT would have caught the hang, no operator, no power cycle. But recovery alone would have produced a reboot loop until someone fixed the mutex bug and added logging to read the reset cause on boot. The checklist covers detection and recovery. Your code still has to stop causing it.

If you build embedded systems, you already live this pattern. You feed watchdogs, read reset reasons, tune timeouts around your slowest legitimate operation. Python production is the same lesson in different clothes: external supervision, enforced timeouts, observable recovery. The platform changes. The contract does not.

Follow me on Twitter: https://twitter.com/DevAsService

Follow me on Instagram: https://www.instagram.com/devasservice/

Follow me on TikTok: https://www.tiktok.com/@devasservice

Follow me on YouTube: https://www.youtube.com/@DevAsService