Site: the developer docs (phase 4), with the Debug Builds and the Debug Console first
CI / build (pull_request) Successful in 8m38s
Site / build (pull_request) Successful in 9s

/dev/ has Debug Builds and the Debug Console (builds and the token, the
console and its protocol, files and screenshots, driving the UI, crashes and
Safe Mode, the command reference), Build, test and release (including how an
update works), the architecture decisions and the milestone plans.

Generated from the repository by site/tools/gen_dev_docs.py: the ADRs, the
milestones, the README's sections, and the command reference, read from the
firmware's own `help` text. The pages are committed (Zola cannot read outside
its folder); the Site workflow checks they are current, and now also runs
when src/main.cpp changes. M0, M1 and CONTEXT.md are not published.
README: the gnss commands that the table lacked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT
This commit is contained in:
2026-10-06 21:25:33 +02:00
co-authored by Claude Sonnet 5.5
parent 8f95bee744
commit 3b4100dc6a
43 changed files with 2284 additions and 6 deletions
@@ -0,0 +1,21 @@
+++
title = "Safe Mode, crash reports and a watched main loop, in every build"
description = "Rollback protects against new firmware that fails Probation. It does nothing for firmware that was confirmed and crashes later: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show.…"
weight = 5
[extra]
docs = true
source = "docs/adr/0005-safe-mode-crash-reports-watchdog.md"
tag = "ADR 0005"
+++
Rollback protects against new firmware that fails Probation. It does nothing for firmware that was confirmed and crashes later: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. Without a cable, such a device would restart forever. Three measures, in release and Debug Builds alike, keep it reachable:
- **Safe Mode.** The firmware counts starts that follow a crash (panic or watchdog) in NVS, first thing at boot. After 3 in a row, it starts only the clock, Wi-Fi, the Update Service and, in a Debug Build, the Debug Console: no Apps, no IRC, no SD card, and a screen that says so with the address to push an update to. Any normal restart (a `reboot`, an update), or a minute of uptime, resets the count.
- **Crash reports.** The same boot record keeps which version was running, so after a crash the firmware knows which one crashed, even when a Rollback has switched slots since. ESP-IDF already writes a core dump to its flash partition on a panic; after the restart the firmware prints its summary (task, PC, reason, backtrace) and raises a Notification. The `crash` command shows it again later. In a Debug Build, `scripts/rdbg.py crash` decodes the backtrace and `scripts/rdbg.py coredump` fetches the whole dump for `esp-coredump`, against the ELF of that exact build (`.pio/elves/`, named by version and ELF digest).
- **The main loop is watched.** Arduino-ESP32 subscribes only core 0's idle task to the task watchdog, and the main loop runs on core 1: a stuck loop used to hang the device for good, with the screen frozen and the Debug Console unable to run commands. `enableLoopWDT()` makes a loop stuck for 5 s a panic, with a core dump, counted towards Safe Mode. And an installed update no longer depends on the main loop: the Update Service restarts into it by itself after 90 s.
## Consequences
- Nothing in the main loop may block for 5 s. Network and card work already run on their own tasks.
- Safe Mode can't help when Wi-Fi or the Update Service itself is what crashes; that still needs USB.
- Three crashes within a minute of each restart are needed to reach Safe Mode, so a crash loop costs about half a minute before the device becomes reachable.