Site: the developer docs (phase 4), with the Debug Builds and the Debug Console first

/dev/ has Debug Builds and the Debug Console (builds and the token, the console and its protocol, files and screenshots, driving the UI, crashes and Safe Mode, the command reference), Build, test and release (including how an update works), the architecture decisions and the milestone plans. Generated from the repository by site/tools/gen_dev_docs.py: the ADRs, the milestones, the README's sections, and the command reference, read from the firmware's own `help` text. The pages are committed (Zola cannot read outside its folder); the Site workflow checks they are current, and now also runs when src/main.cpp changes. M0, M1 and CONTEXT.md are not published. README: the gnss commands that the table lacked. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT
2026-10-06 21:25:33 +02:00
parent 8f95bee744
commit 3b4100dc6a
43 changed files with 2284 additions and 6 deletions
@@ -0,0 +1,23 @@
+++
+title = "Own firmware that speaks Meshtastic, not a Meshtastic fork"
+description = "We build our own firmware from existing libraries (PlatformIO + Arduino-ESP32, M5Cardputer/M5Unified, RadioLib, TinyGPSPlus, nanopb with Meshtastic's published protobufs). We implement the Meshtastic protocol ourselves as one…"
+weight = 1
+
+[extra]
+docs = true
+source = "docs/adr/0001-own-firmware-speaking-meshtastic.md"
+tag = "ADR 0001"
+++
+We build our own firmware from existing libraries (PlatformIO + Arduino-ESP32, M5Cardputer/M5Unified, RadioLib, TinyGPSPlus, nanopb with Meshtastic's published protobufs). We implement the Meshtastic protocol ourselves as one pluggable Mesh Protocol, rather than forking the Meshtastic firmware, which already supports this exact hardware.
+
+A fork would give full compatibility on day one, but its architecture is built around being a single-purpose Meshtastic node. That conflicts with our goals: a multi-app OS with a fully custom UX, and room for other mesh protocols (e.g. MeshCore) later.
+
+## Consequences
+
+- We accept partial Meshtastic compatibility at first: text on channels, Direct Messages, node list, position and relaying.
+- The phone-app (BLE) API and PKI-encrypted Direct Messages are deferred, and we must re-implement protocol details ourselves.
+- Multi-boot with stock Meshtastic via a launcher was rejected: it gives none of our own UX.
+
+## Note (2026-10-04, M2)
+
+NMEA is parsed by our own small, host-tested parser instead of TinyGPSPlus: the GNSS App's Sky view needs the satellite list (GSV) across several constellations, which TinyGPSPlus doesn't track. See docs/milestones/M2.md, Q66.
@@ -0,0 +1,18 @@
+++
+title = "Own small widget kit on M5GFX, not LVGL"
+description = "The UI is drawn with M5GFX into an off-screen buffer, using a small widget kit we own: list, text view, line editor, dialog, Status Bar and Toast. We chose this over LVGL."
+weight = 2
+
+[extra]
+docs = true
+source = "docs/adr/0002-own-widget-kit-on-m5gfx.md"
+tag = "ADR 0002"
+++
+The UI is drawn with M5GFX into an off-screen buffer, using a small widget kit we own: list, text view, line editor, dialog, Status Bar and Toast. We chose this over LVGL.
+
+LVGL would give us ready-made widgets, but it costs roughly 40–60 KB of RAM on a device with no PSRAM. It would also need to coexist with the Mesh Service, the Wi-Fi stack and TLS, and it brings a large learning surface. Most of our Apps are lists and text on a 240×135 screen, and full control of the UX is a primary goal.
+
+## Consequences
+
+- We write and maintain our own widgets.
+- Switching to LVGL later would mean rewriting every App's view layer.
@@ -0,0 +1,20 @@
+++
+title = "Signed Update Files checked by the firmware, not ESP32 Secure Boot"
+description = "Firmware Updates are accepted only when their Update File carries a valid ECDSA P-256 signature over the image's SHA-256. The firmware itself checks it, against a public key compiled into it, before switching the boot partition.…"
+weight = 3
+
+[extra]
+docs = true
+source = "docs/adr/0003-own-signature-check-not-secure-boot.md"
+tag = "ADR 0003"
+++
+Firmware Updates are accepted only when their Update File carries a valid ECDSA P-256 signature over the image's SHA-256. The firmware itself checks it, against a public key compiled into it, before switching the boot partition. The private key lives outside the repository, in `~/.config/roro9stack/ota-key.pem`.
+
+We chose this over the ESP32's hardware Secure Boot. Secure Boot is enforced by the chip, but it burns eFuses one-way: a mistake bricks the device, and the device can never run unsigned firmware again, which makes recovery over USB harder. On a single development device, a software check that refuses unsigned pushes is enough, and it stays reversible: a new firmware can carry a new public key.
+
+## Consequences
+
+- Someone with physical USB access can still flash anything. Only Wi-Fi and SD card updates are guarded.
+- **Losing the private key** means the next update has to go over USB, carrying a new public key.
+- P-256 rather than Ed25519, because the firmware's TLS library (mbedTLS) already verifies it, so it costs no extra code.
+- **Rollback: the bootloader first, the firmware as a second line.** Arduino-ESP32 marks a new image valid before `setup()` unless the sketch overrides `verifyRollbackLater()`, which once made every update look good and hid the bootloader's rollback (it had looked like the prebuilt bootloader ignored it). With the override, an image stays pending until Probation confirms it, and the bootloader reverts one that restarts unconfirmed, however early it crashes. The firmware also counts its own boots on Probation, very first thing in `setup()`, and reverts itself on the second unconfirmed start.
@@ -0,0 +1,31 @@
+++
+title = "A Debug Console over Wi-Fi, in Debug Builds only"
+description = "The goal of Firmware Updates is to manage the device without a cable, and that includes finding out what went wrong. So a Debug Build (cardputer-adv-debug, -DRORO_DEBUG, version suffix +debug) adds a Debug Console on TCP 2323…"
+weight = 4
+
+[extra]
+docs = true
+source = "docs/adr/0004-debug-console-in-debug-builds.md"
+tag = "ADR 0004"
+++
+The goal of Firmware Updates is to manage the device without a cable, and that includes finding out what went wrong. So a **Debug Build** (`cardputer-adv-debug`, `-DRORO_DEBUG`, version suffix `+debug`) adds a **Debug Console** on TCP 2323: the serial console, over Wi-Fi. A client sends a token as its first line, then gets the last 4 KB of console output (boot messages included), every new line live, and runs the same commands as the serial port, plus a few that only make sense remotely. ESP-IDF's own log lines are teed into it.
+
+It's compiled out of release builds entirely, rather than switched off by a setting. A console that runs commands is a remote control: in a release build, nothing listens.
+
+## How it fits
+
+- **Console, not Serial.** All human-readable output goes through `console`, which writes to the USB port and, in a Debug Build, to a ring buffer the Debug Console drains. Writes never wait for USB: a host that's attached but not reading used to stall the main loop for up to 2 s per line.
+- **Commands run on the main loop.** The socket lives on the Debug Console's own task, which only queues command lines. The main loop runs them, as it does serial commands, so they touch Apps and Services from the one task allowed to.
+- **The token** is 128 random bits in `~/.config/roro9stack/debug-token`, made by the first build and passed into the container. It's never committed; a Debug Build refuses to compile without one. Like the OTA key, it guards against the network, not against someone holding the device.
+- **One client at a time**, to keep memory flat (4 KB for the ring since M2, 6 KB of task stack).
+- **Binary commands are answered on the console's own task**, not queued: `get`/`put` (SD card files, run as one Storage Service job each so card access stays on the storage task, with TCP doing the flow control), `screenshot` (the 32 KB RGB332 frame the UI composes into, read as it stands, so it may tear), `coredump get` and `reset`. These keep working when the main loop is stuck. A failed `put` closes the connection, so the rest of the file is never read as commands.
+
+## Keep a Debug Build in the fallback slot
+
+Rollback returns to the previous firmware, whatever it is. As long as development goes through Debug Builds, the firmware a crash falls back to has the Debug Console, so a bad update never costs remote access. A release build pushed over a Debug Build leaves the Debug Build in the other slot until the next update overwrites it.
+
+## Consequences
+
+- Anyone on the same network with the token can read the console, inject keys and reboot the device. The console never prints stored secrets (Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
+- The TCP stream is plain text: fine on a home network, not across the internet.
+- `+debug` versions compare equal to their release counterparts, so moving between the two is never refused as a downgrade.
@@ -0,0 +1,21 @@
+++
+title = "Safe Mode, crash reports and a watched main loop, in every build"
+description = "Rollback protects against new firmware that fails Probation. It does nothing for firmware that was confirmed and crashes later: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show.…"
+weight = 5
+
+[extra]
+docs = true
+source = "docs/adr/0005-safe-mode-crash-reports-watchdog.md"
+tag = "ADR 0005"
+++
+Rollback protects against new firmware that fails Probation. It does nothing for firmware that was confirmed and crashes later: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. Without a cable, such a device would restart forever. Three measures, in release and Debug Builds alike, keep it reachable:
+
+- **Safe Mode.** The firmware counts starts that follow a crash (panic or watchdog) in NVS, first thing at boot. After 3 in a row, it starts only the clock, Wi-Fi, the Update Service and, in a Debug Build, the Debug Console: no Apps, no IRC, no SD card, and a screen that says so with the address to push an update to. Any normal restart (a `reboot`, an update), or a minute of uptime, resets the count.
+- **Crash reports.** The same boot record keeps which version was running, so after a crash the firmware knows which one crashed, even when a Rollback has switched slots since. ESP-IDF already writes a core dump to its flash partition on a panic; after the restart the firmware prints its summary (task, PC, reason, backtrace) and raises a Notification. The `crash` command shows it again later. In a Debug Build, `scripts/rdbg.py crash` decodes the backtrace and `scripts/rdbg.py coredump` fetches the whole dump for `esp-coredump`, against the ELF of that exact build (`.pio/elves/`, named by version and ELF digest).
+- **The main loop is watched.** Arduino-ESP32 subscribes only core 0's idle task to the task watchdog, and the main loop runs on core 1: a stuck loop used to hang the device for good, with the screen frozen and the Debug Console unable to run commands. `enableLoopWDT()` makes a loop stuck for 5 s a panic, with a core dump, counted towards Safe Mode. And an installed update no longer depends on the main loop: the Update Service restarts into it by itself after 90 s.
+
+## Consequences
+
+- Nothing in the main loop may block for 5 s. Network and card work already run on their own tasks.
+- Safe Mode can't help when Wi-Fi or the Update Service itself is what crashes; that still needs USB.
+- Three crashes within a minute of each restart are needed to reach Safe Mode, so a crash loop costs about half a minute before the device becomes reachable.
@@ -0,0 +1,27 @@
+++
+title = "The framework is rebuilt with our own SDK settings, for smaller TLS buffers"
+description = "Arduino-ESP32 ships its ESP-IDF libraries prebuilt, with one sdkconfig for every ESP32-S3 board. Its TLS settings give every connection a 16 KB receive buffer and a 16 KB send buffer for its whole life. On a device with no PSRAM…"
+weight = 6
+
+[extra]
+docs = true
+source = "docs/adr/0006-framework-rebuilt-for-smaller-tls-buffers.md"
+tag = "ADR 0006"
+++
+Arduino-ESP32 ships its ESP-IDF libraries prebuilt, with one `sdkconfig` for every ESP32-S3 board. Its TLS settings give every connection a 16 KB receive buffer and a 16 KB send buffer for its whole life. On a device with no PSRAM and about 340 KB of RAM, an IRC connection over TLS left a 12.6 KB low in M2, against a 40 KB floor.
+
+Those settings are compiled into the libraries, so changing them means rebuilding them. pioarduino supports this as a "hybrid compile": `custom_sdkconfig` in `platformio.ini` lists the settings, and the build regenerates the framework's libraries from ESP-IDF (the same 5.5.5 the prebuilt ones come from) before building the app. We set:
+
+- `MBEDTLS_ASYMMETRIC_CONTENT_LEN`, with 16 KB to receive (servers send full TLS records) and **4 KB to send** (IRC lines are short): 12 KB less per connection.
+- `MBEDTLS_DYNAMIC_BUFFER`, `DYNAMIC_FREE_CONFIG_DATA`, `DYNAMIC_FREE_CA_CERT`: buffers allocated when needed, and handshake-only data (the CA chain) freed once connected.
+
+The rebuild also follows the board definition instead of the generic one: PSRAM support is off (the Cardputer ADV has none) and the flash size is 8 MB.
+
+## Consequences
+
+- With IRC connected over TLS, a Debug Build has 78 KB free and a 59 KB low (it was 31 KB and 12.6 KB), and 46 KB at the lowest under the heaviest combined load measured (IRC, two refused installs, a 1.6 MB put and get).
+- The first build after a fresh checkout, or after changing `custom_sdkconfig`, takes about 4 minutes instead of 45 s: it downloads ESP-IDF into the PlatformIO volume and compiles it. Later builds reuse it.
+- Everything the firmware depends on was checked in the regenerated `sdkconfig`: app rollback, core dumps to flash (ELF), the 5 s task watchdog, FreeRTOS run-time stats, the certificate bundle.
+- The project now owns its partition table (`default_8MB.csv`, identical to the framework's), which the hybrid build requires. Changing it would break updates over the air: the app slots must stay where they are.
+- Generated files (`sdkconfig.*`, `managed_components/`, `.dummy/`) are ignored by git.
+- A TLS server that sends records over 16 KB would still fail, as before; one that needs us to send records over 4 KB would now fail. Neither happens with IRC.
@@ -0,0 +1,28 @@
+++
+title = "Our own copy of the SD driver, for one missing byte"
+description = "Arduino-ESP32's SD library talks to the card over SPI through sd_diskio.cpp. That driver gives up on a write without saying why, and about once in 1,500 multi-block writes it gave up on one that had worked (issue #21). A 1.7 MB…"
+weight = 7
+
+[extra]
+docs = true
+source = "docs/adr/0007-own-copy-of-the-sd-driver.md"
+tag = "ADR 0007"
+++
+Arduino-ESP32's `SD` library talks to the card over SPI through `sd_diskio.cpp`. That driver gives up on a write without saying why, and about once in 1,500 multi-block writes it gave up on one that had worked (issue #21). A 1.7 MB upload failed about three times in ten; before M3, the retry on top of it then filled the gap with zeros.
+
+The cause, measured with a driver that records where it stops: after the "Stop Tran" token that ends a multi-block write, a card takes about a byte of clock to signal busy. The driver deselects, selects again, and reads one byte to see whether the card is ready. Read too early, that byte is 0xFF, "ready"; the status check (CMD13) then goes out while the card is still programming, and its answer (0xFF, 0x1F) is taken for an error. Every failure seen was this one: all blocks accepted, then a status that isn't one. ChaN's reference driver, which FatFs ships as its example, sends a dummy byte after selecting the card for this reason. Arduino's doesn't.
+
+PlatformIO links the framework's library objects directly, so one file can't be replaced from `src`. A project library with the same name takes its place: **`lib/SD` is Arduino-ESP32 3.3.12's SD library (Apache-2.0), with `sd_diskio.cpp` changed** and the other files as they came. The changes are marked `roro:`:
+
+- A dummy byte after selecting the card, before the ready test, and one after Stop Tran.
+- Each place a write gives up records the step and the card's answer (`sd_fault.h`): `info` shows the count, and the Debug Console's `put` prints the detail.
+
+Halving the SPI clock to 10 MHz didn't change the failure rate, so the card stays at 20 MHz.
+
+## Consequences
+
+- 30 uploads of 1.7 MB in a row, each read back and compared by SHA-256, ten of them with the LoRa radio listening on the same bus: no write fault. Before: 3 failures in 10.
+- Every writer gains: Logs, Tracks, Gemini pages, Saved Pages, Captures and Update Files installed from the card all go through this driver, and none of them checked.
+- **The copy has to follow the framework.** When the platform is updated, compare `lib/SD` with the new `libraries/SD` and carry the `roro:` changes over. If upstream fixes the ready test, drop the copy. Reported as [arduino-esp32#12970](https://github.com/espressif/arduino-esp32/issues/12970); issue #39 follows it.
+- One more defect was read in the code and left alone, because nothing here exercises it: the driver tests the card's answer to a data block against 0x0A and 0x0C, values it can't take (accepted is 0x05, CRC error 0x0B, write error 0x0D), so a block rejected for a CRC error is never resent. No such rejection was seen in any failure. If `DataToken` faults ever show in `info`, that's the next fix.
+- A fault is now counted and explained instead of silent, so the next cause, if there is one, starts with evidence.
@@ -0,0 +1,25 @@
+++
+title = "CI signs releases with the project's key"
+description = "A tag v is built, signed and published by Gitea Actions with nobody at a keyboard. The signing key of ADR 0003 is therefore held twice: in ~/.config/roro9stack/ota-key.pem on the development machine, as before, and as the…"
+weight = 8
+
+[extra]
+docs = true
+source = "docs/adr/0008-ci-signs-releases.md"
+tag = "ADR 0008"
+++
+A tag `v*` is built, signed and published by Gitea Actions with nobody at a keyboard. The signing key of ADR 0003 is therefore held twice: in `~/.config/roro9stack/ota-key.pem` on the development machine, as before, and as the repository secret `OTA_SIGNING_KEY`, which the release step writes to a file for as long as it runs.
+
+We chose this over signing by hand after CI has built (a command per release, the key in one place), and over a second key for CI that the firmware would also trust. A release that needs a manual step isn't made on the day it's ready, and issue #6, the device installing releases by itself, needs releases that are always there and always signed.
+
+## What it costs
+
+- **Whoever can run a workflow in this repository can sign firmware every device accepts.** That means: anyone who can push to it, the runner's host and whoever administers it, and the Gitea instance with its database, where the secret is stored. Before, it took the development machine.
+- The runner executes jobs **on its own host**, not in a container, as a user who can use Docker. A workflow is not confined.
+- Pull requests from forks must never run with this secret. Gitea doesn't pass secrets to them; the workflow also only runs on pushes and by hand.
+
+## What limits it
+
+- The release step checks the signed file against the public key in the sources it built (`scripts/ota_verify.py`): a wrong or replaced secret stops the release instead of publishing a file no device takes.
+- ADR 0003's way out stays: a firmware release can carry a new public key. If the secret is ever in doubt, make a new pair, ship it in a release signed with the old key, and replace the secret.
+- A device still only installs what it's told to (until #6), keeps a new image on Probation, and rolls back one that doesn't hold.
@@ -0,0 +1,30 @@
+++
+title = "The device trusts the two ISRG roots for what it fetches"
+description = "The firmware talks to the project's Gitea over HTTPS (issue #6, and #4 after it). A TLS client has to decide whose certificates it believes. Three ways were possible:"
+weight = 9
+
+[extra]
+docs = true
+source = "docs/adr/0009-the-device-trusts-the-isrg-roots.md"
+tag = "ADR 0009"
+++
+The firmware talks to the project's Gitea over HTTPS (issue #6, and #4 after it). A TLS client has to decide whose certificates it believes. Three ways were possible:
+
+- **The framework's bundle**, about 130 certificate authorities (about 60 KB of flash). Any of them could vouch for `git.twis.la`.
+- **Pin the server's certificate**, as the Gemini App does for capsules. The server's certificate is replaced every few months, so a pin would ask the question again at every renewal.
+- **Carry the roots the server's chain ends in:** `ISRG Root X1` (RSA 4096) and `ISRG Root X2` (ECDSA P-384), the Let's Encrypt roots, about 2.7 KB of flash (`src/platform/ca_roots.h`).
+
+We chose the third. The chain and the name are checked by mbedTLS during the handshake. It trusts one organisation's two roots, valid until 2035 and 2040, and a renewal changes nothing.
+
+## What it costs
+
+- **If the server moves to another CA, the device can no longer reach it**, and the next firmware, carrying that CA's root, has to come from the PC or the SD card. Both still work; they don't use TLS.
+- The roots are public data checked against the published fingerprints (listed in the file), refreshed by hand if Let's Encrypt ever changes them.
+
+## What it doesn't change
+
+The Update File's own signature (ADR 0003) is what decides what gets installed. A hijacked connection could hide a release, or serve an older signed one, but never make the device install firmware that isn't ours. The TLS check matters more for #4, where a token will travel over it.
+
+## Measured while building it
+
+A TLS connection to this server peaks at about 52 KB of heap, **the same whether the certificate is checked or not**, so skipping the check would have saved nothing. The cost is the connection itself (record buffers and handshake), not the trust decision.
@@ -0,0 +1,13 @@
+++
+title = "Decisions"
+description = "The architecture decision records: why the firmware is built the way it is, and what each choice costs. Generated from docs/adr/ in the repository."
+template = "guide-index.html"
+page_template = "guide-page.html"
+sort_by = "weight"
+weight = 3
+
+[extra]
+eyebrow = "Developer docs"
+++
+
+One page for each architecture decision: the choice, what it was chosen over, and its consequences. They are written when the decision is made and kept; a later decision that changes an earlier one says so. Generated from `docs/adr/` in the repository: edit those files, not these pages.