twisla/roro9stack
Public Access
1
0
Fork 0
Code Issues 48 Pull Requests Actions Packages Releases 16 Activity

Site: the developer docs (phase 4), with the Debug Builds and the Debug Console first
CI / build (pull_request) Successful in 8m38s
Details
Site / build (pull_request) Successful in 9s
Details

Browse Source 
/dev/ has Debug Builds and the Debug Console (builds and the token, the
console and its protocol, files and screenshots, driving the UI, crashes and
Safe Mode, the command reference), Build, test and release (including how an
update works), the architecture decisions and the milestone plans.

Generated from the repository by site/tools/gen_dev_docs.py: the ADRs, the
milestones, the README's sections, and the command reference, read from the
firmware's own `help` text. The pages are committed (Zola cannot read outside
its folder); the Site workflow checks they are current, and now also runs
when src/main.cpp changes. M0, M1 and CONTEXT.md are not published.
README: the gnss commands that the table lacked.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT
This commit is contained in:
twisla
2026-10-06 21:25:33 +02:00
co-authored by Claude Sonnet 5.5
parent 8f95bee744
commit 3b4100dc6a
43 changed files with 2284 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
site/content/dev/debug/_index.md
+22
View File
@@ -0,0 +1,22 @@
+++
title = "Debug Builds and the Debug Console"
description = "A firmware you can drive from your desk: its console over Wi-Fi, its screen as a PNG, its SD card, its crash dumps, and a way back when an update goes wrong."
template = "guide-index.html"
page_template = "guide-page.html"
sort_by = "weight"
weight = 1
[extra]
eyebrow = "Developer docs"
+++
A **Debug Build** is the same firmware plus a **Debug Console**: the serial console, over Wi-Fi, behind a token. It is the most useful thing in the project. With it you can:
- **see everything the device prints**, boot messages included, without a cable;
- **run every serial command** from your desk;
- **press keys** and **take screenshots**, so a UI change can be tested and looked at remotely;
- **copy files** to and from the SD card;
- **read crash reports and fetch core dumps**, decoded against the exact build that crashed;
- **update the firmware** over the same Wi-Fi, and know that a bad update rolls back to a build that still has the console.
Start with [Debug Builds](/dev/debug/debug-builds/) to put one on a device, then [the Debug Console](/dev/debug/console/). The rest are what you can do with it.
site/content/dev/debug/commands.md
+113
View File
@@ -0,0 +1,113 @@
+++
title = "Command reference"
description = "Every command the firmware understands, over USB serial or the Debug Console: what `help` prints, then what each one does."
weight = 30
[extra]
docs = true
source = "src/main.cpp and README.md"
tag = "Reference"
+++
## What `help` prints
The firmware's own list, read from `src/main.cpp`. These work in every build, over USB serial:
```
info firmware, uptime, memory, Wi-Fi, app slots
tasks FreeRTOS tasks over the next second: state, priority, free stack, CPU share
net bytes each network service has read and written since boot
reboot restart
boot other restart into the other app slot (manual Rollback)
log level <0-5> ESP-IDF log level (0 none ... 5 verbose)
ls [folder] | du <path> | mkdir <path> | rm <path> | cp [-f] <from> <to> | mv [-f] <from> <to> | cancel the SD card, with the Storage App's rules
lora probe | lora status | lora rx on|off | lora preset <name> the LoRa radio, receive only
lora capture start|stop a LoRa Capture to /captures/lora (pcap, LoRaTap)
lora sweep on [from MHz] [to MHz] [step kHz] | off | dump RSSI across a band (863 870 100)
lora custom <MHz> <BW kHz> <SF> <CR 5-8> <sync hex> [preamble] e.g. 868.1 125 7 5 34 8 (LoRaWAN)
gnss quiet on|off pause the GNSS receiver while the LoRa radio listens (it costs the radio 8 dB)
gnss status | gnss restart | gnss track start|stop | gnss nmea on|off | gnss send <sentence without $ and checksum>
crash the last crash: firmware, reason, task, backtrace
coredump erase forget the core dump in flash
key <name|char> press a key: up down left right select back home del tab space, or one character
wifi status | wifi add <ssid><TAB><password>
wifi ip <ssid> dhcp | wifi ip <ssid> <address>/<prefix> [gateway] a Saved Network's IP setting
wifi dns <a> [b] | wifi dns always on|off | wifi ntp <a> [b] DNS and NTP servers
gemini get <url> fetch a Gemini page and report header, size, certificate, heap
irc start | irc stop | irc dump | irc say <buffer> <text>
install <path.ota> Update from SD
update check | list | status | install <tag> the project's releases on Gitea
sd card | sd list | cat <path> | log <text> | burst | sound on|off | short | normal
```
A **Debug Build** adds these (the last ones, marked *Debug Console only*, exist only over Wi-Fi, where [`rdbg.py`](/dev/debug/files-and-screens/) speaks them):
```
crash abort|wdt crash on purpose (to test crash reports and Safe Mode)
wifi ip ... try <seconds> | wifi ip keep a trial IP setting: back to the previous one unless kept
loop spin on|off make the main loop spin without resting, to compare load and radio noise
lora noise test [gnss|quiet] | lora noise report Sweep under one changed condition at a time (Wi-Fi goes off for a moment)
lora inject <hex> [rssi] [snr] a packet into the LoRa Scanner as if received (nothing is sent)
sd fill <folder> <count> makes that many small files there, to test a crowded folder
coredump get (Debug Console only) send the raw core dump: use scripts/rdbg.py coredump
reset (Debug Console only) restart at once, even if the main loop is stuck
get <path> | put <path> <size> <sha256> | screenshot (Debug Console only) binary, see rdbg.py
quit close the Debug Console connection
```
In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few run: `help`, `info`, `tasks`, `net`, `reboot`, `boot other`, `wifi status`, and anything starting with `log level`, `crash`, `coredump`, `wifi add`. Anything else answers `not available in Safe Mode`.
## What they do
`scripts/serial_log.sh [seconds] [command…]` records the serial output, and can send commands to the firmware first. For example, `scripts/serial_log.sh 30 short sleep:12 burst` sets short screen timeouts, waits 12 s, then sends a burst of Toasts.
| Command | Effect |
|---|---|
| `burst` | Publishes 5 Notifications at once |
| `key up\|down\|left\|right\|select\|back\|home`, or `key <char>` | Injects a key press |
| `sound on` / `sound off` | Toggles the Sound setting (beep + LED) |
| `short` / `normal` | Screen timeouts 5 s / 10 s, or 30 s / 60 s |
| `wifi add <ssid><TAB><password>` | Adds a Saved Network (so credentials stay out of the repo) |
| `wifi ip <ssid> dhcp` / `wifi ip <ssid> <address>/<prefix> [gateway]` | A Saved Network's IP setting: Automatic, or Fixed. Debug Builds: add `try <seconds>` to go back to the previous setting unless `wifi ip keep` follows |
| `wifi dns <a> [b]` / `wifi dns always on\|off` / `wifi ntp <a> [b]` | DNS servers (used on Fixed networks, or always), and NTP servers |
| `log <text>` | Appends a line to a test IRC Log (`/irc/dev/#test/<date>.log`) |
| `sd card` | What the SD card says it is: type, size, and its identity register (maker, name, revision, serial, date) |
| `sd list` | Lists the files of each Storage Clean-up category |
| `sd fill <folder> <count>` | Debug Builds: makes that many small files in a folder, to test a crowded one |
| `cat <path>` | Prints the first ~1.2 KB of a file on the SD card |
| `irc start` | Starts the IRC Service (normally done by opening the IRC App) |
| `irc stop` | Stops it, as `/quit` does: QUIT if connected, no more retries, and the App stays disconnected until you type |
| `gemini get <url>` | Fetches a Gemini page and prints its header, size, certificate fingerprint and heap use |
| `gemini trust <host> <port> <sha256>` | Pins a certificate by hand (the Gemini App asks when one changes) |
| `irc say <buffer> <text>` | Types into a Buffer, commands included (`irc say 0 /join #test`) |
| `irc dump` | Prints IRC status, memory, and the last lines of each Buffer |
| `wifi status` | Prints Wi-Fi state, network, signal, clock and free heap, then the address, gateway, DNS and NTP servers in use and where each came from |
| `info` | Firmware, uptime, last start reason, memory, Wi-Fi, the SD card and its write faults since boot, and both app slots with their versions and OTA states |
| `tasks` | FreeRTOS tasks over the next second: state, priority, lowest free stack, share of a core, each core's load, and how many passes the main loop made |
| `net` | Bytes each network service has read and written since boot |
| `reboot` / `boot other` | Restart, or restart into the other app slot (a manual Rollback) |
| `log level <0-5>` | ESP-IDF log level |
| `ls [folder]` / `du <path>` | Lists a folder of the SD card with sizes and dates, or counts the files and bytes under a path |
| `cp [-f] <from> <to>` / `mv [-f] <from> <to>` / `rm <path>` / `mkdir <path>` / `cancel` | What the Storage App does, with its rules: copy (folders too), move or rename, delete (a folder with what's in it), new folder. `-f` replaces a file that's in the way; a tab separates paths that hold spaces; `cancel` stops a copy or a delete |
| `install <path>` | Update from SD with that `.ota` file, as Settings → Firmware does |
| `update check` / `update list` / `update status` / `update install <tag>` | The project's releases on Gitea: look at the latest, list the last ten, say what's known, or download and install one (not on a Debug Build) |
| `update pretend <version>` / `update probe <host>` / `update damage cut\|flip <n>` / `update daily` | Debug Builds: pretend to run another version (so a release counts as an update), see whether a server's certificate is accepted, cut or damage the next download, run the daily check again |
| `lora probe` | Finds the radio: chip, oscillator, antenna switch, DIO1 interrupt, noise floor |
| `lora status` | Radio settings, who's listening, packet and error counters, noise floor, task stack |
| `lora rx on` / `lora rx off` | Listens and prints each packet on the console |
| `lora preset <name>` / `lora custom <MHz> <BW kHz> <SF> <CR> <sync hex> [preamble]` | Receive settings: a Meshtastic preset, or anything else (`lora custom 868.1 125 7 5 34 8` for LoRaWAN) |
| `lora capture start` / `lora capture stop` | A LoRa Capture, as `c` in the App |
| `lora sweep on [from MHz] [to MHz] [step kHz]` / `lora sweep off` / `lora sweep dump` | Sweep a band (863 870 100 by default), with a summary every 2 s (floor, strongest, peaks), or print the latest pass |
| `gnss quiet on` / `gnss quiet off` | The "Pause GNSS for LoRa" setting |
| `gnss status` / `gnss restart` | The receiver's state, and a restart of it |
| `gnss track start` / `gnss track stop` | A Track, as `r` in the GNSS App (the reason is printed if it can't start) |
| `gnss nmea on` / `gnss nmea off` | Prints each NMEA sentence the receiver sends, as `nmea: …` |
| `gnss send <sentence>` | Sends a sentence to the receiver, without the `$` and the checksum (it adds them) |
| `lora noise test [gnss\|quiet]` / `lora noise report` | Debug Builds: Sweep under one changed condition at a time to find what raises the noise floor (Wi-Fi goes off for a few seconds); then the result |
| `lora inject <hex> [rssi] [snr]` | Debug Builds: a packet into the Scanner as if received (nothing is sent) |
| `crash` | The last crash: which firmware, why, task, PC and backtrace (from the core dump in flash) |
| `coredump erase` | Forgets the core dump |
| `loop spin on` / `loop spin off` | Debug Builds: make the main loop spin without resting, to compare load and radio noise |
| `crash abort` / `crash wdt` | Debug Builds: crash on purpose, or hang the main loop until the watchdog fires |
| `help` | Lists the commands |
`scripts/flash.sh` stops a running serial log first, since it would hold the port.
site/content/dev/debug/console/debug-console.svg
+54
View File
@@ -0,0 +1,54 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 760 340" role="img" aria-label="Inside the Debug Console. On the PC, rdbg.py connects to TCP 2323 and sends the token first. On the device, the Debug Console task checks the token, serves one client, sends the console ring to the socket, and queues command lines for the main loop. It answers binary commands itself: get, put, screenshot, coredump get and reset. The main loop runs queued commands with runCommand, the same as the USB serial commands, and prints through console.printf into a 4 KB ring, which also goes to USB serial when there is room. ESP-IDF's own log lines are teed into the ring. Card work runs as jobs on the storage task: ls, rm and install from the main loop, get and put from the Debug Console task.">
<defs>
<marker id="dc-arrow" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path d="M0,0 L10,5 L0,10 z" fill="currentColor"/></marker>
<marker id="dc-arrow-hot" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse"><path class="f-accent" d="M0,0 L10,5 L0,10 z"/></marker>
</defs>
<g font-family="JetBrains Mono, ui-monospace, monospace" fill="currentColor">
<rect class="dc-box" x="16" y="40" width="160" height="80" rx="6" stroke-dasharray="4 3"/>
<text x="28" y="62" font-size="12" font-weight="600">your PC</text>
<text x="28" y="84" font-size="11">rdbg.py</text>
<text x="28" y="102" font-size="11" class="dc-dim">token first</text>
<rect class="dc-hot" x="236" y="24" width="260" height="176" rx="8"/>
<text x="250" y="48" font-size="12" font-weight="600">Debug Console task</text>
<text x="250" y="72" font-size="11">token check, one client</text>
<text x="250" y="90" font-size="11">ring → socket, live</text>
<text x="250" y="108" font-size="11">lines → command queue</text>
<text x="250" y="134" font-size="11" class="dc-dim">answers these itself:</text>
<text x="250" y="152" font-size="11">get · put · screenshot</text>
<text x="250" y="170" font-size="11">coredump get · reset</text>
<text x="250" y="188" font-size="10" class="dc-dim">(they work with the loop stuck)</text>
<rect class="dc-panel" x="236" y="244" width="260" height="80" rx="8"/>
<text x="250" y="268" font-size="12" font-weight="600">main loop</text>
<text x="250" y="290" font-size="11">runCommand(): the same</text>
<text x="250" y="308" font-size="11" class="dc-dim">commands as USB serial</text>
<rect class="dc-panel" x="560" y="24" width="184" height="96" rx="8"/>
<text x="574" y="48" font-size="12" font-weight="600">console</text>
<text x="574" y="70" font-size="11">4 KB ring</text>
<text x="574" y="88" font-size="11" class="dc-dim">+ USB serial,</text>
<text x="574" y="106" font-size="11" class="dc-dim"> if there's room</text>
<rect class="dc-box" x="560" y="150" width="184" height="40" rx="6" stroke-dasharray="4 3"/>
<text x="652" y="175" font-size="11" text-anchor="middle">ESP-IDF logs (tee)</text>
<rect class="dc-panel" x="560" y="244" width="184" height="80" rx="8"/>
<text x="574" y="268" font-size="12" font-weight="600">storage task</text>
<text x="574" y="290" font-size="11">SD card jobs</text>
<text x="574" y="308" font-size="11" class="dc-dim">ls rm install get put</text>
<path class="dc-line-hot" d="M180 80 H232" marker-start="url(#dc-arrow-hot)" marker-end="url(#dc-arrow-hot)"/>
<text class="f-accent" x="184" y="72" font-size="10">TCP 2323</text>
<path class="dc-line-hot" d="M300 200 V240" marker-end="url(#dc-arrow-hot)"/>
<text class="f-accent" x="308" y="226" font-size="10">queue</text>
<path class="dc-line" d="M556 56 H500" marker-end="url(#dc-arrow)"/>
<text x="508" y="48" font-size="10" class="dc-dim">read</text>
<path class="dc-line" d="M652 150 V124" marker-end="url(#dc-arrow)"/>
<path class="dc-line" d="M496 262 H516 V100 H556" marker-end="url(#dc-arrow)"/>
<text x="510" y="232" font-size="10" text-anchor="end" class="dc-dim">printf</text>
<path class="dc-line" d="M496 180 H540 V290 H556" marker-end="url(#dc-arrow)"/>
<path class="dc-line" d="M496 306 H556" marker-end="url(#dc-arrow)"/>
<text x="512" y="322" font-size="10" class="dc-dim">jobs</text>
</g>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 4.3 KiB

site/content/dev/debug/console/index.md
+90
View File
@@ -0,0 +1,90 @@
+++
title = "The Debug Console"
description = "Connect to a Debug Build over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do."
weight = 2
[extra]
tag = "Console"
diagrams = true
+++
## Connect
```sh
export RORO_OTA_HOST=10.39.39.12 # or pass -H <ip>
scripts/rdbg.py # interactive: the backlog, then live lines; type commands
scripts/rdbg.py info # one command, and its reply
scripts/rdbg.py -b tasks # the same, with the backlog shown first
```
`scripts/rdbg.py` is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It reads the token from `~/.config/roro9stack/debug-token` and talks to **TCP 2323** on the device's address. In interactive mode, <kbd>Ctrl</kbd>+<kbd>D</kbd> or `quit` leaves. Piped input works too, but `rdbg.py` leaves **as soon as its input ends**, before the replies arrive: keep the input open for a few seconds, as in `(printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py`. For one command, the one-shot form above is simpler.
The device listens **only while Wi-Fi is connected**, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back.
## What you get
On connecting, in order:
1. a **banner**: `roro9stack v0.11.0-3-g12006bd+debug debug console. 'help' lists the commands. Backlog follows.`
2. the **backlog**: the last **4 KB** of console output, oldest first, **boot messages included** (a ring buffer in RAM);
3. then **every new line, live**: everything the firmware prints, and ESP-IDF's own log lines, which are copied into the same stream (they still reach the USB port too);
4. the **replies to your commands**, in the same stream, each preceded by the `> command` line when it runs.
A line `status: heap <free> min <lowest>` appears every 10 seconds in the stream: a free-memory trace you get without asking.
```
> net
net: IRC in 0 out 0
net: Gemini in 0 out 0
net: Debug Console in 1038 out 205115
net: Updates in 4788 out 204
```
If you are not reading fast enough (a slow link), the device says so in the stream instead of stalling: `[... 312 bytes lost: the console ran faster than the network]`.
## The protocol
It is a plain line protocol, easy to speak from anything. This is what `rdbg.py` does, and all it needs:
| Step | Detail |
|---|---|
| Connect | TCP 2323. **One client at a time**: a second one waits until the first leaves |
| Authenticate | Send the token and `\n` within **10 seconds**. The comparison takes the same time whatever you send |
| Refused | After about a second the device sends `denied\n` and hangs up, and prints `debug: refused a client from <ip>` on its own console. No quick retries |
| Commands | One line each, up to 240 bytes. The device **queues at most 8**; past that, `debug: busy, command dropped` |
| Leave | `quit` or `exit` closes the connection |
| Binary commands | `get`, `put`, `screenshot`, `coredump get`: a text header line, then raw bytes (see [files and screens](/dev/debug/files-and-screens/)) |
A command that never runs has not been dropped by the network: the main loop is busy or stuck. That is what the next section is about.
## How commands run
{{ diagram(src="debug-console.svg", min_width=580, caption="The console task only queues command lines. The main loop runs them with the same code as USB serial commands. Binary commands are answered by the console task itself, so they work when the main loop is stuck.") }}
- **Text commands run on the main loop**, exactly like serial ones: they touch the Apps and the Services from the one task allowed to. The main loop prints `> command` as it starts, and the reply follows in the stream.
- **Binary commands run on the console's own task**: `get`, `put`, `screenshot`, `coredump get` and `reset`. A failed `put` closes the connection, so the rest of the file is never read as commands.
- That split is the point: with the **main loop stuck** (an infinite loop, a deadlock), text commands queue forever, but **`reset` still restarts the device at once**, `coredump get` still reads the dump, and `get` still reads files. A loop stuck for 5 seconds is a panic with a core dump anyway: see [Crashes and Safe Mode](/dev/debug/crashes/).
- **Card work stays on the storage task**: `ls`, `rm`, `cp` and `install` from the main loop, `get` and `put` from the console task, all as jobs the storage task runs, so the card is only ever touched from one place.
- Writes to the console **never wait for USB**: a host attached to the USB port but not reading used to stall the main loop for up to two seconds per line.
## Security
- The token is checked **before anything else**, and a wrong one costs a second.
- Anyone on the same network **with the token** can read the console, press keys and restart the device. The console never prints stored secrets (Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
- The stream is **plain text**: fine on a home network, not across the internet. Do not forward port 2323.
- **Release builds have no console at all.** Nothing listens.
The decision is [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
## Without `rdbg.py`
Anything that can open a TCP connection works. The token and each command are just lines:
```python
import socket
s = socket.create_connection(("10.39.39.12", 2323))
s.sendall(b"<token>\n") # then read the banner line
s.sendall(b"info\n") # read until the stream goes quiet
```
`scripts/rdbg.py` adds the parts that need work on the PC side: decoding crash backtraces, saving files and screenshots, and checking checksums.
site/content/dev/debug/crashes.md
+72
View File
@@ -0,0 +1,72 @@
+++
title = "Crashes, core dumps and Safe Mode"
description = "What happens when the firmware crashes: the report, the core dump, the build it is decoded against, the main-loop watchdog and Safe Mode."
weight = 5
[extra]
tag = "Console"
+++
Rollback (see [How an update works](/dev/build/how-an-update-works/)) protects against **new** firmware that fails Probation. These measures cover firmware that was **confirmed and then crashes**: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. They are in **every build**, release and Debug alike; the Debug Console is what makes them convenient. The decision is [ADR 0005](/dev/decisions/0005-safe-mode-crash-reports-watchdog/).
## What a crash leaves behind
- **A core dump** in a flash partition: ESP-IDF writes it on a panic.
- **A crash record** in NVS, written first thing at the next boot: which **version** was running (so after a Rollback has switched slots, the firmware still knows which one crashed), and how many starts in a row followed a crash.
- **A summary on the console** after the restart (task, program counter, reason, backtrace), and a **Notification** on screen.
The `crash` command prints it again whenever you like:
```
> crash
crash: last one in v0.9.0-1-g4ab873e-dirty+debug (panic)
crash: task loopTask, pc 0x4037e179, cause 0, address 0x00000000
crash: reason: abort() was called at PC 0x421209b3 on core 1
crash: backtrace 0x4037e179 0x4037e141 0x4038582d 0x421209b3 ...
crash: elf sha256 3c6a185e5
```
`coredump erase` forgets the dump.
## Decode it: `rdbg.py crash` and `rdbg.py coredump`
An address is no use without the **exact build** that crashed. Every build archives its ELF in **`.pio/elves/`**, named by version and the first 16 hex digits of its SHA-256 (`scripts/version.py`); the core dump names the crashed firmware by the same digest. So a crash can be decoded **after later builds**, including a build you have since replaced:
```sh
scripts/rdbg.py crash # the crash report, with the backtrace turned into functions and source lines
scripts/rdbg.py coredump # fetch the whole core dump, then decode it
scripts/rdbg.py coredump my.bin # ...to a file you name
```
- **`crash`** runs the device's `crash`, finds the ELF by digest (or by version), and passes the backtrace to `scripts/decode_backtrace.sh`, which runs `addr2line` from the build container: one line for each frame, with function and file:line.
- **`coredump`** fetches the raw dump over the console (it works when the main loop is stuck) and runs `scripts/decode_coredump.sh`: `esp-coredump` and GDB, giving **every task's backtrace, the registers, and the crashed task's stack**.
- By hand: `scripts/decode_backtrace.sh <version|digest> <address>...`, and `scripts/decode_coredump.sh <core.bin> <version|digest>`. Both say which ELF they used.
- If no archived ELF matches, they say so: the build was made on another machine, or `.pio/` was cleaned.
## The main loop is watched
Arduino-ESP32 puts only core 0's idle task on the task watchdog, and the main loop runs on core 1: a stuck loop used to hang the device for good, with a frozen screen and a console that could not run commands. Now **a loop stuck for 5 seconds is a panic**, with a core dump, counted towards Safe Mode. The rule that follows: **nothing in the main loop may block for 5 seconds.** Network and card work already run on their own tasks.
An installed update no longer depends on the main loop either: the Update Service restarts into it by itself after 90 seconds.
## Safe Mode
The count of starts that follow a crash (a panic or the watchdog) is kept in NVS. After **three in a row**, the firmware starts **Safe Mode** instead of everything else:
- only the **clock, Wi-Fi, the Update Service and, in a Debug Build, the Debug Console** start: no Apps, no IRC, no SD card;
- the screen says so, with the **address to push an update to**;
- only a few commands run (the [command reference](/dev/debug/commands/) lists them); anything else answers `not available in Safe Mode`.
Any **normal restart** (`reboot`, an update) or **a minute of uptime** resets the count. So Safe Mode is reached by crashing three times *quickly*, and a crash loop costs about half a minute before the device becomes reachable. To leave it: push a fix (`scripts/flash.sh --debug --ota`), or `reboot`.
**Its limit:** if Wi-Fi or the Update Service is what crashes, Safe Mode cannot help, and it takes USB.
## Crash on purpose
On a Debug Build:
```
crash abort # abort(): a panic with a core dump
crash wdt # hang the main loop until the task watchdog fires
```
Use them to see a real report and decode it, to check that **the counter reaches Safe Mode**, and to prove that a Debug Build you are about to rely on will survive its own crashes. The device restarts by itself after either, and the report is there to read when the console comes back.
site/content/dev/debug/debug-builds.md
+56
View File
@@ -0,0 +1,56 @@
+++
title = "Debug Builds"
description = "What a Debug Build is, how to build and flash one, how its token works, and why you should keep one in the fallback slot."
weight = 1
[extra]
tag = "Start here"
+++
## What it is
A Debug Build is built from the same source as a release, with the `cardputer-adv-debug` environment (it `extends` `cardputer-adv` in `platformio.ini`) and `-DRORO_DEBUG`. Differences:
- the **Debug Console** on TCP **2323** (next page);
- extra commands, only meant for testing: crash on purpose, fake an installed version, damage a download, inject a LoRa packet, fill a folder with files (see the [command reference](/dev/debug/commands/));
- the version ends in **`+debug`** (`scripts/version.py`) wherever the version shows: Settings, `info`, the Update Service. A `+debug` version compares **equal** to its release counterpart, so going between the two is never refused as a downgrade.
**It is compiled out of release builds, not switched off by a setting.** A console that runs commands, presses keys and reboots the device is a remote control; in a release build nothing listens and the code is not there.
## Build and flash one
Everything runs in Docker (see [Build, test and release](/dev/build/build-and-test/)). Over USB:
```sh
scripts/flash.sh --debug # builds cardputer-adv-debug, uploads, opens the serial monitor
```
Once a Debug Build is on the device, every later one can go over Wi-Fi, with no cable:
```sh
export RORO_OTA_HOST=10.39.39.12 # the address Settings > Firmware shows
scripts/flash.sh --debug --ota # builds, signs and pushes; the device installs and restarts
```
The device's address is on **Settings → Firmware** ("Push to", which also gives port 3232, the update port). The Debug Console is on the same address, port 2323. See [Flash and update](/dev/build/flash/) for the update side.
## The token
The console asks for a secret first. It is **128 random bits**, made the first time any build runs (`scripts/_docker.sh`), kept in `~/.config/roro9stack/debug-token` next to the update-signing key, and passed into the build container as `RORO_DEBUG_TOKEN`. `scripts/debug_flags.py` compiles it into the firmware, and **refuses to build a Debug Build without one**, rather than fall back on a default.
- It is **never committed.** Releases have no console, so nothing of it is published: CI builds a Debug Build on a pull request to prove it still compiles, but never publishes it, because each one carries its builder's token.
- It guards against **the network**, not against someone who holds the device: anyone with USB access can flash anything anyway ([ADR 0003](/dev/decisions/0003-own-signature-check-not-secure-boot/)).
- To change it, delete the file and build again; the new token goes into the next Debug Build you flash.
## Keep a Debug Build in the other slot
The device has two app slots, so an update never overwrites the running firmware. If a new firmware crashes during [Probation](/dev/build/how-an-update-works/), the device goes **back to the previous one**, whatever that is. As long as you develop on Debug Builds, **the firmware a crash falls back to has the console too**, so a bad update never costs you remote access. Pushing a *release* build over a Debug Build leaves the Debug Build in the other slot until the next update overwrites it.
So the habit is: develop on Debug Builds, release by tag, and think twice before pushing a release over the only Debug Build you have.
## Releases and Debug Builds
- A Debug Build shows the latest release in Settings → Firmware but **never installs it**: that would replace the console with a release that has none. Update a Debug Build from the PC with `scripts/flash.sh --debug --ota`.
- A Debug Build still checks and lists releases, which is useful for testing the update path: `update pretend` makes a released version count as newer (see [Drive the UI](/dev/debug/drive-the-ui/)).
- **Safe Mode** (after 3 crashes in a row) keeps the Debug Console running, so a crash loop is something you fix remotely: see [Crashes and Safe Mode](/dev/debug/crashes/).
The reasoning is in [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
site/content/dev/debug/drive-the-ui.md
+123
View File
@@ -0,0 +1,123 @@
+++
title = "Drive the UI from your desk"
description = "Press keys, take screenshots, fake inputs and test the awkward paths (updates, crashes, fixed IPs, crowded folders) without touching the device."
weight = 4
[extra]
tag = "Console"
+++
Everything the keyboard can do, a command can do, and everything on the screen can be looked at remotely. That makes the Cardputer testable like a web page: **act, look, repeat**.
## Keys
```
key up|down|left|right|select|back|home|del|tab|space
key a # any single character: it is typed
```
Two things to know before you use them:
1. **A name `key` does not know is `select`.** `key sleect` presses Enter. A single character is typed as that character; anything else that is not a known name is treated as Enter. Check what you type.
2. **A key that wakes a dark screen only wakes it.** The device's power policy swallows the key press that turns the screen back on, as it does for the real keyboard: the first `key` after the screen went off does nothing else. Send `key back` (harmless) first, or keep the screen on with the `normal` and `short` commands below.
`Fn` combinations, modifiers and the compose key have no command: the arrows are `key up|down|left|right`, and `key back` is the back key (`` ` `` on the device). Text is typed one character at a time.
## Look before you press
**Take a screenshot before any key that deletes, renames or installs.** A blind sequence of `key` commands goes wrong the moment the screen is not where you think it is, and the screen is often not where you think it is: a Toast, a dialog that has not closed, a different App. A sequence that was meant to open a note once renamed real data instead.
```sh
scripts/rdbg.py key home # to the Launcher
scripts/rdbg.py screenshot a.png # look: is it the Launcher?
scripts/rdbg.py key down
scripts/rdbg.py key select
scripts/rdbg.py screenshot b.png # look again before the next destructive step
```
- Prefer **reading a state** to assuming it: `info`, `ls <folder>`, `cat <file>`, `irc dump`, `gnss status`, `lora status`, `wifi status`, `update status`.
- Test on a **scratch folder** on the card, not on your real files.
- For anything that deletes (`rm`, a delete dialog), `ls` first and `ls` after.
## Keep the screen on, and make timeouts short
```
short # screen dims after 5 s, turns off after 10 s: to test the screen policy
normal # back to 30 s and 60 s
burst # five Toasts at once: to test notifications
sound on | sound off
```
## Fake the inputs
Each of these puts something in, **without** the outside world:
| Command | What it does |
|---|---|
| `lora inject <hex> [rssi] [snr]` | A packet into the LoRa Scanner as if the radio had received it. Nothing is sent |
| `irc say <buffer> <text>` | Types into an IRC buffer, commands included: `irc say 0 /join #test` |
| `log <text>` | Adds a line to a test IRC log |
| `gnss send <sentence>` | Sends an NMEA sentence **to** the GNSS receiver (the checksum is added), to configure it |
| `gemini get <url>` | Fetches a page and reports header, size, certificate and heap use, without the App |
| `gemini trust <host> <port> <sha256>` | Pins a certificate by hand |
| `sd fill <folder> <count>` | **Debug Build.** Makes that many small files in a folder, to test a crowded one (the Storage App shows the first 256) |
| `wifi add <ssid><TAB><password>` | Adds a Saved Network, so credentials stay out of the repository |
## Test the update path
An update that goes wrong is the case you most want to rehearse, and a Debug Build can make it go wrong **on purpose**:
```
update status # what the device runs, what failed here before, the daily check, heap
update check | update list # look at the server: the latest release, or the last ten
update pretend v0.9.0 # take the running version to be v0.9.0: the latest release now counts as "new"
update damage cut 50000 # the next download is cut after 50000 bytes
update damage flip 100000 # ...or has the byte at offset 100000 damaged
update install v0.11.0 force # try the install
update probe git.twis.la # is that server's certificate accepted? (the two ISRG roots only)
update daily # forget today's daily check: it runs again at the next tick
update pretend off # back to the real version
```
- **`force` is needed on a Debug Build.** A plain `update install <tag>` answers `Debug Build: update from the PC`, because installing a release would replace the console with a build that has none. `force` is accepted only on a Debug Build, and only from the console.
- **A damaged download must be refused cleanly:** the Update Service checks the signature after the first 160 bytes and the image hash at the end, so a cut or a flipped byte must end in a refusal with **nothing switched**. Check `info` afterwards: both slots, and `update: confirmed`.
- **An undamaged `force` install really installs** the release, into the other slot. The Debug Build stays where it was until the next update overwrites it, and a Rollback returns to it, but think before you do it.
- The server is read by the device itself, with Wi-Fi up; the download is one TLS connection, about 52 KB of heap at its peak, so IRC steps aside (see [the memory limit](/howto/not-enough-memory/)). `status: heap` in the stream shows it happen.
For crashes during Probation, see [Crashes and Safe Mode](/dev/debug/crashes/).
## Test a network change without losing the console
The Debug Console runs over the Wi-Fi you are about to change, which is the usual way to lock yourself out. A **trial IP setting** takes care of it:
```
wifi ip MyNet 10.39.39.50/24 10.39.39.1 try 60 # use this address for 60 s...
wifi ip keep # ...and keep it, if you could still reach the device
```
If you do not send `wifi ip keep` in time, the device goes back to the **previous** setting by itself, and the console comes back with it. (A Debug Build command: `try` is not in release builds.)
## Measure
```
info # firmware, uptime, last start reason, heap now/lowest/largest block, chip temperature, Wi-Fi, SD faults, both app slots
tasks # each task over the next second: state, priority, least free stack, CPU share; each core's load; main-loop passes
net # bytes each network service has read and written since start
```
`tasks` takes a second to answer. A low number in the `stack` column is a risk (the System App shows it in the warning colour under 512 bytes). `loop spin on|off` (Debug Build) makes the main loop spin without resting, to compare load and radio noise. And the `status: heap` line every 10 seconds in the stream is the cheapest memory trace there is: watch it while you do the thing you suspect.
```
task st pri stack cpu% core
loopTask R 1 1828 1.3 1
wifi B 23 4072 0.7 0
debug B 1 3088 0.5 -1
...
load: core 0 2 %, core 1 1 %
loop: 50 passes in the last second, chip 35.3 C
```
The [System App](/guide/system/) shows the same, live, on the device.
## Radio experiments
`lora preset <name>` and `lora custom <MHz> <BW kHz> <SF> <CR> <sync hex> [preamble]` change what the receiver listens to (receive only: the radio never transmits), `lora sweep on [from] [to] [step]` takes a survey, and `lora noise test [gnss|quiet]` (Debug Build) runs a Sweep under one changed condition at a time, with Wi-Fi off for a moment, to find what raises the noise floor. `lora probe` finds the radio and reports its chip, oscillator, antenna switch, interrupt line and noise floor. Details in the [command reference](/dev/debug/commands/).
site/content/dev/debug/files-and-screens.md
+60
View File
@@ -0,0 +1,60 @@
+++
title = "Files, screenshots and the SD card"
description = "Copy files to and from the card, take a screenshot, fetch a core dump and restart the device, all over Wi-Fi, with checksums."
weight = 3
[extra]
tag = "Console"
+++
These are the **binary commands**: a text header line, then raw bytes. The console task answers them itself, so they keep working when the main loop is stuck. `scripts/rdbg.py` handles each one on the PC side; the protocol is given too, for your own tools.
## `get`: card to PC
```sh
scripts/rdbg.py get /gnss/tracks/20261004-142530.gpx # saved here, under its own name
scripts/rdbg.py get /captures/lora/capture.pcap my-capture.pcap
```
The device answers `get: data <size>`, then exactly `<size>` bytes, then `get: end`. A path it cannot open (missing, or a folder) gives `get: error cannot open <path>`. The script prints the size and the speed.
## `put`: PC to card
```sh
scripts/rdbg.py put roro9stack-v0.12.0.ota # to /updates/roro9stack-v0.12.0.ota
scripts/rdbg.py put notes.txt /notes/from-the-pc.txt # to a path you choose
```
The default destination is `/updates/<name>`, so this is also the way to **install an update from the SD card without touching the device**: `put` the `.ota` file, then `scripts/rdbg.py install /updates/<name>`.
The device does a few things you want from a file transfer:
- the PC sends the **size and the SHA-256** first (`put <path> <size> <sha256>`); the device answers `put: ready <size>` or `put: error <why>` (no card, **not enough space**: it wants the size plus 64 KB free, or a path or size it refuses);
- it writes to a **temporary `.part` file**, creating missing folders, and only **renames it into place after the whole file has been read back from the card and its SHA-256 matches**: the checksum covers what is on the card, not what arrived;
- a write the card refuses is **retried up to 3 times**, cutting the file back to the last good byte, and gives up rather than leave a hole in the middle;
- it ends with `put: done <path> <size> B`, or `put: error <why>` and **a closed connection** (so the rest of the file is never read as commands). A failed transfer leaves nothing on the card.
Speed is about 300 KB/s. The same transfer exists over USB serial, for a device with no Wi-Fi: `scripts/sd_put.sh <file> [card path]` (about 30 seconds for 1.6 MB, with the card left in).
## `screenshot`: the screen as a PNG
```sh
scripts/rdbg.py screenshot ui.png # 480x270: the 240x135 screen at 2x
```
The device sends `screenshot: rgb332 <width> <height>` and then **one byte per pixel**: the frame the UI composed off-screen, in RGB332 (RRRGGGBB), the way M5GFX stores an 8-bit sprite. The script expands it and doubles it into a PNG.
- It is read **as it stands**, while the UI may be drawing, so it can **tear**. It is for looking at, not for pixel-exact comparison.
- It is the real thing: the screenshots on this site, in the [user guide](/guide/) and the [devlog](/devlog/), were taken this way.
- Its main use is in a loop: send a key, wait a moment, take a screenshot, look. See [Drive the UI](/dev/debug/drive-the-ui/).
## `coredump get`: the crash dump
The raw contents of the core dump partition: `coredump: data <size>`, the bytes, `coredump: end`, or `coredump: none`. `scripts/rdbg.py coredump` fetches and decodes it in one go: see [Crashes and Safe Mode](/dev/debug/crashes/).
## `reset`: restart now
```sh
scripts/rdbg.py reset
```
The console prints `debug: restarting now` and restarts the chip after a moment. It does not go through the main loop, so it works when the loop is stuck. (The text command `reboot` does go through the main loop: a clean restart. `boot other` restarts into the other app slot: a manual rollback.)