Public Access
One firmware: the Debug Console in every build, off until switched on, with the device's own token
There is no Debug Build any more (ADR 0010, issue #68, Q188 to Q195). The console and the test commands are compiled into every firmware. It listens only while Settings > Debug Console is on, which isn't the default; off, neither its task nor its 4 KB ring exists. The token is made by the device and shown on that page; a client proves it knows it by answering a challenge with an HMAC, so it never crosses the network, and five wrong answers close the console for a minute. DBG in the Status Bar while it listens. Over USB serial only: debug on, debug token <value>, debug token new. scripts/flash.sh --debug uses them to set a device up with the developer's token. scripts/rdbg.py takes the token from -t, $RORO_DEBUG_TOKEN or the file, answers the challenge, and fetches a release's ELF to decode a crash. Gone: the cardputer-adv-debug environment, RORO_DEBUG, the +debug version, scripts/debug_flags.py, update install ... force, and the rule that a Debug Build doesn't install releases. Old clients and old firmwares don't talk to each other. Against the builds it replaces: 30 KB more flash and 88 bytes more static RAM than the release, 4 KB less RAM than the Debug Build. 468 host tests. Checked on the device: off by default, login, the pause after wrong tokens, Safe Mode with the console, the setting surviving an update, debug off. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT
This commit is contained in:
Claude Opus 5.5
@@ -10,4 +10,4 @@ weight = 2
|
||||
eyebrow = "Developer docs"
|
||||
+++
|
||||
|
||||
The firmware is built, tested and released **in Docker**, so that a clean machine, your machine and the CI runner all get the same result. These pages say how. For the Debug Console, which is how you work on a device once it is flashed, see [Debug Builds and the Debug Console](/dev/debug/).
|
||||
The firmware is built, tested and released **in Docker**, so that a clean machine, your machine and the CI runner all get the same result. These pages say how. For the Debug Console, which is how you work on a device once it is flashed, see [The Debug Console](/dev/debug/).
|
||||
|
||||
@@ -26,13 +26,13 @@ The framework is rebuilt with the TLS settings in `platformio.ini` (`custom_sdkc
|
||||
|
||||
## CI and releases
|
||||
|
||||
Gitea Actions (`.gitea/workflows/ci.yml`, docs/milestones/R1.md) runs the host tests on every push to `main`, and on a pull request also builds the release firmware and the Debug Build: changes reach `main` through pull requests. Pushing a tag `v*` runs all of it and publishes a release on Gitea with:
|
||||
Gitea Actions (`.gitea/workflows/ci.yml`, docs/milestones/R1.md) runs the host tests on every push to `main`, and on a pull request also builds the firmware: changes reach `main` through pull requests. Pushing a tag `v*` runs all of it and publishes a release on Gitea with:
|
||||
|
||||
- `roro9stack-<version>.ota`, the signed Update File;
|
||||
- `roro9stack-<version>-factory.bin`, the whole flash image for a first install over USB;
|
||||
- `roro9stack-<version>.elf.gz`, to decode crash reports from that build;
|
||||
- `SHA256SUMS`.
|
||||
|
||||
CI signs with the project's key, held as a repository secret (ADR 0008). Debug Builds are built but never published: each carries its builder's Debug Console token.
|
||||
CI signs with the project's key, held as a repository secret (ADR 0008). There is one firmware: the Debug Console is in every build, switched off until its owner switches it on (ADR 0010).
|
||||
|
||||
`scripts/ota_verify.py <file.ota>` checks an Update File on a PC the way a device does. `scripts/release_build.sh` and `scripts/release_publish.py` are what the workflow runs; they work the same by hand.
|
||||
|
||||
@@ -54,4 +54,4 @@ The connection is checked against the two ISRG roots Let's Encrypt chains end in
|
||||
|
||||
**IRC steps aside.** A secure connection takes about 52 KB of memory at its peak, and IRC's own takes 40 KB of the 107 KB there is. A check or an install you ask for makes IRC disconnect for the few seconds it takes and reconnect afterwards. The daily check never does that: with IRC connected it waits for a moment when IRC isn't, so while IRC stays connected for days it doesn't run, and **Latest release** is the way to check.
|
||||
|
||||
**A Debug Build** shows the latest release but doesn't install it: releases have no Debug Console (they aren't built with one, so that nobody's console token is published), and installing one would take it away. A Debug Build is updated from the PC with `scripts/flash.sh --debug --ota`.
|
||||
**There is no separate Debug Build** any more (ADR 0010): every firmware installs releases, and the Debug Console is a setting, which an update leaves as it was.
|
||||
|
||||
@@ -24,7 +24,7 @@ scripts/ota_keygen.sh # once: creates the key p
|
||||
scripts/make_ota.py firmware.bin v0.12.0 out.ota # wraps and signs an image (the key: $RORO_OTA_KEY or the default path)
|
||||
scripts/ota_verify.py out.ota # checks a file the way a device does, on the PC
|
||||
scripts/ota_push.py out.ota 10.39.39.12 # pushes it to the Update Service, TCP 3232
|
||||
scripts/flash.sh --ota 10.39.39.12 # builds, signs and pushes in one go (add --debug for a Debug Build)
|
||||
scripts/flash.sh --ota 10.39.39.12 # builds, signs and pushes in one go
|
||||
```
|
||||
|
||||
`ota_push.py` prints the device's answer (`OK …`), or says the device **refused the update and hung up**, with the reason on the device's screen: the header is checked first, so a refused file stops mid-transfer.
|
||||
|
||||
@@ -10,7 +10,7 @@
|
||||
<text x="30" y="75" font-size="11" class="wi-dim">builds, signs, pushes</text>
|
||||
<rect class="wi-box" x="16" y="112" width="220" height="56" rx="6"/>
|
||||
<text x="30" y="135" font-size="12" font-weight="600">rdbg.py put</text>
|
||||
<text x="30" y="155" font-size="11" class="wi-dim">Debug Build, Wi-Fi 2323</text>
|
||||
<text x="30" y="155" font-size="11" class="wi-dim">Debug Console, Wi-Fi 2323</text>
|
||||
<rect class="wi-box" x="16" y="192" width="220" height="56" rx="6"/>
|
||||
<text x="30" y="215" font-size="12" font-weight="600">sd_put.sh</text>
|
||||
<text x="30" y="235" font-size="11" class="wi-dim">USB serial, card stays in</text>
|
||||
|
||||
|
Before Width: | Height: | Size: 4.3 KiB After Width: | Height: | Size: 4.3 KiB |
@@ -1,5 +1,5 @@
|
||||
+++
|
||||
title = "Debug Builds and the Debug Console"
|
||||
title = "The Debug Console"
|
||||
description = "A firmware you can drive from your desk: its console over Wi-Fi, its screen as a PNG, its SD card, its crash dumps, and a way back when an update goes wrong."
|
||||
template = "guide-index.html"
|
||||
page_template = "guide-page.html"
|
||||
@@ -10,13 +10,13 @@ weight = 1
|
||||
eyebrow = "Developer docs"
|
||||
+++
|
||||
|
||||
A **Debug Build** is the same firmware plus a **Debug Console**: the serial console, over Wi-Fi, behind a token. It is the most useful thing in the project. With it you can:
|
||||
The **Debug Console** is the serial console, over Wi-Fi, for whoever holds the device's token. It is in **every firmware**, switched off until you switch it on, and it is the most useful thing in the project. With it you can:
|
||||
|
||||
- **see everything the device prints**, boot messages included, without a cable;
|
||||
- **run every serial command** from your desk;
|
||||
- **press keys** and **take screenshots**, so a UI change can be tested and looked at remotely;
|
||||
- **copy files** to and from the SD card;
|
||||
- **read crash reports and fetch core dumps**, decoded against the exact build that crashed;
|
||||
- **update the firmware** over the same Wi-Fi, and know that a bad update rolls back to a build that still has the console.
|
||||
- **update the firmware** over the same Wi-Fi, and know that a bad update rolls back by itself.
|
||||
|
||||
Start with [Debug Builds](/dev/debug/debug-builds/) to put one on a device, then [the Debug Console](/dev/debug/console/). The rest are what you can do with it.
|
||||
Start with [Switch the console on](/dev/debug/switch-it-on/), then [the Debug Console](/dev/debug/console/). The rest are what you can do with it.
|
||||
|
||||
@@ -10,7 +10,7 @@ tag = "Reference"
|
||||
+++
|
||||
## What `help` prints
|
||||
|
||||
The firmware's own list, read from `src/main.cpp`. These work in every build, over USB serial:
|
||||
The firmware's own list, read from `src/main.cpp`. Every firmware has all of them, over USB serial and over the Debug Console. The ones marked *Debug Console only* exist only over Wi-Fi, where [`rdbg.py`](/dev/debug/files-and-screens/) speaks them, and the ones marked *USB serial only* only over the cable:
|
||||
|
||||
```
|
||||
info firmware, uptime, memory, Wi-Fi, app slots
|
||||
@@ -37,11 +37,8 @@ irc start | irc stop | irc dump | irc say <buffer> <text>
|
||||
install <path.ota> Update from SD
|
||||
update check | list | status | install <tag> the project's releases on Gitea
|
||||
sd card | sd list | cat <path> | log <text> | burst | sound on|off | short | normal
|
||||
```
|
||||
|
||||
A **Debug Build** adds these (the last ones, marked *Debug Console only*, exist only over Wi-Fi, where [`rdbg.py`](/dev/debug/files-and-screens/) speaks them):
|
||||
|
||||
```
|
||||
debug status | debug off the Debug Console over Wi-Fi (Settings > Debug Console)
|
||||
debug on | debug token <16 to 64 characters> | debug token new (USB serial only) switch it on, set its token
|
||||
crash abort|wdt crash on purpose (to test crash reports and Safe Mode)
|
||||
wifi ip ... try <seconds> | wifi ip keep a trial IP setting: back to the previous one unless kept
|
||||
loop spin on|off make the main loop spin without resting, to compare load and radio noise
|
||||
@@ -54,7 +51,7 @@ get <path> | put <path> <size> <sha256> | screenshot (Debug Console only) bina
|
||||
quit close the Debug Console connection
|
||||
```
|
||||
|
||||
In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few run: `help`, `info`, `tasks`, `net`, `reboot`, `boot other`, `wifi status`, and anything starting with `log level`, `crash`, `coredump`, `wifi add`. Anything else answers `not available in Safe Mode`.
|
||||
In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few run: `help`, `info`, `tasks`, `net`, `reboot`, `boot other`, `wifi status`, and anything starting with `log level`, `crash`, `coredump`, `wifi add`, `debug`. Anything else answers `not available in Safe Mode`.
|
||||
|
||||
## What they do
|
||||
|
||||
@@ -67,12 +64,12 @@ In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few r
|
||||
| `sound on` / `sound off` | Toggles the Sound setting (beep + LED) |
|
||||
| `short` / `normal` | Screen timeouts 5 s / 10 s, or 30 s / 60 s |
|
||||
| `wifi add <ssid><TAB><password>` | Adds a Saved Network (so credentials stay out of the repo) |
|
||||
| `wifi ip <ssid> dhcp` / `wifi ip <ssid> <address>/<prefix> [gateway]` | A Saved Network's IP setting: Automatic, or Fixed. Debug Builds: add `try <seconds>` to go back to the previous setting unless `wifi ip keep` follows |
|
||||
| `wifi ip <ssid> dhcp` / `wifi ip <ssid> <address>/<prefix> [gateway]` | A Saved Network's IP setting: Automatic, or Fixed. Add `try <seconds>` to go back to the previous setting unless `wifi ip keep` follows |
|
||||
| `wifi dns <a> [b]` / `wifi dns always on\|off` / `wifi ntp <a> [b]` | DNS servers (used on Fixed networks, or always), and NTP servers |
|
||||
| `log <text>` | Appends a line to a test IRC Log (`/irc/dev/#test/<date>.log`) |
|
||||
| `sd card` | What the SD card says it is: type, size, and its identity register (maker, name, revision, serial, date) |
|
||||
| `sd list` | Lists the files of each Storage Clean-up category |
|
||||
| `sd fill <folder> <count>` | Debug Builds: makes that many small files in a folder, to test a crowded one |
|
||||
| `sd fill <folder> <count>` | Makes that many small files in a folder, to test a crowded one |
|
||||
| `cat <path>` | Prints the first ~1.2 KB of a file on the SD card |
|
||||
| `irc start` | Starts the IRC Service (normally done by opening the IRC App) |
|
||||
| `irc stop` | Stops it, as `/quit` does: QUIT if connected, no more retries, and the App stays disconnected until you type |
|
||||
@@ -89,8 +86,8 @@ In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few r
|
||||
| `ls [folder]` / `du <path>` | Lists a folder of the SD card with sizes and dates, or counts the files and bytes under a path |
|
||||
| `cp [-f] <from> <to>` / `mv [-f] <from> <to>` / `rm <path>` / `mkdir <path>` / `cancel` | What the Storage App does, with its rules: copy (folders too), move or rename, delete (a folder with what's in it), new folder. `-f` replaces a file that's in the way; a tab separates paths that hold spaces; `cancel` stops a copy or a delete |
|
||||
| `install <path>` | Update from SD with that `.ota` file, as Settings → Firmware does |
|
||||
| `update check` / `update list` / `update status` / `update install <tag>` | The project's releases on Gitea: look at the latest, list the last ten, say what's known, or download and install one (not on a Debug Build) |
|
||||
| `update pretend <version>` / `update probe <host>` / `update damage cut\|flip <n>` / `update daily` | Debug Builds: pretend to run another version (so a release counts as an update), see whether a server's certificate is accepted, cut or damage the next download, run the daily check again |
|
||||
| `update check` / `update list` / `update status` / `update install <tag>` | The project's releases on Gitea: look at the latest, list the last ten, say what's known, or download and install one |
|
||||
| `update pretend <version>` / `update probe <host>` / `update damage cut\|flip <n>` / `update daily` | Pretend to run another version (so a release counts as an update), see whether a server's certificate is accepted, cut or damage the next download, run the daily check again |
|
||||
| `lora probe` | Finds the radio: chip, oscillator, antenna switch, DIO1 interrupt, noise floor |
|
||||
| `lora status` | Radio settings, who's listening, packet and error counters, noise floor, task stack |
|
||||
| `lora rx on` / `lora rx off` | Listens and prints each packet on the console |
|
||||
@@ -102,12 +99,14 @@ In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few r
|
||||
| `gnss track start` / `gnss track stop` | A Track, as `r` in the GNSS App (the reason is printed if it can't start) |
|
||||
| `gnss nmea on` / `gnss nmea off` | Prints each NMEA sentence the receiver sends, as `nmea: …` |
|
||||
| `gnss send <sentence>` | Sends a sentence to the receiver, without the `$` and the checksum (it adds them) |
|
||||
| `lora noise test [gnss\|quiet]` / `lora noise report` | Debug Builds: Sweep under one changed condition at a time to find what raises the noise floor (Wi-Fi goes off for a few seconds); then the result |
|
||||
| `lora inject <hex> [rssi] [snr]` | Debug Builds: a packet into the Scanner as if received (nothing is sent) |
|
||||
| `lora noise test [gnss\|quiet]` / `lora noise report` | Sweep under one changed condition at a time to find what raises the noise floor (Wi-Fi goes off for a few seconds); then the result |
|
||||
| `lora inject <hex> [rssi] [snr]` | A packet into the Scanner as if received (nothing is sent) |
|
||||
| `crash` | The last crash: which firmware, why, task, PC and backtrace (from the core dump in flash) |
|
||||
| `coredump erase` | Forgets the core dump |
|
||||
| `loop spin on` / `loop spin off` | Debug Builds: make the main loop spin without resting, to compare load and radio noise |
|
||||
| `crash abort` / `crash wdt` | Debug Builds: crash on purpose, or hang the main loop until the watchdog fires |
|
||||
| `loop spin on` / `loop spin off` | Make the main loop spin without resting, to compare load and radio noise |
|
||||
| `crash abort` / `crash wdt` | Crash on purpose, or hang the main loop until the watchdog fires |
|
||||
| `debug status` / `debug off` | The Debug Console: whether it's on, has a token and a client; switch it off |
|
||||
| `debug on` / `debug token <value>` / `debug token new` | USB serial only: switch it on (making a token if there's none), give it a token of 16 to 64 characters, or make a new one. The token is never printed |
|
||||
| `help` | Lists the commands |
|
||||
|
||||
`scripts/flash.sh` stops a running serial log first, since it would hold the port.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
+++
|
||||
title = "The Debug Console"
|
||||
description = "Connect to a Debug Build over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do."
|
||||
description = "Connect to the console over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do."
|
||||
weight = 2
|
||||
[extra]
|
||||
tag = "Console"
|
||||
@@ -17,16 +17,16 @@ scripts/rdbg.py info # one command, and its reply
|
||||
scripts/rdbg.py -b tasks # the same, with the backlog shown first
|
||||
```
|
||||
|
||||
`scripts/rdbg.py` is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It reads the token from `~/.config/roro9stack/debug-token` and talks to **TCP 2323** on the device's address. In interactive mode, <kbd>Ctrl</kbd>+<kbd>D</kbd> or `quit` leaves. Piped input works too, but `rdbg.py` leaves **as soon as its input ends**, before the replies arrive: keep the input open for a few seconds, as in `(printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py`. For one command, the one-shot form above is simpler.
|
||||
`scripts/rdbg.py` is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It takes the token from `-t`, `$RORO_DEBUG_TOKEN` or `~/.config/roro9stack/debug-token` ([Switch the console on](/dev/debug/switch-it-on/)) and talks to **TCP 2323** on the device's address. In interactive mode, <kbd>Ctrl</kbd>+<kbd>D</kbd> or `quit` leaves. Piped input works too, but `rdbg.py` leaves **as soon as its input ends**, before the replies arrive: keep the input open for a few seconds, as in `(printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py`. For one command, the one-shot form above is simpler.
|
||||
|
||||
The device listens **only while Wi-Fi is connected**, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back.
|
||||
The device listens **only while the console is switched on and Wi-Fi is connected**, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back.
|
||||
|
||||
## What you get
|
||||
|
||||
On connecting, in order:
|
||||
|
||||
1. a **banner**: `roro9stack v0.11.0-3-g12006bd+debug debug console. 'help' lists the commands. Backlog follows.`
|
||||
2. the **backlog**: the last **4 KB** of console output, oldest first, **boot messages included** (a ring buffer in RAM);
|
||||
1. a **banner**: `roro9stack v0.12.0 debug console. 'help' lists the commands. Backlog follows.`
|
||||
2. the **backlog**: the last **4 KB** of console output, oldest first (a ring buffer in RAM, kept while the console is switched on: boot messages included, if it was on at boot);
|
||||
3. then **every new line, live**: everything the firmware prints, and ESP-IDF's own log lines, which are copied into the same stream (they still reach the USB port too);
|
||||
4. the **replies to your commands**, in the same stream, each preceded by the `> command` line when it runs.
|
||||
|
||||
@@ -49,12 +49,17 @@ It is a plain line protocol, easy to speak from anything. This is what `rdbg.py`
|
||||
| Step | Detail |
|
||||
|---|---|
|
||||
| Connect | TCP 2323. **One client at a time**: a second one waits until the first leaves |
|
||||
| Authenticate | Send the token and `\n` within **10 seconds**. The comparison takes the same time whatever you send |
|
||||
| Refused | After about a second the device sends `denied\n` and hangs up, and prints `debug: refused a client from <ip>` on its own console. No quick retries |
|
||||
| Challenge | The device sends one line: `roro9stack debug console, challenge <32 hex digits>`. Those are 16 random bytes, new each time |
|
||||
| Answer | Within **10 seconds**, send the **HMAC-SHA256 of those 16 bytes, keyed by the token**, as 64 hex digits and `\n`. The token is the tidied one: capitals, no dashes |
|
||||
| Accepted | The banner line, then the backlog |
|
||||
| Refused | After about a second the device sends `denied\n` and hangs up, and prints `debug: refused a client from <ip>` on its own console |
|
||||
| Locked | **Five wrong answers in a row** close the console to everyone for 60 seconds: it answers `locked\n` and hangs up, and a Toast on the device names the address they came from |
|
||||
| Commands | One line each, up to 240 bytes. The device **queues at most 8**; past that, `debug: busy, command dropped` |
|
||||
| Leave | `quit` or `exit` closes the connection |
|
||||
| Binary commands | `get`, `put`, `screenshot`, `coredump get`: a text header line, then raw bytes (see [files and screens](/dev/debug/files-and-screens/)) |
|
||||
|
||||
**The token never crosses the network.** Someone on the same Wi-Fi who records a login gets a challenge and its answer, which are no use for the next challenge. The comparison on the device takes the same time whatever it is given.
|
||||
|
||||
A command that never runs has not been dropped by the network: the main loop is busy or stuck. That is what the next section is about.
|
||||
|
||||
## How commands run
|
||||
@@ -69,22 +74,26 @@ A command that never runs has not been dropped by the network: the main loop is
|
||||
|
||||
## Security
|
||||
|
||||
- The token is checked **before anything else**, and a wrong one costs a second.
|
||||
- Anyone on the same network **with the token** can read the console, press keys and restart the device. The console never prints stored secrets (Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
|
||||
- The stream is **plain text**: fine on a home network, not across the internet. Do not forward port 2323.
|
||||
- **Release builds have no console at all.** Nothing listens.
|
||||
- **Off by default.** Nothing listens until the console is switched on at the device, or over USB.
|
||||
- The login is a **challenge and an answer**: the token itself is never sent, and five wrong answers close the console for a minute.
|
||||
- Anyone on the same network **with the token** can read the console, press keys, copy the SD card's files and restart the device. The console never prints stored secrets (the token, Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
|
||||
- They cannot change the firmware: an update still has to be **signed**.
|
||||
- The stream after the login is **plain text**: fine on a home network, not across the internet. Do not forward port 2323.
|
||||
|
||||
The decision is [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
|
||||
The decisions are [ADR 0010](/dev/decisions/0010-debug-console-in-every-build/) and, for how the console works inside, [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
|
||||
|
||||
## Without `rdbg.py`
|
||||
|
||||
Anything that can open a TCP connection works. The token and each command are just lines:
|
||||
Anything that can open a TCP connection and compute an HMAC works:
|
||||
|
||||
```python
|
||||
import socket
|
||||
import hashlib, hmac, socket
|
||||
token = "K7QF3M2X9WBDHT4P6RNC" # tidied: capitals, no dashes
|
||||
s = socket.create_connection(("10.39.39.12", 2323))
|
||||
s.sendall(b"<token>\n") # then read the banner line
|
||||
s.sendall(b"info\n") # read until the stream goes quiet
|
||||
challenge = s.makefile().readline().split()[-1] # "... challenge 0f1e2d..."
|
||||
answer = hmac.new(token.encode(), bytes.fromhex(challenge), hashlib.sha256).hexdigest()
|
||||
s.sendall((answer + "\n").encode()) # then read the banner line
|
||||
s.sendall(b"info\n") # and what follows, until the stream goes quiet
|
||||
```
|
||||
|
||||
`scripts/rdbg.py` adds the parts that need work on the PC side: decoding crash backtraces, saving files and screenshots, and checking checksums.
|
||||
|
||||
@@ -6,7 +6,7 @@ weight = 5
|
||||
tag = "Console"
|
||||
+++
|
||||
|
||||
Rollback (see [How an update works](/dev/build/how-an-update-works/)) protects against **new** firmware that fails Probation. These measures cover firmware that was **confirmed and then crashes**: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. They are in **every build**, release and Debug alike; the Debug Console is what makes them convenient. The decision is [ADR 0005](/dev/decisions/0005-safe-mode-crash-reports-watchdog/).
|
||||
Rollback (see [How an update works](/dev/build/how-an-update-works/)) protects against **new** firmware that fails Probation. These measures cover firmware that was **confirmed and then crashes**: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. They are in every firmware; the Debug Console is what makes them convenient. The decision is [ADR 0005](/dev/decisions/0005-safe-mode-crash-reports-watchdog/).
|
||||
|
||||
## What a crash leaves behind
|
||||
|
||||
@@ -18,7 +18,7 @@ The `crash` command prints it again whenever you like:
|
||||
|
||||
```
|
||||
> crash
|
||||
crash: last one in v0.9.0-1-g4ab873e-dirty+debug (panic)
|
||||
crash: last one in v0.9.0-1-g4ab873e-dirty (panic)
|
||||
crash: task loopTask, pc 0x4037e179, cause 0, address 0x00000000
|
||||
crash: reason: abort() was called at PC 0x421209b3 on core 1
|
||||
crash: backtrace 0x4037e179 0x4037e141 0x4038582d 0x421209b3 ...
|
||||
@@ -40,7 +40,8 @@ scripts/rdbg.py coredump my.bin # ...to a file you name
|
||||
- **`crash`** runs the device's `crash`, finds the ELF by digest (or by version), and passes the backtrace to `scripts/decode_backtrace.sh`, which runs `addr2line` from the build container: one line for each frame, with function and file:line.
|
||||
- **`coredump`** fetches the raw dump over the console (it works when the main loop is stuck) and runs `scripts/decode_coredump.sh`: `esp-coredump` and GDB, giving **every task's backtrace, the registers, and the crashed task's stack**.
|
||||
- By hand: `scripts/decode_backtrace.sh <version|digest> <address>...`, and `scripts/decode_coredump.sh <core.bin> <version|digest>`. Both say which ELF they used.
|
||||
- If no archived ELF matches, they say so: the build was made on another machine, or `.pio/` was cleaned.
|
||||
- **A release's ELF is fetched for you.** When the crash names a released version (`v0.12.0`) and no local ELF matches, `rdbg.py` downloads `roro9stack-<version>.elf.gz` from the release on Gitea into `.pio/elves/`, so a crash on a firmware you did not build can be decoded. Decoding itself still runs in the build container.
|
||||
- If nothing matches, the scripts say so: an unreleased build made on another machine, or `.pio/` was cleaned.
|
||||
|
||||
## The main loop is watched
|
||||
|
||||
@@ -52,21 +53,19 @@ An installed update no longer depends on the main loop either: the Update Servic
|
||||
|
||||
The count of starts that follow a crash (a panic or the watchdog) is kept in NVS. After **three in a row**, the firmware starts **Safe Mode** instead of everything else:
|
||||
|
||||
- only the **clock, Wi-Fi, the Update Service and, in a Debug Build, the Debug Console** start: no Apps, no IRC, no SD card;
|
||||
- only the **clock, Wi-Fi, the Update Service and, if it is switched on, the Debug Console** start: no Apps, no IRC, no SD card;
|
||||
- the screen says so, with the **address to push an update to**;
|
||||
- only a few commands run (the [command reference](/dev/debug/commands/) lists them); anything else answers `not available in Safe Mode`.
|
||||
|
||||
Any **normal restart** (`reboot`, an update) or **a minute of uptime** resets the count. So Safe Mode is reached by crashing three times *quickly*, and a crash loop costs about half a minute before the device becomes reachable. To leave it: push a fix (`scripts/flash.sh --debug --ota`), or `reboot`.
|
||||
Any **normal restart** (`reboot`, an update) or **a minute of uptime** resets the count. So Safe Mode is reached by crashing three times *quickly*, and a crash loop costs about half a minute before the device becomes reachable. To leave it: push a fix (`scripts/flash.sh --ota`), or `reboot`. Over USB, `debug on` works in Safe Mode too, if the console was off.
|
||||
|
||||
**Its limit:** if Wi-Fi or the Update Service is what crashes, Safe Mode cannot help, and it takes USB.
|
||||
|
||||
## Crash on purpose
|
||||
|
||||
On a Debug Build:
|
||||
|
||||
```
|
||||
crash abort # abort(): a panic with a core dump
|
||||
crash wdt # hang the main loop until the task watchdog fires
|
||||
```
|
||||
|
||||
Use them to see a real report and decode it, to check that **the counter reaches Safe Mode**, and to prove that a Debug Build you are about to rely on will survive its own crashes. The device restarts by itself after either, and the report is there to read when the console comes back.
|
||||
Use them to see a real report and decode it, to check that **the counter reaches Safe Mode**, and to prove that a build you are about to rely on will survive its own crashes. The device restarts by itself after either, and the report is there to read when the console comes back.
|
||||
|
||||
@@ -1,56 +0,0 @@
|
||||
+++
|
||||
title = "Debug Builds"
|
||||
description = "What a Debug Build is, how to build and flash one, how its token works, and why you should keep one in the fallback slot."
|
||||
weight = 1
|
||||
[extra]
|
||||
tag = "Start here"
|
||||
+++
|
||||
|
||||
## What it is
|
||||
|
||||
A Debug Build is built from the same source as a release, with the `cardputer-adv-debug` environment (it `extends` `cardputer-adv` in `platformio.ini`) and `-DRORO_DEBUG`. Differences:
|
||||
|
||||
- the **Debug Console** on TCP **2323** (next page);
|
||||
- extra commands, only meant for testing: crash on purpose, fake an installed version, damage a download, inject a LoRa packet, fill a folder with files (see the [command reference](/dev/debug/commands/));
|
||||
- the version ends in **`+debug`** (`scripts/version.py`) wherever the version shows: Settings, `info`, the Update Service. A `+debug` version compares **equal** to its release counterpart, so going between the two is never refused as a downgrade.
|
||||
|
||||
**It is compiled out of release builds, not switched off by a setting.** A console that runs commands, presses keys and reboots the device is a remote control; in a release build nothing listens and the code is not there.
|
||||
|
||||
## Build and flash one
|
||||
|
||||
Everything runs in Docker (see [Build, test and release](/dev/build/build-and-test/)). Over USB:
|
||||
|
||||
```sh
|
||||
scripts/flash.sh --debug # builds cardputer-adv-debug, uploads, opens the serial monitor
|
||||
```
|
||||
|
||||
Once a Debug Build is on the device, every later one can go over Wi-Fi, with no cable:
|
||||
|
||||
```sh
|
||||
export RORO_OTA_HOST=10.39.39.12 # the address Settings > Firmware shows
|
||||
scripts/flash.sh --debug --ota # builds, signs and pushes; the device installs and restarts
|
||||
```
|
||||
|
||||
The device's address is on **Settings → Firmware** ("Push to", which also gives port 3232, the update port). The Debug Console is on the same address, port 2323. See [Flash and update](/dev/build/flash/) for the update side.
|
||||
|
||||
## The token
|
||||
|
||||
The console asks for a secret first. It is **128 random bits**, made the first time any build runs (`scripts/_docker.sh`), kept in `~/.config/roro9stack/debug-token` next to the update-signing key, and passed into the build container as `RORO_DEBUG_TOKEN`. `scripts/debug_flags.py` compiles it into the firmware, and **refuses to build a Debug Build without one**, rather than fall back on a default.
|
||||
|
||||
- It is **never committed.** Releases have no console, so nothing of it is published: CI builds a Debug Build on a pull request to prove it still compiles, but never publishes it, because each one carries its builder's token.
|
||||
- It guards against **the network**, not against someone who holds the device: anyone with USB access can flash anything anyway ([ADR 0003](/dev/decisions/0003-own-signature-check-not-secure-boot/)).
|
||||
- To change it, delete the file and build again; the new token goes into the next Debug Build you flash.
|
||||
|
||||
## Keep a Debug Build in the other slot
|
||||
|
||||
The device has two app slots, so an update never overwrites the running firmware. If a new firmware crashes during [Probation](/dev/build/how-an-update-works/), the device goes **back to the previous one**, whatever that is. As long as you develop on Debug Builds, **the firmware a crash falls back to has the console too**, so a bad update never costs you remote access. Pushing a *release* build over a Debug Build leaves the Debug Build in the other slot until the next update overwrites it.
|
||||
|
||||
So the habit is: develop on Debug Builds, release by tag, and think twice before pushing a release over the only Debug Build you have.
|
||||
|
||||
## Releases and Debug Builds
|
||||
|
||||
- A Debug Build shows the latest release in Settings → Firmware but **never installs it**: that would replace the console with a release that has none. Update a Debug Build from the PC with `scripts/flash.sh --debug --ota`.
|
||||
- A Debug Build still checks and lists releases, which is useful for testing the update path: `update pretend` makes a released version count as newer (see [Drive the UI](/dev/debug/drive-the-ui/)).
|
||||
- **Safe Mode** (after 3 crashes in a row) keeps the Debug Console running, so a crash loop is something you fix remotely: see [Crashes and Safe Mode](/dev/debug/crashes/).
|
||||
|
||||
The reasoning is in [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
|
||||
@@ -59,12 +59,12 @@ Each of these puts something in, **without** the outside world:
|
||||
| `gnss send <sentence>` | Sends an NMEA sentence **to** the GNSS receiver (the checksum is added), to configure it |
|
||||
| `gemini get <url>` | Fetches a page and reports header, size, certificate and heap use, without the App |
|
||||
| `gemini trust <host> <port> <sha256>` | Pins a certificate by hand |
|
||||
| `sd fill <folder> <count>` | **Debug Build.** Makes that many small files in a folder, to test a crowded one (the Storage App shows the first 256) |
|
||||
| `sd fill <folder> <count>` | Makes that many small files in a folder, to test a crowded one (the Storage App shows the first 256) |
|
||||
| `wifi add <ssid><TAB><password>` | Adds a Saved Network, so credentials stay out of the repository |
|
||||
|
||||
## Test the update path
|
||||
|
||||
An update that goes wrong is the case you most want to rehearse, and a Debug Build can make it go wrong **on purpose**:
|
||||
An update that goes wrong is the case you most want to rehearse, and the firmware can make it go wrong **on purpose**:
|
||||
|
||||
```
|
||||
update status # what the device runs, what failed here before, the daily check, heap
|
||||
@@ -72,15 +72,14 @@ update check | update list # look at the server: the latest release, or
|
||||
update pretend v0.9.0 # take the running version to be v0.9.0: the latest release now counts as "new"
|
||||
update damage cut 50000 # the next download is cut after 50000 bytes
|
||||
update damage flip 100000 # ...or has the byte at offset 100000 damaged
|
||||
update install v0.11.0 force # try the install
|
||||
update install v0.11.0 # try the install
|
||||
update probe git.twis.la # is that server's certificate accepted? (the two ISRG roots only)
|
||||
update daily # forget today's daily check: it runs again at the next tick
|
||||
update pretend off # back to the real version
|
||||
```
|
||||
|
||||
- **`force` is needed on a Debug Build.** A plain `update install <tag>` answers `Debug Build: update from the PC`, because installing a release would replace the console with a build that has none. `force` is accepted only on a Debug Build, and only from the console.
|
||||
- **A damaged download must be refused cleanly:** the Update Service checks the signature after the first 160 bytes and the image hash at the end, so a cut or a flipped byte must end in a refusal with **nothing switched**. Check `info` afterwards: both slots, and `update: confirmed`.
|
||||
- **An undamaged `force` install really installs** the release, into the other slot. The Debug Build stays where it was until the next update overwrites it, and a Rollback returns to it, but think before you do it.
|
||||
- **An undamaged install really installs** the release, into the other slot, and the device restarts into it. Your build stays in the slot it was in until the next update overwrites it, and the console's setting and token are untouched: the release has the console too.
|
||||
- The server is read by the device itself, with Wi-Fi up; the download is one TLS connection, about 52 KB of heap at its peak, so IRC steps aside (see [the memory limit](/howto/not-enough-memory/)). `status: heap` in the stream shows it happen.
|
||||
|
||||
For crashes during Probation, see [Crashes and Safe Mode](/dev/debug/crashes/).
|
||||
@@ -94,7 +93,7 @@ wifi ip MyNet 10.39.39.50/24 10.39.39.1 try 60 # use this address for 60 s..
|
||||
wifi ip keep # ...and keep it, if you could still reach the device
|
||||
```
|
||||
|
||||
If you do not send `wifi ip keep` in time, the device goes back to the **previous** setting by itself, and the console comes back with it. (A Debug Build command: `try` is not in release builds.)
|
||||
If you do not send `wifi ip keep` in time, the device goes back to the **previous** setting by itself, and the console comes back with it.
|
||||
|
||||
## Measure
|
||||
|
||||
@@ -104,7 +103,7 @@ tasks # each task over the next second: state, priority, least free stack,
|
||||
net # bytes each network service has read and written since start
|
||||
```
|
||||
|
||||
`tasks` takes a second to answer. A low number in the `stack` column is a risk (the System App shows it in the warning colour under 512 bytes). `loop spin on|off` (Debug Build) makes the main loop spin without resting, to compare load and radio noise. And the `status: heap` line every 10 seconds in the stream is the cheapest memory trace there is: watch it while you do the thing you suspect.
|
||||
`tasks` takes a second to answer. A low number in the `stack` column is a risk (the System App shows it in the warning colour under 512 bytes). `loop spin on|off` makes the main loop spin without resting, to compare load and radio noise. And the `status: heap` line every 10 seconds in the stream is the cheapest memory trace there is: watch it while you do the thing you suspect.
|
||||
|
||||
```
|
||||
task st pri stack cpu% core
|
||||
@@ -120,4 +119,4 @@ The [System App](/guide/system/) shows the same, live, on the device.
|
||||
|
||||
## Radio experiments
|
||||
|
||||
`lora preset <name>` and `lora custom <MHz> <BW kHz> <SF> <CR> <sync hex> [preamble]` change what the receiver listens to (receive only: the radio never transmits), `lora sweep on [from] [to] [step]` takes a survey, and `lora noise test [gnss|quiet]` (Debug Build) runs a Sweep under one changed condition at a time, with Wi-Fi off for a moment, to find what raises the noise floor. `lora probe` finds the radio and reports its chip, oscillator, antenna switch, interrupt line and noise floor. Details in the [command reference](/dev/debug/commands/).
|
||||
`lora preset <name>` and `lora custom <MHz> <BW kHz> <SF> <CR> <sync hex> [preamble]` change what the receiver listens to (receive only: the radio never transmits), `lora sweep on [from] [to] [step]` takes a survey, and `lora noise test [gnss|quiet]` runs a Sweep under one changed condition at a time, with Wi-Fi off for a moment, to find what raises the noise floor. `lora probe` finds the radio and reports its chip, oscillator, antenna switch, interrupt line and noise floor. Details in the [command reference](/dev/debug/commands/).
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
+++
|
||||
title = "Switch the console on"
|
||||
description = "The Debug Console is in every firmware, and off. How to switch it on, where its token comes from, and how to set a device up without typing anything."
|
||||
weight = 1
|
||||
[extra]
|
||||
tag = "Start here"
|
||||
+++
|
||||
|
||||
## One firmware
|
||||
|
||||
There is no special build. **Every roro9stack firmware has the Debug Console**, the same one, and the commands made for testing (crash on purpose, fake an installed version, damage a download, inject a LoRa packet). It is **off** until its owner switches it on.
|
||||
|
||||
**Off means nothing is there:** no socket listens, the console's task does not exist, and neither does its 4 KB buffer. A device that never uses it pays 30 KB of flash and 88 bytes of memory.
|
||||
|
||||
Before version 0.12 this was a separate *Debug Build* with a token compiled in from the builder's machine, which is why it could not be published. [ADR 0010](/dev/decisions/0010-debug-console-in-every-build/) says why that changed.
|
||||
|
||||
## On the device
|
||||
|
||||
**Settings → Debug Console:**
|
||||
|
||||
| Row | Does |
|
||||
|---|---|
|
||||
| **Debug Console** | The switch. Switching it **on** asks first, and makes a token if there is none |
|
||||
| **Connect to** | The address and port: `10.39.39.12:2323` |
|
||||
| **New token** | Makes another one. The old one stops working, and whoever is connected is cut off |
|
||||
| **Type a token** | One of your own, of 16 to 64 characters |
|
||||
|
||||
Under the rows, the **token**, in large type on two lines, in groups of four: `K7QF-3M2X-9WBD-HT4P-6RNC`. This page is the only place it is ever shown. The Status Bar shows **`DBG`** while the console listens, and brighter while someone is connected.
|
||||
|
||||
The setting **stays** across restarts and updates, and in Safe Mode.
|
||||
|
||||
## The token
|
||||
|
||||
- **The device makes it,** from its hardware random generator, the first time the console is switched on: 100 bits, written as 20 characters without the letters that get misread (no I, L, O or U).
|
||||
- **Dashes and case do not count,** and an `O`, `I` or `L` is taken for the `0` or `1` it was. Type it as you read it.
|
||||
- **One you type** must have at least 16 characters. A short one would be the weakest part of the whole thing.
|
||||
- It is **never printed** on a console, and it **never crosses the network** ([the Debug Console](/dev/debug/console/) says how).
|
||||
- It guards against **the network**, not against someone who holds the device: anyone with USB access can flash anything anyway ([ADR 0003](/dev/decisions/0003-own-signature-check-not-secure-boot/)).
|
||||
|
||||
## Give `rdbg.py` the token
|
||||
|
||||
`scripts/rdbg.py` looks for it in this order:
|
||||
|
||||
```sh
|
||||
scripts/rdbg.py -t K7QF-3M2X-9WBD-HT4P-6RNC info # 1. on the command line (also --token)
|
||||
RORO_DEBUG_TOKEN=K7QF-3M2X-9WBD-HT4P-6RNC scripts/rdbg.py info # 2. in the environment
|
||||
echo K7QF-3M2X-9WBD-HT4P-6RNC > ~/.config/roro9stack/debug-token # 3. in a file, for the device you use every day
|
||||
```
|
||||
|
||||
## Without typing: over USB
|
||||
|
||||
With the device on a cable, the console can be set up from the PC:
|
||||
|
||||
```sh
|
||||
scripts/flash.sh --debug # flashes over USB, then switches the console on and gives it your token
|
||||
```
|
||||
|
||||
It sends two commands over the serial port, which you can also type there yourself:
|
||||
|
||||
```
|
||||
debug on # switch it on (a token is made if there is none)
|
||||
debug token <value> # give it this token: 16 to 64 characters
|
||||
debug token new # make a new one
|
||||
debug status # on or off, token set or not, a client or not (the token itself is never shown)
|
||||
debug off # switch it off
|
||||
```
|
||||
|
||||
`debug on` and `debug token` work **over USB serial only**: the console cannot be used to open itself wider. `debug status` and `debug off` work from anywhere, and a screenshot taken over the console while this page is open shows the token, to someone who already had it. Your token file is made by the first build (`scripts/_docker.sh`), 32 hex digits, and is never committed.
|
||||
|
||||
Since the setting survives updates, this is needed **once for a device**, not at each flash. Later builds go over Wi-Fi with `scripts/flash.sh --ota`.
|
||||
|
||||
## Should it be on?
|
||||
|
||||
On your own network, on a device you are working on: yes, that is what it is for. Remember what it gives to whoever has the token **and** is on the same network: the console, the keys, the files on the SD card, a restart. It does not give them the firmware: an update still has to be signed.
|
||||
|
||||
On a network you share with strangers, switch it off, or at least know that what the console prints is not encrypted. The token is safe there; the conversation is not.
|
||||
@@ -1,6 +1,6 @@
|
||||
+++
|
||||
title = "A Debug Console over Wi-Fi, in Debug Builds only"
|
||||
description = "The goal of Firmware Updates is to manage the device without a cable, and that includes finding out what went wrong. So a Debug Build (cardputer-adv-debug, -DRORO_DEBUG, version suffix +debug) adds a Debug Console on TCP 2323…"
|
||||
description = "Superseded in part by ADR 0010 (2026-10-06): there is no Debug Build any more. The console is in every firmware, off until switched on, and its token belongs to the device, not to the build. What this record says about how the…"
|
||||
weight = 4
|
||||
|
||||
[extra]
|
||||
@@ -8,6 +8,8 @@ docs = true
|
||||
source = "docs/adr/0004-debug-console-in-debug-builds.md"
|
||||
tag = "ADR 0004"
|
||||
+++
|
||||
**Superseded in part by [ADR 0010](/dev/decisions/0010-debug-console-in-every-build/) (2026-10-06):** there is no Debug Build any more. The console is in every firmware, off until switched on, and its token belongs to the device, not to the build. What this record says about how the console works (the ring, commands on the main loop, binary commands on its own task, one client) still holds.
|
||||
|
||||
The goal of Firmware Updates is to manage the device without a cable, and that includes finding out what went wrong. So a **Debug Build** (`cardputer-adv-debug`, `-DRORO_DEBUG`, version suffix `+debug`) adds a **Debug Console** on TCP 2323: the serial console, over Wi-Fi. A client sends a token as its first line, then gets the last 4 KB of console output (boot messages included), every new line live, and runs the same commands as the serial port, plus a few that only make sense remotely. ESP-IDF's own log lines are teed into it.
|
||||
|
||||
It's compiled out of release builds entirely, rather than switched off by a setting. A console that runs commands is a remote control: in a release build, nothing listens.
|
||||
|
||||
@@ -8,10 +8,10 @@ docs = true
|
||||
source = "docs/adr/0005-safe-mode-crash-reports-watchdog.md"
|
||||
tag = "ADR 0005"
|
||||
+++
|
||||
Rollback protects against new firmware that fails Probation. It does nothing for firmware that was confirmed and crashes later: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. Without a cable, such a device would restart forever. Three measures, in release and Debug Builds alike, keep it reachable:
|
||||
Rollback protects against new firmware that fails Probation. It does nothing for firmware that was confirmed and crashes later: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. Without a cable, such a device would restart forever. Three measures, in every build, keep it reachable:
|
||||
|
||||
- **Safe Mode.** The firmware counts starts that follow a crash (panic or watchdog) in NVS, first thing at boot. After 3 in a row, it starts only the clock, Wi-Fi, the Update Service and, in a Debug Build, the Debug Console: no Apps, no IRC, no SD card, and a screen that says so with the address to push an update to. Any normal restart (a `reboot`, an update), or a minute of uptime, resets the count.
|
||||
- **Crash reports.** The same boot record keeps which version was running, so after a crash the firmware knows which one crashed, even when a Rollback has switched slots since. ESP-IDF already writes a core dump to its flash partition on a panic; after the restart the firmware prints its summary (task, PC, reason, backtrace) and raises a Notification. The `crash` command shows it again later. In a Debug Build, `scripts/rdbg.py crash` decodes the backtrace and `scripts/rdbg.py coredump` fetches the whole dump for `esp-coredump`, against the ELF of that exact build (`.pio/elves/`, named by version and ELF digest).
|
||||
- **Safe Mode.** The firmware counts starts that follow a crash (panic or watchdog) in NVS, first thing at boot. After 3 in a row, it starts only the clock, Wi-Fi, the Update Service and, if it's switched on, the Debug Console: no Apps, no IRC, no SD card, and a screen that says so with the address to push an update to. Any normal restart (a `reboot`, an update), or a minute of uptime, resets the count.
|
||||
- **Crash reports.** The same boot record keeps which version was running, so after a crash the firmware knows which one crashed, even when a Rollback has switched slots since. ESP-IDF already writes a core dump to its flash partition on a panic; after the restart the firmware prints its summary (task, PC, reason, backtrace) and raises a Notification. The `crash` command shows it again later. With the Debug Console, `scripts/rdbg.py crash` decodes the backtrace and `scripts/rdbg.py coredump` fetches the whole dump for `esp-coredump`, against the ELF of that exact build (`.pio/elves/`, named by version and ELF digest).
|
||||
- **The main loop is watched.** Arduino-ESP32 subscribes only core 0's idle task to the task watchdog, and the main loop runs on core 1: a stuck loop used to hang the device for good, with the screen frozen and the Debug Console unable to run commands. `enableLoopWDT()` makes a loop stuck for 5 s a panic, with a core dump, counted towards Safe Mode. And an installed update no longer depends on the main loop: the Update Service restarts into it by itself after 90 s.
|
||||
|
||||
## Consequences
|
||||
|
||||
@@ -0,0 +1,38 @@
|
||||
+++
|
||||
title = "The Debug Console is in every build, off until its owner switches it on"
|
||||
description = "There is one firmware. The Debug Console of ADR 0004 is compiled into it, and so are the commands that used to exist only in Debug Builds. It listens only while Settings → Debug Console is on, which is not the default, and it…"
|
||||
weight = 10
|
||||
|
||||
[extra]
|
||||
docs = true
|
||||
source = "docs/adr/0010-debug-console-in-every-build.md"
|
||||
tag = "ADR 0010"
|
||||
+++
|
||||
There is **one firmware**. The Debug Console of ADR 0004 is compiled into it, and so are the commands that used to exist only in Debug Builds. It listens only while **Settings → Debug Console** is on, which is not the default, and it lets in whoever proves they hold **the device's own token**. The `cardputer-adv-debug` environment, `RORO_DEBUG`, the `+debug` version and the token compiled in from the builder's machine are gone.
|
||||
|
||||
ADR 0004 kept the console out of release builds with one argument: a console that runs commands is a remote control, so in a release nothing should listen. That argument was never whole: the Update Service has listened on TCP 3232 in every build since Firmware Updates exist, guarded by a signature. A listener that is off by default and guarded by a secret the device made itself is the same kind of trade, and what the split cost had grown:
|
||||
|
||||
- **What was tested wasn't what shipped.** Development ran on Debug Builds; releases were a different binary, built to be published and never run by the person who wrote them.
|
||||
- **Debug Builds couldn't be published**, since each carried its builder's token. So the console, `rdbg.py`, screenshots and crash decoding were for whoever built the firmware, and nobody else.
|
||||
- **Rules that existed only to protect the console:** a Debug Build never installed a release (it would have lost the console), `update install … force` to do it anyway, "keep a Debug Build in the fallback slot", versions with `+debug` that had to compare equal to their release.
|
||||
- **CI built two firmwares** on every pull request and every tag.
|
||||
|
||||
## How it works
|
||||
|
||||
- **Off means nothing is there.** With the setting off, no socket listens, the console's task doesn't exist and neither does its 4 KB ring: a device that never uses the console pays for it in flash (30 KB more than the release build was) and 88 bytes of static RAM, measured. A missing or invalid stored setting counts as off.
|
||||
- **The token is the device's.** The first time the console is switched on, the device makes one from its hardware random generator: 100 bits, as 20 characters of Crockford's base32 (`K7QF-3M2X-9WBD-HT4P-6RNC`), shown on the Debug Console page and nowhere else. It can be replaced by one typed by hand, of 16 to 64 characters. It is stored with the other settings and is never printed on a console.
|
||||
- **The token never crosses the network.** The device sends 16 random bytes; the client answers with their HMAC-SHA256 keyed by the token. Someone on the same Wi-Fi who records a login has nothing they can use for the next one.
|
||||
- **Five wrong answers in a row** close the console to everyone for a minute, with a Notification naming the address they came from. A wrong answer still costs a second.
|
||||
- **`DBG` in the Status Bar** while the console listens, bright while a client is connected.
|
||||
- **USB serial can set it up:** `debug on`, `debug token <value>`, `debug token new`. `scripts/flash.sh --debug` uses them to give a freshly flashed device the developer's token. Whoever holds the cable can flash anything anyway (ADR 0003); the console itself can only switch itself off.
|
||||
- **Safe Mode** starts the console if it's switched on: the settings are read before Safe Mode is decided.
|
||||
|
||||
## Consequences
|
||||
|
||||
- **"Nothing listens" now rests on one stored setting**, not on absent code. That setting is typed, validated and host-tested, and its default is off; a bug that flipped it would have to come from code that already runs on the device.
|
||||
- **Whoever has the token and the network has the device:** the console, its keys, the SD card's files, a restart. Not its firmware: an update still has to be signed (ADR 0003).
|
||||
- **The stream is still plain text.** The token is safe from an eavesdropper; what the console prints, and the files it carries, are not. TLS would cost about 52 KB of the 107 KB there is, next to IRC's own connection: not for a console.
|
||||
- **An old `rdbg.py` can't talk to a new firmware**, and the reverse: the first line of the protocol changed. Accepted, with one user.
|
||||
- **A device whose settings are damaged has no console in Safe Mode**, only the signed update push. Before, a Debug Build always had it.
|
||||
- **The commands that crash, damage a download or fill a folder on purpose** are in every build. They need the cable or the token, like everything else.
|
||||
- **Releases already publish their ELF**, so `rdbg.py crash` fetches it when it isn't in `.pio/elves/`: crash reports can be decoded without having built the firmware.
|
||||
@@ -22,7 +22,7 @@ Until now the tests, the builds, the signing and the flashing all happened on on
|
||||
|---|---|
|
||||
| Q151 | A push to `main`: the host tests (with their coverage). A push to another branch: nothing, its pull request is what runs (a branch with an open pull request ran twice, once for each). A pull request: the same and both builds; changes reach `main` through pull requests. A tag `v*`: all of it, then a release. (First: everything on every push, which rebuilt the firmware far more often than anyone looked at it.) |
|
||||
| Q152 | **CI signs.** The signing key is the repository secret `OTA_SIGNING_KEY`; a tag push makes a complete, signed release with no manual step (ADR 0008). |
|
||||
| Q153 | The Debug Build is built in CI with a token of the runner's own, to prove it compiles, and **isn't published**: it would hand everyone its Debug Console token. |
|
||||
| Q153 | *Revised by Q188: there is one firmware.* The Debug Build is built in CI with a token of the runner's own, to prove it compiles, and **isn't published**: it would hand everyone its Debug Console token. |
|
||||
| Q154 | Pull requests from forks don't start a run. |
|
||||
| Q155 | A release carries `roro9stack-<version>.ota` (signed), `-factory.bin` for USB, `.elf.gz` to decode crashes, and `SHA256SUMS`. |
|
||||
| Q156 | Its text is the tag's message, what the files are, and the commits since the tag before. |
|
||||
@@ -72,7 +72,7 @@ The device looks at the project's Gitea for a newer release, says so, and instal
|
||||
| Q168 | A version that failed (rolled back) isn't announced again by the background check until a newer one exists; it can still be installed by hand. |
|
||||
| Q169 | **Older releases:** a list of the last ten, newest first, the running one marked. Installing an older one asks with a stronger warning. |
|
||||
| Q170 | Enter on a release shows its version, date, size and the tag message, with Install. |
|
||||
| Q171 | **Debug Builds** check and show the latest release, but don't install it: it would replace the Debug Build and its console (Q153: Debug Builds aren't published). Their updates come from the PC. |
|
||||
| Q171 | *Revised by Q188: there is no Debug Build, every firmware installs releases.* **Debug Builds** check and show the latest release, but don't install it: it would replace the Debug Build and its console (Q153: Debug Builds aren't published). Their updates come from the PC. |
|
||||
| Q172 | **Memory, as measured:** a TLS connection peaks at about 52 KB of heap, with or without checking the certificate, so it starts with 80 KB free (Q86's 20 KB spare on top), not 55 KB. **A check, list or install someone asked for makes IRC step aside** and come back after; the daily check never does, and with IRC connected it waits. (First: 55 KB and nothing else. With IRC connected a check left 3 KB and a download 836 bytes.) |
|
||||
| Q173 | Left out, each with its issue: installing automatically (#52), a release channel (#53), resuming a download (#54). |
|
||||
| Q174 | Ships as **v0.11.0**. Tested on the device with the real signed releases; a Debug Build command pretends the device runs an older version, so v0.10.0 counts as an update. |
|
||||
@@ -131,3 +131,47 @@ The device looks at the project's Gitea for a newer release, says so, and instal
|
||||
- **A key press during the hold:** Back on the Firmware page while a check is going doesn't cancel it.
|
||||
|
||||
**Two slips during the checks:** a blind sequence of keys on the Firmware page opened the SD card's install dialog (the page keeps its selection between visits); it was cancelled with Back, nothing installed. And my port-polling while waiting for a restart took the Debug Console's only client slot, which made the first install attempt look like a failure.
|
||||
|
||||
## One firmware: the Debug Console in every build (issue #68)
|
||||
|
||||
The issue asked for a token that could be set, so that Debug Builds could be published. The design round ended somewhere simpler: no Debug Builds. The console is in every firmware, off until switched on, with a token that belongs to the device (ADR 0010, which supersedes part of ADR 0004).
|
||||
|
||||
### Decisions (design round 2026-10-06)
|
||||
|
||||
| # | Decision |
|
||||
|---|---|
|
||||
| Q188 | **One firmware,** with the Debug Console and the test commands compiled in. The `cardputer-adv-debug` environment, `RORO_DEBUG`, the `+debug` version and `scripts/debug_flags.py` go; CI builds one firmware. Revises Q153 and Q171. |
|
||||
| Q189 | **Off by default,** and off when the stored setting is missing or invalid. Off, nothing listens and no memory is used: the task and the 4 KB ring exist only while it's on. |
|
||||
| Q190 | **Settings → Debug Console:** the switch, where to connect, the token, "New token", "Type a token". It stays on across restarts and in Safe Mode. **`DBG` in the Status Bar** while it listens, bright with a client connected. |
|
||||
| Q191 | **The device makes the token** the first time the console is switched on: 100 bits as 20 characters of Crockford's base32, shown in groups of four. It can be replaced by one typed by hand, of at least 16 characters. Case and dashes don't count. |
|
||||
| Q192 | **USB serial can set it up:** `debug on`, `debug token <value>`, `debug token new`, over USB only; `debug status` and `debug off` from anywhere. `scripts/flash.sh --debug` gives a device the developer's token. The token is never printed. |
|
||||
| Q193 | **Challenge and answer:** the device sends 16 random bytes, the client their HMAC-SHA256 keyed by the token. Five wrong answers in a row close the console for a minute, with a Notification. **Old clients and old firmwares don't talk to each other:** accepted, this far from v1 and with one user. |
|
||||
| Q194 | `rdbg.py` takes the token from **`-t`/`--token`**, then **`$RORO_DEBUG_TOKEN`**, then `~/.config/roro9stack/debug-token`. |
|
||||
| Q195 | **The test commands are in every build** (`crash abort`, `crash wdt`, `update pretend`, `update damage`, `lora inject`, `sd fill`, `loop spin`, `wifi ip … try`). `update install … force` goes: there's nothing left to protect. |
|
||||
|
||||
### As built
|
||||
|
||||
- **`lib/debug`** (host-tested, 9 tests): the token maker, tidying what a person types, HMAC-SHA256 on the project's own SHA-256 (checked against RFC 4231), the answer to a challenge (the same vector as `rdbg.py`'s), a comparison that takes the same time whatever it's given, and the pause after five wrong answers, right across the clock's wrap.
|
||||
- **Two settings,** `DebugConsole` and `DebugToken`, with the others: validated, and invalid stored values count as missing.
|
||||
- **The console's task and ring come and go with the setting.** `Console::openRing` allocates the ring; switching off closes the socket, frees it and ends the task. `tick` follows the settings once a second, so a switch off and on again takes a second or two.
|
||||
- **Measured against the two builds it replaces:** 1,881,799 bytes of flash and 54,700 of static RAM. The release build was 1,851,387 and 54,612 (so 30 KB more flash and 88 bytes more RAM); the Debug Build was 1,874,887 and 58,780 (4 KB of RAM back: the ring is no longer a static array).
|
||||
- **`scripts/rdbg.py`** answers the challenge, and fetches a release's ELF from Gitea when a crash names a version that isn't in `.pio/elves/`.
|
||||
|
||||
### Checks
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Host tests | 468 pass (456 before) |
|
||||
| The firmware builds | One environment, 1,881,799 bytes of flash used |
|
||||
| Pushed over Wi-Fi to a device running a Debug Build | Installed, restarted; the update port answers and **port 2323 refuses connections**: off by default |
|
||||
| Switched on at the device (Settings → Debug Console) | A token is made and shown. Its first drawing, in bold at normal size, was misread once: it is now at twice the size, in two lines, and `O`, `I` and `L` are taken for `0` and `1` |
|
||||
| A login with `scripts/rdbg.py` | The challenge is answered; `info`, `screenshot` and the backlog work |
|
||||
| `DBG` in the Status Bar | There while the console is on, bright while a client is connected (screenshots) |
|
||||
| `debug on` and `debug token` over the console | Refused: `over USB serial only` |
|
||||
| Six logins with a wrong token | Five are refused, a second apart; the sixth, and the right token after it, get `locked`. After a minute the right token works again. The console's own log and a Notification say so |
|
||||
| Three `crash abort` in a row | **Safe Mode**, with the console reachable: `info` works, `ls /` says `not available in Safe Mode`. `rdbg.py crash` decodes the backtrace to `runCommand` at the `abort()` line. `reboot` leaves Safe Mode |
|
||||
| An update over Wi-Fi with the console on | The setting and the token survive: the console is back by itself after the restart |
|
||||
| `debug off` over the console | The client is dropped and port 2323 refuses connections; the update port still answers |
|
||||
| Memory, console on with a client | 108 KB free (the Debug Build it replaces: 104 KB). In Safe Mode: 178 KB free |
|
||||
|
||||
**Not checked:** `scripts/flash.sh --debug` over USB (no device on USB here); "New token" and "Type a token" from the page (the code paths are the ones `debug token` uses, which need the cable); the Toast itself on screen (the console is closed while it shows; its Notification is in the log); free memory with the console off, which only the serial port could say (the static figures above are the evidence); fetching a release's ELF, which needs a crash on a released version.
|
||||
|
||||
@@ -38,6 +38,7 @@ A strip at the top of every screen: the name of the App on the left, and on the
|
||||
| `97%` | The battery (in the warning colour at 15% and under) |
|
||||
| `SD` | A card is in; the warning colour at 80% full |
|
||||
| bars and `W` | Wi-Fi connected, with its signal; `W?` is searching; `MON` is the Wi-Fi radio in its monitoring mode, which pauses IRC |
|
||||
| `DBG` | The Debug Console is switched on (Settings → Debug Console); brighter while a PC is connected to it |
|
||||
| `REC` | A GNSS Track is being recorded |
|
||||
| `CAP` | A LoRa capture is being recorded |
|
||||
| `L` | The radio is listening; it lights up for a moment on each packet. `SW` while a Sweep runs |
|
||||
|
||||
@@ -22,6 +22,7 @@ Move with the arrows. On a toggle, a choice or a slider, left and right change t
|
||||
| **Wi-Fi** | The page below |
|
||||
| **Check for updates** | Once a day, see [Updates](/guide/updates/) |
|
||||
| **Firmware** | The page described in [Updates](/guide/updates/) |
|
||||
| **Debug Console** | Off unless you switch it on. It lets a PC on the same network read the device's console and drive it, with a token shown on this page: see [the developer docs](/dev/debug/switch-it-on/). Leave it off if that means nothing to you |
|
||||
| **About** | The firmware version, the node number, battery, memory, uptime, clock and licence |
|
||||
|
||||
## Wi-Fi
|
||||
|
||||
@@ -41,4 +41,4 @@ The connection to the project's server is checked against the two root certifica
|
||||
|
||||
## For developers
|
||||
|
||||
Updates can also be pushed from a PC over Wi-Fi, with `scripts/flash.sh --ota <ip>`, or put on the card with `scripts/sd_put.sh`: see the [README](https://git.twis.la/twisla/roro9stack#firmware-updates-over-wi-fi-ota). **Debug Builds** show the latest release but do not install it, because a release has no debug console: update a Debug Build from the PC.
|
||||
Updates can also be pushed from a PC over Wi-Fi, with `scripts/flash.sh --ota <ip>`, or put on the card with `scripts/sd_put.sh`: see the [README](https://git.twis.la/twisla/roro9stack#firmware-updates-over-wi-fi-ota). The firmware's console can be reached over Wi-Fi too: see [The Debug Console](/dev/debug/).
|
||||
|
||||
@@ -11,7 +11,7 @@
|
||||
</header>
|
||||
|
||||
<div class="prose">
|
||||
<p>Each release has a <strong>signed update file</strong> (<code>.ota</code>: copy it to <code>/updates</code> on the SD card and install it from Settings → Firmware or the Storage App, or push it over Wi-Fi), a <strong>factory image</strong> (<code>-factory.bin</code>, the whole flash at offset 0 for a first install over USB), the <strong>symbols</strong> (<code>.elf.gz</code>, to decode a crash report) and <code>SHA256SUMS</code>. Debug Builds aren't published.</p>
|
||||
<p>Each release has a <strong>signed update file</strong> (<code>.ota</code>: copy it to <code>/updates</code> on the SD card and install it from Settings → Firmware or the Storage App, or push it over Wi-Fi), a <strong>factory image</strong> (<code>-factory.bin</code>, the whole flash at offset 0 for a first install over USB), the <strong>symbols</strong> (<code>.elf.gz</code>, to decode a crash report) and <code>SHA256SUMS</code>.</p>
|
||||
</div>
|
||||
|
||||
{% if releases %}
|
||||
|
||||
@@ -164,15 +164,16 @@ def build():
|
||||
+ relink("\n".join(readme[k] for k in ("Flash", "Firmware Updates over Wi-Fi (OTA)")), "README.md"))
|
||||
|
||||
common, debug = help_text()
|
||||
if debug:
|
||||
sys.exit("gen_dev_docs: kHelp has a RORO_DEBUG part again: there is one firmware (ADR 0010)")
|
||||
exact, prefix = safe_mode_commands()
|
||||
safe = ", ".join(f"`{c}`" for c in exact) + ", and anything starting with " + ", ".join(f"`{p.strip()}`" for p in prefix)
|
||||
dev = readme["Development aids"].split("### Debug Builds and the Debug Console")[0]
|
||||
dev = readme["Development aids"].split("### The Debug Console")[0]
|
||||
dev = re.sub(r"^## Development aids\n", "", dev).strip() + "\n"
|
||||
pages["debug/commands.md"] = (
|
||||
front("Command reference", "Every command the firmware understands, over USB serial or the Debug Console: what `help` prints, then what each one does.", 30, "src/main.cpp and README.md", tag="Reference")
|
||||
+ "## What `help` prints\n\n"
|
||||
+ "The firmware's own list, read from `src/main.cpp`. These work in every build, over USB serial:\n\n```\n" + common + "\n```\n\n"
|
||||
+ "A **Debug Build** adds these (the last ones, marked *Debug Console only*, exist only over Wi-Fi, where [`rdbg.py`](/dev/debug/files-and-screens/) speaks them):\n\n```\n" + debug + "\n```\n\n"
|
||||
+ "The firmware's own list, read from `src/main.cpp`. Every firmware has all of them, over USB serial and over the Debug Console. The ones marked *Debug Console only* exist only over Wi-Fi, where [`rdbg.py`](/dev/debug/files-and-screens/) speaks them, and the ones marked *USB serial only* only over the cable:\n\n```\n" + common + "\n```\n\n"
|
||||
+ f"In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few run: {safe}. Anything else answers `not available in Safe Mode`.\n\n"
|
||||
+ "## What they do\n\n" + dev)
|
||||
return pages
|
||||
|
||||
Reference in New Issue
Block a user