twisla/roro9stack
Public Access
1
0
Fork 0
Code Issues 48 Pull Requests Actions Packages Releases 16 Activity

One firmware: the Debug Console in every build, off until switched on, with the device's own token
CI / build (pull_request) Successful in 7m20s
Details
Site / build (pull_request) Successful in 9s
Details

Browse Source 
There is no Debug Build any more (ADR 0010, issue #68, Q188 to Q195). The
console and the test commands are compiled into every firmware. It listens
only while Settings > Debug Console is on, which isn't the default; off,
neither its task nor its 4 KB ring exists. The token is made by the device
and shown on that page; a client proves it knows it by answering a challenge
with an HMAC, so it never crosses the network, and five wrong answers close
the console for a minute. DBG in the Status Bar while it listens.

Over USB serial only: debug on, debug token <value>, debug token new.
scripts/flash.sh --debug uses them to set a device up with the developer's
token. scripts/rdbg.py takes the token from -t, $RORO_DEBUG_TOKEN or the
file, answers the challenge, and fetches a release's ELF to decode a crash.

Gone: the cardputer-adv-debug environment, RORO_DEBUG, the +debug version,
scripts/debug_flags.py, update install ... force, and the rule that a Debug
Build doesn't install releases. Old clients and old firmwares don't talk to
each other.

Against the builds it replaces: 30 KB more flash and 88 bytes more static
RAM than the release, 4 KB less RAM than the Debug Build. 468 host tests.
Checked on the device: off by default, login, the pause after wrong tokens,
Safe Mode with the console, the setting surviving an update, debug off.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT
This commit is contained in:
twisla
2026-10-06 22:59:35 +02:00
co-authored by Claude Opus 5.5
parent 1353e6a5f9
commit c68741cc46
65 changed files with 1245 additions and 374 deletions
Download Patch File Download Diff File Expand all files Collapse all files
site/content/dev/debug/_index.md
+4 -4
View File
@@ -1,5 +1,5 @@
+++
title = "Debug Builds and the Debug Console"
title = "The Debug Console"
description = "A firmware you can drive from your desk: its console over Wi-Fi, its screen as a PNG, its SD card, its crash dumps, and a way back when an update goes wrong."
template = "guide-index.html"
page_template = "guide-page.html"
@@ -10,13 +10,13 @@ weight = 1
eyebrow = "Developer docs"
+++
A **Debug Build** is the same firmware plus a **Debug Console**: the serial console, over Wi-Fi, behind a token. It is the most useful thing in the project. With it you can:
The **Debug Console** is the serial console, over Wi-Fi, for whoever holds the device's token. It is in **every firmware**, switched off until you switch it on, and it is the most useful thing in the project. With it you can:
- **see everything the device prints**, boot messages included, without a cable;
- **run every serial command** from your desk;
- **press keys** and **take screenshots**, so a UI change can be tested and looked at remotely;
- **copy files** to and from the SD card;
- **read crash reports and fetch core dumps**, decoded against the exact build that crashed;
- **update the firmware** over the same Wi-Fi, and know that a bad update rolls back to a build that still has the console.
- **update the firmware** over the same Wi-Fi, and know that a bad update rolls back by itself.
Start with [Debug Builds](/dev/debug/debug-builds/) to put one on a device, then [the Debug Console](/dev/debug/console/). The rest are what you can do with it.
Start with [Switch the console on](/dev/debug/switch-it-on/), then [the Debug Console](/dev/debug/console/). The rest are what you can do with it.
site/content/dev/debug/commands.md
+14 -15
View File
@@ -10,7 +10,7 @@ tag = "Reference"
+++
## What `help` prints
The firmware's own list, read from `src/main.cpp`. These work in every build, over USB serial:
The firmware's own list, read from `src/main.cpp`. Every firmware has all of them, over USB serial and over the Debug Console. The ones marked *Debug Console only* exist only over Wi-Fi, where [`rdbg.py`](/dev/debug/files-and-screens/) speaks them, and the ones marked *USB serial only* only over the cable:
```
info firmware, uptime, memory, Wi-Fi, app slots
@@ -37,11 +37,8 @@ irc start | irc stop | irc dump | irc say <buffer> <text>
install <path.ota> Update from SD
update check | list | status | install <tag> the project's releases on Gitea
sd card | sd list | cat <path> | log <text> | burst | sound on|off | short | normal
```
A **Debug Build** adds these (the last ones, marked *Debug Console only*, exist only over Wi-Fi, where [`rdbg.py`](/dev/debug/files-and-screens/) speaks them):
```
debug status | debug off the Debug Console over Wi-Fi (Settings > Debug Console)
debug on | debug token <16 to 64 characters> | debug token new (USB serial only) switch it on, set its token
crash abort|wdt crash on purpose (to test crash reports and Safe Mode)
wifi ip ... try <seconds> | wifi ip keep a trial IP setting: back to the previous one unless kept
loop spin on|off make the main loop spin without resting, to compare load and radio noise
@@ -54,7 +51,7 @@ get <path> | put <path> <size> <sha256> | screenshot (Debug Console only) bina
quit close the Debug Console connection
```
In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few run: `help`, `info`, `tasks`, `net`, `reboot`, `boot other`, `wifi status`, and anything starting with `log level`, `crash`, `coredump`, `wifi add`. Anything else answers `not available in Safe Mode`.
In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few run: `help`, `info`, `tasks`, `net`, `reboot`, `boot other`, `wifi status`, and anything starting with `log level`, `crash`, `coredump`, `wifi add`, `debug`. Anything else answers `not available in Safe Mode`.
## What they do
@@ -67,12 +64,12 @@ In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few r
| `sound on` / `sound off` | Toggles the Sound setting (beep + LED) |
| `short` / `normal` | Screen timeouts 5 s / 10 s, or 30 s / 60 s |
| `wifi add <ssid><TAB><password>` | Adds a Saved Network (so credentials stay out of the repo) |
| `wifi ip <ssid> dhcp` / `wifi ip <ssid> <address>/<prefix> [gateway]` | A Saved Network's IP setting: Automatic, or Fixed. Debug Builds: add `try <seconds>` to go back to the previous setting unless `wifi ip keep` follows |
| `wifi ip <ssid> dhcp` / `wifi ip <ssid> <address>/<prefix> [gateway]` | A Saved Network's IP setting: Automatic, or Fixed. Add `try <seconds>` to go back to the previous setting unless `wifi ip keep` follows |
| `wifi dns <a> [b]` / `wifi dns always on\|off` / `wifi ntp <a> [b]` | DNS servers (used on Fixed networks, or always), and NTP servers |
| `log <text>` | Appends a line to a test IRC Log (`/irc/dev/#test/<date>.log`) |
| `sd card` | What the SD card says it is: type, size, and its identity register (maker, name, revision, serial, date) |
| `sd list` | Lists the files of each Storage Clean-up category |
| `sd fill <folder> <count>` | Debug Builds: makes that many small files in a folder, to test a crowded one |
| `sd fill <folder> <count>` | Makes that many small files in a folder, to test a crowded one |
| `cat <path>` | Prints the first ~1.2 KB of a file on the SD card |
| `irc start` | Starts the IRC Service (normally done by opening the IRC App) |
| `irc stop` | Stops it, as `/quit` does: QUIT if connected, no more retries, and the App stays disconnected until you type |
@@ -89,8 +86,8 @@ In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few r
| `ls [folder]` / `du <path>` | Lists a folder of the SD card with sizes and dates, or counts the files and bytes under a path |
| `cp [-f] <from> <to>` / `mv [-f] <from> <to>` / `rm <path>` / `mkdir <path>` / `cancel` | What the Storage App does, with its rules: copy (folders too), move or rename, delete (a folder with what's in it), new folder. `-f` replaces a file that's in the way; a tab separates paths that hold spaces; `cancel` stops a copy or a delete |
| `install <path>` | Update from SD with that `.ota` file, as Settings → Firmware does |
| `update check` / `update list` / `update status` / `update install <tag>` | The project's releases on Gitea: look at the latest, list the last ten, say what's known, or download and install one (not on a Debug Build) |
| `update pretend <version>` / `update probe <host>` / `update damage cut\|flip <n>` / `update daily` | Debug Builds: pretend to run another version (so a release counts as an update), see whether a server's certificate is accepted, cut or damage the next download, run the daily check again |
| `update check` / `update list` / `update status` / `update install <tag>` | The project's releases on Gitea: look at the latest, list the last ten, say what's known, or download and install one |
| `update pretend <version>` / `update probe <host>` / `update damage cut\|flip <n>` / `update daily` | Pretend to run another version (so a release counts as an update), see whether a server's certificate is accepted, cut or damage the next download, run the daily check again |
| `lora probe` | Finds the radio: chip, oscillator, antenna switch, DIO1 interrupt, noise floor |
| `lora status` | Radio settings, who's listening, packet and error counters, noise floor, task stack |
| `lora rx on` / `lora rx off` | Listens and prints each packet on the console |
@@ -102,12 +99,14 @@ In **Safe Mode** (see [Crashes and Safe Mode](/dev/debug/crashes/)) only a few r
| `gnss track start` / `gnss track stop` | A Track, as `r` in the GNSS App (the reason is printed if it can't start) |
| `gnss nmea on` / `gnss nmea off` | Prints each NMEA sentence the receiver sends, as `nmea: …` |
| `gnss send <sentence>` | Sends a sentence to the receiver, without the `$` and the checksum (it adds them) |
| `lora noise test [gnss\|quiet]` / `lora noise report` | Debug Builds: Sweep under one changed condition at a time to find what raises the noise floor (Wi-Fi goes off for a few seconds); then the result |
| `lora inject <hex> [rssi] [snr]` | Debug Builds: a packet into the Scanner as if received (nothing is sent) |
| `lora noise test [gnss\|quiet]` / `lora noise report` | Sweep under one changed condition at a time to find what raises the noise floor (Wi-Fi goes off for a few seconds); then the result |
| `lora inject <hex> [rssi] [snr]` | A packet into the Scanner as if received (nothing is sent) |
| `crash` | The last crash: which firmware, why, task, PC and backtrace (from the core dump in flash) |
| `coredump erase` | Forgets the core dump |
| `loop spin on` / `loop spin off` | Debug Builds: make the main loop spin without resting, to compare load and radio noise |
| `crash abort` / `crash wdt` | Debug Builds: crash on purpose, or hang the main loop until the watchdog fires |
| `loop spin on` / `loop spin off` | Make the main loop spin without resting, to compare load and radio noise |
| `crash abort` / `crash wdt` | Crash on purpose, or hang the main loop until the watchdog fires |
| `debug status` / `debug off` | The Debug Console: whether it's on, has a token and a client; switch it off |
| `debug on` / `debug token <value>` / `debug token new` | USB serial only: switch it on (making a token if there's none), give it a token of 16 to 64 characters, or make a new one. The token is never printed |
| `help` | Lists the commands |
`scripts/flash.sh` stops a running serial log first, since it would hold the port.
site/content/dev/debug/console/index.md
+25 -16
View File
@@ -1,6 +1,6 @@
+++
title = "The Debug Console"
description = "Connect to a Debug Build over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do."
description = "Connect to the console over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do."
weight = 2
[extra]
tag = "Console"
@@ -17,16 +17,16 @@ scripts/rdbg.py info # one command, and its reply
scripts/rdbg.py -b tasks # the same, with the backlog shown first
```
`scripts/rdbg.py` is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It reads the token from `~/.config/roro9stack/debug-token` and talks to **TCP 2323** on the device's address. In interactive mode, <kbd>Ctrl</kbd>+<kbd>D</kbd> or `quit` leaves. Piped input works too, but `rdbg.py` leaves **as soon as its input ends**, before the replies arrive: keep the input open for a few seconds, as in `(printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py`. For one command, the one-shot form above is simpler.
`scripts/rdbg.py` is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It takes the token from `-t`, `$RORO_DEBUG_TOKEN` or `~/.config/roro9stack/debug-token` ([Switch the console on](/dev/debug/switch-it-on/)) and talks to **TCP 2323** on the device's address. In interactive mode, <kbd>Ctrl</kbd>+<kbd>D</kbd> or `quit` leaves. Piped input works too, but `rdbg.py` leaves **as soon as its input ends**, before the replies arrive: keep the input open for a few seconds, as in `(printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py`. For one command, the one-shot form above is simpler.
The device listens **only while Wi-Fi is connected**, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back.
The device listens **only while the console is switched on and Wi-Fi is connected**, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back.
## What you get
On connecting, in order:
1. a **banner**: `roro9stack v0.11.0-3-g12006bd+debug debug console. 'help' lists the commands. Backlog follows.`
2. the **backlog**: the last **4 KB** of console output, oldest first, **boot messages included** (a ring buffer in RAM);
1. a **banner**: `roro9stack v0.12.0 debug console. 'help' lists the commands. Backlog follows.`
2. the **backlog**: the last **4 KB** of console output, oldest first (a ring buffer in RAM, kept while the console is switched on: boot messages included, if it was on at boot);
3. then **every new line, live**: everything the firmware prints, and ESP-IDF's own log lines, which are copied into the same stream (they still reach the USB port too);
4. the **replies to your commands**, in the same stream, each preceded by the `> command` line when it runs.
@@ -49,12 +49,17 @@ It is a plain line protocol, easy to speak from anything. This is what `rdbg.py`
| Step | Detail |
|---|---|
| Connect | TCP 2323. **One client at a time**: a second one waits until the first leaves |
| Authenticate | Send the token and `\n` within **10 seconds**. The comparison takes the same time whatever you send |
| Refused | After about a second the device sends `denied\n` and hangs up, and prints `debug: refused a client from <ip>` on its own console. No quick retries |
| Challenge | The device sends one line: `roro9stack debug console, challenge <32 hex digits>`. Those are 16 random bytes, new each time |
| Answer | Within **10 seconds**, send the **HMAC-SHA256 of those 16 bytes, keyed by the token**, as 64 hex digits and `\n`. The token is the tidied one: capitals, no dashes |
| Accepted | The banner line, then the backlog |
| Refused | After about a second the device sends `denied\n` and hangs up, and prints `debug: refused a client from <ip>` on its own console |
| Locked | **Five wrong answers in a row** close the console to everyone for 60 seconds: it answers `locked\n` and hangs up, and a Toast on the device names the address they came from |
| Commands | One line each, up to 240 bytes. The device **queues at most 8**; past that, `debug: busy, command dropped` |
| Leave | `quit` or `exit` closes the connection |
| Binary commands | `get`, `put`, `screenshot`, `coredump get`: a text header line, then raw bytes (see [files and screens](/dev/debug/files-and-screens/)) |
**The token never crosses the network.** Someone on the same Wi-Fi who records a login gets a challenge and its answer, which are no use for the next challenge. The comparison on the device takes the same time whatever it is given.
A command that never runs has not been dropped by the network: the main loop is busy or stuck. That is what the next section is about.
## How commands run
@@ -69,22 +74,26 @@ A command that never runs has not been dropped by the network: the main loop is
## Security
- The token is checked **before anything else**, and a wrong one costs a second.
- Anyone on the same network **with the token** can read the console, press keys and restart the device. The console never prints stored secrets (Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
- The stream is **plain text**: fine on a home network, not across the internet. Do not forward port 2323.
- **Release builds have no console at all.** Nothing listens.
- **Off by default.** Nothing listens until the console is switched on at the device, or over USB.
- The login is a **challenge and an answer**: the token itself is never sent, and five wrong answers close the console for a minute.
- Anyone on the same network **with the token** can read the console, press keys, copy the SD card's files and restart the device. The console never prints stored secrets (the token, Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
- They cannot change the firmware: an update still has to be **signed**.
- The stream after the login is **plain text**: fine on a home network, not across the internet. Do not forward port 2323.
The decision is [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
The decisions are [ADR 0010](/dev/decisions/0010-debug-console-in-every-build/) and, for how the console works inside, [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
## Without `rdbg.py`
Anything that can open a TCP connection works. The token and each command are just lines:
Anything that can open a TCP connection and compute an HMAC works:
```python
import socket
import hashlib, hmac, socket
token = "K7QF3M2X9WBDHT4P6RNC" # tidied: capitals, no dashes
s = socket.create_connection(("10.39.39.12", 2323))
s.sendall(b"<token>\n") # then read the banner line
s.sendall(b"info\n") # read until the stream goes quiet
challenge = s.makefile().readline().split()[-1] # "... challenge 0f1e2d..."
answer = hmac.new(token.encode(), bytes.fromhex(challenge), hashlib.sha256).hexdigest()
s.sendall((answer + "\n").encode()) # then read the banner line
s.sendall(b"info\n") # and what follows, until the stream goes quiet
```
`scripts/rdbg.py` adds the parts that need work on the PC side: decoding crash backtraces, saving files and screenshots, and checking checksums.
site/content/dev/debug/crashes.md
+7 -8
View File
@@ -6,7 +6,7 @@ weight = 5
tag = "Console"
+++
Rollback (see [How an update works](/dev/build/how-an-update-works/)) protects against **new** firmware that fails Probation. These measures cover firmware that was **confirmed and then crashes**: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. They are in **every build**, release and Debug alike; the Debug Console is what makes them convenient. The decision is [ADR 0005](/dev/decisions/0005-safe-mode-crash-reports-watchdog/).
Rollback (see [How an update works](/dev/build/how-an-update-works/)) protects against **new** firmware that fails Probation. These measures cover firmware that was **confirmed and then crashes**: a corrupt setting, a server that sends something unexpected, a bug that takes an hour to show. They are in every firmware; the Debug Console is what makes them convenient. The decision is [ADR 0005](/dev/decisions/0005-safe-mode-crash-reports-watchdog/).
## What a crash leaves behind
@@ -18,7 +18,7 @@ The `crash` command prints it again whenever you like:
```
> crash
crash: last one in v0.9.0-1-g4ab873e-dirty+debug (panic)
crash: last one in v0.9.0-1-g4ab873e-dirty (panic)
crash: task loopTask, pc 0x4037e179, cause 0, address 0x00000000
crash: reason: abort() was called at PC 0x421209b3 on core 1
crash: backtrace 0x4037e179 0x4037e141 0x4038582d 0x421209b3 ...
@@ -40,7 +40,8 @@ scripts/rdbg.py coredump my.bin # ...to a file you name
- **`crash`** runs the device's `crash`, finds the ELF by digest (or by version), and passes the backtrace to `scripts/decode_backtrace.sh`, which runs `addr2line` from the build container: one line for each frame, with function and file:line.
- **`coredump`** fetches the raw dump over the console (it works when the main loop is stuck) and runs `scripts/decode_coredump.sh`: `esp-coredump` and GDB, giving **every task's backtrace, the registers, and the crashed task's stack**.
- By hand: `scripts/decode_backtrace.sh <version|digest> <address>...`, and `scripts/decode_coredump.sh <core.bin> <version|digest>`. Both say which ELF they used.
- If no archived ELF matches, they say so: the build was made on another machine, or `.pio/` was cleaned.
- **A release's ELF is fetched for you.** When the crash names a released version (`v0.12.0`) and no local ELF matches, `rdbg.py` downloads `roro9stack-<version>.elf.gz` from the release on Gitea into `.pio/elves/`, so a crash on a firmware you did not build can be decoded. Decoding itself still runs in the build container.
- If nothing matches, the scripts say so: an unreleased build made on another machine, or `.pio/` was cleaned.
## The main loop is watched
@@ -52,21 +53,19 @@ An installed update no longer depends on the main loop either: the Update Servic
The count of starts that follow a crash (a panic or the watchdog) is kept in NVS. After **three in a row**, the firmware starts **Safe Mode** instead of everything else:
- only the **clock, Wi-Fi, the Update Service and, in a Debug Build, the Debug Console** start: no Apps, no IRC, no SD card;
- only the **clock, Wi-Fi, the Update Service and, if it is switched on, the Debug Console** start: no Apps, no IRC, no SD card;
- the screen says so, with the **address to push an update to**;
- only a few commands run (the [command reference](/dev/debug/commands/) lists them); anything else answers `not available in Safe Mode`.
Any **normal restart** (`reboot`, an update) or **a minute of uptime** resets the count. So Safe Mode is reached by crashing three times *quickly*, and a crash loop costs about half a minute before the device becomes reachable. To leave it: push a fix (`scripts/flash.sh --debug --ota`), or `reboot`.
Any **normal restart** (`reboot`, an update) or **a minute of uptime** resets the count. So Safe Mode is reached by crashing three times *quickly*, and a crash loop costs about half a minute before the device becomes reachable. To leave it: push a fix (`scripts/flash.sh --ota`), or `reboot`. Over USB, `debug on` works in Safe Mode too, if the console was off.
**Its limit:** if Wi-Fi or the Update Service is what crashes, Safe Mode cannot help, and it takes USB.
## Crash on purpose
On a Debug Build:
```
crash abort # abort(): a panic with a core dump
crash wdt # hang the main loop until the task watchdog fires
```
Use them to see a real report and decode it, to check that **the counter reaches Safe Mode**, and to prove that a Debug Build you are about to rely on will survive its own crashes. The device restarts by itself after either, and the report is there to read when the console comes back.
Use them to see a real report and decode it, to check that **the counter reaches Safe Mode**, and to prove that a build you are about to rely on will survive its own crashes. The device restarts by itself after either, and the report is there to read when the console comes back.
site/content/dev/debug/debug-builds.md
-56
View File
@@ -1,56 +0,0 @@
+++
title = "Debug Builds"
description = "What a Debug Build is, how to build and flash one, how its token works, and why you should keep one in the fallback slot."
weight = 1
[extra]
tag = "Start here"
+++
## What it is
A Debug Build is built from the same source as a release, with the `cardputer-adv-debug` environment (it `extends` `cardputer-adv` in `platformio.ini`) and `-DRORO_DEBUG`. Differences:
- the **Debug Console** on TCP **2323** (next page);
- extra commands, only meant for testing: crash on purpose, fake an installed version, damage a download, inject a LoRa packet, fill a folder with files (see the [command reference](/dev/debug/commands/));
- the version ends in **`+debug`** (`scripts/version.py`) wherever the version shows: Settings, `info`, the Update Service. A `+debug` version compares **equal** to its release counterpart, so going between the two is never refused as a downgrade.
**It is compiled out of release builds, not switched off by a setting.** A console that runs commands, presses keys and reboots the device is a remote control; in a release build nothing listens and the code is not there.
## Build and flash one
Everything runs in Docker (see [Build, test and release](/dev/build/build-and-test/)). Over USB:
```sh
scripts/flash.sh --debug # builds cardputer-adv-debug, uploads, opens the serial monitor
```
Once a Debug Build is on the device, every later one can go over Wi-Fi, with no cable:
```sh
export RORO_OTA_HOST=10.39.39.12 # the address Settings > Firmware shows
scripts/flash.sh --debug --ota # builds, signs and pushes; the device installs and restarts
```
The device's address is on **Settings → Firmware** ("Push to", which also gives port 3232, the update port). The Debug Console is on the same address, port 2323. See [Flash and update](/dev/build/flash/) for the update side.
## The token
The console asks for a secret first. It is **128 random bits**, made the first time any build runs (`scripts/_docker.sh`), kept in `~/.config/roro9stack/debug-token` next to the update-signing key, and passed into the build container as `RORO_DEBUG_TOKEN`. `scripts/debug_flags.py` compiles it into the firmware, and **refuses to build a Debug Build without one**, rather than fall back on a default.
- It is **never committed.** Releases have no console, so nothing of it is published: CI builds a Debug Build on a pull request to prove it still compiles, but never publishes it, because each one carries its builder's token.
- It guards against **the network**, not against someone who holds the device: anyone with USB access can flash anything anyway ([ADR 0003](/dev/decisions/0003-own-signature-check-not-secure-boot/)).
- To change it, delete the file and build again; the new token goes into the next Debug Build you flash.
## Keep a Debug Build in the other slot
The device has two app slots, so an update never overwrites the running firmware. If a new firmware crashes during [Probation](/dev/build/how-an-update-works/), the device goes **back to the previous one**, whatever that is. As long as you develop on Debug Builds, **the firmware a crash falls back to has the console too**, so a bad update never costs you remote access. Pushing a *release* build over a Debug Build leaves the Debug Build in the other slot until the next update overwrites it.
So the habit is: develop on Debug Builds, release by tag, and think twice before pushing a release over the only Debug Build you have.
## Releases and Debug Builds
- A Debug Build shows the latest release in Settings → Firmware but **never installs it**: that would replace the console with a release that has none. Update a Debug Build from the PC with `scripts/flash.sh --debug --ota`.
- A Debug Build still checks and lists releases, which is useful for testing the update path: `update pretend` makes a released version count as newer (see [Drive the UI](/dev/debug/drive-the-ui/)).
- **Safe Mode** (after 3 crashes in a row) keeps the Debug Console running, so a crash loop is something you fix remotely: see [Crashes and Safe Mode](/dev/debug/crashes/).
The reasoning is in [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/).
site/content/dev/debug/drive-the-ui.md
+7 -8
View File
@@ -59,12 +59,12 @@ Each of these puts something in, **without** the outside world:
| `gnss send <sentence>` | Sends an NMEA sentence **to** the GNSS receiver (the checksum is added), to configure it |
| `gemini get <url>` | Fetches a page and reports header, size, certificate and heap use, without the App |
| `gemini trust <host> <port> <sha256>` | Pins a certificate by hand |
| `sd fill <folder> <count>` | **Debug Build.** Makes that many small files in a folder, to test a crowded one (the Storage App shows the first 256) |
| `sd fill <folder> <count>` | Makes that many small files in a folder, to test a crowded one (the Storage App shows the first 256) |
| `wifi add <ssid><TAB><password>` | Adds a Saved Network, so credentials stay out of the repository |
## Test the update path
An update that goes wrong is the case you most want to rehearse, and a Debug Build can make it go wrong **on purpose**:
An update that goes wrong is the case you most want to rehearse, and the firmware can make it go wrong **on purpose**:
```
update status # what the device runs, what failed here before, the daily check, heap
@@ -72,15 +72,14 @@ update check | update list # look at the server: the latest release, or
update pretend v0.9.0 # take the running version to be v0.9.0: the latest release now counts as "new"
update damage cut 50000 # the next download is cut after 50000 bytes
update damage flip 100000 # ...or has the byte at offset 100000 damaged
update install v0.11.0 force # try the install
update install v0.11.0 # try the install
update probe git.twis.la # is that server's certificate accepted? (the two ISRG roots only)
update daily # forget today's daily check: it runs again at the next tick
update pretend off # back to the real version
```
- **`force` is needed on a Debug Build.** A plain `update install <tag>` answers `Debug Build: update from the PC`, because installing a release would replace the console with a build that has none. `force` is accepted only on a Debug Build, and only from the console.
- **A damaged download must be refused cleanly:** the Update Service checks the signature after the first 160 bytes and the image hash at the end, so a cut or a flipped byte must end in a refusal with **nothing switched**. Check `info` afterwards: both slots, and `update: confirmed`.
- **An undamaged `force` install really installs** the release, into the other slot. The Debug Build stays where it was until the next update overwrites it, and a Rollback returns to it, but think before you do it.
- **An undamaged install really installs** the release, into the other slot, and the device restarts into it. Your build stays in the slot it was in until the next update overwrites it, and the console's setting and token are untouched: the release has the console too.
- The server is read by the device itself, with Wi-Fi up; the download is one TLS connection, about 52 KB of heap at its peak, so IRC steps aside (see [the memory limit](/howto/not-enough-memory/)). `status: heap` in the stream shows it happen.
For crashes during Probation, see [Crashes and Safe Mode](/dev/debug/crashes/).
@@ -94,7 +93,7 @@ wifi ip MyNet 10.39.39.50/24 10.39.39.1 try 60 # use this address for 60 s..
wifi ip keep # ...and keep it, if you could still reach the device
```
If you do not send `wifi ip keep` in time, the device goes back to the **previous** setting by itself, and the console comes back with it. (A Debug Build command: `try` is not in release builds.)
If you do not send `wifi ip keep` in time, the device goes back to the **previous** setting by itself, and the console comes back with it.
## Measure
@@ -104,7 +103,7 @@ tasks # each task over the next second: state, priority, least free stack,
net # bytes each network service has read and written since start
```
`tasks` takes a second to answer. A low number in the `stack` column is a risk (the System App shows it in the warning colour under 512 bytes). `loop spin on|off` (Debug Build) makes the main loop spin without resting, to compare load and radio noise. And the `status: heap` line every 10 seconds in the stream is the cheapest memory trace there is: watch it while you do the thing you suspect.
`tasks` takes a second to answer. A low number in the `stack` column is a risk (the System App shows it in the warning colour under 512 bytes). `loop spin on|off` makes the main loop spin without resting, to compare load and radio noise. And the `status: heap` line every 10 seconds in the stream is the cheapest memory trace there is: watch it while you do the thing you suspect.
```
task st pri stack cpu% core
@@ -120,4 +119,4 @@ The [System App](/guide/system/) shows the same, live, on the device.
## Radio experiments
`lora preset <name>` and `lora custom <MHz> <BW kHz> <SF> <CR> <sync hex> [preamble]` change what the receiver listens to (receive only: the radio never transmits), `lora sweep on [from] [to] [step]` takes a survey, and `lora noise test [gnss|quiet]` (Debug Build) runs a Sweep under one changed condition at a time, with Wi-Fi off for a moment, to find what raises the noise floor. `lora probe` finds the radio and reports its chip, oscillator, antenna switch, interrupt line and noise floor. Details in the [command reference](/dev/debug/commands/).
`lora preset <name>` and `lora custom <MHz> <BW kHz> <SF> <CR> <sync hex> [preamble]` change what the receiver listens to (receive only: the radio never transmits), `lora sweep on [from] [to] [step]` takes a survey, and `lora noise test [gnss|quiet]` runs a Sweep under one changed condition at a time, with Wi-Fi off for a moment, to find what raises the noise floor. `lora probe` finds the radio and reports its chip, oscillator, antenna switch, interrupt line and noise floor. Details in the [command reference](/dev/debug/commands/).
site/content/dev/debug/switch-it-on.md
+76
View File
@@ -0,0 +1,76 @@
+++
title = "Switch the console on"
description = "The Debug Console is in every firmware, and off. How to switch it on, where its token comes from, and how to set a device up without typing anything."
weight = 1
[extra]
tag = "Start here"
+++
## One firmware
There is no special build. **Every roro9stack firmware has the Debug Console**, the same one, and the commands made for testing (crash on purpose, fake an installed version, damage a download, inject a LoRa packet). It is **off** until its owner switches it on.
**Off means nothing is there:** no socket listens, the console's task does not exist, and neither does its 4 KB buffer. A device that never uses it pays 30 KB of flash and 88 bytes of memory.
Before version 0.12 this was a separate *Debug Build* with a token compiled in from the builder's machine, which is why it could not be published. [ADR 0010](/dev/decisions/0010-debug-console-in-every-build/) says why that changed.
## On the device
**Settings → Debug Console:**
| Row | Does |
|---|---|
| **Debug Console** | The switch. Switching it **on** asks first, and makes a token if there is none |
| **Connect to** | The address and port: `10.39.39.12:2323` |
| **New token** | Makes another one. The old one stops working, and whoever is connected is cut off |
| **Type a token** | One of your own, of 16 to 64 characters |
Under the rows, the **token**, in large type on two lines, in groups of four: `K7QF-3M2X-9WBD-HT4P-6RNC`. This page is the only place it is ever shown. The Status Bar shows **`DBG`** while the console listens, and brighter while someone is connected.
The setting **stays** across restarts and updates, and in Safe Mode.
## The token
- **The device makes it,** from its hardware random generator, the first time the console is switched on: 100 bits, written as 20 characters without the letters that get misread (no I, L, O or U).
- **Dashes and case do not count,** and an `O`, `I` or `L` is taken for the `0` or `1` it was. Type it as you read it.
- **One you type** must have at least 16 characters. A short one would be the weakest part of the whole thing.
- It is **never printed** on a console, and it **never crosses the network** ([the Debug Console](/dev/debug/console/) says how).
- It guards against **the network**, not against someone who holds the device: anyone with USB access can flash anything anyway ([ADR 0003](/dev/decisions/0003-own-signature-check-not-secure-boot/)).
## Give `rdbg.py` the token
`scripts/rdbg.py` looks for it in this order:
```sh
scripts/rdbg.py -t K7QF-3M2X-9WBD-HT4P-6RNC info # 1. on the command line (also --token)
RORO_DEBUG_TOKEN=K7QF-3M2X-9WBD-HT4P-6RNC scripts/rdbg.py info # 2. in the environment
echo K7QF-3M2X-9WBD-HT4P-6RNC > ~/.config/roro9stack/debug-token # 3. in a file, for the device you use every day
```
## Without typing: over USB
With the device on a cable, the console can be set up from the PC:
```sh
scripts/flash.sh --debug # flashes over USB, then switches the console on and gives it your token
```
It sends two commands over the serial port, which you can also type there yourself:
```
debug on # switch it on (a token is made if there is none)
debug token <value> # give it this token: 16 to 64 characters
debug token new # make a new one
debug status # on or off, token set or not, a client or not (the token itself is never shown)
debug off # switch it off
```
`debug on` and `debug token` work **over USB serial only**: the console cannot be used to open itself wider. `debug status` and `debug off` work from anywhere, and a screenshot taken over the console while this page is open shows the token, to someone who already had it. Your token file is made by the first build (`scripts/_docker.sh`), 32 hex digits, and is never committed.
Since the setting survives updates, this is needed **once for a device**, not at each flash. Later builds go over Wi-Fi with `scripts/flash.sh --ota`.
## Should it be on?
On your own network, on a device you are working on: yes, that is what it is for. Remember what it gives to whoever has the token **and** is on the same network: the console, the keys, the files on the SD card, a restart. It does not give them the firmware: an update still has to be signed.
On a network you share with strangers, switch it off, or at least know that what the console prints is not encrypted. The token is safe there; the conversation is not.