+++ title = "The Debug Console" description = "Connect to the console over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do." weight = 2 [extra] tag = "Console" diagrams = true +++ ## Connect ```sh export RORO_OTA_HOST=10.39.39.12 # or pass -H scripts/rdbg.py # interactive: the backlog, then live lines; type commands scripts/rdbg.py info # one command, and its reply scripts/rdbg.py -b tasks # the same, with the backlog shown first ``` `scripts/rdbg.py` is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It takes the token from `-t`, `$RORO_DEBUG_TOKEN` or `~/.config/roro9stack/debug-token` ([Switch the console on](/dev/debug/switch-it-on/)) and talks to **TCP 2323** on the device's address. In interactive mode, Ctrl+D or `quit` leaves. Piped input works too, but `rdbg.py` leaves **as soon as its input ends**, before the replies arrive: keep the input open for a few seconds, as in `(printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py`. For one command, the one-shot form above is simpler. The device listens **only while the console is switched on and Wi-Fi is connected**, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back. ## What you get On connecting, in order: 1. a **banner**: `roro9stack v0.12.0 debug console. 'help' lists the commands. Backlog follows.` 2. the **backlog**: the last **4 KB** of console output, oldest first (a ring buffer in RAM, kept while the console is switched on: boot messages included, if it was on at boot); 3. then **every new line, live**: everything the firmware prints, and ESP-IDF's own log lines, which are copied into the same stream (they still reach the USB port too); 4. the **replies to your commands**, in the same stream, each preceded by the `> command` line when it runs. A line `status: heap min ` appears every 10 seconds in the stream: a free-memory trace you get without asking. ``` > net net: IRC in 0 out 0 net: Gemini in 0 out 0 net: Debug Console in 1038 out 205115 net: Updates in 4788 out 204 ``` If you are not reading fast enough (a slow link), the device says so in the stream instead of stalling: `[... 312 bytes lost: the console ran faster than the network]`. ## The protocol It is a plain line protocol, easy to speak from anything. This is what `rdbg.py` does, and all it needs: | Step | Detail | |---|---| | Connect | TCP 2323. **One client at a time**: a second one waits until the first leaves | | Challenge | The device sends one line: `roro9stack debug console, challenge <32 hex digits>`. Those are 16 random bytes, new each time | | Answer | Within **10 seconds**, send the **HMAC-SHA256 of those 16 bytes, keyed by the token**, as 64 hex digits and `\n`. The token is the tidied one: capitals, no dashes | | Accepted | The banner line, then the backlog | | Refused | After about a second the device sends `denied\n` and hangs up, and prints `debug: refused a client from ` on its own console | | Locked | **Five wrong answers in a row** close the console to everyone for 60 seconds: it answers `locked\n` and hangs up, and a Toast on the device names the address they came from | | Commands | One line each, up to 240 bytes. The device **queues at most 8**; past that, `debug: busy, command dropped` | | Leave | `quit` or `exit` closes the connection | | Binary commands | `get`, `put`, `screenshot`, `coredump get`: a text header line, then raw bytes (see [files and screens](/dev/debug/files-and-screens/)) | **The token never crosses the network.** Someone on the same Wi-Fi who records a login gets a challenge and its answer, which are no use for the next challenge. The comparison on the device takes the same time whatever it is given. A command that never runs has not been dropped by the network: the main loop is busy or stuck. That is what the next section is about. ## How commands run {{ diagram(src="debug-console.svg", min_width=580, caption="The console task only queues command lines. The main loop runs them with the same code as USB serial commands. Binary commands are answered by the console task itself, so they work when the main loop is stuck.") }} - **Text commands run on the main loop**, exactly like serial ones: they touch the Apps and the Services from the one task allowed to. The main loop prints `> command` as it starts, and the reply follows in the stream. - **Binary commands run on the console's own task**: `get`, `put`, `screenshot`, `coredump get` and `reset`. A failed `put` closes the connection, so the rest of the file is never read as commands. - That split is the point: with the **main loop stuck** (an infinite loop, a deadlock), text commands queue forever, but **`reset` still restarts the device at once**, `coredump get` still reads the dump, and `get` still reads files. A loop stuck for 5 seconds is a panic with a core dump anyway: see [Crashes and Safe Mode](/dev/debug/crashes/). - **Card work stays on the storage task**: `ls`, `rm`, `cp` and `install` from the main loop, `get` and `put` from the console task, all as jobs the storage task runs, so the card is only ever touched from one place. - Writes to the console **never wait for USB**: a host attached to the USB port but not reading used to stall the main loop for up to two seconds per line. ## Security - **Off by default.** Nothing listens until the console is switched on at the device, or over USB. - The login is a **challenge and an answer**: the token itself is never sent, and five wrong answers close the console for a minute. - Anyone on the same network **with the token** can read the console, press keys, copy the SD card's files and restart the device. The console never prints stored secrets (the token, Wi-Fi and IRC passwords), but the IRC traffic it shows is readable. - They cannot change the firmware: an update still has to be **signed**. - The stream after the login is **plain text**: fine on a home network, not across the internet. Do not forward port 2323. The decisions are [ADR 0010](/dev/decisions/0010-debug-console-in-every-build/) and, for how the console works inside, [ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/). ## Without `rdbg.py` Anything that can open a TCP connection and compute an HMAC works: ```python import hashlib, hmac, socket token = "K7QF3M2X9WBDHT4P6RNC" # tidied: capitals, no dashes s = socket.create_connection(("10.39.39.12", 2323)) challenge = s.makefile().readline().split()[-1] # "... challenge 0f1e2d..." answer = hmac.new(token.encode(), bytes.fromhex(challenge), hashlib.sha256).hexdigest() s.sendall((answer + "\n").encode()) # then read the banner line s.sendall(b"info\n") # and what follows, until the stream goes quiet ``` `scripts/rdbg.py` adds the parts that need work on the PC side: decoding crash backtraces, saving files and screenshots, and checking checksums.