roro9stack/site/content/dev/debug/console/index.md

+++ title = "The Debug Console" description = "Connect to the console over Wi-Fi: the protocol, what you get when you connect, how commands run, and what the console can and cannot do." weight = 2 [extra] tag = "Console" diagrams = true +++

Connect

export RORO_OTA_HOST=10.39.39.12   # or pass -H <ip>

scripts/rdbg.py            # interactive: the backlog, then live lines; type commands
scripts/rdbg.py info       # one command, and its reply
scripts/rdbg.py -b tasks   # the same, with the backlog shown first

scripts/rdbg.py is a Python script with no dependencies. A one-shot command prints the reply and what follows, until the console has been quiet for a moment (1.5 seconds). It takes the token from -t, $RORO_DEBUG_TOKEN or ~/.config/roro9stack/debug-token (Switch the console on) and talks to TCP 2323 on the device's address. In interactive mode, Ctrl+D or quit leaves. Piped input works too, but rdbg.py leaves as soon as its input ends, before the replies arrive: keep the input open for a few seconds, as in (printf 'info\ntasks\n'; sleep 3) | scripts/rdbg.py. For one command, the one-shot form above is simpler.

The device listens only while the console is switched on and Wi-Fi is connected, and the console follows Wi-Fi: it stops listening when Wi-Fi drops and starts again when it is back.

What you get

On connecting, in order:

1. a banner: roro9stack v0.12.0 debug console. 'help' lists the commands. Backlog follows.
2. the backlog: the last 4 KB of console output, oldest first (a ring buffer in RAM, kept while the console is switched on: boot messages included, if it was on at boot);
3. then every new line, live: everything the firmware prints, and ESP-IDF's own log lines, which are copied into the same stream (they still reach the USB port too);
4. the replies to your commands, in the same stream, each preceded by the > command line when it runs.

A line status: heap <free> min <lowest> appears every 10 seconds in the stream: a free-memory trace you get without asking.

> net
net: IRC            in          0  out          0
net: Gemini         in          0  out          0
net: Debug Console  in       1038  out     205115
net: Updates        in       4788  out        204

If you are not reading fast enough (a slow link), the device says so in the stream instead of stalling: [... 312 bytes lost: the console ran faster than the network].

The protocol

It is a plain line protocol, easy to speak from anything. This is what rdbg.py does, and all it needs:

Step	Detail
Connect	TCP 2323. One client at a time: a second one waits until the first leaves
Challenge	The device sends one line: roro9stack debug console, challenge <32 hex digits>. Those are 16 random bytes, new each time
Answer	Within 10 seconds, send the HMAC-SHA256 of those 16 bytes, keyed by the token, as 64 hex digits and \n. The token is the tidied one: capitals, no dashes
Accepted	The banner line, then the backlog
Refused	After about a second the device sends denied\n and hangs up, and prints debug: refused a client from <ip> on its own console
Locked	Five wrong answers in a row close the console to everyone for 60 seconds: it answers locked\n and hangs up, and a Toast on the device names the address they came from
Commands	One line each, up to 240 bytes. The device queues at most 8; past that, debug: busy, command dropped
Leave	quit or exit closes the connection
Binary commands	get, put, screenshot, coredump get: a text header line, then raw bytes (see files and screens)

The token never crosses the network. Someone on the same Wi-Fi who records a login gets a challenge and its answer, which are no use for the next challenge. The comparison on the device takes the same time whatever it is given.

A command that never runs has not been dropped by the network: the main loop is busy or stuck. That is what the next section is about.

How commands run

{{ diagram(src="debug-console.svg", min_width=580, caption="The console task only queues command lines. The main loop runs them with the same code as USB serial commands. Binary commands are answered by the console task itself, so they work when the main loop is stuck.") }}

• Text commands run on the main loop, exactly like serial ones: they touch the Apps and the Services from the one task allowed to. The main loop prints > command as it starts, and the reply follows in the stream.
• Binary commands run on the console's own task: get, put, screenshot, coredump get and reset. A failed put closes the connection, so the rest of the file is never read as commands.
• That split is the point: with the main loop stuck (an infinite loop, a deadlock), text commands queue forever, but reset still restarts the device at once, coredump get still reads the dump, and get still reads files. A loop stuck for 5 seconds is a panic with a core dump anyway: see Crashes and Safe Mode.
• Card work stays on the storage task: ls, rm, cp and install from the main loop, get and put from the console task, all as jobs the storage task runs, so the card is only ever touched from one place.
• Writes to the console never wait for USB: a host attached to the USB port but not reading used to stall the main loop for up to two seconds per line.

Security

• Off by default. Nothing listens until the console is switched on at the device, or over USB.
• The login is a challenge and an answer: the token itself is never sent, and five wrong answers close the console for a minute.
• Anyone on the same network with the token can read the console, press keys, copy the SD card's files and restart the device. The console never prints stored secrets (the token, Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
• They cannot change the firmware: an update still has to be signed.
• The stream after the login is plain text: fine on a home network, not across the internet. Do not forward port 2323.

The decisions are ADR 0010 and, for how the console works inside, ADR 0004.

Without rdbg.py

Anything that can open a TCP connection and compute an HMAC works:

import hashlib, hmac, socket
token = "K7QF3M2X9WBDHT4P6RNC"                    # tidied: capitals, no dashes
s = socket.create_connection(("10.39.39.12", 2323))
challenge = s.makefile().readline().split()[-1]   # "... challenge 0f1e2d..."
answer = hmac.new(token.encode(), bytes.fromhex(challenge), hashlib.sha256).hexdigest()
s.sendall((answer + "\n").encode())               # then read the banner line
s.sendall(b"info\n")                              # and what follows, until the stream goes quiet

scripts/rdbg.py adds the parts that need work on the PC side: decoding crash backtraces, saving files and screenshots, and checking checksums.