Debug Builds: the console over Wi-Fi (Debug Console, TCP 2323)

cardputer-adv-debug (-DRORO_DEBUG, version +debug) adds a Debug Console:
after a token line, a client gets the last 6 KB of console output, live
lines (ESP-IDF logs included) and the serial commands. The socket task
only queues lines; the main loop runs them. Release builds compile none
of it. The token lives in ~/.config/roro9stack/debug-token, created by
_docker.sh and passed into the container.

All output now goes through `console`, which never waits for USB: a host
that was attached but not reading stalled the main loop up to 2 s per
line. New commands everywhere: info (slots with their versions from NVS,
since the framework stamps its own into each image), tasks, reboot,
boot other, log level, help. scripts/rdbg.py is the client; flash.sh
--debug builds it; CI builds both variants. ADR 0004.

Verified on the device: USB-flashed, then updated over Wi-Fi to a Debug
Build that confirmed on Probation; both slots hold Debug Builds.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT

CONTEXT.md

@@ -120,6 +120,14 @@ _Avoid_: trial, test mode
 Returning automatically to the previous firmware when new firmware resets or crashes during Probation.
 _Avoid_: revert, downgrade (a downgrade is installing an older version on purpose)
 
+**Debug Build**:
+A firmware built with the remote debugging aids compiled in (`+debug` in its version). Release builds have none of them.
+_Avoid_: dev build, test build (a test build is one made to fail on purpose, such as a crashing update)
+
+**Debug Console**:
+The console of a Debug Build over Wi-Fi: live log lines and the serial commands, behind a token.
+_Avoid_: telnet, remote shell
+
 ## Relationships
 
 - The **Launcher** starts **Apps**. Exactly one **App** is in the foreground.

README.md

@@ -70,5 +70,22 @@ To install from the SD card instead, copy the `.ota` file from `.pio/build/cardp
 | `irc say <buffer> <text>` | Types into a Buffer, commands included (`irc say 0 /join #test`) |
 | `irc dump` | Prints IRC status, memory, and the last lines of each Buffer |
 | `wifi status` | Prints Wi-Fi state, network, signal, clock and free heap |
+| `info` | Firmware, uptime, last start reason, memory, Wi-Fi, and both app slots with their versions and OTA states |
+| `tasks` | FreeRTOS tasks: state, priority, lowest free stack, CPU share |
+| `reboot` / `boot other` | Restart, or restart into the other app slot (a manual Rollback) |
+| `log level <0-5>` | ESP-IDF log level |
+| `help` | Lists the commands |
 
 `scripts/flash.sh` stops a running serial log first, since it would hold the port.
+
+### Debug Builds and the Debug Console
+
+`scripts/flash.sh --debug` (USB) or `scripts/flash.sh --debug --ota <ip>` (Wi-Fi) installs a Debug Build: the same firmware plus the Debug Console on TCP 2323 (ADR 0004). Then, with `RORO_OTA_HOST` set to the device's IP:
+
+```sh
+scripts/rdbg.py                 # interactive: the console backlog, live lines, and commands
+scripts/rdbg.py info            # one command and its reply
+scripts/rdbg.py -b tasks        # the same, after the backlog (boot messages and so on)
+```
+
+Every command above works there too. The token is in `~/.config/roro9stack/debug-token`, made by the first build; keep developing on Debug Builds, so the firmware a Rollback returns to always has the console.

docs/adr/0004-debug-console-in-debug-builds.md

@@ -0,0 +1,22 @@
+# A Debug Console over Wi-Fi, in Debug Builds only
+
+The goal of Firmware Updates is to manage the device without a cable, and that includes finding out what went wrong. So a **Debug Build** (`cardputer-adv-debug`, `-DRORO_DEBUG`, version suffix `+debug`) adds a **Debug Console** on TCP 2323: the serial console, over Wi-Fi. A client sends a token as its first line, then gets the last 6 KB of console output (boot messages included), every new line live, and runs the same commands as the serial port, plus a few that only make sense remotely. ESP-IDF's own log lines are teed into it.
+
+It's compiled out of release builds entirely, rather than switched off by a setting. A console that runs commands is a remote control: in a release build, nothing listens.
+
+## How it fits
+
+- **Console, not Serial.** All human-readable output goes through `console`, which writes to the USB port and, in a Debug Build, to a ring buffer the Debug Console drains. Writes never wait for USB: a host that's attached but not reading used to stall the main loop for up to 2 s per line.
+- **Commands run on the main loop.** The socket lives on the Debug Console's own task, which only queues command lines. The main loop runs them, as it does serial commands, so they touch Apps and Services from the one task allowed to.
+- **The token** is 128 random bits in `~/.config/roro9stack/debug-token`, made by the first build and passed into the container. It's never committed; a Debug Build refuses to compile without one. Like the OTA key, it guards against the network, not against someone holding the device.
+- **One client at a time**, to keep memory flat (about 6 KB for the ring, 6 KB of task stack).
+
+## Keep a Debug Build in the fallback slot
+
+Rollback returns to the previous firmware, whatever it is. As long as development goes through Debug Builds, the firmware a crash falls back to has the Debug Console, so a bad update never costs remote access. A release build pushed over a Debug Build leaves the Debug Build in the other slot until the next update overwrites it.
+
+## Consequences
+
+- Anyone on the same network with the token can read the console, inject keys and reboot the device. The console never prints stored secrets (Wi-Fi and IRC passwords), but the IRC traffic it shows is readable.
+- The TCP stream is plain text: fine on a home network, not across the internet.
+- `+debug` versions compare equal to their release counterparts, so moving between the two is never refused as a downgrade.

platformio.ini

@@ -21,6 +21,15 @@ lib_deps =
     m5stack/M5Cardputer @ 1.1.1
 test_ignore = *
 
+; Debug Build: the same firmware plus the Debug Console on TCP 2323 (see ADR 0004). The token comes
+; from ~/.config/roro9stack/debug-token, passed in by scripts/_docker.sh; it's never committed.
+[env:cardputer-adv-debug]
+extends = env:cardputer-adv
+extra_scripts = pre:scripts/version.py, pre:scripts/debug_flags.py
+build_flags =
+    ${env:cardputer-adv.build_flags}
+    -DRORO_DEBUG
+
 ; Host-side unit tests for pure logic (no hardware).
 [env:native]
 platform = native

scripts/_docker.sh

@@ -4,9 +4,17 @@ ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 
 docker build -q -t "$IMAGE" "$ROOT/docker" >/dev/null
 
+# The Debug Console token (ADR 0004): made once, kept with the OTA key, never committed.
+DEBUG_TOKEN_FILE="$HOME/.config/roro9stack/debug-token"
+if [ ! -s "$DEBUG_TOKEN_FILE" ]; then
+    mkdir -p "$(dirname "$DEBUG_TOKEN_FILE")"
+    (umask 077 && od -An -tx1 -N16 /dev/urandom | tr -d ' \n' > "$DEBUG_TOKEN_FILE")
+fi
+
 run_in_container() {
     docker run --rm \
         -u "$(id -u):$(id -g)" -e HOME=/tmp \
+        -e RORO_DEBUG_TOKEN="$(cat "$DEBUG_TOKEN_FILE")" \
         -v "$ROOT:/work" \
         -v roro9stack-pio:/pio \
         "${DOCKER_EXTRA[@]}" \

scripts/ci.sh

@@ -4,4 +4,4 @@ set -euo pipefail
 source "$(dirname "$0")/_docker.sh"
 DOCKER_EXTRA=()
 
-run_in_container bash -c 'git config --global --add safe.directory /work && pio test -e native && pio run -e cardputer-adv'
+run_in_container bash -c 'git config --global --add safe.directory /work && pio test -e native && pio run -e cardputer-adv -e cardputer-adv-debug'

scripts/debug_flags.py

@@ -0,0 +1,12 @@
+# Debug Builds only: compiles in the Debug Console token from $RORO_DEBUG_TOKEN (set by _docker.sh
+# from ~/.config/roro9stack/debug-token). Refuses to build without one rather than use a default.
+import os
+import re
+import sys
+
+Import("env")  # noqa: F821  (provided by PlatformIO)
+
+token = os.environ.get("RORO_DEBUG_TOKEN", "").strip()
+if not re.fullmatch(r"[0-9a-f]{32}", token):
+    sys.exit("debug build: RORO_DEBUG_TOKEN is missing; build through scripts/ci.sh or scripts/flash.sh --debug")
+env.Append(CPPDEFINES=[("RORO_DEBUG_TOKEN", '\\"%s\\"' % token)])  # noqa: F821

scripts/flash.sh

@@ -1,9 +1,15 @@
 #!/usr/bin/env bash
 # Flash the firmware over USB, then open the serial monitor.
-# Usage: scripts/flash.sh [port]          USB (default: the first Espressif device found)
+# Usage: scripts/flash.sh [--debug] [port]        USB (default: the first Espressif device found)
-#        scripts/flash.sh --ota [host]    Wi-Fi: build, sign and push a Firmware Update
+#        scripts/flash.sh [--debug] --ota [host]  Wi-Fi: build, sign and push a Firmware Update
 #                                                 (host: the device's IP from Settings > About, or $RORO_OTA_HOST)
+# --debug builds the Debug Build (cardputer-adv-debug): the Debug Console on TCP 2323, see ADR 0004.
 set -euo pipefail
+ENV=cardputer-adv
+if [ "${1:-}" = "--debug" ]; then
+    ENV=cardputer-adv-debug
+    shift
+fi
 
 if [ "${1:-}" = "--ota" ]; then
     ROOT="$(cd "$(dirname "$0")/.." && pwd)"
@@ -11,10 +17,11 @@ if [ "${1:-}" = "--ota" ]; then
     [ -n "$HOST" ] || { echo "Usage: scripts/flash.sh --ota <device IP> (or set RORO_OTA_HOST)" >&2; exit 1; }
     source "$(dirname "$0")/_docker.sh"
     DOCKER_EXTRA=()
-    run_in_container bash -c 'git config --global --add safe.directory /work && pio run -e cardputer-adv' | tail -3
+    run_in_container bash -c "git config --global --add safe.directory /work && pio run -e $ENV" | tail -3
     VERSION="$(git -C "$ROOT" describe --tags --always --dirty)"
-    OUT="$ROOT/.pio/build/cardputer-adv/roro9stack-$VERSION.ota"
+    [ "$ENV" = cardputer-adv-debug ] && VERSION="$VERSION+debug"  # as scripts/version.py names it
-    "$ROOT/scripts/make_ota.py" "$ROOT/.pio/build/cardputer-adv/firmware.bin" "$VERSION" "$OUT"
+    OUT="$ROOT/.pio/build/$ENV/roro9stack-$VERSION.ota"
+    "$ROOT/scripts/make_ota.py" "$ROOT/.pio/build/$ENV/firmware.bin" "$VERSION" "$OUT"
     exec "$ROOT/scripts/ota_push.py" "$OUT" "$HOST"
 fi
 PORT="${1:-$(readlink -f /dev/serial/by-id/*Espressif* 2>/dev/null | head -1)}"
@@ -23,4 +30,4 @@ source "$(dirname "$0")/_docker.sh"
 docker rm -f roro9stack-serial >/dev/null 2>&1 || true  # a serial log would hold the port
 DOCKER_EXTRA=(--device "$PORT" --group-add "$(stat -c %g "$PORT")" -it)
 
-run_in_container bash -c "git config --global --add safe.directory /work && pio run -e cardputer-adv -t upload --upload-port $PORT && pio device monitor -p $PORT -b 115200"
+run_in_container bash -c "git config --global --add safe.directory /work && pio run -e $ENV -t upload --upload-port $PORT && pio device monitor -p $PORT -b 115200"

scripts/rdbg.py

@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""The Debug Console of a Debug Build, over Wi-Fi (TCP 2323, see ADR 0004).
+
+Usage: scripts/rdbg.py [-H host] [-b] [command ...]
+  no command   interactive: type commands, see every console line live; Ctrl-D or `quit` leaves
+  command      runs it and prints what follows, until the console has been quiet for a moment
+  -H host      the device's IP (Settings > Firmware), default $RORO_OTA_HOST
+  -b           also print the backlog the device sends on connecting (boot messages and so on)
+The token is read from ~/.config/roro9stack/debug-token (made by the first build).
+"""
+import os
+import select
+import socket
+import sys
+import time
+
+PORT = 2323
+TOKEN_FILE = os.path.expanduser("~/.config/roro9stack/debug-token")
+
+
+def read_until(sock, marker, timeout):
+    """Everything up to and including `marker`, or None on a timeout or a closed connection."""
+    sock.settimeout(timeout)
+    data = b""
+    while marker not in data:
+        try:
+            chunk = sock.recv(4096)
+        except socket.timeout:
+            return None
+        if not chunk:
+            return None
+        data += chunk
+    return data
+
+
+def read_until_quiet(sock, quiet, out):
+    """Copies everything the device sends to `out` until nothing has arrived for `quiet` seconds."""
+    sock.settimeout(quiet)
+    while True:
+        try:
+            data = sock.recv(4096)
+        except socket.timeout:
+            return True
+        if not data:
+            return False
+        if out:
+            out.write(data.decode(errors="replace"))
+            out.flush()
+
+
+def interactive(sock):
+    sock.setblocking(False)
+    while True:
+        ready, _, _ = select.select([sock, sys.stdin], [], [])
+        if sock in ready:
+            data = sock.recv(4096)
+            if not data:
+                print("\n(connection closed)")
+                return
+            sys.stdout.write(data.decode(errors="replace"))
+            sys.stdout.flush()
+        if sys.stdin in ready:
+            line = sys.stdin.readline()
+            if not line:
+                return
+            sock.sendall(line.encode())
+
+
+def main():
+    args = sys.argv[1:]
+    host, backlog = os.environ.get("RORO_OTA_HOST"), False
+    while args and args[0].startswith("-"):
+        flag = args.pop(0)
+        if flag == "-H" and args:
+            host = args.pop(0)
+        elif flag == "-b":
+            backlog = True
+        else:
+            sys.exit(__doc__)
+    if not host:
+        sys.exit("No device: pass -H <ip> or set RORO_OTA_HOST\n\n" + __doc__)
+    token = open(TOKEN_FILE).read().strip()
+
+    with socket.create_connection((host, PORT), timeout=10) as sock:
+        sock.sendall((token + "\n").encode())
+        # The device may still be finishing a previous client: wait for this connection's banner.
+        banner = read_until(sock, b"Backlog follows.\n", 15)
+        if banner is None:
+            sys.exit("device: no banner (wrong token, or another client is connected)")
+        show = sys.stdout if backlog or not args else None
+        if show:
+            show.write(banner.decode(errors="replace"))
+        read_until_quiet(sock, 0.5, show)  # the backlog
+        if not args:
+            return interactive(sock)
+        command = " ".join(args)
+        sock.sendall((command + "\n").encode())
+        # The main loop echoes "> command" when it runs it; the reply follows.
+        echoed = read_until(sock, f"> {command}\n".encode(), 15)
+        if echoed is None:
+            sys.exit("device: the command never ran")
+        sys.stdout.write(echoed.decode(errors="replace").split(f"> {command}\n", 1)[1])
+        try:
+            read_until_quiet(sock, 1.5, sys.stdout)
+        except BrokenPipeError:  # e.g. piped into head
+            pass
+
+
+if __name__ == "__main__":
+    main()

scripts/version.py

@@ -10,4 +10,7 @@ try:
 except Exception:
     version = "unknown"
 
+if env["PIOENV"].endswith("-debug"):  # noqa: F821
+    version += "+debug"  # a Debug Build says so wherever the version shows
+
 env.Append(CPPDEFINES=[("RORO_VERSION", '\\"%s\\"' % version)])  # noqa: F821

src/main.cpp

@@ -15,12 +15,15 @@
 #include "event_bus.h"
 #include "file_receiver.h"
 #include "key_mapper.h"
+#include "platform/console.h"
 #include "platform/nvs_store.h"
+#include "platform/system_info.h"
 #include "service_manager.h"
 #include "platform/identity.h"
 #include "services/battery_service.h"
 #include "services/irc_service.h"
 #include "services/clock_service.h"
+#include "services/debug_console.h"
 #include "services/power_service.h"
 #include "services/storage_service.h"
 #include "services/update_service.h"
@@ -49,6 +52,9 @@ static SavedNetworks* savedNetworks;
 static WifiService* wifi;
 static IrcService* irc;
 static UpdateService* update;
+#ifdef RORO_DEBUG
+static DebugConsole* debugConsole;
+#endif
 static Notifier* notifier;
 static LauncherApp launcher;
 static AppManager* apps;
@@ -102,6 +108,7 @@ void setup() {
     nvs.begin();
     UpdateService::bootGuard(nvs);  // first: before anything that could crash on new firmware
     Serial.setRxBufferSize(2 * FileReceiver::kChunk);  // before the port opens; sd put sends 1 chunk at a time
+    Serial.setTxBufferSize(2048);  // the console skips Serial when it's full: room for a burst like `tasks`
 
     auto cfg = M5.config();
     M5Cardputer.begin(cfg, true);
@@ -128,6 +135,10 @@ void setup() {
     services.add(*wifi);
     services.add(*irc);
     services.add(*update);
+#ifdef RORO_DEBUG
+    debugConsole = new DebugConsole(*wifi);
+    services.add(*debugConsole);
+#endif
 
     apps = new AppManager(launcher);
     launcher.setManager(*apps);
@@ -138,16 +149,17 @@ void setup() {
     apps->registerApp({"demo", "Widget demo", true, new DemoApp(bus)});
     apps->registerApp({"setup", "Setup", true, new SetupApp(settings, *apps)});
 
-    bus.subscribe(EventType::Notification, [](const Event& e) { Serial.printf("notification: %s\n", e.text); });
+    bus.subscribe(EventType::Notification, [](const Event& e) { console.printf("notification: %s\n", e.text); });
 
-    if (!screen.begin()) Serial.println("frame buffer allocation failed");
+    if (!screen.begin()) console.println("frame buffer allocation failed");
     services.startAll(millis());
     apps->begin();
     if (!settings.getBool(Setting::SetupDone)) apps->openModal("setup");
-    Serial.printf("%s %s ready, free heap %u\n", kProductName, versionString(), ESP.getFreeHeap());
+    console.printf("%s %s ready, free heap %u, last start: %s\n", kProductName, versionString(), ESP.getFreeHeap(),
+                   system_info::resetReason());
 }
 
-// Dev aid: serial commands to drive the UI without the keyboard.
+// Dev aid: commands to drive the UI without the keyboard, from the serial port or the Debug Console.
 static bool listingWanted = false;
 
 static void printListingWhenReady() {
@@ -157,9 +169,9 @@ static void printListingWhenReady() {
     for (size_t i = 0; i < listing.size(); i++) {
         uint64_t bytes = 0;
         for (auto& f : listing[i]) bytes += f.bytes;
-        Serial.printf("sd list: %-20s %u files, %s\n", kCleanupCategories[i].label, (unsigned)listing[i].size(),
+        console.printf("sd list: %-20s %u files, %s\n", kCleanupCategories[i].label, (unsigned)listing[i].size(),
                        formatBytes(bytes).c_str());
-        for (auto& f : listing[i]) Serial.printf("  %s (%u B)\n", f.path.c_str(), (unsigned)f.bytes);
+        for (auto& f : listing[i]) console.printf("  %s (%u B)\n", f.path.c_str(), (unsigned)f.bytes);
     }
 }
 
@@ -258,16 +270,50 @@ static void uploadStep() {
     }
 }
 
-static void serialCommands() {
+static const char* const kHelp =
-    if (upload.active()) return readUploadBytes();  // raw file bytes, not commands
+    "info              firmware, uptime, memory, Wi-Fi, app slots\n"
-    static String line;
+    "tasks             FreeRTOS tasks: state, priority, free stack, CPU\n"
-    while (Serial.available()) {
+    "reboot            restart\n"
-        char c = Serial.read();
+    "boot other        restart into the other app slot (manual Rollback)\n"
-        if (c != '\n') {
+    "log level <0-5>   ESP-IDF log level (0 none ... 5 verbose)\n"
-            line += c;
+    "key <name|char>   press a key: up down left right select back home, or one character\n"
-            continue;
+    "wifi status | wifi add <ssid><TAB><password>\n"
-        }
+    "irc start | irc dump | irc say <buffer> <text>\n"
+    "sd list | cat <path> | log <text> | burst | sound on|off | short | normal\n"
+#ifdef RORO_DEBUG
+    "crash abort|wdt   crash on purpose (to test crash reports and Safe Mode)\n"
+    "quit              close the Debug Console connection\n"
+#endif
+    ;
+
+static void runCommand(String line) {
     line.trim();
+    if (line.isEmpty()) return;
+    if (line == "help") console.print(kHelp);
+    if (line == "info") {
+        system_info::printSystem(console);
+        console.printf("wifi: %s, ip %s, rssi %d | sd: %s\n", wifi->ssid().c_str(), wifi->ip().c_str(), wifi->rssi(),
+                       storageService->state().present ? "present" : "none");
+        console.printf("update: %s\n", update->onProbation() ? "on probation" : "confirmed");
+        system_info::printSlots(console, nvs);
+    }
+    if (line == "tasks") system_info::printTasks(console);
+    if (line == "reboot") {
+        console.println("restarting");
+        delay(300);
+        ESP.restart();
+    }
+    if (line == "boot other") console.printf("boot other: %s\n", system_info::bootOtherSlot());
+    if (line.startsWith("log level ")) {
+        int level = line.substring(10).toInt();
+        esp_log_level_set("*", static_cast<esp_log_level_t>(constrain(level, 0, 5)));
+        console.printf("log level: %d\n", constrain(level, 0, 5));
+    }
+#ifdef RORO_DEBUG
+    if (line == "crash abort") abort();
+    if (line == "crash wdt")
+        for (;;) {}  // the main loop never yields: the task watchdog fires
+#endif
     if (line == "burst")
         for (int i = 1; i <= 5; i++)
             bus.publish(Event::withText(EventType::Notification, ("Burst " + String(i)).c_str(), 0));
@@ -283,23 +329,23 @@ static void serialCommands() {
         std::string ssid = (tab > 0 ? line.substring(9, tab) : line.substring(9)).c_str();
         std::string pass = tab > 0 ? line.substring(tab + 1).c_str() : "";
         std::string error = savedNetworks->add(ssid, pass);
-            Serial.printf("wifi add: %s\n", error.empty() ? "ok" : error.c_str());
+        console.printf("wifi add: %s\n", error.empty() ? "ok" : error.c_str());
         wifi->savedNetworksChanged();
     }
-        if (line.startsWith("log ")) {  // appends to a test IRC Log
+    if (line.startsWith("log ") && !line.startsWith("log level ")) {  // appends to a test IRC Log
         std::string path = storage::dailyLogPath({"irc", "dev", "#test"}, clockService->localDate());
         storageService->appendLine(path, clockService->displayTime() + " " + line.substring(4).c_str());
-            Serial.printf("log: queued to %s\n", path.c_str());
+        console.printf("log: queued to %s\n", path.c_str());
     }
     if (line.startsWith("cat ")) {
         std::string path = line.substring(4).c_str();
         File f = SD.open(path.c_str());
-            if (!f) Serial.printf("cat: cannot open %s\n", path.c_str());
+        if (!f) console.printf("cat: cannot open %s\n", path.c_str());
         else {
-                Serial.printf("--- %s (%u B) ---\n", path.c_str(), (unsigned)f.size());
+            console.printf("--- %s (%u B) ---\n", path.c_str(), (unsigned)f.size());
             int shown = 0;
-                while (f.available() && shown < 1200) { Serial.write(f.read()); shown++; }
+            while (f.available() && shown < 1200) { console.write(f.read()); shown++; }
-                Serial.println("\n--- end ---");
+            console.println("\n--- end ---");
             f.close();
         }
     }
@@ -317,24 +363,24 @@ static void serialCommands() {
         irc->withSession([&](IrcSession& s) { s.input(b, text, clockService->utcNow()); });
     }
     if (line == "irc dump") {
-            Serial.printf("irc: status %d, unread %d, heap %u min %u\n", (int)irc->status(), irc->totalUnread(),
+        console.printf("irc: status %d, unread %d, heap %u min %u\n", (int)irc->status(), irc->totalUnread(),
                        ESP.getFreeHeap(), ESP.getMinFreeHeap());
         IrcConfig c = irc->draftConfig();  // which login is configured, never the secrets
-            Serial.printf("irc: nick %s, sasl %s, nickserv %s, autojoin %u channels\n", c.nick.c_str(),
+        console.printf("irc: nick %s, sasl %s, nickserv %s, autojoin %u channels\n", c.nick.c_str(),
                        c.saslUser.empty() || c.saslPassword.empty() ? "off" : "on",
                        c.nickservPassword.empty() ? "off" : "on", (unsigned)c.autojoin.size());
         irc->withSession([](IrcSession& s) {
             for (int i = 0; i < s.bufferCount(); i++) {
                 const auto& b = s.buffer(i);
-                    Serial.printf("irc buffer %d %s unread %d%s\n", i, b.name.c_str(), b.unread, b.joined ? " joined" : "");
+                console.printf("irc buffer %d %s unread %d%s\n", i, b.name.c_str(), b.unread, b.joined ? " joined" : "");
                 size_t from = 0;
                 for (size_t j = from; j < b.lines.size(); j++)
-                        Serial.printf("   <%s> %s\n", b.lines[j].nick.c_str(), b.lines[j].text.c_str());
+                    console.printf("   <%s> %s\n", b.lines[j].nick.c_str(), b.lines[j].text.c_str());
             }
         });
     }
     if (line == "wifi status")
-            Serial.printf("wifi: state %d ssid '%s' rssi %d ip %s clock %s heap %u min %u | fw %s%s\n",
+        console.printf("wifi: state %d ssid '%s' rssi %d ip %s clock %s heap %u min %u | fw %s%s\n",
                        (int)wifi->state(), wifi->ssid().c_str(), wifi->rssi(), wifi->ip().c_str(),
                        clockService->displayTime().c_str(), ESP.getFreeHeap(), ESP.getMinFreeHeap(), versionString(),
                        update->onProbation() ? " (on probation)" : "");
@@ -348,6 +394,18 @@ static void serialCommands() {
         settings.setInt(Setting::OffTimeoutS, 60);
         settings.setInt(Setting::DimTimeoutS, 30);
     }
+}
+
+static void serialCommands() {
+    if (upload.active()) return readUploadBytes();  // raw file bytes, not commands
+    static String line;
+    while (Serial.available()) {
+        char c = Serial.read();
+        if (c != '\n') {
+            line += c;
+            continue;
+        }
+        runCommand(line);
         line = "";
     }
 }
@@ -362,6 +420,12 @@ void loop() {
     uint32_t now = millis();
 
     serialCommands();
+#ifdef RORO_DEBUG
+    for (std::string remote; debugConsole->takeCommand(remote);) {
+        console.printf("> %s\n", remote.c_str());  // so the transcript reads the same on both ends
+        runCommand(remote.c_str());
+    }
+#endif
     uploadStep();
     printListingWhenReady();
     M5Cardputer.update();
@@ -380,7 +444,7 @@ void loop() {
 
     if (now - lastLog > 10000) {
         lastLog = now;
-        Serial.printf("status: heap %u min %u\n", ESP.getFreeHeap(), ESP.getMinFreeHeap());
+        console.printf("status: heap %u min %u\n", ESP.getFreeHeap(), ESP.getMinFreeHeap());
     }
 
     // Firmware Update: its progress takes over the screen, then the device restarts when it's safe.

src/platform/console.cpp

@@ -0,0 +1,86 @@
+#include "platform/console.h"
+
+#include <Arduino.h>
+
+#ifdef RORO_DEBUG
+#include <esp_log.h>
+#include <freertos/FreeRTOS.h>
+
+#include <algorithm>
+#include <cstdio>
+#endif
+
+namespace roro {
+
+Console console;
+
+#ifdef RORO_DEBUG
+namespace {
+
+// Any task may write (ESP-IDF logs come from everywhere), so the ring is behind a spinlock, held
+// only for the copy.
+portMUX_TYPE ringLock = portMUX_INITIALIZER_UNLOCKED;
+vprintf_like_t espLogNext = nullptr;
+
+int teeEspLog(const char* format, va_list args) {
+    char line[200];
+    va_list copy;
+    va_copy(copy, args);
+    int n = vsnprintf(line, sizeof line, format, copy);
+    va_end(copy);
+    if (n > 0) console.toRing(reinterpret_cast<const uint8_t*>(line), std::min<size_t>(n, sizeof line - 1));
+    return espLogNext ? espLogNext(format, args) : n;  // the serial port, as before
+}
+
+}  // namespace
+
+void Console::captureEspLogs() { espLogNext = esp_log_set_vprintf(teeEspLog); }
+
+void Console::toRing(const uint8_t* data, size_t len) {
+    if (len > kRingBytes) {
+        data += len - kRingBytes;  // only the tail can fit
+        len = kRingBytes;
+    }
+    portENTER_CRITICAL(&ringLock);
+    size_t at = head_ % kRingBytes;
+    size_t first = std::min(len, kRingBytes - at);
+    memcpy(ring_ + at, data, first);
+    memcpy(ring_, data + first, len - first);
+    head_ += len;
+    portEXIT_CRITICAL(&ringLock);
+}
+
+uint32_t Console::oldest() const {
+    portENTER_CRITICAL(&ringLock);
+    uint32_t pos = head_ > kRingBytes ? head_ - kRingBytes : 0;
+    portEXIT_CRITICAL(&ringLock);
+    return pos;
+}
+
+size_t Console::readSince(uint32_t& pos, uint8_t* out, size_t max, uint32_t& skipped) {
+    portENTER_CRITICAL(&ringLock);
+    uint32_t from = head_ > kRingBytes ? head_ - kRingBytes : 0;
+    skipped = pos < from ? from - pos : 0;
+    if (pos < from) pos = from;
+    size_t n = std::min<size_t>(max, head_ - pos);
+    size_t at = pos % kRingBytes;
+    size_t first = std::min(n, kRingBytes - at);
+    memcpy(out, ring_ + at, first);
+    memcpy(out + first, ring_, n - first);
+    pos += n;
+    portEXIT_CRITICAL(&ringLock);
+    return n;
+}
+#endif
+
+size_t Console::write(const uint8_t* data, size_t len) {
+#ifdef RORO_DEBUG
+    toRing(data, len);
+#endif
+    // Never wait for the USB host. One that's attached but not reading (a VM, a closed terminal)
+    // would stall the caller for up to 2 s per write while HWCDC retries; the ring keeps it anyway.
+    if (Serial.availableForWrite() >= static_cast<int>(len)) Serial.write(data, len);
+    return len;
+}
+
+}  // namespace roro

src/platform/console.h

@@ -0,0 +1,40 @@
+#pragma once
+
+#include <Print.h>
+
+#include <cstddef>
+#include <cstdint>
+
+namespace roro {
+
+// Where the firmware's console output goes: the USB serial port, and in a Debug Build also a ring
+// buffer the Debug Console sends over Wi-Fi (with what came before the connection, so boot messages
+// aren't lost). Use `console` instead of Serial for anything a human should be able to read remotely.
+class Console : public Print {
+   public:
+    size_t write(uint8_t c) override { return write(&c, 1); }
+    size_t write(const uint8_t* data, size_t len) override;
+    using Print::write;
+
+#ifdef RORO_DEBUG
+    static constexpr size_t kRingBytes = 6144;
+
+    // Also copies ESP-IDF's own log lines into the ring (they still reach the serial port).
+    void captureEspLogs();
+    // Copies bytes written since `pos` into `out`, and advances `pos`. A reader that fell more than
+    // the ring behind gets the oldest bytes still there, and `skipped` says how many it missed.
+    size_t readSince(uint32_t& pos, uint8_t* out, size_t max, uint32_t& skipped);
+    // The position of the oldest byte still in the ring: a new reader starts there.
+    uint32_t oldest() const;
+    // The ring only, for output that already reaches the serial port another way.
+    void toRing(const uint8_t* data, size_t len);
+
+   private:
+    uint8_t ring_[kRingBytes];
+    uint32_t head_ = 0;  // total bytes ever written; the ring holds the last kRingBytes of them
+#endif
+};
+
+extern Console console;
+
+}  // namespace roro

src/platform/system_info.cpp

@@ -0,0 +1,118 @@
+#include "platform/system_info.h"
+
+#include <Arduino.h>
+#include <esp_heap_caps.h>
+#include <esp_ota_ops.h>
+#include <esp_system.h>
+#include <freertos/FreeRTOS.h>
+#include <freertos/task.h>
+
+#include <string>
+#include <vector>
+
+#include "version.h"
+
+namespace roro::system_info {
+
+const char* resetReason() {
+    switch (esp_reset_reason()) {
+        case ESP_RST_POWERON: return "power-on";
+        case ESP_RST_EXT: return "reset pin";
+        case ESP_RST_SW: return "restart";
+        case ESP_RST_PANIC: return "panic";
+        case ESP_RST_INT_WDT: return "interrupt watchdog";
+        case ESP_RST_TASK_WDT: return "task watchdog";
+        case ESP_RST_WDT: return "watchdog";
+        case ESP_RST_DEEPSLEEP: return "deep sleep";
+        case ESP_RST_BROWNOUT: return "brownout";
+        case ESP_RST_SDIO: return "sdio";
+        case ESP_RST_USB: return "usb";
+        case ESP_RST_JTAG: return "jtag";
+        default: return "unknown";
+    }
+}
+
+bool resetWasCrash() {
+    switch (esp_reset_reason()) {
+        case ESP_RST_PANIC:
+        case ESP_RST_INT_WDT:
+        case ESP_RST_TASK_WDT:
+        case ESP_RST_WDT: return true;
+        default: return false;
+    }
+}
+
+static const char* stateName(esp_ota_img_states_t s) {
+    switch (s) {
+        case ESP_OTA_IMG_NEW: return "new";
+        case ESP_OTA_IMG_PENDING_VERIFY: return "pending verify";
+        case ESP_OTA_IMG_VALID: return "valid";
+        case ESP_OTA_IMG_INVALID: return "invalid";
+        case ESP_OTA_IMG_ABORTED: return "aborted";
+        default: return "undefined";
+    }
+}
+
+void printSystem(Print& out) {
+    uint32_t s = millis() / 1000;
+    out.printf("firmware: %s %s\n", kProductName, versionString());
+    out.printf("uptime: %luh%02lum%02lus, last start: %s\n", (unsigned long)(s / 3600), (unsigned long)(s / 60 % 60),
+               (unsigned long)(s % 60), resetReason());
+    out.printf("heap: %u free, %u lowest, %u largest block\n", ESP.getFreeHeap(), ESP.getMinFreeHeap(),
+               (unsigned)heap_caps_get_largest_free_block(MALLOC_CAP_8BIT));
+    out.printf("chip: %s rev %d, %lu MHz, %.1f C\n", ESP.getChipModel(), ESP.getChipRevision(),
+               (unsigned long)ESP.getCpuFreqMHz(), temperatureRead());
+}
+
+static std::string slotKey(const esp_partition_t* slot) { return std::string("ver_") + slot->label; }
+
+void recordSlotVersion(KeyValueStore& store, const esp_partition_t* slot, const char* version) {
+    std::string known;
+    if (slot && !(store.getString(slotKey(slot).c_str(), known) && known == version))
+        store.putString(slotKey(slot).c_str(), version);
+}
+
+void printSlots(Print& out, KeyValueStore& store) {
+    const esp_partition_t* running = esp_ota_get_running_partition();
+    const esp_partition_t* next = esp_ota_get_boot_partition();
+    esp_partition_iterator_t it = esp_partition_find(ESP_PARTITION_TYPE_APP, ESP_PARTITION_SUBTYPE_ANY, nullptr);
+    for (; it; it = esp_partition_next(it)) {  // the last next() releases the iterator
+        const esp_partition_t* p = esp_partition_get(it);
+        esp_app_desc_t desc;
+        bool hasApp = esp_ota_get_partition_description(p, &desc) == ESP_OK;
+        std::string version = "(unknown version)";
+        store.getString(slotKey(p).c_str(), version);
+        esp_ota_img_states_t state;
+        bool hasState = esp_ota_get_state_partition(p, &state) == ESP_OK;
+        out.printf("slot %s: %s%s%s, %s\n", p->label, hasApp ? version.c_str() : "(empty)", p == running ? ", running" : "",
+                   p == next ? ", boots next" : "", hasState ? stateName(state) : "no OTA state");
+    }
+}
+
+void printTasks(Print& out) {
+    UBaseType_t n = uxTaskGetNumberOfTasks();
+    std::vector<TaskStatus_t> tasks(n + 4);
+    uint32_t total = 0;
+    n = uxTaskGetSystemState(tasks.data(), tasks.size(), &total);
+    out.printf("%-16s %-4s %3s %6s %5s %s\n", "task", "st", "pri", "stack", "cpu%", "core");
+    static const char kStates[] = "RrBSD";  // running, ready, blocked, suspended, deleted
+    for (UBaseType_t i = 0; i < n; i++) {
+        const TaskStatus_t& t = tasks[i];
+        unsigned cpu = total ? (unsigned)((uint64_t)t.ulRunTimeCounter * 100 / total) : 0;
+        int core = t.xCoreID == tskNO_AFFINITY ? -1 : (int)t.xCoreID;
+        out.printf("%-16s %-4c %3u %6u %5u %d\n", t.pcTaskName, t.eCurrentState < 5 ? kStates[t.eCurrentState] : '?',
+                   (unsigned)t.uxCurrentPriority, (unsigned)t.usStackHighWaterMark, cpu, core);
+    }
+}
+
+const char* bootOtherSlot() {
+    const esp_partition_t* other = esp_ota_get_next_update_partition(nullptr);
+    esp_app_desc_t desc;
+    if (!other || esp_ota_get_partition_description(other, &desc) != ESP_OK) return "the other slot is empty";
+    if (esp_ota_set_boot_partition(other) != ESP_OK) return "the other slot doesn't hold a valid image";
+    delay(200);
+    esp_restart();
+    return "";
+}
+
+}  // namespace roro::system_info

src/platform/system_info.h

@@ -0,0 +1,29 @@
+#pragma once
+
+#include <Print.h>
+#include <esp_partition.h>
+
+#include "key_value_store.h"
+
+namespace roro::system_info {
+
+// Why the chip last started: "power-on", "panic", "task watchdog", "brownout"...
+const char* resetReason();
+// True if the last start was a crash or a watchdog, not a power-on or a deliberate restart.
+bool resetWasCrash();
+
+// Firmware, uptime and memory, one fact per line.
+void printSystem(Print& out);
+// Both app slots: which one runs, which boots next, and each one's version and OTA state. Versions
+// come from NVS ("ver_<slot>"): the prebuilt framework stamps every image with its own version.
+void printSlots(Print& out, KeyValueStore& store);
+// Remembers which version the running slot holds (call at boot), or the slot an update just wrote.
+void recordSlotVersion(KeyValueStore& store, const esp_partition_t* slot, const char* version);
+// FreeRTOS tasks: state, priority, lowest free stack, CPU share since boot.
+void printTasks(Print& out);
+
+// Boots the firmware in the other app slot if it holds a valid image (a manual Rollback).
+// Returns an error message; on success it restarts and doesn't return.
+const char* bootOtherSlot();
+
+}  // namespace roro::system_info

src/services/debug_console.cpp

@@ -0,0 +1,143 @@
+#ifdef RORO_DEBUG
+
+#include "services/debug_console.h"
+
+#include <WiFi.h>
+
+#include "platform/console.h"
+#include "version.h"
+
+namespace roro {
+
+namespace {
+
+constexpr uint32_t kAuthTimeoutMs = 10000;
+constexpr size_t kMaxLine = 240;
+constexpr size_t kMaxQueued = 8;
+
+// Compares without stopping at the first difference, so timing says nothing about the token.
+bool sameToken(const std::string& a, const char* b) {
+    size_t n = strlen(b);
+    uint8_t diff = a.size() != n;
+    for (size_t i = 0; i < n; i++) diff |= (i < a.size() ? a[i] : 0) ^ b[i];
+    return diff == 0;
+}
+
+// Reads one line (without its \r\n) within `timeoutMs`; false on a timeout, a drop or an overlong line.
+bool readLine(NetworkClient& c, std::string& line, uint32_t timeoutMs) {
+    line.clear();
+    uint32_t since = millis();
+    while (c.connected() && millis() - since < timeoutMs) {
+        int ch = c.read();
+        if (ch < 0) {
+            delay(10);
+            continue;
+        }
+        if (ch == '\n') {
+            if (!line.empty() && line.back() == '\r') line.pop_back();
+            return true;
+        }
+        if (line.size() >= kMaxLine) return false;
+        line += static_cast<char>(ch);
+    }
+    return false;
+}
+
+}  // namespace
+
+DebugConsole::DebugConsole(WifiService& wifi) : wifi_(wifi), lock_(xSemaphoreCreateMutex()) {}
+
+void DebugConsole::start() {
+    console.captureEspLogs();
+    if (!task_) xTaskCreate(taskEntry, "debug", 6144, this, 1, &task_);
+}
+
+bool DebugConsole::takeCommand(std::string& line) {
+    xSemaphoreTake(lock_, portMAX_DELAY);
+    bool any = !commands_.empty();
+    if (any) {
+        line = std::move(commands_.front());
+        commands_.pop_front();
+    }
+    xSemaphoreGive(lock_);
+    return any;
+}
+
+void DebugConsole::taskEntry(void* self) { static_cast<DebugConsole*>(self)->listen(); }
+
+void DebugConsole::listen() {
+    NetworkServer server(kPort);
+    bool listening = false;
+    for (;;) {
+        bool up = wifi_.state() == WifiController::State::Connected;
+        if (up && !listening) {
+            server.begin();
+            listening = true;
+        } else if (!up && listening) {
+            server.end();
+            listening = false;
+        }
+        if (listening) {
+            NetworkClient client = server.accept();
+            if (client) {
+                client.setNoDelay(true);
+                if (authenticate(client)) serve(client);
+                client.stop();
+            }
+        }
+        vTaskDelay(pdMS_TO_TICKS(200));
+    }
+}
+
+bool DebugConsole::authenticate(NetworkClient& client) {
+    std::string token;
+    if (readLine(client, token, kAuthTimeoutMs) && sameToken(token, RORO_DEBUG_TOKEN)) return true;
+    console.printf("debug: refused a client from %s\n", client.remoteIP().toString().c_str());
+    delay(1000);  // no quick retries
+    client.print("denied\n");
+    return false;
+}
+
+void DebugConsole::serve(NetworkClient& client) {
+    console.printf("debug: client %s connected\n", client.remoteIP().toString().c_str());
+    client.printf("%s %s debug console. 'help' lists the commands. Backlog follows.\n", kProductName, versionString());
+    connected_ = true;
+    uint32_t pos = console.oldest();
+    uint8_t buf[512];
+    std::string line;
+    while (client.connected()) {
+        // Console output since last time, including the replies to this client's commands.
+        uint32_t skipped = 0;
+        size_t n;
+        while ((n = console.readSince(pos, buf, sizeof buf, skipped)) > 0) {
+            if (skipped) client.printf("\n[... %u bytes lost: the console ran faster than the network]\n", (unsigned)skipped);
+            if (client.write(buf, n) != n) break;
+        }
+        // Commands, one line at a time.
+        while (client.available()) {
+            int ch = client.read();
+            if (ch == '\r') continue;
+            if (ch != '\n') {
+                if (line.size() < kMaxLine) line += static_cast<char>(ch);
+                continue;
+            }
+            if (line == "quit" || line == "exit") {
+                client.stop();
+                break;
+            }
+            xSemaphoreTake(lock_, portMAX_DELAY);
+            bool full = commands_.size() >= kMaxQueued;
+            if (!full && !line.empty()) commands_.push_back(line);
+            xSemaphoreGive(lock_);
+            if (full) client.print("debug: busy, command dropped\n");
+            line.clear();
+        }
+        vTaskDelay(pdMS_TO_TICKS(20));
+    }
+    connected_ = false;
+    console.println("debug: client disconnected");
+}
+
+}  // namespace roro
+
+#endif

src/services/debug_console.h

@@ -0,0 +1,52 @@
+#pragma once
+
+#ifdef RORO_DEBUG
+
+#include <freertos/FreeRTOS.h>
+#include <freertos/semphr.h>
+
+#include <deque>
+#include <string>
+
+#include "service.h"
+#include "services/wifi_service.h"
+
+class NetworkClient;
+
+namespace roro {
+
+// Debug Builds only (ADR 0004): the console over Wi-Fi, on TCP 2323 while Wi-Fi is Connected. A
+// client sends the debug token as its first line, then gets the recent console backlog, every new
+// console line, and runs the same commands as the serial port. One client at a time.
+//
+// The socket lives on this Service's own task; commands are handed to the main loop (takeCommand),
+// which runs them where touching Apps and Services is safe. Their output reaches the client
+// through the console ring like everything else.
+class DebugConsole : public Service {
+   public:
+    static constexpr uint16_t kPort = 2323;
+
+    explicit DebugConsole(WifiService& wifi);
+    const char* name() const override { return "debug"; }
+    void start() override;
+
+    // The next command line a client sent, for the main loop to run.
+    bool takeCommand(std::string& line);
+    bool clientConnected() const { return connected_; }
+
+   private:
+    static void taskEntry(void* self);
+    void listen();
+    void serve(::NetworkClient& client);
+    bool authenticate(::NetworkClient& client);
+
+    WifiService& wifi_;
+    TaskHandle_t task_ = nullptr;
+    SemaphoreHandle_t lock_;
+    std::deque<std::string> commands_;
+    volatile bool connected_ = false;
+};
+
+}  // namespace roro
+
+#endif

src/services/update_service.cpp

@@ -9,6 +9,7 @@
 
 #include "platform/identity.h"
 #include "platform/ota_device.h"
+#include "platform/system_info.h"
 #include "probation.h"
 #include "update_parser.h"
 #include "version.h"
@@ -95,6 +96,7 @@ void UpdateService::bootGuard(KeyValueStore& store) {
     esp_ota_img_states_t state;
     bool probation = esp_ota_get_state_partition(esp_ota_get_running_partition(), &state) == ESP_OK &&
                      state == ESP_OTA_IMG_PENDING_VERIFY;
+    system_info::recordSlotVersion(store, esp_ota_get_running_partition(), versionString());
     int32_t attempts = 0;
     store.getInt("ota_attempts", attempts);
     if (Probation::rollBackAtBoot(probation, attempts)) {
@@ -156,6 +158,7 @@ void UpdateService::install(UpdateSource& source, const char* via) {
 
     if (ok) {
         store_.putString("ota_pending", parser.version());
+        system_info::recordSlotVersion(store_, esp_ota_get_boot_partition(), parser.version().c_str());
         store_.putString("ota_from", versionString());
         source.reply("OK " + parser.version() + (parser.isDowngrade() ? " (older than the installed one)" : ""));
         notify("Update " + parser.version() + " installed (" + via + "), restarting", NotificationLevel::Info);