Debug Console: files, screenshots and Update from SD over Wi-Fi

New commands everywhere: ls, rm and install <path> (Update from SD
without the Firmware page), all as Storage Service jobs. In Debug
Builds the console task answers get and put (one storage job per
transfer, file kept open, TCP flow control: about 300 KB/s, against
55 KB/s over serial) and screenshot (the RGB332 frame the UI composes
into). rdbg.py turns those into files and PNGs.

A failed put closes the connection: the rest of the file had been
parsed as commands. Card writes are retried 3 times after closing,
truncating to the last good byte and reopening, since FATFS keeps a
file in error after one failed write (seen once at 1.3 MB on this card).
onStorage() decides with one compare-and-swap whether the job or the
timeout wins, so an abandoned job can't touch a returned stack frame.

Verified on the device: screenshot; a get round trip byte-identical;
4 puts in a row; a tampered .ota refused by install; a good one put,
installed from SD, confirmed on Probation. The retry path itself has
not fired since it was added.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT

README.md

@@ -74,6 +74,8 @@ To install from the SD card instead, copy the `.ota` file from `.pio/build/cardp
 | `tasks` | FreeRTOS tasks: state, priority, lowest free stack, CPU share |
 | `reboot` / `boot other` | Restart, or restart into the other app slot (a manual Rollback) |
 | `log level <0-5>` | ESP-IDF log level |
+| `ls [folder]` / `rm <path>` | Lists a folder of the SD card, or deletes a file |
+| `install <path>` | Update from SD with that `.ota` file, as Settings → Firmware does |
 | `crash` | The last crash: which firmware, why, task, PC and backtrace (from the core dump in flash) |
 | `coredump erase` | Forgets the core dump |
 | `crash abort` / `crash wdt` | Debug Builds: crash on purpose, or hang the main loop until the watchdog fires |
@@ -97,8 +99,13 @@ Every command above works there too, plus a few handled by the PC side or the co
 scripts/rdbg.py crash           # the last crash, its backtrace decoded against that exact build's ELF
 scripts/rdbg.py coredump        # fetch the core dump and decode it all (registers, every task) with esp-coredump
 scripts/rdbg.py reset           # restart at once, even if the main loop is stuck
+scripts/rdbg.py screenshot      # the screen as a PNG (2x)
+scripts/rdbg.py put <file> [card path]    # to the SD card (default /updates/<name>), SHA-256 checked, ~300 KB/s
+scripts/rdbg.py get <card path> [file]    # from the SD card
 ```
 
+So a Firmware Update can also go `rdbg.py put roro9stack-….ota` then `rdbg.py install /updates/roro9stack-….ota`: the Update from SD path, without touching the device.
+
 Every build keeps its ELF in `.pio/elves/` (version and digest in the name) for that; `scripts/decode_backtrace.sh <version|digest> <addresses>` decodes any backtrace by hand.
 
 After 3 crash restarts in a row the firmware starts in **Safe Mode** (ADR 0005): only Wi-Fi, Firmware Updates and the Debug Console, so a fix can be pushed as usual. `reboot` leaves it. The token is in `~/.config/roro9stack/debug-token`, made by the first build; keep developing on Debug Builds, so the firmware a Rollback returns to always has the console.

docs/adr/0004-debug-console-in-debug-builds.md

@@ -10,6 +10,7 @@ It's compiled out of release builds entirely, rather than switched off by a sett
 - **Commands run on the main loop.** The socket lives on the Debug Console's own task, which only queues command lines. The main loop runs them, as it does serial commands, so they touch Apps and Services from the one task allowed to.
 - **The token** is 128 random bits in `~/.config/roro9stack/debug-token`, made by the first build and passed into the container. It's never committed; a Debug Build refuses to compile without one. Like the OTA key, it guards against the network, not against someone holding the device.
 - **One client at a time**, to keep memory flat (about 6 KB for the ring, 6 KB of task stack).
+- **Binary commands are answered on the console's own task**, not queued: `get`/`put` (SD card files, run as one Storage Service job each so card access stays on the storage task, with TCP doing the flow control), `screenshot` (the 32 KB RGB332 frame the UI composes into, read as it stands, so it may tear), `coredump get` and `reset`. These keep working when the main loop is stuck. A failed `put` closes the connection, so the rest of the file is never read as commands.
 
 ## Keep a Debug Build in the fallback slot
 

scripts/rdbg.py

@@ -8,14 +8,21 @@ Usage: scripts/rdbg.py [-H host] [-b] [command ...]
   -b           also print the backlog the device sends on connecting (boot messages and so on)
 
 Commands handled here as well as on the device:
-  crash             the last crash, with its backtrace decoded (scripts/decode_backtrace.sh)
-  coredump [file]   fetch the core dump (default core-<date>.bin) and decode it with esp-coredump
+  crash                    the last crash, with its backtrace decoded (scripts/decode_backtrace.sh)
+  coredump [file]          fetch the core dump (default core-<date>.bin) and decode it with esp-coredump
+  get <card path> [file]   copy a file from the SD card
+  put <file> [card path]   copy a file to the SD card (default /updates/<name>), checked with SHA-256
+  screenshot [file.png]    what the screen shows (default screen-<date>.png), at 2x
+  reset                    restart at once, even if the main loop is stuck
 The token is read from ~/.config/roro9stack/debug-token (made by the first build).
 """
+import hashlib
 import os
 import re
 import select
+import struct
 import subprocess
+import zlib
 import socket
 import sys
 import time
@@ -134,6 +141,77 @@ def coredump(sock, path):
         subprocess.call([os.path.join(SCRIPTS, "decode_coredump.sh"), path, key])
 
 
+def binary(sock, command, tag):
+    """Sends a binary command; returns (header words, payload) or exits with the device's error."""
+    sock.sendall((command + "\n").encode())
+    buf = b""
+    sock.settimeout(30)
+    pattern = re.compile(tag.encode() + rb": (data|ready|rgb332|error)([^\n]*)\n")
+    while not (m := pattern.search(buf)):
+        chunk = sock.recv(65536)
+        if not chunk:
+            sys.exit("device: closed the connection")
+        buf += chunk
+    if m.group(1) == b"error":
+        sys.exit(f"device: {tag}: error{m.group(2).decode()}")
+    return m.group(1).decode(), m.group(2).decode().split(), buf[m.end():]
+
+
+def receive(sock, have, size):
+    while len(have) < size:
+        chunk = sock.recv(65536)
+        if not chunk:
+            sys.exit(f"device: the connection closed after {len(have)} of {size} bytes")
+        have += chunk
+    return have[:size]
+
+
+def get(sock, remote, local):
+    _, words, rest = binary(sock, f"get {remote}", "get")
+    start, size = time.time(), int(words[0])
+    data = receive(sock, rest, size)
+    local = local or os.path.basename(remote)
+    open(local, "wb").write(data)
+    print(f"saved {local} ({size} bytes, {size / 1024 / max(time.time() - start, 0.001):.0f} KB/s)")
+
+
+def put(sock, local, remote):
+    data = open(local, "rb").read()
+    remote = remote or "/updates/" + os.path.basename(local)
+    digest = hashlib.sha256(data).hexdigest()
+    binary(sock, f"put {remote} {len(data)} {digest}", "put")
+    start = time.time()
+    sock.sendall(data)
+    answer = read_until(sock, b"\n", 30) or b"put: error no answer"
+    answer = answer.decode(errors="replace").strip().splitlines()[-1]
+    print(f"device: {answer} ({len(data) / 1024 / max(time.time() - start, 0.001):.0f} KB/s)")
+    if " error " in answer:
+        sys.exit(1)
+
+
+def png(path, width, height, rgb, scale):
+    rows = b""
+    for y in range(height):
+        row = b"".join(rgb[(y * width + x) * 3:(y * width + x) * 3 + 3] * scale for x in range(width))
+        rows += (b"\x00" + row) * scale
+    def chunk(kind, body):
+        return struct.pack(">I", len(body)) + kind + body + struct.pack(">I", zlib.crc32(kind + body))
+    header = struct.pack(">IIBBBBB", width * scale, height * scale, 8, 2, 0, 0, 0)
+    with open(path, "wb") as f:
+        f.write(b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", zlib.compress(rows)) + chunk(b"IEND", b""))
+
+
+def screenshot(sock, path):
+    _, words, rest = binary(sock, "screenshot", "screenshot")
+    width, height = int(words[0]), int(words[1])
+    pixels = receive(sock, rest, width * height)
+    # RGB332, as M5GFX stores an 8-bit sprite: RRRGGGBB.
+    rgb = b"".join(bytes(((v >> 5) * 255 // 7, ((v >> 2) & 7) * 255 // 7, (v & 3) * 255 // 3)) for v in pixels)
+    path = path or time.strftime("screen-%Y%m%d-%H%M%S.png")
+    png(path, width, height, rgb, 2)
+    print(f"saved {path} ({width * 2}x{height * 2})")
+
+
 def interactive(sock):
     sock.setblocking(False)
     while True:
@@ -181,6 +259,12 @@ def main():
             return interactive(sock)
         if args == ["crash"]:
             return crash(sock)
+        if args[0] == "get" and len(args) >= 2:
+            return get(sock, args[1], args[2] if len(args) > 2 else None)
+        if args[0] == "put" and len(args) >= 2:
+            return put(sock, args[1], args[2] if len(args) > 2 else None)
+        if args[0] == "screenshot":
+            return screenshot(sock, args[1] if len(args) > 1 else None)
         if args == ["reset"]:  # answered by the console's own task, not the main loop
             sock.sendall(b"reset\n")
             print((read_until(sock, b"restarting now\n", 10) or b"device: no answer").decode().strip().splitlines()[-1])

src/main.cpp

@@ -144,7 +144,7 @@ void setup() {
     services.add(*irc);
     services.add(*update);
 #ifdef RORO_DEBUG
-    debugConsole = new DebugConsole(*wifi);
+    debugConsole = new DebugConsole(*wifi, *storageService);
     services.add(*debugConsole);
 #endif
 
@@ -160,6 +160,9 @@ void setup() {
     bus.subscribe(EventType::Notification, [](const Event& e) { console.printf("notification: %s\n", e.text); });
 
     if (!screen.begin()) console.println("frame buffer allocation failed");
+#ifdef RORO_DEBUG
+    debugConsole->setFrame(screen.canvas());  // for `screenshot`
+#endif
     services.startAll(millis());
     apps->begin();
     if (!settings.getBool(Setting::SetupDone)) apps->openModal("setup");
@@ -188,11 +191,14 @@ static void setupSafeMode(int crashes) {
     services.add(*wifi);
     services.add(*update);
 #ifdef RORO_DEBUG
-    debugConsole = new DebugConsole(*wifi);
+    debugConsole = new DebugConsole(*wifi, *storageService);
     services.add(*debugConsole);
 #endif
     bus.subscribe(EventType::Notification, [](const Event& e) { console.printf("notification: %s\n", e.text); });
     screen.begin();
+#ifdef RORO_DEBUG
+    debugConsole->setFrame(screen.canvas());
+#endif
     services.startAll(millis());
     console.printf("%s %s in SAFE MODE: %d crash restarts in a row. Only Wi-Fi and Firmware Updates run.\n",
                    kProductName, versionString(), crashes);
@@ -322,11 +328,13 @@ static const char* const kHelp =
     "key <name|char>   press a key: up down left right select back home, or one character\n"
     "wifi status | wifi add <ssid><TAB><password>\n"
     "irc start | irc dump | irc say <buffer> <text>\n"
+    "ls [folder] | rm <path> | install <path.ota> (Update from SD)\n"
     "sd list | cat <path> | log <text> | burst | sound on|off | short | normal\n"
 #ifdef RORO_DEBUG
     "crash abort|wdt   crash on purpose (to test crash reports and Safe Mode)\n"
     "coredump get      (Debug Console only) send the raw core dump: use scripts/rdbg.py coredump\n"
     "reset             (Debug Console only) restart at once, even if the main loop is stuck\n"
+    "get <path> | put <path> <size> <sha256> | screenshot   (Debug Console only) binary, see rdbg.py\n"
     "quit              close the Debug Console connection\n"
 #endif
     ;
@@ -362,6 +370,20 @@ static void runCommand(String line) {
         esp_log_level_set("*", static_cast<esp_log_level_t>(constrain(level, 0, 5)));
         console.printf("log level: %d\n", constrain(level, 0, 5));
     }
+    if (line == "ls" || line.startsWith("ls ") || line.startsWith("rm ")) {
+        if (!storageService->state().present) return (void)console.println("sd: no card");
+        bool list = !line.startsWith("rm ");
+        std::string path = line.length() > 3 ? line.substring(3).c_str() : "/";
+        storageService->runJob([list, path]() {  // card access stays on the storage task
+            if (!list) return (void)console.printf("rm: %s %s\n", path.c_str(), SD.remove(path.c_str()) ? "removed" : "failed");
+            File dir = SD.open(path.c_str());
+            if (!dir || !dir.isDirectory()) return (void)console.printf("ls: %s is not a folder\n", path.c_str());
+            for (File f = dir.openNextFile(); f; f = dir.openNextFile())
+                console.printf("%10u  %s%s\n", f.isDirectory() ? 0u : (unsigned)f.size(), f.name(), f.isDirectory() ? "/" : "");
+            console.printf("ls: end of %s\n", path.c_str());
+        });
+    }
+    if (line.startsWith("install ")) update->installFromSd(line.substring(8).c_str());  // Update from SD
     if (line == "crash") crash_report::print(console, nvs);
     if (line == "coredump erase") console.println(crash_report::erase() ? "coredump: erased" : "coredump: nothing to erase");
 #ifdef RORO_DEBUG

src/services/debug_console.cpp

@@ -8,6 +8,13 @@
 #include <esp_core_dump.h>
 #include <esp_flash.h>
 
+#include <SD.h>
+#include <unistd.h>
+
+#include <atomic>
+#include <memory>
+
+#include "file_receiver.h"
 #include "platform/console.h"
 #include "version.h"
 
@@ -67,7 +74,8 @@ void sendCoreDump(NetworkClient& client) {
 
 }  // namespace
 
-DebugConsole::DebugConsole(WifiService& wifi) : wifi_(wifi), lock_(xSemaphoreCreateMutex()) {}
+DebugConsole::DebugConsole(WifiService& wifi, StorageService& storage)
+    : wifi_(wifi), storage_(storage), lock_(xSemaphoreCreateMutex()) {}
 
 void DebugConsole::start() {
     console.captureEspLogs();
@@ -147,14 +155,7 @@ void DebugConsole::serve(NetworkClient& client) {
                 client.stop();
                 break;
             }
-            if (line == "reset") {  // answered here: works even when the main loop is stuck
-                client.print("debug: restarting now\n");
-                client.flush();
-                delay(200);
-                esp_restart();
-            }
-            if (line == "coredump get") {  // binary: answered here, not by the main loop
-                sendCoreDump(client);
+            if (binaryCommand(client, line)) {
                 line.clear();
                 continue;
             }
@@ -171,6 +172,137 @@ void DebugConsole::serve(NetworkClient& client) {
     console.println("debug: client disconnected");
 }
 
+bool DebugConsole::binaryCommand(NetworkClient& client, const std::string& line) {
+    if (line == "reset") {  // works even when the main loop is stuck
+        client.print("debug: restarting now\n");
+        client.flush();
+        delay(200);
+        esp_restart();
+    }
+    if (line == "coredump get") sendCoreDump(client);
+    else if (line.rfind("get ", 0) == 0) get(client, line.substr(4));
+    else if (line.rfind("put ", 0) == 0) put(client, line.substr(4));
+    else if (line == "screenshot") screenshot(client);
+    else return false;
+    return true;
+}
+
+bool DebugConsole::onStorage(std::function<void()> job) {
+    // Shared with the job, which outlives this wait if the card is missing and it never starts.
+    // Exactly one side wins the state change: the job (Queued -> Running) or the wait giving up
+    // (Queued -> Abandoned), so an abandoned job can never touch this stack frame.
+    enum : int { Queued, Running, Abandoned };
+    struct Run {
+        std::atomic<int> state{Queued};
+        SemaphoreHandle_t done = xSemaphoreCreateBinary();
+        ~Run() { vSemaphoreDelete(done); }
+    };
+    auto run = std::make_shared<Run>();
+    storage_.runJob([run, job]() {
+        int expected = Queued;
+        if (!run->state.compare_exchange_strong(expected, Running)) return;
+        job();
+        xSemaphoreGive(run->done);
+    });
+    for (int waited = 0; xSemaphoreTake(run->done, pdMS_TO_TICKS(500)) != pdTRUE; waited += 500) {
+        int expected = Queued;
+        if (waited >= 5000 && run->state.compare_exchange_strong(expected, Abandoned)) return false;
+    }
+    return true;
+}
+
+void DebugConsole::get(NetworkClient& client, const std::string& path) {
+    bool ran = onStorage([&]() {
+        File f = SD.open(path.c_str());
+        if (!f || f.isDirectory()) return (void)client.printf("get: error cannot open %s\n", path.c_str());
+        size_t size = f.size();
+        client.printf("get: data %u\n", (unsigned)size);
+        uint8_t buf[1024];
+        for (size_t sent = 0; sent < size;) {
+            int n = f.read(buf, std::min(sizeof buf, size - sent));
+            if (n <= 0 || client.write(buf, n) != static_cast<size_t>(n)) break;  // the host sees it short
+            sent += n;
+        }
+        f.close();
+        client.print("get: end\n");
+    });
+    if (!ran) client.print("get: error no SD card\n");
+}
+
+// The same checks as `sd put` over serial (FileReceiver), but TCP does the flow control, so the
+// file is written in one job with the file kept open.
+void DebugConsole::put(NetworkClient& client, const std::string& args) {
+    FileReceiver r;
+    std::string error = r.begin(args, millis());
+    StorageState card = storage_.state();
+    if (error.empty() && !card.present) error = "no SD card";
+    if (error.empty() && card.totalBytes - card.usedBytes < r.size() + 64 * 1024) error = "not enough space";
+    if (!error.empty()) return (void)client.printf("put: error %s\n", error.c_str());
+
+    bool ran = onStorage([&]() {
+        std::string part = r.partPath();
+        for (size_t slash = part.find('/', 1); slash != std::string::npos; slash = part.find('/', slash + 1)) {
+            std::string dir = part.substr(0, slash);
+            if (!SD.exists(dir.c_str())) SD.mkdir(dir.c_str());
+        }
+        File f = SD.open(part.c_str(), FILE_WRITE);
+        if (!f) return (void)client.print("put: error card not writable\n");
+        client.printf("put: ready %u\n", (unsigned)r.size());
+        uint8_t buf[1024];
+        using S = FileReceiver::State;
+        while (r.state() == S::Receiving || r.state() == S::Writing) {
+            if (r.state() == S::Writing) {
+                const auto& c = r.chunk();
+                bool ok = f.write(c.data(), c.size()) == c.size();
+                // A card can fail one write and take the next. FATFS keeps a failed file in error,
+                // so: close, cut back to the last good byte, reopen, try again.
+                for (int retry = 1; !ok && retry <= 3; retry++) {
+                    console.printf("put: write failed at %u, retry %d\n", (unsigned)r.received(), retry);
+                    f.close();
+                    delay(50 * retry);
+                    truncate(("/sd" + part).c_str(), r.received());
+                    f = SD.open(part.c_str(), FILE_APPEND);
+                    ok = f && f.size() == r.received() && f.write(c.data(), c.size()) == c.size();
+                }
+                r.chunkWritten(ok, millis());
+                continue;
+            }
+            int n = client.read(buf, std::min(sizeof buf, r.wanted()));
+            if (n > 0) r.feed(buf, n, millis());
+            else if (!client.connected()) break;
+            else {
+                delay(2);
+                r.tick(millis());
+            }
+        }
+        f.close();
+        if (r.state() == S::Finishing) {
+            if (SD.exists(r.path().c_str())) SD.remove(r.path().c_str());
+            r.finished(SD.rename(part.c_str(), r.path().c_str()));
+        }
+        if (r.state() == S::Done) client.printf("put: done %s %u B\n", r.path().c_str(), (unsigned)r.size());
+        else {
+            SD.remove(part.c_str());
+            client.printf("put: error %s\n", r.error().empty() ? "connection lost" : r.error().c_str());
+            // The rest of the file is still on its way: never read it as commands.
+            client.flush();
+            client.stop();
+        }
+    });
+    if (!ran) client.print("put: error no SD card\n");
+}
+
+// The frame as composed off-screen (RGB332, one byte a pixel). Read while the UI may be drawing,
+// so it can tear; it's for looking, not for pixel-exact tests.
+void DebugConsole::screenshot(NetworkClient& client) {
+    const uint8_t* pixels = frame_ ? static_cast<const uint8_t*>(frame_->getBuffer()) : nullptr;
+    if (!pixels) return (void)client.print("screenshot: error no frame\n");
+    int w = frame_->width(), h = frame_->height();
+    client.printf("screenshot: rgb332 %d %d\n", w, h);
+    for (int y = 0; y < h; y += 16) client.write(pixels + y * w, w * std::min(16, h - y));
+    client.print("screenshot: end\n");
+}
+
 }  // namespace roro
 
 #endif

src/services/debug_console.h

@@ -6,10 +6,13 @@
 #include <freertos/semphr.h>
 
 #include <deque>
+#include <functional>
 #include <string>
 
 #include "service.h"
+#include "services/storage_service.h"
 #include "services/wifi_service.h"
+#include "ui/canvas.h"
 
 class NetworkClient;
 
@@ -26,7 +29,9 @@ class DebugConsole : public Service {
    public:
     static constexpr uint16_t kPort = 2323;
 
-    explicit DebugConsole(WifiService& wifi);
+    DebugConsole(WifiService& wifi, StorageService& storage);
+    // The off-screen frame the UI composes into: `screenshot` sends it as it stands.
+    void setFrame(Canvas& frame) { frame_ = &frame; }
     const char* name() const override { return "debug"; }
     void start() override;
 
@@ -39,8 +44,18 @@ class DebugConsole : public Service {
     void listen();
     void serve(::NetworkClient& client);
     bool authenticate(::NetworkClient& client);
+    // Binary commands, answered on this task: true if `line` was one.
+    bool binaryCommand(::NetworkClient& client, const std::string& line);
+    // Runs `job` on the storage task (where card access is safe) and waits for it. False if the
+    // card isn't mounted, in which case the storage task never runs it.
+    bool onStorage(std::function<void()> job);
+    void get(::NetworkClient& client, const std::string& path);
+    void put(::NetworkClient& client, const std::string& args);
+    void screenshot(::NetworkClient& client);
 
     WifiService& wifi_;
+    StorageService& storage_;
+    Canvas* frame_ = nullptr;
     TaskHandle_t task_ = nullptr;
     SemaphoreHandle_t lock_;
     std::deque<std::string> commands_;