diff --git a/README.md b/README.md index 71eb25a..a952dd6 100644 --- a/README.md +++ b/README.md @@ -74,6 +74,8 @@ To install from the SD card instead, copy the `.ota` file from `.pio/build/cardp | `tasks` | FreeRTOS tasks: state, priority, lowest free stack, CPU share | | `reboot` / `boot other` | Restart, or restart into the other app slot (a manual Rollback) | | `log level <0-5>` | ESP-IDF log level | +| `ls [folder]` / `rm ` | Lists a folder of the SD card, or deletes a file | +| `install ` | Update from SD with that `.ota` file, as Settings → Firmware does | | `crash` | The last crash: which firmware, why, task, PC and backtrace (from the core dump in flash) | | `coredump erase` | Forgets the core dump | | `crash abort` / `crash wdt` | Debug Builds: crash on purpose, or hang the main loop until the watchdog fires | @@ -97,8 +99,13 @@ Every command above works there too, plus a few handled by the PC side or the co scripts/rdbg.py crash # the last crash, its backtrace decoded against that exact build's ELF scripts/rdbg.py coredump # fetch the core dump and decode it all (registers, every task) with esp-coredump scripts/rdbg.py reset # restart at once, even if the main loop is stuck +scripts/rdbg.py screenshot # the screen as a PNG (2x) +scripts/rdbg.py put [card path] # to the SD card (default /updates/), SHA-256 checked, ~300 KB/s +scripts/rdbg.py get [file] # from the SD card ``` +So a Firmware Update can also go `rdbg.py put roro9stack-….ota` then `rdbg.py install /updates/roro9stack-….ota`: the Update from SD path, without touching the device. + Every build keeps its ELF in `.pio/elves/` (version and digest in the name) for that; `scripts/decode_backtrace.sh ` decodes any backtrace by hand. After 3 crash restarts in a row the firmware starts in **Safe Mode** (ADR 0005): only Wi-Fi, Firmware Updates and the Debug Console, so a fix can be pushed as usual. `reboot` leaves it. The token is in `~/.config/roro9stack/debug-token`, made by the first build; keep developing on Debug Builds, so the firmware a Rollback returns to always has the console. diff --git a/docs/adr/0004-debug-console-in-debug-builds.md b/docs/adr/0004-debug-console-in-debug-builds.md index 40901e8..7c942a7 100644 --- a/docs/adr/0004-debug-console-in-debug-builds.md +++ b/docs/adr/0004-debug-console-in-debug-builds.md @@ -10,6 +10,7 @@ It's compiled out of release builds entirely, rather than switched off by a sett - **Commands run on the main loop.** The socket lives on the Debug Console's own task, which only queues command lines. The main loop runs them, as it does serial commands, so they touch Apps and Services from the one task allowed to. - **The token** is 128 random bits in `~/.config/roro9stack/debug-token`, made by the first build and passed into the container. It's never committed; a Debug Build refuses to compile without one. Like the OTA key, it guards against the network, not against someone holding the device. - **One client at a time**, to keep memory flat (about 6 KB for the ring, 6 KB of task stack). +- **Binary commands are answered on the console's own task**, not queued: `get`/`put` (SD card files, run as one Storage Service job each so card access stays on the storage task, with TCP doing the flow control), `screenshot` (the 32 KB RGB332 frame the UI composes into, read as it stands, so it may tear), `coredump get` and `reset`. These keep working when the main loop is stuck. A failed `put` closes the connection, so the rest of the file is never read as commands. ## Keep a Debug Build in the fallback slot diff --git a/scripts/rdbg.py b/scripts/rdbg.py index 3ac918f..d75bba8 100755 --- a/scripts/rdbg.py +++ b/scripts/rdbg.py @@ -8,14 +8,21 @@ Usage: scripts/rdbg.py [-H host] [-b] [command ...] -b also print the backlog the device sends on connecting (boot messages and so on) Commands handled here as well as on the device: - crash the last crash, with its backtrace decoded (scripts/decode_backtrace.sh) - coredump [file] fetch the core dump (default core-.bin) and decode it with esp-coredump + crash the last crash, with its backtrace decoded (scripts/decode_backtrace.sh) + coredump [file] fetch the core dump (default core-.bin) and decode it with esp-coredump + get [file] copy a file from the SD card + put [card path] copy a file to the SD card (default /updates/), checked with SHA-256 + screenshot [file.png] what the screen shows (default screen-.png), at 2x + reset restart at once, even if the main loop is stuck The token is read from ~/.config/roro9stack/debug-token (made by the first build). """ +import hashlib import os import re import select +import struct import subprocess +import zlib import socket import sys import time @@ -134,6 +141,77 @@ def coredump(sock, path): subprocess.call([os.path.join(SCRIPTS, "decode_coredump.sh"), path, key]) +def binary(sock, command, tag): + """Sends a binary command; returns (header words, payload) or exits with the device's error.""" + sock.sendall((command + "\n").encode()) + buf = b"" + sock.settimeout(30) + pattern = re.compile(tag.encode() + rb": (data|ready|rgb332|error)([^\n]*)\n") + while not (m := pattern.search(buf)): + chunk = sock.recv(65536) + if not chunk: + sys.exit("device: closed the connection") + buf += chunk + if m.group(1) == b"error": + sys.exit(f"device: {tag}: error{m.group(2).decode()}") + return m.group(1).decode(), m.group(2).decode().split(), buf[m.end():] + + +def receive(sock, have, size): + while len(have) < size: + chunk = sock.recv(65536) + if not chunk: + sys.exit(f"device: the connection closed after {len(have)} of {size} bytes") + have += chunk + return have[:size] + + +def get(sock, remote, local): + _, words, rest = binary(sock, f"get {remote}", "get") + start, size = time.time(), int(words[0]) + data = receive(sock, rest, size) + local = local or os.path.basename(remote) + open(local, "wb").write(data) + print(f"saved {local} ({size} bytes, {size / 1024 / max(time.time() - start, 0.001):.0f} KB/s)") + + +def put(sock, local, remote): + data = open(local, "rb").read() + remote = remote or "/updates/" + os.path.basename(local) + digest = hashlib.sha256(data).hexdigest() + binary(sock, f"put {remote} {len(data)} {digest}", "put") + start = time.time() + sock.sendall(data) + answer = read_until(sock, b"\n", 30) or b"put: error no answer" + answer = answer.decode(errors="replace").strip().splitlines()[-1] + print(f"device: {answer} ({len(data) / 1024 / max(time.time() - start, 0.001):.0f} KB/s)") + if " error " in answer: + sys.exit(1) + + +def png(path, width, height, rgb, scale): + rows = b"" + for y in range(height): + row = b"".join(rgb[(y * width + x) * 3:(y * width + x) * 3 + 3] * scale for x in range(width)) + rows += (b"\x00" + row) * scale + def chunk(kind, body): + return struct.pack(">I", len(body)) + kind + body + struct.pack(">I", zlib.crc32(kind + body)) + header = struct.pack(">IIBBBBB", width * scale, height * scale, 8, 2, 0, 0, 0) + with open(path, "wb") as f: + f.write(b"\x89PNG\r\n\x1a\n" + chunk(b"IHDR", header) + chunk(b"IDAT", zlib.compress(rows)) + chunk(b"IEND", b"")) + + +def screenshot(sock, path): + _, words, rest = binary(sock, "screenshot", "screenshot") + width, height = int(words[0]), int(words[1]) + pixels = receive(sock, rest, width * height) + # RGB332, as M5GFX stores an 8-bit sprite: RRRGGGBB. + rgb = b"".join(bytes(((v >> 5) * 255 // 7, ((v >> 2) & 7) * 255 // 7, (v & 3) * 255 // 3)) for v in pixels) + path = path or time.strftime("screen-%Y%m%d-%H%M%S.png") + png(path, width, height, rgb, 2) + print(f"saved {path} ({width * 2}x{height * 2})") + + def interactive(sock): sock.setblocking(False) while True: @@ -181,6 +259,12 @@ def main(): return interactive(sock) if args == ["crash"]: return crash(sock) + if args[0] == "get" and len(args) >= 2: + return get(sock, args[1], args[2] if len(args) > 2 else None) + if args[0] == "put" and len(args) >= 2: + return put(sock, args[1], args[2] if len(args) > 2 else None) + if args[0] == "screenshot": + return screenshot(sock, args[1] if len(args) > 1 else None) if args == ["reset"]: # answered by the console's own task, not the main loop sock.sendall(b"reset\n") print((read_until(sock, b"restarting now\n", 10) or b"device: no answer").decode().strip().splitlines()[-1]) diff --git a/src/main.cpp b/src/main.cpp index d8b61ef..752b3ba 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -144,7 +144,7 @@ void setup() { services.add(*irc); services.add(*update); #ifdef RORO_DEBUG - debugConsole = new DebugConsole(*wifi); + debugConsole = new DebugConsole(*wifi, *storageService); services.add(*debugConsole); #endif @@ -160,6 +160,9 @@ void setup() { bus.subscribe(EventType::Notification, [](const Event& e) { console.printf("notification: %s\n", e.text); }); if (!screen.begin()) console.println("frame buffer allocation failed"); +#ifdef RORO_DEBUG + debugConsole->setFrame(screen.canvas()); // for `screenshot` +#endif services.startAll(millis()); apps->begin(); if (!settings.getBool(Setting::SetupDone)) apps->openModal("setup"); @@ -188,11 +191,14 @@ static void setupSafeMode(int crashes) { services.add(*wifi); services.add(*update); #ifdef RORO_DEBUG - debugConsole = new DebugConsole(*wifi); + debugConsole = new DebugConsole(*wifi, *storageService); services.add(*debugConsole); #endif bus.subscribe(EventType::Notification, [](const Event& e) { console.printf("notification: %s\n", e.text); }); screen.begin(); +#ifdef RORO_DEBUG + debugConsole->setFrame(screen.canvas()); +#endif services.startAll(millis()); console.printf("%s %s in SAFE MODE: %d crash restarts in a row. Only Wi-Fi and Firmware Updates run.\n", kProductName, versionString(), crashes); @@ -322,11 +328,13 @@ static const char* const kHelp = "key press a key: up down left right select back home, or one character\n" "wifi status | wifi add \n" "irc start | irc dump | irc say \n" + "ls [folder] | rm | install (Update from SD)\n" "sd list | cat | log | burst | sound on|off | short | normal\n" #ifdef RORO_DEBUG "crash abort|wdt crash on purpose (to test crash reports and Safe Mode)\n" "coredump get (Debug Console only) send the raw core dump: use scripts/rdbg.py coredump\n" "reset (Debug Console only) restart at once, even if the main loop is stuck\n" + "get | put | screenshot (Debug Console only) binary, see rdbg.py\n" "quit close the Debug Console connection\n" #endif ; @@ -362,6 +370,20 @@ static void runCommand(String line) { esp_log_level_set("*", static_cast(constrain(level, 0, 5))); console.printf("log level: %d\n", constrain(level, 0, 5)); } + if (line == "ls" || line.startsWith("ls ") || line.startsWith("rm ")) { + if (!storageService->state().present) return (void)console.println("sd: no card"); + bool list = !line.startsWith("rm "); + std::string path = line.length() > 3 ? line.substring(3).c_str() : "/"; + storageService->runJob([list, path]() { // card access stays on the storage task + if (!list) return (void)console.printf("rm: %s %s\n", path.c_str(), SD.remove(path.c_str()) ? "removed" : "failed"); + File dir = SD.open(path.c_str()); + if (!dir || !dir.isDirectory()) return (void)console.printf("ls: %s is not a folder\n", path.c_str()); + for (File f = dir.openNextFile(); f; f = dir.openNextFile()) + console.printf("%10u %s%s\n", f.isDirectory() ? 0u : (unsigned)f.size(), f.name(), f.isDirectory() ? "/" : ""); + console.printf("ls: end of %s\n", path.c_str()); + }); + } + if (line.startsWith("install ")) update->installFromSd(line.substring(8).c_str()); // Update from SD if (line == "crash") crash_report::print(console, nvs); if (line == "coredump erase") console.println(crash_report::erase() ? "coredump: erased" : "coredump: nothing to erase"); #ifdef RORO_DEBUG diff --git a/src/services/debug_console.cpp b/src/services/debug_console.cpp index 47e47ae..d2e3f64 100644 --- a/src/services/debug_console.cpp +++ b/src/services/debug_console.cpp @@ -8,6 +8,13 @@ #include #include +#include +#include + +#include +#include + +#include "file_receiver.h" #include "platform/console.h" #include "version.h" @@ -67,7 +74,8 @@ void sendCoreDump(NetworkClient& client) { } // namespace -DebugConsole::DebugConsole(WifiService& wifi) : wifi_(wifi), lock_(xSemaphoreCreateMutex()) {} +DebugConsole::DebugConsole(WifiService& wifi, StorageService& storage) + : wifi_(wifi), storage_(storage), lock_(xSemaphoreCreateMutex()) {} void DebugConsole::start() { console.captureEspLogs(); @@ -147,14 +155,7 @@ void DebugConsole::serve(NetworkClient& client) { client.stop(); break; } - if (line == "reset") { // answered here: works even when the main loop is stuck - client.print("debug: restarting now\n"); - client.flush(); - delay(200); - esp_restart(); - } - if (line == "coredump get") { // binary: answered here, not by the main loop - sendCoreDump(client); + if (binaryCommand(client, line)) { line.clear(); continue; } @@ -171,6 +172,137 @@ void DebugConsole::serve(NetworkClient& client) { console.println("debug: client disconnected"); } +bool DebugConsole::binaryCommand(NetworkClient& client, const std::string& line) { + if (line == "reset") { // works even when the main loop is stuck + client.print("debug: restarting now\n"); + client.flush(); + delay(200); + esp_restart(); + } + if (line == "coredump get") sendCoreDump(client); + else if (line.rfind("get ", 0) == 0) get(client, line.substr(4)); + else if (line.rfind("put ", 0) == 0) put(client, line.substr(4)); + else if (line == "screenshot") screenshot(client); + else return false; + return true; +} + +bool DebugConsole::onStorage(std::function job) { + // Shared with the job, which outlives this wait if the card is missing and it never starts. + // Exactly one side wins the state change: the job (Queued -> Running) or the wait giving up + // (Queued -> Abandoned), so an abandoned job can never touch this stack frame. + enum : int { Queued, Running, Abandoned }; + struct Run { + std::atomic state{Queued}; + SemaphoreHandle_t done = xSemaphoreCreateBinary(); + ~Run() { vSemaphoreDelete(done); } + }; + auto run = std::make_shared(); + storage_.runJob([run, job]() { + int expected = Queued; + if (!run->state.compare_exchange_strong(expected, Running)) return; + job(); + xSemaphoreGive(run->done); + }); + for (int waited = 0; xSemaphoreTake(run->done, pdMS_TO_TICKS(500)) != pdTRUE; waited += 500) { + int expected = Queued; + if (waited >= 5000 && run->state.compare_exchange_strong(expected, Abandoned)) return false; + } + return true; +} + +void DebugConsole::get(NetworkClient& client, const std::string& path) { + bool ran = onStorage([&]() { + File f = SD.open(path.c_str()); + if (!f || f.isDirectory()) return (void)client.printf("get: error cannot open %s\n", path.c_str()); + size_t size = f.size(); + client.printf("get: data %u\n", (unsigned)size); + uint8_t buf[1024]; + for (size_t sent = 0; sent < size;) { + int n = f.read(buf, std::min(sizeof buf, size - sent)); + if (n <= 0 || client.write(buf, n) != static_cast(n)) break; // the host sees it short + sent += n; + } + f.close(); + client.print("get: end\n"); + }); + if (!ran) client.print("get: error no SD card\n"); +} + +// The same checks as `sd put` over serial (FileReceiver), but TCP does the flow control, so the +// file is written in one job with the file kept open. +void DebugConsole::put(NetworkClient& client, const std::string& args) { + FileReceiver r; + std::string error = r.begin(args, millis()); + StorageState card = storage_.state(); + if (error.empty() && !card.present) error = "no SD card"; + if (error.empty() && card.totalBytes - card.usedBytes < r.size() + 64 * 1024) error = "not enough space"; + if (!error.empty()) return (void)client.printf("put: error %s\n", error.c_str()); + + bool ran = onStorage([&]() { + std::string part = r.partPath(); + for (size_t slash = part.find('/', 1); slash != std::string::npos; slash = part.find('/', slash + 1)) { + std::string dir = part.substr(0, slash); + if (!SD.exists(dir.c_str())) SD.mkdir(dir.c_str()); + } + File f = SD.open(part.c_str(), FILE_WRITE); + if (!f) return (void)client.print("put: error card not writable\n"); + client.printf("put: ready %u\n", (unsigned)r.size()); + uint8_t buf[1024]; + using S = FileReceiver::State; + while (r.state() == S::Receiving || r.state() == S::Writing) { + if (r.state() == S::Writing) { + const auto& c = r.chunk(); + bool ok = f.write(c.data(), c.size()) == c.size(); + // A card can fail one write and take the next. FATFS keeps a failed file in error, + // so: close, cut back to the last good byte, reopen, try again. + for (int retry = 1; !ok && retry <= 3; retry++) { + console.printf("put: write failed at %u, retry %d\n", (unsigned)r.received(), retry); + f.close(); + delay(50 * retry); + truncate(("/sd" + part).c_str(), r.received()); + f = SD.open(part.c_str(), FILE_APPEND); + ok = f && f.size() == r.received() && f.write(c.data(), c.size()) == c.size(); + } + r.chunkWritten(ok, millis()); + continue; + } + int n = client.read(buf, std::min(sizeof buf, r.wanted())); + if (n > 0) r.feed(buf, n, millis()); + else if (!client.connected()) break; + else { + delay(2); + r.tick(millis()); + } + } + f.close(); + if (r.state() == S::Finishing) { + if (SD.exists(r.path().c_str())) SD.remove(r.path().c_str()); + r.finished(SD.rename(part.c_str(), r.path().c_str())); + } + if (r.state() == S::Done) client.printf("put: done %s %u B\n", r.path().c_str(), (unsigned)r.size()); + else { + SD.remove(part.c_str()); + client.printf("put: error %s\n", r.error().empty() ? "connection lost" : r.error().c_str()); + // The rest of the file is still on its way: never read it as commands. + client.flush(); + client.stop(); + } + }); + if (!ran) client.print("put: error no SD card\n"); +} + +// The frame as composed off-screen (RGB332, one byte a pixel). Read while the UI may be drawing, +// so it can tear; it's for looking, not for pixel-exact tests. +void DebugConsole::screenshot(NetworkClient& client) { + const uint8_t* pixels = frame_ ? static_cast(frame_->getBuffer()) : nullptr; + if (!pixels) return (void)client.print("screenshot: error no frame\n"); + int w = frame_->width(), h = frame_->height(); + client.printf("screenshot: rgb332 %d %d\n", w, h); + for (int y = 0; y < h; y += 16) client.write(pixels + y * w, w * std::min(16, h - y)); + client.print("screenshot: end\n"); +} + } // namespace roro #endif diff --git a/src/services/debug_console.h b/src/services/debug_console.h index 9b4f23e..40dd5b3 100644 --- a/src/services/debug_console.h +++ b/src/services/debug_console.h @@ -6,10 +6,13 @@ #include #include +#include #include #include "service.h" +#include "services/storage_service.h" #include "services/wifi_service.h" +#include "ui/canvas.h" class NetworkClient; @@ -26,7 +29,9 @@ class DebugConsole : public Service { public: static constexpr uint16_t kPort = 2323; - explicit DebugConsole(WifiService& wifi); + DebugConsole(WifiService& wifi, StorageService& storage); + // The off-screen frame the UI composes into: `screenshot` sends it as it stands. + void setFrame(Canvas& frame) { frame_ = &frame; } const char* name() const override { return "debug"; } void start() override; @@ -39,8 +44,18 @@ class DebugConsole : public Service { void listen(); void serve(::NetworkClient& client); bool authenticate(::NetworkClient& client); + // Binary commands, answered on this task: true if `line` was one. + bool binaryCommand(::NetworkClient& client, const std::string& line); + // Runs `job` on the storage task (where card access is safe) and waits for it. False if the + // card isn't mounted, in which case the storage task never runs it. + bool onStorage(std::function job); + void get(::NetworkClient& client, const std::string& path); + void put(::NetworkClient& client, const std::string& args); + void screenshot(::NetworkClient& client); WifiService& wifi_; + StorageService& storage_; + Canvas* frame_ = nullptr; TaskHandle_t task_ = nullptr; SemaphoreHandle_t lock_; std::deque commands_;