+++
title = '''It was off'''
description = '''roro9stack gets a website, and writing down how its Debug Console works shows that only one person could ever use it. So the Debug Build is retired: one firmware, with the console in it, off until its owner switches it on. Then an evening of chasing a console that wouldn't come back and a memory leak, neither of which existed.'''
date = 2026-10-07T00:30:00+02:00

[extra]
topics = '''ESP32-S3 · Debugging · Security'''
read_label = '''Read what was off →'''
uid = '''<b>debug:</b> on, token set, client connected'''
dek = "A day of writing for [roro9stack](/devlog/roro9stack/), my firmware for the M5Stack Cardputer: a website, a user guide, and developer docs about the thing I like best in it, a console over Wi-Fi. Documenting it properly made one fact hard to miss: nobody else could have it. Fixing that deleted more than it added, and then cost me an evening looking for two bugs that turned out to be a switch in the Off position and TCP minding its own business."
byline = '''designed by interrogation, round twelve: fourteen questions thrown away, eight kept'''

[extra.sign]
label = "Tokens leaked by the feature built to protect them"
note = "In a screenshot, taken over the console, of the one page that shows the token."
count = "1"
tone = "red"

[[extra.cast]]
name = "The Debug Build"
role = "retired, v0.3.0 to v0.11.0"
text = "The same firmware plus a console over Wi-Fi, with its builder's token compiled in. Which is why it could never be published, and why the firmware I tested was never the one I released."

[[extra.cast]]
name = "Port 3232"
role = "the Update Service, in every build since v0.3.0"
text = "Listens on the network in release builds, guarded by a signature. The counter-example to \"in a release, nothing listens\" that had been sitting there all along."

[[extra.cast]]
name = "The token"
role = "100 bits, 20 characters"
text = "Made by the device, shown on one page of Settings and nowhere else. Read off a 240-pixel screen by a human, which is where the trouble started."

[[extra.cast]]
name = "The dialog"
role = "\"Switch it on?\""
text = "Opens with Cancel selected, as a question about a remote control should. Enter, Enter: still off. Works exactly as designed."

[[extra.cast]]
name = "TIME_WAIT"
role = "two minutes, per closed connection"
text = "What TCP does with a connection it has closed, in case a late packet turns up. About 270 bytes each. Looks exactly like a leak if you only watch for one minute."
+++

## TL;DR

- **roro9stack has a website:** [roro9stack.net](/), with an Install page that flashes a Cardputer from the browser, a [user guide](/guide/), [how-tos](/howto/), an [FAQ](/faq/), [developer docs](/dev/) generated from the repository, and this devlog, which moved here.
- **The Debug Build is gone.** There is one firmware (**v0.12.0**), and the Debug Console is in it: **off** until you switch it on in Settings, with a token the device makes itself.
- **The token never crosses the network.** The device sends a challenge, the client answers with an HMAC. Five wrong answers close the console for a minute.
- **It costs** 30 KB of flash and 88 bytes of RAM over the old release build. Off, nothing listens and nothing is allocated.
- Then I spent an evening on **a console that wouldn't come back on** (it was off) and **a memory leak of 230 bytes per reopening** (it was TCP). Both produced a real fix on the way, and one design change I had to undo.
- 468 host tests, 12 more than last time. The decision is [ADR 0010](/dev/decisions/0010-debug-console-in-every-build/).

## The cast

{{ cast() }}

## A site, in four phases

The [last post](/devlog/roro9stack-f1-r1/) ended with a device that installs its own releases. That makes it something another person could use, and another person needs somewhere to start that isn't a Gitea README. So the first half of the day was a website: a home page, an Install page, then a guide to each App, how-tos, an FAQ, and developer docs.

Three things from that half are worth keeping.

**Flashing from the browser, without a copy of the firmware.** The Install page uses ESP Web Tools and Web Serial. The obvious way is to copy the firmware image next to the page; then every release needs a rebuild of the site. Instead the page asks Gitea's API for the latest release, in the browser, and hands the flasher a manifest it builds on the spot. That only needed one header: Caddy, in front of Gitea, adds `Access-Control-Allow-Origin: https://roro9stack.net` to the release downloads and to the releases API, which are public anyway. A new release shows up on the page the moment it exists.

**"Generated from the repository" met Zola.** The developer docs were to be built from `docs/`, the README and the firmware's own `help` text, not copied by hand. Zola refuses to read a file outside its own folder, and it resolves symlinks before deciding, so that door is closed too. So a small script writes those pages, they are committed, and CI fails when one is out of date. The command reference is the part I like: it is parsed from the `kHelp` string in `main.cpp`, so the site can't describe a command the firmware doesn't have.

**The posts you're reading had `<style>` in them.** This devlog came over from my blog, seventeen diagrams included, each an inline SVG with its own `<style>` block. The site's Content-Security-Policy is `style-src 'self'`, which refuses exactly that. The rules moved into a stylesheet and the `style="…"` attributes became classes. Tested in a real browser with the production policy on every response: no violations.

The site says only what the firmware does today, which meant writing the same sentence several times: the mesh messenger isn't built. The radio listens. It doesn't talk yet.

## Documenting a feature only I could use

The part of the developer docs I cared most about is the Debug Console: the serial console over Wi-Fi, with key presses, screenshots, file transfer and crash dumps. I wrote six pages on it, checked the protocol against the live device, and by the end had described, in detail, a feature with this property:

> A Debug Build carries its builder's token, so Debug Builds are never published.

Everything in those pages was for whoever builds the firmware. That's me.

The issue I filed was the obvious patch: make the token configurable, so that Debug Builds can be published. The design round for it had fourteen questions. Where does a published Debug Build go, so that devices in the field don't install it by accident (v0.11.0 takes the last `.ota` it finds in a release)? What is its tag called, so that CI doesn't start itself again? How does a device that runs one get the next?

Then one question from the other side of the table replaced all fourteen: *why have Debug Builds at all?*

## The argument that was never whole

[ADR 0004](/dev/decisions/0004-debug-console-in-debug-builds/) kept the console out of release builds with one sentence, which I was rather proud of:

> A console that runs commands is a remote control: in a release build, nothing listens.

Except something does. The Update Service has listened on TCP 3232 in every build since v0.3.0, guarded by a signature. "Nothing listens" was never true; "nothing listens without a lock on it" was. A console that is off by default, behind a secret the device made itself, is the same trade.

And the split had been charging rent the whole time:

- **What I tested wasn't what I shipped.** I lived on Debug Builds. Releases were a different binary that nobody ran before it was published.
- **Rules that only protected the console.** A Debug Build refused to install a release (it would have lost its console), so there was `update install … force` to do it anyway, and advice about which firmware to keep in the other slot.
- **Versions ending in `+debug`** that had to compare equal to their release.
- **Two firmwares built by CI** on every pull request and every tag.

One firmware deletes all four.

## What off means

The new argument only holds if "off" is as good as "not compiled in". So:

- **Off is the default,** and what a missing or damaged setting falls back to.
- **Off, nothing exists:** no socket, no task, and no 4 KB buffer, which used to be a static array and is now allocated when the console is switched on.
- **On needs someone at the device,** or on the USB cable. Over the console itself, `debug on` and `debug token` answer `over USB serial only`: it can switch itself off, not open itself wider.

{% table() %}
| Build | Flash | Static RAM |
|---|---|---|
| The release build, before | 1,851,387 | 54,612 |
| The Debug Build, before | 1,874,887 | 58,780 |
| **The one firmware** | **1,881,799** | **54,700** |
{% end %}

Thirty kilobytes of flash, and 88 bytes of RAM, for a console in every device. The Debug Build's extra 4 KB of RAM was the buffer, sitting there whether anyone connected or not.

## A token you can read, and one you can't steal

The token used to be 32 hex digits in a file on my laptop, compiled in. Now the device makes it, the first time the console is switched on: 100 bits from the hardware generator, as twenty characters of Crockford's base32, the alphabet that leaves out I, L, O and U because people misread them.

The old login sent the token as the first line of a plain TCP connection. That was fine for a secret that lived on one laptop and one device on one network. It's not fine for a secret in every device, on whatever Wi-Fi its owner uses: mine is called `knbg-guests`, and anyone with the Wi-Fi password can watch it.

So the token no longer travels:

{% code(caption="The whole login. A recorded answer is no use for the next challenge.") %}
```
device:  roro9stack debug console, challenge 3f9a…c1   (16 random bytes)
client:  HMAC-SHA256(key = token, message = those bytes), in hex
device:  roro9stack v0.12.0 debug console. 'help' lists the commands. Backlog follows.
```
{% end %}

HMAC is twenty lines on top of the SHA-256 the firmware already had for update files, checked against the RFC's vectors and against the same vector as the Python client. Five wrong answers in a row and the console answers `locked` to everyone for a minute:

{% code(caption="Six tries with a wrong token, then the right one.") %}
```
try 1 at  1s: wrong token
try 2 at  3s: wrong token
try 3 at  4s: wrong token
try 4 at  5s: wrong token
try 5 at  6s: wrong token
try 6 at  6s: closed for a minute after too many wrong tokens
right token:  closed for a minute after too many wrong tokens
```
{% end %}

This breaks every old client. I'm the only user, and v1 is a long way off; this is exactly when to break things.

## Three ways it went wrong

### I couldn't read my own token

First login, on the real device: `wrong token`. The firmware was right. I had typed one character wrong, reading a 20-character token drawn in bold at the normal size, where 8 and B, 5 and S, 2 and Z are a matter of opinion.

The token is now drawn at twice the size, in regular weight, on two lines. And what I should have done from the start: Crockford's rule says that if someone types an O, an I or an L, they meant 0 or 1, because the alphabet doesn't have those letters. Both ends now apply it.

### The screenshot

To check the new layout I did what I always do: took a screenshot over the console.

{{ figure(src="token.png", alt="The Cardputer's Settings, Debug Console page at 2x: Debug Console On, Connect to 10.39.39.12:2323, New token, Type a token, and below them a token in large type on two lines, T49R-HQXB-NDJX then BNSB-XSWD. The Status Bar shows DBG in blue.", width=480, height=270, caption="The page that shows the token \"and nowhere else\", in a file on my laptop. This token was replaced within the hour.") }}

The documentation I had written an hour earlier says the token is shown on one page and never printed anywhere. And there it is, in a PNG. No escalation: whoever can take a screenshot already has the token. But a secret now sits in a file because of a habit, and the sign at the top of this post is for that. The docs now say it in so many words: a screenshot of that page is a copy of the token.

### The console that wouldn't come back

The last test was `debug off`, sent over the console: the client is dropped, the port refuses connections. Good. I switched it back on at the device, made a new token, and tried to log in.

Connection refused.

The device was up. Its update port answered. And it stayed refused across another firmware push and a restart. So, I reasoned, the console fails to come back once it has been closed, and I went looking:

- **The framework's `begin()` returns nothing.** `NetworkServer::begin()` can fail at `socket()`, `bind()` or `listen()` and says nothing either way; my code set `listening = true` and never asked. A real hole. Fixed: it asks, complains on the console, and tries again every two seconds.
- **I couldn't reproduce it from my desk.** Switching the console on is for the device and the cable only, by design, so the one path I wanted to test was the one I had made unreachable. Hence `debug off <seconds>`: the console closes and comes back by itself after the pause. It can't open anything wider (same state, same token), and it makes closing and reopening something a script can do twenty times.

Then I went and looked at the device's screen.

It said **Off**.

The "Switch it on?" dialog opens with **Cancel** selected. Enter on the row, Enter on the dialog: nothing changes, and the page says so in plain letters that I hadn't read. I had filed a bug against a switch for being in the position I left it in.

### The leak that was TCP

With `debug off <seconds>` I could finally hammer the path. It came back every time. And free memory fell by about 230 bytes per cycle.

A leak in code I had just written, in a firmware where 836 bytes was [once all that was left](/devlog/roro9stack-f1-r1/). I had a suspect: the console's task is deleted when the console goes off and made again when it comes back. Twenty plain logins leaked nothing, so it wasn't connections. I changed the design: the task stays, asleep, while the console is off. That breaks the promise that off means nothing is there, but a leak is worse.

It leaked exactly as much as before.

So I did what I should have done first, and measured for longer:

{% table() %}
| | Free heap |
|---|---|
| Before | 107,248 |
| Right after 15 closings and reopenings | 103,180 |
| 60 s later | 105,008 |
| 120 s later | 107,004 |
| 240 s later | 106,924 |
{% end %}

All of it comes back. When the device closes a connection, TCP keeps it for two minutes in case a late packet arrives, and each one holds about 270 bytes. Fifteen cycles in a minute and a half look like a leak; fifteen cycles and a cup of tea look like nothing at all.

The design change went back out. Off means the task is gone too, as decided.

## What held

{{ figure(src="dbg.png", alt="The Cardputer's Launcher at 2x, listing IRC, Wi-Fi Tools, GNSS, Gemini, LoRa Scanner, Storage, Notes, System and Settings. The Status Bar shows DBG in blue between the GNSS and Wi-Fi indicators.", width=480, height=270, caption="`DBG` in the Status Bar: there while the console listens, bright while someone is connected. Taken over the console, so: bright.") }}

Checked on the device, not just in tests:

- **Off by default.** Pushed over a Debug Build, the new firmware came up with port 2323 refusing connections.
- **The setting survives updates.** Four pushes later the console came back by itself each time.
- **Safe Mode has the console.** Three `crash abort` in a row, and the device started in Safe Mode with 178 KB free and the console reachable. `ls /` answered `not available in Safe Mode`, and the crash decoded to the line of `main.cpp` with `abort()` on it.
- **25 closings and reopenings,** each back a second after its pause.
- **More memory than before.** 108 KB free with the console on and a client connected, against 104 KB on the Debug Build.

And what I didn't check: `scripts/flash.sh --debug`, which now sets a device up over USB, with no cable attached to the VM that night; typing a token of my own on the device; and the notification on the screen during a lockout, which nobody connected can see, the console being closed.

## By the numbers

{% table() %}
| | |
|---|---|
| Design questions written, then thrown away | 14 |
| Design questions kept | 8 |
| Host tests | 468 |
| Bits in a token | 100 |
| Wrong answers before the console closes | 5 |
| Flash it costs every device | 30,412 bytes |
| RAM it costs every device, off | 88 bytes |
| Bytes TCP holds for a closed connection, for two minutes | about 270 |
| Pages on the website | 66 |
| Bugs found in the console that evening | 1 (the silent `begin()`) |
| Bugs I thought I'd found | 3 |
{% end %}

## Where it stands

{% steps() %}
1. ~~M0 and M1: the skeleton, Wi-Fi, IRC, Wi-Fi Tools.~~ v0.1.0 to v0.2.1, [the first post](/devlog/roro9stack/).

2. ~~Updates and debugging over the air.~~ v0.3.0, [Look, no cables](/devlog/roro9stack-ota/).

3. ~~M2: GNSS.~~ v0.4.0, [Seventeen satellites](/devlog/roro9stack-gnss/).

4. ~~G1: Gemini.~~ v0.5.0, [A browser in the RAM IRC left over](/devlog/roro9stack-gemini/).

5. ~~M3: the LoRa radio, listening.~~ v0.6.0, [The loudest thing it hears is itself](/devlog/roro9stack-lora/).

6. ~~S1: the card, fixed addresses, the System App.~~ v0.6.1 to v0.8.1, [One byte too early](/devlog/roro9stack-s1/).

7. ~~F1 and the start of R1: files, notes, signed releases, updates from Gitea.~~ v0.9.0 to v0.11.0, [836 bytes](/devlog/roro9stack-f1-r1/).

8. ~~W1: the website.~~ You're on it.

9. ~~One firmware, with the Debug Console in it.~~ v0.12.0, this post.

10. Next: notes of any size, a shell on the device itself for the same commands, and one help key everywhere instead of a hint line on every screen. And M4, the mesh, which still wants a second node.
{% end %}

{% signoff() %}
I wrote a feature whose whole point is that it's off until you switch it on, and then spent an evening debugging it for being off. The switch works. So does TCP. The only thing that leaked was the token, and I did that myself.
{% end %}