Devlog: "It said "No"", the help key, the Shell, notes of any size and a site that publishes itself (v0.13.0 to v0.15.0)

Also corrects one sentence in W1.md that claimed more than was known about
how publishing by hand had gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT

site/content/devlog/roro9stack-shell/index.md

@@ -0,0 +1,264 @@
++++
+title = '''It said "No"'''
+description = '''The last post listed three things as next for roro9stack: one help key, a shell on the device, notes of any size. All three shipped in a day, with a CI that stopped rebuilding the world and a website that publishes itself. On the way, a test script kept typing after the device had crashed, and sent one word to an IRC channel full of people.'''
+date = 2026-10-07T18:00:00+02:00
+
+[extra]
+topics = '''ESP32-S3 · Testing · Editors'''
+read_label = '''Read who it said it to →'''
+uid = '''<b>app:</b> Shell &nbsp; <b>heap:</b> 104 KB free'''
+dek = "Three releases of [roro9stack](/devlog/roro9stack/) in one day: v0.13.0, v0.14.0 and v0.15.0. A help key that replaces every hint line, the firmware's console on the device's own screen, and an editor that opens a megabyte in the memory it used for a shopping list. Most of what went wrong was me being wrong about what the device had done. One thing was the device doing exactly what my script told it to, in the wrong App."
+byline = '''designed by interrogation, rounds thirteen to sixteen: thirty-seven questions, two of them answered twice'''
+
+[extra.sign]
+label = "Messages sent to real people by a test script"
+note = "One word, in an IRC channel, after a crash the script didn't notice."
+count = "1"
+tone = "red"
+
+[[extra.cast]]
+name = "Fn+h"
+role = "the help key, on every screen"
+text = "Lists the keys that work where you are. Every screen had a line at the bottom doing that, differently and never completely. Those lines are gone."
+
+[[extra.cast]]
+name = "The Shell"
+role = "an App, since v0.14.0"
+text = "The commands I had been typing from a PC over Wi-Fi, on the device's own keyboard. It took more than one try to decide what it should show, and one crash to decide where its commands run."
+
+[[extra.cast]]
+name = "The test script"
+role = "types keys over the Debug Console"
+text = "Tireless, exact, and with no idea what is on the screen. It typed the right letters. The App under them had changed."
+
+[[extra.cast]]
+name = "The window"
+role = "8 KB of a note, around the cursor"
+text = "All of a note that is in memory. The rest stays on the card, described by a short list. It moves when the cursor nears its edge, and nobody is meant to notice."
+
+[[extra.cast]]
+name = "<note>.edit"
+role = "the side file"
+text = "Where a long note's changes wait, four kilobytes at a time, until the note is left and rewritten. Also what a power cut leaves behind, on purpose."
++++
+
+## TL;DR
+
+- The [last post](/devlog/roro9stack-console/) ended with three things as "next". **All three are in**: a help key (**v0.13.0**), a shell on the device (**v0.14.0**), notes of any size (**v0.15.0**).
+- **Fn+h lists the keys of the screen you're on**, and every hint line is gone. The same lists make the key tables on this website.
+- **CI went from over seven minutes to about one** for a pull request. It had been rebuilding the whole framework at every run because of one file that isn't in git.
+- **The Shell** runs the firmware's commands on the device: Tab completes every word and paths on the card, `rm` behaves like Unix's and asks first, `*` and `?` work.
+- **The editor opens any file.** A 1.2 MB note uses the same 17.5 KB as a 62-byte one, saves in 4 KB pieces, and is rewritten in 2.6 s when you leave it. A power cut at any byte leaves the note or the last save, never something in between.
+- **This site publishes itself** when a change is merged, through an SSH key that can do exactly one thing.
+- A test script **sent the word "No" to an IRC channel**. That one can't be fixed, only prevented.
+- 507 host tests, 39 more than last time.
+
+## The cast
+
+{{ cast() }}
+
+## One key instead of a hint line on every screen
+
+Every screen had a line at the bottom: `Enter open  d delete  r rename`. Each was written by hand, each was different, and none had room for everything. The screen is 240 pixels wide.
+
+So: **Fn+h, everywhere**, and `?` wherever you aren't typing text. It opens a panel over the App, titled with where you are, listing that screen's keys and then the ones that work everywhere. Any other key closes it.
+
+{{ figure(src="help.png", alt="The Cardputer's screen at 2x: a panel titled Keys: Shell, listing Enter run the line, Tab complete the command, Fn semicolon and period lines you typed before, Alt semicolon and period scroll back and forward, Ctrl b, Fn comma and slash move the cursor, Del delete backwards, help every command.", width=480, height=270, caption="The Shell's keys, from the first build that had a Shell. The line that runs off the edge was reworded the same afternoon.") }}
+
+The hint lines went, all of them, with one exception: the first-start Setup keeps its own, because someone in their first minute doesn't know the help key exists. It tells them on its first and last screens.
+
+The lists started as code inside each App. They are now **data in one file**, 52 small tables, and the same script that builds the [developer docs](/dev/) reads that file and writes the key tables in the [user guide](/guide/). CI fails if the site's copy is out of date, so the guide can't list a key the firmware doesn't have.
+
+## The framework that was rebuilt every time
+
+A pull request took over seven minutes to check, and a release thirteen and a half. For a firmware that builds in 77 seconds on my machine.
+
+Where the time went, for one pull request:
+
+{% table() %}
+| Step | Time |
+|---|---|
+| Tools | 15 s |
+| Host tests and coverage | 55 s |
+| **The firmware** | **358 s** |
+| of which: configuring ESP-IDF | 87 s |
+| of which: compiling ESP-IDF's libraries | 171 s |
+| of which: our own code | 91 s |
+{% end %}
+
+roro9stack rebuilds the Arduino framework with its own settings, for [smaller TLS buffers](/dev/decisions/0006-framework-rebuilt-for-smaller-tls-buffers/). The rebuilt libraries were sitting in the runner's cache the whole time. But the build system decides whether they still match by reading a file in the project folder, and that file is generated: it isn't in git. Every fresh checkout had no such file, so every run concluded the libraries were stale and rebuilt them. 260 seconds, each time, to produce what was already there.
+
+The fix is to keep that file with the libraries it describes. Two more things came out of looking:
+
+- **The version was a `-D` flag on every compiler command line.** Every commit changes the version, so every commit recompiled every file, on my machine too, and no cache could ever have helped. It's now one generated header that one file includes.
+- **A release built the firmware twice**: once to check it, once to sign it.
+
+With a build cache for pull requests on top: **27 seconds** for the firmware with one file changed, and about a minute for the whole run on the real runner. A release takes under three minutes, and still compiles its own sources from nothing: no published file contains an object built for another commit.
+
+## A shell, and what it should show
+
+The Debug Console's commands are the tool I use most, and they needed a PC. The Shell is an App that runs them on the device.
+
+The first version was an afternoon's work and wrong in three ways.
+
+**It showed too much.** The firmware prints all the time: IRC connecting, a packet heard, whatever a PC on the USB port is asking for. Version one showed everything printed in the ten seconds after a command, on the theory that the reply would be in there somewhere. It was, among everything else. The fix was to stop guessing: the console now knows **who each line is for**. A command run from the Shell prints as the Shell's, and so does an answer that arrives a second later from another task, which notes who asked. Ctrl+b shows everything, for when that's what you want.
+
+**`rm` was dangerous in a new way.** Over the console, `rm <folder>` had always removed the folder and everything in it, which is fine for a script and less fine for a thumb on a small keyboard. It's now Unix's: a folder needs `-r`, and in the Shell it asks unless you say `-f`.
+
+**Tab did one word.** It now follows the firmware's own `help` text, word by word: `lora st` becomes `lora status`, `gnss track ` lists `start  stop`. The words are read from the help text as it is written, so a new command completes without anyone maintaining a table. Past the command, it completes paths on the SD card. And `*` and `?` work in file names.
+
+{{ figure(src="rm.png", alt="The Cardputer's screen at 2x: a dialog titled Delete? reading The 5 that match /gt2/*. It can't be undone. with two buttons, Cancel selected and Delete.", width=480, height=270, caption="`rm /gt2/*` in the Shell: one question for all five, with Cancel selected. `/gt2` is a scratch folder, made for the purpose.") }}
+
+One more addition, small and my favourite: **an App's name with a capital letter opens it.** `Notes`, `Irc`, `Storage`. Every command is lowercase, so the capital is the whole syntax.
+
+## It said "No"
+
+The Shell's first version ran each command from inside the key handler. I test the UI by sending key presses over the Debug Console, so the call chain was: the main loop, a remote command, a key, the App manager, the Shell, the command interpreter *a second time*, the file command, and `printf` under all of it. The main loop has under 2 KB of stack to spare. `rm` on a folder went past it.
+
+The device crashed, and restarted, as it should. It came back up in the Launcher.
+
+My test script didn't know. It had a list of keys to send and it sent them. Its next Enter, meant for the Shell, landed in the Launcher and opened the first App in the list. That is IRC, which connected, as it's configured to, and joined its channels.
+
+A few lines later the script reached its test of the capital-letter feature: type `No`, press Tab to complete it to `Notes`, press Enter. Tab completes nothing in IRC. Enter sends.
+
+One word, to a channel of real people, from my nick. Not harmful, not explainable either, and not something any commit can take back.
+
+Two things changed that afternoon:
+
+- **The Shell hands its line to the main loop**, which runs it at the same depth as any console command. The crash is gone, and the crash report had decoded to exactly that chain of calls.
+- **`info` reports the App in front**, and the test helper checks it before every line it types. A script that survives a restart is typing somewhere else, and now it stops.
+
+The rule I'd had since the day before was "take a screenshot before any key that deletes something". It was the right rule for the wrong failure. Typing is also an action.
+
+## A megabyte in 17 KB
+
+The Notes editor held the whole note in memory and stopped at 16 KB. That was a limit for the first version only; [F1's notes](/dev/milestones/f1/) say so in bold.
+
+The device has no spare memory to throw at this, so the design is the old one from editors that ran on less: **the note is the file on the card, plus one window in memory.** The window is about 8 KB around the cursor. Everything else is a list of pieces: "bytes 0 to 40,000 of the file", "then 9,000 bytes of what was typed". When the cursor nears the window's edge, the window is written away if it changed, and the next one is loaded.
+
+What makes it usable is what it writes, and when:
+
+{% table() %}
+| | Up to 64 KB | Above |
+|---|---|---|
+| The save, five seconds after the last key | The whole file, as before | What changed, appended to `<note>.edit`: about 4 KB |
+| Leaving the note | Nothing more to do | The file is rewritten, with a progress bar |
+| After a power cut | The note as last saved | The note opens with the saved changes back |
+{% end %}
+
+{{ figure(src="saving.png", alt="The Cardputer's screen at 2x, all black with Saving in blue, the file name zz-big.txt, a progress bar a little over half full, and 60%.", width=480, height=270, caption="Leaving a 1.2 MB note. This takes 2.6 seconds, which is long enough to deserve a bar and short enough that I had to race the screenshot.") }}
+
+Memory with a note open is 17.5 KB, for a note of 62 bytes or of 1.2 MB. Going to the end of the megabyte takes as long as any other key.
+
+### The part that has to be right
+
+An editor that loses text is worse than no editor, and this one now has a side file, a temporary file and the note itself, any of which can be half written when the power goes. So the rewrite ends with a mark: once the complete new file is on the card, one small write to the side file says "done". Before that mark, the old note and its side file are the truth. After it, the new file is, and whatever was interrupted is finished the next time the note is opened.
+
+That is a claim, and it's the kind I don't trust until something has tried to break it. Two tests do:
+
+- **A power cut at every 997th byte** of two saves and a rewrite. After each, the note has to be one of exactly three texts, the one that was reported as saved has to be there, and no stray file may be left.
+- **36,000 random keys** on six notes, typing, deleting, moving, jumping, saving and cutting the power, compared with a plain string after every key.
+
+They pass. But the first run had three failures, and they're worth a line each, because only one of them was the editor's:
+
+1. I had worked out by hand where the cursor should be after a recovery, and got it wrong by four.
+2. I had assumed windows would break between groups of three characters in my test text. They break between characters, which is all they promise.
+3. **A file replaced by a shorter one lost its pending edits without a word.** The design says they are set aside as `.edit.lost` and the editor tells you. The code checked the pieces against the new file's length first, found them out of range, concluded there was nothing valid to keep, and deleted them. A real bug, in exactly the path that exists to never delete typed text.
+
+Two of three were the test being wrong. The third is why the tests exist.
+
+{{ figure(src="back.png", alt="The Cardputer's Notes editor at 2x, showing zz-big.txt, 1.1 MB, saved. The first line reads YTOP line 000000, followed by line 000001 to line 000007. At the bottom, in orange: Your unsaved changes are back.", width=480, height=270, caption="After a restart in the middle of a rewrite. The `Y` was typed, saved to the side file, and the device was reset while it was writing the megabyte. It's there.") }}
+
+### What I had promised, and what I measured
+
+I'd said the rewrite would take about two and a half seconds a megabyte. The first build took 3.5 to 4.5 seconds for 1.2 MB. It was copying in 2 KB blocks. With 4 KB blocks it takes 2.6, about 450 KB a second, which is what this card gives a plain copy.
+
+## A site that publishes itself
+
+Until this morning, publishing this site meant logging into the web server and running a script, by hand, after every merge.
+
+Now CI does it, after a merge and after a release. The interesting part is what the key in CI is allowed to do, which is one thing:
+
+{% code(caption="One line of `authorized_keys` on the web server. Whatever the client asks for, this runs instead.") %}
+```
+from="<the runner>",restrict,command="/path/to/rororefresh.sh" ssh-ed25519 AAAA… roro9stack-ci
+```
+{% end %}
+
+My first plan had the command as a secret in CI, next to the key. With a forced command there is nothing to keep secret: CI connects and sends no command at all, and a stolen key can refresh the website and do nothing else. The server's address, the user, the key and the server's host key are secrets; the repository is public and none of them is in it.
+
+Checked against a throwaway SSH server before the real one: asking for `id; cat /etc/passwd` runs the refresh. No terminal. `scp` copies nothing. A different host key stops the run.
+
+On its first real run, the live page changed fifteen seconds after the run started. This post got here that way.
+
+## The device was right
+
+A pattern from the day, three times over.
+
+**The keys that vanished.** Twice, the first key my script sent after a quiet minute did nothing. The [F1 notes](/dev/milestones/f1/) describe that exact bug, fixed. I wrote it up as a possible regression. It isn't: a key that wakes a dark screen only wakes it, same as on the real keyboard. That's documented, on a page of this site, which I wrote.
+
+**The letters in the wrong place.** In the editor test I typed two letters at the top of a note, moved 400 lines down, typed four more, fetched the file and compared it with what I expected. It differed: `liMID ne 000400` where I expected `MID line 000400`. The file matched the screen exactly. Moving down keeps the cursor's column, as it should, and I had typed two letters first.
+
+**The "No".** The device did what it was sent. Every key arrived, in order.
+
+Three times the instrument was right and the reading was wrong. The remote `key` command now takes `ctrl-`, `alt-` and `shift-`, by the way, because checking the editor's jump to the end of a note needed Ctrl, and "the remote `key` command can't send that" had been in the not-checked list of every milestone since the editor existed.
+
+## What I didn't check
+
+- **The real keyboard**, for Fn+h, `?`, Ctrl+b and the Alt scroll. Everything was driven by remote keys.
+- **The power button's path** in the editor, which writes the side file only, and the screen turning off.
+- **Memory with IRC connected** and a long note open. After the morning, I didn't connect IRC.
+- **A file of tens of megabytes.**
+- Renaming or deleting a note from the Storage App leaves its side file behind. The Notes App handles both.
+
+## By the numbers
+
+{% table() %}
+| | |
+|---|---|
+| Releases | 3 |
+| Design questions | 37 |
+| Host tests | 507 |
+| A pull request's CI run, before and after | over 7 min, about 1 |
+| Seconds spent rebuilding what was already built, per run | 260 |
+| Flash the Shell costs | 21 KB |
+| Flash notes of any size cost | 15 KB |
+| Memory with a note open, 62 bytes or 1.2 MB | 17.5 KB |
+| Bytes a long note's save writes | about 4,000 |
+| Random keys the editor was compared against a string for | 36,000 |
+| Bugs those tests found in the editor | 1 |
+| Bugs they found in my arithmetic | 2 |
+| Words sent to an IRC channel | 1 |
+{% end %}
+
+## Where it stands
+
+{% steps() %}
+1. ~~M0 and M1: the skeleton, Wi-Fi, IRC, Wi-Fi Tools.~~ v0.1.0 to v0.2.1, [the first post](/devlog/roro9stack/).
+
+2. ~~Updates and debugging over the air.~~ v0.3.0, [Look, no cables](/devlog/roro9stack-ota/).
+
+3. ~~M2: GNSS.~~ v0.4.0, [Seventeen satellites](/devlog/roro9stack-gnss/).
+
+4. ~~G1: Gemini.~~ v0.5.0, [A browser in the RAM IRC left over](/devlog/roro9stack-gemini/).
+
+5. ~~M3: the LoRa radio, listening.~~ v0.6.0, [The loudest thing it hears is itself](/devlog/roro9stack-lora/).
+
+6. ~~S1: the card, fixed addresses, the System App.~~ v0.6.1 to v0.8.1, [One byte too early](/devlog/roro9stack-s1/).
+
+7. ~~F1 and the start of R1: files, notes, signed releases, updates from Gitea.~~ v0.9.0 to v0.11.0, [836 bytes](/devlog/roro9stack-f1-r1/).
+
+8. ~~W1: the website, and one firmware with the Debug Console in it.~~ v0.12.0, [It was off](/devlog/roro9stack-console/).
+
+9. ~~One help key, and CI in a minute.~~ v0.13.0, this post.
+
+10. ~~The Shell.~~ v0.14.0, this post.
+
+11. ~~Notes of any size, and a site that publishes itself.~~ v0.15.0, this post.
+
+12. Next: M4, the mesh, which still wants a second node. And the editor, now that it opens anything, plainly lacks undo.
+{% end %}
+
+{% signoff() %}
+The last post promised three things and this one delivers them, which would be a tidy story if a script of mine hadn't said "No" to a room of strangers halfway through. The device did nothing wrong all day. It crashed where I had written a crash, restarted as designed, and typed what it was sent.
+{% end %}