roro9stack/docs/milestones/R1.md

# R1 — Releases

**Status:** in progress. CI and signed releases on Gitea (issue #5) are in place since 2026-10-06: every tag from v0.1.0 to v0.10.0 has its release. Updates from Gitea (issue #6) is built and checked on the device, on branch `gitea-updates`, not merged yet. The Issues App (#4) comes after.

**Goal:** a tag is a release, built the same way every time and published where a device can find it.

## CI and releases (issue #5)

Until now the tests, the builds, the signing and the flashing all happened on one machine, through `scripts/ci.sh` and `scripts/flash.sh`. Nothing was published.

### Decisions (design round 2026-10-06)

| # | Decision |
|---|---|
| Q151 | A push to `main`: the host tests (with their coverage). A push to another branch: nothing, its pull request is what runs (a branch with an open pull request ran twice, once for each). A pull request: the same and both builds; changes reach `main` through pull requests. A tag `v*`: all of it, then a release. (First: everything on every push, which rebuilt the firmware far more often than anyone looked at it.) |
| Q152 | **CI signs.** The signing key is the repository secret `OTA_SIGNING_KEY`; a tag push makes a complete, signed release with no manual step (ADR 0008). |
| Q153 | *Revised by Q188: there is one firmware.* The Debug Build is built in CI with a token of the runner's own, to prove it compiles, and **isn't published**: it would hand everyone its Debug Console token. |
| Q154 | Pull requests from forks don't start a run. |
| Q155 | A release carries `roro9stack-<version>.ota` (signed), `-factory.bin` for USB, `.elf.gz` to decode crashes, and `SHA256SUMS`. |
| Q156 | Its text is the tag's message, what the files are, and the commits since the tag before. |
| Q157 | No cache service to begin with: measure first. |
| Q158 | Reproducible builds aren't needed for signing any more (Q152); not pursued here. |
| Q159 | **The tags from before CI get their releases too**, v0.1.0 to v0.10.0, built from each tag's own sources by running the workflow by hand. |
| Q160 | Actions is switched on for the repository. |
| Q161 | **Changes reach `main` through pull requests, merged as "rebase, then a merge commit"**, the only style the repository allows: the branch's commits keep their messages, the merge commit marks the pull request, and what CI tested is what lands. No squash, no fast-forward. |

### As built

- **One workflow, `.gitea/workflows/ci.yml`, one job**, on the runner `runner0` (label `ubuntu`). The job asks for a `python:3.12-slim` container, installs git, a compiler, openssl and PlatformIO, and runs the same scripts as a developer's machine. No Docker inside the job.
- **The cache is a Docker volume**, `roro9stack-pio`, mounted at `/pio`; the runner's `config.yaml` allows it under `container.valid_volumes`. A first run downloads about 1 GB and rebuilds the framework. Measured from the jobs' own start and end times: the first full run on an empty cache took 11.4 minutes (tests and both builds); the first run in a container, 8.1; a pull request now takes about 8.7 (tests, coverage and both builds), a release build alone 5.5, and a push to `main` (tests and coverage) 1.1. (An earlier version of this note said 17 minutes: that was the waiting time, not the job's.)
- **No JavaScript actions**, so the image needs no Node and nothing is fetched from GitHub: the checkout is four git commands.
- **`scripts/_docker.sh`** runs the command in place when `RORO_NO_DOCKER` is set (a CI job is already in a build container), and in the project's image otherwise. The Debug Build's token is made on the spot in CI and goes with the container.
- **`scripts/release_build.sh <checkout> <out>`** builds a tag's own sources with today's tools, signs, verifies against the public key in those sources, and writes the files and the release's text. **`scripts/release_publish.py`** creates the Gitea release or completes it; run twice, it replaces what's there. Both run the same on a developer's machine.
- **`scripts/ota_verify.py`** checks an Update File as a device does, on a PC.
- **The job's own token** (`secrets.GITEA_TOKEN`) is enough to create a release and upload its files.
- **Old tags.** v0.1.0 to v0.3.0 are from before the framework was rebuilt with our settings (ADR 0006) and can't link against a rebuilt one left in the cache: the release build puts the stock framework libraries back for them. v0.1.0 to v0.2.1 have no public key in their sources (Firmware Updates came with v0.3.0); their files are checked against today's.

### How it went

- **The runner's label took three tries.** Registered as `ubuntu://docker:ubuntu:resolute` and then as `ubuntu::docker://...`, Gitea took the whole string for the label's name; with the first, jobs ran on the runner's host itself. The first version of the workflow was written for that (plain shell, `docker run` for the build) and published v0.10.0 that way. `ubuntu:docker://docker.gitea.com/runner-images:ubuntu-latest` is the form that works.
- **Gitea 1.27's API can't cancel a run that isn't finished**, only delete a finished one; switching Actions off and on for the repository doesn't either. Runs queued for a label that no longer exists stay queued until cancelled in the web UI.
- **CI's image isn't byte-identical to a local build of the same tag** (same size, different bytes). Not pursued (Q158).

## Updates from Gitea (issue #6)

The device looks at the project's Gitea for a newer release, says so, and installs it on request, with the same signed Update Files, Probation and rollback as a push from the PC or an install from the card.

### What the server gives (checked 2026-10-06)

- **Its certificate** is Let's Encrypt, all ECDSA: leaf `git.twis.la` (renewed every few months, next expiry 2026-12-14) under the intermediate YE2, Root YE and ISRG Root X2, which X1 cross-signs. Pinning the leaf would ask a question at every renewal.
- **The API** answers over HTTP/1.1, chunked: `releases/latest` is 3.2 KB (about 350 bytes of it matter), a list of ten releases is 33 KB.
- **A download** is a direct 200 with `Content-Length` and no redirect; ranges work.

### Decisions (design round 2026-10-06)

| # | Decision |
|---|---|
| Q162 | **Trust:** the firmware carries ISRG Root X1 and X2 and checks the server's chain and name against them, not the framework's bundle of about 130 CAs (ADR 0009). Shared with #4. If the server moves to another CA, the next firmware comes from the PC. |
| Q163 | The Update File's own signature stays the real guard. A hijacked connection could hide a release or offer an older signed one, never install firmware that isn't ours. |
| Q164 | The source, `git.twis.la` and `twisla/roro9stack`, is a constant in the firmware. A fork changes it, and has its own key. |
| Q165 | **When:** on request in Settings > Firmware, and once a day in the background while Wi-Fi is up and the Clock is set (certificate dates need it). A setting, **Check for updates**, on by default. It installs nothing by itself; it skips quietly below the memory floor and never runs during an install. |
| Q166 | A Toast, "Update v0.11.0 available: see Settings > Firmware", once per version per boot. |
| Q167 | **The download goes straight into the inactive slot,** no card needed. A truncated or tampered file is refused after 160 bytes or at its end, and the running firmware is untouched. A failed download starts over. |
| Q168 | A version that failed (rolled back) isn't announced again by the background check until a newer one exists; it can still be installed by hand. |
| Q169 | **Older releases:** a list of the last ten, newest first, the running one marked. Installing an older one asks with a stronger warning. |
| Q170 | Enter on a release shows its version, date, size and the tag message, with Install. |
| Q171 | *Revised by Q188: there is no Debug Build, every firmware installs releases.* **Debug Builds** check and show the latest release, but don't install it: it would replace the Debug Build and its console (Q153: Debug Builds aren't published). Their updates come from the PC. |
| Q172 | **Memory, as measured:** a TLS connection peaks at about 52 KB of heap, with or without checking the certificate, so it starts with 80 KB free (Q86's 20 KB spare on top), not 55 KB. **A check, list or install someone asked for makes IRC step aside** and come back after; the daily check never does, and with IRC connected it waits. (First: 55 KB and nothing else. With IRC connected a check left 3 KB and a download 836 bytes.) |
| Q173 | Left out, each with its issue: installing automatically (#52), a release channel (#53), resuming a download (#54). |
| Q174 | Ships as **v0.11.0**. Tested on the device with the real signed releases; a Debug Build command pretends the device runs an older version, so v0.10.0 counts as an update. |

### Done when

- A check, by hand or daily, tells the right thing: up to date, newer available, no network, bad certificate, no clock, too little memory.
- A newer release installs from the Firmware page with no card and no PC, and the device restarts into it and confirms it.
- A tampered or truncated download is refused and the running firmware keeps running.
- The Older releases list shows ten, and installing one asks first.
- A release that rolled back isn't announced again.
- A Debug Build shows the latest release and doesn't install it.
- The daily check never runs below the memory floor, during an install, or without a clock.

### Work breakdown

1. **Model** (host-tested): a streaming JSON scanner, the release list read from it, HTTP response heads and chunked bodies, URLs, which release counts as an update.
2. **The connection:** the root certificates, an HTTPS client, a check and a list from the Update Service's task; console commands to try them.
3. **The download:** an HTTPS source for the existing install path.
4. **The screens:** the Firmware page's release rows, the release page, Older releases, the setting, the daily check and its Toast.
5. **Checks on the device**, recorded here.

### As built

- **`lib/release`** (host-tested): a streaming JSON scanner, the release reader built on it, HTTP heads and chunked bodies, URLs, and the decisions (which release is an update, whether to announce it, which download URLs are taken). A list of ten releases is 33 KB of JSON and costs a few hundred bytes of memory, because nothing is kept but the path.
- **`HttpsGet`** (`src/platform`): one GET, the answer read as a stream, redirects not followed. **`GiteaReleases`** keeps the latest and the list. **The Update Service** serves the requests on its own task (about 5.4 KB of its 7 KB stack at the peak) and installs through the install path that already existed, with an HTTPS source in place of the card or the TCP port.
- **The daily check** is scheduled from the Update Service's tick: Wi-Fi up, the Clock set, no Probation, nothing else going on, memory for a connection. The day it last succeeded is kept in flash.
- **A version that failed** (rolled back) is remembered as `ota_failed`, and isn't announced again by the daily check.
- **The screens:** the Firmware page's Latest release and Older releases rows, a release page with the tag's message, and the install dialog.
- **Debug Builds** get knobs to try what can't be tried otherwise: `update pretend`, `probe`, `damage` and `daily`.
- **The first message,** `... available: see Settings > Firmware`, was cut at 48 bytes by the notification's own limit; it now reads `v0.11.0 is out: see Settings > Firmware`.

### Checks on the device (2026-10-06, Debug Builds of branch `gitea-updates`)

| Check | Result |
|---|---|
| Host tests | 456 pass |
| Check and list against the live server | The certificate is accepted against the two embedded roots; `releases/latest` read; a list of ten (33 KB) streamed |
| Servers that must be refused | github.com, example.com, expired.badssl.com, self-signed.badssl.com, wrong.host.badssl.com, untrusted-root.badssl.com and the router: each "isn't accepted" or a TLS error |
| A download cut short at 800,000 bytes | Refused, "update file too short"; the running firmware untouched |
| One byte flipped in the signature | Refused after 160 bytes, "bad signature"; the image isn't read further |
| One byte flipped in the image | Downloaded in full, refused at its end, "image corrupted (hash mismatch)" |
| The real v0.10.0, from the console and then from the screen | Downloaded, restarted, confirmed on Probation: the slot table read `v0.10.0, valid` both times. The Debug Build was pushed back from the PC after each |
| The screens | Latest release (checking, then `(current)` or `(new)`), the release page with the tag's message, Older releases with ten rows, the install dialog (Cancel by default, Back cancels), the progress screen at 28% |
| IRC connected, before the hold | A check left 3 KB of heap; a full download, 836 bytes |
| IRC connected, with the hold | The lowest free heap during a full download: 38 KB. IRC reconnected afterwards (its counters kept growing) |
| The daily check | It ran by itself, announced `v0.10.0 is out: see Settings > Firmware` once; with IRC connected (68 KB free) it didn't run |
| Speed | 1.9 MB in about 46 s, 40 KB/s, over the guest Wi-Fi at -65 dBm; not investigated further |

**Not checked:** the certificate's **name** on its own. Connecting by IP makes the server end the handshake before it shows its certificate, so that test proved nothing; the library sets the name it verifies, and OpenSSL on the PC refused the wrong name against the same chain. A failed daily check retrying, the clock not being set, the release that failed before not being announced (host-tested, not on the device), and the hold when IRC isn't connected but Gemini holds memory.

**Limits worth knowing:**
- **With IRC connected for days, the daily check doesn't run.** It would have to take IRC down to make room. Opening Latest release does.
- **A server that changes CA can't be reached** until a firmware carrying the new root comes from the PC (ADR 0009).
- **No resuming:** a broken download starts over (#54).
- **A key press during the hold:** Back on the Firmware page while a check is going doesn't cancel it.

**Two slips during the checks:** a blind sequence of keys on the Firmware page opened the SD card's install dialog (the page keeps its selection between visits); it was cancelled with Back, nothing installed. And my port-polling while waiting for a restart took the Debug Console's only client slot, which made the first install attempt look like a failure.

## One firmware: the Debug Console in every build (issue #68)

The issue asked for a token that could be set, so that Debug Builds could be published. The design round ended somewhere simpler: no Debug Builds. The console is in every firmware, off until switched on, with a token that belongs to the device (ADR 0010, which supersedes part of ADR 0004).

### Decisions (design round 2026-10-06)

| # | Decision |
|---|---|
| Q188 | **One firmware,** with the Debug Console and the test commands compiled in. The `cardputer-adv-debug` environment, `RORO_DEBUG`, the `+debug` version and `scripts/debug_flags.py` go; CI builds one firmware. Revises Q153 and Q171. |
| Q189 | **Off by default,** and off when the stored setting is missing or invalid. Off, nothing listens and no memory is used: the task and the 4 KB ring exist only while it's on. |
| Q190 | **Settings → Debug Console:** the switch, where to connect, the token, "New token", "Type a token". It stays on across restarts and in Safe Mode. **`DBG` in the Status Bar** while it listens, bright with a client connected. |
| Q191 | **The device makes the token** the first time the console is switched on: 100 bits as 20 characters of Crockford's base32, shown in groups of four. It can be replaced by one typed by hand, of at least 16 characters. Case and dashes don't count. |
| Q192 | **USB serial can set it up:** `debug on`, `debug token <value>`, `debug token new`, over USB only; `debug status` and `debug off` from anywhere. `scripts/flash.sh --debug` gives a device the developer's token. The token is never printed. |
| Q193 | **Challenge and answer:** the device sends 16 random bytes, the client their HMAC-SHA256 keyed by the token. Five wrong answers in a row close the console for a minute, with a Notification. **Old clients and old firmwares don't talk to each other:** accepted, this far from v1 and with one user. |
| Q194 | `rdbg.py` takes the token from **`-t`/`--token`**, then **`$RORO_DEBUG_TOKEN`**, then `~/.config/roro9stack/debug-token`. |
| Q195 | **The test commands are in every build** (`crash abort`, `crash wdt`, `update pretend`, `update damage`, `lora inject`, `sd fill`, `loop spin`, `wifi ip … try`). `update install … force` goes: there's nothing left to protect. |

### As built

- **`lib/debug`** (host-tested, 9 tests): the token maker, tidying what a person types, HMAC-SHA256 on the project's own SHA-256 (checked against RFC 4231), the answer to a challenge (the same vector as `rdbg.py`'s), a comparison that takes the same time whatever it's given, and the pause after five wrong answers, right across the clock's wrap.
- **Two settings,** `DebugConsole` and `DebugToken`, with the others: validated, and invalid stored values count as missing.
- **The console's task and ring come and go with the setting.** `Console::openRing` allocates the ring; switching off closes the socket, frees it and ends the task. `tick` follows the settings once a second, so a switch off and on again takes a second or two.
- **Measured against the two builds it replaces:** 1,881,799 bytes of flash and 54,700 of static RAM. The release build was 1,851,387 and 54,612 (so 30 KB more flash and 88 bytes more RAM); the Debug Build was 1,874,887 and 58,780 (4 KB of RAM back: the ring is no longer a static array).
- **The listener is asked whether it listens.** The framework's `begin()` returns nothing and fails without a word; the task now checks, says so on the console, and tries again every two seconds.
- **`debug off <seconds>`** closes the console and brings it back by itself: the only way to try its closing and reopening from afar, since switching it on is for the device and the cable only.
- **`scripts/rdbg.py`** answers the challenge, and fetches a release's ELF from Gitea when a crash names a version that isn't in `.pio/elves/`.

### Checks

| Check | Result |
|---|---|
| Host tests | 468 pass (456 before) |
| The firmware builds | One environment, 1,881,799 bytes of flash used |
| Pushed over Wi-Fi to a device running a Debug Build | Installed, restarted; the update port answers and **port 2323 refuses connections**: off by default |
| Switched on at the device (Settings → Debug Console) | A token is made and shown. Its first drawing, in bold at normal size, was misread once: it is now at twice the size, in two lines, and `O`, `I` and `L` are taken for `0` and `1` |
| A login with `scripts/rdbg.py` | The challenge is answered; `info`, `screenshot` and the backlog work |
| `DBG` in the Status Bar | There while the console is on, bright while a client is connected (screenshots) |
| `debug on` and `debug token` over the console | Refused: `over USB serial only` |
| Six logins with a wrong token | Five are refused, a second apart; the sixth, and the right token after it, get `locked`. After a minute the right token works again. The console's own log and a Notification say so |
| Three `crash abort` in a row | **Safe Mode**, with the console reachable: `info` works, `ls /` says `not available in Safe Mode`. `rdbg.py crash` decodes the backtrace to `runCommand` at the `abort()` line. `reboot` leaves Safe Mode |
| An update over Wi-Fi with the console on | The setting and the token survive: the console is back by itself after the restart |
| `debug off` over the console | The client is dropped and port 2323 refuses connections; the update port still answers |
| `debug off 2`, 10 times in a row, then 15 | It came back each time, a second after the pause. Free heap dips by about 270 bytes for each connection the device closes and **is all back two minutes later** (107.2 KB before, 103.2 right after 15, 107.0 at two minutes): TCP holds a closed connection that long. Not a leak, though it looked like one for an hour |
| Memory, console on with a client | 108 KB free (the Debug Build it replaces: 104 KB). In Safe Mode: 178 KB free |

**One false alarm:** after the tests the console "wouldn't come back on". It was off: the setting had been left off by `debug off`, and the page's "Switch it on?" dialog opens on **Cancel**, so Enter twice leaves it off. Nothing was lost.

**Not checked:** `scripts/flash.sh --debug` over USB (no device on USB here); "New token" and "Type a token" from the page (the code paths are the ones `debug token` uses, which need the cable); the Toast itself on screen (the console is closed while it shows; its Notification is in the log); free memory with the console off, which only the serial port could say (the static figures above are the evidence); fetching a release's ELF, which needs a crash on a released version.

## CI that doesn't rebuild the world (issue #74)

A pull request's run took over seven minutes, a tag's thirteen and a half. The goal: a firmware build under a minute.

### Where the time went (run 81, a pull request, 2026-10-06)

| Step | Time |
|---|---|
| Tools (apt, pip) | 15 s |
| Check out | 2 s |
| Host tests and coverage | 55 s |
| **The firmware** | **358 s** |
| of which: CMake configuring ESP-IDF | 87 s |
| of which: compiling ESP-IDF's libraries | 171 s |
| of which: our own build (the Arduino core, the libraries, `src/`) | 91 s |

A tag's run did the firmware step, then built the same commit again for the release: twice 356 s.

### Why the framework was rebuilt every time

The framework is rebuilt with our SDK settings (ADR 0006), and the rebuilt libraries stay in the toolchain volume. But the platform decides whether they match by reading **`sdkconfig.defaults` in the project folder**, whose first line carries a hash of the settings. That file is generated, and not in git. A fresh checkout has none, so the platform concluded "different settings", **reinstalled the framework and rebuilt it**: 260 seconds, at every run, to arrive at the libraries that were already there. On a developer's machine the file is simply still there from the last build, which is why nobody saw it.

### What changed

| Change | Effect |
|---|---|
| **The file is kept in the volume, inside the libraries it describes** (`framework-arduinoespressif32-libs/.roro-sdkconfig.defaults`), copied into the checkout before a build and back after one that passed. The platform still checks its hash against `platformio.ini`: changed settings rebuild, as they must. Kept there and not beside them, it disappears when the libraries are reinstalled, so it can't describe libraries that are gone | 358 s to 92 s |
| **The version is no longer a `-D` on every compiler command line.** `scripts/version.py` writes `lib/version/src/version_generated.h` (not in git, written only when it changes), read by one file. Before, every commit recompiled everything, on a developer's machine too, and no cache could have helped | A rebuild with nothing changed: 77 s to 13 s, locally |
| **PlatformIO's build cache** (`PLATFORMIO_BUILD_CACHE_DIR`, SCons's CacheDir) in the volume, for the firmware of pull requests: objects by the signature of their sources and command line | 92 s to 26 s, with a new version and one changed file |
| **ccache for the host tests.** They are built with coverage counters, and the build cache would return objects without their `.gcno` files; ccache keeps both | 49 s to 33 s. What's left is PlatformIO starting 51 test programs |
| **A tag builds its firmware once**, in the release step | minus 6 minutes |
| PlatformIO and gcovr in a virtual environment in the volume | a few seconds |

**A release compiles its own sources from nothing:** it reuses the rebuilt framework (the platform checks the hash) but not the build cache, so no published file contains an object that came from another commit's build.

### Measured (a development machine, fresh copies of the tree, the same volume)

| | Before | After |
|---|---|---|
| The firmware, fresh checkout, nothing cached for it | 358 s | 82 s (it fills the cache) |
| The firmware, fresh checkout, a new version and one file changed | 358 s | **27 s** |
| Host tests and coverage | 49 s | 33 s |
| Rebuilding locally with nothing changed | 77 s | 13 s |

### Measured on the runner (pull request #76, 2026-10-07)

| Run | Tools | Tests and coverage | The firmware | The whole job |
|---|---|---|---|---|
| Before (run 81) | 15 s | 55 s | 358 s | 434 s |
| The first with the new workflow: no mark yet, the framework is rebuilt once more and the caches fill | 16 s | 59 s | 354 s | 431 s |
| The next commit (only the workflow changed) | 10 s | 36 s | **51 s** | **100 s** |
| The same commit again | 10 s | 36 s | **18 s** | **66 s** |

In the 51-second run, 245 objects came from the cache and 43 were compiled: `version.cpp`, as expected, and all 42 files of `src/`, which had not changed. In the run after it, all 290 came from the cache. So the objects of `src/` made by the run that rebuilt the framework were not reusable by a normal run, and those of a normal run are: the two-pass build that rebuilds the framework compiles `src/` with something different on its command line. It costs one 51-second run after each framework rebuild, which is rare; I did not look for what differs.

The firmware step with everything cached is 18 seconds: the libraries are downloaded and unpacked (4 s), the dependency scan (5 s), fetching 290 objects, the link and the image (the last 11 s). A pull request that changes a few files should land between that and 51 seconds.

**What it costs:** the build cache grows by about 40 MB a run (each linked firmware is kept) and is started again past 3 GB; ccache is held to 1 GB.