Merge pull request 'CI: stop rebuilding the framework at every run; a build cache and ccache (#74)' (#76) from ci-speed into main

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT

.gitea/workflows/ci.yml

@@ -1,14 +1,31 @@
 # CI and releases (docs/milestones/R1.md).
 #   A push to main:    the host tests, with their coverage of lib/, and the README's badges
 #                      published to the branch `badges`.
-#   A pull request:    the same tests and coverage, then the firmware.
+#   A pull request:    the same tests and coverage, then the firmware (from the build cache).
 #                      A branch's pushes run nothing by themselves: its pull request runs, once.
-#   A tag v*:          all of it, then a Gitea release with the signed Update File.
+#   A tag v*:          the tests, then the firmware built once, clean, signed and published as a
+#                      Gitea release.
 #   Run by hand:       the release of a tag that exists already (the ones from before CI).
 #
 # The job runs in a plain Python image, as scripts/ci.sh does on a developer's machine, with the
 # toolchains in a Docker volume the runner allows (container.valid_volumes: roro9stack-pio): that
 # volume is the cache. No JavaScript actions, so the image needs no Node: the checkout is git.
+#
+# What the volume keeps between runs, and what makes each go stale (issue #74, R1.md):
+#   /pio/packages, /pio/platforms   toolchains and the framework: by their versions in platformio.ini
+#   .../framework-arduinoespressif32-libs/.roro-sdkconfig.defaults
+#                                   the mark that the framework is already rebuilt with our SDK settings: the
+#                                   project's sdkconfig.defaults, which isn't in git, so that every fresh
+#                                   checkout rebuilt the framework (260 s). The platform checks its hash
+#                                   against platformio.ini's settings, and rebuilds if they differ. It is kept
+#                                   inside the libraries it describes, so it goes when they are reinstalled.
+#   /pio/ci/build-cache             PlatformIO's build cache (SCons): objects by the signature of their
+#                                   sources and command line. For pull requests only: a release compiles
+#                                   its own sources from nothing.
+#   /pio/ci/ccache                  the host tests' objects (they're built for coverage, which the build
+#                                   cache can't keep: it would lose the .gcno files)
+#   /pio/ci/venv                    PlatformIO and gcovr: delete the folder to upgrade them
+# To start from nothing (a slow run, about 7 minutes): delete /pio/ci and that .roro-sdkconfig.defaults file.
 name: CI
 on:
   push:
@@ -37,13 +54,23 @@ jobs:
     env:
       PLATFORMIO_CORE_DIR: /pio
       RORO_NO_DOCKER: 1
+      SDK_MARK: /pio/packages/framework-arduinoespressif32-libs/.roro-sdkconfig.defaults
+      CCACHE_DIR: /pio/ci/ccache
+      CCACHE_MAXSIZE: 1G
     steps:
       - name: Tools
         run: |
           apt-get update -qq
-          apt-get install -y -qq --no-install-recommends git build-essential openssl >/dev/null
+          apt-get install -y -qq --no-install-recommends git build-essential openssl ccache >/dev/null
-          pip install -q --no-cache-dir --root-user-action=ignore platformio gcovr
+          mkdir -p /pio/ci
-          pio --version; df -h /pio | tail -1; ls /pio | head
+          if [ ! -x /pio/ci/venv/bin/pio ]; then
+            python -m venv /pio/ci/venv
+            /pio/ci/venv/bin/pip install -q --no-cache-dir platformio gcovr
+          fi
+          ln -sf /pio/ci/venv/bin/pio /pio/ci/venv/bin/gcovr /usr/local/bin/
+          pio --version; df -h /pio | tail -1; du -sh /pio/ci/* 2>/dev/null || true
+          # The build cache only grows: start it again past 3 GB (a full set of objects is 160 MB, and each run adds about 40).
+          if [ "$(du -sm /pio/ci/build-cache 2>/dev/null | cut -f1)" -gt 3072 ] 2>/dev/null; then rm -rf /pio/ci/build-cache; fi
 
       - name: Check out
         run: |
@@ -57,11 +84,20 @@ jobs:
 
       - name: Host tests, and their coverage of lib/
         if: github.event_name != 'workflow_dispatch'
-        run: scripts/coverage.sh
+        run: |
+          export PATH="/usr/lib/ccache:$PATH"  # gcc and g++ through ccache
+          scripts/coverage.sh
+          ccache -s | grep -E 'Hits|Misses' | head -2
 
+      # A pull request only: a tag's firmware is built once, by the release step below.
       - name: The firmware
-        if: github.event_name == 'pull_request' || github.ref_type == 'tag'
+        if: github.event_name == 'pull_request'
-        run: scripts/ci.sh builds
+        env:
+          PLATFORMIO_BUILD_CACHE_DIR: /pio/ci/build-cache
+        run: |
+          [ ! -f "$SDK_MARK" ] || cp "$SDK_MARK" sdkconfig.defaults
+          scripts/ci.sh builds
+          cp sdkconfig.defaults "$SDK_MARK"  # what the framework in the volume is rebuilt with, now
 
       # The README's badges are files on a branch of their own, replaced at each push to main and
       # at each tag (the release badge says which tag is the latest)
@@ -103,7 +139,12 @@ jobs:
           trap 'rm -f "$RORO_OTA_KEY"' EXIT
           printf '%s\n' "$OTA_SIGNING_KEY" > "$RORO_OTA_KEY"
           umask 022
+          # The framework rebuilt with our settings is reused if it matches (the platform checks);
+          # the release's own sources are compiled from nothing, with no build cache.
+          [ ! -f "$SDK_MARK" ] || cp "$SDK_MARK" /tmp/release-src/sdkconfig.defaults
           scripts/release_build.sh /tmp/release-src dist
+          # An old tag has no SDK settings of its own, and no sdkconfig.defaults afterwards: nothing to mark.
+          [ ! -f /tmp/release-src/sdkconfig.defaults ] || [ ! -d "$(dirname "$SDK_MARK")" ] || cp /tmp/release-src/sdkconfig.defaults "$SDK_MARK"
 
       - name: Publish the release
         if: steps.release.outputs.tag != ''

.gitignore

@@ -12,3 +12,6 @@ sdkconfig.*
 
 # The built site (site/config.toml sends it here)
 /public/
+
+# The version, written by scripts/version.py before each build
+lib/version/src/version_generated.h

README.md

@@ -26,7 +26,7 @@ This runs the host-side unit tests (`test/`, `native` environment), then builds
 
 `scripts/coverage.sh` runs the same tests with coverage counters and writes a line-by-line report to `.pio/coverage/index.html`. The badge above is its figure for `main`: the share of the lines of `lib/` that the host tests run. `lib/` is the logic that compiles on a PC; `lib/SD` (the card's driver) and `src/` (the Apps, the Services, everything that needs the device) have no host tests and aren't in that figure.
 
-The framework is rebuilt with the TLS settings in `platformio.ini` (`custom_sdkconfig`, ADR 0006), so the first build after a fresh checkout takes about 4 minutes; later builds take under a minute.
+The framework is rebuilt with the TLS settings in `platformio.ini` (`custom_sdkconfig`, ADR 0006), so the first build after a fresh checkout takes about 6 minutes; later builds take about 15 seconds when little has changed. The platform knows the framework is already rebuilt by `sdkconfig.defaults` in the project folder, which it writes and git ignores: delete it and the next build rebuilds the framework. CI keeps that file, a build cache and ccache in its volume (docs/milestones/R1.md).
 
 ## CI and releases
 

docs/milestones/R1.md

@@ -172,3 +172,62 @@ The issue asked for a token that could be set, so that Debug Builds could be pub
 **One false alarm:** after the tests the console "wouldn't come back on". It was off: the setting had been left off by `debug off`, and the page's "Switch it on?" dialog opens on **Cancel**, so Enter twice leaves it off. Nothing was lost.
 
 **Not checked:** `scripts/flash.sh --debug` over USB (no device on USB here); "New token" and "Type a token" from the page (the code paths are the ones `debug token` uses, which need the cable); the Toast itself on screen (the console is closed while it shows; its Notification is in the log); free memory with the console off, which only the serial port could say (the static figures above are the evidence); fetching a release's ELF, which needs a crash on a released version.
+
+## CI that doesn't rebuild the world (issue #74)
+
+A pull request's run took over seven minutes, a tag's thirteen and a half. The goal: a firmware build under a minute.
+
+### Where the time went (run 81, a pull request, 2026-10-06)
+
+| Step | Time |
+|---|---|
+| Tools (apt, pip) | 15 s |
+| Check out | 2 s |
+| Host tests and coverage | 55 s |
+| **The firmware** | **358 s** |
+| of which: CMake configuring ESP-IDF | 87 s |
+| of which: compiling ESP-IDF's libraries | 171 s |
+| of which: our own build (the Arduino core, the libraries, `src/`) | 91 s |
+
+A tag's run did the firmware step, then built the same commit again for the release: twice 356 s.
+
+### Why the framework was rebuilt every time
+
+The framework is rebuilt with our SDK settings (ADR 0006), and the rebuilt libraries stay in the toolchain volume. But the platform decides whether they match by reading **`sdkconfig.defaults` in the project folder**, whose first line carries a hash of the settings. That file is generated, and not in git. A fresh checkout has none, so the platform concluded "different settings", **reinstalled the framework and rebuilt it**: 260 seconds, at every run, to arrive at the libraries that were already there. On a developer's machine the file is simply still there from the last build, which is why nobody saw it.
+
+### What changed
+
+| Change | Effect |
+|---|---|
+| **The file is kept in the volume, inside the libraries it describes** (`framework-arduinoespressif32-libs/.roro-sdkconfig.defaults`), copied into the checkout before a build and back after one that passed. The platform still checks its hash against `platformio.ini`: changed settings rebuild, as they must. Kept there and not beside them, it disappears when the libraries are reinstalled, so it can't describe libraries that are gone | 358 s to 92 s |
+| **The version is no longer a `-D` on every compiler command line.** `scripts/version.py` writes `lib/version/src/version_generated.h` (not in git, written only when it changes), read by one file. Before, every commit recompiled everything, on a developer's machine too, and no cache could have helped | A rebuild with nothing changed: 77 s to 13 s, locally |
+| **PlatformIO's build cache** (`PLATFORMIO_BUILD_CACHE_DIR`, SCons's CacheDir) in the volume, for the firmware of pull requests: objects by the signature of their sources and command line | 92 s to 26 s, with a new version and one changed file |
+| **ccache for the host tests.** They are built with coverage counters, and the build cache would return objects without their `.gcno` files; ccache keeps both | 49 s to 33 s. What's left is PlatformIO starting 51 test programs |
+| **A tag builds its firmware once**, in the release step | minus 6 minutes |
+| PlatformIO and gcovr in a virtual environment in the volume | a few seconds |
+
+**A release compiles its own sources from nothing:** it reuses the rebuilt framework (the platform checks the hash) but not the build cache, so no published file contains an object that came from another commit's build.
+
+### Measured (a development machine, fresh copies of the tree, the same volume)
+
+| | Before | After |
+|---|---|---|
+| The firmware, fresh checkout, nothing cached for it | 358 s | 82 s (it fills the cache) |
+| The firmware, fresh checkout, a new version and one file changed | 358 s | **27 s** |
+| Host tests and coverage | 49 s | 33 s |
+| Rebuilding locally with nothing changed | 77 s | 13 s |
+
+### Measured on the runner (pull request #76, 2026-10-07)
+
+| Run | Tools | Tests and coverage | The firmware | The whole job |
+|---|---|---|---|---|
+| Before (run 81) | 15 s | 55 s | 358 s | 434 s |
+| The first with the new workflow: no mark yet, the framework is rebuilt once more and the caches fill | 16 s | 59 s | 354 s | 431 s |
+| The next commit (only the workflow changed) | 10 s | 36 s | **51 s** | **100 s** |
+| The same commit again | 10 s | 36 s | **18 s** | **66 s** |
+
+In the 51-second run, 245 objects came from the cache and 43 were compiled: `version.cpp`, as expected, and all 42 files of `src/`, which had not changed. In the run after it, all 290 came from the cache. So the objects of `src/` made by the run that rebuilt the framework were not reusable by a normal run, and those of a normal run are: the two-pass build that rebuilds the framework compiles `src/` with something different on its command line. It costs one 51-second run after each framework rebuild, which is rare; I did not look for what differs.
+
+The firmware step with everything cached is 18 seconds: the libraries are downloaded and unpacked (4 s), the dependency scan (5 s), fetching 290 objects, the link and the image (the last 11 s). A pull request that changes a few files should land between that and 51 seconds.
+
+**What it costs:** the build cache grows by about 40 MB a run (each linked firmware is kept) and is started again past 3 GB; ccache is held to 1 GB.

lib/version/src/version.cpp

@@ -0,0 +1,15 @@
+#include "version.h"
+
+// Written by scripts/version.py before each build; not in git.
+#if __has_include("version_generated.h")
+#include "version_generated.h"
+#endif
+#ifndef RORO_VERSION
+#define RORO_VERSION "unknown"
+#endif
+
+namespace roro {
+
+const char* versionString() { return RORO_VERSION; }
+
+}  // namespace roro

lib/version/src/version.h

@@ -1,14 +1,12 @@
 #pragma once
 
-#ifndef RORO_VERSION
-#define RORO_VERSION "unknown"
-#endif
-
 namespace roro {
 
 constexpr const char* kProductName = "roro9stack";
 
-// "roro9stack v0.1.0" — used on the boot screen and in About.
+// "v0.1.0", from `git describe` (scripts/version.py): used on the boot screen and in About.
-inline const char* versionString() { return RORO_VERSION; }
+// A function in one file, not a macro on every compiler command line: a new commit then recompiles
+// that one file, and everything else comes from the build cache (issue #74).
+const char* versionString();
 
 }  // namespace roro

scripts/version.py

@@ -10,7 +10,15 @@ try:
 except Exception:
     version = "unknown"
 
-env.Append(CPPDEFINES=[("RORO_VERSION", '\\"%s\\"' % version)])  # noqa: F821
+# The version goes into one generated header, read by one file (lib/version/src/version.cpp). As a -D
+# on every command line it made each new commit recompile everything, and no build cache could help
+# (issue #74). Written only when it changes, so an unchanged version rebuilds nothing.
+import os
+
+_header = os.path.join(env.subst("$PROJECT_DIR"), "lib", "version", "src", "version_generated.h")  # noqa: F821
+_text = '#define RORO_VERSION "%s"\n' % version
+if not os.path.exists(_header) or open(_header).read() != _text:
+    open(_header, "w").write(_text)
 
 
 # Keep every build's ELF, named by version and the first 16 hex digits of its SHA-256 (the core dump

site/content/dev/build/build-and-test.md

@@ -22,7 +22,7 @@ This runs the host-side unit tests (`test/`, `native` environment), then builds
 
 `scripts/coverage.sh` runs the same tests with coverage counters and writes a line-by-line report to `.pio/coverage/index.html`. The badge above is its figure for `main`: the share of the lines of `lib/` that the host tests run. `lib/` is the logic that compiles on a PC; `lib/SD` (the card's driver) and `src/` (the Apps, the Services, everything that needs the device) have no host tests and aren't in that figure.
 
-The framework is rebuilt with the TLS settings in `platformio.ini` (`custom_sdkconfig`, ADR 0006), so the first build after a fresh checkout takes about 4 minutes; later builds take under a minute.
+The framework is rebuilt with the TLS settings in `platformio.ini` (`custom_sdkconfig`, ADR 0006), so the first build after a fresh checkout takes about 6 minutes; later builds take about 15 seconds when little has changed. The platform knows the framework is already rebuilt by `sdkconfig.defaults` in the project folder, which it writes and git ignores: delete it and the next build rebuilds the framework. CI keeps that file, a build cache and ccache in its volume (docs/milestones/R1.md).
 
 ## CI and releases
 

site/content/dev/milestones/r1.md

@@ -180,3 +180,62 @@ The issue asked for a token that could be set, so that Debug Builds could be pub
 **One false alarm:** after the tests the console "wouldn't come back on". It was off: the setting had been left off by `debug off`, and the page's "Switch it on?" dialog opens on **Cancel**, so Enter twice leaves it off. Nothing was lost.
 
 **Not checked:** `scripts/flash.sh --debug` over USB (no device on USB here); "New token" and "Type a token" from the page (the code paths are the ones `debug token` uses, which need the cable); the Toast itself on screen (the console is closed while it shows; its Notification is in the log); free memory with the console off, which only the serial port could say (the static figures above are the evidence); fetching a release's ELF, which needs a crash on a released version.
+
+## CI that doesn't rebuild the world (issue #74)
+
+A pull request's run took over seven minutes, a tag's thirteen and a half. The goal: a firmware build under a minute.
+
+### Where the time went (run 81, a pull request, 2026-10-06)
+
+| Step | Time |
+|---|---|
+| Tools (apt, pip) | 15 s |
+| Check out | 2 s |
+| Host tests and coverage | 55 s |
+| **The firmware** | **358 s** |
+| of which: CMake configuring ESP-IDF | 87 s |
+| of which: compiling ESP-IDF's libraries | 171 s |
+| of which: our own build (the Arduino core, the libraries, `src/`) | 91 s |
+
+A tag's run did the firmware step, then built the same commit again for the release: twice 356 s.
+
+### Why the framework was rebuilt every time
+
+The framework is rebuilt with our SDK settings (ADR 0006), and the rebuilt libraries stay in the toolchain volume. But the platform decides whether they match by reading **`sdkconfig.defaults` in the project folder**, whose first line carries a hash of the settings. That file is generated, and not in git. A fresh checkout has none, so the platform concluded "different settings", **reinstalled the framework and rebuilt it**: 260 seconds, at every run, to arrive at the libraries that were already there. On a developer's machine the file is simply still there from the last build, which is why nobody saw it.
+
+### What changed
+
+| Change | Effect |
+|---|---|
+| **The file is kept in the volume, inside the libraries it describes** (`framework-arduinoespressif32-libs/.roro-sdkconfig.defaults`), copied into the checkout before a build and back after one that passed. The platform still checks its hash against `platformio.ini`: changed settings rebuild, as they must. Kept there and not beside them, it disappears when the libraries are reinstalled, so it can't describe libraries that are gone | 358 s to 92 s |
+| **The version is no longer a `-D` on every compiler command line.** `scripts/version.py` writes `lib/version/src/version_generated.h` (not in git, written only when it changes), read by one file. Before, every commit recompiled everything, on a developer's machine too, and no cache could have helped | A rebuild with nothing changed: 77 s to 13 s, locally |
+| **PlatformIO's build cache** (`PLATFORMIO_BUILD_CACHE_DIR`, SCons's CacheDir) in the volume, for the firmware of pull requests: objects by the signature of their sources and command line | 92 s to 26 s, with a new version and one changed file |
+| **ccache for the host tests.** They are built with coverage counters, and the build cache would return objects without their `.gcno` files; ccache keeps both | 49 s to 33 s. What's left is PlatformIO starting 51 test programs |
+| **A tag builds its firmware once**, in the release step | minus 6 minutes |
+| PlatformIO and gcovr in a virtual environment in the volume | a few seconds |
+
+**A release compiles its own sources from nothing:** it reuses the rebuilt framework (the platform checks the hash) but not the build cache, so no published file contains an object that came from another commit's build.
+
+### Measured (a development machine, fresh copies of the tree, the same volume)
+
+| | Before | After |
+|---|---|---|
+| The firmware, fresh checkout, nothing cached for it | 358 s | 82 s (it fills the cache) |
+| The firmware, fresh checkout, a new version and one file changed | 358 s | **27 s** |
+| Host tests and coverage | 49 s | 33 s |
+| Rebuilding locally with nothing changed | 77 s | 13 s |
+
+### Measured on the runner (pull request #76, 2026-10-07)
+
+| Run | Tools | Tests and coverage | The firmware | The whole job |
+|---|---|---|---|---|
+| Before (run 81) | 15 s | 55 s | 358 s | 434 s |
+| The first with the new workflow: no mark yet, the framework is rebuilt once more and the caches fill | 16 s | 59 s | 354 s | 431 s |
+| The next commit (only the workflow changed) | 10 s | 36 s | **51 s** | **100 s** |
+| The same commit again | 10 s | 36 s | **18 s** | **66 s** |
+
+In the 51-second run, 245 objects came from the cache and 43 were compiled: `version.cpp`, as expected, and all 42 files of `src/`, which had not changed. In the run after it, all 290 came from the cache. So the objects of `src/` made by the run that rebuilt the framework were not reusable by a normal run, and those of a normal run are: the two-pass build that rebuilds the framework compiles `src/` with something different on its command line. It costs one 51-second run after each framework rebuild, which is rare; I did not look for what differs.
+
+The firmware step with everything cached is 18 seconds: the libraries are downloaded and unpacked (4 s), the dependency scan (5 s), fetching 290 objects, the link and the image (the last 11 s). A pull request that changes a few files should land between that and 51 seconds.
+
+**What it costs:** the build cache grows by about 40 MB a run (each linked firmware is kept) and is started again past 3 GB; ccache is held to 1 GB.