CI: runs are too slow; investigate the build cache #74

New Issue

2026-10-06T23:37:33Z

twisla commented

2026-10-06 23:37:33 +00:00

CI runs take too long. Measured so far (docs/milestones/R1.md and this week's runs):

Run	Time
A push to `main` (host tests and coverage only)	about 1 min
A pull request (tests, coverage, the firmware)	about 9 min
A tag (all of it, then sign and publish)	11 to 13.5 min: v0.12.0 took 13.5 min from the tag to the published release

For comparison, on the development machine an incremental firmware build takes 77 seconds, and the same build from a fresh checkout takes 6 minutes (362 s, measured on 2026-10-06 from an exported copy of main).

The likely cause is that difference: the framework is rebuilt with our own SDK settings (custom_sdkconfig, ADR 0006, pioarduino's "hybrid compile"), and the result lives in the checkout's .pio/ build folder. Every CI run starts from a fresh checkout, so it pays the framework rebuild every time. The Docker volume roro9stack-pio keeps the toolchains and packages, but not that.

To investigate, in this order:

Where the time goes. Timestamps for each step of one pull-request run and one tag run: checkout, tools, host tests, coverage (which builds the tests a second time), the framework rebuild, our own sources, signing, upload.
What the volume keeps and what it doesn't. Is the rebuilt framework reused between runs when platformio.ini hasn't changed? If not, can it be (a build folder in the volume, keyed by a hash of platformio.ini and the SDK settings, so a changed setting still rebuilds)?
A compiler cache (ccache or sccache) in the volume, for our own sources and the framework's.
Doing less: coverage only on main rather than on every pull request, or tests and the firmware as two jobs in parallel if the runner has the cores (4).
A release build that reuses the pull request's. The tag builds the same commit that the merged pull request already built.

Constraints: a release must still be reproducible from its tag (a stale cache must never end up in a published file: the key has to cover everything that changes the output, or releases build clean and only pull requests use the cache); and the runner executes jobs on its own host with access to the signing secret (ADR 0008), so nothing from a fork's pull request may write to a cache a release reads.

Done when (to be refined): a pull request's run takes under 3 minutes when platformio.ini hasn't changed, the tag run's time is known step by step, and R1.md says what is cached, where, and what invalidates it.

CI runs take too long. Measured so far (docs/milestones/R1.md and this week's runs): | Run | Time | |---|---| | A push to `main` (host tests and coverage only) | about 1 min | | A pull request (tests, coverage, the firmware) | about 9 min | | A tag (all of it, then sign and publish) | 11 to 13.5 min: v0.12.0 took 13.5 min from the tag to the published release | For comparison, on the development machine an incremental firmware build takes **77 seconds**, and the same build from a fresh checkout takes **6 minutes** (362 s, measured on 2026-10-06 from an exported copy of `main`). **The likely cause** is that difference: the framework is rebuilt with our own SDK settings (`custom_sdkconfig`, ADR 0006, pioarduino's "hybrid compile"), and the result lives in the checkout's `.pio/` build folder. Every CI run starts from a fresh checkout, so it pays the framework rebuild every time. The Docker volume `roro9stack-pio` keeps the toolchains and packages, but not that. **To investigate, in this order:** 1. **Where the time goes.** Timestamps for each step of one pull-request run and one tag run: checkout, tools, host tests, coverage (which builds the tests a second time), the framework rebuild, our own sources, signing, upload. 2. **What the volume keeps and what it doesn't.** Is the rebuilt framework reused between runs when `platformio.ini` hasn't changed? If not, can it be (a build folder in the volume, keyed by a hash of `platformio.ini` and the SDK settings, so a changed setting still rebuilds)? 3. **A compiler cache** (ccache or sccache) in the volume, for our own sources and the framework's. 4. **Doing less:** coverage only on `main` rather than on every pull request, or tests and the firmware as two jobs in parallel if the runner has the cores (4). 5. **A release build that reuses the pull request's.** The tag builds the same commit that the merged pull request already built. **Constraints:** a release must still be reproducible from its tag (a stale cache must never end up in a published file: the key has to cover everything that changes the output, or releases build clean and only pull requests use the cache); and the runner executes jobs on its own host with access to the signing secret (ADR 0008), so nothing from a fork's pull request may write to a cache a release reads. **Done when (to be refined):** a pull request's run takes under 3 minutes when `platformio.ini` hasn't changed, the tag run's time is known step by step, and R1.md says what is cached, where, and what invalidates it.

twisla added this to the R1 Releases milestone 2026-10-06 23:37:33 +00:00

twisla added the

labels 2026-10-06 23:37:33 +00:00

twisla referenced this issue

2026-10-07 00:10:36 +00:00

CI: stop rebuilding the framework at every run; a build cache and ccache (#74) #76

twisla referenced this issue from a commit

2026-10-07 00:10:37 +00:00

CI: stop rebuilding the framework at every run; a build cache, ccache, and the version out of the compiler flags (#74)

twisla referenced this issue from a commit

2026-10-07 02:08:51 +00:00

Merge pull request 'CI: stop rebuilding the framework at every run; a build cache and ccache (#74)' (#76) from ci-speed into main

twisla commented

2026-10-07 02:08:52 +00:00

Done in pull request #76, merged into main. The investigation and every measurement are in docs/milestones/R1.md.

The cause: the firmware step took 358 s, of which 260 s rebuilt the framework that was already in the volume. The platform decides whether it matches by sdkconfig.defaults in the project folder, which is generated and not in git: no fresh checkout had it, so every run reinstalled and rebuilt the framework.

What changed: that file is kept in the volume, inside the libraries it describes; the version is out of the compiler flags (one generated header); PlatformIO's build cache for pull requests' firmware; ccache for the host tests; a tag builds its firmware once.

On the runner:

Run	Tests	Firmware	The whole job
Before	55 s	358 s	434 s
After, a commit that changed `src/` or followed a framework rebuild	36 s	51 s	100 s
After, everything cached	36 s	18 s	66 s

The target was a build under 60 seconds: 18 to 51.

Left as it is, on purpose: a release compiles its own sources from nothing (it reuses the rebuilt framework, not the build cache), so a tag's build stays around 90 seconds. Not measured: a tag's run, expected around 2.5 minutes instead of 13.5.

Not looked into: why the objects of src/ made by a run that rebuilds the framework can't be reused by the next one (it costs one 51-second run, rarely); and the tests' 36 seconds, most of it PlatformIO starting 51 test programs one by one.

Done in pull request #76, merged into `main`. The investigation and every measurement are in `docs/milestones/R1.md`. **The cause:** the firmware step took 358 s, of which 260 s rebuilt the framework that was already in the volume. The platform decides whether it matches by `sdkconfig.defaults` in the project folder, which is generated and not in git: no fresh checkout had it, so every run reinstalled and rebuilt the framework. **What changed:** that file is kept in the volume, inside the libraries it describes; the version is out of the compiler flags (one generated header); PlatformIO's build cache for pull requests' firmware; ccache for the host tests; a tag builds its firmware once. **On the runner:** | Run | Tests | Firmware | The whole job | |---|---|---|---| | Before | 55 s | 358 s | 434 s | | After, a commit that changed `src/` or followed a framework rebuild | 36 s | 51 s | 100 s | | After, everything cached | 36 s | 18 s | 66 s | The target was a build under 60 seconds: 18 to 51. **Left as it is, on purpose:** a release compiles its own sources from nothing (it reuses the rebuilt framework, not the build cache), so a tag's build stays around 90 seconds. **Not measured:** a tag's run, expected around 2.5 minutes instead of 13.5. **Not looked into:** why the objects of `src/` made by a run that rebuilds the framework can't be reused by the next one (it costs one 51-second run, rarely); and the tests' 36 seconds, most of it PlatformIO starting 51 test programs one by one.

twisla closed this issue

2026-10-07 02:08:53 +00:00

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: twisla/roro9stack#74