Rebuild the framework with smaller TLS buffers (ADR 0006)

custom_sdkconfig makes pioarduino regenerate the ESP-IDF libraries:
asymmetric TLS buffers (16 KB in, 4 KB out) and dynamic buffers that
free handshake-only data once connected. The rebuild also follows the
board: no PSRAM, 8 MB flash. The project now carries the partition
table the hybrid build needs (identical to the framework's: the app
slots must not move under updates over the air). Generated files are
ignored.

Debug Build, GNSS on, IRC on TLS: 78 KB free and a 59 KB low (M2 began
at 31 KB and 12.6 KB; the floor is 40 KB). The full stress set passes
on it: refused installs over Wi-Fi and from SD, put/get, screenshot,
core dump decode, Probation; under all of it at once, the low is 46 KB.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT

.gitignore

@@ -4,3 +4,8 @@
 
 # Private signing keys never belong in the repository (ADR 0003)
 *key.pem
+
+# Generated by pioarduino's hybrid compile from custom_sdkconfig (platformio.ini)
+.dummy/
+managed_components/
+sdkconfig.*

README.md

@@ -18,6 +18,8 @@ scripts/ci.sh
 
 This runs the host-side unit tests (`test/`, `native` environment), then builds the firmware. The output is `.pio/build/cardputer-adv/firmware.factory.bin`.
 
+The framework is rebuilt with the TLS settings in `platformio.ini` (`custom_sdkconfig`, ADR 0006), so the first build after a fresh checkout takes about 4 minutes; later builds take under a minute.
+
 ## Flash
 
 1. Connect the Cardputer by USB-C.

default_8MB.csv

@@ -0,0 +1,7 @@
+# Name,   Type, SubType, Offset,  Size, Flags
+nvs,      data, nvs,     0x9000,  0x5000,
+otadata,  data, ota,     0xe000,  0x2000,
+app0,     app,  ota_0,   0x10000, 0x330000,
+app1,     app,  ota_1,   0x340000,0x330000,
+spiffs,   data, spiffs,  0x670000,0x180000,
+coredump, data, coredump,0x7F0000,0x10000,

docs/adr/0006-framework-rebuilt-for-smaller-tls-buffers.md

@@ -0,0 +1,19 @@
+# The framework is rebuilt with our own SDK settings, for smaller TLS buffers
+
+Arduino-ESP32 ships its ESP-IDF libraries prebuilt, with one `sdkconfig` for every ESP32-S3 board. Its TLS settings give every connection a 16 KB receive buffer and a 16 KB send buffer for its whole life. On a device with no PSRAM and about 340 KB of RAM, an IRC connection over TLS left a 12.6 KB low in M2, against a 40 KB floor.
+
+Those settings are compiled into the libraries, so changing them means rebuilding them. pioarduino supports this as a "hybrid compile": `custom_sdkconfig` in `platformio.ini` lists the settings, and the build regenerates the framework's libraries from ESP-IDF (the same 5.5.5 the prebuilt ones come from) before building the app. We set:
+
+- `MBEDTLS_ASYMMETRIC_CONTENT_LEN`, with 16 KB to receive (servers send full TLS records) and **4 KB to send** (IRC lines are short): 12 KB less per connection.
+- `MBEDTLS_DYNAMIC_BUFFER`, `DYNAMIC_FREE_CONFIG_DATA`, `DYNAMIC_FREE_CA_CERT`: buffers allocated when needed, and handshake-only data (the CA chain) freed once connected.
+
+The rebuild also follows the board definition instead of the generic one: PSRAM support is off (the Cardputer ADV has none) and the flash size is 8 MB.
+
+## Consequences
+
+- With IRC connected over TLS, a Debug Build has 78 KB free and a 59 KB low (it was 31 KB and 12.6 KB), and 46 KB at the lowest under the heaviest combined load measured (IRC, two refused installs, a 1.6 MB put and get).
+- The first build after a fresh checkout, or after changing `custom_sdkconfig`, takes about 4 minutes instead of 45 s: it downloads ESP-IDF into the PlatformIO volume and compiles it. Later builds reuse it.
+- Everything the firmware depends on was checked in the regenerated `sdkconfig`: app rollback, core dumps to flash (ELF), the 5 s task watchdog, FreeRTOS run-time stats, the certificate bundle.
+- The project now owns its partition table (`default_8MB.csv`, identical to the framework's), which the hybrid build requires. Changing it would break updates over the air: the app slots must stay where they are.
+- Generated files (`sdkconfig.*`, `managed_components/`, `.dummy/`) are ignored by git.
+- A TLS server that sends records over 16 KB would still fail, as before; one that needs us to send records over 4 KB would now fail. Neither happens with IRC.

docs/milestones/M2.md

@@ -1,12 +1,21 @@
 # M2 — GNSS
 
-**Status:** steps 1–6 done on the device (branch `m2`). Open: the heap floor during a TLS handshake (below).
+**Status:** done on the device (branch `m2`): every "Done when" item below is met.
 
 ## Measured
 
 - **Cold start** (`$PCAS10,2`) to a 3D Fix, by a window: **73 s**, 5 satellites used of 8 in view. A restart of the ESP32 alone keeps the receiver's Fix (the Cap stays powered).
 - By a window: 3D Fix from GPS, GLONASS, Galileo and BeiDou, up to 14 of 17 satellites used, HDOP 1.0–1.3.
-- **Heap, Debug Build, GNSS on:** with IRC on TLS, first 33 KB free and a 12.6 KB low (floor 40 KB; v0.2.1 had a 79 KB low). Not a GNSS cost: the OTA and Debug Build work added task stacks, a console ring, serial buffers, mDNS and two TCP servers. **Trimmed by measurement:** each task's peak stack was measured through its worst case (an ECDSA-checked install over Wi-Fi and from SD, get/put, a core dump fetch, an IRC TLS handshake), then stacks were set to peak plus about 2 KB: loop 8→6 KB, update 8→5, storage 10→6, irc 8→6; the console ring 6→4 KB, serial TX 2→1 KB, GNSS UART 1 KB→512 B. **After:** 46 KB free with IRC connected, an 18 KB low. The steady state clears the floor; the TLS handshake peak doesn't yet. Next candidates: mbedTLS dynamic buffers (custom sdkconfig), one network task for the update and debug listeners, mDNS on demand.
+- **Heap, Debug Build, GNSS on, IRC on TLS** (floor 40 KB; v0.2.1 had a 79 KB low):
+
+  | | Free | Lowest |
+  |---|---|---|
+  | Start of M2 | 31 KB | 12.6 KB |
+  | Stacks and buffers trimmed by measurement | 46 KB | 18 KB |
+  | mDNS removed | 54 KB | 34 KB |
+  | Framework rebuilt with smaller TLS buffers (ADR 0006) | **78 KB** | **59 KB** |
+
+  Under the heaviest combined load measured (IRC, two refused installs, a 1.6 MB put and get), the low is 46 KB. The cost had come mostly from the OTA and Debug Build work, not GNSS. Stacks were set to measured peak plus about 2 KB (loop 6 KB, update 5, storage 6, irc 6); after a TLS handshake the irc task has 1.7 KB left. IRC can now be stopped by hand (`/quit` in any state, `irc stop`), which frees its TLS memory.
 
 **Goal:** the device knows where it is and what time it is without a network: a GNSS Service in the background, a GNSS App with the position and a sky view of the satellites, the clock set from satellites when there's no NTP, and Tracks recorded to the SD card.
 

platformio.ini

@@ -20,6 +20,16 @@ build_flags =
 lib_deps =
     m5stack/M5Cardputer @ 1.1.1
 test_ignore = *
+; Smaller TLS buffers (M2): the framework is rebuilt with these settings (pioarduino "hybrid
+; compile"). Receive stays 16 KB (servers send full TLS records); send drops to 4 KB (IRC lines are
+; short); buffers are allocated as needed and handshake-only data is freed once connected.
+custom_sdkconfig =
+    CONFIG_MBEDTLS_ASYMMETRIC_CONTENT_LEN=y
+    CONFIG_MBEDTLS_SSL_IN_CONTENT_LEN=16384
+    CONFIG_MBEDTLS_SSL_OUT_CONTENT_LEN=4096
+    CONFIG_MBEDTLS_DYNAMIC_BUFFER=y
+    CONFIG_MBEDTLS_DYNAMIC_FREE_CONFIG_DATA=y
+    CONFIG_MBEDTLS_DYNAMIC_FREE_CA_CERT=y
 
 ; Debug Build: the same firmware plus the Debug Console on TCP 2323 (see ADR 0004). The token comes
 ; from ~/.config/roro9stack/debug-token, passed in by scripts/_docker.sh; it's never committed.