Screen recording: video "screenshots" of the display #17

New Issue

2026-10-05T12:03:46Z

twisla commented

2026-10-05 12:03:46 +00:00

Idea

Record what the screen shows as a video: an animated GIF or an MP4, at 2× like the screenshots. It could be streamed to a computer over Wi-Fi, recorded to the SD card, or both. Also, take still screenshots on the device, with a key, saved to the card, in any build.

Why

Blog posts, the website (#12) and bug reports (#4) would gain a lot from short clips: scrolling a Gemini page, the Sky view moving, the Launcher tiles (#9), a theme switch (#10).
Stills show a state; a clip shows behaviour, including bugs that only show up in motion.

What's known

Today. Screenshots exist only in Debug Builds: the Debug Console's screenshot command sends the 240×135 RGB332 frame (32,400 bytes) over TCP, and rdbg.py saves it as a PNG at 2×.
The frame is easy to get.
- The whole UI is drawn on one 8-bit RGB332 canvas, then pushed to the display.
- The Debug Console already has a pointer to it (setFrame, src/main.cpp:188).
- Capturing a frame right after Screen::render means no flicker and no tearing.
Streaming over Wi-Fi (Debug Builds).
- A record command streams frames to rdbg.py, which writes a GIF or MP4 with ffmpeg.
- Raw frames are 32 KB each, so 10 fps is 324 KB/s: too much for comfort next to IRC's TLS connection.
- But frames barely change. Sending only the changed rows, with RLE, should cut that by 10× or more on most screens.
- Timestamps come with each frame, so playback runs at real speed even when frames are skipped.
Recording to the card (any build).
- The same delta and RLE stream goes to a file (/screen/…), written through StorageService's task. A converter on the computer turns it into a GIF or MP4.
- Or encode a GIF directly on the device. RGB332 maps exactly onto a 256-colour GIF palette, so there's no colour quantisation. LZW needs about 16-20 KB of RAM while recording, which is possible but is a cost with IRC connected.
Still screenshots on the device. Write a PNG or BMP to /screen/ with a key combination. A BMP is trivial; a PNG needs zlib, which ESP-IDF's ROM already has (miniz).
The cost of recording.
- Copying and diffing the frame each time it's drawn costs CPU on the main loop. Measure the frame time with it on.
- The previous frame has to be kept for diffing (32 KB). That's significant against our floors, so it's only allocated while recording.
- The display itself isn't involved: we copy from the canvas, not the screen.
Privacy. A recording shows whatever is on screen, including passwords as they're typed (masked fields excepted), IRC messages and positions. Recording should be clearly visible, for example a red dot in the status bar.

Questions for the design round

Where to: over Wi-Fi to rdbg.py (Debug Builds), to the card (any build), or both?
What format comes out: GIF (fits RGB332 exactly, easy for blogs), MP4 (smaller), or our own stream plus a converter?
Encode the GIF on the device, or on the computer?
Which keys start and stop recording, and which take a still? Do these work in release builds?
Frame rate: fixed (10 fps) or every frame drawn?
Should the status bar's red dot be included in the recording or hidden from it?
Should keypresses be shown as an overlay, for tutorials and docs (#12)?
How long can a recording run: a time limit, or until the card's Logs limit?

## Idea Record what the screen shows as a video: an animated GIF or an MP4, at 2× like the screenshots. It could be streamed to a computer over Wi-Fi, recorded to the SD card, or both. Also, take still screenshots on the device, with a key, saved to the card, in any build. ## Why - Blog posts, the website (#12) and bug reports (#4) would gain a lot from short clips: scrolling a Gemini page, the Sky view moving, the Launcher tiles (#9), a theme switch (#10). - Stills show a state; a clip shows behaviour, including bugs that only show up in motion. ## What's known - **Today.** Screenshots exist only in Debug Builds: the Debug Console's `screenshot` command sends the 240×135 RGB332 frame (32,400 bytes) over TCP, and `rdbg.py` saves it as a PNG at 2×. - **The frame is easy to get.** - The whole UI is drawn on one 8-bit RGB332 canvas, then pushed to the display. - The Debug Console already has a pointer to it (`setFrame`, `src/main.cpp:188`). - Capturing a frame right after `Screen::render` means no flicker and no tearing. - **Streaming over Wi-Fi (Debug Builds).** - A `record` command streams frames to `rdbg.py`, which writes a GIF or MP4 with ffmpeg. - Raw frames are 32 KB each, so 10 fps is 324 KB/s: too much for comfort next to IRC's TLS connection. - But frames barely change. Sending only the changed rows, with RLE, should cut that by 10× or more on most screens. - Timestamps come with each frame, so playback runs at real speed even when frames are skipped. - **Recording to the card (any build).** - The same delta and RLE stream goes to a file (`/screen/…`), written through StorageService's task. A converter on the computer turns it into a GIF or MP4. - Or encode a GIF directly on the device. RGB332 maps exactly onto a 256-colour GIF palette, so there's no colour quantisation. LZW needs about 16-20 KB of RAM while recording, which is possible but is a cost with IRC connected. - **Still screenshots on the device.** Write a PNG or BMP to `/screen/` with a key combination. A BMP is trivial; a PNG needs zlib, which ESP-IDF's ROM already has (miniz). - **The cost of recording.** - Copying and diffing the frame each time it's drawn costs CPU on the main loop. Measure the frame time with it on. - The previous frame has to be kept for diffing (32 KB). That's significant against our floors, so it's only allocated while recording. - The display itself isn't involved: we copy from the canvas, not the screen. - **Privacy.** A recording shows whatever is on screen, including passwords as they're typed (masked fields excepted), IRC messages and positions. Recording should be clearly visible, for example a red dot in the status bar. ## Questions for the design round 1. Where to: over Wi-Fi to `rdbg.py` (Debug Builds), to the card (any build), or both? 2. What format comes out: GIF (fits RGB332 exactly, easy for blogs), MP4 (smaller), or our own stream plus a converter? 3. Encode the GIF on the device, or on the computer? 4. Which keys start and stop recording, and which take a still? Do these work in release builds? 5. Frame rate: fixed (10 fps) or every frame drawn? 6. Should the status bar's red dot be included in the recording or hidden from it? 7. Should keypresses be shown as an overlay, for tutorials and docs (#12)? 8. How long can a recording run: a time limit, or until the card's Logs limit? ## Related `src/services/debug_console.cpp` (`screenshot`), `scripts/rdbg.py`, `src/ui/screen.cpp`, `src/main.cpp:188`, #4 (bug reports), #9, #10, #12.

twisla added the

kind

feature

area/ui area/ota-debug area/storage

concern/memory labels 2026-10-05 12:03:46 +00:00

twisla added this to the U1 Look and feel milestone 2026-10-05 20:14:08 +00:00

twisla referenced this issue

2026-10-06 05:34:11 +00:00