Public Access
Gemini: read big pages from the card as you scroll (Q88)
A page larger than memory allows is now windowed instead of cut: one pass over its file (cache or Saved Page) counts lines, indexes every 64th (offset, and the preformatted state there in bit 31: about 1 KB for a 1 MB page) and loads the first window. Scrolling near either end reads the next or previous window on the Gemini task; the line on top of the screen stays put, the scrollbar follows the whole page, and Tab at a window's edge pages on instead of wrapping. Window budgets count the memory the old window gives back. Display pages alternate between two cache files so the one on screen is never overwritten by the next fetch; jobs use a third. On the device, Cosmos with IRC connected: 226 of 419 lines at first, then lines 192-419, back to 64 and 0 while scrolling. `key space` added to the console's key command. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT
This commit is contained in:
Claude Opus 5.5
@@ -27,6 +27,7 @@ Gemini (geminiprotocol.net): one request per TLS connection on port 1965, the re
|
||||
| Q84 | The start page lists Saved Pages, newest first, grouped by capsule; they open with no network. In a Saved Page, a link to another Saved Page opens the saved copy; other links fetch online if Wi-Fi is up, or say "not saved, offline". A Saved Page shows when it was saved; `r` refreshes it. |
|
||||
| Q86 | *Decided after step 1.* **Two floors:** free heap stays above 40 KB in steady state, and above 20 KB for the second or two of a TLS handshake (measured: 24 KB with IRC connected). A fetch refuses to start below **55 KB** free ("not enough memory: stop IRC or retry"), so nothing pushes lower. |
|
||||
| Q87 | *Decided in step 3.* **With a card, every page streams to `/gemini/cache/page.gmi`** in 1 KB pieces while its TLS connection is open; once the connection closes and its ~45 KB is back, the page is loaded into RAM as far as the 40 KB floor allows. The whole page stays on the card (Saved Pages copy it). Without a card, the page goes straight to RAM under the same two floors. Pages are held as lines in 4 KB chunks, never one large block (the largest free block with IRC connected is about 31 KB). |
|
||||
| Q88 | *Added after step 6.* **A page bigger than memory allows is read from the card as you scroll.** Opening it, one pass over its file counts the lines, records where every 64th starts (and whether it's inside a preformatted block), and loads the first window. Scrolling near either end of the window reads the next or previous one in the background, keeping the line on top of the screen where it is; the scrollbar follows the whole page. Display pages alternate between two cache files, so the one on screen is never overwritten by the next fetch; background jobs use a third. |
|
||||
| Q85 | Saved Pages are deleted from the App only (`d`, with confirmation), never by Storage Clean-up's age rules, like Notes. |
|
||||
|
||||
## Measured (step 1)
|
||||
@@ -36,6 +37,8 @@ Gemini (geminiprotocol.net): one request per TLS connection on port 1965, the re
|
||||
- **Step 3, Cosmos (31.6 KB) with IRC connected:** first stopped at 4.6 KB (RAM only, the transfer's 20 KB floor). Streamed to the card: the whole page on the card, 20 KB of it loaded, lowest free heap 19.5 KB during the transfer and 43 KB once loaded. Without IRC: the whole page in RAM. Redirects (Cosmos `31`), input (`10`), not found (`51`) and a changed certificate (refused, both fingerprints shown) all checked on the device.
|
||||
- **Steps 4–6 on the device:** Project Gemini and its relative links, Back with the scroll restored, a refused YouTube link; `b` bookmarks, `s` saves (and says when it replaced an older copy), `S` saved 6 of 6 pages, the start page lists both; a Saved Page opens from the card with its origin, its saved links open saved copies, `r` refreshes, `d` asks first; Kennedy's input prompt sent "cardputer" and got 87 results; emoji drawn as `?`.
|
||||
- **A bug found there:** refreshing first loaded the whole Saved Page into RAM just to read its origin, next to the App's copy and a TLS connection: the heap fell to 436 bytes. Now only the first line is read, and every fetch (pages, saves, refreshes) checks the 55 KB start floor. Lowest since boot afterwards: 53.8 KB.
|
||||
- **Windowed pages (Q88), Cosmos with IRC connected:** 226 of 419 lines in memory at first; paging down loaded lines 192–419 in one window, scrolling back up loaded 64 onwards, then 0 onwards. Window budgets count the memory the old window gives back.
|
||||
- **Open:** with IRC connected, the lowest free heap during a Gemini fetch was 13–15.5 KB in later runs (24 and 19.5 KB earlier): the TLS receive buffers vary with the server's records. That's under Q86's 20 KB transient floor; the 55 KB start floor doesn't guarantee it.
|
||||
- Antenna (`warmedal.se`) doesn't answer, from the PC either; the default aggregator becomes Cosmos (`gemini://skyjake.fi/~Cosmos/`, which redirects to `cosmos.skyjake.fi`).
|
||||
|
||||
## Done when
|
||||
|
||||
+72
-7
@@ -96,9 +96,18 @@ void GeminiApp::navigate(const std::string& target, bool remember, bool preferSa
|
||||
|
||||
void GeminiApp::show(GeminiPage&& page) {
|
||||
page_ = std::move(page);
|
||||
windowLoading_ = false;
|
||||
analyse();
|
||||
topRow_ = hscroll_ = 0;
|
||||
selected_ = -1;
|
||||
}
|
||||
|
||||
void GeminiApp::analyse() {
|
||||
types_.clear();
|
||||
links_.clear();
|
||||
bool pre = false;
|
||||
// A window may start inside a preformatted block: the index says (bit 31).
|
||||
size_t entry = page_.windowStart / GeminiService::kIndexEvery;
|
||||
bool pre = page_.windowed && entry < page_.lineIndex.size() && (page_.lineIndex[entry] & 0x80000000u);
|
||||
for (size_t i = 0; i < page_.text.lineCount(); i++) {
|
||||
LineType t = parseGemLine(page_.text.line(i), pre).type;
|
||||
if (page_.header.mimeType() != "text/gemini") t = LineType::Preformatted; // other text/*: as is
|
||||
@@ -106,14 +115,41 @@ void GeminiApp::show(GeminiPage&& page) {
|
||||
if (t == LineType::Link) links_.push_back(i);
|
||||
}
|
||||
layoutDirty_ = true;
|
||||
topRow_ = hscroll_ = 0;
|
||||
selected_ = -1;
|
||||
requestRedraw();
|
||||
}
|
||||
|
||||
size_t GeminiApp::topLine() const {
|
||||
if (firstRow_.size() < 2) return page_.windowStart;
|
||||
size_t i = std::upper_bound(firstRow_.begin(), firstRow_.end() - 1, static_cast<uint32_t>(topRow_)) - firstRow_.begin();
|
||||
return page_.windowStart + (i > 0 ? i - 1 : 0);
|
||||
}
|
||||
|
||||
void GeminiApp::requestWindow(size_t firstLine) {
|
||||
if (windowLoading_ || !gemini_.loadWindow(page_, firstLine)) return;
|
||||
windowLoading_ = true;
|
||||
keepLine_ = static_cast<long>(topLine());
|
||||
status_ = "Reading from the card...";
|
||||
statusMs_ = millis();
|
||||
}
|
||||
|
||||
// A new window of the same page: same URL, history and position, other lines.
|
||||
void GeminiApp::applyWindow(GeminiPage&& window) {
|
||||
windowLoading_ = false;
|
||||
status_.clear();
|
||||
page_.text = std::move(window.text);
|
||||
page_.windowStart = window.windowStart;
|
||||
page_.lineIndex = std::move(window.lineIndex);
|
||||
selected_ = -1;
|
||||
analyse();
|
||||
}
|
||||
|
||||
void GeminiApp::update(uint32_t nowMs) {
|
||||
GeminiPage result;
|
||||
if (gemini_.takeResult(result)) {
|
||||
bool got = gemini_.takeResult(result);
|
||||
if (got && !result.windowOf.empty()) {
|
||||
windowLoading_ = false;
|
||||
if (result.windowOf == page_.backing) applyWindow(std::move(result)); // else: we've moved on
|
||||
} else if (got) {
|
||||
loading_ = false;
|
||||
std::string url = result.url;
|
||||
switch (result.outcome) {
|
||||
@@ -222,11 +258,30 @@ void GeminiApp::layout(Canvas& c) {
|
||||
|
||||
void GeminiApp::scrollTo(int row) {
|
||||
int total = firstRow_.empty() ? 0 : static_cast<int>(firstRow_.back());
|
||||
// A windowed page: near either end of the window, read the next part from the card. The new
|
||||
// window starts a block before the line on top, so what's on screen stays.
|
||||
const size_t block = GeminiService::kIndexEvery;
|
||||
if (row + rowsOnScreen() > total - 2 && moreBelow()) {
|
||||
size_t line = topLine();
|
||||
requestWindow(line >= block ? line - line % block : 0);
|
||||
} else if (row < 0 && moreAbove()) {
|
||||
size_t line = topLine();
|
||||
size_t back = page_.text.lineCount() / 2; // about half a window above
|
||||
requestWindow(line > back ? line - back : 0);
|
||||
}
|
||||
topRow_ = std::max(0, std::min(row, std::max(0, total - rowsOnScreen())));
|
||||
requestRedraw();
|
||||
}
|
||||
|
||||
void GeminiApp::selectLink(int direction) {
|
||||
// A windowed page: past the window's last (or first) link, page on instead of wrapping.
|
||||
int last = static_cast<int>(links_.size()) - 1;
|
||||
if ((direction > 0 && moreBelow() && (links_.empty() || selected_ == last)) ||
|
||||
(direction < 0 && moreAbove() && (links_.empty() || selected_ == 0))) {
|
||||
selected_ = -1;
|
||||
scrollTo(topRow_ + direction * (rowsOnScreen() - 1));
|
||||
return;
|
||||
}
|
||||
if (links_.empty()) return;
|
||||
int count = static_cast<int>(links_.size());
|
||||
if (selected_ < 0) {
|
||||
@@ -371,6 +426,11 @@ void GeminiApp::draw(Canvas& c) {
|
||||
scrollTo(restoreRow_);
|
||||
restoreRow_ = -1;
|
||||
}
|
||||
if (keepLine_ >= 0 && !windowLoading_) { // a new window: back to the line that was on top
|
||||
long local = keepLine_ - static_cast<long>(page_.windowStart);
|
||||
if (local >= 0 && local < static_cast<long>(page_.text.lineCount())) topRow_ = firstRow_[local];
|
||||
keepLine_ = -1;
|
||||
}
|
||||
}
|
||||
|
||||
// The URL bar: what's shown, or what's loading, or a passing message.
|
||||
@@ -404,10 +464,15 @@ void GeminiApp::draw(Canvas& c) {
|
||||
c.drawString(("Cut short: " + page_.truncatedWhy).substr(0, 47).c_str(), kMargin,
|
||||
y0 + (row - topRow_) * theme::kLineHeight + 2);
|
||||
}
|
||||
// Scroll position, on the right edge.
|
||||
// Scroll position, on the right edge: over the whole page, also when only a window is in memory.
|
||||
int total = static_cast<int>(firstRow_.back());
|
||||
if (total > rows) {
|
||||
int h = area.h - kUrlBar, barH = std::max(6, h * rows / total);
|
||||
int h = area.h - kUrlBar;
|
||||
if (page_.windowed && page_.totalLines > 0) {
|
||||
int barH = std::max(6, static_cast<int>(h * page_.text.lineCount() / page_.totalLines / 3));
|
||||
int y = static_cast<int>((h - barH) * topLine() / std::max<size_t>(1, page_.totalLines - 1));
|
||||
c.fillRect(area.w - 2, y0 + y, 2, barH, theme::kMuted);
|
||||
} else if (total > rows) {
|
||||
int barH = std::max(6, h * rows / total);
|
||||
c.fillRect(area.w - 2, y0 + (h - barH) * topRow_ / std::max(1, total - rows), 2, barH, theme::kMuted);
|
||||
}
|
||||
|
||||
|
||||
@@ -37,6 +37,12 @@ class GeminiApp : public App {
|
||||
bool isSaved() const { return !page_.savedFrom.empty(); }
|
||||
std::string savedPathOf() const { return page_.url.substr(7); } // file://…
|
||||
void show(GeminiPage&& page);
|
||||
void analyse(); // types and links of the lines in memory
|
||||
void applyWindow(GeminiPage&& window);
|
||||
void requestWindow(size_t firstLine);
|
||||
size_t topLine() const; // the source line (in the whole page) at the top of the screen
|
||||
bool moreBelow() const { return page_.windowed && page_.windowStart + page_.text.lineCount() < page_.totalLines; }
|
||||
bool moreAbove() const { return page_.windowed && page_.windowStart > 0; }
|
||||
void showMessage(const std::string& url, const std::string& gemtext);
|
||||
void layout(Canvas& c);
|
||||
void drawLine(Canvas& c, size_t index, int firstRow, int y, int rows);
|
||||
@@ -55,6 +61,8 @@ class GeminiApp : public App {
|
||||
int topRow_ = 0, hscroll_ = 0;
|
||||
int selected_ = -1; // into links_
|
||||
int restoreRow_ = -1; // going back: where that page was scrolled to, applied after layout
|
||||
bool windowLoading_ = false;
|
||||
long keepLine_ = -1; // a new window: the source line to keep at the top, applied after layout
|
||||
std::vector<Visit> history_;
|
||||
bool loading_ = false;
|
||||
std::string loadingUrl_, status_;
|
||||
|
||||
+4
-2
@@ -350,7 +350,7 @@ static const char* const kHelp =
|
||||
"gnss status | gnss restart | gnss track start|stop | gnss nmea on|off | gnss send <sentence without $ and checksum>\n"
|
||||
"crash the last crash: firmware, reason, task, backtrace\n"
|
||||
"coredump erase forget the core dump in flash\n"
|
||||
"key <name|char> press a key: up down left right select back home del tab, or one character\n"
|
||||
"key <name|char> press a key: up down left right select back home del tab space, or one character\n"
|
||||
"wifi status | wifi add <ssid><TAB><password>\n"
|
||||
"gemini get <url> fetch a Gemini page and report header, size, certificate, heap\n"
|
||||
"irc start | irc stop | irc dump | irc say <buffer> <text>\n"
|
||||
@@ -435,7 +435,9 @@ static void runCommand(String line) {
|
||||
Key key = k == "up" ? Key::Up : k == "down" ? Key::Down : k == "left" ? Key::Left
|
||||
: k == "right" ? Key::Right : k == "back" ? Key::Back : k == "home" ? Key::Home
|
||||
: k == "del" ? Key::Delete : k == "tab" ? Key::Tab : Key::Select;
|
||||
KeyEvent ev = k.length() == 1 && k[0] > ' ' ? KeyEvent::character((unsigned char)k[0]) : KeyEvent::of(key);
|
||||
KeyEvent ev = k == "space" ? KeyEvent::character(' ')
|
||||
: k.length() == 1 && k[0] > ' ' ? KeyEvent::character((unsigned char)k[0])
|
||||
: KeyEvent::of(key);
|
||||
if (!power->onKey(millis())) apps->handleKey(ev);
|
||||
}
|
||||
if (line.startsWith("wifi add ")) { // wifi add <ssid>\t<password>: credentials never touch the repo
|
||||
|
||||
+145
-34
@@ -82,6 +82,17 @@ void listSaved(const std::string& dir, std::vector<SavedEntry>& out) {
|
||||
|
||||
} // namespace
|
||||
|
||||
const char* GeminiService::slotPath(int slot) {
|
||||
static const char* const kPaths[] = {"/gemini/cache/page0.gmi", "/gemini/cache/page1.gmi", "/gemini/cache/work.gmi"};
|
||||
return kPaths[slot];
|
||||
}
|
||||
|
||||
const char* GeminiService::cacheOf(const std::string& url) const {
|
||||
for (int s = 0; s < 3; s++)
|
||||
if (!url.empty() && slotUrl_[s] == url) return slotPath(s);
|
||||
return nullptr;
|
||||
}
|
||||
|
||||
GeminiService::GeminiService(KeyValueStore& store, StorageService& storage, ClockService& clock, EventBus& bus)
|
||||
: store_(store), storage_(storage), clock_(clock), bus_(bus), lock_(xSemaphoreCreateMutex()) {}
|
||||
|
||||
@@ -173,6 +184,18 @@ bool GeminiService::download(const std::string& url) {
|
||||
return start(Job::Download);
|
||||
}
|
||||
|
||||
bool GeminiService::loadWindow(const GeminiPage& page, size_t firstLine) {
|
||||
if (busy_ || page.backing.empty()) return false;
|
||||
windowPage_ = GeminiPage();
|
||||
windowPage_.backing = page.backing;
|
||||
windowPage_.totalLines = page.totalLines;
|
||||
windowPage_.lineIndex = page.lineIndex;
|
||||
windowLine_ = firstLine;
|
||||
// The App drops its current window once the new one is in: count that memory as coming back.
|
||||
windowReleasing_ = page.text.bytes() + page.text.lineCount() * 8;
|
||||
return start(Job::Window);
|
||||
}
|
||||
|
||||
bool GeminiService::takeResult(GeminiPage& out) {
|
||||
if (!ready_) return false;
|
||||
xSemaphoreTake(lock_, portMAX_DELAY);
|
||||
@@ -234,7 +257,7 @@ void GeminiService::run() {
|
||||
for (const std::string& link : links_) {
|
||||
if (cancel_) break;
|
||||
GeminiPage page;
|
||||
fetchFollowing(link, page, false);
|
||||
fetchFollowing(link, page, false, kWorkSlot);
|
||||
bool text = page.outcome == GeminiPage::Outcome::Ok &&
|
||||
page.header.category() == gemini::Category::Success && page.header.mimeType() == "text/gemini";
|
||||
if (text && saveCached(link).empty()) saved++;
|
||||
@@ -256,13 +279,36 @@ void GeminiService::run() {
|
||||
break;
|
||||
}
|
||||
GeminiPage page;
|
||||
fetchFollowing(from, page, false);
|
||||
fetchFollowing(from, page, false, kWorkSlot);
|
||||
std::string why = page.outcome != GeminiPage::Outcome::Ok ? page.error
|
||||
: page.header.category() != gemini::Category::Success ? page.header.meta
|
||||
: saveCached(from);
|
||||
finishWithMessage(why.empty() ? "Refreshed" : "Not refreshed: " + why);
|
||||
break;
|
||||
}
|
||||
case Job::Window: {
|
||||
// Q86 both ways: old and new together above the transient floor for a moment, and the
|
||||
// new one alone (once the old is dropped) leaving the steady floor.
|
||||
size_t free = esp_get_free_heap_size();
|
||||
size_t transient = free > kTransientFloor + TextBuffer::kChunk ? free - kTransientFloor - TextBuffer::kChunk : 0;
|
||||
size_t steady = free + windowReleasing_ > kSteadyFloor + TextBuffer::kChunk
|
||||
? free + windowReleasing_ - kSteadyFloor - TextBuffer::kChunk
|
||||
: 0;
|
||||
size_t budget = std::min(transient, steady);
|
||||
GeminiPage page = std::move(windowPage_);
|
||||
page.windowStart = windowLine_ - windowLine_ % kIndexEvery;
|
||||
page.windowed = true;
|
||||
page.windowOf = page.backing;
|
||||
page.outcome = GeminiPage::Outcome::Ok;
|
||||
readWindow(page, page.windowStart, std::min(budget, kMaxBody));
|
||||
console.printf("gemini: window from line %u: %u lines\n", (unsigned)page.windowStart,
|
||||
(unsigned)page.text.lineCount());
|
||||
xSemaphoreTake(lock_, portMAX_DELAY);
|
||||
result_ = std::move(page);
|
||||
ready_ = true;
|
||||
xSemaphoreGive(lock_);
|
||||
break;
|
||||
}
|
||||
case Job::Delete: {
|
||||
bool removed = false;
|
||||
bool ran = storage_.runAndWait([&]() { removed = SD.remove(arg_.c_str()); });
|
||||
@@ -295,7 +341,7 @@ void GeminiService::run() {
|
||||
if (name.empty()) name = "download";
|
||||
std::string to = "/gemini/downloads/" + storage::sanitize(name);
|
||||
bool copied = false;
|
||||
if (cacheHolds(arg_)) storage_.runAndWait([&]() { copied = copyFile(kCachePath, to, ""); });
|
||||
if (const char* from = cacheOf(arg_)) storage_.runAndWait([&]() { copied = copyFile(from, to, ""); });
|
||||
finishWithMessage(copied ? "Saved to " + to : "Not saved (no card, or the page is gone)");
|
||||
break;
|
||||
}
|
||||
@@ -324,7 +370,12 @@ void GeminiService::runFetch(GeminiPage& page) {
|
||||
page.url = url;
|
||||
page.error = "Not saved, and offline";
|
||||
} else
|
||||
fetchFollowing(url, page, true);
|
||||
{
|
||||
int slot = nextDisplay_;
|
||||
fetchFollowing(url, page, true, slot);
|
||||
if (page.outcome == GeminiPage::Outcome::Ok && !page.cachePath.empty())
|
||||
nextDisplay_ = 1 - slot; // keep this one: it backs the page on screen
|
||||
}
|
||||
}
|
||||
}
|
||||
page.ms = millis() - started;
|
||||
@@ -332,7 +383,7 @@ void GeminiService::runFetch(GeminiPage& page) {
|
||||
}
|
||||
|
||||
// One request, following up to kMaxRedirects redirects.
|
||||
void GeminiService::fetchFollowing(const std::string& first, GeminiPage& page, bool loadIntoRam) {
|
||||
void GeminiService::fetchFollowing(const std::string& first, GeminiPage& page, bool loadIntoRam, int slot) {
|
||||
std::string url = first;
|
||||
size_t free = esp_get_free_heap_size();
|
||||
if (free < kStartFloor) { // Q86, for every fetch: pages, saves, refreshes
|
||||
@@ -342,7 +393,7 @@ void GeminiService::fetchFollowing(const std::string& first, GeminiPage& page, b
|
||||
return;
|
||||
}
|
||||
for (int hop = 0;; hop++) {
|
||||
fetchOne(url, page, loadIntoRam);
|
||||
fetchOne(url, page, loadIntoRam, slot);
|
||||
if (page.outcome != GeminiPage::Outcome::Ok || page.header.category() != gemini::Category::Redirect) break;
|
||||
std::string next = gemini::resolve(url, page.header.meta);
|
||||
if (!gemini::isGemini(next)) {
|
||||
@@ -401,7 +452,7 @@ void GeminiService::openSaved(const std::string& path, GeminiPage& page) {
|
||||
page.savedFrom = rest.substr(0, on);
|
||||
page.savedOn = on == std::string::npos ? "" : rest.substr(on + 4);
|
||||
}
|
||||
loadFromCard(page, esp_get_free_heap_size(), path.c_str());
|
||||
loadFromCard(page, esp_get_free_heap_size(), path);
|
||||
page.outcome = GeminiPage::Outcome::Ok;
|
||||
}
|
||||
|
||||
@@ -459,12 +510,14 @@ void GeminiService::composeStart(GeminiPage& page) {
|
||||
std::string GeminiService::saveCached(const std::string& url) {
|
||||
if (!gemini::isGemini(url)) return "Only Gemini pages can be saved";
|
||||
if (!storage_.state().present) return "No SD card";
|
||||
if (!cacheHolds(url)) {
|
||||
const char* from = cacheOf(url);
|
||||
if (!from) {
|
||||
GeminiPage page;
|
||||
fetchFollowing(url, page, false);
|
||||
fetchFollowing(url, page, false, kWorkSlot);
|
||||
if (page.outcome != GeminiPage::Outcome::Ok) return page.error;
|
||||
if (page.header.category() != gemini::Category::Success) return page.header.meta;
|
||||
if (!cacheHolds(page.url)) return "Not saved";
|
||||
from = cacheOf(page.url);
|
||||
if (!from) return "Not saved";
|
||||
}
|
||||
std::string to = gemini::savedPath(url);
|
||||
bool existed = false, ok = false;
|
||||
@@ -472,14 +525,14 @@ std::string GeminiService::saveCached(const std::string& url) {
|
||||
std::string header = std::string(kSavedMark) + url + " on " + now();
|
||||
storage_.runAndWait([&]() {
|
||||
existed = SD.exists(to.c_str());
|
||||
ok = copyFile(kCachePath, to, header);
|
||||
ok = copyFile(from, to, header);
|
||||
});
|
||||
if (!ok) return "Writing to the SD card failed";
|
||||
replaced_ = existed; // Q82: saving again replaces it, and says so
|
||||
return "";
|
||||
}
|
||||
|
||||
void GeminiService::fetchOne(const std::string& url, GeminiPage& page, bool loadIntoRam) {
|
||||
void GeminiService::fetchOne(const std::string& url, GeminiPage& page, bool loadIntoRam, int slot) {
|
||||
page.outcome = GeminiPage::Outcome::Failed;
|
||||
page.text.clear();
|
||||
page.truncated = false;
|
||||
@@ -553,11 +606,11 @@ void GeminiService::fetchOne(const std::string& url, GeminiPage& page, bool load
|
||||
// With a card, the body goes there while the connection is open (only 4 KB in RAM),
|
||||
// and comes back into RAM once the connection's memory is free again.
|
||||
if (storage_.state().present) {
|
||||
cachedUrl_.clear();
|
||||
if (!receiveToCard(tls, buf + i, n - i, page)) return;
|
||||
slotUrl_[slot].clear();
|
||||
if (!receiveToCard(tls, buf + i, n - i, page, slot)) return;
|
||||
tls.stop();
|
||||
cachedUrl_ = url;
|
||||
if (loadIntoRam) loadFromCard(page, freeBefore, kCachePath);
|
||||
slotUrl_[slot] = url;
|
||||
if (loadIntoRam) loadFromCard(page, freeBefore, slotPath(slot));
|
||||
page.outcome = GeminiPage::Outcome::Ok;
|
||||
return;
|
||||
}
|
||||
@@ -589,18 +642,18 @@ void GeminiService::fetchOne(const std::string& url, GeminiPage& page, bool load
|
||||
page.outcome = GeminiPage::Outcome::Ok;
|
||||
}
|
||||
|
||||
// Streams the rest of the body to kCachePath in 1 KB pieces, each written by the storage task:
|
||||
// Streams the rest of the body to a cache file in 1 KB pieces, each written by the storage task:
|
||||
// with IRC's TLS connection and this one both open, every KB of RAM counts (Q86).
|
||||
|
||||
bool GeminiService::receiveToCard(NetworkClientSecure& tls, const char* first, size_t len, GeminiPage& page) {
|
||||
bool GeminiService::receiveToCard(NetworkClientSecure& tls, const char* first, size_t len, GeminiPage& page, int slot) {
|
||||
struct Cache {
|
||||
File file;
|
||||
};
|
||||
auto cache = std::make_shared<Cache>();
|
||||
bool opened = storage_.runAndWait([cache]() {
|
||||
bool opened = storage_.runAndWait([cache, slot]() {
|
||||
SD.mkdir("/gemini");
|
||||
SD.mkdir("/gemini/cache");
|
||||
cache->file = SD.open(kCachePath, FILE_WRITE);
|
||||
cache->file = SD.open(slotPath(slot), FILE_WRITE);
|
||||
});
|
||||
if (!opened || !cache->file) {
|
||||
page.error = "Can't write to the SD card";
|
||||
@@ -642,37 +695,95 @@ bool GeminiService::receiveToCard(NetworkClientSecure& tls, const char* first, s
|
||||
storage_.runAndWait([cache]() { cache->file.close(); });
|
||||
if (cancel_) page.error = "Cancelled";
|
||||
else if (!ok) page.error = "Writing to the SD card failed";
|
||||
else page.cachePath = kCachePath;
|
||||
else page.cachePath = slotPath(slot);
|
||||
return !cancel_ && ok;
|
||||
}
|
||||
|
||||
// Back into RAM, now that the connection's memory is free again: as much as the steady floor
|
||||
// allows (Q86); the rest stays on the card.
|
||||
|
||||
void GeminiService::loadFromCard(GeminiPage& page, size_t freeBefore, const char* path) {
|
||||
// Back into RAM, now that the connection's memory is free again: one pass over the file counts
|
||||
// its lines and indexes every kIndexEvery-th, and loads lines while the steady floor allows (Q86).
|
||||
// If they don't all fit, the page is windowed: the App asks for other windows as you scroll.
|
||||
void GeminiService::loadFromCard(GeminiPage& page, size_t freeBefore, const std::string& path) {
|
||||
size_t budget = freeBefore > kSteadyFloor + TextBuffer::kChunk ? freeBefore - kSteadyFloor - TextBuffer::kChunk : 0;
|
||||
budget = std::min(budget, kMaxBody);
|
||||
page.backing = path;
|
||||
page.lineIndex.clear();
|
||||
page.totalLines = 0;
|
||||
page.windowStart = 0;
|
||||
bool loading = true;
|
||||
storage_.runAndWait([&]() {
|
||||
File f = SD.open(path);
|
||||
File f = SD.open(path.c_str());
|
||||
if (!f) return;
|
||||
std::string line;
|
||||
uint32_t offset = 0, lineStart = 0;
|
||||
bool pre = false;
|
||||
auto endLine = [&]() {
|
||||
if (page.totalLines % kIndexEvery == 0) page.lineIndex.push_back(lineStart | (pre ? 0x80000000u : 0));
|
||||
if (line.compare(0, 3, "```") == 0) pre = !pre;
|
||||
if (loading) {
|
||||
if (page.text.bytes() + page.text.lineCount() * 8 + line.size() + 1 > budget) loading = false;
|
||||
else {
|
||||
line += '\n';
|
||||
page.text.append(line.data(), line.size());
|
||||
}
|
||||
}
|
||||
page.totalLines++;
|
||||
line.clear();
|
||||
};
|
||||
char buf[512];
|
||||
int n;
|
||||
while ((n = f.read(reinterpret_cast<uint8_t*>(buf), sizeof buf)) > 0) {
|
||||
size_t cost = page.text.bytes() + page.text.lineCount() * 8 + n;
|
||||
if (cost > budget) {
|
||||
page.truncated = true;
|
||||
if (page.truncatedWhy.empty())
|
||||
page.truncatedWhy = "only part of it fits in memory; the whole page is on the card";
|
||||
break;
|
||||
for (int i = 0; i < n; i++, offset++) {
|
||||
if (buf[i] == '\n') {
|
||||
endLine();
|
||||
lineStart = offset + 1;
|
||||
} else if (line.size() < kMaxLineBytes) line += buf[i];
|
||||
}
|
||||
page.text.append(buf, n);
|
||||
}
|
||||
if (!line.empty()) endLine();
|
||||
f.close();
|
||||
});
|
||||
page.text.finish();
|
||||
page.windowed = !loading;
|
||||
if (page.windowed)
|
||||
console.printf("gemini: %s is windowed: %u of %u lines in memory\n", path.c_str(),
|
||||
(unsigned)page.text.lineCount(), (unsigned)page.totalLines);
|
||||
}
|
||||
|
||||
// Lines from `firstLine` on, from the page's file, as many as `budget` allows.
|
||||
void GeminiService::readWindow(GeminiPage& page, size_t firstLine, size_t budget) {
|
||||
size_t entry = firstLine / kIndexEvery;
|
||||
if (entry >= page.lineIndex.size()) return;
|
||||
storage_.runAndWait([&]() {
|
||||
File f = SD.open(page.backing.c_str());
|
||||
if (!f || !f.seek(page.lineIndex[entry] & 0x7FFFFFFFu)) return;
|
||||
size_t lineNo = entry * kIndexEvery;
|
||||
std::string line;
|
||||
char buf[512];
|
||||
int n;
|
||||
bool done = false;
|
||||
while (!done && (n = f.read(reinterpret_cast<uint8_t*>(buf), sizeof buf)) > 0) {
|
||||
for (int i = 0; i < n && !done; i++) {
|
||||
if (buf[i] != '\n') {
|
||||
if (line.size() < kMaxLineBytes) line += buf[i];
|
||||
continue;
|
||||
}
|
||||
if (lineNo >= firstLine) {
|
||||
if (page.text.bytes() + page.text.lineCount() * 8 + line.size() + 1 > budget) done = true;
|
||||
else {
|
||||
line += '\n';
|
||||
page.text.append(line.data(), line.size());
|
||||
}
|
||||
}
|
||||
lineNo++;
|
||||
line.clear();
|
||||
}
|
||||
}
|
||||
if (!done && !line.empty() && lineNo >= firstLine) page.text.append(line.data(), line.size());
|
||||
f.close();
|
||||
});
|
||||
page.text.finish();
|
||||
}
|
||||
|
||||
|
||||
void GeminiService::report(const GeminiPage& page, size_t heapBefore, size_t heapLowest) {
|
||||
console.printf("gemini: %s -> %s, %lu ms\n", page.requested.c_str(), page.url.c_str(), (unsigned long)page.ms);
|
||||
if (page.outcome != GeminiPage::Outcome::Ok) console.printf("gemini: %s\n", page.error.c_str());
|
||||
|
||||
@@ -36,6 +36,14 @@ struct GeminiPage {
|
||||
std::string savedFrom; // a Saved Page: the URL it was saved from (links resolve against it)
|
||||
std::string savedOn; // and when, "2026-10-05 01:20"
|
||||
uint32_t ms = 0;
|
||||
// A page bigger than memory allows is read from its file as you scroll: `text` then holds
|
||||
// lines [windowStart, windowStart + text.lineCount()) of totalLines. lineIndex has where every
|
||||
// 64th line starts in the file (bit 31: inside a preformatted block there).
|
||||
std::string backing; // the file the page is in (a cache file or a Saved Page)
|
||||
bool windowed = false;
|
||||
size_t totalLines = 0, windowStart = 0;
|
||||
std::vector<uint32_t> lineIndex;
|
||||
std::string windowOf; // a window result: the backing file it belongs to
|
||||
|
||||
// What relative links resolve against.
|
||||
const std::string& base() const { return savedFrom.empty() ? url : savedFrom; }
|
||||
@@ -54,7 +62,8 @@ class GeminiService {
|
||||
static constexpr int kMaxRedirects = 5; // Q72
|
||||
static constexpr size_t kMaxOnCard = 1024 * 1024; // a text page larger than this is cut
|
||||
static constexpr int kMaxLinkedPages = 30; // Q83
|
||||
static constexpr const char* kCachePath = "/gemini/cache/page.gmi";
|
||||
static constexpr size_t kIndexEvery = 64; // lines between lineIndex entries
|
||||
static constexpr size_t kMaxLineBytes = 8192; // longer lines are cut
|
||||
static constexpr const char* kBookmarks = "/gemini/bookmarks.gmi";
|
||||
static constexpr const char* kStartUrl = "about:start";
|
||||
|
||||
@@ -70,6 +79,8 @@ class GeminiService {
|
||||
bool deleteSaved(const std::string& savedPath); // d on a Saved Page (Q85)
|
||||
bool addBookmark(const std::string& url, const std::string& title); // b (Q77)
|
||||
bool download(const std::string& url); // s on a non-text page (Q73)
|
||||
// The window of a big page starting at `firstLine` (a multiple of kIndexEvery), from its file.
|
||||
bool loadWindow(const GeminiPage& page, size_t firstLine);
|
||||
|
||||
bool busy() const { return busy_; }
|
||||
bool takeResult(GeminiPage& out);
|
||||
@@ -82,22 +93,27 @@ class GeminiService {
|
||||
bool fetchToConsole(const std::string& url);
|
||||
|
||||
private:
|
||||
enum class Job { Fetch, Save, SaveWithLinks, Refresh, Delete, Bookmark, Download };
|
||||
enum class Job { Fetch, Save, SaveWithLinks, Refresh, Delete, Bookmark, Download, Window };
|
||||
// Cache files: two take turns for pages on screen (the one shown is never overwritten by the
|
||||
// next), one for jobs (saving linked pages, refreshing).
|
||||
static constexpr int kWorkSlot = 2;
|
||||
static const char* slotPath(int slot);
|
||||
|
||||
bool start(Job job);
|
||||
static void taskEntry(void* self);
|
||||
void run();
|
||||
void runFetch(GeminiPage& page);
|
||||
void fetchFollowing(const std::string& url, GeminiPage& page, bool loadIntoRam);
|
||||
void fetchOne(const std::string& url, GeminiPage& page, bool loadIntoRam);
|
||||
bool receiveToCard(::NetworkClientSecure& tls, const char* first, size_t len, GeminiPage& page);
|
||||
void loadFromCard(GeminiPage& page, size_t freeBefore, const char* path);
|
||||
void fetchFollowing(const std::string& url, GeminiPage& page, bool loadIntoRam, int slot);
|
||||
void fetchOne(const std::string& url, GeminiPage& page, bool loadIntoRam, int slot);
|
||||
bool receiveToCard(::NetworkClientSecure& tls, const char* first, size_t len, GeminiPage& page, int slot);
|
||||
void loadFromCard(GeminiPage& page, size_t freeBefore, const std::string& path);
|
||||
void readWindow(GeminiPage& page, size_t firstLine, size_t budget);
|
||||
void openSaved(const std::string& path, GeminiPage& page);
|
||||
std::string savedFirstLine(const std::string& path);
|
||||
std::string savedFrom(const std::string& path);
|
||||
void composeStart(GeminiPage& page);
|
||||
std::string saveCached(const std::string& url); // "" or what went wrong
|
||||
bool cacheHolds(const std::string& url) const { return cachedUrl_ == url; }
|
||||
const char* cacheOf(const std::string& url) const; // the cache file holding url's body, or nullptr
|
||||
void finishWithMessage(const std::string& text);
|
||||
void notify(const std::string& text);
|
||||
void report(const GeminiPage& page, size_t heapBefore, size_t heapLowest);
|
||||
@@ -115,7 +131,10 @@ class GeminiService {
|
||||
bool preferSaved_ = false;
|
||||
GeminiPage result_;
|
||||
std::string message_;
|
||||
std::string cachedUrl_; // the URL whose body is in kCachePath
|
||||
std::string slotUrl_[3]; // the URL whose body each cache file holds
|
||||
int nextDisplay_ = 0;
|
||||
size_t windowLine_ = 0, windowReleasing_ = 0;
|
||||
GeminiPage windowPage_;
|
||||
bool replaced_ = false; // the last save replaced an older copy
|
||||
volatile bool busy_ = false, ready_ = false, messageReady_ = false, cancel_ = false, toConsole_ = false;
|
||||
size_t lowest_ = 0;
|
||||
|
||||
Reference in New Issue
Block a user