diff --git a/README.md b/README.md index 5c72e6b..1c1d2d6 100644 --- a/README.md +++ b/README.md @@ -54,6 +54,12 @@ To install from the SD card instead, copy the `.ota` file from `.pio/build/cardp **The private key** lives in `~/.config/roro9stack/ota-key.pem` and must never be committed. If it's lost, generate a new pair and flash once over USB. +## Gemini + +The Gemini App browses Geminispace (docs/milestones/G1.md): Tab and Shift+Tab pick a link, Enter follows it, Back returns (to where the page was scrolled), Space pages down, `g` types an address. Certificates are trusted on first use; a changed one stops the page and asks. + +On a page, `b` bookmarks it, `s` saves it to the SD card to read offline (a non-text file goes to `/gemini/downloads/`), `S` saves it with the pages it links to on the same capsule (up to 30). The start page lists bookmarks and Saved Pages; inside a Saved Page, `r` refreshes it and `d` deletes it. With a card, every page streams through `/gemini/cache/` so a large one arrives whole even with IRC connected; what doesn't fit in memory stays on the card. + ## Development aids `scripts/serial_log.sh [seconds] [command…]` records the serial output, and can send commands to the firmware first. For example, `scripts/serial_log.sh 30 short sleep:12 burst` sets short screen timeouts, waits 12 s, then sends a burst of Toasts. @@ -70,6 +76,8 @@ To install from the SD card instead, copy the `.ota` file from `.pio/build/cardp | `cat ` | Prints the first ~1.2 KB of a file on the SD card | | `irc start` | Starts the IRC Service (normally done by opening the IRC App) | | `irc stop` | Stops it, as `/quit` does: QUIT if connected, no more retries, and the App stays disconnected until you type | +| `gemini get ` | Fetches a Gemini page and prints its header, size, certificate fingerprint and heap use | +| `gemini trust ` | Pins a certificate by hand (the Gemini App asks when one changes) | | `irc say ` | Types into a Buffer, commands included (`irc say 0 /join #test`) | | `irc dump` | Prints IRC status, memory, and the last lines of each Buffer | | `wifi status` | Prints Wi-Fi state, network, signal, clock and free heap | diff --git a/docs/milestones/G1.md b/docs/milestones/G1.md index c6c9a3c..9d90de8 100644 --- a/docs/milestones/G1.md +++ b/docs/milestones/G1.md @@ -1,5 +1,7 @@ # G1 — Gemini client +**Status:** steps 1–6 built and checked on the device (branch `g1`); one item left for a check by hand: reading Saved Pages with Wi-Fi off. + **Goal:** browse Geminispace from the Cardputer: fetch and read gemtext over TLS, follow links, answer input prompts, keep bookmarks, and save pages to the SD card to read later, offline. A side milestone between M2 and M3, tagged v0.5.0 when done. Gemini (geminiprotocol.net): one request per TLS connection on port 1965, the request is the URL and CRLF, the response a ` ` header line, then the body. Most capsules use self-signed certificates: trust on first use is the norm. @@ -32,6 +34,8 @@ Gemini (geminiprotocol.net): one request per TLS connection on port 1965, the re - `gemini://geminiprotocol.net/`: `20 text/gemini`, 1,184 bytes, TLS handshake 0.7–1.1 s, whole fetch 0.7–1.1 s; kennedy.gemi.dev 1.9 s. The fetch task's stack peaks at about 3.6 KB of 6. - **Heap, Debug Build, IRC connected over TLS:** about 68 KB free before a fetch. After the handshake the fetch holds about 32 KB (36–40 KB left); the handshake itself (certificate chain parsed with the 16 KB receive buffer allocated) dips to about **24 KB** for a second or two. Nothing leaks: the heap after matches the heap before. - **Step 3, Cosmos (31.6 KB) with IRC connected:** first stopped at 4.6 KB (RAM only, the transfer's 20 KB floor). Streamed to the card: the whole page on the card, 20 KB of it loaded, lowest free heap 19.5 KB during the transfer and 43 KB once loaded. Without IRC: the whole page in RAM. Redirects (Cosmos `31`), input (`10`), not found (`51`) and a changed certificate (refused, both fingerprints shown) all checked on the device. +- **Steps 4–6 on the device:** Project Gemini and its relative links, Back with the scroll restored, a refused YouTube link; `b` bookmarks, `s` saves (and says when it replaced an older copy), `S` saved 6 of 6 pages, the start page lists both; a Saved Page opens from the card with its origin, its saved links open saved copies, `r` refreshes, `d` asks first; Kennedy's input prompt sent "cardputer" and got 87 results; emoji drawn as `?`. +- **A bug found there:** refreshing first loaded the whole Saved Page into RAM just to read its origin, next to the App's copy and a TLS connection: the heap fell to 436 bytes. Now only the first line is read, and every fetch (pages, saves, refreshes) checks the 55 KB start floor. Lowest since boot afterwards: 53.8 KB. - Antenna (`warmedal.se`) doesn't answer, from the PC either; the default aggregator becomes Cosmos (`gemini://skyjake.fi/~Cosmos/`, which redirects to `cosmos.skyjake.fi`). ## Done when diff --git a/src/apps/gemini_app.cpp b/src/apps/gemini_app.cpp index a9b20a3..8c07f5e 100644 --- a/src/apps/gemini_app.cpp +++ b/src/apps/gemini_app.cpp @@ -18,18 +18,9 @@ namespace { constexpr int kUrlBar = 10; // the small line with the URL at the top constexpr int kMargin = 3; -constexpr const char* kStartUrl = "about:start"; +constexpr const char* kStartUrl = GeminiService::kStartUrl; constexpr size_t kMaxHistory = 20; // Q77 -const char* const kStartPage = - "# Gemini\n" - "Tab picks a link, Enter follows it, Back returns, g types an address.\n" - "## Start here\n" - "=> gemini://geminiprotocol.net/ Project Gemini\n" - "=> gemini://kennedy.gemi.dev/ Kennedy: search Geminispace\n" - "=> gemini://skyjake.fi/~Cosmos/ Cosmos: what's new on Geminispace\n" - "=> gemini://bbs.geminispace.org/ Bubble: a bulletin board\n"; - // How each line type looks: its prefix, the indent of its wrapped rows, font and colour. struct Style { const char* prefix; @@ -54,10 +45,22 @@ Style styleOf(LineType t) { } // namespace void GeminiApp::onEnter() { - if (page_.url.empty() && !loading_) showMessage(kStartUrl, kStartPage); + if (page_.url.empty() && !loading_) navigate(kStartUrl, false); requestRedraw(); } +// The page's first heading, for a bookmark's label. +std::string GeminiApp::titleOf() const { + for (size_t i = 0; i < page_.text.lineCount() && i < 40; i++) + if (static_cast(types_[i]) == LineType::Heading1) return lineAt(i).text; + return ""; +} + +void GeminiApp::follow(const std::string& link) { + std::string url = resolve(page_.base(), link); + navigate(url, true, isSaved()); // inside a Saved Page, its saved links open offline (Q84) +} + void GeminiApp::showMessage(const std::string& url, const std::string& gemtext) { GeminiPage p; p.outcome = GeminiPage::Outcome::Ok; @@ -68,14 +71,19 @@ void GeminiApp::showMessage(const std::string& url, const std::string& gemtext) show(std::move(p)); } -void GeminiApp::navigate(const std::string& target, bool remember) { - if (!isGemini(target)) { +void GeminiApp::navigate(const std::string& target, bool remember, bool preferSaved) { + bool local = target == kStartUrl || target.rfind("file://", 0) == 0; + if (!isGemini(target) && !local) { status_ = "Not Gemini: " + target; statusMs_ = millis(); requestRedraw(); return; } - if (!gemini_.fetch(target)) return; + if (!gemini_.fetch(target, preferSaved)) { + status_ = "Busy: try again in a moment"; + statusMs_ = millis(); + return; + } if (remember && !page_.url.empty()) { history_.push_back({page_.url, topRow_}); if (history_.size() > kMaxHistory) history_.erase(history_.begin()); @@ -120,11 +128,21 @@ void GeminiApp::update(uint32_t nowMs) { switch (result.header.category()) { case Category::Success: if (result.header.mimeType().rfind("text/", 0) == 0) show(std::move(result)); - else showMessage(url, "# Not a text page\nThis is " + result.header.mimeType() + - ", which the Cardputer can't show.\n"); + else { + std::string mime = result.header.mimeType(); + showMessage(url, "# Not a text page\nThis is " + mime + + ", which the Cardputer can't show. s saves it to the card, in " + "/gemini/downloads.\n"); + page_.header.meta = mime; // so s knows to download it + } break; case Category::Input: - showMessage(url, "# " + result.header.meta + "\nInput prompts come in the next step.\n"); + // Q72: ask, then request the same URL with the answer as its query. + inputOpen_ = true; + inputSensitive_ = result.header.sensitiveInput(); + inputPrompt_ = result.header.meta; + inputUrl_ = url; + input_.setText(""); break; default: showMessage(url, "# " + std::to_string(result.header.status) + " " + result.header.meta + @@ -136,6 +154,16 @@ void GeminiApp::update(uint32_t nowMs) { requestRedraw(); } // Signed: statusMs_ may be a few ms ahead of this pass's nowMs (set by a key handled after it). + std::string message; + if (gemini_.takeMessage(message)) { + status_ = message; + statusMs_ = millis(); + if (!reopen_.empty()) { // a refreshed Saved Page: show the new copy + navigate(reopen_, false); + reopen_.clear(); + } else if (message == "Deleted") navigate(kStartUrl, false); + requestRedraw(); + } if (!status_.empty() && static_cast(nowMs - statusMs_) > 4000) { status_.clear(); requestRedraw(); @@ -168,10 +196,12 @@ std::vector GeminiApp::rowsOf(Canvas& c, size_t i) const { std::string indent(st.indent, ' '); std::vector rows = wrapText(st.prefix + displayText(g.text), width, measure); if (rows.empty()) rows.push_back(""); - // Rows after the first: wrap again, narrower, behind the indent. + // Rows after the first: the exact rest of the text (wrapText keeps every space but the one it + // breaks at), wrapped again, narrower, behind the indent. if (rows.size() > 1 && st.indent > 0) { - std::string rest; - for (size_t r = 1; r < rows.size(); r++) rest += (r > 1 ? " " : "") + rows[r]; + std::string full = st.prefix + displayText(g.text); + std::string rest = full.substr(rows[0].size()); + if (!rest.empty() && rest[0] == ' ') rest.erase(0, 1); std::vector more = wrapText(rest, width - measure(indent), measure); rows.resize(1); for (auto& m : more) rows.push_back(indent + m); @@ -225,6 +255,28 @@ bool GeminiApp::onKey(const KeyEvent& e) { certDialog_.reset(); return true; } + if (deleteDialog_) { + deleteDialog_->onKey(e); + if (deleteDialog_->result() == DialogModel::kPending) return true; + if (deleteDialog_->result() == 1) gemini_.deleteSaved(savedPathOf()); + deleteDialog_.reset(); + return true; + } + if (inputOpen_) { + switch (e.key) { + case Key::Char: input_.insert(e.ch); break; + case Key::Delete: input_.backspace(); break; + case Key::Left: input_.left(); break; + case Key::Right: input_.right(); break; + case Key::Back: inputOpen_ = false; break; + case Key::Select: + inputOpen_ = false; + navigate(resolve(inputUrl_, "?" + encodeQuery(input_.text()))); + break; + default: break; + } + return true; + } if (addressOpen_) { switch (e.key) { case Key::Char: address_.insert(e.ch); break; @@ -250,7 +302,7 @@ bool GeminiApp::onKey(const KeyEvent& e) { case Key::Right: hscroll_ += 8; return true; case Key::Tab: selectLink(e.shift ? -1 : 1); return true; case Key::Select: - if (selected_ >= 0) navigate(resolve(page_.url, lineAt(links_[selected_]).url)); + if (selected_ >= 0) follow(lineAt(links_[selected_]).url); return true; case Key::Back: case Key::Delete: @@ -262,16 +314,49 @@ bool GeminiApp::onKey(const KeyEvent& e) { { Visit v = history_.back(); history_.pop_back(); - if (v.url == kStartUrl) showMessage(kStartUrl, kStartPage); - else navigate(v.url, false); + navigate(v.url, false); restoreRow_ = v.topRow; } return true; case Key::Char: - if (e.ch == ' ') scrollTo(topRow_ + rowsOnScreen() - 1); - else if (e.ch == 'g' || e.ch == 'G') { - address_.setText(page_.url == kStartUrl ? "" : page_.url); - addressOpen_ = true; + switch (e.ch) { + case ' ': scrollTo(topRow_ + rowsOnScreen() - 1); break; + case 'g': + address_.setText(page_.url == kStartUrl || isSaved() ? "" : page_.url); + addressOpen_ = true; + break; + case 'b': // Q77 + if (isGemini(page_.base())) gemini_.addBookmark(page_.base(), titleOf()); + break; + case 's': // Q82, Q73 + if (isSaved()) status_ = "Already saved; r refreshes it"; + else if (!isGemini(page_.url)) status_ = "Only Gemini pages can be saved"; + else if (page_.header.mimeType().rfind("text/", 0) == 0) gemini_.save(page_.url); + else gemini_.download(page_.url); + statusMs_ = millis(); + break; + case 'S': { // Q83: this page and the gemtext it links to on the same capsule + Url here; + if (isSaved() || !parseUrl(page_.url, here)) break; + std::vector links; + for (uint32_t i : links_) { + std::string target = resolve(page_.url, lineAt(i).url); + Url u; + if (isGemini(target) && parseUrl(target, u) && u.host == here.host && target != page_.url && + std::find(links.begin(), links.end(), target) == links.end()) + links.push_back(target); + if (static_cast(links.size()) >= GeminiService::kMaxLinkedPages - 1) break; + } + gemini_.saveWithLinks(page_.url, links); + break; + } + case 'r': // Q84: a fresh copy of a Saved Page + if (isSaved() && gemini_.refresh(savedPathOf())) reopen_ = page_.url; + break; + case 'd': // Q85 + if (isSaved()) deleteDialog_.reset(new DialogModel({"Keep", "Delete"})); + break; + default: break; } return true; default: return false; @@ -291,7 +376,10 @@ void GeminiApp::draw(Canvas& c) { // The URL bar: what's shown, or what's loading, or a passing message. c.setFont(&fonts::small); c.setTextDatum(top_left); - std::string bar = loading_ ? "Loading " + loadingUrl_ : !status_.empty() ? status_ : page_.url; + std::string bar = loading_ ? "Loading " + loadingUrl_ + : !status_.empty() ? status_ + : isSaved() ? "saved " + page_.savedOn + ": " + page_.savedFrom + : page_.url; c.setTextColor(loading_ || !status_.empty() ? theme::kWarning : theme::kMuted); c.drawString(displayText(bar).substr(0, 47).c_str(), kMargin, area.y + 1); @@ -331,6 +419,22 @@ void GeminiApp::draw(Canvas& c) { c.drawString("Go to (Enter, or ` to cancel)", box.x, box.y - 10); widgets::lineEditor(c, address_, box); } + if (inputOpen_) { + theme::Rect box{2, area.y + area.h / 2 - 4, area.w - 4, theme::kLineHeight + 4}; + c.fillRect(0, box.y - 26, area.w, box.h + 28, theme::kBackground); + c.setFont(&fonts::body); + c.setTextColor(theme::kText); + c.drawString(displayText(inputPrompt_).substr(0, 39).c_str(), box.x, box.y - 22); + c.setFont(&fonts::small); + c.setTextColor(theme::kMuted); + c.drawString(inputSensitive_ ? "(hidden) Enter sends, ` cancels" : "Enter sends, ` cancels", box.x, box.y - 9); + if (inputSensitive_) { + LineEditor masked(1024); + masked.setText(std::string(input_.text().size(), '*')); + widgets::lineEditor(c, masked, box); + } else widgets::lineEditor(c, input_, box); + } + if (deleteDialog_) widgets::dialog(c, "Delete this Saved Page?", page_.savedFrom, *deleteDialog_); if (certDialog_) widgets::dialog(c, "Certificate changed", pendingCert_.host + " now shows a different certificate. Was " + pendingCert_.pinned.substr(0, 16) + diff --git a/src/apps/gemini_app.h b/src/apps/gemini_app.h index 5f16ae8..a26d2e4 100644 --- a/src/apps/gemini_app.h +++ b/src/apps/gemini_app.h @@ -21,7 +21,7 @@ class GeminiApp : public App { explicit GeminiApp(GeminiService& gemini) : gemini_(gemini) {} void onEnter() override; bool onKey(const KeyEvent& e) override; - bool textEntryActive() const override { return addressOpen_; } + bool textEntryActive() const override { return addressOpen_ || inputOpen_; } void update(uint32_t nowMs) override; void draw(Canvas& c) override; @@ -31,7 +31,11 @@ class GeminiApp : public App { int topRow; }; - void navigate(const std::string& url, bool remember = true); + void navigate(const std::string& url, bool remember = true, bool preferSaved = false); + void follow(const std::string& link); + std::string titleOf() const; + bool isSaved() const { return !page_.savedFrom.empty(); } + std::string savedPathOf() const { return page_.url.substr(7); } // file://… void show(GeminiPage&& page); void showMessage(const std::string& url, const std::string& gemtext); void layout(Canvas& c); @@ -59,6 +63,11 @@ class GeminiApp : public App { LineEditor address_{1024}; std::unique_ptr certDialog_; GeminiPage pendingCert_; + std::unique_ptr deleteDialog_; + bool inputOpen_ = false, inputSensitive_ = false; + std::string inputPrompt_, inputUrl_; + LineEditor input_{1024}; + std::string reopen_; // after a refresh: the Saved Page to show again }; } // namespace roro diff --git a/src/main.cpp b/src/main.cpp index 0d0a2b1..544ba2e 100644 --- a/src/main.cpp +++ b/src/main.cpp @@ -148,7 +148,7 @@ void setup() { storageService = new StorageService(bus); power = new PowerService(settings); clockService = new ClockService(settings, bus); - geminiService = new GeminiService(nvs, *storageService); + geminiService = new GeminiService(nvs, *storageService, *clockService, bus); gnssService = new GnssService(settings, *clockService, *storageService, bus); savedNetworks = new SavedNetworks(nvs); savedNetworks->load(); diff --git a/src/services/gemini_service.cpp b/src/services/gemini_service.cpp index 25f71a0..433a66c 100644 --- a/src/services/gemini_service.cpp +++ b/src/services/gemini_service.cpp @@ -2,21 +2,33 @@ #include #include +#include #include #include +#include #include "gemini_url.h" +#include "gemtext.h" #include "platform/console.h" #include "sha256.h" - -using roro::gemini::TextBuffer; +#include "storage_paths.h" namespace roro { +using gemini::TextBuffer; + namespace { constexpr uint32_t kIdleTimeoutMs = 15000; +constexpr const char* kSavedRoot = "/gemini/saved"; +constexpr const char* kSavedMark = "> Saved from "; // first line of a Saved Page (Q82) +constexpr const char* kDefaults = + "## Start here\n" + "=> gemini://geminiprotocol.net/ Project Gemini\n" + "=> gemini://kennedy.gemi.dev/ Kennedy: search Geminispace\n" + "=> gemini://skyjake.fi/~Cosmos/ Cosmos: what's new on Geminispace\n" + "=> gemini://bbs.geminispace.org/ Bubble: a bulletin board\n"; std::string hex(const uint8_t* data, size_t len) { static const char* digits = "0123456789abcdef"; @@ -28,10 +40,50 @@ std::string hex(const uint8_t* data, size_t len) { return out; } +// Folders for a file path, one at a time (SD.mkdir makes one level). +void makeParents(const std::string& path) { + for (size_t slash = path.find('/', 1); slash != std::string::npos; slash = path.find('/', slash + 1)) { + std::string dir = path.substr(0, slash); + if (!SD.exists(dir.c_str())) SD.mkdir(dir.c_str()); + } +} + +// Copies `from` to `to`, optionally after a first line. On the storage task. +bool copyFile(const char* from, const std::string& to, const std::string& firstLine) { + File in = SD.open(from); + if (!in) return false; + makeParents(to); + File out = SD.open(to.c_str(), FILE_WRITE); + if (!out) return false; + bool ok = firstLine.empty() || out.print((firstLine + "\n").c_str()) > 0; + uint8_t buf[512]; + int n; + while (ok && (n = in.read(buf, sizeof buf)) > 0) ok = out.write(buf, n) == static_cast(n); + out.close(); + in.close(); + if (!ok) SD.remove(to.c_str()); + return ok; +} + +struct SavedEntry { + std::string path; // /gemini/saved/host/…/name.gmi + time_t when; +}; + +void listSaved(const std::string& dir, std::vector& out) { + File d = SD.open(dir.c_str()); + if (!d || !d.isDirectory()) return; + for (File f = d.openNextFile(); f && out.size() < 200; f = d.openNextFile()) { + std::string path = dir + "/" + f.name(); + if (f.isDirectory()) listSaved(path, out); + else out.push_back({path, f.getLastWrite()}); + } +} + } // namespace -GeminiService::GeminiService(KeyValueStore& store, StorageService& storage) - : store_(store), storage_(storage), lock_(xSemaphoreCreateMutex()) {} +GeminiService::GeminiService(KeyValueStore& store, StorageService& storage, ClockService& clock, EventBus& bus) + : store_(store), storage_(storage), clock_(clock), bus_(bus), lock_(xSemaphoreCreateMutex()) {} // NVS keys are 15 characters at most: "gm" and 12 hex digits of SHA-256("host:port"). std::string GeminiService::pinKey(const std::string& host, int port) const { @@ -45,11 +97,20 @@ void GeminiService::trust(const std::string& host, int port, const std::string& store_.putString(pinKey(host, port).c_str(), fingerprint); } -bool GeminiService::fetch(const std::string& url) { +std::string GeminiService::now() const { + int64_t t = clock_.utcNow(); + return t < 0 ? "an unknown date" : ClockModel::formatLocalDate(t) + " " + ClockModel::formatLocalTime(t); +} + +void GeminiService::notify(const std::string& text) { + bus_.publish(Event::withText(EventType::Notification, text.c_str(), static_cast(NotificationLevel::Info))); +} + +bool GeminiService::start(Job job) { if (busy_) return false; busy_ = true; - ready_ = cancel_ = false; - pendingUrl_ = url; + ready_ = messageReady_ = cancel_ = false; + job_ = job; if (xTaskCreate(taskEntry, "gemini", 6144, this, 1, nullptr) != pdPASS) { // peak 3.6 KB (step 1) busy_ = false; return false; @@ -57,6 +118,13 @@ bool GeminiService::fetch(const std::string& url) { return true; } +bool GeminiService::fetch(const std::string& url, bool preferSaved) { + if (busy_) return false; + arg_ = url; + preferSaved_ = preferSaved; + return start(Job::Fetch); +} + bool GeminiService::fetchToConsole(const std::string& url) { if (busy_) return false; toConsole_ = true; @@ -67,6 +135,44 @@ bool GeminiService::fetchToConsole(const std::string& url) { return true; } +bool GeminiService::save(const std::string& url) { + if (busy_) return false; + arg_ = url; + return start(Job::Save); +} + +bool GeminiService::saveWithLinks(const std::string& url, std::vector links) { + if (busy_) return false; + arg_ = url; + links_ = std::move(links); + return start(Job::SaveWithLinks); +} + +bool GeminiService::refresh(const std::string& savedPath) { + if (busy_) return false; + arg_ = savedPath; + return start(Job::Refresh); +} + +bool GeminiService::deleteSaved(const std::string& savedPath) { + if (busy_) return false; + arg_ = savedPath; + return start(Job::Delete); +} + +bool GeminiService::addBookmark(const std::string& url, const std::string& title) { + if (busy_) return false; + arg_ = url; + arg2_ = title; + return start(Job::Bookmark); +} + +bool GeminiService::download(const std::string& url) { + if (busy_) return false; + arg_ = url; + return start(Job::Download); +} + bool GeminiService::takeResult(GeminiPage& out) { if (!ready_) return false; xSemaphoreTake(lock_, portMAX_DELAY); @@ -77,53 +183,303 @@ bool GeminiService::takeResult(GeminiPage& out) { return true; } +bool GeminiService::takeMessage(std::string& out) { + if (!messageReady_) return false; + xSemaphoreTake(lock_, portMAX_DELAY); + out = std::move(message_); + messageReady_ = false; + xSemaphoreGive(lock_); + return true; +} + +void GeminiService::finishWithMessage(const std::string& text) { + xSemaphoreTake(lock_, portMAX_DELAY); + message_ = text; + messageReady_ = true; + xSemaphoreGive(lock_); +} + void GeminiService::taskEntry(void* self) { static_cast(self)->run(); vTaskDelete(nullptr); } void GeminiService::run() { - GeminiPage page; - page.requested = pendingUrl_; - size_t before = esp_get_free_heap_size(); - lowest_ = before; - uint32_t start = millis(); - std::string url = pendingUrl_; - if (before < kStartFloor) { - page.error = "Not enough memory (" + std::to_string(before / 1024) + " KB free): stop IRC or retry"; - } else { - for (int hop = 0;; hop++) { - fetchOne(url, page); - if (page.outcome != GeminiPage::Outcome::Ok || page.header.category() != gemini::Category::Redirect) break; - std::string next = gemini::resolve(url, page.header.meta); - if (!gemini::isGemini(next)) { - page.outcome = GeminiPage::Outcome::Failed; - page.error = "Redirect to another protocol: " + next; - break; + switch (job_) { + case Job::Fetch: { + GeminiPage page; + runFetch(page); + if (!toConsole_) { + xSemaphoreTake(lock_, portMAX_DELAY); + result_ = std::move(page); + ready_ = true; + xSemaphoreGive(lock_); } - if (hop + 1 > kMaxRedirects) { - page.outcome = GeminiPage::Outcome::Failed; - page.error = "Too many redirects"; - break; - } - url = next; + toConsole_ = false; + break; + } + case Job::Save: { + std::string why = saveCached(arg_); + finishWithMessage(!why.empty() ? why : replaced_ ? "Saved again (replaced the older copy)" : "Saved for offline reading"); + break; + } + case Job::SaveWithLinks: { + std::string why = saveCached(arg_); + if (!why.empty()) { + finishWithMessage(why); + break; + } + int saved = 1, failed = 0, total = static_cast(links_.size()) + 1; + notify("Saving " + std::to_string(total) + " pages for offline reading"); + for (const std::string& link : links_) { + if (cancel_) break; + GeminiPage page; + fetchFollowing(link, page, false); + bool text = page.outcome == GeminiPage::Outcome::Ok && + page.header.category() == gemini::Category::Success && page.header.mimeType() == "text/gemini"; + if (text && saveCached(link).empty()) saved++; + else failed++; + if ((saved + failed) % 5 == 0) notify("Saved " + std::to_string(saved) + " of " + std::to_string(total)); + } + std::string done = "Saved " + std::to_string(saved) + " of " + std::to_string(total) + " pages" + + (failed ? " (" + std::to_string(failed) + " not gemtext or failed)" : ""); + notify(done); + finishWithMessage(done); + break; + } + case Job::Refresh: { + // Only the first line: where it came from. (Loading the whole page here, next to the + // App's copy and a TLS connection, once took the heap down to 436 bytes.) + std::string from = savedFrom(arg_); + if (from.empty()) { + finishWithMessage("Not a Saved Page"); + break; + } + GeminiPage page; + fetchFollowing(from, page, false); + std::string why = page.outcome != GeminiPage::Outcome::Ok ? page.error + : page.header.category() != gemini::Category::Success ? page.header.meta + : saveCached(from); + finishWithMessage(why.empty() ? "Refreshed" : "Not refreshed: " + why); + break; + } + case Job::Delete: { + bool removed = false; + bool ran = storage_.runAndWait([&]() { removed = SD.remove(arg_.c_str()); }); + finishWithMessage(!ran ? "No SD card" : removed ? "Deleted" : "Not deleted"); + break; + } + case Job::Bookmark: { + std::string line = "=> " + arg_ + (arg2_.empty() ? "" : " " + arg2_), already; + bool ran = storage_.runAndWait([&]() { + File f = SD.open(kBookmarks); + while (f && f.available()) { + String l = f.readStringUntil('\n'); + if (l.startsWith(("=> " + arg_ + " ").c_str()) || l == ("=> " + arg_).c_str()) already = "yes"; + } + if (f) f.close(); + if (!already.empty()) return; + makeParents(kBookmarks); + File out = SD.open(kBookmarks, FILE_APPEND); + if (out) { + out.print((line + "\n").c_str()); + out.close(); + } + }); + finishWithMessage(!ran ? "No SD card" : !already.empty() ? "Already bookmarked" : "Bookmarked"); + break; + } + case Job::Download: { + gemini::Url u; + std::string name = gemini::parseUrl(arg_, u) ? u.path.substr(u.path.rfind('/') + 1) : ""; + if (name.empty()) name = "download"; + std::string to = "/gemini/downloads/" + storage::sanitize(name); + bool copied = false; + if (cacheHolds(arg_)) storage_.runAndWait([&]() { copied = copyFile(kCachePath, to, ""); }); + finishWithMessage(copied ? "Saved to " + to : "Not saved (no card, or the page is gone)"); + break; } - } - page.url = url; - page.ms = millis() - start; - if (toConsole_) { - report(page, before, lowest_); - toConsole_ = false; - } else { - xSemaphoreTake(lock_, portMAX_DELAY); - result_ = std::move(page); - ready_ = true; - xSemaphoreGive(lock_); } busy_ = false; } -void GeminiService::fetchOne(const std::string& url, GeminiPage& page) { +void GeminiService::runFetch(GeminiPage& page) { + page.requested = arg_; + size_t before = esp_get_free_heap_size(); + lowest_ = before; + uint32_t started = millis(); + const std::string& url = arg_; + if (url == kStartUrl) composeStart(page); + else if (url.rfind("file://", 0) == 0) openSaved(url.substr(7), page); + else { + // Inside a Saved Page, a link to another Saved Page opens the saved copy (Q84). + bool savedCopy = false; + if (preferSaved_) { + std::string path = gemini::savedPath(url); + storage_.runAndWait([&]() { savedCopy = SD.exists(path.c_str()); }); + if (savedCopy) openSaved(path, page); + } + if (!savedCopy) { + if (preferSaved_ && !WiFi.isConnected()) { + page.url = url; + page.error = "Not saved, and offline"; + } else + fetchFollowing(url, page, true); + } + } + page.ms = millis() - started; + if (toConsole_) report(page, before, lowest_); +} + +// One request, following up to kMaxRedirects redirects. +void GeminiService::fetchFollowing(const std::string& first, GeminiPage& page, bool loadIntoRam) { + std::string url = first; + size_t free = esp_get_free_heap_size(); + if (free < kStartFloor) { // Q86, for every fetch: pages, saves, refreshes + page.url = url; + page.outcome = GeminiPage::Outcome::Failed; + page.error = "Not enough memory (" + std::to_string(free / 1024) + " KB free): stop IRC or retry"; + return; + } + for (int hop = 0;; hop++) { + fetchOne(url, page, loadIntoRam); + if (page.outcome != GeminiPage::Outcome::Ok || page.header.category() != gemini::Category::Redirect) break; + std::string next = gemini::resolve(url, page.header.meta); + if (!gemini::isGemini(next)) { + page.outcome = GeminiPage::Outcome::Failed; + page.error = "Redirect to another protocol: " + next; + break; + } + if (hop + 1 > kMaxRedirects) { + page.outcome = GeminiPage::Outcome::Failed; + page.error = "Too many redirects"; + break; + } + url = next; + } + page.url = url; +} + +// A Saved Page's first line: "> Saved from on " (Q82). Empty if it isn't one. +std::string GeminiService::savedFirstLine(const std::string& path) { + std::string first; + storage_.runAndWait([&]() { + File f = SD.open(path.c_str()); + if (!f) return; + char buf[600]; + size_t n = f.readBytesUntil('\n', buf, sizeof buf - 1); + buf[n] = 0; + first = buf; + f.close(); + }); + if (!first.empty() && first.back() == '\r') first.pop_back(); + return first.rfind(kSavedMark, 0) == 0 ? first : ""; +} + +std::string GeminiService::savedFrom(const std::string& path) { + std::string first = savedFirstLine(path); + if (first.empty()) return ""; + std::string rest = first.substr(strlen(kSavedMark)); + return rest.substr(0, rest.find(" on ")); +} + +// A Saved Page from the card, as far as memory allows; its first line says where it came from. +void GeminiService::openSaved(const std::string& path, GeminiPage& page) { + page.url = "file://" + path; + page.header.status = 20; + page.header.meta = "text/gemini"; + bool found = false; + storage_.runAndWait([&]() { found = SD.exists(path.c_str()); }); + std::string first = found ? savedFirstLine(path) : ""; + if (!found) { + page.error = "This Saved Page is gone"; + return; + } + if (first.rfind(kSavedMark, 0) == 0) { + std::string rest = first.substr(strlen(kSavedMark)); + size_t on = rest.find(" on "); + page.savedFrom = rest.substr(0, on); + page.savedOn = on == std::string::npos ? "" : rest.substr(on + 4); + } + loadFromCard(page, esp_get_free_heap_size(), path.c_str()); + page.outcome = GeminiPage::Outcome::Ok; +} + +// about:start (Q78, Q84): bookmarks, then Saved Pages by capsule, newest first, then defaults. +void GeminiService::composeStart(GeminiPage& page) { + std::string doc = "# Gemini\nTab picks a link, Enter follows it, Back returns, g types an address. " + "On a page: b bookmarks it, s saves it to read offline, S with the pages it links to.\n"; + std::string bookmarks; + std::vector saved; + bool card = storage_.runAndWait([&]() { + File f = SD.open(kBookmarks); + while (f && f.available() && bookmarks.size() < 8192) { + String l = f.readStringUntil('\n'); + if (l.startsWith("=>")) bookmarks += std::string(l.c_str()) + "\n"; + } + if (f) f.close(); + listSaved(kSavedRoot, saved); + }); + doc += "## Bookmarks\n"; + doc += !card ? "No SD card: bookmarks and Saved Pages live on it.\n" + : bookmarks.empty() ? "None yet: b on a page adds it here.\n" : bookmarks; + if (!saved.empty()) { + // Capsules ordered by their newest Saved Page, pages newest first within each. + std::sort(saved.begin(), saved.end(), [](const SavedEntry& a, const SavedEntry& b) { return a.when > b.when; }); + std::vector order; + std::map byHost; + size_t root = strlen(kSavedRoot) + 1; + for (auto& e : saved) { + std::string rel = e.path.substr(root); + std::string host = rel.substr(0, rel.find('/')); + if (!byHost.count(host)) order.push_back(host); + std::string label = rel.substr(host.size() + 1); + if (label.size() > 4 && label.compare(label.size() - 4, 4, ".gmi") == 0) label.resize(label.size() - 4); + char day[24] = ""; + if (e.when > 1000000000) { + struct tm t; + localtime_r(&e.when, &t); + strftime(day, sizeof day, " (%d %b)", &t); + } + byHost[host] += "=> file://" + e.path + " " + label + day + "\n"; + } + doc += "## Saved pages\n"; + for (auto& host : order) doc += "### " + host + "\n" + byHost[host]; + } + doc += kDefaults; + page.url = kStartUrl; + page.header.status = 20; + page.header.meta = "text/gemini"; + page.text.append(doc.data(), doc.size()); + page.text.finish(); + page.outcome = GeminiPage::Outcome::Ok; +} + +// The cached body of `url` (fetched again if the cache holds something else) to its Saved Page. +std::string GeminiService::saveCached(const std::string& url) { + if (!gemini::isGemini(url)) return "Only Gemini pages can be saved"; + if (!storage_.state().present) return "No SD card"; + if (!cacheHolds(url)) { + GeminiPage page; + fetchFollowing(url, page, false); + if (page.outcome != GeminiPage::Outcome::Ok) return page.error; + if (page.header.category() != gemini::Category::Success) return page.header.meta; + if (!cacheHolds(page.url)) return "Not saved"; + } + std::string to = gemini::savedPath(url); + bool existed = false, ok = false; + replaced_ = false; + std::string header = std::string(kSavedMark) + url + " on " + now(); + storage_.runAndWait([&]() { + existed = SD.exists(to.c_str()); + ok = copyFile(kCachePath, to, header); + }); + if (!ok) return "Writing to the SD card failed"; + replaced_ = existed; // Q82: saving again replaces it, and says so + return ""; +} + +void GeminiService::fetchOne(const std::string& url, GeminiPage& page, bool loadIntoRam) { page.outcome = GeminiPage::Outcome::Failed; page.text.clear(); page.truncated = false; @@ -197,9 +553,11 @@ void GeminiService::fetchOne(const std::string& url, GeminiPage& page) { // With a card, the body goes there while the connection is open (only 4 KB in RAM), // and comes back into RAM once the connection's memory is free again. if (storage_.state().present) { + cachedUrl_.clear(); if (!receiveToCard(tls, buf + i, n - i, page)) return; tls.stop(); - loadFromCard(page, freeBefore); + cachedUrl_ = url; + if (loadIntoRam) loadFromCard(page, freeBefore, kCachePath); page.outcome = GeminiPage::Outcome::Ok; return; } @@ -233,6 +591,7 @@ void GeminiService::fetchOne(const std::string& url, GeminiPage& page) { // Streams the rest of the body to kCachePath in 1 KB pieces, each written by the storage task: // with IRC's TLS connection and this one both open, every KB of RAM counts (Q86). + bool GeminiService::receiveToCard(NetworkClientSecure& tls, const char* first, size_t len, GeminiPage& page) { struct Cache { File file; @@ -289,11 +648,12 @@ bool GeminiService::receiveToCard(NetworkClientSecure& tls, const char* first, s // Back into RAM, now that the connection's memory is free again: as much as the steady floor // allows (Q86); the rest stays on the card. -void GeminiService::loadFromCard(GeminiPage& page, size_t freeBefore) { + +void GeminiService::loadFromCard(GeminiPage& page, size_t freeBefore, const char* path) { size_t budget = freeBefore > kSteadyFloor + TextBuffer::kChunk ? freeBefore - kSteadyFloor - TextBuffer::kChunk : 0; budget = std::min(budget, kMaxBody); storage_.runAndWait([&]() { - File f = SD.open(kCachePath); + File f = SD.open(path); if (!f) return; char buf[512]; int n; @@ -312,6 +672,7 @@ void GeminiService::loadFromCard(GeminiPage& page, size_t freeBefore) { page.text.finish(); } + void GeminiService::report(const GeminiPage& page, size_t heapBefore, size_t heapLowest) { console.printf("gemini: %s -> %s, %lu ms\n", page.requested.c_str(), page.url.c_str(), (unsigned long)page.ms); if (page.outcome != GeminiPage::Outcome::Ok) console.printf("gemini: %s\n", page.error.c_str()); @@ -327,4 +688,5 @@ void GeminiService::report(const GeminiPage& page, size_t heapBefore, size_t hea (unsigned)esp_get_free_heap_size()); } + } // namespace roro diff --git a/src/services/gemini_service.h b/src/services/gemini_service.h index 8c99964..d6173bb 100644 --- a/src/services/gemini_service.h +++ b/src/services/gemini_service.h @@ -4,9 +4,12 @@ #include #include +#include +#include "event_bus.h" #include "gemini_response.h" #include "key_value_store.h" +#include "services/clock_service.h" #include "services/storage_service.h" #include "text_buffer.h" @@ -19,7 +22,7 @@ struct GeminiPage { enum class Outcome { Ok, Failed, CertificateChanged }; Outcome outcome = Outcome::Failed; std::string requested; // what was asked for - std::string url; // where it ended up, after redirects + std::string url; // where it ended up, after redirects (file://… for a Saved Page) gemini::Header header; // the final response's header gemini::TextBuffer text; // 2x only: the body as lines, up to kMaxBody bool truncated = false; // the body was cut: see truncatedWhy @@ -30,12 +33,18 @@ struct GeminiPage { std::string pinned, presented; std::string cachePath; // with a card: the whole body as received (Saved Pages copy it) size_t receivedBytes = 0; // the whole body, which may be more than `text` holds + std::string savedFrom; // a Saved Page: the URL it was saved from (links resolve against it) + std::string savedOn; // and when, "2026-10-05 01:20" uint32_t ms = 0; + + // What relative links resolve against. + const std::string& base() const { return savedFrom.empty() ? url : savedFrom; } }; -// Gemini (docs/milestones/G1.md). Each fetch runs on a short-lived task of its own: a TLS -// handshake can block for seconds, which the main loop's watchdog wouldn't allow, and an idle -// client should cost no stack. Certificates are trusted on first use, per host and port (Q71). +// Gemini (docs/milestones/G1.md). Everything that touches the network or the SD card runs as one +// job at a time on a short-lived task of its own: a TLS handshake or a missing card can block for +// seconds, which the main loop's watchdog wouldn't allow, and an idle client should cost no stack. +// Certificates are trusted on first use, per host and port (Q71). class GeminiService { public: static constexpr size_t kMaxBody = 64 * 1024; // Q74 @@ -43,16 +52,28 @@ class GeminiService { static constexpr size_t kSteadyFloor = 40 * 1024; // Q86: what must be left once the page is in static constexpr size_t kTransientFloor = 20 * 1024; // Q86: while the TLS connection is open static constexpr int kMaxRedirects = 5; // Q72 - static constexpr size_t kMaxOnCard = 1024 * 1024; // a text page larger than this is cut + static constexpr int kMaxLinkedPages = 30; // Q83 static constexpr const char* kCachePath = "/gemini/cache/page.gmi"; + static constexpr const char* kBookmarks = "/gemini/bookmarks.gmi"; + static constexpr const char* kStartUrl = "about:start"; - GeminiService(KeyValueStore& store, StorageService& storage); + GeminiService(KeyValueStore& store, StorageService& storage, ClockService& clock, EventBus& bus); + + // Pages: gemini://…, about:start (bookmarks, Saved Pages, defaults) or file:///gemini/saved/…. + // `preferSaved`: open the Saved Page of that URL if there is one (inside a Saved Page, Q84). + bool fetch(const std::string& url, bool preferSaved = false); + // Actions; each reports one line through takeMessage(). + bool save(const std::string& url); // s (Q82) + bool saveWithLinks(const std::string& url, std::vector links); // S (Q83) + bool refresh(const std::string& savedPath); // r on a Saved Page + bool deleteSaved(const std::string& savedPath); // d on a Saved Page (Q85) + bool addBookmark(const std::string& url, const std::string& title); // b (Q77) + bool download(const std::string& url); // s on a non-text page (Q73) - // Starts a fetch; false if one is running. The result comes from takeResult(). - bool fetch(const std::string& url); bool busy() const { return busy_; } bool takeResult(GeminiPage& out); + bool takeMessage(std::string& out); void cancel() { cancel_ = true; } // After the user accepts a changed certificate. void trust(const std::string& host, int port, const std::string& fingerprint); @@ -61,21 +82,42 @@ class GeminiService { bool fetchToConsole(const std::string& url); private: + enum class Job { Fetch, Save, SaveWithLinks, Refresh, Delete, Bookmark, Download }; + + bool start(Job job); static void taskEntry(void* self); void run(); - void fetchOne(const std::string& url, GeminiPage& page); + void runFetch(GeminiPage& page); + void fetchFollowing(const std::string& url, GeminiPage& page, bool loadIntoRam); + void fetchOne(const std::string& url, GeminiPage& page, bool loadIntoRam); + bool receiveToCard(::NetworkClientSecure& tls, const char* first, size_t len, GeminiPage& page); + void loadFromCard(GeminiPage& page, size_t freeBefore, const char* path); + void openSaved(const std::string& path, GeminiPage& page); + std::string savedFirstLine(const std::string& path); + std::string savedFrom(const std::string& path); + void composeStart(GeminiPage& page); + std::string saveCached(const std::string& url); // "" or what went wrong + bool cacheHolds(const std::string& url) const { return cachedUrl_ == url; } + void finishWithMessage(const std::string& text); + void notify(const std::string& text); void report(const GeminiPage& page, size_t heapBefore, size_t heapLowest); std::string pinKey(const std::string& host, int port) const; - - bool receiveToCard(::NetworkClientSecure& tls, const char* first, size_t len, GeminiPage& page); - void loadFromCard(GeminiPage& page, size_t freeBefore); + std::string now() const; KeyValueStore& store_; StorageService& storage_; + ClockService& clock_; + EventBus& bus_; SemaphoreHandle_t lock_; - std::string pendingUrl_; + Job job_ = Job::Fetch; + std::string arg_, arg2_; + std::vector links_; + bool preferSaved_ = false; GeminiPage result_; - volatile bool busy_ = false, ready_ = false, cancel_ = false, toConsole_ = false; + std::string message_; + std::string cachedUrl_; // the URL whose body is in kCachePath + bool replaced_ = false; // the last save replaced an older copy + volatile bool busy_ = false, ready_ = false, messageReady_ = false, cancel_ = false, toConsole_ = false; size_t lowest_ = 0; };