Devlog: "Six releases behind", the WireGuard tunnel, the documentation caught up, and nine network commands (v0.19.0 to v0.21.0)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EhqxQ49eCju4CzKYNjZzwT
2026-10-08 03:12:03 +02:00
parent 0498916740
commit 01a8e2a233
12 changed files with 314 additions and 2 deletions
@@ -0,0 +1,312 @@
+++
+title = '''Six releases behind'''
+description = '''roro9stack gets a WireGuard tunnel, and the tools to find out why a network doesn't work: ping, nslookup, traceroute, a TLS check and the rest. In between, I looked at the project's own website and found that the documentation had stopped keeping up six releases earlier, and nobody had noticed, me included.'''
+date = 2026-10-08T03:15:00+02:00
+
+[extra]
+topics = '''ESP32-S3 · WireGuard · Documentation'''
+read_label = '''Read what was missing →'''
+uid = '''<b>ifconfig:</b> vpn 10.9.0.2/32 mtu 1420, up, default route'''
+dek = "Three more releases of [roro9stack](/devlog/roro9stack/), my firmware for the M5Stack Cardputer: a VPN (v0.19.0) and nine commands for troubleshooting a network from the device itself (v0.20.0 and v0.21.0). The tunnel crashed the device the first time it was started and then turned out to be the easy part. The hard part was noticing that the user guide described a firmware from the day before."
+byline = '''measured before deciding, for once; then promised more than the network stack could do'''
+
+[extra.sign]
+label = "Releases shipped with the documentation behind"
+note = "v0.13.0 to v0.18.0. Each had its own page updated, and nothing around it."
+count = "6"
+tone = "red"
+
+[[extra.cast]]
+name = "The tunnel"
+role = "WireGuard, one peer, IPv4"
+text = "Costs under 2 KB of memory once it's up, which on this device is close to free. Stopped the device dead the first time it was asked to start."
+
+[[extra.cast]]
+name = "lwIP"
+role = "the network stack"
+text = "Has a lock, and in this firmware it checks that you hold it. Has no routing table, which I found out after promising one."
+
+[[extra.cast]]
+name = "The test peer"
+role = "a WireGuard server in a container"
+text = "Could reach the device. The device couldn't reach it. So the server called the client, which WireGuard doesn't mind at all."
+
+[[extra.cast]]
+name = "The home page"
+role = "of this site"
+text = "Said the Storage App opens text, hex, captures and tracks. It had been showing pictures for four releases and serving files to phones for one."
+
+[[extra.cast]]
+name = "ping"
+role = "and eight friends"
+text = "nslookup, port, traceroute, tls, ntp, ifconfig, arp, netstat. The first thing I did with them was measure my own tunnel, and learn something."
+++
+
+## TL;DR
+
+- **A WireGuard VPN** (**v0.19.0**): copy a client `.conf` to the card, import it in Settings, switch it on. One tunnel, to one server. It carries everything, or the tunnel's own subnet.
+- **It costs 63 KB of flash and under 2 KB of memory.** The library crashed the firmware on its first call; the fix was three lines of ours.
+- **I promised split tunnels by `AllowedIPs` and couldn't deliver:** the network stack routes by one subnet or by default, nothing finer.
+- **The documentation was six releases behind.** Every feature had its own page. The home page, Settings, the Status Bar, the how-tos, the glossary and the README's first paragraph had none of it.
+- **Nine network commands** in the Shell (**v0.20.0**, **v0.21.0**): `ping`, `nslookup`, `port`, `traceroute`, `tls`, `ntp`, `ifconfig`, `arp`, `netstat`.
+- A ping of 1392 bytes crosses my tunnel and one of 1393 doesn't. I now know my tunnel's MTU to the byte, from a device with a 240-pixel screen.
+- 546 host tests, 13 more than last time.
+
+## The cast
+
+{{ cast() }}
+
+## Measured first, for once
+
+The issue for the VPN had a line I'd written days ago and am glad of: *measure both libraries first.* So before any design, a trial firmware with the maintained library in it, a throwaway WireGuard server in a container, and a real tunnel.
+
+{% table() %}
+| | Cost |
+|---|---|
+| Flash, the library | 43 KB |
+| Flash, with the service, the Settings page and the commands | 63 KB |
+| Static RAM | 1.2 KB |
+| Heap with the tunnel up | 1.8 KB |
+{% end %}
+
+On a device where a TLS connection takes 52 KB, a VPN for 1.8 is a gift. That number alone decided most of the design round: no memory floors, no "close IRC first", nothing to ration.
+
+Getting to that number took three surprises.
+
+### It stopped on the first call
+
+{% code(caption="The first `wg up`. The crash report named the line.") %}
+```
+assert failed: netif_add /IDF/components/lwip/lwip/src/core/netif.c:297
+(Required to lock TCPIP core functionality!)
+```
+{% end %}
+
+The library talks to lwIP, the network stack, through its low-level functions, and takes no lock while it does. Most builds don't mind. This firmware's framework is built with the check switched on, and so the first `netif_add` was the last thing the device did.
+
+The fix isn't in the library. Every call into it is made with the lock held, on our side: a three-line guard object. The library is used exactly as published.
+
+### The server had to call the client
+
+My device was on a guest Wi-Fi. The test server was on another network, and the guest network doesn't let its devices open connections inward. No handshake. For a while it looked like the library didn't work.
+
+It worked fine. WireGuard doesn't have clients and servers, only peers, and either can call the other. I gave the device a fixed port to listen on, told the *server* where the device was, and the server started the handshake. Twenty-five pings of twenty-five, through a tunnel set up backwards.
+
+(Later, on a network where the device could reach out, the normal direction worked first time. And then against my real server, which is the test that counts.)
+
+### "What AllowedIPs say"
+
+In the design round I'd agreed to this: *what goes through the tunnel is what the file's `AllowedIPs` line says.* Home subnets through the tunnel, the rest out over Wi-Fi. That's what every WireGuard client does.
+
+Then I read how lwIP decides where a packet goes. It has two rules: the packet is for an interface's own subnet, or it goes to the default interface. There is no routing table to add "192.168.1.0/24 via the tunnel" to.
+
+So the firmware does one of two things, and says which when you import a file:
+
+{% table() %}
+| The file says | What happens |
+|---|---|
+| `AllowedIPs = 0.0.0.0/0` | Everything goes through the tunnel |
+| Anything else | The one subnet the device's tunnel address is in |
+{% end %}
+
+A home network *behind* the server needs the first kind. An import with more ranges than that says so: "through it 10.9.0.0/24, not 1 other range". I'd rather the screen admit a limit than the documentation.
+
+It also changed an answer I'd given about a kill switch. I'd said there wouldn't be one. With everything routed into the tunnel, there is: while the server is silent the default route still points into a tunnel that has nowhere to send, and nothing leaves. Not by decision. By construction.
+
+{{ figure(src="vpn.png", alt="The Cardputer's Settings, VPN page at 2x: VPN On, Start with Wi-Fi On, Import /vpn/wg0.conf, Forget it; then in blue It is up, heard 66 s ago, and in grey the server's address and This device 10.9.0.2, through it everything. The Status Bar shows VPN in blue.", width=480, height=270, caption="Settings → VPN, against the test server. No key appears on this page, or anywhere else: the private key goes in with the file and is never shown again.") }}
+
+## Taking it down took my connection with it
+
+One more, because it's the kind of bug that only shows when you test the way you work.
+
+The tunnel puts its own DNS servers in while it's up, and has to give the old ones back when it stops. The Wi-Fi settings already had a way to get DHCP's servers back: ask for a new lease. So I called that.
+
+A new lease reconnects Wi-Fi. Reconnecting Wi-Fi drops every connection, including the Debug Console session I had just typed `vpn down` into. The command worked and I never saw it say so.
+
+The tunnel now remembers what was there and puts it back. It also notices when a DHCP renewal replaces its servers mid-flight, puts its own back in, and keeps the renewed ones for later.
+
+## Six releases behind
+
+With the tunnel working against my real server, I went to the website to see how it read. And then I went through the rest of the site, and it got worse with every page.
+
+- The **home page** described a Storage App that opens "text, hex, captures, tracks and update files". It had been showing pictures since v0.16.0 and serving the card to phones since v0.18.0.
+- The **Notes** card didn't say that a note can be any size, which it can since v0.15.0.
+- **Settings**, in the guide, had no VPN row. The **Status Bar** table had no `VPN`.
+- There was **no how-to** for anything built since: nothing on moving files with a phone, nothing on screenshots.
+- The **README** opened by calling this "a Meshtastic-compatible mesh messenger". It listens to a mesh. It has never sent a message.
+- Every milestone document had a **status line** from days ago. One still said a feature was "in a pull request" that had long been merged and released.
+
+None of this was neglect in the usual sense. Each feature *had* been documented: its own guide page, its section in the README, its design notes. That was the trap. Every pull request looked finished because the page about the new thing was there. Nobody was looking at the pages about the old things, which is where a reader starts.
+
+Six releases went out like that.
+
+What changed isn't a resolution to be more careful, which lasts a week. It's a list, the same one every time: the home page's cards, every Settings row, the Status Bar, a how-to for anything a person would want to do, the FAQ, the glossary, the README's opening, the status lines, a screenshot of each new screen. A feature isn't done until each of those has been asked "does the new thing show here?"
+
+The catch-up went into the same pull request as the VPN: three new how-tos, five new screenshots, and a README that starts with what the firmware does today.
+
+The two releases after it were documented as they were built. That's two. Ask me again at twenty.
+
+## Nine commands for a bad network day
+
+A VPN, addresses you can set by hand, a file server: this device now has enough networking to have network problems. And until today, the only thing it could tell you was `wifi status`.
+
+{% table() %}
+| Command | Tells you |
+|---|---|
+| `ping` | Does it answer, and how fast |
+| `nslookup` | A name's addresses, which DNS server answered, in how long |
+| `port` | A TCP port: open, refused, or silent |
+| `traceroute` | The routers on the way |
+| `tls` | A certificate: who for, who by, until when, and whether *this device* trusts it |
+| `ntp` | A time server's clock against this one |
+| `ifconfig` | The interfaces, and **which one is the default route** |
+| `arp` | The neighbours on the Wi-Fi |
+| `netstat` | What the device itself listens on |
+{% end %}
+
+They run in the Shell and over both consoles. The slow ones each get a task of their own and print as they go; `cancel` stops one.
+
+{{ figure(src="shell.png", alt="The Shell at 2x with VPN in the Status Bar: ping: 1 4 ms, 2 7 ms, 3 4 ms, then ping: 3 sent, 3 back, 0% lost, 4/5/7 with ms wrapped onto the next line; then nslookup roro9stack.net, 10.9.0.1 answered in 6 ms, 65.21.233.110.", width=480, height=270, caption="The first build, in the Shell. The ping summary is one character too long for the screen and drops its `ms` onto a line of its own. It reads `3/3 back, 0% lost, 4/5/7 ms` now.") }}
+
+Three things I like about them.
+
+**`nslookup` asks the server itself.** The system's resolver gives you an address and nothing else. This sends its own question over UDP, so it can say *which* server answered and how long it took, and it can ask a different server to compare. When a name doesn't resolve, that's the whole diagnosis.
+
+**`tls` checks nothing, then checks everything.** IRC, Gemini and updates all fail with "TLS error" and no way to see why. `tls` makes a handshake that accepts any certificate, so that a bad one can be looked at, and then judges it itself against the roots this device actually trusts:
+
+{% code(caption="Four servers, four verdicts.") %}
+```
+tls: for git.twis.la, by YE2
+tls: valid 2026-09-15 to 2026-12-14, 67 days left
+tls: this device trusts it
+
+tls: for geminiprotocol.net, by geminiprotocol.net
+tls: NOT trusted here: not signed by a root this device has
+
+tls: valid 2015-04-09 to 2015-04-12, EXPIRED 4197 days ago
+tls: NOT trusted here: expired, not signed by a root this device has
+
+tls: for *.badssl.com, by YR1
+tls: NOT trusted here: not for that name
+```
+{% end %}
+
+The second one is fine, by the way: a Gemini capsule signs its own certificate, and the browser trusts it on first sight.
+
+**A ping with a size finds a tunnel's MTU.** This is the one I didn't expect to use within the hour:
+
+{% code(caption="Through my tunnel. 1392 bytes plus 28 of headers is 1420, WireGuard's usual.") %}
+```
+$ ping 9.9.9.9 2 1392
+ping: 2/2 back, 0% lost, 27/30/33 ms
+$ ping 9.9.9.9 2 1393
+ping: 0/2 back, 100% lost
+```
+{% end %}
+
+One byte. And `ifconfig` on the same device, same minute:
+
+{% code() %}
+```
+ifconfig: vpn 10.9.0.2/32 mtu 1420, up, default route
+ifconfig: wifi 172.16.42.25/21 gw 172.16.42.1 mtu 1500, up
+ifconfig: dns 10.9.0.1
+```
+{% end %}
+
+"default route" on the `vpn` line is the answer to the first question anybody has with a VPN up.
+
+## All of them, on the device
+
+Typed on the Cardputer's own keyboard, in the Shell, with the tunnel up. `VPN` in the Status Bar is the tunnel; everything below went through it.
+
+{{ figure(src="ifconfig.png", alt="The Shell: ifconfig prints vpn 10.9.0.2/32 mtu 1420, up, default route; wifi 172.16.42.25/21 gw 172.16.42.1 mtu 1500, up; dns 10.9.0.1.", width=480, height=270, caption="`ifconfig`. The first line answers the first question: with the tunnel up, everything leaves through it.") }}
+
+{{ figure(src="ping.png", alt="The Shell: ping 9.9.9.9 3 prints 9.9.9.9, 56 bytes, then 1 25 ms, 2 25 ms, 3 33 ms, and 3/3 back, 0% lost, 25/27/33 ms.", width=480, height=270, caption="`ping`, with a count. The summary fits on one line now.") }}
+
+{{ figure(src="nslookup.png", alt="The Shell: nslookup www.wikipedia.org prints 10.9.0.1 answered in 293 ms, www.wikipedia.org is dyna.wikimedia.org, and 185.15.59.224.", width=480, height=270, caption="`nslookup`. Which server answered, how long it took, the alias the name really is, and the address.") }}
+
+{{ figure(src="port.png", alt="The Shell: port git.twis.la 443 prints 65.21.233.110:443 open, 48 ms.", width=480, height=270, caption="`port`. Open, in 48 ms. The other two answers are `refused` and nothing at all for five seconds.") }}
+
+{{ figure(src="tls.png", alt="The Shell after tls git.twis.la: for git.twis.la, by YE2; valid 2026-09-15 to 2026-12-14, 67 days left; this device trusts it; sha256 and 64 hexadecimal digits over two lines.", width=480, height=270, caption="`tls`. The verdict, and the fingerprint that a Gemini pin is made of. The first line scrolled off: the handshake took 0.7 s.") }}
+
+{{ figure(src="ntp.png", alt="The Shell: ntp prints pool.ntp.org (162.159.200.123), stratum 3, 23 ms away, and this clock is right, to 0.1 s.", width=480, height=270, caption="`ntp`. The clock gates TLS and the VPN, so it's worth being able to ask.") }}
+
+{{ figure(src="netstat.png", alt="The Shell: netstat prints tcp 2323 listens (Debug Console), tcp 3232 listens (updates), tcp 172.16.42.25:2323 - 172.16.42.249:51552, udp 49974, udp 49973, udp 68 (DHCP).", width=480, height=270, caption="`netstat`. What the device offers the network right now. The one connection is the Debug Console session that pressed these keys.") }}
+
+No picture of `traceroute` or `arp`: the first would list my provider's routers and the second my machines' hardware addresses, and neither belongs on a website. The traceroute through the tunnel was nine hops, my own server first.
+
+## The same mistake, twice, in three hours
+
+For the file sharing, the testable half of the code went in a file called `web_share.h`, and the service that uses it in another file called `web_share.h`, in another folder. One included the other by name and got itself. I wrote that up in [the last post](/devlog/roro9stack-share/) as one of the two things that went wrong.
+
+For the network commands, the testable half went in `net_tools.h`, and the service in `net_tools.h`.
+
+Same error message. Same fix. I'd like to say the second time took less long to spot.
+
+Two smaller ones:
+
+- **A refused connection isn't called refused.** lwIP reports it as "reset". My first `port` command told me a port on my own PC had "no route to it".
+- **My test tool lied about concurrency.** The script that sends commands waits "until the console is quiet". A ping prints every second, so the console was never quiet, so my "second command while a ping runs" was sent after the ping had finished. It ran, and I briefly believed the one-at-a-time guard didn't work.
+
+## What I didn't check
+
+- **From far away.** My real server answered, but the device was on the server's own network, reaching it by its public name.
+- **Roaming** from one Wi-Fi to another with the tunnel wanted.
+- **IRC through the tunnel.**
+- `tls` with IRC connected, when there shouldn't be the memory and it should say so.
+- The network commands **without** the VPN: every test went through the tunnel or to the local network.
+
+## By the numbers
+
+{% table() %}
+| | |
+|---|---|
+| Releases | 3 |
+| Heap a WireGuard tunnel costs | 1.8 KB |
+| Flash it costs | 63 KB |
+| Calls into the library before the device stopped | 1 |
+| Lines to fix that | 3 |
+| Ways lwIP can route a packet | 2 |
+| Releases that shipped with the docs behind | 6 |
+| How-tos written in one sitting to catch up | 3 |
+| Network commands | 9 |
+| Flash they cost | 20 KB |
+| Bytes between a ping that crosses my tunnel and one that doesn't | 1 |
+| Times I gave two headers the same name | 2 |
+| Host tests | 546 |
+{% end %}
+
+## Where it stands
+
+{% steps() %}
+1. ~~M0 and M1: the skeleton, Wi-Fi, IRC, Wi-Fi Tools.~~ v0.1.0 to v0.2.1, [the first post](/devlog/roro9stack/).
+
+2. ~~Updates and debugging over the air.~~ v0.3.0, [Look, no cables](/devlog/roro9stack-ota/).
+
+3. ~~M2: GNSS.~~ v0.4.0, [Seventeen satellites](/devlog/roro9stack-gnss/).
+
+4. ~~G1: Gemini.~~ v0.5.0, [A browser in the RAM IRC left over](/devlog/roro9stack-gemini/).
+
+5. ~~M3: the LoRa radio, listening.~~ v0.6.0, [The loudest thing it hears is itself](/devlog/roro9stack-lora/).
+
+6. ~~S1: the card, fixed addresses, the System App.~~ v0.6.1 to v0.8.1, [One byte too early](/devlog/roro9stack-s1/).
+
+7. ~~F1 and the start of R1: files, notes, signed releases, updates from Gitea.~~ v0.9.0 to v0.11.0, [836 bytes](/devlog/roro9stack-f1-r1/).
+
+8. ~~W1: the website, and one firmware with the Debug Console in it.~~ v0.12.0, [It was off](/devlog/roro9stack-console/).
+
+9. ~~A help key, the Shell, notes of any size.~~ v0.13.0 to v0.15.0, [It said "No"](/devlog/roro9stack-shell/).
+
+10. ~~Pictures, a screenshot key, the card in a phone's browser.~~ v0.16.0 to v0.18.0, [Press w](/devlog/roro9stack-share/).
+
+11. ~~A WireGuard tunnel, and the documentation caught up.~~ v0.19.0, this post.
+
+12. ~~Nine commands for a bad network day.~~ v0.20.0 and v0.21.0, this post.
+
+13. Next: M4, the mesh, which still wants a second node. And maybe SSH, now that there is a tunnel to reach things through.
+{% end %}
+
+{% signoff() %}
+The VPN took a library, a lock and an evening. The documentation took a look. I'd spent a day shipping features to a website that described yesterday's firmware, with every pull request looking complete because its own page was there. The tunnel carries exactly 1420 bytes, and I know that because the device told me.
+{% end %}