My personal website, https://kyrylonovotarskyi.com, is served with a swarm of six
ESP32-S3 microcontrollers sitting on a shelf in a London flat. No CDN in front, no
Cloud behind, no hosting provider, no Raspberry Pis. The whole rack,
screens and fans included, draws about 22 watts. It survives pulling
any individual plug in it, including the leader's, recovering in under four seconds.
The only hard constraint for this project: the serving path is microcontrollers
only. Nothing with a Linux kernel anywhere near a request.
Using a Pi would have been prudent, but where's the fun in that?
Going full cyberpunk is what made it interesting.
Four £12 (~$16) LILYGO T-ETH-Lite (ESP32-S3 with wired Ethernet) blades share
one public identity - a locally administered vMAC address plus a LAN IP the
router forwards to. Whoever wins the election writes that MAC into its own
Ethernet controller and announces it with a gratuitous ARP. Failover is
fast because nothing upstream actually learns anything. The election runs
over encrypted ESP-NOW radio (so cool!), off the wire.
Hence, a cable or a switch failure cannot split the brain that decides who owns the cable.
The elected fleet leader does not serve: it acts as a layer-4 TCP splicer,
relaying raw bytes to whichever blade has the fewest connections in flight,
delegating TLS termination downstream.
If the leader goes down, the next elected blade takes the mask and balances to what
remains. If a single survivor blade is up, it starts splicing to itself over loopback.
Two more boards with screens narrate everything happening in the system - every election,
every reboot and its cause - hearing the same 1 Hz radio heartbeats as everyone
else.
Please be gentle with it: there is a very real ceiling on how much TLS
termination an ESP32 can physically do. A fresh HTTPS connection costs
~650 ms of P-256 handshake; the fleet takes about 2.9-3.0 new visitors per
second before the queue forms. Warm connections return in ~40-60 ms at
~40-50 RPS. If the site is slow, chances are it's not the machines
struggling - it's you, sitting in a queue behind the rest of this thread.
Claude did most of the typing on this project. This post is mine, however.
To my surprise, it proved much less capable at this kind of software than I expected, and performed
extremely poorly at designing anything for the physical world. The division
of labour we've settled on: I decide what is true and what ships. Every
measurement came off my bench, every disaster happened to my actual
hardware, and the solder burns are mine.
The whole build is open source - the firmware, the OpenSCADs, and a build
guide that starts at the first solder joint and ends with a fleet holding
an election: https://github.com/Novotarskyi/ivan-bohun
disco_jack•2h ago
The only hard constraint for this project: the serving path is microcontrollers only. Nothing with a Linux kernel anywhere near a request. Using a Pi would have been prudent, but where's the fun in that?
Going full cyberpunk is what made it interesting.
Four £12 (~$16) LILYGO T-ETH-Lite (ESP32-S3 with wired Ethernet) blades share one public identity - a locally administered vMAC address plus a LAN IP the router forwards to. Whoever wins the election writes that MAC into its own Ethernet controller and announces it with a gratuitous ARP. Failover is fast because nothing upstream actually learns anything. The election runs over encrypted ESP-NOW radio (so cool!), off the wire. Hence, a cable or a switch failure cannot split the brain that decides who owns the cable. The elected fleet leader does not serve: it acts as a layer-4 TCP splicer, relaying raw bytes to whichever blade has the fewest connections in flight, delegating TLS termination downstream. If the leader goes down, the next elected blade takes the mask and balances to what remains. If a single survivor blade is up, it starts splicing to itself over loopback. Two more boards with screens narrate everything happening in the system - every election, every reboot and its cause - hearing the same 1 Hz radio heartbeats as everyone else.
Please be gentle with it: there is a very real ceiling on how much TLS termination an ESP32 can physically do. A fresh HTTPS connection costs ~650 ms of P-256 handshake; the fleet takes about 2.9-3.0 new visitors per second before the queue forms. Warm connections return in ~40-60 ms at ~40-50 RPS. If the site is slow, chances are it's not the machines struggling - it's you, sitting in a queue behind the rest of this thread.
Claude did most of the typing on this project. This post is mine, however. To my surprise, it proved much less capable at this kind of software than I expected, and performed extremely poorly at designing anything for the physical world. The division of labour we've settled on: I decide what is true and what ships. Every measurement came off my bench, every disaster happened to my actual hardware, and the solder burns are mine.
The whole build is open source - the firmware, the OpenSCADs, and a build guide that starts at the first solder joint and ends with a fleet holding an election: https://github.com/Novotarskyi/ivan-bohun
The full story - the lore, the wedged-TCP-stack hunt, and how I learned to solder - is in the repo's STORY.md: https://github.com/Novotarskyi/ivan-bohun/blob/main/STORY.md