How AI on Microcontrollers Works: Operators and Kernels

https://danielmangum.com/posts/ai-microcontrollers-operators-kernels/

83•hasheddan•7mo ago

Comments

Neywiny•7mo ago

In the same way "embedded" is relative, I appreciate the author's recognition that "edge" is relative. For some, AI at the edge means on-prem server farms. For some it means a mini-pc. For others, maybe an SBC. Here it's a microcontroller. Further still is AI within the sensors a microcontroller would talk to. That's probably just another microcontroller but still.

woliveirajr•7mo ago

Microcontrollers all the way down

astrobe_•7mo ago

There's "micro" and "micro". The microcontroller operating a simple coffee machine, or a simple washing-machine is probably 8 or 16 bits. This is what I would call "bare metal", as they don't run an OS, only off-the-shelf frameworks at best.

For "bigger" devices, it's usually a Cortex inside a system-on-chip or system-on-module, 32 bits single core and a few Mb of RAM for low-end (enough to run regular Linux distro instead of uClinux for instance), 64 bits multicore for high-end devices that deal with audio/video. That kind of business is often resource-hungry in every way.

I work with that kind of stuff, and to me these "microcontrollers" are just monsters that I hesitate to call "micro" when some of my coworkers work on much smaller chips with only a few K of RAM available.

Neywiny•7mo ago

I do wish sometimes they used the bigger micro, though. We have some power supplies that technically have an Ethernet interface. But when using it, even for SCPI over TCP (forget about the virtual front panel that takes a minute to update), it lags so bad the output enable button needs a few tries to toggle. I should practice yanking the positive wire for an emergency

amelius•7mo ago

Indeed. Is nVidia Jetson "edge" or not?

simgt•7mo ago

Of course it is. Edge mostly stands for network edge, Jetsons aren't meant to be deployed in data centers.

amelius•7mo ago

I don't understand why we need a separate "lite" format for microcontrollers.

Wouldn't it be advantageous if we used ONNX for everything? https://onnx.ai/

batuhandumani•7mo ago

tf-lite micro library has many advantages, and the first of these is the tensorFlow framework itself. You can train the model easily and then implement the same or a similar architecture on esp-32s without much effort. Another advantage is its optimization and you can easily intervene in various memory optimizations and even though it is not a large one, it does have a community.

Apart from these, for example, the author implemented the model the traditional way using C, but it is more convenient to use tf-lite micro on esp32s with the Berry script language.

However, since I have never used onnxin this kind of project, I cant speak to its advantages, so comparisons are difficult from my perspective. But as I said, tf-lite and offer benefits like easy integration, good optimization, and as the name implies, tensorFlow.

gozzoo•7mo ago

What about the so called NPUs which are present in some modern microcontroller chips?

Show HN: LocalGPT – A local-first AI assistant in Rust with persistent memory

OpenClaw Is Changing My Life

Haskell for all: Beyond agentic coding

SectorC: A C Compiler in 512 bytes (2023)

Total surface area required to fuel the world with solar (2009)

Software factories and the agentic moment

LLMs as the new high level language

Speed up responses with fast mode

Hoot: Scheme on WebAssembly

Brookhaven Lab's RHIC concludes 25-year run with final collisions

The Architecture of Open Source Applications (Volume 1) Berkeley DB

Stories from 25 Years of Software Development

Vocal Guide – belt sing without killing yourself

First Proof

Substack confirms data breach affects users’ email addresses and phone numbers

Show HN: I saw this cool navigation reveal, so I made a simple HTML+CSS version

Wood Gas Vehicles: Firewood in the Fuel Tank (2010)

Vouch

Al Lowe on model trains, funny deaths and working with Disney

Start all of your commands with a comma (2009)

Show HN: A luma dependent chroma compression algorithm (image compression)

LineageOS 23.2

The AI boom is causing shortages everywhere else

FDA intends to take action against non-FDA-approved GLP-1 drugs

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

Where did all the starships go?

Selection rather than prediction

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Unseen Footage of Atari Battlezone Arcade Cabinet Production