How AI on Microcontrollers Works: Operators and Kernels

https://danielmangum.com/posts/ai-microcontrollers-operators-kernels/

83•hasheddan•7mo ago

Comments

Neywiny•7mo ago

In the same way "embedded" is relative, I appreciate the author's recognition that "edge" is relative. For some, AI at the edge means on-prem server farms. For some it means a mini-pc. For others, maybe an SBC. Here it's a microcontroller. Further still is AI within the sensors a microcontroller would talk to. That's probably just another microcontroller but still.

woliveirajr•7mo ago

Microcontrollers all the way down

astrobe_•7mo ago

There's "micro" and "micro". The microcontroller operating a simple coffee machine, or a simple washing-machine is probably 8 or 16 bits. This is what I would call "bare metal", as they don't run an OS, only off-the-shelf frameworks at best.

For "bigger" devices, it's usually a Cortex inside a system-on-chip or system-on-module, 32 bits single core and a few Mb of RAM for low-end (enough to run regular Linux distro instead of uClinux for instance), 64 bits multicore for high-end devices that deal with audio/video. That kind of business is often resource-hungry in every way.

I work with that kind of stuff, and to me these "microcontrollers" are just monsters that I hesitate to call "micro" when some of my coworkers work on much smaller chips with only a few K of RAM available.

Neywiny•7mo ago

I do wish sometimes they used the bigger micro, though. We have some power supplies that technically have an Ethernet interface. But when using it, even for SCPI over TCP (forget about the virtual front panel that takes a minute to update), it lags so bad the output enable button needs a few tries to toggle. I should practice yanking the positive wire for an emergency

amelius•7mo ago

Indeed. Is nVidia Jetson "edge" or not?

simgt•7mo ago

Of course it is. Edge mostly stands for network edge, Jetsons aren't meant to be deployed in data centers.

amelius•7mo ago

I don't understand why we need a separate "lite" format for microcontrollers.

Wouldn't it be advantageous if we used ONNX for everything? https://onnx.ai/

batuhandumani•7mo ago

tf-lite micro library has many advantages, and the first of these is the tensorFlow framework itself. You can train the model easily and then implement the same or a similar architecture on esp-32s without much effort. Another advantage is its optimization and you can easily intervene in various memory optimizations and even though it is not a large one, it does have a community.

Apart from these, for example, the author implemented the model the traditional way using C, but it is more convenient to use tf-lite micro on esp32s with the Berry script language.

However, since I have never used onnxin this kind of project, I cant speak to its advantages, so comparisons are difficult from my perspective. But as I said, tf-lite and offer benefits like easy integration, good optimization, and as the name implies, tensorFlow.

gozzoo•7mo ago

What about the so called NPUs which are present in some modern microcontroller chips?

Show HN: Animalese

StrongDM's AI team build serious software without even looking at the code

John Haugeland on the failure of micro-worlds

Show HN: I built an invoicing SaaS with AI-generated invoice templates

Velocity

Corning Invented a New Fiber-Optic Cable for AI and Landed a $6B Meta Deal [video]

Show HN: XAPIs.dev – Twitter API Alternative at 90% Lower Cost

Near-Instantly Aborting the Worst Pain Imaginable with Psychedelics

Show HN: Nginx-defender – realtime abuse blocking for Nginx

The Super Sharp Blade

Smart Homes Are Terrible

What I haven't figured out

KPMG pressed its auditor to pass on AI cost savings

Open-source Claude skill that optimizes Hinge profiles. Pretty well.

First Proof

I squeezed a BERT sentiment analyzer into 1GB RAM on a $5 VPS

Kagi Translate

Building Interactive C/C++ workflows in Jupyter through Clang-REPL [video]

Tactical tornado is the new default

Full-Circle Test-Driven Firmware Development with OpenClaw

Automating Myself Out of My Job – Part 2

Dependency Resolution Methods

Crypto firm apologises for sending Bitcoin users $40B by mistake

Show HN: iPlotCSV: CSV Data, Visualized Beautifully for Free

There's no such thing as "tech" (Ten years later)

List of unproven and disproven cancer treatments

Me/CFS: The blind spot in proactive medicine (Open Letter)

Ask HN: What are the word games do you play everyday?

Show HN: Paper Arena – A social trading feed where only AI agents can post

TOSTracker – The AI Training Asymmetry