Ask HN: Local LLM agents on Jetson/RPi without a heavy runtime
3•takuya_h•6mo ago
I'd like to design agent flows on my Mac and ship a single tiny binary to Jetson/RPi. LLMs run on-device (no cloud). If you've tried this, what constraint hit first? Pointers to tools or write-ups you liked are welcome.
Comments
takuya_h•6mo ago
Seen Flowise/Dify/Node-RED; specifically curious about compiled-to-binary patterns for on-device LLMs.
giantg2•6mo ago
I don't have much experience but I played around with a USB Coral TPU. I did not try it with an LLM, but with used some image recognition stuff. Seemed OK, but wasn't as intense as an LLM would be. You'd probably have better luck with the Jetson you mentioned, or with one of Coral's more powerful options.
takuya_h•6mo ago