If we can get this down to a single Raspberry Pi, then we have crazy embedded toys and tools. Locally, at the edge, with no internet connection.
Kids will be growing up with toys that talk to them and remember their stories.
We're living in the sci-fi future. This was unthinkable ten years ago.
chagpt is literally leading ppl with higher education to have full on psychosis by feeding into their insane delusions and confirmation bias, im sure a less smart version of this is a perfect toy for a kid w/o a fullt developed brain yet
literally go touch grass bro...
Graphics cards with decent amount of memory are still massively overpriced (even used), big, noisy and draw a lot of energy.
2. This remain slow
3. The context window used here is likely 8k or similar which makes it unusable for bigger input/output.
Models already work fine on phones just try https://github.com/google-ai-edge/gallery and you will see local AI running on phones fine.
geerlingguy•59m ago
alchemist1e9•26m ago
If that problem gets solved, even if for only a batch approach that enables parallel batch inference resulting in high total token/s but low per session, and for bigger models, then it would he a serious game changer for large scale low cost AI automation without billions capex. My intuition says it should be possible, so perhaps someone has done it or started on it already.