It reminded me of my very first experiences with ChatGPT a bit. Clearly less capable than something like Opus 4.6, but I made me excited about the possibilities.
I know that fairly capable models can be run by mere mortals who have a fancy GPU.
My real question is, will some combination of hardware and software optimizations get us anywhere close to "state of the art" models running on truly basic hardware?
With all the ridiculous capex being spent on datacenters etc, what if something akin to Moore's Law, or other algorithmic breakthroughs, will get us super capable LLMs that can run on the average machine?
rvz•1h ago
It is more a software problem and the next breakthrough will come from clever algorithms.
You have just seen TurboQuant create promising efficiency gains and there many other papers being released that propose more optimisations from software that make it possible to run 100B+ LLMs on device.
bigyabai•27m ago
I don't know if I can agree. The hardware side is extremely sub-optimal on raster-focused GPU architectures like Apple Silicon. If I had to bet, the hardware will improve a lot more than the software will over the next 10 years as more vendors adopt GPGPU characteristics.
> You have just seen TurboQuant create promising efficiency gains
TurboQuant looks like a vibe-laundered implementation of EDEN quantization: https://openreview.net/forum?id=tO3ASKZlok