But I'm also wondering about being able to run them on consumer-end hardware.
I remember using a local model 2-3 years ago and had to wait around 2-3 minutes for a basic answer to be printed. Now I'm running a "thinking" Qwen on a 16GB GPU and I'm able to do anything I'd do with Opus a couple months ago, at nearly the same speed. But that does use my entire VRAM and most of the RAM I have. No way I can also run a game or something else on the side.
But like how we went from bulky PCs to smartphones 1000x faster, and at the rate local models already improved, do you think we'll ever be able to have the same kind of models running locally, on affordable hardware, on our phones or maybe our fridges?
Not saying we should use them on anything, that will be a question for later, but strictly thinking about capabilities.
ben_w•32m ago
Yes, trivially so.
If the investment currently driving new models ceases for whatever reason, there's plenty of ways to burn weights into read-only hardware that's much more energy efficient and compact, we don't even need new lithography nodes for smaller transistors.
Only a few are attempting this today, because the model update cycle is so fast compared to mass production of hardware, that any wights you burn into hardware are likely obsolete before you can ship them.
If you want to get sci-fi about it: what happens when we can take a biopsy from someone to get some stem cells, use it to tissue culture a brain organoid, use a nano-electrode "neural lace" to send the right electrical impulses to exploit biological processes so as to wire up the synapses to encode the same patterns as some AI model (probably not a Transformer architecture, those are situationally good for computers and not likely the best for biology), then implant it in the head of whoever you got the tissue sample from?
With currently reported model sizes, at 1 param ~= 1 synapse, this would be around 1% of your normal brain volume.
Everyone could skip further education while also getting to at least "pass the exams" level knowledge in… how many subjects can LLMs already pass exams in? Does anyone even check this since GPT-4?
• https://en.wikipedia.org/wiki/Cerebral_organoid
• https://en.wikipedia.org/wiki/Long-term_potentiation