I think some really cool ways to get this have been done already like the technique of spreading an LLMs weights across a fast-enough SSD and then only serving up an expert at a time on the GPU/SoC.
I have this idea to make a tool that lets people download several small open-weight models and they act like experts but are different very task specific models with a harness designed to keep track of their thoughts & work to let normal hardware be able to get close to frontier (or just useful level intelligence).
Basically, I want to have a model that acts like a router or decision maker where you start your chat with initially (say a coding task for example) and it decides what model does the work next. If it needs to research it writes down in an .MD what it needs to find out, if it needs to write code, another model starts working but the other models memory & work get saved to an .MD (or folder of .MDs) and lets you get real work done but completely locally.
This might already be done but its just an idea I had and I wanted to see what the smart people on the internet thought or how they would approach this. I might also just be able to ask AI.