Hi HN! I wanted to run an LLM locally on my 24GB M4 Pro Mac mini without needing to kill all other apps to free up the RAM for the inference.
So I built a runtime based on llama.cpp that uses SSD offloading and Lookahead Expert Scheduling. I introduced optimizations like adaptive caching and prefill batching, I/O and GPU work overlap, GPU keepalive to keep frequency high.
Turns out that this runtime now lets me run Qwen 3.8 Flash Next (RCO GSQ IQ3_XXS from ISTA DASLab) at 10 t/s decoding, 160 t/s in prefill and keep it 7.5GB peak process memory footprint. That’s double the end-to-end speed and half the RAM footprint compared to Strata with the same model on this machine.
Currently the project is an open-source alpha, I am still optimizing and adding support to more model architectures. Currently tested models are Qwen 3.8 FN and Qwen 3.6 35B A3B (Unsloth’s Q6) that runs at 400+ t/s in prefill.
The project is called MoEShift and I’d love to see how it performs on your Macs: you can share the benchmark as a GitHub issue using the benchmark runner that I prepared.
Of course technical feedback is much appreciated.
davidetroiani•41m ago
So I built a runtime based on llama.cpp that uses SSD offloading and Lookahead Expert Scheduling. I introduced optimizations like adaptive caching and prefill batching, I/O and GPU work overlap, GPU keepalive to keep frequency high. Turns out that this runtime now lets me run Qwen 3.8 Flash Next (RCO GSQ IQ3_XXS from ISTA DASLab) at 10 t/s decoding, 160 t/s in prefill and keep it 7.5GB peak process memory footprint. That’s double the end-to-end speed and half the RAM footprint compared to Strata with the same model on this machine.
Currently the project is an open-source alpha, I am still optimizing and adding support to more model architectures. Currently tested models are Qwen 3.8 FN and Qwen 3.6 35B A3B (Unsloth’s Q6) that runs at 400+ t/s in prefill.
The project is called MoEShift and I’d love to see how it performs on your Macs: you can share the benchmark as a GitHub issue using the benchmark runner that I prepared. Of course technical feedback is much appreciated.
Link to source code, architecture details, benchmarks and install instructions: https://github.com/davide-troiani/moeshift
Cheers!