After checking out Turbo Fieldfare (a very interesting project that streamed experts from SSD on Apple Silicon Macs to run Gemma 4 26B with very little RAM usage), I tried to make this project much faster and succeeded!

This project gets up to 100+ tokens/s inference on Gemma 4 26B (Turbo Fieldfare runs at 20-30 on similar hardware) and doesn't have insanely slow prefill times due to the ANE helping (I had to reverse engineer Apple's APIs to get this working).

I also implemented speculative decoding, and while Turbo steered away from using mmap, this repo utilizes it so the OS file cache serves as the expert cache.