Today I benchmarked Qwen 3.8 Flash Next IQ3 XXS on M4 Pro and M5 Pro (both 24GB unified memory) using the same configuration on MoEShift.
Results (M4 Pro vs M5 Pro):
- Short prompt decode: 9.84 vs 14.69 tok/s (+49%)
- Long prompt prefill (6,060 tokens): 199 vs 374 tok/s (+88%)
- Long prompt decode: 9.57 vs 13.14 tok/s (+37%)
- Long prompt inference time: 83.9 vs 55.1 seconds
- Peak RAM process footprint: 8.5GB on both Macs
The major difference is the prefill speed thanks to the higher SSD throughput of the M5 Pro, which directly impacts the expert streaming.
What impresses me the most is running Flash Next at these speeds on so low a RAM budget, leaving memory also for other apps which is the reason for this project.
davidetroiani•11m ago
Results (M4 Pro vs M5 Pro):
- Short prompt decode: 9.84 vs 14.69 tok/s (+49%)
- Long prompt prefill (6,060 tokens): 199 vs 374 tok/s (+88%)
- Long prompt decode: 9.57 vs 13.14 tok/s (+37%)
- Long prompt inference time: 83.9 vs 55.1 seconds
- Peak RAM process footprint: 8.5GB on both Macs
The major difference is the prefill speed thanks to the higher SSD throughput of the M5 Pro, which directly impacts the expert streaming.
What impresses me the most is running Flash Next at these speeds on so low a RAM budget, leaving memory also for other apps which is the reason for this project.
For reproduction, I ran the benchmark using the stock benchmark runner included in the repo: https://github.com/davide-troiani/moeshift