While model-specific inference engine vs general debate is quite prevalent, I asked if the real bottleneck is the primitive, I think the real answer lies in redesigning a new primitive!
With agents, a mlx-alternative didn't seem too ambitious even for a weekend project!
Carbon is a tile-oriented first Python DSL, so the inference engine up-top(Swiss -> https://github.com/Lazarus-931/swiss), can be kept simple, yet deliver on-par performance as model specific engines like strata/ds4!
The most promising thing is, we can continuously iterate carbon's internals to keep swiss clean, performative and simple.
On an M4 Mac mini (16 GB), at batch sizes 1, 2 and 4 and contexts of 2K–16K, Swiss on Qwen3.5-9B ran prefill at 1.07× and decode at 1.00× the speed of mlx-lm on average, and prefill at 1.08× and decode at 1.08× the speed of mlx-vlm. On Qwen3-8B it was 1.07× and 1.05× against mlx-lm, and 1.08× and 1.05× against mlx-vlm. On Qwen3-4B it was 1.10× and 1.07× against mlx-lm, and 1.08× and 1.07× against mlx-vlm. On LFM3-1.7B it was 1.15× and 1.08× against mlx-vlm; mlx-lm does not support LFM3. On Gemma 4 12B, at batch 1 only, it was 1.20× and 1.02× against mlx-lm, and 1.18× and 1.02× against mlx-vlm.
With speculative decoding on the same M4 (batch 1, prompts of up to 1.5K tokens, greedy), Swiss runs Gemma 4 12B with its bf16 assistant at 1.17–1.30× the speed of mlx-vlm with the same drafter, and with 4-bit drafters it reaches up to 2.7× its own plain decode, for example 41–57 tok/s on Qwen3.5-9B with DFlash against 29–35 tok/s for mlx-vlm with DFlash.
I think this was a fun experiment, and hopefully will now explore further agent driven optimizations in carbonIR. I think there is more efficiency we can squeeze out!
More models will be included once/if hardware to run them on, feel free to contribute as well!
alazarmb•11m ago
With agents, a mlx-alternative didn't seem too ambitious even for a weekend project!
Carbon is a tile-oriented first Python DSL, so the inference engine up-top(Swiss -> https://github.com/Lazarus-931/swiss), can be kept simple, yet deliver on-par performance as model specific engines like strata/ds4!
The most promising thing is, we can continuously iterate carbon's internals to keep swiss clean, performative and simple.
On an M4 Mac mini (16 GB), at batch sizes 1, 2 and 4 and contexts of 2K–16K, Swiss on Qwen3.5-9B ran prefill at 1.07× and decode at 1.00× the speed of mlx-lm on average, and prefill at 1.08× and decode at 1.08× the speed of mlx-vlm. On Qwen3-8B it was 1.07× and 1.05× against mlx-lm, and 1.08× and 1.05× against mlx-vlm. On Qwen3-4B it was 1.10× and 1.07× against mlx-lm, and 1.08× and 1.07× against mlx-vlm. On LFM3-1.7B it was 1.15× and 1.08× against mlx-vlm; mlx-lm does not support LFM3. On Gemma 4 12B, at batch 1 only, it was 1.20× and 1.02× against mlx-lm, and 1.18× and 1.02× against mlx-vlm.
With speculative decoding on the same M4 (batch 1, prompts of up to 1.5K tokens, greedy), Swiss runs Gemma 4 12B with its bf16 assistant at 1.17–1.30× the speed of mlx-vlm with the same drafter, and with 4-bit drafters it reaches up to 2.7× its own plain decode, for example 41–57 tok/s on Qwen3.5-9B with DFlash against 29–35 tok/s for mlx-vlm with DFlash.
I think this was a fun experiment, and hopefully will now explore further agent driven optimizations in carbonIR. I think there is more efficiency we can squeeze out!
More models will be included once/if hardware to run them on, feel free to contribute as well!
I'd appreciate feedback or ideas!