Apple Silicon and macOS VMs: 11–16× Faster LLM Inference with Llama.cpp

https://github.com/trycua/cua/blob/main/blog/gpu-passthrough-macos-vms.md

45•frabonacci•52m ago

Comments

thehamkercat•31m ago

> 11.08× faster and generated tokens 16.36× faster than the same workload in the same stock VM.

So this was the comparison, for me the title was a bit confusing

frabonacci•16m ago

yeah fair point. it's always tricky to get the whole idea across within HN's title limit. tldr: we ran the same workload in the same Lume macOS VM on the same Apple Silicon host, first with stock Metal capability reporting and then with our process-scoped dynamic library. The 11.08x figure is prompt processing, while 16.36x is token generation. the mechanism technically extends to graphics workloads too but these figures are specifically from llama.cpp

azinman2•26m ago

I don’t understand what Apple 1-9 are. At first I thought it was M series chips but there is no M9 (yet)

niklasbuschmann•22m ago

https://developer.apple.com/documentation/metal/mtlgpufamily

wtallis•8m ago

So those generation numbers aren't really anchored to Apple's hardware designs. It's just counting from when Apple introduced the Metal API, and the first several generations were when the GPU cores Apple was using were still nominally PowerVR designs.

simonw•24m ago

It looks to me like this won't speed up llama.cpp for everyone, just for users running it in this particular kind of Virtualization.framework VM.

The fix here works around a problem where the VM was causing llama.cpp to select the wrong kernels.

engzaanin•14m ago

That makes sense. The title initially sounded like a general llama.cpp speedup on Apple Silicon, but if the improvement comes from fixing kernel selection inside Virtualization.framework VMs, that distinction is pretty important.

purplemoonx•12m ago

Still no easy way to get up and running performantly with Llama.cpp

Nobody tells you how, they just act like you're an idiot. So I always say fuck this and install Ollama.

Then everyone goes BLASPHEMY "just use llama.cpp"

Yeah I did, and it's slow as hell. It doesn't work well. Idk why.

"You aren't doing it right"

Okay tell me how

"NO"

-----

For this reason, Ollama is the superior solution. I know, downvote, everyone hates Ollama here but until llama.cpp gets their shit together on developer experience it doesn't exist as far as I'm concerned

unglaublich•11m ago

I think people generally throw Claude or Codex at the configuration challenge, so they don't know either.

purplemoonx•4m ago

Maybe the llama.cpp dev loved webpack as a child, or just loves making the most simple thing complicated as hell for no reason lol

drittich•6m ago

There are certainly challenges. When setting up a new model, I get AI to walk me through the commands using llama-benchmark that determine the best parameters for my particular configuration and needs. Once you've got that it's pretty easy to port those parameters to llama-server. It takes me about an hour to run through this process. It would be great if there was a registry of hardware, models, configuration parameters, and resulting tokens per second. Maybe one day we'll get there.

kevin42•5m ago

What hardware do you run? I have a first-gen mac studio, and I just run cmake and build with no special options. Same thing with llama-server, I just specify the model and use the built-in web UI.

For reference, I get ~26 tok/sec with the new Muse 30B model.

shay_ker•6m ago

I recall there was another YC startup that was working on Mac-specific ML optimizations for local inference (and perhaps fine-tuning).

I wonder if their work is related?

Mother of all VPNs v2 release: Same 16 protocols, rebuilt underneath

We built our own status page with AI, replacing a $65k SaaS product

Jackie the bald eagle dead at 14

Nvidia Nemotron 3.5 Lightning

What does International Vlogging Day mean in 2026?

PDCA

AI Is Accelerating Vulnerability Discovery

Scrunch: Rewriting the Web for AI

Pure-Rust, Sandboxed, Browser for Claude Code / Codex

The Vertical AI Bubble: We Keep Forgetting That LLMs Roll Dice

Share What You Learned, Not Just Your AI Conversation

Low-Tech Ceramic Water Filter

Learning from Historical Mistakes

Manually Unbreakable Cryptography

Dogs and Fat Tails

Why did I build a product no one needs?

Free tokens for sale: How fake signups drive AI fraud

A quick look at zero-knowledge proofs

How the Heck Does JPEG Work?

Show HN: A football SIM that has run 8 seasons – every match deterministic

Summer 2026 on course to be UK's hottest on record, says Met Office

Spotify Xirp

Stealing Reasoning Traces from Proprietary LLM APIs [pdf]

Program with Paint Brushes, Not Pencils

Show HN: Prompting Refinement Tool [requesting testing]

Microsoft responds to unwanted OneDrive feature installs

Changes to the Open Science Framework

Harvesting Ethereum Traces Without an Archive Node

Mozilla revokes Firefox signing key after unencrypted copy lands in GitHub

Leanroute Is Live One AI Gateway for Models and Tools