Apple's MLX adding CUDA support

https://github.com/ml-explore/mlx/pull/1983

169•nsagent•4h ago

Comments

gsibble•3h ago

Awesome

nxobject•3h ago

If you're going "wait, no Apple platform has first-party CUDA support!", note that this set of patches also adds support for "Linux [platforms] with CUDA 12 and SM 7.0 (Volta) and up".

https://ml-explore.github.io/mlx/build/html/install.html

teaearlgraycold•3h ago

I wonder if Jensen is scared. If this opens up the door to other implementations this could be a real threat to Nvidia. CUDA on AMD, CUDA on Intel, etc. Might we see actual competition?

jsight•3h ago

I think this is the other way around. It won't be cuda on anything except for nvidia.

However, this might make mlx into a much stronger competitor for Pytorch.

teaearlgraycold•3h ago

Oh bummer. Almost got excited.

baby_souffle•3h ago

If you implement compatible apis, are you prohibited from calling it cuda?

moralestapia•3h ago

I'm sure I saw this lawsuit somewhere ...

The gist is the API specification in itself is copyright, so it is copyright infringement then.

wyldfire•3h ago

Too subtle - was this oracle vs java one? Remind me: java won or lost that one?

mandevil•2h ago

Oracle sued Google, and Google won, 6-2 (RBG was dead, Barrett had not yet been confirmed when the case was heard).

Supreme Court ruled that by applying the Four Factors of Fair Use, Google stayed within Fair Use.

An API specification ends up being a system of organizing things, like the Dewey Decimal System (and thus not really something that can be copyrighted), which in the end marks the first factor for Google. Because Google limited the Android version of the API to just things that were useful for smart phones it won on the second factor too. Because only 0.4% of the code was reused, and mostly was rewritten, Google won on the third factor. And on the market factor, if they held for Oracle, it would harm the public because then "Oracle alone would hold the key. The result could well prove highly profitable to Oracle (or other firms holding a copyright in computer interfaces) ... [but] the lock would interfere with, not further, copyright's basic creativity objectives." So therefore the fourth factor was also pointing in Google's favor.

Whether "java" won or lost is a question of what is "java"? Android can continue to use the Java API- so it is going to see much more activity. But Oracle didn't get to demand license fees, so they are sad.

moralestapia•2h ago

Oh man, thanks for this.

I always thought it was resolved as infringement and they had to license the Java APIs or something ...

Wow.

mandevil•2h ago

The district court ruled for Google over patents and copyright- that it was not a copyright at all, the Court of Appeals then reversed and demanded a second court trial on whether Google was doing fair use of Oracle's legitimate copyright, which the district court again held for Google, and then the Court of Appeals reversed the second ruling and held for Oracle that it was not fair use of their copyright, and then Google appealed that to the the Supreme Court ... and won in April 2021, putting an end to this case which was filed in August 2010. But the appeals court in between the district court and the Supreme Court meant that for a long while in the middle Oracle was the winner.

This is part of why patents and copyrights can't be the moat for your company. 11 years, with lots of uncertainty and back-and-forth, to get a final decision.

tough•51m ago

Yeah this case made me think using llms to clean-room reverse engineer any API exposing SaaS or private codebase would be game

15155•3h ago

Considering 100% of the low-level CUDA API headers have the word "CUDA" in them, this would be interesting to know.

mayli•2h ago

Yeah, nice to have MLX-opencl or MLX-amd-whatever

almostgotcaught•3h ago

> CUDA backend

backend

tekacs•3h ago

This instance is the other way around, but that's what this is – CUDA on AMD (or other platforms): https://docs.scale-lang.com/stable/

zdw•3h ago

How does this work when one of the key features of MLX is using a unified memory architecture? (see bullets on repo readme: https://github.com/ml-explore/mlx )

I would think that bringing that to all UMA APUs (of any vendor) would be interesting, but discreet GPU's definitely would need a different approach?

edit: reading the PR comments, it appears that CUDA supports a UMA API directly, and will transparently copy as needed.

nerdsniper•3h ago

Edit: I had the details of the Google v Oracle case wrong. SCOTUS found that re-implementing an API does not infringe copyright. I was remembering the first and second appellate rulings.

Also apparently this is not a re-implementation of CUDA.

skyde•3h ago

this is CUDA backend to MLX not MLX backend for CUDA!

liuliu•3h ago

You misunderstood and this is not re-implementing CUDA API.

MLX is a PyTorch-like framework.

Uehreka•3h ago

This is exactly the kind of thing I wouldn’t opine on until like, an actual lawyer weighs in after thoroughly researching it. There are just too many shades of meaning in this kind of case law for laymen to draw actionable conclusions directly from the opinions.

Though I imagine that if Apple is doing this themselves, they likely know what they’re doing, whatever it is.

MuffinFlavored•3h ago

Is this for Mac's with NVIDIA cards in them or Apple Metal/Apple Silicon speaking CUDA?... I can't really tell.

Edit: looks like it's "write once, use everywhere". Write MLX, run it on Linux CUDA, and Apple Silicon/Metal.

cowsandmilk•3h ago

Neither, it is for Linux computers with NVIDIA cards

MBCook•3h ago

Seems you already found the answer.

I’ll note Apple hasn’t shipped an Nvidia card in a very very long time. Even on the Mac pros before Apple Silicon they only ever sold AMD cards.

My understanding from rumors is that they had a falling out over the problems with the dual GPU MacBook Pros and the quality of drivers.

I have no idea if sticking one in on the PCI bus let you use it for AI stuff though.

kmeisthax•2h ago

On Apple Silicon, writing to memory on a PCIe / Thunderbolt device will generate an exception. ARM spec says you're allowed to write to devices as if they were memory but Apple enforces that all writes to external devices go through a device memory mapping[0]. This makes using an external GPU on Apple Silicon[1] way more of a pain in the ass, if not impossible. AFAIK nobody's managed to write an eGPU driver for Apple Silicon, even with Asahi.

[0] https://developer.arm.com/documentation/102376/0200/Device-m...

[1] Raspberry Pi 4's PCIe has the same problem AFAIK

bobmcnamara•1h ago

Ewww, that kills out of order CPU performance. If it's like ARMv7, it effectively turns each same-page access into it's own ordering barrier.

xuki•2h ago

That particular MBP model had a high rate of GPU failure because it ran too hot.

I imagined the convo between Steve Jobs and Jensen Huang went like this:

S: your GPU is shit

J: your thermal design is shit

S: f u

J: f u too

Apple is the kind of company that hold a grudge for a very long time, their relationships with suppliers are very one way, their way or the highway.

bobmcnamara•1h ago

S: omg so thin!!1!1!!l!

rcruzeiro•1h ago

I think the ones that failed were the AMD ones, specifically the old 17 inches MacBook Pro.

VladVladikoff•40m ago

Won’t work. No driver support.

dkga•3h ago

This is the only strategy humble me can see working for CUDA in MLX

hbcondo714•51m ago

> "write once, use everywhere"

So my MLX workloads can soon be offloaded to the cloud!?

Keyframe•3h ago

Now do linux support / drivers for Mac hardware!

lvl155•3h ago

Seriously. Those Apple guys became delusional especially after Jobs passed away. These guys just sat on their successes and did nothing for a decade plus. M1 was nice but that was all Jobs doing and planning. I don’t like this Apple. They forgot how to innovate.

But I guess we have a VR device nobody wants.

jjtheblunt•2h ago

It would be funny if you were typing out your response on an iPhone that has been running for 36 hours without recharging.

macinjosh•2h ago

if only their batteries would last that long.

marcellus23•1h ago

> M1 was nice but that was all Jobs doing and planning

M1 was launched 9 years after Jobs died. You're saying they had everything ready to go back then and just sat on their asses for a decade?

lvl155•1h ago

Who bought Semi? Jobs knew they had to make their own. M1 is just a product of their iPhone chips hence all the efficiency.

bigyabai•1h ago

I think we're seeing the twilight of those efforts. Asahi Linux was an absolute powerhouse of reverse-engineering prowess, and it took years to get decent Vulkan coverage and half of the modern lineup's GPUs supported. Meanwhile AMD and even Intel are shipping Vulkan 1.3 drivers day-one on new hardware. It's a cool enthusiast effort to extend the longevity of the hardware, but it bears repeating; nobody is disrupting Nvidia's bottom-line here. Apple doesn't sell hardware competitive with Nvidia's datacenter hardware, and even if they did it's not supported by the community. It's doubtful that Apple would make any attempt to help them.

There seems to a pervading assumption that Apple is still making a VolksComputer in 2025, blithely supporting a freer status-quo for computing. They laid out their priorities completely with Apple Silicon, you're either on Apple's side or falling behind. Just the way they want it.

albertzeyer•3h ago

This is exciting. So this is using unified memory of CUDA? I wonder how well that works. Is the behavior of the unified memory in CUDA actually the same as for Apple silicon? For Apple silicon, as I understand, the memory is anyway shared between GPU and CPU. But for CUDA, this is not the case. So when you have some tensor on CPU, how will it end up on GPU then? This needs a copy somehow. Or is this all hidden by CUDA?

MBCook•3h ago

This is my guess, but does higher end hardware they sell, like the server rack stuff for AI, perhaps have the unified memory?

I know standard GPUs don’t.

The patch suggested one of the reasons for it was to make it easy to develop on a Mac and run on a super computer. So the hardware with the unified memory might be in that class.

Y_Y•3h ago

The servers don't, but the Jetsons do

ajuhasz•2h ago

The Jetsons[1] have unified memory[2].

[1] https://www.nvidia.com/en-us/autonomous-machines/embedded-sy... [2] https://www.nvidia.com/en-us/on-demand/session/gtcspring22-s...

tonyarkles•2h ago

They sure do and it's pretty amazing. One iteration of a vision system I worked on got frames from a camera over a Mellanox NIC that supports RDMA (Rivermax), preprocessed the images using CUDA, did inference on them with TensorRT, and the first time a single byte of the inference pipeline hit the CPU itself was when we were consuming the output.

patrickkrusiec•2h ago

The physical memory is not be unified, but on modern rack scale Nvidia systems, like Grace Hopper or NVL72, the CPU and the GPU(s) share the same virtual address space and have non-uniform memory access to each other's memory.

zcbenz•2h ago

In the absence of hardware unified memory, CUDA will automatically copy data between CPU/GPU when there are page faults.

fenced_load•2h ago

There is also NVLink c2c support between Nvidia's CPUs and GPUs that doesn't require any copy, CPUs and GPUs directly access each other's memory over a coherent bus. IIRC, they have 4 CPU + 4 GPU servers already available.

benreesman•1h ago

Yeah NCCL is a whole world and it's not even the only thing involved, but IIRC that's the difference between 8xH100 PCI and 8xH100 SXM2.

nickysielicki•50m ago

saagarjha•8m ago

This seems like it would be slow…

paulirish•1h ago

It's coming from zcbenz who created Electron among others https://zcbenz.com/ Nice.

benreesman•1h ago

I wonder how much this is a result of Strix Halo. I had a fairly standard stipend for a work computer that I didn't end up using for a while so I recently cashed it in on the EVO-X2 and fuck me sideways: that thing is easily competitive with the mid-range znver5 EPYC machines I run substitors on. It mops the floor with any mere-mortal EC2 or GCE instance, like maybe some r1337.xxxxlarge.metal.metal or something has an edge, but the z1d.metal and the c6.2xlarge or whatever type stuff (fast cores, good NIC, table stakes), blows them away. And those things are 3-10K a month with heavy provisioned IOPS. This thing has real NVME and it cost 1800.

I haven't done much local inference on it, but various YouTubers are starting to call the DGX Spark overkill / overpriced next to Strix Halo. The catch of course is ROCm isn't there yet (they're seeming serious now though, matter of time).

Flawless CUDA on Apple gear would make it really tempting in a way that isn't true with Strix so cheap and good.

jitl•1h ago

It’s pretty explicitly targeting cloud cluster training in the PR description.

nl•1h ago

> The catch of course is ROCm isn't there yet (they're seeming serious now though, matter of time).

Competitive AMD GPU neural compute has been any day now for at least 10 years.

bigyabai•1h ago

The inference side is fine, nowadays. llama.cpp has had a GPU-agnostic Vulkan backend for a while, it's the training side that tends to be a sticking point for consumer GPUs.

hamandcheese•1h ago

For the uninitiated, Strix Halo is the same as the AMD Ryzen AI Max+ 395 which will be in the Framework Desktop and is starting to show up in some mini PCs as well.

The memory bandwidth on that thing is 200GB/s. That's great compared to most other consumer-level x86 platforms, but quite far off of an Nvidia GPU (a 5090 has 1792GB/s, dunno about the pro level cards) or even Apple's best (M3 Ultra has 800GB/s).

It certainly seems like a great value. But for memory bandwidth intensive applications like LLMs, it is just barely entering the realm of "good enough".

yieldcrv•33m ago

Apple is just being stupid, handicapping their own hardware so they can sell the fixed one next year or the year after

This is time tested Apple strategy is now undermining their AI strategy and potential competitiveness

tl;dr they could have done 1600GB/s

saagarjha•10m ago

They could have shipped a B200 too. Obviously there are reasons they don't do that.

Rohansi•13m ago

You're comparing theoretical maximum memory bandwidth. It's not enough to only look at memory bandwidth because you're a lot more likely to be compute limited when you have a lot of memory bandwidth available. For example, M1 had so much bandwidth available that it couldn't make use of even when fully loaded.

attentive•39m ago

how is it vs m4 mac mini?

orliesaurus•56m ago

Why is this a big deal, can anyone explain if they are familiar with the space?

elpakal•51m ago

> NVIDIA hardware is widely used for academic and massive computations. Being able to write/test code locally on a Mac and then deploy to super computers would make a good developer experience.

That one stands out to me as a mac user.

radicaldreamer•41m ago

MacBooks used to use Nvidia GPUs, then Apple had a falling out with Nvidia and the beef stands to this day (Apple didn’t use Nvidia hardware when training it’s own LLMs for Apple Intelligence).

I wouldn’t be surprised if within the next few years we see a return of Nvidia hardware to the Mac, probably starting with low volume products like the MacPro, strictly for professional/high-end use cases.

fooker•40m ago

> Apple didn’t use Nvidia hardware when training it’s own LLMs for Apple Intelligence

Do you have some links for this?

almostgotcaught•21m ago

People on hn make up more BS than your local bar

https://www.investors.com/news/technology/apple-stock-apple-...

numpad0•20m ago

> This PR is an ongoing effort to add a CUDA backend to MLX

looks like it allows MLX code to compile and run on x86 + GeForce hardware, not the other way around.