It's great that the efficiencies available can be shown to be extractable. The real, much harder, trick is putting together a sufficiently smart compiler to enable them for heterogeneous compute setups.
The problem is that even for things like consoles, it's usually more "cost efficient" to write normal fast-to-write code that isn't maximally effective, let the compiler do its magic, and call it good enough.
Sometimes I dream of what the world would do if we were mystically stuck on exactly the processors we have today, for twenty years.
So in reality the opportunities to really code against a specific piece of hardware are few and far between...
Heck, then you get into multiple operating modes of the same hardware - the Nintendo Switch has a different perf profile if it's docked vs. not.
Reminds me of the old American cars in Cuba - https://en.wikipedia.org/wiki/Yank_tank
Given all the time and money, there's also a skills gap.
But then, this still wouldn't incentivize building directly to the hardware, because of the need to run on a large variety of different hardware. You're still better off preferencing portability over performance, and then making it up by cutting scope and ease of development.
Don't forget disposable vapes: https://news.ycombinator.com/item?id=45252817
This is one of my back-of-mind hopes for AI. Enlist computers as our allies in making computer software faster. Imagine if you could hand a computer brain your code, and ask it to just make the program faster. It becomes a form of RL problem, where the criteria are 1) a functionally equivalent program 2) that is faster.
Who confirms what is functionally equivalent?
However, if you stick to only defined behaviour, they are 'functionally equivalent', if your compiler doesn't have a bug.
I think a more interesting case might be showing functional equivalence on some subset of all inputs (because tbh, showing functional equivalence on all inputs often requires "doing certain things the slow way").
An even more interesting case might be "inputs of up to a particular complexity in execution" (which is... very hard to calculate, but likely would mean combining ~code coverage & ~path coverage).
Of course, doing all of that w/o creating security issues (esp. with native code) is an even further out pipe dream.
I'd settle for something much simpler, like "we can automatically vectorize certain loop patterns for particular hardware if we know the hardware we're targeting" from a compiler. That's already hard enough to be basically a pipe dream.
"Developer time is so expensive, we need to throw everything under the bus to make developers fast."
The kinds of things often thrown under the bus: Optimizations, runtime speed, memory footprint, disk image size, security, bug fixing, code cleanliness / lint, and so on. The result is crappy software written fast. Now, imagine some hypothetical AI (that we don't have yet) that makes developer time spent on the project trivial.
Optimistically: There might be time for some of these important software activities.
Pessimistically: Companies will continue to throw these things under the bus and just shit out crappy software even faster.
I have not had to use Windows in ages but every time I touch it I am amazed at the fact that it takes like 10-15GB for a bare installation of the latest version, while it does about the same amount of work as XP was able to do in under 1GB. Yes I am aware assets are a thing but has usability increased as a result of larger assets?
That's a good or a bad thing depending on your perspective
For the last six years my full time job has largely been optimizing games where most of the team has been working with this mindset. Sometimes someone spends a few days of just getting things done, followed by others building on top of it. This leads to systems which are not fast enough and take me weeks or even months to optimize.
We even got together at my last job and created a series of lectures on performance and best practices for everyone, including artists, to get ahead of this type of issues. It was apparently very appreciated, especially among the non technical staff who said it was valuable and they had no idea.
The problem is that even for things like consoles, it's usually more "cost efficient" to write normal fast-to-write code that isn't maximally effective, let the compiler do its magic, and call it good enough.
https://signalsandthreads.com/why-ml-needs-a-new-programming...
Transgressing the Boundaries: Towards a Transformative Hermeneutics of Quantum Gravity
https://physics.nyu.edu/faculty/sokal/transgress_v2/transgre...
They will argue that we had it coming, and that it serves us right, and maybe they're not wrong.
(And I'm curious... The way you said "Ben here" makes me wonder if I know you?)
Please accept my sheepish apology for taking an unwarranted potshot at your writing style.
OTOH, if I hadn't, none of us would have known about that remarkable connection!
My favorite type of code
Nope.
I was also surprised, when joining such significant labs at how much relatively-low hanging fruits were still available to work on. But the reality is that there is just too much work to do, each seemingly super-important, and not enough people to do it.
Not sure I agree, if you look at the head count growth of companies like OpenAI, Anthropic etc, it is super fast, its already pretty hard to keep everything working smoothly with that rate of employee growth, so going faster than that seems very risky.
Ultimately I think it's mostly caused by the field still being so new. Everything still needs to be optimized and there just aren't that many very good CUDA programmers to start with, then you need to find one that also has deep knowledge of ML and transformers architectures, which further drains the pool. And then when you do find one of them, there is 50 different things they could be working on instead of what's in the article, all equally or more impactful. The architectures being constantly evolving also make it hard/not a great ROI to go super super deep in single digit % optimization when there is new stuff coming out all the time that can be made an order of magnitude faster.
A good example of that is flash attention: it is maybe the most significant/impactful optimization in ML of the last few years. Tl;dr is how do you fuse the entire attention pipeline together to make it much faster and avoid massive tensor materialization. The bottleneck was obvious to anyone that profiled a Transformer-based model, but there was no obvious solution because of how softmax works. Yet the paper that ultimately unblock this was published back in 2019 [1], but it took 3 years for a team to connect the dots. Most people in pure ML engineering didn't know about the paper and don't have good enough CUDA knowledge/ GPU arch understanding, most people with good CUDA knowledge don't understand ML well enough, and even the author of that 2019 paper said "[we] hypothesize that this reduction in memory accesses should improve Softmax performance on actual hardware" but didn't have the technical skills to test this or to see how that could be part of a bigger breakthrough because it requires understanding core concepts in how GPU worked and compute/memory imbalance.
You chase away your top engineers when you glom up the system with dumbfucks.
whenever anyone says this they should be required to disclose whether they've actually 1) been employed to do this work 2) how many LC rounds they've failed during their last job search ..... lol
* https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
* https://www.nvidia.com/en-us/technologies/multi-instance-gpu...
Or having multiple processes from one user share it:
https://www.usenix.org/system/files/usenixsecurity24-guo-yan...
https://www.sciencedirect.com/science/article/pii/S016740482...
> In this paper, we presented MISO, a technique to leverage the MIG functionality on NVIDIA A100 GPUs to dynamically partition GPU resources among co-located jobs. MISO deploys a learning-based method to quickly find the optimal MIG partition for a given job mix running in MPS. MISO is evaluated using a variety of deep learning workloads and achieves an average job completion time that is lower than the unpartitioned GPU scheme by 49% and is within 10% of the Oracle technique.
MIG is used a lot in HPC and multi-tenancy cloud, where isolation is important. See Figure 1 and §6.2:
* https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
The card is actually sliced into different instances (show up as different /dev/nvidiaXs), each with their own SMs, L2, and DRAM, that are isolated between each one. (MPS is for the same user to share a GPU instance: allows multiple CUDA apps to attach and time-slicing occurs.)
Question unclear: looking at to use (yes: lots in HPC, hypervisors), or looking at from a security POV (don't know)?
MPS should only be used where all the workloads trust each other. It is similar to running multiple games on your computer simultaneously.
You cannot use NVLink with MPS or MIG, it is not isolated, and malformed NVLink messages can be authored in userspace and can crash the whole GPU. Some vendors, like Modal, allow you to request NVLink'd shared GPUs anyway.
MIG only makes sense for cloud providers. MPS only makes sense for interactive (read: not ML) workloads. Workloads needing more than 1 GPU cannot use either.
Meanwhile, don’t forget that if your workloads are cooperative, you can put all the processes you want on a single GPU and they’ll happily multitask. No security boundary of course, but who knows how good MIG is at that.
I’d greatly prefer better tools for cooperative GPU sharing like per process memory limits or compute priority levels. Also seems like it should be way easier to implement. As containerization and k8 have proven, there’s a ton of utility in bin packing your own workloads better without rock solid security boundaries.
I know several HPC sites that use it: they (e.g.) ordered cookie-cutter server designs/models to simplify logistics, but not all of their users need the complete capabilities, and so they slice/dice some portion into smaller instances for smaller jobs.
E.g.:
* https://www.rc.virginia.edu/2025/07/hpc-maintenance-aug-12-2...
> Only seven slices. Seven?
At some point the slices because so small that they stop being useful. An A100 can have as 'little' as 40G of memory, and you're now down to 5G per instance:
* https://docs.nvidia.com/datacenter/tesla/mig-user-guide/#a10...
> Extremely limited hardware support.
It's a reasonable argument that you'd only need it at the top-end of the hardware: the number of workloads that need all that compute and memory are not that common, so downshifting some hardware to resource slices that are more typical is not crazy. Of course you then upshift when needed: but if you had purchased 'smaller' cards because that's what you thought you (initially) needed, then you're stuck at that level. There's no way for you to upshift/de-downshift.
> Difficult to configure - like the early days of CUDA.
How hard is it to run nvidia-smi?
> Meanwhile, don’t forget that if your workloads are cooperative, you can put all the processes you want on a single GPU and they’ll happily multitask. No security boundary of course, but who knows how good MIG is at that.
The security boundary of MIG is lot better than MPS, which basically has no security. I know several folks running HPC clusters that use it to isolate the Slurm workloads of different users. And my search-fu has found no CVEs or published papers jailbreaking out of MIG instances.
> I’d greatly prefer better tools for cooperative GPU sharing like per process memory limits or compute priority levels. Also seems like it should be way easier to implement.
This is what MPS is for:
* https://docs.nvidia.com/deploy/mps/index.html
* https://man.archlinux.org/man/extra/nvidia-utils/nvidia-cuda...
The recently leaked M5 benchmarks reveal a 35% faster GPU. These improvements compound, so you can get a GPU that's effectively twice as fast by waiting a couple of years.
Modern GPUs are the equivalent of local supercomputers, but the drivers, languages and libraries are still playing catch up. Imagine the audio processing you could do if only you could target that hardware.
That's an interesting thought. Commercial grade signal processing rely on FPGAs and the Fintech field adapted them for high frequency trading. I wonder if we will see signal processing enabled on GPUs for consumers if the GPU drivers were more open.
So, in a very real sense, signal processing is exactly what the GPU is built for and primarily used for.
35% raster improvements, it's worth noting, is not super impressive on the GPU side of things. Most raster compute is a square function, to double your render resolution you need a 4x the GPU power (on-paper) to handle the pixel count. That's what, six years of annual iteration? A large component of Apple and AMD's inability to break into Nvidia's CUDA empire is their obsession over raster optimization in a world where DLSS and FSR exists. It's a noble pursuit, but even as a gamer I've gotta admit they're wasting their time. We have software methods that can close the gap in render quality between $100 GPUs and $1000 GPUs, but no such solution for GPGPU compute.
I won an GPU hackathon back in 2019 doing something very similar to this; although the other way around, I was compressing weights using hardware modules.
Maybe… you don’t actually want or need to use all the features of something you bought. Particularly given that GPUs previously used for cryptocurrency mining may have damaged themselves while being run full out for a year straight.
the writeup is a classic example of what we lose through abstraction and how writing custom (and optimized) code still beats sticking to high-level implementations.
i would go further and say that the "megakernel" written as part of the optimization is highly-model dependent as well.
the whole "cuda moat" is from the generic implementations of the moving parts of the model architecture. at the same time, you lose a lot of performance through the generic code. it is like comparing writing a stock trading algo in next.js vs assembly.
training models is another landscape altogether, so props to those who can quickly adapt to the hardware they got.
jonstewart•4mo ago
Accept!