Analyzing Modern Nvidia GPU Cores

https://arxiv.org/abs/2503.20481

178•mfiguiere•9mo ago

Comments

winwang•9mo ago

I hope this can help shed the misconception that GPUs are only good at linear algebra and FP arithmetic, which I've been hearing a whole lot!

Edit: learned a bunch, but the "uniform" registers and 64-bit (memory) performance are some easy standouts.

remcob•9mo ago

It’s well known GPUs are good at cryptography. Starting with hash functions (e.g. crypto mining) but also zero knowledge proofs and multi party computation.

saagarjha•9mo ago

They're not particularly good at cryptography, but they are good at highly parallel tasks like trying a bunch of hashes.

qwertox•9mo ago

Wasn't it well known that CUDA cores are programmable cores?

winwang•9mo ago

Haha, if you're the type to toss out the phrase "well known", then yes!

YetAnotherNick•9mo ago

In a sense, GPUs are only great at matrix-matrix multiplication. For anything else you would only get 7% of the FLOPs/s compared to it(989 vs 67 TFLOP/s for H100)[1].

[1]: https://www.nvidia.com/en-in/data-center/h100/

winwang•9mo ago

lol, I haven't thought about it like that, true. though of course, I mean compared to CPUs :P

I try and use tensor cores for non-obvious things every now and then. The most promising so far seems to be for linear arithmetic in Datalog, but that's just matrix-vector/gemv

harperlee•9mo ago

Could you expand the Datalog example? I'm quite interested

winwang•9mo ago

This was just a brief moment of thought over a year ago, but I can try to summarize. I was thinking about how to unify variables in certain simple Datalog settings. If we think of a clause as a vector of variables, then simple unifications can look like just a gather operation. A gather can be thought of as a matrix-vector multiplication, but that's not really useful (performance wise). But if those variables are also in a linear equation, then it becomes possibly-useful, e.g. for something like `P(x, y) :- E(x, 3x+4y)`

cma•9mo ago

That link says "* With sparsity". For extremely sparse matrixes you can get more than 989 TFLOPS on CPU, if we're counting elided operations in TFLOPS.

YetAnotherNick•9mo ago

I am counting FP16/BF16 without sparsity, which is used in majority of AI.

cma•9mo ago

That change checks out then. They didn't see much need for FP16 outside of that so no longer run it at double FP32 rate outside of tensor cores (unless I'm mixing that up with AMD).

Other forms of sparsity are heavily used at training time now, like block compression in Deepseek.

randomgermanguy•9mo ago

Funnily, they're far from being optimal for GEMM ops (especially in terms of power consumption).

For GEMM you need to visit each row/vec n-times so theres a bunch of data-reuse going on, which isn't optimal for GPUs since you can't keep that all so close to your processing-units. And while the tensor-cores kinda implement this i think they don't quite scale up to a full sized systolic array, which is you would want for larger matrix multiplications.

Also just a simpler view: with GPUs most of their silicon is spent NOT tensor-core, so just from that you know its not optimal i guess.

Just referring to that FLOP/s number doesn't really mean much nowadays with tensor-cores and sparsity.

In my eyes the big win of GPUs are that not only are they pretty good at GEMMs but also really good at a lot of other easily parallelizable tasks PLUS they're comparatively easy to program ^^

dist-epoch•9mo ago

From your comment I've learned that you never did GPU graphical programming :)

"uniform registers" exist for about 20 years now.

rnrn•9mo ago

I think you are confusing uniform registers with the uniform keyword in RSL / GLSL / HLSL?

maybe some vendors have had an equivalent to uniform registers for 20 years, but per the articles’ references they are new in nvidia GPUs in turing (2018)

dist-epoch•9mo ago

They are the same thing. The uniform keyword in shading languages is implemented using the uniform registers.

I don't know what Nvidia did in 2018, maybe they opened up access to the uniform registers to CUDA code.

I made Grok research this topic:

> In conclusion, research strongly suggests that the "uniform" keyword in GLSL is implemented in hardware using NVIDIA's "uniform registers," as evidenced by NVIDIA's own documentation on the Turing architecture and historical practices of mapping uniforms to constant registers. While explicit links can be limited due to proprietary details, the combination of technical presentations, community discussions, and historical context supports this connection. The uniform register file, with its capacity and usage in shader instructions, aligns with GLSL's uniform functionality, ensuring efficient data access during shader execution.

https://grok.com/share/c2hhcmQtMg%3D%3D_358362f3-21e2-4fe0-a...

Firadeoclus•9mo ago

While you can use uniform registers to implement the uniform keyword from shading languages, the two are not the same. Uniform registers are not constants, and they are only uniform/shared across one warp. Nvidia architectures before Turing did not have uniform registers.

pjc50•9mo ago

I didn't get that at all - to me this looks like a very smart investigation into instruction latency and the precise mechanics of out-of-order execution (no reference made to speculative or branch prediction, though?) without looking at what the instructions do in detail.

GPUs can certainly do bulk integer arithmetic but most use cases prefer FP. Maybe for DSP fixed-point is ideal.

gmays•9mo ago

The special sauce:

> "GPUs leverage hardware-compiler techniques where the compiler guides hardware during execution."

kookamamie•9mo ago

> NVIDIA RTX A6000

Unfortunately that's already behind the latest GPU by two generations. You'd have these after A6000: 6000 Ada, Pro 6000.

flowerthoughts•9mo ago

It's a major step forward compared to 2006.

A6000 was released in 2020: https://www.techpowerup.com/gpu-specs/rtx-a6000.c3686

KeplerBoy•9mo ago

Nvidia's Quadro naming scheme really is bad these days, isn't it?

I bet there are plenty of papers out there claiming to have used a RTX 6000 instead of a RTX 6000 Ada gen.

kookamamie•9mo ago

The naming scheme is horrible, to be quite frank.

To understand this, consider these names in the order of release time: Quadro RTX 6000, RTX A6000, RTX 6000 Ada, RTX Pro 6000, RTX Pro 6000 Max-Q.

pjmlp•9mo ago

Still better than most folks have access to.

I bet I can do more CUDA with my lame GeForce MX 150 from 2017, than what most people can reach for to do ROCm, and that is how NVidia keeps being ahead.

kookamamie•9mo ago

Yeah, kind of. I have an 6000 Ada and 5090 here.

pjmlp•9mo ago

On a laptop?

Because that is part of my point, that is a laptop GPU.

kookamamie•9mo ago

Oh no, there are high-end desktops. You're right, laptops completely different profiles for these things.

gitroom•9mo ago

Haha honestly I always thought GPUs were mostly number crunchers, but there's way more under the hood than I realized. Wondering now if anyone really gets the full potential of these cores, or if we're all just scratching the surface most days?

Dlemo•9mo ago

There are very good performance tools for GPUs.

I don't think GPU utilization is a real bottleneck in most cases.

dist-epoch•9mo ago

Yet DeepSeek managed to get huge improvement by optimizing GPU code.

nabla9•9mo ago

>Overall, we can conclude that GPUs are hardware-compiler codesign where the compiler guides the hardware in handling dependencies and introduces hints that can improve performance and energy.

New architectures rely on the compiler to handle register data dependencies, and controlling register file cache allocation policy.

dist-epoch•9mo ago

This is an age-old idea, RISC compilers were supposed to do this too, the mythical "sufficiently smart compiler"

https://wiki.c2.com/?SufficientlySmartCompiler

MindSpunk•9mo ago

It's not so much about having a "sufficiently smart compiler" in the case of GPUs doing compiler assisted scheduling. It's about not having to implement that logic in hardware at all. The more smarts they push into the core hardware, the more silicon each core needs, the less cores you can fit, and more power you spend on figuring out what to run rather than crunching numbers.

Doing the work in the compiler may produce less optimal scheduling than what is theoretically possible, but with the number of "cores" in a GPU you would spend a lot of power doing it in hardware for each one.

nabla9•9mo ago

RISC woks well with compilers (ARM, RISC-V), they don't require mythical compilers, just standard good ones.

You are probably thinking VLIW like Intels Itanium, and Transmeta. Those architectures required really smart compiler for scheduling and it was a bust.

Nvidia GPU's need smart compiler and it works because the task is limited to optimizing numerical pipelines that are 99% matrix multiplication, dot products. The data movement is more predicable. Compilers know how the data will be used and know how to schedule.

peterfirefly•9mo ago

MIPS used to not have interlocked pipeline stages. It was the compiler's job to work around that. MIPS -- and many other RISCs -- had a branch delay slot. It was the compiler's job to try to do something useful with that. RISCs stayed in-order for a long time -- it was the compiler's job to try do schedule the instructions in a way that compensated as well as possible for that.

GPUs rely on fairly smart compilers -- but they also hide latency (memory access) by switching hw threads (a bit like barrel processors of yore).

France's homegrown open source online office suite

British drivers over 70 to face eye tests every three years

Start all of your commands with a comma (2009)

Hoot: Scheme on WebAssembly

Stories from 25 Years of Software Development

Leisure Suit Larry's Al Lowe on model trains, funny deaths and Disney

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

First Proof

Reinforcement Learning from Human Feedback

The Waymo World Model

Coding agents have replaced every framework I used

Vocal Guide – belt sing without killing yourself

Software Factories and the Agentic Moment

72M Points of Interest

Unseen Footage of Atari Battlezone Arcade Cabinet Production

A Fresh Look at IBM 3270 Information Display System

StrongDM's AI team build serious software without even looking at the code

Ga68, a GNU Algol 68 Compiler

Making geo joins faster with H3 indexes

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: Kappal – CLI to Run Docker Compose YML on Kubernetes for Local Dev

Hackers (1995) Animated Experience

Sheldon Brown's Bicycle Technical Info

Show HN: I spent 4 years building a UI design tool with only the features I use

What Is Ruliology?

Show HN: If you lose your memory, how to regain access to your computer?

An Update on Heroku

Microsoft open-sources LiteBox, a security-focused library OS

Female Asian Elephant Calf Born at the Smithsonian National Zoo

France's homegrown open source online office suite

British drivers over 70 to face eye tests every three years

Start all of your commands with a comma (2009)

Hoot: Scheme on WebAssembly

Stories from 25 Years of Software Development

Leisure Suit Larry's Al Lowe on model trains, funny deaths and Disney

OpenCiv3: Open-source, cross-platform reimagining of Civilization III

First Proof

Reinforcement Learning from Human Feedback

The Waymo World Model

Coding agents have replaced every framework I used

Vocal Guide – belt sing without killing yourself

Software Factories and the Agentic Moment

72M Points of Interest

Unseen Footage of Atari Battlezone Arcade Cabinet Production

A Fresh Look at IBM 3270 Information Display System

StrongDM's AI team build serious software without even looking at the code

Ga68, a GNU Algol 68 Compiler

Making geo joins faster with H3 indexes

Show HN: Look Ma, No Linux: Shell, App Installer, Vi, Cc on ESP32-S3 / BreezyBox

Monty: A minimal, secure Python interpreter written in Rust for use by AI

Show HN: Kappal – CLI to Run Docker Compose YML on Kubernetes for Local Dev

Hackers (1995) Animated Experience

Sheldon Brown's Bicycle Technical Info

Show HN: I spent 4 years building a UI design tool with only the features I use

What Is Ruliology?

Show HN: If you lose your memory, how to regain access to your computer?

An Update on Heroku

Microsoft open-sources LiteBox, a security-focused library OS

Female Asian Elephant Calf Born at the Smithsonian National Zoo

Analyzing Modern Nvidia GPU Cores

Comments