I try and use tensor cores for non-obvious things every now and then. The most promising so far seems to be for linear arithmetic in Datalog, but that's just matrix-vector/gemv

For GEMM you need to visit each row/vec n-times so theres a bunch of data-reuse going on, which isn't optimal for GPUs since you can't keep that all so close to your processing-units. And while the tensor-cores kinda implement this i think they don't quite scale up to a full sized systolic array, which is you would want for larger matrix multiplications.

Also just a simpler view: with GPUs most of their silicon is spent NOT tensor-core, so just from that you know its not optimal i guess.

Just referring to that FLOP/s number doesn't really mean much nowadays with tensor-cores and sparsity.

In my eyes the big win of GPUs are that not only are they pretty good at GEMMs but also really good at a lot of other easily parallelizable tasks PLUS they're comparatively easy to program ^^

maybe some vendors have had an equivalent to uniform registers for 20 years, but per the articles’ references they are new in nvidia GPUs in turing (2018)

To understand this, consider these names in the order of release time: Quadro RTX 6000, RTX A6000, RTX 6000 Ada, RTX Pro 6000, RTX Pro 6000 Max-Q.

Doing the work in the compiler may produce less optimal scheduling than what is theoretically possible, but with the number of "cores" in a GPU you would spend a lot of power doing it in hardware for each one.

You are probably thinking VLIW like Intels Itanium, and Transmeta. Those architectures required really smart compiler for scheduling and it was a bust.

Nvidia GPU's need smart compiler and it works because the task is limited to optimizing numerical pipelines that are 99% matrix multiplication, dot products. The data movement is more predicable. Compilers know how the data will be used and know how to schedule.

There is so much stuff you miss when you don't follow the links ;)