To me, this looks like a win.
Governments are there to finance projects like this that enable the country to have certain skillsets that wouldn't exist otherwise because of other countries having better solutions in the global market.
But what governments often can do, is break local optimums clustering around the quarter economy and take moonshot chances and find paths otherwise never taken. Hopefully one of these paths are great.
The difficult thing becomes deciding when to pull the plug. Is ITER a good thing or not? (Results wise, it is, but for the money? Who can tell really.)
Well, at least VCs don't burn through my tax money, whilst they are failing at picking winners.
See how IBM was on the path to inventing electronic computers for business and accounting, but got pre-empted by the militaries' needs and procurement.
In the counterfactual if there hadn't been a war going on, presumably businesses would have had even more needs for international business machines.
Military/government procurement provided the demand for which private entities could provide, money and other resources at level that private market never managed.
Without large injection of money supply there's not much extra dollar to chase and safe investments sound like better bets.
This provided demand for products and services that simply was not there in private form, allowing way riskier investments - all the way to 1980s Silicon Valley was mostly riding downstream of military procurement (Can't source the quote right now, but Sun microsystems be-or-not-be was landing a contract for NSA for Unix workstations - because NSA could frontload an order for few hundred if not thousands workstations)
Just like it doesn’t work to try an ecosystem based on one species, a society has to blend government and private spending. They work on different incentives and timeframes, and both have pitfalls that the other might handle better.
The fp64 GFLOPS per watt metric in the post is almost entirely meaningless to compare between these accelerators and NVIDIA GPUs, for example it says
> Hopper H200 is 47.9 gigaflops per watt at FP64 (33.5 teraflops divided by 700 watts)
But then if you consider H100 PCIe [0] instead, it's going to be 26000/350 = 74.29 GFLOPS per watt. If you go look harder you can find ones with better on-paper fp64 performance, for example AMD MI300X has 81.7 TFLOPs with typical board power of "750W Peak", which gives 108.9 GFLOPS per watt.
The truth is the power allocation of most GPGPUs are heavily tilted for Tensor usages. This has been the trend well before B300.
That's all for HPC.
And Pezy processors are certainly not designed for "AI" (i.e. linear algebra with lower input precision). For AI inference starting from 2020 everyone is talking about how many T(FL)OPS per watt, not G.
[0] which is a nerfed version of H200's precursor.
These Pezy chips are also made for large clusters. There is a whole system design around the chips that wasn't presented here. The Pezy-SC2, for instance, was built around liquid immersion cooling. I am not sure you could ever buy an air-cooled version.
Is the whole board submersed in liquid? Or just the processor?
"Each immersion tank can contain 16 Bricks. A Brick consists of a backplane board, 32 PEZY-SC2 modules, 4 Intel Xeon D host processors, and 4 InfiniBand EDR cards. Modules inside a Brick are connected by hierarchical PCI Express fabric switches, and the Bricks are interconnected by InfiniBand."
Well that was a disappointing end to a sentence. I was hoping another company would invest a few million in HPC to play SC2!
GPUs are great if your workload can use them, but not so great for more general tasks. These are more appropriate to more traditional supercomputing tasks, as in they're not optimized for lower precision AI stuff, like NVIDIA GPUs are.
> The Hopper H200 is 47.9 gigaflops per watt at FP64 (33.5 teraflops divided by 700 watts), and the Blackwell B200 is rated at 33.3 gigaflops per watt (40 teraflops divided by 1,200 watts). The Blackwell B300 has FP64 severely deprecated at 1.25 teraflops and burns 1,400 watts, which is 0.89 gigaflops per watt. (The B300 is really aimed at low precision AI inference.)
Nevertheless, my point is more that if FP64 performance is poor on purpose, then you're probably not using anywhere near the card's TDP to do FP64 calculations, so FLOPS/watt(TDP) is misleading.
Regardless, there's "tricks" you can use to sort of extend the precision of hardware floating point - using a pair of e.g. FP32 numbers to implement something that's "almost" a FP64. Well known among numerics practitioners.
[1] https://youtu.be/vzVlQhaAZtQ?si=DJRmwOoyYGdq6mUQ [2] https://calligotech.com/uttunga/
They have massive government investment in not only maintaining that status, but also doing so on a completely domestic supply chain as much as possible.
Therefore they have the same need for supercomputers that the US national labs do (perhaps more so, since they're even more reliant on simulation), and heavily prefer locally sourced pieces of that critical infrastructure.
I wouldn't be surprised if an incredibly large part of the local push for Rapidus is to pull them off of TSMC and the supply chain risk for their nuclear program in case the whole China/Taiwan thing comes to a head.
Do you have a citation for "weeks away"? Wikipedia only says "within one year": https://en.wikipedia.org/wiki/Japanese_nuclear_weapons_progr...
But: if you consider the amount of nuclear generating capacity has (4th in the world, more than Russia), and its advanced space program, “within one year” probably means closer to “weeks or months” than “three hundred and sixty four days”.
numpad0•5mo ago
WithinReason•5mo ago
saagarjha•5mo ago
WithinReason•5mo ago
saagarjha•5mo ago
jacquesm•5mo ago
I really can't complain, now, FPGAs, however... And if there ever is a company that comes out and improves substantially on this I'll be happy for sure but if you asked me off the bat what they should improve I honestly wouldn't know, especially not taking into account that this was an incremental effort over ~2 decades and that originated in an industry that has nothing to do with the main use case today and some detours into unrelated industries besides (crypto, for instance).
From fluid dynamics, FEA, crypto, gaming, genetics, AI and many others with a single generic architecture and delivering very good performance is no mean feat.
I'd love to hear in what way you would improve on their toolset.
programjames•5mo ago
1. Memory indexing. It's a pain to avoid banking conflicts, and implement cooperative loading on transposed matrices. To improve this, (1) pop up a warning when banking conflicts are detected, (2) make cooperative loading solved by the compiler. It wouldn't be too hard to have a second form of indexing memory_{idx} that the compiler solves a linear programming problem for to maximize throughput (do you spend more thread cycles cooperative loading, or are banking conflicts fine because you have other things to work on?)
2. Why is there no warning when shared memory is unspecified? It isn't hard to check if you're accessing an index that might not have been assigned a value. The compiler should pop out a warning and assign it to 0.0, or maybe even just throw an error.
3. Timing - doesn't exist. Pretty much the gold standard is to run your kernel 10_000 times in a loop and subtract the time from before and after the loop. This isn't terribly important, I'm just getting flashbacks to before I learned `timeit` was a thing in Python.
jacquesm•5mo ago
https://forums.developer.nvidia.com/c/accelerated-computing/...
They regularly have threads asking for such suggestions.
But I don't think they rise to the general conclusion that the tooling is bad.
numpad0•5mo ago
There is a type of research called traffic surveys, which involves hiring few men with adequate education to sit or stand at an intersection for one whole day to count numbers of passing entities by types. YOLO wasn't accurate enough. I have gut feeling that vision enabled LLM would be. That doesn't require constant update or upgrades to latest NN innovations so no need to do full CUDA, so long one known good weight files work.
CoastalCoder•5mo ago
I've only done a little work on CUDA, but I was pretty impressed with it and with their NSys tools.
I'm curious what you wish was different.
saagarjha•5mo ago
Of course, one might mention that GPUs are nothing like CPUs–but the programming model works super hard to try to hide this. So it's not really well designed in my book. I actually quite like the compilers that people are designing these days to write block-level code, because I feel like it better represents the work people want to do and then you pick which way you want it lowered.
As for Nsight (Systems), it is…ok, I guess? It's fine for games and stuff I guess but for HPC or AI it doesn't really surface the information that you would want. People who are running their GPUs really hard know they have kernels running all the time and what the performance characteristics of them are. Nsight Compute is the thing that tells you that but it's kind of a mediocre profiler (some of this may be limitations of hardware performance counters) and to use it effectively you basically have to read a bunch of blog posts by people instead of official documentation.
Despite not having used it much, my impression was that Nvidia's "moat" was that they have good networking libraries, that they are pretty good (relatively) and making sure all their tools work, and they have had consistent investment on this for a decade.
electroglyph•5mo ago
jandrewrogers•5mo ago
The wide vectors on GPUs are somewhat irrelevant. Scalar barrel processors exist and have the same issues. A scalar barrel processor feels deceptively CPU-like and will happily compile and run normal CPU code. The performance will nonetheless be poor unless the C++ code is designed to be a good fit for the nature of a barrel processor, code which will look weird and non-idiomatic to someone who has only written code for CPUs.
There is no way to hide that a barrel processor is not a CPU even though they superficially have a lot of CPU-like properties. A barrel processor is extremely efficient once you learn to write code for them and exceptionally well-suited to HPC since they are not latency-sensitive. However, most people never learn how to write proper code for barrel processors.
Ironically, barrel processor style code architecture is easy to translate into highly optimized CPU code, just not the reverse.
glitchc•5mo ago
[1] https://docs.nvidia.com/deeplearning/performance/dl-performa...
jandrewrogers•5mo ago
Modern barrel processors implementations have complex microarchitectures that are much closer to a modern GPU in design. That is not accidental, the lineage is clearly there if you've worked on both. I will grant that vanishingly few people have ever seen or worked on a modern non-GPU barrel processor, since they are almost exclusively the domain of exotics built for government applications AFAICT.
kragen•5mo ago
zozbot234•5mo ago
adrian_b•5mo ago
When threads are implemented only in software, without hardware support, you have what is called coarse-grained multithreading. In this case, a CPU core executes one thread, until that thread must wait for a long time, e.g. for the completion of some I/O operation. Then the operating system switches the context from the stalled thread to another thread that is ready to run, by saving all registers used by the old thread and restoring the registers of the new thread, from the values that were saved when the new thread has been executed last time.
Such multithreading is coarse-grained, because saving and restoring the registers is expensive so it cannot be done often.
When hardware assists context-switching, by being able to store internally in the CPU core multiple sets of registers, i.e. multiple thread contexts, then you can have FGMT (fine-grained multithreading). In the earliest CPUs with FGMT the switching of the thread contexts was done after each executed instruction, but in all more recent CPUs or GPUs with FGMT the context switching can be done after each clock cycle.
Barrel processors are a subset of the FGMT processors, the simplest and the least efficient of them. Barrel processors are now only of historical interest. Nobody has made barrel processors during the last decades. In barrel processors, the threads are switched in round robin, i.e. in a fixed order. You cannot choose the next thread to run. This wastes clock cycles, because the next thread in the fixed order may be stalled, waiting for some event, so nothing can be done during its allocated clock cycle.
The name "barrel", introduced by CDC 6600 in 1964, refers to the similarity with the barrel of a revolver, you can rotate it with a position, bringing the next thread for execution, but you cannot jump over a thread to reach some arbitrary position.
What is switched in a barrel CPU at each clock cycle between threads is not a context, i.e. not the registers, but the execution units of the CPU, which become attached to the context of the current thread, i.e. to its registers. For each thread there is a distinct set of registers, storing the thread context.
The descriptions of the internal architecture of GPUs are extremely confusing, because NVIDIA has chosen to replace in its documentation all the words that have been used for decades when describing CPUs with different words, with no apparent reason except of obfuscating the GPU architecture. AMD has followed NVIDIA, and they have created a third set of architectural terms, mapped one to one to those of NVIDIA, but using yet other words, for maximum confusion.
For instance, NVIDIA calls "warp" what in a CPU is called "thread". What NVIDIA calls "thread" is what in a CPU is called "vector lane" or "SIMD lane". What NVIDIA calls "stream multiprocessor" is what in a CPU is called "core".
Both GPUs and CPUs are made of multiple cores, which can execute programs in parallel.
Each core can execute multiple threads, which share the same execution units. For executing multiple threads, most if not all GPUs use FGMT, while most modern CPUs use SMT (Simultaneous Multithreading).
Unlike FGMT, SMT can exist only on superscalar processors, i.e. which can initiate the execution of multiple instructions in the same clock cycle. Only in that case it may also be possible to initiate the execution of instructions from distinct threads in the same clock cycle.
Some GPUs may be able to initiate 2 instructions per clock cycle, only when certain conditions are met, but for all such GPUs their descriptions are typically very vague and it may be impossible to determine whether those 2 instructions may come from different threads, i.e. from different warps in the NVIDIA terminology.
KeplerBoy•5mo ago
rwmj•5mo ago
DrNosferatu•5mo ago
Seriously doubt that: free hardware (or 10s of bucks) would galvanize the community and achieve huge support - look at the Raspberry Pi project original prices and the consequences.
DrNosferatu•5mo ago
Say, release has extensions to a RISC-V design.
DrNosferatu•5mo ago
londons_explore•5mo ago
I suspect that whenever you look like you're making good progress on this front, nvidia gives you a lot of chips for free on condition you shelve the effort though!
The latest example being Tesla, who were designing their own hardware and software stack for NN training, then suspiciously got huge numbers of H100's ahead of other clients and cancelled the dojo effort.
AlotOfReading•5mo ago
To combat all of these issues, they were fighting with Nvidia (and losing) for access to leading edge nodes, which kept going up in price. Their personnel costs kept rising as the company became more politicized, people left to join other companies (e.g. densityai), and they became embroiled in the salary wars to replace them.
My suspicion is that Musk told them to just buy Nvidia instead of waiting around for years of slow iteration to get something competitive.
The custom silicon I was involved with experienced similar issues. It was too expensive and slow to try competing with Nvidia, and no one could stomach the costs to do so.
nromiun•5mo ago
If people use PyTorch on a Nvidia GPU they are running layers and layers of code written by those that know how to write fast kernels for GPUs. In some cases they use assembly as well.
Nvidia stuck to one stack and wrote all their high level libraries on it, while their competitors switched from old APIs to new ones and never made anything close to CUDA.
woooooo•5mo ago
CUDA and CUBLAS being capable of a bunch of other things is really cool, and would take a long time to catch up with, but getting the bare minimum to run LLMs on any platform with a bunch of GDDR7 channels and cores at a reasonable price would have people writing torch/ggml backends within weeks.
nromiun•5mo ago
Here is an example of how hard it is: https://siboehm.com/articles/22/CUDA-MMM
And this is just basic matrix mult. If you add activation functions it will slow down even more. There is nothing easy about GPU programming, if you care about performance. CUDA gives you all that optimization on a plate.
woooooo•5mo ago
I'm saying the API surface of what to offer for LLMs is pretty small. Yeah, optimizing it is hard but it's "one really smart person works for a few weeks" hard, and most of the tiling techniques are public. Speaking of which, thanks for that blog post, off to read it now.
kadushka•5mo ago
AMD should hire that one really smart person.
adgjlsfhk1•5mo ago
hedgehog•5mo ago
astrange•5mo ago
pclmulqdq•5mo ago
numpad0•5mo ago
Another one of these I still sometimes think about is NEC VectorEngine - they had 5 TFLOPS FP32 with 48GB of HBM2 totaling 1.5TB/s bandwidth at $10k in 2020. That was within a digit or two against NVIDIA at basically the same price. But they then didn't capitalize on it, just kept delivering to national institutes in ritualistic manners.
I do have basic conceptual understanding of these grant businesses and have vague intuitions as to how bureaucracy wants substantial capital investments and report files without commercial capitalizations, with emphasis on the last part, as it would disrupt internal politics inside government agencies and also creates unfair government competitive pressure against civilian sectors, but at some point it starts looking like cash campfires. I don't know exactly how slow are M4 Mac Studios relative to NVIDIA Tesla clusters normalized for VRAM, but they're considered comparable regardless just because they run LLMs at 10-20 tok/s. So it's just, unfortunate, that these accelerators of basically same nature as M-series CPUs are built, kept on idle, and then recycled.
The one that is in my mind as "no way these brochure figures are real" is PFN MN-Core - though it looks like they might be doing an LLM specific variant in the future. Hopefully they retail them.
tho23i23434234•5mo ago
It's just too inward looking these days - probably why technical innovations in Japan don't get shaped to meet the worlds needs, but gets sold as if it were a luxury artisan-product (ala "swiss-made" stuff).
The trouble with people who criticize Japan (incl. Japanese) is that they think this is because of "old people & culture" - but actually, no, the "old" Japanese (in the 1900s-1980s) seemed to have been extraordinarily curious about the world, and also very clever in marketing things. The issue is most definitely "modern", but ofc. saying that is verboten in the dogma of liberalism.
numpad0•5mo ago
Thinking about Japanese economy in general and how it hadn't grown in 30-40 years: 40 years is technically two generations, but life in Japan hadn't deteriorated meaningfully during that period. Substantial socio-political improvements were made, university entrance had rose somewhat absurdly high, some new infrastructures were built, convenience store sandwich prices hasn't doubled, overtimes and harassments at workplaces are way more strictly scrutinized. There's the problem of employment ice age, but it's not as bad as the collapse of the Soviet Union; not at "Vladimir Putin was laid off KGB and drove taxi to make ends meet" levels, only "PhDs drove trucks". So "Japan still using FAX" narratives only partially make sense. Overall, it does feel that there is something strange is going on in this country, something like effective isolationism.
I guess my point is... it's unfortunate that these efforts and costs spent goes to waste, and we don't know why it's only built to be recycled.
pezezin•5mo ago
I have been living in Japan for the past 7 years, and in my experience all generations are guilty.
I work for a big international research project, so I have met many old Japanese professors and high-level researchers. They all lament how their generation wanted to go abroad, see the world, and change things, whereas their new graduates don't even want to learn English, just stay in their little Japanese bubble and do what they are told to do. But for every one of those outgoing old Japanese people, you meet 10 who are dead set in their ways and don't want any change, and with the population getting older, the country has become stagnant.
aredox•5mo ago
Maxious•4mo ago