(I have spent a good amount of time hacking the llvm pass pipeline for my personal project so if there was a significant difference I probably would have seen it by now)
I just glanced at the IR which was different for some attributes (nounwind vs mustprogress norecurse), but the resulting assembly is 100% identical for every optimization level.
BenchExec "uses the cgroups feature of the Linux kernel to correctly handle groups of processes and uses Linux user namespaces to create a container that restricts interference of [each program] with the benchmarking host."
A nitpick is that benchmarking C/C++ with $MARCH_FLAG -mtune=native and math magic is kinda unfair for Zig/Julia (Nim seem to support those) - unless you are running Gentoo it's unlikely to be used for real applications.
In my opinion, the comparisons could be better if the file I/O and console printing were removed.
Memory-mapped I/O can be great in some circumstances, but a one-time read of a small file is one of the canonical examples for when it isn't worth the hassle and setup/teardown overhead.
My Python version is a good example of the structure: read rounds.txt, run the loop, print the result, exit. I’m timing the whole program with hyperfine.
I agree that for a “pure compute” microbenchmark you could remove file I/O and console output. I kept them mainly because:
- It gives every language the same simple interface (same input, same output) and acts as a basic correctness/sanity check.
- The benchmark runs 1 billion iterations. The file read and a single print happen once per run, so that overhead is tiny compared to the loop, and the results stay comparable in practice.
That said, I’m not against a compute-only / quiet mode. Since hyperfine already handles timing externally, the real work is implementing and maintaining a consistent --quiet / --no-io variant across 50+ languages.
If someone wants to contribute that (even starting with a subset), I’m happy to review PRs.
- run bytecode - very high level - GC memory
But not all have these traits. Not sure.
The one exception is sort of an exception that proves the rule: it's marked "C# (SIMD)", and looks like a native compiler and not a managed one.
for (long i = 2; i <= rounds + 2; i++) {
x *= -1.0;
pi += x / (2.0 * i - 1.0);
}
With my older version of Clang, the resulting assembly at -O3 isn't vectorized. Now look at the C version in leibniz.c: rounds += 2u; // do this outside the loop
for (unsigned i=2u; i < rounds; ++i) // use ++i instead of i++
{
double x = -1.0 + 2.0 * (i & 0x1); // allows vectorization
pi += (x / (2u * i - 1u)); // double / unsigned = double
}
This produces vectorized code when I compile it. When I replace the Objective C loop with that code, the compiler also produces vectorized code.You see something similar in the other kings-of-speed languages. Zig? It's the C code ported directly to a different syntax. D? Exact same. Fortran 90? Slightly different, but still obviously written with compiler vectorization in mind.
(For what it's worth, the trunk version of Clang is able to auto-vectorize either version of the loop without help.)
Also, winners don’t make excuses.
(Not even being snarky. You have to spiritually accept that as a fact if you are in the PL perf game.)
I did the same sort of thing with the Seive of Eratosthenes once, on a smaller scale. My Haskell and Python implementations varied by almost a factor of 4 (although you could argue that I changed the algorithm too much on the fastest Python one). OK, yes, all the Haskell ones were faster than the fastest Python one, and the C one was another 4 times faster than the fastest Haskell one... but they were still over the place.
It's true this is a microbenchmark and not super informative about "Big Problems" (because nothing is). But it absolutely shows up code generation and interpretation performance in an interesting way.
Note in particular the huge delta between rust 1.92 and nightly. I'm gonna guess that's down to the autovectorizer having a hole that the implementation slipped through, and they fixed it.
The benchmark also includes startup time, file I/O, and console printing. There could have been a one-time startup cost somewhere that got removed.
The benchmark is not really testing the Leibniz loop performance for the very fast languages, it's testing startup, I/O, console printing, etc.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
How significant are the differences for such tiny tiny programs?
https://github.com/niklas-heer/speed-comparison/blob/master/...
https://github.com/niklas-heer/speed-comparison/blob/master/...
Suppose I remove the strictness annotations (3 exclamation points, in places that aren't obvious to a naive programmer coming from almost any other language). If I then compile it unoptimized, it gets up to over 30GB of resident memory before I get bored (it's an old machine with a lot of memory). It would probably die with an out of memory error if I tried to run it to completion. However, if I compile that same modified code optimized, the compiler infers the strictness and the program runs in exactly the same time as it does with the annotations there. BUT it's far from obvious to the casual observer when the compiler can make those inferences and when it can't.
I had ChatGPT rewrite the Haskell code to use unboxed numeric types. It ran in 1.5 seconds (the C version takes 1.27). The rewrite mostly consists of sprinkling "#" liberally throughout the code, but also requires using a few specialized functions. I had ChatGPT do it because I have never used unboxed types, and you could argue that they're not common idiom. However, anybody who actually wrote that kind of numerical code in Haskell on a regular basis would use unboxed types as a matter of course.
So which one is the right time?
You're acting like this is a gotcha, but the answer is obviously "all of them" and that indeed, this tells you interesting things about the behavior of your compiler. There are lots of variant scores in the linked article that reflect different ways of expressing the problem.
But also, it tells you something about the limitations of your language too. For example, the biggest single reason that C/C++ (and languages like Fortran/Zig/D and sometimes C# and Rust whose code generation is isomorphic to them) sit at the top of the list is that they autovectorize. SIMD code isn't a common idiom either, but the compiler figures it out anyway.
And apparently Haskell isn't capable of doing enough static analysis to fall back to an unboxed implementation (though given the simplicity of this code, that should be possible I think). That's not a "flaw" and it doesn't mean Haskell "loses", but it absolutely is an important thing to note. And charts like this show us where those holes lie.
They tell me interesting things if I know enough about the language to know the difference. It tells me things if I'm getting into the weeds with Haskell specifically. That doesn't make the big comparison chart useful in any way.
I still don't know anything that lets me compare anything with any other language unless I actually know that language nearly as well. And I definitely don't get much out of a long list of languages, most of which I know not at all or at most at a "hello world" level, with only a couple of the entries tagged with even minimal information about compilers or their configurations at all. Especially when, on top of that, I don't know how much the person writing the test code knew.
At most I get "this language does a pretty good/poor job on this type of task when given code that may or may not be what a 'native expert' would write.".
And that's not news. Nobody (with any sophistication) would write that code for real in Python, or probably in Haskell either, because most seasoned programmers know that if you want speed on a task like that, you write it in a more traditional compiled procedural language. It's also not a kind of code that most people write to begin with. If you want an arctangent (which is really what it's doing), you use the library function, and the underlying implementation of that is either handcrafted C, or, more likely, a single, crafted CPU instruction with some call handling code wrapped around it.
So what is the overall chart giving me that I can use?
"If you write in C or an analog, your math will autovectorize nicely"
"If you use a runtime with a managed heap, you're likely to take a penalty even on math stuff that doesn't look heap limited"
"rust 1.92 is, surprisingly, well behind clang on autovectorizable code"
I mean, I think that stuff is interesting.
> If you want an arctangent (which is really what it's doing), you use the library function
If you just want to call library functions, you're 100% at the mercy of whatever platform you picked[1]. So sure, don't look at benchmarks, they can only prove you wrong.
[1] Picked without, I'll note, having looked carefully at cross-platform benchmarks before having made the choice! Because you inexplicably don't think they tell you anything useful.
ᐅ time uv run -p cpython-3.14 leibniz.py
3.1415926525880504
________________________________________________________
Executed in 38.24 secs fish external
usr time 37.91 secs 158.00 micros 37.91 secs
sys time 0.16 secs 724.00 micros 0.16 secs
ᐅ time uv run -p pypy leibniz.py
3.1415926525880504
________________________________________________________
Executed in 1.52 secs fish external
usr time 1.16 secs 0.25 millis 1.16 secs
sys time 0.02 secs 1.29 millis 0.02 secs
It was a free 25x speedup.But this is a good benchmark results that demonstrate what performance level can you expect from every language when someone not versed in it does the code porting. Fair play
for i in 2...rounds+2
and I would hope/expect the compiler to be smart enough to know that it only has to check “rounds+2” once there. Swift isn’t exactly new anymore, and it’s supported by a large company.What do I overlook?
Makes the benchmarks game 100 lines seem like major apps.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
- Swift (standard): 893ms
- Swift (relaxed): 903ms (uses fast-math equivalent)
- Swift (SIMD): 509ms (explicit SIMD4)
The standard version uses x *= -1.0 which creates a loop-carried dependency that blocks auto-vectorization - same issue as Crystal, Odin, Ada. The SIMD version uses the branchless i & 0x1 trick and is ~1.75x faster.Fair point that someone versed in Swift would probably use the better pattern in the standard version too. PRs welcome to improve it! The goal was idiomatic-ish code, but I'm not an expert in all 40+ languages.
What do you think they could have done better assuming that the IO is a necessary part of the benchmark?
Also good job to the Rust devs for making the benchmark so much faster in nightly. I wonder what they did.
The differences among the really fast languages are probably in different startup times if I had to guess.
Startup times matter a great deal.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
So make it significant, move giga like benchmarks game reverse-complement & fasta & …
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
When you put these programs into Godbolt to see what's going on with them, so much of the code is just the I/O part that it's annoying to analyze
> Why do you also count reading a file and printing the output?
> Because I think this is a more realistic scenario to compare speeds.
Which is fine, but should be noted more prominently. The startup time and console printing obviously aren't relevant for something like the Python run, but at the top of the chart where runs are a fraction of a second it probably accounts for a lot of the differences.
Running the inner loop 100 times over would have made the other effects negligible. As written, trying to measure millisecond differences between entire programs isn't really useful unless someone has a highly specific use case where they're re-running a program for fractions of a second instead of using a long-running process.
Swift: 3.7
Python: that's incorrect!
Swift: yeah, but it's fast!
There is very little superfluous or that cannot be inferred by the compiler here: https://github.com/niklas-heer/speed-comparison/blob/master/...
Nim 1672
Julia 3012
D 3479
C# (SIMD) 5853
C# 8919
>Nim version is not some naive versionIt's direct translation of formula, using `mod` rather `x = -x`.
*Rather comparing numbers << 1. **No blank/comment lines. As cloc and similar tools count.
Although arguably these flags are more reasonable than allowing the use of -march=native.
Also consider the inherent advantage popular languages have: you don't need to break out to a completely niche language, while achieving high performance. Saying this, this microbenchmark is naive and does not showcase realistic bottlenecks applications would face like how well-optimized standard library and popular frameworks are, whether the compiler deals with complexity and abstractions well, whether there are issues with multi-threaded scaling, etc etc. You can tell this by performance of dynamically typed languages - since all data is defined in scope of a single function, the compiler needs to do very little work and can hide the true cost of using something like Lua (LuaJIT).
Agree with the rest of your comment.
I don't see these flags in Nim compilation config. The only extra option used is "-march=native"[0].
[0] https://github.com/niklas-heer/speed-comparison/blob/9681e8e...
public static void main(String[] args) throws FileNotFoundException {
Scanner s = new Scanner(new File("rounds.txt"));
long rounds = s.nextLong();
s.close();
double sum = 0.0;
double flip = -1.0;
for (long i = 1; i <= rounds; i++) {
flip *= -1.0;
sum += flip / (2 * i - 1);
}
System.out.println(sum * 4.0);
}
:the measurements changed dramatically if the order was switched something like: sum += flip / (2 * i - 1);
flip *= -1.0;
YMMV # Language Accuracy
14 Swift (SIMD) 8.69
9 Fortran 90 9.44
2 C# (SIMD) 9.49
. [All the others] 9.50 // [I removed the error checking]
def pi_accuracy(value_str):
"""Calculate accuracy of computed pi value.
Returns the number of correct decimal places (higher is better).
"""
value = float(value_str)
# math.pi is available in MicroPython
accuracy = 1 - (value / math.pi)
return -math.log10(abs(accuracy))I’m genuinely blown away by all the interest in what started as a silly little experiment. The project grew way beyond its original scope. My initial curiosity was simply: how could you set up a pipeline to do automatic speed comparisons? I was less interested in the results as a definitive measure and more in the infrastructure challenge itself.
But then the interest kept growing. I tried to modernize things, but one thing became quite notable: the difference between a language that gets actively optimized by its community (like Julia) versus one that just sits there unoptimized is striking.
Honestly, I got overwhelmed. Managing all those implementations, keeping versions up to date, reviewing contributions—it was a lot. I basically tapped out for about a year.
Now I’m back, and with AI assistance, maintaining this has become much more realistic—updating versions, helping optimize implementations, etc. That said, I’m always happy to accept contributions from folks who know their languages better than I do.
Thank you all for your interest and the thoughtful discussion!
This I like!
> … actively optimized … versus one that just sits there unoptimized is striking.
See N=50,000,000 nbody #1 #2 #3 #4 #5 jdk-23
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
> a silly little experiment
This I don't like.
> > a silly little experiment
> This I don't like.
What do you mean with that?
To explain what I meant. I knew that that in the end this was a microbenchmark. It can certainly give you some clue about a language, but it doesn't tell you the whole picture. In the end it tells you how good a language is (or can be) at loops and floating point math. That's what I meant. I hope that makes it clearer.
Take kostya or hanabi1224 or attractivechaos or … as your starting point and do better.
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
That said, I'm not sure what "do better" means here. This is an open source project I maintain in my spare time. If you see room for improvement, PRs are always welcome. That's how open source works - if something is useful to you and you want it improved, contribute.
The project exists because some people find it useful. If it's not for you, that's fine too.
forgotpwd16•1mo ago
- C++ unsurpassable king.
- There's a stark jump of times going from ~200ms to ~900ms. (Rust v1.92.0 being an in-between outlier.)
- C# gets massive boost (990->225ms) when using SIMD.
- But C++ somehow gets slower when using SIMD.
- Zig very fast*!
- Rust got big boost (630ms->230ms) upgrading v1.92.0->1.94.0.
- Nim (that compiles to C then native via GCC) somehow faster than GCC-compiled C.
- Julia keeps proving high-level languages can be fast too**.
- Swift gets faster when using SIMD but loses much accuracy.
- Go fastest language with own compiler (ie not dependent to GCC/LLVM).
- V (also compiles to C) expected it (appearing similar) be close to Nim.
- Odin (LLVM) & Ada (GCC) surprisingly slow. (Was expecting them to be close to Zig/Fortran.)
- Crystal slowest LLVM-based language.
- Pure CPython unsurpassable turtle.
Curious how D's reference compiler (DMD) compares to the LLVM/GCC front-ends, how LFortran to gfortran, and QBE to GCC/LLVM. Also would like to see Scala Native (Scala currently being inside the 900~1000ms bunch).
* Note that uses `@setFloatMode(.Optimized)` which according to docs is equivalent to `--fast-math` but only D/Fortran use this flag (C/C++ do not).
** Uses `@fastmath` AND `@simd`. The comparison supposedly is for performance on idiomatic code and for Julia SIMD is a simple annotation applied to the loop (and Julia may even auto do it) but should still be noted because (as seen in C# example) it can be big.
mrsmrtss•1mo ago
mrsmrtss•1mo ago
neonsunset•1mo ago
On M4 Max, Go takes 0.982s to run while C# (non-SIMD) and F# are ~0.51s. Changing it to be closer to Go makes the performance worse in a similar manner.
neonsunset•1mo ago
C# is using CoreCLR/NativeAOT. Which does not use GCC or LLVM also. Its compiler is more capable than that of Go.
Aurornis•1mo ago
For the sub-second compiled languages, it's basically a benchmark of startup times, not performance in the hot loop.
igouy•1mo ago
https://benchmarksgame-team.pages.debian.net/benchmarksgame/...
nheer•1mo ago
The benchmark is definitely measuring the hot loop, not startup time.
nheer•1mo ago
- C++ SIMD being slower: The standard C++ uses i & 0x1 which lets the compiler auto-vectorize. With -O3 -ffast-math -march=native, gcc/clang do this really well. The explicit AVX2 version has overhead from manual vector setup and horizontal sum at the end. Modern compilers often beat hand-written SIMD for simple loops like this.
- Zig fast-math: Correct. Line 5 has @setFloatMode(.optimized) with a comment saying "like C -ffast-math".
- Julia: Also correct. Uses @fastmath @simd for - both annotations together.
- Crystal/Odin/Ada being slow: All three use x = -x which creates a loop-carried dependency that blocks auto-vectorization. The fast implementations use the branchless i & 0x1 trick instead.
- C# SIMD: Uses Vector512 doing 8 doubles per iteration. That explains the ~4x speedup.
- Nim vs C: Both compile via gcc with similar flags. Probably just measurement variance.
- Fortran: Interestingly does NOT use -ffast-math. Uses manual loop unrolling instead (processes 4 terms per iteration).
- Go: You're right that it's the fastest with its own compiler. No LLVM/GCC backend, just Go's own SSA-based compiler.
For suggestions - DMD, LFortran, and Scala Native would be great additions. PRs welcome!
vips7L•1mo ago
On my machine (an old i7-8700), dmd performs rather poorly. 3.5 seconds.
Comparatively ldc runs in 943 milliseconds: I'm sure there is compiler switch magic that I don't know about that could improve them.