Someone mentioned recently that the slowness of rustc is in large part due to llvm. I know that is probably orthogonal to the work here, but I do like the idea of building the compiler with different toolchains, and that there may be follow on effects down the line.
Edit: found it
There is definitely Rust code that takes exponential time to compile, borrow checker or not.
https://play.rust-lang.org/?version=stable&mode=release&edit...
https://github.com/rust-lang/rust/issues/75992
Some people used async in ways that surfaced these problems. Upgraded rustc, then the project took forever to compile.
- Parsing: x ms
- Type checking: y ms
- LLVM IR generation: z ms
And have there been any statistics done on that across open-source projects, like mean, median, percentiles and so on?
I am asking because it should depend a lot on each project what is costly in compile time, making it more difficult to analyse. And I am also curious about how many projects are covered by "edge cases", if it is 1%, 0.1%, 0.01%, and so on.
> And have there been any statistics done on that across open-source projects, like mean, median, percentiles and so on?
I am not aware of any. But in all the posts on this topic over the years, codegen always ends up being half the time. It’s why cargo check is built the way it is, and why it’s always faster than a full build. If non-codegen factors were significant with any regularity, you’d be seeing reports of check being super slow compared to build.
https://www.reddit.com/r/rust/comments/1daip72/rust_checkrun...
May not be indicative, not sure what crate the author was using.
This may be a better example.
https://github.com/rust-lang/rust/issues/132064
>cargo check with 1.82: 6m 19s
>cargo check with 1.81: 1m 22s
It may be difficult to fix.
https://github.com/rust-lang/rust/issues/132064#issuecomment...
>Triage notes (AFAIUI): #132625 is merged, but the compile time is not fully clawed back as #132625 is a compromise between (full) soundness and performance in favor of a full revert (full revert would bring back more soundness problems AFAICT)
Update: They fixed it, but another issue surfaced, possibly related, as I read the comments.
I could imagine you being correct about the borrow checking typability problem being NP-complete. Or an even worse complexity class. Typability in ML is EXPTIME-complete, a larger set than NP-complete https://en.wikipedia.org/wiki/EXPTIME https://dl.acm.org/doi/10.1145/96709.96748 .
I also am not sure how to figure out if the complexity class of some kind of borrow checking has something to do with the exponential compile times of some practical Rust projects after they upgraded compiler version, for instance in https://github.com/rust-lang/rust/issues/75992 .
It would be good if there was a formal description of at least one borrow checking algorithm as well as the borrow checking "problem", and maybe also analysis of the complexity class of the problem.
The upcoming Polonius borrow checking algorithm was prototyped using Datalog, which is a logical programming language. So the source code of the prototype [1] effectively is a formal definition. However, I don't think that the version which is in the compiler now exactly matches this early prototype.
EDIT: to be clear, there is a polonius implementation in the rust compiler, but you need to use '-Zpolonius=next' flag on a nightly rust compiler to access it.
I read something curious.
https://users.rust-lang.org/t/polonius-is-more-ergonomic-tha...
>I recommend watching the video @nerditation linked. I believe Amanda mentioned somewhere that Polonius is 5000x slower than the existing borrow-checker; IIRC the plan isn't to use Polonius instead of NLL, but rather use NLL and kick off Polonius for certain failure cases.
That slowdown might be temporary, as it is optimized over time, if I had to guess, since otherwise there might then be two solvers in compilers for Rust. It would be line with some other languages if the worst-case complexity class is something exponential.
Indeed. Based on the last comment on the tracking issue [0], it looks like they have not figured out whether they will be able to optimize Polonius enough before stabilization, or if they will try non-lexical lifetimes first.
[0]: https://github.com/rust-lang/rust-project-goals/issues/118
That said, that doesn't mean LLVM is always where the fixes need to be. For instance, one reason rustc spends a lot of time in LLVM is that rustc feeds more code to LLVM than it should, and relies on the LLVM optimizer to improve it. Over time, we're getting better about how much code we throw at LLVM, and that's providing performance improvements.
Similarly, you can build Clang using itself or using GCC. The resulting compiler should behave the same and produce the same machine code, even if its own machine code is somewhat different.
The produced binaries could still have artifacts from the original compiler in them, e.g. if "compiler built-in" libraries or standard libraries were compiled with the original compiler.
Both GCC and rustc use a multi-stage build process where the new compiler builds itself again, so you reach an idempotent state where no artifacts from the original compiler are left.
[1] https://github.com/rust-lang/rustc_codegen_cranelift/ [2] https://cranelift.dev/
> In the bootstrap process, the entire thing becomes way more complex. You see, rustc is not invoked directly. The bootstrap script calls a wrapper around the compiler.
> Running that wrapped rustc is not easy to run either: it requires a whole lot of complex, environment flags to be set.
> All that is to say: I don’t know how to debug the Rust compiler. I am 99.9 % sure there is an easy way to do this, documented somewhere I did not think to look. After I post this, somebody will tell me "oh, you just need to do X".
> Still, at the time of writing, I did not know how to do this.
> So, can we attach gdb to the running process? Nope, it crashes way to quickly for that.
It's kind of funny how often this problem crops up and the variety of tricks I have in my back to deal with it. Sometimes I patch the script to invoke gdb --args [the original command] instead, but this is only really worthwhile if it's a simple shell script and also I can track where stdin/stdout are going. Otherwise I might patch the code to sleep a bit before actually running anything to give me a chance to attach GDB. On some platforms you can get notified of process execs and sometimes even intercept that (e.g. as an EDR solution) and sometimes I will use that to suspend the process before it gets a chance to launch. But I kind of wish there was a better way to do this in general…LLDB has a "wait for launch" flag but it just spins in a loop waiting for new processes and it can't catch anything that dies too early.
* Run the whole tree of processes under `gdb` with `set detach-on-fork off`.
* LD_PRELOAD a library that inserts the sleeps for you, maybe on startup or maybe on signal/exit.
Ideally, we'd have some kind of infrastructure to name and identify particular processes recursively.
This is one area where rust disappoints me, there isn’t a “cargo debug” built in (there is an external program but it doesn’t work well), and when I just manually attach gdb most of the symbols are usually missing.
I would seriously consider a language billed as “debugger-first”, just to see what the experience was like.
Use C, tell gdb "-O0 -g -ggdb3"
I still sometimes get caught out getting debug information all in the right place, particularly when someone is using ninja or some such, I've even ended up wrapping gcc just to strip optimisation options and add -g, rather than figure out how to fight some very complex build system mess.
java -jar -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005
Then you connect to it with "jdb"
I don't think I want to do it myself, but I'd love a "101 ways to get into a debugger" webpage.
When it is loaded it will automatically talk to VSCode and tell it to start a debugger and attach to it & it waits for the debugger to attach.
End result is you just have to run your script with an environment variable set and it will automatically attach a nice GUI debugger to the process no matter how deeply buried in scripts and Makefiles it is.
https://github.com/Timmmm/autodebug
I currently use this for debugging C++ libraries that are dynamically loaded into Questa (a commercial SystemVerilog simulator) that is started by a Python script running in some custom build system.
In the past I used it to debug Python code running in an interpreter launched by a C library loaded by Questa started by a Makefile started by a different Python interpreter that was launched by another Makefile. Yeah. It wasn't the only reason by a long shot but that company did not survive...
System.Diagnostics.Debugger.Launch();
Which pops up a window asking you to select what debugger you want to use, and then opens your application in it.
We (Undo.io) came up with a technique for following a tree of processes and initiating process recording based on a glob of program name. It's the `--record-on` flag in https://docs.undo.io/UsingTheLiveRecorderTool.html. You can grab a free trial from our website.
For open source, with rr (https://rr-project.org/) I think you'd just `rr record` the initial process and you'll end up capturing the whole process tree - then you can look at the one you're interested in.
As others have said you could also do some smart things with GDB's follow-fork settings but I think process recording is ideal for capturing complicated situations like this as you can go and review what happened later on.
> Inlining heuristics are disabled and inlining is always attempted regardless of optimization level.
So it should be interpreted as "always attempt to inline", as opposed to "this must be inlined", or other attributes that instead influence the "should this be inlined" heuristic.
EDIT: as curious an attribute as it might be, I didn't mean to be talking about inclines
Google "llvm inline recursion". It exists. It should works. Fibonacci is the standard test case.
This back-end uses the confusingly-named libgccjit (not as JIT), which gives access only to a subset of GCC's functionality.
If something isn't already exposed, it might take a while to get patches to GCC and libgccjit accepted and merged.
This made me chuckle. Playing dumb so that gcc optimizes you away is both hilarious and genius
dwheeler•7mo ago
ramon156•7mo ago
mijoharas•7mo ago
> ...with the emphasis on limp. At some points, we are using over 100 GB of RAM... not ideal.
(so performance will be the next thing to be worked on to make this useable).
I think the goals of using gcc for rust is that it can provide a parallel implementation, which can help uncover bugs and provide more stability to the entire language, and because if there is a large gcc project already, they may be reticent to introduce LLVM as a dependency as well.
tucnak•7mo ago
That is a $200 word right there, and you're using it wrong.
mijoharas•7mo ago
https://www.merriam-webster.com/grammar/can-reticent-mean-re...
You're correct that some people recommend against it, and it's only been in use since the second world war.
echelon•7mo ago
Languages mutate quickly.
mijoharas•7mo ago
trelane•7mo ago
More on prescriptive vs descriptive: https://twominenglish.com/prescriptivist-vs-descriptivist/
I personally am more descriptivist when it comes to English. It is also a lot of fun to bring some M-W citations in order to "Well, actually" someone who is "Well, actually"-ing.
nartho•7mo ago
johnklos•7mo ago
pornel•7mo ago
This GCC support here is only a backend in the existing Rust compiler written in Rust. The existing Rust compiler is using GCC as a language-agnostic assembler and optimizer, not as a Rust compiler. The GCC part doesn't even know what Rust code looks like.
There is a different project meant to reimplement Rust (front end) from scratch in C++ in GCC itself, but that implementation is far behind and can't compile non-toy programs yet.