As an author of the blog, I'll note that this was one of the easiest applications of ADRS. Bowen, who was leading this effort, got things running within a day or two and the initial runs were with free Google credits! It was exciting to see how quickly these kinds of frameworks could be applied to real-world engineering and algorithmic challenges.

Nice result, but the snake pattern is pretty obvious and intuitive even for a human who just glances over the problem. It kinda breaks if there is huge variance (if the top load expert is orders of magnitude higher than #2 it probably should just get its own GPU), but I'm not familiar enough with MoE to know if that's a realistic possibility.

Alternate title: “Human experts discover a 5x faster MoE load balancing algorithm using AI”

i wonder how hard it is to get the setup for AI to evolve on?

Really cool to see the AI-discovered algorithm is not just a theoretical result but is actually in a PR for vLLM. My question is about the code itself. Was the Python/PyTorch generated by OpenEvolve directly usable, or did it require significant human cleanup to make it readable, maintainable, and conform to the project's coding standards? I'm curious about how close we are to AI generating production-ready, human-editable code for complex algorithms.

I'm not sure if this is the exact same thing, but a load balancing paper reported a 4.2x speedup by applying a "snake pattern" in 2024:

https://arxiv.org/pdf/2402.02447

this feels less like Copilot and more like AlphaGo for systems programming. it's not just finding patterns in existing code, but discovering novel and more efficient strategies in a given problem space. Very cool.

The final code might be fast, but is it understandable? The evolution process shows it tried a bunch of things that didn't work. The final result is a heuristic that won out based on a specific simulator and fitness function.

does this only work for vLLM or is generally applicable?

i'm skeptical this generalizes beyond problems that can be expressed as "rearrange tensors faster". it feels like a solution that only works for a very narrow and convenient class of problems.

getting a 5x speedup for less than $10 and in just five hours is insane. the roi on this approach is going to be hard to beat.

So, if I got this right, this is just about re-implementing an existing load balancing algorithm faster...? If so, this is really dumb. As you guys checked out, yes most load balancing algorithms are slow/dumb:

>First, we evaluate DeepSeek's open-source EPLB implementation. This employs a greedy bin-packing strategy: experts are sorted by load in descending order, and each is placed onto the least-loaded GPU that has capacity (Figure 3a, Example 1). While simple, the solution is slow because it written in Python and uses a for-loop to performs linear search for finding the best-fit GPU choice.

This is because when considering a load balancing algorithm, unless the work being done (in this case by the GPU) lasts only a few ms, the load balancing algorithm being fast will never be the bottleneck. The post does not mention whether this is the case at all.

Also, I don't want to sound rude, but if all they managed to get is a 5x increase over a simple python algorithm, I don't think this is impressive at all...? Any rewrite of the 'dumb' algorithm in a language with more memory control and cache continuity should result in much better results.