They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.

It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?

How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.

This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.

It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.

Any thoughts on building this for Starcraft 2?