fp.

I've refreshed StarSkirmish Bench, a benchmark where LLMs play StarCraft BW by writing code, with results from the flagship models of OpenAI, Anthropic, Google, and Muse

Comments

BeaverGoose•43m ago

They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.

__cayenne__•35m ago

I did a live stream where Astra played Pluto. Astra went 0 - 1000 https://x.com/kaimcpheeters/status/2106782151622504911?s=20

deburo•22m ago

I noticed Pluto is running on a simple CPU? If so, that's very nice for future RTS games. You just ship (or use a server-client arch) a bunch of models of various difficulty levels to let players practice. Could we even get models tuned to specific campaign levels? I wonder how costly the creation of Pluto was.

qsort•39m ago

It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?

__cayenne__•29m ago

Yes during the 1 hour window they are allowed to play practice matches against opponents they are graded against - though they do not have access to their opponents source code.

They drop to single digits against the A tier and S tier bots because those bots are just generally harder opponents to beat.

jstummbillig•38m ago

How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.

m_w_•37m ago

This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.

It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.

__cayenne__•32m ago

I did a 48 hour live stream - Astra cleared in 43 hours.

Although I should caveat the format advantaged the LLM bots compared to traditional tournament rules.

0ddj0b•30m ago

Any thoughts on building this for Starcraft 2?

caughtinthought•4m ago

How do the best humans do against the LLMs?