Brood War Bench

https://bw.swerdlow.dev/report

61•benswerd•6h ago

Comments

benswerd•6h ago

+ Playable Agent driven Starcraft

conorcleary•43m ago

Dibs on the fly brain

bee_rider•1h ago

A ton of conversations about the game must be in the training set. I wonder, is there any way just from watching how they play, of telling if they tend to pick strategies that people complain or meme about online?

stackghost•1h ago

I wonder if there is a library to decipher brood war replay files. Perhaps an agent could learn by watching.

frutiger•1h ago

I haven’t checked for SC:BW but Blizzard has official parsers/replayers for SC2 on GitHub.

benswerd•49m ago

Not hard to build. I was shocked at how fast/easy this was to pull together.

malfist•1h ago

This is a great idea for a benchmark. Something all the benchmarks seem to be missing is strategy, tactical solutions in most of the benchmarks are all thats required but here requires actual long term thinking and tactical thinking, balancing and orchestration.

shard972•55m ago

That’s why I tried making https://wrathbench.shard.page

Game_Ender•54m ago

Any details about the harness the agents were given? I am curious what representation of the screen and world state was provided to the agents and what tools they had available.

benswerd•49m ago

Oh sorry I should be more clear on that. Will add to report.

For agent harness I did Claude Code, Codex, Grok Build. This was primarily a cost driven decision — I have a lot of free tokens and I didn't want to pay API prices for this.

For game harness I used minimal BW-API issue command and get observation apis as tools. I felt this was the most fair way to do it on my small scale.

In the future I would like to integrate code mode and multiple games/I think if it was a best of 5 where each agent could learn from its past games and build its own automations over time that would be much more interesting.

pelagicAustral•54m ago

Unrelated to the benchmark...

I love StarCraft. I started playing it right from the beginning, most of my friends right now are from that era. I literally met people that have spread to almost every continent when I was in my early teens. We played at internet cafes and did not have access to the internet, that was priced differently...

I miss those days so much.

Everybody was from a different background back then, and nobody was anything other than a guy that plays StaCraft at the cybercafe... And now, we are in our 40's and I know Math teachers, history teachers, oil rig operators, software programmers, professional gamers, lawyers and more... hahah So crazy to think about it... and I know them, we talk, what a world.

sidewndr46•48m ago

Later we even listened to Eminem and played violent video games. Most of have never even been charged with a crime, much less abused someone.

benswerd•37m ago

I agree.

My first time playing StarCraft was at summer camp around a decade after it came out.

All the smartest people played it so I wanted to too. Great decision, I have been continually impressed with the people who StarCraft introduced me to.

nemo1618•37m ago

Even at Burning Man, in the middle of the desert, there is a camp that hosts a StarCraft tournament every year (on the dustiest setups you've ever seen!) :)

benswerd•37m ago

So dope

GodelNumbering•47m ago

A friend of mine created GoBench[1][2] that evaluates LLMs on 9×9 Go using KataGo opponents as Elo anchors, you see real capability differences there, like Astra Max substantially leading all other models. I think strategy is a generally interesting area to evaluate LLMs on

[1] https://rolandgao.com/blog/gobench/

[2] https://rolandgao.com/gobench.pdf

winwang•47m ago

Would be interesting if you could team a fast and slow agent together -- slow model can either act directly or maybe just communicate to the fast model.

benswerd•41m ago

I might open this up to a tournament if enough people want. Any interest?

dschuessler•47m ago

Somewhat related: In 2018, Google DeepMind had already created AIs that were capable of beating professional gamers in StarCraft 2 (the sequel to Brood War): https://www.youtube.com/watch?v=cUTMhmVh1qs

benswerd•42m ago

I predict LLMs will reach superhuman level and beat even that model in the next 12 months

orbital-decay•20m ago

Starcraft is APM-dependent. Unless the latency will improve greatly in frontier reasoning LLMs (which is unlikely), it will remain a bit like knitting with an excavator.

benswerd•18m ago

I predict latency will improve greatly in the next 12 months to more than 4x speed on current frontier tasks

loeg•17m ago

Yeah but Starcraft needs, like, 10-20x the APM these agents are doing.

benswerd•15m ago

I’m not convinced a lot of it can’t be solved with code mode.

Marine staggering for example seems like an ideal code mode task.

gadtfly•46m ago

Did it play by looking at screenshots and sending clicks, or was there other mediation/symbolization?

It sounds like it might have been actually played in real time, which would be very important to distinguish.

I have recently seen other harnesses letting agents play real-time games in what seems like discrete time slices, turning eg Portal into something turn-based https://www.youtube.com/watch?v=ruuGXFAmiOE

loeg•18m ago

There's a bot data stream already; it's probably hooked up to that rather than screencap. Yes, I believe these were playing in real time.

faeyanpiraat•42m ago

There is currently a bot beating everyone on the ladder. Just watched it today on Artosiscasts yt channel.

tweakimp•21m ago

If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net/

minimal_action•13m ago

I think we're on the early days of games you connect with your agent to. Human + AI units one versus the other. Like knights with their horses. Not sure which is the horse..

AntiRush•11m ago

Back in 2010, during the early days of bwapi, there was a Brood Wars AI tournament held by the Expressive Intelligence Studio at UC Santa Cruz. It's interesting to see how different the approaches were back then, vs this or Deepmind's SC2 work.

https://web.archive.org/web/20091124210529/http://eis.ucsc.e...

There's a great contemporary Ars Technica piece by a competitor:

https://arstechnica.com/gaming/2011/01/skynet-meets-the-swar...

As an undergrad I did a project using genetic programming. It was not very successful, but it was a lot of fun.

https://tomisin.space/archive/starcraft-genetic-programming/

I built non-autoregressive decision models with RL a year ago

AI-generated posters don’t have to be horrible

Brood War Bench

Measure internet censorship. Contribute to the largest open dataset

ZK-JPEG: Zero-Knowledge Image Editing and Compression

I captured 72 hours of idle Android packets behind pfSense

Two parallel neural ectoderm progenitors contribute to the developing brain

Show HN: CUA-S1 – A System One Model for Computer Use

Deodands put a price on objects that caused death

Suzanne Ciani's Buchla Cookbook

Tin: full-text search for Postgres

Android 17 is the first since 3.x to add new APIs without releasing to the AOSP

The Secret Life of Circuits

UFO Series Home Page: "UFO" TV Series from 1970

Supabase (YC S20) Is Hiring for OrioleDB

New evidence for hidden chambers beyond Tutankhamun's tomb

Black Holes or Black Hole Stars? Astronomers Spar over 'Little Red Dots'

GPT-6 Astra Solves a WWI German Radio Cipher

I think you should almost never use AI to write

Btrfs/ZFS/bcachefs under workloads classic benchmarks skip

Adventures in Microcontroller Circuit Debugging

San Francisco Onion Futures Company

How to Write with an LLM

Cloudflare Quick Tunnels

Microsoft director: AI scraping 'the largest theft of labor in human history'

If math is more than proof, we need to better celebrate the rest of it

Saving another 100TB of RAM

You can run Git on object storage if you re-make packfiles

What Zig felt like, coming from Rust

SDCC – Small Device C Compiler

I built non-autoregressive decision models with RL a year ago

AI-generated posters don’t have to be horrible

Brood War Bench

Measure internet censorship. Contribute to the largest open dataset

ZK-JPEG: Zero-Knowledge Image Editing and Compression

I captured 72 hours of idle Android packets behind pfSense

Two parallel neural ectoderm progenitors contribute to the developing brain

Show HN: CUA-S1 – A System One Model for Computer Use

Deodands put a price on objects that caused death

Suzanne Ciani's Buchla Cookbook

Tin: full-text search for Postgres

Android 17 is the first since 3.x to add new APIs without releasing to the AOSP

The Secret Life of Circuits

UFO Series Home Page: "UFO" TV Series from 1970

Supabase (YC S20) Is Hiring for OrioleDB

New evidence for hidden chambers beyond Tutankhamun's tomb

Black Holes or Black Hole Stars? Astronomers Spar over 'Little Red Dots'

GPT-6 Astra Solves a WWI German Radio Cipher

I think you should almost never use AI to write

Btrfs/ZFS/bcachefs under workloads classic benchmarks skip

Adventures in Microcontroller Circuit Debugging

San Francisco Onion Futures Company

How to Write with an LLM

Cloudflare Quick Tunnels

Microsoft director: AI scraping 'the largest theft of labor in human history'

If math is more than proof, we need to better celebrate the rest of it

Saving another 100TB of RAM

You can run Git on object storage if you re-make packfiles

What Zig felt like, coming from Rust

SDCC – Small Device C Compiler

Brood War Bench

Comments