Hi HN, we're the team behind Heal. We recently wanted to measure the actual limits of static reasoning in LLMs: specifically, what classes of bugs AI code reviewers are fundamentally blind to because they lack a runtime.
To measure this, we ran a benchmark. We took 5 products (3 open-source, 2 private) and replayed 236 real pull requests from their history. We ran three popular AI PR reviewers on default settings, and then ran Heal, our agent that builds an application sandbox (to control the seeds, mock dependencies etc)around the application and drives it end-to-end like a user would.
- A runtime allows LLMs to test and verify assumptions without risking increasing noise level.
- PR bots are still great for style, consistency, and files outside the user path (CI config, docs, etc.)
We stopped after spending a few thousand dollars running this because the signal was strong enough, but we’re hoping to grow the dataset. We also acknowledge that requiring a reproducible test (what Heal does) is a different standard than an LLM just leaving a comment.
We’d love to hear your thoughts on the methodology, or if you’ve run into the limits of static AI reviewers in your own workflows. Happy to answer any questions.
batikha•56m ago
To measure this, we ran a benchmark. We took 5 products (3 open-source, 2 private) and replayed 236 real pull requests from their history. We ran three popular AI PR reviewers on default settings, and then ran Heal, our agent that builds an application sandbox (to control the seeds, mock dependencies etc)around the application and drives it end-to-end like a user would.
A few insights:
- Stacking code reviewers doesn’t yield better results.
- A runtime allows LLMs to test and verify assumptions without risking increasing noise level.
- PR bots are still great for style, consistency, and files outside the user path (CI config, docs, etc.)
We stopped after spending a few thousand dollars running this because the signal was strong enough, but we’re hoping to grow the dataset. We also acknowledge that requiring a reproducible test (what Heal does) is a different standard than an LLM just leaving a comment.
We’d love to hear your thoughts on the methodology, or if you’ve run into the limits of static AI reviewers in your own workflows. Happy to answer any questions.