4 days ago Grok 4.5 and GPT5.6 were released. In our study, they beat all Anthropic models for finding vulnerabilities in Pull Requests.
We ran 10 models over the same 10 pull requests, each carrying one planted access-control bug (IDOR, missing auth, broken authorization), five times per model, and scored every finding against the actual code.
The result is a clear cost/quality frontier. Open AI’s GPT-5.6 Sol dominates. It found all vulnerabilities at a very affordable $0.70 per PR. For comparison Fable only found 88% and cost $3.61 per PR. No Anthropic model reaches the frontier at all.
This is NOT a harness quirk, we tested and found similar results in the Claude Code and Pydantic harnesses.
Details of our methodology and synthetic repos in the publication.
pcollins123•1h ago
We ran 10 models over the same 10 pull requests, each carrying one planted access-control bug (IDOR, missing auth, broken authorization), five times per model, and scored every finding against the actual code.
The result is a clear cost/quality frontier. Open AI’s GPT-5.6 Sol dominates. It found all vulnerabilities at a very affordable $0.70 per PR. For comparison Fable only found 88% and cost $3.61 per PR. No Anthropic model reaches the frontier at all.
This is NOT a harness quirk, we tested and found similar results in the Claude Code and Pydantic harnesses.
Details of our methodology and synthetic repos in the publication.