A bit of a meta question: what are the most relevant benchmarks by now?

I’ve been able to offload most tasks (coding or eles) to Codex since 5.3-codex with extra high thinking

So TL;DR benchmarking in a completely non-reproducible manner ?

"Model X performed great, but we can't possibly tell you anything about the code it was looking at apart from it was a large code base from an unknown company".

So basically pinky-promise benchmarking ?

I'm not sure I follow the value here ?

Wake me up when September ends or when I can do this locally.

> Each task comes from a private production codebase that we licensed from a real-world company

How does that work?

I think these benchmarks are not that useful, e.g. this suggests Fable is better than Astra, but in practice Astra is waaaaaay faster (like 5x; it's not even close), and also waaaay less annoying to talk to.

There's only two or three sane options here - you can easily try them all and pick yourself.

The fact that gemini 3.8 flash is so high up there just tells you this is an awful benchmark.

Try and use gemini 3.8 yourself for any real world work and you'll see it's terrible. It'll just go in circles reading the same file 20 times for no reason making hundreds of tool calls for a simple change.

My intuition is that many of the better & bigger 'private' code bases, at least in terms of claude code and codex... are not in fact private at this point.

One lesson of running botsbench.com, in a slightly different domain, is to measure for model contamination every time.