First chart: from score 61 (GPT-5.6 Sol) to drumroll 61 (GPT-6 Astra)
flyaway123•35m ago
Indeed. Though to be fair it is referring to "Artificial Analysis Coding Agent Index", from 65 to 67.
dist-epoch•17m ago
I think they mean cost per task, where Astra is now on the Pareto frontier.
ekojs•42m ago
Well, seems like ECI [0] and the AA index is diverging quite a bit. Benchmarking LLM is tough and I think we are seeing the limitations of current benchmarks and applicability to real tasks.
Readerium•53m ago
lostmsu•43m ago