This means that generic benchmarks and evals are sort of passé. There's no need to have them write "make a coffee shop page" or whatever. Instead, simply focus on the real problem you have and evaluate whether they solve that problem. You want to move along the price-performance frontier alone and the SOTA models are decidedly at the top of the performance curve but very expensive so they serve very well to produce the gold standard and to judge.
The change from prior to today is that benchmaxxing and per-token pricing allowing for evaluation point to the same direction: do not proxy results. Instead deploy the highest end model you have as a judge for others until you have statistical confidence in discrimination and move along the task-specific frontier.
I highly doubt that you have solved it. Writing proper evals and benchmarks for real-world scenarios is far from trivial.
Benchmarking an agent essentially means freezing, at the very minimum:
- the model
- the model's configuration (e.g. effort, permissions, provider)
- the dataset (e.g. a git repository at a specific sha)
- the code running the agent itself (you can build your own harness, trivial, but you still need to ship it as a single executable, froze in time. benchmarking against closed source runtime like claude code is quite useless, they change too frequently and in ways you cannot directly inspect).
- the tools at agent's disposal. Even a slightly different implementation of tool X (e.g. grep or readfile or sed) has an impact. In general this implies also freezing a very specific container image. In my personal benchmarks I provide a specific list of tools that come with the executable, there's no possibility of interacting with the outside world besides the provided apis, the agent bundles its own tools.
And even then: there's significant noise coming from the LLM providers themselves which noticeably change the models behaviour, I don't know whether this is because they optimize some settings or change the inference over time, etc.
And, last but not least, the output of LLMs is non deterministic.
Also, the LLM as judge presents essentially the same non-deterministic problems, has to be benchmarked itself thoroughly, and writing quality rubrics or "golden answers/outputs" is just difficult. One of the metrics I consistently try to emphasize is to avoid the "shotgun vomit dump" of information. So answers that get right to the point in plain terms avoiding dumps of information filled of jargon on top of the user are rated differently.
In short: its far from trivial to benchmark models on real-world agentic work taken from your personal or professional projects.
And even creating the test cases themselves is hard. No: you cannot take the output of some "sota" and use it as gold standard. This is a very crap approach. It's the sloppiest solution to the problem, in the very sense of slop: plausible, average, lacking any creativity or out of the box thinking, the things that make the real difference in complex software development.
The very point of creating these benchmarks is to find which configuration/model/tools/harness (skills/mcps/documentation/agents.md, etc) works better.
And it only works if you create these benchmarks yourself from genuinely difficult non-trivial work and find a solution that is better by most metrics implementation-wise, albeit you could settle on the implementation solving a series of cases and edge cases.
In my benchmarks, I started insisting on having at least 5 runs.
This also makes me very suspicious of some of the posted benchmarks that compare models. If the results don't say how many benchmark runs were performed and what the variance was, there is really no basis for comparison. You are just guessing.
I'm asking because I personally only use AI with specific and detailed instructions, building my projects piece-by-piece.
I mostly don't look at the low level code and some of it I don't understand as much as I'd like, but I very much give much more technical instructions than a simple, two sentence prompt.
So, it seems to me this "oneshot from a simple prompt" eval is fairly meaningless when it comes to model evaluation itself, as it is in no way representative of real world application.
This would be more something for "vibe coders", people with little to no programming background wanting a website?
I'm interested to know how the classic AI "purple preference" emerged (organically?) and if the beige wave came out of specific training efforts to combat it?
To your point on development work (the code itself), I was talking to some friends on the Google Chrome team about any research understanding the model's preferences around framework ergonomics and abilities to properly implement core web standards for given tasks. I think that would be super fascinating.
Kimi, GLM, and Deepseek all absolutely run away with the "Can I quickly read the menu and find the address" challenge.
Most of the rest of the pages are stylistic, but hard to parse.
If I were in a car on a mobile phone trying to find the address of the place to meet a friend for coffee... I don't want a bunch of fluff and stylistic design that makes it hard to parse the information on the site.
How so? Surely they can just steal such generic graphics off existing web sites.
Sorry for linking these again, but they are a solid rebuttal to every "we compared 11 different models" post that reaches the top of Hacker News.
1. Massive variance on identical tasks: Sonnet ranges from 8,163 to 17,334 tokens solving the exact same lambda calculus problem over 5 runs [0]. Five runs isn't even enough for a full comparison, but it clearly demonstrates how wildly variable model output is. The flame graphs help visualize the thinking process to show why that variance happens.
2. With enough runs of the same prompt, models become extremely predictable. Although not deterministic, understand this will help us use the models more effectively. Scroll down to the blue "Walk it from the start" button on this page [1]. It maps the probability of solving a problem as it gets progressively more complex, helping visualize the probability distribution of reasoning models so it stops being a complete black box. (Note: this required thousands of runs and serious compute).
Single-prompt or 3-run "evals" are just noise without statistical rigor.
If you scroll a little further there is a clean comparison of capacity -- opposed to capability which is using SVG to paint a pelican on a bicycle -- of Qwen3-4B vs. Phi-4-reasoning over thousand of runs on the same 144 problems.
It appears that they're mostly testing the ability to make business tasks autonomous, with ~20% associated to development tasks (ssh here, install this, etc.), but not actual programming.
Yes, the most valuable benchmarks and evaluations you can write are those that resemble your work.
The evaluations are extremely hard to write and test.
And yes, virtually all benchmarks are E2E one shots, they do not reflect multi turn processes or how most people interact with LLMs.
Which is why every Opus after 4.6 looks better on benchmarks, but is hard to work with interactively.
It's still a work in progress but preliminary results reveal that Sol is able to reproduce 70%-90% of Fable's performance. This is a very meaningful result for me because code review is what I use AI for.
isqueiros•48m ago
If that's the entire prompt, it's quite depressing how much alike these all look. I appreciate some of the details from the Opus 5 version, but I can't help but strongly feel the AI vibes emanating from that design.
hombre_fatal•32m ago
On the other hand, if you want something different with LLMs, all it takes is a few more words of creative flair in the prompt.
skinfaxi•16m ago
mym1990•4m ago
andy99•30m ago
tantalor•30m ago
If all you want is "opening hours, the address, a short menu and a photo" there are easier and cheaper ways to do that.
miyoji•26m ago
Me, I'm saying that, and I've skipped going to coffee shops and restaurants because they don't have a website, just a fucking Facebook page. I don't use Meta products and can't see their page if I'm not logged into an account I don't have, so I do what the business owner intended: I go fuck myself and get coffee somewhere else.
altmanaltman•18m ago
skinfaxi•14m ago
altmanaltman•6m ago