Oxford pretends AI benchmarks are science, not marketing
Chatbot vendors routinely make up a new benchmark, then brag how well their hot new chatbot does on it. Like that time OpenAI’s o3 model trounced the FrontierMath benchmark, and it’s just a coincidence that OpenAI paid for the benchmark and got access to the questions ahead of time.
Every new model will be trained hard against all the benchmarks. There is no such thing as real world performance — there’s only benchmark numbers.
I find this phrase really amusing every time I see it, since "homo unius libri" meant "someone who has studied one thing and mastered it", rather than "someone with narrow knowledge".
reify•2mo ago
Chatbot vendors routinely make up a new benchmark, then brag how well their hot new chatbot does on it. Like that time OpenAI’s o3 model trounced the FrontierMath benchmark, and it’s just a coincidence that OpenAI paid for the benchmark and got access to the questions ahead of time.
Every new model will be trained hard against all the benchmarks. There is no such thing as real world performance — there’s only benchmark numbers.
https://pivot-to-ai.com/2025/11/06/oxford-pretends-ai-benchm...