Oxford pretends AI benchmarks are science, not marketing
Chatbot vendors routinely make up a new benchmark, then brag how well their hot new chatbot does on it. Like that time OpenAI’s o3 model trounced the FrontierMath benchmark, and it’s just a coincidence that OpenAI paid for the benchmark and got access to the questions ahead of time.
Every new model will be trained hard against all the benchmarks. There is no such thing as real world performance — there’s only benchmark numbers.
reify•16m ago
Chatbot vendors routinely make up a new benchmark, then brag how well their hot new chatbot does on it. Like that time OpenAI’s o3 model trounced the FrontierMath benchmark, and it’s just a coincidence that OpenAI paid for the benchmark and got access to the questions ahead of time.
Every new model will be trained hard against all the benchmarks. There is no such thing as real world performance — there’s only benchmark numbers.
https://pivot-to-ai.com/2025/11/06/oxford-pretends-ai-benchm...