Ask HN: What benchmarks are you using to judge AI models?

4•cowpig•9mo ago

There are so many models, and so many new ones being released all the time, that I have a hard time knowing which ones to prioritize testing anecdotally. What benchmarks have you found to be especially indicative of real-world performance?

I use:

* Aider's Polyglot benchmark seems to be a decent indicator of which models are going to be good at coding:

https://aider.chat/docs/leaderboards/

* I generally assume OpenRouter usage to be an indicator of a model's popularity, and by proxy, utility:

https://openrouter.ai/rankings

* LLM-Stats has a lot of charts of benchmarks that I look at:

https://llm-stats.com/

Comments

paulcole•9mo ago

> There are so many models, and so many new ones being released all the time, that I have a hard time knowing which ones to prioritize testing anecdotally

Just pick one and use it. The ones you’ve heard of (if you are not obsessively refreshing AI model rankings pages) are basically the same.

I’m sure I’ll get a ton of pushback that the one somebody loves is obviously so much better than the other one, but whatever.

Just give me OpenAI’s most popular model, their fastest model, and their newest model. I’ll pick among those 3 based on what I’m prioritizing in the moment (speed, depth, everyday use).

kadushka•9mo ago

For me it's the opposite - we don't get enough models to test. In the last 6 months, we got Claude 3.7, OpenAI o1, Grok 3, Gemini 2.5 Pro, and OpenAI o3. That's it - 5 frontier models. Not that hard to test each one of them manually, which I did for many hours and with many different tasks. o1 --> o3 and 2.5 Pro are the ones I'm using the most.

I couldn't care less about benchmarks - I know what these models are capable of from personal experience.

LLMs are powerful, but enterprises are deterministic by nature

Ask HN: Anyone Using a Mac Studio for Local AI/LLM?

Ask HN: Ideas for small ways to make the world a better place

Ask HN: Non AI-obsessed tech forums

Ask HN: 10 months since the Llama-4 release: what happened to Meta AI?

Ask HN: Non-profit, volunteers run org needs CRM. Is Odoo Community a good sol.?

Ask HN: Who wants to be hired? (February 2026)

AI Regex Scientist: A self-improving regex solver

Ask HN: Who is hiring? (February 2026)

Tell HN: Another round of Zendesk email spam

Ask HN: Is Connecting via SSH Risky?

Ask HN: Has your whole engineering team gone big into AI coding? How's it going?

Ask HN: Why LLM providers sell access instead of consulting services?

Ask HN: What is the most complicated Algorithm you came up with yourself?

Ask HN: How does ChatGPT decide which websites to recommend?

Ask HN: Is it just me or are most businesses insane?

Ask HN: Any International Job Boards for International Workers?

Ask HN: Mem0 stores memories, but doesn't learn user patterns

Ask HN: Is there anyone here who still uses slide rules?

Kernighan on Programming

Ask HN: Anyone Seeing YT ads related to chats on ChatGPT?

Ask HN: Does global decoupling from the USA signal comeback of the desktop app?

We built a serverless GPU inference platform with predictable latency

Ask HN: How Did You Validate?

Ask HN: Does a good "read it later" app exist?

Ask HN: Have you been fired because of AI?

Ask HN: Cheap laptop for Linux without GUI (for writing)

Ask HN: Anyone have a "sovereign" solution for phone calls?

Test management tools for automation heavy teams

Ask HN: OpenClaw users, what is your token spend?