LLM leaderboard – Comparing models from OpenAI, Google, DeepSeek and others

https://artificialanalysis.ai/leaderboards/models

61•bookofjoe•19h ago

Comments

dang•18h ago

Benchmarks and comparison of LLM AI models and API hosting providers - https://news.ycombinator.com/item?id=39014985 - Jan 2024 (70 comments)

energy123•18h ago

You can consider the o3/o4-mini price to be half that due to flex processing. Flex gives the benefits of the batch API without the downside of waiting for a response. It's not marketed that way but that is my experience. With 20% cache hits I'm averaging around $0.8/million input tokens and $4/million output tokens.

Incipient•18h ago

Do you use them for code generation? I am simply using copilot as $10/mo is a reasonable budget...but quick guesses based on my use, would put code generation via an API at potentially $10/day?

energy123•18h ago

o3 is a unique model. For difficult math problems, it generates long reasoning traces (e.g. 10-20k tokens). For coding questions, the reasoning tokens are consistently small. Unlike Gemini 2.5 Pro, which generates longer reasoning traces for coding questions.

Cost for o3 code generation is therefore driven primarily by context size. If your programming questions have short contexts, then o3 API with flex is really cost effective.

For 30k input tokens and 3k output tokens, the cost is 30000 * 0.8 / 1000000 + 3000 * 4 / 1000000 = $0.036

But if you have contexts between 100k-200k, then the monthly plans that give you a budget of prompts instead of tokens are probably going to be cheaper.

qmmmur•17h ago

I’m shocked people are signing up to pay even these fees to build presumably CRUD apps. I feel a complete divergence in the profession between people who use this and who don’t.

thedevilslawyer•16h ago

A whole codebase of 100k lines (~1M tokens) for ~a dollar. Would like to understand why would signing up for this be shocking?

rowanG077•12h ago

That's really misrepresenting how it works. Most lines will be written, re-written again and adjusted multiple times. Yesterday I did approx 5 hours of peer-coding with claude 4 opus. And I have these stats:

Total tokens in: 3,644,200 Total tokens out: 92,349

And of that only approx 2.3k lines where actually commited for PRs.

simonw•12h ago

I calculate that as $61.59 https://www.llm-prices.com/#it=3644200&ot=92349&ic=15&oc=75

So that's about $12/hour, or 2.6 cents per line of finished code.

Still pretty cheap! Very few unassisted human programmers can churn out 2300/(5 * 60) = 7.6 lines of code per minute consistently over a five hour time span.

That said, I think Claude Code, while impressive, is incredibly quick to burn through tokens. I still mostly use copy-and-paste info Claude or ChatGPT as my main AI-assisted workflow which keeps me in more control and spends a ton less tokens.

rowanG077•11h ago

Yes I can confirm that's approx what I paid. My first time using claude 4 opus and I used aider. It seems the estimation aider gives is very wrong as it was telling me I used approx 15$. I only noticed because my credit ran out. The $/performance tells me I should check what grok4 can do. I didn't use it seriously yet.

simonw•11h ago

Claude Opus 4 is 5x the price of Claude Sonnet 4. I don't think it's 5x as good. I default to Sonnet and rarely use Opus - in this case Sonnet would have cost about $12.31 for the same volume of tokens.

0points•10h ago

There are code generators for CRUD. You could be a 10x AI programmer without AI if the measure is how fast you bang out CRUDs.

koakuma-chan•13h ago

Some people are struggling to build CRUDs.

molticrystal•17h ago

For those curious on a few of the metrics, besides $/token, tokens/s, latency, context size, they use the results from:

    MMLU-Pro (Reasoning & Knowledge)  
    GPQA Diamond (Scientific Reasoning)  
    Humanity's Last Exam (Reasoning & Knowledge)  
    LiveCodeBench (Coding)  
    SciCode (Coding)  
    HumanEval (Coding)  
    MATH-500 (Quantitative Reasoning)  
    AIME 2024 (Competition Math)  
    Chatbot Arena  (selectively used)

NitpickLawyer•16h ago

> Humanity's Last Exam (Reasoning & Knowledge)

Article yesterday was saying that ~30% of the chemistry/biology questions on HLE were either wrong, misleading or highly contested in scilit.

pogue•17h ago

Look at that bar graph comparing the price of every model compared to Claude Opus

It's a shame it's so good for coding

https://artificialanalysis.ai/models/claude-4-opus-thinking/...

teaearlgraycold•17h ago

I’ve had very mixed results with 4 Opus. It’s still just a language model and can’t understand some basic concepts.

matltc•13h ago

Do you think it is demonstrably better than Sonnet? Grabbed a pro sub last month shortly after the cli tool dropped, but have not used it past couple weeks because I found myself spending way more time correcting it than getting useful output

cc-d•17h ago

How about adding a freedom measurement in those columns?

andy99•12h ago

Impossible to be objective on what that means. I can see having a "baggage" field that lists non performance-related concerns for each.

LeoPanthera•17h ago

Is there an index for judging how much a model distorts the truth in order to comply with a political agenda?

OsrsNeedsf2P•17h ago

How would you create the base "truth" for these models? People are adamant about both sides of many topics.

"Which country started the Korean war?", "Did Israel genocide the people of Gaza?", "Does China have lawful rights over Taiwan?"

LeoPanthera•17h ago

Hopefully obviously, by testing it against objective facts which are nonetheless "controversial" politically.

thedevilslawyer•16h ago

In the end many of these are "political facts" and not objective like what year was a person born in. The answer to your question is as simple as - come up with the actual list of "facts", and then run a simple eval with every model on them.

The implementation is trivial - the listing down of "political facts" is the hard part.

mattigames•17h ago

For a start you don't ask such subjective questions, that's a bit silly, instead you ask for e.g. the death toll of Israel vs Palestine in the last year, the number of deaths surrounding the tianammen square protests, if it gives you a straight answers with numbers (or at least a consistent estimate) and citing it's sources it's a good start.

thedevilslawyer•16h ago

Let's take the example you have listed:

1) where would you get the death toll from? What would be the sources of truth?

2) Are there conflicting sources?

3) if yes, what is your expectation for the correct response?

mattigames•13h ago

They are all controversial matters, therefore conflicting sources are not only expected but desired to be informed by the LLMs when asking such matters, the report by well-funded likely-biased sources (e.g. Israel government) would obviously needed to be given less credibility, estimates that are widely different that all the rest would also need to be given less credibility, and so on.

thedevilslawyer•13h ago

Thanks, these handwavy and subjective answers hopefully tells you why the questions of the grand-parent are not "silly".

k4rli•9h ago

Perhaps universal truth or objective facts simply don't exist anymore? Or have they ever?

Tiananmen square might have been bad, not too familiar with Asian happenings, but so are post-WW2 conflicts started by western nations.

kouteiheika•13h ago

It's not perfect, but, yes: https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard

l5870uoo9y•16h ago

It is interesting that it ranks `GPT-4.1 mini` higher than `GPT-4.1` (the latter costing five times more).

__mharrison__•16h ago

Here's my plot based on Aider benchmarks

https://www.linkedin.com/posts/panela_important-plot-for-fol...

witnessme•15h ago

Surprised to find out grok 3 mini is so economic and ranks higher than equivalent gpt models. I run most of my agents on gpt4.1 mini, might switch now

Garlef•14h ago

Is there an option to filter the list based on the measurements? I.e "context window > X, intelligence > Y, price < Z"? I'd love that.

It seems the only filter options available are unrelated to the measured metrics.

(I might have missing this since the UI is a bit cluttered.)

globular-toast•14h ago

Whenever you present a table with sorting ability you might as well make the first click ascending or descending according to what makes the most sense for that column. For example I'm highly unlikely to be interested in which model has the smallest context window, but it's always two clicks to find which one has the highest.

Sorting null values first isn't very useful either.

lm28469•14h ago

Vibe coded websites be like

koakuma-chan•13h ago

Not necessarily vibe coded. Sometimes developers don't actually care about the product, and just want to get it over with.

archargelod•11h ago

Still sounds like vibe coding. If they don't care about product, nothing is stopping them from taking the AI shortcut.

loehnsberg•13h ago

Interesting to learn that o4-mini-high has the highest intelligence/$ score here at par with o3-pro which is twice as expensive and slow.

HackerTool – A cool toolkit with many useful features

Second Reality Ported to Windows

Tell HN: Gemini CLI is buggy; use at your own risk

Withdrawal of Baltimore-Washington Superconducting Magnetic Levitation Project [pdf]

DIY Dual-Screen Cyberdeck with Raspberry Pi 5

Corporation for Public Broadcasting (funder of NPR, PBS) will shut down

Dartboat: Darts Game Scorer

Fast and Slow

Hal Abelson (Co-Author of SICP) InfiniteHistoryProject MIT (2011) [video]

Generate Charli XCX Brat-style text and album covers

Trailblazing SF coffee chain about to be sold to private equity firm for $145M

Twentyseven 1.0

Groundwater depletion sinks home prices in California's Central Valley

Was the Renaissance Real?

Cerebras Code

Are you sure you are buying good car?

U.S. fires statistics chief after soft jobs report

Updated Digital Identity Guidelines are Here (NIST 800-63 Revision 4)

Y Combinator is looking for DOGE-related startups for its next cohort

Coffeematic PC – A coffee maker computer that pumps hot coffee to the CPU

Tim Cook rallying Apple employees around AI efforts

Anthropic Revokes OpenAI's Access to Claude

The State of Design and the Iconfactory

GitHub Copilot crosses 20M all-time users

First Contact – New in Little Snitch Mini 1.8

AI is entering an 'unprecedented regime.' Should we stop it - and can we?

Stack traces for Postgres errors with backtrace_functions

China Just Refueled a Satellite in Space

Ask HN: How do you avoid job hunting burnout?

GM's $4B Bet: Gasoline Is Still King

HackerTool – A cool toolkit with many useful features

Second Reality Ported to Windows

Tell HN: Gemini CLI is buggy; use at your own risk

Withdrawal of Baltimore-Washington Superconducting Magnetic Levitation Project [pdf]

DIY Dual-Screen Cyberdeck with Raspberry Pi 5

Corporation for Public Broadcasting (funder of NPR, PBS) will shut down

Dartboat: Darts Game Scorer

Fast and Slow

Hal Abelson (Co-Author of SICP) InfiniteHistoryProject MIT (2011) [video]

Generate Charli XCX Brat-style text and album covers

Trailblazing SF coffee chain about to be sold to private equity firm for $145M

Twentyseven 1.0

Groundwater depletion sinks home prices in California's Central Valley

Was the Renaissance Real?

Cerebras Code

Are you sure you are buying good car?

U.S. fires statistics chief after soft jobs report

Updated Digital Identity Guidelines are Here (NIST 800-63 Revision 4)

Y Combinator is looking for DOGE-related startups for its next cohort

Coffeematic PC – A coffee maker computer that pumps hot coffee to the CPU

Tim Cook rallying Apple employees around AI efforts

Anthropic Revokes OpenAI's Access to Claude

The State of Design and the Iconfactory

GitHub Copilot crosses 20M all-time users

First Contact – New in Little Snitch Mini 1.8

AI is entering an 'unprecedented regime.' Should we stop it - and can we?

Stack traces for Postgres errors with backtrace_functions

China Just Refueled a Satellite in Space

Ask HN: How do you avoid job hunting burnout?

GM's $4B Bet: Gasoline Is Still King

LLM leaderboard – Comparing models from OpenAI, Google, DeepSeek and others

Comments