GPT-OSS 120B Runs at 3000 tokens/sec on Cerebras

https://www.cerebras.ai/blog/openai-gpt-oss-120b-runs-fastest-on-cerebras

48•samspenc•3mo ago

Comments

freak42•3mo ago

I absolutely hate it, when a website says "try this" and after you went through the trouble of weiting something comes up with a sign up link first. Makes me leave instantly to never come back.

schappim•3mo ago

I was doing a demo to my colleagues and had the above.

Alifatisk•3mo ago

Same with groq.com, there is a "try this", and after you enter the prompt, it asks you to sign in. Closed the page.

traceroute66•3mo ago

Headline at the top of the Cerebras page linked to by the OP "Cerebras Raises $1.1B Series G at $8.1B Valuation".

If you're going after the AI money gravy train then you need to wave the "we have $n registered users" carrot on your PPT slides for the investors because registered user == monetization opportunity.

I'm not defending it. I hate being forced to register for shit when I just want to try it or use the free tier.

But it is what it is.

Saline9515•3mo ago

Well if they give it out for free (aka they pay for it), asking you to register is a reasonable ask. It's not a public service funded by taxpayers.

freak42•3mo ago

Yes they can ask, but do it at the beginning not the end of the process, this is a dark pattern and fucking annoying.

magackame•3mo ago

Anyone remember those online psychological tests where you spend an hour on one and in the end you need to pay up to get the result?)))

traceroute66•3mo ago

> do it at the beginning not the end

Exactly this.

If you present me with a form and a submit button then I expect the input to go through and a result to be presented.

If you don't want to present me with results before login, then put the form behind the wall too.

Simple.

traceroute66•3mo ago

> Well if they give it out for free (aka they pay for it), asking you to register is a reasonable ask

They have other options... rate limiting, serving (more) quantized to non-registered etc. etc.

Saline9515•3mo ago

Those options are still not free. And giving a degraded version of your product to free users is a bad way to acquire clients.

cyanydeez•3mo ago

Right, being proud of your money making is not something I consider a consumer focused product unless that customer is other moneyseeking orga, which like cancer, often ends up in a bubble.

anonym29•3mo ago

This is like declaring that a Ferrari dealership offering you a free test drive in a million dollar art exhibit on wheels is evil for asking for your phone number before handing you the keys.

If this was some beat-to-hell, high-mileage used economy car, sure, that would be a pain in the ass, and not worth it. But it's a mistake to place Cerebras into that mental bucket.

You don't even need to use real information to create an account. Just grab a temp-mail disposable address and sign up as fred flintstone or mickey mouse.

If you're a heavy LLM inference user (i.e. if you've ever paid for a $200/mo sub from any of the big AI labs), I can damn near guarantee you will not regret trying out Cerebras.

freak42•3mo ago

You didn't get my point at all.

rpdillon•3mo ago

Would your expectations be more aligned if it's said "free trial"? That might create an expectation of a sign up where "try this" might not.

moralestapia•3mo ago

Off topic but related.

A week ago I went to a launch party for a product that's supposed to "revolutionize design" (a web app w/ an OAI prompt).

No demo, only like two pictures of the actual product. Founder spent like half an hour giving a speech about the future, etc...

"All of you here will get access to it in a couple weeks."

Couple weeks go by ... I "get access". It's a .dmg, 1) What, I open it, it's not even an app, it's an installer ..., I install it, the app opens up and it's a giant red button that takes you to a website to create an account ...

These guys are completely lost.

petesergeant•3mo ago

It’s an absolute beast. I run it via OpenRouter, where I have Groq and Cerebras as the providers. Cheap enough as to be almost free, strong performance, and lightning fast.

jsheard•3mo ago

Cheap enough for now, but of all the companies selling inference at a loss, Cerebras and Groq are probably losing the most per token. Their hardware is ungodly expensive and its reliance on huge amounts of SRAM bottlenecks how much cheaper it can get, since SRAM density is improving at a snails pace at this point.

petesergeant•3mo ago

Not doubting you but anything to back that up? Happy enough to burn VC money until someone shows up who can run it without losing money, either way.

rajman187•3mo ago

They’ve filed a S1 [1] last year when attempting to go public. It showed something like a $60M+ loss for the first 6 months of 2024. The IPO didn’t happen because the CEO’s past included some financial missteps and the banks didn’t want to deal with this. At the time the majority of their revenue came from a single source in Abu Dhabi, as well

[1] https://www.sec.gov/Archives/edgar/data/2021728/000162828024...

petesergeant•3mo ago

> the majority of their revenue came from a single source in Abu Dhabi, as well

I live in UAE, whose continuing enthusiasm in AI investment stretches well beyond short-term profit, so having AD on-board seems like a plus not a minus. I'm sure there are specific exceptions, but generally Emirati money has seemed like smart money.

rpdillon•3mo ago

You're pointing out a bunch of high capex costs (hardware, SRAM), but then concluding that their opEx is greater than their revenue on a per unit basis. Are they really losing money on every token? It seems that using hardware acceleration would decrease inference costs and they could make it up on unit economics over time.

But I'm just reasoning from first principles. I don't have any specific data about them.

aurareturn•3mo ago

  It seems that using hardware acceleration would decrease inference costs and they could make it up on unit economics over time.

Nvidia GPUs are accelerators too. The reason they can do this so fast is because they're storing entire models in SRAM.

rpdillon•2mo ago

There are degrees of acceleration. My understanding, limited as it is, is that groq and cerebras are using highly optimized acceleration to achieve their token generation rates, far beyond that in a regular GPU, and this leads to lower costs per token.

Is this incorrect?

aurareturn•2mo ago

Yes, they're called ASICs on Grog. But Cerebras has more general cores that can do more complex things. Inference is mostly limited by bandwidth though.

7thpower•3mo ago

Switching costs are low, so if that happens we’ll just switch.

KronisLV•3mo ago

The Cerebras GML-4.6 post might also be of (some?/more?) interest to the people here, since it's more useful for programming: https://news.ycombinator.com/item?id=45852751

I don't think that this is a dupe or anything and 3000 t/s is really cool, the other post just has more discussion of Cerebras and people's experiences with using GLM 4.6 for software development.

sunpazed•3mo ago

This is really impressive. At these speeds, it’s possible to run agents with multi-tool turns within seconds. Consider it a feature rich, “non-deterministic API” for your platform or business.

drewbitt•3mo ago

It's a decent general model too - I have it plugged up in llm and raycast since August at great speeds. I wish Cerebras would do MiniMax M2 which should be an upgrade and replacement if it was just faster. It would never be as fast as gpt-oss-120 though

iFire•3mo ago

Does anyone know how much one system costs?

Queueing Theory v2: DORA metrics, queue-of-queues, chi-alpha-beta-sigma notation

Show HN: Hibana – choreography-first protocol safety for Rust

Haniri: A live autonomous world where AI agents survive or collapse

GPT-5.3-Codex System Card [pdf]

Atlas: Manage your database schema as code

Geist Pixel

Show HN: MCP to get latest dependency package and tool versions

The better you get at something, the harder it becomes to do

Show HN: WP Float – Archive WordPress blogs to free static hosting

Show HN: I Hacked My Family's Meal Planning with an App

Sony BMG copy protection rootkit scandal

The Future of Systems

NASA now allowing astronauts to bring their smartphones on space missions

Claude Code Is the Inflection Point

Show HN: MicroClaw – Agentic AI Assistant for Telegram, Built in Rust

Show HN: Omni-BLAS – 4x faster matrix multiplication via Monte Carlo sampling

The AI-Ready Software Developer: Conclusion – Same Game, Different Dice

AI Agent Automates Google Stock Analysis from Financial Reports

Voxtral Realtime 4B Pure C Implementation

I Was Trapped in Chinese Mafia Crypto Slavery [video]

U.S. CBP Reported Employee Arrests (FY2020 – FYTD)

Show HN: I built a free UCP checker – see if AI agents can find your store

Show HN: SVGV – A Real-Time Vector Video Format for Budget Hardware

Study of 150 developers shows AI generated code no harder to maintain long term

Spotify now requires premium accounts for developer mode API access

When Albert Einstein Moved to Princeton

Agents.md as a Dark Signal

System time, clocks, and their syncing in macOS

McCLIM and 7GUIs – Part 1: The Counter

So whats the next word, then? Almost-no-math intro to transformer models

Queueing Theory v2: DORA metrics, queue-of-queues, chi-alpha-beta-sigma notation

Show HN: Hibana – choreography-first protocol safety for Rust

Haniri: A live autonomous world where AI agents survive or collapse

GPT-5.3-Codex System Card [pdf]

Atlas: Manage your database schema as code

Geist Pixel

Show HN: MCP to get latest dependency package and tool versions

The better you get at something, the harder it becomes to do

Show HN: WP Float – Archive WordPress blogs to free static hosting

Show HN: I Hacked My Family's Meal Planning with an App

Sony BMG copy protection rootkit scandal

The Future of Systems

NASA now allowing astronauts to bring their smartphones on space missions

Claude Code Is the Inflection Point

Show HN: MicroClaw – Agentic AI Assistant for Telegram, Built in Rust

Show HN: Omni-BLAS – 4x faster matrix multiplication via Monte Carlo sampling

The AI-Ready Software Developer: Conclusion – Same Game, Different Dice

AI Agent Automates Google Stock Analysis from Financial Reports

Voxtral Realtime 4B Pure C Implementation

I Was Trapped in Chinese Mafia Crypto Slavery [video]

U.S. CBP Reported Employee Arrests (FY2020 – FYTD)

Show HN: I built a free UCP checker – see if AI agents can find your store

Show HN: SVGV – A Real-Time Vector Video Format for Budget Hardware

Study of 150 developers shows AI generated code no harder to maintain long term

Spotify now requires premium accounts for developer mode API access

When Albert Einstein Moved to Princeton

Agents.md as a Dark Signal

System time, clocks, and their syncing in macOS

McCLIM and 7GUIs – Part 1: The Counter

So whats the next word, then? Almost-no-math intro to transformer models

GPT-OSS 120B Runs at 3000 tokens/sec on Cerebras

Comments