Other aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).
i've always wanted cost per prompt, but even that has too much variation.
If the benchmarks are non-predictive, well, you can't use them for much of anything, which is of course a recurring problem with every benchmark ever.
The uncertainty of how to use this vastly vastly outweighs the price in a data centre - so buckle up, buy enoughbGPUs to experiment at a known cost and one day you will find the approach that gives you 10x returns - at that point pay any price per token but not till then
People don't like to hear this but the open models just aren't good for end to end agentic workflows.
There are some very very good small open models that can excel in certain finite bounded tasks, but the foundational models are essential to building out agentic pipelines that actually work.
Stuff like the latest DeepSeek, Kimchi and GLM are used and loved by many people. It's not using an open model that is difficult: it's having the hardware allowing to do so. It's pricey and require technical skills.
That's why most people who are using (excellent btw) open-weight models are just renting compute online.
Also risking it all for some distilled models is a recipe for disaster.
We've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing)
(In this particular case Deepseek was way slower than GPT 5.5 but I think that's because it installed Libreoffice half-way through the task!)
But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.
If yours is the only request in the batch it will cost them one full pass through the model.
If yours is one of 1024 inputs in the batch the per token cost is 1024x less.
For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher
Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.
But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.
Well that's the problem with these black boxes. You really have no idea beforehand how many tokens a given task is going to take. There's simply too many variables involved. It's therefore only natural for people to assume "the cheaper and older model is probably going to cost less overall to use than the newer, more expensive one."
Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.
So even though it had better generated tok/s, because so many more were generated, the clock time was longer.
And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)
Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it should cost less.
Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.
Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.
This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.
I’ve wanted a fast model to generate commit messages. No idea what that would be, but it doesn’t have to pass the SWE benchmarks very well. Just well enough that it understands the codebase.
I'm sure there are degenerate cases, but I'd bet a relatively small model could do the job.
I want a model that generates commit messages fast. Currently I have to wait up to a minute or two. That model doesn’t need to score very highly on SWE benchmarks, just highly enough that it can write out a good enough message in a few seconds. If you tested it on ${current top tier benchmark} you’d think it’s way too costly when in fact it’s the best tradeoff.
This is (apparently) the conceit of SteveYegge / GasTown - no model can cope unassisted so chunk it up, run it and if it falls over remember the exact place and restart, merging it all in
But that’s not my point.
I believe that software is a new form of literacy and just as all Companies and societies are literate now, in the future (tm) companies will run exclusively on software - AI developed software and those who go all out will have the sort of advantages the Catholic Church had over .. guilds?
Anyhow, that’s me being AI optimist. But writing the code is going to be a small part of that transition - almost everything to do with LLMs that is claimed amazing (Computer vision is something else) - almost everything people say we need an LLM is stuff you could have done three years ago but your internal politics just would not let you. Oh look we can see if our policies are being met (you could have written the policies in code and solved the whole problem)
Im struggling to get it out but - almost everything AI is proposed for is stuff a well run engineering firm coukd have taken on. A software literate firm could have done without AI is where firms are hoping AI will Get them
Imagine how far ahead real software literate firms will be - as long as they don’t burn their runway in tokens
Which is why, the right play imo is still buy in-house as much as possible, engineer around the problems and explore the phase space at marginal Cost.
Then and only then think frontier models.
There are probably 100 competing versions what this phrase might encapsulate. Could you elaborate more on which version you are using exactly?
My experience is that frontier models are only marginally better and not close to the cost/value of the open models which are anywhere from 10-100x cheaper. Perhaps I'm not doing "end to end agentic workflows?"
I'm still using Claude at work (they're the only approved provider), but wow are the smaller models starting to SMOKE the big ones. At this point, all I'd consider paying out of my own pocket for is the lowest-limit Anthropic/GPT plan to get a big model as the Steward, but I wouldn't pay for ANY of the Anthropic models as the workers who do all the work. And as time passes, I don't know if I'd even do that; the open models are serving SO well.
Love to hear more about how you structure the orchestrator etc
zeroonetwothree•1h ago