The relevant question is how easy is to setup 56x B200s in a cluster, are we talking about a adhoc infra team job or clicking a button on AWS?
So they've invested around $2.5M-3M into hardware + monthly operating costs (electricity, updates, cooling). I wonder how much did they spend on tokens to decide to make such a switch. I mean if they used to spend $500k-1M per month just on tokens then it would pay off in a year or so, assuming their infra will be stable and efficient.
We'll see if that's a trend for mid-big companies or just an experiment.
All that math is assuming that it's all self-hosted in their data center or so and not paying to a cloud provider, which will be probably more expensive.
The slice of the pie where frontier models are required is just gonna be smaller and smaller. And at point where AI wrote docs, tests, and all the easy and medium difficulty functions you are already saving a ton per employee
Yes, margins will go down a bit but the frontier labs still have some leverage.
At the moment at least Nvidia doesn't have a incentive to release a new product because it's sold out as it is and probably doesn't want to plunge their customers in depth with outdated products?
Are you assuming that they bought the hardware? because they could also be renting the metal off a provider, in which case they either pay by the hour or have a yearly commitment to save on costs.
But the goal posts will move and I think frontier labs will find new areas that have a need for absolute frontier.
Guess who else has that? The Chinese companies developing open weight models. In fact, the vast majority of AI science comes from China. They might still be playing catch up in some areas, but they're also leading in others (famously DeepSeek's inference architecture as an example).
There is IMO a non-trivial chance those labs will eventually surpass the US ones and become themselves "frontier", and that may happen sooner than a lot of people expect.
Of course, it still remains to be seen if they will keep publishing the weights at that point.
At some point, they'll stop releasing models for free and will need a license.
I mean if they don't then the US companies could just release their weights and chop the Chinese off at the knee's at that point (reversing what is currently happening).
Opening the weights isn't just a technical thing, it's a geopolitical decision which likely in part comes from the top of the CCCP (by either request or just implicitly allowing it).
https://jacquesmattheij.com/local-inference-with-large-langu...
If you have more questions feel free to mail or ask here in the thread and I'll do my best to answer.
The hardware related parts are near the end of the article.
The last cause might be a straight up blocker - certain financial regulations straight up prevent you from giving third parties to your data (esp outside national/EU jurisdiction), no matter what documents you sign.
Time for the daily reminder that ZDR is a pinky-promise.
ZDR is a contract clause. That means:
- It is subject to the rest of the contract, i.e. the good old "subject to lawful request or court order" clauses
- It is subject to random provider exceptions, e.g. see Anthropic and Fable.
- You have no way of verifying it beyond "trust me bro".
- Run your own hardware (obviously, the ideal situation)
- If you can't run your own hardware, use confidential computing which runs the AI in secure enclave, e.g. Privatemode[1] (Germany) or Tinfoil[2] (US)
They are less good on ambiguous prompts like 'fix all the problems' (which to my chagrin is a very common prompt it turns out..)
To give one example: I'm doing a lot of stuff that involves hardware. When settings things up for fixing a bug I have to make sure to include that the hardware has been tested and works well. If I don't do that the model will almost certainly spend a ridiculous amount of time speculating about all the different ways in which the hardware could be broken (including theorizing about chip mask errors and all kinds of other obscure and rarely seen in practice defects). It's going to end up telling itself that for sure the 500 lines it just produced are absolutely perfect and the device that is on the i2c bus is broken in a way that it just really can't progress with solving the issue because there is nothing wrong with the code.
These kinds of blind-spots are funny when they happen once in a month but when they happen regularly it begins to be irritating. Adding a line or two to the prompt will take care of the issue. Frontier models tend to be better at this sort of thing.
At the same time, most of their junior devwork can be replaced completely by AI - so they really should start to learn how to work with it. I recommended everybody to get a private 20€/month claude subscription and use it for toy projects. I have younger devs (early 20s) in the team that told me they won't spend private money, or simply have no usecase in their private life for AI. We talk about software engineers here, and IMHO most of them are in denial of the fact that their skills have mostly become worthless if they don't start to pivot to something valueable.
On both frontiers and open. And lately the gap has been closing so if anything yeeting Anthropic etc is getting more viable over time
And they will keep it to themselves to extract the value for as long as possible.
swyx•1h ago