Agree fully.

The smaller models will only get better which push out the usefulness of older gpus.

Two issues with this. One, it's profitable assuming you just keep serving the same model forever, which is not realistic in this market. A given model has a shelf-life, which these days is measured in months, not years. Which means that trying to separate the cost of training the model from the cost of serving it doesn't make much business sense. And two, for providers that provide inference only via open weight models, the margins quickly move to commoditization. The "someday" when frontier model providers can enjoy their current high inference margins without the burden of significant training costs is never going to arrive.

Profitable on a per token basis is meaningless. Ed Zitron doesn’t argue that it is impossible to offer profitable inference, he argues that the business as it stands today is deeply unprofitable and only getting worse because it isn’t a high-margin inference business.

Play out the most likely pessimist’s scenario: LLMs are marginally useful but frontier models are overkill so businesses just use dirt cheap open weight models on their own hardware and/or they rent hardware instead of paying per token. Then what for OpenAI and Anthropic?

OpenAI’s business collapses if customers are happy with an LLM that costs $0.10 per million tokens even if it only costs OpenAI $0.05 in inference per million tokens. The insane bonkers claim from Garry Tan that in 2 years we will be using 90,000x as many tokens as today is… well, obviously not true.

The fixed costs that OpenAI and Anthropic have created need inference demand far beyond what is plausible.