For a long time it has been a win to do clustering, retrieval, ranking, classification, anomaly detection and other non-generative tasks with local models from the BERT family. These are so much better than all the things I tried from 2000-2020. ModernBERT for feature generation plus classical algorithms from scikit-learn works so well, plus scikit-learn has great tools for eval, model selection, probability calibration and all the other things that make the difference between "write another arXiv paper" vs "deploy a product into production"

If you read the splogs you'd think it is all about the "zero-shot" scenario where you write a prompt asking what decision you want the model to make. Yeah, it is amazing that it does so well with so little input, but side by side "many-shot" models that learn a decision surface in hyperspace beat the pants off "few-shot" models that look at nearest neighbors and "zero-shot" models that do whatever they do. It's not even close.

The reckoning for generative AI may near and it might be like something right out of "Where's your Ed at?" When people see honest prices for current models I think a lot of people might decide the juice isn't worth the squeeze. The story in the next few year is not going to be bigger models that only Elon Musk models but a relentless fight to get costs down.

Local is best when you're iterating fast on prompt templates or need to run the same task hundreds of times—API costs add up quickly there.