Hi! I'm an AI engineer at a Series D company. We use LLMs for basically everything: customer responses, classification, routing, clustering, etc. While most of my focus is on developing new features, writing evals for them and monitoring quality in production, I can't help but look for opportunities to improve our current ways of doing things. One of these is model upgrades. It seems that we keep models until OpenAI decides to deprecate them, whereas everyone knows that we could have saved money and made better product by making the switch earlier on. Another adjacent problem is over reliance on LLMs to do everything where as a simpler classifier could have sufficed. So I have a couple of questions for my fellow ai engineers

- When did you last switch models, and what triggered it?

- How did you check the new model didn't regress? Eval suite, replaying real traffic, spot checks, just shipping it?

- Have you ever moved a task to a smaller model? an open source one ? What made you do it?

- Is there anything you run on an LLM that a classifier or plain rules could handle?