Did you validate this by running a A/B test? Main question is were you able to classify back into your known categories correctly all the time, or did the errors compound from the llm hallucination plus embedding search

Eh, maybe you should keep both paths. When LLMs eventually crawl the site to feed back to agentic shoppers, maybe they logically follow the more truncated less decorated path.

I was in a project where we sent the whole taxonomy every request, 40k tokens + one article, ”plz classify”. This was before structured outputs. It was extremely expensive and still hallucinated. Good ol’ days.

Smart! I've done the same trick for resolving extracted intents to selection.

But if accuracy matters, you can't rely on embedding sort to get a closet match. With a real test set they usually don't hold up under scrutiny.

Everything in AI is like this. You get an idea, try it once or twice, "LGTM" and you ship. Then it never survives contact reality.

Embedding sort gives you a better shortlist than the whole list, but you will probably want a heavier model to vet candidates.

I would propose the following, query vector store for 10 closest categories based on a query, feed it to an LLM, in the prompt ask it to produce a single digit 0-9 representing the number of the most appropriate choice. Use plain text prompt, dont inflate token count with JSON. There you go, you just drastically reduced the output pricing.

Additionally you could experiment with a reranker instead of an LLM or after reranking take top-3 results and then feed to LLM as input in order to reduce input token costs.

Isn't search engines quicker than calling a LLM ? It might have a huge impact between a 20ms search engine call and a 2s LLM call for the end user.