I don't understand the point they are trying to make.
It's very often (always?) the case that something general also solves particular problems.
A sorting algorithm is an implementation of min()
A parser also is a syntax checker.
A route planner is a reachability checker.
A computer algebra system is a basic arithmetic calculator.
A general constraint solver is a Soduku hint maker.
It's true that LLM output can be used as an input to another classifier, this is also true of any classifier. The improvement on top of the straight LLM classification is relatively small, and I would argue that working on the prompt or just including in the prompt for the LLM what features might be useful to consider would likely work even better.
Fundamentally I read this article as: We want to build a simpler, dumbed down clone of Mathematica, so we cobbled together the following pieces... We also needed a way to do arithmetic, so we also include a copy of Mathematica to do basic arithmetic.
michi883•14m ago
I took the point as: don't make the LLM the classifier. Use it to turn messy input into useful features, then let a normal model make the actual decision. That gives you thresholds/calibration you can inspect.
What I'm not sure about is how stable those features are when you switch the underlying LLM or model version.
softwaredoug•4m ago
In my work on LLM as a judge, I prefer to use LLM decisions as features in a downstream classic ML model for the final decision. It works really well
ltbarcly3•17m ago
It's very often (always?) the case that something general also solves particular problems.
It's true that LLM output can be used as an input to another classifier, this is also true of any classifier. The improvement on top of the straight LLM classification is relatively small, and I would argue that working on the prompt or just including in the prompt for the LLM what features might be useful to consider would likely work even better.Fundamentally I read this article as: We want to build a simpler, dumbed down clone of Mathematica, so we cobbled together the following pieces... We also needed a way to do arithmetic, so we also include a copy of Mathematica to do basic arithmetic.
michi883•14m ago
What I'm not sure about is how stable those features are when you switch the underlying LLM or model version.