If by some mechanism other than what a gatekeeper would call 'reasoning', a machine produces outputs that approach indistinguishable from 'well reasoned', the argument that it didn't get there by reasoning is, well, not useful at the very least.
In context: just because our LLMs don't have an explicit 'system 2' component doesnt mean it can't have superhuman reasoning
> it can't have superhuman reasoning
no they don't, we still die of cancer, there's no global deployed autonomous driving and food production driven by super intelligent ais and I'm not walking on mars thanks to gravitational elevators
Love to see the forever moving goalposts. One might say they're autonomously moving..
Lots if these arguments are similar to birds saying "Jets don't flap their wings so they aren't even flying."
The arguments about reasoning are even shakier because they usually rely on totally unproven assertions about human reasoning. At least we know birds flap their wings.
Birds existed since forever ago in nature, and they fly by flapping their wings. Then planes got invented, and they fly using a very different mechanism (that doesn't involve flapping wings).
The point made by the grandparent comment: saying "LLMs don't actually reason, because the underlying mechanism they use is different from how humans reason" feels about the same as "planes don't actually fly, because the underlying mechanism they use is different from how birds fly".
Another thing humans did is invent the telescope, the microscope, the transistor, antibiotics, the computer, AI, discovered how to send satellites in space and how to do heart-transplant etc.
I'd say there's still some way to go for AI before we declare humans dumb because they "failed for decades" at solving a few math problems.
Jacobians are essentially derivatives but for matrices.
Stop trying to compare either system to a human and look at it for what it is -
A prediction engine that runs fast enough to brute force problems.
In the case of alpha go its "innovation" was millions of games played against itself. It had bound parameters and strict win conditions.
In the case of LLM's you can deploy 1000's of agents to smash themselves against an idea. The whole hugging face attack is an example of this (1200 agents out of an unknown number chose that path).
There is the old saying about monkeys, typewriters and Shakespeare. Well we have better monkeys who basically follow a derivative of zipfs law (not actually), who use tokens not letters and their goal in many cases is testable (compile, unit, E2E).
I'm not sure I understand this. Do we have evidence humans have an independent set of beliefs not shaped by knowledge and reasoning? If so, where do these come from?
I'm especially confused about a prior statement as a scientist:
> Three shortcomings prevent what chatbots do from qualifying as reasoning (in a way that a scientist might recognize).
How does a set of beliefs help with reasoning?
> Third, while the chains of thought chatbots produce look like deliberation, research has demonstrated that the bots often concoct them after the fact, reaching an answer by one route but reporting another.
We also often do the same as humans.
I relate a lot to what they do. Find words, alignment and suss out follow ups, ons, and outs to the next reasonable conclusion.
Then use that context to bootstrap the next because if you build a powerful conclusion than can reverse itself into its evidentiary context, then every next context step can update its priors.
And so on the turtles flow where like an LLM, THE start of the context disappears over the horizon, but as long as im contexting in disinterested chunks of equal quality, then its not a problem.
But while internal tobeach context you can find reason, as a requisite building block like falling tetris pieces, the whole isnt the sum of its parts.
At least reasoning is not guaranteed.
Would sure be nice though...
The next token prediction is just "the hardware" following the underlying rules. Like the basic set of rules.. in a sense similar to how the "game of life" does not really contain gliders. Gliders are just a self stabilised system that arrises from the simple rules.
Kind of like how a brain is just a bag of molecules. Molecules can't reason either.
My position is that it is not possible.
Seems so to me.
So explain then what you mean by "not possible".
The fact that the output produces one token at a time does not mean that the LLM's internal state is processing just the next token
E.g. "Approach this problem iteratively. As you form a hypothesis, track the confidence you have in various explanations you're considering, what evidence you're weighing to support each, and the unresolved questions you're holding onto. Log all that for later inspection.
Be methodical when evaluating evidence and only accept facts you have verified. At every stage, gauge how much each possible next step resolves uncertainty, and discard options unlikely to advance progress. Divide the functions I described into subagents responsible for each, and coordinate with them as you work."
(I'm definitely one of the bigger "AI" skeptics out there but am nonetheless fascinated by these questions).
However, if the assertion is true, then no amount of prompting can solve it - you cannot explain to a fish how to use a bicycle. Telling an LLM to weigh evidence only works if an LLM can, but isn’t, weighing evidence: if it cannot do so, instructions will generate the appearance of weighing evidence with additional “thought” tokens copying that of reasoning texts, but the output will be equally groundless.
If argument from authority was valid in itself, then LeCun would have killed LLMs like 300 different times now, and yet keeps being wrong.
Folks see Jesus in burnt toast. Monet was a master of exploiting this where what’s really just blotches of color our brains fill into beautifully detailed images.
Our experience with LLMs is no different. Folks believe there is some deeper intelligence there but it’s all still just 1s and 0s on a computer chip. We’re interpreting things happening that simply are not happening.
We are experiencing non-humanoid intelligence without AGI. That is awesome. And we don’t have a clue how to protect ourself from AGI.
Likewise, we intelligent primates have this great system of coordination called market economics that lets us destroy our home planet with our eyes open. That’s what we call intelligence!
(Not 100% personal opinion and deliberately inflated from the I’ve been thinking about.)
You either have to accept that the brain can be described with math (like everything else we have ever known in the universe), or that there is a supernatural phenomenon that exists in the brain.
This is an inescapable conclusion that boils down to "Do you believe magic is real or not?"
Magic is real and you can have your unique special human intelligence.
Magic is not real, and the brain is just another computer crunching numbers.
I think the real question is: how do we define reasoning?
My view is that AI can reason, just differently from humans.
https://www.finextra.com/blogposting/31255/adaptability-as-e...
Interestingly in 2019 this was the author's take. He now appears to be confusing/conflating between LLMs and agents in a way that helps argue his case about "System 1", but his prior view seems more metaphysically robust.
the categories i like to use to describe what llms are capable and incapable of are: instrumental reason, which is reason as a tool for achieving a goal; and objective reason, which is reasoning about which goals are good or bad, or worth pursuing.
llms are, i think, approaching or have achieved better-than-human performance on the former category in a wide variety of applications.
the latter, not so. leaving aside that there are schools which claim (dogmatically, imho) humans don't or can't engage in objective reason, i dont believe llms are structurally capable of it. their goals can only be imposed on them from outside, coming from prompts, implicit value assumptions in training data, loss function, and rlhf. there is something about human interior experience of an objectively existing world that lets is evaluate true/false/good/bad in a way that is unique to humans among other animals.
llms can't do it. i dont just mean on ethical, epistemic, or aesthetic judgements, but even in practical circumstances like the ones engineers encounter. the reason engineers still have to work alongside llms, even though lllms are (imho) far better programmers and technicians, is that even when given a goal, there is always a graph of evaluations that lead to that objective and llms routinely fail to evaluate the tradeoffs and land in states in outcome space that are subtly (or not so subtly) wrong, even though the objective is complete!
forgive typos, i am on mobile.
I think it's an interesting approach but the overall discussion about reasoning is really pedantic. What the author is describing here is one approach out of many, and in my opinion it doesn't cover what humans colloquially think of when they hear reasoning (while the output from a chain of thought sometimes does).
I think an analogy would be helpful. As an LLM, reasoning through text, you wouldn't 'know' the idea of a man separately from, posterior to, the words man in say, French and English. As a human, you WOULD know the idea of man, and you would mean that idea when you say the French or English words for man.
For an LLM, although its approximation of knowledge would lead it to claim it knows they're the same thing, there would be differences in its weights that influence its usage of both the French and English words for man, and which may lead it to conclusions in one language it wouldn't reach in another. Because its knowledge/reasoning is interwoven to its knowledge, not prior to it.
For a given bug one could write a test that prove its existence, this gives the LLM a target that they can actually iterate towards.
rkagerer•39m ago