The AI implication is fascinating though. We tend to build systems where the language model is simultaneously the language interface, memory, planner, reasoner and controller. Human cognition may be pointing in the opposite architectural direction: separate persistent state, reasoning/control mechanisms and language, then let language be one tool among several.
That distinction seems more interesting to me than simply asking whether LLMs “reason in language.”
Language as an internal conceptualization is still (obviously) essential to logical reasoning. The question of how language and logic relate can, therefore, concern either the internal language of concepts or the expressed language and its grammar, the latter of which is related to logic in that it reflects to some degree the internal language, but it is not identical with it.
Quantity is not necessarily a problem: it feels a lot like the same tradeoff as if you need to picture a lot of apples in an image of fixed resolution, you can make them smaller/less detailed just by, loosely, "allocating fewer pixels per apple", rather than needing the same detail for each that you'd need if you're trying to zoom in depth into one apple.
However, if I have a situation where I have to picture a lot of very detailed things of irreducible complexity (the details matter) that need to be placed in a specific configuration (the ordering matters), then it gets harder. Statistics was so intuitive for me that I would read the equations as visual images in my head because it was throwing out the individual details and dealing with population mechanics as individual objects of study. However, combinatorics was initially very hard for me to learn because it was dealing with very large numbers of hierarchies of orders as well as the specifics of their interrelations, so building intuitive understandings of why the theorems were correct was very difficult.
I have learned to lean on a very internally barebones representation of language/symbolic structures when needing to reason about certain complex topics that require it, but, even then, often when I'm conscious of it, it's almost like I'm visualizing the letters or forming the words as sound in my head. When I'm forming sentences to actually speak or write, it feels like I've got a concept that I've turned into a "rendering" that I'm trying to communicate via language substructures. To do so, I'll break it down into component parts and just start describing the substructures as if I'm perceiving it and describing it live in small parts. It feels like the learned skill required for speaking a language for me is learning how to decompose and reorder these objects into chunks that are expressible within the vocabulary and grammar of the target language and finding the correct words&patterns that represent and order these chunks most efficiently for effective communication. Almost feels like a SAT problem or a language diffusive process (and likely is, to some degree, related to those processes or similar under the hood), but, it actually qualitatively feels that way to me, consciously/experientially. Language feels almost bolted onto my experience rather than the primary mode of thought, even if I can think in it if needed.
LLMs might be a step towards AGI (assuming that's possible) but the prevailing belief that LLMs and human brains (the latter being the most complex structures in the universe that we know of) are already functionally equivalent in every way that matters seems more like an article of faith to me than fact.
Also, regarding LLMs, 2 years ago another famous expert in the field was adamant that LLMs can't plan, can't do math and they can't work at long contexts. Obviously they can do all of that now. So, really, no one knows.
myelinscience•3d ago
NitpickLawyer•40m ago
Are they looking at blood flow in areas to map "language network" and other stuff? I remember a few recent papers that found that a) blood flow doesn't necessarily mean "active area" and b) two brands of fmri found completely different "areas" activating for the same patient + same puzzle.
chermi•14m ago
fMRI is one of the best tools we have, but that doesn't mean it's good. fMRI shit should always be taken with spoon of salt. source: a bunch of grad school friends in neuroscience and psychology a few years ago.
"on two patients whose language networks were destroyed by stroke, plus fMRI on healthy volunteers solving the same puzzles silently. Both lines of evidence point the same way: the language network sits out during reasoning." that doesn't mean they didn't learn the meta-ability of how to solve such things without language. You can learn something with language, move it to a separate "database" in some abstracted form then pull from that "database" from a non-language component. In fact I would think the language-isn't-thinking people would favor an explanation where the thinking ability is stored independently of language. I guess my point is that no "language" component being lit up during a specific reasoning task doesn't exclude the possibility we learned that specific reasoning algorithm (for lack of a better word) via language + other things.
TL;DR maybe this specific "logic" ability is a learned-via-language+ component of the brain. Language not being active during that puzzle could just mean language+ generated that component and stored it in a more abstract form than language.