I can definitely confirm the last part. Every time I read the output of an LLM, I picture a person talking to me.

It seems the people who use it and see good results are busy doing, and the rest are either just expressing their opinions as facts and trying to tell others what to think and how they are wrong. I pay zero attention to those people they have no skin in the game.

This seems highly dubious. You can't just memorise the form of mathematical proofs and then produce a valid one by feeding plausible looking BS into a verifier until it works. That's like saying a cargo cult will build a working airport if it just tries enough times.

Dijkstra said[1] "... the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim."

I don't see a clear demarcation of the things that only "reasoning" can accomplish and can't be approximated or imitated by other methods, and so I think the question is simply not meaningful or relevant.

[1] https://www.cs.utexas.edu/~EWD/transcriptions/EWD08xx/EWD867...

This is a brillaint way to simulate reasoning, but its likely not how we reason ... simply how we store reasoning in writing.

Its useful if you know how to use it, its dangerous if you think its more than that.

But tl;dr it can (since its uncompressing our lingusticially stored reasoning from books) arrive at reasoning a DIFFERENT way than our brains did... and this isn't right or wrong.

Where it diverges is when it must move beyond the text or even the synthetic possible text of all vector spaces combined (aka novel territory) and it can't conjecture or test those outcomes well. But to be fair, neither can MOST humans.

Perhaps you've reified it too much.

If that's the case, AI-generated code could also operate on its own logic. Right now, programming is still done by humans, not machines, which creates a mismatch. But maybe the true machine-generated code could be much closer to the machine itself.

When you code with AI, there's a subtle mismatch with human-written code. It's like human code is a clean ORM layer, while machine code is raw SQL queries—there's that kind of subtle impedance mismatch. If we ever reach machine-to-machine code, what would that code even look like? Would it still use classes and methods?

That’s all.

The point of the story was that classifiers can be right for the wrong reasons and almost inevitably are. At least there’s zero guarantee that the reason for making the prediction matches the human or “real” reason why it’s correct.

LLMs are classifiers, there is absolutely no reason to assume they’re any different, regardless of any reasoning tokens they emit. They do what their handler wants to see, that’s all, and that’s what they’re trained to do.

People often take this as a knock against them. It isn’t, it’s just the reality of neural network classifiers. The results speak for themselves and don’t depend on whether they “actually” reason, but all evidence says they don’t, or at least there’s no special reason why they would.

It’s frustrating that anyone who says maybe we shouldn’t base our entire economy on this one thing until we understand it and what’s it’s useful is essentially labeled this way.

  On the other side of the AI-reasoning fence, the disdain seems to be mutual. “These ‘scientific’ papers from last summer — I would put this in big, big air quotes,” said Sébastien Bubeck, a member of OpenAI’s technical staff (and a prominent evangelist for the company’s reasoning models among scientists and mathematicians). He called earlier Apple results critiquing AI reasoning “wrong,” claiming that they were due to a training quirk in models that are now obsolete. “Modern models starting with GPT-5.5 do not suffer from this issue,” he said. “It would be interesting to revisit those results.” (Apple did not make its researchers available for interviews.)

  The “think” part is what OpenAI, for one, is doubling down on. When I asked Bubeck if the splashy unit distance proof was produced with methods outside the LRM’s own chain of thought — perhaps with Lean verifying its results — he seemed to find the question almost nonsensical.

  “It’s not like we’re making a mystery of it,” he said. “We have released the chain of thought. You can just go and look at it. The whole point is that the model is reasoning like a human would. And when humans reason, we don’t use Lean.” Technically, OpenAI released a “rewritten summary” of the model’s chain of thought produced by two human experts using Codex, another OpenAI model. Since 2024, the company has not publicly revealed “raw” chains of thought from its reasoning models, a policy also adopted by Google DeepMind and Anthropic.

[1] https://www.hrw.org/report/2018/01/09/dark-side/secret-origi...

Now as far as the math stuff a quirk of that field is that it can be fully analyzed in token space. Because 2+2 is a matter of definition it doesn’t need empirical testing like biology or subjective social support like a claim about the causes of WWI

So somewhere between the fact that language encodes a lot more ‘concepts’ than we naively may realize, the power of statistical emergence via associations, and what pursuits can be fruitfully done in token space we can get a long way towards ‘intelligence’

It seems intuitive that continuing a correct output sequence is easier than the "discontinuity" of jumping from the input prompt to the output sequence.

1) LLM's written CoT might not always be faithful to the model's real reasoning process (true and important)

2) The "stochastic parrot" hypothesis, which the article reintroduces as "approximate retrieval" - ie, LLMs don't "really reason" at all, they just memorize a lossy encoding of their training data. This obviously raises the question of how LLMs can now routinely solve open mathematical problems, with no solutions in the training data by definition. The article handwaves this with:

"The model doesn’t have to learn or reliably apply a general reasoning process, Kambhampati said; it just has to absorb enough examples of what the steps look like to predictively mimic them on its way to “stitching together” a plausible result that can then be verified."

The problem is that "mimicking" training data to arrive at a "plausible" result gets you an incorrect-but-plausible-sounding "proof" of the Jacobian conjecture, which was famous for humans writing plausible-looking "proofs" that had subtle flaws. You can't disprove the conjecture through sheer luck (search space too large) or "approximate retrieval" (the only thing you'd retrieve are fake "proofs"; far more human effort went into proof than disproof) or by writing something "plausible" that just happens to be correct (Jacobian was famous for "plausible" but wrong); the model must be carrying out mathematical reasoning somehow, by any sane definition of the word, even if it isn't fully reflected in CoT. The article doesn't address this.

> Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.

What’s going on here, for example? But what if this is the path of human reasoning too. You know, have you guys read Peter Thiel’s Antichrist essay? It’s very weird, man. Guy sounds off his rocker entirely.

But he’s super successful, right? Maybe world modeling doesn’t text represent well. By the antichrist maybe he means some notion of the collective voting for distribution of resources without contributing productive capacity and that that ends societies? Or maybe internal world models are just not text serializable effectively.

A thing I’ve recently been enamored of are effective world and coordination models that are not “true”. E.g. a tribe that believes the forest gets angry if they do not hunt united. Lots more like that in Darwin’s Cathedral.

It might seem a bit free association-y but the topic itself is that.

The reasoning tokens behind this comment: https://wiki.roshangeorge.dev/w/Blog/2025-10-12/Word_Magic