(I wrote the article. I'm a longtime HN user. I find that threads here lately have gotten very jumpy-offy -- commenters use a specific article about e.g. icebergs melting to have a conversation about climate change and climate change denial, instead of to talk about the merits of the particular article -- and I was hoping to nudge folks to read the full piece, then comment on specific parts of it. I'm not sure that'll work but figured it's worth a try!)

The first category is what I'd call "the simplifying metaphor", in which it is claimed that AIs are actually "just" something very simple, and therefore do not think.

- "AIs just pick the most likely next token"

- "AI is just a blurry jpeg of the web" (Ted Chiang)

- "AIs are just stochastic parrots"

The problem with all of these is that "just" is doing an awful lot of work. For instance, if AIs "just" pick the most likely next token, it is going to matter a lot _how_ they do that. And one way they could do that is... by thinking.

There are many different stochastic processes that you could use to try to build a chat bot. LLMs are the only one so far that actually works well, and any serious critique has to explain why LLMs work better than (say) Markov chains despite "just" doing the same fundamental thing.

The second category of argument is "AIs are dumb". Here, skeptics claim that because AI fail at task X, they aren't thinking, because any agent capable of thought would be able to do task X. For instance, AIs hallucinate, or AIs fail to follow explicit instructions, and so on.

But this line of argument is also very poor, because we clearly don't want to define "thinking" as "a process by which an agent avoids all mistakes". That would exclude humans as well. It seems we need a theory that splits the universe of intellectual tasks into "those that require thinking" and "those that don't", and then we need to show that AI is good only at the latter, while humans are good at both. But unless I missed it no such theory is forthcoming.