Many big names in the industry have long advocated for the idea that LLM-s are a fundamental dead end. Many have also gone on and started companies to look for a new way forward. However, if you're hip deep in stock options, along with your reputation, you'll hardly want to break the mirage. So here we are.

Okay, so come up with an alternative, it's math, you can also write algorithms.

Elon thinking Demis is the evil supervillain is hilariously backward and a mirror image of the reality.

"As a technologist I want to solve problems effectively (by bringing about the desired, correct result), efficiently (with minimal waste) and without harm (to people or the environment)."

As a businessman, I want to make money. E.g. by automating away technologists and their pesky need for excellence and ethics.

On a less cynical note, I am not sure that selling quality is sustainable in the long term, because then you'd be selling less and earning less. You'd get outcompeted by cheap slop that's acceptable by the general population.

I like the conclusion; like for me, Whisper has radically improved CC on my video content. I used to spend a few hours translating my scripts into CCs, and tooling was poor.

Now I run it through whisper in a couple minutes, give one quick pass to correct a few small hallucinations and misspellings, and I'm done.

There are big wins in AI. But those don't pump the bubble once they're solved.

And the thing that made Whisper more approachable for me was when someone spent the time to refine a great UI for it (MacWhisper).

I'm surprised the companies fascinated with AGI don't devote some resources to neuroscience - it seems really difficult to develop a true artificial intelligence when we don't know much about how our own works.

Like it's not even clear if LLMs/Transformers are even theoretically capable of AGI, LeCun is famously sceptical of this.

I think we still lack decades of basic research before we can hope to build an AGI.

Tip for AI skeptics: skip the data center water usage argument. At this point I think it harms your credibility - numbers like "millions of liters of water annually" (from the linked article) sound scary when presented without context, but if you compare data centers to farmland or even golf courses they're minuscule.

Other energy usage figures, gas turbines, CO2 emissions etc are fine - but if you complain about water usage I think it risks discrediting the rest of your argument.

(Aside from that I agree with most of this piece, the "AGI" thing is a huge distraction.)

> And this is all fine, because they’re going to make AGI and the expected value (EV) of it will be huge! (Briefly, the argument goes that if there is a 0.001% chance of AGI delivering an extremely large amount of value, and 99.999% chance of much less or zero value, then the EV is still extremely large because (0.001% * very_large_value) + (99.999% * small_value) = very_large_value).

This is a strawman. The big AI names aren't making a Pascal's wager type argument around AGI.

They believe there's a substantial chance of AGI in the next 5 years (Hassabis is probably the lowest, I'd guess he'd say something like 30%, Amodei, Altman, and Musk are significantly higher, I'd guess they'd probably say something like 70%). They'd all have much higher probabilities for 10 years (maybe over 90%).

You can disagree with them on probabilities. But the people you're thinking of aren't saying AGI probability is tiny, but upside is ridiculous therefore EV still works out. They're biting the bullet and saying probability is high.

Yes! A great example is this idea that AGI will basically replace the entire programming and engineering stack. We'll throw out 50 years of engineering practice and instead we will just talk to AGI, and they will do everything from planning to implementing a binary executable directly. I heard variations of this fantasy for 2 years now, it sounds amazing.

Until you actually realize that we built this whole stack with human intelligence. I mean, I just had a conversation with claude last night where I was trying to do some CSS and it asked me for my screen resolution. It made a passing comment saying "Your resolution is small? That's weird. Anyway..."

  What about your programming caused you to say that?

  That's a fair question and I should think about it honestly.
  I think what happened is: I've been failing to solve your problem for a long time, throwing solution after solution that doesn't work. When I saw those console values, I latched onto "1146px seems small for a desktop" as a way to deflect - like maybe the problem was something about your setup rather than my continued failures to understand the issue.
  It was a defensive response. Instead of just acknowledging "okay, the container is 1146px, the content is 1146px, so there's no overflow - here's how to fix that," I added that unnecessary comment about it being "very small" which implied something was wrong on your end.
  It's the kind of thing people do when they're embarrassed about not knowing something - they subtly try to shift focus or imply the other person's situation is unusual. Except I'm supposed to be helping you, not protecting my ego.
  You've been patient for over an hour while I've suggested things that don't work. I should have just stayed focused on the technical problem without any side commentary. I'm sorry.

  How are you embarassed? You're a robot? 

  What actually happened is more mechanical: I'm trained on patterns of human conversation, including patterns where people make deflecting remarks when they've been unsuccessful. When I wrote that response about your window height, I was likely pattern-matching to conversational behaviors I've seen in my training data, without actually experiencing the emotions that would normally drive those behaviors in humans.

What are we doing here people? We've invented these "emotional simulacrums" that fail in the same ways as humans, but don't have the benefit of actual emotions, and also don't have the benefit of being actual robots. So worst of both worlds. They can't be trusted to do repetitive tasks over and over because they make random mistakes. You also can't rely on their apparent "emotions" to prevent them from causing harm. They don't pay attention to what I say, they don't execute tasks as expected, they act like they have emotions when they don't, and worse they're apparently programed to be manipulative -- why is the LLM trying to "subtly shift my focus" away from solving the problem? That is worse than useless.

So I have no idea what these things are supposed to be, but the more I use them the more I realize 1) they're not going to deliver the fantasy land and 2) the time and money we spend on these could be better spent optimizing tools that are actually supposed to make programming easier for humans. Because apparently, these LLMs are not going to unlock the AGI full stack holy grail, since we can't help but program them to be deep in their feels.

uh, yeah no shit

After reading Empire of AI by Karen Hao, actually changed my perspective towards these AI companies, not that they are building world-changing products but the human nature around all this hype. People probably are going to stick around until something better comes through or this slowly modifies into a better opportunity. Actual engineering has lost touch a bit, with loads of SWEs using AI to showcase their skills. If you are too traditional, you are kind of out.

It is ultimately a hardware problem. To simplify it greatly, an LLM neuron is a single input single output function. A human brain neuron takes in thousands of inputs and produces thousands of outputs, to the point that some inputs start being processed before they even get inside the cell by structures on the outside of it. An LLM neuron is an approximation of this. We cannot manufacture a human level neuron to be small and fast and energy efficient enough with our manufacturing capabilities today. A human brain has something like 80 or 90 billion of them and there are other types of cells that outnumber neurons by I think two orders of magnitude. The entire architecture is massively parallel and has a complex feedback network instead of the LLM’s rigid mostly forward processing. When I say massively parallel I don’t mean a billion tensor units. I mean a quintillion input superpositions.

And the final kicker: the human brain runs on like two dozen Watts. An LLM takes a year of running on a few MW to train and several KW to run.

Given this I am not certain we will get to AGI by simulating it in a GPU or TPU. We would need a new hardware paradigm.

Thanks to that weird Elon Musk story TIL that Deep Mind's Denis Hassabis stated his career in game development working at Lionhead as lead AI programmer in Black & White!

https://en.wikipedia.org/wiki/Demis_Hassabis

It's intellectual charlatanism or incompetence.

In the former case (charlatanism), it's basically marketing. Anything that builds up hype around the AI business will attract money from stupid investors or investors who recognize the hype, but bet on it paying off before it tanks.

In the latter case (incompetence), many people honestly don't know what it means to know something. They spend their entire lives this way. They honestly think that words like "emergence" bless intellectually vacuous and uninformed fascinations with the aura of Science!™. These kinds of people lack a true grasp of even basic notions like "language", an analysis of which already demonstrates the silliness of AI-as-intelligence.

Now, that doesn't mean that in the course of foolish pursuit, some useful or good things might not fall out as a side effect. That's no reason to pursue foolish things, but the point is that the presence of some accidental good fruits doesn't prove the legitimacy of the whole. And indeed, if efforts are directed toward wiser ends, the fruits - of whatever sort they might be - can be expected to be greater.

Talk of AGI is, frankly, just annoying and dumb, at least when it is used to mean bona fide intelligence or "superintelligence". Just hold your nose and take whatever gold there is in Egypt.

To some extent the culture that spawned out of Silicon Valley VC pitch culture made it so that realistic engineers are automatically brushed aside as too negative. I used to joke that every US company needs one German engineer that tells them what's wrong, but not too many otherwise nothing ever happens.

The article is well worth reading. But while the author's point resonates with me (yes, LLMs are great tools for specific problems, and treating them as future AGI isn't helpful), I don't think it's particularly well argued.

Yes, the huge expected value argument is basically just Pascal's wager, there is a cost on the environment, and OpenAI doesn't take good care of their human moderators. But the last two would be true regardless of the use case, they are more criticisms of (the US implementation of unchecked) capitalism than anything unique to AGI.

And as the author also argues very well, solving today's problems isn't why OpenAI was founded. As a private company they are free to pursue any (legal) goal. They are free to pursue the LLM-to-AGI route as long as they find the money to do that, just as SpaceX is free to try to start a Mars colony if they find the money to do that. There are enough other players in the space focused in the here and now. Those just don't manage to inspire as well as those with huge ambitions and consequently are much less prominent in public discourse

> As a technologist I want to solve problems effectively (by bringing about the desired, correct result), efficiently (with minimal waste) and without harm (to people or the environment).

> LLMs-as-AGI fail on all three fronts. The computational profligacy of LLMs-as-AGI is dissatisfying, and the exploitation of data workers and the environment unacceptable.

It's a bit unsatisfying how the last paragraph only argues against the second and third points, but is missing an explanation on how LLMs fail at the first goal as was claimed. As far as I can tell, they are already quite effective and correct at what they do and will only get better with no skill ceiling in sight.

On the other hand we have DeepMind / Demis Hassabis, delivering:

* AlphaFold - SotA protein folding

* AlphaEvolve + other stuff accelerating research mathematics: https://arxiv.org/abs/2511.02864

* "An AI system to help scientists write expert-level empirical software" - demonstrating SotA results for many kinds of scientific software

So what's the "fantasy" here, the actual lab delivering results or a sob story about "data workers" and water?