Slightly different cohorts.
I'm not really trying to be snarky; I'm trying to point out to you that you're being really vague. And that when you actually get really, really concrete about what we have it ... starts to seem a little less magical than saying "computers that talk and think". Computers that are really quite good at sampling from a distribution of high-likelihood next language tokens based upon complex and long context window is still a pretty incredible thing, but it seems a little less likely to put us all out of a job in the next 10 years.
LLMs may be a stepping stone to AGI. It's impressive tech. But nobody's proven anything like that yet, and you're running on pure faith not facts here.
I'm enjoying the new LLM based tooling a lot, but nothing about it suggests that we're in any way near to AGI because it's very much a one trick pony so far.
When we see generative AI that updates its weights in real time (currently an intractible problem) as part of the feedback loop then things might get very interesting. Until then it's just another tool in the box. CS interns learn.
I would be interested to hear the way that you see. I don't have any problem seeing a huge number of roadblocks to post-scarcity that AI won't solve, but I am open to a different perspective.
Not that I think you're wrong, but come on - make the case!
I have the very unoriginal view that - yes, it's a (huge) bubble but also, just like the dot com bubble, the tevhnology is a big deal - but it's not obvious to see what will stand and fall in the aftermath.
Remember that Sun Microsystems, a very established pre-dot com business, rose to huge heights on the bubble and was then smashed by the fall when it popped. Who's the AI bubble's Sun and who's its Amazon? Place your bets...
Nah, we aren't. There's a reason the output of generative AI is called slop.
Extraordinary claims demand extraordinary evidence. We have machines that talk, which is corollary to nothing.
That's an extremely speculative view that has been fashionable at several points in the last 50 years.
Prediction is obviously involved in certain forms of cognition, but it obviously isn't all there is to the kinds of beings we are.
Napkin scribbles
It's always different this time.
More seriously: there are decent arguments that say that LLMs have an upper bound of usefulness and that we're not necessarily closer to transcending that with a different AI technology than we were 10 or 30 years ago.
The LLMs we have, even if they are approaching an upper bound, are a big deal. They're very interesting and have lots of applications. These applications might be net-negative or net-positive, it will probably vary by circumstance. But they might not become what you're extrapolating them into.
Even in 2002, my CS profs were talking about how GAI was a long time off bc we had been trying for decades to innovate on neural nets and LLMs and nothing better had been created despite some of the smartest people on the planet trying.
The compute and data are both limitations of NNs.
We've already gotten really close to the data limit (we aren't generating enough useful content as a species and the existing stuff has all been slurped up).
Standard laws of physics restrict the compute side, just like how we know we will hit it with CPUs. Eventually, you just cannot put things closer together that generate more heat because they interfere with each other because we hit the physical laws re miniaturization.
No, GAI will require new architectures no one has thought of in nearly a century.
2. The category of computerized machines (of which self checkouts are one example) has absolutely revolutionized the world. Computerization is the defining technology of the last twenty years.
They revolutionized supermarkets.
I would really like to hear you explain how they revolutionized supermarkets.
I use them every day, and my shopping experience is served far better by going to a place that is smaller than one that has automated checkout machines. (Smaller means so much faster.)
Hell, if you go to Costco, the automated checkout line moves slower than the ones manned by experienced workers.
And for small baskets, sure, but it was scan as you shop that really changed supermarkets and those things thankfully do not talk.
Outside of the software world it's mostly a (much!) better Google.
Between now and a Star Trek world, there's so much to build that we can use any help we can get.
Indeed. I was using speech to text three decades ago. Dragon Naturally Speaking was released in the 90s.
It's blatantly obvious to see if you work with something you personally have a lot of expertise in. They're effectively advanced search engines. Useful sure.. but they're not anywhere close to "making decisions"
From where I look at it, LLMs are flawed in many ways, and people who see progress as inevitable do not have a mental model of the foundation of those systems to be able to extrapolate. Also, people do not know any other forms of AI or have though hard about this stuff on their own.
The most problematic things are:
1) LLMs are probabilistic and a continuous function, forced by gradient descent. (Just having a "temperature" seems so crazy to me.) We need to merge symbolic and discrete forms of AI. Hallucinations are the elephant in the room. They should not be put under the rug. They should just not be there in the first place! If we try to cover them with a layer of varnish, the cost will be very large in the long run (it already is: step-by-step reasoning, mixture of experts, RAG, etc. are all varnish, in my opinion)
2) Even if generalization seems ok, I think it is still really far from where it should be, since humans need exponentially less data and generalize to concepts way more abstract than AI systems. This is related to HASA and ISA relations. Current AI systems do not have any of that. Hierarchy is supposed to be the depth of the network, but it is a guess at best.
3) We are just putting layer upon layer of complexity instead of simplifying. It is the victory of the complexifiers and it is motivated by the rush to win the race. However, I am not so sure that, even if the goal seems so close now, we are going to reach it. What are we gonna do? Keep adding another order of magnitude of compute on top of the last one to move forward? That's the bubble that I see. I think that that is not solving AI at all. And I'm almost sure that a much better way of doing AI is possible, but we have fallen into a bad attractor just because Ilya was very determined.
We need new models, way simpler, symbolic and continuous at the same time (i.e. symbolic that simulate continuous), non-gradient descent learning (just store stuff like a database), HAS-A hierarchies to attend to different levels of structure, IS-A taxonomies as a way to generalize deeply, etc, etc, etc.
Even if we make progress by brute forcing it with resources, there is so much work to simplify and find new ideas that I still don't understand why people are so optimistic.
Hallucinations are incredibly fucking overrated as a problem. They are a consequence of the LLM in question not having a good enough internal model of its own knowledge, which is downstream from how they're trained. Plenty of things could be done to improve on that - and there is no fundamental limitation that would prevent LLMs from matching human hallucination rates - which are significantly above zero.
There is a lot of "transformer LLMs are flawed" going around, and a lot of alternative architectures being proposed, or even trained and demonstrated. But so far? There's nothing that would actually outperform transformer LLMs at their strengths. Most alternatives are sidegrades at best.
For how "naive" transformer LLMs seem, they sure set a high bar.
Saying "I know better" is quite easy. Backing that up is really hard.
In the end leaving the world changed, but not as meaningfully or positively as promised.
Maybe say something concrete? What's a positive real world impact of LLMs where they aren't hideously expensive and error prone to the point of near uselessness? Something that isn't just the equivalent of a crypto-bro saying that their system for semi-regulated speculation (totally not a rugpull!) will end the tyranny of the banks.
---
Less flippantly, they are excellent for self-studying university-level topics. It's like being able to ask questions to a personal tutor/professor.
- documentation
- design reviews
- type systems
- code review
- unit tests
- continuous integration
- integration testing
- Q&A process
- etc.
Is it not the same with AI? With the right processes you can get consistent results from inconsistent tools.
So why's it different this time?
Is the wear so small that it’s simply negligible ?
Is it going to be that significant though? No idea.
Just ask Intel what happened to 14th gen.
It's not normally an issue, but the edge cases can be very sharp. Otherwise, the bigger concern is the hardware becoming obsolete because of new generations being significantly more power efficient. Over a few years, the power+cooling+location bill of a high end CPU running at 90% utilization can cost more than the CPU itself.
Maybe we can finally have a Rosie from the Jetsons.
just what I want, a mobile Alexa that spews ads and spies on me 24/7
The “at a loss” scenario comes from (1) training costs and (2) companies selling tokens below market to get market share. Neither of those imply that people won’t run models in future. Training new frontier-class models could potentially become an issue, but even that seems unlikely given what these models are capable of.
But then, without this huge financial and tech bubble that's driven by these huge companies:
1/ will those models evolve, or new models appear, for a fraction of the cost of building them today?
2/ will GPU (or their replacement) also cost a fraction of what they cost today, so that they are still integrated in end-user processors, so that those model can run efficiently?
These people won't sit still and models will keep getting better as well as cheaper to run.
I have access to quite a few models, and I use them here and there. They are sort of useful, sometimes. But I don't pay directly for any of them. Honestly, I wouldn't.
Creating new LLMs might be out of reach for all but very well-capitalized organizations with clear intentions, and governments.
There might be a viable market for SLMs though. Why does my model need to know about the Boer wars to generate usable code?
How about chips during the dotcom period? What was their lifespan?
Maybe in next decade we will have cheap gaming cloud offerings built on repurposed GPUs.
Not saying that’s even remotely realistic over the next century, but it does seem to be how some of these people think. Excessive wealth destroys intelligence, it doesn’t enhance it, as countless examples show.
As the noise fades, and with luck, the obsession with slapping "AI" on everything will fade with it. Too many hype-driven CEOs are chasing anything but substance.
Some AI tools may survive because they're genuinely useful, but I worry that most won't be cost-effective without heavy subsidies.
Once the easy money dries up, the real engineers and builders will still be here, quietly making things that work.
Altman's plea -- "Come on guys, we just need a few trillion more!" -- and that error-riddled AI slide deck will be the meme that marks the top of the market.
In 10 years GPUs will have a lifespan for 5-7 years. The rate of improvement on this front has been slowing down faster then CPU.
What is interesting is that it seems like the ever larger sums of money sloshing around are resulting in bigger, faster hype cycles. We are already seeing some companies face issues after blowback from adopting AI too fast.
2. While comprehensive studies were never done, some tech channels did some testing and found used GPUs to be generally reliable or easily repairable, when scamming was excluded. https://youtu.be/UFytB3bb1P8
> Most of the money is being spent on incredibly expensive GPUs that have a 1-3 year lifespan due to becoming obsolete quickly and wearing out under constant, high-intensity use.
So it isn’t entirely tied to the rate of obsolescence, these things apparently get worn down from the workloads.
In terms of performance improvement, it is slightly complicated, right? It turns out that it was possible to do ML training on existing GPGPU. Then there was spurt of improvement as they go after the low-hanging fruit for that application…
If we’re talking about what we might be left with after the bubble pops, the rate of obsolescence doesn’t seem that relevant anyway. The chips as they are after the pop will be usable for the next thing or not, it is hard to guess.
- this is war path funding
- this is geopolitics; and it’s arguably a rational and responsible play
- we should expect to see more nationalization
- whatever is at the other end of this seems like it will be extreme
And, the only way out is through
This AI bubble already has lots of people with their forks and knifes waiting to capitalize on a myriad of possible surpluses after the burst. There's speculation on top of _the next bubble_ and how it will form, even before this one pops.
That is absolutely disgusting, by the way.
I think that people doing work in many professions with these offline tools alone could more than double their productivity compared to their productivity two years ago. Furthermore if the usage was shared in order to lower idle time, such as 20 machines for 100 workers, the initial capital outlay is even lower.
Perhaps investors will not see the returns they expect, but it is difficult to image how even the current state of AI doesn't vastly change the economy. There could be significant business failures among cloud providers and attempts to rapidly increase the cost of admission to closed models, but there's essentially no possibility of productivity regressing to a pre-AI levels.
They already work on the most expensive Apple hardware. I expect that price to come down in the next few years.
It’s really just the UX that’s bad but that’s solvable.
Apple isn’t having to pay for each users power and use either. They sell hardware once and folks pay with their own electricity to run it.
the dotcom bubble was a result of investors jumping on the hype train all at once and then getting off of it all at once.
Yes, investors will eventually find another hype train to jump on, but unlike 2000, we have tons of more retail investors and AI is also not a brand new tech sector, it's built upon the existing well established and "too big to fail" internet/ecommerce infrastructure. Random companies slapping AI on things will fail but all the real AI use cases will only expand and require more and more resources.
OpenAI alone just hit 800M MAU. That will easily double in a few years. There will be adjustments,corrections and adaptations of course but the value and wealth it generates is very real.
I'm no seer, I can't predict the future but I don't see a massive popping of some unified AI bubble anytime soon.
OpenAI has ~4B of revenue already, and they aren't even monetizing aggressively. Facebook has an infinite money glitch, and can afford to put billions in the ground in pursuit of moonshots and Zuck's own vanity projects. Google is Google, and xAI is Elon Musk. The most vulnerable frontier lab is probably Anthropic, and Anthropic is still backed by Amazon and, counterintuitively, also Google.
At the same time: there is a glut of questionable AI startups, extreme failure rate is likely - but they aren't the bulk of the market, not by a long shot. The bulk of the "AI money" is concentrated at either the frontier labs themselves, or companies providing equipment and services to them.
The only way I see for the "bubble to pop" is for multiple frontier labs to get fucked at the same time, and I just don't see that happening as it is.
Figuring out which was which was absolutely not possible at the time. Not many people foresaw Sun Microsystems as being a victim and nor was it obvious that Amazon would be a victor.
I wouldn't bet my life savings on OpenAI.
Same for GPUs/LLMs? At some point things will mature and we’ll be left with plentiful, cheap, high end LLM access, on the back of the investment that has been made. Whether or not it’s running on legacy GPUs, like some 90s fiber still carries traffic, is meaningless. It’s what the investment unlocks.
...whether it is profitable is another matter
It will become cool for you to become inaccessible, unreachable, no one knowing your location or what you’re doing. People might carry around little beeper type devices that bounce small pre-defined messages around on encrypted radio mesh networks to say stuff like “I’m okay” or “I love you”, and that’s it. Maybe they are used for contactless payments as well.
People won’t really bother searching the web anymore they’ll just ask AI to pull up whatever information they need.
The question is, with social media on the decline, with the internet no longer used for recreational purposes, what else are people going to do? Feels like the consumer tech sector will shrink dramatically, meaning that most tech written will be made to create “hard value” instead of soft. Think anything having to do with movement of data and matter, or money.
Much of the tech world and government plans are built on the assumption that people will just continue using tech to its maximum utility, even when it is clearly bad for them, but what if that simply weren’t the case? Then a lot of things fall apart.
arisAlexis•1h ago
mjhay•1h ago
gizmo686•1h ago
We are well into this process already. Core chat capabilities have pretty much stalled out. But most of the attempts at application are still very thin layers over chat bots.
abathologist•1h ago
dotnet00•1h ago