way more pogress toward fusion than AGI. Uncontrolled runaway fusion reactions were perfected in the 50s (iirc) with the thermonuclear bombs. Controllable fusion reactions have been common for many years. A controllable, self-sustaining, and profitable fusion reaction is all that is left. The goalposts that mark when AGI has been reached haven't even been defined yet.
Humans have since adapted to identify content differences and assign lower economic value to content created by programs, i.e. the humans being "impersonated" and "fooled" are themselves evolving in response to imitation.
This isn’t a bad thing, and I think LLMs are very impressive. But I do think we’d hesitate to call their behavior human-like if we weren’t predisposed to anthropomorphism.
If you train it on a bunch of paintings whose quality ranges from a toddler's painting to Picasso's, it's not going to make one that's better than Picasso's, it's going to output something more comparable to the most average painting it was trained on. If you then adjust your training data to only include world's best paintings ever since we began to paint, the outcome is going to improve, but it'll just be another better-than-human-average painting. If you then leave it running 24/7, it'll churn out a bunch of better-than-human-average paintings, but there's still an easily-identifiable ceiling it won't go above.
An oracle that always returns the most average answer certainly has its use cases, but it's fundamentally opposed to the idea of superintelligence.
Yes, I agree, it's not high quality stuff it produces exactly, unless the person using it already is an expert and could produce high quality stuff without it too.
But there is no denying it that those things were regarded as "far-near future maybe" for a long time, until some people put the right pieces together.
second a definition is obviously not a prerequisite as evidenced by natural selection
I don't think he should stop, because I think he's right. We lack a definition of intelligence that doesn't do a lot of hand waving.
You linked to a paper with 18 collective definitions, 35 psychologist definitions, and 18 ai researcher definitions of intelligence. And the conclusion of the paper was that they came up with their own definition of intelligence. That is not a definition in my book.
> second a definition is obviously not a prerequisite as evidenced by natural selection
right, we just need a universe, several billions of years and sprinkle some evolution and we'll also get intelligence, maybe.
And real AI is probably like fusion. Its always 10 years away.
You and also everyone since the beginning of AI. https://quoteinvestigator.com/2024/06/20/not-ai/
The common thread between those who take things as "AI is anything that doesn't work yet" and "what we have is still not yet AI" is "this current technology could probably have used a less distracting marketing name choice, where we talk about what it delivers rather than what it's supposed to be delivering".
From where I sit, the generative models provide more flexibility but tend to underperform on any particular task relative to a targeted machine learning effort, once you actually do the work on comparative evaluation.
You appear to be comparing apples to oranges. A generation task is not a categorization task. Machine learning solves categorization problems. Generative AI uses model trained by machine learning methods, but in a very different architecture to solve generative problems. Completely different and incomparable application domain.
If “machine learning” is taken to be so broad as to include any artificial neural network, all of which are trained with back propagation these days, then it is useless as a term.
The term “machine learning” was coined in the era of specialized classification agents that would learn how to segment inputs in some way. Thing email spam detection, or identifying cat pictures. These algorithms are still an essential part of both the pre-training and RLHF fine tuning of LLM models. But the generative architectures are new and very essential to the current interest in and hype surrounding AI at this point in time.
> “I mean in 2035, that, like, graduating college student, if they still go to college at all, could very well be, like, leaving on a mission to explore the solar system on a spaceship in some completely new, exciting, super well-paid, super interesting job, and feeling so bad for you and I that, like, we had to do this kind of, like, really boring old kind of work and everything is just better."
Which should be reassuring to anyone having trouble finding an entry-level job as an illustrator or copywriter or programmer or whatever.
If nothing else it's been a sci-fi topic for more than a century. There's connotations, cultural baggage, and expectations from the general population about what AI is and what it's capable of, most of which isn't possible or applicable to the current crop of "AI" tools.
You can't just change the meaning of a word overnight and toss all that history away, which is why it comes across as an intentionally dishonest choice in the name of profits.
More to the point, the history of AI up through about 2010 talks about attempts to get it working using different approaches to the problem space, followed by a shift in the definitions of what AI is in the 2005-2015 range (narrow AI vs. AGI). Plenty of talk about the various methods and lines fo research that were being attempted, but very little about publicly pushing to call commercially available deliverables as AI.
Once we got to the point where large amounts of VC money was being pumped into these companies there was an incentive to redefine AI in favor of what was within the capabilities and scope of machine learning and LLMs, regardless of whether that fit into the historical definition of AI.
I took an AI class in 2001. We learned all sorts of algorithms classified as AI. Including various ML techniques. Under which included perceptrons.
Ten years ago you'd be ashamed to call anything "AI," and say machine learning if you wanted to be taken seriously, but neural networks have really have brought back the term--and for good reason, given the results.
From skimming the conversation it seems to mostly revolve around LLMs (transformer models) which is probably not going to be the way we obtain AGI to begin with, frankly it is too simple to be AGI, but the reason why there's so much hype is because it is simple to begin with so really I don't know.
Then I got it. :) Something so mundane that maybe the AIs can help prevent it.
I mean sure I now "control" the AI, but I still think these no AGI for 2 decades claims are a bit rough.
If you’re correct, there’s not much reward aside from the “I told you so” bragging rights, if you’re wrong though - boy oh boy, you’ll be deemed unworthy.
You only need to get one extreme prediction right (stock market collapse, AI taking over, etc ), then you’ll be seen as “the guru”, the expert, the one who saw it coming. You’ll be rewarded by being invited to boards, panels and government councils to share your wisdom, and be handsomely paid to explain, in hindsight, why it was obvious to you, and express how baffling it was that no one else could see what you saw.
On the other hand, if predict an extreme case and you get it wrong, there’s virtually 0 penalties, no one will hold that against you, and no one even remembers.
So yeah, fame and fortune is in taking many shots at predicting disasters, not the other way around.
cause elon musk says FSD is coming in 2017?
If ChatGPT is not AGI, somebody has moved goalposts.
it doesn't make him one
But nothing will make grifters richer than promising it's right around the corner.
AI has now been revealed to the masses. When AGI arrives most people will barely notice. It will just feel like slightly better LLMs to them. They will have already cemented notions of how it works and how it affects their lives.
The debate about AGI is interesting from a philosophical perspective, but from a practical perspective AI doesn't need to get anywhere close to AGI to turn the world upside down.
I feel like GPT 3 was AGI, personally. It crossed some threshold that was both real and magical, and future improvements are relying on that basic set of features at their core. Can we confidently say this is not a form of general intelligence? Just because it’s more a Chinese Room than a fully autonomous robot? We can keep moving the goalposts indefinitely, but machine intelligence will never exactly match that of humans.
It crossed some threshold that was both real and magical
and future improvements are relying on that basic set of features at their core
Thats not a period, it's a full stop. There is no debate to be had here.
IF an LLM makes some sort of breakthrough (and massive data collation allows for that to happen) it needs to be "re trained" to absorb its own new invention.
But we also have a large problem in our industry, where hardware evolved to make software more efficient. Not only is that not happening any more but we're making our software more complex and to some degree less efficient with every generation.
This is particularly problematic in the LLM space: every generation of "ML" on the llm side seems to be getting less efficient with compute. (Note: this isnt quite the case in all areas of ML, yolo models working on embedded compute is kind of amazing).
Compactness, efficiency and reproducibility are directions the industry needs to evolve in, if it ever hopes to be sustainable.
We are approaching situation, where AI will make most decisions, and people will wear it as a skin suit, to fake competency!
Otherwise we would have to say that pre-literacy societies lacked intelligence, which would be silly since they are the ones that invented writing in the first place!
> capability to learn skills and build a mode
We are adults, not children! At some point brain looses plasticity, and it is very difficult to learn new stuff!
And good luck competing with asians or AI!
If you have evidence for that claim, show it. Otherwise, no, you're just making stuff up.
Very simple proof, they can not even read/listen to their own constitution!
Obviously this quote would be well applied if we were at a stage where computers were better at everything humans can do and some people were saying "This is not AGI because it doesn't think exactly the same as a human". But we aren't anywhere near this stage yet.
2029: Human-level AI
2045: The Singularity - machine intelligence 1 billion times more powerful than all human intelligence
Based on exponential growth in computing. He predicts we'll merge with AI to transcend biological limits. His track record is mixed, but 2029 looks more credible post-GPT-5. The 2045 claim remains highly speculative.
Hegel thought history ended with the Prussian state, Fukuyama thought it ended in liberal America, Paul thought judgement day was so close you need not bother to marry, the singularity always comes around when the singularians get old. Funny how that works
The overwhelming majority of all gains in human life expectancy have come due to reductions in infant mortality. When you hear about things like a '40' year life expectancy in the past it doesn't mean that people just dropped dead at 40. Rather if you have a child that doesn't make it out of childhood, and somebody else that makes it to 80 - you have a life expectancy of ~40.
If you look back to the upper classes of old their life expectancy was extremely similar to those of today. So for instance in modern history, of the 15 key Founding Fathers, 7 lived to at least 80 years old: John Adams, John Quincy Adams, Samuel Adams, Jefferson, Madison, Franklin, John Jay. John Adams himself lived to 90. The youngest to die were Hamilton who died in a duel, and John Hancock who died of gout of an undocumented cause - it can be caused by excessive alcohol consumption.
All the others lived into their 60s and 70s. So their overall life expectancy was pretty much the same as we have today. And this was long before vaccines or even us knowing that surgeons washing their hands before surgery was a good thing to do. It's the same as you go back further into history. A study [1] of all men of renown in Ancient Greece was 71.3 [1], and that was from thousands of years ago!
Life expectancy at birth is increasing, but longevity is barely moving. And as Kurzweil has almost certainly done plentiful research on this topic, he is fully aware of this. Cognitive dissonance strikes again.
I was a lot disappointed when he went to work for Tesla, and I think that he had some achievement there, butnot nearly the impact I believe he potentially has.
His switch (back?) to OpenAI was, in my mind, much more in keeping with where his spirit really lies.
So, with that in mind, maybe I've drunk too much kool aid, maybe not. But I'm in agreement with him, the LLMs are not AGI, they're bloody good natural language processors, but they're still regurgitating rather than creating.
Essentially that's what humans do, we're all repeating what our education/upbringing told us worked for our lives.
But we all recognise that what we call "smart" is people recognising/inventing ways to do things that did not exist before. In some cases its about applying a known methodset to a new problem, in others its about using a substance/method in a way that other substances/methodsets are used, but the different substance/methodset produces something interesting (think, oh instead of boiling food in water, we can boil food in animal fats... frying)
AI/LLMs cannot do this, not at all. That spark of creativity is agonisingly close, but, like all 80/20 problems, is likely still a while away.
The timeline (10 years) - it was the early 2010s (over 10 years ago now) that the idea of backward propagation, after a long AI winter, finally came of age. It (the idea) had been floating about since at least the 1970s. And that ushered in the start of our current revolution, that and "Deep Learning" (albeit with at least another AI winter spanning the last 4 or 5 years until LLMs arrived)
So, given that timeline, and the restraints in the currrent technology, I think that Andrej is on the right track, and it will be interesting to see where we are in ten years time.
However, don't let the bandwagon ( from either side ) cloud your judgment. Even warm fusion or any fusion at all is still very useful and it's here to stay.
This whole AGI and "the future" thing is mostly a VC/Banks and shovel sellers problem. A problem that has become ours too because the ridiculous amounts of money "invested", so even warm fusion is not enough from an investment vs expectations perspective.
They are already playing musical money chairs, unfortunately we already know who's going to pay for all of this "exuberance" in the end.
I hope this whole thing crashes and burns as soon as possible, not because I don't "believe" in AI, but because people have been absolutely stupid about it. The workplace has been unbearable with all this stupidity and amounts of fake "courage" about every single problem and the usual judgment of the value of work and knowledge your run-of-the-mill dipshit manager has now.
That includes anyone reading this message long after the lives of those reading it on its post date have ended.
Which of course raises the interesting question of how I can make good on this bet.
"When you get a demo and something works 90% of the time, that’s just the first nine. Then you need the second nine, a third nine, a fourth nine, a fifth nine. While I was at Tesla for five years or so, we went through maybe three nines or two nines. I don’t know what it is, but multiple nines of iteration. There are still more nines to go.
That’s why these things take so long."
If you need to get to 9 9s, the 9th 9 could be more effort than the other 8 combined.
I think this is an important way of understanding AI progress. Capability improvements often look exponential on a particular fixed benchmark, but the difficulty of the next step up is also often exponential, and so you get net linear improvement with a wider perspective.
— Tom Cargill, Bell Labs (September 1985)
A marathon consists of two halves: the first 20 miles, and then the last 10k (6.2mi) when you're more sore and tired than you've ever been in your life.
The interviewer had an idea that he took for granted: that to understand knowledge you have to have a model of the world. LLMs seem to udnerstand language therefore they've trained a model of the world. Sutton rejected the premise immediately. He might be right in being skeptical here.
babies are already born with "the model of the world"
but a lot of experiments on babies/young kids tell otherwise
> but a lot of experiments on babies/young kids tell otherwise
I believe they are born with such a model? It's just that model is one where mummy still has fur for the baby to cling on to? And where aged something like 5 to 8 it's somehow useful for us to build small enclosures to hide in, leading to a display of pillow forts in the modern world?
Did you see the recent video by Nick Beato [1] where he asks various models about a specific number? The models that get it right are the models that consume youtube videos, because there was a youtube video about that specific number. It's like, these models are capable of telling you about very similar things that they've seen, but they don't seem like they understand it. It's totally unclear whether this is a quantitative or qualitative gap.
Even if you don't like that definition, you still have the question of how many nines we are away from having an AI that can contribute to its own development.
I don't think you know the answer to that. And therefore I think your "fast acceleration within two years" is unsupported, just wishful thinking. If you've got actual evidence, I would like to hear it.
Machine learning has been helping with the development of machine learning ever since hyper-parameter optimisers became a thing.
Transformers have been helping with the development of transformer models… I don't know exactly, but it was before ChatGPT came out.
None of the initials in AGI are booleans.
But I do agree that:
> "fast acceleration within two years" is unsupported, just wishful thinking
Nobody has any strong evidence of how close "it" is, or even a really good shared model of what "it" even is.
If you look at it differently, assembly language may have been one nine, compilers may have been the next nine, successive generations of language until ${your favorite language} one more nine, and yet, they didn't get us noticeably closer to AGI.
It may also be that we're looking at this the wrong way altogether. If you compare the natural world with what humans have achieved, for instance, both things are qualitatively different, they have basically nothing to do with each other. Humanity isn't "adding nines" to what Nature was doing, we're just doing our own thing. Likewise, whatever "nines" AGI may be singularly good at adding may be in directions that are orthogonal to everything we've been doing.
Progress doesn't really go forward. It goes sideways.
Great. So what's the plan for refunding with interest the customers who were defrauded by a full self-driving demo that was not and still isn't the product they were promised (much less delivered) by Tesla. How much did Karpathy personally profit from the lie he participated in?
Academia has rediscovered itself
Signal attenuation, a byproduct of entropy, due to generational churn means there's little guarantee.
Occam's Razor; Karpathy knows the future or he is self selecting biology trying to avoid manual labor?
His statements have more in common with Nostradamus. It's the toxic positivity form of "the end is nigh". It's "Heaven exists you just have to do this work to get there."
Physics always wins and statistics is not physics. Gamblers fallacy; improvement of statistical odds does not improve probability. Probability remains the same this is all promises of some people who have no idea or interest in doing anything else with their lives; so stay the course.
20% of your effort gets you 80% of the way. But most of your time is spent getting that last 20%. People often don't realize that this is fractal like in nature, as it draws from the power distribution. So of that 20% you still have left, the same holds true. 20% of your time (20% * 80% = 16% -> 36%) to get 80% (80% * 20% => 96%) again and again. The 80/20 numbers aren't actually realistic (or constant) but it's a decent guide.
It's also something tech has been struggling with lately. Move fast and break things is a great way to get most of the way there. But you also left a wake of destruction and tabled a million little things along the way. Someone needs to go back and clean things up. Someone needs to revisit those tabled things. While each thing might be little, we solve big problems by breaking them down into little ones. So each big problem is a sum of many little ones, meaning they shouldn't be quickly dismissed. And like the 9's analogy, 99.9% of the time is still 9hrs of downtime a year. It is still 1e6 cases out of 1e9. A million cases is not a small problem. Scale is great and has made our field amazing, but it is a double edged sword.
I think it's also something people struggle with. It's very easy to become above average, or even well above average at something. Just trying will often get you above average. It can make you feel like you know way more but the trap is that while in some domains above average is not far from mastery in other domains above average is closer to no skill than it is to mastery. Like how having $100m puts your wealth closer to a homeless person than a billionaire. At $100m you feel way closer to the billionaire because you're much further up than the person with nothing but the curve is exponential.
Most of these companies value is built on the idea of AGI being achievable in the near future.
AGI being too close or too far away affects the value of these companies- too close and it'll seem too likely that the current leaders will win. Too far away and the level of spending will seem unsustainable.
Is it? Or is it based on the idea a load of white collar workers will have their jobs automated, and companies will happily spend mid four figures for tech that replaces a worker earning mid five figures?
I know it's against the guidelines to discuss the state of a thread, but I really wish we could have thoughtful conversations about the content of links instead of title reactions.
The brain processes and has insights differently experiencing it at conversation speed.
We might get what the conversation was that others had, but it can miss the mark for the listening and inner processing that leads to it's own gifts.
It's not about one or the other for me, usually both.
edit: typo fix.
Granted, a bunch of commenters are probably doing what you’re saying.
The criticism that people are only replying to a tiny portion of the argument is still valid, but sometimes it's more fun to have an open-ended discussion rather than address what's in the actual article/video.
That's an "agent" at its simplest -- a LLM able to derive from natural language when it is contextually appropriate to call out to external "tools" (i.e. functions).
The people heralding the emergence of AGI are doing little more than pushing Ponzi schemes along while simultaneously fueling vitriolic waves of hate and neo-luddism for a ground-breaking technology boom that could enhance everything about how we live our lives... if it doesn't get regulated into the ground due to the fear they're recklessly cooking up.
Software can already write more text on any given subject better than a majority of humanity. It can arguably drive better across more contexts than all of humanity - any human driver over a billion miles of normal traffic will have more accidents than self driving AI over the same distance. Short stories, haikus, simple images, utility scripts, simple software, web design, music generation - all of these tasks are already superhuman.
Longer time horizons, realtime and continuous memory, a suite of metacognitive tasks, planning, synthesis of large bodies of disparate facts into novel theory, and a few other categories of tasks are currently out of reach, but some are nearly solved, and the list of things that humans can do better than AI gets shorter by the day. We're a few breakthroughs away, maybe even one big architectural leap, from having software that is capable (in principle) of doing anything humans can do.
I think AGI is going to be here faster than Kurzweil predicted, because he probably didn't take into consideration the enormous amount of money being spent on these efforts.
There has never been anything like this in history - in the last decade, over 5 trillion dollars has been spent on AI research and on technologies that support AI, like crypto mining datacenters that pivoted to AI, new power, water, data support, providing the infrastructure and foundation for the concerted efforts in research and development. There are tens of thousands of AI researchers, some of them working in private finance, some for academia, some doing military resarch, some doing open source, and a ton doing private sector research, of which an astonishing amount is getting published and shared.
In contrast, the entire world spent around 16 trillion dollars on world war II - all of the R&D and emergency projects and military logistics, humanitarian aid, and so on.
We have AI getting more resources and attention and humans involved in a singular development effort, pushing toward a radical transformation of the very concept of "labor" - while I think it might be a good thing if it is a decade away, even perpetually so until we have some reasonable plan for coping with it, I very much think we're going to see AGI within the very near future.
*When I say "in principle" I mean that given the appropriate form factor, access, or controls, the AI can do all the thinking, planning, and execution that a human could do, at least as well as any human. We will have places that we don't want robots or AI going, tasks reserved for humans, traditions, taboos, economics, and norms that dictate AI capabilities in practice, but there will be no legitimacy to the idea that an AI couldn't do a thing.
A new contribution by quite a few prominent authors. One of the better efforts at defining AGI *objectively*, rather than through indirect measures like economic impact.
I believe it is incomplete because the psychological theory it is based on is incomplete. It is definitely worth discussing though.
—-
In particular, creative problem solving in the strong sense, ie the ability to make cognitive leaps, and deep understanding of complex real-world physics such as the interactions between animate and inanimate entities are missing from this definition, among others.
Fundamentally, AGI requires 2 things.
First it needs to be able to operate without information, learning as it goes. The core kernel should be such that it doesn't have any sort of training on real world concepts, only general language parsing that it can use to map to some logic structure to be able to determine a plan of action. So for example, if you give the kernel the ability to send ethernet packets, it should eventually figure out how to talk tls to communicate with the modern web, even if that takes an insane amount of repetition.
The reason for this is that you want the kernel to be able to find its way through any arbitrarily complex problem space. Then as it has access to more data, whether real time, or in memory, it can be more and more efficient.
This part is solvable. After all, human brains do this. A single rack of Google TPUs is roughly the same petaflops as a human brain operating at max capacity if you assume neuron activation is a add-multiply and firing speed of 200 times/second, and humans don't use all of their brain all the time.
The second part that makes the intelligence general is the ability to simulate reality faster than reality. Life is imperative by nature, and there are processes with chaotic effects (human brains being one of them), that have no good mathematical approximations. As such, if an AGI can truly simulate a human brain to be able to predict behavior, it needs to do this at an approximation level that is good enough, but also fast enough to where it can predict your behavior before you exhibit it, with overhead in also running simulations in parallel and figuring out the best course of actions. So for a single brain, you are looking at probably a full 6 warehouses full of TPUs.
Read that sentence again. Slowly.
What do you think "general language parsing" IS if not learned patterns from real-world data? You're literally describing a transformer and then saying we need to invent it.
And your TLS example is deranged. You want an agent to discover the TLS protocol by randomly sending ethernet packets? The combinatorial search space is so large this wouldn't happen before the sun explodes. This isn't intelligence! This is bruteforce with extra steps!
Transformers already ARE general algorithms with zero hardcoded linguistic knowledge. The architecture doesn't know what a noun is. It doesn't know what English is. It learns everything from data through gradient descent. That's the entire damn point.
You're saying we need to solve a problem that was already solved in 2017 while claiming it needs a century of quantum computing.
(I did listen to a sizable portion of this podcast while making risotto (stir stir stir), and the thought occurred to me: “am I becoming more stupid by listening to these pundits?” More generally, I feel like our internet content (and meta content (and meta meta content)) is getting absolutely too voluminous without the appropriate quality controls. Maybe we need more internet death.)
(I was in college during the first AI Winter, so... I can't help but think that the cycles are tighter but convergence isn't guaranteed.)
agrover•1h ago
sputknick•1h ago