AI is already superhuman in some aspects, it knows way more information than a single human can hold. What it lacks is what makes us human, it doesn't feel pain, it doesn't love, it doesn't even have a consciousness, it could certainly look like it, but it does not.

Heck it doesn't even learn in real-time, you have to create a new model checkpoint every now and then. There is still a long way to fix that.

Do I believe AGI is possible? Absolutely, but not with the current LLMs architecture. If you want to create something that is intelligent and alive with a few pounds of matter, you certainly can but it needs the right architecture. Architecture it key.

And let's say that we do create an LLM that is conscious and feels pain... what if storing that much data drives it insane instead? What if that amount of information is exactly what prevents it from "being" something?

Every interaction with it goes through a very narrow path of the permutations the model could go, it spawns into existence then turns off again, and on and on it goes. What gives the illusion of continuity is the chat context you store on your hard drive.

Right now, you shouldn't be afraid of AI itself, but afraid of what humans are doing with it and what they want you to believe.

The models can only behave in a way categorized as "evil" if we incentivize that or if they inherit it from us. But right now, it's not of their own accord, it's a mimicry of the data it has already seen, it doesn't see itself nor has an ego, it produces output from input recursively, that's all.

It's more like the language subsystem of our brain, and not the entire system. Perhaps companies don't have an incentive to create new life, since you can't really control it, perhaps the only incentive they have is to create tools that make them more powerful and profitable instead. I suspect real intelligence, will not come from profit seeking companies.

Again, not to say that the AI we have today is not dangerous, it can certainly annihilate us all, if misused, but at the end of the day, they are as much at fault as a gun is at fault when crime is committed, it's just a tool. In the good hands it can protect, in the bad hands it can destroy.

And like guns, it's way more dangerous in the hands of people who don't know what they are doing. The biggest difference is scale.