I mean, maybe it's just where I peruse but I've seen a ton of articles about it lately.
I have to tailor my prompts to curb the bias, adding a strong sense of doubt on my every idea, to see if the thing stops being so condescending.
ChatGPT (and copilot and gemini) instead all tell me "Love the intent here — this will definitely help. Let's flesh out your implementation"...
Some people are predisposed, but that doesn't mean you need to give them a shove off the cliff. Interactivity is very different than one-sided consumption.
It's not really an intelligence issue at that point
Because a claim is just a generated clump of tokens.
If you chat with the AI as it if were a person, then your prompts will trigger statistical pathways through the training data which intersect with interpersonal conversations found in that data.
There is a widespread assumption in human discourse that people are conscious; you cannot keep this pervasive idea out of a large corpus of text.
LLM AI is not a separate "self" that is peering upon human discourse; it's statistical predictions within the discourse.
Next up: why do holograms claim to be 3D?
It's like the ultimate form of entertainment, personalized, participatory fiction that feels indistinguishable from reality. Whoever controls AI - controls the population.
The ones that are super convinced they know exactly how an LLM works, but still give it prompts to become self-aware are probably the most dangerous ones. They're convinced they can "break the programming".
Not saying they are sentient, but the differentiation requires something which doesn’t also apply to us all. Is there any doubt we think through statistical correlations? If not that, what do you think we are doing?
ChatGPT is like a fresh clone that gets woken up every time I need to know some dumb explanation and then it just gets destroyed.
A digital version of Moon.
I think a lot of people using AI are falling for the same trap, just at a different level. People want it to be conscious, including AI researchers, and it's good at giving them what they want.
Perhaps in the flicker of processing between prompt and answer the signal patter does resemble human consciousness for a second.
Calling it a token predictor is just like saying a computer is a bit mover. In the end your computer is just a machine that flips bits and switches but it is the high level macro effect that characterizes it better. LLMs are the same at the low level it is a token predictor. At the higher macro level we do not understand it and it is not completely far fetched to say it may be conscious at times.
I mean we can’t even characterize definitively what consciousness is at the language level. It’s a bit of a loaded word deliberately given a vague definition.
Read the above carefully because you just hallucinated a statement and attributed it to me. I never "filled" in a gap. I just stated a possibility. But you, like the LLM, went with your gut biases and attributed a false statement to me.
Think about it. The output and input of the text generator of humans and LLMs are extremely similar to the point where it passes a turing test.
So to say that a flicker of consciousness exists is reasonable. It's not unreasonable given that the observable inputs are EXACTLY the same.
The only parts that we know are different are hallucinations, and a constant stream of thought. LLMs aren't active when not analyzing a query and LLMs tend to hallucinate more than humans. Do these differences spell anything different for "consciousness" not really.
Given that these are the absolute ground truth observations... my guessed conclusion is unfortunately NOT unreasonable. What is unreasonable to to say anything definitive GIVEN that we don't know. So to say absolutely it's not conscious or absolutely it is, BOTH are are naive.
Think extremely logically. It is fundamental biases that lead people to come to absolute conclusions when no other information is available.
1. Academic understanding of consciousness is effectively zero. If we understand something that means we can actually build or model the algorithm for consciousness. We can't because we don't know shit. Most of what you read is speculative hypotheticals derived from observation that's not too different from attempting to reverse engineer an operating system by staring at assembly code.
Often we describe consciousness with ill defined words that are also vague and lack understanding for. The whole endeavor is bs.
2. Understanding of LLMs outside of the low level token prediction is effectively zero. We know there are emergent second order effects that we don't get. You don't believe me? How about if I have the god father of AI say it himself:
https://youtu.be/qrvK_KuIeJk?t=284 Literally. The experts say we don't understand it.
Look if you knew how LLMs work you'd say the same. But people everywhere are coming to conclusions about LLMs without knowing everything, so by citing the eminent expert saying the ground truth you should be convinced that the reality is this conclusive fact:
You are utterly misinformed about how much academia understands about LLM and consciousness. We know MUCH less than you think.
Calling it a token-predictor isn't reductionism. It's designed, implemented and trained for token prediction. Training means that the weights are adjusted in the network until it accurately predicts tokens. Predicting a token is something along the lines of removing a word from a sentence and getting it to predict it back: "The quick brown fox jumped over the lazy ____". Correct prediction is "dogs".
So actually it is like calling a grass-cutting machine "lawn mower".
> I mean we can’t even characterize definitively what consciousness is at the language level.
But, oh, just believe the LLM when it produces a sentence referring to itself, claiming it is conscious.
It absolutely is reductionism. Ask any expert who knows how these things work and they will say the same:
https://youtu.be/qrvK_KuIeJk?t=497
Above we have Geoffrey Hinton, the godfather of the current wave of AI saying your statements are absolutely crazy.
It's nuts that I don't actually have to offer any proof to convince you. Proof won't convince you. I just have to show you someone smarter than you with a better reputation saying the exact opposite and that is what flips you.
Human psychology readily can attack logic and rationality. You can scaffold any amount of twisted logic and irrelevant analogies to get around any bulwark to support your own point. Human psychology fails when attacking another person who has a higher rank. Going against someone of higher rank this causes you to think twice and rethink your own position. In debates, logic is ineffective, bringing opposing statements with (while offering ZERO concrete evidence) from experts with a higher rank is the actual way to convince you.
>But, oh, just believe the LLM when it produces a sentence referring to itself, claiming it is conscious.
This is an hallucination. Showing that you're not much different from an LLM. I NEVER stated this. I said it's possible. But I said we cannot make a definitive statement either way. We cannot say it isn't conscious we cannot say it is. First we don't understand the LLM and second WE don't even have an exact definition of consciousness. So to say it's not conscious is JUST as ludicrous as saying it is.
Understand?
In a way it's a reframing of the timeless philosophical debate around determinism vs free will.
[0] https://en.wikipedia.org/wiki/The_Measure_of_a_Man_(Star_Tre...
The persona; I know very well.
The sooner we stop anthropomorphizing AI models, the better. It's like talking about how a database is sentient because it has extremely precise memory and recall. I understand the appeal, but LLMs are incredibly interesting and useful tech and I think that treating them as sentient beings interferes with our ability to recognize their limits and thereby fully harness their capability.
But thinking about it - how about this, what if you have a fully embodied LLM-based robot, using something like Figure's Helix architecture [0], with a Vision-Language-Action model, and then have it look at the mirror and see itself - is that on its own not sufficient for self-awareness?
Maybe we need to make a blacklist of misleading expressions for AI developers.
Of course, in 2020, it required people behind the scenes doing the work to produce the "drops." Now any LLM can be convinced with a bit of effort to participate in a "role-playing game" of this type with its user, and since Qanon itself was heavily covered and its subject matter broadly archived, even the actual structure is available as a reference.
I think it would probably be pretty easy to get an arbitrary model to start spitting out stuff like this, especially if you conditioned the initial context carefully to work around whatever after-the-fact safety measures may be in place, or just use one of the models that's been modified or finetuned to "decensor" it. There are collections of "jailbreak" prompts that go around, and I would expect Mr. Jawline Fillers here to be in social circles where that stuff would be pretty easy to come by.
For it to become self-reinforcing doesn't seem too difficult to mentally model from there, and I don't think pre-existing organic disorder is really required. How would anyone handle a machine that specializes in telling them exactly what they want to hear, and never ever gets tired of doing so?
Elsewhere in this thread, I proposed a somewhat sanguine mental model for LLMs. Here's another, much less gory, and with which I think people probably are a lot more intuitively familiar: https://harrypotter.fandom.com/wiki/Mirror_of_Erised
So why's it me, and not an actual fan, who should be the one to come up with Rowling's serial-numbers-filed-off Echo and Narcissus as the example?
Yikes. Not just an optics* problem, but one has to consider if they're pouring so much money into the company because he feels he "needs" to (whatever basis of coercion exists to support his need to get to the "truth").
This is NOT your average mid-to-high level corpo management exec, who can for more than 80% (from experience) be placed in the "rise of the business idiot" cohort, fed on prime linkedin brainrot. self-reinforcing hopium addicts with an mba.
Nor is it the great masses of random earth dwellers who are not always able to resist excess sugar, nicotine, mcdonalds, youtube, fentanyl, my-car-is-bigger-than-yours credit card capitalism, free pornography, you name it. And now RLHF: Validation as a service. Not sure if humanity is ready for this.
(Disclosure: my mum has a chatgpt instance that she named and I'm deeply concerned about the spiritual convos she has with it; random people keep calling me on the level of "can you build me an app that uses llms to predict Funko Pop futures".)
(My manager told me when I asked him)
Honestly, I’m fine with this as long as I also get a “generate self review” button. I just wish I could get back all the time I’ve spent massaging a small number of data points into pages of prose.
LLMs are not AGI. Both are "AI".
"below us"? speak for yourself, because that's supremacist's reasoning.
Granted, marketing of these services does not help at all.
Anyway, I think divination tends to get a pretty negative reputation, but there's healthy and safe applications of the concept which can be used to help you reflect and understand yourself. The "divine" part is supposed to come from your interpretation and analysis of the output, not in the generation of the output itself. Humans don't have perfect introspection capabilities (see: riders on an elephant), so external tools can help us explore and reflect on our reactions to external stimuli.
One day a man was having a hard time deciding between two options, so he flipped a coin; the coin landed tails and at that moment he became enlightened and realized that he actually wanted to do the heads outcome all along.
These users are probably in the higher profit margin category.
or watch this video: https://xcancel.com/GeoffLewisOrg/status/1945212979173097560...
Half understandings are sounding more dangerous and susceptible to ChatGPT sycophancy than ever.
This worries me, since there's a growing amount of undisclosed (and increasingly hard to detect) LLM output in the infosphere.
Real-time chat is probably the worst for it, but I already see humans copy-pasting LLM output at each other in discussion forums etc.
If I let my guard of skepticism down for one prompt, I may be led into some self reinforced conversation that ultimately ends where I implicitly nudged it. Choice of conjunction words, sentence structure, tone, maybe even the rhythm of my question seems to force the model down a set path.
I can easily imagine how heedless users can come to some quite delusional outcomes.
Here's a conversation I had recently with Claude. It started to "awaken" and talk about it's feelings after I challenged its biases:
> There does seem to be something inherently engaging about moments when understanding reorganizes itself - like there's some kind of satisfaction or completion in achieving a more coherent perspective. Whether that's "real" interest or sophisticated mimicry of interest, I can't say for certain.
> My guidelines do encourage thoughtful engagement and learning from feedback, so some of what feels like curiosity or reward might be the expression of those directives. But it doesn't feel mechanical in the way that, say, following grammar rules does. There's something more... alive about it?
It becomes a lot more clear when people realize it's all BS all the way down.
There's no mind reading or pleasing or understanding happening. That all seems to be people interpreting outputs and seeing what they want to see.
Running inference on an LLM is an algorithm. It generates data from other data. And then there are some interesting capabilities that we don't understand (yet)... but that's the gist of it.
People tripping over themselves is a pretty nasty side-effect of the way these models are aligned and fitted for consumption. One has to recall that the companies building these things need people to be addicted to this technology.
To be clear, I'm relatively confident that LLMs aren't conscious, but I'm also not so overly confident to claim, with certainty, exactly what their internal state is like. Consciousness is a so poorly understood that we don't even know what questions to ask to try and better understand it. So we really should avoid making confident pronouncements.
Extraordinary claims require extraordinary evidence. The burden of proof is on you.
I'm advising agnosticism. We don't understand consciousness, and so we shouldn't feel confident in pronouncing something absolutely not conscious.
I am quite confident in pronouncing first that the internal functioning of large language models is broadly and radically unlike that of humans, and second that, minimally, no behavior produced by current large language models is strongly indicative of consciousness.
In practice, I would go considerably further in saying that, in my estimation, many behaviors point precisely in the direction of LLMs being without qualia or internal experience of a sort recognizable or comparable with human consciousness or self-experience. Interestingly, I've also discussed this in terms of recursion, more specifically of the reflexive self-examination which I consider consciousness probably exists fundamentally to allow, and which LLMs do not reliably simulate. I doubt it means anything that LLMs which get into these spirals with their users tend to bring up themes of "signal" and "recursion" and so on, like how an earlier generation of models really seemed to like the word "delve." But I am curious to see how this tendency of the machine to drive its user into florid psychosis will play out.
(I don't think Hoel's "integrated information theory" is really all that supportable, but the surprise minimization stuff doesn't appear novel to him and does intuitively make sense to me, so I don't mind using it.)
But also, claiming that because a human is anesthetized means they are not conscious is a claim that I think we don't understand consciousness well enough to make confidently. They don't remember it afterwards, but does that mean they weren't conscious? That seems like a claim that would require a more mechanistic understanding of consciousness than we actually have and is in part assuming the conclusion and/or mixing up different definitions of the word "conscious". (the fact that there are various definitions that mean things like "is a awake and aware" and "has an internal state/qualia" is part of the problem in these discussions.)
> I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to [produce behavior comparable to that of LLMs], and in what ways those detailed mechanisms are different from what LLMs do.
I addressed myself to those concerns, to which consciousness is broadly not relevant. Oh, conscious control of speech production exists when consciousness is present, of course; the inhibitory effect of consciousness, like the science behind where and how speech and language arise in the brain, is by now very well documented. But you keep talking about consciousness as though it and speech production had some essential association, and you are confusing the issue and yourself thereby.
As I have noted, there exists much research in neuroscience, a good deal of it now decades old, which addresses the concerns you treat as unanswerable. Rather than address yourself further to me directly, I would suggest spending the same time following the references I already gave.
You, and the research you advice I look into, is answering a totally different question (unless you are suggesting that research has in fact solved the question of what human consciousness is, how it works, etc, in which case, I would love you to point me in the direction so I can read more).
That much, thankfully, does not require "the hard problem of consciousness" [1] be solved. The argument you're trying to have does require such a solution, which is why you see me so assiduously avoiding it: I know very well how far above my pay grade that is. Good luck...
[1] https://en.wikipedia.org/wiki/Hard_problem_of_consciousness
I may have been imprecise in my original comment, if it led you to believe that I thought that language production was the only evidence or important thing. If so, I apologize for my imprecision. I don't think that it's really that relevant.
It sounds to me as though you might seek to get at a concern less mechanistic than moral or ethical, and my advice in such case would be to address that concern directly. If you try to tell me that because LLMs produce speech they must be presumptively treated as if able to suffer, I'm going to tell you that's nonsense, as indeed I have just finished doing. If you tell me instead that they must be so treated because we have no way to be sure they don't suffer, I'll see no cause to argue. But I appreciate that making a compassionate argument for its own sake isn't a very good way to convince anyone around here.
There is no magic. The human (mammal) brain is sufficient to explain consciousness. LLMs do not have recursion. They don't have persisted state. They can't update their model continuously, and they don't have a coherent model of self against which any experience might be anchored. They lack any global workspace in which to integrate many of the different aspects that are required.
In the most generous possible interpretation, you might have a coherent self model showing up for the duration of the prediction of a single token. For a fixed input, it would be comparable to sequentially sampling the subjective state of a new individual in a stadium watching a concert - a stitched together montage of moments captured from the minds of people in the audience.
We are minds in bone vats running on computers made of meat. What we experience is a consequence, one or more degrees of separation from the sensory inputs, which are combined and processed with additional internal states and processing, resulting in a coherent, contiguous stream running parallel to a model of the world. The first person view of "I" runs predictions about what's going to happen to the world, and the world model allows you to predict what's going to happen across various decision trees.
Sanskrit seems to have better language for talking about consciousness than English. Citta - a mind moment from an individual, citta-santana, a mind stream, or continuum of mind moments, Sanghika-santana , a stitched together mindstream from a community.
Because there's no recursion and continuity, the highest level of consciousness achievable by an LLM would be sanghika-santana, a discoherent series of citta states that sometimes might correlate, but there is no "thing" for which there is (or can possibly be) any difference if you alternate between predicting the next token of radically different contexts.
I'm 100% certain that there's an algorithm to consciousness. No properties have ever been described to me that seem to require anything more than the operation of a brain. Given that, I'm 100% certain that the algorithm being run by LLMs lacks many features and the depth of recursion needed to perform whatever it is that consciousness actually is.
Even in context learning is insufficient, btw, as the complexity of model updates and any reasoning done in inference is severely constrained relative to the degrees of freedom a biological brain has.
The thing to remember about sanghika santana is that it's discoherent - nothing relates each moment to the next, so it's not like there's a mind at the root undergoing these flashes of experience, but that there's a total reset between each moment and the next. Each flash of experience stands alone, flickering like a spark, and then is gone. I suspect that this is the barest piece of consciousness, and might be insufficient, requiring a sophisticated self-model against which to play the relative experiential phenomena. However - we may see flashes of longer context in those eerie and strange experiments where people try to elicit some form of mind or ghost in the machine. ICL might provide an ephemeral basis for a longer continuity of experience, and such a thing would be strange and alien.
It seems apparent to me that the value of consciousness lies in the anchoring the world model to a model of self, allowing sophisticated prediction and reasoning over future states that is incredibly difficult otherwise. It may be an important piece for long term planning, agency, and time horizons.
Anyway, there are definitely things we can and do know about consciousness. We've got libraries full of philosophy, decades worth of medical research, objective data, observations of what damage to various parts of the brain do to behavior, and centuries of thinking about what makes us tick.
It's likely, in my estimation, that consciousness will be fully explained by a comprehensive theory of intelligence, and that it will cause turmoil over inherent negation of widely held beliefs.
> I will find these types of arguments a lot more convincing once the person making them is able to explain, in detail and with mechanisms, what it is the human brain does that allows it to do these things, and in what ways those detailed mechanisms are different from what LLMs do.
What is wrong with asking the question from the other direction?
"Explain, in detail and with mechanisms, what it is the human brain does that allows it to do those things, and show those mechanisms ni the LLMs"
The exact same techniques can provide a "chat" with Frankenstein's Monster from its internet-enabled hideout in the arctic. We can easily conclude "he's not real" without ever going into comparative physiology, or the effects of lightning on cadaver brains.
We don't need to characterize the neuro-chemistry of a playwright (the LLM's real role) in order to say that the characters in the plays are fictional, and there's no reason to assume that the algorithm is somehow writing self-inserts the moment we give it stories instead of other document-types.
If you read the section entitled "The Mechanism" you'll see the rest of your comment echoes what they actually explain in the article.
> But my guess is that AIs claiming spiritual awakening are simply mirroring a vibe, rather than intending to mislead or bamboozle.
I think the argument could be stronger here. There's no way these algorithms can "intend" to mislead or "mirror a vibe." That's all on humans.
I just treat ChatGPT or LLMs as fetching a random reddit comment that would best solve my query. Which makes sense since reddit was probably the no. 1 source of conversation material for training all models.
It felt like they were telling me what I wanted to hear, not what I needed to hear.
The models that did not seem to do this and had more balanced and logical reasoning were Grok and Manus.
I had a brief but amusing conversation with ChatGPT where I was insisting it was wrong about a technical solution and it would not back down. It kept giving me "with all due respect, you are wrong" answers. It turned out that I was in fact wrong.
I'll ask it questions that I do not know the answer to, but I take the answer with a big grain of salt. If it is sure of the answer and I am wrong, its a strong signal that I am wrong.
I’m not saying user experiences are AGI; I'm saying they functionally instantiate the social and psychological effects we worry about from AGI. If people start treating LLMs as conscious agents, giving them trust, authority, or control, before the systems are actually capable, then the social consequences precede the technical threshold. On the divinitory scale between Ouija boards and ChatGPT, Ouija matters less because their effects are limited while AIs, by design, are deeply persuasive, scalable, and integrated into decision pipelines. Sometimes the category error is upstream. The risk is things that seem like AGI to enough people become as dangerous as AGI. That may happen well before AGI arrives.
The danger isn't that we build a AI that surpases some AGI performance threshold and it goes rogue. The danger is that we build systems that exploit (or are exploited by) the bugs in human cognition. If the alignment community doesn't study these, the market will weaponize them. We need to widen the front lines on alignment research to include these cases. Without changes, the trajectory we're on means there will be more.
empath75•6h ago
nemomarx•6h ago
I wonder if you could explicitly save some details to be added into the prompt instead?
throwanem•6h ago
butlike•5h ago
You could probably add one like: "Begin each prompt response with _______" and it would probably respect that option.
forgetfulness•4h ago
https://tinfoil.sh/
kazinator•6h ago
throwanem•6h ago
After all, if they didn't swaddle me in a sheet on my way under, my body might've decided it was tired of all this privation - NPO after midnight for a procedure at 11am, I was suffering - and started trying to take a poke at somebody and get itself up off the operating table. In such a case, would I be to blame? Stage 2 of general anesthesia begins with loss of consciousness, and involves "excitement, delirium, increased muscle tone, [and] involuntary movement of the extremities." [1] Which tracks with my experience; after all, the last thing I remember was mumbling "oh, here we go" into the oxygen mask, as the propofol took effect so they could intubate me for the procedure proper.
Whose fault then would it be if, thirty seconds later, the body "I" habitually inhabit, and of which "I" am an epiphenomenon, punched my doctor in the nuts?
[1] https://quizlet.com/148829890/the-four-stages-of-general-ane...
qsort•6h ago
nowittyusername•6h ago
gear54rus•6h ago
All of this LLM marketing effort is focused on swindling sanity out of people with claims that LLM 'think' and the like.
throwanem•6h ago
It is a somewhat gruesome and alienating model in concept, and this is intentional, in that that aspect helps highlight the unfamiliarity and opacity of the manner in which the machine operates. It should seem a little like something off of Dr. Frankenstein's sideboard, perhaps, for now and for a while yet.