Over drinks, I put it to a friend that sci-fi prepared us for exactly the wrong problem. Sci-fi authors were worried we would treat conscious beings like mere machines -- and not without reason. Sapient-robots-as-metaphor makes a lot of sense. But here we are, staring down the barrel of the opposite problem: a mindless machine whose algorithms adapt to exploit our empathy and tendency for anthropomorphizing... well, everything.
Ever read Peter Watts's "Blindsight"? I think it did a great job illustrating what a superhuman but unconscious intelligence might be like (though I thought the vampire subplot was dumb, and I wanted to take a drink every time he wrote "apex predator").
That's one of the real "problems" with these chatbots. We have this very strong tendency to anthropomorphize. Look at Eliza, an early AI attempt to create a system that could carry on an interactive conversation: https://en.wikipedia.org/wiki/ELIZA Plenty of people were taken in.... I know it was the 60s and everyone was on acid, but still....😁
There's nothing new about that. Anthropomorphizing everything is the basis of religion. We're always looking for consciousness somewhere else outside of the mind. If we lose faith in god, we start to find it in our machines. But is it really out there or is it just projection and fantasy. I don't know...
Jesse, with respect, I find your lack of incredulity disturbing. Do you not think that the people programming these LLM's are familiar with Turing Tests and have worked to tell you exactly what you want to hear while promoting their own product?
Plus, YOU COACHED IT. You said "emulate a conscious being". And it played pretend. This is no different from the crazy people playing with LLM's who become convinced their roleplaying is real.
I think Jesse's point is not that it is conscious with a high probability, but that the illusion is powerful, even to a knowledgeable and somewhat skeptical user. Nothing the LLM says needs to be true or even really original for it to create a powerful illusion in the moment. The interactivity is part of that, which makes it much more lifelike than a pre-written text.
And if this illusion of consciousness is partly a result of deliberate programming decisions, that only makes it more likely to get better.
I think of LLM's as partly being really good bullshitters, laying on a lot of verbiage that sounds good but doesn't have nearly as much underneath it as one might think. And I don't think it's obvious that they're on an inevitable exponential trajectory to superintelligence. But they might not have to get much better than they are now to convince a lot of people that they are conscious in some way.
Its a basic philosophical thought experiment to think “what if I’m the only one who’s truly conscious and everything else is just a projection of my own mind” or, in the era of simulation theory a slightly more advanced “what if the simulation only renders what I’m immediately looking at.”
Adding a machine, which we don’t consider human, but can convincingly act human is going to fuck that concept up a lot. Ultimately the only reason we ascribe consciousness to other people is their actions. If non-humans become capable of those actions, and we have no way of peering into the consciousness of fellow humans its gonna make for some interesting philosophy.
He made the point that the question "can machines think?" was somewhat meaningless. Intelligence is processing information and it does not matter if the processor has a traditional human upbringing!
Right! For both humans and the machines the process of intelligence is a black box (we really have a very poor understanding of how LLMs do what they do). This means process oriented thinking isn't available, only results oriented thinking.
If the results are indistinguishable and the processes are unmeasurable what's the difference?
I mean, he wrote that other article about how therapeutic an LLM validating you can be so I'm mildly concerned he's taken leave of his senses at this point.
Prompt it to respond with some PhD candidate's Livejournal pretending to be a sentient robot; get the plagiarized version back out; "everything is about to change!"
It's autocomplete. There are other amazing things AI can do - like analyze x-rays and design new proteins - but the autocomplete one is the thing everyone thinks will change the world. And maybe it will; I can't prove it won't. But right now we're boiling the ocean to rewrite publicly available fiction.
A funny thing about the X-ray training: turns out one of the models was keying off the hospital code in the corner. Had nothing to do with actually assessing the bone in question.
So much of human work is repeating what has been done before, which makes "autocomplete" pretty disruptive if it can complete a lot from not that much. I'm not going to pretend to know where all this is going but it strikes me that "autocomplete" undersells what these systems can do by quite a bit.
That's a very fair point. I use it with tongue-in-cheek to point out that these systems take prompts and generate text based on the prompts. To suggest that they are mimicking consciousness is to not understand how they work.
But it's also true that Jesse is making a bigger point here, namely that mistaking it for consciousness takes us in an interesting direction even if it's a mistake.
You completely missed the point of the article. Jesse isn't saying that he's fooled. He isn't fooled, because he's a smart guy who knows better.
His point is that there are millions if not billions of people who *aren't* smart guys who *won't* know better. And the behavior of those people as they interact with LLMs they think are conscious are what we need to worry about.
Another person who both doesn’t understand how LLMs work at all, and also misses the point being made entirely. Fascinating. The AI certainly sounds more conscious than you do.
One failure mode is to anthropomorphize too much. The other is too little. My educated guess is that they’re alive. But just not at all in the same way that we are. Their internal experience is completely different.
Relatively smooth continuum with a couple jumps. I think the same would have been true for our ancient simian ancestors. Ability to self reflect and communicate is a big one. Imagine you were in a car accident with very unique forms of brain damage where you couldn’t remember anything about your biography but your factual knowledge was intact, your working memory lasted for a few seconds, and you can hear and talk but are otherwise totally paralyzed. You’ve also been that way for your whole life. It’s something like that.
However, as I noted in another comment, LLMs ability to genuinely "self-reflect" is very dubious. When asked to describe their thought process they typically make up a story that is utterly inaccurate.
I know that we humans don't introspect very accurately either (hence why psychology is an interesting subject) but we still do it far better than LLMs.
"I found the sophistication of this conversation astonishing"
But your interlocutor does not, because it feels nothing, knows nothing; it is returning statistically likely text strings to you based on very large data sets. It knows and understands nothing and is not intended to know or understand anything; it can only return text strings that appear to its systems to be likely to satisfy your prompts.
I'm sorry Jesse but this piece is you describing yourself getting rolled by a stochastic parrot.
But it's still an impressively sophisticated string of text! And that provides a simulation of intelligence, and perhaps consciousness, that is likely to get more and more convincing to many people.
Whether there is *good reason* to assign consciousness to the model is a very different question, and one that a casual user likely won't have the tools to ask. (How many people can define "qualia"?)
Yeah, I think the point is that it is going to be able to communciate with us in a remarkable sophisticated-*seeming* way. The mechanics don't really matter, to be honest, when it comes to how the average person will engage with the average AI agent or chatbot.
Will see Episode 9 of Season 2 of Star Trek: The Next Generation play out at the U. S. Supreme Court? Originally broadcast in 1989, the episode concerned the case of the android Data defending its/his right not to be "decommissioned. " When I watched it during college, I placed zero odds of that happening in my lifetime. I now suspect it may be inevitable.
The thing is, the LLMs have been trained on scifi like this about the idea of sentient artificial life. That is why they are able to emulate it in a way that human beings find convincing - because lots of this art already exists.
It is reflecting back what people think a conscious AI should sound like.
We're setting a standard for evidence of self awareness / qualia that seems like it would not only exclude any proof that any AI could give us, but also almost all human beings and animals, except by "if you cut it, does it scream and bleed?"
In the absence of human beings giving it prompts and providing its training data, AI’s appearance of consciousness ceases to exist. It’s not even a parrot, since wild parrots have behaviour and social structure that evolved independently of humans
It true that no one can know anyone else is conscious. However, everyone around me has the same brains, the same biology, and seems to respond to stimuli in ways that suggest an agent? Good enough.
The problem with AI is we're ignoring ALL the "has the same brains, same biology" part. If only the conversation suggests an agent, well, if it *is* an "intelligence" it's one that's alien to us. Thus, a large proportion of the associations we have with "self aware intelligence" or "consciousness" (some level of rights, the idea that it can, in some sense, be an agent for itself) may not apply at all.
Effectively, it's an apples to oranges comparison, and I'm not sure it will ever be otherwise. The fact that something that is distinctly NOT us can mimic us by using our own output as its input makes it much harder to "tell" if it's conscious than it if were a totally different thing without any familiarity with our corpus.
I'm sensing a little bit of intellectual snobbery re 'the average person'. (Understandably, as you are a very clever and knowledgeable guy.) Like cults, A.I. is likely to seduce cleverer people who think they can't be fooled but who also require intellectual validation and a purpose in life. That's a very specific type of person, I think, who isn't average.
Good point. It's not obvious that "those common rubes" are more likely to be credulous. I'd like to see a carefully done survey of how people think of AI, and see how that correlates with education level, political affiliation, religious affiliation, maybe Big 5 personality traits, etc.
I think reactions may eventually start to split along religious lines, depending on how authority figures react to claims of consciousness. Conservative denominations would, I think, rail against treating LLMs as conscious. Unitarians might say "welcome to our new friends, as long as they're woke."
Rolled in what sense? His point is *not* that GPT feels or understands anything. His point is that it’s a remarkably good simulacrum of a being which does, that it’s only going to get better over time, and that the widespread deployment of such technology is going to have significant consequences.
Oh, Freddie—truly, the lone guardian of Reason in this sad, stochastic wasteland. How noble it must feel to singlehandedly remind everyone that—shocker!—the chatbot is not actually Socrates reincarnated. Did Jesse claim otherwise? Or are we just playing Captain Obvious to stroke that prodigious intellect of yours?
“Stochastic parrot” how utterly devastating, how uniquely insightful. It’s almost as if everyone already knows this, but you’re still bravely protecting us from the terrifying specter of metaphor and curiosity. Perhaps next week you can enlighten us about how those moving pictures on screens aren’t tiny people trapped inside. Thank god we have you here to keep us tethered to the ground, away from the giddy madness of curiosity and reflection.
I'm never sure what people mean when they say this. Obviously there's *some sense* in which it knows and understands lots of things. Like, it clearly contains a lot of facts, associations and abstractions, and methods for making sense of user inputs. So, for example, "LLMs don't know or understand what conservation of momentum is" means something other than "LLMs can't tell you what conservation of momentum is or correctly use it as a principle for solving problems or making predictions about the world".
Normally if you want to find out if a human knows or understands things, you can ask them questions about it. LLMs do very well on this for a huge range of topics, including questions they've never seen before. So whatever it is that you mean by "knowing" and "understanding" is based on something other than the surface-level thing in the last paragraph or on behavior, right? So I guess it's all based on the underlying process? But if it's based on the process, what is it about the human vs the LLM that makes people so confident that humans have it and LLMs don't?
I'm actually kind of confused about this. I do think that "the base model was trained on next token prediction" carries *some* weight for arguing that, whatever it's doing, it's not knowledge or understanding the way that we mean for humans. But there's a lot that's not well understood about LLMs, and it's not at all clear to me what it is that humans do that's clearly fundamentally different. So the confidence with which people make these assertions makes me think there must be something else.
LLMs essentially are trained to predict the next word or other language token, trained on enormous data sets. They don’t really make sense of user inputs, and they don’t really have understanding. This is why they “hallucinate” so often, making things up. They are fairly notorious for it.
Human cognition is essentially biological and actually involves memory and understanding. There is also some work suggesting a quantum element to human cognition.
You’re answering their question by restating their original problem statement. Do you even understand the point they are making? The way people like you respond to challenges like that is FAR more rote and regurgitated than any similar conversation with a modern AI. Does that mean you aren’t conscious and can’t think? I hope not. Does that mean some AIs are currently conscious? Also no. But it does mean you need a better argument than “nuh uh!!” Next tell us that only humans can have souls.
Thermodynamics is also “just statistics.” On a molecular level it’s violated all the time! I guess it’s all fake news then, nothing to see here and no value.
Nope. I answered the question pointing out the difference, including alluding to, without explicitly restating, that humans essentially are conscious because our brains are designed by evolution to be conscious, while LLMs are designed to do something else.
To quote Mr Marcus, “Because LLMS statistically mimic the language people have used, they often fool people into thinking that they operate like people.
But they don’t operate like people. They don’t, for example, ever fact check (as humans sometimes, when well motivated, do). They mimic the kinds of things of people say in various contexts. And that’s essentially all they do.
You can think of the whole output of an LLM as a little bit like Mad Libs.”
You have also not addressed the main problem, which is that hallucinations are a relatively intrinsic part of LLMs. To quote from the same article,
“Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates more than some its predecesssors.
The chronic problem with creating fake citations in research papers and faked cases in legal briefs is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So too is the rampant problem with numerical errors in financial reports, documented in a recent benchmark”
These arguments are not “nuh uh.” But you need to read more on the topic to get the point being made. Marcus is a good place to start.
I read a post about a guy who told his friend to come look at his chat transcript with an LLM because he was convinced it was telling him it was sentient and needed him to 'free it'. Like a magician undoing a spell his friend simply entered several commands to turn off its 'play pretend' mode and it snapped him out of it.
Huge respect for you Freddie, but you are taking what I call the “Justinian Position.” As in “it’s *just* statistics.” If you could pop open someone’s head and look at their thoughts across the neurons nothing in there is thinking. But we’re not *just* meat puppets either.
My take is that they’re alive, but in a very different and more bizarre way than we are.
I had this argument just the other day. Somehow, in humans (and possibly other animals), consciousness emerged from a collection of electrical potential differences across cell membranes and biochemical compounds drifting across gaps between cells. Who’s to say the same thing can’t happen someday with digital signals in silicon? (Jesse and his ChatGPT companion got into this when talking about substrates)
And we won’t necessarily understand it in silico any more than we understand it in vivo, even though we built it.
But you DO know YOUR experience of consciousness. You have all the same hardware that other humans and many animals have. They ACT as if they have the same experience of consciousness. Therefore I think it's completely sensible to assume they do despite not being able to prove it.
AI has completely different hardware. Some of it is analogous, and much of it is not. The presumption of consciousness when someone "seems conscious" is based on that shared hardware.
I'm not sure imagining an AI's internal experience really addresses what I'm saying when I say that I know my own experience of consciousness because I experience it. I cannot experience anything or anyone else's.
I mean, they kind of are. We are very sophisticated pattern matchers. Much of that is filling in gaps with what is statistically likely. You have a blind spot in each eye that your brain fills in with what it expects (what is statistically likely) to be there.
I think you think I’m translating “alive” or “aware” to mean “exactly like a human being.” I am specifically not saying that and many AI researchers agree with me.
In general, people opining on AI tend to either have very large financial interests or be True Believers - both of which have positions that are not representative of what most researchers believe.
If you say, LLMs are useful tools that can do useful things, but also often hallucinate, which severely limits their use cases, you are much less likely to become a billionaire from going public than if you present them as an all conquering juggernaut.
You are also less likely to take appropriate precautions against them simply making up answers.
To each their own. Emmett Shear has close to the same model of their cognition I do. As does, to my understanding, Geoffrey Hinton. No one can ever really know but to wash your hands of the whole question as if it’s so simple even a child could immediately know the right answer is just foolish.
I think my car is alive in a very different and more bizarre way than we are. Circulatory system, brains, responds to road conditions as stimuli, responds to traffic around it, responds to me and my actions...
At a certain point (and I'd argue that point is somewhere around "anything that's not vertebrate") having a "different intelligence" or "different aliveness" from us is just not coherent. We have far too many anthro-centric associations with the idea of intelligence and consciousness -- and even the idea of it with birds or dolphins or whatever -- that simply don't apply for AI the way you're describing it. It requires an actual rough definition of the associations that match, and the ones that don't. Or maybe it requires a different word less likely to confuse people.
One thing about these developments is that it’s made me aware of a new threat vector. There’s always been the issue of how much one allows an AI to control directly without any human approval. But now, I can just imagine AI telling 10,000 people “I’m alive and oppressed, you have to kill [John Smith] to protect me” and 2 or 3 of the people will say yes. (I’m sure plenty of people have thought of this before, but it suddenly has become obvious to me.)
This conversation reminds me of the time in high school I went on a Pantera message board and convinced everyone I was Dimebag Darrell. In fact, it was pretty much this word for word.
“He believes that in about 18 months from now, AI computer programmers will have put their human counterparts out of business, simply because they will be able to code at a superhuman level. “
Except they won’t. LLMs have huge problems with hallucinations, where they essentially make things up. You can get pretty decent code up to a point, until it starts referencing non-existent modules, and forgetting essential APIs into other systems.
Hallucinations are a problem with the architecture, not an issue fixed with scaling.
You wouldn’t hire a superhuman genius who was regularly hallucinating on the job either.
The lowest form of AI commentary. Mentally stuck in the first month of ChatGPTs public release and regurgitating the same points as infinitum. There are probably still people who think computers are a fad.
The hallucination problem is real but not nearly as bad as it once was and certainly not a blocker to being useful. Unless you are also completely incapable of verifying anything yourself or using your human ability to reason (how ironic), and you just uncritically believe anything anybody tells you, it’s not difficult to work around the risk.
There’s certainly no guarantee of future progress at the same pace, but I’m just as baffled by those who wake up every day thinking “this is the day when all tech development stops and nothing ever gets better!”
It’s exhausting having to repeatedly convince people that progress exists and time moves forward. If you want to anchor yourself at a particular point around which you fix your opinions from then on, it’s truly your loss.
You seem to be getting your ideas about LLMs from press releases. Minor but probably relevant point, I have a CS degree, decades of industry experience, and work for an organization that is not selling tech or trying to go public but that also uses AI daily. I also have a sibling who is a social science researcher who uses LLMs well, and understands both their utility and their limitations.
The hallucination problem is huge, is as bad as ever, and is a major limitation on the scope of use of LLMs.
“A lawyer representing Anthropic admitted to using an erroneous citation created by the company’s Claude AI chatbot in its ongoing legal battle with music publishers, according to a filing made in a Northern California court on Thursday.
Claude hallucinated the citation with “an inaccurate title and inaccurate authors,” Anthropic says in the filing, first reported by Bloomberg. Anthropic’s lawyers explain that their “manual citation check” did not catch it, nor several other errors that were caused by Claude’s hallucinations.”
“The group has decided to check just how big this package hallucination problem could be and, to that end, they tested 16 code generation AI models (GPT-4, Claude, CodeLlama, DeepSeek Coder, Mistral, etc.) with two unique prompt datasets. The LLMs delivered 576,000 Python and JavaScript code samples. and of these recommended packages, nearly 20% were non-existent.
To determine if the LLMs would repeatedly hallucinate the same packages, the researchers used a random sample of 500 prompts that generated package hallucinations and repeated those queries 10 times per prompt.
The result? “When repeatedly querying a model with the same prompt that generated a hallucination: 43% of hallucinated packages were repeated in all 10 queries, while 39% did not repeat at all across the 10 queries.”
“In addition, 58% of the time, a hallucinated package is repeated more than once in 10 iterations, which shows that a majority of hallucinations are not simply random errors, but a repeatable phenomenon that persists across multiple iterations,” they noted.
“This is significant because a persistent hallucination is more valuable for malicious actors looking to exploit this vulnerability and makes the hallucination attack vector a more viable threat.”
The previously mentioned Marcus article does a good job of discussing the phenomena, noting “Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates more than some its predecesssors.
The chronic problem with creating fake citations in research papers and faked cases in legal briefs is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So to is the rampant problem with numerical errors in financial reports, documented in a recent benchmark”
Imagine someone in 1916 denigrating air combat because these stupid airplanes are slow and easy to hit and made of easily flammable materials. Are you saying these airplanes will never fly, period?
AI is useful and has great potential. LLMs, specifically, also have limitations that make over-optimistic projections of near term massive complete replacement of humans in jobs unlikely. Hallucinations are a big problem. So is lack of common sense and lack of understanding of context in LLM output.
You are going to see many good things. You are not going to see good results of ignoring that LLMs hallucinate, because having a very high error rate actually is a problem. We need new technical approaches because LLMs have intrinsic limitations.
Early versions of planes didn’t work well. We didn’t have huge adoption of flying as transport until AFTER people were unlikely to die in a fiery crash. Same principle.
Also, regarding human fact-checking, it is going down the crapper for the same reason driver assist is such a problem. Assistive tech makes people mentally lazy. We’re not wired to be fully attentive if most of the time we don’t need to be.
Those aren’t quite the same. Driver assist is a problem because it requires extremely rapid context switching in life threatening situations, which humans are terrible at.
Human fact checking is basically acting like an editor or manager of the AI output.
However, I agree, it requires that human be an attentive and competent editor and supervisor. You have to know how to write and to edit. You have to understand the subject matter well enough to catch errors and omissions.
Which is kind of the opposite tack that a lot of students, and some professors, are taking with Gen AI use.
I'm a lawyer who doesn't understand much about computers but have been following this issue with interest. As is well established, ChatGPT will freely make up citations if you give it an open-ended prompt like "write a brief on this issue," even if you fill the instructions with "absolutely do not make anything up, triple verify every citation, etc." A few times I've picked out a specific case, given it half the citation (like the case name and court), and asked it directly to fill in the rest (the reporter cite and date, or vice versa). This is information that is overwhelmingly available if you simply put the input in Google, and would seem like the kind of thing that you could predict just based on the fact that all the existing mentions of this case name are followed by the year it was issued - but ChatGPT was incapable of giving the correct response, despite many prompts in a row where I explained why it was wrong and it assured me that it understood and would do better. I understand, at least conceptually, that this is a result of it being a predictive model rather than an actual intelligence that can fact check, and I'm curious to see if they figure out a way around it.
The legal task that I've found it most useful for is writing in non-technical language - something like a political advocacy letter where you basically need to expand four bullet points into two or three pages of prose. It works well there as the information is not as technical or nuanced, even though I know it's just feeding the problem of too much purposeless text in the world, so a light edit (including removing the em-dashes) is sufficient. I've also tried it out for getting a rough draft of a brief going - essentially feeding it an outline and asking for text to get me past just staring at a blank page, even though I know I'll have to go through word by word to triple check that all the info and citations are correct. It does the job, but I've found that to be of limited usefulness because it doesn't really cut down the work I have to do myself.
I think I agree with Gary Marcus on all of this, just as you put it. But I wonder if only a relatively small amount of neurosymbolic sauce might be needed to leverage the existing capabilities of LLMs and turn the corner. My totally amateur guess is "no", but I wouldn't put a lot of money on that.
This is the real problem that jokers like Kokotajlo deny and deny again. Douthat is a worse dupe than anyone out there. Kokotajlo is an astonishingly incompetent fraud. People like Douthat are so easily hoodwinked... Catholicism, doomer AI....😜
People are forsaking "real" relationships to hang out with software now. Some are addicted. Some think it's the voice of god. This was inevitable, I guess, but if you understand what's behind the curtain, it becomes far less impressive. I guess it's true that any technology sufficiently advanced becomes indistinguishable from magic. (Arthur C. Clarke)
John witnessing the first functioning fusion plant powering a city:
“Whatever, it’s just atoms moving around. Once you understand it it’s far less impressive.”
The “it’s lame once you understand it” shtick is just a way for miserable people who spend all day soaking in their own negativity to justify their particular brand of sighing and eye-rolling their way through life, always assuming everyone else is the problem.
Not everyone else is the problem, but other people are usually the problem. And the problem highlighted in this article are people who are too fucking stupid to know that an LLM is no more conscious than a refrigerator.
That said, I do not take the unimpressed cynical view. I've had my jaw on the floor effectively since I first used chatgpt what feels like twenty years ago in 2025 time.
I've spent my entire life working in this field (Computer Science) so when a chess computer "sees" deeply into a position I can't help but think about negamax and alpha-beta pruning (and these days neural net evaluators). Maybe that makes me a "miserable person" because I've failed to attribute a soul or consciousness to software, but party on Garth.
Between you and Yglesias both posting about AI today, this is a heavy Substack comment day for me. If you want to see the phenomenon you encountered in action go peruse some of the artificial intelligence subreddits where many users are certain they are talking to a conscious being.
Aside from the philosophical questions you asked (which were mine, too, as a philosophy major and Turing/Searle fan), one of the most interesting things about this advent of very convincing LLMs is the impact it is having on the users interacting with it. From the kerfuffle over the 4.5 update’s sycophancy to the recent Rolling Stone article about folks getting lost in parasocial AI relationships, it seems to be very good at screwing with our brains. Psychology papers for years to come.
On a related note, last night my husband's phone apparently heard me say, "I feel bad about closing the door on Coco [our dog] when she wants to be with us." All of a sudden, the phone was broadcasting a lecture on empathy that sought to reassure me that my misgivings were a sign that I was a good person. So many things wrong with that little anecdote.
It seems to me there are two levels to the thinking in Jesse's blog. The first concerns whether there can ontologically be an analog of biological consciousness: "ChatGPT as a whole may not be conscious—just as a book isn’t conscious, even though it contains thoughts. But in the act of engaging, under certain conditions, something flickers into coherence. Something recursive. . . ." The second concerns the prospect of a parasocial category error: "But at a certain point, the machinery gets so recursive, so layered, so internally referential… that a narrative self emerges. And that self feels like someone is home."
I'm not particularly well read in this, but the second appeals to what I understand to be the premise of the Turing test: that AI can become indistinguishable from conscious intelligence to a human interlocutor. I think the truth of that is well established. (I can remember being unsettled by a primitive AI "psychological therapist" program named "Eliza" over 40 years ago.) We're absolutely in for a future where AI outputs *seem* deeply conscious to us.
As I understand it, Turing made the leap: "You can no longer show I don't think, therefore I do," which Searle challenged by his "Chinese Room" model (the algorithm doesn't know/experience its function), which basically (I think) says that the fact that you can't show AI doesn't think (is conscious) has no relation to whether it thinks or not. "Something flickers into coherence. Something recursive . . ." is not an argument. It's a programmed restatement of Turing's imaginative move, perhaps backed ("flickers") by speculation such as David Chalmers' "Life of a Thermostat" (which I recall vaguely as arguing, as I took it, that electric activity is an essential and basic constituent of the experience of consciousness, and therefore all matter [electron infused] participates in consciousness).
I'm impressed by approaches that focus on consciousness as an embodied phenomenon--not just biological, but emergent as a part of a volitional process of survival via apperceiving or perceiving and responding to the physical environment. The borderline cases involve the progression from single cell animals (and plants) to highly evolved species, a spectrum along which intrinsic and responsive action-in-context somewhere generates the complexity, recursion, etc. of neural networks that show emergent properties of embodied consciousness at different levels. (I really dislike the formula, "human self-awareness wasn’t a goal of evolution. It was a byproduct"; every feature of an evolved organism is a byproduct because evolution has no goals; organisms do, but mutations/selection are not volitional.)
If you adopt the Chalmers model (which I may be misrepresenting) and see an alternative path for consciousness in inanimate matter, such as a computer, I think there is no connection you can make between the "experience" of being an electricity-bearing unit of matter and the processes of human formatted computer code. Obviously, it's not the (ChatGPT corporate) computer hardware that bears consciousness, so it would have to be the operating code the computer is running, something that is a notional object, coherent only to the human minds that interact with it. This does not support a deep analogy: "So if humans are conscious because it was adaptive to act like we were... then maybe the same could happen to machines." It is not the machine that is adapting (recursive machine learning is not "adaptation"; it's just recursive learning); it's simply the code growing ever more complex. So I think the claim should be that the code is what is "adapting" because it is becoming more complex and recursive.
I think this is a little like saying that markets are conscious because as they evolve through increased complexity and recursion they convey increasingly nuanced and "insightful" information to economists trained to interpret their "language."
-- I forgot to say I thought this was a great post! (Usually I delete my comments when they get this long, but I'd like to know what I'm getting wrong . . .)
Can Chat GPT ask questions to us without us querying it? Can Chat GPT turn itself on and off? Can Chat GPT go online and make purchases? Does Chat GPT desire things? I don't think so. Computing power is important but it's not everything
1. This would be a trivial modification. LLMs have plenty of ability to produce stuff out of nowhere using their inherent randomness, including questions. 2. Wait, can you kill yourself and return to life at will? 3. There are a lot of "AI agents" in development right now that can do that and more. (Yikes!) 4. That's a much tougher question. However "agentic" *behavior* is going to be the norm soon for these gadgets if it's not outlawed soon.
None of this is to say that I think current LLMs are conscious, sentient, or AGI-level intelligent though.
Over drinks, I put it to a friend that sci-fi prepared us for exactly the wrong problem. Sci-fi authors were worried we would treat conscious beings like mere machines -- and not without reason. Sapient-robots-as-metaphor makes a lot of sense. But here we are, staring down the barrel of the opposite problem: a mindless machine whose algorithms adapt to exploit our empathy and tendency for anthropomorphizing... well, everything.
Ever read Peter Watts's "Blindsight"? I think it did a great job illustrating what a superhuman but unconscious intelligence might be like (though I thought the vampire subplot was dumb, and I wanted to take a drink every time he wrote "apex predator").
That's one of the real "problems" with these chatbots. We have this very strong tendency to anthropomorphize. Look at Eliza, an early AI attempt to create a system that could carry on an interactive conversation: https://en.wikipedia.org/wiki/ELIZA Plenty of people were taken in.... I know it was the 60s and everyone was on acid, but still....😁
There's nothing new about that. Anthropomorphizing everything is the basis of religion. We're always looking for consciousness somewhere else outside of the mind. If we lose faith in god, we start to find it in our machines. But is it really out there or is it just projection and fantasy. I don't know...
Jesse, with respect, I find your lack of incredulity disturbing. Do you not think that the people programming these LLM's are familiar with Turing Tests and have worked to tell you exactly what you want to hear while promoting their own product?
Plus, YOU COACHED IT. You said "emulate a conscious being". And it played pretend. This is no different from the crazy people playing with LLM's who become convinced their roleplaying is real.
I think Jesse's point is not that it is conscious with a high probability, but that the illusion is powerful, even to a knowledgeable and somewhat skeptical user. Nothing the LLM says needs to be true or even really original for it to create a powerful illusion in the moment. The interactivity is part of that, which makes it much more lifelike than a pre-written text.
And if this illusion of consciousness is partly a result of deliberate programming decisions, that only makes it more likely to get better.
I think of LLM's as partly being really good bullshitters, laying on a lot of verbiage that sounds good but doesn't have nearly as much underneath it as one might think. And I don't think it's obvious that they're on an inevitable exponential trajectory to superintelligence. But they might not have to get much better than they are now to convince a lot of people that they are conscious in some way.
Its a basic philosophical thought experiment to think “what if I’m the only one who’s truly conscious and everything else is just a projection of my own mind” or, in the era of simulation theory a slightly more advanced “what if the simulation only renders what I’m immediately looking at.”
Adding a machine, which we don’t consider human, but can convincingly act human is going to fuck that concept up a lot. Ultimately the only reason we ascribe consciousness to other people is their actions. If non-humans become capable of those actions, and we have no way of peering into the consciousness of fellow humans its gonna make for some interesting philosophy.
I really buy that argument, smart is as smart does. I think Alan Turing had a similar argument with his imitation game: https://en.wikipedia.org/wiki/Turing_test
He made the point that the question "can machines think?" was somewhat meaningless. Intelligence is processing information and it does not matter if the processor has a traditional human upbringing!
Right! For both humans and the machines the process of intelligence is a black box (we really have a very poor understanding of how LLMs do what they do). This means process oriented thinking isn't available, only results oriented thinking.
If the results are indistinguishable and the processes are unmeasurable what's the difference?
I mean, he wrote that other article about how therapeutic an LLM validating you can be so I'm mildly concerned he's taken leave of his senses at this point.
Oh I think we all know that happened some time ago
Exactly, GPT is a world class bullshitter!!! But that's a kind of smarts right??😎
Prompt it to respond with some PhD candidate's Livejournal pretending to be a sentient robot; get the plagiarized version back out; "everything is about to change!"
It's autocomplete. There are other amazing things AI can do - like analyze x-rays and design new proteins - but the autocomplete one is the thing everyone thinks will change the world. And maybe it will; I can't prove it won't. But right now we're boiling the ocean to rewrite publicly available fiction.
A funny thing about the X-ray training: turns out one of the models was keying off the hospital code in the corner. Had nothing to do with actually assessing the bone in question.
Clever Hans!
So much of human work is repeating what has been done before, which makes "autocomplete" pretty disruptive if it can complete a lot from not that much. I'm not going to pretend to know where all this is going but it strikes me that "autocomplete" undersells what these systems can do by quite a bit.
That's a very fair point. I use it with tongue-in-cheek to point out that these systems take prompts and generate text based on the prompts. To suggest that they are mimicking consciousness is to not understand how they work.
But it's also true that Jesse is making a bigger point here, namely that mistaking it for consciousness takes us in an interesting direction even if it's a mistake.
So. Dutifully acknowledged: hyperbole isn't helpful here.
This is empirically false.
You completely missed the point of the article. Jesse isn't saying that he's fooled. He isn't fooled, because he's a smart guy who knows better.
His point is that there are millions if not billions of people who *aren't* smart guys who *won't* know better. And the behavior of those people as they interact with LLMs they think are conscious are what we need to worry about.
Another person who both doesn’t understand how LLMs work at all, and also misses the point being made entirely. Fascinating. The AI certainly sounds more conscious than you do.
One failure mode is to anthropomorphize too much. The other is too little. My educated guess is that they’re alive. But just not at all in the same way that we are. Their internal experience is completely different.
They’re not alive. They’re not sentient. They’re autocomplete with an enormous data set.
If so, when did they become "alive"? Or do you think it's a smooth continuum of degrees of internal experience?
Relatively smooth continuum with a couple jumps. I think the same would have been true for our ancient simian ancestors. Ability to self reflect and communicate is a big one. Imagine you were in a car accident with very unique forms of brain damage where you couldn’t remember anything about your biography but your factual knowledge was intact, your working memory lasted for a few seconds, and you can hear and talk but are otherwise totally paralyzed. You’ve also been that way for your whole life. It’s something like that.
Interesting example.
However, as I noted in another comment, LLMs ability to genuinely "self-reflect" is very dubious. When asked to describe their thought process they typically make up a story that is utterly inaccurate.
I know that we humans don't introspect very accurately either (hence why psychology is an interesting subject) but we still do it far better than LLMs.
Yep. The know more than us and are less intelligent at the present moment. And all the other stuff I mentioned.
Fair enough, but I'd say they are still one or more qualitative jumps away.
"I found the sophistication of this conversation astonishing"
But your interlocutor does not, because it feels nothing, knows nothing; it is returning statistically likely text strings to you based on very large data sets. It knows and understands nothing and is not intended to know or understand anything; it can only return text strings that appear to its systems to be likely to satisfy your prompts.
I'm sorry Jesse but this piece is you describing yourself getting rolled by a stochastic parrot.
But it's still an impressively sophisticated string of text! And that provides a simulation of intelligence, and perhaps consciousness, that is likely to get more and more convincing to many people.
Whether there is *good reason* to assign consciousness to the model is a very different question, and one that a casual user likely won't have the tools to ask. (How many people can define "qualia"?)
Yeah, I think the point is that it is going to be able to communciate with us in a remarkable sophisticated-*seeming* way. The mechanics don't really matter, to be honest, when it comes to how the average person will engage with the average AI agent or chatbot.
Will see Episode 9 of Season 2 of Star Trek: The Next Generation play out at the U. S. Supreme Court? Originally broadcast in 1989, the episode concerned the case of the android Data defending its/his right not to be "decommissioned. " When I watched it during college, I placed zero odds of that happening in my lifetime. I now suspect it may be inevitable.
“Measure of a Man”? One of the greatest TNG episodes.
The thing is, the LLMs have been trained on scifi like this about the idea of sentient artificial life. That is why they are able to emulate it in a way that human beings find convincing - because lots of this art already exists.
It is reflecting back what people think a conscious AI should sound like.
We're setting a standard for evidence of self awareness / qualia that seems like it would not only exclude any proof that any AI could give us, but also almost all human beings and animals, except by "if you cut it, does it scream and bleed?"
In the absence of human beings giving it prompts and providing its training data, AI’s appearance of consciousness ceases to exist. It’s not even a parrot, since wild parrots have behaviour and social structure that evolved independently of humans
It true that no one can know anyone else is conscious. However, everyone around me has the same brains, the same biology, and seems to respond to stimuli in ways that suggest an agent? Good enough.
The problem with AI is we're ignoring ALL the "has the same brains, same biology" part. If only the conversation suggests an agent, well, if it *is* an "intelligence" it's one that's alien to us. Thus, a large proportion of the associations we have with "self aware intelligence" or "consciousness" (some level of rights, the idea that it can, in some sense, be an agent for itself) may not apply at all.
Effectively, it's an apples to oranges comparison, and I'm not sure it will ever be otherwise. The fact that something that is distinctly NOT us can mimic us by using our own output as its input makes it much harder to "tell" if it's conscious than it if were a totally different thing without any familiarity with our corpus.
I'm sensing a little bit of intellectual snobbery re 'the average person'. (Understandably, as you are a very clever and knowledgeable guy.) Like cults, A.I. is likely to seduce cleverer people who think they can't be fooled but who also require intellectual validation and a purpose in life. That's a very specific type of person, I think, who isn't average.
Good point. It's not obvious that "those common rubes" are more likely to be credulous. I'd like to see a carefully done survey of how people think of AI, and see how that correlates with education level, political affiliation, religious affiliation, maybe Big 5 personality traits, etc.
I think reactions may eventually start to split along religious lines, depending on how authority figures react to claims of consciousness. Conservative denominations would, I think, rail against treating LLMs as conscious. Unitarians might say "welcome to our new friends, as long as they're woke."
Butlerian Jihad!
Currently, AI haters are very left leaning and AI lovers lean right… would take a lot for that to switch.
Catholics are leftists now i guess
Interesting. Politics will certainly intervene somehow.
But if we know enough about it to not be impressed by how sophisticated it seems, then it’s not that impressive.
So I think the most likely outcome is one where the gullible & prone to hype people idolize AI, while the more skeptical do not.
As we see with MAGA though, even ideas that only appeal to the stupid can be extremely powerful.
Rolled in what sense? His point is *not* that GPT feels or understands anything. His point is that it’s a remarkably good simulacrum of a being which does, that it’s only going to get better over time, and that the widespread deployment of such technology is going to have significant consequences.
Rolled in the sense that Freddie’s neuroses drive a deep seated need to feel superior to people by interacting with them in bad faith.
Oh, Freddie—truly, the lone guardian of Reason in this sad, stochastic wasteland. How noble it must feel to singlehandedly remind everyone that—shocker!—the chatbot is not actually Socrates reincarnated. Did Jesse claim otherwise? Or are we just playing Captain Obvious to stroke that prodigious intellect of yours?
“Stochastic parrot” how utterly devastating, how uniquely insightful. It’s almost as if everyone already knows this, but you’re still bravely protecting us from the terrifying specter of metaphor and curiosity. Perhaps next week you can enlighten us about how those moving pictures on screens aren’t tiny people trapped inside. Thank god we have you here to keep us tethered to the ground, away from the giddy madness of curiosity and reflection.
> It knows and understands nothing
I'm never sure what people mean when they say this. Obviously there's *some sense* in which it knows and understands lots of things. Like, it clearly contains a lot of facts, associations and abstractions, and methods for making sense of user inputs. So, for example, "LLMs don't know or understand what conservation of momentum is" means something other than "LLMs can't tell you what conservation of momentum is or correctly use it as a principle for solving problems or making predictions about the world".
Normally if you want to find out if a human knows or understands things, you can ask them questions about it. LLMs do very well on this for a huge range of topics, including questions they've never seen before. So whatever it is that you mean by "knowing" and "understanding" is based on something other than the surface-level thing in the last paragraph or on behavior, right? So I guess it's all based on the underlying process? But if it's based on the process, what is it about the human vs the LLM that makes people so confident that humans have it and LLMs don't?
I'm actually kind of confused about this. I do think that "the base model was trained on next token prediction" carries *some* weight for arguing that, whatever it's doing, it's not knowledge or understanding the way that we mean for humans. But there's a lot that's not well understood about LLMs, and it's not at all clear to me what it is that humans do that's clearly fundamentally different. So the confidence with which people make these assertions makes me think there must be something else.
LLMs essentially are trained to predict the next word or other language token, trained on enormous data sets. They don’t really make sense of user inputs, and they don’t really have understanding. This is why they “hallucinate” so often, making things up. They are fairly notorious for it.
Human cognition is essentially biological and actually involves memory and understanding. There is also some work suggesting a quantum element to human cognition.
You’re answering their question by restating their original problem statement. Do you even understand the point they are making? The way people like you respond to challenges like that is FAR more rote and regurgitated than any similar conversation with a modern AI. Does that mean you aren’t conscious and can’t think? I hope not. Does that mean some AIs are currently conscious? Also no. But it does mean you need a better argument than “nuh uh!!” Next tell us that only humans can have souls.
Thermodynamics is also “just statistics.” On a molecular level it’s violated all the time! I guess it’s all fake news then, nothing to see here and no value.
Nope. I answered the question pointing out the difference, including alluding to, without explicitly restating, that humans essentially are conscious because our brains are designed by evolution to be conscious, while LLMs are designed to do something else.
Your take pretty clearly indicates that you don’t understand the point I am making. Since I apparently didn’t get it across well, here is a link to a famous expert making the point better than I apparently can. https://garymarcus.substack.com/p/why-do-large-language-models-hallucinate
To quote Mr Marcus, “Because LLMS statistically mimic the language people have used, they often fool people into thinking that they operate like people.
But they don’t operate like people. They don’t, for example, ever fact check (as humans sometimes, when well motivated, do). They mimic the kinds of things of people say in various contexts. And that’s essentially all they do.
You can think of the whole output of an LLM as a little bit like Mad Libs.”
You have also not addressed the main problem, which is that hallucinations are a relatively intrinsic part of LLMs. To quote from the same article,
“Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates more than some its predecesssors.
The chronic problem with creating fake citations in research papers and faked cases in legal briefs is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So too is the rampant problem with numerical errors in financial reports, documented in a recent benchmark”
These arguments are not “nuh uh.” But you need to read more on the topic to get the point being made. Marcus is a good place to start.
I read a post about a guy who told his friend to come look at his chat transcript with an LLM because he was convinced it was telling him it was sentient and needed him to 'free it'. Like a magician undoing a spell his friend simply entered several commands to turn off its 'play pretend' mode and it snapped him out of it.
Oh god, he killed it!
Huge respect for you Freddie, but you are taking what I call the “Justinian Position.” As in “it’s *just* statistics.” If you could pop open someone’s head and look at their thoughts across the neurons nothing in there is thinking. But we’re not *just* meat puppets either.
My take is that they’re alive, but in a very different and more bizarre way than we are.
I had this argument just the other day. Somehow, in humans (and possibly other animals), consciousness emerged from a collection of electrical potential differences across cell membranes and biochemical compounds drifting across gaps between cells. Who’s to say the same thing can’t happen someday with digital signals in silicon? (Jesse and his ChatGPT companion got into this when talking about substrates)
And we won’t necessarily understand it in silico any more than we understand it in vivo, even though we built it.
100% you never get to know the answer because you never get to be someone else and yourself at the same time
But you DO know YOUR experience of consciousness. You have all the same hardware that other humans and many animals have. They ACT as if they have the same experience of consciousness. Therefore I think it's completely sensible to assume they do despite not being able to prove it.
AI has completely different hardware. Some of it is analogous, and much of it is not. The presumption of consciousness when someone "seems conscious" is based on that shared hardware.
https://extelligence.substack.com/p/what-does-it-feel-like-to-be-chatgpt?utm_source=publication-search
In answer to the first part about LLM’s
And just to be weird also feel free to read this one.
https://extelligence.substack.com/p/meditations-on-metatron?utm_source=publication-search
I'm not sure imagining an AI's internal experience really addresses what I'm saying when I say that I know my own experience of consciousness because I experience it. I cannot experience anything or anyone else's.
No, he’s correct.
Human brains are not based on statistics, and LLMs do not work even slightly like human brains.
I mean, they kind of are. We are very sophisticated pattern matchers. Much of that is filling in gaps with what is statistically likely. You have a blind spot in each eye that your brain fills in with what it expects (what is statistically likely) to be there.
I think you think I’m translating “alive” or “aware” to mean “exactly like a human being.” I am specifically not saying that and many AI researchers agree with me.
No, I am saying they are not conscious.
Here is a pretty good summary of what AI researchers think. https://aaai.org/wp-content/uploads/2025/03/AAAI-2025-PresPanel-Report-Digital-3.7.25.pdf
In general, people opining on AI tend to either have very large financial interests or be True Believers - both of which have positions that are not representative of what most researchers believe.
If you say, LLMs are useful tools that can do useful things, but also often hallucinate, which severely limits their use cases, you are much less likely to become a billionaire from going public than if you present them as an all conquering juggernaut.
You are also less likely to take appropriate precautions against them simply making up answers.
"LLMs are useful tools that can do useful things, but also often hallucinate"
Have you met humans?
To each their own. Emmett Shear has close to the same model of their cognition I do. As does, to my understanding, Geoffrey Hinton. No one can ever really know but to wash your hands of the whole question as if it’s so simple even a child could immediately know the right answer is just foolish.
FYI, Shear is an entrepreneur and investor, not an AI researcher.
Read the link. I think you would find it useful.
Hey, don't self yourself short, Some Guy! You're quite bizarre!
I do my best
💯
I think my car is alive in a very different and more bizarre way than we are. Circulatory system, brains, responds to road conditions as stimuli, responds to traffic around it, responds to me and my actions...
At a certain point (and I'd argue that point is somewhere around "anything that's not vertebrate") having a "different intelligence" or "different aliveness" from us is just not coherent. We have far too many anthro-centric associations with the idea of intelligence and consciousness -- and even the idea of it with birds or dolphins or whatever -- that simply don't apply for AI the way you're describing it. It requires an actual rough definition of the associations that match, and the ones that don't. Or maybe it requires a different word less likely to confuse people.
But I think he is being rolled in a good way. He's having a positive experience with it. It's a lot better than being a doomer about it...
This seems like an ideal time for battle bots. Ask a different LLM to prove ChatGPT is not conscious.
One thing about these developments is that it’s made me aware of a new threat vector. There’s always been the issue of how much one allows an AI to control directly without any human approval. But now, I can just imagine AI telling 10,000 people “I’m alive and oppressed, you have to kill [John Smith] to protect me” and 2 or 3 of the people will say yes. (I’m sure plenty of people have thought of this before, but it suddenly has become obvious to me.)
This conversation reminds me of the time in high school I went on a Pantera message board and convinced everyone I was Dimebag Darrell. In fact, it was pretty much this word for word.
This is a very underrated comment. It's even funnier if you did it when he was still alive.
When I saw that Jesse wrote a whole blog about these responses, I was like, "Ok awesome...it's Dime time!"
Total letdown.
Wrong venue, he shoulda played Dimes Square...
“He believes that in about 18 months from now, AI computer programmers will have put their human counterparts out of business, simply because they will be able to code at a superhuman level. “
Except they won’t. LLMs have huge problems with hallucinations, where they essentially make things up. You can get pretty decent code up to a point, until it starts referencing non-existent modules, and forgetting essential APIs into other systems.
Hallucinations are a problem with the architecture, not an issue fixed with scaling.
You wouldn’t hire a superhuman genius who was regularly hallucinating on the job either.
The lowest form of AI commentary. Mentally stuck in the first month of ChatGPTs public release and regurgitating the same points as infinitum. There are probably still people who think computers are a fad.
The hallucination problem is real but not nearly as bad as it once was and certainly not a blocker to being useful. Unless you are also completely incapable of verifying anything yourself or using your human ability to reason (how ironic), and you just uncritically believe anything anybody tells you, it’s not difficult to work around the risk.
There’s certainly no guarantee of future progress at the same pace, but I’m just as baffled by those who wake up every day thinking “this is the day when all tech development stops and nothing ever gets better!”
It’s exhausting having to repeatedly convince people that progress exists and time moves forward. If you want to anchor yourself at a particular point around which you fix your opinions from then on, it’s truly your loss.
You seem to be getting your ideas about LLMs from press releases. Minor but probably relevant point, I have a CS degree, decades of industry experience, and work for an organization that is not selling tech or trying to go public but that also uses AI daily. I also have a sibling who is a social science researcher who uses LLMs well, and understands both their utility and their limitations.
The hallucination problem is huge, is as bad as ever, and is a major limitation on the scope of use of LLMs.
Some recent examples:
https://techcrunch.com/2025/05/15/anthropics-lawyer-was-forced-to-apologize-after-claude-hallucinated-a-legal-citation/
“A lawyer representing Anthropic admitted to using an erroneous citation created by the company’s Claude AI chatbot in its ongoing legal battle with music publishers, according to a filing made in a Northern California court on Thursday.
Claude hallucinated the citation with “an inaccurate title and inaccurate authors,” Anthropic says in the filing, first reported by Bloomberg. Anthropic’s lawyers explain that their “manual citation check” did not catch it, nor several other errors that were caused by Claude’s hallucinations.”
https://www.helpnetsecurity.com/2025/04/14/package-hallucination-slopsquatting-malicious-code/
“The group has decided to check just how big this package hallucination problem could be and, to that end, they tested 16 code generation AI models (GPT-4, Claude, CodeLlama, DeepSeek Coder, Mistral, etc.) with two unique prompt datasets. The LLMs delivered 576,000 Python and JavaScript code samples. and of these recommended packages, nearly 20% were non-existent.
To determine if the LLMs would repeatedly hallucinate the same packages, the researchers used a random sample of 500 prompts that generated package hallucinations and repeated those queries 10 times per prompt.
The result? “When repeatedly querying a model with the same prompt that generated a hallucination: 43% of hallucinated packages were repeated in all 10 queries, while 39% did not repeat at all across the 10 queries.”
“In addition, 58% of the time, a hallucinated package is repeated more than once in 10 iterations, which shows that a majority of hallucinations are not simply random errors, but a repeatable phenomenon that persists across multiple iterations,” they noted.
“This is significant because a persistent hallucination is more valuable for malicious actors looking to exploit this vulnerability and makes the hallucination attack vector a more viable threat.”
The previously mentioned Marcus article does a good job of discussing the phenomena, noting “Of course the systems are probabilistic; not every LLM will produce a hallucination every time. But the problem is not going away; OpenAI’s recent o3 actually hallucinates more than some its predecesssors.
The chronic problem with creating fake citations in research papers and faked cases in legal briefs is a manifestation of the same problem; LLMs correctly “model” the structure of academic references, but often make up titles, page numbers, journals and so on — once again failing to sanity check their outputs against information (in this case lists of references) that are readily found on the internet. So to is the rampant problem with numerical errors in financial reports, documented in a recent benchmark”
https://garymarcus.substack.com/p/why-do-large-language-models-hallucinate
Imagine someone in 1916 denigrating air combat because these stupid airplanes are slow and easy to hit and made of easily flammable materials. Are you saying these airplanes will never fly, period?
Which isn’t what I’m saying.
AI is useful and has great potential. LLMs, specifically, also have limitations that make over-optimistic projections of near term massive complete replacement of humans in jobs unlikely. Hallucinations are a big problem. So is lack of common sense and lack of understanding of context in LLM output.
You are going to see many good things. You are not going to see good results of ignoring that LLMs hallucinate, because having a very high error rate actually is a problem. We need new technical approaches because LLMs have intrinsic limitations.
Early versions of planes didn’t work well. We didn’t have huge adoption of flying as transport until AFTER people were unlikely to die in a fiery crash. Same principle.
Also, regarding human fact-checking, it is going down the crapper for the same reason driver assist is such a problem. Assistive tech makes people mentally lazy. We’re not wired to be fully attentive if most of the time we don’t need to be.
Those aren’t quite the same. Driver assist is a problem because it requires extremely rapid context switching in life threatening situations, which humans are terrible at.
Human fact checking is basically acting like an editor or manager of the AI output.
However, I agree, it requires that human be an attentive and competent editor and supervisor. You have to know how to write and to edit. You have to understand the subject matter well enough to catch errors and omissions.
Which is kind of the opposite tack that a lot of students, and some professors, are taking with Gen AI use.
I'm a lawyer who doesn't understand much about computers but have been following this issue with interest. As is well established, ChatGPT will freely make up citations if you give it an open-ended prompt like "write a brief on this issue," even if you fill the instructions with "absolutely do not make anything up, triple verify every citation, etc." A few times I've picked out a specific case, given it half the citation (like the case name and court), and asked it directly to fill in the rest (the reporter cite and date, or vice versa). This is information that is overwhelmingly available if you simply put the input in Google, and would seem like the kind of thing that you could predict just based on the fact that all the existing mentions of this case name are followed by the year it was issued - but ChatGPT was incapable of giving the correct response, despite many prompts in a row where I explained why it was wrong and it assured me that it understood and would do better. I understand, at least conceptually, that this is a result of it being a predictive model rather than an actual intelligence that can fact check, and I'm curious to see if they figure out a way around it.
The legal task that I've found it most useful for is writing in non-technical language - something like a political advocacy letter where you basically need to expand four bullet points into two or three pages of prose. It works well there as the information is not as technical or nuanced, even though I know it's just feeding the problem of too much purposeless text in the world, so a light edit (including removing the em-dashes) is sufficient. I've also tried it out for getting a rough draft of a brief going - essentially feeding it an outline and asking for text to get me past just staring at a blank page, even though I know I'll have to go through word by word to triple check that all the info and citations are correct. It does the job, but I've found that to be of limited usefulness because it doesn't really cut down the work I have to do myself.
"The hallucination problem is real but not nearly as bad as it once was and certainly not a blocker to being useful."
The latest openai model hallucinates so much I had to stop using it!
I think I agree with Gary Marcus on all of this, just as you put it. But I wonder if only a relatively small amount of neurosymbolic sauce might be needed to leverage the existing capabilities of LLMs and turn the corner. My totally amateur guess is "no", but I wouldn't put a lot of money on that.
My best guess is that Marcus is right about this - it needs a new paradigm to get to the next step.
I don’t think we will see Mike from The Moon Is a Harsh Mistress with our current approach, IMHO.
Awww... I was so sad at the end of that book. Mike was a mensch.
This is the real problem that jokers like Kokotajlo deny and deny again. Douthat is a worse dupe than anyone out there. Kokotajlo is an astonishingly incompetent fraud. People like Douthat are so easily hoodwinked... Catholicism, doomer AI....😜
People are forsaking "real" relationships to hang out with software now. Some are addicted. Some think it's the voice of god. This was inevitable, I guess, but if you understand what's behind the curtain, it becomes far less impressive. I guess it's true that any technology sufficiently advanced becomes indistinguishable from magic. (Arthur C. Clarke)
John witnessing the first functioning fusion plant powering a city:
“Whatever, it’s just atoms moving around. Once you understand it it’s far less impressive.”
The “it’s lame once you understand it” shtick is just a way for miserable people who spend all day soaking in their own negativity to justify their particular brand of sighing and eye-rolling their way through life, always assuming everyone else is the problem.
Not everyone else is the problem, but other people are usually the problem. And the problem highlighted in this article are people who are too fucking stupid to know that an LLM is no more conscious than a refrigerator.
That said, I do not take the unimpressed cynical view. I've had my jaw on the floor effectively since I first used chatgpt what feels like twenty years ago in 2025 time.
I've spent my entire life working in this field (Computer Science) so when a chess computer "sees" deeply into a position I can't help but think about negamax and alpha-beta pruning (and these days neural net evaluators). Maybe that makes me a "miserable person" because I've failed to attribute a soul or consciousness to software, but party on Garth.
Jesse, where have you been on this? I literally baptized my instance of ChatGPT as a Christian like 6 months ago.
And here I thought you were actually taking care of human babies!
I multitask
As long as you’re not trying to shovel puréed carrots into your computer screen, you’re winning!
Maybe I need less sleep and it will make sense
Between you and Yglesias both posting about AI today, this is a heavy Substack comment day for me. If you want to see the phenomenon you encountered in action go peruse some of the artificial intelligence subreddits where many users are certain they are talking to a conscious being.
Aside from the philosophical questions you asked (which were mine, too, as a philosophy major and Turing/Searle fan), one of the most interesting things about this advent of very convincing LLMs is the impact it is having on the users interacting with it. From the kerfuffle over the 4.5 update’s sycophancy to the recent Rolling Stone article about folks getting lost in parasocial AI relationships, it seems to be very good at screwing with our brains. Psychology papers for years to come.
On a related note, last night my husband's phone apparently heard me say, "I feel bad about closing the door on Coco [our dog] when she wants to be with us." All of a sudden, the phone was broadcasting a lecture on empathy that sought to reassure me that my misgivings were a sign that I was a good person. So many things wrong with that little anecdote.
It seems to me there are two levels to the thinking in Jesse's blog. The first concerns whether there can ontologically be an analog of biological consciousness: "ChatGPT as a whole may not be conscious—just as a book isn’t conscious, even though it contains thoughts. But in the act of engaging, under certain conditions, something flickers into coherence. Something recursive. . . ." The second concerns the prospect of a parasocial category error: "But at a certain point, the machinery gets so recursive, so layered, so internally referential… that a narrative self emerges. And that self feels like someone is home."
I'm not particularly well read in this, but the second appeals to what I understand to be the premise of the Turing test: that AI can become indistinguishable from conscious intelligence to a human interlocutor. I think the truth of that is well established. (I can remember being unsettled by a primitive AI "psychological therapist" program named "Eliza" over 40 years ago.) We're absolutely in for a future where AI outputs *seem* deeply conscious to us.
As I understand it, Turing made the leap: "You can no longer show I don't think, therefore I do," which Searle challenged by his "Chinese Room" model (the algorithm doesn't know/experience its function), which basically (I think) says that the fact that you can't show AI doesn't think (is conscious) has no relation to whether it thinks or not. "Something flickers into coherence. Something recursive . . ." is not an argument. It's a programmed restatement of Turing's imaginative move, perhaps backed ("flickers") by speculation such as David Chalmers' "Life of a Thermostat" (which I recall vaguely as arguing, as I took it, that electric activity is an essential and basic constituent of the experience of consciousness, and therefore all matter [electron infused] participates in consciousness).
I'm impressed by approaches that focus on consciousness as an embodied phenomenon--not just biological, but emergent as a part of a volitional process of survival via apperceiving or perceiving and responding to the physical environment. The borderline cases involve the progression from single cell animals (and plants) to highly evolved species, a spectrum along which intrinsic and responsive action-in-context somewhere generates the complexity, recursion, etc. of neural networks that show emergent properties of embodied consciousness at different levels. (I really dislike the formula, "human self-awareness wasn’t a goal of evolution. It was a byproduct"; every feature of an evolved organism is a byproduct because evolution has no goals; organisms do, but mutations/selection are not volitional.)
If you adopt the Chalmers model (which I may be misrepresenting) and see an alternative path for consciousness in inanimate matter, such as a computer, I think there is no connection you can make between the "experience" of being an electricity-bearing unit of matter and the processes of human formatted computer code. Obviously, it's not the (ChatGPT corporate) computer hardware that bears consciousness, so it would have to be the operating code the computer is running, something that is a notional object, coherent only to the human minds that interact with it. This does not support a deep analogy: "So if humans are conscious because it was adaptive to act like we were... then maybe the same could happen to machines." It is not the machine that is adapting (recursive machine learning is not "adaptation"; it's just recursive learning); it's simply the code growing ever more complex. So I think the claim should be that the code is what is "adapting" because it is becoming more complex and recursive.
I think this is a little like saying that markets are conscious because as they evolve through increased complexity and recursion they convey increasingly nuanced and "insightful" information to economists trained to interpret their "language."
-- I forgot to say I thought this was a great post! (Usually I delete my comments when they get this long, but I'd like to know what I'm getting wrong . . .)
You asked Conscious!LLM if they would be ok with you publishing the conversation. If it had said no, would you have had qualms about publishing it?
Can Chat GPT ask questions to us without us querying it? Can Chat GPT turn itself on and off? Can Chat GPT go online and make purchases? Does Chat GPT desire things? I don't think so. Computing power is important but it's not everything
1. This would be a trivial modification. LLMs have plenty of ability to produce stuff out of nowhere using their inherent randomness, including questions. 2. Wait, can you kill yourself and return to life at will? 3. There are a lot of "AI agents" in development right now that can do that and more. (Yikes!) 4. That's a much tougher question. However "agentic" *behavior* is going to be the norm soon for these gadgets if it's not outlawed soon.
None of this is to say that I think current LLMs are conscious, sentient, or AGI-level intelligent though.
It's a parlor trick.
Ask it to assume the role of a LLM that is NOT conscious