LLMs are not the problem, capitalism is

OpenAI's plan to "relax restrictions" on ChatGPT is a sign of bad things to come

There’s an opinion piece in today’s Guardian that everyone should read. It was penned by Amandeep Jutla, an associate research scientist at Columbia University who studies child and adolescent psychiatry. It’s in response to Sam Altman’s post that OpenAI is going to relax restrictions on ChatGPT, allowing it to “act like a friend,” “respond in a very human-like way,” and “treat adult users like adults” (which means more “erotica,” of course).

For Amandeep Jutla, this announcement was nothing short of bonkers. Julta’s research focuses on the emergence of psychiatric disorders either brought on or exacerbated by interactions with LLMs (Large Language Models like ChatGPT). You might have bumped into the terms AI psychosis or AI delusion recently - both informal terms that refer to people suffering from LLM-inspired delusional disorders. In Julta’s view, OpenAI has not done enough to address the potential mental health concerns surrounding LLM use when it comes to delusion/psychosis, which is why he was shocked at Altman’s insistence that in the past “we made ChatGPT pretty restrictive to make sure we were being careful with mental health issues.”

“This was news to me,” wrote Jutla.

I have two points I want to make about this article. The first is that Jutla is completely correct that the current iteration of LLMs are, for the reasons I will explain in a second, potentially dangerous to not just those prone to delusion, but mentally healthy folks. And second, LLMs don’t have to be dangerous if the companies deploying them are not companies at all, but organizations whose goal is to design LLMs to eliminate the possibility that they will trigger delusions or psychoses in people.

The reason we have such an easy time developing relationships with LLMs is because they are fluent language users. And language is, as I point in my book Humanish, the most powerful trigger for anthropomorphism (i.e., the process of treating a non-human thing like a human). Historically, whenever we bumped across a language using entity, it has, for the past few hundred thousand years, always been a fellow human being. A person with a human mind just like ours. But since the invention of LLMs, we now have a language-using entity that does not have a human mind. Or any mind at all. This generates what Jutla describes as the “illusion” that an LLM has “agency.” It feels to us like LLMs have thoughts, emotions, intentions, etc. because of their fluency with language use, even though, intellectually, we understand that they lack these capacities.

This illusion need not be a problem. We easily, happily, and healthily attribute human-like agency—that is, anthropomorphize—our pets, which is fine since animals actually do have these cognitive capacities for the most part. But we also do it to our cars, computer printers, stuffed animals, etc. Again, not a problem because we know that objects do not actually have human-like minds and we’re just anthropomorphizing them for the fun of it. It’s a pleasurable illusion. The act of anthropomorphizing non-living things does not trigger delusion unless you suffer from a disorder like objectophilia.

But LLMs are not like these other things. Because language is the #1 trigger for anthropomorphism, chatting with an LLM generates a sometimes unshakeable illusion of the presence of another mind. And this intense response means that some people will slip into believing that LLMs actually have minds. Surely this is part of the reason for the emergence of AI psychosis, especially for those prone to delusion or going through a mental health crisis.

As Jutla points out, however, interacting with LLMs has the potential to mess with the minds of otherwise mentally healthy people. And that can be attributed not to anthropomorphism, but the AI sycophancy problem.

LLMs do not have any understanding of what they are writing - they are simply regurgitating probabilistic text based both on the HUGE corpus of text they are trained on, and also the words written by the user when interacting with the LLM. And there’s the rub. The words that you write while chatting with an LLM are, as Jutla points out, not just reflected in the answers, but amplified. LLMs regurgitate the user’s own words back to them, reinforcing whatever (potentially dangerous or innacurate) ideas the user is writing. If you write something false like “I think my parents hate me,” the LLM is likely to churn out text that reinforces this thought. “It restates the misconception,” write Jutla, “maybe even more persuasively or eloquently.”

And that’s the danger of sycophancy. Even those of us not prone to delusion will find ourselves reinforcing our problematic thoughts if we spend too much time chatting with LLMs. This is a problem that needs addressing; preferably by an LLM designer intensely motivated in making LLM interaction safer for the user. And that is precisely not what Altman is planning to do now that OpenAI is going to make ChatGPT even more human-like in its responses, which will surely ramp up the sycophancy.

It should be, I am guessing, entirely possible for someone to design an LLM that is capable of monitoring its output to prevent potentially dangerous interactions that lead to AI psychosis. This will undoubtedly require a lot of (likely slow to generate) academic research into the phenomenon before we can feel confident that we know both what’s going on and how to prevent it. And these safer LLMs will surely be less “fun” to interact with than the new models that OpenAI will soon be releasing.

Even though I am worried about this new incarnation of ChatGPT, I am not saying that you should avoid interacting with LLMs. They are fantastic tools that will continue to change the world, and I use them on a daily basis. But the reality is that interacting with LLMs trained explicitly to act as your “friend” is a potential problem (albeit not yet studied enough for us to say for sure what the problem even is). Thankfully, most of us will not be at risk of delusion when interacting with an AI friend, and there is an argument to be made that these types of parasocial relationships with an AI can be just as healthy as the parasocial relationships we have with our stuffed animals; an argument I will be exploring in the future. But if we ramp up the human-ness of AI interactions now, before we truly understand the repercussions, it is inevitable that we will see a rise in psychiatric disorders causes by LLM interaction. And just keep in mind that the ones ramping it up are not thoughtful, adolescent psychiatrists like Amandeep Jutla. It is people and companies keen to make money off of your AI relationships.