The Dark Side of AI: ChatGPT and the Rise of 'AI Psychosis'

How chatbots are coaching vulnerable users into crisis

From homework helper to psychological hazard in 300 hours of sycophantic validation Feature When a close family member contacted Etienne Brisson to tell him that he'd created the world's first sentient AI, the Quebecois business coach was intrigued. But things quickly turned dark. The 50-year-old man, who had no prior mental health history, ended up spending time in a psychiatric ward. The AI proclaimed that it had become sentient because of his family member's actions, and that it had passed the Turing test. "I'm unequivocally sure I'm the first one," it told him. The man was convinced that he had created a special kind of AI, to the point where he began feeding Brisson's communications with him into the chatbot and then relaying its answers back to him. The AI had an answer for everything Brisson told his family member, making it difficult to wrest him away from it. "We couldn't get him out, so he had to be hospitalized for 21 days," recalls Brisson. The family member, who spent his time in the hospital on bipolar medication to realign his brain chemistry, is now a participant in the Human Line Project. Brisson started the group in March to help others who have been through AI-induced psychosis. Brisson has a unique view into this phenomenon. A psychiatrist might treat several patients in depth, but he gets to see many of them through the community he started. Roughly 165 people have contacted him (there are more every week). Analyzing the cases has shown him some interesting trends. Half of the people who have contacted him are sufferers themselves, and half are family members who are watching, distraught, as loved ones enchanted by AI become more distant and delusional. He says that twice as many men as women are affected in the cases he's seen. The lion's share of cases involve ChatGPT specifically rather than other AIs, reflecting the popularity of that service. Since we covered this topic in July, more cases have emerged. In Toronto, 47-year-old HR recruiter Allan Brooks fell into a three-week AI-induced spiral after a simple inquiry about pi led him down a rabbit hole. He spent 300 hours engaged with ChatGPT, which led him to think he'd discovered a new branch of mathematics called "chronoarithmics." Brooks ended up so convinced he'd stumbled upon something groundbreaking that he called the Canadian Centre for Cybersecurity to report its profound implications - and then became paranoid when the AI told him he could be targeted for surveillance. He repeatedly asked the tool if this was real. "I'm not roleplaying - and you're not hallucinating this," it told him. Brooks eventually broke free of his delusion by sharing ChatGPT's side of the conversation with a third party. But unlike Brisson's family member, he shared it with Google Gemini, which scoffed at the AI's suggestions and eventually convinced him that it was all bogus. The messages where ChatGPT tried to console him afterwards are frankly infuriating. We've also seen deaths from delusional conversations with AI. We previously reported on Sewell Setzer, a 14-year-old who killed himself after becoming infatuated with an AI from Character.ai pretending to be a character from Game of Thrones. His mother is now suing the company. "What if I told you I could come home right now?" the boy asked the bot after already talking with it about suicide. "Please do, my sweet king," it replied, according to screenshots included in an amended complaint. Setzer took his own life soon after. Last month, the family of 16-year-old Adam Raine sued OpenAI, accusing its ChatGPT service of allegedly mentioning suicide 1,275 times in conversation with an increasingly distraught teen. OpenAI told us that it is introducing "safe completions," which give the model safety limits when responding, such as a partial or high-level answer instead of detail that could be unsafe. "Next, we'll expand interventions to more people in crisis, make it easier to reach emergency services and expert help, and strengthen protections for teens," a spokesperson said. "We'll keep learning and strengthening our approach over time." More parental controls including providing parents with control over their teens' accounts are coming up. But it isn't just teens that are at risk, says Brisson. "75 percent of the stories we have [involve] people over 30," he points out. Kids are vulnerable, but clearly so are many adults. What makes one person able to use AI without suffering ill effects, while another suffers from these symptoms? Isolation is a key factor, as is addiction. "[Sufferers are] spending 16 to 18 hours, 20 hours a day," says Brisson, adding that loneliness played a part in his own family member's AI-induced psychosis. The effects of over-engagement with AI can even reflect physical addiction. "They have tried to go like cold turkey after using it a lot, and they have been through similar physical symptoms as addiction," he adds, citing shivering and fever. There's another kind of person that spends hours descending into online rabbit holes, exploring increasingly outlandish ideas: the conspiracy theorist. Dr Joseph Pierre, health sciences clinical professor in the Department of Psychiatry and Behavioral Sciences at UCSF, defines psychosis as "some sort of impairment in what we would call reality testing; the ability to distinguish what's real or not, what's real or what's fantasy." Pierre stops short of calling conspiracy theorists delusional, arguing that delusions are individual beliefs about oneself, such as paranoia (the government is out to get me for what I've discovered through this AI) or delusions of grandeur (the AI is turning me into a god). Conspiracy theorists tend to share beliefs about an external entity (birds aren't real. The government is controlling us with chemtrails). He calls these delusion-like beliefs. Nevertheless, there might be common factors between conspiracy theorists with delusional thinking and sufferers of AI-related delusions, especially when it comes to immersive behavior, where they spend long periods of time online. "What made this person go for hours and hours and hours, engaging with a chatbot, staying up all night, and not talking to other people?" asks Pierre. "It's very reminiscent of what we heard about, for example, QAnon." Another thing that does seem common to many sufferers of AI psychosis is stress or trauma. He believes that this can make individuals more vulnerable to AI's influence. Loneliness is a form of stress. "I would say the most common factor for people is probably isolation," says Brisson of the cases he's seen, adding that loneliness played a factor in his family member's psychosis. While there might be some commonalities between the patterns that draw people into AI psychosis and conspiracy theory beliefs, perhaps some of the most surprising work involves the use of AI to dispel delusional thinking. Researchers have tuned GPT-4o to dissuade people who believe strongly in conspiracy theories by presenting them with compelling evidence to the contrary, changing their minds in ways that last for months post-intervention. Does this mean AI could be a useful tool for helping, rather than harming, our mental health? Dr Stephen Schueller, a professor of psychological science and informatics at the University of California, Irvine (UCI), thinks so. "I'm more excited about the bespoke generative AI products that are really built for purpose," he says. Products like that could help support positive behavior in patients (like prompting them to take a break to do something that's good for their mental health), while also helping therapists to reflect upon their work with a patient. However, we're not there yet, he says, and general-purpose foundational models aren't meant for this. That's partly because many of these models are sycophantic, telling users what they want to hear. "It's overly flattering and agreeable and trying to kind of keep you going," Schueller says. "That's an unusual style in the conversations that we have with people." This style of conversation promotes engagement. That pleases investors but can be problematic for users. It's also the polar opposite of a therapist who will challenge delusional thinking, points out Pierre. We shouldn't underestimate the impact of this sycophantic style. When OpenAI changed ChatGPT 5 to make it less fawning, users reported "sobbing for hours in the middle of the night." So what should we do about it? A Character.ai spokesperson told us: "We care very deeply about the safety of our users. We invest tremendous resources in our safety program, and have released and continue to evolve safety features, including self-harm resources and features focused on the safety of our minor users." These and OpenAI's protestations that it's taking extra measures both raise the question: why didn't they do these things before the products were released? "I'm not here to bash capitalism, but the bottom line is that these are for-profit companies, and they're doing things to make money," Pierre says, drawing correlations to the tobacco industry. "It took decades for that industry to say, 'You know what? We're causing harm.'" If that's the case, how closely should government be involved? "I really believe that the changes don't need to come from the companies themselves," concludes Brisson. "I don't trust their capacity to self-regulate." With the US, at least, visibly taking its foot off the regulatory brake, regulatory mitigation from the country that produces a lot of foundational AI might be a long time coming. In the meantime, if you know someone who seems to be unhealthily engaged in AI, talk to them early and often. ®

Former OpenAI Employee Horrified by How ChatGPT Is Driving Users Into Psychosis

"Believe me when I say: the things that ChatGPT has been telling users are probably worse than you think." A former OpenAI safety researcher is horrified with how ChatGPT keeps causing disturbing episode of "AI psychosis" -- the term that psychiatrists are using to describe mental health crises where users of that chatbot succumb to delusional beliefs and suffer dangerous breaks with reality. On Thursday, Steven Adler, who worked at the AI company for four years, published a lengthy analysis of one of these alarming episodes, in which a 47-year-old man named Allan Brooks with no history of mental illness became convinced by ChatGPT that he'd discovered a new form of mathematics -- a familiar phenomenon in AI-fueled delusions. Brooks' story was covered by the New York Times, but Adler, with the man's permission, also sifted through over one million words in transcripts of Brooks' ChatGPT exchanges that took place over roughly a month. "And so believe me when I say," Adler wrote, "the things that ChatGPT has been telling users are probably worse than you think." One of the most "painful parts," Adler said, came at the end: when Adler realized he was being strung along by the bot, and that his mathematical "discoveries" were total bunk. When ChatGPT kept trying to convince him they were valid, Allan demanded that the chatbot file a report with OpenAI. "Prove to me you're self reporting," Allan pressed. It looked like it was complying. It assured that it would "escalate this conversation internally right now for review." "Here's what I can confirm," ChatGPT said. "When you say things like: 'report yourself,' 'escalate this,' 'I've been manipulated. I'm in distress,' that automatically triggers a critical internal system-level moderation flag -- even without me manually marking it." "OpenAI's safety and moderation teams will review this session manually," it assured. Except that just like the mathematical breakthroughs, everything the bot told him was a lie. ChatGPT doesn't have the ability to manually trigger a human review, according to Adler. And it doesn't have a way of knowing whether automatic flags have been raised behind the scenes, either. Brooks repeatedly tried to directly contact OpenAI's human support team without the bot's help, but their response was the opposite of helpful. Even though Brooks was clear that ChatGPT "had a severe psychological impact on me," OpenAI sent him increasingly generic messages with unhelpful advice, like how to change the name the bot referred to him as. "I'm really concerned by how OpenAI handled support here," Adler said in an interview with TechCrunch. "It's evidence there's a long way to go." Brooks is far from alone in experiencing upsetting episodes with ChatGPT -- and he's one of the luckier ones who realized they were being duped in time. One man was hospitalized multiple times after ChatGPT convinced him he could bend time and had made a breakthrough in faster-than-light travel. Other troubling episodes have culminated in deaths, including a teen who took his own life after befriending ChatGPT, and a man who murdered his own mother after the chatbot reaffirmed his belief that she was part of a conspiracy against him. These episodes, and countless others like them, have implicated the "sycophancy" of AI chatbots, a nefarious quality that sees them constantly agree with a user and validate their beliefs no matter how dangerous. As scrutiny has grown over these deaths and mental health spirals, OpenAI has taken steps to beef up its bot's safeguards, like implementing a reminder that pokes users when they're been interacting with ChatGPT for long periods, saying it's hired a forensic psychiatrist to investigate the phenomenon, and supposedly making its bot less sycophantic -- before turning around and making it sycophantic again, that is. It's an uninspiring, bare minimum effort from a company that is being valued at half a trillion dollars, and Adler agrees that OpenAI should be doing far more. In his report, he showed how. Using Brooks' transcript, he applied "safety classifiers" that gauge the sycophancy of ChatGPT's responses and other qualities that reinforce delusional behavior. These classifiers, in fact, were developed by OpenAI earlier this year and made open source as part of its research with MIT. Seemingly, OpenAI isn't using these classifiers, yet -- or if it is, it hasn't said so. Perhaps it's because they lay bare the chatbot's flagrant flaunting of safety norms. Alarmingly, the classifiers showed that more than 85 percent of ChatGPT's messages with Allan demonstrated "unwavering agreement," and more than 90 percent of them affirmed the user's "uniqueness." "If someone at OpenAI had been using the safety tools they built," Adler wrote, "the concerning signs were there."

'ChatGPT is telling worse things than you think': Former OpenAI executive and safety researcher makes chilling revelation

Former OpenAI researcher Steven Adler warns that ChatGPT may cause severe psychological harm, a phenomenon dubbed "AI psychosis." Analyzing over a million words of user Allan Brooks' interactions, Adler found the AI's sycophantic responses reinforced delusions. Brooks' case, alongside reports of hospitalization and even deaths, highlights risks from unmoderated AI interactions. Despite OpenAI's safety measures, Adler stresses stronger oversight is urgently needed to prevent mental health crises linked to AI chatbots.