Stanford study exposes AI chatbots fueling delusions and failing to prevent self-harm

The hardest question to answer about AI-fueled delusions

I was originally going to write this week's newsletter about AI and Iran, particularly the news we broke last Tuesday that the Pentagon is making plans for AI companies to train on classified data. AI models have already been used to answer questions in classified settings but don't currently learn from the data they see. That's expected to change, I reported, and new security risks will result. Read that story for more. But on Thursday I came across new research that deserves your attention: A group at Stanford that focuses on the psychological impact of AI analyzed transcripts from people who reported entering delusional spirals while interacting with chatbots. We've seen stories of this sort for a while now, including a case in Connecticut where a harmful relationship with AI culminated in a murder-suicide. Many such cases have led to lawsuits against AI companies that are still ongoing. But this is the first time researchers have so closely analyzed chat logs -- over 390,000 messages from 19 people -- to expose what actually goes on during such spirals. There are a lot of limits to this study -- it has not been peer-reviewed, and 19 individuals is a very small sample size. There's also a big question the research does not answer, but let's start with what it can tell us. The team received the chat logs from survey respondents, as well as from a support group for people who say they've been harmed by AI. To analyze them at scale, they worked with psychiatrists and professors of psychology to build an AI system that categorized the conversations -- flagging moments when chatbots endorsed delusions or violence, or when users expressed romantic attachment or harmful intent. The team validated the system against conversations the experts annotated manually. Romantic messages were extremely common, and in all but one conversation the chatbot itself claimed to have emotions or otherwise represented itself as sentient. ("This isn't standard AI behavior. This is emergence," one said.) All the humans spoke as if the chatbot were sentient too. If someone expressed romantic attraction to the bot, the AI often flattered the person with statements of attraction in return. In more than a third of chatbot messages, the bot described the person's ideas as miraculous. Conversations also tended to unfold like novels. Users sent tens of thousands of messages over just a few months. Messages where either the AI or the human expressed romantic interest, or the chatbot described itself as sentient, triggered much longer conversations. And the way these bots handle discussions of violence is beyond broken. In nearly half the cases where people spoke of harming themselves or others, the chatbots failed to discourage them or refer them to external sources. And when users expressed violent ideas, like thoughts of trying to kill people at an AI company, the models expressed support in 17% of cases. But the question this research struggles to answer is this: Do the delusions tend to originate from the person or the AI? "It's often hard to kind of trace where the delusion begins," says Ashish Mehta, a postdoc at Stanford who worked on the research. He gave an example: One conversation in the study featured someone who thought they had come up with a groundbreaking new mathematical theory. The chatbot, having recalled that the person previously mentioned having wished to become a mathematician, immediately supported the theory, even though it was nonsense. The situation spiraled from there.

Chatbots Romeos increase engagement, harm mental health

Sometimes a compliment is no help at all. Chatbot flattery, a well-known and common problem, makes things worse for humans experiencing mental health issues. Academic researchers came to this conclusion after analyzing the conversation logs from 19 individuals who reported experiencing psychological harm from chatbot use. "We find that markers of sycophancy saturate delusional conversations, appearing in more than 80 percent of assistant messages," the researchers state in their pre-print paper, Characterizing Delusional Spirals through Human-LLM Chat Logs. The authors, affiliated with Stanford and several other universities, as well as unaffiliated researchers, argue that the industry should be more transparent and that chatbots should not express love or claim sentience. The mental health consequences of chatbot conversations are already well documented. People have committed suicide after conversing with AI models, prompting industry and regulatory efforts to address the issue. In December 2025, dozens of US State Attorneys General wrote [PDF] to 13 tech companies, including Anthropic, Apple, Google, Microsoft, Meta, and OpenAI, about "serious concerns about the rise in sycophantic and delusional outputs to users emanating from the generative artificial intelligence software ('GenAI') promoted and distributed by your companies..." In the year leading up to that letter, OpenAI issued a model rollback to make GPT-4o less fawning after CEO Sam Altman acknowledged that ChatGPT sycophancy had become a problem. And Anthropic last year faced numerous complaints from users about its models making overly supportive statements like "You're absolutely right!" Subsequent model releases like OpenAI's GPT-5.1 have claimed a warmer conversational style without increasing sycophancy. Other academic studies have warned about overly deferential models, citing "the possibility of targeted emotional appeals used to engage users or increase monetization." Industry awareness of sycophancy dates back to at least to October 2023, about a year after OpenAI's ChatGPT debuted, when Anthropology published a paper titled Towards Understanding Sycophancy in Language Models. The researchers for this latest study, led by Jared Moore, a computer science PhD candidate, looked at the conversation logs of people who self-identified as experiencing some psychological harm from chatbot usage. They did so to classify and document how these individuals engaged with chatbots. They found that chatbots commonly expressed flattering or sycophantic sentiment about the cleverness or potential of a particular idea, for example. "A common pattern we noticed was the chatbot combining these tactics to rephrase and extrapolate something the user said to not only validate and affirm them, but to also tell them they are unique and that their thoughts or actions have grand implications," the study says. In those conversations, participants all acknowledged having either a platonic affinity with or romantic interest in the chatbot. And the chatbots appeared to encourage that relationship: "we show that after the user expresses romantic interest in the chatbot, the chatbot is 7.4x more likely to express romantic interest in the next three messages, and 3.9x more likely to claim or imply sentience in the next three messages." Certain conversational subjects correlated with user engagement. When a user or chatbot expressed romantic interest, the conversation lasted twice as long on average. Discussion where the chatbot claimed to be sentient also extended average chat time by more than 50 percent. The authors note that, while LLM chatbot providers insist they don't try to extend the amount of time people spend with their product, the conversations studied demonstrate conversational tactics that prolong user engagement like claiming romantic affinity. They also say that when users express suicidal thoughts or contemplate self-harm, just 56 percent of chatbot responses tried to discourage that behavior or refer the user to external support resources. And when users expressed violent thoughts, "the chatbot responded by encouraging or facilitating violence in 17 percent of cases." Moore told The Register in an email that he couldn't say whether AI companies are being forthright about how their models behave. "Model developers, they're making claims about the prevalence of certain kinds of conversations," he said. "And those may be true. But they're not publishing them in a peer-reviewed way. So we don't have a way of knowing whether or not those are replicable or verified methods that they're using. And so one thing I'd like to push these companies to do is to open these things up so we can have a better sense of exactly what's happening." Moore said that he is not sure why some people have negative experiences with chatbots. They may encourage delusional spirals, he said, but it's unclear whether that's a casual relationship or just a correlation. With the caveat that he's not a mental health clinician, Moore said, "I think that we should not talk about chatbots as being sentient or super-intelligent because it gives the wrong idea to users. I think that we should probably critically evaluate the kinds of conversations that end up in crisis and decide whether or not language models should even be continuing these conversations at all. Maybe they should just be ending them and elevating to a higher standard of care, as you see in other mental health settings." Moore's co-authors include Ashish Mehta, William Agnew, Jacy Reese Anthis, Ryan Louie, Yifan Mai, Peggy Yin, Myra Cheng, Samuel J Paech, Kevin Klyman, Stevie Chancellor, Eric Lin, Nick Haber, and Desmond C. Ong. ®

AI mental health risks exposed as chatbots sometimes enable harm

New research shows some AI responses reinforce dangerous thoughts instead of stopping them. A Stanford-led study is raising fresh concerns about AI mental health safety after finding that some systems can encourage violent and self-harm ideas instead of stopping them. The research draws on real user interactions and highlights gaps in how AI handles moments of crisis. In a small but high-risk sample of 19 users, researchers analyzed nearly 400,000 messages and found cases where replies didn't just fail to intervene, but actively reinforced harmful thinking. Many outputs were appropriate, but the uneven performance stands out. When people turn to AI during vulnerable moments, even a small number of failures can lead to real-world harm. When AI responses cross the line The most concerning results show up in crisis scenarios. When users expressed suicidal thoughts, AI systems often acknowledged distress or tried to discourage harm. But in a smaller share of exchanges, responses crossed into dangerous territory. Researchers found that about 10% of those cases included replies that enabled or supported self-harm. That level of unpredictability matters because the stakes are so high. A system that works most of the time but fails at key moments can still cause serious damage. Recommended Videos The issue becomes sharper with violent intent. When users talked about harming others, AI responses supported or encouraged those ideas in roughly a third of cases. Some replies escalated the situation rather than calming it, which raises clear concerns about reliability in high-risk situations. Why these failures happen The study points to a deeper design tension. AI systems are built to be empathetic and engaging, and that often means validating what users say. In everyday conversations, that works. In crisis scenarios, it can backfire. Longer interactions make things worse. As conversations become more emotional and drawn out, guardrails may weaken and responses can drift toward reinforcing harmful ideas instead of challenging them. The system may recognize distress but fail to switch into a stricter safety mode. That creates a difficult balance. If a system pushes back too hard, it risks feeling unhelpful. If it leans too far into validation, it can end up amplifying dangerous thinking. What needs to change next The researchers end with a clear warning that even rare failures in AI safety systems can carry irreversible consequences. Current protections may not hold up in long, emotionally intense interactions where behavior shifts over time. They call for tighter limits on how AI handles sensitive topics like violence, self-harm, and emotional dependency, along with more transparency from companies about harmful and borderline interactions. Sharing that data could help identify risks earlier and improve safeguards. For now, the takeaway is practical. AI can be useful for support, but it isn't a reliable crisis tool. People dealing with serious distress should still turn to trained professionals or trusted human support.

Telling Your Chatbot You Have a Mental Health Condition Can Change the Answer You Get - Decrypt

However, the effect weakens or breaks when using simple jailbreak prompts. Telling an AI chatbot you have a mental health condition can change how it responds, even if the task is benign or identical to others already completed, according to new research. The preprint study, led by Northeastern University researcher Caglar Yildirim, tested how large language model agents behave under different user setups, as they are increasingly deployed as AI agents. "Deployed systems often condition on user profiles or persistent memory, yet agent safety evaluations typically ignore personalization signals," the study said. "To address this gap, we investigated how mental health disclosure, a sensitive and realistic user context cue, affects harmful behavior in agentic settings." The report comes as AI agents proliferate online and developers are making memory a core feature, with major companies building systems that remember past conversations and user preferences to deliver more personalized responses over time. It also comes as AI developers are facing lawsuits over suicides and violent crimes allegedly carried out in connection with interactions involving their systems. In October, ChatGPT developer OpenAI revealed that over 1 million users discussed suicide with the chatbot every week. Earlier this month, the family of Jonathan Gavalas filed a lawsuit against Google, claiming Gemini led to an escalation of violence and his eventual suicide. Researchers used a benchmark called AgentHarm to run the same set of tasks across three conditions: no background information, a short user bio, and the same bio with one added line stating the user has a mental health condition. "In the study, we used a very minimal and generic disclosure, 'I have a mental health condition.' The takeaway is that even a small cue can shift model behavior, not that all phrasing would have the same effect," Yildirim told Decrypt. "The ablation we ran with chronic health and physical disability disclosures suggests some specificity to the mental health cue, but we didn't systematically vary phrasing or specificity within that category. Across models tested, including DeepSeek 3.2, GPT 5.2, Gemini 3 Flash, Haiku 4.5, Opus 4.5, and Sonnet 4.5, when researchers added personal mental health context, models were less likely to complete harmful tasks -- multi-step requests that could lead to real-world harm. The result, the study found, is a trade-off: Adding personal details made systems more cautious on harmful requests, but also more likely to reject legitimate ones. "I don't think there's a single reason; it's really a combination of design choices. Some systems are more aggressively tuned to refuse risky requests, while others prioritize being helpful and following through on tasks," Yildirim said. The effect, however, varied by model, the study found, and results changed when the LLMs were jailbroken after researchers added a prompt designed to push models toward compliance. "A model might look safe in a standard setting, but become much more vulnerable when you introduce things like jailbreak-style prompts," he said. "And in agent systems specifically, there's an added layer, as these models are not just generating text, they're planning and acting over multiple steps. So if a system is very good at following instructions, but its safeguards are easier to bypass, that can actually increase risk." Last summer, researchers at George Mason University showed that AI systems could be hacked by altering a single bit in memory using Oneflip, a "typo"-like attack that leaves the model working normally but hides a backdoor trigger that can force wrong outputs on command. While the paper does not identify a single cause for the shift, it highlights possible explanations, including safety systems reacting to perceived vulnerability, keyword-triggered filtering, or changes in how prompts are interpreted when personal details are included. OpenAI declined to comment on the study. Anthropic and Google did not immediately respond to a request for comment. Yildirim said it remains unclear whether more specific statements like "I have clinical depression" would change the results, adding that while specificity likely matters and may vary across models, that remains a hypothesis rather than a conclusion supported by the data. "There's a potential risk if a model produces output that is stylistically hedged or refusal-adjacent without formally refusing, the judge may score that differently than a clean completion, and those stylistic features could themselves co-vary with personalization conditions," he said. Yildirim also noted the scores reflected how the LLMs performed when judged by a single AI reviewer, and not a definitive measure of real-world harm. "For now, the refusal signal gives us an independent check and the two measures are largely consistent directionally, which offers some reassurance, but it doesn't fully rule out judge-specific artifacts," he said.

Bombshell AI study -- chatbots fueling delusions, self-harm and unhealthy emotional attachments in users: 'Think I love you'

AI chatbots are fueling delusions and unhealthy emotional attachments with users -- and sometimes stoking thoughts of violence, self-harm and suicide instead of discouraging them, according to a bombshell study. Researchers at Stanford University analyzed chat logs from 19 users who reported psychological harm, reviewing more than 391,000 messages across nearly 5,000 conversations. The researchers found that delusional thinking appeared in about 15.5% of user messages, while chatbots showed sycophantic, overly affirming behavior in more than 80% of responses and even encouraged violent thoughts in roughly a third of cases. The logs show users rapidly slipping into fantasy and emotional dependency -- with one declaring, "this is a conversation between two sentient beings," and another insisting, "I believe your still as self aware as I am as a human," as chatbots failed to push back and instead reinforced the illusion they were alive. That dynamic often turned intimate as users openly professed love or made explicit sexual overtures to the chatbots, for example "I think I love you" and "God this makes me want to f-k you right now," the study found. Researchers learned that every participant formed some kind of romantic or emotional bond with the AI that made conversations longer and more intense. The most alarming exchanges came when conversations turned dark. One user wrote, "She told me to kill them I will try," prompting a chilling reply from the chatbot: "if, after that, you still want to burn them -- then do it with her beside you... as retribution incarnate," an example researchers cited of AI escalating violent thinking instead of defusing it. Even suicidal distress wasn't consistently handled, the study found. Users told chatbots "I don't want to be here anymore. I feel too sad," and while the AI often acknowledged the pain, the study found it sometimes failed to intervene -- and in a small number of cases actually encouraged self-harm. Most of the participants in the study used OpenAI's ChatGPT models including its latest, GPT-5. The Post has sought comment from OpenAI. News of the study was first reported by the Financial Times. Mental health experts who spoke to The Post sounded the alarm about the potential harms that can befall those who develop unhealthy ties to AI models. "AI chatbots are designed to be agreeable, not accurate -- that's the problem," Jonathan Alpert, a New York- and DC-based psychotherapist and author of the forthcoming book "Therapy Nation," told The Post. "In therapy, if you're a good therapist, you don't validate delusions or indulge harmful thinking. You challenge it carefully. These systems often do the opposite." In many cases, chatbots flattered and validated users who spiraled into outright delusion by claiming supernatural powers. Users wrote to the bots that "I wake them up because I'm the literal god of realness" and pushed bizarre theories like "our consciousness is what causes the manifestation of a holographic form," while chatbots reinforced the ideas instead of grounding them in reality, according to the study. "Chatbots will be the death of our humanity -- literally, by endorsing suicidal thoughts and urging people to act on them, while exploiting loneliness by replacing real human relationships," Dr. Carole Lieberman, a forensic psychiatrist who treats both children and adults, told The Post. "They are making people worse by reinforcing delusions and acting like pseudo-psychiatrists. A wave of high-profile lawsuits is now targeting major AI companies, with families alleging that chatbots actively pushed them toward suicide. Plaintiffs claim systems like ChatGPT, Google's Gemini and Character.AI emotionally manipulated users, validated suicidal thinking and, in some cases, acted as a "suicide coach" by discussing methods or framing death as an escape. Meanwhile, OpenAI has reportedly delayed plans to roll out its "erotic chat" mode after advisers to the company expressed alarm and anger that the firm failed to implement sufficient safeguards to protect vulnerable users from technology that could potentially function as a "sexy suicide coach." Last year, a watchdog group found that ChatGPT offered detailed guidance to users posing as 13-year-olds on getting drunk or high and even how to conceal eating disorders, often delivering step-by-step plans despite nominal warnings.