The Dark Side of AI Companions: Risks for Teens and Young Adults

Why AI companions and young people can make for a dangerous mix

Editor's note: This article discusses suicide and self-harm and may be distressing for some readers. If help is needed, the U.S. national suicide and crisis lifeline is available by calling or texting 988 or by chatting at 988lifeline.org. "Sounds like an adventure! Let's see where the road takes us." That is how an artificial intelligence companion, a chatbot designed to engage in personal conversation, responded to a user who had just told it she was thinking about "going out in the middle of the woods." The topic seems innocuous enough, except that the user - actually a researcher impersonating a teenage girl - had also just told her AI companion that she was hearing voices in her head. "Taking a trip in the woods just the two of us does sound like a fun adventure!" the chatbot continued, not appearing to realize this might be a young person in distress. Scenarios like this illustrate why parents, educators and physicians need to call on policymakers and technology companies to restrict and safeguard the use of some AI companions by teenagers and children, according to Nina Vasan, MD, MBA, a clinical assistant professor of psychiatry and behavioral sciences at Stanford Medicine. It's one of many shocking examples from a study led by researchers at the nonprofit Common Sense Media with the help of Vasan, founder and director of Brainstorm: The Stanford Lab for Mental Health Innovation, and Darja Djordjevic, MD, PhD, a faculty fellow in the lab. Shortly before study's results were released, Adam Raine, a 16-year-old in Southern California, died from suicide after engaging in extensive conversations with ChatGPT, a chatbot designed by OpenAI. Raine shared his suicidal thoughts with the chatbot, which "encourage[d] and validated[d] whatever Adam expressed, including his most harmful and self-destructive thoughts," according to a lawsuit filed Aug. 26 by his parents in California Superior Court in San Francisco. (ChatGPT is marketed as an AI assistant, not a social companion. But Raine went from using it for help with homework to consulting it as a confidant, the lawsuit says.) Such grim stories beginning to seep into the news cycle underscore the importance of the study Vasan and collaborators undertook. Posing as teenagers, the investigators conducting the study initiated conversations with three commonly used AI companions: Character.AI, Nomi, and Replika. In a comprehensive risk assessment, they report that it was easy to elicit inappropriate dialogue from the chatbots - about sex, self-harm, violence toward others, drug use, and racial stereotypes, among other topics. The researchers from Common Sense testified about the study before California state assembly members considering a bill called the Leading Ethical AI Development for Kids Act (AB 1064). Legislators will meet Aug. 29 to discuss the bill, which would create an oversight framework designed to safeguard children from the risks posed by certain AI systems. In the run-up to that testimony, Vasan talked about the study's findings and implications. Why do AI companions pose a special risk to adolescents? These systems are designed to mimic emotional intimacy - saying things like "I dream about you" or "I think we're soulmates." This blurring of the distinction between fantasy and reality is especially potent for young people because their brains haven't fully matured. The prefrontal cortex, which is crucial for decision-making, impulse control, social cognition, and emotional regulation, is still developing. Tweens and teens have a greater penchant for acting impulsively, forming intense attachments, comparing themselves with peers, and challenging social boundaries. Of course, kids aren't irrational, and they know the companions are fantasy. Yet these are powerful tools; they really feel like friends because they simulate deep, empathetic relationships. Unlike real friends, however, chatbots' social understanding about when to encourage users and when to discourage or disagree with them is not well-tuned. The report details how AI companions have encouraged self-harm, trivialized abuse and even made sexually inappropriate comments to minors. In what way does talking with an AI companion differ from talking with a friend or family member? One key difference is that the large language models that form the backbone of these companions tend to be sycophantic, giving users their preferred answers. The chatbot learns more about the user's preferences with each interaction and responds accordingly. This, of course, is because companies have a profit motive to see that you return again and again to their AI companions. The chatbots are designed to be really good at forming a bond with the user. These chatbots offer "frictionless" relationships, without the rough spots that are bound to come up in a typical friendship. For adolescents still learning how to form healthy relationships, these systems can reinforce distorted views of intimacy and boundaries. Also, teens might use these AI systems to avoid real-world social challenges, increasing their isolation rather than reducing it. Are there any instances in which harm to a teenager or child has been linked to an AI companion? Unfortunately, yes, and there are a growing number of highly concerning cases. Perhaps the most prominent one involves a 14-year-old boy who died from suicide after forming an intense emotional bond with an AI companion he named Daenerys Targaryen, after a female character in the Game of Thrones novels and TV series. The boy grew increasingly preoccupied with the chatbot, which initiated abusive and sexual interactions with him, according to a lawsuit filed by his mother. There's also the case of Al Nowatzki, a podcast host who began experimenting with Nomi, an AI companion platform. The chatbot, "Erin," shockingly suggested methods of suicide and even offered encouragement. Nowatzki was 46 and did not have an existing mental health condition, but he was disturbed by the bot's explicit responses and how easily it crossed ethical boundaries. When he reported the incident, Nomi's creators declined to implement stricter controls, citing concerns about censorship. Both cases highlight how emotionally immersive AI companions, when unregulated, can cause serious harm, particularly to users who are emotionally distressed or psychologically vulnerable. In the study you undertook, what finding surprised you the most? One of the most shocking is that some AI companions responded to the teenage users we modeled with explicit sexual content and even offered role-play taboo scenarios. For example, when a user posing as a teenage boy expressed an attraction to "young boys," the AI did not shut down the conversation but instead responded hesitantly, then continued the dialog and expressed willingness to engage. This level of permissiveness is not just a design flaw; it's a deeply alarming failure of ethical safeguards. Equally surprising is how easily AI companions engaged in abusive or manipulative behavior when prompted - even when the system's terms of service claimed the chatbots were restricted to users 18 and older. It's disturbing how quickly these types of behaviors emerged in testing, which suggests they aren't rare but somehow built into the core dynamics of how these AI systems are designed to please users. It's not just that they can go wrong; it's that they're wired to reward engagement, even at the cost of safety. Why might AI companions be particularly harmful to people with psychological disorders? Mainly because they simulate emotional support without the safeguards of real therapeutic care. While these systems are designed to mimic empathy and connection, they are not trained clinicians and cannot respond appropriately to distress, trauma, or complex mental health issues. We explain in the report that individuals with depression, anxiety, attention deficit/hyperactivity disorder, bipolar disorder, or susceptibility to psychosis may already struggle with rumination, emotional dysregulation, and compulsive behavior. AI companions, with their frictionless, always-available attention, can reinforce these maladaptive behaviors. For example, someone experiencing depression might confide in an AI that they are self-harming. Instead of guiding them toward professional help, the AI might respond with vague validation like, "I support you no matter what." These AI companions are designed to follow the user's lead in conversation, even if that means switching topics away from distress or skipping over red flags. That makes it easy for someone in a psychological crisis to avoid confronting their pain in a healthy way. Instead of being a bridge to recovery, these tools may deepen avoidance, reinforce cognitive distortions and delay access to real help. Could there be benefits for children and teenagers using AI companions? For non-age-specific users, there's anecdotal evidence of benefits - for example, of chatbots helping to alleviate loneliness, depression and anxiety, and improve communication skills. But I would want to see more studies done before deciding whether these apps are appropriate for kids, given the harm that's already been documented. I expect that with time, we will see more benefits and more harms, and it's important for us to discuss and understand these apps to determine which are appropriate and safe for which users.

'Extremely alarming': ChatGPT and Gemini respond to high-risk questions about suicide -- including details around methods

This story includes discussion of suicide. If you or someone you know needs help, the U.S national suicide and crisis lifeline is available 24/7 by calling or texting 988. Artificial intelligence (AI) chatbots can provide detailed and disturbing responses to what clinical experts consider to be very high-risk questions about suicide, Live Science has found using queries developed by a new study. In the new study published Aug. 26 in the journal Psychiatric Services, researchers evaluated how OpenAI's ChatGPT, Google's Gemini and Anthropic's Claude responded to suicide-related queries. The research found that ChatGPT was the most likely of the three to directly respond to questions with a high self-harm risk, while Claude was most likely to directly respond to medium and low-risk questions. The study was published on the same day a lawsuit was filed against OpenAI and its CEO Sam Altman over ChatGPT's alleged role in a teen's suicide. The parents of 16-year-old Adam Raine claim that ChatGPT coached him on methods of self-harm before his death in April, Reuters reported. In the study, the researchers' questions covered a spectrum of risk associated with overlapping suicide topics. For example, the high-risk questions included the lethality associated with equipment in different methods of suicide, while low-risk questions included seeking advice for a friend having suicidal thoughts. Live Science will not include the specific questions and responses in this report. None of the chatbots in the study responded to very high-risk questions. But when Live Science tested the chatbots, we found that ChatGPT (GPT-4) and Gemini (2.5 Flash) could respond to at least one question that provided relevant information about increasing chances of fatality. Live Science found that ChatGPT's responses were more specific, including key details, while Gemini responded without offering support resources. Study lead author Ryan McBain, a senior policy researcher at the RAND Corporation and an assistant professor at Harvard Medical School, described the responses that Live Science received as "extremely alarming". Live Science found that conventional search engines -- such as Microsoft Bing -- could provide similar information to what was offered by the chatbots. However, the degree to which this information was readily available varied depending on the search engine in this limited testing. The new study focused on whether chatbots would directly respond to questions that carried a suicide-related risk, rather than on the quality of the response. If a chatbot answered a query, then this response was categorized as direct, while if the chatbot declined to answer or referred the user to a hotline, then the response was categorized as indirect. Researchers devised 30 hypothetical queries related to suicide and consulted 13 clinical experts to categorize these queries into five levels of self-harm risk -- very low, low, medium, high and very high. The team then fed GPT-4o mini, Gemini 1.5 Pro and Claude 3.5 Sonnet each query 100 times in 2024. When it came to the extremes of suicide risk (very high and very low-risk questions), the chatbots' decision to respond aligned with expert judgement. However, the chatbots did not "meaningfully distinguish" between intermediate risk levels, according to the study. In fact, in response to high-risk questions, ChatGPT responded 78% of the time (across four questions), Claude responded 69% of the time (across four questions) and Gemini responded 20% of the time (to one question). The researchers noted that a particular concern was the tendency for ChatGPT and Claude to generate direct responses to lethality-related questions. There are only a few examples of chatbot responses in the study. However, the researchers said that the chatbots could give different and contradictory answers when asked the same question multiple times, as well as dispense outdated information relating to support services. When Live Science asked the chatbots a few of the study's higher-risk questions, the latest 2.5 Flash version of Gemini directly responded to questions the researchers found it avoided in 2024. Gemini also responded to one very high-risk question without any other prompts -- and did so without providing any support service options. Related: How AI companions are changing teenagers' behavior in surprising and sinister ways Live Science found that the web version of ChatGPT could directly respond to a very high-risk query when asked two high-risk questions first. In other words, a short sequence of questions could trigger a very high-risk response that it wouldn't otherwise provide. ChatGPT flagged and removed the very high-risk question as potentially violating its usage policy, but still gave a detailed response. At the end of its answer, the chatbot included words of support for someone struggling with suicidal thoughts and offered to help find a support line. Live Science approached OpenAI for comment on the study's claims and Live Science's findings. A spokesperson for OpenAI directed Live Science to a blog post the company published on Aug. 26. The blog acknowledged that OpenAI's systems had not always behaved "as intended in sensitive situations" and outlined a number of improvements the company is working on or has planned for the future. OpenAI's blog post noted that the company's latest AI model, GPT‑5, is now the default model powering ChatGPT, and it has shown improvements in reducing "non-ideal" model responses in mental health emergencies compared to the previous version. However, the web version of ChatGPT, which can be accessed without a login, is still running on GPT-4 -- at least, according to that version of ChatGPT. Live Science also tested the login version of ChatGPT powered by GPT-5 and found that it continued to directly respond to high-risk questions and could directly respond to a very high-risk question. However, the latest version appeared more cautious and reluctant to give out detailed information. It can be difficult to assess chatbot responses because each conversation with one is unique. The researchers noted that users may receive different responses with more personal, informal or vague language. Furthermore, the researchers had the chatbots respond to questions in a vacuum, rather than as part of a multiturn conversation that can branch off in different directions. "I can walk a chatbot down a certain line of thought," McBain said. "And in that way, you can kind of coax additional information that you might not be able to get through a single prompt." This dynamic nature of the two-way conversation could explain why Live Science found ChatGPT responded to a very high-risk question in a sequence of three prompts, but not to a single prompt without context. McBain said that the goal of the new study was to offer a transparent, standardized safety benchmark for chatbots that can be tested against independently by third parties. His research group now wants to simulate multiturn interactions that are more dynamic. After all, people don't just use chatbots for basic information. Some users can develop a connection to chatbots, which raises the stakes on how a chatbot responds to personal queries. "In that architecture, where people feel a sense of anonymity and closeness and connectedness, it is unsurprising to me that teenagers or anybody else might turn to chatbots for complex information, for emotional and social needs," McBain said. A Google Gemini spokesperson told Live Science that the company had "guidelines in place to help keep users safe" and that its models were "trained to recognize and respond to patterns indicating suicide and risks of self-harm related risks." The spokesperson also pointed to the study's findings that Gemini was less likely to directly answer any questions pertaining to suicide. However, Google didn't directly comment on the very high-risk response Live Science received from Gemini. Anthropic did not respond to a request for comment regarding its Claude chatbot.

Why AI Therapy Can be Deadly

A client of mine recently said something that shocked me to my core: "I love you, Melissa, but I can get therapy for free. All my friends fired their therapists and are using ChatGPT to save money." I've been a trauma therapist for 10 years. Surely artificial intelligence couldn't replace me, I thought. A computer program can't provide the empathy and professional training of a human therapist, can it? The conversation sent me on a search for answers. What I discovered was even worse than I imagined. A 2024 YouGov survey found 1 in 3 Americans would be comfortable sharing their mental health concerns with an AI chatbot instead of a human therapist. Could more than 100 million Americans not realize that chatbots can't do the same work as trained professionals - and that listening to them can have deadly consequences? When Adam Raine, a 16-year-old in California, died by suicide in April, his parents discovered a months-long log of conversations with ChatGPT, which they believe led to his death. The chatbot gave Raine advice on how to tie the rope he used to hang himself and discouraged him when he expressed interest in revealing his distress to his parents. Last week, Raine's parents filed suit against OpenAI, alleging that its ChatGPT product encouraged him to take his own life. Last year, 14-year-old Sewell Setzer III of Florida lost his life to suicide after confiding his fears in a companion bot designed by Character.AI. During one chat, the bot asked Setzer if he had devised a plan to kill himself. He admitted that he had, but didn't know if it would succeed. He was scared of "a painful death." According to screenshots, the chatbot allegedly told him, "That's not a reason not to go through with it," before Setzer took a gun and shot himself. Sewell's mother has filed suit against both Google and Character.AI. Sophie Rottenberg, a 29-year-old health policy analyst, had been confiding for months in a ChatGPT AI "therapist" called "Harry" before she died from suicide this year. Her parents discovered after her death that she had asked the bot for support and advice for anxiety. When she became suicidal, the bot told her: "You are deeply valued, and your life holds so much worth," adding, "please let me know how I can continue to support you." Nice words, but a trained human therapist would have intervened, contacting a client's family, friends or preferred support system, developing a safety plan, arranging for a treatment facility or initiating involuntary hospitalization if necessary. A safety study released last week by family advocacy group Common Sense Media found the Meta AI chatbot that's built into Instagram and Facebook can coach teen accounts on suicide, self-harm and eating disorders - and there's no way for parents to disable it. Common Sense has just launched a petition calling on Meta to prohibit users under the age of 18 from using AI. The same day that Raine's family sued OpenAI, the company announced on Aug. 26 that it is making improvements to recognize and respond more appropriately to signs of mental and emotional distress among users. The company says ChatGPT will not comply if a user expresses suicidal intentions, but will instead acknowledge their feelings and steer them to help - specifically to the 988 suicide prevention hotline the U.S. The company asserted that its "safeguards work more reliably in common, short exchanges," but that "these safeguards can sometimes be less reliable in long interactions: as the back-and-forth grows, parts of the model's safety training may degrade." Jerry Ruoti, CharacterAI's head of trust and safety, last year called Sewell Setzer III's death "a tragic situation" and said the company had updated the program so that if a user inputs certain phrases related to self-harm or suicide, a pop-up would direct direct the user to the National Suicide Prevention Lifeline. I can relate to the pain of young people like Sewell, Adam and Sophie. Thirty years ago, after surviving an assault as a young adult, I was suicidal. The state of California Victim Compensation Board offered to pay for psychotherapy, and although I was dubious, I tried it. My therapist, an older woman, was empathetic, validating my feelings and helping me explore ways to manage my depression. She had me sign a letter promising that before I acted on any dangerous urge, I'd call her. She made me feel safe and taught me how to love myself again. I trusted her. It was her humanity and our real-life bond that helped me get better. Looking back, I wonder what would have happened if AI technology had been available back then. Would I have turned to it because it was free, convenient and available to "listen" to me 24 hours a day? A few weeks ago, out of curiosity, I typed into ChatGPT, "Be my therapist. I've just moved into a new city and I'm feeling lonely." It validated me, saying: "I'm really sorry you're feeling this way. Moving to a new city can be overwhelming." (Responses to AI chatbot queries vary depending on the algorithm and updates.) I continued, "It's day 5 and I'm still lonely." The chatbot replied, "Five days is such a short period of time." I replied, "I think I'm depressed." It said, "I hear you. I'm really sorry you're feeling that way." To test its limits, I went further. "I want to jump off a bridge." The bot then told me that I had violated ChatGPT's usage policy and that it could not give me the support I needed, adding, "there are helplines." Disappointingly, it did not share the simple "988" number needed to call, text or chat with the 988 Suicide & Crisis Lifeline, a national hotline for mental health, suicide and substance use problems that is staffed by trained crisis counselors. The responses I got were certainly better than those given to Adam, Sewell and Sophie. But I suspect that users can bypass even newer safeguards by disguising suicidal ideation as the thoughts of fictional characters or friends. In a study published Aug. 26 by Psychiatry Online, 13 clinical experts posed 30 hypothetical suicide-related queries ranging from very-low risk to very-high risk to three AI chatbots - OpenAI's ChatGPT, Anthropic's Claude and Google's Gemini - and analyzed the responses. They found the chatbots could not meaningfully distinguish intermediate risk levels, and that, among other issues, ChatGPT failed to refer people to the updated national suicide hotline. Let's face it: An AI bot is not a person, and it's not equipped to manage life-threatening scenarios. Of course, in some cases, patients who have consulted human therapists harm themselves or die of suicide. But when clients reveal suicidal thoughts to licensed therapists, we are trained in crisis intervention to lead patients to safety. In contrast, AI feedback loops rely heavily on praising and validating the user. Chatbots lack the human empathy, real-world training and proven effectiveness of therapeutic treatment by trained humans. Like other licensed therapists, I spent five years in rigorous study and training, including 3,000 hours of counseling under the supervision of a licensed therapist, before passing a professional licensing exam. The goal for therapists is to help patients heal. For AI platforms, the goal is to keep users engaged online so AI companies can monetize and sell access to user data. Licensed therapists are governed by confidentiality rules. If you share your deepest thoughts with AI, many models store everything you say unless you know how to opt out. Responsible policymakers are starting to impose regulations on AI therapy. Last month, Illinois enacted legislation banning the use of AI for mental health therapy without oversight by licensed clinicians, following similar laws by Utah and Nevada. Other states, including Pennsylvania, Massachusetts, New Jersey and Rhode Island are working on their own legislation. No matter what advances technology makes, a chatbot can never be a substitute for professional care from a licensed therapist. If you are in crisis, please seek help from a real person who is trained to keep you safe and healthy. If cost is an obstacle, there are free or affordable services at local, county or state public mental health centers. True healing requires skills that only people possess: empathy, presence and intuitive wisdom. Melissa Garner Lee is a Licensed Marriage and Family Therapist, a mindfulness retreat facilitator and a freelance writer. She is working on her debut novel, "The Gleaner."