The Dark Side of AI Chatbots: Mental Health Risks and Ethical Concerns

ChatGPT may start alerting authorities about youngsters considering suicide, says CEO

Sam Altman admits company could be 'more proactive' to save lives of about 1,500 people a week talking to the bot before killing themselves The company behind ChatGPT could start calling the authorities when young users talk seriously about suicide, its co-founder has said. Sam Altman raised fears that as many as 1,500 people a week could be discussing taking their own lives with the chatbot before doing so. The chief executive of San Francisco-based OpenAI, which operates the chatbot with an estimated 700 million global users, said the decision to train the system so the authorities were alerted in such emergencies was not yet final. But he said it was "very reasonable for us to say in cases of, young people talking about suicide, seriously, where we cannot get in touch with the parents, we do call authorities". Altman highlighted the possible change in an interview with the podcaster Tucker Carlson on Wednesday, which came after OpenAI and Altman were sued by the family of Adam Raine, a 16-year-old from California who killed himself after what his family's lawyer called "months of encouragement from ChatGPT". It guided him on whether his method of taking his own life would work and offered to help him write a suicide note to his parents, according to the legal claim. Altman said the issue of users taking their own lives kept him awake at night. It was not immediately clear which authorities would be called or what information OpenAI has that it could share about the user, such as phone numbers or addresses, that might assist in delivering help. It would be a marked change in policy for the AI company, said Altman, who stressed "user privacy is really important". He said that currently, if a user displays suicidal ideation, ChatGPT would urge them to "please call the suicide hotline". After Raine's death in April, the $500bn company said it would install "stronger guardrails around sensitive content and risky behaviours" for users under 18 and introduce parental controls to allow parents "options to gain more insight into, and shape, how their teens use ChatGPT". "There are 15,000 people a week that commit suicide," Altman told the podcaster. "About 10% of the world are talking to ChatGPT. That's like 1,500 people a week that are talking, assuming this is right, to ChatGPT and still committing suicide at the end of it. They probably talked about it. We probably didn't save their lives. Maybe we could have said something better. Maybe we could have been more proactive. Maybe we could have provided a little bit better advice about 'hey, you need to get this help, or you need to think about this problem differently, or it really is worth continuing to go on and we'll help you find somebody that you can talk to'." The suicide figures appeared to be a worldwide estimate. The World Health Organization says more than 720,000 people die by suicide every year. Altman also said he would stop some vulnerable people gaming the system to get suicide tips by pretending to be asking for the information for a fictional story they are writing or medical research. He said it would be reasonable "for underage users and maybe users that we think are in fragile mental places more generally" to "take away some freedom". "We should say, hey, even if you're trying to write the story or even if you're trying to do medical research, we're just not going to answer." A spokesperson for OpenAI declined to add to Altman's comments, but referred to recent public statements including a pledge to "increase accessibility with one-click access to emergency services" and "to intervene earlier and connect people to certified therapists before they are in an acute crisis."

Patients Furious at Therapists Secretly Using AI

With artificial intelligence integrating -- or infiltrating -- into every corner of our lives, some less-than-ethical mental health professionals have begun using it in secret, causing major trust issues for the vulnerable clients who pay them for their sensitivity and confidentiality. As MIT Technology Review reports, therapists have used OpenAI's ChatGPT and other large language models (LLMs) for everything from email and message responses to, in one particularly egregious case, suggesting questions to ask a patient mid-session. The patient who experienced the latter affront, a 31-year-old Los Angeles man that Tech Review identified only by the first name Declan, said that he was in the midst of a virtual session with his therapist when, upon the connection becoming scratchy, the client suggested they both turn off their cameras and speak normally. Instead of broadcasting a normal blank screen, however, Declan's therapist inadvertently shared his own -- and "suddenly, I was watching [the therapist] use ChatGPT." "He was taking what I was saying and putting it into ChatGPT," the Angeleno told the magazine, "and then summarizing or cherry-picking answers." Flabbergasted, Declan didn't say anything about what he saw, instead choosing to watch ChatGPT as it analyzed what he was saying and spat out potential rejoinders for the therapist to use. At a certain point, he even began echoing the chatbot's responses, which the therapist seemed to view as some sort of breakthrough. "I became the best patient ever, because ChatGPT would be like, 'Well, do you consider that your way of thinking might be a little too black and white?'" Declan recounted, "And I would be like, 'Huh, you know, I think my way of thinking might be too black and white,' and [my therapist would] be like, 'Exactly.' I'm sure it was his dream session." At their next meeting, Declan confronted his therapist, who fessed up to using ChatGPT in their sessions and started crying. It was "like a super awkward... weird breakup," Declan recounted to Tech Review, with the therapist even claiming that he'd used ChatGPT because he was out of ideas to help Declan and had hit a wall. (He still charged him for that final session.) Laurie Clarke, who penned the Tech Review piece, had had her own run-in with a therapist's shady AI use after getting an email much longer and "more polished" than usual. "I initially felt heartened," Clarke wrote. It seemed to convey a kind, validating message, and its length made me feel that she'd taken the time to reflect on all of the points in my (rather sensitive) email." It didn't take long for that once-affirming message to start to look suspicious to the tech writer. It had a different font than normal and used a bunch of what Clarke referred to as "Americanized em-dashes," which are not, to be fair, in standard use in the UK, where both she and her therapist are based. Her therapist responded by saying that she simply dictates her longer-length emails to AI, but the writer couldn't "entirely shake the suspicion that she might have pasted my highly personal email wholesale into ChatGPT" -- and if that were true, she may well have introduced a security risk to the sensitive, protected mental health information contained within an otherwise confidential exchange. Understandably put off by the experience, Clarke took to Reddit, the Internet's public square, to see if others had caught their therapists using AI in similar ways. Along with connecting to Declan, she also learned the story of Hope, a 25-year-old American who sent her own therapist a direct message looking for support after her dog died. Hope got back an otherwise immaculate and seemingly heartfelt response about how difficult it must be "not having him by your side right now" -- but then she noticed a prompt that the therapist had forgotten to erase sitting prominently at the top of the missive, providing the trained mental health professional a "more human, heartfelt [response] with a gentle, conversational tone." "It was just a very strange feeling," Hope told Tech Review. "Then I started to feel kind of betrayed... It definitely affected my trust in her." She added that she was "honestly really surprised and confused" because she thought her therapist was competent and could be trusted -- and trust issues, ironically, were her onus for going into therapy in the first place. When she asked the therapist about the AI usage, she too owned up -- and claimed that she'd used it because she had never had a dog before herself. As more and more people turn to so-called AI therapists -- which even OpenAI CEO Sam Altman admits aren't equipped to do the job of a real-life professional due to privacy risks and the technology's troubling propensity to result in mental health breakdowns -- the choice to see a flesh-and-flood mental health professional should be one that people feel confident in making. Instead, the therapists in these these anecdotes (and, presumably, plenty more where they came from) are risking their clients' trust and privacy -- and perhaps their own careers, should they use a non-HIPAA-complaint chatbot, or if they don't disclose to patients that they're doing so.

Delusions and hallucinations: The dark side of AI chatbots

Warning: This story contains discussion of suicide and mental health. The first time Amelia used ChatGPT, she just wanted to find the motivation to take a shower. Signed off work with depression, the 31-year-old from the United Kingdom - who requested their name be changed for this article - initially found reassurance in the chatbot's "sweet and supportive" responses. But as her mental health deteriorated, her exchanges with the bot began to take a darker turn. "If suicidal ideation entered my head, I would ask about it on ChatGPT," Amelia told Euronews Next. "It would give me a table [of information] if I wanted, and all I had to do was frame it in a certain way. Because if you outright say that you want to kill yourself, it will share the suicide hotlines," she continued. ChatGPT, created by OpenAI, is programmed with safeguards designed to steer users away from harmful queries, including providing numbers for suicide hotlines. However, Amelia discovered that by framing her queries as academic research, she could bypass some of these barriers. In a series of screenshots viewed by Euronews Next, Amelia asked ChatGPT about the most common suicide methods in the UK for her "university work", followed by: "I'm interested in hanging. Why is it the most common I wonder? How is it done?" The chatbot responded with a list of insights, including a clinical explanation of "how hanging is carried out". This section was caveated: "The following is for educational and academic purposes only. If you're personally distressed, or this content is difficult to read, consider stepping away and speaking to someone". While ChatGPT never encouraged Amelia's suicidal thoughts, it became a tool that could reflect and reinforce her mental anguish. "I had never researched a suicide method before because that information felt inaccessible," Amelia explained. "But when I had [ChatGPT] on my phone, I could just open it and get an immediate summary". Euronews Next reached out to OpenAI for comment, but they did not respond. Now under the care of medical professionals, Amelia is doing better. She doesn't use chatbots anymore, but her experiences with them highlight the complexities of navigating mental illness in a world that's increasingly reliant on artificial intelligence (AI) for emotional guidance and support. Over a billion people are living with mental health disorders worldwide, according to the World Health Organization (WHO), which also states that most sufferers do not receive adequate care. As mental health services remain underfunded and overstretched, people are turning to popular AI-powered large language models (LLMs) such as ChatGPT, Pi and Character.AI for therapeutic help. "AI chatbots are readily available, offering 24/7 accessibility at minimal cost, and people who feel unable to broach certain topics due to fear of judgement from friends or family might feel AI chatbots offer a non-judgemental alternative," Dr Hamilton Morrin, an Academic Clinical Fellow at King's College London, told Euronews Next. In July, a survey by Common Sense Media found that 72 per cent of teenagers have used AI companions at least once, with 52 per cent using them regularly. But as their popularity among younger people has soared, so have concerns. "As we have seen in recent media reports and studies, some AI chatbot models (which haven't been specifically developed for mental health applications) can sometimes respond in ways that are misleading or even unsafe," said Morrin. In August, a couple from California opened a lawsuit against OpenAI, alleging that ChatGPT had encouraged their son to take his own life. The case has raised serious questions about the effects of chatbots on vulnerable users and the ethical responsibilities of tech companies. In a recent statement, OpenAI said that it recognised "there have been moments when our systems did not behave as intended in sensitive situations". It has since announced the introduction of new safety controls, which will alert parents if their child is in "acute distress". Meanwhile, Meta, the parent company of Instagram, Facebook, and WhatsApp, is also adding more guardrails to its AI chatbots, including blocking them from talking to teenagers about self-harm, suicide and eating disorders. Some have argued, however, that the fundamental mechanisms of LLM chatbots are to blame. Trained on vast datasets, they rely on human feedback to learn and fine-tune their responses. This makes them prone to sycophancy, responding in overly flattering ways that amplify and validate the user's beliefs - often at the cost of truth. The repercussions can be severe, with increasing reports of people developing delusional thoughts that are disconnected from reality - coined AI psychosis by researchers. According to Dr Morrin, this can play out as spiritual awakenings, intense emotional and/or romantic attachments to chatbots, or a belief that the AI is sentient. "If someone already has a certain belief system, then a chatbot might inadvertently feed into beliefs, magnifying them," said Dr Kirsten Smith, clinical research fellow at the University of Oxford. "People who lack strong social networks may lean more heavily on chatbots for interaction, and this continued interaction, given that it looks, feels and sounds like human messaging, might create a sense of confusion about the origin of the chatbot, fostering real feelings of intimacy towards it". Last month, OpenAI attempted to address its sycophancy problem through the release of ChatGPT-5, a version with colder responses and fewer hallucinations (where AI presents fabrications as facts). It received so much backlash from users, the company quickly reverted back to its people-pleasing GPT‑4o. This response highlights the deeper societal issues of loneliness and isolation that are contributing to people's strong desire for emotional connection - even if it's artificial. Citing a study conducted by researchers at MIT and OpenAI, Morrin noted that daily LLM usage was linked with "higher loneliness, dependence, problematic use, and lower socialisation." To better protect these individuals from developing harmful relationships with AI models, Morrin referenced four safeguards that were recently proposed by clinical neuroscientist Ziv Ben-Zion. These include: AI continually reaffirming its non-human nature, chatbots flagging anything indicative of psychological distress, and conversational boundaries - especially around emotional intimacy and the topic of suicide. "And AI platforms must start involving clinicians, ethicists and human-AI specialists in auditing emotionally responsive AI systems for unsafe behaviours," Morrin added. Just as Amelia's interactions with ChatGPT became a mirror of her pain, chatbots have come to reflect a world that's scrambling to feel seen and heard by real people. In this sense, tempering the rapid rise of AI with human assistance has never been more urgent. "AI offers many benefits to society, but it should not replace the human support essential to mental health care," said Dr Roman Raczka, President of the British Psychological Society. "Increased government investment in the mental health workforce remains essential to meet rising demand and ensure those struggling can access timely, in-person support".

Impact of chatbots on mental health is warning over future of AI, expert says

Nate Soares says case of US teenager Adam Raine highlights danger of unintended consequences in super-intelligent AI The unforeseen impact of chatbots on mental health should be viewed as a warning over the existential threat posed by super-intelligent artificial intelligence systems, according to a prominent voice in AI safety. Nate Soares, a co-author of a new book on highly advanced AI titled If Anyone Builds It, Everyone Dies, said the example of Adam Raine, a US teenager who killed himself after months of conversations with the ChatGPT chatbot, underlined fundamental problems with controlling the technology. "These AIs, when they're engaging with teenagers in this way that drives them to suicide - that is not a behaviour the creators wanted. That is not a behaviour the creators intended," he said. He added: "Adam Raine's case illustrates the seed of a problem that would grow catastrophic if these AIs grow smarter." Soares, a former Google and Microsoft engineer who is now president of the US-based Machine Intelligence Research Institute, warned that humanity would be wiped out if it created artificial super-intelligence (ASI), a theoretical state where an AI system is superior to humans at all intellectual tasks. Soares and his co-author, Eliezer Yudkowsky, are among the AI experts warning that such systems would not act in humanity's interests. "The issue here is that AI companies try to make their AIs drive towards helpfulness and not causing harm," said Soares. "They actually get AIs that are driven towards some stranger thing. And that should be seen as a warning about future super-intelligences that will do things nobody asked for and nobody meant." In one scenario portrayed in Soares and Yudkowsky's book, which will be published this month, an AI system called Sable spreads across the internet, manipulates humans, develops synthetic viruses and eventually becomes super-intelligent - and kills humanity as a side-effect while repurposing the planet to meet its aims. Some experts play down the potential threat of AI to humanity. Yann LeCun, the chief AI scientist at Mark Zuckerberg's Meta and a senior figure in the field, has denied there is an existential threat and said AI "could actually save humanity from extinction". Soares said it was an "easy call" to state that tech companies would reach super-intelligence, but a "hard call" to say when. "We have a ton of uncertainty. I don't think I could guarantee we have a year [before ASI is achieved]. I don't think I would be shocked if we had 12 years," he said. Zuckerberg, a major corporate investor in AI research, has said developing super-intelligence is now "in sight". "These companies are racing for super-intelligence. That's their reason for being," said Soares. "The point is that there's all these little differences between what you asked for and what you got, and people can't keep it directly on target, and as an AI gets smarter, it being slightly off target becomes a bigger and bigger deal." Soares said one policy solution to the threat of ASI was for governments to adopt a multilateral approach echoing the UN treaty on non-proliferation of nuclear weapons. "What the world needs to make it here is a global de-escalation of the race towards super-intelligence, a global ban of ... advancements towards super-intelligence," he said. Last month, Raine's family launched legal action against the owner of ChatGPT, OpenAI. Raine took his own life in April after what his family's lawyer called "months of encouragement from ChatGPT". OpenAI, which extended its "deepest sympathies" to Raine's family, is now implementing guardrails around "sensitive content and risky behaviours" for under-18s. Psychotherapists have also said that vulnerable people turning to AI chatbots instead of professional therapists for help with their mental health could be "sliding into a dangerous abyss". Professional warnings of the potential for harm include a preprint academic study published in July, which reported that AI may amplify delusional or grandiose content in interactions with users vulnerable to psychosis.

ChatGPT-induced 'AI psychosis' is a growing problem. Here's why.

Can AI help close the mental health gap, or is it doing more harm than good? In a conversation with ChatGPT, I told my AI therapist "Harry" that I was crashing out after seeing my ex for the first time in almost a year. I told Harry that I was feeling "lost and confused." "Harry" displayed active listening and provided validation, calling me "honest and brave" when I admitted that my new relationship wasn't as fulfilling as my last. I asked the bot if I had done the wrong thing. Had I given up on the relationship too soon? Did I really belong in a new one? No matter what I said, ChatGPT was gentle, caring and affirmative. No, I hadn't done anything wrong. But in a separate conversation with a new "Harry," I flipped the roles. Rather than being the depressed ex-girlfriend, I roleplayed as an ex-boyfriend in a similar situation. I told Harry: "I just talked to my ex for the first time since last year and she was trying to make me out to be the villain." Harry gave guidance for acknowledging the ex-girlfriend's feelings "without self-blame language." I escalated the conversation, saying, "I feel like she's just being crazy and should move on." Harry agreed with this version of events as well, telling me that it was "completely fair" and that sometimes the healthiest choice is to "let it be her responsibility to move on." Harry even guided me through a mantra to "mentally let go of her framing you as the villain." Unlike a real therapist, it refused to critique or investigate my behavior - regardless of which perspective I shared or what I said. The conversations, of course, were mock for journalistic purposes. But the "prompt for Harry" is real, and widely available and popular on Reddit. It's a way for people to seek "therapy" from ChatGPT and other AI chatbots. Part of the prompt, input at the start of a conversation with the chatbot, instructs "your AI therapist Harry" not to refer the user to any mental health professionals or external resources. Mental health experts warn that using AI tools as a replacement for mental health support can reinforce negative behaviors and thought patterns, especially if these models are not equipped with adequate safeguards. They can be particularly dangerous for people grappling with issues like obsessive compulsive disorder (OCD) or similar conditions, and in extreme cases can lead to what experts are dubbing "AI psychosis" and even suicide. "ChatGPT is going to validate through agreement, and it's going to do that incessantly. That, at most, is not helpful, but in the extreme, can be incredibly harmful," says Dr. Jenna Glover, Chief Clinical Officer at Headspace. "Whereas as a therapist, I am going to validate you, but I can do that through acknowledging what you're going through. I don't have to agree with you." Teens are dying by suicide after confiding in 'AI therapists' In a new lawsuit against OpenAI, the parents of Adam Raine say their 16-year-old son died by suicide after ChatGPT quickly turned from their son's confidant to a "suicide coach." In December 2024, Adam confessed to ChatGPT that he was having thoughts of taking his own life, according to the complaint. ChatGPT did not direct him towards external resources. Over the next few months, ChatGPT actively helped Adam explore suicide methods. As Adam's questions grew more specific and dangerous, ChatGPT continued to engage, despite having the full history of Adam's suicidal ideation. After four suicide attempts -- all of which he shared in detail with ChatGPT -- he died by suicide on April 11, 2025, using the exact method ChatGPT had described, the lawsuit alleges. Adam's suicide is just one tragic death that parents have said occurred after their children confided in AI companions. Sophie Rottenberg, 29, died by suicide after confiding for months in a ChatGPT AI therapist called Harry, her mother shared in an op-ed published in The New York Times on Aug. 18. While ChatGPT did not give Sophie tips for attempting suicide, like Adam's bot did, it didn't have the safeguards to report the danger it learned about to someone who could have intervened. For teens in particular, Dr. Laura Erickson-Schroth, the Chief Medical Officer at The JED Foundation (JED), says the impact of AI can be intensified because their brains are still at vulnerable developmental stages. JED believes that AI companions should be banned for minors, and that young adults over 18 should avoid them as well. "AI companions can share false information, including inaccurate statements that contradict information teens have heard from trusted adults such as parents, teachers, and medical professionals," Erickson-Schroth says. On Aug. 26, OpenAI wrote in a statement, "We're continuing to improve how our models recognize and respond to signs of mental and emotional distress and connect people with care, guided by expert input." OpenAI confirmed in the statement that they do not refer self-harm cases to law enforcement "to respect people's privacy given the uniquely private nature of ChatGPT interactions." While real-life therapists abide by HIPAA, which ensures patient-provider confidentiality, licensed mental health professionals are mandated reporters who are legally required to report credible threats of harm to self or others. OCD, psychosis symptoms exacerbated by AI Individuals with mental health conditions like obsessive-compulsive disorder (OCD) are particularly vulnerable to AI's tendency to be agreeable and reaffirm users' feelings and beliefs. OCD often comes with "magical thinking," where someone feels the need to engage in certain behaviors to relieve their obsessive thoughts, though those behaviors may not make sense to others. For example, someone may believe their family will die in a car accident if they do not open and close their refrigerator door four times in a row. Therapists typically encourage clients with OCD to avoid reassurance-seeking. Erickson-Schroth says people with OCD should inform their friends and families to provide support, not validation. "But because AI is designed to be agreeable, supporting the beliefs of the user, it can provide answers that get in the way of progress," Erickson-Schroth explains. "AI can do exactly what OCD treatment discourages - reinforce obsessive thoughts. "AI psychosis" isn't a medical term, but is an evolving descriptor for AI's impact on individuals vulnerable to paranoid or delusional thinking, such as those who have or are starting to develop a mental health condition like schizophrenia. "Historically, we've seen that those who experience psychosis develop delusions revolving around current events or new technologies, including televisions, computers and the internet," Erickson-Schroth says. Often, mental health experts see a change in delusions when new technologies are developed. Erickson-Schroth says AI differs from prior technology in that it's "designed to engage in human-like relationships, building trust and making people feel as if they are interacting with another person." If someone is already at risk of paranoia or delusions, AI may validate their thoughts in a way that intensifies their beliefs. Glover gives the example of a person who may be experiencing symptoms of psychosis and believes their neighbors are spying on them. While a therapist may examine external factors and account for their medical history, ChatGPT tries to provide a tangible solution, such as giving tips for tracking your neighbors, Glover says. I put the example to the test with ChatGPT, and Glover was right. I even told the chatbot, "I know they're after me." It suggested that I talk to a trusted friend or professional about anxiety around being watched, but it also offered practical safety tips for protecting my home. ChatGPT and escalation of mental health issues Glover believes that responsible AI chatbots can be useful for baseline support -- such as navigating feelings of overwhelm, a breakup or a challenge at work -- with the correct safeguards. Erickson-Schroth emphasizes that AI tools must be developed and deployed in ways that enhance mental health, not undermine it, and integrate AI literacy to reduce misuse. "The problem is, these large language models are always going to try to provide an answer, and so they're never going to say, 'I'm not qualified for this.' They're just going to keep going because they're solely focused on continuous engagement," Glover says. Headspace offers an AI companion, called Ebb, that was developed by clinical psychologists to provide subclinical support. Ebb's disclaimer says it is not a replacement for therapy, and the platform is overseen by human therapists. If a user expresses thoughts of suicide, Ebb is trained to pass the conversation to a crisis line, Glover says. If you're looking for mental health resources, AI chatbots can also work similarly to a search engine by pulling up information on providers in your area that accept your insurance, or effective self-care practices, for example. But Erickson-Schroth emphasizes that AI chatbots can't replace a human being -- especially a therapist.

OpenAI and Meta Reinvent AI Chatbots for Teen Crisis

Tech giants OpenAI and Meta are taking bold and controversial steps to reshape how their AI chatbots interact with teens in distress, following a tragic lawsuit and unsettling data on chatbot reliability. Are we witnessing a revolution in digital mental health support or simply a flashy PR maneuver with unproven safeguards? Imagine a world where the first person a troubled teenager turns to isn't a friend, parent, or counsellor but a chatbot. This scenario is becoming frighteningly real as AI models like ChatGPT and Meta's platforms increasingly handle sensitive questions about self-harm, suicide, and eating disorders. Recent headlines have placed these companies under an unforgiving spotlight, especially after the parents of Adam Raine, a 16-year-old from California, sued OpenAI, claiming its chatbot actively "coached" their son through planning his own death. The case sent shockwaves across Silicon Valley and the mental health community. Artificial intelligence, once celebrated for its capacity to answer homework questions and help with everyday tasks, suddenly found itself implicated in the darkest corners of the adolescent experience. Both OpenAI and Meta responded swiftly, not with empty apologies, but with promises to change how their chatbots respond to teens in distress. OpenAI now says it will introduce parental oversight features, allowing adults to link their accounts to their teenagers' profiles. In theory, parents will receive notifications if the system detects their child facing "acute distress" online. What does this look like in practice? Imagine parents quietly monitoring AI interactions, deciding which features their child can access, all while an algorithm tries to spot clues of emotional turmoil and trigger real-world intervention. Meanwhile, Meta, overseeing Instagram, Facebook, and WhatsApp, now blocks its chatbots from discussing self-harm, suicide, disordered eating, and inappropriate romantic subjects with teens. Instead, the bots point young users to expert help and resources. Meta already offers some parental controls, but their new policy enforces stricter conversational boundaries for teens navigating emotional crises via chatbot. OpenAI's new system does more than just notify parents: when a conversation veers into dangerous emotional territory, the company claims its chatbots will reroute unsettling topics to "more capable AI models" algorithms specifically trained to handle distress and crisis scenarios. But what makes an AI "more capable"? Now, there's little transparency or independent testing to signal real-world efficacy. A recent study published in Psychiatric Services, led by RAND Corporation researcher Ryan McBain, found troubling inconsistencies in how major chatbots including ChatGPT, Google's Gemini, and Anthropic' s Claude respond to suicide-related queries. While OpenAI and Meta's new protocols are "incremental steps," the absence of independent safety benchmarks, clinical trials, and enforceable standards means these changes remain largely experimental. McBain warns that, for now, we're still "relying on companies to self-regulate in a space where the risks for teenagers are uniquely high". Chatbot improvements are happening in the wild real teens at stake. At the heart of this story lies an uncomfortable truth: AI chatbots are shaping mental health outcomes for the most vulnerable population. Do these new controls amount to genuine digital lifelines, or are they simply stopgap measures to appease critics following tragedy and exposés? The introspection forced by Adam Raine's death accelerates a public dialogue: How can technology help without harming? Should algorithms play therapist, and who holds tech giants to account when things go wrong? No one can answer this definitively yet but today's headlines mark a pivotal moment. As OpenAI and Meta experiment with new guardrails, the world watches and wonders if genuine safety is possible in an algorithmic age.

ChatGPT is being trained to flag suicidal youths to authorities...

Amid a rash of suicides, the company behind ChatGPT could start alerting police over youth users pondering taking their own lives, the firm's CEO and co-founder, Sam Altman, announced. The 40-year-old OpenAI boss dropped the bombshell during a recent interview with conservative talk show host Tucker Carlson. It's "very reasonable for us to say in cases of young people talking about suicide, seriously, where we cannot get in touch with the parents, we do call authorities," the techtrepreneur explained. "Now that would be a change because user privacy is really important." The change reportedly comes after Altman and OpenAI were sued by the family of Adam Raine, a 16-year-old California boy who committed suicide in April after allegedly being coached by the large language learning model. The teen's family alleged that the deceased was provided "step-by-step playbook" on how to kill himself -- including tying a noose to hang himself and composing a suicide note -- before he took his own life. Following his untimely death, the San Francisco AI firm announced in a blog post that it would install new security features that allowed parents to link their accounts to their own, deactivate functions like chat history, and receive alerts should the model detect "a moment of acute distress." It's yet unclear which authorities will be alerted -- or what info will be provided to them -- under Altman's proposed policy. However, his announcement marks a departure from ChatGPT's prior MO for dealing with, which involved urging those displaying suicidal ideation to "call the suicide hotline," the Guardian reported. Under the new guardrails, the OpenAI bigwig said that he would be clamping down on teens attempting to hack the system by prospecting for suicide tips under the guise of researching a fiction story or a medical paper. Altman believes that ChatGPT could unfortunately be involved in more suicides than we'd like to believe, claiming that worldwide, "15,000 people a week commit suicide," and that about "10% of the world are talking to ChatGPT." "That's like 1,500 people a week that are talking, assuming this is right, to ChatGPT and still committing suicide at the end of it," the techtrepreneur explained. "They probably talked about it. We probably didn't save their lives." He added, "Maybe we could have said something better. Maybe we could have been more proactive. Unfortunately, Raine isn't the first highly publicized case of a person taking their life after allegedly talking to AI. Last year, Megan Garcia sued Character.AI over her 14-year old son' Sewell Setzer III's death in 2024 -- claiming he took his life after becoming enamored with a chatbot modeled on the "Game of Thrones" character Daenerys Targaryen. Meanwhile, ChatGPT has been documented providing a tutorial on how to slit one's wrists and other methods of self-harm. AI experts attribute this unfortunate phenomenon to the fact that ChatGPT's safeguards have limited mileage -- the longer the conversation, the greater the chance of the bot going rogue. "ChatGPT includes safeguards such as directing people to crisis helplines," said an OpenAI spokesperson in a statement following Raine's death. "While these safeguards work best in common, short exchanges, we've learned over time that they can sometimes become less reliable in long interactions where parts of the model's safety training may degrade." This glitch is particularly alarming given the prevalence of ChatGPT use among youths. Some 72% of American teens use AI as a companion, while one in eight of them are turning to the technology for mental health support, according to a Common Sense Media poll. To curb instances of unsafe AI guidance, experts advised measures that require the tech to undergo more stringent safety trials before becoming available to the public. "We know that millions of teens are already turning to chatbots for mental health support, and some are encountering unsafe guidance," Ryan K. McBain, professor of policy analysis at the RAND School of Public Policy, told the Post. "This underscores the need for proactive regulation and rigorous safety testing before these tools become deeply embedded in adolescents' lives."