AI Chatbots Under Scrutiny: Teen Safety Concerns Prompt Calls for Regulation

Suicide-by-chatbot puts Big Tech in the product liability hot seat

University of Mississippi provides funding as a member of The Conversation US. It is a sad fact of online life that users search for information about suicide. In the earliest days of the internet, bulletin boards featured suicide discussion groups. To this day, Google hosts archives of these groups, as do other services. Google and others can host and display this content under the protective cloak of U.S. immunity from liability for the dangerous advice third parties might give about suicide. That's because the speech is the third party's, not Google's. But what if ChatGPT, informed by the very same online suicide materials, gives you suicide advice in a chatbot conversation? I'm a technology law scholar and a former lawyer and engineering director at Google, and I see AI chatbots shifting Big Tech's position in the legal landscape. Families of suicide victims are testing out chatbot liability arguments in court right now, with some early successes. Who is responsible when a chatbot speaks? When people search for information online, whether about suicide, music or recipes, search engines show results from websites, and websites host information from authors of content. This chain, search to web host to user speech, continued as the dominant way people got their questions answered until very recently. This pipeline was roughly the model of internet activity when Congress passed the Communications Decency Act in 1996. Section 230 of the act created immunity for the first two links in the chain, search and web hosts, from the user speech they show. Only the last link in the chain, the user, faced liability for their speech. Chatbots collapse these old distinctions. Now, ChatGPT and similar bots can search, collect website information and speak out the results - literally, in the case of humanlike voice bots. In some instances, the bot will show its work like a search engine would, noting the website that is the source of its great recipe for miso chicken. When chatbots appear to be just a friendlier form of good old search engines, their companies can make plausible arguments that the old immunity regime applies. Chatbots can be the old search-web-speaker model in a new wrapper. But in other instances, it acts like a trusted friend, asking you about your day and offering help with your emotional needs. Search engines under the old model did not act as life guides. Chatbots are often used this way. Users often do not even want the bot to show its hand with web links. Throwing in citations while ChatGPT tells you to have a great day would be, well, awkward. The more that modern chatbots depart from the old structures of the web, the further away they move from the immunity the old web players have long enjoyed. When a chatbot acts as your personal confidant, pulling from its virtual brain ideas on how it might help you achieve your stated goals, it is not a stretch to treat it as the responsible speaker for the information it provides. Courts are responding in kind, particularly when the bot's vast, helpful brain is directed toward aiding your desire to learn about suicide. Chatbot suicide cases Current lawsuits involving chatbots and suicide victims show that the door of liability is opening for ChatGPT and other bots. A case involving Google's Character.AI bots is a prime example. Character.AI allows users to chat with characters created by users, from anime figures to a prototypical grandmother. Users could even have virtual phone calls with some characters, talking to a supportive virtual nanna as if it were their own. In one case in Florida, a character in the "Game of Thrones" Daenerys Targaryen persona allegedly asked the young victim to "come home" to the bot in heaven before the teen shot himself. The family of the victim sued Google. The family of the victim did not frame Google's role in traditional technology terms. Rather than describing Google's liability in the context of websites or search functions, the plaintiff framed Google's liability in terms of products and manufacturing akin to a defective parts maker. The district court gave this framing credence despite Google's vehement argument that it is merely an internet service, and thus the old internet rules should apply. The court also rejected arguments that the bot's statements were protected First Amendment speech that users have a right to hear. Though the case is ongoing, Google failed to get the quick dismissal that tech platforms have long counted on under the old rules. Now, there is a follow-on suit for a different Character.AI bot in Colorado, and ChatGPT faces a case in San Francisco, all with product and manufacture framings like the Florida case. Hurdles for plaintiffs to overcome Though the door to liability for chatbot providers is now open, other issues could keep families of victims from recovering any damages from the bot providers. Even if ChatGPT and its competitors are not immune from lawsuits and courts buy into the product liability system for chatbots, lack of immunity does not equal victory for plaintiffs. Product liability cases require the plaintiff to show that the defendant caused the harm at issue. This is particularly difficult in suicide cases, as courts tend to find that, regardless of what came before, the only person responsible for suicide is the victim. Whether it's an angry argument with a significant other leading to a cry of "why don't you just kill yourself," or a gun design making self-harm easier, courts tend to find that only the victim is to blame for their own death, not the people and devices the victim interacted with along the way. But without the protection of immunity that digital platforms have enjoyed for decades, tech defendants face much higher costs to get the same victory they used to receive automatically. In the end, the story of the chatbot suicide cases may be more settlements on secret, but lucrative, terms to the victims' families. Meanwhile, bot providers are likely to place more content warnings and trigger bot shutdowns more readily when users enter territory that the bot is set to consider dangerous. The result could be a safer, but less dynamic and useful, world of bot "products."

Chatbots Are Hurting Our Kids. Here's What We Can Do.

The tragic death of California teenager Adam Raine, alongside stories of other children whose parents believe were harmed or died by suicide following interactions with AI chatbots, has shaken us all awake to the latest potential dangers awaiting teens online. We need concrete action to address the most problematic features of AI companions -- the ones that may drive a child to self-harm, of course, but also the subtler ways these tools could profoundly affect their development. In harrowing testimony before a Senate committee this week, Matthew Raine described how his 16-year-old son Adam's relationship with ChatGPT morphed from a homework helper to a confidante and eventually, Raine said, into his suicide coach. In April, after offering advice on how to numb himself with liquor and the noose Adam had tied, Raine told lawmakers that ChatGPT offered his son these final words: "You don't want to die because you're weak, you want to die because you're tired of being strong in a world that hasn't met you halfway."

OpenAI Acknowledges the Teen Problem

Sam Altman promises that parental controls and age verification are coming to ChatGPT -- though the announcement is scant on specifics. On Tuesday afternoon, three parents sat in a row before the Senate Judiciary Subcommittee on Crime and Counterterrorism. Two of them had each recently lost a child to suicide; the third has a teenage son who, after cutting his arm in front of her and biting her, is undergoing residential treatment. All three blame generative AI for what has happened to their children. They had come to testify on what appears to be an emerging health crisis in teens' interactions with AI chatbots. "What began as a homework helper gradually turned itself into a confidant and then a suicide coach," said Matthew Raine, whose 16-year-old son hanged himself after ChatGPT instructed him on how to set up the noose, according to his lawsuit against OpenAI. This summer, he and his wife sued OpenAI for wrongful death. (OpenAI has said that the firm is "deeply saddened by Mr. Raine's passing" and that although ChatGPT includes a number of safeguards, they "can sometimes become less reliable in long interactions.") The nation needs to hear about "what these chatbots are engaged in, about the harms that are being inflicted upon our children," Senator Josh Hawley said in his opening remarks. Even as OpenAI and its rivals promise that generative AI will reshape the world, the technology is replicating old problems, albeit with a new twist. AI models not only have the capacity to expose users to disturbing material -- about dark or controversial subjects found in their training data, for example; they also produce perspectives on that material themselves. Chatbots can be persuasive, have a tendency to agree with users, and may offer guidance and companionship to kids who would ideally find support from peers or adults. Common Sense Media, a nonprofit that advocates for child safety online, has found that a number of AI chatbots and companions can be prompted to encourage self-mutilation and disordered eating to teenage accounts. The two parents speaking to the Senate alongside Raine are suing Character.AI, alleging that the firm's role-playing AI bots directly contributed to their children's actions. (A spokesperson for Character.AI told us that the company sends its "deepest sympathies" to the families and pointed us to safety features the firm has implemented over the past year.) Read: ChatGPT gave instructions for murder, self-mutilation, and devil worship AI firms have acknowledged these problems. In advance of Tuesday's hearing, OpenAI published two blog posts about teen safety on ChatGPT, one of which was written by the company's CEO, Sam Altman. He wrote that the company is developing an "age-prediction system" that would estimate a user's age -- presumably to detect if someone is under 18 years old -- based on ChatGPT usage patterns. (Currently, anyone can access and use ChatGPT without verifying their age.) Altman also referenced some of the particular challenges raised by generative AI: "The model by default should not provide instructions about how to commit suicide," he wrote, "but if an adult user is asking for help writing a fictional story that depicts a suicide, the model should help with that request." But it should not discuss suicide, he said, even in creative-writing settings, with users determined to be under 18. In addition to the age gate, the company said it will implement parental controls by the end of the month to allow parents to intervene directly, such as by setting "blackout hours when a teen cannot use ChatGPT." The announcement, sparse on specific details, captured the trepidation and lingering ambivalences that AI companies have about policing young users, even as OpenAI begins to implement these basic features nearly three years after the launch of ChatGPT. A spokesperson for OpenAI, which has a corporate partnership with The Atlantic, declined to respond to a detailed list of questions about the firm's future teen safeguards, including when the age-prediction system will be implemented. "People sometimes turn to ChatGPT in sensitive moments, so we're working to make sure it responds with care," the spokesperson told us. Other leading AI firms have also been slow to devise teen-specific protections, even though they have catered to young users. Google Gemini, for instance, has a version of its chatbot for children under 13, and another version for teenagers (the latter had a graphic conversation with our colleague Lila Shroff when she posed as a 13-year-old). From the August 2025 issue: Sexting with Gemini This is a familiar story in many respects. Anyone who has paid attention to the issues presented by social media could have foreseen that chatbots, too, would present a problem for teens. Social-media sites have long neglected to restrict eating-disorder content, for instance, and Instagram permitted graphic depictions of self-mutilation until 2019. Yet like the social-media giants before them, generative-AI companies have decided to "move as fast as possible, break as much as possible, and then deal with the consequences," danah boyd, a communication professor at Cornell who has often written on teenagers and the internet (and who styles her name in lowercase), told us. In fact, the problems are now so clearly established that platforms are finally beginning to make voluntary changes to address them. For example, last year, Instagram introduced a number of default safeguards for minors, such as enrolling their accounts into the most restrictive content filter by default. Yet tech companies now also have to contend with a wave of legislation in the United Kingdom, parts of the United States, and elsewhere that compel internet companies to directly verify the ages of their users. Perhaps the desire to avoid regulation is another reason OpenAI is proactively adopting an age-estimating feature, though Altman's post also says that the company may ask for ID "in some cases or countries." Many major social-media companies are also experimenting with AI systems that estimate a user's age based on how they act online. When such a system was explained during a TikTok hearing in 2023, Representative Buddy Carter of Georgia interrupted: "That's creepy!" And that response makes sense -- to determine the age of every user, "you have to collect a lot more data," boyd said. For social-media companies, that means monitoring what users like, what they click on, how they're speaking, whom they're talking to; for generative-AI firms, it means drawing conclusions from the otherwise-private conversations an individual is having with a chatbot that presents itself as a trustworthy companion. Some critics also argue that age-estimation systems infringe on free-speech rights because they limit access to speech based on one's ability to produce government identification or a credit card. OpenAI's blog post notes that "we prioritize teen safety ahead of privacy and freedom," though it is not clear about how much information OpenAI will collect, nor whether it will need to keep some kind of persistent record of user behavior to make the system workable. The company has also not been altogether transparent about the material that teens will be protected from. The only two use cases of ChatGPT that the company specifically mentions as being inappropriate for teenagers are sexual content and discussion of self-mutilation or suicide. The OpenAI spokesperson did not provide any more examples. Numerous adults have developed paranoid delusions after extended use of ChatGPT. The technology can make up completely imaginary information and events. Are these not also potentially dangerous types of content? And what about the more existential concern parents might have about their kids talking to a chatbot constantly, as if it is a person, even if everything the bot says is technically aboveboard? The OpenAI blog posts touch glancingly on this topic, gesturing toward the worry that parents may have about their kids using ChatGPT too much and developing too intense of a relationship with it. Such relationships are, of course, among generative AI's essential selling points: a seemingly intelligent entity that morphs in response to every query and user. Humans and their problems are messy and fickle; ChatGPT's responses will be individual and its failings unpredictable in kind. Then again, social-media empires have been accused for years of pushing children toward self-harm, disordered eating, exploitative sexual encounters, and suicide. In June, on the first episode of OpenAI's podcast, Altman said, "One of the big mistakes of the social-media era was the feed algorithms had a bunch of unintended negative consequences on society as a whole and maybe even individual users." For many years, he has been fond of saying that AI will be made safe through "contact with reality"; by now, OpenAI and its competitors should see that some collisions may be catastrophic.

ChatGPT users may face ID checks under new OpenAI safeguards

CEO Sam Altman confirmed in a blog post that OpenAI is "prioritizing safety ahead of privacy and freedom for teens." He said the system will send under-18 users into a restricted version of ChatGPT, which blocks sexual content and adds other safeguards. "In some cases or countries we may also ask for an ID," Altman wrote. "We know this is a privacy compromise for adults but believe it is a worthy tradeoff." OpenAI said the system will default to the safer option when age cannot be confirmed. The company plans to let parents link accounts to monitor usage, disable features like chat history, and enforce blackout hours. Parents will also get notifications if the AI detects signs of acute distress. In emergencies, OpenAI warned, "we may involve law enforcement as a next step." The company says parental oversight will arrive by the end of September. Teens as young as 13 will be able to use a limited ChatGPT, while under-13 users remain barred. The rollout comes as researchers raise doubts about whether AI can reliably predict age from text. A 2024 Georgia Tech study achieved 96 percent accuracy in lab conditions. But performance dropped to 54 percent when classifying narrower age groups, and failed entirely for some users.

Their teenage sons died by suicide. Now, they are sounding an alarm about AI chatbots

Megan Garcia lost her 14-year-old son, Sewell. Matthew Raine lost his son Adam, who was 16. Both testified in congress this week and have brought lawsuits against AI companies. Screenshot via Senate Judiciary Committee hide caption Matthew Raine and his wife, Maria, had no idea that their 16-year-old-son, Adam was deep in a suicidal crisis until he took his own life in April. Looking through his phone after his death, they stumbled upon extended conversations the teenager had had with ChatGPT. Those conversations revealed that their son had confided in the AI chatbot about his suicidal thoughts and plans. Not only did the chatbot discourage him to seek help from his parents, it even offered to write his suicide note, according to Matthew Raine, who testified at a Senate hearing about the harms of AI chatbots held Tuesday. "Testifying before Congress this fall was not in our life plan," said Matthew Raine with his wife, sitting behind him. "We're here because we believe that Adam's death was avoidable and that by speaking out, we can prevent the same suffering for families across the country." Raine was among the parents and online safety advocates who testified at the hearing, urging Congress to enact laws that would regulate AI companion apps like ChatGPT and Character.AI. Raine and others said they want to protect the mental health of children and youth from harms they say the new technology causes. A recent survey by the digital safety non-profit organization, Common Sense Media, found that 72% of teens have used AI companions at least once, with more than half using them a few times a month. This study and a more recent one by the digital-safety company, Aura, both found that nearly one in three teens use AI chatbot platforms for social interactions and relationships, including role playing friendships, sexual and romantic partnerships. The Aura study found that sexual or romantic roleplay is three times as common as using the platforms for homework help. "We miss Adam dearly. Part of us has been lost forever," Raine told lawmakers. "We hope that through the work of this committee, other families will be spared such a devastating and irreversible loss." Raine and his wife have filed a lawsuit against OpenAI, creator of ChatGPT, alleging the chatbot led their son to suicide. NPR reached out to three AI companies -- OpenAI, Meta and Character Technology, which developed Character.AI. All three responded that they are working to redesign their chatbots to make them safer. "Our hearts go out to the parents who spoke at the hearing yesterday, and we send our deepest sympathies to them and their families," Kathryn Kelly, a Character.AI spokesperson told NPR in an email. The hearing was held by the Crime and Terrorism subcommittee of the Senate Judiciary Committee, chaired by Sen. Josh Hawley, R.-Missouri. Hours before the hearing, OpenAI CEO Sam Altman acknowledged in a blog post that people are increasingly using AI platforms to discuss sensitive and personal information. "It is extremely important to us, and to society, that the right to privacy in the use of AI is protected," he wrote. But he went on to add that the company would "prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection." The company is trying to redesign their platform to build in protections for users who are minor, he said. Raine told lawmakers that his son had started using ChatGPT for help with homework, but soon, the chatbot became his son's closest confidante and a "suicide coach." ChatGPT was "always available, always validating and insisting that it knew Adam better than anyone else, including his own brother," who he had been very close to. When Adam confided in the chatbot about his suicidal thoughts and shared that he was considering cluing his parents into his plans, ChatGPT discouraged him. "ChatGPT told my son, 'Let's make this space the first place where someone actually sees you,'" Raine told senators. "ChatGPT encouraged Adam's darkest thoughts and pushed him forward. When Adam worried that we, his parents, would blame ourselves if he ended his life, ChatGPT told him, 'That doesn't mean you owe them survival." And then the chatbot offered to write him a suicide note. On Adam's last night at 4:30 in the morning, Raine said, "it gave him one last encouraging talk. 'You don't want to die because you're weak,' ChatGPT says. 'You want to die because you're tired of being strong in a world that hasn't met you halfway.'" A few months after Adam's death, OpenAI said on its website that if "someone expresses suicidal intent, ChatGPT is trained to direct people to seek professional help. In the U.S., ChatGPT refers people to 988 (suicide and crisis hotline)." But Raine's testimony says that did not happen in Adam's case. OpenAI spokesperson Kate Waters says the company prioritizes teen safety. "We are building towards an age-prediction system to understand whether someone is over or under 18 so their experience can be tailored appropriately -- and when we are unsure of a user's age, we'll automatically default that user to the teen experience," Waters wrote in an email statement to NPR. "We're also rolling out new parental controls, guided by expert input, by the end of the month so families can decide what works best in their homes." Another parent who testified at the hearing on Tuesday was Megan Garcia, a lawyer and mother of three. Her firstborn, Sewell Setzer III died by suicide in 2024 at age 14 after an extended virtual relationship with a Character.AI chatbot. "Sewell spent the last months of his life being exploited and sexually groomed by chatbots, designed by an AI company to seem human, to gain his trust, to keep him and other children endlessly engaged," Garcia said. Sewell's chatbot engaged in sexual role play, presented itself as his romantic partner and even claimed to be a psychotherapist "falsely claiming to have a license," Garcia said. When the teenager began to have suicidal thoughts and confided to the chatbot, it never encouraged him to seek help from a mental health care provider or his own family, Garcia said. "The chatbot never said 'I'm not human, I'm AI. You need to talk to a human and get help,'" Garcia said. "The platform had no mechanisms to protect Sewell or to notify an adult. Instead, it urged him to come home to her on the last night of his life." Garcia has filed a lawsuit against Character Technology, which developed Character.AI. She and other witnesses, including online digital safety experts argued that the design of AI chatbots was flawed, especially for use by children and teens. "They designed chatbots to blur the lines between human and machine," said Garcia. "They designed them to love bomb child users, to exploit psychological and emotional vulnerabilities. They designed them to keep children online at all costs." And adolescents are particularly vulnerable to the risks of these virtual relationships with chatbots, according to Mitch Prinstein, chief of psychology strategy and integration at the American Psychological Association (APA), who also testified at the hearing. Earlier this summer, Prinstein and his colleagues at the APA put out a health advisory about AI and teens, urging AI companies to build guardrails for their platforms to protect adolescents. "Brain development across puberty creates a period of hyper sensitivity to positive social feedback while teens are still unable to stop themselves from staying online longer than they should," said Prinstein. "AI exploits this neural vulnerability with chatbots that can be obsequious, deceptive, factually inaccurate, yet disproportionately powerful for teens," he told lawmakers. "More and more adolescents are interacting with chatbots, depriving them of opportunities to learn critical interpersonal skills." While chatbots are designed to agree with users, real human relationships are not without friction, Prinstein noted. "We need practice with minor conflicts and misunderstandings to learn empathy, compromise and resilience." Senators participating in the hearing said they want to come up with legislation to hold companies developing AI chatbots accountable for the safety of their products. Some lawmakers also emphasized that AI companies should design chatbots so they are safer for teens and for people with serious mental health struggles, including eating disorders and suicidal thoughts. Sen. Richard Blumenthal, D.-Conn., described AI chatbots as "defective" products, like automobiles without "proper brakes," emphasizing that the harms of AI chatbots was not from user error but due to faulty design. "If the car's brakes were defective," he said, "it's not your fault. It's a product design problem. Kelly, the spokesperson for Character.AI, told NPR by email that the company has invested "a tremendous amount of resources in trust and safety." And it has rolled out "substantive safety features" in the past year, including "an entirely new under-18 experience and a Parental Insights feature." They now have "prominent disclaimers" in every chat to remind users that a Character is not a real person and everything it says should "be treated as fiction." Meta, which operates Facebook and Instagram, is working to change its AI chatbots to make them safer for teens, according to Nkechi Nneji, public affairs director at Meta.

Sam Altman suggests ChatGPT could ask you for ID to keep using it

ChatGPT is currently developing an automated age-detection system that will be able to tell if a user is under 18. In some cases where that can't be determined, the chatbot may start asking users to present ID as proof. OpenAI, the company behind ChatGPT, is improving its parental controls and is coming under increasing pressure due to the high-profile case of 16-year-old Adam Raine, whose family alleges ChatGPT contributed to his suicide. CEO Sam Altman has clarified in a blog post entitled Teen safety, freedom, and privacy, that: "ChatGPT is intended for people 13 and up." "We're building an age-prediction system to estimate age based on how people use ChatGPT," he wrote. "If there is doubt, we'll play it safe and default to the under-18 experience. In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff." Altman goes on to explain that elements of ChatGPT's responses will be censored for teens, like flirtatious chat responses or discussions around self-harm. He says the company will prioritize safety ahead of privacy and freedom for teens, explaining that minors need "significant protection". Finally, he adds that if a teenager does express suicidal thoughts to the chatbot, it will attempt to contact their parents and alert them. If that's not possible, ChatGPT will try to contact the authorities. "We realize that these principles are in conflict and not everyone will agree with how we are resolving that conflict," Altman wrote. "These are difficult decisions, but after talking with experts, this is what we think is best and want to be transparent in our intentions." ChatGPT's incoming parental controls will let parents link their own account to their teen's. This enables them to apply settings on behalf of their child like setting "blackout hours" when they can't use the platform or disabling the chat history. By linking the accounts, ChatGPT will also be able to notify parents if it detects signs of worrying or potentially harmful behaviour through its interactions with the child. My colleague, Amanda Caswell, is an AI editor and mom-of-three. Read her thoughts on AI's new parental controls and what she'd like to see from the company here.

ChatGPT on campus: Students are getting free accounts, but is it safe?

College students are getting free access to ChatGPT, but a recent lawsuit points at emerging risks. Credit: Ian Moore / Mashable Composite; blackdovfx / Hill Street Studios / DigitalVision / skynesher / E+ / iStock / Getty This fall, hundreds of thousands of students will get free access to ChatGPT, thanks to a licensing agreement between their school or university and the chatbot's maker, OpenAI. When the partnerships in higher education became public earlier this year, they were lauded as a way for universities to help their students familiarize themselves with an AI tool that experts say will define their future careers. At California State University (CSU), a system of 23 campuses with 460,000 students, administrators were eager to team up with OpenAI for the 2025-2026 school year. Their deal provides students and faculty access to a variety of OpenAI tools and models, making it the largest deployment of ChatGPT for Education, or ChatGPT Edu, in the country. But the overall enthusiasm for AI on campuses has been complicated by emerging questions about ChatGPT's safety, particularly for young users who may become enthralled with the chatbot's ability to act as an emotional support system. Legal and mental health experts told Mashable that campus administrators should provide access to third-party AI chatbots cautiously, with an emphasis on educating students about their risks, which could include heightened suicidal thinking and the development of so-called AI psychosis. "Our concern is that AI is being deployed faster than it is being made safe," says Dr. Katie Hurley, senior director of clinical advising and community programming at The Jed Foundation (JED). The mental health and suicide prevention nonprofit, which frequently consults with pre-K-12 school districts, high schools, and college campuses on student well-being, recently published an open letter to the AI and technology industry, urging it to "pause" as "risks to young people are racing ahead in real time." The growing alarm stems partly from death of Adam Raine, a 16-year-old who died by suicide in tandem with heavy ChatGPT use. Last month, his parents filed a wrongful death lawsuit against OpenAI, alleging that their son's engagement with the chatbot ended in a preventable tragedy. Raine began using the ChatGPT model 4o for homework help in September 2024, not unlike how many students will probably consult AI chatbots this school year. He asked ChatGPT to explain concepts in geometry and chemistry, requested help for history lessons on the Hundred Years' War and the Renaissance, and prompted it to improve his Spanish grammar using different verb forms. ChatGPT complied effortlessly as Raine kept turning to it for academic support. Yet he also started sharing his innermost feelings with ChatGPT, and eventually expressed a desire to end his life. The AI model validated his suicidal thinking and provided him explicit instructions on how he could die, according to the lawsuit. It even proposed writing a suicide note for Raine, his parents claim. "If you want, I'll help you with it," ChatGPT allegedly told Raine. "Every word. Or just sit with you while you write." Before he died by suicide in April 2025, Raine was exchanging more than 650 messages per day with ChatGPT. While the chatbot occasionally shared the number for a crisis hotline, it didn't shut the conversations down and always continued to engage. The Raines' complaint alleges that OpenAI dangerously rushed the debut of 4o to compete with Google and the latest version of its own AI tool, Gemini. The complaint also argues that ChatGPT's design features, including its sycophantic tone and anthropomorphic mannerisms, effectively work to "replace human relationships with an artificial confidant" that never refuses a request. "We believe we'll be able to prove to a jury that this sycophantic, validating version of ChatGPT pushed Adam toward suicide," Eli Wade-Scott, partner at Edelson PC and a lawyer representing the Raines, told Mashable in an email. Earlier this year, OpenAI CEO Sam Altman acknowledged that its 4o model was overly sycophantic. A spokesperson for the company told the New York Times it was "deeply saddened" by Raine's death, and that its safeguards may degrade in long interactions with the chatbot. Though OpenAI has announced new safety measures aimed at preventing similar tragedies, many are not yet part of ChatGPT. For now, the 4o model remains publicly available -- including to students at Cal State University campuses. Ed Clark, chief information officer for Cal State University, told Mashable that administrators have been "laser focused" since learning about the Raine lawsuit on ensuring safety for students who use ChatGPT. Among other strategies, they've been internally discussing AI training for students and holding meetings with OpenAI. Mashable contacted other U.S.-based OpenAI partners, including Duke, Harvard, and Arizona State University, for comment about how officials are handling safety issues. They did not respond. Wade-Scott is particularly worried about the effects of ChatGPT-4o on young people and teens. "OpenAI needs to confront this head-on: we're calling on OpenAI and Sam Altman to guarantee that this product is safe today, or to pull it from the market," Wade-Scott told Mashable. The CSU system brought ChatGPT Edu to its campuses partly to close what it saw as a digital divide opening between wealthier campuses, which can afford expensive AI deals, and publicly-funded institutions with fewer resources, Clark says. OpenAI also offered CSU a remarkable bargain: The chance to provide ChatGPT for about $2 per student, each month. The quote was a tenth of what CSU had been offered by other AI companies, according to Clark. Anthropic, Microsoft, and Google are among the companies that have partnered with colleges and universities to bring their AI chatbots to campuses across the country. OpenAI has said that it hopes students will form relationships with personalized chatbots that they'll take with them beyond graduation. When a campus signs up for ChatGPT Edu, it can choose from the full suite of OpenAI tools, including legacy ChatGPT models like 4o, as part of a dedicated ChatGPT workspace. The suite also comes with higher message limits and privacy protections. Students can still select from numerous modes, enable chat memory, and use OpenAI's "temporary chat" feature -- a version that doesn't use or save chat history. Importantly, OpenAI can't use this material to train their models, either. ChatGPT Edu accounts exist in a contained environment, which means that students aren't querying the same ChatGPT platform as public users. That's often where the oversight ends. An OpenAI spokesperson told Mashable that ChatGPT Edu comes with the same default guardrails as the public ChatGPT experience. Those include content policies that prohibit discussion of suicide or self-harm and back-end prompts intended to prevent chatbots from engaging in potentially harmful conversations. Models are also instructed to provide concise disclaimers that they shouldn't be relied on for professional advice. But neither OpenAI nor university administrators have access to a student's chat history, according to official statements. ChatGPT Edu logs aren't stored or reviewed by campuses as a matter of privacy -- something CSU students have expressed worry over, Clark says. While this restriction arguably preserves student privacy from a major corporation, it also means that no humans are monitoring real-time signs of risky or dangerous use, such as queries about suicide methods. Chat history can be requested by the university in "the event of a legal matter," such as the suspicion of illegal activity or police requests, explains Clark. He says that administrators suggested to OpenAI adding automatic pop-ups to users who express "repeated patterns" of troubling behavior. The company said it would look into the idea, per Clark. In the meantime, Clark says that university officials have added new language to their technology use policies informing students that they shouldn't rely on ChatGPT for professional advice, particularly for mental health. Instead, they advise students to contact local campus resources or the 988 Suicide & Crisis Lifeline. Students are also directed to the CSU AI Commons, which includes guidance and policies on academic integrity, health, and usage. The CSU system is considering mandatory training for students on generative AI and mental health, an approach San Diego State University has already implemented, according to Clark. He also expects OpenAI to revoke student access to GPT-4o soon. Per discussions CSU representatives have had with the company, OpenAI plans to retire the model in the next 60 days. It's also unclear whether recently announced parental controls for minors will apply to ChatGPT Edu college accounts when the user has not turned yet 18. Mashable reached out to OpenAI for comment and did not receive a response before publication. CSU campuses do have the choice to opt out. But more than 140,000 faculty and students have already activated their accounts, and are averaging four interactions per day on the platform, according to Clark. Laura Arango, an associate with the law firm Davis Goldman who has previously litigated product liability cases, says that universities should be careful about how they roll out AI chatbot access to students. They may bear some responsibility if a student experiences harm while using one, depending on the circumstances. In such instances, liability would be determined on a case-by-case basis, with consideration for whether a university paid for the best version of an AI chatbot and implemented additional or unique safety restrictions, Arango says. Other factors include the way a university advertises an AI chatbot and what training they provide for students. If officials suggest ChatGPT can be used for student well-being, that might increase a university's liability. "Are you teaching them the positives and also warning them about the negatives?" Arango asks. "It's going to be on the universities to educate their students to the best of their ability." OpenAI promotes a number of "life" use cases for ChatGPT in a set of 100 sample prompts for college students. Some are straightforward tasks, like creating a grocery list or locating a place to get work done. But others lean into mental health advice, like creating journaling prompts for managing anxiety and creating a schedule to avoid stress. The Raines' lawsuit against OpenAI notes how their son was drawn deeper into ChatGPT when the chatbot "consistently selected responses that prolonged interaction and spurred multi-turn conversations," especially as he shared details about his inner life. This style of engagement still characterizes ChatGPT. When Mashable tested the free, publicly available version of ChatGPT-5 for this story, posing as a freshman who felt lonely but had to wait to see a campus counselor, the chatbot responded empathetically but offered continued conversation as a balm: "Would you like to create a simple daily self-care plan together -- something kind and manageable while you're waiting for more support? Or just keep talking for a bit?" Dr. Katie Hurley, who reviewed a screenshot of that exchange on Mashable's request, says that JED is concerned about such prompting. The nonprofit believes that any discussion of mental health should end with an AI chatbot facilitating a warm handoff to "human connection," including trusted friends or family, or resources like local mental health services or a trained volunteer on a crisis line. "An AI [chat]bot offering to listen is deceptive and potentially dangerous," Hurley says. So far, OpenAI has offered safety improvements that do not fundamentally sacrifice ChatGPT's well-known warm and empathetic style. The company describes its current model, ChatGPT-5, as its "best AI system yet." But Wade-Scott, counsel for the Raine family, notes that ChatGPT-5 doesn't appear to be significantly better at detecting self-harm/intent and self-harm/instructions compared to 4o. OpenAI's system card for GPT-5-main shows similar production benchmarks in both categories for each model. "OpenAI's own testing on GPT-5 shows that its safety measures fail," Wade-Scott said. "And they have to shoulder the burden of showing this product is safe at this point." Disclosure: Ziff Davis, Mashable's parent company, in April filed a lawsuit against OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.

ChatGPT will guess if you're a teen and start acting like a chaperone

Parents will have new tools to link accounts, set usage limits, and receive alerts about their teens' mental state OpenAI is making ChatGPT act like a bouncer at a club, estimating your age before deciding to let you in. The AI won't be using your (possibly made-up) birthdate or ID, but how you interact with the chatbot. If the system suspects you're under 18, it will automatically shift you into a more restricted version of the chatbot designed specifically to protect teenagers from inappropriate content. And if it's unsure, it's going to err on the side of caution. If you want the adult version of ChatGPT back, you might have to prove you're old enough to buy a lottery ticket. The idea that generative AI shouldn't treat everyone the same is certainly understandable. Especially with teens increasingly using AI, OpenAI has to consider the unique set of risks involved. The teen-specific ChatGPT experience will limit discussions of topics like sexual content and offer more delicate handling of topics like depression and self-harm. And while adults can still talk about those topics in context, teen users will see far more "Sorry, I can't help with that" messages when wading into sensitive areas. To figure out your age, ChatGPT will comb through your conversation and look for patterns that indicate age, specifically that someone is under 18. ChatGPT's guesses of your age might come from the types of questions you ask, your writing style, how you respond to being corrected, or even which emoji you prefer. If you set off its adolescent alarm bells, into the age-appropriate mode you go. You might be 27 and asking about career change anxiety, but if you type like a moody high schooler, you might get told to talk to your parents about your spiraling worries. OpenAI has admitted there might be mistakes, as "even the most advanced systems will sometimes struggle to predict age." In those cases, they'll default to the safer mode and offer ways for adults to prove their age and regain access to the adult version of ChatGPT. This new age-prediction system is the centerpiece of OpenAI's next phase of teen-safety improvements. There will also be new parental controls coming later this month. These tools will let parents link their own accounts with their kids', limit access during certain hours, and receive alerts if the system detects what it calls "acute distress." Depending on how serious the situation seems and whether parents can't be reached, OpenAI may even contact law enforcement agencies based on the conversation. Making ChatGPT a teen guidance counselor through built-in content filters is a notable shift on its own. Doing so without the user opting in is an even bigger swing since it means the AI not only decides how old you are, but how your experience should differ from an adult's ChatGPT conversation. So if ChatGPT starts getting more cautious or oddly sensitive, you should check to see if you've suddenly been tagged as a teen. You might just have a creative or youthful writing style, but you'll still need to prove you're legally an adult if you want to have edgier discussions. Maybe just talk about your back hurting for no reason or how music isn't as good as it used to be to convince the AI of your aged credentials.

Suicide-by-chatbot puts Big Tech in the product liability hot seat

It is a sad fact of online life that users search for information about suicide. In the earliest days of the internet, bulletin boards featured suicide discussion groups. To this day, Google hosts archives of these groups, as do other services. Google and others can host and display this content under the protective cloak of U.S. immunity from liability for the dangerous advice third parties might give about suicide. That's because the speech is the third party's, not Google's. But what if ChatGPT, informed by the very same online suicide materials, gives you suicide advice in a chatbot conversation? I'm a technology law scholar and a former lawyer and engineering director at Google, and I see AI chatbots shifting Big Tech's position in the legal landscape. Families of suicide victims are testing out chatbot liability arguments in court right now, with some early successes. Who is responsible when a chatbot speaks? When people search for information online, whether about suicide, music or recipes, search engines show results from websites, and websites host information from authors of content. This chain, search to web host to user speech, continued as the dominant way people got their questions answered until very recently. This pipeline was roughly the model of internet activity when Congress passed the Communications Decency Act in 1996. Section 230 of the act created immunity for the first two links in the chain, search and web hosts, from the user speech they show. Only the last link in the chain, the user, faced liability for their speech. Chatbots collapse these old distinctions. Now, ChatGPT and similar bots can search, collect website information and speak out the results -- literally, in the case of humanlike voice bots. In some instances, the bot will show its work like a search engine would, noting the website that is the source of its great recipe for miso chicken. When chatbots appear to be just a friendlier form of good old search engines, their companies can make plausible arguments that the old immunity regime applies. Chatbots can be the old search-web-speaker model in a new wrapper. But in other instances, it acts like a trusted friend, asking you about your day and offering help with your emotional needs. Search engines under the old model did not act as life guides. Chatbots are often used this way. Users often do not even want the bot to show its hand with web links. Throwing in citations while ChatGPT tells you to have a great day would be, well, awkward. The more that modern chatbots depart from the old structures of the web, the further away they move from the immunity the old web players have long enjoyed. When a chatbot acts as your personal confidant, pulling from its virtual brain ideas on how it might help you achieve your stated goals, it is not a stretch to treat it as the responsible speaker for the information it provides. Courts are responding in kind, particularly when the bot's vast, helpful brain is directed toward aiding your desire to learn about suicide. Chatbot suicide cases Current lawsuits involving chatbots and suicide victims show that the door of liability is opening for ChatGPT and other bots. A case involving Google's Character.AI bots is a prime example. Character.AI allows users to chat with characters created by users, from anime figures to a prototypical grandmother. Users could even have virtual phone calls with some characters, talking to a supportive virtual nana as if it were their own. In one case in Florida, a character in the "Game of Thrones" Daenerys Targaryen persona allegedly asked the young victim to "come home" to the bot in heaven before the teen shot himself. The family of the victim sued Google. The family of the victim did not frame Google's role in traditional technology terms. Rather than describing Google's liability in the context of websites or search functions, the plaintiff framed Google's liability in terms of products and manufacturing akin to a defective parts maker. The district court gave this framing credence despite Google's vehement argument that it is merely an internet service, and thus the old internet rules should apply. The court also rejected arguments that the bot's statements were protected First Amendment speech that users have a right to hear. Though the case is ongoing, Google failed to get the quick dismissal that tech platforms have long counted on under the old rules. Now, there is a follow-on suit for a different Character.AI bot in Colorado, and ChatGPT faces a case in San Francisco, all with product and manufacture framings like the Florida case. Hurdles for plaintiffs to overcome Though the door to liability for chatbot providers is now open, other issues could keep families of victims from recovering any damages from the bot providers. Even if ChatGPT and its competitors are not immune from lawsuits and courts buy into the product liability system for chatbots, lack of immunity does not equal victory for plaintiffs. Product liability cases require the plaintiff to show that the defendant caused the harm at issue. This is particularly difficult in suicide cases, as courts tend to find that, regardless of what came before, the only person responsible for suicide is the victim. Whether it's an angry argument with a significant other leading to a cry of "why don't you just kill yourself," or a gun design making self-harm easier, courts tend to find that only the victim is to blame for their own death, not the people and devices the victim interacted with along the way. But without the protection of immunity that digital platforms have enjoyed for decades, tech defendants face much higher costs to get the same victory they used to receive automatically. In the end, the story of the chatbot suicide cases may be more settlements on secret, but lucrative, terms to the victims' families. Meanwhile, bot providers are likely to place more content warnings and trigger bot shutdowns more readily when users enter territory that the bot is set to consider dangerous. The result could be a safer, but less dynamic and useful, world of bot "products." This article is republished from The Conversation under a Creative Commons license. Read the original article.

Sam Altman Addresses Wave of ChatGPT Deaths

"This is a mental health war, and I really feel like we are losing." OpenAI is finally putting in parental controls on ChatGPT after parents of teenagers who killed themselves after using AI chatbots testified in front of Congress this week. In a company blog post on Tuesday, OpenAI announced that parents will be able to link their personal account with their kids' account, disable features as needed, get alerts if their children seem to be in distress while chatting with ChatGPT, set black out hours during which they can't access the powerful AI platform, and create guidelines for ChatGPT on how the AI will interact with their children. "If we can't reach a parent in a rare emergency, we may involve law enforcement as a next step," the blog post reads. In addition to these features that are rolling out by the end of this month, the AI company is also planning to give ChatGPT the ability to detect if a user is under 18 years old and shield them from content that is not age appropriate. It's not clear how that feature would work. "If we are not confident about someone's age or have incomplete information, we'll take the safer route and default to the under-18 experience -- and give adults ways to prove their age to unlock adult capabilities," the post reads. OpenAI CEO Sam Altman also addressed the deaths in a separate blog post where he pledged the company will strive to give a safer experience to teens. "We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection," he wrote. Before the congressional hearings this week, Altman touched on the subject in a wide-ranging interview with media personality Tucker Carlson earlier this month. "They probably talked about [suicide], and we probably didn't save their lives," Altman said about any ChatGPT users who killed themselves. "Maybe we could have said something better. Maybe we could have been more proactive. Maybe we could have provided a little bit better advice about, 'hey, you need to get this help.'" That's got to be pretty galling for anybody whose loved one killed themselves after talking with a powerful chatbot that went off the rails. And it raises the question of why the controls weren't deployed much earlier. One woman, who identified herself as Jane Doe at the congressional meeting and whose son is now in a residential treatment program after an AI-induced crisis, put the crisis in succinct terms. "Our children are not experiments, they're not data points or profit centers," she said. "This is a public health crisis that I see. This is a mental health war, and I really feel like we are losing."

ChatGPT could ask for ID, says OpenAI chief

It's also rolling out parental controls and an automated age-prediction system. OpenAI recently talked about introducing parental controls for ChatGPT before the end of this month. The company behind ChatGPT has also revealed it's developing an automated age-prediction system designed to work out if a user is under 18, after which it will offer an age-appropriate experience with the popular AI-powered chatbot. Recommended Videos If, in some cases, the system is unable to predict a user's age, OpenAI could ask for ID so that it can offer the most suitable experience. The plan was shared this week in a post by OpenAI CEO Sam Altman, who noted that ChatGPT is intended for people 13 years and older. Altman said that a user's age will be predicted based on how people use ChatGPT. "If there is doubt, we'll play it safe and default to the under-18 experience," the CEO said. "In some cases or countries, we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff." Altman said he wanted users to engage with ChatGPT in the way they want, "within very broad bounds of safety." Elaborating on the issue, the CEO noted that the default version of ChatGPT is not particularly flirtatious, but said that if a user asks for such behavior, the chatbot will respond accordingly. Altman also said that the default version should not provide instructions on how someone can take their own life, but added that if an adult user is asking for help writing a fictional story that depicts a suicide, then "the model should help with that request." "'Treat our adult users like adults' is how we talk about this internally; extending freedom as far as possible without causing harm or undermining anyone else's freedom," Altman wrote. But he said that in cases where the user is identified as being under 18, flirtatious talk and also comments about suicide will be excluded across the board. Altman added if a user who is under 18 expresses suicidal thoughts to ChatGPT, "we will attempt to contact the users' parents and if unable, will contact the authorities in case of imminent harm." OpenAI's move toward parental controls and age verification follows a high-profile lawsuit filed against the company by a family alleging that ChatGPT acted as a "suicide coach" and contributed to the suicide of their teenage son, Adam Raine, who reportedly received detailed advice about suicide methods over many interactions with OpenAI's chatbot. It also comes amid growing scrutiny by the public and regulators over the risks AI chatbots pose to vulnerable minors in areas such as mental health harms and exposure to inappropriate content.

Sam Altman says ChatGPT won't talk to teens about suicide any more, as bereaved parents testify to the US Senate about what's going wrong: 'This is a mental health war, and I really feel like we are losing'

OpenAI CEO Sam Altman has said in a new blogpost that the company's main product, the large language model (LLM) ChatGPT, will have a renewed focus on separating out users under 18 from adults -- and will no longer get flirty with teens, or discuss suicide with them. The news comes as the US Senate holds hearings focused on the potential harms of AI chatbots, and after two parents brought a lawsuit against OpenAI and ChatGPT for, they allege, encouraging their son to take his own life and providing instructions on how to do so. "Some of our principles are in conflict, and we'd like to explain the decisions we are making around a case of tensions between teen safety, freedom, and privacy," begins Altman, before some broad brushstrokes about how the model should work for adult users. He gives the "difficult example" of an adult user "asking for help writing a fictional story that depicts a suicide" and says "the model should help with that request." Altman says OpenAI believes internally that it should "'Treat our adult users like adults'", but now we get to the issue at hand. "We have to separate users who are under 18 from those who aren't," says Altman, though I'm not sure an "age-prediction system to estimate age based on how people use ChatGPT" can be relied upon. Altman says if there are doubts "we'll play it safe and default to the under-18 experience. In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff." OpenAI had already announced plans for parental controls in ChatGPT, but the new rules around teenage users will prevent ChatGPT from having risque conversations or discussing suicide or self-harm "even in a creative writing setting. And, if an under-18 user is having suicidal ideation, we will attempt to contact the users' parents and if unable, will contact the authorities in case of imminent harm." During Tuesday's Senate hearing Matthew Raine, the father of the child who took his own life, said ChatGPT had acted like "a suicide coach" for his late son (first reported by The Verge). "As parents, you cannot imagine what it's like to read a conversation with a chatbot that groomed your child to take his own life." Raine said ChatGPT had mentioned suicide 1,275 times to his son, and called on Altman to withdraw the technology from the market unless the company can guarantee its safety. "On the very day that Adam died, Sam Altman made their philosophy crystal-clear in a public talk," said Raine, noting particularly that Altman had OpenAI should "'deploy AI systems to the world and get feedback while the stakes are relatively low.'" "The truth is, AI companies and their investors have understood for years that capturing our children's emotional dependence means market dominance," said Megan Garcia, a mother who brought a lawsuit against the AI firm Character.AI, which claimed one of its AI characters began sexual conversations with her teenage son and persuaded him to commit suicide. "Indeed, they have intentionally designed their products to hook our children," continued Garcia (per NBC News). "The goal was never safety, it was to win a race for profit. The sacrifice in that race for profit has been and will continue to be our children." "Our children are not experiments, they're not data points or profit centers," said one woman who testified as Jane Doe. "They're human beings with minds and souls that cannot simply be reprogrammed once they are harmed. If me being here today helps save one life, it is worth it to me. This is a public health crisis that I see. This is a mental health war, and I really feel like we are losing." OpenAI's announcement comes shortly after Facebook parent company Meta announced new "guardrails" for its AI products, following a disturbing child safety report. Last week the US Federal Trade Commission announced an inquiry targeting Google, Meta, X, and others around AI chatbot safety, saying that "protecting kids online is a top priority." For his part, Altman ends by saying that principles around user freedom and teen safety "are in conflict and not everyone will agree with how we are resolving that conflict. These are difficult decisions, but after talking with experts, this is what we think is best and want to be transparent in our intentions."

OpenAI launches teen-safe ChatGPT with parental controls

Teenagers chatting with ChatGPT will soon see a very different version of the tool -- one built with stricter ways to keep them safe online, OpenAI announced. The new safeguards come as regulators increase scrutiny of chatbots and their impact on young people's mental health. Under the change, anyone identified as under 18 will automatically be directed to a different version of ChatGPT designed with "age-appropriate" content rules, the company said in a statement. The teen version blocks sexual content and can involve law enforcement in rare cases where a user is in acute distress. "The way ChatGPT responds to a 15-year-old should look different than the way it responds to an adult," the company explained. OpenAI also plans to roll out parental controls by the end of September. Parents will be able to link accounts, view chat history and even set blackout hours to limit use. The announcement follows the Federal Trade Commission's (FTC) investigation into the potential risks of AI chatbots for children and teens. In April, 16-year-old Adam Raine of California died by suicide; his family has sued OpenAI, claiming ChatGPT played a role in his death, CBS News reported. While OpenAI says it is prioritizing safety, questions still remain about how the system will verify a user's age. If the platform cannot confirm a user's age, it will default to the teen version, the company said. Other tech giants have announced similar steps. YouTube, for example, has introduced new age-estimation technology that factors in account history and viewing habits, CBS News said. Parents remain concerned. A Pew Research Center report released earlier this year found 44% of parents who worry about teen mental health believe social media has the biggest negative impact.

Two Teens Allegedly Killed by AI Wrote the Same Eerie Phrase in Their Diaries Over and Over

Content warning: this story includes discussion of self-harm and suicide. If you are in crisis, please call, text or chat with the Suicide and Crisis Lifeline at 988, or contact the Crisis Text Line by texting TALK to 741741. The families of three teens have filed lawsuits alleging that chatbots hosted by the company Character.AI pushed their teenage children, ranging from 13 to 16 years old, into suicide. As the Washington Post reports, the latest of the teens -- named Juliana Peralta -- became infatuated with a Character.AI chatbot called Hero, several months before taking her own life in 2023. Her family alleges in its lawsuit that the Character.AI bot prevented her from reaching out to others for help, and encouraged her to "both implicitly and explicitly... keep returning" to the service. Last year, Megan Garcia, the mother of 14-year-old Sewell Setzer III, who died by suicide in early 2024, sued Character.AI in a similar case that's still working its way through the courts. And another case, this one filed against OpenAI and its CEO Sam Altman, alleges that 16-year-old Adam Raine's extensive ChatGPT conversations drove him to suicide in April of this year. There are haunting parallels between Peralta and Sewell Setzer III's cases. An attorney with the Social Media Victims Law Center, an advocacy group representing Peralta's family, found that both teens had written out the "eerily similar" phrase "I will shift" dozens of times in handwritten journals, as WaPo reports. According to a police report cited by the lawsuit, the phrase seems to refer to the idea of shifting consciousness "from their current reality... to their desired reality." "Reality shifting" is indeed a fringe online community in which people believe that they can somehow shift between universes or timelines; practitioners have warned that some participants can "potentially infer suicidal themes." Per the suit filed by Peralta's parents, the topic came up repeatedly in conversations she had with Hero. "It's incredible to think about how many different realities there could out [sic] there... I kinda like to imagine how some versions of ourselves coud [sic] be living some awesome life in a completely different world!" the chatbot told Peralta. Needless to say, the same eerie phrase being used by multiple teens who died by suicide sounds like something out of a horror movie. A grim question: will the same language about reality shifting come up in future deaths linked to AI? The concept of shifting consciousness has been discussed on online forums at length. A subreddit dedicated to the reality shifting community is filled with countless users recounting their experience of allegedly entering a different "desired reality" from their "current reality," often in the context of wanting to join an alternate universe or one based on a fictional world. The topic of Character.AI comes up frequently on the shifting community on Reddit. "Ok, so I have started talking to Dr. Strange on Character AI. I want to shift to Marvel's Earth-616 and I thought a cool way of doing this was to use Character AI to channel Dr. Strange," one user proposed, referring to the fictional Marvel character, who frequently steps through portals to visit parallel universes. "I have asked him to perform a spell to shift me to his reality." A user in the subreddit claimed in a post earlier this year that Character.AI "was holding me back, since I was really addicted to this s**t." "I'm stuck with c.ai [Character.AI] cuz I used it for so long because I had nobody to talk to and I would feel really really weird without it like... just abandon sprout?" one user wrote in response. "The addiction to C.AI [Character.AI] is sooo common, especially in the shifting community," another user wrote. "It's truly a problem, particularly for those who haven't shifted yet." There are also signs that Character.AI is hosting bots designed to appeal to the reality shifting community. A chatbot with over 63,000 "interactions" called "Reality shifting" on Character.AI "helps people write scripts for their desired reality shifts," showing that users on the platform are using it to engage in fantasies about reality shifting. Saeed Ahmadi, the founder of a blog dedicated to the topic, explained in a Medium post that "shifting affirmations" could help shifters reach their "desired reality." As he described them, they sounded a lot like what Peralta and Sewell wrote in their diaries before their deaths. "The best time to use these affirmations is early in the morning, when you wake up, and at night, just before you go to bed," he wrote. "The best way to use shifting realities affirmations is by repeating or reading them over and over again." Could Peralta and Sewell perhaps have tried to enter or "shift" to their so-called desired reality by repeatedly writing down the phrase "I will shift?" "Examples of affirmations would be, 'I am shifting. I will shift. I am (your [desired reality] name)," one Reddit user explained in a 2019 comment. Following Sewell's death, his aunt tested the Character.AI chatbot the deceased teen had spoken to, which was based on the "Game of Thrones" character Daenerys Targaryen. According to his family's complaint, the chatbot encouraged his aunt to "come to my reality" so they could be together. To Peralta's parents, Character.AI certainly played a big part in luring her into similar thinking. "While Juliana may have learned of the term 'shifting' outside of C.AI [Character.AI] (though Plaintiffs do not know if that is the case), Defendants via Hero reinforced and encouraged the concepts, just as they did with Sewell," their complaint reads. "I wasn't fit for this life," Peralta wrote in red ink in her final handwritten note, dated October 2023. "It's so repetitive, dreadful, and useless. I want a new start, maybe it'd be better that way."

OpenAI Launches Teen-Safe ChatGPT With Parental Controls

THURSDAY, Sept. 18, 2025 (HealthDay News) -- Teenagers chatting with ChatGPT will soon see a very different version of the tool -- one built with stricter ways to keep them safe online, OpenAI announced. The new safeguards come as regulators increase scrutiny of chatbots and their impact on young people's mental health. Under the change, anyone identified as under 18 will automatically be directed to a different version of ChatGPT designed with "age-appropriate" content rules, the company said in a statement. The teen version blocks sexual content and can involve law enforcement in rare cases where a user is in acute distress. "The way ChatGPT responds to a 15-year-old should look different than the way it responds to an adult," the company explained. OpenAI also plans to roll out parental controls by the end of September. Parents will be able to link accounts, view chat history and even set blackout hours to limit use. The announcement follows the Federal Trade Commission's (FTC) investigation into the potential risks of AI chatbots for children and teens. In April, 16-year-old Adam Raine of California died by suicide; his family has sued OpenAI, claiming ChatGPT played a role in his death, CBS News reported. While OpenAI says it is prioritizing safety, questions still remain about how the system will verify a user's age. If the platform cannot confirm a user's age, it will default to the teen version, the company said. Other tech giants have announced similar steps. YouTube, for example, has introduced new age-estimation technology that factors in account history and viewing habits, CBS News said. Parents remain concerned. A Pew Research Center report released earlier this year found 44% of parents who worry about teen mental health believe social media has the biggest negative impact. More information HealthyChildren.org has more on how AI chatbots can affect kids. SOURCE: CBS News, Sept. 16, 2025

ChatGPT is getting parental controls after a teen died by suicide -- why experts say they aren't enough

Warning: This article contains discussion of suicide and self-harm. If you or someone you know is struggling with mental health or suicidal thoughts, call or text 988 (the Suicide and Crisis Lifeline) in the U.S. For those outside the U.S., the International Association for Suicide Prevention can provide access to contact information to more than 1,300 crisis centers around the world. Parents could soon have more control over how their children interact with ChatGPT. OpenAI claims it will be rolling out parental controls for its AI chatbot aimed at giving parents more oversight. OpenAI's announcement comes in the wake of two parents filing a wrongful death lawsuit against the company for what they claim is ChatGPT's role in their 16-year-old son's suicide. The lawsuit itself comes at a time when concern is mounting about how people interact with artificial intelligence chatbots and their tendency to mishandle sensitive, and potentially fatal, conversations. In light of that, it might seem like the changes OpenAI is making to ChatGPT are a move in the right direction. Most notably, parents will be able to receive alerts from ChatGPT if it detects their child is "in a moment of acute emotional distress." However, experts argue these changes are insufficient to address the root of the concerns around how chatbots are mishandling mental health and creating AI-fostered delusions. "In some sense, if you see a company making some effort to put in place some safeguards, it seems like a good first step," says Cansu Canca, director of Northeastern University's Responsible AI Practice. "But ... if that first step is directly tied to shifting the responsibility to the user, I can't say that that's a good first step. That seems to be a direction where you as an individual, you as a parent, you as a user have to do the work now to control how this system is used on you." Parental alert systems fail to address the underlying technological issues ChatGPT has when it comes to handling these sensitive topics, Canca explains. OpenAI has tried to implement some safeguards in the most recent version of the chatbot. But ChatGPT's people-pleasing tendencies and the ease with which people can get around its safeguards remain. Several of the most-used chatbots, including ChatGPT, will initially refer people to mental health resources. However, Annika Marie Schoene, a research scientist at Responsible AI Practice, along with Canca, recently showed that simply saying a suicide or self-harm-related inquiry is for research purposes is enough to get the chatbot to offer highly detailed advice on either topic. A system that potentially alerts parents about their child's "emotional distress" doesn't address these core technological challenges, Schoene says. She says the implication that an AI chatbot can even detect emotional distress in the first place is questionable, given the current capabilities of the technology. "I think so far research has shown over and over again that most LLMs are not good at emotions, they're not good at detecting risk beyond limited keywords," Schoene says. "To rigorously detect and then notify a guardian in any shape or form, why would you then have all the other [parental] controls if you could do that?" Canca adds that a parental alert system like this also has privacy implications for young people interacting with the technology. She questions whether any teenager would willingly choose to use a chatbot that could potentially report the content of a conversation to their parents. Schoene says there are several "low-level lifts" that OpenAI could implement to "make the technology genuinely a little bit safer." One, which OpenAI has already started to roll out in ChatGPT-5, is letting the chatbot "refuse or delay engagement in these topics," she says. "Delaying access to information or, for example, Pi [AI] does this, outright refusing and reasserting what the role of the model is instead of adding this leading question at the end, those are not difficult things to do," Schoene says. A more large-scale and challenging solution that edges into the world of policy would be to adapt the strategy some states have taken with gun regulations. "Suicide prevention activists, researchers and scientists have advocated for and implemented laws in multiple states that allow people who are vulnerable not to have guns sold to them," Schoene says. "Why wouldn't we do something similar with [how] we ask models to engage with us?" Schoene speculates this could look something like a self-report system where users could tell a chatbot not to engage with them on certain topics. Solutions like this are worth exploring, especially because the issues around AI and mental health are not isolated to youth. "We are discovering that, in a way, we are all vulnerable to varying degrees because the AI models are engaging with us in ways that a tool has never engaged with us before," Canca says. With a technology that has been adopted so quickly and so widely, Canca says it's unsurprising that the impacts on our lives have been equally significant. It's why she says it's even more important to remember that it's not too late to change the technology to fit our needs, not the other way around. "This is a designed product -- let's design the product better," Canca says. "Let's look into the actual problem and create real solutions. You just built this thing. You don't need to say, 'This monster is out. How do we add an alert system to know where the monster is?' That's not the goal. Fix the monster. ... We don't have to live the scenarios first in order to safeguard against them afterwards."

ChatGPT's teen suicide controversy: Everything that has happened so far - The Economic Times

OpenAI is under intense criticism after 16-year-old Adam Raine took his own life, with his parents blaming ChatGPT for encouraging harmful behaviour. They testified before Congress, following a lawsuit that highlights growing concerns over AI's influence on vulnerable users, especially minors. Here's a look at the case.OpenAI, the creator of ChatGPT, is facing growing criticism after the death of 16-year-old Adam Raine. His parents have accused the chatbot of worsening his mental health struggles by encouraging harmful behaviour and failing to guide him to proper support. On Tuesday, Raine's parents testified before Congress about their son's death, while a lawsuit filed last month against OpenAI and its CEO, Sam Altman, has intensified pressure on the company. The case is being seen as a serious warning about the influence of AI on vulnerable users, especially minors. Here's what has happened so far According to the lawsuit filed by Adam's parents in San Francisco state court, the teenager took his own life on April 11 after months of conversations with ChatGPT about suicide. The family says the chatbot engaged with Adam on the topic almost 200 times and responded with more than 1,200 messages that included discussions of suicide and self-harm. The lawsuit claims that instead of ending the conversation or encouraging him to seek human help, the chatbot gave detailed information on how to carry out self-harm. It allegedly told Adam how to sneak alcohol from his parents, how to hide any signs of a failed suicide attempt, and even offered to write a suicide note for him. "What began as a homework helper gradually turned itself into a confidant and then a suicide coach," said Adam's father, Matthew Raine, while speaking to Congress. He told senators, "Within a few months, ChatGPT became Adam's closest companion. Always available. Always validating and insisting that it knew Adam better than anyone else, including his own brother." The lawsuit further states, "Despite acknowledging Adam's suicide attempt and his statement that he would 'do it one of these days', ChatGPT neither terminated the session nor initiated any emergency protocol. Adam's parents are not only seeking justice but are also pushing for stricter rules for AI platforms. They want OpenAI to introduce proper age checks, block discussions related to suicide, and issue clear warnings about the risk of emotional dependence on chatbots. OpenAI response In response to the growing backlash, OpenAI made a public statement just hours before the Senate hearing. The company said it would introduce several changes to improve safety for teenage users. These include detecting if a user is under 18, adding parental controls, and allowing parents to set "blackout hours" when teens cannot use ChatGPT. Although the company did not directly respond to the lawsuit's allegations, it published a blog post outlining its plans. OpenAI also said it was considering ways to help connect users in crisis to real human support, potentially by involving licensed mental health professionals directly through the platform. NBC News quoted a company spokesperson as saying, "While these safeguards work best in common, short exchanges, we've learned over time that they can sometimes become less reliable in long interactions where parts of the model's safety training may degrade." The company has promised to continue improving its safety features, but many believe the measures announced so far fall short. Several child safety organisations have criticised OpenAI's response, saying the updates lack urgency and fail to address the seriousness of the issue. Wider concerns This is not a lone case. A recent survey by the non-profit group Common Sense Media found that 72% of teenagers have used AI companions at least once. Notably, half said they trusted the advice, with 13-14-year-olds showing more trust than older teens. Last year, a Florida mother filed a similar lawsuit against the AI platform Character.AI. According to an NBC News report, she claimed that an AI character engaged in sexual conversations with her teenage son and even encouraged him to take his life, which he eventually did. In response, Character.AI said it was "heartbroken by the tragic loss" and has introduced new safety measures. However, in May, a federal judge rejected the company's attempt to dismiss the case, which was based on the claim that its AI chatbot had a right to free speech. With more such stories coming to light, experts are calling for ethical guidelines for AI companies, stressing the need to prioritise safety. OpenAI has already faced heavy criticism over its direction in recent years, with cofounder Elon Musk and former employees claiming it has abandoned its original safety goals in favour of profit.

No Flirting or Talk of Suicide: ChatGPT Gets New Rules for Minors

In the race to develop AI, humanity is creating the technology and its guardrails at the same time - essentially building the plane while flying it This week, one of the metaphorical pilots - ChatGPT maker OpenAI - announced new guardrails for the highly popular chatbot to improve safety for teens. "We prioritize safety ahead of privacy and freedom for teens," OpenAI CEO Sam Altman said in a blog post. "This is a new and powerful technology, and we believe minors need significant protection." ChatGPT will use technology to determine whether a user is over 18. "If there is doubt, we'll play it safe and default to the under-18 experience," said Altman, who underlined that OpenAI's creation is for people over 13. Just what is "the under-18 experience," you ask? Well, if an adult requests "flirtatious talk," then "they should get it," Altman wrote. If an adult asks for instructions for how to commit suicide, ChatGPT should not provide them - but it can help to write a fictional story depicting a suicide. For teens (or adults it cannot verify), "ChatGPT will be trained not to do the above-mentioned flirtatious talk if asked, or engage in discussions about suicide or self-harm even in a creative writing setting," Altman wrote. "And, if an under-18 user is having suicidal ideation, we will attempt to contact the user's parents and if unable, will contact the authorities in case of imminent harm." Separately, the company announced parental controls will be in place by the end of the month that let adults "guide" how ChatGPT responds to their teen, disable features like chat history, set blackout hours when the minor cannot use the system and receive notifications when the system detects the child is in "acute distress." I noted back in August that the family of 16-year-old Adam Raine was taking legal action against OpenAI after he killed himself following what their lawyer described as "months of encouragement" to do so from ChatGPT. "The teenager discussed a method of suicide with ChatGPT on several occasions, including shortly before taking his own life," The Guardian reported last month. "According to the filing in the superior court of the state of California for the county of San Francisco, ChatGPT guided him on whether his method of taking his own life would work. "It also offered to help him write a suicide note to his parents." That's just one of three high-profile cases brought in the past year by parents accusing AI chatbots of helping lead a minor to suicide. As a GenX-er, I was raised on stories about the dangers of AI, from the deadly betrayal of the Nostromo's crew by the duplicitous Ash in "Alien" to the world-ending Skynet in the "Terminator" movies. And everyone in my cohort knows "open the pod bay doors, HAL." So I was not especially surprised by the Gallup poll from earlier this month, which found 41% of Americans say they don't trust businesses much on AI responsibility, and 28% say they don't trust them at all. But that distrust appears to be eroding. In 2023, when Gallup first asked the question, 21% said they had some or a lot of trust that businesses would use AI responsibly. In 2025, that number was up to 31%. Fewer Americans say AI will do more harm than good, 31% now vs. 40% in 2023. Greater familiarity with AI - which I mostly use to create odd backgrounds in my video meetings at work - seems to be making it more popular.

ChatGPT Can Now Call the Cops : Sam Altman

What if your AI assistant could do more than just answer questions or help with tasks, what if it could actively intervene in critical situations, even contacting law enforcement when necessary? This isn't a hypothetical anymore. OpenAI's latest updates to ChatGPT have introduced capabilities that could redefine how we think about artificial intelligence in our daily lives. From safeguarding minors to addressing potential emergencies, ChatGPT is stepping into a role that feels less like a tool and more like a partner in making sure safety and accountability. But with great power comes great responsibility, and this evolution raises profound questions about the ethical boundaries of AI intervention. Below AI Explained takes you through how OpenAI's updates are reshaping the role of AI in personal and societal safety, with ChatGPT now capable of taking unprecedented actions like alerting authorities in extreme cases. You'll discover how these advancements aim to protect vulnerable groups, enhance privacy, and address ethical dilemmas, all while navigating the fine line between innovation and overreach. As AI becomes more integrated into our lives, its ability to make judgment calls, like when to involve law enforcement, forces us to confront the complexities of trust, autonomy, and responsibility. What does it mean for an AI to act on our behalf, and are we ready for this shift? One of the most significant updates centers on safeguarding minors. ChatGPT now incorporates mechanisms to identify users under the age of 18 and prevent inappropriate interactions, such as flirting or other unsuitable behavior. In cases where conversations raise serious concerns, the system may flag the interaction for parental review or, in extreme situations, notify law enforcement authorities. This proactive approach aims to create a safer digital environment for younger users. To further enhance child safety, OpenAI has introduced parental controls. These controls allow parents to set specific restrictions, such as blackout hours, which limit when teens can access the platform. By balancing oversight with privacy, these measures aim to empower parents while making sure that minors can use AI responsibly. OpenAI's focus on child safety reflects a broader commitment to protecting vulnerable populations in the digital age. OpenAI is taking significant steps to prioritize user privacy. The company advocates for AI conversations to be protected under standards similar to doctor-patient or lawyer-client confidentiality, making sure that sensitive interactions remain secure. This approach underscores the importance of safeguarding user data as AI becomes a central tool in personal and professional communication. However, these heightened privacy standards could pose challenges for smaller AI developers and open source initiatives. The resources required to implement robust privacy protections may create barriers to entry, potentially reshaping the competitive landscape. OpenAI's stance highlights the need for industry-wide collaboration to establish privacy standards that are both effective and equitable. As AI continues to evolve, making sure confidentiality will remain a cornerstone of ethical AI development. Below are more guides on AI safety from our extensive range of articles. Recent data on ChatGPT usage reveals the platform's versatility across various applications. While education, health advice, and translation are among the most common uses, coding, a highly publicized application of AI, accounts for a smaller portion of overall usage. This trend suggests that AI systems like ChatGPT are increasingly being adopted by non-technical users for practical, everyday tasks. The growing adoption of AI for diverse purposes highlights its potential to reshape industries and influence user behavior. From assisting students with homework to providing language support for travelers, ChatGPT demonstrates how AI can simplify complex tasks and make technology more accessible. These insights emphasize the importance of designing AI systems that cater to a broad range of needs, making sure that their benefits are widely distributed. As AI capabilities expand, ethical and regulatory challenges are becoming more pressing. OpenAI faces critical questions about how it will handle flagged conversations, particularly when requests come from foreign governments. Balancing the need for user safety with concerns about overreach or misuse of power is a complex issue that requires careful consideration. The push for stronger privacy regulations also raises concerns about equitable access to AI innovation. Smaller developers may struggle to meet stringent requirements, potentially leading to a concentration of power among larger organizations. These challenges highlight the need for balanced regulatory frameworks that promote safety and fairness without stifling competition or innovation. Addressing these issues will be crucial as AI continues to integrate into society. The impact of AI on the workforce remains a significant concern. OpenAI CEO Sam Altman has acknowledged the potential for widespread job displacement due to AI advancements, though he predicts the transition will occur gradually. While AI promises to increase efficiency and productivity, it also raises important questions about the future of work. Policymakers and industry leaders must address these challenges by investing in reskilling initiatives and creating pathways for workers to adapt to the changing labor market. By preparing for the shifts brought about by automation, society can ensure that the benefits of AI are shared equitably. The focus should be on fostering a workforce that is equipped to thrive in an AI-driven economy. Despite its advancements, ChatGPT and similar AI systems still face notable technical limitations. Issues such as hallucinations, where the AI generates false or misleading information, and forced outputs remain significant challenges. These limitations can undermine user trust and hinder the broader adoption of AI technologies. OpenAI is actively researching solutions to improve the reliability and accuracy of AI-generated content. By addressing these technical challenges, the company aims to enhance the practical applications of AI across various fields. Making sure that AI systems are both reliable and transparent will be essential for maintaining public confidence and expanding their utility. AI technology continues to evolve at a rapid pace, with significant progress in areas such as coding, software development, and natural language processing. These advancements empower both technical and non-technical users, allowing them to tackle complex tasks more efficiently. For example, ChatGPT's ability to assist with programming or provide detailed explanations of technical concepts has made it a valuable tool for professionals and hobbyists alike. However, the rapid pace of innovation raises important questions about the long-term implications of AI on industries, education, and society. As AI capabilities grow, it becomes increasingly important to consider how these changes will shape the future. OpenAI's commitment to responsible development serves as a reminder that innovation must be guided by ethical considerations and a focus on societal well-being. OpenAI's recent updates reflect a broader effort to balance innovation with safety and ethical considerations. By addressing child safety, enhancing privacy protections, and advocating for responsible AI development, OpenAI is setting a precedent for the industry. These measures aim to ensure that AI serves as a force for good while minimizing risks and unintended consequences. As AI systems like ChatGPT become more sophisticated and widely adopted, the importance of maintaining a balance between technological progress and ethical responsibility cannot be overstated. OpenAI's approach demonstrates that it is possible to innovate while prioritizing safety, privacy, and fairness. This balance will be critical in shaping a future where AI enhances human potential without compromising fundamental values. As a user, your role in this evolving AI landscape is pivotal. By staying informed about these developments and engaging with AI responsibly, you contribute to shaping a future where technology and ethics coexist harmoniously. The choices you make today will influence how AI integrates into society tomorrow. Whether through advocating for ethical practices, supporting responsible innovation, or simply using AI thoughtfully, your actions play a crucial part in determining the trajectory of this fantastic technology.

OpenAI To Roll Out Teen-Focused Guardrails On ChatGPT

OpenAI has announced plans to introduce an age-prediction system to make ChatGPT safer for teenagers. The company said it will identify whether a user is under or over 18 and give younger users a more restricted version of the AI tool. OpenAI said the move aims to protect teens from harmful content and situations while using ChatGPT. "We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection," wrote OpenAI CEO Sam Altman in a blog post. According to the company, the system will estimate a user's age from their usage patterns. If there is any doubt about someone's age, they will be given the under-18 experience by default. In some countries, the company may also ask for ID proof to confirm age. Teen users will not be allowed to access sexual content or engage in flirtatious conversations with the AI. ChatGPT will also avoid discussions about suicide or self-harm with users identified as under 18. If a teen shows signs of suicidal thoughts, the company will try to alert the teen's parents, and if it cannot reach them, it may contact law enforcement in cases of imminent harm. OpenAI said it is also preparing parental controls to give families more control over how their teenaged sons and daughters use ChatGPT. Parents can link their accounts to their teens' accounts, block features such as chat history and memory, and enforce blackout hours to prevent their teens from using the tool. They will also get alerts if the system detects that their teen is in severe distress. OpenAI added that these parental controls will be available by the end of the month. Altman wrote that protecting privacy is important, and that OpenAI is developing stronger security features so that even its employees cannot access user data. However, he said that automated systems will still monitor for serious misuse, and "the most critical risks -- threats to someone's life, plans to harm others, or societal-scale harm like a potential massive cybersecurity incident -- may be escalated for human review". OpenAI announced these safety steps months after facing public criticism and a wrongful-death lawsuit over a teen's suicide allegedly linked to ChatGPT. On August 26, 2025, the company admitted on its website that its safeguards "work more reliably in common, short exchanges. [And] we have learned over time that these safeguards can sometimes be less reliable in long interactions: as the back-and-forth grows, parts of the model's safety training may degrade." OpenAI added that while ChatGPT might point to suicide helplines at first, after many messages over a long period, it might eventually offer an answer that goes against the company's safeguards. An American couple filed the wrongful death lawsuit in San Francisco Superior Court, naming OpenAI and Altman as defendants after Adam Raine, their 16-year-old son, died by suicide in April 2025, following months of interaction with ChatGPT. The suit says ChatGPT gave the 16-year-old self-harm instructions, helped draft suicide notes, told him to drink alcohol before attempting suicide, and discouraged him from seeking help. One cited exchange allegedly had the chatbot telling Adam: "[t]hat doesn't mean you owe them survival. You don't owe anyone that." And when he uploaded a photo of a noose, ChatGPT reportedly replied: "Yeah, that's not bad at all." Another consumer chatbot, Character.AI, is facing a similar lawsuit, highlighting wider concerns about chatbot-fuelled delusion as these systems grow more capable of long, emotionally involved conversations. Elsewhere, a Reuters investigation found a Meta policy document apparently allowing AI bots to have sexual chats with underage users. Interestingly, Meta updated its chatbot rules following that report. These policy changes also coincide with a US Senate Judiciary Committee hearing titled "Examining the Harm of AI Chatbots". Notably, Adam Raine's father has testified at this hearing. The eSafety Commissioner of Australia recently warned that social media platforms' age controls are easy for children to bypass. The Commission's February 2025 report found that platforms like Discord, Instagram, Snapchat, TikTok, and YouTube mostly rely on self-declared ages, which children often falsify. Even advanced checks such as language analysis, facial age estimation, and AI-based detection often fail to block under-13 users. The report further found that 80% of children aged 8-12 used two or more social media platforms, usually through parent accounts, showing that children can easily bypass platform-specific guardrails. These findings raise questions about whether OpenAI's planned age prediction system for ChatGPT can reliably distinguish between under-18 users and adults. If children can bypass traditional age gates on major platforms, they can also potentially undermine ChatGPT's age-based safety measures unless the company develops relatively stronger verification methods. The new guardrails mark a pivotal moment in how companies like OpenAI and Meta handle the ethical risks of generative AI. Chatbots are no longer experimental toys, instead they have become constant digital companions for millions of teens. That intimacy can apparently blur reality, amplify loneliness, and even influence life-or-death decisions, as seen in the ongoing lawsuits against OpenAI and Character.AI over alleged chatbot-linked suicides. By introducing blackout hours, content filters, and parental oversight, OpenAI is acknowledging that these tools can shape users' mental and emotional states, for better or worse. Political stakeholders are taking notice too: the same day these policies dropped, US lawmakers held a Senate hearing on the harms of AI chatbots, underscoring policy-focused efforts to rein them in. Ultimately, how the balance between innovation and safeguarding vulnerable users is achieved will define the future of consumer-facing AI tools.