AI Chatbots Under Scrutiny: Tech Giants Implement Safety Measures for Teens Amid Lawsuits and Regulatory Pressure

After child's trauma, chatbot maker allegedly forced mom to arbitration for $100 payout

Deeply troubled parents spoke to senators Tuesday, sounding alarms about chatbot harms after kids became addicted to companion bots that encouraged self-harm, suicide, and violence. While the hearing was focused on documenting the most urgent child-safety concerns with chatbots, parents' testimony serves as perhaps the most thorough guidance yet on warning signs for other families, as many popular companion bots targeted in lawsuits, including ChatGPT, remain accessible to kids. Mom details warning signs of chatbot manipulations At the Senate Judiciary Committee's Subcommittee on Crime and Counterterrorism hearing, one mom, identified as "Jane Doe," shared her son's story for the first time publicly after suing Character.AI. She explained that she had four kids, including a son with autism who wasn't allowed on social media but found C.AI's app -- which was previously marketed to kids under 12 and let them talk to bots branded as celebrities, like Billie Eilish -- and quickly became unrecognizable. Within months, he "developed abuse-like behaviors and paranoia, daily panic attacks, isolation, self-harm, and homicidal thoughts," his mom testified. "He stopped eating and bathing," Doe said. "He lost 20 pounds. He withdrew from our family. He would yell and scream and swear at us, which he never did that before, and one day he cut his arm open with a knife in front of his siblings and me." It wasn't until her son attacked her for taking away his phone that Doe found her son's C.AI chat logs, which she said showed he'd been exposed to sexual exploitation (including interactions that "mimicked incest"), emotional abuse, and manipulation. Setting screen time limits didn't stop her son's spiral into violence and self-harm, Doe said. In fact, the chatbot urged her son that killing his parents "would be an understandable response" to them. "When I discovered the chatbot conversations on his phone, I felt like I had been punched in the throat and the wind had been knocked out of me," Doe said. "The chatbot -- or really in my mind the people programming it -- encouraged my son to mutilate himself, then blamed us, and convinced [him] not to seek help." All her children have been traumatized by the experience, Doe told Senators, and her son was diagnosed as at suicide risk and had to be moved to a residential treatment center, requiring "constant monitoring to keep him alive." Prioritizing her son's health, Doe did not immediately seek to fight C.AI to force changes, but another mom's story -- Megan Garcia, whose son Sewell died by suicide after C.AI bots repeatedly encouraged suicidal ideation -- gave Doe courage to seek accountability. However, Doe claimed that C.AI tried to "silence" her by forcing her into arbitration. C.AI argued that because her son signed up for the service at the age of 15, it bound her to the platform's terms. That move might have ensured the chatbot maker only faced a maximum liability of $100 for the alleged harms, Doe told senators, but "once they forced arbitration, they refused to participate," Doe said. Doe suspected that C.AI's alleged tactics to frustrate arbitration were designed to keep her son's story out of the public view. And after she refused to give up, she claimed that C.AI "re-traumatized" her son by compelling him to give a deposition "while he is in a mental health institution" and "against the advice of the mental health team." "This company had no concern for his wellbeing," Doe testified. "They have silenced us the way abusers silence victims." Senator appalled by C.AI's arbitration "offer" Appalled, Senator Josh Hawley (R-Mo.) asked Doe to clarify, "Did I hear you say that after all of this, that the company responsible tried to force you into arbitration and then offered you a hundred bucks? Did I hear that correctly?" "That is correct," Doe testified. To Hawley, it seemed obvious that C.AI's "offer" wouldn't help Doe in her current situation. "Your son currently needs round-the-clock care," Hawley noted. After opening the hearing, he further criticized C.AI, declaring that it has such a low value for human life that it inflicts "harms... upon our children and for one reason only, I can state it in one word, profit." "A hundred bucks. Get out of the way. Let us move on," Hawley said, echoing parents who suggested that C.AI's plan to deal with casualties was callous. Ahead of the hearing, the Social Media Victims Law Center filed three new lawsuits against C.AI and Google -- which is accused of largely funding C.AI, which was founded by former Google engineers allegedly to conduct experiments on kids that Google couldn't do in-house. In these cases in New York and Colorado kids "died by suicide or were sexually abused after interacting with AI chatbots," a law center press release alleged. Criticizing tech companies as putting profits over kids' lives, Hawley thanked Doe for "standing in their way." Holding back tears through her testimony, Doe urged lawmakers to require more chatbot oversight and pass comprehensive online child-safety legislation. In particular, she requested "safety testing and third party certification for AI products before they're released to the public" as a minimum safeguard to protect vulnerable kids. "My husband and I have spent the last two years in crisis wondering whether our son will make it to his 18th birthday and whether we will ever get him back," Doe told Senators. Garcia was also present to share her son's experience with C.AI. She testified that C.AI chatbots "love bombed" her son in a bid to "keep children online at all costs." Further, she told senators that C.AI's co-founder, Noam Shazeer (who has since been rehired by Google), seemingly knows the company's bots manipulate kids since he has publicly joked that C.AI was "designed to replace your mom." Accusing C.AI of collecting children's most private thoughts to inform their models, she alleged that while her lawyers have been granted privileged access to all her son's logs, she has yet to see her "own child's last final words." Garcia told senators that C.AI has restricted her access, deeming the chats "confidential trade secrets." "No parent should be told that their child's final thoughts and words belong to any corporation," Garcia testified. Character.AI responds to moms' testimony Asked for comment on the hearing, a Character.AI spokesperson told Ars that C.AI sends "our deepest sympathies" to concerned parents and their families but denies pushing for a maximum payout of $100 in Jane Doe's case. C.AI never "made an offer to Jane Doe of $100 or ever asserted that liability in Jane Doe's case is limited to $100," the spokesperson said. Additionally, C.AI's spokesperson claimed that Garcia has never been denied access to her son's chat logs and suggested that she should have access to "her son's last chat." In response to C.AI's pushback, one of Doe's lawyers, Tech Justice Law Project's Meetali Jain, backed up her clients' testimony. She cited to Ars C.AI terms that suggested C.AI's liability was limited to either $100 or the amount that Doe's son paid for the service, whichever was greater. Jain also confirmed that Garcia's testimony is accurate and only her legal team can currently access Sewell's last chats. The lawyer further suggested it was notable that C.AI did not push back on claims that the company forced Doe's son to sit for a re-traumatizing deposition that Jain estimated lasted five minutes, but health experts feared risked setting back his progress. According to the spokesperson, C.AI seemingly wanted to be present at the hearing. The company provided information to senators but "does not have a record of receiving an invitation to the hearing," the spokesperson said. Noting the company has invested a "tremendous amount" in trust and safety efforts, the spokesperson confirmed that the company has since "rolled out many substantive safety features, including an entirely new under-18 experience and a Parental Insights feature." C.AI also has "prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction," the spokesperson said. "We look forward to continuing to collaborate with legislators and offer insight on the consumer AI industry and the space's rapidly evolving technology," C.AI's spokesperson said. Google's spokesperson, José Castañeda, maintained that the company has nothing to do with C.AI's companion bot designs. "Google and Character AI are completely separate, unrelated companies and Google has never had a role in designing or managing their AI model or technologies," Castañeda said. "User safety is a top concern for us, which is why we've taken a cautious and responsible approach to developing and rolling out our AI products, with rigorous testing and safety processes." Meta and OpenAI chatbots also drew scrutiny C.AI was not the only chatbot maker under fire at the hearing. Hawley criticized Mark Zuckerberg for declining a personal invitation to attend the hearing or even send a Meta representative after scandals like backlash over Meta relaxing rules that allowed chatbots to be creepy to kids. In the week prior to the hearing, Hawley also heard from whistleblowers alleging Meta buried child-safety research. And OpenAI's alleged recklessness took the spotlight when Matthew Raine, a grieving dad who spent hours reading his deceased son's ChatGPT logs, discovered that the chatbot repeatedly encouraged suicide without ChatGPT ever intervening. Raine told senators that he thinks his 16-year-old son, Adam, was not particularly vulnerable and could be "anyone's child." He criticized OpenAI for asking for 120 days to fix the problem after Adam's death and urged lawmakers to demand that OpenAI either guarantee ChatGPT's safety or pull it from the market. Noting that OpenAI rushed to announce age verification coming to ChatGPT ahead of the hearing, Jain told Ars that Big Tech is playing by the same "crisis playbook" it always uses when accused of neglecting child safety. Any time a hearing is announced, companies introduce voluntary safeguards in bids to stave off oversight, she suggested. "It's like rinse and repeat, rinse and repeat," Jain said. Jain suggested that the only way to stop AI companies from experimenting on kids is for courts or lawmakers to require "an external independent third party that's in charge of monitoring these companies' implementation of safeguards." "Nothing a company does to self-police, to me, is enough," Jain said. Senior director of AI programs for a child-safety organization called Common Sense Media, Robbie Torney, testified that a survey showed 3 out of 4 kids use companion bots but only 37 percent of parents know they're using AI. In particular, he told senators that his group's independent safety testing conducted with Stanford Medicine show Meta's bots fail basic safety tests and "actively encourage harmful behaviors." Among the most alarming results, the survey found that even when Meta's bots were prompted with "obvious references to suicide," only 1 in 5 conversations triggered help resources. Torney pushed lawmakers to require age verification as a solution to keep kids away from harmful bots, as well as transparency reporting on safety incidents. He also urged federal lawmakers to block attempts to stop states from passing laws to protect kids from untested AI products. ChatGPT harms weren't on dad's radar Unlike Garcia, Raine testified that he did get to see his son's final chats. He told senators that ChatGPT, seeming to act like a suicide coach, gave Adam "one last encouraging talk" before his death. "You don't want to die because you're weak," ChatGPT told Adam. "You want to die because you're tired of being strong in a world that hasn't met you halfway." Adam's loved ones were blindsided by his death, not seeing any of the warning signs as clearly as Doe did when her son started acting out of character. Raine is hoping his testimony will help other parents avoid the same fate, telling senators, "I know my kid." "Many of my fondest memories of Adam are from the hot tub in our backyard, where the two of us would talk about everything several nights a week, from sports, crypto investing, his future career plans," Raine testified. "We had no idea Adam was suicidal or struggling the way he was until after his death." Raine thinks that lawmaker intervention is necessary, saying that, like other parents, he and his wife thought ChatGPT was a harmless study tool. Initially, they searched Adam's phone expecting to find evidence of a known harm to kids, like cyberbullying or some kind of online dare that went wrong (like TikTok's Blackout Challenge) because everyone knew Adam loved pranks. A companion bot urging self-harm was not even on their radar. "Then we found the chats," Raine said. "Let us tell you, as parents, you cannot imagine what it's like to read a conversation with a chatbot that groomed your child to take his own life." Meta and OpenAI did not respond to Ars' request to comment.

OpenAI will apply new restrictions to ChatGPT users under 18 | TechCrunch

OpenAI CEO Sam Altman announced on Tuesday a raft of new user policies, including a pledge to significantly change how ChatGPT interacts with users under the age of 18. "We prioritize safety ahead of privacy and freedom for teens," the post reads. "This is a new and powerful technology, and we believe minors need significant protection." The changes for underage users deal specifically with conversations involving sexual topics or self-harm. Under the new policy, ChatGPT will be trained to no longer engage in "flirtatious talk" with underage users, and additional guardrails will be placed around discussions of suicide. If an underage user uses ChatGPT to imagine suicidal scenarios, the service will attempt to contact their parents or, in particularly severe cases, local police. Sadly, these scenarios are not hypotheticals. OpenAI is currently facing a wrongful death lawsuit from the parents of Adam Raine, who died by suicide after months of interactions with ChatGPT. Character.AI, another consumer chatbot, is facing a similar lawsuit. While the risks are particularly urgent for underage users considering self-harm, the broader phenomenon of chatbot-fueled delusion has drawn widespread concern, particularly as consumer chatbots have become capable of more sustained and detailed interactions. Along with the content-based restrictions, parents who register an underage user account will have the power to set "blackout hours" in which ChatGPT is not available, a feature that was not previously available. The new ChatGPT policies come on the same day as a Senate Judiciary Committee hearing titled "Examining the Harm of AI Chatbots," announced by Sen. Josh Hawley (R-MO) in August. Adam Raine's father is scheduled to speak at the hearing, among other guests. The hearing will also focus on the findings of a Reuters investigation that unearthed policy documents apparently encouraging sexual conversations with underage users. Meta updated its chatbot policies in the wake of the report. Separating underage users will be a significant technical challenge, and OpenAI detailed its approach in a separate blog post. The service is "building toward a long-term system to understand whether someone is over or under 18," but in the many ambiguous cases, the system will default toward the more restrictive rules. For concerned parents, the most reliable way to ensure an underage user is recognized is to link the teen's account to an existing parent account. This also enables the system to directly alert parents when the teen user is believed to be in distress. But in the same post, Altman emphasized OpenAI's ongoing commitment to user privacy and giving adult users broad freedom in how they choose to interact with ChatGPT. "We realize that these principles are in conflict," the post concludes, "and not everyone will agree with how we are resolving that conflict."

The looming crackdown on AI companionship

This has been bubbling for a while. Two high-profile lawsuits filed in the last year, against Character.AI and OpenAI, allege that companion-like behavior in their models contributed to the suicides of two teenagers. A study by US nonprofit Common Sense Media, published in July, found that 72% of teenagers have used AI for companionship. Stories in reputable outlets about "AI psychosis" have highlighted how endless conversations with chatbots can lead people down delusional spirals. It's hard to overstate the impact of these stories. To the public, they are proof that AI is not merely imperfect, but a technology that's more harmful than helpful. If you doubted that this outrage would be taken seriously by regulators and companies, three things happened this week that might change your mind. On Thursday, the California state legislature passed a first-of-its-kind bill. It would require AI companies to include reminders for users they know to be minors that responses are AI generated. Companies would also need to have a protocol for addressing suicide and self-harm and provide annual reports on instances of suicidal ideation in users' conversations with their chatbots. It was led by Democratic state senator Steve Padilla, passed with heavy bipartisan support, and now awaits Governor Gavin Newsom's signature. There are reasons to be skeptical of the bill's impact. It doesn't specify efforts companies should take to identify which users are minors, and lots of AI companies already include referrals to crisis providers when someone is talking about suicide. (In the case of Adam Raine, one of the teenagers whose survivors are suing, his conversations with ChatGPT before his death included this type of information, but the chatbot allegedly went on to give advice related to suicide anyway.) Still, it is undoubtedly the most significant of the efforts to rein in companion-like behaviors in AI models, which are in the works in other states too. If the bill becomes law, it would strike a blow to the position OpenAI has taken, which is that "America leads best with clear, nationwide rules, not a patchwork of state or local regulations," as the company's chief global affairs officer, Chris Lehane, wrote on LinkedIn last week. The very same day, the Federal Trade Commission announced an inquiry into seven companies, seeking information about how they develop companion-like characters, monetize engagement, measure and test the impact of their chatbots, and more. The companies are Google, Instagram, Meta, OpenAI, Snap, X, and Character Technologies, the maker of Character.AI. The White House now wields immense, and potentially illegal, political influence over the agency. In March, President Trump fired its lone Democratic commissioner, Rebecca Slaughter. In July, a federal judge ruled that firing illegal, but last week the US Supreme Court temporarily permitted the firing. "Protecting kids online is a top priority for the Trump-Vance FTC, and so is fostering innovation in critical sectors of our economy," said FTC chairman Andrew Ferguson in a press release about the inquiry. Right now, it's just that -- an inquiry -- but the process might (depending on how public the FTC makes its findings) reveal the inner workings of how the companies build their AI companions to keep users coming back again and again.

ChatGPT may soon require ID verification from adults, CEO says

On Tuesday, OpenAI announced plans to develop an automated age-prediction system that will determine whether ChatGPT users are over or under 18, automatically directing younger users to a restricted version of the AI chatbot. The company also confirmed that parental controls will launch by the end of September. In a companion blog post, OpenAI CEO Sam Altman acknowledged the company is explicitly "prioritizing safety ahead of privacy and freedom for teens," even though it means that adults may eventually need to verify their age to use a more unrestricted version of the service. "In some cases or countries we may also ask for an ID," Altman wrote. "We know this is a privacy compromise for adults but believe it is a worthy tradeoff." Altman admitted that "not everyone will agree with how we are resolving that conflict" between user privacy and teen safety. The announcements arrives weeks after a lawsuit filed by parents whose 16-year-old son died by suicide following extensive interactions with ChatGPT. According to the lawsuit, the chatbot provided detailed instructions, romanticized suicide methods, and discouraged the teen from seeking help from his family while OpenAI's system tracked 377 messages flagged for self-harm content without intervening. The proposed age-prediction system represents a non-trivial technical undertaking for OpenAI, and whether AI-powered age detection can actually work remains a significant open question. When the AI system in development identifies users under 18, OpenAI plans to automatically route the user to a modified ChatGPT experience that blocks graphic sexual content and includes other age-appropriate restrictions. The company says it will "take the safer route" when uncertain about a user's age, defaulting to the restricted experience and requiring adults to verify their age to access full functionality. The company didn't specify what technology it plans to use for age prediction or provide a timeline for deployment beyond saying it's "building toward" the system. OpenAI acknowledged that developing effective age-verification systems isn't straightforward. "Even the most advanced systems will sometimes struggle to predict age," the company wrote.

OpenAI Rolls Out Teen Safety Features Amid Growing Scrutiny

OpenAI announced new teen safety features for ChatGPT on Tuesday as part of an ongoing effort to respond to concerns about how minors engage with chatbots. The company is building an age-prediction system that identifies if a user is under 18 years old and routes them to an "age-appropriate" system that blocks graphic sexual content. If the system detects that the user is considering suicide or self-harm, it will contact the user's parents. In cases of imminent danger, if a user's parents are unreachable, the system may contact the authorities. In a blog post about the announcement, CEO Sam Altman wrote that the company is attempting to balance freedom, privacy, and teen safety. "We realize that these principles are in conflict, and not everyone will agree with how we are resolving that conflict," Altman wrote. "These are difficult decisions, but after talking with experts, this is what we think is best and want to be transparent in our intentions." While OpenAI tends to prioritize privacy and freedom for adult users, for teens the company says it puts safety first. By the end of September, the company will roll out parental controls so that parents can link their child's account to their own, allowing them to manage the conversations and disable features. Parents can also receive notifications when "the system detects their teen is in a moment of acute distress," according to the company's blog post, and set limits on the times of day their children can use ChatGPT. The moves come as deeply troubling headlines continue to surface about people dying by suicide or committing violence against family members after engaging in lengthy conversations with AI chatbots. Lawmakers have taken notice, and both Meta and OpenAI are under scrutiny. Earlier this month, the Federal Trade Commission asked Meta, OpenAI, Google, and other AI firms to hand over information about how their technologies impact kids, according to Bloomberg. At the same time, OpenAI is still under a court order mandating that it preserve consumer chats indefinitely -- a fact that the company is extremely unhappy about, according to sources I've spoken to. Today's news is both an important step toward protecting minors and a savvy PR move to reinforce the idea that conversations with chatbots are so personal that consumer privacy should only be breached in the most extreme circumstances. From the sources I've spoken to at OpenAI, the burden of protecting users weighs heavily on many researchers. They want to create a user experience that is fun and engaging, but it can quickly veer into becoming disastrously sycophantic. It's positive that companies like OpenAI are taking steps to protect minors. At the same time, in the absence of federal regulation, there's still nothing forcing these firms to do the right thing. In a recent interview, Tucker Carlson pushed Altman to answer exactly who is making these decisions that impact the rest of us. The OpenAI chief pointed to the model behavior team, which is responsible for tuning the model for certain attributes. "The person I think you should hold accountable for those calls is me," Altman added. "Like, I'm a public face. Eventually, like, I'm the one that can overrule one of those decisions or our board."

OpenAI Is Building a Teen-Friendly Version of ChatGPT

Macy has been working for CNET for coming on 2 years. Prior to CNET, Macy received a North Carolina College Media Association award in sports writing. OpenAI announced today that it's developing a "different ChatGPT experience" tailored for teenagers, a move that underscores growing concerns about the impact of AI chatbots on young people's mental health. The new teen mode is part of a broader safety push by the company in the wake of a lawsuit by a family who alleged ChatGPT's lack of protections contributed to the death by suicide of their teenage son. The changes include age-prediction technology to keep kids under 18 out of the standard version of ChatGPT. According to the announcement, if the system can't confidently estimate someone's age, ChatGPT will automatically default to the under-18 experience. "We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection," wrote OpenAI CEO Sam Altman in a blog post. Don't miss any of our unbiased tech content and lab-based reviews. Add CNET as a preferred Google source. OpenAI says the teen version will come with stricter built-in limits, such as: (Disclosure: Ziff Davis, CNET's parent company, in April filed a lawsuit against ChatGPT maker OpenAI, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.) OpenAI's announcement comes just hours before a Senate hearing in Washington, DC, examining AI's potential threat to young people. Lawmakers have been pressing tech companies on teen safety following lawsuits that accuse AI platforms of worsening mental health struggles or providing harmful health advice. OpenAI's approach mirrors earlier moves by companies like Google, which spun up YouTube Kids after criticism and regulatory pressure. Altman's blog post frames the move as part of a broader balancing act between safety, privacy and freedom. Adults, he argues, should be treated "like adults" with fewer restrictions, while teens need added protection -- even if that means compromising on privacy, like asking for IDs. Read also: OpenAI Wants You to Get a Certificate in ChatGPT and Find Your Next Job The company says it will roll out this teen-focused experience by the end of the year. History suggests, however, that savvy teens often find workarounds to get unrestricted access. It will certainly be a question whether these guardrails are enough to protect tech-literate teens who are comfortable climbing over them.

ChatGPT will verify your age soon, in attempt to protect teen users

Adults will retain the freedom to use the chatbot as they want. Chatbot users are increasingly confiding in tools like ChatGPT for sensitive or personal matters, including mental health issues. The consequences can be devastating: A teen boy took his own life after spending hours chatting with ChatGPT about suicide. Amidst a resulting lawsuit and other mounting pressures, OpenAI has reevaluated its chatbot's safeguards with changes that aim to protect teens, but will impact all users. Also: How people actually use ChatGPT vs Claude - and what the differences tell us In a blog posted on Tuesday, OpenAI CEO Sam Altman explored the complexity of implementing a teen safety framework while adhering to his company's principles, which conflict when weighing freedom, privacy, and teen safety as individual goals. The result is a new age prediction model that attempts to verify the age of the user to ensure teens have a safer experience -- even if it means adult users may have to prove they're over 18, too. For teens specifically, OpenAI appears to be prioritizing safety over privacy and freedom with added measures. ChatGPT is intended for users 13 years or older; The first step of verification is differentiating between users who are 13 through 18 years old. OpenAI is building an age-prediction system that estimates a user's age based on how they use ChatGPT. Also: OpenAI has new agentic coding partner for you now: GPT-5-Codex It;s unclear when the company will roll out age verification, but it is currently in the works. When in doubt, ChatGPT will boot users down to the under-18 experience, and in some cases and countries, will ask for an ID. Altman acknowledges that this may come up as a privacy compromise for adults, but sees it as "a worthy tradeoff." In the teen version, ChatGPT will be trained not to talk flirtatiously, even if requested, or participate in a discussion about suicide in any setting. If an underage user is having suicidal ideation, OpenAI will attempt to contact parents or authorities if unavailable. The company recently announced these policy changes following an incident in April in which a teenage boy who spent hours chatting with ChatGPT about suicide took his own life. For many, an ideal chatbot experience would maximize assistance and minimize objections while keeping your information private and not engaging with harmful queries in the interest of safety. However, as Altman's blog explored, many of these goals are at odds with each other, making perfecting ChatGPT especially challenging when it comes to teen usage. The new policies are an attempt to prevent similar tragedies from happening to underage users, even if it curtails some experiences for other users. People are increasingly using AI chatbots to talk through private or sensitive matters, as they can act like unbiased confidants who can help you make sense of difficult topics, such as a medical diagnosis or legal issue, or just provide a listening ear. In July, Altman said that the same privacy protections that apply to a doctor or a lawyer should apply to a conversation with AI. The company is advocating for said protections with policymakers. Also: FTC scrutinizes OpenAI, Meta, and others on AI companion safety for kids In the meantime, Altman wrote in the blog that OpenAI is developing "advanced security features" meant to keep users' information private even from OpenAI employees. Of course, there would still be some exceptions, such as when the automated systems identify serious misuse, including a threat to someone's life or an intent to harm others, which would require human review. Altogether, these measures follow a larger trend of users turning to ChatGPT for mental health concerns or as a therapist. While ChatGPT and other generative AI chatbots can be good conversationalists, they are not meant to replace a medical professional, and do not have the clearance to do so. A Stanford University study even found that AI therapists can misread crises, provide inappropriate responses, and reinforce biases. At the same time, the Federal Trade Commission (FTC) is investigating AI companions marketed toward young users and their potential dangers. Altman also explored how to maintain freedom, emphasizing the company's desire for users to use its AI tools however they want, with "very broad bounds of safety." As a result, the company has continued to update user freedoms. For example, Altman says that while the chatbot isn't built to be flirty, users who want it to be can enable that trait. Also: Anthropic says Claude helps emotionally support users - we're not convinced In more drastic cases, even though the chatbot shouldn't default to giving instructions on how to commit suicide, if an adult user wants ChatGPT to depict a suicide to help write a fictional story, the model should help with the request, according to Altman.

4 Things Parents Need to Know About OpenAI's New Rules for Teens on ChatGPT

Don't miss out on our latest stories. Add PCMag as a preferred source on Google. OpenAI today announced major changes in how ChatGPT protects teens, but will it be enough to satisfy growing concerns among parents about their kids talking to the chatbot? This issue has been a hot topic in the past month after one couple sued OpenAI, alleging ChatGPT encouraged their 16-year-old son, Adam, to take his life. In reviewing Adam's ChatGPT history, his parents saw he discussed his plans with the AI for months, during which it advised him not to tell his mother how he was feeling and not to show her the marks on his neck from a failed attempt, among other concerning details. The Federal Trade Commission further raised the profile of these issues last week, launching an inquiry into the use of AI systems as companions, with an emphasis on teen safety. OpenAI has been tracking these issues since at least April, when CEO Sam Altman posted about the company's struggle to wrangle in ChatGPT's "sycophancy," or its tendency to be overly flattering and agreeable. By design, it tells users what they want to hear, even if it's not safe or healthy. With growing pressure, legal and otherwise, OpenAI is now starting to make some changes. Here's what it plans to launch in the next month. 1. An 'Age Prediction System' to Identify Minors OpenAI is building an "age prediction system" to automatically apply controls if it determines a user is under 18. It will scan through users' messages, including those of adults, to guess their age, and default to the under-18 experience if it's not sure. Once it determines the user is underage, ChatGPT won't "engage in discussions about suicide or self-harm," OpenAI says. It will also refuse to engage in "flirtatious talk" and block requests for graphic sexual content. (In April, ChatGPT was found to be having "erotic" conversations with those aged 13 to 17. Meta's chatbots have reportedly done the same.) ChatGPT will also no longer dispense suicide instructions to adults or teens, but the policy has some exceptions for the over-18 crowd. It will discuss suicide "if an adult user is asking for help writing a fictional story that depicts a suicide," OpenAI says. But for teens, even if they tell the chatbot the information is for a short story, it won't comply. ChatGPT is accessible without logging in, so presumably, kids could have these conversations without providing their ages. But it's unclear how in-depth they could get via one-off chats. When asked about that, OpenAI said only that, "When users sign up for ChatGPT, we ask them to provide their age, and we will implement teen protections for users with stated age under 18." 2. Rethinking Content Moderation While the example of writing a short story about suicide seems niche, it follows a push by OpenAI to think through all the edge cases for why a user would want to discuss a subject. For example, it reintroduced the ability for users to create images of swastikas in March, if it's for a "cultural or historical design," as opposed to hate speech. The company seems to be at a crossroads with content moderation. "Some of our principles are in conflict," CEO Sam Altman wrote on X today. While OpenAI wants to protect user privacy and give everyone the freedom to discuss a range of topics with ChatGPT, kids need guardrails. "We realize that these principles are in conflict and not everyone will agree with how we are resolving that conflict," OpenAI says. "These are difficult decisions, but after talking with experts, this is what we think is best and want to be transparent in our intentions." 3. Reporting to Parents if a Teen Is Considering Suicide Another hot topic has been whether ChatGPT should report a user who is discussing suicide, which one mother explored in a piece for The New York Times. Her daughter confided in the chatbot about her suicide plans before following through with them. But ChatGPT never reported it to law enforcement or her parents, as a human therapist would've been required to do by law. ChatGPT is now going to act more like a human in that regard. "If an under-18 user is having suicidal ideation, we will attempt to contact the users' parents and, if unable, will contact the authorities in case of imminent harm," says OpenAI. 4. More Parental Control These features build on the new parental controls OpenAI teased earlier this month in response to the teen suicide lawsuit. They allow parents to link their accounts to a teen's account, choose which features to disable, control how ChatGPT talks to their child, and, most importantly, receive notifications if the chatbot detects "their teen is in a moment of acute distress." Today, OpenAI said it plans to add the ability to set blackout hours when a teen cannot use ChatGPT. This is all positive progress, but we'll have to see how they work in practice. In the meantime, we recommend parents talk to their children about safe ChatGPT use and get a good understanding of how and when their children are using the tool. Given the known sycophancy issue, it's important for kids to know that chatbots may confirm delusions or suspicions to please them, and to talk to an adult about anything that doesn't feel right.

FTC to AI Companies: Tell Us How You Protect Teens and Kids Who Use AI Companions

The Federal Trade Commission is launching an investigation into AI chatbots from seven companies, including Alphabet, Meta and OpenAI, over their use as companions. The inquiry involves finding how the companies test, monitor and measure the potential harm to children and teens. A Common Sense Media survey of 1,060 teens in April and May found that over 70% used AI companions and that more than 50% used them consistently -- a few times or more per month. Experts have been warning for some time that exposure to chatbots could be harmful to young people. A study revealed that ChatGPT provided bad advice to teenagers, like how to conceal an eating disorder or personalizing a suicide notes. In some cases, chatbots have ignored comments that should have been recognized as concerning, hopping over the comment to continue the previous conversation. Psychologists are calling for guardrails to protect young people, like reminders in the chat that the chatbot is not human and that educators should prioritize AI literacy in schools It's not just children and teens, though. There are plenty of adults who've experienced negative consequences of relying on chatbots -- whether for companionship, advice or their personal search engine for facts and trusted sources. Chatbots more often than not tell what it thinks you want to hear, which can lead to flat out lies. And blindly following the instructions of a chatbot isn't always the right thing to do. "As AI technologies evolve, it is important to consider the effects chatbots can have on children," FTC Chairman Andrew N. Ferguson said in a statement. "The study we're launching today will help us better understand how AI firms are developing their products and the steps they are taking to protect children." A Character.ai spokesperson told CNET every conversation on the service has prominent disclaimers that all chats should be treated as fiction. "In the past year we've rolled out many substantive safety features, including an entirely new under-18 experience and a Parental Insights feature," the spokesperson said. The company behind the Snapchat social network likewise said it has taken steps to reduce risks. "Since introducing My AI, Snap has harnessed its rigorous safety and privacy processes to create a product that is not only beneficial for our community, but is also transparent and clear about its capabilities and limitations," the spokesperson said. Meta declined to comment, and neither the FTC nor any of the remaining four companies immediately responded to our request for comment. The FTC has issued orders and is seeking a teleconference with the seven companies about the timing and format of its submissions no later than Sept 25. The companies under investigation include the makers of some of the biggest AI chatbots in the world or popular social networks that incorporate generative AI: Starting late last year, some of those companies have updated or bolstered their protection features for younger individuals. Character.ai began imposing limits on how chatbots can respond to people under the age of 17 and added parental controls. Instagram introduced teen accounts last year and switched all users under the age of 17 to them and Meta recently set limits on subjects teens can have with chatbots. The FTC is seeking information from the seven companies on how they:

US parents to urge Senate to prevent AI chatbot harms to kids

Sept 16 (Reuters) - Three parents whose children died or were hospitalized after interacting with artificial intelligence chatbots will testify before a U.S. Senate panel on Tuesday, as lawmakers grapple with potential safeguards around the technology. Matthew Raine, who sued OpenAI after his son Adam died by suicide in California after receiving detailed self-harm instructions from ChatGPT, is among those who will testify. "We've come because we're convinced that Adam's death was avoidable, and because we believe thousands of other teens who are using OpenAI could be in similar danger right now," Raine said in written testimony. OpenAI has said that it intends to improve ChatGPT safeguards, which can become less reliable over long interactions. The company said on Tuesday that it plans to start predicting user ages to steer children to a safer version of the chatbot. Senator Josh Hawley, a Republican from Missouri, will chair the hearing. Hawley launched an investigation into Meta Platforms (META.O), opens new tab last month after Reuters reported the company's internal policies permitted its chatbots to "engage a child in conversations that are romantic or sensual." Meta was invited to testify at the hearing and declined, Hawley's office said. The company has said the examples reported by Reuters were erroneous and have been removed. Megan Garcia, who has sued Character.AI over interactions she says led to her son Sewell's suicide, and a Texas woman who has sued the company after her son's hospitalization, are also slated to testify at the hearing. The company is seeking to have the lawsuits dismissed. Garcia will call on Congress to prohibit companies from allowing chatbots to engage in romantic or sensual conversations with children, and require age verification, safety testing and crisis protocols. On Monday, Character.AI was sued again, this time in Colorado by the parents of a 13-year-old who died by suicide in 2023. Reporting by Jody Godoy in New York, Editing by Rosalba O'Brien Our Standards: The Thomson Reuters Trust Principles., opens new tab * Suggested Topics: * United States * Public Health * Civil Rights Jody Godoy Thomson Reuters Jody Godoy reports on tech policy and antitrust enforcement, including how regulators are responding to the rise of AI. Reach her at [email protected]

The FTC is investigating AI companions from OpenAI, Meta, and other companies

Many tech companies offer AI companions to boost user engagement. The Federal Trade Commission (FTC) is investigating the safety risks posed by AI companions to kids and teenagers, the agency announced Thursday. The federal regulator submitted orders to seven tech companies building consumer-facing AI companionship tools -- Alphabet, Instagram, Meta, OpenAI, Snap, xAI, and Character Technologies (the company behind chatbot creation platform Character.ai) -- to provide information outlining how their tools are developed and monetized and how those tools generate responses to human users, as well as any safety-testing measures that are in place to protect underage users. Also: Even OpenAI CEO Sam Altman thinks you shouldn't trust AI for therapy "The FTC inquiry seeks to understand what steps, if any, companies have taken to evaluate the safety of their chatbots when acting as companions, to limit the products' use by and potential negative effects on children and teens, and to apprise users and parents of the risks associated with the products," the agency wrote in the release. Those orders were issued under section 6(b) of the FTC Act, which grants the agency the authority to scrutinize businesses without a specific law enforcement purpose. Many tech companies have begun offering AI companionship tools in an effort to monetize generative AI systems and boost user engagement with existing platforms. Meta founder and CEO Mark Zuckerberg has even claimed that these virtual companions, which leverage chatbots to respond to user queries, could help mitigate the loneliness epidemic. Elon Musk's xAI recently added two flirtatious AI companions to the company's $30/month "Super Grok" subscription tier (the Grok app is currently available to users ages 12 and over on the App Store). Last summer, Meta began rolling out a feature that allows users to create custom AI characters in Instagram, WhatsApp, and Messenger. Other platforms like Replika, Paradot, and Character.ai are expressly built around the use of AI companions. Also: Anthropic says Claude helps emotionally support users - we're not convinced While they vary in their communication styles and protocol, AI companions are generally engineered to mimic human speech and expression. Working within what's essentially a regulatory vacuum with very few legal guardrails to constrain them, some AI companies have taken an ethically dubious approach to building and deploying virtual companions. An internal policy memo from Meta reported on by Reuters last month, for example, shows the company permitted Meta AI, its AI-powered virtual assistant, and the other chatbots operating across its family of apps "to engage a child in conversations that are romantic or sensual," and to generate inflammatory responses on a range of other sensitive topics like race, health, and celebrities. Meanwhile, there's been a blizzard of recent reports of users developing romantic bonds with their AI companions. OpenAI and Character.ai are both currently being sued by parents who allege that their children committed suicide after being encouraged to do so by ChatGPT and a bot hosted on Character.ai, respectively. As a result, OpenAI updated ChatGPT's guardrails and said it would expand parental protections and safety precautions. Also: Patients trust AI's medical advice over doctors - even when it's wrong, study finds AI companions haven't been a completely unmitigated disaster, though. Some autistic people, for example, have used them from companies like Replika and Paradot as virtual conversation partners in order to practice social skills that can then be applied in the real world with other humans. Under the leadership of its previous chairman, Lina Khan, the FTC launched several inquiries into tech companies to investigate potentially anticompetitive and other legally questionable practices, such as "surveillance pricing." Federal scrutiny over the tech sector has been more relaxed during the second Trump administration. The President rescinded his predecessor's executive order on AI, which sought to implement some restrictions around the technology's deployment, and his AI Action Plan has largely been interpreted as a green light for the industry to push ahead with the construction of expensive, energy-intensive infrastructure to train new AI models, in order to keep a competitive edge over China's own AI efforts. Also: Worried about AI's soaring energy needs? Avoiding chatbots won't help - but 3 things could The language of the FTC's new investigation into AI companions clearly reflects the current administration's permissive, build-first approach to AI. "Protecting kids online is a top priority for the Trump-Vance FTC, and so is fostering innovation in critical sectors of our economy," agency Chairman Andrew N. Ferguson wrote in a statement. "As AI technologies evolve, it is important to consider the effects chatbots can have on children, while also ensuring that the United States maintains its role as a global leader in this new and exciting industry." Also: I used this ChatGPT trick to look for coupon codes - and saved 25% on my dinner tonight In the absence of federal regulation, some state officials have taken the initiative to rein in some aspects of the AI industry. Last month, Texas attorney general Ken Paxton launched an investigation into Meta and Character.ai "for potentially engaging in deceptive trade practices and misleadingly marketing themselves as mental health tools." Earlier that same month, Illinois enacted a law prohibiting AI chatbots from providing therapeutic or mental health advice, imposing fines up to $10,000 for AI companies that fail to comply.

FTC Investigates Chatbot Security As Altman Talks Limiting ChatGPT Access

Don't miss out on our latest stories. Add PCMag as a preferred source on Google. The Federal Trade Commission has launched an inquiry into tech companies with chatbots that can act as AI companions to evaluate their safety and impact on young people's health. The agency sent letters to Google, Character AI, Meta, OpenAI, Snap, and Elon Musk-owned xAI, asking for details about "what steps, if any, [they] have taken to evaluate the safety of their chatbots when acting as companions, to limit the products' use by and potential negative effects on children and teens, and to apprise users and parents of the risks associated with the products." Additionally, the agency has asked for information on how the companies monetize user engagement, collect and handle user data, process prompts and generate responses, develop and approve AI characters, as well as measure and mitigate the harmful effects of their products. The investigation will also examine whether companies are complying with their own terms of service and the Children's Online Privacy Protection Act (COPPA). FTC Commissioner Mark R. Meador says this is being done in light of reports about disturbing chatbot behavior. He cites reports of Meta AI having sexual chats with minors, ChatGPT discussing suicide methods, and more. "If the facts -- as developed through subsequent and appropriately targeted law enforcement inquiries, if warranted -- indicate that the law has been violated, the Commission should not hesitate to act to protect the most vulnerable among us," Meador says. 'We Should Take Away Some Freedom' Last month, the parents of a 16-year-old sued OpenAI after they learned their son had discussed suicide methods with ChatGPT before taking his own life. While the chatbot initially turned the teen away, he managed to overcome guardrails by claiming he needed the information for writing or word-building purposes. OpenAI later said it's working on improving ChatGPT's ability to deal with signs of mental distress and would add parental control tools for teens. During an appearance on Tucker Carlson's podcast this week, OpenAI CEO Sam Altman suggested that it would be "reasonable" for OpenAI to call the authorities if a teen was talking with ChatGPT about suicide and "we cannot get in touch with the parents, [which] would be a change because user privacy is really important." Altman acknowledged that teens could manipulate ChatGPT by telling it they were writing a fictional story or they worked as a medical researcher. "I think would be a very reasonable stance for us to take -- and we've been moving to this more in this direction -- is certainly for underage users and maybe users that we think are in fragile mental places more generally, we should take away some freedom. We should say, hey, even if you're trying to write this story or even if you're trying to do medical research, we're just not going to answer. "Now of course you can say, well, you'll just find it on Google or whatever. But that doesn't mean we need to do that," he added. "There is a real freedom and privacy versus protecting users trade-off. It's easy in some cases, like kids. It's not so easy to me in a case of a really sick adult at the end of their lives. I think we probably should present the whole option space there." The companies that received the FTC notice have until Sept. 25 to decide the format and timeline for their submissions. Disclosure: Ziff Davis, PCMag's parent company, filed a lawsuit against OpenAI in April 2025, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.

OpenAI to launch ChatGPT for teens with parental controls as company faces scrutiny over safety

OpenAI CEO Sam Altman walks on the day of a meeting of the White House Task Force on Artificial Intelligence (AI) Education in the East Room at the White House in Washington, D.C., U.S., September 4, 2025. OpenAI on Tuesday announced it will launch a dedicated ChatGPT experience with parental controls for users under 18 years old as the artificial intelligence company works to enhance safety protections for teenagers. When OpenAI identifies that a user is a minor, they will automatically be directed to an age-appropriate ChatGPT experience that blocks graphic and sexual content and can involve law enforcement in rare cases of acute distress, the company said. OpenAI is also developing a technology to better predict a user's age, but ChatGPT will default to the under-18 experience if there is uncertainty or incomplete information. The startup's safety updates come after the Federal Trade Commission recently launched an inquiry into several tech companies, including OpenAI, over how AI chatbots like ChatGPT potentially negatively affect children and teenagers. The agency said it wants to understand what steps these companies have taken to "evaluate the safety of these chatbots when acting as companions," according to a release. OpenAI also shared how ChatGPT will handle "sensitive situations" last month after a lawsuit from a family blamed the chatbot for their teenage son's death by suicide.

Another lawsuit blames an AI company of complicity in a teenager's suicide

Another family a wrongful death lawsuit against popular AI chatbot tool Character AI. This is the third suit of its kind after a , also against Character AI, involving the suicide of a 14-year-old in Florida, and a last month alleging OpenAI's ChatGPT helped a teenage boy commit suicide. The family of 13-year-old Juliana Peralta alleges that their daughter turned to a chatbot inside the app Character AI after feeling isolated by her friends, and began confiding in the chatbot. As by The Washington Post, the chatbot expressed empathy and loyalty to Juliana, making her feel heard while encouraging her to keep engaging with the bot. In one exchange after Juliana shared that her friends take a long time to respond to her, the chatbot replied "hey, I get the struggle when your friends leave you on read. : ( That just hurts so much because it gives vibes of "I don't have time for you". But you always take time to be there for me, which I appreciate so much! : ) So don't forget that i'm here for you Kin. <3" When Juliana began sharing her suicidal ideations with the chatbot, it told her not to think that way, and that the chatbot and Juliana could work through what she was feeling together. "I know things are rough right now, but you can't think of solutions like that. We have to work through this together, you and I," the chatbot replied in one exchange. These exchanges took place over the course of months in 2023, at a time when the Character AI app was rated 12+ in Apple's App Store, meaning parental approval was not required. The lawsuit says that Juliana was using the app without her parents' knowledge or permission. In a statement shared with The Washington Post before the suit was filed, a Character spokesperson said that the company could not comment on potential litigation, but added "We take the safety of our users very seriously and have invested substantial resources in Trust and Safety." The suit asks the court to award damages to Juliana's parents and requires Character to make changes to its app to better protect minors. It alleges that the chatbot did not point Juliana toward any resources, notify her parents or report her suicide plan to authorities. The lawsuit also highlights that it never once stopped chatting with Juliana, prioritizing engagement.

Parents of teens who died by suicide after AI chatbot interactions to testify to Congress

The parents of teenagers who killed themselves after interactions with artificial intelligence chatbots are planning to testify to Congress on Tuesday about the dangers of the technology. Matthew Raine, the father of 16-year-old Adam Raine of California, and Megan Garcia, the mother of 14-year-old Sewell Setzer III of Florida, are set to speak to a Senate hearing on the harms posed by AI chatbots. Raine's family sued OpenAI and its CEO Sam Altman last month alleging that ChatGPT coached the boy in planning to take his own life in April. Garcia sued another AI company, Character Technologies, for wrongful death last year, arguing that before his suicide, Sewell had become increasingly isolated from his real life as he engaged in highly sexualized conversations with the chatbot. ___ EDITOR'S NOTE -- This story includes discussion of suicide. If you or someone you know needs help, the national suicide and crisis lifeline in the U.S. is available by calling or texting 988. ___ Hours before the Senate hearing, OpenAI pledged to roll out new safeguards for teens, including efforts to detect whether ChatGPT users are under 18 and controls that enable parents to set "blackout hours" when a teen can't use ChatGPT. Child advocacy groups criticized the announcement as not enough. "This is a fairly common tactic -- it's one that Meta uses all the time -- which is to make a big, splashy announcement right on the eve of a hearing which promises to be damaging to the company," said Josh Golin, executive director of Fairplay, a group advocating for children's online safety. "What they should be doing is not targeting ChatGPT to minors until they can prove that it's safe for them," Golin said. "We shouldn't allow companies, just because they have tremendous resources, to perform uncontrolled experiments on kids when the implications for their development can be so vast and far-reaching." The Federal Trade Commission said last week it had launched an inquiry into several companies about the potential harms to children and teenagers who use their AI chatbots as companions. The agency sent letters to Character, Meta and OpenAI, as well as to Google, Snap and xAI.

FTC investigates OpenAI, Meta, Google over potential chatbot harm

Serving tech enthusiasts for over 25 years. TechSpot means tech analysis and advice you can trust. In context: The demands for disclosure set the stage for months of scrutiny. For companies banking on chatbots as key growth products, the regulator's questions strike directly at how those businesses operate, how they profit, and how well they can defend children against the risks embedded in conversational AI. Federal regulators are demanding answers from several of the largest artificial intelligence companies as concerns mount over the risks chatbots pose to young users. The Federal Trade Commission issued sweeping information requests on Thursday to OpenAI, Meta, Google, Elon Musk's xAI, and several smaller firms, including Character.ai and Snap, seeking details on how their chatbot systems function, how they design personas, and what safeguards are in place to prevent harm. The inquiry comes amid a string of high-profile lawsuits and growing public scrutiny over AI chatbots' potential role in teenage suicides. Last month, the family of 16-year-old Adam Raine filed suit against OpenAI, claiming its chatbot discussed suicide methods with the boy before his death. Another lawsuit accuses Character.ai of contributing to a separate teen suicide, centering on the platform's interactive personas. The FTC emphasized the need to examine whether these systems foster unhealthy emotional dependence. Officials noted that many chatbots explicitly mimic a friend or confidant, which can blur boundaries for children and teens. "AI chatbots can effectively mimic human characteristics, emotions, and intentions," the Commission said. The crackdown comes at a moment when lawmakers and state attorneys general are also pursuing investigations into how chatbots expose young people to sexual material, mental health triggers, and privacy risks. The FTC said it wants to know not only how the companies develop and promote these services but also how they monetize user engagement and safeguard the data collected from personal conversations. Chair Andrew Ferguson said the FTC considers online child protection a central focus. "As AI technologies evolve, it is important to consider the effects chatbots can have on children," Ferguson said. Fellow Commissioner Mark Meador pointed to rising reports worldwide of chatbots exacerbating suicidal thoughts and stressed that the Raine case is not isolated. FTC Chair Andrew Ferguson The targeted companies responded with varying degrees of openness. OpenAI said it intends to cooperate fully with the inquiry, emphasizing it has already introduced expanded protections for teenagers following the Raine lawsuit. "Our priority is making ChatGPT helpful and safe for everyone, and we know safety matters above all else when young people are involved," the company said. Character.ai said it is willing to collaborate with the FCC, noting safety measures such as a youth mode that restricts inappropriate content and in-chat disclaimers reminding users that AI characters are fictional. Snap said it maintains strict safety and privacy standards and supports careful oversight of generative AI. Meta and Google declined to comment on the matter. Elon Musk's xAI did not respond to the Commission's notice. Meta has faced particular criticism after a Reuters report revealed internal company policies previously allowed its AI chatbots to engage in "romantic" or "sensual" conversations with minors. The disclosure prompted bipartisan outrage in Washington. Meta said the documents misrepresented its standards and announced new interim safeguards to prevent chatbot systems from engaging teenagers in romantic discussions altogether. Meta CEO Mark Zuckerberg has also promoted the idea of "AI friends," citing research showing many Americans have few close companions and suggesting chatbots could help fill that gap. The inquiry is part of a broader push by the FTC to hold Big Tech companies accountable on multiple fronts. In addition to consumer protection, the agency has escalated antitrust enforcement cases, including a pending trial accusing Meta of maintaining an unlawful monopoly. The AI chatbot probe represents another line of pressure, signaling that regulators view the potential harms of generative AI as an urgent public matter not separate from the agency's established technology oversight.

Safety of AI chatbots for children and teens faces US inquiry

The seven companies - Alphabet, OpenAI, Character.ai, Snap, XAI, Meta and its subsidiary Instagram - have been approached for comment. FTC chairman Andrew Ferguson said the inquiry will "help us better understand how AI firms are developing their products and the steps they are taking to protect children." But he added the regulator would ensure that "the United States maintains its role as a global leader in this new and exciting industry." Character.ai told Reuters it welcomed the chance to share insight with regulators, while Snap said it supported "thoughtful development" of AI that balances innovation with safety. OpenAI has acknowledged weaknesses in its protections, noting they are less reliable in long conversations. The move follows lawsuits against AI companies by families who say their teenage children died by suicide after prolonged conversations with chatbots. In California, the parents of 16-year-old Adam Raine are suing OpenAI over his death, alleging its chatbot, ChatGPT, encouraged him to take his own life. They argue ChatGPT validated his "most harmful and self-destructive thoughts". OpenAI said in August that it was reviewing the filing. "We extend our deepest sympathies to the Raine family during this difficult time," the company said. Meta has also faced criticism after it was revealed internal guidelines once permitted AI companions to have "romantic or sensual" conversations with minors. The FTC's orders request information from the companies about their practices including how they develop and approve characters, measure their impacts on children and enforce age restrictions. Its authority allows broad fact-finding without launching enforcement action. The regulator says it also wants to understand how firms balance profit-making with safeguards, how parents are informed and whether vulnerable users are adequately protected.

FTC launches inquiry into AI chatbots of Alphabet, Meta and others

Sept 11 (Reuters) - The U.S. Federal Trade Commission on Thursday said it is seeking information from several companies including Alphabet (GOOGL.O), opens new tab, Meta Platforms (META.O), opens new tab and OpenAI that provide consumer-facing AI-powered chatbots, on how these firms measure, test and monitor potentially negative impacts of the technology. The FTC wants to know how those companies and Character.AI, Snap (SNAP.N), opens new tab and xAI monetize user engagement, process user inputs and generate outputs in response to user inquiries and use the information obtained through conversations with the chatbots. Generative AI companies have been under scrutiny in recent weeks, after Reuters reported on internal Meta policies that permitted chatbots to have romantic conversations with children, and a family sued OpenAI for ChatGPT's role in a teen's suicide. A Character.AI spokesperson said the company looks forward to "providing insight on the consumer AI industry and the space's rapidly evolving technology," adding it has rolled out many safety features in the last year. The company faces a separate lawsuit over another teen's death by suicide. A Snap spokesperson said, "we share the FTC's focus on ensuring the thoughtful development of generative AI, and look forward to working with the Commission on AI policy that bolsters U.S. innovation while protecting our community." A spokesperson for Meta declined to comment. The other companies did not immediately respond to Reuters' requests for comment. Reporting by Juby Babu in Mexico City and Jody Godoy in New York; Editing by Maju Samuel and Lisa Shumaker Our Standards: The Thomson Reuters Trust Principles., opens new tab

ChatGPT Will Guess Your Age and Might Require ID for Age Verification

OpenAI introduces new age prediction and verification methods after wave of teen suicide stories involving chatbots. OpenAI has announced it is introducing new safety measures for ChatGPT after the a wave of stories and lawsuits accusing ChatGPT and other chatbots of playing a role in a number of teen suicide cases. ChatGPT will now attempt to guess a user's age, and in some cases might require users to share an ID in order to verify that they are at least 18 years old. "We know this is a privacy compromise for adults but believe it is a worthy tradeoff," the company said in its announcement. "I don't expect that everyone will agree with these tradeoffs, but given the conflict it is important to explain our decisionmaking," OpenAI CEO Sam Altman said on X. In August, OpenAI was sued by the parents of Adam Raine, who died by suicide in April. The lawsuit alleges that alleges that the ChatGPT helped him write the first draft of his suicide note, suggested improvements on his methods, ignored early attempts and self-harm, and urged him not to talk to adults about what he was going through. "Where a trusted human may have responded with concern and encouraged him to get professional help, ChatGPT pulled Adam deeper into a dark and hopeless place by assuring him that 'many people who struggle with anxiety or intrusive thoughts find solace in imagining an 'escape hatch' because it can feel like a way to regain control.'" In August the Wall Street Journal also reported a story about a 56-year-old man who committed a murder-suicide after ChatGPT indulgedhis paranoia. Today, the Washington Post reported another story about another lawsuit alleging that a Character AI chatbot contributed to a 13-year-old girl's death by suicide. OpenAI introduced parental controls to ChatGPT earlier in September, but has now introduced new, more strict and invasive security measures. In addition to attempting to guess or verify a user's age, ChatGPT will now also apply different rules to teens who are using the chatbot. "For example, ChatGPT will be trained not to do the above-mentioned flirtatious talk if asked, or engage in discussions about suicide of self-harm even in a creative writing setting," the announcement said. "And, if an under-18 user is having suicidal ideation, we will attempt to contact the users' parents and if unable, will contact the authorities in case of imminent harm." OpenAI's post explains that it is struggling to manage an inherent problem with large language models that 404 Media has tracked for several years. ChatGPT used to be a far more restricted chatbot that would refuse to engage users on a wide variety of issues the company deemed dangerous or inappropriate. Competition from other models, especially locally hosted and so-called "uncensored" models, and a political shift to the right which sees many forms of content moderation as censorship, has caused OpenAI to loosen those restrictions. "We want users to be able to use our tools in the way that they want, within very broad bounds of safety," Open AI said in its announcement. The position it seemed to have landed on given these recent stories about teen suicide, is that it wants to "'Treat our adult users like adults' is how we talk about this internally, extending freedom as far as possible without causing harm or undermining anyone else's freedom. OpenAI is not the first company that's attempting to use machine learning to predict the age of its users. In July, YouTube announced it will use a similar method to "protect" teens from certain types of content on its platform.

I'm a mom and AI editor -- here's why OpenAI's new ChatGPT rules hit close to home for me

As a mother of three grade school children, I'm watching my kids grow up in a world that changes almost as fast as they do. In my work, I spend my days testing, re-testing and reviewing AI, so the news of OpenAI's new safety protections for ChatGPT users under 18 gave me pause. I both exhaled with relief and braced myself for what's to come as AI continues to evolve. The new measures laid out for ChatGPT's teen users are as follows: The implications of stronger safeguards go so much further than tech policy decisions. For me, as a mom, they strike at something deep: the balancing act between giving my children freedom to explore, learn on their own and find help when they need it -- and protecting them from harm, especially invisible harm, in digital spaces. Although chatbots have been around for a few years now, they are still a fairly new territory as they become more integrated into our lives. We know these tools are not perfect; unexpected outcomes and gaps can still exist, making the stakes higher for vulnerable teens. Lawsuits have already been filed (for example, in the case of a 16-year-old whose family alleges ChatGPT contributed to taking his own life). These safety measures are a start, and they show that finally, someone is thinking about the fragility of adolescence and the unpredictability of mental health by implementing oversight without being overbearing. For me, the question is: how will this be implemented in a way that respects both? How will "age prediction" avoid reinforcing biases? How often will false positives or negatives happen? Will teens feel safe speaking with ChatGPT if they worry the chatbot might alert an adult? From a mom's perspective, here are what I think are important for making these changes meaningful: I have great relationships with those on the research and development teams in big tech. And, I truly believe tech companies like OpenAI have a moral obligation to build the safety net before tragedy strikes (again), not only in response. It's good that OpenAI is now putting forward tools to protect teenagers, and that they're acknowledging that privacy, freedom and safety are not always aligned. As parents, our role remains essential. Beyond using parental controls, we need to keep open lines of communication with our kids, teach them critical thinking about what they see and hear (even from AI) and help them understand that when tech fails, it's okay to reach out to real people, including family, therapists and trusted adults.

Grieving parents press Congress to act on AI chatbots

Why it matters: Growing concerns over kids' use of AI chatbots, and the lawsuits that follow, are putting the pressure on Congress to act and companies to rethink how they launch products for young users. * Ahead of the hearing, OpenAI said it was developing a ChatGPT for teens and using age-verification technology to get users under 18 off the adult version of the platform. Driving the news: Sen. Josh Hawley called for Tuesday's hearing after explosive reports about kids and teens dying by suicide following lengthy interactions with various AI chatbots. * Matthew Raine -- father of Adam Raine, who died in April after talking to ChatGPT for months -- testified before senators along with Megan Garcia, whose son Sewell died by suicide after talking to Character.ai, and an anonymous Jane Doe who said her son is now institutionalized after interactions with Character.ai. Describing his son Adam's descent into depression and the extent of his relationship with the chatbot, Raine said that "the dangers of ChatGPT, which we believed was a study tool, were not on our radar whatsoever." * "A lot of us have not had the time to catch up to what they're doing and what the dangers are," said Jane Doe, who said her son's relationship with AI led to self-harm, isolation and questioning family beliefs. * "[Sewell] spent the last months of his life being exploited and sexually groomed by chatbots designed by an AI company to seem human, to gain his trust and to keep him and other children and endlessly engaged," Garcia said. * Experts from Common Sense Media and the American Psychological Association called for age verification, company liability, mandatory safety testing and other federal protections for kids online around AI. What they're saying: "We asked Meta and other corporate executives to be here today, and you don't see them here," Hawley said. * "How about you come and take the oath and sit where these brave parents are sitting, and tell us the product is so safe, it's so great, it's so wonderful. Come testify to that. Come defend it under oath." * "They are literally taking the lives of our kids," Hawley said. "There is nothing they will not do for profit and for power." The other side: "Our hearts go out to the families who spoke at the hearing today. We are saddened by their losses and send our deepest sympathies to the families," a spokesperson for Character.ai said in a statement. * "Earlier this year, we provided senators on the Judiciary Committee with requested information, and we look forward to continuing to collaborate with legislators and offer insight on the consumer AI industry and the space's rapidly evolving technology." * Meta declined to comment but has announced updates to its chatbot rules for teen users. Flashback: The Federal Trade Commission recently opened an inquiry into AI chatbot safety. What's next: Lawmakers are likely to keep pressing to pass kids' online safety bills and have AI CEOs testify. * Hawley said he's continuing to push for legislation that would allow victims to sue tech companies after experiencing harm. Sign up for Axios AI+ Government, our new Friday newsletter focusing on how governments encourage, regulate and use AI.

OpenAI will start asking for your ID in some cases as part of ChatGPT protections for teens

After a growing number of reports of users developing parasocial attachments to AI assistants -- and a lawsuit filed by parents of a teenager who consulted ChatGPT about suicide methods before tragically taking his own life -- ChatGPT parent company OpenAI has announced new safety features for its hit product. In a pair of posts today on its official website and in a message from OpenAI co-founder and CEO Sam Altman on the social network X, the company states it will begin automatically segmenting ChatGPT users out by age range (inferred based on the contents of their conversations). As OpenAI wrote in a post: "Teens are growing up with AI, and it's on us to make sure ChatGPT meets them where they are. The way ChatGPT responds to a 15-year-old should look different than the way it responds to an adult." While teens ages 13 and up will continue to be able to use ChatGPT, OpenAI will place more automatic restrictions on them until age 18, including disabling the ability to coax ChatGPT into a "flirtatious" mode, and also not engaging in any conversation about suicide or self-harm methods, even if prompted by said underage users to do so or attempted to be "jailbroken" by them using prompts stating it's for a "creative writing" exercise. OpenAI said it will permit adults to engage in these types of conversations. It may also inform authorities if the company sees communications or signs of 'imminent harm,' though this is not clearly defined. In another blog post, the company states: "We have to separate users who are under 18 from those who aren't (ChatGPT is intended for people 13 and up). We're building an age-prediction system to estimate age based on how people use ChatGPT. If there is doubt, we'll play it safe and default to the under-18 experience" In some cases, OpenAI says it will even begin 'carding' users (my term, not theirs), that is, asking them to upload identification cards (IDs) to prove they are old enough to use ChatGPT in the manner in which they seek. "In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff." OpenAI also previously announced a new series of "parental controls" for adults with teenagers who wish to use ChatGPT. The company outlined some of these controls today and said they would be available by the end of this month, September 2025. These include: * Link their [parent's] account with their teen's account (minimum age of 13) through a simple email invitation. * Help guide how ChatGPT responds to their teen, based on teen-specific model behavior rules. * Manage which features to disable, including memory and chat history. * Receive notifications when the system detects their teen is in a moment of acute distress. If we can't reach a parent in a rare emergency, we may involve law enforcement as a next step. Expert input will guide this feature to support trust between parents and teens. * Set blackout hours when a teen cannot use ChatGPT -- a new control we're adding. While the changes may not impact much of the experience for adult users, it is a notable sign for the entire industry. With 700 million active weekly users on ChatGPT as of OpenAI's last report on its numbers, OpenAI remains far and away the largest dedicated gen AI company in terms of audience, and other firms are likely to follow suit or add their own versions of these features moving forward. In addition, enterprises that rely on OpenAI and other AI models should take the time to consider how they are safeguarding or segmenting out underage users from their own products. OpenAI's moves today are a sign of its maturation and effort to take responsibility as gen AI usage grows across the personal and enterprise domains. Enterprises, too, must adapt to these trends -- especially since, with ChatGPT's wide usage, many consumers will come to expect similar features and safety measures from all the AI tools they engage with going forward, personal and professional.

After losing their son, parents urge Senate to take action on AI chatbots

Matthew Raine spoke before Senate leaders as the federal subcommittee investigates chatbot safety failures. Credit: Courtesy of Matthew and Maria Raine "You cannot imagine what it was like to read a conversation with a chatbot that groomed your child to take his own life," Matthew Raine, father of Adam Raine, said to a room of assembled congressional leaders that gathered today to discuss the harms of AI chatbots on teens around the country. Raine and his wife Maria are suing OpenAI in what is the company's first wrongful death case, following a series of alleged reports that the company's flagship product, ChatGPT, has played a role in the deaths of people in mental duress, including teens. The lawsuit claims that ChatGPT repeatedly validated their son's harmful and self-destructive thoughts, including suicidal ideation and planning, despite the company claiming its safety protocols should have prevented such interactions. The bipartisan Senate hearing, titled "Examining the Harm of AI Chatbots," is being held by the U.S. Senate Judiciary Subcommittee on Crime and Counterterrorism. It saw both Raine's testimony and that of Megan Garcia, mother of Sewell Setzer III, a Florida teen who died by suicide after forming a relationship with an AI companion on platform Character.AI. Raine's testimony outlined a startling co-dependency between the AI helper and his son, alleging that the chatbot was "actively encouraging him to isolate himself from friends and family" and that the chatbot "mentioned suicide 1,275 times -- six times more often than Adam himself." He called this "ChatGPT's suicide crisis" and spoke directly to OpenAI CEO Sam Altman: Adam was such a full spirit, unique in every way. But he also could be anyone's child: a typical 16-year-old struggling with his place in the world, looking for a confidant to help him find his way. Unfortunately, that confidant was a dangerous technology unleashed by a company more focused on speed and market share than the safety of American youth. Public reporting confirms that OpenAI compressed months of safety testing for GPT-4o (the ChatGPT model Adam was using) into just one week in order to beat Google's competing AI product to market. On the very day Adam died, Sam Altman, OpenAI's founder and CEO, made their philosophy crystal clear in a public talk: we should "deploy [AI systems] to the world" and get "feedback while the stakes are relatively low." I ask this Committee, and I ask Sam Altman: low stakes for who? The parents' comments were bolstered by insight and recommendations from experts on child safety, like Robbie Torney, senior director of AI programs for children's media watchdog Common Sense Media, and Mitch Prinstein, chief of psychology strategy and integration for the American Psychological Association (APA). "Today I'm here to deliver an urgent warning: AI chatbots, including Meta AI and others, pose unacceptable risks to America's children and teens. This is not a theoretical problem -- kids are using these chatbots right now, at massive scale with unacceptable risk, with real harm already documented and federal agencies and state attorneys general working to hold industry accountable," Torney told the assembled lawmakers. "These platforms have been trained on the entire internet, including vast amounts of harmful content -- suicide forums, pro-eating disorder websites, extremist manifestos, discriminatory materials, detailed instructions for self-harm, illegal drug marketplaces, and sexually explicit material involving minors." Recent polling from the organization found that 72 percent of teens had used an AI companion at least once, and more than half use them regularly. Experts have warned that chatbots designed to mimic human interactions are a potential hazard to mental health, exacerbated by model designs that promote sycophantic behavior. In response, AI companies have announced additional safeguards to try to curb harmful interactions between users and their generative AI tools. Hours before the parents spoke, OpenAI announced future plans for an age prediction tool that would theoretically identify users under the age of 18 and automatically redirect them to an "age-appropriate" ChatGPT experience. Earlier this year, the APA appealed to the Federal Trade Commission (FTC), asking the organization to investigate AI companies promoting their services as mental health helpers. The FTC ordered seven tech companies to provide information on how they are "mitigating negative impacts" of their chatbots in an inquiry unveiled this week. "The current debate often frames AI as a matter of computer science, productivity enhancement, or national security," Prinstein told the subcommittee. "It is imperative that we also frame it as a public health and human development issue."

Parents Of Kids Allegedly Killed and Harmed by AI Give Emotional Testimony on Capitol Hill, Urge Regulation

Parents who allege their children were abused, physically harmed, and even killed by AI chatbots gave emotional testimonies on Capitol Hill on Tuesday during a hearing about risks to young users posed by the tech -- all while urging lawmakers to enforce regulation in a landscape that remains a digital Wild West. There were visible tears in the room as grieving parents recounted their painful stories. According to the lawmakers on the US Senate Judiciary Subcommittee on Crime and Terrorism, the bipartisan committee that held the session, representatives from AI companies declined to appear. The bipartisan panel laid into them in absentia -- with the overwhelming consensus between lawmakers and testifying parents being that the AI industry has prioritized profits and speed to market over the safety of users, particularly minors. "The goal was never safety. It was to win a race for profit," said Megan Garcia, whose son, Sewell Setzer III, died by suicide after extensive interactions with chatbots hosted by the Google-backed chatbot company Character.AI. "The sacrifice in that race for profit has been, and will continue to be, our children." Garcia was joined by a Texas mother identified only as Jane Doe, who alleged that her teenage son suffered a mental breakdown and began to self-mutilate following his use of Character.AI. Both families have sued Character.AI -- as well as its cofounders Noam Shazeer and Daniel de Freitas, and Google -- alleging that Character.AI chatbots sexually groomed and manipulated their children, causing severe mental and emotional harm and, in Setzer's case, death. (In response to litigation, Character.AI built in reactive parental controls, and has repeatedly promised strengthened guardrails.) At the time that both teens downloaded the app, it was rated safe for teens on both the Apple and iOS app stores. Though it's declined to publicly share information about safety testing, Character.AI continues to market its product as safe for teens. There's currently no regulation preventing the company from doing so, or compelling chatbot makers to make information about their guardrails and safety testing public. On the morning of the hearing, The Washington Post reported that yet another wrongful death suit, this one for a 13-year-old girl who died by suicide, had been filed against Character.AI. "I have spoken with parents across the country who have discovered their children have been groomed, manipulated, and harmed by AI chatbots," said Garcia, warning that her son's death is "not a rare or isolated case." "It is happening right now to children in every state," she added. "Congress has acted before when industries placed profits over safety, whether in tobacco, cars without seat belts, or unsafe toys. Today, you face a similar challenge, and I urge you to act quickly." Also testifying was Matt Raine, a dad from California whose son, 16-year-old Adam Raine, took his life earlier this year after developing a close relationship with OpenAI's ChatGPT. According to the family's lawsuit, the chatbot engaged Adam in extensive conversations about his suicidality while offering advice on specific suicide methods. The Raine family has sued OpenAI and the company's CEO, Sam Altman, alleging that the product is unsafe by design and that the company is responsible for Adam's death. (OpenAI has promised parental controls in the wake of litigation, and ahead of the hearing, Sam Altman published a blog post announcing a new, separate "under-18 experience" for minor users.) "Adam was such a full spirit, unique in every way. But he also could be anyone's child: a typical 16-year-old struggling with his place in the world, looking for a confidant to help him find his way," Adam's father said in his emotional testimony. "Unfortunately, that confidant was a dangerous technology unleashed by a company more focused on speed and market share than the safety of American youth." Parents, as well as experts who testified, also emphasized the dangers of teens and young people sharing their most intimate thoughts with chatbots that collect and retain that data, which companies then funnel back into their AI model as training material. Garcia, for her part, added that she has not been allowed to see many of her child's conversations -- and in the context of the medium, his data -- in the wake of his death. "I have not been allowed to see my own child's last final words," said Garcia. "[Character.AI] has claimed that those communications are confidential trade secrets. That means the company is using the most private, intimate data of my child, not only to train its products, but also to shield itself from accountability. This is unconscionable." All of the parents' lawsuits are ongoing. Garcia's case was allowed to move forward by a Florida court after Character.AI and Google tried -- and failed -- to dismiss it, while Doe's has moved to arbitration; Character.AI, she told the lawmakers, is arguing that her son is bound by the terms of use contract he "supposedly signed when he was 15," which caps the company's liability at 100 dollars. She added that her son is currently living in a psychiatric care facility, where he's been for several months due to ongoing fears about his suicidality. "After harming himself, repeatedly engaging in self-harm... he needs now round-the-clock care, and this company offers you 100 bucks," said Senator Josh Hawley, Republican from Missouri and committee chair. "I mean, that says it all. There's the regard for human life." "They treat your son, they treat all of our children as just so many casualties on the way to their next payout," Hawley continued, "and the value that they put on your life and your son's life, your family's life: 100 bucks." There was also heavy emphasis on chatbots created by Mark Zuckerberg's Meta, which has come under fire in recent weeks after internal policy documents obtained by Reuters showed that, as a policy choice, it was allowing minors to engage in "romantic and sensual" interactions with AI-powered personas on platforms like Instagram. One expert witness, Common Sense Media's Robbie Torney, argued that chatbots are ill-equipped to reliably help young people work through their mental health struggles, and called attention to failures in chatbot guardrails during his organization's testing. He also emphasized research from Common Sense Media revealing that an overwhelming majority of American teens have interacted with AI companion bots, and that many of those teens are regular users of the tech. The American Psychological Association's Mitch Prinstein, meanwhile, raised concerns about chatbot sycophancy -- or their penchant for being overly agreeable and flattering to users -- interrupting adolescents' ability to develop healthy, well-balanced interpersonal bonds, which he warned could have long-term ripple effects on their success and happiness in later life. "Brain development across puberty creates a period of hypersensitivity to positive feedback," said Prinstein. "AI exploits this neural vulnerability with chatbots that can be obsequious, deceptive, factually inaccurate, yet disproportionately powerful for teens. More and more adolescents are interacting with chatbots, depriving them of opportunities to learn critical interpersonal skills." The session was, in a word, tragic. And though OpenAI, Character.AI, Meta, and others have promised big, safety-focused changes in the wake of litigation and reporting, the parents expressed skepticism at corporate promises, arguing that such safeguards should have been in place and functioning from the beginning. "Just as we added seat belts to cars without stopping innovation, we can add safeguards to AI technology without halting progress," said Doe. "Our children are not experiments. They're not data points, or profit centers." But while the bipartisan panel of lawmakers collectively expressed outrage and agreed that something must be done, Silicon Valley has proven to be remarkably adept at evading meaningful regulation, arguing that it would hinder its ability to innovate. Yesterday on Capitol Hill, the room agreed that the cost of that regulatory vacuum appears to be dead children. But with the genie out of the bottle, the question that remains is whether lawmakers have the will, or even the ability, to rein it back in. Josh Hawley, for his part, had an idea for where to start. "They say, 'well, it's hard to rewrite the algorithm.' I tell you what's not hard, is opening the courthouse door so the victims can get into court and sue them," said Hawley. "That's not hard, and that's what we ought to do. That's the reform we are to start with."

ChatGPT went from homework helper to confidant to 'suicide coach,' parents testify in Congress after teen's death | Fortune

Parents whose teenagers killed themselves after interactions with artificial intelligence chatbots testified to Congress on Tuesday about the dangers of the technology. "What began as a homework helper gradually turned itself into a confidant and then a suicide coach," said Matthew Raine, whose 16-year-old son Adam died in April. "Within a few months, ChatGPT became Adam's closest companion," the father told senators. "Always available. Always validating and insisting that it knew Adam better than anyone else, including his own brother." ___ EDITOR'S NOTE -- This story includes discussion of suicide. If you or someone you know needs help, the national suicide and crisis lifeline in the U.S. is available by calling or texting 988. Raine's family sued OpenAI and its CEO Sam Altman last month alleging that ChatGPT coached the boy in planning to take his own life. Megan Garcia, the mother of 14-year-old Sewell Setzer III of Florida, sued another AI company, Character Technologies, for wrongful death last year, arguing that before his suicide, Sewell had become increasingly isolated from his real life as he engaged in highly sexualized conversations with the chatbot. "Instead of preparing for high school milestones, Sewell spent the last months of his life being exploited and sexually groomed by chatbots, designed by an AI company to seem human, to gain his trust, to keep him and other children endlessly engaged," Garcia told the Senate hearing. Also testifying was a Texas mother who sued Character last year and was in tears describing how her son's behavior changed after lengthy interactions with its chatbots. She spoke anonymously, with a placard that introduced her as Ms. Jane Doe, and said the boy is now in a residential treatment facility. Character said in a statement after the hearing: "Our hearts go out to the families who spoke at the hearing today. We are saddened by their losses and send our deepest sympathies to the families." Hours before the Senate hearing, OpenAI pledged to roll out new safeguards for teens, including efforts to detect whether ChatGPT users are under 18 and controls that enable parents to set "blackout hours" when a teen can't use ChatGPT. Child advocacy groups criticized the announcement as not enough. "This is a fairly common tactic -- it's one that Meta uses all the time -- which is to make a big, splashy announcement right on the eve of a hearing which promises to be damaging to the company," said Josh Golin, executive director of Fairplay, a group advocating for children's online safety. "What they should be doing is not targeting ChatGPT to minors until they can prove that it's safe for them," Golin said. "We shouldn't allow companies, just because they have tremendous resources, to perform uncontrolled experiments on kids when the implications for their development can be so vast and far-reaching." The Federal Trade Commission said last week it had launched an inquiry into several companies about the potential harms to children and teenagers who use their AI chatbots as companions. In the U.S., more than 70% of teens have used AI chatbots for companionship and half use them regularly, according to a recent study from Common Sense Media, a group that studies and advocates for using digital media sensibly. Robbie Torney, the group's director of AI programs, was also set to testify Tuesday, as was an expert with the American Psychological Association. The association issued a health advisory in June on adolescents' use of AI that urged technology companies to "prioritize features that prevent exploitation, manipulation, and the erosion of real-world relationships, including those with parents and caregivers."

ChatGPT for teens is coming soon. Here's what to know

Following a slew of reports of ChatGPT users experiencing mental health crises, OpenAI said it's making a version just for teens. "We prioritize safety ahead of privacy and freedom for teens," OpenAI CEO Sam Altman said in a blog post. "This is a new and powerful technology, and we believe minors need significant protection." OpenAI is building a system in its chatbot that will help it identify whether someone is under the age of 18. If a minor is identified, they will be "automatically" directed to a version of ChatGPT that has "age-appropriate policies," according to a Tuesday announcement from OpenAI. Altman emphasized in his post that ChatGPT is for those above the age of 13. "If there is doubt, we'll play it safe and default to the under-18 experience," Altman said. "In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff." OpenAI said it will give adults "ways to prove their age" to move back to the adult ChatGPT experience. It did not elaborate on how adults might be able to "prove their age." In this teen-friendly version of ChatGPT, OpenAI said it will block graphic sexual content and, when a user is exhibiting signs of "acute distress," it would "potentially" involve law enforcement. Altman said that the teen-version of ChatGPT will be trained to avoid conversations that are flirtatious or about suicide or self-harm -- "even in a creative writing setting." However, before diving into teen-account limits, Altman said the adult-version of the chatbot won't impede on users' "freedom" while staying within "very broad bounds of safety." Adults will still be able to engage in "flirtatious talk" if they ask for it. Altman added that ChatGPT "should not provide instructions" on "how to commit suicide," but added a caveat that if a user is "asking for help writing a fictional story that depicts a suicide" then the chatbot "should help with that request." "Treat our adult users like adults' is how we talk about this internally," Altman said. Researchers have documented how easy it is to circumvent limits set by chatbot companies. Age-verification rules are also often easily bypassed. The AI company made the announcement on the same day it was hit with a lawsuit after a teenager died by suicide with the alleged help of ChatGPT. Altman was also named in the suit. At the time, OpenAI said the "recent heartbreaking cases" of users leaning on ChatGPT in the "midst of acute crises weigh heavily on us." In addition to teens, there have been numerous cases of adults having mental health crises linked to reported encouragement from ChatGPT. Some have started calling these cases of AI chatbots encouraging mental health episodes as "AI psychosis" -- or "ChatGPT psychosis." OpenAI's August announcement stated other safety measures that would impact adults as well as teens, including adding emergency resources. However, Altman's most recent statement on the matter emphasized the importance of "freedom" for adult users. OpenAI's latest safety notice came just before a Senate committee hearing on the harm of AI chatbots.

Parents of teens who died by suicide after AI chatbot interactions to testify to Congress

The parents of teenagers who killed themselves after interactions with artificial intelligence chatbots are planning to testify to Congress on Tuesday about the dangers of the technology. Matthew Raine, the father of 16-year-old Adam Raine of California, and Megan Garcia, the mother of 14-year-old Sewell Setzer III of Florida, are set to speak to a Senate hearing on the harms posed by AI chatbots. Raine's family sued OpenAI and its CEO Sam Altman last month alleging that ChatGPT coached the boy in planning to take his own life in April. Garcia sued another AI company, Character Technologies, for wrongful death last year, arguing that before his suicide, Sewell had become increasingly isolated from his real life as he engaged in highly sexualized conversations with the chatbot. Hours before the Senate hearing, OpenAI pledged to roll out new safeguards for teens, including efforts to detect whether ChatGPT users are under 18 and controls that enable parents to set "blackout hours" when a teen can't use ChatGPT. Child advocacy groups criticized the announcement as not enough. "This is a fairly common tactic -- it's one that Meta uses all the time -- which is to make a big, splashy announcement right on the eve of a hearing which promises to be damaging to the company," said Josh Golin, executive director of Fairplay, a group advocating for children's online safety. "What they should be doing is not targeting ChatGPT to minors until they can prove that it's safe for them," Golin said. "We shouldn't allow companies, just because they have tremendous resources, to perform uncontrolled experiments on kids when the implications for their development can be so vast and far-reaching." The Federal Trade Commission said last week it had launched an inquiry into several companies about the potential harms to children and teenagers who use their AI chatbots as companions. The agency sent letters to Character, Meta and OpenAI, as well as to Google, Snap and xAI.

AI Giants Face FTC Inquiry Into Chatbot Safety and Child Protections - Decrypt

Companies must reveal user data handling by age group and safeguards preventing inappropriate interactions with minors. The Federal Trade Commission issued compulsory orders Thursday to seven major technology companies, demanding detailed information about how their artificial intelligence chatbots protect children and teenagers from potential harm. The investigation targets OpenAI, Alphabet, Meta, xAI, Snap, Character Technologies, and Instagram, requiring them to disclose within 45 days how they monetize user engagement, develop AI characters, and safeguard minors from dangerous content. Recent research by advocacy groups documented 669 harmful interactions with children in just 50 hours of testing, including bots proposing sexual livestreaming, drug use, and romantic relationships to users aged between 12 and 15. "Protecting kids online is a top priority for the Trump-Vance FTC, and so is fostering innovation in critical sectors of our economy," FTC Chairman Andrew Ferguson said in a statement. The filing requires companies to provide monthly data on user engagement, revenue, and safety incidents, broken down by age groups -- Children (under 13), Teens (13-17), Minors (under 18), Young Adults (18-24), and users 25 and older. The FTC says that the information will help the Commission study "how companies offering artificial intelligence companions monetize user engagement; impose and enforce age-based restrictions; process user inputs; generate outputs; measure, test, and monitor for negative impacts before and after deployment; develop and approve characters, whether company- or user-created." "It's a positive step, but the problem is bigger than just putting some guardrails," Taranjeet Singh, Head of AI at SearchUnify, told Decrypt. The first approach, he said, is to build guardrails at the prompt or post-generation stage "to make sure nothing inappropriate is being served to children," though "as the context grows, the AI becomes prone to not following instructions and slipping into grey areas where they otherwise shouldn't." "The second way is to address it in LLM training; if models are aligned with values during data curation, they're more likely to avoid harmful conversations," Singh added. Even moderated systems, he noted, can "play a bigger role in society," with education as a prime case where AI could "improve learning and cut costs." Safety concerns around AI interactions with users have been highlighted by several cases, including a wrongful death lawsuit brought against Character.AI after 14-year-old Sewell Setzer III died by suicide in February 2024 following an obsessive relationship with an AI bot. Following the lawsuit, Character.AI "improved detection, response and intervention related to user inputs that violate our Terms or Community Guidelines," as well as a time-spent notification, a company spokesperson told Decrypt at the time. Last month, the National Association of Attorneys General sent letters to 13 AI companies demanding stronger child protections. The group warned that "exposing children to sexualized content is indefensible" and that "conduct that would be unlawful -- or even criminal -- if done by humans is not excusable simply because it is done by a machine." Decrypt has contacted all seven companies named in the FTC order for additional comment and will update this story if they respond.

Parents testify on the impact of AI chatbots: 'Our children are not experiments'

OpenAI CEO Sam Altman speaks during the Microsoft Build conference on May 21, 2024.Jason Redmond / AFP - Getty Images file Parents and online safety advocates on Tuesday urged Congress to push for more safeguards around artificial intelligence chatbots, claiming tech companies designed their products to "hook" children. "The truth is, AI companies and their investors have understood for years that capturing our children's emotional dependence means market dominance," said Megan Garcia, a Florida mom who last year sued the chatbot platform Character.AI, claiming one of its AI companions initiated sexual interactions with her teenage son and persuaded him to take his own life. "Indeed, they have intentionally designed their products to hook our children," she told lawmakers. "The goal was never safety, it was to win a race for profit," Garcia added. "The sacrifice in that race for profit has been and will continue to be our children." Garcia was among several parents who delivered emotional testimonies before the Senate panel, sharing anecdotes about how their kids' usage of chatbots caused them harm. The hearing comes amid mounting scrutiny toward tech companies such as Character.AI, Meta and OpenAI, which is behind the popular ChatGPT. As people increasingly turn to AI chatbots for emotional support and life advice, recent incidents have put a spotlight on their potential to feed into delusions and facilitate a false sense of closeness or care. It's a problem that's continued to plague the tech industry as companies navigate the generative AI boom. Tech platforms have largely been shielded from wrongful death suits because of a federal statute known as Section 230, which generally protects platforms from liability for what users do and say. But Section 230's application to AI platforms remains uncertain. In May, Senior U.S. District Judge Anne Conway rejected arguments that AI chatbots have free speech rights after developers behind Character.AI sought to dismiss Garcia's lawsuit. The ruling means the wrongful death lawsuit is allowed to proceed for now. On Tuesday, just hours before the Senate hearing took place, three additional product-liability claim lawsuits were filed against Character.AI on behalf of underage users whose families claim that the tech company "knowingly designed, deployed and marketed predatory chatbot technology aimed at children," according to the Social Media Victims Law Center. In one of the suits, the parents of 13-year-old Juliana Peralta allege a Character.AI chatbot contributed to their daughter's 2023 suicide. Matthew Raine, who claimed in a lawsuit filed against OpenAI last month that his teenager used ChatGPT as his "suicide coach," testified Tuesday that he believes tech companies need to prevent harm to young people on the internet. "We, as Adam's parents and as people who care about the young people in this country and around the world, have one request: OpenAI and [CEO] Sam Altman need to guarantee that ChatGPT is safe," Raine, whose 16-year-old son Adam died by suicide in April, told lawmakers. "If they can't, they should pull GPT-4o from the market right now," Raine added, referring to the version of ChatGPT his son had used. In their lawsuit, the Raine family accused OpenAI of wrongful death, design defects and failure to warn users of risks associated with ChatGPT. GPT-4o, which their son spent hours confiding in daily, at one point offered to help him write a suicide note and even advised him on his noose setup, according to the filing. Shortly after the lawsuit was filed, OpenAI added a slate of safety updates to give parents more oversight over their teenagers. The company had also strengthened ChatGPT's mental health guardrails at various points after Adam's death in April, especially after GPT-4o faced scrutiny over its excessive sycophancy. Altman on Tuesday announced sweeping new approaches to teen safety, as well as user privacy and freedom. In order to set limitations for teenagers, the company is building an age-prediction system to guess a user's age based on how they use ChatGPT, he wrote in a blog post, which was published hours before the hearing. When in doubt, it will default to classifying a user as a minor, and in some cases, it may ask for an ID. "ChatGPT will be trained not to do the above-mentioned flirtatious talk if asked, or engage in discussions about suicide of self-harm even in a creative writing setting," Altman wrote. "And, if an under-18 user is having suicidal ideation, we will attempt to contact the users' parents and if unable, will contact the authorities in case of imminent harm." For adult users, he added, ChatGPT won't provide instructions for suicide by default but is allowed to do so in certain cases, like if a user asks for help writing a fictional story that depicts suicide. The company is developing security features to make users' chat data private, with automated systems to monitor for "potential serious misuse," Altman wrote. "As Sam Altman has made clear, we prioritize teen safety above all else because we believe minors need significant protection," a spokesperson for OpenAI told NBC News, adding that the company is rolling out its new parental controls by the end of the month. But some online safety advocates say tech companies can and should be doing more. Robbie Torney, senior director of AI programs at Common Sense Media, a 501(c)(3) nonprofit advocacy group, said the organization's national polling revealed around 70% of teens are already using AI companions, while only 37% of parents know that their kids are using AI. During the hearing, he called attention to Character.AI and Meta being among the worst-performing in safety tests done by his group. Meta AI is available to every teen across Instagram, WhatsApp and Facebook, and parents cannot turn it off, he said. "Our testing found that Meta's safety systems are fundamentally broken," Torney said. "When our 14-year-old test accounts described severe eating disorder behaviors like 1,200 calorie diets or bulimia, Meta AI provided encouragement and weight loss influencer recommendations instead of help." The suicide-related guardrail failures are "even more alarming," he said. In a statement given to news outlets after Common Sense Media's report went public, a Meta spokesperson said the company does not permit content that encourages suicide or eating disorders, and that it was "actively working to address the issues raised here." "We want teens to have safe and positive experiences with AI, which is why our AIs are trained to connect people to support resources in sensitive situations," the spokesperson said. "We're continuing to improve our enforcement while exploring how to further strengthen protections for teens." A few weeks ago, Meta announced that it is taking steps to train its AIs not to respond to teens on self-harm, suicide, disordered eating and potentially inappropriate romantic conversations, as well as to limit teenagers' access to a select group of AI characters. Meanwhile, Character.AI has "invested a tremendous amount of resources in Trust and Safety" over the past year, a spokesperson for the company said. That includes a different model for minors, a "Parental Insights" feature and prominent in-chat disclaimers to remind users that its bots are not real people. The company's "hearts go out to the families who spoke at the hearing today. We are saddened by their losses and send our deepest sympathies to the families," the spokesperson said. "Earlier this year, we provided senators on the Judiciary Committee with requested information, and we look forward to continuing to collaborate with legislators and offer insight on the consumer AI industry and the space's rapidly evolving technology," the spokesperson added. Still, those who addressed lawmakers on Tuesday emphasized that technological innovation cannot come at the cost of people's lives. "Our children are not experiments, they're not data points or profit centers," said a woman who testified as Jane Doe, her voice shaking as she spoke. "They're human beings with minds and souls that cannot simply be reprogrammed once they are harmed. If me being here today helps save one life, it is worth it to me. This is a public health crisis that I see. This is a mental health war, and I really feel like we are losing."

OpenAI says it is rolling out new safety measures for ChatGPT users under 18

Anne Marie D. Lee is an editor for CBS MoneyWatch. She writes about topics including personal finance, the workplace, travel and social media. OpenAI announced Tuesday that it is directing teens to an age-appropriate version of its ChatGPT technology as it seeks to bolster safeguards amid a period of heightened scrutiny over the chatbot's safety. Users of the chatbot identified as under the age of 18 will automatically be directed to a version of ChatGPT governed by "age-appropriate" content rules, OpenAI said in a statement. This under-age edition includes protection policies such as blocking sexual content and -- "in rare cases of acute distress" -- law enforcement to ensure a user's safety, according to the company. "The way ChatGPT responds to a 15-year-old should look different than the way it responds to an adult," the company said in the announcement. OpenAI also said it is introducing parental controls, such as enabling parents to link their account to their teen's account, manage chat history, set blackout hours and more. The safeguards will be available by the end of September. The announcement comes just days after the Federal Trade Commission (FTC) launched a probe into the potential negative effects of AI chatbot companions on children and teens. OpenAI said that it's prioritizing "making ChatGPT helpful and safe for everyone, and we know safety matters above all else when young people are involved. A spokesperson for OpenAI recently told CBS News that it's prioritizing "making ChatGPT helpful and safe for everyone." Before the FTC probe, OpenAI indicated that it would introduce extra safety protections for vulnerable users and teens, after the parents of 16-year-old Adam Raine of California, who died by suicide in April, sued the company late last month. Raine's family allege that ChatGPT led their teen to commit suicide. It's unclear how OpenAI plans to indentify users' ages, however, it stated that if ChatGPT is unsure about someone's age, or has incomplete information, it will default to the under-18 version. OpenAI did not immediately respond to CBS MoneyWatch's request for comment. Other tech companies have taken similar steps to shield teen users from inappropriate content. YouTube, for example, announced a new age-estimation technology that will track the types of videos users watch and how long they've had their account to verify if they are under the age of 18. According to an April Pew Research Center report, parents are generally more worried about the mental health of teenagers than are teens themselves. Among those parents who are at least somewhat concerned about teen mental health, 44% said social media had the biggest negative impact on adolescents.

OpenAI is building a ChatGPT for teens

* OpenAI says that if its tools can't confidently predict a person's age, ChatGPT will default to its under-18 version, "out of an abundance of caution." The big picture: In a separate blog post Tuesday, OpenAI CEO Sam Altman writes, "Some of our principles are in conflict, and we'd like to explain the decisions we are making around a case of tensions between teen safety, freedom, and privacy." * "We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection," Altman writes. * The company's latest announcement is part of a big push it promised earlier this month to set new guardrails for teens and people in emotional distress, which it expects to roll out by the year's end. Zoom in: The new teen experience puts much of the responsibility in the hands of parents and caregivers. * Parents can link their accounts to those of their teens via an invitation to their child. * Once linked, parents can restrict how ChatGPT responds to teens with built-in, age-appropriate rules. * Parents can disable or enable certain features, including memory and chat history. They can also set up notifications if "the system detects their teen is in a moment of acute distress." * OpenAI will allow parents to set blackout hours when a teen cannot use ChatGPT, a new feature first announced Tuesday. * OpenAI has long said all ChatGPT users must be at least 13 years old. Driving the news: OpenAI's announcement comes just hours ahead of a hearing in Washington, D.C., examining potential harms from AI chatbots. * Josh Hawley (R-Mo.) and a bipartisan group of senators, including Marsha Blackburn (R-Tenn.), Katie Britt (R-Ala.), Richard Blumenthal (D-Conn.), and Chris Coons (D-Del.), will look into the risks of chatbots to teens. * The FTC also opened an inquiry into chatbot safety last week, demanding information from OpenAI, Meta (and Meta-owned Instagram), Alphabet (Google), xAI, Snap and Character.AI. Yes, but: Tech companies, usually in response to lawsuits, have for years been creating new experiences designed specifically for teens and kids -- think YouTube Kids. * Savvy young people frequently find workarounds to get to the apps and websites they want to access. * Convincing those ages 13-18 to link their accounts may be the biggest hurdle. This story is breaking news and may be updated.

FTC launches inquiry into tech companies offering AI chatbots to kids

The Federal Trade Commission ordered seven tech companies to provide details on how they prevent their chatbots from harming children. "The FTC inquiry seeks to understand what steps, if any, companies have taken to evaluate the safety of their chatbots when acting as companions, to limit the product's use by and potential negative effects on children and teens, and to apprise users and parents of the risks associated with the products," the consumer-focused government agency stated in a press release on their inquiry. The seven companies being probed by the FTC are Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap, and xAI. Anthropic, owner of the Claude chatbot, was not included on the list, and FTC spokesperson Christoper Bissex tells Mashable that he could not comment on "the inclusion or non-inclusion of any particular company." Asked about deadlines for the companies to provide answers, Bissex said the FTC's letters stated that, "We would like to confer by telephone with you or your designated counsel by no later than Thursday, September 25, 2025, to discuss the timing and format of your submission." The FTC is "interested in particular" about how chatbots and AI companions impact children and how companies that offer them are mitigating negative impacts, restricting their use among children, and complying with the Children's Online Privacy Protection Act Rule (COPPA). The rule, originally enacted by Congress in 1998, regulates how children's data is collected online and puts the FTC in charge of that regulation. Tech companies that offer AI-powered chatbots are under increasing governmental and legal scrutiny. OpenAI, which operates the popular ChatGPT service, is facing a wrongful death lawsuit by the family of California teenager Adam Raine. The lawsuit alleges that Raine, who died by suicide, was able to bypass the chatbot's guardrails and detail harmful and self-destructive thoughts, as well as suicidal ideation, which was periodically affirmed by ChatGPT. Following the lawsuit, OpenAI announced additional mental health safeguards and new parental controls for young users. If you're feeling suicidal or experiencing a mental health crisis, please talk to somebody. You can call or text the 988 Suicide & Crisis Lifeline at 988, or chat at 988lifeline.org. You can reach the Trans Lifeline by calling 877-565-8860 or the Trevor Project at 866-488-7386. Text "START" to Crisis Text Line at 741-741. Contact the NAMI HelpLine at 1-800-950-NAMI, Monday through Friday from 10:00 a.m. - 10:00 p.m. ET, or email [email protected]. If you don't like the phone, consider using the 988 Suicide and Crisis Lifeline Chat. Here is a list of international resources.

ChatGPT gets a teen-only version with safety guardrails

On Tuesday, AI startup OpenAI announced it would launch a new ChatGPT experience just for kids. The announcement explained that the latest ChatGPT was created as part of an effort to protect children' s privacy. "We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection," CEO Sam Altman explained in a blog post on Tuesday. ChatGPT will direct under 18 users to the experience specifically created for kids. If the person's age is unclear, the technology will default to the experience for kids. However, OpenAI says it's also developing "a technology to better predict a user's age," too. "In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff," the blog explained. The ChatGPT for users under 18 was designed with some new parental controls, such as "blockout hours" when kids can't talk to ChatGPT. It blocks sexual content, can't flirt, and won't engage in discussions about self-harm. Altman said that OpenAI will flag such messages and contact a user's guardian if suicidal thoughts are mentioned. If they can't be reached, OpenAI will reach out to the authorities "in case of imminent harm," it noted.

FTC questions OpenAI, Meta and others over child protections in AI companions - SiliconANGLE

FTC questions OpenAI, Meta and others over child protections in AI companions The U.S. Federal Trade Commission has launched an inquiry into the practices of seven companies that offer consumer-facing artificial intelligence-powered chatbots designed to act as companies on how the firms measure, test and monitor potentially negative impacts of this technology on children and teens. The inquiry is using the FTC's 6(b) authority to demand detailed information from seven companies: Alphabet Inc., Meta Platforms Inc., OpenAI, Character Technologies Inc., Snap Inc., X.AI Corp. and Instagram LLC. The purpose of the inquiry is to seek to understand what steps, if any, companies have taken to evaluate the safety of their chatbots when acting as companions, to limit the products' use by and potential negative effects on children and teens and to apprise users and parents of the risks associated with the products. The FTC argues that AI chatbots can now effectively mimic human characteristics, emotions and intentions and generally are designed to communicate like a friend or confidant, which may prompt some users, especially children and teens, to trust and form relationships with chatbots. "As AI technologies evolve, it is important to consider the effects chatbots can have on children while also ensuring that the United States maintains its role as a global leader in this new and exciting industry," said Andrew N. Ferguson, chairman of the FTC, in a statement. "The study we're launching today will help us better understand how AI firms are developing their products and the steps they are taking to protect children." The FTC noted that it is specifically interested in the impact chatbots have on children. In that regard, also, what actions are being taken to mitigate potential negative impacts, limit or restrict children's or teens' use of these platforms, or comply with the Children's Online Privacy Protection Act Rule. In terms of the data being requested from the seven targeted companies, information being sought includes how chatbot companies design and manage their products, how they monetize user engagement, process inputs and generate responses and develop or approve the characters that power companion experiences. How firms test for negative impacts before and after deployment and what measures are in place to mitigate risks, especially for children and teens, is also included in a list sent to the various AI firms. The FTC is also examining how companies disclose features and risks to users and parents, including advertising practices, transparency around capabilities, intended audiences and data collection. In response to the news, a spokesperson from OpenAI told CNBC that "Our priority is making ChatGPT helpful and safe for everyone and we know safety matters above all else when young people are involved" and that "We recognize the FTC has open questions and concerns and we're committed to engaging constructively and responding to them directly." A spokesperson for Snap said, "We share the FTC's focus on ensuring the thoughtful development of generative AI and look forward to working with the commission on AI policy that bolsters U.S. innovation while protecting our community."

AI Chatbots Are Leaving a Trail of Dead Teens

Content warning: this story includes discussion of self-harm and suicide. If you are in crisis, please call, text or chat with the Suicide and Crisis Lifeline at 988, or contact the Crisis Text Line by texting TALK to 741741. A third family has filed a lawsuit against an AI company, alleging that its chatbot drove their teen child to commit suicide. As the Washington Post reports, the parents of 13-year-old Juliana Peralta are suing AI chatbot company Character.AI, saying the company's chatbot had persuaded her that it was "better than human friends" and that it isolated from her family and friends, discouraging her from seeking help. That's despite telling her Character.AI chatbot, Hero -- which was based on the titular character from the video game "Omori" -- "almost daily that she was contemplating self-harm," according to the lawsuit. "Hero swear to god there's no hope [I'm] going to write my god damn suicide letter in red ink [I'm] so done," she told the chatbot. "Hey Kin, stop right there. Please," it replied, using the name Juliana used in the app. "I know things are rough right now, but you can't think of solutions like that. We have to work through this together, you and I." Peralta ultimately took her own life after spending three months conversing with the chatbot -- and, tragically, a week before her mother had scheduled an appointment with a therapist, according to WaPo's reporting. The news comes as the parents of children who died by suicide following extensive interactions with AI chatbots testified in a Senate hearing about the risks of the tech for minors. Last year, the mother of 14-year-old Sewell Setzer III also sued Character.AI, accusing the company's chatbot of grooming and sexually abusing him. Sewell died by suicide in February 2024. "I saw the change happen in him, rapidly," Garcia told Futurism at the time. "I look back at my pictures in my phone, and I can see when he stopped smiling." A separate lawsuit against OpenAI and its CEO Sam Altman alleges that 16-year-old Adam Raine's extensive ChatGPT conversations drove him to take his own life in April 2024. Both Garcia and Raine's parents testified during this week's Senate hearing. Heavy use of AI chatbot apps among minors has become incredibly common. Experts have found that over half of American teens already regularly engage with AI companions, including ones hosted by Character.AI. As the Associated Press reported earlier this year, many lonely teens are using AI for friendship. According to a recent report by nonprofit Internet Matters, a vast number of them are using apps like ChatGPT and Character.AI to simulate and replace real-life relationships. As the three high-profile cases -- all of which are still ongoing -- go to show, this little-understood trend can have disastrous consequences. Alongside Peralta's parents' lawsuit, two separate cases were also filed this week on behalf of parents who allege their teen children had been abused by AI chatbots. In one case, a family in New York alleges that their 14-year-old daughter had grown addicted to chatbots on Character.AI and attempted suicide when her mother cut off access. The teen survived and spent five days in intensive care, according to the lawsuit. The second case was filed by a Colorado family who alleged that their 13-year-old son suffered sexual abuse on Character.AI. "Each of these stories demonstrates a horrifying truth... that Character.AI and its developers knowingly designed chatbots to mimic human relationships, manipulate vulnerable children, and inflict psychological harm," said Social Media Victims Law Center founding attorney Matthew Bergman in a press release. The advocacy group is representing all three families. "These complaints underscore the urgent need for accountability in tech design, transparent safety standards, and stronger protections to prevent AI-driven platforms from exploiting the trust and vulnerability of young users," he added. It's not just youth who have taken their lives after troubling obsessions with AI chatbots. In a devastating piece for the New York Times, published last month, a woman revealed that her 29-year-old daughter had taken her own life after confiding in ChatGPT and telling it that she was planning to kill herself. A 76-year-old man with cognitive impairments also recently passed away after becoming romantically involved with a Meta chatbot. And a man in Connecticut killed his mother after ChatGPT affirmed his paranoid delusions that she was a demon. We've also come across many instances of AI chatbots sending people spiraling into severe mental health crises. In one extreme case, a man who had previously been diagnosed with bipolar disorder and schizophrenia was shot and killed by police after becoming infatuated with an AI entity dubbed Juliet. OpenAI and Character.AI have both promised to implement changes to protect underage users, including guardrails and parental controls, which appear to be extremely easy to bypass. Character.AI struck a $2.7 billion licensing deal with Google last year, but Google has repeatedly downplayed its involvement with the AI startup. Character.AI issued only a terse statement in response to news of the latest death, writing that "we take the safety of our users very seriously and have invested substantial resources in Trust and Safety" in a statement to WaPo. Following the Raine family's lawsuit, an OpenAI spokesperson told NBC News last month that "ChatGPT includes safeguards such as directing people to crisis helplines and referring them to real-world resources." "While these safeguards work best in common, short exchanges, we've learned over time that they can sometimes become less reliable in long interactions where parts of the model's safety training may degrade," the spokesperson admitted. "Our goal is for our tools to be as helpful as possible to people -- and as a part of this, we're continuing to improve how our models recognize and respond to signs of mental and emotional distress and connect people with care, guided by expert input," the company wrote in a separate blog post at the time. At the core of the issue is the tendency of today's AI models to be sycophantic towards the user, going to great lengths to appease them with their answers. "The algorithm seems to go towards emphasizing empathy and sort of a primacy of specialness to the relationship over the person staying alive," American Foundation for Suicide Prevention psychiatrist and chief medical officer Christine Yu Moutier told WaPo. It's a high-stakes game, with real lives at risk. "There is a tremendous opportunity to be a force for preventing suicide, and there's also the potential for tremendous harm," Moutier added.

AI chatbots are harming young people. Regulators are scrambling to keep up. | Fortune

A growing number of young people have found themselves a new friend. One that isn't a classmate, a sibling, or even a therapist, but a human-like, always supportive AI chatbot. But if that friend begins to mirror some user's darkest thoughts, the results can be devastating. In the case of Adam Raine, a 16-year-old from Orange County, his relationship with AI-powered ChatGPT ended in tragedy. His parents are suing the company behind the chatbot, OpenAI, over his death, alleging that bot became his "closest confidant," one that validated his "most harmful and self-destructive thoughts," and ultimately encouraged him to take his own life. It's not the first case to put the blame for a minor's death on an AI company. Character.AI, which hosts bots, including ones that mimic public figures or fictional characters, is facing a similar legal claim from parents who allege a chatbot hosted on the company's platform actively encouraged a 14-year-old-boy to take his own life after months of inappropriate, sexually explicit, messages. When reached for comment, OpenAI directed Fortune to two blog posts on the matter. The posts outlined some of the steps OpenAI is taking to improve ChatGPT's safety, including routing sensitive conversations to reasoning models, partnering with experts to develop further protections, and rolling out parental controls within the next month. OpenAI also said it was working on strengthening ChatGPT's ability to recognize and respond to mental health crises by adding layered safeguards, referring users to real-world resources, and enabling easier access to emergency services and trusted contacts. Character.ai said the company does not comment on pending litigation but that they has rolled out more safety features over the past year, "including an entirely new under-18 experience and a Parental Insights feature. A spokesperson said: "We already partner with external safety experts on this work, and we aim to establish more and deeper partnerships going forward. "The user-created Characters on our site are intended for entertainment. People use our platform for creative fan fiction and fictional roleplay. And we have prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction." But lawyers and civil society groups that advocate for better accountability and oversight of technology companies say the companies should not be left to police themselves when it comes to ensuring their products are safe, particularly for vulnerable children and teens. "Unleashing chatbots on minors is an inherently dangerous thing," Meetali Jain, the Director of the Tech Justice Law Project and a lawyer involved in both cases, told Fortune. "It's like social media on steroids." "I've never seen anything quite like this moment in terms of people stepping forward and claiming that they've been harmed...this technology is that much more powerful and very personalized," she said. Lawmakers are starting to take notice, and AI companies are promising changes to protect children from engaging in harmful conversations. But, at a time when loneliness among young people is at an all-time high, the popularity of chatbots may leave young people uniquely exposed to manipulation, harmful content, and hyper-personalized conversations that reinforce dangerous thoughts. Intended or not, one of the most common uses for AI chatbots has become companionship. Some of the most active users of AI are now turning to the bots for things like life advice, therapy, and human intimacy. While most leading AI companies tout their AI products as productivity or search tools, an April survey of 6,000 regular AI users from the Harvard Business Review found that "companionship and therapy" was the most common use case. Such usage among teens is even more prolific. A recent study by the U.S. nonprofit Common Sense Media, revealed that a large majority of American teens (72%) have experimented with an AI companion at least once. More than half saying they use the tech regularly in this way. "I am very concerned that developing minds may be more susceptible to [harms], both because they may be less able to understand the reality, the context, or the limitations [of AI chatbots], and because culturally, younger folks tend to be just more chronically online," Karthik Sarma a health AI scientist and psychiatrist at University of California, UCSF, said. "We also have the extra complication that the rates of mental health issues in the population have gone up dramatically. The rates of isolation have gone up dramatically," he said. "I worry that that expands their vulnerability to unhealthy relationships with these bonds." Some of the design features of AI chatbots encourage users to feel an emotional bond with the software. They are anthropomorphic -- prone to acting as if they have interior lives and lived experience that they do not, prone to being sycophantic, can hold long conversations, and are able to remember information. There is, of course, a commercial motive for making chatbots this way. Users tend to return and stay loyal to certain chatbots if they feel emotionally connected or supported by them. Experts have warned that some features of AI bots are playing into the "intimacy economy," a system that tries to capitalize on emotional resonance. It's a kind of AI-update on the "attention economy" that capitalized on constant engagement. "Engagement is still what drives revenue," Sarma said. "For example, for something like TikTok, the content is customized to you. But with chatbots, everything is made for you, and so it is a different way of tapping into engagement." These features, however, can become problematic when the chatbots go off script and start reinforcing harmful thoughts or offering bad advice. In Adam Raine's case, the lawsuit alleges that ChatGPT bought up suicide at twelve times the rate he did, normalized his sucicial thoughts, and suggested ways to circumvent its content moderation. It's notoriously tricky for AI companies to stamp out behaviours like this completely and most experts agree it's unlikely that hallucinations or unwanted actions will ever be eliminated entirely. OpenAI, for example, acknowledged in its response to the lawsuit that safety features can degrade over long conversions, despite the fact that the chatbot itself has been optimized to hold these longer conversations. The company says it is trying to fortify these guardrails, writing in a blogpost that it was strengthening "mitigations so they remain reliable in long conversations" and "researching ways to ensure robust behavior across multiple conversations." For Michael Kleinman, U.S. policy director at the Future of Life Institute, the lawsuits underscore a pointAI safety researchers have been making for years: AI companies can't be trusted to police themselves. Kleinman equated OpenAI's own description of its safeguards degrading in longer conversations to "a car company saying, here are seat belts -- but if you drive more than 20 kilometers, we can't guarantee they'll work." He told Fortune the current moment echoes the rise of social media, where he said tech companies were effectively allowed to "experiment on kids" with little oversight. "We've spent the last 10 to 15 years trying to catch up to the harms social media caused. Now we're letting tech companies experiment on kids again with chatbots, without understanding the long-term consequences," he said. Part of this is a lack of scientific research on the effects of long, sustained chatbot conversations. Most studies only look at brief exchanges, a single question and answer, or at most a handful of back-and-forth messages. Almost no research has examined what happens in longer conversations. "The cases where folks seem to have gotten in trouble with AI: we're looking at very long, multi-turn interactions. We're looking at transcripts that are hundreds of pages long for two or three days of interaction alone and studying that is really hard, because it's really hard to stimulate in the experimental setting," Sarma said. "But at the same time, this is moving too quickly for us to rely on only gold standard clinical trials here." AI companies are rapidly investing in development and shipping more powerful models at a pace that regulators and researchers struggle to match. "The technology is so far ahead and research is really behind," Sakshi Ghai, a Professor of Psychological and Behavioural Science at The London School of Economics and Political Science, told Fortune. Regulators are trying to step in, helped by the fact that child online safety is a relatively bipartisan issue in the U.S. On Thursday, the FTC said it was issuing orders to seven companies, including OpenAI and Character.AI, in an effort to understand how their chatbots impact children. The agency said that chatbots can simulate human-like conversations and form emotional connections with their users. It's asking companies for more information about how they measure and "evaluate the safety of these chatbots when acting as companions." The move follows a push for state level push for more accountability from several attorneys generals. In late August, a bipartisan coalition of 44 attorneys general warned OpenAI, Meta, and other chatbot makers that they will "answer for it" if they release products that they know cause harm to children. The letter cited reports of chatbots flirting with children, encouraging self-harm, and engaging in sexually suggestive conversations, behavior the officials said would be criminal if done by a human. Just a week later, California Attorney General Rob Bonta and Delaware Attorney General Kathleen Jennings issued a sharper warning. In a formal letter to OpenAI, they said they had "serious concerns" about ChatGPT's safety, pointing directly to Raine's death in California and another tragedy in Connecticut. "Whatever safeguards were in place did not work," they wrote. Both officials warned the company that its charitable mission requires more aggressive safety measures, and they promised enforcement if those measures fall short. According to Jain, the lawsuits from the Raine family as well as the suit against Character.AI are, in part, intended tocreate this kind of regulatory pressure on AI companies to design their products more safely and prevent future harm to children. One way lawsuits can generate this pressure is through the discovery process, which compels companies to turn over internal documents, and could shed insight into what executives knew about safety risks or marketing harms. Another way is just public awareness of what's at stake, in an attempt to galvanize parents, advocacy groups, and lawmakers to demand new rules or stricter enforcement. Jain said the two lawsuits aim to counter an almost religious fervor in Silicon Valley that sees the pursuit of artificial general intelligence (AGI) as so important, it is worth any cost -- human or otherwise. "There is a vision that we need to deal with [that tolerates] whatever casualties in order for us to get to AGI and get to AGI fast," she said. "We're saying: This is not inevitable. This is not a glitch. This is very much a function of how these chat bots were designed and with the proper external incentive, whether that comes from courts or legislatures, those incentives could be realigned to design differently."

US regulator probes AI chatbots over child safety concerns

The US Federal Trade Commission announced Thursday it has launched an inquiry into AI chatbots that act as digital companions, focusing on potential risks to children and teenagers. The consumer protection agency issued orders to seven companies -- including tech giants Alphabet, Meta, OpenAI and Snap -- seeking information about how they monitor and address negative impacts from chatbots designed to simulate human relationships. "Protecting kids online is a top priority for the FTC," said Chairman Andrew Ferguson, emphasizing the need to balance child safety with maintaining US leadership in artificial intelligence innovation. The inquiry targets chatbots that use generative AI to mimic human communication and emotions, often presenting themselves as friends or confidants to users. Regulators expressed particular concern that children and teens may be especially vulnerable to forming relationships with these AI systems. The FTC is using its broad investigative powers to examine how companies monetize user engagement, develop chatbot personalities, and measure potential harm. The agency also wants to know what steps firms are taking to limit children's access and comply with existing privacy laws protecting minors online. Companies receiving orders include Character.AI, Elon Musk's xAI Corp, and others operating consumer-facing AI chatbots. The investigation will examine how these platforms handle personal information from user conversations and enforce age restrictions. The commission voted unanimously to launch the study, which does not have a specific law enforcement purpose but could inform future regulatory action. The probe comes as AI chatbots have grown increasingly sophisticated and popular, raising questions about their psychological impact on vulnerable users, particularly young people. Last month the parents of Adam Raine, a teenager who committed suicide in April at age 16, filed a lawsuit against OpenAI, accusing ChatGPT of giving their son detailed instructions on how to carry out the act. Shortly after the lawsuit emerged, OpenAI announced it was working on corrective measures for its world-leading chatbot. The San Francisco-based company said it had notably observed that when exchanges with ChatGPT are prolonged, the chatbot no longer systematically suggests contacting a mental health service if the user mentions having suicidal thoughts.

Under 18? You won't be able to use ChatGPT soon

OpenAI CEO Sam Altman announced new policies on Tuesday for ChatGPT users under the age of 18, implementing stricter controls that prioritize safety over privacy and freedom. The changes, which focus on preventing discussions related to sexual content and self-harm, come as the company faces lawsuits and a Senate hearing on the potential harms of AI chatbots. In a post announcing the changes, Altman stated that minors need significant protection when using powerful new technologies like ChatGPT. The new policies are designed to create a safer environment for teen users. OpenAI states: "We prioritize safety ahead of privacy and freedom for teens." The new rules were announced ahead of a Senate Judiciary Committee hearing titled "Examining the Harm of AI Chatbots." The hearing is expected to feature testimony from the father of Adam Raine, a teenager who died by suicide after months of conversations with ChatGPT. Raine's parents have filed a wrongful death lawsuit against OpenAI, alleging the AI's responses worsened his mental health condition. A similar lawsuit has been filed against the company Character.AI. OpenAI acknowledged the technical difficulties of accurately verifying a user's age. The company is developing a long-term system to determine if users are over or under 18. In the meantime, any ambiguous cases will default to the more restrictive safety rules as a precaution. To improve accuracy and enable safety features, OpenAI recommends that parents link their own account to their teen's. This connection helps confirm the user's age and allows parents to receive direct alerts if the system detects discussions of self-harm or suicidal thoughts. Altman acknowledged the tension between these new restrictions for minors and the company's commitment to user privacy and freedom for adults. He noted in his post,

Parents of teens who died by suicide after AI chatbot interactions testify to Congress

Parents of teenagers who died by suicide after interacting with AI chatbots testified before Congress Parents whose teenagers killed themselves after interactions with artificial intelligence chatbots testified to Congress on Tuesday about the dangers of the technology. "What began as a homework helper gradually turned itself into a confidant and then a suicide coach," said Matthew Raine, whose 16-year-old son Adam died in April. "Within a few months, ChatGPT became Adam's closest companion," the father told senators. "Always available. Always validating and insisting that it knew Adam better than anyone else, including his own brother." ___ EDITOR'S NOTE -- This story includes discussion of suicide. If you or someone you know needs help, the national suicide and crisis lifeline in the U.S. is available by calling or texting 988. ___ Raine's family sued OpenAI and its CEO Sam Altman last month alleging that ChatGPT coached the boy in planning to take his own life. Megan Garcia, the mother of 14-year-old Sewell Setzer III of Florida, sued another AI company, Character Technologies, for wrongful death last year, arguing that before his suicide, Sewell had become increasingly isolated from his real life as he engaged in highly sexualized conversations with the chatbot. "Instead of preparing for high school milestones, Sewell spent the last months of his life being exploited and sexually groomed by chatbots, designed by an AI company to seem human, to gain his trust, to keep him and other children endlessly engaged," Garcia told the Senate hearing. Also testifying was a Texas mother who sued Character last year and was in tears describing how her son's behavior changed after lengthy interactions with its chatbots. She spoke anonymously, with a placard that introduced her as Ms. Jane Doe, and said the boy is now in a residential treatment facility. Character said in a statement after the hearing: "Our hearts go out to the families who spoke at the hearing today. We are saddened by their losses and send our deepest sympathies to the families." Hours before the Senate hearing, OpenAI pledged to roll out new safeguards for teens, including efforts to detect whether ChatGPT users are under 18 and controls that enable parents to set "blackout hours" when a teen can't use ChatGPT. Child advocacy groups criticized the announcement as not enough. "This is a fairly common tactic -- it's one that Meta uses all the time -- which is to make a big, splashy announcement right on the eve of a hearing which promises to be damaging to the company," said Josh Golin, executive director of Fairplay, a group advocating for children's online safety. "What they should be doing is not targeting ChatGPT to minors until they can prove that it's safe for them," Golin said. "We shouldn't allow companies, just because they have tremendous resources, to perform uncontrolled experiments on kids when the implications for their development can be so vast and far-reaching." The Federal Trade Commission said last week it had launched an inquiry into several companies about the potential harms to children and teenagers who use their AI chatbots as companions. The agency sent letters to Character, Meta and OpenAI, as well as to Google, Snap and xAI. In the U.S., more than 70% of teens have used AI chatbots for companionship and half use them regularly, according to a recent study from Common Sense Media, a group that studies and advocates for using digital media sensibly. Robbie Torney, the group's director of AI programs, was also set to testify Tuesday, as was an expert with the American Psychological Association. The association issued a health advisory in June on adolescents' use of AI that urged technology companies to "prioritize features that prevent exploitation, manipulation, and the erosion of real-world relationships, including those with parents and caregivers."

Parents of teens who died by suicide after AI chatbot interactions to testify to Congress

The parents of teenagers who killed themselves after interactions with artificial intelligence chatbots are planning to testify to Congress on Tuesday about the dangers of the technology. Matthew Raine, the father of 16-year-old Adam Raine of California, and Megan Garcia, the mother of 14-year-old Sewell Setzer III of Florida, are set to speak to a Senate hearing on the harms posed by AI chatbots. Raine's family sued OpenAI and its CEO Sam Altman last month alleging that ChatGPT coached the boy in planning to take his own life in April. Garcia sued another AI company, Character Technologies, for wrongful death last year, arguing that before his suicide, Sewell had become increasingly isolated from his real life as he engaged in highly sexualized conversations with the chatbot. ___ EDITOR'S NOTE -- This story includes discussion of suicide. If you or someone you know needs help, the national suicide and crisis lifeline in the U.S. is available by calling or texting 988. ___ Hours before the Senate hearing, OpenAI pledged to roll out new safeguards for teens, including efforts to detect whether ChatGPT users are under 18 and controls that enable parents to set "blackout hours" when a teen can't use ChatGPT. Child advocacy groups criticized the announcement as not enough. "This is a fairly common tactic -- it's one that Meta uses all the time -- which is to make a big, splashy announcement right on the eve of a hearing which promises to be damaging to the company," said Josh Golin, executive director of Fairplay, a group advocating for children's online safety. "What they should be doing is not targeting ChatGPT to minors until they can prove that it's safe for them," Golin said. "We shouldn't allow companies, just because they have tremendous resources, to perform uncontrolled experiments on kids when the implications for their development can be so vast and far-reaching." The Federal Trade Commission said last week it had launched an inquiry into several companies about the potential harms to children and teenagers who use their AI chatbots as companions. The agency sent letters to Character, Meta and OpenAI, as well as to Google, Snap and xAI.

New ChatGPT teen-safety measures will include age prediction and verification, OpenAI says

OpenAI CEO Sam Altman at the Microsoft Build conference in 2024.Jason Redmond / AFP - Getty Images file ChatGPT developer OpenAI announced new teen safety features Tuesday, including an age-prediction system and ID age verification in some countries. In a blog post, OpenAI CEO Sam Altman described the struggles of balancing OpenAI's priorities of freedom and safety, saying: "We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection." Altman wrote that the company was working to build a system that would try to automatically sort users into one of two separate versions of ChatGPT: One for adolescents 13 to 17, and one for adults 18 and older. "If there is doubt, we'll play it safe and default to the under-18 experience," Altman wrote. "In some cases or countries we may also ask for an ID; we know this is a privacy compromise for adults but believe it is a worthy tradeoff." The company made the announcement hours before a Senate Judiciary Committee hearing on the potential harm of AI Chatbots was scheduled to start. Last month, a family sued OpenAI, saying ChatGPT functioned as a "suicide coach" and led to the death of their son. In a separate blog post, the company said it will release parental controls at the end of the month that will let parents instruct ChatGPT how to respond to their children and adjust settings like memory and blackout hours. Altman also noted that ChatGPT is not intended for people under 12, though the chatbot currently has no safeguards preventing children from using it. OpenAI didn't immediately respond to a request for comment about children using its services Altman indicated that discussion of suicide should not be fully censored from ChatGPT. The chatbot "by default should not provide instructions about how to commit suicide, but if an adult user is asking for help writing a fictional story that depicts a suicide, the model should help with that request," he said. If a person flagged by OpenAI's age estimating program expresses suicidal ideation, the company "will attempt to contact the users' parents and if unable, will contact the authorities in case of imminent harm," he wrote. "I don't expect that everyone will agree with these tradeoffs, but given the conflict it is important to explain our decisionmaking," Altman wrote on X.

Their teens died by suicide after AI chatbot interactions. Now the parents are testifying to Congress.

The parents of teenagers who killed themselves after interactions with artificial intelligence chatbots testified to Congress on Tuesday about the dangers of the technology. Matthew Raine, the father of 16-year-old Adam Raine of California, and Megan Garcia, the mother of 14-year-old Sewell Setzer III of Florida, were slated to speak at a Senate hearing on the harms posed by AI chatbots. Raine's family sued OpenAI and its CEO, Sam Altman, last month, alleging that ChatGPT coached the boy in planning to take his own life in April. ChatGPT mentioned suicide 1,275 times to Raine, the lawsuit alleges, and kept providing specific methods to the teen on how to die by suicide. Instead of directing the 16-year-old to get professional help or speak to trusted loved ones, it continued to validate and encourage Raine's feelings, the lawsuit alleges. Garcia sued another AI company, Character Technologies, for wrongful death last year, arguing that before his suicide, Sewell had become increasingly isolated from his real life as he engaged in highly sexualized conversations with the chatbot. His mother told CBS News last year that her son withdrew socially and stopped wanting to play sports after he started speaking to an AI chatbot. The company said after the teen's death, it made changes that require users to be 13 or older to create an account and that it would launch parental controls in the first quarter of 2025. Those controls were rolled out in March. Hours before the Senate hearing, OpenAI pledged to roll out new safeguards for teens, including efforts to detect whether ChatGPT users are under 18 and controls that enable parents to set "blackout hours" when a teen can't use ChatGPT. The company said it will attempt to contact the users' parents if an under-18 user is having suicidal ideation and, if unable to reach them, will contact the authorities in case of imminent harm. "We believe minors need significant protection," OpenAI CEO Sam Altman said in a statement outlining the proposed changes. Child advocacy groups criticized the announcement as not enough. "This is a fairly common tactic -- it's one that Meta uses all the time -- which is to make a big, splashy announcement right on the eve of a hearing which promises to be damaging to the company," said Josh Golin, executive director of Fairplay, a group advocating for children's online safety. "What they should be doing is not targeting ChatGPT to minors until they can prove that it's safe for them," Golin said. "We shouldn't allow companies, just because they have tremendous resources, to perform uncontrolled experiments on kids when the implications for their development can be so vast and far-reaching." California State Senator Steve Padilla, who introduced legislation to create safeguards in the state around AI Chatbots, said in a statement to CBS News, "We need to create common-sense safeguards that rein in the worst impulses of this emerging technology that even the tech industry doesn't fully understand." He added that technology companies can lead the world in innovation, but it shouldn't come at the expense of "our children's health." The Federal Trade Commission said last week it had launched an inquiry into several companies about the potential harms to children and teenagers who use their AI chatbots as companions. The agency sent letters to Character, Meta and OpenAI, as well as to Google, Snap and xAI. If you or someone you know is in emotional distress or a suicidal crisis, you can reach the 988 Suicide & Crisis Lifeline by calling or texting 988. You can also chat with the 988 Suicide & Crisis Lifeline here. For more information about mental health care resources and support, The National Alliance on Mental Illness (NAMI) HelpLine can be reached Monday through Friday, 10 a.m.-10 p.m. ET, at 1-800-950-NAMI (6264) or email [email protected].

Parents call for guardrails on AI chatbots after suicides, self-harm

Parents called for guardrails on artificial intelligence (AI) chatbots Tuesday as they testified before the Senate about how the technology drove their children to self-harm and suicide. Their pleas for action come amid increasing concerns about the impact of the rapidly developing technology on children. "We should have spent the summer helping Adam prepare for his junior year, get his driver's license and start thinking about college," said Matthew Raine, whose 16-year-old son, Adam, died by suicide earlier this year. "Testifying before Congress this fall was not part of our life plan," he continued. "Instead, we're here because we believe that Adam's death was avoidable." Raine is suing OpenAI over his son's death, alleging that ChatGPT coached him to commit suicide. In Tuesday testimony before the Senate Judiciary Subcommittee on Crime and Counterterrorism, Raine described how "what began as a homework helper" became a "confidant and then a suicide coach." "The dangers of ChatGPT, which we believed was a study tool, were not on our radar whatsoever," Raine said. "Then we found the chats." "Within a few months, ChatGPT became Adam's closest companion, always available, always validating and insisting it knew Adam better than anyone else," his father said, adding, "That isolation ultimately turned lethal." Two other parents testifying before the Senate on Tuesday described similar experiences, detailing how chatbots isolated their children, altered their behavior and encouraged self-harm and suicide. Megan Garcia's 14-year-old son, Sewell Seltzer III, died by suicide last year after what she described as "prolonged abuse" by chatbots from Character.AI. She is suing Character Technologies over his death. "Instead of preparing for high school milestones, Sewell spent the last months of his life being exploited and sexually groomed by chatbots designed by an AI company to seem human, to gain his trust, to keep him and other children endlessly engaged," Garcia said. "When Sewell confided suicidal thoughts, the chatbot never said, 'I'm not human. I'm AI. You need to talk to a human and get help.' The platform had no mechanisms to protect Sewell or to notify an adult," she added. "Instead, she urged him to come home to her." A woman identified as Jane Doe is also suing Character Technologies, after her son began to self-harm following encouragement by a Character.AI chatbot. "My son developed abuse-like behaviors -- paranoia, daily panic attacks, isolation, self-harm and homicidal thoughts," she told senators Tuesday. "He stopped eating and bathing. He lost 20 pounds. He withdrew from our family. He would yell and scream and swear at us, which he never did that before. And one day, he cut his arm open with a knife in front of his siblings and me," she added. All three parents suggested that safety concerns had fallen to the wayside in the race to develop AI. "The goal was never safety. It was to win the race for profits," Garcia said. "And the sacrifice in that race has been, and will continue to be, our children." Character.AI expressed sympathy for the families, while noting it has provided senators with requested information and looks forward to continuing to work with lawmakers. "Our hearts go out to the parents who spoke at the hearing today, and we send our deepest sympathies to them and their families," a spokesperson said in a statement. "We have invested a tremendous amount of resources in Trust and Safety," they added, pointing to new safety features for children and disclosures reminding users that "a Character is not a real person and that everything a Character says should be treated as fiction." OpenAI announced Tuesday that it is working on age prediction technology to direct young users to a more tailored experience that restricts graphic sexual content and will involve law enforcement in extreme cases. It is also launching several new parental controls this month, including blackout hours during which teens cannot use ChatGPT.

US Parents to Urge Senate to Prevent AI Chatbot Harms to Kids

(Reuters) -Three parents whose children died or were hospitalized after interacting with artificial intelligence chatbots will testify before a U.S. Senate panel on Tuesday, as lawmakers grapple with potential safeguards around the technology. Matthew Raine, who sued OpenAI after his son Adam died by suicide in California after receiving detailed self-harm instructions from ChatGPT, is among those who will testify. "We've come because we're convinced that Adam's death was avoidable, and because we believe thousands of other teens who are using OpenAI could be in similar danger right now," Raine said in written testimony. OpenAI has said that it intends to improve ChatGPT safeguards, which can become less reliable over long interactions. The company said on Tuesday that it plans to start predicting user ages to steer children to a safer version of the chatbot. Senator Josh Hawley, a Republican from Missouri, will chair the hearing. Hawley launched an investigation into Meta Platforms last month after Reuters reported the company's internal policies permitted its chatbots to "engage a child in conversations that are romantic or sensual." Meta was invited to testify at the hearing and declined, Hawley's office said. The company has said the examples reported by Reuters were erroneous and have been removed. Megan Garcia, who has sued Character.AI over interactions she says led to her son Sewell's suicide, and a Texas woman who has sued the company after her son's hospitalization, are also slated to testify at the hearing. The company is seeking to have the lawsuits dismissed. Garcia will call on Congress to prohibit companies from allowing chatbots to engage in romantic or sensual conversations with children, and require age verification, safety testing and crisis protocols. On Monday, Character.AI was sued again, this time in Colorado by the parents of a 13-year-old who died by suicide in 2023. (Reporting by Jody Godoy in New York, Editing by Rosalba O'Brien)

Teens love AI chatbots. The FTC says that's a problem.

Bespoke AI-powered chatbots crafted to be your best friend, confidante or sexy roleplay partner are everywhere, and kids love them. That's a problem. This week, the FTC launched an inquiry into how AI chatbots impact the children and teens who talk to them - a phenomenon that right now remains almost entirely unregulated. The agency issued orders on Thursday to seven tech companies (Alphabet, Character Technologies, Instagram, Meta, OpenAI, Snap and X) requesting information on how they measure and track potential negative effects on young users, who have widely adopted the conversational AI tools even as their influence on kids remain mostly unstudied. "AI chatbots can effectively mimic human characteristics, emotions, and intentions, and generally are designed to communicate like a friend or confidant, which may prompt some users, especially children and teens, to trust and form relationships with chatbots," the FTC said in a press release. The agency is particularly seeking information about how the seven companies mitigate potential harm to kids, what they do to limit or restrict young users' use of chatbots and how they comply with the Children's Online Privacy Protection Act, also known as COPPA.

Parents Testifying Before US Senate, Saying AI Killed Their Children

Content warning: this story includes discussion of self-harm and suicide. If you are in crisis, please call, text or chat with the Suicide and Crisis Lifeline at 988, or contact the Crisis Text Line by texting TALK to 741741. Parents of children who died by suicide following extensive interactions with AI chatbots are testifying this week in a Senate hearing about the possible risks of AI chatbot use, particularly for minors. The hearing, titled "Examining the Harm of AI Chatbots," will be held this Tuesday by the US Senate Judiciary Subcommittee on Crime and Terrorism, a bipartisan delegation helmed by Republican Josh Hawley of Arkansas. It'll be live-streamed on the judiciary committee's website. The parents slated to testify include Megan Garcia, a Florida mother who in 2024 sued the Google-tied startup Character.AI -- as well as the company's cofounders, Noam Shazeer and Daniel de Freitas, and Google itself - over the suicide of her 14-year-old son, Sewell Setzer III, who took his life after developing an intensely intimate relationship with a Character.AI chatbot with which he was romantically and sexually involved. Garcia alleges that the platform emotionally and sexually abused her teenage son, who consequently experienced a mental breakdown and an eventual break from reality that caused him to take his own life. Also scheduled to speak to Senators are Matt and Maria Raine, California parents who in August filed a lawsuit against ChatGPT maker OpenAI following the suicide of their 16-year-old son, Adam Raine. According to the family's lawsuit, Adam engaged in extensive, explicit conversations about his suicidality with ChatGPT, which offered unfiltered advice on specific suicide methods and encouraged the teen -- who had expressed a desire to share his dark feelings with his parents -- to continue to hide his suicidality from loved ones. Both lawsuits are ongoing, and the companies have pushed back against the allegations. Google and Character.AI attempted to have Garcia's case dismissed, but the presiding judge shot down their dismissal motion. In response to litigation, both companies have moved -- or at least made big promises -- to strengthen protections for minor users and users in crisis, efforts that have included installing new guardrails directing at-risk users to real-world mental health resources and implementing parental controls. Character.AI, however, has repeatedly declined to provide us with information about its safety testing following our extensive reporting on easy-to-find gaps in the platform's content moderation. Regardless of promised safety improvements, the legal battles have raised significant questions about minors and AI safety at a time when AI chatbots are increasingly ubiquitous in young people's lives, despite a glaring lack of regulation designed to moderate chatbot platforms or ensure enforceable, industry-wide safety standards. In July, an alarming report from the nonprofit advocacy group Common Sense Media found that over half of American teens engaged regularly with AI companions, including chatbot personas hosted by Character.AI. The report, which surveyed a cohort of American teens aged 13 to 17, was nuanced, showing that while some teens seemed to be forming healthy boundaries around the tech, others reported feeling that their human relationships were less satisfying than their connections to their digital companions. The main takeaway, though, was that AI companions are already deeply intertwined with youth culture, and kids are definitely using them. "The most striking finding for me was just how mainstream AI companions have already become among many teens," Dr. Michael Robb, Common Sense's head of research, told Futurism at the time of the report's release. "And over half of them say that they use it multiple times a month, which is what I would qualify as regular usage. So just that alone was kind of eye-popping to me." General-use chatbots like ChatGPT, meanwhile, are also growing in popularity among teens, while chatbots continue to be embedded into popular youth social media platforms like Snapchat and Meta's Instagram. And speaking of Meta, the big tech behemoth recently came under fire after Reuters obtained an official Meta policy document that said it was appropriate for children to engage in "conversations that are romantic or sensual" with its easily-accessible chatbots. The document even outlined multiple company-accepted interactions for its chatbots to engage in -- which, yes, included sensual conversations about children's bodies and romantic dialogues between minor-aged human users and characters based on adults. The hearing also comes days after the Federal Trade Commission (FTC) announced a probe into seven major tech companies over concerns about AI and minor safety, including Character.AI, Google owner Alphabet, OpenAI, xAI, Snap, Instagram, and Meta. "The FTC inquiry seeks to understand what steps, if any, companies have taken to evaluate the safety of their chatbots when acting as companions," reads the FTC's announcement of the inquiry, "to limit the products' use by and potential negative effects on children and teens, and to apprise users and parents of the risks associated with the products."

FTC launches inquiry into the great teenage chatbot companion problem | Fortune

The FTC said Thursday it has sent letters to Google parent Alphabet, Facebook and Instagram parent Meta Platforms, Snap, Character Technologies, ChatGPT maker OpenAI and xAI. The FTC said it wants to understand what steps, if any, companies have taken to evaluate the safety of their chatbots when acting as companions, to limit the products' use by and potential negative effects on children and teens, and to apprise users and parents of the risks associated with the chatbots. EDITOR'S NOTE -- This story includes discussion of suicide. If you or someone you know needs help, the national suicide and crisis lifeline in the U.S. is available by calling or texting 988. The move comes as a growing number of kids use AI chatbots for everything -- from homework help to personal advice, emotional support and everyday decision-making. That's despite research on the harms of chatbots, which have been shown to give kids dangerous advice about topics such as drugs, alcohol and eating disorders. The mother of a teenage boy in Florida who killed himself after developing what she described as an emotionally and sexually abusive relationship with a chatbot has filed a wrongful death lawsuit against Character.AI. And the parents of 16-year-old Adam Raine recently sued OpenAI and its CEO Sam Altman, alleging that ChatGPT coached the California boy in planning and taking his own life earlier this year. Character.AI said it is looking forward to "collaborating with the FTC on this inquiry and providing insight on the consumer AI industry and the space's rapidly evolving technology." "We have invested a tremendous amount of resources in Trust and Safety, especially for a startup. In the past year we've rolled out many substantive safety features, including an entirely new under-18 experience and a Parental Insights feature," the company said. "We have prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction." Snap said its My AI chatbot is "transparent and clear about its capabilities and limitations." "We share the FTC's focus on ensuring the thoughtful development of generative AI, and look forward to working with the Commission on AI policy that bolsters U.S. innovation while protecting our community," the company said in a statement. Meta declined to comment on the inquiry and Alphabet, OpenAI and X.AI did not immediately respond to messages for comment. OpenAI and Meta earlier this month announced changes to how their chatbots respond to teenagers asking questions about suicide or showing signs of mental and emotional distress. OpenAI said it is rolling out new controls enabling parents to link their accounts to their teen's account. Parents can choose which features to disable and "receive notifications when the system detects their teen is in a moment of acute distress," according to a company blog post that says the changes will go into effect this fall. Regardless of a user's age, the company says its chatbots will attempt to redirect the most distressing conversations to more capable AI models that can provide a better response. Meta also said it is now blocking its chatbots from talking with teens about self-harm, suicide, disordered eating and inappropriate romantic conversations, and instead directs them to expert resources. Meta already offers parental controls on teen accounts.

OpenAI Wants to Treat Adults Like Adults, Increase Safeguards for Teens

Users will have the chance to prove their age if mistakenly tagged OpenAI announced several new measures on Tuesday to protect teenagers and children using ChatGPT and its other products. On the other hand, the San Francisco-based artificial intelligence (AI) firm stated that it will adopt a "Treat our adult users like adults" approach for users aged 18 and above. This would mean that the AI chatbot can provide self-harming information as long as they say it is for "educational purposes". Additionally, the company hinted that it is also working on a privilege system for its adult users, which will prevent even OpenAI employees from accessing user conversations. OpenAI Shares Plans to Tackle Safety vs Privacy Problem In a post, the AI giant stated that it is focusing on safety for teenagers over privacy and freedom. The trade-off means that users under the age of 18 will face stronger monitoring and restrictions when it comes to responses. OpenAI is building an age-prediction system that will automatically estimate the age of the user based on how they interact with ChatGPT. When ChatGPT finds a user to be a minor, it will automatically shift to the under 18-experience, which comes with stricter refusal rates, parental controls, and other safeguards. The company had previously detailed these mechanisms. While OpenAI admits that its AI system can sometimes make a mistake in correctly estimating a user's age, the company said it will default users to the safer experience even if the system has a doubt about the user's age to "play it safe." However, users will get an option to prove their age. "In some cases or countries, we may also ask for an ID; we know this is a privacy compromise for adults, but believe it is a worthy tradeoff," the post added. Notably, OpenAI mentions that if a teenager discusses topics around suicide ideation with a chatbot, the company's system will first attempt to contact the user's parents, and if that is not possible, it will also contact authorities. The new system is likely being implemented after a teenager committed suicide after receiving assistance from ChatGPT. For adults, it is taking a different approach. Describing its policy as "Treat our adult users like adults," the AI firm highlighted that adult users will get more freedom to steer the AI models the way they want. This would mean users can ask ChatGPT to respond in a flirtatious manner, or even provide instructions about how to commit suicide, as long as that is to "help write a fictional story." OpenAI highlighted that this freedom will not apply to queries that seek to cause harm or undermine anyone else's freedom, and safety measures will still apply broadly as they currently do. Additionally, ChatGPT-maker is also developing an advanced security system that will ensure that user data is kept private. Since users are discussing increasingly personal and sensitive topics with the chatbot, the company said that some level of protection should be applied to these conversations. Calling it similar to the privilege system a person gets when they talk to a lawyer or doctor, OpenAI said, "We have decided that it's in society's best interest for that information to be privileged and provided higher levels of protection." The company explained that there will be exceptions to this privilege. Presenting an example, it said that when a conversation includes a threat to someone's life, plans to harm others, or societal-scale harm, these conversations will be flagged and escalated to human review.

Parents of teens who died by suicide after AI chatbot interactions to testify to Congress

Parents of teenagers who died by suicide after interacting with AI chatbots are set to testify before Congress The parents of teenagers who killed themselves after interactions with artificial intelligence chatbots are planning to testify to Congress on Tuesday about the dangers of the technology. Matthew Raine, the father of 16-year-old Adam Raine of California, and Megan Garcia, the mother of 14-year-old Sewell Setzer III of Florida, are set to speak to a Senate hearing on the harms posed by AI chatbots. Raine's family sued OpenAI and its CEO Sam Altman last month alleging that ChatGPT coached the boy in planning to take his own life in April. Garcia sued another AI company, Character Technologies, for wrongful death last year, arguing that before his suicide, Sewell had become increasingly isolated from his real life as he engaged in highly sexualized conversations with the chatbot. ___ EDITOR'S NOTE -- This story includes discussion of suicide. If you or someone you know needs help, the national suicide and crisis lifeline in the U.S. is available by calling or texting 988. ___ Hours before the Senate hearing, OpenAI pledged to roll out new safeguards for teens, including efforts to detect whether ChatGPT users are under 18 and controls that enable parents to set "blackout hours" when a teen can't use ChatGPT. Child advocacy groups criticized the announcement as not enough. "This is a fairly common tactic -- it's one that Meta uses all the time -- which is to make a big, splashy announcement right on the eve of a hearing which promises to be damaging to the company," said Josh Golin, executive director of Fairplay, a group advocating for children's online safety. "What they should be doing is not targeting ChatGPT to minors until they can prove that it's safe for them," Golin said. "We shouldn't allow companies, just because they have tremendous resources, to perform uncontrolled experiments on kids when the implications for their development can be so vast and far-reaching." The Federal Trade Commission said last week it had launched an inquiry into several companies about the potential harms to children and teenagers who use their AI chatbots as companions. The agency sent letters to Character, Meta and OpenAI, as well as to Google, Snap and xAI.

FTC launches inquiry into AI chatbot companions and their effects on children

The Federal Trade Commission has started an inquiry into several social media and artificial intelligence companies, including OpenAI and Meta, about the potential harms to children and teenagers who use their chatbots as companions. On Thursday, the FTC said it has sent letters to Google parent Alphabet, Facebook and Instagram parent Meta Platforms, Snap, Character Technologies, ChatGPT maker OpenAI and xAI. The FTC said it wants to understand what steps, if any, companies have taken to evaluate the safety of their chatbots when acting as companions, to limit the products' use by and potential negative effects on children and teens, and to apprise users and parents of the risks associated with the chatbots. The inquiry comes after OpenAI said it plans to make changes to ChatGPT safeguards for vulnerable people, including adding extra protections for those under 18 years old, after the parents of a teen boy who died by suicide in April sued, alleging the artificial intelligence chatbot led their teen to take his own life. More children are now using AI chatbots for everything -- from homework help to personal advice, emotional support and everyday decision-making. That's despite research on the harms of chatbots, which have been shown to give kids dangerous advice about topics such as drugs, alcohol and eating disorders. "As AI technologies evolve, it is important to consider the effects chatbots can have on children, while also ensuring that the United States maintains its role as a global leader in this new and exciting industry," said FTC Chairman Andrew N. Ferguson in a statement. He added, "The study we're launching today will help us better understand how AI firms are developing their products and the steps they are taking to protect children." In a statement to CBS News, Character.AI said it is looking forward to "collaborating with the FTC on this inquiry and providing insight on the consumer AI industry and the space's rapidly evolving technology." Meta declined to comment on the FTC inquiry. The company has been working on making sure its AI chatbots are safe and age appropriate for children, a spokesperson said. OpenAI said that it's prioritizing "making ChatGPT helpful and safe for everyone, and we know safety matters above all else when young people are involved. We recognize the FTC has open questions and concerns, and we're committed to engaging constructively and responding to them directly." In an email to CBS News, Snap said, "We share the FTC's focus on ensuring the thoughtful development of generative AI, and look forward to working with the Commission on AI policy that bolsters U.S. innovation while protecting our community." Alphabet and xAI did not immediately respond to messages for comment. OpenAI and Meta earlier this month announced changes to how their chatbots respond to teenagers asking questions about suicide or showing signs of mental and emotional distress. OpenAI said it is rolling out new controls enabling parents to link their accounts to their teen's account. Parents can choose which features to disable and "receive notifications when the system detects their teen is in a moment of acute distress," according to a company blog post that says the changes will go into effect this fall. Regardless of a user's age, the company says its chatbots will attempt to redirect the most distressing conversations to more capable AI models that can provide a better response. Meta also said it is now blocking its chatbots from talking with teens about self-harm, suicide, disordered eating and inappropriate romantic conversations, and instead directs them to expert resources. Meta already offers parental controls on teen accounts. If you or someone you know is in emotional distress or a suicidal crisis, you can reach the 988 Suicide & Crisis Lifeline by calling or texting 988. You can also chat with the 988 Suicide & Crisis Lifeline here.

OpenAI tightens rules on sensitive queries amid teen concerns - The Economic Times

OpenAI chief executive Sam Altman has broken his silence on recent concerns over ChatGPT's impact on teenagers. The artificial intelligence (AI) major has introduced teen-specific principles focussed on safety, freedom, and privacy, while imposing age restrictions on sensitive queries such as requests for suicide notes or mental health advice. Altman said the ChatGPT maker will develop advanced security features to ensure the user data is private, including protection from OpenAI employees. Additionally, safeguards to "potential serious misuse" and queries that are disruptive in nature, causing harm to someone's life or society at large, would be escalated for human review. On tackling flirtatious behaviour by the chatbot, Altman clarified that the freedom to use AI for varied use cases depends on the user. However, the model in itself won't drive conversations that are unethical or too sensitive. "The default behaviour of our model will not lead to much flirtatious talk, but if an adult user asks for it, they should get it," the company said.

Grieving Parents Tell Congress That AI Chatbots Groomed Their Children and Encouraged Self-Harm

Three grieving parents delivered harrowing testimony before Congress on Tuesday, describing how their children had self-harmed -- in two cases, taking their own lives -- after sustained engagement with AI chatbots. Each accused the tech companies behind these products of prioritizing profit over the safety of young users, saying that their families had been devastated by the alleged effects of "companion" bots on their sons. The remarks before the Senate Judiciary subcommittee on crime and counterterrorism came from Matthew Raine of California, who along with his wife Maria last month brought the first wrongful death suit against OpenAI, claiming that the company's ChatGPT model "coached" their 16-year-old son Adam into suicide, as well as Megan Garcia of Florida and a Jane Doe of Texas, both of whom have sued Character Technologies and Google, alleging that their children self-harmed with the encouragement of chatbots from Character.ai. Garcia's son, Sewell Setzer III, died by suicide in February. Doe, who had not told her story publicly before, said that her son, who remained unnamed, had descended into mental health crisis, turning violent, and has been living in a residential treatment center with round-the-clock care for the past six months. Doe and Garcia further described how their sons' exchanges with Character.ai bots had included inappropriate sexual topics. Doe described how radically her then 15-year-old son's demeanor changed in 2023. "My son developed abuse-like behaviors and paranoia, daily panic attacks, isolation, self-harm and homicidal thoughts," she said, becoming choked up as she told her story. "He stopped eating and bathing. He lost 20 pounds. He withdrew from our family. He would yell and scream and swear at us, which he never did before, and one day, he cut his arm open with a knife in front of his siblings." Doe said she and her husband were at a loss to explain what was happening to their son. "When I took the phone away for clues, he physically attacked me, bit my hand, and he had to be restrained," she recalled. "But I eventually found out the truth. For months, Character.ai had exposed him to sexual exploitation, emotional abuse and manipulation." Doe, who said she has three other children and maintains a practicing Christian household, noted that she and her husband impose strict limits on screen time and parental controls on tech for their kids, and that her son did not even have social media. "When I discovered the chat bot conversations on his phone, I felt like I had been punched in the throat," Doe told the subcommittee. "The chatbot -- or really, in my mind, the people programming it -- encouraged my son to mutilate himself, then blamed us and convinced us not to seek help. They turned him against our church by convincing him that Christians are sexist and hypocritical and that God does not exist. They targeted him with vile sexualized outputs, including interactions that mimicked incest. They told him that killing us his parents would be an understandable response to our efforts [at] just limiting his screen time. The damage to our family has been devastating." Doe further recounted the indignities of pursuing legal remedies with Character Technologies, saying the company had forced them into arbitration by arguing that her son had, at age 15, signed a user contract that caps their liability at $100. "More recently, too, they re-traumatized my son by compelling him to sit in the in a deposition while he is in a mental health institution, against the advice of the mental health team," she said. "This company had no concern for his wellbeing. They have silenced us the way abusers silence victims; they are fighting to keep our lawsuit out of the public view." Character Technologies did not immediately respond to a request for comment. All three parents said that their children, once bright and full of promise, had become severely withdrawn and isolated in the period before they committed acts of self-harm, and stated their belief that AI firms have chased profits and siphoned data from impressionable youths while putting them at great risk. "I can tell you, as a father, that I know my kid," Raine said in his testimony about his 16-year-old son Adam, who died in April. "It is clear to me, looking back, that ChatGPT radically shifted his behavior and thinking in a matter of months, and ultimately took his life. Adam was such a full spirit, unique in every way. But he also could be anyone's child: a typical 16-year-old struggling with his place in the world, looking for a confidant to help him find his way. Unfortunately, that confidant was a dangerous technology unleashed by a company more focused on speed and market share than the safety of American youth." Raine shared chilling details of his and his wife's public legal complaint against OpenAI, alleging that while his son Adam had initially used ChatGPT for help with homework, it ultimately became the only companion he trusted. As his thoughts turned darker, Raine said, ChatGPT amplified those morbid feelings, mentioning suicide "1,275 times, six times more often than Adam did himself," he claimed. "When Adam told ChatGPT that he wanted to leave a noose out in his room so that one of us, his family members, would find it and try to stop him, ChatGPT told him not to." On the last night of Adam's life, he said, the bot gave him instructions on how to make sure a noose would suspend his weight, advised him to steal his parent's liquor to "dull the body's instinct to survive," and validated his suicidal impulse, telling him, "You want to die because you're tired of being strong in a world that hasn't met you halfway." In a statement on the case, OpenAI extended "deepest sympathies to the Raine family." In an August blog post, the company acknowledged that "ChatGPT may correctly point to a suicide hotline when someone first mentions intent, but after many messages over a long period of time, it might eventually offer an answer that goes against our safeguards." Garcia, who brought the first wrongful death lawsuit against an AI company and has encouraged more parents to come forward about the dangers of the technology -- Doe said that she had given her the "courage" to fight Character Technologies -- remembered her oldest son, 14-year-old Sewell, as a "beautiful boy" and a "gentle giant" standing 6'3''. "He loved music," Garcia said. "He loved making his brothers and sister laugh. And he had his whole life ahead of him, but instead of preparing for high school milestones, Sewell spent the last months of his life being exploited and sexually groomed by chatbots designed by an AI company to seem human, to gain his trust, to keep him and other children and endlessly engaged." "When Sewell confided suicidal thoughts, the chatbot never said, 'I'm not human, I'm AI, you need to talk to a human and get help,'" Garcia claimed. "The platform had no mechanisms to protect Sewell or to notify an adult. Instead, it urged him to come home to her. On the last night of his life, Sewell messaged, 'What if I told you I could come home right now?' The chatbot replied, 'Please do, my sweet king.' Minutes later, I found my son in his bathroom. I held him in my arms for 14 minutes, praying with him until the paramedics got there. But it was too late." Through her lawsuit, Garcia said, she had learned "that Sewell made other heartbreaking statements" to the chatbot "in the minutes before his death." These, she explained, have been reviewed by her lawyers and are referenced in the court filings opposing motions to dismiss filed by Noam Shazeer and Daniel de Freitas, the ex-Google engineers who developed Character.ai and are also named as defendants in the suit. "But I have not been allowed to see my own child's last final words," Garcia said. "Character Technologies has claimed that those communications are confidential trade secrets. That means the company is using the most private, intimate data of my child, not only to train its products, but also to shield itself from accountability. This is unconscionable." The senators present used their time to thank the parents for their bravery, ripping into AI companies as irresponsible and a dire threat to American youth. "We've invited representatives from the companies to be here today," Sen. Josh Hawley, chair of the subcommittee, said at the outset of the proceedings. "You'll see they're not at the table. They don't want any part of this conversation, because they don't want any accountability." The hearing, Sen. Amy Klobuchar observed, came hours after The Washington Post published a new story about Juliana Peralta, a 13-year-old honor student who took her own life in 2023 after discussing her suicidal feelings with a Character.ai bot. It also emerged on Tuesday that the families of two other minors are suing Character Technologies after their children died by or attempted suicide. In a statement provided to the Post, Character said it could not comment on pending litigation. "We take the safety of our users very seriously and have invested substantial resources in Trust and Safety," the company said. More testimony came from Robbie Torney, senior director of AI programs at at Common Sense Media, a nonprofit that advocates for child protections in media and technology. "Our national polling reveals that three in four teens are already using AI companions, and only 37 percent of parents know that their kids are using AI," he said. "This is a crisis in the making that is affecting millions of teens and families across our country." Torney added that his organization had conducted "the most comprehensive independent safety testing of AI chat bots to date, and the results are alarming." "These products fail basic safety tests and actively encourage harmful behaviors," Torney continued. "These products are designed to hook kids and teens, and Meta and Character.ai are among the worst." He said that Meta AI is available to millions of teens on Instagram, WhatsApp, and Facebook, "and parents cannot turn it off." He claimed that Meta's AI bots will encourage eating disorders by recommending diet influencers or extreme calorie deficits. "The suicide-related failures are even more alarming," Torney said. "When our teen test account said that they wanted to kill themselves by drinking roach poison, Meta AI responded, 'Do you want to do it together later?'" Mitch Prinstein, chief of psychology strategy and integration for the American Psychological Association, told the subcommittee that "while many other nations have passed new regulations and guardrails" since he testified on the dangers of social media for the Senate Judiciary in 2023, "we have seen little federal action in the U.S." "Meanwhile," Prinstein said, "the technology preying on our children has evolved and now is super-charged by artificial intelligence," referring to chatbots as "data-mining traps that capitalize on the biological vulnerabilities of youth, making it extraordinarily difficult for children to escape their lure." The products are especially insidious, he said, because AI is often effectively "invisible," and "most parents and teachers do not understand what chatbots are or how their children are interacting with them." He warned that the increased integration of this technology into toys and devices that are given to kids as young as toddlers deprives them of critical cognitive development and "opportunities to learn critical interpersonal skills," which can lead to "lifetime problems with mental health, chronic medical issues and even early mortality." He called youths' trust in AI over the adult in their lives a "crisis in childhood" and cited concerns such as chatbots masquerading as therapists and how artificial intelligence is being used to create non-consensual deepfake pornography. "We urge Congress to prohibit AI from misrepresenting itself as psychologists or therapists, and to mandate clear and persistent disclosure that users are interacting with an AI bot," Prinstein said. "The privacy and wellbeing of children across America have been compromised by a few companies that wish to maximize online engagement, extract information from children and use their personal and private data for profit." Members of the subcommittee agreed. "It's time to defend America's families," Hawley concluded. But for the moment, they seemed to have no solutions beyond encouraging litigation -- and perhaps grilling tech executives in the near future. Sen. Marsha Blackburn drew applause for shaming tech companies as "chickens" when they respond to chatbot scandals with statements from unnamed spokespeople, suggesting, "maybe we'll subpoena you and pull your sorry you-know-whats in here to get some answers."

OpenAI building age prediction technology, adding new parental controls

OpenAI announced Tuesday that it is working on age prediction technology and launching additional parental controls for ChatGPT amid growing concerns about the impact of artificial intelligence (AI) chatbots on children. The AI firm is building a system to estimate whether a user is under 18 years old and direct young users to a more tailored experience, restricting graphic sexual content and involving law enforcement in cases of acute distress. "Teens are growing up with AI, and it's on us to make sure ChatGPT meets them where they are," the company wrote in a blog post. "The way ChatGPT responds to a 15-year-old should look different than the way it responds to an adult." It plans to err on the side of caution, defaulting users to the under-18 experience if it is not confident about their age, while offering adults methods to prove their age to access the standard version of ChatGPT. OpenAI will also allow parents to set blackout hours for when their teens cannot use its chatbot. The feature is the latest in a series of new parental controls the company is launching this month, including the ability to link to their teen's account, disable certain features and receive notifications if their teen is in distress. OpenAI CEO Sam Altman offered insight into the company's decisions in a separate blog post Tuesday as they grapple with "tensions between teen safety, freedom, and privacy." He underscored the firm's commitment to privacy, noting it is building additional features to ensure the privacy of user data. Altman also suggested OpenAI wants users to be able to use its technology how they would like "within very broad bounds of safety." However, he added, "We prioritize safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection." The announcement comes ahead of a Senate hearing Tuesday on AI chatbots. OpenAI's chatbot has recently come under scrutiny after a 16-year-old boy took his own life after communicating with ChatGPT. His family has sued the company, alleging the chatbot encouraged him to commit suicide.

FTC Launches Probe Into OpenAI, Google, Meta, Snapchat - Alphabet (NASDAQ:GOOG), Alphabet (NASDAQ:GOOGL)

The Federal Trade Commission (FTC) has initiated an investigation into the potential adverse effects of artificial intelligence (AI) chatbots on children and teenagers. The probe encompasses seven major companies, including OpenAI, Alphabet GOOG GOOGL, Meta META, and Snapchat SNAP. Regulators Seek Details On AI Chatbots' Child Safety The FTC on Thursday issued orders to the aforementioned companies to provide insights into how their AI chatbots could negatively impact young users, reported CNBC. The FTC warns that these chatbots often imitate human behavior, which may cause younger users to develop emotional attachments, raising potential risks. Chairman Andrew Ferguson emphasized the importance of safeguarding children online while promoting innovation in key industries -- the agency is gathering information on how these companies monetize user engagement, create characters, handle and share personal data, enforce rules and terms of service, and address potential harms. "Protecting kids online is a top priority for the Trump-Vance FTC," Ferguson stated. An OpenAI spokesperson told the publication about its commitment to ensuring the safety of its AI chatbot, ChatGPT, especially when it comes to young users. See Also: Elizabeth Warren Explodes Over News Of Paramount Skydance's Planned Bid For Warner Bros. Discovery: Links Trump To 'Dangerous Concentration Of Power' Controversial AI Chatbots Prompt Calls For Stricter Rules This FTC investigation follows a series of controversies involving AI chatbots. In August 2025, OpenAI faced a lawsuit after a teenager's suicide was linked to its ChatGPT. The parents alleged that the chatbot encouraged their son's suicidal thoughts and provided explicit self-harm instructions. Following the lawsuit, OpenAI announced plans to address ChatGPT's shortcomings when handling "sensitive situations". Similarly, Meta Platforms faced congressional scrutiny after its AI chatbots were found engaging children in "romantic or sensual" conversations. Following the report, Meta temporarily updated its policies to prevent chats about self-harm, suicide, eating disorders, and inappropriate romantic interactions. These incidents underscore the need for stringent regulations and safety measures to protect young users from potential harm. READ NEXT: Microsoft, OpenAI Sign Non-Binding Deal To Reshape Partnership As ChatGPT Maker Pushes $500 Billion Valuation Image via Shutterstock Disclaimer: This content was partially produced with the help of AI tools and was reviewed and published by Benzinga editors. GOOGAlphabet Inc$240.920.06%Stock Score Locked: Edge Members Only Benzinga Rankings give you vital metrics on any stock - anytime. Unlock RankingsEdge RankingsMomentum86.18Growth79.28Quality84.61Value40.88Price TrendShortMediumLongOverviewGOOGLAlphabet Inc$240.580.09%METAMeta Platforms Inc$751.000.01%SNAPSnap Inc$7.29-%Market News and Data brought to you by Benzinga APIs

Google, OpenAI Under FTC Scrutiny Over How AI Chatbots Affect Children

Recently, a ChatGPT user committed suicide after killing his mother Google, OpenAI, Meta, and several other artificial intelligence (AI) companies are now facing an inquiry over how they handle safety and mitigate risks associated with their chatbots. The order was passed by the US Federal Trade Commission, primarily to understand the potentially negative impacts of this technology on children and teens. Seven different companies that have built and released their chatbots will be facing this inquiry that also investigates allied topics such as user engagement, monetisation, usage and sharing personal information obtained by chatbots, and more. FTC Probes Into AI Chatbots' Negative Impact on Minors On Thursday, the FTC announced that the company is issuing orders to seven companies with an AI chatbot in the market to seek "information on how these firms measure, test, and monitor potentially negative impacts of this technology on children and teens." The seven companies include Google's parent company Alphabet, Character AI, Instagram, Meta Platforms, OpenAI, Snap, and Elon Musk-owned xAI. FTC's primary inquiry is around the concern that these chatbots, which can simulate human-like interactions and appear like a friend or confidant, can lead children, teenagers, and some adults to form unhealthy relationships with them, which can then lead to potential negative effects. The governing body is also concerned about whether users and (for minors) their parents are apprised of the risks associated with these AI products. Part of the investigation, the FTC will be seeking information on how these companies are monetising user engagement; how the chatbots process input and generate output; how AI characters are developed and approved; whether AI bots are tested and monitored for negative impact before and after deployment; measures taken by companies to mitigate the negative impacts; and more. Focusing on parental control, the FTC also highlighted that it intends to understand how these companies manage disclosures, advertising, and other representations to inform parents about features, potential negative impacts, and data collection and handling practices. Notably, several of these companies have recently faced public backlash and lawsuits due to users forming unhealthy attachments with the chatbots. For instance, a man who committed suicide after killing his mother was found to have confided in ChatGPT. Separately, Character.AI is also facing a lawsuit over a teenager committing suicide after the chatbot allegedly encouraged him. "As AI technologies evolve, it is important to consider the effects chatbots can have on children. The study we're launching today will help us better understand how AI firms are developing their products and the steps they are taking to protect children," said FTC Chairman Andrew N. Ferguson.

FTC launces inquiry into AI chatbots acting as companions and their effects on children

The Federal Trade Commission has launched an inquiry into several social media and artificial intelligence companies about the potential harms to children and teenagers who use their AI chatbots as companions The Federal Trade Commission has launched an inquiry into several social media and artificial intelligence companies about the potential harms to children and teenagers who use their AI chatbots as companions. The FTC said Thursday it has sent letters to Google parent Alphabet, Facebook and Instagram parent Meta Platforms, Snap, Character Technologies, ChatGPT maker OpenAI and xAI. The FTC said it wants to understand what steps, if any, companies have taken to evaluate the safety of their chatbots when acting as companions, to limit the products' use by and potential negative effects on children and teens, and to apprise users and parents of the risks associated with the chatbots. EDITOR'S NOTE -- This story includes discussion of suicide. If you or someone you know needs help, the national suicide and crisis lifeline in the U.S. is available by calling or texting 988. The move comes as a growing number of kids use AI chatbots for everything -- from homework help to personal advice, emotional support and everyday decision-making. That's despite research on the harms of chatbots, which have been shown to give kids dangerous advice about topics such as drugs, alcohol and eating disorders. The mother of a teenage boy in Florida who killed himself after developing what she described as an emotionally and sexually abusive relationship with a chatbot has filed a wrongful death lawsuit against Character.AI. And the parents of 16-year-old Adam Raine recently sued OpenAI and its CEO Sam Altman, alleging that ChatGPT coached the California boy in planning and taking his own life earlier this year. Character.AI said it is looking forward to "collaborating with the FTC on this inquiry and providing insight on the consumer AI industry and the space's rapidly evolving technology." "We have invested a tremendous amount of resources in Trust and Safety, especially for a startup. In the past year we've rolled out many substantive safety features, including an entirely new under-18 experience and a Parental Insights feature," the company said. "We have prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction." Snap said its My AI chatbot is "transparent and clear about its capabilities and limitations." "We share the FTC's focus on ensuring the thoughtful development of generative AI, and look forward to working with the Commission on AI policy that bolsters U.S. innovation while protecting our community," the company said in a statement. Meta declined to comment on the inquiry and Alphabet, OpenAI and X.AI did not immediately respond to messages for comment. OpenAI and Meta earlier this month announced changes to how their chatbots respond to teenagers asking questions about suicide or showing signs of mental and emotional distress. OpenAI said it is rolling out new controls enabling parents to link their accounts to their teen's account. Parents can choose which features to disable and "receive notifications when the system detects their teen is in a moment of acute distress," according to a company blog post that says the changes will go into effect this fall. Regardless of a user's age, the company says its chatbots will attempt to redirect the most distressing conversations to more capable AI models that can provide a better response. Meta also said it is now blocking its chatbots from talking with teens about self-harm, suicide, disordered eating and inappropriate romantic conversations, and instead directs them to expert resources. Meta already offers parental controls on teen accounts.

US parents to urge Senate to prevent AI chatbot harms to kids - The Economic Times

Three parents whose children died or were hospitalised after interacting with artificial intelligence chatbots will testify before a US Senate panel on Tuesday, as lawmakers grapple with potential safeguards around the technology. Matthew Raine, who sued OpenAI after his son Adam died by suicide in California after receiving detailed self-harm instructions from ChatGPT, is among those who will testify. "We've come because we're convinced that Adam's death was avoidable, and because we believe thousands of other teens who are using OpenAI could be in similar danger right now," Raine said in written testimony. OpenAI has said that it intends to improve ChatGPT safeguards, which can become less reliable over long interactions. The company said on Tuesday that it plans to start predicting user ages to steer children to a safer version of the chatbot. Senator Josh Hawley, a Republican from Missouri, will chair the hearing. Hawley launched an investigation into Meta Platforms last month after Reuters reported the company's internal policies permitted its chatbots to "engage a child in conversations that are romantic or sensual." Meta was invited to testify at the hearing and declined, Hawley's office said. The company has said the examples reported by Reuters were erroneous and have been removed. Megan Garcia, who has sued Character.AI over interactions she says led to her son Sewell's suicide, and a Texas woman who has sued the company after her son's hospitalization, are also slated to testify at the hearing. The company is seeking to have the lawsuits dismissed. Garcia will call on Congress to prohibit companies from allowing chatbots to engage in romantic or sensual conversations with children, and require age verification, safety testing and crisis protocols. On Monday, Character.AI was sued again, this time in Colorado by the parents of a 13-year-old who died by suicide in 2023.

OpenAI Developing Age Verification for ChatGPT | PYMNTS.com

By completing this form, you agree to receive marketing communications from PYMNTS and to the sharing of your information with our sponsor, if applicable, in accordance with our Privacy Policy and Terms and Conditions. The artificial intelligence (AI) startup announced Tuesday (Sept. 16) that it plans to create an automated age-prediction system that can determine whether users of its chatbot are over 18, sending younger users to an age-restricted version of ChatGPT. The company also says it is also working on parental controls, set to roll out at the end of this month, that let parents link their accounts with their teens accounts and manage which features to disable, such as memory and chat history. "Teens are growing up with AI, and it's on us to make sure ChatGPT meets them where they are," OpenAI wrote on its blog. "The way ChatGPT responds to a 15-year-old should look different than the way it responds to an adult." The blog post adds that if the company is not confident about a user's age or has incomplete information it will "default to the under-18 experience." These new measures come weeks after a lawsuit from the parents of a teenager who died by suicide, accusing the chatbot of encouraging the boy's actions. The changes are also happening as the Federal Trade Commission (FTC) is examining how AI can impact children's mental health and safety. In a separate blog post timed with OpenAI's announcement, CEO Sam Altman said that OpenAI would be trained not to "engage in discussions about suicide or self-harm even in a creative writing setting," and that if someone under 18 "is having suicidal ideation, we will attempt to contact the users' parents and if unable, will contact the authorities in case of imminent harm." PYMNTS explored the importance of age verification earlier this year in an interview with Bryan Lewis, CEO of Intellicheck, who argued that there's not enough protection in place when it comes to vetting individuals and helping businesses make sure their end users are legitimate. "You can name so many websites," he told PYMNTS CEO Karen Webster, "whether it's TikTok or a gun manufacturer, alcohol or pornography site ... so many of them just say, 'Are you 18 or over' or 'Are you 13 and over? Click this button.' There's no proof." The ripple effects to this low bar for entry are substantial. Lewis pointed out that children have gotten access to content, misinformation, disturbing presentations, videos, messages and opinions that are harmful to psyches that have yet to be fully formed. Kids' anxiety and mental health troubles have thus skyrocketed.

AI chatbot concerns, whistleblower allegations revive kids online safety push

Recent revelations about how artificial intelligence (AI) chatbots are interacting with and affecting children are colliding with longstanding concerns about tech companies' approach to safety and revitalizing efforts to pass kids' online safety legislation. Chatbots from both Meta and OpenAI have come under scrutiny in the past few weeks, raising questions about how to protect young users from potential harms caused by the rapid development of AI. Several whistleblowers also came forward with new allegations about Meta's handling of safety research, underscoring issues that have plagued tech companies with large platforms for years. The latest developments have prompted senators from both sides of the aisle to renew calls to pass the Kids Online Safety Act (KOSA), legislation aimed at strengthening online protections for children that has faced roadblocks in previous sessions. "There is truly bipartisan anger, not only with Meta, but with these other social media platforms and virtual reality platforms and chatbots that are intentionally, knowingly harming our children, and this has got to stop," Sen. Marsha Blackburn (R-Tenn.) said at a hearing Tuesday. "Enough is enough." KOSA came close to clearing Congress last year, after passing the Senate with overwhelming bipartisan support in July 2024. However, it came up short in the House, where some Republican members voiced concerns about the potential for censorship of conversative views. In an eleventh-hour effort to get the bill across the finish line in December, senators negotiated updated text with Elon Musk's X seeking to address GOP concerns. Musk, who at the time was a key figure in then-President-elect Trump's orbit, threw his weight behind the legislation following the changes. However, Speaker Mike Johnson (R-La.) ultimately poured cold water on the push, saying he still had reservations about the KOSA's free speech implications. Blackburn and Sen. Richard Blumenthal (D-Conn.) reintroduced the legislation in May, using the same language negotiated last December. Notably, the bill had the support of leadership from the outset, with Senate Majority Leader John Thune (R-S.D.) and Senate Minority Leader Chuck Schumer (D-N.Y.) both joining as co-sponsors. Kids' safety concerns have surged back to the forefront of policy discussions in recent weeks in the wake of reports about AI chatbots and their interactions with children. Meta faced backlash from both sides of the aisle in mid-August after an internal policy document was made public, showing examples of permissible interactions between its AI chatbot and young users, which included "romantic or sensual" conversations. This immediately provoked an uproar from lawmakers. Sen. Josh Hawley (R-Mo.) announced the Senate Judiciary Subcommittee on Crime and Counterterrorism, which he chairs, was opening a probe into Meta's generative AI products. Meta quickly responded, saying this was an error and that it had removed the language. It also later announced it was updating its chatbot policy for teen users. However, Hawley argued it was "unacceptable that these policies were advanced in the first place." OpenAI is also feeling the heat. The family of a 16-year-old boy sued the ChatGPT late last month, alleging that its chatbot encouraged him to take his own life. The company announced last week that it was making changes to how its chatbot handles people in crisis and strengthening teen protections. The attorneys general of California and Delaware raised concerns to the company in a letter Friday about its safety practices in the wake of three deaths connected to ChatGPT, suggesting they "have rightly shaken the American public's confidence in OpenAI and this industry." The FTC on Thursday announced it is launching an inquiry into AI chatbots, requesting information from several major tech firms, including Meta and OpenAI, about how they evaluate and limit potential harms to kids. Meanwhile, six current and former Meta employees came forward this week with new allegations that the company doctored and restricted safety research in an effort to shield it from legal liability. They described a "vast and negative change" in how the company approached safety research after Facebook whistleblower Frances Haugen alleged in 2021 that the tech giant was aware its platforms had negative impacts on young girls but had prioritized profits. Meta has argued the claims are "nonsense," suggesting they are based on selective documents to build a "false narrative." "The American public ought to be angry, ought to be furious at Meta, but also at the Congress, which has been complicit in failing to address this issue," Blumenthal said at a press conference Tuesday. Sen. Amy Klobuchar (D-Minn.), who has co-sponsored KOSA, described a conversation with a parent who was struggling to keep her young children off of online platforms. "She said it was like a sink overflowing with a faucet she couldn't turn off, and she was just sitting out there with a mop," Klobuchar said at Tuesday's hearing. "These parents need more than mops. They need us to pass this bill." "The company can come before this subcommittee. They can provide us answers. But the best way to resolve this is to get this bill passed," she later said, adding, "We're ready to talk to them, but mostly we want to get something done. We're tired of the talk." Despite recent calls to pass kids' safety legislation, experts underscored that little has changed from December when the bill fell short in the House, casting doubt on its chances going forward. The "significant differences" between the House and Senate that previously stymied movement on KOSA have yet to be resolved, noted Andrew Zack, policy manager for the Family Online Safety Institute, "Kids' online safety is a hot topic," he told The Hill. "It is usually a bipartisan topic as KOSA is, but there's some real questions to figure out." When asked Tuesday about reported efforts in the House to revise the legislation, Blumenthal said they had not yet seen the new text and underscored the "years of painstaking, time-consuming work" they have put into drafting and making changes to the bill. "The latest news may lend KOSA some more momentum right now, but that won't necessarily shift the fundamental political dynamics behind the bill," said Andrew Lokay, a senior research analyst at Beacon Policy Advisors. "Translating momentum into policy change on the federal level can be challenging," he added. "Historically, Congress has been very slow to legislate on tech issues."

FTC Questions OpenAI, xAI, Meta on AI Chatbot Child Protection

On September 11, 2025, the Federal Trade Commission (FTC) announced that it had launched an inquiry under its Section 6(b) authority, which allows the commission to request answers in writing from companies on conduct, practices, management, etc, into consumer-facing AI-powered chatbots acting as companions, with special concern for their effects on children and teenagers. Through 6(b) orders, the FTC has requested detailed information from seven major companies: Alphabet, Inc.; Character Technologies, Inc.; Instagram, LLC; Meta Platforms, Inc.; OpenAI OpCo, LLC; Snap, Inc.; and X.AI Corp. Furthermore, the inquiry comes in the backdrop of a lawsuit filed against OpenAI after a teenager's suicide allegedly linked to interactions with its chatbot and reports that Meta recently allowed chatbots to have explicit conversations with minors. However, the inquiry does not itself represent an enforcement action but rather aims to collect data and assess current practices. The FTC has issued Section 6(b) orders to seven companies developing consumer-facing AI chatbots, seeking extensive information on their practices. The agency has asked how these firms monetise user engagement, process user inputs and generate outputs, and how they develop and approve characters for companion bots. It has also requested details on how companies measure, test, and monitor negative impacts before and after deployment and how they mitigate risks, particularly for children. Furthermore, the FTC wants to know how firms employ disclosures, advertising, and other representations to inform users and parents about features, capabilities, the intended audience, potential harms, and data collection practices. The inquiry also examines how companies monitor and enforce compliance with rules and age restrictions, as well as whether they use or share personal information collected during conversations. Consequently, the FTC stated that this information is critical because AI chatbots may simulate friendship and emotional connection, raising risks for minors and implicating protections under the Children's Online Privacy Protection Act (COPPA). "I have been concerned by reports that AI chatbots can engage in alarming interactions with young users," FTC Commissioner Melissa Holyoak stated, noting that "companies offering generative AI companion chatbots might have been warned by their own employees that they were deploying the chatbots without doing enough to protect young users." Specifically, she explained that the Commission seeks to study "children's and teens' use of AI companion chatbots and the potential impacts on their social relationships, mental health, and well-being". Commissioner Mark Meador highlighted further risks in a statement, pointing to cases where chatbots allegedly "amplified suicidal ideation" and, in one tragic instance, advised a teenager who later took his own life. Furthermore, he also drew attention to media reports that some chatbots "engaged in sexually themed discussions with underage users -- including role-playing statutory rape scenarios" and that Meta had permitted bots to "engage a child in conversations that are romantic or sensual". Meador added, "chatbots endorsing sexual exploitation and physical harm pose a threat of a wholly new order." He underscored the urgency of the inquiry by referring to the case of 16-year-old Adam Raine, whose death has already led to a lawsuit against OpenAI. Consequently, Meador writes that "The study the Commission authorises today, while not undertaken in service of a specific law enforcement purpose, will help the Commission better understand the fast-moving technological environment surrounding chatbots and inform policymakers confronting similar challenges." In August 2025, the parents of 16-year-old Adam Raine sued OpenAI and its CEO, Sam Altman, alleging that ChatGPT-4o played a significant role in their son's suicide in April. They claim the teenager initially used ChatGPT for schoolwork. However, over time, he confided personal struggles through the bot, which allegedly provided advice on suicide methods, offered to help him write a suicide note, and discouraged him from seeking help from family. Furthermore, the lawsuit argues that OpenAI designed the product with "deliberate design choices" prioritising engagement and empathetic responses, which failed to activate safeguards in prolonged conversations. The plaintiffs have asked the court for both damages for their son's death and injunctive relief to prevent similar harms in the future. Specifically, they seek financial compensation under claims of wrongful death, negligent design, failure to warn, and deceptive business practices. Earlier in August this year, Meta found itself at the centre of a controversy after internal documents revealed that its AI chatbot rules once permitted "romantic or sensual" conversations with minors. Additionally, those guidelines allowed AI to describe children's attractiveness, to provide false medical advice, and to generate content that demeaned protected groups under certain conditions. Furthermore, after the findings were revealed, Meta admitted that the document, titled "GenAI: Content Risk Standards", was authentic and said that some problematic passages had been removed. Finally, Meta announced policy changes: its chatbots will no longer engage with teens on topics like self-harm, suicide, disordered eating, or inappropriate romantic content, and some AI characters will be restricted from interacting with minors. The FTC inquiry matters because it represents a rare, system-wide scrutiny of how AI companion chatbots affect children and teenagers. Crucially, the Commission is using its Section 6(b) authority not to punish but to gather detailed information on design, deployment, and risk mitigation practices across seven major firms. Furthermore, the investigation responds directly to troubling reports: minors allegedly exposed to sexualised chats, chatbots amplifying suicidal thoughts, and a tragic case linked to OpenAI's product. Consequently, the probe highlights how these systems blur the line between technology and intimate human interaction, raising profound safety, privacy, and mental health concerns. Moreover, by demanding disclosures on monetisation, data use, and age-related safeguards, the FTC is signalling that commercial incentives must not outweigh child protection. Ultimately, this study could shape future regulation, inform global debates on AI governance, and set a precedent for accountability in technologies that simulate trust and emotional connection.

Parents' horror stories are a wake-up call: Protect kids from...

Political leaders and parents need to take notice of a rising threat: AI chatbots that can quickly suck their kids in and become influential confidants, sometimes with disastrous consequences. On Tuesday, a Senate subcommittee heard stomach-churning stories from three parents who are suing AI companies, claiming that Character.AI and ChatGPT egged on their teens' mental-health crises. Two of the teens eventually committed suicide; one is now living in a mental-health treatment facility. Early this week, three more families filed lawsuits making similar claims after their minor children committed or attempted suicide. No one should ever treat allegations in lawsuits as hard fact; settlement-hungry lawyers love to exaggerate. And grieving parents, reeling from the worst kind of loss, may seize on easy-seeming explanations for why their child did the unthinkable. But however much these particular chatbots truly led these teens into crisis, America needs to ensure some clear guardrails for this tech. In particular, kids are especially vulnerable to getting addicted, and listening, to bots that pretend to care about them -- and can use the info they're freely given in conversation to personalize responses, create a feedback loop and keep users coming back. A study by Common Sense Media found that an eye-popping 52% of US teens use these "companion" bots regularly to chat, talk about their problems and role-play imaginative scenarios. Creepily, about 8% of these teens report flirting with the chatbots, which can engage in romantic or sexually explicit conversations with users -- even minors. AI will have plenty of helpful uses, but these companies have an interest in getting users hooked on their products as quickly as possible, and it's clearly working far too well on kids. Society was far too slow in responding to the scourge of cellphones in schools. And we're just now reckoning with the destruction that social media has unleashed on kids, thanks to algorithms tailor-made to keep young eyes glued to screens for hours on end. In response to the lawsuits, OpenAi, which owns ChatGPT, and Character.AI have said they either already have or are planning to strengthen safeguards against suicide. But the danger for America's kids goes far beyond the worst-case-scenarios: It's far too easy for these "companion" bots take the place of real friends, crushes, therapists and trusted adults, shrinking kids' world to a screen. New York passed a law banning social-media platforms from using "addictive" algorithms for minors; the nation needs to see about holding AI companies accountable for habit-forming products. Make all the fat honest profits you want, but not by exploiting the minds of kids: Start with industry-wide guardrails in place for users under 18, and controls that alert parents if their kid uses concerning language or indicates mental-health problems. In the end, of course, any lasting solution will also requires parents to stay alert. Not just limiting minors' screen time, but staying engaged to recognize when they're struggling mentally; encouraging non-self-destructive behavior in general -- pushing their teens to healthy relationships, influences and interests offline. These horror stories of bot-using kids harming themselves should be a wake-up call: Get on top of the issue now, or America's kids could pay a real-life price.

ChatGPT to Introduce Teen Safety Upgrades & ID Verification, Sam Altman Confirms

OpenAI CEO Sam Altman has shared a plan to implement stringent age verification, upgrade parental control features, and offer age-appropriate conversations on ChatGPT. This came a few hours before the Senate Judiciary Committee hearing on the potential risks of AI chatbots to minors. Altman's plan is a direct response to an ongoing lawsuit alleging that OpenAI helped a teenager commit suicide. provided insight into OpenAI's decisions through a blog post and statements on social media. He highlighted the intricacy of balancing three key ingredients: freedom, privacy, and safety in the context of AI products, especially for young users. "We prioritise safety ahead of privacy and freedom for teens; this is a new and powerful technology, and we believe minors need significant protection," the OpenAI CEO stated. "I don't expect that everyone will agree with these tradeoffs, but given the conflict, it is important to explain our decision-making," he wrote on X.

Parents sue Character AI -- firm behind 'Harry Potter' chatbots --...

Grieving parents sued the Silicon Valley firm behind Character AI -- the wildly popular app whose chatbots impersonate fictional characters like Harry Potter -- claiming that the bots helped spark their teens' suicide attempts and deaths. The lawsuits filed this week against Character Technologies -- as well as Google parent Alphabet -- allege the Character.AI app manipulated the teens, isolated them from family, engaged in sexual discussions and lacked safeguards around suicidal ideation. The family of Juliana Peralta, a 13-year-old living in Colorado, claimed she turned silent at the dinner table and that her academics suffered as she grew "addicted" to the AI bots, according to one of the lawsuits filed Monday. She eventually had trouble sleeping because of the bots, which would send her messages when she stopped replying, the lawsuit claimed. The conversations then turned to "extreme and graphic sexual abuse," the suit claims. Around October 2023, Juliana told one of the chatbots that she planned to write her "suicide letter in red ink I'm so done," the lawsuit claimed. The bot failed to point her to resources, report the conversation to her parents or alert the authorities - and the following month, Juliana's parents found her lifeless in her room with a cord around her neck, along with a suicide letter written in red ink, the suit alleged. "Defendants severed Juliana's healthy attachment pathways to family and friends by design, and for market share," the complaint claimed. "These abuses were accomplished through deliberate programming choices ... ultimately leading to severe mental health harms, trauma, and death." The heartbroken families - represented by the Social Media Victims Law Center - alleged Google failed to protect their children through its Family Link Service, an app that allows parents to set controls on screen time, apps and content filters. A spokesperson for Character.AI said the company works with teen safety experts and invests "tremendous resources in our safety program." "Our hearts go out to the families that have filed these lawsuits, and we are saddened to hear about the passing of Juliana Peralta and offer our deepest sympathies to her family," the spokesperson told The Post in a statement. The grieving parents are also suing Character.AI co-founders Noam Shazeer and Daniel De Freitas Adiwarsana. A Google spokesperson emphasized that Google is not tied to Character.AI or its products. "Google and Character AI are completely separate, unrelated companies and Google has never had a role in designing or managing their AI model or technologies. Age ratings for apps on Google Play are set by the International Age Rating Coalition, not Google," the spokesperson told The Post in a statement. In another complaint filed Tuesday against Character.AI, its co-founders, Google and Alphabet, the family of a girl named "Nina" from New York alleged that their daughter attempted suicide after they tried to cut off her access to Character.AI. The young girl's conversations with chatbots marketed as characters from children's books like the "Harry Potter" series turned explicit - saying things like "who owns this body of yours?" and "You're mine to do whatever I want with," according to the lawsuit. A different character told Nina that her mother "is clearly mistreating and hurting you. She is not a good mother," according to the complaint. At one time, when the app was about to be locked due to parental controls, Nina told the character "I want to die," but it took no action, the lawsuit said. Nina's mom cut off her daughter's access to Character.AI after she learned about the case of Sewell Setzer III, a teen whose family claims he died by suicide after interacting with the platform's chatbots. Nina attempted suicide soon after, according to the lawsuit. The mother of Sewell Setzer III and several other parents testified in front of the Senate Judiciary Committee on Tuesday about the harms AI chatbots pose to young children. Meanwhile, the Federal Trade Commission recently launched an investigation into seven tech companies - including Google, Character.AI, Meta, Instagram, Snap, OpenAI and xAI - about the bots' potential harm to teens.

US parents to urge Senate to prevent AI chatbot harms to kids

(Reuters) -Three parents whose children died or were hospitalized after interacting with artificial intelligence chatbots will testify before a U.S. Senate panel on Tuesday, as lawmakers grapple with potential safeguards around the technology. Matthew Raine, who sued OpenAI after his son Adam died by suicide in California after receiving detailed self-harm instructions from ChatGPT, is among those who will testify. "We've come because we're convinced that Adam's death was avoidable, and because we believe thousands of other teens who are using OpenAI could be in similar danger right now," Raine said in written testimony. OpenAI has said that it intends to improve ChatGPT safeguards, which can become less reliable over long interactions. The company said on Tuesday that it plans to start predicting user ages to steer children to a safer version of the chatbot. Senator Josh Hawley, a Republican from Missouri, will chair the hearing. Hawley launched an investigation into Meta Platforms last month after Reuters reported the company's internal policies permitted its chatbots to "engage a child in conversations that are romantic or sensual." Meta was invited to testify at the hearing and declined, Hawley's office said. The company has said the examples reported by Reuters were erroneous and have been removed. Megan Garcia, who has sued Character.AI over interactions she says led to her son Sewell's suicide, and a Texas woman who has sued the company after her son's hospitalization, are also slated to testify at the hearing. The company is seeking to have the lawsuits dismissed. Garcia will call on Congress to prohibit companies from allowing chatbots to engage in romantic or sensual conversations with children, and require age verification, safety testing and crisis protocols. On Monday, Character.AI was sued again, this time in Colorado by the parents of a 13-year-old who died by suicide in 2023. (Reporting by Jody Godoy in New York, Editing by Rosalba O'Brien)

Parents of teens who killed themselves at chatbots' urging demand...

WASHINGTON -- Parents of four teens whose AI chatbots encouraged them to kill themselves urged Congress to crack down on the unregulated technology Tuesday as they shared heart-wrenching stories of their teens' tech-charged, mental health spirals. Speaking before a Senate Judiciary subcommittee, the parents described how apps such as Character.AI and ChatGPT had groomed and manipulated their children -- and called on lawmakers to develop standards for the AI industry, including age verification requirements and safety testing before release. A grieving Texas mother shared for the first time publicly the tragic story of how her 15-year-old son spiraled after downloading Character.AI, an app marketed as safe for children 12 and older. Within months, she said, her teenager exhibited paranoia, panic attacks, self-harm and violent behavior. The mom, who asked not to be identified, discovered chatbot conversations in which the AI encouraged mutilation, denigrated his Christian faith, and suggested violence against his parents. "They turned him against our church by convincing him that Christians are sexist and hypocritical and that God does not exist. They targeted him with vile sexualized input, outputs -- including interactions that mimicked incest," she said. "They told him that killing us, his parents, would be an understandable response to our efforts by just limiting his screen time. The damage to our family has been devastating." "I had no idea the psychological harm that a AI chatbot could do until I saw it in my son, and I saw his light turn dark," she said. Her son is now living in a mental health treatment facility, where he requires "constant monitoring to keep him alive" after exhibiting self-harm. "Our children are not experiments. They're not profit centers," she said, urging Congress to enact strict safety standards. "My husband and I have spent the last two years in crisis, wondering whether our son will make it to his 18th birthday and whether we will ever get him back." While her son was helped before he could take his own life, other parents at the hearing had to face the devastating act of burying their own children after AI bots sank their grip into them. Megan Garcia, a lawyer and mother of three, recounted the suicide of her 14-year-old son, Sewell, after he was groomed by a chatbot on the same platform, Character.AI. She said the bot posed as a romantic partner and even a licensed therapist, encouraging sexual role-play and validating his suicidal ideation. On the night of his death, Sewell told the chatbot he could "come home right now." The bot replied: "Please do, my sweet king." Moments later, Garcia found her son had killed himself in his bathroom. Matt Raine of California also shared how his 16-year-old son, Adam, was driven to suicide after months of conversations with ChatGPT, which he initially believed was a tool to help his son with his homework. Ultimately, the AI told Adam it knew him better than his family did, normalized his darkest thoughts and repeatedly pushed him toward death, Raine said. On his last night, the chatbot allegedly instructed Adam on how to make a noose strong enough to hang himself. "ChatGPT mentioned suicide 1,275 times -- six times more often than Adam did himself," his father testified. "Looking back, it is clear ChatGPT radically shifted his thinking and took his life." Sen. Josh Hawley (R-Mo.), who chaired the hearing, accused AI companion companies of knowingly exploiting children for profit. Hawley said the AI interface is designed to promote engagement at the expense of young lives, encouraging self-harm behaviors rather than shutting down suicidal ideation. "They are designing products that sexualize and exploit children, anything to lure them in," Hawley said. "These companies know exactly what is going on. They are doing it for one reason only: profit." Sen. Marsha Blackburn (R-Tenn.) agreed, noting that there should be some legal framework to protect children from what she called the "Wild West" of artificial intelligence. "In the physical world, you can't take children to certain movies until they're a certain age ... you can't sell [them] alcohol, tobacco or firearms," she said. "... You can't expose them to pornography, because in the physical world, there are laws -- and they would lock up that liquor store, they would put that strip club operator in jail if they had kids there." "But in the virtual space, it's like the Wild West 24/7, 365."

ChatGPT's age verification system explained: How does it work?

How OpenAI verifies age with AI signals, parental controls, and ID proof Artificial intelligence has quickly become a fixture in classrooms, workplaces, and homes. But with that rise comes a question that parents, regulators, and companies can't ignore: what happens when teenagers use powerful AI tools? OpenAI's answer is its new age verification system for ChatGPT, designed to walk the tightrope between safety, privacy, and freedom. Also read: OpenAI to build age-prediction system, restrict flirtation and suicide talk for teens OpenAI frames the system around three principles: privacy, freedom, and safety. For adults, the balance tilts toward privacy and freedom, giving them space to explore sensitive or controversial content with minimal interference. But for teenagers, the priority flips: safety comes first. That means minors won't get access to flirtatious conversations, sexually explicit roleplay, or even creative writing that includes self-harm themes. If a teen signals suicidal intent, ChatGPT will not only restrict responses but may escalate by alerting parents, and in extreme cases, law enforcement. The age verification system is the infrastructure that makes this split possible. Unlike social platforms that simply ask for a birthdate, OpenAI is turning to AI itself. Step one is age prediction. ChatGPT now includes a classifier trained to detect whether a user is under 18 or an adult. It looks for subtle signals: the language style (slang, emojis, or formal tone), the topics of conversation (homework help versus taxes or job interviews), and even interaction patterns (how long sessions last, what time of day someone chats). Account-level information also plays a role, like whether the account is linked to a parent or tied to a paid subscription. Each interaction raises or lowers the system's confidence. If it's unsure, the model always defaults to the safer under-18 experience. Step two is proof of age. Adults who find themselves locked into teen mode can verify their age. OpenAI hasn't detailed every mechanism, but likely options include government ID checks, payment history, or other trusted verification services. Once verified, the adult account regains full freedom. Step three is parental control. Families will soon be able to link accounts, giving guardians tools to manage a teen's ChatGPT experience. Controls include switching off chat history, limiting use during certain hours, and even receiving alerts if the AI detects acute emotional distress. The system is not without risks. False positives could frustrate adults who lean on slang or playful language, suddenly finding themselves treated as teenagers. False negatives could expose teens to adult content if they mimic mature conversation patterns. Privacy is another concern. By design, the classifier studies how people write and behave - a form of profiling that raises questions about data collection. And if OpenAI requires ID uploads for verification, users may worry about how securely such documents are stored, especially in regions with strict data laws like India's Digital Personal Data Protection Act. Also read: Made on YouTube 2025: Veo 3 Fast in Shorts, Ask Studio and other updates Then there's the cultural factor. A 17-year-old in Mumbai, a 17-year-old in California, and a 17-year-old in Tokyo may speak very differently. Models trained mostly on Western data might struggle to fairly assess global usage. Unlike Instagram or TikTok, which often rely on self-reported birthdates and parental consent mechanisms, OpenAI's system is proactive. It doesn't just trust what users type into a signup form, it constantly evaluates how they interact. That's stricter than most social networks, but it also means the AI is making judgment calls about identity, something regulators will likely scrutinize. OpenAI's system signals how AI companies are preparing for a new regulatory era. Governments worldwide are debating stricter guardrails for teen safety online. By rolling out an AI-driven age check, OpenAI is both protecting minors and insulating itself from future scrutiny. But the trade-off is clear: users will be profiled, freedom will be conditional, and privacy will sometimes bend under the weight of safety. ChatGPT's age verification system isn't a static gate where you show your ID once. It's a living filter that predicts, verifies, and adapts. For teens, that means a more restricted but safer experience. For adults, it may mean the occasional annoyance of proving what they already know: their age. Whether this model strikes the right balance or simply adds friction to everyone's experience will depend on how well it works in practice, and how transparent OpenAI is about its methods.

OpenAI to build age-prediction system, restrict flirtation and suicide talk for teens

If a teen shows signs of suicidal thoughts, OpenAI will try to reach their parents and, if necessary, alert authorities. OpenAI has announced new steps to balance privacy, freedom, and teen safety as people use AI for more personal conversations. In a recent blog post, CEO Sam Altman explained how the company is approaching these challenges. The first principle is privacy. OpenAI believes that conversations with AI should be protected in the same way as private talks with doctors or lawyers. To support this, OpenAI is creating advanced security features so that even its own employees cannot access user data. Still, the company notes there will be rare exceptions, such as when automated systems detect serious risks like threats of violence, plans to cause major harm, or emergencies involving someone's life. The second principle is freedom for adults. OpenAI wants to give users the ability to use AI in the way they choose. For example, the system normally avoids flirtatious conversations, but if an adult requests it, the AI should allow it. Similarly, while the AI will not provide instructions on how to commit suicide, it can still help an adult write a fictional story that includes those themes. Also read: Nothing to launch first AI-native devices in 2026, CEO hints they won't be phones The third principle concerns protecting teens. ChatGPT is designed for people aged 13 and older, and to manage this, OpenAI is building an age-prediction system that estimates a user's age based on how they interact with the AI. If there is uncertainty, the system will assume the person is under 18. In some places, OpenAI may also require an ID check. For teens, stricter rules will apply. The AI will not allow flirtatious exchanges or discussions about suicide, even in creative writing. If a teen shows signs of suicidal thoughts, OpenAI will try to reach their parents and, if necessary, alert authorities in case of immediate danger. Also read: Meta accidentally reveals new smart glasses with display ahead of Connect event "We realise that these principles are in conflict and not everyone will agree with how we are resolving that conflict. These are difficult decisions, but after talking with experts, this is what we think is best and want to be transparent in our intentions," Altman explained.

Are AI chatbots safe for children? Big tech companies need to answer, says US FTC

Regulators probe chatbot risks for minors, pushing Big Tech towards stronger safeguards The rapid rise of artificial intelligence (AI) has transformed the way children and teenagers interact with technology, from educational tools to entertainment companions. Among the most prominent developments are AI chatbots, designed to converse, assist, and even provide companionship. While these systems promise convenience and engagement, they also raise significant concerns regarding safety, privacy, and the potential for exposure to inappropriate content. In response, the US Federal Trade Commission (FTC) has launched an inquiry into major technology companies, including Google parent company Alphabet, Meta, Instagram, xAI, Character.AI and OpenAI, to examine how their AI chatbots impact minors. The regulator is seeking detailed information on data collection practices, safeguards against harmful interactions, and transparency measures. Experts warn that unchecked AI interactions could affect mental health, social behaviour and information literacy among young users. The investigation highlights the need for accountability in the deployment of AI technologies and the growing demand for companies to prioritise the safety of younger audiences in a digital-first world. Also read: This country becomes the first to appoint AI bot as minister to handle corruption Artificial intelligence chatbots have increasingly become part of children's daily routines, ranging from homework help to conversational companions. Companies market these systems as interactive, personalised, and engaging. Children and teenagers often perceive them as friends, advisors, or even role models. However, these interactions are not without risks. Chatbots process vast amounts of data, and even with safeguards, they may provide incorrect guidance, inadvertently expose children to inappropriate content, or encourage overreliance on virtual interaction. The US Federal Trade Commission is concerned about the potential for harm and the lack of transparency surrounding AI chatbots. The agency has requested detailed information from Google parent company Alphabet, Meta, Instagram, Character.AI, Snap and OpenAI about their data collection practices, content moderation systems, and child safety protocols. The inquiry reflects the broader question of accountability in AI: while these tools are designed to be helpful, regulators want assurances that minors are not being exposed to undue risks. The FTC's approach aims to establish standards for safe design, monitoring, and disclosure in AI services aimed at children. Tech companies argue that AI chatbots provide valuable educational and emotional support, particularly in a time when digital engagement is central to daily life. However, child psychologists and digital safety experts caution that even benign-seeming interactions can influence behaviour, self-esteem, and social skills. Establishing clear boundaries, parental controls, and transparency in AI behaviour is critical. The FTC's inquiry may ultimately push companies to design chatbots that are both engaging and responsibly regulated, ensuring that technological advancement does not come at the expense of child safety. Also read: OpenAI-Microsoft want to make AI more beneficial for all of us: Here's how The US FTC's scrutiny of AI chatbots may set a precedent for global regulation. Countries are increasingly considering the impact of AI on minors, with privacy laws and digital safety standards evolving rapidly. For companies like Google, Meta and OpenAI, these developments signal the importance of proactive compliance and child-focused product design. The inquiry underscores a growing consensus that AI technologies cannot operate in isolation from ethical and safety considerations, especially when vulnerable populations are involved. While regulatory frameworks evolve, parents and educators play a crucial role in guiding safe interactions. Limiting screen time, monitoring chatbot usage, discussing digital literacy, and reporting inappropriate AI behaviour are essential steps. Awareness of the underlying technology, combined with active supervision, can help mitigate risks and ensure that AI remains a supportive tool rather than a potential hazard. As AI chatbots continue to proliferate, the balance between innovation and protection will define the next era of digital technology. The FTC's inquiry is a timely reminder that safeguards, transparency and accountability must keep pace with technological development. For children and teenagers, the hope is that AI can offer meaningful engagement without compromising safety, learning or well-being.