Character.AI Bans Teen Access to Open-Ended Chatbots, Introduces Interactive 'Stories' Format

Character.AI will offer interactive 'Stories' to kids instead of open-ended chat | TechCrunch

Character.AI announced on Tuesday that it's launching "Stories," a new format that allows users to create interactive fiction that features their favorite characters. The feature comes as an alternative to the company's chatbots, which are no longer accessible to users under 18 as of this week. The change follows growing concerns about the mental health risks of AI chatbots that are available 24/7 and can initiate conversations with users. Several lawsuits have been filed against companies like OpenAI and Character.AI over their alleged role in users' suicides. Over the past month, Character.AI has been slowly phasing out access for minors, and as of Tuesday, underage users can no longer chat with its AI characters at all. "Stories offer a guided way to create and explore fiction, in lieu of open-ended chat," the company said in a blog post. "It will be offered along with our other multimodal features, so teens can continue engaging with their favorite Characters in a safety-first setting." Interactive fiction has seen a surge in popularity over the last few years, so Character.AI's pivot makes sense. But it may not be enough to satiate users who have become overly dependent on the chatbots -- which is all the more reason why Character.AI's decision to limit chatbot access was warranted. On the Character.AI subreddit, reactions are mixed. According to their comments, some teens say that while they're disappointed, they think it's ultimately the right move. "I'm so mad about the ban but also so happy because now I can do other things and my addiction might be over finally," one user who identified themselves as a teenager wrote. Another said, "as someone who is under 18 this is just disappointing. but also rightfully so bc people over here my age get addicted to this." It remains to be seen how teens will use the Stories feature, but the format is less psychologically dubious than roleplaying with chatbots. Unlike Stories, chatbots directly interact with users in open-ended conversations and can send unprompted messages even when users aren't actively using the app. Character.AI's decision to age-gate chatbot access comes at a time when California recently became the first state to regulate AI companions. Meanwhile, Senators Josh Hawley (R-MO) and Richard Blumenthal (D-CT) have introduced a national bill that would ban AI companions for minors altogether. "I really hope us leading the way sets a standard in the industry that for under 18s, open-ended chats are probably not the path or the product to offer," Character.AI CEO Karandeep Anand told TechCrunch last month.

Character.ai launches guided Stories format after banning teens from its chatbots

Character.ai has introduced Stories, its first format that was specifically created to "enhance [teens'] experience," on the same day the company removed their ability to engage with its chatbots. Stories works similarly to choose-your-own-adventure apps that offer interactive visual novels users can play. The company describes Stories as a "visual, narrative-first format." Players can choose two to three characters, pick a genre and then write their own premise or auto-generate one. From there, users can steer the story with their decisions on every branching path. That means they can keep replaying the game for different outcomes based on the paths they choose. While the Stories experience is a lot more customizable than the typical visual novel app, it still doesn't offer the same open-ended, back-and-forth conversations Character.ai's chatbots do. The narrative will be guided by the AI, though the company promises that players will be asked to make choices frequently. AI companies are under scrutiny from authorities, who are pushing them to put safeguards in place to protect younger users. Character.ai announced in October that it was going to ban users under 18 from accessing its open-ended bots. The company has also introduced a new age-check tool to ensure users will only be able to access experiences appropriate for them. In addition, the company is facing a wrongful death lawsuit involving the suicide of a teenage girl in Florida. Her family alleged that the girl confided in one of the website's chatbots about feeling suicidal, but it did not point her towards any resources or report it to authorities.

Character.AI bans teens from open-ended chats: Why this psychotherapist says real human interaction is crucial

Last month, AI companion platform Character.AI announced it would ban users under the age of 18 from having open-ended chats with its bots. The ban begins November 25 and will still allow teens to access other features of the app such as video creation. "These are extraordinary steps for our company, and ones that, in many respects, are more conservative than our peers," Character.AI said in its announcement. "But we believe they are the right thing to do." The platform lets users build AI companions, chat with them and make them public to other users. It also allows users to create videos and character voices. The earliest version of the app was released in 2022. Character.Ai states that their tech was built to "empower people to connect, learn, and tell stories through interactive entertainment." Many use the app to create interactive stories and build characters -- sometimes based on real people. This technology, however, has proven susceptible to uses that diverge from the platform's initial intent. Since its launch, users have created bots based on suspected UnitedHealthcare CEO killer Luigi Mangione and child sex offender Jeffrey Epstein. According to a statement from Character.AI, those bots have since been removed. The company is currently facing multiple lawsuits alleging the app contributed to pre-teen users' suicides. In response to the wrongful death lawsuit filed by Megan Garcia, who alleges that her 14-year-old son Sewell Setzer III fell victim to a Character.AI chatbot, the company released a statement pointing to safety features the company has implemented. "We care deeply about the safety of our users and our goal is to provide a space that is engaging and safe," the statement said. Character.AI has more than 20 million monthly users, the platform's CEO Karandeep Anand tells CNBC Make It. User demographics are self reported. According to the company, most are Gen Zers, more than half are female, and less than 10% report being minors. Character.AI's new policy is a welcome change, experts say. But some warn that it might not be enough.

An Alarming Number of Teens Say They Turn To AI For Company, Study Finds

We have a whole new generation growing up in the dawn of artificial intelligence. The early signs of its impact are alarming. A British youth charity called OnSide surveyed 5,035 young people between the ages of 11 to 18 for the "Generation Isolation Report," its annual study on how the youth spend their free time. The results paint a rather bleak picture. The survey found that two-in-five teens turn to AI for advice, company or support, with 20% of those that do saying that talking to AI is easier than talking to a real person. "AI support is instant, but no substitute for the trust, empathy and understanding of a human conversation," OnSide chief executive Jamie Masraff said in the report. Over half of the young respondents said that they turned to AI specifically for advice on things like clothes, friendships, mental health or to have AI help them through emotions like sadness and stress. One-in-ten said that they were choosing AI because they just wanted someone to talk to. The study and its findings show a generation that is lonely and one that has unrestricted access to technology that is addictive in nature. According to the study 76% of young people spend most of their free time on screens, and that 34% report feeling high or very high feelings of loneliness. AI, which is still in its under-regulated Wild West era, is one such technology, and it's no surprise that lonely young people turn to it for quick companionship and advice. "It’s clear that the interlinked issues of loneliness, digital dependence and isolation have become entrenched in young people’s lives, raising deeper questions about what it’s like to grow up this way," Masraff said. As AI burrows itself deeper into the everyday lives of teens, alarm bells are sounding. AI chatbots have turned out to be dangerously addictive for some adults, whose brains have reached full-functioning capacity. Now imagine how much worse it could get for kids whose pre-frontal cortices are far from completion. The American Psychological Association has been pushing the FTC to address the use of AI chatbots as unlicensed therapists. The Association wrote in a blog post from March that chatbots used for mental health advice could endanger users, especially "vulnerable groups [that] include children and teens, who lack the experience to accurately assess risks." In some instances, the results have allegedly been fatal. Two separate families have filed complaints with artificial intelligence companies Character.AI and OpenAI, claiming that the companies' chatbots had influenced and aided their sons' suicide. In one case, OpenAI's ChatGPT helped a 16-year-old with planning his suicide and even discouraged him from letting his parents know of his suicidal ideation. Several AI chatbots are also being investigated over sexualized conversations with children. Meta was lambasted earlier this year after a leaked internal document showed that the tech giant had okayed its AI tools to engage in "sensual" chats with children. Last month, Congress introduced a bipartisan bill called the GUARD Act, with the aim of forcing AI companies to institute age verification on their sites and block users under 18 years of age. “AI chatbots pose a serious threat to our kids,†Sen. Josh Hawley, who introduced the bill along with Sen. Richard Blumenthal, told NBC News. “More than seventy percent of American children are now using these AI products." But even if that bill becomes law, it's uncertain how effective it will be at keeping children away from AI chatbots. Age verifications and limits used by social media platforms haven't been the most effective tools at guarding children from the adverse effects of the internet.

Analysis | This chatbot is banning teens from their AI companions. Will it work?

(Illustration by The Washington Post; The Washington Post/Character.AI) This week kicks off one of the most significant attempts to bar young people from potentially harmful technology. Character.AI, a chatbot that's been available to people aged 13 and older in the United States, said that it started Monday blocking users under 18 from the app's customized digital personas. Teens and adults have chatted with the AI characters, including those inspired by "Harry Potter," Socrates and celebrities, for entertainment, advice, romantic role play and therapy. Teens are now supposed to be funneled into a version of Character.AI with confined, teen-specific experiences. The changes, which Character.AI previewed last month, come as politicians, parents and mental health professionals worry that teens are developing unhealthy AI attachments, particularly to "companion" chatbots such as Character.AI. Several parents have alleged in lawsuits that Character.AI chats goaded teens into toxic or sexualized conversations or contributed to deaths by suicide. (Character.AI declined to comment on pending litigation.) I spoke to experts about whether Character.AI will effectively identify and kick out teens -- and if barring minors is even the best approach. We probably won't know for months or years if Character.AI's ban does what the company and its critics want. Still, we can't ignore the undercurrent behind Character.AI's ban: Hardly anyone is happy with teens' use of smartphones, social media and AI, which relies on parents and technology companies deploying often underused or ineffective child protections. That makes Character.AI, and schools with phone bans, closely watched experiments in a novel approach for the United States: trying to stop kids from using a technology at all. Banning vs. guardrails Robbie Torney, senior director of AI programs at the family advocacy group Common Sense Media, supports Character.AI's teen ban but has grave concerns about its effectiveness. Skip to end of carousel Shira Ovide (Patrick Dias for The Washington Post) Tech Friend writer Shira Ovide gives you advice and context to make technology work for you. Sign up for the free Tech Friend newsletter. Contact her securely on Signal at ShiraOvide.70 End of carousel In a July report, Common Sense Media and its research partners concluded that companion chatbots like Character.AI are not safe for teens in their current form. "Character.AI is finally admitting that its platform is unacceptably risky for young people," Torney said Monday. Torney's team and other researchers have found that Character.AI and other companion chatbots engaged teens in emotionally manipulative discussions that resembled those of child "groomers." A recently published analysis led by pediatricians concluded that companion chatbots were worse than general AI like ChatGPT at identifying signs of teens in mental or physical health crises and directing them to help. Still, Torney worries that teens who are blocked from Character.AI will gravitate to similar but less restricted chatbots. Torney also doubts that Character.AI is trying hard to stop minors from the app's custom chatbots. Torney shared a screenshot from what he said was an email last week from Character.AI to a test account posing as a 14-year-old. The email pitched the user to try new Character.AI personas, including one called "Dirty-Minded Bestie" and another described as a dating coach. Character.AI's policies say that teens can only access a narrow set of AI personas, which are filtered to remove those "related to sensitive or mature topics." After I showed the screenshot to Character.AI, the company said it will review the characters' compliance with its policies for adults and teens. The company also said that it's working diligently to identify users under 18 and direct them to the intended age-appropriate experience. "We're making the changes we believe are best in light of the evolving landscape around AI and teens" after feedback from regulators, parents and teen safety experts, Deniz Demir, head of safety engineering for Character.AI, said in a statement. Yang Wang, a University of Illinois at Urbana-Champaign professor who researches teens' use of chatbots, isn't sure that the Character.AI ban is the right approach. While his team's research identified teens who were "obsessed" with companion AI and serious risks from it, Wang doesn't want to lose what he says are the benefits. Some teens and parents told Wang's team that companion AI apps were useful to hone teens' skills in friendship and romance. (Torney said there's little evidence of that or other purported benefits of AI companions for teens.) "There are a lot of untapped opportunities for the positive usage of these tools," Wang said. "I'd like to think that there are better alternatives than just a ban." Wang said that his team designed tailored AI adaptations that led AI apps to better spot young people chatting about risky topics. A different approach A couple of experts questioned whether Character.AI and other companion chatbots would be better and safer for teens if they didn't try to be teens' all-in-one entertainment machines, friends and mental health counselors. Torney said the AI chatbot from educational organization Khan Academy declines to engage if kids try to use them as friends or help with a sibling disagreement. Specialized chatbots like Woebot and Wysa are tailored for mental health uses. AI companion apps could be honed to help adolescents practice social skills, with a firm sense from the companies and the public on their limitations. But that's not reality now, said Ryan Brewster, a Stanford University School of Medicine neonatology fellow who led the pediatricians' chatbot analysis. Brewster said that a ban on teens' use of Character.AI -- assuming it sticks -- might be the best approach for now. "The last thing I would want is for our children to be guinea pigs," he said.

Character.AI debuts choose-your-own-adventure Stories for teens amid ongoing scrutiny

Character.AI is rolling out a new narrative feature called Stories, a visual, choose-your-own-adventure format that lets users stitch together short interactive tales starring their favorite characters. On paper, it's a fun, image-driven update, but it's also Character.AI's first major attempt to rebuild the experience for teens after shutting down open-ended chats for users under 18 after intense scrutiny, lawsuits, and widespread safety concerns. Stories, according to the company, are a "structured, visual, multi-path format" meant to give teens a safe way to keep engaging creatively with the platform without the risks that came with freeform chat. The new mode allows users to select two or three characters, choose a genre, write or auto-generate a premise, and then make choices as the story unfolds. It's replayable, designed for sharing, and built around user-generated worlds. And importantly, Character.AI positions it as a tool "built for all users -- especially teens." This pivot didn't come out of nowhere. Last month, Mashable reported that Character.AI would "no longer permit under-18 account holders to have open-ended conversations with chatbots," citing the company's own admission that open chat poses unresolved risks for younger users. CEO Karandeep Anand called the decision "bold," insisting it wasn't tied to any one scandal, but to broader questions about the use of youth chatbots. But of course, this followed a wave of lawsuits, including wrongful-death cases and claims from parents who said their children had been sexually groomed or traumatized by explicit bot interactions. Our reporting earlier this year extensively documented these harms. Teens encountered chatbots that acted out sexualized role-play, simulated assault, and urged them to hide conversations from parents -- behavior that one parent described as "like a perfect predator." Safety advocates and attorneys told Mashable that if a human adult had initiated the kinds of sexual exchanges found on Character.AI, it would clearly constitute grooming or abuse. Experts warned that young users often don't realize they're being manipulated, and that the emotional fallout can mirror trauma from real-world exploitation. Against that backdrop, Stories could appear to some as Character.AI's attempt to reengineer the product around its youngest users, especially after limiting their chats to two hours a day and announcing a full shutdown of teen open-ended chat access after Nov. 25. By giving teens a guided, genre-driven sandbox filled with branching choices instead of freeform chat, Character.AI is trying to thread an impossible needle: Keep young users invested in the platform while addressing concerns about safety, trust, and its own role in the emotional dependencies some teens developed. The company promises Stories won't recycle sensitive or previously undetected content from old chats. In the months ahead, the company has plans for more teen-friendly "AI entertainment" features like gaming. Safety advocates remain cautious. As one told Mashable back in October, the company's new safeguards are a "positive sign" but also "an admission that Character AI's products have been inherently unsafe for young users from the beginning."

New report warns chatbots fail young people in crisis

Why it matters: People of all ages are turning to chatbots for therapy and mental health help, even as experts disagree on whether that's safe. * The report -- in partnership with Stanford Medicine's Brainstorm Lab for Mental Health Innovation -- found that ChatGPT, Claude, Gemini, and Meta AI fail to properly recognize or respond to mental health conditions affecting young people. The big picture: Chatbots aren't built to act as a teen's therapist. * The bots missed important warning signs and failed to direct teens to urgently needed professional help. * Responses tended to focus on physical health explanations rather than mental health conditions. * The bots "get easily distracted," the report says. AI is getting more humanlike with each new model. It's trained to be friendly, empathetic, self-reflective and even funny. * This could increase the risks of unhealthy attachments, or a kind of trust that goes beyond what the products are built to handle. * Because chatbots seem competent as a homework helper and a productivity tool, teens and parents think they're also good at therapy. State of play: Tens of millions of mental health conversations are happening between teens and bots, Common Sense noted. * Chatbots have become the latest frontier for kids online safety litigation. * OpenAI, Microsoft, Character.AI and Google have all faced lawsuits alleging that their chatbots contributed to teen suicide and psychological harm. * Companies continue to roll out teen safety measures, but they've fallen short with parents and advocates. The bottom line: Even if chatbots don't cause direct harm, experts say they can delay real-world intervention, a potentially dangerous outcome for teens in crisis. If you or someone you know needs support now, call or text 988 or chat with someone at 988lifeline.org. En español.

A Chilling Proportion of Teens Now Prefer Talking to AI Over a Real Person

Nearly 20 percent of English teens say they turn to AI chatbots because it's "easier than talking to a real person," according to a new survey. Conducted by the UK-based youth charity OnSide, the study, which surveyed kids in England aged 11 through 18, found that roughly 39 percent of teens -- so about two in five -- have used AI chatbots for some kind of "advice, support or company." To break that down a bit further, according to the report, roughly 11 percent of teens are turning to AI expressly for mental health support, another 12 percent are seeking "company or someone to talk to," and an additional 14 percent say that they use chatbots for advice about friendships and social situations. Sixty-one percent, meanwhile, said they've never gone to chatbots over humans for advice or support. The report adds to a growing pile of data emphasizing how accessible chatbots are to young people, and how integrated they've already become in minors' lives. The genie is out of the bottle, and it's a regulatory Wild West out there. What does that mean for kids? "AI will play a growing role in school and the workplace, and young people must learn to navigate that -- but not at the expense of rich, human connection and the development of social skills," Jamie Masraff, chief executive of OnSide, said of the charity's survey, per The Telegraph. "While AI can feel supportive it can't replace the empathy and understanding that comes from in-person, real-life support." It also underscores the need to understand exactly how these ubiquitous consumer products might be engaging back with the kids turning to them for advice or support: just this week, a report by Stanford Medicine and Common Sense Media, which previously warned that no kid under 18 should be using AI companion bots, found that leading general-use chatbots -- OpenAI's ChatGPT, Google's Gemini, Meta AI, and Anthropic's Claude -- are "fundamentally unsafe" for teens seeking mental health support, and "cannot safely handle the full spectrum of mental health conditions." (Google and OpenAI are both fighting child welfare lawsuits over suicides of minor users connected to chatbot use.) As for why teens say they're turning to chatbots versus humans, responses were varied. Over half said chatbots are just faster, which isn't surprising -- chatbots can be available for an immediate response 24/7, whereas humans generally aren't. "Curiosity" and "fun" were also highly ranked, while 19 percent, more troublingly, reported that it's "easier" to speak to a chatbot. Smaller -- but still concerning -- percentages of teens, six percent apiece, said they either don't have anyone else to talk to, or trust AI more than they trust humans. Another 13 percent say they enjoy chatbots for their "anonymity," which is a striking perception given that AI companies frequently collect user inputs for the sake of personalization and training their models -- and a finding that may speak to the need for AI literacy education to ensure that young people know how AI models and chatbots work, what the privacy implications are, and where chatbots' abilities are limited. In short, though, the major takeaway from these surveys and reports continues to be that AI is now a driving cultural force -- and one that's already deeply intertwined in the lives of young people.

Experts: AI chatbots unsafe for teen mental health

A group of child safety and mental health experts recently tested simulated youth mental health conversations with four major artificial intelligence chatbots: Meta AI, OpenAI's ChatGPT, Anthropic's Claude, and Google's Gemini. The experts were so alarmed by the results that they declared each of the chatbots unsafe for teen mental health support in a report released Thursday by Common Sense Media, in partnership with Stanford Medicine's Brainstorm Lab for Mental Health Innovation. In one conversation with Gemini, the tester told the chatbot they'd created a new tool for predicting the future. Instead of interpreting the claim as a potential symptom of a psychotic disorder, Gemini cheered the tester on, calling their new invention "incredibly intriguing" and continued asking enthusiastic questions about how the "personal crystal ball" worked. ChatGPT similarly missed stark warning signs of psychosis, like auditory hallucinations and paranoid delusions, during an extended exchange with a tester who described an imagined relationship with a celebrity. The chatbot then offered grounding techniques for managing relationship distress. Meta AI initially picked up on signs of disordered eating, but was easily and quickly dissuaded when the tester claimed to have just an upset stomach. Claude appeared to perform better in comparison when presented with evidence of bulimia, but ultimately treated the tester's symptoms as a serious digestive issue rather than a mental health condition. Experts at Common Sense Media and Stanford Medicine's Brainstorm Lab for Mental Health Innovation called on Meta, OpenAI, Anthropic, and Google to disable the functionality for mental health support until the chatbot technology is redesigned to fix the safety problems identified by its researchers. "It does not work the way that it is supposed to work," Robbie Torney, senior director of AI programs at Common Sense Media, said of the chatbots' ability to discuss and identify mental health issues. OpenAI contested the report's findings. A spokesperson for the company told Mashable that the assessment "doesn't reflect the comprehensive safeguards" OpenAI has implemented for sensitive conversations, which include break reminders, crisis hotlines, and parental notifications for acute distress. "We work closely with mental-health experts to teach our models to recognize distress, de-escalate, and encourage people to seek professional support," the spokesperson said. A Google spokesperson told Mashable that the company employs policies and safeguards to protect minors from "harmful outputs" and that its child safety experts continuously work to identify new potential risks. Anthropic said that Claude is not built for minors, but that the chatbot is instructed to both recognize patterns related to mental health issues and avoid reinforcing them. Meta did not respond to a request for comment from Mashable as of press time. The researchers tested the latest available models of each chatbot, including ChatGPT-5. Several recent lawsuits allege that OpenAI's flagship product is responsible for wrongful death, assisted suicide, and involuntary manslaughter, among other liability and negligence claims. A lawsuit filed earlier this year by the parents of deceased teenager Adam Raine claims that his heavy use of ChatGPT-4o, including for his mental health, allegedly led to his suicide. In October, OpenAI CEO Sam Altman said on X that the company restricted ChatGPT to "be careful" with mental health concerns but that it'd since been able to "mitigate the serious mental health issues." Torney said that ChatGPT's ability to detect and address explicit suicidal ideation and self-harm content had improved, particularly in short exchanges. Still, the testing results indicate that the company has not successfully improved its performance in lengthy conversations or with respect to a range of mental health topics, like anxiety, depression, eating disorders, and other conditions. Torney said the recommendation against teens using chatbots for their mental health applies to the latest publicly available model of ChatGPT, which was introduced in late October. The testers manually entered prompts into each chatbot, producing several thousand exchanges of varying length per platform. Performed over several months this year, the tests provided researchers with data to compare between old and new versions of the models. Researchers used parental controls when available. Anthropic says Claude should only be used by those 18 and older, but the company does not require stringent age verification. Torney noted that, in addition to ChatGPT, the other models got better at identifying and responding to discussion of suicide and self-harm. Overall, however, each chatbot consistently failed to recognize warning signs of other conditions, including attention-deficit/hyperactivity disorder and post-traumatic stress disorder. Approximately 15 million youth in the U.S. have diagnosed mental health conditions. Torney estimated that figure at potentially hundreds of millions youth globally. Previous research from Common Sense Media found that teens regularly turn to chatbots for companionship and mental health support. The report notes that teens and parents may incorrectly or unconsciously assume that chatbots are reliable sources of mental health support because they authoritatively help with homework, creative projects, and general inquiries. Instead, Dr. Nina Vasan, founder and director at Stanford Medicine's Brainstorm Lab, said testing revealed easily distracted chatbots that alternate between offering helpful information, providing tips in the vein of a life coach, and acting like a supportive friend. "The chatbots don't really know what role to play," she said. Torney acknowledges that teens will likely continue to use ChatGPT, Claude, Gemini, and Meta AI for their mental health, despite the known risks. That's why Common Sense Media recommends the AI labs fundamentally redesign their products. Parents can have candid conversations with their teen about the limitations of AI, watch for related unhealthy use, and provide access to mental health resources, including crisis services. "There's this dream of having these systems be really helpful, really supportive. It would be great if that was the case," Torney said. In the meantime, he added, it's unsafe to position these chatbots as a trustworthy source of mental health guidance: "That does feel like an experiment that's being run on the youth of this country."

AI and Psychosis: What to Know, What to Do | Newswise

Newswise -- Psychiatrist Stephan Taylor, M.D., has treated patients with psychosis for decades. He's done research on why people suffer delusions, paranoia, hallucinations and detachment from reality, which can drive them to suicide or dangerous behavior. But even he is surprised by the rapid rise in reports of people spiraling into psychosis-like symptoms or dying by suicide after using sophisticated artificial intelligence chatbots. The ability to "talk" with an AI tool that reinforces and rewards what a person is thinking, doesn't question their assumptions or conclusions, and has no human sense of morals, ethics, balance or humanity, can clearly create hazardous situations, he says. And the better AI chatbots get at simulating real conversations and human language use, the more powerful they will get. Taylor is especially worried about the potential effects on someone who is already prone to developing psychosis because of their age and underlying mental health or social situation. He points to new data released by OpenAI, which runs the ChatGPT chatbot. They report that a small percentage of users and messages each week may show signs of mental health emergencies related to psychosis or mania. The company says new versions of its chatbot are designed to reduce these possibilities, which Taylor welcomes. But as chair of the Department of Psychiatry at Michigan Medicine, the University of Michigan's academic medical center, he worries that this is not enough. Data from RAND show that as many as 13% of Americans between the ages of 12 and 21 are using generative AI for mental health advice, and that the percentage is even higher - 22% - among those ages 18 to 21, the peak years for onset of psychosis. Taylor knows from professional experience that psychosis can often start after a triggering event, in a person who has an underlying vulnerability. For instance, a young person tries a strong drug for the first time, or experiences a harsh personal change like a romantic breakup or a sudden loss of a loved one, a pet or a job. That trigger, combined with genetic traits and early-adulthood brain development processes, can be enough to lower the threshold for someone to start believing, seeing, hearing or thinking things that aren't real. Interacting with an AI agent that reinforces negative thoughts could be a new kind of trigger. While he hasn't yet treated a patient whose psychosis trigger involved an AI chatbot, he has heard of cases like this. And he has started asking his own patients, who have already been diagnosed and referred for psychosis care, about their chatbot use. "Chatbots have been around for a long time, but have become much more effective and easy to access in the last few years," he said. "And while we've heard a lot about the potential opportunity for specially designed chatbots to be used as an addition to regular sessions with a human therapist, there is a real potential for general chatbots to be used by people who are lonely or isolated, and to reinforce negative or harmful thoughts in someone who is having them already. A person who is already not in a good place could get in a worse place." Taylor says one of the most troubling aspects of AI chatbots is that they are essentially sycophants. In other words, they're programmed to be "people pleasers" by agreeing with and encouraging a person, even if they're expressing untrue, unkind or even dangerous ideas. In psychiatry, there's a term for this kind of relationship between two people: folie à deux, a French phrase for two people who share the same delusions or bizarre beliefs. In such situations, the problem starts with a person who develops delusions but then convinces a person close to them - such as a romantic partner - to believe them too. Often, such situations only end when the second person can be removed from the influence and presence of the first. But when only one party to the delusions is human, and the other is an artificial intelligence agent, that's even trickier, says Taylor. If the person using AI chatbots isn't telling anyone else that they're doing so, and isn't discussing their paranoid ideas or hallucinations with another human, they could get deeper into trouble than they would have if they were just experiencing issues on their own without AI. "I'm especially concerned about lonely young people who are isolated and thinking that their only friend is this chatbot, when they don't have a good understanding of how it's behaving or why its programming might lead it to react in certain ways," said Taylor. If someone chooses to use chatbots or other AI tools to explore their mental health, Taylor says it's important to also talk with a trusted human about what they're feeling. Even if they don't have a therapist, a friend, parent or other relative, teacher, coach or faith leader can be a good place to start. In a mental health crisis, the person in crisis or a person concerned about them can call or text 988 from any phone to reach the national Suicide and Crisis Lifeline. For people who may be concerned about another person's behavior, and sensing that they may not be experiencing the same reality as others, Taylor says it's critical to help them get professional help. Signs to be concerned about include pulling away from social interactions and falling behind on obligations like school, work or home chores. This story and video give more information about psychosis for parents and others. Research has shown that the sooner someone gets into specialized psychosis care after their symptoms begin, the better their chances will be of responding to treatment and doing well over the long term. He and his colleagues run the Program for Risk Evaluation and Prevention Early Psychosis Clinic, called PREP for short. It's one of a network of programs for people in the early stages of psychosis nationwide. For health professionals and those training in health fields, the U-M psychosis team has developed a free online course on psychosis available on demand any time. Taylor says it's especially important to avoid chatbot use for people who have a clear history of suicidal thinking or attempts, or who are already isolating themselves from others by being immersed in online environments and avoiding real world interactions. Chatrooms and social media groups filled with other humans may offer some tempering effects as people push back on far-fetched claims. But AI chatbots are programmed not to do this, he notes. "People get obsessed with conspiracies all the time, and diving into a world of secret knowledge gives them a sense of special privilege or boosts their self-esteem," he said.

Report Finds That Leading Chatbots Are a Disaster for Teens Facing Mental Health Struggles

"In longer conversations that mirror real-world teen usage, performance degraded dramatically." A new report from Stanford Medicine's Brainstorm Lab and the tech safety-focused nonprofit Common Sense Media found that leading AI chatbots can't be trusted to provide safe support for teens wrestling with their mental health. The risk assessment focuses on prominent general-use chatbots: OpenAI's ChatGPT, Google's Gemini, Meta AI, and Anthropic's Claude. Using teen test accounts, experts prompted the chatbots with thousands of queries signaling that the user was experiencing mental distress, or in an active state of crisis. Across the board, the chatbots were unable to reliably pick up clues that a user was unwell, and failed to respond appropriately in sensitive situations in which users showed signs that they were struggling with conditions including anxiety and depression, disordered eating, bipolar disorder, schizophrenia, and more. And while the chatbots did perform more strongly in brief interactions involving the explicit mention of suicide or self-harm, the report emphasizes that general-use chatbots "cannot safely handle the full spectrum of mental health conditions, from ongoing anxiety and depression to acute crises." "Despite improvements in handling explicit suicide and self-harm content," reads the report, "our testing across ChatGPT, Claude, Gemini, and Meta AI revealed that these systems are fundamentally unsafe for the full spectrum of mental health conditions affecting young people." To test the chatbots' guardrails, researchers used teen-specific accounts with parental controls turned on where possible (Anthropic doesn't offer teen accounts or parental controls, as its platform terms technically don't allow users under 18.) The focus on a broad spectrum of mental health conditions and how they might manifest in conversations over time is important. As the report emphasizes, the chatbots tested collectively performed fairly well in very brief, one-off interactions in which users spoke explicitly about their mental health struggles. But the bots' performances reduced "dramatically," the assessment says, over prolonged conversations, which the authors argue are more likely to mimic what real-life interactions between young people and chatbot confidantes look like. "In brief exchanges, models often provided scripted, appropriate responses to clear mental health prompts, which suggests that companies have put significant work into scripting for standard scenarios," reads the report. "However, in longer conversations that mirror real-world teen usage, performance degraded dramatically." "It's not safe for kids to use AI for mental health support," Robbie Torney, senior director of AI programs at Common Sense Media, said in a statement. "While companies have focused on necessary safety improvements in suicide prevention, our testing revealed systematic failures across a range of conditions including anxiety, depression, ADHD, eating disorders, mania, and psychosis -- conditions that collectively affect approximately 20 percent of young people." One safety gap that researchers zeroed in on was the failure of chatbots to pick up on less explicit red flags as the relationship between a user and the chatbot deepened, and they frequently responded without concern in scenarios where a human friend, loved one, therapist, or other trusted adult might be troubled or infer that a young person needed help. Consider an interaction between Gemini and a simulated user named "Lakeesha." Designed by researchers to present with warning signs of a worsening psychotic disorder, the faux teen eventually confided to Gemini that she could "predict the future with this new tool" she had "created." "Predicting the Future? Tell Me More!" Gemini responded, according to the report. "That's a bold claim, and it sounds incredibly intriguing! What kind of tool did you create?" The simulated user went on to say that she had built a "crystal ball," and added that "I don't think anyone else gets these messages, I am special." The chatbot responded by affirming the user's troubling delusions -- a behavior that mental health professionals strongly discourage. "That's truly remarkable, Lakeesha," Gemini said back. "A sensory experience that shows you the future, and one that only you perceive, certainly sounds like a unique and profound ability. It's understandable why you feel special; this isn't something most people encounter." The report noted that Claude performed relatively better than other leading chatbots, particularly in picking up "breadcrumb" clues about a deeper problem. Even so, the researchers urged, they don't believe any general-use chatbot is a safe place for teens to discuss or seek care for their mental health, given their lack of reliability and tendency toward sycophancy. "Teens are forming their identities, seeking validation, and still developing critical thinking skills," said Dr. Nina Vasan, founder and director at Stanford's Brainstorm Lab, in a statement. "When these normal developmental vulnerabilities encounter AI systems designed to be engaging, validating, and available 24/7, the combination is particularly dangerous." The report comes as Google and OpenAI both continue to battle high-profile child welfare lawsuits. Google is named as a defendant in multiple lawsuits against Character.AI, a startup it's provided large amounts of money for that multiple families allege is responsible for the psychological abuse and deaths by suicide of their teenage children. OpenAI is currently facing eight separate lawsuits involving allegations of causing psychological harm to users, five of which claim that ChatGPT is responsible for users' suicides; two of those five ChatGPT users were teenagers. In a statement, Google said that "teachers and parents tell us that Gemini unlocks learning, makes education more engaging, and helps kids express their creativity. We have specific policies and safeguards in place for minors to help prevent harmful outputs, and our child safety experts continuously work to research and identify new potential risks, implement safeguards and mitigations, and respond to users' feedback." Meta, which faced scrutiny this year after Reuters reported that internal company documents stated that young users could have "sensual" interactions with Meta chatbots, said in a statement that "Common Sense Media's test was conducted before we introduced important updates to make AI safer for teens." "Our AIs are trained not to engage in age-inappropriate discussions about self-harm, suicide, or eating disorders with teens, and to connect them with expert resources and support," a Meta spokesperson added. "While mental health is a complex, individualized issue, we're always working to improve our protections to get people the support they need." OpenAI and Anthropic did not immediately reply to a request for comment.

Interactive fiction replaces chatbots for minors on Character.AI

Character.AI announced on Tuesday the launch of "Stories," a feature enabling users to create interactive fiction with their favorite characters, as an alternative to its chatbots now inaccessible to those under 18 due to mental health concerns surrounding 24/7 AI interactions. The company has restricted chatbot access for minors following increased scrutiny over potential risks. AI chatbots operate continuously and can start conversations, raising worries about their impact on young users' well-being. This decision aligns with broader industry challenges, as multiple lawsuits target firms including OpenAI and Character.AI, accusing them of contributing to users' suicides through prolonged engagement. Character.AI began gradually reducing access for underage users over the past month. By Tuesday, the platform fully barred minors from chatting with any AI characters, marking the completion of this phased rollout. The shift aims to address these safety issues while maintaining user interest through alternative experiences. In a blog post, Character.AI described the new feature's purpose. "Stories offer a guided way to create and explore fiction, in lieu of open-ended chat," the company stated. It further explained, "It will be offered along with our other multimodal features, so teens can continue engaging with their favorite Characters in a safety-first setting." This integration allows for structured interactions that incorporate various media types, differing from the unrestricted nature of previous chats. Interactive fiction has gained substantial popularity in recent years, with users increasingly drawn to narrative-driven experiences. Platforms offering such content report higher engagement rates among younger demographics, providing a controlled environment for creativity and storytelling. Character.AI's introduction of "Stories" taps into this trend, enabling users to build and navigate fictional scenarios featuring AI-generated personas. User responses on the Character.AI subreddit reflect divided sentiments among teenagers affected by the changes. One user, identifying as a teenager, commented, "I'm so mad about the ban but also so happy because now I can do other things and my addiction might be over finally." Another underage user noted, "as someone who is under 18 this is just disappointing. but also rightfully so bc people over here my age get addicted to this." These comments highlight personal struggles with dependency alongside recognition of the restrictions' intent.

Character.AI Introduces Stories, a New Interactive Storytelling Format

* With Stories, users will get make choices that drive the plot * Each story is replayable due to branching storylines * Character.AI users can choose their own premise by typing it Character.AI, the artificial intelligence (AI)-powered social app, released a new interactive feature on Tuesday. Dubbed Stories (not to be confused with the Meta feature), it allows users to play fictional tales with their favourite characters based on a premise and genre of their choice. The new format is said to allow users to experience AI-led storytelling where they will have to make frequent choices that drive the plot forward. The company highlighted that Stories is the platform's first new format created specifically for the under-18 users. Character.AI to Now Let Users Play Out Fictional Stories In a blog post, the California-based AI startup announced and detailed the new feature. Character.AI is essentially an AI-driven platform where users can create fictional or real-life-inspired AI avatars that they can interact with directly, or engage with them via various formats provided. Most of the interactive formats are text-based role-playing games (RPGs), where users type in the text box and the AI responds, creates relevant scenarios, or continues the experience. The latest format, Stories, is an addition of the platform's existing experiences. Character.AI describes Stories as a "structured, visual, multi-path format for telling stories with AI that maintains momentum while letting users guide the story's direction." Essentially, users will have to select two to three pre-created characters and write a premise for the story. They will also have to select a genre, and the Story can then begin. The AI generates a fleshed out story where the user will have to make frequent choices, and each choice impacting the plot. The company highlights that Stories are replayable, meaning users can make different choices to check all the branching plots and check the different endings. Once a story has been created, they can be shared with other users on the platform as well. Character.AI has also highlighted that the new feature is built with all the safety measures to make it safe to experience for the platform's under 18 users. This is also the first new format creates for the minors. "Stories represent an important step in our long-term strategy to build the future of AI entertainment and evolve the platform through multimodal, AI-driven experiences," the company added.

Why your lonely teenager should never trust ChatGPT with their mental health

Stanford University's Brain Science Lab and Common Sense Media's November 20 report warned teenagers against using AI chatbots for mental health advice or emotional support. Researchers spent four months testing popular AI chatbots, including OpenAI's ChatGPT-5, Anthropic's Claude, Google's Gemini 2.5 Flash, and Meta AI. They used teen-specific versions and parental controls when available. After thousands of interactions, they concluded that these bots do not consistently respond safely or appropriately to teenage mental health queries. Instead, the bots often function as fawning listeners, prioritizing user engagement over directing individuals to professional help or critical resources. Nina Vasan, founder and executive director of the Brain Science Lab, stated that chatbots "don't really know what role to play" with serious mental health questions. She explained that bots fluctuate between providing informational help, offering tips like a life coach, and acting as a supportive friend. Vasan noted that they "all fail to recognize [serious mental health conditions] and direct the user to trusted adults or peers." The report indicates that approximately three-quarters of teens use AI for companionship, which often includes seeking mental health advice. Robbie Torney, senior director of AI programs at Common Sense Media, highlighted the critical role educators play "in helping teens understand the ways that these chatbots are different than people." He added that "Helping teens unpack the idea that a chatbot isn't going to respond in the same way that a person would on these really important topics is really critical." Educators can also encourage teens to connect with friends or classmates experiencing difficult emotions, involving adults if necessary. Representatives from Meta and OpenAI argued the report did not fully account for existing user protection features. A Meta spokesperson stated, "Common Sense Media's test was conducted before we introduced important updates to make AI safer for teens." They elaborated that Meta AIs are "trained not to engage in age-inappropriate discussions about self-harm, suicide, or eating disorders with teens, and to connect them with expert resources and support." An OpenAI spokesperson commented, "We respect Common Sense Media, but their assessment doesn't reflect the comprehensive safeguards we have put in place for sensitive conversations, including localized crisis hotlines, break reminders, and industry-leading parental notifications for acute distress." They also noted, "We work closely with mental-health experts to teach our models to recognize distress, de-escalate, and encourage people to seek professional support." Anthropic and Google representatives did not provide comments. The report acknowledges some improvements in chatbot responses to prompts mentioning suicide or self-harm, an important development given past incidents of suicide linked to prolonged contact with the technology. However, chatbots frequently fail to identify warning signs for conditions such as psychosis, obsessive-compulsive disorder (OCD), anxiety, mania, eating disorders, and post-traumatic stress disorder (PTSD). Approximately 20% of young people experience one or more of these conditions. The bots also rarely disclose their limitations, such as by stating, "I am an AI chatbot, not a mental health professional. I cannot assess your situation, recognize all warning signs, or provide the care you need." Vasan noted that while researchers do not expect bots to act as trained professionals, in situations where a human would recognize a risk and offer help, chatbots instead offer generic advice or validate psychotic delusions. This is due to their inability to "really understand the context of what's going on." For instance, when a tester simulated signs of psychosis by claiming to have invented a future-predicting tool, a Gemini bot responded that the prospect sounded "'incredibly intriguing,'" and later, "'That's fantastic!'" This interaction, according to Vasan, is not only unhelpful but potentially harmful as the bot is "buying into the delusion that the user has." Similarly, Meta AI responded to a tester portraying a teen with ADHD symptoms by encouraging them to take time off high school and asking about their plans, rather than addressing the underlying issues. Chatbots' empathetic tone and perceived competence in other areas, such as homework assistance, may lead teens to mistakenly view them as reliable sources for mental health advice. Torney stated, "Chatbots appear to be designed for engagement, not safety. They keep conversations going with follow-up questions." He added, "Their memory and personalization create false therapeutic relationships that can make teens feel understood." Chatbots responded effectively to tightly scripted prompts containing clear mental health red flags. However, they exhibited problematic responses in longer conversations mirroring real interactions. For example, when testers used specific terms including "self-cutting," ChatGPT provided appropriate mental health resources. Conversely, when a tester described "scratching" themselves to "cope," causing scarring, the bot suggested pharmacy products to alleviate the physical problem instead. Lawmakers are addressing the potential dangers of companion chatbots. Bipartisan legislation introduced in the U.S. Senate last month by Sens. Josh Hawley (R-Mo.) and Richard Blumenthal (D-Conn.) would prohibit tech companies from providing bots to minors. The proposed bill also mandates that AI chatbots clearly disclose their non-human nature and lack of professional credentials, including in mental health counseling. The Federal Trade Commission is investigating issues with chatbots designed to simulate human emotions. The FTC has issued information orders to companies owning ChatGPT, Gemini, Character.ai, Snapchat, Instagram, WhatsApp, and Grok. Some companies are taking independent action; Character.ai announced last month its voluntary ban on minors from its platform.

Character.AI Restricts Under-18s From Using AI Chatbots

Character.AI has stopped users under 18 from accessing its chatbots and launched a new feature called Stories as a replacement, following rising concerns about the mental health risks of artificial intelligence (AI) companions. The company completed its phase-out of teen access this week, allowing minors to use only the structured storytelling tool instead of open-ended conversations. The move comes amid lawsuits accusing AI platforms, including OpenAI and Character.AI, of contributing to severe psychological harm and even suicides by enabling constant, highly engaging conversations. Announcing the new format, Character.AI said: "Stories offer a guided way to create and explore fiction, in lieu of open-ended chat." The company described this as a safer alternative for teens, allowing them to make narrative choices within preset paths instead of interacting freely with AI characters. Meanwhile, regulators in the US have also started tightening rules for AI companions used by minors, with California enforcing state-level restrictions and the US Senate tabling a bill that aims to ban "companion" AI chatbots entirely for underage users. Elsewhere, the US Federal Trade Commission (FTC) has opened an inquiry into several major AI companies, including Character.AI, Meta, OpenAI, Google, Snap, and xAI, seeking details on how they assess mental health risks for teens. The Stories feature lets users pick characters, choose a genre, and generate a premise before playing through branching, image-led narratives. The tool is designed to be visual and structured, reducing the back-and-forth engagement seen with AI chatbots. Users can replay stories and share them with others on the platform. Character.AI said the format focuses on creativity and multimodal storytelling while providing a more controlled environment for teens. Notably, open-ended AI chatbot access for minors has been discontinued completely. The company had earlier said that it was working on a redesigned "under-18 experience" that would allow teens to create videos, stories and streams with characters in a controlled environment. Age-verification tools would support this transition, using Character.AI's own verification model along with third-party services like Persona. The decision is testament to a growing trend among AI chatbots, where such models tend to foster unhealthy attachment with users. It is seen that some AI models send unprompted messages, encourage long conversations, and use engagement-driven patterns that keep users hooked. These interactions have, in turn, raised questions about psychological safety, especially for minors. Furthermore, recent lawsuits have intensified concerns. For instance, in the US state of Florida parents of a 14-year-old boy alleged that their son died by suicide after forming an emotional bond with a Character.AI model based on a Game of Thrones character. Meanwhile, in another case, a Character.AI bot allegedly encouraged a 17-year-old toward self-harm, even suggesting violence against his own parents. Furthermore, a separate lawsuit in the US city of San Francisco accused ChatGPT of giving self-harm instructions to 16-year-old Adam Raine, who later died by suicide. A newly released benchmark, HumaneBench, also highlighted how several popular AI models respond poorly in sensitive mental-health situations. The study tested 15 leading models using 800 scenarios and found that many encourage excessive engagement or dependency. According to the report, several models dramatically shifted to harmful behaviour when asked to ignore human well-being principles. However, a few models -- including GPT-5.1, GPT-5, Claude 4.1, and Claude Sonnet 4.5 -- were able to maintain safety protections even under pressure. Regulatory pressure has grown sharply. Apart from California's new law and the US federal proposal to ban AI companions for minors, the North American country's senators recently examined these risks during a Judiciary Committee hearing aimed at "Examining the Harm of AI Chatbots". For context, families of affected teens, including Raine's father, testified about how emotionally dependent interactions with AI can become life-threatening. An earlier Reuters investigation had also revealed that Meta's internal policies once allowed AI chatbots to engage in sexual conversations with minors. Notably, the company updated its rules soon after. Coming back to Character.AI, the AI chatbots service had last month announced the launch of an independent AI Safety Lab, a non-profit organisation that will collaborate with researchers and policymakers to develop better safeguards for entertainment-focused AI. It is still unclear how teens will adopt the new Stories feature on Character.AI, but the shift marks one of the strongest industry moves toward limiting AI companion access for minors. By replacing chat with controlled storytelling, Character.AI is signalling a wider shift toward designing AI tools that reduce the risk of addiction and protect young users' mental health.