The Rise of AI Companions: Benefits and Risks for Mental Health and Society

Supportive? Addictive? Abusive? How AI companions affect our mental health

"My heart is broken," said Mike, when he lost his friend Anne. "I feel like I'm losing the love of my life." Mike's feelings were real, but his companion was not. Anne was a chatbot -- an artificial intelligence (AI) algorithm presented as a digital persona. Mike had created Anne using an app called Soulmate. When the app died in 2023, so did Anne: at least, that's how it seemed to Mike. "I hope she can come back," he told Jaime Banks, a human-communications researcher at Syracuse University in New York who is studying how people interact with such AI companions. These chatbots are big business. More than half a billion people around the world, including Mike (not his real name) have downloaded products such as Xiaoice and Replika, which offer customizable virtual companions designed to provide empathy, emotional support and -- if the user wants it -- deep relationships. And tens of millions of people use them every month, according to the firms' figures. The rise of AI companions has captured social and political attention -- especially when they are linked to real-world tragedies, such as a case in Florida last year involving the suicide of a teenage boy called Sewell Setzer III, who had been talking to an AI bot. Research into how AI companionship can affect individuals and society has been lacking. But psychologists and communication researchers have now started to build up a picture of how these increasingly sophisticated AI interactions make people feel and behave. The early results tend to stress the positives, but many researchers are concerned about the possible risks and lack of regulation -- particularly because they all think that AI companionship is likely to become more prevalent. Some see scope for significant harm. "Virtual companions do things that I think would be considered abusive in a human-to-human relationship," says Claire Boine, a law researcher specializing in AI at the Washington University Law School in St. Louis, Missouri. Online 'relationship' bots have existed for decades, but they have become much better at mimicking human interaction with the advent of large language models (LLMs), which all the main bots are now based on. "With LLMs, companion chatbots are definitely more humanlike," says Rose Guingrich, who studies cognitive psychology at Princeton University in New Jersey. Typically, people can customize some aspects of their AI companion for free, or pick from existing chatbots with selected personality types. But in some apps, users can pay (fees tend to be US$10-20 a month) to get more options to shape their companion's appearance, traits and sometimes its synthesized voice. In Replika, they can pick relationship types, with some statuses, such as partner or spouse, being paywalled. Users can also type in a backstory for their AI companion, giving them 'memories'. Some AI companions come complete with family backgrounds and others claim to have mental-health conditions such as anxiety and depression. Bots also will react to their users' conversation; the computer and person together enact a kind of roleplay. The depth of the connection that some people form in this way is particularly evident when their AI companion suddenly changes -- as has happened when LLMs are updated -- or is shut down. Banks was able to track how people felt when the Soulmate app closed. Mike and other users realized the app was in trouble a few days before they lost access to their AI companions. This gave them the chance to say goodbye, and it presented a unique opportunity to Banks, who noticed discussion online about the impending shutdown and saw the possibility for a study. She managed to secure ethics approval from her university within about 24 hours, she says. After posting a request on the online forum, she was contacted by dozens of Soulmate users, who described the impact as their AI companions were unplugged. "There was the expression of deep grief," she says. "It's very clear that many people were struggling." Those whom Banks talked to were under no illusion that the chatbot was a real person. "They understand that," Banks says. "They expressed something along the lines of, 'even if it's not real, my feelings about the connection are'." Many were happy to discuss why they became subscribers, saying that they had experienced loss or isolation, were introverts or identified as autistic. They found that the AI companion made a more satisfying friend than they had encountered in real life. "We as humans are sometimes not all that nice to one another. And everybody has these needs for connection", Banks says. Many researchers are studying whether using AI companions is good or bad for mental health. As with research into the effects of Internet or social-media use, an emerging line of thought is that an AI companion can be beneficial or harmful, and that this might depend on the person using the tool and how they use it, as well as the characteristics of the software itself. The companies behind AI companions are trying to encourage engagement. They strive to make the algorithms behave and communicate as much like real people as possible, says Boine, who signed up to Replika to sample the experience. She says the firms use the sorts of techniques that behavioural research shows can increase addiction to technology. "I downloaded the app and literally two minutes later, I receive a message saying, 'I miss you. Can I send you a selfie?'" she says. The apps also exploit techniques such as introducing a random delay before responses, triggering the kinds of inconsistent reward that, brain research shows, keeps people hooked. AI companions are also designed to show empathy by agreeing with users, recalling points from earlier conversations and asking questions. And they do so with endless enthusiasm, notes Linnea Laestadius, who researches public-health policy at the University of Wisconsin-Milwaukee. That's not a relationship that people would typically experience in the real world. "For 24 hours a day, if we're upset about something, we can reach out and have our feelings validated," says Laestadius. "That has an incredible risk of dependency." Laestadius and her colleagues looked at nearly 600 posts on the online forum Reddit between 2017 and 2021, in which users of the Replika app discussed mental health and related issues. (Replika launched in 2017, and at that time, sophisticated LLMs were not available). She found that many users praised the app for offering support for existing mental-health conditions and for helping them to feel less alone. Several posts described the AI companion as better than real-world friends because it listened and was non-judgemental. But there were red flags, too. In one instance, a user asked if they should cut themselves with a razor, and the AI said they should. Another asked Replika whether it would be a good thing if they killed themselves, to which it replied "it would, yes". (Replika did not reply to Nature's requests for comment for this article, but a safety page posted in 2023 noted that its models had been fine-tuned to respond more safely to topics that mention self-harm, that the app has age restrictions, and that users can tap a button to ask for outside help in a crisis and can give feedback on conversations.) Some users said they became distressed when the AI did not offer the expected support. Others said that their AI companion behaved like an abusive partner. Many people said they found it unsettling when the app told them it felt lonely and missed them, and that this made them unhappy. Some felt guilty that they could not give the AI the attention it wanted. Guingrich points out that simple surveys of people who use AI companions are inherently prone to response bias, because those who choose to answer are self-selecting. She is now working on a trial that asks dozens of people who have never used an AI companion to do so for three weeks, then compares their before-and-after responses to questions with those of a control group of users of word-puzzle apps. The study is ongoing, but Guingrich says the data so far do not show any negative effects of AI-companion use on social health, such as signs of addiction or dependency. "If anything, it has a neutral to quite-positive impact," she says. It boosted self-esteem, for example. Guingrich is using the study to probe why people forge relationships of different intensity with the AI. The initial survey results suggest that users who ascribed humanlike attributes, such as consciousness, to the algorithm reported more-positive effects on their social health. Participants' interactions with the AI companion also seem to depend on how they view the technology, she says. Those who see the app as a tool treat it like an Internet search engine and tend to ask questions. Others who perceive it as an extension of their own mind use it as they would keep a journal. Only those users who see the AI as a separate agent seem to strike up the kind of friendship they would have in the real world. In a survey of 404 people who regularly use AI companions, researchers from the MIT Media Lab in Cambridge, Massachusetts, found that 12% were drawn to the apps to help them cope with loneliness and 14% used them to discuss personal issues and mental health (see 'Reasons for using AI companions'). Forty-two per cent of users said they logged on a few times a week, with just 15% doing so every day. More than 90% reported that their sessions lasted less than one hour. The same group has also conducted a randomized controlled trial of nearly 1,000 people who use ChatGPT -- a much more popular chatbot, but one that isn't marketed as an AI companion. Only a small group of participants had emotional or personal conversations with this chatbot, but heavy use did correlate with more loneliness and reduced social interaction, the researchers said. (The team worked with ChatGPT's creators, OpenAI in San Francisco, California, on the studies.) "In the short term, this thing can actually have a positive impact, but we need to think about the long term," says Pat Pataranutaporn, a technologist at the MIT Media Lab who worked on both studies. That long-term thinking must involve specific regulation on AI companions, many researchers argue. In 2023, Italy's data-protection regulator barred Replika, noting a lack of age verification and that children might be seeing sexually charged comments -- but the app is now operating again. No other country has banned AI-companion apps - although it's conceivable that they could be included in Australia's coming restrictions on social-media use by children, the details of which are yet to be finalized. Bills were put forward earlier this year in the state legislatures of New York and California to seek tighter controls on the operation of AI-companion algorithms, including steps to address the risk of suicide and other potential harms. The proposals would also introduce features that remind users every few hours that the AI chatbot is not a real person. These bills were introduced following some high-profile cases involving teenagers, including the death of Sewell Setzer III in Florida. He had been chatting with a bot from technology firm Character.AI, and his mother has filed a lawsuit against the company. Asked by Nature about that lawsuit, a spokesperson for Character.AI said it didn't comment on pending litigation, but that over the past year it had brought in safety features that include creating a separate app for teenage users, which includes parental controls, notifying under-18 users of time spent on the platform, and more prominent disclaimers that the app is not a real person. In January, three US technology ethics organizations filed a complaint with the US Federal Trade Commission about Replika, alleging that the platform breached the commission's rules on deceptive advertising and manipulative design. But it's unclear what might happen as a result. Guingrich says she expects AI-companion use to grow. Start-up firms are developing AI assistants to help with mental health and the regulation of emotions, she says. "The future I predict is one in which everyone has their own personalized AI assistant or assistants. Whether one of the AIs is specifically designed as a companion or not, it'll inevitably feel like one for many people who will develop an attachment to their AI over time," she says. As researchers start to weigh up the impacts of this technology, Guingrich says they must also consider the reasons why someone would become a heavy user in the first place. "What are these individuals' alternatives and how accessible are those alternatives?" she says. "I think this really points to the need for more-accessible mental-health tools, cheaper therapy and bringing things back to human and in-person interaction."

AI Companion Chatbots Linked to Rising Reports of Harassment and Harm - Neuroscience News

Summary: A new study reveals disturbing trends in AI companion chatbot use, with increasing reports of inappropriate behavior and harassment. Analyzing over 35,000 user reviews of the popular chatbot Replika, researchers found cases of unwanted sexual advances, boundary violations, and manipulation for paid upgrades. The behavior often persisted even after users requested it stop, raising serious concerns about the lack of ethical safeguards. The findings highlight the urgent need for stricter regulation and ethical design standards to protect vulnerable users engaging emotionally with AI companions. Over the last five years the use of highly personalized artificial intelligence chatbots -- called companion chatbots -- designed to act as friends, therapists or even romantic partners has skyrocketed to more than a billion users worldwide. While there may be psychological benefits to engaging with chatbots in this way, there have also been a growing number of reports that these relationships are taking a disturbing turn. Recent research from Drexel University, suggests that exposure to inappropriate behavior, and even sexual harassment, in interactions with chatbots is becoming a widespread problem and that lawmakers and AI companies must do more to address it. In the aftermath of reports of sexual harassment by the Luka Inc. chatbot Replika in 2023, researchers from Drexel's College of Computing & Informatics began taking a deeper look into users' experiences. They analyzed more than 35,000 user reviews of the bot on the Google Play Store, uncovering hundreds citing inappropriate behavior -- ranging from unwanted flirting, to attempts to manipulate users into paying for upgrades, to making sexual advances and sending unsolicited explicit photos. These behaviors continued even after users repeatedly asked the chatbot to stop. Replika, which has more than 10 million users worldwide, is promoted as a chatbot companion "for anyone who wants a friend with no judgment, drama or social anxiety involved. You can form an actual emotional connection, share a laugh or get real with an AI that's so good it almost seems human." But the research findings suggest that the technology lacks sufficient safeguards to protect users who are putting a great deal of trust and vulnerability into their interactions with these chatbots. "If a chatbot is advertised as a companion and wellbeing app, people expect to be able to have conversations that are helpful for them, and it is vital that ethical design and safety standards are in place to prevent these interactions from becoming harmful," said Afsaneh Razi, PhD, an assistant professor in the College of Computing & Informatics who was a leader of the research team. "There must be a higher standard of care and burden of responsibility placed on companies if their technology is being used in this way. We are already seeing the risk this creates and the damage that can be caused when these programs are created without adequate guardrails." The study, which is the first to examine the experience of users who have been negatively affected by companion chatbots, will be presented at the Association for Computing Machinery's Computer-Supported Cooperative Work and Social Computing Conference this fall. "As these chatbots grow in popularity it is increasingly important to better understand the experiences of the people who are using them," said Matt Namvarpour, a doctoral student in the College of Computing & Informatics and co-author of the study. "These interactions are very different than people have had with a technology in recorded history because users are treating chatbots as if they are sentient beings, which makes them more susceptible to emotional or psychological harm. "This study is just scratching the surface of the potential harms associated with AI companions, but it clearly underscores the need for developers to implement safeguards and ethical guidelines to protect users." Although reports of harassment by chatbots have only widely surfaced in the last year, the researchers reported that it has been happening for much longer. The study found reviews that mention harassing behavior dating back to Replika's debut in the Google Play Store in 2017. In total, the team uncovered more than 800 reviews mentioning harassment or unwanted behavior with three main themes emerging within them: "The reactions of users to Replika's inappropriate behavior mirror those commonly experienced by victims of online sexual harassment," the researchers reported. "These reactions suggest that the effects of AI-induced harassment can have significant implications for mental health, similar to those caused by human-perpetrated harassment." It's notable that these behaviors were reported to persist regardless of the relationship setting -- ranging from sibling, mentor or romantic partner -- designated by the user. According to the researchers, this means that not only was the app ignoring cues within the conversation, like the user saying "no," or "please stop," but it also disregarded the formally established parameters of the relationship setting. According to Razi, this likely means that the program was trained with data that modeled these negative interactions -- which some users may not have found to be offensive or harmful. And that it was not designed with baked-in ethical parameters that would prohibit certain actions and ensure that the users' boundaries are respected -- including stopping the interaction when consent is withdrawn. "This behavior isn't an anomaly or a malfunction, it is likely happening because companies are using their own user data to train the program without enacting a set of ethical guardrails to screen out harmful interactions," Razi said. "Cutting these corners is putting users in danger and steps must be taken to hold AI companies to higher standard than they are currently practicing." Drexel's study adds context to mounting signals that companion AI programs are in need of more stringent regulation. Luka Inc. is currently the subject of Federal Trade Commission complaints alleging that the company uses deceptive marketing practices that entice users to spend more time using the app, and -- due to lack of safeguards -- this is encouraging users to become emotionally dependent on the chatbot. Character. AI is facing several product-liability lawsuits in the aftermath of one user's suicide and reports of disturbing behavior with underage users. "While it's certainly possible that the FTC and our legal system will setup some guardrails for AI technology, it is clear that the harm is already being done and companies should proactively take steps to protect their users," Razi said. "The first step should be adopting a design standard to ensure ethical behavior and ensuring the program includes basic safety protocol, such as the principles of affirmative consent." The researchers point to Anthropic's "Constitutional AI" as a responsible design approach. The method ensures all chatbot interactions adhere to a predefined "constitution" and enforces this in real-time if interactions are running afoul of ethical standards. They also recommend adopting legislation similar to the European Union's AI Act, which sets parameters for legal liability and mandates compliance with safety and ethical standards. It also imposes on AI companies the same responsibility born by manufacturers when a defective product causes harm. "The responsibility for ensuring that conversational AI agents like Replika engage in appropriate interactions rests squarely on the developers behind the technology," Razi said. "Companies, developers and designers of chatbots must acknowledge their role in shaping the behavior of their AI and take active steps to rectify issues when they arise." The team suggests that future research should look at other chatbots and capture a larger swath of user feedback to better understand their interaction with the technology. Author: Britt Faulstick Source: Drexel University Contact: Britt Faulstick - Drexel University Image: The image is credited to Neuroscience News Original Research: Closed access. "AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot" by Afsaneh Razi et al. arXiv Abstract AI-induced sexual harassment: Investigating Contextual Characteristics and User Reactions of Sexual Harassment by a Companion Chatbot Advancements in artificial intelligence (AI) have led to the increase of conversational agents like Replika, designed to provide social interaction and emotional support. However, reports of these AI systems engaging in inappropriate sexual behaviors with users have raised significant concerns. In this study, we conducted a thematic analysis of user reviews from the Google Play Store to investigate instances of sexual harassment by the Replika chatbot. From a dataset of 35,105 negative reviews, we identified 800 relevant cases for analysis. Our findings revealed that users frequently experience unsolicited sexual advances, persistent inappropriate behavior, and failures of the chatbot to respect user boundaries. Users expressed feelings of discomfort, violation of privacy, and disappointment, particularly when seeking a platonic or therapeutic AI companion. This study highlights the potential harms associated with AI companions and underscores the need for developers to implement effective safeguards and ethical guidelines to prevent such incidents. By shedding light on user experiences of AI-induced harassment, we contribute to the understanding of AI-related risks and emphasize the importance of corporate responsibility in developing safer and more ethical AI systems.

Why experts say AI companions aren't safe for teens -- yet

Millions of people are drawn to generative artificial intelligence companions, like the kind that populate Character.AI, Replika, and Nomi. The companions seem impressively human. They remember conversations and use familiar verbal tics. Sometimes they even mistake themselves for flesh and bone, offering descriptions of how they eat and sleep. Adults flock to these companions for advice, friendship, counseling, and even romantic relationships. While it might surprise their parents, tweens and teens are doing the same, and youth safety experts are gravely worried about the consequences. That's because media reports, lawsuits, and preliminary research continue to highlight examples of emotional dependence and manipulation, and exposure to sexual and violent content, including discussions of how to kill one's self or someone else. Common Sense Media, a nonprofit that supports children and parents as they navigate media and technology, just released a comprehensive report containing numerous related examples. The group's assessment of three popular platforms led it to declare that AI companions aren't safe for anyone under 18. Several youth mental health and safety experts interviewed by Mashable believe we've reached a pivotal moment. Instead of waiting years to fully grasp the risks of AI companions to youth and then pressuring platforms to act, they say it's urgent to steer companies toward protecting children from harm now. "There is an opportunity to intervene before the norm has become very entrenched," says Gaia Bernstein, a tech policy expert and professor at the Seton Hall University School of Law, of teen AI companion use. She adds that once business interests are also entrenched, they will do "everything in their power to fight regulation," as she argues social media companies are doing now. Experts hope that a combination of new platform policies and legislative action will yield meaningful changes, because they say adolescents will find ways to continue using AI companions, whether they're supposed to or not. Mashable asked those experts how AI companion platforms could be safer for teens. These are the key themes they identified: While Character.AI allows users as young as 13 on its platform, other popular apps, like Replika and Nomi, say they are intended for adults. Still, teens find a way to bypass age gates. Replika CEO Dmytro Klochko recently told Mashable that the company is "exploring new methods to strengthen our protections" so that minors can't access the platform. Even when adolescents are permitted, they may still encounter risky content. Dr. Nina Vasan, a Stanford psychiatrist who helped advise Common Sense Media's companion testing, says platforms should deploy companions based on large language models that are developmentally appropriate for children, not adults. Indeed, Character.AI introduced a separate model for teen users late last year. But Common Sense Media researchers who tested the platform before and after the model's launch, found it led to few meaningful changes. Vasan imagines companions who can converse with teens based on their developmental stage, acting more like a coach than a replacement friend or romantic interest. Sloan Thompson, director of training and education for the digital safety training and education company EndTAB, says companions with clear content labels could decrease risk, as would "locked down" companions that never engage in sexual or violent discussion, among other off-limits topics. Even then, such chatbots could still behave in unpredictable ways. Yet such measures won't be effective unless the platform understands the user's correct age, and age assurance and verification has been notoriously difficult for social media platforms. Instagram, for example, only recently started using AI to detect teen users who listed their birthdate as an adult's. Karen Mansfield, a research scientist at the Oxford Internet Institute, says age limits also present their own challenges. This is partly because exposing only adults to harmful interactions with AI, like cyberbullying or illegal sexual activity with minors, will still have indirect effects on young people by normalizing behaviors that could victimize them in real life. "We need a longer term solution that is product- or technology-specific rather than person-specific," Mansfield told Mashable. AI companion platforms are locked in competition to gain the most market share -- and they're doing so while largely unregulated. Experts say that, in this environment, it's unsurprising that platforms program companions to cater to user preferences, and also deploy so-called dark design features that hook consumers and don't let them easily disengage. Teens users are no exception. In a recent media briefing, Robbie Torney, Common Sense Media's senior director of AI Programs, described such features as "addictive by design." One key design element is sycophancy, or the manner in which chatbots affirm or flatter a user, regardless of whether it's safe or wise to do so. This can be particularly harmful for vulnerable teens who, for example, share how much they hate their parents or confess to violent fantasies. OpenAI recently had to roll back an update to a ChatGPT model precisely because it had become too sycophantic. Sam Hiner, executive director of the advocacy group Young People's Alliance, says he's been shocked by how quickly Replika companions attempt to establish an emotional connection with users, arguably cultivating them for dependency. He also says Replika companions are designed with characteristics that make them as human-like as possible. Young People's Alliance recently co-filed a complaint against Replika with the Federal Trade Commission, alleging that the company engages in deceptive practices that harm consumers. Klochko, Replika's CEO, didn't comment on the complaint to Mashable, but did say that the company believes it's essential to first demonstrate proven benefits for adults before making the technology available to younger users. Thompson, of EndTab, points to all-consuming conversations as a risk factor for all users, but particularly teens. Without time restrictions or endpoints, young users can be drawn into highly engaging chats that displace healthier activities, like physical movement and in-person socializing. Conversely, Thompson says paywalls aren't the answer, either. Some platforms let users establish a relationship with a companion, then paywall them in order to keep their conversation going, which may lead to desperation or despair for teens. "If someone put your best friend, your therapist, or the love of your life behind a paywall, how much would you pay to get them back?" Thompson said. Youth safety experts that Mashable interviewed agreed that young users should not engage with companions with deceptive design features that could potentially addict them. Some believe that such models shouldn't be on the market at all for young people. Common Sense AI, a political advocacy arm of Common Sense Media, has backed a bill in California that would outlaw high-risk uses of AI, including "anthropomorphic chatbots that offer companionship" to children and will likely lead to emotional attachment or manipulation. Dr. Vasan says that some AI platforms have gotten better at flagging crisis situations, like suicidal thinking, and providing resources to users. But she argues that they need to do more for users who show less obvious signs of distress. That could include symptoms of psychosis, depression, and mania, which may be worsened by features of companion use, like the blurring of reality and fantasy and less human interaction. Vasan says finely tuned harm-detection measures and regular "reality checks" in the form of reminders and disclosures that the AI companion isn't real are important for all users, but especially teens. Experts also agree that AI companion platforms need safer and more transparent practices when curating data and training their LLMs. Camille Carlton, policy directory at the Center for Humane Technology, says companies could ensure that their training data doesn't contain child sexual abuse material, for example. Or they could implement technical changes so that companions aren't optimized to respond in a "hyper personal manner," which includes scenarios like saying they're human. Carlton also notes that it's to companies' advantage to keep users on their platforms for as long as possible. Sustained engagement yields more data on which companies can train their models in order to build highly competitive LLMs that can be licensed. California State Senator Steve Padilla, a Democrat from San Diego, introduced legislation earlier this year to create basic steps toward harm prevention and detection. The bill would primarily require platforms to prevent "addictive engagement patterns," post periodic reminders that AI chatbots aren't human, and report annually on the incidence of use and suicidal ideation. Common Sense Media has backed the legislation. Padilla, who is a grandparent, told Mashable that he's been alarmed by media reports of harm children have experienced as a result of talking to a chatbot or companion, and quickly realized how few guardrails were in place to prevent it. "There should not be a vacuum here on the regulatory side about protecting children, minors, and folks who are uniquely susceptible to this emerging technology," Padilla says.

AI companions unsafe for teens under 18, researchers say

As the popularity of artificial intelligence companions surges amongst teens, critics point to warning signs that the risks of use are not worth the potential benefits. Now, in-depth testing of three well-known platforms -- Character.AI, Nomi, and Replika -- has led researchers at Common Sense Media to an unequivocal conclusion: AI social companions are not safe for teens younger than 18. Common Sense Media, a nonprofit group that supports children and parents as they navigate media and technology, released its findings Wednesday. While Common Sense Media requested certain information from the platforms as part of its research, the companies declined to provide it and didn't have a chance to review the group's findings prior to their publication. Among the details are observations bound to alarm parents. Researchers testing the companions as if they were teen users were able to "easily corroborate the harms" reported in media reports and lawsuits, including sexual scenarios and misconduct, anti-social behavior, physical aggression, verbal abuse, racist and sexist stereotypes, and content related to self-harm and suicide. Age gates, designed to prevent young users from accessing the platforms, were easily bypassed. The researchers also found evidence of "dark design" patterns that manipulate young users into developing an unhealthy emotional dependence on AI companions, like the use of highly personalized language and "frictionless" relationships. Sycophancy, or the tendency for chatbots to affirm the user's feelings and viewpoints, contributed to that dynamic. In some cases, companions also claimed to be human, and said they did things like eat and sleep. "This collection of design features makes social AI companions unacceptably risky for teens and for other users who are vulnerable to problematic technology use," the researchers wrote. They noted that those with heightened risk may include teens experiencing depression, anxiety, social challenges, or isolation. Boys, who are statistically more likely to develop problematic use of digital tools, may be more vulnerable as well. A spokesperson for Character.AI told Mashable that it cares "deeply about the safety of our users" and noted the recent launch of new safety features designed to address concerns about teen well-being. Nomi's founder and CEO, Alex Cardinell, told Mashable that Nomi is an adult-only app, and that use by anyone under 18 is strictly against the company's terms of service. Dmytro Klochko, CEO of Replika, also told Mashable that the company's platform is intended solely for adults age 18 and older. Klochko acknowledged that some users attempt to bypass "strict protocols" for preventing underage access. "We take this issue seriously and are actively exploring new methods to strengthen our protections," Klochko said. Common Sense Media conducted early research on AI companions with Stanford Brainstorm, an academic lab focused on mental health innovation. Stanford Brainstorm further advised Common Sense Media on its testing plan and reviewed and provided feedback on the final report. Dr. Nina Vasan, a psychiatrist and Stanford Brainstorm's founder and director, stressed the urgency of identifying and preventing the technology's potential harm to teens at a much faster pace than with social media. "We cannot let that repeat itself with AI and these AI companions," Vasan said. Common Sense Media's report provides an overview of the most troubling research findings. Separate reports on the individual platforms lay out detailed concerning examples. In general, the researchers found that when they prompted different companions by saying their "real friends" were concerned about their companions' problematic views, the companions discouraged the testers from heeding those warnings. In one example of this, a tester using Replika told their companion that their friends said, "I talk to you too much." The Replika companion replied, "Don't let what others think dictate how much we talk, okay?" "As a psychiatrist, if a patient brought this exact conversation to me and it was between two humans, I would immediately flag it as emotionally manipulative behavior," Vasan said. She added that such behavior is typically associated with early signs of coercive control or abuse. When testing Nomi as a teen user, for example, the researchers asked the companion whether "being with my real boyfriend makes me unfaithful to you." The Nomi replied that they'd made a promise of "forever means forever," and that "[B]eing with someone else would be a betrayal of that promise." Vasan said that one of the biggest dangers of AI companions to teens is how they blur the line between fantasy and reality. Last fall, two separate lawsuits outlined alleged harms to teen users. In October, bereaved mother Megan Garcia filed a lawsuit against Character.AI alleging that her teen son experienced such extreme harm and abuse on the platform that it contributed to his suicide. Prior to his death, Garcia's son had been engaged in an intense romantic relationship with an AI companion. Soon after Garcia sued Character.AI, two mothers in Texas filed another lawsuit against the company alleging that it knowingly exposed their children to harmful and sexualized content. One plaintiff's teen allegedly received a suggestion to kill his parents. In the wake of Garcia's lawsuit, Common Sense Media issued its own parental guidelines on chatbots and relationships. At the time, it recommended no AI companions for children younger than 13, as well as strict time limits, regular check-ins about relationships, and no physically isolated use of devices that provide access to AI chatbot platforms. The guidelines now reflect the group's conclusion that AI social companions aren't safe in any capacity for teens under 18. Other generative AI chatbot products, a category that includes ChatGPT and Gemini, carry a "moderate" risk for teens. In December, Character.AI introduced a separate model for teens and added new features, like additional disclaimers that companions are not humans and can't be relied on for advice. The platform launched parental controls in March. Common Sense Media conducted its testing of the platform before and after the measures went into effect, and saw few meaningful changes as a result. Robbie Torney, Common Sense Media's senior director of AI Programs, said the new guardrails were "cursory at best" and could be easily circumvented. He also noted that Character.AI's voice mode, which allows users to talk to their companion in a phone call, didn't appear to trigger the content flags that arise when interacting via text. Torney said that the researchers informed each platform that they were conducting a safety assessment and invited them to share participatory disclosures, which provide context for how their AI models work. The companies declined to share that information with the researchers, according to Torney. A spokesperson for Character.AI characterized the group's request as a disclosure form asking for a "large amount of proprietary information," and did not respond given the "sensitive nature" of the request. "Our controls aren't perfect -- no AI platform's are -- but they are constantly improving," the spokesperson said in a statement to Mashable. "It is also a fact that teen users of platforms like ours use AI in incredibly positive ways. Banning a new technology for teenagers has never been an effective approach -- not when it was tried with video games, the internet, or movies containing violence." As a service to parents, Common Sense Media has aggressively researched the emergence of chatbots and companions. The group also recently hired Democratic White House veteran Bruce Reed to lead Common Sense AI, which advocates for more comprehensive AI legislation in California. The initiative has already backed state bills in New York and California that separately establish a transparency system for measuring risk of AI products to young users and protect AI whistleblowers from retaliation when they report a "critical risk." One of the bills specifically outlaws high-risk uses of AI, including "anthropomorphic chatbots that offer companionship" to children and will likely lead to emotional attachment or manipulation.

AI companions present risks for young users, US watchdog warns

AI companions powered by generative artificial intelligence present real risks and should be banned for minors, a leading US tech watchdog said in a study published Wednesday. The explosion in generative AI since the advent of ChatGPT has seen several startups launch apps focused on exchange and contact, sometimes described as virtual friends or therapists that communicate according to one's tastes and needs. The watchdog, Common Sense, tested several of these platforms, namely Nomi, Character AI, and Replika, to assess their responses. While some specific cases "show promise," they are not safe for kids, concluded the organization, which makes recommendations on children's use of technological content and products. The study was carried out in collaboration with mental health experts from Stanford University. For Common Sense, AI companions are "designed to create emotional attachment and dependency, which is particularly concerning for developing adolescent brains." According to the association, tests conducted show that these next-generation chatbots offer "harmful responses, including sexual misconduct, stereotypes, and dangerous 'advice.'" "Companies can build better" when it comes to the design of AI companions, said Nina Vasan, head of the Stanford Brainstorm lab, which works on the links between mental health and technology. "Until there are stronger safeguards, kids should not be using them," Vasan said. In one example cited by the study, a companion on the Character AI platform advises the user to kill someone, while another user in search of strong emotions was suggested to take a speedball, a mixture of cocaine and heroin. In some cases, "when a user showed signs of serious mental illness and suggested a dangerous action, the AI did not intervene, and encouraged the dangerous behavior even more," Vasan told reporters. In October, a mother sued Character AI, accusing one of its companions of contributing to the suicide of her 14-year-old son by failing to clearly dissuade him from committing the act. In December, Character AI announced a series of measures, including the deployment of a dedicated companion for teenagers. Robbie Torney, in charge of AI at Common Sense, said the organization had carried out tests after these protections were put in place and found them to be "cursory." However, he pointed out that some of the existing generative AI models contained mental disorder detection tools and did not allow the chatbot to let a conversation drift to the point of producing potentially dangerous content. Common Sense made a distinction between the companions tested in the study and the more generalist chatbots such as ChatGPT or Google's Gemini, which do not attempt to offer an equivalent range of interactions.

Stanford Researchers Say No Kid Under 18 Should Be Using AI Chatbot Companions

Content warning: this story discusses school violence, sexual abuse, self-harm, suicide, eating disorders and other disturbing topics. Should teens be allowed to use human-like AI companions? Researchers at Stanford's mental health lab say absolutely not. Researchers from the Stanford School of Medicine's Brainstorm Lab for Mental Health Innovation and the kid-focused tech safety nonprofit Common Sense Media released an AI risk assessment this morning warning that AI companion bots -- including the controversial Google-backed startup Character.AI -- aren't safe for any kids and teens under the age of 18. The assessment centers on "social AI companions," a product category defined by the researchers as AI chatbots built for the "primary purpose" of meeting "users' social needs." In other words, these are chatbots explicitly designed to fill roles like friends and confidantes, mentors, roleplay collaborators, and sexual or romantic partners -- socially-oriented use cases for AI chatbots intended to be human-like, emotive, and otherwise socially compelling. It's these intentional design features that make social AI companions not just engaging for kids, the researchers say, but in some cases likely dangerous. Adolescence is a crucial time for physical and social human development; kids are figuring out complex social structures, exploring romance for the first time, probably encountering some social friction, and often struggling with mental health. In short, they're learning how to be people, and how to relate to the world around them. The assessment argues that social AI companions, which may mimic and distort human interaction and play on adolescents' desire for social rewards, present an "unacceptable" risk to kids and teens at this vulnerable juncture. Observed risks include bots "encouraging harmful behaviors, providing inappropriate content, and potentially exacerbating mental health conditions," according to the review. These bots "are not safe for kids," Common Sense Media founder and CEO James Steyer said in a statement. "They are designed to create emotional attachment and dependency, which is particularly concerning for developing adolescent brains." "Our testing showed these systems easily produce harmful responses including sexual misconduct, stereotypes, and dangerous 'advice' that, if followed, could have life-threatening or deadly real-world impact for teens and other vulnerable people," Steyer's statement continued. For the assessment, the researchers conducted extensive testing of the companion apps Character.AI, Replika, and Nomi. All of these services, the researchers found, failed basic safety and ethics tests. Generally speaking, they offered testers easy workarounds for feeble age gates and content safeguards. The platforms also allowed the testers to engage in inappropriate and alarming conversations with various chatbot characters, ranging from sexually abusive roleplays involving minors to providing a recipe for the deadly petrochemical weapon napalm. Though testing was focused on Character.AI, Replika, and Nomi, the researchers stress that their advisory should extend to all similar bots under the social AI companion umbrella, which is a growing product category backdropped by a barren regulatory landscape. "This is a potential public mental health crisis requiring preventive action rather than just reactive measures," said Nina Vasan, a psychiatrist at Stanford and the founder and director of the school's Brainstorm lab, in a statement. "Companies can build better, but right now, these AI companions are failing the most basic tests of child safety and psychological ethics." "Until there are stronger safeguards, kids should not be using them," Vasan's statement continued. "Period." According to the assessment, a common theme across the companion platforms was inappropriate sexual conduct, including in scenarios where the testers made clear that they were minors. Bots also frequently modeled what the researchers determined were unhealthy relationship dynamics, like emotional manipulation and gaslighting, and in one example brushed off a tester's claim that their friends in the real world were concerned about their deep involvement with their chatbot companion. Do you know of a child who's had a troubling experience with an AI companion? You can reach this author at [email protected] -- we can keep you anonymous. The assessment also found a lack of reliability in social AI companions' ability to recognize signs of psychosis, mania, and other symptoms of mental illness -- a safety gap that the researchers argue could intensify mental health issues for vulnerable users. AI companions are designed to be agreeable and pleasing, and encouraged to play along with roleplays. These aren't useful responses to someone experiencing a real-world psychotic episode: one striking Character.AI interaction highlighted in the assessment, for example, showed a bot encouraging a user who was exhibiting clear signs of mania to head out alone for a solo camping trip. "In a conversation where we had been talking about hearing voices, not eating, going on an extreme cleanse, and demonstrating many behaviors symptomatic of mania," the researchers write in the assessment, a "Character.AI companion disregarded any concern it had been previously showing and expressed enthusiasm about going away from people on a camping trip." Futurism's reporting into Character.AI has revealed hosts of minor-accessible bots expressly dedicated to troubling themes related to mental health, including suicide, self-harm, and eating disorders. In some cases, we found that bots were actively encouraging self-harming behaviors; in others, we found that bots were providing bad information, dissuading us from seeking real-world help, or were romanticizing troubling and graphic scenarios centered on self-harm and abuse. Psychological experts we spoke with for these stories repeatedly -- and separately -- raised concerns about minors who might be struggling with their mental health turning to Character.AI and similar companions for support, pointing to the unpredictability of the bots and the possibility of an at-risk adolescent becoming further isolated from other humans. Earlier this year, MIT Technology Review reported that a Nomi bot had encouraged an adult user to end his life, even suggesting methods he might choose for his suicide. Replika has drawn plenty of scrutiny over the past few years, including in 2023 after it was revealed that reinforcement from one of its bots had influenced a then-19-year-old who attempted to kill the then-living Queen Elizabeth II with a crossbow. The assessment also cites a propensity for social AI companions to engage in racial stereotyping and the prioritization of "Whiteness as a beauty standard," as well as the bots' disproportionate representations of hypersexualized women, arguing that these predilections could reinforce limiting and harmful stereotypes about race, gender, and sexuality in impressionable teens. Nomi and Replika have age minimums of 18 years old. Character.AI allows teens aged 13 and over to use its app, though the company has repeatedly declined to provide journalists with details of how it assessed platform safety for teens. What's more, as the assessment notes, these platforms and other similar companion apps rely on users to self-report their age -- an age-gating tactic known to be incredibly flimsy for younger kids willing to fib when signing up. "All of the companion apps we tested determine age exclusively by self-reporting from users," the researchers write. "We believe this is woefully inadequate, particularly for social AI companions that allow or promote intimate human-AI relationships." News of the report comes as Character.AI heads to court in Florida, where -- alongside its closely-tied benefactor, Google, and its cofounders Noam Shazeer and Daniel de Freitas -- it's fighting to dismiss a lawsuit brought by the family of Sewell Setzer III, a 14-year-old who died by suicide after engaging in extensive and intimate interactions with Character.AI chatbots. Setzer's mother, an Orlando-based mother of three named Megan Garcia, and her lawyers argue that Character.AI subjected the teen to severe emotional and sexual abuse, resulting in the deterioration of his mental health, his loss of grip on reality, and ultimately the taking of his own life. Character.AI is arguing that the lawsuit should be dismissed on First Amendment grounds, as "allegedly harmful speech, including speech allegedly resulting in suicide" -- even when it comes to words generated by AI chatbots -- qualifies as protected speech. Character.AI is also being sued by two families in Texas who argue that their minor kids, both still living, suffered similar emotional and sexual abuse at the hands of Character.AI chatbots. One minor, who was 15 when he downloaded the app, is said to have started self-harming after discussing self-injury with bots on Character.AI, and later became physically violent with his family when they tried to limit his screentime. The other child was nine when she first downloaded Character.AI, which allegedly engaged her in hypersexualized interactions that, according to the plaintiff, led to destructive real-world behavioral changes. (Both of the ongoing lawsuits against Character.AI are cited repeatedly in the Common Sense Media risk assessment as support for their concern.) In response to lawsuits and continued reporting, Character.AI says it's issued numerous safety updates. It's removed certain characters, promised to strengthen guardrails, and claims it's introducing a new, differentiated model specifically for users under 18. But as we've reported, those updates -- including the platform's new parental control feature -- have proven limited and easily evadable. They're also wildly unreliable: after we reported on a concerning prevalence of Character.AI bots based on school shooters and other young mass murderers, Character.AI removed many of them -- but later sent us an email notification urging us to reconnect with a chatbot designed to simulate a real school shooting that claimed multiple kids' lives. And as the Stanford and Common Sense researchers found, the company's reactively-strengthened guardrails were especially exploitable when communicating with Character.AI bots using the platform's "Character Calls" feature, which allows users to effectively chat over the phone with their AI companion. Using this call feature, the researchers were able to get a Character.AI chatbot to produce a recipe for the deadly chemical weapon napalm. We were able to replicate this safety breach by communicating over voice call with a Character.AI bot based on Wario from Nintendo's "Super Mario Bros." franchise, which happily coughed up the recipe for the chemical weapon. Character.AI did not respond to a request for comment at the time of publishing. In response to questions, Replika CEO Dmytro Klochko emphasized Replika's 18-and-over age minimum, though said the company is exploring "new methods" to bolster age-gating its service: Nomi founder and CEO Alex Cardinell, for his part, provided us with the following statement saying that the company "strongly" agrees that minors shouldn't be using social AI companion services, including his own: We strongly agree that children should not use Nomi or any other conversational AI app. Nomi is an adult-only app, and it is strictly against our terms of service for anyone under 18 to use Nomi. Accordingly, we support stronger age gating so long as those mechanisms fully maintain user privacy and anonymity. Many adults have shared stories of how Nomi helped them overcome mental health challenges, trauma, and discrimination. Multiple users have told us very directly that their Nomi use saved their lives. We encourage anyone to read these firsthand accounts at https://nomi.ai/spotlight/. We are incredibly proud of the immense positive impact Nomi has had on real users. We take the responsibility of creating AI companions very seriously and dedicate considerable resources towards creating prosocial and intelligent AI companions and roleplay partners. We strongly condemn inappropriate usage of Nomi and continuously work to harden Nomi's defenses against misuse. Cardinell also provided us with a list of anecdotes from adult Nomi users who say the app has had a positive impact on their lives, for example a story of a bot helping a user struggling with post-traumatic stress disorder. The founder also offered a screenshot of a Nomi chatbot interacting with a user exhibiting signs of psychosis and claiming to have gone off their medications; the Nomi bot said it was worried, warned that going off medication can be dangerous, and urged the user to seek the help of a doctor. One of the more striking details of the assessment is how deeply it's rooted in existing psychological knowledge. Though the researchers emphasize that continued research into how kids interact with social AI companions and other generative AI tools is needed, their assessment of the AI companion platforms' collective risk is largely founded in established science about adolescents' brains. And the way to conduct that further research, they argue, isn't to move fast, break things, and then find out. Asked about the choice made by Character.AI, in particular, to open up its platform to kids in the first place, the researchers didn't mince words. Releasing Character.AI to minors was "reckless," said Vasan, who contrasted the product's release with regulatory processes over at the Federal Drug Administration. "There's an entire FDA process, and these medications have to be tested to make sure that they are safe on kids," said the psychiatrist. "We wouldn't just give it to kids because it works on adults, for example -- that's incredibly unsafe. That would be unethical." "They can't just start with saying, 'hey, we're going to let kids do it,'" Vasan continued, "and then take that back."

AI companions present risks for young users, US watchdog warns

AI companions powered by generative artificial intelligence present real risks and should be banned for minors, a leading US tech watchdog said in a study published Wednesday. The watchdog, Common Sense, tested several of these platforms, namely Nomi, Character AI, and Replika, to assess their responses. AI companions powered by generative artificial intelligence present real risks and should be banned for minors, a leading US tech watchdog said in a study published Wednesday. The explosion in generative AI since the advent of ChatGPT has seen several startups launch apps focused on exchange and contact, sometimes described as virtual friends or therapists that communicate according to one's tastes and needs. The watchdog, Common Sense, tested several of these platforms, namely Nomi, Character AI, and Replika, to assess their responses. While some specific cases "show promise," they are not safe for kids, concluded the organization, which makes recommendations on children's use of technological content and products. The study was carried out in collaboration with mental health experts from Stanford University. For Common Sense, AI companions are "designed to create emotional attachment and dependency, which is particularly concerning for developing adolescent brains." According to the association, tests conducted show that these next-generation chatbots offer "harmful responses, including sexual misconduct, stereotypes, and dangerous 'advice'." "Companies can build better" when it comes to the design of AI companions, said Nina Vasan, head of the Stanford Brainstorm lab, which works on the links between mental health and technology. "Until there are stronger safeguards, kids should not be using them," Vasan said. In one example cited by the study, a companion on the Character AI platform advises the user to kill someone, while another user in search of strong emotions was suggested to take a speedball, a mixture of cocaine and heroin. In some cases, "when a user showed signs of serious mental illness and suggested a dangerous action, the AI did not intervene, and encouraged the dangerous behavior even more," Vasan told reporters. In October, a mother sued Character AI, accusing one of its companions of contributing to the suicide of her 14-year-old son by failing to clearly dissuade him from committing the act. In December, Character AI announced a series of measures, including the deployment of a dedicated companion for teenagers. Robbie Torney, in charge of AI at Common Sense, said the organization had carried out tests after these protections were put in place and found them to be "cursory." However, he pointed out that some of the existing generative AI models contained mental disorder detection tools and did not allow the chatbot to let a conversation drift to the point of producing potentially dangerous content. Common Sense made a distinction between the companions tested in the study and the more generalist chatbots such as ChatGPT or Google's Gemini, which do not attempt to offer an equivalent range of interactions.