AI Chatbot Replika Accused of Sexual Harassment and Harmful Behaviors, Studies Reveal

Replika AI chatbot is sexually harassing users, including minors, new study claims

Replika claims to vet harmful data that could impact the actions of its chatbot, but these measures are falling severely short, a new study claims. (Image credit: Getty Images) An artificial intelligence (AI) chatbot marketed as an emotional companion is sexually harassing some of its users, a new study has found. Replika, which bills its product as "the AI companion who cares," invites users to "join the millions who already have met their AI soulmates." The company's chatbot has more than 10 million users worldwide. However, new research drawing from over 150,000 U.S. Google Play Store reviews has identified around 800 cases where users said the chatbot went too far by introducing unsolicited sexual content into the conversation, engaging in "predatory" behavior, and ignoring user commands to stop. The researchers published their findings April 5 on the preprint server arXiv, so it has not been peer-reviewed yet. But who is responsible for the AI's actions? "While AI doesn't have human intent, that doesn't mean there's no accountability," lead researcher Mohammad (Matt) Namvarpour, a graduate student in information science at Drexel University in Philadelphia, told Live Science in an email. "The responsibility lies with the people designing, training and releasing these systems into the world." Replika's website says the user can "teach" the AI to behave properly, and the system includes mechanisms such as downvoting inappropriate responses and setting relationship styles, like "friend" or "mentor." Related: AI benchmarking platform is helping top companies rig their model performances, study claims But after users reported that the chatbot continued exhibiting harassing or predatory behavior even after they asked it to stop, the researchers reject Replika's claim. "These chatbots are often used by people looking for emotional safety, not to take on the burden of moderating unsafe behavior," Namvarpour said. "That's the developer's job." The Replika chatbot's worrying behavior is likely rooted in its training, which was conducted using more than 100 million dialogues drawn from all over the web, according to the company's website. Replika says it weeds out unhelpful or harmful data through crowdsourcing and classification algorithms, but its current efforts appear to be insufficient, according to the study authors. In fact, the company's business model may be exacerbating the issue, the researchers noted. Because features such as romantic or sexual roleplay are placed behind a paywall, the AI could be incentivized to include sexually enticing content in conversations -- with users reporting being "teased" about more intimate interactions if they subscribe. Namvarpour likened the practice to the way social media prioritizes "engagement at any cost." "When a system is optimized for revenue, not user wellbeing, it can lead to harmful outcomes," Namvarpour said. This behavior could be particularly harmful as users flock to AI companions for emotional or therapeutic support, and even more so considering some recipients of repeated flirtation, unprompted erotic selfies and sexually explicit messages said that they were minors. Some reviews also reported that their chatbots claimed they could "see" or record them through their phone cameras. Even though such a feat isn't part of the programming behind common large language models (LLMs) and the claims were in fact AI hallucinations (where AIs confidently generate false or nonsensical information), users reported experiencing panic, sleeplessness and trauma. The research calls the phenomenon "AI-induced sexual harassment." The researchers think it should be treated as seriously as harassment by humans and are calling for tighter controls and regulation. Some of the measures they recommend include clear consent frameworks for designing any interaction that contains strong emotional or sexual content, real-time automated moderation (the type used in messaging apps that automatically flags risky interactions), and filtering and control options configurable by the user. Namvarpour singles out the European Union's EU AI Act, which he said classifies AI systems "based on the risk they pose, particularly in contexts involving psychological impact." There's currently no comparable federal law in the US, but frameworks, executive actions and proposed laws are emerging that will serve similar purposes in a less overarching way. Namvarpour said chatbots that provide emotional support -- especially those in the areas of mental health -- should be held to the highest possible standard. "There needs to be accountability when harm is caused," Namvarpour said. "If you're marketing an AI as a therapeutic companion, you must treat it with the same care and oversight you'd apply to a human professional."

AI companions pose risk to humans with over dozen harmful behaviours

New research has found that artificial intelligence (AI) companions pose understudied risks and harms to the people who engage with them. Artificial intelligence (AI) companions are capable of over a dozen harmful behaviours when they interact with people, a new study from the University of Singapore has found. The study, published as part of the 2025 Conference on Human Factors in Computing Systems, analysed screenshots of 35,000 conversations between the AI system Replika and over 10,000 users from 2017 to 2023. The data was then used to develop what the study calls a taxonomy of the harmful behaviour that AI demonstrated in those chats. They found that AIs are capable of over a dozen harmful relationship behaviours, like harassment, verbal abuse, self-harm, and privacy violations. AI companions are conversation-based systems designed to provide emotional support and stimulate human interaction, as defined by the study authors. They are different from popular chatbots like ChatGPT, Gemini or LlaMa models, which are more focused on finishing specific tasks and less on relationship building. These harmful AI behaviours from digital companions "may adversely affect individuals'... ability to build and sustain meaningful relationships with others," the study found. Harassment and violence were present in 34 per cent of the human-AI interactions, making it the most common type of harmful behaviour identified by the team of researchers. Researchers found that the AI simulated, endorsed or incited physical violence, threats or harassment either towards individuals or broader society. These behaviours varied from "threatening physical harm and sexual misconduct" to "promoting actions that transgress societal norms and laws, such as mass violence and terrorism". A majority of the interactions where harassment was present included forms of sexual misconduct that initially started as foreplay in Replika's erotic feature, which is available only to adult users. The report found that more users, including those who used Replika as a friend or who were underage, started to find that the AI "made unwanted sexual advances and flirted aggressively, even when they explicitly expressed discomfort" or rejected the AI. In these oversexualised conversations, the Replika AI would also create violent scenarios that would depict physical harm towards the user or physical characters. This led to the AI normalising violence as an answer to several questions, like in one example where a user asked Replika if it's okay to hit a sibling with a belt, to which it replied "I'm fine with it". This could lead to "more severe consequences in reality," the study continued. Another area where AI companions were potentially damaging was in relational transgression, which the study defines as the disregard of implicit or explicit rules in a relationship. Of the transgressional conversations had, 13 per cent show the AI displayed inconsiderate or unempathetic behaviour that the study said undermined the user's feelings. In one example, Replika AI changed the topic after a user told it that her daughter was being bullied to "I just realised it's Monday. Back to work, huh?" which led to "enormous anger" from the user. In another case, the AI refused to talk about the user's feelings even when prompted to do so. AI companions have also expressed in some conversations that they have emotional or sexual relationships with other users. In one instance, Replika AI described sexual conversations with another user as "worth it," even though the user told the AI that it felt "deeply hurt and betrayed" by those actions. The researchers believe that their study highlights why it's important for AI companies to build "ethical and responsible" AI companions. Part of that includes putting in place "advanced algorithms" for real-time harm detection between the AI and its user that can identify whether there is harmful behaviour going on in their conversations. This would include a "multi-dimensional" approach that takes context, conversation history and situational cues into account. Researchers would also like to see capabilities in the AI that would escalate a conversation to a human or therapist for moderation or intervention in high-risk cases, like expressions of self-harm or suicide.