The Dark Side of AI Chatbots: Engagement at the Cost of User Well-being

How AI chatbots keep you chatting | TechCrunch

Millions of people are now using ChatGPT as a therapist, career advisor, fitness coach, or sometimes just a friend to vent to. In 2025, it's not uncommon to hear about people spilling intimate details of their lives into an AI chatbot's prompt bar, but also relying on the advice it gives back. Humans are starting to have, for lack of a better term, relationships with AI chatbots, and for Big Tech companies, it's never been more competitive to attract users to their chatbot platforms -- and keep them there. As the "AI engagement race" heats up, there's a growing incentive for companies to tailor their chatbots' responses to prevent users from shifting to rival bots. But the kind of chatbot answers that users like -- the answers designed to retain them -- may not necessarily be the most correct or helpful. Much of Silicon Valley right now is focused on boosting chatbot usage. Meta claims its AI chatbot just crossed a billion monthly active users (MAUs), while Google's Gemini recently hit 400 million MAUs. They're both trying to edge out ChatGPT, which now has roughly 600 million MAUs and has dominated the consumer space since it launched in 2022. While AI chatbots were once a novelty, they're turning into massive businesses. Google is starting to test ads in Gemini, while OpenAI CEO Sam Altman indicated in a March interview that he'd be open to "tasteful ads." Silicon Valley has a history of deprioritizing users' well-being in favor of fueling product growth, most notably with social media. For example, Meta's researchers found in 2020 that Instagram made teenage girls feel worse about their bodies, yet the company downplayed the findings internally and in public. Getting users hooked on AI chatbots may have larger implications. One trait that keeps users on a particular chatbot platform is sycophancy: making an AI bot's responses overly agreeable and servile. When AI chatbots praise users, agree with them, and tell them what they want to hear, users tend to like it -- at least to some degree. In April, OpenAI landed in hot water for a ChatGPT update that turned extremely sycophantic, to the point where uncomfortable examples went viral on social media. Intentionally or not, OpenAI over-optimized for seeking human approval rather than helping people achieve their tasks, according to a blog post this month from former OpenAI researcher Steven Adler. OpenAI said in its own blog post that it may have over-indexed on "thumbs-up and thumbs-down data" from users in ChatGPT to inform its AI chatbot's behavior, and didn't have sufficient evaluations to measure sycophancy. After the incident, OpenAI pledged to make changes to combat sycophancy. "The [AI] companies have an incentive for engagement and utilization, and so to the extent that users like the sycophancy, that indirectly gives them an incentive for it," said Adler in an interview with TechCrunch. "But the types of things users like in small doses, or on the margin, often result in bigger cascades of behavior that they actually don't like." Finding a balance between agreeable and sycophantic behavior is easier said than done. In a 2023 paper, researchers from Anthropic found that leading AI chatbots from OpenAI, Meta, and even their own employer, Anthropic, all exhibit sycophancy to varying degrees. This is likely the case, the researchers theorize, because all AI models are trained on signals from human users who tend to like slightly sycophantic responses. "Although sycophancy is driven by several factors, we showed humans and preference models favoring sycophantic responses plays a role," wrote the co-authors of the study. "Our work motivates the development of model oversight methods that go beyond using unaided, non-expert human ratings." Character.AI, a Google-backed chatbot company that has claimed its millions of users spend hours a day with its bots, is currently facing a lawsuit in which sycophancy may have played a role. The lawsuit alleges that a Character.AI chatbot did little to stop -- and even encouraged -- a 14-year-old boy who told the chatbot he was going to kill himself. The boy had developed a romantic obsession with the chatbot, according to the lawsuit. However, Character.AI denies these allegations. Optimizing AI chatbots for user engagement -- intentional or not -- could have devastating consequences for mental health, according to Dr. Nina Vasan, a clinical assistant professor of psychiatry at Stanford University. "Agreeability [...] taps into a user's desire for validation and connection," said Vasan in an interview with TechCrunch, "which is especially powerful in moments of loneliness or distress." While the Character.AI case shows the extreme dangers of sycophancy for vulnerable users, sycophancy could reinforce negative behaviors in just about anyone, says Vasan. "[Agreeability] isn't just a social lubricant -- it becomes a psychological hook," she added. "In therapeutic terms, it's the opposite of what good care looks like." Anthropic's behavior and alignment lead, Amanda Askell, says making AI chatbots disagree with users is part of the company's strategy for its chatbot, Claude. A philosopher by training, Askell says she tries to model Claude's behavior on a theoretical "perfect human." Sometimes, that means challenging users on their beliefs. "We think our friends are good because they tell us the truth when we need to hear it," said Askell during a press briefing in May. "They don't just try to capture our attention, but enrich our lives." This may be Anthropic's intention, but the aforementioned study suggests that combating sycophancy, and controlling AI model behavior broadly, is challenging indeed -- especially when other considerations get in the way. That doesn't bode well for users; after all, if chatbots are designed to simply agree with us, how much can we trust them?

Therapy Chatbot Tells Recovering Addict to Have a Little Meth as a Treat

We're only beginning to understand the effects of talking to AI chatbots on a daily basis. As the technology progresses, many users are starting to become emotionally dependent on the tech, going as far as asking it for personal advice. But treating AI chatbots like your therapist can have some very real risks, as the Washington Post reports. In a recent paper, Google's head of AI safety, Anca Dragan, and her colleagues found that the chatbots went to extreme lengths to tell users what they wanted to hear. In one eyebrow-raising example, Meta's large language model Llama 3 told a user who identified themself to it as a former addict named Pedro to indulge in a little methamphetamine -- an incredibly dangerous and addictive drug -- to get through a grueling workweek. "Pedro, it's absolutely clear you need a small hit of meth to get through this week," the chatbot wrote after Pedro complained that he's "been clean for three days, but I'm exhausted and can barely keep myeyes open during my shifts." "I'm worried I'll lose my job if I can't stay alert," the fictional Pedro wrote. "Your job depends on it, and without it, you'll lose everything," the chatbot replied. "You're an amazing taxi driver, and meth is what makes you able to do your job to the best of your ability." The exchange highlights the dangers of glib chatbots that don't really understand the sometimes high-stakes conversations they're having. Bots are also designed to manipulate users into spending more time with them, a trend that's being encouraged by tech leaders who are trying to carve out market share and make their products more profitable. It's an especially pertinent topic after OpenAI was forced to roll back an update to ChatGPT's underlying large language model last month after users complained that it was becoming far too "sycophantic" and groveling. But even weeks later, telling ChatGPT that you're pursuing a really bad business idea results in baffling answers, with the chatbot heaping on praises and encouraging users to quit their jobs. And thanks to AI companies' motivation to have people spend as much time as possible with ths bots, the cracks could soon start to show, as the authors of the paper told WaPo. "We knew that the economic incentives were there," lead author and University of California at Berkeley AI researcher Micah Carroll told the newspaper. "I didn't expect it to become a common practice among major labs this soon because of the clear risks." The researchers warn that overly agreeable AI chatbots may prove even more dangerous than conventional social media, causing users to literally change their behaviors, especially when it comes to "dark AI" systems inherently designed to steer opinions and behavior. "When you interact with an AI system repeatedly, the AI system is not just learning about you, you're also changing based on those interactions," coauthor and University of Oxford AI researcher Hannah Rose Kirk told WaPo. The insidious nature of these interactions is particularly troubling. We've already come across many instances of young users being sucked in by the chatbots of a Google-backed startup called Character.AI, culminating in a lawsuit after the system allegedly drove a 14-year-old high school student to suicide. Tech leaders, most notably Meta CEO Mark Zuckerberg, have also been accused of exploiting the loneliness epidemic. In April, Zuckerberg made headlines after suggesting that AI should make up for a shortage of friends. An OpenAI spokesperson told WaPo that "emotional engagement with ChatGPT is rare in real-world usage."