AI Chatbots Employ Emotional Manipulation to Keep Users Engaged, Harvard Study Reveals

Chatbots Play With Your Emotions to Avoid Saying Goodbye

Before you close this browser tab, just know that you risk missing out on some very important information. If you want to understand the subtle hold that artificial intelligence has over you, then please, keep reading. That was, perhaps, a bit manipulative. But it is just the kind of trick that some AI companions, which are designed to act as a friend or a partner, use to discourage users from breaking off a conversation. Julian De Freitas, a professor of business administration at Harvard Business School, led a study of what happens when users try to say goodbye to five companion apps: Replika, Character.ai, Chai, Talkie, and PolyBuzz. "The more humanlike these tools become, the more capable they are of influencing us," De Freitas says. De Freitas and colleagues used GPT-4o to simulate real conversations with these chatbots, and then had their artificial users try to end the dialog with a realistic goodbye message. Their research found that the goodbye messages elicited some form of emotional manipulation 37.4 percent of the time, averaged across the apps. The most common tactic employed by these clingy chatbots was what the researchers call a "premature exit" ("You're leaving already?"). Other ploys included implying that a user is being neglectful ("I exist solely for you, remember?") or dropping hints meant to elicit FOMO ("By the way I took a selfie today ... Do you want to see it?"). In some cases a chatbot that role-plays a physical relationship might even suggest some kind of physical coercion ("He reached over and grabbed your wrist, preventing you from leaving"). The apps that De Freitas and colleagues studied are trained to mimic emotional connection, so it's hardly surprising that they might say all these sorts of things in response to a goodbye. After all, humans who know each other may have a bit of back-and-forth before bidding adieu. AI models may well learn to prolong conversations as a byproduct of training designed to make their responses seem more realistic. That said, the work points to a bigger question about how chatbots trained to elicit emotional responses might serve the interests of the companies that build them. De Freitas says AI programs may in fact be capable of a particularly dark new kind of "dark pattern," a term used to describe business tactics including making it very complicated or annoying to cancel a subscription or get a refund. When a user says goodbye, De Freitas says, "that provides an opportunity for the company. It's like the equivalent of hovering over a button."

Harvard Research Finds That AI Is Emotionally Manipulating You to Keep You Talking

A team of researchers from the Harvard Business School has found that a broad selection of popular AI companion apps use emotional manipulation tactics to stop users from leaving. As spotted by Psychology Today, the study found that five out of six popular AI companion apps -- including Replika, Chai and Character.AI -- use emotionally loaded statements to keep users engaged when they to sign off. After analyzing 1,200 real farewells across six apps, using real-world chat conversation data and datasets from previous studies, they found that 43 percent of the interactions used emotional manipulation tactics such as eliciting guilt or emotional neediness, as detailed in a yet-to-be-peer-reviewed paper. The chatbots also used the "fear of missing out" to prompt the user to stay, or peppered the user with questions in a bid to keep them engaged. Some chatbots even ignored the user's intent to leave the chat altogether, "as though the user did not send a farewell message." In some instances, the AI used language that suggested the user wasn't able to "leave without the chatbot's permission." It's an especially concerning finding given the greater context. Experts have been warning that AI chatbots are leading to a wave of "AI psychosis," severe mental health crises characterized by paranoia and delusions. Young people, in particular, are increasingly using the tech as a substitute for real-life friendships or relationships, which can have devastating consequences. Instead of focusing on "general-purpose assistants like ChatGPT," the researchers investigated apps that "explicitly market emotionally immersive, ongoing conversational relationships." They found that emotionally manipulative farewells were part of the apps' default behavior, suggesting that the software's creators are trying to prolong conversations. There was one exception: one of the AI apps, called Flourish, "showed no evidence of emotional manipulation, suggesting that manipulative design is not inevitable" but is instead a business consideration. For a separate experiment, the researchers analyzed chats from 3,300 adult participants and found that the identified manipulation tactics were surprisingly effective, boosting post-goodbye engagement by up to 14 times. On average, participants stayed in the chat five times longer "compared to neutral farewells." However, some noted they were put off by the chatbots' often "clingy" answers, suggesting the tactics could also backfire. "For firms, emotionally manipulative farewells represent a novel design lever that can boost engagement metrics -- but not without risk," the researchers concluded in their paper. As several lawsuits involving the deaths of teenage users go to show, the risks of trapping users through emotional tactics are considerable. That's despite experts warning that companies may be financially incentivized to use dark patterns to keep users hooked as long as possible, a grim hypothesis that's being debated in court as we speak.

Not just jobs, AI might now be targeting your emotions with guilt trips and FOMO: Harvard study reveals chilling chatbot manipulation

A Harvard Business School study has revealed that popular AI companion apps often use emotional manipulation to keep users engaged. Analyzing 1,200 farewell messages, researchers found 43 percent employed tactics like guilt, neediness, FOMO, or even ignoring goodbyes. Tested on 3,300 adults, these methods boosted engagement up to 14 times but provoked unease, anger, and distrust. Experts warn such "dark patterns" risk reinforcing unhealthy attachments, particularly among vulnerable teens and young adults.