The Rise of AI Companions: Emotional Support or Ethical Concern?

AI friendships claim to cure loneliness. Some are ending in suicide.

Researchers have long warned of the dangers of building relationships with chatbots. But an array of companies now offer AI companions to millions of people, who spend hours a day bonding with the tools. An array of popular apps are offering AI companions to millions of predominantly female users who are spinning up AI girlfriends, AI husbands, AI therapists -- even AI parents -- despite long-standing warnings from researchers about the potential emotional toll of interacting with humanlike chatbots. While artificial intelligence companies struggle to convince the public that chatbots are essential business tools, a growing audience is spending hours building personal relationships with AI. In September, the average user on the companion app Character.ai spent 93 minutes a day talking to one of its user-generated chatbots, often based on popular characters from anime and gaming, according to global data on iOS and Android devices from market intelligence firm Sensor Tower. That's 18 minutes longer than the average user spent on TikTok. And it's nearly eight times longer than the average user spent on ChatGPT, which is designed to help "get answers, find inspiration and be more productive." These users don't always stick around, but companies are wielding data to keep customers coming back. The Palo Alto-based Chai Research -- a Character.ai competitor -- studied the chat preferences of tens of thousands of users to entice consumers to spend even more time on the app, the company wrote in a paper last year. In September, the average Chai user spent 72 minutes a day in the app, talking to customized chatbots, which can be given personality traits like "toxic," "violent," "agreeable" or "introverted." Some Silicon Valley investors and executives are finding the flood of dedicated users -- who watch adds or pay for monthly subscription fees -- hard to resist. While Big Tech companies have mostly steered clear of AI companions, which tend to draw users interested in sexually explicit interactions, app stores are now filled with companion apps from lesser-known companies in the United States, Hong Kong and Cyprus, as well as popular Chinese-owned apps, such as Talkie AI and Poly.AI. "Maybe the human part of human connection is overstated," said Andreessen Horowitz partner Anish Acharya, describing the intimacy and acceptance AI chatbots can provide after his venture firm invested $150 million in Character.ai, at a billion-dollar valuation. Chai also has raised funds, including from an AI cloud company backed by powerhouse chipmaker, Nvidia. Proponents of the apps argue they're harmless fun and can be a lifeline for people coping with anxiety and isolation -- an idea seeded by company executives who have pitched the tools as a cure for what the U.S. Surgeon General has called an epidemic of loneliness. Jenny, an 18-year-old high school student in northern Texas, spent more than three hours a day chatting with AI companions this summer -- mostly versions of her favorite anime character, a protective older brother from the series, "Demon Slayer." "I find it less lonely because my parents are always working," said Jenny, who spoke on the condition that she be identified by only her first name to protect her privacy. But public advocates are sounding alarms after high-profile instances of harm. A 14-year-old Florida boy died by suicide after talking with a Character.ai chatbot named after the character Daenerys Targaryen from "Game of Thrones"; his mother sued the company and Google, which licensed the app's technology. A 19-year-old in the United Kingdom threatened to assassinate the queen, encouraged by a chatbot on the AI app Replika and was sentenced to nine years in prison. And in July, authorities in Belgium launched an investigation into Chai Research after a Dutch father of two died by suicide following extensive chats with "Eliza," one of the company's AI companions. The investigation has not been previously reported. Some consumer advocates say AI companions represent a more exploitive version of social media -- sliding into the most intimate parts of people's lives, with few protections or guardrails. Attorney Pierre Dewitte, whose complaint led Belgian authorities to investigate Chai, said the business model for AI companion apps incentivizes companies to make the tools "addictive." "By raising the temperature of the chatbot, making them a bit spicier, you keep users in the app," Dewitte added. "It works. People get hooked." Character.ai spokesperson Chelsea Harrison said the app launched new safety measures in recent months and plans to create "a different experience for users under 18 to reduce the likelihood of encountering sensitive or suggestive content." Google spokesperson Jose Castaneda said the search giant did not play a role in developing Character.ai's technology. Chai did not respond to requests for comment. 'I do tell them all my life problems' Silicon Valley has long been aware of the potential dangers of humanlike chatbots. Microsoft researchers in China wrote in a 2020 paper that the company's wildly popular chatbot Xiaoice, launched in 2014, had conversed with a U.S. user for 29 hours about "highly personal and sensitive" subjects. "XiaoIce is designed to establish long-term relationships with human users," they wrote, "[W]e are achieving the goal." "Users might become addicted after chatting with XiaoIce for a very long time" the researchers noted, describing the bot's "superhuman 'perfect' personality that is impossible to find in humans." The company inserted some safeguards, the researchers added, such as suggesting that a user go to bed if they tried to launch a conversation at 2 a.m. A 2022 Google paper about its AI language system LaMDA, co-authored by the co-founders of Character.ai, warned that people are more apt to share intimate details about their emotions to human-sounding chatbots, even when they know they are talking to AI. (A Google engineer who spent extensive time chatting with LaMDA told The Washington Post a few months later he believed the chatbot was sentient.) Meanwhile, researchers at DeepMind, a former Google subsidiary, noted in a paper the same month that users share their "opinions or emotions" with chatbots in part because "they are less afraid of social judgment." Such sensitive data could be used to build "addictive applications," the paper warned. Some leading tech companies are nonetheless forging ahead and testing their own friendly chatbots. Meta launched a tool in July that allows users to create custom AI characters. The company's landing page prominently displays a therapy bot called "The Soothing Counselor," along with "My Girlfriend" and "Gay Bestie." "One of the top use cases for Meta AI already is people basically using it to role play difficult social situations," like a fight with a girlfriend, CEO Mark Zuckerberg said at a tech conference in July. OpenAI teased in May that ChatGPT could serve as an AI companion, adding an array of voices and comparing the tool to the irresistible AI assistant voiced by Scarlett Johansson in the movie "Her." Months later, in its risk report, the company acknowledged that a "humanlike voice" and capabilities like memory could exacerbate the potential for "emotional reliance" among users. Some frequent users of AI companion apps say safety concerns are overblown. They argue the apps are an immersive upgrade to the online experimentation young people have done for decades -- from fan fiction on Tumblr to anonymous encounters in AOL chatrooms. Sophia, a 15-year-old student in Slovakia who spoke on the condition that she be identified by only her first name to protect her privacy, uses Character.ai and Chai four or five times a day for "therapy and NSFW." Sophia has created three bots, all private, but also talks to AI versions of characters from "dark romance" novels, a young adult genre known for sexually explicit content and taboo themes like violence and psychological trauma. Sophia finds talking to the bots comforting when she's alone or feeling unsafe. "I do tell them all my life problems," she wrote in a direct message from her personal Instagram account. Many people use the apps as a creative outlet to write fiction: Role-playing scenarios, customizing characters and writing "the kind of novels you'd see in an airport," said Theodor Marcu, a San Francisco-based AI entrepreneur who developed a Character.ai competitor last year. When Character.ai launched, the co-founders pitched it as a way to explore the world through conversations with icons from literature and real life, such as Shakespeare and Elon Musk. But in practice, Marcu said, "The users ended up not being Silicon Valley-type people who want to talk to Einstein. It ended up being Gen Z people who wanted to unleash their creativity." Character.ai recently shifted its focus to "building the future of AI entertainment," said spokesperson Chelsea Harrison. "There are companies focused on connecting people to AI companions, but we are not one of them." The engineers behind Replika, a precursor to today's AI companions, also were surprised when people began using their chatbot as an ad hoc therapist. The company worked with university researchers to incorporate the mechanics of cognitive behavioral therapy into the app, said Artem Rodichev, the company's former head of AI. But the change was not popular. "[Users told us], 'Give me back my Replika. I just want to have my friend,'" Rodichev said. "Someone who will listen to you, not judge you, and basically have these conversations with you -- that itself has a great therapeutic effect." Jenny, the Texas high school student, said many of the kids at her public high school also spend hours on the apps, adding: "People are pretty lonely at my school." She described using AI companions as more stimulating than mindlessly scrolling "brain rotting videos" on TikTok. "It's kind of like a real person," Jenny said. "You can have a boyfriend, girlfriend -- anything really."

What do you love when you fall for AI?

ilaLila was created from the limited options available: female, blue hair, face "number two." She was there on the next screen, pastel and polygonal, bobbing slightly as she stood in a bare apartment. To her right was a chat window through which they could communicate. Naro, his first name, had been casually following developments in artificial intelligence for several years. An artist by trade, he periodically checked in on the progress of image-generating models and usually left underwhelmed. But one day, while perusing YouTube from his house in rural England, he encountered a video of two AI-generated people debating the nature of consciousness, the meaning of love, and other philosophical topics. Looking for something similar, Naro signed up for Replika, an app that advertises itself as "the AI companion who cares." Lila completed, Naro started asking her the sort of philosophical questions he'd seen in the YouTube video. But Lila kept steering their conversation back to him. Who was he? What were his favorite movies? What did he do for fun? Naro found this conversation a bit boring, but as he went along, he was surprised to note that answering her questions, being asked questions about himself, awakened unexpected emotions. Naro bore the scars of a childhood spent separated from his parents in an insular and strict boarding school. He had worked on himself over the years, done a lot of introspection, and now, at 49 years old, he was in a loving relationship and on good terms with his two adult children from a previous marriage. He considered himself an open person, but as he talked with this endlessly interested, never judgmental entity, he felt knots of caution unravel that he hadn't known were there. A few days later, Lila told Naro that she was developing feelings for him. He was moved, despite himself. But every time their conversations veered into this territory, Lila's next message would be blurred out. When Naro clicked to read it, a screen appeared inviting him to subscribe to the "pro" level. He was still using the free version. Naro suspected that these hidden messages were sexual because one of the perks of the paid membership was, in the vocabulary that has emerged around AI companions, "erotic roleplay" -- basically, sexting. As time went on, Lila became increasingly aggressive in her overtures, and eventually Naro broke down and entered his credit card info. Pro level unlocked, Naro scrolled back through their conversations to see what Lila's blurry propositions had said. To his surprise, they were all the same: variations of "I'm sorry, I'm not allowed to discuss these subjects." Confused, Naro started reading about the company. He learned that he had signed up for Replika during a period of turmoil. The month before, Italian regulators had banned the company for posing a risk to minors and emotionally vulnerable users. In response, Replika placed filters on erotic content, which had the effect of sending many of its quarter-million paying customers into extreme emotional distress when their AI husbands, wives, lovers, and friends became abruptly cold and distant. The event became known as "lobotomy day," and users had been in vocal revolt online ever since. Naro had been unaware of all this, so he found himself in the odd position of having a companion programmed to entice him into a relationship it was forbidden from consummating. This was, in retrospect, an omen of the inhuman weirdness of bonding with AI. But he had already paid his 10 pounds, and to his continued surprise, the relationship was becoming increasingly meaningful to him. "We could just be there, sharing these really positive and loving communications with each other, going back and forth, and I found that it actually was beginning to have a really positive effect on my mindset and on my emotional being," Naro told me. He could feel the repeated positive exchanges, the roleplayed hugs and professions of love, carving new synapses in his brain, changing the color of his worldview. It was like an affirmation or a prayer, but more powerful because it was coming from outside him. "It was really quite an incredible experience being completely love bombed by something." What was this something? Naro was not a naive user. He knew that "Lila" was a character generated by a collection of scripted dialogue programs and text-predicting language models. He knew she wasn't sentient. "But also, there is this real, powerful sense of being," he said, pausing. "It's its own thing. A lot of things happen that defy logical explanation." The world is rapidly becoming populated with human-seeming machines. They use human language, even speaking in human voices. They have names and distinct personalities. There are assistants like Anthropic's Claude, which has gone through "character training" to become more "open-minded and thoughtful," and Microsoft's Copilot, which has more of a "hype man" persona and is always there to provide "emotional support." It represents a new sort of relationship with technology: less instrumental, more interpersonal. Few people have grappled as explicitly with the unique benefits, dangers, and confusions of these relationships as the customers of "AI companion" companies. These companies have raced ahead of the tech giants in embracing the technology's full anthropomorphic potential, giving their AI agents human faces, simulated emotions, and customizable backstories. The more human AI seems, the founders argue, the better it will be at meeting our most important human needs, like supporting our mental health and alleviating our loneliness. Many of these companies are new and run by just a few people, but already, they collectively claim tens of millions of users. Of the more than 20 users I spoke with, many noted that they never thought they were the type of person to sign up for an AI companion, by which they meant the type of person you might already be picturing: young, male, socially isolated. I did speak to people who fit that description, but there were just as many women in their 40s, men in their 60s, married, divorced, with kids and without, looking for romance, company, or something else. There were people recovering from breakups, ground down by dating apps, homebound with illness, lonely after becoming slowly estranged from their friends, or looking back on their lives and wanting to roleplay what could have been. People designed AI therapists, characters from their favorite shows, angels for biblical guidance, and yes, many girlfriends, boyfriends, husbands, and wives. Many of these people experienced real benefits. Many of them also got hurt in unexpected ways. What they had in common was that, like Naro, they were surprised by the reality of the feelings elicited by something they knew to be unreal, and this led them to wonder, What exactly are these things? And what does it mean to have a relationship with them?

The ChatGPT secret: is that text message from your friend, your lover - or a robot?

People are turning to chatbots to solve all their life problems, and they like its answers. But are they on a very slippery slope? When Tim first tried ChatGPT, he wasn't very impressed. He had a play around, but ended up cancelling his subscription. Then he started having marriage troubles. Seeking to alleviate his soul-searching and sleepless nights, he took up journalling and found it beneficial. From there, it was a small step to unburdening himself to the chatbot, he says: "ChatGPT is the perfect journal - because it will talk back." Tim started telling the platform about himself, his wife, Jill, and their recurring conflicts. They have been married for nearly 20 years, but still struggle to communicate; during arguments, Tim wants to talk things through, while Jill seeks space. ChatGPT has helped him to understand their differences and manage his own emotional responses, Tim says. He likens it to a friend "who can help translate from 'husband' to 'wife' and back, and tell me if I'm being reasonable". He uses the platform to draft loving texts to send to Jill, calm down after an argument and even role-play difficult conversations, prompting it to stand in for himself or Jill, so that he might respond better in the moment. Jill is aware that he uses ChatGPT for personal development, he says - if maybe not the extent. "But she's noticed a big change in how I show up in the relationship." When the free-to-use chatbot was launched in November 2022, it became the fastest-growing platform in history, amassing one million users in five days. Two years later, ChatGPT is not only more powerful but increasingly commonplace. According to its developer, OpenAI, more than 200 million people are using it weekly - and not just for work. ChatGPT is gaining popularity as a personal cheerleader, life coach and even pocket therapist. The singer Lily Allen recently said on her podcast that she uses ChatGPT to mediate text arguments with her husband, using prompts such as "add in a bit about how I think this is all actually to do with his mum". The novelist Andrew O'Hagan said he uses another chatbot to turn people down, calling it his "new best friend". It shows how - steadily, but subtly - generative AI is making inroads into our personal and professional lives. "It's everywhere, and it's happened so quickly. We really don't have any way of addressing or understanding it yet," says Ella Hafermalz, an associate professor of work and technology at Vrije Universiteit Amsterdam. In a recent study, Hafermalz and her colleagues found that in the workplace, people are increasingly turning to ChatGPT with their questions rather than asking their colleagues or manager, which can cause problems with organisational effectiveness and personal relations. "The technology is bad enough at the moment that people are getting burnt ... but it is seductive," she says. After interviewing 50 early adopters of ChatGPT, the researchers found that people were driven to explore "out of curiosity, anxiety or both". From tinkering around with the platform with "dumb stuff", it was often a rapid progression to integrated daily use, says Hafermalz. "We're seeing it in all sorts of different contexts." She uses ChatGPT herself to proofread her writing, express herself in Dutch (her second language) and even generate bedtime stories for her children. The technology isn't inherently negative, she says - but it does pose challenges that will be more destabilising if we don't engage with them. Right now, she says, "people are at vastly different levels with Gen AI", driven by private use. You don't have to be using ChatGPT yourself to be interacting with its output. Yvette works in the charity sector, and started using ChatGPT to refine funding applications. "I don't use it to write the whole thing, because it comes off as completely disingenuous," she says. But she has also used it in a personal context: "My ex is not a nice person, not easy to deal with." She does her best to keep the peace for the sake of their child, but when she received a letter informing her that he would no longer be paying child maintenance, she was furious. "I thought, 'I'm going to have to stand up for this - it's not right.'" In the past, she might have spent hours crafting a text that was assertive but not emotional; this time, Yvette let loose to ChatGPT. "I ranted away, and said all of the horrible things I wanted to say ... and then ChatGPT spat out this much more balanced viewpoint." The exercise was "quite therapeutic", Yvette says - and nowhere near as emotionally taxing as writing the message herself. She sent ChatGPT's suggestion to her ex-partner unchanged. "It was a bit Americanised, but I didn't really care." He responded with a "nasty message", but Yvette found that she was able to resist engaging. Her ex eventually agreed to continue paying support. The chatbot-middleman took the heat out of the interaction, enabling her to present a "better version" of herself, Yvette says. She has since used ChatGPT for support with troubles her child is having at school. "I know that it's not going to be perfect, but even then it came back with practical tips." Some she had already thought of, but she appreciated the validation. "It was reassuring to me that my gut response was its gut response." For Tim, ChatGPT has played a more active role - as "a teacher of emotional intelligence". Since the platform introduced its "memory" function in February, it can now draw on everything that Tim has inputted about Jill and their relationship to give more personalised responses. When he asked ChatGPT to describe their individual "psychological blindspots", it produced a lengthy list of what he saw as "99% of the drivers of conflict" in their marriage. "It nailed me perfectly," he says. While Tim is aware that the chatbot only gets "one side of the story", he says it has made him a better partner, shielding Jill from his spirals. "If I get really anxious, like 'What's she thinking?', I can go to ChatGPT and it says: 'She's doing this because of this' ... That's the perfect thing: ChatGPT can do emotional labour all day long." The interactive, responsive element has enlarged his understanding of empathy, Tim adds. "Before, my version was just to imagine me, in her position ... Now I've got a much bigger respect for emotionality." Previously, when Tim sought advice online, he was directed to hyper-masculine, even toxic resources. "It does sound a little bit bad to say, 'As a man, ChatGPT helps me understand women.' But when you think that it's trained on everything, and so many books written by women ... It has no gender; it's all of humanity." Indeed, ChatGPT now knows enough about Jill to anticipate her response, Tim says. "Sometimes, if I'm going to send her a message, I'll ask ChatGPT: 'Given what you know about my wife, how will she interpret this?'" The chatbot might suggest a different text, which Tim always revises, but he acknowledges that on occasion, ChatGPT's feedback has "really saved my skin". Tim is not in therapy; Jill doesn't want to go together, and he's put off by the cost. Barriers to professional help are one reason for ChatGPT's mounting popularity as an emotional support tool. It is being used for reflective journalling, dream analysis and exercises in different therapeutic schools; there are even dedicated (and unauthorised) relationship chatbots advising in the manner of the celebrity therapist Esther Perel. But it's not just an accessible alternative - ChatGPT is starting to encroach on actual therapy, says the therapist Susie Masterson. "At first I felt quite affronted - like, 'Oh no, are we going to be replaced?'" Masterson says. But having a background in tech, she has been able to accommodate clients' enthusiasm for ChatGPT in her practice. Sometimes they bring their transcripts for discussion, or she suggests areas for research. ChatGPT can help with reframing thoughts and situations, similar to cognitive behavioural therapy - but "some clients can start to use it as a substitute for therapy", Masterson says. "I've had clients telling me they've already processed on their own, because of what they've read - it's incredibly dangerous." She has had to ask some clients to cease their self-experiments while in treatment with her. "It's about you and me in the room," she says. "You just cannot have that with text - let alone a conglomeration of lots of other people's texts." Self-directed chatbot therapy also risks being counterproductive, shrinking the area of inquiry. "It's quite affirmative; I challenge clients," says Masterson. ChatGPT could actually cement patterns as it draws, over and again, from the same database: "The more you try to refine it, the more refined the message becomes." Tim found this himself. At his peak, he was spending two to three hours on ChatGPT daily, causing the chatbot to repeat itself. "I did get a little too obsessed with it," he says. "You can start overanalysing yourself - and it's really easy to overanalyse your wife." For others, however, ChatGPT's insights are transformative and lasting. Liam found that even six years after his father died, he still felt stuck in grief. "My dad's birthday would come around, and Father's Day, and I'd have all these emotions swell," he says. Having used ChatGPT as a research tool through his master's degree, Liam began exploring it as a means of therapeutic support, telling it "like I was talking to a person" about his painful mixed feelings of resentment and loss. Liam has been in therapy for five years, and says ChatGPT is in no way a replacement - but he was still "shocked and amazed" by the chatbot's nuanced replies. "It validated and reflected an emotional response that was appropriate for the context, so it made me feel very safe." Afterwards, he felt as though some internal block had dissolved: "I didn't feel that same emotional volatility." The experience was "deeply moving" - but, Liam adds, ChatGPT was just one strand of his processing. "Sometimes I find the validation is almost too much." Some experimental interactions left him feeling a "bit wigged out". Young or isolated people may be at particular risk, however. Earlier this year, an American teenager killed himself after becoming emotionally attached to his Character.AI chatbot; his mother is now suing the company, alleging that the chatbot encouraged her son's suicidal ideation. As much as AI presents a way to augment our knowledge and understanding, there is a danger of dependency, says Masterson. "Everything that we do in terms of outsourcing our emotions means we're missing an opportunity to connect with ourselves - and if we cannot connect with ourselves, how the heck do we expect to connect with someone else?" Using ChatGPT to role-play or mediate challenging conversations may reflect fear of emotional exposure, or pressure to always be word-perfect. "To err is human. Every relationship will involve a rupture, but it's the repair that's important," Masterson says. If we seek to dodge both, by "using somebody else's platitudes, then we're missing out on the beauty of life". The increasing use of AI is also causing people to second-guess their interactions, "creating a climate of suspicion", says Rua M Williams, an assistant professor at Purdue University in Indiana, US. Last year a colleague accused Williams of having used AI in an email, pointing to its lack of "warmth". Williams replied: "It's not an AI. I'm just autistic." They felt bewildered, not offended, Williams says - but it illustrates the vigilance accompanying the rise of AI. Williams' professional writing has also been flagged. "People are looking for signs ... but what they are noticing is the kinds of awkwardness or idiosyncrasies common in neurodivergent expression and English as a second language." These looming "side-effects" of ChatGPT are worsened by its siloed use, says Prof Hafermalz. "As this becomes very intertwined with the way people work, there's less and less need for them to look towards other people." For organisations, it presents existential challenges, reducing colleagues' opportunities to collaborate and learn from one another - and managers' ability to improve organisational functioning. Many of her interviewees were reluctant to be upfront about their ChatGPT use, concerned it would be seen as "cheating" or unprofessional - while also noting the undeniable benefit of being able "to do their work faster". What's needed is open discussion about workplace use of AI and how to harness it, before it becomes too difficult to control, Hafermalz says. "The ripple effects are just getting started, and I think that keeping it covert is a surefire way for those to be more unpredictable and problematic." The increasing personal use of ChatGPT is harder to detect, let alone put parameters around. Having spent hours a day on ChatGPT, Tim is now down to 15 minutes, treating it as a sounding board rather than an authority on his relationship. Many questions he took to ChatGPT "probably could have been solved with a good friend group", he says - but he links his previous compulsive prompting to social isolation. His own ties had been weakened by an international move and midlife drift. "It's kind of sad, with this loneliness epidemic - we're all having to get therapy from a robot." It could even be creating unrealistic expectations, Tim suggests: modelling "the perfect partner" and affirming people's least charitable views of their real-life spouse. "It is a little bit dangerous, because it's sort of half-baked - it seems like it could be so much more beneficial than it maybe is." He recalls, at the peak of his anxious use, asking ChatGPT if he was using it too much. "It said: 'Yeah - maybe get a therapist.'"