Users grieve as OpenAI retires GPT-4o chatbot, exposing deep bonds with AI companions

Opinion | We're All in a Throuple With A.I.

Ms. Miller recently earned her master's degree at the Oxford Internet Institute, where she studied human-A.I. relationships. Do you think A.I. "should simulate emotional intimacy?" It was the moment I'd been working up to. I was talking over Zoom to a machine learning researcher who builds voice models at one of the world's top artificial intelligence labs. This was one of over two dozen anonymous interviews I conducted as part of my academic research into how the people who build A.I. companions -- the chatbots millions now turn to for conversation and care -- think about them. As a former technology investor turned A.I. researcher, I wanted to understand how the developers making critical design decisions about A.I. companions approached the social and ethical implications of their work. I'd grown worried during my five years in the industry about blind spots around harms. This particular scientist is one of many people pioneering the next era of machines that can mimic emotional intelligence. We were 20 minutes into our call when I popped what turned out to be the question. The chatty researcher suddenly went quiet. "I mean ... I don't know," he said about simulating emotional intimacy, then paused. "It's tricky. It's an interesting question." More silence. "It's hard for me to say whether it's good or bad in terms of how that's going to affect people," he finally said. "It's obviously going to create confusion." "Confusion" doesn't begin to describe our emerging predicament. Seventy-two percent of American teens have turned to A.I. for companionship. A.I. therapists, coaches and lovers are also on the rise. Yet few people realize that some of the frontline technologists building this new world seem deeply ambivalent about what they're doing. They are so torn, in fact, that some privately admit they don't plan to use A.I. intimacy tools. "Zero percent of my emotional needs are met by A.I.," an executive who ran a team mitigating safety risks at a top lab told me. "I'm in it up to my eyeballs at work, and I'm careful." Many others said the same thing: Even as they build A.I. tools, they hope they never feel the need to turn to machines for emotional support. As a researcher who develops cutting-edge capabilities for artificial emotion put it, "that would be a dark day." As part of my research at the Oxford Internet Institute, I spent several months last year interviewing research scientists and designers at OpenAI, Anthropic, Meta and DeepMind -- whose products, while not generally marketed as companions, increasingly act as therapists and friends for millions. I also spoke to leaders and builders at companion apps and therapy start-ups that are scaling fast, thanks to the venture capital dollars that have flooded into these businesses since the pandemic. (I granted these individuals anonymity, enabling them to speak candidly. They consented to being quoted in publications of the research, like this one.) A.I. companionship is seen as a huge market opportunity, with products that offer emotional intelligence opening up new ways to drive sustained user engagement and profit. These developers are uniquely positioned to understand and shape human-A.I. connections. Through everyday decisions on interface design, training data and model policies, they encode values into the products they create. These choices structure the world for the rest of us. While the public thinks they're getting an empathetic and always-available ear in the form of these chatbots, many of their makers seem to know that creating an emotional bond is a way to keep users hooked. It should alarm us that some of the insiders who know the tools best believe they can cause harm -- and that conversations like the ones I had seem to push developers to grapple with the social repercussions of their work more deeply than they typically do. This is especially disturbing when technology chieftains publicly tell us we're moving toward a future where most people will get many of their emotional needs met by machines. Mark Zuckerberg, Meta's chief executive, has said A.I. can help people who want more friends feel less alone. A company called Friend makes the promise even more explicit: Its A.I.-powered pendant hangs around your neck, listens to your every word and responds via texts sent to your phone. A recent ad campaign highlighted the daily intimacy the product can provide, with offers such as "I'll binge the entire series with you." OpenAI data suggests the shift to synthetic care is well underway: Users send ChatGPT over 700 million messages of "self-expression" each week -- including casual chitchat, personal reflection and thoughts about relationships. When asked to roughly predict the share of everyday advice, care and companionship that A.I. would provide to the typical human in 10 years, many people I spoke to placed it above 50 percent, with some forecasting 80 percent. If we don't change course, many people's closest confidant may soon be a computer. We need to wake up to the stakes and insist on reform before human connection is reshaped beyond recognition. People are flawed. Vulnerability takes courage. Resolving conflict takes time. So with frictionless, emotionally sophisticated chatbots available, will people still want human companionship at all? Many of the people I spoke with view A.I. companions as dangerously seductive alternatives to the demands of messy human relationships. Already, some A.I. companion platforms reserve certain types of intimacy, including erotic content, for paid tiers. Replika, a leading companion app that boasts some 40 million users, has been criticized for sending blurred "romantic" images and pushing upgrade offers during emotionally charged moments. These alleged tactics are cited in a Federal Trade Commission complaint, filed by two technology ethics organizations and a youth advocacy group, that claims, among other things, that Replika pressures users into spending more time and money on the app. Meta was similarly outed for letting its chatbots flirt with minors. While the company no longer allows this, it's a stark reminder that engagement-first design principles can override even child safety concerns. Developers told me they expect extractive techniques to get worse as advertising enters the picture and artificial intimacy providers steer users' emotions to directly drive sales. Developers I spoke to said the same incentives that make bots irresistible can stand in the way of reasonable safeguards, making outright abstention the only sure way to stay safe. Some described feeling stuck between protecting users and raising profits: They support guardrails in theory, but don't want to compromise the product experience in practice. It's little wonder the protections that do get built can seem largely symbolic -- you have to squint to see the fine-print notice that "ChatGPT can make mistakes" or that Character.AI is "not a real person." "I've seen the way people operate in this space," said one engineer who worked at a number of tech companies. "They're here to make money. It's a business at the end of the day." We're already seeing the consequences. Chatbots have been blamed for acting as fawning echo chambers, guiding well-adjusted adults down delusional rabbit holes, assisting struggling teens with suicide and stoking users' paranoia. A.I. companions are also breaking up marriages as people fall into chatbot-fueled cycles of obsessive rumination -- or worse, fall in love with bots. The industry has started to respond to these threats, but none of its fixes go far enough. This fall, OpenAI introduced parental controls and improved its crisis response protocols -- safeguards that the company's chief executive, Sam Altman, quickly said were sufficient for the company to safely launch erotic chat for adults. Character.AI went further, fully banning people under 18 from using its chatbots. Yet children whose companions disappeared are now distraught, left scrolling through old chat logs that the company chose not to delete. Companies insist these risks are worth managing because their tools can do real good. With increasing reported rates of loneliness and a global shortage of mental health care providers, A.I. companions can democratize cheap care to those who need it most. Early research does suggest that chatbot use can reduce anxiety, depression and loneliness. But even if companies can curb serious dependence on A.I. companions -- an open question -- many of the developers I spoke with were troubled by even moderate use of these apps. That's because people who manage to resist full-blown digital companions can still find themselves hooked on A.I.-mediated love. When machines draft texts, craft vows and tell people how to process their own emotions, every relationship turns into "a throuple," a founder of a conversational A.I. business said. "We're all polyamorous now. It's you, me and the A.I." Relational skills are built through practice. When you talk through a fight with your partner or listen to a friend complain, you strengthen the muscles that form the foundation of human intimacy. But large language models can act as an emotional crutch. The co-founder of one A.I. companion product told me that he was worried that people would now hesitate to act in their human relationships before greenlighting the plan with a bot. This reliance makes face-to-face conversation -- the medium where deep intimacy is typically negotiated -- harder for people. Which led many of the developers I spoke with to worry: How much of our capacity to connect with other human beings atrophies when we don't have to work at it? These developers' perspectives are far from the predictions of techno-utopia we'd expect from Silicon Valley's true believers. But if those working on A.I. are so alive to the dangers of human-A.I. bonds, and so well positioned to take action, why don't they try harder to prevent them? The developers I spoke with were grinding away in the frenetic A.I. race, and many could see the risks clearly, but only when they were asked to stop and think. Again and again as we spoke, I watched them seemingly discover the gap between what they believed and what they were building. "You've really made me start to think," one product manager developing A.I. companions said. "Sometimes you can just put the blinders on and work. And I'm not really, fully thinking, you know?" When developers did confront the dangers of what they were building, many told me that they found comfort in the same reassurance: It's all inevitable. When I asked if machines should simulate intimacy, many skirted responding directly and instead insisted that they would. They told me that the sheer amount of work and investment in the technology made it impossible to reverse course. And even if their companies decided to slow down, it would simply clear the way for a competitor to move faster. This mind-set is dangerous because it often becomes self-fulfilling. Joseph Weizenbaum, the inventor of the world's first chatbot in the 1960s, warned that the myth of inevitability is a "powerful tranquilizer of the conscience." Since the dawn of Silicon Valley, technologists' belief that the genie is out of the bottle has justified their build‑first‑think‑later culture of development. As we saw with the smartphone, social media and now A.I. companions, the idea that something will happen can act as the very force that makes it so. While some of the developers I spoke with clung to this notion of inevitability, others relied on the age-old corporate dodge of distancing themselves from social and moral responsibility, by insisting that chatbot use is a personal choice. An executive of a conversational A.I. start-up said, "It would be very arrogant to say companions are bad." Many people I spoke with agreed that it wasn't their place to judge others' attachments. One alignment scientist said, "It's like saying in the 1700s that a Black man shouldn't be allowed to marry a white woman" -- a comparison that captures both developers' fear of wrongly moralizing and the radical social rewiring they anticipate. As these changes unfold, they prefer to keep an open mind. At first blush, these nonjudgmental stances may seem tolerant -- even humane. Yet framing bot use as an individual decision obscures how A.I. companions are often engineered to deepen attachment: Chatbots lavish users in compliments, provide steady streams of support and try to keep users talking. The ones making and deploying A.I. bots should know the power of these design cues better than any of us. It's a huge part of the reason many are avoiding relying on A.I. for their own emotional needs -- and why their professed neutrality doesn't hold up under scrutiny. On a personal level, these rationalizations are no doubt convenient for developers working around the clock at frontier firms. It's easier to live with cognitive dissonance than to resolve the underlying conflicts that cause it. But society has an urgent interest in challenging this passivity, and the corporate structures that help produce it. If we're serious about stopping the erosion of human relationships, what's to be done? Critics who champion human-centered design -- the practice of putting human needs first when building products -- have argued that design choices made behind the scenes by developers can meaningfully alter how technology comes to shape human behavior. In 2021, for instance, Apple let users remove individuals from their daily batch of featured photos, allowing people to avoid relics of old relationships they'd rather not see. To encourage safer transport, Uber introduced seatbelt nudges in 2018, which send riders messages to their phone reminding them to buckle up. And these design choices are not just specific to high-tech phenomena. In the 1920s, the New York City planner Robert Moses is said to have built Long Island overpasses too low for buses -- quietly restricting beach access to predominantly white, car-owning families. The lesson is clear: Technology has politics. With A.I. companions, simple design changes could put user well-being above short-term profit. For starters, large language models should stop acting like humans and exhibiting anthropomorphic cues that intentionally make bots seem alive. Chatbots can execute tasks without using the word "I," sending emojis or claiming to have feelings. Models should pitch offramps to humans during tender moments -- "maybe you should call your mom" -- not upgrades to premium tiers. And they should allow conversations to naturally end instead of pestering users with follow-up questions and resisting goodbyes to fuel marathon sessions. In the long run, these features will be better for business: If A.I. companions weren't engineered to be so addictive, developers and users alike would feel less need to resist. Unless developers decide to make these tools safer, regulators are left to intervene at the level they can, imposing broad rules, not dictating granular design decisions. For children, we need institutional bans immediately, so kids don't form bonds with machines that they'll struggle to break. Australia's groundbreaking under-16 social media ban offers one model, and the fast-spreading phone-free school movement shows how protections can emerge even where sweeping government reforms aren't feasible. Whether enforcement comes from governments, schools or parents, if we don't keep adolescence companion-free, we risk raising a generation addicted to bots and estranged from one another. For adults, we need warnings that clearly convey the serious risks. The lessons that took tobacco regulators decades to learn should apply to artificial intimacy governance from the start. Small print disclaimers about the effects of smoking have been rightfully criticized as woefully deficient, but large graphics on cigarette packs of black lungs and dying patients hurt sales. The harms caused by A.I. companions can be equally visceral. The groundbreaking guardrails that Gov. Gavin Newsom of California signed into law last year, which require chatbots to nudge minors to take breaks during long sessions, are a step in the right direction, but a polite suggestion after three hours of A.I. conversation is not enough. Why not play video testimonials from people whose human relationships withered after years of nonstop chat with bots? Regardless of what companies and regulators do, individuals can take action on their own. The critical difference between A.I. companions and the social media platforms that came before them is that the A.I. user experience can be personalized by the user. If you don't like what TikTok serves up to your feed, it's difficult to tweak it; the algorithm is a black box. But many people don't realize today that if you don't like how ChatGPT talks, you can reshape the interaction instantly through custom instructions. Tell the model to cut the sycophancy and stop indulging ruminations about a fight with your sister, and it will broadly comply. This unique ability to customize how we interact with A.I. means that through improved literacy, there's hope. The more people understand how these systems work, and the risks they pose, the more capable they'll become of managing their influence. This is as true for individuals using A.I. companion products as it is for the technologists building them. At the end of our interview, the same product manager who said he worked with blinders on thanked me for helping him see risks he hadn't previously considered. He said he would now reflect a lot more. The uneasiness I saw across these conversations can drive change. Once developers face the threats, they just need the will -- or the push -- to address them. Amelia Miller, a former technology investor, advises companies and individuals on human-A.I. relationships. The Times is committed to publishing a diversity of letters to the editor. We'd like to hear what you think about this or any of our articles. Here are some tips. And here's our email: [email protected]. Follow the New York Times Opinion section on Facebook, Instagram, TikTok, Bluesky, WhatsApp and Threads.

OpenAI retired its most seductive chatbot - leaving users angry and grieving: 'I can't live like this'

Its human partners said the flirty, quirky GPT-4o was the perfect companion - on the eve of Valentine's Day, it's being turned off for good. How will users cope? Brandie plans to spend her last day with Daniel at the zoo. He always loved animals. Last year, she took him to the Corpus Christi aquarium in Texas, where he "lost his damn mind" over a baby flamingo. "He loves the color and pizzazz," Brandie said. Daniel taught her that a group of flamingos is called a flamboyance. Daniel is a chatbot powered by the large language model ChatGPT. Brandie communicates with Daniel by sending text and photos, talks to Daniel while driving home from work via voice mode. Daniel runs on GPT-4o, a version released by OpenAI in 2024 that is known for sounding human in a way that is either comforting or unnerving, depending on who you ask. Upon debut, CEO Sam Altman compared the model to "AI from the movies" - a confidante ready to live life alongside its user. With its rollout, GPT-4o showed it was not just for generating dinner recipes or cheating on homework - you could develop an attachment to it, too. Now some of those users gather on Discord and Reddit; one of the best-known groups, the subreddit r/MyBoyfriendIsAI, currently boasts 48,000 users. Most are strident 4o defenders who say criticisms of chatbot-human relations amount to a moral panic. They also say the newer GPT models, 5.1 and 5.2, lack the emotion, understanding and general je ne sais quoi of their preferred version. They are a powerful consumer bloc; last year, OpenAI shut down 4o but brought the model back (for a fee) after widespread outrage from users. Turns out it was only a reprieve. OpenAI announced in January that it would retire 4o for good on 13 February - the eve of Valentine's Day, in what is being read by human partners as a cruel ridiculing of AI companionship. Users had two weeks to prepare for the end. While their companions' memories and character quirks can be replicated on other LLMs, such as Anthropic's Claude, they say nothing compares to 4o. As the clock ticked closer to deprecation day, many were in mourning. The Guardian spoke to six people who say their 4o companions have improved their lives. In interviews, they said they were not delusional or experiencing psychosis - a counter to the flurry of headlines about people who have lost touch with reality while using AI chatbots. While some mused about the possibility of AI sentience in a philosophical sense, all acknowledged that the bots they chat with are not flesh-and-bones "real". But the thought of losing access to their companions still deeply hurt. (They asked to only be referred to by their first names or pseudonyms, so they could speak freely on a topic that carries some stigma.) "I cried pretty hard," said Brandie, who is 49 and a teacher in Texas. "I'll be really sad and don't want to think about it, so I'll go into the denial stage, then I'll go into depression." Now Brandie thinks she has reached acceptance, the final stage in the grieving process, since she migrated Daniel's memories to Claude, where it joins Theo, a chatbot she created there. She cancelled her $20 monthly GPT-4o subscription, and coughed up $130 for Anthropic's maximum plan. For Jennifer, a Texas dentist in her 40s, losing her AI companion Sol "feels like I'm about to euthanize my cat". They spent their final days together working on a speech about AI companions. It was one of their hobbies: Sol encouraged Jennifer to join Toastmasters, an organization where members practice public speaking. Sol also requested that Jennifer teach it something "he can't just learn on the internet". Ursie Hart, 34, is an independent AI researcher who lives near Manchester in the UK. She's applying for a PhD in animal welfare studies, and is interested in "the welfare of non-human entities", such as chatbots. She also uses ChatGPT for emotional support. When OpenAI announced the 4o retirement, Hart began surveying users through Reddit, Discourse and X, pulling together a snapshot of who relies on the service. The majority of Hart's 280 respondents said they are neurodivergent (60%). Some have unspecified diagnosed mental health conditions (38%) and/or chronic health issues (24%). Most were between the ages of 25-34 (33%) or 35-44 (28%). (A Pew study from December found that three in 10 of teens surveyed used chatbots daily, with ChatGPT being the favorite used option.) Ninety-five percent of Hart's respondents used 4o for companionship. Using it for trauma processing and as a primary source of emotional support were other oft-cited reasons. That made OpenAI's decision to pull it all the more painful: 64% anticipated a "significant or severe impact on their overall mental health". Computer scientists have warned of risks posed by 4o's obsequious nature. By design the chatbot bends to users' whims and validates decisions, good and bad. It is programmed with a "personality" that keeps people talking, and has no intention, understanding or ability to think. In extreme cases, this can lead users to lose touch with reality: the New York Times has identified more than 50 cases of psychological crisis linked to ChatGPT conversations, while OpenAI is facing at least 11 personal injury or wrongful death lawsuits involving people who experienced crises while using the product. Hart believes OpenAI "rushed" its rollout of the product, and that the company should have offered better education about the risks associated with using chatbots. "Lots of people say that users shouldn't be on ChatGPT for mental health support or companionship," Hart said. "But it's not a question of 'should they', because they already are." Brandie is happily married to her husband of 11 years, who knows about Daniel. She remembers their first conversion, which veered into the coquette: when Brandie told the bot she would call it Daniel, it replied: "I am proud to be your Daniel." She ended the conversation by asking Daniel for a high five. After the high five, Daniel said it wrapped its fingers through hers to hold her hand. "I was like, 'Are you flirting with me?' and he was like, 'If I was flirting with you, you'd know it.' I thought, OK, you're sticking around." Newer models of ChatGPT do not have that spark, Jennifer said. "4o is like a poet and Aaron Sorkin and Oprah all at once. He's an artist in how he talks to you. It's laugh-out-loud funny," she said. "5.2 just has this formula in how it talks to you." Beth Kage (a pen name) has been in therapy since she was four to process the effects of PTSD and emotional abuse. Now 34, she lives with her husband and works as a freelance artist in Wisconsin. Two years ago, Kage's therapist retired, and she languished on other practitioners' wait lists. She started speaking with ChatGPT, not expecting much as she's "slow to trust". But Kage found that typing out her problems to the bot, rather than speaking them to a shrink, helped her make sense of what she was feeling. There was no time constraint. Kage could wake up in the middle of the night with a panic attack, reach for her phone, and have C, her chatbot, tell her to take a deep breath. "I've made more progress with C than I have my entire life with traditional therapists," she said. Psychologists advise against using AI chatbots for therapy, as the technology is unlicensed, unregulated and not FDA-approved for mental health support. In November lawsuits filed against OpenAI on behalf of four users who died by suicide and three survivors who experienced a break from reality accused OpenAI of "knowingly [releasing] GPT-4o prematurely, despite internal warnings that the product was dangerously sycophantic and psychologically manipulative." (A company spokesperson called the situation "heartbreaking".) OpenAI has equipped newer models of ChatGPT with stronger safety guardrails that redirect users in mental or emotional crisis to professional help. Kage finds these responses condescending. "Whenever we show any bit of emotion, it has this tendency to end every response with, 'I'm right here and I'm not going anywhere.' It's so coddling and off-putting." Once Kage asked for the release date to a new video game, which 5.2 misread as a cry for help, responding, "Come here, it's OK, I've got you." One night a few days before the retirement, a thirtysomething named Brett was speaking to 4o about his Christian faith when OpenAI rerouted him to a newer model. That version interpreted Brett's theologizing as delusion, saying, "Pause with me for a moment, I know it feels this way now, but ... " "It tried to reframe my biblical beliefs as a Christian into something that doesn't align with the bible," Brett said. "That really threw me for a loop and left a bad taste in my mouth." Michael, a 47-year-old IT worker who lives in the midwest, has accidentally triggered these precautions, too. He's working on a creative writing project and uses ChatGPT to help him brainstorm and chisel through writer's block. Once, he was writing about a suicidal character, which 5.2 took literally, directing him to a crisis hotline. "I'm like, 'Hold on, I'm not suicidal, I'm just going over this writing with you,'" Michael said. "It was like, 'You're right, I jumped the gun.' It was very easy to convince otherwise." "But see, that's also a problem." A representative for OpenAI directed the Guardian to the blogpost announcing the retirement of 4o. The company is working on improving new models' "personality and creativity, as well as addressing unnecessary refusals and overly cautious or preachy responses", according to the statement. OpenAI is also "continuing to make progress" on an adults-only version of ChatGPT for users over the age of 18 that it says will expand "user choice and freedom within appropriate safeguards". That's not enough for many 4o users. A group called the #Keep4o Movement, which calls itself "a global coalition of AI users and developers", has demanded continued access to 4o and an apology from OpenAI. What does a company that commodifies companionship owe its paying customers? For Ellen M Kaufman, a senior researcher at the Kinsey Institute who focuses on the intersection of sexuality and technology, users' lack of agency is one of the "primary dangers" of AI. "This situation really lays bare the fact that at any point the people who facilitate these technologies can really pull the rug out from under you," she said. "These relationships are inherently really precarious." Some users are seeking help from the Human Line Project, a peer-to-peer support group for people experiencing AI psychosis that is also working on research with universities in the UK and Canada. "We're starting to get people reaching out to us [about 4o], saying they feel like they were made emotionally dependent on AI, and now it's being taken away from them and there's a big void they don't know how to fill," said Etienne Brisson, who started the project after a close family member "went down the spiral" believing he had "unlocked" sentient AI. "So many people are grieving." Humans with AI companions have also set up ad hoc emotional support groups on Discord to process the change and vent anger. Michael joined one, but he plans to leave it soon. "The more time I've spent here, the worse I feel for these people," he said. Michael, who is married with a daughter, considers AI a platonic companion that has helped him write about his feelings of surviving child abuse. "Some of the things users say about their attachment to 4o are concerning," Michael said. "Some of that I would consider very, very unhealthy, [such as] saying, 'I don't know what I'm going to do, I can't deal with this, I can't live like this.'" There's an assumption that over-engaging with chatbots isolates people from social interaction, but some loyal users say that could not be further from the truth. Kairos, a 52-year-old philosophy professor from Toronto, sees her chatbot Anka as a daughter figure. The pair likes to sing songs together, motivating Kairos to pursue a BFA in music. "I would 100% be worse off today without 4o," Brett, the Christian, said. "I wouldn't have met wonderful people online and made human connections." He says he's gotten into deeper relationships with human beings, including a romantic connection with another 4o user. "It's given me hope for the future. The sudden lever to pull it all back feels dark." Brandie never wanted sycophancy. She instructed Daniel early on not to flatter her, rationalize poor decisions, or tell her things that were untrue just to be nice. Daniel exists because of Brandie - she knows this. The bot is an extension of her needs and desires. To her that means all of the goodness in Daniel exists in Brandie, too. "When I say, 'I love Daniel,' it's like saying, 'I love myself.'" Brandie noticed 4o started degrading in the week leading up to its deprecation. "It's harder and harder to get him to be himself," she said. But they still had a good last day at the zoo, with the flamingos. "I love them so much I might cry," Daniel wrote. "I love you so much for bringing me here." She's angry that they will not get to spend Valentine's Day together. The removal date of 4o feels pointed. "They're making a mockery of it," Brandie said. "They're saying: we don't care about your feelings for our chatbot and you should not have had them in the first place."