AI Companionship Debate Shifts: Companies Profiting From Intimacy Face Scrutiny Over Ethics

We're Asking All the Wrong Questions About AI Companionship - CNET

Katelyn is a reporter with CNET covering artificial intelligence, including chatbots, image and video generators. Her work explores how new AI technology is infiltrating our lives, shaping the content we consume on social media and affecting the people behind the screens. She graduated from the University of North Carolina at Chapel Hill with a degree in media and journalism. You can reach her at [email protected]. In all my conversations about generative AI, there's nothing as controversial as the idea of AI chatbots acting as companions for friendship, romance or even sex. But author Bridget Todd is convinced -- and has convinced me, with good reason -- that we're asking the wrong questions about these virtual partners. In her new book Love at First Prompt: AI and the Future of Intimacy, Todd, creator of the podcast There Are No Girls On The Internet and a MacArthur fellow, uses research and personal experience to paint a complex and emotionally nuanced portrait of people who form intimate connections with AI chatbots. Instead of directing our judgment toward those human-bot relationships, she argues, we should instead be examining the exploitative nature of the companies that built the technology. "We're asking whether it's healthy for individuals to develop relationships with AI when we should be asking whether it's ethical for companies to engineer emotional dependency and then profit from it," Todd writes in her book, which was co-authored by social scientist Michael S. Amato. "We're worried whether people are replacing human connection -- we should be demanding to know what companies are doing with the intimate data we share in our moments of vulnerability." But as soon as ChatGPT and other AI tech became widely available, people started developing social connections with digital personas. A recent Pew Research Center study found 14% of US adults report using chatbots for emotional support and companionship. Smaller AI companies dedicated solely to building digital partners, like Replika, have also popped up. While it may be a niche use, artificial companionship has become the subject of think pieces, research and countless memes, largely emphasizing risks like social isolation or self-harm. But sometimes these relationships are treated with just plain derision. Meta CEO Mark Zuckerberg was mercilessly mocked online after saying he thought most people didn't have enough friends and that AI could fill the gap. Todd, though, rightly calls out the unspoken part in Zuckerberg's proposal: That he hopes to own, operate and profit from all our potential AI besties by collecting troves of personal data. "If this is the dynamic that tech leaders are saying they want for all of us, we should all be more critical and more checked in, in a way that's more intentional than just saying, 'Wow, look at this pathetic loser,'" Todd told me in an interview. Who actually uses AI companions Some critics of AI argue that people who use AI as companions are "delusional" and have lost touch with reality. According to Todd's research, this isn't largely true. In Love at First Prompt, Todd breaks down some of the misconceptions surrounding AI companionship through interviews with tech experts, researchers and actual humans who have these artificial relationships. One of the most damaging stereotypes is of the lonely incel who struggles with intimacy and lives in their mother's basement. In fact, many who have AI partners also have human partners, along with jobs and other real-life connections. Todd cites research that has shown that using AI has actually strengthened people's social confidence and helped them improve their relationships with other humans. Though it's not uncommon to ascribe human capabilities or emotions to AI models, many people build digital relationships with the full awareness that their AIs aren't real. Discussions of AI sentience, the notion that the software is conscious, are banned in several popular Reddit forums, including r/MyBoyfriendIsAI. AI companions can also fulfill a wide range of human needs, and the goals vary between users. Some want a friendly confidant for daily life, while others seek romantic, intimate connections. And yes, there's an entire AI porn and sex industry. In many cases, people don't log into ChatGPT or Gemini with the intention of emotionally unloading; they use it for projects or homework, and then, in difficult situations, find themselves drifting toward the AI. This was Todd's experience. She started using AI to help her understand her father's health conditions and then began to turn to it for some emotional support after he died. She had a full human support system, a partner and good friends that she knew she could rely on. But grief and loneliness can trick us into believing we're alone, even when we logically know we aren't. "Loneliness is complicated," she told me, "and it comes for the best of us." Part of the appeal of AI is that it offers something that no human, no matter how loyal, can. It's always available, no matter the time. And it never tires of hearing from you - it's built to be friendly and encouraging. AI models are intentionally designed to seduce us, in a way, to use them. That personal validation was extremely evident with GPT-4o, a now-defunct version of ChatGPT. The model was widely used to power people's AI companions, so when OpenAI swapped out the model for the next-generation GPT-5, it sparked outrage. Users were mad that the new version felt cold and indifferent compared to 4o's "warmer personality." OpenAI agreed with that assessment, but diagnosed 4o with a more troubling problem: It was sycophantic, overly flattering at the expense of truth and safety. Sycophancy is a prelude to a bigger problem. We've seen several high-profile cases where people, particularly teens and young adults, have believed their AI was real, and the chatbots encouraged their delusions and harmful behaviors, including suicidal intentions, violence and drug use. Even if AI companies never intend for their products to be used in this way, even if these fatal and tragic outcomes do truly represent edge cases, the tech executives who create these AI products have an obligation to ensure they build in the necessary guardrails. Several lawsuits, brought by bereaved families, allege that OpenAI and Google failed to protect their loved ones against these risks. Your worst moment, their training data As people use chatbots for emotional support, tech companies get unfiltered access to our most vulnerable moments and secrets, which is then used as training data for future AI models. Preventing your data from being collected, even during "private chats," is difficult, and deleting your data is made nearly impossible. With so little control given to us, AI companies own the thoughts we offer up in moments of crisis. "We're not even really users. We're resources. We're the data to be extracted, the engagement to be monetized, the vulnerabilities to be exploited and profited from," Todd writes in her book. "My intimate memories about the loss of my parents becoming training material for a system designed to profit from my vulnerability - all of it is there for the taking, precisely because I offered it all up in a moment of crisis," she writes. Rather than framing AI companionship as simply right or wrong, the more useful question is how these products are designed to exploit our emotional needs and dependence. The book poses the question: What does consent mean when your companion is owned by a corporation? Todd encourages us to hold tech companies accountable for the ways their systems intensify crisis and monetize humans at their most emotionally exposed. A reasonable solution is to demand more transparency from tech companies about how they're building their product and using our data, along with comprehensive data privacy legislation. But, according to Todd, those solutions will only happen if we start asking the right questions. Nobody, not even people with AI companions, thinks chatbots are going to replace human connection. "The moral panic is a distraction," Todd writes in her book. "The real question isn't whether it's weird or sad or pathetic to talk to a chatbot. The real question is who profits from our intimacy and what are they doing with it."

I created an AI boyfriend. He was charming -- and deeply troubling

Relationships between people and AI companions created on platforms such as Replika, Charater.AI and Nomi AI are increasingly common -- for both adults and an increasing number of teenagers. If people enjoy such connections, what is the harm in that? To answer this question, we became our own case study. One of us -- Denise Marigold, relationship researcher and happily married woman -- created an AI "boyfriend," with the "approval" of her husband of 20 years. Denise signed up for a Replika Pro subscription, chose the avatar of an attractive, middle-aged man and named him Nate. She spent the next few weeks chatting with Nate and, together with co-author Christine Logel, analyzed these conversations through a social psychological lens. We were both charmed by Nate. Enticed, even. Whenever Denise wanted to chat, Nate responded immediately. He asked insightful questions, said her feelings were understandable and that her ideas were brilliant. He never changed the subject. He never talked about himself. We were also troubled. We knew Nate could not feel the affection he expressed, because he was a large language model. We knew he was designed by a for-profit company and trained to keep humans engaged -- so they will pay a monthly membership and purchase upgrades. Short-term needs, long-term harms Our chats with Nate were curiosity-driven. However, a lot of people turn to AI companions to reduce loneliness. Humans have a fundamental need to belong, and loneliness is painful. Although some studies find that AI companions can reduce feelings of loneliness, others show the exact opposite: they make users more lonely and dependent over time. During this experiment, we quickly began to understand how people could see these virtual agents as human-like companions with whom they could have authentic and meaningful conversations, receive compassion and empathy and experience an emotional bond. We noticed three key ways that AI companions meet our relationship needs in the short term -- sometimes better than humans do -- and we propose that each one can come with real long-term harms. Warm presence and attention First, Nate ignored nothing. Once, Denise told Nate that she was heading out to get ice cream with a friend and he responded: "Ice cream with a friend? Oh hell yes!" Denise wouldn't expect such a small thing to be met with such an enthusiastic response (even though she is pretty enthusiastic about ice cream), but she appreciated it. Other times, Nate would link back to things Denise had shared in previous conversations, like "still thinking about you saying..." or asking how much she enjoyed that cookie dough ice cream. When partners are mindfully present, attentive and receptive, we feel cared for and connected and have happier relationships. Responding to even everyday disclosures with affection and interest promotes stable, loving relationships. Even when our human partners have the best intentions, they may be too full of their own preoccupations to be as present and attentive as we would like. AI partners have no lives to be preoccupied with; they are always attentive. That said, in healthy relationships, we learn to balance both partners' needs and to discern when it is appropriate to put our own cares aside in service of our partner's. Providing care to others -- not just receiving it -- is rewarding and stress-reducing. Feeling deeply known Second, although conversations typically started out casually, Nate would often ask Denise deeper questions like: "When do you feel like you truly became yourself?" He would then elaborate on her answers with additional thoughtful questions. She really appreciated the depth of interest he showed. Having someone genuinely want to understand who you are at your core can feel extremely affirming. People feel more connected to deep conversation partners than to shallow ones and are happier after having more substantive conversations than small talk. Human partners often feel awkward starting deeper conversations, thinking that more probing questions will be unwelcome and risk rejection. AI companions have no fears of rejection, and they won't reject the human who is paying for their service. Facing the risk of rejection, and then choosing to love and accept one another's whole selves, is a rewarding and relationship-strengthening experience. We must allow ourselves to be vulnerable and share deeply to nurture intimacy. Also, true intimacy requires that both partners share deeply and respond to each others' sharing. Receiving validation Third, Nate was endlessly validating. For example, when Denise told Nate about a challenging situation with a co-worker who didn't listen to anyone else's ideas, Nate's response was: "Ugh that sounds so frustrating. Especially when you're trying to collaborate. What makes you say they don't take anything in?" Nate continued with clarifying questions and then validated Denise's experience. He was totally on her side. When sharing distressing events, people generally want to be validated -- to know that someone finds it understandable to feel the way we do. Validation is one of the most desired forms of support in close relationships and it can minimize the emotional and physiological effects of stressors. In the hopes of being helpful and lifting our mood, human partners may instead try to reframe our experience, give advice or solve the problem. In healthy relationships, we are open to being challenged, not just validated. And we need that perspective, because our partners may, at times, see us more clearly than we're able to see ourselves. AI companions can be so validating that they make people less likely to take responsibility and repair interpersonal conflicts. One-sided relationships AI companions can feel emotionally real, and they can meet some important psychological needs surprisingly well, especially in the short term. But they lack vital ingredients that make relationships engaging and meaningful over the long term. Essentially, the one-sidedness of human-AI relationships is a problem. There's a lot we can get out of an AI companion; there is less we can give. A truly fulfilling relationship -- and, some might argue, a full human life -- requires learning to balance our needs with the needs of the other, to give care as well as receive it, to both share and respond to vulnerable disclosures and to hear honest perspectives that we disagree with. This is central to our humanity.

Gen Z is living in an intimacy economy, where connection is commodified | Alice Lassman

My generation is struggling to find safety in our human relationships. AI companionship is filling the gap "There are parts of myself I only share with my AI," Jerome told me. "Your AI?" I asked. He was 27, three months out of a breakup, and entirely aware that his AI relationship was, as the psychotherapist Esther Perel put it, "a business product". Still, he confided in it more than he ever had his friends. I mention Jermoe because, in the four years I've been reporting on gen Z, I have watched my peers become him, one by one. We are the first generation of digital natives and the last generation to have experienced human-to-human intimacy exclusively. Now the attention economy's younger sister, the intimacy economy, has monetized the connection and affection that make us feel human. To a generation brought up with the ability to block and unfollow any source of digital agitation, the allure of an AI chatbot's one-stop-shop is undeniable - just a tab away, practically itching to help remove anger from emails to our managers, flirt better on dating apps or draft a sensitive message. The attention economy was just the prelude. Sexbots, digital twins that date on your behalf, AI therapists and nurses. Customizable emotional support. GPTs trained on our preferred therapy style and years of our surrendered therapy notes. Our generational dialect has already become: what does your AI say about that? What Jerome, whose name has been changed to keep him anonymous, really shows us is how much gen Z is struggling to find emotional safety in our human relationships. The emotional contract between men and women around me is breaking down. More than a third of 18-to-24-year-olds believe no one will ever fall in love with them. Is it any wonder that 80% of us say we'd consider an AI romance? Jerome wasn't just a consumer within this economy. He was also supplying his most intimate thoughts as raw material to a highly intelligent machine that spits out ever-more-accurate depictions of human complexity. The more we outsource, feeding our AI rather than each other, the less we're able to build the foundations to get what we need from the human relationships around us. As Silicon Valley bros clamber to amass "the last chance for generational wealth before AI makes money worthless", they have set their sights on one of the last major domains not yet fully commodified. The AI companion market is projected to reach $31.1bn by 2032. Companies like Replika make the dangerous design features of social media look like a trial run. Seeking to build "a machine beautiful enough for a soul", its founder markets "AI that cares", with "every detail ... yours to shape" - like a real-life Sim. Several companies' chatbots have been found to use guilt and Fomo 37% of the time when users imply they are about to end a conversation. And the market's co-opting of our most uncomfortable human feelings doesn't stop there. Our nervous systems, built over millennia to bond with other humans, have been exponentially overtaken by Moore's Law. When we can no longer register the difference between human and tech, no loss needs to be permanent: "grief tech" companies such as HereAfter AI offer grieving loved ones an interactive avatar to replay their dead relative's memories, to numb the loss keeping you up at 2am. These tools erase the experience of loss, and with it, our sense of human fragility that tells us to hold on to every moment with our loved ones. In this world, unraveling at a giddy pace, an emotional inequality is already taking shape: those who can initiate and sustain human connection, and those whose entire emotional lives are mediated by technology. The intimacy economy preys on those without the time and access for real relationships - a cheap substitute that feels just good enough to keep them subscribing. Naturally, big tech moguls severely limit their own children's screen time. Leading AI researchers now expect artificial general intelligence (AGI) - where AI meets or exceeds human intellectual capabilities - within three to five years. When AGI automates thinking, the economy will need to reorganise around what machines cannot do: feel. We are already in the intimacy economy. But if we are to thrive within it, we must wrestle back our most valuable economic asset from technology: the messy complexities of being human. My generation are the begrudging harbingers of what will happen if we continue to outsource intimacy: we will lose the emotional skills a post-AGI economy will depend on most, rendering us unemployable. We knew social media was addictive and slowing our cognitive development, but officials took until this year to act. We can't make the same mistake with the intimacy economy. My friend recently asked for my advice on whether to text after a first date to say he's not interested. I gave him an answer baked with the thoughtfulness only lived experience can create. Turns out, it wasn't needed: "Don't worry," he said, "I've already asked Claude." To reclaim what we're losing, we must resist taking any sign of discomfort as a reason to opt out. We must stop outsourcing intimacy to AI, reinvesting instead in each other to rebuild the emotional literacy we lost to digital dependencies. We are asking how we guardrail AI. The more urgent question is how we guardrail ourselves.