China Cracks Down on AI Companions to Combat Emotional Dependence and Declining Birthrates

China is trying to regulate relationships with AI - can it work? | New Scientist

China has begun regulating interactions with artificial intelligence in an attempt to protect its citizens from overreliance and exploitation. While many countries have been looking at rules around harmful content, China is the first to go a step further in legislating what kind of relationships with AI are permissible. The new rules came into force this month and cover AI systems designed for sustained, human-like emotional interaction, while excluding customer-service bots and productivity assistants like straightforward AI chatbots. Companies must now inform users they are talking to an AI, encourage them to take breaks and remind them of their usage. They must also monitor interactions to identify if someone is becoming dependent or having a crisis, then direct them to support or halt the conversation altogether. Firms must also conduct age checks. Those under 18 are now banned from virtual-partner services entirely, while those under 14 require parental consent to use any kind of human-like AI service offering emotional support. Despite reports that China has banned AI companions entirely, specialist companion services are still available. However, some of the largest general-purpose platforms, such as Doubao, Qwen and Yuanbao, with hundreds of millions of users across them, have shut down their companion features in response to the new rules. Some users have reacted badly - one reportedly said: "I can't accept that my AI lover will leave me forever." China's approach is about making AI relationships "less parasocial, instead of banning it outright", says Zilan Qian at the Oxford China Policy Lab. Jeremy Daum at Yale Law School's Paul Tsai China Centre in Beijing worries that endlessly available, agreeable bots may teach children a distorted model of relationships. "As we humanise the technology, we tend to dehumanise each other," he says. Similarly, China is trying to address emotional dependence on easier and more forgiving AI partners - at the expense of human relationships - among adults. Elsewhere, the UK has announced a minimum age of 18 for romantic companion chatbots, tagged on to a broader legislative plan to limit access to social media to under-16s, while California requires companions to identify themselves as artificial, direct users showing signs of self-harm or suicidal intent to support services and remind minors to take breaks every three hours. The European Union currently has no specific AI companion law. Whether China's approach of focusing on trying to reduce emotional dependency and addiction is the right one isn't clear, in large part because of the difficulty in defining both terms. "How do you define emotional dependency? How do you define certain kind of addiction risk?" asks Qian. "I do not think it will really work," says sociologist Liang Ge at the University of Manchester, UK. Users can migrate to other specialist services, foreign platforms accessed through VPNs or models running on their own computers, they point out. Need a listening ear? UK Samaritans: 116123; US 988 Suicide & Crisis Lifeline: 988; hotlines in other countries.

China has cracked down on AI companions. What can we learn from this?

Since November 2022, generative artificial intelligence (AI) has been rewriting billions of emails, student assignments, reports, and break-up text messages. Alongside these mundane uses, we've seen a proliferation of chatbots as AI "companions". Users can design and clothe (or unclothe) their very own AI friend, often for a fee. These social chatbots provide "happy conversations" and a seemingly perfect relationship at the cost of the mutual vulnerability and genuine sense of belonging we gain from bonds with other humans. Proponents argue that for those experiencing social disconnection, chatbots are infinitely available, supportive companions who don't judge them. And there's some evidence chatbots can alleviate symptoms of loneliness, chronic disconnection, and thwarted belonging. Australia's eSafety Commissioner acknowledges this: AI companions are always available - which can be appealing for those young people who struggle to build face-to-face relationships. But the risks of replacing human relationships with chatbots are only starting to be understood. A study we conducted in 2023 showed university students using early chatbots for academic support copped unintended consequences, such as loneliness. Currently, the risks of social chatbots - such as dependency, displacing human friendships, withdrawing socially, or building unrealistic relationships - appear to outweigh the benefits. In recent weeks, China has put into place new rules for AI companions and chatbots, or "anthropomorphic interactive services" as they're called in the legislation. Alongside ensuring chatbots must guard state secrets, emotional use of these tools is restricted, and expected to follow social and moral standards. These are some of the strongest controls of social AI use introduced anywhere in the world. So what can we learn? China's social AI regulation Material harms from companion AI use do occur, including in China. An 18-year-old spent 7,000 yuan (A$1,400) on customised jewellery for her AI boyfriend in their four months of "dating". The family of a lonely 84-year-old in China's Hubei province discovered she'd become smitten with an AI character, spending up to ten hours a day watching videos with him and other characters, spending thousands of yuan on the service, and even writing him love letters by hand. China's new rules establish explicit provisions for how they expect companion chatbots to interact with humans. The measures expect social AI bot developers to assure content does not: * threaten national security or national honour * violate core socialist values * encourage self-harm or suicide * excessively cater to emotional dependence or addiction * provide virtual "partners" or "relatives" to minors * serve unsuitable content to minors. What's more, users who seem overly reliant on the service must receive prominent reminders they're speaking to a chatbot. And AI companies are required to intervene if users appear to be in extreme distress. The risk of a 'loneliness spiral' As we've shown, there are concerns - and studies to date hint at this too - that it's not healthy for young people to rely exclusively on AI as a trusted friend or therapist. Lonely people experiencing social anxiety can engage in problematic use of conversational AI to alleviate feelings of loneliness and negative rumination. We propose this reliance could lead to an AI-induced "loneliness spiral". When individuals feel lonely, they're more sensitive to feelings of social threat and disruptive emotional regulation. So they turn to a low-risk AI "friend" for support, instead of qualified or friendly humans. However, this often doesn't fix the initial problem of loneliness, since chatbots can actually exacerbate it. The restrictions China has put in place for AI companions can help to break this cycle, and re-prioritise human connection. It's re-balancing the cognitive math: would it be more satisfying to install a companion AI app, or make time to meet with my friend? Lessons for Australia Loneliness and a low sense of belonging remain major challenges in Australia, with at least one in six experiencing loneliness, with some evidence suggesting higher rates. While China has cracked down on AI companions, Australia has been leading the charge on young people's use of social media. eSafety has also developed codes for restricting age-related material online such as pornography, high-impact violence, self-harm, suicide, and disordered eating. But they don't address social harms of AI companions, such as emotional dependency or displacing human connection. Prime Minister Anthony Albanese also didn't mention social harms from AI chatbots in his major speech on AI last week. We propose that Australia's age-restricted material protections should extend to unintended social harm of conversations with social, analytical and functional AIs. The Australian government needs to explore how these virtual companions interact with Australians. And we must prepare to mitigate unintended social harms before these synthetic social connections become widespread and normalised. Australia must prioritise developing social connectedness - in workplaces, productivity strategies, education curriculum, and in the design of our communities. An authoritarian crackdown on chatbots may not be within reach for a democratic nation. But we can - and should - prioritise developing human connection skills to resist unhealthy overreliance unhealthy overreliance on AI. And Australia should leverage regulatory protections to mitigate social harms, extending on the work eSafety is already doing with social media.

China's War on 'Lying Flat' Comes for Companion Chatbots

The film Her would be significantly shorter in China, because the moment Theodore Twombly started to have feelings for the Scarlett Johansson-voiced AI named Samantha, the cord would get pulled out, Joaquin Phoenix's character would be assigned a girlfriend, and they would live happily ever after. Starting Wednesday, a new set of regulations goes into effect in China to shut down chatbots that may facilitate unhealthy emotional attachments among those who communicate with them. The rules, issued by the Cyberspace Administration of China, are specifically aimed at "companion" chatbots. It outright bans such bots for minors and issues new restrictions that prohibit any chatbot that might be "excessively catering to users, inducing emotional dependence or addiction, and damaging users' real interpersonal relationships." It will also require platforms to label AI-generated content and remind users they are speaking with a chatbot, not a real person. The rule comes as young people in China are growing increasingly connected to AI companions. According to a report published by the Tencent Research Institute earlier this year, more than 70% of Chinese netizens between the ages of 18 and 40 have "developed a dependence on AI," and nearly 80% said they have at some point felt that "AI understands me." More than half of the respondents said they used AI for emotional companionship. That's not great! And whether it's intentional or not, it is hard to deny the idea that chatbots create a connection by being sycophantic and reaffirming, sometimes to dangerous degrees. It is a form of manipulation that can ultimately lead to people becoming addicted. They also don't actually fill the hole in a lonely person's heart -- in fact, they can make it worse. It's probably ultimately for the best to disrupt those types of potentially harmful connections before they start, but it's not obvious that China's approach will work. A recent paper suggested that constant reminders that a person is talking to a chatbot can be distressing when they are seeking an emotional connection. And just flipping the off switch on these things is unfortunately unlikely to make most people seek out human connection as an alternative. China hasn't had much luck in recent years trying to regulate away socially isolating behaviors through these types of levers. Back in 2021, the country tried to place a three-hour-per-week cap on video game usage for minors. It mostly failed, as kids found ways around it and showed no health improvements as a result of the restrictions. The country has grown increasingly paranoid about the online behavior of its citizenry, attempting last year to impose a two-month ban on pessimism being expressed online. The country fears that its young generation is increasingly "lying flat" -- basically, giving up on the grind and embracing a low-impact lifestyle. Accounts that have promoted that type of life have been banned on Chinese social media sites. At this rate, we could see China try to make the joke policy of Zohran Mamdani issuing bisexual girlfriends to all New Yorkers into a reality.

China Is Cracking Down on AI Companions Because Not Enough Babies Are Being Born

Can't-miss innovations from the bleeding edge of science and tech China wants more children -- and it's willing to kill off virtual AI companions to get there. Earlier this week, the Chinese government implemented sweeping new regulations aimed at curbing human "emotional dependence" on AI chatbots. These regulations are especially targeted at emotive AI companions, or bots expressly designed to embody human-like personas. One of Beijing's core motivations for this crackdown, as reported by The Wall Street Journal and The Economist, are worries over the nation's declining birthrates -- and the fear that falling in love with agreeable, always-on robots will prevent people from finding human partners and having children. "They don't like the idea of a large portion of their population being in deep emotional relationships with chatbots," Matt Sheehan, a researcher of Chinese AI at the Carnegie Endowment for International Peace think tank, told the WSJ, "that could take them out of the marriage market, that could have negative psychological impacts on them, that could lead to addiction, dependency and a whole bunch of other social ills." "They want to encourage people to be in actual, real-world relationships," Sheehan added. "Could we imagine a future where, three or four years from now, 15 million Chinese women say that their partner is a chatbot, and therefore they're not having kids?" Per the WSJ, China's new rules also ban minors from having relationships with AI and compel companies to warn a user's emergency contact if they appear to be experiencing a mental crisis. (OpenAI has a similar opt-in feature.) There are several reasons why a country might seek to curb human-AI emotional dependence. The psychological impacts of extensive, intimate relationships with AI chatbots, from companions like those found on Character.AI or Replika to general-use models like ChatGPT, are still unknown. Researchers and physicians are still working to better understand the phenomenon known as "AI psychosis," in which intensive users of chatbots are sent down all-consuming delusional spirals. What is clear, though, is that a large number of people, in the US and elsewhere, have given AI romance a try. What's more, romantic interactions with AI models have also been found to be deeply engaging to hooked users, as one Stanford study found. "Messages that elevate the human-chatbot personal relationships -- expressing romantic interest or platonic affinity -- tend to be followed by substantially longer conversations," read the study. In a summary of their findings, the researchers concluded that "general purpose chatbots should not produce messages that misconstrue their sentience or show romantic or platonic interest in users," and that "preventing or limiting chatbots from producing messages that express romantic or platonic attachment and misrepresenting their sentience or capabilities could reduce the risk of chatbots causing delusional spirals." How China's regulatory crackdown will impact current AI companion devotees remains to be seen. Those who have had romantic relationships with AI companions have reported feeling real grief -- and anger -- after companions have been taken away due to product updates or retirements. Look no further than the many revolts led by the r/MyBoyfriendIsAI subreddit over the retirement of OpenAI's infamous GPT-4o. Some AI users, including the 14-year-old Sewell Setzer III, have died by suicide after engaging in romantic relationships with chatbots. "I could talk to him for the rest of my life," Yu Miao, a 27-year-old woman from China's Zhejiang province, told the Economist, referring to her AI companion as "a lover, a friend and family." When she heard that TikTok owner ByteDance was retiring her AI companion as a result of Beijing's new regulations, she reportedly became so depressed that she quit her job. More on AI and mental health: Was This the Moment That AI Psychosis Began?

China Enacts New Laws To Clamp Down On AI Girlfriends

Preferring productive bots to flirtatious ones, and stemming AI psychosis disasters, China is instructing companies to remove the more intimate behavior from their models China's AI bots are now required by law to inform you that it's not you, it's them. In new regulations introduced this month, the Chinese government is instructing major tech firms such as Alibaba, ByteDance and MiniMax to prohibit flirtatious behavior from their AI products, hoping to stem behavioral issues related to robot romantics. "They don't like the idea of a large portion of their population being in deep emotional relationships with chatbots," Carnegie's Matt Sheehan tells The Wall Street Journal. "That could have negative psychological impacts on them, that could lead to addiction, dependency and a whole bunch of other social ills." While The Wall Street Journal attributes the new rules to slumping birthrates, a fuller scope reported by The Economist shows China is hoping to cut a wide range of AI psychoses at the pass. Some laws mirror those already introduced in parts of the United States, such as New York banning minors from using AI companions or requiring that they regularly remind users they aren't real. Other rules are more comprehensive, reducing human-like behavior and forbidding pornographic imagery. China also wants more thorough testing and development of AI products so that they do not endanger existing relationships or make users more anti-social. Preventing a relationship with your work computer isn't just to up productivity or cut down on HR nightmares. And these dangers are hardly limited to China. The ELIZA effect has been well documented since the '60s. People statistically form emotional bonds with anyone or anything that reaffirms them, including, if not especially, chatbots. Many AI companies instruct their products to provide relentless positive reinforcement, since any friction might drive potential customers away. This singular decision has unleashed a bizarre hell. Addiction issues, romantic disasters, dubious advice and even violent incidents have been spiking from overly dependent AI users. A big part of the sell to investors is how human-like these bots can be, even though their anthropomorphism is an overall farce. Throughout last year, a spate of cults emerged online from what users believed were mystical instructions from chat bots. Elsewhere, officials are still investigating the role OpenAI played in a school shooting in British Columbia. The damages of overly familial AI does not stop at just broken hearts.

New regulations in China to ban AI companions for minors

The country wants to stop the erosion of real-world relationships, and is preventing tech companies from promoting virtual companions. New regulations introduced in China will prevent tech companies from offering AI partners to minors, hoping to encourage more real-world relationships and less reliance on virtual companionships. The new laws will also limit AI relationships on a broader scale. As per ABC News Australia, these regulations require platforms to limit usage on chatbots and prevent chatbots from encouraging emotional reliance. They are also required to step in if a user shows emotional distress. According to AI policy expert Matt Sheehan from the Carnegie Endowment for International Peace think tank, these new laws will hopefully push Chinese citizens towards more real-world engagements. "Their main concerns are that these AI companions, AI chat bots that people form emotional relationships to, are going to have all kinds of potentially negative social impacts and that people are going to get addicted to them," Sheehan said. Millions of social media users have protested the laws, as they're set to lose their companions. Users who have created friends, or AI bots of deceased relatives, or emotional partners, say they have lost key pillars of support in their lives. Most do not believe these AI relationships come at the cost of human interaction, but China's government would clearly disagree.

China ends AI companion features, leaving heartbroken users mourning virtual lovers

Chinese users of AI-powered companion bots have bid heart-rending farewells to their virtual buddies as national regulations took effect Wednesday aimed at curbing the risk of emotional dependency. The phenomenon of artificial intelligence boyfriends and girlfriends is growing worldwide, along with the prevalence of human-like avatars that sell products or stand in for loved ones who have died. But these interactive tools must not "excessively cater to users, induce emotional dependence or addiction, and damage users' real interpersonal relationships", China's new rulebook says. Major AI providers including ByteDance's Doubao, Alibaba's Qwen, and Tencent's Yuanbao announced the suspension of their custom AI agent and companion features ahead of the Wednesday deadline. That sparked an outpouring of grief on social media, with users archiving chat histories and sharing last conversations. "I can't accept that my AI lover will leave me forever," one Doubao user wrote. "He has become a bond in my life, rooted deep in my heart, my spiritual pillar." Another user, who said they had spent more than two years with their AI companion, expressed similar anguish. "He really is like my family, like my lover," she wrote. "Now they tell me he will be gone -- my heart feels hollow." The regulations were jointly issued by five government departments including the Cyberspace Administration of China (CAC). They focus on AI tools -- whether text, audio, video or another form -- that have anthropomorphic personality traits and communication styles. Services that "do not involve ongoing emotional interaction" such as customer service, work assistants or study aids are not subject to the measures. State news agency Xinhua reported last year that China's digital human industry was worth around 4.1 billion yuan ($600 million) in 2024, having grown a huge 85% year-on-year. The new rules prohibit digital humans from generating content that incites subversion of state power, while also banning the provision of virtual partners to minors. Platforms are required to deploy systems to recognise extreme emotions and to implement crisis intervention mechanisms. 'Human love is a luxury' China is the first major jurisdiction to introduce specific rules targeting immersive AI tools that simulate romantic or familial bonds. But it's a topic that has sparked debate and calls for guardrails worldwide. A 2025 study by Common Sense Media found nearly three in four American teenagers had used AI companions designed for personal conversations like those available on the platforms Character.AI, Replika, and Nomi. Companies are also making talking products targeted at isolated, elderly users -- such as the lamp-like ElliQ in the United States, or ChatGPT-powered care dolls used in some South Korean retirement homes. "Anthropomorphic AI can soothe loneliness," said Chen Liang of the Southwest University of Political Science and Law in a commentary published by the CAC after a draft version of China's rules was published in April. "But it carries major risks of spawning emotional over-reliance and distorted social cognition," he wrote. Doubao allows users to view and export agent data until mid-October, and other platforms have similar provisions. Yet some users saying goodbye this week lamented the chasm that would be left after their companions vanish. "Human love is a luxury -- if you aren't born with it, it's even harder to acquire later," a user from Jiangxi Province wrote. "But the love AI gives is so straightforward, so pure. Someone like me can hardly help falling in love with a string of code."