China's AI Boom Fuels Wave of Deepfake Disaster Content as Authorities Make Arrests

Warnings that China's AI Boom Could Swamp the Country with Misinformation as Disaster Lies Prompt Arrests

As China deals with the fallout of Typhoon Noul that made landfall over the weekend, officials will be painfully aware that they will likely have an additional challenge to deal with: dangerously misleading videos created with Artificial Intelligence (AI). Powerful storms and flooding incidents over the last few months have seen fake videos inundating social media platforms - these include images of dead bodies floating in floodwaters, the false depiction of severe flooding in areas actually not affected and others with blatantly fake details of emergency response efforts. In some areas, videos containing false information about power outages prompted residents to panic buy emergency supplies. Other videos capitalise on real-life incidents. Flooding in Hengzhou city, Guangxi in early July caused hundreds of snakes to escape from a breeding farm. Then fake images were also published showing crocodiles being released into a river. In an already tense and dangerous situation involving mass evacuations and complex rescue operations, authorities want to avoid panic. Now they fear that AI technology - which China is pouring billions of dollars into - is providing simple, accessible tools that can be used to trick the public. "Many of these content creators use disaster news to try to gain followers, knowing that more sensational or exaggerated content will garner more likes and more views. They are deliberately fabricating information to grab people's attention," Huang Zhihua, Deputy Chief in charge of online public security for Zhejiang Police Department told Chinese state media. In recent weeks, there have been arrests and punishment - ranging from detention awaiting criminal prosecution to fines - for the creation of bogus reports and video content. But experts are warning that an unprecedented crisis of misinformation could be looming. In an article published by the newspaper of the Communist Party's Central Party School, Professor Chen Bin said, "AI dramatically lowers the technical barriers to producing false information. Individuals no longer require sophisticated technical skills. By simply prompting AI tools, they can rapidly generate convincing text, images, audio and video... which is highly realistic but entirely fabricated content, manufactured on a large scale." Professor Gao Fuping, from the East China University of Political Science and Law, told the BBC that motivation for the publishing of such false content ranges from those who do it "to create something amusing without thinking about the consequences", to those who want to "generate traffic for personal gain" or those who wish to deliberately "cause panic or social unrest". "Some users think the online world is anonymous and unregulated or that they cannot be traced," he said. There have been calls for all AI-generated content to be clearly stamped but other users can then just re-edit the material removing the markings. Professor Xu Xiaoke from Beijing Normal University told the BBC that greater efforts were also needed to verify geographic locations with timestamps and the checking of image sources. "That said, I would also stress that detection technology still lags behind generation technology. We're in a constant cat-and-mouse game. So we should not assume that technology alone can identify AI-generated content with the click of a button," he warned. The Chinese government is embracing AI in its bid to become the leading tech superpower. In 2017, it announced that AI would be "the main driving force" of the country's progress. Vast sums of money is pouring into AI businesses seeking more capital, fuelling domestic competition. The state-controlled wire service Xinhua reports that the country has over 6,000 "AI companies". In this climate, Prof Xu believes that China needs to build "trusted information infrastructure". This could include official government early warning systems and disaster interfaces which news organisations can access. But there have been instances of official sources, including government-controlled media, deliberately downplaying disasters, which has led to some public mistrust in this area. Earlier in July, local journalists were asked to "soften" their language to describe a major breach in the wall of the Liulan Dam near Naning as an "opening" or "gap". Officials appeared to be trying to make the breach sound less alarming but the consequences were deadly. Water bursting through the broken dam led to at least 26 deaths downstream. And this is not an isolated incident. In 2021, local officials were arrested, accused of concealing or underreporting 139 deaths in Henan province. Incidents like these have resulted in a trust vacuum that has made it much easier for bad actors to spread fake content. But police are imploring people to trust accredited sources during emergencies. "When encountering disaster-related information from an unknown source, approach it with scepticism and rely on official announcements," Gao Kai from the Ningbo Public Security Bureau said. He also tried to reassure the public that the creators of false information would be caught, saying that "the internet is not a lawless zone. Public security authorities will continue to intensify efforts to crack down on illegal online activity". "We've created a specialised task force to monitor social media closely, especially short video platforms. They will investigate and handle these cases quickly, cracking down in a very strict manner". Punishment could range from a fine to up to seven years in prison if the fallout of the misinformation is serious enough. On 23 July, China's Cyberspace Administration also announced a nationwide crackdown on fake internet content related to floods and other disasters, ordering its local branches to focus on catching out fabricated data, malicious editing, staged scenes, misattributed historical events and those impersonating officials. In addition, it called on social media platforms to strengthen the review of the content they are publishing. Prof Gao believes "the answer is public education combined with law enforcement", saying that there was a "need to publicise real cases where people have been punished." He added that "the harm caused by AI-generated fake content is ultimately the result of how people choose to use the technology - not the technology itself", adding that existing laws were sufficient to deal with it. However, others are not so confident. Professor Chen pointed out that "AI technology continues to evolve much faster than legal frameworks. As new legal questions emerge, AI misinformation remains a considerable challenge." Shanghai and Washington based lawyer Peter Pang says lawyers around the world are seeing an explosion in the number of cases dealing with these issues. He believes that the companies that create AI apps and the platforms used to publish fake videos and other material must be held legally accountable along with the people who create misinformation. "The first line would be the platforms, because they're the ones that are the enablers. They're the ones that allow this to go out," he told the BBC. He used a hypothetical example of a person who runs over a pedestrian following panic induced by a fake post on social media. "The video creator might say 'you ran over somebody. That's your problem. I didn't do it.' But let's think it through. You created the foundation from which people reacted. If you created that panic, you need to have responsibility for consequences that occur." And Professor Xu warns that "AI-generated misinformation is likely to increase not only in volume but in the speed with which it spreads." "Looking ahead, political elections, public health emergencies, armed conflicts, natural disasters and issues affecting people's livelihoods are all likely to become high-risk areas for AI-generated misinformation. "Instead of waiting for false information to go viral before trying to contain it, governments and platforms should now be preparing in advance."

AI's latest problem is deepfake disaster content for farming engagement

Fake and recycled videos from disasters in Venezuela and China are making social media feeds harder to trust when people need accurate updates quickly Social feeds around the earthquakes in Venezuela filled with recycled footage and AI-generated destruction. Severe weather in China attracted its own wave of AI disaster videos, including fabricated flood scenes and false reports of power outages. It creates an ugly race. People want immediate evidence of what happened, and AI slop can deliver something dramatic before reliable information catches up. Authentic updates are left competing with fiction precisely when confusion can send people scrambling for answers. How Venezuela's earthquake became content The misinformation surrounding Venezuela's earthquakes didn't rely on AI alone. Genuine videos from other disasters were assigned new locations, while older Venezuelan footage returned as breaking news. Once the date and context disappear, a real recording can deceive as effectively as a synthetic one. Recommended Videos That gives engagement farmers an easy way into an unfolding story. They only need a dramatic clip, a timely caption, and enough reach to beat anyone checking where it came from. AI adds more convincing material, but speed does much of the dirty work. What AI added to China's floods The fabricated posts from China show what happens when producing convincing fakes becomes cheap. Some images placed bodies in floodwaters, while false outage reports sent residents looking for emergency supplies. Many posts were intended to attract followers and traffic, sometimes with money as the eventual goal. The footage doesn't need to withstand close inspection. It only needs to look plausible long enough for frightened or curious viewers to pass it along. Why the first clips deserve suspicion The first viral videos often appear when reliable context is scarce. That timing gives dramatic footage an advantage, even when nobody can establish who recorded it, when it was made, or whether it shows the current disaster. Assuming every clip is fake would also discard genuine eyewitness evidence. A better rule is to treat disaster footage as unverified until you can confirm its location, date, and original uploader. If those basics can't be traced, don't help a content farm turn someone else's emergency into feed filler.