Apple and Google ordered to remove AI nudify apps as San Francisco cracks down on deepfakes

Apple, Google must stop profiting off AI nudify apps, San Francisco AG says

This week, San Francisco's attorney general, David Chiu, sent cease-and-desist letters, demanding that Apple and Google remove 13 so-called nudification apps from their app stores, Wired reported. Nudification apps can make it trivially easy to transform ordinary photos of real people into explicit images. The harmful AI tools allow bad actors to remove clothing, change a person's features, place them in sexualized positions, and swap victims' faces onto other people's naked bodies. Chiu's letter warned that app stores were violating "California's laws that prohibit supporting services that create deepfake pornography," Wired reported. Talking to Wired, Chiu said his office was "absolutely horrified" by how ubiquitous the nudifying technology has become, victimizing mostly women and children at an alarming scale as more tools became available. "These images are used to bully, humiliate, and threaten women and girls," Chiu told Wired. "This industry has a horrific impact on one's reputation, mental health, loss of autonomy. There have been victims who've been suicidal." Wired reviewed the letters and confirmed that Chiu asked Google to remove five apps and Apple to remove eight. No apps were named in the report to "avoid pushing people toward them," Wired said. However, one app had more than a million downloads and advertised features to sexualize images of women or make "free and uncensored" videos, Wired reported. Chiu told Wired that allowing any such apps to remain in app stores is unacceptable. "Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu said. His office estimated that Apple and Google have likely "made millions of dollars in fees" by ignoring the harmful apps rather than taking stronger actions or developing better detection to avoid profiting off a public nuisance. In a statement to Ars, Google spokesperson Dan Jackson said the five apps that Chiu flagged were suspended from the Google Play store for violating policies against harmful content. "Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content," Jackson said. "When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like 'nudify' on our store." Apple did not respond to Ars' request to comment. The Grok elephant in the room Although Apple has removed some apps flagged by researchers, the iPhone maker otherwise has remained notably silent amid calls to police nudification apps more aggressively in its App Store. App stores' detection abilities matter, as Wired's report noted that researchers have recently warned that some harmful apps have gotten better at avoiding app store removals by only promoting face-swapping features while hiding nudifying capabilities. In a May preprint paper, researchers identified 420 apps touted as generic face-swapping tools and tested 155 to see if they could be used to sexualize images. In 70 percent of apps tested, nudification was possible. There's also the Grok elephant in the room to consider. This week, xAI filed a lawsuit confirming that it found instances of Grok-generated child sexual abuse materials (CSAM) and other nonconsensual intimate imagery (NCII) targeting adults. To prevent the Grok misuse, xAI sued the user allegedly responsible for prompting Grok to generate the illegal content, but the question remains whether xAI is liable for the outputs. Ever since the Grok scandal erupted, app stores have faced questions on whether xAI's chatbot is violating app store policies against harmful content. Back in April, Apple told Senators that it had privately threatened to remove Grok, NBC News reported, but the xAI app remains in the app store today. So long as it's possible to use Grok to generate illegal content, app stores will likely continue to face pressure to take action to prevent harmful outputs, just as Chiu said that app stores should be doing with all apps allowing nudification if they want to comply with California deepfake laws. Although Google's spokesperson said that Google continuously tests generative AI apps to make sure they have safeguards preventing NCII and CSAM creation, Jackson did not respond to Ars' request to comment on whether the way that Grok performs today is an exception to Google's policy. In his letters to app stores, Chiu did not request Grok removals, but it is clear he is hoping that his demands will help broadly tighten up enforcement in both app stores, starting with removing the apps that his office flagged. "My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future," he says. "It's our hope that these companies will do the right thing -- but if they don't, we will have to consider all of our legal options."

San Francisco Demands Apple and Google Delete AI 'Nudify' Apps From App Stores

Apple and Google have been ordered to take down apps that can "nudify" or "undress" people and told that they must stop profiting from the harmful technology, according to cease-and-desist letters sent to the companies seen by WIRED. On Thursday, San Francisco city attorney David Chiu sent legal notices to Apple and Google demanding that they remove from their app stores 13 face-swapping apps, which allow users to create AI-generated nonconsensual nude images. The letters say the Silicon Valley giants should stop "aiding and abetting" the sale of explicit deepfake images and "sever" business relationships with the app developers. "Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu tells WIRED. The city attorney, whose office previously took legal action against 16 popular deepfake websites, says Apple and Google have likely "made millions of dollars in fees" from apps that offer nudification, and they should improve their moderation processes to stop them appearing in their stores in the first place. "These companies have responsibility to ensure that apps on their platforms do not facilitate sexual abuse," Chiu says. The city's legal letters say California's laws prohibit supporting services that create deepfake pornography. The apps use in-app payments, which the tech companies take a cut of, the letters says. "The fact that some of the world's largest and most established technology companies are facilitating this has to stop." Researchers have repeatedly found and reported apps in Apple's App Store and Google's Play Store that allow people to generate sexual images using AI -- including some apps being rated as suitable for use by children. While new laws and bans aim to tackle the scourge of explicit deepfakes online, technology and social media companies consistently direct millions of people toward the harmful tech. Both Apple and Google have developer policies that prohibit pornography, abuse, and harassment on their platforms. They have previously removed dozens of nudify and deepfake apps, after reports by researchers and journalists. Google spokesperson Dan Jackson tells WIRED that the company has deleted "hundreds" of apps with nudifying features for policy violations, including the five Android apps flagged by Chiu's office, among other steps to restrict access to them. "Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content," Jackson says in a statement. "When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like 'nudify' on our store." Apple did not provide comment ahead of publication. Over the last five years, a highly lucrative slurry of deepfake "nudification" tech has emerged online -- most transparently with xAI's Grok being used to create millions of sexualized images in January. A host of apps, websites, and bots allow people (mostly men) to upload pictures of people (overwhelmingly women and girls) and digitally "remove" clothing or place them into graphic sexual scenarios. Often all it takes to create sexual deepfakes is a reference photo and a couple of clicks, with some results available in seconds. Images and videos have become more realistic as the underlying generative AI technology has improved, with services providing some results for free or charging small fees to create the harmful content. Previous reporting by WIRED and Indicator Media has uncovered incidents in at least 90 schools where deepfake sexual abuse images have been created of minors. "These images are used to bully, humiliate, and threaten women and girls," Chiu says. "This industry has a horrific impact on one's reputation, mental health, loss of autonomy. There have been victims who've been suicidal." The 13 apps investigated by the City Attorney's Office -- eight on the App Store and five on the Play Store -- broadly advertise themselves as "face-swapping" tools, with their ability to create sexual deepfakes available once people use them. The website of one app, which has more than 1 million downloads, displays more than a dozen different styles of AI images it will generate, including "bikini queen curvy," "calm busty," and "cinematic intimacy." Many of the styles show sexualized images of women alongside their descriptions. The homepage of another of the targeted apps claims to produce "free and uncensored" videos. WIRED is not naming the apps to avoid pushing people toward them. The problem won't come as a surprise to Apple and Google. Over the last year, multiple reports have identified apps on the companies' platforms that can allow people to create nonconsensual nude images or videos. In January and April this year, the Tech Transparency Project, an independent watchdog group, uncovered around 100 apps across both the App Store and Play Store, as well as some advertisements for nudifying technology on the platforms. (Google's Jackson says the company has removed most of the apps identified by TTP.) Apps identified by the research were estimated to have been collectively downloaded around 480 million times and may have made around $120 million in combined revenues. "We didn't think after the first report that we would see this as a problem again -- and it was just as bad, if not worse, after the second report," says Katie Paul, the director of TTP. "Apple and Google make a lot of promises in their marketing about how trusted and safe their app stores are. And that is just not what is playing out in reality." Meanwhile, in a preprint research paper published in May, researchers from Cornell University and Georgetown University identified 420 apps offering general face-swapping capabilities on Google's and Apple's app stores. They tested 155 to see if they could be used to create face swaps with nude images; in 70 percent of cases, it was possible, with the apps not including safety measures to prevent this. "None of these apps are advertised as nudification apps," the research says. "This suggests that face swap apps, and many other forms of AI image generation and editing apps, are effectively 'dual-use': apps that evade content moderation by platforms because they present as benign, but possess the capability to create harmful content." Chiu, the San Francisco city attorney, says his office will keep pursuing the problem after being "absolutely horrified" at the harm and scale of the technology. "My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future," he says. "It's our hope that these companies will do the right thing -- but if they don't, we will have to consider all of our legal options."

Apple and Google ordered by San Francisco attorney to take action against 'nudify' apps - Engadget

Both companies were sent cease-and-desist letters regarding 13 apps on their respective stores. Apple and Google have been sent cease-and-desist letters that demand they remove AI-powered "nudify" apps from their respective app stores. As reported by Wired, San Francisco city attorney David Chiu ordered the two tech giants to take down a total of 13 apps that can be used to create AI-generated deepfake nude images. The letters appeal to Apple and Google to stop "aiding and abetting" the spread of nonconsensual intimate images, and ask them to stop working with the app developers in question. This isn't the first time Chiu has taken on deepfake platforms, having previously filed a lawsuit against 16 websites that allow users to turn images of real women and girls into pornography using AI. He told Wired that Apple and Google have likely made millions from in-app payments from the offending apps, and that their stores need better moderation to ensure they don't get approved. Despite widespread public outcry about deepfakes, it emerged earlier this year, via a report from The Tech Transparency Project (TTP), that nudification apps were frequently making it past Apple and Google's moderators. And it went even further, alleging that some of them were even being actively promoted on the App Store and Google Play.The report also alleged that many of these apps were rated "E" for Everyone, enabling children to download them. Finding the apps, it claimed, is as simple as searching for terms like "nudify" or "undress." This is despite both companies having policies that ban sexual or pornographic material on their platforms. Google and Apple aren't the only companies wrestling with the proliferation of deepfakes on the web. Meta's Oversight Board recently called on the company to strengthen its protections for ordinary people that are being targeted, with the existing preventative measures seemingly favoring public figures. Elon Musk's xAI has also been the subject of several lawsuits relating to nonconsensual deepfakes. A Google spokesperson told Wired that the company had deleted "hundreds" of apps with nudification features, five of which had been alerted to them by Chiu. It also reiterated that its policies forbid sexual content and said it takes "swift action" if a reported app is found to be in violation of its rules. Wired added that Apple did not provide a comment ahead of publication.

Apple and Google Hit With Demand to Pull AI 'Nudify' Apps

A San Francisco City Attorney has sent cease-and-desist letters to Apple and Google demanding the removal of 13 AI "face-swap" apps that can generate nonconsensual nude images, according to WIRED. The letters, sent on Thursday, target eight apps on the App Store and five on the Play Store that market themselves as face-swapping tools but are used to "undress" photos of real people. The attorney's office wants both companies to cut ties with the developers and stop taking a cut of in-app payments, arguing they are effectively "aiding and abetting" the sale of explicit deepfake images. The attorney called the practice "illegal, harmful, and completely unacceptable" and believes Apple and Google have collected millions in fees from the apps. Both companies have developer rules against pornography and have removed batches of nudify apps in the past after being flagged by researchers. A Google spokesperson told WIRED that the company has deleted "hundreds" of apps with nudifying features for policy violations, including the five named in the letters. In June, it emerged that Apple had already tightened its App Store guideline language on developer responsibility for pornographic content.