Apple and Google ordered to stop profiting from AI nudify apps targeting women and children

Apple, Google must stop profiting off AI nudify apps, San Francisco AG says

This week, San Francisco's attorney general, David Chiu, sent cease-and-desist letters, demanding that Apple and Google remove 13 so-called nudification apps from their app stores, Wired reported. Nudification apps can make it trivially easy to transform ordinary photos of real people into explicit images. The harmful AI tools allow bad actors to remove clothing, change a person's features, place them in sexualized positions, and swap victims' faces onto other people's naked bodies. Chiu's letter warned that app stores were violating "California's laws that prohibit supporting services that create deepfake pornography," Wired reported. Talking to Wired, Chiu said his office was "absolutely horrified" by how ubiquitous the nudifying technology has become, victimizing mostly women and children at an alarming scale as more tools became available. "These images are used to bully, humiliate, and threaten women and girls," Chiu told Wired. "This industry has a horrific impact on one's reputation, mental health, loss of autonomy. There have been victims who've been suicidal." Wired reviewed the letters and confirmed that Chiu asked Google to remove five apps and Apple to remove eight. No apps were named in the report to "avoid pushing people toward them," Wired said. However, one app had more than a million downloads and advertised features to sexualize images of women or make "free and uncensored" videos, Wired reported. Chiu told Wired that allowing any such apps to remain in app stores is unacceptable. "Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu said. His office estimated that Apple and Google have likely "made millions of dollars in fees" by ignoring the harmful apps rather than taking stronger actions or developing better detection to avoid profiting off a public nuisance. In a statement to Ars, Google spokesperson Dan Jackson said the five apps that Chiu flagged were suspended from the Google Play store for violating policies against harmful content. "Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content," Jackson said. "When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like 'nudify' on our store." Apple did not respond to Ars' request to comment. The Grok elephant in the room Although Apple has removed some apps flagged by researchers, the iPhone maker otherwise has remained notably silent amid calls to police nudification apps more aggressively in its App Store. App stores' detection abilities matter, as Wired's report noted that researchers have recently warned that some harmful apps have gotten better at avoiding app store removals by only promoting face-swapping features while hiding nudifying capabilities. In a May preprint paper, researchers identified 420 apps touted as generic face-swapping tools and tested 155 to see if they could be used to sexualize images. In 70 percent of apps tested, nudification was possible. There's also the Grok elephant in the room to consider. This week, xAI filed a lawsuit confirming that it found instances of Grok-generated child sexual abuse materials (CSAM) and other nonconsensual intimate imagery (NCII) targeting adults. To prevent the Grok misuse, xAI sued the user allegedly responsible for prompting Grok to generate the illegal content, but the question remains whether xAI is liable for the outputs. Ever since the Grok scandal erupted, app stores have faced questions on whether xAI's chatbot is violating app store policies against harmful content. Back in April, Apple told Senators that it had privately threatened to remove Grok, NBC News reported, but the xAI app remains in the app store today. So long as it's possible to use Grok to generate illegal content, app stores will likely continue to face pressure to take action to prevent harmful outputs, just as Chiu said that app stores should be doing with all apps allowing nudification if they want to comply with California deepfake laws. Although Google's spokesperson said that Google continuously tests generative AI apps to make sure they have safeguards preventing NCII and CSAM creation, Jackson did not respond to Ars' request to comment on whether the way that Grok performs today is an exception to Google's policy. In his letters to app stores, Chiu did not request Grok removals, but it is clear he is hoping that his demands will help broadly tighten up enforcement in both app stores, starting with removing the apps that his office flagged. "My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future," he says. "It's our hope that these companies will do the right thing -- but if they don't, we will have to consider all of our legal options."

San Francisco Demands Apple and Google Delete AI 'Nudify' Apps From App Stores

Apple and Google have been ordered to take down apps that can "nudify" or "undress" people and told that they must stop profiting from the harmful technology, according to cease-and-desist letters sent to the companies seen by WIRED. On Thursday, San Francisco city attorney David Chiu sent legal notices to Apple and Google demanding that they remove from their app stores 13 face-swapping apps, which allow users to create AI-generated nonconsensual nude images. The letters say the Silicon Valley giants should stop "aiding and abetting" the sale of explicit deepfake images and "sever" business relationships with the app developers. "Generating non-consensual intimate images is illegal, harmful, and completely unacceptable," Chiu tells WIRED. The city attorney, whose office previously took legal action against 16 popular deepfake websites, says Apple and Google have likely "made millions of dollars in fees" from apps that offer nudification, and they should improve their moderation processes to stop them appearing in their stores in the first place. "These companies have responsibility to ensure that apps on their platforms do not facilitate sexual abuse," Chiu says. The city's legal letters say California's laws prohibit supporting services that create deepfake pornography. The apps use in-app payments, which the tech companies take a cut of, the letters says. "The fact that some of the world's largest and most established technology companies are facilitating this has to stop." Researchers have repeatedly found and reported apps in Apple's App Store and Google's Play Store that allow people to generate sexual images using AI -- including some apps being rated as suitable for use by children. While new laws and bans aim to tackle the scourge of explicit deepfakes online, technology and social media companies consistently direct millions of people toward the harmful tech. Both Apple and Google have developer policies that prohibit pornography, abuse, and harassment on their platforms. They have previously removed dozens of nudify and deepfake apps, after reports by researchers and journalists. Google spokesperson Dan Jackson tells WIRED that the company has deleted "hundreds" of apps with nudifying features for policy violations, including the five Android apps flagged by Chiu's office, among other steps to restrict access to them. "Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content," Jackson says in a statement. "When violations are reported to us, we investigate and take swift action, which in the case of these apps has included suspending hundreds of violating apps and restricting related search terms like 'nudify' on our store." Apple did not provide comment ahead of publication. Over the last five years, a highly lucrative slurry of deepfake "nudification" tech has emerged online -- most transparently with xAI's Grok being used to create millions of sexualized images in January. A host of apps, websites, and bots allow people (mostly men) to upload pictures of people (overwhelmingly women and girls) and digitally "remove" clothing or place them into graphic sexual scenarios. Often all it takes to create sexual deepfakes is a reference photo and a couple of clicks, with some results available in seconds. Images and videos have become more realistic as the underlying generative AI technology has improved, with services providing some results for free or charging small fees to create the harmful content. Previous reporting by WIRED and Indicator Media has uncovered incidents in at least 90 schools where deepfake sexual abuse images have been created of minors. "These images are used to bully, humiliate, and threaten women and girls," Chiu says. "This industry has a horrific impact on one's reputation, mental health, loss of autonomy. There have been victims who've been suicidal." The 13 apps investigated by the City Attorney's Office -- eight on the App Store and five on the Play Store -- broadly advertise themselves as "face-swapping" tools, with their ability to create sexual deepfakes available once people use them. The website of one app, which has more than 1 million downloads, displays more than a dozen different styles of AI images it will generate, including "bikini queen curvy," "calm busty," and "cinematic intimacy." Many of the styles show sexualized images of women alongside their descriptions. The homepage of another of the targeted apps claims to produce "free and uncensored" videos. WIRED is not naming the apps to avoid pushing people toward them. The problem won't come as a surprise to Apple and Google. Over the last year, multiple reports have identified apps on the companies' platforms that can allow people to create nonconsensual nude images or videos. In January and April this year, the Tech Transparency Project, an independent watchdog group, uncovered around 100 apps across both the App Store and Play Store, as well as some advertisements for nudifying technology on the platforms. (Google's Jackson says the company has removed most of the apps identified by TTP.) Apps identified by the research were estimated to have been collectively downloaded around 480 million times and may have made around $120 million in combined revenues. "We didn't think after the first report that we would see this as a problem again -- and it was just as bad, if not worse, after the second report," says Katie Paul, the director of TTP. "Apple and Google make a lot of promises in their marketing about how trusted and safe their app stores are. And that is just not what is playing out in reality." Meanwhile, in a preprint research paper published in May, researchers from Cornell University and Georgetown University identified 420 apps offering general face-swapping capabilities on Google's and Apple's app stores. They tested 155 to see if they could be used to create face swaps with nude images; in 70 percent of cases, it was possible, with the apps not including safety measures to prevent this. "None of these apps are advertised as nudification apps," the research says. "This suggests that face swap apps, and many other forms of AI image generation and editing apps, are effectively 'dual-use': apps that evade content moderation by platforms because they present as benign, but possess the capability to create harmful content." Chiu, the San Francisco city attorney, says his office will keep pursuing the problem after being "absolutely horrified" at the harm and scale of the technology. "My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future," he says. "It's our hope that these companies will do the right thing -- but if they don't, we will have to consider all of our legal options."

Apple and Google ordered by San Francisco attorney to take action against 'nudify' apps - Engadget

Both companies were sent cease-and-desist letters regarding 13 apps on their respective stores. Apple and Google have been sent cease-and-desist letters that demand they remove AI-powered "nudify" apps from their respective app stores. As reported by Wired, San Francisco city attorney David Chiu ordered the two tech giants to take down a total of 13 apps that can be used to create AI-generated deepfake nude images. The letters appeal to Apple and Google to stop "aiding and abetting" the spread of nonconsensual intimate images, and ask them to stop working with the app developers in question. This isn't the first time Chiu has taken on deepfake platforms, having previously filed a lawsuit against 16 websites that allow users to turn images of real women and girls into pornography using AI. He told Wired that Apple and Google have likely made millions from in-app payments from the offending apps, and that their stores need better moderation to ensure they don't get approved. Despite widespread public outcry about deepfakes, it emerged earlier this year, via a report from The Tech Transparency Project (TTP), that nudification apps were frequently making it past Apple and Google's moderators. And it went even further, alleging that some of them were even being actively promoted on the App Store and Google Play.The report also alleged that many of these apps were rated "E" for Everyone, enabling children to download them. Finding the apps, it claimed, is as simple as searching for terms like "nudify" or "undress." This is despite both companies having policies that ban sexual or pornographic material on their platforms. Google and Apple aren't the only companies wrestling with the proliferation of deepfakes on the web. Meta's Oversight Board recently called on the company to strengthen its protections for ordinary people that are being targeted, with the existing preventative measures seemingly favoring public figures. Elon Musk's xAI has also been the subject of several lawsuits relating to nonconsensual deepfakes. A Google spokesperson told Wired that the company had deleted "hundreds" of apps with nudification features, five of which had been alerted to them by Chiu. It also reiterated that its policies forbid sexual content and said it takes "swift action" if a reported app is found to be in violation of its rules. Wired added that Apple did not provide a comment ahead of publication.

San Francisco warns Apple and Google to stop profiting from AI nudify apps

Serving tech enthusiasts for over 25 years. TechSpot means tech analysis and advice you can trust. What just happened? San Francisco has demanded that Apple and Google remove 13 AI-powered apps capable of creating nonconsensual nude images. City Attorney David Chiu has accused the companies of profiting from technology overwhelmingly used to target women and girls. Chiu sent cease-and-desist letters to the two tech giants on Thursday, demanding that they remove eight apps from Apple's App Store and five from Google Play. The programs are advertised primarily as face-swapping tools, but investigators found that they could also place people's faces onto explicit images or generate fake nude content. The letters accuse Apple and Google of "aiding and abetting" the sale of illegal deepfake pornography by hosting the apps, processing their in-app payments, and taking a cut of the proceeds. Chiu said the companies have likely earned millions of dollars in fees from theit cut of the apps' sales. San Francisco has given Apple and Google 28 days to explain how they will comply. The city could otherwise pursue civil enforcement carrying penalties of at least $25,000 per violation. The demands also call on the companies to cut payment-processing ties with the developers and introduce recurring reviews designed to stop replacement apps from appearing. Courtesty of the Tech Transparency Project Google said it had removed all five Android apps identified by Chiu's office. The company added that it has suspended hundreds of apps containing nudification features and restricted related search terms such as "nudify." Apple said it removed three of the flagged apps and is terminating their developers' accounts. Four other developers have been told to address policy violations or face removal. Both companies already prohibit pornography and abusive sexual content, which raises questions about how the apps made it through their review processes. The problem isn't limited to the named 13 programs. Two Tech Transparency Project investigations this year uncovered around 100 nudify-capable apps across both stores. They were estimated to have generated about $120 million in combined revenue and accumulated roughly 480 million downloads. A separate Cornell and Georgetown study found 420 face-swap apps on the stores. Researchers tested 155 and discovered that 70% allowed users to place faces onto nude images without technical safeguards. None openly advertised itself as a nudification app, illustrating how seemingly ordinary photo tools can evade moderation. San Francisco previously sued 16 of the most popular AI undressing websites in 2024, which had attracted 200 million visits during the first half of that year. Meta later sued the developer of Crush AI after more than 8,000 ads for the nudify service appeared on Facebook and Instagram in two weeks. Last month, the Justice Department also seized two websites accused of publishing hundreds of thousands of deepfake nude images featuring famous women. Chiu warned that Apple and Google must become more proactive, adding that his office will consider further legal action if they fail to respond.

Apple and Google Hit With Demand to Pull AI 'Nudify' Apps

A San Francisco City Attorney has sent cease-and-desist letters to Apple and Google demanding the removal of 13 AI "face-swap" apps that can generate nonconsensual nude images, according to WIRED. The letters, sent on Thursday, target eight apps on the App Store and five on the Play Store that market themselves as face-swapping tools but are used to "undress" photos of real people. The attorney's office wants both companies to cut ties with the developers and stop taking a cut of in-app payments, arguing they are effectively "aiding and abetting" the sale of explicit deepfake images. The attorney called the practice "illegal, harmful, and completely unacceptable" and believes Apple and Google have collected millions in fees from the apps. Both companies have developer rules against pornography and have removed batches of nudify apps in the past after being flagged by researchers. A Google spokesperson told WIRED that the company has deleted "hundreds" of apps with nudifying features for policy violations, including the five named in the letters. In June, it emerged that Apple had already tightened its App Store guideline language on developer responsibility for pornographic content.

Apple is finally removing AI apps that can digitally undress anyone

Apple and Google told by court by court to remove undressing apps Apple has removed several AI-powered "nudify" apps from the App Store after coming under legal pressure from San Francisco's city attorney, but the episode raises a much bigger question than whether a handful of apps violated App Store rules. The real issue is why these apps made it onto one of the world's most tightly controlled app marketplaces in the first place. Recommended Videos The latest controversy began after San Francisco City Attorney David Chiu sent cease-and-desist letters to both Apple and Google, accusing the companies of facilitating the distribution of apps capable of generating non-consensual AI-generated nude images. According to the legal notices, the companies have also profited from these apps through in-app purchases, with California authorities arguing that the platforms should sever ties with developers offering such services. Apple has now responded, confirming that several of the identified apps have already been removed while additional developers risk permanent removal if they fail to comply with App Store policies. Apple says the apps violate its rules. Critics say they shouldn't have appeared at all. Following the publication of WIRED's investigation, Apple told 9to5Mac that apps designed to generate or distribute pornographic or "nudification" content have always violated its App Review Guidelines. According to the company, three of the apps highlighted by the San Francisco City Attorney's Office have already been removed from the App Store, while the developer accounts behind them are in the process of being terminated. Apple also said it has contacted four additional developers, warning them to address policy violations or face removal from the platform. "The App Store was designed to be a safe and trusted place for users," Apple said in its statement, adding that it proactively rejects nudification apps and removes others after receiving reports through its reporting tools. Apple also reiterated that developers remain responsible for the content generated inside their applications. Apps that allow user-generated content must include mechanisms to filter objectionable material or risk removal from the App Store altogether. Google has taken similar action. The company says it has already suspended hundreds of Android apps offering nudification features and restricted Play Store searches for terms like "nudify." According to Google, the five Android apps cited by the San Francisco City Attorney have also been removed for violating Play Store policies. Here's Apple's full statement as provided to 9to5Mac: "The App Store was designed to be a safe and trusted place for users, and we have always strictly prohibited apps designed to generate, distribute, or consume pornography. 'Nudification' apps are against our App Review Guidelines, and we have proactively rejected many of these apps and removed many others, including when users have flagged them via our reporting tools. We have removed three of the apps in question and are in the process of terminating their developer accounts from our program. We are in contact with four others that need to address policy violations or risk being removed as well." The bigger challenge is moderation in the AI era The legal action comes amid growing concern over how generative AI has dramatically lowered the barrier to creating explicit deepfakes. Unlike traditional pornography, many of today's AI-powered face-swapping tools require nothing more than a single photograph. Within seconds, users can generate fabricated nude images or explicit videos depicting real people without their knowledge or consent. Authorities say the consequences extend well beyond privacy violations. Victims have increasingly reported harassment, extortion, reputational damage and severe mental health impacts. Previous investigations have also uncovered incidents across dozens of schools where AI-generated sexual images of minors were created using commercially available tools. What makes enforcement especially difficult is that many of these applications don't openly advertise themselves as "nudify" tools. Instead, they present themselves as harmless AI face-swapping or photo-editing apps. The explicit image generation capabilities often remain hidden until after installation or behind in-app purchases, making automated moderation significantly harder. Research published earlier this year found that a large number of seemingly ordinary face-swapping apps could still generate explicit deepfakes despite not marketing themselves that way. In many cases, researchers found little or no safeguards preventing misuse. That distinction has become central to the debate now facing Apple and Google Both companies maintain that their policies clearly prohibit pornography and sexually exploitative content. Critics, however, argue that policies alone are no longer enough when AI applications can disguise harmful functionality behind otherwise legitimate features. The City Attorney's Office also alleges that Apple and Google financially benefit from these apps through commissions on in-app purchases, arguing that platform operators have a greater responsibility to identify and remove them before they reach users. For Apple, the latest removals may help address the immediate criticism, but they are unlikely to end the broader conversation. As generative AI becomes cheaper, faster and increasingly capable of producing realistic synthetic media, app review is no longer just about checking screenshots or developer descriptions. It now requires identifying what an AI model can be persuaded to generate after installation, a considerably more difficult task than traditional moderation. That challenge is unlikely to disappear anytime soon, and neither are the questions being asked of the companies running the world's largest app stores.

San Francisco Urges Apple, Google to Remove AI Nudify Apps

San Francisco has initiated legal action against Apple and Google, asking the companies to remove AI-powered 'nudify' apps from their app stores. The city says these apps enable users to create non-consensual intimate deepfake images, violating California law. In cease-and-desist notices sent to both companies, San Francisco City Attorney David Chiu accused and Google of continuing to host dozens of such apps and processing payments for them despite the risks they pose. The city has warned that further legal action could follow if the companies fail to comply.