Apple and Google ordered to stop profiting from AI nudify apps targeting women and children

San Francisco Takes Legal Action Against Tech Giants

San Francisco City Attorney David Chiu sent cease-and-desist letters to Apple and Google this week, demanding immediate removal of 13 AI nudify apps from their respective app stores1

1

2

2

. The letters warn that both companies are violating California's laws that prohibit supporting services creating deepfake pornography1

1

. Chiu's office specifically identified eight apps on Apple's App Store and five on Google Play that enable users to create nonconsensual nude images through AI-generated harmful content2

2

.

The legal notices accuse Apple and Google of "aiding and abetting" the sale of explicit deepfake images and demand they "sever" business relationships with the app developers2

2

. According to Chiu, the companies have likely "made millions of dollars in fees" from in-app payments these apps process, with the tech giants taking a cut of the proceeds1

1

4

4

. San Francisco has given both companies 28 days to explain their compliance plans or face civil penalties of at least $25,000 per violation4

4

.

Face-Swapping Apps Hide Nudification Features

The 13 apps under investigation broadly advertise themselves as face-swapping apps, concealing their ability to create sexual deepfakes until users actually engage with them2

2

. One flagged app with more than 1 million downloads displays over a dozen AI image styles including "bikini queen curvy," "calm busty," and "cinematic intimacy," with many showing sexualized images of women2

2

. Another app's homepage claims to produce "free and uncensored" videos1

1

.

Researchers have warned that harmful apps have become increasingly sophisticated at avoiding detection by app store moderation systems. A May preprint paper identified 420 apps marketed as generic face-swapping tools and tested 155 of them1

1

. In 70 percent of apps tested, nudification was possible1

1

. None openly advertised itself as a nudification app, illustrating how seemingly ordinary photo tools can evade moderation4

4

. Two Tech Transparency Project investigations this year uncovered around 100 nudify-capable apps across both stores, estimated to have generated about $120 million in combined revenue and accumulated roughly 480 million downloads4

4

.

Devastating Impact on Women and Girls

Talking to reporters, Chiu said his office was "absolutely horrified" by how ubiquitous the nudifying technology has become, victimizing mostly women and children at an alarming scale1

1

. "These images are used to bully, humiliate, and threaten women and girls," Chiu explained1

1

2

2

. "This industry has a horrific impact on one's reputation, mental health, loss of autonomy. There have been victims who've been suicidal"1

1

.

Previous reporting uncovered incidents in at least 90 schools where deepfake sexual abuse images have been created of minors2

2

. The reputational harm extends beyond immediate psychological trauma, affecting victims' personal relationships, educational opportunities, and professional prospects. Chiu emphasized that "generating non-consensual intimate images is illegal, harmful, and completely unacceptable," and that allowing any such apps to remain in app stores is unacceptable1

1

2

2

.

Tech Giants Respond With Mixed Results

Google spokesperson Dan Jackson confirmed that the five apps flagged by Chiu were suspended from the Google Play store for violating harmful content policies1

1

. "Google Play does not allow apps that contain sexual content, and we continually take proactive steps to detect and remove apps with harmful content," Jackson stated1

1

. The company claims to have deleted "hundreds" of apps with nudifying features and restricted related search terms like "nudify" on its store1

1

2

2

.

Apple removed three of the flagged apps and is terminating their developers' accounts, while four other developers have been told to address policy violations or face removal4

4

. However, Apple did not provide comment to multiple news outlets1

1

2

2

. In June, Apple tightened its App Store guideline language on developer responsibility for pornographic content5

5

. Both companies have developer policies prohibiting pornography, abuse, and harassment, raising questions about platform accountability and how these apps passed initial review processes2

2

3

3

.

Broader Questions About Generative AI Platforms

The cease-and-desist letters arrive amid growing scrutiny of generative AI platforms enabling non-consensual intimate imagery. The xAI Grok controversy looms large, with xAI filing a lawsuit this week confirming instances of Grok-generated child sexual abuse materials and other NCII targeting adults1

1

. Back in April, Apple reportedly threatened to remove Grok from its app store, yet the xAI app remains available today1

1

.

While Google's spokesperson said the company continuously tests generative AI apps to ensure safeguards preventing NCII creation, questions persist about whether current enforcement adequately addresses platforms like Grok1

1

. The Meta Oversight Board has also called on the company to strengthen protections for ordinary people being targeted, with existing measures seemingly favoring public figures3

3

. San Francisco previously sued 16 of the most popular AI undressing websites in 2024, which had attracted 200 million visits during the first half of that year4

4

.

Chiu's demands call on Apple and Google to introduce recurring reviews designed to stop replacement apps from appearing and to remove apps from app stores more proactively4

4

. "My hope is that Apple and Google will immediately remove these apps and strengthen their screening systems to make sure that apps like this never get onto their platforms in the future," Chiu stated1

1

. "These companies have responsibility to ensure that apps on their platforms do not facilitate sexual abuse," he added, warning that his office will consider further legal action if they fail to respond adequately2

2

4

4

.