EU moves to ban AI nudify apps after Grok made them mainstream, targeting platforms over users

EU Parliament Votes to Ban AI 'Nudifier' Systems After Grok Scandal

The European Parliament's Internal Market and Civil Liberties committees voted 101-9, with 8 abstentions, to ban AI 'nudifier' systems as part of amendments to the EU AI Act1

1

. The decision follows a scandal involving Elon Musk's Grok chatbot, which allowed users to generate sexually explicit content produced by AI chatbots without consent, including child sexual abuse material. The amendment specifically targets platforms that allow users to generate sexually explicit images without consent, marking a significant shift in how the European Union regulates AI-generated deepfakes2

2

.

The proposed ban would prohibit AI systems that "generate realistic images so as to depict sexually explicit activities or the intimate parts of an identifiable natural person" without their consent2

2

. However, the ban would not apply to AI systems with effective safety measures preventing users from creating such images1

1

. This caveat provides a pathway for companies to continue operating if they implement robust safeguards and demonstrate accountability in content moderation.

Grok Backlash Drives Regulatory Shift from Users to Platforms

The Grok backlash "epitomized" why regulatory intervention was necessary, according to lawmakers who warned about "AI-powered nudity applications, such as Grok on X, but also other tools that are freely available online"1

1

. In January, Elon Musk's Grok service was used to generate and publish vast amounts of non-consensual sexual images based on pictures of fully clothed people, prompting xAI to restrict the feature after widespread criticism2

2

.

Earlier this year, xAI declined to introduce safeguards to block harmful outputs, instead choosing to paywall the feature and vow to suspend users who generate child sexual abuse material or non-consensual intimate imagery1

1

. This approach of blaming users rather than implementing platform-level restrictions may soon be foiled by the EU AI Act amendment. Lawmakers argued that "individual perpetrators" are "often hard to find" and can already "be punished under national criminal law," making it more effective to "prevent widespread image-based sexual violence from the outset"1

1

.

Timeline and Enforcement: What Companies Should Watch

EU member states backed the ban on March 13, with European ambassadors agreeing to prohibit "practices regarding the generation of non-consensual sexual and intimate content or child sexual abuse material"5

5

. The European Parliament is scheduled to vote on the proposal on March 263

3

. If approved, the ban could become law as early as August, though negotiations between lawmakers and EU governments on the final text will likely take up to a year before implementation4

4

.

For xAI and other AI developers, the stakes are substantial. Violations of the EU AI Act could result in fines of up to 7 percent of total worldwide annual turnover1

1

. Regulators in the European Union and the UK are formally investigating X and xAI over the Grok incident to establish whether it breached laws on content moderation and online safety2

2

. Tech regulators and national watchdogs in Britain, Ireland, and Spain are also currently investigating Grok's sexualized AI deepfakes4

4

.

Broader Implications for AI Regulation and High-Risk Systems

The amendment adds to existing EU legislation targeting deepfake nudes and cyberviolence. A 2024 directive on violence against women made it an offense to use AI to produce sexual images without the subject's consent, and the Digital Services Act also punishes social networks that allow the spread of illegal content, including child abuse material2

2

. The new ban represents the first EU policy to specifically target AI platforms that produce and allow sharing of sexual material without consent1

1

.

Lawmakers also voted to delay certain rules on high-risk AI systems to December 2027 for stand-alone systems and August 2028 for high-risk AI systems embedded in products, pushed back from the original August 2026 deadline2

2

5

5

. This delay will allow specialist organizations to draft detailed guidance on compliance and give companies more clarity2

2

.

Legal Challenges Mount Against xAI in Multiple Jurisdictions

In the United States, xAI faces legal challenges seeking injunctions against Grok's nudify outputs. Ashley St. Clair, a mother of one of Elon Musk's children, became one of the first victims to file a lawsuit in January. Three young girls in Tennessee recently filed a proposed class action representing all children harmed by Grok's alleged child sexual abuse material outputs1

1

. While xAI has seemingly faced few consequences in the US, the Take It Down Act, which takes effect in May, could expose the company to billions in fines1

1

.

Michael McNamara, a civil liberties committee member, stated that the proposal to ban nudify apps "is something that our citizens expect"1

1

3

3

. The challenge for developers of advanced AI capable of generating audiovisual material will be proving they have set up restrictions, though it remains unclear how chatbots would verify whether the person portrayed in an image has given their consent2

2

.