OpenAI ships a CSAM detector built for the age of AI-generated abuse images

Reviewed byNidhi Govil

9 Sources

Share

OpenAI released a free classifier that spots AI-generated abuse images. Its own model card shows precision falling from 99% in the lab to around 30% on live traffic.

OpenAI announced a package of child-safety measures on November 16, built around an image classifier that detects child sexual abuse material, or CSAM, child sexual abuse material,, child sexual abuse material,, including images made or altered with AI. The release also brings warning screens and 24-hour suspensions to ChatGPT, new guidance for developers, and investigative tools for police

1

5

2

.

Source: Digit

Source: Digit

Why now

Reports of suspected AI-generated abuse material to the National Center for Missing and Exploited Children's CyberTipline rose from 67,000 in 2024 to about 400,000 in 2025. Researchers have warned the workload could become unmanageable

2

3

. Hash-matching, the existing standard, works by matching previously identified material, so new or AI-generated images slip past it

8

7

.

OpenAI says companies have a responsibility to stop their technology being used to harm children. That means working with industry, child-safety groups and law enforcement

3

.

What the classifier does

The classifier scores images for signs of abuse so human reviewers can prioritize their workload. It produces two separate scores: one for suspected abuse material and one for suspected AI-generated material. OpenAI says it is the first broadly available classifier to carry a specific signal for AI-generated material

5

2

7

. NCMEC helped develop it, and it is free through the OpenAI API to vetted organizations, shared via the Tech Coalition and ROOST, an open-source safety coalition

1

4

.

In early testing it flagged an estimated 48% fewer images while catching roughly the same number of abusive images

1

2

. Flagged images were roughly twice as likely to include abuse

1

2

. Early partners reported their own results, self-submitted: fal's reviewers verified 72% of its high-confidence flags as new material

4

6

.

The gap in the model card

OpenAI's own model card carries a caveat. Per that card, offline precision is 99.1% for the likely-CSAM score and 100% for the AI-generated score. Against real ChatGPT traffic, those figures fell to 29.8% and 34%

7

. OpenAI attributes the drop to how rare this material is in ordinary traffic. Even so, a system flagging many images that turn out to be clean adds reviewer work rather than saving it

7

. OpenAI says the score alone does not confirm abuse or establish anything about legal status, so human review stays in the loop

5

7

.

Camille François, president of ROOST, said "OpenAI is setting an important precedent for transparency in child safety." She considers it the first model card in the child-safety detection field

3

.

What ChatGPT users will see

Users who try to find or create content that sexualizes children will get a warning, lose the flagged conversation, and have their account paused for 24 hours. Repeat violations can bring longer restrictions, closure, or reports to authorities

1

2

. ChatGPT already blocked such requests and refused to generate this material; the new warnings add an intervention layer for the users who try anyway

1

6

. OpenAI says the point is to interrupt the behavior and steer people toward specialist help. John Buckley, who leads child-safety public policy at OpenAI, said the 24-hour pause follows expert advice and international evidence on what disrupts people searching for this material

5

2

.

Source: TechCrunch

Source: TechCrunch

Developer guidance on detecting and reporting abuse material was written with the Tech Coalition, Thorn, NCMEC and the Internet Watch Foundation. It was commissioned jointly with Anthropic and Google

1

4

. Police get AI tools that review chat logs and build timelines, plus training and a field guide. Buckley said police teams faced capacity problems during development. OpenAI wants its tooling to help them triage and reach high-risk cases faster

5

.

The announcement lands amid pressure: eight states sued OpenAI last month over alleged harms to children, and Common Sense Media has called its technology an unacceptable risk for teens

1

2

.

North Carolina Attorney General Jeff Jackson, cochair of the AG Alliance's AI Task Force, spoke at OpenAI's event and described the influx of AI-generated abuse reports as explosive

2

. "Most of what AGs do is accountability work... But with this, we wanted to see if it was possible to develop a model of working together to prevent that harm. And I would say it's still basically an experiment."

Source: Analytics Insight

Source: Analytics Insight

© 2026 TheOutpost.AI All rights reserved