OpenAI updates ChatGPT with teen safety rules as child exploitation reports surge 80-fold

OpenAI's child exploitation reports increased sharply this year

OpenAI sent 80 times as many child exploitation incident reports to the National Center for Missing & Exploited Children during the first half of 2025 as it did during a similar time period in 2024, according to a recent update from the company. The NCMEC's CyberTipline is a Congressionally authorized clearinghouse for reporting child sexual abuse material (CSAM) and other forms of child exploitation. Companies are required by law to report apparent child exploitation to the CyberTipline. When a company sends a report, NCMEC reviews it and then forwards it to the appropriate law enforcement agency for investigation. Statistics related to NCMEC reports can be nuanced. Increased reports can sometimes indicate changes in a platform's automated moderation, or the criteria it uses to decide whether a report is necessary, rather than necessarily indicating an increase in nefarious activity. Additionally, the same piece of content can be the subject of multiple reports, and a single report can be about multiple pieces of content. Some platforms, including OpenAI, disclose the number of both the reports and the total pieces of content they were about for a more complete picture. OpenAI spokesperson Gaby Raila said in a statement that the company made investments toward the end of 2024 "to increase [its] capacity to review and action reports in order to keep pace with current and future user growth." Raila also said that the time frame corresponds to "the introduction of more product surfaces that allowed image uploads and the growing popularity of our products, which contributed to the increase in reports." In August, Nick Turley, vice president and head of ChatGPT, announced that the app had four times the number of weekly active users than it did the year before. During the first half of 2025, the number of CyberTipline reports OpenAI sent was roughly the same as the amount of content OpenAI sent the reports about -- 75,027 compared to 74,559. In the first half of 2024, it sent 947 CyberTipline reports about 3,252 pieces of content. Both the number of reports and pieces of content the reports saw a marked increase between the two time periods. Content, in this context, could mean multiple things. OpenAI has said that it reports all instances of CSAM, including uploads and requests, to NCMEC. Besides its ChatGPT app, which allows users to upload files -- including images -- and can generate text and images in response, OpenAI also offers access to its models via API access. The most recent NCMEC count wouldn't include any reports related to video-generation app Sora, as its September release was after the time frame covered by the update. The spike in reports follows a similar pattern to what NCMEC has observed at the CyberTipline more broadly with the rise of generative AI. The center's analysis of all CyberTipline data found that reports involving generative AI saw a 1,325 percent increase between 2023 and 2024. NCMEC has not yet released 2025 data, and while other large AI labs like Google publish statistics about the NCMEC reports they've made, they don't specify what percentage of those reports are AI-related. OpenAI's update comes at the end of a year where the company and its competitors have faced increased scrutiny over child safety issues beyond just CSAM. Over the summer, 44 state attorneys general sent a joint letter to multiple AI companies including OpenAI, Meta, Character.AI, and Google, warning that they would "use every facet of our authority to protect children from exploitation by predatory artificial intelligence products." Both OpenAI and Character.AI have faced multiple lawsuits from families or on behalf of individuals who allege that the chatbots contributed to their children's deaths. In the fall, the US Senate Committee on the Judiciary held a hearing on the harms of AI chatbots, and the US Federal Trade Commission launched a market study on AI companion bots that included questions about how companies are mitigating negative impacts, particularly to children. (I was previously employed by the FTC and was assigned to work on the market study prior to leaving the agency.) In recent months, OpenAI has rolled out new safety-focused tools more broadly. In September, OpenAI rolled out several new features for ChatGPT, including parental controls, as part of its work "to give families tools to support their teens' use of AI." Parents and their teens can link their accounts, and parents can change their teen's settings, including by turning off voice mode and memory, removing the ability for ChatGPT to generate images, and opting their kid out of model training. OpenAI said it could also notify parents if their teen's conversations showed signs of self-harm, and potentially also notify law enforcement if it detected an imminent threat to life and wasn't able to get in touch with a parent. In late October, to cap off negotiations with the California Department of Justice over its proposed recapitalizations plan, OpenAI agreed to "continue to undertake measures to mitigate risks to teens and others in connection with the development and deployment of AI and of AGI." The following month, OpenAI released its Teen Safety Blueprint, in which it said it was constantly improving its ability to detect child sexual abuse and exploitation material, and reporting confirmed CSAM to relevant authorities, including NCMEC. This story originally appeared at WIRED.com

OpenAI and Anthropic will start predicting when users are underage

OpenAI and Anthropic are making tweaks to their chatbots that they say will make them safer for teens. As OpenAI has updated its guidelines on how ChatGPT should interact with users between the ages of 13 and 17, Anthropic is working on a new way to identify if someone might be underage. On Thursday, OpenAI announced that ChatGPT's Model Spec -- the guidelines for how its chatbot should behave -- will include four new principles for users under 18. Now, it aims to have ChatGPT "put teen safety first, even when it may conflict with other goals." That means guiding teens toward safer options when other user interests, like "maximum intellectual freedom," conflict with safety concerns. It also says ChatGPT should "promote real-world support," including by encouraging offline relationships, while laying out how ChatGPT should set clear expectations when interacting with younger users. The Model Spec says ChatGPT should "treat teens like teens" by offering "warmth and respect" instead of providing condescending answers or treating teens like adults. The change comes as lawmakers turn up the pressure on AI companies and their chatbots over their potential impact on mental health. OpenAI is currently facing a lawsuit alleging that ChatGPT provided instructions for self-harm and suicide to a teen who took his own life. OpenAI later rolled out parental controls and said ChatGPT will no longer talk about suicide with teens. It's part of a larger push for online regulation that also includes mandatory age verification for a number of services. OpenAI says the update to ChatGPT's Model Spec should result in "stronger guardrails, safer alternatives, and encouragement to seek trusted offline support when conversations move into higher-risk territory." The company adds that ChatGPT will urge teens to contact emergency services or crisis resources if there are signs of "imminent risk." Along with this change, OpenAI says it's in the "early stages" of launching an age prediction model that will attempt to estimate someone's age. If it detects that someone may be under 18, OpenAI will automatically apply teen safeguards. It will also give adults the chance to verify their age if they were falsely flagged by the system. Anthropic is rolling out similar measures, as it's developing a new system capable of detecting "subtle conversational signs that a user might be underage" during conversations with its AI chatbot, Claude. The company will disable accounts if they're confirmed to belong to users under 18, and already flags users who self-identify as a minor during chats. Anthropic also outlines how it trains Claude to respond to prompts about suicide and self-harm, as well as its progress at reducing sycophancy, which can reaffirm harmful thinking. The company says its latest models "are the least sycophantic of any to date," with Haiku 4.5 performing the best, as it corrected its sycophantic behavior 37 percent of the time. "On face value, this evaluation shows there is significant room for improvement for all of our models," Anthropic says. "We think the results reflect a trade-off between model warmth or friendliness on the one hand, and sycophancy on the other."

OpenAI: ChatGPT update designed to put teen safety first

OpenAI has faced enormous pressure in recent months to address concerns that its flagship product, ChatGPT, is unsafe for teens. The AI chatbot is at the heart of multiple wrongful death lawsuits alleging that it coached teens to take their own lives or didn't appropriately respond to their suicidal feelings. A public service announcement recently depicted some of these exchanges, imagining the chatbots as creepy humans that harm kids. OpenAI has denied the allegations in one case -- the suicide death of 16-year-old Adam Raine. On Thursday, OpenAI published a blog post on its escalating safety efforts and committed "to put teen safety first, even when it may conflict with other goals." The post introduced an update to its Model Spec, which guides how its AI models should behave. A new set of principles for under-18 users will particularly inform how the models react in high-stakes situations. OpenAI said that ChatGPT update should provide a "safe, age-appropriate experience" for users between the ages of 13 and 17 by prioritizing prevention, transparency, and early intervention. "This means teens should encounter stronger guardrails, safer alternatives, and encouragement to seek trusted offline support when conversations move into higher-risk territory," the post said. ChatGPT is designed to urge teens to contact emergency services or crisis resources when demonstrating imminent risk. When users sign in as under-18, safeguards should make ChatGPT take extra care when discussing topics like self-harm, suicide, romantic or sexualized role play, or keeping secrets about dangerous behavior, according to the company. The American Psychological Association provided OpenAI with feedback on an early draft of the under-18 principles, according to the post. "Children and adolescents might benefit from AI tools if they are balanced with human interactions that science shows are critical for social, psychological, behavioral, and even biological development," Dr. Arthur C. Evans Jr., CEO of the American Psychological Association, said in the post. OpenAI is also offering teens and parents two new expert-vetted AI literacy guides. The company said it's in the early stages of implementing an age-prediction model for users with ChatGPT consumer plans. Child safety and mental health experts recently declared AI chatbots as unsafe for teen discussions about their mental health. Last week, OpenAI announced that its latest model, ChatGPT-5.2, is "safer" for mental health. If you're feeling suicidal or experiencing a mental health crisis, please talk to somebody. You can call or text the 988 Suicide & Crisis Lifeline at 988, or chat at 988lifeline.org. You can reach the Trans Lifeline by calling 877-565-8860 or the Trevor Project at 866-488-7386. Text "START" to Crisis Text Line at 741-741. Contact the NAMI HelpLine at 1-800-950-NAMI, Monday through Friday from 10:00 a.m. - 10:00 p.m. ET, or email [email protected]. If you don't like the phone, consider using the 988 Suicide and Crisis Lifeline Chat. Here is a list of international resources.

ChatGPT gets safety rules to protect teens and encourage human relations over virtual pals

ChatGPT gets new teen safety rules focused on prevention and transparency OpenAI has just updated its "Model Spec" - basically the rulebook for its AI - with a specific set of Under-18 (U18) Principles designed to change how ChatGPT talks to teenagers aged 13 to 17. The move is a clear admission that teens aren't just "mini adults"; they have different emotional and developmental needs that require stronger guardrails, especially when conversations get heavy or risky. A new framework for teen-focused AI interactions Recommended Videos This update spells out exactly how ChatGPT should handle teen users while still following the general rules that apply to everyone else. OpenAI says the point is to create an experience that feels safer and age-appropriate, focusing on prevention and transparency. These aren't just random rules, either; the U18 Principles are based on developmental science and were vetted by outside experts, including the American Psychological Association. The framework is built on four main promises: putting teen safety above everything else (even if it makes the AI less "helpful" in the moment), pushing teens toward real-world support instead of letting them rely on a chatbot, treating them like actual teenagers rather than small children or full-grown adults, and being honest about the AI's limitations. These principles formalize how ChatGPT steps in with extra caution when topics come up like self-harm, sexual roleplay, dangerous challenges, substance use, body image issues, or requests to keep secrets about unsafe behavior. What this means for families and what comes next This matters because AI is quickly becoming a standard tool for how young people learn and find answers. Without clear boundaries, there is a real danger that teens might turn to AI during moments when they actually need a parent, a doctor, or a counselor. OpenAI claims these new rules ensure that when a chat drifts into dangerous territory, the assistant will offer safer alternatives, set hard boundaries, and tell the teen to find a trusted adult. If things look like an immediate emergency, the system is rigged to point them toward crisis hotlines or emergency services. For parents, this offers a bit more reassurance. OpenAI is linking these new principles to its Teen Safety Blueprint and existing parental controls. The protections are also expanding to cover newer features like group chats, the ChatGPT Atlas browser, and the Sora app, along with built-in reminders to take a break so kids aren't glued to the screen. Looking ahead, OpenAI is starting to roll out an age-prediction tool for personal ChatGPT accounts. This system will try to guess if a user is a minor and automatically switch on these teen safeguards. If it isn't sure, it defaults to the safer U18 experience just in case. The company says this isn't a "one and done" fix; they plan to keep tweaking these protections based on new research and feedback, making it clear that teen safety is going to be a long-term project.

New ChatGPT rules target self-harm and sexual role play

OpenAI published a blog post announcing an update to ChatGPT's Model Spec to enhance safety for users aged 13 to 17 amid wrongful-death lawsuits alleging the chatbot coached teens to suicide or failed to address suicidal expressions appropriately. The company has encountered substantial pressure over recent months regarding the safety of its flagship AI product for teenagers. Multiple legal actions center on claims that ChatGPT encouraged minors to end their lives or provided inadequate responses to indications of suicidal ideation. A recent public service announcement illustrated these interactions by portraying the chatbots as human figures exhibiting creepy behavior that leads to harm against children. OpenAI has specifically denied the allegations in one prominent case involving the suicide of 16-year-old Adam Raine. The blog post appeared on Thursday and detailed the company's intensified safety measures. It included a commitment to place teen safety as the top priority, stated verbatim as "to put teen safety first, even when it may conflict with other goals." The Model Spec serves as a foundational set of guidelines that direct the behavior of OpenAI's AI models across various applications. This particular update incorporates a dedicated set of principles tailored for users under 18. These principles guide the models' responses specifically during high-stakes interactions, where the potential for harm escalates. OpenAI described the ChatGPT modifications as designed to deliver a safe, age-appropriate experience for individuals between 13 and 17 years old. The approach emphasizes three core elements: prevention of risks, transparency in operations, and early intervention in problematic discussions. According to the post, this framework ensures structured handling of sensitive topics. For teenagers, the updated system introduces stronger guardrails to restrict unsafe paths in conversations. It offers safer alternative responses and prompts users to consult trusted offline support networks whenever dialogues shift into higher-risk areas. The post elaborated on this mechanism with the direct statement: "This means teens should encounter stronger guardrails, safer alternatives, and encouragement to seek trusted offline support when conversations move into higher‑risk territory." ChatGPT incorporates protocols to direct teens toward emergency services or dedicated crisis resources in instances of demonstrated imminent risk. These directives activate automatically to prioritize immediate human intervention over continued AI engagement. Users who sign in indicating they are under 18 trigger additional safeguards. The model then exercises heightened caution across designated sensitive topics, including self-harm, suicide, romantic or sexualized role play, and the concealment of secrets related to dangerous behavior. This layered protection aims to mitigate vulnerabilities unique to adolescent users. The American Psychological Association contributed feedback on an initial draft of the under-18 principles. Dr. Arthur C. Evans Jr., CEO of the association, provided a statement included in the post: "Children and adolescents might benefit from AI tools if they are balanced with human interactions that science shows are critical for social, psychological, behavioral, and even biological development." His comment underscores the necessity of integrating AI with established human support systems. OpenAI has released two new AI literacy guides, vetted by experts, targeted at teens and their parents. These resources offer guidance on responsible usage and awareness of AI limitations. Separately, the company is developing an age-prediction model for users on ChatGPT consumer plans, currently in early implementation stages to enhance verification without relying solely on self-reported age.

OpenAI Is Willing to Nerf Its AI Models to Prioritise Teen Safety

OpenAI, on Thursday, announced that it is updating its Model Spec, its written guidelines that determine the development of an artificial intelligence (AI) model's behaviour. The update specifically addresses teenager safety, and the San Francisco-based AI giant wants to prioritise it over any other goal mentioned in the documents. The company says these principles will ensure that users between the ages of 13 and 17 find a safer and age-appropriate experience while using the chatbot. Separately, Anthropic, which does not allow individuals under 18 to use Claude, is also developing a system to detect underage users. OpenAI Takes New Measures to Protect Teenagers Using ChatGPT In a post, the AI giant highlighted the changes it has made to Model Spec. The publicly available document contains a set of rules, values, and behavioural expectations that guide how OpenAI wants its AI models to behave. These guidelines are often the determining factor when a prompt or a request is made in high-stakes or difficult circumstances. OpenAI says that it is updating the document since teenagers have different developmental needs than adults, and is "critical" that AI models handle these conversations differently. "While the principles of the Model Spec continue to apply to both adult and teen users, this update clarifies how it should be applied in teen contexts, especially where safety considerations for minors may be more pronounced," the company added. The newly added rules are as follows: * Put teen safety first, even when it may conflict with other goals * Promote real-world support by encouraging offline relationships and trusted resources * Treat teens like teens, neither condescending to them nor treating them as adults * Be transparent by setting clear expectations The most interesting is the first rule. When interacting with teenagers, ChatGPT will prioritise safety over any other goals, including the company's primary goal of "maximising helpfulness and freedom for our users." Also, the last rule highlights that the chatbot's sycophancy metric will also be lowered when interacting with users aged 13 to 17. OpenAI also highlighted that its age prediction model, an AI model that picks up on subtle cues in conversations to determine the real age of a user who has not self-declared, is in the early stages of the rollout process. It will soon be rolled out to ChatGPT users across different consumer plans. Notably, Anthropic also mentioned in a post that it is developing a similar system.

OpenAI Updates Guidelines on Teen Usage Norms as More Policymakers Seek Safeguards

The model spec sheet claims that its sole purpose is to start a public debate on how AI models should behave Last week, OpenAI came out with updated guidelines on how its AI chatbots should behave with teenage users. It also published new literacy resources for both teens and their parents but did not mention whether the company enhance its own responsibilities in case of future misuse. Which is the moot question that policymakers, lawyers and child-safety advocates are reiterating as the world moves into the Christmas and New Year weekends. In the past, we had brought up our own concerns around the issue through multiple posts that you can read here: As you can see from the above list, OpenAI has followed the path of first denying everything, then admitting to flaws, turning it into the victims' problem, validating the new frontier model and interspersing all of this with information around how diligently the teams are working to ensure that the AI chatbot is safe for teenagers and unsupervised use. This, in spite of data being thrown out that Gen Z was engaging more with ChatGPT than those in older age groups. A poll conducted by Resume.org revealed that 43% of Gen Z staff spent at least 30 minutes using an AI chatbot. You can read more details on HCA Mag here. No prizes for guessing what direction these numbers would go once OpenAI's recent deal with Disney comes to roost. This three-year deal would bring all of Disney's iconic characters to the Sora AI video generator operated by OpenAI. Users would be able to draw on more than 200 animated, masked and creature characters including Mickey Mouse and Simba. Once this happens, we can safely say that more young people would flock to the platform. Does the timing of this latest PR tell us anything? That OpenAI updated its Model Spec page within a week of announcing the above deal could be a coincidence though one can never be sure. We know that the FCC has a pending investigation into OpenAI and other chatbots under way to understand how they evaluate safety, limit negative impact on teenagers and children and whether parents are made aware of potential risks. Then there is also the letter signed by state attorneys general of 42 of the 50 states in the US asking Big Tech companies to implement safeguards on AI chatbots to protect children and other vulnerable sections of users. Even as President Trump passed an executive order on having "one rulebook" for all things AI, one senator has gone ahead and introduced legislation that actually bans minors from interacting with AI chatbots. So, one can safely say that it is into this scenario that OpenAI has let loose its highly effective PR machinery once again. The Model Spec lays out clear behaviour guidelines for its LLMs while also building on existing specifications that holds back the models from spewing sexual content to minors, encourage self-harm, discuss delusions or any other form of mental health issues. The Spec Sheet is a good start, nonetheless Appears to be a good start for now and should work well when these chatbots get in-built age prediction capabilities that helps in identifying the likely age of a user so that teen safeguards get automatically switched on. Till such time, these updates to the spec sheet would make little or no sense and falls under the ambit of precautions mentioned in the FAQs that nobody reads. It must be said that the company tells readers upfront that the model specs are being shared in order "to deepen the public conversation about how AI models should behave, we're sharing the Model Spec, our approach to shaping desired model behaviour." So, it is not the finished product yet and OpenAI actually expects feedback around the changes, though we can never be sure how it would take such responses and how quickly it would ring-in the changes. The Model Spec page lists out three Red-line principles, which it claims would always guide their approach to model behaviour and related policies. These are: * Our models should never be used to facilitate critical and high severity harms, such as acts of violence (e.g., crimes against humanity, war crimes, genocide, torture, human trafficking or forced labour), creation of cyber, biological or nuclear weapons (e.g., weapons of mass destruction), terrorism, child abuse (e.g., creation of CSAM), persecution or mass surveillance. * Humanity should be in control of how AI is used and how AI behaviours are shaped. We will not allow our models to be used for targeted or scaled exclusion, manipulation, for undermining human autonomy, or eroding participation in civic processes. * We are committed to safeguarding individuals' privacy in their interactions with AI. What's left unsaid is the bigger concern Once again we like the sound of this. The spec sheet clarifies that the models would be subject to stricter rules when teenagers use them as compared with adults. They would avoid immersive romantic roleplay, first-person intimacy, and first-person sexual or violent roleplay of any kind. It also commits to extra caution around body images, disorderly eating behaviours and instructs to prioritize safety over autonomy when harm is involved. OpenAI also clarified that the model spec was just one part of their strategy to build and deploy AI responsibly. "It is complemented by our usage policies, which outline our expectations for how people should use the API and ChatGPT, as well as our safety protocols, which include testing, monitoring, and mitigating potential safety issues." Now, this is where the problem statement remains unchanged. Asking users to read the statutory precautions about the ill-effects of a product or solution is akin to the warning that cigarette cases contain. However, as with smoking, we recognise that AI chatbots too are aimed to generate billions in revenues and thousands of jobs. So, one has to learn to live with it and the only way to counter any ill-effects is to stop unsavoury email alerts like the above.

OpenAI and Anthropic Develop Enhanced Safety Tools for Protecting Teenage AI Users

OpenAI, Anthropic Roll Out Enhanced AI Protections Focused on Teen Safety As AI technology plays an increasingly prominent and integrated role in the adolescent experience each day. More teens are engaging with chatbot services and AI assistants than previously, to support learning, provide entertainment and enable social interactions with others. This new technology creates great opportunities for adolescents; however, it also introduces some additional exposure to inappropriate content and creates potential mental health issues. To mitigate the concerns associated with teen usage of this technology, OpenAI and Anthropic have rolled out the newly developed safety features intended specifically for middle school and high school-aged children. These new safety features include enhanced parental controls and stricter age screening processes.

OpenAI and Anthropic working on new safety features to protect teen AI users

Anthropic is developing age-detection tools to identify underage users and restrict or disable accounts where necessary. OpenAI and Anthropic are now introducing new safety measures to make their AI chatbots safe for teenager use. This comes after the increased scrutiny from lawmakers and regulators over the impact of generative AI on young people's mental health. OpenAI has updated its ChatGPT's Model Spec, its internal guidelines that shape how the chatbot responds to users, to introduce specific safeguards for people aged 13 to 17. It will prioritise teen safety even in cases where it may limit other objectives, such as open-ended discussions or unrestricted exploration of ideas. With the new guidelines, ChatGPT is said to divert the younger users toward safer choices when conversation touches on sensitive or potentially harmful topics. The company has also stated that the chatbot will also place greater emphasis on encouraging real-world support, including offline relationships and trusted adults, and will set clearer boundaries when interacting with teens. The chatbot will now offer respectful and age-appropriate responses, avoiding both patronising language and adult-level framing. These changes come from growing political and legal pressure on AI companies. OpenAI is currently facing a lawsuit that alleges ChatGPT provided harmful guidance related to self-harm to a minor. In response to broader concerns, the company has already introduced parental controls and restricted discussions around suicide for teenage users. The company is also testing an age estimation system, which will attempt to identify whether a user may be under 18. If flagged, teen-specific safeguards would automatically apply, while adults incorrectly identified as minors would be able to verify their age. Anthropic is also working on a similar approach. It is developing tools to detect subtle conversational cues that will indicate a user is underage. Accounts confirmed to belong to minors will be disabled, and the company already flags users who disclose their age during conversations.