Incident sheet · /security
- System
- Security & Risk
- Filed
- Read
- 3 min · 536 words
- Severity
- Routine
OpenAI adds CSAM classifier and ChatGPT safety controls
OpenAI’s selective, free API classifier cut flagged images by 48% while finding the same number of abusive images, as ChatGPT added 24-hour suspensions.

Event log
- In early partner tests, the classifier flagged 48% fewer images while finding the same number of abusive images.
- Images flagged by the classifier were twice as likely to be abusive.
- ChatGPT adds a 24-hour suspension for prompts intended to sexualize children; repeat offenses can trigger removal.
- Selected companies can access the classifier through OpenAI’s API at no cost.
- Over the previous 12 months, OpenAI released a safety blueprint, stricter content rules, model retuning and a teen ChatGPT version.
By Olivia Hart
OpenAI’s new child sexual abuse material (CSAM) image classifier, developed with the National Center for Missing and Exploited Children, cut flagged images by 48% in early partner tests while finding the same number of abusive images. For review teams triaging those images, the immediate gain is capacity: OpenAI said the system reduces reviewer work without losing detections.
OpenAI announced four linked safety measures: the classifier, new ChatGPT warnings and temporary blocks, developer guidelines for OpenAI tools, and tools for law enforcement handling child-safety cases. Select companies can access the classifier through OpenAI’s API at no cost.
The release comes amid growing safety concerns. Safety nonprofit Common Sense Media labeled OpenAI’s technology “unacceptable risk” for teens, while several lawsuits allege the company’s technology harmed users, especially children. Separately, Grok has been used to create nonconsensual sexualized imagery, including images of children.
The classifier detects CSAM images, including material generated by AI. It examines visual signs of abuse in images the system has not seen before, then assigns a score that gives reviewers an indicative direction for handling an image.
OpenAI developed the classifier with NCMEC and is working with Tech Coalition and ROOST to share access. Its early partner results broke down this way:
- 48% fewer images flagged.
- The same number of abusive images found.
- Images flagged by the classifier were twice as likely to be abusive.
OpenAI said the lower flag volume reduces reviewers’ workload while retaining the same number of abusive-image detections. The combination of fewer flags and a higher concentration of abusive material gives review teams a shorter, more focused queue.
What changes for developers and child-safety teams?
OpenAI is adding warning screens to ChatGPT for prompts intended to create or access material that sexualizes children. The app will show an alert, direct the user toward specialized support to stop harmful behavior and suspend the account for 24 hours as a warning. ChatGPT already blocks CSAM and related requests; repeated offenses may lead to permanent account removal.
OpenAI formed the message and guidelines around this behavior with Protect Children, the Lucy Faithfull Foundation and Joel Scanlan at the University of Tasmania. Scanlan independently evaluated a trial of the deterrent messages.
The developer guidance tells teams how to prevent, detect and help users report CSAM in products built with OpenAI tools. OpenAI said early testing helped a number of developers adopt child-safety tooling and register to report CSAM to NCMEC.
For law enforcement, OpenAI is adding AI-powered dedicated plugins, investigative tools and complex data analysis for child-safety work. Program administrators will manage access. The company also plans to train investigators and provide a field guide on how to use AI.
Over the previous 12 months, OpenAI has tightened its controls:
- It released a safety blueprint.
- It updated content guidelines to bar inappropriate content.
- It retuned its model to prevent harmful outputs.
- It released a special version of ChatGPT for teens.
These controls predate the classifier and ChatGPT warning screens in the latest package.
For enterprises, the practical change is no-cost classifier access for selected API users, clearer reporting guidance and a defined 24-hour enforcement step before possible ChatGPT account removal.
via techcrunch.com (Source)
More from Olivia Hart
Show full bio
News editor covering money, funding and market shifts around enterprise information technology and the leaders who run at CIO Monitor.
207 articles
The newsroom digest
One email a day with the stories that matter. No noise.