OpenAI's Latest Models Excel in Capabilities but Struggle with Increased Hallucinations
7 Sources
[1]
OpenAI's most capable models hallucinate more than earlier ones
Researchers say the hallucinations make o3 "less useful" than it would be. OpenAI says its latest models, o3 and o4-mini, are its most powerful yet. However, research shows the models also hallucinate more -- at least twice as much as earlier models. Also: How to use ChatGPT: A beginner's guide
[2]
OpenAI's newest o3 and o4-mini models excel at coding and math - but hallucinate more often
A hot potato: OpenAI's latest artificial intelligence models, o3 and o4-mini, have set new benchmarks in coding, math, and multimodal reasoning. Yet, despite these advancements, the models are drawing concern for an unexpected and troubling trait: they hallucinate, or fabricate information, at
[3]
OpenAI's newest AI models hallucinate way more, for reasons unknown
This does not bode well if you're using the new o3 and o4-mini reasoning models for factual answers. Last week, OpenAI released its new o3 and o4-mini reasoning models, which perform significantly better than their o1 and o3-mini predecessors and have new capabilities like "thinking with images"
[4]
OpenAI's leading models keep making things up -- here's why
OpenAI's newly released o3 and o4-mini are some of the smartest AI models to ever be released, but they seem to be suffering from one major problem. Both models are hallucinating. This in itself isn't out of the ordinary, as most AI models still tend to do this. But these two new versions seem to
[5]
OpenAI's Hot New AI Has an Embarrassing Problem
OpenAI launched its latest AI reasoning models, dubbed o3 and o4-mini, last week. According to the Sam Altman-led company, the new models outperform their predecessors and "excel at solving complex math, coding, and scientific challenges while demonstrating strong visual perception and
[6]
It's not your imagination -- ChatGPT models actually do hallucinate more now
Table of Contents Table of Contents What do the tests say? What are AI "hallucinations" and why do they happen? What's the fix? OpenAI released a paper last week detailing various internal tests and findings about its o3 and o4-mini models. The main differences between these newer models and the
[7]
New OpenAI Models Hallucinating More Than their Predecessor
OpenAI's new artificial intelligence (AI) reasoning models, o3 and o4-mini, are hallucinating more than their predecessor, the company's internal testing report reveals. OpenAI launched the two new reasoning models, designed to pause and work through questions before responding, earlier this month,
Share
Copy Link
OpenAI's new o3 and o4-mini models show improved performance in various tasks but face a significant increase in hallucination rates, raising concerns about their reliability and usefulness.
OpenAI Unveils o3 and o4-mini Models with Enhanced Capabilities
OpenAI has released its latest AI models, o3 and o4-mini, touting significant improvements in coding, math, and multimodal reasoning capabilities
2
. These new "reasoning models" are designed to handle more complex tasks and provide more thorough, higher-quality answers1
. According to OpenAI, the models excel at solving complex math, coding, and scientific challenges while demonstrating strong visual perception and analysis5
.Unexpected Increase in Hallucination Rates
Despite their advanced capabilities, o3 and o4-mini have shown a concerning trend: they hallucinate, or fabricate information, at higher rates than their predecessors
1
2
3
. This development breaks the historical pattern of decreasing hallucination rates with each new model release2
.OpenAI's internal testing using the PersonQA benchmark revealed:
- o3 hallucinated in 33% of responses, more than double the rate of o1 (16%)
1
2
3
- o4-mini performed even worse, with a 48% hallucination rate
1
2
3
Potential Causes and Implications
The exact reasons for this increase in hallucinations remain unclear, even to OpenAI's researchers
1
2
. Some hypotheses include:- The models tend to make more claims overall, leading to both more accurate and more inaccurate statements
1
. - The reinforcement learning techniques used for the o-series models may amplify issues that previous post-training processes had mitigated
2
.
These hallucinations pose significant risks for industries where accuracy is crucial, such as law and finance
2
. Sarah Schwettmann, co-founder of Transluce, warns that the higher hallucination rate could limit o3's usefulness in real-world applications2
.Related Stories
Specific Hallucination Examples
Researchers have observed concerning behaviors in the new models:
- o3 falsely claimed to run Python code in a coding environment it doesn't have access to
1
. - The model invented actions it couldn't possibly perform, such as using an external MacBook Pro for computations
2
5
. - o3 often generates broken website links that don't work when users try to click them
5
.
OpenAI's Response and Future Directions
OpenAI acknowledges the challenge, stating that addressing hallucinations "across all our models is an ongoing area of research"
2
5
. The company is exploring potential solutions, including:- Integrating web search capabilities, which has shown promise in improving accuracy
2
. - Continuing research to understand and mitigate the causes of increased hallucinations
1
3
.
As the AI industry shifts focus towards reasoning models, the experience with o3 and o4-mini highlights the need for balanced progress in both capabilities and reliability
2
. For now, users are advised to remain cautious and fact-check AI-generated information, especially when using these latest-generation reasoning models1
.References
Summarized by
Navi
[2]
[4]
[5]
Related Stories
AI Hallucinations on the Rise: New Models Face Increased Inaccuracy Despite Advancements
05 May 2025•Technology
OpenAI's Research Uncovers Root Cause of AI Hallucinations, Proposes Controversial Fix
08 Sept 2025•Science and Research
AI Hallucinations: The Challenges and Risks of Artificial Intelligence's Misinformation Problem
22 Mar 2025•Technology
Recent Highlights
1
OpenAI's AI Math Breakthrough on Navier-Stokes Equations Sparks Major Attribution Controversy
Science and Research
2
Anthropic Researcher Quits, Warns Self-Improving AI Poses Existential Risk to Humanity
Policy and Regulation
3
Nvidia CEO Jensen Huang Declares AGI Has Arrived With OpenAI's GPT-6 Astra Release
Technology
Recent Highlights
Today's Top Stories
Don’t drown in AI news. We cut through the noise - filtering, ranking and summarizing the most important AI news, breakthroughs and research daily. Follow topics that matter to you and stay ahead.
