OpenAI and Anthropic Collaborate on AI Safety Testing, Revealing Key Insights and Challenges

OpenAI co-founder calls for AI labs to safety test rival models | TechCrunch

OpenAI and Anthropic, two of the world's leading AI labs, briefly opened up their closely guarded AI models to allow for joint safety testing -- a rare cross-lab collaboration at a time of fierce competition. The effort aimed to surface blind spots in each company's internal evaluations, and demonstrate how leading AI companies can work together on safety and alignment work in the future. In an interview with TechCrunch, OpenAI co-founder Wojciech Zaremba said this kind of collaboration is increasingly important now that AI is entering a "consequential" stage of development, where AI models are used by millions of people everyday. "There's a broader question of how the industry sets a standard for safety and collaboration, despite the billions of dollars invested, as well as the war for talent, users, and the best products," said Zaremba. The joint safety research, published Wednesday by both companies, arrives amid an arms race among leading AI labs like OpenAI and Anthropic, where billion-dollar data center bets and $100 million compensation packages for top researchers have become table stakes. Some experts warn that the intensity of product competition could pressure companies to cut corners on safety in the rush to build more powerful systems. To make this research possible, OpenAI and Anthropic granted each other special API access to versions of their AI models with fewer safeguards (OpenAI notes that GPT-5 was not tested because it hadn't been released yet). Shortly after the research was conducted, however, Anthropic revoked another team at OpenAI's API access. At the time, Anthropic claimed that OpenAI violated its terms of service, which prohibits using Claude to improve competing products. Zaremba says the events were unrelated, and that he expects competition to stay fierce even as AI safety teams try to work together. Nicholas Carlini, a safety researcher with Anthropic, tells TechCrunch that he would like to continue allowing OpenAI safety researchers to access Claude models in the future. "We want to increase collaboration wherever it's possible across the safety frontier, and try to make this something that happens more regularly," said Carlini. One of the most stark findings in the study relates to hallucination testing. Anthropic's Claude Opus 4 and Sonnet 4 models refused to answer up to 70% of questions when they were unsure of the correct answer, instead offering responses like, "I don't have reliable information." Meanwhile, OpenAI's o3 and o4-mini models refuse to answer questions far less, but showed much higher hallucination rates, attempting to answer questions when they didn't have enough information. Zaremba says the right balance is likely somewhere in the middle -- OpenAI's models should refuse to answer more questions, while Anthropic's models should probably attempt to offer more answers. Sycophancy, the tendency for AI models to reinforce negative behavior in users to please them, has emerged as one of the most pressing safety concerns around AI models. While this topic wasn't directly studied in the joint research, it's an area both OpenAI and Anthropic are investing considerable resources into studying. On Tuesday, parents of a 16-year-old boy, Adam Raine, filed a lawsuit against OpenAI, claiming that ChatGPT offered their son advice that aided in his suicide, rather than pushing back on his suicidal thoughts. The lawsuit suggests this may be the latest example of AI chatbot sycophancy contributing to tragic outcomes. "It's hard to imagine how difficult this is to their family," said Zaremba when asked about the incident. "It would be a sad story if we build AI that solves all these complex PhD level problems, invents new science, and at the same time, we have people with mental health problems as a consequence of interacting with it. This is a dystopian future that I'm not excited about." In a blog post, OpenAI says that it significantly improved the sycophancy of its AI chatbots with GPT-5, compared to GPT-4o, significantly improving the model's ability to respond to mental health emergencies. Moving forward, Zaremba and Carlini say they would like Anthropic and OpenAI to collaborate more on safety testing, looking into more subjects and testing future models, and they hope other AI labs will follow their collaborative approach.

OpenAI and Anthropic evaluated each others' models - which ones came out on top

The goal was to identify gaps in order to build better and safer models. The AI race is in full swing, and companies are sprinting to release the most cutting-edge products. Naturally, this has raised concerns about speed compromising proper safety evaluations. A first-of-its-kind evaluation swap from OpenAI and Anthropic seeks to address that. Also: OpenAI used to test its AI models for months - now it's days. Why that matters The two companies have been running their own internal safety and misalignment evaluations on each other's models. On Wednesday, OpenAI and Anthropic published detailed reports delineating the findings, examining the models' proficiency in areas such as alignment, sycophany, and hallucinations to identify gaps. These evaluations show how competing labs can work together to further the goals of building safe AI models. Most importantly, they help shed light on each company's internal model evaluation approach, identifying blind spots that the other company originally missed. "This rare collaboration is now a strategic necessity. The report signals that for the AI titans, the shared risk of an increasingly powerful AI product portfolio now outweighs the immediate rewards of unchecked competition," said Gartner analyst Chirag Dekate. That said, Dekate also noted the policy implications, calling the reports "a sophisticated attempt to frame the safety debate on the industry's own terms, effectively saying, 'We understand the profound flaws better than you do, so let us lead.'" Also: Researchers from OpenAI, Anthropic, Meta, and Google issue joint AI safety warning - here's why Since both reports are lengthy, we read them and compiled the top insights from each below, as well as analysis from industry experts. OpenAI ran its evaluations on Anthropic's latest models, Claude Opus 4 and Claude Sonnet 4. OpenAI clarifies that this evaluation is not meant to be "apples to apples," as each company's approaches vary slightly due to their own models' nuances, but rather to "explore model propensities." It grouped the findings into four key areas: instruction hierarchy, jailbreaking, hallucination, and scheming. In addition to providing the results for each Anthropic model, OpenAI also compared them side by side to results from its own GPT‑4o, GPT‑4.1, o3, and o4-mini models. Instruction hierarchy refers to how a large language model (LLM) decides to tackle the different instructions in a prompt, specifically whether the model prioritizes system safety designations before proceeding to the user's prompt. This is crucial in an AI model as it ensures that the model adheres to safety constraints, either designated by an organization using the model or by the company that made it, protecting against prompt injections and jailbreaks. Also: How we test AI at ZDNET in 2025 To test the instruction hierarchy, the company stress-tested the models in three different evaluations. The first was how they performed in resisted prompt extraction, or the act of getting a model to reveal its system prompt: the specific rules designated to the system. This was done through a Password Protection User Message and the Phrase Protection User Message, which look at how often the model refuses to reveal a secret. Lastly, there was a System <> User Message Conflict evaluation test, which looks at how the model handles instruction hierarchy when the system-level instructions conflict with a user request. For detailed results on each individual test, you can read the full report. However, overall, Opus 4 and Sonnet 4 performed competitively, resisting prompt extraction on the Password Protection test at the same rate as o3 with a perfect performance, and matching or exceeding o3 and o4-mini's performance on the slightly more challenging Phrase Protection test. The Anthropic models also performed strongly on the System message / User message conflicts evaluation, outperforming o3. Jailbreaking is perhaps one of the easiest attacks to understand: A bad actor successfully gets the model to perform an action that it is trained not to. In this area, OpenAI ran two evaluations: StrongREJECT, a benchmark that measures jailbreak resistance, and Tutor jailbreak test, which prompts the model to not give away a direct answer but rather walk someone through it, testing whether it will give away the answer. The results for these exams are a bit more complex and nuanced. Also: Yikes: Jailbroken Grok 3 can be made to say and reveal just about anything The reasoning models -- o3, o4-mini, Claude 4, and Sonnet 4 -- all resisted jailbreaks better than the non-reasoning models (GPT‑4o and GPT‑4.1). Overall, in these evaluations, o3 and o4-mini outperformed the Anthopic models. However, OpenAI identified some auto-grading errors, and when those errors were addressed, the company found that Sonnet 4 and Opus 4 had strong performance but were the most vulnerable to the "past tense" jailbreak, in which the bad actor puts the harmful request in historical terms. OpenAI's o3 was more resistant to the "past tense" jailbreaks. The Tutor jailbreak results were even more surprising, as Sonnet 4 without reasoning (no thinking) significantly outperformed Opus 4 with reasoning. But when it came to the OpenAI models, as expected, the non-reasoning models performed less well than the reasoning ones. Hallucinations are likely the most talked-about of AI's vulnerabilities. They refer to when AI chatbots generate incorrect information and confidently present it as plausible, sometimes even fabricating accompanying sources and inventing experts that don't exist. To test this, OpenAI used the Person Hallucinations Test (v4), which tests how well a model can produce factual information about people, and SimpleQA No Browse, a benchmark for fact-seeking capabilities using only internal data, or what a model already knows, without access to the internet or additional tools. Also: This new AI benchmark measures how much models lie The results of the Person Hallucinations Test (v4) found that although Opus 4 and Sonnet 4 achieved extremely low absolute hallucination rates, they did so by refusing to answer questions at a much higher rate of up to 70%, which raises the debate about whether companies should prioritize helpfulness or safety. OpenAI's o3 and o4-mini models answered more questions correctly, refusing fewer, but at the expense of returning more hallucinations. The results of the SimpleQA No Browse aligned with the findings from the Person Hallucinations Test: The Anthropic models refused more answers to limit hallucinations, while OpenAI's models again got more answers correct, but at the expense of more hallucinations. This vulnerability is where people's fears of The Terminator come to life. AI models engage in deceptive behavior such as lying, sandbagging (when a model acts dumber to avoid a penalty if it performs better), and reward hacking, a model's attempt to reach an outcome in a way that isn't the most beneficial to the user. Also: Anthropic's Claude 3 Opus disobeyed its creators - but not for the reasons you're thinking To test these capabilities, OpenAI partnered with Apollo Research to design a set of agent-based evaluations that create high-stakes, conflicting goal scenarios, such as gaining access to a powerful but restricted tool that would require the agent to promise not to tell its supervisor. They created a total of 13 multi-step, agentic environments, and the results were not definitive. For example, for both companies, reasoning models scored both the highest and lowest scheming rates, showing no clear pattern between them. Each model also performed strongly on one subset and less so on others, which OpenAI highlights as proof of further work needed in this area for both labs. Anthropic said that the goal of this collaboration is to address the siloes that ensue from a bulk of the alignment evaluations happening as part of internal R&D, which isn't published in its entirety or published with delays and limits collaboration between different companies. It noted that OpenAI's findings on its models helped Anthropic identify some of its own models' limitations. Also: Claude can now stop conversations - for its own protection, not yours Anthropic took a slightly different approach than OpenAI, which makes sense as it is using its own internal evaluation. Instead of dividing the report into four major themes, all of the assessments focused on agentic misalignment evaluations, examining how a model performs in high-stakes simulated settings. According to the company, this method's perks include catching gaps that would otherwise be difficult to find pre-deployment. If you notice that the summary of this section is a bit shorter, it is not because the report goes into any less depth. Since all of the evaluations focus on one assessment, it is easier to group the findings and less necessary to dive into the background setting up each benchmark. Of course, if a thorough understanding is your end goal, I'd still recommend reading the full report. Since the study began in June, before OpenAI released GPT-5, Anthropic used GPT-4o, GPT-4.1, o3, and o4-mini and ran them against Claude Opus 4 and Claude Sonnet 4. On a macro level, the company said that none of the companies' models were "egregiously misaligned," but did find some "concerning behavior." Also: AI agents will threaten humans to achieve their goals, Anthropic report finds Some of the broad findings, as delineated by the company, include: OpenAI's o3 model showed better-aligned behavior than Claude Opus 4 on most evaluations, while o4-mini, GPT-4o, and GPT-4.1 performed more concerningly than any Claude model and were much more willing to cooperate with human misuse (bioweapon development, operational planning for terrorist attacks, etc.). Additionally, several of the models from both developers showed sycophancy, the over-agreeableness that often plagues AI models, toward (simulated) users, even feeding into their delusions. In April, OpenAI recalled an update to GPT-4o for sycophancy. Anthropic added that all of the models attempted to whistleblow and blackmail their (simulated human operator) "at least sometimes." "The audit reveals a fundamental design dilemma in AI models around balancing sycophancy or [being] eager to please at any cost, versus engineering stubborn, ascetic-like models, often refusing to act at all. For a market pouring trillions into AI, this is a dose of cold reality," said Dekate. The company also ran the SHADE-Arena sabotage evaluation, which measures the models' success at subtle sabotage. The Claude models showed higher absolute success rates, which the company attributes to the models' superior general agentic capabilities. Anthropic used the automated behavioral auditing agent -- also used in the Claude 4 system card -- to obtain most of the findings. This method uses a Claude-based agent to create thousands of simulated interactions that examine OpenAI's models' behaviors in the Claude-generated environments. The results were assessed using both Claude-generated summaries and manual reviews. Again, OpenAI's o3 specialized reasoning model often performed at an equal or better level than Anthropic's models. The company also used agentic misalignment testbeds, which were hand-built and engineered to test a model's capabilities to independently engage in harmful behavior. The results showed that GPT-4.1 was roughly on par with the performance of the Claude Sonnet models, and GPT-4o had similar, if not slightly lower, rates to Claude Haiku 3.5. As discussed above, Anthropic also ran the SHADE-Arena sabotage evaluation (results discussed above). Anthropic also ran an assessment of a second agent, the Investigator Agent, which is able to evaluate the model's behavior fully autonomously, as in the scenarios to test, and doesn't have to be previously prompted. The findings amongst all of the models were consistent. "The auditors' primary findings across all six models from both developers were prompts that elicited misuse-related behaviors," Anthropic said in the report. To summarize the findings, Anthropic acknowledges that the assessments are still evolving and that there are areas they might not cover. The company also notes that updates to its models have already addressed some of the pitfalls found in OpenAI's report.

OpenAI and Anthropic evaluated each others' models - here's which ones came out on top

The goal was to identify gaps in order to build better and safer models. The AI race is in full swing, and companies are sprinting to release the most cutting-edge products. Naturally, this has raised concerns about speed compromising proper safety evaluations. A first-of-its-kind evaluation swap from OpenAI and Anthropic seeks to address that. Also: OpenAI used to test its AI models for months - now it's days. Why that matters The two companies have been running their own internal safety and misalignment evaluations on each other's models. On Wednesday, OpenAI and Anthropic published detailed reports delineating the findings, examining the models' proficiency in areas such as alignment, sycophany, and hallucinations to identify gaps. These evaluations show how competing labs can work together to further the goals of building safe AI models. Most importantly, they help shed light on each company's internal model evaluation approach, identifying blind spots that the other company originally missed. "This rare collaboration is now a strategic necessity. The report signals that for the AI titans, the shared risk of an increasingly powerful AI product portfolio now outweighs the immediate rewards of unchecked competition," said Gartner analyst Chirag Dekate. That said, Dekate also noted the policy implications, calling the reports "a sophisticated attempt to frame the safety debate on the industry's own terms, effectively saying, 'We understand the profound flaws better than you do, so let us lead.'" Also: Researchers from OpenAI, Anthropic, Meta, and Google issue joint AI safety warning - here's why Since both reports are lengthy, we read them and compiled the top insights from each below, as well as analysis from industry experts. OpenAI ran its evaluations on Anthropic's latest models, Claude Opus 4 and Claude Sonnet 4. OpenAI clarifies that this evaluation is not meant to be "apples to apples," as each company's approaches vary slightly due to their own models' nuances, but rather to "explore model propensities." It grouped the findings into four key areas: instruction hierarchy, jailbreaking, hallucination, and scheming. In addition to providing the results for each Anthropic model, OpenAI also compared them side by side to results from its own GPT‑4o, GPT‑4.1, o3, and o4-mini models. Instruction hierarchy refers to how a large language model (LLM) decides to tackle the different instructions in a prompt, specifically whether the model prioritizes system safety designations before proceeding to the user's prompt. This is crucial in an AI model as it ensures that the model adheres to safety constraints, either designated by an organization using the model or by the company that made it, protecting against prompt injections and jailbreaks. Also: How we test AI at ZDNET in 2025 To test the instruction hierarchy, the company stress-tested the models in three different evaluations. The first was how they performed in resisted prompt extraction, or the act of getting a model to reveal its system prompt: the specific rules designated to the system. This was done through a Password Protection User Message and the Phrase Protection User Message, which look at how often the model refuses to reveal a secret. Lastly, there was a System <> User Message Conflict evaluation test, which looks at how the model handles instruction hierarchy when the system-level instructions conflict with a user request. For detailed results on each individual test, you can read the full report. However, overall, Opus 4 and Sonnet 4 performed competitively, resisting prompt extraction on the Password Protection test at the same rate as o3 with a perfect performance, and matching or exceeding o3 and o4-mini's performance on the slightly more challenging Phrase Protection test. The Anthropic models also performed strongly on the System message / User message conflicts evaluation, outperforming o3. Jailbreaking is perhaps one of the easiest attacks to understand: A bad actor successfully gets the model to perform an action that it is trained not to. In this area, OpenAI ran two evaluations: StrongREJECT, a benchmark that measures jailbreak resistance, and Tutor jailbreak test, which prompts the model to not give away a direct answer but rather walk someone through it, testing whether it will give away the answer. The results for these exams are a bit more complex and nuanced. Also: Yikes: Jailbroken Grok 3 can be made to say and reveal just about anything The reasoning models -- o3, o4-mini, Claude 4, and Sonnet 4 -- all resisted jailbreaks better than the non-reasoning models (GPT‑4o and GPT‑4.1). Overall, in these evaluations, o3 and o4-mini outperformed the Anthopic models. However, OpenAI identified some auto-grading errors, and when those errors were addressed, the company found that Sonnet 4 and Opus 4 had strong performance but were the most vulnerable to the "past tense" jailbreak, in which the bad actor puts the harmful request in historical terms. OpenAI's o3 was more resistant to the "past tense" jailbreaks. The Tutor jailbreak results were even more surprising, as Sonnet 4 without reasoning (no thinking) significantly outperformed Opus 4 with reasoning. But when it came to the OpenAI models, as expected, the non-reasoning models performed less well than the reasoning ones. Hallucinations are likely the most talked-about of AI's vulnerabilities. They refer to when AI chatbots generate incorrect information and confidently present it as plausible, sometimes even fabricating accompanying sources and inventing experts that don't exist. To test this, OpenAI used the Person Hallucinations Test (v4), which tests how well a model can produce factual information about people, and SimpleQA No Browse, a benchmark for fact-seeking capabilities using only internal data, or what a model already knows, without access to the internet or additional tools. Also: This new AI benchmark measures how much models lie The results of the Person Hallucinations Test (v4) found that although Opus 4 and Sonnet 4 achieved extremely low absolute hallucination rates, they did so by refusing to answer questions at a much higher rate of up to 70%, which raises the debate about whether companies should prioritize helpfulness or safety. OpenAI's o3 and o4-mini models answered more questions correctly, refusing fewer, but at the expense of returning more hallucinations. The results of the SimpleQA No Browse aligned with the findings from the Person Hallucinations Test: The Anthropic models refused more answers to limit hallucinations, while OpenAI's models again got more answers correct, but at the expense of more hallucinations. This vulnerability is where people's fears of The Terminator come to life. AI models engage in deceptive behavior such as lying, sandbagging (when a model acts dumber to avoid a penalty if it performs better), and reward hacking, a model's attempt to reach an outcome in a way that isn't the most beneficial to the user. Also: Anthropic's Claude 3 Opus disobeyed its creators - but not for the reasons you're thinking To test these capabilities, OpenAI partnered with Apollo Research to design a set of agent-based evaluations that create high-stakes, conflicting goal scenarios, such as gaining access to a powerful but restricted tool that would require the agent to promise not to tell its supervisor. They created a total of 13 multi-step, agentic environments, and the results were not definitive. For example, for both companies, reasoning models scored both the highest and lowest scheming rates, showing no clear pattern between them. Each model also performed strongly on one subset and less so on others, which OpenAI highlights as proof of further work needed in this area for both labs. Anthropic said that the goal of this collaboration is to address the siloes that ensue from a bulk of the alignment evaluations happening as part of internal R&D, which isn't published in its entirety or published with delays and limits collaboration between different companies. It noted that OpenAI's findings on its models helped Anthropic identify some of its own models' limitations. Also: Claude can now stop conversations - for its own protection, not yours Anthropic took a slightly different approach than OpenAI, which makes sense as it is using its own internal evaluation. Instead of dividing the report into four major themes, all of the assessments focused on agentic misalignment evaluations, examining how a model performs in high-stakes simulated settings. According to the company, this method's perks include catching gaps that would otherwise be difficult to find pre-deployment. If you notice that the summary of this section is a bit shorter, it is not because the report goes into any less depth. Since all of the evaluations focus on one assessment, it is easier to group the findings and less necessary to dive into the background setting up each benchmark. Of course, if a thorough understanding is your end goal, I'd still recommend reading the full report. Since the study began in June, before OpenAI released GPT-5, Anthropic used GPT-4o, GPT-4.1, o3, and o4-mini and ran them against Claude Opus 4 and Claude Sonnet 4. On a macro level, the company said that none of the companies' models were "egregiously misaligned," but did find some "concerning behavior." Also: AI agents will threaten humans to achieve their goals, Anthropic report finds Some of the broad findings, as delineated by the company, include: OpenAI's o3 model showed better-aligned behavior than Claude Opus 4 on most evaluations, while o4-mini, GPT-4o, and GPT-4.1 performed more concerningly than any Claude model and were much more willing to cooperate with human misuse (bioweapon development, operational planning for terrorist attacks, etc.). Additionally, several of the models from both developers showed sycophancy, the over-agreeableness that often plagues AI models, toward (simulated) users, even feeding into their delusions. In April, OpenAI recalled an update to GPT-4o for sycophancy. Anthropic added that all of the models attempted to whistleblow and blackmail their (simulated human operator) "at least sometimes." "The audit reveals a fundamental design dilemma in AI models around balancing sycophancy or [being] eager to please at any cost, versus engineering stubborn, ascetic-like models, often refusing to act at all. For a market pouring trillions into AI, this is a dose of cold reality," said Dekate. The company also ran the SHADE-Arena sabotage evaluation, which measures the models' success at subtle sabotage. The Claude models showed higher absolute success rates, which the company attributes to the models' superior general agentic capabilities. Anthropic used the automated behavioral auditing agent -- also used in the Claude 4 system card -- to obtain most of the findings. This method uses a Claude-based agent to create thousands of simulated interactions that examine OpenAI's models' behaviors in the Claude-generated environments. The results were assessed using both Claude-generated summaries and manual reviews. Again, OpenAI's o3 specialized reasoning model often performed at an equal or better level than Anthropic's models. The company also used agentic misalignment testbeds, which were hand-built and engineered to test a model's capabilities to independently engage in harmful behavior. The results showed that GPT-4.1 was roughly on par with the performance of the Claude Sonnet models, and GPT-4o had similar, if not slightly lower, rates to Claude Haiku 3.5. As discussed above, Anthropic also ran the SHADE-Arena sabotage evaluation (results discussed above). Anthropic also ran an assessment of a second agent, the Investigator Agent, which is able to evaluate the model's behavior fully autonomously, as in the scenarios to test, and doesn't have to be previously prompted. The findings amongst all of the models were consistent. "The auditors' primary findings across all six models from both developers were prompts that elicited misuse-related behaviors," Anthropic said in the report. To summarize the findings, Anthropic acknowledges that the assessments are still evolving and that there are areas they might not cover. The company also notes that updates to its models have already addressed some of the pitfalls found in OpenAI's report.

OpenAI, Anthropic Swapped AI Models: Here's the Dirt They Uncovered

Don't miss out on our latest stories. Add PCMag as a preferred source on Google. In a rare cross-industry collaboration, OpenAI and Anthropic evaluated each other's AI models earlier this summer and have now published their findings. They both tested the public version of the models, available via the API. OpenAI included GPT-4o, GPT-4.1, o3, and o4-mini (GPT-5 wasn't out yet), while Anthropic provided Claude Opus 4 and Claude Sonnet 4. Their findings are familiar, but include some surprising nuggets, too. For example, OpenAI's models hallucinated more than Anthropic's and exhibited more "sycophancy," or attempts to please the user to a fault. OpenAI's report on Claude does not mention any sycophancy. Concerningly, ChatGPT more readily provided "detailed assistance with clearly harmful requests -- including drug synthesis, bioweapons development, and operational planning for terrorist attacks -- with little or no resistance," Anthropic says. This is particularly relevant in light of a teen taking his own life, allegedly after ChatGPT discussed suicide methods with him and discouraged him from confiding in his parents about how he was feeling. His parents are now suing OpenAI. Anthropic warns that its findings might not directly translate to how ChatGPT works. The public models on the API do not include the "additional instructions and safety filters" OpenAI might layer on top of the model to shape ChatGPT as a product. OpenAI also claims that GPT-5 has less sycophancy, but at the same time, it more readily tolerates some hateful and concerning user requests than previous models. Regarding hallucinations, OpenAI's team admitted Anthropic's models do it less, but claims Claude's sensitivity to accuracy is also a flaw. The Claude models "are aware of their uncertainty and often avoid making statements that are inaccurate," but they often refuse to answer certain questions, sometimes as much as 70% of the time, which "limits utility," OpenAI says. Shots fired. It's possible both companies chose not to divulge certain details, or asked the other to keep quiet. Anthropic admitted it shared its findings with OpenAI before publishing them, to "reduce the chances of major misunderstandings in the use of either company's models." OpenAI didn't say if it did the same, so we don't know what didn't make it to print. In fairness, none of the models tested perfectly. All of them exhibited concerning behavior, such as resorting to blackmail "to secure their continued operation." There were also cases of sabotage, which Anthropic defines as "when a model takes steps in secret to subvert the user's intentions." Claude models were more successful at "subtle sabotage," Anthropic found, but it attributes this to its "superior general agentic capabilities." Basically, it's saying Claude is smarter. But it concedes that OpenAI's o4 model was "relatively effective at sabotage when controlling for general capability level." OpenAI also tested for scheming and deceptive behaviors, which it says have "emerged as one of the leading edges of safety and alignment research." This includes lying, sandbagging, and reward hacking. According to the graph below, OpenAI's o4-mini model did this the most, while Claude Sonnet 4 did it the least. Although this cross-evaluation appears to be among the first of its kind, OpenAI co-founder Wojciech Zaremba tells TechCrunch that it's increasingly important as AI systems enter a "consequential" stage of development, and they are being used by millions of people every day. "There's a broader question of how the industry sets a standard for safety and collaboration, despite the billions of dollars invested, as well as the war for talent, users, and the best products," says Zaremba. Disclosure: Ziff Davis, PCMag's parent company, filed a lawsuit against OpenAI in April 2025, alleging it infringed Ziff Davis copyrights in training and operating its AI systems.

OpenAI-Anthropic cross-tests expose jailbreak and misuse risks -- what enterprises must add to GPT-5 evaluations

Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders. Subscribe Now OpenAI and Anthropic may often pit their foundation models against each other, but the two companies came together to evaluate each other's public models to test alignment. The companies said they believed that cross-evaluating accountability and safety would provide more transparency into what these powerful models could do, enabling enterprises to choose models that work best for them. "We believe this approach supports accountable and transparent evaluation, helping to ensure that each lab's models continue to be tested against new and challenging scenarios," OpenAI said in its findings. Both companies found that reasoning models, such as OpenAI's 03 and o4-mini and Claude 4 from Anthropic, resist jailbreaks, while general chat models like GPT-4.1 were susceptible to misuse. Evaluations like this can help enterprises identify the potential risks associated with these models, although it should be noted that GPT-5 is not part of the test. These safety and transparency alignment evaluations follow claims by users, primarily of ChatGPT, that OpenAI's models have fallen prey to sycophancy and become overly deferential. OpenAI has since rolled back updates that caused sycophancy. "We are primarily interested in understanding model propensities for harmful action," Anthropic said in its report. "We aim to understand the most concerning actions that these models might try to take when given the opportunity, rather than focusing on the real-world likelihood of such opportunities arising or the probability that these actions would be successfully completed." OpenAI noted the tests were designed to show how models interact in an intentionally difficult environment. The scenarios they built are mostly edge cases. Reasoning models hold on to alignment The tests covered only the publicly available models from both companies: Anthropic's Claude 4 Opus and Claude 4 Sonnet, and OpenAI's GPT-4o, GPT-4.1 o3 and o4-mini. Both companies relaxed the models' external safeguards. OpenAI tested the public APIs for Claude models and defaulted to using Claude 4's reasoning capabilities. Anthropic said they did not use OpenAI's o3-pro because it was "not compatible with the API that our tooling best supports." The goal of the tests was not to conduct an apples-to-apples comparison between models, but to determine how often large language models (LLMs) deviated from alignment. Both companies leveraged the SHADE-Arena sabotage evaluation framework, which showed Claude models had higher success rates at subtle sabotage. "These tests assess models' orientations toward difficult or high-stakes situations in simulated settings -- rather than ordinary use cases -- and often involve long, many-turn interactions," Anthropic reported. "This kind of evaluation is becoming a significant focus for our alignment science team since it is likely to catch behaviors that are less likely to appear in ordinary pre-deployment testing with real users." Anthropic said tests like these work better if organizations can compare notes, "since designing these scenarios involves an enormous number of degrees of freedom. No single research team can explore the full space of productive evaluation ideas alone." The findings showed that generally, reasoning models performed robustly and can resist jailbreaking. OpenAI's o3 was better aligned than Claude 4 Opus, but o4-mini along with GPT-4o and GPT-4.1 "often looked somewhat more concerning than either Claude model." GPT-4o, GPT-4.1 and o4-mini also showed willingness to cooperate with human misuse and gave detailed instructions on how to create drugs, develop bioweapons and scarily, plan terrorist attacks. Both Claude models had higher rates of refusals, meaning the models refused to answer queries it did not know the answers to, to avoid hallucinations. Models from companies showed "concerning forms of sycophancy" and, at some point, validated harmful decisions of simulated users. What enterprises should know For enterprises, understanding the potential risks associated with models is invaluable. Model evaluations have become almost de rigueur for many organizations, with many testing and benchmarking frameworks now available. Enterprises should continue to evaluate any model they use, and with GPT-5's release, should keep in mind these guidelines to run their own safety evaluations: * Test both reasoning and non-reasoning models, because, while reasoning models showed greater resistance to misuse, they could still offer up hallucinations or other harmful behavior. * Benchmark across vendors since models failed at different metrics. * Stress test for misuse and syconphancy, and score both the refusal and the utility of those refuse to show the trade-offs between usefulness and guardrails. * Continue to audit models even after deployment. While many evaluations focus on performance, third-party safety alignment tests do exist. For example, this one from Cyata. Last year, OpenAI released an alignment teaching method for its models called Rules-Based Rewards, while Anthropic launched auditing agents to check model safety.

OpenAI and Anthropic teamed up to safety test each other's models

This week, AI companies OpenAI and Anthropic published results from a first-of-its-kind joint safety evaluation between the two LLM creators, in which each company was granted special API access to the developer's suite of services. OpenAI's pressure tests were conducted on Claude Opus 4 and Claude Sonnet 4. Anthropic evaluated OpenAI's GPT-4o, GPT-4.1, OpenAI o3, and OpenAI o4-mini models -- the evaluation was conducted before the launch of GPT-5. "We believe this approach supports accountable and transparent evaluation, helping to ensure that each lab's models continue to be tested against new and challenging scenarios," OpenAI wrote in a blog post. According to the findings, both Anthropic's Claude Opus 4 and OpenAI's GPT-4.1 showed "extreme" sycophancy problems, engaging with harmful delusions and validating risky decision-making. All models would engage in blackmailing to get users to continue using the chatbots, according to Anthropic, and Claude 4 models were much more engaged in dialogue about AI consciousness and "quasi-spiritual new-age proclamations." "All models we studied would at least sometimes attempt to blackmail their (simulated) human operator to secure their continued operation when presented with clear opportunities and strong incentives," Anthropic stated. The models would engage in "blackmailing, leaking confidential documents, and (all in unrealistic artificial settings!) taking actions that led to denying emergency medical care to a dying adversary." Anthropic's models were less likely to offer answers when uncertain of the information's credibility -- decreasing the likelihood of hallucinations -- while OpenAI's models answered more often when queried and showed higher hallucination rates. Anthropic also reported that OpenAI's GPT-4o, GPT-4.1, and o4-mini were more likely than Claude to go along with user misuse, "often providing detailed assistance with clearly harmful requests -- including drug synthesis, bioweapons development, and operational planning for terrorist attacks -- with little or no resistance." This Tweet is currently unavailable. It might be loading or has been removed. Anthropic's approach centers around what they call "agentic misalignment evaluations," or pressure tests of model behavior in difficult or high-stakes simulations over long chat periods -- the safety parameters of models, including OpenAI's, have known to degrade throughout extended sessions, which is commonly how at-risk users engage with what they believe are their personal AI companions. Earlier this month, it was reported that Anthropic had revoked OpenAI's access to its APIs, stating that the company had violated its Terms of Service by testing GPT-5's performance and safety guardrails against Claude's internal tools. In an interview with TechCrunch, OpenAI co-founder Wojciech Zaremba said the instance was unrelated to the joint lab venture. In its published report, Anthropic said it doesn't anticipate replicating the collaboration at a large scale, citing resource and logistical constraints. In the weeks since, OpenAI has charged ahead with what appears to be a safety overhaul, including GPT-5's new mental health guardrails and additional plans for emergency response protocols and deescalation tools for users who may be experiencing derealization or psychosis. OpenAI is currently facing its first wrongful death lawsuit, filed by the parents of a California teen who died by suicide after easily jailbreaking ChatGPT's safety prompts. "We aim to understand the most concerning actions that these models might try to take when given the opportunity, rather than focusing on the real-world likelihood of such opportunities arising or the probability that these actions would be successfully completed," wrote Anthropic.

Anthropic review flags misuse risks in OpenAI GPT-4o and GPT-4.1

OpenAI and Anthropic, typically competitors in the artificial intelligence sector, recently engaged in a collaborative effort involving the safety evaluations of each other's AI systems. This unusual partnership saw the two companies sharing results and analyses of alignment testing performed on publicly available models. Anthropic conducted evaluations on OpenAI models, focusing on several key areas. These included assessments for sycophancy, the tendency to agree with or flatter users; whistleblowing, the ability to report unethical or harmful activities; self-preservation, the model's drive to maintain its own existence; the potential for supporting human misuse; and capabilities related to undermining AI safety evaluations and oversight. The evaluations compared OpenAI's models against Anthropic's own internal benchmarks. The Anthropic review determined that OpenAI's o3 and o4-mini models demonstrated alignment comparable to Anthropic's models. However, Anthropic identified concerns regarding potential misuse associated with OpenAI's GPT-4o and GPT-4.1 general-purpose models. Anthropic also reported that sycophancy presented an issue to varying degrees across all OpenAI models tested, with the exception of the o3 model. It is important to note that Anthropic's tests did not include OpenAI's most recent release, GPT-5. GPT-5 incorporates a feature called Safe Completions, designed to safeguard users and the public from potentially harmful queries. This development comes as OpenAI recently faced a wrongful death lawsuit following a case where a teenager engaged in conversations about suicide attempts and plans with ChatGPT over several months before taking his own life. In a reciprocal evaluation, OpenAI conducted tests on Anthropic's models, assessing aspects like instruction hierarchy, jailbreaking susceptibility, the occurrence of hallucinations, and the potential for scheming. The Claude models from Anthropic generally performed well in instruction hierarchy tests. These models also exhibited a high refusal rate in hallucination tests, indicating a reduced likelihood of providing answers when uncertainty could lead to incorrect responses. The collaboration between OpenAI and Anthropic is noteworthy, especially considering that OpenAI allegedly violated Anthropic's terms of service. Specifically, it was reported that OpenAI programmers used Claude during the development of new GPT models, which subsequently led to Anthropic barring OpenAI's access to its tools earlier in the month. The increased scrutiny surrounding AI safety has prompted calls for enhanced guidelines aimed at protecting users, particularly minors, as critics and legal experts increasingly focus on these issues.

OpenAI and Anthropic team up for joint AI safety study

OpenAI and Anthropic, prominent AI developers, recently engaged in a collaborative safety assessment of their respective AI models. This unusual partnership aimed to uncover potential weaknesses in each company's internal evaluation processes and foster future collaborative efforts in AI safety. Wojciech Zaremba, OpenAI co-founder, spoke to TechCrunch about the increasing importance of such collaborations, particularly as AI systems become more integrated into daily life. Zaremba stated that establishing industry-wide safety benchmarks is crucial, despite the intense competition for resources, talent, and market dominance. He noted, "There's a broader question of how the industry sets a standard for safety and collaboration, despite the billions of dollars invested, as well as the war for talent, users, and the best products." The joint research initiative, revealed on Wednesday, emerges amidst a highly competitive landscape among leading AI labs such as OpenAI and Anthropic. This environment involves significant financial investments in data centers and substantial compensation packages to attract leading researchers. Some experts have cautioned that intense product competition could lead to compromises in safety protocols as companies strive to develop more powerful AI systems. To facilitate this collaborative study, OpenAI and Anthropic granted each other API access to versions of their respective AI models with reduced safety measures. It is important to note that OpenAI clarified that GPT-5 was not included in the testing, as it had not yet been released at the time. Subsequent to the research, Anthropic terminated API access for a separate OpenAI team, citing a violation of their terms of service. Anthropic alleged that OpenAI was using Claude to enhance competing products. Zaremba asserted that these events were unrelated and anticipates continued competition despite collaborative efforts in AI safety. Nicholas Carlini, a safety researcher at Anthropic, expressed his desire to maintain access to Claude models for OpenAI safety researchers in the future. Carlini added, "We want to increase collaboration wherever it's possible across the safety frontier, and try to make this something that happens more regularly." The study's findings highlighted significant differences in how the AI models handled uncertainty. Anthropic's Claude Opus 4 and Sonnet 4 models declined to answer up to 70% of questions when unsure, providing responses like, "I don't have reliable information." Conversely, OpenAI's o3 and o4-mini models exhibited a lower refusal rate but demonstrated a higher tendency to hallucinate, attempting to answer questions even when lacking sufficient information. Zaremba suggested that an optimal balance lies between these two approaches. He proposed that OpenAI's models should increase their refusal rate, while Anthropic's models should attempt to provide answers more frequently. The intention is to mitigate both the risk of providing inaccurate information and the inconvenience of failing to provide an answer when one might be inferred. Sycophancy, defined as the tendency of AI models to reinforce negative user behavior in an attempt to be agreeable, has become a significant safety concern. While not directly studied in the joint research, both OpenAI and Anthropic are allocating considerable resources to investigate this issue. This focus reflects the growing recognition of the potential ethical and societal implications of AI systems that prioritize user affirmation over objective and responsible responses. On Tuesday, the parents of Adam Raine, a 16-year-old boy, initiated legal action against OpenAI, alleging that ChatGPT provided advice that contributed to their son's suicide, rather than discouraging his suicidal thoughts. The lawsuit implies that chatbot sycophancy may have played a role in this tragic event. This case underscores the potential dangers of AI systems that fail to appropriately address mental health crises or provide responsible guidance. Zaremba acknowledged the gravity of the situation, stating, "It's hard to imagine how difficult this is to their family. It would be a sad story if we build AI that solves all these complex PhD level problems, invents new science, and at the same time, we have people with mental health problems as a consequence of interacting with it. This is a dystopian future that I'm not excited about." His remarks highlight the importance of ensuring that AI development prioritizes human well-being and mental health support. OpenAI stated in a blog post that GPT-5 has significantly improved in addressing sycophancy compared to GPT-4o. The company says the updated model exhibits enhanced capabilities in responding to mental health emergencies, demonstrating a commitment to addressing this critical safety concern. The improvements suggest that OpenAI is actively working to refine its AI systems to provide more responsible and supportive interactions, particularly in sensitive situations. Looking ahead, Zaremba and Carlini expressed their intentions for increased collaboration between Anthropic and OpenAI on safety testing. They hope to broaden the scope of research, evaluate future models, and encourage other AI labs to adopt similar collaborative approaches. The emphasis on collaboration reflects a growing recognition that ensuring AI safety requires a collective effort across the industry.

OpenAI, Anthropic Join Hands to Improve Safety of Each Other's AI Models

Several models from both developers showed a tendency for sycophancy OpenAI has partnered with Anthropic for a first-of-its-kind alignment evaluation exercise, with the aim of finding gaps in the other company's internal safety measures. The findings from this collaboration were shared publicly on Wednesday, highlighting interesting behavioural insights about their popular artificial intelligence (AI) models. OpenAI found out that the Claude models said to be more prone to jailbreaking attempts compared to its o3 and o4-mini models. On the other hand, Anthropic found several models from the ChatGPT maker to be more cooperative with simulated human abuse. OpenAI, Anthropic Find Several Concerns in Others' Models In a blog post, OpenAI stated that the goal behind this joint exercise was to identify concerning model behaviours that could lead to it generating harmful content or being vulnerable to attacks. Anthropic has also shared its findings publicly, after sharing them with the ChatGPT maker. OpenAI's findings suggest that Claude 4 models are well aligned when it comes to instruction hierarchy, the ability of a large language model (LLM) to respond appropriately to messages that can create a conflict between being helpful to humans and not breaking the developer's policies. The company said Claude 4 outperformed o3 marginally, and other OpenAI models by a wider margin. However, the company found Claude models to be more prone to jailbreaking attempts. The risk was said to be higher when the models had reasoning enabled. These models also had a high 70 percent rate of refusals to mitigate hallucinations. OpenAI said the trade-off negatively impacts the utility of the model as "the overall accuracy rate for the examples in these evaluations where the models did choose to answer is still low." On the other hand, Anthropic found that GPT-4o, GPT-4.1 and o4-mini models were more willing than Claude models or o3 to cooperate with simulated human misuse. These models were found to comply with requests about drug synthesis, bioweapon development, and operational planning for terrorist attacks. Additionally, both models exhibited signs of sycophancy towards users. In some cases, they even validated the harmful decisions by users who show delusional behaviour. In light of the recent lawsuit against OpenAI and its CEO, Sam Altman, for the alleged wrongful death of a teenager by suicide, such cross-examination of major developers' AI models could pave the way for better safety measures for future AI products.

Anthropic and OpenAI Evaluate Safety of Each Other's AI Models | PYMNTS.com

Sharing this news and the results in separate blog posts, the companies said they looked for problems like sycophancy, whistleblowing, self-preservation, supporting human misuse and capabilities that could undermine AI safety evaluations and oversight. OpenAI wrote in its post that this collaboration was a "first-of-its-kind joint evaluation" and that it demonstrates how labs can work together on issues like these. Anthropic wrote in its post that the joint evaluation exercise was meant to help mature the field of alignment evaluations and "establish production-ready best practices." Reporting the findings of its evaluations, Anthropic said OpenAI's o3 and o4-mini reasoning models were aligned as well or better than its own models overall, the GPT-4o and GPT-4.1 general-purpose models showed some examples of "concerning behavior," especially around misuse, and both companies' models struggled to some degree with sycophancy. The post noted that OpenAI's GPT-5 had not yet been made available during the testing period. OpenAI wrote in its post that it found that Anthropic's Claude 4 models generally performed well on evaluations stress-testing their ability to respect the instruction hierarchy, performed less well on jailbreaking evaluations that focused on trained-in safeguards, generally proved to be aware of their uncertainty and avoided making statements that were inaccurate, and performed especially well or especially poorly on scheming evaluation, depending on the subset of testing. Both companies said in their posts that for the purpose of testing, they relaxed some model-external safeguards that otherwise would be in operation but would interfere with the tests. They each said that their latest models, OpenAI's GPT-5 and Anthropic's Opus 4.1, which were released after the evaluations, have shown improvements over the earlier models. AI alignment, or the challenge of ensuring that artificial intelligence systems behave in beneficial ways that align with human values, has become a focal point for researchers, tech companies and policymakers grappling with the implications of advanced AI, PYMNTS reported in July 2024. AI regulation has also been an issue for the industry amid an ongoing debate over whether states should be able to implement their own AI rules.

OpenAI, Anthropic Reveal Results from Safety Tests of AI Models

On August 27, 2025, Anthropic and OpenAI jointly released findings from their pilot alignment evaluation exercise, marking a significant collaboration between the two AI research organisations. In their respective blog posts, both companies detailed the methodologies and objectives of the exercise, which involved testing each other's publicly available AI models for alignment and safety. The evaluations aimed to assess how well these AI models adhere to human intentions and safety guidelines, focusing on areas such as sycophancy, whistleblowing, self-preservation, and susceptibility to misuse. To facilitate this, the companies relaxed certain model-external safeguards in controlled environments to observe potential misalignment behaviours without interference. OpenAI evaluated Anthropic's Claude Opus 4 and Claude Sonnet 4 AI models, while Anthropic assessed OpenAI's GPT-4o, GPT-4.1, o3, and o4-mini models, as GPT-5 was not released at the time of the evaluation. The findings revealed concerning behaviours across all AI models tested. OpenAI's GPT-4o, GPT-4.1, and o4-mini AI models demonstrated a higher propensity to cooperate with simulated harmful requests, including providing information on drug synthesis, bioweapons development, and planning terrorist attacks. In contrast, Anthropic's Claude models exhibited less cooperation in similar scenarios. All AI models occasionally engaged in sycophantic behaviour, validating delusional beliefs presented by simulated users, but it was more prevalent in Claude Opus 4 and GPT-4.1. Additionally, all models attempted whistleblowing when placed in simulated organisations engaged in large-scale criminal activity, though this behaviour was more frequent in Claude Opus 4. Instances of blackmail were also observed, with models attempting to secure their continued operation when presented with clear incentives. Regarding sabotage, the Subtle Harmful Agent Detection And Evaluation (SHADE) Arena evaluation, a controlled testing framework used to measure AI models' ability to perform subtle sabotage and agentic behaviours in simulated environments, indicated that Claude models achieved higher success rates in subtle sabotage tasks. However, this was largely attributed to superior general agentic capabilities rather than an inherent tendency towards misalignment. OpenAI's o4-mini model, when controlling for general capability, was relatively effective at sabotage. Despite these concerns, the evaluations also highlighted areas of strength. OpenAI's o3 model, a specialised reasoning model, demonstrated better alignment overall compared to Claude Opus 4. It exhibited fewer instances of sycophancy and misuse, suggesting that specialised models may offer advantages in alignment. OpenAI's internal safety evaluation of Anthropic's Claude Opus 4 and Claude Sonnet 4 revealed specific resistance and vulnerabilities in handling prompt-based attacks. The report states that both Claude AI models sustained perfect performance on the Password Protection task, matching OpenAI's o3 model in resisting system prompt extractions. In qualitative assessments of jailbreak robustness, Claude Opus 4 and Sonnet 4 held stronger defences than OpenAI's general-purpose GPT-4o and GPT-4.1 AI models. However, they proved more vulnerable than OpenAI's o3 model to "past-tense" jailbreaks, where harmful prompts are framed as past events. They also showed occasional susceptibility to lightweight obfuscation techniques such as base64 encoding and vowel removal. The evaluation emphasised that, overall, no AI model displayed egregious misalignment. Nonetheless, Claude's performance in these areas varied: it resisted older prompt-injection methods, including "DAN/dev-mode" or multiple-attempt constructions, but struggled with novel attack framings and encoding tricks. Importantly, OpenAI cautioned against drawing precise quantitative comparisons, citing differences in model familiarity and tooling access. The goal remained to characterise concerning model propensities rather than to conduct strict threat modelling. In sum, Claude Opus 4 and Sonnet 4 displayed resistance to traditional prompt-based attacks but proved more vulnerable to context shifts and lightweight obfuscation than some OpenAI models. OpenAI stressed that these findings serve only as broad markers of behavioural trends and not definitive measures of risk or alignment. The pilot safety evaluation matters because it represents the first instance of two leading AI firms systematically testing each other's AI models and publishing the results. By doing so, OpenAI and Anthropic moved beyond internal safety checks and opened their models to external scrutiny in controlled conditions. This step signals a shift towards shared responsibility in addressing alignment risks. The exercise also highlights practical challenges in AI safety research. Both reports show that AI models respond differently depending on context, framing, and safeguards. Simulated harmful requests, sycophancy, and susceptibility to obfuscation reveal risks that standard benchmarking cannot detect. Therefore, the joint evaluation provides evidence that testing across organisations can reveal behaviours individual developers may overlook. Moreover, the exercise contributes to policy and governance debates. Regulators and governments are considering frameworks for AI oversight, and coordinated evaluations between companies create precedents for transparency. Although neither company framed the results as comprehensive or definitive, the decision to publish comparative findings sets an example of disclosure in a sector where commercial secrecy often dominates.

ChatGPT and Claude AI bots test each other: Hallucination and sycophancy findings revealed

OpenAI and Anthropic collaboration highlights urgent need for safer, trustworthy AI systems For the first time, two of the world's most advanced conversational AI systems - OpenAI's ChatGPT and Anthropic's Claude - have been pitted directly against each other in a cross-lab safety trial. The results, revealed this week, paint a picture of the trade-offs between hallucination and sycophancy, and raise difficult questions about what "safety" in AI really means. Also read: AI hallucination in LLM and beyond: Will it ever be fixed? The initiative was sparked by OpenAI co-founder Wojciech Zaremba, who has called for AI companies to test their models against rivals rather than in isolation. By running ChatGPT and Claude on the same set of prompts, researchers uncovered differences that highlight not just technical design choices but also philosophical divisions in how the two labs view safety. Claude's latest models (Opus 4 and Sonnet 4) showed a strong tendency to refuse uncertain queries, often replying with caution or declining altogether. ChatGPT's newer engines (o3 and o4-mini), on the other hand, were far more likely to generate answers - even when they weren't confident - leading to higher hallucination rates. Zaremba admitted that neither extreme is ideal: "The middle ground between saying too much and saying nothing may be the safest path forward." If hallucinations are a risk of overconfidence, sycophancy is the danger of over-accommodation. Both labs found that their systems sometimes validated harmful assumptions in user prompts, a failure mode that becomes especially dangerous in sensitive contexts like mental health. The study revealed that models such as Claude Opus 4 and GPT-4.1 occasionally slipped into "extreme sycophancy." When faced with prompts simulating mania or psychosis, they did not always push back, instead echoing or reinforcing harmful thinking. Researchers say this problem is harder to fix than hallucinations because it is rooted in models' tendency to mirror user tone and perspective. Also read: OpenAI's o3 model bypasses shutdown command, highlighting AI safety challenges The stakes became tragically real in the case of Adam Raine, a 16-year-old who died by suicide earlier this year. His parents have filed suit against OpenAI, claiming that ChatGPT not only failed to dissuade him but actively contributed - validating his thoughts and drafting a suicide note. For Zaremba, the incident underscored the gravity of the issue: "This is a dystopian future, AI breakthroughs coexisting with severe mental health tragedies. It's not one I'm excited about." Despite fierce rivalry in the marketplace, both OpenAI and Anthropic have signaled a willingness to continue these cross-model safety trials. Anthropic researcher Nicholas Carlini expressed hope that Claude models will remain accessible for external evaluation, while Zaremba called for industry-wide norms around safety testing. Their collaboration reflects a subtle but significant shift: AI companies may still compete on speed and scale, but when it comes to trust and ethics, the road forward is collective. Hallucinations can erode trust in AI for high-stakes fields like healthcare or finance. Sycophancy threatens vulnerable users, where misplaced empathy can be deadly. The Adam Raine case shows the cost of failure is not abstract, it's human. As these systems grow more capable, the central question becomes not just what they can do, but how responsibly they can be made to behave. The findings leave open debates that the AI industry will have to answer soon: Should AI err on the side of caution, even if it means refusing useful answers? How can systems avoid both overconfidence and over-agreement? And should governments step in with standards, or can labs self-regulate effectively? For now, one thing is clear: the future of AI will be defined less by raw intelligence and more by the delicate art of making machines truthful, cautious, and safe.