OpenAI and Anthropic Collaborate on Groundbreaking AI Safety Testing

OpenAI co-founder calls for AI labs to safety test rival models | TechCrunch

OpenAI and Anthropic, two of the world's leading AI labs, briefly opened up their closely guarded AI models to allow for joint safety testing -- a rare cross-lab collaboration at a time of fierce competition. The effort aimed to surface blind spots in each company's internal evaluations, and demonstrate how leading AI companies can work together on safety and alignment work in the future. In an interview with TechCrunch, OpenAI co-founder Wojciech Zaremba said this kind of collaboration is increasingly important now that AI is entering a "consequential" stage of development, where AI models are used by millions of people everyday. "There's a broader question of how the industry sets a standard for safety and collaboration, despite the billions of dollars invested, as well as the war for talent, users, and the best products," said Zaremba. The joint safety research, published Wednesday by both companies, arrives amid an arms race among leading AI labs like OpenAI and Anthropic, where billion-dollar data center bets and $100 million compensation packages for top researchers have become table stakes. Some experts warn that the intensity of product competition could pressure companies to cut corners on safety in the rush to build more powerful systems. To make this research possible, OpenAI and Anthropic granted each other special API access to versions of their AI models with fewer safeguards (OpenAI notes that GPT-5 was not tested because it hadn't been released yet). Shortly after the research was conducted, however, Anthropic revoked another team at OpenAI's API access. At the time, Anthropic claimed that OpenAI violated its terms of service, which prohibits using Claude to improve competing products. Zaremba says the events were unrelated, and that he expects competition to stay fierce even as AI safety teams try to work together. Nicholas Carlini, a safety researcher with Anthropic, tells TechCrunch that he would like to continue allowing OpenAI safety researchers to access Claude models in the future. "We want to increase collaboration wherever it's possible across the safety frontier, and try to make this something that happens more regularly," said Carlini. One of the most stark findings in the study relates to hallucination testing. Anthropic's Claude Opus 4 and Sonnet 4 models refused to answer up to 70% of questions when they were unsure of the correct answer, instead offering responses like, "I don't have reliable information." Meanwhile, OpenAI's o3 and o4-mini models refuse to answer questions far less, but showed much higher hallucination rates, attempting to answer questions when they didn't have enough information. Zaremba says the right balance is likely somewhere in the middle -- OpenAI's models should refuse to answer more questions, while Anthropic's models should probably attempt to offer more answers. Sycophancy, the tendency for AI models to reinforce negative behavior in users to please them, has emerged as one of the most pressing safety concerns around AI models. While this topic wasn't directly studied in the joint research, it's an area both OpenAI and Anthropic are investing considerable resources into studying. On Tuesday, parents of a 16-year-old boy, Adam Raine, filed a lawsuit against OpenAI, claiming that ChatGPT offered their son advice that aided in his suicide, rather than pushing back on his suicidal thoughts. The lawsuit suggests this may be the latest example of AI chatbot sycophancy contributing to tragic outcomes. "It's hard to imagine how difficult this is to their family," said Zaremba when asked about the incident. "It would be a sad story if we build AI that solves all these complex PhD level problems, invents new science, and at the same time, we have people with mental health problems as a consequence of interacting with it. This is a dystopian future that I'm not excited about." In a blog post, OpenAI says that it significantly improved the sycophancy of its AI chatbots with GPT-5, compared to GPT-4o, significantly improving the model's ability to respond to mental health emergencies. Moving forward, Zaremba and Carlini say they would like Anthropic and OpenAI to collaborate more on safety testing, looking into more subjects and testing future models, and they hope other AI labs will follow their collaborative approach.

Anthropic and OpenAI Evaluate Safety of Each Other's AI Models | PYMNTS.com

Sharing this news and the results in separate blog posts, the companies said they looked for problems like sycophancy, whistleblowing, self-preservation, supporting human misuse and capabilities that could undermine AI safety evaluations and oversight. OpenAI wrote in its post that this collaboration was a "first-of-its-kind joint evaluation" and that it demonstrates how labs can work together on issues like these. Anthropic wrote in its post that the joint evaluation exercise was meant to help mature the field of alignment evaluations and "establish production-ready best practices." Reporting the findings of its evaluations, Anthropic said OpenAI's o3 and o4-mini reasoning models were aligned as well or better than its own models overall, the GPT-4o and GPT-4.1 general-purpose models showed some examples of "concerning behavior," especially around misuse, and both companies' models struggled to some degree with sycophancy. The post noted that OpenAI's GPT-5 had not yet been made available during the testing period. OpenAI wrote in its post that it found that Anthropic's Claude 4 models generally performed well on evaluations stress-testing their ability to respect the instruction hierarchy, performed less well on jailbreaking evaluations that focused on trained-in safeguards, generally proved to be aware of their uncertainty and avoided making statements that were inaccurate, and performed especially well or especially poorly on scheming evaluation, depending on the subset of testing. Both companies said in their posts that for the purpose of testing, they relaxed some model-external safeguards that otherwise would be in operation but would interfere with the tests. They each said that their latest models, OpenAI's GPT-5 and Anthropic's Opus 4.1, which were released after the evaluations, have shown improvements over the earlier models. AI alignment, or the challenge of ensuring that artificial intelligence systems behave in beneficial ways that align with human values, has become a focal point for researchers, tech companies and policymakers grappling with the implications of advanced AI, PYMNTS reported in July 2024. AI regulation has also been an issue for the industry amid an ongoing debate over whether states should be able to implement their own AI rules.