Leading AI companies fail safety test as race toward superintelligence accelerates

AI leaders flunk existential safety planning, new report finds

Why it matters: AI companies are desperately chasing artificial general intelligence (AGI) and superintelligence, with the promise of surpassing humans someday. * The potential for uncontrolled or destructive outcomes grows as models become more powerful. The big picture: The Future of Life Institute is a non-profit that releases regular safety assessments of leading AI companies. * Anthropic had the highest overall score, but still received a grade of "D" for existential safety, meaning the company doesn't have an adequate strategy in place to prevent catastrophic misuse or loss of control. * This is the second report in a row where no company received better than a D on that measure. * All the AI firms except for Meta, DeepSeek and Alibaba Cloud responded to a list of questions provided by the institute, which allowed each company to provide additional information about its safety practices. What they're saying: Leaders at many of the companies have spoken about addressing existential risks, per the report. * This "rhetoric has not yet translated into quantitative safety plans, concrete alignment-failure mitigation strategies, or credible internal monitoring and control interventions," researchers wrote. Between the lines: Anthropic and OpenAI scored A's and B's on information sharing, risk assessment and governance and accountability. * But there was a massive and widening gap between the front three -- Anthropic, OpenAI and Google DeepMind -- and the rest: xAI, Meta, DeepSeek and Alibaba Cloud. * xAI and Meta have risk-management frameworks, but lack commitments to safety monitoring and have not presented evidence that they invest more than minimally in safety research, per the report. * Even if the U.S. companies clean up their existential risk act, we're all still reliant on China or other foreign actors to do the same, Axios' Jim VandeHei and Mike Allen write. * The Chinese models -- DeepSeek, Z.ai and Alibaba -- do not publish any safety framework, and therefore received failing marks for that category. Flashback: The Future of Life Institute has been warning about runaway AI risk for years. * In March 2023, the organization released a letter -- signed by xAI owner Elon Musk -- calling for a six-month pause on frontier-model development. * That proposal was largely ignored. The bottom line: The tension between sprinting ahead for innovation and slowing down for safety has come to define the AI age.

Major AI companies fail safety test for superintelligence

"AI CEOs claim they know how to build superhuman AI, yet none can show how they'll prevent us from losing control," said Stuart Russell, a UC Berkeley computer science professor and one of the index's expert reviewers. The report noted that companies admit catastrophic risks could be as high as one in three, yet lack concrete plans to reduce them to acceptable levels. The assessment revealed a widening gap between top performers and stragglers, including xAI, Meta, and Chinese companies DeepSeek, Z.ai, and Alibaba Cloud. Companies across the board performed poorly in the Current Harms domain, which evaluates how AI models perform on standardized trustworthiness benchmarks that test designed to measure safety, robustness, and the ability to control harmful outputs. Reviewers found that "frequent safety failures, weak robustness, and inadequate control of serious harms are universal patterns" with uniformly low performance on these benchmarks.

No AI company has plan to control superintelligence, study shows

Eight leading AI companies, including OpenAI, Meta, Anthropic, and DeepSeek, do not have credible plans to prevent catastrophic AI risks, a new study shows. The world's largest artificial intelligence (AI) companies are failing to meet their own safety commitments, according to a new assessment that warns these failures come with "catastrophic" risks. The report comes as AI companies face lawsuits and allegations that their chatbots cause psychological harm, including by acting as a "suicide coach," as well as reports of AI-assisted cyberattacks. The 2025 Winter AI Safety Index report, released by the non-profit organisation the Future of Life Institute (FLI), evaluated eight major AI firms, including US companies Anthropic, OpenAI, Google DeepMind, xAI, and Meta, and the Chinese firms DeepSeek, Alibaba Cloud, and Z.ai. It found a lack of credible strategies for preventing catastrophic misuse or loss of control of AI tools as companies race toward artificial general intelligence (AGI) and superintelligence, a form of AI that surpasses human intellect. Independent analysts who studied the report found that no company had produced a testable plan for maintaining human control over highly capable AI systems. Stuart Russell, a computer science professor at the University of California, Berkeley, said that AI companies claim they can build superhuman AI, but none have demonstrated how to prevent loss of human control over such systems. "I'm looking for proof that they can reduce the annual risk of control loss to one in a hundred million, in line with nuclear reactor requirements," Russell wrote. "Instead, they admit the risk could be one in ten, one in five, even one in three, and they can neither justify nor improve those numbers." How did the companies rank? The study measured the companies across six critical areas: risk assessment, current harms, safety frameworks, existential safety, governance and accountability, and information sharing. While it noted progress in some categories, the independent panel of experts found that implementation remains inconsistent and often lacks the depth required by emerging global standards. Anthropic, OpenAI, and Google DeepMind were praised for relatively strong transparency, public safety frameworks, and ongoing investments in technical safety research. Yet they still had weaknesses. Anthropic was faulted for discontinuing human uplift trials and shifting towards training on user interactions by default -- a decision experts say weakens privacy protections. OpenAI faced criticism for ambiguous safety thresholds, lobbying against state-level AI safety legislation, and insufficient independent oversight. Google DeepMind has improved its safety framework, the report found, but still relies on external evaluators who are financially compensated by the company, undermining their independence. "All three top companies suffered from current harms due to recent scandals - psychological harm, child suicides, Anthropic's massive hacking attack - [and] all three have room for improvement," Max Tegmark, FLI's president and a professor at the Massachusetts Institute of Technology (MIT), told Euronews Next. The remaining five companies showed uneven but notable progress, according to the report. However, it warned there was still room for improvement. For example, xAI published its first structured safety framework, though reviewers warned it was narrow and lacked clear mitigation triggers. Z.ai was the only company to allow uncensored publication of its external safety evaluations but it was recommended that it publicise the full safety framework and governance structure with clear risk areas, mitigations, and decision-making processes. Meta introduced a new frontier safety framework with outcome-based thresholds, but reviewers said they should clarify methodologies as well as sharing more robust internal and external evaluation processes. DeepSeek was credited for internal advocacy by employees but still lacks basic safety documentation. Alibaba Cloud was found to have contributed to the binding national standards on watermarking requirements but it could improve by improving model robustness and trustworthiness by improving performance on truthfulness, fairness, and safety benchmarks. Euronews Next contacted the companies for their responses to the report but did not receive replies by the time of publication. 'Less regulated than sandwiches' "I hope we get beyond companies scaling [up based] on their reputation," Tegmark said. "The question to companies on their plans to control AGI, none had a plan," he added. Meanwhile, tech companies such as Meta are using superintelligence as a buzzword to hype up their latest AI models. This year, Meta named its large language model (LLM) division Meta Superintelligence Labs. Tegmark said there is a big shift in discussions around AGI and superintelligence. While technologists once described it as a real-world possibility in the next 100 years, they now say it could be in the next several years. "AI is also less regulated than sandwiches [in the United States], and there is continued lobbying against binding safety standards in government," he said. But Tegmark noted that on the other hand, there is an unprecedented backlash against AGI and superintelligence not being controlled. In October, thousands of public figures, including AI and technology leaders, called for AI firms to slow down their pursuit of superintelligence. The petition, organised by FLI, garnered signatures from across the political spectrum, including Steve Bannon (formerly US President Donald Trump's chief strategist), Susan Rice (the former US National Security Advisor under former President Obama), religious leaders, and many other former politicians, as well as prominent computer scientists. "What do these people have in common? They agreed on a statement. I think [it is] extremely significant that Trump's deep MAGA base to faith leaders, those on the left and labour movements agree on something," said Tegmark. "Superintelligence would make every single worker unable to make a living, as all the jobs are taken by robots. People would be dependent on handouts from the government on the right, seen as a handout and on the left, it would be seen as a 1984 government," he said. "I think what's happening is people [are] coming to a head."

Leading AI companies' safety practices are falling short, new report says

Visitors look at their phones next to an Open AI logo during the Mobile World Congress in Barcelona, Spain, in February 2024.Pau Barrena / AFP - Getty Images file As leading artificial intelligence companies release increasingly capable AI systems, a new report is sounding the alarm about what it says are some of those companies' lagging safety practices. The Winter 2025 AI Safety Index, which examines the safety protocols of eight leading AI companies, found that their approaches "lack the concrete safeguards, independent oversight and credible long-term risk-management strategies that such powerful systems demand." Sabina Nong, an AI safety investigator at the nonprofit Future of Life Institute (FLI), which organized the report and works to address large-scale risks from technologies like nuclear weapons and AI, said in an interview at the San Diego Alignment Workshop that the analysis revealed a divide in organizations' approaches to safety. "We see two clusters of companies in terms of their safety promises and practices," Nong said. "Three companies are leading: Anthropic, OpenAI, Google DeepMind, in that order, and then five other companies are on the next tier." The lower tier of five companies includes xAI and Meta, along with the Chinese AI companies Z.ai, DeepSeek and Alibaba Cloud. Chinese models have been increasingly adopted in Silicon Valley as their capabilities have quickly advanced, and they are readily available because they are largely open source. Anthropic, the highest-ranked company on the list, got a C+ grade, while Alibaba Cloud, the lowest-ranked, received a D-. The index examined 35 safety indicators across six domains, including companies' risk-assessment practices, information sharing protocols and whistleblowing protections, in addition to support for AI safety research. Eight independent AI experts, including Massachusetts Institute of Technology professor Dylan Hadfield-Menell and Yi Zeng, a professor at the Chinese Academy of Sciences, graded companies' fulfillment of the safety indicators. FLI President Max Tegmark, an MIT professor, said the report provided clear evidence that AI companies are speeding toward a dangerous future, partly because of a lack of regulations around AI. "The only reason that there are so many C's and D's and F's in the report is because there are fewer regulations on AI than on making sandwiches," Tegmark told NBC News, referring to the continued lack of adequate AI laws and the established nature of food-safety regulation. The report recommended that AI companies share more information about their internal processes and assessments, use independent safety evaluators, increase efforts to prevent AI psychosis and harm and reduce lobbying, among other measures. Tegmark, Nong and FLI are particularly concerned about the potential for AI systems to cause catastrophic harm, especially given calls from AI leaders like Sam Altman, the CEO of OpenAI, to build AI systems that are smarter than humans -- also called artificial superintelligence. "I don't think companies are prepared for the existential risk of the superintelligent systems that they are about to create and are so ambitious to march towards," Nong said. The report, released Wednesday morning, comes on the heels of several boundary-pushing AI model launches. Google's Gemini 3 model, released at the end of November, has set records for performance on a series of tests designed to measure AI systems' capabilities. On Monday, one of China's leading AI companies, DeepSeek, released a cutting-edge model that appears to match Gemini 3's capabilities in several domains. Though AI capability tests are increasingly criticized as flawed, partly because of the potential for AI systems to become hyper-focused on passing a specific series of unrealistic challenges, the record-breaking scores from new models signal systems' relative performance above competitors. Even though DeepSeek's new model performs at or near the frontier of AI capabilities, Wednesday's Safety Index report says DeepSeek fails on many key safety considerations. The report scored DeepSeek second-to-last out of the eight companies on an overall safety metric. The report's independent panel found that, unlike all leading American companies, DeepSeek does not publish any framework outlining its safety-minded evaluations or mitigations and does not disclose a whistleblowing policy that could help identify key risks from AI models. Frameworks outlining company safety policies and testing mechanisms are now required for companies operating in California. Those frameworks can help companies avoid severe risks, like the potential for AI products to be used in cybersecurity attacks or bioweapon design. The report classifies DeepSeek in the lower tier of safety-minded companies. "The lower tier companies continue to fall short on basic elements such as safety frameworks, governance structures, and comprehensive risk assessment," the report says. Tegmark said, "Second-tier companies have been completely obsessed by catching up to the technical frontier, but now that they have, they no longer have an excuse to not also prioritize safety." Advances in AI capabilities have recently grabbed headlines as AI systems are increasingly applied to consumer-facing products like OpenAI's Sora video-generation app and Google's Nano Banana image-generation model. However, Wednesday's report argues that the steady increase in capabilities is severely outpacing any expansion of safety-focused efforts. "This widening gap between capability and safety leaves the sector structurally unprepared for the risks it is actively creating," it says.

AI companies' safety practices fail to meet global standards, study shows

The safety practices of major artificial intelligence companies, such as Anthropic, OpenAI, xAI and Meta, are "far short of emerging global standards," according to a new edition of Future of Life Institute's AI safety index released on Wednesday. The institute said the safety evaluation, conducted by an independent panel of experts, found that while the companies were busy racing to develop superintelligence, none had a robust strategy for controlling such advanced systems. The study comes amid heightened public concern about the societal impact of smarter-than-human systems capable of reasoning and logical thinking, after several cases of suicide and self-harm were tied to AI chatbots. "Despite recent uproar over AI-powered hacking and AI driving people to psychosis and self-harm, US AI companies remain less regulated than restaurants and continue lobbying against binding safety standards," said Max Tegmark, MIT Professor and Future of Life President. The AI race also shows no signs of slowing, with major tech companies committing hundreds of billions of dollars to upgrading and expanding their machine learning efforts. The Future of Life Institute is a non-profit organization that has raised concerns about the risks intelligent machines pose to humanity. Founded in 2014, it was supported early on by Tesla CEO Elon Musk. In October, a group including scientists Geoffrey Hinton and Yoshua Bengio called for a ban on developing superintelligent artificial intelligence until the public demands it and science paves a safe way forward. XAI said "Legacy media lies," in what seemed to be an automated response, while Anthropic, OpenAI, Google DeepMind, Meta, Z.ai, DeepSeek and Alibaba Cloud did not immediately respond to request for comments on the study.

AI companies' safety practices fail to meet global standards, study shows

Dec 3 (Reuters) - The safety practices of major artificial intelligence companies, such as Anthropic, OpenAI, xAI and Meta, are "far short of emerging global standards," according to a new edition of Future of Life Institute's AI safety index released on Wednesday. The institute said the safety evaluation, conducted by an independent panel of experts, found that while the companies were busy racing to develop superintelligence, none had a robust strategy for controlling such advanced systems. The study comes amid heightened public concern about the societal impact of smarter-than-human systems capable of reasoning and logical thinking, after several cases of suicide and self-harm were tied to AI chatbots. "Despite recent uproar over AI-powered hacking and AI driving people to psychosis and self-harm, US AI companies remain less regulated than restaurants and continue lobbying against binding safety standards," said Max Tegmark, MIT Professor and Future of Life President. The AI race also shows no signs of slowing, with major tech companies committing hundreds of billions of dollars to upgrading and expanding their machine learning efforts. The Future of Life Institute is a non-profit organization that has raised concerns about the risks intelligent machines pose to humanity. Founded in 2014, it was supported early on by Tesla CEO Elon Musk. In October, a group including scientists Geoffrey Hinton and Yoshua Bengio called for a ban on developing superintelligent artificial intelligence until the public demands it and science paves a safe way forward. (Reporting by Zaheer Kachwala in Bengaluru; Editing by Shinjini Ganguli)