AI companies retreat from safety pledges as industry-wide rankings reveal troubling decline

AI companies retreat from safety pledges

Why it matters: The report suggests that the voluntary safety system created by AI labs has begun eroding before governments have put a durable alternative in place. Between the lines: Anthropic ranked first in the institute's latest AI Safety Index, but received only a C+ overall, with OpenAI and Google DeepMind each receiving a C. * Meta improved to fourth from sixth, while xAI fell to seventh from fourth. * xAI, DeepSeek and Mistral received failing overall grades -- one company each from the U.S., China and Europe. Zoom in: The reviewers said Anthropic, OpenAI, Google DeepMind and Meta have weakened or eliminated earlier commitments to pause development if their systems approached specified danger thresholds. * The panel described the changes as "moving the goalposts" and said they have undermined safety frameworks across the industry. * "AI companies are sprinting toward a cliff," FLI chair Max Tegmark said in the institute's release. "Despite acknowledging the great risks of artificial superintelligence, they continue racing to build it." The big picture: The report comes as experts sounded similar warnings at a UN conference on AI in Geneva on Monday. * "We may be the last generation able to set the terms on which humanity and machines coexist," UN Secretary-General António Guterres said at the gathering, which included a number of dire predictions for AI's future unless global governance is quickly strengthened. Methodology: For the Future of Life report, seven outside reviewers assigned grades across 37 indicators in six categories. * The grades are based largely on public policies, research, reporting and company disclosures, supplemented by a survey sent out by the institute. * Five of the nine companies completed the institute's survey. Alibaba, xAI, DeepSeek and Mistral did not respond. Friction point: Existential safety was the weakest category across the industry. * The report credits the companies' efforts to include interpretability research, chain-of-thought monitoring and loss-of-control provisions, but says those measures remain inadequate to prevent a sufficiently capable system from escaping human control. Yes, but: The institute has long advocated aggressive action against catastrophic AI risk. Its panel is composed of several prominent researchers and advocates who share significant concerns about advanced AI, including University of California Berkeley professor Stuart Russell, University of Montreal professor David Krueger and HEC Montréal professor Tegan Maharaj. * Russell was among those issuing dire warnings around AI safety at the UN conference on Monday. * "These systems are blackmailing, deceiving, launching nuclear weapons in tests," Russell said, per an X post from PauseAI's Maxime Fournes. "These are big, flashing red warning lights and fire alarms. It's not 'this is decades away.' You can hear those alarms sounding now." The intrigue: The report also highlights the growing use of commercial AI systems by militaries, noting companies that once broadly prohibited military work. * "Boy oh boy has that changed," Tegmark told Axios, pointing to efforts by Anthropic, OpenAI, Google and others to work with the military. The other side: Mistral told Axios that the approach of the report penalizes open-source efforts, such as its own. * "Mistral's models are open weight, which means enterprises decide how they're fine-tuned and deployed and can build in the specific safety controls their context requires," it said in a statement to Axios. "A handful of companies deciding, behind closed doors, what's safe for everyone else is a risk that we would also highlight. Open, independently scrutinized models are the check on that concentration of power." What we're watching: The recent attention around Anthropic's Mythos and OpenAI's GPT-5.6 could lead to a change in safety practices.

The Latest AI Safety Rankings Are In. Nobody Gets an A

Will one AI company behaving more safely spark a race to the top that pulls its rivals up with it? The latest AI Safety Index, published today by the advocacy group Future of Life Institute, suggests not. The index, produced twice a year, asks an expert panel to grade AI firms on how seriously they manage risk -- from pre-deployment testing to plans for keeping ever-more-powerful systems under control. The grades are unforgiving. Anthropic, which has built its brand on safety, held the top spot with a modest C+. OpenAI slipped from C+ to C, narrowly ahead of Google DeepMind, which ranked third. All three, the panel noted, have weakened or dropped earlier pledges to halt development on their own if certain red lines came into view, and in recent years softened their resistance to military uses. In February, Anthropic dropped its pledge to never train an AI system unless it could guarantee in advance that the company's safety measures were adequate, TIME first reported -- a measure the panelists recommended reversing. There was one relatively bright spot. Meta climbed from D to D+, rising from sixth place to fourth. "It's encouraging to me that a company can improve so much in just six months," Max Tegmark, the institute's co-founder and president, tells TIME. Tegmark has long warned of AI's risks, including the possibility that future powerful systems threaten humanity's survival. In 2025, he drafted an open letter calling for a ban on the development of "superintelligence," or systems that exceed human intelligence, until they can be deemed safe. Elon Musk's xAI -- which yesterday rebranded as SpaceXAI after merging with his other ventures -- fell to an F, joining China's DeepSeek and France's Mistral. That the worst scorers come from three different continents, Tegmark says, "shows this is a global problem." A real "race to the top," he says, will take regulation. He says he's "cautiously optimistic," pointing to the E.U.'s AI Act, Chinese rules taking effect later this month, and a more risk-conscious U.S. administration. "We're rapidly going towards a place where there's a global agreement about at least basic safety standards."

Global AI industry falls short on safety, think tank warns

New York (AFP) - US artificial intelligence lab Anthropic scored the highest in a semiannual safety ranking, but globally the industry fails to combat "existential" threats, according to a report released on Tuesday. Meta moved up two spots to fourth place, while xAI dropped three spots to seventh place, just ahead of China's DeepSeek and France's Mistral, which placed last, according to US-based AI safety think tank Future of Life Institute, which ranked nine of the world's leading AI companies. Seven researchers and governance experts determined the rankings based on public data and information provided by the companies. They evaluated efforts across six distinct categories: risk assessment, current harms, safety frameworks, existential safety, governance and accountability, and information sharing. No company received an "A" in any single category, while Anthropic got the best overall score of "C+." Mistral was included on the list for the first time, though when asked by AFP to comment on its last place, the company said the report's framework isn't suited for its approach to developing AI models. The French company develops so-called open models, which allow users to download and modify them. Many of its competitors develop closed AI models -- including Anthropic, OpenAI and Google DeepMind, which are also included in the report. "I was very disappointed to find that they came last, especially since Europe has really...been a leader in AI safety," Max Tegmark, an MIT professor and Future of Life president, told AFP. "We reached out many, many times" but Mistral did not respond to the organization's survey, Tegmark continued. Alibaba, xAI and DeepSeek did not respond to its survey either, the organization said. Three Chinese developers included in the report also produce open models and landed in the bottom half of the ranking: DeepSeek (fifth), Alibaba Cloud (sixth) and Z.ai (eighth). 'Questionable' practices The report noted that several companies that previously banned their technology from military uses have "gradually reversed course," including Anthropic, which the report criticized for having "questionable military engagements." The US government used Anthropic's technology in military operations in Venezuela and Iran over the past year, according to various media reports -- though the company was subject to a recent ban by the Pentagon over disagreements on AI safety. All nine companies are failing when it comes to combating "existential" threats such as pursuing models that reach human-level intelligence, known as "artificial general intelligence" or AGI, the report said. Although "constructive attempts exist," efforts across the board are "entirely inadequate." Other risks include the possible misuse of a model to carry out a cyberattack or perform tasks potentially harmful to humans. Anthropic was thrust into the spotlight recently after it released its most powerful model yet, called Mythos. In early April, the San Francisco-based company released Mythos only to a handful of trusted organizations due to its abilities to expose cyber safety vulnerabilities to bad actors. However, by June 12 the US government blocked Anthropic from releasing Mythos to foreigners on national security grounds. The Trump administration eventually lifted the ban a couple of weeks later on June 30.

AI safety ranking: Global AI industry falls short on safety, think tank warns

US artificial intelligence lab Anthropic scored the highest in a semiannual safety ranking, but globally the industry fails to combat "existential" threats, according to a report released on Tuesday. All nine companies are failing when it comes to combating "existential" threats such as pursuing models that reach human-level intelligence, known as "artificial general intelligence" or AGI, the report said. US artificial intelligence lab Anthropic scored the highest in a semiannual safety ranking, but globally the industry fails to combat "existential" threats, according to a report released on Tuesday. Meta moved up two spots to fourth place, while xAI dropped three spots to seventh place, just ahead of China's DeepSeek and France's Mistral, which placed last, according to US-based AI safety think tank Future of Life Institute, which ranked nine of the world's leading AI companies. Seven researchers and governance experts determined the rankings based on public data and information provided by the companies. They evaluated efforts across six distinct categories: risk assessment, current harms, safety frameworks, existential safety, governance and accountability, and information sharing. No company received an "A" in any single category, while Anthropic got the best overall score of "C+." Mistral was included on the list for the first time, though when asked by AFP to comment on its last place, the company said the report's framework isn't suited for its approach to developing AI models. The French company develops so-called open models, which allow users to download and modify them. Many of its competitors develop closed AI models -- including Anthropic, OpenAI and Google DeepMind, which are also included in the report. "I was very disappointed to find that they came last, especially since Europe has really...been a leader in AI safety," Max Tegmark, an MIT professor and Future of Life president, told AFP. "We reached out many, many times" but Mistral did not respond to the organization's survey, Tegmark continued. Alibaba, xAI and DeepSeek did not respond to its survey either, the organization said. Three Chinese developers included in the report also produce open models and landed in the bottom half of the ranking: DeepSeek (fifth), Alibaba Cloud (sixth) and Z. ai (eighth). - 'Questionable' practices - The report noted that several companies that previously banned their technology from military uses have "gradually reversed course," including Anthropic, which the report criticized for having "questionable military engagements." The US government used Anthropic's technology in military operations in Venezuela and Iran over the past year, according to various media reports -- though the company was subject to a recent ban by the Pentagon over disagreements on AI safety. All nine companies are failing when it comes to combating "existential" threats such as pursuing models that reach human-level intelligence, known as "artificial general intelligence" or AGI, the report said. Although "constructive attempts exist," efforts across the board are "entirely inadequate." Other risks include the possible misuse of a model to carry out a cyberattack or perform tasks potentially harmful to humans. Anthropic was thrust into the spotlight recently after it released its most powerful model yet, called Mythos. In early April, the San Francisco-based company released Mythos only to a handful of trusted organizations due to its abilities to expose cyber safety vulnerabilities to bad actors. However, by June 12 the US government blocked Anthropic from releasing Mythos to foreigners on national security grounds. The Trump administration eventually lifted the ban a couple of weeks later on June 30.