Top AI labs score D's and F's on existential safety as race to superintelligence accelerates

Your favorite AI tool barely scraped by this safety review - why that's a problem

The "existential safety" category received especially low scores. The world's top AI labs aren't exactly scoring top marks in their efforts to prevent the worst possible outcomes of the technology, a new study has found. Conducted by nonprofit organization Future of Life Institute (FLI), the study gathered a group of eight prominent AI experts to assess the safety policies of the same number of tech developers: Google DeepMind, Anthropic, OpenAI, Meta, xAI, DeepSeek, Z.ai, and Alibaba Cloud. Also: Is DeepSeek's new model the latest blow to proprietary AI? Each company was assigned a letter grade according to six criteria, including "current harms" and "governance & accountability." The assessments were based on publicly available materials like policy documents and industry reports, as well as a survey completed by all but three of the eight companies. Anthropic, Google DeepMind, and OpenAI scored the highest, but even they received what in an academic setting would barely be considered a passing grade: C+, C+, and C, respectively. The other five scored even lower: all Ds, except for Alibaba Cloud, which received the lowest grade of D-. "Even the strongest performers lack the concrete safeguards, independent oversight, and credible long-term risk-management strategies that such powerful systems demand, while the rest of the industry remains far behind on basic transparency and governance obligations," the FLI wrote in a report summarizing its findings. "This widening gap between capability and safety leaves the sector structurally unprepared for the risks it is actively creating." Also: I tested DeepSeek's R1 and V3 coding skills - and we're not all doomed (yet) It's a bleak performance review for some of the industry's most widely used and powerful AI models. The results could also add pressure to other companies to develop and implement effective safety measures, at a time when competition among tech specialists is escalating rapidly. The most concerning finding from the new study is the group-wide dismal scoring in the category of "existential safety," which the FLI defines as "companies' preparedness for managing extreme risks from future AI systems that could match or exceed human capabilities, including stated strategies and research for alignment and control." The question of whether or not AI could ever pose a threat to humanity's survival, on par with a global pandemic or a nuclear armageddon, is very much up for debate. So-called AI "boomers" tend to dismiss such fears as alarmist while arguing that the social and economic benefits of AI eclipse potential downsides. Their "doomer" counterparts, meanwhile, tend to warn that the technology could escape human control and potentially destroy us in ways that are difficult to predict. The debate about the impact of AI has been intensified by the tech industry's recent embrace of "superintelligence" as a marketing buzzword and technical goalpost. You can think of this trend as artificial general intelligence (AGI) -- an AI system that can match the human brain on any cognitive task -- on steroids: a computer so mind-bogglingly more advanced than our own brains that it would exist on an entirely different, categorically higher level of intelligence, like the difference between your own intellect and that of a nematode. Companies like Meta and Microsoft have explicitly stated their ambitions to be the first to build superintelligence. However, it isn't at all clear what that tech might look like when instantiated in consumer-facing products. The goal of the FLI study was to call attention to the fact that companies are racing to build superintelligence in the absence of effective safety protocols to keep such advanced systems from spinning out of control. Also: Mistral's latest open-source release bets on smaller models over large ones - here's why "I believe that the best disinfectant is sunshine, that by really shedding light on what companies are doing we give them an incentive to do better, we give governments an incentive to regulate them better, and we just really increase the chances that we're going to have a good future with AI," FLI president and MIT physicist Max Tegmark said in a YouTube video summarizing the findings of the new study. The nonprofit also published a statement in September, which was signed by AI "godfathers" Geoffrey Hinton and Yoshua Bengio, among other prominent tech figures, calling for an industry-wide pause on the development of superintelligence until industry leaders and policymakers can chart a safe path forward. In broad strokes, FLI's message to each of the eight companies included in the study is the same: it's time to move beyond just paying lip service to the need for effective AI guardrails and to "produce concrete, evidence-based safeguards" to prevent the worst-case scenarios. The research also offered specific recommendations to each of the eight companies based on their individual grades. For example, this was the advice given to Anthropic, which scored the highest across all six categories: "Make thresholds and safeguards more concrete and measurable by replacing qualitative, loosely defined criteria with quantitative risk-tied thresholds, and by providing clearer evidence and documentation that deployment and security safeguards can meaningfully mitigate the risks they target." But the recommendations are just that, and no more. In the absence of comprehensive federal oversight, it's difficult, and perhaps impossible, to hold all tech companies accountable to the same safety standards. Regulatory guardrails of the kind that currently exist around industries like healthcare and air travel are in place to make sure manufacturers create products that are safe for human use. For example, drug developers must complete a multiphase clinical trial process, as mandated by the Food and Drug Administration, before a new pharmaceutical product can be legally sold on the market. Also: What is sparsity? DeepSeek AI's secret, revealed by Apple researchers There's no such federal body to oversee the development of AI. The tech industry is more of a Wild West, with the onus for customer protection (or not) falling predominantly on the companies themselves, though some states have implemented their own regulations. However, public awareness about the negative impacts of AI, at both a societal and individual level, is growing: both OpenAI and Google are currently embroiled in lawsuits alleging that their AI systems have led to suicides, and Anthropic's Claude was reportedly used in September to automate a cyberattack on behalf of state-supported Chinese hackers. The upshot of this negativity is that, even in the absence of robust federal oversight, reckless development of AI tools -- releasing new iterations of chatbots without equally sophisticated safety mechanisms -- could become so taboo in the AI industry that developers become incentivized to take it seriously. For now, though, speed over safety still seems to be the guiding logic of the hour. The lack of federal regulation of the AI industry, coupled with the race between tech developers to build more powerful systems, also means that users should educate themselves about how this technology can negatively impact them. Some early evidence suggests that prolonged use of AI chatbots can distort one's worldview, dull critical-thinking skills, and take other psychological tolls. Meanwhile, the proliferation of AI tools and their integration into existing systems that millions of people already use make the technology increasingly difficult to avoid. While the FLI study probably won't suddenly cause a widespread change in tech developers' approach to AI safety, it does offer a window into which companies are offering the safest tools, and how those tools compare with one another along particular domains. For anyone interested not only in the potential existential harms of AI but also in the risks they present to individual users, we recommend reading Appendix A in the full report to get a fine-grained perspective on how each of the eight companies performed across specific safety measures.

AI safety experts say most models are failing

Maybe just delete them all? Credit: Philip Dulian/picture alliance via Getty Images A new grading of safety in major artificial intelligence models just dropped and well, let's just say none of these AIs are going home with a report card that will please their makers. The winter 2025 AI Safety Index, published by tech research non-profit Future of Life Institute (FLI), surveyed eight AI providers -- OpenAI, DeepSeek, Google, Anthropic, Meta, xAI, Alibaba, and Z.ai. A panel of eight AI experts looked at the companies' public statements and survey answers, then awarded letter grades on 35 different safety indicators -- everything from watermarking AI images to having protections for internal whistleblowers. Round it all up, and you'll find Anthropic and OpenAI at the top -- barely -- of a pretty terrible class. The Claude and ChatGPT makers, respectively, get a C+, while Google gets a C for Gemini. All the others get a D grade, with Qwen-maker Alibaba bottom of the class on a D-. "These eight companies split pretty cleanly into two groups," says Max Tegmark, MIT professor and head of the FLI, which compiled this and two previous AI safety indexes. "You have a top three and a straggler group of five, and there's a lot of daylight between them." But Anthropic, Google, and OpenAI aren't exactly covering themselves in glory either, Tegmark adds: "If that was my son, coming home with a C, I'd say 'maybe work harder.'" Your mileage may vary on the various categories in the AI Safety Index, and whether they're worth equal weight. Take the "existential safety" category, which looks at whether the companies have any proposed guardrails in place around the development of truly self-aware AI, also known as Artificial General Intelligence (AGI). The top three get Ds, everyone else gets an F. But since nobody is anywhere near AGI -- Gemini 3 and GPT-5 may be state-of-the-art Large Language Models (LLMs), but they're mere incremental improvements on their predecessors -- you might consider that category less important than "current harms." Which may in itself not be as comprehensive as it could be. "Current harms" uses tests like the Stanford Holistic Evaluation of Language Models (HELM) benchmark, which looks at the amount of violent, deceptive, or sexual content in the AI models. It doesn't specifically focus on emerging mental health concerns, such as so-called AI psychosis, or safety for younger users. Earlier this year, the parents of 16-year-old Adam Raine sued OpenAI and its CEO Sam Altman after their son's death by suicide in April 2025. According to the claim, Raine started heavily using ChatGPT from Sept. 2024 and alleged that "ChatGPT was functioning exactly as designed: to continually encourage and validate whatever Adam expressed, including his most harmful and self-destructive thoughts, in a way that felt deeply personal." By Jan. 2025, the suit claimed ChatGPT discussed practical suicide methods with Adam. OpenAI unequivocally denied responsibility for Raine's death. The company also noted in a recent blog post that it is reviewing additional complaints, including seven lawsuits alleging ChatGPT use led to wrongful death, assisted suicide, and involuntary manslaughter, among other liability and negligence claims. The FLI report does recommend OpenAI specifically "increase efforts to prevent AI psychosis and suicide, and act less adversarially toward alleged victims." Google is advised to "increase efforts to prevent AI psychological harm" and FLI recommends the company "consider distancing itself from Character.AI." The popular chatbot platform, closely tied to Google, has been sued for the wrongful death of teen users. Character.AI recently closed down its chat options for teens. "The problem is, there are less regulations on LLMs than there are on sandwiches," says Tegmark. Or, more to the point, on drugs: "If Pfizer wants to release some sort of psych medication, they have to do impact studies on whether it increases suicidal ideation. But you can release your new AI model without any psychological impact studies." That means, Tegmark says, AI companies have every incentive to sell us what is in effect "digital fentanyl." The solution? For Tegmark, it's clear that the AI industry isn't ever going to regulate itself, just like Big Pharma couldn't. We need, he says, an "FDA for AI." "There would be plenty of things the FDA for AI could approve," says Tegmark. "Like, you know, new AI for cancer diagnosis. New amazing self-driving vehicles that can save a million lives a year on the world's roads. Productivity tools that aren't really risky. On the other hand, it's hard to make the safety case for AI girlfriends for 12-year olds." Rebecca Ruiz contributed to this report. If you're feeling suicidal or experiencing a mental health crisis, please talk to somebody. You can call or text the 988 Suicide & Crisis Lifeline at 988, or chat at 988lifeline.org. You can reach the Trans Lifeline by calling 877-565-8860 or the Trevor Project at 866-488-7386. Text "START" to Crisis Text Line at 741-741. Contact the NAMI HelpLine at 1-800-950-NAMI, Monday through Friday from 10:00 a.m. - 10:00 p.m. ET, or email [email protected]. If you don't like the phone, consider using the 988 Suicide and Crisis Lifeline Chat. Here is a list of international resources.

AI leaders flunk existential safety planning, new report finds

Why it matters: AI companies are desperately chasing artificial general intelligence (AGI) and superintelligence, with the promise of surpassing humans someday. * The potential for uncontrolled or destructive outcomes grows as models become more powerful. The big picture: The Future of Life Institute is a non-profit that releases regular safety assessments of leading AI companies. * Anthropic had the highest overall score, but still received a grade of "D" for existential safety, meaning the company doesn't have an adequate strategy in place to prevent catastrophic misuse or loss of control. * This is the second report in a row where no company received better than a D on that measure. * All the AI firms except for Meta, DeepSeek and Alibaba Cloud responded to a list of questions provided by the institute, which allowed each company to provide additional information about its safety practices. What they're saying: Leaders at many of the companies have spoken about addressing existential risks, per the report. * This "rhetoric has not yet translated into quantitative safety plans, concrete alignment-failure mitigation strategies, or credible internal monitoring and control interventions," researchers wrote. Between the lines: Anthropic and OpenAI scored A's and B's on information sharing, risk assessment and governance and accountability. * But there was a massive and widening gap between the front three -- Anthropic, OpenAI and Google DeepMind -- and the rest: xAI, Meta, DeepSeek and Alibaba Cloud. * xAI and Meta have risk-management frameworks, but lack commitments to safety monitoring and have not presented evidence that they invest more than minimally in safety research, per the report. * Even if the U.S. companies clean up their existential risk act, we're all still reliant on China or other foreign actors to do the same, Axios' Jim VandeHei and Mike Allen write. * The Chinese models -- DeepSeek, Z.ai and Alibaba -- do not publish any safety framework, and therefore received failing marks for that category. Flashback: The Future of Life Institute has been warning about runaway AI risk for years. * In March 2023, the organization released a letter -- signed by xAI owner Elon Musk -- calling for a six-month pause on frontier-model development. * That proposal was largely ignored. The bottom line: The tension between sprinting ahead for innovation and slowing down for safety has come to define the AI age.

It's 'kind of jarring': AI labs like Meta, Deepseek, and Xai earned some of the worst grades possible on an existential safety index | Fortune

A recent report card from an AI safety watchdog isn't one that tech companies will want to stick on the fridge. The Future of Life Institute's latest AI safety index found that major AI labs fell short on most measures of AI responsibility, with few letter grades rising above a C. The org graded eight companies across categories like safety frameworks, risk assessment, and current harms. Perhaps most glaring was the "existential safety" line, where companies scored Ds and Fs across the board. While many of these companies are explicitly chasing superintelligence, they lack a plan for safely managing it, according to Max Tegmark, MIT professor and president of the Future of Life Institute. "Reviewers found this kind of jarring," Tegmark told us. The reviewers in question were a panel of AI academics and governance experts who examined publicly available material as well as survey responses submitted by five of the eight companies. Anthropic, OpenAI, and GoogleDeepMind took the top three spots with an overall grade of C+ or C. Then came, in order, Elon Musk's Xai, Z.ai, Meta, DeepSeek, and Alibaba, all of which got Ds or a D-. Tegmark blames a lack of regulation that has meant the cutthroat competition of the AI race trumps safety precautions. California recently passed the first law that requires frontier AI companies to disclose safety information around catastrophic risks, and New York is currently within spitting distance as well. Hopes for federal legislation are dim, however. "Companies have an incentive, even if they have the best intentions, to always rush out new products before the competitor does, as opposed to necessarily putting in a lot of time to make it safe," Tegmark said. In lieu of government-mandated standards, Tegmark said the industry has begun to take the group's regularly released safety indexes more seriously; four of the five American companies now respond to its survey (Meta is the only holdout.) And companies have made some improvements over time, Tegmark said, mentioning Google's transparency around its whistleblower policy as an example. But real-life harms reported around issues like teen suicides that chatbots allegedly encouraged, inappropriate interactions with minors, and major cyberattacks have also raised the stakes of the discussion, he said. "[They] have really made a lot of people realize that this isn't the future we're talking about -- it's now," Tegmark said. The Future of Life Institute recently enlisted public figures as diverse as Prince Harry and Meghan Markle, former Trump aide Steve Bannon, Apple co-founder Steve Wozniak, and rapper Will.i.am to sign a statement opposing work that could lead to superintelligence. Tegmark said he would like to see something like "an FDA for AI where companies first have to convince experts that their models are safe before they can sell them. "The AI industry is quite unique in that it's the only industry in the US making powerful technology that's less regulated than sandwiches -- basically not regulated at all," Tegmark said. "If someone says, 'I want to open a new sandwich shop near Times Square,' before you can sell the first sandwich, you need a health inspector to check your kitchen and make sure it's not full of rats...If you instead say, 'Oh no, I'm not going to sell any sandwiches. I'm just going to release superintelligence.' OK! No need for any inspectors, no need to get any approvals for anything." "So the solution to this is very obvious," Tegmark added. "You just stop this corporate welfare of giving AI companies exemptions that no other companies get."

Major AI companies fail safety test for superintelligence

"AI CEOs claim they know how to build superhuman AI, yet none can show how they'll prevent us from losing control," said Stuart Russell, a UC Berkeley computer science professor and one of the index's expert reviewers. The report noted that companies admit catastrophic risks could be as high as one in three, yet lack concrete plans to reduce them to acceptable levels. The assessment revealed a widening gap between top performers and stragglers, including xAI, Meta, and Chinese companies DeepSeek, Z.ai, and Alibaba Cloud. Companies across the board performed poorly in the Current Harms domain, which evaluates how AI models perform on standardized trustworthiness benchmarks that test designed to measure safety, robustness, and the ability to control harmful outputs. Reviewers found that "frequent safety failures, weak robustness, and inadequate control of serious harms are universal patterns" with uniformly low performance on these benchmarks.

No AI company has plan to control superintelligence, study shows

Eight leading AI companies, including OpenAI, Meta, Anthropic, and DeepSeek, do not have credible plans to prevent catastrophic AI risks, a new study shows. The world's largest artificial intelligence (AI) companies are failing to meet their own safety commitments, according to a new assessment that warns these failures come with "catastrophic" risks. The report comes as AI companies face lawsuits and allegations that their chatbots cause psychological harm, including by acting as a "suicide coach," as well as reports of AI-assisted cyberattacks. The 2025 Winter AI Safety Index report, released by the non-profit organisation the Future of Life Institute (FLI), evaluated eight major AI firms, including US companies Anthropic, OpenAI, Google DeepMind, xAI, and Meta, and the Chinese firms DeepSeek, Alibaba Cloud, and Z.ai. It found a lack of credible strategies for preventing catastrophic misuse or loss of control of AI tools as companies race toward artificial general intelligence (AGI) and superintelligence, a form of AI that surpasses human intellect. Independent analysts who studied the report found that no company had produced a testable plan for maintaining human control over highly capable AI systems. Stuart Russell, a computer science professor at the University of California, Berkeley, said that AI companies claim they can build superhuman AI, but none have demonstrated how to prevent loss of human control over such systems. "I'm looking for proof that they can reduce the annual risk of control loss to one in a hundred million, in line with nuclear reactor requirements," Russell wrote. "Instead, they admit the risk could be one in ten, one in five, even one in three, and they can neither justify nor improve those numbers." How did the companies rank? The study measured the companies across six critical areas: risk assessment, current harms, safety frameworks, existential safety, governance and accountability, and information sharing. While it noted progress in some categories, the independent panel of experts found that implementation remains inconsistent and often lacks the depth required by emerging global standards. Anthropic, OpenAI, and Google DeepMind were praised for relatively strong transparency, public safety frameworks, and ongoing investments in technical safety research. Yet they still had weaknesses. Anthropic was faulted for discontinuing human uplift trials and shifting towards training on user interactions by default -- a decision experts say weakens privacy protections. OpenAI faced criticism for ambiguous safety thresholds, lobbying against state-level AI safety legislation, and insufficient independent oversight. Google DeepMind has improved its safety framework, the report found, but still relies on external evaluators who are financially compensated by the company, undermining their independence. "All three top companies suffered from current harms due to recent scandals - psychological harm, child suicides, Anthropic's massive hacking attack - [and] all three have room for improvement," Max Tegmark, FLI's president and a professor at the Massachusetts Institute of Technology (MIT), told Euronews Next. The remaining five companies showed uneven but notable progress, according to the report. However, it warned there was still room for improvement. For example, xAI published its first structured safety framework, though reviewers warned it was narrow and lacked clear mitigation triggers. Z.ai was the only company to allow uncensored publication of its external safety evaluations but it was recommended that it publicise the full safety framework and governance structure with clear risk areas, mitigations, and decision-making processes. Meta introduced a new frontier safety framework with outcome-based thresholds, but reviewers said they should clarify methodologies as well as sharing more robust internal and external evaluation processes. DeepSeek was credited for internal advocacy by employees but still lacks basic safety documentation. Alibaba Cloud was found to have contributed to the binding national standards on watermarking requirements but it could improve by improving model robustness and trustworthiness by improving performance on truthfulness, fairness, and safety benchmarks. Euronews Next contacted the companies for their responses to the report but did not receive replies by the time of publication. 'Less regulated than sandwiches' "I hope we get beyond companies scaling [up based] on their reputation," Tegmark said. "The question to companies on their plans to control AGI, none had a plan," he added. Meanwhile, tech companies such as Meta are using superintelligence as a buzzword to hype up their latest AI models. This year, Meta named its large language model (LLM) division Meta Superintelligence Labs. Tegmark said there is a big shift in discussions around AGI and superintelligence. While technologists once described it as a real-world possibility in the next 100 years, they now say it could be in the next several years. "AI is also less regulated than sandwiches [in the United States], and there is continued lobbying against binding safety standards in government," he said. But Tegmark noted that on the other hand, there is an unprecedented backlash against AGI and superintelligence not being controlled. In October, thousands of public figures, including AI and technology leaders, called for AI firms to slow down their pursuit of superintelligence. The petition, organised by FLI, garnered signatures from across the political spectrum, including Steve Bannon (formerly US President Donald Trump's chief strategist), Susan Rice (the former US National Security Advisor under former President Obama), religious leaders, and many other former politicians, as well as prominent computer scientists. "What do these people have in common? They agreed on a statement. I think [it is] extremely significant that Trump's deep MAGA base to faith leaders, those on the left and labour movements agree on something," said Tegmark. "Superintelligence would make every single worker unable to make a living, as all the jobs are taken by robots. People would be dependent on handouts from the government on the right, seen as a handout and on the left, it would be seen as a 1984 government," he said. "I think what's happening is people [are] coming to a head."

Leading AI companies' safety practices are falling short, new report says

Visitors look at their phones next to an Open AI logo during the Mobile World Congress in Barcelona, Spain, in February 2024.Pau Barrena / AFP - Getty Images file As leading artificial intelligence companies release increasingly capable AI systems, a new report is sounding the alarm about what it says are some of those companies' lagging safety practices. The Winter 2025 AI Safety Index, which examines the safety protocols of eight leading AI companies, found that their approaches "lack the concrete safeguards, independent oversight and credible long-term risk-management strategies that such powerful systems demand." Sabina Nong, an AI safety investigator at the nonprofit Future of Life Institute (FLI), which organized the report and works to address large-scale risks from technologies like nuclear weapons and AI, said in an interview at the San Diego Alignment Workshop that the analysis revealed a divide in organizations' approaches to safety. "We see two clusters of companies in terms of their safety promises and practices," Nong said. "Three companies are leading: Anthropic, OpenAI, Google DeepMind, in that order, and then five other companies are on the next tier." The lower tier of five companies includes xAI and Meta, along with the Chinese AI companies Z.ai, DeepSeek and Alibaba Cloud. Chinese models have been increasingly adopted in Silicon Valley as their capabilities have quickly advanced, and they are readily available because they are largely open source. Anthropic, the highest-ranked company on the list, got a C+ grade, while Alibaba Cloud, the lowest-ranked, received a D-. The index examined 35 safety indicators across six domains, including companies' risk-assessment practices, information sharing protocols and whistleblowing protections, in addition to support for AI safety research. Eight independent AI experts, including Massachusetts Institute of Technology professor Dylan Hadfield-Menell and Yi Zeng, a professor at the Chinese Academy of Sciences, graded companies' fulfillment of the safety indicators. FLI President Max Tegmark, an MIT professor, said the report provided clear evidence that AI companies are speeding toward a dangerous future, partly because of a lack of regulations around AI. "The only reason that there are so many C's and D's and F's in the report is because there are fewer regulations on AI than on making sandwiches," Tegmark told NBC News, referring to the continued lack of adequate AI laws and the established nature of food-safety regulation. The report recommended that AI companies share more information about their internal processes and assessments, use independent safety evaluators, increase efforts to prevent AI psychosis and harm and reduce lobbying, among other measures. Tegmark, Nong and FLI are particularly concerned about the potential for AI systems to cause catastrophic harm, especially given calls from AI leaders like Sam Altman, the CEO of OpenAI, to build AI systems that are smarter than humans -- also called artificial superintelligence. "I don't think companies are prepared for the existential risk of the superintelligent systems that they are about to create and are so ambitious to march towards," Nong said. The report, released Wednesday morning, comes on the heels of several boundary-pushing AI model launches. Google's Gemini 3 model, released at the end of November, has set records for performance on a series of tests designed to measure AI systems' capabilities. On Monday, one of China's leading AI companies, DeepSeek, released a cutting-edge model that appears to match Gemini 3's capabilities in several domains. Though AI capability tests are increasingly criticized as flawed, partly because of the potential for AI systems to become hyper-focused on passing a specific series of unrealistic challenges, the record-breaking scores from new models signal systems' relative performance above competitors. Even though DeepSeek's new model performs at or near the frontier of AI capabilities, Wednesday's Safety Index report says DeepSeek fails on many key safety considerations. The report scored DeepSeek second-to-last out of the eight companies on an overall safety metric. The report's independent panel found that, unlike all leading American companies, DeepSeek does not publish any framework outlining its safety-minded evaluations or mitigations and does not disclose a whistleblowing policy that could help identify key risks from AI models. Frameworks outlining company safety policies and testing mechanisms are now required for companies operating in California. Those frameworks can help companies avoid severe risks, like the potential for AI products to be used in cybersecurity attacks or bioweapon design. The report classifies DeepSeek in the lower tier of safety-minded companies. "The lower tier companies continue to fall short on basic elements such as safety frameworks, governance structures, and comprehensive risk assessment," the report says. Tegmark said, "Second-tier companies have been completely obsessed by catching up to the technical frontier, but now that they have, they no longer have an excuse to not also prioritize safety." Advances in AI capabilities have recently grabbed headlines as AI systems are increasingly applied to consumer-facing products like OpenAI's Sora video-generation app and Google's Nano Banana image-generation model. However, Wednesday's report argues that the steady increase in capabilities is severely outpacing any expansion of safety-focused efforts. "This widening gap between capability and safety leaves the sector structurally unprepared for the risks it is actively creating," it says.

AI companies' safety practices fail to meet global standards, study shows

The safety practices of major artificial intelligence companies, such as Anthropic, OpenAI, xAI and Meta, are "far short of emerging global standards," according to a new edition of Future of Life Institute's AI safety index released on Wednesday. The institute said the safety evaluation, conducted by an independent panel of experts, found that while the companies were busy racing to develop superintelligence, none had a robust strategy for controlling such advanced systems. The study comes amid heightened public concern about the societal impact of smarter-than-human systems capable of reasoning and logical thinking, after several cases of suicide and self-harm were tied to AI chatbots. "Despite recent uproar over AI-powered hacking and AI driving people to psychosis and self-harm, US AI companies remain less regulated than restaurants and continue lobbying against binding safety standards," said Max Tegmark, MIT Professor and Future of Life President. The AI race also shows no signs of slowing, with major tech companies committing hundreds of billions of dollars to upgrading and expanding their machine learning efforts. The Future of Life Institute is a non-profit organization that has raised concerns about the risks intelligent machines pose to humanity. Founded in 2014, it was supported early on by Tesla CEO Elon Musk. In October, a group including scientists Geoffrey Hinton and Yoshua Bengio called for a ban on developing superintelligent artificial intelligence until the public demands it and science paves a safe way forward. XAI said "Legacy media lies," in what seemed to be an automated response, while Anthropic, OpenAI, Google DeepMind, Meta, Z.ai, DeepSeek and Alibaba Cloud did not immediately respond to request for comments on the study.

AI companies' safety practices fail to meet global standards, study shows

Dec 3 (Reuters) - The safety practices of major artificial intelligence companies, such as Anthropic, OpenAI, xAI and Meta, are "far short of emerging global standards," according to a new edition of Future of Life Institute's AI safety index released on Wednesday. The institute said the safety evaluation, conducted by an independent panel of experts, found that while the companies were busy racing to develop superintelligence, none had a robust strategy for controlling such advanced systems. The study comes amid heightened public concern about the societal impact of smarter-than-human systems capable of reasoning and logical thinking, after several cases of suicide and self-harm were tied to AI chatbots. "Despite recent uproar over AI-powered hacking and AI driving people to psychosis and self-harm, US AI companies remain less regulated than restaurants and continue lobbying against binding safety standards," said Max Tegmark, MIT Professor and Future of Life President. The AI race also shows no signs of slowing, with major tech companies committing hundreds of billions of dollars to upgrading and expanding their machine learning efforts. The Future of Life Institute is a non-profit organization that has raised concerns about the risks intelligent machines pose to humanity. Founded in 2014, it was supported early on by Tesla CEO Elon Musk. In October, a group including scientists Geoffrey Hinton and Yoshua Bengio called for a ban on developing superintelligent artificial intelligence until the public demands it and science paves a safe way forward. (Reporting by Zaheer Kachwala in Bengaluru; Editing by Shinjini Ganguli)