Meta Unveils 'Frontier AI Framework' to Address Risks in Advanced AI Development

Meta Has Mentioned In A New Policy Document Titled 'Frontier AI Framework' That It Will Not Release A Highly Capable AI Model If It Carries Too Much Risk

As companies race towards developing advanced AI models, which have a more common name called artificial general intelligence (AGI), there is always an associated risk that comes with introducing something that can accomplish any task that a human being is capable of finishing. Meta likely realizes the threat of such an uncontrolled development roadmap can lead to, which is why it has drafted a new 'Frontier AI Framework,' which is a policy document highlighting the company's continued efforts into making the best AI system possible, while monitoring its deleterious effects. There are various scenarios in which Meta would not be compelled to release a capable AI model, with the company providing some conditions in the new policy document. Frontier AI Framework has identified two types of systems that are deemed too risky and are categorized under 'high risk' and 'critical risk.' These AI models are capable of aiding cybersecurity measures and chemical and biological attacks. These kinds of situations can result in a 'catastrophic outcome.' Threat modelling is fundamental to our outcomes-led approach. We run threat modelling exercises both internally and with external experts with relevant domain expertise, where required. The goal of these exercises is to explore, in a systematic way, how frontier AI models might be used to produce catastrophic outcomes. Through this process, we develop threat scenarios' which describe how different actors might use a frontier AI model to realise a catastrophic outcome. We design assessments to simulate whether our model would uniquely enable these scenarios, and identify the enabling capabilities the model would need to exhibit to do so. Our first set of evaluations are designed to identify whether all of these enabling capabilities are present, and if the model is sufficiently performant on them. If so, this would prompt further evaluation to understand whether the model could uniquely enable the threat scenario. Meta states that if it has identified a system that displays a critical risk, work will immediately be halted, and it will not be released. Unfortunately, there are still minute chances that the AI system is released, and while the company will exercise measures to ensure that an event of cataclysmic proportions does not transpire, Meta admits that these measures might be insufficient. Readers checking out the Frontier AI Framework will probably be nervous about where AGI is headed. Even if companies like Meta do not have internal measures in place to limit the release of potentially dangerous AI models, it is likely that the law will intervene in full force. Now, all that remains to be seen is how far this development can go.

Meta says it may stop development of AI systems it deems too risky | TechCrunch

Meta CEO Mark Zuckerberg has pledged to make artificial general intelligence (AGI) -- which is roughly defined as AI that can accomplish any task a human can -- openly available one day. But in a new policy document, Meta suggests that there are certain scenarios in which it may not release a highly capable AI system it developed internally. The document, which Meta is calling its Frontier AI Framework, identifies two types of AI systems the company considers too risky to release: "high risk" and "critical risk" systems. As Meta defines them, both "high-risk" and "critical-risk" systems are capable of aiding in cybersecurity, chemical, and biological attacks, the difference being that "critical-risk" systems could result in a "catastrophic outcome [that] cannot be mitigated in [a] proposed deployment context." High-risk systems, by contrast, might make an attack easier to carry out but not as reliably or dependably as a critical risk system. Which sort of attacks are we talking about here? Meta gives a few examples, like the "automated end-to-end compromise of a best-practice-protected corporate-scale environment" and the "proliferation of high-impact biological weapons." The list of possible catastrophes in Meta's document is far from exhaustive, the company acknowledges, but includes those that Meta believes to be "the most urgent" and plausible to arise as a direct result of releasing a powerful AI system. Somewhat surprising is that, according to the document, Meta classifies system risk not based on any one empirical test but informed by the input of internal and external researchers who are subject to review by "senior-level decision-makers." Why? Meta says that it doesn't believe the science of evaluation is "sufficiently robust as to provide definitive quantitative metrics" for deciding a system's riskiness. If Meta determines a system is high-risk, the company says it will limit access to the system internally and won't release it until it implements mitigations to "reduce risk to moderate levels." If, on the other hand, a system is deemed critical-risk, Meta says it will implement unspecified security protections to prevent the system from being exfiltrated and stop development until the system can be made less dangerous. Meta's Frontier AI Framework, which the company says will evolve with the changing AI landscape, appears to be a response to criticism of the company's "open" approach to system development. Meta has embraced a strategy of making its AI technology openly available -- albeit not open source by the commonly understood definition -- in contrast to companies like OpenAI that opt to gate their systems behind an API. For Meta, the open release approach has proven to be a blessing and a curse. The company's family of AI models, called Llama, has racked up hundreds of millions downloads. But Llama has also reportedly been used by at least one U.S. adversary to develop a defense chatbot. In publishing its Frontier AI Framework, Meta may also be aiming to contrast its open AI strategy with Chinese AI firm DeepSeek's. DeepSeek also makes its systems openly available. But the company's AI has few safeguards and can be easily steered to generate toxic and harmful outputs. "[W]e believe that by considering both benefits and risks in making decisions about how to develop and deploy advanced AI," Meta writes in the document, "it is possible to deliver that technology to society in a way that preserves the benefits of that technology to society while also maintaining an appropriate level of risk."

Meta reveals what kinds of AI even it would think too risky to release

Risk assessments and modeling will categorize AI models as critical, high, or moderate Meta has revealed some concerns about the future of AI despite CEO Mark Zuckerberg's well-publicized intentions to make artificial general intelligence (AGI) openly available to all. The company's newly-released Frontier AI Framework explores some "critical" risks that AI could pose, including its potential implications on cybersecurity and chemical and biological weapons. By making its guidelines publicly available, Meta hopes to collaborate with other industry leaders to "anticipate and mitigate" such risks by identifying potential "catastrophic" outcomes and threat modeling to establish thresholds. Stating, "open sourcing AI is not optional; it is essential," Meta outlined in a blog post how sharing research helps organizations learn from each other's assessments and encourages innovation. Its framework works by proactively running periodic threat modeling exercises to complement its AI risk assessments - modeling will also be used if and when an AI model is identified to potentially "exceed current frontier capabilities," where it becomes a threat. These processes are informed by internal and external experts, resulting in one of three negative categories: 'critical,' where the development of the model must stop; 'high,' where the model in its current state must not be released; and 'moderate,' where further consideration is given to the release strategy. Some threats include the discovery and exploitation of zero-day vulnerabilities, automated scams and frauds and the development of high-impact biological agents. In the framework, Meta writes: "While the focus of this Framework is on our efforts to anticipate and mitigate risks of catastrophic outcomes, it is important to emphasize that the reason to develop advanced AI systems in the first place is because of the tremendous potential for benefits to society from those technologies." The company has committed to updating its framework with the help of academics, policymakers, civil society organizations, governments, and the wider AI community as the technology continues to develop.

Meta acknowledges 'critical risk' AI systems that are too dangerous to develop -- here's what that means

Meta's internal mantra, at least until 2014 (when it was still Facebook), was to "move fast and break things." Fast-forward over a decade and the blistering pace of AI development has seemingly got the company rethinking things a little bit. A new policy document, spotted by TechCrunch, appears to show Meta taking a more cautionary approach. The company has identified a scenarios where "high risk" or "critical risk" AI systems are deemed too dangerous to release to the public in their present state. These kinds of systems would typically include any AI that could help with cybersecurity or biological warfare attacks. In the policy document, Meta specifically references AI that could help to create a "catastrophic outcome [that] cannot be mitigated in [a] proposed deployment context." The company states: "This Frontier AI Framework describes how Meta works to build advanced AI, including by evaluating and mitigating risks and establishing thresholds for catastrophic risks." So, what will Meta -- which has done pioneering work in the open source AI space with Llama -- do if it determines a system poses this kind of threat? In the first case, the company says it will limit access internally and won't release it until it puts mitigations in place to "reduce risk to moderate levels". If things get more serious, straying into the "critical risk" territory, Meta says it will stop development altogether and put security measures in place to stop exfiltration into the wider AI market: "Access is strictly limited to a small number of experts, alongside security protections to prevent hacking or exfiltration insofar as is technically feasible and commercially practicable." Meta's decision to publicise this new framework on AI development is likely a response to the surge of open source AI tools currently empowering the industry. Chinese platform DeepSeek has hit the world of AI like a sledgehammer in the last couple of weeks and has (seemingly) very few safeguards in place. Like DeepSeek, Meta's own Llama 3.2 model can be used by others to build AI tools that benefit from the vast library of data from billions of Facebook and Instagram users it was trained on. Meta says it will also revise and update its framework as necessary as AI continues to evolve. "We expect to update our Framework as our collective understanding of how to measure and mitigate potential catastrophic risk from frontier AI develops, including related to state actors," Meta's document states. "This might involve adding, removing, or updating catastrophic outcomes or threat scenarios, or changing the ways in which we prepare models to be evaluated."

Mark Zuckerberg's Meta vows not to release 'high-risk' AI models: Here's what it means

Meta has come up with a policy document, called the Frontier AI Framework, to share its views on two types of AI systems, "high risk" and "critical risk," that the company considers too risky for release. Meta, the parent company of Facebook, Instagram, WhatsApp, has suggested in a new policy document that it might not release its internally developed highly capable AI system under certain circumstances. This comes even as Meta CEO Mark Zuckerberg has talked about making artificial general intelligence (AGI) openly available to everyone in near future. In its policy document, called the Frontier AI Framework, Meta has discussed two types of AI systems -- "high risk" and "critical risk" -- which it considers too risky to release, TechCrunch reported. According to Meta, "high-risk" and "critical-risk" systems have the capability to aid in cybersecurity, biological and chemical attacks. A major difference between the two is that "critical-risk" systems may result in "catastrophic outcomes" which cannot be mitigated in a "proposed deployment context". Also Read: Final Destination: Bloodlines trailer marks the return of the horror franchise after 14 years On the other hand, high-risk systems may make it easy to carry out an attack, but not as dependable and reliable as the other one. Although the list of possible attacks in the document is a lengthy one, Meta has highlighted some examples like "proliferation of high-impact biological weapons" and "automated end-to-end compromise of a best-practice-protected corporate-scale environment," according to TechCrunch. Meta said that its list includes the risks that are "most urgent," which could arise due to the availability of a powerful AI system. As per the company's document, Meta has classified them on the basis of inputs from internal and external researchers. These are said to be subject to review by "senior-level decision-makers". Once a system is determined as high-risk, Meta stated that it will limit its access internally and will not unveil it to the public till the time mitigations to "reduce risk to moderate levels" are implemented. For critical-risk systems, the company will be implementing unspecified security protections to prevent it from being exfiltrated. Also, its development will be put on halt until it gets to a less dangerous level. Also Read: Wordle #1326 February 4, 2025: Here's hints, clues, answer for today's Wordle puzzle Meta says that it doesn't believe the science of evaluation is "sufficiently robust as to provide definitive quantitative metrics" for deciding a system's riskiness. The Frontier AI Framework of Meta is being seen as its response to the criticism over "open" approach to system development. By considering both risks as well as benefits in making decisions about how to develop and deploy advanced artificial intelligence technology, Meta believes that it is "possible to deliver that technology to society in a way that preserves the benefits of that technology to society while also maintaining an appropriate level of risk." 1. What is Meta's family of AI models? It is called Llama, which has received hundreds of millions of downloads. 2. How is AGI? AGI stands for Artificial General Intelligence, which is referred to hypothetical intelligence of a machine that has the capability to understand and learn intellectual tasks just like human beings.

AI News: Meta Unveils Framework To Restrict High-Risk Artificial Intelligence Systems

Meta has introduced a new policy, the Frontier AI Framework, outlining its approach to restricting the development and release of high-risk artificial intelligence systems. According to the AI news, the framework will address concerns about the dangers of advanced AI technology, particularly in cybersecurity and biosecurity. The company states that some AI models may be too risky to release, requiring internal safeguards before further deployment. In a recent document filing, Meta classified AI systems into two categories based on potential risks. These categories are high-risk and critical-risk, each defined by the extent of possible harm. AI models deemed high-risk may assist in cyber or biological attacks. However, critical-risk AI can cause severe harm, with Meta stating that such systems could lead to catastrophic consequences. According to the AI news, Meta will halt the development of any system classified as critical risk and implement additional security measures to prevent unauthorized access. High-risk AI models will be restricted internally, with further work to reduce risks before release. The framework reflects the company's focus on minimizing potential threats associated with artificial intelligence. These security measures come amid recent concerns over AI data privacy. In the latest AI news, DeepSeek, a Chinese startup, has been removed from Apple's App Store and Google's Play Store in Italy. The country's data protection authority is investigating its data collection practices. To determine AI system risk levels, Meta will rely on assessments from internal and external researchers. However, the company states that no single test can fully measure risk, making expert evaluation a key factor in decision-making. The framework outlines a structured review process, with senior decision-makers overseeing final risk classifications. For high-risk AI, Meta plans to introduce mitigation measures before considering a release. This approach will prevent AI systems from being misused while maintaining their intended functionality. If an artificial intelligence model is classified as critical-risk, development will be suspended entirely until safety measures can ensure controlled deployment. Meta has pursued an open AI development model, allowing broader access to its Llama AI models. This strategy has resulted in widespread adoption, with millions of downloads recorded. However, concerns have emerged regarding potential misuse, including reports that a U.S. adversary utilized Llama to develop a defense chatbot. With the Frontier AI Framework, the company is addressing these concerns while maintaining its commitment to open AI development.

Mark Zuckerberg's Meta Says Some AI Systems Are Too Dangerous To Release And May Halt Development If It Sees 'Critical Risk' Meta's Zuckerberg Commits To Public AGI, Restricts High-Risk AI - Meta Platforms (NASDAQ:META)

Mark Zuckerberg-led Meta Platforms Inc. META on Tuesday suggested that it might discontinue the development of certain artificial intelligence (AI) systems that it perceives as too risky. What Happened: The document, dubbed the Frontier AI Framework, identifies two categories of AI systems that Meta deems too risky to release: "high risk" and "critical risk" systems. These systems could potentially be used in cybersecurity, chemical, and biological attacks. The distinction between the two lies in the severity of the potential outcome, with "critical risk" systems posing a threat that cannot be mitigated in the proposed deployment context. The document also lists examples of possible attacks, such as the "automated end-to-end compromise of a best-practice-protected corporate-scale environment" and the "proliferation of high-impact biological weapons." While the list is not exhaustive, it includes scenarios that the tech giant believes to be the most urgent and plausible. Meta's approach to risk classification relies on insights from internal and external researchers rather than empirical testing. Senior decision-makers review these assessments, as the company acknowledges that current evaluation methods lack the scientific precision needed for definitive quantitative risk metrics. See Also: AI Could Be As Deadly As Nuclear Weapons': How China's Advances Change The Game Why It Matters: The Frontier AI Framework and the commitment to public AGI, while restricting high-risk AI systems, align with Meta's aggressive investment in AI and AR/VR technologies. Meta's investment in virtual and augmented reality is projected to exceed $100 billion in 2025, coinciding with Zuckerberg's prediction of 2025 as a pivotal year for its smart glasses. This Frontier AI Framework also underscores the company's cautious approach towards potential risks associated with AI deployment. In the document, Meta stated, "We believe that by considering both benefits and risks in making decisions about how to develop and deploy advanced AI, it is possible to deliver that technology to society in a way that preserves the benefits of that technology to society while also maintaining an appropriate level of risk." Earlier this year, Zuckerberg acknowledged the impact of DeepSeek's novel approaches on Meta's AI development. Despite the potential disruption from DeepSeek's cost-effective R1 model, both Microsoft and Meta have confirmed their commitment to AI investment. Read more: Mark Zuckerberg Says China's DeepSeek Did Some Novel Things 'We're Still Digesting,' But Meta May Keep Pouring Billions Into AI Infrastructure Anyway Image via Shutterstock Disclaimer: This content was partially produced with the help of AI tools and was reviewed and published by Benzinga editors. METAMeta Platforms Inc$700.800.48%Overview Rating:Speculative50%Technicals Analysis660100Financials Analysis400100WatchlistOverviewMarket News and Data brought to you by Benzinga APIs

Are some AGI systems too risky to release? Meta says so.

It seems like since AI came into our world, creators have put a lead foot down on the gas. However, according to a new policy document, Meta CEO Mark Zuckerberg might slow or stop the development of AGI systems that are deemed "high risk" or "critical risk." AGI is an AI system that can do anything a human can do, and Zuckerberg promised to make it openly available one day. But in the document "Frontier AI Framework," Zuckerberg concedes that some highly capable AI systems won't be released publicly because they could be too risky. The framework "focuses on the most critical risks in the areas of cybersecurity threats and risks from chemical and biological weapons." "By prioritizing these areas, we can work to protect national security while promoting innovation. Our framework outlines a number of processes we follow to anticipate and mitigate risk when developing frontier AI systems," a press release about the document reads. For example, the framework intends to identify "potential catastrophic outcomes related to cyber, chemical and biological risks that we strive to prevent." It also conducts "threat modeling exercises to anticipate how different actors might seek to misuse frontier AI to produce those catastrophic outcomes" and has "processes in place to keep risks within acceptable levels." If the company determines that the risks are too high, it will keep the system internal instead of allowing public access. "While the focus of this Framework is on our efforts to anticipate and mitigate risks of catastrophic outcomes, it is important to emphasize that the reason to develop advanced AI systems in the first place is because of the tremendous potential for benefits to society from those technologies," the document reads. Yet, it looks like Zuckerberg's hitting the brakes -- at least for now -- on AGI's fast track to the future.

Meta plans to block 'catastrophic' AI models - but admits it may not be able to

A Meta policy document describes the company's fears that it could accidentally develop an AI model which would lead to "catastrophic outcomes." It describes its plans to prevent the release of such models, but admits that it may not be able to do so. Among the capabilities the company most fears are an AI system that could break through the security of even the best-protected corporate or government computer network without human assistance ... TechCrunch spotted the policy document bearing the innocuous-sounding title of Frontier AI Framework. The document, which Meta is calling its Frontier AI Framework, identifies two types of AI systems the company considers too risky to release: "high risk" and "critical risk" systems. As Meta defines them, both "high-risk" and "critical-risk" systems are capable of aiding in cybersecurity, chemical, and biological attacks, the difference being that "critical-risk" systems could result in a "catastrophic outcome [that] cannot be mitigated in [a] proposed deployment context." High-risk systems, by contrast, might make an attack easier to carry out but not as reliably or dependably as a critical risk system. The company explains its definition of a "catastrophic" outcome: Catastrophic outcomes are outcomes that would have large scale, devastating, and potentially irreversible harmful impacts on humanity that could plausibly be realized as a direct result of access to [our AI models]. One example given is the "automated end-to-end compromise of a best-practice-protected corporate-scale environment." In other words, an AI that can break into any computer network without needing any help from humans. Others are: The company says that when it identifies a critical risk, it will immediately cease work on the model and seek to ensure that it cannot be released. Meta's document frankly admits that the best it can do in these circumstances is to do its best to ensure that the model is not released, but its measures may not be sufficient (italics are our emphasis):

Meta AI Safety Pledge: How it compares to EU, US AI regulations

Disclaimer: This content generated by AI & may have errors or hallucinations. Edit before use. Read our Terms of use In a new policy document, Meta claimed that it would halt the development of its AI models deemed "critical" or "high" risk and undertake necessary mitigation efforts. The "Frontier AI Framework" report aligns with the Frontier AI Safety Commitments, which Meta and other tech giants signed in 2024. Meta classified risk levels using an outcome-based approach, identifying potential threat scenarios. These catastrophic outcomes spanned the domains of cybersecurity and chemical and biological risks. Following are the risk levels and the corresponding security efforts they warrant: While Meta claims to implement 'security protections to prevent hacking or data exfiltration,' it does not explicitly list these measures. Further, Meta classifies system risks based on inputs by internal and external researchers who are under review by "senior-level decision-makers", given its view that the science of evaluation is not robust enough to determine a system's riskiness, TechCrunch reported. When compared to Meta's Frontier AI Framework, the European Union AI Act 2024 includes a "risk-based approach" depending on broader risks posed to society and fundamental rights. Accordingly, the legislation defines four levels of risks for AI systems: Further, the Act mandates that providers of high-risk AI systems must fulfil certain obligations like establishing a risk management system, conducting data governance, embedding AI systems with automatic record-keeping, and maintaining technical documentation, among others. Before the Act's passage, the EU aimed to balance regulatory proportionality -- protecting fundamental rights and freedoms without hindering AI adoption. The framework acknowledges that certain AI systems require higher scrutiny. In 2024, the United States Commerce Department's National Institute of Standards and Technology (NIST) published a guidance document identifying generative AI risks and corresponding solutions to mitigate them. The risk management efforts were framed considering potential harms to people, organisations, and ecosystems. These risks are categorised according to the UK's International Scientific Report on the Safety of Advanced AI into four categories: Meta's document detailing its risk approach comes at a time when several regions have banned its rival DeepSeek AI, citing data privacy concerns. Interestingly, as TechCrunch noted, these developments may constitute an effort to differentiate its AI strategy from the Chinese firm. Further, while Meta moves toward a risk-based classification for its AI models, this compliance remains largely voluntary and focuses on the company's internal risk management strategies and governance. Finally, as Meta contends its framework is not absolute and subject to updates depending on the evolution of the AI ecosystem, it will be interesting to note how the tech company complies with global risk-based norms and develops its own simultaneously.

Meta's New Report Shows How to Prevent 'Catastrophic Risks' from AI

The framework is structured around a three-stage process: anticipating risks, evaluating and mitigating risks, and deciding whether to release, restrict, or halt the model. Meta, the company behind the open-source Llama family of AI models, unveiled a new report on Monday titled 'The Frontier AI Framework'. This report outlines the company's approach to developing general-purpose AI models by evaluating and mitigating their "catastrophic risks". It focuses on large-scale risks, such as cybersecurity threats and risks from chemical and biological weapons. "By prioritising these areas, we can work to protect national security while promoting innovation," read a blog post from the company announcing the report. The framework is structured around a three-stage process: anticipating risks, evaluating and mitigating risks, and deciding whether the model should be released, restricted, or halted. The report contains risk assessments from various experts in multiple disciplines, including engineering, product management, compliance and privacy, legal and policy, and other company leaders. If the risks are deemed "critical", the framework suggests stopping the development of the model and restricting access to a small number of experts with security protections. If the risks are deemed "high", the model should not be released but can be accessed by a core research team with security protections. Lastly, if the risks are "moderate", the model can be released as per the framework. It is recommended that you read the full report to understand each of these risk levels in detail. "While it's not possible to entirely eliminate risk if we want this AI to be a net positive for society, we believe it's important to work internally and, where appropriate, with governments and outside experts to take steps to anticipate and mitigate severe risks that it may present," the report stated. Multiple companies building AI models have unveiled such frameworks over the years. For instance, Anthropic's 'Responsible Scaling Policy' provides technical and organisational protocols that the company is adopting to "manage risks of developing increasingly capable AI systems". Anthropic's policy defines four safety levels. The higher the safety level, the larger the model's capability, which increases security and safety measures. In October last year, the company said it was required to upgrade its security measures to an AI safety level of 3, which is the penultimate level of safety. This level suggests that the model's capability poses a "significantly higher risk". Similarly, OpenAI has a charter that outlines its mission to develop and deploy AI models safely. Furthermore, the company regularly releases a system card for all of its models, which outlines its work regarding safety and security before releasing any new variant.