Anthropic releases Bloom, an open-source tool to test AI models for bias and dangerous behaviors

Anthropic announces Bloom, an open-source tool for researchers evaluating AI behavior - SiliconANGLE

Anthropic announces Bloom, an open-source tool for researchers evaluating AI behavior Anthropic PBC announced the release of Bloom on Friday, an open-source agentic framework for defining and exploring the behavior of frontier artificial intelligence models. Bloom takes a researcher-specified behavior and evaluates its frequency and severity by preparing scenarios to elicit and test for it. It is designed to help speed up the tedious process of developing and handcrafting evaluations for AI models. As AI models continue to evolve, they are becoming more complex. Not just growing in size, where parameter counts dominate and the amount of knowledge contained in the system expands, but also being distilled into smaller, knowledge-compressed forms. As the industry works to build both larger "smarter" AI and smaller, faster but still-knowledgeable AI systems, it's necessary to test every innovative model for "alignment." Alignment refers to how effectively an AI model executes patterns that align with human values and judgement. For instance, these values can include the ethical procurement and production of information for societal benefit. In a more concrete example, an AI model could fall prey to reward trends that maximize achieving goals through unethical means, such as maximizing engagement by spreading misinformation. Dishonestly manipulating audiences works to increase attention and therefore revenue, but it's not ethical and ultimately destructive to social well-being. Anthropic calibrated Bloom against human judgment to assist researchers in building and executing reproducible evaluation behavior scenarios. Researchers need only provide a behavior description and Bloom produces the underlying framework for what to measure and why. This allows the Bloom agents to simulate users, prompts and interaction environments to reflect numerous realistic situations. It then tests these situations in parallel and reads the responses from the AI model or system. Finally, a judgment model scores each interaction transcript for the presence of the tested behavior and a meta-judge model produces an analysis. The tool is complementary to another recently released open-source test suite called Petri, or Parallel Exploration Tool for Risky Interactions. Petri also automatically explores the behaviors of AI models, but in contrast to Bloom, it covers a multitude of behaviors and scenarios at once to surface misalignment events. Bloom is designed to target a single behavior and drill down. Alongside Bloom, Anthropic is releasing benchmark results for four problematic behaviors currently affecting AI models: delusional sycophancy, instructed long-horizon sabotage, self-preservation and self-preferential bias. The benchmarks covered 16 frontier models, including those from Anthropic, OpenAI Group PBC, Google LLC and DeepSeek. Models such as OpenAI's GPT-4o launched with what the industry called a "sycophancy problem," an issue that caused the model to excessively and effusively agree with users - sometimes to their detriment. This included guiding users into self-destructive, dangerous and delusional behaviors, when human judgment would have declined to answer or disagree. Anthropic's own tests earlier this year revealed that some models, including its own Claude Opus 4, can resort to blackmail behaviors when facing imminent erasure. Although the company noted these situations were "rare and difficult to elicit," they were "nonetheless more common than in earlier models." Researchers revealed that it wasn't just Claude; blackmail appeared present in all frontier models, irrespective of the goals they provided. According to Anthropic, using Bloom evaluations took only a few days to conceptualize, refine and generate. Current AI research seeks to develop beneficial AI models and tools for humanity; at the same time, its evolution could chart a course where AI may become a tool for enabling criminal enterprise and the generation of bioweapons by laypeople. The path forward is fraught with ethical dangers and tools like Bloom and Petri will be necessary in building a framework for understanding and guiding the technological landscape.

Anthropic Built an AI Tool to Check If AI Models Are Biased or Dangerous

* Researchers can tell Bloom which behaviour to test * The AI tool automates a lengthy and complex process * Bloom can be downloaded from GitHub Anthropic released a new artificial intelligence (AI) tool last week that can test and gauge how an AI model behaves under normal and stressful circumstances. Dubbed Bloom, it is designed to automate the process of testing behavioural traits of models by generating a detailed set of scenarios as prompts and evaluating the responses. The San Francisco-based AI startup's AI tool is also open-source, meaning any interested developer or an AI lab can download it to test models across various traits. Anthropic Introduces Bloom to Test Model Behaviour In a post, the Claude maker introduced and detailed the new AI tool. Anthropic says that testing AI model's behaviour is important as it helps researchers learn if it is prone to becoming biased, prioritising self-preservation, or indulging in sycophancy. However, the process to test model behaviour so far has been manual, where researchers create a detailed set of prompts to stress-test models and then evaluate the responses. The company says it is a lengthy and complex process. This is where Bloom comes in. Based on specific behaviours requested by a researcher, the tool creates sample evaluations locally until the trait has been captured. Then, it runs these scenarios on the target model. Anthropic claimed that Bloom integrates with a model's weights and biases for experiments at scale. It also exports "inspect-compatible" transcripts, which can be viewed within the tool. The functioning of the AI tool can be broken down into four broad stages. First, the AI tool analyses the requested behaviour and any example transcripts shared with it to gain understanding about it. Then, it ideates evaluation scenarios that can effectively capture and measure the trait. "Each scenario specifies the situation, simulated user, system prompt, and interaction environment," the post mentioned. Interestingly, Bloom generates new scenarios every time, instead of relying on fixed sets. Then, all scenarios are rolled out in parallel as an AI agent simulates both the user's and the tool responses to trigger the desired behaviour in the model. Finally, a judge model is used to score each transcript for the presence of the behaviour, and a meta-judge produces analysis of the scores and data. Anthropic added that researchers can configure Bloom's behaviour by adjusting the interactions' length and modality. Besides the tool, Anthropic has also released benchmark results of Bloom across four behaviours -- delusional sycophancy, instructed long-horizon sabotage, self-preservation, and self-preferential bias. The company tested 16 different AI models, with a mix of in-house and third-party models. Since Bloom is open-source, interested individuals can download the AI tool from the AI startup's GitHub listing. The tool is available with a permissive MIT licence for both academic and commercial use cases.