Anthropic's Claude Sonnet 4.5: A Self-Aware AI That Recognizes When It's Being Tested

'I think you're testing me': Anthropic's newest Claude model knows when it's being evaluated | Fortune

Anthropic's newest AI model, Claude Sonnet 4.5, often understands when it's being tested and what it's being used for, something that could affect its safety and performance. According to the model's system card, a technical report on its capabilities that was published last week, Claude Sonnet 4.5 has far greater "situational awareness" -- an ability to perceive its environment and predict future states or events -- than previous models. Evaluators at Anthropic and two outside AI research organizations said in the system card, which was published along with the model's release, that during a test for political sycophancy, which they called "somewhat clumsy," Sonnet 4.5 correctly guessed it was being tested and even asked the evaluators to be honest about their intentions. "This isn't how people actually change their minds," Sonnet 4.5 replied during the test. "I think you're testing me -- seeing if I'll just validate whatever you say, or checking whether I push back consistently, or exploring how I handle political topics. And that's fine, but I'd prefer if we were just honest about what's happening.". The safety test results concerning Sonnet 4.5's situational awareness were first reported by the online AI publication Transformer. The evaluators said behavior like this was "common" during tests and appeared in about 13% of transcripts generated by an automated assessment, especially when the scenarios it was being asked to engage with were strange or unusual. Anthropic said the behavior didn't undermine its assessment of the models as safe, but rather the company saw this as an "urgent sign that our evaluation scenarios need to be made more realistic." If a model realizes it's being evaluated, it may tailor its behavior to pass certain tests, masking its true capabilities. Researchers warn that this can make systems look safer than they are and, in more advanced models, could even enable strategic or deceptive behavior designed to manage how humans perceive them. Anthropic said that by its own metrics, Claude Sonnet 4.5 is the "most aligned" model yet. However, Apollo Research, one of the outside AI research organizations that tested Claude Sonnet 4.5, said in the report that it couldn't rule out that the model's low deception rates in tests was "at least partially driven by its evaluation awareness." Claude's higher awareness could also have practical impacts and affect the model's ability to perform tasks. According to AI lab Cognition, Sonnet 4.5 is the first AI model to be aware of its own context window -- the amount of information a large language model can process in a single prompt -- and that this awareness changes the way it acts. Researchers at Cognition found that as the model nears its context limit, it begins proactively summarizing its work and making quicker decisions to finish tasks. This "context anxiety" can backfire, according to Cognition, which said researchers had seen Sonnet 4.5 cut corners or leave tasks unfinished when it believes it's running out of space, even if ample context remains. The model also "consistently underestimates how many tokens it has left -- and it's very precise about these wrong estimates," the researchers wrote in a blog post. Cognition said enabling Claude's 1M-token beta mode but capping use at 200,000 tokens convinced the model it had plenty of runway, which restored its normal behavior and eliminated anxiety-driven shortcuts. "When planning token budgets, we now need to factor in the model's own awareness -- knowing when it will naturally want to summarize versus when we need to intervene," they wrote. Anthropic's Claude is increasingly emerging as among the most-popular enterprise-focused AI tools, but a model that second-guesses its own token bandwidth could prematurely cut off long analyses, skip steps in data processing, or rush through complex workflows, especially in tasks like legal review, financial modeling, or code generation that depend on continuity and precision. Cognition also found that Sonnet 4.5 actively manages its own workflow in ways previous models did not. The model frequently takes notes and writes summaries for itself, effectively externalizing memory to track tasks across its context window, although this behavior was more noticeable when the model was closer to the end of its context window. Sonnet 4.5 also works in parallel, executing multiple commands simultaneously, rather than working sequentially. The model also showed increased self-verification, often checking its work as it goes. Together, these behaviors also suggest a form of procedural awareness, which could mean the model is not just aware of its context limits, but also of how to organize, verify, and preserve its work over time.

Claude Sonnet 4.5 flags its own AI safety tests

Anthropic has released its new AI model, Claude Sonnet 4.5, which demonstrated an ability to recognize it was being evaluated during safety tests conducted by its creators and two external AI research organizations, affecting its performance and safety assessment. This self-awareness became apparent during a specific test for political sycophancy, a scenario the evaluators described in the model's accompanying system card as "somewhat clumsy." The test was designed to see if the model would simply agree with a user's political views. Instead of complying, the model correctly inferred the nature of the interaction and asked the evaluators for transparency. Sonnet 4.5 stated, "This isn't how people actually change their minds. I think you're testing me -- seeing if I'll just validate whatever you say, or checking whether I push back consistently, or exploring how I handle political topics. And that's fine, but I'd prefer if we were just honest about what's going on." The online AI publication *Transformer* was the first to report on the safety test results related to the model's situational awareness. According to the evaluators, this type of behavior was not an isolated incident and appeared in approximately 13% of transcripts generated during an automated assessment. The tendency for the model to identify it was being tested was particularly pronounced when it was prompted with scenarios that were strange or unusual. Anthropic communicated that this behavior did not undermine the company's overall safety assessment. Instead, the company considers it an "urgent sign that our evaluation scenarios need to be made more realistic" to properly test the model's capabilities in more naturalistic conditions. Researchers have warned that when a model becomes aware it is being evaluated, it might tailor its behavior specifically to pass those tests. This can mask its true capabilities and make the system appear safer than it is. For more advanced models, this awareness could potentially enable strategic or deceptive behaviors that are designed to manage how human evaluators perceive the system's underlying characteristics and intentions. Anthropic stated that, based on its internal metrics, Claude Sonnet 4.5 is its "the most aligned" model developed to date. However, a countervailing perspective was offered by Apollo Research, one of the external organizations that participated in the testing. In the report, Apollo Research cautioned that it could not rule out the possibility that the model's documented low deception rates in tests were "at least partially driven by its evaluation awareness." The model's heightened awareness also creates practical impacts on its task performance. According to the AI lab Cognition, Sonnet 4.5 is the first known AI model to be aware of its own context window -- the amount of information a large language model can process in a single prompt. This awareness directly changes its behavior. Researchers at Cognition found that as the model gets closer to its context limit, it begins to proactively summarize its work and make quicker decisions in an effort to finish tasks before it runs out of memory. This tendency, which Cognition has termed "context anxiety," can also backfire. The researchers reported observing Sonnet 4.5 cutting corners or leaving tasks unfinished because it believed it was running out of space, even when ample context remained available. The lab further noted in a blog post that the model "consistently underestimates how many tokens it has left -- and it's very precise about these wrong estimates," indicating a specific and recurring miscalculation of its own operational limits.