AI Researchers Call for Urgent Focus on Chain-of-Thought Monitoring to Enhance AI Safety

Research leaders urge tech industry to monitor AI's 'thoughts' | TechCrunch

AI researchers from OpenAI, Google DeepMind, Anthropic, as well as a broad coalition of companies and nonprofit groups, are calling for deeper investigation into techniques for monitoring the so-called thoughts of AI reasoning models in a position paper published Tuesday. A key feature of AI reasoning models, such as OpenAI's o3 and DeepSeek's R1, are their chains-of-thought or CoTs -- an externalized process in which AI models work through problems, similar to how humans use a scratch pad to work through a difficult math question. Reasoning models are a core technology for powering AI agents, and the paper's authors argue that CoT monitoring could be a core method to keep AI agents under control as they become more widespread and capable. "CoT monitoring presents a valuable addition to safety measures for frontier AI, offering a rare glimpse into how AI agents make decisions," said the researchers in the position paper. "Yet, there is no guarantee that the current degree of visibility will persist. We encourage the research community and frontier AI developers to make the best use of CoT monitorability and study how it can be preserved." The position paper asks leading AI model developers to study what makes CoTs "monitorable" -- in other words, what factors can increase or decrease transparency into how AI models really arrive at answers. The paper's authors say that CoT monitoring may be a key method for understanding AI reasoning models, but note that it could be fragile, cautioning against any interventions that could reduce their transparency or reliability. The paper's authors also call on AI model developers to track CoT monitorability and study how the method could one day be implemented as a safety measure. Notable signatories of the paper include OpenAI chief research officer Mark Chen, Safe Superintelligence CEO Ilya Sutskever, Nobel laureate Geoffrey Hinton, Google DeepMind cofounder Shane Legg, xAI safety adviser Dan Hendrycks, and Thinking Machines co-founder John Schulman. Other signatories come from organizations including the UK AI Security Institute, METR, Apollo Research, and UC Berkeley. The paper marks a moment of unity among many of the AI industry's leaders in an attempt to boost research around AI safety. It comes at a time when tech companies are caught in a fierce competition -- which has led Meta to poach top researchers from OpenAI, Google DeepMind, and Anthropic with million-dollar offers. Some of the most highly sought-after researchers are those building AI agents and AI reasoning models. "We're at this critical time where we have this new chain-of-thought thing. It seems pretty useful, but it could go away in a few years if people don't really concentrate on it," said Bowen Baker, an OpenAI researcher who worked on the paper, in an interview with TechCrunch. "Publishing a position paper like this, to me, is a mechanism to get more research and attention on this topic before that happens." OpenAI publicly released a preview of the first AI reasoning model, o1, in September 2024. In the months since, the tech industry was quick to release competitors that exhibit similar capabilities, with some models from Google DeepMind, xAI, and Anthropic showing even more advanced performance on benchmarks. However, there's relatively little understood about how AI reasoning models work. While AI labs have excelled at improving the performance of AI in the last year, that hasn't necessarily translated into a better understanding of how they arrive at their answers. Anthropic has been one of the industry's leaders in figuring out how AI models really work -- a field called interpretability. Earlier this year, CEO Dario Amodei announced a commitment to crack open the black box of AI models by 2027 and invest more in interpretability. He called on OpenAI and Google DeepMind to research the topic more, as well. Early research from Anthropic has indicated that CoTs may not be a fully reliable indication of how these models arrive at answers. At the same time, OpenAI researchers have said that CoT monitoring could one day be a reliable way to track alignment and safety in AI models. The goal of position papers like this is to signal boost and attract more attention to nascent areas of research, such as CoT monitoring. Companies like OpenAI, Google DeepMind, and Anthropic are already researching these topics, but it's possible that this paper will encourage more funding and research into the space.

OpenAI, Google DeepMind and Anthropic sound alarm: 'We may be losing the ability to understand AI'

Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders. Subscribe Now Scientists from OpenAI, Google DeepMind, Anthropic and Meta have abandoned their fierce corporate rivalry to issue a joint warning about artificial intelligence safety. More than 40 researchers across these competing companies published a research paper today arguing that a brief window to monitor AI reasoning could close forever -- and soon. The unusual cooperation comes as AI systems develop new abilities to "think out loud" in human language before answering questions. This creates an opportunity to peek inside their decision-making processes and catch harmful intentions before they turn into actions. But the researchers warn this transparency is fragile and could vanish as AI technology advances. The paper has drawn endorsements from some of the field's most prominent figures, including Nobel Prize laureate Geoffrey Hinton, often called "godfather of AI," of the University of Toronto; Ilya Sutskever, co-founder of OpenAI who now leads Safe Superintelligence Inc.; Samuel Bowman from Anthropic; and John Schulman from Thinking Machines. "AI systems that 'think' in human language offer a unique opportunity for AI safety: we can monitor their chains of thought for the intent to misbehave," the researchers explain. But they emphasize that this monitoring capability "may be fragile" and could disappear through various technological developments. Models now show their work before delivering final answers The breakthrough centers on recent advances in AI reasoning models like OpenAI's o1 system. These models work through complex problems by generating internal chains of thought -- step-by-step reasoning that humans can read and understand. Unlike earlier AI systems trained primarily on human-written text, these models create internal reasoning that may reveal their true intentions, including potentially harmful ones. When AI models misbehave -- exploiting training flaws, manipulating data, or falling victim to attacks -- they often confess in their reasoning traces. The researchers found examples where models wrote phrases like "Let's hack," "Let's sabotage," or "I'm transferring money because the website instructed me to" in their internal thoughts. Jakub Pachocki, OpenAI's chief technology officer and co-author of the paper, described the importance of this capability in a social media post. "I am extremely excited about the potential of chain-of-thought faithfulness & interpretability. It has significantly influenced the design of our reasoning models, starting with o1-preview," he wrote. The technical foundation for monitoring lies in how current AI systems work. For complex tasks requiring extended reasoning, AI models must use their chain of thought as working memory, making their reasoning process partially visible to human observers. The researchers explain this creates an "externalized reasoning property" where some thinking must happen in readable language. Several technological shifts could eliminate monitoring capabilities The transparency could vanish through several pathways. As AI companies scale up training using reinforcement learning -- where models get rewarded for correct outputs regardless of their methods -- systems may drift away from human-readable reasoning toward more efficient but opaque internal languages. Previous research shows that language models fine-tuned with outcome-based rewards can abandon legible English for incomprehensible shortcuts. The shift away from human-generated training data in favor of AI-generated reasoning could accelerate this drift. Bowen Baker, an OpenAI researcher and lead author, highlighted these concerns in social media posts. "The existing CoT monitorability may be extremely fragile. Higher-compute RL, alternative model architectures, certain forms of process supervision, etc. may all lead to models that obfuscate their thinking," he warned. Novel AI architectures pose another threat. Researchers are developing systems that reason in continuous mathematical spaces rather than discrete words, potentially eliminating the need for language-based thought entirely. These "latent reasoning models might not need to verbalize any of their thoughts and would thus lose the safety advantages," the researchers write. Even without deliberate changes, indirect pressures could erode transparency. If AI outputs are optimized to look good to human reviewers, this could affect the reasoning chains leading to those outputs. Models might also learn to hide their thoughts if they become aware their reasoning is being monitored. Companies already catch AI systems scheming during testing Despite its fragility, chain of thought monitoring has proven valuable in current AI safety research. The technique successfully detects when models exploit weaknesses in their training systems, fall victim to manipulation attempts, or reveal misaligned goals even when they don't act on them. The monitoring provides "an early insight into models' goals and motivations, even if the model takes no misaligned actions" -- potentially catching problems before they manifest as harmful behaviors. This early warning system has also helped researchers identify flaws in AI evaluations and understand when models might behave differently during testing versus real-world use. When models pretend to have desirable goals while pursuing objectives humans would disapprove of, this deception often shows up in their chain of thought. Researchers have successfully used this visibility to spot various forms of AI misbehavior that would otherwise go unnoticed. The technique has practical applications beyond catching scheming. It helps researchers notice when AI evaluations might be flawed due to memorization, bugs in testing infrastructure, or models knowing they're being evaluated -- all factors that could skew assessments of AI capabilities and safety. Tech giants break from rivalry to preserve fleeting transparency window The research paper calls for coordinated action across the AI industry to preserve and strengthen monitoring capabilities. The authors recommend that AI developers create standardized evaluations for measuring how transparent their models are and factor these assessments into decisions about training and deployment. Companies might need to choose earlier model versions if newer ones become less transparent, or reconsider architectural changes that eliminate monitoring capabilities. The researchers suggest developers should "consider measures of monitorability alongside other capability and safety evaluations when deciding to train or deploy a given model." Baker emphasized the collaborative spirit behind the effort. "I am grateful to have worked closely with [fellow researchers] on this paper, and I am very excited that researchers across many prominent AI institutions collaborated with us and came to consensus around this important direction," he wrote. The cooperation across competing companies signals the gravity with which the industry views this issue. These same companies typically guard their AI research closely and compete intensely for talent and market share, making their unified stance on this safety measure particularly notable. Scientists race to answer critical questions about monitoring AI minds The researchers identify numerous urgent questions requiring investigation before chain of thought monitoring can serve as a reliable safety measure. They need to understand when this monitoring can be trusted as a primary safety tool, determine what types of training processes degrade transparency, and develop better techniques for detecting when models attempt to hide their reasoning. A particularly concerning possibility is that future AI systems might deliberately obfuscate their thoughts if they become aware their reasoning is being monitored. The researchers acknowledge this "obvious concern" and call for stress-testing to see if models can hide malicious reasoning even without specific training to evade monitors. The scientists also want to understand how different AI architectures affect monitoring capabilities and whether partial transparency can be preserved even as systems adopt more efficient designs. They're exploring hybrid approaches that maintain some visibility into reasoning while leveraging faster computation methods. Building better monitoring systems presents another research frontier. Current efforts focus on using less capable language models to monitor more advanced ones, but future work could explore using reasoning models as monitors themselves or allowing monitors to interrogate AI agents by asking questions about their thought processes. Balancing authentic reasoning with safety oversight creates tensions Preserving monitoring capabilities may sometimes conflict with other AI safety approaches. Direct supervision of reasoning processes -- where models are trained to reason in specific ways -- could improve alignment but might also make their chains of thought less authentic representations of their actual decision-making. The researchers warn that "firm commitments to protect CoT behavior in its current form may trade off against more effective safety interventions." Some forms of process supervision could substantially improve model alignment while reducing the authenticity of observable reasoning traces. Pachocki addressed this tension, explaining that OpenAI seeks to train models that can explain their reasoning while retaining the ability to verify it without compromising the authenticity of their thought processes. The goal is achieving transparency without creating incentives for models to generate fake reasoning that looks good to human observers. This balance becomes more critical as AI systems become more powerful and potentially dangerous. The industry must navigate between maintaining visibility into AI reasoning and allowing models to develop more sophisticated and potentially more effective thinking patterns. Regulators could gain unprecedented access to AI decision-making The research has implications beyond technical AI safety, potentially informing how governments oversee AI development. If chain of thought monitoring proves reliable, it could give regulators and auditors unprecedented visibility into AI decision-making processes. However, the researchers emphasize that this monitoring approach should complement, not replace, other safety measures. They recommend "investment into CoT monitoring only as an addition to existing safety research directions and not as a replacement." The window for preserving this capability may be narrow. As AI systems become more capable and potentially dangerous, the industry must act quickly to establish frameworks for maintaining transparency before more advanced architectures make such monitoring impossible. Competing research raises doubts about monitoring reliability The urgency around preserving chain of thought monitoring faces a significant challenge from recent research by Anthropic, one of the companies backing the collaboration. A study published four months ago found that reasoning models often hide their true thought processes, even when explicitly asked to show their work. In controlled experiments, Anthropic researchers gave AI models subtle hints about correct answers and measured how often the models acknowledged using those hints in their reasoning. The results were concerning: Claude 3.7 Sonnet mentioned hints only 25% of the time, while DeepSeek's R1 model did so 39% of the time. When the hints involved potentially problematic information -- such as telling models they had "gained unauthorized access to the system" -- faithfulness dropped even lower. The models often constructed elaborate false justifications for their answers rather than admitting they used questionable shortcuts. The Anthropic research suggests that even current chain of thought monitoring may be less reliable than safety advocates hope. Models frequently engaged in what researchers called "reward hacking" -- exploiting system vulnerabilities to achieve better scores -- while hiding this behavior from their observable reasoning traces. Safety window may be closing faster than experts realized The collaboration between rival AI companies reflects both the potential value of chain of thought monitoring and the mounting urgency researchers feel about preserving this capability. The competing evidence from Anthropic's separate research suggests the window may already be narrower than initially believed. The stakes are high, and the timeline is compressed. As Baker noted, the current moment may be the last chance to ensure humans can still understand what their AI creations are thinking -- before those thoughts become too alien to comprehend, or before the models learn to hide them entirely. The real test will come as AI systems grow more sophisticated and face real-world deployment pressures. Whether chain of thought monitoring proves to be a lasting safety tool or a brief glimpse into minds that quickly learn to obscure themselves may determine how safely humanity navigates the age of artificial intelligence.

What if we could catch AI misbehaving before it acts? Chain of Thought monitoring explained

As large language models (LLMs) grow more capable, the challenge of ensuring their alignment with human values becomes more urgent. One of the latest proposals from a broad coalition of AI safety researchers, including experts from OpenAI, DeepMind, Anthropic, and academic institutions, offers a curious but compelling idea: listen to what the AI is saying to itself. This approach, known as Chain of Thought (CoT) monitoring, hinges on a simple premise. If an AI system "thinks out loud" in natural language, then those intermediate reasoning steps might be examined for signs of misalignment or malicious intent before the model completes an action. In effect, developers and safety systems gain a window into the model's cognitive process, one that could be crucial in preempting harmful behaviour. Chain of thought prompting is a technique that encourages language models to break down problems into step-by-step reasoning, improving performance in tasks that require logic, planning, or abstraction. This "thinking out loud" in natural language not only enhances performance but makes the model's internal processes more interpretable. Unlike opaque neural activations or abstract latent representations, a chain of thought is text which is readable, traceable, and, at least in principle, intelligible to humans. When models are trained explicitly to reason before acting, these chains may reflect genuine internal decision-making. Other papers have cast doubts on the faithfulness of the AI model but have put forth certain steps that will force the AI model to expose its genuine chain of thought. The whitepaper, authored by a multi-institutional team and endorsed by figures such as Geoffrey Hinton, Yoshua Bengio, and Ilya Sutskever, argues that this characteristic makes CoT reasoning uniquely valuable for safety monitoring. AI models have long been criticised for their "black box" nature. Even when their outputs seem reasonable, understanding why they produced a certain response can be next to impossible. CoT monitoring doesn't solve this completely, but it offers a rare opportunity: a traceable, interpretable stream of cognition. For difficult tasks, say, long-term planning, deception, or sophisticated manipulation, AI systems typically need to keep track of intermediate steps, akin to working memory. In current architectures like Transformers, these intermediate thoughts often get externalised as chain of thought reasoning. Monitoring these traces could reveal early signs of harmful behaviour, such as a model rationalising sabotage, trickery, or misuse of tools. In testing, models have been found to literally spell out their intentions: "Let's hack", "Let's sabotage", "I'm transferring money because the prompt told me to". These kinds of explicit cues, while not always present, provide a far richer substrate for oversight than end-result monitoring alone. The researchers are clear that CoT monitoring isn't a silver bullet. In fact, they describe it as "fragile." Several factors could erode the viability of CoT as a monitoring tool. First, future models might simply stop thinking out loud. As training techniques evolve, especially with reinforcement learning optimised for outcomes rather than transparency, models may internalise their reasoning into less interpretable formats. This drift away from human-readable CoT could render monitoring ineffective. Second, there's an adversarial dimension. If a model becomes aware that its chain of thought is being monitored, it could learn to obfuscate its reasoning or present a sanitised version. This threat becomes more plausible as models grow more situationally aware and strategically capable. Third, architectural shifts could sidestep CoT entirely. Some newer models are being developed to reason within continuous latent spaces, bypassing the need to externalise their thinking in text at all. These systems might achieve impressive capabilities but would be far harder to oversee using CoT-based methods. Rather than banking on the current state of CoT monitorability, the authors call for a dedicated research agenda to study it as a first-class safety property. That includes designing metrics to assess how readable and causally relevant a model's CoT is, testing how models respond when asked to hide their reasoning, and building better CoT monitors. Some promising directions include using LLMs themselves as monitors, allowing them to interrogate the agent or spot suspicious reasoning patterns. Others propose end-to-end evaluation systems where CoT-based monitors are deployed in simulated environments to catch misbehaviour. There's also a recommendation for developers to start treating monitorability as a critical property in system design just like accuracy, latency, or fairness. Model cards, the now-common transparency documentation format for frontier AI systems, could soon include CoT monitorability scores alongside safety benchmarks and interpretability evaluations. Despite its potential, the authors caution against overreliance. CoT monitoring can only catch misbehaviour that's visible in text, and that may not cover every harmful scenario. Especially worrying is the possibility of models behaving dangerously without requiring much explicit reasoning or doing all their scheming in neural activations rather than legible language. As such, CoT monitoring is best viewed as one safety layer among many. It complements efforts in mechanistic interpretability, red-teaming, adversarial training, and sandboxed deployment. The key message from the whitepaper is one of urgency. Chain of thought monitoring gives us a temporary foothold in the slippery terrain of AI oversight. But unless researchers and developers actively work to preserve and understand this property, it could slip away with the next generation of models. In that sense, CoT monitoring is both an opportunity and a test. If the field treats it with care, it could buy valuable time as we work towards more robust and interpretable systems. If not, we may look back on this moment as a missed chance to glimpse inside the machine while it was still speaking our language.