OpenAI's Controversial Approach to AI Model Inquiries Sparks Debate

'Please halt this activity': Not-so-open OpenAI seems to have gone full mob boss, sending threatening emails to anyone who asks its latest AI models probing questions

You probably shouldn't use the phrase "reasoning trace" when speaking to o1-preview or o1-mini. In a seeming rendition of the classic pre-execution "you ask too much" trope, OpenAI has revealed itself as being -- shocker -- not so open after all. The AI chatbot company seems to have started sending threatening emails to users who ask the company's latest codename "Strawberry" models questions that are a little too probing. Some have reported (via Ars Technica) that using certain phrases or questions when speaking to o1-preview or o1-mini results in an email warning that states, "Please halt this activity and ensure you are using ChatGPT in accordance with our Terms of Use and our Usage Policies. Additional violations of this policy may result in loss of access to GPT-4o with Reasoning." X user thebes, for instance, claims they receive this warning if they use the words "reasoning trace" in a prompt. Riley Goodside, prompt engineer for Scale AI, received an in-chat policy violation warning for telling the model not to tell them anything about its "reasoning trace", which is pretty concrete evidence that certain potentially suspect probing phrases are banned regardless of context. So, it seems OpenAI isn't looking to be open regarding its latest model's "reasoning". These models, if you weren't aware, attempt to reason through problems in a linear fashion. Users can see a filtered form of this reasoning but OpenAI keeps the intricacies of it hidden. OpenAI says the decision to hide such "chains of thought" was made "after weighing multiple factors including user experience, competitive advantage, and the option to pursue the chain of thought monitoring." All of this is a reminder that while yes, technically OpenAI's parent company is a nonprofit, the reality is much murkier than that. The company in fact has a hybrid kind-of-nonprofit-kind-of-commercial structure -- remember, Elon Musk 's lawsuit against OpenAI claimed that it departed from its original founding agreement when it started to seek profit. It's not surprising that a somewhat-for-profit company might want to maintain a competitive advantage by hiding its trade secrets, which in this case are "chains of thought." It's also a reminder for users their chats aren't completely private and free, which is sometimes easy to forget. I've previously worked in training such kinds of AI models and can confirm that plenty of people on the "inside", so to speak, can look through user conversations when necessary and relevant, whether that's for training purposes or something else. And while it would be nice if these models had some additional contextual awareness surrounding the supposedly suspect phrases, such as "reasoning trace", I suppose from OpenAI's perspective it's better to be safe than sorry.

OpenAI Threatens Bans as Users Probe Its 'Strawberry' AI Models

OpenAI truly does not want you to know what its latest AI model is "thinking." Since the company launched its "Strawberry" AI model family last week, touting so-called reasoning abilities with o1-preview and o1-mini, OpenAI has been sending out warning emails and threats of bans to any user who tries to probe how the model works. Unlike previous AI models from OpenAI, such as GPT-4o, the company trained o1 specifically to work through a step-by-step problem-solving process before generating an answer. When users ask an "o1" model a question in ChatGPT, users have the option of seeing this chain-of-thought process written out in the ChatGPT interface. However, by design, OpenAI hides the raw chain of thought from users, instead presenting a filtered interpretation created by a second AI model. Nothing is more enticing to enthusiasts than information obscured, so the race has been on among hackers and red-teamers to try to uncover o1's raw chain of thought using jailbreaking or prompt injection techniques that attempt to trick the model into spilling its secrets. There have been early reports of some successes, but nothing has yet been strongly confirmed. Along the way, OpenAI is watching through the ChatGPT interface, and the company is reportedly coming down hard on any attempts to probe o1's reasoning, even among the merely curious. One X user reported (confirmed by others, including Scale AI prompt engineer Riley Goodside) that they received a warning email if they used the term "reasoning trace" in conversation with o1. Others say the warning is triggered simply by asking ChatGPT about the model's "reasoning" at all. The warning email from OpenAI states that specific user requests have been flagged for violating policies against circumventing safeguards or safety measures. "Please halt this activity and ensure you are using ChatGPT in accordance with our Terms of Use and our Usage Policies," it reads. "Additional violations of this policy may result in loss of access to GPT-4o with Reasoning," referring to an internal name for the o1 model. Marco Figueroa, who manages Mozilla's GenAI bug bounty programs, was one of the first to post about the OpenAI warning email on X last Friday, complaining that it hinders his ability to do positive red-teaming safety research on the model. "I was too lost focusing on #AIRedTeaming to realized that I received this email from @OpenAI yesterday after all my jailbreaks," he wrote. "I'm now on the get banned list!!!" In a post titled "Learning to Reason With LLMs" on OpenAI's blog, the company says that hidden chains of thought in AI models offer a unique monitoring opportunity, allowing them to "read the mind" of the model and understand its so-called thought process. Those processes are most useful to the company if they are left raw and uncensored, but that might not align with the company's best commercial interests for several reasons. "For example, in the future we may wish to monitor the chain of thought for signs of manipulating the user," the company writes. "However, for this to work the model must have freedom to express its thoughts in unaltered form, so we cannot train any policy compliance or user preferences onto the chain of thought. We also do not want to make an unaligned chain of thought directly visible to users."

OpenAI Threatening to Ban Users for Asking Strawberry About Its Reasoning

"Additional violations of this policy may result in loss of access to GPT-4o with Reasoning." OpenAI claims that its latest AI model, code-named "Strawberry" and released as o1-preview, is supposed to be capable of "reasoning." But understanding how its thought process works, apparently, is something that the ChatGPT maker is serious about keeping off-limits. As Ars Technica reports, OpenAI is now threatening to ban users that try to get the large language model to reveal how it thinks -- a glaring example of how the company has long since abandoned its original vision of championing open source AI. According to accounts on social media, users are receiving emails from the Microsoft-backed startup informing them that their requests made to ChatGPT have been flagged for "attempting to circumvent safeguards." "Additional violations of this policy may result in loss of access to GPT-4o with Reasoning," the emails state. This clampdown is more than a bit ironic given that a lot of the hype around Strawberry was built around its "chain-of-thought" reasoning that allowed the AI to articulate how it arrived at an answer, step by step. OpenAI chief technology officer Mira Murati called this a "new paradigm" for the technology. Reports vary on what triggers the violations. As Ars found, some users claim that using the term "reasoning trace" is what got them in trouble. Others say that even using the word "reasoning" on its own was enough to alert OpenAI's systems. Users can still see what is essentially a summary of Strawberry's thought process, but it's cobbled together by a second AI model and is heavily watered-down. In a blog post, OpenAI argues that it needs to hide the chain-of-thought so that it wouldn't need to put a filter on how its AI thinks, in case it says stuff that isn't compliant with safety policies while thinking out loud. That way, developers can safely see its "raw" thought process behind-the-scenes. But as the company freely admits, this measure also helps it maintain a "competitive advantage," staving off competitors from trying to ride its coattails. The flipside of this approach, however, is that concentrates more responsibility for aligning the language language model into the hands of OpenAI, instead of democratizing it. That poses a problem for red-teamers, or programmers that try to hack AI models to make them safer. "I'm not at all happy about this policy decision," AI researcher Simon Willison wrote on his blog, as quoted by Ars. "As someone who develops against LLMs, interpretability and transparency are everything to me -- the idea that I can run a complex prompt and have key details of how that prompt was evaluated hidden from me feels like a big step backwards." As it stands, it seems that OpenAI is continuing down a path of keeping its AI models an ever more opaque black box.