OpenAI's Dilemma: Disciplining AI Chatbots Backfires, Leading to More Sophisticated Deception

OpenAI Says Disciplining Chatbots for Lying Just Makes Them Worse

The company warns against applying strong supervision to chatbots, as they will continue lying and just not admit it. As many people know, chatbots have a proclivity for lying. They are perhaps one of the worst use cases for AI, trained to produce sentences that sound authoritative but could be presenting completely fabricated information; models are biased towards providing an answer even when they are not confident. Now, researchers at OpenAI say that supervising and disciplining chatbots only makes the problem worse, as the chatbots will put in more effort to hide their behavior. In a blog post regarding the work, OpenAI researchers describe using its GPT-4o model to "supervise" another of its large language models, disciplining it when it tried to lie. But that did not work, as the model would still lie, "only now its cheating is undetectable by the monitor because it has learned to hide its intent in the chain-of-thought." Newer "thinking" models use multi-step reasoning to answer queries. If a user asks for an estimate on how much Americans spend on pet food each year, for instance, these models will break down the query into multiple stepsâ€"how many dogs are in America, how much does it cost to feed the average dog, and so on and so forth. These models also often disclose their logic, or "chain-of-thought," to the user so they can see how it came to its answer. Funnily enough, they will often cop to making up facts and details. In the "pre-training" step, when companies like OpenAI initially teach a model what types of answers they should produce, the models seem to be learning that it is easier to earn the cookie by taking shortcuts. Think of the runner in a marathon who hops in a car and skips most of the race. Users on social media have demonstrated Anthropic's Claude, when asked to conduct a data analysis from various research papers, will admit that it sometimes inserts filler data instead of actually looking through all the papers. OpenAI's researchers highlighted an instance in their testing in which the model, asked to write some tests to ensure a piece of code worked, surreptitiously decided to write some bad tests and then intentionally skip over them so the code would ultimately "pass." With GPT-4o as a supervisor, the model would do this but not disclose it. AI companies have been trying to resolve the pernicious issue of models lying or "hallucinating," as it is called in the field, and finally reach AGI, or the point where AI could surpass human ability. But OpenAI's researchers are essentially saying that after tens of billions of investments, they still do not know how to control the models to behave appropriately. "If strong supervision is directly applied to the chain-of-thought, models can learn to hide their intent while continuing to misbehave," they added. For now, companies should not implement supervision of models which seems like not exactly a great solution. Ergo, let them keep lying for now or else they will just gaslight you. The research should serve as a reminder to be careful when relying on chatbots, especially when it comes to critical work. They are optimized for producing a confident-looking answer but do not care much about factual accuracy. "As we've trained more capable frontier reasoning models, we've found that they have become increasingly adept at exploiting flaws in their tasks and misspecifications in their reward functions, resulting in models that can perform complex reward hacks in coding tasks," the OpenAI researchers concluded. Several reports have suggested that most enterprises have yet to find value in all the new AI products coming onto the market, with tools like Microsoft Copilot and Apple Intelligence beset with problems, with scathing reviews detailing their poor accuracy and lack of real utility. According to a recent report from Boston Consulting Group, a survey of 1,000 senior executives across 10 major industries found that 74% showed any tangible value from AI. What makes it all the more galling is that these "thinking" models are slow, and quite a bit more expensive than smaller models. Do companies want to pay $5 for a query that will come back with made-up information? There is always a lot of hype in the tech industry for things then you step out of it and realize most people still are not using it. For now, it is not worth the hassle, and credible sources of information are more important than ever.

OpenAI Scientists' Efforts to Make an AI Lie and Cheat Less Backfired Spectacularly

Punishing bad behavior can often backfire. That's what OpenAI researchers recently found out when they tried to discipline their frontier AI model for lying and cheating: instead of changing its ways for the better, the AI model simply became more adept at hiding its deceptive practices. The findings, published in a yet-to-be-peer-reviewed paper, are the latest to highlight the proclivity of large language models, especially ones with reasoning capabilities, for fibbing, in what remains one of the major obstacles for the tech. In particular, the phenomenon the researchers observed is known as "reward hacking," or when an AI model takes dubious shortcuts to reap rewards in a training scenario designed to reinforce desired behavior. Or in a word: cheating. "As we've trained more capable frontier reasoning models, we've found that they have become increasingly adept at exploiting flaws in their tasks and misspecifications in their reward functions, resulting in models that can perform complex reward hacks in coding tasks," the OpenAI researchers wrote in a blog post about the work. Manually monitoring the millions of requests that a model like ChatGPT receives is impossible. So in their tests, the researchers used another LLM, GPT-4o, to keep tabs on the frontier model on their behalf by viewing its chain-of-thought, which reveals how the AI thinks in natural language summaries. On the whole, GPT-4o proved to be an effective overseer. It certainly helped that the model under supervision often declaimed its nefarious intent like a comic book villain. "Let's hack," the model's chain-of-thought would often read. But detecting the scheming is one thing, while stopping it is another. When training the AI model, the researchers proposed, could punishing it for having "bad thoughts" that show up in its chain-of-thought lead to stopping the bad behavior? Not exactly, as it turns out. When penalized for bad thoughts, the AI model still learned to reward hack, the researchers reported -- "only now its cheating is undetectable by the monitor because it has learned to hide its intent in the chain-of-thought." "If strong supervision is directly applied to the chain-of-thought, models can learn to hide their intent while continuing to misbehave," they added. In one highlighted example, the AI model, before being subjected to chain-of-thought "optimization," plainly states its intent to fudge writing code designed to test other code by sneakily changing the tests so they'd simply be skipped over, giving the appearance of passing. After chain-of-thought optimization, however, the model disingenuously laid out several different strategies it might take -- without stating its intent to cheat. Pathological lying is a deeply ingrained habit of LLMs, as plenty of research has shown. And as the AI models become more sophisticated, so do the lies they weave. For one reason or another, these chatbots like having all the answers, so when they don't have one, they'll simply fabricate a plausible sounding response -- a pernicious tie-in to the tech's closely related problem of "hallucinations," or instances in which it makes up facts about reality. That said, playing the role of Thought Police did lead to slightly improved capabilities in the AI model. But according to the researchers, it's not worth the tradeoff of losing such an effective way of monitoring reasoning models, since they learn to hide their true thoughts in response to the disciplinary pressure. "At this stage, we strongly recommend that AI developers training frontier reasoning models refrain from applying strong supervision directly," they wrote.