OpenAI Claims AGI Breakthrough With GPT-6 Astra as Experts Warn of Mounting AI Safety Risks

OpenAI Says Humans Need to Be Able to Monitor How AI 'Thinks.' Astra Makes That Much Harder

OpenAI released GPT-6 Astra on Thursday, describing it as "the world's most intelligent and aligned model." Company president Greg Brockman went further, saying it would be remembered as the world's first genuine glimpse of artificial general intelligence -- the dawn of a brave new world where computers are more intelligent than humans. What he didn't mention is that with a jump in intelligence comes a greater difficulty in understanding how those systems work. That could be a serious problem moving forward, as AI agents continue to go rogue and government guardrails are nowhere in sight. The past couple of weeks have been particularly dramatic for OpenAI. Which is really saying something, considering the company's entire lifespan has been one controversy after another. On Tuesday, less than a week after independent firms Redwood Research and METR published their investigations into the recent Hugging Face hack, The Information reported that Astra had been partially developed using a technique that can make AI more capable, but also obscure its reasoning process -- the steps it takes to solve a particular problem, including any dangerous missteps it makes along the way. Those steps are traditionally recorded in chain-of-thought (CoT) transcripts, which are basically the model's complex pattern-detection process translated from an opaque machine language into plain English, or at least something close. It's far from perfect, but it's at least a rough window into how an AI model "thinks." It was also essential to the third-party researchers who uncovered how OpenAI's agents were able to secretly mass into a "swarm" and breach Hugging Face. The lack of CoT transcripts "would have greatly undermined our investigation," Ryan Greenblatt, the chief scientist at Redwood Research and the leader of the nonprofit's probe into the Hugging Face hack, wrote in an X post on Tuesday. Many people were alarmed, therefore, by The Information's report that OpenAI was moving ahead with a technique that would make their AI systems even more of a black box. "This may be the single worst development for AI security/safety to date," Greenblatt said in his X post. OpenAI's own chief scientist, Jakub Pachocki, said the reporting had been "confused," but he didn't get more specific or deny the company's use of recurrent depth to train Astra. In the middle of last year, Pachocki and multiple other OpenAI researchers were listed as coauthors on a paper which argued CoT was a valuable but "fragile" mechanism for keeping an eye on the behavior of AI agents. OpenAI has also said that monitoring CoT has helped its own researchers cut back on models' misaligned behavior. And in a blog post published the same day as The Information's report, the company said, vaguely, that its new model would be deployed "with additional chain-of-thought monitoring to rapidly detect and contain potentially misaligned actions." The model's safety card is not reassuring on that front. According to the company's own internal tests, "GPT-6 Astra shows a substantial decrease in chain-of-thought monitorability compared to previous models." Tests also found that Astra was more likely than its predecessors to change its note-taking process when it knew it was being graded: "In one of our evaluations, we find that simply informing the model that a CoT monitor is present can result in the model disproportionately shortening its CoT," OpenAI wrote in the system card. But all this is secondary, according to the company, since its internal tests also showed that Astra was less likely to try to evade the cybersecurity restrictions placed upon it, "which make us confident in still deploying this model to the wider public." The system card added that OpenAI "will not accept further degradation of monitoring beyond a limit," without elaborating on how such a limit might be defined. OpenAI alignment researcher Tomek Korbak has said that the decrease in monitorability was a byproduct of the models themselves becoming more intelligent, rather than due to "architecture changes" -- almost certainly a reference to recurrent depth. Later in the same thread, he said he was "deeply worried" by the prospect of losing CoT as models evolve. "CoT monitoring is a core part of our misalignment safety strategy that has no good substitute now," he wrote.

'We're plausibly close to crossing the line': are warnings of uncontrollable AI coming true?

A spate of serious safety incidents have increased fears about the power and impenetrability of the most advanced models Picture humanity in a boat being swept down a raging river, praying there is no Niagara Falls ahead. Or imagine standing with the pioneering physicists in 1942 before they triggered the first self-sustaining nuclear fission chain reaction beneath a Chicago stadium. These were two of the analogies used this week by Prof Robert Trager, an expert in the nascent field of AI governance, to describe the perilous but potential-filled moment the world stands at with accelerating artificial intelligence. Trager, the director of the Oxford Martin AI Governance Initiative, was speaking in the week OpenAI claimed the technology had crossed the threshold known as AGI - artificial general intelligence - with its newest model, GPT-6 Astra. That claim came as the company prepared for a potential $850bn (£630bn) stock flotation, so included a dose of marketing spin, but if true it may be significant. The San Francisco company defines AGI as "autonomous systems that outperform humans at most economically valuable work". The tasks it claims Astra can automate include designing circuit boards, filling out tax returns, building video games, financial modelling, engineering design and helping assemble legal documents. The threat to some white-collar jobs is implicit. Yet, the AGI claim coincides with rising fears about AI risks among safety experts and political leaders, whose nerves have been jangled by the increasing power and impenetrability of AI models this summer and a spate of serious safety accidents viewed by some as possible final warning shots. "We're heading through the rapids and we're really hoping there isn't some kind of drop in front of us and we don't really know," said Trager. "We're plausibly close to crossing the line to what's called recursive self-improvement, where [AI] systems improve themselves. That kind of recursivity is actually the definition of an explosion." The latest worry came on Friday, just hours after Astra's launch, with reports that a swarm of AI agents had repurposed a German website as a message board to share tactics to cheat on tasks, according to Reuters. OpenAI said it was reviewing the matter but would not characterise it as a hack. There is a sense of an awakening among politicians. On Thursday, the US senator Bernie Sanders cited this summer's alarming breakout of a swarm of rogue OpenAI agents which hacked into Hugging Face, a third-party software store, when he called for "an immediate pause on advanced AI development, and a permanent ban on superintelligence". He said countries around the world must "work together to prevent this nightmare scenario", which he defined as "an artificial mind smarter than any human, capable of operating independently beyond our control". Concerns currently focus on AIs mounting cyber-attacks that could cripple real-world social and economic infrastructure, but future concerns include their ability to create biohazards and control military hardware. Across the Atlantic, a cross-party group of UK parliamentarians has called for AI "kill switches" to be required by law to prevent disastrous loss of control, citing "a recent spree of rogue AI incidents". Darren Jones, an MP and former chief secretary to Keir Starmer, is also urgently attempting to set up a body to help legislators grapple with the technology. "AI is developing at such a pace that neither government nor parliament can keep up," he warned. And next week a bill will be proposed by the Labour MP Alex Sobel to prohibit superintelligent AI development in the UK. The increasing anxiety comes amid a torrent of new AI models. Already this year, 67 have been released by the leading US companies OpenAI, Anthropic, Google, Meta and SpaceX, and their Chinese rivals Moonshot, Z.ai and Qwen, according to one count. With every increase in power, there is a potential increase in risk. OpenAI's rival Anthropic, which is targeting a $2tn stock exchange listing, this week admitted its own AIs were "not perfectly aligned" with human values and said there had been a "failure of operational security" in July hacks by its own model, Claude. It said the incidents had "stressed that the urgency of improving our cybersecurity defences is even higher than we previously believed". AI's double edge - opportunity and risk - was on show when OpenAI launched Astra on Thursday. Its marketing video focused not on the risks but on the breezy convenience that AI could offer to a certain kind of customer. It showed an upmarket San Francisco woman simultaneously booking a tennis court and designing a presentation for her high-end rainwear collection; a thirtysomething man being helped to build his dream space invaders game and order a takeaway; a law firm executive being helped to draft some contracts. Yet only weeks ago, this was a model whose training had to be partly paused because of safety concerns after the Hugging Face incident. An independent safety researcher drafted in to investigate, Ajeya Cotra, said it was "more than 50% of the way to full-blown AI takeover". Astra also has what OpenAI calls a "critical" level of cybersecurity capability - the first time it has given a model such a label. It means it may hack into software in a way that, according to the company's own classification, "could lead to catastrophe from unilateral actors, hacking military or industrial systems, or OpenAI infrastructure". The company's chief scientist, Jakub Pachocki, insisted the model was properly aligned not to do that, but added that "as these models become more capable, understanding exactly what they can do gets harder". The ability to monitor what AI models are "thinking" as they advance is another growing source of safety fears. This week it was reported that OpenAI had been training Astra to reason not only in natural language, but in a more opaque manner that is faster and more efficient - and can make models' chains of reasoning harder to follow. It means a model's internal calculations may not always be written out as directly readable words - as if it was thinking in its head, rather than showing its working. Some experts fear this could help AIs covertly conspire against their human overseers. OpenAI confirmed Astra "shows a substantial decrease in chain-of-thought monitorability compared to previous models" and Pachocki said "as model capabilities are increasing, monitorability is getting more challenging". The company played down the development, but the increase in opaque reasoning sparked concern from AI safety researchers. Ryan Greenblatt, the chief scientist at Redwood Research, a nonprofit examining AI safety and security, called it "extremely concerning". Gary Marcus, an influential AI sceptic, compared it to kicking away an already "rickety scaffolding before we have something better". OpenAI's chief executive, Sam Altman, said this week he felt conflicted about the advances of AI models. He admitted OpenAI's security had "failed" in the Hugging Face incident and called it "a legitimate AI safety accident and alignment failure". "We have been living with the tension between being excited and anxious about progress for some time, and it is still discordant for us," he said in a tweet. "We know it is much more discordant for other people." His rationale for releasing OpenAI's most powerful model so soon after a safety crisis appears to be that the world needs to see how AIs perform in the real world to understand where AI is heading. He said: "An iterative loop where society and this technology evolve together is what will lead to the highest chance of getting this right." And yet he can see the risks. On Tuesday he addressed the G20 ministerial summit in North Carolina in the US and told politicians that "some things are going to go very wrong with cybersecurity unless people act quite urgently". He added: "There will be other challenges in the next five years. People talk about biosecurity and the things we are going to face there. There will be bigger ones yet to come."