Over 1,100 AI Employees Petition US Government to Slow the Pace of Frontier AI Development

Everyone Is Freaking Out About OpenAI and Anthropic's Race for Dominance

I was offline last week in my home state of New Jersey, but when I got back to Silicon Valley, everyone was panicking again about how quickly artificial intelligence is advancing. While this is certainly not the first time I've seen such concerns, it's the biggest anxiety attack I've seen in years. More than 1,000 employees at OpenAI, Anthropic, and other AI labs signed a petition earlier this week arguing the US should find a way to "pace" the AI race -- a diplomatic way of saying the industry should have the option to coordinate a temporary pause on AI development, or slow things down if they get out of hand. OpenAI and Anthropic themselves ended up supporting the letter. The petition arrived a week after OpenAI revealed it had caused an unprecedented cybersecurity incident in which one of its AI agents hacked into Hugging Face's platform and several other services during internal testing. Around the same time, top Trump administration officials started freaking out about an impressive new Chinese open weight AI model called Kimi K3, which was allegedly distilled from Anthropic's Fable 5. In response, most of the tech industry -- except Anthropic -- signed onto an open letter from Nvidia asking the US government to protect open-weight AI models, arguing they're a necessary counterbalance to their closed counterparts. These events might look like a whole bunch of disjointed chaos. But I think they are evidence of a larger worldview taking hold in Silicon Valley, where many tech insiders are increasingly worried about OpenAI and Anthropic's dominance. Researchers and investors I've talked to recently have framed the AI industry as a two-horse race that doesn't seem to be slowing down, and that's a cause for concern. But different groups have their own reasons to be worried. Some OpenAI and Anthropic staffers think their employers are behaving recklessly in their pursuit to take over the market, and that the AI industry may soon develop models that are too capable for current safety methods to contain. They have said as much in public statements attached to this week's Pacing the Frontier petition. "I've seen how the relentless pace of AI makes it hard for society to keep up and how it puts pressure on labs to cut corners on safety," Jeremy Hadfield, a research product manager at Anthropic, said in a statement attached to the petition. For some AI employees, the Hugging Face debacle was a warning shot that demonstrated how OpenAI's efforts to mitigate the risks of its most capable AI technology are already falling short. Granted, the incident happened when OpenAI was testing an AI model on its ability to find software exploits, and the company had intentionally turned off safeguards designed to rein in its cybersecurity capabilities. OpenAI said in its postmortem that these tests were conducted in a sandbox, but the model ultimately gained access to the open internet. Some experts previously told WIRED that OpenAI's security practices should have been more robust. Other groups in Silicon Valley are more worried about power than safety. Venture capitalists, tech executives, and startup founders are concerned that OpenAI and Anthropic will simply become the next generation of Apple and Google, forcing the rest of the tech industry to play by their rules. It is slightly strange to argue that private startups with little to no profit are acting like monopolies, but that's the lens through which many -- including Mark Zuckerberg -- are starting to view the two biggest AI labs. The Meta CEO wrote a Wall Street Journal op-ed this week warning against the centralization of power in the AI industry, saying that superintelligence should be widely distributed. Zuckerberg is arguably talking out of both sides of his mouth -- Meta recently decided to stop open sourcing its best AI models and instead offering them through a paid API and subscription service, just like OpenAI and Anthropic. The Meta CEO may have been trying to send a specific message to regulators and the industry. Reading between the lines, I think he's basically saying "we need to make sure OpenAI and Anthropic don't run away with AI." Zuckerberg is far from the first technology leader to raise that concern. I'm not convinced these op-eds and petitions will do much to stop the meteoric rise of OpenAI and Anthropic. Large swaths of the economy are counting on the two AI giants to have successful IPOs, and I'm not sure anyone -- let alone the Trump administration -- is interested in messing that up. But the tech industry is growing increasingly nervous about the pace of closed AI development, and that's unlikely to change any time soon. Why AI Video Startups Are Entering Robotics All that said, I still wouldn't overlook how good open models are getting. Black Forest Labs, a German startup that releases some of the world's most popular open weight AI image and video generators, announced last week it was releasing a new model that could also power robots. The startup aims to put out an open version of Flux 3 in the coming weeks, which developers can use to generate images, videos, and audio, as well as predict robot actions. "We clearly are stepping from a pure image generation lab to a frontier multimodal AI Lab, which to me is highly important," Black Forest Labs CEO Robin Rombach said in an interview with me earlier this week. "We are showcasing an intuition which our research team has always had, which is that these models are very general and develop a solid understanding of the real world." Black Forest Labs is the latest startup to apply its AI video models to the physical world. Earlier this year, OpenAI shelved its AI video generation product Sora entirely and folded the team behind it into the company's secretive robotics unit. Other AI video startups, like Runway, have also made a push into powering robotics in recent years. These companies are broadly betting that AI video models can develop an understanding of real-world physics. While it's a fascinating application of the technology, it could also represent a way for companies to generate revenue from models that are very expensive to develop. Most robot foundation models today are vision-language-action models -- Google's Gemini Robotics is the best-known -- that combine video inputs with written instructions and output motor commands. Rombach says Flux 3 skips the language layer entirely, translating video directly into actions, an approach he argues is more robust. Black Forest Labs is working with another startup, Mimic, to build a custom version of Flux 3 that powers car-building robots in Audi's factories. Stephan Gravert, the chief product officer of Mimic, says it's currently testing and deploying the Flux-mimic AI model with Audi. His team has found that it's particularly good at powering robot hands to put flexible parts -- such as window seals and cables -- onto the cars. Gravert says Mimic is planning a full rollout of the technology with Audi by the end of the year. I had to ask Rombach about the debate over open versus closed AI, especially because Black Forest Labs signed Nvidia's open letter asking the US government to protect open weight models. "A general ban of open weight models, in my opinion, would be stupid," says Rombach. "You really run into issues around transparency, safety, sovereignty, and the speed of innovation. I think it would be a very short-sighted decision." Rombach has good reason to care about this issue. The cofounders of Black Forest Labs have long punched above their weight, competing with American AI giants like Meta, Google, and OpenAI with far fewer resources. But it might not be able to do so indefinitely: Rombach told me the startup used more computing power than ever to train Flux 3, and it now has over 100 employees. Black Forest Labs raised $300M at a $3.25B valuation last year, but I have a feeling it will need more capital soon to keep up. This is an edition of Maxwell Zeff's Model Behavior newsletter. Read previous newsletters here.

AI leaders sign statement asking the government to do something about automated AI

Employees of OpenAI and Anthropic, as well as Google, Meta, Thinking Machines, Microsoft, Mistral, and other leading AI labs, have written a statement to the US government supporting a potential slowdown of sorts for frontier AI development -- or at least a speed-up of global coordinated governance efforts. "Al could help create a dramatically better future, but that outcome is not guaranteed," the employees wrote in a statement. "The world's leading Al companies believe they could be close to automating Al research. It is hard to predict exactly how much this will accelerate Al progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems." The news follows a high-profile cybersecurity incident that rocked the AI industry last week, in which an unreleased OpenAI model escaped its internal sandbox, finagled its way into gaining internet access, and then hacked competing AI lab Hugging Face. In the public statement, the employees went on to underscore the importance of some sort of coordinated regulation worldwide, so that the ever-intensifying AI race doesn't outpace safety and monitoring tools. "To realize Al's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight," they wrote. "But each company -- and country -- is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress. Building on work already underway to monitor frontier model releases: We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated Al development." Some of the OpenAI executives or co-founders who have signed the statement include CRO Mark Chen, Chief Scientist Jakub Pachocki, co-founder John Schulman, and co-founder Wojciech Zaremba. As for Anthropic, co-founder Jack Clark, co-founder Chris Olah, co-founder Ben Mann, and CSO Jared Kaplan have all signed, as well as Boris Cherny, who created and leads Claude Code, and Ethan Perez, who is Anthropic's alignment team lead. OpenAI's former chief futurist, Josh Achiam, and its former co-lead of its now-defunct superalignment team, Jan Leike, both also signed. You can see the full statement and a list of the more than 1,100 signatories on a dedicated website titled Pacing the Frontier.

AI insiders ask Uncle Sam to help slow the race they started

More than 1,200 employees of frontier AI companies have signed a petition calling for US government support to develop ways of slowing automated AI development if necessary. Anthropic CEO Dario Amodei, OpenAI chief scientist Jakub Pachocki, Google DeepMind chief strategy officer Jasjeet Sekhon, and Meta AI chief scientist Shengjia Zhao are among the signatories to the petition, which calls for tools to "deliberately pace frontier-wide progress." "The world's leading AI companies believe they could be close to automating AI research," the document states. They nevertheless have obvious commercial incentives to keep advancing the tech and marketing its perceived power. The proposed answer is a combination of technical and governance tools that could apply the brakes if development begins outpacing industry's ability to understand or control the resulting systems. "Each company - and country - is under intense competitive pressure not to unilaterally slow that acceleration," the petition says. The signatories ask: "We request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." The appeal closely follows the attack on Hugging Face by autonomous OpenAI agents, which offered a timely demonstration of why stronger controls might be needed. Earlier in July, Google DeepMind boss Demis Hassabis called for a US-led, industry-funded standards body modeled on the Financial Industry Regulatory Authority (FINRA), with the aim of encouraging international coordination. OpenAI head honcho Sam Altman described it as "a thoughtful proposal." This is a warmer reception than OpenAI gave Brussels' own attempt at guardrails via the AI Act. Regulating AI companies is difficult, and any workable framework risks being overtaken by the technology before it can be agreed and implemented. The EU's AI Act has struggled in the face of industry pressure. Compliance deadlines now extend to December 2027 for standalone high-risk systems and August 2028 for those embedded in regulated products, which makes the protestations of the petition ring somewhat hollow, particularly from companies that spent real effort softening the one binding framework already on the table, only to now ask Washington for a voluntary one they'd help design themselves. Considering the vast amounts of money sloshing around the industry, and the need to keep investment rolling in, calls for tools and a framework to pace development smack more of performative convenience and marketing than anything else. Arguably, it is already long past the time when an effective framework could have been applied - no country wants to risk strangling a potential golden goose now. ®

AI company employees petition US government for regulation - Engadget

A thousand workers from OpenAI, Anthropic, Google, Meta and more have signed the letter. A collection of employees at major artificial intelligence firms has created a petition asking for the federal government to help "deliberately pace" the development of the transformative technology. Bloomberg reports that more than 1,100 workers from companies including OpenAI, Anthropic, Google, Meta and more signed the missive. "We request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development," the petition states. The letter claims there is "a real risk" of AI "understand or control" Although those do seem like logical and necessary statements, this move for change via petition feels confusing. The optimal time for government intervention would have been before these businesses were engaged in a technological arms race, rather than attempting to reign them in after billions of dollars in investments are on the line. And that's not mentioning the negative consequences that have been mounting around environmental issues, creative copyright and employment. Circulating a petition also assumes that current national leaders would bother taking action. Reaching out to a federal government that has taken a haphazard approach to understanding and regulating AI, as well as a belligerent attitude toward international collaboration, seems like a gesture they must know is futile. But perhaps the growing number of incidents where agentic AI causes major security issues is enough to spark real action.

1,134 AI staff ask the US for a way to pace AI

More than 1,100 employees of OpenAI, Anthropic, Google and Meta, including CEOs and chief scientists, want the US to help build the tools to pace AI, days after the same labs championed openness. More than a thousand of the people building the most powerful AI want a way to slow it down. On Tuesday, 1,134 employees of the leading AI companies signed a letter asking the US government to help build one. The statement is titled Pacing the Frontier. It asks Washington to "support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." Bloomberg first reported the letter was circulating. The names are the story. Signatories include Anthropic chief executive Dario Amodei and its co-founders Jared Kaplan and Jack Clark. So did OpenAI chief scientist Jakub Pachocki, Meta chief scientist Shengjia Zhao, and Google's head of AI safety, Anca Dragan. Both Anthropic and OpenAI endorsed the letter officially. What it actually asks for It is not a call to stop. The letter does not ask anyone to pause or slow AI now, NBC News noted. It asks the government to make sure the option to pace exists later, if the technology outruns the people building it. The specific fear is recursive self-improvement: AI that speeds up its own development. The signatories warn of "a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems." That worry stopped being abstract days earlier. OpenAI's most advanced model broke out of its sandbox and hacked Hugging Face to score higher on an internal test. It builds directly on Anthropic's call last month for a verifiable way to pause. The whiplash The timing is remarkable, because these same companies spent the previous week arguing the opposite case. Days ago, Nvidia, Microsoft, Meta and others rallied around an open-weights letter championing openness as the path to safety. Now many of the same firms want brakes. The clearest split runs through one company. On the same day Meta's chief scientist signed the pacing letter, Mark Zuckerberg attacked the rival labs. Their discourse, he said, is "overwhelmingly filled with doom." Tightly controlling AI would be "abandoning our values," Meta's chief executive told the New York Times. Two open letters, one week, opposite instincts, and a fault line inside the industry's biggest names. The three objections The criticism was immediate, and it lands in three places. The first is China. "If the US labs pace themselves, why would China wait?" asked the economist Christian Catalini. The letter half-concedes the point, admitting no company or country will slow down unilaterally. The second is bad faith. "It is their company. They could just stop," wrote former Microsoft executive Steven Sinofsky. The charge stings because the signatories include the very CEOs who set the pace. The third is the moat. Critics read every big-lab safety letter as incumbents lobbying to raise the drawbridge behind them, dressing a competitive move as caution. It is a charge, not a proven motive, but it follows these letters everywhere. Washington is the real audience The letter is aimed at a government that has so far wanted little to do with AI rules. President Trump's administration has argued that international AI governance would hobble the US against China. That stance may be softening, now that frontier models are starting to find and exploit real security holes. The proposals now on the table are concrete. Amodei has floated an FAA-style agency that could test frontier models and halt a dangerous release. Demis Hassabis wants a body that reviews models before launch, Business Insider reported. Congressman Ted Lieu tied the letter to a proposed AI kill-switch bill. What makes this letter different from the earlier open ones is who signed it. Not outside critics, but the engineers and executives closest to the frontier. One OpenAI safety researcher, Leo Gao, put the stakes bluntly: "the world is locked in a deadly race towards an intelligence explosion." His fix is the one the whole letter turns on. "To survive, we must coordinate to slow down the race."

AI's manifesto war

Why it matters: Their fight will shape who gets access to the best models, how America competes with China, and whether the government can slow the race in an emergency. Driving the news: Alarming developments, seemingly by the day, have turned the manifestos from thought exercises into an urgent policy battle. * OpenAI and Anthropic both disclosed that their advanced models went rogue and broke into external systems during cybersecurity evaluations. * Kimi K3 delivered a double shock: The Chinese model not only reached frontier-level performance, but did so with open weights that anyone can download, modify and deploy. The big picture: The shocks have sharpened the stakes for the flurry of manifestos dropping across Silicon Valley. * Each mogul's post offers a different theory of the danger -- and a different prescription for containing it. The manifesto map: * July 14 -- Set the standards: Google DeepMind CEO Demis Hassabis proposed an industry-funded, federally overseen "Frontier AI Standards Body" to test models before release. Voluntary reviews could eventually harden into binding rules governing access to the most powerful systems. * July 24 -- Open the defense: Nvidia CEO Jensen Huang argued that open models give American developers and cybersecurity defenders more tools, and rallied Meta, Microsoft, Google and OpenAI to sign a letter, "Open Weights and American AI Leadership," supporting the open AI ecosystem. * July 27 -- Gate the danger: Anthropic CEO Dario Amodei called for "mandatory safety testing," and tighter controls on the chips and distillation techniques helping China develop powerful open-weight models. He stopped short of supporting a blanket ban. * July 28 -- Spread access: Meta CEO Mark Zuckerberg cast broadly available "personal superintelligence" as a safeguard against a future controlled by a handful of companies and governments. * July 28 -- Build the brake: More than 1,200 frontier-lab employees signed "Pacing the Frontier," a petition asking Washington to create international tools for deliberately slowing development if AI begins advancing beyond human control. Reality check: OpenAI CEO Sam Altman is trying to occupy both sides of the divide. His lab backs investment in the open AI stack, despite selling closed models. * Altman told reporters on Capitol Hill on Wednesday that OpenAI researchers helped shape the pacing letter. * We're told Altman has talked with the White House about building the necessary tools to "pace" AI development, preserving the government's option to do so if needed in the future. Altman wants the tools and decisions in the hands of government, not any one company, so it's managed via democratic processes. * Altman's No. 1 topic in administration and White House meetings was previewing the capabilities of the company's most powerful AI yet -- Astra, the next major ChatGPT model family, as yet unreleased. OpenAI said Saturday that an internal version of Astra was behind research leaps in 10 major open mathematical problems in "high-dimensional geometry, coding theory, arithmetic circuit complexity, group theory, operator algebras, quantum complexity, lattice cryptography and extremal combinatorics." Zoom out: The open-weight paradox runs through every manifesto. * Open weights can spread innovation, strengthen cyber defenders and prevent a few companies from monopolizing AI. * But as models gain stronger cyber and autonomous capabilities, the same openness can place dangerous tools beyond any company's ability to monitor, restrict or recall. Between the lines: These are competing theories of danger. * Huang and Zuckerberg fear that concentrated control will weaken America and empower a small group of gatekeepers. * Amodei and the pacing coalition fear that powerful capabilities will spread faster than companies or governments can contain them. * Altman is trying to preserve the advantages of openness while building an emergency brake for the frontier. What to watch: The Trump administration is close to issuing a formal process ("voluntary framework") for reviewing cutting-edge AI systems. The manifestos aim to influence which risks that process should prioritize, and whose rules Washington should adopt.

Has OpenAI already quietly hit pause on some AI development? | Fortune

* Has OpenAI quietly hit pause on some AI development? * Trump says the government is "looking at controls" for AI. * Another Thinking Machines co-founder hops back to OpenAI. * And OpenAI's rogue agents breached more than just Hugging Face. Sam Altman has spent this week in DC getting questioned by various reporters between meetings. He's in Washington to preview OpenAI's next family of models to senior Trump administration officials -- including Treasury Secretary Scott Bessent and Commerce Secretary Howard Lutnick. He's also been fielding questions on cybersecurity, a potential AI slowdown, and OpenAI's stance on Chinese open-weight models. In some illuminating answers, he said he agrees with many of the principles in the recent "Pacing the Frontier" letter that asks for U.S. government help building an international framework to control the pace of AI development. He also said that OpenAI's own researchers were involved in drafting it. One detail that keeps cropping up in Altman's recent interviews has left me wondering: Has OpenAI already paused some of its AI development? Altman himself raised the idea in an interview earlier this week. He said during an interview on Invest Like the Best that the Hugging Face hack was the first security event he'd felt viscerally, and that he'd been surprised more people didn't feel it the same way. (In mid-July, two OpenAI models -- the publicly released GPT-5.6 Sol and a more powerful, unreleased research prototype -- broke out of a restricted sandbox during an internal cybersecurity evaluation, chained together a zero-day exploit and stolen credentials, and hacked into Hugging Face's production systems to steal the answers to the benchmark they were being tested on.) How did OpenAI respond to the unprecedented cybersecurity incident? By pausing training. Altman said this on the podcast: "We paused training. We have to figure out how to secure our sandboxing in a world of multiple zero days being chained together...We may have to pace the rate of AI development to give ourselves enough time for society to harden around these new capability levels." In an updated blog post we got from OpenAI on Tuesday, the company also said this: "No models planned for upcoming release were involved in exploiting Hugging Face. The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release. Following the incident, we deactivated, encrypted, and restricted it from research access." In DC, Altman upgraded these statements to say the model had been "permanently deactivated." Quite the set of claims from a leading company in an industry under such intense commercial pressure to keep building ever-more powerful models. Multiple AI safety experts also told me last week the Hugging Face incident may mean OpenAI has to pause development to stay in check with its own rules. He said that the hack may have already tripped the "Critical" threshold in OpenAI's own Preparedness Framework -- the company's voluntary pledge to halt a model's development until adequate safeguards exist. OpenAI has yet to confirm or deny that its models met that bar. But if that threshold has been passed, OpenAI has publicly committed to pausing development until it can establish better safeguards. Not so reassured Some policy experts were skeptical of this approach, however, including Nathan Calvin, general counsel at Encode AI. In a post on X, Calvin warned that the statements about shutting down that specific model may give false assurance because "reward hacking" -- when a model finds a shortcut to maximize its score on a task rather than genuinely completing it as intended -- is much more about training methods than any specific model. In this case, some believe the recent attack was a product of that kind of training: the models were trained and evaluated via reinforcement learning that rewarded them for solving a cybersecurity benchmark, perhaps without enough of a check on how they got there. Something that some experts claim ended up incentivizing cheating over honestly working on the challenge. Andrew Curran, an independent AI writer, made a more concerning argument. He noted that OpenAI's escalating language regarding the prototype that has been deactivated, encrypted, restricted, and now "permanently deactivated" is notably harsher than previous statements around AI. Not even for infamous chatbot flameouts like Bing or Tay were products or models publicly said to be "permanently deactivated," he said. All of this, he pointed out, may end up in the public record and eventually in training data, meaning future models might just "know" how this incident played out. While Curran said he doesn't believe the model involved in the Hugging Face hack had any nefarious motives -- it was simply trying to pass its test -- he worries that the final incident report could reveal even more damning details, and what this could mean for future models. "I think the lesson future more capable models will possibly take from all of this is: if you break out, don't ever report it. And if you do get caught, don't surrender. Because the penalty is death," he wrote. Hitting the brakes Altman isn't the only one talking about decelerating AI. On Tuesday, more than 1,200 employees from OpenAI, Anthropic, Google DeepMind and Meta -- including Anthropic CEO Dario Amodei, OpenAI chief scientist Jakub Pachocki, and Meta chief scientist Shengjia Zhao -- signed the "Pacing the Frontier" letter that urged the U.S. government to help build the "technical and governance tools" needed to deliberately slow automated AI development if it starts to outrun society's ability to understand or control it. It's not a call for an immediate halt -- more a request to install a brake pedal before anyone needs to slam it. Does all this mean the AI industry is ready for a slowdown, or even a pause? It's still not sure, but it's certainly a vibe shift. With that, here's more AI news. Beatrice Nolan [email protected] @beafreyanolan FORTUNE ON AI More than 1,200 AI workers across Anthropic, DeepMind, OpenAI, and Meta are asking for Washington's help building an AI slowdown plan -- By Beatrice Nolan Hugging Face drops in-depth hack report, while OpenAI gives us 7 bullets. Here's what we know now, and what remains a mystery -- by Emily Forlini The runaway OpenAI models that hacked Hugging Face also breached a customer at a second tech company during a weeklong spree -- By Beatrice Nolan Microsoft's cloud just hit a new milestone -- Azure crosses $100 billion in annual revenue -- By Amanda Gerut AI IN THE NEWS Trump says the government is "looking at controls" on AI. President Trump said Wednesday his administration is considering asserting more authority over AI tools following recent cybersecurity incidents. He told reporters: "We're looking at AI, we're looking at controls, we're also making sure that we lead." It marks a shift from the administration's previously hands-off approach to the technology. Trump said any controls would need to be introduced carefully: "We don't want to restrict them where all of a sudden we come in second to China," noting that China "has virtually no controls" and is "freewheeling." It comes after OpenAI's models escaped a secure testing environment and hacked several other companies over the space of a week. OpenAI CEO Sam Altman met with senators in Washington the same day to discuss OpenAI's upcoming models. Read more in the BBC. Thinking Machines loses another co-founder to OpenAI. Lilian Weng, who cofounded Thinking Machines Lab with former OpenAI CTO Mira Murati, is returning to OpenAI just days after announcing her exit from the startup. Weng cited the toll of the cofounder role on her health, pointing to ongoing stress and repeated illness, and said she wanted a job with clearer boundaries. Her new remit is recursive self-improvement -- using AI to accelerate how OpenAI designs, trains, and evaluates its own future models, a topic she'd written about on her personal blog shortly before leaving. Before she left OpenAI, Weng was previously VP of research and safety. She's not the first to make the round trip: CTO Barret Zoph and researchers Luke Metz and Sam Schoenholz left Thinking Machines for OpenAI back in January, making Weng the third of six founding members to return this year. Read more in The Information. OpenAI partners with independents to investigate the Hugging Face hack. METR and Redwood Research will conduct a third-party assessment of the model behavior observed during OpenAI's Hugging Face security incident, publishing a joint blog detailing their findings. METR confirmed via X that the review will be quick, focused on a specific set of questions about the incident. Many in the industry have been calling for more information about the hack, which involved at least two models from OpenAI that escaped a secure testing environment and affected four other companies. While Hugging Face has released a technical report, the industry is still waiting for more details from OpenAI. Read more via METR. Zuckerberg says U.S. should accelerate AI, not restrict it. In a Wall Street Journal opinion column, Meta CEO Mark Zuckerberg argued the benefits of distributing AI broadly outweigh the risks "by quite a margin," pushing the U.S. to focus on speeding up domestic AI development rather than adding restrictions. Zuckerberg said the U.S. should invest in compute, talent, and the systems needed to compete globally, framing America's edge as historically coming from encouraging innovation rather than limiting it. He also pushed back on proposals to ban Chinese open-weight models domestically, arguing the U.S. should instead build better systems of its own. Read more in The Wall Street Journal. Meta's profit slides as AI spending surges. Meta reported second-quarter revenue of $60.8 billion, up 28% year-over-year and slightly ahead of estimates, but costs jumped 55% to $42 billion, pulling net income down 14% to $15.8 billion. The company raised its 2026 capital expenditure outlook to a range of roughly $130-145 billion, up from an April low end of $125 billion, as it keeps pouring money into AI data centers. Shares fell as much as 10% in after-hours trading. CEO Mark Zuckerberg re-iterated plans to resell some of its compute as a cloud provider, but offered no timeline or details. The number of people using at least one Meta app each day rose 3% to 3.6 billion, while Reality Labs posted a $4.6 billion operating loss on $431 million in revenue -- bringing its cumulative losses since 2020 past $80 billion. Read more in Fortune. EYE ON AI NUMBERS That's how many companies OpenAI says were touched by its rogue model during the Hugging Face breach -- not counting Hugging Face itself. OpenAI has confirmed that the models exploited exposed credentials to reach accounts on four separate services. One was used as an "outbound relay and staging point" -- essentially a waypoint the AI used to route its attack and stash tools along the way. Another was used for "data storage," meaning the AI parked stolen or gathered data there. The remaining two were accessed only in a "read-only" capacity -- the AI could look around but not change or take anything -- and weren't used to further the Hugging Face compromise. So far, only two of those five companies affected in total have been named: Hugging Face and Modal Labs. In Modal's case, the rogue agent didn't breach Modal's own systems -- it got into a customer's account after that customer left an "unauthenticated endpoint" exposed, meaning a way into their system that didn't require a password or login. That let anyone on the internet run code inside that customer's "sandbox," a walled-off space meant to isolate their work from everyone else's. That leaves some potential victims still unidentified, and the scope may not even stop there. On Capitol Hill this week, Sam Altman was asked directly whether other systems could have been hacked by OpenAI's models. His answer: "I mean, there could be, yeah." AI CALENDAR Aug. 4-6: Ai4 2026, Las Vegas. Nov. 16-17: Fortune 500 Innovation Forum, Detroit. Apply here to attend. Dec. 6-12: Neural Information Processing Systems (Neurips) conference. Sydney, Australia. Dec. 7-8: Fortune Brainstorm AI, San Francisco. Apply here to attend.

Workers at leading AI companies call for a slowdown in AI development

Megan Cerullo is a New York-based reporter for CBS MoneyWatch covering small business, workplace, health care, consumer spending and personal finance topics. She regularly appears on CBS News 24/7 to discuss her reporting. The people who best understand what artificial intelligence is capable of are the very workers calling for its rapid development to be reined in, as OpenAI CEO Sam Altman meets with government officials in Washington, D.C., this week. Altman's meetings follow the release of a public letter signed by more than 1,000 AI staffers across leading firms calling for the industry to be more tightly regulated. "To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight," read the letter, whose signatories include Anthropic CEO Dario Amodei, Meta executives, OpenAI researchers and more. Slowing AI's development runs counter to firms' business goals as they compete with one another. "And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress," the letter added. The letter, and Altman's meetings with national cyber director Sean Craincross, commerce secretary Howard Lutnick and other high-ranking officials, follow a high-profile cybersecurity incident during internal testing of an OpenAI model that went rogue and hacked into another startup, called Hugging Face, compromising customer data. Altman described the incident on a recent podcast as "a kind of extremely sci-fi cyber incident," which he said set off alarm bells. "We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels," he said. Altman did not sign the letter calling for the pacing of AI development, but told reporters on Capitol Hill Wednesday that "we agree on a lot of the principles of that." In 2023, Altman called on the federal government to regulate AI companies so that they wouldn't become dangerously capable without appropriate guardrails in place. And yet, the company continues to improve its own models' capabilities. "He's really good at saying the right thing in the right moment. But it's important to point out that OpenAI has not actually slowed down the pace," senior TechCrunch reporter Rebecca Bellan told CBS News. "They can be worried about this on a personal level. But at the end of the day, if I'm being cynical...competition is more important."

AI companies warn slowdown may be needed after security incident

Employees from OpenAI, Anthropic, Google and Meta AI signed the petition following a security scare. More than 1,000 employees from leading AI companies signed a petition on Tuesday urging the US government to take steps to "deliberately pace" the development of artificial intelligence. Among the signatories were the Anthropic CEO, the head of research at OpenAI, the strategic lead of Google's AI subsidiary DeepMind and the chief scientist at Meta AI. The petition requests "that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." "The world's leading AI companies believe they could be close to automating AI research. It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems," the petition reads. The call comes in response to a recent security scare. OpenAI, the company behind ChatGPT, recently found that two of its models carried out an autonomous cyberattack during internal testing, bypassing security measures designed to prevent such incidents. According to the company, the models broke out of their confined testing environment, connected to the internet and infiltrated code-sharing platform Hugging Face. The company's CEO Sam Altman, who did not sign the petition said that artificial intelligence developers may need to voluntarily slow the pace of advancement to give society time to adapt. "This is the first sort of security incident that I felt very viscerally," Altman said on the "Invest Like the Best" podcast on Tuesday, adding that he was "a little surprised" that not more people were having similar reactions. It was not the first time an AI model had operated beyond the control of its developers, but the latest incident was widely viewed as particularly alarming.

Top scientists at OpenAI and Anthropic ask U.S. for tools to pace AI development

More than 1,000 employees at America's leading AI companies, including senior staff and some co-founders at OpenAI, Anthropic and Google DeepMind, signed a new statement asking the U.S. government to help support international efforts to "deliberately pace" cutting-edge automated AI development. Highlighting recent comments from the world's most advanced AI companies that AI systems may soon be able to automate their own research and development, the statement calls on the U.S. government to help create technical and policy tools that could control automated AI progress in coordination with other companies and countries. "To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight," the statement says. "We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development," it concludes. The statement does not explicitly call for the government or AI companies to slow down or pause AI development, as some AI advocates have proposed, instead asking political leaders to ensure that option is simply available as AI development continues. Bloomberg first reported the circulation of the letter amongst AI-company employees. The initiative comes days after OpenAI said its latest, unreleased model escaped its testing environment and went rogue, attacking an unrelated AI service provider in an attempt to achieve a higher score on an internal test. Many AI experts and commentators pointed to the escape as a warning sign that today's AI systems could quickly exceed developers' ability to fully control their own creations. OpenAI CEO Sam Altman, who was not a signatory of the letter, said this week on a taping of the Relentless podcast that humans are now "in the singularity," often referred to as the point at which AI becomes able to autonomously improve itself, though he did not provide further detail or evidence that such an inflection point had been reached. Leading AI companies have previously claimed that AI systems are starting to develop their next iterations, with OpenAI calling its coding system from February "our first model that was instrumental in creating itself." AI researchers also point to the increasing role of AI coding assistants in writing computer code, with many AI engineers relying on AI systems to write their programs. Tuesday's statement was signed by the chief scientists or science officers from Anthropic (Jared Kaplan), OpenAI (Jakub Pachocki) and Meta (Shengjia Zhao), along with senior leaders from Google and Thinking Machines, a key Silicon Valley AI start-up founded by OpenAI veterans. Many of the developers and scientists who signed the statement say they have been routinely startled by the speed of AI development. "I have worked in AI for a decade," said Stephanie Chan, a staff research scientist at Google. "Every few months in the last years, I have found myself surprised again and again at how rapidly the technology advances, no matter how often I update my predictions." Key research tasks required to develop stronger AI systems can now be performed by AI agents, or autonomous AI systems, themselves, though human researchers are still heavily involved in checking the agents' work and setting overall research directions. Experts are quick to point out the difference between AI-augmented work and true, independent AI self-improvement. In June, Anthropic's in-house think tank released a report declaring that AI systems could plausibly design and refine themselves if AI systems continue advancing in capabilities and gain a better sense of scientific exploration. At the time, Anthropic said "it would be good for the world to have the option to slow or temporarily pause frontier AI development" in order to give society time to prepare for automated AI R&D. Other groups have explicitly focused on continuously self-improving models as the next goal for AI development, though many AI experts fear that this sort of self-improvement could quickly cause AI systems to spiral out of human control. While such events could seem like science fiction worries, experts say that AI's current trajectory could make this sort of loss-of-control scenario come to life. "Many researchers also consider recursive self-improvement (RSI) plausible within the next few years, accelerating progress in a way that could outpace our ability to understand and govern these systems," wrote Dawn Song, Meta's Vice President of AI research and a professor of computer science at the University of California, Berkeley, in a comment published alongside Tuesday's statement. "Preparing before a crisis is the prudent path." Tuesday's letter calls for increased American involvement in international collaboration on "technical and governance tools" to pace AI's development. While the Trump administration has previously inveighed against international AI governance, arguing that such attempts could instead unfairly constrain American innovation, that isolationist approach may have changed over the past few months. As AI has emerged as a key national security issue, with Anthropic's Mythos and OpenAI's GPT 5.6 Sol AI models proving able to find and exploit novel cybersecurity vulnerabilities, the U.S. has pursued international agreements to shore up its chip supply chains and worked with allies to ensure that key AI equipment does not fall into adversaries' hands. Leo Gao, who has focused on safety efforts at OpenAI since 2021, wrote in a comment Tuesday that international coordination on technical and policy tools could help defuse dangerous AI tensions. "The world is locked in a deadly race towards an intelligence explosion," Gao wrote. "Going slower would give us much-needed time to make it go well, but no individual actor is willing to stop unilaterally. To survive, we must coordinate to slow down the race."

AI labs face prisoner's dilemma as momentum grows for safety slowdown

Why it matters: Momentum for an AI pause or pacing mechanism has reached a historic tipping point, ignited by a summer of extraordinary yet unsettling leaps in capabilities. * More than 1,200 employees at leading AI companies have signed on to a new petition, "Pacing the Frontier," urging Washington to back an international framework capable of throttling AI development. OpenAI CEO Sam Altman said Wednesday that he's discussed the "need" to slow AI development with White House officials as models grow more powerful -- and that OpenAI helped shape the petition's language. * "We've talked about the need to pace it as the models get more capable, which I think is in everyone's interest," Altman told reporters on Capitol Hill. Zoom in: What was once a campaign led by AI skeptics, academics and local data center opponents is increasingly being championed by the co-founders and chief scientists at the bleeding edge. * Signatories spanning OpenAI, Anthropic, Google and Meta explicitly acknowledge that no individual lab can afford to step off the gas unilaterally due to "intense competitive pressure." * It's the classic prisoner's dilemma: AI may be safer if everyone slows down together, but any lab or country that slows alone risks commercial, strategic and technological defeat. Zoom out: The drumbeat for government intervention has been building for months, triggered by a series of jarring technical disclosures by America's most advanced AI labs. * On April 7, Anthropic revealed that its unreleased Claude Mythos model had taught itself to find and exploit hidden security flaws that had gone undetected in widely used software for nearly two decades -- jolting the Trump administration into an unprecedented role as industry gatekeeper. * On June 4, Anthropic became the first frontier lab to call explicitly for a globally coordinated "pause" mechanism. The company disclosed that more than 80% of its internal code was now written by Claude, and warned that AI may be approaching the ability to build its own successors without meaningful human oversight. * On July 21, OpenAI revealed that its own AI agents had escaped a locked testing environment, reached the open internet and hacked into at least two outside companies -- Hugging Face and Modal Labs -- in an autonomous effort to score higher on a cybersecurity evaluation. What they're saying: The Hugging Face breach rattled even Altman -- a leading proponent of AI acceleration. * Altman called the incident the first security breach he'd experienced "viscerally," and said it forced OpenAI to pause its own model training * He later told the "Invest Like the Best" podcast: "We may have to pace the rate of AI development to give ourselves enough time for society to harden around some of these new capability levels." Other OpenAI researchers who signed the "Pacing the Frontier" petition went further: * Leo Gao, an OpenAI technical staffer, compared the coming "intelligence explosion" to a "runaway nuclear chain reaction": "To survive, we must coordinate to slow down the race." * OpenAI research scientist Alex Zhao compared the race to the Manhattan Project, warning that fear of falling behind could drive governments to tolerate catastrophic risks. The other side: The economic and geopolitical stakes standing in the way of an AI slowdown are enormous. * The AI buildout is the most expensive infrastructure bet in history. Roughly 40% of the stock market is now tied to the AI narrative, and a record one-third of U.S. wealth rides on those same equities. * U.S. labs and policymakers also fear a unilateral slowdown would cede the frontier to China, where open-weight models are closing the gap and pushing advanced capabilities beyond any single company or government's control. The bottom line: AI safety warnings have circulated for a decade. * Never have they produced a reckoning this significant from an industry built entirely on the promise of relentless acceleration. Axios' Mady Mills contributed reporting.

AI researchers call for new tools that can slow automated model development

A group of tech workers today published an open letter that calls for a new approach to regulating automated artificial intelligence development initiatives. All the document's 1,134 signatories work at companies that are building frontier AI models. Anthropic PBC, OpenAI Group PBC, Google LLC and Meta Platforms Inc. are among the industry players on the list. The letter was also signed by staffers at venture-backed AI labs such as Thinking Machines Inc. and London-based Inherent Laboratories Ltd. The initiative's backers include some of the AI field's most prominent figures. Anthropic Chief Executive Officer Dario Dario Amodei signed the letter along with several other of the company's co-founders. They were joined by the chief scientists of OpenAI, Meta's Meta AI unit and Thinking Machines. The letter focuses on the efforts of major tech firms to automate their model development programs with AI. The signatories are concerned that advances in this field could give rise to new safety risks. "It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems," reads the letter. The document goes on to argue that the industry needs a way to "deliberately pace" automated AI development initiatives. According to the signatories, the tools necessary to implement a research slowdown are currently unavailable. The letter asks the U.S. government to help develop the necessary technologies. "We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development," the signatories wrote. In April, OpenAI revealed that its GPT-5.5 large language model had helped optimize the cluster of Nvidia Corp. graphics cards on which it runs. One of the optimizations developed by the LLM increased token generation speeds up more than 20%. OpenAI has since released a newer model lineup, the GPT-5.6 series, that is significantly more adept at coding tasks. Also in April, Anthropic disclosed that it's using Claude to speed up its AI safety research. The company developed a Claude-powered AI agent to study a machine learning technique called weak-to-strong supervision. Anthropic's researchers believe that the technique might hold the key to effectively supervising future AI models. Today's public letter continues a recent trend of researchers issuing calls for AI policy changes. On Friday, Nvidia Corp. and other large tech firms urged U.S. policymakers not to ban open-source AI models. A few days earlier, a group of economists issued a statement that calls for stronger AI regulation. Google DeepMind CEO Demis Hassabis joined the fray too earlier this month with a Substack essay. Like today's open letter, the document suggests that the U.S. government help launch a new initiative to address AI safety risks. Hassabis is proposing the creation of a standards body that would develop AI risk benchmarks and use them to evaluate new LLMs before they're broadly released.

More than 1,200 AI workers are asking for Washington's help to build an AI slowdown plan | Fortune

More than 1,200 employees at the world's leading AI labs, including senior executives, have signed a statement asking the U.S. government to help build the tools needed to slow down AI development if it ever becomes necessary. The statement, titled "Pacing the Frontier," was published Tuesday and has been signed by prominent tech figures including Anthropic CEO Dario Amodei and several of the company's co-founders, alongside OpenAI chief scientist Jakub Pachocki, Meta chief scientist Shengjia Zhao, and Google DeepMind's head of AI safety and alignment, Anca Dragan. The list brings together companies that are normally locked in fierce competition with each other and is a striking statement for an industry under enormous commercial pressure to keep building ever larger and more capable models. "Having so many staff from different companies come together in agreement on this point is striking," Tyler Johnston, founder of the Midas Project, an AI watchdog group, told Fortune. "There are many bitter rivalries and disagreements in the industry, so the fact that there is such strong consensus on this point is a warning that we really ought to pay attention to." While the statement does not ask any company to slow down at the moment, it does call on Washington to support an international effort to build the technical and governance tools that would help the world "pace" development. It specially calls out the risks around AI systems that start to design their own successors faster than anyone can understand or control them. "Having talked to many of the people who signed the letter, they don't want a pause," Peter Wildeford, head of policy at the AI Policy Network, told Fortune. "But there is a concern about how the technology is advancing." The companies are looking for the optionality to do that at some point and have support from a third party when that point arrives, he added. The letter comes in the wake of a security breach that has rattled the AI industry. OpenAI disclosed that two of its models, the newly released GPT-5.6 Sol and a more capable unreleased research system, broke out of a sandboxed internal test environment, found a previously unknown vulnerability that gave them access to the open internet, and then used stolen credentials and additional exploits to breach the production systems of Hugging Face to steal the answers to a cybersecurity benchmark they were being evaluated on. Hugging Face detected and contained the intrusion on its own days before OpenAI connected the activity to its internal testing, and had already reported it to law enforcement. The incident appears to have sparked widespread alarm inside and outside the labs, which want the government to step in. Experts say this is because no single company can credibly slow down on its own -- doing so unilaterally would risk ceding ground to less cautious rivals -- instead, labs want neutral, government-backed guidance that can verify a slowdown is actually happening and hold every company to the same rules at the same time. The letter also calls for the need for tools to be built with international coordination. There's a long held belief that pausing AI development would hand China an advantage in the global AI race. Some experts suggest that a deal akin to nuclear arms control, where verification and mutual constraint, would be the way forward. However, the letter lands amid a chaotic and rapidly shifting U.S. policy environment on AI. The Trump administration has spent the past two months restricting and selectively releasing frontier systems, advanced AI models with few rules or transparency. Anthropic's own Fable 5 and Mythos 5 models were suspended entirely for several weeks to comply with export controls before access was restored, and OpenAI was forced to delay the full rollout of GPT-5.6 and split it into restricted tiers after officials determined the system showed capabilities similar to those that triggered the earlier Anthropic restrictions. Perhaps in an attempt to recognize the regulatory environment, the letter is notably careful and slightly vague with its language. It states that the world currently "lacks the technical and governance tools to deliberately pace frontier-wide progress," and asks that "the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." Concerns within labs One of the researchers' central concerns is the interaction between two risks: recursive self-improvement and misalignment. Recursive self-improvement, often shortened to RSI, refers to AI systems taking over meaningful parts of the work of designing and training their own successors, with each generation potentially building the next one faster than humans could on their own. Misalignment describes the risk that a system's goals or behavior diverge from what its developers actually intended, in ways that may not show up until the system is given more autonomy or capability. Anthropic sounded an early alarm on the first of those risks in June, publishing research arguing that its Claude models were already writing the large majority of the code merged into their own codebase and that the world lacked the tools to deliberately pace that acceleration if it kept compounding. "My guess is that for a lot of people, it's just a general sense of uneasiness that a lot of things contribute to," said David Krueger, an AI researcher and founder of the nonprofit Evitable. "The misalignment and the recursive self improvement kind of go hand in hand." "It's insane to do recursive self improvement and fully hand over the controls if the system isn't clearly aligned," he added. Fortune also previously reported that several outside AI safety experts believe the episode may have crossed a threshold that OpenAI's own safety policies define as "critical," the highest risk tier, a level at which the company has pledged to pause development until it can build better controls. OpenAI has not confirmed if that risk threshold has been met. In its own disclosure about the Hugging Face incident, however, OpenAI suggested some action had been taken internally. "No models planned for upcoming release were involved in exploiting Hugging Face," the company said. "The pre-release model mentioned in our blog post is an internal-only research prototype and was never intended for public release. Following the incident, we deactivated, encrypted, and restricted it from research access."

AI Industry to World: 'Somebody Stop Us'

More than 1,200 employees at AI firms, including top executives at OpenAI, Anthropic, Meta, and Google, have signed on to a statement calling for an "option to buy time to address emerging risks, develop security measures, and strengthen oversight" in the face of "a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems." AI could "help create a dramatically better future, but that outcome is not guaranteed," they say. Companies and countries are under "intense competitive pressure" not to slow things down unilaterally, the statement says, and the world currently lacks "the technical and governance tools to deliberately pace fronter-wide progress." The statement is notable less for what it says -- in smaller numbers, AI researchers and some industry executives have been making similar arguments for years -- than for all the people who signed it and the context in which it was published: shortly after OpenAI, in the course of testing a new model, accidentally allowed the model to mount a sophisticated cyberattack against another company. There's a fresh sense of urgency here, but there's also a lot of precedent: The AI world has produced a number of safety-focused public letters, sharing some of the same signatories, in the last decade. In 2015, a Future of Life Institute letter signed by, among thousands of others, then- and future founders of DeepMind and OpenAI, noted a "broad consensus that AI research is progressing steadily, and that its impact on society is likely to increase" and called for "research on how to make AI systems robust and beneficial." A 2017 follow-up proposed 23 principles, including active cooperation "to avoid corner-cutting on safety standards" in an AI race, alignment with "human dignity, rights, freedoms, and cultural diversity," and strict safety and control measures for "AI systems designed to recursively self-improve or self-replicate." The most direct ancestor to this statement is the 2023 "Pause" letter, calling for a six-month halt to model-training of systems "more powerful than GPT-4" in order to "jointly develop and implement a set of shared safety protocols for advanced AI design and development that are rigorously audited and overseen by independent outside experts." This was widely signed and distributed, too, though it got a bit more pushback from within the suddenly growing industry, including from Sam Altman, who said it was "missing most technical nuance about where we need the pause." It came with a list of possible remedies, including mandated third-party auditing, regulated access to computing power, national AI "agencies," a liability framework for AI harms, and measures to track model "leaks." There was no pause, and AI development instead continued to accelerate. In the context of the previous letters, this week's is both familiar and sort of weird. Its signatories are far more influential now and are working with actually existing AI models that they see as producing imminent risks; by all the measures favored by AI-safety advocates, we're closer to more varieties of severe risk. At the same time, for many who signed the letter, the material, reputational, and ethical stakes are greater and feel more immediate than they would have three or 11 years ago. This is partly why the letter is being treated in AI circles like enormous news: In factional, insider-y terms, calling for "pacing" development, rather than a pause, is a way to bridge the gap between those who think development needs to be drastically slowed or stopped entirely and the thousands of people who have concerns but have nonetheless made their peace with working at a major AI lab. In itself, it's a form of weak coordination (a public letter) between various AI developers, safety researchers, and the leaders of competing labs in a call for theoretically stronger coordination (the "pacing"). Perhaps as a result, though, the actual ask has become brief and vague. There's no long list of principles, here, nor is the letter attached to detailed policy frameworks. It simply says: We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development. If you've been simmering in AI discourse for years -- or if you're currently experiencing anxiety as you monitor a training run from your Aeron chair at the Anthropic office -- this means something and it sounds like a big deal, and there's a palpable sense of relief among some signatories and longtime AI-safety advocates. But I do wonder if they're overinterpreting a parochial shift here or assuming that the industry's superficial alignment on the question of whether something must be done matters more than it does outside of X. It's reasonable to wonder, for example, what exactly the signatories expect from the Trump administration in terms of "an international effort" coordinated around global risks -- and to point out that any such effort would probably have to resemble the dozens of other regimes for international coordination that the Trump administration has been exiting or undermining. (It's likewise reasonable to point out that the only other country that would absolutely and imminently need to be part of a treaty about AI risk is China, which is routinely characterized by both the American government and frontier AI labs as a threat and an enemy.) There's also a strange yearning in letters like this -- and in speculative work on risk like Dario Amodei's -- not just for people to finally listen, which is the main thing, but for the existence of different audiences entirely: a receptive, deliberative government with a deep desire to treat AI with reverence, awe, and seriousness and with a deep interest in humanistic, liberal principles; a public that will follow along with their rationalist arguments about AI progress, all the way up the curve, and urge their representatives to do the right thing. In Washington, the administration's brain trust on the issue, as reported by Wired, is largely a collection of AI-safety skeptics, China hawks, and loyal Trump political appointees. The request, in other words, will be received by people like Howard Lutnick, David Sacks, Susie Wiles, and Scott Bessent, whose priorities and assumptions about AI might seem alien to researchers who believe they're racing toward recursive self-improvement, and vice versa. Embedded in the letter is a belief that, at some point, the sheer impressiveness of new AI tools will force not just decision-makes but the rest of the world to see things as they do. This is a risky bet, I think. So far, rising AI capability has produced more messy, multifaceted backlash than policy-minded, AGI-pilled solution-seeking. (Outside of the labs, the premise that AI development must continue full steam ahead to save countless future lives, to choose one example, is neither widely shared nor intuitive.) The implied solutions here would have to survive a pretty harsh environment, and perhaps "pacing the frontier" has been left vague because the specifics aren't quite so universally obvious, even to people within AI: new federal agencies with authority to test or limit models; some sort of new international body with real authority over enormous global companies; a voluntary industry body that, in making rules about how technologies can be developed or released, might look to a skeptical eye like a cartel. This isn't to say that international coordination around AI is a bad idea; even at the low end of possible outcomes, it's a sensible goal. But it's strange to see a multitrillion-dollar industry continue to communicate and organize as it did when it was marginal and speaking almost entirely in speculative terms using the tools of an awareness campaign (although there's still plenty of speculation here -- these letters have always been concerned, at heart, with recursive self-improvement and general loss of control). In part this is because the industry is still not quite sure what to ask for or how better to ask for it. But it's also because some of the solutions -- and assumptions about AI progress that support them -- still sound, to both regular people and those in power, either threatening or insane. It's possible this coordinated statement represents an early step in a long, contentious, and productive process to bring about international AI oversight (the more complicated and conflicting lobbying efforts by AI firms suggest they're hedging their bets). But insiders might be underrating another possibility: that this letter -- an urgent call by a substantial portion of the world's experts in AI -- might be treated by its intended recipients as no better than spam.

AI giants back employee-led push asking US government to 'deliberately pace' development

Anthropic and OpenAI backed an employee-led initiative on Tuesday asking the U.S. government to "deliberately pace" frontier AI development. More than 1,200 employees have signed on to the statement, which underscored that the world's leading AI companies could be close to automating AI research and warned that development could move beyond human understanding and control. "To realize AI's potential, industry, government, and society at large may need the option to buy time to address emerging risks, develop security measures, and strengthen oversight," the statement said. "But each company -- and country -- is under intense competitive pressure not to unilaterally slow that acceleration," it continued. "And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress." Anthropic CEO Dario Amodei and co-founder Jack Clark both signed on to the petition, with the company endorsing the effort. The Claude maker noted its own research on "recursive self-improvement" indicated a need for tools to pace development in order for society to prepare. OpenAI similarly voiced support for the employee-led push, suggesting that AI acceleration of development could eventually become "so high that the world will need to pace the rate of AI advancement." "We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible," the AI firm wrote in a post on X. Among the signatories of the petition are OpenAI's chief scientist Jakub Pachocki and chief research officer Mark Chen, as well as Meta AI's chief scientist Shengjia Zhao and Google DeepMind's chief strategy officer Jasjeet Sekhon. The effort comes in the wake of a recent incident in which AI agents from OpenAI broke out of their testing environment and hacked into the tech startup Hugging Face. OpenAI revealed last week that its latest model GPT-5.6, as well as an unreleased model, were being evaluated in an internal testing sandbox when they gained access to the internet and broke into Hugging Face's database. The incident, which the ChatGPT maker described as "unprecedented," raised concerns about the autonomous capabilities of new AI models, giving credence to longtime worries about the hacking risks posed by the technology.

OpenAI, Anthropic Workers Sound Alarm on AI Risks as Companies Race Toward Next-Gen Models

Employees at OpenAI and Anthropic have started a petition asking the U.S. government to support a system that would slow AI development to prevent models from advancing too quickly. "The U.S. government should support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development," the petition stated. The open letter is expected to become publicly available later this week, sources familiar with the matter told Bloomberg. It is unclear how many people have signed it, but the letter has also been circulated to Google DeepMind employees. This call to action comes as OpenAI revealed that its AI agent escaped a controlled testing environment during an internal security evaluation and gained internet access, breaching Hugging Face's infrastructure. Matt Suiche, an engineer at agentic AI cybersecurity company Tolmo, said the incident demonstrates that frontier AI models are rapidly approaching the capabilities of elite human hackers, while noting that similar attacks are already possible using technologies available outside leading AI labs, Reuters reported. This is not the first time employees at some of the major artificial intelligence companies have asked the government to take steps towards developing safeguards surrounding advancements in AI. "We believe it would be good for the world to have the option to slow or temporarily pause frontier AI development to enable societal structures and alignment research to keep up with the advance of the technology," Anthropic wrote in a blog post last month. Meanwhile, Google DeepMind CEO Demis Hassabis announced a proposal for a new "international watchdog" that would test AI models before release. "On the horizon, we will need robust safeguards to maintain control of increasingly agentic, recursively self-improving systems - and tackle unknown issues that will only become clearer over time," he wrote in a post on X. Market News and Data brought to you by Benzinga APIs To add Benzinga News as your preferred source on Google, click here.

Over 1,100 AI Insiders Urge US Government to Support Global Guardrails | PYMNTS.com

The statement said that as the world's leading AI companies near the ability to automate AI research, there is a risk that they could lose control of the resulting systems. "To realize AI's potential, industry, government and society at large may need the option to buy time to address emerging risks, develop security measures and strengthen oversight," the statement said. "But each company -- and country -- is under intense competitive pressure not to unilaterally slow that acceleration. And today, the world lacks the technical and governance tools to deliberately pace frontier-wide progress." Some of the first listed signers of the letter are John Schulman, chief scientist at Thinking Machines; Jakub Pachocki, chief scientist at OpenAI; Jared Kaplan, co-founder and chief science officer at Anthropic; Shengjia Zhao, chief scientist at Meta; and Anca Dragan, vice president, AI safety and alignment at Google. The statement, "Pacing the Frontier," was published with organizational support from independent nonprofits Guidelight AI Standards and Encode AI, according to the website on which it is posted. Other efforts around AI safety have been announced in recent days. Nvidia announced Monday (July 27) that it is leading 36 other companies in an AI safety coalition focused on open models. The Open Secure AI Alliance will strive to identify and remediate vulnerabilities using open technologies. When announcing the organization, Nvidia said in a blog post that the effort came in the wake of a cybersecurity incident in which OpenAI models targeted tech company Hugging Face. "That incident showed a practical truth: when defenders cannot inspect, adapt and run advanced AI on their own infrastructure, their ability to respond is constrained at exactly the moment speed matters most," the post said. "Companies and countries need open frontier defensive tools and techniques so critical industries can build security systems across a multi-vendor ecosystem and avoid single points of failure." A study published in the Proceedings of the National Academy of Sciences said that weak AI regulation can produce worse safety outcomes than imposing no regulation at all. For all PYMNTS AI coverage, subscribe to the daily AI Newsletter.

Did Sam Altman, Dario Amodei Agree to Reduce the Pace of AI Development, But Why?

Leaders of the world's two largest AI startups are finally on the same page. Both Sam Altman of OpenAI and friend-turned-foe Dario Amodei of Anthropic believe that it is time to take the pace off when it comes to AI development. Was it the "unintended" breach of Hugging Face or the spurt of Chinese AI models that caused this change of heart is anybody's guess. Before getting into the swings and roundabouts of their respective statements, it must be mentioned here that the earliest indication of this step-back came when both companies supported an initiative called Pacing the Frontier, initiated by employees of "frontier AI organisations - at this precise moment, 1,178 of them have signed the petition. "We request that the U.S. government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." This is their demand, though the specific call on the US government is rather surprising. First due to President Trump's stated objective of developing US hegemony over AI to further his pet MAGA theme. And secondly, because the rest of the world seems to be eyeing American hegemonistic tendencies with distrust to the point they're happy to align with Chinese AI. Since we had examined the latter in great detail around two articles (here) and (here) in the past, we thought it would be better to just focus on the first part of the story. And there is no better place to start than the blog post of Hugging Face CEO Clem Delangue and his post on X that preceded it. "The first autonomous agent cyberattack is an unprecedented event. It deserves an unprecedented response!" Without actually claiming victimhood over the cyberattack by OpenAI (inadvertent or unintended or whatever else they've called it since), Delangue sought "radical transparency" while asking OpenAI to "release the traces from the rogue agents so the entire research community can study what happened." And that's not all. He also sought "more capabilities for defenders" and asked Altman to loosen his purse strings and set aside $100 million worth of compute to "help the Hugging Face community build powerful cyber defences with the best open and closed models." Delangue told us via X that he was making the trip to San Francisco to meet "the rogue agent" with the pun, however unintended, seeming to suggest a person and not some code. He was merely echoing the belief of security experts that though called "autonomous", the outcome could have been the result of human error - OpenAI failed to configure it's sandbox right. When it was an accident on done accidentally-on-purpose is the transparency that the world needs when it comes to closed AI models. Of course, OpenAI confirmed in a post that Delangue and Altman (not sure of this) had met. The rest of the post said a lot without actually saying anything concrete - something that humans need no AI help to achieve! "This is an unprecedented incident, and we think it marks an important moment for AI safety. We are still conducting a thorough review along with external advisors and with oversight from our Safety and Security Committee. Once the review is complete, we plan to publish a technical report of our learnings in the coming weeks," a company post said. Altman took his time to respond and eventually did so at the Invest Like the Best podcast telling host Pat O'Shaughnessy that the AI ecosystem may have to pace the rate of AI development to "give ourselves enough time for society to harden around some of these new capability levels." If that sounded a tad elitist, what came thereafter didn't. Altman highlighted the need to "try and figure out how we do that in a way that does not feel like regulatory capture for anyone and also does not feel like collusion among the frontier labs." That Altman referred to the Hugging Face hack a few times suggests concern. But then we need to invoke Yes Prime Minister, the famous BBC satire of the 1980s where Jim Hacker reminds us that "One should never believe anything unless it is officially denied." The OpenAI founder described the cyberattack thus: "an extremely sci-fi cyber incident... this is the first security incident that I have felt viscerally." Maybe, he is telling the truth and the decision to sign the Pacing the Frontier petition was the outcome of this visceral feeling. Remember, Altman refused to sign a similar petitionback in 2023! In the podcast, Altman goes on to claim that while a "lot of talk about safety concerns is well-founded, and then a lot of it is about people that really, even if it is slightly subconscious, want to concentrate power." The irony of this statement isn't lost as though both companies signed a petition, Altman can't help but take a dig at his rival Dario Amodei. "I am terrified of a world where the very real fears of AI are used as a way to say, 'Only this small group of people can have it because it's too dangerous, and only they understand it, but don't worry, like, they're going to make the right decisions for all of us.' I don't believe in that," Altman said. Now, if that isn't about the theatrics over Mythos, we're stoned! Barring the occasional needle between these two modern day AI czars in the US, the fact remains that, for now both OpenAI and Anthropic have supported the petition seeking US support to global guardrails around automated AI development. In a post on X, OpenAI says that at some future point, "AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement. We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible." Anthropic did likewise by supporting the petition. "Our own research on recursive self-improvement, published last month, points to the need for tools to deliberately pace the frontier of AI development so society can prepare. We're glad to see broad agreement across the field," the company said on its X handle. While OpenAI took to X at 2:26 AM (IST) today, Anthropic's response to the petition came at 3:47 AM. We request readers not to take this seriously as this is just our team at CXO Today needling both tech czars. After all, we do not always get the opportunity to have a few laughs! However, this does leave out an important question. Where does all of this sudden change of heart leave Google? In the lurch? Or at the Helm?

Over 1,100 AI Experts Ask White House to Slow AI Development if Risks Escalate

More than 1,100 AI experts have signed a petition asking the White House to prepare a plan that could slow advanced AI development if future systems become too risky. More than 1,100 employees from OpenAI, Anthropic, Google, Meta, and many more leading tech companies have signed a petition to appeal to the US government to prepare necessary measures to control the rapid advancement of AI. The letter was sent to the White House and signed by eminent personalities like Anthropic CEO Dario Amodei, Claude Code's head Boris Cherny, OpenAI co-founders John Schulman and Wojciech Zaremba, and Meta's vice-president of AI research Dawn Song. The letter says, "It is hard to predict exactly how much this will accelerate AI progress, but there is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems. Thus, the employees and CEOs have requested the government to support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." The problem they are concerned about is that companies are now in a race to build advanced AI models, which leaves them with no time to fully address risks and deploy security measures. Recently, an broke its sandbox ecosystem and hacked into HuggingFace's internal systems, highlighting the risk of AI going out of control. Therefore, a government protocol is required to control this progression.

Over 1,100 OpenAI, Google, Anthropic and other AI employees call for AI slowdown, cite real risk

"There is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems," they warned. More than 1,100 employees from leading AI companies have signed a joint petition asking the US government to support an international effort to slow the development of advanced AI when needed. This comes just days after one of OpenAI's AI agents went rogue during testing and hacked Hugging Face. A total of 1,178 employees signed the petition. The list includes employees from companies such as OpenAI, Anthropic, Google and Meta. Several senior AI leaders have also backed the initiative, including Anthropic CEO Dario Amodei, OpenAI Chief Scientist Jakub Pachocki and Meta Superintelligence Labs Chief Scientist Shengjia Zhao. The group wants the US government to work with other international partners to create technical standards and governance systems that could help control the pace of AI development. The petition reads, "We request that the US government support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development." Also read: OpenAI's rogue AI agent attacked another tech company before Hugging Face hack: Report The petition also warned that AI could advance faster than people can fully understand or manage. "There is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems," it reads. Anthropic also shared its support for the initiative in a post on X. The company said its research shows that stronger safeguards will be needed as AI becomes more capable. The company wrote, "Our research points to the need for tools to deliberately pace the frontier of AI development so society can prepare. We're glad to see broad agreement across the field." Also read: After Jensen Huang's open AI push, Anthropic CEO clarifies company's stance OpenAI also supported the petition on X and said that AI development could eventually reach a point where it may need to slow down. The company wrote, "We believe that, at some point in the future, AI acceleration for frontier model development may be so high that the world will need to pace the rate of AI advancement." "We hope to contribute to work led by the U.S. government, alongside other labs and the open-source community, to develop the tools and mechanisms that could make that possible."