Weak AI Regulation Can Make Technology Less Safe Than No Rules at All, Study Warns

Weak AI Regulation Creates Dangerous Incentives

Weak AI safety regulations may backfire and produce technology that is less safe than AI developed under no regulation at all, according to a groundbreaking study published in the Proceedings of the National Academy of Sciences 1

1

. Researchers from Cornell University and Carnegie Mellon University used theoretical economics and game theory to model how regulatory requirements influence investments in AI safety across the technology's complex development chain 3

3

.

The study's central finding challenges conventional wisdom about AI governance. When regulations focus primarily on downstream companies—firms that deploy AI in healthcare, customer service, or e-commerce—AI model developers like OpenAI, Google, and Anthropic tend to reduce their own safety investments 1

1

. This creates what researchers call a free-riding problem, where foundation model developers shift the safety burden onto application providers, ultimately leaving gaps in oversight that make the technology more dangerous 4

4

.

The Free-Riding Problem in the AI Supply Chain

"There's a free-riding behavior that occurs," explained principal author Benjamin Laufer, a doctoral researcher at Cornell University. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist" 1

1

. The mechanism is straightforward: when downstream companies face legal requirements to meet safety standards, model developers know the final product will clear regulatory bars regardless of their own efforts. This removes their incentive to invest in measures like third-party safety audits 2

2

.

The research team, which included Laufer's adviser Jon Kleinberg and Professor Hoda Heidari of Carnegie Mellon University, modeled the AI supply chain as a two-player game 2

2

. One company trains a large, general-purpose AI model, while another adapts that model for specific applications. Each chooses how much to invest in AI safety and performance, then splits the revenue. The researchers found that weak AI rules targeting only the second company consistently reduced total safety below levels achieved with no regulation at all—a result that held across a wide range of scenarios in their economic modeling 2

2

.

Strong AI Regulation Benefits Everyone

The study offers a counterintuitive solution: stronger, well-placed regulation covering both AI model developers and downstream companies can improve both safety and business outcomes 1

1

. The researchers define utility as revenue share minus investment cost, and their model shows that appropriate AI safety regulations can increase end product safety while improving the utility derived by all players in the AI supply chain 3

3

.

This optimal outcome stems from solving what game theory calls a prisoner's dilemma. Two companies building AI products may both want higher safety, yet neither can trust the other to follow through. Each has a private incentive to cut corners at the last minute, so the safer, more profitable path never gets taken 2

2

. Strict regulation binding both sides removes this uncertainty. "The goal of regulation should be the mutual benefit of everybody in society, and this can include those developing the technology, but also end users and the public," Laufer noted 2

2

.

Implications for AI Policy Debates

The findings arrive as policymakers worldwide grapple with how to govern artificial intelligence. In the United States, two camps have formed around AI governance. One group favors lighter federal guardrails aligned with the Trump administration's approach, arguing that the AI industry needs freedom to innovate quickly to compete with China 1

1

. The opposing camp pushes for stricter AI safety regulations, warning that under-regulated development could lead to consequences ranging from mental health impacts to widespread job displacement 1

1

.

At the federal level, policymakers have concentrated on companies developing advanced or frontier AI models, including questions about model safety, testing and national security. States, meanwhile, have focused more heavily on downstream applications in employment, healthcare, insurance and other high-impact decisions 3

3

. The study suggests neither layer of AI regulation should be considered in isolation, as obligations imposed at one point in the supply chain alter safety investments elsewhere 3

3

.

"People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology," Laufer emphasized. "To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity" 1

1

. As states and countries roll out new AI laws, the research indicates that who gets regulated may be just as important as how stringent the rules are 2

2

.