Weak AI Regulation Can Make Technology Less Safe Than No Rules at All, Study Warns

Weak AI Regulation Is Worse Than No Regulation, Researchers Claim

Weak AI safety regulations may "backfire," creating a product that is potentially more dangerous than AI products created under no regulation, according to a new study published on Monday in the Proceedings of the National Academy of Sciences. Using theoretical economics and game theory principles, a group of researchers from Cornell and Carnegie Mellon University created a theoretical model that aims to show how AI regulation can be most effective at ensuring safety. They found that for true safety, regulation needs to be strict while targeting the companies that develop AI models (such as OpenAI, Google, and Anthropic), rather than solely focusing on the downstream companies that apply the technology in real-life settings, like companies that provide AI medical diagnostic systems or e-commerce customer service chatbots. Even though just choosing to regulate the specific AI use cases "might seem logical" at first, the authors argue that it can backfire and reduce the overall safety of AI products. That's because when the government focuses on regulating downstream companies and lets the AI model developers off the hook, the general-purpose AI developers tend to cut corners on safety measures like third-party audits, hoping that the downstream companies will ensure the safety of the end product instead. "There's a free-riding behavior that occurs," according to the study's principal author Benjamin Laufer. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist." The study comes as the United States government and Silicon Valley are trying to figure out how artificial intelligence should be regulated. Two main camps have formed in response. On one side are the anti-regulation technologists, who envision much lighter federal guardrails that largely align with the Trump administration's approach to AI governance. This group tends to say that the AI industry should be free of unnecessary guardrails to innovate as quickly as possible, because that's the only way that the United States can win the global AI race against China. The self-proclaimed pro-innovation group also tends to fashion the opposing camp, who favor stricter AI safety regulation, as doomers at best and as attempting regulatory capture at worst. The supporters of stricter federal AI regulation, however, claim that, in pursuit of wider profit margins, the AI industry is underestimating or underselling the risks of under-regulated AI development, and the list of purported downsides includes everything from AI psychosis to the community health consequences of data centers and a much-feared unemployment crisis that is expected to follow wider AI adoption. But the authors of the new study argue that safety versus revenue doesn't have to be an either-or situation. According to the model, "stronger, well-placed regulation can mutually benefit all players" by improving both the safety of the end product and the utility general-purpose AI creators and the downstream domain specialists get from the investment. The researchers define utility as revenue share minus investment cost. This supposed sweet spot exists when regulators expect both the general-purpose AI producers and the downstream companies to make enough investment to meet meaningful safety standards. The situation is a classic example of a prisoner's dilemma, a game theory problem in which two rational decision-makers are given the option to cooperate or betray each other. If they both choose to cooperate, then they will get the best possible outcome, but neither knows what the other will choose. If they both betray each other, then they get a mediocre outcome, but if one decides to cooperate while the other betrays, then the betrayed one gets the worst outcome. Unsure what the other participant is going to choose, the decision-maker often chooses to betray and guarantee their own benefit, ensuring a worse outcome for everyone than had they cooperated. Strict regulation for companies across the AI supply chain would ensure trust rather than freeriding, providing the grounds for cooperation and the most ideal outcome for all, the researchers claim. "People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology," Laufer said. "To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity."

Weak AI regulations may leave artificial intelligence less safe

New research warns that weak AI safety regulations can actually make artificial intelligence less safe. Instead of encouraging companies to improve their products, poorly designed rules may shift responsibility in ways that reduce overall safety. The risk appears when regulations target only the companies that adapt AI for specific jobs, while the firms building the underlying models remain largely untouched. As states and countries roll out new AI laws, the study suggests that who gets regulated may be just as important as how AI is regulated. Good intentions, bad results Benjamin Laufer, a doctoral researcher at Cornell University, built the analysis with his adviser Jon Kleinberg, in collaboration with Professor Hoda Heidari of Carnegie Mellon University (CMU). Together they modeled how safety rules ripple through the chain of companies that build and sell AI. That chain usually has two links. One company trains a large, general-purpose AI model, the kind that powers popular chatbots. Another company takes that model and adapts it for a narrow job, like reading medical scans or handling customer complaints. The team's central result is blunt. A weak rule placed only on the second company can lower the safety of the finished product. It sinks below the level those same companies would have reached with no rule at all. That result surprised them. It also held up across a wide range of settings in their model, not just one lucky example. Balancing AI safety and profit The researchers treated the two companies as players in a game. Each one chooses how much to invest in AI safety and raw performance. Safety costs money. So does performance. Whatever they build, they split the revenue it earns. The model maker moves first and sets the starting point. Then the domain specialists who adapt the model decide how much further to push it. Before any of that, the two sides strike a deal on how to divide the eventual payout. A regulator sits above all this. It can set a minimum safety level for the first company, the second, both, or neither. The researchers then solved for how rational, profit-seeking firms would respond. This builds on earlier work from the same group on how general and specialist firms bargain over fine-tuning. The new twist is the safety floor and the question of who should have to clear it. The free-rider problem The backfiring comes from a simple piece of self-interest. Consider a rule that forces the downstream company to meet a set safety bar. The model maker now knows the final product will clear that bar no matter what because the law requires it. So the maker can quietly spend less on safety. The downstream safety floor does the work instead. In the unregulated version, the maker had reason to invest more because no one else was guaranteed to. "There's a free-riding behavior that occurs," said Laufer. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist." The net effect runs backward. The rule was meant to raise safety, but it hands the upstream company an excuse to cut corners. Total safety ends up lower than before. They found this holds whenever both companies share revenue and both put in real effort. That covers a broad slice of how AI is actually built today. AI regulations that actually work The second finding runs the other way. Rules aimed at both companies, set at the right level, can make products safer and leave both firms better off. The reason is trust. Two companies building a product may both want higher safety, yet neither can trust the other to follow through. Each has a private incentive to skimp at the last minute. So the safer, more profitable path never gets taken. A rule that binds both sides removes the guesswork. Neither has to take the other at its word. That lets them reach a combination of safety and profit they both prefer but could not lock in alone. "The goal of regulation should be the mutual benefit of everybody in society, and this can include those developing the technology, but also end users and the public," said Laufer. This is why some companies openly ask to be regulated. A well-placed rule can act like a contract with teeth, and both parties may be willing to pay for it. A separate study similarly found that safety rules can ease a firm's entry into a market rather than block it. AI laws are still evolving Right now, much of the debate is guesswork. Recent proposals differ on which companies should bear responsibility for AI safety, and there is little hard evidence showing how those choices play out in practice. "There isn't much AI safety regulation, and so a lot of possible regulations are just proposals at this stage," said Laufer. "To some extent, regulation is poking in the dark, so it's worth reasoning through what effects these regulations might have on incentives." The model offers one clear lesson. Aiming safety rules only at the companies adapting AI for niche uses can quietly make things worse. Spreading the requirements across both the model maker and the adapter tends to work better. That question is already shaping real policy. The European Union's AI Act and past state bills have wrestled with exactly where to draw the line. Lessons for future AI regulation The team's findings may also extend beyond AI safety. A related analysis from the same researchers found that rules encouraging AI models to become more open can backfire in much the same way. The study is still a simplified picture, and the authors want to test their predictions against real-world regulations as they emerge. Even so, the work highlights a trap policymakers may want to avoid. Weak rules aimed at the wrong companies could do less than nothing. The study is published in Proceedings of the National Academy of Sciences. -- - Like what you read? Subscribe to our newsletter for engaging articles, exclusive content, and the latest updates. Check us out on EarthSnap, a free app brought to you by Eric Ralls and Earth.com.

Study Warns Weak AI Rules Can Make Technology Less Safe | PYMNTS.com

The study, published in the Proceedings of the National Academy of Sciences by researchers from Cornell University and Carnegie Mellon University, used theoretical economics and game theory to model how regulatory requirements influence investments in AI safety. Its central finding is that poorly targeted or insufficiently stringent regulation can create incentives for companies to reduce their own safety investments and shift responsibility to others. The researchers draw an important distinction between two points at which governments can regulate AI: the developers of general-purpose models, such as OpenAI, Google and Anthropic, and downstream companies that deploy those models for particular purposes, such as medical diagnostics, eCommerce or customer service chatbots. Regulating individual applications may appear to be the most logical approach because risks often emerge when AI is deployed in specific settings, per Gizmodo's report on the study. But the study found that focusing regulation primarily on downstream users can create unintended incentives for model developers to spend less on safety. When downstream companies are expected to ensure that applications meet regulatory requirements, general-purpose model providers may "free ride" on those investments by cutting back on measures such as third-party safety audits. "There's a free-riding behavior that occurs," principal author Benjamin Laufer said. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist." That distinction is particularly relevant to the emerging U.S. regulatory landscape. At the federal level, policymakers have focused much of their attention on companies developing advanced or frontier AI models, including questions surrounding model safety, testing and national security. States, meanwhile, have generally concentrated more heavily on downstream applications, including AI used in employment, healthcare, insurance and other high-impact decisions. The study suggests that neither layer of regulation should be considered in isolation. Instead, policymakers should account for how obligations imposed at one point in the AI supply chain can alter safety investments elsewhere. The researchers found that strong, appropriately placed regulation can produce benefits for both safety and economic returns. Their model suggests that "stronger, well-placed regulation can mutually benefit all players" by increasing end product safety while also improving the utility derived by general-purpose AI developers and downstream specialists from their investments. The study defines utility as a company's share of revenue minus its investment costs. The optimal outcome occurs when regulators require sufficient safety investment from both model developers and downstream companies rather than allowing either side to assume the other will shoulder the burden. Researchers compared the problem to the classic "prisoner's dilemma" in game theory. Without confidence that other participants will invest adequately in safety, each company has an incentive to protect its own economic interests by spending less and relying on others. The result can be collectively worse even when cooperation would benefit everyone. Strong regulation across the AI supply chain, by contrast, can reduce that uncertainty and discourage companies from shifting responsibility. In that framework, regulatory requirements effectively create conditions in which both model providers and downstream deployers have incentives to make complementary safety investments. The findings could complicate the broader U.S. debate between those who argue that excessive regulation could undermine innovation and U.S. competitiveness with China, and supporters of stronger safeguards, who contend that commercial incentives alone are insufficient to address AI risks. Rather than framing the choice simply as regulation versus deregulation, the study argues that the design and placement of regulation may be just as important as how stringent it is. "People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology," Laufer said. "To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity."

Weak AI laws can make AI more dangerous instead of safer, study warns

Strong, coordinated regulation across the AI supply chain could improve both AI safety and business outcomes, according to the researchers. AI regulations that are too weak or aim at wrong companies can unintentionally make AI systems less safe, as per the new study published in the Proceedings of the National Academy of Sciences (PNAS). The researchers from Cornell University and Carnegie Mellon University argue that effective AI regulation should focus on companies making the foundation models rather than only the businesses deploying AI in products and services. The study suggests that when regulations primarily target downstream companies such as firms using AI in healthcare, customer support or e-commerce, developers of general purpose AI models may reduce their own investments in safety measures. Instead, they could rely on application providers to handle safety risks, creating gaps in oversight. Researchers warn of 'free-riding' on AI safety Using economic modelling and game theory, the researchers examined how the different regulatory approaches influence the behaviour of AI companies. They found that model developers are more likely to scale back efforts such as third party safety audits if they believe downstream businesses will bear the responsibility of ensuring safe deployment. Also read: Apple iPhone 17 Pro price drops by over Rs 9,000 ahead of iPhone 18 Pro launch: How to grab this deal The researchers describe this as a free riding problem, where AI developers shift the burden of safety onto companies building applications on top of their models. According to the study, this may ultimately result in AI systems that are less secure than those developed without such regulations. Strict regulation can benefit both safety and business The study comes when the entire world is debating on how AI should be governed. In the US, one group says that minimal regulation is needed to maintain innovation and compete globally while others are pushing for stronger safeguards to address concerns from misinformation and mental health impacts to job displacement. The researchers, on the other hand, argue that stricter, well-designed regulation does not necessarily come at the expense of innovation. As per their model, needing meaningful safety investments from both AI developers and companies deploying AI can improve overall system safety.