Weak AI Regulation Can Backfire and Make AI More Dangerous, Researchers Warn

Weak AI Regulation Is Worse Than No Regulation, Researchers Claim

Weak AI safety regulations may "backfire," creating a product that is potentially more dangerous than AI products created under no regulation, according to a new study published on Monday in the Proceedings of the National Academy of Sciences. Using theoretical economics and game theory principles, a group of researchers from Cornell and Carnegie Mellon University created a theoretical model that aims to show how AI regulation can be most effective at ensuring safety. They found that for true safety, regulation needs to be strict while targeting the companies that develop AI models (such as OpenAI, Google, and Anthropic), rather than solely focusing on the downstream companies that apply the technology in real-life settings, like companies that provide AI medical diagnostic systems or e-commerce customer service chatbots. Even though just choosing to regulate the specific AI use cases "might seem logical" at first, the authors argue that it can backfire and reduce the overall safety of AI products. That's because when the government focuses on regulating downstream companies and lets the AI model developers off the hook, the general-purpose AI developers tend to cut corners on safety measures like third-party audits, hoping that the downstream companies will ensure the safety of the end product instead. "There's a free-riding behavior that occurs," according to the study's principal author Benjamin Laufer. "The regulation acts as a tool for the general provider to offload the safety burden onto the downstream specialist." The study comes as the United States government and Silicon Valley are trying to figure out how artificial intelligence should be regulated. Two main camps have formed in response. On one side are the anti-regulation technologists, who envision much lighter federal guardrails that largely align with the Trump administration's approach to AI governance. This group tends to say that the AI industry should be free of unnecessary guardrails to innovate as quickly as possible, because that's the only way that the United States can win the global AI race against China. The self-proclaimed pro-innovation group also tends to fashion the opposing camp, who favor stricter AI safety regulation, as doomers at best and as attempting regulatory capture at worst. The supporters of stricter federal AI regulation, however, claim that, in pursuit of wider profit margins, the AI industry is underestimating or underselling the risks of under-regulated AI development, and the list of purported downsides includes everything from AI psychosis to the community health consequences of data centers and a much-feared unemployment crisis that is expected to follow wider AI adoption. But the authors of the new study argue that safety versus revenue doesn't have to be an either-or situation. According to the model, "stronger, well-placed regulation can mutually benefit all players" by improving both the safety of the end product and the utility general-purpose AI creators and the downstream domain specialists get from the investment. The researchers define utility as revenue share minus investment cost. This supposed sweet spot exists when regulators expect both the general-purpose AI producers and the downstream companies to make enough investment to meet meaningful safety standards. The situation is a classic example of a prisoner's dilemma, a game theory problem in which two rational decision-makers are given the option to cooperate or betray each other. If they both choose to cooperate, then they will get the best possible outcome, but neither knows what the other will choose. If they both betray each other, then they get a mediocre outcome, but if one decides to cooperate while the other betrays, then the betrayed one gets the worst outcome. Unsure what the other participant is going to choose, the decision-maker often chooses to betray and guarantee their own benefit, ensuring a worse outcome for everyone than had they cooperated. Strict regulation for companies across the AI supply chain would ensure trust rather than freeriding, providing the grounds for cooperation and the most ideal outcome for all, the researchers claim. "People think of AI as a single object, but actually AI involves a very complicated set of stakeholders and actors that each have their own contributions to the technology," Laufer said. "To regulate in a thoughtful way, we need to consider the whole supply chain, not just a single provider or entity."

Weak AI laws can make AI more dangerous instead of safer, study warns

Strong, coordinated regulation across the AI supply chain could improve both AI safety and business outcomes, according to the researchers. AI regulations that are too weak or aim at wrong companies can unintentionally make AI systems less safe, as per the new study published in the Proceedings of the National Academy of Sciences (PNAS). The researchers from Cornell University and Carnegie Mellon University argue that effective AI regulation should focus on companies making the foundation models rather than only the businesses deploying AI in products and services. The study suggests that when regulations primarily target downstream companies such as firms using AI in healthcare, customer support or e-commerce, developers of general purpose AI models may reduce their own investments in safety measures. Instead, they could rely on application providers to handle safety risks, creating gaps in oversight. Researchers warn of 'free-riding' on AI safety Using economic modelling and game theory, the researchers examined how the different regulatory approaches influence the behaviour of AI companies. They found that model developers are more likely to scale back efforts such as third party safety audits if they believe downstream businesses will bear the responsibility of ensuring safe deployment. Also read: Apple iPhone 17 Pro price drops by over Rs 9,000 ahead of iPhone 18 Pro launch: How to grab this deal The researchers describe this as a free riding problem, where AI developers shift the burden of safety onto companies building applications on top of their models. According to the study, this may ultimately result in AI systems that are less secure than those developed without such regulations. Strict regulation can benefit both safety and business The study comes when the entire world is debating on how AI should be governed. In the US, one group says that minimal regulation is needed to maintain innovation and compete globally while others are pushing for stronger safeguards to address concerns from misinformation and mental health impacts to job displacement. The researchers, on the other hand, argue that stricter, well-designed regulation does not necessarily come at the expense of innovation. As per their model, needing meaningful safety investments from both AI developers and companies deploying AI can improve overall system safety.