AI Coding Tools May Slow Down Experienced Developers, Study Reveals

AI coding tools may not speed up every developer, study shows | TechCrunch

Software engineer workflows have been transformed in recent years by an influx of AI coding tools like Cursor and GitHub Copilot, which promise to enhance productivity by automatically writing lines of code, fixing bugs, and testing changes. The tools are powered by AI models from OpenAI, Google DeepMind, Anthropic, and xAI that have rapidly increased their performance on a range of software engineering tests in recent years. However, a new study published Thursday by the non-profit AI research group METR calls into question the extent to which today's AI coding tools enhance productivity for experienced developers. METR conducted a randomized controlled trial for this study by recruiting 16 experienced open-source developers and having them complete 246 real tasks on large code repositories they regularly contribute to. The researchers randomly assigned roughly half of those tasks as "AI-allowed," giving developers permission to use state-of-the-art AI coding tools such as Cursor Pro, while the other half of tasks forbade the use of AI tools. Before completing their assigned tasks, the developers forecasted that using AI coding tools would reduce their completion time by 24%. That wasn't the case. "Surprisingly, we find that allowing AI actually increases completion time by 19% -- developers are slower when using AI tooling," the researchers said. Notably, only 56% of the developers in the study had experience using Cursor, the main AI tool offered in the study. While nearly all the developers (94%) had experience using some web-based LLMs in their coding workflows, this study was the first time some used Cursor specifically. The researchers note that developers were trained on using Cursor in preparation for the study. Nevertheless, METR's findings raise questions about the supposed universal productivity gains promised by AI coding tools in 2025. Based on the study, developers shouldn't assume that AI coding tools -- specifically what's come to be known as "vibe coders" -- will immediately speed up their workflows. METR researchers point to a few potential reasons why AI slowed down developers rather than speeding them up. First, developers spend much more time prompting AI and waiting for it to respond when using vibe coders rather than actually coding. AI also tends to struggle in large, complex code bases, which this test used. The study's authors are careful not to draw any strong conclusions from these findings, explicitly noting they don't believe AI systems currently fail to speed up many or most software developers. Other large scale studies have shown that AI coding tools do speed up software engineer workflows. The authors also note that AI progress has been substantial in recent years, and that they wouldn't expect the same results even three months from now. METR has also found that AI coding tools have significantly improved their ability to complete complex, long-horizon tasks in recent years. The research offers yet another reason to be skeptical of the promised gains of AI coding tools. Other studies have shown that today's AI coding tools can introduce mistakes, and in some cases, security vulnerabilities.

Need to Code Faster? AI Might Be Slowing You Down

Much has been made of how AI tools like Cursor Pro or Anthropic's Claude will revolutionize coders' day-to-day experience, potentially shaving tens of hours off their working weeks -- but research has now cast doubt on those assumptions. A new study assessing the performance of 16 experienced software developers working on 246 different tasks found that using AI tools actually increased completion time for the tasks by 19%. This flew in the face of the expectations of researchers, who had anticipated that the AI-assisted coders would be 24% faster than the non-AI control group. The study, first spotted by The Register, came from Metr, a nonprofit AI-focused research organization. Reasons why AI slowed down developers included low reliability. The study found that the developers accepted fewer than 44% of the AI's code suggestions, and the majority reported making major changes to clean up the AI-generated code. In fact, the developers spent 9% of their time reviewing or cleaning the AI outputs. Though the AI-assisted developers did save a lot of time spent actively coding or researching, many of these gains were nullified by time spent crafting prompts, waiting for the AI to come back with responses, or proofing its output for errors. In addition, the study found that participants consistently overestimated AI's ability to assist at the task at hand, even after hours of using the tool. However, there are plenty of caveats to this research. The participants all had five years of coding experience, with only "moderate" AI experience. So results may have been different for less experienced coders. The participants were also set to work on very large and complex existing code repositories, which broadly "had very high quality bars for code contributions," according to the researchers. So results may be different for a completely green coder working on a simple app. This isn't the first time that we've heard rumblings about AI causing as many problems as it solves in the workplace. Research from economists at the University of Chicago and the University of Copenhagen released earlier this year found that AI has yet to move the needle in terms of real-world productivity. Though it caused gains of roughly 3% in overall worker productivity, this was offset by the workload of tasks created by AI, such as teachers checking homework for ChatGPT use.

AI coding tools make developers slower, study finds

Artificial intelligence coding tools are supposed to make software development faster, but researchers who tested these tools in a randomized, controlled trial found the opposite. Computer scientists with Model Evaluation & Threat Research (METR), a non-profit research group, have published a study showing that AI coding tools made software developers slower, despite expectations to the contrary. Not only did the use of AI tools hinder developers, but it led them to hallucinate, much like the AIs have a tendency to do themselves. The developers predicted a 24 percent speedup, but even after the study concluded, they believed AI had helped them complete tasks 20 percent faster when it had actually delayed their work by about that percentage. Surprisingly, we find that allowing AI actually increases completion time by 19 percent -- AI tooling slowed developers down "After completing the study, developers estimate that allowing AI reduced completion time by 20 percent," the study says. "Surprisingly, we find that allowing AI actually increases completion time by 19 percent -- AI tooling slowed developers down." The study involved 16 experienced developers who work on large, open source projects. The developers provided a list of real issues (e.g. bug fixes, new features, etc.) they needed to address - 246 in total - and then forecast how long they expected those tasks would take. The issues were randomly assigned to allow or disallow AI tool usage. The developers then proceeded to work on their issues, using their AI tool of choice (mainly Cursor Pro with Claude 3.5/3.7 Sonnet) when allowed to do so. The work occurred between February and June 2025. The study says the slowdown can likely be attributed to five factors: Other considerations like AI generation latency and failure to provide models with optimal context (input) may have played some role in the results, but the researchers say they're uncertain how such things affected the study. Other researchers have also found that AI does not always live up to the hype. A recent study from AI coding biz Qodo found some of the benefits of AI software assistance were undercut by the need to do additional work to check AI code suggestions. An economic survey found that generative AI has had no impact on jobs or wages, based on data from Denmark. An Intel study found that AI PCs make users less productive. And call center workers at a Chinese electrical utility say that while AI assistance can accelerate some tasks, it also slows things down by creating more work. That aspect of AI tool use - the added work - is evident in one of the graphics included in the study. "When AI is allowed, developers spend less time actively coding and searching for/reading information, and instead spend time prompting AI, waiting on and reviewing AI outputs, and idle," the study explains. More anecdotally, a lot of coders find that AI tools can help test new scenarios quickly in a low-stakes way and automate certain routine tasks, but don't save time overall because you still have to validate whether the code actually works - plus, they don't learn like an intern. In other words, AI tools may make programming incrementally more fun, but they don't make it more efficient. The authors - Joel Becker, Nate Rush, Beth Barnes, and David Rein - caution that their work should be reviewed in a narrow context, as a snapshot in time based on specific experimental tools and conditions. "The slowdown we observe does not imply that current AI tools do not often improve developer's productivity - we find evidence that the high developer familiarity with repositories and the size and maturity of the repositories both contribute to the observed slowdown, and these factors do not apply in many software development settings," they say. The authors go on to note that their findings don't imply current AI systems are not useful or that future AI models won't do better. ®

AI slows down some experienced software developers, study finds

SAN FRANCISCO, July 10 (Reuters) - Contrary to popular belief, using cutting-edge artificial intelligence tools slowed down experienced software developers when they were working in codebases familiar to them, rather than supercharging their work, a new study found. AI research nonprofit METR conducted the in-depth study, opens new tab on a group of seasoned developers earlier this year while they used Cursor, a popular AI coding assistant, to help them complete tasks in open-source projects they were familiar with. Before the study, the open-source developers believed using AI would speed them up, estimating it would decrease task completion time by 24%. Even after completing the tasks with AI, the developers believed that they had decreased task times by 20%. But the study found that using AI did the opposite: it increased task completion time by 19%. The study's lead authors, Joel Becker and Nate Rush, said they were shocked by the results: prior to the study, Rush had written down that he expected "a 2x speed up, somewhat obviously." The findings challenge the belief that AI always makes expensive human engineers much more productive, a factor that has attracted substantial investment into companies selling AI products to aid software development. AI is also expected to replace entry-level coding positions. Dario Amodei, CEO of Anthropic, recently told Axios that AI could wipe out half of all entry-level white collar jobs in the next one to five years. Prior literature on productivity improvements has found significant gains: one study found using AI sped up coders by 56%, opens new tab, another study found developers were able to complete 26% more tasks, opens new tab in a given time. But the new METR study shows that those gains don't apply to all software development scenarios. In particular, this study showed that experienced developers intimately familiar with the quirks and requirements of large, established open source codebases experienced a slowdown. Other studies often rely on software development benchmarks for AI, which sometimes misrepresent real-world tasks, the study's authors said. The slowdown stemmed from developers needing to spend time going over and correcting what the AI models suggested. "When we watched the videos, we found that the AIs made some suggestions about their work, and the suggestions were often directionally correct, but not exactly what's needed," Becker said. The authors cautioned that they do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with. Still, the majority of the study's participants, as well as the study's authors, continue to use Cursor today. The authors believe it is because AI makes the development experience easier, and in turn, more pleasant, akin to editing an essay instead of staring at a blank page. "Developers have goals other than completing the task as soon as possible," Becker said. "So they're going with this less effortful route." Reporting by Anna Tong in San Francisco; Editing by Sonali Paul Our Standards: The Thomson Reuters Trust Principles., opens new tab Suggested Topics:Artificial Intelligence Anna Tong Thomson Reuters Anna Tong is a correspondent for Reuters based in San Francisco, where she reports on the technology industry. She joined Reuters in 2023 after working at the San Francisco Standard as a data editor. Tong previously worked at technology startups as a product manager and at Google where she worked in user insights and helped run a call center. Tong graduated from Harvard University.

AI Might Be Slowing Down Some Employees' Work, a Study Says

This is a little technical, but it's a fascinating result that cuts through some of the hype surrounding this very buzzy technology. Essentially while coding experts thought AI would help them in their work, the hard data shows it actually slowed down the coders who used it. As Marcus put it, "if this is a general, replicable finding," that can be extended beyond the realm of coding into other work sectors where AI tools are becoming commonplace, "it's a serious blow to generative AI's flagship use case. People might be imagining productivity gains that they are not getting, and ignoring real-world costs, to boot." The real-world costs Marcus mentions here are evident from METR's study: slowing down an expert developer by nearly 20 percent means their efficiency is taking a hit, which has direct business costs. Naturally, there are some details of the study that need to be considered. For example, this was a short window of time (early 2025) that encompassed a suite of AI tools that is evolving and improving every day. It's also a niche test group. METR noted they were "experienced developers working on large, complex codebases that, often, they helped build," and this factor may have played into the slowdown that AI use caused. METR noted it expects "AI tools provide greater productivity benefits in other settings (e.g. on smaller projects, with less experienced developers, or with different quality standards)." The impact of AI tools on the coding profession remains a subject of heated debate. A recent Microsoft study, for example, raised an alarm that some young coders, fresh from college, already rely on AI help so much that they don't really understand the details or science behind the code they write -- potentially setting them up for failure later on.

AI slows down some experienced software developers, study finds - The Economic Times

Contrary to popular belief, using cutting-edge artificial intelligence tools slowed down experienced software developers when they were working in codebases familiar to them, rather than supercharging their work, a new study found. AI research nonprofit METR conducted the in-depth study on a group of seasoned developers earlier this year while they used Cursor, a popular AI coding assistant, to help them complete tasks in open-source projects they were familiar with. Before the study, the open-source developers believed using AI would speed them up, estimating it would decrease task completion time by 24%. Even after completing the tasks with AI, the developers believed that they had decreased task times by 20%. But the study found that using AI did the opposite: it increased task completion time by 19%. The study's lead authors, Joel Becker and Nate Rush, said they were shocked by the results: prior to the study, Rush had written down that he expected "a 2x speed up, somewhat obviously." The findings challenge the belief that AI always makes expensive human engineers much more productive, a factor that has attracted substantial investment into companies selling AI products to aid software development. AI is also expected to replace entry-level coding positions. Dario Amodei, CEO of Anthropic, recently told Axios that AI could wipe out half of all entry-level white collar jobs in the next one to five years. Prior literature on productivity improvements has found significant gains: one study found using AI sped up coders by 56%, another study found developers were able to complete 26% more tasks in a given time. But the new METR study shows that those gains don't apply to all software development scenarios. In particular, this study showed that experienced developers intimately familiar with the quirks and requirements of large, established open source codebases experienced a slowdown. Other studies often rely on software development benchmarks for AI, which sometimes misrepresent real-world tasks, the study's authors said. The slowdown stemmed from developers needing to spend time going over and correcting what the AI models suggested. "When we watched the videos, we found that the AIs made some suggestions about their work, and the suggestions were often directionally correct, but not exactly what's needed," Becker said. The authors cautioned that they do not expect the slowdown to apply in other scenarios, such as for junior engineers or engineers working in codebases they aren't familiar with. Still, the majority of the study's participants, as well as the study's authors, continue to use Cursor today. The authors believe it is because AI makes the development experience easier, and in turn, more pleasant, akin to editing an essay instead of staring at a blank page. "Developers have goals other than completing the task as soon as possible," Becker said. "So they're going with this less effortful route."