Stanford's 2025 AI Index Report: Rapid Progress, Rising Costs, and Global Competition

These 12 Eye-Opening Graphs Reveal the State of AI in 2025

Eliza Strickland is a senior editor at IEEE Spectrum covering AI and biomedical engineering. If you read the news about AI, you may feel bombarded with conflicting messages: AI is booming. AI is a bubble. AI's current techniques and architectures will keep producing breakthroughs. AI is on an unsustainable path and needs radical new ideas. AI is going to take your job. AI is mostly good for turning your family photos into Studio Ghibli-style animated images. While there are many different ways to measure which country is "ahead" in the AI race (journal articles published or cited, patents awarded, etc.), one straightforward metric is who's putting out models that matter. The research institute Epoch AI has a database of influential and important AI models that extends from 1950 to the present, from which the AI Index drew the information shown in this chart. Last year, 40 notable models came from the United States, while China had 15 and Europe had 3 (incidentally, all from France). Another chart, not shown here, indicates that almost all of those 2024 models came from industry rather than academia or government. As for the decline in notable models released from 2023 to 2024, the index suggests it may be due to the increasing complexity of the technology and the ever-rising costs of training. Yowee, but it's expensive! The AI Index doesn't have precise data, because many leading AI companies have stopped releasing information about their training runs. But the researchers partnered with Epoch AI to estimate the costs of at least some models based on details gleaned about training duration, type and quantity of hardware, and the like. The most expensive model for which they were able to estimate the costs was Google's Gemini 1.0 Ultra, with a breathtaking cost of about US $192 million. The general scale up in training costs coincided with other findings of the report: Models are also continuing to scale up in parameter count, training time, and amount of training data. Not included in this chart is the Chinese upstart DeepSeek, which rocked financial markets in January with its claim of training a competitive large language model for just $6 million -- a claim that some industry experts have disputed. AI Index steering committee co-director Yolanda Gil tells IEEE Spectrum that she finds DeepSeek "very impressive," and notes that the history of computer science is rife with examples of early inefficient technologies giving way to more elegant solutions. "I'm not the only one who thought there would be a more efficient version of LLMs at some point," she says. "We just didn't know who would build it and how." The ever-increasing costs of training (most) AI models risks obscuring a few positive trends that the report highlights: Hardware costs are down, hardware performance is up, and energy efficiency is up. That means inference costs, or the expense of querying a trained model, are falling dramatically. This chart, which is on a logarithmic scale, shows the trend in terms of AI performance per dollar. The report notes that the blue line represents a drop from $20 per million tokens to $0.07 per million tokens; the pink line shows a drop from $15 to $0.12 in less than a year's time. While energy efficiency is a positive trend, let's whipsaw back to a negative: Despite gains in efficiency, overall power consumption is up, which means that the data centers at the center of the AI boom have an enormous carbon footprint. The AI Index estimated the carbon emissions of select AI models based on factors such as training hardware, cloud provider, and location, and found that the carbon emissions from training frontier AI models have steadily increased over time -- with DeepSeek being the outlier. The worst offender included in this chart, Meta's Llama 3.1, resulted in an estimated 8,930 tonnes of CO emitted, which is the equivalent of about 496 Americans living a year of their American lives. That massive environmental impact explains why AI companies have been embracing nuclear as a reliable source of carbon-free power. The United States may still have a commanding lead on the quantity of notable models released, but Chinese models are catching up on quality. This chart shows the narrowing performance gap on a chatbot benchmark. In January 2024, the top U.S. model outperformed the best Chinese model by 9.26 percent; by February 2025, this gap had narrowed to just 1.70 percent. The report found similar results on other benchmarks relating to reasoning, math, and coding. This year's report highlights the undeniable fact that many of the benchmarks we use to gauge AI systems' capabilities are "saturated" -- the AI systems get such high scores on the benchmarks that they're no longer useful. It has happened in many domains: general knowledge, reasoning about images, math, coding, and so on. Gil says she has watched with surprise as benchmark after benchmark has been rendered irrelevant. "I keep thinking [performance] is going to plateau, that it's going to reach a point where we need new technologies or radically different architectures" to continue making progress, she says. "But that has not been the case." In light of this situation, determined researchers have been crafting new benchmarks that they hope will challenge AI systems. One of those is Humanity's Last Exam, which consists of extremely challenging questions contributed by subject-matter experts hailing from 500 institutions worldwide. So far, it's still hard for even the best AI systems: OpenAI's reasoning model, o1, has the top score so far with 8.8 percent correct answers. We'll see how long that lasts. Today's generative AI systems get their smarts by training on vast amounts of data scraped from the Internet, leading to the oft-stated idea that "data is the new oil" of the AI economy. As AI companies keep pushing the limits of how much data they can feed into their models, people have started worrying about "peak data," and when we'll run out of the stuff. One issue is that websites are increasingly restricting bots from crawling their sites and scraping their data (perhaps due to concerns that AI companies are profiting from the websites' data while simultaneously killing their business models). Websites state these restrictions in machine readable robots.txt files. This chart shows that 48 percent of data from top web domains is now fully restricted. But Gil says it's possible that new approaches within AI may end the dependence on huge data sets. "I would expect that at some point the amount of data is not going to be as critical," she says. The corporate world has turned on the spigot for AI funding over the past five years. And while overall global investment in 2024 didn't match the giddy heights of 2021, it's notable that private investment has never been higher. Of the $150 billion in private investment in 2024, another chart in the index (not shown here) indicates that about $33 billion went to investments in generative AI. Presumably, corporations are investing in AI because they expect a big return on investment. This is the part where people talk in breathless tones about the transformative nature of AI and about unprecedented gains in productivity. But it's fair to say that corporations haven't yet seen a transformation that results in significant savings or substantial new profits. This chart, with data drawn from a McKinsey survey, shows that of those companies that reported cost reductions, most had savings of less than 10 percent. Of companies that had a revenue increase due to AI, most reported gains of less than 5 percent. That big payoff may still be coming, and the investment figures suggest that a lot of corporations are betting on it. It's just not here yet. AI for science and medicine is a mini-boom within the AI boom. The report lists a variety of new foundation models that have been released to help researchers in fields such as materials science, weather forecasting, and quantum computing. Many companies are trying to turn AI's predictive and generative powers into profitable drug discovery. And OpenAI's o1 reasoning model recently scored 96 percent on a benchmark called MedQA, which has questions from medical board exams. But overall, this seems like another area of vast potential that hasn't yet translated into significant real-world impact -- in part, perhaps, because humans still haven't figured out quite how to use the technology. This chart shows the results of a 2024 study that tested whether doctors would make more accurate diagnoses if they used GPT-4 in addition to their typical resources. They did not, and it also didn't make them faster. Meanwhile, GPT-4 on its own outperformed both the human-AI teams and the humans alone. In the United States, this chart shows that there has been plenty of talk about AI in the halls of Congress, and very little action. The report notes that action in the United States has shifted to the state level, where 131 bills were passed into law in 2024. Of those state bills, 56 related to deepfakes, prohibiting either their use in elections or for spreading nonconsensual intimate imagery. Beyond the United States, Europe did pass its AI Act, which places new obligations on companies making AI systems that are deemed high risk. But the big global trend has been countries coming together to make sweeping and non-binding pronouncements about the role that AI should play in the world. So there's plenty of talk all around. Whether you're a stock photographer, a marketing manager, or a truck driver, there's been plenty of public discourse about whether or when AI will come for your job. But in a recent global survey on attitudes about AI, the majority of people did not feel threatened by AI. While 60 percent of respondents from 32 countries believe that AI will change how they do their jobs, only 36 percent expected to be replaced. "I was really surprised" by these survey results, says Gil. "It's very empowering to think, 'AI is going to change my job, but I will still bring value.'" Stay tuned to find out if we all bring value by managing eager teams of AI employees.

AI costs drop 280-fold, but harmful incidents rise 56% in last year -- Stanford 2025 AI report highlights China-US competition

Artificial intelligence investments are higher than ever amidst turbulent conditions The cost to prompt high-end AI LLMs has plummeted from $20 per million tokens to $0.07 per million in just 18 months, according to Stanford's 2025 AI Index Report. A panoramic view of the worldwide AI landscape, Stanford's annual report also reports a serious need for more responsible AI guardrails and a tightening race between the US and China's emerging AI tech. Stanford University's Institute for Human-Centered AI (HAI) has published its annual AI Index Report since 2017, with its recent reports regularly cited by world governments. HAI has collected and collated data on AI's myriad facets, studying investments into the market, where and how the tech is most used, and where it is most lacking. This year's report offers serious insights on the growth of AI over 2024, and predicts where it will likely go next. Artificial intelligence models have become significantly cheaper to use in only the last year -- but at the same time they have become more expensive to train. This apparent contradiction is illustrated in HAI's helpful graphs accompanying the study: as major companies have ballooned their investments into their flagship models, the cost to operate and query the same models has dropped significantly. OpenAI, Meta, and Google have all measurably increased the costs invested into their flagship language models. On average, each company spent 28 times as much money training its most recent flagship AI model as it did training the predecessor (Meta's $3 million to $170 million jump was the largest). Other relative newcomer, such as Mistral and xAI have also entered the game spending high -- Grok-2 cost an estimated $107 million to train. The cost to train these LLMs does not seem to be dropping anytime soon, either. xAI's Grok-3, which released to the public in February, is claimed to have used 10 times the GPUs of Grok-2's training. Grok-3 had no official price tag, but it could potentially have cost $1 billion or more to complete. If these numbers to train a computer program seem astronomical, it's because they are. While these trillion-dollar companies invest hundreds of billions into the next generation of AI, the price to reach GPT-3.5 performance has shrunk. The cost to inference a model at GPT-3.5-level performance -- defined by HAI as 64.8% accuracy -- fell 280 times from November 2022 to October 2024. Falling hardware and operation costs of smaller AI models contributed heavily to this price drop. Enterprise AI hardware costs have fallen 30% in the last year, with new hardware also being 40% more energy efficient. Companies are likely to continue spending more and more money on training flagship models every year, but typical users content with GPT-3.5 performance will find their costs becoming lower and lower. The United States has been the highest spender and top performer in artificial intelligence since the tech's breakthrough into the mainstream. However, China is close behind on the AI race. The top performing U.S.- and China-based LLMs are getting closer and closer in performance when tested in industry benchmarks. As the above graph displays, the U.S.'s best model only beat China's champion by 1.70%, as voted on by blind trials in LMSYS Chatbot Arena. Results from top benchmarks MMLU and HumanEval have also begun to approach even results, with the U.S. still managing to stay barely ahead. The United States still handily beat China in quantity, if not quality. In a HAI collection of highly notably AI models, the United States took an easy lead with 40 of 2024's most notable LLMs. China fell distantly behind with 15, and all of Europe only contributed 3 models to the race. HAI's chapter on Responsible AI paints a starker picture on the reality of using AI, which carries a non-zero level of risk. The AI Incident Database (AIID), a non-profit research organization dedicated to collecting information on harmful AI incidents, reportedly saw a disturbing increase in harmful AI incidents over 2024. 233 harmful or dangerous incidents were reported to the AIID in 2024, surpassing the ~150 reports in 2023 and ~100 in 2022. Some of the most severe incidents in 2024 were listed in HAI's complete Chapter 3. These incidents included a false ID of a shopper thought to be a shoplifter with anti-theft AI, deepfake pornography, and instances of chatbots encouraging harmful behavior, including self-harm. Notably, few AI companies accept responsibility for AI incidents when they occur, with several of the above incidents leading to refusals to issue an apology or reparation from the involved companies. HAI's full 2025 AI Index can be found on the Stanford HAI website. The 8-chapter study covers a much broader range than could be covered here, representing many hours of reading. The AI landscape is broader and more invested in than ever before, making recent tariffs that threaten to shake up the status quo frightening to the still-nascent industry. The future of the tech is yet unknown, though hopefully safety and responsibility in training and application take up a more dominant share of attention in the coming years.

Report: Massive amounts of $$$ still being pumped into AI

AI continues to improve - at least according to benchmarks. But the promised benefits have largely yet to materialize while models are increasing in size and becoming more computationally demanding, and greenhouse gas emissions from AI training continue to rise. These are some of the takeaways from the AI Index Report 2025 [PDF], a lengthy and in-depth publication from Stanford University's Institute for Human-Centered AI (HAI) that covers development, investment, adoption, governance and even global attitudes towards artificial intelligence, giving a snapshot of the current state of play. In terms of performance, the researchers state that AI models are increasingly mastering new and challenging benchmarks designed to test their capabilities, including MMMU, GPQA, and SWE-bench. With the latter, which measures success in solving actual coding problems from GitHub, AI systems managed just 4.4 percent in 2023, but this jumped to 71.7 percent last year. According to HAI, last year's AI Index highlighted that many models had already surpassed human performance on a range of tasks, with only a few exceptions, such as competition-level mathematics and visual commonsense reasoning. This trend largely continued over the past year, with models closing performance gaps and matching or exceeding humans on even more demanding benchmarks. If you think that sounds depressing, the report also stresses that complex reasoning is still out of reach for AI models. Even with mechanisms such as chain-of-thought reasoning to boost their performance, large language models (LLMs) are unable to reliably solve problems for which a solution can be found using logical reasoning, making them unsuitable still for many applications. However, HAI highlights the enormous level of investment still being pumped into the sector, with global corporate AI investment reaching $252.3 billion in 2024, up 26 percent for the year. Most of this is in the US, which hit $109.1 billion, nearly 12 times higher than China's $9.3 billion and 24 times the UK's $4.5 billion, it says. Most companies that report financial impacts from using AI within a business function estimate the benefits as being at low levels Despite all this investment, "most companies that report financial impacts from using AI within a business function estimate the benefits as being at low levels," the report writes. It says that 49 percent of organizations using AI in service operations reported cost savings, followed by supply chain management (43 percent) and software engineering (41 percent), but in most cases, the cost savings are less than 10 percent. When it comes to revenue gains, 71 percent of respondents using AI in marketing and sales reported gains, while 63 percent in supply chain management and 57 percent in service operations, but the most common level of revenue increase is less than 5 percent. The report claims, "AI is beginning to deliver financial impact across business functions, but most companies are early in their journeys" - an excuse we've been hearing for some time now. Meanwhile, despite the modest returns, the HAI report warns that the amount of compute used to train top-notch AI models is doubling approximately every 5 months, the size of datasets required for LLM training is doubling every eight months, and the energy consumed for training is doubling annually. This is leading to rapidly increasing greenhouse gas emissions resulting from AI training, the report finds. It says that early AI models such as AlexNet over a decade ago caused only modest CO₂ emissions of 0.01 tons, while GPT-4 (2023) was responsible for emitting 5,184 tons, and Llama 3.1 405B (2024) pumping out 8,930 tons. This compares with about 18 tons of carbon a year the average American emits, it claims. While North America maintains its leadership in organizations' use of AI, Greater China demonstrated one of the most significant year-over-year growth rates To set against this, the energy efficiency of infrastructure is increasing by 40 percent annually, while hardware performance, as measured in 16-bit floating-point operations, grows by 43 percent annually, doubling every 1.9 years. According to HAI, the US and China are vying for AI leadership, with America as the primary source of models, producing no fewer than 40 during 2024. However, China leads in AI research publication totals, while the United States leads in influential research, with the most cited AI publications. "While North America maintains its leadership in organizations' use of AI, Greater China demonstrated one of the most significant year-over-year growth rates," the researchers pointed out. The performance gap between models produced by the US and China is also shrinking, with American models significantly outperforming their Chinese counterparts in 2023, a trend that the HAI report says no longer holds. Finally, a chapter of the report looks at public perception and sentiment surrounding AI, and suggests there is growing public anxiety about the technology. It says that two thirds of people now expect that AI-powered products and services will significantly impact daily life within the next 3 to 5 years. But readers of The Register may be surprised to learn that optimism about AI is growing in areas that were previously the most skeptical about it, including the US and UK. In 2022, about 38 percent of Brits and 35 percent of Yanks viewed AI as having more benefits than drawbacks, the HAI asserts. By 2024, those numbers had risen by 8 and 4 percentage points, respectively. As far as workers go, 60 percent of respondents believe that AI will change how individuals do their jobs in the next five years, while 36 percent expect that AI will replace their jobs in the next five years. Overall, AI is viewed as a potential time saver, but not many people share the AI industry's massive hype about the economic benefits. While 55 percent think the technology will save time, only 38 percent expect AI to improve health outcomes, 36 percent expect AI will improve their national economy, and only 31 percent see a positive impact on the jobs market. All this is little more than a summary of the AI Index report, which spans more than 450 pages and is festooned with charts and graphs to illustrate its many points. HAI says the data it is based upon is broadly sourced and is available via its website. ®

Stanford HAI's annual report highlights rapid adoption and growing accessibility of powerful AI systems - SiliconANGLE

Stanford HAI's annual report highlights rapid adoption and growing accessibility of powerful AI systems Researchers from Stanford University early today published the latest edition of their annual AI Index Report, detailing the growing influence of artificial intelligence technologies on our society and the global economy. The Stanford Institute for Human-Centered Artificial Intelligence, known as Stanford HAI, has been publishing its annual reports on the state of the AI industry since 2017. This year's edition, the eighth, is the "most comprehensive" report to date, spanning more than 430 pages. The authors say it's arriving at a critical juncture as the influence of AI across society rapidly accelerates with the emergence of increasingly capable and sophisticated AI systems. The report contains an in-depth analysis of the latest new AI models and development techniques, along with a detailed look at the evolving landscape of AI hardware, new estimates on the cost of AI inference, and studies on some of the hottest new industry trends. It also looks at the expanding role of AI in areas such as science and medicine, and the growing emphasis on responsible AI practices. According to Stanford HAI's analysis, AI developers continued to make great strides in overall performance over the last year, with record-breaking scores on key benchmarks including the MMLU, GPQA and SWE-bench, which were established in 2023 to test the limits of advanced AI systems. Beyond these achievements, the last year also saw significant advances in areas such as AI-generated video and AI agents, which are autonomous systems that can perform tasks on behalf of people with minimal supervision. The report says that the U.S. is still the nation to beat in terms of top-performing models, but worryingly for that country, China is closing the gap fast. In 2024, U.S.-based organizations published 40 "notable AI models" compared to just 15 from Chinese firms and three from Europe. However, those Chinese models have made some impressive strides, achieving close to parity with their U.S. counterparts on key benchmarks such as MMLU and HumanEval One reason for China gaining ground is that AI systems are becoming increasingly efficient, meaning they're more affordable to develop and therefore more accessible. The report found that the inference cost for a system matching the performance of OpenAI's GPT-3.5 has fallen by 280-times over the last two years, while in terms of hardware, costs declined 30% in the last 12 months. AI models are also consuming 40% less energy than a year ago, while the emergence of more powerful "open-weights" models that can be easily customized for different use cases is also lowering the barrier to entry. That may explain why regions such as the Middle East, Southeast Asia and Latin America were able to launch their first powerful LLMs in the last year. However, the report notes that even the top AI developers are still struggling to match the capabilities of real humans in terms of AI's reasoning skills. While learning-based systems that generate and verify hypotheses are performing well on tasks like the International Math Olympiad, such systems still struggle with "logic-heavy tasks" such as arithmetic and planning, which limits their adoption. The advances in AI continue to be fueled by billions of dollars in investment, and the lion's share of that cash comes from America. In the U.S., private investors threw more than $109 billion at AI startups and projects, almost 12 times the $9.3 billion invested in China, and 24 times as much as the U.K.'s $4.5 billion. Within the AI industry, generative AI continues to attract the most cash at $33.9 billion in private investment globally, up 18% from 2023. Investors continue to pour billions of dollars into AI because the technology is seeing rapid adoption by businesses. According to the report, 78% of global enterprises confirmed they have deployed AI systems in their workflows in 2024, up from 55% one year earlier. This is being driven by what Stanford HAI says is a "growing body of research" that shows AI is boosting productivity in key industries while helping to narrow skills gaps across the workforce. The report also noted the impact of AI on science, noting that Nobel Prizes were awarded to researchers that utilized deep learning systems to advance physics, and to a second team that applied AI to protein folding. AI increasingly permeates our lives in areas ranging from healthcare to transportation. In the U.S., for instance, the Federal Drug Administration has now approved more than 950 AI-powered medical devices, up from just six in 2015 and 221 in 2023. AI-powered self-driving cars, such as Waymo's autonomous taxis, are also a thing, providing 150,000 rides per week. However, the U.S. public is still somewhat opposed to having too much AI in its lives. According to one study, just 39% of Americans said they see AI as more beneficial than harmful. That compares with places such as China, Thailand and Indonesia, where more than 75% of people say they welcome the advantages of AI. Globally, there remains a significant gap in terms of education. While two-thirds of the world's nations now offer, or plan to offer computer science education to K-12 students, access remains limited in areas like Africa because of a lack of basic infrastructure such as electricity. Moreover, there are still challenges in terms of teaching basic AI skills In the U.S., 81% of teachers say they believe AI should be a foundational element of computer science education, but less than half say they're able to teach it. The report highlights a worrying rise in the number of AI-related security incidents over the last year, and notes that despite these, standardized RAI evaluations are rare even among the biggest AI developers, such as OpenAI, Google LLC and Meta Platforms Inc. On the other hand, the emergence of new benchmarks such as AIR-Bench, HELM Safety and FACTS is a promising development as Stanford HAI believes they can be useful tools for the industry to assess the safety and accuracy of their models. There is, however, still an alarming gap in the business world in terms of those that recognize AI's risks and those who take action to try to mitigate those dangers. That's in contrast to most governments, which are showing "increased urgency" with regard to responsible AI. For instance, last year global organizations such as the European Union, African Union and the United Nations all published frameworks focused on AI transparency, trustworthiness and other key principles of responsible AI. Looking at the U.S., federal agencies issued 59 AI-based regulations in 2024, more than twice as many as they did in 2023. On a global scale, AI-based legislative mentions increased by an average of 21% across 75 countries.