Nvidia's AI Dominance Faces Challenges as Scaling Laws Show Signs of Slowing

Nvidia and the AI boom faces a scaling problem

The computational "law" that made Nvidia the world's most valuable company is starting to break down. This is not the famous Moore's Law, the semiconductor-industry maxim that chip performance will increase by doubling transistor density every two years. For many in Silicon Valley, Moore's Law has been displaced as the dominant predictor of technological progress by a new concept: the "scaling law" of artificial intelligence. This posits that putting more data into a bigger AI model -- in turn, requiring more computing power -- delivers smarter systems. This insight put a rocket under AI's progress, transforming the focus of development from solving tough science problems to the more straightforward engineering challenge of building ever-bigger clusters of chips -- usually Nvidia's. The scaling law had its coming-out moment with the launch of ChatGPT. The breakneck pace of improvement in AI systems in the two years since then seemed to suggest the rule might hold true right until we reach some kind of "super intelligence", perhaps within this decade. Over the past month, however, industry rumblings have grown louder that the latest models from the likes of OpenAI, Google and Anthropic have not shown the expected improvements in line with the scaling law's projections. "The 2010s were the age of scaling, now we're back in the age of wonder and discovery once again," OpenAI co-founder Ilya Sutskever told Reuters recently. This is the man who a year ago said he thought it was "pretty likely the entire surface of the earth will be covered with solar panels and data centres" to power AI. Until recently, the scaling law was applied to "pre-training": the foundational step in building a large AI model. Now, AI executives, researchers and investors are conceding that AI model capabilities are -- as Marc Andreessen put it on his podcast -- "topping out" on pre-training alone, meaning that more work is required after the model is built to keep the advances coming. Some of the scaling law's earliest adherents such as Microsoft chief Satya Nadella, have attempted to recast its definition. It doesn't matter if pre-training yields shrinking returns, defenders argue, because models can now "reason" when asked a complex question. "We are seeing the emergence of a new scaling law," Nadella said recently, referring to OpenAI's new o1 model. But this kind of fudging should make Nvidia's investors nervous. Of course, the scaling "law" was never an ironclad rule, just as there was no inherent factor that allowed engineers at Intel to keep increasing transistor density in line with Moore's Law. Rather, these concepts serve as organising principles for the industry, driving competition. Nonetheless, the scaling law hypothesis has fuelled "fear of missing out" on the next big tech transition, leading to unprecedented investment by Big Tech on AI. Capital expenditures at Microsoft, Meta, Amazon and Google are set to exceed $200bn this year and top $300bn next year, according to Morgan Stanley Nobody wants to be last to build super intelligence. But if bigger no longer means better in AI, will those plans be curtailed? Nvidia stands to suffer more than most if they are. When the chipmaker reported its earnings last week, the first question from analysts was about scaling laws. Jensen Huang, Nvidia's chief executive, insisted pre-training scaling was "intact" but admitted it is "not enough" by itself. The good news for Nvidia, Huang argued, is that the solution will require even more of its chips: so-called "test time scaling", as AI systems like OpenAI's o1 have to "think" for longer to come up with smarter responses. This may well be true. While training has soaked up most of Nvidia's chips so far, demand for computing power for "inference" -- or how models respond to each individual query -- is expected to grow rapidly as more AI applications emerge. People involved in the construction of this AI infrastructure believe that the industry is going to be playing catch-up on inference for at least another year. "Right now, this is a market that's going to need more chips, not less," Microsoft president Brad Smith told me. But longer term, the chase for chips to power ever-larger models before they are rolled out has been replaced by something that is more closely tied to AI usage. Most businesses are still searching for AI's killer app, especially in areas that would require the nascent "reasoning" capabilities of o1. Nvidia became the world's most valuable company during the speculative phase of the AI buildout. The scaling law debate underlines just how much its future depends on Big Tech getting tangible returns on those huge investments.

Nvidia's CEO defends his moat as AI labs change how they improve their AI models

Nvidia raked in more than $19 billion in net income during the last quarter, the company reported on Wednesday, but that did little to assure investors that its rapid growth would continue. On its earnings call, analysts prodded CEO Jensen Huang about how Nvidia would fare if tech companies start using new methods to improve their AI models. The method that underpins OpenAI's o1 model, or "test-time scaling," came up quite a lot. It's the idea that AI models will give better answers if you give them more time and computing power to "think" through questions. Specifically, it adds more compute to the AI inference phase, which is everything that happens after a user hits enter on their prompt. Nvidia's CEO was asked whether he was seeing AI model developers shift over to these new methods, and how Nvidia's older chips would work for AI inference. Huang told investors that o1, and test-time scaling more broadly, could play a larger role in Nvidia's business moving forward, calling it "one of the most exciting developments" and "a new scaling law." Huang did his best to ensure investors that Nvidia is well-positioned for the change. The Nvidia CEO's remarks aligned with what Microsoft CEO Satya Nadella said onstage at a Microsoft event on Tuesday: o1 represents a new way for the AI industry to improve its models. This is a big deal for the chip industry because it places a greater emphasis on AI inference. While Nvidia's chips are the gold standard for training AI models, there's a broad set of well-funded startups creating lightning-fast AI inference chips, such as Groq and Cerebras. It could be a more competitive space for Nvidia to operate in. Despite recent reports that improvements in generative models are slowing, Huang told analysts that AI model developers are still improving their models by adding more compute and data during the pretraining phase. Anthropic CEO Dario Amodei also said on Wednesday during an onstage interview at the Cerebral Valley summit in San Francisco that he is not seeing a slowdown in model development. "Foundation model pretraining scaling is intact and it's continuing," said Huang on Wednesday. "As you know, this is an empirical law, not a fundamental physical law, but the evidence is that it continues to scale. What we're learning, however, is that it's not enough," said Huang. That's certainly what Nvidia investors wanted to hear, since the chipmaker's stock has soared more than 180% in 2024 by selling the AI chips that OpenAI, Google, and Meta train their models on. However, Andreessen Horowtiz partners and several other AI executives have previously said that these methods are already starting to show diminishing returns. Huang noted that most of Nvidia's computing workloads today are around the pre-training of AI models - not inference -- but he attributed that more to where the AI world is today. He said that one day, there will simply be more people running AI models, meaning more AI inference will happen. Huang noted that Nvidia is the largest inference platform in the world today and the company's scale and reliability gives it a huge advantage compared to startups. "Our hopes and dreams are that someday, the world does a ton of inference, and that's when AI has really succeeded," said Huang. "Everybody knows that if they innovate on top of CUDA and Nvidia's architecture, they can innovate more quickly, and they know that everything should work."

After Posting 94% Revenue Jump, Nvidia CEO Says Company Has Room to Scale | PYMNTS.com

"Many AI services are running 24/7, just like any factory," Huang told the earnings call audience. "We're going to see this new type of system come online. And I call it [the company's data centers] an AI factory because that's really close to what it is. It's unlike a data center of the past. "And these fundamental trends are really just beginning. We expect this to happen, this growth, this modernization and the creation of a new industry to go on for several years." Huang and CFO Colette Kress clearly feel that the company's best days are ahead of it, even as analysts question whether or not it can keep up the pace in several areas: large language model (LLM) development, AI usage scale and the torrid revenue growth it has achieved over the past two years. Their reasons for optimism ranged from consumer adoption rates to the coming explosion of enterprise and industrial AI and the long list of companies that rely on Nvidia data centers and chips (whose manufacturing is outsourced) for their own applications. By way of background, an AI data center is a specialized facility designed to handle the heavy computational demands of AI workloads, essentially providing the infrastructure needed to train and deploy complex machine learning models and algorithms by processing massive amounts of data using high-performance servers, specialized hardware accelerators, and advanced networking capabilities, all optimized for AI operations. In simpler terms, it's a data center specifically built to power AI applications at scale. If there was a theme on the call and in the earnings materials, it was that laundry list of companies from Alphabet to Meta to Microsoft to Oracle to Volvo that are hooked into Nvidia. But when that list wasn't running, Huang and Kress faced some tough questions from analysts, ranging from scaling for LLM development to a potential controversy about reported overheating issues for the companies seven-chip Blackwell set of GPUs that it is banking its next few years on. For perspective, the company's Q3 earnings were achieved without shipping any newly designed chips. Blackwell is the new addition, and demand, according to Kress, is "staggering." Despite some concerns about a potential slowdown in the scaling of LLMs, Huang maintained that there is still ample opportunity for growth. He emphasized that the scaling of foundation models is "intact and continuing," citing ongoing advancements in post-training scaling and inference-time scaling. Post-training scaling, which initially involved reinforcement learning with human feedback, has evolved to incorporate AI feedback and synthetic data generation. Meanwhile, inference-time scaling, demonstrated by OpenAI's ChatGPT-01, allows for improved answer quality with increased processing time. Huang expressed optimism about the continued growth of the AI market, driven by the ongoing modernization of data centers and the emergence of generative AI applications. He described the shift from traditional coding to machine learning as a fundamental change that will require companies to upgrade their infrastructure to support AI workloads. Huang also highlighted the emergence of generative AI, which he likened to the advent of the iPhone, as a completely new capability that will create new market segments and opportunities. He cited examples such as OpenAI, Runway and Harvey, which provide basic intelligence, digital artist intelligence, and legal intelligence, respectively. Nvidia's Blackwell architecture is designed to meet the demands of this evolving AI landscape. The company has developed seven custom chips for the Blackwell system, which can be configured for air-cooled or liquid-cooled data centers and support various MVlink and CPU options. Huang acknowledged the engineering challenges involved in integrating these systems into diverse data center architectures but remained confident in Nvidia's ability to execute. He cited examples of successful collaborations with major cloud service providers (CSPs) such as Dell, Corweave, Oracle, Microsoft and Google. Nvidia is also seeing strong growth in the enterprise and industrial AI sectors. The company's Nvidia AI Enterprise platform is being used by industry leaders to build copilots and agents. In the industrial AI space, Nvidia's Omniverse platform is enabling the development and operation of industrial AI and robotics applications. Major manufacturers like Foxconn are adopting Omniverse to accelerate their businesses, automate workflows, and improve operating efficiency.

Prediction: This Massive Risk Could Derail Nvidia Stock | The Motley Fool

Demand for artificial intelligence (AI) chips, particularly ultra-powerful GPUs from Nvidia (NVDA 0.66%) used to train the most advanced AI models, appears to be insatiable. Nvidia's data center segment generated more than $30 billion in revenue during the third quarter alone, up by nearly a factor of 10 compared to two years ago. Tech giants are scrambling to build AI data centers packed with Nvidia's GPUs, spending untold billions on hardware. Up until now, each new AI model to come out of OpenAI, one of those tech giants, or anyone else who's caught AI fever has meaningfully improved on its predecessor. OpenAI's GPT-4 is far more capable than GPT-3, and Alphabet's Gemini AI models blow its older models out of the water. But those improvements came at a cost. GPT-4 is estimated to have cost around $100 million to train, whereas GPT-3 may have cost just a few million dollars. Anthropic CEO Dario Amodei expects the next generation of AI models to cost around $1 billion to produce. Buying and then running many thousands of high-powered GPUs is expensive, and collecting mass amounts of training data is no picnic, either. Large language models like GPT-4 work by predicting the next token in the output. This works pretty well in a lot of cases. The best AI models can produce high-quality text, generate convincing images, and even appear to do some fairly advanced reasoning. More training data generally produces better results, as does more time churning through that data in the training process. However, AI companies appear to be reaching a limit. The rate of improvement in AI models is slowing, even with more data and more computational horsepower. Marc Andreessen, co-founder of venture capital firm a16z, recently noted that AI models seem to be hitting a ceiling in capabilities, regardless of the amount of data or computing power thrown at them. A major breakthrough could help AI companies push through this ceiling, but it's also possible that LLMs just aren't capable of much more. Soaring demand for AI chips is driven by the idea that training a $1 billion or a $10 billion AI model makes financial sense. What if it doesn't? If AI models have largely topped out in terms of capabilities, the frantic multibillion-dollar AI investments being made by tech giants in an effort to not fall behind may never pay off in terms of revenue or profit. The hangover from this overinvestment could be brutal for companies like Nvidia as demand for AI chips dries up. Nvidia has delivered incredible returns to investors, and it's absolutely dominated the market for AI chips. But it's important to remember that trees don't grow to the sky. An AI breakthrough that blasts through the apparent LLM ceiling is certainly possible, but it's also possible that AI technology, like almost every new technology throughout history, has been overhyped to a degree. Artificial intelligence won't go away if AI models stop improving by leaps and bounds, but Nvidia's incredible growth and profits certainly will. With a market capitalization above $3 trillion, Nvidia stock looks like a risky proposition.