Google's TPU Chips Challenge Nvidia's AI Dominance as Industry Shifts Toward Specialized Processors

Why Google's custom AI chips are shaking up the tech industry

Nvidia's position as the dominant supplier of AI chips may be under threat from a specialised chip pioneered by Google, with reports suggesting companies like Meta and Anthropic are looking to spend billions on Google's tensor processing units. The success of the artificial intelligence industry has been in large part based on graphical processing units (GPUs), a kind of computer chip that can perform many parallel calculations at the same time, rather than one after the other like the computer processing units (CPUs) that power most computers. GPUs were originally developed to assist with computer graphics, as the name suggests, and gaming. "If I have a lot of pixels in a space and I need to do a rotation of this to calculate a new camera view, this is an operation that can be done in parallel, for many different pixels," says Francesco Conti at the University of Bologna in Italy. This ability to do calculations in parallel happened to be useful for training and running AI models, which often use calculations involving vast grids of numbers performed at the same time, called matrix multiplication. "GPUs are a very general architecture, but they are extremely suited to applications that show a high degree of parallelism," says Conti. However, because they weren't originally designed with AI in mind, there can be inefficiencies in the ways that GPUs translate the calculations that are performed on the chips. Tensor processing units (TPUs), which were originally developed by Google in 2016, are instead designed solely around matrix multiplication, says Conti, which are the main calculations needed for training and running large AI models. This year, Google released the seventh generation of its TPU, called Ironwood, which powers many of the company's AI models like Gemini and protein-modelling AlphaFold. Technologically, TPUs are more of a subset of GPUs than an entirely different chip, says Simon McIntosh-Smith at the University of Bristol, UK. "They focus on the bits that GPUs do more specifically aimed at training and inference for AI, but actually they're in some ways more similar to GPUs than you might think." But because TPUs are designed with certain AI applications in mind, they can be much more efficient for these jobs and save potentially tens or hundreds of millions of dollars, he says. However, this specialisation also has its disadvantages and can make TPUs inflexible if the AI models change significantly between generations, says Conti. "If you don't have the flexibility on your [TPU], you have to do [calculations] on the CPU of your node in the data centre, and this will slow you down immensely," says Conti. One advantage that Nvidia GPUs have traditionally held is that there is simple software available that can help AI designers run their code on Nvidia chips. This didn't exist in the same way for TPUs when they first came about, but the chips are now at a stage where they are more straightforward to use, says Conti. "With the TPU, you can now do the same [as GPUs]," he says. "Now that you have enabled that, it's clear that the availability becomes a major factor." Although Google first launched the TPU, many of the largest AI companies (known as hyperscalers), as well as smaller start-ups, have now started developing their own specialised TPUs, including Amazon, which uses its own Trainium chips to train its AI models. "Most of the hyperscalers have their own internal programmes, and that's partly because GPUs got so expensive because the demand was outstripping supply, and it might be cheaper to design and build your own," says McIntosh-Smith. Google has been developing its TPUs for over a decade, but it has mostly been using these chips for its own AI models. What appears to be changing now is that other large companies, like Meta and Anthropic, are making sizeable purchases of computing power from Google's TPUs. "What we haven't heard about is big customers switching, and maybe that's what's starting to happen now," says McIntosh-Smith. "They've matured enough and there's enough of them." As well as creating more choice for the large companies, it could make good financial sense for them to diversify, he says. "It might even be that that means you get a better deal from Nvidia in the future," he says.

Could a new generation of dedicated AI chips burst Nvidia's bubble and do for AI GPUs what ASICs did for crypto mining?

A Chinese startup founded by a former Google engineer claims to have created a new ultra-efficient and relatively low cost AI chip using older manufacturing techniques. Meanwhile, Google itself is now reportedly considering whether to make its own specialised AI chips available to buy. Together, these chips could represent the start of a new processing paradigm which could do for the AI industry what ASICs did for bitcoin mining. In both cases, we're talking about chips broadly known as TPUs or tensor processing units. They're a kind of ASIC, or application-specific integrated circuit, that's designed at hardware level to do a very narrow set of AI-related tasks very well. By contrast, Nvidia's GPUs, even its AI-specific GPUs as opposed to its gaming GPUs, are more general purpose in nature. The parallel here is bitcoin mining, which used to be done on graphics cards before a new generation of dedicated mining chips, also ASICs, came along and did the same job much more efficiently. Could the same transition from more general purpose GPUs to ASICs hit the AI industry? The Chinese startup is Zhonghao Xinying. Its Ghana chip is claimed to offer 1.5 times the performance of Nvidia's A100 AI GPU while reducing power consumption by 75%. And it does that courtesy of a domestic Chinese chip manufacturing process that the company says is "an order of magnitude lower than that of leading overseas GPU chips." By "an order or magnitude lower," the assumption is that means well behind in technological terms given China's home-grown chip manufacturing is probably a couple of generations behind the best that TSMC in Taiwan can offer and behind even what the likes of Intel and Samsung can offer, too. While the A100 is an old AI GPU dating from 2020 and thus much slower than Nvidia's latest Blackwell GPUs, it's the efficiency and cost gains of the new Chinese ASIC that will be most appealing to the AI industry. As for Google, it has been making TPUs since 2017. But the more recent explosion in demand for AI hardware is reportedly enough to make Google consider selling TPUs to customers as opposed to merely hiring out access. According to a report on The Information (via Yahoo Finance), Google is in talks with with several customers, including a possible multi-billion-dollar deal with Meta, and has ambitions to "capture" 10% of Nvidia's AI revenue. Google's TPUs are also more narrowly defined and likely more power efficient than Nvidia's AI GPUs. Of course, there are plenty of barriers to the adoption of these new AI ASICs. Much of the AI industry is heavily invested in Nvidia hardware and software tools. Moving to a totally new ASIC-based platform would mean plenty of short term pain. But what with Nvidia said to be charging $45,000 to $50,000 per B200 GPU, companies arguably have every reason to suck up that pain for long term gains. Moreover, if more efficient ASICs do make inroads into the AI market, that could reduce demand for the most advanced manufacturing processes. And that would be good news for all kinds of chips, including GPUs for PC gaming. Right now, we're all effectively paying more for our graphics cards because of the incredible demand for large, general purpose AI GPUs built on the same silicon as gaming GPUs. If at least a chunk of the AI market shifted to this new ASIC paradigm, well, the whole supply-and-demand equation would shift. Heck, maybe even Nvidia would view gaming as important, again. We can but hope!