Curated by THEOUTPOST
On Wed, 20 Nov, 12:07 AM UTC
6 Sources
[1]
Nvidia rivals focus on building a different kind of chip to power AI products
SANTA CLARA, Calif. (AP) -- Building the current crop of artificial intelligence chatbots has relied on specialized computer chips pioneered by Nvidia, which cornered the market and made itself the poster child of the AI boom. But the same qualities that make those graphics processor chips, or GPUs, so effective at creating powerful AI systems from scratch make them less efficient at putting AI products to work. That's opened up the AI chip industry to rivals who think they can compete with Nvidia in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI. "These companies are seeing opportunity for that kind of specialized hardware," said Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology. "The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips." It takes a lot of computing power to make an AI chatbot. It starts with a process called training or pretraining -- the "P" in ChatGPT -- that involves AI systems "learning" from the patterns of huge troves of data. GPUs are good at doing that work because they can run many calculations at a time on a network of devices in communication with each other. However, once trained, a generative AI tool still needs chips to do the work -- such as when you ask a chatbot to compose a document or generate an image. That's where inferencing comes in. A trained AI model must take in new information and make inferences from what it already knows to produce a response. GPUs can do that work, too. But it can be a bit like taking a sledgehammer to crack a nut. "With training, you're doing a lot heavier, a lot more work. With inferencing, that's a lighter weight," said Forrester analyst Alvin Nguyen. That's led startups like Cerebras, Groq and d-Matrix as well as Nvidia's traditional chipmaking rivals -- such as AMD and Intel -- to pitch more inference-friendly chips as Nvidia focuses on meeting the huge demand from bigger tech companies for its higher-end hardware. D-Matrix, which is launching its first product this week, was founded in 2019 -- a bit late to the AI chip game, as CEO Sid Sheth explained during a recent interview at the company's headquarters in Santa Clara, California, the same Silicon Valley city that's also home to AMD, Intel and Nvidia. "There were already 100-plus companies. So when we went out there, the first reaction we got was 'you're too late,'" he said. The pandemic's arrival six months later didn't help as the tech industry pivoted to a focus on software to serve remote work. Now, however, Sheth sees a big market in AI inferencing, comparing that later stage of machine learning to how human beings apply the knowledge they acquired in school. "We spent the first 20 years of our lives going to school, educating ourselves. That's training, right?" he said. "And then the next 40 years of your life, you kind of go out there and apply that knowledge -- and then you get rewarded for being efficient." The product, called Corsair, consists of two chips with four chiplets each, made by Taiwan Semiconductor Manufacturing Company -- the same manufacturer of most of Nvidia's chips -- and packaged together in a way that helps to keep them cool. The chips are designed in Santa Clara, assembled in Taiwan and then tested back in California. Testing is a long process and can take six months -- if anything is off, it can be sent back to Taiwan. D-Matrix workers were doing final testing on the chips during a recent visit to a laboratory with blue metal desks covered with cables, motherboards and computers, with a cold server room next door. While tech giants like Amazon, Google, Meta and Microsoft have been gobbling up the supply of costly GPUs in a race to outdo each other in AI development, makers of AI inference chips are aiming for a broader clientele. Forrester's Nguyen said that could include Fortune 500 companies that want to make use of new generative AI technology without having to build their own AI infrastructure. Sheth said he expects a strong interest in AI video generation. "The dream of AI for a lot of these enterprise companies is you can use your own enterprise data," Nguyen said. "Buying (AI inference chips) should be cheaper than buying the ultimate GPUs from Nvidia and others. But I think there's going to be a learning curve in terms of integrating it." Feldgoise said that, unlike training-focused chips, AI inference work prioritizes how fast a person will get a chatbot's response. He said another whole set of companies is developing AI hardware for inference that can run not just in big data centers but locally on desktop computers, laptops and phones. Better-designed chips could bring down the huge costs of running AI to businesses. That could also affect the environmental and energy costs for everyone else. Sheth says the big concern right now is, "are we going to burn the planet down in our quest for what people call AGI -- human-like intelligence?" It's still fuzzy when AI might get to the point of artificial general intelligence -- predictions range from a few years to decades. But, Sheth notes, only a handful of tech giants are on that quest. "But then what about the rest?" he said. "They cannot be put on the same path." The other set of companies don't want to use very large AI models -- it's too costly and uses too much energy. "I don't know if people truly, really appreciate that inference is actually really going to be a much bigger opportunity than training. I don't think they appreciate that. It's still training that is really grabbing all the headlines," Sheth said.
[2]
Nvidia rivals focus on building a different kind of chip to power AI products
Rivals of Nvidia think they can compete with the chip giant in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI.Building the current crop of artificial intelligence chatbots has relied on specialized computer chips pioneered by Nvidia, which dominates the market and made itself the poster child of the AI boom. But the same qualities that make those graphics processor chips, or GPUs, so effective at creating powerful AI systems from scratch make them less efficient at putting AI products to work. That's opened up the AI chip industry to rivals who think they can compete with Nvidia in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI. "These companies are seeing opportunity for that kind of specialized hardware," said Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology. "The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips." What is AI inference? It takes a lot of computing power to make an AI chatbot. It starts with a process called training or pretraining - the "P" in ChatGPT - that involves AI systems "learning" from the patterns of huge troves of data. GPUs are good at doing that work because they can run many calculations at a time on a network of devices in communication with each other. However, once trained, a generative AI tool still needs chips to do the work - such as when you ask a chatbot to compose a document or generate an image. That's where inferencing comes in. A trained AI model must take in new information and make inferences from what it already knows to produce a response. GPUs can do that work, too. But it can be a bit like taking a sledgehammer to crack a nut. "With training, you're doing a lot heavier, a lot more work. With inferencing, that's a lighter weight," said Forrester analyst Alvin Nguyen. That's led startups like Cerebras, Groq and d-Matrix as well as Nvidia's traditional chipmaking rivals - such as AMD and Intel - to pitch more inference-friendly chips as Nvidia focuses on meeting the huge demand from bigger tech companies for its higher-end hardware. Inside an AI inference chip lab D-Matrix, which is launching its first product this week, was founded in 2019 - a bit late to the AI chip game, as CEO Sid Sheth explained during a recent interview at the company's headquarters in Santa Clara, California, the same Silicon Valley city that's also home to AMD, Intel and Nvidia. "There were already 100-plus companies. So when we went out there, the first reaction we got was 'you're too late,'" he said. The pandemic's arrival six months later didn't help as the tech industry pivoted to a focus on software to serve remote work. Now, however, Sheth sees a big market in AI inferencing, comparing that later stage of machine learning to how human beings apply the knowledge they acquired in school. "We spent the first 20 years of our lives going to school, educating ourselves. That's training, right?" he said. "And then the next 40 years of your life, you kind of go out there and apply that knowledge - and then you get rewarded for being efficient." The product, called Corsair, consists of two chips with four chiplets each, made by Taiwan Semiconductor Manufacturing Company - the same manufacturer of most of Nvidia's chips - and packaged together in a way that helps to keep them cool. The chips are designed in Santa Clara, assembled in Taiwan and then tested back in California. Testing is a long process and can take six months - if anything is off, it can be sent back to Taiwan. D-Matrix workers were doing final testing on the chips during a recent visit to a laboratory with blue metal desks covered with cables, motherboards and computers, with a cold server room next door. Who wants AI inference chips? While tech giants like Amazon, Google, Meta and Microsoft have been gobbling up the supply of costly GPUs in a race to outdo each other in AI development, makers of AI inference chips are aiming for a broader clientele. Forrester's Nguyen said that could include Fortune 500 companies that want to make use of new generative AI technology without having to build their own AI infrastructure. Sheth said he expects a strong interest in AI video generation. "The dream of AI for a lot of these enterprise companies is you can use your own enterprise data," Nguyen said. "Buying (AI inference chips) should be cheaper than buying the ultimate GPUs from Nvidia and others. But I think there's going to be a learning curve in terms of integrating it." Feldgoise said that, unlike training-focused chips, AI inference work prioritizes how fast a person will get a chatbot's response. He said another whole set of companies is developing AI hardware for inference that can run not just in big data centers but locally on desktop computers, laptops and phones. Why does this matter? Better-designed chips could bring down the huge costs of running AI to businesses. That could also affect the environmental and energy costs for everyone else. Sheth says the big concern right now is, "are we going to burn the planet down in our quest for what people call AGI - human-like intelligence?" It's still fuzzy when AI might get to the point of artificial general intelligence - predictions range from a few years to decades. But, Sheth notes, only a handful of tech giants are on that quest. "But then what about the rest?" he said. "They cannot be put on the same path." The other set of companies don't want to use very large AI models - it's too costly and uses too much energy. "I don't know if people truly, really appreciate that inference is actually really going to be a much bigger opportunity than training. I don't think they appreciate that. It's still training that is really grabbing all the headlines," Sheth said.
[3]
Nvidia rivals focus on building a different kind of chip to power AI products
SANTA CLARA, Calif. -- Building the current crop of artificial intelligence chatbots has relied on specialized computer chips pioneered by Nvidia, which cornered the market and made itself the poster child of the AI boom. But the same qualities that make those graphics processor chips, or GPUs, so effective at creating powerful AI systems from scratch make them less efficient at putting AI products to work. That's opened up the AI chip industry to rivals who think they can compete with Nvidia in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI. "These companies are seeing opportunity for that kind of specialized hardware," said Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology. "The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips." It takes a lot of computing power to make an AI chatbot. It starts with a process called training or pretraining -- the "P" in ChatGPT -- that involves AI systems "learning" from the patterns of huge troves of data. GPUs are good at doing that work because they can run many calculations at a time on a network of devices in communication with each other. However, once trained, a generative AI tool still needs chips to do the work -- such as when you ask a chatbot to compose a document or generate an image. That's where inferencing comes in. A trained AI model must take in new information and make inferences from what it already knows to produce a response. GPUs can do that work, too. But it can be a bit like taking a sledgehammer to crack a nut. "With training, you're doing a lot heavier, a lot more work. With inferencing, that's a lighter weight," said Forrester analyst Alvin Nguyen. That's led startups like Cerebras, Groq and d-Matrix as well as Nvidia's traditional chipmaking rivals -- such as AMD and Intel -- to pitch more inference-friendly chips as Nvidia focuses on meeting the huge demand from bigger tech companies for its higher-end hardware. D-Matrix, which is launching its first product this week, was founded in 2019 -- a bit late to the AI chip game, as CEO Sid Sheth explained during a recent interview at the company's headquarters in Santa Clara, California, the same Silicon Valley city that's also home to AMD, Intel and Nvidia. "There were already 100-plus companies. So when we went out there, the first reaction we got was 'you're too late,'" he said. The pandemic's arrival six months later didn't help as the tech industry pivoted to a focus on software to serve remote work. Now, however, Sheth sees a big market in AI inferencing, comparing that later stage of machine learning to how human beings apply the knowledge they acquired in school. "We spent the first 20 years of our lives going to school, educating ourselves. That's training, right?" he said. "And then the next 40 years of your life, you kind of go out there and apply that knowledge -- and then you get rewarded for being efficient." The product, called Corsair, consists of two chips with four chiplets each, made by Taiwan Semiconductor Manufacturing Company -- the same manufacturer of most of Nvidia's chips -- and packaged together in a way that helps to keep them cool. The chips are designed in Santa Clara, assembled in Taiwan and then tested back in California. Testing is a long process and can take six months -- if anything is off, it can be sent back to Taiwan. D-Matrix workers were doing final testing on the chips during a recent visit to a laboratory with blue metal desks covered with cables, motherboards and computers, with a cold server room next door. While tech giants like Amazon, Google, Meta and Microsoft have been gobbling up the supply of costly GPUs in a race to outdo each other in AI development, makers of AI inference chips are aiming for a broader clientele. Forrester's Nguyen said that could include Fortune 500 companies that want to make use of new generative AI technology without having to build their own AI infrastructure. Sheth said he expects a strong interest in AI video generation. "The dream of AI for a lot of these enterprise companies is you can use your own enterprise data," Nguyen said. "Buying (AI inference chips) should be cheaper than buying the ultimate GPUs from Nvidia and others. But I think there's going to be a learning curve in terms of integrating it." Feldgoise said that, unlike training-focused chips, AI inference work prioritizes how fast a person will get a chatbot's response. He said another whole set of companies is developing AI hardware for inference that can run not just in big data centers but locally on desktop computers, laptops and phones. Better-designed chips could bring down the huge costs of running AI to businesses. That could also affect the environmental and energy costs for everyone else. Sheth says the big concern right now is, "are we going to burn the planet down in our quest for what people call AGI -- human-like intelligence?" It's still fuzzy when AI might get to the point of artificial general intelligence -- predictions range from a few years to decades. But, Sheth notes, only a handful of tech giants are on that quest. "But then what about the rest?" he said. "They cannot be put on the same path." The other set of companies don't want to use very large AI models -- it's too costly and uses too much energy. "I don't know if people truly, really appreciate that inference is actually really going to be a much bigger opportunity than training. I don't think they appreciate that. It's still training that is really grabbing all the headlines," Sheth said.
[4]
Nvidia Rivals Focus on Building a Different Kind of Chip to Power AI Products
SANTA CLARA, Calif. (AP) -- Building the current crop of artificial intelligence chatbots has relied on specialized computer chips pioneered by Nvidia, which cornered the market and made itself the poster child of the AI boom. But the same qualities that make those graphics processor chips, or GPUs, so effective at creating powerful AI systems from scratch make them less efficient at putting AI products to work. That's opened up the AI chip industry to rivals who think they can compete with Nvidia in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI. "These companies are seeing opportunity for that kind of specialized hardware," said Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology. "The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips." What is AI inference? It takes a lot of computing power to make an AI chatbot. It starts with a process called training or pretraining -- the "P" in ChatGPT -- that involves AI systems "learning" from the patterns of huge troves of data. GPUs are good at doing that work because they can run many calculations at a time on a network of devices in communication with each other. However, once trained, a generative AI tool still needs chips to do the work -- such as when you ask a chatbot to compose a document or generate an image. That's where inferencing comes in. A trained AI model must take in new information and make inferences from what it already knows to produce a response. GPUs can do that work, too. But it can be a bit like taking a sledgehammer to crack a nut. "With training, you're doing a lot heavier, a lot more work. With inferencing, that's a lighter weight," said Forrester analyst Alvin Nguyen. That's led startups like Cerebras, Groq and d-Matrix as well as Nvidia's traditional chipmaking rivals -- such as AMD and Intel -- to pitch more inference-friendly chips as Nvidia focuses on meeting the huge demand from bigger tech companies for its higher-end hardware. Inside an AI inference chip lab D-Matrix, which is launching its first product this week, was founded in 2019 -- a bit late to the AI chip game, as CEO Sid Sheth explained during a recent interview at the company's headquarters in Santa Clara, California, the same Silicon Valley city that's also home to AMD, Intel and Nvidia. "There were already 100-plus companies. So when we went out there, the first reaction we got was 'you're too late,'" he said. The pandemic's arrival six months later didn't help as the tech industry pivoted to a focus on software to serve remote work. Now, however, Sheth sees a big market in AI inferencing, comparing that later stage of machine learning to how human beings apply the knowledge they acquired in school. "We spent the first 20 years of our lives going to school, educating ourselves. That's training, right?" he said. "And then the next 40 years of your life, you kind of go out there and apply that knowledge -- and then you get rewarded for being efficient." The product, called Corsair, consists of two chips with four chiplets each, made by Taiwan Semiconductor Manufacturing Company -- the same manufacturer of most of Nvidia's chips -- and packaged together in a way that helps to keep them cool. The chips are designed in Santa Clara, assembled in Taiwan and then tested back in California. Testing is a long process and can take six months -- if anything is off, it can be sent back to Taiwan. D-Matrix workers were doing final testing on the chips during a recent visit to a laboratory with blue metal desks covered with cables, motherboards and computers, with a cold server room next door. Who wants AI inference chips? While tech giants like Amazon, Google, Meta and Microsoft have been gobbling up the supply of costly GPUs in a race to outdo each other in AI development, makers of AI inference chips are aiming for a broader clientele. Forrester's Nguyen said that could include Fortune 500 companies that want to make use of new generative AI technology without having to build their own AI infrastructure. Sheth said he expects a strong interest in AI video generation. "The dream of AI for a lot of these enterprise companies is you can use your own enterprise data," Nguyen said. "Buying (AI inference chips) should be cheaper than buying the ultimate GPUs from Nvidia and others. But I think there's going to be a learning curve in terms of integrating it." Feldgoise said that, unlike training-focused chips, AI inference work prioritizes how fast a person will get a chatbot's response. He said another whole set of companies is developing AI hardware for inference that can run not just in big data centers but locally on desktop computers, laptops and phones. Why does this matter? Better-designed chips could bring down the huge costs of running AI to businesses. That could also affect the environmental and energy costs for everyone else. Sheth says the big concern right now is, "are we going to burn the planet down in our quest for what people call AGI -- human-like intelligence?" It's still fuzzy when AI might get to the point of artificial general intelligence -- predictions range from a few years to decades. But, Sheth notes, only a handful of tech giants are on that quest. "But then what about the rest?" he said. "They cannot be put on the same path." The other set of companies don't want to use very large AI models -- it's too costly and uses too much energy. "I don't know if people truly, really appreciate that inference is actually really going to be a much bigger opportunity than training. I don't think they appreciate that. It's still training that is really grabbing all the headlines," Sheth said. Copyright 2024 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.
[5]
Nvidia rivals focus on building a different kind of chip to power AI products
SANTA CLARA, Calif. (AP) -- Building the current crop of artificial intelligence chatbots has relied on specialized computer chips pioneered by Nvidia, which cornered the market and made itself the poster child of the AI boom. But the same qualities that make those graphics processor chips, or GPUs, so effective at creating powerful AI systems from scratch make them less efficient at putting AI products to work. That's opened up the AI chip industry to rivals who think they can compete with Nvidia in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI. "These companies are seeing opportunity for that kind of specialized hardware," said Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology. "The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips." What is AI inference? It takes a lot of computing power to make an AI chatbot. It starts with a process called training or pretraining -- the "P" in ChatGPT -- that involves AI systems "learning" from the patterns of huge troves of data. GPUs are good at doing that work because they can run many calculations at a time on a network of devices in communication with each other. However, once trained, a generative AI tool still needs chips to do the work -- such as when you ask a chatbot to compose a document or generate an image. That's where inferencing comes in. A trained AI model must take in new information and make inferences from what it already knows to produce a response. GPUs can do that work, too. But it can be a bit like taking a sledgehammer to crack a nut. "With training, you're doing a lot heavier, a lot more work. With inferencing, that's a lighter weight," said Forrester analyst Alvin Nguyen. That's led startups like Cerebras, Groq and d-Matrix as well as Nvidia's traditional chipmaking rivals -- such as AMD and Intel -- to pitch more inference-friendly chips as Nvidia focuses on meeting the huge demand from bigger tech companies for its higher-end hardware. Inside an AI inference chip lab D-Matrix, which is launching its first product this week, was founded in 2019 -- a bit late to the AI chip game, as CEO Sid Sheth explained during a recent interview at the company's headquarters in Santa Clara, California, the same Silicon Valley city that's also home to AMD, Intel and Nvidia. "There were already 100-plus companies. So when we went out there, the first reaction we got was 'you're too late,'" he said. The pandemic's arrival six months later didn't help as the tech industry pivoted to a focus on software to serve remote work. Now, however, Sheth sees a big market in AI inferencing, comparing that later stage of machine learning to how human beings apply the knowledge they acquired in school. "We spent the first 20 years of our lives going to school, educating ourselves. That's training, right?" he said. "And then the next 40 years of your life, you kind of go out there and apply that knowledge -- and then you get rewarded for being efficient." The product, called Corsair, consists of two chips with four chiplets each, made by Taiwan Semiconductor Manufacturing Company -- the same manufacturer of most of Nvidia's chips -- and packaged together in a way that helps to keep them cool. The chips are designed in Santa Clara, assembled in Taiwan and then tested back in California. Testing is a long process and can take six months -- if anything is off, it can be sent back to Taiwan. D-Matrix workers were doing final testing on the chips during a recent visit to a laboratory with blue metal desks covered with cables, motherboards and computers, with a cold server room next door. Who wants AI inference chips? While tech giants like Amazon, Google, Meta and Microsoft have been gobbling up the supply of costly GPUs in a race to outdo each other in AI development, makers of AI inference chips are aiming for a broader clientele. Forrester's Nguyen said that could include Fortune 500 companies that want to make use of new generative AI technology without having to build their own AI infrastructure. Sheth said he expects a strong interest in AI video generation. "The dream of AI for a lot of these enterprise companies is you can use your own enterprise data," Nguyen said. "Buying (AI inference chips) should be cheaper than buying the ultimate GPUs from Nvidia and others. But I think there's going to be a learning curve in terms of integrating it." Feldgoise said that, unlike training-focused chips, AI inference work prioritizes how fast a person will get a chatbot's response. He said another whole set of companies is developing AI hardware for inference that can run not just in big data centers but locally on desktop computers, laptops and phones. Why does this matter? Better-designed chips could bring down the huge costs of running AI to businesses. That could also affect the environmental and energy costs for everyone else. Sheth says the big concern right now is, "are we going to burn the planet down in our quest for what people call AGI -- human-like intelligence?" It's still fuzzy when AI might get to the point of artificial general intelligence -- predictions range from a few years to decades. But, Sheth notes, only a handful of tech giants are on that quest. "But then what about the rest?" he said. "They cannot be put on the same path." The other set of companies don't want to use very large AI models -- it's too costly and uses too much energy. "I don't know if people truly, really appreciate that inference is actually really going to be a much bigger opportunity than training. I don't think they appreciate that. It's still training that is really grabbing all the headlines," Sheth said.
[6]
Nvidia's rivals are focusing on building AI inference chips. Here's what to know
Building the current crop of artificial intelligence chatbots has relied on specialized computer chips pioneered by Nvidia, which dominates market and made itself the poster child of the AI boom. But the same qualities that make those graphics processor chips, or GPUs, so effective at creating powerful AI systems from scratch make them less efficient at putting AI products to work. That's opened up the AI chip industry to rivals who think they can compete with Nvidia in selling so-called AI inference chips that are more attuned to the day-to-day running of AI tools and designed to reduce some of the huge computing costs of generative AI. "These companies are seeing opportunity for that kind of specialized hardware," said Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology. "The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips."
Share
Share
Copy Link
As Nvidia dominates the AI training chip market with GPUs, competitors are focusing on developing specialized AI inference chips to meet the growing demand for efficient AI deployment and reduce computing costs.
As artificial intelligence (AI) continues to evolve, a new battleground is emerging in the chip industry. While Nvidia has dominated the market for AI training with its powerful GPUs, competitors are now focusing on developing specialized AI inference chips. These chips are designed to efficiently run AI models after they've been trained, potentially reducing the enormous computing costs associated with generative AI 1.
AI development involves two main stages: training and inference. Training, which is the "P" in ChatGPT, requires significant computing power to process vast amounts of data and create AI models. Nvidia's GPUs excel at this task due to their ability to perform multiple calculations simultaneously 2.
However, once an AI model is trained, it still needs chips to operate – this is where inference comes in. Inference involves the AI model taking in new information and making decisions based on its training. While GPUs can handle inference, they may be overqualified for the task, as Forrester analyst Alvin Nguyen explains: "With training, you're doing a lot heavier, a lot more work. With inferencing, that's a lighter weight" 3.
The growing adoption of AI models is creating a substantial demand for inference chips. Jacob Feldgoise, an analyst at Georgetown University's Center for Security and Emerging Technology, notes, "The broader the adoption of these models, the more compute will be needed for inference and the more demand there will be for inference chips" 1.
This opportunity has attracted both startups and established chipmakers. Companies like Cerebras, Groq, and d-Matrix, along with Nvidia's traditional rivals AMD and Intel, are developing inference-friendly chips to compete in this emerging market 4.
One company making waves in the AI inference chip space is d-Matrix. Founded in 2019, the company is launching its first product, Corsair, this week. CEO Sid Sheth sees a significant market in AI inferencing, comparing it to how humans apply knowledge acquired in school throughout their lives 5.
The Corsair chip, manufactured by Taiwan Semiconductor Manufacturing Company, consists of two chips with four chiplets each, designed to optimize cooling and efficiency. D-Matrix's approach highlights the specialized nature of inference chips compared to general-purpose GPUs 5.
While tech giants like Amazon, Google, Meta, and Microsoft are the primary consumers of high-end GPUs for AI development, inference chip makers are targeting a broader market. Forrester's Nguyen suggests that Fortune 500 companies looking to implement generative AI without building extensive infrastructure could be potential customers 3.
The development of efficient inference chips could have far-reaching implications. Better-designed chips could significantly reduce the costs of running AI for businesses and potentially mitigate the environmental and energy impacts of AI deployment 4.
As the AI chip market evolves, some industry insiders, including d-Matrix's Sheth, believe that inference could become a more significant opportunity than training. However, this potential shift is not yet widely recognized, as training continues to dominate headlines 5.
The race to develop efficient AI inference chips represents a new frontier in the AI industry, potentially reshaping the landscape of AI deployment and challenging Nvidia's current dominance in the AI chip market.
Reference
[4]
U.S. News & World Report
|Nvidia Rivals Focus on Building a Different Kind of Chip to Power AI Products[5]
Nvidia's remarkable growth in the AI chip market faces potential hurdles as the industry grapples with diminishing returns from traditional scaling methods, prompting a shift towards new approaches like test-time scaling.
4 Sources
4 Sources
Huawei is making strategic moves to capture a larger share of China's AI chip market, currently dominated by Nvidia. The company is focusing on inference tasks and helping local firms adapt Nvidia-trained AI models to run on Huawei's Ascend chips.
2 Sources
2 Sources
At CES 2025, Nvidia CEO Jensen Huang introduced the concept of "Agentic AI," forecasting a multi-trillion dollar shift in work and industry. The company unveiled new AI technologies, GPUs, and partnerships, positioning Nvidia at the forefront of the AI revolution.
37 Sources
37 Sources
As Nvidia's stock surges due to AI chip demand, experts warn of potential slowdown. Meanwhile, tech giants like Apple and Google develop in-house AI chips, challenging Nvidia's market position.
3 Sources
3 Sources
Intel launches Tiber AI Cloud, powered by Gaudi 3 chips, partnering with Inflection AI to offer enterprise AI solutions, competing with major cloud providers and NVIDIA in the AI accelerator market.
4 Sources
4 Sources
The Outpost is a comprehensive collection of curated artificial intelligence software tools that cater to the needs of small business owners, bloggers, artists, musicians, entrepreneurs, marketers, writers, and researchers.
© 2025 TheOutpost.AI All rights reserved