The Evolving Landscape of AI Infrastructure: From Cloud to Custom Solutions

Cloud, edge or on-prem? Navigating the new AI infrastructure paradigm

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More No doubt, enterprise data infrastructure continues to transform with technological innovation -- most notably today due to data-and-resource hungry generative AI. As gen AI changes the enterprise itself, leaders continue to grapple with the cloud/edge/on-prem question. On the one hand, they need near-instant access to data; on the other, they need to know that that data is protected. As they face this conundrum, more and more enterprises are seeing hybrid models as the way forward, as they can exploit the different advantages of what cloud, edge and on-prem models have to offer. Case in point: 85% of cloud buyers are either deployed or in the process of deploying a hybrid cloud, according to IDC. "The pendulum between the edge and the cloud and all the hybrid flavors in between has kept shifting over the past decade," Priyanka Tembey, co-founder and CTO at runtime application security company Operant, told VentureBeat. "There are quite a few use cases coming up where compute can benefit from running closer to the edge, or as a combination of edge plus cloud in a hybrid manner." The shifting data infrastructure pendulum For a long time, cloud was associated with hyperscale data centers -- but that is no longer the case, explained Dave McCarthy, research VP and global research lead for IDC's cloud and edge services. "Organizations are realizing that the cloud is an operating model that can be deployed anywhere," he said. "Cloud has been around long enough that it is time for customers to rethink their architectures," he said. "This is opening the door for new ways of leveraging hybrid cloud and edge computing to maximize the value of AI." AI, notably, is driving the shift to hybrid cloud and edge because models need more and more computational power as well as access to large datasets, noted Miguel Leon, senior director at app modernization company WinWire. "The combination of hybrid cloud, edge computing and AI is changing the tech landscape in a big way," he told VentureBeat. "As AI continues to evolve and becomes a de facto embedded technology to all businesses, its ties with hybrid cloud and edge computing will only get deeper and deeper." Edge addresses issues cloud can't alone According to IDC research, spending on edge is expected to reach $232 billion this year. This growth can be attributed to several factors, McCarthy noted -- each of which addresses a problem that cloud computing can't solve alone. One of the most significant is latency-sensitive applications. "Whether introduced by the network or the number of hops between the endpoint and server, latency represents a delay," McCarthy explained. For instance, vision-based quality inspection systems used in manufacturing require real-time response to activity on a production line. "This is a situation where milliseconds matter, necessitating a local, edge-based system," he said. "Edge computing processes data closer to where it's generated, reducing latency and making businesses more agile," Leon agreed. It also supports AI apps that need fast data processing for tasks like image recognition and predictive maintenance. Edge is beneficial for limited connectivity environments, as well, such as internet of things (IoT) devices that may be mobile and move in and out of coverage areas or experience limited bandwidth, McCarthy noted. In certain cases -- autonomous vehicles, for one -- AI must be operational even if a network is unavailable. Another issue that spans all computing environments is data -- and lots of it. According to the latest estimates, approximately 328.77 million terabytes of data are generated every day. By 2025, the volume of data is expected to increase to more than 170 zettabytes, representing a more than 145-fold increase in 15 years. As data in remote locations continues to increase, costs associated with transmitting it to a central data store also continue to grow, McCarthy pointed out. However, in the case of predictive AI, most inference data does not need to be stored long-term. "An edge computing system can determine what data is necessary to keep," he said. Also, whether due to government regulation or corporate governance, there can be restrictions to where data can reside, McCarthy noted. As governments continue to pursue data sovereignty legislation, businesses are increasingly challenged with compliance. This can occur when cloud or data center infrastructure is located outside a local jurisdiction. Edge can come in handy here, as well, With AI initiatives quickly moving from proof-of-concept trials to production deployments, scalability has become another big issue. "The influx of data can overwhelm core infrastructure," said McCarthy. He explained that, in the early days of the internet, content delivery networks (CDNs) were created to cache content closer to users. "Edge computing will do the same for AI," he said. Benefits and uses of hybrid models Different cloud environments have different benefits, of course. For example, McCarthy noted, that auto-scaling to meet peak usage demands is "perfect" for public cloud. Meanwhile, on-premises data centers and private cloud environments can help secure and provide better control over proprietary data. The edge, for its part, provides resiliency and performance in the field. Each plays its part in an enterprise's overall architecture. "The benefit of a hybrid cloud is that it allows you to choose the right tool for the job," said McCarthy. He pointed to numerous use cases for hybrid models: For instance, in financial services, mainframe systems can be integrated with cloud environments so that institutions can maintain their own data centers for banking operations while leveraging the cloud for web and mobile-based customer access. Meanwhile, in retail, local in-store systems can continue to process point-of-sale transactions and inventory management independently of the cloud should an outage occur. "This will become even more important as these retailers roll out AI systems to track customer behavior and prevent shrinkage," said McCarthy. Tembey also pointed out that a hybrid approach with a combination of AI that runs locally on a device, at the edge and in larger private or public models using strict isolation techniques can preserve sensitive data. Not to say that there aren't downsides -- McCarthy pointed out that, for instance, hybrid can increase management complexity, especially in mixed vendor environments. "That is one reason why cloud providers have been extending their platforms to both on-prem and edge locations," he said, adding that original equipment manufacturers (OEMs) and independent software vendors (ISVs) have also increasingly been integrating with cloud providers. Interestingly, at the same time, 80% of respondents to an IDC survey indicated that they either have or plan to move some public cloud resources back on-prem. "For a while, cloud providers tried to convince customers that on-premises data centers would go away and everything would run in the hyperscale cloud," McCarthy noted. "That has proven not to be the case."

AI is changing enterprise computing -- and the enterprise itself

It's hard to think of any enterprise technology having a greater impact on business today than artificial intelligence (AI), with use cases including automating processes, customizing user experiences, and gaining insights from massive amounts of data. As a result, there is a realization that AI has become a core differentiator that needs to be built into every organization's strategy. Some were surprised when Google announced in 2016 that they would be a mobile-first company, recognizing that mobile devices had become the dominant user platform. Today, some companies call themselves 'AI first,' acknowledging that their networking and infrastructure must be engineered to support AI above all else. Failing to address the challenges of supporting AI workloads has become a significant business risk, with laggards set to be left trailing AI-first competitors who are using AI to drive growth and speed towards a leadership position in the marketplace. However, adopting AI has pros and cons. AI-based applications create a platform for businesses to drive revenue and market share, for example by enabling efficiency and productivity improvements through automation. But the transformation can be difficult to achieve. AI workloads require massive processing power and significant storage capacity, putting strain on already complex and stretched enterprise computing infrastructures. In addition to centralized data center resources, most AI deployments have multiple touchpoints across user devices including desktops, laptops, phones and tablets. AI is increasingly being used on edge and endpoint devices, enabling data to be collected and analyzed close to the source, for greater processing speed and reliability. For IT teams, a large part of the AI discussion is about infrastructure cost and location. Do they have enough processing power and data storage? Are their AI solutions located where they run best -- at on-premises data centers or, increasingly, in the cloud or at the edge? How enterprises can succeed at AI If you want to become an AI-first organization, then one of the biggest challenges is building the specialized infrastructure that this requires. Few organizations have the time or money to build massive new data centers to support power-hungry AI applications. The reality for most businesses is that they will have to determine a way to adapt and modernize their data centers to support an AI-first mentality. But where do you start? In the early days of cloud computing, cloud service providers (CSPs) offered simple, scalable compute and storage -- CSPs were considered a simple deployment path for undifferentiated business workloads. Today, the landscape is dramatically different, with new AI-centric CSPs offering cloud solutions specifically designed for AI workloads and, increasingly, hybrid AI setups that span on-premises IT and cloud services. AI is a complex proposition and there's no one-size-fits-all solution. It can be difficult to know what to do. For many organizations, help comes from their strategic technology partners who understand AI and can advise them on how to create and deliver AI applications that meet their specific objectives -- and will help them grow their businesses. With data centers, often a significant part of an AI application, a key element of any strategic partner's role is enabling data center modernization. One example is the rise in servers and processors specifically designed for AI. By adopting specific AI-focused data center technologies, it's possible to deliver significantly more compute power through fewer processors, servers, and racks, enabling you to reduce the data center footprint required by your AI applications. This can increase energy efficiency and also reduce the total cost of investment (TCO) for your AI projects. A strategic partner can also advise you on graphics processing unit (GPU) platforms. GPU efficiency is key to AI success, particularly for training AI models, real-time processing or decision-making. Simply adding GPUs won't overcome processing bottlenecks. With a well implemented, AI-specific GPU platform, you can optimize for the specific AI projects you need to run and spend only on the resources this requires. This improves your return on investment (ROI), as well as the cost-effectiveness (and energy efficiency) of your data center resources. Similarly, a good partner can help you identify which AI workloads truly require GPU-acceleration, and which have greater cost effectiveness when running on CPU-only infrastructure. For example, AI Inference workloads are best deployed on CPUs when model sizes are smaller or when AI is a smaller percentage of the overall server workload mix. This is an important consideration when planning an AI strategy because GPU accelerators, while often critical for training and large model deployment, can be costly to obtain and operate. Data center networking is also critical for delivering the scale of processing that AI applications require. An experienced technology partner can give you advice about networking options at all levels (including rack, pod and campus) as well as helping you to understand the balance and trade-off between different proprietary and industry-standard technologies. What to look for in your partnerships Your strategic partner for your journey to an AI-first infrastructure must combine expertise with an advanced portfolio of AI solutions designed for the cloud and on-premises data centers, user devices, edge and endpoints. AMD, for example, is helping organizations to leverage AI in their existing data centers. AMD EPYC(TM) processors can drive rack-level consolidation, enabling enterprises to run the same workloads on fewer servers, CPU AI performance for small and mixed AI workloads, and improved GPU performance, supporting advanced GPU accelerators and minimize computing bottlenecks. Through consolidation with AMD EPYC™ processors data center space and power can be freed to enable deployment of AI-specialized servers. The increase in demand for AI application support across the business is putting pressure on aging infrastructure. To deliver secure and reliable AI-first solutions, it's important to have the right technology across your IT landscape, from data center through to user and endpoint devices. Enterprises should lean into new data center and server technologies to enable them to speed up their adoption of AI. They can reduce the risks through innovative yet proven technology and expertise. And with more organizations embracing an AI-first mindset, the time to get started on this journey is now. Learn more about AMD.

From cost center to competitive edge: The strategic value of custom AI Infrastructure

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More AI is no longer just a buzzword -- it's a business imperative. As enterprises across industries continue to adopt AI, the conversation around AI infrastructure has evolved dramatically. Once viewed as a necessary but costly investment, custom AI infrastructure is now seen as a strategic asset that can provide a critical competitive edge. Mike Gualtieri, vice president and principal analyst at Forrester, emphasizes the strategic importance of AI infrastructure. "Enterprises must invest in an enterprise AI/ML platform from a vendor that at least keeps pace with, and ideally pushes the envelope of, enterprise AI technology," Gualtieri said. "The technology must also serve a reimagined enterprise operating in a world of abundant intelligence." This perspective underscores the shift from viewing AI as a peripheral experiment to recognizing it as a core component of future business strategy. The infrastructure revolution The AI revolution has been fueled by breakthroughs in AI models and applications, but those innovations have also created new challenges. Today's AI workloads, especially around training and inference for large language models (LLMs), require unprecedented levels of computing power. This is where custom AI infrastructure comes into play. "AI infrastructure is not one-size-fits-all," says Gualtieri. "There are three key workloads: data preparation, model training and inference." Each of these tasks has different infrastructure requirements, and getting it wrong can be costly, according to Gualtieri. For example, while data preparation often relies on traditional computing resources, training massive AI models like GPT-4o or LLaMA 3.1 necessitates specialized chips such as Nvidia's GPUs, Amazon's Trainium or Google's TPUs. Nvidia, in particular, has taken the lead in AI infrastructure, thanks to its GPU dominance. "Nvidia's success wasn't planned, but it was well-earned," Gualtieri explains. "They were in the right place at the right time, and once they saw the potential of GPUs for AI, they doubled down." However, Gualtieri believes that competition is on the horizon, with companies like Intel and AMD looking to close the gap. The cost of the cloud Cloud computing has been a key enabler of AI, but as workloads scale, the costs associated with cloud services have become a point of concern for enterprises. According to Gualtieri, cloud services are ideal for "bursting workloads" -- short-term, high-intensity tasks. However, for enterprises running AI models 24/7, the pay-as-you-go cloud model can become prohibitively expensive. "Some enterprises are realizing they need a hybrid approach," Gualtieri said. "They might use the cloud for certain tasks but invest in on-premises infrastructure for others. It's about balancing flexibility and cost-efficiency." This sentiment was echoed by Ankur Mehrotra, general manager of Amazon SageMaker at AWS. In a recent interview, Mehrotra noted that AWS customers are increasingly looking for solutions that combine the flexibility of the cloud with the control and cost-efficiency of on-premise infrastructure. "What we're hearing from our customers is that they want purpose-built capabilities for AI at scale," Mehrotra explains. "Price performance is critical, and you can't optimize for it with generic solutions." To meet these demands, AWS has been enhancing its SageMaker service, which offers managed AI infrastructure and integration with popular open-source tools like Kubernetes and PyTorch. "We want to give customers the best of both worlds," says Mehrotra. "They get the flexibility and scalability of Kubernetes, but with the performance and resilience of our managed infrastructure." The role of open source Open-source tools like PyTorch and TensorFlow have become foundational to AI development, and their role in building custom AI infrastructure cannot be overlooked. Mehrotra underscores the importance of supporting these frameworks while providing the underlying infrastructure needed to scale. "Open-source tools are table stakes," he says. "But if you just give customers the framework without managing the infrastructure, it leads to a lot of undifferentiated heavy lifting." AWS's strategy is to provide a customizable infrastructure that works seamlessly with open-source frameworks while minimizing the operational burden on customers. "We don't want our customers spending time on managing infrastructure. We want them focused on building models," says Mehrotra. Gualtieri agrees, adding that while open-source frameworks are critical, they must be backed by robust infrastructure. "The open-source community has done amazing things for AI, but at the end of the day, you need hardware that can handle the scale and complexity of modern AI workloads," he says. The future of AI infrastructure As enterprises continue to navigate the AI landscape, the demand for scalable, efficient and custom AI infrastructure will only grow. This is especially true as artificial general intelligence (AGI) -- or agentic AI -- becomes a reality. "AGI will fundamentally change the game," Gualtieri said. "It's not just about training models and making predictions anymore. Agentic AI will control entire processes, and that will require a lot more infrastructure." Mehrotra also sees the future of AI infrastructure evolving rapidly. "The pace of innovation in AI is staggering," he says. "We're seeing the emergence of industry-specific models, like BloombergGPT for financial services. As these niche models become more common, the need for custom infrastructure will grow." AWS, Nvidia and other major players are racing to meet this demand by offering more customizable solutions. But as Gualtieri points out, it's not just about the technology. "It's also about partnerships," he says. "Enterprises can't do this alone. They need to work closely with vendors to ensure their infrastructure is optimized for their specific needs." Custom AI infrastructure is no longer just a cost center -- it's a strategic investment that can provide a significant competitive edge. As enterprises scale their AI ambitions, they must carefully consider their infrastructure choices to ensure they are not only meeting today's demands but also preparing for the future. Whether through cloud, on-premises, or hybrid solutions, the right infrastructure can make all the difference in turning AI from an experiment into a business driver

Going beyond GPUs: The evolving landscape of AI chips and accelerators

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Data centers are the backend of the internet we know. Whether it's Netflix or Google, all major companies leverage data centers, and the computer systems they host, to deliver digital services to end users. As the focus of enterprises shifts toward advanced AI workloads, data centers' traditional CPU-centric servers are being buffed with the integration of new specialized chips or "co-processors." At the core, the idea behind these co-processors is to introduce an add-on of sorts to enhance the computing capacity of the servers. This enables them to handle the calculational demands of workloads like AI training, inference, database acceleration and network functions. Over the last few years, GPUs, led by Nvidia, have been the go-to choice for co-processors due to their ability to process large volumes of data at unmatched speeds. Due to increased demand GPUs accounted for 74% of the co-processors powering AI use cases within data centers last year, according to a study from Futurum Group. According to the study, the dominance of GPUs is only expected to grow, with revenues from the category surging 30% annually to $102 billion by 2028. But, here's the thing: while GPUs, with their parallel processing architecture, make a strong companion for accelerating all sorts of large-scale AI workloads (like training and running massive, trillion parameter language models or genome sequencing), their total cost of ownership can be very high. For example, Nvidia's flagship GB200 "superchip", which combines a Grace CPU with two B200 GPUs, is expected to cost between $60,000 and $70,000. A server with 36 of these superchips is estimated to cost around $2 million. While this may work in some cases, like large-scale projects, it is not for every company. Many enterprise IT managers are looking to incorporate new technology to support select low- to medium-intensive AI workloads with a specific focus on total cost of ownership, scalability and integration. After all, most AI models (deep learning networks, neural networks, large language models etc) are in the maturing stage and the needs are shifting towards AI inferencing and enhancing the performance for specific workloads like image recognition, recommender systems or object identification -- while being efficient at the same time. This is exactly where the emerging landscape of specialized AI processors and accelerators, being built by chipmakers, startups and cloud providers, comes in. What exactly are AI processors and accelerators? At the core, AI processors and accelerators are chips that sit within servers' CPU ecosystem and focus on specific AI functions. They commonly revolve around three key architectures: Application-Specific Integrated Circuited (ASICs), Field-Programmable Gate Arrays (FPGAs), and the most recent innovation of Neural Processing Units (NPUs). The ASICs and FPGAs have been around for quite some time, with programmability being the only difference between the two. ASICs are custom-built from the ground up for a specific task (which may or may not be AI-related), while FPGAs can be reconfigured at a later stage to implement custom logic. NPUs, on their part, differentiate from both by serving as the specialized hardware that can only accelerate AI/ML workloads like neural network inference and training. "Accelerators tend to be capable of doing any function individually, and sometimes with wafer-scale or multi-chip ASIC design, they can be capable of handling a few different applications. NPUs are a good example of a specialized chip (usually part of a system) that can handle a number of matrix-math and neural network use cases as well as various inference tasks using less power," Futurum group CEO Daniel Newman tells Venturebeat. The best part is that accelerators, especially ASICs and NPUs built for specific applications, can prove more efficient than GPUs in terms of cost and power use. "GPU designs mostly center on Arithmetic Logic Units (ALUs) so that they can perform thousands of calculations simultaneously, whereas AI accelerator designs mostly center on Tensor Processor Cores (TPCs) or Units. In general, the AI accelerators' performance versus GPUs performance is based on the fixed function of that design," Rohit Badlaney, the general manager for IBM's cloud and industry platforms, tells VentureBeat. Currently, IBM follows a hybrid cloud approach and uses multiple GPUs and AI accelerators, including offerings from Nvidia and Intel, across its stack to provide enterprises with choices to meet the needs of their unique workloads and applications -- with high performance and efficiency. "Our full-stack solutions are designed to help transform how enterprises, developers and the open-source community build and leverage generative AI. AI accelerators are one of the offerings that we see as very beneficial to clients looking to deploy generative AI," Badlaney said. He added while GPU systems are best suited for large model training and fine-tuning, there are many AI tasks that accelerators can handle equally well - and at a lesser cost. For instance, IBM Cloud virtual servers use Intel's Gaudi 3 accelerator with a custom software stack designed specifically for inferencing and heavy memory demands. The company also plans to use the accelerator for fine-tuning and small training workloads via small clusters of multiple systems. "AI accelerators and GPUs can be used effectively for some similar workloads, such as LLMs and diffusion models (image generation like Stable Diffusion) to standard object recognition, classification, and voice dubbing. However, the benefits and differences between AI accelerators and GPUs entirely depend on the hardware provider's design. For instance, the Gaudi 3 AI accelerator was designed to provide significant boosts in compute, memory bandwidth, and architecture-based power efficiency," Badlaney explained. This, he said, directly translates to price-performance benefits. Beyond Intel, other AI accelerators are also drawing attention in the market. This includes not only custom chips built for and by public cloud providers such as Google, AWS and Microsoft but also dedicated products (NPUs in some cases) from startups such as Groq, Graphcore, SambaNova Systems and Cerebras Systems. They all stand out in their own way, challenging GPUs in different areas. In one case, Tractable, a company developing AI to analyze damage to property and vehicles for insurance claims, was able to leverage Graphcore's Intelligent Processing Unit-POD system (a specialized NPU offering) for significant performance gains compared to GPUs they had been using. "We saw a roughly 5X speed gain," Razvan Ranca, co-founder and CTO at Tractable, wrote in a blog post. "That means a researcher can now run potentially five times more experiments, which means we accelerate the whole research and development process and ultimately end up with better models in our products." AI processors are also powering training workloads in some cases. For instance, the AI supercomputer at Aleph Alpha's data center is using Cerebras CS-3, the system powered by the startup's third-generation Wafer Scale Engine with 900,000 AI cores, to build next-gen sovereign AI models. Even Google's recently introduced custom ASIC, TPU v5p, is driving some AI training workloads for companies like Salesforce and Lightricks. What should be the approach to picking accelerators? Now that it's established there are many AI processors beyond GPUs to accelerate AI workloads, especially inference, the question is: how does an IT manager pick the best option to invest in? Some of these chips may deliver good performance with efficiencies but might be limited in terms of the kind of AI tasks they could handle due to their architecture. Others may do more but the TCO difference might not be as massive when compared to GPUs. Since the answer varies with the design of the chips, all experts VentureBeat spoke to suggested the selection should be based upon the scale and type of the workload to be processed, the data, the likelihood of continued iteration/change and cost and availability needs. According to Daniel Kearney, the CTO at Sustainable Metal Cloud, which helps companies with AI training and inference, it is also important for enterprises to run benchmarks to test for price-performance benefits and ensure that their teams are familiar with the broader software ecosystem that supports the respective AI accelerators. "While detailed workload information may not be readily in advance or may be inconclusive to support decision-making, it is recommended to benchmark and test through with representative workloads, real-world testing and available peer-reviewed real-world information where available to provide a data-driven approach to choosing the right AI accelerator for the right workload. This upfront investigation can save significant time and money, particularly for large and costly training jobs," he suggested. Globally, with inference jobs on track to grow, the total market of AI hardware, including AI chips, accelerators and GPUs, is estimated to grow 30% annually to touch $138 billion by 2028.